im_file configuration

Post a different question
8
responses

i defined an input for im_file that is being sent as syslog

i have multiple files in a folder what i want to do is read each file and on EOF copy to another folder.

didn`t find the option to identify EOF

AskedDecember 18, 2014 - 4:22pm

Answer (1)

This is not possible but there are plans to implement support for that.

im_file was designed with the assumption to read files that are constantly appended to. In this case reaching EOF does not mean there won't be any more data appended.

AnsweredDecember 18, 2014 - 4:34pm

Comments (7)

  • guyl's picture

    i understand but the issue is that for some reason the logs always rotates and i am getting the same results.

    **file getting read from the start every time.

    December 19, 2014 - 9:41am
  • guyl's picture

    let me describe the situation:

    I have a folder containing logs

    1.log 2.log 3.log

    I defined the im_file as

    *.log from that folder and collection is working the problem is that it never stops (collecting same raws)

    checking the logs shows information about "1.log has been rotated...etc....reading from start" (same for other logs)

    Eventually even if the logs are static It just keep reading them in a loop

     

    saving last POS doesn't help also defining logs Identification by size and inode didn’t work.

     

    Thanks for helping, i hope that we can solve this one :)

    December 19, 2014 - 6:13pm
  • adm's picture
    (NXLog)

    Ok, but that still does not explain how the logs are rotated.

    Is this your rotation strategy: 1.log => 2.log, 2.log => 3.log, create 1.log ?

    Also you did not show your conf, that could reveal some other issues.

    December 19, 2014 - 6:35pm
  • guyl's picture

    I will send the config on sunday 

    my output actins is im_file => syslog

    the logs are just statis logs it is more easy to understand 

    ERROR20\12.log

    ERROR19\12.log

    ERROR18\12.log

    the big problem is that i don`t do any reotation but in the nxlog for some reason identify that the logs changes and reads it again

    December 20, 2014 - 1:59pm
  • guyl's picture

    Configuration:

    define ROOT C:\Program Files (x86)\nxlog

    Moduledir %ROOT%\modules
    CacheDir %ROOT%\data
    Pidfile %ROOT%\data\nxlog.pid
    SpoolDir %ROOT%\data
    LogFile %ROOT%\data\nxlog.log

    <Extension syslog>
        Module        xm_syslog 
    </Extension>

    <Extension File operations>
        Module        xm_fileop 
    </Extension>

    <Input in1>
        Module      im_file
        File         '\\PATH\VB*'
        Exec         if file_name() =~ /([^\\]+)$/ $LogFile = $1;
        SavePos        True
        InputType    LineBased
        Recursive    FALSE
        PollInterval 5
        ActiveFiles 1
        Exec $Message = 'LOG: '+$raw_event;
    </Input>

    <Input in2>
        Module      im_file
        File         '\\PATH\TCP*'
        Exec         if file_name() =~ /([^\\]+)$/ $LogFile = $1;
        SavePos        True
        InputType    LineBased
        Recursive    FALSE
        PollInterval 5
        ActiveFiles 1
        Exec $Message = 'LOG: '+$raw_event;
    </Input>

    <Output out>
        Module      om_udp
        Host        IP
        Port        514
        Exec to_syslog_bsd();
    </Output>

    <Route 1>
        Path        in1,in2 => out
    </Route>

     

    nxlog.log:

    2014-12-21 14:34:50 INFO nxlog-ce-2.8.1248 started
    2014-12-21 14:35:10 WARNING inode changed for '\\Path\VB.LOG141210': reopening possibly rotated file
    2014-12-21 14:35:10 WARNING inode changed for '\\Path\TCP.LOG141203': reopening possibly rotated file
    2014-12-21 14:35:20 WARNING inode changed for '\\Path\VB.LOG141210': reopening possibly rotated file
    2014-12-21 14:35:20 WARNING inode changed for '\\Path\VB.LOG141210': reopening possibly rotated file

    *****logs are static so what it should do is read each log 1 time and continue*******

    ****what acctually is happing is the reopen you can see and logs are reread again******

    December 21, 2014 - 4:03pm