4
responses

Hi Everyone,
New forum user, so sorry if I do not respect some uses rules…and I’m French by the way, so sorry about my bad English level too…;-)
I use Nxlog in the following context :
A machine A send periodically a file on a machine B (same file name)
Machine B is used like a first level of log centralization. Nxlog is installed on it
Then, Nxlog send log to Machine C who host Logstash
I’m looking for a process who remove input files after end of reading and processing, but I’m not sure about how process “in-out” operate, is-it after each line or after a n entire file reading ?
How can I test a “end of reading” condition ?
Right now, I’m going to test something like that :
<Extension fileop>
 Module xm_fileop
</Extension>

<Input in>
    Module im_file
    SavePos TRUE
    ReadFromLast TRUE
    PollInterval 900 # 15mn
    File "/var/log/inpufile*.log"
 Exec $InputFileName1 = file_name()
</Input>

<Output logstash>
    Module om_tcp
 Port 6002
Host 10.x.y.z
Exec file_remove($InputFileName1);
</Output>

<Route 1>
    Path in => logstash
</Route>

Is it correct ?

Thanks in advance for your help

AskedMay 11, 2016 - 3:19pm

Answer (1)

NXLog reads the data from files one line per record by default. EOF might not always mean that the file is complete as there are situations when there is more data appended later.

Currently there is no support in im_file to delete the input on EOF, adding this feature is on the roadmap though. For now you should use external log rotation tools or <Schedule>.

Comments (3)

  • RemyVeo's picture

    Thanks for your answer.

    In my context, I'm sure that Nxlog will end its processing on datafile before new records come, because of "intermediate centralization with temorary file system" use.

    Lets try with <Schedule>... thanks again.

  • Severin Simko's picture

    Hi,

    yes, the NXLog EE from version 3.99. supports the OnEof block for im_file module. So it will be possible to use delete the input file after reading.

    Severin