How to delete input files after reading ?

Tags:

#1 RemyVeo

Hi Everyone,
New forum user, so sorry if I do not respect some uses rules…and I’m French by the way, so sorry about my bad English level too…;-)
I use Nxlog in the following context :
A machine A send periodically a file on a machine B (same file name)
Machine B is used like a first level of log centralization. Nxlog is installed on it
Then, Nxlog send log to Machine C who host Logstash
I’m looking for a process who remove input files after end of reading and processing, but I’m not sure about how process “in-out” operate, is-it after each line or after a n entire file reading ?
How can I test a “end of reading” condition ?
Right now, I’m going to test something like that :
<Extension fileop>
 Module xm_fileop
</Extension>

<Input in>
    Module im_file
    SavePos TRUE
    ReadFromLast TRUE
    PollInterval 900 # 15mn
    File "/var/log/inpufile*.log"
 Exec $InputFileName1 = file_name()
</Input>

<Output logstash>
    Module om_tcp
 Port 6002
Host 10.x.y.z
Exec file_remove($InputFileName1);
</Output>

<Route 1>
    Path in => logstash
</Route>

Is it correct ?

Thanks in advance for your help

#2 adm Nxlog ✓
#1 RemyVeo
Hi Everyone, New forum user, so sorry if I do not respect some uses rules…and I’m French by the way, so sorry about my bad English level too…;-) I use Nxlog in the following context : A machine A send periodically a file on a machine B (same file name) Machine B is used like a first level of log centralization. Nxlog is installed on it Then, Nxlog send log to Machine C who host Logstash I’m looking for a process who remove input files after end of reading and processing, but I’m not sure about how process “in-out” operate, is-it after each line or after a n entire file reading ? How can I test a “end of reading” condition ? Right now, I’m going to test something like that : <Extension fileop>  Module xm_fileop </Extension> <Input in>     Module im_file     SavePos TRUE     ReadFromLast TRUE     PollInterval 900 # 15mn     File "/var/log/inpufile*.log"  Exec $InputFileName1 = file_name() </Input> <Output logstash>     Module om_tcp  Port 6002 Host 10.x.y.z Exec file_remove($InputFileName1); </Output> <Route 1>     Path in => logstash </Route> Is it correct ? Thanks in advance for your help

NXLog reads the data from files one line per record by default. EOF might not always mean that the file is complete as there are situations when there is more data appended later.

Currently there is no support in im_file to delete the input on EOF, adding this feature is on the roadmap though. For now you should use external log rotation tools or <Schedule>.