8responses guyl i defined an input for im_file that is being sent as syslog i have multiple files in a folder what i want to do is read each file and on EOF copy to another folder. didn`t find the option to identify EOF AskedDecember 18, 2014 - 4:22pm Answer the question Leave a comment
adm This is not possible but there are plans to implement support for that. im_file was designed with the assumption to read files that are constantly appended to. In this case reaching EOF does not mean there won't be any more data appended. AnsweredDecember 18, 2014 - 4:34pm Leave a comment Comments (7) guyl Leave a comment i understand but the issue is that for some reason the logs always rotates and i am getting the same results. **file getting read from the start every time. December 19, 2014 - 9:41am adm (NXLog) Leave a comment You should use a wildcard pattern for the file name that matches the file being written but not those that are rotated. December 19, 2014 - 9:48am guyl Leave a comment let me describe the situation: I have a folder containing logs 1.log 2.log 3.log I defined the im_file as *.log from that folder and collection is working the problem is that it never stops (collecting same raws) checking the logs shows information about "1.log has been rotated...etc....reading from start" (same for other logs) Eventually even if the logs are static It just keep reading them in a loop saving last POS doesn't help also defining logs Identification by size and inode didn’t work. Thanks for helping, i hope that we can solve this one :) December 19, 2014 - 6:13pm adm (NXLog) Leave a comment Ok, but that still does not explain how the logs are rotated. Is this your rotation strategy: 1.log => 2.log, 2.log => 3.log, create 1.log ? Also you did not show your conf, that could reveal some other issues. December 19, 2014 - 6:35pm guyl Leave a comment I will send the config on sunday my output actins is im_file => syslog the logs are just statis logs it is more easy to understand ERROR20\12.log ERROR19\12.log ERROR18\12.log the big problem is that i don`t do any reotation but in the nxlog for some reason identify that the logs changes and reads it again December 20, 2014 - 1:59pm guyl Leave a comment Configuration: define ROOT C:\Program Files (x86)\nxlog Moduledir %ROOT%\modules CacheDir %ROOT%\data Pidfile %ROOT%\data\nxlog.pid SpoolDir %ROOT%\data LogFile %ROOT%\data\nxlog.log <Extension syslog> Module xm_syslog </Extension> <Extension File operations> Module xm_fileop </Extension> <Input in1> Module im_file File '\\PATH\VB*' Exec if file_name() =~ /([^\\]+)$/ $LogFile = $1; SavePos True InputType LineBased Recursive FALSE PollInterval 5 ActiveFiles 1 Exec $Message = 'LOG: '+$raw_event; </Input> <Input in2> Module im_file File '\\PATH\TCP*' Exec if file_name() =~ /([^\\]+)$/ $LogFile = $1; SavePos True InputType LineBased Recursive FALSE PollInterval 5 ActiveFiles 1 Exec $Message = 'LOG: '+$raw_event; </Input> <Output out> Module om_udp Host IP Port 514 Exec to_syslog_bsd(); </Output> <Route 1> Path in1,in2 => out </Route> nxlog.log: 2014-12-21 14:34:50 INFO nxlog-ce-2.8.1248 started 2014-12-21 14:35:10 WARNING inode changed for '\\Path\VB.LOG141210': reopening possibly rotated file 2014-12-21 14:35:10 WARNING inode changed for '\\Path\TCP.LOG141203': reopening possibly rotated file 2014-12-21 14:35:20 WARNING inode changed for '\\Path\VB.LOG141210': reopening possibly rotated file 2014-12-21 14:35:20 WARNING inode changed for '\\Path\VB.LOG141210': reopening possibly rotated file *****logs are static so what it should do is read each log 1 time and continue******* ****what acctually is happing is the reopen you can see and logs are reread again****** December 21, 2014 - 4:03pm adm (NXLog) Leave a comment Why does the inode of the file change? That's the only reason why nxlog thinks the file was rotated. December 22, 2014 - 7:53pm
Comments (7)
i understand but the issue is that for some reason the logs always rotates and i am getting the same results.
**file getting read from the start every time.
You should use a wildcard pattern for the file name that matches the file being written but not those that are rotated.
let me describe the situation:
I have a folder containing logs
1.log 2.log 3.log
I defined the im_file as
*.log from that folder and collection is working the problem is that it never stops (collecting same raws)
checking the logs shows information about "1.log has been rotated...etc....reading from start" (same for other logs)
Eventually even if the logs are static It just keep reading them in a loop
saving last POS doesn't help also defining logs Identification by size and inode didn’t work.
Thanks for helping, i hope that we can solve this one :)
Ok, but that still does not explain how the logs are rotated.
Is this your rotation strategy: 1.log => 2.log, 2.log => 3.log, create 1.log ?
Also you did not show your conf, that could reveal some other issues.
I will send the config on sunday
my output actins is im_file => syslog
the logs are just statis logs it is more easy to understand
ERROR20\12.log
ERROR19\12.log
ERROR18\12.log
the big problem is that i don`t do any reotation but in the nxlog for some reason identify that the logs changes and reads it again
Configuration:
define ROOT C:\Program Files (x86)\nxlog
Moduledir %ROOT%\modules
CacheDir %ROOT%\data
Pidfile %ROOT%\data\nxlog.pid
SpoolDir %ROOT%\data
LogFile %ROOT%\data\nxlog.log
<Extension syslog>
Module xm_syslog
</Extension>
<Extension File operations>
Module xm_fileop
</Extension>
<Input in1>
Module im_file
File '\\PATH\VB*'
Exec if file_name() =~ /([^\\]+)$/ $LogFile = $1;
SavePos True
InputType LineBased
Recursive FALSE
PollInterval 5
ActiveFiles 1
Exec $Message = 'LOG: '+$raw_event;
</Input>
<Input in2>
Module im_file
File '\\PATH\TCP*'
Exec if file_name() =~ /([^\\]+)$/ $LogFile = $1;
SavePos True
InputType LineBased
Recursive FALSE
PollInterval 5
ActiveFiles 1
Exec $Message = 'LOG: '+$raw_event;
</Input>
<Output out>
Module om_udp
Host IP
Port 514
Exec to_syslog_bsd();
</Output>
<Route 1>
Path in1,in2 => out
</Route>
nxlog.log:
2014-12-21 14:34:50 INFO nxlog-ce-2.8.1248 started
2014-12-21 14:35:10 WARNING inode changed for '\\Path\VB.LOG141210': reopening possibly rotated file
2014-12-21 14:35:10 WARNING inode changed for '\\Path\TCP.LOG141203': reopening possibly rotated file
2014-12-21 14:35:20 WARNING inode changed for '\\Path\VB.LOG141210': reopening possibly rotated file
2014-12-21 14:35:20 WARNING inode changed for '\\Path\VB.LOG141210': reopening possibly rotated file
*****logs are static so what it should do is read each log 1 time and continue*******
****what acctually is happing is the reopen you can see and logs are reread again******
Why does the inode of the file change? That's the only reason why nxlog thinks the file was rotated.