Does the parser quit after not matching x amount of logs?

#1 jhartman 1 year ago

I am having issues where when I try to parse a big file ~1500 lines the regexp doesn't find any lines that match and then eventually “gives up”. There is no ERROR log in the nxlog.log saying that it essentially stopped but it never writes anything to my output file. In the input file I have 6 lines that match my parser though.

<Exec>
if $raw_event =~ /srv_name="([^"]+).+?user_auth_entr="([^"]+)/
{
$event_type = 'VPN_SESSION_IP_ASSIGNED';
$version = 'v1';
$time = 'test';
$account = $2;
$account_domain = 'null';
$assigned_ip = 'null';
$source_ip = 'null';
$authentication_result = 'FAILURE';
$authentication_target = $1;
}
</Exec>

I was confused at first and thought there was an issue with my statement but it checks out.

When I specifically grabbed only those 6 log lines that would match and ran the service against ONLY those logs, everything parsed and worked just fine giving my the 6 new log lines in my output file.

So this leads me to believe that the only reason it didn't work the first time is because the first time it actually matches those log lines is the 112th line. So my assumption is that it tries to process the incoming log lines against my regex and after so many not matching it just stops.

Can anyone confirm if this is accurate and if so, how can I increase the threshold and/or remove this dependency?

Permalink

#2 NenadMDeactivated Nxlog ✓ 1 year ago (Last updated 1 year ago )

Hello,

This issue looks like a problem with the im_file module configuration or a problem with the lines in the file or with the file itself….NXLog doesn't really give up after any number of lines.
Since you haven't shared the entire configuration file - I could only assume the root cause. My best guess is that the problem could be that you are trying to read a static file without changing the default values of the SavePos and the ReadFromLast directives. Please check the following documentation page https://docs.nxlog.co/ce/current/index.html#im_file_config

ReadFromLast

This optional boolean directive instructs the module to only read logs which arrive after NXLog is started. This directive comes into effect if a saved position is not found, for example on first start, or when the SavePos directive is FALSE. When the SavePos directive is TRUE and a previously saved position is found, the module will always resume reading from the saved position. If ReadFromLast is FALSE, the module will read all logs from the beginning of the file. This can result in a lot of messages and is usually not the expected behavior. If this directive is not specified, it defaults to TRUE.

The following matrix shows the outcome of this directive in conjunction with the SavePos directive:

ReadFromLast	SavePos	Saved Position	Outcome
`TRUE`	`TRUE`	No	Reads events that are logged after NXLog is started.
`TRUE`	`TRUE`	Yes	Reads events from saved position.
`TRUE`	`FALSE`	No	Reads events that are logged after NXLog is started.
`TRUE`	`FALSE`	Yes	Reads events that are logged after NXLog is started.
`FALSE`	`TRUE`	No	Reads all events.
`FALSE`	`TRUE`	Yes	Reads events from saved position.
`FALSE`	`FALSE`	No	Reads all events.
`FALSE`	`FALSE`	Yes	Reads all events.