xm_multiline, EndLine, and wildcarded input files

#1 rochbu 7 years ago

Given a number of application logs sharing the same HeaderLine and EndLine regular expressions we are trying out a xm_multiline with im_file config using wild cards.

<Extension multi> Module xm_multiline HeaderLine /^\d{4}-\d{2}-\d{2} \d{2}:\d{2}:\d{2},\d{3} @ batch_task\._init_logger : \[INFO\]\+ / EndLine /^\d{4}-\d{2}-\d{2} \d{2}:\d{2}:\d{2},\d{3} @ run_batch\.<module> : \[INFO\]- / </Extension>

<Input inPython> Module im_file File "C:\\data\\server1\\*.log" InputType multi Exec $FileName = file_name(); </Input>

It works consistently without wildcards pointed at one file.
It works intermittently with wildcards pointed at multiple files being written to concurrenlty.

I'm wondering if this is a supported use case. i.e. multiline events from wildcarded files being written to concurrenlty. Or should we be specifiying each input file individually?

thanks,

Rob

Permalink

#2 b0ti Nxlog ✓ 7 years ago

#1 rochbu

Given a number of application logs sharing the same HeaderLine and EndLine regular expressions we are trying out a xm_multiline with im_file config using wild cards. <Extension multi> Module xm_multiline HeaderLine /^\d{4}-\d{2}-\d{2} \d{2}:\d{2}:\d{2},\d{3} @ batch_task\._init_logger : \[INFO\]\+ / EndLine /^\d{4}-\d{2}-\d{2} \d{2}:\d{2}:\d{2},\d{3} @ run_batch\.<module> : \[INFO\]- / </Extension> <Input inPython> Module im_file File "C:\\data\\server1\\*.log" InputType multi Exec $FileName = file_name(); </Input> It works consistently without wildcards pointed at one file. It works intermittently with wildcards pointed at multiple files being written to concurrenlty. I'm wondering if this is a supported use case. i.e. multiline events from wildcarded files being written to concurrenlty. Or should we be specifiying each input file individually? thanks, Rob

The InputType directive ensures that the data is processed on the file level (and per connection for the networking modules) instead of combining data coming from all the files (*.log) and matching that as a whole, so yes, that's a supported use-case and there is no need to specify each file individually.

There was another report with a similar complaint recently, see this other forum post. Can you create a test case to reproduce and put it in a support ticket?