Correct use of xm_multiline and im_file output to Graylog
Hello nxlog world. My application has a custom log file on a Windows 2012 box that I need to send to Graylog which has a GELF UDP input running. The log file is multiline and there is a specific header and footer that I'm seperating the log file entries with. As of right now despite using the multiline module, when my logs arrive in Graylog it is parsing each line of the log file individually instead of by header and footer. Here is my nxlog.conf:
define ROOT C:\Program Files (x86)\nxlog
Moduledir %ROOT%\modules
CacheDir %ROOT%\data
Pidfile %ROOT%\data\nxlog.pid
SpoolDir %ROOT%\data
LogFile %ROOT%\data\nxlog.log
<Extension gelf>
Module xm_gelf
</Extension>
<Extension multiline>
Module xm_multiline
HeaderLine /\d{4}-\d{2}-\d{2}\s\d*:\d\d:\d*,\d{3}\s[a-zA-Z]*\s\s\S\s\[Begin Lead\]/
EndLine /\d{4}-\d{2}-\d{2}\s\d*:\d\d:\d*,\d{3}\s[a-zA-Z]*\s\s\S\s\[End Lead\]/
</Extension>
<Extension json>
Module xm_json
</Extension>
<Input im_msvistalog>
Module im_msvistalog
Query <QueryList>\
<Query Id="0">\
<Select Path="System">*[System[(Level=1 or Level=2 or Level=3)]]</Select>\
<Select Path="Application">*[System[(Level=1 or Level=2 or Level=3)]]</Select>\
</Query>\
</QueryList>
Exec if $raw_event =~ /(\d\d\d\d\-\d\d-\d\d \d\d:\d\d:\d\d)/ $EventTime = parsedate($1);
</Input>
<Input im_file>
Module im_file
File "C:\\LOGS\\application_service.log"
SavePos FALSE
ReadFromLast FALSE
InputType multiline
</Input>
<Output om_udp>
Module om_udp
Host 10.25.32.21
Port 12201
OutputType GELF
</Output>
<Route 1>
Path im_msvistalog => om_udp
</Route>
<Route 2>
Path im_file => om_udp
</Route>
The application_service.log file looks like this:
2017-06-20 17:25:50,168 INFO – [Begin Lead]
2017-06-20 17:25:50,168 INFO – Getting lead by id '5551212'
2017-06-20 17:25:51,215 INFO – Lead field = First Name. Value = Michael
2017-06-20 17:25:54,778 INFO – Lead field = Last Name. Value = Westin
2017-06-20 17:25:54,793 INFO – Lead field = CustomerNumber. Value = 99999
2017-06-20 17:26:05,887 INFO – [End Lead]
2017-06-20 18:25:50,168 INFO – [Begin Lead]
2017-06-20 18:25:50,168 INFO – Getting lead by id '4279551'
2017-06-20 18:25:51,215 INFO – Lead field = First Name. Value = Emmit
2017-06-20 18:25:54,778 INFO – Lead field = Last Name. Value = Stussy
2017-06-20 18:25:54,793 INFO – Lead field = CustomerNumber. Value = 94231
2017-06-20 18:26:05,887 INFO – [End Lead]
2017-06-20 19:25:50,168 INFO – [Begin Lead]
2017-06-20 19:25:50,168 INFO – Getting lead by id '0081227'
2017-06-20 19:25:51,215 INFO – Lead field = First Name. Value = Saul
2017-06-20 19:25:54,778 INFO – Lead field = Last Name. Value = Goodman
2017-06-20 19:25:54,793 INFO – Lead field = CustomerNumber. Value = 33487
2017-06-20 19:26:05,887 INFO – [End Lead]
Ideally when it his Graylog I'd like the content between [Begin Lead] and [End Lead] to be stored in one field. But with my nxlog config above it is putting each line of the log file in its own message ID in Graylog. Do I have my config wrong? Is there a better method to parse the log file and send to Graylog? Any help is appreciated.