Correct use of xm_multiline and im_file output to Graylog

Tags:

#1 phoeneous

Hello nxlog world. My application has a custom log file on a Windows 2012 box that I need to send to Graylog which has a GELF UDP input running. The log file is multiline and there is a specific header and footer that I'm seperating the log file entries with. As of right now despite using the multiline module, when my logs arrive in Graylog it is parsing each line of the log file individually instead of by header and footer. Here is my nxlog.conf:


define ROOT C:\Program Files (x86)\nxlog

Moduledir %ROOT%\modules
CacheDir %ROOT%\data
Pidfile %ROOT%\data\nxlog.pid
SpoolDir %ROOT%\data
LogFile %ROOT%\data\nxlog.log

<Extension gelf>
    Module xm_gelf
</Extension>

<Extension multiline>
    Module    xm_multiline
        HeaderLine    /\d{4}-\d{2}-\d{2}\s\d*:\d\d:\d*,\d{3}\s[a-zA-Z]*\s\s\S\s\[Begin Lead\]/
        EndLine        /\d{4}-\d{2}-\d{2}\s\d*:\d\d:\d*,\d{3}\s[a-zA-Z]*\s\s\S\s\[End Lead\]/
</Extension>

<Extension json>
    Module    xm_json
</Extension>

<Input im_msvistalog>
    Module      im_msvistalog
    Query   <QueryList>\
            <Query Id="0">\
                    <Select Path="System">*[System[(Level=1 or Level=2 or Level=3)]]</Select>\
                    <Select Path="Application">*[System[(Level=1 or Level=2 or Level=3)]]</Select>\
            </Query>\
        </QueryList>
    Exec if $raw_event =~ /(\d\d\d\d\-\d\d-\d\d \d\d:\d\d:\d\d)/ $EventTime = parsedate($1);
</Input>

<Input im_file>
    Module   im_file
    File "C:\\LOGS\\application_service.log"
    SavePos FALSE
     ReadFromLast FALSE
    InputType multiline
</Input>

<Output om_udp>
    Module      om_udp
    Host        10.25.32.21
    Port        12201
    OutputType  GELF
</Output>

<Route 1>
    Path    im_msvistalog  => om_udp
</Route>

<Route 2>
    Path im_file => om_udp
</Route>


The application_service.log file looks like this:

2017-06-20 17:25:50,168 INFO  – [Begin Lead]
2017-06-20 17:25:50,168 INFO  – Getting lead by id '5551212'
2017-06-20 17:25:51,215 INFO  – Lead field = First Name. Value = Michael
2017-06-20 17:25:54,778 INFO  – Lead field = Last Name. Value = Westin
2017-06-20 17:25:54,793 INFO  – Lead field = CustomerNumber. Value = 99999
2017-06-20 17:26:05,887 INFO  – [End Lead]
2017-06-20 18:25:50,168 INFO  – [Begin Lead]
2017-06-20 18:25:50,168 INFO  – Getting lead by id '4279551'
2017-06-20 18:25:51,215 INFO  – Lead field = First Name. Value = Emmit
2017-06-20 18:25:54,778 INFO  – Lead field = Last Name. Value = Stussy
2017-06-20 18:25:54,793 INFO  – Lead field = CustomerNumber. Value = 94231
2017-06-20 18:26:05,887 INFO  – [End Lead]
2017-06-20 19:25:50,168 INFO  – [Begin Lead]
2017-06-20 19:25:50,168 INFO  – Getting lead by id '0081227'
2017-06-20 19:25:51,215 INFO  – Lead field = First Name. Value = Saul
2017-06-20 19:25:54,778 INFO  – Lead field = Last Name. Value = Goodman
2017-06-20 19:25:54,793 INFO  – Lead field = CustomerNumber. Value = 33487
2017-06-20 19:26:05,887 INFO  – [End Lead]


Ideally when it his Graylog I'd like the content between [Begin Lead] and [End Lead] to be stored in one field.  But with my nxlog config above it is putting each line of the log file in its own message ID in Graylog.  Do I have my config wrong?  Is there a better method to parse the log file and send to Graylog?  Any help is appreciated.

#2 b0ti Nxlog ✓
#1 phoeneous
Hello nxlog world. My application has a custom log file on a Windows 2012 box that I need to send to Graylog which has a GELF UDP input running. The log file is multiline and there is a specific header and footer that I'm seperating the log file entries with. As of right now despite using the multiline module, when my logs arrive in Graylog it is parsing each line of the log file individually instead of by header and footer. Here is my nxlog.conf: define ROOT C:\Program Files (x86)\nxlog Moduledir %ROOT%\modules CacheDir %ROOT%\data Pidfile %ROOT%\data\nxlog.pid SpoolDir %ROOT%\data LogFile %ROOT%\data\nxlog.log <Extension gelf>     Module xm_gelf </Extension> <Extension multiline>     Module    xm_multiline         HeaderLine    /\d{4}-\d{2}-\d{2}\s\d*:\d\d:\d*,\d{3}\s[a-zA-Z]*\s\s\S\s\[Begin Lead\]/         EndLine        /\d{4}-\d{2}-\d{2}\s\d*:\d\d:\d*,\d{3}\s[a-zA-Z]*\s\s\S\s\[End Lead\]/ </Extension> <Extension json>     Module    xm_json </Extension> <Input im_msvistalog>     Module      im_msvistalog     Query   <QueryList>\             <Query Id="0">\                     <Select Path="System">*[System[(Level=1 or Level=2 or Level=3)]]</Select>\                     <Select Path="Application">*[System[(Level=1 or Level=2 or Level=3)]]</Select>\             </Query>\         </QueryList>     Exec if $raw_event =~ /(\d\d\d\d\-\d\d-\d\d \d\d:\d\d:\d\d)/ $EventTime = parsedate($1); </Input> <Input im_file>     Module   im_file     File "C:\\LOGS\\application_service.log"     SavePos FALSE      ReadFromLast FALSE     InputType multiline </Input> <Output om_udp>     Module      om_udp     Host        10.25.32.21     Port        12201     OutputType  GELF </Output> <Route 1>     Path    im_msvistalog  => om_udp </Route> <Route 2>     Path im_file => om_udp </Route> The application_service.log file looks like this: 2017-06-20 17:25:50,168 INFO  – [Begin Lead] 2017-06-20 17:25:50,168 INFO  – Getting lead by id '5551212' 2017-06-20 17:25:51,215 INFO  – Lead field = First Name. Value = Michael 2017-06-20 17:25:54,778 INFO  – Lead field = Last Name. Value = Westin 2017-06-20 17:25:54,793 INFO  – Lead field = CustomerNumber. Value = 99999 2017-06-20 17:26:05,887 INFO  – [End Lead] 2017-06-20 18:25:50,168 INFO  – [Begin Lead] 2017-06-20 18:25:50,168 INFO  – Getting lead by id '4279551' 2017-06-20 18:25:51,215 INFO  – Lead field = First Name. Value = Emmit 2017-06-20 18:25:54,778 INFO  – Lead field = Last Name. Value = Stussy 2017-06-20 18:25:54,793 INFO  – Lead field = CustomerNumber. Value = 94231 2017-06-20 18:26:05,887 INFO  – [End Lead] 2017-06-20 19:25:50,168 INFO  – [Begin Lead] 2017-06-20 19:25:50,168 INFO  – Getting lead by id '0081227' 2017-06-20 19:25:51,215 INFO  – Lead field = First Name. Value = Saul 2017-06-20 19:25:54,778 INFO  – Lead field = Last Name. Value = Goodman 2017-06-20 19:25:54,793 INFO  – Lead field = CustomerNumber. Value = 33487 2017-06-20 19:26:05,887 INFO  – [End Lead] Ideally when it his Graylog I'd like the content between [Begin Lead] and [End Lead] to be stored in one field.  But with my nxlog config above it is putting each line of the log file in its own message ID in Graylog.  Do I have my config wrong?  Is there a better method to parse the log file and send to Graylog?  Any help is appreciated.

I think your regexp is not matching. The following should work:

<Extension multiline>
        Module        xm_multiline
        HeaderLine    /\[Begin Lead\]/
        EndLine       /\[End Lead\]/
        Exec          if $raw_event =~ /\d\d\d\d-\d\d-\d\d \d\d\:\d\d\:\d\d,\d+ \S+\s+(.+)/ $raw_event = $1;
</Extension>


Note that it is possible to rewrite the log inside xm_multiline to remove the headers so that you get only this:

[Begin Lead]Getting lead by id '5551212'Lead field = First Name. Value = MichaelLead field = Last Name. Value = WestinLead field = CustomerNumber. Value = 99999[End Lead]