Issues With "Multi-line message parser (xm_multiline)"


#1 gmelasecca

My company is looking to setup NxLog. We are having issues reading in multiline exception logs from applications such as Tomcat, Java, Apache etc. I am able to read in the files but unfortunately the output in our GrayLog application is showing every event as one line. I tried to implement the xm_multiline module but i seem to be having issues getting it to work.

 

installed NxLog and checked my configuration to the following below. restarted the services, let the service run all night and still the output is the same as shown below.

 

Sample Input Log:

07/07/2016 13:35:11.654 [tomcat-http--43] [ERROR] [4114723 ms] Warning - unprocessed rows in esolutions.care.assess.WeAssessment
esolutions.EsolutionsException: There were 83 unprocessed rows out of 84
    at esolutions.base.WeObject.sleep(WeObject.java:2767)
    at esolutions.base.WeObject.clear(WeObject.java:3250)
    at esolutions.care.assess.WeAssessment.clear(WeAssessment.java:7699)
    at esolutions.base.WeObject.close(WeObject.java:2815)
    at esolutions.util.WeHTMLTable.getTableHTML(WeHTMLTable.java:541)
    at esolutions.util.WeHTMLTable.toHTML(WeHTMLTable.java:508)
    at org.apache.jsp.admin.client.cp_005fassessment_jsp._jspService(cp_005fassessment_jsp.java:4412)
    at org.apache.jasper.runtime.HttpJspBase.service(HttpJspBase.java:70)

07/07/2016 13:36:21.828 [tomcat-http--26] [ERROR] [4184897 ms] Warning - unprocessed rows in esolutions.care.assess.WeAssessment

esolutions.EsolutionsException: There were 82 unprocessed rows out of 83

    at esolutions.base.WeObject.sleep(WeObject.java:2767)
    at esolutions.base.WeObject.clear(WeObject.java:3250)
    at esolutions.care.assess.WeAssessment.clear(WeAssessment.java:7699)
    at esolutions.base.WeObject.close(WeObject.java:2815)
    at esolutions.util.WeHTMLTable.getTableHTML(WeHTMLTable.java:541)
    at esolutions.util.WeHTMLTable.toHTML(WeHTMLTable.java:508)
    at org.apache.jsp.admin.client.cp_005fassessment_jsp._jspService(cp_005fassessment_jsp.java:4412)
    at org.apache.jasper.runtime.HttpJspBase.service(HttpJspBase.java:70)
    at javax.servlet.http.HttpServlet.service(HttpServlet.java:727)
    at org.apache.jasper.servlet.JspServletWrapper.service(JspServletWrapper.java:432)

 

 

Sample Output From GrayLog in CSV format. The output in the webui is each event as it shows in the "message" column.

timestamp source EventReceivedTime level message SourceModuleName SourceModuleType
2016-07-19T21:27:08.000Z GDPCCA02 07/19/16 17:27 6 2016/07/19 17:27:08.032 | srvmain | INFO   | 07/19/2016 17:27:08 pcc-wrapper-log im_file
2016-07-19T21:27:08.000Z GDPCCA02 07/19/16 17:27 6 2016/07/19 17:27:08.032 | srvmain | INFO   | java.lang.NumberFor pcc-wrapper-log im_file
2016-07-19T21:27:08.000Z GDPCCA02 07/19/16 17:27 6 2016/07/19 17:27:08.032 | srvmain | INFO   | at com.pointclickc pcc-wrapper-log im_file
2016-07-19T21:27:08.000Z GDPCCA02 07/19/16 17:27 6 2016/07/19 17:27:08.032 | srvmain | INFO   | at org.apache.cata pcc-wrapper-log im_file

 

Configuration File. I tried multiple regular expressions with no success.

## This is a sample configuration file. See the nxlog reference manual about the
## configuration options. It should be installed locally and is also available
## online at http://nxlog.org/docs/

## Please set the ROOT to the folder your nxlog was installed into,
## otherwise it will not start.

#define ROOT C:\Program Files\nxlog
define ROOT C:\Program Files (x86)\nxlog

Moduledir %ROOT%\modules
CacheDir %ROOT%\data
Pidfile %ROOT%\data\nxlog.pid
SpoolDir %ROOT%\data
LogFile %ROOT%\data\nxlog.log

<Extension gelf>
    Module        xm_gelf
</Extension>

<Extension fileop>
    Module        xm_fileop
</Extension>

<Extension multiline>
    Module    xm_multiline
    HeaderLine    /^\d{0,2}\/\d{0,2}\/\d{0,4}/
#    HeaderLine    '^\d{0,2}\/\d{0,2}\/\d{0,4}\ \d{0,3}\:\d{0,3}\:\d{0,3}\.\d{0,4}\ \['
</Extension>


<Input pcc-wrapper-log>
    Module            im_file
    File            "C:\\pivotal-tc-server-standard-3.1.0.RELEASE\\pccweb\\logs\\wrapper.log"
    SavePos            TRUE
    InputType    multiline
</Input>
<Input pcc-mdstrace-log>
    Module            im_file
    File            "C:\\pivotal-tc-server-standard-3.1.0.RELEASE\\pccweb\\logs\\mdstrace.log"
    SavePos            TRUE
    InputType    multiline
</Input>
<Input pcc-exceptionHidingUtil-log>
    Module            im_file
    File            "C:\\pivotal-tc-server-standard-3.1.0.RELEASE\\pccweb\\logs\\exceptionHidingUtil.log"
    SavePos            TRUE
    InputType    multiline
</Input>
<Input pcc-esolutions-log>
    Module            im_file
    File            "C:\\pivotal-tc-server-standard-3.1.0.RELEASE\\pccweb\\logs\\esolutions.log"
    SavePos            TRUE
    InputType    multiline
</Input>
#<Input pcc-localHostAccess-log>
#    Module            im_file
#    File            "C:\\pivotal-tc-server-standard-3.1.0.RELEASE\\pccweb\\logs\\localhost_access_log.*"
#    SavePos            TRUE
#    InputType    multiline
#</Input>

<Output graylog>
    Module      om_udp
    Host        graylog.genesishcc.com
    Port        12201
    OutputType    GELF
</Output>

<Route PCC>
    Path        pcc-wrapper-log => pcc-mdstrace-log => pcc-exceptionHidingUtil-log => pcc-esolutions-log => graylog
##    Path        pcc-wrapper-log => pcc-mdstrace-log => pcc-exceptionHidingUtil-log => pcc-esolutions-log => pcc-localHostAccess-log => graylog
</Route>

#2 adm Nxlog ✓
#1 gmelasecca
My company is looking to setup NxLog. We are having issues reading in multiline exception logs from applications such as Tomcat, Java, Apache etc. I am able to read in the files but unfortunately the output in our GrayLog application is showing every event as one line. I tried to implement the xm_multiline module but i seem to be having issues getting it to work.   installed NxLog and checked my configuration to the following below. restarted the services, let the service run all night and still the output is the same as shown below.   Sample Input Log: 07/07/2016 13:35:11.654 [tomcat-http--43] [ERROR] [4114723 ms] Warning - unprocessed rows in esolutions.care.assess.WeAssessment esolutions.EsolutionsException: There were 83 unprocessed rows out of 84     at esolutions.base.WeObject.sleep(WeObject.java:2767)     at esolutions.base.WeObject.clear(WeObject.java:3250)     at esolutions.care.assess.WeAssessment.clear(WeAssessment.java:7699)     at esolutions.base.WeObject.close(WeObject.java:2815)     at esolutions.util.WeHTMLTable.getTableHTML(WeHTMLTable.java:541)     at esolutions.util.WeHTMLTable.toHTML(WeHTMLTable.java:508)     at org.apache.jsp.admin.client.cp_005fassessment_jsp._jspService(cp_005fassessment_jsp.java:4412)     at org.apache.jasper.runtime.HttpJspBase.service(HttpJspBase.java:70) 07/07/2016 13:36:21.828 [tomcat-http--26] [ERROR] [4184897 ms] Warning - unprocessed rows in esolutions.care.assess.WeAssessment esolutions.EsolutionsException: There were 82 unprocessed rows out of 83     at esolutions.base.WeObject.sleep(WeObject.java:2767)     at esolutions.base.WeObject.clear(WeObject.java:3250)     at esolutions.care.assess.WeAssessment.clear(WeAssessment.java:7699)     at esolutions.base.WeObject.close(WeObject.java:2815)     at esolutions.util.WeHTMLTable.getTableHTML(WeHTMLTable.java:541)     at esolutions.util.WeHTMLTable.toHTML(WeHTMLTable.java:508)     at org.apache.jsp.admin.client.cp_005fassessment_jsp._jspService(cp_005fassessment_jsp.java:4412)     at org.apache.jasper.runtime.HttpJspBase.service(HttpJspBase.java:70)     at javax.servlet.http.HttpServlet.service(HttpServlet.java:727)     at org.apache.jasper.servlet.JspServletWrapper.service(JspServletWrapper.java:432)     Sample Output From GrayLog in CSV format. The output in the webui is each event as it shows in the "message" column. timestamp source EventReceivedTime level message SourceModuleName SourceModuleType 2016-07-19T21:27:08.000Z GDPCCA02 07/19/16 17:27 6 2016/07/19 17:27:08.032 | srvmain | INFO   | 07/19/2016 17:27:08 pcc-wrapper-log im_file 2016-07-19T21:27:08.000Z GDPCCA02 07/19/16 17:27 6 2016/07/19 17:27:08.032 | srvmain | INFO   | java.lang.NumberFor pcc-wrapper-log im_file 2016-07-19T21:27:08.000Z GDPCCA02 07/19/16 17:27 6 2016/07/19 17:27:08.032 | srvmain | INFO   | at com.pointclickc pcc-wrapper-log im_file 2016-07-19T21:27:08.000Z GDPCCA02 07/19/16 17:27 6 2016/07/19 17:27:08.032 | srvmain | INFO   | at org.apache.cata pcc-wrapper-log im_file   Configuration File. I tried multiple regular expressions with no success. ## This is a sample configuration file. See the nxlog reference manual about the ## configuration options. It should be installed locally and is also available ## online at http://nxlog.org/docs/ ## Please set the ROOT to the folder your nxlog was installed into, ## otherwise it will not start. #define ROOT C:\Program Files\nxlog define ROOT C:\Program Files (x86)\nxlog Moduledir %ROOT%\modules CacheDir %ROOT%\data Pidfile %ROOT%\data\nxlog.pid SpoolDir %ROOT%\data LogFile %ROOT%\data\nxlog.log <Extension gelf>     Module        xm_gelf </Extension> <Extension fileop>     Module        xm_fileop </Extension> <Extension multiline>     Module    xm_multiline     HeaderLine    /^\d{0,2}\/\d{0,2}\/\d{0,4}/ #    HeaderLine    '^\d{0,2}\/\d{0,2}\/\d{0,4}\ \d{0,3}\:\d{0,3}\:\d{0,3}\.\d{0,4}\ \[' </Extension> <Input pcc-wrapper-log>     Module            im_file     File            "C:\\pivotal-tc-server-standard-3.1.0.RELEASE\\pccweb\\logs\\wrapper.log"     SavePos            TRUE     InputType    multiline </Input> <Input pcc-mdstrace-log>     Module            im_file     File            "C:\\pivotal-tc-server-standard-3.1.0.RELEASE\\pccweb\\logs\\mdstrace.log"     SavePos            TRUE     InputType    multiline </Input> <Input pcc-exceptionHidingUtil-log>     Module            im_file     File            "C:\\pivotal-tc-server-standard-3.1.0.RELEASE\\pccweb\\logs\\exceptionHidingUtil.log"     SavePos            TRUE     InputType    multiline </Input> <Input pcc-esolutions-log>     Module            im_file     File            "C:\\pivotal-tc-server-standard-3.1.0.RELEASE\\pccweb\\logs\\esolutions.log"     SavePos            TRUE     InputType    multiline </Input> #<Input pcc-localHostAccess-log> #    Module            im_file #    File            "C:\\pivotal-tc-server-standard-3.1.0.RELEASE\\pccweb\\logs\\localhost_access_log.*" #    SavePos            TRUE #    InputType    multiline #</Input> <Output graylog>     Module      om_udp     Host        graylog.genesishcc.com     Port        12201     OutputType    GELF </Output> <Route PCC>     Path        pcc-wrapper-log => pcc-mdstrace-log => pcc-exceptionHidingUtil-log => pcc-esolutions-log => graylog ##    Path        pcc-wrapper-log => pcc-mdstrace-log => pcc-exceptionHidingUtil-log => pcc-esolutions-log => pcc-localHostAccess-log => graylog </Route>

Tested the following config with the sample input provided:

<Extension multi>
    Module      xm_multiline
    HeaderLine   /^\d{0,2}\/\d{0,2}\/\d{0,4}/
</Extension>

<Extension json>
    Module      xm_json
</Extension>

<Input in>
    Module      im_file
    File        "tmp/input.txt"
    ReadFromLast FALSE
    InputType   multi
</Input>

<Output out>
    Module      om_file
    Exec $Message = $raw_event; to_json();
    File        "tmp/output"
</Output>

<Route 66>
    Path        in => out
</Route>

It produced 2 JSON records as expected.

I suggest trimming your config to one input and testing it locally first.

Also this looks a bit odd:

Path        pcc-wrapper-log => pcc-mdstrace-log => pcc-exceptionHidingUtil-log => pcc-esolutions-log => graylog

Should be:

Path        pcc-wrapper-log, pcc-mdstrace-log, pcc-exceptionHidingUtil-log, pcc-esolutions-log => graylog