Issues With "Multi-line message parser (xm_multiline)"

View thread

gmelasecca

My company is looking to setup NxLog. We are having issues reading in multiline exception logs from applications such as Tomcat, Java, Apache etc. I am able to read in the files but unfortunately the output in our GrayLog application is showing every event as one line. I tried to implement the xm_multiline module but i seem to be having issues getting it to work.

 

installed NxLog and checked my configuration to the following below. restarted the services, let the service run all night and still the output is the same as shown below.

 

Sample Input Log:

07/07/2016 13:35:11.654 [tomcat-http--43] [ERROR] [4114723 ms] Warning - unprocessed rows in esolutions.care.assess.WeAssessment
esolutions.EsolutionsException: There were 83 unprocessed rows out of 84
    at esolutions.base.WeObject.sleep(WeObject.java:2767)
    at esolutions.base.WeObject.clear(WeObject.java:3250)
    at esolutions.care.assess.WeAssessment.clear(WeAssessment.java:7699)
    at esolutions.base.WeObject.close(WeObject.java:2815)
    at esolutions.util.WeHTMLTable.getTableHTML(WeHTMLTable.java:541)
    at esolutions.util.WeHTMLTable.toHTML(WeHTMLTable.java:508)
    at org.apache.jsp.admin.client.cp_005fassessment_jsp._jspService(cp_005fassessment_jsp.java:4412)
    at org.apache.jasper.runtime.HttpJspBase.service(HttpJspBase.java:70)

07/07/2016 13:36:21.828 [tomcat-http--26] [ERROR] [4184897 ms] Warning - unprocessed rows in esolutions.care.assess.WeAssessment

esolutions.EsolutionsException: There were 82 unprocessed rows out of 83

    at esolutions.base.WeObject.sleep(WeObject.java:2767)
    at esolutions.base.WeObject.clear(WeObject.java:3250)
    at esolutions.care.assess.WeAssessment.clear(WeAssessment.java:7699)
    at esolutions.base.WeObject.close(WeObject.java:2815)
    at esolutions.util.WeHTMLTable.getTableHTML(WeHTMLTable.java:541)
    at esolutions.util.WeHTMLTable.toHTML(WeHTMLTable.java:508)
    at org.apache.jsp.admin.client.cp_005fassessment_jsp._jspService(cp_005fassessment_jsp.java:4412)
    at org.apache.jasper.runtime.HttpJspBase.service(HttpJspBase.java:70)
    at javax.servlet.http.HttpServlet.service(HttpServlet.java:727)
    at org.apache.jasper.servlet.JspServletWrapper.service(JspServletWrapper.java:432)

 

 

Sample Output From GrayLog in CSV format. The output in the webui is each event as it shows in the "message" column.

timestamp source EventReceivedTime level message SourceModuleName SourceModuleType
2016-07-19T21:27:08.000Z GDPCCA02 07/19/16 17:27 6 2016/07/19 17:27:08.032 | srvmain | INFO   | 07/19/2016 17:27:08 pcc-wrapper-log im_file
2016-07-19T21:27:08.000Z GDPCCA02 07/19/16 17:27 6 2016/07/19 17:27:08.032 | srvmain | INFO   | java.lang.NumberFor pcc-wrapper-log im_file
2016-07-19T21:27:08.000Z GDPCCA02 07/19/16 17:27 6 2016/07/19 17:27:08.032 | srvmain | INFO   | at com.pointclickc pcc-wrapper-log im_file
2016-07-19T21:27:08.000Z GDPCCA02 07/19/16 17:27 6 2016/07/19 17:27:08.032 | srvmain | INFO   | at org.apache.cata pcc-wrapper-log im_file

 

Configuration File. I tried multiple regular expressions with no success.

## This is a sample configuration file. See the nxlog reference manual about the
## configuration options. It should be installed locally and is also available
## online at http://nxlog.org/docs/

## Please set the ROOT to the folder your nxlog was installed into,
## otherwise it will not start.

#define ROOT C:\Program Files\nxlog
define ROOT C:\Program Files (x86)\nxlog

Moduledir %ROOT%\modules
CacheDir %ROOT%\data
Pidfile %ROOT%\data\nxlog.pid
SpoolDir %ROOT%\data
LogFile %ROOT%\data\nxlog.log

<Extension gelf>
    Module        xm_gelf
</Extension>

<Extension fileop>
    Module        xm_fileop
</Extension>

<Extension multiline>
    Module    xm_multiline
    HeaderLine    /^\d{0,2}\/\d{0,2}\/\d{0,4}/
#    HeaderLine    '^\d{0,2}\/\d{0,2}\/\d{0,4}\ \d{0,3}\:\d{0,3}\:\d{0,3}\.\d{0,4}\ \['
</Extension>


<Input pcc-wrapper-log>
    Module            im_file
    File            "C:\\pivotal-tc-server-standard-3.1.0.RELEASE\\pccweb\\logs\\wrapper.log"
    SavePos            TRUE
    InputType    multiline
</Input>
<Input pcc-mdstrace-log>
    Module            im_file
    File            "C:\\pivotal-tc-server-standard-3.1.0.RELEASE\\pccweb\\logs\\mdstrace.log"
    SavePos            TRUE
    InputType    multiline
</Input>
<Input pcc-exceptionHidingUtil-log>
    Module            im_file
    File            "C:\\pivotal-tc-server-standard-3.1.0.RELEASE\\pccweb\\logs\\exceptionHidingUtil.log"
    SavePos            TRUE
    InputType    multiline
</Input>
<Input pcc-esolutions-log>
    Module            im_file
    File            "C:\\pivotal-tc-server-standard-3.1.0.RELEASE\\pccweb\\logs\\esolutions.log"
    SavePos            TRUE
    InputType    multiline
</Input>
#<Input pcc-localHostAccess-log>
#    Module            im_file
#    File            "C:\\pivotal-tc-server-standard-3.1.0.RELEASE\\pccweb\\logs\\localhost_access_log.*"
#    SavePos            TRUE
#    InputType    multiline
#</Input>

<Output graylog>
    Module      om_udp
    Host        graylog.genesishcc.com
    Port        12201
    OutputType    GELF
</Output>

<Route PCC>
    Path        pcc-wrapper-log => pcc-mdstrace-log => pcc-exceptionHidingUtil-log => pcc-esolutions-log => graylog
##    Path        pcc-wrapper-log => pcc-mdstrace-log => pcc-exceptionHidingUtil-log => pcc-esolutions-log => pcc-localHostAccess-log => graylog
</Route>