1
response

My company is looking to setup NxLog. We are having issues reading in multiline exception logs from applications such as Tomcat, Java, Apache etc. I am able to read in the files but unfortunately the output in our GrayLog application is showing every event as one line. I tried to implement the xm_multiline module but i seem to be having issues getting it to work.

 

installed NxLog and checked my configuration to the following below. restarted the services, let the service run all night and still the output is the same as shown below.

 

Sample Input Log:

07/07/2016 13:35:11.654 [tomcat-http--43] [ERROR] [4114723 ms] Warning - unprocessed rows in esolutions.care.assess.WeAssessment

esolutions.EsolutionsException: There were 83 unprocessed rows out of 84

    at esolutions.base.WeObject.sleep(WeObject.java:2767)

    at esolutions.base.WeObject.clear(WeObject.java:3250)

    at esolutions.care.assess.WeAssessment.clear(WeAssessment.java:7699)

    at esolutions.base.WeObject.close(WeObject.java:2815)

    at esolutions.util.WeHTMLTable.getTableHTML(WeHTMLTable.java:541)

    at esolutions.util.WeHTMLTable.toHTML(WeHTMLTable.java:508)

    at org.apache.jsp.admin.client.cp_005fassessment_jsp._jspService(cp_005fassessment_jsp.java:4412)

    at org.apache.jasper.runtime.HttpJspBase.service(HttpJspBase.java:70)

07/07/2016 13:36:21.828 [tomcat-http--26] [ERROR] [4184897 ms] Warning - unprocessed rows in esolutions.care.assess.WeAssessment

esolutions.EsolutionsException: There were 82 unprocessed rows out of 83

    at esolutions.base.WeObject.sleep(WeObject.java:2767)

    at esolutions.base.WeObject.clear(WeObject.java:3250)

    at esolutions.care.assess.WeAssessment.clear(WeAssessment.java:7699)

    at esolutions.base.WeObject.close(WeObject.java:2815)

    at esolutions.util.WeHTMLTable.getTableHTML(WeHTMLTable.java:541)

    at esolutions.util.WeHTMLTable.toHTML(WeHTMLTable.java:508)

    at org.apache.jsp.admin.client.cp_005fassessment_jsp._jspService(cp_005fassessment_jsp.java:4412)

    at org.apache.jasper.runtime.HttpJspBase.service(HttpJspBase.java:70)

    at javax.servlet.http.HttpServlet.service(HttpServlet.java:727)

    at org.apache.jasper.servlet.JspServletWrapper.service(JspServletWrapper.java:432)

 

 

Sample Output From GrayLog in CSV format. The output in the webui is each event as it shows in the "message" column.

timestamp source EventReceivedTime level message SourceModuleName SourceModuleType
2016-07-19T21:27:08.000Z GDPCCA02 07/19/16 17:27 6 2016/07/19 17:27:08.032 | srvmain | INFO   | 07/19/2016 17:27:08 pcc-wrapper-log im_file
2016-07-19T21:27:08.000Z GDPCCA02 07/19/16 17:27 6 2016/07/19 17:27:08.032 | srvmain | INFO   | java.lang.NumberFor pcc-wrapper-log im_file
2016-07-19T21:27:08.000Z GDPCCA02 07/19/16 17:27 6 2016/07/19 17:27:08.032 | srvmain | INFO   | at com.pointclickc pcc-wrapper-log im_file
2016-07-19T21:27:08.000Z GDPCCA02 07/19/16 17:27 6 2016/07/19 17:27:08.032 | srvmain | INFO   | at org.apache.cata pcc-wrapper-log im_file

 

Configuration File. I tried multiple regular expressions with no success.

## This is a sample configuration file. See the nxlog reference manual about the

## configuration options. It should be installed locally and is also available

## online at http://nxlog.org/docs/

## Please set the ROOT to the folder your nxlog was installed into,

## otherwise it will not start.

#define ROOT C:\Program Files\nxlog

define ROOT C:\Program Files (x86)\nxlog

Moduledir %ROOT%\modules

CacheDir %ROOT%\data

Pidfile %ROOT%\data\nxlog.pid

SpoolDir %ROOT%\data

LogFile %ROOT%\data\nxlog.log

<Extension gelf>

    Module        xm_gelf

</Extension>

<Extension fileop>

    Module        xm_fileop

</Extension>

<Extension multiline>

    Module    xm_multiline

    HeaderLine    /^\d{0,2}\/\d{0,2}\/\d{0,4}/

#    HeaderLine    '^\d{0,2}\/\d{0,2}\/\d{0,4}\ \d{0,3}\:\d{0,3}\:\d{0,3}\.\d{0,4}\ \['

</Extension>


<Input pcc-wrapper-log>

    Module            im_file

    File            "C:\\pivotal-tc-server-standard-3.1.0.RELEASE\\pccweb\\logs\\wrapper.log"

    SavePos            TRUE

    InputType    multiline

</Input>

<Input pcc-mdstrace-log>

    Module            im_file

    File            "C:\\pivotal-tc-server-standard-3.1.0.RELEASE\\pccweb\\logs\\mdstrace.log"

    SavePos            TRUE

    InputType    multiline

</Input>

<Input pcc-exceptionHidingUtil-log>

    Module            im_file

    File            "C:\\pivotal-tc-server-standard-3.1.0.RELEASE\\pccweb\\logs\\exceptionHidingUtil.log"

    SavePos            TRUE

    InputType    multiline

</Input>

<Input pcc-esolutions-log>

    Module            im_file

    File            "C:\\pivotal-tc-server-standard-3.1.0.RELEASE\\pccweb\\logs\\esolutions.log"

    SavePos            TRUE

    InputType    multiline

</Input>

#<Input pcc-localHostAccess-log>

#    Module            im_file

#    File            "C:\\pivotal-tc-server-standard-3.1.0.RELEASE\\pccweb\\logs\\localhost_access_log.*"

#    SavePos            TRUE

#    InputType    multiline

#</Input>

<Output graylog>

    Module      om_udp

    Host        graylog.genesishcc.com

    Port        12201

    OutputType    GELF

</Output>

<Route PCC>

    Path        pcc-wrapper-log => pcc-mdstrace-log => pcc-exceptionHidingUtil-log => pcc-esolutions-log => graylog

##    Path        pcc-wrapper-log => pcc-mdstrace-log => pcc-exceptionHidingUtil-log => pcc-esolutions-log => pcc-localHostAccess-log => graylog

</Route>

AskedJuly 19, 2016 - 11:32pm

Answer (1)

Tested the following config with the sample input provided:

<Extension multi>
    Module      xm_multiline
    HeaderLine   /^\d{0,2}\/\d{0,2}\/\d{0,4}/
</Extension>

<Extension json>
    Module      xm_json
</Extension>

<Input in>
    Module      im_file
    File        "tmp/input.txt"
    ReadFromLast FALSE
    InputType   multi
</Input>

<Output out>
    Module      om_file
    Exec $Message = $raw_event; to_json();
    File        "tmp/output"
</Output>

<Route 66>
    Path        in => out
</Route>

It produced 2 JSON records as expected.

I suggest trimming your config to one input and testing it locally first.

Also this looks a bit odd:

Path        pcc-wrapper-log => pcc-mdstrace-log => pcc-exceptionHidingUtil-log => pcc-esolutions-log => graylog

Should be:

Path        pcc-wrapper-log, pcc-mdstrace-log, pcc-exceptionHidingUtil-log, pcc-esolutions-log => graylog