Issues With "Multi-line message parser (xm_multiline)"
My company is looking to setup NxLog. We are having issues reading in multiline exception logs from applications such as Tomcat, Java, Apache etc. I am able to read in the files but unfortunately the output in our GrayLog application is showing every event as one line. I tried to implement the xm_multiline module but i seem to be having issues getting it to work.
installed NxLog and checked my configuration to the following below. restarted the services, let the service run all night and still the output is the same as shown below.
Sample Input Log:
07/07/2016 13:35:11.654 [tomcat-http--43] [ERROR] [4114723 ms] Warning - unprocessed rows in esolutions.care.assess.WeAssessment
esolutions.EsolutionsException: There were 83 unprocessed rows out of 84
at esolutions.base.WeObject.sleep(WeObject.java:2767)
at esolutions.base.WeObject.clear(WeObject.java:3250)
at esolutions.care.assess.WeAssessment.clear(WeAssessment.java:7699)
at esolutions.base.WeObject.close(WeObject.java:2815)
at esolutions.util.WeHTMLTable.getTableHTML(WeHTMLTable.java:541)
at esolutions.util.WeHTMLTable.toHTML(WeHTMLTable.java:508)
at org.apache.jsp.admin.client.cp_005fassessment_jsp._jspService(cp_005fassessment_jsp.java:4412)
at org.apache.jasper.runtime.HttpJspBase.service(HttpJspBase.java:70)
07/07/2016 13:36:21.828 [tomcat-http--26] [ERROR] [4184897 ms] Warning - unprocessed rows in esolutions.care.assess.WeAssessment
esolutions.EsolutionsException: There were 82 unprocessed rows out of 83
at esolutions.base.WeObject.sleep(WeObject.java:2767)
at esolutions.base.WeObject.clear(WeObject.java:3250)
at esolutions.care.assess.WeAssessment.clear(WeAssessment.java:7699)
at esolutions.base.WeObject.close(WeObject.java:2815)
at esolutions.util.WeHTMLTable.getTableHTML(WeHTMLTable.java:541)
at esolutions.util.WeHTMLTable.toHTML(WeHTMLTable.java:508)
at org.apache.jsp.admin.client.cp_005fassessment_jsp._jspService(cp_005fassessment_jsp.java:4412)
at org.apache.jasper.runtime.HttpJspBase.service(HttpJspBase.java:70)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:727)
at org.apache.jasper.servlet.JspServletWrapper.service(JspServletWrapper.java:432)
Sample Output From GrayLog in CSV format. The output in the webui is each event as it shows in the "message" column.
timestamp | source | EventReceivedTime | level | message | SourceModuleName | SourceModuleType |
2016-07-19T21:27:08.000Z | GDPCCA02 | 07/19/16 17:27 | 6 | 2016/07/19 17:27:08.032 | srvmain | INFO | 07/19/2016 17:27:08 | pcc-wrapper-log | im_file |
2016-07-19T21:27:08.000Z | GDPCCA02 | 07/19/16 17:27 | 6 | 2016/07/19 17:27:08.032 | srvmain | INFO | java.lang.NumberFor | pcc-wrapper-log | im_file |
2016-07-19T21:27:08.000Z | GDPCCA02 | 07/19/16 17:27 | 6 | 2016/07/19 17:27:08.032 | srvmain | INFO | at com.pointclickc | pcc-wrapper-log | im_file |
2016-07-19T21:27:08.000Z | GDPCCA02 | 07/19/16 17:27 | 6 | 2016/07/19 17:27:08.032 | srvmain | INFO | at org.apache.cata | pcc-wrapper-log | im_file |
Configuration File. I tried multiple regular expressions with no success.
## This is a sample configuration file. See the nxlog reference manual about the
## configuration options. It should be installed locally and is also available
## online at http://nxlog.org/docs/
## Please set the ROOT to the folder your nxlog was installed into,
## otherwise it will not start.
#define ROOT C:\Program Files\nxlog
define ROOT C:\Program Files (x86)\nxlog
Moduledir %ROOT%\modules
CacheDir %ROOT%\data
Pidfile %ROOT%\data\nxlog.pid
SpoolDir %ROOT%\data
LogFile %ROOT%\data\nxlog.log
<Extension gelf>
Module xm_gelf
</Extension>
<Extension fileop>
Module xm_fileop
</Extension>
<Extension multiline>
Module xm_multiline
HeaderLine /^\d{0,2}\/\d{0,2}\/\d{0,4}/
# HeaderLine '^\d{0,2}\/\d{0,2}\/\d{0,4}\ \d{0,3}\:\d{0,3}\:\d{0,3}\.\d{0,4}\ \['
</Extension>
<Input pcc-wrapper-log>
Module im_file
File "C:\\pivotal-tc-server-standard-3.1.0.RELEASE\\pccweb\\logs\\wrapper.log"
SavePos TRUE
InputType multiline
</Input>
<Input pcc-mdstrace-log>
Module im_file
File "C:\\pivotal-tc-server-standard-3.1.0.RELEASE\\pccweb\\logs\\mdstrace.log"
SavePos TRUE
InputType multiline
</Input>
<Input pcc-exceptionHidingUtil-log>
Module im_file
File "C:\\pivotal-tc-server-standard-3.1.0.RELEASE\\pccweb\\logs\\exceptionHidingUtil.log"
SavePos TRUE
InputType multiline
</Input>
<Input pcc-esolutions-log>
Module im_file
File "C:\\pivotal-tc-server-standard-3.1.0.RELEASE\\pccweb\\logs\\esolutions.log"
SavePos TRUE
InputType multiline
</Input>
#<Input pcc-localHostAccess-log>
# Module im_file
# File "C:\\pivotal-tc-server-standard-3.1.0.RELEASE\\pccweb\\logs\\localhost_access_log.*"
# SavePos TRUE
# InputType multiline
#</Input>
<Output graylog>
Module om_udp
Host graylog.genesishcc.com
Port 12201
OutputType GELF
</Output>
<Route PCC>
Path pcc-wrapper-log => pcc-mdstrace-log => pcc-exceptionHidingUtil-log => pcc-esolutions-log => graylog
## Path pcc-wrapper-log => pcc-mdstrace-log => pcc-exceptionHidingUtil-log => pcc-esolutions-log => pcc-localHostAccess-log => graylog
</Route>
Tested the following config with the sample input provided:
<Extension multi> Module xm_multiline HeaderLine /^\d{0,2}\/\d{0,2}\/\d{0,4}/ </Extension> <Extension json> Module xm_json </Extension> <Input in> Module im_file File "tmp/input.txt" ReadFromLast FALSE InputType multi </Input> <Output out> Module om_file Exec $Message = $raw_event; to_json(); File "tmp/output" </Output> <Route 66> Path in => out </Route>
It produced 2 JSON records as expected.
I suggest trimming your config to one input and testing it locally first.
Also this looks a bit odd:
Path pcc-wrapper-log => pcc-mdstrace-log => pcc-exceptionHidingUtil-log => pcc-esolutions-log => graylog
Should be:
Path pcc-wrapper-log, pcc-mdstrace-log, pcc-exceptionHidingUtil-log, pcc-esolutions-log => graylog