Ask questions. Get answers. Find technical product solutions from passionate experts in the NXLog community.

Difficulty Parsing IIS Logs and Sending to Loggly
I'm using NXLog to sending Windows events and IIS logs to Loggly. We've recently onboarded a new MSSP and they have asked us to check off all IIS logging fields. This seems to break parsing of IIS logs that need to be sent to Loggly. I've contact Loggly support and they can't seem to come to a resolution. Below is the code that we had been using for Loggly previously. This is a sample NXLog configuration file created by Loggly. June 2013 See the nxlog reference manual about the configuration options. It should be installed locally and is also available online at http://nxlog.org/nxlog-docs/en/nxlog-reference-manual.html Please set the ROOT to the folder your nxlog was installed into, otherwise it will not start. #define ROOT C:\Program Files\nxlog #define ROOT_STRING C:\Program Files\nxlog define ROOT C:\Program Files (x86)\nxlog define ROOT_STRING C:\Program Files (x86)\nxlog define CERTDIR %ROOT%\cert define LOGFILE %ROOT%\data\nxlog.log Moduledir %ROOT%\modules CacheDir %ROOT%\data Pidfile %ROOT%\data\nxlog.pid SpoolDir %ROOT%\data LogFile %ROOT%\data\nxlog.log Include fileop while debugging, also enable in the output module below <Extension fileop> Module xm_fileop </Extension> <Extension json> Module xm_json </Extension> <Extension syslog> Module xm_syslog </Extension> <Input internal> Module im_internal Exec $Message = to_json(); </Input> Windows Event Log <Input eventlog> #Uncomment im_msvistalog for Windows Vista/2008 and later Module im_msvistalog Query &lt;QueryList&gt;\ &lt;Query Id=&quot;0&quot;&gt;\ &lt;Select Path=&quot;Application&quot;&gt;*&lt;/Select&gt;\ &lt;Select Path=&quot;System&quot;&gt;*&lt;/Select&gt;\ &lt;Select Path=&quot;Security&quot;&gt;*&lt;/Select&gt;\ &lt;/Query&gt;\ &lt;/QueryList&gt; #Uncomment im_mseventlog for Windows XP/2000/2003 #Module im_mseventlog Exec $Message = to_json(); </Input> <Processor buffer> Module pm_buffer # 100Mb disk buffer MaxSize 102400 Type disk </Processor> <Processor buffer_iis> Module pm_buffer # 100Mb disk buffer MaxSize 102400 Type disk </Processor> <Output out> Module om_tcp Host logs-01.loggly.com Port 514 Exec to_syslog_ietf(); Exec $raw_event =~ s/(\[.*] )//g; $raw_event = replace($raw_event, '{', '[CUSTOMER ID tag=&quot;windows&quot;] {', 1); #Use the following line for debugging (uncomment the fileop extension above as well) #Exec file_write(&quot;C:\\Program Files (x86)\\nxlog\\data\\nxlog_output.log&quot;, $raw_event); </Output> <Output _nxlog> Module om_file File '%LOGFILE%' &lt;Schedule&gt; When @hourly Exec if (file_size('%LOGFILE%') &gt;= 1M) { file_cycle('%LOGFILE%', 5); _nxlog-&gt;reopen(); } &lt;/Schedule&gt; </Output> <Route 1> Path internal, eventlog => buffer => out </Route> <Route 2> Path internal => _nxlog </Route> Create the parse rule for IIS logs. You can copy these from the header of the IIS log file. <Extension w3c> Module xm_csv Fields $date, $time, $s-computername, $cs-method, $cs-uri-stem, $cs-uri-query, $c-ip, $cs(User-Agent), $cs(Referer), $cs-host, $sc-status, $sc-substatus, $sc-bytes, $cs-bytes, $time-taken, X-Forwarded-For, RequestId, PrincipalId FieldTypes string, string, string, string, string, string, string, string, string, string, string, string, integer, integer, integer, string, string, string Delimiter ' ' QuoteChar '"' EscapeControl FALSE UndefValue - </Extension> Convert the IIS logs to JSON and use the original event time <Input SC> Module im_file File "C:\inetpub\logs\LogFiles\W3SVC1\u_ex*" SavePos TRUE Exec if $raw_event =~ /^#/ drop(); \ else \ { \ w3c-&gt;parse_csv(); \ $SourceName = &quot;IIS&quot;; \ $Message = to_json(); \ } </Input> <Input SC_WebAPI> Module im_file File "C:\inetpub\logs\LogFiles\W3SVC2\u_ex*" SavePos TRUE Exec if $raw_event =~ /^#/ drop(); \ else \ { \ w3c-&gt;parse_csv(); \ $SourceName = &quot;IIS&quot;; \ $Message = to_json(); \ } </Input> <Route IIS> Path SC,SC_WebAPI => buffer_iis => out </Route> The error we received in data.log looks like below: 2021-02-18 14:11:12 "SERVERNAME" ERROR if-else failed at line 144, character 261 in C:\Program Files (x86)\nxlog\conf\nxlog.conf. statement execution has been aborted procedure 'parse_csv' failed at line 144, character 156 in C:\Program Files (x86)\nxlog\conf\nxlog.conf. statement execution has been aborted couldn't parse integer: ELB-HealthChecker/2.0 2021-02-18 14:11:26 "SERVERNAME" ERROR if-else failed at line 131, character 261 in C:\Program Files (x86)\nxlog\conf\nxlog.conf. statement execution has been aborted procedure 'parse_csv' failed at line 131, character 156 in C:\Program Files (x86)\nxlog\conf\nxlog.conf. statement execution has been aborted couldn't parse integer: ELB-HealthChecker/2.0 It seems as if the cs-useragent is being evaluated as an integer. or possible the input has fewer fields than expected. Any help would be appreciated.

BB_838545 created
Parsing Windows Logs (from FILE)
I am having no luck with a simple parsing of EVT log files. Is there an easy way to read in EVT (Binary Log files) and output them in Syslog Format? This is the config file I am using: (I Used python evtx to extract into text XML) However that yields XML attributes which apparently are not parse-able. Problem Set: Give 3 files (System.evt, Application.evt, and Security.EVT) parse the EVT format into Syslog_BSD(or IETF) formats.     <Extension multiline>     Module    xm_multiline     HeaderLine    /^<event>/     EndLine    /^</event>/ </Extension> <Extension syslog> Module    xm_syslog </Extension> <Extension xmlparser> Module    xm_xml </Extension> <Extension json> Module    xm_json </Extension> <Extension fileop> Module xm_fileop </Extension> <Input in>     Module im_file     File "%ROOT%/test.xml" #    File "/tmp/cab.xml"     SavePos    FALSE     ReadFromLast FALSE     InputType    multiline     <Exec>       # Discard everything that doesn't seem to be an xml event          if $raw_event !~ /^<event>/ drop();       # Parse the xml event       parse_xml(); to_syslog_ietf();       # Rewrite some fields        $EventTime = parsedate($timestamp);       delete($timestamp);       delete($EventReceivedTime);       # Convert to JSON       to_json();     </Exec> </Input> <Output out>     Module  om_file     File    "%ROOT%/out.log"     Exec    parse_xml();     Exec     log_info("FIELD" +  to_json()); </Output> <Route 1>     Path    in => out </Route>

Colin.Bitterfield created
Replies: 1
View post »
last updated
NXLog Parsing XML
I've seen some posts from about a year ago that NXLog is unable to parse attributes using xm_xml, I just wanted to check if this is still true? I am running NXLog as a service on Windows machines and want to be able to parse the following message, is it possible? <log4j:event logger="com.sentry.test.LogContextListener" timestamp="1437661699866" level="TRACE" thread="localhost-startStop-1"> <log4j:message><![CDATA[This is a trace message about how we should use C#]]></log4j:message> </log4j:event>

Jakauppila created
Replies: 1
View post »
last updated