Ask questions. Get answers. Find technical product solutions from passionate experts in the NXLog community.
Difficulty Parsing IIS Logs and Sending to Loggly
BB_838545 created
I'm using NXLog to sending Windows events and IIS logs to Loggly. We've recently onboarded a new MSSP and they have asked us to check off all IIS logging fields. This seems to break parsing of IIS logs that need to be sent to Loggly. I've contact Loggly support and they can't seem to come to a resolution.
Below is the code that we had been using for Loggly previously.
This is a sample NXLog configuration file created by Loggly. June 2013
See the nxlog reference manual about the configuration options.
It should be installed locally and is also available
online at http://nxlog.org/nxlog-docs/en/nxlog-reference-manual.html
Please set the ROOT to the folder your nxlog was installed into,
otherwise it will not start.
#define ROOT C:\Program Files\nxlog
#define ROOT_STRING C:\Program Files\nxlog
define ROOT C:\Program Files (x86)\nxlog
define ROOT_STRING C:\Program Files (x86)\nxlog
define CERTDIR %ROOT%\cert
define LOGFILE %ROOT%\data\nxlog.log
Moduledir %ROOT%\modules
CacheDir %ROOT%\data
Pidfile %ROOT%\data\nxlog.pid
SpoolDir %ROOT%\data
LogFile %ROOT%\data\nxlog.log
Include fileop while debugging, also enable in the output module below
<Extension fileop>
Module xm_fileop
</Extension>
<Extension json>
Module xm_json
</Extension>
<Extension syslog>
Module xm_syslog
</Extension>
<Input internal>
Module im_internal
Exec $Message = to_json();
</Input>
Windows Event Log
<Input eventlog>
#Uncomment im_msvistalog for Windows Vista/2008 and later
Module im_msvistalog
Query <QueryList>\
<Query Id="0">\
<Select Path="Application">*</Select>\
<Select Path="System">*</Select>\
<Select Path="Security">*</Select>\
</Query>\
</QueryList>
#Uncomment im_mseventlog for Windows XP/2000/2003
#Module im_mseventlog
Exec $Message = to_json();
</Input>
<Processor buffer>
Module pm_buffer
# 100Mb disk buffer
MaxSize 102400
Type disk
</Processor>
<Processor buffer_iis>
Module pm_buffer
# 100Mb disk buffer
MaxSize 102400
Type disk
</Processor>
<Output out>
Module om_tcp
Host logs-01.loggly.com
Port 514
Exec to_syslog_ietf();
Exec $raw_event =~ s/(\[.*] )//g; $raw_event = replace($raw_event, '{', '[CUSTOMER ID tag="windows"] {', 1);
#Use the following line for debugging (uncomment the fileop extension above as well)
#Exec file_write("C:\\Program Files (x86)\\nxlog\\data\\nxlog_output.log", $raw_event);
</Output>
<Output _nxlog>
Module om_file
File '%LOGFILE%'
<Schedule>
When @hourly
Exec if (file_size('%LOGFILE%') >= 1M) { file_cycle('%LOGFILE%', 5); _nxlog->reopen(); }
</Schedule>
</Output>
<Route 1>
Path internal, eventlog => buffer => out
</Route>
<Route 2>
Path internal => _nxlog
</Route>
Create the parse rule for IIS logs. You can copy these from the header of the IIS log file.
<Extension w3c>
Module xm_csv
Fields $date, $time, $s-computername, $cs-method, $cs-uri-stem, $cs-uri-query, $c-ip, $cs(User-Agent), $cs(Referer), $cs-host, $sc-status, $sc-substatus, $sc-bytes, $cs-bytes, $time-taken, X-Forwarded-For, RequestId, PrincipalId
FieldTypes string, string, string, string, string, string, string, string, string, string, string, string, integer, integer, integer, string, string, string
Delimiter ' '
QuoteChar '"'
EscapeControl FALSE
UndefValue -
</Extension>
Convert the IIS logs to JSON and use the original event time
<Input SC>
Module im_file
File "C:\inetpub\logs\LogFiles\W3SVC1\u_ex*"
SavePos TRUE
Exec if $raw_event =~ /^#/ drop(); \
else \
{ \
w3c->parse_csv(); \
$SourceName = "IIS"; \
$Message = to_json(); \
}
</Input>
<Input SC_WebAPI>
Module im_file
File "C:\inetpub\logs\LogFiles\W3SVC2\u_ex*"
SavePos TRUE
Exec if $raw_event =~ /^#/ drop(); \
else \
{ \
w3c->parse_csv(); \
$SourceName = "IIS"; \
$Message = to_json(); \
}
</Input>
<Route IIS>
Path SC,SC_WebAPI => buffer_iis => out
</Route>
The error we received in data.log looks like below:
2021-02-18 14:11:12 "SERVERNAME" ERROR if-else failed at line 144, character 261 in C:\Program Files (x86)\nxlog\conf\nxlog.conf. statement execution has been aborted
procedure 'parse_csv' failed at line 144, character 156 in C:\Program Files (x86)\nxlog\conf\nxlog.conf. statement execution has been aborted
couldn't parse integer: ELB-HealthChecker/2.0
2021-02-18 14:11:26 "SERVERNAME" ERROR if-else failed at line 131, character 261 in C:\Program Files (x86)\nxlog\conf\nxlog.conf. statement execution has been aborted
procedure 'parse_csv' failed at line 131, character 156 in C:\Program Files (x86)\nxlog\conf\nxlog.conf. statement execution has been aborted
couldn't parse integer: ELB-HealthChecker/2.0
It seems as if the cs-useragent is being evaluated as an integer. or possible the input has fewer fields than expected.
Any help would be appreciated.
BB_838545 created
Parsing Windows Logs (from FILE)
Colin.Bitterfield created
I am having no luck with a simple parsing of EVT log files.
Is there an easy way to read in EVT (Binary Log files) and output them in Syslog Format?
This is the config file I am using: (I Used python evtx to extract into text XML) However that yields XML attributes which apparently are not parse-able.
Problem Set:
Give 3 files (System.evt, Application.evt, and Security.EVT) parse the EVT format into Syslog_BSD(or IETF) formats.
<Extension multiline>
Module xm_multiline
HeaderLine /^<event>/
EndLine /^</event>/
</Extension>
<Extension syslog>
Module xm_syslog
</Extension>
<Extension xmlparser>
Module xm_xml
</Extension>
<Extension json>
Module xm_json
</Extension>
<Extension fileop>
Module xm_fileop
</Extension>
<Input in>
Module im_file
File "%ROOT%/test.xml"
# File "/tmp/cab.xml"
SavePos FALSE
ReadFromLast FALSE
InputType multiline
<Exec>
# Discard everything that doesn't seem to be an xml event
if $raw_event !~ /^<event>/ drop();
# Parse the xml event
parse_xml(); to_syslog_ietf();
# Rewrite some fields
$EventTime = parsedate($timestamp);
delete($timestamp);
delete($EventReceivedTime);
# Convert to JSON
to_json();
</Exec>
</Input>
<Output out>
Module om_file
File "%ROOT%/out.log"
Exec parse_xml();
Exec log_info("FIELD" + to_json());
</Output>
<Route 1>
Path in => out
</Route>
Colin.Bitterfield created
NXLog Parsing XML
Jakauppila created
I've seen some posts from about a year ago that NXLog is unable to parse attributes using xm_xml, I just wanted to check if this is still true?
I am running NXLog as a service on Windows machines and want to be able to parse the following message, is it possible?
<log4j:event logger="com.sentry.test.LogContextListener" timestamp="1437661699866" level="TRACE" thread="localhost-startStop-1"> <log4j:message><![CDATA[This is a trace message about how we should use C#]]></log4j:message> </log4j:event>
Jakauppila created