How Do I Send IIS v8.5 Logs in W3C Format to a Linux Syslog Server?
Tags:
nxlog configuration
#1
bk
I would like to send IIS v8.5 logs over to a Linux syslog server. I have all parts installed, but need help with the nxlog agent configuration on the IIS server (Win2012 R2). My current configuration is attached.
There are errors in the Win2012 nxlog agent's log file, and I am unable to fix them all, they are attached as well. I had to comment out parts like writing to a local file in order to get the agent running. I would like to have that work as well. It created the file but it's empty.
Config (errors are further below):
## Please set the ROOT to the folder your nxlog was installed into,
## otherwise it will not start.
#define ROOT C:\Program Files\nxlog
define ROOT C:\Program Files (x86)\nxlog
define CERTDIR %ROOT%\cert
Moduledir %ROOT%\modules
CacheDir %ROOT%\data
Pidfile %ROOT%\data\nxlog.pid
SpoolDir %ROOT%\data
LogFile %ROOT%\data\nxlog.log
Module xm_syslog
#
Module xm_fileop
Module xm_csv
Fields date, time, s-ip, cs-method, cs-uri-stem, cs-uri-query, \
s-port, cs-username, c-ip, cs(User-Agent), cs(Referer), \
sc-status, sc-substatus, sc-win32-status, time-taken
FieldTypes string, string, string, string, string, string, integer, \
string, string, string, string, integer, integer, integer, \
integer
Delimiter ' '
EscapeChar '"'
QuoteChar '"'
EscapeControl FALSE
UndefValue -
Module im_file
File "C:\\inetpub\\logs\\LogFiles\\W3SVC1\\\\*.log"
if $raw_event =~ /^#/ drop();
else
{
w3c_parser->parse_csv();
$EventTime = parsedate($date + "T" + $time + ".000Z");
}
#
# Module om_tcp
# File 'C:\outputiis.log'
# Exec to_syslog_bsd();
#
Module om_tcp
Host 10.0.3.163
Port 514
Exec to_syslog_bsd();
Path iis_w3c => out_tcp
Path iis_w3c => out_file_iis
Module xm_json
Module xm_charconv
AutodetectCharsets utf-8, euc-jp, utf-16, utf-32, iso8859-2, ucs-2le
##########################################
## NXLOG INTERNAL LOG ##
##########################################
# Nxlog internal logs - Recommended to keep this turned ON so error(s)/Issues with NXLog are reported.
Module im_internal
Exec $Hostname = hostname_fqdn();
Exec $EventReceivedTime = integer($EventReceivedTime) / 1000000; to_json();
##########################################
## FLAT FILES ##
##########################################
## http://nxlog-ce.sourceforge.net/nxlog-docs/en/nxlog-reference-manual.html#im_file
## Input to watch a file of your choosing. After Input, name it whatever you want to describe that NXLog
## is pulling, then add that name to the path in Route 1 after eventlog. Can be separated for filtering diff logs.
## After setting the Message as raw_event this converts the message to UTF-8, drops empty messages
## removes extra whitespace, grabs the file name as LogFile, adds the FQDN, and deletes a useless var
## Pulls all logfiles from the default ITS Log Location
## It is Recommended to LEAVE THIS ENABLED
## Ensure that "ITS_Logs" is specified in the correct Route at the bottom for output
Module im_file
File "C:\\ITS\\Logs\\\\*.log"
SavePos TRUE
Recursive TRUE
Exec $Message = $raw_event;
Exec $Message = convert($Message, "ucs-2le", "utf-8");
Exec if $Message == '' drop();
Exec if $Message =~ s/^\s+//g log_debug("whitespace removed");
Exec if file_name() =~ /([^\\]+)$/ $LogFile = $1;
Exec $Hostname = hostname_fqdn();
Exec delete($SourceModuleType);
Exec $EventTime = $EventReceivedTime;
Exec $EventReceivedTime = integer($EventReceivedTime) / 1000000;
Exec to_json();
Error Message:
2019-09-11 12:08:56 ERROR if-else failed at line 46, character 9 in C:\Program Files (x86)\nxlog\conf\nxlog.conf. statement execution has been aborted; procedure 'parse_csv' failed at line 44, character 36 in C:\Program Files (x86)\nxlog\conf\nxlog.conf. statement execution has been aborted; cannot parse integer "/", invalid modifier: '/'
2019-09-11 12:09:26 ERROR last message repeated 5 times
#1
bk
I would like to send IIS v8.5 logs over to a Linux syslog server. I have all parts installed, but need help with the nxlog agent configuration on the IIS server (Win2012 R2). My current configuration is attached.
There are errors in the Win2012 nxlog agent's log file, and I am unable to fix them all, they are attached as well. I had to comment out parts like writing to a local file in order to get the agent running. I would like to have that work as well. It created the file but it's empty.
Config (errors are further below):
## Please set the ROOT to the folder your nxlog was installed into,
## otherwise it will not start.
#define ROOT C:\Program Files\nxlog
define ROOT C:\Program Files (x86)\nxlog
define CERTDIR %ROOT%\cert
Moduledir %ROOT%\modules
CacheDir %ROOT%\data
Pidfile %ROOT%\data\nxlog.pid
SpoolDir %ROOT%\data
LogFile %ROOT%\data\nxlog.log
<Extension syslog>
Module xm_syslog
</Extension>
<Extension fileop>#
Module xm_fileop
</Extension>
<Extension w3c_parser>
Module xm_csv
Fields date, time, s-ip, cs-method, cs-uri-stem, cs-uri-query, \
s-port, cs-username, c-ip, cs(User-Agent), cs(Referer), \
sc-status, sc-substatus, sc-win32-status, time-taken
FieldTypes string, string, string, string, string, string, integer, \
string, string, string, string, integer, integer, integer, \
integer
Delimiter ' '
EscapeChar '"'
QuoteChar '"'
EscapeControl FALSE
UndefValue -
</Extension>
<Input iis_w3c>
Module im_file
File "C:\\inetpub\\logs\\LogFiles\\W3SVC1\\\\*.log"
<Exec>
if $raw_event =~ /^#/ drop();
else
{
w3c_parser->parse_csv();
$EventTime = parsedate($date + "T" + $time + ".000Z");
}
</Exec>
</Input>
#<Output out_file_iis>
# Module om_tcp
# File 'C:\outputiis.log'
# Exec to_syslog_bsd();
#</Output>
<Output out_tcp>
Module om_tcp
Host 10.0.3.163
Port 514
Exec to_syslog_bsd();
</Output>
<Route send_iis_to_syslog_server>
Path iis_w3c => out_tcp
</Route>
<Route iis>
Path iis_w3c => out_file_iis
</Route>
<Extension json>
Module xm_json
</Extension>
<Extension charconv>
Module xm_charconv
AutodetectCharsets utf-8, euc-jp, utf-16, utf-32, iso8859-2, ucs-2le
</Extension>
##########################################
## NXLOG INTERNAL LOG ##
##########################################
# Nxlog internal logs - Recommended to keep this turned ON so error(s)/Issues with NXLog are reported.
<Input internal>
Module im_internal
Exec $Hostname = hostname_fqdn();
Exec $EventReceivedTime = integer($EventReceivedTime) / 1000000; to_json();
</Input>
##########################################
## FLAT FILES ##
##########################################
## http://nxlog-ce.sourceforge.net/nxlog-docs/en/nxlog-reference-manual.html#im_file
## Input to watch a file of your choosing. After Input, name it whatever you want to describe that NXLog
## is pulling, then add that name to the path in Route 1 after eventlog. Can be separated for filtering diff logs.
## After setting the Message as raw_event this converts the message to UTF-8, drops empty messages
## removes extra whitespace, grabs the file name as LogFile, adds the FQDN, and deletes a useless var
## Pulls all logfiles from the default ITS Log Location
## It is Recommended to LEAVE THIS ENABLED
## Ensure that "ITS_Logs" is specified in the correct Route at the bottom for output
<Input ITS_Logs>
Module im_file
File "C:\\ITS\\Logs\\\\*.log"
SavePos TRUE
Recursive TRUE
Exec $Message = $raw_event;
Exec $Message = convert($Message, "ucs-2le", "utf-8");
Exec if $Message == '' drop();
Exec if $Message =~ s/^\s+//g log_debug("whitespace removed");
Exec if file_name() =~ /([^\\]+)$/ $LogFile = $1;
Exec $Hostname = hostname_fqdn();
Exec delete($SourceModuleType);
Exec $EventTime = $EventReceivedTime;
Exec $EventReceivedTime = integer($EventReceivedTime) / 1000000;
Exec to_json();
</Input>
Error Message:
2019-09-11 12:08:56 ERROR if-else failed at line 46, character 9 in C:\Program Files (x86)\nxlog\conf\nxlog.conf. statement execution has been aborted; procedure 'parse_csv' failed at line 44, character 36 in C:\Program Files (x86)\nxlog\conf\nxlog.conf. statement execution has been aborted; cannot parse integer "/", invalid modifier: '/'
2019-09-11 12:09:26 ERROR last message repeated 5 times
2019-09-11 12:08:56 ERROR if-else failed at line 46, character 9 in C:\Program Files (x86)\nxlog\conf\nxlog.conf. statement execution has been aborted; procedure 'parse_csv' failed at line 44, character 36 in C:\Program Files (x86)\nxlog\conf\nxlog.conf. statement execution has been aborted; cannot parse integer "/", invalid modifier: '/'
This error message seems to indicate that one of the fields is expected to be an integer, but is seen as / instead.
In our example in the manual (which looks like you are using), we are expecting the default fields. If the fields have changed or been modified in the IIS Manager, you will need to update the CSV fields appropriately. You can see the fields if you open the log file.
Below is an example log file from one of my test servers. You will notice that I have changed the fields at some point, so my #Fields line was updated.
These are the fields you have to match in the xm_csv Fields/FieldTypes sections. Again, make sure it matches the fields you see in the file on your system.
#Software: Microsoft Internet Information Services 10.0
#Version: 1.0
#Date: 2019-08-30 22:52:31
#Fields: date time s-ip cs-method cs-uri-stem cs-uri-query s-port cs-username c-ip cs(User-Agent) cs(Referer) sc-status sc-substatus sc-win32-status time-taken
2019-08-30 22:52:31 127.0.0.1 GET / - 80 - 127.0.0.1 Mozilla/5.0+(Windows+NT+10.0;+WOW64;+Trident/7.0;+rv:11.0)+like+Gecko - 200 0 0 93
2019-08-30 22:52:31 127.0.0.1 GET /iisstart.png - 80 - 127.0.0.1 Mozilla/5.0+(Windows+NT+10.0;+WOW64;+Trident/7.0;+rv:11.0)+like+Gecko http://127.0.0.1/ 200 0 0 0
2019-08-30 22:52:31 127.0.0.1 GET /favicon.ico - 80 - 127.0.0.1 Mozilla/5.0+(Windows+NT+10.0;+WOW64;+Trident/7.0;+rv:11.0)+like+Gecko - 404 0 2 15
#Software: Microsoft Internet Information Services 10.0
#Version: 1.0
#Date: 2019-08-30 22:55:44
#Fields: date time s-sitename s-computername s-ip cs-method cs-uri-stem cs-uri-query s-port cs-username c-ip cs(User-Agent) cs(Referer) sc-status sc-substatus sc-win32-status time-taken
2019-08-30 22:55:43 W3SVC1 WIN-LU43V8BOQ6J 127.0.0.1 GET /favicon.ico - 80 - 127.0.0.1 Mozilla/5.0+(Windows+NT+10.0;+WOW64;+Trident/7.0;+rv:11.0)+like+Gecko - 404 0 2 0
