responses
Hello,
I currently am sending ms eventlogs from a server to our syslog server, and this is working fine. So I wanted to add to the nxlog conf file to allow it to collect the IIS W3C logs from this server as well and send them to the syslog server. A lot of the support examples out there added scripting to parse the w3c logs to json format. I do not want to do this. Just need to the raw logs to go to our syslog server. Below is my conf file. I'd appreciate pointers and/or the pointing out of errors.
define ROOT C:\Program Files (x86)\nxlog
Moduledir %ROOT%\modules
CacheDir %ROOT%\data
Pidfile %ROOT%\data\nxlog.pid
SpoolDir %ROOT%\data
LogFile %ROOT%\data\nxlog.log
<Extension syslog>
Module xm_syslog
</Extension>
<Extension w3c>
Module xm_csv
Fields $date, $time, $s-sitename, $s-computername, $s_ip, $cs_method, $cs_uri_stem, $cs_uri_query, $s_port, $cs_username, $c_ip, $cs-version, $cs_User_Agent, $cs(Cookie), $cs_Referer, $cs-host, $sc_status, $sc_substatus, $sc_win32_status, $sc-bytes, $cs-bytes, $time_taken
FieldTypes string, string, string, string, string, string, string, string, integer, string, string, string, string, string, string, string, integer, integer, integer, integer, integer, integer
Delimiter ' '
</Extension>
<Input EventLogs>
Module im_msvistalog
</Input>
<Input IIS>
Module im_file
File "C:\\inetpub\\logs\\LogFiles\\W3SVC1\\u_ex*"
SavePos TRUE
UndefValue -
</Input>
<Output Syslog>
Module om_udp
Host 172.16.x.x
Port 514
Exec to_syslog_snare();
</Output>
<Route 1>
Path EventLogs, IIS => Syslog
</Route>
Comments (11)
While you are troubleshooting, the NXLog log file and the configuration verify option is useful (
-v
).For the first error I change
<Extension syslog>
to<Extension _syslog>
.For the second error,
UndefValue
is a property of xm_csv andim_file
does not use it.I would start here and move forward. Note that there are some examples in the manual that do not convert to JSON. Might browse through and see if something there helps as well.
https://nxlog.co/documentation/nxlog-user-guide#iis
I tried running the conf file above and these are the errors it produced.
2018-07-31 07:53:35 ERROR module 'Syslog' is already defined at C:\Program Files (x86)\nxlog\conf\nxlog.conf:30
2018-07-31 07:53:35 ERROR invalid keyword: UndefValue at C:\Program Files (x86)\nxlog\conf\nxlog.conf:26
2018-07-31 07:53:35 ERROR module 'IIS' has configuration errors, not adding to route '1' at C:\Program Files (x86)\nxlog\conf\nxlog.conf:37
2018-07-31 07:53:35 ERROR route 1 is not functional without output modules, ignored at C:\Program Files (x86)\nxlog\conf\nxlog.conf:37
2018-07-31 07:53:35 WARNING no routes defined!
2018-07-31 07:53:35 WARNING not starting module syslog because it is part of an incomplete route
2018-07-31 07:53:35 WARNING not starting unused module EventLogs
2018-07-31 07:53:35 WARNING not starting unused module IIS
2018-07-31 07:53:35 INFO nxlog-ce-2.10.2102 started
Yes. Please see my previous comment.
This is the modified conf file I am using now. There are no errors being produced in the error log. The windows event logs are being sent to the syslog server but none of the iis_w3c logs are, even after generating activity on the webserver. Any thoughts?
define ROOT C:\Program Files (x86)\nxlog
Moduledir %ROOT%\modules
CacheDir %ROOT%\data
Pidfile %ROOT%\data\nxlog.pid
SpoolDir %ROOT%\data
LogFile %ROOT%\data\nxlog.log
<Extension _syslog>
Module xm_syslog
</Extension>
<Extension w3c>
Module xm_csv
Fields $date $time $s-sitename $s-computername $s-ip $cs-method $cs-uri-stem c$s-uri-query $s-port $cs-username $c-ip $cs-version $cs(User-Agent) $cs(Cookie) $cs(Referer) $cs-host $sc-status $sc-substatus $sc-win32-status $sc-bytes $cs-bytes $time-taken
FieldTypes string, string, string, string, string, string, string, string, integer, string, string, string, string, string, string, string, integer, integer, integer, integer, integer, integer
Delimiter ' '
</Extension>
<Input eventlogs>
Module im_msvistalog
</Input>
<Input iis_w3c>
Module im_file
File "C:\\inetpub\\logs\\LogFiles\\W3SVC1\\u_ex*"
</Input>
<Output syslog>
Module om_udp
Host 172.16.x.x
Port 514
Exec to_syslog_snare();
</Output>
<Route 1>
Path eventlogs, iis_w3c => syslog
</Route>
Need to call the CSV parser.
instancename->parse_csv();
in your case:Exec w3c->parse_csv();
https://nxlog.co/documentation/nxlog-user-guide#iis-log-file-format
I would also make sure the files at "C:\inetpub\logs\LogFiles\W3SVC1\u_ex*" are actually being written to and updated.
I made suggested changes to the conf file and the IIS W3C settings and where it is logging to. So far the new conf file is not producing any errors. But I am not seeing anything being sent the syslog server. All ports are open b/c the windows events logs are being collected and sent to the syslog server. Any ideas?
Here is the most recent conf.
define ROOT C:\Program Files (x86)\nxlog
Moduledir %ROOT%\modules
CacheDir %ROOT%\data
Pidfile %ROOT%\data\nxlog.pid
SpoolDir %ROOT%\data
LogFile %ROOT%\data\nxlog.log
<Extension _syslog>
Module xm_syslog
</Extension>
<Extension w3c>
Module xm_csv
Fields $date $time $s-sitename $s-computername $s-ip $cs-method $cs-uri-stem $cs-uri-query $s-port $cs-username $c-ip $cs-version $cs(User-Agent) $cs(Cookie) $cs(Referer) $cs-host $sc-status $sc-substatus $sc-win32-status $sc-bytes $cs-bytes $time-taken
FieldTypes string, string, string, string, string, string, string, string, integer, string, string, string, string, string, string, string, integer, integer, integer, integer, integer, integer
Delimiter ' '
</Extension>
<Input eventlogs>
Module im_msvistalog
</Input>
<Input iis_w3c>
Module im_file
File 'C:\\inetpub\\logs\\LogFiles\\W3SVC1\\u_ex*'
<Exec>
w3c->parse_csv();
$EventTime = strptime($Date + " " + $Time, "%m/%d/%Y %H:%M:%S");
</Exec>
</Input>
<Output syslog>
Module om_udp
Host 172.16.217.9
Port 514
Exec to_syslog_snare();
</Output>
<Route 1>
Path eventlogs, iis_w3c => syslog
</Route>
Here is a sample IIS W3C Log.
#Software: Microsoft Internet Information Services 7.5
#Version: 1.0
#Date: 2018-07-26 00:00:00
#Fields: date time s-sitename s-computername s-ip cs-method cs-uri-stem cs-uri-query s-port cs-username c-ip cs-version cs(User-Agent) cs(Cookie) cs(Referer) cs-host sc-status sc-substatus sc-win32-status sc-bytes cs-bytes time-taken
2018-07-26 00:00:00 W3SVC1 ServerName 172.16.x.x POST /autodiscover/autodiscover.xml - 443 AMCASTLE\user 74.140.145.252 HTTP/1.1 Microsoft+Office/14.0+(Windows+NT+6.1;+Microsoft+Outlook+14.0.6106;+Pro) OutlookSession="{22762E45-D11C-4EA8-9ECC-2CE476F3BC29}" - autodiscover.amcastle.com 200 0 0 4309 1557 109
2018-07-26 00:00:01 W3SVC1 ServerName 172.16.x.x POST /EWS/Exchange.asmx - 443 - 172.16.x.x HTTP/1.1 Mozilla/4.0+(compatible;+MSIE+6.0;+MS+Web+Services+Client+Protocol+2.0.50727.8762) - - outlookcas.amcastle.com 401 0 0 341 330 0
2018-07-26 00:00:01 W3SVC1 ServerName 172.16.x.x POST /EWS/Exchange.asmx - 443 - 172.16.x.x HTTP/1.1 Mozilla/4.0+(compatible;+MSIE+6.0;+MS+Web+Services+Client+Protocol+2.0.50727.8762) - - outlookcas.amcastle.com 401 1 2148074254 677 389 0
I would take this one step at a time. Add a new
<Output>
usingom_file
, comment out your<Output syslog>
and update the<Route 1>
accordingly.Verify that you are getting logs in the output file in the format that you want.
If your output file shows logs as you want, then you need to troubleshoot the connection. Make sure that the packets are being sent and received by the destination (
172.16.217.9
). You can use wireshark or similar for these tests. Make sure your destination is wanting to listen on UDP as well instead of TCP or other.I made the changes suggested. Below are the errors I am getting. I appears that the parser is failing on the header of the W3C logs which are not in CSV format.
2018-08-23 19:00:28 ERROR procedure 'parse_csv' failed at line 26, character 19 in C:\Program Files (x86)\nxlog\conf\nxlog.conf. statement execution has been aborted; Not enough fields in CSV input, expected 22, got 6 in input '#Software: Microsoft Internet Information Services 7.5'
2018-08-23 19:00:28 ERROR procedure 'parse_csv' failed at line 26, character 19 in C:\Program Files (x86)\nxlog\conf\nxlog.conf. statement execution has been aborted; Not enough fields in CSV input, expected 22, got 2 in input '#Version: 1.0'
2018-08-23 19:00:28 ERROR procedure 'parse_csv' failed at line 26, character 19 in C:\Program Files (x86)\nxlog\conf\nxlog.conf. statement execution has been aborted; Not enough fields in CSV input, expected 22, got 3 in input '#Date: 2018-08-24 00:00:00'
2018-08-23 19:00:28 ERROR procedure 'parse_csv' failed at line 26, character 19 in C:\Program Files (x86)\nxlog\conf\nxlog.conf. statement execution has been aborted; cannot parse integer "cs-uri-query", invalid modifier: 'y'
2018-08-24 13:46:29 WARNING stopping nxlog service
2018-08-24 13:46:29 WARNING nxlog-ce received a termination request signal, exiting...
2018-08-24 13:46:30 INFO nxlog-ce-2.10.2102 started
Sample header of W3C logs:
#Software: Microsoft Internet Information Services 7.5
#Version: 1.0
#Date: 2018-07-25 00:00:00
#Fields: date time s-sitename s-computername s-ip cs-method cs-uri-stem cs-uri-query s-port cs-username c-ip cs-version cs(User-Agent) cs(Cookie) cs(Referer) cs-host sc-status sc-substatus sc-win32-status sc-bytes cs-bytes time-taken
2018-07-25 00:00:00 W3SVC1
See the Microsoft IIS section in our user guide on how to parse IIS logs.
I added a new output to send the IIS logs to a local text file, commenting out the old output line. The text file was populating with the correct logs as expected. They appear to be parsed in the correct format too.
Additionally, I am using the same conf file to collect Windows event logs and send them to the Syslog server. They are getting through fine. So there does not appear to be a network issue. ( I can double check with the wireshark. I believe I ran it earlier though and confirmed.
Could there be a specific requirement needed when sending to syslog like utility number, or maybe this line in the conf file, 'Exec to_syslog_snare();' which works for windows event logs?
Pages