Sending IIS W3C to Syslog

Post a different question
11
responses

Hello,

I currently am sending ms eventlogs from a server to our syslog server, and this is working fine. So I wanted to add to the nxlog conf file to allow it to collect the IIS W3C logs from this server as well and send them to the syslog server. A lot of the support examples out there added scripting to parse the w3c logs to json format. I do not want to do this. Just need to the raw logs to go to our syslog server. Below is my conf file. I'd appreciate pointers and/or the pointing out of errors.

define ROOT C:\Program Files (x86)\nxlog

Moduledir %ROOT%\modules
CacheDir %ROOT%\data
Pidfile %ROOT%\data\nxlog.pid
SpoolDir %ROOT%\data
LogFile %ROOT%\data\nxlog.log

<Extension syslog>
Module xm_syslog
</Extension>
<Extension w3c>
Module xm_csv
Fields $date, $time, $s-sitename, $s-computername, $s_ip, $cs_method, $cs_uri_stem, $cs_uri_query, $s_port, $cs_username, $c_ip, $cs-version, $cs_User_Agent, $cs(Cookie), $cs_Referer, $cs-host, $sc_status, $sc_substatus, $sc_win32_status, $sc-bytes, $cs-bytes, $time_taken
FieldTypes string, string, string, string, string, string, string, string, integer, string, string, string, string, string, string, string, integer, integer, integer, integer, integer, integer
Delimiter ' '
</Extension>

<Input EventLogs>
Module im_msvistalog
</Input>
<Input IIS>
Module im_file
File "C:\\inetpub\\logs\\LogFiles\\W3SVC1\\u_ex*"
SavePos TRUE
UndefValue -
</Input>

<Output Syslog>
Module om_udp
Host 172.16.x.x
Port 514
Exec to_syslog_snare();
</Output>

<Route 1>
Path EventLogs, IIS => Syslog
</Route>

AskedJuly 30, 2018 - 8:29pm

Comments (11)

  • Zhengshi's picture
    (NXLog)

    While you are troubleshooting, the NXLog log file and the configuration verify option is useful (-v).

    C:\Program Files\nxlog>nxlog.exe -f -v -c conf/test.conf
2018-07-10 11:59:36 ERROR module 'Syslog' is already defined at conf/test.conf:30

C:\Program Files\nxlog>nxlog.exe -f -v -c conf/test.conf
2018-07-10 12:00:09 ERROR invalid keyword: UndefValue at conf/test.conf:26

    For the first error I change <Extension syslog> to <Extension _syslog>.
    For the second error, UndefValue is a property of xm_csv and im_file does not use it.

    I would start here and move forward. Note that there are some examples in the manual that do not convert to JSON. Might browse through and see if something there helps as well.
    https://nxlog.co/documentation/nxlog-user-guide#iis

    July 30, 2018 - 11:00pm
  • jmeyers's picture

    I tried running the conf file above and these are the errors it produced.

    2018-07-31 07:53:35 ERROR module 'Syslog' is already defined at C:\Program Files (x86)\nxlog\conf\nxlog.conf:30
    2018-07-31 07:53:35 ERROR invalid keyword: UndefValue at C:\Program Files (x86)\nxlog\conf\nxlog.conf:26
    2018-07-31 07:53:35 ERROR module 'IIS' has configuration errors, not adding to route '1' at C:\Program Files (x86)\nxlog\conf\nxlog.conf:37
    2018-07-31 07:53:35 ERROR route 1 is not functional without output modules, ignored at C:\Program Files (x86)\nxlog\conf\nxlog.conf:37
    2018-07-31 07:53:35 WARNING no routes defined!
    2018-07-31 07:53:35 WARNING not starting module syslog because it is part of an incomplete route
    2018-07-31 07:53:35 WARNING not starting unused module EventLogs
    2018-07-31 07:53:35 WARNING not starting unused module IIS
    2018-07-31 07:53:35 INFO nxlog-ce-2.10.2102 started

    July 31, 2018 - 3:20pm
  • jmeyers's picture

    This is the modified conf file I am using now. There are no errors being produced in the error log. The windows event logs are being sent to the syslog server but none of the iis_w3c logs are, even after generating activity on the webserver. Any thoughts?

    define ROOT C:\Program Files (x86)\nxlog

    Moduledir %ROOT%\modules
    CacheDir %ROOT%\data
    Pidfile %ROOT%\data\nxlog.pid
    SpoolDir %ROOT%\data
    LogFile %ROOT%\data\nxlog.log

    <Extension _syslog>
    Module xm_syslog
    </Extension>
    <Extension w3c>
    Module xm_csv
    Fields $date $time $s-sitename $s-computername $s-ip $cs-method $cs-uri-stem c$s-uri-query $s-port $cs-username $c-ip $cs-version $cs(User-Agent) $cs(Cookie) $cs(Referer) $cs-host $sc-status $sc-substatus $sc-win32-status $sc-bytes $cs-bytes $time-taken
    FieldTypes string, string, string, string, string, string, string, string, integer, string, string, string, string, string, string, string, integer, integer, integer, integer, integer, integer
    Delimiter ' '
    </Extension>

    <Input eventlogs>
    Module im_msvistalog
    </Input>
    <Input iis_w3c>
    Module im_file
    File "C:\\inetpub\\logs\\LogFiles\\W3SVC1\\u_ex*"
    </Input>

    <Output syslog>
    Module om_udp
    Host 172.16.x.x
    Port 514
    Exec to_syslog_snare();
    </Output>

    <Route 1>
    Path eventlogs, iis_w3c => syslog
    </Route>

    July 31, 2018 - 11:26pm
  • Zhengshi's picture
    (NXLog)

    Need to call the CSV parser. instancename->parse_csv(); in your case: Exec w3c->parse_csv();

    https://nxlog.co/documentation/nxlog-user-guide#iis-log-file-format

    I would also make sure the files at "C:\inetpub\logs\LogFiles\W3SVC1\u_ex*" are actually being written to and updated.

    August 1, 2018 - 3:41am
  • jmeyers's picture

    I made suggested changes to the conf file and the IIS W3C settings and where it is logging to. So far the new conf file is not producing any errors. But I am not seeing anything being sent the syslog server. All ports are open b/c the windows events logs are being collected and sent to the syslog server. Any ideas?

    Here is the most recent conf.
    define ROOT C:\Program Files (x86)\nxlog

    Moduledir %ROOT%\modules
    CacheDir %ROOT%\data
    Pidfile %ROOT%\data\nxlog.pid
    SpoolDir %ROOT%\data
    LogFile %ROOT%\data\nxlog.log

    <Extension _syslog>
    Module xm_syslog
    </Extension>
    <Extension w3c>
    Module xm_csv
    Fields $date $time $s-sitename $s-computername $s-ip $cs-method $cs-uri-stem $cs-uri-query $s-port $cs-username $c-ip $cs-version $cs(User-Agent) $cs(Cookie) $cs(Referer) $cs-host $sc-status $sc-substatus $sc-win32-status $sc-bytes $cs-bytes $time-taken
    FieldTypes string, string, string, string, string, string, string, string, integer, string, string, string, string, string, string, string, integer, integer, integer, integer, integer, integer
    Delimiter ' '
    </Extension>

    <Input eventlogs>
    Module im_msvistalog
    </Input>
    <Input iis_w3c>
    Module im_file
    File 'C:\\inetpub\\logs\\LogFiles\\W3SVC1\\u_ex*'
    <Exec>
    w3c->parse_csv();
    $EventTime = strptime($Date + " " + $Time, "%m/%d/%Y %H:%M:%S");
    </Exec>
    </Input>

    <Output syslog>
    Module om_udp
    Host 172.16.217.9
    Port 514
    Exec to_syslog_snare();
    </Output>

    <Route 1>
    Path eventlogs, iis_w3c => syslog
    </Route>

    Here is a sample IIS W3C Log.
    #Software: Microsoft Internet Information Services 7.5
    #Version: 1.0
    #Date: 2018-07-26 00:00:00
    #Fields: date time s-sitename s-computername s-ip cs-method cs-uri-stem cs-uri-query s-port cs-username c-ip cs-version cs(User-Agent) cs(Cookie) cs(Referer) cs-host sc-status sc-substatus sc-win32-status sc-bytes cs-bytes time-taken
    2018-07-26 00:00:00 W3SVC1 ServerName 172.16.x.x POST /autodiscover/autodiscover.xml - 443 AMCASTLE\user 74.140.145.252 HTTP/1.1 Microsoft+Office/14.0+(Windows+NT+6.1;+Microsoft+Outlook+14.0.6106;+Pro) OutlookSession="{22762E45-D11C-4EA8-9ECC-2CE476F3BC29}" - autodiscover.amcastle.com 200 0 0 4309 1557 109
    2018-07-26 00:00:01 W3SVC1 ServerName 172.16.x.x POST /EWS/Exchange.asmx - 443 - 172.16.x.x HTTP/1.1 Mozilla/4.0+(compatible;+MSIE+6.0;+MS+Web+Services+Client+Protocol+2.0.50727.8762) - - outlookcas.amcastle.com 401 0 0 341 330 0
    2018-07-26 00:00:01 W3SVC1 ServerName 172.16.x.x POST /EWS/Exchange.asmx - 443 - 172.16.x.x HTTP/1.1 Mozilla/4.0+(compatible;+MSIE+6.0;+MS+Web+Services+Client+Protocol+2.0.50727.8762) - - outlookcas.amcastle.com 401 1 2148074254 677 389 0

    August 9, 2018 - 6:51pm
  • Zhengshi's picture
    (NXLog)

    I would take this one step at a time. Add a new <Output> using om_file, comment out your <Output syslog> and update the <Route 1> accordingly.
    Verify that you are getting logs in the output file in the format that you want.

    If your output file shows logs as you want, then you need to troubleshoot the connection. Make sure that the packets are being sent and received by the destination (172.16.217.9). You can use wireshark or similar for these tests. Make sure your destination is wanting to listen on UDP as well instead of TCP or other.

    August 16, 2018 - 5:23pm
  • jmeyers's picture

    I made the changes suggested. Below are the errors I am getting. I appears that the parser is failing on the header of the W3C logs which are not in CSV format.

    2018-08-23 19:00:28 ERROR procedure 'parse_csv' failed at line 26, character 19 in C:\Program Files (x86)\nxlog\conf\nxlog.conf. statement execution has been aborted; Not enough fields in CSV input, expected 22, got 6 in input '#Software: Microsoft Internet Information Services 7.5'
    2018-08-23 19:00:28 ERROR procedure 'parse_csv' failed at line 26, character 19 in C:\Program Files (x86)\nxlog\conf\nxlog.conf. statement execution has been aborted; Not enough fields in CSV input, expected 22, got 2 in input '#Version: 1.0'
    2018-08-23 19:00:28 ERROR procedure 'parse_csv' failed at line 26, character 19 in C:\Program Files (x86)\nxlog\conf\nxlog.conf. statement execution has been aborted; Not enough fields in CSV input, expected 22, got 3 in input '#Date: 2018-08-24 00:00:00'
    2018-08-23 19:00:28 ERROR procedure 'parse_csv' failed at line 26, character 19 in C:\Program Files (x86)\nxlog\conf\nxlog.conf. statement execution has been aborted; cannot parse integer "cs-uri-query", invalid modifier: 'y'
    2018-08-24 13:46:29 WARNING stopping nxlog service
    2018-08-24 13:46:29 WARNING nxlog-ce received a termination request signal, exiting...
    2018-08-24 13:46:30 INFO nxlog-ce-2.10.2102 started

    Sample header of W3C logs:
    #Software: Microsoft Internet Information Services 7.5
    #Version: 1.0
    #Date: 2018-07-25 00:00:00
    #Fields: date time s-sitename s-computername s-ip cs-method cs-uri-stem cs-uri-query s-port cs-username c-ip cs-version cs(User-Agent) cs(Cookie) cs(Referer) cs-host sc-status sc-substatus sc-win32-status sc-bytes cs-bytes time-taken
    2018-07-25 00:00:00 W3SVC1

    August 24, 2018 - 9:39pm
  • jmeyers's picture

    I added a new output to send the IIS logs to a local text file, commenting out the old output line. The text file was populating with the correct logs as expected. They appear to be parsed in the correct format too.
    Additionally, I am using the same conf file to collect Windows event logs and send them to the Syslog server. They are getting through fine. So there does not appear to be a network issue. ( I can double check with the wireshark. I believe I ran it earlier though and confirmed.

    Could there be a specific requirement needed when sending to syslog like utility number, or maybe this line in the conf file, 'Exec to_syslog_snare();' which works for windows event logs?

    October 10, 2018 - 11:50pm

Pages

Answers (0)