IIS logs sent via NXlog not showing up in Graylog

Tags: graylog iis

#1 Nathan.Reid 9 years ago

I'm having trouble getting my IIS logs into Graylog.

My Windows event logs show up just fine, but the IIS logs never show up.

I'm using NXlog per the Graylog docs. I've verified that the logs are getting sent (I have them also writing to a file, and I've checked with Wireshark to make sure the packets are being sent).

At one point I changed from using a GELF input to a raw input, and then the messages showed up but of course were unreadable as they were still in the compressed GELF format, like so:

x��n�0E��*��Í�r� �$��#�m"��$ȿ��h9q\��J�g43�zA;h9a5J��xh�V��b��@Z��Ƕ�^��A�z:t��[Vv$��:�S��j�&�[b�>��)��M�a��+��vb�Ji̦��\@%E��f��b��W��`�X��`:)��hX+P��Si�V�ɡ�'9�ݲgB)vcǳ.��.ٞ[w��8�ky�L�Kk�4��pC��c'L��폑E�#X3(٥�m۲ �H� ?r�|k�׸%��lkh��C��3��É�'ua~<l�l��z!SmWM�g��"�̦�j�o�DVp�7*�%G��Q��c"G��B��̼�(�� `*�z�GN(�N�k]e xZU�iu�A�|ړ��Z�[��A+��C��&��}�&��'��d�Б��̻_��m�0��]��9<ޭ��Ub�,�U�n�Q��8��F��(J�%.U?��^��wJ�a

I can send the messages in completely raw, but then all the data is stuck in one field.

Why would Graylog accept my GELF-formatted Windows event logs, but not my GELF-formatted IIS logs?

I'm running the latest Graylog VMware OVA, and the only changes I've made were changing the password and timezone, enforcing HTTPS, and setting up LDAP and my inputs.

Here's my NXlog config file:

define ROOT C:\Program Files (x86)\nxlog
 
Moduledir %ROOT%\modules
CacheDir %ROOT%\data
Pidfile %ROOT%\data\nxlog.pid
SpoolDir %ROOT%\data
LogFile %ROOT%\data\nxlog.log
 
<Extension gelf>
    Module         xm_gelf
</Extension>
 
<Extension fileop>
    Module         xm_fileop
</Extension>
 
 <Extension json>
    Module      xm_json
</Extension>
 
# Create the parse rule for IIS logs. You can copy these from the header of the IIS log file.
<Extension w3c>
    Module             xm_csv
    Fields             $date, $time, $s-ip, $cs-method, $cs-uri-stem, $cs-uri-query, $s-port, $cs-username, $c-ip, $csUser-Agent, $cs-Referer, $sc-status, $sc-substatus, $sc-win32-status, $time-taken
    FieldTypes         string, string, string, string, string, string, integer, string, string, string, string, integer, integer, integer, integer
    Delimiter         ' '
    QuoteChar         '"'
    EscapeControl     FALSE
    UndefValue         -
</Extension>
 
<Input iis>
    Module        im_file
    File        "C:\\inetpub\\logs\\LogFiles\\W3SVC12\\u_ex*"
    SavePos      TRUE
 
    Exec        if $raw_event =~ /^#/ drop();                    \
                else                                             \
                {                                                \
                    w3c->parse_csv();                            \
                    $EventTime = parsedate($date + " " + $time); \
                    $SourceName = "IIS";                         \
                    $Message = to_json();                         \
                }
</Input>
 
<Input eventlog>
    Module      im_msvistalog
</Input>
 
<Output graylog>
    Module      om_udp
    Host        graylog
    Port        12201
    OutputType    GELF
 
    #Use the following line for debugging (uncomment the fileop extension above as well)
    Exec file_write("C:\\Program Files (x86)\\nxlog\\data\\nxlog_output.log", $raw_event);
</Output>


#<Route eventlog>
#    Path        eventlog => graylog
#</Route>


<Route iis-to-graylog>
    Path        iis => graylog
</Route>
 
 

Any assistance will be greatly appreciated.

Permalink

#2 Nathan.Reid 9 years ago

#1 Nathan.Reid

I'm having trouble getting my IIS logs into Graylog. My Windows event logs show up just fine, but the IIS logs never show up. I'm using NXlog per the Graylog docs. I've verified that the logs are getting sent (I have them also writing to a file, and I've checked with Wireshark to make sure the packets are being sent). At one point I changed from using a GELF input to a raw input, and then the messages showed up but of course were unreadable as they were still in the compressed GELF format, like so: x��n�0E��*��Í�r� �$��#�m"��$ȿ��h9q\��J�g43�zA;h9a5J��xh�V��b��@Z��Ƕ�^��A�z:t��[Vv$��:�S��j�&�[b�>��)��M�a��+��vb�Ji̦��\@%E��f��b��W��`�X��`:)��hX+P��Si�V�ɡ�'9�ݲgB)vcǳ.��.ٞ[w��8�ky�L�Kk�4��pC��c'L��폑E�#X3(٥�m۲ �H� ?r�|k�׸%��lkh��C��3��É�'ua~<l�l��z!SmWM�g��"�̦�j�o�DVp�7*�%G��Q��c"G��B��̼�(�� `*�z�GN(�N�k]e xZU�iu�A�|ړ��Z�[��A+��C��&��}�&��'��d�Б��̻_��m�0��]��9<ޭ��Ub�,�U�n�Q��8��F��(J�%.U?��^��wJ�a I can send the messages in completely raw, but then all the data is stuck in one field. Why would Graylog accept my GELF-formatted Windows event logs, but not my GELF-formatted IIS logs? I'm running the latest Graylog VMware OVA, and the only changes I've made were changing the password and timezone, enforcing HTTPS, and setting up LDAP and my inputs. Here's my NXlog config file: define ROOT C:\Program Files (x86)\nxlog Moduledir %ROOT%\modules CacheDir %ROOT%\data Pidfile %ROOT%\data\nxlog.pid SpoolDir %ROOT%\data LogFile %ROOT%\data\nxlog.log <Extension gelf> Module xm_gelf </Extension> <Extension fileop> Module xm_fileop </Extension> <Extension json> Module xm_json </Extension> # Create the parse rule for IIS logs. You can copy these from the header of the IIS log file. <Extension w3c> Module xm_csv Fields $date, $time, $s-ip, $cs-method, $cs-uri-stem, $cs-uri-query, $s-port, $cs-username, $c-ip, $csUser-Agent, $cs-Referer, $sc-status, $sc-substatus, $sc-win32-status, $time-taken FieldTypes string, string, string, string, string, string, integer, string, string, string, string, integer, integer, integer, integer Delimiter ' ' QuoteChar '"' EscapeControl FALSE UndefValue - </Extension> <Input iis> Module im_file File "C:\\inetpub\\logs\\LogFiles\\W3SVC12\\u_ex*" SavePos TRUE Exec if $raw_event =~ /^#/ drop(); \ else \ { \ w3c->parse_csv(); \ $EventTime = parsedate($date + " " + $time); \ $SourceName = "IIS"; \ $Message = to_json(); \ } </Input> <Input eventlog> Module im_msvistalog </Input> <Output graylog> Module om_udp Host graylog Port 12201 OutputType GELF #Use the following line for debugging (uncomment the fileop extension above as well) Exec file_write("C:\\Program Files (x86)\\nxlog\\data\\nxlog_output.log", $raw_event); </Output> #<Route eventlog> # Path eventlog => graylog #</Route> <Route iis-to-graylog> Path iis => graylog </Route> Any assistance will be greatly appreciated.

I solved the issue. I noticed that the number of events stored in Graylog was still going up, even though they weren't being displayed, so I adjusted my search to look into the future. IIS records the timestamp in UTC, but when it was parsed by NXlog, I wasn't including any timezone data, so graylog was using the local timezone instead, resulting in events being recorded as having occurred 5 hours in the future.

I updated the IIS parsing to mark the Event Time as UTC by appending "Z" and it now works correctly:

<Input iis>
Module im_file
File "C:\\inetpub\\logs\\LogFiles\\W3SVC12\\u_ex*"
SavePos TRUE
Exec if $raw_event =~ /^#/ drop(); \
else \
{ \
w3c->parse_csv(); \
$EventTime = parsedate($date + " " + $time); \
$EventTime = strftime($EventTime, "%Y-%m-%dT%H:%M:%SZ"); \
$SourceName = "IIS"; \
}
</Input>