IIS logs sent via NXlog not showing up in Graylog
Tags:
graylog iis
#1
Nathan.Reid
I'm having trouble getting my IIS logs into Graylog.
My Windows event logs show up just fine, but the IIS logs never show up.
I'm using NXlog per the Graylog docs. I've verified that the logs are getting sent (I have them also writing to a file, and I've checked with Wireshark to make sure the packets are being sent).
At one point I changed from using a GELF input to a raw input, and then the messages showed up but of course were unreadable as they were still in the compressed GELF format, like so:
x����n�0E��*���Í�r � �$���#�m"�����$ȿ��h9q\����J�g43�zA;h9a5J��xh�V� �b� ��@Z��Ƕ�^�� �A�z:t����[Vv$��:�S��j�&�[b�>��)�������M�a�����+��vb�Ji̦�����\@%E���f��b����W��`�X��`:)���hX+P��Si�V�ɡ�'9�ݲgB)vcdz.��.ٞ[w��8�ky�L�Kk�4��pC��c'L�����폑E�#X3( ٥�m۲ �H� ?r�|k�%��lkh����C������3��É�'u a~<l�l��z!SmWM�g��"�̦�j�o�DVp�7*�%G ��Q��c"G� �B���̼�(��� �`*�z�GN(�N�k] e xZU�iu�A�|ړ� �Z�[� ��A+���C����&����}�&���' ����d�Б��̻_��m�0��� ����]��9<ޭ������Ub�,�U�n�Q��8��F��(J�%.U?��^��wJ�a
I can send the messages in completely raw, but then all the data is stuck in one field.
Why would Graylog accept my GELF-formatted Windows event logs, but not my GELF-formatted IIS logs?
I'm running the latest Graylog VMware OVA, and the only changes I've made were changing the password and timezone, enforcing HTTPS, and setting up LDAP and my inputs.
Here's my NXlog config file:
define ROOT C:\Program Files (x86)\nxlogModuledir %ROOT%\modulesCacheDir %ROOT%\dataPidfile %ROOT%\data\nxlog.pidSpoolDir %ROOT%\dataLogFile %ROOT%\data\nxlog.log<Extension gelf> Module xm_gelf</Extension><Extension fileop> Module xm_fileop</Extension> <Extension json> Module xm_json</Extension># Create the parse rule for IIS logs. You can copy these from the header of the IIS log file.<Extension w3c> Module xm_csv Fields $date, $time, $s-ip, $cs-method, $cs-uri-stem, $cs-uri-query, $s-port, $cs-username, $c-ip, $csUser-Agent, $cs-Referer, $sc-status, $sc-substatus, $sc-win32-status, $time-taken FieldTypes string, string, string, string, string, string, integer, string, string, string, string, integer, integer, integer, integer Delimiter ' ' QuoteChar '"' EscapeControl FALSE UndefValue -</Extension><Input iis> Module im_file File "C:\\inetpub\\logs\\LogFiles\\W3SVC12\\u_ex*" SavePos TRUE Exec if $raw_event =~ /^#/ drop(); \ else \ { \ w3c->parse_csv(); \ $EventTime = parsedate($date + " " + $time); \ $SourceName = "IIS"; \ $Message = to_json(); \ }</Input><Input eventlog> Module im_msvistalog</Input><Output graylog> Module om_udp Host graylog Port 12201 OutputType GELF #Use the following line for debugging (uncomment the fileop extension above as well) Exec file_write("C:\\Program Files (x86)\\nxlog\\data\\nxlog_output.log", $raw_event);</Output>#<Route eventlog># Path eventlog => graylog#</Route><Route iis-to-graylog> Path iis => graylog</Route>Any assistance will be greatly appreciated.
#2
Nathan.Reid
#1
Nathan.Reid
I'm having trouble getting my IIS logs into Graylog.
My Windows event logs show up just fine, but the IIS logs never show up.
I'm using NXlog per the Graylog docs. I've verified that the logs are getting sent (I have them also writing to a file, and I've checked with Wireshark to make sure the packets are being sent).
At one point I changed from using a GELF input to a raw input, and then the messages showed up but of course were unreadable as they were still in the compressed GELF format, like so:
x����n�0E��*���Í�r � �$���#�m"�����$ȿ��h9q\����J�g43�zA;h9a5J��xh�V� �b� ��@Z��Ƕ�^�� �A�z:t����[Vv$��:�S��j�&�[b�>��)�������M�a�����+��vb�Ji̦�����\@%E���f��b����W��`�X��`:)���hX+P��Si�V�ɡ�'9�ݲgB)vcdz.��.ٞ[w��8�ky�L�Kk�4��pC��c'L�����폑E�#X3( ٥�m۲ �H� ?r�|k�%��lkh����C������3��É�'u a~<l�l��z!SmWM�g��"�̦�j�o�DVp�7*�%G ��Q��c"G� �B���̼�(��� �`*�z�GN(�N�k] e xZU�iu�A�|ړ� �Z�[� ��A+���C����&����}�&���' ����d�Б��̻_��m�0��� ����]��9<ޭ������Ub�,�U�n�Q��8��F��(J�%.U?��^��wJ�a
I can send the messages in completely raw, but then all the data is stuck in one field.
Why would Graylog accept my GELF-formatted Windows event logs, but not my GELF-formatted IIS logs?
I'm running the latest Graylog VMware OVA, and the only changes I've made were changing the password and timezone, enforcing HTTPS, and setting up LDAP and my inputs.
Here's my NXlog config file:
define ROOT C:\Program Files (x86)\nxlog
Moduledir %ROOT%\modules
CacheDir %ROOT%\data
Pidfile %ROOT%\data\nxlog.pid
SpoolDir %ROOT%\data
LogFile %ROOT%\data\nxlog.log
<Extension gelf>
Module xm_gelf
</Extension>
<Extension fileop>
Module xm_fileop
</Extension>
<Extension json>
Module xm_json
</Extension>
# Create the parse rule for IIS logs. You can copy these from the header of the IIS log file.
<Extension w3c>
Module xm_csv
Fields $date, $time, $s-ip, $cs-method, $cs-uri-stem, $cs-uri-query, $s-port, $cs-username, $c-ip, $csUser-Agent, $cs-Referer, $sc-status, $sc-substatus, $sc-win32-status, $time-taken
FieldTypes string, string, string, string, string, string, integer, string, string, string, string, integer, integer, integer, integer
Delimiter ' '
QuoteChar '"'
EscapeControl FALSE
UndefValue -
</Extension>
<Input iis>
Module im_file
File "C:\\inetpub\\logs\\LogFiles\\W3SVC12\\u_ex*"
SavePos TRUE
Exec if $raw_event =~ /^#/ drop(); \
else \
{ \
w3c->parse_csv(); \
$EventTime = parsedate($date + " " + $time); \
$SourceName = "IIS"; \
$Message = to_json(); \
}
</Input>
<Input eventlog>
Module im_msvistalog
</Input>
<Output graylog>
Module om_udp
Host graylog
Port 12201
OutputType GELF
#Use the following line for debugging (uncomment the fileop extension above as well)
Exec file_write("C:\\Program Files (x86)\\nxlog\\data\\nxlog_output.log", $raw_event);
</Output>
#<Route eventlog>
# Path eventlog => graylog
#</Route>
<Route iis-to-graylog>
Path iis => graylog
</Route>
Any assistance will be greatly appreciated.
I solved the issue. I noticed that the number of events stored in Graylog was still going up, even though they weren't being displayed, so I adjusted my search to look into the future. IIS records the timestamp in UTC, but when it was parsed by NXlog, I wasn't including any timezone data, so graylog was using the local timezone instead, resulting in events being recorded as having occurred 5 hours in the future.
I updated the IIS parsing to mark the Event Time as UTC by appending "Z" and it now works correctly:
<Input iis>
Module im_file
File "C:\\inetpub\\logs\\LogFiles\\W3SVC12\\u_ex*"
SavePos TRUE
Exec if $raw_event =~ /^#/ drop(); \
else \
{ \
w3c->parse_csv(); \
$EventTime = parsedate($date + " " + $time); \
$EventTime = strftime($EventTime, "%Y-%m-%dT%H:%M:%SZ"); \
$SourceName = "IIS"; \
}
</Input>
Module im_file
File "C:\\inetpub\\logs\\LogFiles\\W3SVC12\\u_ex*"
SavePos TRUE
Exec if $raw_event =~ /^#/ drop(); \
else \
{ \
w3c->parse_csv(); \
$EventTime = parsedate($date + " " + $time); \
$EventTime = strftime($EventTime, "%Y-%m-%dT%H:%M:%SZ"); \
$SourceName = "IIS"; \
}
</Input>
