Inconsistent log sending from windows to graylog2

Tags:

#1 avner

Hi,

 

We are using the community edition of nxlog 2.8.1248 on windows 2008 R2 server. We are having forwarding event log and IIS logs to graylog2.

This is the conf file is pasted below.

  • If we just have the IIS udp forwarding, it *sometimes* works. We think its not working and then a few hours later we see data coming through, then it might stop again.
  • This IIS issue is also incosistent across machines. Some machines send data, while others never do
  • There are no errors in the nxlog.log file
  • The event log forwarding worked when we used om_udp and GELF format, but when its turned on in combination with IIS (as per conf below) it sends nothing.
  • Graylog2 server is up and running with the respective inputs.
  • I've tested UDP packets can get through to the Graylog2 server

I've checked the IIS csv parsing is correct, and as mentioned I don't see errors in the log.

Would appreciate ideas on what be going on, and how we might troubleshoot this issue?

Thanks,

Av

 

## This is a sample configuration file. See the nxlog reference manual about the
## configuration options. It should be installed locally and is also available
## online at http://nxlog.org/nxlog-docs/en/nxlog-reference-manual.html

## Please set the ROOT to the folder your nxlog was installed into,
## otherwise it will not start.

#define ROOT C:\Program Files\nxlog
define ROOT C:\Program Files (x86)\nxlog

Moduledir %ROOT%\modules
CacheDir %ROOT%\data
Pidfile %ROOT%\data\nxlog.pid
SpoolDir %ROOT%\data
LogFile %ROOT%\data\nxlog.log
LogLevel INFO

<Extension syslog>
    Module xm_syslog
</Extension>

<Extension gelf>
    Module xm_gelf
</Extension>

# Create the parse rule for IIS logs. You can copy these from the header of the IIS log file.
<Extension w3c>
    Module xm_csv
    Fields $date, $time, $s-sitename, $s-computername, $s-ip, $cs-method, $cs-uri-stem, $cs-uri-query, $s-port, $cs-username, $c-ip, $cs-version, $cs(User-Agent), $cs(Cookie), $cs(Referer), $cs-host, $sc-status, $sc-substatus, $sc-win32-status, $sc-bytes, $cs-bytes $time-taken
    FieldTypes string, string, string, string, string, string, string, string, string, string, string, string, string, string, string, string, integer, integer, integer, string, string, string
    Delimiter ' '
    QuoteChar   '"'
    EscapeControl FALSE
    UndefValue  -
</Extension>

# Enable json extension
<Extension json>
    Module      xm_json
</Extension>
 
# Convert the IIS logs to JSON and use the original event time
# Uncomment IIS_IN section if logging for IIS logging

<Input eventlog>
     # Use 'im_mseventlog' for Windows XP, 2000 and 2003
    Module      im_msvistalog
    # Uncomment the following to collect specific event logs only
    Query       <QueryList>\
                    <Query Id="0">\
                        <Select Path="Application">*</Select>\
                        <Select Path="System">*</Select>\
                    </Query>\
                </QueryList> 
</Input>

<Input IIS_Site1>
    Module im_file
    File "C:\\inetpub\\logs\\LogFiles\\W3SVC6\\u_ex*.log"
    SavePos TRUE
    Exec if $raw_event =~ /^#/ drop();    \
        else    \
        {    \
            w3c->parse_csv();    \
            $EventTime = parsedate($date + " " + $time);    \
            $SourceName = "IIS";    \
            $Message = to_json();    \
        }
</Input>


<Output Event_Out>
    Module      om_udp
    Host        10.85.105.215
    Port        12201      
    OutputType  GELF
</Output>

<Output IIS_Out>
    Module      om_udp
    Host        10.85.105.215 
    Port        514
</Output>


<Route IIS>
     Path IIS_Site1 => IIS_Out
</Route>

<Route Events>
    Path eventlog => Event_Out
</Route>

 

#2 adm Nxlog ✓
#1 avner
Hi,   We are using the community edition of nxlog 2.8.1248 on windows 2008 R2 server. We are having forwarding event log and IIS logs to graylog2. This is the conf file is pasted below. If we just have the IIS udp forwarding, it *sometimes* works. We think its not working and then a few hours later we see data coming through, then it might stop again. This IIS issue is also incosistent across machines. Some machines send data, while others never do There are no errors in the nxlog.log file The event log forwarding worked when we used om_udp and GELF format, but when its turned on in combination with IIS (as per conf below) it sends nothing. Graylog2 server is up and running with the respective inputs. I've tested UDP packets can get through to the Graylog2 server I've checked the IIS csv parsing is correct, and as mentioned I don't see errors in the log. Would appreciate ideas on what be going on, and how we might troubleshoot this issue? Thanks, Av   ## This is a sample configuration file. See the nxlog reference manual about the ## configuration options. It should be installed locally and is also available ## online at http://nxlog.org/nxlog-docs/en/nxlog-reference-manual.html ## Please set the ROOT to the folder your nxlog was installed into, ## otherwise it will not start. #define ROOT C:\Program Files\nxlog define ROOT C:\Program Files (x86)\nxlog Moduledir %ROOT%\modules CacheDir %ROOT%\data Pidfile %ROOT%\data\nxlog.pid SpoolDir %ROOT%\data LogFile %ROOT%\data\nxlog.log LogLevel INFO <Extension syslog> Module xm_syslog </Extension> <Extension gelf> Module xm_gelf </Extension> # Create the parse rule for IIS logs. You can copy these from the header of the IIS log file. <Extension w3c> Module xm_csv Fields $date, $time, $s-sitename, $s-computername, $s-ip, $cs-method, $cs-uri-stem, $cs-uri-query, $s-port, $cs-username, $c-ip, $cs-version, $cs(User-Agent), $cs(Cookie), $cs(Referer), $cs-host, $sc-status, $sc-substatus, $sc-win32-status, $sc-bytes, $cs-bytes $time-taken FieldTypes string, string, string, string, string, string, string, string, string, string, string, string, string, string, string, string, integer, integer, integer, string, string, string Delimiter ' ' QuoteChar '"' EscapeControl FALSE UndefValue - </Extension> # Enable json extension <Extension json> Module xm_json </Extension> # Convert the IIS logs to JSON and use the original event time # Uncomment IIS_IN section if logging for IIS logging <Input eventlog> # Use 'im_mseventlog' for Windows XP, 2000 and 2003 Module im_msvistalog # Uncomment the following to collect specific event logs only Query <QueryList>\ <Query Id="0">\ <Select Path="Application">*</Select>\ <Select Path="System">*</Select>\ </Query>\ </QueryList> </Input> <Input IIS_Site1> Module im_file File "C:\\inetpub\\logs\\LogFiles\\W3SVC6\\u_ex*.log" SavePos TRUE Exec if $raw_event =~ /^#/ drop(); \ else \ { \ w3c->parse_csv(); \ $EventTime = parsedate($date + " " + $time); \ $SourceName = "IIS"; \ $Message = to_json(); \ } </Input> <Output Event_Out> Module om_udp Host 10.85.105.215 Port 12201 OutputType GELF </Output> <Output IIS_Out> Module om_udp Host 10.85.105.215 Port 514 </Output> <Route IIS> Path IIS_Site1 => IIS_Out </Route> <Route Events> Path eventlog => Event_Out </Route>  

To debug whether an output has anything to send you could add something like this:

Exec log_info("sending data: " + $raw_event);

Note that your IIS_out module only sends the contents of $raw_event. Assuming the remote end is expecting syslog it is likely that these events are dropped there since IIS logsfiles are w3c which is not syslog.  Use tcpdump/wireshark to verify what is being sent.