2
responses

I have NXLog CE latest version monitoring both Windows Event Logs and the DNS Debug log file on Server 2012 R2 and sending to TCP GELF format to a Graylog server.

I'm seeing periodic significant discrepancies between the actual amount of logs generated vs the logs that are being sent and received in my central logging platform (Graylog). from 10,000 messages per minute to 150 messages per minute, when it happens. I have verified this by getting a local copy of the dnsdebug log and checking the amount of lives vs running a query for the same time period in Graylog.

I see this drops specifically against the DNS_Debug file not against the Windows EVTX file, they come through at a normal rate.

Does anyone see anything wrong with the configuration below?

Are there Debug sources for NXlog that can be reviewed to see if internal errors are being generated?

I'm also seeing NXlog send messages with blank short_message and full_message.

Example
{"version":"1.1","_EventReceivedTime":"2018-08-16 16:36:51","_SourceModuleName":"DNS_Debug","_SourceModuleType":"im_file","host":"<Hostname>","short_message":"","full_message":"","timestamp":1534401411,"level":6}

This generates errors on the Graylog as well as an invalid input against a mandatory field in the GELF specification.

In terms of volume we're talking approximately 500,000 messages per 30mins.

Version : nxlog-ce-2.10.2102.msi

NB: We had to use [Exec $ShortMessage = $raw_event;] because by default the short_message field was coming through as 64 character truncated, which appears to be a default configuration.

Any help community would be greatly appreciated.

Next course of action is to set the internal logging to debug and check out what is happening.

--------
## Title: nxlog_winsrv
## Version: 0.1
## OS: Server 2012R2
##
## For any concerns please contact [REDACTED]

#define ROOT C:\Program Files\nxlog
define ROOT C:\Program Files (x86)\nxlog

Moduledir %ROOT%\modules
CacheDir %ROOT%\data
Pidfile %ROOT%\data\nxlog.pid
SpoolDir %ROOT%\data
LogFile %ROOT%\data\nxlog.log
<Extension gelf>
Module xm_gelf
</Extension>
<Extension _json>
Module xm_json
</Extension>
<Input evtx_in>
Module im_msvistalog
Query <QueryList>\
<Query Id="0">\
<Select Path="Setup">*</Select>\
<Select Path="System">*</Select>\
<Select Path="Security">*</Select>\
</Query>\
</QueryList>
</Input>
<Input dnsDebug_in>
Module im_file
File "C:\dns_debug.txt"
SavePos TRUE
Exec $ShortMessage = $raw_event;
</Input>

<Output default_out>
Module om_tcp
Host [HOSTNAME}
Port [Host_Port]
OutputType GELF_TCP
</Output>

<Route default_route>
Path evtx_in, dnsDebug_in => default_out
</Route>
--------

AskedAugust 28, 2018 - 3:55am

Comments (1)

  • Zhengshi's picture
    (NXLog)

    Got a few things going on so will break it up a bit. :)

    Are there Debug sources for NXlog that can be reviewed to see if internal errors are being generated?

    Yes! Please use LogLevel DEBUG to enable debug logging. Note that this will produce a lot of output.
    https://nxlog.co/docs/nxlog-ce/nxlog-reference-manual.html#config_global_loglevel

    NB: We had to use [Exec $ShortMessage = $raw_event;] because by default the short_message field was coming through as 64 character truncated, which appears to be a default configuration.

    64 is the default ShortMessageLength, though this can be adjusted. This field is a requirement of the GELF specifications. The fields being blank are another issue...
    https://nxlog.co/docs/nxlog-ce/nxlog-reference-manual.html#xm_gelf_config

    Blank fields/missing entries.
    I am a big fan of adding om_file and a new route when troubleshooting. I would write output to a temporary file and compare it against the input or graylog.
    You can also use something like exec log_info($raw_event); to print as the events are being processed if you want, though depending on number of events, the om_file is probably better option for you.

Answer (1)

Note that whatever is read from the input file will be sent out over TCP. NXLog does not drop logs, unless you explicitly disable flow-control.

We had a couple similar reports recently all connected with Graylog so I'm tempted to say that it's them discarding the data. I suggest verifying with wireshark or some other means where you can check what is being sent.

Also see the DNS Monitoring section in the user guide about the possibilities in this area.