1
answer

Hello,

Has anyone observed any memory leaks with the community edition of nxlog v2.1.2148 on Windows (2008R2, 2012, and 2012R2)?

On our busier servers, we periodically will see a burst of errors like the following in the nxlog.log file:
2015-04-12 12:22:09 ERROR EvtNext failed with error 14: Not enough storage is available to complete this operation.
2015-04-12 12:22:10 ERROR EvtUpdateBookmark failed: The handle is invalid.
2015-04-12 12:22:11 ERROR EvtNext failed with error 14: Not enough storage is available to complete this operation.
2015-04-12 12:22:11 ERROR EvtUpdateBookmark failed: The handle is invalid.
(These two errors can take up megabytes of space in the logfile.)

Once I see these errors, nxlog is effectively "mute" until I restart it.

I currently have a system where this has happened, and the nxlog process is taking over 700MB of RAM. I do have nxlog configured with pm_buffer (memory), with a buffer size of 100MB. If it's helpful, I've included my config below (flattened and comments removed -- it was spread across two files with one including the other).

For troubleshooting memory leaks on Linux, I've seen comments about using Valgrind. Is there something comparable for Windows?
Thanks,

- Daniel

###############################################################################

define ROOT C:\Program Files (x86)\nxlog
define EVLOGHOST ip_address_of_my_loghost

Moduledir %ROOT%\modules
CacheDir %ROOT%\data
Pidfile %ROOT%\data\nxlog.pid
SpoolDir %ROOT%\data
LogFile %ROOT%\data\nxlog.log

<Extension json>
Module xm_json
</Extension>

<Extension xml>
Module xm_xml
</Extension>

<Processor membuffer>
Module pm_buffer
MaxSize 102400
Type Mem
WarnLimit 76800
</Processor>

<Input internal>
Module im_internal
Exec $EventReceivedTime = integer($EventReceivedTime) / 1000000; to_json();
</Input>

<Input eventlog>
Module im_msvistalog

Exec $EventReceivedTime = integer($EventReceivedTime) / 1000000;
Exec $EventType = lc(string($EventType));
Exec $FileName = lc(string($FileName));
Exec $Hostname = lc(string($Hostname));
Exec $Severity = lc(string($Severity));
Exec delete($SourceModuleType);
Exec delete($EventTimeWritten);
Exec delete($EventTime);
Exec rename_field("Message", "full_message");
Exec if ($IpAddress =~ /::ffff:(.*)/) $IpAddress = $1;
Exec to_json();
</Input>

<Output EventLogOut>
Module om_tcp
Host %EVLOGHOST%
Port 3515
</Output>

<Route EventLogRoute>
Path internal, eventlog => membuffer => EventLogOut
</Route>

###############################################################################

AskedApril 15, 2015 - 3:07pm

Answer (1)

  1. Upgrade to 2.9.1347
  2. Get rid of pm_buffer in your conf. It's not useful with flow-control and om_tcp
AnsweredApril 15, 2015 - 3:12pm

Comments (9)

  • nxlog0406's picture

    Unfortunately, the memory leak is still there.

    I've upgraded to 2.9.1347 -- thanks for mentioning that (I was looking on SourceForge for newer releases).
    I've also removed the pm_buffer and xm_xml modules.

    I've been running for slightly over 24 hours with the new version (2.9.1347), and the nxlog process is already using 800+MB of RAM. (Note: This is an active host -- it's generating over 10 million events every day.)
    Below is the config I've been running since the nxlog upgrade yesterday. This is the actual config -- no "include" lines or anything else that you don't see here (except that I've replaced the EVLOGHOST ip address).

    --------------------------------------------------------------------------------------------------------------------------------------------
    ## Flat config file, with minimal config

    define ROOT C:\Program Files (x86)\nxlog

    define EVLOGHOST my_eventlog_host_ip_address_goes_here

    Moduledir %ROOT%\modules
    CacheDir %ROOT%\data
    Pidfile %ROOT%\data\nxlog.pid
    SpoolDir %ROOT%\data
    LogFile %ROOT%\data\nxlog.log

    <Extension json>
    Module xm_json
    </Extension>

    <Input internal>
    Module im_internal
    Exec $EventReceivedTime = integer($EventReceivedTime) / 1000000; to_json();
    </Input>

    <Input eventlog>
    Module im_msvistalog

    Exec $EventReceivedTime = integer($EventReceivedTime) / 1000000;
    Exec $EventType = lc(string($EventType));
    Exec $FileName = lc(string($FileName));
    Exec $Hostname = lc(string($Hostname));
    Exec $Severity = lc(string($Severity));
    Exec delete($SourceModuleType);
    Exec delete($EventTimeWritten);
    Exec delete($EventTime);
    Exec rename_field("Message", "full_message");
    Exec if ($IpAddress =~ /::ffff:(.*)/) $IpAddress = $1;
    Exec to_json();
    </Input>

    <Output EventLogOut>
    Module om_tcp
    Host %EVLOGHOST%
    Port 3515
    </Output>

    <Route EventLogRoute>
    Path internal, eventlog => EventLogOut
    </Route>

    April 21, 2015 - 9:01pm
  • adm's picture
    (NXLog)

    Thanks for testing the new version. Unfortunately the string() function is the culprit, this is the source of the memory leak.

    The fix will be available in the next version.

    On the other hand your use of the string() function is redundant since all arguments are strings.

    Exec $EventType = lc(string($EventType));
    Exec $FileName = lc(string($FileName));
    Exec $Hostname = lc(string($Hostname));
    Exec $Severity = lc(string($Severity));

    Correcting the above to this should get rid of the leak:

    Exec $EventType = lc($EventType);
    Exec $FileName = lc($FileName);
    Exec $Hostname = lc($Hostname);
    Exec $Severity = lc($Severity);

    We will be happy to offer more help under a commercial support contract.

    April 21, 2015 - 10:42pm
  • three.sixteen's picture

    Hey there, I wanted to report that I'm experiencing this issue as well in nxlog-2.9.1372-trial

    However, I'm not using the string function..

     

    define ROOT C:\Program Files (x86)\nxlog

    Moduledir %ROOT%\modules
    CacheDir %ROOT%\data
    Pidfile %ROOT%\data\nxlog.pid
    SpoolDir %ROOT%\data
    LogFile %ROOT%\data\nxlog.log

    <Extension json>
        Module      xm_json
    </Extension>

    <Extension _exec>
        Module      xm_exec
    </Extension>
      
    <Input internal>
        Module      im_internal
    </Input>

    <Input eventlog>
        Module      im_msvistalog
    </Input>

    <Output eshttp>
      Module om_http
      URL XXXXXXXXXXXXXXXXXX
      ContentType application/json
      Exec $EventTime = now();
      Exec $EventTime = $EventTime + "-07:00";
      Exec set_http_request_path(strftime(now(), "/nxlog-%Y%m%d/" + $SourceModuleName)); rename_field("timestamp","@timestamp"); to_json();
    </Output>

    <Route es>
        Path        internal, eventlog => eshttp
    </Route>

    June 5, 2015 - 11:35pm
  • adm's picture
    (NXLog)

    Thanks for reporting this. This time the set_http_request_path() call is responsible for the memory leak. The hotfix will be available shortly in EE and it will be in the next release of the Community Edition also.

    June 7, 2015 - 4:40pm
  • dwoodruff's picture

    Hello,

    I'm seeing the same error messages in my log when my busy hosts (domain controllers) stop logging randomly. Below is my config file - any idea what might be causing the issue in my configuration? Thanks!

    define ROOT C:\Program Files (x86)\nxlog

    Moduledir %ROOT%\modules
    CacheDir %ROOT%\data
    Pidfile %ROOT%\data\nxlog.pid
    SpoolDir %ROOT%\data
    LogFile %ROOT%\data\nxlog.log

    ## Extensions
    <Extension _charconv>
    Module xm_charconv
    </Extension>
    <Extension syslog>
    Module xm_syslog
    </Extension>
    <Extension fileop>
    Module xm_fileop
    </Extension>
    <Extension kvp>
    Module xm_kvp
    KVDelimiter =
    KVPDelimiter \t
    </Extension>

    # Input module - read from event log
    <Input eventlog>
    Module im_msvistalog

    # when started, read from last position read from log file
    SavePos TRUE
    ReadFromLast TRUE

    # only collect the three main logs
    Query <QueryList>\
    <Query Id="0">\
    <Select Path="Application">*</Select>\
    <Select Path="System">*</Select>\
    <Select Path="Security">*</Select>\
    </Query>\
    </QueryList>
    </Input>

    ## Inputs

    #SCCM logs
    <Input cm_logs>
    Module im_file
    File "C:\\Windows\\CCM\\Logs\\\*.log"
    Exec $Hostname = hostname_fqdn();
    Exec $SyslogFacility = 'local0';
    Exec $SourceName = file_basename(file_name());
    </Input>

    ## Outputs
    # generic output to just send the raw message, no manipulation
    <Output syslogserver>
    Module om_tcp
    Host <hostname is here>
    Port 10515
    Exec to_syslog_bsd();
    </Output>

    # output for the event log so QRadar can parse it. messages are sent with key/value pairs
    <Output syslogserver_kvp>
    Module om_tcp
    Host <hostname is here>
    Port 10515
    Exec kvp->to_kvp(); $Message = $raw_event; to_syslog_bsd();
    </Output>

    # send both inputs (event logs and files) to output
    # ONLY event logs use key/value pair form
    <Route 1>
    Path eventlog => syslogserver_kvp
    </Route>
    <Route 2>
    Path cm_logs => syslogserver
    </Route>

    June 24, 2015 - 6:31pm
  • adm's picture
    (NXLog)

    "error 14: Not enough storage is available to complete this operation" means that the process ran out of memory. Can you run some process monitoring tool to check the memory usage of the nxlog process? You may want to trim your config in order to isolate the issue and find the offending module (if it's a leak outside of im_msvistalog).

    June 24, 2015 - 11:21pm
  • dwoodruff's picture

    I was able to iterate and trim down the config and found that the kvp module is what was causing the memory leak. I had it disabled on one machine overnight and memory usage on the process only increased by 1MB, where as the previous night with the kvp module in use had an increase of about 750MB.

    June 26, 2015 - 2:41pm
  • adm's picture
    (NXLog)

    Thanks for the additional information. The memory leak has been identified and is indeed caused by to_kvp(). The fix will be available in the next release of the NXLog Community Edition. Feel free to get in touch if you need a hotfix asap.

    June 27, 2015 - 12:25am
  • dwoodruff's picture

    Thanks for the response. Task manager showed over 800 MB memory used on one of the hosts before I restarted the service. I'll see what I can do for trimming the config. Unfortunately the data from these busy hosts is important so I'm limited in what I can cut out!

    June 25, 2015 - 2:49am