4
responses

Hello all,

Hoping someone will be able to pinpoint the issue we're experiencing.
We have deployed the latest NXLog Enterprise Edition agents (nxlog-5.5.7535) on numerous Windows servers, including multiple domain controllers (DCs).
We're running into an issue where the agents on two DCs have a continuous increase in RAM usage. No other servers experience this problem.
Over multiple days the memory usage by the agent slowly increases to gigabytes. This will keep increasing, but we restart the agent and we're back at ~10MB of RAM usage. The DCs with this issue run Windows Server 2012 R2.
There are newer DCs in place as well, running Windows Server 2022. These new DCs do not have this problem.

Does anyone have an idea as to why this is happening? If you require further information, please let me know.

All the DCs have the same agent template deployed on them, see below:

<?xml version="1.0" encoding="UTF-8" standalone="yes"?>
<agent>
    <agentType>TEMPLATE</agentType>
    <global-config>
        <log-level>INFO</log-level>
        <is-log-to-file>true</is-log-to-file>
        <connection-mode>LISTEN_FROM</connection-mode>
        <connection-address>172.16.1.44</connection-address> <!-- IP address of NXlog Manager -->
        <connection-port>4041</connection-port>
        <verbatim></verbatim>
    </global-config>
    <route>
        <name>route1</name>
        <priority>0</priority>
        <module-route>
            <module-id>RIN</module-id>
        </module-route>
        <module-route>
            <module-id>eventlog</module-id>
        </module-route>
        <module-route>
            <module-id>dnsaudit</module-id>
        </module-route>
    </route>
    <module>
        <name>RIN</name>
        <module>om_tcp</module>
        <config xsi:type="module-connect-address-config-mapping" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">
            <type>OUTPUT</type>
            <verbatim>Exec to_syslog_snare();</verbatim>
            <connect-address>172.16.1.43</connect-address> <!-- IP address of log collector -->
            <port>517</port> <!-- Port used to collect logs by collector. Traffic is syslog but we've opted to not use standard port 514 to facilitate easier splitting of collected logs. -->
            <output-format>LineBased</output-format>
        </config>
    </module>
    <module>
        <name>eventlog</name>
        <module>im_msvistalog</module>
        <config xsi:type="module-windows-vista-log-in-config-mapping" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">
            <type>INPUT</type>
            <verbatim>&lt;QueryXML&gt;
        &lt;QueryList&gt;
          &lt;Query Id="0"&gt;
        &lt;Select Path="Security"&gt;*&lt;/Select&gt;
          &lt;/Query&gt;
          &lt;Query Id="1"&gt;
        &lt;Select Path="Application"&gt;*&lt;/Select&gt;
         &lt;/Query&gt;
          &lt;Query Id="2"&gt;
        &lt;Select Path="System"&gt;*&lt;/Select&gt;
          &lt;/Query&gt;  

    &lt;/QueryList&gt;
&lt;/QueryXML&gt;</verbatim>
            <savePosition>true</savePosition>
            <read-from-last-position>true</read-from-last-position>
            <query></query>
            <channel></channel>
            <remote>false</remote>
            <server></server>
            <user-name></user-name>
            <domain></domain>
            <password></password>
            <auth-method>Default</auth-method>
        </config>
    </module>
    <module>
        <name>dnsaudit</name>
        <module>im_etw</module>
        <config xsi:type="module-event-tracing-for-windows-in-config-mapping" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">
            <type>INPUT</type>
            <verbatim></verbatim>
            <provider>Microsoft-Windows-DNSServer</provider>
            <level>Verbose</level>
        </config>
    </module>
</agent>
AskedSeptember 14, 2022 - 4:59pm

Comments (3)

  • Klevin's picture
    (NXLog)

    Hello Sir,

    First step on this issue is to update to the newest NXLog agent version if possible, this just to test if the issue persist. Please can you share the exact windows version for 2022 and 2012? Can you share as well the nxlog.conf ( please redact sensitive information).

    Standing by for the results.

    Sincerely Klevin

  • nervevector's picture

    Hello,

    Thank you very much for your reply. I see in the change log the following is listed:
    [4378] Fixed memory leak issue in im_msvistalog triggered by events which are larger than 42KB

    We do employ this import module on all of our Windows servers, so this might be a fix to the issue. I will upgrade the agents on the DCs to the latest version of NXLog EE and report back.

Answer (1)

As suggested by Klevin, I upgraded the NXLog agents to the latest EE version (5.6.7727).
This appears to have resolved the memory leak. I have not seen runaway RAM usage on the DCs since deployment.