Possible memory leak? - Windows Server 2012 R2 - NXLog EE


#1 nervevector

Hello all,

Hoping someone will be able to pinpoint the issue we're experiencing.
We have deployed the latest NXLog Enterprise Edition agents (nxlog-5.5.7535) on numerous Windows servers, including multiple domain controllers (DCs).
We're running into an issue where the agents on two DCs have a continuous increase in RAM usage. No other servers experience this problem.
Over multiple days the memory usage by the agent slowly increases to gigabytes. This will keep increasing, but we restart the agent and we're back at ~10MB of RAM usage. The DCs with this issue run Windows Server 2012 R2.
There are newer DCs in place as well, running Windows Server 2022. These new DCs do not have this problem.

Does anyone have an idea as to why this is happening? If you require further information, please let me know.

All the DCs have the same agent template deployed on them, see below:

<?xml version="1.0" encoding="UTF-8" standalone="yes"?>
<agent>
    <agentType>TEMPLATE</agentType>
    <global-config>
        <log-level>INFO</log-level>
        <is-log-to-file>true</is-log-to-file>
        <connection-mode>LISTEN_FROM</connection-mode>
        <connection-address>172.16.1.44</connection-address> <!-- IP address of NXlog Manager -->
        <connection-port>4041</connection-port>
        <verbatim></verbatim>
    </global-config>
    <route>
        <name>route1</name>
        <priority>0</priority>
        <module-route>
            <module-id>RIN</module-id>
        </module-route>
        <module-route>
            <module-id>eventlog</module-id>
        </module-route>
        <module-route>
            <module-id>dnsaudit</module-id>
        </module-route>
    </route>
    <module>
        <name>RIN</name>
        <module>om_tcp</module>
        <config xsi:type="module-connect-address-config-mapping" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">
            <type>OUTPUT</type>
            <verbatim>Exec to_syslog_snare();</verbatim>
            <connect-address>172.16.1.43</connect-address> <!-- IP address of log collector -->
            <port>517</port> <!-- Port used to collect logs by collector. Traffic is syslog but we've opted to not use standard port 514 to facilitate easier splitting of collected logs. -->
            <output-format>LineBased</output-format>
        </config>
    </module>
    <module>
        <name>eventlog</name>
        <module>im_msvistalog</module>
        <config xsi:type="module-windows-vista-log-in-config-mapping" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">
            <type>INPUT</type>
            <verbatim>&lt;QueryXML&gt;
        &lt;QueryList&gt;
          &lt;Query Id="0"&gt;
		&lt;Select Path="Security"&gt;*&lt;/Select&gt;
          &lt;/Query&gt;
          &lt;Query Id="1"&gt;
		&lt;Select Path="Application"&gt;*&lt;/Select&gt;
         &lt;/Query&gt;
          &lt;Query Id="2"&gt;
		&lt;Select Path="System"&gt;*&lt;/Select&gt;
          &lt;/Query&gt;  
         
	&lt;/QueryList&gt;
&lt;/QueryXML&gt;</verbatim>
            <savePosition>true</savePosition>
            <read-from-last-position>true</read-from-last-position>
            <query></query>
            <channel></channel>
            <remote>false</remote>
            <server></server>
            <user-name></user-name>
            <domain></domain>
            <password></password>
            <auth-method>Default</auth-method>
        </config>
    </module>
    <module>
        <name>dnsaudit</name>
        <module>im_etw</module>
        <config xsi:type="module-event-tracing-for-windows-in-config-mapping" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">
            <type>INPUT</type>
            <verbatim></verbatim>
            <provider>Microsoft-Windows-DNSServer</provider>
            <level>Verbose</level>
        </config>
    </module>
</agent>
#2 nervevector
#1 nervevector
Hello all, Hoping someone will be able to pinpoint the issue we're experiencing. We have deployed the latest NXLog Enterprise Edition agents (nxlog-5.5.7535) on numerous Windows servers, including multiple domain controllers (DCs). We're running into an issue where the agents on two DCs have a continuous increase in RAM usage. No other servers experience this problem. Over multiple days the memory usage by the agent slowly increases to gigabytes. This will keep increasing, but we restart the agent and we're back at ~10MB of RAM usage. The DCs with this issue run Windows Server 2012 R2. There are newer DCs in place as well, running Windows Server 2022. These new DCs do not have this problem. Does anyone have an idea as to why this is happening? If you require further information, please let me know. All the DCs have the same agent template deployed on them, see below: <?xml version="1.0" encoding="UTF-8" standalone="yes"?> <agent> <agentType>TEMPLATE</agentType> <global-config> <log-level>INFO</log-level> <is-log-to-file>true</is-log-to-file> <connection-mode>LISTEN_FROM</connection-mode> <connection-address>172.16.1.44</connection-address> <!-- IP address of NXlog Manager --> <connection-port>4041</connection-port> <verbatim></verbatim> </global-config> <route> <name>route1</name> <priority>0</priority> <module-route> <module-id>RIN</module-id> </module-route> <module-route> <module-id>eventlog</module-id> </module-route> <module-route> <module-id>dnsaudit</module-id> </module-route> </route> <module> <name>RIN</name> <module>om_tcp</module> <config xsi:type="module-connect-address-config-mapping" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"> <type>OUTPUT</type> <verbatim>Exec to_syslog_snare();</verbatim> <connect-address>172.16.1.43</connect-address> <!-- IP address of log collector --> <port>517</port> <!-- Port used to collect logs by collector. Traffic is syslog but we've opted to not use standard port 514 to facilitate easier splitting of collected logs. --> <output-format>LineBased</output-format> </config> </module> <module> <name>eventlog</name> <module>im_msvistalog</module> <config xsi:type="module-windows-vista-log-in-config-mapping" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"> <type>INPUT</type> <verbatim>&lt;QueryXML&gt; &lt;QueryList&gt; &lt;Query Id="0"&gt; &lt;Select Path="Security"&gt;*&lt;/Select&gt; &lt;/Query&gt; &lt;Query Id="1"&gt; &lt;Select Path="Application"&gt;*&lt;/Select&gt; &lt;/Query&gt; &lt;Query Id="2"&gt; &lt;Select Path="System"&gt;*&lt;/Select&gt; &lt;/Query&gt; &lt;/QueryList&gt; &lt;/QueryXML&gt;</verbatim> <savePosition>true</savePosition> <read-from-last-position>true</read-from-last-position> <query></query> <channel></channel> <remote>false</remote> <server></server> <user-name></user-name> <domain></domain> <password></password> <auth-method>Default</auth-method> </config> </module> <module> <name>dnsaudit</name> <module>im_etw</module> <config xsi:type="module-event-tracing-for-windows-in-config-mapping" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"> <type>INPUT</type> <verbatim></verbatim> <provider>Microsoft-Windows-DNSServer</provider> <level>Verbose</level> </config> </module> </agent>

Scratch that, we are seeing this issue on the 2022 DCs as well. Does anyone have an idea? Seems to be a DC specific issue.