exec_async powershell.exe causes huge number of open handles which leads to memory exhaustion

Tags:

#1 MathieuH

We use NXlog on Windows server to send audit logs from MariaDB to a syslog server. On certain messages I execute a powershell script with exec_async. We discoverred that the powershell.exe subprocesses is not closed correctly which leads to excessive memory use which can be seen by using RamMap and then the Page Table value. In the processes tab a lot of powershell.exe's can be seen with 4KB memory use. With Process Explorer I can see a large number of Handles in the NXlog.exe process. A restart of the nxlog service temporary resolves the issue by cleaning up the (zombie?) powershell.exe processes. I could reproduce the issue with a very simple powershell script with only 1 line: "exit". When I run "C:\Windows\System32\WindowsPowerShell\v1.0\powershell.exe -ExecutionPolicy ByPass -NonInteractive -File C:\temp\test.ps1" from a cmd.exe it exits normally and the powershell.exe subprocess doesn't stay visible in RamMap.

Could this be a bug? Or should I call powershell.exe differently?

Excerpt of my config:

define ROOT     C:\Program Files (x86)\nxlog
define CERTDIR  %ROOT%\cert
define CONFDIR  %ROOT%\conf
define LOGDIR   %ROOT%\data
define LOGFILE  %LOGDIR%\nxlog.log
LogFile %LOGFILE%

Moduledir %ROOT%\modules
CacheDir  %ROOT%\data
Pidfile   %ROOT%\data\nxlog.pid
SpoolDir  %ROOT%\data

<Extension _exec>
  Module xm_exec
</Extension>

.....

<Processor discard_messages>
	Module	pm_null
	<Exec>
		# Discard messages with 'keepalive'
		if( $auditlog_object == "SELECT 0 FROM DUAL" ) exec_async("C:\\Windows\\System32\\WindowsPowerShell\\v1.0\\powershell.exe", "-ExecutionPolicy", "ByPass", "-NonInteractive", "-File", "C:\\temp\\test.ps1");
		if( $auditlog_object == "SELECT 0 FROM DUAL" ) drop(); 
	</Exec>
</Processor>

....

<Input mariadb_audit_file>
Module im_file
File 'C:\mariadb\data\server_audit.log'
PollInterval 1
SavePos	True
ReadFromLast True
Recursive False
RenameCheck False
    Exec $FileName = file_name(); # Send file name with each message	
</Input>

<Output output_graylog>
	Module om_udp
	Host x.x.x.x
	Port 12201
	OutputType  GELF
	Exec $short_message = $raw_event; # Avoids truncation of the short_message field.
	Exec $Hostname = hostname_fqdn();
</Output>

<Route route-0>
  Path mariadb_audit_file => preprocess => extract_fields => discard_messages => whitelist_queries => output_graylog
</Route>
#2 MathieuH
#1 MathieuH
We use NXlog on Windows server to send audit logs from MariaDB to a syslog server. On certain messages I execute a powershell script with exec_async. We discoverred that the powershell.exe subprocesses is not closed correctly which leads to excessive memory use which can be seen by using RamMap and then the Page Table value. In the processes tab a lot of powershell.exe's can be seen with 4KB memory use. With Process Explorer I can see a large number of Handles in the NXlog.exe process. A restart of the nxlog service temporary resolves the issue by cleaning up the (zombie?) powershell.exe processes. I could reproduce the issue with a very simple powershell script with only 1 line: "exit". When I run "C:\Windows\System32\WindowsPowerShell\v1.0\powershell.exe -ExecutionPolicy ByPass -NonInteractive -File C:\temp\test.ps1" from a cmd.exe it exits normally and the powershell.exe subprocess doesn't stay visible in RamMap. Could this be a bug? Or should I call powershell.exe differently? Excerpt of my config: define ROOT C:\Program Files (x86)\nxlog define CERTDIR %ROOT%\cert define CONFDIR %ROOT%\conf define LOGDIR %ROOT%\data define LOGFILE %LOGDIR%\nxlog.log LogFile %LOGFILE% Moduledir %ROOT%\modules CacheDir %ROOT%\data Pidfile %ROOT%\data\nxlog.pid SpoolDir %ROOT%\data <Extension _exec> Module xm_exec </Extension> ..... <Processor discard_messages> Module pm_null <Exec> # Discard messages with 'keepalive' if( $auditlog_object == "SELECT 0 FROM DUAL" ) exec_async("C:\\Windows\\System32\\WindowsPowerShell\\v1.0\\powershell.exe", "-ExecutionPolicy", "ByPass", "-NonInteractive", "-File", "C:\\temp\\test.ps1"); if( $auditlog_object == "SELECT 0 FROM DUAL" ) drop(); </Exec> </Processor> .... <Input mariadb_audit_file> Module im_file File 'C:\mariadb\data\server_audit.log' PollInterval 1 SavePos True ReadFromLast True Recursive False RenameCheck False Exec $FileName = file_name(); # Send file name with each message </Input> <Output output_graylog> Module om_udp Host x.x.x.x Port 12201 OutputType GELF Exec $short_message = $raw_event; # Avoids truncation of the short_message field. Exec $Hostname = hostname_fqdn(); </Output> <Route route-0> Path mariadb_audit_file => preprocess => extract_fields => discard_messages => whitelist_queries => output_graylog </Route>

The issue is not limited to powershell.exe but also occurs at other programs (e.g. cmd.exe). It seems to be an issue in exec_async. When I run the command with exec (withoud _async) the handles don't count upwards. I have made this workaround with a batchfile so the calling is as quick as possible (the powershell script could take up to 10 seconds).

nxlog.conf:

if( $auditlog_object == "SELECT 0 FROM DUAL" ) exec("C:\\Windows\\System32\\cmd.exe", "/C", "C:\\temp\\call_testPS.bat");

call_testPS.bat:

start C:\Windows\System32\WindowsPowerShell\v1.0\powershell.exe -ExecutionPolicy ByPass -NonInteractive -File C:\temp\test.ps1