Windows Event log processing slow | Log collection solutions

#1 dgpv 1 year ago

Hello, We are using nxlog-ce 3.0.2284 on Windows 2016 to collect event logs and forward them to our new SIEM.

We experience an increasing delay in the reception of these events by the SIEM. It starts with a few seconds delay and after a moment, the delay has progressively increased to hours. We collect all the event logs, without any filter and the observed throughput is around 150eps, with peaks at 700eps. The output is om_tcp with TLS. Last, we disabled the buffer processor to see if it helps, but with no improvement.

The server does not seem to be very loaded in terms of RAM, CPU and disk.

Are we reaching nxlog maximum performance ? Looking at the docs, it seems and the default values for BatchSize is 31 and PollInterval is 1 second. Does it mean that the maximum input thoughput is 31eps ? Lowering the PollInterval to 0.1 seconds seems to help, but is it the recommended strategy to improve performance ?

How can we configure nxlog to improve the performance ?

Thank you for your help !

David

Permalink

#2 KlevinDeactivated Nxlog ✓ 1 year ago

#1 dgpv

Hello, We are using nxlog-ce 3.0.2284 on Windows 2016 to collect event logs and forward them to our new SIEM. We experience an increasing delay in the reception of these events by the SIEM. It starts with a few seconds delay and after a moment, the delay has progressively increased to hours. We collect all the event logs, without any filter and the observed throughput is around 150eps, with peaks at 700eps. The output is om_tcp with TLS. Last, we disabled the buffer processor to see if it helps, but with no improvement. The server does not seem to be very loaded in terms of RAM, CPU and disk. Are we reaching nxlog maximum performance ? Looking at the docs, it seems and the default values for BatchSize is 31 and PollInterval is 1 second. Does it mean that the maximum input thoughput is 31eps ? Lowering the PollInterval to 0.1 seconds seems to help, but is it the recommended strategy to improve performance ? How can we configure nxlog to improve the performance ? Thank you for your help ! David

Hello Sir,

Please can you share the config you are using? Please remember to redact sensitive data since this forum is public.

Sincerely Klevin