Hi.

We are facing this problem that NXLog takes a lot of memory when using it to collect logs from PostgreSQL database. I tried to modify the polling interval in the config but it did not help.

Our config is like this now:

<Input PostgreSQL>

Module  im_dbi
Driver  pgsql
SavePos false
PollInterval 5
Option  host 127.0.0.1
Option  username *****
Option  password **************
Option  dbname messagelog
SQL     SELECT id, discriminator, time, queryid, message, timestamprecord, response, memberclass, membercode, subsystemcode FROM logrecord
Exec $SourceName =  'PostgreSQL';
Exec to_json();

</Input>

<Output out> Module om_tcp Host 192.168.1.1 Port 1468 Exec to_syslog_ietf(); </Output>

<Route 1> Path PostgreSQL => out </Route>

I enabled debug and it produces a huge amount of these lines per second:

2019-04-09 15:00:57 DEBUG worker 0 processing event 0x7f67240a6d80 2019-04-09 15:00:57 DEBUG PROCESS_EVENT: POLL (out) 2019-04-09 15:00:57 DEBUG nx_module_pollset_poll: out 2019-04-09 15:00:57 DEBUG worker 2 got signal for new job 2019-04-09 15:00:57 DEBUG worker 2 got no event to process 2019-04-09 15:00:57 DEBUG worker 2 waiting for new event 2019-04-09 15:00:57 DEBUG [out] no poll events, pollset_poll timed out 2019-04-09 15:00:57 DEBUG nx_event_to_jobqueue: POLL (out) 2019-04-09 15:00:57 DEBUG event added to jobqueue 2019-04-09 15:00:57 DEBUG worker 0 processing event 0x7f67240a4fb0 2019-04-09 15:00:57 DEBUG PROCESS_EVENT: POLL (out) 2019-04-09 15:00:57 DEBUG nx_module_pollset_poll: out 2019-04-09 15:00:57 DEBUG worker 1 got signal for new job 2019-04-09 15:00:57 DEBUG worker 1 got no event to process 2019-04-09 15:00:57 DEBUG worker 1 waiting for new event 2019-04-09 15:00:57 DEBUG [out] no poll events, pollset_poll timed out 2019-04-09 15:00:57 DEBUG nx_event_to_jobqueue: POLL (out) 2019-04-09 15:00:57 DEBUG event added to jobqueue 2019-04-09 15:00:57 DEBUG worker 0 processing event 0x7f67240a6d80 2019-04-09 15:00:57 DEBUG PROCESS_EVENT: POLL (out) 2019-04-09 15:00:57 DEBUG nx_module_pollset_poll: out 2019-04-09 15:00:57 DEBUG worker 2 got signal for new job 2019-04-09 15:00:57 DEBUG worker 2 got no event to process 2019-04-09 15:00:57 DEBUG worker 2 waiting for new event 2019-04-09 15:00:57 DEBUG [out] no poll events, pollset_poll timed out 2019-04-09 15:00:57 DEBUG nx_event_to_jobqueue: POLL (out) 2019-04-09 15:00:57 DEBUG event added to jobqueue 2019-04-09 15:00:57 DEBUG worker 0 processing event 0x7f67240a4fb0

We are not seeing the memory usage problem on other log collecting methods. Any ideas what could cause this?

PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 21239 root 20 0 162120 2396 1600 R 0.3 0.1 0:00.04 top 28670 nxlog 20 0 1589652 1.3g 3676 S 0.3 34.4 14:33.41 nxlog