nxlog dropping UDP packets | Log collection solutions

#1 manoj.muthukumaran 5 years ago

This is less a question and more of an observation.

I am currently running nxlog 4.1.4016 on Ubuntu 18.04.1 LTS in a vmware environment. Say I boot the VM up and the nxlog service kicks off correctly and works as intended ultimately writing to a network share that I have mounted. If I do a "sudo systemctl restart nxlog.service" or even "./nxlog -r" in order to reload nxlog with a slightly modified config file, UDP packet receive errors and UDP receive buffer errors start climbing from 0 like crazy (netstat -suna). A reboot of the VM from this state does not even fix the issue, the errors immediately appear. In order to fix the issue, I had to purge the nxlog install and do a reinstall in order to prevent any further issues. My config consists of listening for UDP on 2 ports, going through a memory buffer, and writing to the mounted share.

Permalink

#2 manoj.muthukumaran 5 years ago

#1 manoj.muthukumaran

This is less a question and more of an observation. I am currently running nxlog 4.1.4016 on Ubuntu 18.04.1 LTS in a vmware environment. Say I boot the VM up and the nxlog service kicks off correctly and works as intended ultimately writing to a network share that I have mounted. If I do a "sudo systemctl restart nxlog.service" or even "./nxlog -r" in order to reload nxlog with a slightly modified config file, UDP packet receive errors and UDP receive buffer errors start climbing from 0 like crazy (netstat -suna). A reboot of the VM from this state does not even fix the issue, the errors immediately appear. In order to fix the issue, I had to purge the nxlog install and do a reinstall in order to prevent any further issues. My config consists of listening for UDP on 2 ports, going through a memory buffer, and writing to the mounted share.

I wanted to post a follow-up over the weekend. I left nxlog running untouched over the weekend after a fresh install and ensuring my UDP errors remained at 0. This morning, I noticed I was missing data in sporadic ~30 minute windows across the weekend and of course I checked that UDP errors were through the roof. At this moment, doing a tail -f on the file(s) that is/are being written to, I noticed a huge delay (anywhere from 5-30 mins) between the data coming into my VM (tcpdump -n src IP -v) and nxlog writing it out. I cannot confirm whether on not this is an issue with the buffer in my route or some other underlying issue in nxlog. I can confirm that I am getting UDP drops in the kernel (__udp4_lib_rcv) using the tool drop-watch (https://github.com/pavel-odintsov/drop_watch). My next route of testing took me to installing the community edition instead of the enterprise version and it's working like a charm and as expected right now.