Single source with 40K EPS
Hi Team, I have a single log source that is pumping around 40K EPS, which our NX server is unable to handle, my question is how do I increase the log ingestion capacity. Current setup on an AWS VM: Ubuntu 20.04 LTS 8 CPU, 32GB Ram, 32gb SSD As per my understanding we needed to increase the number of routes tied to the input, as well as the average event size and batch sizes, hence edited the nxlog.con file with following 1 input, 8 routes, 2048 byte average event size, 25000 event batch size.
Even with these settings, we are not processing more then 6k EPS.
Can anyone advice, what else we can do, please? Note: filtering of events at the source is not an option.
Hi Junaid,
One option is to use a load balancer and distribute the load to different ports on the agent server, then add corresponding input modules to read those events. You implement a HAProxy LB installed on the same server as the agent.
I hope this helps.
Regards,
Jeffron
