2
responses

Hi Team,
I have a single log source that is pumping around 40K EPS, which our NX server is unable to handle, my question is how do I increase the log ingestion capacity.
Current setup on an AWS VM:
Ubuntu 20.04 LTS
8 CPU, 32GB Ram, 32gb SSD
As per my understanding we needed to increase the number of routes tied to the input, as well as the average event size and batch sizes, hence edited the nxlog.con file with following
1 input, 8 routes, 2048 byte average event size, 25000 event batch size.

Even with these settings, we are not processing more then 6k EPS.

Can anyone advice, what else we can do, please?
Note: filtering of events at the source is not an option.

AskedJune 30, 2022 - 9:05am

Comments (2)

  • jeffron's picture
    (NXLog)

    Hi Junaid,

    One option is to use a load balancer and distribute the load to different ports on the agent server, then add corresponding input modules to read those events. You implement a HAProxy LB installed on the same server as the agent.

    I hope this helps.

    Regards,

    Jeffron

  • Klevin's picture
    (NXLog)

    If the log source can split the load into 2 ore more input modules in the NXLog agent should be better, since a single module ( input, output, route, proc ) is using a single thread and the more input, output and routes you use than the processing power should be higher.

    So the agent will listen to more source ips / ports .

    You can tell if you reached max capacity by checking the cpu /ram load of the machine.

    If you have Exec rules ( regex for example ) add them in pm_null and add the route accordingly, this further to increase the threads and split the load.

    Sincerely
    Klevin

Answers (0)