This post is the first in a series of answers to questions that our customers asked.
Clarifying EPS
EPS stands for Events Per Second and is considered a standard for measuring the speed of event processing. More precisely, it tells how many events can flow through a particular system in a second. In our case, the number relates to how many events NXLog receives, handles, and outputs in one second.
EPS is broken into three phases: input stream EPS, processing EPS, and output stream EPS. Combining these gives you a main EPS number which is likely to be identical to the bottleneck of the three. In this post, I will only elaborate on the first one. It is complex enough in itself.
At this point, this post will not be able to provide your exact numbers, as it depends on so many factors. Nevertheless, I will get you a step closer and show you an NXLog configuration that can measure your input stream EPS.
Why do you need to track EPS?
Measuring the input stream EPS rate can be significant for several reasons.
- Cost monitoring
-
Some SIEM solutions base their pricing on the number of events ingested. If you aggregate all your logs to a single or a few NXLog server(s) before forwarding them to your SIEM—which you should do—you can keep an eye on your EPS rate over a more extended period. It will help you monitor whether you exceed your EPS quota.
- Testing
-
Analyze your system’s capabilities before upgrading, reorganizing, or scaling. It may also affect your budgeting.
- Statistical analysis
-
Monitoring change can provide you with valuable information. EPS fluctuation compared to the previous period or baseline could indicate suspicious activities. For example, the explosion of a request rate in the case of a web server can be a sign of a DDoS attack. EPS information can also be beneficial from the IT operations side, but it can equally be valuable on the IT Security flank. They both see things from different perspectives and, in both cases, can lead to useful conclusions.
- Diagnostics
-
To see whether logs sent to the receiving node are arriving. Other methods are available for this; nevertheless, it is a possible use-case.
EPS tracking is not an exact science
When talking about EPS, most people will ask questions like: "What is the highest EPS rate NXLog can achieve?" Maximum EPS count is significant in some cases, but there is no silver bullet in measuring this absolute number. The best achievable EPS count is also outside this post’s scope.
I often use an analogy comparing EPS to your network bandwidth and how much data can flow through that. Imagine, just because you have a gigabit network adapter in your computer, it cannot necessarily receive data at that rate. It is highly likely in a corporate network that many applications are running simultaneously on a computer/server also running NXLog. A particular system’s capabilities are unlikely to be only dedicated to NXLog. In addition to considering your network and system load, there is one more critical factor—the size of your logs, which heavily affects your EPS rate.
Monitoring your EPS rate gives you a good insight into what is happening in your network.
How to track your input EPS stream with NXLog?
I set up two Windows 10 machines in my lab to show how it works. Node1 generates log messages with NXLog’s im_testgen module and sends them to Node2 via the om_tcp module. Node2 receives the data with the im_tcp module and includes the EPS tracking configuration. It then writes the data to the local machine utilizing the om_file module. It is a straightforward configuration. To make it easy to replicate, I will include the complete configuration files from both devices.
The input module block of this configuration generates log data using the im_testgen module at a random time, with a maximum of 2000 microseconds between two messages. The output module is configured to send the generated log data to the IP address of Node2.
<Input testgen>
Module im_testgen
Exec sleep(get_rand(2000));
</Input>
<Output tcpout>
Module om_tcp
Host 192.168.88.148:1514
</Output>This configuration first defines the path for the LOGDIR directory to save our files.
For the sake of simplicity, it is the same directory we will use to keep the log file in the output module.
The first module block configures the xm_fileop module, as it is required to do file operations in the rest of the configuration.
The input module block receives the logs from Node1 via the im_tcp module.
The Exec block within that sets the filename and a file path for the input stream EPS data.
It also creates the file if it does not exist.
Then the Schedule directive is where the data creation for the input stream EPS data is triggered.
In this configuration, it does this every ten minutes.
Finally, the output module block writes down the log data using the om_file module.
Once applied, it will write the rate of events every ten minutes into a file called eps-$SourceModuleType-$SourceModuleName.csv.
define LOGDIR C:\test
<Extension fileop>
Module xm_fileop
</Extension>
<Input tcpin>
Module im_tcp
ListenAddr 0.0.0.0:1514
<Exec>
if defined get_stat('eps') { add_stat('eps', 1); }
if not defined get_var('fn') { create_var('fn'); set_var('fn', '%LOGDIR%\eps-' + $SourceModuleType + '-' + $SourceModuleName + '.csv'); }
if not ( file_exists(get_var('fn')) ) or ( file_size (get_var('fn')) == 0 ) { file_write(get_var('fn'), "DateTime,EPS\n"); }
</Exec>
<Schedule>
Every 10 min
<Exec>
if get_stat('eps') { file_write(get_var('fn'), now()+','+get_stat('eps')+"\n"); }
create_stat('eps', 'RATE', 1);
</Exec>
</Schedule>
</Input>
<Output fileout>
Module om_file
File 'C:\test\testlogdata.txt'
</Output>Implementing on Linux
The configuration for Node2 above is for Windows; however, you can easily modify it for use on a Linux machine.
In the second line of the first Exec block, the log directory is defined as '%LOGDIR%\eps-' on Windows, which you need to change to '%LOGDIR%/eps-' on Linux, changing the backslash to forward slash.
Then change the LOGDIR variable at the top to a Linux path.
Finally, change the Output block file path to a Linux file path.
Looking at the gathered data and its visualization
After running NXLog for a few hours on both computers, enough data is captured to populate a chart.
I tried to develop a data set that resembles an "end of the day" scenario in a corporate environment.
Logging during two hours when people leave work, thus generating fewer logs as time advances.
The file in my setup is eps-im_tcp-tcp.csv, and contains the following structure.
DateTime,EPS
2023-01-19 15:51:05,22543
2023-01-19 16:01:05,23917
2023-01-19 16:11:05,19941
2023-01-19 16:21:05,16696
2023-01-19 16:31:05,17336
2023-01-19 16:41:05,15475
2023-01-19 16:51:05,12718
2023-01-19 17:01:05,8053
2023-01-19 17:11:05,7829
2023-01-19 17:21:05,7737
2023-01-19 17:31:05,6640
2023-01-19 17:41:05,5439
2023-01-19 17:51:05,4833You can use the above data to generate the graph below. Please note that this is just a tiny sample, and the sole goal is to be a thought starter so you can build on it.
Conclusion
In this short post, I looked at how you could track your input stream EPS with NXLog and discussed its usefulness. I came up with a simple solution that works without any third-party software, so the only software you need to use is NXLog. Finally, I showed you a configuration that you can easily copy and paste into your configuration and the visualization of the test data set. I hope this post serves you as a thought starter on your journey of EPS tracking.
