Data logging, and by extension, logging events have become essential to enterprise-level IT operations in order to provide security and performance monitoring of business operations. However, with the large volume of logs being collected, there is cause for concern that companies are not only collecting too many logs, but also that they are neglecting to collect the very logs that would be most useful for monitoring security-related events. Ironically, many adhere to the notion that the more events collected, the better. This couldn’t be further from the truth since unfiltered logging creates an unnecessary burden on business operations instead of safeguarding them. In reality, the majority of logs collected when events are not filtered are low-value events of little use to security analysts. Such low-value events, known as log noise, can severely hamper security analysts' timely access to the most critical security events when the ratio of high-value events to log noise is so low.
It’s all about quality, not quantity
So, how do you go about weeding out all of the low-value events while keeping only the high-value ones? Are there any additional pros or cons to such an approach?
To realize the goal of collecting mainly high-value events, you need to collaborate with the key stakeholders in your organization who will be using the logs in order to establish a list of event types that are most critical to the security and performance of your businesses operations. Don’t forget that your organization may be legally responsible for collecting additional types of events in order to fulfill compliance mandates and auditing requirements.
Once these event types have been identified, work with your IT Security team on documenting the attributes of these various events so that they can be programmatically identified and collected for future processing and analysis. For more details on this approach, see our white paper, How to develop a log collection strategy.
Regardless of your organizations size, industry, or its current logging infrastructure, NXLog’s flexibility provides numerous ways of reducing log noise and volume which can significantly reduce operation costs.
How NXLog can help
There is practically no other log management solution that is capable of collecting logs from all platforms found in larger organizations (various UNIX platforms, Linux, Windows, Apple macOS, along with various network-ready devices), filtering the logs, aggregating them, and sending them to almost any SIEM or endpoint of your choice.
This means NXLog is a one-stop solution for log collection, processing and distribution. It requires neither third-party "plugins" nor different "connectors" to be installed each time you decide to change from a different log source or from a different endpoint to another.
These are the three most common ways that NXLog can be utilized to efficiently reduce log noise and save costs:
- Trimming logs
-
Trimming events refers to reducing log size by removing unwanted data comprised of fields containing low-value information, as well as redundant or duplicated fields. You can easily configure this feature by specifying a list of "whitelisted" fields to keep. All other event fields will be removed thus reducing the volume of data that will be processed and forwarded.
- Filtering logs
-
Depending on your business requirements, it might be feasible to filter out entire log events when there is no valuable information in the log record, or for example, when the log event is duplicated. You can configure NXLog to drop such unwanted log events by defining a set of attributes that they will match. Any matching events will trigger the drop procedure, leaving only the events that security analysts need.
- Data compression
-
Log data can also be compressed on transmission in order to reduce network bandwidth usage. Consequently, log data is transmitted faster and disk storage requirements are reduced if the endpoints are writing the logs to file.
There are just a few of the advantages NXLog offers for reducing the total cost of logging operations. The cost savings can be significant if your organization is currently sending all events unfiltered to a SIEM, since many of them charge by data volume ingested. If you take a look at NXLog’s rich set of features, you will see that is truly in a class of its own in terms of its modular, distributed architecture and its ability to integrate with such a vast number third-party solutions.
