The benefits of log aggregation

Determining which devices, types of logs, and mode of collection is the first step. For workstations and servers, you normally need to install a logging agent. Window Event Forwarding (WEF) can be used for very basic log aggregation on Windows without the need to install an agent, but it has some drawbacks. And, like syslog, it is operating system dependent.

Beginning with data ingestion, the log is collected in its base form. Most modern logs already contain structured or semi-structured data. However, even with structured data, some fields may contain embedded fields of information that can be parsed into new fields, making the event records even more valuable later on during the analytical stage. For unstructured or semi-structured data that is known to contain such fields, it is important to parse the fields sooner rather than later, ideally on the same host where the logs are generated. A workstation will need only negligible resources to parse its own data in real time, while an intermediate or centralized log collection server might become overloaded if given the task of processing the logs from hundreds or thousands of hosts it is receiving concurrently.

With the many different forms that a log can take, a standardization process is necessary. This can include data normalization. The ability to merge specific types of logs, like DNS lookups from macOS and Windows hosts, by normalizing data can greatly reduce the complexity of queries once a SIEM has ingested these DNS client events that now have a common schema. You may also need to standardize logs to a data format like JSON to make them digestible by your SIEM.

Logs can be modified during the aggregation process. In some circumstances, you may want to redact or remove certain key-value fields before further processing the log data—​for example, censoring sensitive user information like passwords or encryption keys. Similarly, specific fields may contain extraneous information that can be truncated. Maybe some log sources are valuable to your organization, yet they have some unimportant fields. Truncating or dropping entire fields can conserve network bandwidth and disk storage can be conserved. Fields can also be added to enhance the data quality of your logs, like timestamps and hostnames. This is especially important for log sources from macOS workstations, since they rarely contain a hostname field or any other field that can be used to identify the host that generated the logs. This essential information is required for performing aggregation queries that provide you with metrics like how many (or what percentage) of hosts have experienced any given security event within a specific time frame. Using a standard query with the hostname information would enable you to identify which workstations were affected by the event.

You probably will not need to collect duplicated logs showing the same information hundreds of times, informational logs of a benign nature, or debug messages that are of interest only to the developers who maintain the software generating the events. These can all be dropped at the source, thereby decreasing the network throughput of unnecessary logs and, by extension, increasing the throughput of relevant logs. This means more efficient data collection and a huge boost to the quality of logs you are sending to your SIEM.

This is the final stage of the log aggregation process. Once the logs have been processed, they are ready for forwarding over the network—​known as “shipping”—to their final destination. The confidentiality of logs in transit is addressed with secure encryption protocols like TLS. Typically they are shipped to a SIEM that will ingest them. Once in the SIEM you can run correlations on them, generate metrics, and be able to visualize events. SIEMs are not always the final resting place of aggregated logs. They can be shipped to a database like Raijin, which specializes in log data and provides a foundation for developing real-time data analytics.

Many SIEM platforms are priced based on throughput, whether EPS (Events Per Second) or GB/day (Gigabytes per day). Additionally, archival space is priced per GB of disk space used. By filtering out the noise, you can significantly decrease the number of unnecessary logs sent to your analytics and storage platforms, thereby lowering the running costs of these systems.

Logs are only useful if meaning can be extracted from them. Log aggregation platforms can parse free-text event messages into meaningful, structured data that your analytics platform can more efficiently use. This results in simpler queries that run faster, which in turn means faster notifications and reduced response times for your security operations to investigate an event.

More frequently, logs are being created ephemerally—​especially in a distributed, cloud-based environment using systems like Docker and Kubernetes. These logs must be captured within a certain period or the information will be lost. A log aggregation solution can gather these logs at creation time and ship them to a permanent store.

Aggregating logs in a central location increases the integrity of your logs. In the event of a breach, log messages can show the steps that an attacker took. Most attackers attempt to cover their tracks by deleting or manipulating logs. By centralizing data, log aggregation tools counteract these methods.

Many regulations, including PCI-DSS and HIPAA, require logs to be aggregated and stored for a set retention period. Your auditors may also require this of you. Log aggregation allows your organization to conform to these requirements. Raijin database is optimized for log data, integrates seamlessly with NXLog, and supports expression-based partitioning that can automate this task for you.

With the potential for thousands of devices across an organization’s infrastructure, a system for aggregating logs should be easy to deploy at enterprise scale.

Most organizations' IT infrastructure employs multiple operating systems, containerized applications, embedded systems, a wide range of third-party solutions, and more. A log aggregation solution should be modular, and work equally on each system with the same degree of autonomy and accuracy.

Millions of events can be generated in short spaces of time. Any log management platform not capable of sustaining an ingestion rate that meets or exceeds the average rate of logs generated is not a viable solution. This is an important requirement to consider if you want to prevent loss of valuable data.

Integrating log aggregation with third-party solutions should be simple and complementary. The ideal log aggregation solution will need to support a wide variety of SIEM platforms, security tools like firewalls and IDPS systems, and embedded devices on both the input and output sides. They should work together to create a better log management service.

Enterprise-level software is often complicated to configure. Log aggregation platforms should be straightforward and intuitive, based on familiar concepts, languages, and configuration styles that are familiar to IT professionals.

Incorporating log aggregation into an existing log management process requires careful planning and testing prior to deployment. As such, a solution should provide well-written, easy-to-understand documentation that is frequently updated and has a community of technical professionals ready to help.