What are logs for?
People outside the security industry probably would answer this question: "To fill up disk space". Partly true, logging can be a point of failure in the system if the log files are not properly rotated and managed. On the security side, logs are a crucial part of our world's IT systems. For example, how would we know who accessed our e-mail account and when? This would be impossible to answer without logging data about these events. From the above example it is clear that logging is essential in order to have a clear understanding of the digital space around us constructed by our devices. Logging can show us if somebody tried to hack into our systems, if the servers are getting old are starting to fail, or it can uncover tracks for further investigation if an incident had already happened so that we can learn from it. This is the reason why regulatory and compliance requirements are present in several industries. With the help of logging systems, everything can be tracked back and questions like 'who sent the money' or 'who accessed top secret industry documents' can be answered. But all of this counts for nothing if the log messages are lost, or there is nobody to analyze the data.
What makes a good centralized log collection system?
It is an important requirement of a system to log events, but storing the logs on the same systems makes the information very vulnerable to almost everything from hardware failure to human factor. This is where log collectors and forwarders come into play. These software tools can process and forward the event data in real-time, to a central remote log collector, which aggregates the messages and stores the logs separately. There are several advantages of sending logs to a central server and using a log forwarder to do this. One of the key performance indicators of log forwarders is throughput. Log messages are generated and usually forwarded one by one, and the throughput is measured in EPS (Events Per Second). It is important to select a tool that can process events at a high EPS rate, not only for the forwarding side, but for the receiver side also. Log collection system performance can also be improved with log forwarders as they can filter out unneeded messages or "noise" from the events, and can even reduce the size of the messages. If this is done at the source of the log messages, the useless messages are not forwarded and stored, thus valuable system resources are not wasted. Centralized log management can also provide a good solution for the challenges of current technologies like containers. Container orchestration systems are designed to work efficiently when it comes to fault-tolerance and scaling, but they were not designed to store data inside the containers. In other words, the containers are state-less. The logs would be gone forever in case of an event in the container farm resulting in the destruction and recreation of the container if the logs were stored inside the container. However, collected logs are barely useful without getting them indexed and making them searchable. Centrally stored logs can help understand the bigger picture in the overall infrastructure, find causes and effects easier, and create custom reports and analytics. This process not only helps when investigating a single node, but becomes useful across many devices, including servers, client computers, networking equipment etc.
From a security aspect the biggest advantage is that the logs are stored in a secure and reliable location which is isolated and inaccessible from where the logs were generated. This is an important feature of a central logging system as attackers will usually try to remove every trace of their activity and wipe all evidence of their attack in order to keep the compromised system up as long as possible to avoid being detected. Without a centralized logging system the attackers can erase or modify locally stored logs and remain uncovered. It becomes much easier for the security team to maintain access policies related to who and at what level can access the logs when event data is centrally stored. Log centralization is also gaining more popularity since the GDPR in the EU has come into force, not to mention other regulations such as PCI-DSS where compliance through a centralized logging system is much easier to achieve as data from every system is available at a single location for the required audits.
Shipping logs to a central log collector can be achieved basically in two ways. The first method being agent-based which uses a separate software tool installed on the source system to read and collect the log messages. This log collector agent should also filter, transform and enrich the logs before forwarding it to the central location. Most people with relevant experience in the field will agree that agent-based log collection is the optimal method, since filtering, normalization and other processing can be done at the source. The second method is called agent-less log collection when no additional software is required to ship the logs. In this case the built-in log forwarding capabilities of the systems are utilized where logs need to be collected from. The disadvantage of agent-less logging is that there are different log transport protocols and formats in use and these are limited in what they can forward. For example the Syslog protocol is the standard on Unix systems whereas Microsoft Windows uses Windows Event Forwarding natively. The good news is that the NXLog Enterprise Edition supports both collection modes and can handle these agent-less log forwarding protocols so that it can be used in environments where agents are not welcome. For remote Syslog based logging the UDP transport has been the most widely utilized for quite a few years and is still in use in environments where reliability and security is not a concern. Recently TCP adoption has increased and is now more common. TCP-based log transfer is more reliable and can also benefit from TLS encrpytion to ensure nobody can read the log messages when forwarded over the network to the collector. NXLog can be very useful if you have to deal with legacy systems or systems that don't support more modern and secure protocols. NXLog can help, for instance, by transforming an older UDP based source into a secure SSL/TLS stream and forward to a collector. The NXLog Enterprise Edition provides a solution to solve several common problems at once in an agent-based setup. Using the im_batchcompress and om_batchcompress modules it becomes possible to forward event data in a compressed form over the network over TCP which can optionally be encrypted with SSL/TLS. Since data is compressed on the sender side, this helps saving valuable bandwidth. In addition this improves reliability as the protocol uses acknowledgement to guarantee message delivery.
Heterogeneous system logging
Security analysts and system administrators are often faced with heterogeneous environments that they must maintain. These environments contain diverse systems including combinations of Windows, Linux, macOS, AIX and Solaris in both client and server configurations. The NXLog Enterprise Edition supports all of the above mentioned operating systems. These various systems, and the organizations that run them, all have different logging requirements and it is imperative that they can be managed together in a unified way. Whether it is Syslog from a Unix or Linux host, or EventLog from a Windows host, there will be a need to pull logging events across the infrastructure. NXLog can collect logs from Windows systems through various methods in either agent-based, or agent-less setups. Local and remote collection of the Windows Eventlog can be accomplished with im_msvistalog. The Windows clients can be also configured via Group Policy to forward Windows Eventlog to a collector using WEF (Windows Event Forwarding). This can be collected by the im_wseventing module, which can be run on Windows or Linux hosts. Both of these modules can provide further benefits by filtering data at the source.
Logging plays a vital role in any environment and will only continue to be more important over time. NXLog can solve your logging problems in an easy, efficient and highly available manner by being integrated into existing systems, or becoming a key building block for a new architecture. Feel free to download either the NXLog Community Edition or fill out a trial request for the NXLog Enterprise Edition and do a test run if you are interested in collecting event data at scale.