NXLog vs. Splunk Universal Forwarder

NXLog is a robust log collector designed to work under heavy loads and sudden spurts of high log volumes. Curiously, our fully-featured agent outperforms the minimalist Splunk Universal Forwarder containing "only the essential components needed to forward data."

When we benchmarked Splunk Enterprise’s processing and indexing rate with a sudden flood of over 30,000 Windows Sysmon events, it processed and indexed the same dataset far more quickly when NXLog sent the events versus the Splunk Universal Forwarder. In our test environment, Splunk Enterprise consistently indexed events forwarded by NXLog over ten times faster than those sent by the Splunk Universal Forwarder, despite the extra processing for the NXLog agent to emulate the log format of the Universal Forwarder.

In the world of enterprise software, infrastructure is dynamic. It is constantly being enhanced, upgraded, or even completely redesigned. Although Splunk has been a key player in the SIEM market for a while, no one can predict the future. For example, suppose management decided that Splunk needs to be complemented with another SIEM solution to fill a functional gap or even replaced entirely. What would be the ramifications of such a decision for the hundreds or thousands of log sources on which the Splunk Universal Forwarder is the sole log forwarding agent?

NXLog is vendor-agnostic and supports practically any operating system in enterprise computing environments. It can function as the sole log collector and forwarder and seamlessly integrates with any third-party SIEM or log storage, such as:

Today, software solutions must be able to integrate with other systems, such as log management and threat analysis platforms. For example, Elasticsearch may be introduced into an existing environment to store and analyze logs. If you’re using NXLog EE as your log collector, you only need to make minimal changes to the NXLog configurations to start forwarding logs to it. Simply add another output and route, and you can simulcast logs in different formats to Elasticsearch and any other platform.

Most enterprises aim for a consolidated view of their data from all sources, including logs. Such a goal is only possible with the ability to normalize data into a common structure. For example, most log sources include a field to determine where the log record originated. However, the naming varies from one log source to another and can be Computer, ComputerName, Host, Hostname, or any other name determined by the vendor. In addition, Splunk creates default fields during indexing, including host, source, and sourcetype. Yet, a log record in Splunk can have an IP address in the host field, while another ComputerName field contains the hostname. This makes searching your data challenging because you need to include all possible field names in your queries.

NXLog automatically adds three core fields to every event to facilitate normalization: EventReceivedTime, SourceModuleName, and SourceModuleType. Additionally, if a hostname is defined, it also adds a Hostname field. These field names are common across all events processed by NXLog.

Log enrichment goes beyond simply normalizing field names across disparate log sources. Imagine the benefits of creating custom fields specific to your organization. It would allow analysts to isolate events from log sources associated with a particular project, group, external business partner, business unit, store number, geographical region/zone, etc.

Some Splunk pricing models are volume-based. Consequently, to get the best value out of Splunk, the number of events sent to it needs to be kept at a minimum with a high ratio of high-quality event records. The challenge of this strategy is that—​more often than not—​log sources contain a low proportion of quality logs mixed in among an ocean of relatively useless informational records. Although you can blacklist specific log sources and perform basic filtering on Windows logs, with the Splunk Universal Forwarder, you cannot implement complex, highly selective filters on noisy log sources containing high-value events.

In comparison, NXLog provides advanced filtering capabilities. For example, it supports native XPath filters (QueryXML) to filter Windows logs. For example, the following configuration collects logs based on a combination of source, log level, and event ID while ignoring other unimportant events.