NXLog Agent vs Splunk Universal Forwarder

NXLog Agent is a robust log collector designed to handle heavy loads and sudden spikes in log volume. Curiously, our agent outperforms the minimalist Splunk Universal Forwarder containing "only the essential components needed to forward data."

When we benchmarked Splunk Enterprise’s processing and indexing rate with a sudden flood of over 30,000 Windows Sysmon events, it processed and indexed the same dataset far more quickly when NXLog Agent sent the events versus the Splunk Universal Forwarder. In our test environment, Splunk Enterprise consistently indexed events forwarded by NXLog Agent over ten times faster than those sent by the Splunk Universal Forwarder, despite the extra processing required for NXLog Agent to emulate the Splunk Universal Forwarder log format.

In the world of enterprise software, infrastructure is dynamic. It is constantly being enhanced, upgraded, or even completely redesigned. Although Splunk has been a key player in the SIEM market for a while, no one can predict the future. For example, suppose management decided that Splunk needs to be complemented with another SIEM solution to fill a functional gap or even replaced entirely. What would be the ramifications of such a decision for the hundreds or thousands of log sources using solely the Splunk Universal Forwarder to forward logs?

NXLog Agent is vendor-agnostic and supports all popular operating systems used in enterprise environments. It can function as the sole log collector and forwarder and seamlessly integrates with any third-party SIEM or log storage, such as:

Today, software solutions must be able to integrate with diverse systems, such as log management and threat analysis platforms. For example, Elasticsearch may be introduced into an existing environment to store and analyze logs. If you’re using NXLog Agent as your log collector, you only need to make minimal configuration changes to start forwarding logs to it. Simply add another output and route, and you can simulcast logs in different formats to Elasticsearch and any other platform.

Most enterprises aim to achieve a consolidated view of their data across all sources, including logs. Such a goal is only possible by normalizing data into a common structure. For example, most log sources include a field to determine where the log record originated. However, the naming varies from one log source to another and can be Computer, ComputerName, Host, Hostname, or any other name determined by the vendor. In addition, Splunk creates default fields during indexing, including host, source, and sourcetype. Furthermore, a log record in Splunk can have the host field set to an IP address, while the ComputerName field contains the hostname. This makes searching your data challenging because you need to include all possible field names in your queries.

NXLog Agent automatically adds four core fields to every event to facilitate normalization: EventReceivedTime, Hostname, SourceModuleName, and SourceModuleType. These field names are common across all events processed by NXLog Agent.

Log enrichment goes beyond simply normalizing field names across different log sources. Imagine the benefits of creating custom fields specific to your organization. It would allow analysts to isolate events from log sources associated with a particular project, group, business partner, business unit, or geographical region.

Some Splunk pricing models are volume-based. Consequently, to get the best value from Splunk, the number of events sent to it should be kept to a minimum, with a high ratio of high-quality event records. The challenge with this strategy is that, more often than not, log sources contain a low proportion of high-quality logs amid an ocean of useless informational records. Although you can blacklist specific log sources and perform some filtering on Windows logs with the Splunk Universal Forwarder, you cannot implement complex, highly selective filters on noisy log sources containing high-value events.

In comparison, NXLog Agent provides advanced filtering capabilities, such as native XPath filtering (QueryXML) for Windows logs. For example, the following configuration collects logs based on a combination of source, log level, and event ID while ignoring other unimportant events.