NXLog vs Splunk Universal Forwarder

There have not been any forum posts about NXLog failing under heavy load when confronted with a sudden high volume of events. Ironically, it is the feature-rich NXLog agent that outperforms the minimalist, "streamlined" Splunk universal forwarder having "only the essential components needed to forward data."

Benchmarking Splunk Enterprise’s rate of processing and indexing a sudden flood of over 30,000 Windows Sysmon events in a controlled test environment indicated that events forwarded by the Splunk universal forwarder lagged far behind the indexing of the exact same set of Sysmon events when forwarded by NXLog, despite the extra overhead of emulating the format of the universal forwarder. Splunk Enterprise was consistently able to index events forwarded by NXLog over 10 times faster than the same events sent by the Splunk universal forwarder.

In the world of enterprise software, infrastructure is dynamic. It is constantly being enhanced, upgraded, or even completely redesigned. Although Splunk has been a key player in the SIEM market for a while, no one can predict the future. If corporate leadership decided that Splunk needs to be complemented with another SIEM solution to fill some functional gap, or even replaced by another SIEM, what would be the ramifications of such a decision for the hundreds or thousands of log sources where the Splunk universal forwarder has been deployed as the sole agent for forwarding events?

NXLog’s forte is its support for practically any operating system found in enterprise computing environments and its seamless integration with third party solutions, especially SIEMs. The NXLog agent can function as the sole log collector and forwarder for these SIEMs:

Today’s deployments need to integrate with multiple, diverse solutions, from log management suites to endpoint security applications. For example, Nagios Log Server and Elasticsearch with Kibana could be deployed into an existing enterprise environment where NXLog EE is already integrated as Splunk’s event forwarder with only minimal changes to NXLog configurations. By adding only two additional outputs and routes, events can be simulcast to Splunk, Nagios, and Elasticsearch.

Most enterprises have the goal of a single view for their data. This can also be applied to the meta data for log sources. This will remain an illusive goal without the ability to normalize field names of data common to most log sources like hostname, event time, log source name, etc. Windows Event Log sources contain a field called Computer. Splunk renames that field to ComputerName. During the indexing process, Splunk creates default fields which include host, source, and sourcetype, yet it is possible for a Splunk event to have an IP address for host while the ComputerName is displaying the correct hostname. How can queries be written to encompass the entire set of log data when searching by host?

NXLog automatically adds three core fields to every event to facilitate this basic need for normalization: EventReceivedTime, SourceModuleName, and SourceModuleType. If a hostname is defined, then a Hostname field is added as well. These field names are common across all events collected by NXLog.

Event log enrichment is much broader than the simple normalization of field names across disparate log sources. Imagine the benefits of creating custom fields specific to your organization which could allow analysts to isolate events emanating from log sources associated with a specific project, group, external business partner, internal business unit, store number, geographical region/zone, etc.

Most of the Splunk pricing models are based on the volume of log data the indexer ingests. Consequently, to get the best value out of Splunk, the volume of events forwarded needs to be kept as low as possible, while the ratio of high-quality events to low-quality events needs to be kept as high as possible. The challenge of this strategy is that—​more often than not—​log sources contain a low proportion of quality events mixed in among an ocean of relatively useless, informational events. Although the Splunk universal forwarder can be configured to blacklist specific log sources, it cannot implement complex, highly selective filters on a noisy log sources that also contain events of high value.

In a Windows environment, NXLog can use native XPath filters (QueryXML) to achieve this goal. In the following example, events are collected based on their security content which are easily identified by their Windows EventID while the less valuable events are dropped.