NXLog vs Splunk Universal Forwarder

NXLog supports both log enrichment and direct forwarding of log events to Splunk indexes, Splunk SIEM and other Splunk products for further analysis.

If you are reading this, you may either be looking for a new log collection solution or seeking to replace and improve an existing Splunk deployment. If so, we hope that this article can provide further information to help you make a more informed decision on the next steps improving your Splunk deployment.

Feature Comparison

There are multiple choices of log collection agents available on the market, and the Splunk universal forwarder agent is no exception. Splunk offers a universal forwarder, a light forwarder, and a heavy forwarder. NXLog offers both a free Community Edition (CE) and a the paid Enterprise Edition (EE) of its log collection software.

Note
The light forwarder has been deprecated as of Splunk Enterprise version 6.0.0.

For anyone unfamiliar with Splunk forwarders, it is noteworthy that their names might be considered less than descriptive. For instance, the Splunk Types of forwarders document reveals that a "heavy forwarder (sometimes referred to as a 'regular forwarder') has a smaller footprint than an indexer …​" and in the same document we learn that a universal forwarder has the smallest footprint (memory, CPU load), even smaller than a light forwarder.

In this comparison, we will focus on the Splunk universal forwarder, which Splunk defines as "a dedicated, streamlined version of Splunk Enterprise that contains only the essential components needed to forward data." From these descriptions it is clear that its design goals were centered on performance rather than possessing a rich set of functional features.

The universal forwarder is the best tool for forwarding data to indexers. Its main limitation is that it forwards only unparsed data.

— Splunk's Splexicon: the Splunk glossary

For more technical information about Splunk’s forwarders and how they compare to each other, see their Splunk Forwarder Comparison document.

The following table compares the Splunk universal forwarder agent (version 8.0.3) and how it stacks up against both editions of the NXLog agent. In this matrix, we will look at which operating systems are supported, which formats each agent can write, along with 28 functional capabilities one might expect to find in a forwarding agent.

Table 1. NXLog Manager/CE and EE vs Splunk Universal Forwarder
Feature Splunk UF (excl UF Package) NXLog Enterprise Edition NXLog Community Edition

OS Support

Microsoft Windows

20

20

20

Linux

20

20

20

Windows Nano Server

20

20

20

IBM AIX

20

20

20

BSD

20

20

20

Apple macOS

20

20

20

Solaris

20

20

20

ARM

20

20

20

Docker

20

20

20

Output Format Support

Snare Output Support

20

20

20

JSON Output Support

20

20

20

GELF Output Support

20

20

20

XML Output Support

20

20

20

Syslog Formatting (RFC5424)

20

20

20

Syslog Formatting (RFC3164)

20

20

20

Log Processing Features

Windows XP/2000/2003 Event Log Support

20

20

20

Per-Event Filtering

20

20

20

Event Parsing

20

20

20

Event Log Caching

20

20

20

Use as Windows Event Collector for WEF

20

20

20

Event Tracing for Windows (ETW)

20

20

20

UTC Logging

20

20

20

Field/Value Rewrite or Injection

20

20

20

Normalizing Windows Logs to Syslog

20

20

20

Event Correlation

20

20

20

Truncation of Verbose Event Text

20

20

20

Filter for Events of Interest

20

20

20

Debug Mode

20

20

20

Group Policy Support

20

20

20

Agent Networking and Output Features

Failover

20

20

20

TCP/UDP Message Delivery

20

20

20

HTTP Event Collector Support

20

20

20

Forwards to Splunk Enterprise

20

20

20

Forwards to 3rd Party Systems

20

20

20

Event Routing

20

20

20

SSL/TLS Encryption

20

20

20

Log Message Simulcasting

20

20

20

Centralized Configuration Management

20

20

20

Enhanced Event Throttling

20

20

20

Agent Heartbeat

20

20

20

Alerting

20

20

20

Support for Thousands of Agents

20

20

20

Vendor Support

Vendor Product Support

20

20

20

Why use NXLog when Splunk Already has a Forwarding Agent?

Performance

There have not been any forum posts about NXLog failing under heavy load when confronted with a sudden high volume of events. Ironically, it is the feature-rich NXLog agent that outperforms the minimalist, "streamlined" Splunk universal forwarder having "only the essential components needed to forward data."

Benchmarking Splunk Enterprise’s rate of processing and indexing a sudden flood of over 30,000 Windows Sysmon events in a controlled test environment indicated that events forwarded by the Splunk universal forwarder lagged far behind the indexing of the exact same set of Sysmon events when forwarded by NXLog, despite the extra overhead of emulating the format of the universal forwarder. Splunk Enterprise was consistently able to index events forwarded by NXLog over 10 times faster than the same events sent by the Splunk universal forwarder.

Table 2. Indexing a Flood of 30,000 Windows Sysmon Events
Rate of Indexing (EPS) Splunk universal forwarder NXLog

Maximum

259

3,377

Mean

121

1,439

Median

121

1,192

Minimum

0

1,116

Multiple Integrations

In the world of enterprise software, infrastructure is dynamic. It is constantly being enhanced, upgraded, or even completely redesigned. Although Splunk has been a key player in the SIEM market for a while, no one can predict the future. If corporate leadership decided that Splunk needs to be complemented with another SIEM solution to fill some functional gap, or even replaced by another SIEM, what would be the ramifications of such a decision for the hundreds or thousands of log sources where the Splunk universal forwarder has been deployed as the sole agent for forwarding events?

NXLog’s forte is its support for practically any operating system found in enterprise computing environments and its seamless integration with third party solutions, especially SIEMs. The NXLog agent can function as the sole log collector and forwarder for these SIEMs:

  • ArcSight Enterprise Security Manager (ESM)

  • Elasticsearch and Kibana

  • Graylog

  • IBM QRadar

  • McAfee ESM

  • Nagios Log Server

  • NetWitness

  • Rapid7

  • RSA NetWitness

  • Splunk Enterprise

Today’s deployments need to integrate with multiple, diverse solutions, from log management suites to endpoint security applications. For example, Nagios Log Server and Elasticsearch with Kibana could be deployed into an existing enterprise environment where NXLog EE is already integrated as Splunk’s event forwarder with only minimal changes to NXLog configurations. By adding only two additional outputs and routes, events can be simulcast to Splunk, Nagios, and Elasticsearch.

Event Log Enrichment

Most enterprises have the goal of a single view for their data. This can also be applied to the meta data for log sources. This will remain an illusive goal without the ability to normalize field names of data common to most log sources like hostname, event time, log source name, etc. Windows Event Log sources contain a field called Computer. Splunk renames that field to ComputerName. During the indexing process, Splunk creates default fields which include host, source, and sourcetype, yet it is possible for a Splunk event to have an IP address for host while the ComputerName is displaying the correct hostname. How can queries be written to encompass the entire set of log data when searching by host?

NXLog automatically adds three core fields to every event to facilitate this basic need for normalization: EventReceivedTime, SourceModuleName, and SourceModuleType. If a hostname is defined, then a Hostname field is added as well. These field names are common across all events collected by NXLog.

Event log enrichment is much broader than the simple normalization of field names across disparate log sources. Imagine the benefits of creating custom fields specific to your organization which could allow analysts to isolate events emanating from log sources associated with a specific project, group, external business partner, internal business unit, store number, geographical region/zone, etc.

Reduce Operating Costs

Most of the Splunk pricing models are based on the volume of log data the indexer ingests. Consequently, to get the best value out of Splunk, the volume of events forwarded needs to be kept as low as possible, while the ratio of high-quality events to low-quality events needs to be kept as high as possible. The challenge of this strategy is that—​more often than not—​log sources contain a low proportion of quality events mixed in among an ocean of relatively useless, informational events. Although the Splunk universal forwarder can be configured to blacklist specific log sources, it cannot implement complex, highly selective filters on a noisy log sources that also contain events of high value.

In a Windows environment, NXLog can use native XPath filters (QueryXML) to achieve this goal. In the following example, events are collected based on their security content which are easily identified by their Windows EventID while the less valuable events are dropped.

nxlog.conf
1
2
3
4
5
6
7
8
9
10
11
12
13
<Input SecurityAuditEvents>
    Module im_msvistalog
    <QueryXML>
        <QueryList>
            <Query Id="0" Path="Security">
                <Select Path="Security">*[System[Provider[@Name='Microsoft-Windows
                -Security-Auditing'] and (Level=1 or Level=2 or Level=3) and
                ((EventID=4928 and EventID=4931) or (EventID=4932 and EventID=4937)
                or EventID=4662 or (EventID=5136 and EventID = 5141))]]</Select>
            </Query>
         </QueryList>
    </QueryXML>
</Input>
Note

For more information on how to filter Windows Event Log events, see the Filtering Events section in the NXLog User Guide.

For more details about how to reduce SIEM operating costs, see our Reduce data size and cut SIEM licensing costs white paper.

Next steps with NXLog

NXLog can completely replace the Splunk universal forwarder by providing the exact same functionality, do it faster, enrich your logs, and provide you with the tools to create complex filters, import/export in various formats, and the ability to route enriched logs to multiple, diverse endpoints. What might initially appear to be an additional expense can be the start of a wise investment strategy for throttling the long-term operating costs of a hungry SIEM.

Our documentation abounds with detailed, step-by-step deployment instructions specific to each platform, an extensive configuration section, over 70 integration topics with examples, a reference manual for specifics on the modules, and real-world configuration samples that have been tested so that you don’t have to do the heavy lifting.

For further information or questions, please contact us.

NXLog Ltd. develops multi-platform log collection tools that support many different log sources, formats, transports, and integrations. The tools help administrators collect, parse, and forward logs so they can more easily respond to security issues, investigate operational problems, and analyze event data. NXLog distributes the free and open source NXLog Community Edition and offers additional features and support with the NXLog Enterprise Edition.

This document is provided for informational purposes only and is subject to change without notice. Trademarks are the properties of their respective owners.

Download a fully functional trial of the Enterprise Edition for free