NXLog vs Splunk Universal Forwarder
If you are reading this, you may either be looking for a new log collection solution or seeking to replace and improve an existing Splunk deployment. If so, we hope that this article can provide further information to help you make a more informed decision on the next steps improving your Splunk deployment.
Feature Comparison
There are multiple choices of log collection agents available on the market, and the Splunk universal forwarder agent is no exception. Splunk offers a universal forwarder, a light forwarder, and a heavy forwarder. NXLog offers both a free Community Edition (CE) and a the paid Enterprise Edition (EE) of its log collection software.
Note
|
The light forwarder has been deprecated as of Splunk Enterprise version 6.0.0. |
For anyone unfamiliar with Splunk forwarders, it is noteworthy that their names might be considered less than descriptive. For instance, the Splunk Types of forwarders document reveals that a "heavy forwarder (sometimes referred to as a 'regular forwarder') has a smaller footprint than an indexer …" and in the same document we learn that a universal forwarder has the smallest footprint (memory, CPU load), even smaller than a light forwarder.
In this comparison, we will focus on the Splunk universal forwarder, which Splunk defines as "a dedicated, streamlined version of Splunk Enterprise that contains only the essential components needed to forward data." From these descriptions it is clear that its design goals were centered on performance rather than possessing a rich set of functional features.
The universal forwarder is the best tool for forwarding data to indexers. Its main limitation is that it forwards only unparsed data.
For more technical information about Splunk’s forwarders and how they compare to each other, see their Splunk Forwarder Comparison document.
The following table compares the Splunk universal forwarder agent (version 8.0.3) and how it stacks up against both editions of the NXLog agent. In this matrix, we will look at which operating systems are supported, which formats each agent can write, along with 28 functional capabilities one might expect to find in a forwarding agent.
Feature | Splunk UF (excl UF Package) | NXLog Enterprise Edition | NXLog Community Edition |
---|---|---|---|
OS Support |
|||
Microsoft Windows |
|||
Linux |
|||
Windows Nano Server |
|||
IBM AIX |
|||
BSD |
|||
Apple macOS |
|||
Solaris |
|||
ARM |
|||
Docker |
|||
Output Format Support |
|||
Snare Output Support |
|||
JSON Output Support |
|||
GELF Output Support |
|||
XML Output Support |
|||
Syslog Formatting (RFC5424) |
|||
Syslog Formatting (RFC3164) |
|||
Log Processing Features |
|||
Windows XP/2000/2003 Event Log Support |
|||
Per-Event Filtering |
|||
Event Parsing |
|||
Event Log Caching |
|||
Use as Windows Event Collector for WEF |
|||
Event Tracing for Windows (ETW) |
|||
UTC Logging |
|||
Field/Value Rewrite or Injection |
|||
Normalizing Windows Logs to Syslog |
|||
Event Correlation |
|||
Truncation of Verbose Event Text |
|||
Filter for Events of Interest |
|||
Debug Mode |
|||
Group Policy Support |
|||
Agent Networking and Output Features |
|||
Failover |
|||
TCP/UDP Message Delivery |
|||
HTTP Event Collector Support |
|||
Forwards to Splunk Enterprise |
|||
Forwards to 3rd Party Systems |
|||
Event Routing |
|||
SSL/TLS Encryption |
|||
Log Message Simulcasting |
|||
Centralized Configuration Management |
|||
Enhanced Event Throttling |
|||
Agent Heartbeat |
|||
Alerting |
|||
Support for Thousands of Agents |
|||
Vendor Support |
|||
Vendor Product Support |
Why use NXLog when Splunk Already has a Forwarding Agent?
Performance
There have not been any forum posts about NXLog failing under heavy load when confronted with a sudden high volume of events. Ironically, it is the feature-rich NXLog agent that outperforms the minimalist, "streamlined" Splunk universal forwarder having "only the essential components needed to forward data."
Benchmarking Splunk Enterprise’s rate of processing and indexing a sudden flood of over 30,000 Windows Sysmon events in a controlled test environment indicated that events forwarded by the Splunk universal forwarder lagged far behind the indexing of the exact same set of Sysmon events when forwarded by NXLog, despite the extra overhead of emulating the format of the universal forwarder. Splunk Enterprise was consistently able to index events forwarded by NXLog over 10 times faster than the same events sent by the Splunk universal forwarder.
Rate of Indexing (EPS) | Splunk universal forwarder | NXLog |
---|---|---|
Maximum |
259 |
3,377 |
Mean |
121 |
1,439 |
Median |
121 |
1,192 |
Minimum |
0 |
1,116 |
Multiple Integrations
In the world of enterprise software, infrastructure is dynamic. It is constantly being enhanced, upgraded, or even completely redesigned. Although Splunk has been a key player in the SIEM market for a while, no one can predict the future. If corporate leadership decided that Splunk needs to be complemented with another SIEM solution to fill some functional gap, or even replaced by another SIEM, what would be the ramifications of such a decision for the hundreds or thousands of log sources where the Splunk universal forwarder has been deployed as the sole agent for forwarding events?
NXLog’s forte is its support for practically any operating system found in enterprise computing environments and its seamless integration with third party solutions, especially SIEMs. The NXLog agent can function as the sole log collector and forwarder for these SIEMs:
-
ArcSight Enterprise Security Manager (ESM)
-
Elasticsearch and Kibana
-
Graylog
-
IBM QRadar
-
McAfee ESM
-
Nagios Log Server
-
NetWitness
-
Rapid7
-
RSA NetWitness
-
Splunk Enterprise
Today’s deployments need to integrate with multiple, diverse solutions, from log management suites to endpoint security applications. For example, Nagios Log Server and Elasticsearch with Kibana could be deployed into an existing enterprise environment where NXLog EE is already integrated as Splunk’s event forwarder with only minimal changes to NXLog configurations. By adding only two additional outputs and routes, events can be simulcast to Splunk, Nagios, and Elasticsearch.
Event Log Enrichment
Most enterprises have the goal of a single view for their data. This can also be applied to the meta data for log sources. This will remain an illusive goal without the ability to normalize field names of data common to most log sources like hostname, event time, log source name, etc. Windows Event Log sources contain a field called Computer. Splunk renames that field to ComputerName. During the indexing process, Splunk creates default fields which include host, source, and sourcetype, yet it is possible for a Splunk event to have an IP address for host while the ComputerName is displaying the correct hostname. How can queries be written to encompass the entire set of log data when searching by host?
NXLog automatically adds three core fields to every event to facilitate this basic need for normalization: EventReceivedTime, SourceModuleName, and SourceModuleType. If a hostname is defined, then a Hostname field is added as well. These field names are common across all events collected by NXLog.
Event log enrichment is much broader than the simple normalization of field names across disparate log sources. Imagine the benefits of creating custom fields specific to your organization which could allow analysts to isolate events emanating from log sources associated with a specific project, group, external business partner, internal business unit, store number, geographical region/zone, etc.
Reduce Operating Costs
Most of the Splunk pricing models are based on the volume of log data the indexer ingests. Consequently, to get the best value out of Splunk, the volume of events forwarded needs to be kept as low as possible, while the ratio of high-quality events to low-quality events needs to be kept as high as possible. The challenge of this strategy is that—more often than not—log sources contain a low proportion of quality events mixed in among an ocean of relatively useless, informational events. Although the Splunk universal forwarder can be configured to blacklist specific log sources, it cannot implement complex, highly selective filters on a noisy log sources that also contain events of high value.
In a Windows environment, NXLog can use native XPath filters (QueryXML
) to
achieve this goal. In the following example, events are collected based on
their security content which are easily identified by their Windows EventID
while the less valuable events are dropped.
1
2
3
4
5
6
7
8
9
10
11
12
13
<Input SecurityAuditEvents>
Module im_msvistalog
<QueryXML>
<QueryList>
<Query Id="0" Path="Security">
<Select Path="Security">*[System[Provider[@Name='Microsoft-Windows
-Security-Auditing'] and (Level=1 or Level=2 or Level=3) and
((EventID=4928 and EventID=4931) or (EventID=4932 and EventID=4937)
or EventID=4662 or (EventID=5136 and EventID = 5141))]]</Select>
</Query>
</QueryList>
</QueryXML>
</Input>
Note
|
For more information on how to filter Windows Event Log events, see the Filtering Events section in the NXLog User Guide. For more details about how to reduce SIEM operating costs, see our Reduce data size and cut SIEM licensing costs white paper. |
Next steps with NXLog
NXLog can completely replace the Splunk universal forwarder by providing the exact same functionality, do it faster, enrich your logs, and provide you with the tools to create complex filters, import/export in various formats, and the ability to route enriched logs to multiple, diverse endpoints. What might initially appear to be an additional expense can be the start of a wise investment strategy for throttling the long-term operating costs of a hungry SIEM.
Our documentation abounds with detailed, step-by-step deployment instructions specific to each platform, an extensive configuration section, over 70 integration topics with examples, a reference manual for specifics on the modules, and real-world configuration samples that have been tested so that you don’t have to do the heavy lifting.
For further information or questions, please contact us.