How to reduce Splunk ingest cost without losing detection coverage

Splunk measures license usage at the indexing pipeline. Splunk’s Admin Manual states that the measured volume is the raw data entering the indexing pipeline, not the compressed data written to disk. However, data filtered and dropped before indexing does not count against your quota. Four more rules on the same page affect your bill:

Every technique in this guide reduces the data reaching the indexing pipeline. Anything that happens after the meter, such as compression, storage tiering, or retention changes, saves disk and money elsewhere, but not license.

Your pricing model decides how much this matters. Splunk sells activity-based, workload, ingest, and entity pricing. Ingest pricing charges by gigabytes per day, and the steps below hit it directly. Activity-based pricing meters ingest and search activity together, so cutting volume moves one of its two meters. Workload pricing charges for the compute and storage your searches need, so volume reduction still lowers storage and indexer load, but it is not the primary meter. Entity pricing counts monitored hosts, containers, and protected devices, so ingest volume does not move it at all. Splunk does not publish list prices, so we do not quote any. Measure your own volume instead; step 1 shows how.

Telemetry data passes through four stages before Splunk indexes it, and you can cut at any of them:

The earlier you cut, the more you save, because a dropped event costs no bandwidth, no intermediate-tier CPU, and no indexer time. Each stage limits what you can do, though.

Splunk documents the limits of the endpoint stage. The universal forwarder does not parse data except in limited situations, and if you need to change or route data based on its contents you need a heavy forwarder. The universal forwarder can filter Windows Event Log by event code through inputs.conf, but regex routing to the nullQueue runs on a heavy forwarder or indexer, after the data has already crossed the network.

Edge Processor and Ingest Actions, Splunk’s newer options, cover the same use cases at the intermediate tier. In Splunk’s own examples, Edge Processor can match a certain event code, mask the extensive message field at the end of Windows events, and route an unfiltered copy of data to an AWS S3 bucket. The comparison of Splunk’s data management solutions lists the same filter, mask, and Amazon S3 routing capabilities for Ingest Actions. Pipeline templates on Splunk Lantern can filter low-priority Cisco ASA events to reduce license usage. These work, and if you already run them, keep them.

NXLog Agent moves the same work to stage 2. It parses Windows Event Log, Linux and macOS system logs, syslog, and flat files into structured fields on the host, so you can filter on any field, delete fields, deduplicate, and send different subsets to different destinations before anything leaves the machine. The rest of this guide shows how, next to the Splunk-native equivalent where one exists.

Start with Splunk’s own license accounting. The license_usage.log file records bytes indexed (b) by source type (st), index (idx), host (h), and source (s). This search ranks the last 30 days by source type:

Swap st for h to rank hosts, or for idx to rank indexes. If host and source values come back empty, you have hit Splunk’s squashing behavior: above a threshold of distinct source/host combinations, Splunk reports only source type and index. The License Usage Report View in Splunk Web uses the same data.

Then go one level down. For your top source types, sample events and check len(_raw). A source type can be expensive because it produces many events, because each event is large, or both. Each case needs a different fix, which is why the steps below are separate.

If you enrolled your agents in NXLog Platform, the Agents view shows events per second, CPU, and memory per agent, which tells you which hosts to look at first.

The cheapest event is the one Windows never writes. Microsoft’s audit policy documentation rates each subcategory by event volume and recommends a setting per computer type.

The clearest example is Audit Filtering Platform Connection, the subcategory behind events 5156 and 5158. Microsoft rates its volume as high and notes that Success auditing generates one event for every connection made to the system. Its recommendations table has two tiers for each computer type, labeled General and Stronger. The General recommendation for Success auditing is No on domain controllers, member servers, and workstations alike. The Stronger recommendation is IF: enable Success auditing only on high-value hosts where you need to monitor successful inbound or outbound connections to and from untrusted IP addresses. Failure auditing (blocked connections, event 5157) is Yes in both tiers, so keep it on everywhere.

Another documented flood is event 4703. Under Audit Authorization Policy Change, Microsoft warns that applications which adjust token privileges through the AdjustPrivilegesToken API, Configuration Manager being the named example, generate large numbers of 4703 events, and does not recommend Success auditing where such applications run.

Before you filter these downstream, check whether a Group Policy or a third-party agent enabled them in the first place. The command auditpol /get /category:* on a sample host shows the effective policy. If nobody needs the events, turn the subcategory off. If some team does need them, keep the policy and filter at the endpoint instead, which is step 3.

One guardrail from the joint Best Practices for Event Logging and Threat Detection (ASD’s ACSC, CISA, FBI, NSA, and partners, August 2024): keep process creation events with command-line auditing on. Event 4688 can be a large share of Security channel volume on busy hosts, but do not save money there.

Once the audit policy is right, filter what remains before it leaves the host.

On a Splunk universal forwarder, inputs.conf supports blacklist entries for Windows Event Log inputs, either as plain event codes or as key-value regex pairs. Splunk’s Windows event log monitoring documentation covers the formats; the inputs.conf configuration reference documents the allow list and deny list formats and the renderXml setting.

The second entry keeps 4662 only for the object types your detections use. Step 4 builds the same allowlist on the NXLog side; keep the two in step if you run both.

On NXLog Agent, the Event Log for Windows input module accepts the same structured XML query that Event Viewer uses. Windows evaluates the query inside the Event Log API, so suppressed events never reach the agent. Microsoft documents the <Select> and <Suppress> elements and the XPath subset they accept. The NXLog Agent Event Log for Windows input module reference covers the QueryXML directive.

A practical tip: build the filter in Event Viewer as a Custom View, switch to the XML tab, and paste the result into QueryXML. You get a query that a real host has already validated.

On a Splunk heavy forwarder or indexer, the equivalent is a nullQueue transform. Splunk Cloud customers reach the same outcome through Ingest Actions rulesets or Edge Processor pipelines.

The three differ in where the bytes stop. The forwarder blacklist and the NXLog Agent query stop them on the host. The nullQueue stops them after they have crossed your network and consumed parsing capacity, but before the license meter.

Event IDs cover Windows. Most other noise has no ID: debug-level application logs, load balancer health checks in web server logs, informational syslog from network gear.

NXLog Agent’s drop() procedure discards the record being processed, and you can call it after any condition on any parsed field. NXLog’s event filtering guide uses syslog severity as the example. parse_syslog() sets a normalized $SeverityValue, so one line keeps warnings and above:

The same pattern lets you filter by field rather than by event ID, which matters for events that are noisy and valuable at the same time. Event 4662 on a domain controller shows why: a few object types drive specific detections, while the rest is volume. Rather than suppress the whole ID, keep what your detection rules reference:

For 4662 the readable object type appears in the rendered message, so this example matches on $Message. The Event Log for Windows input module also parses the event’s EventData items into fields, so where a value arrives in plain form, such as $SubjectUserName or $ProcessName, compare the field directly. Adjust the allowlist to match the rules in your Splunk detection content. Filter on what the detection uses, not on the event ID alone.

Because Splunk meters bytes, you can keep every event and still pay less by sending less of each one. Two things inflate Windows events in particular: the rendered description text at the end of each message, and the bookkeeping fields the collector adds.

Splunk documents the first problem. Its Edge Processor examples name masking "the extensive message field at the end of Windows events" as a use case, Ingest Actions offers the same masking, and the universal forwarder has suppress_text and related settings for the same purpose.

The description is large. NXLog’s event trimming guide publishes a sample 4769 (Kerberos service ticket) message before and after removing the boilerplate. We counted the bytes: the full message is 1,123 bytes, the trimmed version is 603, so the three explanatory paragraphs are 520 bytes, or 46 percent of the event. Sizes vary by event ID and rendering, so run the same count on your own samples, but the pattern holds: the text that explains what a 4769 is repeats identically in every 4769.

NXLog Agent gives you three levels of trimming:

Here is a Security channel collector that applies all three and sends the result to Splunk’s HTTP Event Collector. The HEC output follows NXLog’s Splunk integration guide; the event object is what Splunk indexes, while time, host, source, and sourcetype travel as HEC metadata keys as described in Splunk’s HEC event format documentation.

Decide field by field with your detection engineers. Keywords, OpcodeValue, and Version rarely appear in a search; SubjectUserName, ProcessName, and IpAddress appear constantly. If a rule or a dashboard references a field, it stays.

Some sources emit the same event hundreds of times in a row: a service that logs a failed connection every second, a firewall that reports the same denied flow, a misconfigured scheduled task. Each copy costs the same as the first.

NXLog Agent’s De-Duplicator processor module compares each event against the one before it on the fields you choose. It forwards the first, drops the duplicates, and emits one summary event reading last message repeated n times, the same behavior classic syslog daemons use. Insert it as a processor in the route:

Two things to know before you rely on it. The module waits one second for duplicates to arrive, so it collapses bursts inside that window rather than an unbounded run, and a source that repeats every few seconds still sends every copy. And CheckFields matters: left unset it compares $Message alone, which merges identical messages from different hosts, so set it explicitly as above.

NXLog is also phasing this module out. It still ships and still works, but its documentation marks it for removal in a future release and points to module variables as the way to build the same thing: hold the previous event’s key in a module variable, compare the current event against it, and count the repeats yourself. For a configuration you expect to run for years, build it that way. To get a quick measurement today, the De-Duplicator processor module is still the quickest option.

For high-volume numeric telemetry such as performance counters, remember the 150-byte cap on metric events in Splunk’s license accounting. If a source only ever feeds a chart, sending it as metrics rather than events changes what it costs.

Some telemetry data you must keep but rarely search: compliance retention, full firewall allow logs, verbose application logs you want for incident reconstruction. The joint ASD/CISA guidance recommends exactly this split. It describes a centralized logging facility such as a secured data lake with select, processed telemetry data forwarded to the SIEM, hot and cold storage tiers, and it states that organizations should consider filtering event logs before sending them to a SIEM to limit cost and capacity issues.

NXLog Agent implements the split with two outputs on one input. Each output receives its own copy of the record, so a drop() in one output does not affect the other. The Splunk output keeps the detection set; the archive output, here the Amazon S3 output module, keeps everything in JSON.

The event ID list is a starting point, not a recommendation. Derive yours from the rules and dashboards you run, and review it whenever detection content changes.

The archive stays reachable. Splunk Cloud Platform’s Federated Search for Amazon S3 searches S3 data without ingesting it. Splunk positions it for low-frequency, ad hoc searches rather than real-time detection, it requires an AWS Glue Data Catalog, and the validated architecture refers to a per-search scan cost, so check the current terms. NXLog Platform’s log storage is another home for the archive tier, with its own search and retention controls.

Compression and batching reduce bandwidth and connection overhead. They do not reduce your Splunk license, because Splunk measures raw bytes before compression. Splunk states this in its licensing documentation, and it still trips people up.

Once you forward from many hosts over WAN links or into Splunk Cloud, the bandwidth savings are real. The HTTP(s) output module supports gzip compression and batching, and NXLog’s Splunk integration guide uses both against HEC:

For agent-to-relay hops inside your own network, the NXLog Transport output module does the same job between NXLog Agent instances; the bandwidth usage guide covers the options.

Everything above is a configuration file. The operational problem is applying it to hundreds or thousands of hosts and confirming that it took effect.

NXLog Platform manages the agent fleet centrally. You build a configuration once, and assign it to agents by group, so a new suppression rule reaches every domain controller in one change, without touching each host. The agent management views show which agents run which configuration, their status, and per-agent throughput, which is how you confirm a filter change produced the drop you expected.

The licensing model also matters here. We license NXLog Platform per data source, not per gigabyte, so trimming and filtering change nothing on the NXLog side of the bill. The only meter your pipeline work moves is Splunk’s.

A filter that removes evidence saves nothing. Three rules limit the risk.

The table compares each technique by where it runs, how much license volume it saves, the risk to detection coverage, and the effort to apply it.

Splunk’s license counts bytes at one point in the pipeline. Every step in this guide gets you to that point with fewer of them, in an order that protects detection coverage: policy first, event IDs second, fields and duplicates third, routing fourth. The Splunk-native tools handle part of the job after the data has traveled; NXLog Agent handles all of it on the host, and NXLog Platform pushes the change to the fleet.

To test this against your own volume, start a free NXLog Platform account, point one agent at a noisy domain controller, and compare the step 1 numbers a day later. Our Splunk integration guide has the HEC setup, and we’re happy to help with the field list if you get in touch.