Pre-SIEM log filtering and log reduction means dropping, trimming, deduplicating, or rerouting log events before they cross your SIEM’s ingest boundary. The work happens on the collecting agent or on a relay, not inside the SIEM. Most commercial SIEMs bill on the volume they ingest, in gigabytes per day or events per second, so every byte you remove upstream is a byte you never pay to index, store, or search.
This article covers what to keep, what to drop, and how to do both with NXLog Agent, with configurations you can copy.
What is pre-SIEM log filtering and log reduction?
Pre-SIEM reduction is the set of techniques that shrink telemetry data volume between the source and the SIEM. You have four levers:
| Technique | What it removes | Main risk |
|---|---|---|
Filtering |
Entire events that match a rule |
Dropping an event that a detection rule or audit requires |
Deduplication |
Repeated identical events |
Losing the frequency signal if you discard the count |
Trimming |
Fields or text inside events you keep |
Breaking a parser that expects the full record |
Routing |
Nothing. It moves low-value events to cheaper storage |
Slower retrieval during an investigation |
Where the processing runs matters. Filtering on the endpoint agent saves bandwidth, relay capacity, and SIEM ingestion costs at once. Filtering on a relay is easier to manage centrally, but you still pay to move the data off the host. Filtering at the SIEM edge, with the vendor’s own ingestion-time transformations, can cut the ingest bill, but you have still paid to move every byte off the host and across your network, and the rules only work for that one SIEM. For tables on the Microsoft Sentinel data lake tier, the Microsoft Sentinel billing documentation lists a data processing charge per GB on everything that arrives, before any filter runs. Moving data to a cheaper tier after ingestion cuts cost, but it is not pre-SIEM reduction.
Why filter before the SIEM at all?
Filtering upstream pays off in three areas:
- Cost
-
Microsoft Sentinel bills its analytics tier per GB, either pay-as-you-go or through commitment tiers that start at 100 GB per day. Splunk’s ingest model is priced on GB of data ingested, and its pricing FAQ also lists activity-based, workload, and entity pricing, where volume is not the primary meter. Check which contract you are on before you estimate savings.
- Detection quality
-
In our work with SOC teams, noisy sources slow searches and bury the events analysts need.
- Compliance
-
Reduction is a policy decision, not a cost shortcut. The joint Best Practices for Event Logging and Threat Detection guide (ASD’s ACSC with CISA, FBI, NSA, and partners, August 2024) lists an enterprise-approved logging policy first among its four key factors, with centralized log collection and correlation second. NIST’s SP 800-92 (2006) is under revision; the Revision 1 draft (October 2023) defines log management as generating, transmitting, storing, accessing, and disposing of log data. A filter rule is part of your disposal policy. Document it as one.
What to keep and what to drop
Start from detection and retention requirements, not from volume charts. Two anchors help:
- Windows
-
Microsoft’s Appendix L: Events to Monitor lists Security event IDs with a potential criticality of High, Medium, or Low. Treat it as your keep-list baseline. Microsoft’s guidance on monitoring Active Directory says the Directory Service Access category, which produces event 4662, should in general only be enabled on domain controllers and generates a lot of noise. Event 4662 only appears on domain controllers, where DCSync detections depend on it, so reduce its volume by tightening the SACLs rather than by dropping events.
- Detection coverage
-
Map each source to the MITRE ATT&CK data sources your rules depend on. If no rule, hunt, or retention requirement consumes an event type, it is a routing candidate.
Common high-volume, low-value candidates we see in customer environments: Windows Filtering Platform events 5156 and 5158, handle-close events (4658), machine-account network logons, syslog informational and debug severities, firewall allow-rule events, and load-balancer health checks.
The rule of thumb: route what you might need for forensics, drop only what nothing consumes.
Five NXLog Agent configurations that cover the four levers
Each snippet below builds on examples in the current NXLog Platform documentation and matches the NXLog Agent 6.16 reference manual. NXLog Agent applies these rules on the host or on a relay, before anything crosses the SIEM boundary.
1. Drop events by severity or field
The simplest reduction is a severity floor. This configuration listens for syslog, parses it, and discards anything less severe than warning, following the filter events how-to:
<Extension syslog>
Module xm_syslog
</Extension>
<Input udp_listen>
Module im_udp
ListenAddr 0.0.0.0:514
<Exec>
parse_syslog();
if ($SyslogSeverityValue > 4) { (1)
drop();
}
</Exec>
</Input>| 1 | Drops events with a $SyslogSeverityValue above 4.
Syslog severity runs from 0 (emergency) to 7 (debug), and lower values are more severe, so this condition keeps warning and anything more severe. |
What it risks: some applications log security-relevant activity at informational severity. Check your detection rules before you apply a blanket floor.
2. Filter Windows events at the source, then refine with drop()
For Windows Event Log, filter twice.
First, subscribe narrowly with an XPath query so NXLog Agent never collects the noise.
The Windows Event Log integration guide notes that XPath queries have a maximum length, and that queries with more than 22 clauses fail with error code 15001, so a second Exec stage handles anything XPath cannot express.
<Input windows_security>
Module im_msvistalog
<QueryXML>
<QueryList>
<Query Id="0">
<Select Path="Security">*[System[(Level<=4)]]</Select>
<Suppress Path="Security">
*[System[(EventID=4658 or EventID=5156 or EventID=5158)]]
</Suppress>
</Query>
</QueryList>
</QueryXML>
<Exec> (1)
if $EventID == 4624 and $LogonType == 3 and $TargetUserName =~ /\$$/ { (2)
drop();
}
if $EventID == 4688 and $NewProcessName =~ /^C:\\Windows\\System32\\conhost\.exe$/i { (3)
drop();
}
</Exec>
</Input>| 1 | The second stage handles field logic that XPath cannot express. |
| 2 | Drops machine-account network logons, which have a $TargetUserName that ends in $. |
| 3 | Drops process creation events for the system conhost.exe, which typically add noise.
The full path in the match keeps events for a binary with the same name in any other location. |
The field names above are the ones the Event Log for Windows input module extracts from the event data, so you filter on them directly instead of running regular expressions against the whole message.
What it risks: machine-account logons matter for some lateral-movement detections, so confirm your rule set before you drop them. If you reuse this configuration on domain controllers, do not add event 4662 to the suppression: it carries the property access details that DCSync detections read.
3. Deduplicate repeated events
The De-Duplicator processor module compares consecutive events on the fields you choose.
It waits one second for repeats to arrive.
When it finds them, it forwards the first event, drops the rest, and emits a single last message repeated n times event, so you keep the frequency signal.
One caution before you build on it: the De-Duplicator module is being phased out and will be removed in a future release. The NXLog language variables can do the same job, so plan the migration.
<Extension syslog>
Module xm_syslog
</Extension>
<Input udp_listen>
Module im_udp
ListenAddr 0.0.0.0:514
Exec parse_syslog();
</Input>
<Processor norepeat>
Module pm_norepeat
CheckFields Hostname, Message, SourceName (1)
</Processor>
<Output agent_relay>
Module om_batchcompress
Host relay.example.com:2514
</Output>
<Route r>
Path udp_listen => norepeat => agent_relay
</Route>| 1 | Compares only these fields.
Leave out $EventTime, because it differs on every repeat. |
What it risks: brute-force detections that count identical failures per second need the raw events, not the summary. Exclude those sources from the norepeat route.
4. Trim fields and descriptive text from events you keep
Not every event you keep needs every byte. The trim events how-to shows two approaches. The first uses the Rewrite extension module to define an allow-list of fields and discard the rest:
<Extension allowlist>
Module xm_rewrite
Keep AccountName, Channel, EventID, EventReceivedTime, EventTime, \
Hostname, Severity, SeverityValue, SourceName
</Extension>
<Input eventlog>
Module im_msvistalog
<QueryXML>
<QueryList>
<Query Id='0'>
<Select Path='Security'>*[System/Level<=4]</Select>
</Query>
</QueryList>
</QueryXML>
Exec allowlist->process();
</Input>This list is deliberately minimal and keeps no EventData fields, so a logon event loses its account name, logon type, and source address.
Extend it with the fields your detections read, such as TargetUserName, LogonType, and IpAddress.
The second removes the explanatory text Windows appends to certain events. That boilerplate is identical across every instance of an event ID and adds nothing to detection:
<Exec>
if ($Channel == 'Security') and ($EventID == 4769) {
$Message =~ s/\s*This event is generated every time access is.*$//s;
}
$raw_event = $EventTime + ' ' + $Message; (1)
</Exec>| 1 | Stream outputs, such as the TCP, UDP, and File output modules, send $raw_event, so trimming parsed fields alone does not shrink what they send.
This line rebuilds $raw_event from the trimmed fields. |
How much does trimming save? In NXLog’s Windows Event Log guide, truncating the Message field cut a syslog-formatted logon event (4624) from 2,170 bytes to 1,006 bytes, a 54 percent reduction. In JSON, deleting the Message field plus redundant standard fields cut the same event from 3.78 KB to 1.12 KB, or 70 percent. NXLog’s SIEM optimization white paper (September 2024) reports a larger figure: a failed-logon event (4625) fell from 2,917 characters in Snare syslog format to 557 characters, an 80.9 percent reduction, after NXLog dropped fields and reshaped the record into the JSON schema Rapid7 InsightIDR expects. Each figure covers one event type in one format, so treat them as the range you can reach per event, not a fleet-wide average.
What it risks: a SIEM parser that expects the full message text can fail on a shortened one. Detection rules and dashboards that reference a removed field no longer match. The trimmed event is also no longer the original record, which matters when an audit or investigation needs it unmodified. Test the trimmed output against your detection content, and keep a full copy in an archive, as in the next technique.
5. Route by value: high-value events to the SIEM, everything else to cheap storage
Routing is the safest reduction technique because it keeps everything.
The filter events how-to shows the pattern with per-output drop() conditions.
Here it is with a TLS-protected SIEM destination and an Amazon S3 output module archive:
define CERTDIR %CONFDIR%/cert
<Extension json>
Module xm_json
</Extension>
<Input windows_security>
Module im_msvistalog
<QueryXML>
<QueryList>
<Query Id="0">
<Select Path="Security">*</Select>
</Query>
</QueryList>
</QueryXML>
</Input>
<Output siem>
Module om_ssl
Host siem.example.com:6514
CAFile %CERTDIR%/ca.pem
<Exec>
if $EventID NOT IN (4624, 4625, 4648, 4672, 4688, 4697, 4698,
4720, 4728, 4732, 4756, 4768, 4769, 4776) { (1)
drop();
}
to_json();
</Exec>
</Output>
<Output archive>
Module om_amazons3
Region eu-west-1
Bucket security-archive
Server SRV01
Exec to_json();
</Output>
<Route split>
Path windows_security => siem, archive
</Route>| 1 | Forwards only the event IDs that your detection content consumes. |
Because drop() runs inside the siem output block, it only affects that destination.
The archive receives the full stream.
The event ID list is a starting point, not a recommendation for your environment.
Build yours from Microsoft’s Appendix L: Events to Monitor and the rules your SIEM runs.
Managing filter rules across thousands of agents with NXLog Platform
Per-host nxlog.conf edits do not scale past a few dozen machines.
NXLog Platform manages enrolled agents centrally: you create a configuration, assign it to agents matched by properties, and start from configuration templates for common sources.
A data flow visualization shows which agents feed which outputs, which helps when you check that the archive route exists everywhere.
On results: in the published case study, QNB Finansbank’s Expert Architect said NXLog lets them filter out up to 80 percent of events on the endpoints and feed security systems only the data that matters for security operations. That is one customer’s figure for their environment, not a controlled benchmark. Your ratio depends on your sources and audit policy.
How to validate that reduction has not broken detection
A filter rule decides which events your detection rules can see, so a mistake in one weakens your detections as much as a mistake in a detection rule. Test and review filter rules with the same rigor as your detection content.
-
Count before and after. NXLog Agent’s Remote Management extension module reports per-module and per-route event counters. The
evt-dropcounter covers events removed bydrop(), the De-Duplicator module,duplicate_guard(), andFlowControl. The Internal Metrics input module collects these counters as events, so you can ship them to a dashboard. Compare event counts on a sample host with the rule set on and off. -
Replay a known-bad sample. Run a controlled test that triggers each of your high-priority rules with the filter in place, and confirm every rule still fires. If a rule goes quiet, the filter is wrong, not the rule.
-
Track changes to the rules. NXLog Platform keeps configuration versions: each time you save, it records a new version with the date, the user who changed it, and an optional comment. Use the comment to record why each suppression exists, and compare two versions or restore an earlier one if a change breaks a detection. Keep a written exceptions list that names an owner for each suppression.
-
Re-review on source changes. New audit policies, OS upgrades, and application updates change what gets logged. Schedule a review when they land.
Common mistakes
-
Matching strings in
$raw_eventwhen the module already parsed the field you need. Field comparisons are cheaper and less fragile. -
Dropping instead of routing. You cannot recover deleted events during an incident.
-
Filtering only at the SIEM edge, after you have paid for transport and relay capacity.
-
Leaving the rule set without an owner. Filters rot quietly, and the first sign is usually a missed detection.
Where to go next
Start with one noisy source, route it rather than drop it, and measure the difference with event counters before you touch anything else. The NXLog Platform documentation has the full module reference for every snippet above.
Want to test these configurations on your own agents? Start a free NXLog Platform trial. And if you would like a second pair of eyes on your filter design, we’re happy to talk.
FAQ
- Does pre-SIEM filtering reduce SIEM licensing costs?
-
Yes, when the SIEM meters by volume. Microsoft Sentinel’s analytics tier and Splunk’s ingest pricing both bill on GB ingested, so events you remove upstream are never billed. Under Splunk workload or entity pricing the saving is smaller and indirect.
- Is it safe to filter security logs before they reach the SIEM?
-
It is safe when you route low-value events to retained storage rather than deleting them, and when you test the filter rules against your detection content. The joint 2024 logging guidance recommends an approved logging policy. Your filters should be part of that policy.
- What is the difference between log filtering and log reduction?
-
Filtering removes whole events. Log reduction is the broader set of techniques (filtering, deduplication, trimming, and routing) that shrink the volume reaching the SIEM.
- Can I filter Windows Event Log events before they leave the host?
-
Yes. NXLog Agent’s Event Log for Windows input module accepts an XPath subscription query, so NXLog Agent never collects unwanted events, and an
Execblock handles conditions XPath cannot express. - Should filtering run on the agent or on a collector?
-
Agent-side filtering saves the most because it reduces bandwidth, relay load, and ingestion costs together. A relay is easier to manage if you have no central agent management. NXLog Platform closes that gap by pushing configurations to enrolled agents.
