You can send AWS VPC Flow Logs to Splunk in three main ways:
-
Publish them to Amazon S3 and pull them with the SQS-based S3 input in the Splunk Add-on for AWS.
-
Stream them through Amazon Data Firehose to the HTTP Event Collector (HEC).
-
Run NXLog Agent between S3 and HEC to parse, filter, and route records before Splunk indexes them.
The Firehose path is the fastest, because it skips S3 file batching and polling. The add-on’s documented extractions stop at version 5, while the NXLog Agent path parses newer fields without custom Splunk extractions.
Key takeaways
-
Firehose to HEC is the push path. You don’t need a Lambda transform with Splunk Add-on for AWS 7.3.0 or later, but the HEC token needs indexer acknowledgment.
-
SQS-based S3 is Splunk’s recommended pull input. Plan for S3 file batching and polling delay.
-
The add-on extracts v2 to v5 layouts only. To use reject reasons, encryption status, or the June 2026 EC2 tag and next-hop fields, parse records before Splunk or write your own extractions.
-
Keep S3 as your system of record. Index what your detections use and replay the rest when an investigation needs it.
What VPC Flow Logs record, and what they miss
Each flow log record summarizes one network flow on one network interface during an aggregation window. AWS writes each record as a line of space-separated fields and uses a dash for any value that doesn’t apply.
# version account-id interface-id srcaddr dstaddr srcport dstport protocol packets bytes start end action log-status
2 111122223333 eni-0a1b2c3d4e5f67890 10.0.12.45 10.0.40.7 51544 443 6 14 6031 1791417660 1791417720 ACCEPT OK
This default-format record shows an accepted flow of 14 TCP packets and 6,031 bytes to port 443 on 10.0.40.7.
The start and end values are Unix timestamps in seconds.
Field versions at a glance
The default format contains only version 2 fields. A custom format lets you pick fields from any version and set their order.
| Version | Fields added | Why it matters for detection |
|---|---|---|
2 |
|
Who talked to whom, how much, and whether it was allowed |
3 |
|
Ties flows to workloads. Packet-level addresses see through NAT gateways and EKS pod networking |
4 |
|
Region, Availability Zone, Local Zone, Wavelength Zone, and Outposts context |
5 |
|
Identifies AWS service traffic, ingress versus egress, and the path egress traffic took |
7 |
|
Attributes flows to ECS clusters, services, and tasks |
8 |
|
Shows rejects caused by VPC Block Public Access (BPA) or VPC encryption controls (EC) |
9 |
|
Identifies the regional NAT gateway behind a flow |
10 |
|
Shows whether a flow was Nitro-encrypted, application-encrypted, both, or neither |
11 |
|
Embeds EC2 tag values and next-hop interface details in each record |
There is no version 6 row because version 6 fields apply to Transit Gateway flow logs only.
AWS announced the version 11 fields in June 2026. Treat fields that don’t come from packet headers, such as ECS and tag metadata, as context: AWS describes them as best-effort values that can be missing or inaccurate. Tag values in particular can be missing or inaccurate for the first hour after you create a subscription.
Delay, gaps, and record quirks
Flow logs aren’t a real-time feed.
AWS aggregates each flow for up to 10 minutes by default, or 1 minute if you set it, and Nitro-based instances use 1 minute or less regardless of the setting.
Best-effort delivery then typically takes about 5 minutes to CloudWatch Logs and about 10 minutes to S3.
Timestamps are approximate too: start can land up to 60 seconds before or after the first packet was seen.
Flow logs also skip some traffic by design, including instance traffic to the Amazon DNS server, instance metadata (169.254.169.254), Amazon Time Sync, DHCP, ARP, and Traffic Mirroring source traffic. Traffic to DNS servers you run yourself is logged. For DNS-layer visibility through the Amazon resolver, add Route 53 Resolver query logs.
Three record details shape detection logic:
- A REJECT can come from timing, not policy
-
AWS notes that packets arriving after a connection closes are also recorded as
REJECT. - Interface addresses can hide the real endpoints
-
srcaddranddstaddrcan show an intermediate interface instead, such as a NAT gateway or the EKS node rather than the pod, whilepkt-srcaddrandpkt-dstaddrkeep the original addresses. - TCP flags are aggregated
-
Flags can be OR-ed across the window, and only FIN, SYN, RST, and SYN-ACK are recorded, so traffic carrying only ACK or PSH shows
0.
Which path should you use?
The three paths differ in where records wait, where parsing happens, and how much control you have before Splunk indexes the data. If your detections read only version 2 to 5 fields, Path 1 or Path 2 covers them.
| Path 1: S3 and SQS-based S3 input | Path 2: Firehose to HEC | Path 3: NXLog Agent pipeline | |
|---|---|---|---|
Data flow |
S3, then SNS and SQS, then the Splunk Add-on for AWS |
Amazon Data Firehose, then the HEC raw endpoint |
S3, then SQS, then NXLog Agent, then the HEC event endpoint |
What adds delay |
S3 file batching and delivery, plus the input’s collection interval (300 seconds by default) |
Aggregation window plus Firehose buffering |
S3 file batching and delivery, plus SQS polling |
Fields Splunk extracts |
v2 to v5 layouts |
v2 to v5 layouts, or OCSF |
Any fields you list in the parser |
Filtering before indexing |
Flow log filter and field selection |
Flow log filter and field selection, or a Lambda transform you maintain |
Drop, enrich, and normalize rules in NXLog Agent |
Choose it when |
You want Splunk-native collection from S3 |
You want the shortest path from flow to search |
You need v7 to v11 fields, control over what Splunk indexes, or several destinations |
Skip the CloudWatch Logs pull input. Splunk advises against pull-based API inputs for CloudWatch Logs because of rate limits, and points you to Firehose instead.
Path 1: Amazon S3 and the SQS-based S3 input
Splunk names the SQS-based S3 input the best practice for VPC flow logs delivered to S3. S3 announces each new log file through SNS and SQS, and the add-on downloads and indexes it. Three details decide whether it works:
- Field order
-
The add-on extracts fields only when records follow the default format or one of two documented custom orders. The documented orders are a 29-field order covering versions 2 to 5, or the same 29 fields in alphabetical order, which Splunk calls Select All. That name predates the version 7 to 11 fields, so don’t select every field in today’s console.
- Queue setup
-
Splunk’s prerequisites are an SNS topic for S3 object-created notifications, an SQS queue subscribed to it, a dead-letter queue, and the listed IAM permissions. Set the visibility timeout to at least 5 minutes, or unfinished files return to the queue and the add-on indexes them twice.
- Bucket notifications
-
Saving a notification configuration replaces the bucket’s whole configuration, so merge in existing rules first, and allow S3 to publish to the SNS topic before you save.
The add-on sets the sourcetype to aws:cloudwatchlogs:vpcflow and doesn’t accept a custom one.
The input only collects files that S3 announces through SQS after you set it up, so use the Generic S3 input for history.
To scale, point inputs on more collection nodes at the same queue: the input is stateless, so the others keep going if one fails.
Path 2: Amazon Data Firehose to Splunk HEC
VPC Flow Logs can publish straight to Firehose, which pushes records to HEC without waiting for S3 files. AWS updated its Splunk walkthrough in December 2023 to drop the Lambda transform, because Splunk Add-on for AWS 7.3.0 and later handles flow log records from Firehose. A guide that still adds a Lambda function predates that change, and older material may use the name the service had until February 2024: Amazon Kinesis Data Firehose.
Firehose is strict about the HEC endpoint:
- Trusted certificate
-
Firehose requires a CA-signed certificate that matches the HEC endpoint’s DNS name, so self-signed certificates don’t work. On Splunk Cloud, the Firehose endpoint is
https://http-inputs-firehose-<stack>.splunkcloud.com:443. - Indexer acknowledgment
-
Firehose waits for HEC to confirm each delivery. On Splunk Cloud, Firehose is the only supported client for acknowledgment-enabled tokens, so give it its own token. On Splunk Enterprise, the global
[http]stanza in the HEC app’sinputs.confdefaults todisabled = 1, which turns off every token, so enable it too. Behind a load balancer, AWS recommends sticky sessions so acknowledgment checks reach the indexer that received the data.
When you create the stream, Splunk recommends the raw endpoint unless a transform turns records into HEC event JSON.
Size the retry window, 0 to 7,200 seconds, to cover a realistic HEC outage.
Backup mode is one-way: you can switch from FailedEventsOnly to AllEvents but not back, so start with FailedEventsOnly or validate on a separate test stream.
Keep the token out of the stream definition with SecretsManagerConfiguration.
The stream must use Direct PUT as its source, and AWS lists the IAM permissions and cross-account roles the flow log needs.
OCSF variant. From add-on version 8.3.0, Splunk also accepts flow logs that CloudWatch Logs converted to OCSF, the Open Cybersecurity Schema Framework. It suits detections that already target OCSF, but the extra CloudWatch Logs hop adds ingestion and storage costs.
Path 3: NXLog Agent between S3 and Splunk HEC
Choose this path when you want to decide what Splunk indexes, use fields beyond version 5, or send the same records to more than one destination. NXLog Agent picks up each new flow log file as S3 announces it, parses every record against the field list you configured, adds CIM field names, and sends compressed batches to HEC.
If you already run Path 1, keep it and give Path 3 its own flow log under an nxlog/ S3 prefix, with its own notification rule and SQS queue.
Scope Path 1’s notification rule to its own prefix first: S3 rejects overlapping prefix filters for the same event type, and a rule with no prefix overlaps every other.
Separate prefixes also mean each consumer reads only the field layout it parses.
AWS allows up to 250 flow log subscriptions per resource in an account, but each one publishes and bills its own copy of the data.
- Pick the fields you need
-
NXLog Agent parses by field position from a list you control, so your format can include fields the add-on doesn’t extract, such as
reject-reason(version 8) and EC2 instance tags (version 11). Tag fields need extra permissions, starting withec2:DescribeTags. AWS percent-encodes special characters in tag values, so use Splunk’surldecode()eval function when a report needs the original text. - Read new files from SQS
-
NXLog Agent’s Python input module runs a short Python reader that you write with boto3, the AWS SDK for Python. The NXLog AWS integration guide shows a setup for CloudWatch Logs, and your reader can follow the same pattern. Your reader long-polls SQS, streams each gzip file from S3, skips the header line, and posts one event per flow record. It deletes a message only after posting every record from that file, so with
PersistLogqueueandSyncLogqueueenabled on the output, delivery is at-least-once, and a redelivered message can repeat records in Splunk. To scale out, run more agents against the same queue. NXLog Agent’s Amazon S3 input module polls a bucket on a schedule instead, so this path uses SQS notifications to pick up each file as it lands and to share the work across agents. - Parse, normalize, and forward
-
The configuration parses each record with the CSV extension, using a space delimiter and
UndefValueset to-, so the dashes AWS writes for missing values become undefined fields. It then drops what you don’t want indexed, adds CIM field names, wraps each record in the HEC event format withtimeset from the record’sstartfield, and sends gzip-compressed batches with the HTTP(s) output module. An excerpt from the input’sExecblock:# No traffic during the window: nothing to analyze if ($log_status == "NODATA") { drop(); } # CIM addresses: prefer packet-level values (NAT gateway, EKS pods) $src_ip = defined($pkt_srcaddr) ? $pkt_srcaddr : $srcaddr; $dest_ip = defined($pkt_dstaddr) ? $pkt_dstaddr : $dstaddr; # Example policy: drop accepted flows from load balancer subnets, # such as health checks. Replace the pattern with your own. # Rejects are kept. if (($action == "ACCEPT") and ($src_ip =~ /^10\.0\.(8|9)\./)) { drop(); } if ($action == "ACCEPT") { $action = "allowed"; } else if ($action == "REJECT") { $action = "blocked"; }
On the delivery side:
- Retries
-
The HTTP(s) output module treats any response other than 200, 201, or 202 as an error and resends. It splits a failed batch into one request per record, and an
OnErrorblock reroutes any record HEC rejects as malformed (HTTP 400) to a local file. - Durability
-
With
PersistLogqueue, NXLog Agent removes records from its on-disk queue only after delivery succeeds. Our guide to log shipper offline buffering covers sizing and testing. - Acknowledgment
-
This path’s HEC token doesn’t use indexer acknowledgment, so an HTTP 200 means HEC accepted the request, not that Splunk has indexed it.
Tell Splunk how to read the events.
Send them with your own sourcetype, such as aws:vpcflow:nxlog.
Then, deploy search-time field aliases, an event type, and the network and communicate tags to your search heads so the events join the CIM Network Traffic data model.
Search-time settings don’t add to what you index.
Manage it with NXLog Pipeline Manager. When NXLog Agent runs in several AWS accounts or Regions, NXLog Pipeline Manager lets you save your NXLog Agent configuration once and apply it to every agent in one step. You can also limit who can change it with role-based access control (RBAC). Make sure your Python reader script is on each agent host, at the path the configuration references.
Map flow log fields to the Splunk CIM
With Paths 1 and 2, the add-on’s aws:cloudwatchlogs:vpcflow sourcetype is marked CIM compliant.
With Path 3, you own the mapping.
Apply it in your NXLog Agent configuration and Splunk settings as follows (the excerpt above covers the address and action rows):
| VPC Flow Logs field | CIM field | Mapping |
|---|---|---|
|
|
Packet-level source, so NAT gateway and EKS traffic shows the original endpoint |
|
|
Packet-level destination |
|
|
Numeric, unchanged |
|
|
IANA number to name ( |
|
|
|
|
|
|
|
|
Unchanged |
|
|
Search-time alias |
|
|
Search-time alias |
|
|
Search-time alias (BPA or EC) |
The direction field describes traffic relative to the interface that logged it, not your VPC boundary.
Detection searches to start with
This search queries the Network Traffic data model with tstats, so it works for any path once your data maps to the CIM.
It assumes an accelerated data model (set summariesonly=false if yours isn’t), and its threshold is a starting point for your own baseline.
Scanning from one source. Blocked flows to many distinct ports within 15 minutes:
| tstats summariesonly=true count AS blocked_flows dc(All_Traffic.dest_port) AS distinct_ports
from datamodel=Network_Traffic.All_Traffic
where All_Traffic.action="blocked"
by All_Traffic.src_ip _time span=15m
| rename "All_Traffic.*" AS *
| where distinct_ports >= 25
| sort - distinct_ports
The same pattern covers allowed SSH and RDP flows from public addresses, and large outbound transfers.
An outbound search relies on direction, which you set in your Path 3 configuration (see the CIM mapping table), so with add-on data, check that direction is populated first.
Block Public Access hits (Path 3 data).
This one uses the version 8 reject-reason field, which the add-on doesn’t extract:
index=<your_index> sourcetype="aws:vpcflow:nxlog" reject_reason="BPA"
| stats count min(_time) AS first_seen max(_time) AS last_seen BY src_ip dest_ip dest_port vpc_id
| convert ctime(first_seen) ctime(last_seen)
Keep volume and cost under control
AWS bills VPC Flow Logs as CloudWatch vended logs at tiered per-GB rates, whether they go to CloudWatch Logs, S3, or Firehose, and Firehose adds its own per-GB ingestion charge for vended-log sources. Rates vary by Region and change over time, so this post quotes none.
- At the source
-
Capture
ACCEPT,REJECT, orALLtraffic for a VPC, a subnet, or a single network interface, and use a custom format with only the fields you need. The 1-minute interval produces more records than the 10-minute default. - Parquet is for Athena, not Splunk
-
AWS reports 10 to 100 times faster queries and 20 percent less storage than gzip text, but Parquet adds conversion charges, and these Splunk paths read plain text.
- In Splunk
-
Send trend data to a metrics index, or filter before indexing with SPL2 pipelines in Edge Processor and Ingest Processor.
- In the pipeline (Path 3)
-
One starting policy: keep every
REJECTand every flow that crosses your VPC boundary, then drop records that no detection or report uses. The original files stay in S3 for replay when an investigation needs them.
Ports of Auckland, a NXLog customer, credits NXLog Agent’s filtering for letting the team ingest only valuable events and save on EPS volume.
Validate the VPC Flow Logs to Splunk pipeline
Check each hop in order.
At the source, aws ec2 describe-flow-logs should report DeliverLogsStatus as SUCCESS.
In Splunk, compare paths by the p95 of _indextime - _time per sourcetype.
Then check each path’s own signals:
- Path 1
-
Splunk suggests a CloudWatch alarm on the queue’s
ApproximateAgeOfOldestMessageorApproximateNumberOfMessagesVisibleas the trigger for adding collection nodes. - Path 2
-
Turn on Firehose error logging, and send the backup bucket’s S3 notifications to an SQS-based S3 input with
parse_firehose_error_dataenabled to re-ingest failed events. - Path 3
-
Check the NXLog Agent log and the local file of rejected records.
TcpSendStallDetect TRUEon the HTTP(s) output instance logs a warning when sends to HEC are blocked.
Choose the path that fits your detections
Start from the fields and latency your detections need. If fields from versions 2 to 5 cover them, Path 1 or Path 2 uses Splunk’s own add-on. If you need newer fields, control over what Splunk indexes, or more than one destination, put NXLog Agent in the middle and manage it with NXLog Pipeline Manager.
Ready to try it? Start with the Free plan and filter your own flow logs before Splunk indexes them.
FAQ
- Does the Splunk Add-on for AWS support VPC Flow Logs?
-
Yes. The add-on indexes VPC Flow Logs as
aws:cloudwatchlogs:vpcflowthrough SQS-based S3, Firehose to HEC, Kinesis, or CloudWatch Logs inputs, and for flow logs delivered to S3, Splunk recommends SQS-based S3. Field extraction requires the default format or one of two documented custom orders that cover versions 2 to 5. - What sourcetype do VPC Flow Logs use in Splunk?
-
The add-on uses
aws:cloudwatchlogs:vpcflowfor events andaws:cloudwatchlogs:vpcflow:metricfor a metrics index. OCSF-transformed records useocsf:aws:vpcflow. If you parse records before Splunk, use your own sourcetype, such asaws:vpcflow:nxlog, and map it to the CIM. - Do I need a Lambda function to send VPC Flow Logs to Splunk through Firehose?
-
No. AWS removed the Lambda step from its Firehose walkthrough in December 2023 because Splunk Add-on for AWS 7.3.0 and later handles flow log records from Firehose. Point Firehose at the HEC raw endpoint and use a token whose sourcetype is
aws:cloudwatchlogs:vpcflow. - Can Splunk parse VPC Flow Logs version 11 fields such as EC2 tags?
-
Not through the add-on’s documented extractions, which cover field orders up to version 5. To use fields available in versions 7 to 11, including ECS metadata,
reject-reason,encryption-status, EC2 tags, and next-hop data, parse records before Splunk, as in Path 3, or write your own search-time extractions.