News and blog
NXLog main page
  • Products
    NXLog Platform
    Log collection
    NXLog Agent
    NXLog Community Edition
    Integrations
    Professional Services
  • Solutions
    Use cases
    Specific OS support
    SCADA/ICS
    Windows event log
    DNS logging
    MacOS logging
    Open Telemetry
    Cost reduction
    Industries
    Financial Services
    Government & Education
    Entertainment & Gambling
    Telecommunications
    Medical & Healthcare
    Military & Defense
    Law Firms & Legal Counsel
    Industrial & Manufacturing
  • Pricing
    Licensing
    Plans
  • Partners
    Find a Reseller
    Partner Program
    Partner Portal
  • Resources
    Documentation
    Blog
    White papers
    Videos
    Webinars
    Case Studies
    Community Program
    Community Forum
  • About
    Company
    Careers
  • Support
    Support portals
    Contact us

NXLog Platform
Log collection
NXLog Agent
NXLog Community Edition
Integrations
Professional Services

Use Cases
Specific OS support
SCADA/ICS
Windows event log
DNS logging
MacOS logging
Open Telemetry
Cost reduction
Solutions by industry
Financial Services
Government & Education
Entertainment & Gambling
Telecommunications
Medical & Healthcare
Military & Defense
Law Firms & Legal Counsel
Industrial & Manufacturing

Licensing
Plans

Find a Reseller
Partner Program
Partner Portal

Documentation
Blog
White papers
Videos
Webinars
Case Studies
Community Program
Community Forum

Company
Careers

Support portals
Contact us
Let's Talk
  • Start free
  • Interactive demo
Let's Talk
  • Start free
  • Interactive demo
NXLog search
  • Loading...
Let's Talk
  • Start free
  • Interactive demo
November 19, 2026 siem

VPC Flow Logs to Splunk: three ingestion paths compared

By João Correia

Share
ALL ANNOUNCEMENT COMPARISON COMPLIANCE DEPLOYMENT SECURITY SIEM STRATEGY RSS

You can send AWS VPC Flow Logs to Splunk in three main ways:

  • Publish them to Amazon S3 and pull them with the SQS-based S3 input in the Splunk Add-on for AWS.

  • Stream them through Amazon Data Firehose to the HTTP Event Collector (HEC).

  • Run NXLog Agent between S3 and HEC to parse, filter, and route records before Splunk indexes them.

The Firehose path is the fastest, because it skips S3 file batching and polling. The add-on’s documented extractions stop at version 5, while the NXLog Agent path parses newer fields without custom Splunk extractions.

Key takeaways

  • Firehose to HEC is the push path. You don’t need a Lambda transform with Splunk Add-on for AWS 7.3.0 or later, but the HEC token needs indexer acknowledgment.

  • SQS-based S3 is Splunk’s recommended pull input. Plan for S3 file batching and polling delay.

  • The add-on extracts v2 to v5 layouts only. To use reject reasons, encryption status, or the June 2026 EC2 tag and next-hop fields, parse records before Splunk or write your own extractions.

  • Keep S3 as your system of record. Index what your detections use and replay the rest when an investigation needs it.

What VPC Flow Logs record, and what they miss

Each flow log record summarizes one network flow on one network interface during an aggregation window. AWS writes each record as a line of space-separated fields and uses a dash for any value that doesn’t apply.

# version account-id interface-id srcaddr dstaddr srcport dstport protocol packets bytes start end action log-status
2 111122223333 eni-0a1b2c3d4e5f67890 10.0.12.45 10.0.40.7 51544 443 6 14 6031 1791417660 1791417720 ACCEPT OK

This default-format record shows an accepted flow of 14 TCP packets and 6,031 bytes to port 443 on 10.0.40.7. The start and end values are Unix timestamps in seconds.

Field versions at a glance

The default format contains only version 2 fields. A custom format lets you pick fields from any version and set their order.

Version Fields added Why it matters for detection

2

version, account-id, interface-id, 5-tuple, packets, bytes, start, end, action, log-status

Who talked to whom, how much, and whether it was allowed

3

vpc-id, subnet-id, instance-id, tcp-flags, type, pkt-srcaddr, pkt-dstaddr

Ties flows to workloads. Packet-level addresses see through NAT gateways and EKS pod networking

4

region, az-id, sublocation-type, sublocation-id

Region, Availability Zone, Local Zone, Wavelength Zone, and Outposts context

5

pkt-src-aws-service, pkt-dst-aws-service, flow-direction, traffic-path

Identifies AWS service traffic, ingress versus egress, and the path egress traffic took

7

ecs-cluster-arn, ecs-service-name, ecs-task-id, and other ECS fields

Attributes flows to ECS clusters, services, and tasks

8

reject-reason

Shows rejects caused by VPC Block Public Access (BPA) or VPC encryption controls (EC)

9

resource-id

Identifies the regional NAT gateway behind a flow

10

encryption-status

Shows whether a flow was Nitro-encrypted, application-encrypted, both, or neither

11

instance-tag, interface-tag, asg-tag, interface-type, next-hop fields

Embeds EC2 tag values and next-hop interface details in each record

There is no version 6 row because version 6 fields apply to Transit Gateway flow logs only.

AWS announced the version 11 fields in June 2026. Treat fields that don’t come from packet headers, such as ECS and tag metadata, as context: AWS describes them as best-effort values that can be missing or inaccurate. Tag values in particular can be missing or inaccurate for the first hour after you create a subscription.

Delay, gaps, and record quirks

Flow logs aren’t a real-time feed. AWS aggregates each flow for up to 10 minutes by default, or 1 minute if you set it, and Nitro-based instances use 1 minute or less regardless of the setting. Best-effort delivery then typically takes about 5 minutes to CloudWatch Logs and about 10 minutes to S3. Timestamps are approximate too: start can land up to 60 seconds before or after the first packet was seen.

Flow logs also skip some traffic by design, including instance traffic to the Amazon DNS server, instance metadata (169.254.169.254), Amazon Time Sync, DHCP, ARP, and Traffic Mirroring source traffic. Traffic to DNS servers you run yourself is logged. For DNS-layer visibility through the Amazon resolver, add Route 53 Resolver query logs.

Three record details shape detection logic:

A REJECT can come from timing, not policy

AWS notes that packets arriving after a connection closes are also recorded as REJECT.

Interface addresses can hide the real endpoints

srcaddr and dstaddr can show an intermediate interface instead, such as a NAT gateway or the EKS node rather than the pod, while pkt-srcaddr and pkt-dstaddr keep the original addresses.

TCP flags are aggregated

Flags can be OR-ed across the window, and only FIN, SYN, RST, and SYN-ACK are recorded, so traffic carrying only ACK or PSH shows 0.

Which path should you use?

The three paths differ in where records wait, where parsing happens, and how much control you have before Splunk indexes the data. If your detections read only version 2 to 5 fields, Path 1 or Path 2 covers them.

Path 1: S3 and SQS-based S3 input Path 2: Firehose to HEC Path 3: NXLog Agent pipeline

Data flow

S3, then SNS and SQS, then the Splunk Add-on for AWS

Amazon Data Firehose, then the HEC raw endpoint

S3, then SQS, then NXLog Agent, then the HEC event endpoint

What adds delay

S3 file batching and delivery, plus the input’s collection interval (300 seconds by default)

Aggregation window plus Firehose buffering

S3 file batching and delivery, plus SQS polling

Fields Splunk extracts

v2 to v5 layouts

v2 to v5 layouts, or OCSF

Any fields you list in the parser

Filtering before indexing

Flow log filter and field selection

Flow log filter and field selection, or a Lambda transform you maintain

Drop, enrich, and normalize rules in NXLog Agent

Choose it when

You want Splunk-native collection from S3

You want the shortest path from flow to search

You need v7 to v11 fields, control over what Splunk indexes, or several destinations

Skip the CloudWatch Logs pull input. Splunk advises against pull-based API inputs for CloudWatch Logs because of rate limits, and points you to Firehose instead.

Path 1: Amazon S3 and the SQS-based S3 input

Splunk names the SQS-based S3 input the best practice for VPC flow logs delivered to S3. S3 announces each new log file through SNS and SQS, and the add-on downloads and indexes it. Three details decide whether it works:

Field order

The add-on extracts fields only when records follow the default format or one of two documented custom orders. The documented orders are a 29-field order covering versions 2 to 5, or the same 29 fields in alphabetical order, which Splunk calls Select All. That name predates the version 7 to 11 fields, so don’t select every field in today’s console.

Queue setup

Splunk’s prerequisites are an SNS topic for S3 object-created notifications, an SQS queue subscribed to it, a dead-letter queue, and the listed IAM permissions. Set the visibility timeout to at least 5 minutes, or unfinished files return to the queue and the add-on indexes them twice.

Bucket notifications

Saving a notification configuration replaces the bucket’s whole configuration, so merge in existing rules first, and allow S3 to publish to the SNS topic before you save.

The add-on sets the sourcetype to aws:cloudwatchlogs:vpcflow and doesn’t accept a custom one. The input only collects files that S3 announces through SQS after you set it up, so use the Generic S3 input for history. To scale, point inputs on more collection nodes at the same queue: the input is stateless, so the others keep going if one fails.

Path 2: Amazon Data Firehose to Splunk HEC

VPC Flow Logs can publish straight to Firehose, which pushes records to HEC without waiting for S3 files. AWS updated its Splunk walkthrough in December 2023 to drop the Lambda transform, because Splunk Add-on for AWS 7.3.0 and later handles flow log records from Firehose. A guide that still adds a Lambda function predates that change, and older material may use the name the service had until February 2024: Amazon Kinesis Data Firehose.

Firehose is strict about the HEC endpoint:

Trusted certificate

Firehose requires a CA-signed certificate that matches the HEC endpoint’s DNS name, so self-signed certificates don’t work. On Splunk Cloud, the Firehose endpoint is https://http-inputs-firehose-<stack>.splunkcloud.com:443.

Indexer acknowledgment

Firehose waits for HEC to confirm each delivery. On Splunk Cloud, Firehose is the only supported client for acknowledgment-enabled tokens, so give it its own token. On Splunk Enterprise, the global [http] stanza in the HEC app’s inputs.conf defaults to disabled = 1, which turns off every token, so enable it too. Behind a load balancer, AWS recommends sticky sessions so acknowledgment checks reach the indexer that received the data.

When you create the stream, Splunk recommends the raw endpoint unless a transform turns records into HEC event JSON. Size the retry window, 0 to 7,200 seconds, to cover a realistic HEC outage. Backup mode is one-way: you can switch from FailedEventsOnly to AllEvents but not back, so start with FailedEventsOnly or validate on a separate test stream. Keep the token out of the stream definition with SecretsManagerConfiguration. The stream must use Direct PUT as its source, and AWS lists the IAM permissions and cross-account roles the flow log needs.

OCSF variant. From add-on version 8.3.0, Splunk also accepts flow logs that CloudWatch Logs converted to OCSF, the Open Cybersecurity Schema Framework. It suits detections that already target OCSF, but the extra CloudWatch Logs hop adds ingestion and storage costs.

Path 3: NXLog Agent between S3 and Splunk HEC

Choose this path when you want to decide what Splunk indexes, use fields beyond version 5, or send the same records to more than one destination. NXLog Agent picks up each new flow log file as S3 announces it, parses every record against the field list you configured, adds CIM field names, and sends compressed batches to HEC.

If you already run Path 1, keep it and give Path 3 its own flow log under an nxlog/ S3 prefix, with its own notification rule and SQS queue. Scope Path 1’s notification rule to its own prefix first: S3 rejects overlapping prefix filters for the same event type, and a rule with no prefix overlaps every other. Separate prefixes also mean each consumer reads only the field layout it parses. AWS allows up to 250 flow log subscriptions per resource in an account, but each one publishes and bills its own copy of the data.

Pick the fields you need

NXLog Agent parses by field position from a list you control, so your format can include fields the add-on doesn’t extract, such as reject-reason (version 8) and EC2 instance tags (version 11). Tag fields need extra permissions, starting with ec2:DescribeTags. AWS percent-encodes special characters in tag values, so use Splunk’s urldecode() eval function when a report needs the original text.

Read new files from SQS

NXLog Agent’s Python input module runs a short Python reader that you write with boto3, the AWS SDK for Python. The NXLog AWS integration guide shows a setup for CloudWatch Logs, and your reader can follow the same pattern. Your reader long-polls SQS, streams each gzip file from S3, skips the header line, and posts one event per flow record. It deletes a message only after posting every record from that file, so with PersistLogqueue and SyncLogqueue enabled on the output, delivery is at-least-once, and a redelivered message can repeat records in Splunk. To scale out, run more agents against the same queue. NXLog Agent’s Amazon S3 input module polls a bucket on a schedule instead, so this path uses SQS notifications to pick up each file as it lands and to share the work across agents.

Parse, normalize, and forward

The configuration parses each record with the CSV extension, using a space delimiter and UndefValue set to -, so the dashes AWS writes for missing values become undefined fields. It then drops what you don’t want indexed, adds CIM field names, wraps each record in the HEC event format with time set from the record’s start field, and sends gzip-compressed batches with the HTTP(s) output module. An excerpt from the input’s Exec block:

# No traffic during the window: nothing to analyze
if ($log_status == "NODATA") {
    drop();
}

# CIM addresses: prefer packet-level values (NAT gateway, EKS pods)
$src_ip = defined($pkt_srcaddr) ? $pkt_srcaddr : $srcaddr;
$dest_ip = defined($pkt_dstaddr) ? $pkt_dstaddr : $dstaddr;

# Example policy: drop accepted flows from load balancer subnets,
# such as health checks. Replace the pattern with your own.
# Rejects are kept.
if (($action == "ACCEPT") and ($src_ip =~ /^10\.0\.(8|9)\./)) {
    drop();
}

if ($action == "ACCEPT") {
    $action = "allowed";
}
else if ($action == "REJECT") {
    $action = "blocked";
}

On the delivery side:

Retries

The HTTP(s) output module treats any response other than 200, 201, or 202 as an error and resends. It splits a failed batch into one request per record, and an OnError block reroutes any record HEC rejects as malformed (HTTP 400) to a local file.

Durability

With PersistLogqueue, NXLog Agent removes records from its on-disk queue only after delivery succeeds. Our guide to log shipper offline buffering covers sizing and testing.

Acknowledgment

This path’s HEC token doesn’t use indexer acknowledgment, so an HTTP 200 means HEC accepted the request, not that Splunk has indexed it.

Tell Splunk how to read the events. Send them with your own sourcetype, such as aws:vpcflow:nxlog. Then, deploy search-time field aliases, an event type, and the network and communicate tags to your search heads so the events join the CIM Network Traffic data model. Search-time settings don’t add to what you index.

Manage it with NXLog Pipeline Manager. When NXLog Agent runs in several AWS accounts or Regions, NXLog Pipeline Manager lets you save your NXLog Agent configuration once and apply it to every agent in one step. You can also limit who can change it with role-based access control (RBAC). Make sure your Python reader script is on each agent host, at the path the configuration references.

Map flow log fields to the Splunk CIM

With Paths 1 and 2, the add-on’s aws:cloudwatchlogs:vpcflow sourcetype is marked CIM compliant. With Path 3, you own the mapping. Apply it in your NXLog Agent configuration and Splunk settings as follows (the excerpt above covers the address and action rows):

VPC Flow Logs field CIM field Mapping

pkt-srcaddr, falling back to srcaddr

src_ip, src

Packet-level source, so NAT gateway and EKS traffic shows the original endpoint

pkt-dstaddr, falling back to dstaddr

dest_ip, dest

Packet-level destination

srcport, dstport

src_port, dest_port

Numeric, unchanged

protocol

transport, protocol

IANA number to name (6 tcp, 17 udp, 1 and 58 icmp), and protocol set to ip

action

action

ACCEPT to allowed, REJECT to blocked

flow-direction

direction

ingress to inbound, egress to outbound

bytes, packets

bytes, packets

Unchanged

account-id

vendor_account

Search-time alias

interface-id

dvc

Search-time alias

reject-reason

rule

Search-time alias (BPA or EC)

The direction field describes traffic relative to the interface that logged it, not your VPC boundary.

Detection searches to start with

This search queries the Network Traffic data model with tstats, so it works for any path once your data maps to the CIM. It assumes an accelerated data model (set summariesonly=false if yours isn’t), and its threshold is a starting point for your own baseline.

Scanning from one source. Blocked flows to many distinct ports within 15 minutes:

| tstats summariesonly=true count AS blocked_flows dc(All_Traffic.dest_port) AS distinct_ports
    from datamodel=Network_Traffic.All_Traffic
    where All_Traffic.action="blocked"
    by All_Traffic.src_ip _time span=15m
| rename "All_Traffic.*" AS *
| where distinct_ports >= 25
| sort - distinct_ports

The same pattern covers allowed SSH and RDP flows from public addresses, and large outbound transfers. An outbound search relies on direction, which you set in your Path 3 configuration (see the CIM mapping table), so with add-on data, check that direction is populated first.

Block Public Access hits (Path 3 data). This one uses the version 8 reject-reason field, which the add-on doesn’t extract:

index=<your_index> sourcetype="aws:vpcflow:nxlog" reject_reason="BPA"
| stats count min(_time) AS first_seen max(_time) AS last_seen BY src_ip dest_ip dest_port vpc_id
| convert ctime(first_seen) ctime(last_seen)

Keep volume and cost under control

AWS bills VPC Flow Logs as CloudWatch vended logs at tiered per-GB rates, whether they go to CloudWatch Logs, S3, or Firehose, and Firehose adds its own per-GB ingestion charge for vended-log sources. Rates vary by Region and change over time, so this post quotes none.

At the source

Capture ACCEPT, REJECT, or ALL traffic for a VPC, a subnet, or a single network interface, and use a custom format with only the fields you need. The 1-minute interval produces more records than the 10-minute default.

Parquet is for Athena, not Splunk

AWS reports 10 to 100 times faster queries and 20 percent less storage than gzip text, but Parquet adds conversion charges, and these Splunk paths read plain text.

In Splunk

Send trend data to a metrics index, or filter before indexing with SPL2 pipelines in Edge Processor and Ingest Processor.

In the pipeline (Path 3)

One starting policy: keep every REJECT and every flow that crosses your VPC boundary, then drop records that no detection or report uses. The original files stay in S3 for replay when an investigation needs them.

Ports of Auckland, a NXLog customer, credits NXLog Agent’s filtering for letting the team ingest only valuable events and save on EPS volume.

Validate the VPC Flow Logs to Splunk pipeline

Check each hop in order. At the source, aws ec2 describe-flow-logs should report DeliverLogsStatus as SUCCESS. In Splunk, compare paths by the p95 of _indextime - _time per sourcetype. Then check each path’s own signals:

Path 1

Splunk suggests a CloudWatch alarm on the queue’s ApproximateAgeOfOldestMessage or ApproximateNumberOfMessagesVisible as the trigger for adding collection nodes.

Path 2

Turn on Firehose error logging, and send the backup bucket’s S3 notifications to an SQS-based S3 input with parse_firehose_error_data enabled to re-ingest failed events.

Path 3

Check the NXLog Agent log and the local file of rejected records. TcpSendStallDetect TRUE on the HTTP(s) output instance logs a warning when sends to HEC are blocked.

Choose the path that fits your detections

Start from the fields and latency your detections need. If fields from versions 2 to 5 cover them, Path 1 or Path 2 uses Splunk’s own add-on. If you need newer fields, control over what Splunk indexes, or more than one destination, put NXLog Agent in the middle and manage it with NXLog Pipeline Manager.

Ready to try it? Start with the Free plan and filter your own flow logs before Splunk indexes them.

FAQ

Does the Splunk Add-on for AWS support VPC Flow Logs?

Yes. The add-on indexes VPC Flow Logs as aws:cloudwatchlogs:vpcflow through SQS-based S3, Firehose to HEC, Kinesis, or CloudWatch Logs inputs, and for flow logs delivered to S3, Splunk recommends SQS-based S3. Field extraction requires the default format or one of two documented custom orders that cover versions 2 to 5.

What sourcetype do VPC Flow Logs use in Splunk?

The add-on uses aws:cloudwatchlogs:vpcflow for events and aws:cloudwatchlogs:vpcflow:metric for a metrics index. OCSF-transformed records use ocsf:aws:vpcflow. If you parse records before Splunk, use your own sourcetype, such as aws:vpcflow:nxlog, and map it to the CIM.

Do I need a Lambda function to send VPC Flow Logs to Splunk through Firehose?

No. AWS removed the Lambda step from its Firehose walkthrough in December 2023 because Splunk Add-on for AWS 7.3.0 and later handles flow log records from Firehose. Point Firehose at the HEC raw endpoint and use a token whose sourcetype is aws:cloudwatchlogs:vpcflow.

Can Splunk parse VPC Flow Logs version 11 fields such as EC2 tags?

Not through the add-on’s documented extractions, which cover field orders up to version 5. To use fields available in versions 7 to 11, including ECS metadata, reject-reason, encryption-status, EC2 tags, and next-hop data, parse records before Splunk, as in Path 3, or write your own search-time extractions.

NXLog Platform is an on-premises solution for centralized log management with
versatile processing forming the backbone of security monitoring.

With our industry-leading expertise in log collection and agent management, we comprehensively
address your security log-related tasks, including collection, parsing, processing, enrichment, storage, management, and analytics.

Start free Contact us
  • Splunk
  • Log forwarding
  • AWS
Share

Facebook Twitter LinkedIn Reddit Mail
Related Posts

Making the most of Windows Event Forwarding for centralized log collection in 2026
7 minutes | July 8, 2026
DNS Log Collection on Windows
9 minutes | May 28, 2020
Sending ETW Logs to Splunk with NXLog
5 minutes | March 3, 2020

Stay connected:

Featured posts

Announcing NXLog Platform 1.15
October 8, 2026
Announcing NXLog Platform 1.14
August 19, 2026
Announcing NXLog Platform 1.13
June 9, 2026
Enterprise IIS log analysis software: top tools, use cases, and NXLog Agent integration
May 7, 2026
Announcing NXLog Platform 1.12
April 21, 2026
How to visualize telemetry data flow and volume with NXLog Platform
March 23, 2026
Security dashboards go dark: why visibility isn't optional, even when your defenses keep running
February 26, 2026
Building a practical OpenTelemetry pipeline with NXLog Platform
February 25, 2026
Announcing NXLog Platform 1.11
February 23, 2026
Adopting OpenTelemetry without changing your applications
February 10, 2026
Linux security monitoring with NXLog Platform: Extracting key events for better monitoring
January 9, 2026
2025 and NXLog - a recap
December 18, 2025
Announcing NXLog Platform 1.10
December 11, 2025
Announcing NXLog Platform 1.9
October 22, 2025
Gaining valuable host performance metrics with NXLog Platform
September 30, 2025
Security Event Logs: Importance, best practices, and management
July 22, 2025
Enhancing security with Microsoft's Expanded Cloud Logs
June 10, 2025

Categories

  • ANNOUNCEMENT
  • COMPARISON
  • COMPLIANCE
  • DEPLOYMENT
  • SECURITY
  • SIEM
  • STRATEGY
  • Products
  • NXLog Platform
  • NXLog Agent
  • NXLog Community Edition
  • Integration
  • Professional Services
  • Licensing
  • Plans
  • Resources
  • Documentation
  • Blog
  • White Papers
  • Videos
  • Webinars
  • Case Studies
  • Community Program
  • Community Forum
  • Compare NXLog Platform
  • Partners
  • Find a Reseller
  • Partner Program
  • Partner Portal
  • About NXLog
  • Company
  • Careers
  • Support Portals
  • Contact Us

Follow us

LinkedIn Facebook YouTube Reddit
logo

© Copyright NXLog Ltd.

Privacy Policy • General Terms of Business