Kubernetes DaemonSet log collection runs one log-collector pod on every node in your cluster. The collector reads all container logs from the node’s /var/log/containers directory through read-only hostPath mounts and forwards them to your SIEM, with no changes to your application pods. When the cluster adds a node, the DaemonSet adds a collector, so coverage scales automatically.

This guide shows you how the pattern works, how Kubernetes writes container logs in 2026, and how to deploy NXLog Agent as a DaemonSet that ships container and audit logs to Microsoft Sentinel or Splunk.

What is DaemonSet-based log collection in Kubernetes?

A DaemonSet is a Kubernetes controller that keeps a copy of a pod running on every node, or on a selected set of nodes. When a node joins the cluster, the DaemonSet schedules the pod on it. When a node leaves, the pod goes with it. That behavior makes DaemonSets the standard pattern for node-level agents, and the Kubernetes logging architecture documentation lists a node-level logging agent first among its cluster-level logging options.

You need cluster-level logging because Kubernetes lacks native centralized log storage. According to the Kubernetes logging architecture, when a container restarts, the kubelet keeps only the most recent terminated container and its logs. When Kubernetes evicts a pod from a node, it evicts its containers and their logs too. If the logs haven’t left the node by then, they are lost.

Three patterns cover cluster-level log collection:

Pattern How it works Strengths Trade-offs

DaemonSet node agent

One collector pod per node reads every container’s log files from the host filesystem

Full cluster coverage, no application changes, one agent per node regardless of pod count

Collector needs host log paths mounted and node-level read access

Sidecar collector

A collector container runs in the same pod as the application

Reaches file-based logs that never hit stdout/stderr

One collector per pod, higher total resource use, per-application configuration

Direct-from-application

The application ships its own logs to a backend

No collection infrastructure

Logging logic lives in application code; no coverage for Kubernetes system components; gaps when the application crashes

The DaemonSet is the right default. Use the sidecar only for applications that write logs exclusively to files inside the container. The NXLog Kubernetes integration guide also covers that deployment. NXLog Agent can stream those file-based logs to stdout or forward them directly to your SIEM.

What the collector reads: how Kubernetes writes container logs

A DaemonSet collector is a file reader. To configure it correctly, you need to know exactly which files exist on the node and what format they’re in. This is where most Kubernetes logging setups break, because the log format changed when Kubernetes removed the dockershim.

Here’s the pipeline. Your application writes to stdout and stderr. The container runtime captures both streams and writes them to a file per container under /var/log/pods/. To make those files easier to find, Kubernetes maintains symlinks in /var/log/containers/ named <pod-name>_<namespace>_<container-name>-<container-id>.log. The NXLog Kubernetes integration guide documents this structure and uses the filename to enrich each record with pod, namespace, and container metadata.

The format of those files depends on the runtime. Kubernetes removed the dockershim in v1.24 (May 2022), so current clusters run a CRI runtime such as containerd or CRI-O. The Kubernetes logging documentation states that the integration between the runtime and the kubelet is standardized as the CRI logging format. The kubelet CRI logging design proposal defines this format as one plain-text line per record: a timestamp, the stream name, a completeness tag, and a message. Only legacy Docker-runtime nodes still produce JSON:

# CRI format (containerd, CRI-O): one plain-text line per record
2026-08-14T09:26:31.123456789Z stderr F ERROR: connection to db-svc:5432 refused

# Legacy Docker json-file format: one JSON object per line
{"log":"ERROR: connection to db-svc:5432 refused\n","stream":"stderr","time":"2026-08-14T09:26:31.123456789Z"}

The third field in a CRI record is a tag. Per the design proposal, F marks a completed entry, either a single-line entry or the last line of a multi-line one, and P marks an entry the runtime has split and not yet finished. The proposal does not say what causes a split; in practice it is a log line too long for the runtime’s read buffer.

Rotation matters too. With CRI runtimes, the kubelet rotates container log files based on the containerLogMaxSize (default 10Mi) and containerLogMaxFiles (default 5) settings documented in the Kubernetes logging architecture and the kubelet configuration reference. With the default settings, a busy container keeps at most ~50 MiB of history on the node. The kubelet deletes older logs, which is another reason logs must leave the node quickly.

Why node-level collection matters for SecOps

Containers do not sit still long enough for you to investigate later. Sysdig’s 2025 Cloud-Native Security and Usage Report found that 60% of containers live for 60 seconds or less. Whatever a workload logged in that minute sits on the node, and only on the node, until something ships it.

For a security team, that makes transient pod logs lost evidence. An attacker who crashes a pod, or a cluster autoscaler that removes a node, destroys any log data that hasn’t shipped. To keep container logs usable as forensic material, you must ship them off the node within seconds of the write.

Compliance frameworks assume that an off-node copy exists. PCI DSS v4.0.1 Requirement 10, "Log and monitor all access to system components and cardholder data," sets the floor in 10.5.1: retain at least 12 months of audit log history, with the most recent three months immediately available for analysis. A node-local file capped at five 10 MiB rotations cannot meet that. Centralized collection can. Our guide to PCI DSS 4.0 logging requirements covers the rest of Requirement 10.

The pipeline itself is also part of your attack surface, so build it with least privilege:

  • Mount host log paths read-only. The collector reads logs. It should never alter them.

  • Grant Kubernetes API permissions only if the collector queries the API, for example to enrich events with pod metadata. The configuration below enriches records from filenames and environment variables, which needs no cluster RBAC at all.

  • Encrypt transport to the SIEM. Both output examples below use TLS.

One constraint to plan for: the NXLog Kubernetes integration guide notes that NXLog Agent needs to run as root to read the logs generated by all pods. The files under /var/log/pods are root-owned on the host, so pair that access with the read-only mounts above.

Deploy NXLog Agent as a Kubernetes DaemonSet

The steps below follow the deployment in the NXLog Kubernetes integration guide, with the manifest and parsing updated for current CRI runtimes. What changed and why is called out inline.

Before you start: build the NXLog Agent Docker image by following the Docker installation guide. The Dockerfile copies every .conf file in the build folder to /opt/nxlog/etc/nxlog.d, so you can bake your configuration into the image or mount it later as a ConfigMap. To read all pod logs, replace USER nxlog with USER root in the Dockerfile before building, as the integration guide describes. Push the image to a registry your cluster can pull from.

Step 1: ServiceAccount and RBAC (optional for this configuration)

The integration guide provisions a ServiceAccount with permission to read, list, and watch pods and namespaces. You only need this if you plan to enrich events from the Kubernetes API. The configuration in step 3 enriches from filenames and environment variables alone, so it runs with no cluster permissions. That is the least-privilege option. Create the RBAC now only if API enrichment is on your roadmap:

apiVersion: v1
kind: ServiceAccount
metadata:
  name: nxlog
  namespace: kube-system
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: nxlog
rules:
- apiGroups: [""]
  resources: ["pods", "namespaces"]
  verbs: ["get", "list", "watch"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: nxlog
roleRef:
  kind: ClusterRole
  name: nxlog
  apiGroup: rbac.authorization.k8s.io
subjects:
- kind: ServiceAccount
  name: nxlog
  namespace: kube-system

Step 2: The DaemonSet manifest

apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: nxlog
  namespace: kube-system
  labels:
    k8s-app: nxlog-logging
spec:
  selector:
    matchLabels:
      name: nxlog
  template:
    metadata:
      labels:
        name: nxlog
    spec:
      serviceAccountName: nxlog   # omit if you skipped step 1
      tolerations:
      # Collect from control-plane nodes too. kubeadm stopped applying the
      # legacy node-role.kubernetes.io/master taint in v1.25. Add a second
      # toleration for it only on clusters older than that.
      - key: node-role.kubernetes.io/control-plane
        operator: Exists
        effect: NoSchedule
      containers:
      - name: nxlog
        image: registry.example.com/nxlog:latest
        resources:
          limits:
            memory: 200Mi
          requests:
            cpu: 100m
            memory: 200Mi
        env:
        - name: NODE_NAME
          valueFrom:
            fieldRef:
              fieldPath: spec.nodeName
        volumeMounts:
        - name: varlog
          mountPath: /var/log
          readOnly: true
        # Only needed on legacy Docker-runtime nodes, where the symlinks in
        # /var/log/containers point into /var/lib/docker/containers
        - name: dockercontainers
          mountPath: /var/lib/docker/containers
          readOnly: true
        # Writable: holds the agent's cache file, so file read positions
        # survive a pod restart. See "Common pitfalls" below.
        - name: nxlog-cache
          mountPath: /var/lib/nxlog
      volumes:
      - name: varlog
        hostPath:
          path: /var/log
      - name: dockercontainers
        hostPath:
          path: /var/lib/docker/containers
      - name: nxlog-cache
        hostPath:
          path: /var/lib/nxlog
          type: DirectoryOrCreate
      terminationGracePeriodSeconds: 30

Three deliberate updates against the integration guide’s manifest:

  • The toleration targets node-role.kubernetes.io/control-plane, because kubeadm stopped applying the legacy master taint in v1.25.

  • The image comes from a registry rather than imagePullPolicy: Never, which only works with locally built images.

  • A small writable hostPath holds the agent’s cache.

Both log mounts stay readOnly, exactly as the guide ships them.

Mounting /var/log covers /var/log/containers, /var/log/pods, and any Kubernetes system logs written as files under /var/log. One caveat from the Kubernetes documentation on log locations: on systemd-based nodes, the kubelet and container runtime write to journald rather than to files, so those two components need a separate collection path if you want their logs too.

Step 3: The NXLog Agent configuration

This is where the runtime change bites. The integration guide’s example calls parse_json() on every record, which works on Docker’s json-file format. On a containerd or CRI-O node, which is what you run unless you deliberately kept Docker, the record is plain text and parse_json() has nothing to parse. The configuration below handles the CRI format first and falls back to JSON for any legacy Docker nodes still in the fleet:

# Global setting; place this in nxlog.conf. It points the cache file at the
# hostPath volume from step 2 so read positions survive a pod restart.
CacheDir  /var/lib/nxlog

envvar NODE_NAME

<Extension json>
    Module    xm_json
</Extension>

<Input k8s_containers>
    Module    im_file
    File      '/var/log/containers/*.log'
    <Exec>
        # CRI runtimes (containerd, CRI-O) write plain text:
        # <RFC3339Nano-timestamp> <stream> <P|F> <message>
        if $raw_event =~ /^(\S+) (stdout|stderr) ([FP]) (.*)$/
        {
            $EventTime = parsedate($1);
            $stream    = $2;
            $Message   = $4;
        }
        else
        {
            # Legacy Docker json-file records are JSON objects keyed
            # log/stream/time. Normalize them to the CRI field names so
            # both runtimes produce the same record shape.
            parse_json();
            $EventTime = parsedate($time);
            $Message   = $log;
            delete($log);
            delete($time);
        }

        $log_type = "k8s_container";
        $log_file = file_name();
        $k8s_node = '%NODE_NAME%';

        # Symlink names encode the source:
        # <pod-name>_<namespace>_<container-name>-<container-id>.log
        if $log_file =~ /\/.*\/(.+)_(.+)_(.+)-(.+).log$/
        {
            $k8s_pod          = $1;
            $k8s_namespace    = $2;
            $k8s_container    = $3;
            $k8s_container_id = $4;
        }
    </Exec>
</Input>

The File input module instance watches the wildcard path, so the agent picks up new containers as their symlinks appear. The filename regex and enrichment fields come straight from the integration guide. The CRI branch is the update for current runtimes.

Now the output. For Microsoft Sentinel, NXLog Agent sends events through the Azure Monitor Logs Ingestion API using the Microsoft Azure Logs Ingestion output module, authenticated with a Microsoft Entra application. Shape your records to match the columns of your data collection rule’s stream. The Microsoft Sentinel integration guide walks through the Azure-side setup and a worked field mapping:

<Output sentinel>
    Module            om_azuremonitor
    API               LogsIngestion
    ClientId          <entra-app-client-id>
    ClientSecret      <entra-app-secret>
    TenantId          <entra-tenant-id>
    URL               https://<your-dce>.ingest.monitor.azure.com
    DcrImmutableId    dcr-xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
    StreamName        Custom-K8sContainers-Stream
    Exec              to_json();
</Output>

For Splunk, send events to the HTTP Event Collector with the HTTP(s) output module, passing the HEC token in an Authorization header. The Splunk integration guide shows the full mapping, including Splunk’s time, host, and sourcetype metadata. The block below is the minimum that gets enriched records into an index:

<Output splunk_hec>
    Module         om_http
    URL            https://splunk.example.com:8088/services/collector/event
    AddHeader      Authorization: Splunk <hec-token>
    HTTPSCAFile    %CERTDIR%/splunk-ca.pem
    <Exec>
        $raw_event = '{"event":' + to_json() + '}'; # (1)
    </Exec>
</Output>

  1. Wrap the record in the HEC event envelope. Without a time key, Splunk stamps events on receipt. See the Splunk integration guide for the full mapping.

Step 4: Deploy and verify

$ kubectl apply -f nxlog-daemonset.yml
daemonset.apps/nxlog created

$ kubectl -n kube-system get daemonset nxlog
NAME    DESIRED   CURRENT   READY   UP-TO-DATE   AVAILABLE   NODE SELECTOR   AGE
nxlog   4         4         4       4            4           <none>          30s

DESIRED should equal your node count, control plane included. On the SIEM side, a collected event looks like this after enrichment (illustrative record, fields as produced by the configuration above):

{
  "EventTime": "2026-10-05T09:26:31.123456+00:00",
  "stream": "stderr",
  "Message": "ERROR: connection to db-svc:5432 refused",
  "log_type": "k8s_container",
  "k8s_node": "node003",
  "k8s_pod": "payments-api-7f6d9c5b8-x2kkq",
  "k8s_namespace": "prod",
  "k8s_container": "payments-api",
  "k8s_container_id": "bdc9da43d94800e213477bbddbc9f96fa7a32b680509580cfce97fa4186d63e5"
}

Every event now answers the questions an analyst asks first: which workload, which namespace, which node, which container instance.

Collecting Kubernetes audit logs with the same DaemonSet

Container logs tell you what your workloads did. Kubernetes audit logs tell you what everyone did to the cluster. They record every request to the API server, who made it, and how the API responded. Repeated forbidden responses against sensitive resources are the kind of signal a SecOps team wants in the SIEM, not sitting in a file on a control-plane node.

Enable auditing by defining an audit policy and pointing kube-apiserver at it with the --audit-policy-file and --audit-log-path flags, as described in the Kubernetes auditing documentation. The log backend writes one JSON object per line, and because the DaemonSet already mounts /var/log, writing the audit log under a path such as /var/log/kubernetes/audit.log makes it visible to the collector on control-plane nodes with no manifest changes. Add an input for it:

<Input k8s_audit>
    Module        im_file
    File          '/var/log/kubernetes/audit.log'
    BufferSize    150000
    <Exec>
        parse_json();
        $log_type = "k8s_audit";
        $k8s_node = '%NODE_NAME%';
    </Exec>
</Input>

Note the BufferSize. Kubernetes audit records can be large, and NXLog Agent’s input and output modules default to a 65,000-byte buffer. The integration guide documents the failure mode: the agent truncates records and logs data size (69015) is over the limit (65000), will be truncated. Raise BufferSize on both the audit input and the output module it feeds.

Plan for the disk as well. The API server rotates its own audit log, and the kube-apiserver flag reference puts the defaults at 100 MB per file (--audit-log-maxsize), 100 retained files (--audit-log-maxbackup), and 366 days (--audit-log-maxage). A busy cluster can therefore park roughly 10 GB of audit history on each control-plane node, and once the backup count fills, the API server deletes the oldest file. Shipping the log off the node as it is written stops that rotation from setting your retention window.

Managing the agent fleet with NXLog Platform

A DaemonSet solves deployment. It doesn’t solve operations: changing a parsing rule across the fleet, spotting the one agent that stopped sending, or keeping configurations consistent while the autoscaler adds and removes nodes overnight.

That’s the job of NXLog Platform. Agents enroll with NXLog Platform, and auto-enrollment rules automatically assign a configuration to each new agent, so the collector on a freshly scaled node comes up with the right configuration without anyone touching it. You can assign one configuration to many agents, which turns "update the CRI regex everywhere" from a redeploy into a single edit. NXLog Platform also ships ready-made configuration templates for common log sources and SIEM destinations, Microsoft Sentinel and Splunk among them, and shows you per-agent status so a silent node stands out. The NXLog Platform User Guide covers enrollment and fleet management in detail.

Common pitfalls

parse_json() fails on every container log

Your cluster runs containerd or CRI-O, and the records are plain text in the CRI format. This affects any configuration written for the Docker era, including older published examples. To fix this, parse the CRI fields with a regex first, as in step 3, and keep the JSON branch only as a fallback.

Stack traces arrive as one event per line

The runtime writes one record per line, so a 40-line Java exception becomes 40 events. Use the Multiline Parser extension with a header pattern that matches the start of your application’s log entries to join them back together. Separately, the runtime splits log entries that are too long for its read buffer and tags every fragment but the last with P. If your applications emit long lines, plan to reassemble on the P/F tags.

The DaemonSet skips control-plane nodes

Control-plane nodes carry a NoSchedule taint, and since kubeadm v1.25 that taint is node-role.kubernetes.io/control-plane, not the legacy master taint older manifests tolerate. If DESIRED is smaller than your node count, check which taint your control-plane nodes carry with kubectl describe node and tolerate that one.

Audit events arrive truncated

Audit records can exceed NXLog Agent’s 65,000-byte default buffer. Raise BufferSize on both the audit input and the output module it feeds, per the integration guide.

The agent re-sends old events after a pod restart

NXLog Agent remembers file read positions in its cache file, but if the cache lives on the container’s ephemeral filesystem, a pod restart wipes it and the agent re-reads every file. The manifest in step 2 mounts /var/lib/nxlog from the host for exactly this reason. Point the CacheDir global directive to this path.

Ship your cluster’s logs before the cluster deletes them

A DaemonSet gives you one collector per node, every container covered, and no application changes. NXLog Agent adds the parsing, enrichment, and SIEM delivery on top, and NXLog Platform keeps the whole fleet configured and visible as your cluster scales. The full deployment reference lives in our Kubernetes integration guide, and you can try NXLog Platform for free to manage your first DaemonSet fleet.

FAQ

Should I use a DaemonSet or a sidecar for Kubernetes logging?

Use a DaemonSet by default: one collector per node covers every container that writes to stdout/stderr, with no application changes. Add a sidecar only for applications that write logs exclusively to files inside the container, where the node-level collector can’t see them.

Where does Kubernetes store container logs?

The container runtime writes each container’s stdout/stderr to a file under /var/log/pods/ on the node, and Kubernetes maintains symlinks in /var/log/containers/ named <pod>_<namespace>_<container>-<container-id>.log. By default, the kubelet rotates these files at 10 MiB and keeps 5 files per container.

Can a DaemonSet collect logs from control-plane nodes?

Yes. Add a toleration for the node-role.kubernetes.io/control-plane:NoSchedule taint to the DaemonSet’s pod spec. Clusters created with kubeadm 1.24 or older may also carry the legacy node-role.kubernetes.io/master taint.

Does the log collector need to run as root?

To read the log files of all pods under /var/log/pods, yes. The files are root-owned on the host. NXLog Agent’s Docker image runs as the nxlog user by default, and the Kubernetes integration guide describes switching it to root for this deployment. Keep the host log mounts read-only to limit what that access can do.

How do I collect Kubernetes audit logs?

Enable auditing on the API server with an audit policy and the --audit-log-path flag, then read the resulting JSON-lines file with the DaemonSet collector on control-plane nodes. Raise the collector’s buffer size, because audit records can exceed the 65,000-byte default.

NXLog Platform is an on-premises solution for centralized log management with
versatile processing forming the backbone of security monitoring.

With our industry-leading expertise in log collection and agent management, we comprehensively
address your security log-related tasks, including collection, parsing, processing, enrichment, storage, management, and analytics.

Start free Contact us
Share
Related Posts