Key takeaways

  • Container logs share the container’s lifecycle. The Kubernetes documentation states that when a pod is evicted from a node, it removes the containers along with their logs.

  • Container lifespans keep shrinking. Sysdig’s 2025 Cloud-Native Security and Usage Report found that 60% of containers live for 60 seconds or less. Collection has to be continuous, not scheduled.

  • A node-level agent (a DaemonSet on Kubernetes) covers every container on a node without changing your applications. NXLog publishes ready-to-use configurations for this pattern.

  • containerd and CRI-O write plain-text CRI-format lines, not the JSON that Docker Engine produces. Your parsing has to match your runtime.

  • Application logs are only part of the picture. Collect Kubernetes audit and system logs too, and raise NXLog Agent’s buffer size for large audit records.

What is containerized log collection?

Containerized log collection is the practice of capturing log data from containers and forwarding it to storage with an independent lifecycle from the containers themselves. That data includes the standard output and standard error streams the container runtime writes to files on the host, plus any log files applications create inside containers.

That independence is the whole point. The Kubernetes logging architecture documentation calls this cluster-level logging: log storage and lifecycle kept separate from nodes, pods, and containers. Kubernetes does not ship a native storage backend for log data, so collection is your responsibility.

For a SecOps team, the stakes show up during incident response. By the time you investigate, the container that produced the evidence is often long gone. If its logs went with it, your timeline has a hole exactly where the attacker was active.

Why do container logs disappear?

Three mechanics work against you.

Logs live and die with the workload

On Kubernetes, the kubelet keeps one terminated container’s logs after a restart, and when a pod is evicted, it removes the containers along with their logs. On standalone Docker hosts, the default json-file driver writes each container’s logs inside that container’s own directory on the host.

Rotation trims what remains

The kubelet rotates container logs with defaults of 10 MiB per file (containerLogMaxSize) and 5 files per container (containerLogMaxFiles), and kubectl logs returns only the contents of the latest file. Docker keeps json-file without log rotation as its default for backward compatibility, so instead of losing old entries to rotation, you get log files that grow until they fill the host’s disk. Docker’s own recommendation is to switch to the local driver, which rotates at 20 MB per file across 5 compressed files by default, or to set max-size and max-file on json-file yourself. Note that max-file has no effect unless max-size is also set, which is a common reason rotation silently doesn’t happen.

{
  "log-driver": "local",
  "log-opts": {
    "max-size": "20m",
    "max-file": "5"
  }
}

Either way, you are choosing how much history to keep on the node, not whether to keep it. Capping the files protects the disk and shortens the window in which an agent can collect the evidence.

Lifespans keep shrinking

Sysdig’s 2025 Cloud-Native Security and Usage Report, an annual analysis of real-world usage data from Sysdig customers, found that for the first time, 60% of containers live for 60 seconds or less. When Sysdig first measured lifespans in 2019, half of containers lasted at least five minutes. The window for collecting a container’s logs has shrunk from minutes to seconds.

The population running on this foundation keeps growing, too. The CNCF’s 2025 Annual Cloud Native Survey (results published January 2026) reports that 82% of container users run Kubernetes in production, up from 66% in 2023.

There is a compliance angle as well. NIST’s Application Container Security Guide (SP 800-190), by Souppaya, Morello, and Scarfone (2017), maps container technology security to NIST SP 800-53 controls, including the Audit and Accountability family. Those controls are difficult to evidence when the logs vanish with the workload.

Where do container logs live?

Environment Default location on the host Format

Docker Engine (json-file driver)

/var/lib/docker/containers/<container-id>/<container-id>-json.log

JSON, one object per line

Kubernetes (any CRI runtime)

/var/log/pods/, with symlinks in /var/log/containers/

CRI logging format, written by the runtime

Kubernetes node components

journald on systemd nodes, or log files under /var/log

Plain text / journal

On Kubernetes, the kubelet directs the container runtime to write logs into /var/log/pods, and a parallel symlink structure in /var/log/containers gives collectors one flat directory to tail. The symlink filenames follow the pattern <pod-name>_<namespace>_<container-name>-<container-id>.log, and the NXLog Agent configurations below turn that filename into per-event metadata.

What are the four ways to collect container logs?

The Kubernetes documentation describes three cluster-level logging patterns: a node-level agent, a logging sidecar, and pushing logs directly from the application. On standalone Docker hosts, there’s a fourth option: the logging driver push.

Pattern How it works App changes Trade-offs

Node-level agent

One agent per node (a DaemonSet on Kubernetes) tails the runtime’s log files

None

Needs host path access, and usually root, on each node; parsing must match the runtime’s log format

Sidecar agent

An agent container in the pod reads app log files from a shared volume

Pod spec change

Per-pod resource overhead; the right fit when apps write files instead of stdout

Logging driver push (Docker)

The Docker daemon ships logs via a syslog, gelf, or fluentd driver

None (daemon config)

Driver choice affects docker logs behavior; GELF over UDP offers no delivery guarantees

Direct from the application

The app ships its own logs to a backend

Code or library change

Couples logging to application code; collects nothing from the runtime or host

The node-level agent is the first pattern the Kubernetes documentation lists, and it’s the one we recommend as your default. One agent per node, complete coverage of everything writing to stdout and stderr, and no changes to application pods. The pattern is the same whether that agent is NXLog Agent, Fluent Bit, or Fluentd. What differs is what else the agent can collect, and how you manage the fleet, which we cover at the end. Beyond containers, NXLog Agent also handles Windows Event Log, syslog, and flat files. Our integration guides document more than 100 log sources and destinations across the rest of your estate.

Reserve the sidecar for the exception: applications that write log files inside the container instead of logging to stdout. NXLog documents both patterns with working manifests.

How do you collect Docker container logs with NXLog Agent?

Run NXLog Agent on the Docker host and tail the json-file logs directly. This configuration comes from our Docker integration guide:

<Extension fileop>
    Module    xm_fileop
</Extension>

<Extension json>
    Module    xm_json
</Extension>

<Input container_logs>
    Module    im_file
    File      '/var/lib/docker/containers/*/*-json.log'
    <Exec>
        parse_json();
        $HostID = file_basename(file_name());
        $HostID =~ s/-json.log//;
    </Exec>
</Input>

The File input module supports wildcards, so one input block covers every container on the host, including ones that don’t exist yet. The JSON extension parse_json() call turns Docker’s log, stream, and time fields into structured event data, and the Exec block extracts the container ID from the filename into $HostID. Pair it with the output that matches your pipeline: Elasticsearch, HTTP(s), or any other output module.

If you’d rather push than tail, Docker’s GELF logging driver can send container logs straight to an NXLog Agent listener over the network. Set the driver in /etc/docker/daemon.json:

{
  "log-driver": "gelf",
  "log-opts": {
    "gelf-address": "udp://<agent-host>:12201"
  }
}

Then have NXLog Agent listen for GELF, using the GELF extension:

<Extension gelf>
    Module        xm_gelf
</Extension>

<Input docker_gelf>
    Module        im_udp
    ListenAddr    0.0.0.0:12201
    InputType     GELF_UDP
</Input>

One caveat before you standardize on this: UDP makes no delivery guarantees. The GELF extension also supports GELF_TCP with the TCP input module if you need reliable transport.

How do you collect Kubernetes logs with an NXLog Agent DaemonSet?

Deploying NXLog Agent as a DaemonSet puts one collector on every node, reading every pod’s logs with no changes to your workloads. NXLog provides a Docker package for building the agent image, and the Kubernetes integration guide walks through the full setup: a ServiceAccount, a ClusterRole granting get, list, and watch on pods and namespaces, and the binding between them. The agent needs to run as root to read logs from all pods.

Here is the DaemonSet, abridged for length. The full manifest, including RBAC, tolerations, and resource limits, is in the guide:

apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: nxlog
  namespace: kube-system
spec:
  selector:
    matchLabels:
      name: nxlog
  template:
    metadata:
      labels:
        name: nxlog
    spec:
      volumes:
      - name: dockercontainers
        hostPath:
          path: /var/lib/docker/containers
      - name: varlog
        hostPath:
          path: /var/log
      containers:
      - name: nxlog
        image: nxlog:latest
        volumeMounts:
        - name: dockercontainers
          mountPath: /var/lib/docker/containers
          readOnly: true
        - name: varlog
          mountPath: /var/log
          readOnly: true
        env:
        - name: NODE_NAME
          valueFrom:
            fieldRef:
              fieldPath: spec.nodeName

The hostPath mounts give the agent read-only access to the node’s log directories, and the downward API injects the node name so every event carries it. Mounting /var/log is the part that matters on a current cluster: it covers /var/log/containers and the /var/log/pods files its symlinks point to. The /var/lib/docker/containers mount only helps on nodes still running Docker Engine as the runtime, so you can drop it from clusters on containerd or CRI-O. When you test this configuration, if the capture fails, add FollowSymLinks TRUE to the input block.

The NXLog Agent configuration below, from the same guide, tails /var/log/containers, parses the JSON, and extracts pod, namespace, container, and container ID from the filename:

envvar NODE_NAME

<Extension json>
    Module    xm_json
</Extension>

<Input k8s_containers>
    Module    im_file
    File      '/var/log/containers/*.log'
    <Exec>
        parse_json();

        $log_type = "k8s_container";
        $log_file = file_name();
        $k8s_node = '%NODE_NAME%';

        # Log filenames are in the format
        # <pod-name>_<namespace>_<container-name>-<container-id>.log
        if $log_file =~ /\/.*\/(.+)_(.+)_(.+)-(.+).log$/
        {
            $k8s_pod = $1;
            $k8s_namespace = $2;
            $k8s_container = $3;
            $k8s_container_id = $4;
        }
    </Exec>
</Input>

<Output elasticsearch>
    Module    om_elasticsearch
    URL       http://192.168.1.123:9200/_bulk
    Exec      to_json();
</Output>

The filename-based enrichment is what makes these events usable during an incident: "which pod, in which namespace, on which node" is the difference between a searchable timeline and a pile of anonymous text.

What about containerd and CRI-O?

Kubernetes removed dockershim, its built-in Docker Engine integration, in v1.24. Nodes running containerd or CRI-O don’t produce JSON log files. Instead, the runtime writes plain-text lines in the CRI logging format, which the Kubernetes documentation names as the standardized integration between container runtimes and the kubelet. The kubelet CRI logging design proposal specifies the line layout: an RFC 3339Nano timestamp, the stream name, a P (partial) or F (full) tag, and the message.

The parse_json() call above works on Docker-runtime nodes. For CRI-format lines, swap the JSON parsing for a regular expression. This configuration is adapted from NXLog’s documented Kubernetes setup; validate it against a log sample from your own nodes before rollout:

envvar NODE_NAME

<Input k8s_containers_cri>
    Module    im_file
    File      '/var/log/containers/*.log'
    <Exec>
        # CRI-format line: <timestamp> <stream> <P|F> <message>
        if $raw_event =~ /^(\S+) (stdout|stderr) ([PF]) (.*)$/
        {
            $EventTime = parsedate($1);
            $stream    = $2;
            $cri_tag   = $3;
            $Message   = $4;
        }

        $k8s_node = '%NODE_NAME%';
        $log_file = file_name();
        if $log_file =~ /\/.*\/(.+)_(.+)_(.+)-(.+).log$/
        {
            $k8s_pod = $1;
            $k8s_namespace = $2;
            $k8s_container = $3;
            $k8s_container_id = $4;
        }
    </Exec>
</Input>

<Extension json>
    Module    xm_json
</Extension>

<Output elasticsearch>
    Module    om_elasticsearch
    URL       http://192.168.1.123:9200/_bulk
    Exec      to_json();
</Output>

The JSON extension is back for the output side only. This input parses CRI lines with a regular expression rather than parse_json(), but to_json() still needs the module loaded.

Lines tagged P are fragments of long messages the runtime split across multiple lines. This configuration keeps each fragment as a separate event, which is acceptable for most detection use cases. If your workloads emit long single-line records, plan for reassembly.

Are application logs enough?

No. Application logs tell you what your workloads did. Kubernetes audit logs tell you who did what to the cluster: every API server request, the identity behind it, the resource it touched, and the response it got. Repeated 403 Forbidden responses against sensitive resources are the kind of signal you only see here.

Enable auditing by applying an audit policy to kube-apiserver. Our Kubernetes guide includes a working policy and the API server flags. The audit backend writes JSON records, one per line, which NXLog Agent reads like this:

<Extension json>
    Module    xm_json
</Extension>

<Input k8s_audit>
    Module    im_file
    File      '/var/log/kubernetes/audit.log'
    <Exec>
        parse_json();
        $hostname = hostname();
        $log_type = "k8s_audit";
        to_json();
    </Exec>
</Input>

<Output siem>
    Module        om_http
    URL           http://siem.example.com/
    BufferSize    100000
</Output>

One practical detail from the guide that saves you a support ticket: Kubernetes audit records can be large, and NXLog Agent’s input and output modules have a default buffer of 65,000 bytes. NXLog Agent truncates oversized records, with a log entry like data size (69015) is over the limit (65000), will be truncated. Raise BufferSize on the relevant modules, as shown above.

Round out coverage with the node and control plane: on systemd nodes, the kubelet and container runtime log to journald, while control plane components write under /var/log. Our guide includes a configuration for cluster system logs covering kube-apiserver, kube-scheduler, kubelet, etcd, and the other control plane components.

How do you manage collection across a whole fleet?

A DaemonSet solves one cluster. Most security teams run several clusters, plus Docker hosts, plus the VMs, Windows servers, and network gear that containers didn’t replace. NXLog Platform gives you one place to manage, configure, and monitor your NXLog Agent fleet across all of it, with agent management APIs for automating changes at scale. You can even run NXLog Platform on the cluster itself; there’s a Helm chart for that.

Conclusion

Containers made your infrastructure faster than your logging. The fix is a collection layer with its own lifecycle: an agent on every node, parsing matched to your runtime, and audit and system logs alongside the application streams. The configurations above come from the NXLog Platform documentation, where you’ll find the complete guides. If you’d like to see fleet-wide agent management on your own clusters, you can try NXLog Platform for free. And if you have questions about your setup, reach out through our support portals; we’re happy to help!

FAQ

How do I collect logs from Docker containers?

Either tail the files or push them. Run an agent on the host that reads /var/lib/docker/containers/*/*-json.log, or configure a Docker logging driver such as syslog or gelf to send logs to a listener. NXLog’s Docker guide documents both.

Where does Kubernetes store container logs?

On each node. The kubelet directs the runtime to write logs under /var/log/pods, with symlinks in /var/log/containers. The kubelet rotates them (10 MiB per file and 5 files per container by default), so anything older is gone unless an agent collected it.

Should I use a DaemonSet or a sidecar for Kubernetes logging?

Default to a DaemonSet: one agent per node, full coverage of stdout and stderr, no application changes. Add a sidecar for specific pods whose applications write log files inside the container instead of logging to stdout.

Do container logs survive after a container stops?

Only briefly. On Kubernetes, the kubelet keeps the logs of one terminated container after a restart, and eviction removes containers along with their logs. With Docker’s json-file driver, logs sit in the container’s directory on the host and are subject to the same lifecycle. Treat on-node logs as a cache, not a record.

How do I collect Kubernetes audit logs?

Apply an audit policy and a log backend to kube-apiserver, then read the resulting JSON-lines file with a file input, raising the agent’s buffer size for large records. NXLog’s Kubernetes guide includes a working policy and configuration.

NXLog Platform is an on-premises solution for centralized log management with
versatile processing forming the backbone of security monitoring.

With our industry-leading expertise in log collection and agent management, we comprehensively
address your security log-related tasks, including collection, parsing, processing, enrichment, storage, management, and analytics.

Start free Contact us
Share
Related Posts