Kubernetes is nowadays the de facto standard for the deployment and management of containerized applications. A Kubernetes deployment may contain hundreds, if not thousands, of nodes and pods. As with any other system, collecting logs from your Kubernetes environment is imperative to monitor the health of your cluster and to troubleshoot issues when they arise. In this post we will explore the logging challenges that Kubernetes poses, and how NXLog can be a key player in your logging solution.
Why collect logs from Kubernetes?
A Kubernetes deployment is a highly dynamic environment. Containers can be created, deleted, or rescheduled at any point in time, making the transient nature of containers a challenge to manage in itself. When containers crash or are deleted, the system removes all the data related to that container, including logs that could potentially hold valuable information for troubleshooting.
Kubernetes clusters also consist of multiple components, each creating their own logging in different locations and formats. This logging is extremely important for monitoring and troubleshooting cluster-level problems, however the volume of logging that is generated makes it impossible to manage manually while staying on top of any potential issues.
In view of these challenges, it is clear that you need to have a log collection strategy in place that unifies and aggregates logs to a single repository, be it a SIEM or a log management system.
How does NXLog fit in?
Kubernetes provides a logging framework that captures application logs written to the standard output and standard error streams and writes them to a log file. However, the Kubernetes platform does not have a native, centralized storage solution for aggregating cluster-level logging.
NXLog Enterprise Edition is a versatile and lightweight log collection solution that can ship Kubernetes logs collected from various components to a central location. It provides powerful log filtering and manipulation capabilities that you can use for normalization of log data, improving the quality of logs prior to ingestion, or for reducing log volume.
Additionally, Kubernetes application and system log records do not include data that can be traced back to the originating pod or node, making data enrichment a must if you want to make sense of your logs. NXLog provides the ability to enrich log records with metadata, such as the node name, pod name, and namespace, which will greatly help when analyzing your logs and troubleshooting issues related to specific nodes or pods.
Deploying NXLog in your Kubernetes cluster
There are various ways you can deploy NXLog Enterprise Edition to collect logs from your Kubernetes cluster. It can be installed on a node to collect logs directly from the host, or deployed in an application container to open up further possibilities. NXLog provides a Docker package that can be used to easily build an image and deploy NXLog Enterprise Edition in a container.
We will now look at two different approaches for deploying an NXLog application container to collect logs from your Kubernetes cluster. The two can be combined according to your requirements, to provide an all-encompassing log collection solution.
NXLog as a DaemonSet
The most common method for deploying a log collector in Kubernetes is as a DaemonSet, which ensures that the pod runs on each node in your cluster. This method can be used for collecting application logs streamed to standard output and standard error, as well as to collect Kubernetes system and audit logs.
Kubernetes creates log files in /var/logs by default. When NXLog is
deployed as a DaemonSet, it can be configured to collect logs from this
location, process the log records according to the source, and forward
them to a central repository. Deploying NXLog in this manner means that
with a single configuration, you ensure that you are collecting the majority
of the logs from all your nodes, without any need to modify application
containers.
Interested in seeing a practical example? See how you can deploy NXLog as a DaemonSet in our Kubernetes integration guide.
NXLog as a sidecar
In the event that you need to collect logs from an application that does not stream its logs to standard output and standard error, you can deploy NXLog Enterprise Edition as a sidecar container. When using this method, the log collection container runs in the same pod as the application. After logs are collected and processed, they can be streamed to standard output, which the Kubernetes engine writes to the container log file.
Alternatively, you can configure NXLog to forward the logs directly to a central repository, removing the need for further processing.
See a complete example of how to deploy NXLog as a sidecar in our Kubernetes integration guide.
Conclusion
In this post we have highlighted some of the challenges Kubernetes logging poses, and why implementing a log collection solution is important. The preferred method of collecting logs from Kubernetes is through a DaemonSet, however for applications that do not write logs to the standard output, a log collector can be deployed as a sidecar.
NXLog is a versatile log collector with a small footprint that can meet all your Kubernetes logging requirements. While cloud-based services like AWS and Microsoft Azure may offer log collection tools that function well within their own proprietary ecosystems, NXLog Enterprise Edition offers a vendor-agnostic logging solution that integrates with most third-party platforms. With its flexible, modular design you can continue to use NXLog with only minimal configuration changes should you decide to replace your SIEM or log management system later on.
