NXLog main page
  • Products
    LOG COLLECTOR
    NXLog Enterprise Edition
    Full feature multi-platform log collection
    NXLog Community Edition
    Open-source free log collector
    ADD-ONS FOR NXLOG ENTERPRISE EDITION
    NXLog Add-Ons
    Integration with various software
    AGENT MANAGER FOR NXLOG ENTERPRISE EDITION
    NXLog Manager
    Manage and monitor NXLog instances
    NXLog Minder
    Hyper-scalable, API-first agent management
    DATABASE FOR NXLOG ENTERPRISE EDITION
    Raijin Database Engine
    The schemaless SQL database for storing events
    more from nxlog
    Professional Services
    Compare NXLog EE and CE
    NXLog Solution Packs
  • Downloads
    NXLog Enterprise Edition
    Full feature multi-platform log collection
    NXLog Manager
    Manage and monitor NXLog instances
    NXLog Community Edition
    Open-source free log collector
  • Solutions
    Integrations
    With SIEM, Devices, SaaS...
    Specfic OS support
    AIX, Linux, FreeBSD
    SCADA/ICS
    Energy, Oil & Gas, Transport...
    Windows Event log
    Collect locally or remotely, ..
    DNS Logging
    Enterprise-grade DNS log...
    Log Collection Modes
    Agent-based, Agentless or Cloud
    Agent Management
    Agents management and monitoring
    FIM
    File Integrity Monitoring
    macOS Logging
    ULS events, Apple System Logs ...
    By Industry
    Financial Services
    Government & Education
    Entertainment & Gambling
    Telecommunications
    Medical & Healthcare
    Military & Defense
    Law Firms & Legal Counsel
    Industrial & Manufacturing
  • Partners
    Find a Reseller
    Look for our resellers worldwide
    Technology Ecosystem
    See all our partners and integrations
    Partner Program
    Join our community of partners
  • Resources
    Documentation
    Products guides and integrations
    Blog
    Tutorials, updates and releases
    White papers
    Datasheets, infographics and more
    Videos
    Trainings and tutorial on specific topics
    Webinars
    Community events and webinars
    Case Studies
    Customer success stories
    Community Forum →
  • Support
  • Why Nxlog
    About Us
    Our journey, team and mission
    Customers
    Testimonials and case studies
    Careers
    We are hiring!
    Contact Us →
LOG COLLECTOR
NXLog Enterprise Edition
Full feature multi-platform log collection
NXLog Community Edition
Open-source free log collector
ADD-ONS FOR NXLOG ENTERPRISE EDITION
NXLog Add-Ons
Integration with various software
AGENT MANAGER FOR NXLOG ENTERPRISE EDITION
NXLog Manager
Manage and monitor NXLog instances
NXLog Minder
Hyper-scalable, API-first agent management
DATABASE FOR NXLOG ENTERPRISE EDITION
Raijin Database Engine
The schemaless SQL database for storing events
more from nxlog
Professional Services
Compare NXLog EE and CE
NXLog Solution Packs
NXLog Enterprise Edition
Full feature multi-platform log collection
NXLog Manager
Manage and monitor NXLog instances
NXLog Community Edition
Open-source free log collector
Integrations
With SIEM, Devices, SaaS...
Specfic OS support
AIX, Linux, FreeBSD
SCADA/ICS
Energy, Oil & Gas, Transport...
Windows Event log
Collect locally or remotely, ..
DNS Logging
Enterprise-grade DNS log...
Log Collection Modes
Agent-based, Agentless or Cloud
Agent Management
Agents management and monitoring
FIM
File Integrity Monitoring
macOS Logging
ULS events, Apple System Logs ...
By Industry
Financial Services
Government & Education
Entertainment & Gambling
Telecommunications
Medical & Healthcare
Military & Defense
Law Firms & Legal Counsel
Industrial & Manufacturing
Find a Reseller
Look for our resellers worldwide
Technology Ecosystem
See all our partners and integrations
Partner Program
Join our community of partners
Documentation
Products guides and integrations
Blog
Tutorials, updates and releases
White papers
Datasheets, infographics and more
Videos
Trainings and tutorial on specific topics
Webinars
Community events and webinars
Case Studies
Customer success stories
Community Forum →
About Us
Our journey, team and mission
Customers
Testimonials and case studies
Careers
We are hiring!
Contact Us →
Request trial
  • Loading...
Request Trial
July 12, 2023 strategy

Understanding memory usage in NXLog

By Gábor Horváth

Share
ALL SIEM STRATEGY SECURITY ANNOUNCEMENT DEPLOYMENT COMPLIANCE COMPARISON RSS

Understanding how NXLog allocates memory is essential to optimize your configuration for performance and utilize system resources efficiently.

NXLog is designed for high-performance log collection and processing and is optimized to use system resources efficiently. However, various external factors affect how NXLog uses system resources, including memory, which can impact NXLog’s and its host’s performance. Misconfiguration is the leading factor we see when troubleshooting excessive memory consumption. Therefore, in this blog post, we will dive deeper into how NXLog allocates memory to help you create the optimal configuration for your system or determine whether high memory usage results from a misconfiguration.

What drives memory consumption up?

A passive NXLog instance uses between 10-20 MB of RAM. However, in reality, the memory it requires depends on the modules in use and the configuration’s complexity. Mainly, two configuration settings determine memory allocation, the LogqueueSize and BufferSize.

LogqueueSize

The log queue size has four main variables:

  1. StringLimit sets the maximum size of an event. It defaults to 5 MB.

  2. BatchSize sets the maximum number of events in a batch. The default is 50 events.

  3. LogqueueSize sets the maximum number of batches NXLog will keep in the queue for each output module. The default is 100 batches.

  4. The number of output modules.

BufferSize

Input and output modules allocate a buffer per input stream. For example, network input modules, such as im_http, im_ssl, and im_tcp, create a buffer per client connection. File-based inputs, such as im_file, create a buffer for each open file up to the maximum number of active files. The default buffer size is 65,000 bytes.

So, in essence, you can calculate your theoretical maximum memory allocation as follows:

\(StringLimit ∗ BatchSize ∗ LogqueueSize ∗ (NumberOfOutputModules+NumberOfProcessorModules)\)

For example, for a default configuration with one output instance:

\(5242880 * 50 * 100 * 1 = 26214400000B = 24.41GB\)

This calculation basically means:

\(BufferSize * (NumberOfStreams)\)

Which, when translated into numbers, comes down to:

\(65000 * (1+1) = 130000B\)

And taking into account the base NXLog footprint:

\(BaseNXLogFootprint=~20MB\)

The above calculation represents the worst-case scenario for a minimal configuration. Keep in mind that to see this manifest in real life, the following conditions must occur:

  • A full log queue, i.e., a blocked output instance

  • Events equal to the size limit (5 MB by default)

An unobstructed log queue will use little memory. And fortunately, events are usually much smaller and range between 120 bytes and a few hundred kilobytes. So the calculation with 200 KB results in only about 1 GB.

What happens in case of memory overuse?

The OS will intervene in some cases. For example, the OoM (Out of Memory) killer will destroy the offending process on Linux. If this happens, NXLog will lose any data in the log queue.

When is this likely to happen?

NXLog allows the log queue to grow until it reaches the LogqueueSize limit if an output instance is blocked. However, if the limit is set too high, it might stress the system’s memory management to the point where an OoM event is triggered.

How can you distinguish between a memory leak and normal growth?

Simultaneously observing the log queue size and memory usage is essential to inspect a potential memory leak. Therefore, the first step is to check the settings and state of the log queue. It will be very hard to spot a leak if you cannot control the LogqueueSize and its inherent variability.

You may observe a monotonic upward trend in memory use, continuing until it plateaus at a maximum value while the log queue fills up. In addition, an OoM event may occur at some point, terminating the NXLog process. However, this is not proof of a memory leak but merely an over-allocation.

Once the log queue is full, memory consumption should stabilize with only minor fluctuations. There may be slight variations during this period, but the overall memory consumption should remain steady. If NXLog never reaches equilibrium, there is a legitimate reason to suspect a memory leak.

NXLog memory usage reaches equilibrium once the log queue is full

memory usage 1

If log processing is unobstructed, your system should reach a steady near-equilibrium state, primarily dependent on the configuration and the event flow. Therefore, you should not see a continuous increase in memory usage.

Memory usage in an unobstructed event flow

memory usage 2

On the other hand, if memory usage grows despite the event flow not being obstructed, it indicates a potential memory leak.

Memory usage grows despite an empty queue and a steady event flow

memory usage 3

If the memory usage balloons because of an obstructed log queue, it may not be immediately released once it stabilizes.

Memory reallocation after the log queue is emptied

memory usage 4

Special cases

om_kafka

om_kafka uses the librdkafka library, which uses a separate message queue. That is an additional memory overhead that you must consider when the module cannot deliver data to the broker.

pm_buffer

pm_buffer supports memory-based buffering. If you enable this, the module uses additional memory apart from the log queue.

Conclusion

In conclusion, there are four factors you must keep in mind when optimizing or troubleshooting NXLog memory consumption:

  • Understand the mechanics - log queue, batch, and event size.

  • Understand the configuration - which modules you’re using, how many output and processor modules you have, and what is the log queue size limit.

  • Understand the events - event size and EPS profile.

  • Understand what to monitor - track memory usage in conjunction with EPS and observe whether there is a monotonic upward trend.

For more information on NXLog’s memory usage, see Using Buffers in the NXLog User Guide.

GET STARTED TODAY:
CONTACT US Our experts are happy to help REQUEST A FREE TRIAL Give NXLog Enterprise Edition a try GET PRICING Request a quote

NXLog Ltd. develops multi-platform log collection tools that support many different log sources, formats, transports, and integrations. The tools help administrators collect, parse, and forward logs so they can more easily respond to security issues, investigate operational problems, and analyze event data. NXLog distributes the free and open source NXLog Community Edition and offers additional features and support with the NXLog Enterprise Edition.

This document is provided for informational purposes only and is subject to change without notice. Trademarks are the properties of their respective owners.

  • nxlog configuration
  • memory management
Share

Facebook Twitter LinkedIn Reddit Mail
Related Posts

Need to replace syslog-ng? Changing to NXLog is easier than you think
9 minutes | November 23, 2022
Putting together your first NXLog configuration
4 minutes | September 25, 2021

Stay connected:

Sign up

Keep up to date with our monthly digest of articles.

By clicking singing up, I agree to the use of my personal data in accordance with NXLog Privacy Policy.

Featured posts

Upgrading from NXLog Enterprise Edition 5 to NXLog Enterprise Edition 6
September 11, 2023
Announcing NXLog Enterprise Edition 6.0
September 11, 2023
The cybersecurity challenges of modern aviation systems
September 8, 2023
Raijin announces release of version 1.2
August 11, 2023
The Sarbanes-Oxley (SOX) Act and security observability
August 9, 2023
Log Management and PCI DSS 4.0 compliance
August 2, 2023
Detect threats using NXLog and Sigma
July 27, 2023
HIPAA compliance logging requirements
July 19, 2023
Announcing NXLog Enterprise Edition 5.9
June 20, 2023
Industrial cybersecurity - The facts
June 8, 2023
Raijin announces release of version 1.1
May 30, 2023
CISO starter pack - Security Policy
May 2, 2023
Announcing NXLog Enterprise Edition 5.8
April 24, 2023
CISO starter pack - Log collection fundamentals
April 3, 2023
Raijin announces release of version 1.0
March 9, 2023
Avoid vendor lock-in and declare SIEM independence
February 13, 2023
Announcing NXLog Enterprise Edition 5.7
January 20, 2023
NXLog - 2022 in review
December 22, 2022
Need to replace syslog-ng? Changing to NXLog is easier than you think
November 23, 2022
The EU's response to cyberwarfare
November 22, 2022
Looking beyond Cybersecurity Awareness Month
November 8, 2022
GDPR compliance and log data
September 23, 2022
NXLog in an industrial control security context
August 10, 2022
Raijin vs Elasticsearch
August 9, 2022
NXLog provides native support for Google Chronicle
May 11, 2022
Aggregating macOS logs for SIEM systems
February 17, 2022
How a centralized log collection tool can help your SIEM solutions
April 1, 2020

Categories

  • SIEM
  • STRATEGY
  • SECURITY
  • ANNOUNCEMENT
  • DEPLOYMENT
  • COMPLIANCE
  • COMPARISON
logo

Subscribe to our newsletter to get the latest updates, news, and products releases.

© Copyright 2023 NXLog Ltd.

PRIVACY POLICY TERMS OF USE

  • PRODUCTS
  • NXLOG ENTERPRISE EDITION
  • NXLOG COMMUNITY EDITION
  • NXLOG ADD-ONS
  • NXLOG MANAGER
  • NXLOG MINDER
  • RAIJIN DATABASE
  • MORE NXLOG
  • COMPARE SOLUTIONS
  • INDUSTRIES
  • INTERGRATIONS
  • FIND A RESELLER
  • PARTNER PROGRAM
  • RESOURCES
  • DOCUMENTATION
  • WHITE PAPERS
  • WEBINARS
  • CASE STUDIES
  • TUTORIALS
  • BLOG
  • COMMUNITY FORUM
  • ABOUT US
  • WHY NXLOG
  • CUSTOMERS
  • CAREERS
  • CONTACT US
  • DOWNLOADS
  • NXLOG ENTERPRISE EDITION
  • NXLOG COMMUNITY EDITION
  • NXLOG MINDER
  • NXLOG MANAGER
  • NXLOG ADD-ONS
  • RAIJIN DATABASE