News and blog
NXLog main page
  • Products
    NXLog Platform
    Log collection
    Log management and analytics
    Log storage
    NXLog Community Edition
    Integrations
    Professional Services
  • Solutions
    Use cases
    Specific OS support
    SCADA/ICS
    Windows event log
    DNS logging
    MacOS logging
    Solutions by industry
    Financial Services
    Government & Education
    Entertainment & Gambling
    Telecommunications
    Medical & Healthcare
    Military & Defense
    Law Firms & Legal Counsel
    Industrial & Manufacturing
  • Plans
  • Partners
    Find a Reseller
    Partner Program
  • Resources
    Documentation
    Blog
    White papers
    Videos
    Webinars
    Case Studies
    Community Program
    Community Forum
  • About
    Company
    Careers
  • Support
    Support portals
    Contact us

NXLog Platform
Log collection
Log management and analytics
Log storage
NXLog Community Edition
Integrations
Professional Services

Use Cases
Specific OS support
SCADA/ICS
Windows event log
DNS logging
MacOS logging
Solutions by industry
Financial Services
Government & Education
Entertainment & Gambling
Telecommunications
Medical & Healthcare
Military & Defense
Law Firms & Legal Counsel
Industrial & Manufacturing


Find a Reseller
Partner Program

Documentation
Blog
White papers
Videos
Webinars
Case Studies
Community Program
Community Forum

Company
Careers

Support portals
Contact us
Let's Talk Start free
NXLog search
  • Loading...
Let's Talk Start free
February 2, 2022 strategy

Reliable delivery of logs - can you trust TCP?

By Collins Maina

Share
ALL SIEM STRATEGY SECURITY ANNOUNCEMENT DEPLOYMENT COMPLIANCE COMPARISON RSS

When considering your log collection strategy, a decision you have to make is which transport protocol to use to transfer logs from source to destination. The choice is often between the two most commonly used protocols, UDP (User Datagram Protocol) and TCP (Transfer Control Protocol). Which one to use depends on the type of logs you need to transfer, and whether performance or reliability is more important.

This blog post will compare these protocols, discuss why TCP is usually the preferred choice, and provide some options to further increase log delivery reliability with NXLog Enterprise Edition.

Protocol differences - TCP vs. UDP

The main difference between these protocols lies in how data packets move from one point to another. UDP is a connection-less protocol that does not guarantee the delivery of data packets to the destination; therefore, it is considered unreliable. On the other hand, TCP is a connection-oriented protocol designed to ensure the delivery of data packets to the destination, and so it is considered a reliable protocol.

Although UDP is an unreliable protocol, it has its uses when performance is preferred over reliability. UDP reduces the overhead of establishing a network session and can attain faster transmission speeds. Since it does not require receipt verification, it also reduces network load. These factors make it suitable for transferring high-volume, non-critical logs. UDP is usually preferred in monitoring applications to minimize overhead on the system you’re monitoring, or when having the latest logging data is of greater importance than having complete data.

When reliable data transfer is important, TCP is the natural choice since it ensures the delivery of packets to their destination. It is not the intent of this blog post to go into the technical details of how TCP works. However, in a nutshell, it achieves this reliability by performing what is known as a three-way handshake, where the client and server exchange a series of SYN (synchronize) and ACK (acknowledge) packets to initiate communication and establish a reliable connection for the actual data transfer to start. After sending data, the client waits a specific length of time for an acknowledgment. If it does not receive an acknowledgment from the server during that period, it will assume the packet was lost and retransmit it. When considering which protocol to use for log forwarding, TCP reliability is an important benefit to have.

Unreliable aspects of TCP

The original intent of the TCP protocol was to ensure that a connection is established and data is transferred reliably. However, you should be aware of some adverse effects when transferring log data over TCP.

The reliability-first approach of the TCP protocol creates some overhead on the system since it needs to create a connection and maintain its state, deliver packets in the correct order, ensure data integrity, and resend packages if required. Even so, it is still possible to lose data if the server prematurely closes the connection while it is transmitting data. In such cases, unsent data stored in the socket buffers will be lost.

Additionally, even though TCP can detect duplicate packets, there are instances where data duplication is possible. For example, this can happen if the ACK packets are lost and data is retransmitted. In other words, although TCP is designed to guarantee that each packet arrives only once, if the connection breaks, there is no way of knowing what exactly arrived.

In most cases, TCP is suitable for transferring logs because its benefits outweigh the disadvantages. Although TCP cannot guarantee that data loss or duplication will never happen, there are other ways to make your log transfer more reliable, and NXLog can help with this.

Improving log delivery reliability with NXLog

With NXLog, whether you choose to transfer your logs via TCP, UDP, or another protocol altogether, you gain access to functionality that helps you mitigate data loss or log duplication, which includes:

  • Methods to create buffers that allow syncing logs before forwarding them to their destination. The pm_buffer module offers both memory and disk-based buffering. They can even be combined to significantly reduce the risk of data loss due to a system crash or network failure.

  • Log duplication prevention is provided by the duplicate_guard() function. See Protection against duplication in the NXLog User Guide.

  • Addressing unreliability in the TCP protocol by using application-level acknowledgment. Two sets of modules are available for this purpose:

    • The om_http / im_http modules guarantee message delivery over the network by sending logs (om_http) in an HTTP POST request. The server (im_http or a third-party HTTP server) responds with a status code to indicate if the data was received and processed successfully.

    • The om_batchcompress / im_batchcompress modules also acknowledge receipt of data with the added benefit of data compression for faster transfer. In addition, these modules serialize fields so that structured data is preserved for later processing at the destination.

    You can configure both options to transfer data securely using TLS/SSL.

Addressing log delivery reliability is required for compliance standards and certifications such as PCI-DSS, SOX, HIPAA, and ISO27001. With these features, you can tailor your log collection solution to ensure the timely delivery of logs while reducing the risk of data loss during transit. Whatever your requirements, NXLog can help you achieve the right balance between efficiency and reliability that best suits your needs. If you are interested in learning more about implementing these solutions, see Reliable message delivery in the NXLog User Guide.

GET STARTED TODAY:
CONTACT US Our experts are happy to help REQUEST A FREE TRIAL Give NXLog Enterprise Edition a try GET PRICING Request a quote
  • reliability
  • tcp
  • udp
Share

Facebook Twitter LinkedIn Reddit Mail
Related Posts

Using Raijin Database Engine to aggregate and analyze Windows security events
11 minutes | July 29, 2021
File-based logs? Yes, they’re still being used!
3 minutes | August 25, 2021
Sending logs to Microsoft Sentinel with NXLog
10 minutes | February 1, 2021

Stay connected:

Sign up

Keep up to date with our monthly digest of articles.

By clicking singing up, I agree to the use of my personal data in accordance with NXLog Privacy Policy.

Featured posts

Announcing NXLog Platform 1.6
April 22, 2025
Announcing NXLog Platform 1.5
February 27, 2025
Announcing NXLog Platform 1.4
December 20, 2024
NXLog redefines log management for the digital age
December 19, 2024
2024 and NXLog - a review
December 19, 2024
Announcing NXLog Platform 1.3
October 25, 2024
NXLog redefines the market with the launch of NXLog Platform: a new centralized log management solution
September 24, 2024
Welcome to the future of log management with NXLog Platform
August 28, 2024
Announcing NXLog Enterprise Edition 5.11
June 20, 2024
Raijin announces release of version 2.1
May 31, 2024
Ingesting log data from Debian UFW to Loki and Grafana
May 21, 2024
Announcing NXLog Enterprise Edition 6.3
May 13, 2024
Raijin announces release of version 2.0
March 14, 2024
NXLog Enterprise Edition on Submarines
March 11, 2024
The evolution of event logging: from clay tablets to Taylor Swift
February 6, 2024
Migrate to NXLog Enterprise Edition 6 for our best ever log collection experience
February 2, 2024
Raijin announces release of version 1.5
January 26, 2024
2023 and NXLog - a review
December 22, 2023
Announcing NXLog Enterprise Edition 5.10
December 21, 2023
Raijin announces release of version 1.4
December 12, 2023
Announcing NXLog Enterprise Edition 6.2
December 4, 2023
Announcing NXLog Manager 5.7
November 3, 2023
Announcing NXLog Enterprise Edition 6.1
October 20, 2023
Raijin announces release of version 1.3
October 6, 2023
Upgrading from NXLog Enterprise Edition 5 to NXLog Enterprise Edition 6
September 11, 2023
Announcing NXLog Enterprise Edition 6.0
September 11, 2023
The cybersecurity challenges of modern aviation systems
September 8, 2023
Raijin announces release of version 1.2
August 11, 2023
The Sarbanes-Oxley (SOX) Act and security observability
August 9, 2023
Log Management and PCI DSS 4.0 compliance
August 2, 2023
Detect threats using NXLog and Sigma
July 27, 2023
HIPAA compliance logging requirements
July 19, 2023
Announcing NXLog Enterprise Edition 5.9
June 20, 2023
Industrial cybersecurity - The facts
June 8, 2023
Raijin announces release of version 1.1
May 30, 2023
CISO starter pack - Security Policy
May 2, 2023
Announcing NXLog Enterprise Edition 5.8
April 24, 2023
CISO starter pack - Log collection fundamentals
April 3, 2023
Raijin announces release of version 1.0
March 9, 2023
Avoid vendor lock-in and declare SIEM independence
February 13, 2023
Announcing NXLog Enterprise Edition 5.7
January 20, 2023
NXLog - 2022 in review
December 22, 2022
Need to replace syslog-ng? Changing to NXLog is easier than you think
November 23, 2022
The EU's response to cyberwarfare
November 22, 2022
Looking beyond Cybersecurity Awareness Month
November 8, 2022
GDPR compliance and log data
September 23, 2022
NXLog in an industrial control security context
August 10, 2022
Raijin vs Elasticsearch
August 9, 2022
NXLog provides native support for Google Chronicle
May 11, 2022
Aggregating macOS logs for SIEM systems
February 17, 2022
How a centralized log collection tool can help your SIEM solutions
April 1, 2020

Categories

  • SIEM
  • STRATEGY
  • SECURITY
  • ANNOUNCEMENT
  • DEPLOYMENT
  • COMPLIANCE
  • COMPARISON
logo

Subscribe to our newsletter to get the latest updates, news, and products releases. 

© Copyright 2024 NXLog FZE.

Privacy Policy. General Terms of Use

Follow us

  • Product
  • NXLog Platform 
  • Log collection
  • Log management and analysis
  • Log storage
  • Integration
  • Professional Services
  • Plans
  • Resources
  • Documentation
  • Blog
  • White papers
  • Videos
  • Webinars
  • Case studies
  • Community Program
  • Community forum
  • Support
  • Getting started guide
  • Support portals
  • About NXLog
  • About us
  • Careers
  • Find a reseller
  • Partner program
  • Contact us