• Products
    LOG COLLECTOR
    NXLog Enterprise Edition
    Full feature multi-platform log collection
    NXLog Community Edition
    Open-source free log collector
    ADD-ONS FOR NXLOG ENTERPRISE EDITION
    NXLog Add-Ons
    Integration with various software
    AGENT MANAGER FOR NXLOG ENTERPRISE EDITION
    NXLog Manager
    Manage and monitor NXLog instances
    NXLog Minder
    Hyper-scalable, API-first agent management
    DATABASE FOR NXLOG ENTERPRISE EDITION
    Raijin Database Engine
    The schemaless SQL database for storing events
    more from nxlog
    Professional Services
    Compare NXLog EE and CE
  • Downloads
    NXLog Enterprise Edition
    Full feature multi-platform log collection
    NXLog Manager
    Manage and monitor NXLog instances
    NXLog Community Edition
    Open-source free log collector
  • Solutions
    Integrations
    With SIEM, Devices, SaaS...
    Specfic OS support
    AIX, Linux, FreeBSD
    SCADA/ICS
    Energy, Oil & Gas, Transport...
    Windows Event log
    Collect locally or remotely, ..
    DNS Logging
    Enterprise-grade DNS log...
    Log Collection Modes
    Agent-based, Agentless or Cloud
    Agent Management
    Agents management and monitoring
    FIM
    File Integrity Monitoring
    macOS Logging
    ULS events, Apple System Logs ...

    By Industry

    Financial Services
    Government & Education
    Entertainment & Gambling
    Telecommunications
    Medical & Healthcare
    Military & Defense
    Law Firms & Legal Counsel
    Industrial & Manufacturing
  • Partners
    Find a Reseller
    Look for our resellers worldwide
    Technology Ecosystem
    See all our partners and integrations
    Partner Program
    Join our community of partners
    Partner Portal →
  • Resources
    Documentation
    Products guides and integrations
    Blog
    Tutorials, updates and releases
    White papers
    Datasheets, infographics and more
    Videos
    Trainings and tutorial on specific topics
    Webinars
    Community events and webinars
    Community Forum →
  • Support
  • Why Nxlog
    About Us
    Our journey, team and mission
    Customers
    Testimonials and case studies
    Careers
    We are hiring!
    Contact Us →
Log In Sign Up
Request Trial
LOG COLLECTOR
NXLog Enterprise Edition
Full feature multi-platform log collection
NXLog Community Edition
Open-source free log collector
ADD-ONS FOR NXLOG ENTERPRISE EDITION
NXLog Add-Ons
Integration with various software
AGENT MANAGER FOR NXLOG ENTERPRISE EDITION
NXLog Manager
Manage and monitor NXLog instances
NXLog Minder
Hyper-scalable, API-first agent management
DATABASE FOR NXLOG ENTERPRISE EDITION
Raijin Database Engine
The schemaless SQL database for storing events
more from nxlog
Professional Services
Compare NXLog EE and CE
NXLog Enterprise Edition
Full feature multi-platform log collection
NXLog Manager
Manage and monitor NXLog instances
NXLog Community Edition
Open-source free log collector
Integrations
With SIEM, Devices, SaaS...
Specfic OS support
AIX, Linux, FreeBSD
SCADA/ICS
Energy, Oil & Gas, Transport...
Windows Event log
Collect locally or remotely, ..
DNS Logging
Enterprise-grade DNS log...
Log Collection Modes
Agent-based, Agentless or Cloud
Agent Management
Agents management and monitoring
FIM
File Integrity Monitoring
macOS Logging
ULS events, Apple System Logs ...

By Industry

Financial Services
Government & Education
Entertainment & Gambling
Telecommunications
Medical & Healthcare
Military & Defense
Law Firms & Legal Counsel
Industrial & Manufacturing
Find a Reseller
Look for our resellers worldwide
Technology Ecosystem
See all our partners and integrations
Partner Program
Join our community of partners
Partner Portal →
Documentation
Products guides and integrations
Blog
Tutorials, updates and releases
White papers
Datasheets, infographics and more
Videos
Trainings and tutorial on specific topics
Webinars
Community events and webinars
Community Forum →
Support
About Us
Our journey, team and mission
Customers
Testimonials and case studies
Careers
We are hiring!
Contact Us →
  • Loading...
Request Trial
August 25, 2021 log filelog collectionsecurity

File-based logs? Yes, they’re still being used!

By Tamás Burtics

Share
ALL SIEM STRATEGY SECURITY ANNOUNCEMENT DEPLOYMENT COMPLIANCE COMPARISON RSS

File-based logs are where it all began. These logs can yield information of great value to security analysts and administrators alike. Armed with this information, IT professionals are better equipped to troubleshoot issues, evaluate system performance, identify bugs, and even detect security breaches.

In today’s world, we tend to focus on the modern, integrated logging facilities like Microsoft’s Windows Event Log or Apple’s Unified Logging System (ULS). However, all the major operating systems still generate log files that may or may not be integrated into these logging facilities. File-based logs, it seems, are still alive and well, probably far into the foreseeable future.

Whether it be on Windows platforms that log high-frequency events in the form of ETL binary files, or Linux systems that store log files in the /var/log directory, organizations rely even more so today on this conventional logging method for capturing information about unique OS operations.

Beyond the IT departments of large corporations, manufacturing plants also use log files extensively. In these industries, log files are the primary data source for supervisory control and data acquisition systems (SCADA), which monitor and control industrial processes while gathering real-time data.

Excerpt from a log file from a SCADA system
2020-05-12 12:59:38.103        +03:00        *************************************
2020-05-12 12:59:38.112        +03:00        *** Citect process is starting up ***
2020-05-12 12:59:38.135        +03:00        Running in 32 bit process architecture
2020-05-12 12:59:38.136        +03:00        Running in Console Mode
2020-05-12 12:59:38.164        +03:00        Data Execution Prevention (DEP), has been enabled successfully

Gleaning useful information from log files may sound easy, but there are some challenges:

Log files often lack structure and standardization. This is mainly due to proprietary platforms or applications whose vendors did not adopt any standard for their log file format. Having to read and translate various incompatible log formats can hinder the aggregation of logged data and hence, the ability to detect security breaches.

Another consideration is the "signal to noise ratio" of log files. Although log files can contain extremely valuable data, these gems are often buried under an enormous heap of rubble.

Therefore, today’s businesses need a solution that can effectively meet these challenges that log files present. This solution should be capable enough—​and powerful enough—​to read any file format, any data structure, on any platform, aggregate the data, and forward it to your choice of SIEM or Analytics platform. It should also provide a flexible, powerful facility for filtering messages so that each organization can get what it needs.

NXLog can collect your log files from any source and route them to literally any destination simply by using its dedicated im_file module and specifying the path to the log files and their destination.

Due to business requirements, your log data’s cumulative storage needs may have to be kept within prescribed storage limits. With NXlog, you can easily filter your logs based on field values so that any event matching certain criteria is dropped. You may also decide to trim your logs which allows you to choose a list of fields to discard or specify a list of fields to keep, thus reducing the overall size of log files. Another approach NXLog can use to mitigate file size is log file compression for situations in which log records cannot be dropped or altered due to policy restrictions.

Regardless of your organization’s choice of software solutions and their underlying platforms, you can add value to your current log collection strategy by complementing it with file-based logs. All you need is the right log collection tool that can read any file, in any format, from any location. With such a flexible solution, you can rest assured, you will reap the additional benefits of processing file-based logs while forgetting all of the challenges they normally present.

GET STARTED TODAY:
CONTACT US Our experts are happy to help REQUEST A FREE TRIAL Give NXLog Enterprise Edition a try GET PRICING Request a quote

NXLog Ltd. develops multi-platform log collection tools that support many different log sources, formats, transports, and integrations. The tools help administrators collect, parse, and forward logs so they can more easily respond to security issues, investigate operational problems, and analyze event data. NXLog distributes the free and open source NXLog Community Edition and offers additional features and support with the NXLog Enterprise Edition.

This document is provided for informational purposes only and is subject to change without notice. Trademarks are the properties of their respective owners.

  • log file
  • log collection
Share

Facebook Twitter LinkedIn Reddit Mail
Related Posts

Making the most of Windows Event Forwarding for centralized log collection
7 minutes | December 17, 2018
DNS Log Collection - Part 2
9 minutes | May 28, 2020
Log Collection Glossary 2020
8 minutes | April 2, 2020

Stay connected:

Sign up

Keep up to date with our weekly digest of articles.

By clicking singing up, I agree to the use of my personal data in accordance with NXLog Privacy Policy.

Featured posts

Announcing NXLog Enterprise Edition 5.7
January 20, 2023
NXLog - 2022 in review
December 22, 2022
Need to replace syslog-ng? Changing to NXLog is easier than you think
November 23, 2022
The EU's response to cyberwarfare
November 22, 2022
Looking beyond Cybersecurity Awareness Month
November 8, 2022
GDPR compliance and log data
September 23, 2022
NXLog in an industrial control security context
August 10, 2022
Raijin vs Elasticsearch
August 9, 2022
NXLog provides native support for Google Chronicle
May 11, 2022
Aggregating macOS logs for SIEM systems
February 17, 2022
How a centralized log collection tool can help your SIEM solutions
April 1, 2020

Categories

  • SIEM
  • STRATEGY
  • SECURITY
  • ANNOUNCEMENT
  • DEPLOYMENT
  • COMPLIANCE
  • COMPARISON

Subscribe to our newsletter to get the latest updates, news, and products releases.

© Copyright 2023 NXLog Ltd.

PRIVACY POLICY TERMS OF USE

  • PRODUCTS

  • NXLOG ENTERPRISE EDITION
  • NXLOG COMMUNITY EDITION
  • NXLOG ADD-ONS
  • NXLOG MANAGER
  • NXLOG MINDER
  • RAIJIN DATABASE
  • MORE NXLOG

  • COMPARE SOLUTIONS
  • INDUSTRIES
  • INTERGRATIONS
  • FIND A RESELLER
  • PARTNER PROGRAM
  • RESOURCES

  • DOCUMENTATION
  • WHITE PAPERS
  • WEBINARS
  • TUTORIALS
  • BLOG
  • COMMUNITY FORUM
  • ABOUT US

  • WHY NXLOG
  • CUSTOMERS
  • CAREERS
  • CONTACT US
  • DOWNLOADS

  • NXLOG ENTERPRISE EDITION
  • NXLOG COMMUNITY EDITION
  • NXLOG MINDER
  • NXLOG MANAGER
  • NXLOG ADD-ONS
  • RAIJIN DATABASE