Optimize log management and cut costs

The process of security logging optimization is an ongoing effort within organizations as data reliability, performance, scaling and TCO associated issues continuously emerge. While these challenges can stem from a number of different sources, it is possible to solve them with a single solution: a focused log collection strategy. This white paper aims to help you develop such a strategy that ensures reliable log analytics and optimized performance while reducing the overall logging stack costs.

The challenges of logging

Collecting everything is a common mistake in a logging environment.

Although many regulations require a full copy of logs (for forensics purposes) a compliant SIEM operation can be conducted without it. One common fallacy is that by gathering and analyzing as many log messages as possible, there will be a better chance of understanding what is actually happening within the IT infrastructure, and that security personnel can react accordingly. In reality, this is as far from the truth as possible, and here is why.

Within an organization a SOC / SIEM receives on average, 5,000 to 10,000 alerts a day, yet a single security analyst can only validate around 100 per day. The main issue usually boils down to being overwhelmed with too many false positives or redundant (repetitively broadcast) alerts. Among the 10,000 events per day, only about 20-25 are actual threats. While this is a troubling number, what is much more troubling is the fact that they may get lost among all these other alerts.

Moreover, the sheer volume of collected logs introduces substantial challenges in terms of data storage, retention, and maintenance. The complexity and expense of managing an end-to-end logging infrastructure can quickly escalate, making it crucial for businesses to find efficient solutions.

How to develop a log collection strategy

The Open Web Application Security Project (OWASP) proposes the following logging best practices:

Context	Auditable events should be logged with sufficient context.
Retention	Events should be retained for long enough that delayed forensic analysis can be performed when necessary.
Format	Log data should be generated in a format suitable for centralized log management.
Integrity	High-value transactions should have an audit trail with controls to prevent tampering.
Monitoring	Suspicious activities should be detected and responded to promptly.
Compliance	All major regulatory standards and frameworks, including PCI DSS, HIPAA, ISO 27001, SOX, NERC and NIST CSF mandate establishing centralized logging in order to comply.

Taking these best practices into account, the following key aspects should guide your log collection strategy:

Clearly define your log collection use cases	Avoid collecting logs from all sources and indiscriminately forwarding them to a centralized server for processing. A good example of how to mitigate this would be to differentiate between critical and non-critical logs. In general audit logs hold far more security value than operation logs. Consider using a SIEM for higher value audit logs in tandem with a log management server for lower value events like operation logs.
Identify your log rollover and archiving approach	Define which logs can be discarded, which logs can be archived, and for how long they need to be retained depending on factors like legal agreements, local regulations, and whether or not they contain personally identifiable information (PII).
Consider adopting a multi-layer log management approach	This allows you to store low priority logs using more cost-effective solutions (since many solutions charge based on the retention period). Logs of higher importance can be sent directly to the SIEM.
Understand the difference between high quality log collection and log availability	Logs that need to be available may be scattered in your infrastructure, like in cold storage. A very specific query might take hours to complete due to the sheer volume of relevant logs available. If there is a problem with low quality logs, establishing centralized log collection can help.
Invest in a centralized log collection solution to further your strategic efforts	Such solutions offer a wide range of functionality that can simplify SIEM operations and make analytics more efficient, resulting in reduced overall SIEM costs.

Implementing the best practices with centralized log collection

Centralized log collection, log aggregation, or log centralization is the process of sending event log data to a dedicated server or service for storage and optionally search and analytics. Storing logs on a centralized system provides several benefits versus storing the data locally:

Event data can be accessed even if the originating server is offline, compromised, or decommissioned.
Data can be analyzed and correlated across more than one system.
It is more difficult for malicious actors to remove evidence from logs that have already been forwarded.
Incident investigation and auditing is easier, as all event data is collected in one location.
Scalable, high-availability, and redundancy solutions are easier to implement and maintain because they can be implemented at the point of the collection server.
Compliance with internal and external standards for log data retention only need to be managed at a single point.

Besides these direct benefits of centralized log collection, you can also leverage the following optimizations:

Reducing the number of logs achieved through log event filtering, selective event collection, classification, correlation, and removing duplicate logs.
Reducing the size of the log data by parsing out fields containing the same content or fields that are not essential for the SIEM, and by truncating long fields if the SIEM needs only a subset of the data.
Providing data compression via network and batch compression.
Providing a lightweight architecture to reduce memory footprint and increased flexibility for different uses.
Offering better storage space management by forwarding logs to multiple endpoints combined with resource splitting to manage which logs should actually go to the SIEM.
Offering better resource efficiency by managing log enrichment and log conversion to determine how and when they should be processed during the log collection cycle.
Cutting back on SIEM costs by significantly reducing the event per second (EPS) and Gigabytes per day (GB/day) log volume.

Many of our customers turn to us to meet their SIEM optimization and cost saving challenges by using NXLog Platform as their centralized log collection solution.

Reducing SIEM licensing costs with NXLog Platform

Lowering the total cost of ownership of your SIEM infrastructure is an added benefit when you use NXLog Platform to optimize log collection specifically for your use cases

NXLog Platform is a complete log collection and log management solution that includes out-of-the-box storage and analytics solutions, eliminating the need for extra third-party products with the added burden of integrations and ongoing maintenance.
NXLog Platform offers over 100 popular integrations with log sources and target systems out-of-the-box, ensuring a fast and streamlined deployment of the solution, thus helping save on integration efforts.
NXLog Platform provides better SIEM optimization by simplifying the log data and reducing its size while retaining important metadata. The end result is faster search times.
NXLog Platform is a vendor neutral solution that allows you to connect and combine solutions that best fit your needs. It can be deployed across a wide variety of nodes and destinations, such as by forwarding logs to third-party SIEMs or by leveraging the built-in log management capabilities. Besides this, NXLog Platform increases confidence in SIEM analysis by making sure that captured logs remain tamperproof.

The following diagram provides a non-exhaustive overview of the possible integrations and log management use cases that NXLog Platform supports out of the box:

Generally, SIEMs on the market either follow an Event per second (EPS) or Gigabytes per day (GB/day) pricing structure. In both cases the core principle is the same: the more recklessly you collect and analyze data the more expensive it becomes.

The table below summarizes how SIEM licensing costs are determined vs the role NXLog Platform plays in making a difference to reduce such costs.

Table 1. SIEM licenses and the NXLog Platform difference
Company	Licensing issue	What can NXLog Platform do
Elasticsearch	Elastic is known to be resource-demanding (CPU/RAM/Storage, mind hardware costs). Its price depends on cloud resource consumption and on number of cluster nodes used (actually, on-prem price is not public at all). Also Elastic may require you to manually manage various agents to collect logs and that will bring additional pressure on the budget.	NXLog Platform bundles the most versatile agent on the market that substitutes all possible Elastic shippers (like *beats, Logstash and others). It allows building a unified log collection pipeline across all your infrastructure. With the NXLog Platform agent you can filter out noise from the logs and reduce storage and computing demands for Elasticsearch, thus lowering the price. NXLog Platform also provides a central management UI for agents to simplify operations and reduce maintenance costs.
Google Chronicle	The pricing for Google Chronicle is not public, but the product does not come with any native log collection tools. This means that you have to acquire a third-party solution to accomplish the task.	Google recommends NXLog for the log shipping purpose, especially for Windows Events, Active Directory, and DNS logs.
MicroFocus ArcSight	ArcSight Logger and ArcSight Data Platform (ADP) appliances and software are licensed based on GB per day, data ingested, and security events correlated per second. See ArcSight license and enforcement information.	No licensing limitations or subscriptions involving log data usage. Used to pre-process logs (filtering, deduplicating metadata, dropping fields) prior to forwarding to ArcSight.
Microsoft Sentinel (Azure Sentinel)	With Sentinel’s pay-as-you-go pricing, you are billed for the volume of data in gigabytes (GB) that is ingested for security analysis by Microsoft Sentinel and stored in the Azure Monitor Log Analytics workspace. Besides this, native Azure collectors have limited integration and filtering capabilities, so you may end up billed for a huge volume of data stored.	NXLog Platform takes over the log collection task, allowing you to reduce data volume ingested to Sentinel by filtering out the noise and thus lowering your costs.
LogRhythm	LogRhythm has an unlimited data plan.	Conduct agent-side filtering to reduce EPS. The compression modules reduce bandwidth needed to send logs from remote sites.
Splunk	Annual subscription pricing is based on GB/day. See calculator. Splunk free licenses have limitations stipulated in the end user agreement.	No endpoint restrictions or limitations. Modules are available to help decrease log data size.

Examples of NXLog Platform optimizations

The following sections describe specific NXLog Platform features that work together to reduce data storage and licensing costs, as well as a use case demonstrating the results of such optimizations:

Reducing the number of logs

Filter log events: Depending on the logging requirements and the log source, it is possible to simply discard certain events. NXLog Platform can filter events based on nearly any set of criteria. Read more on how to filter logs.

Typical example where dropping certain logs with a duplicated error message is needed

2018-10-26 09:48:26 ERROR Service is already running
2018-10-26 10:10:44 ERROR Service is already running
2018-10-26 13:22:27 ERROR Service is already running
2018-10-26 16:35:07 ERROR Service is already running
2018-10-26 16:36:45 WARNING stopping nxlog service

Selective event collection: NXLog Platform can collect certain Windows Event IDs and drop the rest. Important security events such as lateral movements can be configured instead of collecting all event logs. Read more in the Event IDs to Monitor section.

Using structured logging

Structured log collection: When logs are more human-readable, and security experts are spending less time having to look for a "needle in a haystack," it can dramatically reduce the operational cost of the SIEM. NXLog Platform provides structured log collection which allows events to be classified and correlated using any of the values provided by the event set. Read more in the Structured data section.

Raw log sample, difficult to correlate:

<38>Nov 22 10:30:12 myhost sshd[8459]: Failed password for invalid user linda from 192.168.1.60 port 38176 ssh2

An excerpt of the same event as structured data, easier to correlate

{  "SyslogFacility": "USER",  "SyslogSeverity": "NOTICE",  "EventTime": "2019-11-22 10:30:12",  "Hostname": "myhost",  "SourceName": "sshd",  "ProcessID": 8459,  "Status": "failed",  "AuthenticationMethod": "password",  "Reason": "invalid user",  "User": "linda",  "SourceIPAddress": "192.168.1.60",  "SourcePort": 38176,  "Protocol": "ssh2"}

Reducing the size of log data

Trimming log events: NXLog Platform can trim events by removing parts of redundant text from the event record or state which fields to discard. This helps reduce log data bloat. Read more on how to trim events.

Exec example of truncating a Windows Event Log event:

if ($Channel == 'Security') and ($EventID == 4688)
$Message =~ s/\s*Token Elevation Type indicates the type of .*$//s;
else if $(Channel == 'Security') and ($EventID == 4769)
$Message =~ s/\s*This event is generated every time access is .*$//s;

Deduplicate metadata: Some log data will have duplicated metadata across multiple events of the same time. One example of this is the "descriptive event data" in Windows Event Log. By removing this verbose text from common events, event sizes can be reduced significantly while still preserving the important metadata.

Using data compression over the network

SSL and HTTPS compression modules: NXLog Platform allows you to enable data compression when sending data over the network.

Batch compression: NXLog Platform can transfer events in compressed and encrypted batches to save bandwidth. This is especially useful for transferring logs in low-bandwidth environments or on networks with suboptimal performance. Read more about compressing log data during transport.

Using lightweight deployment solutions

Lightweight architecture: The NXLog Platform agent provides a reduced memory footprint and increased flexibility for different uses. Besides this, you can also consider deploying an agent-less or mixed approach. Keep in mind that only agent-based solutions can provide advanced features like (most of the) trimming, filtering, and compressing. Use the same agent for your configurations instead of keeping track of multiple agents. Read more on our blog post Agent-based versus agent-less log collection - which option is best?

Simple agent fleet management: With NXLog Platform, you only need one agent software to handle all log collection, filtering, and processing tasks. This reduces the budgets and human resources required for integration and maintenance. Efficient agent fleet management capabilities further reduce costs associated with configuring and monitoring a large number of agents across your network and infrastructure.

Saving storage

Log forwarding: NXLog Platform allows you to forward logs from a single host to multiple endpoints. This mitigates the risk of log tampering or removal following an attack.

Resource splitting: NXLog Platform can centralize, store, and manage logs, while forwarding relevant logs to the SIEM.

On-disk compression: Using external programs or scripts, NXLog Platform can automate the compression of old log files to reduce disk usage.

Deciding when, how, and where log enrichment takes place

Log enrichment: With NXLog Platform you can decide how and at what point log enrichment happens during the log collection cycle. For example, determine which logs should be enriched on the agent-side at the point of forwarding or at the point of writing to log storage.

Resource efficiency: Converting certain data formats from one to another can be resource intensive. NXLog Platform forwards and converts log data on the fly, without any need to write the converted data to local disk before forwarding, thus maintaining high efficiency and conserving resources.

Example use case

The following use case is an example of how NXLog Platform can optimize the log data and achieve significant savings.

Use Case: Rapid7 InsightIDR SIEM

Windows account login attempts generate events containing copious metadata. See below for an authentication event in Snare Syslog format.

Full event sample of a Windows Failed Authentication Event in Syslog

01 Aug 2019 17:46:45.291{
  "timestamp": "2019-08-01T21:46:43.000Z",
  "hostname": "NXLOG-AGENT",
  "event_code": "4625",
  "description": "An account failed to log on.",
  "subject_user_sid": "S-1-0-0",
  "subject_user_name": "-",
  "subject_domain_name": "-",
  "subject_logon_id": "0x0",
  "logon_type": "Network",
  "target_user_sid": "S-1-0-0",
  "target_user_name": "ADMINISTRATOR",
  "target_domain_name": "",
  "failure_reason": "Unknown user name or bad password.",
  "status": "username or password incorrect",
  "sub_status": "user name is correct but the password is wrong",
  "process_id": "0x0",
  "process_name": "-",
  "workstation_name": "-",
  "ip_address": "212.92.116.56",
  "ip_port": "0",
  "logon_process_name": "NtLmSsp",
  "authentication_package_name": "NTLM",
  "transmitted_services": "-",
  "lm_package_name": "-",
  "key_length": "0",
  "source_data": "<11>Aug  1 17:46:43 NXLOG-AGENT MSWinEventLog\t3\tSecurity\t77\tThu Aug 01 17:46:43 2019\t4625\tMicrosoft-Windows-Security-Auditing\tN/A\tN/A\tFailure Audit\tNXLOG-AGENT\tLogon\t\tAn account failed to log on.    Subject:   Security ID:  S-1-0-0   Account Name:  -   Account Domain:  -   Logon ID:  0x0    Logon Type:   3    Account For Which Logon Failed:   Security ID:  S-1-0-0   Account Name:  ADMINISTRATOR   Account Domain:      Failure Information:   Failure Reason:  Unknown user name or bad password.   Status:   0xC000006D   Sub Status:  0xC000006A    Process Information:   Caller Process ID: 0x0   Caller Process Name: -    Network Information:   Workstation Name: -   Source Network Address: 212.92.116.56   Source Port:  0    Detailed Authentication Information:   Logon Process:  NtLmSsp    Authentication Package: NTLM   Transited Services: -   Package Name (NTLM only): -   Key Length:  0    This event is generated when a logon request fails. It is generated on the computer where access was attempted.    The Subject fields indicate the account on the local system which requested the logon. This is most commonly a service such as the Server service, or a local process such as Winlogon.exe or Services.exe.    The Logon Type field indicates the kind of logon that was requested. The most common types are 2 (interactive) and 3 (network).    The Process Information fields indicate which account and process on the system requested the logon.    The Network Information fields indicate where a remote logon request originated. Workstation name is not always available and may be left blank in some cases.    The authentication information fields provide detailed information about this specific logon request.   - Transited services indicate which intermediate services have participated in this logon request.   - Package name indicates which sub-protocol was used among the NTLM protocols.   - Key length indicates the length of the generated session key. This will be 0 if no session key was requested.\t223050"
}

However, only important ingress authentication details are needed. The NXLog Platform xm_rewrite module, xm_json JSON module, and drop_fields(); procedure can decrease the data size. They will also structure the data, leaving only the important event details required by the Rapid7 Insight IDR UEF.

Windows Failed Authentication Event after log enrichment

{"timestamp":"2019-08-01T20:33:41.000Z","user":"NXLOG-AGENT","account":"NXLOG-AGENT","result":"FAILED_OTHER","source_ip":"212.92.117.25","service":"CUSTOM UNIVERSAL EVENT","geoip_organization":"NFOrce Entertainment B.V.","geoip_country_code":"NL","geoip_country_name":"Netherlands","geoip_city":"","geoip_region":"","authentication_target":"-","source_json":{"time":"2019-08-01T20:33:41Z","account":"NXLOG-AGENT","version":"v1","authentication_target":"-","source_ip":"212.92.117.25","event_type":"INGRESS_AUTHENTICATION","authentication_result":"FAILURE"}}

The difference in size between the two is 2,360 characters (2,917 vs only 557 characters), which is a reduction of 80.9%.

In conclusion

To summarize, SIEM technology consolidates and correlates log data to support security analysts in managing the vast amounts of information generated by virtually any type of network device or computer.

There is a common misconception that collecting every single log for the SIEM will result in better analytics, but in reality all it does is increase SIEM costs and generate a large number of false alerts. The solution is the proper implementation and utilization of a log collection tool combined with SIEM analytics. The result is better log collection, reduced log noise, reliable analytics, and reduced SIEM costs.

Deploying a proper security logging infrastructure is a huge challenge for any IT team. And, a good SIEM system requires an even better log collection environment. Following the best practices of the industry and fine-tuning your security frequently, leads to better IT Security operations.

About NXLog

NXLog provides a reliable centralized log collection across the entire enterprise. For more information on NXLog visit our website or schedule a meeting with our professionals.

NXLog Platform is an on-premises solution for centralized log management with
versatile processing forming the backbone of security monitoring.

With our industry-leading expertise in log collection and agent management, we comprehensively
address your security log-related tasks, including collection, parsing, processing, enrichment, storage, management, and analytics.

Start free Contact us