Raijin vs Elasticsearch

Log collection is most closely linked to enterprise security practices—​for example, aggregation and analysis in a SIEM. However, it is often valuable to collect certain logs for reasons other than security. It may even be a requirement of your organization, for the purposes of auditing, legal compliance, or data retention.

Storing all these logs in a database is the most efficient way to manage the data. Without a database, finding and managing logs stored as flat files, or even as structured data, can be challenging. Even so, not all database engines are suited to ingesting and organizing vast amounts of log data. When choosing a database engine to manage your logs, consider the following "must have" features:

Flexible schema

This is a "must have" feature. Some log sources can exhibit an inflexible schema like those we associate with relational databases. However, if you consult Microsoft’s Windows Event Log Reference, you will soon realize that the XML-based documents Microsoft uses to store event data cannot be easily mapped to a relational database schema. For instance, many Windows log sources contain an EventData field that is a complex datatype, which in this case defines a nested data structure containing multiple subfields. If we examine Windows Sysmon events, the EventData field may contain as few as 3 subfields or as many as 17 subfields depending on the Event ID. Ideally, the database should be able to store such events without loss of data and with minimum restructuring. Yes, a relational database could store such events in multiple tables joined by foreign key constraints, but a document database or a database engine that supports flexible schemas could do this without having to manually define schemas for every log source.

Another core feature of analytics-based systems is their ability to aggregate data. Without flexible schemas, a database cannot aggregate logs from disparate log sources that monitor a common category of events. For instance, the only way to aggregate DNS events is to normalize the field/column names and their respective datatypes common to multiple log sources, such as query name, query type, query class, query severity, process ID, etc. Fields/columns not common to multiple log sources like client ID, query flags, Windows Domain, Provider GUID, etc., can contain valuable data that should not be discarded. Only a flexible schema can safeguard such log source-specific data.

Even for log sources with fixed schemas, you should consider the probability that their providers will make changes to their schemas over time. Why wouldn’t a log source’s schema undergo updates to keep in step with its parent application? Do you really want to be maintaining schema changes for every log source you collect?

High ingestion rate

Ideally, log data should be ingested in real-time. Otherwise, consistently accumulating a backlog of events to be ingested will not result in a sustainable log collection solution. Any database infrastructure that cannot ingest all log data of interest in real-time will become more of a liability than an asset.

Time series data optimization

One type of log data that is usually important exists as time series data. Most database engines are not equipped to handle this type of data efficiently. The most popular database engines that are optimized for time series data are MongoDB, Prometheus, and a newcomer, Raijin.

There are many other factors to consider, like total cost of ownership, scalability, configurability, and reliability, to name a few. Still, these three basic requirements are essential and led to the selection of Raijin and Elasticsearch for this comparison. Although Raijin and Elasticsearch are very different in their design and goals, they share some important commonalities.

Introducing Raijin

Raijin is a columnar database that supports semi-structured data of varied forms, like logs. The impetus for creating Raijin was the need for a high-performance database engine tailor-made for log data that would also integrate easily and efficiently with NXLog agents. With native JSON support and a built-in REST API, there is no need to install any client-side tools, making it a perfect solution for integrating with web applications. For large organizations already using NXLog, Raijin is the logical choice for managing their logs. It provides the foundation needed for generating real-time analytics.

Under the hood, Raijin has been engineered for high-performance. It leverages modern improvements in CPU design—​like optimized SIMD instructions (SSE2/AVX2)--to excel when working with vectorized data, which means more data processed with fewer cycles. With its hybrid columnar data storage and block-level data compression, you will see a marked reduction in your data storage requirements, and better performance with data ingestion as well as queries. Combined with vectorized execution, the data throughput that analytical workloads demand can be easily met.

Perhaps the most exciting characteristic of the Raijin database engine is that it achieves its query performance without indexes. Most other database engines would only reach a fraction of their purported performance without explicit, user-defined index definitions.

Database types

Elasticsearch is a document-oriented database, which traditionally falls into the NoSQL category because its flexible schema supports nested data structures, similar to the XML-based documents Windows uses for event logs. Raijin, on the other hand, is in a class all by itself with its flexible schemas, REST API, hybrid columnar data storage, and native SQL capabilities. It has many similarities with relational database engines. Because it doesn’t support nested data structures, Raijin stores its rows and columns of data in tables instead of documents. What sets it part from the relational crowd is its unique ability to insert data into a newly-created table without having to first define a schema, although it does support schema definitions. The other unique feature is its ability to ingest large batches of JSON data via its REST API, which gives it exceptional write performance. In addition to the standard INSERT INTO <table> SET syntax, it offers an alternate JSON syntax for inserting JSON-formatted data.

INSERT INTO events {
    "hostname":  "srv-04",
    "eventtime": "2015-06-24 12:20:12Z",
    "eventtype": "ERROR",
    "message":   "Database timeout!"
};

Query languages

Raijin natively speaks SQL. Like other popular NoSQL databases, native Elasticsearch queries follow JSON syntax, but the structure and reserved names used for the keys are vendor-specific, as shown in this unaltered example taken directly from Elastic’s documentation on Query and filter context:

Example 1. Example of query and filter contexts

Below is an example of query clauses being used in query and filter context in the search API. This query will match documents where all of the following conditions are met:

  • The title field contains the word search.

  • The content field contains the word elasticsearch.

  • The status field contains the exact word published.

  • The publish_date field contains a date from 1 Jan 2015 onwards.

GET /_search
{
  "query": { (1)
    "bool": { (2)
      "must": [
        { "match": { "title":   "Search"        }},
        { "match": { "content": "Elasticsearch" }}
      ],
      "filter": [ (3)
        { "term":  { "status": "published" }},
        { "range": { "publish_date": { "gte": "2015-01-01" }}}
      ]
    }
  }
}
1 The query parameter indicates query context.
2 The bool and two match clauses are used in query context, which means that they are used to score how well each document matches.
3 The filter parameter indicates filter context. Its term and range clauses are used in filter context. They will filter out documents which do not match, but they will not affect the score for matching documents.

Although Query DSL (Domain Specific Language) may be the best way to query for documents in Elasticsearch, Elastic is fully aware of the significant, long-term investments most larger organizations have made in SQL-based solutions. For this reason, they offer Elasticsearch SQL as a way to bridge this gap.

If a Raijin database were to contain the same documents used in the Elasticsearch example above, the equivalent SQL query shown here would work with Raijin right out of the box:

SELECT * FROM tbl
WHERE MATCH (title) AGAINST ('Search')
AND   MATCH (content) AGAINST ('Elasticsearch')
AND   status = 'published'
AND   DATE(publish_date) >= '2015-01-01';

Performance

We conducted the following benchmarks to see how Raijin stacks up against Elasticsearch:

The CPU used in our tests that produced these benchmarks was an AMD Ryzen 7 4800H. We used the same set of 120,000 logs for benchmarking data ingestion and query processing speed.

Ingestion speed

Raijin vs Elasticsearch ingestion speed
Figure 1. Performance variations and scaling for Raijin and Elasticsearch

The labels on the x-axis represent the test parameters in the format {1} T-{2} R-{3}, where:

  • {1} is the number of processes Raijin or Elasticsearch used to ingest the records

  • {2} is the number of records sent

  • {3} is the batch size (number of events sent per HTTP request)

When using six threads, Raijin reached its maximum performance of 355,603 events per second when run with six threads, enabling it to ingest 158% more data per second than Elasticsearch. The test was designed to ingest 120,000 records using 12 HTTP requests, i.e., 10,000 records per batch. Even Raijin’s single-thread performance proved to be more than sufficient.

Query processing speed

Raijin vs Elasticsearch query speed
Figure 2. Query performance for Raijin and Elasticsearch
The queries used in these tests
query0: SELECT AVG(case when c3 < 0 then c3 end) FROM logs GROUP BY uid
query1: SELECT AVG(c3) FROM logs GROUP BY uid
query2: SELECT COUNT(c3) FROM logs GROUP BY uid
query3: SELECT MAX(c3) FROM logs GROUP BY uid
query4: SELECT MIN(c3) FROM logs GROUP BY uid
query5: SELECT SUM(c3) FROM logs GROUP BY uid
query6: SELECT COUNT(c3) FROM logs WHERE uid = 24 GROUP BY uid
query7: SELECT COUNT(DISTINCT c3) FROM logs WHERE uid = 24 GROUP BY uid
query8: SELECT COUNT(*) FROM logs WHERE ip LIKE '32%.%.%.127'
query9: SELECT COUNT(c3) FROM logs WHERE c3 < 0
query10: SELECT COUNT(case when c3 < 0 then c3 end) FROM logs
query11: SELECT COUNT(CAST(c3 AS INT)) FROM logs WHERE c3 > -200
query12: SELECT COUNT(CAST(c3 AS FLOAT)) FROM logs WHERE c3 > -200
query13: SELECT COUNT(CAST(c3 AS DOUBLE)) FROM logs WHERE c3 > -200
query14: SELECT COUNT(CAST(c3 AS STRING)) FROM logs WHERE c3 > -200
query15: SELECT COUNT(c3) FROM logs WHERE MOD(c3,2) <> 0
query16: SELECT COUNT(c3) FROM logs WHERE MOD(c3,2) = 0
query17: SELECT FIRST(c3) FROM logs GROUP BY uid
query18: SELECT LAST(c3) FROM logs GROUP BY uid
query19: SELECT COUNT(IIF(success=false,True,null)) FROM logs where '2022-06-02'<time and time<'2022-06-02'
query20: SELECT COUNT(action) FROM logs where '2022-06-01'<time and time<'2020-06-30' and success=false

To summarize these results, it is readily apparent that Raijin is a better choice for aggregation queries in general. Elasticsearch performed poorly in 11 of these 21 queries, with results taking over 20 seconds to be returned. When executing six of these queries, Elasticsearch needed more than half a minute to return any results. The poorest performance was the roughly 55 seconds Elasticsearch needed to complete query0.

However, even when Elasticsearch was faster than Raijin while running query11 through query14 (probably due to the CAST function within COUNT()), Raijin’s poorer performance was still in the sub-5-second range, a far cry from taking more than half a minute to return results!

The biggest surprise was Elasticsearch performing better than Raijin on query20, which counts how many documents contain the action field and uses time to filter the results to a specific date range. However, the query only took under three seconds with Raijin, which is still a perfectly reasonable length of time.

Average CPU usage

Raijin vs Elasticsearch average CPU usage
Figure 3. Average CPU usage for Raijin and Elasticsearch

Elasticsearch CPU usage increased with the number of CPU threads, reaching a maximum value of 637 cycles. On the other hand, Raijin reached a maximum of 485 cycles, which translates to a 23.8% lower average CPU usage than Elasticsearch.

On a side note, while benchmarking the query speeds, Raijin demonstrated lower CPU usage in 17 out of 21 tests, while Elasticsearch’s CPU usage remained high in most tests.

Average memory usage

Raijin vs Elasticsearch memory usage
Figure 4. Average memory usage for Raijin and Elasticsearch

Elasticsearch maintained a stable memory consumption of 12,288 MB across the board during data ingestion. In comparison, Raijin only consumed 4,096 MB.

Disk I/O

Raijin vs Elasticsearch disk I/O
Figure 5. Disk I/O for Raijin and Elasticsearch

By default, Raijin uses compression to reduce disk usage requirements. As expected, these results show a dramatic difference in disk usage for both read and write tests when batch size is set to one. With increasing batch size, Elasticsearch uses less disk space. However, regardless of batch size, Raijin’s disk usage remains only a fraction of Elasticsearch’s.

Who is the winner?

Actually, both are winners. They both offer schemaless (schema flexibility) storage of log records, which is essential for maintaining the integrity of event logs. Both perform well when querying date/time ranges of time-series data.

However, each database engine has its area of expertise, which became evident during our performance testing. If data ingestion rates are a problem in your enterprise, Raijin is the better choice for collecting and managing your logs. Also, Raijin’s performance with true aggregation queries that involve a GROUP BY clause is phenomenal. So, if real-time data analytics is your goal, you will be better served using Raijin for that task.

Elasticsearch still reigns as the king of full-text search in the world of database engines. If full-text search performance is your most important feature, then you are probably already using Elasticsearch and should stick with it if that is your highest priority. For most use cases, Raijin’s full-text search performance is acceptable. In fact, when paired with NXLog’s powerful parsing capabilities, the need to perform full-text searches will be greatly diminished. This is especially true once NXLog has parsed catch-all fields used for storing embedded, but flattened, structured data and has sent these new fields to Raijin as dedicated columns in a flexible schema. Ideally, most log data should be parsed as structured data. The need for excessive full-text searches is often an indicator of a data model that could use some design improvements.

If you are unaware of this, Raijin offers a perk that you might find very appealing. Raijin can be downloaded for free and used as an on-premise database engine. This means you are free to deploy it anywhere you like, and you can manage your logs without any subscription or data storage costs.

Solutions for common needs

Log storage and analytics

Most NoSQL solutions are inefficient or lack support for analytical queries. Raijin Database is optimized for high-volume log ingestion and exhibits exceptional query times when executing standard SQL aggregation queries. It can function as the aggregator and final resting place of logs in a centralized logging environment. It is the logical facilitator of enterprise-wide logging analytics.

Data retention compliance

With Raijin’s expression-based partitioning, partitions can be dynamically created based on a combination of date and other columns. This enables the precise selection of records to be purged based on your specific data retention requirements. For further details, including examples showing how easy it is to automate this process, see the Purging data guide in the Raijin Database Engine User Manual.

Summary

After looking at the benefits of storing log data in a database, we established some basic database features needed for effectively managing logs and gleaning analytics from their data, with both Raijin and Elasticsearch as contenders for handling these tasks.

The performance measured in this limited set of benchmarks may not be conclusive regarding overall capabilities since database engines are complex tools, each having its own design goals and set of tasks that it has been optimized for. However, given the challenge of finding the best overall database engine for collecting and aggregating logs in a centralized logging environment, we can only conclude that Raijin has a definite edge over Elasticsearch. Our criteria for this decision rests not only on the "must have" features we introduced early on, but also the common tasks that larger organizations are grappling with when it comes to high-performance log aggregation in real time and their need to easily comply with data retention mandates.

Raijin may be the new kid on the block, and may not provide all the functionality other database engines have yet, but then again, no single database engine is the perfect choice for every use case imaginable. If you want to collect logs in a database, why not try them both yourself? Go download Elasticsearch and set it up. Then download Kibana and set it up. Send Elasticsearch a fairly large set of events from your log sources, run some aggregations queries having a GROUP BY clause and make some notes of your overall experience. Then go download Raijin and follow the Quickstart guide. Send Raijin the same logs you sent Elasticsearch, and run the same aggregation queries. Let us know which one was easier for you to deploy and how they integrated with your log sources. Share your experience with us regarding the differences between both solutions regarding data ingestion and running queries.

GET STARTED TODAY:
Disclaimer

While we endeavor to keep the information in this topic up to date and correct, NXLog makes no representations or warranties of any kind, express or implied about the completeness, accuracy, reliability, suitability, or availability of the content represented here.

The accurateness of the content was tested and proved to be working in our lab environment at the time of the last revision with the following software versions:

Raijin M5
Elasticsearch 8.3.1
Ubuntu 18.04.1

Last revision: 9 July 2022

NXLog Ltd. develops multi-platform log collection tools that support many different log sources, formats, transports, and integrations. The tools help administrators collect, parse, and forward logs so they can more easily respond to security issues, investigate operational problems, and analyze event data. NXLog distributes the free and open source NXLog Community Edition and offers additional features and support with the NXLog Enterprise Edition.

This document is provided for informational purposes only and is subject to change without notice. Trademarks are the properties of their respective owners.

Download a fully functional trial of the Enterprise Edition for free