Raijin has announced the release of version 2.1 of its powerful, schemaless SQL-like database engine. This focuses on performance improvements.
Read on for the highlights and check out the Raijin release notes for a complete list of the features and improvements.
Performance improvements
As mentioned, this release focused on optimizing the performance of partitioned database tables. Partitioned tables store data in separate locations with their own set of metadata based on the values present in the data. One common pattern is to use partitioning to separate data based on time, creating, for example, one partition per day. This allows time-based queries to skip most of the data in tables.
Our tests confirmed that query performance improved across the board - an average of 83% reduction in response times across a mix of 28 typical queries.
Before we completed these optimizations, our query time increased nearly linearly with the amount of stored data. This is no longer the case as Raijin’s performance no longer decays as more data is added to the tables.
Data preparation for the query tests
We used a piece of the 60GB of JSON data for data generation.
We extracted eight million records and modified the SourceModuleName field to simulate 13 input modules.
During generation, We overwrote the first part of the timestamp (year, month, day) for all 8 million records to the same day, starting with a timestamp of 2023-04-01T02:00:01.000000+02:00.
We then inserted these records into the database, shifting the timestamp by one day in each cycle. The last timestamp was 2023-08-28T17:34:02.130184+02:00.
The image below displays the performance improvement chart.
Collect logs with NXLog
As a database engine, Raijin is geared towards semi-structured data, making it a perfect solution for log storage. Log data is notoriously unstructured, making it hard to store in rigid table structures like traditional SQL systems.
Raijin complements NXLog’s powerful log collection and centralization solutions. With NXLog Enterprise Edition’s Raijin integration, you can seamlessly and easily send logs from NXLog Enterprise Edition to a Raijin database instance. The two fit hand in glove, providing the methods of capturing, processing, and storing log data for a complete log management solution.
