65. Elasticsearch and Kibana

Table of Contents

Elasticsearch is a search engine and document database that is commonly used to store logging data. Kibana is a popular user interface and querying front-end for Elasticsearch. Kibana is often used with the Logstash data collection engine—together forming the ELK stack (Elasticsearch, Logstash, and Kibana).

However, Logstash is not actually required to load data into Elasticsearch. NXLog can do this as well, and offers several advantages over Logstash—this is the KEN stack (Kibana, Elasticsearch, and NXLog).

Because Logstash is written in Ruby and requires Java, it has high system resource requirements. NXLog has a small resource footprint and is recommended by many ELK users as the log collector of choice for Windows and Linux.
Due to the Java dependency, Logstash requires system administrators to deploy the Java runtime onto their production servers and keep up with Java security updates. NXLog does not require Java.
Elastic’s Logstash wmi input plugin creates events based on the results of a WMI query. This method incurs a significant performance penalty. NXLog uses the native Windows Event Log API in order to more efficiently capture Windows events.

The following sections explain how to configure NXLog to:

send logs directly to Elasticsearch, replacing Logstash; or
forward collected logs to Logstash, acting as a log collector for Logstash.

65.1. Sending logs to Elasticsearch

Consult the Elasticsearch Reference and the Kibana User Guide for more information about installing and configuring Elasticsearch and Kibana. For NXLog Enterprise Edition 3.x, see Using Elasticsearch With NXLog Enterprise Edition 3.x in the Reference Manual.

Configure NXLog.

Example 289. Using om_elasticsearch

The om_elasticsearch module is only available in NXLog Enterprise Edition. Because it sends data in batches, it reduces the effect of the latency inherent in HTTP responses, allowing the Elasticsearch server to process the data much more quickly (10,000 EPS or more on low-end hardware).

nxlog.conf


  1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16

  <Extension _json>
    Module      xm_json
</Extension>

<Output out>
    Module        om_elasticsearch
    URL           http://localhost:9200/_bulk
    FlushInterval   2
    FlushLimit    100

    # Create an index daily
    Index       strftime($EventTime, "nxlog-%Y%m%d")

    # Use the following if you do not have $EventTime set
    #Index      strftime($EventReceivedTime, "nxlog-%Y%m%d")
</Output>

Example 290. Using om_http

For NXLog Community Edition, the om_http module can be used instead to send logs to Elasticsearch. Because it sends a request to the Elasticsearch HTTP REST API for each event, the maximum logging throughput is limited by HTTP request and response latency. Therefore this method is suitable only for low-volume logging scenarios.

nxlog.conf


  1
2
3
4
5
6
7
8
9
10
11
12
13
14
15

  <Extension _json>
    Module      xm_json
</Extension>

<Output out>
    Module      om_http
    URL         http://localhost:9200
    ContentType application/json
    <Exec>
        set_http_request_path(strftime($EventTime, "/nxlog-%Y%m%d/" +
                                       $SourceModuleName));
        rename_field("timestamp", "@timestamp");
        to_json();
    </Exec>
</Output>

Restart NXLog, and make sure the event sources are sending data. This can be checked with curl -X GET 'localhost:9200/_cat/indices?v&pretty'. There should be an index matching nxlog* and its docs.count counter should be increasing.
Configure the appropriate index pattern for Kibana.
1. Open Management on the left panel and click on Index Patterns.
2. Set the Index pattern to nxlog*. A matching index should be listed. Click > Next step.
3. Set the Time Filter field name selector to EventTime (or EventReceivedTime if the $EventTime field is not set by the input module). Click Create index pattern.
Test that the NXLog and Elasticsearch/Kibana configuration is working by opening Discover on the left panel.

65.2. Forwarding logs to Logstash

NXLog can be configured to act as a log collector, forwarding logs to Logstash in JSON format.

Set up a configuration on the Logstash server to process incoming event data from NXLog.

logstash.conf

input {
  tcp {
    codec => json_lines { charset => CP1252 }
    port => "3515"
    tags => [ "tcpjson" ]
  }
}
filter {
  date {
    locale => "en"
    timezone => "Etc/GMT"
    match => [ "EventTime", "YYYY-MM-dd HH:mm:ss" ]
  }
}
output {
  elasticsearch {
    host => localhost
  }
  stdout { codec => rubydebug }
}

Note

The json codec in Logstash sometimes fails to properly parse JSON—it will concatenate more than one JSON record into one event. Use the json_lines codec instead.

Although the im_msvistalog module converts data to UTF-8, Logstash seems to have trouble parsing that data. The charset => CP1252 seems to help.

Configure NXLog.

nxlog.conf


  1
2
3
4
5
6

  <Output out>
    Module  om_tcp
    Host    10.1.1.1
    Port    3515
    Exec    to_json();
</Output>

Restart NXLog.

You are here

65. Elasticsearch and Kibana | Log Collection Solutions