How can I extend a JSON document with an extra field?

#1 object 9 years ago

Hello,

Our application writes logs in JSON format so it's quite straightforward to send them to Elasticsearch using om_http module. However we need to enrich JSON logs with additional information like application name. I was searching for the solution and found that I could do the following:

<Output elasticsearch>
    Module      om_http
   URL            (server_url)
   ContentType application/json
    Exec       set_http_request_path(strftime(now(), "/test-%Y.%m/log/"));
   Exec       parse_json(); $Application="MyApp"; to_json();
</Output>

The last line in the output specification make sure the json payload is first parsed and then generated again, enriched with a new field "Application". I wonder if this is a right approach or there are other alternatives.

Thanks in advance

Permalink

#2 adm Nxlog ✓ 9 years ago

#1 object

Hello, Our application writes logs in JSON format so it's quite straightforward to send them to Elasticsearch using om_http module. However we need to enrich JSON logs with additional information like application name. I was searching for the solution and found that I could do the following: <Output elasticsearch> Module om_http URL (server_url) ContentType application/json Exec set_http_request_path(strftime(now(), "/test-%Y.%m/log/")); Exec parse_json(); $Application="MyApp"; to_json(); </Output> The last line in the output specification make sure the json payload is first parsed and then generated again, enriched with a new field "Application". I wonder if this is a right approach or there are other alternatives. Thanks in advance

Generally this should be fine. Note that parse_json() does not support nested JSON, only flat key-values.

The NXLog Enterprise Edition has an om_elasticsearch module that uses the bulk API. If you have a lot of data to load this will be a lot faster than om_http which submits a single event per http request.