Common format for Windows, internal logs and my app logs

#1 igorgatis 9 years ago

TL;DR: what's the recommended way of converting logs to a common (e.g. GELF) format?

I'm using NXLog together with Logstash and EalsticSearch. I'm collecting logs from Windows, NXLogs (internal) and my app logs using line based JSON.

Windows logs and NXLogs seem to share a lot of field names. I can write my app so that it uses the same fields. This greatly facilitates viewing data in elasticsearch. I could stick with windows fields or convert them all to GELF. AFAIK, the convertion from Windows Logs to GELF seems seems to require a lot of per-field convertion work. There is a good chance I won't get it right until enough data is produced.

I was looking for a convert_to_gelf() function which would take care of converting Windows Logs, Internal logs, IIS, etc to GELF. Is there such thing? Is manual conversion my only option?

Permalink

#2 adm Nxlog ✓ 9 years ago (Last updated 9 years ago )

#1 igorgatis

TL;DR: what's the recommended way of converting logs to a common (e.g. GELF) format? I'm using NXLog together with Logstash and EalsticSearch. I'm collecting logs from Windows, NXLogs (internal) and my app logs using line based JSON. Windows logs and NXLogs seem to share a lot of field names. I can write my app so that it uses the same fields. This greatly facilitates viewing data in elasticsearch. I could stick with windows fields or convert them all to GELF. AFAIK, the convertion from Windows Logs to GELF seems seems to require a lot of per-field convertion work. There is a good chance I won't get it right until enough data is produced. I was looking for a convert_to_gelf() function which would take care of converting Windows Logs, Internal logs, IIS, etc to GELF. Is there such thing? Is manual conversion my only option?

There is an xm_gelf module which provides an output formatter for GELF.

For UDP:

<Output out>
    Module      om_udp
    Host        127.0.0.1
    Port        12201
    OutputType  GELF_UDP
</Output>

For TCP:

<Output out>
    Module      om_tcp
    Host        127.0.0.1
    Port        12201
    OutputType  GELF_TCP
</Output>

See the reference manual for more information.

The only reason why these are implemented this way is because GELF for TCP is very recent and GELF_UDP uses a zlib compressed format. This is a binary payload which cannot be produced with a function like to_gelf() if that was available.

If you are not using Graylog, I'm not sure what the point is in using GELF since NXLog already produces a set of fields that are normalized to a common name in all modules (if possible). You can simply use xm_json and call to_json().