Pushing JSON log to Gelf

Post a different question
5
responses

Hi
I am trying to post events from my logs files to gelf_tcp (Graylog).
My log records are in flat json format.
I can push logs to gelf, however with few issues;
I appreciate if you can help.

Issue 1: My log record has a field called "level". When the record is sent to Graylog, level does not match the one that I have in log file.
I do parse_json() first and also I tried to explicitly set the value of level based on NXLog documentation https://nxlog.co/documentation/nxlog-user-guide/xm_gelf.html
There is also something wrong with documentation. It says gelf understand field "SeverityLevel" but in the example in the same page it is using "SyslogSeverityLevel".
I tried to explicitly set both fields with $level field but nothing changes in graylog.

Issue 2: I have timestamp filed in my log record which is ISO 8601 format. I could not find an easy way to parse it. the parsedata() function does not help there.
I end up extracting date and time part from my field and then send it to parsedate($1 + " " + $2). If you know a better way, please let me know.

Thanks

Kev

AskedJuly 23, 2020 - 6:59am

Comments (5)

  • ksaffarian's picture

    Hi Sure

    Here is sample line of log file:

    { "timestamp": "2020-07-23T02:52:31.7718702Z", "levelname": "ERROR", "level": "3", "message": "Stopping Channel MASTER_CONTROL_CHANNEL", "threadid": "8", "name": "BroadcastServer.Channel" }

    Here is my nxlog config:

    <Input MMS_Logs>

    Module  im_file

    File 'C:\\Logs\\*.log'
    CloseWhenIdle   TRUE
    SavePos TRUE
    ReadFromLast TRUE
    RenameCheck TRUE

    <Exec>
        parse_json();
        rename_field('message', 'Short_Message');
        $FullMessage = $raw_data;
        $SyslogSeverityValue = $level;
                $SeverityValue = $level;
        $LogFileName = file_name();
        $EventTimeStamp = $timestamp;
        if $timestamp =~ /(\d+\-\d+\-\d+)T(\d+\:\d+\:\d+)/
            $EventTime = parsedate($1 + " " + $2);
    </Exec>
</Input>

<Output Graylog>

    Module  om_tcp
    Host    %GraylogEndpoint%
    Port    12201
    OutputType  GELF_TCP

    <Exec> 
        $Event_Source = 'app-1';
        $aws_account = 'account-1';
    </Exec>

</Output>

    July 24, 2020 - 4:20am
  • ksaffarian's picture

    Hi buddy
    this seems to be working, to_json();
    thanks for the suggestion.
    Few things though:
    - I gave it a test and it only works if I set $SyslogSeverityLevel = $level. So, I believe this document need to be updated! https://nxlog.co/documentation/nxlog-user-guide/xm_gelf.html
    - $FullMessage does not show up in Graylog as well. Maybe the document is not really update and needs a good refresh.
    Cheers

    July 28, 2020 - 10:23am

Answers (0)