5
responses

Hi
I am trying to post events from my logs files to gelf_tcp (Graylog).
My log records are in flat json format.
I can push logs to gelf, however with few issues;
I appreciate if you can help.

Issue 1: My log record has a field called "level". When the record is sent to Graylog, level does not match the one that I have in log file.
I do parse_json() first and also I tried to explicitly set the value of level based on NXLog documentation https://nxlog.co/documentation/nxlog-user-guide/xm_gelf.html
There is also something wrong with documentation. It says gelf understand field "SeverityLevel" but in the example in the same page it is using "SyslogSeverityLevel".
I tried to explicitly set both fields with $level field but nothing changes in graylog.

Issue 2: I have timestamp filed in my log record which is ISO 8601 format. I could not find an easy way to parse it. the parsedata() function does not help there.
I end up extracting date and time part from my field and then send it to parsedate($1 + " " + $2). If you know a better way, please let me know.

Thanks

Kev

AskedJuly 23, 2020 - 6:59am

Comments (5)

  • ksaffarian's picture

    Hi Sure

    Here is sample line of log file:

    { "timestamp": "2020-07-23T02:52:31.7718702Z", "levelname": "ERROR", "level": "3", "message": "Stopping Channel MASTER_CONTROL_CHANNEL", "threadid": "8", "name": "BroadcastServer.Channel" }
    

    Here is my nxlog config:

    <Input MMS_Logs>
    
        Module  im_file
    
        File 'C:\\Logs\\*.log'
        CloseWhenIdle   TRUE
        SavePos TRUE
        ReadFromLast TRUE
        RenameCheck TRUE
    
        <Exec>
            parse_json();
            rename_field('message', 'Short_Message');
            $FullMessage = $raw_data;
            $SyslogSeverityValue = $level;
                    $SeverityValue = $level;
            $LogFileName = file_name();
            $EventTimeStamp = $timestamp;
            if $timestamp =~ /(\d+\-\d+\-\d+)T(\d+\:\d+\:\d+)/
                $EventTime = parsedate($1 + " " + $2);
        </Exec>
    </Input>
    
    <Output Graylog>
    
        Module  om_tcp
        Host    %GraylogEndpoint%
        Port    12201
        OutputType  GELF_TCP
    
        <Exec> 
            $Event_Source = 'app-1';
            $aws_account = 'account-1';
        </Exec>
    
    </Output>
    

  • ksaffarian's picture

    Hi buddy
    this seems to be working, to_json();
    thanks for the suggestion.
    Few things though:
    - I gave it a test and it only works if I set $SyslogSeverityLevel = $level. So, I believe this document need to be updated! https://nxlog.co/documentation/nxlog-user-guide/xm_gelf.html
    - $FullMessage does not show up in Graylog as well. Maybe the document is not really update and needs a good refresh.
    Cheers

Answers (0)