I am trying to use nxlog to process some historical files and load them into graylog. It is a nested json file and I only need a few fields out of it. I am hoping for some pointers on how to get the following data out of the json file and posted into graylog.

username rename text to message rename first title to method rename title_link to method_link

From the fields section Severity: value Region: Value Rename Last Seen to timestamp: value I would like graylog to use this as the timestamp for the data coming in.

Below is my whole nxlog.conf but the relevant input is the guardduty input.

nxlog.conf User nxlog Group nxlog Panic Soft

default values:

define INSTALLDIR /opt/nxlog

PidFile %INSTALLDIR%/var/run/nxlog/nxlog.pid

CacheDir %INSTALLDIR%/var/spool/nxlog

ModuleDir %INSTALLDIR%/lib/nxlog/modules

SpoolDir %INSTALLDIR%/var/spool/nxlog

define CERTDIR %INSTALLDIR%/var/lib/nxlog/cert define CONFDIR %INSTALLDIR%/etc/nxlog.d

Note that these two lines define constants only; the log file location

is ultimately set by the LogFile directive (see below). The

MYLOGFILE define is also used to rotate the log file automatically

(see the _fileop block).

define LOGDIR %INSTALLDIR%/var/log/nxlog define MYLOGFILE %LOGDIR%/nxlog.log

If you are not using NXLog Manager, disable the include line

and enable LogLevel and LogFile.

#include %CONFDIR%/*.conf LogLevel DEBUG LogFile %MYLOGFILE%

<Extension _syslog> Module xm_syslog </Extension>

This block rotates %MYLOGFILE% on a schedule. Note that if LogFile

is changed in managed.conf via NXLog Manager, rotation of the new

file should also be configured there.

<Extension _fileop> Module xm_fileop

# Check the size of our log file hourly, rotate if larger than 5MB
<Schedule>
    Every   1 hour
    <Exec>
        if ( file_exists('%MYLOGFILE%') and
             (file_size('%MYLOGFILE%') >= 5M) )
        {
             file_cycle('%MYLOGFILE%', 8);
        }
    </Exec>
</Schedule>

# Rotate our log file every week on Sunday at midnight
<Schedule>
    When    @weekly
    Exec    if file_exists('%MYLOGFILE%') file_cycle('%MYLOGFILE%', 8);
</Schedule>

</Extension>

<Extension json_parser> Module xm_json Flatten True PrettyPrint True </Extension>

<Extension exec> Module xm_exec </Extension>

<Extension gelf> Module xm_gelf </Extension>

<Extension antivirus_csv> Module xm_csv Fields $Severity,$timestamp,$ip,$endtime,$User,$User Groups,$Device,$Device Group Delimiter , </Extension>

<Extension xml> Module xm_xml </Extension>

<Input alienvault> Module im_file File "/var/nxlog/alienvault/*.log" SavePos TRUE ReadFromLast TRUE <Exec> parse_xml(); to_json(); </Exec> </Input>

<Input antivirus> Module im_file File "/var/nxlog/antivirus/*.csv"

ReadFromLast TRUE

SavePos True

<Exec> antivirus_csv->parse_csv(); to_json(); </Exec> </Input>

<Input guardduty> Module im_file File "/var/nxlog/guardduty/*.json" SavePos True InputType json_parser <Exec> #Delete some fields that aren't necessary delete($type); delete($subtype); delete($text); delete($mrkdwn_in); delete($EventReceivedTime);

    #Convert fields back to JSON because of the deletion
    to_json();
</Exec>

</Input>

<Output udp_12202> Module om_udp Host graylog.mydomain.com:12202 OutputType GELF_UDP </Output>

<Output udp_5515> Module om_udp Host graylog.mydomain.com:5515 OutputType GELF_UDP </Output>

<Route antivirus_to_udp_5515> Path antivirus => udp_5515 </Route>

<Route to_udp_12202> Path alienvault, guardduty => udp_12202 </Route>

Sample JSON. The files are larger but I kept the first and last entry. [ { "type": "message", "subtype": "bot_message", "text": "", "ts": "1614611466.000100", "username": "GuardDuty", "icons": { "image_48": "https://s3-us-east-1.amazonaws.com/slack-files2/bot_icons/2019-07-02/671758367922_48.png" }, "bot_id": "BL5799J6A", "attachments": [ { "fallback": "Recon:IAMUser/UserPermissions - <https://console.aws.amazon.com/guardduty/home?region=us-east-1#/findings?search=id%5t45ffg59a575art6789404dr5oci5a4zz>", "text": "APIs commonly used to discover the users, groups, policies and permissions in an account, was invoked by IAM principal Some-Role under unusual circumstances. Such activity is not typically seen from this principal.", "pretext": "Finding in us-east-1 for Acct: 505986456095", "title": "Recon:IAMUser/UserPermissions", "id": 1, "title_link": "https://console.aws.amazon.com/guardduty/home?region=us-east-1#/findings?search=id%5t45ffg59a575art6789404dr5oci5a4zz", "color": "e2d43b", "fields": [ { "title": "Severity", "value": "Medium", "short": true }, { "title": "Region", "value": "us-east-1", "short": true }, { "title": "Last Seen", "value": "<!date^1614611294^{date} at {time} | 2021-03-01T15:08:14.526Z>", "short": true } ], "mrkdwn_in": [ "pretext" ] } ] }, { "type": "message", "subtype": "bot_message", "text": "", "ts": "1614629763.001600", "username": "GuardDuty", "icons": { "image_48": "https://s3-us-east-1.amazonaws.com/slack-files2/bot_icons/2019-07-02/671758367922_48.png" }, "bot_id": "BL5799J6A", "attachments": [ { "fallback": "Recon:IAMUser/UserPermissions - <https://console.aws.amazon.com/guardduty/home?region=us-east-1#/findings?search=id%5t45ffg59a575art6789404dr5oci5a4zz>", "text": "APIs commonly used to discover the users, groups, policies and permissions in an account, was invoked by IAM principal Some-Role under unusual circumstances. Such activity is not typically seen from this principal.", "pretext": "Finding in us-east-1 for Acct: 505986456095", "title": "Recon:IAMUser/UserPermissions", "id": 1, "title_link": "https://console.aws.amazon.com/guardduty/home?region=us-east-1#/findings?search=id%5t45ffg59a575art6789404dr5oci5a4zz", "color": "e2d43b", "fields": [ { "title": "Severity", "value": "Medium", "short": true }, { "title": "Region", "value": "us-east-1", "short": true }, { "title": "Last Seen", "value": "<!date^1614629482^{date} at {time} | 2021-03-01T20:11:22.426Z>", "short": true } ], "mrkdwn_in": [ "pretext" ] } ] } ]