2
responses

I am trying to parse AWS S3 Logs. They are in JSON format (One line no CR/LF) and Gzip'd.

I need to ingest these into syslog TCP or UDP (Testing with file out)

I can't get a reliable working nxlog.conf that will process the JSON files.

# NXLOG
define ROOT /usr/local/libexec/nxlog/
Pidfile /var/run/nxlog.pid
LogFile ./nxlog.log
define WORK /Users/cbitterfield/awslogs-project
SpoolDir %WORK%/data
CacheDir %WORK%/data
LogLevel DEBUG

Module xm_syslog

Module xm_json

Module im_file
File "%WORK%/data19/*.json"
Exec parse_json();
# Dump $raw_event
Exec to_syslog_bsd();
SavePos FALSE
ReadFromLast False

Module om_file
File "./output"

Path in => out

Yields the following errors and no output.

2016-04-10 22:13:00 DEBUG '^KEt^GC^D' does not match wildcard '859121128579_CloudTrail_ap-northeast-*.json'
2016-04-10 22:13:00 DEBUG checking '^KEt^GC^D' against wildcard '859121128579_CloudTrail_ap-northeast-*.json':
2016-04-10 22:13:00 DEBUG '^KEt^GC^D' does not match wildcard '859121128579_CloudTrail_ap-northeast-*.json'
2016-04-10 22:13:00 DEBUG checking '^KEt^GC^D' against wildcard '859121128579_CloudTrail_ap-northeast-*.json':
2016-04-10 22:13:00 DEBUG '^KEt^GC^D' does not match wildcard '859121128579_CloudTrail_ap-northeast-*.json'
2016-04-10 22:13:00 DEBUG checking '^KEt^GC^D' against wildcard '859121128579_CloudTrail_ap-northeast-*.json':

AskedApril 11, 2016 - 2:59pm

Comments (2)

  • cbitterfield's picture

    I still haven't solved this issue. It is the latest version of NXLog

    nxlog-ce-2.9.1716

    the issue is that AWS JSON logs are one long line; or I format nicely; either way; it does not parse back to Syslog. I get lots of variations.

Answers (0)