#1 cbitterfield 8 years ago

I am trying to parse AWS S3 Logs. They are in JSON format (One line no CR/LF) and Gzip'd.

I need to ingest these into syslog TCP or UDP (Testing with file out)

I can't get a reliable working nxlog.conf that will process the JSON files.

NXLOG

define ROOT /usr/local/libexec/nxlog/ Pidfile /var/run/nxlog.pid LogFile ./nxlog.log define WORK /Users/cbitterfield/awslogs-project SpoolDir %WORK%/data CacheDir %WORK%/data LogLevel DEBUG

Module xm_syslog Module xm_json Module im_file File "%WORK%/data19/*.json" Exec parse_json(); # Dump $raw_event Exec to_syslog_bsd(); SavePos FALSE ReadFromLast False Module om_file File "./output" Path in => out

Yields the following errors and no output.

2016-04-10 22:13:00 DEBUG '^KE<F4>t^G<C7>C^D' does not match wildcard '859121128579_CloudTrail_ap-northeast-.json' 2016-04-10 22:13:00 DEBUG checking '^KE<F4>t^G<C7>C^D' against wildcard '859121128579_CloudTrail_ap-northeast-.json': 2016-04-10 22:13:00 DEBUG '^KE<F4>t^G<C7>C^D' does not match wildcard '859121128579_CloudTrail_ap-northeast-.json' 2016-04-10 22:13:00 DEBUG checking '^KE<F4>t^G<C7>C^D' against wildcard '859121128579_CloudTrail_ap-northeast-.json': 2016-04-10 22:13:00 DEBUG '^KE<F4>t^G<C7>C^D' does not match wildcard '859121128579_CloudTrail_ap-northeast-.json' 2016-04-10 22:13:00 DEBUG checking '^KE<F4>t^G<C7>C^D' against wildcard '859121128579_CloudTrail_ap-northeast-.json':

Permalink

#2 adm Nxlog ✓ 8 years ago

#1 cbitterfield

I am trying to parse AWS S3 Logs. They are in JSON format (One line no CR/LF) and Gzip'd. I need to ingest these into syslog TCP or UDP (Testing with file out) I can't get a reliable working nxlog.conf that will process the JSON files. NXLOG define ROOT /usr/local/libexec/nxlog/ Pidfile /var/run/nxlog.pid LogFile ./nxlog.log define WORK /Users/cbitterfield/awslogs-project SpoolDir %WORK%/data CacheDir %WORK%/data LogLevel DEBUG Module xm_syslog Module xm_json Module im_file File "%WORK%/data19/*.json" Exec parse_json(); # Dump $raw_event Exec to_syslog_bsd(); SavePos FALSE ReadFromLast False Module om_file File "./output" Path in => out Yields the following errors and no output. 2016-04-10 22:13:00 DEBUG '^KE<F4>t^G<C7>C^D' does not match wildcard '859121128579_CloudTrail_ap-northeast-.json' 2016-04-10 22:13:00 DEBUG checking '^KE<F4>t^G<C7>C^D' against wildcard '859121128579_CloudTrail_ap-northeast-.json': 2016-04-10 22:13:00 DEBUG '^KE<F4>t^G<C7>C^D' does not match wildcard '859121128579_CloudTrail_ap-northeast-.json' 2016-04-10 22:13:00 DEBUG checking '^KE<F4>t^G<C7>C^D' against wildcard '859121128579_CloudTrail_ap-northeast-.json': 2016-04-10 22:13:00 DEBUG '^KE<F4>t^G<C7>C^D' does not match wildcard '859121128579_CloudTrail_ap-northeast-.json' 2016-04-10 22:13:00 DEBUG checking '^KE<F4>t^G<C7>C^D' against wildcard '859121128579_CloudTrail_ap-northeast-.json':

Where does ^KEt^GC^D come from? Which NXLog version is this?

Request a simple example for processing AWS Logs from the S3 Bucket.

NXLOG