csv to syslog (Linux) - Cisco Umbrella files.

#1 schrammbo 6 years ago

So...

I have about 3 days of experience with NXLog and what I am attempting to do is pull Cisco Umbrella Logs via an s3fs mount, unzip them and then read the logs using nxlog.

I am still working out the specifics of how to get all the logs unzipped and into one working file but my test file is failing to be read and sent to the syslog server.

I have taken a new nxlog.conf and set it up with the following:

########################################
# Global directives #
########################################
User nxlog
Group nxlog

LogFile /var/log/nxlog/nxlog.log
LogLevel INFO

########################################
# Modules #
########################################
<Extension csv>
Module xm_csv
</Extension>

<Input in>
Module im_file
File "/tmp/incoming.csv"
</Input>

<Output out>
Module om_udp
Host 10.x.x.x
Port 10514
Exec to_syslog_bsd();
</Output>

########################################
# Routes #
########################################
<Route 1>
Path in => out
</Route>

My nxlog.log file shows no issues with the connection or service but I am not getting any data from my test file.

Has anyone been able to do this or can you provide guidance on why I am not getting any data to transfer using my current setup?

Permalink

#2 b0ti Nxlog ✓ 6 years ago

#1 schrammbo

So... I have about 3 days of experience with NXLog and what I am attempting to do is pull Cisco Umbrella Logs via an s3fs mount, unzip them and then read the logs using nxlog. I am still working out the specifics of how to get all the logs unzipped and into one working file but my test file is failing to be read and sent to the syslog server. I have taken a new nxlog.conf and set it up with the following: ######################################## # Global directives # ######################################## User nxlog Group nxlog LogFile /var/log/nxlog/nxlog.log LogLevel INFO ######################################## # Modules # ######################################## <Extension csv> Module xm_csv </Extension> <Input in> Module im_file File "/tmp/incoming.csv" </Input> <Output out> Module om_udp Host 10.x.x.x Port 10514 Exec to_syslog_bsd(); </Output> ######################################## # Routes # ######################################## <Route 1> Path in => out </Route> My nxlog.log file shows no issues with the connection or service but I am not getting any data from my test file. Has anyone been able to do this or can you provide guidance on why I am not getting any data to transfer using my current setup?

In order to avoid shipping GBs of logs that already exist prior to deployment it will only read event records that were generated after nxlog was started, i.e. the default of ReadFromLast is TRUE. I assume this is causing the "problem".

You may want to add the following for testing:

ReadFromLast FALSE
SavePos FALSE