nxlog to read multiple CSV files and send it SIEM

Tags:

#1 sunilj777 3 years ago

We have a requirement to read CSV from one folder which is auto generated by the application with a new file name at schedule time say either Ex: 1 hour or 4 hours

Following are the fields of first CSV file Example File Name: Tue Aug 25 20-21-49 2020_SessionHistory.CSV SessionID,User ID,Scan Time,Total Files Scanned,Total Files Processed,Total Files Skipped,Total Files Allowed,Total Files Blocked,Device ID,Device Type,Device Manufacturer,Device Model,Device Serial Number 56749F51A4-53CC-4CCF-B606-42334680F890,"Labuser1","08/28/2020 08:24:35",2,2,0,2,0,"USBSTOR\DISK&VEN_SANDISK&PROD_CRUZER_BLADE&REV_1.00\4C530000010112115155&0","USB Device","(Standard disk drives)","SanDisk Cruzer Blade USB Device","4C530000010112115155" 56749F51A4-53CC-4CCF-B606-42334680F890,"Labuser2","08/28/2020 09:24:35",2,2,0,2,0,"USBSTOR\DISK&VEN_SANDISK&PROD_CRUZER_BLADE&REV_1.00\4C530000010112115155&0","USB Device","(Standard disk drives)","SanDisk Cruzer Blade USB Device","4C530000010112115155" 56749F51A4-53CC-4CCF-B606-42334680F890,"Labuser3","08/28/2020 09:24:35",2,2,0,2,0,"USBSTOR\DISK&VEN_SANDISK&PROD_CRUZER_BLADE&REV_1.00\4C530000010112115155&0","USB Device","(Standard disk drives)","SanDisk Cruzer Blade USB Device","4C530000010112115155"

Following are the fields of 2nd CSV file SessionID,File Name,SHA-256,Scan Result,Scan Result Description 50167E52-0380-4191-9DFC-8E1454948C50,E:\eicarcom2.zip,E1105070BA828007508566E28A2B8D4C65D192E9EAF3B7868382B7CAE747B397,0, Threat Detected 50167E52-0380-4191-9DFC-8E1454948C51,E:\eicarcom3.zip,E1105070BA828007508566E28A2B8D4C65D192E9EAF3B7868382B7CAE747B397,0, Threat Detected 50167E52-0380-4191-9DFC-8E1454948C51,E:\eicarcom4.zip,E1105070BA828007508566E28A2B8D4C65D192E9EAF3B7868382B7CAE747B397,0, Threat Detected

I am looking for nxlog config to read both the CSV files and send it SIEM in multiple line, i.e means the Headings should show only once for CSV file like the above ones. In my case the headings is showing for each line. Ex: SessionID,File Name,SHA-256,Scan Result,Scan Result Description 50167E52-0380-4191-9DFC-8E1454948C51,E:\eicarcom3.zip,E1105070BA828007508566E28A2B8D4C65D192E9EAF3B7868382B7CAE747B397,0, Threat Detected SessionID,File Name,SHA-256,Scan Result,Scan Result Description 50167E52-0380-4191-9DFC-8E1454948C51,E:\eicarcom3.zip,E1105070BA828007508566E28A2B8D4C65D192E9EAF3B7868382B7CAE747B397,0, Threat Detected

Also in cases the application generates the reports every 1 hour but the data will be empty only the headings in such cases I want ignore such fields which has empty data only with headings.

#Here is my nxconfig #define ROOT C:\Program Files\nxlog define ROOT C:\Program Files (x86)\nxlog

Moduledir %ROOT%\modules CacheDir %ROOT%\data Pidfile %ROOT%\data\nxlog.pid SpoolDir %ROOT%\data LogFile %ROOT%\data\nxlog.log

<Extension _syslog> Module xm_syslog </Extension>

<Extension csv> Module xm_csv Fields $SessionID, $File Name, $SHA-256, $Scan Result, $Scan Result Description FieldTypes string, string, string, string, string Delimiter ; EscapeControl TRUE </Extension>

<Input in1> Module im_file File 'C:\Program Files (x86)\ExportedLogs\\*.csv' Exec csv->parse_csv(); </Input>

<Output out> Module om_udp Host xx.xx.xx.xx Port 514

</Output>

Regards Sunil

Permalink

#2 manuel.munozDeactivated Nxlog ✓ 3 years ago

#1 sunilj777

We have a requirement to read CSV from one folder which is auto generated by the application with a new file name at schedule time say either Ex: 1 hour or 4 hours Following are the fields of first CSV file Example File Name: Tue Aug 25 20-21-49 2020_SessionHistory.CSV SessionID,User ID,Scan Time,Total Files Scanned,Total Files Processed,Total Files Skipped,Total Files Allowed,Total Files Blocked,Device ID,Device Type,Device Manufacturer,Device Model,Device Serial Number 56749F51A4-53CC-4CCF-B606-42334680F890,"Labuser1","08/28/2020 08:24:35",2,2,0,2,0,"USBSTOR\DISK&VEN_SANDISK&PROD_CRUZER_BLADE&REV_1.00\4C530000010112115155&0","USB Device","(Standard disk drives)","SanDisk Cruzer Blade USB Device","4C530000010112115155" 56749F51A4-53CC-4CCF-B606-42334680F890,"Labuser2","08/28/2020 09:24:35",2,2,0,2,0,"USBSTOR\DISK&VEN_SANDISK&PROD_CRUZER_BLADE&REV_1.00\4C530000010112115155&0","USB Device","(Standard disk drives)","SanDisk Cruzer Blade USB Device","4C530000010112115155" 56749F51A4-53CC-4CCF-B606-42334680F890,"Labuser3","08/28/2020 09:24:35",2,2,0,2,0,"USBSTOR\DISK&VEN_SANDISK&PROD_CRUZER_BLADE&REV_1.00\4C530000010112115155&0","USB Device","(Standard disk drives)","SanDisk Cruzer Blade USB Device","4C530000010112115155" Following are the fields of 2nd CSV file SessionID,File Name,SHA-256,Scan Result,Scan Result Description 50167E52-0380-4191-9DFC-8E1454948C50,E:\eicarcom2.zip,E1105070BA828007508566E28A2B8D4C65D192E9EAF3B7868382B7CAE747B397,0, Threat Detected 50167E52-0380-4191-9DFC-8E1454948C51,E:\eicarcom3.zip,E1105070BA828007508566E28A2B8D4C65D192E9EAF3B7868382B7CAE747B397,0, Threat Detected 50167E52-0380-4191-9DFC-8E1454948C51,E:\eicarcom4.zip,E1105070BA828007508566E28A2B8D4C65D192E9EAF3B7868382B7CAE747B397,0, Threat Detected I am looking for nxlog config to read both the CSV files and send it SIEM in multiple line, i.e means the Headings should show only once for CSV file like the above ones. In my case the headings is showing for each line. Ex: SessionID,File Name,SHA-256,Scan Result,Scan Result Description 50167E52-0380-4191-9DFC-8E1454948C51,E:\eicarcom3.zip,E1105070BA828007508566E28A2B8D4C65D192E9EAF3B7868382B7CAE747B397,0, Threat Detected SessionID,File Name,SHA-256,Scan Result,Scan Result Description 50167E52-0380-4191-9DFC-8E1454948C51,E:\eicarcom3.zip,E1105070BA828007508566E28A2B8D4C65D192E9EAF3B7868382B7CAE747B397,0, Threat Detected Also in cases the application generates the reports every 1 hour but the data will be empty only the headings in such cases I want ignore such fields which has empty data only with headings. #Here is my nxconfig #define ROOT C:\Program Files\nxlog define ROOT C:\Program Files (x86)\nxlog Moduledir %ROOT%\modules CacheDir %ROOT%\data Pidfile %ROOT%\data\nxlog.pid SpoolDir %ROOT%\data LogFile %ROOT%\data\nxlog.log <Extension _syslog> Module xm_syslog </Extension> <Extension csv> Module xm_csv Fields $SessionID, $File Name, $SHA-256, $Scan Result, $Scan Result Description FieldTypes string, string, string, string, string Delimiter ; EscapeControl TRUE </Extension> <Input in1> Module im_file File 'C:\Program Files (x86)\ExportedLogs\\*.csv' Exec csv->parse_csv(); </Input> <Output out> Module om_udp Host xx.xx.xx.xx Port 514 </Output> <Route r> Path in1 => out </Route> Regards Sunil

It seems, if you have two different possible formats for a csv file, you will need two xm_csv modules. Inside your input module some logic will be needed in order to decide which xm_csv should parse a particular event. Regarding removing heading lines, please use something like the following, before calling parse_csv().

   if $raw_event =~ /^SessionID/ { drop(); }