2
responses

We have a requirement to read CSV from one folder which is auto generated by the application with a new file name at schedule time say either Ex: 1 hour or 4 hours

Following are the fields of first CSV file
Example File Name: Tue Aug 25 20-21-49 2020_SessionHistory.CSV
SessionID,User ID,Scan Time,Total Files Scanned,Total Files Processed,Total Files Skipped,Total Files Allowed,Total Files Blocked,Device ID,Device Type,Device Manufacturer,Device Model,Device Serial Number
56749F51A4-53CC-4CCF-B606-42334680F890,"Labuser1","08/28/2020 08:24:35",2,2,0,2,0,"USBSTOR\DISK&VEN_SANDISK&PROD_CRUZER_BLADE&REV_1.00\4C530000010112115155&0","USB Device","(Standard disk drives)","SanDisk Cruzer Blade USB Device","4C530000010112115155"
56749F51A4-53CC-4CCF-B606-42334680F890,"Labuser2","08/28/2020 09:24:35",2,2,0,2,0,"USBSTOR\DISK&VEN_SANDISK&PROD_CRUZER_BLADE&REV_1.00\4C530000010112115155&0","USB Device","(Standard disk drives)","SanDisk Cruzer Blade USB Device","4C530000010112115155"
56749F51A4-53CC-4CCF-B606-42334680F890,"Labuser3","08/28/2020 09:24:35",2,2,0,2,0,"USBSTOR\DISK&VEN_SANDISK&PROD_CRUZER_BLADE&REV_1.00\4C530000010112115155&0","USB Device","(Standard disk drives)","SanDisk Cruzer Blade USB Device","4C530000010112115155"

Following are the fields of 2nd CSV file
SessionID,File Name,SHA-256,Scan Result,Scan Result Description
50167E52-0380-4191-9DFC-8E1454948C50,E:\eicarcom2.zip,E1105070BA828007508566E28A2B8D4C65D192E9EAF3B7868382B7CAE747B397,0, Threat Detected
50167E52-0380-4191-9DFC-8E1454948C51,E:\eicarcom3.zip,E1105070BA828007508566E28A2B8D4C65D192E9EAF3B7868382B7CAE747B397,0, Threat Detected
50167E52-0380-4191-9DFC-8E1454948C51,E:\eicarcom4.zip,E1105070BA828007508566E28A2B8D4C65D192E9EAF3B7868382B7CAE747B397,0, Threat Detected

I am looking for nxlog config to read both the CSV files and send it SIEM in multiple line, i.e means the Headings should show only once for CSV file like the above ones. In my case the headings is showing for each line. Ex:
SessionID,File Name,SHA-256,Scan Result,Scan Result Description
50167E52-0380-4191-9DFC-8E1454948C51,E:\eicarcom3.zip,E1105070BA828007508566E28A2B8D4C65D192E9EAF3B7868382B7CAE747B397,0, Threat Detected
SessionID,File Name,SHA-256,Scan Result,Scan Result Description
50167E52-0380-4191-9DFC-8E1454948C51,E:\eicarcom3.zip,E1105070BA828007508566E28A2B8D4C65D192E9EAF3B7868382B7CAE747B397,0, Threat Detected

Also in cases the application generates the reports every 1 hour but the data will be empty only the headings in such cases I want ignore such fields which has empty data only with headings.

#Here is my nxconfig
#define ROOT C:\Program Files\nxlog
define ROOT C:\Program Files (x86)\nxlog

Moduledir %ROOT%\modules
CacheDir %ROOT%\data
Pidfile %ROOT%\data\nxlog.pid
SpoolDir %ROOT%\data
LogFile %ROOT%\data\nxlog.log

<Extension _syslog>
Module xm_syslog
</Extension>

<Extension csv>
Module xm_csv
Fields $SessionID, $File Name, $SHA-256, $Scan Result, $Scan Result Description
FieldTypes string, string, string, string, string
Delimiter ;
EscapeControl TRUE
</Extension>

<Input in1>
Module im_file
File 'C:\Program Files (x86)\ExportedLogs\\\\*.csv'
Exec csv->parse_csv();
</Input>

<Output out>
Module om_udp
Host xx.xx.xx.xx
Port 514

</Output>

<Route r>
Path in1 => out
</Route>

Regards
Sunil

AskedAugust 28, 2020 - 12:29pm

Answer (1)

It seems, if you have two different possible formats for a csv file, you will need two xm_csv modules. Inside your input module some logic will be needed in order to decide which xm_csv should parse a particular event. Regarding removing heading lines, please use something like the following, before calling parse_csv().

   if $raw_event =~ /^SessionID/ { drop(); }

Comments (1)

  • sunilj777's picture

    Hello Manuel,

    Thanks for your quick reply, I tried adding few logics to read both CSV but still I am getting same errors and nxlog was showing configuration errors
    Can you help me with the writing the configuration file as per the requriement.
    It will be of great help.

    Regards
    Sunil