Importing csv file and converting to syslog and sending to log server
Hello all,
I am having an issue reading in a csv file and converting it out up to a log server. The first line/event in the csv gets parsed and converted correctly but then the second line/event doesn't get parsed and is converted to the same line as the first event. I am trying to have it read in the csv file (being exported from sccm for scep alerts) and convert it to syslog and send it up to log server. Please find all my configs below:
NXlog conf (Not pasting full config file)
########################################
# Application Configuration Includes #
########################################
## Uncomment additional input modules below if desired.
## Additional configuration may be required for each application in its conf file.
# include %ROOT%\conf\ms_dhcpv4.conf
## Must add "MS_DHCPv4" as INPUT to route below.
# include %ROOT%\conf\ms_dhcpv6.conf
## Must add "MS_DHCPv6" as INPUT to route below.
# include %ROOT%\conf\ms_scep.conf
## Must add "ms_scep" as INPUT to route below.
include %ROOT%\conf\ms_scep_csv.conf
## Must add "ms_scep_csv" as INPUT to route below.
# include %ROOT%\conf\ms_dns.conf
## Must add "MS_DNS" as INPUT to route below.
# include %ROOT%\conf\ms_exchange15.conf
## Must add "MS_EXCH_MT" as INPUT to route below.
# include %ROOT%\conf\ms_netlogon.conf
## Must add "MS_NETLOGON" as INPUT to route below.
# include %ROOT%\conf\ms_iis.conf
## Must add "MS_IIS" or "MS_FTP" or "MS_SMTP" as INPUT to route below.
########################################
# Output Module Includes #
########################################
## Uncomment additional OUTPUT modules below if desired.
## You MUST configure an IP or Hostname in each output conf file.
include %ROOT%\conf\output_tcp.conf
## Must add "tcp_sender1" as OUTPUT to route below
# include %ROOT%\conf\output_udp.conf
## Must add "udp_sender1" as OUTPUT to route below
# include %ROOT%\conf\output_encrypted.conf
## Must add "ssl_sender1" as OUTPUT to route below
include %ROOT%\conf\output_file.conf
## Must add "file_sender1" as OUTPUT to route below
########################################
# Default Route #
########################################
## Add additional INPUTS comma separated on LEFT of arrow symbol.
## Add additional OUTPUTS comma separated on RIGHT of arrow symbol.
<Route 1>
#Primary route for log processing and forwarding.
Path ms_scep_csv => file_sender1,tcp_sender1
</Route>
###############################################################################
###############################################################################
## DO NOT MODIFY BELOW CONFIGURATIONS UNLESS INSTRUCTED TO DO SO.
########################################
# Global Extensions #
########################################
## Do not modify extensions as they may be required by included configurations.
<Extension _charconv>
Module xm_charconv
AutodetectCharsets iso8859-2, utf-8, utf-16, utf-32
</Extension>
<Extension _syslog>
Module xm_syslog
# IETFTimestampInGMT TRUE
</Extension>
<Extension _json>
Module xm_json
</Extension>
<Extension _exec>
Module xm_exec
</Extension>
ms_scep_csv conf file ########################################################### # INPUT Microsft System Center Endpoint Protection # ###########################################################
## DO NOT MODIFY MODULE NAMES AS IT MAY BREAK TAP FUNCTIONALITY
<Extension csv>
Module xm_csv
Fields $Type, $RowID, $Name, $Description, $Timestamp, $SchemaVersion, $ObserverHost, $ObserverUser, $ObserverProductName, $ObserverProductVersion, $ObserverProtectionType, $ObserverProtectionVersion, $ObserverProtectionSignatureVersion, $ObserverDetection, $ObserverDetectionTime, $ActorHost, $ActorUser, $ActorProcess, $ActorResource, $ActionType, $TargetHost, $TargetUser, $TargetProcess, $TargetResource, $ClassificationID, $ClassificationType, $ClassificationSeverity, $ClassificationCategory, $RemediationType, $RemediationResult, $RemediationErrorCode, $RemediationPendingAction, $IsActiveMalware
Delimiter ,
</Extension>
<Input ms_scep_csv>
Module im_file
File "C:\\Temp\\Desktop.csv"
ReadFromLast TRUE
SavePos TRUE
CloseWhenIdle TRUE
<Exec>
csv->parse_csv();
to_syslog_ietf();
</Exec>
</Input>
Desktop.csv file "Type","RowID","Name","Description","Timestamp","SchemaVersion","ObserverHost","ObserverUser","ObserverProductName","ObserverProductversion","ObserverProtectionType","ObserverProtectionVersion", "ObserverProtectionSignatureVersion","ObserverDetection","ObserverDetectionTime","ActorHost","ActorUser","ActorProcess","ActorResource","ActionType","TargetHost","TargetUser","TargetProcess","TargetRe source","ClassificationID","ClassificationType","ClassificationSeverity","ClassificationCategory","RemediationType","RemediationResult","RemediationErrorCode","RemediationPendingAction","IsActiveMalware" "SecurityIncident","08be9aba-1326-4ac8-81e1- ace5c4550c76","MalwareInfection","NotImplemented","3/3/2020","1.0","Testing","","SystemCenterEndpointProtection","4.10.209.0","AM","","","Realtime","3/3/2020","","","","","MalwareInfection","Testing","NT AUTHORITY\SYSTEM","System","file:_C:\Path\ofw2d3qz.iqf","2147626289","Trojan:Win32/Giframe.A","Severe","Trojan","NoAction","Testing","0","NoActionRequired","Testing 3/3 9:33am" "SecurityIncident","08be9aba-1326-4ac8-81e1- ace5c4550c76","MalwareInfection","NotImplemented","3/3/2020","1.0","Testing","","SystemCenterEndpointProtection","4.10.209.0","AM","","","Realtime","3/3/2020","","","","","MalwareInfection","Testing","NT AUTHORITY\SYSTEM","System","file:_C:\Path\ofw2d3qz.iqf","2147626289","Trojan:Win32/Giframe.A","Severe","Trojan","NoAction","Testing","0","NoActionRequired","Testing 3/3 9:34am" "SecurityIncident","08be9aba-1326-4ac8-81e1- ace5c4550c76","MalwareInfection","NotImplemented","3/3/2020","1.0","Testing","","SystemCenterEndpointProtection","4.10.209.0","AM","","","Realtime","3/3/2020","","","","","MalwareInfection","Testing","NT AUTHORITY\SYSTEM","System","file:_C:\Path\ofw2d3qz.iqf","2147626289","Trojan:Win32/Giframe.A","Severe","Trojan","NoAction","Testing","0","NoActionRequired","Testing 3/3 9:35am" "SecurityIncident","08be9aba-1326-4ac8-81e1- ace5c4550c76","MalwareInfection","NotImplemented","3/3/2020","1.0","Testing","","SystemCenterEndpointProtection","4.10.209.0","AM","","","Realtime","3/3/2020","","","","","MalwareInfection","Testing","NT AUTHORITY\SYSTEM","System","file:_C:\Path\ofw2d3qz.iqf","2147626289","Trojan:Win32/Giframe.A","Severe","Trojan","NoAction","Testing","0","NoActionRequired","Testing 3/3 9:36am"
Testfile.log output <13>1 2020-03-03T10:19:28.428851-08:00 DESKTOP-TVVB676 - - - [NXLOG@14506 EventReceivedTime="2020-03-03 10:19:28" SourceModuleName="ms_scep_csv" SourceModuleType="im_file" Type="SecurityIncident" RowID="08be9aba-1326-4ac8-81e1-ace5c4550c76" Name="MalwareInfection" Description="NotImplemented" Timestamp="3/3/2020" SchemaVersion="1.0" ObserverHost="Testing" ObserverUser="" ObserverProductName="SystemCenterEndpointProtection" ObserverProductVersion="4.10.209.0" ObserverProtectionType="AM" ObserverProtectionVersion="" ObserverProtectionSignatureVersion="" ObserverDetection="Realtime" ObserverDetectionTime="3/3/2020" ActorHost="" ActorUser="" ActorProcess="" ActorResource="" ActionType="MalwareInfection" TargetHost="Testing" TargetUser="NT AUTHORITY\SYSTEM" TargetProcess="System" TargetResource="file:_C:\Path\ofw2d3qz.iqf" ClassificationID="2147626289" ClassificationType="Trojan:Win32/Giframe.A" ClassificationSeverity="Severe" ClassificationCategory="Trojan" RemediationType="NoAction" RemediationResult="Testing" RemediationErrorCode="0" RemediationPendingAction="NoActionRequired" IsActiveMalware="Testing 3/3 9:35am"] "SecurityIncident","08be9aba-1326-4ac8-81e1- ace5c4550c76","MalwareInfection","NotImplemented","3/3/2020","1.0","Testing","","SystemCenterEndpointProtection","4.10.209.0","AM","","","Realtime","3/3/2020","","","","","MalwareInfection","Testing","NT AUTHORITY\SYSTEM","System","file:_C:\Path\ofw2d3qz.iqf","2147626289","Trojan:Win32/Giframe.A","Severe","Trojan","NoAction","Testing","0","NoActionRequired","Testing 3/3 9:35am"
The number of fields in the Fields directive do not match the number in the source data.
The source looks to have about 33 lines of generic text and then it seems there are multiple Incidents on that same line.
It appears as if every Incident is preceded by "SecurityIncident" and includes about 32 fields.
This is unless what you pasted in is supposed to be on 5 lines? (my suspicion)
Top row being the definition with "Type","RowID", ... and the following lines starting with "SecurityIncident" each.
If this is the case then you will want to ignore the definition line by something similar to the Example 323. Collecting W3C Format Logs With xm_csv
example in the NXLog EE Manual.
Before the parse_csv() line it would be something like if $raw_event =~ /^"Type","RowID".*/ drop();
That would leave you with only actual events to parse.
