Importing csv file and converting to syslog and sending to log server
jbloe812
Hello all,
I am having an issue reading in a csv file and converting it out up to a log server. The first line/event in the csv gets parsed and converted correctly but then the second line/event doesn't get parsed and is converted to the same line as the first event. I am trying to have it read in the csv file (being exported from sccm for scep alerts) and convert it to syslog and send it up to log server. Please find all my configs below:
**NXlog conf (Not pasting full config file)**
########################################
# Application Configuration Includes #
########################################
## Uncomment additional input modules below if desired.
## Additional configuration may be required for each application in its conf file.
# include %ROOT%\conf\ms_dhcpv4.conf
## Must add "MS_DHCPv4" as INPUT to route below.
# include %ROOT%\conf\ms_dhcpv6.conf
## Must add "MS_DHCPv6" as INPUT to route below.
# include %ROOT%\conf\ms_scep.conf
## Must add "ms_scep" as INPUT to route below.
include %ROOT%\conf\ms_scep_csv.conf
## Must add "ms_scep_csv" as INPUT to route below.
# include %ROOT%\conf\ms_dns.conf
## Must add "MS_DNS" as INPUT to route below.
# include %ROOT%\conf\ms_exchange15.conf
## Must add "MS_EXCH_MT" as INPUT to route below.
# include %ROOT%\conf\ms_netlogon.conf
## Must add "MS_NETLOGON" as INPUT to route below.
# include %ROOT%\conf\ms_iis.conf
## Must add "MS_IIS" or "MS_FTP" or "MS_SMTP" as INPUT to route below.
########################################
# Output Module Includes #
########################################
## Uncomment additional OUTPUT modules below if desired.
## You MUST configure an IP or Hostname in each output conf file.
include %ROOT%\conf\output_tcp.conf
## Must add "tcp_sender1" as OUTPUT to route below
# include %ROOT%\conf\output_udp.conf
## Must add "udp_sender1" as OUTPUT to route below
# include %ROOT%\conf\output_encrypted.conf
## Must add "ssl_sender1" as OUTPUT to route below
include %ROOT%\conf\output_file.conf
## Must add "file_sender1" as OUTPUT to route below
########################################
# Default Route #
########################################
## Add additional INPUTS comma separated on LEFT of arrow symbol.
## Add additional OUTPUTS comma separated on RIGHT of arrow symbol.
#Primary route for log processing and forwarding.
Path ms_scep_csv => file_sender1,tcp_sender1
###############################################################################
###############################################################################
## DO NOT MODIFY BELOW CONFIGURATIONS UNLESS INSTRUCTED TO DO SO.
########################################
# Global Extensions #
########################################
## Do not modify extensions as they may be required by included configurations.
Module xm_charconv
AutodetectCharsets iso8859-2, utf-8, utf-16, utf-32
Module xm_syslog
# IETFTimestampInGMT TRUE
Module xm_json
Module xm_exec
**ms_scep_csv conf file**
###########################################################
# INPUT Microsft System Center Endpoint Protection #
###########################################################
## DO NOT MODIFY MODULE NAMES AS IT MAY BREAK TAP FUNCTIONALITY
Module xm_csv
Fields $Type, $RowID, $Name, $Description, $Timestamp, $SchemaVersion, $ObserverHost, $ObserverUser, $ObserverProductName, $ObserverProductVersion, $ObserverProtectionType, $ObserverProtectionVersion, $ObserverProtectionSignatureVersion, $ObserverDetection, $ObserverDetectionTime, $ActorHost, $ActorUser, $ActorProcess, $ActorResource, $ActionType, $TargetHost, $TargetUser, $TargetProcess, $TargetResource, $ClassificationID, $ClassificationType, $ClassificationSeverity, $ClassificationCategory, $RemediationType, $RemediationResult, $RemediationErrorCode, $RemediationPendingAction, $IsActiveMalware
Delimiter ,
Module im_file
File "C:\\Temp\\Desktop.csv"
ReadFromLast TRUE
SavePos TRUE
CloseWhenIdle TRUE
csv->parse_csv();
to_syslog_ietf();
**Desktop.csv file**
"Type","RowID","Name","Description","Timestamp","SchemaVersion","ObserverHost","ObserverUser","ObserverProductName","ObserverProductversion","ObserverProtectionType","ObserverProtectionVersion", "ObserverProtectionSignatureVersion","ObserverDetection","ObserverDetectionTime","ActorHost","ActorUser","ActorProcess","ActorResource","ActionType","TargetHost","TargetUser","TargetProcess","TargetRe source","ClassificationID","ClassificationType","ClassificationSeverity","ClassificationCategory","RemediationType","RemediationResult","RemediationErrorCode","RemediationPendingAction","IsActiveMalware"
"SecurityIncident","08be9aba-1326-4ac8-81e1- ace5c4550c76","MalwareInfection","NotImplemented","3/3/2020","1.0","Testing","","SystemCenterEndpointProtection","4.10.209.0","AM","","","Realtime","3/3/2020","","","","","MalwareInfection","Testing","NT AUTHORITY\SYSTEM","System","file:_C:\Path\ofw2d3qz.iqf","2147626289","Trojan:Win32/Giframe.A","Severe","Trojan","NoAction","Testing","0","NoActionRequired","Testing 3/3 9:33am"
"SecurityIncident","08be9aba-1326-4ac8-81e1- ace5c4550c76","MalwareInfection","NotImplemented","3/3/2020","1.0","Testing","","SystemCenterEndpointProtection","4.10.209.0","AM","","","Realtime","3/3/2020","","","","","MalwareInfection","Testing","NT AUTHORITY\SYSTEM","System","file:_C:\Path\ofw2d3qz.iqf","2147626289","Trojan:Win32/Giframe.A","Severe","Trojan","NoAction","Testing","0","NoActionRequired","Testing 3/3 9:34am"
"SecurityIncident","08be9aba-1326-4ac8-81e1- ace5c4550c76","MalwareInfection","NotImplemented","3/3/2020","1.0","Testing","","SystemCenterEndpointProtection","4.10.209.0","AM","","","Realtime","3/3/2020","","","","","MalwareInfection","Testing","NT AUTHORITY\SYSTEM","System","file:_C:\Path\ofw2d3qz.iqf","2147626289","Trojan:Win32/Giframe.A","Severe","Trojan","NoAction","Testing","0","NoActionRequired","Testing 3/3 9:35am"
"SecurityIncident","08be9aba-1326-4ac8-81e1- ace5c4550c76","MalwareInfection","NotImplemented","3/3/2020","1.0","Testing","","SystemCenterEndpointProtection","4.10.209.0","AM","","","Realtime","3/3/2020","","","","","MalwareInfection","Testing","NT AUTHORITY\SYSTEM","System","file:_C:\Path\ofw2d3qz.iqf","2147626289","Trojan:Win32/Giframe.A","Severe","Trojan","NoAction","Testing","0","NoActionRequired","Testing 3/3 9:36am"
**Testfile.log output**
1 2020-03-03T10:19:28.428851-08:00 DESKTOP-TVVB676 - - - [NXLOG@14506 EventReceivedTime="2020-03-03 10:19:28" SourceModuleName="ms_scep_csv" SourceModuleType="im_file" Type="SecurityIncident" RowID="08be9aba-1326-4ac8-81e1-ace5c4550c76" Name="MalwareInfection" Description="NotImplemented" Timestamp="3/3/2020" SchemaVersion="1.0" ObserverHost="Testing" ObserverUser="" ObserverProductName="SystemCenterEndpointProtection" ObserverProductVersion="4.10.209.0" ObserverProtectionType="AM" ObserverProtectionVersion="" ObserverProtectionSignatureVersion="" ObserverDetection="Realtime" ObserverDetectionTime="3/3/2020" ActorHost="" ActorUser="" ActorProcess="" ActorResource="" ActionType="MalwareInfection" TargetHost="Testing" TargetUser="NT AUTHORITY\\SYSTEM" TargetProcess="System" TargetResource="file:_C:\\Path\\ofw2d3qz.iqf" ClassificationID="2147626289" ClassificationType="Trojan:Win32/Giframe.A" ClassificationSeverity="Severe" ClassificationCategory="Trojan" RemediationType="NoAction" RemediationResult="Testing" RemediationErrorCode="0" RemediationPendingAction="NoActionRequired" IsActiveMalware="Testing 3/3 9:35am"] "SecurityIncident","08be9aba-1326-4ac8-81e1- ace5c4550c76","MalwareInfection","NotImplemented","3/3/2020","1.0","Testing","","SystemCenterEndpointProtection","4.10.209.0","AM","","","Realtime","3/3/2020","","","","","MalwareInfection","Testing","NT AUTHORITY\SYSTEM","System","file:_C:\Path\ofw2d3qz.iqf","2147626289","Trojan:Win32/Giframe.A","Severe","Trojan","NoAction","Testing","0","NoActionRequired","Testing 3/3 9:35am"