Importing csv file and converting to syslog and sending to log server
Hello all,
I am having an issue reading in a csv file and converting it out up to a log server. The first line/event in the csv gets parsed and converted correctly but then the second line/event doesn't get parsed and is converted to the same line as the first event. I am trying to have it read in the csv file (being exported from sccm for scep alerts) and convert it to syslog and send it up to log server. Please find all my configs below:
NXlog conf (Not pasting full config file)
########################################
# Application Configuration Includes #
########################################
## Uncomment additional input modules below if desired.
## Additional configuration may be required for each application in its conf file.
# include %ROOT%\conf\ms_dhcpv4.conf
## Must add "MS_DHCPv4" as INPUT to route below.
# include %ROOT%\conf\ms_dhcpv6.conf
## Must add "MS_DHCPv6" as INPUT to route below.
# include %ROOT%\conf\ms_scep.conf
## Must add "ms_scep" as INPUT to route below.
include %ROOT%\conf\ms_scep_csv.conf
## Must add "ms_scep_csv" as INPUT to route below.
# include %ROOT%\conf\ms_dns.conf
## Must add "MS_DNS" as INPUT to route below.
# include %ROOT%\conf\ms_exchange15.conf
## Must add "MS_EXCH_MT" as INPUT to route below.
# include %ROOT%\conf\ms_netlogon.conf
## Must add "MS_NETLOGON" as INPUT to route below.
# include %ROOT%\conf\ms_iis.conf
## Must add "MS_IIS" or "MS_FTP" or "MS_SMTP" as INPUT to route below.
########################################
# Output Module Includes #
########################################
## Uncomment additional OUTPUT modules below if desired.
## You MUST configure an IP or Hostname in each output conf file.
include %ROOT%\conf\output_tcp.conf
## Must add "tcp_sender1" as OUTPUT to route below
# include %ROOT%\conf\output_udp.conf
## Must add "udp_sender1" as OUTPUT to route below
# include %ROOT%\conf\output_encrypted.conf
## Must add "ssl_sender1" as OUTPUT to route below
include %ROOT%\conf\output_file.conf
## Must add "file_sender1" as OUTPUT to route below
########################################
# Default Route #
########################################
## Add additional INPUTS comma separated on LEFT of arrow symbol.
## Add additional OUTPUTS comma separated on RIGHT of arrow symbol.
<Route 1>
#Primary route for log processing and forwarding.
Path ms_scep_csv => file_sender1,tcp_sender1
</Route>
###############################################################################
###############################################################################
## DO NOT MODIFY BELOW CONFIGURATIONS UNLESS INSTRUCTED TO DO SO.
########################################
# Global Extensions #
########################################
## Do not modify extensions as they may be required by included configurations.
<Extension _charconv>
Module xm_charconv
AutodetectCharsets iso8859-2, utf-8, utf-16, utf-32
</Extension>
<Extension _syslog>
Module xm_syslog
# IETFTimestampInGMT TRUE
</Extension>
<Extension _json>
Module xm_json
</Extension>
<Extension _exec>
Module xm_exec
</Extension>
ms_scep_csv conf file ########################################################### # INPUT Microsft System Center Endpoint Protection # ###########################################################
## DO NOT MODIFY MODULE NAMES AS IT MAY BREAK TAP FUNCTIONALITY
<Extension csv>
Module xm_csv
Fields $Type, $RowID, $Name, $Description, $Timestamp, $SchemaVersion, $ObserverHost, $ObserverUser, $ObserverProductName, $ObserverProductVersion, $ObserverProtectionType, $ObserverProtectionVersion, $ObserverProtectionSignatureVersion, $ObserverDetection, $ObserverDetectionTime, $ActorHost, $ActorUser, $ActorProcess, $ActorResource, $ActionType, $TargetHost, $TargetUser, $TargetProcess, $TargetResource, $ClassificationID, $ClassificationType, $ClassificationSeverity, $ClassificationCategory, $RemediationType, $RemediationResult, $RemediationErrorCode, $RemediationPendingAction, $IsActiveMalware
Delimiter ,
</Extension>
<Input ms_scep_csv>
Module im_file
File "C:\\Temp\\Desktop.csv"
ReadFromLast TRUE
SavePos TRUE
CloseWhenIdle TRUE
<Exec>
csv->parse_csv();
to_syslog_ietf();
</Exec>
</Input>
Desktop.csv file "Type","RowID","Name","Description","Timestamp","SchemaVersion","ObserverHost","ObserverUser","ObserverProductName","ObserverProductversion","ObserverProtectionType","ObserverProtectionVersion", "ObserverProtectionSignatureVersion","ObserverDetection","ObserverDetectionTime","ActorHost","ActorUser","ActorProcess","ActorResource","ActionType","TargetHost","TargetUser","TargetProcess","TargetRe source","ClassificationID","ClassificationType","ClassificationSeverity","ClassificationCategory","RemediationType","RemediationResult","RemediationErrorCode","RemediationPendingAction","IsActiveMalware" "SecurityIncident","08be9aba-1326-4ac8-81e1- ace5c4550c76","MalwareInfection","NotImplemented","3/3/2020","1.0","Testing","","SystemCenterEndpointProtection","4.10.209.0","AM","","","Realtime","3/3/2020","","","","","MalwareInfection","Testing","NT AUTHORITY\SYSTEM","System","file:_C:\Path\ofw2d3qz.iqf","2147626289","Trojan:Win32/Giframe.A","Severe","Trojan","NoAction","Testing","0","NoActionRequired","Testing 3/3 9:33am" "SecurityIncident","08be9aba-1326-4ac8-81e1- ace5c4550c76","MalwareInfection","NotImplemented","3/3/2020","1.0","Testing","","SystemCenterEndpointProtection","4.10.209.0","AM","","","Realtime","3/3/2020","","","","","MalwareInfection","Testing","NT AUTHORITY\SYSTEM","System","file:_C:\Path\ofw2d3qz.iqf","2147626289","Trojan:Win32/Giframe.A","Severe","Trojan","NoAction","Testing","0","NoActionRequired","Testing 3/3 9:34am" "SecurityIncident","08be9aba-1326-4ac8-81e1- ace5c4550c76","MalwareInfection","NotImplemented","3/3/2020","1.0","Testing","","SystemCenterEndpointProtection","4.10.209.0","AM","","","Realtime","3/3/2020","","","","","MalwareInfection","Testing","NT AUTHORITY\SYSTEM","System","file:_C:\Path\ofw2d3qz.iqf","2147626289","Trojan:Win32/Giframe.A","Severe","Trojan","NoAction","Testing","0","NoActionRequired","Testing 3/3 9:35am" "SecurityIncident","08be9aba-1326-4ac8-81e1- ace5c4550c76","MalwareInfection","NotImplemented","3/3/2020","1.0","Testing","","SystemCenterEndpointProtection","4.10.209.0","AM","","","Realtime","3/3/2020","","","","","MalwareInfection","Testing","NT AUTHORITY\SYSTEM","System","file:_C:\Path\ofw2d3qz.iqf","2147626289","Trojan:Win32/Giframe.A","Severe","Trojan","NoAction","Testing","0","NoActionRequired","Testing 3/3 9:36am"
Testfile.log output <13>1 2020-03-03T10:19:28.428851-08:00 DESKTOP-TVVB676 - - - [NXLOG@14506 EventReceivedTime="2020-03-03 10:19:28" SourceModuleName="ms_scep_csv" SourceModuleType="im_file" Type="SecurityIncident" RowID="08be9aba-1326-4ac8-81e1-ace5c4550c76" Name="MalwareInfection" Description="NotImplemented" Timestamp="3/3/2020" SchemaVersion="1.0" ObserverHost="Testing" ObserverUser="" ObserverProductName="SystemCenterEndpointProtection" ObserverProductVersion="4.10.209.0" ObserverProtectionType="AM" ObserverProtectionVersion="" ObserverProtectionSignatureVersion="" ObserverDetection="Realtime" ObserverDetectionTime="3/3/2020" ActorHost="" ActorUser="" ActorProcess="" ActorResource="" ActionType="MalwareInfection" TargetHost="Testing" TargetUser="NT AUTHORITY\SYSTEM" TargetProcess="System" TargetResource="file:_C:\Path\ofw2d3qz.iqf" ClassificationID="2147626289" ClassificationType="Trojan:Win32/Giframe.A" ClassificationSeverity="Severe" ClassificationCategory="Trojan" RemediationType="NoAction" RemediationResult="Testing" RemediationErrorCode="0" RemediationPendingAction="NoActionRequired" IsActiveMalware="Testing 3/3 9:35am"] "SecurityIncident","08be9aba-1326-4ac8-81e1- ace5c4550c76","MalwareInfection","NotImplemented","3/3/2020","1.0","Testing","","SystemCenterEndpointProtection","4.10.209.0","AM","","","Realtime","3/3/2020","","","","","MalwareInfection","Testing","NT AUTHORITY\SYSTEM","System","file:_C:\Path\ofw2d3qz.iqf","2147626289","Trojan:Win32/Giframe.A","Severe","Trojan","NoAction","Testing","0","NoActionRequired","Testing 3/3 9:35am"
