4
responses

Hello all,

I am having an issue reading in a csv file and converting it out up to a log server. The first line/event in the csv gets parsed and converted correctly but then the second line/event doesn't get parsed and is converted to the same line as the first event. I am trying to have it read in the csv file (being exported from sccm for scep alerts) and convert it to syslog and send it up to log server. Please find all my configs below:

NXlog conf (Not pasting full config file)

########################################
# Application Configuration Includes   #
########################################

## Uncomment additional input modules below if desired.
## Additional configuration may be required for each application in its conf file.

# include %ROOT%\conf\ms_dhcpv4.conf
## Must add "MS_DHCPv4" as INPUT to route below.

# include %ROOT%\conf\ms_dhcpv6.conf
## Must add "MS_DHCPv6" as INPUT to route below.

# include %ROOT%\conf\ms_scep.conf
## Must add "ms_scep" as INPUT to route below.

include %ROOT%\conf\ms_scep_csv.conf
## Must add "ms_scep_csv" as INPUT to route below.

# include %ROOT%\conf\ms_dns.conf
## Must add "MS_DNS" as INPUT to route below.

# include %ROOT%\conf\ms_exchange15.conf
## Must add "MS_EXCH_MT" as INPUT to route below.

# include %ROOT%\conf\ms_netlogon.conf
## Must add "MS_NETLOGON" as INPUT to route below.

# include %ROOT%\conf\ms_iis.conf
## Must add "MS_IIS" or "MS_FTP" or "MS_SMTP" as INPUT to route below.

########################################
# Output Module Includes               #
########################################

## Uncomment additional OUTPUT modules below if desired.
## You MUST configure an IP or Hostname in each output conf file.

include %ROOT%\conf\output_tcp.conf 
## Must add "tcp_sender1" as OUTPUT to route below

# include %ROOT%\conf\output_udp.conf 
## Must add "udp_sender1" as OUTPUT to route below

# include %ROOT%\conf\output_encrypted.conf 
## Must add "ssl_sender1" as OUTPUT to route below

include %ROOT%\conf\output_file.conf 
## Must add "file_sender1" as OUTPUT to route below

########################################
# Default Route                        #
########################################

## Add additional INPUTS comma separated on LEFT of arrow symbol.
## Add additional OUTPUTS comma separated on RIGHT of arrow symbol.

<Route 1>
    #Primary route for log processing and forwarding.
    Path    ms_scep_csv => file_sender1,tcp_sender1
</Route>

###############################################################################
###############################################################################

## DO NOT MODIFY BELOW CONFIGURATIONS UNLESS INSTRUCTED TO DO SO.


########################################
# Global Extensions                    #
########################################

## Do not modify extensions as they may be required by included configurations.

<Extension _charconv>
    Module  xm_charconv
    AutodetectCharsets iso8859-2, utf-8, utf-16, utf-32
</Extension>

<Extension _syslog>
    Module  xm_syslog
#    IETFTimestampInGMT  TRUE
</Extension>

<Extension _json>
    Module  xm_json
</Extension>

<Extension _exec>
    Module  xm_exec
</Extension>

ms_scep_csv conf file ########################################################### # INPUT Microsft System Center Endpoint Protection # ###########################################################

## DO NOT MODIFY MODULE NAMES AS IT MAY BREAK TAP FUNCTIONALITY
<Extension csv>
    Module      xm_csv
    Fields      $Type, $RowID, $Name, $Description, $Timestamp, $SchemaVersion, $ObserverHost, $ObserverUser, $ObserverProductName, $ObserverProductVersion, $ObserverProtectionType,             $ObserverProtectionVersion, $ObserverProtectionSignatureVersion, $ObserverDetection, $ObserverDetectionTime, $ActorHost, $ActorUser, $ActorProcess, $ActorResource, $ActionType, $TargetHost,     $TargetUser, $TargetProcess, $TargetResource, $ClassificationID, $ClassificationType, $ClassificationSeverity, $ClassificationCategory, $RemediationType, $RemediationResult, $RemediationErrorCode, $RemediationPendingAction, $IsActiveMalware
    Delimiter   ,
</Extension>

<Input ms_scep_csv>
    Module im_file
    File "C:\\Temp\\Desktop.csv"
    ReadFromLast TRUE
    SavePos TRUE
    CloseWhenIdle TRUE
    <Exec>
        csv->parse_csv();
        to_syslog_ietf();
    </Exec>
</Input>

Desktop.csv file "Type","RowID","Name","Description","Timestamp","SchemaVersion","ObserverHost","ObserverUser","ObserverProductName","ObserverProductversion","ObserverProtectionType","ObserverProtectionVersion", "ObserverProtectionSignatureVersion","ObserverDetection","ObserverDetectionTime","ActorHost","ActorUser","ActorProcess","ActorResource","ActionType","TargetHost","TargetUser","TargetProcess","TargetRe source","ClassificationID","ClassificationType","ClassificationSeverity","ClassificationCategory","RemediationType","RemediationResult","RemediationErrorCode","RemediationPendingAction","IsActiveMalware" "SecurityIncident","08be9aba-1326-4ac8-81e1- ace5c4550c76","MalwareInfection","NotImplemented","3/3/2020","1.0","Testing","","SystemCenterEndpointProtection","4.10.209.0","AM","","","Realtime","3/3/2020","","","","","MalwareInfection","Testing","NT AUTHORITY\SYSTEM","System","file:_C:\Path\ofw2d3qz.iqf","2147626289","Trojan:Win32/Giframe.A","Severe","Trojan","NoAction","Testing","0","NoActionRequired","Testing 3/3 9:33am" "SecurityIncident","08be9aba-1326-4ac8-81e1- ace5c4550c76","MalwareInfection","NotImplemented","3/3/2020","1.0","Testing","","SystemCenterEndpointProtection","4.10.209.0","AM","","","Realtime","3/3/2020","","","","","MalwareInfection","Testing","NT AUTHORITY\SYSTEM","System","file:_C:\Path\ofw2d3qz.iqf","2147626289","Trojan:Win32/Giframe.A","Severe","Trojan","NoAction","Testing","0","NoActionRequired","Testing 3/3 9:34am" "SecurityIncident","08be9aba-1326-4ac8-81e1- ace5c4550c76","MalwareInfection","NotImplemented","3/3/2020","1.0","Testing","","SystemCenterEndpointProtection","4.10.209.0","AM","","","Realtime","3/3/2020","","","","","MalwareInfection","Testing","NT AUTHORITY\SYSTEM","System","file:_C:\Path\ofw2d3qz.iqf","2147626289","Trojan:Win32/Giframe.A","Severe","Trojan","NoAction","Testing","0","NoActionRequired","Testing 3/3 9:35am" "SecurityIncident","08be9aba-1326-4ac8-81e1- ace5c4550c76","MalwareInfection","NotImplemented","3/3/2020","1.0","Testing","","SystemCenterEndpointProtection","4.10.209.0","AM","","","Realtime","3/3/2020","","","","","MalwareInfection","Testing","NT AUTHORITY\SYSTEM","System","file:_C:\Path\ofw2d3qz.iqf","2147626289","Trojan:Win32/Giframe.A","Severe","Trojan","NoAction","Testing","0","NoActionRequired","Testing 3/3 9:36am"

Testfile.log output <13>1 2020-03-03T10:19:28.428851-08:00 DESKTOP-TVVB676 - - - [NXLOG@14506 EventReceivedTime="2020-03-03 10:19:28" SourceModuleName="ms_scep_csv" SourceModuleType="im_file" Type="SecurityIncident" RowID="08be9aba-1326-4ac8-81e1-ace5c4550c76" Name="MalwareInfection" Description="NotImplemented" Timestamp="3/3/2020" SchemaVersion="1.0" ObserverHost="Testing" ObserverUser="" ObserverProductName="SystemCenterEndpointProtection" ObserverProductVersion="4.10.209.0" ObserverProtectionType="AM" ObserverProtectionVersion="" ObserverProtectionSignatureVersion="" ObserverDetection="Realtime" ObserverDetectionTime="3/3/2020" ActorHost="" ActorUser="" ActorProcess="" ActorResource="" ActionType="MalwareInfection" TargetHost="Testing" TargetUser="NT AUTHORITY\SYSTEM" TargetProcess="System" TargetResource="file:_C:\Path\ofw2d3qz.iqf" ClassificationID="2147626289" ClassificationType="Trojan:Win32/Giframe.A" ClassificationSeverity="Severe" ClassificationCategory="Trojan" RemediationType="NoAction" RemediationResult="Testing" RemediationErrorCode="0" RemediationPendingAction="NoActionRequired" IsActiveMalware="Testing 3/3 9:35am"] "SecurityIncident","08be9aba-1326-4ac8-81e1- ace5c4550c76","MalwareInfection","NotImplemented","3/3/2020","1.0","Testing","","SystemCenterEndpointProtection","4.10.209.0","AM","","","Realtime","3/3/2020","","","","","MalwareInfection","Testing","NT AUTHORITY\SYSTEM","System","file:_C:\Path\ofw2d3qz.iqf","2147626289","Trojan:Win32/Giframe.A","Severe","Trojan","NoAction","Testing","0","NoActionRequired","Testing 3/3 9:35am"

AskedMarch 3, 2020 - 7:57pm

Answer (1)

The number of fields in the Fields directive do not match the number in the source data.
The source looks to have about 33 lines of generic text and then it seems there are multiple Incidents on that same line.
It appears as if every Incident is preceded by "SecurityIncident" and includes about 32 fields.

This is unless what you pasted in is supposed to be on 5 lines? (my suspicion) Top row being the definition with "Type","RowID", ... and the following lines starting with "SecurityIncident" each.
If this is the case then you will want to ignore the definition line by something similar to the Example 323. Collecting W3C Format Logs With xm_csv example in the NXLog EE Manual.
Before the parse_csv() line it would be something like if $raw_event =~ /^"Type","RowID".*/ drop(); That would leave you with only actual events to parse.

Comments (3)

  • jbloe812's picture

    Hi Zhengshi, Thank you for your reply. So I think it might help if I could post the csv from the excel view. Basically the first line, "Type, RowID, Name, etc." are the descriptors for the actual data. So SecurityIncident is defined by Type, MalwareInfection is defined by Name. To explain at an even lower level, Type is the column name and SecurityIncident is under the Type Column. RowID is another Column name and all the gibberish is under that column, Name is another Column name and MalwareInfection is under that column. So the Fields are Type, RowID, Name, etc. with the information after that being the actual alert. So it looks like this:
    Type RowID Name Description Timestamp SchemaVersion
    SecurityIncident <gibberish> MalwareInfection NotImplemented 3/3/2020 9:30 1
    SecurityIncident <gibberish> MalwareInfection NotImplemented 3/3/2020 9:31 1

  • Zhengshi's picture
    (NXLog)

    Ok good. My second scenario seems accurate then. You want to take each line and parse it as CSV while ignoring the first row with code similar to what I posted with the drop() command.

  • jbloe812's picture

    Understood thank you. My main issue is it parses and converts the first line of data correctly but not the second line. I want it to send a log/event for each line of data though like below:
    Type="SecurityIncident" RowID="08be9aba-1326-4ac8-81e1-ace5c4550c76" Name="MalwareInfection" Description="NotImplemented" Timestamp="3/3/2020" SchemaVersion="1.0" ObserverHost="Testing" ObserverUser="" ObserverProductName="SystemCenterEndpointProtection"

    The problem is, for the second line, it just sends the data with no field descriptors, like above, but I need the field descriptors and it to show up on another line instead of all one line in the testfile output so it shows up in the syslog server as two separate events.