5
responses

This is so strange, I was looking at the previous questions regarding the type of failure and we're seeing it too, but not related to IIS logs but ftp logs. Here I'll include a sample of the logs and part of the config.

2018-07-25 12:42:02 ERROR if-else failed at line 449, character 236 in C:\Program Files (x86)\nxlog\conf\nxlog.conf. statement execution has been aborted; procedure 'parse_csv' failed at line 449, character 90 in C:\Program Files (x86)\nxlog\conf\nxlog.conf. statement execution has been aborted; Not enough fields in CSV input, expected 12, got 1 in input 'java.lang.NullPointerException'
2018-07-25 12:42:02 ERROR if-else failed at line 449, character 236 in C:\Program Files (x86)\nxlog\conf\nxlog.conf. statement execution has been aborted; procedure 'parse_csv' failed at line 449, character 90 in C:\Program Files (x86)\nxlog\conf\nxlog.conf. statement execution has been aborted; Not enough fields in CSV input, expected 12, got 8 in input '20180725 12:42:01 ERROR Exception Caught in channel 1189859467: '
2018-07-25 12:42:02 ERROR if-else failed at line 449, character 236 in C:\Program Files (x86)\nxlog\conf\nxlog.conf. statement execution has been aborted; procedure 'parse_csv' failed at line 449, character 90 in C:\Program Files (x86)\nxlog\conf\nxlog.conf. statement execution has been aborted; Not enough fields in CSV input, expected 12, got 5 in input 'java.net.ConnectException: Connection refused: connect: /xx.xx.xx.xx:22'

# Windows FTP events log:
<Input FTP_Logs>
Module im_file
File "D:\\GAMFT\\logs\\goanywhere_*"
SavePos TRUE

Exec if $raw_event =~/^#|(\s+at\s+)/ drop();\
else\
{\
w3c_ftp->parse_csv();\
$EventTime = parsedate($date + " " + $time);\
$SourceName = "WINFTP";\
$raw_event = to_json();\
}

the config looks right too me. Suggestions?

AskedJuly 31, 2018 - 4:48pm

Comments (5)

  • nicholasG's picture

    oops, here's the full config.

    #
    # Configuration for converting and sending Windows logs
    # to AlienVault USM Appliance instances.
    #
    # Core features:
    # - only forward specific Windows event IDs
    # - only forward logs collected via WinRM
    # - protect against event storms
    # - transform messages to a "common" CSV format for AV
    #
    # 20140711 - tested for windows 2008r2
    # 20140714 - tested for windows 2012r2
    #

    #
    # Configuration available:
    # - NXLOG
    # - ClAMWIN-NXLOG
    # - DHCP-NXLOG
    # - DNS-NXLOG
    # - EXCHANGE-NXLOG
    # - FTP-NXLOG
    # - IIS-NXLOG
    # - IIS-SMTP-NXLOG
    # - MSSQL-NXLOG
    # - NETWRIX-NXLOG
    # - NPS-NXLOG
    # - SCOM-NXLOG
    # - SYSMON-NXLOG
    # - WINDOWS-FW-NXLOG
    # - WINGFTP-NXLOG
    #

    #
    # Common values:
    #

    define ROOT C:\Program Files (x86)\nxlog
    define LOGFILE %ROOT%\data\nxlog.log

    define OUTPUT_DESTINATION_ADDRESS 10.30.140.20
    define OUTPUT_DESTINATION_PORT 514

    Moduledir %ROOT%\modules
    CacheDir %ROOT%\data
    Pidfile %ROOT%\data\nxlog.pid
    SpoolDir %ROOT%\data
    LogFile %ROOT%\data\nxlog.log

    ##############################################################################################
    #### NXLOG #####
    #### Uncomment the following lines for Windows Security and Application log forwarding #####
    ##############################################################################################

    #
    # Extensions:
    #

    # Support character conversions:
    #<Extension charconv>
    # Module xm_charconv
    #</Extension>

    #
    # Inputs:
    #

    # This nxlog servers heartbeat:
    # MarkInterval defines the interval in minutes of the heartbeat-messages.
    # Mark defines the text which is sent.
    <Input in_nxlog_heartbeat>
    Module im_mark
    MarkInterval 10
    Mark The nxlog service is alive.
    Exec $EventType = 'Application'; $Channel = 'nxlog-ce'; $EventID = 8347;
    </Input>

    # Eventstorm warning message:
    # The module im_internal forwards internal log messages.
    <Input in_nxlog_internal>
    Module im_internal
    Exec if not ($Message == "Eventstorm detected.") drop();
    Exec $EventType = 'Application'; $Channel = 'nxlog-ce'; $EventID = 8103;
    </Input>

    # Windows event log:
    <Input in_windows_events>
    Module im_msvistalog
    SavePos FALSE
    ReadFromLast TRUE

    # Limit the log forwarding to collected events:
    #Query <QueryList> \
    # <Query Id='0' Path='ForwardedEvents'> \
    # <Select Path='ForwardedEvents'>*</Select> \
    # </Query> \
    # </QueryList>

    # Another example for limiting events:
    #Query <QueryList> \
    # <Query Id="0"> \
    # <Select Path="Security">*</Select> \
    # <Select Path="System">*</Select> \
    # <Select Path="Application">*</Select> \
    # <Select Path="SomeOtherPath/XY">*</Select> \
    # </Query> \
    # </QueryList>
    </Input>

    #
    # Transformation:
    #

    # Custom CSV format for nxlog and sysmon-nxlog plugin.
    <Extension transform_alienvault_csv>
    Module xm_csv
    Fields $EventTime, $EventType, $Severity, $Channel, $Hostname, $EventID, $SourceName, $AccountName, $AccountType, $Domain, $Message, $Task, $Category, $Keywords, $UserID, $SeverityValue, $ProviderGuid, $Version, $OpcodeValue, $Opcode, $ActivityID, $RelatedActivityID, $ProcessID, $ThreadID, $RecordNumber
    FieldTypes string, string, string, string, string, string, string, string, string, string, string, string, string, string, string, string, string, string, string, string, string, string, string, string, string
    Delimiter ;
    </Extension>

    #
    # Filters:
    #

    # Match events by Windows event ID.
    # This sets $PatternID in case it matches.
    <Processor match_events>
    Module pm_pattern
    PatternFile %ROOT%\conf\patterndb.xml
    </Processor>

    #
    # Outputs:
    #

    # Process and forward Windows logs:
    <Output out_alienvault_csv>
    Module om_udp
    Host %OUTPUT_DESTINATION_ADDRESS%
    Port %OUTPUT_DESTINATION_PORT%

    # If the EventID doesn't exist in 'patterndb.xml' it gets dropped:
    Exec if not defined $PatternID or not defined $Message { drop(); }
    #Exec if not defined $Message { drop(); }

    # Eventstorm handling:
    # While nxlog processes more than 200 events per second (EPS) it drops all windows logs.
    # If the rate in the next second is lower than 200 EPS it stops dropping.
    # If a new eventstorm is detected it sends a warning and if the storm lasts longer than a minute it sends a warning every minute
    #
    # Variables:
    # rate: this variable is used to count the events and gets reset every second
    # stormed: this variable is '1' if there was an eventstorm a second before else '0'
    # sec: this variable has a lifetime of one second and is used to calculate the rate (EPS)
    # warning: this wariable has a lifetime of 60 seconds and is used to limit the eventstorm-warnings to one per second
    Exec \
    {\
    if not defined get_var('rate') { create_var('rate'); set_var('rate',1); }\
    if not defined get_var('stormed'){ create_var('stormed',2); set_var('stormed',0); set_var('rate',1); }\
    set_var('rate',get_var('rate')+1);\
    if not defined get_var('sec')\
    {\
    create_var('sec',1);\
    set_var('sec',1);\
    if get_var('rate') >= 200 { delete_var('stormed'); create_var('stormed',2); set_var('stormed',1); set_var('rate',1); drop(); } else { set_var('stormed',0); set_var('rate',1); }\
    }\
    else if get_var('stormed') == 1\
    {\
    drop();\
    }\
    if get_var('rate') >= 200\
    {\
    if not defined get_var('warning')\
    {\
    log_warning("Eventstorm detected.");\
    create_var('warning',60);\
    set_var('warning',1);\
    }\
    drop();\
    }\
    }

    # Replace newlines, tabs and carriage returns with blanks:
    Exec $Message = replace($Message, "\t", " "); $Message = replace($Message, "\n", " "); $Message = replace($Message, "\r", " ");

    # Ensure that commonly undefined values are set:
    Exec if not defined $AccountName { $AccountName = "-"; }
    Exec if not defined $AccountType { $AccountType = "-"; }
    Exec if not defined $Domain { $Domain = "-"; }

    # Ensure we send in the proper format:
    Exec transform_alienvault_csv->to_csv(); $raw_event = $Hostname + ' WIN-NXLOG ' + $raw_event + ' Task: ' + $Task + ' Category: '+ $Category + ' Keywords: ' + $Keywords + ' UserID: ' + $UserID + ' Severity: ' + $SeverityValue + ' ProviderGuid: ' + $ProviderGuid + ' Version: ' + $Version + ' OpcodeValue: ' + $OpcodeValue + ' Opcode: ' + $Opcode + ' ActivityID: ' + $ActivityID + ' RelatedActivityID: ' + $RelatedActivityID + ' ProcessID: ' + $ProcessID + ' ThreadID: ' + $ThreadID + ' RecordNumber: ' + $RecordNumber;
    </Output>

    # Output internal nxlog messages:
    <Output out_alienvault_nxlog_csv>
    Module om_udp
    Host %OUTPUT_DESTINATION_ADDRESS%
    Port %OUTPUT_DESTINATION_PORT%

    Exec if not defined $Message { drop(); }

    # Replace newlines, tabs and carriage returns with blanks:
    Exec $Message = replace($Message, "\t", " "); $Message = replace($Message, "\n", " "); $Message = replace($Message, "\r", " ");

    # Ensure that commonly undefined values are set:
    Exec if not defined $AccountName { $AccountName = "-"; }
    Exec if not defined $AccountType { $AccountType = "-"; }
    Exec if not defined $Domain { $Domain = "-"; }

    # Ensure we send in the proper format:
    Exec transform_alienvault_csv->to_csv(); $raw_event = $Hostname + ' WIN-NXLOG ' + $raw_event + ' Task: ' + $Task + ' Category: '+ $Category + ' Keywords: ' + $Keywords + ' UserID: ' + $UserID + ' Severity: ' + $SeverityValue + ' ProviderGuid: ' + $ProviderGuid + ' Version: ' + $Version + ' OpcodeValue: ' + $OpcodeValue + ' Opcode: ' + $Opcode + ' ActivityID: ' + $ActivityID + ' RelatedActivityID: ' + $RelatedActivityID + ' ProcessID: ' + $ProcessID + ' ThreadID: ' + $ThreadID + ' RecordNumber: ' + $RecordNumber;
    </Output>

    #
    # Routes:
    #

    # Route for Windows logs:
    <Route route_windows_logs>
    Path in_windows_events => match_events => out_alienvault_csv
    #Path in_windows_events => out_alienvault_csv
    </Route>

    # Route for internal nxlog messages (eventstorm, heartbeat):
    <Route route_nxlog_messages>
    Path in_nxlog_internal, in_nxlog_heartbeat => out_alienvault_nxlog_csv
    </Route>

    #######################################################################
    #### /NXLOG #####
    #######################################################################

    #######################################################################
    #### CLAMWIN-NXLOG #####
    #### Uncomment the following lines for CLAMWIN log forwarding #####
    #######################################################################

    #<Input CLAMWIN_Logs>
    # Module im_file
    # # REPLACE THE PATH IN THE FOLLOWING INPUT TO THE STORING PATH OF YOUR CLAMWIN LOGS:
    # File "C:\\ProgramData\\.clamwin\\log\\ClamScanLog.txt"
    # InputType LineBased
    # SavePos FALSE

    # Exec $Message = $raw_event;

    # # Replace white spaces
    # Exec $Message = replace($Message, "\t", " "); $Message = replace($Message, "\n", " "); $Message = replace($Message, "\r", " ");
    #</Input>

    ## Output internal CLAMWIN nxlog messages:
    #<Output out_alienvault_CLAMWIN_nxlog>
    # Module om_udp
    # Host %OUTPUT_DESTINATION_ADDRESS%
    # Port %OUTPUT_DESTINATION_PORT%
    # Exec $Hostname = hostname_fqdn();
    # Exec $raw_event =$Hostname + ' CLAM-NXLOG ' + $raw_event;
    #</Output>

    ## Route for CLAMWIN nxlog logs:
    #<Route route_CLAMWIN_nxlog>
    # Path CLAMWIN_Logs => out_alienvault_CLAMWIN_nxlog
    #</Route>
    #######################################################################
    #### /CLAMWIN-NXLOG #####
    #######################################################################

    ######################################################################################################################
    #### DHCP-NXLOG / DNS-NXLOG / FTP-NXLOG / IIS-NXLOG / IIS-SMTP-NXLOG / WINDOWS-FW-NXLOG #####
    #### Uncomment the following lines for DNS, DHCP, FTP, IIS, IIS-SMTP, and/or Windows Firewall log forwarding #####
    ######################################################################################################################
    <Extension json>
    Module xm_json
    </Extension>

    #######################################################################
    #### DHCP-NXLOG #####
    #### Uncomment the following lines for DHCP log forwarding #####
    #######################################################################

    #<Extension transform_alienvault_dhcp_csv>
    #
    # Module xm_csv
    # Fields $EventReceivedTime, $Message
    # FieldTypes string, string
    # Delimiter ;
    #
    #</Extension>

    ## DHCP logs assumed they are located in default location
    ## Use "sysnative" for DHCP Log location for 32-bit applications to access the SYSTEM32 directory on a 64 Bit System
    ## Use "system32" for DHCP Log location on 32 Bit systems
    #<Input DHCP_IN>
    # Module im_file
    # File "C:\\Windows\\Sysnative\\dhcp\\DhcpSrvLog-*.log"
    # SavePos TRUE
    # InputType LineBased
    # Exec if $raw_event =~ /^[0-3][0-9],/\
    # {\
    # $Message = $raw_event;\
    # if $Message =~ s/^00/1000/;\
    # $raw_event = to_json();\
    # }\
    # else\
    # drop();
    #</Input>

    #<Output DHCP_OUT>
    # Module om_udp
    # Host %OUTPUT_DESTINATION_ADDRESS%
    # Port %OUTPUT_DESTINATION_PORT%
    # Exec $Hostname = hostname_fqdn();
    # Exec transform_alienvault_dhcp_csv->to_csv(); $raw_event = $Hostname + ' DHCP-NXLOG: ' + $raw_event;
    #</Output>

    #<Route DHCP>
    # Path DHCP_IN => DHCP_OUT
    #</Route>

    #######################################################################
    #### /DHCP-NXLOG #####
    #######################################################################

    #######################################################################
    #### DNS-NXLOG #####
    #### Uncomment the following lines for DNS log forwarding #####
    #######################################################################

    ## Custom CSV format for the windns-nxlog AlienVault plugin.
    #<Extension transform_alienvault_csv_dns>
    # Module xm_csv
    # Fields $Hostname, $SourceName, $Message
    # FieldTypes string, string, string
    # Delimiter ,
    #</Extension>

    #<Input DNS_Logs>
    # Module im_file
    # File "C:\\Windows\\Sysnative\\dns\\dns.log"
    # SavePos TRUE
    # InputType LineBased

    # Exec if ($raw_event =~ /^#/) OR ($raw_event == '') drop();\
    # else\
    # {\
    # $Message = $raw_event;\
    # $SourceName = "DNS";\
    # $raw_event = to_json();\
    # }
    #</Input>

    #<Output out_alienvault_dns_nxlog>
    # Module om_udp
    # Host %OUTPUT_DESTINATION_ADDRESS%
    # Port %OUTPUT_DESTINATION_PORT%

    # Exec if not defined $Message { drop(); }

    ## Replace newlines, tabs and carriage returns with blanks:
    # Exec $Message = replace($Message, "\t", " "); $Message = replace($Message, "\n", " "); $Message = replace($Message, "\r", " ");

    ## Ensure that commonly undefined values are set:
    # Exec if not defined $AccountName { $AccountName = "-"; }
    # Exec if not defined $AccountType { $AccountType = "-"; }
    # Exec if not defined $Domain { $Domain = "-"; }

    ## Ensure we send in the proper format:
    # Exec $Hostname = hostname_fqdn();
    # Exec transform_alienvault_csv_dns->to_csv(); $raw_event = $Hostname + ' DNS-NXLOG: ' + $raw_event;
    #</Output>

    ## Route for dns nxlog logs:
    #<Route route_dns_nxlog>
    # Path DNS_Logs => out_alienvault_dns_nxlog
    #</Route>
    #######################################################################
    #### /DNS-NXLOG #####
    #######################################################################

    #######################################################################
    #### EXCHANGE-NXLOG #####
    #### Uncomment the following lines for Exchange log forwarding #####
    #######################################################################

    #<Extension syslog>
    # Module xm_syslog
    #</Extension>

    #<Input EXCHANGE_IN>
    # Module im_file
    # # Modify the file path as needed
    # File "C:\\Program Files\\Microsoft\\Exchange Server\\V14\\TransportRoles\\Logs\\MessageTracking\\MSGTRK????????*-*.LOG"
    # SavePos TRUE
    # Exec if $raw_event =~ /HealthMailbox/ drop();
    # Exec if $raw_event =~ /^#/ drop();
    #</Input>

    #<Output EXCHANGE_OUT>
    # Module om_udp
    # Host %OUTPUT_DESTINATION_ADDRESS%
    # Port %OUTPUT_DESTINATION_PORT%
    # Exec $Hostname = hostname_fqdn();
    # Exec $SyslogFacility = 2;
    # Exec $SourceName = 'EXCHANGE-NXLOG';
    # Exec to_syslog_bsd();
    #</Output>

    #<Route EXCHANGE>
    # Path EXCHANGE_IN => EXCHANGE_OUT
    #</Route>

    #######################################################################
    #### /EXCHANGE-NXLOG #####
    #######################################################################

    #######################################################################
    #### FTP-NXLOG #####
    #### Uncomment the following lines for FTP log forwarding #####
    #######################################################################

    <Extension w3c_ftp>
    Module xm_csv
    Fields date, time, cs-username, s-ip, s-port, cs-method, cs-uri-stem, sc-status, sc-win32-status, sc-substatus, x-session, x-fullpath
    FieldTypes string, string, string, string, string, string, string, string, string, string, string, string
    Delimiter ' '
    </Extension>

    # Windows FTP events log:
    <Input FTP_Logs>
    Module im_file
    File "D:\\GAMFT\\logs\\goanywhere_*"
    SavePos TRUE

    Exec if $raw_event =~/^#|(\s+at\s+)/ drop();\
    else\
    {\
    w3c_ftp->parse_csv();\
    $EventTime = parsedate($date + " " + $time);\
    $SourceName = "WINFTP";\
    $raw_event = to_json();\
    }
    </Input>

    # Output internal ftp nxlog messages:
    <Output out_alienvault_ftp_nxlog>
    Module om_udp
    Host %OUTPUT_DESTINATION_ADDRESS%
    Port %OUTPUT_DESTINATION_PORT%
    Exec $Hostname = hostname_fqdn();
    Exec $raw_event =$Hostname + ' WINFTP-NXLOG ' + $raw_event;
    </Output>

    # Route for ftp nxlog logs:
    <Route route_ftp_nxlog>
    Path FTP_Logs => out_alienvault_ftp_nxlog
    </Route>
    #######################################################################
    #### /FTP-NXLOG #####
    #######################################################################

    #######################################################################
    #### IIS-NXLOG #####
    #### Uncomment the following lines for IIS log forwarding #####
    #######################################################################
    <Extension w3c>
    Module xm_csv
    Fields $date, $time, $s_ip, $cs_method, $cs_uri_stem, $cs_uri_query, $s_port, $cs_username, $c_ip, $cs_User_Agent, $cs_Referer, $sc_status, $sc_substatus, $sc_win32_status, $time_taken
    FieldTypes string, string, string, string, string, string, string, string, string, string, string, string, string, string, string
    Delimiter ' '
    </Extension>

    # Windows IIS events log:
    <Input IIS_Logs>
    Module im_file
    File "C:\\inetpub\\logs\\LogFiles\\W3SVC1\\u_ex*"
    SavePos TRUE

    Exec if $raw_event =~/^#/ drop();\
    else\
    {\
    w3c->parse_csv();\
    $EventTime = parsedate($date + " " + $time);\
    $SourceName = "IIS";\
    $raw_event = to_json();\
    }
    </Input>

    # Output internal iis nxlog messages:
    <Output out_alienvault_iis_nxlog>
    Module om_udp
    Host %OUTPUT_DESTINATION_ADDRESS%
    Port %OUTPUT_DESTINATION_PORT%
    Exec $Hostname = hostname_fqdn();
    Exec $raw_event =$Hostname + ' IIS-NXLOG ' + $raw_event;
    </Output>

    # Route for iis nxlog logs:
    <Route route_iis_nxlog>
    Path IIS_Logs => out_alienvault_iis_nxlog
    </Route>
    #######################################################################
    #### /IIS-NXLOG #####
    #######################################################################

    #######################################################################
    #### IIS-SMTP-NXLOG #####
    #### Uncomment the following lines for IIS SMTP log forwarding #####
    #######################################################################

    <Extension w3c_smtp>
    Module xm_csv
    Fields $date, $time, $c-ip, $cs-username, $s-sitename, $s-computername, $s-ip, $s-port, $cs-method, $cs-uri-stem, $cs-uri-query, $sc-status, $sc-win32-status, $sc-bytes, $cs-bytes, $time-taken, $cs-version, $cs-host, $cs(User-Agent), $cs(Cookie), $cs(Referer)
    FieldTypes string, string, string, string, string, string, string, string, string, string, string, string, string, string, string, string, string, string, string, string, string
    Delimiter ' '
    </Extension>

    # Windows IIS SMTP events log:
    <Input IIS_SMTP_Logs>
    Module im_file
    File "C:\\Windows\\System32\\LogFiles\\SmtpSvc1\\ex*"
    SavePos TRUE

    Exec if $raw_event =~/^#/ drop();\
    else\
    {\
    w3c_smtp->parse_csv();\
    $EventTime = parsedate($date + " " + $time);\
    $SourceName = "IIS_SMTP";\
    $raw_event = to_json();\
    }
    </Input>

    # Output internal iis nxlog messages:
    <Output out_alienvault_iis_smtp_nxlog>
    Module om_udp
    Host %OUTPUT_DESTINATION_ADDRESS%
    Port %OUTPUT_DESTINATION_PORT%
    Exec $Hostname = hostname_fqdn();
    Exec $raw_event =$Hostname + ' IIS-NXLOG ' + $raw_event;
    </Output>

    # Route for iis nxlog logs:
    <Route route_iis_smtp_nxlog>
    Path IIS_SMTP_Logs => out_alienvault_iis_smtp_nxlog
    </Route>
    #######################################################################
    #### /IIS-SMTP-NXLOG #####
    #######################################################################

    #######################################################################
    #### MSSQL-NXLOG #####
    #### Uncomment the following lines for MSSQL log forwarding #####
    #######################################################################

    ### NOTE: You must enable SQL Server Audit and send results to the Windows application log

    <Input in_mssql>
    Module im_msvistalog
    SavePos FALSE
    ReadFromLast TRUE

    Query <QueryList> \
    <Query Id="0"> \
    <Select Path="Application">*[System[(EventID='33205')]]</Select>\
    </Query> \
    </QueryList>
    Exec $Message = $raw_event;

    # Finding some values:
    Exec if $raw_event =~ /action_id:(\S+)/ $Action_ID = $1;
    Exec if $raw_event =~ /database_name:(\S+)/ $DataBase = $1;
    Exec if $raw_event =~ /server_instance_name:(\S+)/ $SV_Instace = $1;
    Exec if $raw_event =~ /session_server_principal_name:(\S+)/ $User = $1;
    Exec if $raw_event =~ /AUDIT_SUCCESS/\
    {\
    $Result = 'Success';\
    }\
    else\
    $Result = 'Failure';
    # Replace white spaces
    Exec $Message = replace($Message, "\t", " "); $Message = replace($Message, "\n", " "); $Message = replace($Message, "\r", " ");
    </Input>

    <Output out_mssql>
    Module om_udp
    Host %OUTPUT_DESTINATION_ADDRESS%
    Port %OUTPUT_DESTINATION_PORT%

    # Ensure we send in the proper format:
    Exec $Hostname = hostname_fqdn();
    Exec mssql_csv->to_csv(); $raw_event = $Hostname + ' MSSQL-NXLOG: ' + $raw_event;
    </Output>

    <Extension mssql_csv>
    Module xm_csv
    Fields $Hostname, $SourceName, $Action_ID, $Result, $DataBase, $SV_Instace, $User, $Message
    FieldTypes string, string, string, string, string, string, string, string
    Delimiter ;
    </Extension>

    <Route mssql>
    Path in_mssql => out_mssql
    </Route>

    #######################################################################
    #### /MSSQL-NXLOG #####
    #######################################################################

    ############################################################################
    #### NETWRIX-NXLOG #####
    #### Uncomment the following lines for NETWRIX log forwarding #####
    ############################################################################

    #<Extension transform_alienvault_csv_netwrix>
    # Module xm_csv
    # Fields $EventTime, $EventType, $Severity, $Channel, $Hostname, $EventID, $SourceName, $Task, $Keywords, $Message
    # FieldTypes string, string, string, string, string, string, string, string, string, string
    # Delimiter ;
    #</Extension>

    ## Netwrix events log
    #<Input NETWRIX_Logs>
    # Module im_msvistalog
    # SavePos FALSE
    # ReadFromLast TRUE
    # Query <QueryList> \
    # <Query Id="0"> \
    # <Select Path="Netwrix_Auditor_Integration">*</Select> \
    # </Query> \
    # </QueryList>
    #</Input>

    ## Output internal Netwrix nxlog messages:
    #<Output out_alienvault_netwrix_nxlog>
    # Module om_udp
    # Host %OUTPUT_DESTINATION_ADDRESS%
    # Port %OUTPUT_DESTINATION_PORT%

    ##Replace newlines with ">>"
    # Exec $Message = replace($Message, "\n", ">>");
    #
    # Exec $Hostname = hostname_fqdn();
    # Exec transform_alienvault_csv_netwrix->to_csv(); $raw_event = $Hostname + ' NETWRIX-NXLOG: ' + $raw_event;
    #</Output>

    ## Route for Netwrix nxlog logs:
    #<Route route_netwrix_nxlog>
    # Path NETWRIX_Logs => out_alienvault_netwrix_nxlog
    #</Route>
    #######################################################################
    #### /NETWRIX-NXLOG #####
    #######################################################################

    #######################################################################
    #### NPS-NXLOG #####
    #### Uncomment the following lines for NPS log forwarding #####
    #######################################################################

    #<Extension transform_alienvault_nps>
    # Module xm_nps
    #</Extension>

    ## Assumed NPS logs are located in default location
    #<Input NPS_IN>
    # Module im_file
    # File "C:\\Windows\\System32\\LogFiles\\IN*"
    #
    # Exec if $raw_event =~ /([^,]*,){20}("[^"]*")?([^,]*,){5}(\d+)/ $SID = $4;
    # Exec if $SID =~ /0/ $SID = 1000;
    # Exec parse_nps();
    #</Input>

    #<Output NPS_OUT>
    # Module om_udp
    # Host %OUTPUT_DESTINATION_ADDRESS%
    # Port %OUTPUT_DESTINATION_PORT%
    # Exec $Hostname = hostname_fqdn();
    # Exec to_json();
    # Exec $raw_event = $Hostname + ' NPS-NXLOG: ' + 'SID: ' + $SID + ' ' +$raw_event;
    #</Output>

    #<Route NPS>
    # Path NPS_IN => NPS_OUT
    #</Route>

    #######################################################################
    #### /NPS-NXLOG #####
    #######################################################################

    #######################################################################
    #### SCOM-NXLOG #####
    #### Uncomment the following lines for SCOM log forwarding #####
    #######################################################################

    #<Extension transform_alienvault_csv_scom>
    # Module xm_csv
    # Fields $EventTime, $EventType, $Severity, $Channel, $Hostname, $EventID, $SourceName, $AccountName, $AccountType, $Domain, $Message
    # FieldTypes string, string, string, string, string, string, string, string, string, string, string
    # Delimiter ,
    #</Extension>

    ## Windows SCOM events log:
    #<Input SCOM_Logs>
    # Module im_msvistalog
    # Query <QueryList>\
    # <Query Id="0">\
    # <Select Path="Operations Manager">*</Select>\
    # </Query>\
    # </QueryList>
    #
    # Exec if $raw_event =~ /^#/ drop();\
    # else\
    # {\
    # $Message = $raw_event;\
    # }
    # Exec if $raw_event =~ /User name:\s*(.*?)\s+Session/i $AccountName = $1;
    # Exec if $raw_event =~ /RunAs account\s*(.*?)\s+for/i $AccountName = $1;
    # Exec if $raw_event =~ /CurrentUser=(.*?)\)/i $AccountName = $1;
    #
    #</Input>

    ## Output internal scom nxlog messages:
    #<Output out_alienvault_scom_nxlog>
    # Module om_udp
    # Host %OUTPUT_DESTINATION_ADDRESS%
    # Port %OUTPUT_DESTINATION_PORT%
    #
    # Exec if not defined $Message { drop(); }
    #
    # Exec $Message = replace($Message, "\t", " "); $Message = replace($Message, "\n", " "); $Message = replace($Message, "\r", " ");
    # Exec if not defined $AccountName { $AccountName = "-"; }
    # Exec if not defined $AccountType { $AccountType = "-"; }
    # Exec if not defined $Domain { $Domain = "-"; }
    #
    # Exec $Hostname = hostname_fqdn();
    # Exec transform_alienvault_csv_scom->to_csv(); $raw_event = $Hostname + ' SCOM-NXLOG: ' + $raw_event;
    #</Output>

    ## Route for scom nxlog logs:
    #<Route route_scom_nxlog>
    # Path SCOM_Logs => out_alienvault_scom_nxlog
    #</Route>

    #######################################################################
    #### /SCOM-NXLOG #####
    #######################################################################

    #######################################################################
    #### /SYSMON-NXLOG #####
    #######################################################################

    <Input in_sysmon_events>
    Module im_msvistalog
    SavePos FALSE
    ReadFromLast TRUE

    Query <QueryList>\
    <Query Id="0">\
    <Select Path="Microsoft-Windows-Sysmon/Operational">*</Select>\
    </Query>\
    </QueryList>
    </Input>

    <Output out_sysmon_events>
    Module om_udp
    Host %OUTPUT_DESTINATION_ADDRESS%
    Port %OUTPUT_DESTINATION_PORT%

    # Replace newlines, tabs and carriage returns with blanks:
    Exec $Message = replace($Message, "\t", " "); $Message = replace($Message, "\n", " "); $Message = replace($Message, "\r", " ");

    # Ensure that commonly undefined values are set:
    Exec if not defined $AccountName { $AccountName = "-"; }
    Exec if not defined $AccountType { $AccountType = "-"; }
    Exec if not defined $Domain { $Domain = "-"; }

    # Ensure we send in the proper format:
    Exec transform_alienvault_csv->to_csv(); $raw_event = $Hostname + ' SYSMON-NXLOG ' + $raw_event;
    </Output>

    <Route route_sysmon_logs>
    Path in_sysmon_events => out_sysmon_events
    </Route>

    #######################################################################
    #### /SYSMON-NXLOG #####
    #######################################################################

    ############################################################################
    #### WINDOWS-FW-NXLOG #####
    #### Uncomment the following lines for WINDOWS FIREWALL log forwarding #####
    ############################################################################

    #<Extension transform_alienvault_csv_windows_firewall>
    # Module xm_csv
    # Fields date, time, action, protocol, src-ip, dst-ip, src-port, dst-port, size, tcpflags, tcpsyn, tcpack, tcpwin, icmptype, icmpcode, info, path
    # FieldTypes string, string, string, string, string, string, string, string, string, string, string, string, string, string, string, string, string
    # Delimiter ' '
    #</Extension>

    ## Windows firewall events log:
    #<Input WINDOWS-FW_Logs>
    # Module im_file
    # File "C:\\Windows\\System32\\LogFiles\\Firewall\\pfirewall.log"
    #
    # Exec if $raw_event =~ /^#/ drop();\
    # {\
    # transform_alienvault_csv_windows_firewall->parse_csv();\
    # $EventTime = parsedate($date + " " + $time);\
    # $SourceName = "WINDOWS-FW";\
    # $raw_event = to_json();\
    # }
    #
    #</Input>

    ## Output internal windows firewall nxlog messages:
    #<Output out_alienvault_windows_firewall_nxlog>
    # Module om_udp
    # Host %OUTPUT_DESTINATION_ADDRESS%
    # Port %OUTPUT_DESTINATION_PORT%
    # Exec $Hostname = hostname_fqdn();
    # Exec $raw_event = $Hostname + ' WIN-FW-NXLOG: ' + $raw_event;
    #</Output>

    ## Route for windows firewall nxlog logs:
    #<Route route_windows_fw_nxlog>
    # Path WINDOWS-FW_Logs => out_alienvault_windows_firewall_nxlog
    #</Route>
    #######################################################################
    #### /WINDOWS-FW-NXLOG #####
    #######################################################################

    #######################################################################
    #### WINGFTP-NXLOG #####
    #######################################################################

    #define WING_DOMAIN <EDIT THIS WITH YOUR DOMAIN NAME!>

    #<Input in_wingftp_admin>
    # Module im_file
    # # Modify the file path as needed
    # File "C:\\Program Files (x86)\\Wing FTP Server\\Log\\Admin\\Admin-*.log"
    # SavePos TRUE
    #</Input>

    #<Input in_wingftp_system>
    # Module im_file
    # # Modify the file path as needed
    # File "C:\\Program Files (x86)\\Wing FTP Server\\Log\\System\\System-*.log"
    # SavePos TRUE
    #</Input>

    #<Input in_wingftp_domain>
    # Module im_file
    # # Modify the file path as needed
    # File "C:\\Program Files (x86)\\Wing FTP Server\\Log\\Domains\\%WING_DOMAIN%\\AV-*.log"
    # SavePos TRUE
    #</Input>

    #<Output out_wingftp>
    # Module om_udp
    # Host %OUTPUT_DESTINATION_ADDRESS%
    # Port %OUTPUT_DESTINATION_PORT%

    # # Ensure we send in the proper format:
    # Exec $Hostname = hostname_fqdn();
    # Exec $raw_event = $Hostname + ' WINGFTP-NXLOG: ' + $raw_event;
    #</Output>

    #<Route WINGFTP>
    # Path in_wingftp_admin,in_wingftp_system,in_wingftp_domain => out_wingftp
    #</Route>

    #######################################################################
    #### /WINGFTP-NXLOG #####
    #######################################################################

  • Zhengshi's picture
    (NXLog)

    That is a massive config :)
    I didn't see anything glaring in the config, I believe its like b0ti was eluding to, the source log files "D:\\GAMFT\\logs\\goanywhere_*" are not one single line, and are instead multiline logs. Your CSV is trying to parse for 12 fields separated by spaces and each line does not have the 12 fields it is expecting.

    It would be much easier if you had a sanitized source log example to view as well.

Answers (0)