12
responses

Hi All,

I have previously got smtp logs to go into Graylog using NXlog, it was worknig fine. I then had a disk sapce issue on the graylog host so had to redo some bits, including the nxlog.conf for our SMTP server. 

The SMTP log header specifes the following

#Software: Microsoft Internet Information Services 8.5
#Version: 1.0
#Date: 2017-03-26 23:00:10
#Fields: date time c-ip cs-username s-sitename s-computername s-port cs-method cs-uri-query sc-bytes time-taken 

This is the error I am getting

ERROR if-else failed at line 44, character 436 in C:\Program Files (x86)\nxlog\conf\nxlog.conf. statement execution has been aborted; procedure 'parse_csv' failed at line 44, character 224 in C:\Program Files (x86)\nxlog\conf\nxlog.conf. statement execution has been aborted; Not enough fields in CSV input, expected 11, got 8 in input

I ahve checked and rechecked and their should be 11 items as per the .conf.

define ROOT C:\Program Files (x86)\nxlog

Moduledir %ROOT%\modules
CacheDir %ROOT%\data
Pidfile %ROOT%\data\nxlog.pid
SpoolDir %ROOT%\data
LogFile %ROOT%\data\nxlog.log

<Extension gelf>
    Module         xm_gelf
</Extension>

<Extension fileop>
    Module         xm_fileop
</Extension>

 <Extension json>
    Module      xm_json
</Extension>

# Create the parse rule for IIS logs. You can copy these from the header of the IIS log file.
<Extension w3c>
    Module             xm_csv
    Fields             $date, $time, $c-ip, $cs-username, $s-sitename, $s-computername, $s-port, $cs-method, $cs-uri-query, $sc-bytes, $time-taken
    FieldTypes         string, string, string, string, string, string, integer, string, string, integer, integer
    Delimiter         ' '
    QuoteChar         '"'
    EscapeControl     FALSE
    UndefValue         '-'
</Extension>

<Input smtp>
    Module        im_file
    File        "C:\\Logs\\SMTPSVC1\\\ex*.log"
    SavePos      TRUE

    Exec        if $raw_event =~ /^#/ drop();                    \
                else                                             \
                {                                                \
                    w3c->parse_csv();                            \
                    $EventTime = parsedate($date + " " + $time); \
                    $SourceName = "smtp";                         \
                    $Message = to_json();                         \
                }
</Input>

<Input eventlog>
    Module      im_msvistalog
</Input>

<Output graylog>
    Module      om_udp
    Host        graylog.mydomain.com
    Port        12201
    OutputType    GELF

    #Use the following line for debugging (uncomment the fileop extension above as well)
    #Exec file_write("C:\\Program Files (x86)\\nxlog\\data\\nxlog_output.log", $raw_event);
</Output>

<Route eventlog>
   Path        eventlog => graylog
</Route>

<Route smtp-to-graylog>
    Path        smtp => graylog
</Route>

Its so frustracting that I know this was working correctly.

Any help would be great.

Thanks

AskedMarch 27, 2017 - 12:46pm

Comments (4)

  • Mr_M_Cox's picture

    Here is a bigger chunk of the input log i am using with domains and senstive data replaced.

    2017-03-27 11:54:03 195.130.217.241 OutboundConnectionResponse SMTPSVC1 WF15 25 - 250+Sender+OK 13 31
    2017-03-27 11:54:03 195.130.217.241 OutboundConnectionCommand SMTPSVC1 WF15 25 RCPT TO:<bob@hope.com> 4 31
    2017-03-27 11:54:03 192.168.10.48 WF14 SMTPSVC1 WF15 0 QUIT WF14 72 0
    2017-03-27 11:54:03 192.168.10.48 WF14 SMTPSVC1 WF15 0 EHLO +WF14 202 0
    2017-03-27 11:54:03 192.168.10.48 WF14 SMTPSVC1 WF15 0 MAIL +FROM:<no-reply@mydomain.com> 51 0
    2017-03-27 11:54:03 192.168.10.48 WF14 SMTPSVC1 WF15 0 RCPT +TO:<bob@hope.com> 40 0
    2017-03-27 11:54:03 192.168.10.48 WF14 SMTPSVC1 WF15 0 DATA <WF15f1aAsNBosCXkdX600002156@wf15.mydomain.co.uk> 136 0
    2017-03-27 11:54:03 192.168.10.48 WF14 SMTPSVC1 WF15 0 QUIT WF14 72 0
    2017-03-27 11:54:03 195.130.217.241 OutboundConnectionResponse SMTPSVC1 WF15 25 - 250+Recipient+OK 16 172
    2017-03-27 11:54:03 195.130.217.241 OutboundConnectionCommand SMTPSVC1 WF15 25 DATA - 4 172
    2017-03-27 11:54:03 195.130.217.241 OutboundConnectionResponse SMTPSVC1 WF15 25 - 354+Start+mail+data,+end+with+CRLF.CRLF 39 188
    2017-03-27 11:54:05 195.130.217.241 OutboundConnectionResponse SMTPSVC1 WF15 25 - 250+SmtpThread-6944127-1490615645939@uk-mta-52.uk.mimecast.lan+Received+OK 74 2125
    2017-03-27 11:54:05 195.130.217.241 OutboundConnectionCommand SMTPSVC1 WF15 25 RSET - 4 2125
    2017-03-27 11:54:05 195.130.217.241 OutboundConnectionResponse SMTPSVC1 WF15 25 - 250+Transaction+Reset+OK 24 2125
    2017-03-27 11:54:05 195.130.217.241 OutboundConnectionCommand SMTPSVC1 WF15 25 MAIL FROM:<no-reply@mydomain.com> 4 2125
    2017-03-27 11:54:05 195.130.217.241 OutboundConnectionResponse SMTPSVC1 WF15 25 - 250+Sender+OK 13 2141
    2017-03-27 11:54:05 195.130.217.241 OutboundConnectionCommand SMTPSVC1 WF15 25 RCPT TO:<bob@hope.com> 4 2141
    2017-03-27 11:54:05 195.130.217.241 OutboundConnectionResponse SMTPSVC1 WF15 25 - 250+Recipient+OK 16 2188
    2017-03-27 11:54:05 195.130.217.241 OutboundConnectionCommand SMTPSVC1 WF15 25 DATA - 4 2188
    2017-03-27 11:54:05 195.130.217.241 OutboundConnectionResponse SMTPSVC1 WF15 25 - 354+Start+mail+data,+end+with+CRLF.CRLF 39 2188
    2017-03-27 11:54:06 192.168.10.48 WF14 SMTPSVC1 WF15 0 EHLO +WF14 202 0
    2017-03-27 11:54:06 192.168.10.48 WF14 SMTPSVC1 WF15 0 MAIL +FROM:<no-reply@mydomain.com> 51 0
    2017-03-27 11:54:06 192.168.10.48 WF14 SMTPSVC1 WF15 0 RCPT +TO:<john@smith.co.uk> 42 0
    2017-03-27 11:54:06 192.168.10.48 WF14 SMTPSVC1 WF15 0 DATA <WF15mkqlZ4kcltA05Bq00002157@wf15.mydomain.co.uk> 136 0
    2017-03-27 11:54:06 192.168.10.48 WF14 SMTPSVC1 WF15 0 QUIT WF14 72 0

  • Mr_M_Cox's picture

    Any further help or advice with this would be great.

    At the moment i am getting the odd log through but its not assigning the fields correctly and there are way too few logs, should be hundreds, not 6.

    I know this was working and I cant see what is different this time other than having to get the graylog server back up and running following an issue with the journal.

  • b0ti's picture
    (NXLog)

    We provide commercial support, feel free to reach out. In either case a reproducible test case with input sample and config file will be needed.

  • Mr_M_Cox's picture

    Hey all,

    As you can see this is a fairly long and cofusing thread as I have probably answered the queries in the wrong place making it hard to read and follow but I really do still need any help you can offer.

    the bottom line of my problem is that NXlog appears to be cutting off the lines of log and not inputing and parsing it correctly meaning it either never reaches Graylog or when it does the fields are all wrong.

    Please, if you have any suggestions please let me know.

    Thanks. 

Answer (1)

> I ahve checked and rechecked and their should be 11 items as per the .conf.

It got 8 as per the error message you quoted:

> Not enough fields in CSV input, expected 11, got 8 in input

Comments (7)

  • Mr_M_Cox's picture

    Is i not 11 as per these lines?

      Fields             $date, $time, $c-ip, $cs-username, $s-sitename, $s-computername, $s-port, $cs-method, $cs-uri-query, $sc-bytes, $time-taken
        FieldTypes         string, string, string, string, string, string, integer, string, string, integer, integer

  • b0ti's picture
    (NXLog)

    Here is the clue:

    > Not enough fields in CSV input, expected 11, got 8 in input 'mand SMTPSVC1 WF15 25 EHLO wf15.mydomain.co.uk 4 94'

    For some reason it ends up reading a line from the middle of it.

    This looks like a possible bug. Might be worth testing the NXLog EE to see if this has been fixed there (and not yet backported into the CE). The NXLog EE also has a module called xm_w3c that can automatically parse W3C data without the need to set up the fields manually.

  • Mr_M_Cox's picture

    I have jsut requested a trial of the enterprise edition and will let you know if that fixes it.

    what is odd is that I have had it working n the past, only a few days ago.

  • Mr_M_Cox's picture

    Just re read your comment and i see what you mean, in that the problem is that only 8 were passed. It looks like it is cutting off on the log files but I dont know why. For example.

    2017-03-27 12:44:42 INFO nxlog-ce-2.9.1716 started
    2017-03-27 12:45:50 ERROR procedure 'parse_csv' failed at line 44, character 154 in C:\Program Files (x86)\nxlog\conf\nxlog.conf. statement execution has been aborted; Not enough fields in CSV input, expected 11, got 8 in input 'mand SMTPSVC1 WF15 25 EHLO wf15.mydomain.co.uk 4 94'

    The above error appears to be when processing a line such as...

    2017-03-27 11:54:10 192.168.1.34 OutboundConnectionCommand SMTPSVC1 WF15 25 MAIL FROM:<no-reply@mydomain.com>+SIZE=636 4 47