8
responses

Hello,

I am working on shipping snort logs to an Elastic stack environment. I have found this https://nxlog.co/documentation/nxlog-user-guide.pdf but unfortunately it doesn't seem to help me. It all looks promising on the pdf, but the output does not look like what it shows at the end there. Instead, all of the logs are processed and shipped one line at a time, and as mentioned in that pdf, that is not helpful in this case.

I am using the linux version of the Community Edition and here are sample files:

/var/log/snort/alert:

[**] [1:1000001:1] I saw mommy kissing Santa Clause [**]
[Classification: Generic ICMP event] [Priority: 3]
03/09-15:47:56.187476 src -> dest
ICMP TTL:124 TOS:0x0 ID:16888 IpLen:20 DgmLen:60
Type:8 Code:0 ID:1 Seq:60 ECHO

[**] [1:1000001:1] I saw mommy kissing Santa Clause [**]
[Classification: Generic ICMP event] [Priority: 3]
03/09-15:47:56.187583 src -> dst
ICMP TTL:64 TOS:0x0 ID:62815 IpLen:20 DgmLen:60
Type:0 Code:0 ID:1 Seq:60 ECHO REPLY

/etc/nxlog.conf:

<Extension snort>
Module xm_multiline
HeaderLine /^\[\*\*\] \[\S+] (.*) \[\*\*\]/
Exec if $raw_event =~ /^\s+$/ drop();
</Extension>

<Extension _json>
Module xm_json
</Extension>

<Input in>
Module im_file
File "/var/log/snort/alert"
InputType snort
SavePos FALSE
ReadFromLast FALSE
<Exec>
if $raw_event =~ /(?x)^\[\*\*\]\ \[\S+\]\ (.*)\ \[\*\*\]\s+
(?:\[Classification:\ ([^\]]+)\]\ )?
\[Priority:\ (\d+)\]\s+
(\d\d).(\d\d)\-(\d\d:\d\d:\d\d\.\d+)
\ (\d+.\d+.\d+.\d+):?(\d+)?\ ->
\ (\d+.\d+.\d+.\d+):?(\d+)?\s+\ /

{
$EventName = $1;
$Classification = $2;
$Priority = $3;
$EventTime = parsedate(year(now()) + "-" + $4 + "-" + $5 + " " + $6);
$SourceIPAddress = $7;
$SourcePort = $8;
$DestinationIPAddress = $9;
$DestinationPort = $10;
}

</Exec>

</Input>

<Output out>
Module om_file
File "/root/nxlog/snort"
# Exec to_json();

### This had to be commented out or all the log entries looked like this...{"EventReceivedTime":"2018-03-09 16:14:21","SourceModuleName":"in","SourceModuleType":"im_file"} ###

</Output>

<Route>
Path in => out
</Route>

but the output in /root/nxlog/snort looks just like it did when it went in. there is no separation of any of the data and everything is on the sames lines as it went in. Nothing is in quotes like in the example.

Any help would be great. Thanks!

AskedMarch 9, 2018 - 11:25pm

Answer (1)

I am working on shipping snort logs to an Elastic stack environment.

The user guide has a section on shipping logs to Elasticsearch.

but the output in /root/nxlog/snort looks just like it did when it went in.

vs.

### This had to be commented out or all the log entries looked like this...{"EventReceivedTime":"2018-03-09 16:14:21","SourceModuleName":"in","SourceModuleType":"im_file"} ###

It's because om_file writes to contents of $raw_event. You'll need to format it differently , e.g. use to_json().

Comments (7)

  • motts's picture

    Maybe I am not being clear with this. When I have the to_json part uncommented, all of my log entries look like this:

    {"EventReceivedTime":"2018-03-09 16:19:17","SourceModuleName":"in","SourceModuleType":"im_file"} {"EventReceivedTime":"2018-03-09 16:19:17","SourceModuleName":"in","SourceModuleType":"im_file"} {"EventReceivedTime":"2018-03-09 16:19:18","SourceModuleName":"in","SourceModuleType":"im_file"} {"EventReceivedTime":"2018-03-09 16:19:18","SourceModuleName":"in","SourceModuleType":"im_file"} {"EventReceivedTime":"2018-03-09 16:19:19","SourceModuleName":"in","SourceModuleType":"im_file"} {"EventReceivedTime":"2018-03-09 16:19:19","SourceModuleName":"in","SourceModuleType":"im_file"} {"EventReceivedTime":"2018-03-09 16:19:20","SourceModuleName":"in","SourceModuleType":"im_file"}

    However, the I comment the to_json line, the log entries look exactly as they did when they went in and are shipped one line at a time

    [**] [1:1000001:1] I saw mommy kissing Santa Clause [**] [Classification: Generic ICMP event] [Priority: 3] 03/09-15:47:56.187476 scr -> dest ICMP TTL:124 TOS:0x0 ID:16888 IpLen:20 DgmLen:60 Type:8 Code:0 ID:1 Seq:60 ECHO

    What I got from the PDF is that it should separate things like this

    { "Classification":"Generic ICMP event" "Priority":"3" }

    etc.. which is exactly what I need. I am just not sure why it doesn't seem to be working.

  • b0ti's picture
    (NXLog)

    If you don't see any new fields in the JSON after adding to_json() then your regexp probably does not match and the new fields are not populated.

  • motts's picture

    So I did some testing on the regex and got these results using only this part of the code

     if $raw_event =~ /(?x)^\[\*\*\]\ \[\S+\]\ (.*)\ \[\*\*\]\s+
                    (?:\[Classification:\ ([^\]]+)\]\ )?
                    \[Priority:\ (\d+)\]\s+
                    (\d\d).(\d\d)\-(\d\d:\d\d:\d\d\.\d+)
                    \ (\d+.\d+.\d+.\d+):?(\d+)?\ ->
                    \ (\d+.\d+.\d+.\d+):?(\d+)?\s+\ /
    
    
    Full match  0-149   [**] [1:1000001:1] I saw mommy kissing Santa Clause [**]
    [Classification: Generic ICMP event] [Priority: 3]
    03/13-14:03:38.356247 1.1.1.1 -> 2.2.2.2
    Group 1.    19-51   I saw mommy kissing Santa Clause
    Group 2.    74-92   Generic ICMP event
    Group 3.    105-106 3
    Group 4.    108-110 03
    Group 5.    111-113 13
    Group 6.    114-129 14:03:38.356247
    Group 7.    130-137 1.1.1.1
    Group 9.    141-148 2.2.2.2
    

    However, when I send the data, I still only get the crappy entries with no data in them:

    {"EventReceivedTime":"2018-03-13 16:11:18","SourceModuleName":"in","SourceModuleType":"im_file"}
    {"EventReceivedTime":"2018-03-13 16:11:18","SourceModuleName":"in","SourceModuleType":"im_file"}
    {"EventReceivedTime":"2018-03-13 16:11:18","SourceModuleName":"in","SourceModuleType":"im_file"}
    {"EventReceivedTime":"2018-03-13 16:11:18","SourceModuleName":"in","SourceModuleType":"im_file"}
    {"EventReceivedTime":"2018-03-13 16:11:18","SourceModuleName":"in","SourceModuleType":"im_file"}
    

  • tape's picture

    Hello,

    i think it is still something with the regexp, as nxlog can ship logs to the elasticsearch, and can detect the modules, but without message.

    Peter

  • motts's picture

    Could be. I am not sure on this, so if anyone could offer some assistance, that would be much appreciated.

    I got this regex from this guide. I copied it straight from the guide and into the conf.

    https://nxlog.co/documentation/nxlog-user-guide.pdf

  • b0ti's picture
    (NXLog)

    "Though most of the content applies to all versions of NXLog Community Edition and NXLog Enterprise Edition, this guide was written specifically for NXLog Enterprise Edition version 3.99.3334"

    There might be some difference (e.g indentation in the regexp) but it should also work with the CE pretty much the same way.

    You should try trimming the regexp ato get the first one match and work your way slowly. Also note that this is the pcre flavor if you are using some online regexp matching tool.

    If interested we (NXLog Ltd) do provide technical support services.