5
responses

Hello everyone,

As you can see from my NXlog config below, I am pulling the Linux auth logs in and passing them to our SIEM platform. The auth logs work fine, however as you can also see I am trying to pull in a CSV file that is located on the Linux box. Each individual event is surrounded by curly brackets e.g: {'event}, so I would like to just pull each event into a field called message and then parse these messages on our SIEM platform. I can't seem to get this to work whatsoever, the TCP connection initiates, but NXlog doesn't pickup the CSV file. Any ideas?

 

Cheers

G

########################################
# Global directives                    #
########################################
User nxlog
Group nxlog
LogFile /var/log/nxlog/nxlog.log
LogLevel INFO
########################################
# Modules                              #
########################################
<Extension _syslog>
Module  xm_syslog
</Extension>

<Input auth_logs>
        Module  im_file
        File    "/var/log/auth.log"
        SavePos TRUE
        ReadFromLast    TRUE
</Input>

<Output to_relay>
        Module  om_tcp
        Host    127.0.0.1
        Port    20009
        OutputType      LineBased
</Output>

########################################
# Routes                               #
########################################
<Route 1>
        Path    auth_logs => to_relay
</Route>

<Extension csv1>
        Module  xm_csv
        Fields  $Message
        Delimiter       '{'
</Extension>

<Input filein>
        Module  im_file
        File    "/etc/ingest/sucuri/sucuri.csv"
        Exec    csv1->parse_csv();
</Input>

<Output test>
        Module  om_tcp
        Host    127.0.0.1
        Port    20002
        OutputType      Binary
</Output>

<Route 2>
        Path    filein => test
</Route>

AskedOctober 9, 2017 - 11:17am

Answer (1)

I doubt that the delimiter is `{` , this sounds rather like JSON to me.

Unfortunately without a sample it is hard to tell and give advice.

Comments (4)

  • multiplierx's picture

    Here is some of the CSV below:

    As you can see there is a small amount of info at the start of the file, I do not want this picking up. As for the delimiter, it is ',' however I thought I could use { as then it would pick up each event in its entirety so as then it can be parsed in Graylog.

    Cheers,

     

    {"status":1,"action":"audit_trails","messages":[],"request_time":1507216447,"output":{"total_lines":137,"access_logs":[{"is_usable":0},{"is_usable":1,"checksum":"67f5245beb7edb115e99e9e429d7cdecf982d5db","remote_addr":"x.x.x.x","remote_hostname":"","remote_logname":"-","remote_user":"-","request_date":"03\/Oct\/2017","request_time":"02:43:11","request_timezone":"-0400","request_timestamp":1507012991,"request_method":"POST","resource_path":"\/wp-admin\/admin-ajax.php","http_protocol":"HTTP\/1.1","http_status":"403","http_status_title":"Forbidden","http_bytes_sent":"2456","http_referer":"-","http_user_agent":"Mozilla\/5.0 (X11; Linux x86_64) AppleWebKit\/537.36 (KHTML, like Gecko) Chrome\/50.0.2661.75 Safari\/537.36","sucuri_is_allowed":true,"sucuri_is_blocked":false,"sucuri_block_title":"PROXYBLOCKID","sucuri_block_code":"FBP007","sucuri_block_reason":"","request_country_name":"United States","request_country_code":"us","request_country_flag":"<img src=\"\/static\/images\/blank.png\" title=\"United States\" alt=\"United States\" class=\"flag flag-us\" \/>","geo_location":{"continent_code":"NA","country_code":"US","country_code3":"USA","country_name":"United States","region":"CA","city":"San Francisco","postal_code":"94124","latitude":37.735298156738,"longitude":-122.37319946289,"dma_code":807,"area_code":415}},{"is_usable":1,"checksum":"1d2a4712d3af801c9ebc80f773b1dbb4de8b0267","remote_addr":"x.x.x.x","remote_hostname":"","remote_logname":"-","remote_user":"-","request_date":"03\/Oct\/2017","request_time":"04:22:49","request_timezone":"-0400","request_timestamp":1507018969,"request_method":"GET","resource_path":"\/administrator","http_protocol":"HTTP\/1.1","http_status":"403","http_status_title":"Forbidden","http_bytes_sent":"2512","http_referer":"-","http_user_agent":"Mozilla\/5.0 (Windows NT 6.1; WOW64) AppleWebKit\/537.36 (KHTML, like Gecko) Chrome\/36.0.1985.125 Safari\/537.36","sucuri_is_allowed":false,"sucuri_is_blocked":true,"sucuri_block_title":"PROXYBLOCKID","sucuri_block_code":"IPB17","sucuri_block_reason":"IP Address not whitelisted","request_country_name":"United States","request_country_code":"us","request_country_flag":"<img src=\"\/static\/images\/blank.png\" title=\"United States\" alt=\"United States\" class=\"flag flag-us\" \/>","geo_location":{"continent_code":"NA","country_code":"US","country_code3":"USA","country_name":"United States","region":"CA","city":"San Francisco","postal_code":"94119","latitude":37.774898529053,"longitude":-122.41940307617,"dma_code":807,"area_code":415}}

  • b0ti's picture
    (NXLog)

    If you want to ship the logs unparsed, just remove the CSV (it is not csv). All you need is this:

    Exec $Message = $raw_event;

    With that each line will be sent as a separate event record to Graylog.

    Othewise use xm_json and parse_json();

  • multiplierx's picture

    Hi Boti, I am pretty sure that this is CSV, when looking at the file in VI, the entire thing is held on one line.

    Should I parse each field then?

     

    Cheers,

     

    G

  • multiplierx's picture

    The data i'm trying to pull with NXlog is definitely CSV, is there anyway to pull each event outof the long continous string or do I have to parse into each field?

     

    cheers,

    G