1
answer

Hello, I resisted posting here for a while but am finally at a loss to explain what I'm observing. I'm trying to send nginx access logs to graylog, and am mostly using code adapted from the nxlog ce user guide but I haven't been able to get the fields to successfully capture and arrive in graylog.

The nginx server in question logs two sorts of traffic:

  1. x.x.x.x - - [04/Feb/2020:03:23:22 +0000] "GET /" 400 271 "-" "-" "-" - These are status checks from a load balancer which I'm wanting to drop.
  2. x.x.x.x - <username> [04/Feb/2020:03:23:01 +0000] "POST /rest/api/endpoint HTTP/1.1" 201 508 "-" "okhttp/3.3.0" "-" - This is legitimate traffic to the application behind nginx which I want to parse and capture.

I have the following input defined in my config file.

<Input nginx_access>
    Module im_file
    File '/var/log/nginx/access.log'
    PollInterval 1
    SavePos True
    ReadFromLast True
    Recursive False
    RenameCheck False
    <Exec>
        if $raw_event =~ /(?x)^(\S+)\ \S+\ (\S+)\ \[([^\]]+)\]\ \"(\S+)\ (.+)\ HTTP\/\d\.\d\"\ (\S+)\ (\S+)\ \"([^\"]+)\"\ \"([^\"]+)\"\ \"\S+\"$/
        {
            $Hostname = $1;
            if $2 != '-' $AccountName = $2;
            $EventTime = parsedate($3);
            $HTTPMethod = $4;
            $HTTPURL = $5;
            $HTTPResponseStatus = $6;
            if $7 != '-' $FileSize = $7;
            if $8 != '-' $HTTPReferer = $8;
            if $9 != '-' $HTTPUserAgent = $9;
            delete($Message);
        }
        else drop();
    </Exec>
</Input>

If I remove the else drop(); then this config forwards only the load balancer lines that I don't want, which indicates to me that the lines I do want are matching my regexp. But as shown, this configuration forwards nothing.

I also don't think anything is wrong with my outputs or routes since just straight piping this all to graylog in the message field works fine.

If anyone can help me understand what is going on here that would be much appreciated.

Thanks

AskedFebruary 4, 2020 - 6:06pm

Comments (4)

  • mrkey148's picture

    Sure, here is the full file.

    define ROOT /usr/bin
    
    <Extension gelfExt>
      Module xm_gelf
      # Avoid truncation of the short_message field to 64 characters.
      ShortMessageLength 65536
    </Extension>
    
    <Extension syslogExt>
      Module xm_syslog
    </Extension>
    
    User nxlog
    Group nxlog
    
    Moduledir /usr/libexec/nxlog/modules
    CacheDir /var/spool/nxlog/data
    PidFile /var/run/nxlog/nxlog.pid
    LogFile /var/log/nxlog/nxlog.log
    LogLevel INFO
    
    
    <Input jira_log>
        Module im_file
        File '/jira_local/log/atlassian-jira.log'
        PollInterval 1
        SavePos True
        ReadFromLast True
        Recursive False
        RenameCheck False
        Exec $FileName = file_name(); # Send file name with each message
    </Input>
    
    <Input nginx_access>
        Module im_file
        File '/var/log/nginx/access.log'
        PollInterval 1
        SavePos True
        ReadFromLast True
        Recursive False
        RenameCheck False
        <Exec>
            if $raw_event =~ /(?x)^(\S+)\ \S+\ (\S+)\ \[([^\]]+)\]\ \"(\S+)\ (.+)\ HTTP\/\d\.\d\"\ (\S+)\ (\S+)\ \"([^\"]+)\"\ \"([^\"]+)\"\ \"\S+\"$/
            {
                $Hostname = $1;
                if $2 != '-' $AccountName = $2;
                $EventTime = parsedate($3);
                $HTTPMethod = $4;
                $HTTPURL = $5;
                $HTTPResponseStatus = $6;
                if $7 != '-' $FileSize = $7;
                if $8 != '-' $HTTPReferer = $8;
                if $9 != '-' $HTTPUserAgent = $9;
                delete($Message);
            }
            else drop();
        </Exec>
    </Input>
    
    #<Input syslog-udp>
    #   Module im_udp
    #   Host 127.0.0.1
    #   Port 514
    #   Exec parse_syslog_bsd();
    #</Input>
    
    <Output gelf>
        Module om_tcp
        Host graylog-server-redacted
        Port 12201
        OutputType  GELF_TCP
        <Exec>
          # These fields are needed for Graylog
          $gl2_source_collector = '${sidecar.nodeId}';
          $collector_node_id = '${sidecar.nodeName}';
        </Exec>
    </Output>
    
    
    <Route route-1>
      Path jira_log => gelf
    </Route>
    
    <Route route-2>
      Path nginx_access => gelf
    </Route>
    
    #<Route route-2>
    #  Path syslog-udp => gelf
    #</Route>
    

  • Zhengshi's picture
    (NXLog)

    I would verify that you are getting the events you are looking for first, then you can verify if they are being sent.
    First step, in your match, write out your matches to a new file:

    <Exec>
        if $raw_event =~ /(?x)^(\S+)\ \S+\ (\S+)\ \[([^\]]+)\]\ \"(\S+)\ (.+)\ HTTP\/\d\.\d\"\ (\S+)\ (\S+)\ \"([^\"]+)\"\ \"([^\"]+)\"\ \"\S+\"$/
        {
            $Hostname = $1;
            if $2 != '-' $AccountName = $2;
            $EventTime = parsedate($3);
            $HTTPMethod = $4;
            $HTTPURL = $5;
            $HTTPResponseStatus = $6;
            if $7 != '-' $FileSize = $7;
            if $8 != '-' $HTTPReferer = $8;
            if $9 != '-' $HTTPUserAgent = $9;
            delete($Message);
            file_write("/tmp/matches.out", to_json() + "\n");
        }
        else drop();
    </Exec>
    

    Make sure to add xm_json and xm_fileop if you don't have them already.

    Check the file after. Are you getting matches? If so, next step would be make sure it is making it to the wire. Something like the following could show you. The events should come through easy to read.
    tcpdump -vv -x -X 'port 12201' or even more simply: tcpdump -A 'port 12201'

  • mrkey148's picture

    Hmm interesting. After following your directions I see the data I want in both the local output file and in tcpdump. Seems then like my problem must be on the other end with graylog right? This host is already successfully sending logs from the other jira_log input with no issues, so I doubt it's a connection issue between them.

Answer (1)

To close the loop on this; I restarted my graylog server for an unrelated reason, and noticed after that point the logs I wanted started showing up. Still not sure what exactly the issues was, but it was ultimately not with nxlog.