Help me understand why regexp captured fields are not being forwarded.

View thread

mrkey148

Hello, I resisted posting here for a while but am finally at a loss to explain what I'm observing. I'm trying to send nginx access logs to graylog, and am mostly using code adapted from the nxlog ce user guide but I haven't been able to get the fields to successfully capture and arrive in graylog.

The nginx server in question logs two sorts of traffic:

  1. x.x.x.x - - [04/Feb/2020:03:23:22 +0000] "GET /" 400 271 "-" "-" "-" - These are status checks from a load balancer which I'm wanting to drop.
  2. x.x.x.x - <username> [04/Feb/2020:03:23:01 +0000] "POST /rest/api/endpoint HTTP/1.1" 201 508 "-" "okhttp/3.3.0" "-" - This is legitimate traffic to the application behind nginx which I want to parse and capture.

I have the following input defined in my config file.

<Input nginx_access>
	Module im_file
	File '/var/log/nginx/access.log'
	PollInterval 1
	SavePos	True
	ReadFromLast True
	Recursive False
	RenameCheck False
	<Exec>
        if $raw_event =~ /(?x)^(\S+)\ \S+\ (\S+)\ \[([^\]]+)\]\ \"(\S+)\ (.+)\ HTTP\/\d\.\d\"\ (\S+)\ (\S+)\ \"([^\"]+)\"\ \"([^\"]+)\"\ \"\S+\"$/
        {
            $Hostname = $1;
            if $2 != '-' $AccountName = $2;
            $EventTime = parsedate($3);
            $HTTPMethod = $4;
            $HTTPURL = $5;
            $HTTPResponseStatus = $6;
            if $7 != '-' $FileSize = $7;
            if $8 != '-' $HTTPReferer = $8;
            if $9 != '-' $HTTPUserAgent = $9;
            delete($Message);
        }
        else drop();
	</Exec>
</Input>

If I remove the else drop(); then this config forwards only the load balancer lines that I don't want, which indicates to me that the lines I do want are matching my regexp. But as shown, this configuration forwards nothing.

I also don't think anything is wrong with my outputs or routes since just straight piping this all to graylog in the message field works fine.

If anyone can help me understand what is going on here that would be much appreciated.

Thanks