Sending apache logs using multiline

Post a different question
4
responses

Hello,

I am trying to send some logs into ELK and I am running into a bit of a snag. The logs are delimited by space and there doesn't seem to be an options to change that easily.

I am not really sure how to go about getting the logs sent to my monitoring solution in a formatted way, preferably JSON.

These are apache error logs:

[Fri May 31 14:21:38 2019] [error] [client 1.1.1.1] File does not exist: /home/test/test.xml, referer: https://www.test-group.com/

NxLog conf:

define REGEX /(?x)^\[\S+\ ([^\]]+)\]\ \[(\S+):(\S+)\]\ (\[client\ (\S+)\]\ )?(.+)$/

<Extension multiline>

Module xm_multiline
HeaderLine %REGEX%

</Extension>

<Input in>

Module im_file
File "C:\\path\\\*.log"
InputType multiline
SavePos FALSE
ReadFromLast FALSE
<Exec>
if $raw_event =~ /(?x)^\[\S+\ ([^\]]+)\]\ \[(\S+):(\S+)\]\ (\[client\ (\S+)\]\ )?(.+)$/
{
$EventTime = parsedate($1);
$ApacheModule = $2;
$ApacheLogLevel = $3;
$Message = $4;
}
</Exec>

It's sending the logs to ELK, but the data isn't in a usable format there. Everything looks just like it does in plain text, no fields or values. If I add the "exec to_json();" option, then I have empty logs in ELK. Maybe something is wrong with my regex, but I copied what I could from the reference manual for this log, though this log is missing data from the example in the guide.

Thanks for your time

AskedJune 3, 2019 - 11:47pm

Answer (1)

The guide is setup to use a specific ErrorLogFormat in your apache config file.
https://nxlog.co/documentation/nxlog-user-guide/apache-http.html#error-log

The example you gave and what I see for error logs also do not span multiple lines, are you sure that you need xm_multiline?

As far as ELK not seeing it proper, we are not seeing your Output directive. What are you sending it as?

AnsweredJune 4, 2019 - 10:31pm

Comments (3)

  • motts's picture

    I wouldn't normally think that I needed to use Multiline, but that's what I found in the manual to use for apache logs. The logs I have seem to not have the PID/SID portion, so I removed that from the code, at least to the best of my ability. The apache logs are only single, long lines, but I imagine the headerline would match every single line. Sorta defeats the real purpose of multiline, but logically it should work if I have things set up correctly.

    This is my output, nothing fancy.

    <Output out>
    Module om_tcp
    Host 1.1.1.1
    Port 5018
    </Output>

    At this time, there is no filtering set up on the ELK side.

    June 5, 2019 - 2:54pm
  • Zhengshi's picture
    (NXLog)

    I think mult-iline is only useful in Apache Tomcat. For the error log, I would disable multi-line and update the REGEX portion to match your output more closely.
    The manual is assuming ErrorLogFormat "[%{u}t] [%-m:%l] [pid %P:tid %T] [client %a] %M" where [%-m:%l] is Module:LogLevel. It seems that you just have log level.

    For reference, here is the example from the manual using the above ErrorLogFormat.

    <Input apache_error>
    Module  im_file
    File    '/var/log/apache2/error.log'
    <Exec>
        if $raw_event =~ /(?x)^\[\S+\ ([^\]]+)\]\ \[(\S+):(\S+)\]\ \[pid\ (\d+):
                          tid\ (\d+)\]\ (\[client\ (\S+)\]\ )?(.+)$/
        {
            $EventTime = parsedate($1);
            $ApacheModule = $2;
            $ApacheLogLevel = $3;
            $ApachePID = $4;
            $ApacheTID = $5;
            if $7 != '' $ClientAddress = $7;
            $Message = $8;
        }
    </Exec>
</Input>

    For your log format, you will likely need something more like :

    <Input apache_error>
    Module  im_file
    File    '/var/log/apache2/error.log'
    <Exec>
        if $raw_event =~ /(?x)^\[\S+\ ([^\]]+)\]\ \[(\S+)\]\s\[client\ (\S+)\]\ ?(.+)$/
        {
            $EventTime = parsedate($1);
            $ApacheLogLevel = $2;
            if $4 != '' $ClientAddress = $3;
            $Message = $4;
        }
    </Exec>
</Input>

    This should give you the fields you need for to_json() to give you some proper output. Check by writing to an om_file output or adding a log_info($raw_event);.

    June 5, 2019 - 5:10pm
  • b0ti's picture
    (NXLog)

    Perhaps the multiline example was for Apache Tomcat. To the best of my knowledge the Apache http server writes both the access.log and error.log in a single-line format.

    June 5, 2019 - 6:12pm