Sending apache logs using multiline

Tags:

#1 motts

Hello,

I am trying to send some logs into ELK and I am running into a bit of a snag. The logs are delimited by space and there doesn't seem to be an options to change that easily.

I am not really sure how to go about getting the logs sent to my monitoring solution in a formatted way, preferably JSON.

These are apache error logs:

[Fri May 31 14:21:38 2019] [error] [client 1.1.1.1] File does not exist: /home/test/test.xml, referer: https://www.test-group.com/

NxLog conf:

define REGEX /(?x)^[\S+\ ([^]]+)]\ [(\S+):(\S+)]\ ([client\ (\S+)]\ )?(.+)$/

<Extension multiline>

Module xm_multiline
HeaderLine %REGEX%

</Extension>

<Input in>

Module          im_file
File            &quot;C:\\path\\\*.log&quot;
InputType       multiline
SavePos         FALSE
ReadFromLast    FALSE
&lt;Exec&gt;
    if $raw_event =~ /(?x)^\[\S+\ ([^\]]+)\]\ \[(\S+):(\S+)\]\ (\[client\ (\S+)\]\ )?(.+)$/
    {
        $EventTime = parsedate($1);
        $ApacheModule = $2;
        $ApacheLogLevel = $3;
        $Message = $4;
    }
&lt;/Exec&gt;

It's sending the logs to ELK, but the data isn't in a usable format there. Everything looks just like it does in plain text, no fields or values. If I add the "exec to_json();" option, then I have empty logs in ELK. Maybe something is wrong with my regex, but I copied what I could from the reference manual for this log, though this log is missing data from the example in the guide.

Thanks for your time

#2 Zhengshi
#1 motts
Hello, I am trying to send some logs into ELK and I am running into a bit of a snag. The logs are delimited by space and there doesn't seem to be an options to change that easily. I am not really sure how to go about getting the logs sent to my monitoring solution in a formatted way, preferably JSON. These are apache error logs: [Fri May 31 14:21:38 2019] [error] [client 1.1.1.1] File does not exist: /home/test/test.xml, referer: https://www.test-group.com/ NxLog conf: define REGEX /(?x)^[\S+\ ([^]]+)]\ [(\S+):(\S+)]\ ([client\ (\S+)]\ )?(.+)$/ <Extension multiline> Module xm_multiline HeaderLine %REGEX% </Extension> <Input in> Module im_file File &quot;C:\\path\\\*.log&quot; InputType multiline SavePos FALSE ReadFromLast FALSE &lt;Exec&gt; if $raw_event =~ /(?x)^\[\S+\ ([^\]]+)\]\ \[(\S+):(\S+)\]\ (\[client\ (\S+)\]\ )?(.+)$/ { $EventTime = parsedate($1); $ApacheModule = $2; $ApacheLogLevel = $3; $Message = $4; } &lt;/Exec&gt; It's sending the logs to ELK, but the data isn't in a usable format there. Everything looks just like it does in plain text, no fields or values. If I add the "exec to_json();" option, then I have empty logs in ELK. Maybe something is wrong with my regex, but I copied what I could from the reference manual for this log, though this log is missing data from the example in the guide. Thanks for your time

The guide is setup to use a specific ErrorLogFormat in your apache config file.
https://nxlog.co/documentation/nxlog-user-guide/apache-http.html#error-log

The example you gave and what I see for error logs also do not span multiple lines, are you sure that you need xm_multiline?

As far as ELK not seeing it proper, we are not seeing your Output directive. What are you sending it as?