responses
Hello,
I am trying to send some logs into ELK and I am running into a bit of a snag. The logs are delimited by space and there doesn't seem to be an options to change that easily.
I am not really sure how to go about getting the logs sent to my monitoring solution in a formatted way, preferably JSON.
These are apache error logs:
[Fri May 31 14:21:38 2019] [error] [client 1.1.1.1] File does not exist: /home/test/test.xml, referer: https://www.test-group.com/
NxLog conf:
define REGEX /(?x)^\[\S+\ ([^\]]+)\]\ \[(\S+):(\S+)\]\ (\[client\ (\S+)\]\ )?(.+)$/
<Extension multiline>
Module xm_multiline
HeaderLine %REGEX%
</Extension>
<Input in>
Module im_file
File "C:\\path\\\*.log"
InputType multiline
SavePos FALSE
ReadFromLast FALSE
<Exec>
if $raw_event =~ /(?x)^\[\S+\ ([^\]]+)\]\ \[(\S+):(\S+)\]\ (\[client\ (\S+)\]\ )?(.+)$/
{
$EventTime = parsedate($1);
$ApacheModule = $2;
$ApacheLogLevel = $3;
$Message = $4;
}
</Exec>
It's sending the logs to ELK, but the data isn't in a usable format there. Everything looks just like it does in plain text, no fields or values. If I add the "exec to_json();" option, then I have empty logs in ELK. Maybe something is wrong with my regex, but I copied what I could from the reference manual for this log, though this log is missing data from the example in the guide.
Thanks for your time
Comments (3)
I wouldn't normally think that I needed to use Multiline, but that's what I found in the manual to use for apache logs. The logs I have seem to not have the PID/SID portion, so I removed that from the code, at least to the best of my ability. The apache logs are only single, long lines, but I imagine the headerline would match every single line. Sorta defeats the real purpose of multiline, but logically it should work if I have things set up correctly.
This is my output, nothing fancy.
<Output out>
Module om_tcp
Host 1.1.1.1
Port 5018
</Output>
At this time, there is no filtering set up on the ELK side.
I think mult-iline is only useful in Apache Tomcat. For the error log, I would disable multi-line and update the REGEX portion to match your output more closely.
The manual is assuming
ErrorLogFormat "[%{u}t] [%-m:%l] [pid %P:tid %T] [client %a] %M"
where[%-m:%l]
is Module:LogLevel. It seems that you just have log level.For reference, here is the example from the manual using the above
ErrorLogFormat
.For your log format, you will likely need something more like :
This should give you the fields you need for
to_json()
to give you some proper output. Check by writing to anom_file
output or adding alog_info($raw_event);
.Perhaps the multiline example was for Apache Tomcat. To the best of my knowledge the Apache http server writes both the access.log and error.log in a single-line format.