Modify raw_event to send custom logs
Hello,
I would like to modify the IIS logs for further transfer to the destination. Now I am parsing the IIS log with the xm_csv module, as in the template. UndefValue is disabled to not get empty. How can I interact with parsed data from w3c_parser? For example, I want to combine into a variable $request = '"' + $cs-method + ' ' + $cs-uri-stem + ' ' + $cs-version + '"'; such a value, but I get an error. When I try to write a field from w3c_parser to $raw_event, I also get an error. Any other data is added without error. For example $raw_event = $c-ip error $raw_event = $EventTime + ' ' + $http_host no error Thank you in advance! Example error, logs and config file below
2022-03-23 16:49:45 WARNING stopping nxlog service 2022-03-23 16:49:45 WARNING nxlog-ce received a termination request signal, exiting... 2022-03-23 16:49:56 ERROR Couldn't parse Exec block at C:\Program Files\nxlog\conf\nxlog.conf:59; couldn't parse statement at line 71, character 32 in C:\Program Files\nxlog\conf\nxlog.conf; syntax error, unexpected +, expecting ( 2022-03-23 16:49:56 ERROR module 'iis_w3c' has configuration errors, not adding to route 'uds_to_file' at C:\Program Files\nxlog\conf\nxlog.conf:84 2022-03-23 16:49:56 ERROR route uds_to_file is not functional without input modules, ignored at C:\Program Files\nxlog\conf\nxlog.conf:84 2022-03-23 16:49:56 WARNING no routes defined! 2022-03-23 16:49:56 WARNING not starting unused module iis_w3c 2022-03-23 16:49:56 WARNING not starting unused module file 2022-03-23 16:49:56 INFO nxlog-ce-3.0.2272 started
Current log format date time s-computername s-ip cs-method cs-uri-stem cs-uri-query s-port cs-username c-ip cs-version cs(User-Agent) cs(Cookie) cs(Referer) cs-host sc-status sc-bytes cs-bytes time-taken 2022-03-23 08:00:01 HOST.DOMAIN 99.XX.XX.4 GET /AnalyticsService - 443 - XX.XX.XX.XXX HTTP/1.1 Zabbix - - site.host.domain 200 3918 144 4
Required log format $http_host $remote_addr $remote_user [$time_local] UNIX-TIME-$msec "$request" $status "$sent_http_content_type" $body_bytes_sent "$http_referer" "$http_user_agent" "$http_cookie" $request_time "$upstream_addr" NGINX-CACHE-$upstream_cache_status "$request_id" "$request_body" host.domain 99.99.99.249 - [11/Mar/2022:20:09:56+0300] UNIX-TIME-1647018596.031 "GET /api/company.php?id=853747 HTTP/1.1" 200 "text/xml; charset=UTF-8" 1455 "-" "-" "20b6b325ea192383cb1244412247c5ea=3002538ef353c9daab4f742176a840; etpsid=f488b343a23d1a4a2332e089a0" 0.059 0.059 "10.10.10.111:80" NGINX-CACHE-- "d0b5ac12cf82671067aa5e6c5c" "-"
Panic Soft #NoFreeOnExit TRUE
define ROOT C:\Program Files\nxlog define CERTDIR %ROOT%\cert define CONFDIR %ROOT%\conf\nxlog.d define LOGDIR %ROOT%\data
define LOGFILE %LOGDIR%\nxlog.log LogFile %LOGFILE%
Moduledir %ROOT%\modules CacheDir %ROOT%\data Pidfile %ROOT%\data\nxlog.pid SpoolDir %ROOT%\data
<Extension _syslog> Module xm_syslog </Extension>
<Extension fileop> Module xm_fileop </Extension>
<Extension _charconv> Module xm_charconv AutodetectCharsets iso8859-2, utf-8, utf-16, utf-32 </Extension>
<Extension _exec> Module xm_exec </Extension>
#Create the parse rule for IIS logs. You can copy these from the header of the IIS log file. <Extension w3c_parser> Module xm_csv Fields $date, $time, $s-computername, $s-ip, $cs-method, $cs-uri-stem, $cs-uri-query, $s-port, $cs-username, $c-ip, $cs-version, $cs(User-Agent), $cs(Cookie), $cs(Referer), $cs-host, $sc-status, $sc-bytes, $cs-bytes, $time-taken FieldTypes string, string, string, string, string, string, string, integer, string, string, string, string, string, string, string, integer, integer, integer, integer Delimiter ' ' EscapeChar '"' QuoteChar '"' EscapeControl FALSE
UndefValue -
</Extension>
<Extension w3c_out> Module xm_csv Fields $http_host, $c-ip, $cs-username, $EventTime1, $sc-status, $Unix FieldTypes string, string, string, string, string, string Delimiter ' '
UndefValue -
QuoteMethod None
</Extension>
<Input iis_w3c> Module im_file File 'C:\inetpub\logs\LogFiles\W3SVC1\u_ex*.log' SavePos TRUE <Exec> if $raw_event =~ /^#/ drop(); else { w3c_parser->parse_csv(); $EventTime = parsedate($date + " " + $time); $EventTime = $EventTime + (3 * 3600); $EventTime1 = strftime($EventTime, '[%d/%b/%Y:%H:%M:%S]');
$EventTime1 = '$EventTime1' + ' +0003]';
$Unix = integer($EventTime);
$Unix = 'UNIX-TIME-' + $Unix;
$http_host = "site.host.domain";
$request = '"' + $cs-method + ' ' + $cs-uri-stem + ' ' + $cs-version + '"';
$request = $cs-method;
w3c_out->to_csv();
}
</Exec>
</Input>
<Output file> Module om_file File 'C:\inetpub\logs\LogFiles\Parser\w3c.txt' </Output>
<Route uds_to_file> Path iis_w3c => file </Route>
the "-" character cannot be used in a variable. Now everything works.