Modify raw_event to send custom logs

Tags: W3C IIS

#1 a.volkov

Hello,

I would like to modify the IIS logs for further transfer to the destination. Now I am parsing the IIS log with the xm_csv module, as in the template. UndefValue is disabled to not get empty. How can I interact with parsed data from w3c_parser? For example, I want to combine into a variable $request = '"' + $cs-method + ' ' + $cs-uri-stem + ' ' + $cs-version + '"'; such a value, but I get an error. When I try to write a field from w3c_parser to $raw_event, I also get an error. Any other data is added without error. For example $raw_event = $c-ip error $raw_event = $EventTime + ' ' + $http_host no error Thank you in advance! Example error, logs and config file below

2022-03-23 16:49:45 WARNING stopping nxlog service 2022-03-23 16:49:45 WARNING nxlog-ce received a termination request signal, exiting... 2022-03-23 16:49:56 ERROR Couldn't parse Exec block at C:\Program Files\nxlog\conf\nxlog.conf:59; couldn't parse statement at line 71, character 32 in C:\Program Files\nxlog\conf\nxlog.conf; syntax error, unexpected +, expecting ( 2022-03-23 16:49:56 ERROR module 'iis_w3c' has configuration errors, not adding to route 'uds_to_file' at C:\Program Files\nxlog\conf\nxlog.conf:84 2022-03-23 16:49:56 ERROR route uds_to_file is not functional without input modules, ignored at C:\Program Files\nxlog\conf\nxlog.conf:84 2022-03-23 16:49:56 WARNING no routes defined! 2022-03-23 16:49:56 WARNING not starting unused module iis_w3c 2022-03-23 16:49:56 WARNING not starting unused module file 2022-03-23 16:49:56 INFO nxlog-ce-3.0.2272 started

Current log format date time s-computername s-ip cs-method cs-uri-stem cs-uri-query s-port cs-username c-ip cs-version cs(User-Agent) cs(Cookie) cs(Referer) cs-host sc-status sc-bytes cs-bytes time-taken 2022-03-23 08:00:01 HOST.DOMAIN 99.XX.XX.4 GET /AnalyticsService - 443 - XX.XX.XX.XXX HTTP/1.1 Zabbix - - site.host.domain 200 3918 144 4

Required log format $http_host $remote_addr $remote_user [$time_local] UNIX-TIME-$msec "$request" $status "$sent_http_content_type" $body_bytes_sent "$http_referer" "$http_user_agent" "$http_cookie" $request_time "$upstream_addr" NGINX-CACHE-$upstream_cache_status "$request_id" "$request_body" host.domain 99.99.99.249 - [11/Mar/2022:20:09:56+0300] UNIX-TIME-1647018596.031 "GET /api/company.php?id=853747 HTTP/1.1" 200 "text/xml; charset=UTF-8" 1455 "-" "-" "20b6b325ea192383cb1244412247c5ea=3002538ef353c9daab4f742176a840; etpsid=f488b343a23d1a4a2332e089a0" 0.059 0.059 "10.10.10.111:80" NGINX-CACHE-- "d0b5ac12cf82671067aa5e6c5c" "-"

Panic Soft #NoFreeOnExit TRUE

define ROOT C:\Program Files\nxlog define CERTDIR %ROOT%\cert define CONFDIR %ROOT%\conf\nxlog.d define LOGDIR %ROOT%\data

define LOGFILE %LOGDIR%\nxlog.log LogFile %LOGFILE%

Moduledir %ROOT%\modules CacheDir %ROOT%\data Pidfile %ROOT%\data\nxlog.pid SpoolDir %ROOT%\data

<Extension _syslog> Module xm_syslog </Extension>

<Extension fileop> Module xm_fileop </Extension>

<Extension _charconv> Module xm_charconv AutodetectCharsets iso8859-2, utf-8, utf-16, utf-32 </Extension>

<Extension _exec> Module xm_exec </Extension>

#Create the parse rule for IIS logs. You can copy these from the header of the IIS log file. <Extension w3c_parser> Module xm_csv Fields $date, $time, $s-computername, $s-ip, $cs-method, $cs-uri-stem, $cs-uri-query, $s-port, $cs-username, $c-ip, $cs-version, $cs(User-Agent), $cs(Cookie), $cs(Referer), $cs-host, $sc-status, $sc-bytes, $cs-bytes, $time-taken FieldTypes string, string, string, string, string, string, string, integer, string, string, string, string, string, string, string, integer, integer, integer, integer Delimiter ' ' EscapeChar '"' QuoteChar '"' EscapeControl FALSE

UndefValue -

</Extension>

<Extension w3c_out> Module xm_csv Fields $http_host, $c-ip, $cs-username, $EventTime1, $sc-status, $Unix FieldTypes string, string, string, string, string, string Delimiter ' '

UndefValue -

QuoteMethod		None

</Extension>

<Input iis_w3c> Module im_file File 'C:\inetpub\logs\LogFiles\W3SVC1\u_ex*.log' SavePos TRUE <Exec> if $raw_event =~ /^#/ drop(); else { w3c_parser->parse_csv(); $EventTime = parsedate($date + " " + $time); $EventTime = $EventTime + (3 * 3600); $EventTime1 = strftime($EventTime, '[%d/%b/%Y:%H:%M:%S]');

$EventTime1 = '$EventTime1' + ' +0003]';

		$Unix = integer($EventTime);
		$Unix = 'UNIX-TIME-' + $Unix;
                    $http_host = &quot;site.host.domain&quot;;

$request = '"' + $cs-method + ' ' + $cs-uri-stem + ' ' + $cs-version + '"';

$request = $cs-method;

		w3c_out-&gt;to_csv();
    }
&lt;/Exec&gt;

</Input>

<Output file> Module om_file File 'C:\inetpub\logs\LogFiles\Parser\w3c.txt' </Output>

<Route uds_to_file> Path iis_w3c => file </Route>

#2 a.volkov
#1 a.volkov
Hello, I would like to modify the IIS logs for further transfer to the destination. Now I am parsing the IIS log with the xm_csv module, as in the template. UndefValue is disabled to not get empty. How can I interact with parsed data from w3c_parser? For example, I want to combine into a variable $request = '"' + $cs-method + ' ' + $cs-uri-stem + ' ' + $cs-version + '"'; such a value, but I get an error. When I try to write a field from w3c_parser to $raw_event, I also get an error. Any other data is added without error. For example $raw_event = $c-ip error $raw_event = $EventTime + ' ' + $http_host no error Thank you in advance! Example error, logs and config file below 2022-03-23 16:49:45 WARNING stopping nxlog service 2022-03-23 16:49:45 WARNING nxlog-ce received a termination request signal, exiting... 2022-03-23 16:49:56 ERROR Couldn't parse Exec block at C:\Program Files\nxlog\conf\nxlog.conf:59; couldn't parse statement at line 71, character 32 in C:\Program Files\nxlog\conf\nxlog.conf; syntax error, unexpected +, expecting ( 2022-03-23 16:49:56 ERROR module 'iis_w3c' has configuration errors, not adding to route 'uds_to_file' at C:\Program Files\nxlog\conf\nxlog.conf:84 2022-03-23 16:49:56 ERROR route uds_to_file is not functional without input modules, ignored at C:\Program Files\nxlog\conf\nxlog.conf:84 2022-03-23 16:49:56 WARNING no routes defined! 2022-03-23 16:49:56 WARNING not starting unused module iis_w3c 2022-03-23 16:49:56 WARNING not starting unused module file 2022-03-23 16:49:56 INFO nxlog-ce-3.0.2272 started Current log format date time s-computername s-ip cs-method cs-uri-stem cs-uri-query s-port cs-username c-ip cs-version cs(User-Agent) cs(Cookie) cs(Referer) cs-host sc-status sc-bytes cs-bytes time-taken 2022-03-23 08:00:01 HOST.DOMAIN 99.XX.XX.4 GET /AnalyticsService - 443 - XX.XX.XX.XXX HTTP/1.1 Zabbix - - site.host.domain 200 3918 144 4 Required log format $http_host $remote_addr $remote_user [$time_local] UNIX-TIME-$msec "$request" $status "$sent_http_content_type" $body_bytes_sent "$http_referer" "$http_user_agent" "$http_cookie" $request_time "$upstream_addr" NGINX-CACHE-$upstream_cache_status "$request_id" "$request_body" host.domain 99.99.99.249 - [11/Mar/2022:20:09:56+0300] UNIX-TIME-1647018596.031 "GET /api/company.php?id=853747 HTTP/1.1" 200 "text/xml; charset=UTF-8" 1455 "-" "-" "20b6b325ea192383cb1244412247c5ea=3002538ef353c9daab4f742176a840; etpsid=f488b343a23d1a4a2332e089a0" 0.059 0.059 "10.10.10.111:80" NGINX-CACHE-- "d0b5ac12cf82671067aa5e6c5c" "-" Panic Soft #NoFreeOnExit TRUE define ROOT C:\Program Files\nxlog define CERTDIR %ROOT%\cert define CONFDIR %ROOT%\conf\nxlog.d define LOGDIR %ROOT%\data define LOGFILE %LOGDIR%\nxlog.log LogFile %LOGFILE% Moduledir %ROOT%\modules CacheDir %ROOT%\data Pidfile %ROOT%\data\nxlog.pid SpoolDir %ROOT%\data <Extension _syslog> Module xm_syslog </Extension> <Extension fileop> Module xm_fileop </Extension> <Extension _charconv> Module xm_charconv AutodetectCharsets iso8859-2, utf-8, utf-16, utf-32 </Extension> <Extension _exec> Module xm_exec </Extension> #Create the parse rule for IIS logs. You can copy these from the header of the IIS log file. <Extension w3c_parser> Module xm_csv Fields $date, $time, $s-computername, $s-ip, $cs-method, $cs-uri-stem, $cs-uri-query, $s-port, $cs-username, $c-ip, $cs-version, $cs(User-Agent), $cs(Cookie), $cs(Referer), $cs-host, $sc-status, $sc-bytes, $cs-bytes, $time-taken FieldTypes string, string, string, string, string, string, string, integer, string, string, string, string, string, string, string, integer, integer, integer, integer Delimiter ' ' EscapeChar '"' QuoteChar '"' EscapeControl FALSE UndefValue - </Extension> <Extension w3c_out> Module xm_csv Fields $http_host, $c-ip, $cs-username, $EventTime1, $sc-status, $Unix FieldTypes string, string, string, string, string, string Delimiter ' ' UndefValue - QuoteMethod None </Extension> <Input iis_w3c> Module im_file File 'C:\inetpub\logs\LogFiles\W3SVC1\u_ex*.log' SavePos TRUE <Exec> if $raw_event =~ /^#/ drop(); else { w3c_parser->parse_csv(); $EventTime = parsedate($date + " " + $time); $EventTime = $EventTime + (3 * 3600); $EventTime1 = strftime($EventTime, '[%d/%b/%Y:%H:%M:%S]'); $EventTime1 = '$EventTime1' + ' +0003]'; $Unix = integer($EventTime); $Unix = 'UNIX-TIME-' + $Unix; $http_host = &quot;site.host.domain&quot;; $request = '"' + $cs-method + ' ' + $cs-uri-stem + ' ' + $cs-version + '"'; $request = $cs-method; w3c_out-&gt;to_csv(); } &lt;/Exec&gt; </Input> <Output file> Module om_file File 'C:\inetpub\logs\LogFiles\Parser\w3c.txt' </Output> <Route uds_to_file> Path iis_w3c => file </Route>

the "-" character cannot be used in a variable. Now everything works.