Hello,

I would like to modify the IIS logs for further transfer to the destination. Now I am parsing the IIS log with the xm_csv module, as in the template. UndefValue is disabled to not get empty. How can I interact with parsed data from w3c_parser? For example, I want to combine into a variable $request = '"' + $cs-method + ' ' + $cs-uri-stem + ' ' + $cs-version + '"'; such a value, but I get an error. When I try to write a field from w3c_parser to $raw_event, I also get an error. Any other data is added without error. For example $raw_event = $c-ip error $raw_event = $EventTime + ' ' + $http_host no error Thank you in advance! Example error, logs and config file below

2022-03-23 16:49:45 WARNING stopping nxlog service 2022-03-23 16:49:45 WARNING nxlog-ce received a termination request signal, exiting... 2022-03-23 16:49:56 ERROR Couldn't parse Exec block at C:\Program Files\nxlog\conf\nxlog.conf:59; couldn't parse statement at line 71, character 32 in C:\Program Files\nxlog\conf\nxlog.conf; syntax error, unexpected +, expecting ( 2022-03-23 16:49:56 ERROR module 'iis_w3c' has configuration errors, not adding to route 'uds_to_file' at C:\Program Files\nxlog\conf\nxlog.conf:84 2022-03-23 16:49:56 ERROR route uds_to_file is not functional without input modules, ignored at C:\Program Files\nxlog\conf\nxlog.conf:84 2022-03-23 16:49:56 WARNING no routes defined! 2022-03-23 16:49:56 WARNING not starting unused module iis_w3c 2022-03-23 16:49:56 WARNING not starting unused module file 2022-03-23 16:49:56 INFO nxlog-ce-3.0.2272 started

Current log format date time s-computername s-ip cs-method cs-uri-stem cs-uri-query s-port cs-username c-ip cs-version cs(User-Agent) cs(Cookie) cs(Referer) cs-host sc-status sc-bytes cs-bytes time-taken 2022-03-23 08:00:01 HOST.DOMAIN 99.XX.XX.4 GET /AnalyticsService - 443 - XX.XX.XX.XXX HTTP/1.1 Zabbix - - site.host.domain 200 3918 144 4

Required log format $http_host $remote_addr $remote_user [$time_local] UNIX-TIME-$msec "$request" $status "$sent_http_content_type" $body_bytes_sent "$http_referer" "$http_user_agent" "$http_cookie" $request_time "$upstream_addr" NGINX-CACHE-$upstream_cache_status "$request_id" "$request_body" host.domain 99.99.99.249 - [11/Mar/2022:20:09:56+0300] UNIX-TIME-1647018596.031 "GET /api/company.php?id=853747 HTTP/1.1" 200 "text/xml; charset=UTF-8" 1455 "-" "-" "20b6b325ea192383cb1244412247c5ea=3002538ef353c9daab4f742176a840; etpsid=f488b343a23d1a4a2332e089a0" 0.059 0.059 "10.10.10.111:80" NGINX-CACHE-- "d0b5ac12cf82671067aa5e6c5c" "-"

Panic Soft #NoFreeOnExit TRUE

define ROOT C:\Program Files\nxlog define CERTDIR %ROOT%\cert define CONFDIR %ROOT%\conf\nxlog.d define LOGDIR %ROOT%\data

define LOGFILE %LOGDIR%\nxlog.log LogFile %LOGFILE%

Moduledir %ROOT%\modules CacheDir %ROOT%\data Pidfile %ROOT%\data\nxlog.pid SpoolDir %ROOT%\data

<Extension _syslog> Module xm_syslog </Extension>

<Extension fileop> Module xm_fileop </Extension>

<Extension _charconv> Module xm_charconv AutodetectCharsets iso8859-2, utf-8, utf-16, utf-32 </Extension>

<Extension _exec> Module xm_exec </Extension>

#Create the parse rule for IIS logs. You can copy these from the header of the IIS log file. <Extension w3c_parser> Module xm_csv Fields $date, $time, $s-computername, $s-ip, $cs-method, $cs-uri-stem, $cs-uri-query, $s-port, $cs-username, $c-ip, $cs-version, $cs(User-Agent), $cs(Cookie), $cs(Referer), $cs-host, $sc-status, $sc-bytes, $cs-bytes, $time-taken FieldTypes string, string, string, string, string, string, string, integer, string, string, string, string, string, string, string, integer, integer, integer, integer Delimiter ' ' EscapeChar '"' QuoteChar '"' EscapeControl FALSE

UndefValue -

</Extension>

<Extension w3c_out> Module xm_csv Fields $http_host, $c-ip, $cs-username, $EventTime1, $sc-status, $Unix FieldTypes string, string, string, string, string, string Delimiter ' '

UndefValue -

QuoteMethod		None

</Extension>

<Input iis_w3c> Module im_file File 'C:\inetpub\logs\LogFiles\W3SVC1\u_ex*.log' SavePos TRUE <Exec> if $raw_event =~ /^#/ drop(); else { w3c_parser->parse_csv(); $EventTime = parsedate($date + " " + $time); $EventTime = $EventTime + (3 * 3600); $EventTime1 = strftime($EventTime, '[%d/%b/%Y:%H:%M:%S]');

$EventTime1 = '$EventTime1' + ' +0003]';

		$Unix = integer($EventTime);
		$Unix = 'UNIX-TIME-' + $Unix;
                    $http_host = "site.host.domain";

$request = '"' + $cs-method + ' ' + $cs-uri-stem + ' ' + $cs-version + '"';

$request = $cs-method;

		w3c_out->to_csv();
    }
</Exec>

</Input>

<Output file> Module om_file File 'C:\inetpub\logs\LogFiles\Parser\w3c.txt' </Output>

<Route uds_to_file> Path iis_w3c => file </Route>

I want to know is possible send, with the iis log message, a custom field and value. Actually this is my config: ``` define ROOT C:\Program Files (x86)\nxlog define CERTDIR %ROOT%\cert define CONFDIR %ROOT%\conf define LOGDIR %ROOT%\data define LOGFILE %LOGDIR%\nxlog.log LogFile %LOGFILE% Moduledir %ROOT%\modules CacheDir %ROOT%\data Pidfile %ROOT%\data\nxlog.pid SpoolDir %ROOT%\data #Module xm_syslog Module xm_gelf #Module xm_json Module xm_csv Fields date, time, s-ip, cs-method, cs-uri-stem, cs-uri-query, \ s-port, cs-username, c-ip, cs(User-Agent), cs(Referer), \ sc-status, sc-substatus, sc-win32-status, time-taken FieldTypes string, string, string, string, string, string, integer, \ string, string, string, string, integer, integer, integer, \ integer Delimiter ' ' EscapeChar '"' QuoteChar '"' EscapeControl FALSE UndefValue - Module im_file File "c:\\logs\\app\\iislog\\u_extend1.log" if $raw_event =~ /^#/ drop(); else { w3c_parser->parse_csv(); $EventTime = parsedate($date + "T" + $time + ".000Z"); } Module om_udp Host 172.28.36.25 Port 12201 #Exec to_syslog_snare(); OutputType GELF Path in2 => out ``` This work great but i need to add a custom and static value in the message, like : FIELD: "system" , VALUE: "program1". and this value is always the same ``` date, time, s-ip, cs-method, cs-uri-stem, cs-uri-query, s-port, cs-username, c-ip, cs(User-Agent), cs(Referer), sc-status, sc-substatus, sc-win32-status, time-taken,**system** ``` Is this possible? Thanks!

I'm using NXLog to sending Windows events and IIS logs to Loggly. We've recently onboarded a new MSSP and they have asked us to check off all IIS logging fields. This seems to break parsing of IIS logs that need to be sent to Loggly. I've contact Loggly support and they can't seem to come to a resolution.

Below is the code that we had been using for Loggly previously.

This is a sample NXLog configuration file created by Loggly. June 2013

See the nxlog reference manual about the configuration options.

It should be installed locally and is also available

online at http://nxlog.org/nxlog-docs/en/nxlog-reference-manual.html

Please set the ROOT to the folder your nxlog was installed into,

otherwise it will not start.

#define ROOT C:\Program Files\nxlog #define ROOT_STRING C:\Program Files\nxlog define ROOT C:\Program Files (x86)\nxlog define ROOT_STRING C:\Program Files (x86)\nxlog define CERTDIR %ROOT%\cert define LOGFILE %ROOT%\data\nxlog.log

Moduledir %ROOT%\modules CacheDir %ROOT%\data Pidfile %ROOT%\data\nxlog.pid SpoolDir %ROOT%\data LogFile %ROOT%\data\nxlog.log

Include fileop while debugging, also enable in the output module below

<Extension fileop> Module xm_fileop </Extension>

<Extension json> Module xm_json </Extension>

<Extension syslog> Module xm_syslog </Extension>

<Input internal> Module im_internal Exec $Message = to_json(); </Input>

Windows Event Log

<Input eventlog> #Uncomment im_msvistalog for Windows Vista/2008 and later Module im_msvistalog

Query <QueryList>\
		<Query Id="0">\
			<Select Path="Application">*</Select>\
			<Select Path="System">*</Select>\
			<Select Path="Security">*</Select>\
		</Query>\
   </QueryList>  	

#Uncomment im_mseventlog for Windows XP/2000/2003
#Module im_mseventlog

Exec  $Message = to_json();

</Input>

<Processor buffer> Module pm_buffer # 100Mb disk buffer MaxSize 102400 Type disk </Processor>

<Processor buffer_iis> Module pm_buffer # 100Mb disk buffer MaxSize 102400 Type disk </Processor>

<Output out> Module om_tcp Host logs-01.loggly.com Port 514

Exec to_syslog_ietf();
Exec $raw_event  =~ s/(\[.*] )//g; $raw_event = replace($raw_event, '{', '[CUSTOMER ID tag="windows"] {', 1);

#Use the following line for debugging (uncomment the fileop extension above as well)
#Exec file_write("C:\\Program Files (x86)\\nxlog\\data\\nxlog_output.log",  $raw_event);

</Output>

<Output _nxlog> Module om_file File '%LOGFILE%'

<Schedule>
	When @hourly
	Exec if (file_size('%LOGFILE%') >= 1M) { file_cycle('%LOGFILE%', 5); _nxlog->reopen(); }
</Schedule>		

</Output>

<Route 1> Path internal, eventlog => buffer => out </Route>

<Route 2> Path internal => _nxlog </Route>

Create the parse rule for IIS logs. You can copy these from the header of the IIS log file.

<Extension w3c> Module xm_csv Fields $date, $time, $s-computername, $cs-method, $cs-uri-stem, $cs-uri-query, $c-ip, $cs(User-Agent), $cs(Referer), $cs-host, $sc-status, $sc-substatus, $sc-bytes, $cs-bytes, $time-taken, X-Forwarded-For, RequestId, PrincipalId FieldTypes string, string, string, string, string, string, string, string, string, string, string, string, integer, integer, integer, string, string, string Delimiter ' ' QuoteChar '"' EscapeControl FALSE UndefValue - </Extension>

Convert the IIS logs to JSON and use the original event time

<Input SC> Module im_file File "C:\inetpub\logs\LogFiles\W3SVC1\u_ex*" SavePos TRUE

Exec if $raw_event =~ /^#/ drop();    \
   else                               \
   {                                  \
        w3c->parse_csv();             \
        $SourceName = "IIS";          \
        $Message = to_json();         \
   }

</Input> <Input SC_WebAPI> Module im_file File "C:\inetpub\logs\LogFiles\W3SVC2\u_ex*" SavePos TRUE

Exec if $raw_event =~ /^#/ drop();    \
   else                               \
   {                                  \
        w3c->parse_csv();             \
        $SourceName = "IIS";          \
        $Message = to_json();         \
   }

</Input>

<Route IIS> Path SC,SC_WebAPI => buffer_iis => out </Route>

The error we received in data.log looks like below:

2021-02-18 14:11:12 "SERVERNAME" ERROR if-else failed at line 144, character 261 in C:\Program Files (x86)\nxlog\conf\nxlog.conf. statement execution has been aborted procedure 'parse_csv' failed at line 144, character 156 in C:\Program Files (x86)\nxlog\conf\nxlog.conf. statement execution has been aborted couldn't parse integer: ELB-HealthChecker/2.0 2021-02-18 14:11:26 "SERVERNAME" ERROR if-else failed at line 131, character 261 in C:\Program Files (x86)\nxlog\conf\nxlog.conf. statement execution has been aborted procedure 'parse_csv' failed at line 131, character 156 in C:\Program Files (x86)\nxlog\conf\nxlog.conf. statement execution has been aborted couldn't parse integer: ELB-HealthChecker/2.0

It seems as if the cs-useragent is being evaluated as an integer. or possible the input has fewer fields than expected.

Any help would be appreciated.

Hi,

I'm using a CE where xm_w3c module is not available. So i'm getting below error:

2019-04-09 14:59:30 ERROR Failed to load module from C:\Program Files (x86)\nxlog\modules\extension\xm_w3c.dll, The specified module could not be found. ; The specified module could not be found.
2019-04-09 14:59:30 ERROR Invalid InputType 'w3c_parser' at C:\Program Files (x86)\nxlog\conf\nxlog.conf:94

Is there a way to overcome this error in CE by downloading the particular module ? Or should I try using the Enterprise edition?

When attempting to input IIS logs I see the following error: 'ERROR invalid keyword: w3c->parse_csv()'. Has anyone come across this before and knows how I can parse IIS log into a JSON format to input to redis my config

<Extension _exec> Module xm_exec </Extension>

<Extension _json> Module xm_json </Extension>

<Extension _csv> Module xm_csv Fields $date, $time, $s_ip, $cs_method, $cs_uri_stem, $cs_uri_query, $s_port, $cs_username, $c_ip, $cs(User_Agent), $cs(Referer), $sc_status, $sc_substatus, $sc_win32_status, $time_taken FieldTypes string, string, string, string, string, string, integer, string, string, string, string, string, string, string, integer </Extension>

<Extension w3c> Module xm_csv #Fields: date time s-ip cs-method cs-uri-stem cs-uri-query s-port cs-username c-ip cs(User-Agent) cs(Referer) sc-status sc-substatus sc-win32-status time-taken Fields $date, $time, $s_ip, $cs_method, $cs_uri_stem, $cs_uri_query, $s_port, $cs_username, $c_ip, $cs(User_Agent), $cs(Referer), $sc_status, $sc_substatus, $sc_win32_status, $time_taken FieldTypes string, string, string, string, string, string, integer, string, string, string, string, string, string, string, integer Delimiter ' ' QuoteChar ''' </Extension>

<Extension w3cinput> Module xm_w3c Delimiter ' ' </Extension>

<Input iis_logs>
Module im_file File "C:\inetpub\logs\W3SVC2\u_ex*.log" ReadFromLast True Recursive True SavePos True Exec if $raw_event =~ /^#/ drop();
else
{
w3c->parse_csv();
$EventTime = parsedate($date + " " + $time);
$SourceName = "IIS";
to_json();
} </Input>

<output out_iis> Module om_redis Host # Port # Key "iis" </Output>

<Route 1>
Path iis_logs => out_iis </Route>

Hello Guys,

I have a question about NXLog IIS7 W3C logs. I set NXLog up and it works basicaly but the NXLog logs are full with error messages like this:

2016-04-26 09:46:36 ERROR if-else failed at line 64, character 257 in C:\Program Files (x86)\nxlog\conf\nxlog.conf. statement execution has been aborted; procedure 'parse_csv' failed at line 64, character 113 in C:\Program Files (x86)\nxlog\conf\nxlog.conf. statement execution has been aborted; couldn't parse integer: language=UK&region=802&idfwbonavigation=180173.2

It look, the IIS logs contain an "=" sign in the $cs-uri-query field, and NXLog wants integer after the "=", however the field set to be string in the config file.

Have you ever met anything like this? And if yes, what could be the solution?

 

NXLog extension and input config:

<Extension exiis>
    Module        xm_csv
    Fields        $date $time $s-ip $cs-method $cs-uri-stem $cs-uri-query $s-port $cs-username $c-ip $cs(User-Agent) $cs(Cookie) $cs(Referer) $sc-status $sc-substatus $sc-win32-status $sc-bytes $cs-bytes $time-taken
    FieldTypes    string, string, string, string, string, string, integer, string, string, string, string, string, integer, integer, integer, integer, integer, integer
    Delimiter    ' '
    QuoteChar    '"'
    EscapeControl    FALSE
    UndefValue    -
</Extension>

<Input IIS>
    Module        im_file
    File        "D:\\Logs\\IIS\\W3SVC300\\u_ex*"
    SavePos        TRUE
    Recursive    FALSE
    Exec if $raw_event =~ /^#/ drop();                                                                  \
        else                                                                                                         \
        {                                                                                                              \
            exiis->parse_csv();                                                                                \
            $EventTime = parsedate($date + " " + $time);                                          \
            $EventTime = strftime($EventTime, "%Y-%m-%dT%H:%M:%SZ");            \
        }
</Input>