Ask questions. Get answers. Find technical product solutions from passionate experts in the NXLog community.
Modify raw_event to send custom logs
a.volkov created
Hello,
I would like to modify the IIS logs for further transfer to the destination.
Now I am parsing the IIS log with the xm_csv module, as in the template. UndefValue is disabled to not get empty.
How can I interact with parsed data from w3c_parser? For example, I want to combine into a variable $request = '"' + $cs-method + ' ' + $cs-uri-stem + ' ' + $cs-version + '"'; such a value, but I get an error.
When I try to write a field from w3c_parser to $raw_event, I also get an error. Any other data is added without error.
For example $raw_event = $c-ip error
$raw_event = $EventTime + ' ' + $http_host no error
Thank you in advance!
Example error, logs and config file below
2022-03-23 16:49:45 WARNING stopping nxlog service
2022-03-23 16:49:45 WARNING nxlog-ce received a termination request signal, exiting...
2022-03-23 16:49:56 ERROR Couldn't parse Exec block at C:\Program Files\nxlog\conf\nxlog.conf:59; couldn't parse statement at line 71, character 32 in C:\Program Files\nxlog\conf\nxlog.conf; syntax error, unexpected +, expecting (
2022-03-23 16:49:56 ERROR module 'iis_w3c' has configuration errors, not adding to route 'uds_to_file' at C:\Program Files\nxlog\conf\nxlog.conf:84
2022-03-23 16:49:56 ERROR route uds_to_file is not functional without input modules, ignored at C:\Program Files\nxlog\conf\nxlog.conf:84
2022-03-23 16:49:56 WARNING no routes defined!
2022-03-23 16:49:56 WARNING not starting unused module iis_w3c
2022-03-23 16:49:56 WARNING not starting unused module file
2022-03-23 16:49:56 INFO nxlog-ce-3.0.2272 started
Current log format
date time s-computername s-ip cs-method cs-uri-stem cs-uri-query s-port cs-username c-ip cs-version cs(User-Agent) cs(Cookie) cs(Referer) cs-host sc-status sc-bytes cs-bytes time-taken
2022-03-23 08:00:01 HOST.DOMAIN 99.XX.XX.4 GET /AnalyticsService - 443 - XX.XX.XX.XXX HTTP/1.1 Zabbix - - site.host.domain 200 3918 144 4
Required log format
$http_host $remote_addr $remote_user [$time_local] UNIX-TIME-$msec "$request" $status "$sent_http_content_type" $body_bytes_sent "$http_referer" "$http_user_agent" "$http_cookie" $request_time "$upstream_addr" NGINX-CACHE-$upstream_cache_status "$request_id" "$request_body"
host.domain 99.99.99.249 - [11/Mar/2022:20:09:56+0300] UNIX-TIME-1647018596.031 "GET /api/company.php?id=853747 HTTP/1.1" 200 "text/xml; charset=UTF-8" 1455 "-" "-" "20b6b325ea192383cb1244412247c5ea=3002538ef353c9daab4f742176a840; etpsid=f488b343a23d1a4a2332e089a0" 0.059 0.059 "10.10.10.111:80" NGINX-CACHE-- "d0b5ac12cf82671067aa5e6c5c" "-"
Panic Soft
#NoFreeOnExit TRUE
define ROOT C:\Program Files\nxlog
define CERTDIR %ROOT%\cert
define CONFDIR %ROOT%\conf\nxlog.d
define LOGDIR %ROOT%\data
define LOGFILE %LOGDIR%\nxlog.log
LogFile %LOGFILE%
Moduledir %ROOT%\modules
CacheDir %ROOT%\data
Pidfile %ROOT%\data\nxlog.pid
SpoolDir %ROOT%\data
<Extension _syslog>
Module xm_syslog
</Extension>
<Extension fileop>
Module xm_fileop
</Extension>
<Extension _charconv>
Module xm_charconv
AutodetectCharsets iso8859-2, utf-8, utf-16, utf-32
</Extension>
<Extension _exec>
Module xm_exec
</Extension>
#Create the parse rule for IIS logs. You can copy these from the header of the IIS log file.
<Extension w3c_parser>
Module xm_csv
Fields $date, $time, $s-computername, $s-ip, $cs-method, $cs-uri-stem, $cs-uri-query, $s-port, $cs-username, $c-ip, $cs-version, $cs(User-Agent), $cs(Cookie), $cs(Referer), $cs-host, $sc-status, $sc-bytes, $cs-bytes, $time-taken
FieldTypes string, string, string, string, string, string, string, integer, string, string, string, string, string, string, string, integer, integer, integer, integer
Delimiter ' '
EscapeChar '"'
QuoteChar '"'
EscapeControl FALSE
UndefValue -
</Extension>
<Extension w3c_out>
Module xm_csv
Fields $http_host, $c-ip, $cs-username, $EventTime1, $sc-status, $Unix
FieldTypes string, string, string, string, string, string
Delimiter ' '
UndefValue -
QuoteMethod None
</Extension>
<Input iis_w3c>
Module im_file
File 'C:\inetpub\logs\LogFiles\W3SVC1\u_ex*.log'
SavePos TRUE
<Exec>
if $raw_event =~ /^#/ drop();
else
{
w3c_parser->parse_csv();
$EventTime = parsedate($date + " " + $time);
$EventTime = $EventTime + (3 * 3600);
$EventTime1 = strftime($EventTime, '[%d/%b/%Y:%H:%M:%S]');
$EventTime1 = '$EventTime1' + ' +0003]';
$Unix = integer($EventTime);
$Unix = 'UNIX-TIME-' + $Unix;
$http_host = "site.host.domain";
$request = '"' + $cs-method + ' ' + $cs-uri-stem + ' ' + $cs-version + '"';
$request = $cs-method;
w3c_out->to_csv();
}
</Exec>
</Input>
<Output file>
Module om_file
File 'C:\inetpub\logs\LogFiles\Parser\w3c.txt'
</Output>
<Route uds_to_file>
Path iis_w3c => file
</Route>
a.volkov created
Send custom value in iis_log message
Anbers created
I want to know is possible send, with the iis log message, a custom field and value. Actually this is my config:
define ROOT C:\Program Files (x86)\nxlog
define CERTDIR %ROOT%\cert
define CONFDIR %ROOT%\conf
define LOGDIR %ROOT%\data
define LOGFILE %LOGDIR%\nxlog.log
LogFile %LOGFILE%
Moduledir %ROOT%\modules
CacheDir %ROOT%\data
Pidfile %ROOT%\data\nxlog.pid
SpoolDir %ROOT%\data
<Extension _gelf>
#Module xm_syslog
Module xm_gelf
#Module xm_json
</Extension>
<Extension w3c_parser>
Module xm_csv
Fields date, time, s-ip, cs-method, cs-uri-stem, cs-uri-query, \
s-port, cs-username, c-ip, cs(User-Agent), cs(Referer), \
sc-status, sc-substatus, sc-win32-status, time-taken
FieldTypes string, string, string, string, string, string, integer, \
string, string, string, string, integer, integer, integer, \
integer
Delimiter ' '
EscapeChar '"'
QuoteChar '"'
EscapeControl FALSE
UndefValue -
</Extension>
<Input in2>
Module im_file
File "c:\\logs\\app\\iislog\\u_extend1.log"
<Exec>
if $raw_event =~ /^#/ drop();
else
{
w3c_parser->parse_csv();
$EventTime = parsedate($date + "T" + $time + ".000Z");
}
</Exec>
</Input>
<Output out>
Module om_udp
Host 172.28.36.25
Port 12201
#Exec to_syslog_snare();
OutputType GELF
</Output>
<Route 1>
Path in2 => out
</Route>
This work great but i need to add a custom and static value in the message, like : FIELD: "system" , VALUE: "program1". and this value is always the same
date, time, s-ip, cs-method, cs-uri-stem, cs-uri-query, s-port, cs-username, c-ip, cs(User-Agent), cs(Referer), sc-status, sc-substatus, sc-win32-status, time-taken,**system**
Is this possible?
Thanks!
Anbers created
Difficulty Parsing IIS Logs and Sending to Loggly
BB_838545 created
I'm using NXLog to sending Windows events and IIS logs to Loggly. We've recently onboarded a new MSSP and they have asked us to check off all IIS logging fields. This seems to break parsing of IIS logs that need to be sent to Loggly. I've contact Loggly support and they can't seem to come to a resolution.
Below is the code that we had been using for Loggly previously.
This is a sample NXLog configuration file created by Loggly. June 2013
See the nxlog reference manual about the configuration options.
It should be installed locally and is also available
online at http://nxlog.org/nxlog-docs/en/nxlog-reference-manual.html
Please set the ROOT to the folder your nxlog was installed into,
otherwise it will not start.
#define ROOT C:\Program Files\nxlog
#define ROOT_STRING C:\Program Files\nxlog
define ROOT C:\Program Files (x86)\nxlog
define ROOT_STRING C:\Program Files (x86)\nxlog
define CERTDIR %ROOT%\cert
define LOGFILE %ROOT%\data\nxlog.log
Moduledir %ROOT%\modules
CacheDir %ROOT%\data
Pidfile %ROOT%\data\nxlog.pid
SpoolDir %ROOT%\data
LogFile %ROOT%\data\nxlog.log
Include fileop while debugging, also enable in the output module below
<Extension fileop>
Module xm_fileop
</Extension>
<Extension json>
Module xm_json
</Extension>
<Extension syslog>
Module xm_syslog
</Extension>
<Input internal>
Module im_internal
Exec $Message = to_json();
</Input>
Windows Event Log
<Input eventlog>
#Uncomment im_msvistalog for Windows Vista/2008 and later
Module im_msvistalog
Query <QueryList>\
<Query Id="0">\
<Select Path="Application">*</Select>\
<Select Path="System">*</Select>\
<Select Path="Security">*</Select>\
</Query>\
</QueryList>
#Uncomment im_mseventlog for Windows XP/2000/2003
#Module im_mseventlog
Exec $Message = to_json();
</Input>
<Processor buffer>
Module pm_buffer
# 100Mb disk buffer
MaxSize 102400
Type disk
</Processor>
<Processor buffer_iis>
Module pm_buffer
# 100Mb disk buffer
MaxSize 102400
Type disk
</Processor>
<Output out>
Module om_tcp
Host logs-01.loggly.com
Port 514
Exec to_syslog_ietf();
Exec $raw_event =~ s/(\[.*] )//g; $raw_event = replace($raw_event, '{', '[CUSTOMER ID tag="windows"] {', 1);
#Use the following line for debugging (uncomment the fileop extension above as well)
#Exec file_write("C:\\Program Files (x86)\\nxlog\\data\\nxlog_output.log", $raw_event);
</Output>
<Output _nxlog>
Module om_file
File '%LOGFILE%'
<Schedule>
When @hourly
Exec if (file_size('%LOGFILE%') >= 1M) { file_cycle('%LOGFILE%', 5); _nxlog->reopen(); }
</Schedule>
</Output>
<Route 1>
Path internal, eventlog => buffer => out
</Route>
<Route 2>
Path internal => _nxlog
</Route>
Create the parse rule for IIS logs. You can copy these from the header of the IIS log file.
<Extension w3c>
Module xm_csv
Fields $date, $time, $s-computername, $cs-method, $cs-uri-stem, $cs-uri-query, $c-ip, $cs(User-Agent), $cs(Referer), $cs-host, $sc-status, $sc-substatus, $sc-bytes, $cs-bytes, $time-taken, X-Forwarded-For, RequestId, PrincipalId
FieldTypes string, string, string, string, string, string, string, string, string, string, string, string, integer, integer, integer, string, string, string
Delimiter ' '
QuoteChar '"'
EscapeControl FALSE
UndefValue -
</Extension>
Convert the IIS logs to JSON and use the original event time
<Input SC>
Module im_file
File "C:\inetpub\logs\LogFiles\W3SVC1\u_ex*"
SavePos TRUE
Exec if $raw_event =~ /^#/ drop(); \
else \
{ \
w3c->parse_csv(); \
$SourceName = "IIS"; \
$Message = to_json(); \
}
</Input>
<Input SC_WebAPI>
Module im_file
File "C:\inetpub\logs\LogFiles\W3SVC2\u_ex*"
SavePos TRUE
Exec if $raw_event =~ /^#/ drop(); \
else \
{ \
w3c->parse_csv(); \
$SourceName = "IIS"; \
$Message = to_json(); \
}
</Input>
<Route IIS>
Path SC,SC_WebAPI => buffer_iis => out
</Route>
The error we received in data.log looks like below:
2021-02-18 14:11:12 "SERVERNAME" ERROR if-else failed at line 144, character 261 in C:\Program Files (x86)\nxlog\conf\nxlog.conf. statement execution has been aborted
procedure 'parse_csv' failed at line 144, character 156 in C:\Program Files (x86)\nxlog\conf\nxlog.conf. statement execution has been aborted
couldn't parse integer: ELB-HealthChecker/2.0
2021-02-18 14:11:26 "SERVERNAME" ERROR if-else failed at line 131, character 261 in C:\Program Files (x86)\nxlog\conf\nxlog.conf. statement execution has been aborted
procedure 'parse_csv' failed at line 131, character 156 in C:\Program Files (x86)\nxlog\conf\nxlog.conf. statement execution has been aborted
couldn't parse integer: ELB-HealthChecker/2.0
It seems as if the cs-useragent is being evaluated as an integer. or possible the input has fewer fields than expected.
Any help would be appreciated.
BB_838545 created
nxlog error when tried to use xm_w3c module
Sangeetha created
Hi,
I'm using a CE where xm_w3c module is not available. So i'm getting below error:
2019-04-09 14:59:30 ERROR Failed to load module from C:\Program Files (x86)\nxlog\modules\extension\xm_w3c.dll, The specified module could not be found. ; The specified module could not be found.
2019-04-09 14:59:30 ERROR Invalid InputType 'w3c_parser' at C:\Program Files (x86)\nxlog\conf\nxlog.conf:94
Is there a way to overcome this error in CE by downloading the particular module ?
Or should I try using the Enterprise edition?
Sangeetha created
Invalid Keyword w3c->parse_csv
peterc created
When attempting to input IIS logs I see the following error: 'ERROR invalid keyword: w3c->parse_csv()'. Has anyone come across this before and knows how I can parse IIS log into a JSON format to input to redis
my config
<Extension _exec>
Module xm_exec
</Extension>
<Extension _json>
Module xm_json
</Extension>
<Extension _csv>
Module xm_csv
Fields $date, $time, $s_ip, $cs_method, $cs_uri_stem, $cs_uri_query, $s_port, $cs_username, $c_ip, $cs(User_Agent), $cs(Referer), $sc_status, $sc_substatus, $sc_win32_status, $time_taken
FieldTypes string, string, string, string, string, string, integer, string, string, string, string, string, string, string, integer
</Extension>
<Extension w3c>
Module xm_csv
#Fields: date time s-ip cs-method cs-uri-stem cs-uri-query s-port cs-username c-ip cs(User-Agent) cs(Referer) sc-status sc-substatus sc-win32-status time-taken
Fields $date, $time, $s_ip, $cs_method, $cs_uri_stem, $cs_uri_query, $s_port, $cs_username, $c_ip, $cs(User_Agent), $cs(Referer), $sc_status, $sc_substatus, $sc_win32_status, $time_taken
FieldTypes string, string, string, string, string, string, integer, string, string, string, string, string, string, string, integer
Delimiter ' '
QuoteChar '''
</Extension>
<Extension w3cinput>
Module xm_w3c
Delimiter ' '
</Extension>
<Input iis_logs>
Module im_file
File "C:\inetpub\logs\W3SVC2\u_ex*.log"
ReadFromLast True
Recursive True
SavePos True
Exec if $raw_event =~ /^#/ drop();
else
{
w3c->parse_csv();
$EventTime = parsedate($date + " " + $time);
$SourceName = "IIS";
to_json();
}
</Input>
<output out_iis>
Module om_redis
Host #
Port #
Key "iis"
</Output>
<Route 1>
Path iis_logs => out_iis
</Route>
peterc created
IIS7 W3C log parsing fails
CSimpiFoN created
Hello Guys,
I have a question about NXLog IIS7 W3C logs. I set NXLog up and it works basicaly but the NXLog logs are full with error messages like this:
2016-04-26 09:46:36 ERROR if-else failed at line 64, character 257 in C:\Program Files (x86)\nxlog\conf\nxlog.conf. statement execution has been aborted; procedure 'parse_csv' failed at line 64, character 113 in C:\Program Files (x86)\nxlog\conf\nxlog.conf. statement execution has been aborted; couldn't parse integer: language=UK®ion=802&idfwbonavigation=180173.2
It look, the IIS logs contain an "=" sign in the $cs-uri-query field, and NXLog wants integer after the "=", however the field set to be string in the config file.
Have you ever met anything like this? And if yes, what could be the solution?
NXLog extension and input config:
<Extension exiis>
Module xm_csv
Fields $date $time $s-ip $cs-method $cs-uri-stem $cs-uri-query $s-port $cs-username $c-ip $cs(User-Agent) $cs(Cookie) $cs(Referer) $sc-status $sc-substatus $sc-win32-status $sc-bytes $cs-bytes $time-taken
FieldTypes string, string, string, string, string, string, integer, string, string, string, string, string, integer, integer, integer, integer, integer, integer
Delimiter ' '
QuoteChar '"'
EscapeControl FALSE
UndefValue -
</Extension>
<Input IIS>
Module im_file
File "D:\\Logs\\IIS\\W3SVC300\\u_ex*"
SavePos TRUE
Recursive FALSE
Exec if $raw_event =~ /^#/ drop(); \
else \
{ \
exiis->parse_csv(); \
$EventTime = parsedate($date + " " + $time); \
$EventTime = strftime($EventTime, "%Y-%m-%dT%H:%M:%SZ"); \
}
</Input>
CSimpiFoN created
