responses
Hello,
I'm quite new to nxlog, so forgive me if my question is trivial but I'm having hard time to get the values I extract from my logs using exec and a regex. I have a very large stash of old windows logs in text file (in multiline format), what I want to do is use nxlog to load them in graylog, but I want to format the log in a different way.
It works perfectly when I do not use the the exec and the regexp, but it fails with the message:
2015-08-31 12:12:42 ERROR invalid keyword: $timestamp at C:\Program Files (x86)\nxlog\conf\nxlog.conf:36
The regex works when I test it using http://www.regexr.com/
The error seems in the way I'm trying to assign/write the variables matched by the regex
My nxlog.conf is like the below:
## This is a sample configuration file. See the nxlog reference manual about the
## configuration options. It should be installed locally and is also available
## online at http://nxlog.org/nxlog-docs/en/nxlog-reference-manual.html
## Please set the ROOT to the folder your nxlog was installed into,
## otherwise it will not start.
#define ROOT C:\Program Files\nxlog
define ROOT C:\Program Files (x86)\nxlog
Moduledir %ROOT%\modules
CacheDir %ROOT%\data
Pidfile %ROOT%\data\nxlog.pid
SpoolDir %ROOT%\data
LogFile %ROOT%\data\nxlog.log
<Extension gelf>
Module xm_gelf
</Extension>
<Extension multiline>
Module xm_multiline
HeaderLine /^{/
EndLine /^}/
</Extension>
<Input in>
Module im_file
File "C:\\tmp\\\\example-log.txt"
SavePos TRUE
Recursive TRUE
InputType multiline
exec if $raw_event =~ /(?:\{"([0-9]+?), ([0-9]+?), "(.+?)", "(.+?)", "(?:.+?)","(.+?)\n(?:[a-zA-Z0-9:,"\n\r\s\f\t\-\{\}\.\(\)]+?)New\sLogon:\s*(?:.+?)\n\s*Account\sName:\s*(.+?)\n(?:[a-zA-Z0-9:,"\n\r\s\f\t\-\{\}\.\(\)]+?)Source\sNetwork\sAddress:\s*([0-9\.]{7,15})\n(?:[a-zA-Z0-9:,"\n\r\s\f\t\-\{\}\.\(\)]+?)"\})/g;
$timestamp = $1;
$event = $2;
$status = $3;
$type = $4;
$short = $5;
$user = $6;
$source = $7;
</Input>
#<Output out>
# Module om_tcp
# Host 192.168.1.15
# Port 12201
# OutputType GELF_TCP
#</Output>
<Output out>
Module om_file
File 'C:\\tmp\\output'
</Output>
<Route 1>
Path in => out
</Route>
Could someone kindly help me out on this? I tried to read the nxlog manual but I need to confess that I could not understand what I'm doing wrong
Basically I just would the output to be composed by the raw message and the few fields I match with the regex
Thanks very much!
Mark
Comments (8)
Also you may want to correct this to include braces:
Hello
Thanks so much for the help, but the config still fails, the error is:
2015-08-31 19:17:43 ERROR invalid keyword: { at C:\Program Files (x86)\nxlog\conf\nxlog.conf:45
2015-08-31 19:17:43 ERROR module 'in' has configuration errors, not adding to route '1' at C:\Program Files (x86)\nxlog\conf\nxlog.conf:62
I have modified nxlog.conf as for your explanation:
## This is a sample configuration file. See the nxlog reference manual about the
## configuration options. It should be installed locally and is also available
## online at http://nxlog.org/nxlog-docs/en/nxlog-reference-manual.html
## Please set the ROOT to the folder your nxlog was installed into,
## otherwise it will not start.
#define ROOT C:\Program Files\nxlog
define ROOT C:\Program Files (x86)\nxlog
Moduledir %ROOT%\modules
CacheDir %ROOT%\data
Pidfile %ROOT%\data\nxlog.pid
SpoolDir %ROOT%\data
LogFile %ROOT%\data\nxlog.log
<Extension gelf>
Module xm_gelf
</Extension>
<Extension multiline>
Module xm_multiline
HeaderLine /^{/
EndLine /^}/
</Extension>
<Input in>
Module im_file
File "C:\\tmp\\\\example-log.txt"
SavePos TRUE
Recursive TRUE
InputType multiline
exec if $raw_event =~ /(?:\{"([0-9]+?), ([0-9]+?), "(.+?)", "(.+?)", "(?:.+?)","(.+?)\n(?:[a-zA-Z0-9:,"\n\r\s\f\t\-\{\}\.\(\)]+?)New\sLogon:\s*(?:.+?)\n\s*Account\sName:\s*(.+?)\n(?:[a-zA-Z0-9:,"\n\r\s\f\t\-\{\}\.\(\)]+?)Source\sNetwork\sAddress:\s*([0-9\.]{7,15})\n(?:[a-zA-Z0-9:,"\n\r\s\f\t\-\{\}\.\(\)]+?)"\})/g; \
{ \
$timestamp = $1; \
$event = $2; \
$status = $3; \
$type = $4; \
$short = $5; \
$user = $6; \
$source = $7; \
}
</Input>
#<Output out>
# Module om_tcp
# Host 192.168.1.15
# Port 12201
# OutputType GELF_TCP
#</Output>
<Output out>
Module om_file
File 'C:\\tmp\\output'
</Output>
<Route 1>
Path in => out
</Route>
I tried different configurations, with and without brackets, escapes etc, I wish I could figure out what I'm doing wrong but I need to admit I'm stuck!
Thanks!
Mark
There is an empty line in your config that's causing the error before this: { \
Hello,
Thanks again, still errors, now I get
2015-08-31 22:34:28 ERROR Couldn't parse Exec block at C:\Program Files (x86)\nxlog\conf\nxlog.conf:43; couldn't parse statement at line 43, character 316 in C:\Program Files (x86)\nxlog\conf\nxlog.conf; syntax error, unexpected ;, expecting (
The config has been modified taking the empty line away, I also tried to simplify the regex so make sure the issue wasnt with it but I still get the same error, my config is:
## This is a sample configuration file. See the nxlog reference manual about the
## configuration options. It should be installed locally and is also available
## online at http://nxlog.org/nxlog-docs/en/nxlog-reference-manual.html
## Please set the ROOT to the folder your nxlog was installed into,
## otherwise it will not start.
#define ROOT C:\Program Files\nxlog
define ROOT C:\Program Files (x86)\nxlog
Moduledir %ROOT%\modules
CacheDir %ROOT%\data
Pidfile %ROOT%\data\nxlog.pid
SpoolDir %ROOT%\data
LogFile %ROOT%\data\nxlog.log
<Extension gelf>
Module xm_gelf
</Extension>
<Extension multiline>
Module xm_multiline
HeaderLine /^{/
EndLine /^}/
</Extension>
<Input in>
Module im_file
File "C:\\tmp\\\\example-log.txt"
SavePos TRUE
Recursive TRUE
InputType multiline
exec if $raw_event =~ /(?:\{"([0-9]+?), ([0-9]+?), "(.+?)", "(.+?)", "(?:.+?)","(.+?)\n(?:[a-zA-Z0-9:,"\n\r\s\f\t\-\{\}\.\(\)]+?)New\sLogon:\s*(?:.+?)\n\s*Account\sName:\s*(.+?)\n(?:[a-zA-Z0-9:,"\n\r\s\f\t\-\{\}\.\(\)]+?)Source\sNetwork\sAddress:\s*([0-9\.]{7,15})\n(?:[a-zA-Z0-9:,"\n\r\s\f\t\-\{\}\.\(\)]+?)"\})/g; \
{ \
$timestamp = $1; \
$event = $2; \
$status = $3; \
$type = $4; \
$short = $5; \
$user = $6; \
$source = $7; \
}
</Input>
#<Output out>
# Module om_tcp
# Host 192.168.1.15
# Port 12201
# OutputType GELF_TCP
#</Output>
<Output out>
Module om_file
File 'C:\\tmp\\output'
</Output>
<Route 1>
Path in => out
</Route>
Any suggestion?
Thanks again very much
Mark
It's the semicolon (i.e. statement terminator) after the g. Removing that should fix the syntax error.
Actually in my first response I had that included by negligence, it's been edited now for correctness.
Hello,
Thanks very much, I still have troubles :-)
I removed the ;
I have tried different configurations etc but I still get this error message:
2015-09-01 19:01:34 ERROR Couldn't parse Exec block at C:\Program Files (x86)\nxlog\conf\nxlog.conf:43; couldn't parse statement at line 43, character 321 in C:\Program Files (x86)\nxlog\conf\nxlog.conf; syntax error, unexpected {, expecting (
Maybe there is something with the regex that I'm using that is upsetting the config, I will try to split the regex in small parts and see if it works, then I will add bit by bit all the pieces.
Just to be clear, I'm trying to load old windows event logs, hopefully will work and it will be of use for other people.
Kindly also let me now if think the thread is not meaningful and you want to close it, I understand that the forum is to share knowledge rather than to fix scripts!
Thanks again, great support!
Mark
Looking at that regex it is possible that there is something that causes a syntax error. I suggest starting with the most simple one (e.g. $raw_event =~ /foo/) and then gradually add stuff until you hit the error.
Since NXLog uses pcre internally, you could also test your regex with the pcretest tool directly. Most online regular expression test forms use the javascript regex engine which slightly differs from the pcre syntax.
Hello
I just would like to let you know that the config errors have been solved, there were blank spaces at the end of the regex and one on the left of the { \ and one at the end of the matching bracket, removed those the config does not generate anymore errors.
I did not realise even blank spaces are so important.
Still it seem I'm not able to pass the variables values to the ouput file but I will open a new question for that.
Thanks very much