how to get values extracted using regex?

Tags: regex | exec | variables

#1 mark

Hello,

I'm quite new to nxlog, so forgive me if my question is  trivial but I'm having hard time to get the values I extract from my  logs using exec and a regex. I have a very large stash of old windows logs in text file (in multiline format), what I want to do is use nxlog to load them in graylog, but I want to format the log in a different way.

It works perfectly when I do not use the the exec and the regexp, but it fails with the message:

2015-08-31 12:12:42 ERROR invalid keyword: $timestamp at C:\Program Files (x86)\nxlog\conf\nxlog.conf:36

The regex works when I test it using http://www.regexr.com/

The error seems in the way I'm trying to assign/write the variables matched by the regex

 

My nxlog.conf is like the below:

## This is a sample configuration file. See the nxlog reference manual about the
## configuration options. It should be installed locally and is also available
## online at http://nxlog.org/nxlog-docs/en/nxlog-reference-manual.html

## Please set the ROOT to the folder your nxlog was installed into,
## otherwise it will not start.

#define ROOT C:\Program Files\nxlog
define ROOT C:\Program Files (x86)\nxlog

Moduledir %ROOT%\modules
CacheDir %ROOT%\data
Pidfile %ROOT%\data\nxlog.pid
SpoolDir %ROOT%\data
LogFile %ROOT%\data\nxlog.log


<Extension gelf>
    Module       xm_gelf
</Extension>

<Extension multiline>
    Module      xm_multiline
    HeaderLine  /^{/
    EndLine     /^}/
</Extension>

<Input in>
        Module  im_file
        File    "C:\\tmp\\\\example-log.txt"
        SavePos  TRUE
        Recursive TRUE
        InputType       multiline
        exec if $raw_event =~ /(?:\{"([0-9]+?), ([0-9]+?), "(.+?)", "(.+?)", "(?:.+?)","(.+?)\n(?:[a-zA-Z0-9:,"\n\r\s\f\t\-\{\}\.\(\)]+?)New\sLogon:\s*(?:.+?)\n\s*Account\sName:\s*(.+?)\n(?:[a-zA-Z0-9:,"\n\r\s\f\t\-\{\}\.\(\)]+?)Source\sNetwork\sAddress:\s*([0-9\.]{7,15})\n(?:[a-zA-Z0-9:,"\n\r\s\f\t\-\{\}\.\(\)]+?)"\})/g; 
     
    $timestamp = $1; 
    $event = $2; 
    $status = $3; 
    $type = $4; 
    $short = $5; 
    $user = $6; 
    $source = $7; 
    
</Input>

#<Output out>
#    Module      om_tcp
#    Host        192.168.1.15
#    Port        12201
#    OutputType  GELF_TCP
#</Output>

<Output out>
    Module    om_file
    File    'C:\\tmp\\output'
</Output>

<Route 1>
    Path        in => out
</Route>

 

 Could someone kindly help me out on this? I tried to read the nxlog manual but I need to confess that I could not understand what I'm doing wrong 

Basically I just would the output to be composed by the raw message and the few fields I match with the regex

 

Thanks very much!

 

Mark

#2 adm Nxlog ✓
#1 mark
Hello, I'm quite new to nxlog, so forgive me if my question is  trivial but I'm having hard time to get the values I extract from my  logs using exec and a regex. I have a very large stash of old windows logs in text file (in multiline format), what I want to do is use nxlog to load them in graylog, but I want to format the log in a different way. It works perfectly when I do not use the the exec and the regexp, but it fails with the message: 2015-08-31 12:12:42 ERROR invalid keyword: $timestamp at C:\Program Files (x86)\nxlog\conf\nxlog.conf:36 The regex works when I test it using http://www.regexr.com/ The error seems in the way I'm trying to assign/write the variables matched by the regex   My nxlog.conf is like the below: ## This is a sample configuration file. See the nxlog reference manual about the ## configuration options. It should be installed locally and is also available ## online at http://nxlog.org/nxlog-docs/en/nxlog-reference-manual.html ## Please set the ROOT to the folder your nxlog was installed into, ## otherwise it will not start. #define ROOT C:\Program Files\nxlog define ROOT C:\Program Files (x86)\nxlog Moduledir %ROOT%\modules CacheDir %ROOT%\data Pidfile %ROOT%\data\nxlog.pid SpoolDir %ROOT%\data LogFile %ROOT%\data\nxlog.log <Extension gelf>     Module       xm_gelf </Extension> <Extension multiline>     Module      xm_multiline     HeaderLine  /^{/     EndLine     /^}/ </Extension> <Input in>         Module  im_file         File    "C:\\tmp\\\\example-log.txt"         SavePos  TRUE         Recursive TRUE         InputType       multiline         exec if $raw_event =~ /(?:\{"([0-9]+?), ([0-9]+?), "(.+?)", "(.+?)", "(?:.+?)","(.+?)\n(?:[a-zA-Z0-9:,"\n\r\s\f\t\-\{\}\.\(\)]+?)New\sLogon:\s*(?:.+?)\n\s*Account\sName:\s*(.+?)\n(?:[a-zA-Z0-9:,"\n\r\s\f\t\-\{\}\.\(\)]+?)Source\sNetwork\sAddress:\s*([0-9\.]{7,15})\n(?:[a-zA-Z0-9:,"\n\r\s\f\t\-\{\}\.\(\)]+?)"\})/g;            $timestamp = $1;      $event = $2;      $status = $3;      $type = $4;      $short = $5;      $user = $6;      $source = $7;       </Input> #<Output out> #    Module      om_tcp #    Host        192.168.1.15 #    Port        12201 #    OutputType  GELF_TCP #</Output> <Output out>     Module    om_file     File    'C:\\tmp\\output' </Output> <Route 1>     Path        in => out </Route>    Could someone kindly help me out on this? I tried to read the nxlog manual but I need to confess that I could not understand what I'm doing wrong  Basically I just would the output to be composed by the raw message and the few fields I match with the regex   Thanks very much!   Mark

The Exec directive needs a single line. You need to escape the linebreaks as follows if you want to supply it multiple lines:

exec if $raw_event =~ /(?:\{"([0-9]+?), ([0-9]+?), "(.+?)", "(.+?)", "(?:.+?)","(.+?)\n(?:[a-zA-Z0-9:,"\n\r\s\f\t\-\{\}\.\(\)]+?)New\sLogon:\s*(?:.+?)\n\s*Account\sName:\s*(.+?)\n(?:[a-zA-Z0-9:,"\n\r\s\f\t\-\{\}\.\(\)]+?)Source\sNetwork\sAddress:\s*([0-9\.]{7,15})\n(?:[a-zA-Z0-9:,"\n\r\s\f\t\-\{\}\.\(\)]+?)"\})/g; \    
    $timestamp = $1; \
    $event = $2; \
    $status = $3; \
    $type = $4; \
    $short = $5; \
    $user = $6; \
    $source = $7;