how to get values extracted using regex?

Post a different question
1
answer

Hello,

I'm quite new to nxlog, so forgive me if my question is  trivial but I'm having hard time to get the values I extract from my  logs using exec and a regex. I have a very large stash of old windows logs in text file (in multiline format), what I want to do is use nxlog to load them in graylog, but I want to format the log in a different way.

It works perfectly when I do not use the the exec and the regexp, but it fails with the message:

2015-08-31 12:12:42 ERROR invalid keyword: $timestamp at C:\Program Files (x86)\nxlog\conf\nxlog.conf:36

The regex works when I test it using http://www.regexr.com/

The error seems in the way I'm trying to assign/write the variables matched by the regex

 

My nxlog.conf is like the below:

## This is a sample configuration file. See the nxlog reference manual about the
## configuration options. It should be installed locally and is also available
## online at http://nxlog.org/nxlog-docs/en/nxlog-reference-manual.html

## Please set the ROOT to the folder your nxlog was installed into,
## otherwise it will not start.

#define ROOT C:\Program Files\nxlog
define ROOT C:\Program Files (x86)\nxlog

Moduledir %ROOT%\modules
CacheDir %ROOT%\data
Pidfile %ROOT%\data\nxlog.pid
SpoolDir %ROOT%\data
LogFile %ROOT%\data\nxlog.log


<Extension gelf>
    Module       xm_gelf
</Extension>

<Extension multiline>
    Module      xm_multiline
    HeaderLine  /^{/
    EndLine     /^}/
</Extension>

<Input in>
        Module  im_file
        File    "C:\\tmp\\\\example-log.txt"
        SavePos  TRUE
        Recursive TRUE
        InputType       multiline
        exec if $raw_event =~ /(?:\{"([0-9]+?), ([0-9]+?), "(.+?)", "(.+?)", "(?:.+?)","(.+?)\n(?:[a-zA-Z0-9:,"\n\r\s\f\t\-\{\}\.\(\)]+?)New\sLogon:\s*(?:.+?)\n\s*Account\sName:\s*(.+?)\n(?:[a-zA-Z0-9:,"\n\r\s\f\t\-\{\}\.\(\)]+?)Source\sNetwork\sAddress:\s*([0-9\.]{7,15})\n(?:[a-zA-Z0-9:,"\n\r\s\f\t\-\{\}\.\(\)]+?)"\})/g; 
     
    $timestamp = $1; 
    $event = $2; 
    $status = $3; 
    $type = $4; 
    $short = $5; 
    $user = $6; 
    $source = $7; 
    
</Input>

#<Output out>
#    Module      om_tcp
#    Host        192.168.1.15
#    Port        12201
#    OutputType  GELF_TCP
#</Output>

<Output out>
    Module    om_file
    File    'C:\\tmp\\output'
</Output>

<Route 1>
    Path        in => out
</Route>

 

 Could someone kindly help me out on this? I tried to read the nxlog manual but I need to confess that I could not understand what I'm doing wrong 

Basically I just would the output to be composed by the raw message and the few fields I match with the regex

 

Thanks very much!

 

Mark

AskedAugust 31, 2015 - 4:17am

Answer (1)

The Exec directive needs a single line. You need to escape the linebreaks as follows if you want to supply it multiple lines:

exec if $raw_event =~ /(?:\{"([0-9]+?), ([0-9]+?), "(.+?)", "(.+?)", "(?:.+?)","(.+?)\n(?:[a-zA-Z0-9:,"\n\r\s\f\t\-\{\}\.\(\)]+?)New\sLogon:\s*(?:.+?)\n\s*Account\sName:\s*(.+?)\n(?:[a-zA-Z0-9:,"\n\r\s\f\t\-\{\}\.\(\)]+?)Source\sNetwork\sAddress:\s*([0-9\.]{7,15})\n(?:[a-zA-Z0-9:,"\n\r\s\f\t\-\{\}\.\(\)]+?)"\})/g; \    
    $timestamp = $1; \
    $event = $2; \
    $status = $3; \
    $type = $4; \
    $short = $5; \
    $user = $6; \
    $source = $7; 

AnsweredAugust 31, 2015 - 9:25am

Comments (8)

  • adm's picture
    (NXLog)

    Also you may want to correct this to include braces:

    if $raw_event =~ /<REGEX>/g \    
    {     \
        $timestamp = $1; \
        $event = $2; \
        $status = $3; \
        $type = $4; \
        $short = $5; \
        $user = $6; \
        $source = $7; \
    }

    August 31, 2015 - 9:30am
  • mark's picture

    Hello 

    Thanks so much for the help, but the config still fails, the error is:

    2015-08-31 19:17:43 ERROR invalid keyword: { at C:\Program Files (x86)\nxlog\conf\nxlog.conf:45
    2015-08-31 19:17:43 ERROR module 'in' has configuration errors, not adding to route '1' at C:\Program Files (x86)\nxlog\conf\nxlog.conf:62

    I have modified nxlog.conf as for your explanation:

    ## This is a sample configuration file. See the nxlog reference manual about the
    ## configuration options. It should be installed locally and is also available
    ## online at http://nxlog.org/nxlog-docs/en/nxlog-reference-manual.html

    ## Please set the ROOT to the folder your nxlog was installed into,
    ## otherwise it will not start.

    #define ROOT C:\Program Files\nxlog
    define ROOT C:\Program Files (x86)\nxlog

    Moduledir %ROOT%\modules
    CacheDir %ROOT%\data
    Pidfile %ROOT%\data\nxlog.pid
    SpoolDir %ROOT%\data
    LogFile %ROOT%\data\nxlog.log

    <Extension gelf>
        Module       xm_gelf
    </Extension>

    <Extension multiline>
        Module      xm_multiline
        HeaderLine  /^{/
        EndLine     /^}/
    </Extension>

    <Input in>
            Module  im_file
            File    "C:\\tmp\\\\example-log.txt"
            SavePos  TRUE
            Recursive TRUE
            InputType       multiline
            
        exec if $raw_event =~ /(?:\{"([0-9]+?), ([0-9]+?), "(.+?)", "(.+?)", "(?:.+?)","(.+?)\n(?:[a-zA-Z0-9:,"\n\r\s\f\t\-\{\}\.\(\)]+?)New\sLogon:\s*(?:.+?)\n\s*Account\sName:\s*(.+?)\n(?:[a-zA-Z0-9:,"\n\r\s\f\t\-\{\}\.\(\)]+?)Source\sNetwork\sAddress:\s*([0-9\.]{7,15})\n(?:[a-zA-Z0-9:,"\n\r\s\f\t\-\{\}\.\(\)]+?)"\})/g; \    
       
        { \
        $timestamp = $1; \
        $event = $2; \
        $status = $3; \
        $type = $4; \
        $short = $5; \
        $user = $6; \
        $source = $7; \
        }
        
    </Input>

    #<Output out>
    #    Module      om_tcp
    #    Host        192.168.1.15
    #    Port        12201
    #    OutputType  GELF_TCP
    #</Output>

    <Output out>
        Module    om_file
        File    'C:\\tmp\\output'
    </Output>

    <Route 1>
        Path        in => out
    </Route>

     

    I tried different configurations, with and without brackets, escapes etc, I wish I could figure out what I'm doing wrong but I need to admit I'm stuck!

     

    Thanks!

    Mark

    August 31, 2015 - 11:22am
  • adm's picture
    (NXLog)

    There is an empty line in your config that's causing the error before this: { \

     

    August 31, 2015 - 11:27am
  • mark's picture

    Hello,

     

    Thanks again, still errors, now I get 

    2015-08-31 22:34:28 ERROR Couldn't parse Exec block at C:\Program Files (x86)\nxlog\conf\nxlog.conf:43; couldn't parse statement at line 43, character 316 in C:\Program Files (x86)\nxlog\conf\nxlog.conf; syntax error, unexpected ;, expecting (

     

    The config has been modified taking the empty line away, I also tried to simplify the regex so make sure the issue wasnt with it but I still get the same error, my config is:

     

    ## This is a sample configuration file. See the nxlog reference manual about the
    ## configuration options. It should be installed locally and is also available
    ## online at http://nxlog.org/nxlog-docs/en/nxlog-reference-manual.html

    ## Please set the ROOT to the folder your nxlog was installed into,
    ## otherwise it will not start.

    #define ROOT C:\Program Files\nxlog
    define ROOT C:\Program Files (x86)\nxlog

    Moduledir %ROOT%\modules
    CacheDir %ROOT%\data
    Pidfile %ROOT%\data\nxlog.pid
    SpoolDir %ROOT%\data
    LogFile %ROOT%\data\nxlog.log

    <Extension gelf>
        Module       xm_gelf
    </Extension>

    <Extension multiline>
        Module      xm_multiline
        HeaderLine  /^{/
        EndLine     /^}/
    </Extension>

    <Input in>
            Module  im_file
            File    "C:\\tmp\\\\example-log.txt"
            SavePos  TRUE
            Recursive TRUE
            InputType       multiline
            exec if $raw_event =~ /(?:\{"([0-9]+?), ([0-9]+?), "(.+?)", "(.+?)", "(?:.+?)","(.+?)\n(?:[a-zA-Z0-9:,"\n\r\s\f\t\-\{\}\.\(\)]+?)New\sLogon:\s*(?:.+?)\n\s*Account\sName:\s*(.+?)\n(?:[a-zA-Z0-9:,"\n\r\s\f\t\-\{\}\.\(\)]+?)Source\sNetwork\sAddress:\s*([0-9\.]{7,15})\n(?:[a-zA-Z0-9:,"\n\r\s\f\t\-\{\}\.\(\)]+?)"\})/g; \
        { \
        $timestamp = $1; \
        $event = $2; \
        $status = $3; \
        $type = $4; \
        $short = $5; \
        $user = $6; \
        $source = $7; \
        }
    </Input>

    #<Output out>
    #    Module      om_tcp
    #    Host        192.168.1.15
    #    Port        12201
    #    OutputType  GELF_TCP
    #</Output>

    <Output out>
        Module    om_file
        File    'C:\\tmp\\output'
    </Output>

    <Route 1>
        Path        in => out
    </Route>

     

    Any suggestion?

     

    Thanks again very much

    Mark

     

     

    August 31, 2015 - 2:38pm
  • adm's picture
    (NXLog)

    It's the semicolon (i.e. statement terminator) after the g. Removing that should fix the syntax error.

    Actually in my first response I had that included by negligence, it's been edited now for correctness.

     

    August 31, 2015 - 2:45pm
  • mark's picture

    Hello,

    Thanks very much, I still have troubles :-)

    I removed the ; 

    I have tried different configurations etc but I still get this error message:

    2015-09-01 19:01:34 ERROR Couldn't parse Exec block at C:\Program Files (x86)\nxlog\conf\nxlog.conf:43; couldn't parse statement at line 43, character 321 in C:\Program Files (x86)\nxlog\conf\nxlog.conf; syntax error, unexpected {, expecting (

     

    Maybe there is something with the regex that I'm using that is upsetting the config, I will try to split the regex in small parts and see if it works, then I will add bit by bit all the pieces.

    Just to be clear, I'm trying to load old windows event logs, hopefully will work and it will be of use for other people. 

     

    Kindly also let me now if think the thread is not meaningful and you want to close it, I understand that the forum is to share knowledge rather than to fix scripts!

    Thanks again, great support!

    Mark

    September 1, 2015 - 1:14pm
  • adm's picture
    (NXLog)

    Looking at that regex it is possible that there is something that causes a syntax error. I suggest starting with the most simple one (e.g. $raw_event =~ /foo/) and then gradually add stuff until you hit the error.

    Since NXLog uses pcre internally, you could also test your regex with the pcretest tool directly. Most online regular expression test forms use the javascript regex engine which slightly differs from the pcre syntax.

    September 1, 2015 - 1:51pm
  • mark's picture

    Hello

    I just would like to let you know that the config errors have been solved, there were blank spaces at the end of the regex and one on the left of the { \ and one at the end of the matching bracket, removed those the config does not generate anymore errors.

    I did not realise even blank spaces are so important.

    Still it seem I'm not able to pass the variables values to the ouput file but I will open a new question for that.

    Thanks very much

     

     

    September 2, 2015 - 12:02pm