14
responses

Is there a way to put a wildcard in a regular expression? Here is an example of my Headerline that I need to set:

Jun 11 16:10:05 tst-tst01-rp adevents: ---Begin event transaction---

So I have this for my headerline, but it doesn't seem to work:

Headerline /^\w\w\w \d\d \d\d:\d\d:\d\d \w\w\w-\w\w\w\d\d-\w\w adevents: ---Begin event transaction---/

I was hoping to be able to use a wildcard like this, but it doesn't work:

Headerline /^\*---Begin event transaction---/

Thanks for your time.

AskedJune 11, 2018 - 11:06pm

Answer (1)

Something more like the following:
HeaderLine /^.+---Begin event transaction---/

Edit: I should also mention that the built-in regex is PCRE compatible so when you are testing, keep that in mind.

Comments (13)

  • Zhengshi's picture
    (NXLog)

    I setup a test to verify. This setup worked for me. Note that the JSON is just because it makes things easier to see. Each event is its own JSON record.

    nxlog.conf :

    Panic Soft
    
    define LOGFILE /tmp/nxlog.log
    LogFile %LOGFILE%
    
    <Extension log>
      Module xm_multiline
      #FixedLineCount
      HeaderLine /^.+---Begin event transaction---/
    </Extension>
    
    <Extension json>
      Module xm_json
    </Extension>
    
    <Input in>
      Module im_file
      File "motts_ml.txt"
      InputType log
      SavePos FALSE
      ReadFromLast FALSE
      Exec $message = $raw_event; to_json();
    </Input>
    <Output file>
        Module          om_file
        File            "/tmp/testout.log"
    </Output>
    <Route 1>
        Path    in => file
    </Route>
    

    motts_ml.txt :

    Jun 11 16:10:05 tst-tst01-rp adevents: ---Begin event transaction---
    Stuff
    Jun 12 16:10:05 tst-tst01-rp adevents: ---Begin event transaction---
    Also Stuff
    Jun 13 16:10:05 tst-tst01-rp adevents: ---Begin event transaction---
    More Stuff
    

    testout.log :

    {"EventReceivedTime":"2018-06-11 16:14:46.250544-05:00","SourceModuleName":"in","SourceModuleType":"im_file","message":"Jun 11 16:10:05 tst-tst01-rp adevents: ---Begin event transaction---\nStuff"}
    {"EventReceivedTime":"2018-06-11 16:14:46.250729-05:00","SourceModuleName":"in","SourceModuleType":"im_file","message":"Jun 12 16:10:05 tst-tst01-rp adevents: ---Begin event transaction---\nAlso Stuff"}
    {"EventReceivedTime":"2018-06-11 16:14:46.250931-05:00","SourceModuleName":"in","SourceModuleType":"im_file","message":"Jun 13 16:10:05 tst-tst01-rp adevents: ---Begin event transaction---\nMore Stuff"}
    

  • motts's picture

    Strange because I have almost the same setup:

    <Extension log>

    Module xm_multiline
    Headerline /^.+---Begin event transaction---/
    Exec if $raw_event !~ /\: |---|Failed to join|\d{3,4}/ drop();

    </Extension>

    <Extension json>

    Module xm_json

    </Extension>

    <Input in>

    Module im_file

    File "C:\\Users\\Administrator\\Desktop\\srrlogs\\\*.log"
    InputType log
    SavePos FALSE
    ReadFromLast FALSE

    </Input>

    <Output out>

    Module om_tcp

    Host (ip)
    Port (port)

    </Output>

    <Output out2>

    Module om_file
    File "C:\\Users\\Administrator\\Desktop\\output.txt"

    </Output>

    <Route>

    Path in => out

    </Route>

    <Route 2>

    Path in => out2

    </Route>

    It seems to be sending the logs 1 line at a time instead of being grouped by the headerline

  • motts's picture

    This actually seemed to create a bigger headache for me that I thought because every line of the log starts with some data that I don't care about, like this:

    Jun 11 03:24:03 tst-tst01-rp adevents:

    Is there a way to get rid of all of that junk before the headerline is defined? like make an exec to delete that stuff?

  • Zhengshi's picture
    (NXLog)

    I am not sure exactly the output you are looking for, but there are many ways to manage the content and output as needed.
    As far as removing those bits before being treated as an event, I don't know of a way.

    In the example below, the parse_syslog(); does most the work. It changes the first line into syslog style fields that JSON then packages nicely.
    This should get you enough info to form your output how you need. There is also a nice section in User Guide :
    https://nxlog.co/documentation/nxlog-user-guide#processing_rewrite

    For example, this config:

    <Input in>
      Module im_file
      File "motts_ml.txt"
      InputType log
      SavePos FALSE
      ReadFromLast FALSE
      Exec parse_syslog(); $raw_event = $Message;
      Exec $raw_event =~ s/---Begin event transaction---\n//;
      Exec $message = $raw_event; to_json();
    </Input>
    

    Example log output ( using PrettyPrint TRUE in xm_json directive):

    {
      "EventReceivedTime": "2018-06-12 12:06:02.631125-05:00",
      "SourceModuleName": "in",
      "SourceModuleType": "im_file",
      "SyslogFacilityValue": 1,
      "SyslogFacility": "USER",
      "SyslogSeverityValue": 5,
      "SyslogSeverity": "NOTICE",
      "SeverityValue": 2,
      "Severity": "INFO",
      "Hostname": "tst-tst01-rp",
      "EventTime": "2018-06-13 16:10:05.000000-05:00",
      "SourceName": "adevents",
      "Message": "More Stuff"
    }
    

  • motts's picture

    Hey,

    Thanks for working with me.

    I am running into an error when starting nxlog that says "parse_syslog(); doesn't exist or takes different arguments.

    Here is my input:
    <Input in>

    Module im_file

    File "C:\\Users\\Administrator\\Desktop\\srrlogs\\\*.log"
    InputType log
    SavePos FALSE
    ReadFromLast FALSE
    Exec parse_syslog(); $raw_event = $Message;
    Exec $raw_event =~ s/---Begin event transaction---\n///;
    Exec $message = $raw_event; to_json();

    </Input>

    I checked out the material you provided and it looks like I have it entered correctly, so I am not sure where the issue is.

  • Zhengshi's picture
    (NXLog)

    parse_syslog(); is a part of the Syslog Extension Module.
    https://nxlog.co/documentation/nxlog-user-guide#xm_syslog

    I am pretty sure you are missing part of the config.

    <Extension syslog>
        Module  xm_syslog
    </Extension>
    

  • motts's picture

    Maybe I'm missing something else here because my data is not coming out like yours.

    Sample data:

    Jun 10 03:39:04 tst-tst01-rp adevents: ---Begin event transaction---
    Jun 10 03:39:04 tst-tst01-rp adevents: Setting up rg parser...
    Jun 10 03:39:04 tst-tst01-rp adevents: Setting up rg2 parser...
    Jun 10 03:39:04 tst-tst01-rp adevents: Setting up rg3 parser...
    Jun 10 03:39:04 tst-tst01-rp adevents: Parsing event message...
    Jun 10 03:39:04 tst-tst01-rp adevents: Creating an Array of the parsed AD event...
    Jun 10 03:39:04 tst-tst01-rp adevents: Alert Level: 3
    Jun 10 03:39:04 tst-tst01-rp adevents: Rule: 18107 - Windows Logon Success.
    

    nxlog conf:

    <Extension log>
    
        Module  xm_multiline
        Headerline /^.+---Begin event transaction---/
        Exec    if $raw_event !~ /\: |---|Failed to join|\d{3,4}/ drop();
    
    </Extension>
    
    <Extension json>
    
        Module xm_json
    
    </Extension>
    
    <Extension syslog>
    
        Module  xm_syslog
    
    </Extension>
    
    
    <Input in>
    
        Module im_file
    
            File   "C:\\Users\\Administrator\\Desktop\\srrlogs\\\*.log"
            InputType       log
            SavePos         FALSE
            ReadFromLast    FALSE
            Exec            parse_syslog(); $raw_event = $Message;
            Exec            $raw_event =~ s/---Begin event transaction---\n//;
            Exec            $message = $raw_event; to_json();
    
    </Input>
    

    but my output is just like how it went in except with some "\n" json data in it. I would like to have all of "Jun 10 03:39:04 tst-tst01-rp adevents: " gone for every line if it were possible. It is separated by the Headerline, so that is working, just the Month, day, time, server, and "adevents: " is still there.

  • Zhengshi's picture
    (NXLog)

    Everything you are wanting is regex based. It might be useful to take a few minutes out and check out some PCRE regex tutorials and play around in some testers like https://regex101.com/ or https://learnxinyminutes.com/docs/pcre/.
    The \n characters are newline characters and they denote end of lines. If you do not need those, you can follow the same method used below to remove them and replace with a space or tab, or whatever you feel is best in your environment.

    Config:

    <Input in>
      Module im_file
      File "motts_ml.txt"
      InputType log
      SavePos FALSE
      ReadFromLast FALSE
      <Exec>
        parse_syslog();                                                     # Pulls apart $raw_event and gives us fields like $message
        $message =~ s/---Begin event transaction---\n//;                    # Removes --- line
        $message =~ s/(\w.+ \d\d \d\d:\d\d:\d\d) (\w+-\w+-\w+) (\w+): //g;  # Removes date,etc. note /g to replace global
        to_json();                                                          # Assembles fields into JSON
      </Exec>
    </Input>
    

    Output :

    {
        "EventReceivedTime": "2018-06-13 13:27:46.143385-05:00",
        "SourceModuleName": "in",
        "SourceModuleType": "im_file",
        "SyslogFacilityValue": 1,
        "SyslogFacility": "USER",
        "SyslogSeverityValue": 5,
        "SyslogSeverity": "NOTICE",
        "SeverityValue": 2,
        "Severity": "INFO",
        "Hostname": "tst-tst01-rp",
        "EventTime": "2018-06-10 03:39:04.000000-05:00",
        "SourceName": "adevents",
        "Message": "Setting up rg parser...\nSetting up rg2 parser...\nSetting up rg3 parser...\nParsing event message...\nCreating an Array of the parsed AD event...\nAlert Level: 3\nRule: 18107 - Windows Logon Success."
    }