NxLog multiline to ship data that only contains a colon

Post a different question
7
responses

Hello there,

I am working with a multiline module. This particular file has 38 lines, but I'd like to only ship the lines that contain a colon. Is there a way to write an exec that if the line does not contain a colon then drop the line?

Here is my config so far:

<Extension log>

Module xm_multiline
#FixedLineCount
HeaderLine /^---Begin event transaction---/
#EndLine /^---Event Reporting Complete---/
</Extension>

<Extension json>

Module xm_json

</Extension>

<Input in>

Module im_file

File "C:\\Users\\Administrator\\Desktop\\SRR_Error.txt"
InputType log
SavePos FALSE
ReadFromLast FALSE
Exec $message = $raw_event; to_json();

</Input>

Thanks,

AskedMay 23, 2018 - 4:50pm

Answer (1)

You can use the Exec directive inside the xm_multiline instance to ignore lines as follows:

<Extension log>
    Module      xm_multiline
    HeaderLine  ...
    Exec        if $raw_event !~ /\;/ drop();
</Extension>
AnsweredMay 23, 2018 - 4:57pm

Comments (6)

  • motts's picture

    Hmm that doesn't seem to work. I think it's because the entirety of the message contains a colon, so it is a true statement.

    What I need is only keeping the data that contains a colon after it's been Exec to_json.

    So...

    ---Begin
    walking
    stop:hammertime
    ---end

    This will ship as
    message = ---Begin\r\nwalking\r\nstop:hammertime\r\n---end\r\n

    And I would only like to have it ship "stop:hammertime"

    Does that make sense?

    May 23, 2018 - 9:45pm
  • b0ti's picture
    (NXLog)

    The regexp actually had a semicolon instead of the colon. This should do that now:

    Exec        if $raw_event !~ /\:/ drop();

    To also remove the header and trailer lines you could add this:

    Exec      if $raw_event =~ /^---\S+$/ drop();

    May 23, 2018 - 9:52pm
  • motts's picture

    Haha yeah, my bad on spotting that.

    That worked pretty well actually, with the exception that every line is now it's own log instead of them being in 1 log.

    In the Input section, I do have:

    Exec $message = $raw_event; to_json;

    but logs are still separate.

    If I comment out "Exec if $raw_event !~ /\:/ drop();" in the Extension section, then the entry ships as one log as intended.

    May 23, 2018 - 10:13pm
  • motts's picture

    I added Exec $raw_event = "--" + $raw_event; to place -- before each raw event and got this in return

    --Member Name: -
    --Member ID: %{S-1-5-21-26028188-150678075-188441444-171629}
    --Target Account Name: TestAccount
    --Target Domain: TestDomain
    --Target Account ID: %{S-1-5-21-26028188-150678075-188441444-110557}
    --Caller User Name: TestServer
    --Caller Domain: TestDomain
    --Caller Logon ID: (0x0,0x4FC530E)
    --Privileges: -
    --Failed to join event information because: local variable 'domain1' referenced before assignment
    --Failure, the script has reached an error: local variable 'domain1' referenced before assignment

    When theoretically it should only have the 2 dashes on the first entry.

    May 23, 2018 - 10:42pm
  • motts's picture

    Ah, figured that out. The Exec you suggest I add is dropping the headerline since it doesn't contain a colon. I added a colon to the headerline and the test file and it worked as expected. Is there a way to make that exec have an or statement like:

    Exec if $raw_event !~ /\: / or /---/ drop();

    This doesn't work obviously :(

    May 23, 2018 - 10:51pm
  • motts's picture

    Sorry for the quick comments back. I fixed it by adding a | and then the dashes in the header/end lines. Now if the lines contain a colon or 3 dashes, it keeps the line.

    Exec if $raw_event !~ /\: |---/ drop();

    Thanks a lot for you help. I appreciate it very much.

    May 23, 2018 - 10:57pm