How to Capture Specific Data

Post a different question
4
responses

Hello All. I am at my wits end here. I am trying to capture specific data in a line of text so I can display it in a Dashboard.

I am trying to capture the data from a .txt file. In the file the line looks like this:
Aug 09 09:00:08.076 CCSPAVCS01 VCS: [0x0000068c] ThrowingIncrementPolicy::increment1. newActiveCalls = 33 maxCalls = 220

The data I want is "newActiveCalls = 33". Basically, I am interested in all occurrences of that.

The section in NxLog for the log (called "messages.txt") basically looks like this:
<Input ccsp_logs>
Module im_file
File "C:\\temp\\messages.txt"
Exec parse_syslog();
Exec to_json();
</Input>

The output looks like this:
{"EventReceivedTime":"2018-08-13 11:25:52","SourceModuleName":"ccsp_logs","SourceModuleType":"im_file","SyslogFacilityValue":1,"SyslogFacility":"USER","SyslogSeverityValue":5,"SyslogSeverity":"NOTICE","SeverityValue":2,"Severity":"INFO","Hostname":"CCSPAVCS01","EventTime":"2018-08-09 09:00:08","SourceName":"VCS","Message":"[0x0000068c] ThrowingIncrementPolicy::increment1. newActiveCalls = 33 maxCalls = 220 "}

As you can see, the data I want (newActiveCalls = 33) is encapulated in the "Message" field created by NxLog. I have tried numerous things (to many to list here). Does anyone know a way to capture that specific data so I may extract the numeral (in this case, 33; which is the important part) to create a quantified graphic for my servers? (In this case, I am using Kibana. The end result being a graph showing: Server-1 with 33 active calls, Server-2 with 20 active calls, etc.)

Thank you all for your time and help.

AskedAugust 13, 2018 - 11:31am

Answer (1)

If you are still wanting the output in JSON where the entire $Message field is just the numerical value, you will need to use REGEX and capture groups.
Below is an example assuming that the input file is just the one type of event entry. If you have other types you will need to add additional if statements to drop or manage the events as needed.
The log_info() line is just so I can see the $Message field while nxlog is running in the foreground (-f) to make sure the regex works and is useful while troubleshooting things like this.

<Input ccsp_logs>
        Module im_file
        File 'capturegroup.txt'
        ReadFromLast    False
        SavePos False
        Exec parse_syslog();
        <Exec>
                if $raw_event =~ /^.+newActiveCalls\s=\s(\d+)\s.+/ $Message  = $1;
        </Exec>
        Exec log_info("IN MSG : " + $Message);
        Exec to_json();
</Input>
AnsweredAugust 13, 2018 - 3:28pm

Comments (3)

  • NxlogKinz's picture

    Hi Zhengshi. Thank you very much for taking the time to help me. Very kind of you. I am not familiar with REGEX. But I did try to implement your suggestions to some degree. Needless to say I was not able to isolate the data I wanted. However, I think there might be another solution. In another forum someone suggested to use pairing by use of the KVP module (xm_kvp). I am not familiar with KVP, so I am learning about it now, but it seems a more simpler and efficient solution. I would appreciate any insight you may have on KVP.

    So the string in the log file I am interested in is:
    Aug 09 09:00:08.076 CCSPAVCS01 VCS: [0x0000068c] ThrowingIncrementPolicy::increment1. newActiveCalls = 33 maxCalls = 220

    The pair would be:
    newActiveCalls = 33

    I have this in nxlog.conf:

    <Extension kvp>
    Module xm_kvp
    KVDelimiter =
    KVPDelimiter ' '
    </Extension>

    <Input ccsp_logs>
    Module im_file
    File "C:\\temp\\messages.txt"
    SavePos FALSE
    ReadFromLast FALSE
    #Exec pairs->parse_kvp();
    Exec to_json();
    </Input>

    The result is twofold:

    In the nxlog.log file I get these error messages like this, though not for every line in the log file:
    2018-08-13 17:40:24 ERROR procedure 'parse_kvp' failed at line 33, character 26 in C:\Program Files (x86)\nxlog\conf\nxlog.conf. statement execution has been aborted; invalid KVP input: 'Aug 09 09:00:04.999 CCSPAVCS01 IMS: [0x000005c0] ~FastFileReader. Object: 0x16573d80 ' [state: 2]

    Needless to day, the nxlog.log file grows very fast!

    In my text.log file (my output) I get this:
    {"EventReceivedTime":"2018-08-14 08:09:08","SourceModuleName":"ccsp_logs","SourceModuleType":"im_file","Aug 09 09:00:08.076 CCSPAVCS01 VCS: [0x0000068c] ThrowingIncrementPolicy::increment1. newActiveCalls":"33","maxCalls":"220"}

    As you can see, the result I want is *almost* there. Unfortunately, this is the paring:
    "Aug 09 09:00:08.076 CCSPAVCS01 VCS: [0x0000068c] ThrowingIncrementPolicy::increment1. newActiveCalls":"33"

    instead of

    "newActiveCalls":"33"

    Thus, the question is: how can I pair the values "newActiveCalls" and "33" together? (This idea of paring using KVP appeals to me as there are my values in my particular log file that will benefit from this.)

    Thank you again for any suggestions.

    August 14, 2018 - 8:51am
  • b0ti's picture
    (NXLog)

    You'll need to separate the kvp part first using a regexp before you can parse it so that you only have the following:

    newActiveCalls = 33 maxCalls = 220

    August 14, 2018 - 9:00am