responses
Hello All. I am at my wits end here. I am trying to capture specific data in a line of text so I can display it in a Dashboard.
I am trying to capture the data from a .txt file. In the file the line looks like this:
Aug 09 09:00:08.076 CCSPAVCS01 VCS: [0x0000068c] ThrowingIncrementPolicy::increment1. newActiveCalls = 33 maxCalls = 220
The data I want is "newActiveCalls = 33". Basically, I am interested in all occurrences of that.
The section in NxLog for the log (called "messages.txt") basically looks like this:
<Input ccsp_logs>
Module im_file
File "C:\\temp\\messages.txt"
Exec parse_syslog();
Exec to_json();
</Input>
The output looks like this:
{"EventReceivedTime":"2018-08-13 11:25:52","SourceModuleName":"ccsp_logs","SourceModuleType":"im_file","SyslogFacilityValue":1,"SyslogFacility":"USER","SyslogSeverityValue":5,"SyslogSeverity":"NOTICE","SeverityValue":2,"Severity":"INFO","Hostname":"CCSPAVCS01","EventTime":"2018-08-09 09:00:08","SourceName":"VCS","Message":"[0x0000068c] ThrowingIncrementPolicy::increment1. newActiveCalls = 33 maxCalls = 220 "}
As you can see, the data I want (newActiveCalls = 33) is encapulated in the "Message" field created by NxLog. I have tried numerous things (to many to list here). Does anyone know a way to capture that specific data so I may extract the numeral (in this case, 33; which is the important part) to create a quantified graphic for my servers? (In this case, I am using Kibana. The end result being a graph showing: Server-1 with 33 active calls, Server-2 with 20 active calls, etc.)
Thank you all for your time and help.
Comments (3)
Hi Zhengshi. Thank you very much for taking the time to help me. Very kind of you. I am not familiar with REGEX. But I did try to implement your suggestions to some degree. Needless to say I was not able to isolate the data I wanted. However, I think there might be another solution. In another forum someone suggested to use pairing by use of the KVP module (xm_kvp). I am not familiar with KVP, so I am learning about it now, but it seems a more simpler and efficient solution. I would appreciate any insight you may have on KVP.
So the string in the log file I am interested in is:
Aug 09 09:00:08.076 CCSPAVCS01 VCS: [0x0000068c] ThrowingIncrementPolicy::increment1. newActiveCalls = 33 maxCalls = 220
The pair would be:
newActiveCalls = 33
I have this in nxlog.conf:
<Extension kvp>
Module xm_kvp
KVDelimiter =
KVPDelimiter ' '
</Extension>
<Input ccsp_logs>
Module im_file
File "C:\\temp\\messages.txt"
SavePos FALSE
ReadFromLast FALSE
#Exec pairs->parse_kvp();
Exec to_json();
</Input>
The result is twofold:
In the nxlog.log file I get these error messages like this, though not for every line in the log file:
2018-08-13 17:40:24 ERROR procedure 'parse_kvp' failed at line 33, character 26 in C:\Program Files (x86)\nxlog\conf\nxlog.conf. statement execution has been aborted; invalid KVP input: 'Aug 09 09:00:04.999 CCSPAVCS01 IMS: [0x000005c0] ~FastFileReader. Object: 0x16573d80 ' [state: 2]
Needless to day, the nxlog.log file grows very fast!
In my text.log file (my output) I get this:
{"EventReceivedTime":"2018-08-14 08:09:08","SourceModuleName":"ccsp_logs","SourceModuleType":"im_file","Aug 09 09:00:08.076 CCSPAVCS01 VCS: [0x0000068c] ThrowingIncrementPolicy::increment1. newActiveCalls":"33","maxCalls":"220"}
As you can see, the result I want is *almost* there. Unfortunately, this is the paring:
"Aug 09 09:00:08.076 CCSPAVCS01 VCS: [0x0000068c] ThrowingIncrementPolicy::increment1. newActiveCalls":"33"
instead of
"newActiveCalls":"33"
Thus, the question is: how can I pair the values "newActiveCalls" and "33" together? (This idea of paring using KVP appeals to me as there are my values in my particular log file that will benefit from this.)
Thank you again for any suggestions.
You'll need to separate the kvp part first using a regexp before you can parse it so that you only have the following:
newActiveCalls = 33 maxCalls = 220
Hi b0ti. Thank you taking the time to respond and for the tip. Learning how to do that now.
A big thanks to all who responded to my Post!