Ask questions. Get answers. Find technical product solutions from passionate experts in the NXLog community.
multiline message
toreno93 created
Hello. I have a question.
I get multiline messages
how can I combine into a single line, multiline message ??
for example this message, In this message 4 lines
Jul 21 17:59:10 <14> 1 2016-07-04T00: 53: 02.000000 + 03: 00 node = sec-sflow type = SYSCALL msg = audit (1467579182.055: 3248181): arch = 111
2 syscall = success = yes exit = 4 a0 = 7fc7783127a8 a1 = 2 a2 = a3 = 0 8 items = 1 ppid = 11013 pid = 30363 auid = 0 0 uid = gid = 0 = 0 euid
suid = 0 fsuid = 0 = 0 egid sgid = 0 = 0 fsgid tty = (none) ses = 28 comm = "sshd"
exe = "/ usr / sbin / sshd" key = "root_action"
Thank!
toreno93 created
Multiline Headerline Regex Error
chris.ried created
I am trying to use the multlog module in order to start ingesting a custom log:
I have the following regex: \^(\d{2}|\d).(\d{2}|\d).(\d{4})\s(\d\d|\d):(\d\d|\d):(\d\d|\d)\s(AM|PM).\[(.*)\](.*)
This works in a regex test; however I cannot get it to work with the log file that looks something like this
9/10/2015 11:29:16 AM [0-3-1-SecondaryPortStatus.cs-17] GetStatus for IP: 192.168.0.231 on port: 5016
9/10/2015 11:29:16 AM [0-3-1-SecondaryPortStatus.cs-47] <TRANSACTION>
<FUNCTION_TYPE>SECONDARYPORT</FUNCTION_TYPE>
<COMMAND>STATUS</COMMAND>
<MAC_LABEL>P_061</MAC_LABEL>
<MAC>az4FMuLbvrPz720bBeKWz3c+zBh6MsKVo4nJEW96B04=</MAC>
<COUNTER>217</COUNTER>
</TRANSACTION>
9/10/2015 11:29:16 AM [0-3-1-SecondaryPortStatus.cs-57] <RESPONSE>
<RESPONSE_TEXT>Operation SUCCESSFUL</RESPONSE_TEXT>
<RESULT>OK</RESULT>
<RESULT_CODE>-1</RESULT_CODE>
<TERMINATION_STATUS>SUCCESS</TERMINATION_STATUS>
<COUNTER>217</COUNTER>
<SECONDARY_DATA>10</SECONDARY_DATA>
<SERIAL_NUMBER>285498613</SERIAL_NUMBER>
</RESPONSE>
9/10/2015 11:29:16 AM [0-1-1-LandingPage.xaml.cs-49] POS opened
However when running the nxlog.conf for this I am getting the following error
2015-09-15 08:00:43 ERROR couldn't parse expression at line 12, character 13 in C:\Program Files (x86)\nxlog\conf\nxlog.conf; invalid character: '\'
I am unsure what i need to do in order to get this correct; does anyone have any insight or resources I should further explore. Is there a REGEX specific doc for NXLOG?
chris.ried created
multiline extension not getting the endline regex condition
mvf.right created
Hi,
I am trying to parse a log4net file into json.
Here's my sample log4net:
----------------
2015-01-27 01:06:18,859 [7] ERROR Web.Cms.Content.Base.Taxonomy.TaxonomyDetectionProvider [(null)] - Get taxonomy Type Failed for Tools
2015-01-27 06:34:31,051 [26] ERROR www.Status404 [(null)] - ErrorId: 20150127_102b01c6-3208-48c5-8c8b-ae4f92cf2b20
UserAgent: Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/39.0.2171.99 Safari/537.36
HostAddress: 192.168.10.2
RequestUrl: /ErrorPages/404.aspx
MachineName: QA01
Raw Url:/undefined/
Referrer: http://qa1.www.something.com/toolset.aspx
2015-01-27 06:34:33,270 [26] DEBUG Web.Caching.Core.CacheManagerBase [(null)] - Custom CacheProvider:Web.Caching.Core.AppFabricCacheManager,Web.Caching.Core Disabled
Now I am using xm_multiline to capture each log entries.
----------------
<Extension multiline>
Module xm_multiline
HeaderLine /^\d{4}\-\d{2}\-\d{2} \d{2}\:\d{2}\:\d{2},\d{3}/
EndLine /\r?\n\r?\n^\d{4}\-\d{2}\-\d{2} \d{2}\:\d{2}\:\d{2},\d{3}/
</Extension>
I use a regex to capture the timestamp as the header then I use a regex to capture twice newline then the next timestamp as endline. However it still treat the second and last entry as ONE log entry.
Here's the output:
----------------
{ "EventReceivedTime":"2015-01-27 01:06:35", "SourceModuleName":"log4net", "SourceModuleType":"im_file", "time":"2015-01-27 01:06:18,859", "thread":"7", "level":"ERROR", "logger":"Web.Cms.Content.Base.Taxonomy.TaxonomyDetectionProvider", "ndc":"(null)", "message":"Get taxonomy Type Failed for Tools"}{ "EventReceivedTime":"2015-01-27 06:34:35", "SourceModuleName":"log4net", "SourceModuleType":"im_file", "time":"2015-01-27 06:34:31,051", "thread":"26", "level":"ERROR", "logger":"www.Status404", "ndc":"(null)", "message":" ErrorId: 20150127_102b01c6-3208-48c5-8c8b-ae4f92cf2b20\r\n UserAgent: Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/39.0.2171.99
Safari/537.36\r\n HostAddress: 192.168.10.2\r\n RequestUrl: /ErrorPages/404.aspx\r\n MachineName: QA01\r\n
Raw Url:/undefined/\r\n Referrer: http://qa1.www.something.com/toolset.aspx\r\n\r\n2015-01-27 06:34:33,270 [26] DEBUG Web.Caching.Core.CacheManagerBase [(null)] - Custom CacheProvider:Web.Caching.Core.AppFabricCacheManager,Web.Caching.Core Disabled"}
I used this to produce that output:
----------------
Exec if $raw_event =~ /^(\d{4}\-\d{2}\-\d{2} \d{2}\:\d{2}\:\d{2},\d{3}) \[(\S+)\] (\S+) (\S+) \[(\S+)\] \- (.*)/s \
{ \
$time = $1; \
$thread = $2; \
$level = $3; \
$logger = $4; \
$ndc = $5; \
$message = $6; \
to_json(); \
} \
else \
{ \
drop(); \
}
I've also tried to tweak it by using this to avoid the combining the last two entries as one. However I am not able to get the last entry anymore.
----------------
Exec if $raw_event =~ /^(\d{4}\-\d{2}\-\d{2} \d{2}\:\d{2}\:\d{2},\d{3}) \[(\S+)\] (\S+) (\S+) \[(\S+)\] \- ([\s\S]*?)(\r?\n\r?\n|$)/ \
{ \
$time = $1; \
$thread = $2; \
$level = $3; \
$logger = $4; \
$ndc = $5; \
$message = $6; \
to_json(); \
} \
else \
{ \
drop(); \
}
mvf.right created