Splunk Message parsing

raf

Hi,

It would be easier to debug your config if you pasted it as a text - the forum supports markdown. Config file as a two-part screen isn't convenient to analyze ;)

Also, I'm not sure what's the problem? You've posted a screen, but letting know what have you expected would help with suggesting a solution. In other words - what's the question?

Best,

Rafal

AnsweredOctober 19, 2020 - 10:48pm

Leave a comment

Comments (9)

MB_318874
Leave a comment
Hi Raf,

Thank you for your time and answer.
The goal is to parse everything inside the "Messages"

This is my conf

## Please set the ROOT to the folder your nxlog was installed into,
## otherwise it will not start.
define ROOT C:\Program Files (x86)\nxlog
define ROOT_STRING C:\\Program Files (x86)\\nxlog
define CERTDIR %ROOT\\cert
Moduledir %ROOT%\modules
CacheDir %ROOT%\data
Pidfile %ROOT%\data\nxlog.pid
SpoolDir %ROOT%\data
LogFile %ROOT%\data\nxlog.log

<Extension _syslog>
Module xm_syslog
</Extension>

<Extension _json>
Module xm_json
</Extension>

<Input internal>
Module im_internal
Exec $Message = to_json();
</Input>

<Input eventlog>
Module im_msvistalog

Query <QueryList>\
<Query Id="0">\
<Select Path="Security">*</Select>\
<Select Path="Application">*</Select>\
<Select Path="System">*</Select>\
<Select Path="Windows Powershell">*</Select>\
<Select Path="Microsoft-Windows-Sysmon/Operational">*</Select>\
</Query>\
</QueryList>

Exec $Message = to_json();

Exec if ($Application =~ /nxlog\\nxlog.exe/) drop();

Exec if ($CommandLine =~ /"C:\\Program Files $x86$\\nxlog\\nxlog.exe" -c "C:\\Program Files $x86$\\nxlog\\conf\\nxlog.conf"/) drop();

</Input>

<Output out>
Module om_tcp
Host syslogIP(i change it ;) )
Port 514

Exec to_syslog_ietf(); $raw_event = replace($raw_event, 'NXLOG@12345', '4e7d9816-e12f-3cfe-bd84-5d9042eb2c97@41058 tag="windows"] [', 1);
</Output>

<Route 1>
Path internal, eventlog => out
</Route>

#######
This my RAW log after i keep only the json part
{"EventTime":"2020-10-21 01:15:12","Hostname":"wks01.domain.local","Keywords":36028797018963968,"EventType":"INFO","SeverityValue":2,"Severity":"INFO","EventID":800,"SourceName":"PowerShell","Task":8,"RecordNumber":3911888,"ProcessID":0,"ThreadID":0,"Channel":"Windows PowerShell","Message":"Pipeline execution details for command line: dir. \r\n\r\nContext Information: \r\n\tDetailSequence=1\r\n\tDetailTotal=1\r\n\r\n\tSequenceNumber=21\r\n\r\n\tUserId=domain\\username\r\n\tHostName=ConsoleHost\r\n\tHostVersion=5.1.18362.145\r\n\tHostId=1cfb6616-f938-4221-9970-b204702e8c61\r\n\tHostApplication=C:\\Windows\\System32\\WindowsPowerShell\\v1.0\\powershell.exe\r\n\tEngineVersion=5.1.18362.145\r\n\tRunspaceId=c9a23581-4e32-4eba-8b7e-778a7cfd541d\r\n\tPipelineId=6\r\n\tScriptName=\r\n\tCommandLine=dir \r\n\r\nDetails: \r\nCommandInvocation(Get-ChildItem): \"Get-ChildItem\"\r\n","Category":"Pipeline Execution Details","Opcode":"Info","EventReceivedTime":"2020-10-21 01:15:13","SourceModuleName":"eventlog","SourceModuleType":"im_msvistalog"}

And this is how i see it.

{ [-]
Category: Pipeline Execution Details
Channel: Windows PowerShell
EventID: 800
EventReceivedTime: 2020-10-21 01:15:13
EventTime: 2020-10-21 01:15:12
EventType: INFO
Hostname: wks01.domain.local
Keywords: 36028797018963970
Message: Pipeline execution details for command line: dir.

Context Information:
DetailSequence=1
DetailTotal=1

SequenceNumber=21

UserId=domain\username
HostName=ConsoleHost
HostVersion=5.1.18362.145
HostId=1cfb6616-f938-4221-9970-b204702e8c61
HostApplication=C:\Windows\System32\WindowsPowerShell\v1.0\powershell.exe
EngineVersion=5.1.18362.145
RunspaceId=c9a23581-4e32-4eba-8b7e-778a7cfd541d
PipelineId=6
ScriptName=
CommandLine=dir

Details:
CommandInvocation(Get-ChildItem): "Get-ChildItem"

Opcode: Info
ProcessID: 0
RecordNumber: 3911888
Severity: INFO
SeverityValue: 2
SourceModuleName: eventlog
SourceModuleType: im_msvistalog
SourceName: PowerShell
Task: 8
ThreadID: 0
}

My question is, can we have a configuration where the things inside the Message field already parse by the nxlog?

This part :

Message: Pipeline execution details for command line: dir.

Context Information:
DetailSequence=1
DetailTotal=1

SequenceNumber=21

UserId=domain\username
HostName=ConsoleHost
HostVersion=5.1.18362.145
HostId=1cfb6616-f938-4221-9970-b204702e8c61
HostApplication=C:\Windows\System32\WindowsPowerShell\v1.0\powershell.exe
EngineVersion=5.1.18362.145
RunspaceId=c9a23581-4e32-4eba-8b7e-778a7cfd541d
PipelineId=6
ScriptName=
CommandLine=dir

Details:
CommandInvocation(Get-ChildItem): "Get-ChildItem"

I have find some tips on the forum but i can't get a result :/... Windows XML log :'(

Thank you for your time and help!

October 21, 2020 - 10:23am

b0ti (NXLog)
Leave a comment
I think this is what you are looking for: https://nxlog.co/documentation/nxlog-user-guide/powershell-activity.html

October 21, 2020 - 9:11pm

MB_318874
Leave a comment
Ok thank, it try but nothing change

is my conf right?

le nxlog log file tel me he "couldn't parse statement at line 73, character 57 in C....nx.conf; procedure 'parse_kvp()' does not exist or takes different arguments

## Please set the ROOT to the folder your nxlog was installed into,
## otherwise it will not start.
define ROOT C:\Program Files (x86)\nxlog
define ROOT_STRING C:\\Program Files (x86)\\nxlog
define CERTDIR %ROOT\\cert
Moduledir %ROOT%\modules
CacheDir %ROOT%\data
Pidfile %ROOT%\data\nxlog.pid
SpoolDir %ROOT%\data
LogFile %ROOT%\data\nxlog.log

<Extension _syslog>
Module xm_syslog
</Extension>

<Extension _json>
Module xm_json
</Extension>

<Input internal>
Module im_internal
Exec $Message = to_json();
</Input>

<Input eventlog>
Module im_msvistalog

Query <QueryList>\
<Query Id="0">\
<Select Path="Security">*</Select>\
<Select Path="Application">*</Select>\
<Select Path="System">*</Select>\
<Select Path="Windows Powershell">*</Select>\
<Select Path="Microsoft-Windows-Sysmon/Operational">*</Select>\
</Query>\
</QueryList>

Exec $Message = to_json();

Exec if ($Application =~ /nxlog\\nxlog.exe/) drop();

Exec if ($CommandLine =~ /"C:\\Program Files $x86$\\nxlog\\nxlog.exe" -c "C:\\Program Files $x86$\\nxlog\\conf\\nxlog.conf"/) drop();

</Input>

<Extension kvp>
Module xm_kvp
KVPDelimiter ,
KVDelimiter =
</Extension>

<Extension json>
Module xm_json
</Extension>

<Input in>
Module im_msvistalog
<QueryXML>
<QueryList>
<Query Id="0" Path="Microsoft-Windows-PowerShell/Operational">
<Select Path="Microsoft-Windows-PowerShell/Operational">
*[System[EventID=4103]]</Select>
</Query>
</QueryList>
</QueryXML>
<Exec>
if defined($ContextInfo)
{
$ContextInfo = replace($ContextInfo, "\r\n", ",");
$ContextInfo = replace($ContextInfo, "\n", ",");
$ContextInfo = replace($ContextInfo, " ", "");
kvp->parse_kvp($ContextInfo, "ContextInfo_");
delete($ContextInfo);
delete($Message);
}
json->to_json();
</Exec>
</Input>

<Output out>
Module om_tcp
Host syslogIP(i change it ;) )
Port 514

Exec to_syslog_ietf(); $raw_event = replace($raw_event, 'NXLOG@12345', '4e7d9816-e12f-3cfe-bd84-5d9042eb2c97@41058 tag="windows"] [', 1);
</Output>

<Route 1>
Path internal, eventlog, in => out
</Route>

October 22, 2020 - 3:35pm

MB_318874
Leave a comment
Hello,

Any idea? :/ I'm stuck....

November 18, 2020 - 1:49pm

raf (NXLog)
Leave a comment
Hello,

first - the error messages themselves give us some clues about what's going on. Here, it's stated does not exist or takes different arguments.

The error comes from parse_kvp() method, and can be found in the reference manual. Here, we can see the whole configuration of the module.

In your case, simply remove the method's arguments, so change

kvp->parse_kvp($ContextInfo, "ContextInfo_");

to

kvp->parse_kvp();

You have at least one more error - the syntax used by you is inappropriate - instead of json->to_json(); you need use just to_json();.

Good luck!
Rafal
November 19, 2020 - 9:01am

MB_318874
Leave a comment
Thank you Raf!! it's working better :D

So i guess your powershell documentation is wrong, because i take it from there..?

November 23, 2020 - 4:04pm

MB_318874
Leave a comment
Because i can't parse what is have inside the Message part i guess because we remove the match of kvp->parse_kvp($ContextInfo, "ContextInfo_"); no ?

November 23, 2020 - 4:20pm

raf (NXLog)
Leave a comment
Could you point me to the exact place where did you take it from?

Thanks,
Rafal

November 23, 2020 - 4:40pm

MB_318874
Leave a comment
Hi Rafal,

first : thank you for your time and answer :)

Sure : https://nxlog.co/documentation/nxlog-user-guide/powershell-activity.html

In this guide they have the output like that :

"ContextInfo_Severity": "Informational",
"ContextInfo_Host Name": "ConsoleHost",
"ContextInfo_Host Version": "5.1.17763.592",
"ContextInfo_Host ID": "67d049eb-f3d6-4718-8cd2-b9dae30c4c7b",
"ContextInfo_Host Application": "C: \\Windows\\System32\\WindowsPowerShell\\v1.0\\powershell.exe",
"ContextInfo_Engine Version": "5.1.17763.592",
"ContextInfo_Runspace ID": "3145a9e1-18e3-4fa1-8700-fc78c783684b",

but because we remove the line where we have kvp->parse_kvp($ContextInfo, "ContextInfo_");

I don't have the output of this document. Maybe you need to update it?

Other topics on the powershell too. I use your documentation to configure the transcript powershell but when i use the configuration from this chapter : 113.2.3.
The configuration file : https://gitlab.com/nxlog-public/contrib/raw/master/guide_configs/powershell_transcription-nxlog.conf?inline=false

I got this when i start the nxlog agent

2020-11-24 21:19:02 ERROR Couldn't parse Exec block at C:\Program Files (x86)\nxlog\conf\nxlog.conf:140; couldn't parse statement at line 152, character 16 in C:\Program Files (x86)\nxlog\conf\nxlog.conf; invalid character: '$' (0x24)
2020-11-24 21:19:02 ERROR module 'transcription' has configuration errors, not adding to route '1' at C:\Program Files (x86)\nxlog\conf\nxlog.conf:194
2020-11-24 21:19:02 INFO connecting to 192.168.134.20:514
$ more C:\Users\Public\log.txt
0-11-24 21:18:41 WARNING stopping nxlog service
2020-11-24 21:18:41 WARNING nxlog-ce received a termination request signal, exiting...
2020-11-24 21:19:02 WARNING not starting unused module transcription
2020-11-24 21:19:02 INFO nxlog-ce-2.10.2102 started

Thank you for your answer

November 25, 2020 - 10:47am

You are here

Splunk Message parsing

Answer (1)

Comments (9)