10
responses

Hello,

After looking on the community forum i didn't really get the answer i was seeking.

I'm sending windows log to a syslog and to a splunk,

I got some issue about parsing the Message part as you can see : https://cdn.discordapp.com/attachments/700242491227635714/766300884971159562/unknown.png

This is my conf : https://cdn.discordapp.com/attachments/700242491227635714/766301478897451048/unknown.png
https://cdn.discordapp.com/attachments/700242491227635714/766301597541335060/unknown.png

Maybe this is not the good way to do it, i have to send in syslog because i'm sending it after to an elasticsearch and the splunk.

Thank you for your help :)

AskedOctober 15, 2020 - 4:10pm

Answer (1)

Hi,

It would be easier to debug your config if you pasted it as a text - the forum supports markdown. Config file as a two-part screen isn't convenient to analyze ;)

Also, I'm not sure what's the problem? You've posted a screen, but letting know what have you expected would help with suggesting a solution. In other words - what's the question?

Best,

Rafal

Comments (9)

  • MB_318874's picture

    Hi Raf,

    Thank you for your time and answer.
    The goal is to parse everything inside the "Messages"

    This is my conf

    ## Please set the ROOT to the folder your nxlog was installed into,
    ## otherwise it will not start.
    define ROOT C:\Program Files (x86)\nxlog
    define ROOT_STRING C:\\Program Files (x86)\\nxlog
    define CERTDIR %ROOT\\cert
    Moduledir %ROOT%\modules
    CacheDir %ROOT%\data
    Pidfile %ROOT%\data\nxlog.pid
    SpoolDir %ROOT%\data
    LogFile %ROOT%\data\nxlog.log

    <Extension _syslog>
    Module xm_syslog
    </Extension>

    <Extension _json>
    Module xm_json
    </Extension>

    <Input internal>
    Module im_internal
    Exec $Message = to_json();
    </Input>

    <Input eventlog>
    Module im_msvistalog

    Query <QueryList>\
    <Query Id="0">\
    <Select Path="Security">*</Select>\
    <Select Path="Application">*</Select>\
    <Select Path="System">*</Select>\
    <Select Path="Windows Powershell">*</Select>\
    <Select Path="Microsoft-Windows-Sysmon/Operational">*</Select>\
    </Query>\
    </QueryList>

    Exec $Message = to_json();

    Exec if ($Application =~ /nxlog\\nxlog.exe/) drop();

    Exec if ($CommandLine =~ /"C:\\Program Files \(x86\)\\nxlog\\nxlog.exe" -c "C:\\Program Files \(x86\)\\nxlog\\conf\\nxlog.conf"/) drop();

    </Input>

    <Output out>
    Module om_tcp
    Host syslogIP(i change it ;) )
    Port 514

    Exec to_syslog_ietf(); $raw_event = replace($raw_event, 'NXLOG@12345', '4e7d9816-e12f-3cfe-bd84-5d9042eb2c97@41058 tag="windows"] [', 1);
    </Output>

    <Route 1>
    Path internal, eventlog => out
    </Route>

    #######
    This my RAW log after i keep only the json part
    {"EventTime":"2020-10-21 01:15:12","Hostname":"wks01.domain.local","Keywords":36028797018963968,"EventType":"INFO","SeverityValue":2,"Severity":"INFO","EventID":800,"SourceName":"PowerShell","Task":8,"RecordNumber":3911888,"ProcessID":0,"ThreadID":0,"Channel":"Windows PowerShell","Message":"Pipeline execution details for command line: dir. \r\n\r\nContext Information: \r\n\tDetailSequence=1\r\n\tDetailTotal=1\r\n\r\n\tSequenceNumber=21\r\n\r\n\tUserId=domain\\username\r\n\tHostName=ConsoleHost\r\n\tHostVersion=5.1.18362.145\r\n\tHostId=1cfb6616-f938-4221-9970-b204702e8c61\r\n\tHostApplication=C:\\Windows\\System32\\WindowsPowerShell\\v1.0\\powershell.exe\r\n\tEngineVersion=5.1.18362.145\r\n\tRunspaceId=c9a23581-4e32-4eba-8b7e-778a7cfd541d\r\n\tPipelineId=6\r\n\tScriptName=\r\n\tCommandLine=dir \r\n\r\nDetails: \r\nCommandInvocation(Get-ChildItem): \"Get-ChildItem\"\r\n","Category":"Pipeline Execution Details","Opcode":"Info","EventReceivedTime":"2020-10-21 01:15:13","SourceModuleName":"eventlog","SourceModuleType":"im_msvistalog"}

    And this is how i see it.

    { [-]
    Category: Pipeline Execution Details
    Channel: Windows PowerShell
    EventID: 800
    EventReceivedTime: 2020-10-21 01:15:13
    EventTime: 2020-10-21 01:15:12
    EventType: INFO
    Hostname: wks01.domain.local
    Keywords: 36028797018963970
    Message: Pipeline execution details for command line: dir.

    Context Information:
    DetailSequence=1
    DetailTotal=1

    SequenceNumber=21

    UserId=domain\username
    HostName=ConsoleHost
    HostVersion=5.1.18362.145
    HostId=1cfb6616-f938-4221-9970-b204702e8c61
    HostApplication=C:\Windows\System32\WindowsPowerShell\v1.0\powershell.exe
    EngineVersion=5.1.18362.145
    RunspaceId=c9a23581-4e32-4eba-8b7e-778a7cfd541d
    PipelineId=6
    ScriptName=
    CommandLine=dir

    Details:
    CommandInvocation(Get-ChildItem): "Get-ChildItem"

    Opcode: Info
    ProcessID: 0
    RecordNumber: 3911888
    Severity: INFO
    SeverityValue: 2
    SourceModuleName: eventlog
    SourceModuleType: im_msvistalog
    SourceName: PowerShell
    Task: 8
    ThreadID: 0
    }

    My question is, can we have a configuration where the things inside the Message field already parse by the nxlog?

    This part :

    Message: Pipeline execution details for command line: dir.

    Context Information:
    DetailSequence=1
    DetailTotal=1

    SequenceNumber=21

    UserId=domain\username
    HostName=ConsoleHost
    HostVersion=5.1.18362.145
    HostId=1cfb6616-f938-4221-9970-b204702e8c61
    HostApplication=C:\Windows\System32\WindowsPowerShell\v1.0\powershell.exe
    EngineVersion=5.1.18362.145
    RunspaceId=c9a23581-4e32-4eba-8b7e-778a7cfd541d
    PipelineId=6
    ScriptName=
    CommandLine=dir

    Details:
    CommandInvocation(Get-ChildItem): "Get-ChildItem"

    I have find some tips on the forum but i can't get a result :/... Windows XML log :'(

    Thank you for your time and help!

  • MB_318874's picture

    Ok thank, it try but nothing change

    is my conf right?

    le nxlog log file tel me he "couldn't parse statement at line 73, character 57 in C....nx.conf; procedure 'parse_kvp()' does not exist or takes different arguments

    ## Please set the ROOT to the folder your nxlog was installed into,
    ## otherwise it will not start.
    define ROOT C:\Program Files (x86)\nxlog
    define ROOT_STRING C:\\Program Files (x86)\\nxlog
    define CERTDIR %ROOT\\cert
    Moduledir %ROOT%\modules
    CacheDir %ROOT%\data
    Pidfile %ROOT%\data\nxlog.pid
    SpoolDir %ROOT%\data
    LogFile %ROOT%\data\nxlog.log

    <Extension _syslog>
    Module xm_syslog
    </Extension>

    <Extension _json>
    Module xm_json
    </Extension>

    <Input internal>
    Module im_internal
    Exec $Message = to_json();
    </Input>

    <Input eventlog>
    Module im_msvistalog

    Query <QueryList>\
    <Query Id="0">\
    <Select Path="Security">*</Select>\
    <Select Path="Application">*</Select>\
    <Select Path="System">*</Select>\
    <Select Path="Windows Powershell">*</Select>\
    <Select Path="Microsoft-Windows-Sysmon/Operational">*</Select>\
    </Query>\
    </QueryList>

    Exec $Message = to_json();

    Exec if ($Application =~ /nxlog\\nxlog.exe/) drop();

    Exec if ($CommandLine =~ /"C:\\Program Files \(x86\)\\nxlog\\nxlog.exe" -c "C:\\Program Files \(x86\)\\nxlog\\conf\\nxlog.conf"/) drop();

    </Input>

    <Extension kvp>
    Module xm_kvp
    KVPDelimiter ,
    KVDelimiter =
    </Extension>

    <Extension json>
    Module xm_json
    </Extension>

    <Input in>
    Module im_msvistalog
    <QueryXML>
    <QueryList>
    <Query Id="0" Path="Microsoft-Windows-PowerShell/Operational">
    <Select Path="Microsoft-Windows-PowerShell/Operational">
    *[System[EventID=4103]]</Select>
    </Query>
    </QueryList>
    </QueryXML>
    <Exec>
    if defined($ContextInfo)
    {
    $ContextInfo = replace($ContextInfo, "\r\n", ",");
    $ContextInfo = replace($ContextInfo, "\n", ",");
    $ContextInfo = replace($ContextInfo, " ", "");
    kvp->parse_kvp($ContextInfo, "ContextInfo_");
    delete($ContextInfo);
    delete($Message);
    }
    json->to_json();
    </Exec>
    </Input>

    <Output out>
    Module om_tcp
    Host syslogIP(i change it ;) )
    Port 514

    Exec to_syslog_ietf(); $raw_event = replace($raw_event, 'NXLOG@12345', '4e7d9816-e12f-3cfe-bd84-5d9042eb2c97@41058 tag="windows"] [', 1);
    </Output>

    <Route 1>
    Path internal, eventlog, in => out
    </Route>

  • raf's picture
    (NXLog)

    Hello,

    first - the error messages themselves give us some clues about what's going on. Here, it's stated does not exist or takes different arguments.

    The error comes from parse_kvp() method, and can be found in the reference manual. Here, we can see the whole configuration of the module.

    In your case, simply remove the method's arguments, so change

    kvp->parse_kvp($ContextInfo, "ContextInfo_");
    

    to

    kvp->parse_kvp();
    

    You have at least one more error - the syntax used by you is inappropriate - instead of json->to_json(); you need use just to_json();.

    Good luck!
    Rafal

  • MB_318874's picture

    Hi Rafal,

    first : thank you for your time and answer :)

    Sure : https://nxlog.co/documentation/nxlog-user-guide/powershell-activity.html

    In this guide they have the output like that :

    "ContextInfo_Severity": "Informational",
    "ContextInfo_Host Name": "ConsoleHost",
    "ContextInfo_Host Version": "5.1.17763.592",
    "ContextInfo_Host ID": "67d049eb-f3d6-4718-8cd2-b9dae30c4c7b",
    "ContextInfo_Host Application": "C: \\Windows\\System32\\WindowsPowerShell\\v1.0\\powershell.exe",
    "ContextInfo_Engine Version": "5.1.17763.592",
    "ContextInfo_Runspace ID": "3145a9e1-18e3-4fa1-8700-fc78c783684b",

    but because we remove the line where we have kvp->parse_kvp($ContextInfo, "ContextInfo_");

    I don't have the output of this document. Maybe you need to update it?

    Other topics on the powershell too. I use your documentation to configure the transcript powershell but when i use the configuration from this chapter : 113.2.3.
    The configuration file : https://gitlab.com/nxlog-public/contrib/raw/master/guide_configs/powershell_transcription-nxlog.conf?inline=false

    I got this when i start the nxlog agent

    2020-11-24 21:19:02 ERROR Couldn't parse Exec block at C:\Program Files (x86)\nxlog\conf\nxlog.conf:140; couldn't parse statement at line 152, character 16 in C:\Program Files (x86)\nxlog\conf\nxlog.conf; invalid character: '$' (0x24)
    2020-11-24 21:19:02 ERROR module 'transcription' has configuration errors, not adding to route '1' at C:\Program Files (x86)\nxlog\conf\nxlog.conf:194
    2020-11-24 21:19:02 INFO connecting to 192.168.134.20:514
    $ more C:\Users\Public\log.txt
    0-11-24 21:18:41 WARNING stopping nxlog service
    2020-11-24 21:18:41 WARNING nxlog-ce received a termination request signal, exiting...
    2020-11-24 21:19:02 WARNING not starting unused module transcription
    2020-11-24 21:19:02 INFO nxlog-ce-2.10.2102 started

    Thank you for your answer