5
responses

While not systematic, nxlogs often sends partial json strings over its tcp output. The receiving end is unable to parse it.

The client configuration looks like this:

<Extension json>
  Module      xm_json
</Extension>
<Input eventlogs_json>
  Module      im_msvistalog
  Exec        delete($Keywords);
  Exec        delete($EventReceivedTime);
  Exec        delete($SourceModuleName);
  Exec        delete($SourceModuleType);
  Exec        $Environment = "development";
  Exec        to_json();
</Input>

Here's an example truncated log. There are 2 opening brackets but only 1 closed. And there's a weird \r in there too at the end.

"{\"EventTime\":\"2017-04-06 13:39:31\",\"Hostname\":\"redacted\",\"EventType\":\"INFO\",\"SeverityValue\":2,\"Severity\":\"INFO\",\"EventID\":40962,\"SourceName\":\"Microsoft-Windows-PowerShell\",\"ProviderGuid\":\"{A0C1853B-5C40-4B15-8766-3CF1C58F985A}\",\"Version\":1,\"Task\":4,\"OpcodeValue\":2,\"RecordNumber\":1206682,\"ActivityID\":\"{792C42B9-9C21-0001-F1DE-3079219CD201}\",\"ProcessID\":15728,\"ThreadID\":30916,\"Channel\":\"Microsoft-Windows-PowerShell/Operational\",\"Domain\":{\"EventTime\":\"2017-04-06 13:39:30\",\"Hostname\":\"redacted\",\"EventType\":\"INFO\",\"SeverityValue\":2,\"Severity\":\"INFO\",\"EventID\":400,\"SourceName\":\"PowerShell\",\"Task\":4,\"RecordNumber\":2672,\"ProcessID\":0,\"ThreadID\":0,\"Channel\":\"Windows PowerShell\",\"Message\":\"Engine state is changed from None to Available. \\r\\n\\r\\nDetails: \\r\\n\\tNewEngineState=Available\\r\\n\\tPreviousEngineState=None\\r\\n\\r\\n\\tSequenceNumber=13\\r\\n\\r\\n\\tHostName=ConsoleHost\\r\\n\\tHostVersion=4.0\\r\\n\\tHostId=6fc80cc3-85e0-4c9a-a437-1c05ddec479d\\r\\n\\tHostApplication=powershell.exe -command ($(Invoke-WebRequest http://127.0.0.1:redacted/fullstatus -UseBasicParsing).Content | ConvertFrom-Json).\\\"Status\\\"\\r\\n\\tEngineVersion=4.0\\r\\n\\tRunspaceId=cf67d6e2-814b-4d76-82cc-b138ceb2e122\\r\\n\\tPipelineId=\\r\\n\\tCommandName=\\r\\n\\tCommandType=\\r\\n\\tScriptName=\\r\\n\\tCommandPath=\\r\\n\\tCommandLine=\",\"Category\":\"Engine Lifecycle\",\"Opcode\":\"Info\",\"Environment\":\"development\"}\r"

I was able to parse it in python after removeing \r and adding an extra }.

AskedApril 6, 2017 - 4:23pm

Comments (5)

  • jonapich's picture

    I can provide more details and context. Let me know if there's anything else that's missing that could help pinpoint the problem.

    Here's the full nxlog config:

    define ROOT C:\Program Files (x86)\nxlog
    
    Moduledir %ROOT%\modules
    CacheDir %ROOT%\data
    Pidfile %ROOT%\data\nxlog.pid
    SpoolDir %ROOT%\data
    LogFile %ROOT%\data\nxlog.log
    
    <Extension json>
      Module      xm_json
    </Extension>
    
    <Input eventlogs_json>
      Module      im_msvistalog
      Exec        delete($Keywords);
      Exec        delete($EventReceivedTime);
      Exec        delete($SourceModuleName);
      Exec        delete($SourceModuleType);
      Exec        $Environment = "development";
      Exec        to_json();
    </Input>
    
    <Output fluentd>
      Module      om_tcp
      Host        redacted.fqdn.com
      Port        24224
    </Output>
    
    <Route eventlogs_to_fluentd>
      Path        eventlogs_json => fluentd
    </Route>

    The fluentd server's receiving end is really simple:

    <source>
      @type tcp
      format json
      port 24224
      tag parse.tcp.windows
    </source>
    

    The error I pasted (the escaped json string) directly comes from fluentd's tcp module's log when it fails trying to parse the payload to json.

    I am puzzled because the vast majority of the logs do come through (Kibana indicates a sustained 1300+ records per 5 minutes). But some of them are not valid json, and thus they get discarded by fluentd's input module as invalid events.

  • jonapich's picture

    Great suggestion!

    In the resulting file, there is _not_ exactly one record per line. Long records are broken in 2 lines if they're larger than 1024 characters. Is this expected/normal?

    (still investigating those logs)

  • b0ti's picture
    (NXLog)

    I tested on a Windows 2008R2 using im_msvistalog with to_json() and the file produced does not contain any linebreaks. The longest line was 4247 characters so it exceeds 1024.

    I don't think the delete() statements would be responsible for the issue but it would be a good idea to trim your config as much as possible to a bare minimum.

Answers (0)