Event Logs / Json / Unable to parse at the other end

Tags:

#1 jonapich

While not systematic, nxlogs often sends partial json strings over its tcp output. The receiving end is unable to parse it.

The client configuration looks like this:

<Extension json>
  Module      xm_json
</Extension>
<Input eventlogs_json>
  Module      im_msvistalog
  Exec        delete($Keywords);
  Exec        delete($EventReceivedTime);
  Exec        delete($SourceModuleName);
  Exec        delete($SourceModuleType);
  Exec        $Environment = "development";
  Exec        to_json();
</Input>

Here's an example truncated log. There are 2 opening brackets but only 1 closed. And there's a weird \r in there too at the end.

"{\"EventTime\":\"2017-04-06 13:39:31\",\"Hostname\":\"redacted\",\"EventType\":\"INFO\",\"SeverityValue\":2,\"Severity\":\"INFO\",\"EventID\":40962,\"SourceName\":\"Microsoft-Windows-PowerShell\",\"ProviderGuid\":\"{A0C1853B-5C40-4B15-8766-3CF1C58F985A}\",\"Version\":1,\"Task\":4,\"OpcodeValue\":2,\"RecordNumber\":1206682,\"ActivityID\":\"{792C42B9-9C21-0001-F1DE-3079219CD201}\",\"ProcessID\":15728,\"ThreadID\":30916,\"Channel\":\"Microsoft-Windows-PowerShell/Operational\",\"Domain\":{\"EventTime\":\"2017-04-06 13:39:30\",\"Hostname\":\"redacted\",\"EventType\":\"INFO\",\"SeverityValue\":2,\"Severity\":\"INFO\",\"EventID\":400,\"SourceName\":\"PowerShell\",\"Task\":4,\"RecordNumber\":2672,\"ProcessID\":0,\"ThreadID\":0,\"Channel\":\"Windows PowerShell\",\"Message\":\"Engine state is changed from None to Available. \\r\\n\\r\\nDetails: \\r\\n\\tNewEngineState=Available\\r\\n\\tPreviousEngineState=None\\r\\n\\r\\n\\tSequenceNumber=13\\r\\n\\r\\n\\tHostName=ConsoleHost\\r\\n\\tHostVersion=4.0\\r\\n\\tHostId=6fc80cc3-85e0-4c9a-a437-1c05ddec479d\\r\\n\\tHostApplication=powershell.exe -command ($(Invoke-WebRequest http://127.0.0.1:redacted/fullstatus -UseBasicParsing).Content | ConvertFrom-Json).\\\"Status\\\"\\r\\n\\tEngineVersion=4.0\\r\\n\\tRunspaceId=cf67d6e2-814b-4d76-82cc-b138ceb2e122\\r\\n\\tPipelineId=\\r\\n\\tCommandName=\\r\\n\\tCommandType=\\r\\n\\tScriptName=\\r\\n\\tCommandPath=\\r\\n\\tCommandLine=\",\"Category\":\"Engine Lifecycle\",\"Opcode\":\"Info\",\"Environment\":\"development\"}\r"

I was able to parse it in python after removeing \r and adding an extra }.

#2 b0ti Nxlog ✓
#1 jonapich
While not systematic, nxlogs often sends partial json strings over its tcp output. The receiving end is unable to parse it. The client configuration looks like this: <Extension json> Module xm_json </Extension> <Input eventlogs_json> Module im_msvistalog Exec delete($Keywords); Exec delete($EventReceivedTime); Exec delete($SourceModuleName); Exec delete($SourceModuleType); Exec $Environment = "development"; Exec to_json(); </Input> Here's an example truncated log. There are 2 opening brackets but only 1 closed. And there's a weird \r in there too at the end. "{\"EventTime\":\"2017-04-06 13:39:31\",\"Hostname\":\"redacted\",\"EventType\":\"INFO\",\"SeverityValue\":2,\"Severity\":\"INFO\",\"EventID\":40962,\"SourceName\":\"Microsoft-Windows-PowerShell\",\"ProviderGuid\":\"{A0C1853B-5C40-4B15-8766-3CF1C58F985A}\",\"Version\":1,\"Task\":4,\"OpcodeValue\":2,\"RecordNumber\":1206682,\"ActivityID\":\"{792C42B9-9C21-0001-F1DE-3079219CD201}\",\"ProcessID\":15728,\"ThreadID\":30916,\"Channel\":\"Microsoft-Windows-PowerShell/Operational\",\"Domain\":{\"EventTime\":\"2017-04-06 13:39:30\",\"Hostname\":\"redacted\",\"EventType\":\"INFO\",\"SeverityValue\":2,\"Severity\":\"INFO\",\"EventID\":400,\"SourceName\":\"PowerShell\",\"Task\":4,\"RecordNumber\":2672,\"ProcessID\":0,\"ThreadID\":0,\"Channel\":\"Windows PowerShell\",\"Message\":\"Engine state is changed from None to Available. \\r\\n\\r\\nDetails: \\r\\n\\tNewEngineState=Available\\r\\n\\tPreviousEngineState=None\\r\\n\\r\\n\\tSequenceNumber=13\\r\\n\\r\\n\\tHostName=ConsoleHost\\r\\n\\tHostVersion=4.0\\r\\n\\tHostId=6fc80cc3-85e0-4c9a-a437-1c05ddec479d\\r\\n\\tHostApplication=powershell.exe -command ($(Invoke-WebRequest http://127.0.0.1:redacted/fullstatus -UseBasicParsing).Content | ConvertFrom-Json).\\\"Status\\\"\\r\\n\\tEngineVersion=4.0\\r\\n\\tRunspaceId=cf67d6e2-814b-4d76-82cc-b138ceb2e122\\r\\n\\tPipelineId=\\r\\n\\tCommandName=\\r\\n\\tCommandType=\\r\\n\\tScriptName=\\r\\n\\tCommandPath=\\r\\n\\tCommandLine=\",\"Category\":\"Engine Lifecycle\",\"Opcode\":\"Info\",\"Environment\":\"development\"}\r" I was able to parse it in python after removeing \r and adding an extra }.

The above JSON is escaped so it's not something that comes straight out of to_json() over TCP. You are not telling the full story.