xm_cef, xm_json unexpected behaviour while converting CEF to JSON

Tags:

#1 himanshu.arora

While evaluating the NXLOG enterprise trial edition, we faced a blocker and I need some clarification/help on the same.

We are using NXLOG’s CEF module (xm_cef, xm_json ) which convert’s CEF messages into JSON. It is working properly for most of the cases but giving unexpected output for few of them.

raw CEF message :- CEF:0|Himanshu Arora|Sample1|10.5.011|195|Process Sample|5|abc=Sample Data suser=XY fname= dvc= shost=10.1.1.1 dhost= duser= externalId= app= reason= cs1Label=Affected User cs1= cs2Label=Safe Name cs2=Notification Sample cs3Label=Device Sample cs3= cs4Label=Database cs4= cs5Label="Other info" cs5= cn1Label=Request Id cn1= cn2Label=Ticket Id cn2= msg= JSON output :- { "EventReceivedTime": "2019-04-25T13:43:49.483942+05:30", "SourceModuleName": "cef_input", "SourceModuleType": "im_file", "SyslogFacilityValue": 1, "SyslogFacility": "USER", "SyslogSeverityValue": 5, "SyslogSeverity": "NOTICE", "SeverityValue": 3, "Severity": "WARNING", "EventTime": "2019-04-25T13:43:49.483969+05:30", "Hostname": "himanshu-VirtualBox", "SourceName": "CEF", "CEFVersion": 0, "CEFDeviceVendor": "Himanshu Arora", "CEFDeviceProduct": "Sample1", "CEFDeviceVersion": "10.5.011", "CEFSignatureID": "195", "CEFName": "Process Sample", "CEFSeverity": 5, "abc": "Sample Data", "suser": "XY", "fname": "dvc=", "shost": "10.1.1.1", "dhost": "duser=" }

If you notice the raw message has some fields called cs1Label,cs2Label,cs2,cn1Label,cn2Label, cn2 . these fields are missing in the JSON output file. Moreover in JSON the fields "fname" , "dhost" should have had empty value.

I would like to know

  1. If this issue exists only in the enterprise trial edition and it will be resolved if we purchase the Enterprise edition ? or is it issue being fixed and will be released soon?
  2. Is there a way to include any third party libraries into NXLOG that can convert CEF to JSON.
#2 Zhengshi Nxlog ✓
#1 himanshu.arora
While evaluating the NXLOG enterprise trial edition, we faced a blocker and I need some clarification/help on the same. We are using NXLOG’s CEF module (xm_cef, xm_json ) which convert’s CEF messages into JSON. It is working properly for most of the cases but giving unexpected output for few of them. raw CEF message :- CEF:0|Himanshu Arora|Sample1|10.5.011|195|Process Sample|5|abc=Sample Data suser=XY fname= dvc= shost=10.1.1.1 dhost= duser= externalId= app= reason= cs1Label=Affected User cs1= cs2Label=Safe Name cs2=Notification Sample cs3Label=Device Sample cs3= cs4Label=Database cs4= cs5Label="Other info" cs5= cn1Label=Request Id cn1= cn2Label=Ticket Id cn2= msg= JSON output :- { "EventReceivedTime": "2019-04-25T13:43:49.483942+05:30", "SourceModuleName": "cef_input", "SourceModuleType": "im_file", "SyslogFacilityValue": 1, "SyslogFacility": "USER", "SyslogSeverityValue": 5, "SyslogSeverity": "NOTICE", "SeverityValue": 3, "Severity": "WARNING", "EventTime": "2019-04-25T13:43:49.483969+05:30", "Hostname": "himanshu-VirtualBox", "SourceName": "CEF", "CEFVersion": 0, "CEFDeviceVendor": "Himanshu Arora", "CEFDeviceProduct": "Sample1", "CEFDeviceVersion": "10.5.011", "CEFSignatureID": "195", "CEFName": "Process Sample", "CEFSeverity": 5, "abc": "Sample Data", "suser": "XY", "fname": "dvc=", "shost": "10.1.1.1", "dhost": "duser=" } If you notice the raw message has some fields called cs1Label,cs2Label,cs2,cn1Label,cn2Label, cn2 . these fields are missing in the JSON output file. Moreover in JSON the fields "fname" , "dhost" should have had empty value. I would like to know If this issue exists only in the enterprise trial edition and it will be resolved if we purchase the Enterprise edition ? or is it issue being fixed and will be released soon? Is there a way to include any third party libraries into NXLOG that can convert CEF to JSON.

It could be helpful to have NXLog Version, config, and nxlog.log output when posting issues in the future.

  1. Is there a way to include any third party libraries into NXLOG that can convert CEF to JSON.

NXLog EE supports external Ruby, Python, Perl , and shell scripts in order to process messages.

  1. If this issue exists only in the enterprise trial edition and it will be resolved if we purchase the Enterprise edition ? or is it issue being fixed and will be released soon?

We have an internal issue for this currently. It affects trial and EE edition. I do not have an ETA as of yet.