Process ArcSight Common Event Format (CEF) logs

NXLog Enterprise Edition exclusive feature

CEF is a text-based log format developed by ArcSight™ and used by HP ArcSight™ products. It uses syslog as transport. The full format includes a syslog header or "prefix", a CEF "header", and a CEF "extension". The extension contains a list of key-value pairs. Standard key names are provided, and user-defined extensions can be used for additional key names. In some cases, the CEF format is used with the syslog header omitted.

NXLog can be configured to collect or forward logs in Common Event Format (CEF). NXLog has a dedicated xm_cef module for collecting and parsing CEF logs. NXLog can also be configured to generate and forward CEF logs using the same module. Additionally, NXLog’s xm_csv and xm_kvp modules can be used as an alternative to xm_cef for the same purpose of handling CEF logs.

CEF log syntax

Jan 11 10:25:39 host CEF:Version|Device Vendor|Device Product|Device Version|Device Event Class ID|Name|Severity|[Extension]

Log sample

Oct 12 04:16:11 localhost CEF:0|nxlog.org|nxlog|2.7.1243|Executable Code was Detected|Advanced exploit detected|100|src=192.168.255.110 spt=46117 dst=172.25.212.204 dpt=80

Collecting and parsing CEF logs

NXLog Enterprise Edition can be configured to collect and parse CEF logs with the xm_cef module.

The ArcSight™ Logger can be configured to send CEF logs via TCP with the following steps:

Log in to the Logger control panel.
Browse to Configuration Data Forwarders.
Click Add to create a new Forwarder:
- Name: nxlog
- Type: TCP Forwarder
- Type of Filter: Unified Query
Click Next to proceed to edit the new Forwarder:
- Query: (define as required)
- IP/Host: (enter the IP address or hostname of the system running NXLog)
- Port: 1514
Click Save.

Example 1. Collecting CEF logs

With this configuration, NXLog will collect CEF logs via TCP, convert them to plain JSON format, and save them to file.

nxlog.conf

<Extension cef>
    Module    xm_cef
</Extension>

<Extension json>
    Module    xm_json
</Extension>

<Extension syslog>
    Module    xm_syslog
</Extension>

<Input logger_tcp>
    Module    im_tcp
    Host      0.0.0.0
    Port      1514
    Exec      parse_syslog(); parse_cef($Message);
</Input>

<Output json_file>
    Module    om_file
    File      '/var/log/json'
    Exec      to_json();
</Output>

Generating and forwarding CEF logs

NXLog Enterprise Edition can be configured to generate and forward CEF logs with the xm_cef module.

The ArcSight™ Logger can be configured to receive CEF logs via TCP with the following steps:

Log in to the Logger control panel.
Browse to Configuration Data Receivers in the navigation menu.
Click Add to create a new Receiver:
- Name: nxlog
- Type: CEF TCP Receiver
Click Next to proceed to edit the new Receiver:
- Port: 574
- Encoding: UTF-8
- Source Type: CEF
Click Save.

Example 2. Forwarding CEF logs

With this configuration, NXLog will read syslog logs from file, convert them to CEF, and forward them to the ArcSight Logger via TCP. Default values will be used for the CEF header unless corresponding fields are defined in the event record (see the to_cef() procedure in the NXLog Reference Manual for a list of fields).

nxlog.conf

<Extension cef>
    Module    xm_cef
</Extension>

<Extension syslog>
    Module    xm_syslog
</Extension>

<Input messages_file>
    Module    im_file
    File      '/var/log/messages'
    Exec      parse_syslog();
</Input>

<Output logger_tcp>
    Module    om_tcp
    Host      192.168.1.1
    Port      574
    Exec      $Message = to_cef(); to_syslog_bsd();
</Output>

Using xm_csv and xm_kvp modules to handle CEF logs

Because NXLog Community Edition does not include the xm_cef module, the xm_csv and xm_kvp modules may be used instead to handle CEF logs.

The xm_csv and xm_kvp modules may not always correctly parse or generate CEF logs.

Example 3. Using xm_csv and xm_kvp to parse CEF logs

Here, the xm_csv module is used to parse the pipe-delimited CEF header, while the xm_kvp module is used to parse the space-delimited key-value pairs in the CEF extension. The required extension configurations are shown below.

nxlog.conf extensions

<Extension cef_header>
    Module          xm_csv
    Fields          $Version, $Device_Vendor, $Device_Product, $Device_Version, \
                    $Signature_ID, $Name, $Severity, $_Extension
    Delimiter       |
    QuoteMethod     None
</Extension>

<Extension cef_extension>
    Module          xm_kvp
    KVDelimiter     '='
    KVPDelimiter    ' '
    QuoteMethod     None
</Extension>

<Extension syslog>
    Module          xm_syslog
</Extension>

For CEF input, use an input instance like this one:

nxlog.conf input

<Input in>
    Module          im_tcp
    Host            0.0.0.0
    Port            1514
    <Exec>
        parse_syslog();
        cef_header->parse_csv($Message);
        cef_extension->parse_kvp($_Extension);
    </Exec>
</Input>

For CEF output, use an output instance like this one:

nxlog.conf output

<Output out>
    Module          om_tcp
    Host            192.168.1.1
    Port            574
    <Exec>
        $_Extension = cef_extension->to_kvp();
        $Version = 'CEF:0';
        $Device_Vendor = 'NXLog';
        $Device_Product = 'NXLog';
        $Device_Version = '';
        $Signature_ID = '0';
        $Name = '-';
        $Severity = '';
        $Message = cef_header->to_csv();
        to_syslog_bsd();
    </Exec>
</Output>

Disclaimer

While we endeavor to keep the information in this topic up to date and correct, NXLog makes no representations or warranties of any kind, express or implied about the completeness, accuracy, reliability, suitability, or availability of the content represented here. We update our screenshots and instructions on a best-effort basis.

Last revision: 25 September 2019