Apache NiFi
Apache NiFi is a platform for real-time data ingestion and distribution among various source and destination systems supporting a wide variety of data formats and protocols. NXLog can be configured to send and receive log data from Apache NiFi, supporting several different protocols.
Installation and configuration of Apache NiFi
Apache NiFi runs within a JVM on a host connected to your local network.
For details on the Apache NiFi installation process, visit the How to install and start NiFi section of the Apache NiFi System Administrator’s Guide.
After Apache NiFi has been started, it can be accessed at http://localhost:8080/nifi.
Configuring Apache NiFi processors
To work with data, Apache NiFi uses processors.
Processors are categorized by their names. For example, the names of data ingestion processors start with Get or Listen, such as GetHTTP or ListenTCP. The names of data sending processors start with Put, like PutUDP or PutSyslog.
The following steps lay out the procedure of configuring Apache NiFi to exchange log data from NXLog. For this example, the configuration of the ListenTCP processor is used. These configuration steps are carried out in the Apache NiFi environment by placing components on the canvas.
To add and configure a new processor, follow these steps:
-
From the toolbar, drag and drop a Processor component onto the canvas.
This will open the Add Processor dialog window.
-
Find the required processor from the list and click ADD, then right-click the processor pictogram and select Configure.
-
In the Configure Processor dialog window under the SETTINGS tab, configure the automatic termination of relationships according to your needs.
-
Under the PROPERTIES tab, configure all required fields in bold.
-
Once the configuration is complete, click APPLY.
For an in-depth explanation of all options, visit the Configuring a Processor section in the Apache NiFi documentation.
Processors are the basic blocks for creating dataflows. After the processor configuration has been finished, they should be arranged and bound to create dataflows. For the details of creating data flows, visit the Building a DataFlow section in the Apache NiFi documentation.
Sending logs to Apache NiFi
This example demonstrates how to configure NXLog to forward logs to Apache NiFi via TCP. In this instance, the ListenTCP processor is used on the Apache NiFi side to receive the data.
The configuration below reads log messages using the im_kernel module from a Unix-like operating system. After reading, the data is parsed and converted to JSON, and the output is sent to Apache NiFi using the om_tcp module.
<Extension json>
Module xm_json
</Extension>
<Extension syslog>
Module xm_syslog
</Extension>
<Input from_kernel>
Module im_kernel
</Input>
<Output to_tcp>
Module om_tcp
Host 192.168.31.53:10000
<Exec>
parse_syslog();
to_json();
</Exec>
</Output>This is the sample data formatted to JSON before sending it to Apache NiFi.
{
"SourceName": "kernel",
"EventReceivedTime": "2021-03-10T18:53:56.082022+03:00",
"SourceModuleName": "from_kernel",
"SourceModuleType": "im_kernel",
"SyslogFacilityValue": 0,
"SyslogFacility": "KERN",
"SyslogSeverityValue": 7,
"SyslogSeverity": "DEBUG",
"SeverityValue": 1,
"Severity": "DEBUG",
"EventTime": "2021-03-10T18:57:45.064529+03:00",
"Hostname": "Ubuntu-VM",
"Message": "[ 29.055026] ISO 9660 Extensions: RRIP_1991A"
}To forward log data over SSL/TLS, both NXLog and Apache NiFi need additional configuration to work with certificates and private keys. NXLog uses separate directives to specify certificates and keys, while Apache NiFi can be configured to work with keystores and truststores. For more information about the SSL/TLS configuration of Apache NiFI, see the Security Configuration section of the Apache NiFi Documentation.
To establish a secure connection, Apache NiFi can use the ListenTCP processor with the Port property configured to listen on port 10000. Additionally, you should set SSL Context Service to work with the keystore. Also, you need to set the Client Auth property to the correct client authentication policy.
The NXLog configuration below uses the im_etw module to collect DNS traffic data. Using the xm_json module, all data are converted to JSON, and the om_ssl module is applied to send data to Apache NiFi over SSL/TLS.
<Extension json>
Module xm_json
</Extension>
<Input from_dns>
Module im_etw
Provider Microsoft-Windows-DNS-Client
</Input>
<Output to_nifi_ssl>
Module om_ssl
Host 192.168.31.53:10000
Exec to_json();
AllowUntrusted TRUE
CertFile C:\certs\myCertificate.crt
CertKeyFile C:\certs\myPrivateKey.pem
</Output>Below is the output sample of the data forwarded to Apache NiFi.
{
"SourceName": "Microsoft-Windows-DNS-Client",
"ProviderGuid": "{1C95126E-7EEA-49A9-A3FE-A378B03DDB4D}",
"EventID": 1019,
"Version": 0,
"ChannelID": 16,
"OpcodeValue": 0,
"TaskValue": 1019,
"Keywords": "9223372036854775808",
"EventTime": "2021-03-09T07:28:14.084075-08:00",
"ExecutionProcessID": 104,
"ExecutionThreadID": 1228,
"EventType": "INFO",
"SeverityValue": 2,
"Severity": "INFO",
"Hostname": "WIN-ET85AK2E1J1",
"Domain": "NT AUTHORITY",
"AccountName": "NETWORK SERVICE",
"UserID": "S-1-5-20",
"AccountType": "Well Known Group",
"Flags": "EXTENDED_INFO|IS_64_BIT_HEADER|PROCESSOR_INDEX (577)",
"Location": "0",
"Context": "0",
"EventReceivedTime": "2021-03-09T07:28:15.084417-08:00",
"SourceModuleName": "from_dns",
"SourceModuleType": "im_etw"
}Collecting logs from Apache NiFi
While Apache NiFi can send logs using multiple processors to NXLog, NXLog can receive data using almost any network protocol so there several ways to send data from Apache NiFi to NXLog.
One way to send data to NXLog from Apache NiFi is to utilize the PutSyslog processor with its Hostname and Protocol properties specified. This type of NiFi configuration is the basis for the following example.
This configuration uses the im_udp module to collect data from Apache NiFi. Once received, logs can be saved to file or forwarded over the network.
<Extension syslog>
Module xm_syslog
</Extension>
<Input from_udp>
Module im_udp
ListenAddr 192.168.31.41:10000
Exec parse_syslog();
</Input>Verifying data in Apache NiFi
The user interface of Apache NiFi allows verifying data traffic for the whole flow and by processor.
-
To see the overall data traffic, click Global Menu in the upper right corner of the NiFi user interface and select Data Provenance.
This will open the Data Provenance dialog window as per below.
-
To view the input traffic of a specific processor, you can right-click a processor on the NiFi canvas and select View data provenance.
In the NiFi Data Provenance dialog window, you can observe the data traffic details for the selected processor.
