NXLog User Guide
- OS Support
- Enterprise Edition Reference Manual
- 146. Man Pages
- 147. Configuration
- 148. Language
- 149. Extension Modules
- 150. Input Modules
- 150.1. Process accounting (im_acct)
- 150.2. AIX auditing (im_aixaudit)
- 150.3. Azure (im_azure)
- 150.4. Batched compression (im_batchcompress)
- 150.5. Basic Security Module Auditing (im_bsm)
- 150.6. Check Point OPSEC LEA (im_checkpoint)
- 150.7. DBI (im_dbi)
- 150.8. Event Tracing for Windows (im_etw)
- 150.9. External programs (im_exec)
- 150.10. File (im_file)
- 150.11. File integrity monitoring (im_fim)
- 150.12. Go (im_go)
- 150.13. HTTP(s) (im_http)
- 150.14. Internal (im_internal)
- 150.15. Java (im_java)
- 150.16. Kafka (im_kafka)
- 150.17. Kernel (im_kernel)
- 150.18. Linux Audit System (im_linuxaudit)
- 150.19. macOS Endpoint Security (im_maces)
- 150.20. macOS ULS (im_maculs)
- 150.21. Mark (im_mark)
- 150.22. Event Logging for Windows XP/2000/2003 (im_mseventlog)
- 150.23. Event log for Windows 2008/Vista and later (im_msvistalog)
- 150.24. Null (im_null)
- 150.25. ODBC (im_odbc)
- 150.26. Packet capture (im_pcap)
- 150.27. Perl (im_perl)
- 150.28. Named pipes (im_pipe)
- 150.29. Python (im_python)
- 150.30. Redis (im_redis)
- 150.31. Windows Registry Monitoring (im_regmon)
- 150.32. Ruby (im_ruby)
- 150.33. TLS/SSL (im_ssl)
- 150.34. Systemd (im_systemd)
- 150.35. TCP (im_tcp)
- 150.36. Test Generator (im_testgen)
- 150.37. UDP (im_udp)
- 150.38. Unix domain sockets (im_uds)
- 150.39. Windows Performance Counters (im_winperfcount)
- 150.40. Windows Event Collector (im_wseventing)
- 150.41. ZeroMQ (im_zmq)
- 151. Processor Modules
- 152. Output Modules
- NXLog Manager
- NXLog Add-Ons
|To examine the supported platforms, see the list of installer packages in the Available Modules chapter.|
This mandatory directive specifies the list of Kafka brokers to connect to for collecting logs. The list should include ports and be comma-delimited (for example,
This mandatory directive specifies the Kafka topic to collect records from.
This specifies the path of the certificate authority (CA) certificate that will be used to verify the certificate presented by the remote brokers. A remote broker’s self-signed certificate (which is not signed by a CA) can be trusted by specifying the remote broker certificate itself. In case of certificates signed by an intermediate CA, the certificate specified must contain the complete certificate chain (certificate bundle). CAFile is required if Protocol is set to
This specifies the path of the certificate file that will be presented to the remote broker during the SSL handshake.
This specifies the path of the private key file that was used to generate the certificate specified by the CertFile directive. This is used for the SSL handshake.
This directive specifies the passphrase of the private key specified by the CertKeyFile directive. A passphrase is required when the private key is encrypted. Example to generate a private key with Triple DES encryption using OpenSSL:
$ openssl genrsa -des3 -out server.key 2048
This directive is not needed for passwordless private keys.
This directive can be used to pass a custom configuration property to the Kafka library (librdkafka). For example, the group ID string can be set with
Option group.id mygroup. This directive may be used more than once to specify multiple options. For a list of configuration properties, see the librdkafka CONFIGURATION.md file.Warning
Passing librdkafka configuration properties via the Option directive should be done with care since these properties are used for the fine-tuning of the librdkafka performance and may result in various side effects.
This optional integer directive specifies the topic partition to read from. If this directive is not given, messages are collected from partition 0.
This optional directive specifies the protocol to use for connecting to the Kafka brokers. Accepted values include
plaintext(the default) and
sasl_ssl. If Protocol is set to
sasl_ssl, then the CAFile directive must also be provided.
This directive specifies the Kerberos service name to be used for SASL authentication. The service name is required for the
This specifies the client’s Kerberos principal name for the
sasl_sslprotocols. This directive is only available and mandatory on Linux/UNIX. See note below.
Specifies the path to the kerberos keytab file which contains the client’s allocated principal name. This directive is only available and mandatory on Linux/UNIX.
The SASLKerberosServiceName and SASLKerberosPrincipal directives are only available on Linux/UNIX. On Windows, the login user’s principal name and credentials are used for SASL/Kerberos authentication.
For details about configuring Apache Kafka brokers to accept SASL/Kerberos
authentication from clients, please follow the instructions provided by the
When the im_kafka module reads a message from a broker, it creates and
populates the following fields which are then recorded to
Optional key associated with the message.
The following core fields are also created and populated by NXLog:
The time when the event is received. The value is not modified if the field already exists.
The name of the module instance, for input modules. The value is not modified if the field already exists.
The type of module instance (such as im_file), for input modules. The value is not modified if the field already exists.
This configuration collects events from a Kafka cluster using the brokers
specified. Events are read from the first partition of the
librdkafka library can produce its performance statistics and format it
in JSON. All fields from the JSON structure are explained on the
page of the
librdkafka project on the GitHub website. NXLog can be
configured to poll this data at a specified fixed interval. The result can be
saved to the internal logger.
To read statistical data of the
librdkafka library, the millisecond polling
interval needs to be specified against the Option
directive using the
To get the
librdkafka statistics produced and delivered synchronously, the
statistics.interval.ms option and the Schedule block should specify the
same interval amount.