2
responses

We have the following problem.

We are currently migrating from REDIS to KAFKA on our windows server 2012 machines. In order to do this we are changing our nxlog.conf. (see below for entire configuration)

- We cannot seem to get our certificates to work. Something to note is that the same certificates work on a linux machine in the same environment (acceptance)
- Kafka is configured correctly, so that isn’t the problem. (considering that it works in a similar setup for a linux machine that connects to the same KAFKA topic with the same certificates.)
- When we start the nxlog service like this, the service basically is stuck on the starting phase of the service configuration. We need to force kill the PID to breath live back into it.
- We changed the .jks (java keystore) into a .p12 extension via a keystore manipulator (keystore explorer), since we do not have JAVA running on these windows machines.
- When we put Loglevel on DEBUG we get logging up until the kafka_out module but not further. (i.e. No information about handshaking etc.)
- In the configuration we tried
o Protocol SSL with CAFILE and either (CertKeyFile and KeyPass) or (Keyfile as a cer file)
o We didn’t try converting it to a PEM.
o I put the Partition parameter on comment, since we use multiple partitions in our Kafka-bus, so it wouldn’t make sense to aim for the first one all the time. (Noting that even if not commented out it also fails)

- We use a root and an intermediate certificate for our CA-path.
- There are hardly any examples for om_kafka modules, especially when you have more than 1 broker.
- We are running with version nxlog-4.5.4503

Information I could find on the WWW
# CAFile %CERTDIR%/cert-bundle.pem
# CAFile %CERTDIR%/cacert.pem
# CertFile %CERTDIR%/client-cert.pem
# CertKeyFile %CERTDIR%/client-key.pem
# KeyPass yourPasswordOfTheKey

#CAFile: This specifies the path of the certificate authority (CA) certificate, which will be used to check the certificate of the remote brokers. CAFile is required if Protocol is set to ssl or sasl_ssl.
#CertFile: This specifies the path of the certificate file to be used for the SSL handshake.
#CertKeyFile: This specifies the path of the certificate key file to be used for the SSL handshake.

Primary questions:
- Can CAFILE be used with a keystore or only with a .PEM / .CRT file?
- KeyPass is related to CertKeyFile. If there is a password on either CAFILE or KEYFILE how do I configure this then?
- Is there a higher level of debug so we can see what happens after the statements in our logging?

NXLOG.CONF
# --GENERAL SETTINGS-----------------

envvar COMPUTERNAME
define ROOT O:\NXLOG
define ENV PRD
define APPLI MyPortal

# KAFKA omgeving
define KAFKA_TOPIC mon_myportal_topic
define KAFKA_BROKERLIST_PROD 'kafka-0.intranet:9093,kafka-1.intranet:9093,kafka-2.intranet:9093,kafka-3.intranet:9093,kafka-4.intranet:9093,kafka-5.intranet:9093,kafka-6.intranet:9093,kafka-7.intranet:9093,kafka-8.intranet:9093,kafka-9.intranet:9093,kafka-10.intranet:9093,kafka-11.intranet:9093'

# ----------------------------------------------------------------------

Moduledir %ROOT%\modules
CacheDir %ROOT%\data
SpoolDir %ROOT%\data
CertDir %ROOT%\cert
LogLevel DEBUG
LogFile %ROOT%\log\nxlog.log

#Pidfile %ROOT%\test\nxlog.pid

# ----------------------------------------------------------------------

<Extension json>
Module xm_json
</Extension>

<output kafka_out>
Module om_kafka
BrokerList %KAFKA_BROKERLIST_PROD%
Topic %KAFKA_TOPIC%
# Partition 1
Protocol ssl
CAFile %CertDir%\client.truststore.p12
CertFile %CertDir%\intermediairportal-PRD.crt
# CertKeyFile %CertDir%\intermediairportal-PRD.p12
# KeyPass <password obfuscated>
</Output>

include %ROOT%\conf\check1.conf
include %ROOT%\conf\check2.conf

AskedOctober 18, 2019 - 11:18am

Answer (1)

We didn’t try converting it to a PEM.

There you go. :-)

Comments (1)

  • Richard_Blaauw's picture

    Hi,

    - We converted everything to .PEM
    - The problem we are running into right now is that It appears we are being blocked by something.
    - The following are the last lines of the log when we start NXLOG.

    I did a curl towards the kafka servers, and we can reach it, so to me it doesnt seem like its a firewall issue.

    2019-10-23 10:53:47 DEBUG nxlog config OK
    2019-10-23 10:53:47 DEBUG INIT: json
    2019-10-23 10:53:47 DEBUG INIT: kafka_out
    2019-10-23 10:53:47 DEBUG INIT: mail_check_status_input
    2019-10-23 10:53:47 DEBUG INIT: out_local_mail
    2019-10-23 10:53:47 DEBUG Pollset initialized for module out_local_mail (method: poll)
    2019-10-23 10:53:47 DEBUG INIT: file_check_status_input
    2019-10-23 10:53:47 DEBUG INIT: out_local_file
    2019-10-23 10:53:47 DEBUG Pollset initialized for module out_local_file (method: poll)
    2019-10-23 10:53:47 DEBUG INIT: force_check_status_input
    2019-10-23 10:53:47 DEBUG INIT: out_local_force
    2019-10-23 10:53:47 DEBUG Pollset initialized for module out_local_force (method: poll)
    2019-10-23 10:53:47 DEBUG parsing path: mail_check_status_input => out_local_mail
    2019-10-23 10:53:47 DEBUG adding module mail_check_status_input to route route_mail_checks
    2019-10-23 10:53:47 DEBUG adding module out_local_mail to route route_mail_checks
    2019-10-23 10:53:47 DEBUG parsing path: mail_check_status_input => kafka_out
    2019-10-23 10:53:47 DEBUG adding module mail_check_status_input to route route_mail_checks_kafka
    2019-10-23 10:53:47 DEBUG adding module kafka_out to route route_mail_checks_kafka
    2019-10-23 10:53:47 DEBUG parsing path: file_check_status_input => out_local_file
    2019-10-23 10:53:47 DEBUG adding module file_check_status_input to route route_file_checks
    2019-10-23 10:53:47 DEBUG adding module out_local_file to route route_file_checks
    2019-10-23 10:53:47 DEBUG parsing path: file_check_status_input => kafka_out
    2019-10-23 10:53:47 DEBUG adding module file_check_status_input to route route_file_checks_kafka
    2019-10-23 10:53:47 DEBUG adding module kafka_out to route route_file_checks_kafka
    2019-10-23 10:53:47 DEBUG parsing path: force_check_status_input => out_local_force
    2019-10-23 10:53:47 DEBUG adding module force_check_status_input to route route_force_checks
    2019-10-23 10:53:47 DEBUG adding module out_local_force to route route_force_checks
    2019-10-23 10:53:47 DEBUG parsing path: force_check_status_input => kafka_out
    2019-10-23 10:53:47 DEBUG adding module force_check_status_input to route route_force_checks_kafka
    2019-10-23 10:53:47 DEBUG adding module kafka_out to route route_force_checks_kafka

    To me it appears that nxlog is routing kafka_out to our out configuration and then its being stalled. (ie. We dont even reach a handshaking phase)

    Any suggestions?