Ask questions. Get answers. Find technical product solutions from passionate experts in the NXLog community.
Kafka output to Azure Event Hub
sera123k created
I have a SaaS vendor that uses NXLog to send logs to customers and I would like them to send to an Azure Event Hub that has Kafka enabled so we don't need to use VPNs. I'm able to connect and produce messages to the topic with python and logstash but they seem to be having issues with NXLog.
This is the error they tell me they are getting: 2020-04-09 09:41:27 ERROR rdkafka: [thrd:sasl_ssl://servername.servicebus.windows.net:9093/boots]: sasl_ssl://servername.servicebus.windows.net:9093/bootstrap: SASL GSSAPI mechanism handshake failed: Local: Broker transport failure: broker's supported mechanisms: (n/a) (after 0ms in state DOWN)
Azure seems to have specific requirements/limitations, the configuration below works for logstash and it uses sasl_jaas_config to pass the authentication data:
output {
kafka {
codec => json
topic_id => "kafkatopic"
bootstrap_servers => "eventhubname.servicebus.windows.net:9093"
client_id => "azshrelasea01p"
compression_type => "none"
sasl_jaas_config => "org.apache.kafka.common.security.plain.PlainLoginModule required username='$ConnectionString' password='Endpoint=<eventhubconnectionstring>';"
sasl_mechanism => "PLAIN"
security_protocol => "SASL_SSL"
ssl_truststore_password => "password"
ssl_truststore_location => "/elastic/trust.jks"
ssl_truststore_type => "jks"
batch_size => 250
linger_ms => 6000
}
}
Looking around the internet, it seems that librdkafka has some requirements that have to be met but I don't know how the nxlog config would look to implement these:
https://github.com/Azure/azure-event-hubs-for-kafka/issues/51
https://brokenco.de/2019/04/04/azure-eventhubs-rust.html
Anyone have any experience with this or could help me convert these logstash configs to a nxlog one? It looks like I need to use the Option function and pass the settings there for librdkafka but I don't have a lot of experience with NXLog.
Thanks!
sera123k created
om_kafka - ERROR Unable to produce message
bourazaniss created
I am trying to evaluate om_kafka module on RHEL system.
It starts ok and sends some messages from files to kafka but after a while these messages appear:
2018-03-13 14:52:52 ERROR Unable to produce message
2018-03-13 14:52:55 ERROR last message repeated 183044 times
Sometimes it can be fixed with NXLog service restart, while others not.
nxlog-3.99.3332_RHEL7_x86_64_trial
Any suggestion?
bourazaniss created