Ask questions. Get answers. Find technical product solutions from passionate experts in the NXLog community.
Elasticsearch with Community Edition
akumar created
Is the Output module to ElasticSearch available/will be available in the community edition?
When I last checked it was a feature of the commerical edition only.
I have been a nxlog champion for years now and have been forwarding to logstash. However with Elasticsearch ingest nodes, there is one extra redundent step now.
This will tip the scale to moving off into native beats/rsyslog if there are no plans to make this available.
Thanks
Ash Kumar
akumar created
Issues with data upload to ElasticSearch
jppacheco created
Hello everyone,
This is my first time posting in this community forum, so any help would be greatly appreciated.
I've been working with NxLog and ElasticSearch for a few months now and I've had mostly no issues with it until very recently, where a new ElasticSearch index was created in order to accomodate the new structure of our logs. With that, we also had to update our existing nxlog.conf file.
We have about 3 different ElasticSearch endpoints with the same setup that we currently work with, and at one point during the week we had run out of storage space. After increasing the storage size for all 3 endpoints, two environments appeared to continue sending new information up to ElasticSearch with no further problems. However, the third environment's NxLog services appear to be stuck in both existing AWS instances and newly created instances, repeating the following NxLog log entry over and over:
2016-04-29 15:33:12 INFO connecting to search-stage-logging-udf7h4lq2bsm245ciawp2stcvu.us-east-1.es.amazonaws.com:80
2016-04-29 15:33:12 INFO reconnecting in 1 seconds
2016-04-29 15:33:12 ERROR ### PANIC at line 2456 in module.c/nx_module_pollset_add_socket(): "failed to add descriptor to pollset: Not enough space ; [cannot dump backtrace on this platform]" ###
This was the log entry that we initially discovered that alerted us we had run out of space in ElasticSearch. However, the ElasticSearch dashboard does not show a lack of space anymore, so it's a bit confusing why NxLog would continue to output log entries in the other two environments
Basically, I have two questions:
1. Is this a type of scenario where the NxLog service's working state has been stuck unable to see there is space available? Or does the fault lie with ElasticSearch not showing storage space correctly?
2. If the Nxlog service has been stuck in this state, is there a configuration or some other automated procedure for NxLog to get the service to restart itself on multiple failures?
If anyone has gone through a similar experience, any tips would be greatly appreciated. Thank you for your time.
jppacheco created
Order messages from Windows Event Log with nxlog-elasticsearch-Kibana
moses created
Hi,
I'm using nxlog to send logs from Windows eventlog to elasticsearch, and using Kibana view.
I'm getting all the message as it is in the 'Message' column, I want to re-order it so the hostname parameter will be the windows server (and not the elasticsearch server), add 'Type' to the messages, etc.
this is the configuration file of nxlog:
* 55.2.110.4=elasticsearch server
<Extension json>
Module xm_json
</Extension>
# Nxlog internal logs
<Input internal>
Module im_internal
Exec $EventReceivedTime = integer($EventReceivedTime) / 1000000;
</Input>
# Windows Event Log
<Input eventlog>
# Uncomment im_msvistalog for Windows Vista/2008 and later
Module im_msvistalog
# Uncomment im_mseventlog for Windows XP/2000/2003
# Module im_mseventlog
Exec $EventReceivedTime = integer($EventReceivedTime) / 1000000;
</Input>
<Output out>
Module om_http
URL https://55.2.110.4:443
HTTPSAllowUntrusted TRUE
</Output>
<Route 1>
Path internal, eventlog => out
</Route>
this is an example message from kibana:
Field
Action
Value
@timestamp
2015-09-08T07:35:47.064Z
@version
1
_id
AU-r4dtqVULqkki94YkZ
_index
logstash-2015.09.08
_type
logs
host
55.2.110.4
http_port
5005
message
2015-09-08 07:35:43 dc-prod-zone-a.organization.com AUDIT_SUCCESS 4634 An account was logged off. Subject: Security ID: S-1-5-21-1595779987-1987268195-2987234418-1104 Account Name: DC-PROD-ZONE-C$ Account Domain: ORGANIZATION Logon ID: 0x679381 Logon Type: 3 This event is generated when a logon session is destroyed. It may be positively correlated with a logon event using the Logon ID value. Logon IDs are only unique between reboots on the same computer.
Thanks a lot
moses created