Ask questions. Get answers. Find technical product solutions from passionate experts in the NXLog community.
Run-away logging involving NXLog
CityofRome created
For about 5 years, I've been using NXLog to forward Windows logs from all of my Windows servers into a Graylog server. Recently, one of the servers developed an issue where there will be event ID 5156 ("The Windows Filtering Platform has permitted a connection") triggered when NXLog sends logs to the Graylog server, which triggers another event ID 5156, which triggers another and another and another and so on. So, logging from that one server goes from an average of 50,000/hr to as much as 10 million/hr. I don't see anything in the Windows event logs that seems to trigger the issue but all I have to do is restart the NXLog service to break the loop and resume normal log forwarding for a couple of days. I've uninstalled/re-installed NXLog and upgraded to 'nxlog-ce-2.10.2150'. The server is essentially just a file server. It has Checkpoint Endpoint installed but so do all of my other Windows servers.
Does anyone have any suggestions as to what causes this and how I can resolve the issue? I don't want to disable the events from the Windows Filtering Platform in total but I wouldn't mind if I never saw one triggered by NXLog making network connections. Below is the same NXLog config I've used for all of the Windows servers, even the server I'm having the issue on. Any help you can give is greatly appreciated.
#define ROOT C:\Program Files (x86)\nxlog
define ROOT C:\Program Files (x86)\nxlog
Moduledir %ROOT%\modules
CacheDir %ROOT%\data
Pidfile %ROOT%\data\nxlog.pid
SpoolDir %ROOT%\data
LogFile %ROOT%\data\nxlog.log
<Extension gelf>
Module xm_gelf
</Extension>
<Input in>
Use 'im_mseventlog' for Windows XP, 2000 and 2003
Module im_msvistalog
Uncomment the following to collect specific event logs only
Query <QueryList>\
<Query Id="0">\
<Select Path="Application">*</Select>\
<Select Path="System">*</Select>\
<Select Path="Security">*</Select>\
</Query>\
</QueryList>
</Input>
<Output out>
Module om_udp
Host 172.xx.xx.xxx {<-- redacted for this post}
Port 12201
OutputType GELF
</Output>
<Route 1>
Path in => out
</Route>
CityofRome created
Common format for Windows, internal logs and my app logs
igorgatis created
TL;DR: what's the recommended way of converting logs to a common (e.g. GELF) format?
I'm using NXLog together with Logstash and EalsticSearch. I'm collecting logs from Windows, NXLogs (internal) and my app logs using line based JSON.
Windows logs and NXLogs seem to share a lot of field names. I can write my app so that it uses the same fields. This greatly facilitates viewing data in elasticsearch. I could stick with windows fields or convert them all to GELF. AFAIK, the convertion from Windows Logs to GELF seems seems to require a lot of per-field convertion work. There is a good chance I won't get it right until enough data is produced.
I was looking for a convert_to_gelf() function which would take care of converting Windows Logs, Internal logs, IIS, etc to GELF. Is there such thing? Is manual conversion my only option?
igorgatis created