Buffer is not working
Hello,
Here is my config on Windows machine, running nxlog-ce-2.9.1504
#define ROOT C:\Program Files\nxlog
define ROOT C:\Program Files (x86)\nxlog
Moduledir %ROOT%\modules
CacheDir %ROOT%\data
Pidfile %ROOT%\data\nxlog.pid
SpoolDir %ROOT%\data
LogFile %ROOT%\data\nxlog.log
###############
# Extensions #
###############
<Extension syslog>
Module xm_syslog
</Extension>
<Extension json>
Module xm_json
</Extension>
###########
# Inputs #
###########
<Input some_input>
Module im_file
File 'C:\Logs\input.log'
SavePos TRUE
</Input>
###############
# Processors #
###############
<Processor buffer>
Module pm_buffer
# 1Gb disk buffer 1048576 kilo-bytes
MaxSize 1048576
Type Disk
Directory C:\Logs\buffer
</Processor>
############
# Outputs #
############
<Output tcpout>
Module om_tcp
Port 5170
Host fluentd.company.lan
</Output>
############
# Routes #
############
<Route file>
Path some_input => buffer => tcpout
</Route>
Here's testing case initials:
1. Service ' fluentd.company.lan' is up and running and listens on 5170
2. nxlog up and running with given config
3. Data coming to input.log is successfully routed to output via buffer and is seen in Kibana
Then
1. I change 'C:\Windows\system32\drivers\etc\hosts' file and add '127.0.0.1 fluentd.company.lan' line, saving file
2. Using TCPView tool from SysInternals close current TCP connection with 'fluentd.company.lan:5170'
3. See in nxlog.log, that it tries to connect to 'fluentd.company.lan:5170' and fails to connect
4. Wait for some new data in input.log
5. New data arrived and I see buffer file created 'buffer.1.q' in C:\Logs\buffer and see relevant data in it
6. Wait for some time (2-3 minutes)
7. Again I change 'C:\Windows\system32\drivers\etc\hosts' file and comment '127.0.0.1 fluentd.company.lan' line, saving file
8. nxlog successfully connects to fluentd.company.lan:5170
And here's interesting part, nxlog writes new data found in input file, but I don't see logs in Kibana from buffer file with timestamps from intervals in point #6
Please check this case and make sure buffer is not working on Windows and fix this bug
For me nxlog buffering is not working either. Checked 3 times in different (though similar) environments (both disk and mem buffers).
Here is my nxlog configuration:
<Input sys_in>
Module im_udp
Port 516
</Input>
<Processor sys_buf>
Module pm_buffer
# 512MB buffer
MaxSize 524288
Type Disk
WarnLimit 65536
</Processor>
<Output sys_out>
Module om_tcp
Host log.ourdomain.internal
Port 9291
</Output>
<Route sys>
Path sys_in => sys_buf => sys_out
</Route>
The output endpoint is a logstash TCP listener behind an AWS load balancer (so log.ourdomain.internal has a CNAME record pointing to the ELB).
If I stop logstash service, nxlog starts printing the following messages:
2016-08-19 12:05:38 INFO reconnecting in 1 seconds 2016-08-19 12:05:38 INFO connecting to log.ourdomain.internal:9291 2016-08-19 12:05:38 INFO reconnecting in 1 seconds ...
every second or so. After a minute of waiting I start logstash back, nxlog successfully connects to it but none of the previous log messages are sent...
The timeline looks like this: http://imgur.com/download/YkZqwZg
I can confirm that sys_buf.1.q is created (when nxlog is started), but it has 0KB size, always.
I'm not sure where to dig, the configuration looks correct for me.. Could you please help resolve the issue?
PS. We are using Windows Server 2012 R2 + nxlog 2.9.1716 (we had same issue in previous versions) + logstash (2.3.4).
