Nxlog CE is not sending Event to Input File
Hello,
I have a strange situation: I have a folder to index, which has some files, each file has a single line, a JSON encoded string. If I turn the DEBUG level, I can see that the files match the pattern for files, im_file_add_file function is called, but no event reaches my INPUT.
<Input api>
Module im_file
SavePos FALSE
ReadFromLast FALSE
PollInterval 0.25
#ActiveFiles 1000
#CloseWhenIdle TRUE
File "/var/www/project/current/var/api_logs/default/2018-07-25/qm_sync/*"
<Exec>
log_info(file_name());
$file = file_name();
#if $file =~ /^\/var\/www\/project\/current\/var\/api_logs\/default\/2018-07-25\/qm_sync\/[^\/]*\/([^\/]*)\/.*/
#{
$user = $1;
file_write("/var/log/nxlog/nxlog.log", $raw_event);
if $raw_event =~ /^(.*"requestBody":\{[^\[]*"[^"]*":)\[([^\]]*)\](.*)$/
{
$raw_event = $1 + '"[' + replace($2, '"', '\"') + ']"' + $3;
}
file_write("/var/log/nxlog/nxlog.log", $raw_event);
parse_json();
$timestamp = $requestTimestamp;
$eventTime = strftime($EventReceivedTime, '%Y-%m-%d %H:%M:%S');
$host = hostname_fqdn();
rename_field('timestamp', '@timestamp');
delete($EventReceivedTime);
delete($SourceModuleName);
delete($SourceModuleType);
to_json();
#} else {
# file_write("/var/log/nxlog/nxlog_parse_failed.log", "File: " + $file + "; Message: " + $raw_event + "\n");
# drop();
#}
$indexType = 'nxlog-api-external';
</Exec>
</Input>
If I enter a file with VIM, and just write and quit, without changing anything, the file gets indexed.
Any idea why this is happening? I am using the same instance of NxLog CE to index other files and it works fine with them
If you update a file with VIM or a text editor it will rewrite the whole file, i.e. it first truncates the file and then writes all the data into the file, not just the lines changed.
This is not how nxlog is prepared to read files since log files are append-only. You should test this way by appending data only, e.g.
echo test >> file.log