Nxlog CE is not sending Event to Input File

Tags:

#1 alexandru.enciu

Hello,

I have a strange situation: I have a folder to index, which has some files, each file has a single line, a JSON encoded string. If I turn the DEBUG level, I can see that the files match the pattern for files, im_file_add_file function is called, but no event reaches my INPUT.

<Input api>
    Module im_file
    SavePos      FALSE
    ReadFromLast FALSE
    PollInterval 0.25
    #ActiveFiles 1000
    #CloseWhenIdle TRUE
    File   "/var/www/project/current/var/api_logs/default/2018-07-25/qm_sync/*"
    <Exec>
        log_info(file_name());
        $file = file_name();
        #if $file =~ /^\/var\/www\/project\/current\/var\/api_logs\/default\/2018-07-25\/qm_sync\/[^\/]*\/([^\/]*)\/.*/
        #{
            $user = $1;
            file_write("/var/log/nxlog/nxlog.log", $raw_event);
            if $raw_event =~ /^(.*"requestBody":\{[^\[]*"[^"]*":)\[([^\]]*)\](.*)$/
            {
                $raw_event = $1 + '"[' + replace($2, '"', '\"') + ']"' + $3;
            }
            file_write("/var/log/nxlog/nxlog.log", $raw_event);
            parse_json();
            $timestamp = $requestTimestamp;
            $eventTime = strftime($EventReceivedTime, '%Y-%m-%d %H:%M:%S');
            $host = hostname_fqdn();
            rename_field('timestamp', '@timestamp');
            delete($EventReceivedTime);
            delete($SourceModuleName);
            delete($SourceModuleType);

            to_json();
        #} else {
        #    file_write("/var/log/nxlog/nxlog_parse_failed.log", "File: " + $file + "; Message: " + $raw_event + "\n");
        #    drop();
        #}
        $indexType = 'nxlog-api-external';
    </Exec>
</Input>

If I enter a file with VIM, and just write and quit, without changing anything, the file gets indexed.

Any idea why this is happening? I am using the same instance of NxLog CE to index other files and it works fine with them

#2 b0ti Nxlog ✓
#1 alexandru.enciu
Hello, I have a strange situation: I have a folder to index, which has some files, each file has a single line, a JSON encoded string. If I turn the DEBUG level, I can see that the files match the pattern for files, im_file_add_file function is called, but no event reaches my INPUT. <Input api> Module im_file SavePos FALSE ReadFromLast FALSE PollInterval 0.25 #ActiveFiles 1000 #CloseWhenIdle TRUE File "/var/www/project/current/var/api_logs/default/2018-07-25/qm_sync/*" <Exec> log_info(file_name()); $file = file_name(); #if $file =~ /^\/var\/www\/project\/current\/var\/api_logs\/default\/2018-07-25\/qm_sync\/[^\/]*\/([^\/]*)\/.*/ #{ $user = $1; file_write("/var/log/nxlog/nxlog.log", $raw_event); if $raw_event =~ /^(.*"requestBody":\{[^\[]*"[^"]*":)\[([^\]]*)\](.*)$/ { $raw_event = $1 + '"[' + replace($2, '"', '\"') + ']"' + $3; } file_write("/var/log/nxlog/nxlog.log", $raw_event); parse_json(); $timestamp = $requestTimestamp; $eventTime = strftime($EventReceivedTime, '%Y-%m-%d %H:%M:%S'); $host = hostname_fqdn(); rename_field('timestamp', '@timestamp'); delete($EventReceivedTime); delete($SourceModuleName); delete($SourceModuleType); to_json(); #} else { # file_write("/var/log/nxlog/nxlog_parse_failed.log", "File: " + $file + "; Message: " + $raw_event + "\n"); # drop(); #} $indexType = 'nxlog-api-external'; </Exec> </Input> If I enter a file with VIM, and just write and quit, without changing anything, the file gets indexed. Any idea why this is happening? I am using the same instance of NxLog CE to index other files and it works fine with them

If you update a file with VIM or a text editor it will rewrite the whole file, i.e. it first truncates the file and then writes all the data into the file, not just the lines changed.
This is not how nxlog is prepared to read files since log files are append-only. You should test this way by appending data only, e.g.

echo test >> file.log