3
responses

Hello,

I have a strange situation: I have a folder to index, which has some files, each file has a single line, a JSON encoded string. If I turn the DEBUG level, I can see that the files match the pattern for files, im_file_add_file function is called, but no event reaches my INPUT.

<Input api>
    Module im_file
    SavePos      FALSE
    ReadFromLast FALSE
    PollInterval 0.25
    #ActiveFiles 1000
    #CloseWhenIdle TRUE
    File   "/var/www/project/current/var/api_logs/default/2018-07-25/qm_sync/*"
    <Exec>
        log_info(file_name());
        $file = file_name();
        #if $file =~ /^\/var\/www\/project\/current\/var\/api_logs\/default\/2018-07-25\/qm_sync\/[^\/]*\/([^\/]*)\/.*/
        #{
            $user = $1;
            file_write("/var/log/nxlog/nxlog.log", $raw_event);
            if $raw_event =~ /^(.*"requestBody":\{[^\[]*"[^"]*":)\[([^\]]*)\](.*)$/
            {
                $raw_event = $1 + '"[' + replace($2, '"', '\"') + ']"' + $3;
            }
            file_write("/var/log/nxlog/nxlog.log", $raw_event);
            parse_json();
            $timestamp = $requestTimestamp;
            $eventTime = strftime($EventReceivedTime, '%Y-%m-%d %H:%M:%S');
            $host = hostname_fqdn();
            rename_field('timestamp', '@timestamp');
            delete($EventReceivedTime);
            delete($SourceModuleName);
            delete($SourceModuleType);

            to_json();
        #} else {
        #    file_write("/var/log/nxlog/nxlog_parse_failed.log", "File: " + $file + "; Message: " + $raw_event + "\n");
        #    drop();
        #}
        $indexType = 'nxlog-api-external';
    </Exec>
</Input>

If I enter a file with VIM, and just write and quit, without changing anything, the file gets indexed.

Any idea why this is happening? I am using the same instance of NxLog CE to index other files and it works fine with them

AskedJuly 25, 2018 - 1:05am

Answer (1)

If you update a file with VIM or a text editor it will rewrite the whole file, i.e. it first truncates the file and then writes all the data into the file, not just the lines changed.
This is not how nxlog is prepared to read files since log files are append-only. You should test this way by appending data only, e.g.

echo test >> file.log

Comments (2)

  • Alexandru Enciu's picture

    I solved the issue by using xm_multiline, with FixedLineCount 1. For some reason, even though Nxlog read the file contents, it didn't trigger any event to send to INPUT. I think it's related to LineBased event type, which was expecting an EOL, but since there is only 1 line, there isn't one.

  • b0ti's picture
    (NXLog)

    There has to be a CRLF or LF line terminator at the end of the line when using LineBased input, otherwise it will not be considered as a complete event.