4
responses

We have an application that does some multiple updates every morning between 6am and 7am. During this time, it generates massive amounts of log entries.
This in turn causes the box to run out of memory, triggering Linux's OOM daemon. Running the NxLog-ce.

I have added

PersistLogqueue TRUE
SyncLogqueue TRUE
CacheFlushInterval always
CacheSync TRUE

To the config, and will see if that makes a difference, but it seems that is more to safeguard messages from being lost.

I have looked at https://nxlog.co/question/802/nxlog-ce-memory-leak and https://nxlog.co/question/4132/cache-disk-works-not-good, but not sure those two are what I am after?

AskedMay 5, 2020 - 2:07am

Comments (4)

  • Wernervdmerwe's picture

    Hi,

    Can do - it seems to be a memory leak: https://postimg.cc/f3jyP17z For now, I have the service restarting automatically at 1pm, which effectively resolves the issue, but I would like to properly solve it.

    define ROOT /bin
    
    <Extension gelf>
      Module xm_gelf
    </Extension>
    
    User nxlog
    Group nxlog
    
    Moduledir /usr/libexec/nxlog/modules
    CacheDir /var/spool/collector-sidecar/nxlog
    PidFile /var/run/graylog/collector-sidecar/nxlog.pid
    
    define LOGFILE /var/log/graylog/collector-sidecar/nxlog.log
    LogFile %LOGFILE%
    LogLevel INFO
    
    <Extension logrotate>
        Module  xm_fileop
        <Schedule>
            When    @daily
            Exec    file_cycle('%LOGFILE%', 7);
         </Schedule>
    </Extension>
    
    
    <Input 596fec1644045504b0396b35>
            Module im_file
            File '/var/log/graylog/collector-sidecar/nxlog.log'
            PollInterval 1
            SavePos True
            ReadFromLast True
            Recursive True
            RenameCheck False
            Exec $FileName = file_name(); # Send file name with each message
            Exec $FileName = "nxlog.log";
    </Input>
    <Input 5b6e874d44045507d14afa4b>
            Module im_file
            File '/srv/xxxx/www/shared/log/sms_sent.log'
            PollInterval 1
            SavePos True
            ReadFromLast True
            Recursive False
            RenameCheck False
            Exec $FileName = file_name(); # Send file name with each message
    </Input>
    <Input 5bf23da1440455041bc4e577>
            Module im_file
            File '/srv/xxxx/www/shared/log/production.log'
            PollInterval 1
            SavePos True
            ReadFromLast True
            Recursive True
            RenameCheck True
            Exec $FileName = file_name(); # Send file name with each message
            Exec if $raw_event =~ /\w, \[(.+?)(?=\s) #(\d+)\]  (\w+) -- : \[(\w+)\] \[(.+?)(?=\])\] \[(.+?)(?=\])\] (.*)/ {$EventTime = parsedate($1); $PID = integer($2); $LogLevel =$3; $Module = $4; $Service = $5; $ID = $6; $Data = $7; }
    </Input>
    
    <Output 596feba044045504b0396ab7>
            Module om_tcp
            Host XXXXXXXXXX
            Port 5000
            OutputType  GELF_TCP
            Exec $short_message = $raw_event; # Avoids truncation of the short_message field.
            Exec $gl2_source_collector = 'df4db350-f4ea-460d-94ad-bbcc0cacc81b';
            Exec $collector_node_id = 'Sidecar_on_XXXXXXXXXX';
            Exec $Hostname = hostname_fqdn();
    </Output>
    <Output 5b6e816d4404551c28d4adfb>
            Module om_tcp
            Host XXXXXXXXXX
            Port 5000
            OutputType  GELF_TCP
            Exec $short_message = $raw_event; # Avoids truncation of the short_message field.
            Exec $gl2_source_collector = 'df4db350-f4ea-460d-94ad-bbcc0cacc81b';
            Exec $collector_node_id = 'Sidecar_on_XXXXXX';
            Exec $Hostname = hostname_fqdn();
            Exec $Server = "worker";
            Exec delete($SourceModuleType);
    </Output>
    
    
    <Route route-0>
      Path 596fec1644045504b0396b35 => 596feba044045504b0396ab7
    </Route>
    <Route route-1>
      Path 5b6e874d44045507d14afa4b => 5b6e816d4404551c28d4adfb
    </Route>
    <Route route-2>
      Path 5bf23da1440455041bc4e577 => 5b6e816d4404551c28d4adfb
    </Route>
    

  • Wernervdmerwe's picture

    Unfortunately the feed forms part of the monitoring strategy, so I will not be able to isolate the offending input. If I start adding one at a time back, it means that the remaining input will have to be offline for days as I'd image I'd have to give each input a 24h window to make sure all the race conditions are met.

Answers (0)