NXLog performance

View thread

leshqo

Hi 

I'm recently tried to use NXLog community edition to maintain log files. Basically the idea is to rewrite log files one to one from many services on many hosts to one centralized server. 

In my test scenario I've created 3 services on one host and started propagate logs. Logs were rewriten succesfully but what I observed:
- It took 40 minutes to rewrite 1GB of logs (1GB connection with 60MB transfer between hosts)
- NXLog CPU usage on host where the logs were stored was about 10% (Intel Core i7)
- In Sysinternals ProcessMonitor I've observed that for one data read (65000 bytes)

16:02:26,4218297    nxlog.exe    12884    ReadFile    D:\app\Logs\service1.service\Logs_151130_12.log    SUCCESS    Offset: 490 945 000, Length: 65 000

there is about 450 partial data writes by 30/90/120 bytes which makes write process high ineffective

16:02:26,4219115    nxlog.exe    12884    LockFile    \\192.168.199.46\Shared\logs\service1\Logs_151130_12.log    SUCCESS    Exclusive: True, Offset: 0, Length: 4 294 967 295, Fail Immediately: False
16:02:26,4219594    nxlog.exe    12884    QueryStandardInformationFile    \\192.168.199.46\Shared\logs\service1\Logs_151130_12.log    SUCCESS    AllocationSize: 490 930 673, EndOfFile: 490 930 673, NumberOfLinks: 1, DeletePending: False, Directory: False
16:02:26,4219996    nxlog.exe    12884    WriteFile    \\192.168.199.46\Shared\logs\service1\Logs_151130_12.log    SUCCESS    Offset: 490 930 673, Length: 84, Priority: Normal
16:02:26,4228112    nxlog.exe    12884    UnlockFileSingle    \\192.168.199.46\Shared\logs\service1\Logs_151130_12.log    SUCCESS    Offset: 0, Length: 4 294 967 295

Is that lack of performance or do I have missed something. Usage of NXLog with such results is unacceptable for me.

My config file:

#define ROOT C:\Program Files\nxlog
define ROOT C:\Program Files (x86)\nxlog

Moduledir %ROOT%\modules
CacheDir %ROOT%\data
Pidfile %ROOT%\data\nxlog.pid
SpoolDir %ROOT%\data
LogFile %ROOT%\data\nxlog.log

define LOG_SERVER \\\\192.168.199.46
define LOGS_PATH d:\\App\\Logs\\\\*.log

<Extension _syslog>    
    Module      xm_fileop
    Module        xm_exec
</Extension>

<Input logs>
    Module im_file
    File "%LOGS_PATH%"
    InputType LineBased
    PollInterval 15
    SavePos TRUE
    CloseWhenIdle TRUE
    Recursive TRUE
    Exec $fileName = file_basename(file_name());
    Exec if file_name() =~ /^.*Logs\\(.*?)\.service/        \
    {                                                       \
        $serviceName = $1;                                  \
    }
</Input>

<Output out>    
    Module om_file
    CreateDir TRUE
    OutputType LineBased    
    File "%LOG_SERVER%\\Shared\\Logs\\" + $serviceName + "\\" + $fileName
</Output>

<Route 1>
    Path        logs => out
</Route>