NXLog performance

Tags:

#1 leshqo

Hi 

I'm recently tried to use NXLog community edition to maintain log files. Basically the idea is to rewrite log files one to one from many services on many hosts to one centralized server. 

In my test scenario I've created 3 services on one host and started propagate logs. Logs were rewriten succesfully but what I observed:
- It took 40 minutes to rewrite 1GB of logs (1GB connection with 60MB transfer between hosts)
- NXLog CPU usage on host where the logs were stored was about 10% (Intel Core i7)
- In Sysinternals ProcessMonitor I've observed that for one data read (65000 bytes)

16:02:26,4218297    nxlog.exe    12884    ReadFile    D:\app\Logs\service1.service\Logs_151130_12.log    SUCCESS    Offset: 490 945 000, Length: 65 000

there is about 450 partial data writes by 30/90/120 bytes which makes write process high ineffective

16:02:26,4219115    nxlog.exe    12884    LockFile    \\192.168.199.46\Shared\logs\service1\Logs_151130_12.log    SUCCESS    Exclusive: True, Offset: 0, Length: 4 294 967 295, Fail Immediately: False
16:02:26,4219594    nxlog.exe    12884    QueryStandardInformationFile    \\192.168.199.46\Shared\logs\service1\Logs_151130_12.log    SUCCESS    AllocationSize: 490 930 673, EndOfFile: 490 930 673, NumberOfLinks: 1, DeletePending: False, Directory: False
16:02:26,4219996    nxlog.exe    12884    WriteFile    \\192.168.199.46\Shared\logs\service1\Logs_151130_12.log    SUCCESS    Offset: 490 930 673, Length: 84, Priority: Normal
16:02:26,4228112    nxlog.exe    12884    UnlockFileSingle    \\192.168.199.46\Shared\logs\service1\Logs_151130_12.log    SUCCESS    Offset: 0, Length: 4 294 967 295

Is that lack of performance or do I have missed something. Usage of NXLog with such results is unacceptable for me.

My config file:

#define ROOT C:\Program Files\nxlog
define ROOT C:\Program Files (x86)\nxlog

Moduledir %ROOT%\modules
CacheDir %ROOT%\data
Pidfile %ROOT%\data\nxlog.pid
SpoolDir %ROOT%\data
LogFile %ROOT%\data\nxlog.log

define LOG_SERVER \\\\192.168.199.46
define LOGS_PATH d:\\App\\Logs\\\\*.log

<Extension _syslog>    
    Module      xm_fileop
    Module        xm_exec
</Extension>

<Input logs>
    Module im_file
    File "%LOGS_PATH%"
    InputType LineBased
    PollInterval 15
    SavePos TRUE
    CloseWhenIdle TRUE
    Recursive TRUE
    Exec $fileName = file_basename(file_name());
    Exec if file_name() =~ /^.*Logs\\(.*?)\.service/        \
    {                                                       \
        $serviceName = $1;                                  \
    }
</Input>

<Output out>    
    Module om_file
    CreateDir TRUE
    OutputType LineBased    
    File "%LOG_SERVER%\\Shared\\Logs\\" + $serviceName + "\\" + $fileName
</Output>

<Route 1>
    Path        logs => out
</Route>
#2 adm Nxlog ✓
#1 leshqo
Hi  I'm recently tried to use NXLog community edition to maintain log files. Basically the idea is to rewrite log files one to one from many services on many hosts to one centralized server.  In my test scenario I've created 3 services on one host and started propagate logs. Logs were rewriten succesfully but what I observed: - It took 40 minutes to rewrite 1GB of logs (1GB connection with 60MB transfer between hosts) - NXLog CPU usage on host where the logs were stored was about 10% (Intel Core i7) - In Sysinternals ProcessMonitor I've observed that for one data read (65000 bytes) 16:02:26,4218297 nxlog.exe 12884 ReadFile D:\app\Logs\service1.service\Logs_151130_12.log SUCCESS Offset: 490 945 000, Length: 65 000 there is about 450 partial data writes by 30/90/120 bytes which makes write process high ineffective 16:02:26,4219115 nxlog.exe 12884 LockFile \\192.168.199.46\Shared\logs\service1\Logs_151130_12.log SUCCESS Exclusive: True, Offset: 0, Length: 4 294 967 295, Fail Immediately: False 16:02:26,4219594 nxlog.exe 12884 QueryStandardInformationFile \\192.168.199.46\Shared\logs\service1\Logs_151130_12.log SUCCESS AllocationSize: 490 930 673, EndOfFile: 490 930 673, NumberOfLinks: 1, DeletePending: False, Directory: False 16:02:26,4219996 nxlog.exe 12884 WriteFile \\192.168.199.46\Shared\logs\service1\Logs_151130_12.log SUCCESS Offset: 490 930 673, Length: 84, Priority: Normal 16:02:26,4228112 nxlog.exe 12884 UnlockFileSingle \\192.168.199.46\Shared\logs\service1\Logs_151130_12.log SUCCESS Offset: 0, Length: 4 294 967 295 Is that lack of performance or do I have missed something. Usage of NXLog with such results is unacceptable for me. My config file: #define ROOT C:\Program Files\nxlog define ROOT C:\Program Files (x86)\nxlog Moduledir %ROOT%\modules CacheDir %ROOT%\data Pidfile %ROOT%\data\nxlog.pid SpoolDir %ROOT%\data LogFile %ROOT%\data\nxlog.log define LOG_SERVER \\\\192.168.199.46 define LOGS_PATH d:\\App\\Logs\\\\*.log <Extension _syslog> Module xm_fileop Module xm_exec </Extension> <Input logs> Module im_file File "%LOGS_PATH%" InputType LineBased PollInterval 15 SavePos TRUE CloseWhenIdle TRUE Recursive TRUE Exec $fileName = file_basename(file_name()); Exec if file_name() =~ /^.*Logs\\(.*?)\.service/ \ { \ $serviceName = $1; \ } </Input> <Output out> Module om_file CreateDir TRUE OutputType LineBased File "%LOG_SERVER%\\Shared\\Logs\\" + $serviceName + "\\" + $fileName </Output> <Route 1> Path logs => out </Route>

The performance bottleneck is likely caused by om_file having to open a new file for each event due to the dynamic file path.