NXLog performance
Hi
I'm recently tried to use NXLog community edition to maintain log files. Basically the idea is to rewrite log files one to one from many services on many hosts to one centralized server.
In my test scenario I've created 3 services on one host and started propagate logs. Logs were rewriten succesfully but what I observed:
- It took 40 minutes to rewrite 1GB of logs (1GB connection with 60MB transfer between hosts)
- NXLog CPU usage on host where the logs were stored was about 10% (Intel Core i7)
- In Sysinternals ProcessMonitor I've observed that for one data read (65000 bytes)
16:02:26,4218297 nxlog.exe 12884 ReadFile D:\app\Logs\service1.service\Logs_151130_12.log SUCCESS Offset: 490 945 000, Length: 65 000
there is about 450 partial data writes by 30/90/120 bytes which makes write process high ineffective
16:02:26,4219115 nxlog.exe 12884 LockFile \\192.168.199.46\Shared\logs\service1\Logs_151130_12.log SUCCESS Exclusive: True, Offset: 0, Length: 4 294 967 295, Fail Immediately: False 16:02:26,4219594 nxlog.exe 12884 QueryStandardInformationFile \\192.168.199.46\Shared\logs\service1\Logs_151130_12.log SUCCESS AllocationSize: 490 930 673, EndOfFile: 490 930 673, NumberOfLinks: 1, DeletePending: False, Directory: False 16:02:26,4219996 nxlog.exe 12884 WriteFile \\192.168.199.46\Shared\logs\service1\Logs_151130_12.log SUCCESS Offset: 490 930 673, Length: 84, Priority: Normal 16:02:26,4228112 nxlog.exe 12884 UnlockFileSingle \\192.168.199.46\Shared\logs\service1\Logs_151130_12.log SUCCESS Offset: 0, Length: 4 294 967 295
Is that lack of performance or do I have missed something. Usage of NXLog with such results is unacceptable for me.
My config file:
#define ROOT C:\Program Files\nxlog define ROOT C:\Program Files (x86)\nxlog Moduledir %ROOT%\modules CacheDir %ROOT%\data Pidfile %ROOT%\data\nxlog.pid SpoolDir %ROOT%\data LogFile %ROOT%\data\nxlog.log define LOG_SERVER \\\\192.168.199.46 define LOGS_PATH d:\\App\\Logs\\\\*.log <Extension _syslog> Module xm_fileop Module xm_exec </Extension> <Input logs> Module im_file File "%LOGS_PATH%" InputType LineBased PollInterval 15 SavePos TRUE CloseWhenIdle TRUE Recursive TRUE Exec $fileName = file_basename(file_name()); Exec if file_name() =~ /^.*Logs\\(.*?)\.service/ \ { \ $serviceName = $1; \ } </Input> <Output out> Module om_file CreateDir TRUE OutputType LineBased File "%LOG_SERVER%\\Shared\\Logs\\" + $serviceName + "\\" + $fileName </Output> <Route 1> Path logs => out </Route>