Nxlog handling big number of files
Tags:
#1
alexandru.enciu
Hello,
I have an application that logs some API requests and responses. Each request is logged in a different file, as a single line. In the system there are thousands of files, and Nxlog seems to have issues sending the logs to Elasticsearch. It reads the file, I can see im_file_add_file command in logs, but it takes a long time to actually send the message.
How does Nxlog process multiple files in a single directory?
#1
alexandru.enciu
Hello,
I have an application that logs some API requests and responses. Each request is logged in a different file, as a single line. In the system there are thousands of files, and Nxlog seems to have issues sending the logs to Elasticsearch. It reads the file, I can see im_file_add_file command in logs, but it takes a long time to actually send the message.
How does Nxlog process multiple files in a single directory?
Which version are you testing with?
The NXLog Enterprise Edition has some improvements in this area but if it needs to scan a directory with thousands of files it will be quite an overhead.
