- OS Support
- Enterprise Edition Reference Manual
- 132. Man Pages
- 133. Configuration
- 134. Language
- 135. Extension Modules
- 136. Input Modules
- 136.1. Process Accounting (im_acct)
- 136.2. AIX Auditing (im_aixaudit)
- 136.3. Azure (im_azure)
- 136.4. Batched Compression (im_batchcompress)
- 136.5. Basic Security Module Auditing (im_bsm)
- 136.6. Check Point OPSEC LEA (im_checkpoint)
- 136.7. DBI (im_dbi)
- 136.8. Event Tracing for Windows (im_etw)
- 136.9. External Programs (im_exec)
- 136.10. Files (im_file)
- 136.11. File Integrity Monitoring (im_fim)
- 136.12. Go (im_go)
- 136.13. HTTP(s) (im_http)
- 136.14. Internal (im_internal)
- 136.15. Java (im_java)
- 136.16. Kafka (im_kafka)
- 136.17. Kernel (im_kernel)
- 136.18. Linux Audit System (im_linuxaudit)
- 136.19. macOS ULS (im_maculs)
- 136.20. Mark (im_mark)
- 136.21. EventLog for Windows XP/2000/2003 (im_mseventlog)
- 136.22. Event Log for Windows 2008/Vista and later (im_msvistalog)
- 136.23. Null (im_null)
- 136.24. ODBC (im_odbc)
- 136.25. Packet Capture (im_pcap)
- 136.26. Perl (im_perl)
- 136.27. Named Pipes (im_pipe)
- 136.28. Python (im_python)
- 136.29. Redis (im_redis)
- 136.30. Windows Registry Monitoring (im_regmon)
- 136.31. Ruby (im_ruby)
- 136.32. TLS/SSL (im_ssl)
- 136.33. Systemd (im_systemd)
- 136.34. TCP (im_tcp)
- 136.35. Test Generator (im_testgen)
- 136.36. UDP (im_udp)
- 136.37. Unix Domain Sockets (im_uds)
- 136.38. Windows Performance Counters (im_winperfcount)
- 136.39. Windows Event Collector (im_wseventing)
- 136.40. ZeroMQ (im_zmq)
- 137. Processor Modules
- 138. Output Modules
- NXLog Manager
- NXLog Add-Ons
This module can be used to read log messages from files. The file position can be persistently saved across restarts in order to avoid reading from the beginning again when NXLog is restarted. External rotation tools are also supported. When the module is not able to read any more data from the file, it checks whether the opened file descriptor belongs to the same filename it opened originally. If the inodes differ, the module assumes the file was moved and reopens its input.
|To examine the supported platforms, see the list of installer packages in the Available Modules chapter.|
im_file uses a one second interval to monitor files for new messages. This method was implemented because polling a regular file is not supported on all platforms. If there is no more data to read, the module will sleep for 1 second.
By using wildcards, the module can read multiple files simultaneously and will open new files as they appear. It will also enter newly created directories if recursion is enabled.
|The module needs to scan the directory content for wildcarded file monitoring. This can present a significant load if there are many files (hundreds or thousands) in the monitored directory. For this reason it is highly recommended to rotate files out of the monitored directory either using the built-in log rotation capabilities of NXLog or with external tools.|
This mandatory directive specifies the name of the input file to open. It may be given more than once in a single im_file module instance. The value must be a string type expression. For relative filenames you should be aware that NXLog changes its working directory to "/" unless the global SpoolDir is set to something else. On Windows systems the directory separator is the backslash (
\). For compatibility reasons the forward slash (
/) character can be also used as the directory separator, but this only works for filenames not containing wildcards. If the filename is specified using wildcards, the backslash (
\) should be used for the directory separator. Filenames on Windows systems are treated case-insensitively, but case-sensitively on Unix/Linux.
Wildcards are supported in filenames and directories. Wildcards are not regular expressions, but are patterns commonly used by Unix shells to expand filenames (also known as "globbing").
Matches a single character only.
Matches zero or more characters.
Matches the asterisk (
Matches the question mark (
Used to specify a single character. The class description is a list containing single characters and ranges of characters separated by the hyphen (
-). If the first character of the class description is
!, the sense of the description is reversed (any character not in the list is accepted). Any character can have a backslash (
\) preceding it, which is ignored, allowing the characters
-to be used in the character class, as well as
!at the beginning.
By default, the backslash character (
\) is used as an escape sequence. This character is also the directory separator on Windows. Because of this, escaping of wildcard characters is not supported on Windows, see the EscapeGlobPatterns directive. However, string literals are evaluated differently depending on the quotation type. Single quoted strings are interpreted as-is without escaping, e.g.
C:\t???\*.log. Escape sequences in double quoted strings are processed, for example
C:\t???\*.logafter evaluation. In both cases, the evaluated string is the same and gets separated into parts with different glob patterns at different levels. In the previous example the parts are
*.log. NXLog matches these at the proper directory levels to find all matching files.
This directive specifies the maximum number of files NXLog will actively monitor. If there are modifications to more files in parallel than the value of this directive, then modifications to files above this limit will only get noticed after the DirCheckInterval (all data should be collected eventually). Typically there are only a few log sources actively appending data to log files, and the rest of the files are dormant after being rotated, so the default value of 10 files should be sufficient in most cases. This directive is also only relevant in case of a wildcarded File path.
If set to TRUE, this boolean directive specifies that open input files should be closed as soon as possible after there is no more data to read. Some applications request an exclusive lock on the log file when written or rotated, and this directive can possibly help if the application tries again to acquire the lock. The default is FALSE.
This directive specifies how frequently, in seconds, the module will check the monitored directory for modifications to files and new files in case of a wildcarded File path. The default is twice the value of the PollInterval directive (if PollInterval is not set, the default is 2 seconds). Fractional seconds may be specified. It is recommended to increase the default if there are many files which cannot be rotated out and the NXLog process is causing high CPU load.
This directive can specify a file or a set of files (using wildcards) to be excluded. More than one occurrence of the Exclude directive can be specified.
See the InputType directive in the list of common module directives. If this directive is not specified the default is LineBased (the module will use CRLF as the record terminator on Windows, or LF on Unix).
This directive also supports data converters, see the description in the InputType section.
This boolean directive specifies whether the backslash (
\) in file paths should be disabled as an escape sequence. This is especially useful for file paths on Windows. By default, NoEscape is FALSE (backslash escaping is enabled and the path separator on Windows must be escaped).
This optional block directive can be used to specify a group of statements to execute when a file has been fully read (on end-of-file). Only one OnEOF block can be specified per im_file module instance. The following directives are used inside this block.
This mandatory directive specifies the actions to execute after EOF has been detected and the grace period has passed. Like the normal Exec directive, the OnEOF Exec can be specified as a normal directive or a block directive.
This optional directive specifies the time in seconds to wait before executing the actions configured in the Exec block or directive. The default is 1 second.
This directive specifies how frequently the module will check for new files and new log entries, in seconds. If this directive is not specified, it defaults to 1 second. Fractional seconds may be specified (
PollInterval 0.5will check twice every second).
This optional boolean directive instructs the module to only read logs which arrived after NXLog was started if the saved position could not be read (for example on first start). When SavePos is TRUE and a previously saved position value could be read, the module will resume reading from this saved position. If ReadFromLast is FALSE, the module will read all logs from the file. This can result in quite a lot of messages, and is usually not the expected behavior. If this directive is not specified, it defaults to TRUE.
If set to TRUE, this boolean directive specifies that input files set with the File directive should be searched recursively under sub-directories. For example,
/var/log/apache2/error.log. Wildcards can be used in combination with Recursive:
/var/log/apache2/access.log. This directive only causes scanning under the given path and does not affect the processing of wildcarded directories:
/var/*/qemu/debian.logwill not match
/var/log/libvirt/qemu/debian.log. The default is FALSE.
If set to TRUE, this boolean directive specifies that input files should be monitored for possible file rotation via renaming in order to avoid re-reading the file contents. A file is considered to be rotated when NXLog detects a new file whose inode and size matches that of another watched file which has just been deleted. Note that this does not always work correctly and can yield false positives when a log file is deleted and another is added with the same size. The file system is likely to reuse to inode number of the deleted file and thus the module will falsely detect this as a rename/rotation. For this reason the default value of RenameCheck is FALSE: renamed files are considered to be new and the file contents will be re-read.Note
It is recommended to use a naming scheme for rotated files so names of rotated files do not match the wildcard and are not monitored anymore after rotation, instead of trying to solve the renaming issue with this directive.
If this boolean directive is set to TRUE, the file position will be saved when NXLog exits. The file position will be read from the cache file upon startup. The default is TRUE: the file position will be saved if this directive is not specified. Even if SavePos is enabled, it can be explicitly turned off with the global NoCache directive.
The following functions are exported by im_file.
Return the name of the currently open file which the log was read from.
Returns the number of processed records (including the current record) of the currently open file since it was opened or truncated.
This configuration will read from a file and forward messages via TCP. No additional processing is done.