NXLog User Guide
- Introduction
- Deployment
- Configuration
- 23. Configuration overview
- 24. NXLog language
- 25. Reading and receiving logs
- 26. Processing logs
- 27. Forwarding and Storing Logs
- 28. Centralized Log Collection
- 29. NXLog failover mode
- 30. High Availability (HA)
- 31. Encrypted transfer
- 32. Reducing bandwidth and data size
- 33. Reliable message delivery
- 34. Compression and Encryption
- OS Support
- Integration
- Troubleshooting
- Enterprise Edition Reference Manual
- NXLog Manager
- NXLog Add-Ons
23. Configuration overview
NXLog uses
Apache style
configuration files. The configuration file is loaded from its default
location, or it can be explicitly specified with the -c command line
argument.
The NXLog configuration file is comprised of blocks and directives. Blocks are similar to XML tags containing multiple directives. Directive names are not case-sensitive but arguments sometimes are.
23.1. Multiple lines
A directive and its argument must be specified on the same line. Values spanning
multiple lines must have the newline escaped with a backslash (\), see the
sample below.
1
2
Fields $Version, $Device_Vendor, $Device_Product, $Device_Version, \
$Signature_ID, $Name, $Severity, $_Extension
A typical case of the backslash use is the Exec directive.
1
2
Exec if ($QueryName == 'wpad') OR \
($QueryType != '1') drop();
Breaking regular expressions into multiple lines improves their readability. Below is the sample of the define directive which includes backslashes to continue the expression on the new line.
1
2
3
define EVENT_REGEX /(?x)^(\d+.\d+.\d+)\s+(\d+:\d+:\d+).\d+\
\s+PID=(\d+)\s+TID=(\d+)\s+(\w+)\s+\
\S(\w+):\s+(.*)/
Link to this example.
23.2. Comments
Blank lines in configuration files are ignored.
Lines starting with the hash mark (#) are also ignored and can be used as
comments.
1
# This is the comment line
Each comment should start with a new line. Application of inline comments is impossible because each directive can only contain the directive definition and the argument.
Comments spanning multiple lines should start with the # mark on each line.
Escaping multi-line comments with a backslash (\) is not supported.
1
2
# This is a comment
# spanning multiple lines
Comments inside the QueryXML directive must
be XML-style comments using <!-- and -->, see the sample below.
![[Download file]](images/icons/download_icon.png "Download file"){kind=icon}
1
2
3
4
5
<!--
XML-style comments can
span multiple lines in
QueryXML blocks like this.
-->
Link to this example.
23.3. File paths
File paths are used by global and module directives, and may be quoted with
either single (') or double quotes (").
1
File "/tmp/input"
Several modules, however, require that file paths must be unquoted.
1
DeviceFile /dev/auditpipe
Link to this example.
23.4. Quoting and escaping strings
String literals must be quoted using either single or double quotes.
When a string is enclosed in single quotes, NXLog interprets each
character as is. When a string is enclosed in double quotes,
NXLog supports escape sequences starting with the backslash \
character.
| Sequence | Description |
|---|---|
|
The backslash ( |
|
The double quote ( |
|
Line feed (LF) |
|
Carriage return (CR) |
|
Horizontal tab |
|
Audible bell |
|
A single byte in the form of a two digit hexadecimal number. For example the
line-feed character can also be expressed as |
When a string is enclosed in double quotes, NXLog will also interpret
the question mark ? and asterisk * as wildcard characters. This means
that specifying "file*.log" in a file path will match file1.log, filea.log
etc. For more details, see the Configuration
section of the im_file module documentation.
|
Note
|
Extra care should be taken when specifying file paths within double quotes since the backslash is used as the directory separator on Windows. For more information about the possible complications, see this note of the im_file module. |
The examples below demonstrate how to specify various strings and the resulting value.
The statement below sets the $FilePath field to a path on Windows.
1
$FilePath = 'C:\AppLogs\logfile.txt'
No escaping is done when using single quotes, therefore the string is interpreted as is.
C:\AppLogs\logfile.txtNested quotation is possible using double quotes. Nested quotation using single quotes is not supported.
1
$Message = 'Message: "This is a test"'
Message: "This is a test"The statement below sets the $FilePath field to a path on Windows. It uses
escape sequences for the directory separator.
1
$FilePath = "C:\\AppLogs\\logfile.txt"
Since the string is enclosed in double quotes, \\ will be interpreted as an
escape sequence and will result in a valid path.
C:\AppLogs\logfile.txtBoth single and double quotes are supported for nested quotation. Double quotes
need to be escaped with \".
1
$Message = "Message: \"This is a test\" using 'nested quotation'"
Message: "This is a test" using 'nested quotation'Wildcard characters are useful for reading data from multiple files based on
their naming pattern. The File directive below uses the
* asterisk character to read data from several files which names start with
log and have the .txt extension.
1
2
3
4
<Input from_file>
Module im_file
File "/tmp/log*.txt"
</Input>
Based on the naming pattern, both log1.txt and log_new.txt will be read.
23.5. Configuration structure
The configuration file can be logically divided into three parts: global parameters, module instances, and route instances.
This configuration exemplifies the logical structure. The global parameters section contains two directives. The modules section contains both an input and output instance. The route section contains a single route with a path directing a single input to a single output.
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
# Global section
User nxlog
Group nxlog
# Modules section
<Input in>
Module im_null
</Input>
<Output out>
Module om_null
</Output>
# Route section
<Route r>
Path in => out
</Route>
Link to this example.
23.6. Global directives
The global section contains directives that control the overall behavior of NXLog.
The LogFile directive sets a destination file for NXLog internal logs. If this directive is unset, the log file is disabled and internal NXLog logs are not written to file (unless configured via the im_internal module). See also Rotating the internal log file.
With the User and Group directives set, NXLog will drop root privileges after starting and run under the specified user and group. If for some reason, any capabilities are required to be held permanently, the capabilities directive can be used. These directives are ignored if running on Windows.
After starting, NXLog will change its working directory to the directory specified by the SpoolDir directive. Non-absolute paths in the configuration will be relative to this directory.
See the Reference Manual for a complete list of available global directives.
23.7. Modules
NXLog will only load modules which are specified in the
configuration file and used in an active route.
A module instance is specified according to its corresponding module type
(Extension, Input, Processor, or Output). Each module instance must
have a unique name and a Module directive. The
following is a skeleton configuration block for an input module.
1
2
3
4
<Input instancename>
Module im_module
...
</Input>
Link to this example.
For more details about module instance names, see Configuration in the Reference Manual.
23.8. Routes
Routes define the flow and processing order of the log messages. Each route instance must have a unique name and a Path.
This Route instance, named example, takes logs from Input module
instances named in1 and in2, processes the logs with the proc
Processor module instance, and sends the resulting logs to both Output
module instances out1 and out2. These named module instances must
be defined elsewhere in the configuration file.
1
2
3
<Route example>
Path in1, in2 => proc => out1, out2
</Route>
Link to this example.
For more details about route instance names, see Configuration in the Reference Manual.
If no Route block is specified in the configuration, NXLog will automatically generate a route, with all the Input and Output instances specified in a single path.
NXLog can use a configuration with no Route block, such as the following.
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
<Input in1>
Module im_null
</Input>
<Input in2>
Module im_null
</Input>
<Output out1>
Module om_null
</Output>
<Output out2>
Module om_null
</Output>
The following Route block will be generated automatically.
1
2
3
<Route r>
Path in1, in2 => out1, out2
</Route>
Link to this example.
23.9. Constant and macro definitions
A define is useful if there are many instances in the configuration where the same value must be used. Typically, defines are used for directories and hostnames. In such cases the value can be configured with a single definition. In addition to constants, other strings like code snippets or parser rules can be defined in this way.
An NXLog define works in a similar way to the C language, where the pre-processor substitutes the value in places where the macro is used. The NXLog configuration parser replaces all occurrences of the defined name with its value, and then after this substitution the configuration check occurs.
This example shows the use of two defines: BASEDIR and
IGNORE_DEBUG. The first is a simple constant, and its value is used
in two File directives. The second is an
NXLog language statement, it is used in an
Exec directive.
1
2
3
4
5
6
7
8
9
10
11
12
13
define BASEDIR /var/log
define IGNORE_DEBUG if $raw_event =~ /debug/ drop();
<Input messages>
Module im_file
File '%BASEDIR%/messages'
</Input>
<Input proftpd>
Module im_file
File '%BASEDIR%/proftpd.log'
Exec %IGNORE_DEBUG%
</Input>
Link to this example.
The define directive can be used for statements as
shown above, but multiple statements should be specified using a code
block, with curly braces ({}), to result in the expected
behavior.
The following example shows an incorrect use of the define directive. After substitution, the drop() procedure will always be executed; only the warning message will be logged conditionally.
1
2
3
4
5
6
7
define ACTION log_warning("dropping message"); drop();
<Input in>
Module im_file
File '/var/log/messages'
Exec if $raw_event =~ /dropme/ %ACTION%
</Input>
To avoid this problem, the action should be defined using a code block.
1
define ACTION { log_warning("dropping message"); drop(); }
Link to this example.
23.10. Environment variables
The envvar directive works like
define except that the value is retrieved
from the environment. This makes it possible to reference the
environment variable as if it was a define. This directive is only
available in NXLog Enterprise Edition.
This is similar to the previous example using a define, but here the value is fetched from the environment.
1
2
3
4
5
6
envvar BASEDIR
<Input in>
Module im_file
File '%BASEDIR%/messages'
</Input>
Link to this example.
23.11. File inclusion
NXLog provides several features for including configuration directives and blocks from separate files or from executables.
|
Note
|
The SpoolDir directive does not take effect until after the configuration has been parsed, so relative paths specified with these directives are relative to the working directory where NXLog was started from. Generally, it is recommended to use absolute paths. If desired, define directives can be used to simulate relative paths (see Using defines to include a configuration file). |
With the include directive it is possible to specify a file or set of files to be included in the current NXLog configuration.
This example includes the contents of the /opt/nxlog/etc/syslog.conf file in
the current configuration.
1
include /opt/nxlog/etc/syslog.conf
Link to this example.
The include directive also supports filenames
containing the wildcard character (*). For example, multiple .conf files
could be saved in the nxlog.d directory—or some other custom configuration
directory—and then automatically included in the NXLog configuration
in ascending alphabetical order along with the nxlog.conf file.
Each included file might contain a small set of configuration information
focused exclusively on a single log source. This essentially establishes a
modular design for maintaining larger configurations. One benefit of this
modular configuration approach is the ability to add/remove .conf files to/
from such a directory for enabling/disabling specific log sources without ever
needing to modify the main nxlog.conf configuration.
This solution could be used to specify OS-specific configuration
snippets (like windows2003.conf) or application-specific snippets
(such as syslog.conf).
Including subdirectories inside the configuration directory is not supported, neither are wildcarded directories.
This example includes all .conf files from the nxlog.d folder on Windows.
1
include C:\Program Files\nxlog\conf\nxlog.d\*.conf
The files can also be included using the define directive.
1
2
define CONFDIR C:\Program Files\nxlog\conf\nxlog.d
include %CONFDIR%\*.conf
Link to this example.
With the include_stdout directive, an external command can be used to provide configuration content. There are many ways this could be used, including fetching, decrypting, and validating a signed configuration from a remote host, or generating configuration content dynamically.
Here, a separate script is responsible for fetching the NXLog configuration.
1
include_stdout /opt/nxlog/etc/fetch_conf.sh
Link to this example.