- Introduction
- Deployment
- Configuration
- OS Support
- Integration
- Troubleshooting
- Enterprise Edition Reference Manual
- 127. Man Pages
- 128. Configuration
- 129. Language
- 130. Extension Modules
- 131. Input Modules
- 132. Processor Modules
- 132.1. Blocker (pm_blocker)
- 132.2. Buffer (pm_buffer)
- 132.3. Event Correlator (pm_evcorr)
- 132.4. Filter (pm_filter)
- 132.5. HMAC Message Integrity (pm_hmac)
- 132.6. HMAC Message Integrity Checker (pm_hmac_check)
- 132.7. De-Duplicator (pm_norepeat)
- 132.8. Null (pm_null)
- 132.9. Pattern Matcher (pm_pattern)
- 132.10. Format Converter (pm_transformer)
- 132.11. Timestamping (pm_ts)
- 133. Output Modules
- NXLog Manager
- NXLog Add-Ons
132.2. Buffer (pm_buffer)
Messages received over UDP may be dropped by the operating system if packets are not read from the message buffer fast enough. Some logging subsystems using a small circular buffer can overwrite old logs in the buffer if it is not read, also resulting in loss of log data. Buffering can help in such situations.
Note
|
To examine the supported platforms, see the list of installer packages in the Available Modules chapter. |
The pm_buffer module supports disk- and memory-based log message buffering. If both are required, multiple pm_buffer instances can be used with different settings. Because a memory buffer can be faster, though its size is limited, combining memory and disk based buffering can be a good idea if buffering is frequently used.
The disk-based buffering mode stores the log message data in chunks. When all the data is successfully forwarded from a chunk, it is then deleted in order to save disk space.
Note
|
Using pm_buffer is only recommended when there is a chance of message loss. The built-in flow control in NXLog ensures that messages will not be read by the input module until the output side can send, store, or forward. When reading from files (with im_file) or the Windows EventLog (with im_mseventlog or im_msvistalog) it is rarely necessary to use the pm_buffer module unless log rotation is used. During a rotation, there is a possibility of dropping some data while the output module (im_tcp, for example) is being blocked. |
132.2.1. Configuration
The pm_buffer module accepts the following directives in addition to the common module directives. The MaxSize and Type directives are required.
- CreateDir
-
If set to TRUE, this optional boolean directive instructs the module to create the output directory before opening the file for writing if it does not exist. The default is FALSE.
- MaxSize
-
This mandatory directive specifies the size of the buffer in kilobytes.
- Type
-
This directive can be set to either
Mem
orDisk
to select memory- or disk-based buffering.
- Directory
-
This directory will be used to store the disk buffer file chunks. This is only valid if Type is set to
Disk
.
- WarnLimit
-
This directive specifies an optional limit, smaller than MaxSize, which will trigger a warning message when reached. The log message will not be generated again until the buffer size drops to half of WarnLimit and reaches it again in order to protect against a warning message flood.
132.2.2. Functions
The following functions are exported by pm_buffer.
- integer
buffer_count()
-
Return the number of log messages held in the memory buffer.
- integer
buffer_size()
-
Return the size of the memory buffer in bytes.
132.2.3. Examples
This configuration accepts log messages via UDP and forwards them via TCP. An intermediate memory-based buffer allows the im_udp module instance to continue accepting messages even if the om_tcp output stops working (caused by downtime of the remote host or network issues, for example).
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
<Input udp>
Module im_udp
ListenAddr 0.0.0.0:514
</Input>
<Processor buffer>
Module pm_buffer
# 1 MB buffer
MaxSize 1024
Type Mem
# warn at 512k
WarnLimit 512
</Processor>
<Output tcp>
Module om_tcp
Host 192.168.1.1:1514
</Output>
<Route udp_to_tcp>
Path udp => buffer => tcp
</Route>