Table of Contents
- Introduction
- Deployment
- Configuration
- 23. Configuration overview
- 24. NXLog language
- 25. Reading and receiving logs
- 26. Processing logs
- 26.1. Parsing various log formats
- 26.2. Alerting
- 26.3. Using buffers
- 26.4. Character set conversion
- 26.5. Detecting an inactive agent or log source
- 26.6. Event correlation
- 26.7. Extracting data
- 26.8. Filtering logs
- 26.9. Format conversion
- 26.10. Log rotation and retention
- 26.11. Log classification
- 26.12. Parsing multi-line logs
- 26.13. Rate limiting and traffic shaping of logs
- 26.14. Rewriting and modifying logs
- 26.15. Timestamps
- 27. Forwarding and Storing Logs
- 28. Centralized Log Collection
- 29. NXLog failover mode
- 30. High Availability (HA)
- 31. Encrypted transfer
- 32. Reducing bandwidth and data size
- 33. Reliable message delivery
- 34. Compression and Encryption
- OS Support
- Integration
- Troubleshooting
- Enterprise Edition Reference Manual
- NXLog Manager
- NXLog Add-Ons
26.4. Character set conversion
It is recommended to normalize logs to UTF-8. The xm_charconv module provides character set conversion: the convert_fields() procedure for converting an entire message (all event fields) and a convert() function for converting a string.
Example 95. Character set auto-detection of various input encodings
This configuration shows an example of character set auto-detection. The input file may contain differently encoded lines, but by invoking the convert_fields() procedure, each message will have the character set encoding of its fields detected and then converted to UTF-8 as needed.
nxlog.conf ![[Download file]](images/icons/download_icon.png "Download file")
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
<Extension _charconv>
Module xm_charconv
AutodetectCharsets utf-8, euc-jp, utf-16, utf-32, iso8859-2
</Extension>
<Input filein>
Module im_file
File "tmp/input"
Exec convert_fields("auto", "utf-8");
</Input>
<Output fileout>
Module om_file
File "tmp/output"
</Output>
<Route r>
Path filein => fileout
</Route>