Nxlog Multiline Input to 1 Syslog Message

Post a different question
3
responses

Hi all,

I'm using nxlog to send CAS audit log to our syslog server. Each entry in the text file looks similar to this as below:

2019-08-28 14:33:58,959 INFO [org.apereo.inspektr.audit.support.Slf4jLoggingAuditTrailManager] - Audit trail record BEGIN
=============================================================
WHO: user1
WHAT: ST-65-eMcuA7IeZWYUYPldhgaT-11 for https://test.com.vn/news/
ACTION: SERVICE_TICKET_CREATED
APPLICATION: CAS
WHEN: Wed Aug 28 14:33:58 ICT 2019
CLIENT IP ADDRESS: x.x.x.x
SERVER IP ADDRESS: x.x.x.x
=============================================================

I want to combine these multiple lines to 1 line and I've read some of the documentation on nxlog's website regarding multiline but haven't found a specific config to put them all in 1 entry with a syslog header. Is there any solution for nxlog conf to work with this kind of multiline message? Thanks

AskedAugust 28, 2019 - 11:45am

Answer (1)

To replace newlines, tabs and carriage returns with blanks, try adding this to the Output section:
Exec $Message = replace($Message, "\t", " "); $Message = replace($Message, "\n", " "); $Message = replace($Message, "\r", " ");

Or if you want to apply it to the raw log, you can also try this in the Output section instead:
$raw_event = replace($raw_event, "\r", " ");

AnsweredAugust 28, 2019 - 11:51pm

Comments (2)

  • dnhphuc's picture

    Thanks Jacob for your help. After I added Exec $Message = replace($Message, "\t", " "); $Message = replace($Message, "\n", " "); $Message = replace($Message, "\r", " "); to my nxlog configuration, the Output is as below:

    WHO: user1
    WHAT: ST-79-F4udY6UcuTTATqNfjZjL-11 for https://test.com.vn/
    ACTION: SERVICE_TICKET_CREATED
    APPLICATION: CAS
    WHEN: Thu Aug 29 10:45:15 ICT 2019
    CLIENT IP ADDRESS: 192.168.0.100
    SERVER IP ADDRESS: 192.168.0.3

    Please correct me if something is wrong. Here is my nxlog configuration. Thanks

    ########################################
    # Global directives #
    ########################################
    #User nxlog
    #Group nxlog

    LogFile /var/log/nxlog/nxlog.log
    LogLevel INFO

    <Extension _syslog>
    Module xm_syslog
    </Extension>

    <Extension json>
    Module xm_json
    </Extension>

    <Extension fileop>
    Module xm_fileop
    </Extension>

    <Extension multi>
    Module xm_multiline
    HeaderLine /(?x)^(?<EventTime>\d{4}\-\d{2}\-\d{2}\ \d{2}\:\d{2}\:\d{2}),\d{3}\ \ (?<Severity>\S+)\ \[(?<Class>\S+)\]\ \-\ (?<Message>[\s\S]+)/
    EndLine /^===============/
    </Extension>

    <Input CAS1>
    Module im_file
    File '/var/log/cas-server/cas_audit.log'
    InputType multi
    Exec if $raw_event =~ s/========[=]+//g {}
    </Input>

    <Output out2>
    Module om_file
    File '/tmp/test.txt'
    Exec $Message = replace($Message, "\t", " "); $Message = replace($Message, "\n", " "); $Message = replace($Message, "\r", " ");
    </Output>

    ########################################
    # Routes #
    ########################################
    <Route 1>
    Path CAS1 => out2
    </Route>

    August 29, 2019 - 5:54am
  • Zhengshi's picture
    (NXLog)

    I think most of the issue is the capture groups in the HeaderLine won't work and you need to put your replace against $raw_event.
    I believe EndLine wont work as well since there are two of them, it would likely match the first.

    <Extension multi>
Module xm_multiline
HeaderLine /^\d{4}\-\d{2}\-\d{2}\s\d{2}\:\d{2}\:\d{2},\d{3}\s/
</Extension>

<Input CAS1>
Module im_file
File '/opt/nxlog/etc/cas_audit.log'
InputType multi
ReadFromLast False
SavePos False
Exec log_info($raw_event);
Exec if $raw_event =~ s/========[=]+//g {};
</Input>

<Output out2>
Module om_file
File '/tmp/test.txt'
Exec $raw_event =~ s/(\t|\R)/ /g;
</Output>

    This should result in output like the following. Note that I duplicated your example so that I had 2 entries just to test. ReadFromLast and SavePos were both set to False as well so it would read from the beginning of the file.

    2019-08-28 14:33:58,959 INFO [org.apereo.inspektr.audit.support.Slf4jLoggingAuditTrailManager] - Audit trail record BEGIN  WHO: user1 WHAT: ST-65-eMcuA7IeZWYUYPldhgaT-11 for https://test.com.vn/news/ ACTION: SERVICE_TICKET_CREATED APPLICATION: CAS WHEN: Wed Aug 28 14:33:58 ICT 2019 CLIENT IP ADDRESS: x.x.x.x SERVER IP ADDRESS: x.x.x.x 
2019-08-28 14:33:58,959 INFO [org.apereo.inspektr.audit.support.Slf4jLoggingAuditTrailManager] - Audit trail record BEGIN  WHO: user1 WHAT: ST-65-eMcuA7IeZWYUYPldhgaT-11 for https://test.com.vn/news/ ACTION: SERVICE_TICKET_CREATED APPLICATION: CAS WHEN: Wed Aug 28 14:33:58 ICT 2019 CLIENT IP ADDRESS: x.x.x.x SERVER IP ADDRESS: x.x.x.x

    August 29, 2019 - 5:49pm