1
answer

Hi,

i have follwing logentry:

------------- New entry --------------------

line 1

line 2

line 3

line x

{blank line}

------------- New entry --------------------

line 1

line 2

line 3

line x

​{blank line}

 

I try to parse this with the multiline extension with following configuration:

<Extension multiline>
    Module    xm_multiline
    HeaderLine    /^--/
    EndLine    /^$/
</Extension>

<Input in>
    Module    im_file
    File    "input.txt"
    SavePos    FALSE
    ReadFromLast TRUE
    InputType    multiline   
    Exec    if $raw_event !~ /^--/ drop();
    Exec        $raw_event = replace($raw_event, "\r\n", ";");
</Input>

<Output out>
    Module    om_file
    File    "output.txt"
</Output>

<Route 1>
    Path    in => out
</Route>

When i write one line and save then the input-file nxlog outputs only the header. When i write the complete entry a once nxlog works as expected. What iam doing wrong?

 

 

AskedJuly 13, 2015 - 9:50pm

Answer (1)

Incomplete multiline messages will be flushed by im_file if no new data arrives within PollInterval.

AnsweredJuly 14, 2015 - 2:17pm

Comments (3)

  • fiddell's picture

    I red following about this in the documentation (xm_multiline ->:

    Until there is a new header read, the previous message is stored in the buffers because the module does not know where the message ends. The im_file module will forcibly flush this buffer after the configured PollInterval timeout. If this behaviour is unacceptable, consider using some kind of an encapsulation method (JSON, XML, RFC5425, etc) or use an end marker with EndLine if possible.

    I tried the Endline-Parameter and the json-converter. I both cases the PollInterval wins too. Is this right? 

    July 14, 2015 - 8:47pm
  • adm's picture
    (NXLog)

    I'm not sure I understand the issue. Is your log source writing to the file by appending lines so that there is a delay between the lines being appended even for a single event? Otherwise I suspect you are testing manually your typing speed falls short?

    When using the recommended JSON encapsulation the xm_multiline is not needed since xm_json needs single line JSON without linebreaks.

    July 14, 2015 - 9:56pm
  • fiddell's picture

    Yes the log source writes lines with delay. All this lines together are then a single event.  The Logformat is very ugly the togetherness of the lines is only about formatting recognized.

    OK, JSON is therefore not an option for me because i'ver several lines with line breaks. The passage of text in the documentation is misleading.

    I think i have an solution now for my usecase, it is based on the Example 9.5 of the documentation:

    Exec    if $raw_event =~ /^--.*/ \
                { \
                    if defined(get_var('saved')) \
                    { \
                        $tmp = $raw_event; \
                        set_var('saved', $tmp); \
                        $tmp = undef; \
                        drop(); \
                    } \
                    else \
                    { \
                        set_var('saved', $raw_event); \
                        drop(); \
                    } \
                } \
                else \
                { \
                    if $raw_event =~ /^\s*$/ \
                    { \
                        $raw_event = get_var('saved'); \
                        $saved = undef; \
                        log_info($raw_event); \
                    } \
                    else \
                    {\
                        set_var('saved', get_var('saved') + $raw_event + ";");\
                        drop(); \
                    }\
                }

    July 14, 2015 - 10:24pm