3
responses

What is the most efficient way to parse Microsoft DNS Server debug logs into something more tidy, say into a CSV or KVP format on the nxlog agent?

Consider the following log message:

"24/02/2017 16:37:22 09B0 PACKET  0000009657E7BA40 UDP Rcv 10.0.100.15   a490   Q [0001   D   NOERROR] A      (7)example(3)com(0)"

First of all, what would be the most efficient way performance-wise to convert this into a CSV or KVP format?

And also, is there some other way besides using Exec and replacing parenthesis and numbers in a sed-like manner to get the clean query name? We have tried to use the Exec method before, but we were hitting some serious performance issues on busy DNS servers.

I have currently switched on to using the pm_pattern module to drop invalid log lines (the beginning of the log file and empty lines) and I was wondering if there would be some easy way to perform both of the tasks (the formatting and the cleaning) using the pm_pattern module?

An example output could look something like the following:

datetime=24/02/2017 16:37:22,thread_id=09B0,context=PACKET,packet_id=0000009657E7BA40,protocol=UDP,action=Rcv,remote_ip=10.0.100.15,
xid=a490,event_type=-,opcode=Q,flags_hex=0001,is_authorative=-,is_truncated=-,recursion_desired=D,recursion_available=-,
response_code=NOERROR,question_type=A,question_name=example.com

The empty or "-" values result from fields specified in the DNS debug log format that are not present in the above message (e.g. all possible flags would be "ATDR", and event_type is "-" because "R" marks a response but an empty value (whitespace) marks a query.

And of course, if the above even is possible, would it be too resource consuming?

AskedFebruary 24, 2017 - 3:59pm

Comments (2)

  • tsigidibam's picture

    I currently have the following in my nxlog.conf:

     

    ## Input module for Microsoft DNS server audit logs
    <Input dns>
        Module im_file
        File "C:\\DNSLogs\\dns*.log"
        InputType LineBased
        PollInterval 15
        CloseWhenIdle TRUE
    </Input>

     

    <Processor pattern>
        Module pm_pattern
        PatternFile %ROOT%\conf\patterndb.xml
    </Processor>

     

    ## For outputting the DNS log to Syslog
    <Output default_out>
        Module om_udp
        Host loghost.example.com
        Port 514
        Exec to_syslog_ietf();
    </Output>

     

    <Route 2>
        Path        dns => pattern => default_out
    </Route>

     

    And the patterndb.xml looks like this:

     

    <?xml version='1.0' encoding='UTF-8'?>
    <patterndb>
        <created>2017-24-02 14:00:00</created>
        <version>001</version>

        <group>
            <name>microsoft dns server</name>
            <id>1</id>

            <pattern>
                <id>1</id>
                <name>header row or empty row</name>

                <matchfield>
                    <name>raw_event</name>
                    <type>regexp</type>
                    <value>^[^0-9]</value>
                </matchfield>

                <exec>
                    drop();
                </exec>

            </pattern>

            <pattern>
                <id>2</id>
                <name>drop authorative answers</name>

                <matchfield>
                    <name>raw_event</name>
                    <type>regexp</type>
                    <value>\[[a-fA-F0-9]{4}\sA[\sT]</value>
                </matchfield>

                <exec>
                    drop();
                </exec>

            </pattern>        
        </group>

    </patterndb>

     

    For some reason I am still getting a lot of empty messages, i.e. I only receive the Syslog header. I am NOT getting lines from the log that would match e.g. pattern #2. I thought that the drop() statement in the patterndb.xml would stop the processing pipeline for the lines that match either one of the patterns specified, but this seems to not be the case. What could be wrong here?

  • b0ti's picture
    (NXLog)

    The above drop() in the pattern.xml does the exact same thing as the following:

    Exec if $raw_event =~ /^[^0-9]/ drop();

    In earlier versions to_syslog_*() used $Message only. This was causing an empty payload with only the syslog header in such use cases when transforming plain text files into syslog, but now it also honors $raw_event.

    Also note that pm_pattern will stop matching at the first successful match! If you have multiple patterns that can match the same line it won't be deterministic due to pattern reordering.

    Other than that I'm not sure what the issue is, I suggest removing pm_pattern from your conf to see if it is related.

     

Answer (1)