5
responses

Hello
I'm trying to send Windows DNS logs through NXLog, but i'm having a problem.
I followed the documentation and ended up with the following config file.

Events seem to match the regex, but then i can't seem to use any of the named group names ($Date, $QuestionName, ... any).
I tried to log_info(); but it always shows up as an empty string in the log file :

* This: log_info('q is ' + $QuestionName);
* Shows up in logs as "q is" (and nothing else)

Anyone knows what i'm doing wrong ?
I don't see "no match" in my logfile so i guess events always match the EVENT_REGEX.

Been struggling with this for 24 hours .. even tried unnamed capture groups but also the $0, $1... always show empty.

(config file also at https://pastebin.com/s4CaJg9k in case of problems)
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Panic Soft
#NoFreeOnExit TRUE

define ROOT C:\Program Files (x86)\nxlog
define CERTDIR %ROOT%\cert
define CONFDIR %ROOT%\conf
define LOGDIR %ROOT%\data
define LOGFILE %LOGDIR%\nxlog.log
LogFile %LOGFILE%

Moduledir %ROOT%\modules
CacheDir %ROOT%\data
Pidfile %ROOT%\data\nxlog.pid
SpoolDir %ROOT%\data

# Example data :
#14-09-19 09:20:39 0B64 PACKET 0000005487B8E130 UDP Rcv 172.30.2.30 486a Q [0001 D NOERROR] AAAA (7)outlook(9)office365(3)com(8)transfer(2)be(0)
#14-09-19 09:20:39 0B60 PACKET 0000005487FAC120 UDP Rcv 172.30.1.38 9b88 Q [0001 D NOERROR] AAAA (7)outlook(9)office365(3)com(0)

define EVENT_REGEX /(?x)(?<Date>\d+(?:\-\d+){2})\s (?<Time>\d+(?:\:\d+){2})\s (?<ThreadId>\w+)\s+ (?<Context>\w+)\s+ (?<InternalPacketIdentifier>[[:xdigit:]]+)\s+ (?<Protocol>\w+)\s+ (?<SendReceiveIndicator>\w+)\s (?<RemoteIP>[[:xdigit:].:]+)\s+ (?<Xid>[[:xdigit:]]+)\s (?<QueryType>\s|R)\s (?<Opcode>[A-Z]|\?)\s (?<QFlags>\[(.*?)\])\s+ (?<QuestionType>\w+)\s+ (?<QuestionName>.*)/
define EMPTY_EVENT_REGEX /(^$|^\s+$)/
define DOMAIN_REGEX /\(\d+\)([\w-]+)\(\d+\)([\w-]+)/
define SUBDOMAIN_REGEX /\(\d+\)([\w-]+)\(\d+\)([\w-]+)\(\d+\)(\w+)/
define NOT_STARTING_WITH_DATE_REGEX /^(?!\d+\-\d+\-\d+).+/
define QFLAGS_REGEX /(?x)(?<FlagsHex>\d+)\s+ (?<FlagsCharCodes>\s+|([A-Z]{2}|[A-Z]))\s+ (?<ResponseCode>\w+)/

<Extension _json>
Module xm_json
</Extension>

<Input in>
Module im_file
File 'C:\dnslog\dns.log'
<Exec>
# Drop entries that have empty lines
if $raw_event =~ %EMPTY_EVENT_REGEX% drop();
# Drop entries not starting with date
if $raw_event =~ %NOT_STARTING_WITH_DATE_REGEX% drop();
# Split entries into fields & define regular entries
if $raw_event =~ %EVENT_REGEX%
{
$Regular = TRUE;
#$EventTime = parsedate($Date + " " + $Time);
$Raw = $raw_event;
#delete($date);
#delete($time);
if $FlagsCharCodes =~ /^\s+$/ delete($FlagsCharCodes );
# Convert domains from (8)mydomain(1)com to mydomain.com
if $QuestionName =~ %DOMAIN_REGEX% $QuestionName = $1 + "." + $2;
# Convert domains from (8)sub(8)mydomain(1)com to sub.mydomain.com
if $QuestionName =~ %SUBDOMAIN_REGEX%
$QuestionName = $1 + "." + $2 + "." +$3;

# Set query flags
if $QFlags =~ %QFLAGS_REGEX% delete($QFlags);

# Set the query type
if $QueryType =~ %EMPTY_EVENT_REGEX% $QueryType = "query";
else $QueryType = "response";

log_info('q is ' + $QuestionName);
}
else
{
$Regular = FALSE;
$Raw = $raw_event;
log_info("no match");
}
</Exec>
</Input>

<Output out>
Module om_file
Exec to_json();
File 'C:\output-dns-traffic.json'
</Output>

<Route r1>
path in => out
</Route>

AskedSeptember 19, 2019 - 9:56am

Comments (4)

  • lostence's picture

    It seems i can only use/read/log specifically set variables ($Raw, $raw_event, $QueryType).
    None of the group names in the regex ($Date, $Time, $QuestionName) can be used at all.. yet it seems the regex matches the ~=, as shown by the log_info statements that get logged.

    I must be missing something simple.

  • lostence's picture

    I modified the regex a bit, it matches all my log lines .. ( https://regexr.com/4l9h4 )
    Yet, i cannot use any variable extracted from the regex.

    It's like the "if" statement really only is a conditional comparison between the event and the regex, and doesn't create any usable variable.

    Is it something that only works in the enterprise version or what ?
    (running community edition here).

  • Zhengshi's picture
    (NXLog)

    Named capture groups only currently exist in the Enterprise version.
    Add this at the start of your if structure and you should get the output you are looking for, it should still work with named groups. Check my work of course, I kinda just skipped through it and might have miscounted :)

                $Date = $1;
                $Time = $2;
                $ThreadID = $3;
                $Context = $4;
                $InternalPacketIdentifier = $5;
                $Protocol = $6;
                $SendReceiveIndicator = $7;
                $RemoteIP = $8;
                $Xid = $9;
                $Opcode = $10;
                $QFlags = $11;
                $Group12 = $12;
                $QuestionType = $13;
                $QuestionName = $14;
    
    {
      "EventReceivedTime": "2019-09-19 11:50:27",
      "SourceModuleName": "in",
      "SourceModuleType": "im_file",
      "Date": "19-09-19",
      "Time": "14:14:24",
      "ThreadID": "0B18",
      "Context": "PACKET",
      "InternalPacketIdentifier": "00000055B003C160",
      "Protocol": "UDP",
      "SendReceiveIndicator": "Rcv",
      "RemoteIP": "172.30.0.4",
      "Xid": "a270",
      "Opcode": "Q",
      "Group12": "0001   D   NOERROR",
      "QuestionType": "A",
      "QuestionName": "outlook.office365",
      "Regular": true,
      "Raw": "19-09-19 14:14:24 0B18 PACKET  00000055B003C160 UDP Rcv 172.30.0.4      a270   Q [0001   D   NOERROR] A      (7)outlook(9)office365(3)com(0)",
      "QueryType": "response"
    }
    

  • lostence's picture

    Thanks for taking time to help.

    The suggested workaround works fine, wonderful.
    I wish i didnt waste so much time trying in vain to make something work that was not even supported !

Answer (1)