​

What is the most efficient way to parse Microsoft DNS Server debug logs into something more tidy, say into a CSV or KVP format on the nxlog agent?

Consider the following log message:

"24/02/2017 16:37:22 09B0 PACKET  0000009657E7BA40 UDP Rcv 10.0.100.15   a490   Q [0001   D   NOERROR] A      (7)example(3)com(0)"

First of all, what would be the most efficient way performance-wise to convert this into a CSV or KVP format?

And also, is there some other way besides using Exec and replacing parenthesis and numbers in a sed-like manner to get the clean query name? We have tried to use the Exec method before, but we were hitting some serious performance issues on busy DNS servers.

I have currently switched on to using the pm_pattern module to drop invalid log lines (the beginning of the log file and empty lines) and I was wondering if there would be some easy way to perform both of the tasks (the formatting and the cleaning) using the pm_pattern module?

An example output could look something like the following:

datetime=24/02/2017 16:37:22,thread_id=09B0,context=PACKET,packet_id=0000009657E7BA40,protocol=UDP,action=Rcv,remote_ip=10.0.100.15,
xid=a490,event_type=-,opcode=Q,flags_hex=0001,is_authorative=-,is_truncated=-,recursion_desired=D,recursion_available=-,
response_code=NOERROR,question_type=A,question_name=example.com

The empty or "-" values result from fields specified in the DNS debug log format that are not present in the above message (e.g. all possible flags would be "ATDR", and event_type is "-" because "R" marks a response but an empty value (whitespace) marks a query.

And of course, if the above even is possible, would it be too resource consuming?

Hi,

I'm using nxlog to send logs from Windows eventlog to elasticsearch, and using Kibana view.

I'm getting all the message as it is in the 'Message' column, I want to re-order it so the hostname parameter will be the windows server (and not the elasticsearch server), add 'Type' to the messages, etc.

this is the configuration file of nxlog:

* 55.2.110.4=elasticsearch server

<Extension json>
 Module xm_json
</Extension>

# Nxlog internal logs
<Input internal>
   Module im_internal
   Exec $EventReceivedTime = integer($EventReceivedTime) / 1000000;
</Input>
 
# Windows Event Log
<Input eventlog>
# Uncomment im_msvistalog for Windows Vista/2008 and later
   Module im_msvistalog
 
# Uncomment im_mseventlog for Windows XP/2000/2003
#   Module im_mseventlog
 
   Exec $EventReceivedTime = integer($EventReceivedTime) / 1000000;
</Input>
 
<Output out>
   Module om_http
   URL  https://55.2.110.4:443
   HTTPSAllowUntrusted    TRUE
</Output>
 
<Route 1>
   Path internal, eventlog => out
</Route>

this is an example message from kibana:

 

Field Action Value @timestamp   2015-09-08T07:35:47.064Z @version   1 _id   AU-r4dtqVULqkki94YkZ _index   logstash-2015.09.08 _type   logs host   55.2.110.4 http_port   5005 message  

2015-09-08 07:35:43 dc-prod-zone-a.organization.com AUDIT_SUCCESS 4634 An account was logged off. Subject: Security ID: S-1-5-21-1595779987-1987268195-2987234418-1104 Account Name: DC-PROD-ZONE-C$ Account Domain: ORGANIZATION Logon ID: 0x679381 Logon Type: 3 This event is generated when a logon session is destroyed. It may be positively correlated with a logon event using the Logon ID value. Logon IDs are only unique between reboots on the same computer.

 

Thanks a lot