in imfile, can you filter a multiline comment from the start of the file?

Tags:

#1 dlang

I am reading windows dns logs, and at the start of each file, windows adds text at the beginning of each file. Note that windows will do this in the middle of a multiline log message. Ideally what I want is that when the file is opened, im_file will throw away the header and then pass the rest of the file to xm_multiline so that xm_multiline never knows that this was there. As a work-around I could add a bunch of filters to match each known bad line and throw it out, but that's a pretty inefficient way to do things, This would be something similar to onEOF but happen at the start of the file instead.

an example is:

DNS Server log file creation at 11/6/2019 6:19:22 PM Log file wrap at 11/6/2019 6:19:22 PM

Message logging key (for packets - other items use a subset of these fields): Field # Information Values ------- ----------- ------ 1 Date 2 Time 3 Thread ID 4 Context 5 Internal packet identifier 6 UDP/TCP indicator 7 Send/Receive indicator 8 Remote IP 9 Xid (hex) 10 Query/Response R = Response blank = Query 11 Opcode Q = Standard Query N = Notify U = Update ? = Unknown 12 [ Flags (hex) 13 Flags (char codes) A = Authoritative Answer T = Truncated Response D = Recursion Desired R = Recursion Available 14 ResponseCode ] 15 Question Type 16 Question Name

normal log data, potentially starting in the middle of a log

#2 Zhengshi Nxlog ✓
#1 dlang
I am reading windows dns logs, and at the start of each file, windows adds text at the beginning of each file. Note that windows will do this in the middle of a multiline log message. Ideally what I want is that when the file is opened, im_file will throw away the header and then pass the rest of the file to xm_multiline so that xm_multiline never knows that this was there. As a work-around I could add a bunch of filters to match each known bad line and throw it out, but that's a pretty inefficient way to do things, This would be something similar to onEOF but happen at the start of the file instead. an example is: DNS Server log file creation at 11/6/2019 6:19:22 PM Log file wrap at 11/6/2019 6:19:22 PM Message logging key (for packets - other items use a subset of these fields): Field # Information Values ------- ----------- ------ 1 Date 2 Time 3 Thread ID 4 Context 5 Internal packet identifier 6 UDP/TCP indicator 7 Send/Receive indicator 8 Remote IP 9 Xid (hex) 10 Query/Response R = Response blank = Query 11 Opcode Q = Standard Query N = Notify U = Update ? = Unknown 12 [ Flags (hex) 13 Flags (char codes) A = Authoritative Answer T = Truncated Response D = Recursion Desired R = Recursion Available 14 ResponseCode ] 15 Question Type 16 Question Name normal log data, potentially starting in the middle of a log

I don't think there is an better way than just evaluating this within xm_multiline Exec statements. You could probably use some regex so that you don't have to write as many lines though.