Hi All,

Is there a better solution to capturing IIS logs across multiple sites on a single server than adding an input per site in nxlog.conf?

Thanks,

Matt.

currently running 2.10.2150 on windows with a config that reads a debug DNS log (on c:), parses the logs, drops 99% of the logs, and writes the remainder out with file_write()

we are seeing this memory pool allocation error and looking for info about if it's a known issue, something that we should be changing the config to deal with, or what's happening.

our current config is something very close to (filenames may vary):

Panic Soft #NoFreeOnExit TRUE

define ROOT C:\Program Files (x86)\nxlog define CERTDIR %ROOT%\cert #define CONFDIR %ROOT%\conf define CONFDIR "C:\Program Files (x86)\nxlog\conf define LOGDIR %ROOT%\data #define LOGFILE %LOGDIR%\nxlog.log define LOGFILE %LOGDIR%\nxlog.log LogFile %LOGFILE% #LogLevel DEBUG define OUTFILE H:\dnsadvlogs\dns-filtered.log

Moduledir %ROOT%\modules CacheDir %ROOT%\data Pidfile %ROOT%\data\nxlog.pid SpoolDir %ROOT%\data

<Extension _syslog> Module xm_syslog </Extension>

<Extension _json> Module xm_json </Extension>

<Extension _charconv> Module xm_charconv AutodetectCharsets iso8859-2, utf-8, utf-16, utf-32 </Extension>

<Extension _exec> Module xm_exec </Extension>

<Extension _fileop> Module xm_fileop

# Check the log file size every hour and rotate if larger than 5 MB
<Schedule>
    Every 1 hour
    <Exec>
        if (file_exists('%LOGFILE%') and file_size('%LOGFILE%') >= 5M)
            file_cycle('%LOGFILE%', 8);
    </Exec>
</Schedule>

# Rotate log file every week on Sunday at midnight
<Schedule>
    When    @weekly
    Exec    if file_exists('%LOGFILE%') file_cycle('%LOGFILE%', 8);
</Schedule>

</Extension>

<Extension _fileop2> Module xm_fileop

# Rotate log file every week on Sunday at midnight
<Schedule>
    Every 5 min
    Exec    if file_exists('%OUTFILE') file_cycle('%OUTFILE%', 8);
</Schedule>

</Extension>

<Extension win_dns_parser> Module xm_multiline # look for a date at the start of the line HeaderLine /^\d+/\d+/\d+/ #filter blank and header lines from the input Exec if $raw_event =~ /^\s*$/ drop(); Exec if $raw_event =~ /^DNS Server log file creation/ drop(); Exec if $raw_event =~ /^Log file wrap at / drop(); Exec if $raw_event =~ /^Message logging key/ drop(); Exec if $raw_event =~ /^\sField # Information Values/ drop(); Exec if $raw_event =~ /^\s------- ----------- ------/ drop(); Exec if $raw_event =~ /^\s1 Date/ drop(); Exec if $raw_event =~ /^\s2 Time/ drop(); Exec if $raw_event =~ /^\s3 Thread ID$/ drop(); Exec if $raw_event =~ /^\s4 Context$/ drop(); Exec if $raw_event =~ /^\s5 Internal packet identifier$/ drop(); Exec if $raw_event =~ /^\s6 UDP/TCP indicator$/ drop(); Exec if $raw_event =~ /^\s7 Send/Receive indicator$/ drop(); Exec if $raw_event =~ /^\s8 Remote IP$/ drop(); Exec if $raw_event =~ /^\s9 Xid (hex)$/ drop(); Exec if $raw_event =~ /^\s10 Query/Response R = Response$/ drop(); Exec if $raw_event =~ /^\sblank = Query$/ drop(); Exec if $raw_event =~ /^\s11 Opcode Q = Standard Query$/ drop(); Exec if $raw_event =~ /^\sN = Notify$/ drop(); Exec if $raw_event =~ /^\sU = Update$/ drop(); Exec if $raw_event =~ /^\s*? = Unknown$/ drop(); Exec if $raw_event =~ /^\s12 [ Flags (hex)$/ drop(); Exec if $raw_event =~ /^\s13 Flags (char codes) A = Authoritative Answer$/ drop(); Exec if $raw_event =~ /^\sT = Truncated Response$/ drop(); Exec if $raw_event =~ /^\sD = Recursion Desired$/ drop(); Exec if $raw_event =~ /^\sR = Recursion Available$/ drop(); Exec if $raw_event =~ /^\s14 ResponseCode ]$/ drop(); Exec if $raw_event =~ /^\s15 Question Type$/ drop(); Exec if $raw_event =~ /^\s16 Question Name/ drop(); </Extension>

<Input win_dns> Module im_file #File "H:\dnsadvlogs\dns.log" File "H:\dnsadvlogs\dns.log" # for testing we want to re-read from the start of the file each time SavePos False ReadFromLast True InputType win_dns_parser <Exec> $Message = $raw_event; $Message = replace($Message, "\r", ""); </Exec> </Input>

<Output win_dns_trimmed> Module om_file

File "H:\dnsadvlogs\dns-trimmed.log"

File      "H:\dnsadvlogs\dns-trimmed.log"
<Exec>
    # manipulate the log entry here, not in input so that we can do other things with the raw input as well
    $Message =~ /^(?<timestamp>\d+\/\d+\/\d+ \d+\:\d+\:\d+\s+\S+)\s+(?<pid>\d+)\s+(?<win_dns_type>[^ ]*)\s+(?<win_dns_packetID>[^ ]*)\s+(?<win_dns_protocol>[^ ]*)\s+(?<win_dns_direction>[^ ]*)\s+(?<win_dns_IP>[^ ]*)\s+(?<win_dns_hexID>[^ ]*) (?<win_dns_qr>.) (?P<win_dns_opcode>.) \[(?<win_dns_flags_hex>\S+) (?<win_dns_flags>.*) (?<win_dns_resultcode>\S+)\]\s+(?<win_dns_recordType>\S+)\s+(?<win_dns_query>\S*).*ANSWER SECTION:.(?<answer>.+)\s+AUTHORITY SECTION:/s; 
    $timestamp=$1;
    $pid=$2;
    $win_dns_type=$3;
    $win_dns_packetID=$4;
    $win_dns_protocol=$5;
    $win_dns_direction=$6;
    $win_dns_IP=$7;
    $win_dns_hexID=$8;
    $win_dns_qr=$9;
    $win_dns_opcode=$10;
    $win_dns_flags_hex=$11;
    $win_dns_flags=$12;
    $win_dns_resultcode=$13;
    $win_dns_recordType=$14;
    $win_dns_query=$15;
    $answer=$16;
    # drop messages if they are not replies (since the replies contain the query info)
    if $win_dns_qr == " " drop();
    # drop logs that have no answer info
    #if $answer =~ /^\s+empty\s+$/ drop();
    # drop logs that don't parse (if we don't have a requestion IP address, the log is worthless to UBA)
    if not defined $win_dns_IP drop();
    # drop logs from dnsmasq caching servers.
    #if $win_dns_IP IN ("10.16.169.32","10.49.58.4","10.49.58.3","10.16.6.22","199.47.139.239","199.47.139.238","199.47.139.182") drop();
    # for the first pass, just filter the logs, don't change the format
    # this greatly simplifies the Splunk changes needed as the log parsing doesn't need to change
    $orig = replace($raw_event, "\r", "\r\n") + "\r\n";
    file_write("%OUTFILE%",$orig); drop();
    delete($orig); delete($Message);
    delete($EventReceivedTime); delete($SourceModuleName); delete($SourceModuleType); delete($pid);
    delete($win_dns_type); delete($win_dns_packetID); delete($win_dns_protocol); delete($win_dns_direction);
    delete($win_dns_hexID); delete($win_dns_qr); delete($win_dns_opcode); delete($win_dns_flags_hex);
    delete($win_dns_flags);
    $answer = replace($answer,"(3)",".");
    $answer = replace($answer,"(6)",".");
    $answer = replace($answer,"(9)","");
    $answer = replace($answer,"(0)","");
    $answer =~ s/\[\S\S\S\S\]//g;
    $win_dns_query = replace($win_dns_query,"(3)",".");
    $win_dns_query = replace($win_dns_query,"(6)",".");
    $win_dns_query = replace($win_dns_query,"(9)","");
    $win_dns_query =~ s/\(0\)UDP//;
    $win_dns_query =~ s/\(0\)TCP//;
    $win_dns_query =~ s/\[\S\S\S\S\]//g;
    rename_field("win_dns_query","q");
    rename_field("win_dns_resultcode","rc");
    rename_field("win_dns_IP","ip");
    rename_field("win_dns_recordType","type");
    if $rc == "NXDOMAIN" delete($answer);
    to_json();
</Exec>

</Output> <Route win_dns_route> Path win_dns => win_dns_trimmed </Route>

Hello,

I've read a few posts about ingesting multiple .evt files using the im_msvista module. Is there a way to do this?

I can point to one, but I need to look at several.

thank you! Franz

i am getting a cab error when loading on server 2016

Hello, I want to know if it's possible to have a dynamic variable like that : if .... {
$1 = $2 ; } Thanks !

Hello!

How can i get logs from shared folder by domain user? Should i start nxlog service under this user or there is another way?

Hi everybody,

I would like to centralize sysmon events from endpoints using WEF and then forward "forwarded events" from collector server to graylog. Can i do it using Community edition of Nxlog ? If yes,

• can i concerve originated enpoint source ?
• which format does i need to use (SYSLOG or GELF) ?

Thanks

Hello, I parameterized as seen in the examples the nxlog configuration file for the logs of my Windows 2016 servers, but when I restart the services with them. In the nxlog files I find this:

nxlog failed to start: Expected </Extension_gelf> but saw </Extension> at C:\Program Files (x86)\nxlog\conf\nxlog.conf:48

nxlog failed to start: Expected </Extension_gelf> but saw </Extension> at C:\Program Files (x86)\nxlog\conf\nxlog.conf:48

nxlog failed to start: Expected </Extension2> but saw </Extension> at C:\Program Files (x86)\nxlog\conf\nxlog.conf:48

Do you have ideas to list the errors? thank you in advance

Hello,

When using sysmon and nxlog, nxlog.exe is triggering alerts for suspicious file characteristics from sigma
https://github.com/Neo23x0/sigma/blob/master/rules/windows/sysmon/sysmon_susp_file_characteristics.yml

nxlog.exe binary (others?) have missing properties fields like product, fileversion, company, description.

Having those along binary signature would be great!

Thanks a lot for great work!

Hi,

My app runs on Windows and it writes log to the file a.log. When I run NXLog to watch this file (to send log to the server), My app gets CFileException::sharingViolation trying to write log to a.log file. It does not happen always, but when the log is big sometimes it happens and my app can not write log to a.log file.

Searching the internet, I found 'You should open the file with CFile::shareDenyNone' to avoid this problem. So my questions is :

1. Is NXLog opens the file with ' CFile::shareDenyNone'?
2. How can I avoid this problem?

Here's my nxlog.conf 's input configuration.

<Input watchfile> Module im_file File 'C:\Log\a.log' Exec $Message = $raw_event; Exec if file_name() =~ /.\(.)/ $SourceName = $1; SavePos TRUE Recursive TRUE CloseWhenIdle TRUE </Input>

Any comment will be appreciated. Thanks

Harry

Hi Guy's,

We are using nxlog community to ship server logs to a lake, powershell logs are collected on the server but we want to exclude shipping them, after much googling i thought it would be as easy as adding this line to the conf:

Exec if ($SourceName == 'PowerShell (PowerShell)' drop();

This does not work, can any experts in NX out there advise?

Thanks, O.

$message =~ /(?<timestamp>\d+/\d+.\d+ )/s; $ts=$timestamp; $ts2=$1;

$ts2 has the right info, $ts=""

version nxlog-ce-2.10.2150

Hi all, It's possible (using the queries) to ONLY receive logs related to all windows administrators and not related to all users? Thanks.

Hi All,

every once in a while nxlog on some win servers enteres stopped state after this error:

62501: 2019-10-21 04:32:54 ERROR EvtNext failed with error 1717: The interface is unknown.

Does anybody know what might be causing this? All I could find is that this may have been a bug that was resolved in newer versions though. We are using v 4.0.3735

Thanks, Jarka

I am reading windows dns logs, and at the start of each file, windows adds text at the beginning of each file. Note that windows will do this in the middle of a multiline log message. Ideally what I want is that when the file is opened, im_file will throw away the header and then pass the rest of the file to xm_multiline so that xm_multiline never knows that this was there. As a work-around I could add a bunch of filters to match each known bad line and throw it out, but that's a pretty inefficient way to do things, This would be something similar to onEOF but happen at the start of the file instead.

an example is:

DNS Server log file creation at 11/6/2019 6:19:22 PM Log file wrap at 11/6/2019 6:19:22 PM

Message logging key (for packets - other items use a subset of these fields): Field # Information Values ------- ----------- ------ 1 Date 2 Time 3 Thread ID 4 Context 5 Internal packet identifier 6 UDP/TCP indicator 7 Send/Receive indicator 8 Remote IP 9 Xid (hex) 10 Query/Response R = Response blank = Query 11 Opcode Q = Standard Query N = Notify U = Update ? = Unknown 12 [ Flags (hex) 13 Flags (char codes) A = Authoritative Answer T = Truncated Response D = Recursion Desired R = Recursion Available 14 ResponseCode ] 15 Question Type 16 Question Name

normal log data, potentially starting in the middle of a log

Hi. How can I write SQL statement to the variable. For example: define variable $sql_statement = SELECT some_func() and in module om_dbi use this variable like Module om_dbi SQL $sql_statement Driver pgsql

I am parsing DNS logs (windows debug logs) where there may be multiple answers to one query (cname plus IP, or multiple cnames, or multiple IPs) but I want to normalize the output so that each output line contains one and only one answer (with the rest of the output being the same)

so if I start with

source 1.1.1.1 query type A query foo.com answer cname bar.com answer A 2.2.2.2

how can I get my output to be:

source 1.1.1.1 query type A query foo.com answer cname bar.com source 1.1.1.1 query type A query foo.com answer A 2.2.2.2

Hi There,

Using the CE edition on a Server 2012 R2 collector. I see in the logs the following: ERROR Failed to load module from C:\Program Files (x86)\nxlog\modules\output\Module om_udp.dll, The specified module could not be found.

The config file %ROOT% has the correct path as far as I can see. I've reinstalled and once I start/restart services then this message appears.

NXLog Ver: 2.10.2102

Any help appreciated!

I have a Graylog NXLog configuration that contains a multiline input. I'm trying to figure out what the best regex would be for the HeaderLine to delimit each message as a group for each instance of a username that appears in the log.

This is the desired output:

Message 1 should contain: Username : <Username1> Index : <Index> Assigned IP : <IP1> Public IP : <IP2> Group Policy : <GroupPolicy> Login Time : 15:15:34 UTC Fri Nov 1 2019

Message 2 should contain: Username : <Username2> Index : <Index> Assigned IP : <IP1> Public IP : <IP2> Group Policy : <GroupPolicy> Login Time : 15:16:12 UTC Fri Nov 1 2019

Instead, this is what happens:

Message 1: Username Message 2: Index Message 3: Assigned IP Message 4: Public IP

So on and so forth. Looking for advice on what should be put in the HeaderLine so the logs are delimited and can be read properly. Thanks in advance!

Dear All, I'm using this product for a months and it works fine, every windows PC (Win 7 and Win10) sent informations to central linux syslog server. Now it doesn't work because I've changed network settings. All of my Windows computers have got 802.1x authentication so the networking goes up after user login.

Maybe NXlog starts before user login and try to connect syslog server sending some messages, so it doesn't reach syslog remain in running status and don't send any other informations.

Any ideas ? Thanks a lot for you help ! Best Regards Willy