OK, first cry for help. I've an install that I have inherited from a colleague - until four days ago I had never seen nxlog - and it has caused some performance issues.

The issue is that the are problems are occurring on VDIs on a cloud infrastructure. The cloud provider (a local operation, not one of the big boys like Amazon and Azure) says that they keep seeing machines suddenly taking large amounts of memory for periods of up to 5 minutes. This stops new processes being created (so you can't start task manager to ientify which process is hogging the memory, and in some cases it locks the VDI up so that the user has to be moved off it. The cloud providers are saying that this was not aproblem before nxlog was installed at the start of the week.

(However, the reason my colleague installed it is that the infrastructure has been attacked and there's at least some indicaion of compromise: so we are logging it to track any possible misuse) 

Am I right in thinking that adding a sleep statement to the conf file (which the documentation describes as a kind of poor man's rate limiting), passing the conf file out to all the instances on the individual VDIs, and then restarting services, should alleviate this? Or need I look elsewhere for the problem?

Like I said, please make allowances for n00b status..

J

Hello :),

I make a file rotation when my logs size exceed with 'file_cycle'.

I have something like this : 

    Exec    if outUDP->file_size() > 10k  \
            {  \
                log_info('size exceeded');  \
                file_cycle(%LOGDIR% + $programSource + '\\' + get_var('date') + '.log'); \
                outUDP->reopen();  \
            }

So my logs paths are :

C:\logs\programA\20170329.log
C:\logs\programA\20170329.log.1
C:\logs\programA\20170329.log.2
C:\logs\programB\20170329.log.1
C:\logs\programB\20170329.log.2

But I wanted something more like :

C:\logs\programA\20170329.log
C:\logs\programA\20170329.1.log
C:\logs\programA\20170329.2.log
C:\logs\programB\20170329.1.log
C:\logs\programB\20170329.2.log

I wanted to make it by myself using 'rotate_to' instead of 'file_cycle' .
The probleme is if nxlog restarts, I need it to continue the file rotation without overwrite the files that already exist. I can't check if C:\logs\programA\20170329.1.log exists then C:\logs\programA\20170329.2.log until I find the last...

It is possible to change it using 'file_cycle'? Or do you have another idea?

Thanks :)

I get this error:

MSI (s) (E4:9C) [19:13:39:684]: Product: NXLog-CE -- Error 1401. Could not create key: S-1-5-21-1688939787-1905098679-239004068-17022\Software\Microsoft.  System error 87.  Verify that you have sufficient access to that key, or contact your support personnel.

Error 1401. Could not create key: S-1-5-21-1688939787-1905098679-239004068-17022\Software\Microsoft.  System error 87.  Verify that you have sufficient access to that key, or contact your support personnel.

when using this method as part of a script...

$process = ([WMICLASS]"\\$server\ROOT\CIMV2:Win32_Process").Create("cmd.exe /c " + $remoteProcess1)

The command issued is...

msiexec.exe /i nxlog-ce.msi /q

This method works great for other MSI packages

Any one know why this might be happening?  I ensured that the account creating the remote process has administrative rights.

Why would this need to access the HKEY_Users hive?

I also noticed that if I load the same User Profile on the remote machine that I am using to start the remote process, the installation works fine.

Once that is done no other Administrator can remove the NXLog agent.  I try using the uninstall string and it fails.  THe only way to remove NXLog is to log on as the same user and uninstall.

Where can I find the documenation for this om_redis module?

Only resource I can find is: https://nxlog.co/question/1593/redis-module-lpush

Hi All,

I have previously got smtp logs to go into Graylog using NXlog, it was worknig fine. I then had a disk sapce issue on the graylog host so had to redo some bits, including the nxlog.conf for our SMTP server. 

The SMTP log header specifes the following

#Software: Microsoft Internet Information Services 8.5
#Version: 1.0
#Date: 2017-03-26 23:00:10
#Fields: date time c-ip cs-username s-sitename s-computername s-port cs-method cs-uri-query sc-bytes time-taken 

This is the error I am getting

ERROR if-else failed at line 44, character 436 in C:\Program Files (x86)\nxlog\conf\nxlog.conf. statement execution has been aborted; procedure 'parse_csv' failed at line 44, character 224 in C:\Program Files (x86)\nxlog\conf\nxlog.conf. statement execution has been aborted; Not enough fields in CSV input, expected 11, got 8 in input

I ahve checked and rechecked and their should be 11 items as per the .conf.

define ROOT C:\Program Files (x86)\nxlog

Moduledir %ROOT%\modules
CacheDir %ROOT%\data
Pidfile %ROOT%\data\nxlog.pid
SpoolDir %ROOT%\data
LogFile %ROOT%\data\nxlog.log

<Extension gelf>
    Module         xm_gelf
</Extension>

<Extension fileop>
    Module         xm_fileop
</Extension>

 <Extension json>
    Module      xm_json
</Extension>

# Create the parse rule for IIS logs. You can copy these from the header of the IIS log file.
<Extension w3c>
    Module             xm_csv
    Fields             $date, $time, $c-ip, $cs-username, $s-sitename, $s-computername, $s-port, $cs-method, $cs-uri-query, $sc-bytes, $time-taken
    FieldTypes         string, string, string, string, string, string, integer, string, string, integer, integer
    Delimiter         ' '
    QuoteChar         '"'
    EscapeControl     FALSE
    UndefValue         '-'
</Extension>

<Input smtp>
    Module        im_file
    File        "C:\\Logs\\SMTPSVC1\\\ex*.log"
    SavePos      TRUE

    Exec        if $raw_event =~ /^#/ drop();                    \
                else                                             \
                {                                                \
                    w3c->parse_csv();                            \
                    $EventTime = parsedate($date + " " + $time); \
                    $SourceName = "smtp";                         \
                    $Message = to_json();                         \
                }
</Input>

<Input eventlog>
    Module      im_msvistalog
</Input>

<Output graylog>
    Module      om_udp
    Host        graylog.mydomain.com
    Port        12201
    OutputType    GELF

    #Use the following line for debugging (uncomment the fileop extension above as well)
    #Exec file_write("C:\\Program Files (x86)\\nxlog\\data\\nxlog_output.log", $raw_event);
</Output>

<Route eventlog>
   Path        eventlog => graylog
</Route>

<Route smtp-to-graylog>
    Path        smtp => graylog
</Route>

Its so frustracting that I know this was working correctly.

Any help would be great.

Thanks

I'm struggling with rotating syslog files at midnight so that they are named "YYYY-MM-DD.log" (and contain log records for that date).

I wonder if anyone has an example of the best way to achieve this.  The manual could use an example like this.

Thanks

Phil

I'm attempting to use NXlog to perform a one time read of a CSV file which will then be passed to Fluent-D for processing and writing to a MySQL database.

I've read through the documentation for setting up the nxlog.config file but I'm at a loss to how to configure for my CSV file.

Has anyone has a config file that reads a CSV I'd apreciate the help.

Thanks. 

Hello.

While reading log from windows, I got string with two encodings WIN-1251 and UTF-8:

like 2017-02-21 16:40:24 IT-73.domain.name INFO 44 NT AUTHORITY\҈Ғƌ�Центр обновления Windows начал скачивать обновление.

where all message in utf-8, and part with AccountName that (usually NT AUTHORITY\SYSTEM in english version of Windows 7) but in russian version its NT AUTHORITY\СИСТЕМА and in logs it looks like NT AUTHORITY\҈Ғƌ�

If I use 

convert_fields("AUTO", "utf-8")

I got proper AccountName, but all other parts got wrong encoding.

Is where any way to replace that part with correct encoding?

Hello,

nxlog CE v2.9.1504, Windows Server 2008 Enterprise

relevant part of config file:

<Input eventlog>
# Uncomment im_msvistalog for Windows Vista/2008 and later
    Module im_msvistalog
    Exec if ($Severity == 'INFO') drop();
<QueryXML>
    <QueryList>
        <Query Id="0">
            <Select Path="System">*</Select>
            <Select Path="Security">*</Select>
            <Select Path="Application">*</Select>
<!-- EventID 2137 - Shrepoint Health Analyzer - ignore -->
            <Suppress Path="Application">*[System[(EventID=2137)]]</Suppress>
<!-- EventID 2138 - Shrepoint Health Analyzer - ignore -->
            <Suppress Path="Application">*[System[(EventID=2138)]]</Suppress>
            <Select Path="Microsoft-Windows-TaskScheduler/Operational">*</Select>
        </Query>
    </QueryList>
</QueryXML>
# Uncomment im_mseventlog for Windows XP/2000/2003
#   Module im_mseventlog
</Input>

One event (EventID 1309 from Application channel) always returns an empty message field (message:null)  as you can see in debug output.

{"EventTime":"2017-03-21 09:54:13","Hostname":"HOST.DOMAIN.TLD","Keywords":36028797018963968,
"EventType":"WARNING","SeverityValue":3,"Severity":"WARNING","EventID":1309,"SourceName":"ASP.NET 2.0.50727.0",
"Task":3,"RecordNumber":1013344,"ProcessID":0,"ThreadID":0,"Channel":"Application","ERROR_EVT_UNRESOLVED":true,
"Category":"Web Event","EventReceivedTime":"2017-03-21 09:54:13","SourceModuleName":"eventlog","SourceModuleType":"im_msvistalog",
"message":null}

How can I get more informations, why those events has a null message field?
What means "ERROR_EVT_UNRESOLVED":true?

Thank you.

hello:

I find some problem under AIX6.1 system.  

the log:

2017-03-21 14:02:41 WARNING additional group memberships couldn't be set because getgrouplist()and setgroups() are not available on this platform
2017-03-21 14:02:41 INFO nxlog-ce-2.8.1248 started
2017-03-21 14:02:41 ERROR failed to open /u01/app/oracle/diag/rdbms/testdb/testdb/trace/alert_testdb.log;Permission denied
2017-03-21 14:02:43 ERROR apr_stat failed on file /u01/app/oracle/diag/rdbms/testdb/testdb/trace/alert_testdb.log;Permission denied
2017-03-21 14:02:49 ERROR last message repeated 2 times
2017-03-21 14:02:57 ERROR apr_stat failed on file /u01/app/oracle/diag/rdbms/testdb/testdb/trace/alert_testdb.log;Permission denied
2017-03-21 14:03:13 ERROR apr_stat failed on file /u01/app/oracle/diag/rdbms/testdb/testdb/trace/alert_testdb.log;Permission denied
2017-03-21 14:03:45 ERROR apr_stat failed on file /u01/app/oracle/diag/rdbms/testdb/testdb/trace/alert_testdb.log;Permission denied
2017-03-21 14:04:50 ERROR apr_stat failed on file /u01/app/oracle/diag/rdbms/testdb/testdb/trace/alert_testdb.log;Permission denied

and i sccan the source code, find:

static void nxlog_set_groups(nxlog_t *nxlog, apr_uid_t uid)
{
    gid_t grplist[100];
    int ngroups = sizeof(grplist);
    char *user;
    apr_status_t rv;

    if ( (rv = apr_uid_name_get(&user, uid, nxlog->pool)) != APR_SUCCESS )
    {
    log_aprerror(rv, "couldn't resolve uid %d to name", uid);
    return;
    }
#ifdef HAVE_GETGROUPLIST 
# ifdef HAVE_SETGROUPS
    if ( getgrouplist(user, getgid(), grplist, &ngroups) == -1 )
    {
    log_error("couldn't get group membership for user %s (uid: %d), too many groups?", user, uid);
    return;
    }
    if ( setgroups((size_t) ngroups, grplist) != 0 )
    {
    log_errno("couldn't get group membership for user %s (uid: %d), setgroups() failed",
          user, uid);
    return;
    }
# else
    log_warn("additional group memberships couldn't be set because getgrouplist()"
         "and setgroups() are not available on this platform");
# endif
#else
    log_warn("additional group memberships couldn't be set because getgrouplist()"
         "and setgroups() are not available on this platform");
#endif

}

I don't what' wrong with this problem, thanks.

Hi All,

Checking to see if anyone has run into this.  I have a windows eventlog collector, with a subscription setup to move specific security audit events to the "Forwarded Events" log.  From there, I am looking to push those logs to Sumologic.  Unfortunately Sumo's collector does not handle this well due to the out of sequence EventRecordID of the various events coming from multiple desktops/servers we're collecting from.

Question:

I'm trying to take advantage of Sumo's native Windows eventlog parser, however the options for sending the eventlog data using NXlog send in the specific formats, syslog_snare, xml, json, etc.  Is there a configuration i can use which send the messages as windows eventlog format?  You'll see from my config below, I've tried several formats, to no avail.  Any suggestions would be greatly appreciated.

<Input eventlog>
    Module      im_msvistalog
<QueryXML>
   <QueryList>                    
     <Query Id="0"> 
        <Select Path="ForwardedEvents">*</Select>
     </Query>
   </QueryList>
</QueryXML>
</Input>

<Output out>
    Module      om_tcp
    Host        10.x.x.x
    Port        514
#    Exec       to_xml();
       Exec to_syslog_snare();
#    Exec $raw_event = replace($raw_event, "\r\n"," ");
#    Exec $raw_event = replace($raw_event, "\t", " ");
#    Exec    $raw_event();
</Output>

<Route 1>
# Path in => out
 Path eventlog, internal => out
</Route>

Thanks in advance,

-A

Hello All,

I am new to nxlog or rather logics in terms of regex and all. 

I am looking to drop any message which has *.*.*.255 in message field. I tried below , however it does not seem to be working for me:

Exec if $Message =~ /^([01][0-9][0-9]|2[0-4][0-9]|25[0-5]) . ^([01][0-9][0-9]|2[0-4][0-9]|25[0-5]) . ^([01][0-9][0-9]|2[0-4][0-9]|25[0-5]) . 255/  drop();

Please suggest

config file:

## This is a sample configuration file. See the nxlog reference manual about the

## configuration options. It should be installed locally under

## /usr/share/doc/nxlog-ce/ and is also available online at

## http://nxlog.org/nxlog-docs/en/nxlog-reference-manual.html

########################################

# Global directives                    #

########################################

User nxlog

Group nxlog

LogFile /var/log/nxlog/nxlog.log

LogLevel INFO

########################################

# Modules                              #

########################################

<Extension _syslog>

    Module      xm_syslog

</Extension>

<Extension multiline>

 Module xm_multiline

 HeaderLine /^\w{3}\s\w{3}\s\d{2}\s\d{2}:\d{2}:\d{2}\s\d{4}/

</Extension>

<Input oerrorin1>

 Module    im_file

 File     '/u01/app/oracle/diag/rdbms/oraywsoudb/oraywsoudb/trace/alert_oraywsoudb.log'

 InputType multiline

 SavePos TRUE

 ReadFromLast  FALSE

</Input>

<Output udpout>

 Module    om_udp

 Host   10.1.227.45

 Port   514

</Output>s

########################################

# Routes                               #

########################################

<Route 1>

    Path        oerrorin1 =>udpout

</Route>

then i get the log:

2017-03-17 16:16:39 INFO nxlog-ce-2.8.1248 started

2017-03-17 16:16:39 ERROR failed to open /u01/app/oracle/diag/rdbms/oraywsoudb/oraywsoudb/trace/alert_oraywsoudb.log;Permission denied

2017-03-17 16:16:41 ERROR apr_stat failed on file /u01/app/oracle/diag/rdbms/oraywsoudb/oraywsoudb/trace/alert_oraywsoudb.log;Permission denied

2017-03-17 16:16:47 ERROR last message repeated 2 times

2017-03-17 16:16:55 ERROR apr_stat failed on file /u01/app/oracle/diag/rdbms/oraywsoudb/oraywsoudb/trace/alert_oraywsoudb.log;Permission denied

2017-03-17 16:17:11 ERROR apr_stat failed on file /u01/app/oracle/diag/rdbms/oraywsoudb/oraywsoudb/trace/alert_oraywsoudb.log;Permission denied

2017-03-17 16:17:43 ERROR apr_stat failed on file /u01/app/oracle/diag/rdbms/oraywsoudb/oraywsoudb/trace/alert_oraywsoudb.log;Permission denied

2017-03-17 16:18:47 ERROR apr_stat failed on file /u01/app/oracle/diag/rdbms/oraywsoudb/oraywsoudb/trace/alert_oraywsoudb.log;Permission denied

2017-03-17 16:20:55 ERROR apr_stat failed on file /u01/app/oracle/diag/rdbms/oraywsoudb/oraywsoudb/trace/alert_oraywsoudb.log;Permission denied

2017-03-17 16:25:11 ERROR apr_stat failed on file /u01/app/oracle/diag/rdbms/oraywsoudb/oraywsoudb/trace/alert_oraywsoudb.log;Permission denied

2017-03-17 16:34:16 DEBUG reading config cache from /var/spool/nxlog/configcache.dat

2017-03-17 16:34:16 DEBUG Setting up module '_syslog' using xm_syslog

2017-03-17 16:34:16 DEBUG module _syslog has 4 exported functions

2017-03-17 16:34:16 DEBUG registering function syslog_facility_value

2017-03-17 16:34:16 DEBUG function 'syslog_facility_value' registered

2017-03-17 16:34:16 DEBUG registering function syslog_facility_string

2017-03-17 16:34:16 DEBUG function 'syslog_facility_string' registered

2017-03-17 16:34:16 DEBUG registering function syslog_severity_value

2017-03-17 16:34:16 DEBUG function 'syslog_severity_value' registered

2017-03-17 16:34:16 DEBUG registering function syslog_severity_string

2017-03-17 16:34:16 DEBUG function 'syslog_severity_string' registered

2017-03-17 16:34:16 DEBUG module _syslog has 9 exported procedures

2017-03-17 16:34:16 DEBUG registering procedure parse_syslog

2017-03-17 16:34:16 DEBUG procedure 'parse_syslog' registered

2017-03-17 16:34:16 DEBUG registering procedure parse_syslog

2017-03-17 16:34:16 DEBUG procedure 'parse_syslog' registered

2017-03-17 16:34:16 DEBUG registering procedure parse_syslog_bsd

2017-03-17 16:34:16 DEBUG procedure 'parse_syslog_bsd' registered

2017-03-17 16:34:16 DEBUG registering procedure parse_syslog_bsd

2017-03-17 16:34:16 DEBUG procedure 'parse_syslog_bsd' registered

2017-03-17 16:34:16 DEBUG registering procedure parse_syslog_ietf

2017-03-17 16:34:16 DEBUG procedure 'parse_syslog_ietf' registered

2017-03-17 16:34:16 DEBUG registering procedure parse_syslog_ietf

2017-03-17 16:34:16 DEBUG procedure 'parse_syslog_ietf' registered

2017-03-17 16:34:16 DEBUG registering procedure to_syslog_bsd

2017-03-17 16:34:16 DEBUG procedure 'to_syslog_bsd' registered

2017-03-17 16:34:16 DEBUG registering procedure to_syslog_ietf

2017-03-17 16:34:16 DEBUG procedure 'to_syslog_ietf' registered

2017-03-17 16:34:16 DEBUG registering procedure to_syslog_snare

2017-03-17 16:34:16 DEBUG procedure 'to_syslog_snare' registered

2017-03-17 16:34:16 DEBUG Setting up module 'multiline' using xm_multiline

2017-03-17 16:34:16 DEBUG Setting up module 'oerrorin1' using im_file

2017-03-17 16:34:16 DEBUG module oerrorin1 has 1 exported functions

2017-03-17 16:34:16 DEBUG registering function file_name

2017-03-17 16:34:16 DEBUG function 'file_name' registered

2017-03-17 16:34:16 DEBUG module oerrorin1 has 0 exported procedures

2017-03-17 16:34:16 DEBUG FlowControl enabled for oerrorin1

2017-03-17 16:34:16 DEBUG Setting up module 'udpout' using om_udp

2017-03-17 16:34:16 DEBUG Setting up module 'fileout2' using om_file

2017-03-17 16:34:16 DEBUG module fileout2 has 2 exported functions

2017-03-17 16:34:16 DEBUG registering function file_name

2017-03-17 16:34:16 DEBUG function 'file_name' registered

2017-03-17 16:34:16 DEBUG registering function file_size

2017-03-17 16:34:16 DEBUG function 'file_size' registered

2017-03-17 16:34:16 DEBUG module fileout2 has 2 exported procedures

2017-03-17 16:34:16 DEBUG registering procedure rotate_to

2017-03-17 16:34:16 DEBUG procedure 'rotate_to' registered

2017-03-17 16:34:16 DEBUG registering procedure reopen

2017-03-17 16:34:16 DEBUG procedure 'reopen' registered

2017-03-17 16:34:16 DEBUG CONFIG: _syslog

2017-03-17 16:34:16 DEBUG inputreader 'Syslog_TLS' registered

2017-03-17 16:34:16 DEBUG Inputreader 'Syslog_TLS' registered

2017-03-17 16:34:16 DEBUG outputwriter 'Syslog_TLS' registered

2017-03-17 16:34:16 DEBUG Outputwriter 'Syslog_TLS' registered

2017-03-17 16:34:16 DEBUG CONFIG: multiline

2017-03-17 16:34:16 DEBUG inputreader 'multiline' registered

2017-03-17 16:34:16 DEBUG Inputreader 'multiline' registered

2017-03-17 16:34:16 DEBUG CONFIG: oerrorin1

2017-03-17 16:34:16 DEBUG adding string [/u01/app/oracle/diag/rdbms/oraywsoudb/oraywsoudb/trace/alert_oraywsoudb.log]

2017-03-17 16:34:16 DEBUG string literal declared at line 29, character 86 in /etc/nxlog.conf

2017-03-17 16:34:16 DEBUG literal

2017-03-17 16:34:16 DEBUG parsed expression

2017-03-17 16:34:16 DEBUG CONFIG: udpout

2017-03-17 16:34:16 DEBUG CONFIG: fileout2

2017-03-17 16:34:16 DEBUG adding string [/var/log/logmsg2.txt]

2017-03-17 16:34:16 DEBUG string literal declared at line 43, character 27 in /etc/nxlog.conf

2017-03-17 16:34:16 DEBUG literal

2017-03-17 16:34:16 DEBUG parsed expression

2017-03-17 16:34:16 DEBUG pidfile /var/run/nxlog/nxlog.pid removed

2017-03-17 16:34:16 DEBUG daemonizing...

2017-03-17 16:34:16 DEBUG INIT: _syslog

2017-03-17 16:34:16 DEBUG INIT: multiline

2017-03-17 16:34:16 DEBUG INIT: oerrorin1

2017-03-17 16:34:16 DEBUG INIT: udpout

2017-03-17 16:34:16 DEBUG Pollset initialized for module udpout (method: poll)

2017-03-17 16:34:16 DEBUG INIT: fileout2

2017-03-17 16:34:16 DEBUG Pollset initialized for module fileout2 (method: poll)

2017-03-17 16:34:16 DEBUG now running as group nxlog

2017-03-17 16:34:16 WARNING additional group memberships couldn't be set because getgrouplist()and setgroups() are not available on this platform

2017-03-17 16:34:16 DEBUG now running as user nxlog

2017-03-17 16:34:16 DEBUG running as uid: 203, euid: 203, gid: 204, egid: 204

2017-03-17 16:34:16 DEBUG pidfile /var/run/nxlog/nxlog.pid created

2017-03-17 16:34:16 DEBUG parsing path: oerrorin1 =>udpout

2017-03-17 16:34:16 DEBUG adding module oerrorin1 to route 1

2017-03-17 16:34:16 DEBUG adding module udpout to route 1

2017-03-17 16:34:16 DEBUG parsing path: in2 => fileout2

2017-03-17 16:34:16 ERROR [router.c:68/nx_route_add_module()] module 'in2' is not declared at /etc/nxlog.conf:54

2017-03-17 16:34:16 DEBUG adding module fileout2 to route tcproute

2017-03-17 16:34:16 ERROR [router.c:347/nx_add_route()] route tcproute is not functional without input modules, ignored at /etc/nxlog.conf:54

2017-03-17 16:34:16 DEBUG jobgroup created with priority 99

2017-03-17 16:34:16 DEBUG jobgroup created with priority 10

2017-03-17 16:34:16 DEBUG spawning 3 worker threads

2017-03-17 16:34:16 DEBUG worker thread 0 started

2017-03-17 16:34:16 DEBUG worker thread 1 started

2017-03-17 16:34:16 DEBUG worker thread 2 started

I have no idea about this problem. 

/u01/app/oracle/diag/rdbms/oraywsoudb/oraywsoudb/trace/alert_oraywsoudb.log   --> user and group is oracle:oinstall

and i add the nxlog user in oinstall group， but why continue report Permission denied?

Hi there!

I am a college student...i am working on windows event logger to collect & analyse windows logs(event & syslog).

I don't know how to configure nxlog.conf file such that ... i can get all windows generated logs locally on host machine.

Waiting for reply.

Thank You

have recently been trying the Community Edition of nxlog which does exactly what I need (and more) but have an issue with the msi. In order to be something I could deploy it has to pass uninstall and re-install tests. Unfortunatley the uninstall via Windows installer leaves a lot of file and registry entries behind that subsequently causes issues with a re-installation. Does anyone know if this is likely to be addressed within CE? Does anyone know if Enterprise edition has the same msi issue? Thanks. 

Hello All,

I am using nxlog-ce-2.9.1716 client to collect logs from my device and then forward it to logstash. I want to make sure nxlog client accept syslogs from particular source instead any. However when using specific IP in im_udp module, I am getting error as below:

"2017-03-07 16:04:51 ERROR failed to start im_udp; couldn't bind udp socket to 172.20.20.20:514; The requested address is not valid in its context. " 

my input driver is configured as below:

<Input in>
    Module    im_udp
    Host    172.20.20.20
    Port    514
</Input>

I tried by changing host as below as well:

Host 172.20.20.20/32 - No luck

Host 172.20.20.20/255.255.255.0 - No luck

Host 172.20.20.20/255.255.255.255 - No luck

Host 172.20.20.20 255.255.255.255 - no luck

Host 172.20.20.20 255.255.255.0 - No luck

However if I make Host 0.0.0.0 - it works , but this is not what I am looking for. 

Please help if I am missing anything. 

Regards,

Gaurang 

Hi!

I'm new to NXLOG so sorry if this is a dumb question.

So I have a Windows 2012 server that I'm attempting to set up to accept Syslog messages from an outsourced proxy system. I've been able to get NXLOG accept the logs and dump them to a flat file which our SIEM tool can pick up. However it ends up being a gigantic file... 

I'm trying to refine this now.

Instead of a huge file that we purge out every day, I'd like to have NXLOG limit the log dump to something like 200MB, copy the now older 200MB file to a different directory appending a time stamp to it, then start collecting in a new file in the original folder. The catch is, the live log that is in the original folder needs to keep the same file name, otherwise our SIEM tool won't know what to look for.

I've tried to have a PowerShell script do this and it works, however, if the NXLOG service is running, the log dump file is locked. We could get around this by disabling the service while copying the old log file to a new location, however we'd lose 10-30 minutes of logs in the file copy (not ideal).

Going through the reference manual I see a few ways to limit file size, starting a new file, and copying files over directly in the NXLOG config file. However it's turning out to be a problem condencing all of that into one configuration file that works on Windows. I've only been able to get a 1 of 3 pieces working at one time.

I appreciate any help the community can provide! Let me know if there are any questions or if something isn't clear.

​

What is the most efficient way to parse Microsoft DNS Server debug logs into something more tidy, say into a CSV or KVP format on the nxlog agent?

Consider the following log message:

"24/02/2017 16:37:22 09B0 PACKET  0000009657E7BA40 UDP Rcv 10.0.100.15   a490   Q [0001   D   NOERROR] A      (7)example(3)com(0)"

First of all, what would be the most efficient way performance-wise to convert this into a CSV or KVP format?

And also, is there some other way besides using Exec and replacing parenthesis and numbers in a sed-like manner to get the clean query name? We have tried to use the Exec method before, but we were hitting some serious performance issues on busy DNS servers.

I have currently switched on to using the pm_pattern module to drop invalid log lines (the beginning of the log file and empty lines) and I was wondering if there would be some easy way to perform both of the tasks (the formatting and the cleaning) using the pm_pattern module?

An example output could look something like the following:

datetime=24/02/2017 16:37:22,thread_id=09B0,context=PACKET,packet_id=0000009657E7BA40,protocol=UDP,action=Rcv,remote_ip=10.0.100.15,
xid=a490,event_type=-,opcode=Q,flags_hex=0001,is_authorative=-,is_truncated=-,recursion_desired=D,recursion_available=-,
response_code=NOERROR,question_type=A,question_name=example.com

The empty or "-" values result from fields specified in the DNS debug log format that are not present in the above message (e.g. all possible flags would be "ATDR", and event_type is "-" because "R" marks a response but an empty value (whitespace) marks a query.

And of course, if the above even is possible, would it be too resource consuming?

I'm not sure how to characterize what's going on, but here goes...

My route path has two inputs, an itermittently high-volume input, and a low-volume input. The high-volume input can be thousands in a couple minutes, or it can be practically nothing. The low-volume input is, at most, one or two entries per ~3-4 minutes. There are also three outputs, two HTTP and a rotated file. They are disconnecting a fair bit, presumably due to timeouts or lack of pipelining or something.

In my current configuration, I consistently find that my log data for the high-volume log gets dropped after a couple minutes. I'm not sure if the timing correlates with the HTTP disconnects, but it might. Sometimes I get just over a thousand log lines through, sometimes I get a couple hundred log lines, sometimes I get a couple thousand. Interestingly, the low-volume log is unaffected.

I do have flow control enabled, and putting a buffer on the inputs did not seem to help. I didn't try disabling flow control because I don't understand it very well. I have to have both inputs going into the same route because how the messages interleave is important to the meaning of the entries.

Here's the path that I use:

  Path       vg_tsw_client, vg_tsw_combat => vg_tsw_pattern => vg_tsw_unparse_finder => vg_tsw_testfile, vg_tsw_es, vg_tsw_cdb

_client is the low-volume, _combat is the high volume. _pattern is a pm_pattern with ~25-30 regexes and a small script on every pattern (0-10 lines), _unparse_finder is a pm_null uses add_to_route to copy unmatched patterns to a new route (0 hits lately) and does some light data enrichment via Exec. _testfile is the rotated file output, _es and _cdb are the HTTP outputs.

Thanks in advance.

Hello, I have question.

My variable $EventTime contains DateTime in only this format "2017-12-30 01:30:00"
How me with use NxLog, convert to UNIX TIME format or convert to this format  Dec 30 01:30:00.

Thank