Hello, 

I'm trying to convert a date in NXlog from 06/15/16 to 2016-06-15 because NXlog is not able to parse the date (DEBUG couldn't parse date: 06/14/16).

I created a regular expression ($Date =~ s/(\d+)\/(\d+)\/(\d+)/20$3-$2-$1/;) in my module to convert the date. See the module below

 Exec if $raw_event =~ /^[0-9][0-9],/                        \
        {                                                        \
            ParseDHCP->parse_csv();                                 \
            if $raw_event =~ /^00/ $IDdef = "The log was started.";    \
            if $raw_event =~ /^01/ $IDdef = "The log was stopped.";    \
            if $raw_event =~ /^02/ $IDdef = "The log was temporarily paused due to low disk space.";    \
            if $raw_event =~ /^10/ $IDdef = "A new IP address was leased to a client.";    \
            if $raw_event =~ /^11/ $IDdef = "A lease was renewed by a client.";    \
            if $raw_event =~ /^12/ $IDdef = "A lease was released by a client.";    \
            if $raw_event =~ /^13/ $IDdef = "An IP address was found to be in use on the network.";    \
            if $raw_event =~ /^14/ $IDdef = "A lease request could not be satisfied because the scope's address pool was exhausted.";    \
            if $raw_event =~ /^15/ $IDdef = "A lease was denied.";    \
            if $raw_event =~ /^16/ $IDdef = "A lease was deleted.";    \
            if $raw_event =~ /^17/ $IDdef = "A lease was expired and DNS records for an expired leases have not been deleted.";    \
            if $raw_event =~ /^18/ $IDdef = "A lease was expired and DNS records were deleted.";    \
            if $raw_event =~ /^20/ $IDdef = "A BOOTP address was leased to a client.";    \
            if $raw_event =~ /^21/ $IDdef = "A dynamic BOOTP address was leased to a client.";    \
            if $raw_event =~ /^22/ $IDdef = "A BOOTP request could not be satisfied because the scope's address pool for BOOTP was exhausted.";    \
            if $raw_event =~ /^23/ $IDdef = "A BOOTP IP address was deleted after checking to see it was not in use.";    \
            if $raw_event =~ /^24/ $IDdef = "IP address cleanup operation has began.";    \
            if $raw_event =~ /^25/ $IDdef = "IP address cleanup statistics.";    \
            if $raw_event =~ /^30/ $IDdef = "DNS update request to the named DNS server.";    \
            if $raw_event =~ /^31/ $IDdef = "DNS update failed.";    \
            if $raw_event =~ /^32/ $IDdef = "DNS update successful.";    \
            if $raw_event =~ /^33/ $IDdef = "Packet dropped due to NAP policy.";    \
            if $raw_event =~ /^34/ $IDdef = "DNS update request failed.as the DNS update request queue limit exceeded.";    \
            if $raw_event =~ /^35/ $IDdef = "DNS update request failed.";    \
            if $raw_event =~ /^36/ $IDdef = "Packet dropped because the server is in failover standby role or the hash of the client ID does not match.";    \
            if $raw_event =~ /^[5-9][0-9]/ $IDdef = "Codes above 50 are used for Rogue Server Detection information.";    \
            if $raw_event =~ /^.+,.+,.+,.+,.+,.+,.+,.+,0,/ $QResultDef = "NoQuarantine";    \
            if $raw_event =~ /^.+,.+,.+,.+,.+,.+,.+,.+,1,/ $QResultDef = "Quarantine";    \
            if $raw_event =~ /^.+,.+,.+,.+,.+,.+,.+,.+,2,/ $QResultDef = "Drop Packet";    \
            if $raw_event =~ /^.+,.+,.+,.+,.+,.+,.+,.+,3,/ $QResultDef = "Probation";    \
            if $raw_event =~ /^.+,.+,.+,.+,.+,.+,.+,.+,6,/ $QResultDef = "No Quarantine Information ProbationTime:Year-Month-Day Hour:Minute:Second:MilliSecond.";    \
            $host            =    hostname_fqdn();                \
            $Date =~ s/(\d+)\/(\d+)\/(\d+)/20$3-$2-$1/;   \
            $EventTime         =     parsedate($Date + " " + $Time);    \
            $SourceName     =     "DHCPEvents";                    \
            $Message         =     to_json();                        \
        }                                                        \
        else                                                    \
            drop();

However it returns 2016-06-15 17:37:29 INFO EventTime: 20$3-$2-$1

 

Hi,
I use multiple input files.
I wish to log periodically (every minute) in nxlog.log the filename of the current input file to control all process chain, and add a Exec log_info("Current InputFile  : " + $InputFileName1); in a Schedule Block in Output section.
But it seems that it doesn’t work in a Schedule block => error on nxlog.log => “…field not available in this context…”

My config :
<Input in>
 Module  im_file
 SavePos TRUE
 ReadFromLast FALSE
 ActiveFiles 20
 CloseWhenIdle TRUE
 File "/var/log/MUP10/sac/APMUZS4WBS04*.log"
 Exec $InputFileName1 = file_name();
</Input>

<Output logstash>
 Module om_tcp
 Port 6002
 Host 10.x.y.z
 Exec        create_stat("stat", "RATE", 60); add_stat("stat", 1);

    <Schedule>
        Every   60 sec
        Exec    log_info("Events send to logstash for the last minute: " + get_stat("stat"));
        Exec log_info("Current InputFile  : " + $InputFileName1);
    </Schedule>

</Output>

If I move my Exec log_info("Current InputFile  : " + $InputFileName1); line in a Schedule block in my input section => same error
The only way I find is to move the line in Input section without using a schedule block, like this :
<Input in>
 Module  im_file
 SavePos TRUE
 ReadFromLast FALSE
 ActiveFiles 20
 CloseWhenIdle TRUE
 File "/var/log/MUP10/sac/APMUZS4WBS04*.log"
 Exec $InputFileName1 = file_name();
Exec log_info("Current InputFile  : " + $InputFileName1);
</Input>

But it writes too many files in log….

Any idea ?

Thanks in advance

Hi All

Could you confirm please that File_remove with wildcards AND File's created date condition works ?

I'm trying file_remove commands below :
The two first works properly => no problem with file_remove, "simple" or with wildcards
But not the last… using "now() - 18000", to remove file older than 5h never works, and no error in nxlog.log...looks like the line doesn't exist...?

<Schedule>
  Every 1 min
  Exec    file_remove('/var/log/MUP10/sac/APMUZS4WBS03-2016061300.log');
  Exec    file_remove('/var/log/MUP10/sac/APMUZS4WBS04-201606130*', now());
  Exec    file_remove('/var/log/MUP10/sac/APMUZS4WBS*.log', (now() - 18000));
</Schedule>

Thks

In Nxlog documentation :
~file_remove(string file, datetime older);
 description Remove the file ’file’ if its creation time is older than the value specified in ’older’. It is possible to specify a  wildcard in filenames (but not in the path). If you use backslash as the directory separator with wildcards, make sure  to escape this (e.g. ’C:\\test\\*.log’). This procedure will reopen the LogFile if this is removed. An error is logged if the operation fails.

Hi,

I am using om_http module to send windows eventlogs to AWS API Gateway for further processing. I kept HTTPSAllowUntrusted to True. But I keep getting <cloudfront_hostname>:443 connection failure reconnecting in ## seconds. I can POST data to the URI using curl just fine. I believe it is related to SNI support, which was also limiting other tools like wrk, ab, siege https://github.com/wg/wrk/issues/149 . 

Is there any workaround or fix to support SNI?

Thanks,

Shri

My Setup:

• Graylog2 server to collect logs


• Ubuntu machine running Zimbra sending logs from various Zimbra logfiles in GELF format

My problem:

• Messaged received by Graylog are truncated.

Here is an actual message as it appeared on my Ubuntu server's "mailbox.log" file (please note that I have X'd out the email address):

2016-06-12 08:51:17,832 INFO  [ImapSSLServer-95] [name=XXXXXXX@XXX.org;ip=10.10.48.74;ua=iPod touch Mail/13C75;] imap - ID elapsed=0

Here is the log as received by Graylog:

2016-06-12 08:51:17,832 INFO [ImapSSLServer-95] [name=XXXXXXX

All message seem to be truncated after exactly the same number of characters. I cannot seem to figure this out and would love some help. Below I have pasted my nxlog.conf file:

## This is a sample configuration file. See the nxlog reference manual about the

## configuration options. It should be installed locally under

## /usr/share/doc/nxlog-ce/ and is also available online at

## http://nxlog.org/docs

########################################

# Global directives                    #

########################################

User nxlog

Group nxlog

LogFile /var/log/nxlog/nxlog.log

LogLevel INFO

########################################

# Modules                              #

########################################

<Extension gelf>

    Module      xm_gelf

</Extension>

<Extension syslog2>

    Module    xm_syslog

</Extension>

<Input mailbox.log>

        Module  im_file

        File    "/opt/zimbra/log/mailbox.log"

    InputType LineBased

    SavePos    TRUE

</Input>

<Input access_log>

    Module    im_file

    File    "/opt/zimbra/log/access_log*"

    SavePos True

</Input>

<Input audit.log>

    Module    im_file

    File    "/opt/zimbra/log/audit.log"

    SavePos    TRUE

</Input>

<Input clamd.log>

    Module    im_file

    File    "/opt/zimbra/log/clamd.log"

    SavePos    TRUE

</Input>

<Input freshclam.log>

    Module    im_file

    File    "/opt/zimbra/log/freshclam.log"

    SavePos    TRUE

</Input>

<Input mysql_error.log>

    Module    im_file

    File    "/opt/zimbra/log/mysql_error.log"

    SavePos TRUE

</Input>

<Input mail.log>

    Module    im_file

    File    "/var/log/mail.log"

    SavePos TRUE

</Input>

<Input zimbra.log>

    Module im_file

    File    "/var/log/zimbra.log"

    SavePos TRUE

</Input>

<Input syslog>

    Module im_file

    File    "/var/log/syslog"

    SavePos    TRUE

</Input>

<Input zimbra-stats.log>

    Module im_file

    File    "/var/log/zimbra-stats.log"

    SavePos TRUE

</Input>

<Output out>

    Module      om_udp

    Host        10.10.90.45

    Port        5407

    Exec    to_syslog_snare();

</Output>

<Output out2>

    Module    om_udp

    Host    10.10.90.45

    Port    5413

    OutputType    GELF

</Output>

########################################

# Routes                               #

########################################

<Route 1>

    Path        mailbox.log => out

</Route>

<Route 2>

    Path    access_log => out

</Route>

<Route 3>

    Path    audit.log => out

</Route>

<Route 4>

    Path    clamd.log => out

</Route>

<Route 5>

    Path    freshclam.log => out

</Route>

<Route 6>

    Path    mysql_error.log => out

</Route>

<Route 7

    Path    mail.log => out

</Route>

<Route 8>

    Path    zimbra.log => out

</Route>

<Route 9>

    Path    syslog => out2

</Route>

<Route 10>

    Path    zimbra-stats.log => out

</Route>

 

I've seen the blog posting for AIX support following the download source and compile route.

Is a binary distribution of AIX in the roadmap to anyone's knowledge?

Thanks,

 

Rob

Hello,

I would like to obtain information in alert.log multiple lines for an event but without success. Below, my configuration :

<Extension multiline>

        Module       xm_multiline

        HeaderLine   /^\w\w\w\s\w\w\w\s\d\d\s\d\d:\d\d:\d\d\s\d\d\d\d/

</Extension>

<Extension syslog>

        Module  xm_syslog

</Extension>

<Input aix_syslog>

        Module  im_file

        File "/var/log/syslog.out*"

        Exec        to_syslog_bsd();

</Input>

<Input oracle_log_DBA102XX>

        Module      im_file

        InputType   multiline

        File        "/oradata/DBA102XX/admin/trace/bdump/alert_DBA102XX.log"

        Exec        to_syslog_bsd();

        Exec        $SourceName = "DBA102XX";

</Input>

<Output fluent_aix>

        Module om_udp

        Host 172.31.12.104

        Port 5139

</Output>

<Output fluent_oracle>

        Module om_udp

        Host 172.31.12.104

        Port 5127

</Output>

########################################

# Routes                               #

########################################

<Route 1>

        Path    aix_syslog => fluent_aix

</Route>

<Route 2>

    Path        oracle_log_DBA102XX => fluent_oracle

</Route>

Result Fluentd side :

7 Jun 2016 23:21:21     oracle.user.notice      {"host":"xmgpoc01","ident":"Tue","message":"21:21 DFT 2016"}

7 Jun 2016 23:21:21     oracle.user.notice      {"host":"xmgpoc01","ident":"Thread","message":"1 advanced to log sequence 66 (LGWR switch)"}

7 Jun 2016 23:21:21     oracle.user.notice      {"host":"xmgpoc01","ident":"","message":"/oradata/DBA102XX/oraredo01/LOG3A_DBA102XX.ORA"}

7 Jun 2016 23:21:21     oracle.user.notice      {"host":"xmgpoc01","ident":"","message":"/oradata/DBA102XX/oraredo02/LOG3B_DBA102XX.ORA"}

This line "Exec        $SourceName = "DBA102XX";" doesn't works, i dont know why.

Thank you for your help.

 

 

Hi,

I am a newbie and trying to configure NXLog on a windows machine to forward logs to a syslog server. With the below configuration, all events are being forwarded to the server. we need only critical and error level events and only Security and System related events to be forwarded to the server. Below is my configuration:

<Extension _syslog>
    Module      xm_syslog
</Extension>

<Input in>
    Module      im_msvistalog
</Input>

<Output out>
    Module      om_udp
    Host        192.168.1.25
    Port        514
    Exec        to_syslog_snare();
</Output>

<Route 1>
    Path        in => out
</Route>

Could some one help me what changes I should make so that only Critical and Error levels should be forwarded and only Security and System events should be forwarded ?

 

I would like to use nxlog on the recent Ubuntu server edition and the current package does not work on it.  A new deb for Ubuntu 16.04 would be great or recent sources for the current 2.9 version.

How would I change the syslog event message on an output module so that every message gets an additional field?  I want to add another value called "Project X" after the message portion of all syslogs events/messages as they are forwarded to another server?  Would I use $raw_event as I show in my example? 

Current config:

<Output out>

   Module om_udp

   Host 192.168.1.10

   Port 514

</Output>

 

Would I do this?

<Output out>

   Module om_udp

   Host 192.168.1.10

   Port 514

   Exec $raw_event = $raw_event + "Project X";

</Output>

 

OR is it more like this...?

<Output Out>

   Module om_udp

   Host 192.168.1.10

   Port 514

   $Message = $Message + "Project X"

   Exec to_syslog_bsd();

</Output out>

 

 

 

I am sending data to Elasticsearch via port 9200 and I am also using security for Elasticsearch. Any access to the rest API requiries authentication.

Is there a way to configure the om_http output to provide headers to authenticate when sending data to Elasticsearch?

Thank you

What this error means that leads some output modules to stop sending logs without crashing?

 

ASSERTION FAILED at line 21 in writerfuncs.c/nx_module_output_fill_buffer(): "output->buflen == 0" 

HI all, I am trying to transfer Windows eventlog (Securty, Application and System) from a server A to a server B. It is almost working if I send all the events of server A in a flat file on server B.
But my goal is a little bit different : I need to have the serverA-Events loggued in the eventviewer of the server B.
Do yo know if it is possible to achieve this ?
Thanks in advance !

There is a tone of infrastructure and application monitoring tools out there ( uberagent, vmturbo etc., powershell scirpts) to collect proccess details as their main task.

Allthough some could be integrated with NXLog ( lets say by using im_exec, or xm_exec) they have limited  filtering and output  capabilities compared to NXLog.

If NXLog could implement one day one let's say im_perfmon module to collect performance metrics then it could be easily transformed to a top application to this category.

Has anyone succeed in sending .evt file content to graylog ?

Actually, I found that:

 - Using im_file module I can parse .evt file and send its content outside, but logs are bad formatted

 - Using im_vistalog module I can't parse .evt files only the Windows Event log, but logs are well formatted

Any advice someone ?

Maybe it is possible to send the ouput of im_file to im_vistalog ?

Thanks,
--
Mathieu

Every now and then I get reports of logs not reporting. I investigate and 99.9% of the time, it is due to a loss of connectivity to the log server due to an nxlog crash. Typically, it is due to a faulting module, per Windows Event Viewer.

OS - Windows Server 2012 R2 Datacenter

NXLOG Version - How do I check?

Event Viewer ::

Faulting application name: nxlog.exe, version: 0.0.0.0, time stamp: 0x54fedd1a
Faulting module name: libapr-1-0.dll, version: 0.0.0.0, time stamp: 0x54fedd1a
Exception code: 0xc0000005
Fault offset: 0x00015190
Faulting process id: 0x160
Faulting application start time: 0x01d1b804aaa52028
Faulting application path: D:\Program Files (x86)\nxlog\nxlog.exe
Faulting module path: D:\Program Files (x86)\nxlog\libapr-1-0.dll
Report Id: 79778f7a-2701-11e6-80c2-00155d590419
Faulting package full name: 
Faulting package-relative application ID: 

 

Is this a known issue? Are there ways to prevent this from happening?

Thank you!

I'm attempting to compile the latest nxlog on FreeBSD and it fails with the following error. Any thoughts as to a fix or workaround would be appreciated. My environment is as follows:

FreeBSD 10.1-STABLE

OpenSSL 1.0.2h  3 May 2016

[/usr/local/src/nxlog-ce-2.8.1248]#make
Making all in src
Making all in common
make  all-am
/bin/sh ../../libtool --tag=CC    --mode=compile cc -DHAVE_CONFIG_H -I. -I/usr/local/include/apr-1   -I/usr/local/include  -pipe -W -Wall -Wshadow -Wno-address -Wcast-qual -Wbad-function-cast -Wsign-compare -Wconversion -funsigned-char -Werror-implicit-function-declaration -Wswitch-default -fstrict-aliasing -Wpointer-arith -Wcast-align -Wwrite-strings -Wmissing-noreturn -Wmissing-format-attribute -Wformat -Wextra -Werror=format-security -rdynamic -ggdb3 -MT ssl.lo -MD -MP -MF .deps/ssl.Tpo -c -o ssl.lo ssl.c
libtool: compile:  cc -DHAVE_CONFIG_H -I. -I/usr/local/include/apr-1 -I/usr/local/include -pipe -W -Wall -Wshadow -Wno-address -Wcast-qual -Wbad-function-cast -Wsign-compare -Wconversion -funsigned-char -Werror-implicit-function-declaration -Wswitch-default -fstrict-aliasing -Wpointer-arith -Wcast-align -Wwrite-strings -Wmissing-noreturn -Wmissing-format-attribute -Wformat -Wextra -Werror=format-security -rdynamic -ggdb3 -MT ssl.lo -MD -MP -MF .deps/ssl.Tpo -c ssl.c -o ssl.o
cc: warning: argument unused during compilation: '-rdynamic'
ssl.c:46:27: warning: cast from function call of type 'apr_os_thread_t'
      (aka 'struct pthread *') to non-matching type 'unsigned long'
      [-Wbad-function-cast]
    ret = (unsigned long) apr_os_thread_current();
                          ^~~~~~~~~~~~~~~~~~~~~~~
ssl.c:343:52: warning: cast to 'void *' from smaller integer type 'int'
      [-Wint-to-void-pointer-cast]
    SSL_set_ex_data(ssl, nx_ssl_verify_result_idx, (void *) verify_result);
                                                   ^
ssl.c:372:12: error: implicit declaration of function 'SSLv3_method' is invalid
      in C99 [-Werror,-Wimplicit-function-declaration]
    meth = SSLv3_method();
           ^
ssl.c:372:10: warning: incompatible integer to pointer conversion assigning to
      'const SSL_METHOD *' (aka 'const struct ssl_method_st *') from 'int'
      [-Wint-conversion]
    meth = SSLv3_method();
         ^ ~~~~~~~~~~~~~~
3 warnings and 1 error generated.
*** Error code 1

Stop.
make[3]: stopped in /usr/local/src/nxlog-ce-2.8.1248/src/common
*** Error code 1

Stop.
make[2]: stopped in /usr/local/src/nxlog-ce-2.8.1248/src/common
*** Error code 1

Stop.
make[1]: stopped in /usr/local/src/nxlog-ce-2.8.1248/src
*** Error code 1

Stop.
make: stopped in /usr/local/src/nxlog-ce-2.8.1248

 

 

Hello Team,

I am sending MS SQL Server Error logs from NXLog to our TCP server for processing.

I need to select File encoding to Unicode, I tried but it is goving error if I set encoding to Unicode/unicode.

If I set encoding to utf-8/utf-16 it is adding dot(.) after every single character in a source log line.

Any suggestions/help would be really appriciated.

Thanks,

Mahesh

mahibabar@gmail.com

+91-9923622938

NXLog Redis output module  sends data to a Redis server with the RPUSH command allowing only to change the Redis key ( = database).

RPUSH inserts all the specified values at the tail of the list stored at key

I suggest that the Redis command has to be implemented as a parameter and to support at least the LPUSH.

LPUSH inserts all the specified values at the head of the list stored at key

There are clients that support only retreiving data from the head. The most usual case is to send data to Redis with LPUSH and then have Logstash poll record from Redis with LPOP.

 

Thank you.

I have a multiline log sample as follow: 2/22/2016 4:19:30 PM 09F8 EVENT The DNS server has finished the background loading of zones. All zones are now available for DNS updates and zone transfers, as allowed by their individual zone configuration. 2/22/2016 4:19:38 PM 082C EVENT The DNS server has started. 2/22/2016 4:19:38 PM 08E0 PACKET 0000000001564000 UDP Rcv 192.168.2.213 526c Q [0001 D NOERROR] SOA (5)_ldap(4)_tcp(4)snpl(3)com(2)np(0) UDP question info at 0000000001564000 Socket = 324 Remote addr 192.168.2.213, port 57611 Time Query=104249, Queued=0, Expire=0 Buf length = 0x0fa0 (4000) Msg length = 0x0028 (40) Message: XID 0x526c Flags 0x0100 QR 0 (QUESTION) OPCODE 0 (QUERY) AA 0 TC 0 RD 1 RA 0 Z 0 CD 0 AD 0 RCODE 0 (NOERROR) QCOUNT 1 ACOUNT 0 NSCOUNT 0 ARCOUNT 0 QUESTION SECTION: Offset = 0x000c, RR count = 0 Name "(5)_ldap(4)_tcp(4)snpl(3)com(2)np(0)" QTYPE SOA (6) QCLASS 1 ANSWER SECTION: empty AUTHORITY SECTION: empty ADDITIONAL SECTION: empty I wish to add the some string at end of this log say (ENDOFLOG) to the end of logfile. How can i achieve this?