Hi,

I am evaluating nxlog on Windows but I've run into a snag.

I have logs organized like this:

D:\ServerLogs\2015-07-16\*.log

Where the directory is rotated based on today's date. Now, if I use the recursive parameter set to true for the file watcher, including this

D:\ServerLogs\*.log

works. However, I don't want to include all files dues to size and/or relevance. I only want to include some of them. As far as I can gather, the recursive parameter only works when there's a wildcard on the filename.

What I need is to be able to include the following:

D:\ServerLogs\*\Log1.log

so it follows the directory rotation on that file alone. I can't figure out how to accomplish this with nxlog.

Please advise.

Thank you.

Hi,

I'm using nxlog community edition nxlog-ce-2.9.1347, I have few questions related om_http module.
We have a centralized log server(Log Insight) and 10 application servers. Log Insight server exposes a REST API to post the logs data.
I'm using NXLOG as a log forwarder to Log Insight Server from all my application servers. Please clarify the following questions.

POST URL : http://loginsight:9000/api/v1/messages/ingest/4C4C4544-0037-5910-805A-C4C04F585831

Request Body:

{"messages": [{

 "fields": [
 {"name": "Channel", "content": "Security"},
 {"name": "EventID", "content": "4688"},
 {"name": "EventRecordID", "content": "33311266"},
 {"name": "Keywords", "content": "Audit Success"},
 {"name": "Level", "content": "Information"},
 {"name": "OpCode","content": "Info"},
 {"name": "ProcessID", "content": "4"},
 {"name": "ProviderName", "content": "Microsoft-Windows-Security-Auditing"},
 {"name": "Task", "content": "Process Creation"},
 {"name": "ThreadID", "content": "64"}
 ],
 "text": "A new process has been created.",
 "timestamp": 1396622879241
 }
 ]
}

1. How do I format my log data into the request as mentioned above in NXLOG. Request should be formated into json with fields and the data accordingly.
   I'm able to parse IIS logs,eventlogs and logs from files into JSON but got stuck with calling REST API with request body. Please find my nxlog.conf below.

2. Is it a good idea to directly send the log data to log server via REST API? if not what are the disadvantages. 
3. Does om_http module support retrying logic\buffering just in case REST API is down or doesn't respond.
4. What is the best architecture for sending the logs to centralized server, I see lot of people online follow NXLOG=>LOG STASH=>ELASTICSEARCH or Some Centralized server(Log Insight in my case).
5. Should I use pm_buffer, as my log files will be rotated after certain memory limit is reached in case if the REST API is down or om_http module handles this automatically.
 

 Following configuration reads IIS, event logs and logs from files.

## This is a sample configuration file. See the nxlog reference manual about the
## configuration options. It should be installed locally and is also available
## online at http://nxlog.org/nxlog-docs/en/nxlog-reference-manual.html

## Please set the ROOT to the folder your nxlog was installed into,
## otherwise it will not start.

#define ROOT C:\Program Files\nxlog
define ROOT C:\Program Files (x86)\nxlog

Moduledir %ROOT%\modules
CacheDir %ROOT%\data
Pidfile %ROOT%\data\nxlog.pid
SpoolDir %ROOT%\data
LogFile %ROOT%\data\nxlog.log

<Extension json>
    Module      xm_json
</Extension>
  
<Extension w3c>
  # Map the fields from the IIS log file (you can open the IIS log file to see the header and know what fields to map)
    Module      xm_csv
    Fields   $date, $time, $s-ip, $cs-method, $cs-uri-stem, $cs-uri-query, $s-port, $cs-username, $c-ip, $cs(User-Agent), $sc-status, $sc-substatus, $sc-win32-status, $time-taken
    FieldTypes  string, string, string, string, string, string, integer, string, string, string, integer, integer, integer, integer
    Delimiter   ' '
</Extension>

<Extension multiline>
    Module        xm_multiline
    HeaderLine    /^\d{4}\-\d{2}\-\d{2} \d{2}\:\d{2}\:\d{2}.\d{3}/
</Extension>

<Extension charconv>
    Module    xm_charconv
    AutodetectCharsets utf-8, utf-16, utf-32, iso8859-2
</Extension>

<Input eventlog>
    Module      im_msvistalog
    ReadFromLast    True
    Module      im_msvistalog
        Query       <QueryList>\
                        <Query Id="0">\
                            <Select Path="Application">*</Select>\
                            <Select Path="System">*</Select>\
                            <Select Path="Security">*</Select>\
                        </Query>\
                    </QueryList>
</Input>
 
<Input iis>
    Module      im_file
    File 'C:\inetpub\logs\LogFiles\W3SVC1\u_ex*.log'
    ReadFromLast TRUE
    Exec        if $raw_event =~ /^#/ drop();                \
                else                                             \
                {                                                \
                    w3c->parse_csv();                            \
                    $EventTime = parsedate($date + " " + $time); \
                    to_json ();                                  \
                }
</Input>

<Input webconsole>
    Module im_file
    File 'C:\Stash\WebConsole.log' 
    InputType multiline
    SavePos TRUE
    # The call to convert_fields automatically converts the input to utf-8
    Exec        convert_fields("AUTO","utf-8"); \
            if $raw_event =~ /^(\d{4}\-\d{2}\-\d{2} \d{2}\:\d{2}\:\d{2}.\d{3})    \[(\S+)\]    \[(\S+)\]    \[(\S+)\]    \[(\S+)\]    \[(.*)\]    \[(.*)\]    (.*)/s \
                    { \
                        $time = $1; \
                        $hostname = $2; \
                        $activityId = $3; \
                        $userIddeviceId = $4; \
                        $threadId = $5; \
                        $level = $6; \
                        $logger = $7; \
                        $message = $8; \
                        to_json(); \
                    } \
                    else \
                    { \
                        drop(); \
                }
</Input>

<Output eventlog-out>
    Module      om_tcp
    Host        127.0.0.1
    Port        3515
    Exec        $EventReceivedTime = integer($EventReceivedTime) / 1000000; \
                to_json();
</Output>
 
<Output iis-out>
    Module      om_tcp
    Host        127.0.0.1
    Port        3516
</Output>

<Output general-out>
    Module      om_tcp
    Host        127.0.0.1
    Port        3517
</Output>

<Route 1>

    Path eventlog => eventlog-out
</Route>
 
<Route 2>
    Path iis => iis-out
</Route>

<Route 3>
    Path webconsole => general-out
</Route>

Sample logs(Webconsole.log):

2015-07-10 10:24:17.424    [20EX15736]    [00000000-0000-0000-0000-000000000000]    [0000000-0000000]    [00008]    [Info ]    [TestModule]    Testing log stash3    
2015-07-10 10:24:17.425    [20EX15736]    [00000000-0000-0000-0000-000000000000]    [0000000-0000000]    [00008]    [Info ]    [TestModule]    Testing log stash4    
2015-07-10 10:24:17.448    [20EX15736]    [00000000-0000-0000-0000-000000000000]    [0000000-0000000]    [00008]    [Error]    [TestModule]    *** EXCEPTION ***
System.DivideByZeroException: Attempted to divide by zero.
   at TCPPublisher.Program.Main(String[] args) in c:\Users\test\Documents\Visual Studio 2013\Projects\TCPDemo\TCPPublisher\Program.cs:line 26

Thanks in advance!
Mohan G

Hey,

nxlog.log dosent create.     the folder /var/log/nxlog/  is empty.  

My environmet is RedHat 6.

What can be the reason ? 

Thanks.

Hi,

i have follwing logentry:

I have the following sql_exec command outline in my config:

<Output out>
    Module      om_odbc
    ConnectionString    DSN=nxlog;
    <Exec>
      sql_exec("INSERT INTO eventlog (hostname) VALUES (?)", '$Hostname');
    </Exec>
</Output>

This is exactly how the 2 examples in the documentation have it set up, I cannot seem to get it to work though I get the following error in my error log:

procedure 'sql_exec()' does not exist or takes different arguments

Can anyone shed any light on why this would be happening or exactly what arguements it is looking for?  Thanks

hey,

I have a problem reading the Apache access log file.

After I tried to do tcpdump I noticed that the file not read  and not sent to the destination.

I gave full permissions to the file but the situation has not changed.

nxlog.conf structure :

<Input in1>
    Module im_file
    File "/data/srv/httpd/logs/access_log_web1"  
    SavePos TRUE
</Input>

<Output fileout1>
 Module  om_udp  

 Host  192.168.10.1    

 Port      630

</Output>

<Route 1>

Path        in1   =>    fileout1

</Route>

* **There is no nxlog log file exist in /var/log/nxlog/

Thanks

TL;DR summary: NXLog won't load my certificate - why?

I've set up an NXlog CE server to act as an encrypted entrypoint for a Graylog server. After creating a self-signed certificate and adding it to the configuration, I get an unexpected error in the log when NXlogs starts up.

2015-07-10 06:49:22 INFO reloading configuration and restarting modules
2015-07-10 06:49:22 ERROR SSL error, Failed to open certfile: "/certificates/client.pem";No such file or directory
2015-07-10 06:49:22 INFO configuration reloaded successfully.

I assumed it might be a file rights issue, but looking at the files, it seems like the rights are actually a little too loose for comfort:

root@e757dae37501:/# ll /certificates/
total 20
drwxr-xr-x  2 nxlog nxlog 4096 Jul 10 06:32 ./
drwxr-xr-x 49 root  root  4096 Jul 10 06:32 ../
-rw-r--r--  1 nxlog nxlog 1387 Jul  9 15:51 ca.pem
-rw-r--r--  1 nxlog nxlog  887 Jul  9 15:54 client.key
-rw-r--r--  1 nxlog nxlog 1001 Jul  9 15:55 client.pem

My NXlog configuration looks like this:

...
<Input inssl>
    Module im_ssl
    Host 0.0.0.0
    Port 1515
    InputType Binary
    CAFile "/certificates/ca.pem"
    CertFile "/certificates/client.pem"
    CertKeyFile "/certificates/client.key"
    KeyPass secret
    RequireCert FALSE
    AllowUntrusted TRUE
</Input>
...

I've tried looking at the source code for the SSL module, but it looks like the line that fails is not related to validation but simple file access. I have no idea how to proceed. Suggestions?

Trying to run nxlog-ce 2.8.1248 on AIX 7.1.

Compilation is succesful, nxlog binnary is created, but can not run. There are messages like:

"ERROR Failed to load module from /usr/local/libexec/nxlog/modules/extension/xm_syslog.so, Could not load module /usr/local/libexec/nxlog/modules/extension/xm_syslog.so.; System error: No such file or directory;DSO load file"

There are static libraries (.a) in directories but dynamic (.so) not.

Hi,

We have deployed nxlog 2.8.1248 for Windows on many computers (so far about 200) using Windows 2008 R2, 2012 and 2012 R2.

it appears we are facing multiple issues randomly:

- Service is crashing expectably and log an error 1067 and after a few attempt, it can restart successfully

- Service doesn't want stat and log error 1053 at each try (uninstall/reinstall of the client doesn't help)

- The service is running but seems to be hanging but we have the following messages on the nxlog log:

ERROR EvtUpdateBookmark failed: Unrecognized Win32 error code 6

ERROR EvtNext failed with error 14: Unrecognized Win32 error code 14

Any idea?

Thanks

PM

I am having this weird encoding problem. I am exporting utf8 characters into a local mysql table. when i export from nxlog into a file the characters seem ok. when i switch to dbi export its jiberish. i tried debugging the queries from mysql into a log to check if the characters arrive in the db intact and the characters seem fine there. collation in mysql is utf8. any ideas?

Hello,

I am having an issue with the Program name not population with anything on our syslog server.  The service name is just blank.  Below is my config file.  Am I missing something?

## Please set the ROOT to your nxlog installation directory
 
#define ROOT C:\Program Files\nxlog
define ROOT C:\Program Files (x86)\nxlog
 
Moduledir %ROOT%\modules
CacheDir  %ROOT%\data
Pidfile %ROOT%\data\nxlog.pid
SpoolDir  %ROOT%\data
LogFile %ROOT%\data\nxlog.log

 
<Extension syslog>
  Module xm_syslog  
</Extension>
 
# Monitor application log files
#<Input watchfile>
#  Module im_file
#  # File 'C:\\path\\to\\*.log'
#  Exec $Message = $raw_event;
# Exec if file_name() =~ /.*\\(.*)/ $SourceName = $1;
# SavePos TRUE  
#  Recursive TRUE
#</Input>
 
# Monitor Windows event logs
#<Input eventlog>
  # Uncomment for Windows Vista/2008 or later 
#  Module im_msvistalog
  
  # Uncomment for Windows 2000 or later
  # Module im_mseventlog
#</Input>
 
#<Processor eventlog_transformer>
#  Module pm_transformer
#  Exec $Hostname = hostname();
#  OutputFormat syslog_rfc5424  
#</Processor>
 
<Output syslogout_centreon>
  Module om_udp
  Host 10.10.103.112
  Port 514
</Output>

# Monitor CME FlatFile
<Input watchfile_test>
  Module im_file
  File 'C:\\logs\\test.txt'
  Exec $Message = $raw_event;
  Exec $SyslogSeverityValue = 6;
  Exec if $raw_event =~ /INFO/ drop(); 
  Exec if file_name() =~ /.*\\(.*)/ $SourceName = $1; 
  Exec if $raw_event =~ /WARNING/ $SyslogSeverityValue = 4; 
  Exec if $raw_event =~ /ERROR/ $SyslogSeverityValue = 3; 
  Exec if $raw_event =~ /CRITICAL/ $SyslogSeverityValue = 2;
  Exec if $raw_event =~ /ALERT/ $SyslogSeverityValue = 1; 
  SavePos TRUE  
  Recursive TRUE
  PollInterval 10
</Input>

<Processor filewatcher_transformer_test>
  Module pm_transformer
  
   #Uncomment to override the program name
   Exec $SourceName = 'test';
    
  Exec $Hostname = hostname();
  OutputFormat syslog_rfc5424
</Processor>

# Path to send Syslog message for test Flat File Generator
<Route cme_flat_file_generator>
  Path watchfile_test => filewatcher_transformer_test => syslogout_centreon
</Route>

I am using the following im_file configuration to try to collect Windows DHCP Server logs:

## Input module for Microsoft DHCP server audit logs
<Input dhcp>
    Module im_file
    File "C:\\Windows\\System32\\Dhcp\\DhcpSrvLog-*.log"
    SavePos TRUE
    PollInterval 180
    Exec to_syslog_bsd();
</Input>

I also tried this without escaping the backslashes and even with "/" characters instead. I also tried using a specific filename but nothing seems to work, since I get the "input file does not exist" error.

When I try the same config but with the location being at C:\Dhcp\DhcpSrvLog-*.log, everything works. nxlog service is being run as LocalSystem. Any hints on what I would need to do next to get the logs working from their native location?

Hi, We're evaluating nxlogn to forward event logs and IIS logs from same server as SYSLOG format to centralised log server. The event logs are reaching correctly, but not the IIS logs. Please see the configuration settings. Your help would be highly appreciated

============================================================

define ROOT C:\Program Files (x86)\nxlog

Moduledir %ROOT%\modules
CacheDir %ROOT%\data
Pidfile %ROOT%\data\nxlog.pid
SpoolDir %ROOT%\data
LogFile %ROOT%\data\nxlog.log

<Extension _syslog>
    Module      xm_syslog
</Extension>

<Input eventlog>
    Module      im_msvistalog
# For windows 2003 and earlier use the following:
#   Module      im_mseventlog
</Input>

<Extension w3c>
    Module    xm_csv
    Fields    $date, $time, $site, $dstip, $HTTPMethod, $URIStem, $URIQuery, $port, $username, $srcip, $UserAgent, $HTTPStatus, $SubStatus, $win32Status
    FieldTypes    string, string, string, string, string, string, string, string, string, string, string, string, string, string,
    Delimiter    ' '
</Extension>

<Input IIS_Log>
  Module    im_file
    File    "C:\WINDOWS\System32\LogFiles\W3SVC1\ex*"
    ReadFromLast TRUE
        #Drop info legend lines
        Exec    if $raw_event =~ /^#/ drop();\
        else\
        {\
            w3c->parse_csv();\
            $EventTime = parsedate($date + " " + $time);\
        }
</Input>

<Output out>
    Module      om_udp
    Host        X.X.X.X
    Port        514
    Exec        to_syslog_snare();
</Output>

<Route 1>
    Path        eventlog => out
</Route>

<Route 2>
    Path        IIS_Log => out
</Route

I'm currently using the TCP output of NXLog (v2.9.1347) to ship Windows Server 2008 R2 eventlogs to Logstash (v1.4.2) in JSON format; lately I found that NXLog crashes if Logstash has been unavailable for some time and then became available, although it ships a few logs before crashing.

This event is logged in the eventlog:

Faulting application name: nxlog.exe, version: 0.0.0.0, time stamp: 0x54fedd1a
Faulting module name: ntdll.dll, version: 6.1.7601.18247, time stamp: 0x521ea8e7
Exception code: 0xc0000005
Fault offset: 0x0005e8d1
Faulting process id: 0x4e4
Faulting application start time: 0x01d0a2b5080df49c
Faulting application path: C:\Program Files (x86)\nxlog\nxlog.exe
Faulting module path: C:\Windows\SysWOW64\ntdll.dll
Report Id: 7ebdb4d7-1036-11e5-909f-005056a30012

To reproduce the issue, just have NXLog ship logs to Logstash and then stop Logstash for about an hour then start it, NXLog crashes soon after.

Any idea what might be causing this?

I am trying out the enterprise edition, and could not find documentation for reading event logs directly from .evtx file only, can anyone help in a sample config.

Hello

I don't know way to compile my input module for Oracle - need some guidance for it. I guess autogen.sh (using autotools) is designated to generate makefiles etc and configure.

I made Makefile.am based on im_dbi version:

if HAVE_LIBOCI
im_oci_LTLIBRARIES    = im_oci.la
im_oci_la_SOURCES    = im_oci.c im_oci.h
im_oci_la_CFLAGS    = -rdynamic -D_XOPEN_SOURCE -std=c99
im_oci_la_LDFLAGS    = -module -no-undefined -avoid-version -ldl
im_oci_la_LIBADD    = $(LIBOCI) $(LIBNX)
im_ocidir        = $(NX_MODULEDIR)/input
endif

When I call configure or autogen.sh it gives error:

./configure: line 21597: syntax error: unexpected end of file

but last lines are:
 

21590: #echo ----------------------------------------------------------
21591: #echo Shared: ${BUILD_SHARED_LIBS}
21592: #echo ----------------------------------------------------------
21593: echo
21594: echo "${PACKAGE}-${NXLOG_VERSION_STRING} configured successfully"
21595: echo "type \`${MAKE-make}' and \`${MAKE-make} install'"
21596: echo
21597: echo
21598:

I don't see error here. I just placed my module in modules/input folder Should I generate some files with autotools?

Hi guys,

I need help about NXlog with graylog, on my 2008 server my nxlog send correctly log to my graylog server but i cant see lvl and facility off all logs :-(. like :

plz see my nxlog config :

 <Extension gelf>
Module xm_gelf
</Extension>
 
 <Extension syslog>
    Module    xm_syslog
</Extension>

<Input eventlog>
Module im_msvistalog
# this kinda works for me, put * to get everything
Query <QueryList>\
<Query Id="0">\
<Select Path='Application'>*[Application/Level=2][Application/Level=3][Application/Level=1]</Select>\
<Select Path='System'>*[system/Level=2][system/Level=3][system/Level=1]</Select>\
<Select Path='Security'>*</Select>\
<Select Path='Setup'>*</Select>\
</Query>\
</QueryList>
</Input>

<Output out>
    Module      om_udp
    Host        10.0.0.202
    Port           9000
</Output>

<Route 1>
    Path        eventlog => out
</Route>

btw i tried to use GELF out type but all messages in graylog was ���������������

hope u will help me

Regards,

Gael

I am using om_tcp for forwarding Windows logs to a SIEM system. What will be the expected behavior of nxlog if e.g. a firewall blocks the TCP connections from the nxlog agent to the SIEM? Is there any potential danger in nxlog buffering outgoing logs so that large amounts of memory or disk space would be consumed on the sending host while the connections get blocked?

I got the error below, after a while that i begin to run the nxlog service. I use mysql database, tried postgresql as well and have the same error message. Has anyone idea how I can fix it?

>>ERROR ### ASSERTION FAILED at line 205 in om_dbi.c/om_dbi_get_sql(): "len > j"