I am looking into the possibility of being able to be notified if nxlog cannot output to redis. Has anyone managed to setup anything that writes to redis and if it cannot it sends an email.

Hi,

I encountered the following error suddenly in Nxlog:

ERROR ### ASSERTION FAILED at line 74 in om_http.c/om_http_timeout_event(): "modconf->timeout_event == NULL"

I have an INPUT that reads from multiple files and then send to an output, on_http, which makes a request to an ElasticSearch instance.

Do you have more details on what can cause this error?

This is so strange, I was looking at the previous questions regarding the type of failure and we're seeing it too, but not related to IIS logs but ftp logs. Here I'll include a sample of the logs and part of the config.

2018-07-25 12:42:02 ERROR if-else failed at line 449, character 236 in C:\Program Files (x86)\nxlog\conf\nxlog.conf. statement execution has been aborted; procedure 'parse_csv' failed at line 449, character 90 in C:\Program Files (x86)\nxlog\conf\nxlog.conf. statement execution has been aborted; Not enough fields in CSV input, expected 12, got 1 in input 'java.lang.NullPointerException' 2018-07-25 12:42:02 ERROR if-else failed at line 449, character 236 in C:\Program Files (x86)\nxlog\conf\nxlog.conf. statement execution has been aborted; procedure 'parse_csv' failed at line 449, character 90 in C:\Program Files (x86)\nxlog\conf\nxlog.conf. statement execution has been aborted; Not enough fields in CSV input, expected 12, got 8 in input '20180725 12:42:01 ERROR Exception Caught in channel 1189859467: ' 2018-07-25 12:42:02 ERROR if-else failed at line 449, character 236 in C:\Program Files (x86)\nxlog\conf\nxlog.conf. statement execution has been aborted; procedure 'parse_csv' failed at line 449, character 90 in C:\Program Files (x86)\nxlog\conf\nxlog.conf. statement execution has been aborted; Not enough fields in CSV input, expected 12, got 5 in input 'java.net.ConnectException: Connection refused: connect: /xx.xx.xx.xx:22'

Windows FTP events log:

<Input FTP_Logs> Module im_file File "D:\GAMFT\logs\goanywhere_*" SavePos TRUE

Exec if $raw_event =~/^#|(\s+at\s+)/ drop();\
   else\
   {\
        w3c_ftp->parse_csv();\
              $EventTime = parsedate($date + " " + $time);\
              $SourceName = "WINFTP";\
        $raw_event = to_json();\
   }

the config looks right too me. Suggestions?

Hello,

I currently am sending ms eventlogs from a server to our syslog server, and this is working fine. So I wanted to add to the nxlog conf file to allow it to collect the IIS W3C logs from this server as well and send them to the syslog server. A lot of the support examples out there added scripting to parse the w3c logs to json format. I do not want to do this. Just need to the raw logs to go to our syslog server. Below is my conf file. I'd appreciate pointers and/or the pointing out of errors.

define ROOT C:\Program Files (x86)\nxlog

Moduledir %ROOT%\modules CacheDir %ROOT%\data Pidfile %ROOT%\data\nxlog.pid SpoolDir %ROOT%\data LogFile %ROOT%\data\nxlog.log

<Extension syslog> Module xm_syslog </Extension> <Extension w3c> Module xm_csv Fields $date, $time, $s-sitename, $s-computername, $s_ip, $cs_method, $cs_uri_stem, $cs_uri_query, $s_port, $cs_username, $c_ip, $cs-version, $cs_User_Agent, $cs(Cookie), $cs_Referer, $cs-host, $sc_status, $sc_substatus, $sc_win32_status, $sc-bytes, $cs-bytes, $time_taken FieldTypes string, string, string, string, string, string, string, string, integer, string, string, string, string, string, string, string, integer, integer, integer, integer, integer, integer Delimiter ' ' </Extension>

<Input EventLogs> Module im_msvistalog </Input> <Input IIS> Module im_file File "C:\inetpub\logs\LogFiles\W3SVC1\u_ex*" SavePos TRUE UndefValue - </Input>

<Output Syslog> Module om_udp Host 172.16.x.x Port 514 Exec to_syslog_snare(); </Output>

<Route 1> Path EventLogs, IIS => Syslog </Route>

When attempting to input IIS logs I see the following error: 'ERROR invalid keyword: w3c->parse_csv()'. Has anyone come across this before and knows how I can parse IIS log into a JSON format to input to redis my config

<Extension _exec> Module xm_exec </Extension>

<Extension _json> Module xm_json </Extension>

<Extension _csv> Module xm_csv Fields $date, $time, $s_ip, $cs_method, $cs_uri_stem, $cs_uri_query, $s_port, $cs_username, $c_ip, $cs(User_Agent), $cs(Referer), $sc_status, $sc_substatus, $sc_win32_status, $time_taken FieldTypes string, string, string, string, string, string, integer, string, string, string, string, string, string, string, integer </Extension>

<Extension w3c> Module xm_csv #Fields: date time s-ip cs-method cs-uri-stem cs-uri-query s-port cs-username c-ip cs(User-Agent) cs(Referer) sc-status sc-substatus sc-win32-status time-taken Fields $date, $time, $s_ip, $cs_method, $cs_uri_stem, $cs_uri_query, $s_port, $cs_username, $c_ip, $cs(User_Agent), $cs(Referer), $sc_status, $sc_substatus, $sc_win32_status, $time_taken FieldTypes string, string, string, string, string, string, integer, string, string, string, string, string, string, string, integer Delimiter ' ' QuoteChar ''' </Extension>

<Extension w3cinput> Module xm_w3c Delimiter ' ' </Extension>

<Input iis_logs>
Module im_file File "C:\inetpub\logs\W3SVC2\u_ex*.log" ReadFromLast True Recursive True SavePos True Exec if $raw_event =~ /^#/ drop();
else
{
w3c->parse_csv();
$EventTime = parsedate($date + " " + $time);
$SourceName = "IIS";
to_json();
} </Input>

<output out_iis> Module om_redis Host # Port # Key "iis" </Output>

<Route 1>
Path iis_logs => out_iis </Route>

The following config is working for me to send application event logs to server via tcp, but it is sending duplicate messages for every Event Viewer event/entry.

config below as follows.

User nxlog Group nxlog

LogFile /var/log/nxlog/nxlog.log LogLevel INFO

<Extension exec> Module xm_exec </Extension>

<Extension syslog> Module xm_syslog </Extension>

<Input in> Module im_tcp Port 1514 Host 0.0.0.0 Exec parse_syslog_bsd(); </Input>

<Output out> Module om_file File "/application/logs/" + $Hostname + "" + "app.log" <Exec> if out->file_size() > 1G { set_var('newfile', file_name() + strftime(now(), '%Y%m%d%H%M%S')); rotate_to(get_var('newfile')); exec_async('/bin/bzip2', get_var('newfile')); } </Exec> </Output>

<Route 1> Path in => out </Route>

Hi

I'm using nxlog version 2.10.2102.

I have created the following Input:

<Input Test> Module im_file File "D:\TEST\Data\posWS\UploadLogs\61\\." SavePos TRUE <Exec> if $raw_event =~ /[^,]+/g;
{
$datetime = $1;
$log_level = $2;
$version = $3;
$company_id = $4;
$device_code = $5;
$device_manufacturer = $6;
$device_board = $7;
$sdk_version = $8;
$process_id = $9;
$thread_id = $10;
$class_name = $11;
$method_name = $12;
$file_name = $13;
$line = $14;
$description = $15;
$stack_trace = $16;
$EventTime = parsedate($datetime);
$SourceName = "POS_Device";
} delete($datetime); </Exec> </Input>

When I start the nxlog service, I receive the following error:

ERROR Couldn't parse Exec block at C:\Program Files (x86)\nxlog\conf\nxlog.conf:73; couldn't parse statement at line 74, character 29 in C:\Program Files (x86)\nxlog\conf\nxlog.conf; syntax error, unexpected ;, expecting (

I have tested the regex on https://regex101.com/ and it is working fine (flavour = pcre).

Sample data would be:

2018-07-19 10:26:07.830,ASSERT,3.0.2420InternalTest,145,352099091773370,samsung,universal7870,24,8521,1,CrashHandler,uncaughtException,CrashHandler.java,32,"UncaughtException","java.lang.NullPointerException: Attempt to invoke virtual method 'boolean bll.core.Product.isDiscountProduct()' on a null object reference;at bll.core.Sale$SaleItem.isDiscountItem(Sale.java:5514);at bll.core.Sale.containsSpecial(Sale.java:7685);at activity.SalesControlDisplaySale.setItemsPayments(SalesControlDisplaySale.java:454);at activity.SalesControlDisplaySale.displaySale(SalesControlDisplaySale.java:316);at activity.SalesControlDisplaySale.access$400(SalesControlDisplaySale.java:96);at activity.SalesControlDisplaySale$1.onItemSelected(SalesControlDisplaySale.java:213);at android.widget.AdapterView.fireOnSelected(AdapterView.java:1319);at android.widget.AdapterView.dispatchOnItemSelected(AdapterView.java:1308);at android.widget.AdapterView.-wrap1(AdapterView.java);at android.widget.AdapterView$SelectionNotifier.run(AdapterView.java:1275);at android.os.Handler.handleCallback(Handler.java:751);at android.os.Handler.dispatchMessage(Handler.java:95);at android.os.Looper.loop(Looper.java:154);at android.app.ActivityThread.main(ActivityThread.java:6776);at java.lang.reflect.Method.invoke(Native Method);at com.android.internal.os.ZygoteInit$MethodAndArgsCaller.run(ZygoteInit.java:1496);at com.android.internal.os.ZygoteInit.main(ZygoteInit.java:1386);"

If I do not use the "/g" (Global) at the end, it only returns the first match.

Any help would be greatly appropriated.

Many thanks

Stephen

Hello, I have a strange situation: I have a folder to index, which has some files, each file has a single line, a JSON encoded string. If I turn the DEBUG level, I can see that the files match the pattern for files, im_file_add_file function is called, but no event reaches my INPUT. Module im_file SavePos FALSE ReadFromLast FALSE PollInterval 0.25 #ActiveFiles 1000 #CloseWhenIdle TRUE File "/var/www/project/current/var/api_logs/default/2018-07-25/qm_sync/*" log_info(file_name()); $file = file_name(); #if $file =~ /^\/var\/www\/project\/current\/var\/api_logs\/default\/2018-07-25\/qm_sync\/[^\/]*\/([^\/]*)\/.*/ #{ $user = $1; file_write("/var/log/nxlog/nxlog.log", $raw_event); if $raw_event =~ /^(.*"requestBody":\{[^\[]*"[^"]*":)\[([^\]]*)\](.*)$/ { $raw_event = $1 + '"[' + replace($2, '"', '\"') + ']"' + $3; } file_write("/var/log/nxlog/nxlog.log", $raw_event); parse_json(); $timestamp = $requestTimestamp; $eventTime = strftime($EventReceivedTime, '%Y-%m-%d %H:%M:%S'); $host = hostname_fqdn(); rename_field('timestamp', '@timestamp'); delete($EventReceivedTime); delete($SourceModuleName); delete($SourceModuleType); to_json(); #} else { # file_write("/var/log/nxlog/nxlog_parse_failed.log", "File: " + $file + "; Message: " + $raw_event + "\n"); # drop(); #} $indexType = 'nxlog-api-external'; If I enter a file with VIM, and just write and quit, without changing anything, the file gets indexed. Any idea why this is happening? I am using the same instance of NxLog CE to index other files and it works fine with them

Hello,

I'm trying to use the log rotation functionality of nxlog but when nxlog tries to rotate is get the follow error message:

Error failed to rename file from C:\Users\me\Desktop\pfirewall.log to C:\Users\me\Desktop\pfirewall.log.1: The process cannot access the file because it is being used by another process.

Is there anyway to get around this?

Here is my nxlog.conf

define ROOT C:\Program Files (x86)\nxlog

Moduledir %ROOT%\modules CacheDir %ROOT%\data Pidfile %ROOT%\data\nxlog.pid SpoolDir %ROOT%\data LogFile %ROOT%\data\nxlog.log

define FWLOG C:\Users\me\Desktop\pfirewall.log

<Extension _syslog> Module xm_syslog </Extension>

<Extension fileop> Module xm_fileop </Extension>

<Input FWLOG> Module im_file File '%FWLOG%' InputType LineBased SavePos TRUE </Input>

<Output OUT> Module om_file File '%FWLOG%'

<Schedule>
    Every   10 sec
    Exec    if (file_size('%FWLOG%') >= 1K)  \
            { \
                file_cycle('%FWLOG%',10); \
                OUT->reopen(); \
            }
</Schedule>

</Output>

<Route 1> Path FWLOG => OUT </Route>

Hello,

Is there a way to import .evt files with nxlog? Using im_file doesn't throw errors, but data doesnt seem to ship.

I also tried using:

module im_mseventlog file "file_path"

but that didn't work either.

Thanks for your help.

Hello,

Has anyone successfully configured the new event tracing for windows (ETW) input module in nxlog 4.0 to collect Windows DNSServer events? I configured it in nxlog, but the output file doesn't show most of the DNS queries being made. When I look at the nxlog output and compare it with a trace session in Event Viewer, Event Viewer shows all of the events but nxlog is missing almost all of them. There are a few entries in the nxlog file, but not many. I can't seem to reproduce the scenario that causes them to be included in the nxlog output file.

Info on setup: Server 2016 datacenter, v1607 nxlog 4.0.3735-x64

Related nxlog config:

<Input winetw> Module im_etw Provider Microsoft-Windows-DNSServer </Input> <Output file> Module om_file File 'C:\Windows\Logs\nxlog\test.txt' </Output> <Route messages_to_file> Path winetw => file </Route>

Hello,

I'm trying to avoid having duplicate logs send to my OSSIM server. I tried using the pm_norepeat module but to no avail. Here the line I added in nxlog.conf file :

<Processor sans_doublons> Module pm_norepeat </Processor>

<Route route_windows_logs> Path in_windows_events => sans_doublons => out_alienvault_csv </Route>

I also tried adding "CheckFields raw_event" in the processor, but I still get duplicate logs. Does anyone know what could be the problem ?

Thanks

I'm using nxlog-ce-2.10.5 on AIX 6.5. Configure and make with default options. After building successfully, I tried to start nxlog with im_file and om_file, like the following: ``` Module im_file File "/home/logtest/*.log" Module om_file File "/home/logtest/1.out" Path messages => tcp ``` But nxlog shows some error like this: ``` 2018-07-16 08:45:16 ERROR Failed to load module from /usr/local/libexec/nxlog/modules/input/im_file.so, rtld: 0712-001 Symbol nx_assertion_failed was referenced; from module /usr/local/libexec/nxlog/modules/input/im_file.so(), but a runtime definition; of the symbol was not found.;rtld: 0712-001 Symbol nx_module_add_logdata_input was referenced; from module /usr/local/libexec/nxlog/modules/input/im_file.so(), but a runtime definition; of the symbol was not found.;rtld: 0712-001 Symbol nx_log was referenced; from module /usr/local/libexec/nxlog/modules/input/im_file.so(), but a runtim;DSO load failed 2018-07-16 08:45:16 ERROR Failed to load module from /usr/local/libexec/nxlog/modules/output/om_file.so, rtld: 0712-001 Symbol nx_assertion_failed was referenced; from module /usr/local/libexec/nxlog/modules/output/om_file.so(), but a runtime definition; of the symbol was not found.;rtld: 0712-001 Symbol nx_log was referenced; from module /usr/local/libexec/nxlog/modules/output/om_file.so(), but a runtime definition; of the symbol was not found.;rtld: 0712-001 Symbol nx_pool_create_child was referenced; from module /usr/local/libexec/nxlog/modules/output/om_file.so(), but a runtime de;DSO load failed 2018-07-16 08:45:16 WARNING no functional input modules! 2018-07-16 08:45:16 ERROR module 'messages' is not declared at nxlog.conf:12 2018-07-16 08:45:16 ERROR module 'tcp' is not declared at nxlog.conf:12 2018-07-16 08:45:16 ERROR route messages_to_tcp is not functional without input modules, ignored at nxlog.conf:12 2018-07-16 08:45:16 WARNING no routes defined! 2018-07-16 08:45:16 INFO nxlog-ce-2.10.5 started ``` What did I miss?

I'm having this newbie problem with sending logs from Windows Server 2008 R2. I tried to search a solution from here and from the user guide but can't find a solution. I'm getting this error message to nxlog.log: WARNING Module filein1 has no input files to read My config is this: Module xm_multiline HeaderLine /^\d{4}\-\d{2}\-\d{2}\s\d{2}\:\d{2}\:\d{2}\,\d{3}\s\d+/ #escape lines with whitespaces Exec if $raw_event =~ /^\s*$/ drop(); Module im_file File "C:\Program Files (x86)\CA\Some Manager\Connector Server\jcs\logs\LDAP_DYN\jcs_conn" InputType multiline1 SavePos TRUE Exec $SourceName = 'SomeManager'; Module om_tcp Host 192.168.1.1 Port 1468 Exec to_syslog_ietf(); Path filein1 => out What am i missing here? NXLog version is 4.0.3735.

I am trying to use NXlog CE to forward Exchange 2016 message tracking logs via syslog. I am trying to use the example shown here https://nxlog.co/documentation/nxlog-user-guide#exchange_transport_logs When I create the conf file and attempt to run it the service starts but there are several errors logged in the nxlog.log file. My config file is as follows ``` Panic Soft #NoFreeOnExit TRUE define ROOT C:\Program Files (x86)\nxlog define CERTDIR %ROOT%\cert define CONFDIR %ROOT%\conf define LOGDIR %ROOT%\data define LOGFILE %LOGDIR%\nxlog.log LogFile %LOGFILE% Moduledir %ROOT%\modules CacheDir %ROOT%\data Pidfile %ROOT%\data\nxlog.pid SpoolDir %ROOT%\data Module xm_syslog define BASEDIR C:\Program Files\Microsoft\Exchange Server\V15 Module xm_csv Fields date-time, client-ip, client-hostname, server-ip, server-hostname, \ source-context, connector-id, source, event-id, \ internal-message-id, message-id, network-message-id, \ recipient-address, recipient-status, total-bytes, recipient-count, \ related-recipient-address, reference, message-subject, \ sender-address, return-path, message-info, directionality, \ tenant-id, original-client-ip, original-server-ip, custom-data, \ transport-traffic-type, log-id, schema-version Module im_file File '%BASEDIR%\TransportRoles\Logs\MessageTracking\MSGTRK*.LOG' if $raw_event =~ /^(\xEF\xBB\xBF)?(date-time,|#)/ drop(); else { csv_parser->parse_csv(); $EventTime = parsedate(${date-time}); } Module om_udp Host 10.1.1.1 Port 514 Exec to_syslog_snare(); Path messagetracking => out1 ``` The errors logged in the nxlog.log file are as follows. ``` 2018-07-12 18:06:10 ERROR Couldn't parse Exec block at C:\Program Files (x86)\nxlog\conf\nxlog.conf:39; couldn't parse statement at line 44, character 36 in C:\Program Files (x86)\nxlog\conf\nxlog.conf; invalid character: '$' (0x24) 2018-07-12 18:06:10 ERROR module 'messagetracking' has configuration errors, not adding to route '1' at C:\Program Files (x86)\nxlog\conf\nxlog.conf:59 2018-07-12 18:06:10 ERROR route 1 is not functional without input modules, ignored at C:\Program Files (x86)\nxlog\conf\nxlog.conf:59 2018-07-12 18:06:10 WARNING no routes defined! 2018-07-12 18:06:10 WARNING not starting unused module messagetracking 2018-07-12 18:06:10 WARNING not starting unused module out1 2018-07-12 18:06:10 INFO nxlog-ce-2.10.2102 started ``` I have yet to come across a working example of how to forward exchange logs and was hoping someone may be able to assist. I cant see what i am missing here.

Hello,

Where can I find Nxlog-ce source code for Windows?

Hi,

i have a very basic question.

i have to use nxlog as a logshipper on the (solaris) client side to logstash (elk).

Within this configuration there are some bigger logfiles with multilines to parse (java traces / xml ...).

Should i handle the multiline on the nxlog side or on the logstash receiver side?

thanks in advance

Andreas

Hi,

Does NXLog supports sending log files to Amazon S3 cloud storage?

Thanks,

Amishai

Hi,

I have configured nxlog with Graylog2, i am seeing below warning in nxlog.log, wat could be the issue 2018-07-03 11:34:03 WARNING string limit (1048576) exceeded while trying to append 2018-07-03 11:41:37 INFO inode changed for '/var/log/Store/Store.log' (96469298->96469281): reopening possibly rotated file 2018-07-03 12:03:22 INFO inode changed for '/var/log/Store/Store.log' (96469281->96469298): reopening possibly rotated file

Can someone please let me know if Community Edition support TSL v 1.2 as currently i'm using 1.0 and that does not seem to be working after one of the vendors blocked it due to vulnerability in v 1.0. Can you please let me know what options i have now.

Thanks, Zee