I'm trying to do something like this, but I'm getting a literal "$1" substituted instead of the value of the capture group.

if $data =~ s/(\[[^\]]+\])/"$1"/g log_info($1);

$data ends up containing a "$1" while the log_info statement correctly logs the value of captured group $1.

Is there a way to use s/// and capture groups in the substitution? I also tried \1 and \\1.

Hello,

My organization would like to upgrade our version of NXlog CE from 2.9.1347 to 2.10.21250. We have NXlog installed on hundreds of servers, so performing this process manually is not viable.

We have tried ad nauseum to script this process, but we keep running into the same issues.

1. It appears the NXlog can only be uninstalled by the user account that installed it initially. NXlog version 2.9.1347 does not show in Add/Remove Programs for users who did not install the application and there for when you attempt to uninstall it via and MSI call it tries to install it instead.

2. There does not seem to be a completely clean way to uninstall this product. Even running the uninstall from the same user account that installed the application, it leaves traces of the application which cause the entry to remain in Add/Remove Programs.

My question... We would like to deploy the latest version of NXlog CE using our existing software deployment utility. This requires us to create a batch script to handle the uninstall and reinstall. Is there a way to totally uninstall NXlog CE version 2.9.1347 via CLI quietly or is there a quiet uninstall utility that we can call prior to installing the new version?

Thank you.

I don't believe this is currently possible, but hopefully someone can correct me if it's already implemented.

1. Ability to reorder fields in the raw message that gets sent to output module. The way things currently work any fields that are generated during the processing are added on to the end of the message when it gets forwarded to the output destination. The challenge for me is that I generate both a timestamp and a hostname field which then appear at the end of the message. Both of those fields are important for processing during the ingestion of the data on the remote side. Splunk for example by default only reads a certain limited number of characters into each message in order to find a timestamp and host field. I'd love an ability to reorder (or just move to front of the message) the fields that are generated.

2. JSON templating. For use with another pipeline, we have a requirement for a very specific JSON structure that must wrap each message. It's several levels nested and certain fields have to be present in the right place and in the right order for the event message to be accepted/processed. I tried faking it with json flatten and unflatten functions, but they aren't precise enough. Is there a way to define a specific template that should wrap all the messages before being sent to the output?

parsedate is returning undefined for the string directly from the documentation: "2017-Mar-23 06:38:30.143"

If I change the "Mar" to "03", making "2017-03-23 06:38:30.143", it works, returning "2017-02-23 06:38:20"

Any insights as to why the sample string from the documentation isn't working appreciated!

Hi everybody,

I did upgrade my NXLog client from version 3.2.202 to version 4.2.4216, but windows service can not run. But old version worked normally. It was in Microsoft Windows 10 Enterprise 2016 LTSB. And windows application log contains this event:

Faulting application name: nxlog.exe, version: 4.2.4216.0, time stamp: 0x00000000
Faulting module name: libnx-0.dll, version: 0.0.0.0, time stamp: 0x00000000
Exception code: 0xc0000005
Fault offset: 0x000000000001196e
Faulting process id: 0x1368
Faulting application start time: 0x01d4cdb54eb121da
Faulting application path: C:\nxlog\nxlog.exe
Faulting module path: C:\nxlog\libnx-0.dll
Report Id: 4b6f1d21-d02e-4502-9c20-f6e0cc5f9637
Faulting package full name: 
Faulting package-relative application ID: 

Any ideas? Thanks a lot!

i want to send my atp log to syslog server with the help of nxlog . i am storing atplog to atp.log file and the data is in json format.

I am using below configuration but i am not getting anything on my syslog server which is AV.

configuration:
# Configuration for converting and sending Windows logs
# to AlienVault USM Anywhere.
#

define ROOT C:\Program Files (x86)\nxlog
define OUTPUT_DESTINATION_ADDRESS 10.0.2.4
define OUTPUT_DESTINATION_PORT 514

Moduledir %ROOT%\modules
CacheDir %ROOT%\data
Pidfile %ROOT%\data\nxlog.pid
SpoolDir %ROOT%\data
LogFile %ROOT%\data\nxlog.log


<Extension json>
Module xm_json
</Extension>

<Extension syslog>
Module xm_syslog
</Extension>

<Input internal>
Module im_internal
</Input>
<input atplogfile>
Module im_atplogfile
</input>

<Input ATPLogs_IN>
Module im_atplogfile
File "C:\\temp\\log\\atplogs-*.log"
Exec $Message = to_json(); to_syslog_bsd();
</Input>

<Output ATPLogs_OUT>
Module om_atplogfile
File "/var/log/nxlog/syslog.log"
Exec to_syslog_bsd();
</Output>


########################################
# Routes #
########################################
<Route 1>
Path ATPLogs_IN => ATPLogs_OUT
</Route>

<Input eventlog>
Module im_msvistalog
Query <QueryList>\
<Query Id="0">\
<Select Path="Application">*</Select>\
<Select Path="System">*</Select>\
<Select Path="Security">*</Select>\
</Query>\
</QueryList>
Exec if ($EventID == 5156) OR ($EventID == 5158) drop();
</Input>


<Output out>

Module om_udp
Host %OUTPUT_DESTINATION_ADDRESS%
Port %OUTPUT_DESTINATION_PORT%
Exec $EventTime = integer($EventTime) / 1000000;
Exec $Message = to_json(); to_syslog_bsd();

</Output>

<Route 1>
Path eventlog, internal => out
</Route>

Hello, I have recently been trying up a syslog-ng server for various devices and have tried a couple of things for sending Windows Events to the server. Finally decieded that NXLog will do what I need and I have gotten sent some events over without much configuration, but when trying filter within the .conf file, it always fails. I can't really find much good information as to why it might be failing, as it seems that it should be correct.(to me anyway) # Windows Event Log, Module im_msvistalog Exec if $EventID == 4734 or $EventID == 4624 drop(); Exec $Message = to_json(); I have narrowed it down to this block, since the log says nxlog failed to start: without matching section at C:\Program Files (x86)\nxlog\conf\nxlog.conf:43 Which is where this block ends? I can't really make sense of this, so if anyone has some guidance please tell me.

Getting error: Opsec error. rc=-1 err=-96 Connection error step 4 on the user guide page 706

anyone run into issues pulling certs or suggestions? I can telnet using the port no issues, but not able to pull certs. steps prior were all completed.

Hi, I have the following windows log message which i am trying to parse and remove the \r,\t and \n from the log. with i am using i am unable to parse it and it still shows me the log content as is?

input: ACBDEFG 12/03/2015 09:05:13 AM\r\nLogName=Security\r\nSourceName=Microsoft Windows security auditing.\r\nEventCode=4672\r\nEventType=0\r\nType=Information\r\nComputerName=VDEEXCP01.teckcominco.loc\r\nTaskCategory=Special Logon\r\nOpCode=Info\r\nRecordNumber=3259542776\r\nKeywords=Audit Success\r\nMessage=msg_somethinh.\r\r\n\r\r\nSubject:\r\r\n\tSecurity ID:\t\tABCDEFFHG-12345\r\r\n\tAccount Name:\t\ABEDCEDDDD$\r\r\n\tAccount Domain:\t\tABCXDDFFEEFFFF\r\r\n\tLogon ID:\t\t98665svdvdvdv\r\r\n\r\r\nPrivileges:\t\tSeSecurityPrivilege\r\r\n\t\t\tSeBackupPrivilege\r\r\n\t\t\tSeRestorePrivilege\r\r\n\t\t\tSeTakeOwnershipPrivilege\r\r\n\t\t\tSeDebugPrivilege\r\r\n\t\t\tSeSystemEnvironmentPrivilege\r\r\n\t\t\tSeLoadDriverPrivilege\r\r\n\t\t\tSeImpersonatePrivilege\r\r\n\t\t\tSeDelegateSessionUserImpersonatePrivilege

output: ACBDEFG 12/03/2018 10:15:13 AM\r\nLogName=Security\r\nSourceName=Microsoft Windows security auditing.\r\nEventCode=4672\r\nEventType=0\r\nType=Information\r\nComputerName=VDEEXCP01.teckcominco.loc\r\nTaskCategory=Special Logon\r\nOpCode=Info\r\nRecordNumber=3259542776\r\nKeywords=Audit Success\r\nMessage=msg_somethinh.\r\r\n\r\r\nSubject:\r\r\n\tSecurity ID:\t\tABCDEFFHG-12345\r\r\n\tAccount Name:\t\ABEDCEDDDD$\r\r\n\tAccount Domain:\t\tABCXDDFFEEFFFF\r\r\n\tLogon ID:\t\t98665svdvdvdv\r\r\n\r\r\nPrivileges:\t\tSeSecurityPrivilege\r\r\n\t\t\tSeBackupPrivilege\r\r\n\t\t\tSeRestorePrivilege\r\r\n\t\t\tSeTakeOwnershipPrivilege\r\r\n\t\t\tSeDebugPrivilege\r\r\n\t\t\tSeSystemEnvironmentPrivilege\r\r\n\t\t\tSeLoadDriverPrivilege\r\r\n\t\t\tSeImpersonatePrivilege\r\r\n\t\t\tSeDelegateSessionUserImpersonatePrivilege

I am using the community edition and trying out to parse using the following configuration of nxlog. can you please suggest any changes needed in the conf?

Global section

User nxlog Group nxlog LogFile /var/log/nxlog/nxlog.log LogLevel INFO #Extension section <Extension multi> Module xm_multiline HeaderLine /^================/ EndLine /^---------------/ </Extension>

Modules section

<Input in> Module im_tcp HOST 0.0.0.0 Port 1532
#InputType multi # Remove the boundary markers Exec if $raw_event =~ s/========[=]+//g {}

Make a single line

            Exec if $raw_event =~ s/[\r\n]/ /g {}
              #remove the end directive
Exec        if $raw_event =~ s/--------[-]+//g {}
Exec  if $raw_event =~ /^ *$/ drop();
Exec        $raw_event = replace($raw_event, "\t", " ");

</Input> <Output out> Module om_file File '/tmp/output' #Exec $raw_event = "--------------------------------------\n" + $raw_event; </Output>

Route section

<Route r> Path in => out </Route>

I'm trying to set up nxlog with om_ssl using a 3rd party cert. I can connect without issue using openssl s_client, but am getting a certificate error in nxlog.

local:~$ openssl s_client -connect logs.notmyrealdomain.com:514 CONNECTED(00000003) depth=2 C = GB, ST = Greater Manchester, L = Salford, O = COMODO CA Limited, CN = COMODO RSA Certification Authority verify return:1 depth=1 C = GB, ST = Greater Manchester, L = Salford, O = COMODO CA Limited, CN = COMODO RSA Domain Validation Secure Server CA verify return:1 depth=0 OU = Domain Control Validated, OU = PositiveSSL Wildcard, CN = *.notmyrealdomain.com verify return:1

Certificate chain 0 s:/OU=Domain Control Validated/OU=PositiveSSL Wildcard/CN=*.notmyrealdomain.com i:/C=GB/ST=Greater Manchester/L=Salford/O=COMODO CA Limited/CN=COMODO RSA Domain Validation Secure Server CA 1 s:/C=GB/ST=Greater Manchester/L=Salford/O=COMODO CA Limited/CN=COMODO RSA Domain Validation Secure Server CA i:/C=GB/ST=Greater Manchester/L=Salford/O=COMODO CA Limited/CN=COMODO RSA Certification Authority

If I don't specify a CAfile in nxlog.conf, I get one error: 2019-02-11 12:19:55 ERROR SSL certificate verification failed: unable to get local issuer certificate (err: 20) Module om_ssl Host %OUTPUT_DESTINATION_ADDRESS% Port %OUTPUT_DESTINATION_PORT%

CAFile %ROOT%\cert\ca.pem

If I do specify a CAfile with the same intermediate cert as is on the server, I get a different error: 2019-02-11 12:22:24 ERROR SSL certificate verification failed: unable to get issuer certificate (err: 2) Module om_ssl Host %OUTPUT_DESTINATION_ADDRESS% Port %OUTPUT_DESTINATION_PORT% CAFile %ROOT%\cert\ca.pem

"AllowUntrusted TRUE" does not help, but shouldn't be needed

Thanks in advance for any help!

hi all,

not sure if you can help me, but I recently installed the latest (beta) version of nxlog 4-99.4527 to test the multiple evtx files support. my config looks like this:

<Input eventlog>
    Module  im_msvistalog
    File    "C:\Users\test\Desktop\logs\*.evtx"
</Input>

<Output file_from_eventlog>
	Module om_file
	File "C:\logs\evtx_new.log"
	Exec to_json();
</Output>

while it perfectly outputs to the local file in JSON output :), i still get the following error in the nxlog.log

2019-02-01 15:33:01 ERROR failed to query msvistalog events from file (C:\Users\test\Desktop\logs\Microsoft-Windows-SettingSyncOperational.evtx),[error code: 1287]; Zur Bestimmung der Fehlerursache stehen nicht genügend Informationen zur Verfügung.

I'm sorry that the error message is in german but roughly translated it means "Not enough information is available to determine the cause of the error."

any ideas what could cause this error?

many thanks in advance, theresa

I am trying to build NxLog Comminity edition on Centos 6.10 w/ X64_86 target and found there are five dependencies packages - pr-util.x86_64 apr-util-devel.x86_64 pcre-devel.x86_64 openssl-devel.x86_64 perl-ExtUtils-Embed. Does anyone know where to download the source code for these 5 dependent packages and how to build under centos 6.10 environment?

Thanks.

I am trying to process the logs collected from MacOX kernel using the method described on page 251 of the guide, using /usr/bin/log. I pretty much copied the entire code block, including xm_multiline and xm_json modules. The problem I am running into is that parse_json throws an error for a very specific message and I don't understand what it doesn't like about it. It ONLY complains about this event, while successfully processing others. First the error: 2019-02-06 11:47:19 ERROR failed to parse json string, parse error: invalid object key (must be a string); pple.duetactivityscheduler", }; (right here) ------^; [{; "category" : "scoring",; "processImageUUID" : "C8A8C46F-C86F-3DCE-AA7F-5128FEC80F95",; "processUniqueID" : 104,; "threadID" : 322877,; "timestamp" : "2019-02-06 11:47:19.133462-0500",; "traceID" : 1264507192737796,; "messageType" : "Default",; "senderProgramCounter" : 143258,; "processID" : 104,; "machTimestamp" : 10284775415304436,; "timezoneName" : "",; "subsystem" : "com.apple.duetactivityscheduler",;}]

Seems like it only complains about the events with subsystem of "com.apple.duetactivityscheduler.

I was able to view one of those events and nothing looks out of the ordinary for me:

{ "category" : "lifecycle", "processImageUUID" : "C8A8C46F-C86F-3DCE-AA7F-5128FEC80F95", "processUniqueID" : 104, "threadID" : 324554, "timestamp" : "2019-02-06 11:54:38.540442-0500", "traceID" : 1258391159308292, "messageType" : "Default", "senderProgramCounter" : 49785, "processID" : 104, "machTimestamp" : 10285214841060059, "timezoneName" : "", "subsystem" : "com.apple.duetactivityscheduler", "eventMessage" : "CANCELED: 0:com.apple.timed.ntp.needed:B67ADB <private>!", "senderImageUUID" : "B94808C8-E8F6-349C-9CCF-2C2C3EF30A24", "processImagePath" : "/usr/libexec/dasd", "senderImagePath" : "/System/Library/DuetActivityScheduler/Scheduler/DuetActivitySchedulerDaemon.bundle/Contents/MacOS/DuetActivitySchedulerDaemon" }

Hi, I am monitoring few log file on a windows 2012 server using im_file Send them to an ELK cluster

When checking the indexed logs we noticed delays ( 30 sec or more) for some ( not all) The only clue I have found is a bunch of this type of error

2019-01-28 14:50:01 DEBUG future event, event thread sleeping 34361ms in cond_timedwait

Any idea what that means and how to troubleshoot it ??

Many thanks Steven

I setup nxlog on windows 2008 , windows 2008 R2 , windows 2012 and windows 2016.

windows 2008 and 2008 R2 nxlog have some issue with connection with syslog server while 2012 and 2016 works perfectly fine.

> nxlog log file

2019-01-31 22:06:31 ERROR om_tcp send failed; An existing connection was forcibly closed by the remote host. 2019-01-31 22:06:32 INFO connecting to <some loadbalancer IP> 2019-01-31 22:24:57 INFO reconnecting in 1 seconds 2019-01-31 22:24:58 INFO connecting to <some loadbalancer IP>:514 2019-01-31 22:41:51 INFO reconnecting in 1 seconds 2019-01-31 22:41:52 INFO connecting to <some loadbalancer IP>:514 2019-02-01 00:45:43 INFO reconnecting in 1 seconds 2019-02-01 00:45:44 INFO connecting to <some loadbalancer IP>:514 2019-02-01 01:00:56 INFO reconnecting in 1 seconds 2019-02-01 01:00:56 ERROR om_tcp send failed; An existing connection was forcibly closed by the remote host. 2019-02-01 01:00:56 INFO reconnecting in 2 seconds 2019-02-01 01:00:57 INFO connecting to <some loadbalancer IP>:514 2019-02-01 01:19:06 WARNING received a system shutdown request 2019-02-01 01:19:06 WARNING stopping nxlog service 2019-02-01 01:19:06 WARNING nxlog-ce received a termination request signal, exiting... 2019-02-01 01:19:42 INFO nxlog-ce-2.10.2150 started 2019-02-01 01:19:42 INFO connecting to <some loadbalancer IP>:514 2019-02-01 01:20:09 INFO reconnecting in 1 seconds 2019-02-01 01:20:09 ERROR om_tcp send failed; An existing connection was forcibly closed by the remote host. 2019-02-01 01:20:10 INFO connecting to <some loadbalancer IP>:514 2019-02-01 01:20:13 WARNING received a system shutdown request 2019-02-01 01:20:13 WARNING stopping nxlog service 2019-02-01 01:20:13 WARNING nxlog-ce received a termination request signal, exiting... 2019-02-01 01:20:47 INFO nxlog-ce-2.10.2150 started 2019-02-01 01:20:47 INFO connecting to <some loadbalancer IP>o:514 2019-02-01 02:03:05 INFO reconnecting in 1 seconds 2019-02-01 02:03:05 ERROR om_tcp send failed; An existing connection was forcibly closed by the remote host. 2019-02-01 02:03:06 INFO connecting to <some loadbalancer IP>:514

> Configuration file

define ROOT C:\Program Files (x86)\nxlog
define CERTDIR %ROOT%\cert
define CONFDIR %ROOT%\conf
define LOGDIR %ROOT%\data
define LOGFILE %LOGDIR%\nxlog.log

LogFile %LOGFILE%
Moduledir %ROOT%\modules
CacheDir %ROOT%\data
Pidfile %ROOT%\data\nxlog.pid
SpoolDir %ROOT%\data
LogLevel INFO

<Extension _syslog>
    Module xm_syslog
</Extension>

<Input eventlog>
  Module im_msvistalog
  <QueryXML>
  <QueryList>
     <Query Id='0'>
         <Select Path='Application'>*</Select>
         <Select Path='Security'>*</Select>
         <Select Path='System'>*</Select>
     </Query>
  </QueryList>
  </QueryXML>
</Input>

<Output tcp>
  Module om_tcp
  Host <Load Balancer IP>
  Port 514
  Exec to_syslog_snare();
</Output>

<Route 1>
  Path eventlog => tcp
</Route>

What could be the issue? Is there anything more to be added in 2008 and 2008R2?

So we have several systems/appliances that only send to one location. However, we have a need to send logs to more than one location the issue is that the logs are sent in LEEF format and one system uses LEEF and the other system uses CEF.

I know NXLog will do the multiple sending however, will it also convert the logs it is sending?

I am asking this because we were told the Snare Central Server could do it and found out that it cannot do it without the agents installed. Clearly you cannot install agents on an appliance so before we go to the trouble of trying to setup and build out an NXLog server we need to know if this type of thing is possible.

I want to send the window event log generated from the normal PC to the graylog.

My first plan was to install sysmon and send it to graylog, but I had difficulty with the transport part

So I got to know nxlog.

I need the Windows event log from sysmon, which is the Windows security log. Can I check this in nxlog?

Thank you for your guide.

And I want to know the difference between nxlog and sysmon log

I am attempting to use `xm_ifileop` to rotate some logfiles I am collecting with nxlog. I can see that rotation works as expected if I specify the file path but can I use the same logic to rotate all files in directory. Example: `/var/log/osquery/` on linux/mac and `C:\ProgramData\osquery\log` on windows has 3 files in it that start with `osdqueryd.` and I want to watch those and rotate them if they get over 3M. I have tried on windows and Mac to use a `*` in the file path to specify the directory: `define OSQLOGFILE C:\\ProgramData\\osquery\\log\\osqueryd.\*` but that doesn't rotate the log. If I specify each file by name then it works as expected but then I need 3 xm_fileop sections. Is there an easy way to tell nxlog to rotate all files matching a pattern? Here is my logic so far: ``` Module xm_fileop # Check the log file size every hour and rotate if larger than 3 MB Every 1 hour Exec if (file_exists('%OSQLOGFILE%') and (file_size('%OSQLOGFILE%') >= 1M)) file_cycle('%OSQLOGFILE%', 4); ```

Is it possible to manipulate the ID in SQL queries done by NXLog agent? The documentation states this in im_dbi module:

The module automatically appends a WHERE id > ? LIMIT 10 clause to the statement. The result set returned by the SELECT statement must contain an id column which is then stored and used for the next query.

I have a database that does not get new reqords very often so i want to reset that id for testing and development purposes in the receiving end.

Hi,

I have installed community version on Linux 18.04 AMD 64 and I encountered problem with stopping NX Log service. When I hit "sudo service nxlog stop", the command process but the service still persist. When I hit "sudo service nxlog status", the result is: ● nxlog.service - LSB: logging daemon Loaded: loaded (/etc/init.d/nxlog; generated) Active: failed (Result: exit-code) since Sun 2019-01-27 16:50:34 CET; 33s ago Docs: man:systemd-sysv-generator(8) Process: 8744 ExecStart=/etc/init.d/nxlog start (code=exited, status=1/FAILURE) Tasks: 7 (limit: 2321) CGroup: /system.slice/nxlog.service └─30012 /usr/bin/nxlog

led 27 16:50:34 logstash-VirtualBox systemd[1]: Starting LSB: logging daemon... led 27 16:50:34 logstash-VirtualBox nxlog[8744]: * Starting nxlog daemon... led 27 16:50:34 logstash-VirtualBox nxlog[8744]: 2019-01-27 16:50:34 ERROR Another instance is already running (pid 30012);Resource temporarily unavailable led 27 16:50:34 logstash-VirtualBox nxlog[8744]: Failed to start nxlog! led 27 16:50:34 logstash-VirtualBox nxlog[8744]: ...fail! led 27 16:50:34 logstash-VirtualBox systemd[1]: nxlog.service: Control process exited, code=exited status=1 led 27 16:50:34 logstash-VirtualBox systemd[1]: nxlog.service: Failed with result 'exit-code'. led 27 16:50:34 logstash-VirtualBox systemd[1]: Failed to start LSB: logging daemon.

Could you please advice me how I can stop it and run it again? Is the service working right when I do "sudo service nxlog status"?

Thank you for all your advice.

Kind regards,

Marek