Hi all, 

I'm trying to troubleshoot an issue with getting the Checkpoint input module working and pulling syslogs from a Checkpoint management appliance. I'm following the configuration guide provided in the nxlog documentation, but I am running into a few issues. 

Specifically on the step where I attempt to retrieve the authentication key by using opsec_putkey. The correct output from that command should be "FW: Received new control security key from IP; Authentication with IP initialized successfully." However, my output is " FW: Received new control security key from IP; Failed to initialize authentication with IP."

When I go to test whether the log collection works because I do end up receiving the sslauthkeys.c and sslses.c files, and I have the opsec.p12 cert from a previous step, I get the following response - "Peer IP wants to exchange keys but I don't have a password."

Google hasn't been very fruitful so any advice would be appreciated. 

-M

Hello good people!

Is there any chance for Community Edition to get deb package for Debian 9 (Stretch)?

Many thanks in advance!

Hi All, 

I am on Cent OS 7 and installed nxlog to forward the /var/log/messages and /var/log/secure to my logging server. When I start its starting fine and in logs I am gettting weird log which I am not able to solve. 

ERROR

2017-10-19 13:32:16 ERROR om_tcp detected a connection error;End of file found

2017-10-19 13:32:17 INFO connecting to 10.12.86.87:12225

2017-10-19 13:32:17 INFO reconnecting in 1 seconds

2017-10-19 13:32:17 ERROR om_tcp detected a connection error;End of file found

Config: 

########################################

# Global directives                    #

########################################

User nxlog

Group nxlog

LogFile /var/log/nxlog/nxlog.log

LogLevel INFO

########################################

# Modules                              #

########################################

<Input messages>

    Module              im_file

    File                "/var/log/messages"

SavePos           TRUE

</Input>

<Input securelogs>

    Module              im_file

    File                "/var/log/secure"

SavePos TRUE

</Input>

<Output logserver>

    Module              om_tcp

    Host                10.12.86.87

    Port                12225

</Output>

<Route Log-TCP>

    Path        messages, securelogs => logserver

</Route>

Selinux and Firewall is turned off in my machine. And remote server is listenting on the port 12225 and that can be reachable from this machine without any issues. Can you help me with what I can do to solve this issue. 

Thanks in advance 

hi,

I am tesing nxlog on Ubuntu 16.04 LTS with the configuration

<Extension fileop>
        Module xm_fileop
</Extension>

<Input udpin>
        Module im_udp
        Host 127.0.0.1
        Port 1514
        Exec $HOSTIP = string(host_ip());
        Exec file_write("/tmp/debug.txt", "HostIP:" + $HOSTIP);
</Input>

<Output udpfile>
        Module om_file
        File "/var/log/udp.log"
</Output>

<Route udp>
    Path udpin => udpfile
</Route>

and  I generated log using linux command nc -u 127.0.0.1 1514 and can not get IP in /tmp/debug.txt.

1. I have reinstall the Ubuntu 16.04 LTS purely and install the deb file from nxlog.co， and it still does not get the IP address 

2. I install the nxlog from source code, and it still does not get the IP address

3. I test the same configuration on CentOS7, CentOS6, Ubuntu 14.04 LTS. It works correctly.

so， is there someting that does not compatible with the Ubuntu 16.04 LTS?

Hi

I'm trying to do a syslog udp to tcp converter using this method from the user's manual:

<Input in>
    Module im_tcp
    Host 0.0.0.0
    Port 2345
 </Input>

<Output out>
    Module      om_tcp
    Host        mysyslog.domain.local
    Port        514
</Output>

~~<Processor buffer>
 Module pm_buffer
 WarnLimit 800
 MaxSize 1000
 Type Mem
 Exec if buffer_size() >= 80k drop();
</Processor>

<Route 1>
    Path in => buffer => out
</Route>

This is working correctly but all the events recieved by mysyslog.domain.local server are coming from the same source (the IP of the NXLOG server). Is there a way to preserve the source IP?

Thanks in advance

Regards,

Olga

the following context is my Input configuration and i get a functong from the manual doc,  but the return value of hos_ip() is ip4addr, and I want to convert the return value to string, I have try the string() function, but it does not work at all. Is there any way to convert the return value to string so i can and a new filed of the Input configuration.

ip4addr host_ip();
description Return the first non-loopback IP address the hostname resolves to.
return type ip4addr

<Input 59dcb7a6dd48cb088969e300>
        Module im_file
        File '/usr/local/nginx_raw/logs/access.log'
        PollInterval 1
        SavePos True
        ReadFromLast True
        Recursive True
        RenameCheck False
        Exec $FileName = file_name(); # Send file name with each message
        Exec $HOSTIP = host_ip();
</Input>

Hello!

I noticed strange cases nxlog.exe is loading 60%-90% CPU.
Log file has nothings about errors.
I have the same config file for win7 and winXP, but winXP is OK in contrast to win7.
Also if I use om_udp module instead om_tcp in config file for win7 high load is desappear.  

Could anybody say me what is the problem, or how I can get more information about cause of problems.

Hello!

I have a question about IETFTimestampInGMT option.

In the documentation - section "Syslog (xm_syslog)" - I can read : "IETFTimestampInGMT This optional boolean directive can be used to format the timestamps produced by to_syslog_ietf() in GMT instead of local time. This defaults to FALSE so that local time is used by default with a timezone indicator"

So here's a part of my nxlog.conf :

Hello everyone,

As you can see from my NXlog config below, I am pulling the Linux auth logs in and passing them to our SIEM platform. The auth logs work fine, however as you can also see I am trying to pull in a CSV file that is located on the Linux box. Each individual event is surrounded by curly brackets e.g: {'event}, so I would like to just pull each event into a field called message and then parse these messages on our SIEM platform. I can't seem to get this to work whatsoever, the TCP connection initiates, but NXlog doesn't pickup the CSV file. Any ideas?

Cheers

G

########################################

# Global directives                    #

########################################

User nxlog

Group nxlog

LogFile /var/log/nxlog/nxlog.log

LogLevel INFO

########################################

# Modules                              #

########################################

<Extension _syslog>

Module  xm_syslog

</Extension>

<Input auth_logs>

        Module  im_file

        File    "/var/log/auth.log"

        SavePos TRUE

        ReadFromLast    TRUE

</Input>

<Output to_relay>

        Module  om_tcp

        Host    127.0.0.1

        Port    20009

        OutputType      LineBased

</Output>

########################################

# Routes                               #

########################################

<Route 1>

        Path    auth_logs => to_relay

</Route>

<Extension csv1>

        Module  xm_csv

        Fields  $Message

        Delimiter       '{'

</Extension>

<Input filein>

        Module  im_file

        File    "/etc/ingest/sucuri/sucuri.csv"

        Exec    csv1->parse_csv();

</Input>

<Output test>

        Module  om_tcp

        Host    127.0.0.1

        Port    20002

        OutputType      Binary

</Output>

<Route 2>

        Path    filein => test

</Route>

We have been running this product on our domain controller and within the last week or two, the nxlog.exe process consumes 100% of the CPU.   we stop the nxlog service and the CPU instantly drops to <5%.    start the service and it immediately rises to 100%.   We've uninstalled/installed without change.  Any advice on how to correct this is greatly appreciated. 

In case of DOS attack on a device, there would be a surge of logs in a very short time and all the events look simillar with change in one or two parameters source port/destination port/source ip/destination ip. In such case, can we filter such repeatitive logs in NXLOG agent? If yes, How to do that? I tried pm_norepeat but it didnt help. Any other alternate options?

What I'm trying to achive is to have a few Values globaly defined and the should be automatically added to all inputs.
Ie the same thing as Global Tags in Telegraf

Today I first use a define statement in the global part of NXLog.conf

Define Company Acme 

For each input I define i have to add a Exec line

Exec $Company = '%Company%';

I would like to be able to do this only once and have it automatically appended to all inputs. For a multi Company Scenario with a lot of logfiles It gets rather messy to maintain when you need to add anoter global value. It works well in Telegraf so I would like to be able to do something similar in Nxlog

Hi All, 

I am looking to use nxlog to transform a CSV formatted input from an SMB share into a json formatted line-by-line output for parsing by further handlers of our logging information. The CSV in question is an export of Windows Event Logs from a domain controller. An example of the CSV I am trying to parse is:

"Index","TimeGenerated","InstanceId","EntryType","UserName","MachineName","Category","Field1","Field2","Field3","Field4","Field5","Field6","Field7","Field8","Field9","Field10","Field11","Field12","Field13","Field14","Field15","Field16","Field17"
"3297643","20170914-00:00:01","4768","SuccessAudit",,"DC1.Office.ExampleDomain.com","(11111)","Test","Office.ExampleDomain.com","S-1-5-21-910167743-1234567890-1234567890-1388","krbtgt","S-1-5-21-910167743-1234567890-1234567890-502","0x40800000","0x0","0x17","2","10.0.0.1","55393","","","",,,
"3297644","20170914-00:00:31","4768","SuccessAudit",,"DC1.Office.ExampleDomain.com","(11111)","Test","Office.ExampleDomain.com","S-1-5-21-910167743-1234567890-1234567890-1388","krbtgt","S-1-5-21-910167743-1234567890-1234567890-502","0x40800000","0x0","0x17","2","10.0.0.1","45086","","","",,,
"3297645","20170914-00:01:01","4768","SuccessAudit",,"DC1.Office.ExampleDomain.com","(11111)","Test","Office.ExampleDomain.com","S-1-5-21-910167743-1234567890-1234567890-1388","krbtgt","S-1-5-21-910167743-1234567890-1234567890-502","0x40800000","0x0","0x17","2","10.0.0.1","35822","","","",,,
"3297646","20170914-00:01:31","4768","SuccessAudit",,"DC1.Office.ExampleDomain.com","(11111)","Test","Office.ExampleDomain.com","S-1-5-21-910167743-1234567890-1234567890-1388","krbtgt","S-1-5-21-910167743-1234567890-1234567890-502","0x40800000","0x0","0x17","2","10.0.0.1","44883","","","",,,
"3297647","20170914-00:02:01","4768","SuccessAudit",,"DC1.Office.ExampleDomain.com","(11111)","Test","Office.ExampleDomain.com","S-1-5-21-910167743-1234567890-1234567890-1388","krbtgt","S-1-5-21-910167743-1234567890-1234567890-502","0x40800000","0x0","0x17","2","10.0.0.1","48917","","","",,,
"3297648","20170914-00:02:31","4768","SuccessAudit",,"DC1.Office.ExampleDomain.com","(11111)","Test","Office.ExampleDomain.com","S-1-5-21-910167743-1234567890-1234567890-1388","krbtgt","S-1-5-21-910167743-1234567890-1234567890-502","0x40800000","0x0","0x17","2","10.0.0.1","58464","","","",,,
"3297649","20170914-00:03:01","4768","SuccessAudit",,"DC1.Office.ExampleDomain.com","(11111)","Test","Office.ExampleDomain.com","S-1-5-21-910167743-1234567890-1234567890-1388","krbtgt","S-1-5-21-910167743-1234567890-1234567890-502","0x40800000","0x0","0x17","2","10.0.0.1","51655","","","",,,
"3297651","20170914-00:03:23","4732","SuccessAudit",,"DC1.Office.ExampleDomain.com","(11111)","CN=DC1,OU=Users,OU=___,DC=DC1,DC=Office,DC=ExampleDomain,DC=com","S-1-5-21-2131238190-1946908106-23540016-118539","TestUser","DC1","S-1-5-21-910167743-1234567890-1234567890-14815","S-1-5-21-910167743-1234567890-1234567890-1318","$SECURITY_ACCOUNT","DC1","0x134b47790","-",,,,,,,
"3297650","20170914-00:03:23","4735","SuccessAudit",,"DC1.Office.ExampleDomain.com","(11111)","TestUser"","DC1","S-1-5-21-910167743-1234567890-1234567890-14815","S-1-5-21-910167743-1234567890-1234567890-1318","$SECURITY_ACCOUNT","DC1","0x134b47790","-","-","-",,,,,,,

To keep things simple, I have left out the SMB part of the requirements and set up a configuration as follows:

User root
Group root
define ROOT /opt/nxsec/
#NoFreeOnExit TRUE
define CERTDIR /opt/nxsec/var/lib/nxlog/cert
define CONFDIR /opt/nxsec/var/lib/nxlog
define LOGDIR /opt/nxsec/var/log/nxlog
define LOGFILE "%LOGDIR%/nxlog.log"

SpoolDir /opt/nxsec/var/spool/nxlog
PidFile /opt/nxsec/var/run/nxlog/nxlog.pid
CacheDir /opt/nxsec/var/spool/nxlog
ModuleDir /opt/nxsec/lib/nxlog/modules

<Extension json>
    Module      xm_json
</Extension>

<Extension csv>
    Module      xm_csv
    Fields      $Index, $TimeGenerated, $InstanceId, $EntryType, $UserName, $MachineName, $Category, $Field1, $Field2, $Field3, $Field4, $Field5, $Field6, $Field7, $Field8, $Field9, $Field10, $Field11, $Field12, $Field13, $Field14, $Field15, $Field16, $Field17
    EscapeControl FALSE
</Extension>
<Input in>
    Module im_file
    File "/test/test.csv"
    InputType     LineBased        
    PollInterval  1
    Exec csv->parse_csv();
    Exec $Message = to_json();
</Input>

<Output out>
    Module  om_file
    File    "/test/output.json"
    Sync    TRUE
</Output>

<Route 1>
    Path  in => out
</Route>

NXLog -v validates this file correctly and when run, nxlog does not indicate any errors or log any errors. Using strace, I can see that it even reads the source file, however, it is not writing to the output file.

I have tried various permutations of this configuration, including moving the Exec $Message = json->to_json(); line to the output module, but no matter what I do I cannot seem to get the CSV parsed and written back out again. No crashes happen and no log messages appear from nxlog, however.

Is there something I am doing wrong? Does anyone have a self-contained, complete working example to parse a Windows Event Log CSV export?

Hi, my logs:

<188>date=2017-09-26 time=10:53:04 devname=FT01 devid=************* logid=0000000011 type=traffic subtype=forward level=warning vd=root srcip=10.25d.1dd.ddd srcport=59ddd srcintf="portB" dstip=2dd.1dd.dd5.ddd dstport=dd3 dstintf="port9" poluuid=16594dae-dddd-51e7-dddd-da81ec23cc23 sessionid=26ddd5925 proto=6 action=ip-conn policyid=103 policytype=policy appcat="unscanned" crscore=5 craction=262144 crlevel=low devtype="Router/NAT Device" mastersrcmac=00:dd:14:dd:c8:dd srcmac=00:dd:14:dd:c8:ddt="default" appa

<189>date=2017-09-26 time=10:53:04 devname=FT01 devid=************* logid=0000000013 type=traffic subtype=forward level=notice vd=root srcip=10.25d.1dd.ddd srcport=59674 srcintf="portB" dstip=2dd.1dd.dd5.ddd dstport=443 dstintf="port9" poluuid=16594dae-dddd-51e7-dddd-da81ec23cc23 sessionid=260ddd925 proto=6 action=close policyid=103 policytype=policy dstcountry="United States" srccountry="Reserved" trandisp=snat transip=10.2dd.1dd.2dd transport=59674 service="HTTPS" appid=40568 app="HTTPS.BROWSER" appcat="Web.Client" apprisk=medium applist="default" appact=detected duration=140 sentbyte=1244 rcvdbyte=770 sentpkt=9 rcvdpkt=6 devtype="Router/NAT Device" mastersrcmac=00:2d:dd:6b:dd:60 srcmac=00:dd:14:dd:c8:ddc=88:ad:d2:88:eb

<189>date=2017-09-26 time=10:53:04 devname=FT01 devid=************* logid=0000000013 type=traffic subtype=forward level=notice vd=root srcip=10.2dd.dd.dd srcport=42768 srcintf="portB" dstip=2dd.1dd.dd.dd dstport=53 dstintf="port9" poluuid=0dddda60-dddd-51e7-dddd-56c9d0ddde2f sessionid=260ddd113 proto=17 action=accept policyid=84 policytype=policy dstcountry="Hong Kong" srccountry="Reserved" trandisp=snat transip=10.2dd.dd.2d transport=42768 service="DNS" appid=16195 app="DNS" appcat="Network.Service" apprisk=elevated applist="default" appact=detected duration=180 sentbyte=77 rcvdbyte=93 sentpkt=1 rcvdpkt=1 devtype="Router/NAT Device" mastersrcmac=00:dd:14:dd:c8:dd srcmac=00:dd:14:dd:c8:dddd:6060 srcmac=0

I want to use regular expressions:

field  >> regex

action = ^.+\saction=(\S+)\s
app = ^.+\sapp=\"(.+?)\"
appcat = ^.+\sappcat=\"(.+?)\"
applist = ^.+\sapplist=\"(.+?)\"
attack = ^.+\sattack=\"(.+?)\"
devid = ^.+\sdevid=(\S+)\s
dir = ^.+\sdir=(\S+)\s
dstcountry = ^.+\sdstcountry=\"(.+?)\"
dstintf = ^.+\sdstintf=\"(.+?)\"
dstip = ^.+\sdstip=(\S+)\s
dstport = ^.+\sdstport=(\S+)\s
... 175 more

What configuration to use?

<Input i.forti.log>
 Module im_file
 File "/var/log/forti.log"
 InputType LineBased
</Input>

<Output o.forti.log>
 Module om_tcp
 Host 192.168.00.00
 Port XXXXX
 CAFile /data/conf/ca.crt
 AllowUntrusted TRUE
 OutputType LineBased
</Output>

<Route r.forti.log>
 Path i.forti.log => o.forti.log
</Route>

Thank you very much!!

Hello all,

I'm using the im_file module to send log files to my logging server (graylog).  I'd like it to always skip the firts few lines of newly opened files.  Is this possible?  I couldn't find anything in the documentation.

Hello,

We have an issue where NXLog stops listening to the UDP port but nxlog remains running.  It posts a message to its log and I would like to run, via exec_async a powershell script that stops and restarts the nxlog service. However, I am unable to do so receiving this error:

ERROR if-else failed at line 9, character 97 in C:\Program Files (x86)\nxlog\conf\nxlog-self.conf. statement execution has been aborted; procedure 'exec_async' failed at line 9, character 97 in C:\Program Files (x86)\nxlog\conf\nxlog-self.conf. statement execution has been aborted; couldn't execute process C:\temp\restart; %1 is not a valid Win32 application.  

Does anybody know how to avoid this error and have nxlog run the script?

Thanks and regards

Peter

Hi. 
I'm new on this forum, and i'm bad with the english. 
I have a issue with NXLOG installation version nxlog-ce_2.9.1716

I take this link https://nxlog.co/system/files/products/files/348/nxlog-ce_2.9.1716_ubuntu_1404_amd64.deb
She asks me to have " libperl5.20 (>= 5.20.2) " but i have Ubuntu 16.04 with libperl5.22. 

I can't downgrade libperl.

Can you help me ?

In my configuration file I have an entry that looks as such:

<Input eventlog>
     Query <QueryList>\
               <Query Id="0">\
                   #<Select Path="Application"></Select>\
                   #<Select Path="Security"></Select>\
                   #<Select Path="Setup"></Select>\
                   #<Select Path="System"></Select>\
                   <Select Path="Forwarded Events">*</Select>\
               </Query>\
           </QueryList>
</Input>

When I start NXlog, I get all logs (Application, Security, Setup, etc...) How do only allow the "Forwarded Events" logs?

Hello,

first of all, sorry to bother you with a question that might be easy for you, but im a bit lost.

I would like to know if NXlog is compatible with WEF ?

Long story made short, I plan on using NXlog to output to my SIEM Security logs of Windows Domain Controller following this guide : 

https://www.petri.com/configure-event-log-forwarding-windows-server-2012-r2

wich as you can see, is to configured windows event forwarding ( to reduce the number of nxlog installation on critical server )

Once that first part done, I would like to know what config I should set to be able to "fetch" all of the "Forwarded Event" on my "windows log collector" ?

Thank you !