Ask questions. Get answers. Find technical product solutions from passionate experts in the NXLog community.

Problem about function string() on Ubuntu 16.04 LTS
hi, I am tesing nxlog on Ubuntu 16.04 LTS with the configuration <Extension fileop>         Module xm_fileop </Extension> <Input udpin>         Module im_udp         Host 127.0.0.1         Port 1514         Exec $HOSTIP = string(host_ip());         Exec file_write("/tmp/debug.txt", "HostIP:" + $HOSTIP); </Input> <Output udpfile>         Module om_file         File "/var/log/udp.log" </Output> <Route udp>     Path udpin => udpfile </Route> and  I generated log using linux command nc -u 127.0.0.1 1514 and can not get IP in /tmp/debug.txt. 1. I have reinstall the Ubuntu 16.04 LTS purely and install the deb file from nxlog.co, and it still does not get the IP address  2. I install the nxlog from source code, and it still does not get the IP address 3. I test the same configuration on CentOS7, CentOS6, Ubuntu 14.04 LTS. It works correctly. so, is there someting that does not compatible with the Ubuntu 16.04 LTS?

Little_Rock created
Replies: 1
View post »
last updated
Preserve source IP address
Hi I'm trying to do a syslog udp to tcp converter using this method from the user's manual: <Input in>     Module im_tcp     Host 0.0.0.0     Port 2345  </Input> <Output out>     Module      om_tcp     Host        mysyslog.domain.local     Port        514 </Output> ~~<Processor buffer>  Module pm_buffer  WarnLimit 800  MaxSize 1000  Type Mem  Exec if buffer_size() >= 80k drop(); </Processor> <Route 1>     Path in => buffer => out </Route> This is working correctly but all the events recieved by mysyslog.domain.local server are coming from the same source (the IP of the NXLOG server). Is there a way to preserve the source IP? Thanks in advance Regards, Olga

Olga35000 created
Replies: 1
View post »
last updated
How to convert ip4addr to string
the following context is my Input configuration and i get a functong from the manual doc,  but the return value of hos_ip() is ip4addr, and I want to convert the return value to string, I have try the string() function, but it does not work at all. Is there any way to convert the return value to string so i can and a new filed of the Input configuration. ip4addr host_ip(); description Return the first non-loopback IP address the hostname resolves to. return type ip4addr   <Input 59dcb7a6dd48cb088969e300>         Module im_file         File '/usr/local/nginx_raw/logs/access.log'         PollInterval 1         SavePos True         ReadFromLast True         Recursive True         RenameCheck False         Exec $FileName = file_name(); # Send file name with each message         Exec $HOSTIP = host_ip(); </Input>  

Little_Rock created
Replies: 1
View post »
last updated
NXLog and ODBC
Hi , Trying to create an ODBC connect for NXLog to connect to.   NXLog is installed on the same Windows 2012 server as the SQL Server 2008R2 instance.   Scenario 1: 32-bit ODBC is setup as a System DSN with a SQL Server account that has DBO access to the desired database NXLog service is setup to run under the System account.      - I've tried both drivers available on the system ("SQL Server Native Client 10.0"  and "SQL Server") - get the same result in the error log for each: ERROR im_odbc couldn't connect to the database, 28000:2:18456:[Microsoft][SQL Server Native Client 10.0][SQL Server]Login failed for user ''. (odbc error code: -1)and ERROR im_odbc couldn't connect to the database, 28000:2:18456:[Microsoft][ODBC SQL Server Driver][SQL Server]Login failed for user ''. (odbc error code: -1)   Scenario 2: Same ODBC, but with a Windows account that has full Admin access to the desired databases, and is the same account logged into Windows NXLog service is setup to run under this same account. Goal is to have the same user account accessing everything, in the hope of getting it to connect.   Same error messages as above.   Login failed for user ' '.     Since the error messages don't show the user that is failing to login, I'm having trouble narrowing down where the failure is at.   NXLOG.conf file: <Input call_logs> Module im_odbc ConnectionString DSN=SIEM_NXLog;database=recorder; SQL SELECT ident as id ,at.audit_time as EventTime ,am.audit_module_name as Message FROM mytables...  WHERE at.ident>? SavePos TRUE </Input>   There's one line in the documentation that has me scratching my head: SECTION 6.2.18 (ODBC) The data source must be accessible by the user which nxlog is running under.   I'm not sure if this means that the NTService account needs database access? Or, if the service must be under a Windows account user that has database access? Or, by using a ODBC->System DSN , shouldn't the ODBC already be accessible to all users on the system?   Any thoughts or insight would be helpful. Thanks in advance.   Cheers,  Peter          

pbechard created
Replies: 2
View post »
last updated
High CPU load on Windows 7
Hello! I noticed strange cases nxlog.exe is loading 60%-90% CPU. Log file has nothings about errors. I have the same config file for win7 and winXP, but winXP is OK in contrast to win7. Also if I use om_udp module instead om_tcp in config file for win7 high load is desappear.   Could anybody say me what is the problem, or how I can get more information about cause of problems.

lenard.daian created
Replies: 1
View post »
last updated
Get gmt time with IETFTimestampInGMT
Hello! I have a question about IETFTimestampInGMT option. In the documentation - section "Syslog (xm_syslog)" - I can read : "IETFTimestampInGMT This optional boolean directive can be used to format the timestamps produced by to_syslog_ietf() in GMT instead of local time. This defaults to FALSE so that local time is used by default with a timezone indicator" So here's a part of my nxlog.conf : <Extension _syslog>     Module  xm_syslog     IETFTimestampInGMT  TRUE </Extension>   <Output  outUDP>     Module om_file     Exec log_info($Raw_event);parse_syslog_ietf(); log_info($EventTime);     CreateDir TRUE     Sync FALSE     File "C:\\" + $SourceName + "\\out.log" </Output>   Output : <7>1 2017-10-10T12:30:59.308493+02:00 JohnDoe-PC appTest 3788 tag1 - Testing debug message 2017-10-10 12:30:59   (I tried to print $timestamp but nothing is printed.) So I tought that setting IETFTimestampInGMT to true will convert the $EventTime in GMT but it didn't. I checked the documentation and test different configurations, I should have misunderstood the way it works. What did I miss?   Thank you for your help :)

Savane created
Replies: 1
View post »
last updated
CSV file in Linux
Hello everyone, As you can see from my NXlog config below, I am pulling the Linux auth logs in and passing them to our SIEM platform. The auth logs work fine, however as you can also see I am trying to pull in a CSV file that is located on the Linux box. Each individual event is surrounded by curly brackets e.g: {'event}, so I would like to just pull each event into a field called message and then parse these messages on our SIEM platform. I can't seem to get this to work whatsoever, the TCP connection initiates, but NXlog doesn't pickup the CSV file. Any ideas?   Cheers G ######################################## # Global directives                    # ######################################## User nxlog Group nxlog LogFile /var/log/nxlog/nxlog.log LogLevel INFO ######################################## # Modules                              # ######################################## <Extension _syslog> Module  xm_syslog </Extension> <Input auth_logs>         Module  im_file         File    "/var/log/auth.log"         SavePos TRUE         ReadFromLast    TRUE </Input> <Output to_relay>         Module  om_tcp         Host    127.0.0.1         Port    20009         OutputType      LineBased </Output> ######################################## # Routes                               # ######################################## <Route 1>         Path    auth_logs => to_relay </Route> <Extension csv1>         Module  xm_csv         Fields  $Message         Delimiter       '{' </Extension> <Input filein>         Module  im_file         File    "/etc/ingest/sucuri/sucuri.csv"         Exec    csv1->parse_csv(); </Input> <Output test>         Module  om_tcp         Host    127.0.0.1         Port    20002         OutputType      Binary </Output> <Route 2>         Path    filein => test </Route>

multiplierx created
Replies: 1
View post »
last updated
nxlog.exe consumes 100% cpu
We have been running this product on our domain controller and within the last week or two, the nxlog.exe process consumes 100% of the CPU.   we stop the nxlog service and the CPU instantly drops to <5%.    start the service and it immediately rises to 100%.   We've uninstalled/installed without change.  Any advice on how to correct this is greatly appreciated. 

wklaus created
Replies: 6
View post »
last updated
How to filter repetitive events
In case of DOS attack on a device, there would be a surge of logs in a very short time and all the events look simillar with change in one or two parameters source port/destination port/source ip/destination ip. In such case, can we filter such repeatitive logs in NXLOG agent? If yes, How to do that? I tried pm_norepeat but it didnt help. Any other alternate options?    

kdevmu created
Replies: 1
View post »
last updated
Possibility of Global Tags for all Inputs
What I'm trying to achive is to have a few Values globaly defined and the should be automatically added to all inputs. Ie the same thing as Global Tags in Telegraf Today I first use a define statement in the global part of NXLog.conf Define Company Acme    For each input I define i have to add a Exec line Exec $Company = '%Company%'; I would like to be able to do this only once and have it automatically appended to all inputs. For a multi Company Scenario with a lot of logfiles It gets rather messy to maintain when you need to add anoter global value. It works well in Telegraf so I would like to be able to do something similar in Nxlog  

mats created
Replies: 1
View post »
last updated
Parsing Windows Event Log CSV with NXLog (into JSON)
Hi All,  I am looking to use nxlog to transform a CSV formatted input from an SMB share into a json formatted line-by-line output for parsing by further handlers of our logging information. The CSV in question is an export of Windows Event Logs from a domain controller. An example of the CSV I am trying to parse is: "Index","TimeGenerated","InstanceId","EntryType","UserName","MachineName","Category","Field1","Field2","Field3","Field4","Field5","Field6","Field7","Field8","Field9","Field10","Field11","Field12","Field13","Field14","Field15","Field16","Field17" "3297643","20170914-00:00:01","4768","SuccessAudit",,"DC1.Office.ExampleDomain.com","(11111)","Test","Office.ExampleDomain.com","S-1-5-21-910167743-1234567890-1234567890-1388","krbtgt","S-1-5-21-910167743-1234567890-1234567890-502","0x40800000","0x0","0x17","2","10.0.0.1","55393","","","",,, "3297644","20170914-00:00:31","4768","SuccessAudit",,"DC1.Office.ExampleDomain.com","(11111)","Test","Office.ExampleDomain.com","S-1-5-21-910167743-1234567890-1234567890-1388","krbtgt","S-1-5-21-910167743-1234567890-1234567890-502","0x40800000","0x0","0x17","2","10.0.0.1","45086","","","",,, "3297645","20170914-00:01:01","4768","SuccessAudit",,"DC1.Office.ExampleDomain.com","(11111)","Test","Office.ExampleDomain.com","S-1-5-21-910167743-1234567890-1234567890-1388","krbtgt","S-1-5-21-910167743-1234567890-1234567890-502","0x40800000","0x0","0x17","2","10.0.0.1","35822","","","",,, "3297646","20170914-00:01:31","4768","SuccessAudit",,"DC1.Office.ExampleDomain.com","(11111)","Test","Office.ExampleDomain.com","S-1-5-21-910167743-1234567890-1234567890-1388","krbtgt","S-1-5-21-910167743-1234567890-1234567890-502","0x40800000","0x0","0x17","2","10.0.0.1","44883","","","",,, "3297647","20170914-00:02:01","4768","SuccessAudit",,"DC1.Office.ExampleDomain.com","(11111)","Test","Office.ExampleDomain.com","S-1-5-21-910167743-1234567890-1234567890-1388","krbtgt","S-1-5-21-910167743-1234567890-1234567890-502","0x40800000","0x0","0x17","2","10.0.0.1","48917","","","",,, "3297648","20170914-00:02:31","4768","SuccessAudit",,"DC1.Office.ExampleDomain.com","(11111)","Test","Office.ExampleDomain.com","S-1-5-21-910167743-1234567890-1234567890-1388","krbtgt","S-1-5-21-910167743-1234567890-1234567890-502","0x40800000","0x0","0x17","2","10.0.0.1","58464","","","",,, "3297649","20170914-00:03:01","4768","SuccessAudit",,"DC1.Office.ExampleDomain.com","(11111)","Test","Office.ExampleDomain.com","S-1-5-21-910167743-1234567890-1234567890-1388","krbtgt","S-1-5-21-910167743-1234567890-1234567890-502","0x40800000","0x0","0x17","2","10.0.0.1","51655","","","",,, "3297651","20170914-00:03:23","4732","SuccessAudit",,"DC1.Office.ExampleDomain.com","(11111)","CN=DC1,OU=Users,OU=___,DC=DC1,DC=Office,DC=ExampleDomain,DC=com","S-1-5-21-2131238190-1946908106-23540016-118539","TestUser","DC1","S-1-5-21-910167743-1234567890-1234567890-14815","S-1-5-21-910167743-1234567890-1234567890-1318","$SECURITY_ACCOUNT","DC1","0x134b47790","-",,,,,,, "3297650","20170914-00:03:23","4735","SuccessAudit",,"DC1.Office.ExampleDomain.com","(11111)","TestUser"","DC1","S-1-5-21-910167743-1234567890-1234567890-14815","S-1-5-21-910167743-1234567890-1234567890-1318","$SECURITY_ACCOUNT","DC1","0x134b47790","-","-","-",,,,,,, To keep things simple, I have left out the SMB part of the requirements and set up a configuration as follows: User root Group root define ROOT /opt/nxsec/ #NoFreeOnExit TRUE define CERTDIR /opt/nxsec/var/lib/nxlog/cert define CONFDIR /opt/nxsec/var/lib/nxlog define LOGDIR /opt/nxsec/var/log/nxlog define LOGFILE "%LOGDIR%/nxlog.log" SpoolDir /opt/nxsec/var/spool/nxlog PidFile /opt/nxsec/var/run/nxlog/nxlog.pid CacheDir /opt/nxsec/var/spool/nxlog ModuleDir /opt/nxsec/lib/nxlog/modules <Extension json> Module xm_json </Extension> <Extension csv> Module xm_csv     Fields      $Index, $TimeGenerated, $InstanceId, $EntryType, $UserName, $MachineName, $Category, $Field1, $Field2, $Field3, $Field4, $Field5, $Field6, $Field7, $Field8, $Field9, $Field10, $Field11, $Field12, $Field13, $Field14, $Field15, $Field16, $Field17 EscapeControl FALSE </Extension> <Input in> Module im_file File "/test/test.csv" InputType LineBased PollInterval 1 Exec csv->parse_csv(); Exec $Message = to_json(); </Input> <Output out> Module om_file File "/test/output.json" Sync TRUE </Output> <Route 1> Path in => out </Route> NXLog -v validates this file correctly and when run, nxlog does not indicate any errors or log any errors. Using strace, I can see that it even reads the source file, however, it is not writing to the output file. I have tried various permutations of this configuration, including moving the Exec $Message = json->to_json(); line to the output module, but no matter what I do I cannot seem to get the CSV parsed and written back out again. No crashes happen and no log messages appear from nxlog, however. Is there something I am doing wrong? Does anyone have a self-contained, complete working example to parse a Windows Event Log CSV export?

avhk created
Replies: 2
View post »
last updated
HELP. Divided into fields Fortinet logs with regular expressions.
Hi, my logs: <188>date=2017-09-26 time=10:53:04 devname=FT01 devid=************* logid=0000000011 type=traffic subtype=forward level=warning vd=root srcip=10.25d.1dd.ddd srcport=59ddd srcintf="portB" dstip=2dd.1dd.dd5.ddd dstport=dd3 dstintf="port9" poluuid=16594dae-dddd-51e7-dddd-da81ec23cc23 sessionid=26ddd5925 proto=6 action=ip-conn policyid=103 policytype=policy appcat="unscanned" crscore=5 craction=262144 crlevel=low devtype="Router/NAT Device" mastersrcmac=00:dd:14:dd:c8:dd srcmac=00:dd:14:dd:c8:ddt="default" appa <189>date=2017-09-26 time=10:53:04 devname=FT01 devid=************* logid=0000000013 type=traffic subtype=forward level=notice vd=root srcip=10.25d.1dd.ddd srcport=59674 srcintf="portB" dstip=2dd.1dd.dd5.ddd dstport=443 dstintf="port9" poluuid=16594dae-dddd-51e7-dddd-da81ec23cc23 sessionid=260ddd925 proto=6 action=close policyid=103 policytype=policy dstcountry="United States" srccountry="Reserved" trandisp=snat transip=10.2dd.1dd.2dd transport=59674 service="HTTPS" appid=40568 app="HTTPS.BROWSER" appcat="Web.Client" apprisk=medium applist="default" appact=detected duration=140 sentbyte=1244 rcvdbyte=770 sentpkt=9 rcvdpkt=6 devtype="Router/NAT Device" mastersrcmac=00:2d:dd:6b:dd:60 srcmac=00:dd:14:dd:c8:ddc=88:ad:d2:88:eb <189>date=2017-09-26 time=10:53:04 devname=FT01 devid=************* logid=0000000013 type=traffic subtype=forward level=notice vd=root srcip=10.2dd.dd.dd srcport=42768 srcintf="portB" dstip=2dd.1dd.dd.dd dstport=53 dstintf="port9" poluuid=0dddda60-dddd-51e7-dddd-56c9d0ddde2f sessionid=260ddd113 proto=17 action=accept policyid=84 policytype=policy dstcountry="Hong Kong" srccountry="Reserved" trandisp=snat transip=10.2dd.dd.2d transport=42768 service="DNS" appid=16195 app="DNS" appcat="Network.Service" apprisk=elevated applist="default" appact=detected duration=180 sentbyte=77 rcvdbyte=93 sentpkt=1 rcvdpkt=1 devtype="Router/NAT Device" mastersrcmac=00:dd:14:dd:c8:dd srcmac=00:dd:14:dd:c8:dddd:6060 srcmac=0 I want to use regular expressions: field  >> regex action = ^.+\saction=(\S+)\s app = ^.+\sapp=\"(.+?)\" appcat = ^.+\sappcat=\"(.+?)\" applist = ^.+\sapplist=\"(.+?)\" attack = ^.+\sattack=\"(.+?)\" devid = ^.+\sdevid=(\S+)\s dir = ^.+\sdir=(\S+)\s dstcountry = ^.+\sdstcountry=\"(.+?)\" dstintf = ^.+\sdstintf=\"(.+?)\" dstip = ^.+\sdstip=(\S+)\s dstport = ^.+\sdstport=(\S+)\s ... 175 more What configuration to use? <Input i.forti.log>  Module im_file  File "/var/log/forti.log"  InputType LineBased </Input> <Output o.forti.log>  Module om_tcp  Host 192.168.00.00  Port XXXXX  CAFile /data/conf/ca.crt  AllowUntrusted TRUE  OutputType LineBased </Output> <Route r.forti.log>  Path i.forti.log => o.forti.log </Route>   Thank you very much!!

absolis created
Replies: 1
View post »
last updated
Skipping first X lines (im_file)
Hello all, I'm using the im_file module to send log files to my logging server (graylog).  I'd like it to always skip the firts few lines of newly opened files.  Is this possible?  I couldn't find anything in the documentation.

Lorenzo.Henriquez created
Replies: 1
View post »
last updated
exec_async - can't run powershell script
Hello, We have an issue where NXLog stops listening to the UDP port but nxlog remains running.  It posts a message to its log and I would like to run, via exec_async a powershell script that stops and restarts the nxlog service. However, I am unable to do so receiving this error: ERROR if-else failed at line 9, character 97 in C:\Program Files (x86)\nxlog\conf\nxlog-self.conf. statement execution has been aborted; procedure 'exec_async' failed at line 9, character 97 in C:\Program Files (x86)\nxlog\conf\nxlog-self.conf. statement execution has been aborted; couldn't execute process C:\temp\restart; %1 is not a valid Win32 application.   Does anybody know how to avoid this error and have nxlog run the script? Thanks and regards Peter  

PeterF created
Replies: 1
View post »
last updated
libperl install
Hi.  I'm new on this forum, and i'm bad with the english.  I have a issue with NXLOG installation version nxlog-ce_2.9.1716 I take this link https://nxlog.co/system/files/products/files/348/nxlog-ce_2.9.1716_ubuntu_1404_amd64.deb She asks me to have " libperl5.20 (>= 5.20.2) " but i have Ubuntu 16.04 with libperl5.22.  I can't downgrade libperl. Can you help me ?

Jboucard created
Replies: 4
View post »
last updated
Only sending "Forwarded Events" logs
In my configuration file I have an entry that looks as such: <Input eventlog>      Query <QueryList>\                <Query Id="0">\                    #<Select Path="Application"></Select>\                    #<Select Path="Security"></Select>\                    #<Select Path="Setup"></Select>\                    #<Select Path="System"></Select>\                    <Select Path="Forwarded Events">*</Select>\                </Query>\            </QueryList> </Input> When I start NXlog, I get all logs (Application, Security, Setup, etc...) How do only allow the "Forwarded Events" logs?

dsw283 created
Replies: 1
View post »
last updated
Windows Event Forwarder and NXLog
Hello, first of all, sorry to bother you with a question that might be easy for you, but im a bit lost. I would like to know if NXlog is compatible with WEF ?   Long story made short, I plan on using NXlog to output to my SIEM Security logs of Windows Domain Controller following this guide :  https://www.petri.com/configure-event-log-forwarding-windows-server-2012-r2 wich as you can see, is to configured windows event forwarding ( to reduce the number of nxlog installation on critical server )   Once that first part done, I would like to know what config I should set to be able to "fetch" all of the "Forwarded Event" on my "windows log collector" ?     Thank you !  

gh0stid created
Replies: 1
View post »
last updated
LEEF Format for MS Event Logs
Hello, Has anyone ever set up NxLog to forward windows events to any log aggregator or SIEM that accetps LEEF format? I see the enterprise edition has a LEEF module but wanted to see if this had been done or if there are any issues in doing so. 

aolague created
Replies: 1
View post »
last updated
NXLOG service faiils to start on 2012R2
I have installed nxlog on our 2012R2 DC's.  I go into the file and uncomment out the path to the software.  I then replace the IP address of syslog server with ours and then save the file.  I then go and try and start the nxlog service and immediatly get an error 1053:  The service did not respond to a control request in a timely manner.   I look in the nxlog log file and see the following error message --->  nxlog failed to start: Couldn't change to SpoolDir '%ROOT%\data' The system cannot find the path specified.    I know this error message is incorrect because the same path is used for CacheDir, Pidfile, and LogFile and those seem to be working.    Upon further experimentation if I comment out the Logfile path as well as the Logfile path I can get the service to start but no logs are sent over to my syslog server.   I find it funny that even thought the error is for the SpoolDir and the Logfile seems to be working I have to comment out both items to get the service to start otherwise I continue to get the Error 1053. I'm hoping someone can help with this.   Thanks.

pclark created
Replies: 1
View post »
last updated
Help with GELF_TCP fields
Need some help, I want the fields "$srcip, $srcport, $dstip, $dstport" to be put together in another field, called "$netinfo", how do I do it ?? My logs #fields ts uid id.orig_h id.orig_p id.resp_h id.resp_p proto trans_id query qclass qclass_name qtype qtype_name rcode rcode_name AA TC RD RA Z answers TTLs rejected #types time string addr port addr port enum count string count string count string count string bool bool bool bool count vector[string] vector[interval] bool 1482865188.959602 CMyjvLxxxxxxx0MJjb xxx.xx.192.250 3xxx xxx.xxx.162.xxx 53 udp 19626 - - - - - 0 NOERROR F F F F 0 - - F 1482865189.162798 CW1kwxxxxxxxC3Ug0j xxx.xx.192.250 xxxx5 xxx.xxx.xxx.xxx 53 udp 250 r4.sn-a5m7znes.googlevideo.com - - - - 0 NOERROR T F F F 0 xxx.194.xxx.233 1800.000000 F 1482865189.182565 Cir6Sz3xxxxxO60PD6 fe80::xxx:f35c:xxxx:61ad 65535 ff02::1:3 5355 udp 1772 host 1 C_INTERNET 1 A - - F F F F 0 - - F Nxlog .conf <Extension csv.dns.log> Module xm_csv Fields $timestamp,$uid,$srcip,$srcport,$dstip,$dstport,$service,$transid,$qresponse,$query,$qclass,$qclassname,$qtype,$qtypename,$rcode,$rcodename,$aa,$tc,$rd,$ra,$z,$answ FieldsType string,string,string,string,string,string,string,string,string,string,string,string,string,string,string,string,string,string,string,string,string,string,string Delimiter \t </Extension> <Input i.dns.log> Module im_file File "/*PATH*/dns.log" ReadFromLast TRUE Exec csv.dns.log->parse_csv(); </Input> <Output o.dns.log> Module om_ssl Host 192.XXX.X.XXX Port ZZZZ OutputType GELF_TCP CAFile /data/conf/ca.crt AllowUntrusted TRUE </Output> <Route r.dns.log> Path i.dns.log => o.dns.log </Route> Thank you  

absolis created
Replies: 1
View post »
last updated