Hello! It appears that any nested data - e.g. from EventData - will be overwritten if the field exists on the event itself. For example, please see your documentation on sysmon.  Notice that ProcessID is a field on the event, and is also a field under EventData, albeit with different data. The resulting JSON output includes only the ProcessID from the event itself, not from the eventdata.  In the example at the link, notice that the Event.ProcessID is 1680.  The Event.EventID.ProcessID is 25848.  Notice that the data from the latter (generally more specific to this type of event, and thus generally more important) is not available as structured data anywhere. Personally I'm not using this at the moment, but, I could see many situations where the generic Event fields overwrite valuable information from Event Data. Cheers!  

Hello, Could you please clarify how can I cut out some fields from forwarded event? My situation is the following; I have a local log file on the server where installed nxlog agent. Using im_file module I have defined path to file and filename. After that I configured to forward this log to remote syslog server. When I opened forwarded log on the romete syslog server and find out that my log line was changed. It was added time and server name wehere original log file is stored. I have posted a line from the remote server and marked columns which were added during the forwarding. Jan 12 13:16:28 siem-vm Jan 12 00:01:37 mail2-vm-srv postfix/cleanup[7412]: 6EC1E2A23F9: message-id=<20170111220136.5AE682A23F6> Can you help me? Thank you in advance.

Hi, can anyone help me with the output of my nxlog.conf I want to convert epoch time from my Bro logs; Part of the logs: 1482865199.693051 FSYupp4bmRs8tT5Jyg 3 5A00020E4289E78C695848...... 1482865200.300809 FmXyl22Uxsq1cudDd8 3 5A00020E4289E78C695848...... 1482865200.203542 FAuSUU3X9pgdSJ2D2g 3 5A00020E4289E78C695848....... 1482865201.043722 F0KUdW3Nm5edyqPXLl 3 0CEAC9CAD430F24F334575....... My current settings are <Output o.name.log>  Module om_tcp  Host xx.xxx.xxx.xxx  Port xxxx  OutputType LineBased </Output> Thanks!

I'm attempting to demo xnlog and running into a problem where the Windows Server 2016 event logs are being sent to AWS ElasticSearch Service with the EventTime being a string. This basically renders it impossible to index the logs, as the Kibana board requires a time-field name and is not recongizing the string as a datetime.  Any suggestions on this, or is this a potential bug with Server 2016?

Hi everybody, stumbled over a problem that sometimes I get hostnames from nxlog other times it's fqdns. Happens only with internal nxlog messages.I tired to fix this by using the Exec $Hostname = hostname_fqdn(); statement. # Nxlog internal logs <Input internal>    Module im_internal    Exec $EventReceivedTime = integer($EventReceivedTime) / 1000000; to_json();    Exec $Hostname = hostname_fqdn();  </Input> Still get messages with designation nxlog: internal that have a source_host with hostname only and not fqdn. Either I am missing something really obvious or something is broken. Any guidance to troubleshoot or figure out what is going on much appreciated Best regards Tobias

I have a NXLog service running in Windows Server shipping event logs. It has 2 destinations, 1 is TCP sending logs to syslog_ng and another is GLEF UDP. When my syslog_ng server goes offline, the logs I'm receiving at the GLEF UDP output also stops. Is there any way to make NXlog send the logs to the other output/route even if one output/route fails? Config: ​ Module om_tcp Host 192.168.1.11 Port 25002 Exec to_syslog_snare(); Module om_udp Host 192.168.1.12 Port 51416 OutputType GELF Path in => out Path in => analyze  

Hi everybody, while experimenting with nxlog and relaying windows event logs I stumbled over the issue that even in the latest versions the field Keywords from the Window log is defined in ms_msevent as integer which doesn't fit the values stored in the field in windows. Are there any plans to fix this? best regards Tobias  

Given a number of application logs sharing the same HeaderLine and EndLine regular expressions we are trying out a xm_multiline with im_file config using wild cards.  <Extension multi>   Module      xm_multiline     HeaderLine /^\d{4}-\d{2}-\d{2} \d{2}:\d{2}:\d{2},\d{3} @ batch_task\._init_logger : \[INFO\]\+ /     EndLine /^\d{4}-\d{2}-\d{2} \d{2}:\d{2}:\d{2},\d{3} @ run_batch\.<module> : \[INFO\]- / </Extension> <Input inPython>     Module    im_file     File "C:\\data\\server1\\*.log"     InputType multi     Exec $FileName = file_name(); </Input> It works consistently without wildcards pointed at one file. It works intermittently with wildcards pointed at multiple files being written to concurrenlty. I'm wondering if this is a supported use case. i.e. multiline events from wildcarded files being written to concurrenlty. Or should we be specifiying each input file individually? thanks, Rob  

does nxlog-2.9.1716 still uses LibExpat v2.0.1 and LibPCRE v8.02? Impact: LibPCRE v8.02 is vulnerable to DoS and code overflow. LibExpat v2.0.1 has 4 publicly identified vulnerabilities. References https://www.cvedetails.com/vulnerability-list/vendor_id-12037/product_id-22545/version_id-129378/Libexpat-Expat-2.0.1.html https://www.cvedetails.com/vulnerability-list/vendor_id-3265/product_id-5715/version_id-191791/Pcre-Pcre-8.02.html is it possible to update LibExpat to v2.1.0 and LibPCRE to v8.39?

I want to compile nxlog and package it on windows, but I can't find any material to refer to. Who can help me, give me some advice. Thank you!

Hi everyone !  In our implementation of NxLog on our systems we are looking at some ways to centralize config management & deployement on all agents. As nxLog is integrated with some other software, we are looking at some automatic features like scripts or so to deploy new config on the agents all in a secure way.  If anyone has any tips on how to do so, help would be greatly appraciated. We first thought to use NxLog manager but it doesn't really fit our need as it requires the administrator to actually do some handy work and they are really lazy to be honest ;-) Thank you guys in advance if you can give us some tips :) Regards J. Denis

Hi everyone, Do you know if it is possible to schedule a module execution ? Or does the Schedule function only takes Exec commands ? Here is the solution I've come up with to schedule log sending: <Input im_file_test> Module im_file File '/mnt/test/scheduler/*log' InputType LineBased SavePos TRUE <Schedule> When 0 12 * * * Exec fileop->file_copy("/mnt/test/test.log", "/mnt/scheduler/test.log"); </Schedule> </Input> <Output om_tcp_siem> Module om_tcp Host 192.168.0.10 Port 514 OutputType LineBased </Output> <Route RouteTestB> ​ Path im_file_test => om_tcp_siem </Route> Here is what I would like to do (getting rid of xm_fileop for access privilege reasons:) <Input im_file_test> <Schedule> When 0 12 * * * Module im_file File '/mnt/test/test.log' InputType LineBased SavePos TRUE </Schedule> </Input> Thank you for your time :)

Hello. Question is simple. Have nxlog some module for working with mysql database?

Does anyone have nxlog usefully sending Microsoft DNS logs? I have logging turned on, and I have tried with and without the details option checked. Using it without the details is probably enough for us right now, as it shows the source and the requested URL. However, when sent to syslog, only a blank line is sent. The file output looks like below. The issue may be the space between each line? 8/12/2016 12:58:43 PM 0AE0 PACKET  000000F7524D7120 UDP Rcv x.x.x.x     5a68   Q [0001   D   NOERROR] A      (5)ctldl(13)windowsupdate(3)com(0) 8/12/2016 12:58:47 PM 0AE0 PACKET  000000F75221C070 UDP Rcv x.x.x.x     5a68   Q [0001   D   NOERROR] A      (5)ctldl(13)windowsupdate(3)com(0) Any ideas?  

Hello Guys, I have a problem, that NXLog community edition sometimes sends 2 messages to GrayLog instead of 1, from a log file. These are the messages, that came through together for example: 08 2016/12/02 13:13:28.581 Response: << MerchantId^XXXXXX~TransactionType^XX~OrderNumber^XXXXXXXX~StrId^XXXXXXXXXX~PTTID^XXXXXXXXXXX~MOP^XX~CurrencyId^XXX~Amount^XX.XX~AuthCode^XXXXXX~RequestType^X~MessageCode^XXXX~Message^XXXXXXXXX XXXXXXXX~CVNMessageCode^X~CVNMessage^XXXXXXXX/XXXXXX XXXXXXX >> 08 2016/12/02 13:13:28.581 ReportResult: (IDMMSITransaction = XXXXXXXX, Result = , MessageCode = XXXX) This is the nxlog.conf part for this log:   <Extension exlogs>     Module        xm_multiline     HeaderLine    /^.. \d{4}\/\d{2}\/\d{2} \d{2}\:\d{2}\:\d{2}\.\d{3}/ </Extension> <Input Logs>     Module        im_file     File        "D:\\path\\to\\logs\\log_*"     SavePos        TRUE     InputType    exlogs </Input> <Route logs-graylog>     Path        Logs => graylog </Route>   The problem doesn't happen with all the messages, but quite often and prevents me to create proper extractors. Do you have any idea what could cause this?   Thanks, Tamas Juhasz tjuhasz@stanjames.com

~~Please confirm about HPUX server how to install NXLOG server or client.

Hello, I'm catching up logs, but when they arrived to my Graylog, they take the actual date and not the old date.  I saw my logs and they don't have format date. They're csv, and I have this field : FW_date_time:10Nov2016;21:40:10 and I need to assign it on the timestamp, but I don't kno how. Could you help me please to convert this on a good format? I tried this (I have a field called: FW_date_time on my source csv file): Exec          FW->parse_csv(); Exec            $EventReceivedTime =strptime($FW_date_time, '%Y %b %d %T'); Exec              $EventTime =strptime($FW_date_time, '%Y %b %d %T'); But it didn't work. Thank you very much for your help. Cordially,   Ana      

I'm having some issues sending event logs from a windows 2012 R2 client using nxlog ce agent to a linux syslog-ng server.  The tcp connection appears to be getting dropped by syslog-ng due to an invalid frame header. syslog-ng says: 2016-11-22T09:10:35+00:00 server syslog-ng[8317]: Invalid frame header; header='' 2016-11-22T09:10:35+00:00 server syslog-ng[8317]: Syslog connection closed; fd='8', client='AF_INET(ip.of.nxlog.client:55473)', local='AF_INET(0.0.0.0:6514)' 2016-11-22T09:10:36+00:00 server syslog-ng[8317]: Syslog connection accepted; fd='8', client='AF_INET(ip.of.nxlog.client:55474)', local='AF_INET(0.0.0.0:6514)' nxlog says: 2016-11-22 15:24:27 ERROR om_tcp send failed; An established connection was aborted by the software in your host machine.   2016-11-22 15:24:28 INFO connecting to ip.of.syslog-ng.server:6514 2016-11-22 15:24:28 INFO reconnecting in 1 seconds I've tried several variations in the nxlog.conf file to work around this issue.  Here is my nxlog.conf: define ROOT C:\Program Files (x86)\nxlog Moduledir %ROOT%\modules CacheDir %ROOT%\data Pidfile %ROOT%\data\nxlog.pid SpoolDir %ROOT%\data LogFile %ROOT%\data\nxlog.log <Extension _syslog>     Module      xm_syslog </Extension> <Input in>     Module      im_msvistalog # For windows 2003 and earlier use the following: #   Module      im_mseventlog     ReadFromLast TRUE     <QueryXML>        <QueryList>          <Query Id='1'>             <Select Path='Application'>*</Select>             <Select Path='Security'>*</Select>             <Select Path='System'>*</Select>          </Query>        </QueryList>    </QueryXML> </Input> <Processor eventlog_transformer>   Module pm_transformer   Exec $Hostname = hostname();  OutputFormat syslog_rfc5424  </Processor> <Output out>    Module om_tcp    Host        ip.of.syslog-ng.server    Port 6514        Exec $raw_event = replace($raw_event, "\r\n", " ");     Exec $raw_event = replace($raw_event, "\t", " ");     #Exec to_syslog_ietf();     #Exec        to_syslog_bsd(); </Output> <Route 1>     Path        in => eventlog_transformer => out </Route> Here is the syslog-ng configuration: @version:3.5 @include "scl.conf" # syslog-ng configuration file. # # This should behave pretty much like the original syslog on RedHat. But # it could be configured a lot smarter. # # See syslog-ng(8) and syslog-ng.conf(5) for more information. # # Note: it also sources additional configuration files (*.conf) #       located in /etc/syslog-ng/conf.d/ #Options # ## Warnings # # options {     create_dirs(yes);     dir_perm(0755);     perm(0644);     log_msg_size(65536);     log_fifo_size(10000);     threaded(yes);     ts_format(iso);     keep_hostname(no);     use_dns(no);     dns_cache(no);     use_fqdn(no);     flush_lines(100);     stats_freq(60);     mark_freq(36400); }; #Sources source s_sys {     system();     internal(); }; source s_network {     syslog( port(6514) flags(syslog-protocol)  transport("tcp") keep-alive(yes) so-keepalive(yes) so_rcvbuf(1073741824) log-fetch-limit(100) log-iw-size(100) max-connections(5000));     syslog( port(514) transport("udp") so_rcvbuf(1073741824)); }; #Destinations #destination d_all       { file("/var/log/logs/all_logs/$HOST/$HOST-$YEAR-$MONTH-$DAY-all_logs.log" create_dirs(yes)); }; destination d_all       { file("/var/log/logs/all_logs/$HOST/$HOST.log" create_dirs(yes)); }; #Filters #Logs #   { source(s_sys); filter(f_kernel); destination(d_cons); }; log { source(s_sys); destination(d_all); }; log { source(s_network); destination(d_all); flags(flow-control); }; # Source additional configuration files (.conf extension only) @include "/etc/syslog-ng/conf.d/*.conf"   I can get things to work using udp, but can't get it to work over tcp.  Has anyone else had any success getting nxlog to send events from windows to a syslog-ng server over tcp?

Hello,   I have NXLOG installed, and use UDP to receive and sent the servers logs. However, when I run the netstat command I found that there are packets on error and other received through unknown ports. See the lines shown :   UDP:  123234944 packets received 223432 packets to unknown port received. 523455 packet receive errors 2111 packets sent Why does it happen? I didn't configure any other port on my NXLOG and the firewall rules only accept 514 port to communicate with my NXLOG server. How could I verify what is this? Is it normal?   Thank you very much for your help and answer   Cordially,   Ana

After a bit of trial and and lots of reading, I managed to get Graylog2 working like a charm I'm using NXLOG to send the logs to Graylog via GELF UDP Right now I'm just testing and trying diffrent things. Right now I';m testing with just 4 servers (have close to 100) has consumed a fair bit of space So to evaluate usage, I figured I'd just send the security logs (these are all Windows Servers) Original working configuration __________________________________________________________ define ROOT C:\Program Files (x86)\nxlog Moduledir %ROOT%\modules CacheDir %ROOT%\data Pidfile %ROOT%\data\nxlog.pid SpoolDir %ROOT%\data LogFile %ROOT%\data\nxlog.log <Extension gelf> Module xm_gelf </Extension> <Input in> # Use ’im_mseventlog’ for Windows XP, 2000 and 2003 Module im_msvistalog # Uncomment the following to collect specific event logs only Query <QueryList>\ <Query Id="0">\ <Select Path="System">*</Select>\ <Select Path="Application">*</Select>\ <Select Path="Security">*</Select>\ </Query>\ </QueryList> </Input> <Output out> Module om_udp Host 10.60.10.62 Port 12201 OutputType GELF </Output> <Route r> Path in => out </Route> _______________________________________________________ Now I put a REM statement at the beginning of the file # Just capturing security logs The service won't start. If I rem out Application and System path, it won't start. Any suggestions?