Ask questions. Get answers. Find technical product solutions from passionate experts in the NXLog community.

om_http unexpected data from server causes nxlog service to crash
I have the usual error "Unexpected data from server ..." using om_http to send Windows event logs through Nginx to Redis. The problem is that when randomly this error happens ( once every 5 days) the service unexpectently stops ! From the source code I can understand that NXLog receives a response which cannot map to a correnponding request. It has a response with an appropriate length but the corresponding request is null! Possibly is a bad logic implementation, or even a network problem that could cause request/response mismatch.I really don't know. But I also really can't understand why NxLog preferres to die instead of  just return from the function in case this happens and not execute the code following the condition that drives the service to crash at the end. Stelios

bourazaniss created
Replies: 1
View post »
last updated
Exec if $EventID NOT IN - Question on placement in config.
Would this be the correct placement to add the filter event ID string?  Should anything esle be commented out? #Windows Event Logging of Security,System and Application Logs   Module      im_mseventlog   Exec to_syslog_snare(); Exec if $EventID NOT IN (528, 529, 567, 592, 601, 602, 608, 612, 636, 7034, 7035, 7036, 7040, 4097, 64004, 2, 3005) drop(); </Input>      

dmm3369 created
Replies: 1
View post »
last updated
NXLog SeverityValue for Windows Events
When NXLog ships a Windows event, it appears to be changing the Windows original severity level, and replacing it with SeverityValue and Severity with different values.   What is the mapping of these values?  If Windows has severity values, with "Level" being 4 for Information, 3 for Warning, 2 for Error and 1 for Critical, what is the nxlog created SeverityValue?   Also, I couldn't find an explaination of why this value is changing.  

cybergoof created
Replies: 1
View post »
last updated
Log filtering by event ID?
Hi, We are testing NXlog to ship the security logs to our security team.  We are using XP embedded and it working fine.  Security is asking us to only send specific event ID's.  I have looked at the documentation and it appears that we may not be able to do this with XP.  I was going to do the custom view but XP does not support that. We are supported on the embedded version of XP. My question.  Has anyone tried to do this with XP or is it even possible? Thanks, David Martin  

dmm3369 created
Replies: 1
View post »
last updated
Nxlog module
Hi,  I am doing a rule which detects me are making multiple responses of ICMP from the same IP in which there are diferent IPs.   The problem that i have is that don't write in the output with raw_event and file_write. In file_write I put the "otro" file.  Do you know it can be?   Antonio.   File nxlog.conf:   <Extension fileop>     Module      xm_fileop </Extension> <Input in4>     Module    im_file     File    "/home/antonio/Descargas/sn"     SavePos TRUE     Exec    if ($raw_event =~ /^\d\d:\d\d:\d\d.(.+)/) {     \                     $Message = $1;            \                     $raw_event = $Message;       \                 }     exec if $Message =~ /IP (\S{1,}) > \S{1,}:/ $IP=$1; </Input> <Input internal>     Module    im_internal     Exec    $raw_event = $Message; </Input> <Output out4>     Module    om_file     File    "/home/antonio/Descargas/nx" </Output> <Processor evcorr>     Module       pm_evcorr     <Thresholded>                 Condition  $Message =~ /^ICMP echo reply/             Threshold  3            Interval   120     Context $IP     Exec       $raw_event = "3 ECHO REPLY packets from host $IP";     Exec        file_write("/home/antonio/Descargas/otro", "3 ECHO REPLY packets from host $IP");     </Thresholded> </Processor> <Route 4> Path    in4, internal => evcorr => out4 </Route>   File sn:   17:11:33.953173 IP mad01s25-in-f3.1e100.net > 192.168.20.153: ICMP echo reply, id 62478, seq 2, length 64 17:11:33.953173 IP mad01s25-in-f3.1e100.net > 192.168.20.153: ICMP echo reply, id 62478, seq 2, length 64 17:11:33.953173 IP mad01s25-in-f3.1e100.net > 192.168.20.153: ICMP echo reply, id 62478, seq 2, length 64 17:11:33.953173 IP mad01s25-in-f3.1e100.net > 192.168.20.153: ICMP echo reply, id 62478, seq 2, length 64 17:11:33.953173 IP mad01s25-in-f3.1e100.net > 192.168.20.153: ICMP echo reply, id 62478, seq 2, length 64 17:11:33.953173 IP mad01s25-in-f3.1e100.net > 192.168.20.153: ICMP echo reply, id 62478, seq 2, length 64 17:11:33.953173 IP mad01s25-in-f3.1e100.net > 192.168.20.153: ICMP echo reply, id 62478, seq 2, length 64 17:11:33.953173 IP mad01s25-in-f3.1e100.net > 192.168.20.153: ICMP echo reply, id 62478, seq 2, length 64   File nx:   inode changed for '/home/antonio/Descargas/sn': reopening possibly rotated file 953173 IP mad01s25-in-f3.1e100.net > 192.168.20.153: ICMP echo reply, id 62478, seq 2, length 64 953173 IP mad01s25-in-f3.1e100.net > 192.168.20.153: ICMP echo reply, id 62478, seq 2, length 64 953173 IP mad01s25-in-f3.1e100.net > 192.168.20.153: ICMP echo reply, id 62478, seq 2, length 64 953173 IP mad01s25-in-f3.1e100.net > 192.168.20.153: ICMP echo reply, id 62478, seq 2, length 64 953173 IP mad01s25-in-f3.1e100.net > 192.168.20.153: ICMP echo reply, id 62478, seq 2, length 64 953173 IP mad01s25-in-f3.1e100.net > 192.168.20.153: ICMP echo reply, id 62478, seq 2, length 64 953173 IP mad01s25-in-f3.1e100.net > 192.168.20.153: ICMP echo reply, id 62478, seq 2, length 64 953173 IP mad01s25-in-f3.1e100.net > 192.168.20.153: ICMP echo reply, id 62478, seq 2, length 64   File otro:   Thanks :))

antonio_cuestag created
NXlog Implementation
Hi all! Im new and would really appreciate any assistance/feedback on implementation... I'm trying to implement Audit Reduction in my environment which includes (4) Windows 7 boxes , (2) RHEL 7 boxes (SYSLOG) and (1) RHEL 6 box (SYSLOG)... We have all the logs writing to one of the Windows 7 workstations. The logs need to be manually pulled off and put in another location. This environment is replicated 7 times (7 systems with the same configuration with 7 appliances each) They ultimately get to a Dropbox that forwards them off to an ElasticSearch/Kibana instantiation....So 49~ devices/appliances reporting. I would like to know if the Community Edition would work or if I need to purchase Enterprise? Does NXlog need to be installed on each appliance? Thanks!

aimball44 created
How to delete input files after reading ?
Hi Everyone, New forum user, so sorry if I do not respect some uses rules…and I’m French by the way, so sorry about my bad English level too…;-) I use Nxlog in the following context : A machine A send periodically a file on a machine B (same file name) Machine B is used like a first level of log centralization. Nxlog is installed on it Then, Nxlog send log to Machine C who host Logstash I’m looking for a process who remove input files after end of reading and processing, but I’m not sure about how process “in-out” operate, is-it after each line or after a n entire file reading ? How can I test a “end of reading” condition ? Right now, I’m going to test something like that : <Extension fileop>  Module xm_fileop </Extension> <Input in>     Module im_file     SavePos TRUE     ReadFromLast TRUE     PollInterval 900 # 15mn     File "/var/log/inpufile*.log"  Exec $InputFileName1 = file_name() </Input> <Output logstash>     Module om_tcp  Port 6002 Host 10.x.y.z Exec file_remove($InputFileName1); </Output> <Route 1>     Path in => logstash </Route> Is it correct ? Thanks in advance for your help

RemyVeo created
Replies: 1
View post »
last updated
NxLog marked as virus
Hi there,   Has any of you encountered this virus warning? The scan was done using TrendMicro HouseCall. Is my nxlog infected? Thanks for your help

ChristianPangestu created
Replies: 1
View post »
last updated
nxlog crashes continuously
nxlog is crashing with the following error:   Faulting application name: nxlog.exe, version: 0.0.0.0, time stamp: 0x5666d55e Faulting module name: ntdll.dll, version: 6.3.9600.18202, time stamp: 0x569e72c5 Exception code: 0xc0000005 Fault offset: 0x000192cb Faulting process id: 0x1b60 Faulting application start time: 0x01d18540c8297bd3 Faulting application path: C:\Program Files (x86)\nxlog\nxlog.exe Faulting module path: C:\Windows\SYSTEM32\ntdll.dll Report Id: 06d89363-f134-11e5-80dd-005056a619fb Faulting package full name: Faulting package-relative application ID:   config file:   define ROOT C:\Program Files (x86)\nxlog Moduledir %ROOT%\modules CacheDir %ROOT%\data Pidfile %ROOT%\data\nxlog.pid SpoolDir %ROOT%\data LogFile %ROOT%\data\nxlog.log <Extension json> Module xm_json </Extension> <Input eventlog> # Use 'im_mseventlog' for Windows XP and 2003 Module im_msvistalog SavePos FALSE ReadFromLast FALSE Query <QueryList>\ <Query Id="0">\ <Select Path="Pool2PdfCreator.Produce">*</Select>\ </Query>\ </QueryList> </Input> <Output out> Module om_tcp Host 10.36.52.62 Port 12201 Exec $EventTime = integer($EventTime) / 1000000; to_json(); Exec log_info("RecordNumber: " + $RecordNumber); </Output> <Route r> Path eventlog => out </Route>   (during troubleshooting, I have narrowed down the query to one eventsource and also added Exec log_info("RecordNumber: " + $RecordNumber); to be able to pinpoint the exact entry that causes the issue. I was able to locate the entry that causes the crash. The strange thing is, it sometimes goes through, most of the times causes crash. I am not comfortable with sharing the entry here but I can send it via e-mail. This definitely looks like a bug.   nxlog version: nxlog-ce-2.9.1504

achechen created
Mimic rsyslog output on Ubuntu
Using nxlog in front of logstash on a server.  On the same user, I want to use nxlog to replace rsyslog.  Seems pretty simple.  The only issue is the file format is slightly different than the what rsyslog outputs.  I see: <78>May 6 13:50:01 CRON[19454]: (root) CMD ( /opt/observium/discovery.php -h new >> /dev/null 2>&1) vs: Jul 16 18:00:01 monitor01 CRON[6871]: (root) CMD ( /opt/observium/poller-wrapper.py 16 >> /dev/null 2>&1) Main difference is the <NN> at the beginning of the line and the missing hostname (monitor01).  Here is my .conf: <Input in_uds> Module im_uds UDS /dev/log </Input> <Input in_kernel> Module im_kernel </Input> <Output out> Module om_file File "/var/log/syslog" </Output> <Route local_route> Path in_uds, in_kernel => out </Route> Is there a simple change I can make to get the desired format?  Thanks.  -- Bud    

bbach created
Replies: 1
View post »
last updated
Nxlog not handling winevent TimeCreated
I'm using NXlog to ship windows event logs to an ELK stack.  I need to preserve the datetime when the event happened <TimeCreated SystemTime=> that is stored in the event log.   However, the NXLog that is shipped doesn't preserve <TimeCreated SystemTime>, which I assume is because its invalid json.  How can I preserve this in my nxlog.conf?  Otherwise, I'm stuck with EventTime, which appears to be the datetime of when nxlog processes the event, not when the event happened.   How do I handle this?

cybergoof created
Replies: 1
View post »
last updated
Issues with data upload to ElasticSearch
Hello everyone, This is my first time posting in this community forum, so any help would be greatly appreciated. I've been working with NxLog and ElasticSearch for a few months now and I've had mostly no issues with it until very recently, where a new ElasticSearch index was created in order to accomodate the new structure of our logs. With that, we also had to update our existing nxlog.conf file. We have about 3 different ElasticSearch endpoints with the same setup that we currently work with, and at one point during the week we had run out of storage space. After increasing the storage size for all 3 endpoints, two environments appeared to continue sending new information up to ElasticSearch with no further problems. However, the third environment's NxLog services appear to be stuck in both existing AWS instances and newly created instances, repeating the following NxLog log entry over and over: 2016-04-29 15:33:12 INFO connecting to search-stage-logging-udf7h4lq2bsm245ciawp2stcvu.us-east-1.es.amazonaws.com:80 2016-04-29 15:33:12 INFO reconnecting in 1 seconds 2016-04-29 15:33:12 ERROR ### PANIC at line 2456 in module.c/nx_module_pollset_add_socket(): "failed to add descriptor to pollset: Not enough space ; [cannot dump backtrace on this platform]" ### This was the log entry that we initially discovered that alerted us we had run out of space in ElasticSearch. However, the ElasticSearch dashboard does not show a lack of space anymore, so it's a bit confusing why NxLog would continue to output log entries in the other two environments Basically, I have two questions: 1. Is this a type of scenario where the NxLog service's working state has been stuck unable to see there is space available? Or does the fault lie with ElasticSearch not showing storage space correctly? 2. If the Nxlog service has been stuck in this state, is there a configuration or some other automated procedure for NxLog to get the service to restart itself on multiple failures?   If anyone has gone through a similar experience, any tips would be greatly appreciated. Thank you for your time.

jppacheco created
Replies: 1
View post »
last updated
Sysmon Parsing Problem
The article on structured logging (https://nxlog.co/why-use-structured-logging) shows how you should use structured logging so that changes in log format is minimized.  The example of the sysmon event, process creates, shows what I think is a bug in NXLog.   The ProcessID in the "Message" is the ProcessID (25848) of the new process that sysmon sees created.  However, in the structured NXLog key/values, the ProcessID is that of sysmon itself (1680)   The only way to get the ProcessID of the process sysmon observed created, is to use regular expressions.  Can you verify that this is a bug in NXLog?

cybergoof created
Replies: 1
View post »
last updated
IIS7 W3C log parsing fails
Hello Guys, I have a question about NXLog IIS7 W3C logs. I set NXLog up and it works basicaly but the NXLog logs are full with error messages like this: 2016-04-26 09:46:36 ERROR if-else failed at line 64, character 257 in C:\Program Files (x86)\nxlog\conf\nxlog.conf. statement execution has been aborted; procedure 'parse_csv' failed at line 64, character 113 in C:\Program Files (x86)\nxlog\conf\nxlog.conf. statement execution has been aborted; couldn't parse integer: language=UK&region=802&idfwbonavigation=180173.2 It look, the IIS logs contain an "=" sign in the $cs-uri-query field, and NXLog wants integer after the "=", however the field set to be string in the config file. Have you ever met anything like this? And if yes, what could be the solution?   NXLog extension and input config: <Extension exiis>     Module        xm_csv     Fields        $date $time $s-ip $cs-method $cs-uri-stem $cs-uri-query $s-port $cs-username $c-ip $cs(User-Agent) $cs(Cookie) $cs(Referer) $sc-status $sc-substatus $sc-win32-status $sc-bytes $cs-bytes $time-taken     FieldTypes    string, string, string, string, string, string, integer, string, string, string, string, string, integer, integer, integer, integer, integer, integer     Delimiter    ' '     QuoteChar    '"'     EscapeControl    FALSE     UndefValue    - </Extension> <Input IIS>     Module        im_file     File        "D:\\Logs\\IIS\\W3SVC300\\u_ex*"     SavePos        TRUE     Recursive    FALSE     Exec if $raw_event =~ /^#/ drop();                                                                  \         else                                                                                                         \         {                                                                                                              \             exiis->parse_csv();                                                                                \             $EventTime = parsedate($date + " " + $time);                                          \             $EventTime = strftime($EventTime, "%Y-%m-%dT%H:%M:%SZ");            \         } </Input>  

CSimpiFoN created
Replies: 2
View post »
last updated
How to collect only windows security logs
Hello, I'm kind of new to nxlog, but is it possible to collect only Windows security logs ?

mulail created
Replies: 1
View post »
last updated
how to Perform Windows NXLog.conf
Hello, I do admit I am totally lost about NXLog.conf for Windows 2K12 R2 machines. The purpose is to filter some EventIDs from Security Event Log, for that I tried the below nslog.conf : <Extension _syslog>     Module      xm_syslog </Extension> <Input>    Module      im_msvistalog # For windows 2003 and earlier use the following: #   Module      im_mseventlog     Exec if ($EventID == 4634 or $EventID == 4624 or $EventID == 4672 or $EventID == 4801 or $EventID == 64 or $EventID == 7036 or $EventID == 7040) drop();\     else\     {\         parse_syslog_ietf();\         $Message = $FileName + ": " + $Message;\         $SyslogFacility = syslog_facility_string(22);\         $SyslogFacilityValue = syslog_facility_value("local6");\         if ( $EventType == "INFO" ) $SyslogSeverityValue = 6;\         if ( $EventType == "WARNING" ) $SyslogSeverityValue = 4;\         if ( $EventType == "ERROR" ) $SyslogSeverityValue = 3;\     } </Input> <Output out>     Module      om_udp     Host        10.1.1.39     Port        1514     Exec        to_syslog_snare(); </Output> <Route 1>     Path internal, eventlog => out </Route>     Unfortunately despite the host and port are well set it doesn't work, and I also have these messages from nxlog.log : xxxxxx WARNING no routes defined! xxxxxx WARNING not starting unused module out   I would really appreciate any help

CBush created
Replies: 1
View post »
last updated
Is it possible to compile: im_msvistalog on Linux to ingest saved log files?
Am I missing something? I see the source code, but no configuration options to compile the module under linux.  

cbitterfield created
Replies: 1
View post »
last updated
Request a simple example for processing AWS Logs from the S3 Bucket.
I am trying to parse AWS S3 Logs. They are in JSON format (One line no CR/LF) and Gzip'd. I need to ingest these into syslog TCP or UDP (Testing with file out) I can't get a reliable working nxlog.conf that will process the JSON files. NXLOG define ROOT /usr/local/libexec/nxlog/ Pidfile /var/run/nxlog.pid LogFile ./nxlog.log define WORK /Users/cbitterfield/awslogs-project SpoolDir %WORK%/data CacheDir %WORK%/data LogLevel DEBUG Module xm_syslog Module xm_json Module im_file File "%WORK%/data19/*.json" Exec parse_json(); # Dump $raw_event Exec to_syslog_bsd(); SavePos FALSE ReadFromLast False Module om_file File "./output" Path in => out Yields the following errors and no output. 2016-04-10 22:13:00 DEBUG '^KE<F4>t^G<C7>C^D' does not match wildcard '859121128579_CloudTrail_ap-northeast-.json' 2016-04-10 22:13:00 DEBUG checking '^KE<F4>t^G<C7>C^D' against wildcard '859121128579_CloudTrail_ap-northeast-.json': 2016-04-10 22:13:00 DEBUG '^KE<F4>t^G<C7>C^D' does not match wildcard '859121128579_CloudTrail_ap-northeast-.json' 2016-04-10 22:13:00 DEBUG checking '^KE<F4>t^G<C7>C^D' against wildcard '859121128579_CloudTrail_ap-northeast-.json': 2016-04-10 22:13:00 DEBUG '^KE<F4>t^G<C7>C^D' does not match wildcard '859121128579_CloudTrail_ap-northeast-.json' 2016-04-10 22:13:00 DEBUG checking '^KE<F4>t^G<C7>C^D' against wildcard '859121128579_CloudTrail_ap-northeast-.json':

cbitterfield created
Replies: 2
View post »
last updated
DB2 database support?
I am trying to read the table contents from DB2 table and send it as syslog. Is this possible with im_dbi module?

ramajith created
Replies: 1
View post »
last updated
DBI input/output modules missing in windows installed location
I was trying to test the database input module of NXlog in windows environment. After installed, I have verified the modules folder. I can see all other modules, but im_dbi and om_dbi are missing in the 2.8.x and 2.9.x versions, any idea please?

ramajith created
Replies: 1
View post »
last updated