Is im_odbc available for NXLog community version?

Hello, I'm having trouble sending logs in json format generated from a command. The command generate (one json per line, json syntax checked with jsonlint and all json lines are ok. I send the input log to output file per debug and the json is ok) {"metricset":{"module":"system","name":"memory"},"system":{"memory":{"total`":4294967296,"free":1709912064,"used":{"bytes":2585055232,"pct":60.19},"swap":{"total":2046,"free":2012,"used":{"bytes":34,"pct":1.66}}}}} {"metricset":{"module":"system","name":"cpu"},"system":{"cpu":{"cores": 1,"idle":{"pct":99},"irq":{"pct":0},"system":{"pct":0},"user":{"pct":1}}}} When nxlog send the data to logstash with om_tcp, logstash receive (review the system field, it's not the same as the one generated in the input) Oct 01 03:04:54 elk logstash[43975]: { Oct 01 03:04:54 elk logstash[43975]: "SourceModuleName" => "counters", Oct 01 03:04:54 elk logstash[43975]: "system" => "{&quot;cpu&quot;:{&quot;cores&quot;:1,&quot;idle&quot;:{&quot;pct&quot;:99}&quot;irq&quot;:{&quot;pct&quot;:0}&quot;system&quot;:{&quot;pct&quot;:0}&quot;user&quot;:{&quot;pct&quot;:1}", Oct 01 03:04:54 elk logstash[43975]: "@timestamp" => 2019-10-01T01:04:54.022Z, Oct 01 03:04:54 elk logstash[43975]: "SourceModuleType" => "im_exec", Oct 01 03:04:54 elk logstash[43975]: "port" => 3150, Oct 01 03:04:54 elk logstash[43975]: "@metadata" => { Oct 01 03:04:54 elk logstash[43975]: "input" => "tcp", Oct 01 03:04:54 elk logstash[43975]: "week" => "2019.10-40", Oct 01 03:04:54 elk logstash[43975]: "month" => "2019.10", Oct 01 03:04:54 elk logstash[43975]: "stdout" => "true", Oct 01 03:04:54 elk logstash[43975]: "index" => "in-test-nxlog-2019.10-40", Oct 01 03:04:54 elk logstash[43975]: "day" => "2019.10.01" Oct 01 03:04:54 elk logstash[43975]: }, Oct 01 03:04:54 elk logstash[43975]: "@version" => "1", Oct 01 03:04:54 elk logstash[43975]: "metricset" => "{&quot;module&quot;:system,&quot;name&quot;:cpu}", Oct 01 03:04:54 elk logstash[43975]: "client" => { Oct 01 03:04:54 elk logstash[43975]: "ip" => "10.71.218.62" Oct 01 03:04:54 elk logstash[43975]: }, Oct 01 03:04:54 elk logstash[43975]: "EventReceivedTime" => "2019-10-01 03:03:58" Oct 01 03:04:54 elk logstash[43975]: } If we add the to_json() exec in the input configuration, the debug output breaks in the same way. So, I think that the to_json procedure have a bug with nested json object. <Extension json> Module xm_json </Extension> <Extension charconv> Module xm_charconv </Extension> powershell to recover counter metrics from a windows 2003 server at the same way as metrcbeat do it <Input counters> Module im_exec InputType LineBased Command "%SYSTEMROOT%\System32\WindowsPowerShell\v1.0\powershell.exe" Arg "-ExecutionPolicy" Arg "Bypass" Arg "-NoProfile" Arg "-File" Arg %ROOT%\modules\input\counters.ps1 Arg -interval Arg 60 Exec parse_json(); </Input> <Output tcp> Module om_tcp Host elk Port 5045 Exec to_json(); </Output> <Output debug> Module om_file CreateDir TRUE File "C:\Program Files\nxlog\data\debug.log" # if we uncomment this line, the debug file breaks at the same way #Exec to_json(); </Output> <Route 1> Path counters => tcp </Route> <Route 2> Path counters => debug </Route>

Hello, I am evaluating NXLog using the Community Edition. I created input as shown below to monitor certain Windows events and forward them via email. Everything is working as expected except that $Message or $raw_event variables always return word "true" instead of actual details about the event. Is this a limitation of the CE or am I doing something else wrong? Many thanks in advance for your assistance. <Input eventlog> Module im_msvistalog <QueryXML> <QueryList> <Query Id="0" Path="Application"> <Select Path="Application">[System[Provider[@Name='Symantec AntiVirus' or @Name='Symantec Network Protection']]]</Select> </Query> </QueryList> </QueryXML> <Exec> exec("c:/utils/mailsend.exe", "-to", "info@***.com", "-body", $raw_event, "-subject", "Symantec EPP Alert"); </Exec> </Input>

Hello, I have a windows app that send errors to windows eventlog and I need monitoring this. The event structure is this: <Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event"> <System> <Provider Name="AppName" /> <EventID Qualifiers="16384">1</EventID> <Level>4</Level> <Task>0</Task> <Keywords>0x80000000000000</Keywords> <TimeCreated SystemTime="2019-08-02T10:43:01.000000000Z" /> <EventRecordID>91524</EventRecordID> <Channel>Application</Channel> <Computer>server.domain.es</Computer> <Security /> </System> <EventData> <Data>Full description error</Data> </EventData> </Event> The problem is that when I send this event to Graylog for monitoring, I can't see the contain of EventData that its the most important. I'm reading that there are some problems with data without named. Is there any solution? Thanks

Hello, I have some IIS logs that contain a single " and I am getting errors when I try to use parse_csv saying the data is invalid csv input. As soon as I take out the single ", the log sends fine. What can I do to resolve this issue?

Hello I'm trying to send Windows DNS logs through NXLog, but i'm having a problem. I followed the documentation and ended up with the following config file. Events seem to match the regex, but then i can't seem to use any of the named group names ($Date, $QuestionName, ... any). I tried to log_info(); but it always shows up as an empty string in the log file : This: log_info('q is ' + $QuestionName); Shows up in logs as "q is" (and nothing else) Anyone knows what i'm doing wrong ? I don't see "no match" in my logfile so i guess events always match the EVENT_REGEX. Been struggling with this for 24 hours .. even tried unnamed capture groups but also the $0, $1... always show empty. (config file also at https://pastebin.com/s4CaJg9k in case of problems) Panic Soft #NoFreeOnExit TRUE define ROOT C:\Program Files (x86)\nxlog define CERTDIR %ROOT%\cert define CONFDIR %ROOT%\conf define LOGDIR %ROOT%\data define LOGFILE %LOGDIR%\nxlog.log LogFile %LOGFILE% Moduledir %ROOT%\modules CacheDir %ROOT%\data Pidfile %ROOT%\data\nxlog.pid SpoolDir %ROOT%\data Example data : #14-09-19 09:20:39 0B64 PACKET 0000005487B8E130 UDP Rcv 172.30.2.30 486a Q [0001 D NOERROR] AAAA (7)outlook(9)office365(3)com(8)transfer(2)be(0) #14-09-19 09:20:39 0B60 PACKET 0000005487FAC120 UDP Rcv 172.30.1.38 9b88 Q [0001 D NOERROR] AAAA (7)outlook(9)office365(3)com(0) define EVENT_REGEX /(?x)(?<Date>\d+(?:-\d+){2})\s (?<Time>\d+(?::\d+){2})\s (?<ThreadId>\w+)\s+ (?<Context>\w+)\s+ (?<InternalPacketIdentifier>[[:xdigit:]]+)\s+ (?<Protocol>\w+)\s+ (?<SendReceiveIndicator>\w+)\s (?<RemoteIP>[[:xdigit:].:]+)\s+ (?<Xid>[[:xdigit:]]+)\s (?<QueryType>\s|R)\s (?<Opcode>[A-Z]|?)\s (?<QFlags>[(.?)])\s+ (?<QuestionType>\w+)\s+ (?<QuestionName>.)/ define EMPTY_EVENT_REGEX /(^$|^\s+$)/ define DOMAIN_REGEX /(\d+)([\w-]+)(\d+)([\w-]+)/ define SUBDOMAIN_REGEX /(\d+)([\w-]+)(\d+)([\w-]+)(\d+)(\w+)/ define NOT_STARTING_WITH_DATE_REGEX /^(?!\d+-\d+-\d+).+/ define QFLAGS_REGEX /(?x)(?<FlagsHex>\d+)\s+ (?<FlagsCharCodes>\s+|([A-Z]{2}|[A-Z]))\s+ (?<ResponseCode>\w+)/ <Extension _json> Module xm_json </Extension> <Input in> Module im_file File 'C:\dnslog\dns.log' <Exec> # Drop entries that have empty lines if $raw_event =~ %EMPTY_EVENT_REGEX% drop(); # Drop entries not starting with date if $raw_event =~ %NOT_STARTING_WITH_DATE_REGEX% drop(); # Split entries into fields & define regular entries if $raw_event =~ %EVENT_REGEX% { $Regular = TRUE; #$EventTime = parsedate($Date + " " + $Time); $Raw = $raw_event; #delete($date); #delete($time); if $FlagsCharCodes =~ /^\s+$/ delete($FlagsCharCodes ); # Convert domains from (8)mydomain(1)com to mydomain.com if $QuestionName =~ %DOMAIN_REGEX% $QuestionName = $1 + "." + $2; # Convert domains from (8)sub(8)mydomain(1)com to sub.mydomain.com if $QuestionName =~ %SUBDOMAIN_REGEX% $QuestionName = $1 + "." + $2 + "." +$3; # Set query flags if $QFlags =~ %QFLAGS_REGEX% delete($QFlags); # Set the query type if $QueryType =~ %EMPTY_EVENT_REGEX% $QueryType = &quot;query&quot;; else $QueryType = &quot;response&quot;; log_info('q is ' + $QuestionName); } else { $Regular = FALSE; $Raw = $raw_event; log_info(&quot;no match&quot;); } &lt;/Exec&gt; </Input> <Output out> Module om_file Exec to_json(); File 'C:\output-dns-traffic.json' </Output> <Route r1> path in => out </Route>

We installed version 2.10.2150 and are using the standard out of box config file to sent syslogs to clone. Anything we can do to reduce CPU consumption? Here is our config file but I have removed IP for our clone server: Panic Soft #NoFreeOnExit TRUE define ROOT C:\Program Files (x86)\nxlog define CERTDIR %ROOT%\cert define CONFDIR %ROOT%\conf define LOGDIR %ROOT%\data define LOGFILE %LOGDIR%\nxlog.log LogFile %LOGFILE% Moduledir %ROOT%\modules CacheDir %ROOT%\data Pidfile %ROOT%\data\nxlog.pid SpoolDir %ROOT%\data <Extension _syslog> Module xm_syslog </Extension> Windows Event Log <Input eventlog> Module im_msvistalog </Input> <Output tcp> Module om_tcp Host xx.xx.xx.xx Port 514 Exec to_syslog_snare(); </Output> <Route eventlog_to_tcp> Path eventlog => tcp </Route> <Extension _charconv> Module xm_charconv AutodetectCharsets iso8859-2, utf-8, utf-16, utf-32 </Extension> <Extension _exec> Module xm_exec </Extension> <Extension _fileop> Module xm_fileop # Check the size of our log file hourly, rotate if larger than 5MB &lt;Schedule&gt; Every 1 hour Exec if (file_exists('%LOGFILE%') and \ (file_size('%LOGFILE%') &gt;= 5M)) \ file_cycle('%LOGFILE%', 8); &lt;/Schedule&gt; # Rotate our log file every week on Sunday at midnight &lt;Schedule&gt; When @weekly Exec if file_exists('%LOGFILE%') file_cycle('%LOGFILE%', 8); &lt;/Schedule&gt; </Extension>

Hi, I have many nxlog in my infrastructure that we just implement. I have a simple question. My nxlog config file do not show error when i do Nxlog.exe -f in command prompt. But the services is stopped. I would like to know if no data is forwarded at the moment, the service stay shutdown and open when he will need to send data ? Greetings,

Hi, I use module om_http for send events to host via https, but after start NXlog shows error: ERROR SSL certificate verification failed: self signed certificate in certificate chain (err: 19) Thanks for your ideas!

HI, I am using nxlog CE on Wi2016 and have configured it to log data to windows event files. I am sending dummy syslog using kiwi syslog generator with random host from subnet. I can also see data from random hosts in a syslog watcher. but its not logged in files. why its not saving data for syslog traffic please? here is my config. #define ROOT C:\Program Files\nxlog define ROOT C:\Program Files (x86)\nxlog Moduledir %ROOT%\modules CacheDir %ROOT%\data Pidfile %ROOT%\data\nxlog.pid SpoolDir %ROOT%\data LogFile %ROOT%\data\nxlog.log <Extension gelf> Module xm_gelf </Extension> <Extension json> Module xm_json </Extension> <Extension syslog> Module xm_syslog </Extension> <Input in1> Module im_tcp Host 10.43.9.220 Port 514 </Input> <Input in2> Module im_udp Host 0.0.0.0 Port 514 </Input> <Input in> Module im_msvistalog Exec $Message = to_json(); </Input> <Output out> Module om_file CreateDir TRUE File 'C:\nxlog\Syslog' + "_" + strftime(now(),"%Y-%m-%d") + ".log" OutputType LineBased </Output> <Route R1> Path in1 , in2 , in => out </Route> Any help is appreciated.

Hi Team, Recently I started testing NXLog and was tryingto simulate log forwarding to other syslog servers. My logs are stored in *.log files and I want to forward them to another syslog destination. But after so many attempts, I still fail, and my logs are not forwarded. I also tried writing to another file using om_file but that does not help me as well. The NXLog's logs are of not much help, as It is stuck with just "Connecting to X.X.X.X:514 and never does anything ahead of it. IT does not show any warning / error as well. How do I investigate, what went wrong. I am on Ubuntu 16.04 with NXLog CE 2.10.2150 downloaded from this portal. Below is my configuration, <Input infile1> Module im_file File "/opt/logs/pix.log" InputType LineBased </Input> <Output outfile1> Module om_file CreateDir TRUE File "/opt/logs/output.log" </Output> <Output outtcp1> Module om_tcp Host X.X.X.X Port 514 </Output> <Route r1> path infile1 => outtcp1, outfile1 </Route> I have checked on the network side, did Telnet (for TCP) and NC (for UDP) everything works fine, even rsyslog is able to forward data but NXLog fails.

I would like to send IIS v8.5 logs over to a Linux syslog server. I have all parts installed, but need help with the nxlog agent configuration on the IIS server (Win2012 R2). My current configuration is attached. There are errors in the Win2012 nxlog agent's log file, and I am unable to fix them all, they are attached as well. I had to comment out parts like writing to a local file in order to get the agent running. I would like to have that work as well. It created the file but it's empty. Config (errors are further below): ## Please set the ROOT to the folder your nxlog was installed into, ## otherwise it will not start. #define ROOT C:\Program Files\nxlog define ROOT C:\Program Files (x86)\nxlog define CERTDIR %ROOT%\cert Moduledir %ROOT%\modules CacheDir %ROOT%\data Pidfile %ROOT%\data\nxlog.pid SpoolDir %ROOT%\data LogFile %ROOT%\data\nxlog.log <Extension syslog> Module xm_syslog </Extension> <Extension fileop># Module xm_fileop </Extension> <Extension w3c_parser> Module xm_csv Fields date, time, s-ip, cs-method, cs-uri-stem, cs-uri-query, \ s-port, cs-username, c-ip, cs(User-Agent), cs(Referer), \ sc-status, sc-substatus, sc-win32-status, time-taken FieldTypes string, string, string, string, string, string, integer, \ string, string, string, string, integer, integer, integer, \ integer Delimiter ' ' EscapeChar '"' QuoteChar '"' EscapeControl FALSE UndefValue - </Extension> <Input iis_w3c> Module im_file File "C:\\inetpub\\logs\\LogFiles\\W3SVC1\\\\*.log" <Exec> if $raw_event =~ /^#/ drop(); else { w3c_parser->parse_csv(); $EventTime = parsedate($date + "T" + $time + ".000Z"); } </Exec> </Input> #<Output out_file_iis> # Module om_tcp # File 'C:\outputiis.log' # Exec to_syslog_bsd(); #</Output> <Output out_tcp> Module om_tcp Host 10.0.3.163 Port 514 Exec to_syslog_bsd(); </Output> <Route send_iis_to_syslog_server> Path iis_w3c => out_tcp </Route> <Route iis> Path iis_w3c => out_file_iis </Route> <Extension json> Module xm_json </Extension> <Extension charconv> Module xm_charconv AutodetectCharsets utf-8, euc-jp, utf-16, utf-32, iso8859-2, ucs-2le </Extension> ########################################## ## NXLOG INTERNAL LOG ## ########################################## # Nxlog internal logs - Recommended to keep this turned ON so error(s)/Issues with NXLog are reported. <Input internal> Module im_internal Exec $Hostname = hostname_fqdn(); Exec $EventReceivedTime = integer($EventReceivedTime) / 1000000; to_json(); </Input> ########################################## ## FLAT FILES ## ########################################## ## http://nxlog-ce.sourceforge.net/nxlog-docs/en/nxlog-reference-manual.html#im_file ## Input to watch a file of your choosing. After Input, name it whatever you want to describe that NXLog ## is pulling, then add that name to the path in Route 1 after eventlog. Can be separated for filtering diff logs. ## After setting the Message as raw_event this converts the message to UTF-8, drops empty messages ## removes extra whitespace, grabs the file name as LogFile, adds the FQDN, and deletes a useless var ## Pulls all logfiles from the default ITS Log Location ## It is Recommended to LEAVE THIS ENABLED ## Ensure that "ITS_Logs" is specified in the correct Route at the bottom for output <Input ITS_Logs> Module im_file File "C:\\ITS\\Logs\\\\*.log" SavePos TRUE Recursive TRUE Exec $Message = $raw_event; Exec $Message = convert($Message, "ucs-2le", "utf-8"); Exec if $Message == '' drop(); Exec if $Message =~ s/^\s+//g log_debug("whitespace removed"); Exec if file_name() =~ /([^\\]+)$/ $LogFile = $1; Exec $Hostname = hostname_fqdn(); Exec delete($SourceModuleType); Exec $EventTime = $EventReceivedTime; Exec $EventReceivedTime = integer($EventReceivedTime) / 1000000; Exec to_json(); </Input> Error Message: 2019-09-11 12:08:56 ERROR if-else failed at line 46, character 9 in C:\Program Files (x86)\nxlog\conf\nxlog.conf. statement execution has been aborted; procedure 'parse_csv' failed at line 44, character 36 in C:\Program Files (x86)\nxlog\conf\nxlog.conf. statement execution has been aborted; cannot parse integer "/", invalid modifier: '/' 2019-09-11 12:09:26 ERROR last message repeated 5 times

Hello, I have the following nxlog configuration file: define ROOT C:\Program Files (x86)\nxlog define CERTDIR %ROOT%\cert define CONFDIR %ROOT%\conf define LOGDIR %ROOT%\data define LOGFILE %LOGDIR%\nxlog.log LogFile %LOGFILE% Moduledir %ROOT%\modules CacheDir %ROOT%\data Pidfile %ROOT%\data\nxlog.pid SpoolDir %ROOT%\data <Extension _syslog> Module xm_syslog </Extension> <Input in> Module im_msvistalog #Exec if not ($EventID IN (4624, 1102)) drop(); </input> <Output out> Module om_udp Host xx.xxx.xx Port 514 Exec to_syslog_snare(); </Output> <Route> Path in=>out </Route> I am sending windows log events to a syslog. I would like to know how do I send a "connector is ok" test message every 1 hour can you help me?

Hello NXLog folks!! I'm trying to send all powershell alerts to our logserver but I want to exclude those generated by a specific executable. Since the powershell transcription doesn't list the exe as $process, but instead in context info...how does one filter that out? In this case, anything coming from tsm.exe, or tsmv.exe or tsmv1.exe All insight is welcome Thanks TP Here:s a sample of the log: CommandInvocation(Set-StrictMode): "Set-StrictMode" ParameterBinding(Set-StrictMode): name="Version"; value="1.0" Context: Severity = Informational Host Name = Windows PowerShell ISE Host Host Version = 5.1.17134.858 Host ID = 8ae5c6dd-1af0-4e65-aeac-7a67be38f4e4 Host Application = C:\Program Files\TSM\TSM.exe Engine Version = 1.0 Runspace ID = f1c12215-0436-4e63-8bf2-2bfadf608c65 Pipeline ID = 385 Command Name = Set-StrictMode Command Type = Cmdlet Script Name = Command Path = Sequence Number = 53836 User = Connected User = Shell ID = Here's our Log selection snippet: <Input in> Module im_msvistalog Query <QueryList> <Query Id="0"> <Select Path="Security"></Select> <Select Path="System">[System/Level=4]</Select> <Select Path="Application"></Select> <Select Path="Setup"></Select> <Select Path='Windows PowerShell'></Select> <Select Path='Microsoft-Windows-AAD/Operational'></Select> <Select Path='Microsoft-Windows-Application-Experience/Program-Compatibility-Assistant'></Select> <Select Path='Microsoft-Windows-Application-Experience/Program-Telemetry'></Select> <Select Path='Microsoft-Windows-AppLocker/EXE and DLL'></Select> <Select Path='Microsoft-Windows-AppLocker/MSI and Script'></Select> <Select Path='Microsoft-Windows-AppLocker/Packaged app-Deployment'></Select> <Select Path='Microsoft-Windows-AppLocker/Packaged app-Execution'></Select> <Select Path="Microsoft-Windows-Sysmon/Operational"></Select> <Select Path="Microsoft-Windows-PowerShell/Admin"></Select> <Select Path="Microsoft-Windows-PowerShell/Operational"></Select> <Select Path='Microsoft-Windows-Windows Firewall With Advanced Security/Firewall'></Select> <Select Path='Microsoft-Windows-Windows Firewall With Advanced Security/FirewallVerbose'>*</Select> </Query> </QueryList> </Input>

Hi, I'm trying to use nxlog to to extract three metrics from a .set file. My OS is Windows 10. I edited the .conf file in "C:\Program Files (x86)\nxlog\conf" and it looks like this: Panic Soft #NoFreeOnExit TRUE define ROOT C:\Program Files (x86)\nxlog define CERTDIR %ROOT%\cert define CONFDIR %ROOT%\conf define LOGDIR %ROOT%\data define LOGFILE %LOGDIR%\nxlog.log LogFile %LOGFILE% Moduledir %ROOT%\modules CacheDir %ROOT%\data Pidfile %ROOT%\data\nxlog.pid SpoolDir %ROOT%\data <Extension json> Module xm_json </Extension> <Extension sikora> Module xm_set Fields $Nominal, $PlusTol, $Oval # fields of interest (metrics) FieldTypes string, string, string # type of variable Delimiter ; EscapeControl FALSE </Extension> <Input sikora_logs> Module im_file File "C:\Users\50051145\Desktop\nx_log\\*.set" # imput file #ReadFromLast True #Recursive True #SavePos True ReadFromLast False Recursive False SavePos False &lt;Exec&gt; if $raw_event =~ /^Macrosezione : (.+)/ # creazione variabile { # create_var('macrosection'); # set_var('macrosection', $1); # drop(); # } sikora-&gt;parse_set(); delete($EventReceivedTime); delete($SourceModuleName); delete($SourceModuleType); if $raw_event =~ /^Operatore / { # variable definition for the if not defined get_var('start_time') # timestamp { # log_debug(&quot;parsed_time: &quot; + strptime($time, &quot;%d/%m/%Y %I:%M:%S&quot;)); # create_var('start_time'); # set_var('start_time', strptime($time, &quot;%d/%m/%Y %I:%M:%S&quot;)); drop(); } else { if get_var('start_time') != strptime($time,&quot;%d/%m/%Y %I:%M:%S&quot;) { log_debug(&quot;old_time: &quot; + get_var('start_time')); log_debug(&quot;new_time: &quot;, $time); set_var('start_time', strptime($time,&quot;%d/%m/%Y %I:%M:%S&quot;)); drop(); } } } $time = (integer(get_var('start_time')) / 1000000 + integer($time)) * 1000; # formula to convert timestamp in milliseconds $pressure = integer($pressure); $macrosection = get_var('macrosection'); $nominal = get_var('nominal'); $type = get_var('type'); to_json(); &lt;/Exec&gt; </Input> <Output out> Module om_file # CreateDir TRUE # File "C:\Users\50051145\Desktop\temp" + $fileName # output file </Output> <Route 1> Path sikora_logs => out </Route> When I run the program I expect an output file in a folder on my desktop "C:\Users\50051145\Desktop\temp" but I get nothing. I checked the logs and I get this: 2019-09-10 18:20:34 ERROR Failed to load module from C:\Program Files (x86)\nxlog\modules\extension\xm_set.dll, The specified module could not be found. ; The specified module could not be found. 2019-09-10 18:20:34 ERROR Failed to load module from C:\Program Files (x86)\nxlog\modules\output\om_file #.dll, The specified module could not be found. ; The specified module could not be found. 2019-09-10 18:20:34 ERROR Couldn't parse Exec block at C:\Program Files (x86)\nxlog\conf\nxlog.conf:38; couldn't parse statement at line 45, character 28 in C:\Program Files (x86)\nxlog\conf\nxlog.conf; module sikora not found 2019-09-10 18:20:34 ERROR module 'sikora_logs' has configuration errors, not adding to route '1' at C:\Program Files (x86)\nxlog\conf\nxlog.conf:93 2019-09-10 18:20:34 ERROR module 'out' is not declared at C:\Program Files (x86)\nxlog\conf\nxlog.conf:93 2019-09-10 18:20:34 ERROR route 1 is not functional without input modules, ignored at C:\Program Files (x86)\nxlog\conf\nxlog.conf:93 2019-09-10 18:20:34 WARNING no routes defined! 2019-09-10 18:20:34 WARNING not starting unused module sikora_logs 2019-09-10 18:20:34 INFO nxlog-ce-2.10.2150 started 2019-09-11 11:10:27 WARNING stopping nxlog service 2019-09-11 11:10:27 WARNING nxlog-ce received a termination request signal, exiting... It appers that xm_set.dll library is missing, "The specified module could not be found". I found out in "C:\Program Files (x86)\nxlog\modules\extension" that nxlog doesn't come with a .set library. How can I add this library ? Thank you

Hello All, I have a huge .csv file, this contains logs from Service Now instance. I have the following nxlog configuration file. But when i run the parser, error file i generate exceeds more than 1 GB. The source file itself is only about 225 MB. Please set the ROOT to the folder your nxlog was installed into, otherwise it will not start. #define ROOT C:\Program Files\nxlog define ROOT C:\Program Files (x86)\nxlog Moduledir %ROOT%\modules CacheDir %ROOT%\data Pidfile %ROOT%\data\nxlog.pid SpoolDir %ROOT%\data LogFile %ROOT%\data\nxlog.log <Extension multiline> Module xm_multiline HeaderLine /^\d{1,2}/\d{1,2}/\d{4}\s/ </Extension> <Extension json> Module xm_json </Extension> <Extension csv> Module xm_csv Fields $Created,$Level,$Message,$Source,$CreatedBy FieldTypes string, string, string, string, string </Extension> <Extension syslog> Module xm_syslog </Extension> <Input eventlog> Module im_msvistalog ReadFromLast TRUE SavePos TRUE Query <QueryList> <Query Id="0"> <Select Path="Security">[System[(EventID=4768)]]</Select> <Select Path="Security">[System[(EventID=4769)]]</Select> <Select Path="Security">[System[(EventID=4771)]]</Select> <Select Path="Security">[System[(EventID=4624)]]</Select> <Select Path="Security">[System[(EventID=4625)]]</Select> <Select Path="Security">[System[(EventID=4634)]]</Select> <Select Path="Security">[System[(EventID=4647)]]</Select> <Select Path="Security">[System[(EventID=4648)]]</Select> <Select Path="Security">[System[(EventID=4656)]]</Select> <Select Path="Security">[System[(EventID=4719)]]</Select> <Select Path="Security">[System[(EventID=4720)]]</Select> <Select Path="Security">[System[(EventID=4722)]]</Select> <Select Path="Security">[System[(EventID=4723)]]</Select> <Select Path="Security">[System[(EventID=4724)]]</Select> <Select Path="Security">[System[(EventID=4725)]]</Select> <Select Path="Security">[System[(EventID=4726)]]</Select> <Select Path="Security">[System[(EventID=4727)]]</Select> <Select Path="Security">[System[(EventID=4728)]]</Select> <Select Path="Security">[System[(EventID=4729)]]</Select> <Select Path="Security">[System[(EventID=4730)]]</Select> <Select Path="Security">[System[(EventID=4731)]]</Select> <Select Path="Security">[System[(EventID=4732)]]</Select> <Select Path="Security">[System[(EventID=4733)]]</Select> <Select Path="Security">[System[(EventID=4734)]]</Select> <Select Path="Security">[System[(EventID=4735)]]</Select> <Select Path="Security">[System[(EventID=4737)]]</Select> <Select Path="Security">[System[(EventID=4738)]]</Select> <Select Path="Security">[System[(EventID=4739)]]</Select> <Select Path="Security">[System[(EventID=4741)]]</Select> <Select Path="Security">[System[(EventID=4742)]]</Select> <Select Path="Security">[System[(EventID=4743)]]</Select> <Select Path="System">[System[(EventID=7036)]]</Select> <Select Path="Application">[System[(EventID=18454)]]</Select> <Select Path="Application">[System[(EventID=18456)]]</Select> </Query> </QueryList> Exec to_json(); </Input> <Input filein> Module im_file File 'e:\ServiceNow\agent\export\snow_log.csv' InputType multiline ReadFromLast FALSE SavePos FALSE &lt;Exec&gt; # Ignore top line if $raw_event =~ /Created,Level,Message,Source,Created by/ drop(); if $raw_event =~ /Warning/ drop(); if $raw_event =~ /Information/ drop(); # Convert Newline and Tab to printed character #$raw_event =~ s/\R/\\r\\n/g; #$raw_event =~ s/\t/\\t/g; $raw_event = replace($raw_event,&quot;\n&quot;, &quot; &quot;); $raw_event = replace($raw_event,&quot;\r&quot;, &quot; &quot;); $raw_event = replace($raw_event,&quot;\t&quot;, &quot; &quot;); $SourceName = 'SNOWLogs'; # Parse $raw_event as CSV csv-&gt;parse_csv(); # Convert to JSON to_json(); &lt;/Exec&gt; </Input> <Output fileout> Module om_tcp Host logger Port 5140 #Exec to_syslog_bsd(); </Output> <Output out> Module om_tcp Host logger Port 5140 </Output> <Route r1> Path eventlog => out </Route> <Route parse_xml> Path filein => fileout </Route> For few lines it reads the data properly, but in some lines, it does not read the complete data. I am also trying to drop off unwanted data like information or warning, just to ensure i collect only Error information. But still it does not help. Error information from the file is very limited, so that I can reduce the amount of data to be ingested into ELK. Sample of Error messages as follows: Created Level Message 9/10/2019 3:00 Error java.lang.NullPointerException: java.lang.NullPointerException: 9/10/2019 1:07 Error java.lang.NullPointerException: java.lang.NullPointerException: 9/10/2019 1:04 Error [code]Failed Exploring CI Pattern, Pattern name: <b>Docker Pattern</b>, Process ID: <b>12887</b>, To Check Pattern Log Press <a href="$sw_horizontal_discovery_log.do?discoLogId=aeeb6a6d1b33fb40db5e43b4bd4bcb5a&amp;ipAddress=10.144.112.51&amp;pid=12887&amp;preExecution=&amp;host_sys_id=d3fd5bff87e04504065e00f509434dc2&amp;host_name=dm01db02.ga.ssga.root&amp;patternId=dd15665a7fe022004e83e2065f2a0c57&amp;patternName=Docker Pattern&amp;patternType=1&amp;isCloud=false"><u><b>Here</b></u></a>[/code] 9/10/2019 1:04 Error [code]Failed Exploring CI Pattern, Pattern name: <b>Docker Pattern</b>, Process ID: <b>12841</b>, To Check Pattern Log Press <a href="$sw_horizontal_discovery_log.do?discoLogId=aeeb6a6d1b33fb40db5e43b4bd4bcb11&amp;ipAddress=10.145.112.57&amp;pid=12841&amp;preExecution=&amp;host_sys_id=9ac8ef3887bc0904065e00f509434d22&amp;host_name=dm02db08.ga.ssga.root&amp;patternId=dd15665a7fe022004e83e2065f2a0c57&amp;patternName=Docker Pattern&amp;patternType=1&amp;isCloud=false"><u><b>Here</b></u></a>[/code] 9/10/2019 1:04 Error [code]Failed Exploring CI Pattern, Pattern name: <b>Docker Pattern</b>, Process ID: <b>13373</b>, To Check Pattern Log Press <a href="$sw_horizontal_discovery_log.do?discoLogId=eeeb6a6d1b33fb40db5e43b4bd4bcb41&amp;ipAddress=10.145.112.51&amp;pid=13373&amp;preExecution=&amp;host_sys_id=ca716bb387244504065e00f509434dd6&amp;host_name=dm02db02.ga.ssga.root&amp;patternId=dd15665a7fe022004e83e2065f2a0c57&amp;patternName=Docker Pattern&amp;patternType=1&amp;isCloud=false"><u><b>Here</b></u></a>[/code] 9/10/2019 1:04 Error [code]Failed Exploring CI Pattern, Pattern name: <b>Docker Pattern</b>, Process ID: <b>13328</b>, To Check Pattern Log Press <a href="$sw_horizontal_discovery_log.do?discoLogId=acebe6ad1bff7f404d41dd7edd4bcb1f&amp;ipAddress=10.145.112.54&amp;pid=13328&amp;preExecution=&amp;host_sys_id=7e912fb387244504065e00f509434d8c&amp;host_name=dm02db05.ga.ssga.root&amp;patternId=dd15665a7fe022004e83e2065f2a0c57&amp;patternName=Docker Pattern&amp;patternType=1&amp;isCloud=false"><u><b>Here</b></u></a>[/code] 9/10/2019 1:04 Error [code]Failed Exploring CI Pattern, Pattern name: <b>Docker Pattern</b>, Process ID: <b>12911</b>, To Check Pattern Log Press <a href="$sw_horizontal_discovery_log.do?discoLogId=80eb2a6d1b33fb40db5e43b4bd4bcb88&amp;ipAddress=10.144.112.56&amp;pid=12911&amp;preExecution=&amp;host_sys_id=964e9fff87e04504065e00f509434d5f&amp;host_name=dm01db07.ga.ssga.root&amp;patternId=dd15665a7fe022004e83e2065f2a0c57&amp;patternName=Docker Pattern&amp;patternType=1&amp;isCloud=false"><u><b>Here</b></u></a>[/code] 9/10/2019 1:04 Error [code]Failed Exploring CI Pattern, Pattern name: <b>Docker Pattern</b>, Process ID: <b>12899</b>, To Check Pattern Log Press <a href="$sw_horizontal_discovery_log.do?discoLogId=40eb2a6d1b33fb40db5e43b4bd4bcbc2&amp;ipAddress=10.144.112.53&amp;pid=12899&amp;preExecution=&amp;host_sys_id=391e5bff87e04504065e00f509434d3e&amp;host_name=dm01db04.ga.ssga.root&amp;patternId=dd15665a7fe022004e83e2065f2a0c57&amp;patternName=Docker Pattern&amp;patternType=1&amp;isCloud=false"><u><b>Here</b></u></a>[/code] 9/10/2019 1:04 Error [code]Failed Exploring CI Pattern, Pattern name: <b>Docker Pattern</b>, Process ID: <b>13264</b>, To Check Pattern Log Press <a href="$sw_horizontal_discovery_log.do?discoLogId=48eb2a6d1b33fb40db5e43b4bd4bcb6a&amp;ipAddress=10.145.112.56&amp;pid=13264&amp;preExecution=&amp;host_sys_id=f0b1afb387244504065e00f509434df6&amp;host_name=dm02db07.ga.ssga.root&amp;patternId=dd15665a7fe022004e83e2065f2a0c57&amp;patternName=Docker Pattern&amp;patternType=1&amp;isCloud=false"><u><b>Here</b></u></a>[/code] 9/10/2019 1:04 Error [code]Failed Exploring CI Pattern, Pattern name: <b>Docker Pattern</b>, Process ID: <b>12879</b>, To Check Pattern Log Press <a href="$sw_horizontal_discovery_log.do?discoLogId=44eb2a6d1b33fb40db5e43b4bd4bcbf4&amp;ipAddress=10.144.112.50&amp;pid=12879&amp;preExecution=&amp;host_sys_id=6cfddfbb87e04504065e00f509434d75&amp;host_name=dm01db01.ga.ssga.root&amp;patternId=dd15665a7fe022004e83e2065f2a0c57&amp;patternName=Docker Pattern&amp;patternType=1&amp;isCloud=false"><u><b>Here</b></u></a>[/code] 9/10/2019 1:04 Error [code]Failed Exploring CI Pattern, Pattern name: <b>Docker Pattern</b>, Process ID: <b>13267</b>, To Check Pattern Log Press <a href="$sw_horizontal_discovery_log.do?discoLogId=4adba2ad1bff7f404d41dd7edd4bcbb1&amp;ipAddress=10.145.112.55&amp;pid=13267&amp;preExecution=&amp;host_sys_id=19a12fb387244504065e00f509434d28&amp;host_name=dm02db06.ga.ssga.root&amp;patternId=dd15665a7fe022004e83e2065f2a0c57&amp;patternName=Docker Pattern&amp;patternType=1&amp;isCloud=false"><u><b>Here</b></u></a>[/code] 9/10/2019 1:04 Error [code]Failed Exploring CI Pattern, Pattern name: <b>Docker Pattern</b>, Process ID: <b>12901</b>, To Check Pattern Log Press <a href="$sw_horizontal_discovery_log.do?discoLogId=cedba2ad1bff7f404d41dd7edd4bcb90&amp;ipAddress=10.144.112.57&amp;pid=12901&amp;preExecution=&amp;host_sys_id=665edfbf87e04504065e00f509434d29&amp;host_name=dm01db08.ga.ssga.root&amp;patternId=dd15665a7fe022004e83e2065f2a0c57&amp;patternName=Docker Pattern&amp;patternType=1&amp;isCloud=false"><u><b>Here</b></u></a>[/code] 9/10/2019 1:04 Error [code]Failed Exploring CI Pattern, Pattern name: <b>Docker Pattern</b>, Process ID: <b>13323</b>, To Check Pattern Log Press <a href="$sw_horizontal_discovery_log.do?discoLogId=cadbae6d1bff7f404d41dd7edd4bcb7b&amp;ipAddress=10.145.112.53&amp;pid=13323&amp;preExecution=&amp;host_sys_id=10916b7387244504065e00f509434d22&amp;host_name=dm02db04.ga.ssga.root&amp;patternId=dd15665a7fe022004e83e2065f2a0c57&amp;patternName=Docker Pattern&amp;patternType=1&amp;isCloud=false"><u><b>Here</b></u></a>[/code] 9/10/2019 1:04 Error [code]Failed Exploring CI Pattern, Pattern name: <b>Docker Pattern</b>, Process ID: <b>13312</b>, To Check Pattern Log Press <a href="$sw_horizontal_discovery_log.do?discoLogId=fbcbeead1b377f40276510e4bd4bcbd2&amp;ipAddress=10.145.112.50&amp;pid=13312&amp;preExecution=&amp;host_sys_id=d7616bb387244504065e00f509434dd3&amp;host_name=dm02db01.ga.ssga.root&amp;patternId=dd15665a7fe022004e83e2065f2a0c57&amp;patternName=Docker Pattern&amp;patternType=1&amp;isCloud=false"><u><b>Here</b></u></a>[/code] 9/10/2019 1:04 Error [code]Failed Exploring CI Pattern, Pattern name: <b>Docker Pattern</b>, Process ID: <b>12891</b>, To Check Pattern Log Press <a href="$sw_horizontal_discovery_log.do?discoLogId=b7cbeead1b377f40276510e4bd4bcb97&amp;ipAddress=10.144.112.54&amp;pid=12891&amp;preExecution=&amp;host_sys_id=642edbff87e04504065e00f509434dd6&amp;host_name=dm01db05.ga.ssga.root&amp;patternId=dd15665a7fe022004e83e2065f2a0c57&amp;patternName=Docker Pattern&amp;patternType=1&amp;isCloud=false"><u><b>Here</b></u></a>[/code] 9/10/2019 1:04 Error [code]Failed Exploring CI Pattern, Pattern name: <b>Docker Pattern</b>, Process ID: <b>13255</b>, To Check Pattern Log Press <a href="$sw_horizontal_discovery_log.do?discoLogId=23cbae6d1bff7f404d41dd7edd4bcb6c&amp;ipAddress=10.145.112.52&amp;pid=13255&amp;preExecution=&amp;host_sys_id=d581ebb387244504065e00f509434da2&amp;host_name=dm02db03.ga.ssga.root&amp;patternId=dd15665a7fe022004e83e2065f2a0c57&amp;patternName=Docker Pattern&amp;patternType=1&amp;isCloud=false"><u><b>Here</b></u></a>[/code] 9/10/2019 1:04 Error [code]Failed Exploring CI Pattern, Pattern name: <b>Docker Pattern</b>, Process ID: <b>13008</b>, To Check Pattern Log Press <a href="$sw_horizontal_discovery_log.do?discoLogId=47cb266d1b33fb40db5e43b4bd4bcb6c&amp;ipAddress=10.144.112.52&amp;pid=13008&amp;preExecution=&amp;host_sys_id=fe0ed7ff87e04504065e00f509434dd8&amp;host_name=dm01db03.ga.ssga.root&amp;patternId=dd15665a7fe022004e83e2065f2a0c57&amp;patternName=Docker Pattern&amp;patternType=1&amp;isCloud=false"><u><b>Here</b></u></a>[/code] 9/10/2019 1:04 Error [code]Failed Exploring CI Pattern, Pattern name: <b>Docker Pattern</b>, Process ID: <b>12885</b>, To Check Pattern Log Press <a href="$sw_horizontal_discovery_log.do?discoLogId=c7cb266d1b33fb40db5e43b4bd4bcb8c&amp;ipAddress=10.144.112.55&amp;pid=12885&amp;preExecution=&amp;host_sys_id=a03e1fff87e04504065e00f509434d97&amp;host_name=dm01db06.ga.ssga.root&amp;patternId=dd15665a7fe022004e83e2065f2a0c57&amp;patternName=Docker Pattern&amp;patternType=1&amp;isCloud=false"><u><b>Here</b></u></a>[/code] 9/10/2019 1:03 Error java.lang.NullPointerException: java.lang.NullPointerException: 9/10/2019 1:03 Error java.lang.NullPointerException: java.lang.NullPointerException: 9/10/2019 1:02 Error java.lang.NullPointerException: java.lang.NullPointerException: 9/10/2019 1:01 Error java.lang.NullPointerException: java.lang.NullPointerException: 9/10/2019 1:00 Error cmdb_metadata : Found duplicate cmdb_rel_type records with name: Master of::Stack Member of having sys_ids: 357afff213a21300f39f721a6144b076, c8c685710b22130005d90d2835673aa8: no thrown error 9/10/2019 1:00 Error java.lang.NullPointerException: java.lang.NullPointerException: 9/10/2019 1:00 Error LICENSE_DETAILS.ALLOCATED ua_stats_defn Calculation: DEF1000115 not found: no thrown error 9/10/2019 0:34 Error java.lang.NullPointerException: java.lang.NullPointerException: 9/10/2019 0:30 Error cmdb_metadata : Found duplicate cmdb_rel_type records with name: Master of::Stack Member of having sys_ids: 357afff213a21300f39f721a6144b076, c8c685710b22130005d90d2835673aa8: no thrown error 9/10/2019 0:30 Error cmdb_metadata : Found duplicate cmdb_rel_type records with name: Master of::Stack Member of having sys_ids: 357afff213a21300f39f721a6144b076, c8c685710b22130005d90d2835673aa8: no thrown error 9/10/2019 0:03 Error UATablePkgOverrideHandler: Could not find the package with source com.snc.problem: no thrown error 9/10/2019 0:03 Error UATablePkgOverrideHandler: Could not find the package with source com.snc.incident: no thrown error 9/10/2019 0:00 Error [code]Canceled discovery of <a href="discovery_schedule.do?sys_id=71c932b1db5aa3403f737afc0f96195a"><u>SSGA Windows Active Servers</u></a>. Already at maximum number of active 'Scheduled' invocations (3) for a given schedule[/code] Can someone please help me achieve or rectify my config file ? Thanks a million in advance.

Hi, I’m trying to use regex in nxlog. My current configuration is to save firewall logs to a file .txt using the $Sender value to create the file name. ....... <Input *****> Module im_tcp Host 0.0.0.0 Port 1001 <Exec> if $raw_event =~ /LEEF/ parse_leef(); else parse_syslog(); </Exec> </Input> ....... <Output > define OUT_DIR %LOGDIR2%/ Module om_file File "%OUT_DIR%/" + $Sender + ".txt" <Schedule> Every 3600 sec <Exec> if ->file_size() > 0M { set_var('newfile', file_name() + strftime(now(), '_%Y%m%d%H%M%S') + '.log'); rotate_to(get_var('newfile')); exec_async('C:/Program Files/GnuWin32/bin/bzip2.exe', 'E:// *.log'); } </Exec> </Schedule> </Output> ......... This is the Log: <13>Sep 4 16:07:23 Firewall: LEEF:1.0|FORCEPOINT|Firewall|1.1.1|Connection_Discarded|src=122.1.1.1 EventReceivedTime=2019-09-04 16:07:23 SourceModuleName=****** SourceModuleType=im_tcp LEEFVersion=<1> LEEF:0.0 Vendor=FORCEPOINT vSrcName=Firewall Version=1.1.1 EventID=Connection_Discarded devTimeFormat=MMM dd yyyy HH:mm:ss devTime=2019-09-04 16:07:23 proto=1 dstPort=80 srcPort=53438 dst=192.1.1.1 sender=services.fw.mi01.custom.cloud node 1 action=Discard the system sets the value of $Sender like this: $Sender = services.fw.mi01.custom.cloud node 1 action=Discard.txt but I need instead the system to set $Sender this way, only up to "node 1": $Sender = services.fw.mi01.custom.cloud node 1.txt I thought about using a regex to extrapolate the value I need, but it doesn’t work. this one: <Exec> if $Sender =~ /(?<=sender=).[^\t]+/g; $Sender = $1 </Exec> Can I do this thing? If so, what should I do? Thank you Antonio

Hello, I'm experiencing an issues with the community edition. Once you power off the log collection server, the nxlog client agent (for windows) will drop the connection, and stop sending UDP packets after the log collection server is back online. Restarting the windows agent starts resolves the issue. Is there a solution for this besides restarting the windows service every so often to ensure the agent is always sending logs? Thanks

I'm attempting to enrich some Windows event logs with "ClientMachine" which needs to equal the hostname. I'm having issues with only some logs coming through with this enriched fields, and others do not contain the ClientMachine enrichment. My config is below. Any help would be greatly appreciated. Panic Soft define ROOT C:\Program Files\nxlog #ModuleDir %ROOT%\modules #CacheDir %ROOT%\data #SpoolDir %ROOT%\data #define CERTDIR %ROOT%\cert define CONFDIR %ROOT%\conf Note that these two lines define constants only; the log file location is ultimately set by the LogFile directive (see below). The MYLOGFILE define is also used to rotate the log file automatically (see the _fileop block). define LOGDIR %ROOT%\data define MYLOGFILE %LOGDIR%\nxlog.log By default, LogFile %MYLOGFILE% is set in log4ensics.conf. This allows the log file location to be modified via NXLog Manager. If you are not using NXLog Manager, you can instead set LogFile below and disable the include line. LogFile %MYLOGFILE% <Extension _syslog> Module xm_syslog </Extension> <Input in> Module im_msvistalog </Input> <Output out> Exec $ClientMachine = hostname_fqdn(); Module om_tcp Host 192.168.1.20 Port 11105 Exec to_syslog_snare(); </Output> <Route 1> Path in => out </Route>

Hi all, I'm using nxlog to send CAS audit log to our syslog server. Each entry in the text file looks similar to this as below: 2019-08-28 14:33:58,959 INFO [org.apereo.inspektr.audit.support.Slf4jLoggingAuditTrailManager] - Audit trail record BEGIN ============================================================= WHO: user1 WHAT: ST-65-eMcuA7IeZWYUYPldhgaT-11 for https://test.com.vn/news/ ACTION: SERVICE_TICKET_CREATED APPLICATION: CAS WHEN: Wed Aug 28 14:33:58 ICT 2019 CLIENT IP ADDRESS: x.x.x.x SERVER IP ADDRESS: x.x.x.x ============================================================= I want to combine these multiple lines to 1 line and I've read some of the documentation on nxlog's website regarding multiline but haven't found a specific config to put them all in 1 entry with a syslog header. Is there any solution for nxlog conf to work with this kind of multiline message? Thanks