I'm trying to use the stdout to use a powershell script for IIS to check all location for any w3svc* folders and collect the logs inside them. Currently getting the error: "Couldn't process 'include' directly at <nxlog default location>; Invalid 'include_stdout' directive at <nxlog default location>; im_exec process %ROOT%\get_iis_paths.cmd exited normally with exitval: 1; The specified child process is done executing" The verbatim config uses: include_stdout %ROOT%\get_ftp_log_paths.cmd InputType IIS_W3C What am I missing here?

Hi, I'm testing nxlog with IIS servers. It works, however I have following issue: If server has IIS since years and lets assume that logs were stored for 1 year, I have bunch of iis logfiles which all together combined results with around 5GB of Data. While using nxlog as-it-is, it consumes CPU and also SIEM itself is not entirely happy about receiving sudden "bombarding" of new logs within few minutes timeframe. Unfortunately, I have around 50 servers like that and I really do not need to inject past logs. I wonder if there is an option to either throttle nxlog a bit or in best case I'd like to send ONLY new iis logs, even starting from exact today. Below you may find part of config <Input IIS_Logs_1> Module im_file File 'C:\inetpub\logs\LogFiles\W3SVC*\u_ex*.log' ReadFromLast FALSE Recursive TRUE PollInterval 1 Exec $FileName = file_name(); Exec if $raw_event =~ /^#/ drop();\ else\ {\ w3c_1->parse_csv();\ $SourceName = "IIS";\ } </Input> I'd appreciate any hints.

I am using NxLog to read and output logs from various source files (im_file module). I configured NxLog with "ReadFromLast" hint so it is capturing new logs since last "SavePos". Now I want to change this little bit, I want to read just the newest entry in my log files since "SavePos". How can I achieve this? Is there a hint / filter that can help?

Hi Folks, We are testing nxlog for syslog forwarder as replacment for EvtSys. We are facing issue in retriving keys from JSON format message. Suppose in this case we want to get value for a key "Account Name". Can you please help us how can we get value for this key. Below is the code and nxlog generated windows log sample (collected from syslog server). Can you please let us know what went wrong here. how can achieve any key from the JSON. Below is Sample code snippet----------------------------- filter { grok { match => {"message" => "%{MONTH:month}(?:\s|\s\s)%{MONTHDAY:day}\s(?<time> [0-9][:][0-9]*[:][0-9][0-9])\s%{IPV4:src_ip}\s%{GREEDYDATA:json_msg}"} add_field => [ "received_at", "%{@timestamp}" ] add_field => [ "received_from", "%{host}" ] } json{ source => "json_msg" target => "\r\n\tAccount Name:\t\t" } } Below is Sample Log snippet generated by nxlog----------------------------- May 4 10:31:06 10.248.15.57 {"EventTime": "2021-05-04 10:30:16","Hostname":"WindowsHostMachine","Keywords":-9214364837600034816,"EventType":"AUDIT_SUCCESS","SeverityValue":2,"Severity":"INFO","EventID":4624,"SourceName":"Microsoft-Windows-Security-Auditing","ProviderGuid":"{54849625-5478-4994-A5BA-3E3B0328C30D}","Version":1,"Task":12544,"OpcodeValue":0,"RecordNumber":2002203,"ProcessID":668,"ThreadID":8076,"Channel":"Security","Message":"An account was successfully logged on.\r\n\r\nSubject:\r\n\tSecurity ID:\t\tS-1-0-0\r\n\tAccount Name:\t\t-\r\n\tAccount Domain:\t\t-\r\n\tLogon ID:\t\t0x0\r\n\r\nLogon Type:\t\t\t3\r\n\r\nImpersonation Level:\t\tImpersonation\r\n\r\nNew Logon:\r\n\tSecurity ID:\t\tS-1-5-21-3128912327-2939948577-25280133-30353\r\n\tAccount Name:\t\tanil.jr.kumar\r\n\tAccount Domain:\t\tNEXTGENTest\r\n\tLogon ID:\t\t0x3B3055EE\r\n\tLogon GUID:\t\t{00000000-0000-0000-0000-000000000000}\r\n\r\nProcess Information:\r\n\tProcess ID:\t\t0x0\r\n\tProcess Name:\t\t-\r\n\r\nNetwork Information:\r\n\tWorkstation Name:\tHDC3-L-F25D2EZ\r\n\tSource Network Address:\t-\r\n\tSource Port:\t\t-\r\n\r\nDetailed Authentication Information:\r\n\tLogon Process:\t\tNtLmSsp \r\n\tAuthentication Package:\tNTLM\r\n\tTransited Services:\t-\r\n\tPackage Name (NTLM only):\tNTLM V2\r\n\tKey Length:\t\t128\r\n\r\nThis event is generated when a logon session is created. It is generated on the computer that was accessed.\r\n\r\nThe subject fields indicate the account on the local system which requested the logon. This is most commonly a service such as the Server service, or a local process such as Winlogon.exe or Services.exe.\r\n\r\nThe logon type field indicates the kind of logon that occurred. The most common types are 2 (interactive) and 3 (network).\r\n\r\nThe New Logon fields indicate the account for whom the new logon was created, i.e. the account that was logged on.\r\n\r\nThe network fields indicate where a remote logon request originated. Workstation name is not always available and may be left blank in some cases.\r\n\r\nThe impersonation level field indicates the extent to which a process in the logon session can impersonate.\r\n\r\nThe authentication information fields provide detailed information about this specific logon request.\r\n\t- Logon GUID is a unique identifier that can be used to correlate this event with a KDC event.\r\n\t- Transited services indicate which intermediate services have participated in this logon request.\r\n\t- Package name indicates which sub-protocol was used among the NTLM protocols.\r\n\t- Key length indicates the length of the generated session key. This will be 0 if no session key was requested.","Category":"Logon","Opcode":"Info","SubjectUserSid":"S-1-0-0","SubjectUserName":"-","SubjectDomainName":"-","SubjectLogonId":"0x0","TargetUserSid":"S-1-5-21-3128912327-2939948577-25280133-30353","TargetUserName":"anil.jr.kumar","TargetDomainName":"tNEXTGENTest","TargetLogonId":"0x3b3055ee","LogonType":"3","LogonProcessName":"NtLmSsp ","AuthenticationPackageName":"NTLM","WorkstationName":"HDC3-L-F25D2EZ","LogonGuid":"{00000000-0000-0000-0000-000000000000}","TransmittedServices":"-","LmPackageName":"NTLM V2","KeyLength":"128","ProcessName":"-","IpAddress":"-","IpPort":"-","ImpersonationLevel":"%%1833","EventReceivedTime":"2021-05-04 10:30:17","SourceModuleName":"eventlog","SourceModuleType":"im_msvistalog"}

Hi! I am testing WEC on Linux. I need the uid_to_name () function in this version: $ SubjectUserSidRSLVD = uid_to_name ($ SubjectUserSid); When WEC was on Windows, this function returned SID_Resolve_UserName. On Linux, this function requires a UID as input. Since I give the SID as input, I get null at the output. Are there any analogues of this function NXLog on Linux, so that I would input the SID at the input and receive SID_Resolve_UserName at the output? For the same theme: https://nxlog.co/question/6938/wec-linux-uidtoname-returns-null

Hi all, I'm using CE. Could you tell me how to write nxlog.conf? I want to im_exec every 10Sec. I have no idea about what should I write in schedule. <Input messages> Module im_exec Command "C:\Windows\System32\cmd.exe" Arg /k Arg dir <Schedule> Every 10 sec <Exec> I want to do messages(im_exec) again! </Exec> </Schedule> </Input> <Output file> Module om_file File "C:\test_logs\output_test1.txt" </Output> <Route messages_to_file> Path messages => file </Route>

Hey All, Has anyone successfully configured NXLog/Alienvault for reading Oracle Audit Log files? Our issue is this, NXLog successfully reads and sends it over to Alienvault where it goes to the AV log file there using the plugin oracle-nxlog.cfg. However, it does not show up in the Alienvault user interface. Also, I realized this is more of an Alienvault question, but there online docs are pretty scarce for this topic and I thought one of you folks may have had success in doing this. Thanks! kel

I've created a folder in my local machine were I have admin privileges, however when I send an output to the created folder I always get the following error message "ERROR failed to open C:\location path of the folder\Test_folder; Access is denied. " Please the below: <Output om_api> Module om_file File "C:\location path of the folder\Test_folder" </Output> I've reviewed my folder permissions and I can confirm I've got full access. I've run out of things to check, please help :(

Hello everyone! I would like to Delete EventTime Field from BIND log and not send it to remote SIEM Server Here is a log example: 11-mai-2021 00:27:48.084 queries: info: client @0x7f82bc11d4e0 10.80.0.1#53995 (google.com): query: google.com IN A +E(0) (10.80.1.88) Unfortunately, it seems that i'm doing something wrong because "11-mai-2021 00:27:48.084" still persist in log Here is my config: Panic Soft #NoFreeOnExit TRUE define ROOT C:\Program Files (x86)\nxlog define CERTDIR %ROOT%\cert define CONFDIR %ROOT%\conf define LOGDIR %ROOT%\data define LOGFILE %LOGDIR%\nxlog.log LogFile %LOGFILE% Moduledir %ROOT%\modules CacheDir %ROOT%\data Pidfile %ROOT%\data\nxlog.pid SpoolDir %ROOT%\data <Extension _syslog> Module xm_syslog </Extension> <Extension _charconv> Module xm_charconv AutodetectCharsets iso8859-2, utf-8, utf-16, utf-32 </Extension> <Extension _exec> Module xm_exec </Extension> <Input in1> Module im_file File 'C:\NXLOGTEST\fileTEST.log' &lt;Exec&gt; # 2. Parse BIND 9 metadata if $Message =~ /(?x)^(?&lt;EventTime&gt;\S+\s\S+)\s(?&lt;Category&gt;\S+):\s (?&lt;BINDSeverity&gt;[^:]+):\s(?&lt;Message&gt;.+)$/i { I TRIED ALSO LIKE THIS delete($EventTime); # 3. Parse messages from the queries category if $Category == &quot;queries&quot; { $Message =~ /(?x)^client\s((?&lt;ClientID&gt;\S+)\s)?(?&lt;Client&gt;\S+)\s \((?&lt;OriginalQuery&gt;\S+)\):\squery:\s (?&lt;QueryName&gt;\S+)\s(?&lt;QueryClass&gt;\S+)\s (?&lt;QueryType&gt;\S+)\s(?&lt;QueryFlags&gt;\S+)\s \((?&lt;LocalAddress&gt;\S+)\)$/; } } &lt;/Exec&gt; #NOW I HAVE IT CONFIGURED LIKE THIS Exec delete($EventTime); </Input> <Output out1> Module om_udp Host 192.168.0.227 Port 514 </Output> <Route r1> Path in1 => out1 </Route> nxlog.log shows no errors, only this line after starting: 2021-05-11 19:57:20 INFO nxlog-ce-2.10.2150 started Can anyone help me investigate?

Default Payload from Source Host: <134>1 1515988859.626061236 appliance flows src=172.21.84.107 dst=10.52.193.137 mac=5C:E0:C5:22:85:E4 protocol=tcp sport=50395 dport=443 pattern: allow all Payload Generated by NXLog Server: <134>May 7 15:18:02 10.101.100.193 1515988859.626061236 appliance flows src=172.21.84.107 dst=10.52.193.137 mac=5C:E0:C5:22:85:E4 protocol=tcp sport=50395 dport=443 pattern: allow all Hi, I have a source machine which is sending logs to NXLog server and NXlog server forward the logs to QRadar. But the payload seems to be different on NXLog Server and QRadar. Timestamp is being added additionally by NXLog server and forwarded to QRadar. Is there a way to make change on the NXLOg server to forward the default log to QRadar.

Here is the default nxlogs looks like, but I would need the logs as below. Do you know what needs to be done? 2021-05-07 19:30:15 INFO nxlog-4.2.4216 started 2021-05-07 19:30:15 ERROR couldn't bind tcp socket to 0.0.0.0:514;Address already in use 2021-05-07 19:30:19 INFO successfully connected to agent manager at nxlogmgr.amgen.com:4041 in SSL mode May 7 19:29:38 (HOSTNAME) INFO nxlog-4.2.4216 started May 7 19:29:38 (HOSTNAME) ERROR couldn't bind tcp socket to 0.0.0.0:514;Address already in use May 7 19:29:38 (HOSTNAME) INFO successfully connected to agent manager at nxlogmgr.amgen.com:4041 in SSL mode (or) 2021-05-07 19:30:15 (HOSTNAME) INFO nxlog-4.2.4216 started 2021-05-07 19:30:15 (HOSTNAME) ERROR couldn't bind tcp socket to 0.0.0.0:514;Address already in use 2021-05-07 19:30:19 (HOSTNAME) INFO successfully connected to agent manager at nxlogmgr.amgen.com:4041 in SSL mode

Hi All, I'm currently trying to ingest some XML files into our SIEM platform (outputting to a local file at the moment for testing), and am having some issues reading files from a large directory. The application we're ingesting log files from creates a folder structure and a single XML file per log entry. Example path: "F:\PSP Logs{4D2D2D9D-5379-4BDF-A331-0AF51BA015ED}\2021\05\06\13\File.xml" I've written the below config and it works fine if you take an existing XML file and append a new line to it, however if we drop a new XML file into the directory structure, it does not pick up the file. Any idea's on where we might be going wrong? Note: We have tried with SavePOS and ReadFromLast on both True and False, and tried both older XML files and newly created ones. Panic Soft define INSTALLDIR C:\Program Files\nxlog #ModuleDir %INSTALLDIR%\modules #CacheDir %INSTALLDIR%\data #SpoolDir %INSTALLDIR%\data define CERTDIR %INSTALLDIR%\cert define CONFDIR %INSTALLDIR%\conf\nxlog.d Note that these two lines define constants only; the log file location is ultimately set by the LogFile directive (see below). The MYLOGFILE define is also used to rotate the log file automatically (see the _fileop block). define LOGDIR %INSTALLDIR%\data define MYLOGFILE %LOGDIR%\nxlog.log If you are not using NXLog Manager, disable the include line and enable LogLevel and LogFile. include %CONFDIR%*.conf LogLevel INFO #LogFile %MYLOGFILE% <Extension _syslog> Module xm_syslog </Extension> This block rotates %MYLOGFILE% on a schedule. Note that if LogFile is changed in managed.conf via NXLog Manager, rotation of the new file should also be configured there. <Extension _fileop> Module xm_fileop # Check the size of our log file hourly, rotate if larger than 5MB &lt;Schedule&gt; Every 1 hour &lt;Exec&gt; if ( file_exists('%MYLOGFILE%') and (file_size('%MYLOGFILE%') &gt;= 5M) ) { file_cycle('%MYLOGFILE%', 8); } &lt;/Exec&gt; &lt;/Schedule&gt; # Rotate our log file every week on Sunday at midnight &lt;Schedule&gt; When @weekly Exec if file_exists('%MYLOGFILE%') file_cycle('%MYLOGFILE%', 8); &lt;/Schedule&gt; </Extension> <Extension xm_xml> Module xm_xml </Extension> <Extension xm_json> module xm_json </Extension> <Input MFiles> Module im_file File "F:\PSP Logs\{4D2D2D9D-5379-4BDF-A331-0AF51BA015ED}\2021\05\*.xml" ##InputType linebased SavePos FALSE readfromlast FALSE Recursive TRUE <Exec> parse_xml(); to_json(); log_info('Successful Message Output'); &lt;/Exec&gt; </Input> <Output local_file> module om_file file "C:\test\XML Test Output\Output.txt" </Output> <Route local> Path MFiles => local_file </Route>

My applications are running in windows server, one of the app log i wanted to parse to an another format, when i check nxlog has the feature. I am new to nxlog, so thought to understand the working flow of nxlog. I didnt get any. I am getting the articles for windows event log parsing, Can someone help me how to parse that or can share any article regarding. Thanks in advance. Regards, Dhamodharan.

Hi Am trying to send the file seperated with commas as below "33,adasdad,null,03May2021,notification,SG,null,null" In nxlog.conf ---> i wrote <Extension log> Module xm_multiline HeaderLine /^<event>/ EndLine /^</\event>/ </Extension> <Input apptype> Module im_file File "testlog.log" <Exec> $message = $raw_event; to_json; </Exec> </Input> <Output out> Module om_tcp Host localhost:port# Exec to_json(); </Output> My logstash confgurariotn input{ tcp { port => "port# codec => multiline{ pattern => "^[%{TIMESTAMP_ISO8601}]" negate => true what => "previous" } } } filter {} output { stdout {codec =>rubydebug} } Output am gettingis as attached { "@timestamp" => "2021-05-03T09:47:14.575Z", "message" => ""33,adasdad,null,03May2021,notification,SG,null,null", "@version" => "1", "tags" => [ [0] "multiline", [1] "multiline_codec_max_lines_reached" ], "host" => "<localhost>", "port" => <port#>, "EventReceivedTime" => "2021-05-03T17:47:14.472325+08:00", "SourceModuleName" => "<apptype>", "SourceModuleType" => "im_file", }

hi Am trying to install and run nxlog in different directory other than /opt like <muser>/opt/nxlog Read through the https://nxlog.co/documentation/nxlog-user-guide/relocating.html link. Did the changes in nxlog.conf But one of steps in the above link shows to modify rpath Tried with command given in the link. As am trying in AIX server , chrpath and patchelf are not working Tried as below as well db2chglibpath --search=<oldpath> --replace=<newpath> <folderwith/myuserfolder/opt/nxlog/lib Which showed command doesnt exists Please get me solve this issue.

We are testing nxlog for syslog forwarder for replacment of windows own provided forwarder EvtSys. We are getting logs at syslog server, but see many special characters and such such #015, #012, #011 in multiple places in log. Below is configuration of .conf file <Input in> Module im_msvistalog </Input> <Output out> Module om_udp Host ...* Port 514 </Output> #################### ROUTE ########### <Route r2> Path in => out </Route> Can you please guide us how to resolve it. I am hereby providing Log snippet of both EvtSys generated log and nxlog generated log Sample Log snippet generated by Windows syslog forwarder EvtSys Apr 22 09:01:03 WindowsHostMachine Security-Auditing: 4624: AUDIT_SUCCESS An account was successfully logged on. Subject: Security ID: S-1-5-18 Account Name: WindowsHostMachine$ Account Domain: TEST Logon ID: 0x3E7 Logon Type: 10 Impersonation Level: Impersonation New Logon: Security ID: S-1-5-21-3128912327-2939948577-25280133-5861 Sample Log snippet generated by nxlog Apr 20 12:41:55 2021-04-20 12: 41:29 WindowsHostMachine AUDIT_SUCCESS 4624 An account was successfully logged on.#015#012#015#012Subject:#015#012#011Security ID:#011#011S-1-0-0#015#012#011Account Name:#011#011-#015#012#011Account Domain:#011#011-#015#012#011Logon ID:#011#0110x0#015#012#015#012Logon Type:#011#011#0113#015#012#015#012Impersonation Level:#011#011Impersonation#015#012#015#012New Logon:#015#012#011Security ID:#011#011S-1-5-21-3128912327-2939948577-25280133-30353#015#012#011

Hello! Testing WEC on Linux. The uid_to_name () function in the xm_resolver module returns null. How can this problem be solved?

In the NXLog User Guide (https://nxlog.co/documentation/nxlog-user-guide/pm_hmac.html) I read the pm_hmac module is deprecated and will be removed in future releases. I need to implement an hash chaining log flow, how can I do without that module in future? Thank you

Hi, I would like to send logs of nxlog.log to an external SIEM. Here are the sample log file but I need hostname added as prefix to each log file as shown below. Is there a way we can use rsyslog or nxlog.conf file to do ot? Before 2021-04-28 17:18:36 INFO connecting to agent manager at nxlogmgr.amgen.com:4041 2021-04-28 17:19:06 INFO reconnecting to agent manager (nxlogmgr.amgen.com) in 128 seconds 2021-04-28 17:19:06 ERROR couldn't connect to agent manager's SSL socket on nxlogmgr.amgen.com:40 After Apr 28 17:23:36 Hostname bash[XXXXX]: INFO connecting to agent manager at nxlogmgr.amgen.com:4041 Apr 28 17:23:36 Hostname bash[XXXXX]: INFO reconnecting to agent manager (nxlogmgr.amgen.com) in 128 seconds Apr 28 17:23:36 Hostname bash[XXXXX]: ERROR couldn't connect to agent manager's SSL socket on nxlogmgr.amgen.com:40

Hi,everyone. I would appreciate if you could give me useful tips to clarify problem and collect event log (ID 4624) on the NX Log. FYI, the configuration file is pasted below, as something may be wrong with a part of it. IP address and port No. in the config, file were replaced intentionally. <Background> -Event logs such as ID 4624 and 4634 has been output to Security.evtx. -Security log has been sent to Log Collection Sever by NxLog. <Input In_MSEventlogs> Module im_msvistalog Query <QueryList> <Query Id="0"> <Select Path="Application"></Select> <Select Path="System"></Select> <Select Path="Security">*</Select> </Query> </QueryList> </Input> -Currently, a part of windows event log (ID 4624) has not been sent to the windows log collection server, while event log (ID 4634) has been sent. -- nxlog.conf -- This is a sample configuration file. See the nxlog reference manual about the configuration options. It should be installed locally and is also available online at http://nxlog.org/docs/ Please set the ROOT to the folder your nxlog was installed into, otherwise it will not start. #define ROOT C:\Program Files\nxlog define ROOT C:\Program Files (x86)\nxlog Moduledir %ROOT%\modules CacheDir %ROOT%\data Pidfile %ROOT%\data\nxlog.pid SpoolDir %ROOT%\data LogFile %ROOT%\data\nxlog.log <Extension charconv> Module xm_charconv AutodetectCharsets UTF-8, UCS-2LE </Extension> Load the json extension <Extension json> Module xm_json </Extension> <Input msdns> Module im_file File "C:\Windows\Sysnative\dns\dns.log" ReadFromLast False SavePos False Exec $FileName = file_name(); Exec $Hostname = hostname_fqdn(); Exec $raw_event = "NXLOG|" + $Hostname + "|OFFBOX-MSDNS-TO-LCP|" + $FileName + "::::" + $raw_event; </Input> Send the read log lines out to nxlog server <Output out-msdns> Module om_tcp Host IP address of the Log server Port DNS OutputType LineBased </Output> Build the route from nxlog on Windows to nxlog on server <Extension _syslog> Module xm_syslog </Extension> <Input In_PowerShell> Module im_msvistalog Query <QueryList> <Query Id="0"> <Select Path="Windows PowerShell"></Select> <Select Path="Microsoft-Windows-PowerShell/Operational"></Select> </Query> </QueryList> For windows 2003 and earlier use the following: Module im_mseventlog </Input> <Input In_MSEventlogs> Module im_msvistalog Exec if ($EventID == 5156) drop(); Query <QueryList> <Query Id="0"> <Select Path="Application"></Select> <Select Path="System"></Select> <Select Path="Security">*</Select> </Query> </QueryList> </Input> <Output Out_MSEventlogs> Module om_udp Host IP address of the Log server Port WEL Exec to_syslog_snare(); </Output> <Output Out_PowerShell> Module om_udp Host IP address of the Log server Port PS Exec to_syslog_snare(); Exec $raw_event = replace($raw_event, "MSWinEventLog", "PowerShell"); </Output> <Route 1> Path msdns => out-msdns </Route> <Route 2> Path In_PowerShell => Out_PowerShell </Route> <Route 3> Path In_MSEventlogs => Out_MSEventlogs </Route>