I am trying to read the audit.log on linux. I am using the input noted below; however, I keep receiving this error: WARNING Module audit has no input files to read <Input audit> Module im_file File "/var/log/audit/audit.log" </Input> Can somebody please explain to me what this error means and why I would be receiving it? Thanks in advance for any help on this!

According to the NXlog user documentation (https://nxlog.co/documentation/nxlog-user-guide-full#introduction) the community edition only supports TLSv1.0. Will it be updated to support TLSv1.2 since 1.0 has known vulnerabilities? And if so when is this planned for?

I am testing the FIM module but would like it to not only include the name of the system that the event is coming from but also the IP address. Is there an easy way to have the output for the FIM module include the IP address of the system in the event?

We are in the process of ditching solarwinds kiwi syslog because its not performing, and is instable after every .net update. We mostly use syslog as an proxy between the customer network and our own datacenter where the siem is located. One of the things we heavely use is an filter like log received from host x.x.x.x message text contains " modsecurity " then forward the log to our siem but with an bogus ip adres like 127.0.12.1 we use this to split the logs to seperate logsources is such a filter possible with NXLog (community) we want to purchase the software but need to build a poc first what we want to accomplish first, before doing fancy stuff is listen on udp port 514 , forward that logs to an remote syslog server i was thinking it was something simple like : Module xm_syslog <Input in> Module im_udp Host 0.0.0.0 Port 514 Exec parse_syslog(); </Input> <Output outtcp> Module om_tcp Host 10.202.5.10 Port 530 Exec to_syslog_bsd(); </Output> <Route 1> Path in => out, outtcp </Route> but that doesn rly do anything

I am trying to run a script every time an error is found in logs <Extension _exec> Module xm_exec </Extension> <Input in> Module im_file File "/home/rafal/gitprojects/mst-sender/hub.cloudradar-error.log" <Exec> if $raw_event =~ /(\S+)\ (.+) \[ERROR (.+)/ { exec_async("/bin/sh", "/home/rafal/gitprojects/mst-sender/run.sh"); } </Exec> </Input> From the documentation it looks like it should exec async if regex matches but I am seeing only the following log WARNING not starting unused module in and the script is not executed. I don't need to output it, only run that script. I added a route but this aint helping too <Output out1> Module om_null </Output> <Route 1> # Basic route Path in => out1 </Route> I created another config file as follows just to log a warning but again its not getting executed define ACTION { log_warning("dropping message"); drop(); } <Extension _exec> Module xm_exec </Extension> <Input in> Module im_file File 'D:\mst-sender\hub.cloudradar-error.log' Exec if $raw_event =~ /ERROR/ %ACTION% </Input> <Output out1> Module om_null </Output> <Route 1> # Basic route Path in => out1 </Route> and here is my config file 04/Apr/2020:20:55:33 +0000 [ERROR 0 /hub.cloudradar.php] PHP message: PHP Notice: Indirect modification of overloaded element of Silex\Application has no effect in /var/www/hub/src/app.php on line 96 04/Apr/2020:20:55:33 +0000 [ERROR 0 /hub.cloudradar.php] PHP message: PHP Notice: Indirect modification of overloaded element of Silex\Application has no effect in /var/www/hub/src/app.php on line 96 UPDATE I found the problem. It's a silly mistake of mine You need to make changes in the log file to see the log lines being parsed.

We have an application that does some multiple updates every morning between 6am and 7am. During this time, it generates massive amounts of log entries. This in turn causes the box to run out of memory, triggering Linux's OOM daemon. Running the NxLog-ce. I have added PersistLogqueue TRUE SyncLogqueue TRUE CacheFlushInterval always CacheSync TRUE To the config, and will see if that makes a difference, but it seems that is more to safeguard messages from being lost. I have looked at https://nxlog.co/question/802/nxlog-ce-memory-leak and https://nxlog.co/question/4132/cache-disk-works-not-good, but not sure those two are what I am after?

Can't find this in the documentation and it seems like it should be fairly obvious, so apologies if this is a dumb question. Given the below JSON, if I perform parse_json($raw_event) with Flatten set to the default value of FALSE, how do I access the $header.time_seconds_epoch value, or any other nested value within the config? If Flatten is set to TRUE, then $header.time_seconds_epoch has the expected value in it, but with it set to FALSE, $header.time_seconds_epoch is NULL. { "event_severity": 0, "header": {"event_name": "test", "time_seconds_epoch": 1588267608},"host_info": {"host_name": "test"}}

I have a scenario, where NXLog (CE) collects log events on Windows Server 2012R2 using im_msvistalog and sends messages to a syslog server. The system has a rather unconventional language and locale setting: The location and locale are german (germany), but the language preference is set to English (US). All software installed / running generates GUI and messages in english, as expected. However, NXLog / im_msvistalog doesen't seem to decide output language based on language preference but on local formats setting instead and therefore sends the messages in german. I suppose this is a bug, as NXLog (CE) is the only software presenting anything in english on those systems. Or is there anything I can do to correct that?

I need a way to convert a message from an Incoming LEEF payload to outgoing CEF format. When I have tried to use parse_leef; and use to_cef(); I appear to get the right format but my payload contains the following format: Apr 29 06:55:03 x.x.x.x LEEF:1.0|CEF|.... Obviously I need the LEEF field removed and the CEF field in its place. The parse at the other end doesn't know what do to with this formatting.

Hi, Im trying to change the IETF timestamp format (number of precision). I have used to_syslog_ietf to create the formatted log message. Is anyone know how can we change the time from 6 decimal to 3 or no decimal precision)? Sample Logs - (Current timestamp with 6 decimal) 2012-01-01T17:15:52.873750+08:00 myhost - - - [NXLOG@14506 TestField="test value"] test message Required (timestamp with 3 decimal or no decimal) 2012-01-01T17:15:52.000+08:00 myhost - - - [NXLOG@14506 TestField="test value"] test message Thanks.

Hello, Are there any links or discussion related to how nxlog will be configured, so a Windows server can send DNS Analytical logs to a remote collector. Thank you. Any direction is much appreciated. Temy

Hello, we are working on centralizing logging onto a syslog-ng server and using nxlog for the windows part of the infrastructure. We are using the IP of the sender to route the message to the correct file on the syslog server and I am trying to force nxlog to use the host's IP instead of it's name in the syslog header without much success. The end goal is to forward everything (Windows, Linux, Network) to syslog-ng and from there to Splunk using the Splunk Forwarder on the syslog server. For Windows I am using the to_json() function on the part of the message so that the logs are parsed automatically on the splunk side. There are 3 different QueryXML so as to assign different severity/facility to the messages and, again, route the messages to the correct file on the syslog server. I have tried both to_syslog_bsd() and to syslog_ietf(). For the "old" BSD format I am unable to force the IP in the syslog header when I use IETF format, the IP is correctly used but I then end up with a huge header and data (time, source, ..) being repeated several times which is not efficient BSD format: Apr 27 14:38:45 WIN-0KNG645FE0L Microsoft-Windows-Security-Auditing[628]: {"EventTime":"2020-04-27 14:38:45","Hostname":"10.10.231.53","Keywords":-9214364837600034816,"EventType":"AUDIT_SUCCESS","SeverityValue":2,"Severity":"INFO","EventID":4672,"SourceName":"Microsoft-Windows-Security-Auditing","ProviderGuid":"{54849625-5478-4994-A5BA-3E3B0328C30D}","Version":0,"Task":12548,"OpcodeValue":0,"RecordNumber":22998,"ActivityID":"{9B9FF64D-1C66-0003-EBF6-9F9B661CD601}","ProcessID":628,"ThreadID":692,"Channel":"Security","Category":"Special Logon","Opcode":"Info","SubjectUserSid":"S-1-5-18","SubjectUserName":"SYSTEM","SubjectDomainName":"NT AUTHORITY","SubjectLogonId":"0x3e7","PrivilegeList":"SeAssignPrimaryTokenPrivilege SeTcbPrivilege SeSecurityPrivilege SeTakeOwnershipPrivilege SeLoadDriverPrivilege SeBackupPrivilege SeRestorePrivilege SeDebugPrivilege SeAuditPrivilege SeSystemEnvironmentPrivilege SeImpersonatePrivilege SeDelegateSessionUserImpersonatePrivilege","EventReceivedTime":"2020-04-27 14:38:47","SourceModuleName":"in_sec","SourceModuleType":"im_msvistalog","SyslogFacilityValue":17,"SyslogSeverityValue":2} IETF format: Apr 27 13:39:38 10.10.231.53 1 2020-04-27T13:39:36.528628+02:00 WIN-0KNG645FE0L Microsoft-Windows-Security-Auditing 628 - [NXLOG@14506 Keywords="-9214364837600034816" EventType="AUDIT_SUCCESS" EventID="5379" ProviderGuid="{54849625-5478-4994-A5BA-3E3B0328C30D}" Version="0" Task="13824" OpcodeValue="0" RecordNumber="22664" ActivityID="{9B9FF64D-1C66-0003-EBF6-9F9B661CD601}" ThreadID="1580" Channel="Security" Category="User Account Management" Opcode="Info" SubjectUserSid="S-1-5-21-4288340226-4165896404-4155820673-500" SubjectUserName="Administrator" SubjectDomainName="WIN-0KNG645FE0L" SubjectLogonId="0x53ac84" TargetName="ServerManager*" Type="0" CountOfCredentialsReturned="0" ReadOperation="%%8100" ReturnCode="3221226021" ProcessCreationTime="2020-04-27T11:39:34.659068400Z" ClientProcessId="2716" EventReceivedTime="2020-04-27 13:39:38" SourceModuleName="in_sec" SourceModuleType="im_msvistalog"] {"EventTime":"2020-04-27 13:39:36","Hostname":"10.10.231.53","Keywords":-9214364837600034816,"EventType":"AUDIT_SUCCESS","SeverityValue":2,"Severity":"INFO","EventID":5379,"SourceName":"Microsoft-Windows-Security-Auditing","ProviderGuid":"{54849625-5478-4994-A5BA-3E3B0328C30D}","Version":0,"Task":13824,"OpcodeValue":0,"RecordNumber":22664,"ActivityID":"{9B9FF64D-1C66-0003-EBF6-9F9B661CD601}","ProcessID":628,"ThreadID":1580,"Channel":"Security","Category":"User Account Management","Opcode":"Info","SubjectUserSid":"S-1-5-21-4288340226-4165896404-4155820673-500","SubjectUserName":"Administrator","SubjectDomainName":"WIN-0KNG645FE0L","SubjectLogonId":"0x53ac84","TargetName":"ServerManager*","Type":"0","CountOfCredentialsReturned":"0","ReadOperation":"%%8100","ReturnCode":"3221226021","ProcessCreationTime":"2020-04-27T11:39:34.659068400Z","ClientProcessId":"2716","EventReceivedTime":"2020-04-27 13:39:38","SourceModuleName":"in_sec","SourceModuleType":"im_msvistalog","SyslogFacilityValue":17,"SyslogSeverityValue":2} BSD format seems preferred as it make for smaller messages Sending over TCP so as to not loose some of the largest messages Edit: NXLog nxlog-ce-2.10.2150 Windows 2019 nxlog.conf define ROOT C:\Program Files (x86)\nxlog Moduledir %ROOT%\modules CacheDir %ROOT%\data Pidfile %ROOT%\data\nxlog.pid SpoolDir %ROOT%\data LogFile %ROOT%\data\nxlog.log <Extension syslog> Module xm_syslog </Extension> <Extension _json> Module xm_json </Extension> <Input internal> Module im_internal </Input> <Input in_sys> Module im_msvistalog <QueryXML> <QueryList> <Query Id="0"> <Select Path="System">*</Select> </Query> </QueryList> </QueryXML> <Exec> $SyslogFacilityValue = syslog_facility_value("local1"); $SyslogSeverityValue = syslog_severity_value("emerg"); # $Message = to_json(); # $raw_event = to_json(); # to_syslog_bsd(); </Exec> </Input> <Input in_app> Module im_msvistalog <QueryXML> <QueryList> <Query Id="1"> <Select Path="Application">*</Select> </Query> </QueryList> </QueryXML> <Exec> $SyslogFacilityValue = syslog_facility_value("local1"); $SyslogSeverityValue = syslog_severity_value("alert"); # $Message = to_json(); # to_syslog_bsd(); </Exec> </Input> <Input in_sec> Module im_msvistalog <QueryXML> <QueryList> <Query Id="2"> <Select Path="Security">*</Select> </Query> </QueryList> </QueryXML> <Exec> $SyslogFacilityValue = syslog_facility_value("local1"); $SyslogSeverityValue = syslog_severity_value("crit"); # $Hostname = host_ip(); # parse_syslog(); # $Message = to_json(); # to_syslog_bsd(); delete($Message); </Exec> </Input> <Output syslog-srv> Module om_tcp Host 10.10.231.11 port 514 <Exec> $Hostname = host_ip(); # parse_syslog(); # log_info($raw_event); $Message = to_json(); $Message =~ s/\\[r|n|t]/ /g; # to_syslog_ietf(); to_syslog_bsd(); $raw_event =~ s/\s{2,}/ /g; </Exec> </Output> <Route 1> Path in_sys, in_app, in_sec => syslog-srv </Route>

HI, Is there any solution to use wildcard chars in the path in the file input setting? I have these folders: /path/to/logs/t-01test/log/app.log /path/to/logs/t-02apple/log/app_20200427.log /path/to/logs/t-03horse/log/app.log /path/to/logs/t-04table/log/app.log I used this input settings: <Input in_app> Module im_file File "/path/to/logs/t-/log/app.log" SavePos TRUE InputType LineBased </Input> I always get the ERROR failed to open directory: /path/to/logs/t-*/logs: No such file or directory I used this setting before: <Input in_app> Module im_file File "/path/to/logs/app.log" SavePos TRUE InputType LineBased </Input> This is working correctly but we have added some folders under the /path/to/logs/t-01test/ where the nxlog user cant read for security reasons. The nxlog log is full with errors because of enough rights: ERROR failed to open directory: path/to/logs/t-01test/conf: Permission denied Is it allowed to use wildcards in the path? Thanks!

Hello, I'm trying configure NXlog CE (installed on windows server 2012r2) to collect syslog (from cisco asa), saving it to file and send to Azure Log Analytics (aka Microsoft OMS). In first step I try to collect syslog, convert to json and saving it to file. It's work well. For the next i installed last Pyton (3.8) and check that all libs installed. After that I changed the nxlog.conf according to the manual (https://nxlog.co/documentation/nxlog-user-guide/azure-oms.html#forwarding-data-to-log-analytics). Buy NXlog gives me the following error: ERROR apr_file_write failed in om_exec; The pipe is being closed. How can I fix this error? My nxlog.conf: Panic Soft #NoFreeOnExit TRUE define ROOT C:\Program Files (x86)\nxlog define CERTDIR %ROOT%\cert define CONFDIR %ROOT%\conf define LOGDIR %ROOT%\data define LOGFILE %LOGDIR%\nxlog.log define JSONLOGFILE C:\Program Files (x86)\nxlog\data\json.txt LogFile %LOGFILE% Moduledir %ROOT%\modules CacheDir %ROOT%\data Pidfile %ROOT%\data\nxlog.pid SpoolDir %ROOT%\data <Extension _syslog> Module xm_syslog </Extension> <Extension _charconv> Module xm_charconv AutodetectCharsets iso8859-2, utf-8, utf-16, utf-32 </Extension> <Extension _exec> Module xm_exec </Extension> <Extension _fileop> Module xm_fileop # Check the size of our log file hourly, rotate if larger than 5MB <Schedule> Every 1 hour Exec if (file_exists('%LOGFILE%') and \ (file_size('%LOGFILE%') >= 5M)) \ file_cycle('%LOGFILE%', 8); </Schedule> # Rotate our log file every week on Sunday at midnight <Schedule> When @weekly Exec if file_exists('%LOGFILE%') file_cycle('%LOGFILE%', 8); </Schedule> </Extension> <Extension json> Module xm_json </Extension> <Input udp> Module im_udp Port 514 Host 192.168.1.2 Exec parse_syslog(); to_json(); </Input> <Output file> Module om_file File '%JSONLOGFILE%' </Output> <Output azure_oms> Module om_exec Command "C:\\Users\\user\\AppData\\Local\\Programs\\Python\\Launcher\\py.exe" Arg "C:\Program Files (x86)\nxlog\oms-pipe.py" </Output> <Route udp_to_file_and_oms> Path udp => file, azure_oms </Route> My configuration is different from the example in manual in the "Output azure_oms" part. If use this part as in the manual an error appears: <Output azure_oms> Module om_exec Command oms-pipe.py Exec to_json(); </Output> Error: ERROR couldn't execute process oms-pipe.py; The system cannot find the file specified. Please help me fix this error.

I have been using nxlog-ce-2.9.1716 to send logs in graylog. In graylog timestemp is without milliseconds. For parsing string to date I use such code if ( $raw_event =~ /^(\d{4}-\d\d-\d\d\s\d{1,2}:\d{1,2}:\d{1,2},\d{1,3})\s[\d+]\s(\S+)\s+\s+(.*)/ ) { $EventTime = parsedate($1); $EventTime = parsedate(strftime($EventTime, '%Y-%m-%d %H:%M:%SZ')); } In this case I lost milliseconds. How can I use parsedate() method with milliseconds and converting time to UTC?

Hi https://portal.msrc.microsoft.com/en-us/security-guidance/advisory/ADV190023 Set according to the above site. result,Unexpected service outage occurs. I want you to tell me the measures.

When I use xm_perl with this code from /usr/libexec/nxlog/modules/extension/perl/event1.pl: use Log::Nxlog; use strict; use warnings; use feature 'say'; use JSON; use utf8; sub rec2msg { my $msgsrcaddr = Log::Nxlog::get_field($event, 'MessageSourceAddress'); if ( defined($msgsrcaddr) ) { Log::Nxlog::set_field_string($event, 'MessageSourceCountry', "France"); } } It gives me strange error: syntax error at /usr/libexec/nxlog/modules/extension/perl/event1.pl line 16, at EOF However , there is no code at line 16 at all. What does that mean? It happens when I use any perl code.

Hy! Sometimes one has to analyze Microsoft events containing EventData fields including Data fields without name, or let's say unnamed Data fields, e.g. EventID 2889 - unsigned LDAP requests. - <Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event"> - <System> <Provider Name="Microsoft-Windows-ActiveDirectory_DomainService" Guid="{0e84...-3605-4e8c-...-1e730c959516}" EventSourceName="NTDS General" /> <EventID Qualifiers="16384">2889</EventID> <Version>0</Version> <Level>4</Level> <Task>16</Task> <Opcode>0</Opcode> <Keywords>0x8080000000000000</Keywords> <TimeCreated SystemTime="2020-02-18T13:27:25.716041000Z" /> <EventRecordID>242410</EventRecordID> <Correlation /> <Execution ProcessID="436" ThreadID="1108" /> <Channel>Directory Service</Channel> <Computer>PC1.DOMAINXY.local</Computer> <Security UserID="S-1-5-7-..." /> </System> - <EventData> <Data>172.172.172.172:33426</Data> <Data>DOMAINXY\USERXY</Data> <Data>0</Data> </EventData> </Event> Simply sending this to syslog works like charm. But in this case I woul like to get the values from all three <Data> fields into $raw_event with a special text,like: define EventID_2889_REGEX /(?x) \ <Data>(?<IP>(^(?:(?:25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)\.){3}(?:25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)$))<\/Data> \ <Data>(?<USER>([\w\d]+))<\/Data> \ <Data>(?<BINDTYPE>(\d+))<\/Data>/ <Input eventlog2889> Module im_msvistalog Query <QueryList>\ <Query Id="0" Path="Directory Service">\ <Select Path="Directory Service">*[System[(EventID=2889)]]</Select>\ </Query>\ </QueryList> Exec $EventData =~ %EventID_2889_REGEX%; Exec $raw_event = "Unsigned or simple non TLS LDAP bind request: [" + $EventData + "]"; Exec $SyslogFacilityValue = 21; Exec $SyslogFacility = "LOCAL5"; Exec $ProcessID = $SubjectUserName; Exec $SourceName = "2889"; Exec parse_syslog_ietf(); Exec to_syslog_ietf(); </Input> I tried a lot, read the NXLog CE documentation hints, googled for days now...to list it all up here would explode the forum ;-) Could you please give me some hints how to process these values? Maybe this is not even possible in NXLog CE, see https://nxlog.co/question/4158/windows-eventdata-not-captured? Thanks in advance!

Hello, I'm trying to implement some kind of "inventory" module in nxlog for windows-based machines. The idea is to run CMD and get basic info like OS information, members of local admin group or active sessions of users - those are just examples, the goal would be to execute CMD and get data every X hours/days. as a PoC, I have following configuration: define ROOT C:\Program Files (x86)\nxlog Moduledir %ROOT%\modules CacheDir %ROOT%\data Pidfile %ROOT%\data\nxlog.pid SpoolDir %ROOT%\data LogFile %ROOT%\data\nxlog.log <Extension _syslog> Module xm_syslog </Extension> <Extension json> Module xm_json </Extension> <Input message> Module im_exec Command "C:\Windows\System32\cmd.exe" Arg /k Arg dir </Input> <Output out_debug> Module om_file File "D:\\nxlog_debug.log" </Output> <Route 2> Path message => out_debug </Route> It works, I can see output of dir command in a .log file. However I cannot find a right way to add a "schedule". Tried in several ways, but it always fails with config error. Based on this example, is it possible to run a CMD with arguments with a schedule?

Hello for, windows events proceed in JSON but some fields like "CommandLine", "TaskContent", "EventData" e.t.c arrives with XML escaping character, like &lt; is < &gt; is > &amp; is & &quot; is " Is it possible to disable escaping globally? Or the only option to solve the problem is using unescape_xml function for every field like: $CommandLine = unescape_xml($CommandLine); $TaskContent = unescape_xml($TaskContent);