Ask questions. Get answers. Find technical product solutions from passionate experts in the NXLog community.
NXLOg CE 3.0.22 Memory Leak / 2.11 Download
cyberkryptoin created
Dear All,
Has anyone else found issues with the latest Nxlog V3.0.22 in terms of memory. I upgraded a test system on a client and it was usng spiking up to 40% of CPU whereease 2.10 did not do this.
Also, is it possible to download 2.11.90 as the download link appears to have been removed. I even found this issue on Gitlab https://gitlab.com/nxlog-public/nxlog-ce/-/issues/17, however the link does not work.
Does anyone know if there is a repository of releases?
Cheers
Cyberkryption
cyberkryptoin created
Regex and NXlog
farrisk01 created
Given:
<Exec>
if ($EventID == 5156) OR ($EventID == 5158)
drop();
else if ($Channel == 'Security') and ($EventID == 4624)
$Message =~ s/\s*This event is generated when a logon session is created.*$//s;
else if ($Channel == 'Security') and ($EventID == 4634)
$Message =~ s/\s*This event is generated when a logon session is destroyed.*$//s;
# Update the $raw_event field
$raw_event = $EventTime + ' ' + $Message;
</Exec>;
If I understand this correctly when an eventID of 4624 or 4634 is received a substitution is performed on the message field and the orginal contents are replaced with nothing(null). Am I correct or at least close? If so, what is does 's' before the ';' mean?
farrisk01 created
Error 1053 - service nxlog windows
romuloforato created
Hi, thanks for the post, it helped me a lot.
I did according to the video, but when I started the nxlog service, an error appeared:
"Windows could not start the nxlog service on local computer
"Error: 1053 The service did not respond to the start or control request in timely fashion"
I will be grateful if you can answer !
Tks.
romuloforato created
Access to WSL files from Windows
fnix created
Following the documentation to receive the logs from docker, the module configuration for a Linux configuration is the following :
<Input in>
Module im_file
File '/var/lib/docker/containers/*/*-json.log'
<Exec>
parse_json();
$HostID = file_basename(file_name());
$HostID =~ s/-json.log//;
</Exec>
</Input>
However, in Windows, docker is running inside a WSL2 instance. this is why I replace the path with the docker for desktop equivalent :
\\\\wsl$\docker-desktop-data\version-pack-data\community\docker\containers\\*-json.log
Now, this works only if we launch nxlog from a cmd terminal, with the -f option to run in foreground.
When it is running as a Windows service, it throws an error, unable to access the files. It seems it can't resolve the \\wsl$ in the path
How to access Docker WSL2 logs from Windows ?
fnix created
How to uninstall nxlog.exe - please help
virtualchi created
Hi
I have a windows server core 2016 and need help uninstalling nxlog. Not sure who installed it.
I don't have the installer or know exactly what version I have.
c:\Program Files (x86)\nxlog\nxlog.exe" -c "c:\Program Files (x86)\nxlog\conf\nxlog.conf"
I don't think this below helps since I don't know the version. I think is a 32bit.
> msiexec /x nxlog-5.4.7313_windows_x64.msi /qb
virtualchi created
Unexpected Exec block error
molnar_istvan_ofsz created
I am use nxlog ce edition for a few log file handling and parsing on windows.
This files needs a little parsing and rewriting, what i tryed like this:
<Input from_HS_info>
Module im_file
ReadFromLast true
SavePos true
RenameCheck true
InputType ml_main
File "L:\HS[A-Z]*.log"
<Exec>
$ReadFilename = file_name();
if ( $ReadFilename =~ /^([A-Za-z]+)_[A-Za-z_0-9]*RNTOFSZ_(OFSZIMHTR\d+).*\.log$/ ) {
$AccountName = $1;
$Hostname = $2;
}
# Remove extra spaces
$raw_event = replace($raw_event,'\s\s+1',' ');
$Channel = 'Software';
$AcountType = 'Porcess';
$Severity = 'INFO';
$Message = $raw_event
$Tags = '["' + $AccountName + '", "prod", "Instant Money" ]';
$raw_event = 'InstantMoney ' + $raw_event;
</Exec>
This snippet starts on line 72 and ends on line 98.
When I try to start it sneds an error message to the log file (nxlog.log):
2022-01-24 16:31:54 ERROR Couldn't parse Exec block at C:\Program Files (x86)\nxlog\conf\nxlog.conf:80; couldn't parse statement at line 92, character 13 in C:\Program Files (x86)\nxlog\conf\nxlog.conf; syntax error, unexpected TOKEN_FIELDNAME, expecting ;
Can anyone help me to correct this?
molnar_istvan_ofsz created
NXLog CE Packages for Bullseye
codeweavers created
Hiya,
Just wondering when the packages for Bullseye are going to be released for the Community Edition? I can see there's one already for Enterprise Edition.
I have also just checked that it builds OK on Bullseye and I can't see any problems, so it looks like it should be an easy one?
Thanks,
Rob
codeweavers created
WARNING input file was deleted
farrisk01 created
Hi,
I'm receiving the follow errors in nxlog.log:
2022-01-20 10:11:18 WARNING input file was deleted: C:\Program Files\Common Files\Microsoft Shared\Web Server Extensions\15\LOGS\SPAPP12013-20220106-0939.log
2022-01-20 10:41:19 WARNING input file was deleted: C:\Program Files\Common Files\Microsoft Shared\Web Server Extensions\15\LOGS\SPAPP12013-20220106-1009.log
2022-01-20 11:11:17 WARNING input file was deleted: C:\Program Files\Common Files\Microsoft Shared\Web Server Extensions\15\LOGS\SPAPP12013-20220106-1039.log
2022-01-20 11:41:19 WARNING input file was deleted: C:\Program Files\Common Files\Microsoft Shared\Web Server Extensions\15\LOGS\SPAPP12013-20220106-1109.log
2022-01-20 12:11:19 WARNING input file was deleted: C:\Program Files\Common Files\Microsoft Shared\Web Server Extensions\15\LOGS\SPAPP12013-20220106-1139.log
2022-01-20 12:41:18 WARNING input file was deleted: C:\Program Files\Common Files\Microsoft Shared\Web Server Extensions\15\LOGS\SPAPP12013-20220106-1209.log
2022-01-20 13:11:18 WARNING input file was deleted: C:\Program Files\Common Files\Microsoft Shared\Web Server Extensions\15\LOGS\SPAPP12013-20220106-1239.log
2022-01-20 13:41:19 WARNING input file was deleted: C:\Program Files\Common Files\Microsoft Shared\Web Server Extensions\15\LOGS\SPAPP12013-20220106-1309.log
2022-01-20 14:11:19 WARNING input file was deleted: C:\Program Files\Common Files\Microsoft Shared\Web Server Extensions\15\LOGS\SPAPP12013-20220106-1339.log
2022-01-20 14:41:19 WARNING input file was deleted: C:\Program Files\Common Files\Microsoft Shared\Web Server Extensions\15\LOGS\SPAPP12013-20220106-1409.log
2022-01-20 15:11:18 WARNING input file was deleted: C:\Program Files\Common Files\Microsoft Shared\Web Server Extensions\15\LOGS\SPAPP12013-20220106-1439.log
2022-01-20 15:41:18 WARNING input file was deleted: C:\Program Files\Common Files\Microsoft Shared\Web Server Extensions\15\LOGS\SPAPP12013-20220106-1509.log
2022-01-20 16:11:18 WARNING input file was deleted: C:\Program Files\Common Files\Microsoft Shared\Web Server Extensions\15\LOGS\SPAPP12013-20220106-1539.log
Version: nxlog-ce-2.11.2190.msi
Contents of nxlog.conf:
#
# Configuration for converting and sending Windows logs
# to AlienVault USM Anywhere.
#
# Version: 0.2.20
# Last modification: 2021-10-15
#
define ROOT C:\Program Files (x86)\nxlog
define OUTPUT_DESTINATION_ADDRESS SYSLOG IP
define OUTPUT_DESTINATION_PORT 514
Moduledir %ROOT%\modules
CacheDir %ROOT%\data
Pidfile %ROOT%\data\nxlog.pid
SpoolDir %ROOT%\data
LogFile %ROOT%\data\nxlog.log
<Extension json>
Module xm_json
</Extension>
<Extension syslog>
Module xm_syslog
</Extension>
<Input internal>
Module im_internal
</Input>
#######################################################################
#### SHAREPOINT-NXLOG #####
#### Uncomment the following lines for SharePoint-NXLOG #####
#### log forwarding #####
#######################################################################
<Extension transform_alienvault_csv_sharepoint>
Module xm_csv
Fields $Timestamp, $Process, $TID, $Area, $Category, $EventID, $Level, $Message, $Correlation
FieldTypes string, string, string, string, string, string, string, string, string
Delimiter \t
</Extension>
<Input SHAREPOINT_IN>
Module im_file
File "C:\Program Files\Common Files\Microsoft Shared\Web Server Extensions\15\LOGS*-????????-????.log"
<Exec>
# Drop header lines and empty lines
if $raw_event =~ /^(\xEF\xBB\xBF|Timestamp)/ drop();
else
{
$raw_event =~ s/ +(?=\t)//g;
transform_alienvault_csv_sharepoint->parse_csv();
$EventTime = strptime($Timestamp, "%m/%d/%Y %H:%M:%S");
$Hostname = hostname_fqdn();
$SourceName = "SHAREPOINT-NXLOG";
}
</Exec>
</Input>
<Output SHAREPOINT_OUT>
Module om_udp
Host %OUTPUT_DESTINATION_ADDRESS%
Port %OUTPUT_DESTINATION_PORT%
Exec $EventTime = strftime($EventTime, '%Y-%m-%d %H:%M:%S');
Exec $Message = to_json(); to_syslog_bsd();
</Output>
<Route SP_Route>
Path SHAREPOINT_IN => SHAREPOINT_OUT
</Route>
#######################################################################
#### SHAREPOINT-NXLOG #####
#######################################################################
The files are no longer available due to the log retention policy. How do I prevent this error? I'm relatively sure that I'm missing something in the config file. Any help is appreciated.
farrisk01 created
Simple test - reading from a file and writing to another file
Marin created
Hi I'm new to nxlog.
Just wanted to test it and created config that reads from a file and writes to a file.
The input file contains the string This is a test
My config is:
User nxlog
Group nxlog
<Input in>
Module im_file
File "/opt/nxlog/bin/testlog"
<Exec>
if $raw_event =~ /(\w{4})(\s)(\w{2})(\s)(\w{1})(\s)(\w{4})/
{
$f1 = $1;
$f2 = $2;
$f3 = $3;
$f4 = $4;
$f5 = $5;
$f6 = $6;
$f7 = $7;
}
</Exec>
</Input>
<Output out>
Module om_file
File "/opt/nxlog/bin/outest"
</Output>
<Route r>
Path in => out
</Route>
I start nxlog with command nxlog -c myconf and it does not write to the file.
What Am I missing?
Marin created
The format for passing character strings to an external program.
andy_kr created
Please help me with the character string transfer format.
I need to pass a character string to an external program.
I do it like this:
<Extension exec>
Module xm_exec
</Extension>
<Output to_db_repldata_access_log>
Module om_null
EXEC exec('c:\Windows\System32\cmd.exe','/U','/C','C:\Tools\NXLog\log_access_insert.bat', $Access_Resource);
</Output>
When the $Access_Resource field contains a character string without spaces, such as 'qwerty',
then the string is passed without quotes, just qwerty
When the $Access_Resource field contains a character string with spaces, such as 'qwer ty',
then the string is passed in double quotes - "qwerty"
Questions:
How to unify the way the string is passed, either always in double quotes or always without quotes?
Is it possible to configure the transmission of the string always in single quotes?
andy_kr created
High CPU and RAM Utilization
Anon4343 created
Have you noticed performance issues with the Windows Event log service when a log file size is set to a few GBs? I'm not sure if NXLog is a factor, but perhaps it may sometimes struggle with large event logs?
We have our security event log set to 4 GB size on all servers. I've noticed that there are high CPU and RAM utilization on 5 or 20 minute cycles. The process using the CPU is svchost EventLog. Derived from using Resource Monitor and running:
tasklist /svc /fi "imagename eq svchost.exe"
I used Sysinternals RanMap to see that the security log file was using 4 GB of RAM stored in the Mapped File listing.
We're not seeing this issue on all of our servers. But it was strange when a production and staging server with very similar loads experienced drastically utilizations. The utilization didn't match until the affected server had its security log cleared. There are not a lot of events being generated. 4 GBs of events goes back to over 30 days. The 4 GB setting is a recommended server configuration when using NetWrix Auditor.
The biggest difference is the amount of standby vs active memory allocated to the security log. On affected servers, the active memory will be 4 GB. On unaffected servers, the standby memory will be 4 GB.
Thank you in advance for any pointers.
Anon4343 created
Community Edition Upgrade
nick_bennett created
So we are currently running Community Edition version 2.10.2150 and we're trying to upgrade to the latest version of Community Edition. Does anything need to be done with the config files we have in place? Or will they remain intact since we're going from community -> community and not community -> enterprise.
Also do you know if this type of upgrade is going to require a reboot? (servers)
nick_bennett created
Sending Event Logs To Different Destinations
ABCReed created
Hello,
I am new to using NXLog as it was suggested by my company's current SIEM vendor to be utilized when sending logs to our collectors. I am using NXLog on our Windows Event Forwarding Server to send logs the SIEM. I can get it working where i send all logs coming into the Forwarded Events log section to the collectors. However, I am not able to get it setup where i send logs from specific server to a specific destination. Ideally I would like to have each source sending logs to the WEF server going to a different port on the collector so I can manage them within the SIEM individually vs all as one. Below is what i am using for my input and output config. I am not sure if i need two input sections since it is coming from the same location? Currently this is just sending all logs from server1 and 2 to both ports on the collector. Any help would be appreciated.
<Input server1_in>
Module im_msvistalog
Query <QueryList>
<Query Id="0">
<Select Path="ForwardedEvents">*</Select>
</Query>
</QueryList>
</Input>
<Input server2_in>
Module im_msvistalog
Query <QueryList>
<Query Id="0">
<Select Path="ForwardedEvents">*</Select>
</Query>
</QueryList>
</Input>
<Output server1_out>
Module om_udp
Host Collector IP
Port 1111
Exec if ($MessageSourceAddress == "Server1 IP") to_syslog_snare();
</Output>
<Output server2_out>
Module om_udp
Host Collector IP
Port 2222
Exec if ($MessageSourceAddress == "Server2 IP") to_syslog_snare();
</Output>
<Route 1>
Path server1_in => server1_out
</Route>
<Route 2>
Path server2_in => server2_out
</Route>
ABCReed created
Random white space characters appearing in the output of a log message
ghillssc01 created
Hello,
I am using NXLog EE with the im_odbc module to read application logs from an SQL database table.
After writing these logs to a file or forwarding them to a SIEM I seem to see random white space characters in various fields. Is there anyway possible where I can "cut out" this white space so it no longer appears in the log?
For example, in the below there is white space in the USERID field after sa. Example log here:
<13>Jan 4 16:32:56 PAGBSSC1SQL032 2022-01-04 16:32:56 PAGBSSC1SQL032 INFO id="63548" INDEX1="1" PRODNAME=" " CMPNYNAM="ABF plc " USERID="sa " INQYTYPE="2" DATE1="2022-01-04 00:00:00" SECDESC="Successful Attempts to Log In " DEX_ROW_ID="63548"
Here is my NX Log EE config file:
define INSTALLDIR C:\Program Files\nxlog
#ModuleDir %INSTALLDIR%\modules
#CacheDir %INSTALLDIR%\data
#SpoolDir %INSTALLDIR%\data
define CERTDIR %INSTALLDIR%\cert
define CONFDIR %INSTALLDIR%\conf\nxlog.d
Note that these two lines define constants only; the log file location
is ultimately set by the LogFile directive (see below). The
MYLOGFILE define is also used to rotate the log file automatically
(see the _fileop block).
define LOGDIR %INSTALLDIR%\data
define MYLOGFILE %LOGDIR%\nxlog.log
If you are not using NXLog Manager, disable the include line
and enable LogLevel and LogFile.
include %CONFDIR%*.conf
LogLevel INFO
LogFile %MYLOGFILE%
<Extension _syslog>
Module xm_syslog
</Extension>
This block rotates %MYLOGFILE% on a schedule. Note that if LogFile
is changed in managed.conf via NXLog Manager, rotation of the new
file should also be configured there.
<Extension _fileop>
Module xm_fileop
# Check the size of our log file hourly, rotate if larger than 5MB
<Schedule>
Every 1 hour
<Exec>
if ( file_exists('%MYLOGFILE%') and
(file_size('%MYLOGFILE%') >= 5M) )
{
file_cycle('%MYLOGFILE%', 8);
}
</Exec>
</Schedule>
# Rotate our log file every week on Sunday at midnight
<Schedule>
When @weekly
Exec if file_exists('%MYLOGFILE%') file_cycle('%MYLOGFILE%', 8);
</Schedule>
</Extension>
<Input odbc>
Module im_odbc
ConnectionString DSN=NXLog; Driver={ODBC Driver 17 for SQL Server}; Server=PAGBSSC1SQL032;
Trusted_Connection=yes; Database=DYNAMICS
IdType integer
SQL SELECT DEX_ROW_ID AS id, * FROM DYNAMICS.dbo.SY05000 WHERE DEX_ROW_ID > ?
PollInterval 5
Exec delete($id);
Exec if not ($raw_event =~ /sa/) drop ();
</Input>
<Output udp>
Module om_udp
Host 10.180.13.28:514
Exec to_syslog_bsd();
</Output>
<Route transfer>
Path odbc => udp
</Route>
Any help would be greatly appreciated!
TIA
ghillssc01 created
SQL server error log collect problem
Graziano.Tartari created
I'm trying to collect Sql Server error log using the second conf found here: https://nxlog.co/documentation/nxlog-user-guide/mssql.html
<Input mssql_errorlog>
Module im_file
File 'C:\Program Files\Microsoft SQL Server' +
'MSSQL14.MSSQLSERVER\MSSQL\Log\ERRORLOG'
<Exec>
# Convert character encoding
$raw_event = convert($raw_event, 'UTF-16LE', 'UTF-8');
# Discard empty lines
if $raw_event == '' drop();
# Attempt to match regular expression
else if $raw_event =~ /(?x)^(?<EventTime>\d+-\d+-\d+\ \d+:\d+:\d+.\d+)
\ (?<Source>\S+)\s+(?<Message>.+)$/s
{
# Convert $EventTime field to datetime type
$EventTime = parsedate($EventTime);
# Save $EventTime and $Source; may be needed for next event
set_var('last_EventTime', $EventTime);
set_var('last_Source', $Source);
}
# If regular expression does not match, this is a multi-line event
else
{
# Use the entire line for the $Message field
$Message = $raw_event;
# Check if fields were save from the previous event
if defined(get_var('last_EventTime'))
{
# Use $EventTime and $Source from previous event
$EventTime = get_var('last_EventTime');
$Source = get_var('last_Source');
}
else
# Use received timestamp for $EventTime; $Source is unknown
$EventTime = $EventReceivedTime;
}
</Exec>
</Input>
I receive the follow error:
ERROR if-else failed at line 71, character 9 in C:\Program Files (x86)\nxlog\conf\nxlog.conf. statement execution has been aborted; if-else failed at line 71, character 9 in C:\Program Files (x86)\nxlog\conf\nxlog.conf. statement execution has been aborted; assignment failed at line 57, character 47 in C:\Program Files (x86)\nxlog\conf\nxlog.conf. statement execution has been aborted; function 'parsedate' failed at line 57, character 46 in C:\Program Files (x86)\nxlog\conf\nxlog.conf. expression evaluation has been aborted; 'unknown' type argument is invalid
It seems parsedate cannot evaluate $EventTime...
Please, help me.
Graziano.
Graziano.Tartari created
Selecting events from SQL Server used by Symantec DCS SQL Database
tothr2 created
Is the Conf file format work ? Which point will this read the SQL table ? SQL Table as provided by DCS integration with SIEM solutions like Splunk, SSIM or ArcSight (broadcom.com) KB https://knowledge.broadcom.com/external/article?articleId=175333
<Input reading_integer_id>
Module im_odbc
ConnectionString Driver={ODBC Driver 17 for SQL Server}; Server=xxxxx; Trusted_Connection=yes; Database=db1;UID=ROUSER;PWD=xxxxx;
IdType integer
SQL SELECT Event_ID AS id, * FROM dbo.CSPEVENT_VW WHERE Event_ID > ?
Exec delete($id);
</Input>
Initial run of this gives the following error
2021-12-30 04:22:41 WARNING no routes defined!
2021-12-30 04:22:41 WARNING not starting unused module reading_integer_id
2021-12-30 04:22:41 INFO nxlog-4.10.5008-trial started
2021-12-30 04:22:50 WARNING stopping nxlog service
2021-12-30 04:22:51 WARNING nxlog received a termination request signal, exiting.
tothr2 created
"ERROR memory pool allocation error; Not enough space" even when 300 GB space is available in the disk
Dhananjaya created
Hi Team,
We are using nxlog ce 2.11.2190. We are facing this issue where the nxlog service stops due to "ERROR memory pool allocation error; Not enough space" error found in nxlog logs. After the observation we found that 300 GB disk space is free and we are still getting this error. What could be the reason?
Thanks in Advance
Dhananjaya
Dhananjaya created
NXlog locks server logs
transfl1 created
Exchange admins are complaining of NXlog agent is locking access to server logs.
Any suggestion?
transfl1 created
Nxlog and Strawberry Perl - Error C:\Program Files\nxlog\modules\extension\xm_perl.dll, a dependency dll is likely missing
jnegus3 created
I am upgraded Nxlog version 4.3.4308 to version 5.4.7313. I have been using a Strawberry Perl (version 5.28.0.1) script to parse the logs, but now when Nxlog starts I am getting the following error "Failed to load module from C:\Program Files\nxlog\modules\extension\xm_perl.dll, a dependency dll is likely missing; The specified module could not be found". I have tried multiple versions of Strawberry Perl but get the same error. Can anyone explain why this is happening?
Thanks in advance.
jnegus3 created
Not Fetching "System" log filter by Event ID
chirag_darji created
Hello.
My query is i am trying to fetch "System Event log" from the windows 10 using below input. but problem is the system log is not fetching and putting the file by NXLOG. else i have tried to fetch "Application log" Using below method but that time log is fetching and working. but only "system" log can't fetch.
<Input eventlog>
Module im_msvistalog
SavePos TRUE
<QueryXML>
<QueryList>
<Query Id="0">
<Select Path="System">*[System[(Level=1 or Level=2 or Level=3 or Level=4 or Level=0) and (EventID=3 or EventID=20)]]</Select>
</Query>
</QueryList>
</QueryXML>
Exec to_json();
</Input>
chirag_darji created