In the NXLog User Guide (https://nxlog.co/documentation/nxlog-user-guide/pm_hmac.html) I read the pm_hmac module is deprecated and will be removed in future releases. I need to implement an hash chaining log flow, how can I do without that module in future?
Thank you
rdndnl created
Hi,
I would like to send logs of nxlog.log to an external SIEM. Here are the sample log file but I need hostname added as prefix to each log file as shown below. Is there a way we can use rsyslog or nxlog.conf file to do ot?
Before 2021-04-28 17:18:36 INFO connecting to agent manager at nxlogmgr.amgen.com:4041 2021-04-28 17:19:06 INFO reconnecting to agent manager (nxlogmgr.amgen.com) in 128 seconds 2021-04-28 17:19:06 ERROR couldn't connect to agent manager's SSL socket on nxlogmgr.amgen.com:40
After Apr 28 17:23:36 Hostname bash[XXXXX]: INFO connecting to agent manager at nxlogmgr.amgen.com:4041 Apr 28 17:23:36 Hostname bash[XXXXX]: INFO reconnecting to agent manager (nxlogmgr.amgen.com) in 128 seconds Apr 28 17:23:36 Hostname bash[XXXXX]: ERROR couldn't connect to agent manager's SSL socket on nxlogmgr.amgen.com:40
BC_471242 created
Hi,everyone. I would appreciate if you could give me useful tips to clarify problem and collect event log (ID 4624) on the NX Log. FYI, the configuration file is pasted below, as something may be wrong with a part of it. IP address and port No. in the config, file were replaced intentionally.
<Background>
-Event logs such as ID 4624 and 4634 has been output to Security.evtx.
-Security log has been sent to Log Collection Sever by NxLog.
<Input In_MSEventlogs>
Module im_msvistalog
Query <QueryList>
<Query Id="0">
<Select Path="Application"></Select>
<Select Path="System"></Select>
<Select Path="Security">*</Select>
</Query>
</QueryList>
</Input>
-Currently, a part of windows event log (ID 4624) has not been sent to the windows log collection server, while event log (ID 4634) has been sent.
-- nxlog.conf --
This is a sample configuration file. See the nxlog reference manual about the
configuration options. It should be installed locally and is also available
online at http://nxlog.org/docs/
Please set the ROOT to the folder your nxlog was installed into,
otherwise it will not start.
#define ROOT C:\Program Files\nxlog define ROOT C:\Program Files (x86)\nxlog
Moduledir %ROOT%\modules CacheDir %ROOT%\data Pidfile %ROOT%\data\nxlog.pid SpoolDir %ROOT%\data LogFile %ROOT%\data\nxlog.log
<Extension charconv> Module xm_charconv AutodetectCharsets UTF-8, UCS-2LE </Extension>
Load the json extension
<Extension json> Module xm_json </Extension>
<Input msdns> Module im_file File "C:\Windows\Sysnative\dns\dns.log" ReadFromLast False SavePos False Exec $FileName = file_name(); Exec $Hostname = hostname_fqdn(); Exec $raw_event = "NXLOG|" + $Hostname + "|OFFBOX-MSDNS-TO-LCP|" + $FileName + "::::" + $raw_event; </Input>
Send the read log lines out to nxlog server
<Output out-msdns> Module om_tcp Host IP address of the Log server Port DNS OutputType LineBased </Output>
Build the route from nxlog on Windows to nxlog on server
<Extension _syslog> Module xm_syslog </Extension>
<Input In_PowerShell>
Module im_msvistalog
Query <QueryList>
<Query Id="0">
<Select Path="Windows PowerShell"></Select>
<Select Path="Microsoft-Windows-PowerShell/Operational"></Select>
</Query>
</QueryList>
For windows 2003 and earlier use the following:
Module im_mseventlog
</Input>
<Input In_MSEventlogs>
Module im_msvistalog
Exec if ($EventID == 5156) drop();
Query <QueryList>
<Query Id="0">
<Select Path="Application"></Select>
<Select Path="System"></Select>
<Select Path="Security">*</Select>
</Query>
</QueryList>
</Input>
<Output Out_MSEventlogs> Module om_udp Host IP address of the Log server Port WEL Exec to_syslog_snare(); </Output>
<Output Out_PowerShell> Module om_udp Host IP address of the Log server Port PS Exec to_syslog_snare();
Exec $raw_event = replace($raw_event, "MSWinEventLog", "PowerShell"); </Output>
<Route 1> Path msdns => out-msdns </Route>
<Route 2> Path In_PowerShell => Out_PowerShell </Route>
<Route 3> Path In_MSEventlogs => Out_MSEventlogs </Route>
AyakoFukumoto created
2021-04-25 00:00:00 INFO LogFile C:\Program Files (x86)\nxlog\data\nxlog.log reopened
2021-04-25 00:00:14 ERROR apr_stat failed on file C:\Windows\Sysnative\dhcp\DhcpSrvLog-Sun.log; Access is denied.
2021-04-25 00:00:16 WARNING input file was deleted: C:\Windows\Sysnative\dhcp\DhcpSrvLog-Sun.log
2021-04-26 00:00:45 ERROR apr_stat failed on file C:\Windows\Sysnative\dhcp\DhcpSrvLog-Mon.log; Access is denied.
2021-04-26 00:00:47 WARNING input file was deleted: C:\Windows\Sysnative\dhcp\DhcpSrvLog-Mon.log
Smitha.H.C created
Starting our first install and would appreciate a step by step or similar doc.
arch created
When I use gelf extension the millisecond precision is dropped. Basically although I have in log file something like this:
```
...
2021-04-26T14:28:14.269+00:00 [sync_ldap_trigger.sh] [DEBUG] Get list of users from AD/LDAP successfully (with 1 records)
...
```
and for parsing I am using REGEX for multi line detection. Doesn't matter if I am using parsing date function parsedate or strptime or nothing just pure $1
```
#Exec if $raw_event =~ %PEOSYNCREGEX% $EventTime = parsedate($1); $script = $2 ; $Severity = $3 ; $Message = $4 ;
#Exec if $raw_event =~ %PEOSYNCREGEX% $EventTime = strptime($1, '%Y-%m-%d%t%H:%M:%S.%N%Z'); $script = $2 ; $Severity = $3 ; $Message = $4 ;
Exec if $raw_event =~ %PEOSYNCREGEX% $EventTime = $1; $script = $2 ; $Severity = $3 ; $Message = $4 ;
```
I excluded "buffer" from path to be sure and all parsing are done in im_file section. For sure I am testing it with file output from the same source. But on the tcpdump I still have timestamp without the millisecond precision - which I confirmation form me that this information is lost on shipper site.
_tcpdump detailed packet output:_
```
CNSCNc@{"version":"1.1","_EventReceivedTime":"2021-04-26 14:32:15","_SourceModuleName":"peosynclog","_SourceModuleType":"im_file","_script":"sync_ldap_trigger.sh","_Severity":"DEBUG","_SyslogSeverityValue":7,"_FileName":"/home/adsync/bin/adsync2amish/log/sync_general.log","_order":"VNB","_environment":"TEST","host":"amishd-test.vnbrno.cz","short_message":"Get list of users from AD/LDAP successfully (with 1 records)","full_message":"Get list of users from AD/LDAP successfully (with 1 records)","timestamp":1619447535,"level":7}
```
_nxlog.conf:_
```
########################################
# Global directives #
########################################
User nxlog
Group nxlog
LogFile /var/log/nxlog/nxlog.log
LogLevel INFO
# PeoSYNC
define PEOSYNCREGEX /^(?\d{4}\-\d{2}\-\d{2}T\d{2}\:\d{2}\:\d{2}.\d{3}\+\d{4})[\ ,\t]*\[(?[\S,\.,\_]+)\][\ ,\t]*\[(?[\S,\-,\ ]+)\][\ ,\t]*(?[\S,\s]*)/
define ROOT /var/spool/nxlog
CacheDir %ROOT%/data
SpoolDir %ROOT%/data
########################################
# Modules #
########################################
Module xm_multiline
HeaderLine %PEOSYNCREGEX%
Module xm_gelf
########################################
# Inputs #
########################################
Module im_file
File "/home/adsync/bin/engine/log/sync_general.log"
InputType multilinepeosync
SavePos TRUE
#Exec if $raw_event =~ %PEOSYNCREGEX% $EventTime = parsedate($1); $script = $2 ; $Severity = $3 ; $Message = $4 ;
#Exec if $raw_event =~ %PEOSYNCREGEX% $EventTime = strptime($1, '%Y-%m-%d%t%H:%M:%S.%N%Z'); $script = $2 ; $Severity = $3 ; $Message = $4 ;
Exec if $raw_event =~ %PEOSYNCREGEX% $EventTime = $1; $script = $2 ; $Severity = $3 ; $Message = $4 ;
if $Severity !~ /NEVERUSEDSTRING/
{
if $Severity =~ /ALERT/ $SyslogSeverityValue = 1;
if $Severity =~ /CRIT/ $SyslogSeverityValue = 2;
if $Severity =~ /ERR/ $SyslogSeverityValue = 3;
if $Severity =~ /WARN/ $SyslogSeverityValue = 4;
if $Severity =~ /NOTICE/ $SyslogSeverityValue = 5;
if $Severity =~ /INFO/ $SyslogSeverityValue = 6;
if $Severity =~ /---/ $SyslogSeverityValue = 6;
if $Severity =~ /SQL/ $SyslogSeverityValue = 7;
if $Severity =~ /DEBUG/ $SyslogSeverityValue = 7;
} else {
# Parsing and dropping all messages which are NEVERUSEDSTRING (or INFO|DEBUG if you want change) severity
drop();
}
$FileName = file_name();
$Hostname = 'someserver-dev.loc';
$order = 'ORDER';
$environment = 'DEV';
# if not defined($EventTime) $EventTime = now();
########################################
# Outputs #
########################################
Module om_tcp
Host 127.0.0.1
Port 20228
OutputType GELF_TCP
Module om_file
File "/tmp/sync_general.log"
########################################
# Define Processor Modules #
########################################
Module pm_buffer
MaxSize 524288
Type disk
WarnLimit 393216
########################################
# Routes #
########################################
#Path peosynclog => bufferpeosync => graylogtcppeosyncrewrite
Path peosynclog => graylogtcppeosyncrewrite
Path peosynclog => file
```
Please anyone for help or solution for this issue.
Michal
BTW: Of course that I tried more format for timestamp as 2021-04-26T14:28:04.240+02:00 or 2021-04-26T14:28:04.240+0200 etc. The format in log is in my hands too.
juvie created
Design: strict network design with multiple security zones, nxlog client on all Linux servers forward logs to local zone Collector (nxlog listen on 0.0.0.0 port 514), zone collectors forward to single master collectors, master collectors dump all incoming into a dump file for long time storage + forward to a SIEM. This has worked OK for a year, albeit with a few glitches, but recently we have experienced that the Master collectors stop writing to dump file (apparentely at random), yet the nxlog service remains up and running. Fix is to restart the nxlog service. All collectors have defined buffers - could there something wrong with that part of the setup?
All suggestions are welcome :)
DS_534595 created
Can we relocate the directory structure to our preferred folder while configuring nxlog like below
/opt ---> /<ouruser>/opt
What are the changes we need to do if we want to do that ?
I changed the /<ouruser>/nxlog/etc/init file to point to new folder ALSO nxlog.conf --- path also .
but Got error as "Could not load program <ouruser>/nxlog/bin/nxlog : Dependent module libnx.so could not be loaded "
deepap created
tiparker created
giorgiobuffa created
Setup according to documentation here:
vCenter CLI installed
The issue I'm having isn't so much with NXLog, but rather the .pl script in the documentation. When attempting to run the script:
c:\ perl vcenter.pl -s=x.x.x.x -u=ServiceAccount -p=password
I receive the following:
`{"EventTime":"2021-04-15T19:40:39","Message":"Server version unavailable at 'https://x.x.x.x:443/sdk/vimService.wsdl' at C:/Program Files (x86)/VMware/VMware vSphere CLI/Perl/lib/VMware/VICommon.pm line 551.
","UserName":"ServiceAccount"}
{"EventTime":"2021-04-15T19:40:39","Message":"Vim::get_service_content called, but no global session is defined at vcenter.pl line 87
","UserName":"ServiceAccount"}`
I can confirm these credentials were working before, as I used to have an Arcsight Smart Connector pulling these logs. Any insight that can be provided would be greatly appreciated. Thanks in advance!
Tenways created
Hi I'm trying to get nxlog ee running with Microsoft Sentinel still get the error
2021-04-14 19:02:04 INFO [om_http|AzureHTTP] connecting to xxxxxxxx.ods.opinsights.azure.com(xx.xx.xx.xx):443 2021-04-14 19:02:04 ERROR [om_http|AzureHTTP] SSL error, failed to load ca cert from 'C:\Program Files\nxlog\cert\test.crt', reason: no certificate or crl found 2021-04-14 19:02:04 INFO [om_http|AzureHTTP] reconnecting in 4 sec
Forwarding to Sentinel Logspace is not working.
What's wrong ? Johannes
Panic Soft
define INSTALLDIR C:\Program Files\nxlog
#ModuleDir %INSTALLDIR%\modules #CacheDir %INSTALLDIR%\data #SpoolDir %INSTALLDIR%\data
define CERTDIR %INSTALLDIR%\cert define CONFDIR %INSTALLDIR%\conf\nxlog.d define WORKSPACE xxxxxxxxxxxxxxxxxxxx define SHAREDKEY xxxxxxxxxxxxxxxxxxxx define SUBDOMAIN ods.opinsights.azure.com define RESOURCE api/logs define APIVER api-version=2016-04-01 define SIZELIMIT 65000
Note that these two lines define constants only; the log file location
is ultimately set by the LogFile
directive (see below). The
MYLOGFILE
define is also used to rotate the log file automatically
(see the _fileop
block).
define LOGDIR %INSTALLDIR%\data define MYLOGFILE %LOGDIR%\nxlog.log
If you are not using NXLog Manager, disable the include
line
and enable LogLevel and LogFile.
include %CONFDIR%*.conf
#LogLevel INFO #LogFile %MYLOGFILE%
<Extension _json> Module xm_json </Extension>
<Input VAT_Eventlog> Module im_msvistalog <QueryXML> <QueryList> <Query Id="0"> <Select Path="ForwardedEvents"> *[System[Level=0 and (EventID=4624 or EventID=4647)]] </Select> </Query> </QueryList> </QueryXML> RemoteServer xxx RemoteUser xxx RemoteDomain hq RemotePassword xxx </Input>
<Extension plxm> Module xm_perl PerlCode %INSTALLDIR%\modules\extension\perl\sentinelauth.pl </Extension>
<Output AzureHTTP> Module om_http URL https://%WORKSPACE%.%SUBDOMAIN%/%RESOURCE%?%APIVER% ContentType application/json HTTPSAllowUntrusted TRUE HTTPSCAFile %INSTALLDIR%\cert\test.crt <Exec> create_stat('ec', 'COUNT'); create_stat('bc', 'COUNT'); create_var('batch'); create_var('nextbatch'); add_stat('ec',1);
#---BEGIN--- the enrichment of this event with any new fields:
$BatchNumber = get_stat('bc');
$EventNumber = get_stat('ec');
to_json();
#---END--- the enrichment of this event
if (size(get_var('batch')) + size($raw_event) + 3) > %SIZELIMIT%
# Flush this batch of events
{
set_var('nextbatch', $raw_event);
$raw_event = '[' + get_var('batch') + ']';
add_stat('bc',1);
set_var('batch',get_var('nextbatch'));
$Workspace = "%WORKSPACE%";
$SharedKey = "%SHAREDKEY%";
$ContentLength = string(size($raw_event));
$dts = strftime(now(),'YYYY-MM-DDThh:mm:ssUTC');
$dts_no_tz = replace($dts,'Z','');
$parsedate_utc_false = parsedate($dts_no_tz,FALSE);
$x_ms_date = strftime($parsedate_utc_false, '%a, %d %b %Y %T GMT');
plxm->call("genauth");
add_http_header('Authorization',$authorization);
add_http_header('Log-Type',$SourceModuleName);
add_http_header('x-ms-date',$x_ms_date);
}
else
{
$delimiter = get_stat('ec') == 1 ? '' : ",\n";
set_var('batch', get_var('batch') + $delimiter + $raw_event);
drop();
}
</Exec>
</Output>
#<Output TempFile>
Module om_file
File 'C:\Program Files\nxlog\data\dnsetw.json'
#</Output>
<Route DnsRoute1> Path VAT_Eventlog => AzureHTTP </Route>
test4711 created
sejoneshull created
I'm new to NXLog. I have inherited this from previous engineers no longer with my company. I have to update the server OS and wanted to know when updating the server, do I have to immediately update the NXLog software or will the previous software work? If I update the NXLog software, do I have to immediately deploy the newest version of the endpoint application in my environment?
I am also asking if there are any alerts I can configure on the server. We recently learned a connection was lost and want to ensure we are notified when this happens.
Thanks all!
gglover created
We are looking the log file for n number of Application and pushing into one centralized log. We are reading the log file for every 15 min.Now I want read only last line of all logfiles and need to push that into centralized log. Any cone could you please help on this.
Vijay created
I'm seeing we can add an agent https://nxlog.co/documentation/nxlog-user-guide/nxlog_manager_installation.html#nxlog_manager_agentinstall But I'm not seeing it in the downloads https://nxlog.co/products/nxlog-manager/download so, I'm not able to download the agent. Could someone help me to connect to NXLog Service from NXLog Manager to query or search.
Mayuran created
Hello,
I am having issues setting up the config file for NXlog to pull security events from DC and send to Graylog for my school Capstone project. In the log I keep getting an error "Couldnt parse Exec block at ... couldnt parse statement at line 58, character 218 in .... sytnax error, unexpected )". I utilized a config setup from NXlog documentation and adjusted for my needs. I am not sure what is wrong as the exec block looks proper to me. Here is what I have in my config file:
define SecuritySrc Microsoft-Windows-Security-Auditing
<Input DC_events> Module im_msvistalog <QueryXML> <QueryList> <Query Id="0"> <Select Path="Security">*[System[Provider[ @Name='%SecuritySrc%']]] </Select> </Query> </QueryList> </QueryXML> <Exec> if not (defined($SourceName) and ($EventID IN (%SecurityIDs%) and $SourceName == "%SecuritySrc%")) <------ line 58 drop(); </Exec> </Input>
I appreciate any help on this!
tyros77 created
Hi Team,
We are planning to upgrade nxlog enterprise edition from 3.x to 5.x version.
While planning we were looking at the compatibility matrix of nxlog collector , agent , elastic search and kibana.
to be precise what version of Elastic search will be supoourted with Nxlong v5.x.
Regards Anjan Kumar Tripathy
Anjan_nxlog created
Hello everyone,
I have been noticing some seemingly unnecessary logs being sent to our SIEM when using nxlog for Windows DNS syslog forwarding.
Here is my current configuration:
nxlog.conf
define ROOT C:\Program Files (x86)\nxlog Moduledir %ROOT%\modules CacheDir %ROOT%\data Pidfile %ROOT%\data\nxlog.pid SpoolDir %ROOT%\data LogFile %ROOT%\data\nxlog.log
<Extension syslog> Module xm_syslog </Extension>
<Input in> Module im_file File 'C:\Windows\Sysnative\dns\dns.log' SavePos TRUE ReadFromLast TRUE PollInterval 1 Exec $Message = $raw_event; $SyslogFacilityValue = 22; </Input>
<Output out1> Module om_tcp Host 10.5.1.3 Port 1470 Exec to_syslog_bsd(); </Output>
<Route 1> Path in => out1 </Route>
<Extension _fileop> Module xm_fileop </Extension>
Windows DNS Debug Logging Configuration:
Log packets for debugging: check Packet direction: Outgoing: check Incoming: check
Transport protocol: UDP: check TCP: check
Packet contents: Queries/Transfers: check Updates: check Notifications: not checked
Packet Type: Request: check Response: check
Other options: Log unmatched incoming response packets: not checked Details: not checked Filter packets by IP address: not checked
File path and name: c:\windows\system32\dns\dns.log Maximum size (bytes): 500000000
The result is that all of the necessary logs are being sent to our SIEM, but there are a bunch of unnormalized logs being sent that I do not quite understand.
Here are examples:
<181>(date/time) (System Name) As an example, for a server named DC1 on March 19th, 14:39. <181>Mar 19 14:39:17 DC1
A ton of these are being sent, close to 1 for each normal DNS log. As you could imagine this is using a lot of our event log storage. Does anyone know what is causing this? Or what this <181> is referring to?
Thank you!
johnrclark54 created
pihentagy created