Duplicates 4624 or other Event Viewer Log
Hi, I'm using NXLOG Community to transfer logs in and out, from Windows clients to a Linux server with an NXLOG agent for log collection.
The logs arrive correctly, the only thing is that for the event viewer (example) 4624, I see logs with the same time, even in milliseconds, but the message varies only for a few lines of the "message" field.
I wanted to avoid using the repeat module because I would create the same a log recording increasing the database.
This is client configuration file :
define ROOT C:\Program Files\nxlog
define ROOT_STRING C:\Program Files\nxlog
define CERTDIR %ROOT%\cert
Moduledir %ROOT%\modules
CacheDir %ROOT%\data
Pidfile %ROOT%\data\nxlog.pid
SpoolDir %ROOT%\data
LogFile %ROOT%\data\nxlog.log
define MonitoredEventIds 4624, 4647
<Extension _syslog>
Module xm_syslog
</Extension>
<Extension json>
Module xm_json
</Extension>
<Input eventlog>
Module im_msvistalog
<QueryXML>
<QueryList>
<Query Id='0'>
<Select Path="Security">*</Select>
</Query>
</QueryList>
</QueryXML>
Exec if $EventID NOT IN (%MonitoredEventIds%) drop();
Exec if $TargetUserName == "SYSTEM" drop();
Exec if $TargetUserName =~ /$/ drop();
Exec if $TargetUserName =~ /UMFD/ drop();
Exec if $TargetUserName =~ /DWM/ drop();
Exec if $LogonType == "5" drop();
</Input>
<Output out>
Module om_tcp
Host (10.*****)
Port 1514
Exec to_json();
</Output>
<Route eventlog_to_out>
this is server nxlog.conf
User nxlog
Group nxlog
LogFile /var/log/nxlog/nxlog.log
LogLevel INFO
########################################
# Modules #
########################################
<Extension _syslog>
Module xm_syslog
</Extension>
<Extension json>
Module xm_json
</Extension>
<Input in1>
Module im_tcp
Host 0.0.0.0
Port 1514
<Exec>
parse_json();
</Exec>
</Input>
<Output dbi>
Module om_dbi
SQL INSERT INTO SystemEvents (ReceivedAt, DeviceReportedTime, EventID, EventUser, EventSource, EventLogType, FromHost, NTSeverity, Priority, Message)
VALUES (NOW(), NOW(), $EventID, $TargetUserName, $SourceName, $EventType, $Hostname, $Severity, $SeverityValue, $Message )
Driver mysql
Option host 127.0.0.1
Option username ****
Option password *******
Option dbname Syslog
</Output>
########################################
# Routes #
########################################
<Route 1>
Path in1 => dbi
</Route>
it's possible to not register duplicates at the origin?
it's possible to delete duplicates also in mysql database?
thanks you
Hi Ivano,
From your explanation, it appears the events are exact duplicates (since there is a slight difference in the message field). However, you may want to de-duplicator and specify the exact fields you want to test during filtering.
For example, you may want to add the module below to your configuration. Kindly ensure to modify the checkFields arguments to the requirements of your environment.
<Processor norepeat>
Module pm_norepeat
CheckFields Hostname, Message
</Processor>
and modify your route thus...
<Route 1>
Path in1 =>norepeat => dbi
</Route>
Alternatively, you can perform the filtering based on specific conditions using variables.
I hope this helps.
Regards,
Jeffron
