Hello,
I have nxlog installed on a server where we collect multiple logging streams, nxlog writes those logs to a flat file that is tailed and sent out to our other solutions. This system was not built for log retention so I need nxlog to clear these log files every hour or potentially based on file size. I setup the xm_fileop module on them and defined each log file path and setup the rotation as pasted below, however the logs grow and are not being cleared. Please take a look and let me know where I may have gone wrong. Thank you.
define OUTPUTFILE0 /opt/nxlog/data/fortifirewall/forti.log define OUTPUTFILE1 /opt/nxlog/data/ciscovpn/ciscovpn.log define OUTPUTFILE2 /opt/nxlog/data/cylance/cylance.log define OUTPUTFILE3 /opt/nxlog/data/gpcvpcflow/gpcvpcflow.log define OUTPUTFILE4 /opt/nxlog/data/infobloxdhcp/infobloxdhcp.log define OUTPUTFILE5 /opt/nxlog/data/juniperips/juniperips.log define OUTPUTFILE6 /opt/nxlog/data/pulsevpn/pulsevpn.log define OUTPUTFILE7 /opt/nxlog/data/tanium/tanium.log define OUTPUTFILE8 /opt/nxlog/data/windhcp/windhcp.log define OUTPUTFILE9 /opt/nxlog/data/windns/windns.log define OUTPUTFILE10 /opt/nxlog/data/winevents/winevents.log
<Extension fileop> Module xm_fileop # Truncate the file every hour <Schedule> Every 1 hour Exec file_truncate('%OUTPUTFILE0%'); Exec file_truncate('%OUTPUTFILE1%'); Exec file_truncate('%OUTPUTFILE2%'); Exec file_truncate('%OUTPUTFILE3%'); Exec file_truncate('%OUTPUTFILE4%'); Exec file_truncate('%OUTPUTFILE5%'); Exec file_truncate('%OUTPUTFILE6%'); Exec file_truncate('%OUTPUTFILE7%'); Exec file_truncate('%OUTPUTFILE8%'); Exec file_truncate('%OUTPUTFILE9%'); Exec file_truncate('%OUTPUTFILE10%'); </Schedule> </Extension>
anthonyweller created
Hi,
I installed nxlog and nxlog-manager for testing my own logs in the dashboard, i am new to nxlog, and can't able to check this in dashboard. could you please tell me how to check logs in dashboard, i didn't find any options in the dashboard menu.
Thanks in advance.
sureshnakkina created
Hi,
I just installed nxlog, when i started i seen the below error in nxlog, how to create this file and all, can you please help me on this.
ERROR SSL error, failed to load ca cert from '/opt/nxlog/var/lib/nxlog/cert/agent-ca.pem', reason: No such file or directory, no such file, system lib
Thanks in advance.
sureshnakkina created
Hi,
Anyone can please help on this issue am facing, i'm new to nxlog and things, am getting error : "Cannot communicate with database server." in the nxlog-manager dashboard page. this is my nxlog.conf file, please correct me if there are mistakes.
User nxlog Group nxlog Panic Soft
default values:
PidFile /opt/nxlog/var/run/nxlog/nxlog.pid
CacheDir /opt/nxlog/var/spool/nxlog
ModuleDir /opt/nxlog/lib/nxlog/modules
SpoolDir /opt/nxlog/var/spool/nxlog
define CERTDIR /opt/nxlog/var/lib/nxlog/cert define CONFDIR /opt/nxlog/var/lib/nxlog
Note that these two lines define constants only; the log file location
is ultimately set by the LogFile
directive (see below). The
MYLOGFILE
define is also used to rotate the log file automatically
(see the _fileop
block).
define LOGDIR /opt/nxlog/var/log/nxlog define MYLOGFILE %LOGDIR%/nxlog.log
By default, LogFile %MYLOGFILE%
is set in log4ensics.conf. This
allows the log file location to be modified via NXLog Manager. If you
are not using NXLog Manager, you can instead set LogFile
below and
disable the include
line.
#LogFile %MYLOGFILE% include %CONFDIR%/log4ensics.conf
<Extension _syslog> Module xm_syslog </Extension>
<Extension json>
Module xm_json
</Extension>
This block rotates %MYLOGFILE%
on a schedule. Note that if LogFile
is changed in log4ensics.conf via NXLog Manager, rotation of the new
file should also be configured there.
<Extension _fileop> Module xm_fileop
# Check the size of our log file hourly, rotate if larger than 5MB
<Schedule>
Every 1 hour
<Exec>
if ( file_exists('%MYLOGFILE%') and
(file_size('%MYLOGFILE%') >= 5M) )
{
file_cycle('%MYLOGFILE%', 8);
}
</Exec>
</Schedule>
# Rotate our log file every week on Sunday at midnight
<Schedule>
When @weekly
Exec if file_exists('%MYLOGFILE%') file_cycle('%MYLOGFILE%', 8);
</Schedule>
</Extension>
<Input in_file> Module im_file File '/opt/nxlog/var/log/testing.txt' SavePos TRUE ReadFromLast TRUE PollInterval 1
Exec $Message = $raw_event;
</Input>
<Output out_tcp> Module om_tcp Host 192.168.x.x Port 9090 </Output>
<Route synot> Path in_file => out_tcp </Route>
sureshnakkina created
navdeepsingh83 created
I am trying to read the audit.log on linux. I am using the input noted below; however, I keep receiving this error:
WARNING Module audit has no input files to read
<Input audit> Module im_file File "/var/log/audit/audit.log" </Input>
Can somebody please explain to me what this error means and why I would be receiving it?
Thanks in advance for any help on this!
byron.anderson_atmosera.com created
According to the NXlog user documentation (https://nxlog.co/documentation/nxlog-user-guide-full#introduction) the community edition only supports TLSv1.0. Will it be updated to support TLSv1.2 since 1.0 has known vulnerabilities? And if so when is this planned for?
loza created
I am testing the FIM module but would like it to not only include the name of the system that the event is coming from but also the IP address. Is there an easy way to have the output for the FIM module include the IP address of the system in the event?
byron.anderson_atmosera.com created
Martijng created
Rafalf created
We have an application that does some multiple updates every morning between 6am and 7am. During this time, it generates massive amounts of log entries. This in turn causes the box to run out of memory, triggering Linux's OOM daemon. Running the NxLog-ce.
I have added
PersistLogqueue TRUE SyncLogqueue TRUE CacheFlushInterval always CacheSync TRUE
To the config, and will see if that makes a difference, but it seems that is more to safeguard messages from being lost.
I have looked at https://nxlog.co/question/802/nxlog-ce-memory-leak and https://nxlog.co/question/4132/cache-disk-works-not-good, but not sure those two are what I am after?
Wernervdmerwe created
Can't find this in the documentation and it seems like it should be fairly obvious, so apologies if this is a dumb question. Given the below JSON, if I perform parse_json($raw_event) with Flatten set to the default value of FALSE, how do I access the $header.time_seconds_epoch value, or any other nested value within the config? If Flatten is set to TRUE, then $header.time_seconds_epoch has the expected value in it, but with it set to FALSE, $header.time_seconds_epoch is NULL.
{ "event_severity": 0, "header": {"event_name": "test", "time_seconds_epoch": 1588267608},"host_info": {"host_name": "test"}}
gportnoy created
I have a scenario, where NXLog (CE) collects log events on Windows Server 2012R2 using im_msvistalog and sends messages to a syslog server. The system has a rather unconventional language and locale setting: The location and locale are german (germany), but the language preference is set to English (US). All software installed / running generates GUI and messages in english, as expected.
However, NXLog / im_msvistalog doesen't seem to decide output language based on language preference but on local formats setting instead and therefore sends the messages in german.
I suppose this is a bug, as NXLog (CE) is the only software presenting anything in english on those systems. Or is there anything I can do to correct that?
Dirk created
I need a way to convert a message from an Incoming LEEF payload to outgoing CEF format. When I have tried to use parse_leef; and use to_cef(); I appear to get the right format but my payload contains the following format:
Apr 29 06:55:03 x.x.x.x LEEF:1.0|CEF|....
Obviously I need the LEEF field removed and the CEF field in its place. The parse at the other end doesn't know what do to with this formatting.
kmschramm created
Hi,
Im trying to change the IETF timestamp format (number of precision). I have used to_syslog_ietf to create the formatted log message. Is anyone know how can we change the time from 6 decimal to 3 or no decimal precision)?
Sample Logs - (Current timestamp with 6 decimal) 2012-01-01T17:15:52.873750+08:00 myhost - - - [NXLOG@14506 TestField="test value"] test message
Required (timestamp with 3 decimal or no decimal) 2012-01-01T17:15:52.000+08:00 myhost - - - [NXLOG@14506 TestField="test value"] test message
Thanks.
md22 created
Hello,
Are there any links or discussion related to how nxlog will be configured, so a Windows server can send DNS Analytical logs to a remote collector.
Thank you. Any direction is much appreciated.
Temy
tomolte created
vguyard created
HI,
Is there any solution to use wildcard chars in the path in the file input setting?
I have these folders: /path/to/logs/t-01test/log/app.log /path/to/logs/t-02apple/log/app_20200427.log /path/to/logs/t-03horse/log/app.log /path/to/logs/t-04table/log/app.log
I used this input settings:
<Input in_app> Module im_file File "/path/to/logs/t-/log/app.log" SavePos TRUE InputType LineBased </Input>
I always get the ERROR failed to open directory: /path/to/logs/t-*/logs: No such file or directory
I used this setting before: <Input in_app> Module im_file File "/path/to/logs/app.log" SavePos TRUE InputType LineBased </Input>
This is working correctly but we have added some folders under the /path/to/logs/t-01test/ where the nxlog user cant read for security reasons. The nxlog log is full with errors because of enough rights: ERROR failed to open directory: path/to/logs/t-01test/conf: Permission denied
Is it allowed to use wildcards in the path?
Thanks!
Zola created
Anton.I created
I have been using nxlog-ce-2.9.1716 to send logs in graylog. In graylog timestemp is without milliseconds. For parsing string to date I use such code
if ( $raw_event =~ /^(\d{4}-\d\d-\d\d\s\d{1,2}:\d{1,2}:\d{1,2},\d{1,3})\s[\d+]\s(\S+)\s+\s+(.*)/ )
{
$EventTime = parsedate($1);
$EventTime = parsedate(strftime($EventTime, '%Y-%m-%d %H:%M:%SZ'));
}
In this case I lost milliseconds.
How can I use parsedate() method with milliseconds and converting time to UTC?
spodius created