Ask questions. Get answers. Find technical product solutions from passionate experts in the NXLog community.
syslog rotation - dated file naming
Biggsy created
I'm struggling with rotating syslog files at midnight so that they are named "YYYY-MM-DD.log" (and contain log records for that date).
I wonder if anyone has an example of the best way to achieve this. The manual could use an example like this.
Thanks
Phil
Biggsy created
Basic CSV file reading
techscott created
I'm attempting to use NXlog to perform a one time read of a CSV file which will then be passed to Fluent-D for processing and writing to a MySQL database.
I've read through the documentation for setting up the nxlog.config file but I'm at a loss to how to configure for my CSV file.
Has anyone has a config file that reads a CSV I'd apreciate the help.
Thanks.
techscott created
Problem with NT AUTHORITY\СИСТЕМА in windows event log
takezi created
Hello.
While reading log from windows, I got string with two encodings WIN-1251 and UTF-8:
like 2017-02-21 16:40:24 IT-73.domain.name INFO 44 NT AUTHORITY\҈Ғƌ�Центр обновления Windows начал скачивать обновление.
where all message in utf-8, and part with AccountName that (usually NT AUTHORITY\SYSTEM in english version of Windows 7) but in russian version its NT AUTHORITY\СИСТЕМА and in logs it looks like NT AUTHORITY\҈Ғƌ�
If I use
convert_fields("AUTO", "utf-8")
I got proper AccountName, but all other parts got wrong encoding.
Is where any way to replace that part with correct encoding?
takezi created
Problem with Windows Event
Pasi created
Hello,
nxlog CE v2.9.1504, Windows Server 2008 Enterprise
relevant part of config file:
<Input eventlog>
# Uncomment im_msvistalog for Windows Vista/2008 and later
Module im_msvistalog
Exec if ($Severity == 'INFO') drop();
<QueryXML>
<QueryList>
<Query Id="0">
<Select Path="System">*</Select>
<Select Path="Security">*</Select>
<Select Path="Application">*</Select>
<!-- EventID 2137 - Shrepoint Health Analyzer - ignore -->
<Suppress Path="Application">*[System[(EventID=2137)]]</Suppress>
<!-- EventID 2138 - Shrepoint Health Analyzer - ignore -->
<Suppress Path="Application">*[System[(EventID=2138)]]</Suppress>
<Select Path="Microsoft-Windows-TaskScheduler/Operational">*</Select>
</Query>
</QueryList>
</QueryXML>
# Uncomment im_mseventlog for Windows XP/2000/2003
# Module im_mseventlog
</Input>
One event (EventID 1309 from Application channel) always returns an empty message field (message:null) as you can see in debug output.
{"EventTime":"2017-03-21 09:54:13","Hostname":"HOST.DOMAIN.TLD","Keywords":36028797018963968,
"EventType":"WARNING","SeverityValue":3,"Severity":"WARNING","EventID":1309,"SourceName":"ASP.NET 2.0.50727.0",
"Task":3,"RecordNumber":1013344,"ProcessID":0,"ThreadID":0,"Channel":"Application","ERROR_EVT_UNRESOLVED":true,
"Category":"Web Event","EventReceivedTime":"2017-03-21 09:54:13","SourceModuleName":"eventlog","SourceModuleType":"im_msvistalog",
"message":null}
How can I get more informations, why those events has a null message field?
What means "ERROR_EVT_UNRESOLVED":true?
Thank you.
Pasi created
some problem under the AIX6.1 system
alexchen123 created
hello:
I find some problem under AIX6.1 system.
the log:
2017-03-21 14:02:41 WARNING additional group memberships couldn't be set because getgrouplist()and setgroups() are not available on this platform
2017-03-21 14:02:41 INFO nxlog-ce-2.8.1248 started
2017-03-21 14:02:41 ERROR failed to open /u01/app/oracle/diag/rdbms/testdb/testdb/trace/alert_testdb.log;Permission denied
2017-03-21 14:02:43 ERROR apr_stat failed on file /u01/app/oracle/diag/rdbms/testdb/testdb/trace/alert_testdb.log;Permission denied
2017-03-21 14:02:49 ERROR last message repeated 2 times
2017-03-21 14:02:57 ERROR apr_stat failed on file /u01/app/oracle/diag/rdbms/testdb/testdb/trace/alert_testdb.log;Permission denied
2017-03-21 14:03:13 ERROR apr_stat failed on file /u01/app/oracle/diag/rdbms/testdb/testdb/trace/alert_testdb.log;Permission denied
2017-03-21 14:03:45 ERROR apr_stat failed on file /u01/app/oracle/diag/rdbms/testdb/testdb/trace/alert_testdb.log;Permission denied
2017-03-21 14:04:50 ERROR apr_stat failed on file /u01/app/oracle/diag/rdbms/testdb/testdb/trace/alert_testdb.log;Permission denied
and i sccan the source code, find:
static void nxlog_set_groups(nxlog_t *nxlog, apr_uid_t uid)
{
gid_t grplist[100];
int ngroups = sizeof(grplist);
char *user;
apr_status_t rv;
if ( (rv = apr_uid_name_get(&user, uid, nxlog->pool)) != APR_SUCCESS )
{
log_aprerror(rv, "couldn't resolve uid %d to name", uid);
return;
}
#ifdef HAVE_GETGROUPLIST
# ifdef HAVE_SETGROUPS
if ( getgrouplist(user, getgid(), grplist, &ngroups) == -1 )
{
log_error("couldn't get group membership for user %s (uid: %d), too many groups?", user, uid);
return;
}
if ( setgroups((size_t) ngroups, grplist) != 0 )
{
log_errno("couldn't get group membership for user %s (uid: %d), setgroups() failed",
user, uid);
return;
}
# else
log_warn("additional group memberships couldn't be set because getgrouplist()"
"and setgroups() are not available on this platform");
# endif
#else
log_warn("additional group memberships couldn't be set because getgrouplist()"
"and setgroups() are not available on this platform");
#endif
}
I don't what' wrong with this problem, thanks.
alexchen123 created
Forwarding events from Windows eventlog collector's "Forwarded Events" to Sumo
alund created
Hi All,
Checking to see if anyone has run into this. I have a windows eventlog collector, with a subscription setup to move specific security audit events to the "Forwarded Events" log. From there, I am looking to push those logs to Sumologic. Unfortunately Sumo's collector does not handle this well due to the out of sequence EventRecordID of the various events coming from multiple desktops/servers we're collecting from.
Question:
I'm trying to take advantage of Sumo's native Windows eventlog parser, however the options for sending the eventlog data using NXlog send in the specific formats, syslog_snare, xml, json, etc. Is there a configuration i can use which send the messages as windows eventlog format? You'll see from my config below, I've tried several formats, to no avail. Any suggestions would be greatly appreciated.
<Input eventlog>
Module im_msvistalog
<QueryXML>
<QueryList>
<Query Id="0">
<Select Path="ForwardedEvents">*</Select>
</Query>
</QueryList>
</QueryXML>
</Input>
<Output out>
Module om_tcp
Host 10.x.x.x
Port 514
# Exec to_xml();
Exec to_syslog_snare();
# Exec $raw_event = replace($raw_event, "\r\n"," ");
# Exec $raw_event = replace($raw_event, "\t", " ");
# Exec $raw_event();
</Output>
<Route 1>
# Path in => out
Path eventlog, internal => out
</Route>
Thanks in advance,
-A
alund created
regex to drop certain string from message
GaurangMaru created
Hello All,
I am new to nxlog or rather logics in terms of regex and all.
I am looking to drop any message which has *.*.*.255 in message field. I tried below , however it does not seem to be working for me:
Exec if $Message =~ /^([01][0-9][0-9]|2[0-4][0-9]|25[0-5]) . ^([01][0-9][0-9]|2[0-4][0-9]|25[0-5]) . ^([01][0-9][0-9]|2[0-4][0-9]|25[0-5]) . 255/ drop();
Please suggest
GaurangMaru created
Hello, at AIX6.1 system, get some problem.
alexchen123 created
config file:
## This is a sample configuration file. See the nxlog reference manual about the
## configuration options. It should be installed locally under
## /usr/share/doc/nxlog-ce/ and is also available online at
## http://nxlog.org/nxlog-docs/en/nxlog-reference-manual.html
########################################
# Global directives #
########################################
User nxlog
Group nxlog
LogFile /var/log/nxlog/nxlog.log
LogLevel INFO
########################################
# Modules #
########################################
<Extension _syslog>
Module xm_syslog
</Extension>
<Extension multiline>
Module xm_multiline
HeaderLine /^\w{3}\s\w{3}\s\d{2}\s\d{2}:\d{2}:\d{2}\s\d{4}/
</Extension>
<Input oerrorin1>
Module im_file
File '/u01/app/oracle/diag/rdbms/oraywsoudb/oraywsoudb/trace/alert_oraywsoudb.log'
InputType multiline
SavePos TRUE
ReadFromLast FALSE
</Input>
<Output udpout>
Module om_udp
Host 10.1.227.45
Port 514
</Output>s
########################################
# Routes #
########################################
<Route 1>
Path oerrorin1 =>udpout
</Route>
then i get the log:
2017-03-17 16:16:39 INFO nxlog-ce-2.8.1248 started
2017-03-17 16:16:39 ERROR failed to open /u01/app/oracle/diag/rdbms/oraywsoudb/oraywsoudb/trace/alert_oraywsoudb.log;Permission denied
2017-03-17 16:16:41 ERROR apr_stat failed on file /u01/app/oracle/diag/rdbms/oraywsoudb/oraywsoudb/trace/alert_oraywsoudb.log;Permission denied
2017-03-17 16:16:47 ERROR last message repeated 2 times
2017-03-17 16:16:55 ERROR apr_stat failed on file /u01/app/oracle/diag/rdbms/oraywsoudb/oraywsoudb/trace/alert_oraywsoudb.log;Permission denied
2017-03-17 16:17:11 ERROR apr_stat failed on file /u01/app/oracle/diag/rdbms/oraywsoudb/oraywsoudb/trace/alert_oraywsoudb.log;Permission denied
2017-03-17 16:17:43 ERROR apr_stat failed on file /u01/app/oracle/diag/rdbms/oraywsoudb/oraywsoudb/trace/alert_oraywsoudb.log;Permission denied
2017-03-17 16:18:47 ERROR apr_stat failed on file /u01/app/oracle/diag/rdbms/oraywsoudb/oraywsoudb/trace/alert_oraywsoudb.log;Permission denied
2017-03-17 16:20:55 ERROR apr_stat failed on file /u01/app/oracle/diag/rdbms/oraywsoudb/oraywsoudb/trace/alert_oraywsoudb.log;Permission denied
2017-03-17 16:25:11 ERROR apr_stat failed on file /u01/app/oracle/diag/rdbms/oraywsoudb/oraywsoudb/trace/alert_oraywsoudb.log;Permission denied
2017-03-17 16:34:16 DEBUG reading config cache from /var/spool/nxlog/configcache.dat
2017-03-17 16:34:16 DEBUG Setting up module '_syslog' using xm_syslog
2017-03-17 16:34:16 DEBUG module _syslog has 4 exported functions
2017-03-17 16:34:16 DEBUG registering function syslog_facility_value
2017-03-17 16:34:16 DEBUG function 'syslog_facility_value' registered
2017-03-17 16:34:16 DEBUG registering function syslog_facility_string
2017-03-17 16:34:16 DEBUG function 'syslog_facility_string' registered
2017-03-17 16:34:16 DEBUG registering function syslog_severity_value
2017-03-17 16:34:16 DEBUG function 'syslog_severity_value' registered
2017-03-17 16:34:16 DEBUG registering function syslog_severity_string
2017-03-17 16:34:16 DEBUG function 'syslog_severity_string' registered
2017-03-17 16:34:16 DEBUG module _syslog has 9 exported procedures
2017-03-17 16:34:16 DEBUG registering procedure parse_syslog
2017-03-17 16:34:16 DEBUG procedure 'parse_syslog' registered
2017-03-17 16:34:16 DEBUG registering procedure parse_syslog
2017-03-17 16:34:16 DEBUG procedure 'parse_syslog' registered
2017-03-17 16:34:16 DEBUG registering procedure parse_syslog_bsd
2017-03-17 16:34:16 DEBUG procedure 'parse_syslog_bsd' registered
2017-03-17 16:34:16 DEBUG registering procedure parse_syslog_bsd
2017-03-17 16:34:16 DEBUG procedure 'parse_syslog_bsd' registered
2017-03-17 16:34:16 DEBUG registering procedure parse_syslog_ietf
2017-03-17 16:34:16 DEBUG procedure 'parse_syslog_ietf' registered
2017-03-17 16:34:16 DEBUG registering procedure parse_syslog_ietf
2017-03-17 16:34:16 DEBUG procedure 'parse_syslog_ietf' registered
2017-03-17 16:34:16 DEBUG registering procedure to_syslog_bsd
2017-03-17 16:34:16 DEBUG procedure 'to_syslog_bsd' registered
2017-03-17 16:34:16 DEBUG registering procedure to_syslog_ietf
2017-03-17 16:34:16 DEBUG procedure 'to_syslog_ietf' registered
2017-03-17 16:34:16 DEBUG registering procedure to_syslog_snare
2017-03-17 16:34:16 DEBUG procedure 'to_syslog_snare' registered
2017-03-17 16:34:16 DEBUG Setting up module 'multiline' using xm_multiline
2017-03-17 16:34:16 DEBUG Setting up module 'oerrorin1' using im_file
2017-03-17 16:34:16 DEBUG module oerrorin1 has 1 exported functions
2017-03-17 16:34:16 DEBUG registering function file_name
2017-03-17 16:34:16 DEBUG function 'file_name' registered
2017-03-17 16:34:16 DEBUG module oerrorin1 has 0 exported procedures
2017-03-17 16:34:16 DEBUG FlowControl enabled for oerrorin1
2017-03-17 16:34:16 DEBUG Setting up module 'udpout' using om_udp
2017-03-17 16:34:16 DEBUG Setting up module 'fileout2' using om_file
2017-03-17 16:34:16 DEBUG module fileout2 has 2 exported functions
2017-03-17 16:34:16 DEBUG registering function file_name
2017-03-17 16:34:16 DEBUG function 'file_name' registered
2017-03-17 16:34:16 DEBUG registering function file_size
2017-03-17 16:34:16 DEBUG function 'file_size' registered
2017-03-17 16:34:16 DEBUG module fileout2 has 2 exported procedures
2017-03-17 16:34:16 DEBUG registering procedure rotate_to
2017-03-17 16:34:16 DEBUG procedure 'rotate_to' registered
2017-03-17 16:34:16 DEBUG registering procedure reopen
2017-03-17 16:34:16 DEBUG procedure 'reopen' registered
2017-03-17 16:34:16 DEBUG CONFIG: _syslog
2017-03-17 16:34:16 DEBUG inputreader 'Syslog_TLS' registered
2017-03-17 16:34:16 DEBUG Inputreader 'Syslog_TLS' registered
2017-03-17 16:34:16 DEBUG outputwriter 'Syslog_TLS' registered
2017-03-17 16:34:16 DEBUG Outputwriter 'Syslog_TLS' registered
2017-03-17 16:34:16 DEBUG CONFIG: multiline
2017-03-17 16:34:16 DEBUG inputreader 'multiline' registered
2017-03-17 16:34:16 DEBUG Inputreader 'multiline' registered
2017-03-17 16:34:16 DEBUG CONFIG: oerrorin1
2017-03-17 16:34:16 DEBUG adding string [/u01/app/oracle/diag/rdbms/oraywsoudb/oraywsoudb/trace/alert_oraywsoudb.log]
2017-03-17 16:34:16 DEBUG string literal declared at line 29, character 86 in /etc/nxlog.conf
2017-03-17 16:34:16 DEBUG literal
2017-03-17 16:34:16 DEBUG parsed expression
2017-03-17 16:34:16 DEBUG CONFIG: udpout
2017-03-17 16:34:16 DEBUG CONFIG: fileout2
2017-03-17 16:34:16 DEBUG adding string [/var/log/logmsg2.txt]
2017-03-17 16:34:16 DEBUG string literal declared at line 43, character 27 in /etc/nxlog.conf
2017-03-17 16:34:16 DEBUG literal
2017-03-17 16:34:16 DEBUG parsed expression
2017-03-17 16:34:16 DEBUG pidfile /var/run/nxlog/nxlog.pid removed
2017-03-17 16:34:16 DEBUG daemonizing...
2017-03-17 16:34:16 DEBUG INIT: _syslog
2017-03-17 16:34:16 DEBUG INIT: multiline
2017-03-17 16:34:16 DEBUG INIT: oerrorin1
2017-03-17 16:34:16 DEBUG INIT: udpout
2017-03-17 16:34:16 DEBUG Pollset initialized for module udpout (method: poll)
2017-03-17 16:34:16 DEBUG INIT: fileout2
2017-03-17 16:34:16 DEBUG Pollset initialized for module fileout2 (method: poll)
2017-03-17 16:34:16 DEBUG now running as group nxlog
2017-03-17 16:34:16 WARNING additional group memberships couldn't be set because getgrouplist()and setgroups() are not available on this platform
2017-03-17 16:34:16 DEBUG now running as user nxlog
2017-03-17 16:34:16 DEBUG running as uid: 203, euid: 203, gid: 204, egid: 204
2017-03-17 16:34:16 DEBUG pidfile /var/run/nxlog/nxlog.pid created
2017-03-17 16:34:16 DEBUG parsing path: oerrorin1 =>udpout
2017-03-17 16:34:16 DEBUG adding module oerrorin1 to route 1
2017-03-17 16:34:16 DEBUG adding module udpout to route 1
2017-03-17 16:34:16 DEBUG parsing path: in2 => fileout2
2017-03-17 16:34:16 ERROR [router.c:68/nx_route_add_module()] module 'in2' is not declared at /etc/nxlog.conf:54
2017-03-17 16:34:16 DEBUG adding module fileout2 to route tcproute
2017-03-17 16:34:16 ERROR [router.c:347/nx_add_route()] route tcproute is not functional without input modules, ignored at /etc/nxlog.conf:54
2017-03-17 16:34:16 DEBUG jobgroup created with priority 99
2017-03-17 16:34:16 DEBUG jobgroup created with priority 10
2017-03-17 16:34:16 DEBUG spawning 3 worker threads
2017-03-17 16:34:16 DEBUG worker thread 0 started
2017-03-17 16:34:16 DEBUG worker thread 1 started
2017-03-17 16:34:16 DEBUG worker thread 2 started
I have no idea about this problem.
/u01/app/oracle/diag/rdbms/oraywsoudb/oraywsoudb/trace/alert_oraywsoudb.log --> user and group is oracle:oinstall
and i add the nxlog user in oinstall group, but why continue report Permission denied?
alexchen123 created
Collecting & storing windows syslog locally on host machine
abhishek created
Hi there!
I am a college student...i am working on windows event logger to collect & analyse windows logs(event & syslog).
I don't know how to configure nxlog.conf file such that ... i can get all windows generated logs locally on host machine.
Waiting for reply.
Thank You
abhishek created
community edition msi
durhamcc created
have recently been trying the Community Edition of nxlog which does exactly what I need (and more) but have an issue with the msi. In order to be something I could deploy it has to pass uninstall and re-install tests. Unfortunatley the uninstall via Windows installer leaves a lot of file and registry entries behind that subsequently causes issues with a re-installation. Does anyone know if this is likely to be addressed within CE? Does anyone know if Enterprise edition has the same msi issue? Thanks.
durhamcc created
im_udp with specific host IP not working
GaurangMaru created
Hello All,
I am using nxlog-ce-2.9.1716 client to collect logs from my device and then forward it to logstash. I want to make sure nxlog client accept syslogs from particular source instead any. However when using specific IP in im_udp module, I am getting error as below:
"2017-03-07 16:04:51 ERROR failed to start im_udp; couldn't bind udp socket to 172.20.20.20:514; The requested address is not valid in its context. "
my input driver is configured as below:
<Input in>
Module im_udp
Host 172.20.20.20
Port 514
</Input>
I tried by changing host as below as well:
Host 172.20.20.20/32 - No luck
Host 172.20.20.20/255.255.255.0 - No luck
Host 172.20.20.20/255.255.255.255 - No luck
Host 172.20.20.20 255.255.255.255 - no luck
Host 172.20.20.20 255.255.255.0 - No luck
However if I make Host 0.0.0.0 - it works , but this is not what I am looking for.
Please help if I am missing anything.
Regards,
Gaurang
GaurangMaru created
Issues on setting up NXLOG to accept syslogs on Windows
CBMT created
Hi!
I'm new to NXLOG so sorry if this is a dumb question.
So I have a Windows 2012 server that I'm attempting to set up to accept Syslog messages from an outsourced proxy system. I've been able to get NXLOG accept the logs and dump them to a flat file which our SIEM tool can pick up. However it ends up being a gigantic file...
I'm trying to refine this now.
Instead of a huge file that we purge out every day, I'd like to have NXLOG limit the log dump to something like 200MB, copy the now older 200MB file to a different directory appending a time stamp to it, then start collecting in a new file in the original folder. The catch is, the live log that is in the original folder needs to keep the same file name, otherwise our SIEM tool won't know what to look for.
I've tried to have a PowerShell script do this and it works, however, if the NXLOG service is running, the log dump file is locked. We could get around this by disabling the service while copying the old log file to a new location, however we'd lose 10-30 minutes of logs in the file copy (not ideal).
Going through the reference manual I see a few ways to limit file size, starting a new file, and copying files over directly in the NXLOG config file. However it's turning out to be a problem condencing all of that into one configuration file that works on Windows. I've only been able to get a 1 of 3 pieces working at one time.
I appreciate any help the community can provide! Let me know if there are any questions or if something isn't clear.
CBMT created
How to efficiently clean up Windows DNS Server debug logs in nxlog
tsigidibam created
What is the most efficient way to parse Microsoft DNS Server debug logs into something more tidy, say into a CSV or KVP format on the nxlog agent?
Consider the following log message:
"24/02/2017 16:37:22 09B0 PACKET 0000009657E7BA40 UDP Rcv 10.0.100.15 a490 Q [0001 D NOERROR] A (7)example(3)com(0)"
First of all, what would be the most efficient way performance-wise to convert this into a CSV or KVP format?
And also, is there some other way besides using Exec and replacing parenthesis and numbers in a sed-like manner to get the clean query name? We have tried to use the Exec method before, but we were hitting some serious performance issues on busy DNS servers.
I have currently switched on to using the pm_pattern module to drop invalid log lines (the beginning of the log file and empty lines) and I was wondering if there would be some easy way to perform both of the tasks (the formatting and the cleaning) using the pm_pattern module?
An example output could look something like the following:
datetime=24/02/2017 16:37:22,thread_id=09B0,context=PACKET,packet_id=0000009657E7BA40,protocol=UDP,action=Rcv,remote_ip=10.0.100.15,
xid=a490,event_type=-,opcode=Q,flags_hex=0001,is_authorative=-,is_truncated=-,recursion_desired=D,recursion_available=-,
response_code=NOERROR,question_type=A,question_name=example.com
The empty or "-" values result from fields specified in the DNS debug log format that are not present in the above message (e.g. all possible flags would be "ATDR", and event_type is "-" because "R" marks a response but an empty value (whitespace) marks a query.
And of course, if the above even is possible, would it be too resource consuming?
tsigidibam created
Logging suddenly stops for high-volume input but continues to work for low-volume input
progssilb created
I'm not sure how to characterize what's going on, but here goes...
My route path has two inputs, an itermittently high-volume input, and a low-volume input. The high-volume input can be thousands in a couple minutes, or it can be practically nothing. The low-volume input is, at most, one or two entries per ~3-4 minutes. There are also three outputs, two HTTP and a rotated file. They are disconnecting a fair bit, presumably due to timeouts or lack of pipelining or something.
In my current configuration, I consistently find that my log data for the high-volume log gets dropped after a couple minutes. I'm not sure if the timing correlates with the HTTP disconnects, but it might. Sometimes I get just over a thousand log lines through, sometimes I get a couple hundred log lines, sometimes I get a couple thousand. Interestingly, the low-volume log is unaffected.
I do have flow control enabled, and putting a buffer on the inputs did not seem to help. I didn't try disabling flow control because I don't understand it very well. I have to have both inputs going into the same route because how the messages interleave is important to the meaning of the entries.
Here's the path that I use:
Path vg_tsw_client, vg_tsw_combat => vg_tsw_pattern => vg_tsw_unparse_finder => vg_tsw_testfile, vg_tsw_es, vg_tsw_cdb
_client is the low-volume, _combat is the high volume. _pattern is a pm_pattern with ~25-30 regexes and a small script on every pattern (0-10 lines), _unparse_finder is a pm_null uses add_to_route to copy unmatched patterns to a new route (0 hits lately) and does some light data enrichment via Exec. _testfile is the rotated file output, _es and _cdb are the HTTP outputs.
Thanks in advance.
progssilb created
Convert DateTime to Unix TIme
toreno93 created
Hello, I have question.
My variable $EventTime contains DateTime in only this format "2017-12-30 01:30:00"
How me with use NxLog, convert to UNIX TIME format or convert to this format Dec 30 01:30:00.
Thank
toreno93 created
parse xml attributes
Luca.Corsini created
Hello,
I need to read logfiles from oracle, which are structured in xml with attributes
It's nxlog still unable to access the attributes?
I've read here that the entrerprise edition does it
https://nxlog.co/announcing-nxlog-enterprise-edition-v30
Handling structured data formats better
The xm_xml extension has been enhanced so that it can now parse nested XML and data stored in XML attributes. Parsing of nested JSON has been also implemented in xm_json and UTF-8 validation can be enforced in order to avoid parser failures caused by invalid UTF-8 in other tools.
This is a feature of the enterprise edition only or it will be ported to community edition too?
Luca.Corsini created
Pattern DB appears to drop fields
progssilb created
I'm trying to get a PatternDB working correctly, and it looks like I'm getting some fields but not all of them. There's only one pattern that's actually generating extra fields, and even it is dropping the first field (ParsedDate). Not sure what's going on here...
Config file (via file inclusion):
<Extension json>
Module xm_json
</Extension>
<Extension syslog>
Module xm_syslog
</Extension>
<Input vg_tsw_client>
Module im_file
File "C:\Program Files (x86)\Steam\steamapps\common\The Secret World\ClientLog.txt"
Exec if not ($raw_event =~ /Scaleform\.TSWACT/) drop();
Exec parse_syslog();
</Input>
<Input vg_tsw_combat>
Module im_file
File "C:\Program Files (x86)\Steam\steamapps\common\The Secret World\CombatLog-*.txt"
Exec if ($raw_event =~ /Sprinting [VI]+/) drop();
Exec parse_syslog();
</Input>
<Processor vg_tsw_pattern>
Module pm_pattern
PatternFile %ROOT%\conf\SecretWorld\patterndb.xml
</Processor>
<Output vg_tsw_testfile>
Module om_file
File "C:\\ProgramData\\nxlogs\\vg-tsw-logs.log"
Exec to_json();
</Output>
<Route vg_tsw_route>
Path vg_tsw_client, vg_tsw_combat => vg_tsw_pattern => vg_tsw_testfile
</Route>
Pattern DB:
<?xml version='1.0' encoding='UTF-8'?>
<patterndb>
<created>2010-01-01 01:02:03</created>
<version>42</version>
<group>
<name>tswCombat</name>
<id>50284624</id>
<matchfield>
<name>SourceModuleName</name>
<type>exact</type>
<value>vg_tsw_combat</value>
</matchfield>
<pattern>
<id>1000</id>
<name>basic combat swing</name>
<matchfield>
<name>Message</name>
<type>regexp</type>
<!-- [00:00:28] (Critical) Solomon County Cop's Spray and Pray hits (Normal) Ravenous Horde for 522 physical damage. (Normal) -->
<value>^\[([^\]]+)\] ((?:\(Critical\) |\(Normal\) )?)(.+?'s|Your) (.+?) hits \((Normal|Glancing)\) (.*?) for (\d+) (physical|magical) damage. \((Normal|Penetrated|Blocked)\)</value>
<capturedfield>
<name>ParsedTime</name>
<type>datetime</type>
</capturedfield>
<capturedfield>
<name>CriticalHit</name>
<type>string</type>
</capturedfield>
<capturedfield>
<name>AttackerName</name>
<type>string</type>
</capturedfield>
<capturedfield>
<name>AttackName</name>
<type>string</type>
</capturedfield>
<capturedfield>
<name>Glancing</name>
<type>string</type>
</capturedfield>
<capturedfield>
<name>VictimName</name>
<type>string</type>
</capturedfield>
<capturedfield>
<name>Damage</name>
<type>integer</type>
</capturedfield>
<capturedfield>
<name>DamageType</name>
<type>string</type>
</capturedfield>
<capturedfield>
<name>BlockOrPen</name>
<type>string</type>
</capturedfield>
</matchfield>
<set>
<field>
<name>type</name>
<value>Swing</value>
<type>string</type>
</field>
</set>
</pattern>
</group>
<group>
<name>tswClient</name>
<id>50284625</id>
<matchfield>
<name>SourceModuleName</name>
<type>exact</type>
<value>vg_tsw_client</value>
</matchfield>
<pattern>
<id>2000</id>
<name>tswact load plugin</name>
<matchfield>
<name>Message</name>
<type>regexp</type>
<!-- [2017-02-10 05:47:07Z #3886] [ID:0] ERROR: Scaleform.TSWACT - TSWACT Loaded for |Sheriban| -->
<value>^\[([0-9-:]+)Z #\d+\] \[ID:\d+\] ERROR: Scaleform.TSWACT - TSWACT Loaded for - \|(\w+)\|</value>
<capturedfield>
<name>ParsedTime</name>
<type>string</type>
</capturedfield>
<capturedfield>
<name>PlayerName</name>
<type>string</type>
</capturedfield>
</matchfield>
<set>
<field>
<name>type</name>
<value>TswactLoaded</value>
<type>string</type>
</field>
</set>
</pattern>
<pattern>
<id>2001</id>
<name>tswact load playfield</name>
<matchfield>
<name>Message</name>
<type>regexp</type>
<!-- [2017-02-10 05:47:07Z #3886] [ID:0] ERROR: Scaleform.TSWACT - Playfield - |Kingsmouth Town| -->
<value>^\[([0-9-:]+)Z #\d+\] \[ID:\d+\] ERROR: Scaleform.TSWACT - Playfield - \|(\w+)\|</value>
<capturedfield>
<name>ParsedTime</name>
<type>datetime</type>
</capturedfield>
<capturedfield>
<name>ZoneName</name>
<type>string</type>
</capturedfield>
</matchfield>
<set>
<field>
<name>type</name>
<value>SetZoneName</value>
<type>string</type>
</field>
</set>
</pattern>
<pattern>
<id>2002</id>
<name>tswact enter combat</name>
<matchfield>
<name>Message</name>
<type>regexp</type>
<!-- [2017-02-10 05:00:22Z #10910] [ID:0] ERROR: Scaleform.TSWACT - Enter combat - |Sheriban|Buffs:Sprinting VI:Elemental Force:Third Degree :World Domination| -->
<value>^\[([0-9-:]+)Z #\d+\] \[ID:\d+\] ERROR: Scaleform.TSWACT - Enter combat - \|(\w+)\|</value>
<capturedfield>
<name>ParsedTime</name>
<type>datetime</type>
</capturedfield>
<capturedfield>
<name>PlayerName</name>
<type>string</type>
</capturedfield>
</matchfield>
<set>
<field>
<name>type</name>
<value>EnterCombat</value>
<type>string</type>
</field>
</set>
<exec>
$TestField = 'testValue';
</exec>
</pattern>
</group>
</patterndb>
Some of the output I'm getting:
{"EventReceivedTime":"2017-02-10 11:45:00","SourceModuleName":"vg_tsw_combat","SourceModuleType":"im_file","SyslogFacilityValue":1,"SyslogFacility":"USER","SyslogSeverityValue":5,"SyslogSeverity":"NOTICE","SeverityValue":2,"Severity":"INFO","EventTime":"2017-02-10 11:45:00","Hostname":"shepard","Message":"[11:45:00] Your Pop Shot hits (Normal) Undead Islander for 1437 physical damage. (Normal)","CriticalHit":"","AttackerName":"Your","AttackName":"Pop Shot","Glancing":"Normal","VictimName":"Undead Islander","Damage":1437,"DamageType":"physical","BlockOrPen":"Normal","PatternID":1000,"PatternName":"basic combat swing","type":"Swing"}
{"EventReceivedTime":"2017-02-10 11:45:00","SourceModuleName":"vg_tsw_combat","SourceModuleType":"im_file","SyslogFacilityValue":1,"SyslogFacility":"USER","SyslogSeverityValue":5,"SyslogSeverity":"NOTICE","SeverityValue":2,"Severity":"INFO","EventTime":"2017-02-10 11:45:00","Hostname":"shepard","Message":"[11:45:00] (Critical) Your Pop Shot hits (Normal) Undead Islander for 2965 physical damage. (Penetrated)","CriticalHit":"(Critical) ","AttackerName":"Your","AttackName":"Pop Shot","Glancing":"Normal","VictimName":"Undead Islander","Damage":2965,"DamageType":"physical","BlockOrPen":"Penetrated","PatternID":1000,"PatternName":"basic combat swing","type":"Swing"}
{"EventReceivedTime":"2017-02-10 11:45:00","SourceModuleName":"vg_tsw_combat","SourceModuleType":"im_file","SyslogFacilityValue":1,"SyslogFacility":"USER","SyslogSeverityValue":5,"SyslogSeverity":"NOTICE","SeverityValue":2,"Severity":"INFO","EventTime":"2017-02-10 11:45:00","Hostname":"shepard","Message":"[11:45:00] You gain buff Live Wire"}
{"EventReceivedTime":"2017-02-10 11:45:01","SourceModuleName":"vg_tsw_client","SourceModuleType":"im_file","SyslogFacilityValue":1,"SyslogFacility":"USER","SyslogSeverityValue":5,"SyslogSeverity":"NOTICE","SeverityValue":2,"Severity":"INFO","EventTime":"2017-02-10 11:45:01","Hostname":"shepard","Message":"[2017-02-10 16:45:01Z #18498] [ID:0] ERROR: Scaleform.TSWACT - Out of combat - |Sheriban|"}
{"EventReceivedTime":"2017-02-10 11:45:10","SourceModuleName":"vg_tsw_combat","SourceModuleType":"im_file","SyslogFacilityValue":1,"SyslogFacility":"USER","SyslogSeverityValue":5,"SyslogSeverity":"NOTICE","SeverityValue":2,"Severity":"INFO","EventTime":"2017-02-10 11:45:10","Hostname":"shepard","Message":"[11:45:10] Buff Live Wire terminated."}
Some of the vg_tsw_combat input file:
[11:45:00] Your One in the Chamber hits (Normal) Undead Islander for 231 physical damage. (Normal)
[11:45:00] Buff Sudden Return terminated on Undead Islander.
[11:45:00] Buff One in the Chamber terminated on Undead Islander.
[11:45:00] You gained 146 XP.
[11:45:00] Undead Islander died.
[11:45:00] Your Sudden Return hits (Normal) Undead Islander for 259 physical damage. (Normal)
[11:45:00] Your Pop Shot hits (Normal) Undead Islander for 2045 physical damage. (Penetrated)
[11:45:00] Your Pop Shot hits (Normal) Undead Islander for 2175 physical damage. (Penetrated)
[11:45:00] Your Pop Shot hits (Normal) Undead Islander for 1437 physical damage. (Normal)
[11:45:00] (Critical) Your Pop Shot hits (Normal) Undead Islander for 2965 physical damage. (Penetrated)
[11:45:00] You gain buff Live Wire
[11:45:02] You start using Sprinting VI.
[11:45:03] You gain buff Sprinting VI
[11:45:03] You successfully used Sprinting VI.
[11:45:10] Buff Live Wire terminated.
Some of the vg_tsw_client input:
[2017-02-10 16:33:43Z #6790] [ID:0] ERROR: Scaleform.TSWACT - TSWACT Loaded for |Sheriban|
[2017-02-10 16:33:43Z #6790] [ID:0] ERROR: Scaleform.TSWACT - Playfield - |The Savage Coast|
[2017-02-10 16:34:12Z #7313] [ID:0] ERROR: Scaleform.TSWACT - Enter combat - |Sheriban|Buffs:World Domination|
[2017-02-10 16:34:14Z #7373] [ID:0] ERROR: Scaleform.TSWACT - Out of combat - |Sheriban|
[2017-02-10 16:39:06Z #10609] [ID:0] ERROR: MagicCommand - Trying to prepone the execute timeline to the pass. Spell:7760057
[2017-02-10 16:39:06Z #10624] [ID:0] ERROR: Scaleform.TSWACT - Enter combat - |Sheriban|Buffs:Elemental Force:World Domination|
[2017-02-10 16:39:08Z #10655] [ID:0] ERROR: Scaleform.TSWACT - Out of combat - |Sheriban|
[2017-02-10 16:44:58Z #18330] [ID:0] ERROR: MagicCommand - Trying to prepone the execute timeline to the pass. Spell:7760057
[2017-02-10 16:44:59Z #18388] [ID:0] ERROR: Scaleform.TSWACT - Enter combat - |Sheriban|Buffs:Elemental Force:World Domination|
[2017-02-10 16:45:01Z #18498] [ID:0] ERROR: Scaleform.TSWACT - Out of combat - |Sheriban|
Any ideas?
progssilb created
NXLogClient Event captures Multiple platforms
kdevmu created
Hello There,
Please help me to get answers of below questions.
1. What all logs can be captured using NXLog client on Windows, Unix, AIX and Linux platforms?
2. What is the system prerequsites for installing NXLog client on Windows, UNIX, AIX and Linux platforms?
Thank you.
kdevmu created
Error when I install nxlog on REDHAT tikanga
Ely created
Hello ,
Do someone have an issue when install on REDHAT TIKANGA ??
Error message bellow mentioned :
~[root@osgdt01 tmp]# rpm -ivh nxlog-3.0.1814-1_rhel6.x86_64.rpm
error: nxlog-3.0.1814-1_rhel6.x86_64.rpm: Header V4 RSA/SHA1 signature: BAD, key ID 1da9e40e
error: nxlog-3.0.1814-1_rhel6.x86_64.rpm cannot be installed
anyone know what's signature: BAD ??
I have no idea for this error message , even I google it .
Thanks
Ely created
Writing WIndows Event Logs from Separate Servers to Multiple Files
mc63 created
Hello,
I am trying to setup NXLog so that multiple Windows Servers will send their Event logs to a central server, and that server will output them into basic text files. The logs are delivered to the central server just fine, but instead of going to separate routes as I have configured, all logs appear to be delivered to the same route, which happens to be the first one listed. Any help would be greatly appreciated.
## This is a sample configuration file. See the nxlog reference manual about the
## configuration options. It should be installed locally and is also available
## online at http://nxlog.org/docs/
## Please set the ROOT to the folder your nxlog was installed into,
## otherwise it will not start.
#define ROOT C:\Program Files\nxlog
define ROOT C:\Program Files (x86)\nxlog
Moduledir %ROOT%\modules
CacheDir %ROOT%\data
Pidfile %ROOT%\data\nxlog.pid
SpoolDir %ROOT%\data
LogFile %ROOT%\data\nxlog.log
########## BEGIN EXTENSIONS ##########
<Extension syslog>
Module xm_syslog
</Extension>
########## END EXTENSIONS ##########
########## BEGIN INPUTS ##########
<Input in_xxx>
Module im_tcp
Host 0.0.0.0
Port 514
</Input>
<Input in_yyy>
Module im_tcp
Host 0.0.0.0
Port 514
</Input>
<Input in_zzz>
Module im_tcp
Host 0.0.0.0
Port 514
</Input>
########## END INPUTS ##########
########## BEGIN OUTPUTS ##########
<Output out_xxx>
Module om_file
File "C:\\Logs\\xxx_NXLog.txt"
CreateDir FALSE
Truncate FALSE
OutputType LineBased
</Output>
<Output out_yyy>
Module om_file
File "C:\\Logs\\yyy_NXLog.txt"
CreateDir FALSE
Truncate FALSE
OutputType LineBased
</Output>
<Output out_zzz>
Module om_file
File "C:\\Logs\\zzz_NXLog.txt"
CreateDir FALSE
Truncate FALSE
OutputType LineBased
</Output>
########## END OUTPUTS ##########
########## BEGIN ROUTES ##########
<Route 1>
Path in_xxx => out_xxx
</Route>
<Route 2>
Path in_yyy => out_yyy
</Route>
<Route 3>
Path in_zzz => out_zzz
</Route>
########## END ROUTES ##########
Thank you.
mc63 created