Hello,

I need to read logfiles from oracle, which are structured in xml with attributes

It's nxlog still unable to access the attributes?

I've read here that the entrerprise edition does it 

https://nxlog.co/announcing-nxlog-enterprise-edition-v30

Handling structured data formats better

The xm_xml extension has been enhanced so that it can now parse nested XML and data stored in XML attributes. Parsing of nested JSON has been also implemented in xm_json and UTF-8 validation can be enforced in order to avoid parser failures caused by invalid UTF-8 in other tools.

This is a feature of the enterprise edition only or it will be ported to community edition too?

I'm trying to get a PatternDB working correctly, and it looks like I'm getting some fields but not all of them. There's only one pattern that's actually generating extra fields, and even it is dropping the first field (ParsedDate). Not sure what's going on here...

Config file (via file inclusion):

<Extension json>
    Module      xm_json
</Extension>

<Extension syslog>
    Module      xm_syslog
</Extension>

<Input vg_tsw_client>
  Module     im_file
  File       "C:\Program Files (x86)\Steam\steamapps\common\The Secret World\ClientLog.txt"
  Exec       if not ($raw_event =~ /Scaleform\.TSWACT/) drop();
  Exec         parse_syslog();
</Input>

<Input vg_tsw_combat>
  Module     im_file
  File       "C:\Program Files (x86)\Steam\steamapps\common\The Secret World\CombatLog-*.txt"
  Exec       if ($raw_event =~ /Sprinting [VI]+/) drop();
  Exec         parse_syslog();
</Input>

<Processor vg_tsw_pattern>
    Module    pm_pattern
    PatternFile %ROOT%\conf\SecretWorld\patterndb.xml
</Processor>

<Output vg_tsw_testfile>
  Module     om_file
  File       "C:\\ProgramData\\nxlogs\\vg-tsw-logs.log"
  Exec       to_json();
</Output>

<Route vg_tsw_route>
  Path       vg_tsw_client, vg_tsw_combat => vg_tsw_pattern => vg_tsw_testfile
</Route>

Pattern DB:

<?xml version='1.0' encoding='UTF-8'?>
<patterndb>
 <created>2010-01-01 01:02:03</created>
 <version>42</version>
 
<group>
  <name>tswCombat</name>
  <id>50284624</id>
  <matchfield>
    <name>SourceModuleName</name>
    <type>exact</type>
    <value>vg_tsw_combat</value>
  </matchfield>

  <pattern>
    <id>1000</id>
    <name>basic combat swing</name>

    <matchfield>
     <name>Message</name>
     <type>regexp</type>
        <!-- [00:00:28] (Critical) Solomon County Cop's Spray and Pray hits (Normal) Ravenous Horde for 522 physical damage. (Normal) -->
     <value>^\[([^\]]+)\] ((?:\(Critical\) |\(Normal\) )?)(.+?'s|Your) (.+?) hits \((Normal|Glancing)\) (.*?) for (\d+) (physical|magical) damage. \((Normal|Penetrated|Blocked)\)</value>
     <capturedfield>
      <name>ParsedTime</name>
      <type>datetime</type>
     </capturedfield>
     <capturedfield>
      <name>CriticalHit</name>
      <type>string</type>
     </capturedfield>
     <capturedfield>
        <name>AttackerName</name>
        <type>string</type>
     </capturedfield>
     <capturedfield>
        <name>AttackName</name>
        <type>string</type>
     </capturedfield>
     <capturedfield>
        <name>Glancing</name>
        <type>string</type>
     </capturedfield>
     <capturedfield>
        <name>VictimName</name>
        <type>string</type>
     </capturedfield>
     <capturedfield>
        <name>Damage</name>
        <type>integer</type>
     </capturedfield>
     <capturedfield>
        <name>DamageType</name>
        <type>string</type>
     </capturedfield>
     <capturedfield>
        <name>BlockOrPen</name>
        <type>string</type>
     </capturedfield>
    </matchfield>

    <set>
     <field>
       <name>type</name>
       <value>Swing</value>
       <type>string</type>
     </field>
    </set>
  </pattern>
</group>

<group>
  <name>tswClient</name>
  <id>50284625</id>
  <matchfield>
    <name>SourceModuleName</name>
    <type>exact</type>
    <value>vg_tsw_client</value>
  </matchfield>
 
  <pattern>
    <id>2000</id>
    <name>tswact load plugin</name>

    <matchfield>
     <name>Message</name>
     <type>regexp</type>
        <!-- [2017-02-10 05:47:07Z #3886] [ID:0] ERROR: Scaleform.TSWACT - TSWACT Loaded for |Sheriban| -->
     <value>^\[([0-9-:]+)Z #\d+\] \[ID:\d+\] ERROR: Scaleform.TSWACT - TSWACT Loaded for - \|(\w+)\|</value>
     <capturedfield>
      <name>ParsedTime</name>
      <type>string</type>
     </capturedfield>
     <capturedfield>
      <name>PlayerName</name>
      <type>string</type>
     </capturedfield>
    </matchfield>

    <set>
     <field>
       <name>type</name>
       <value>TswactLoaded</value>
       <type>string</type>
     </field>
    </set>
  </pattern>
 
  <pattern>
    <id>2001</id>
    <name>tswact load playfield</name>

    <matchfield>
     <name>Message</name>
     <type>regexp</type>
        <!-- [2017-02-10 05:47:07Z #3886] [ID:0] ERROR: Scaleform.TSWACT - Playfield - |Kingsmouth Town| -->
     <value>^\[([0-9-:]+)Z #\d+\] \[ID:\d+\] ERROR: Scaleform.TSWACT - Playfield - \|(\w+)\|</value>
     <capturedfield>
      <name>ParsedTime</name>
      <type>datetime</type>
     </capturedfield>
     <capturedfield>
      <name>ZoneName</name>
      <type>string</type>
     </capturedfield>
    </matchfield>

    <set>
     <field>
       <name>type</name>
       <value>SetZoneName</value>
       <type>string</type>
     </field>
    </set>
  </pattern>

  <pattern>
    <id>2002</id>
    <name>tswact enter combat</name>

    <matchfield>
     <name>Message</name>
     <type>regexp</type>
        <!-- [2017-02-10 05:00:22Z #10910] [ID:0] ERROR: Scaleform.TSWACT - Enter combat - |Sheriban|Buffs:Sprinting VI:Elemental Force:Third Degree :World Domination| -->
     <value>^\[([0-9-:]+)Z #\d+\] \[ID:\d+\] ERROR: Scaleform.TSWACT - Enter combat - \|(\w+)\|</value>
     <capturedfield>
      <name>ParsedTime</name>
      <type>datetime</type>
     </capturedfield>
     <capturedfield>
      <name>PlayerName</name>
      <type>string</type>
     </capturedfield>
    </matchfield>

    <set>
     <field>
       <name>type</name>
       <value>EnterCombat</value>
       <type>string</type>
     </field>
    </set>
    <exec>
      $TestField = 'testValue';
    </exec>
  </pattern>
</group>

</patterndb>

Some of the output I'm getting:

{"EventReceivedTime":"2017-02-10 11:45:00","SourceModuleName":"vg_tsw_combat","SourceModuleType":"im_file","SyslogFacilityValue":1,"SyslogFacility":"USER","SyslogSeverityValue":5,"SyslogSeverity":"NOTICE","SeverityValue":2,"Severity":"INFO","EventTime":"2017-02-10 11:45:00","Hostname":"shepard","Message":"[11:45:00] Your Pop Shot hits (Normal) Undead Islander for 1437 physical damage. (Normal)","CriticalHit":"","AttackerName":"Your","AttackName":"Pop Shot","Glancing":"Normal","VictimName":"Undead Islander","Damage":1437,"DamageType":"physical","BlockOrPen":"Normal","PatternID":1000,"PatternName":"basic combat swing","type":"Swing"}
{"EventReceivedTime":"2017-02-10 11:45:00","SourceModuleName":"vg_tsw_combat","SourceModuleType":"im_file","SyslogFacilityValue":1,"SyslogFacility":"USER","SyslogSeverityValue":5,"SyslogSeverity":"NOTICE","SeverityValue":2,"Severity":"INFO","EventTime":"2017-02-10 11:45:00","Hostname":"shepard","Message":"[11:45:00] (Critical) Your Pop Shot hits (Normal) Undead Islander for 2965 physical damage. (Penetrated)","CriticalHit":"(Critical) ","AttackerName":"Your","AttackName":"Pop Shot","Glancing":"Normal","VictimName":"Undead Islander","Damage":2965,"DamageType":"physical","BlockOrPen":"Penetrated","PatternID":1000,"PatternName":"basic combat swing","type":"Swing"}
{"EventReceivedTime":"2017-02-10 11:45:00","SourceModuleName":"vg_tsw_combat","SourceModuleType":"im_file","SyslogFacilityValue":1,"SyslogFacility":"USER","SyslogSeverityValue":5,"SyslogSeverity":"NOTICE","SeverityValue":2,"Severity":"INFO","EventTime":"2017-02-10 11:45:00","Hostname":"shepard","Message":"[11:45:00] You gain buff Live Wire"}
{"EventReceivedTime":"2017-02-10 11:45:01","SourceModuleName":"vg_tsw_client","SourceModuleType":"im_file","SyslogFacilityValue":1,"SyslogFacility":"USER","SyslogSeverityValue":5,"SyslogSeverity":"NOTICE","SeverityValue":2,"Severity":"INFO","EventTime":"2017-02-10 11:45:01","Hostname":"shepard","Message":"[2017-02-10 16:45:01Z #18498] [ID:0] ERROR: Scaleform.TSWACT - Out of combat - |Sheriban|"}
{"EventReceivedTime":"2017-02-10 11:45:10","SourceModuleName":"vg_tsw_combat","SourceModuleType":"im_file","SyslogFacilityValue":1,"SyslogFacility":"USER","SyslogSeverityValue":5,"SyslogSeverity":"NOTICE","SeverityValue":2,"Severity":"INFO","EventTime":"2017-02-10 11:45:10","Hostname":"shepard","Message":"[11:45:10] Buff Live Wire terminated."}

Some of the vg_tsw_combat input file:

[11:45:00] Your One in the Chamber hits (Normal) Undead Islander for 231 physical damage. (Normal)
[11:45:00] Buff Sudden Return terminated on Undead Islander.
[11:45:00] Buff One in the Chamber terminated on Undead Islander.
[11:45:00] You gained 146 XP.
[11:45:00] Undead Islander died.
[11:45:00] Your Sudden Return hits (Normal) Undead Islander for 259 physical damage. (Normal)
[11:45:00] Your Pop Shot hits (Normal) Undead Islander for 2045 physical damage. (Penetrated)
[11:45:00] Your Pop Shot hits (Normal) Undead Islander for 2175 physical damage. (Penetrated)
[11:45:00] Your Pop Shot hits (Normal) Undead Islander for 1437 physical damage. (Normal)
[11:45:00] (Critical) Your Pop Shot hits (Normal) Undead Islander for 2965 physical damage. (Penetrated)
[11:45:00] You gain buff Live Wire
[11:45:02] You start using Sprinting VI.
[11:45:03] You gain buff Sprinting VI
[11:45:03] You successfully used Sprinting VI.
[11:45:10] Buff Live Wire terminated.

Some of the vg_tsw_client input:

[2017-02-10 16:33:43Z #6790] [ID:0] ERROR: Scaleform.TSWACT - TSWACT Loaded for |Sheriban|
[2017-02-10 16:33:43Z #6790] [ID:0] ERROR: Scaleform.TSWACT - Playfield - |The Savage Coast|
[2017-02-10 16:34:12Z #7313] [ID:0] ERROR: Scaleform.TSWACT - Enter combat - |Sheriban|Buffs:World Domination|
[2017-02-10 16:34:14Z #7373] [ID:0] ERROR: Scaleform.TSWACT - Out of combat - |Sheriban|
[2017-02-10 16:39:06Z #10609] [ID:0] ERROR: MagicCommand - Trying to prepone the execute timeline to the pass. Spell:7760057
[2017-02-10 16:39:06Z #10624] [ID:0] ERROR: Scaleform.TSWACT - Enter combat - |Sheriban|Buffs:Elemental Force:World Domination|
[2017-02-10 16:39:08Z #10655] [ID:0] ERROR: Scaleform.TSWACT - Out of combat - |Sheriban|
[2017-02-10 16:44:58Z #18330] [ID:0] ERROR: MagicCommand - Trying to prepone the execute timeline to the pass. Spell:7760057
[2017-02-10 16:44:59Z #18388] [ID:0] ERROR: Scaleform.TSWACT - Enter combat - |Sheriban|Buffs:Elemental Force:World Domination|
[2017-02-10 16:45:01Z #18498] [ID:0] ERROR: Scaleform.TSWACT - Out of combat - |Sheriban|

Any ideas?

Hello There,

Please help me to get answers of below questions.

1. What all logs can be captured using NXLog client on Windows, Unix, AIX and Linux platforms?

2. What is the system prerequsites for installing NXLog client on Windows, UNIX, AIX and Linux platforms?

Thank you.

Hello  ,

Do someone have an issue when install on REDHAT TIKANGA ?? 

Error message bellow mentioned :

~[root@osgdt01 tmp]# rpm -ivh nxlog-3.0.1814-1_rhel6.x86_64.rpm
error: nxlog-3.0.1814-1_rhel6.x86_64.rpm: Header V4 RSA/SHA1 signature: BAD, key ID 1da9e40e
error: nxlog-3.0.1814-1_rhel6.x86_64.rpm cannot be installed

anyone know what's signature: BAD ??

I have no idea for this error message , even I google it .

Thanks

Hello,

I am trying to setup NXLog so that multiple Windows Servers will send their Event logs to a central server, and that server will output them into basic text files. The logs are delivered to the central server just fine, but instead of going to separate routes as I have configured, all logs appear to be delivered to the same route, which happens to be the first one listed. Any help would be greatly appreciated.

## This is a sample configuration file. See the nxlog reference manual about the
## configuration options. It should be installed locally and is also available
## online at http://nxlog.org/docs/

## Please set the ROOT to the folder your nxlog was installed into,
## otherwise it will not start.

#define ROOT C:\Program Files\nxlog
define ROOT C:\Program Files (x86)\nxlog

Moduledir %ROOT%\modules
CacheDir %ROOT%\data
Pidfile %ROOT%\data\nxlog.pid
SpoolDir %ROOT%\data
LogFile %ROOT%\data\nxlog.log

########## BEGIN EXTENSIONS ##########

<Extension syslog>
    Module  xm_syslog
</Extension>

########## END EXTENSIONS ##########

########## BEGIN INPUTS ##########

<Input in_xxx>
    Module      im_tcp
    Host 0.0.0.0
    Port 514
</Input>

<Input in_yyy>
    Module      im_tcp
    Host 0.0.0.0
    Port 514
</Input>

<Input in_zzz>
    Module      im_tcp
    Host 0.0.0.0
    Port 514
</Input>

########## END INPUTS ##########

########## BEGIN OUTPUTS ##########

<Output out_xxx>
    Module      om_file
    File "C:\\Logs\\xxx_NXLog.txt"
    CreateDir FALSE
    Truncate FALSE
    OutputType LineBased
</Output>

<Output out_yyy>
    Module      om_file
    File "C:\\Logs\\yyy_NXLog.txt"
    CreateDir FALSE
    Truncate FALSE
    OutputType LineBased
</Output>

<Output out_zzz>
    Module      om_file
    File "C:\\Logs\\zzz_NXLog.txt"
    CreateDir FALSE
    Truncate FALSE
    OutputType LineBased
</Output>

########## END OUTPUTS ##########

########## BEGIN ROUTES ##########

<Route 1>
    Path        in_xxx => out_xxx
</Route>

<Route 2>
    Path        in_yyy => out_yyy
</Route>

<Route 3>
    Path        in_zzz => out_zzz
</Route>

########## END ROUTES ##########

Thank you.

if we setup a function to identify the IP of the client server and based on that answer, to then forward logs based on that IP info

Two questions, I am attempting to install the nxlog-ce via powershell and the process hangs at the accept eula screen and also it seems the way to install requires copying over the default configuration file after install.

- Is there a flag I can pass to accept the eula?

- Is there a way to pass the path to the config file at install to automatically overwrite the config at installation time without stop/starting the process?

i.e. nxlog.msi /accepteula /install /quiet/ /conf=\\path\to\conf

So to add, if I do:

msiexec /i nxlog.msi /quiet

It will install quietly, but if I launch it with Start-Process msiexec -ArguementList "/i nxlog.msi /quiet", it will still launch the EULA splash again. So, not sure if that is a bug in PowerShell, but would still like to pass the config file at installation without having to overwrite it.

I'm trying to get nxlog to read from a mysql table and output any changes from the last table read to a text file in csv tab delineated format.  Right now all it's doing is injecting multiple carriage returns into the text file with no text.  Am I heading in the right direction or have I totally borked the config?  I'm working with the following config:

<Extension csv>
        Module xm_csv
        Fields $facility, $severity, $hostname, $timestamp, $application, $message
        FieldTypes string, string, string, string, string, string
        Delimiter \t
</Extension>

<Input dbiin>
    Module im_dbi
        SavePos TRUE
        Driver mysql
        Option host localhost
        Option username USERNAME
        Option dbname DBNAME
        Option password PASSWORD
        SQL SELECT facility, severity, hostname, timestamp, application, message FROM table
</Input>

<Output out>
       Module om_file
       File "/var/log/test.txt"
</Output>

<Route 1>

      Path dbiin => out

</Route>

Hello. I have a question.

I get multiline messages

how can I combine into a single line, multiline message ??

for example this message, In this message 4 lines

Jul 21 17:59:10 <14> 1 2016-07-04T00: 53: 02.000000 + 03: 00 node = sec-sflow type = SYSCALL msg = audit (1467579182.055: 3248181): arch = 111

2 syscall = success = yes exit = 4 a0 = 7fc7783127a8 a1 = 2 a2 = a3 = 0 8 items = 1 ppid = 11013 pid = 30363 auid = 0 0 uid = gid = 0 = 0 euid

suid = 0 fsuid = 0 = 0 egid sgid = 0 = 0 fsgid tty = (none) ses = 28 comm = "sshd"

exe = "/ usr / sbin / sshd" key = "root_action"

Thank!

Hello!

It appears that any nested data - e.g. from EventData - will be overwritten if the field exists on the event itself.

For example, please see your documentation on sysmon.  Notice that ProcessID is a field on the event, and is also a field under EventData, albeit with different data.

The resulting JSON output includes only the ProcessID from the event itself, not from the eventdata.  In the example at the link, notice that the Event.ProcessID is 1680.  The Event.EventID.ProcessID is 25848.  Notice that the data from the latter (generally more specific to this type of event, and thus generally more important) is not available as structured data anywhere.

Personally I'm not using this at the moment, but, I could see many situations where the generic Event fields overwrite valuable information from Event Data.

Cheers!

Hello,

Could you please clarify how can I cut out some fields from forwarded event?

My situation is the following;

I have a local log file on the server where installed nxlog agent. Using im_file module I have defined path to file and filename. After that I configured to forward this log to remote syslog server. When I opened forwarded log on the romete syslog server and find out that my log line was changed. It was added time and server name wehere original log file is stored. I have posted a line from the remote server and marked columns which were added during the forwarding.

Jan 12 13:16:28 siem-vm Jan 12 00:01:37 mail2-vm-srv postfix/cleanup[7412]: 6EC1E2A23F9: message-id=<20170111220136.5AE682A23F6>

Can you help me?

Thank you in advance.

Hi, can anyone help me with the output of my nxlog.conf
I want to convert epoch time from my Bro logs;

Part of the logs:

1482865199.693051 FSYupp4bmRs8tT5Jyg 3 5A00020E4289E78C695848......
1482865200.300809 FmXyl22Uxsq1cudDd8 3 5A00020E4289E78C695848......
1482865200.203542 FAuSUU3X9pgdSJ2D2g 3 5A00020E4289E78C695848.......
1482865201.043722 F0KUdW3Nm5edyqPXLl 3 0CEAC9CAD430F24F334575.......

My current settings are

<Output o.name.log>
 Module om_tcp
 Host xx.xxx.xxx.xxx
 Port xxxx
 OutputType LineBased
</Output>

Thanks!

I'm attempting to demo xnlog and running into a problem where the Windows Server 2016 event logs are being sent to AWS ElasticSearch Service with the EventTime being a string. This basically renders it impossible to index the logs, as the Kibana board requires a time-field name and is not recongizing the string as a datetime.  Any suggestions on this, or is this a potential bug with Server 2016?

Hi everybody,

stumbled over a problem that sometimes I get hostnames from nxlog other times it's fqdns. Happens only with internal nxlog messages.I tired to fix this by using the Exec $Hostname = hostname_fqdn(); statement.

# Nxlog internal logs
<Input internal>
   Module im_internal
   Exec $EventReceivedTime = integer($EventReceivedTime) / 1000000; to_json();
   Exec $Hostname = hostname_fqdn(); 
</Input>

Still get messages with designation nxlog: internal that have a source_host with hostname only and not fqdn.

Either I am missing something really obvious or something is broken. Any guidance to troubleshoot or figure out what is going on much appreciated

Best regards

Tobias

I have a NXLog service running in Windows Server shipping event logs. It has 2 destinations, 1 is TCP sending logs to syslog_ng and another is GLEF UDP.

When my syslog_ng server goes offline, the logs I'm receiving at the GLEF UDP output also stops. Is there any way to make NXlog send the logs to the other output/route even if one output/route fails?

Config:

​
    Module      om_tcp
    Host        192.168.1.11
    Port        25002
    Exec        to_syslog_snare();


 
    Module      om_udp
    Host        192.168.1.12
    Port        51416
    OutputType  GELF



    Path        in => out



    Path        in => analyze

Hi everybody,

while experimenting with nxlog and relaying windows event logs I stumbled over the issue that even in the latest versions the

field Keywords from the Window log is defined in ms_msevent as integer which doesn't fit the values stored in the field in windows.

Are there any plans to fix this?

best regards

Tobias

Given a number of application logs sharing the same HeaderLine and EndLine regular expressions we are trying out a xm_multiline with im_file config using wild cards. 

<Extension multi>
  Module      xm_multiline
    HeaderLine /^\d{4}-\d{2}-\d{2} \d{2}:\d{2}:\d{2},\d{3} @ batch_task\._init_logger : \[INFO\]\+ /
    EndLine /^\d{4}-\d{2}-\d{2} \d{2}:\d{2}:\d{2},\d{3} @ run_batch\.<module> : \[INFO\]- /
</Extension>

<Input inPython>
    Module    im_file
    File "C:\\data\\server1\\*.log"
    InputType multi
    Exec $FileName = file_name();
</Input>

It works consistently without wildcards pointed at one file.
It works intermittently with wildcards pointed at multiple files being written to concurrenlty.

I'm wondering if this is a supported use case. i.e. multiline events from wildcarded files being written to concurrenlty. Or should we be specifiying each input file individually?

thanks,

Rob

does nxlog-2.9.1716 still uses LibExpat v2.0.1 and LibPCRE v8.02?

Impact:
LibPCRE v8.02 is vulnerable to DoS and code overflow.
LibExpat v2.0.1 has 4 publicly identified vulnerabilities.

References

https://www.cvedetails.com/vulnerability-list/vendor_id-12037/product_id-22545/version_id-129378/Libexpat-Expat-2.0.1.html
https://www.cvedetails.com/vulnerability-list/vendor_id-3265/product_id-5715/version_id-191791/Pcre-Pcre-8.02.html

is it possible to update LibExpat to v2.1.0 and LibPCRE to v8.39?

I want to compile nxlog and package it on windows, but I can't find any material to refer to. Who can help me, give me some advice. Thank you!

Hi everyone ! 

In our implementation of NxLog on our systems we are looking at some ways to centralize config management & deployement on all agents. As nxLog is integrated with some other software, we are looking at some automatic features like scripts or so to deploy new config on the agents all in a secure way. 

If anyone has any tips on how to do so, help would be greatly appraciated. We first thought to use NxLog manager but it doesn't really fit our need as it requires the administrator to actually do some handy work and they are really lazy to be honest ;-)

Thank you guys in advance if you can give us some tips :)

Regards

J. Denis