I am trying to send parse each of these XML fields into a field for graylog to handle, any ideas would help. I've added <Extension xml> Module xm_xml </Extension> and Exec parse_windows_eventlog_xml(); to_xml(); but I'm not sure what else to do, I'm trying to work with this in the 'message' field The Federation Service validated a new credential. See XML for details. Activity ID: 494a36f8-9b89-4477-8676-0080000000e1 Additional Data XML: <?xml version="1.0" encoding="utf-16"?> <AuditBase xmlns:xsd="http://www.w3.org/2001/XMLSchema" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:type="FreshCredentialAudit"> <AuditType>FreshCredentials</AuditType> <AuditResult>Success</AuditResult> <FailureType>None</FailureType> <ErrorCode>N/A</ErrorCode> <ContextComponents> <Component xsi:type="ResourceAuditComponent"> <RelyingParty>https://xxxxxx.xxxxxxx.edu/adfs/services/trust</RelyingParty> <ClaimsProvider>AD AUTHORITY</ClaimsProvider> <UserId>UNIVERSITY\xxxxxxxxxxxxxx</UserId> </Component> <Component xsi:type="AuthNAuditComponent"> <PrimaryAuth>N/A</PrimaryAuth> <DeviceAuth>false</DeviceAuth> <DeviceId>N/A</DeviceId> <MfaPerformed>false</MfaPerformed> <MfaMethod>N/A</MfaMethod> <TokenBindingProvidedId>false</TokenBindingProvidedId> <TokenBindingReferredId>false</TokenBindingReferredId> <SsoBindingValidationLevel>NotSet</SsoBindingValidationLevel> </Component> <Component xsi:type="ProtocolAuditComponent"> <OAuthClientId>N/A</OAuthClientId> <OAuthGrant>N/A</OAuthGrant> </Component> <Component xsi:type="RequestAuditComponent"> <Server>https://xxxxx.xxxxxx.edu/adfs/services/trust</Server> <AuthProtocol>WSFederation</AuthProtocol> <NetworkLocation>Intranet</NetworkLocation> <IpAddress>x.x.94.22</IpAddress> <ForwardedIpAddress>x.x.128.226</ForwardedIpAddress> <ProxyIpAddress>N/A</ProxyIpAddress> <NetworkIpAddress>N/A</NetworkIpAddress> <ProxyServer>N/A</ProxyServer> <UserAgentString>Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:77.0) Gecko/20100101 Firefox/77.0</UserAgentString> <Endpoint>/adfs/ls/</Endpoint> </Component> </ContextComponents> </AuditBase>

Hello, I am collecting syslog from many different firewalls and I want to extract only a single field from these messages and output that to a file separate of the output file I am currently writing. Basically I want to match the field devname= and extract it and output the results into a list. Is there a way to avoid duplication? So I have firewall1, firewall2, firewall3 etc. and they are in the devname= field so in the log file it has devname="firewall1", devname="firewall2", devname="firewall3" etc. I am wondering how can I look for all these names and create a list of them. Thank you.

I have tried multiple ways to get the wildcard includes to work but just cant get it to work in CE. If I put the path in then it works. define ROOT C:\Program Files (x86)\nxlog define PLUGIN %ROOT%\additional works: include C:\Program Files (x86)\nxlog\additional\nxlog_exchange.conf doesn't work: include additional*.conf include additional/.conf include %PLUGIN%*.conf include %PLUGIN%/.conf

Hi, My working nxlog.conf relies on Query directives aimed at explicitly named Channels together with the im_msvistalog Module. <Input blu_eventlog_iis> Module im_msvistalog Query &lt;QueryList&gt;\ &lt;Query Id=&quot;0&quot;&gt;\ &lt;Select Path=&quot;Microsoft-IIS-Configuration/Administrative&quot;&gt;*&lt;/Select&gt;\ &lt;Select Path=&quot;Microsoft-IIS-Configuration/Operational&quot;&gt;*&lt;/Select&gt;\ &lt;Select Path=&quot;Microsoft-IIS-Logging/Logs&quot;&gt;*&lt;/Select&gt;\ &lt;/Query&gt;\ &lt;/QueryList&gt; # Filter noise from IIS schema issues Exec if ($Message =~ /Unable to find schema/) drop(); # Workaround for local time so as to standardize to absolute microseconds since epoch Exec $EventTime = integer($EventTime); # JSON is required because some Windows logs contain new-line characters. Exec $Message = to_json(); </Input> <Output blu_out_eventlog_iis> Module om_tcp Host %SIEM% Port %PORT% Exec to_syslog_bsd(); </Output> <Route route_eventlog_iis> Path blu_eventlog_iis => blu_out_eventlog_iis </Route> I hoped to clean up some 15007 errors in the nxlog.log files included below: 2020-07-06 14:42:55 ERROR failed to subscribe to msvistalog events,the channel was not found [error code: 15007]; The specified channel could not be found. I suspect the 15007 errors are being generated because IIS isn't setup. In other words, the named IIS Channels are queried but not found. Is it possible to add an If/Else statement as part of those Query directives? Looking over the NXlog CE Manual I only see If/Else statement used with Exec. Any guidance would be greatly appreciated!

Hi all, I must send the event logs (only event ID 4626 and 4625) in the log collector Linux. can you help me with configuration? Thank you.

I'm very new to NXLog and have no clue how to fix this. I was setting up an nxlog.conf file to one of our servers and we keep getting this error. It starts with the 'not enough' fields error but then after a few lines it says 'Too many fields'. There are 21 fields on the W3c fields, but I'm not sure why it seems like it's adding Fields as an input? I added an additional string in the Fieldtypes and also combined date-time instead of 'date, time' but didn't work. Is there a way to fix this in the Conf file? Or should I be looking in the IIS? Log expected 20, got 6 in input '#Software: Microsoft Internet Information Services 8.5' expected 20, got 2 in input '#Version: 1.0' expected 20, got 3 in input '#Date: 2020-06-30 08:35:19' Too many fields in CSV input, expected 20, got 21 in input '#Fields: date time c-ip c-port cs-username s-sitename s-computername cs-host s-ip s-port cs-method cs-uri-stem sc-status sc-win32-status sc-substatus sc-bytes cs-bytes time-taken x-session x-fullpath x-debug' Conf <Extension w3c_ftp> Module xm_csv Fields date, time, c-ip, cs-username, s-sitename, s-computername, s-ip, cs-method, cs-uri-stem, sc-status, sc-win32-status, sc-bytes, cs-bytes, time-taken, s-port, cs-host, sc-substatus, x-session, x-fullpath, x-debug, c-port FieldTypes string, string, string, string, string, string, string, string, string, string, string, string, string, string, string, string, string, string, string, string, string Delimiter ' ' </Extension> TIA!

Hi, I'm trying to send my custom jsotn to GrayLog. My json file is: { "Faxes": "68", "Last30DaysEmails": "9728", "TodayEmails": "66" } This is my nxlog config file: define ROOT C:\Program Files (x86)\nxlog Moduledir %ROOT%\modules CacheDir %ROOT%\data Pidfile %ROOT%\data\nxlog.pid SpoolDir %ROOT%\data LogFile %ROOT%\data\nxlog.log <Extension gelf> Module xm_gelf </Extension> <Extension multiline> Module xm_multiline HeaderLine /\d{4}-\d{2}-\d{2}\s\d*:\d\d:\d*,\d{3}\s[a-zA-Z]*\s\s\S\s\[Begin Lead\]/ EndLine /\d{4}-\d{2}-\d{2}\s\d*:\d\d:\d*,\d{3}\s[a-zA-Z]*\s\s\S\s\[End Lead\]/ </Extension> <Extension json> Module xm_json </Extension> <Input im_file> Module im_file File "C:\\logs\\myfile.json" InputType multiline SavePos FALSE ReadFromLast FALSE </Input> <Output om_udp> Module om_udp Host 192.168.1.100 Port 3514 OutputType GELF </Output> <Route 1> Path im_file => om_udp </Route> Now my problems is: When I send the file to GrayLog server I see multi line with different messages (as json variable), and not all in a single message. How can do to configure correctly my system? Thanks for the support. Marco

Hi, I've want to include nxlog to send all Windows logs to syslog plus Microsoft-Windows-Windows Defender/Operational, however as "HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\EventLog" doesn't seem to include "Microsoft-Windows-Windows Defender" it doesn't read that config in at start and send those Windows Defender logs. How can I get nxlog to send everything it does now, plus Microsoft-Windows-Windows Defender logs, without having to define everything? The only way I can see to do this is with two im_msvistalog defines, i.e. define ROOT C:\Program Files (x86)\nxlog define CERTDIR %ROOT%\cert Moduledir %ROOT%\modules CacheDir %ROOT%\data Pidfile %ROOT%\data\nxlog.pid SpoolDir %ROOT%\data LogFile %ROOT%\data\nxlog.log <Extension syslog> Module xm_syslog </Extension> <Input eventlog> Module im_msvistalog </Input> <Input eventlog_defender> Module im_msvistalog Query <QueryList> <Query Id="0"> <Select Path="Microsoft-Windows-Windows Defender/Operational">*</Select> </Query> </QueryList> </Input> <Processor eventlog_transformer> Module pm_transformer Exec $Hostname = hostname(); OutputFormat syslog_rfc3164 </Processor> <Output syslogout> Module om_ssl [removed, but its to Papertrail] </Output> <Route 1> Path eventlog => eventlog_transformer => syslogout </Route> <Route 2> Path eventlog_defender => eventlog_transformer => syslogout </Route> That seems to work except there are three problems compared to when I only have the Microsoft-Windows-Windows Defender going and the other blocks deleted: The Windows Defender logging is a little inconsistent, testing with an EICAR test virus doesn't always appear, probably 50% chance it appears in the syslog (all events appear in Windows log) The severity and source is copied from the previous log line (usually info rather than warn severity, and not Microsoft-Windows-Windows Defender), maybe a threading issue? Each rows of text is in its own line rather than all in a single message which happens when its the only logging to send Is there a better way to do 'everything' plus Microsoft-Windows-Windows Defender?

I am trying to reduce the size of the message sent from my Windows event logs to graylog and I cannot for the life of me figuring out how tell it to drop certain fields The only thing I can figure out is that i'm supposed to use delete() but how to use and where to place it in my config is very frustrating.

I'm currently using nxlog to filter and forward syslog: Source => Filter Logs on intermediate server with nxlog installed => forward udp 514 (syslog). Config looks like the following: <Extension _syslog> Module xm_syslog </Extension> #syslog to Nxlog01 <Input _sys_in> Module im_udp Port 514 <Exec> if (($Message =~ /REGEX/) or ($Message =~ /REGEX/)) drop(); </Exec> Exec if $MessageSourceAddress != "1.1.1.1" drop(); </Input> <Processor _sys_norepeat> Module pm_norepeat CheckFields Hostname, Message, SourceName </Processor> #Syslog Output to DST server <Output _sys_out> Module om_udp Host dstserver_ip Port 514 Exec to_syslog_bsd(); </Output> <Route Logs> Path _sys_in => _sys_norepeat => _sys_out </Route> Firstly is this possible with the CE agent? I already performed a tcpdump on the intermediate server where nxlog is installed and i can see the syslog being received but not sent after filtered. There is udp 514 connectivity between nxlog server and destination server. I tried: Removing all regex filtering (since i thought my regex wasn't good enough), no results. Changed im_udp to im_tcp (i thought maybe you can't used the same port in both input and output modules). Checked whether there actually is network connectivity between nxlog server and destination server using nc by sending a udp 514 packet to dst server. Can anyone help with this?

Hi All, Needed help with parsing/modify. Would greatly appreciate some direction. At the moment, I am parsing a plain-test log-file and sending to a remote server: Jun 19 16:29:28 server12345 [...] Jun 19 16:29:28 server12345 --- Jun 19 16:29:27 server12345 [program.state :3371] The above is what I get with parsing. I was hoping to make every line transform like this Jun 19 16:29:28 server12345 **programName1** [...] Jun 19 16:29:28 server12345 **programName1** --- Jun 19 16:29:27 server12345 **programName1** [program.state :3371] Can someone help me understand, how I can get programName1 appended to each file, after the server hostname? Also, is it possible to parse the server-name, and replace it as follows: Jun 19 16:29:28 **server12345--NA** programName1 [...] Jun 19 16:29:28 **server12345--NA** programName1 --- Jun 19 16:29:27 **server12345--NA** programName1 [program.state :3371] Would appreciate some guidance on how to change this via config. I currently, am reading in a textfile via the om_file method. Update: Relevant snippets of config: <Input log_file> Module im_file File 'C:\program\var\log\file.log' #InputType multiline_parser Exec parse_syslog(); </Input> <Processor norepeat> Module pm_norepeat CheckFields Hostname, Message, SourceName OutputFormat syslog_rfc3164 </Processor> <Route log_output> Path log_file => norepeat => log_output </Route> <Output log_output> Module om_udp Host x.x.x.x Port 514 </Output>

Hello - new to nxlog. Trying to end nxlog process when EOF reached on input file. Any help appreciated. Thanks, Andrew

Hello, I am fairly new to NXlog and we are trying to understand if It's possible to route/forward logs based on the Source when only one input module for 514 traffic is definied. I understand that the Host directive in the Input module is actually related to the host itself where Nxlog is installed. Would adding some kind of Exec instructions in there help?? Perhaps using a processor in the route module? Since this is a new deployment, we'd like to set it up by following best practices from the get go. I appreciate your help.

Hi all, I have an issue when view agent internal logs "View Log" in NXLog Manager. When I clicked on the View Log button, it will pop up error message as follow: Error Executing the requested operation: Could not parse XML; nested exception is org.xml.sax.SAXParseException; lineNumber:x, columnNumber:x; Character reference "&#. Error Executing the requested operation: Could not parse XML; nested exception is org.xml.sax.SAXParseException; lineNumber:x, columnNumber:x; The element type "SOAP-ENV:Envelope" must be terminated by matching end-tag"</SOAP-ENV:Envelope>" Does anyone know what are those two errors about, and how should we fix it ? Where can we view / find the SOAP-ENV:Envelope or mentioned character from NXLog Manger? Thanks.

Good Morning All, I was hoping to get some direction with a log file we want to parse. We have a directory containing log files on a network share. A new log file is created every day. The issue with these logs is that there are no newline or carriage returns, it is one gigantic line. New events are just added at the end of the string. I am familiar with NXLog to the point of inputs / outputs / routes but I am not sure exactly my next step here. I have some regex that when ran manually will break the log down into individual lines but I am not sure how to implement that in NXLog in a way that it will read the log file, split the entirety of it into individual lines and then export those lines without seeing a bunch of duplication. Or in what blocks to implement which step. Some broad stroke guidance (or details) would be appreciated.

Hi Guys, I have a problem, I must to send the Event log a the LogCollecotr Linux, the problem is when Logcollector riceved the log, the formatting is much long, I want riceved an log so. Example Log LInux: " Jun 19 16:23:01 MSI su: pam_unix(su:session): session opened for user ema by (uid=0) " I can receive the log in this format?? Thank you.

Hi Team, We are using Nxlog to send logs to RSA(SIEM), but few of the security logs are not sending to RSA. Below are the event ids we are not receiving: Event ids starting with 4860- 4890. Below is the configuration which we are using in RSA. Can you please check below configuration and let me know if anything needs to be changed to receive the windows security and application logs. define ROOT C:\Program Files (x86)\nxlog Moduledir %ROOT%\modules CacheDir %ROOT%\data Pidfile %ROOT%\data\nxlog.pid SpoolDir %ROOT%\data LogFile %ROOT%\data\nslog.log <Extension syslog> Module xm_syslog </Extension> <Input in> Module im_msvistalog <QueryXML> <QueryList> <Query Id='0'> <Select Path='Application'></Select> <Select Path='Security'></Select> </Query> </QueryList> </QueryXML> </Input> <Output out> Module om_tcp Host hostname(hided) Port 514 Exec to_syslog_snare();$raw_event = replace($raw_event, "\t", ','); </Output> <Route 1> Path in => out </Route>

Trialing the NXLOG CE version i forward logs for MS Windows Eventlog, now i found just over 200 events are logged while there are over 50 000. It there a way to configured nxlog so it consumes the logs older than today ?

We are using nxLog for sending all kind the logging information to a Graylog. This is working for both the SQL Server error log and SQL audit logs but also for other none SQL Server logs. We where initially looking into using SQL audit for getting the application name of the user connection but we are running SQL Server 2016 and it is only available starting with SQL Server 2017. Instead we have tried to use extended events for the logging. The basic ideas are working and when opening the extended events file in SSMS we can see the requested information. The problem is that the extended event files are binaries and we can't use the same type of nxLog configuration as for error log or SQL audit logs. Instead we have been trying to use etw_classic_sync_target and Event Tracing for Windows(ETW). We are using nxLog module im_etw for collecting the ETW information. I have verified the nxLog configuration (nxLog -v) and basically the configuration seems to be working and some events are sent forward. The problem is that it is not the events from our extended event session! The reason most likely is that we are not using the correct ETW provider. I have not been able to find any information of what provider that SQL Server and extended events are using. I have tried to most obvious ones as e.g. sqlserver but none seems to be the correct. Does anyone know the name of the ETW provider that the extended event etw_classic_sync_target is using? OR does someonw see some other reason why this is not working? We have set up the nxLog module as this <Input xe_etw> Module im_etw Provider sqlserver </Input>

I am having problems with the nxlog manager. The application takes forever to load the login page and when it does load once it finally logs in it just hangs and will not load any of the dashboards. When checking the systemctl status I have seen a couple different errors. Exception in thread "agentServerTaskExecutor-145" java.lang.OutOfMemoryError: Java heap space Jun 09, 2020 8:55:23 AM org.apache.jasper.compiler.JDTJavaCompiler setSourceVM WARNING: Unknown source VM 1.7 ignored. : ERROR: 'XML document structures must start and end within the same entity.' Jun 09, 2020 9:51:11 AM com.sun.xml.internal.messaging.saaj.soap.EnvelopeFactory createEnvelope SEVERE: SAAJ0511: Unable to create envelope from given source service.sh[1395]: 2020-06-09 10:55:39.820:WARN:oejh.HttpParser:qtp1329552164-8630: BadMessage: 400 Illegal character for HttpChannelOverHttp@76b3d195{r=0,a=IDLE,uri=-} Exception in thread "C3P0PooledConnectionPoolManager[identityToken->31989haa16jlcxi18f6us8|6366ebe0]-HelperThread-#1" java.lang.OutOfMemoryError: Java heap space