I need to convert epoch time in seconds but get year 1970. I have 1421079464 and epochconverter tells me it is GMT: Mon, 12 Jan 2015 16:17:44 GMTYour time zone: 12. jaanuar 2015 18:17:44 GMT+2:00 but when I use Exec         if $raw_event =~ /\<TimeStamp\>(.+)\</ {$epochtimetmp = $1; $epochtime=integer($epochtimetmp)*1000; $EventTime = datetime($epochtime);} I get: EventTime: 1970-01-17 12:44:39 Can somebody please tell me what is wrong here?  

Hello, Our application writes logs in JSON format so it's quite straightforward to send them to Elasticsearch using om_http module. However we need to enrich JSON logs with additional information like application name. I was searching for the solution and found that I could do the following: <Output elasticsearch>     Module      om_http     URL            (server_url)     ContentType application/json     Exec        set_http_request_path(strftime(now(), "/test-%Y.%m/log/"));     Exec        parse_json(); $Application="MyApp"; to_json(); </Output> The last line in the output specification make sure the json payload is first parsed and then generated again, enriched with a new field "Application". I wonder if this is a right approach or there are other alternatives. Thanks in advance  

I'm struggling to find the download location for the source version of nxlog-ce-2.9.1347 at https://nxlog.co/products/nxlog-community-edition/download There seem to be links to binary versions for various platforms, and the source of the previous version (nxlog-ce-2.8.1248.tar.gz), but not source for the latest version. Is this an error, or will the source for nxlog-ce-2.9.1347  not be published? Thanks!

The following config is working for me to send windows event logs only for a specific Event Log source application, but it is sending duplicate messages for every Event Viewer event/entry. Can anybody see what the problem is with my config (below)? Running nxlog-ce-2.8.1248 define ROOT C:\Program Files (x86)\nxlog Moduledir %ROOT%\modules CacheDir %ROOT%\data Pidfile %ROOT%\data\nxlog.pid SpoolDir %ROOT%\data LogFile %ROOT%\data\nxlog.log <Extension syslog> Module xm_syslog </Extension> <Input eventlog1> Module im_msvistalog ReadFromLast TRUE Exec if (($Channel =~ /Application/) AND ($SourceName =~ /My app name/)); </Input> <Input otherapp_log1> Module im_file File 'D:\Path\to\my\log\output.log' SavePos TRUE ReadFromLast TRUE PollInterval 1 Exec $Message = $raw_event; $SyslogFacilityValue = 22; </Input> <Output graylog2> Module om_tcp Host 10.x.x.x Port 514 Exec to_syslog_bsd(); </Output> <Route 1> Path eventlog1, otherapp_log1 => graylog2 </Route>

When using NXLog as syslog collector, it's a good idea filter messages by hostname and to construct file name/path based on it and info in the message (facility and/or program name). What I thought about is something like this: <Input tcp_in> Module im_tcp Port 514 Exec parse_syslog_ietf(); Exec if $SyslogFacility == 'MAIL' { \ $Filetype = 'mail'; \ } else if $SyslogFacility == 'FTP' { \ $Filetype = 'ftp'; \ } else { \ $Filetype = 'messages'; \ } </Input> <Output remotelog_out> Module om_file CreateDir TRUE File "/var/log/" + $Hostname + "/" + $Filetype + ".log" </Output> <Route> tcp_in => remotelog_out </Route> It works, but performance is ... The problem seems to be that om_file can keep only one file open and for every new message if filename doesn't match already open one, the old one is closed and the new one is opened. This isn't of course optimal - in case of many hosts and log types significant amount of time is spent in closing/opening. Is there a better way? I'm holding it wrong? :)

I'n using nxlog to send logs from Windows Event Log to rsyslogd. To identify my Windows hosts, i use local6 facility. So they are logged to a separate folder based on IP and facility: local6 messages are logged to /var/log/hosts/windows/SERVER_IP/remote.log, all other facilities (non-local6) are logged to /var/hosts/linux/SERVER_IP/remote.log The thing is, i sometimes find new messages in /var/log/hosts/linux/WINSERVER_IP/remote.log, which are (mostly) a part of other windows Event Log messages. Looks like big messages from Event Log are split into fragments, the first one gets the local6 facility (via SyslogFacilityValue) and is logged to windows-log folder (correct behavior), the rest get another facility value (which is unknown for me) and are logged to linux-log folder (incorrect behavior). My nxlog config (part of it): <Extension syslog>                 Module xm_syslog </Extension> <Input in>     Module      im_msvistalog                 Exec $SyslogFacilityValue = 22; </Input>   <Processor eventlog_transformer>                 Module pm_transformer                 Exec $Hostname = hostname();                 OutputFormat syslog_rfc5424 </Processor>   <Output out>     Module      om_tcp     Host        10.7.1.20     Port        514                 Exec $raw_event = replace($raw_event, "\r\n", " ");                 Exec $raw_event = replace($raw_event, "\t", " "); </Output>   <Route 1>     Path        in => eventlog_transformer => out </Route> Is there a way to strictly specify local6 facility to all outgoing syslog-type messages and their parts, if they get split?  

How can we use nxlog to send syslog or text-logs (/var/log/somelog.log) to Apache Flume?

hi, i configure nxlog as describe in documentation and i configure graylog gelf input. i open a ticket to graylog (https://github.com/Graylog2/graylog2-server/issues/984) and they say maybe is a nxlog missconfig this is myconfig, how can i diagnose the issue from your side? define ROOT C:\Program Files (x86)\nxlog Moduledir %ROOT%\modules CacheDir %ROOT%\data Pidfile %ROOT%\data\nxlog.pid SpoolDir %ROOT%\data LogFile %ROOT%\data\nxlog.log <Extension gelf> Module xm_gelf </Extension> <Input in> # Use 'im_mseventlog' for Windows XP, 2000 and 2003 Module im_msvistalog # Uncomment the following to collect specific event logs only  #Query <QueryList>\  #<Query Id="0">\  #<Select Path="Application">*</Select>\  #<Select Path="System">*</Select>\  #<Select Path="Security">*</Select>\  #</Query>\   #</QueryList> </Input> <Output out> Module om_tcp Host 192.168.0.95 Port 12201 OutputType GELF </Output> <Route r> Path in => out </Route>

Since om_ssl has an option 'OutputType' to send GELF logs over ssl connections this seems to be missing in om_http. Is there any chance to add this in the future, since sending logs over http adds an extra layer of safety because of the acknowledgement after receiving a log entry.

I'm looking at ways to get high precision (with fractions of seconds) timestamps out of nxlog. If application provides these, it's of course easy to get this data into nxlog, but it's not easy to get it out - strftime function you can format timestamps with doesn't support fractions of seconds. Am I right? It's even more problematic if application doesn't provide high precision timestamps - which is the case with most of unix programs using im_uds or all data from im_kernel for example. Even EventReceivedTime timestamp used in case of IETF syslog format isn't high precision. Now, before anyone says that these wouldn't very precise timestamps anyway - that's not what I really care about. But as in the path to the log analysis events can be reordered (with redundant message brokers and stuff), it is critical to have high precision timestamps so correct order of the messages can be restored.  Ideally I'd like to use BSD syslog with high precision timestamps, but in any way it seems to be impossible at the moment. Or am I overlooking something?

I'm having trouble getting my IIS logs into Graylog. My Windows event logs show up just fine, but the IIS logs never show up.   I'm using NXlog per the Graylog docs. I've verified that the logs are getting sent (I have them also writing to a file, and I've checked with Wireshark to make sure the packets are being sent). At one point I changed from using a GELF input to a raw input, and then the messages showed up but of course were unreadable as they were still in the compressed GELF format, like so: x����n�0E��*���Í�r� �$���
#�m"�����$ȿ��h9q\����J�g43�zA;h9a5J��xh�V��b���@Z��Ƕ�^���A�z:t����[Vv$��:�S��j�&�[b�>��)�������M�a�����+��vb�Ji̦�����\@%E���f��b����W��`�X��`:)���hX+P��Si�V�ɡ�'9�ݲgB)vcdz.��.ٞ[w��8�ky�L�Kk�4��pC��c'L�����폑E�#X3(٥�m۲ �H� ?r�|k�׸%��lkh����C������3��É�'ua~<l�l��z!SmWM�g��"�̦�j�o�DVp�7*�%G��Q��c"G��B���̼�(���    �`*�z�GN(�N�k]e xZU�iu�A�|ړ��Z�[���A+���C����&����}�&���'����d�Б��̻_��m�0�������]��9<ޭ������Ub�,�U�n�Q��8��F��(J�%.U?��^��wJ�a   I can send the messages in completely raw, but then all the data is stuck in one field.   Why would Graylog accept my GELF-formatted Windows event logs, but not my GELF-formatted IIS logs?   I'm running the latest Graylog VMware OVA, and the only changes I've made were changing the password and timezone, enforcing HTTPS, and setting up LDAP and my inputs.   Here's my NXlog config file:   define ROOT C:\Program Files (x86)\nxlog   Moduledir %ROOT%\modules CacheDir %ROOT%\data Pidfile %ROOT%\data\nxlog.pid SpoolDir %ROOT%\data LogFile %ROOT%\data\nxlog.log   <Extension gelf>     Module         xm_gelf </Extension>   <Extension fileop>     Module         xm_fileop </Extension>    <Extension json>     Module      xm_json </Extension>   # Create the parse rule for IIS logs. You can copy these from the header of the IIS log file. <Extension w3c>     Module             xm_csv     Fields             $date, $time, $s-ip, $cs-method, $cs-uri-stem, $cs-uri-query, $s-port, $cs-username, $c-ip, $csUser-Agent, $cs-Referer, $sc-status, $sc-substatus, $sc-win32-status, $time-taken     FieldTypes         string, string, string, string, string, string, integer, string, string, string, string, integer, integer, integer, integer     Delimiter         ' '     QuoteChar         '"'     EscapeControl     FALSE     UndefValue         - </Extension>   <Input iis>     Module        im_file     File        "C:\\inetpub\\logs\\LogFiles\\W3SVC12\\u_ex*"     SavePos      TRUE       Exec        if $raw_event =~ /^#/ drop();                    \                 else                                             \                 {                                                \                     w3c->parse_csv();                            \                     $EventTime = parsedate($date + " " + $time); \                     $SourceName = "IIS";                         \                     $Message = to_json();                         \                 } </Input>   <Input eventlog>     Module      im_msvistalog </Input>   <Output graylog>     Module      om_udp     Host        graylog     Port        12201     OutputType    GELF       #Use the following line for debugging (uncomment the fileop extension above as well)     Exec file_write("C:\\Program Files (x86)\\nxlog\\data\\nxlog_output.log", $raw_event); </Output> #<Route eventlog> #    Path        eventlog => graylog #</Route> <Route iis-to-graylog>     Path        iis => graylog </Route>     Any assistance will be greatly appreciated.   

Having a reoccurring problem where I get the following error and my output file quits updating for a few minutes then picks back up again. As a side note, I was having a problem with the eventlog because of an entry that was being written that had an object serialized as xml in the message. That is the exclusion in my application log. Since then, the event log has been working flawlessly. I found a similar post except that in my instance I have already use a more specific filter. http://nxlog.org/question/531/iis-log-bad-characters-causes-nxlog-crash-causing-random-event-dump Log 2015-03-06 13:10:32 INFO nxlog-ce-2.8.1248 started 2015-03-06 13:14:07 ERROR if-else failed at line 70, character 336 in C:\Program Files (x86)\nxlog\conf\nxlog.conf. statement execution has been aborted; procedure 'parse_csv' failed at line 70, character 226 in C:\Program Files (x86)\nxlog\conf\nxlog.conf. statement execution has been aborted; Not enough fields in CSV input, expected 17, got 0 in input '' 2015-03-06 13:28:30 ERROR if-else failed at line 70, character 336 in C:\Program Files (x86)\nxlog\conf\nxlog.conf. statement execution has been aborted; procedure 'parse_csv' failed at line 70, character 226 in C:\Program Files (x86)\nxlog\conf\nxlog.conf. statement execution has been aborted; Not enough fields in CSV input, expected 17, got 0 in input '' 2015-03-06 13:50:07 ERROR if-else failed at line 70, character 336 in C:\Program Files (x86)\nxlog\conf\nxlog.conf. statement execution has been aborted; procedure 'parse_csv' failed at line 70, character 226 in C:\Program Files (x86)\nxlog\conf\nxlog.conf. statement execution has been aborted; Not enough fields in CSV input, expected 17, got 0 in input '' 2015-03-06 14:07:39 ERROR if-else failed at line 70, character 336 in C:\Program Files (x86)\nxlog\conf\nxlog.conf. statement execution has been aborted; procedure 'parse_csv' failed at line 70, character 226 in C:\Program Files (x86)\nxlog\conf\nxlog.conf. statement execution has been aborted; Not enough fields in CSV input, expected 17, got 0 in input '' Line 70 refers to the closing bracket in <Input default_iis_in>  Module im_file  File "D:\inetpub\logs\LogFiles\W3SVC1\u_ex*.log"     ReadFromLast TRUE     Exec if $raw_event=~/^#/ drop();                         \    else                                                \    {                                                   \     w3c->parse_csv();                               \     $EventTime = parsedate($date + " " + $time);    \    } </Input> Full config ## This is a sample configuration file. See the nxlog reference manual about the ## configuration options. It should be installed locally and is also available ## online at http://nxlog.org/nxlog-docs/en/nxlog-reference-manual.html ## Please set the ROOT to the folder your nxlog was installed into, ## otherwise it will not start. #define ROOT C:\Program Files\nxlog define ROOT C:\Program Files (x86)\nxlog Moduledir %ROOT%\modules CacheDir %ROOT%\data Pidfile %ROOT%data\nxlog.pid SpoolDir %ROOT%\data LogFile %ROOT%\data\nxlog.log <Extension w3c>     Module      xm_csv     Fields $date, $time, $s-ip, $cs-method, $cs-uri-stem, $cs-uri-query, $s-port, $cs-username, $c-ip, $cs(User-Agent), $cs(Referer), $sc-status, $sc-substatus, $sc-win32-status, $sc-bytes, $cs-bytes, $time-taken     FieldTypes  string, string, string, string, string, string, string, string, string, string, string, string, string, string, string, string, string     Delimiter ' '     QuoteChar   '"'     EscapeControl FALSE     UndefValue  - </Extension> <Extension json>     Module xm_json </Extension> <Extension fileop>     Module      xm_fileop     <Schedule>         Every   10 min         Exec    file_truncate("evt.json");     </Schedule>     <Schedule>         Every   10 min         Exec    file_truncate("iis-default.json");     </Schedule> </Extension> <Input evt_in>     Module      im_msvistalog     Query       <QueryList>\                     <Query Id="0">\                         <Select Path="Application">*</Select>   \                     </Query>\                 </QueryList>     ReadFromLast TRUE     Exec        if $SourceName=~/TfsDeployer/ drop();           \                 else                                            \                 {                                               \                     to_json();                                  \                 } </Input> <Input default_iis_in>  Module im_file  File "D:\inetpub\logs\LogFiles\W3SVC1\u_ex*.log"     ReadFromLast TRUE     Exec if $raw_event=~/^#/ drop();                         \    else                                                \    {                                                   \     w3c->parse_csv();                               \     $EventTime = parsedate($date + " " + $time);    \    } </Input> <Output evt_out>     Module om_file     File "evt.json" </Output> <Output default_iis_out>     Module om_file     Exec $raw_event = to_json();     File "iis-default.json" </Output> <Route 1>     Path        evt_in => evt_out </Route> <Route 2>     Path default_iis_in => default_iis_out </Route>   Any help would be appreciated. Thank you.

Hello, I'm finding sometimes the message "non-active modification on ....<input_filename>".  Could anyone give us explanations about it ? My nxlog is v2.8.1248 on Windows Server 2008 R2. Thanks Patrick  

Hi, I'm trying to foward log events with NXlog to logstash over a SSL connection. Therefore, I generated my own CA certificate and a certificate to be used by logstash for the SSL connection. NXlog is installed on a Ubuntu 14.04 (virtual) machine with the CA certificate added as '/etc/ssl/certs/logstash-ca.crt'. I tested the connection with openssl s_client -CAfile /etc/ssl/certs/logstash-ca.cert -connect localhost:5000 and everything works fine (all entered text lines are received by logstash). However, when I start NXlog with the appropriate om_ssl output configuration it fails with ERROR SSL certificate verification failed: unable to get local issuer certificate (err: 20) My NXlog configuration looks like this: User nxlog Group nxlog LogFile /var/log/nxlog/nxlog.log LogLevel DEBUG # Modules <Extension _syslog> Module xm_syslog </Extension> <Extension json> Module xm_json </Extension> # Nxlog internal logs <Input internal> Module im_internal Exec $EventReceivedTime = integer($EventReceivedTime) / 1000000; to_json(); </Input> <Output sslout> Module om_ssl Host localhost Port 5000 CertFile "/etc/ssl/certs/logstash-ca.crt" OutputType LineBased </Output> <Route logstash> Path internal => sslout </Route>   When I start NXlog with sudo nxlog -f -c nxlog.conf the debug output looks like this: 2015-03-05 20:37:05 DEBUG module sslout got 1 poll events 2015-03-05 20:37:05 DEBUG Module sslout can read 2015-03-05 20:37:05 DEBUG [...] 2015-03-05 20:37:05 DEBUG PROCESS_EVENT: READ (sslout) 2015-03-05 20:37:05 DEBUG om_ssl read 2015-03-05 20:37:05 DEBUG doing handshake 2015-03-05 20:37:05 DEBUG verify callback (ok: 0) 2015-03-05 20:37:05 DEBUG preverification returned non-OK: unable to get local issuer certificate 2015-03-05 20:37:05 DEBUG STOP: sslout 2015-03-05 20:37:05 DEBUG remove socket [21] 2015-03-05 20:37:05 DEBUG executing statements 2015-03-05 20:37:05 DEBUG [...] 2015-03-05 20:37:05 INFO reconnecting in 1 seconds 2015-03-05 20:37:05 ERROR [om_ssl.c:532/io_err_handler()] -;[ssl.c:168/nx_ssl_check_io_error()] SSL certificate verification failed: unable to get local issuer certificate (err: 20) I have no idea why the verification work with OpenSSL but fails with NXlog (I'm using nxlog-ce-2.8.1248 and OpenSSL version is 1.0.1f). Any help is highly appreciated.  

define ROOT C:\Program Files (x86)\nxlog Moduledir %ROOT%\modules CacheDir %ROOT%\data Pidfile %ROOT%\data\nxlog.pid SpoolDir %ROOT%\data LogFile %ROOT%\data\nxlog.log #### Extensions - load and configure global extendsions <Extension eW3c>     Module xm_csv     Fields $date, $time, $s-ip, $cs-method, $cs-uri-stem, $cs-uri-query, $s-port, $cs-username, $c-ip, $csUser-Agent, $sc-status, $sc-substatus, $sc-win32-status, $time-taken     FieldTypes string, string, string, string, string, string, integer, string, string, string, integer, integer, integer, integer     Delimiter ' '     QuoteChar '"'     EscapeControl FALSE     UndefValue - </Extension> <Extension eFileOp>     Module xm_fileop </Extension> #### #### Inputs <Input iIIS-fulfillment.Api.1>     Module    im_file     File     "D:\\APShared\\Log\\W3SVC1\\\\*"     SavePos    TRUE      #    Exec if $raw_event =~ /^#/ drop();     Exec file_write("C:\\temp\\nxlog_RawIIS.log", $raw_event+"some text man"); \         eW3c->parse_csv(); \         $SourceName = "IIS-fulfillment.Api"; \         file_write("C:\\temp\\nxlog_IIS-csv.log", $raw_event); </Input> #### #### Processor #### #### Outputs <Output oIISFile>     module    om_file     File    "c:\\temp\\nx_iis.out" </Output> #### #### Routes <Route rIISFile>     path     iIIS-fulfillment.Api.1 => oIISFile </Route> NXLOG.log 2015-03-04 12:13:45 INFO nxlog-ce-2.8.1248 started 2015-03-04 12:14:11 ERROR procedure 'parse_csv' failed at line 40, character 94 in C:\Program Files (x86)\nxlog\conf\nxlog.conf. statement execution has been aborted; Not enough fields in CSV input, expected 14, got 0 in input '' "C:\\temp\\nxlog_RawIIS.log" contains: some text man   C:\\temp\\nxlog_IIS-csv.log - File not created. Help.  Where's my $raw_event data?

I have xml style messages that can have thousands of <CUID>xxx</CUID> statements in them so I need to cut out from the middle of the message: from the first string of <CUID> to the last </CUID>. Is this possible?

Hello, I have followed instructions to install nxlog in a Windows Server Core 2012 machine (no UI only command prompt). However, when I try to run the service I get: service could not be started Do you know what I could try to fix this? This Windows is the instance created automatically by Amazon Beanstalk and without a UI I find this difficult to debug.. so any suggestion is appreciated Thanks

Dear NXlog community, I am using nxlog on a windows 2003 environment and i am having some problems with windows failed authentication events.  All entry's with EventID 675 contain the AccountName "SYSTEM" in stead of the username that the failed authentication is for. I couldnt get it to work with pattern matching in nxlog but as i have never used this before i am probably doing something wrong. I would really like to get some statistics of this and get the user name in the AccountName field. For example kibana is reporting: AccountName        SYSTEM AccountType        User Category        Account Logon CategoryNumber        9 Domain            NT AUTHORITY EventID            675 EventType        AUDIT_FAILURE FileName        Security Hostname        SomeHostName Severity        ERROR SeverityValue        4 SourceModuleName    eventlog SourceModuleType    im_mseventlog SourceName        Security host            SomeHostName.SomeDomain message            Pre-authentication failed:              User Name: [username]              User ID: %{some user id}              Service Name: krbtgt/office Pre-Authentication              Type: 0x0 Failure Code: 0x19              Client Address: [ip address]   Any help is appreciated!

Hello, I am trying to connect through SSL to ship logs but nxlog only attempts to connect through SSLv3, not TLS. Any way to force TLS 1.1 or 1.2 handshakes? Thank you

I'm noticing something weird with my NXLog endpoints. If I have my NXLog agents pull windows event logs and transfer them over the network it loses the event time somewhere along the way. My current setup looks like this: Remote Agent: Windows Event Log -> im__mseventlog -> om_SSL (Type Binary) (to Log Collector) Log Collector: im_SSL (Type Binary) -> om_UDP (Type GELF) (to Graylog server) When Graylog is offline it will buffer correctly, but after Graylog comes back online, NXlog will send the logs with the current time.