Ask questions. Get answers. Find technical product solutions from passionate experts in the NXLog community.

High precision timestamps
I'm looking at ways to get high precision (with fractions of seconds) timestamps out of nxlog. If application provides these, it's of course easy to get this data into nxlog, but it's not easy to get it out - strftime function you can format timestamps with doesn't support fractions of seconds. Am I right? It's even more problematic if application doesn't provide high precision timestamps - which is the case with most of unix programs using im_uds or all data from im_kernel for example. Even EventReceivedTime timestamp used in case of IETF syslog format isn't high precision. Now, before anyone says that these wouldn't very precise timestamps anyway - that's not what I really care about. But as in the path to the log analysis events can be reordered (with redundant message brokers and stuff), it is critical to have high precision timestamps so correct order of the messages can be restored.  Ideally I'd like to use BSD syslog with high precision timestamps, but in any way it seems to be impossible at the moment. Or am I overlooking something?

ttyserial created
Replies: 2
View post »
last updated
IIS logs sent via NXlog not showing up in Graylog
I'm having trouble getting my IIS logs into Graylog. My Windows event logs show up just fine, but the IIS logs never show up.   I'm using NXlog per the Graylog docs. I've verified that the logs are getting sent (I have them also writing to a file, and I've checked with Wireshark to make sure the packets are being sent). At one point I changed from using a GELF input to a raw input, and then the messages showed up but of course were unreadable as they were still in the compressed GELF format, like so: x����n�0E��*���Í�r � �$���#�m"�����$ȿ��h9q\����J�g43�zA;h9a5J��xh�V��b���@Z��Ƕ�^���A�z:t����[Vv$��:�S��j�&�[b�>��)�������M�a�����+��vb�Ji̦�����\@%E���f��b����W��`�X��`:)���hX+P��Si�V�ɡ�'9�ݲgB)vcdz.��.ٞ[w��8�ky�L�Kk�4��pC��c'L�����폑E�#X3(٥�m۲ �H� ?r�|k�׸%��lkh����C������3��É�'u a~<l�l��z!SmWM�g��"�̦�j�o�DVp�7*�%G��Q��c"G��B���̼�(���    �`*�z�GN(�N�k] e xZU�iu�A�|ړ��Z�[���A+���C����&����}�&���'����d�Б��̻_��m�0�������]��9<ޭ������Ub�,�U�n�Q��8��F��(J�%.U?��^��wJ�a   I can send the messages in completely raw, but then all the data is stuck in one field.   Why would Graylog accept my GELF-formatted Windows event logs, but not my GELF-formatted IIS logs?   I'm running the latest Graylog VMware OVA, and the only changes I've made were changing the password and timezone, enforcing HTTPS, and setting up LDAP and my inputs.   Here's my NXlog config file:   define ROOT C:\Program Files (x86)\nxlog   Moduledir %ROOT%\modules CacheDir %ROOT%\data Pidfile %ROOT%\data\nxlog.pid SpoolDir %ROOT%\data LogFile %ROOT%\data\nxlog.log   <Extension gelf>     Module         xm_gelf </Extension>   <Extension fileop>     Module         xm_fileop </Extension>    <Extension json>     Module      xm_json </Extension>   # Create the parse rule for IIS logs. You can copy these from the header of the IIS log file. <Extension w3c>     Module             xm_csv     Fields             $date, $time, $s-ip, $cs-method, $cs-uri-stem, $cs-uri-query, $s-port, $cs-username, $c-ip, $csUser-Agent, $cs-Referer, $sc-status, $sc-substatus, $sc-win32-status, $time-taken     FieldTypes         string, string, string, string, string, string, integer, string, string, string, string, integer, integer, integer, integer     Delimiter         ' '     QuoteChar         '"'     EscapeControl     FALSE     UndefValue         - </Extension>   <Input iis>     Module        im_file     File        "C:\\inetpub\\logs\\LogFiles\\W3SVC12\\u_ex*"     SavePos      TRUE       Exec        if $raw_event =~ /^#/ drop();                    \                 else                                             \                 {                                                \                     w3c->parse_csv();                            \                     $EventTime = parsedate($date + " " + $time); \                     $SourceName = "IIS";                         \                     $Message = to_json();                         \                 } </Input>   <Input eventlog>     Module      im_msvistalog </Input>   <Output graylog>     Module      om_udp     Host        graylog     Port        12201     OutputType    GELF       #Use the following line for debugging (uncomment the fileop extension above as well)     Exec file_write("C:\\Program Files (x86)\\nxlog\\data\\nxlog_output.log", $raw_event); </Output> #<Route eventlog> #    Path        eventlog => graylog #</Route> <Route iis-to-graylog>     Path        iis => graylog </Route>     Any assistance will be greatly appreciated.   

Nathan.Reid created
Replies: 1
View post »
last updated
if-else failed, not enough fields
Having a reoccurring problem where I get the following error and my output file quits updating for a few minutes then picks back up again. As a side note, I was having a problem with the eventlog because of an entry that was being written that had an object serialized as xml in the message. That is the exclusion in my application log. Since then, the event log has been working flawlessly. I found a similar post except that in my instance I have already use a more specific filter. http://nxlog.org/question/531/iis-log-bad-characters-causes-nxlog-crash-causing-random-event-dump Log 2015-03-06 13:10:32 INFO nxlog-ce-2.8.1248 started 2015-03-06 13:14:07 ERROR if-else failed at line 70, character 336 in C:\Program Files (x86)\nxlog\conf\nxlog.conf. statement execution has been aborted; procedure 'parse_csv' failed at line 70, character 226 in C:\Program Files (x86)\nxlog\conf\nxlog.conf. statement execution has been aborted; Not enough fields in CSV input, expected 17, got 0 in input '' 2015-03-06 13:28:30 ERROR if-else failed at line 70, character 336 in C:\Program Files (x86)\nxlog\conf\nxlog.conf. statement execution has been aborted; procedure 'parse_csv' failed at line 70, character 226 in C:\Program Files (x86)\nxlog\conf\nxlog.conf. statement execution has been aborted; Not enough fields in CSV input, expected 17, got 0 in input '' 2015-03-06 13:50:07 ERROR if-else failed at line 70, character 336 in C:\Program Files (x86)\nxlog\conf\nxlog.conf. statement execution has been aborted; procedure 'parse_csv' failed at line 70, character 226 in C:\Program Files (x86)\nxlog\conf\nxlog.conf. statement execution has been aborted; Not enough fields in CSV input, expected 17, got 0 in input '' 2015-03-06 14:07:39 ERROR if-else failed at line 70, character 336 in C:\Program Files (x86)\nxlog\conf\nxlog.conf. statement execution has been aborted; procedure 'parse_csv' failed at line 70, character 226 in C:\Program Files (x86)\nxlog\conf\nxlog.conf. statement execution has been aborted; Not enough fields in CSV input, expected 17, got 0 in input '' Line 70 refers to the closing bracket in <Input default_iis_in>  Module im_file  File "D:\inetpub\logs\LogFiles\W3SVC1\u_ex*.log"     ReadFromLast TRUE     Exec if $raw_event=~/^#/ drop();                         \    else                                                \    {                                                   \     w3c->parse_csv();                               \     $EventTime = parsedate($date + " " + $time);    \    } </Input> Full config ## This is a sample configuration file. See the nxlog reference manual about the ## configuration options. It should be installed locally and is also available ## online at http://nxlog.org/nxlog-docs/en/nxlog-reference-manual.html ## Please set the ROOT to the folder your nxlog was installed into, ## otherwise it will not start. #define ROOT C:\Program Files\nxlog define ROOT C:\Program Files (x86)\nxlog Moduledir %ROOT%\modules CacheDir %ROOT%\data Pidfile %ROOT%data\nxlog.pid SpoolDir %ROOT%\data LogFile %ROOT%\data\nxlog.log <Extension w3c>     Module      xm_csv     Fields $date, $time, $s-ip, $cs-method, $cs-uri-stem, $cs-uri-query, $s-port, $cs-username, $c-ip, $cs(User-Agent), $cs(Referer), $sc-status, $sc-substatus, $sc-win32-status, $sc-bytes, $cs-bytes, $time-taken     FieldTypes  string, string, string, string, string, string, string, string, string, string, string, string, string, string, string, string, string     Delimiter ' '     QuoteChar   '"'     EscapeControl FALSE     UndefValue  - </Extension> <Extension json>     Module xm_json </Extension> <Extension fileop>     Module      xm_fileop     <Schedule>         Every   10 min         Exec    file_truncate("evt.json");     </Schedule>     <Schedule>         Every   10 min         Exec    file_truncate("iis-default.json");     </Schedule> </Extension> <Input evt_in>     Module      im_msvistalog     Query       <QueryList>\                     <Query Id="0">\                         <Select Path="Application">*</Select>   \                     </Query>\                 </QueryList>     ReadFromLast TRUE     Exec        if $SourceName=~/TfsDeployer/ drop();           \                 else                                            \                 {                                               \                     to_json();                                  \                 } </Input> <Input default_iis_in>  Module im_file  File "D:\inetpub\logs\LogFiles\W3SVC1\u_ex*.log"     ReadFromLast TRUE     Exec if $raw_event=~/^#/ drop();                         \    else                                                \    {                                                   \     w3c->parse_csv();                               \     $EventTime = parsedate($date + " " + $time);    \    } </Input> <Output evt_out>     Module om_file     File "evt.json" </Output> <Output default_iis_out>     Module om_file     Exec $raw_event = to_json();     File "iis-default.json" </Output> <Route 1>     Path        evt_in => evt_out </Route> <Route 2>     Path default_iis_in => default_iis_out </Route>   Any help would be appreciated. Thank you.

cwigley created
Replies: 2
View post »
last updated
need explanations about the log message "non-active modification on..."
Hello, I'm finding sometimes the message "non-active modification on ....<input_filename>".  Could anyone give us explanations about it ? My nxlog is v2.8.1248 on Windows Server 2008 R2. Thanks Patrick  

pleroch created
Replies: 1
View post »
last updated
NXlog cannot verify self-signed CA certificate
Hi, I'm trying to foward log events with NXlog to logstash over a SSL connection. Therefore, I generated my own CA certificate and a certificate to be used by logstash for the SSL connection. NXlog is installed on a Ubuntu 14.04 (virtual) machine with the CA certificate added as '/etc/ssl/certs/logstash-ca.crt'. I tested the connection with openssl s_client -CAfile /etc/ssl/certs/logstash-ca.cert -connect localhost:5000 and everything works fine (all entered text lines are received by logstash). However, when I start NXlog with the appropriate om_ssl output configuration it fails with ERROR SSL certificate verification failed: unable to get local issuer certificate (err: 20) My NXlog configuration looks like this: User nxlog Group nxlog LogFile /var/log/nxlog/nxlog.log LogLevel DEBUG # Modules <Extension _syslog> Module xm_syslog </Extension> <Extension json> Module xm_json </Extension> # Nxlog internal logs <Input internal> Module im_internal Exec $EventReceivedTime = integer($EventReceivedTime) / 1000000; to_json(); </Input> <Output sslout> Module om_ssl Host localhost Port 5000 CertFile "/etc/ssl/certs/logstash-ca.crt" OutputType LineBased </Output> <Route logstash> Path internal => sslout </Route>   When I start NXlog with sudo nxlog -f -c nxlog.conf the debug output looks like this: 2015-03-05 20:37:05 DEBUG module sslout got 1 poll events 2015-03-05 20:37:05 DEBUG Module sslout can read 2015-03-05 20:37:05 DEBUG [...] 2015-03-05 20:37:05 DEBUG PROCESS_EVENT: READ (sslout) 2015-03-05 20:37:05 DEBUG om_ssl read 2015-03-05 20:37:05 DEBUG doing handshake 2015-03-05 20:37:05 DEBUG verify callback (ok: 0) 2015-03-05 20:37:05 DEBUG preverification returned non-OK: unable to get local issuer certificate 2015-03-05 20:37:05 DEBUG STOP: sslout 2015-03-05 20:37:05 DEBUG remove socket [21] 2015-03-05 20:37:05 DEBUG executing statements 2015-03-05 20:37:05 DEBUG [...] 2015-03-05 20:37:05 INFO reconnecting in 1 seconds 2015-03-05 20:37:05 ERROR [om_ssl.c:532/io_err_handler()] -;[ssl.c:168/nx_ssl_check_io_error()] SSL certificate verification failed: unable to get local issuer certificate (err: 20) I have no idea why the verification work with OpenSSL but fails with NXlog (I'm using nxlog-ce-2.8.1248 and OpenSSL version is 1.0.1f). Any help is highly appreciated.  

goerlitz created
Replies: 1
View post »
last updated
$raw_event contains no data
define ROOT C:\Program Files (x86)\nxlog Moduledir %ROOT%\modules CacheDir %ROOT%\data Pidfile %ROOT%\data\nxlog.pid SpoolDir %ROOT%\data LogFile %ROOT%\data\nxlog.log #### Extensions - load and configure global extendsions <Extension eW3c>     Module xm_csv     Fields $date, $time, $s-ip, $cs-method, $cs-uri-stem, $cs-uri-query, $s-port, $cs-username, $c-ip, $csUser-Agent, $sc-status, $sc-substatus, $sc-win32-status, $time-taken     FieldTypes string, string, string, string, string, string, integer, string, string, string, integer, integer, integer, integer     Delimiter ' '     QuoteChar '"'     EscapeControl FALSE     UndefValue - </Extension> <Extension eFileOp>     Module xm_fileop </Extension> #### #### Inputs <Input iIIS-fulfillment.Api.1>     Module    im_file     File     "D:\\APShared\\Log\\W3SVC1\\\\*"     SavePos    TRUE      #    Exec if $raw_event =~ /^#/ drop();     Exec file_write("C:\\temp\\nxlog_RawIIS.log", $raw_event+"some text man"); \         eW3c->parse_csv(); \         $SourceName = "IIS-fulfillment.Api"; \         file_write("C:\\temp\\nxlog_IIS-csv.log", $raw_event); </Input> #### #### Processor #### #### Outputs <Output oIISFile>     module    om_file     File    "c:\\temp\\nx_iis.out" </Output> #### #### Routes <Route rIISFile>     path     iIIS-fulfillment.Api.1 => oIISFile </Route> NXLOG.log 2015-03-04 12:13:45 INFO nxlog-ce-2.8.1248 started 2015-03-04 12:14:11 ERROR procedure 'parse_csv' failed at line 40, character 94 in C:\Program Files (x86)\nxlog\conf\nxlog.conf. statement execution has been aborted; Not enough fields in CSV input, expected 14, got 0 in input '' "C:\\temp\\nxlog_RawIIS.log" contains: some text man   C:\\temp\\nxlog_IIS-csv.log - File not created. Help.  Where's my $raw_event data?

peter.forsythe@albumprinter.com created
Can I cut out part of the message
I have xml style messages that can have thousands of <CUID>xxx</CUID> statements in them so I need to cut out from the middle of the message: from the first string of <CUID> to the last </CUID>. Is this possible?

bigfoot created
Replies: 3
View post »
last updated
Can't start nxlog service
Hello, I have followed instructions to install nxlog in a Windows Server Core 2012 machine (no UI only command prompt). However, when I try to run the service I get: service could not be started Do you know what I could try to fix this? This Windows is the instance created automatically by Amazon Beanstalk and without a UI I find this difficult to debug.. so any suggestion is appreciated Thanks

durden81 created
Replies: 1
View post »
last updated
Change AccountName field
Dear NXlog community, I am using nxlog on a windows 2003 environment and i am having some problems with windows failed authentication events.  All entry's with EventID 675 contain the AccountName "SYSTEM" in stead of the username that the failed authentication is for. I couldnt get it to work with pattern matching in nxlog but as i have never used this before i am probably doing something wrong. I would really like to get some statistics of this and get the user name in the AccountName field. For example kibana is reporting: AccountName        SYSTEM AccountType        User Category        Account Logon CategoryNumber        9 Domain            NT AUTHORITY EventID            675 EventType        AUDIT_FAILURE FileName        Security Hostname        SomeHostName Severity        ERROR SeverityValue        4 SourceModuleName    eventlog SourceModuleType    im_mseventlog SourceName        Security host            SomeHostName.SomeDomain message            Pre-authentication failed:              User Name: [username]              User ID: %{some user id}              Service Name: krbtgt/office Pre-Authentication              Type: 0x0 Failure Code: 0x19              Client Address: [ip address]   Any help is appreciated!

pk21 created
Replies: 1
View post »
last updated
SSL encryption through TLS
Hello, I am trying to connect through SSL to ship logs but nxlog only attempts to connect through SSLv3, not TLS. Any way to force TLS 1.1 or 1.2 handshakes? Thank you

jpmills created
Replies: 4
View post »
last updated
NXlog Problem, Windows event logs not preserving event time
I'm noticing something weird with my NXLog endpoints. If I have my NXLog agents pull windows event logs and transfer them over the network it loses the event time somewhere along the way. My current setup looks like this: Remote Agent: Windows Event Log -> im__mseventlog -> om_SSL (Type Binary) (to Log Collector) Log Collector: im_SSL (Type Binary) -> om_UDP (Type GELF) (to Graylog server) When Graylog is offline it will buffer correctly, but after Graylog comes back online, NXlog will send the logs with the current time.

SDavis created
Replies: 1
View post »
last updated
Can you run a powershell command within nxlog?
Hi guys, because the event log doesn't subscribe to debug logs and because I have an app that mangles it's entries I was wondering if you can run a powershell command encapsulated in an if. eg: if $SourceName == "IHateThisApp" \ { \ $Message = *encapsulated powershell command* ;  } Of if someone know a way to get the debug info out and parsed into nxlog.

keefbaker created
Replies: 1
View post »
last updated
Parsing XML style log
Hi I read the help but still don't understand how to send this log that would like to call impaxauditlog from a windowsbox text file (that is overwritten) to my syslog server. Below are some examples from the log and my current nxlog config setting. Basically the entries take two lines one starts with !SYS and the other with <IHEYr4>. So based on the example the "!SYS 2015-02-23 11:51:49,478"  is the time the event was logged and "<TimeStamp>2015-02-23T10:08:59+02:00</TimeStamp>" is the real event datetime that I would like use as syslog message datetime (you can ask AGFA for why the difference:) if this is not possible then of course I can settle with the 2015-02-23 11:51:49,478. In essence the message itself should be everything that is between <IHEYr4></IHEYr4> tags BUT there is one big but - I need to parse the message a little and cut out everything that is between fist <CUID> and last </CUID> as there can be thousands of these and this is just wasting my diskspace. So the syslog message should look somethng like this: 2015-02-23T10:08:59+02:00 windowsbox impaxauditlog <IHEYr4><DicomQuery><CUID></CUID><SyntaxUID>LittleIndianImplicit</SyntaxUID></DicomQuery><Host>xx.xx.xx.xx</Host><TimeStamp>2015-02-23T10:08:59+02:00</TimeStamp></IHEYr4> Example: !SYS 2015-02-23 11:51:49,478 - veebr 23 10:08:59 172.16.10.33 <?xml version="1.0" encoding="UTF-8" ?> <IHEYr4><DicomQuery><CUID>1.3.12.2.1107.5.1.4.65588.30000015021704415668700013901</CUID><CUID>1.3.12.2.1107.5.1.4.65588.30000015021706453421800094763</CUID><SyntaxUID>LittleIndianImplicit</SyntaxUID></DicomQuery><Host>xx.xx.xx.xx</Host><TimeStamp>2015-02-23T10:08:59+02:00</TimeStamp></IHEYr4> Main part of my current config: <Input in>     Module      im_msvistalog </Input> <Extension syslog> Module xm_syslog </Extension> <Output sslout> Module om_ssl Host xx.xx.xx.xx Port 10514 Exec to_syslog_ietf(); CAFile %CERTDIR%/ca.cert.pem CertFile %CERTDIR%/cert.pem CertKeyFile %CERTDIR%/key.pem AllowUntrusted TRUE OutputType Syslog_TLS </Output> <Route 1>     Path        in => sslout </Route>  

bigfoot created
Replies: 1
View post »
last updated
No SSL data sent (om_ssl) from Windows NXlog to Linux ELK
Hello, Having an issue with using om_ssl from a Windows NXlog client to a Linux ELK server. The issue lies with the encryption part, the connection is established on the associated destination SSL port I set but no SSL traffic can be see/captured. For information here is the logstash config and the NXlog config, I believe the issue lies with the certs. input {   tcp {     port => 5000     type => syslog   }   udp {     port => 5000     type => syslog   }   tcp {     port => 5001     type => syslog     ssl_cacert => "/etc/pki/tls/certs/rootCA.pem"     ssl_cert => "/etc/pki/tls/certs/logstash.crt"     ssl_key => "/etc/pki/tls/private/logstash.key"     ssl_enable => true nxlog.conf: <Output syslogout> #This is for TCP non-SSL traffic   Module om_tcp   Host 192.168.0.20   Port 5000 </Output> <Output omsslout> #This is for SSL traffic only, omit all "#"   Module          om_ssl   Host            192.168.0.20   Port            5001   CAFile          %CERTDIR%\rootCA.pem   OutputType LineBased   AllowUntrusted FALSE </Output> <Route 1>   Path eventlog => eventlog_transformer => omsslout </Route> Only included relative SSL parts as everything else works over TCP and UDP. Many thanks

squidie created
Replies: 6
View post »
last updated
om_ssl with GELF - GELF over TCP/TLS support
Hello, is there any chance that nxlog supports Output om_ssl with OutputType GELF? As descriped here https://www.mail-archive.com/nxlog-ce-users@lists.sourceforge.net/msg00397.html currently NXLog only supports GELF via om_udp . It would be great, because Graylog now (19. February 2015) supports TLS for all standard TCP and AMQP inputs.

1tft created
Replies: 2
View post »
last updated
Windows 8.1 and corrupted event log ?
I am getting an error message to nxlog.log. 2015-02-17 08:16:23 INFO nxlog-ce-2.8.1248 started 2015-02-17 08:16:35 ERROR Couldn't read next event, corrupted eventlog?; The data is invalid. And after this error no more events or log messages are generated. However I can read event log with event viewer and I can see new events. These are messages in my graylog2: 2015-02-17 08:16:35.000 wintoosa Couldn't read next event, corrupted eventlog? The data is inval 2015-02-17 08:16:23.000 wintoosa nxlog-ce-2.8.1248 started Seems that nxlog is running but it can't handle events after this error. How can I fix this ? My very basic nxlog.conf: define ROOT C:\Program Files (x86)\nxlog Moduledir %ROOT%\modules CacheDir %ROOT%\data Pidfile %ROOT%\data\nxlog.pid SpoolDir %ROOT%\data LogFile %ROOT%\data\nxlog.log #LogLevel DEBUG <Extension syslog> Module xm_syslog </Extension> <Extension gelf> Module xm_gelf </Extension> <Input internal> Module im_internal </Input> <Input eventlog> Module im_msvistalog For windows 2003 and earlier use the following: Module im_mseventlog </Input> <Output out> Module om_udp Host 10.0.0.103 Port 12900 outputType GELF </Output> <Route 1> Path internal, eventlog => out </Route>

nautilus created
Replies: 3
View post »
last updated
Write to message queues
Botond, I would like to write logs out to a message queue such as kafka or redis or rabbitmq etc. Is there a road map for that? Alternately would a workaround such as writing out in JSON format to a om_tcp destination work? Can I also track the state of a tcp connection and send output to an alternate destination? I could write an external shell that would track the nxlog.log for "connection lost/made" and decision off that but I suspect that would be very expensive. Thanks Ash 

akumar created
Replies: 1
View post »
last updated
Log detail being dropped
Hi everyone, We have an internal application on a windows box which logs in a way which is slightly mangled. All other windows logs come through nxlog but on these messages the entire message field gets dropped. Is there a way to parse using ifs? eg: if $service == "Homegrown app" { } So that I can either change the output format or parse them untouched into a file etc... Does anyone know if there are any characters that don't get escaped properly that might mangle the json output? Sorry for not giving a huge amount of information, I have to keep this as agnostic as I can.

keefbaker created
Replies: 2
View post »
last updated
Log to MS SQL Server
Is there a way to use the Community Edition to log to MS SQL Server? I have tried writing a Perl script to dump the text log files into the database, but have had no luck. I understand that you can use im_odbc and om_odbc in the Enterprise version, but not the Community Edition (CE). Has anyone had any luck doing this in the CE?

nberlanga created
multiline extension not getting the endline regex condition
Hi, I am trying to parse a log4net file into json. Here's my sample log4net: ---------------- 2015-01-27 01:06:18,859 [7] ERROR Web.Cms.Content.Base.Taxonomy.TaxonomyDetectionProvider [(null)] - Get taxonomy Type Failed for Tools 2015-01-27 06:34:31,051 [26] ERROR www.Status404 [(null)] - ErrorId: 20150127_102b01c6-3208-48c5-8c8b-ae4f92cf2b20     UserAgent: Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/39.0.2171.99 Safari/537.36     HostAddress: 192.168.10.2     RequestUrl: /ErrorPages/404.aspx     MachineName: QA01     Raw Url:/undefined/     Referrer: http://qa1.www.something.com/toolset.aspx 2015-01-27 06:34:33,270 [26] DEBUG Web.Caching.Core.CacheManagerBase [(null)] - Custom CacheProvider:Web.Caching.Core.AppFabricCacheManager,Web.Caching.Core Disabled   Now I am using xm_multiline to capture each log entries. ---------------- <Extension multiline>     Module        xm_multiline     HeaderLine    /^\d{4}\-\d{2}\-\d{2} \d{2}\:\d{2}\:\d{2},\d{3}/     EndLine        /\r?\n\r?\n^\d{4}\-\d{2}\-\d{2} \d{2}\:\d{2}\:\d{2},\d{3}/ </Extension> I use a regex to capture the timestamp as the header then I use a regex to capture twice newline then the next timestamp as endline. However it still treat the second and last entry as ONE log entry. Here's the output: ---------------- {     "EventReceivedTime":"2015-01-27 01:06:35",   "SourceModuleName":"log4net",   "SourceModuleType":"im_file",   "time":"2015-01-27 01:06:18,859",   "thread":"7",   "level":"ERROR",   "logger":"Web.Cms.Content.Base.Taxonomy.TaxonomyDetectionProvider",   "ndc":"(null)",   "message":"Get taxonomy Type Failed for Tools"}{     "EventReceivedTime":"2015-01-27 06:34:35",   "SourceModuleName":"log4net",   "SourceModuleType":"im_file",   "time":"2015-01-27 06:34:31,051",   "thread":"26",   "level":"ERROR",   "logger":"www.Status404",   "ndc":"(null)",   "message":"  ErrorId: 20150127_102b01c6-3208-48c5-8c8b-ae4f92cf2b20\r\n  UserAgent: Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/39.0.2171.99  Safari/537.36\r\n  HostAddress: 192.168.10.2\r\n  RequestUrl: /ErrorPages/404.aspx\r\n  MachineName: QA01\r\n   Raw Url:/undefined/\r\n  Referrer: http://qa1.www.something.com/toolset.aspx\r\n\r\n2015-01-27 06:34:33,270 [26] DEBUG Web.Caching.Core.CacheManagerBase [(null)] - Custom CacheProvider:Web.Caching.Core.AppFabricCacheManager,Web.Caching.Core Disabled"} I used this to produce that output: ---------------- Exec        if $raw_event =~ /^(\d{4}\-\d{2}\-\d{2} \d{2}\:\d{2}\:\d{2},\d{3}) \[(\S+)\] (\S+) (\S+) \[(\S+)\] \- (.*)/s \                 { \                     $time = $1; \                     $thread = $2; \                     $level = $3; \                     $logger = $4; \                     $ndc = $5; \                     $message = $6; \                     to_json(); \                 } \                 else \                 { \                     drop(); \                 }     I've also tried to tweak it by using this to avoid the combining the last two entries as one. However I am not able to get the last entry anymore. ---------------- Exec        if $raw_event =~ /^(\d{4}\-\d{2}\-\d{2} \d{2}\:\d{2}\:\d{2},\d{3}) \[(\S+)\] (\S+) (\S+) \[(\S+)\] \- ([\s\S]*?)(\r?\n\r?\n|$)/ \                 { \                     $time = $1; \                     $thread = $2; \                     $level = $3; \                     $logger = $4; \                     $ndc = $5; \                     $message = $6; \                     to_json(); \                 } \                 else \                 { \                     drop(); \                 }​

mvf.right created
Replies: 1
View post »
last updated