Hi team, Can someone please help me with parsing the below tomcat log (contains multiple lines) ? **tomcat log snippet: ** 19-Aug-2020 12:39:51.412 INFO [Catalina-utility-2] org.apache.catalina.startup.HostConfig.undeploy Undeploying context [/front] 19-Aug-2020 12:40:31.462 INFO [Catalina-utility-2] org.apache.catalina.startup.HostConfig.deployDirectory Deploying web application directory [D:\Tomcat\webapps\front] 19-Aug-2020 12:40:32.813 SEVERE [Catalina-utility-2] org.apache.catalina.startup.HostConfig.deployDirectory Error deploying web application directory [D:\Tomcat\webapps\front] java.lang.IllegalStateException: Error starting child at org.apache.catalina.core.ContainerBase.addChildInternal(ContainerBase.java:720) at org.apache.catalina.core.ContainerBase.addChild(ContainerBase.java:690) at org.apache.catalina.core.StandardHost.addChild(StandardHost.java:705) at org.apache.catalina.startup.HostConfig.deployDirectory(HostConfig.java:1133) at org.apache.catalina.startup.HostConfig$DeployDirectory.run(HostConfig.java:1866) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) I am trying to use the same example (url below) from nxlog official guide, but didn't help. https://nxlog.co/documentation/nxlog-user-guide/apache-tomcat.html **Please see my conf file content below: ** define REGEX /(?x)^(?<EventTime>\d{2}-\d{3}-\d{4}\ \d{2}:\d{2}:\d{2}).\d{3}\ (?<Severity>\S+)\ [(?<Class>\S+)]\ (?<Message>[\s\S]+)/ Module xm_multiline HeaderLine %REGEX% Module im_file File 'D:\Tomcat\logs\catalina.*.log' SavePos TRUE InputType multiline Exec if $raw_event =~ %REGEX% $EventTime = parsedate($EventTime); log_info($raw_event); log_info($Message); Exec $Message = 'TOMCAT_Catalina ' + $raw_event ; $SyslogFacilityValue = 1; $SyslogSeverityValue=5; **And getting below errors: ** 2020-11-13 14:25:54 ERROR failed to compile regular expression '(?x)^(?<EventTime>\d{2}-\d{3}-\d{4}\ \d{2}:\d{2}:\d{2}).\d{3}\ (?<Severity>\S+)\ [(?<Class>\S+)]\ (?<Message>[\s\S]+)', error at position 136: unmatched parentheses 2020-11-13 14:25:54 ERROR invalid expression in 'HeaderLine' at C:\Program Files (x86)\nxlog\conf\custom\tomcatCatalina.conf:7 2020-11-13 14:25:54 ERROR Invalid InputType 'multiline' at C:\Program Files (x86)\nxlog\conf\custom\tomcatCatalina.conf:19 2020-11-13 14:25:54 ERROR module 'tomcat_catalina_log' has configuration errors, not adding to route '4' at C:\Program Files (x86)\nxlog\conf\custom\tomcatCatalina.conf:53 2020-11-13 14:25:54 ERROR route 4 is not functional without input modules, ignored at C:\Program Files (x86)\nxlog\conf\custom\tomcatCatalina.conf:53 Please help. Thanks in advance!

Hi all :) (I'm new to nxlog) I'm currently facing issues handling logs which are being sent to nxlog via syslog line by line. Basically after looking at documentation i found out that possibly xm_multiline can help me out. Raw log example: 2020.05.20 15:22:37:481 CEST | Info | HTTP Body text part 1 2020.05.20 15:22:37:502 CEST | Info | HTTP Body text part 2 2020.05.20 15:22:37:502 CEST | Info | HTTP Body text part 3 2020.05.20 15:22:37:502 CEST |Debug | HTTP Body text part 4 2020.05.20 15:22:37:502 CEST | Info | HTTP I'm using the following headerline /^\d\d\d\d.\d\d.\d\d\s+\d\d:\d\d:\d\d:\d\d\d/ to capture the event into one. <Extension charconv> Module xm_charconv AutodetectCharsets utf-8, euc-jp, utf-16, utf-32, iso8859-2 </Extension> <Extension json> Module xm_json </Extension> <Extension multiline_header> Module xm_multiline HeaderLine /^\d\d\d\d.\d\d.\d\d\s+\d\d:\d\d:\d\d:\d\d\d/ </Extension> <Input log_udp> Module im_udp Host 0.0.0.0 Port 5140 InputType multiline_header Exec $type = 'mylog'; Exec $Message = $raw_event; </Input> <Output log_out> Module om_udp Host 1.1.1.1 Port 514 Exec $raw_event = to_json(); </Output> <Route forward_xsp> Path log_udp=> log_out </Route> Transforming the log into json. The expected output would be: Event no. 1 2020.05.20 15:22:37:481 CEST | Info | HTTP Body text part 1 Event no. 2 2020.05.20 15:22:37:502 CEST | Info | HTTP Body text part 2. etc. The issue end result: Event no. 1 2020.05.20 15:22:37:481 CEST | Info | HTTP Body text part 2020.05.20 15:22:37:502 CEST | Info | HTTP Body text part 2 Event no. 2 2020.05.20 15:22:38:481 CEST | Info | HTTP Body text part 2020.05.20 15:22:38:502 CEST | Info | HTTP Body text part 2 2020.05.20 15:22:38:481 CEST | Info | HTTP Body text part 2020.05.20 15:22:38:502 CEST | Info | HTTP Body text part 2 Event no. 3 2020.05.20 15:22:39:481 CEST | Info | HTTP Body text part 2020.05.20 15:22:39:502 CEST | Info | HTTP Body text part 2 2020.05.20 15:22:39:481 CEST | Info | HTTP Body text part 2020.05.20 15:22:38:502 CEST | Info | HTTP Body text part 2 2020.05.20 15:22:39:481 CEST | Info | HTTP Body text part 2020.05.20 15:22:39:502 CEST | Info | HTTP Body text part 2 2020.05.20 15:22:39:481 CEST | Info | HTTP Body text part 2020.05.20 15:22:38:502 CEST | Info | HTTP Body text part 2 2020.05.20 15:22:39:481 CEST | Info | HTTP Body text part 2020.05.20 15:22:39:502 CEST | Info | HTTP Body text part 2 2020.05.20 15:22:39:481 CEST | Info | HTTP Body text part 2020.05.20 15:22:38:502 CEST | Info | HTTP Body text part 2 2020.05.20 15:22:39:481 CEST | Info | HTTP Body text part 2020.05.20 15:22:39:502 CEST | Info | HTTP Body text part 2 2020.05.20 15:22:39:481 CEST | Info | HTTP Body text part 2020.05.20 15:22:38:502 CEST | Info | HTTP Body text part 2 the successive timestamp headerline is ignored and the logs are grouped by the second. (see above) :( am i doing anything wrong ? do you guys have any suggestions on how to tackle this type of logs.

Hi guys! I really someone can help because I think I have tested all the things I could think of to make it work... Ok, so we have those logs: '[2018-10-11T12:06:47,434][DEBUG][o.e.a.s.TransportSearchAction] [master01] [245674] Failed to execute fetch phase org.elasticsearch.transport.RemoteTransportException: [hot08][10.10.30.168:9300][indices:data/read/search[phase/fetch/id]] Caused by: org.elasticsearch.search.SearchContextMissingException: No search context found for id [245674] at org.elasticsearch.search.SearchService.findContext(SearchService.java:520) ~[elasticsearch-6.2.4.jar:6.2.4] at org.elasticsearch.search.SearchService.executeFetchPhase(SearchService.java:487) ~[elasticsearch-6.2.4.jar:6.2.4] at org.elasticsearch.action.search.SearchTransportService$11.messageReceived(SearchTransportService.java:440) ~[elasticsearch-6.2.4.jar:6.2.4] at org.elasticsearch.action.search.SearchTransportService$11.messageReceived(SearchTransportService.java:437) ~[elasticsearch-6.2.4.jar:6.2.4] at org.elasticsearch.xpack.security.transport.SecurityServerTransportInterceptor$ProfileSecuredRequestHandler$1.doRun(SecurityServerTransportInterceptor.java:258) ~[?:?] at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-6.2.4.jar:6.2.4] at org.elasticsearch.common.util.concurrent.EsExecutors$1.execute(EsExecutors.java:135) ~[elasticsearch-6.2.4.jar:6.2.4] at org.elasticsearch.xpack.security.transport.SecurityServerTransportInterceptor$ProfileSecuredRequestHandler.lambda$messageReceived$0(SecurityServerTransportInterceptor.java:307) ~[?:?] at org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:60) ~[elasticsearch-6.2.4.jar:6.2.4] at org.elasticsearch.xpack.security.transport.ServerTransportFilter$NodeProfile.lambda$inbound$2(ServerTransportFilter.java:166) ~[?:?] at org.elasticsearch.xpack.security.authz.AuthorizationUtils$AsyncAuthorizer.maybeRun(AuthorizationUtils.java:183) ~[?:?] at org.elasticsearch.xpack.security.authz.AuthorizationUtils$AsyncAuthorizer.setRunAsRoles(AuthorizationUtils.java:177) ~[?:?] at org.elasticsearch.xpack.security.authz.AuthorizationUtils$AsyncAuthorizer.authorize(AuthorizationUtils.java:165) ~[?:?] at org.elasticsearch.xpack.security.transport.ServerTransportFilter$NodeProfile.lambda$inbound$3(ServerTransportFilter.java:168) ~[?:?] at org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:60) ~[elasticsearch-6.2.4.jar:6.2.4] at org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.lambda$authenticateAsync$2(AuthenticationService.java:184) ~[?:?] at org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.lambda$lookForExistingAuthentication$4(AuthenticationService.java:217) ~[?:?] at org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.lookForExistingAuthentication(AuthenticationService.java:228) ~[?:?] at org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.authenticateAsync(AuthenticationService.java:182) ~[?:?] at org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.access$000(AuthenticationService.java:143) ~[?:?] at org.elasticsearch.xpack.security.authc.AuthenticationService.authenticate(AuthenticationService.java:113) ~[?:?] at org.elasticsearch.xpack.security.transport.ServerTransportFilter$NodeProfile.inbound(ServerTransportFilter.java:142) ~[?:?] at org.elasticsearch.xpack.security.transport.SecurityServerTransportInterceptor$ProfileSecuredRequestHandler.messageReceived(SecurityServerTransportInterceptor.java:314) ~[?:?] at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:66) ~[elasticsearch-6.2.4.jar:6.2.4] at org.elasticsearch.transport.TcpTransport$RequestHandler.doRun(TcpTransport.java:1555) ~[elasticsearch-6.2.4.jar:6.2.4] at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:672) ~[elasticsearch-6.2.4.jar:6.2.4] at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-6.2.4.jar:6.2.4] at org.elasticsearch.common.util.concurrent.TimedRunnable.doRun(TimedRunnable.java:41) ~[elasticsearch-6.2.4.jar:6.2.4] at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-6.2.4.jar:6.2.4] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) ~[?:1.8.0_181] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) ~[?:1.8.0_181] at java.lang.Thread.run(Thread.java:748) [?:1.8.0_181]' I really need only the first 3 lines, and is possible, make only one at the end. I have tried the following config: <Extension multi> Module xm_multiline HeaderLine /^[\d{0,4}-\d{0,2}-\d{0,2}\D\d\d{0,2}:\d{0,2}:\d{0,2}\D\d{0,4}]*/ EndLine /^.+(at)/ </Extension> <Input elastic-log> InputType multi Module im_file File "/var/log/elasticsearch/mega.log.test" </Input> <Output file> Module om_file File '/tmp/output' </Output> The the output file keep giving me all the lines instead of the first 3 that I expect... I have tested my regular expressions and I know they are working so.. why I can't have my first 3 lines!!!!???? :) Any help will be very appreciated. R.

Hello, I would like to obtain information in alert.log multiple lines for an event but without success. Below, my configuration : <Extension multiline>         Module       xm_multiline         HeaderLine   /^\w\w\w\s\w\w\w\s\d\d\s\d\d:\d\d:\d\d\s\d\d\d\d/ </Extension> <Extension syslog>         Module  xm_syslog </Extension> <Input aix_syslog>         Module  im_file         File "/var/log/syslog.out*"         Exec        to_syslog_bsd(); </Input> <Input oracle_log_DBA102XX>         Module      im_file         InputType   multiline         File        "/oradata/DBA102XX/admin/trace/bdump/alert_DBA102XX.log"         Exec        to_syslog_bsd();         Exec        $SourceName = "DBA102XX"; </Input> <Output fluent_aix>         Module om_udp         Host 172.31.12.104         Port 5139 </Output> <Output fluent_oracle>         Module om_udp         Host 172.31.12.104         Port 5127 </Output> ######################################## # Routes                               # ######################################## <Route 1>         Path    aix_syslog => fluent_aix </Route> <Route 2>     Path        oracle_log_DBA102XX => fluent_oracle </Route> Result Fluentd side : 7 Jun 2016 23:21:21     oracle.user.notice      {"host":"xmgpoc01","ident":"Tue","message":"21:21 DFT 2016"} 7 Jun 2016 23:21:21     oracle.user.notice      {"host":"xmgpoc01","ident":"Thread","message":"1 advanced to log sequence 66 (LGWR switch)"} 7 Jun 2016 23:21:21     oracle.user.notice      {"host":"xmgpoc01","ident":"","message":"/oradata/DBA102XX/oraredo01/LOG3A_DBA102XX.ORA"} 7 Jun 2016 23:21:21     oracle.user.notice      {"host":"xmgpoc01","ident":"","message":"/oradata/DBA102XX/oraredo02/LOG3B_DBA102XX.ORA"} This line "Exec        $SourceName = "DBA102XX";" doesn't works, i dont know why. Thank you for your help.    

I am trying to use the multlog module in order to start ingesting a custom log: I have the following regex: \^(\d{2}|\d).(\d{2}|\d).(\d{4})\s(\d\d|\d):(\d\d|\d):(\d\d|\d)\s(AM|PM).\[(.*)\](.*) This works in a regex test; however I cannot get it to work with the log file that looks something like this 9/10/2015 11:29:16 AM [0-3-1-SecondaryPortStatus.cs-17] GetStatus for IP: 192.168.0.231 on port: 5016 9/10/2015 11:29:16 AM [0-3-1-SecondaryPortStatus.cs-47] <TRANSACTION> <FUNCTION_TYPE>SECONDARYPORT</FUNCTION_TYPE> <COMMAND>STATUS</COMMAND> <MAC_LABEL>P_061</MAC_LABEL> <MAC>az4FMuLbvrPz720bBeKWz3c+zBh6MsKVo4nJEW96B04=</MAC> <COUNTER>217</COUNTER> </TRANSACTION> 9/10/2015 11:29:16 AM [0-3-1-SecondaryPortStatus.cs-57] <RESPONSE> <RESPONSE_TEXT>Operation SUCCESSFUL</RESPONSE_TEXT> <RESULT>OK</RESULT> <RESULT_CODE>-1</RESULT_CODE> <TERMINATION_STATUS>SUCCESS</TERMINATION_STATUS> <COUNTER>217</COUNTER> <SECONDARY_DATA>10</SECONDARY_DATA> <SERIAL_NUMBER>285498613</SERIAL_NUMBER> </RESPONSE> 9/10/2015 11:29:16 AM [0-1-1-LandingPage.xaml.cs-49] POS opened However when running the nxlog.conf for this I am getting the following error  2015-09-15 08:00:43 ERROR couldn't parse expression at line 12, character 13 in C:\Program Files (x86)\nxlog\conf\nxlog.conf; invalid character: '\' I am unsure what i need to do in order to get this correct; does anyone have any insight or resources I should further explore. Is there a REGEX specific doc for NXLOG? 

Hi, I am trying to parse a log4net file into json. Here's my sample log4net: ---------------- 2015-01-27 01:06:18,859 [7] ERROR Web.Cms.Content.Base.Taxonomy.TaxonomyDetectionProvider [(null)] - Get taxonomy Type Failed for Tools 2015-01-27 06:34:31,051 [26] ERROR www.Status404 [(null)] - ErrorId: 20150127_102b01c6-3208-48c5-8c8b-ae4f92cf2b20     UserAgent: Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/39.0.2171.99 Safari/537.36     HostAddress: 192.168.10.2     RequestUrl: /ErrorPages/404.aspx     MachineName: QA01     Raw Url:/undefined/     Referrer: http://qa1.www.something.com/toolset.aspx 2015-01-27 06:34:33,270 [26] DEBUG Web.Caching.Core.CacheManagerBase [(null)] - Custom CacheProvider:Web.Caching.Core.AppFabricCacheManager,Web.Caching.Core Disabled   Now I am using xm_multiline to capture each log entries. ---------------- <Extension multiline>     Module        xm_multiline     HeaderLine    /^\d{4}\-\d{2}\-\d{2} \d{2}\:\d{2}\:\d{2},\d{3}/     EndLine        /\r?\n\r?\n^\d{4}\-\d{2}\-\d{2} \d{2}\:\d{2}\:\d{2},\d{3}/ </Extension> I use a regex to capture the timestamp as the header then I use a regex to capture twice newline then the next timestamp as endline. However it still treat the second and last entry as ONE log entry. Here's the output: ---------------- {     "EventReceivedTime":"2015-01-27 01:06:35",   "SourceModuleName":"log4net",   "SourceModuleType":"im_file",   "time":"2015-01-27 01:06:18,859",   "thread":"7",   "level":"ERROR",   "logger":"Web.Cms.Content.Base.Taxonomy.TaxonomyDetectionProvider",   "ndc":"(null)",   "message":"Get taxonomy Type Failed for Tools"}{     "EventReceivedTime":"2015-01-27 06:34:35",   "SourceModuleName":"log4net",   "SourceModuleType":"im_file",   "time":"2015-01-27 06:34:31,051",   "thread":"26",   "level":"ERROR",   "logger":"www.Status404",   "ndc":"(null)",   "message":"  ErrorId: 20150127_102b01c6-3208-48c5-8c8b-ae4f92cf2b20\r\n  UserAgent: Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/39.0.2171.99  Safari/537.36\r\n  HostAddress: 192.168.10.2\r\n  RequestUrl: /ErrorPages/404.aspx\r\n  MachineName: QA01\r\n   Raw Url:/undefined/\r\n  Referrer: http://qa1.www.something.com/toolset.aspx\r\n\r\n2015-01-27 06:34:33,270 [26] DEBUG Web.Caching.Core.CacheManagerBase [(null)] - Custom CacheProvider:Web.Caching.Core.AppFabricCacheManager,Web.Caching.Core Disabled"} I used this to produce that output: ---------------- Exec        if $raw_event =~ /^(\d{4}\-\d{2}\-\d{2} \d{2}\:\d{2}\:\d{2},\d{3}) \[(\S+)\] (\S+) (\S+) \[(\S+)\] \- (.*)/s \                 { \                     $time = $1; \                     $thread = $2; \                     $level = $3; \                     $logger = $4; \                     $ndc = $5; \                     $message = $6; \                     to_json(); \                 } \                 else \                 { \                     drop(); \                 }     I've also tried to tweak it by using this to avoid the combining the last two entries as one. However I am not able to get the last entry anymore. ---------------- Exec        if $raw_event =~ /^(\d{4}\-\d{2}\-\d{2} \d{2}\:\d{2}\:\d{2},\d{3}) \[(\S+)\] (\S+) (\S+) \[(\S+)\] \- ([\s\S]*?)(\r?\n\r?\n|$)/ \                 { \                     $time = $1; \                     $thread = $2; \                     $level = $3; \                     $logger = $4; \                     $ndc = $5; \                     $message = $6; \                     to_json(); \                 } \                 else \                 { \                     drop(); \                 }​

Hi, I'm trying to use the xm_multiline module with nxlog to forward content of a logfile to logstash The log contains different xml elements which are properly indented (opening and closing elements are located at the start of the line) . E.g. <data version="x" xmlns:bla="http://www.example.com/bla"> <val:InfoSet> ... ... ... </val:InfoSet> </data> <message ...> <ns>bla</ns> ... ... </message> Because the elements have different names, I can only use < and </ to find the start and end line. I was hoping a filter like this should be enough to select the correct lines: HeaderLine /^</ EndLine /^<// But somehow nxlog gets confused with the / in the regex pattern. I also tried escaping which dindn't help. More testing showed that it needs at least one letter. I tried to specify all letters via regex but that didn't work: HeaderLine /^<[a-z]/ Only way that seems to work is to specify all letters in the square braket (with the exception of the lettern, which breaks).  HeaderLine /^<[abcdefghijklmopqrstuvwxyz]/ (left out n) Here all my test results. These lines worked: HeaderLine /^<m/ EndLine /^</m/ HeaderLine /^<m/ EndLine /^<\/m/ HeaderLine /^<[abcdefghijklm]/ EndLine /^<\/[abcdefghijklm]/ HeaderLine /^<[abcdefghijklmo]/ EndLine /^<\/[abcdefghijklmo]/ HeaderLine /^<[abcdefghijklmopqrstuvwxyz]/ (left out n) EndLine /^<\/[abcdefghijklmopqrstuvwxyz]/ HeaderLine /^<[abcdefghijklmopqrstuvwxyz]/ (left out n + not escaped EndLine /^</[abcdefghijklmopqrstuvwxyz]/ These lines didn't work: HeaderLine /^</ EndLine /^</m/ HeaderLine /^<[a-z]/ EndLine /^</m/ HeaderLine /^<\w/ EndLine /^</m/ HeaderLine /^<[abcdefghijklmn]/ EndLine /^<\/[abcdefghijklmn]/ HeaderLine /^<[bcdefghijklmn]/ EndLine /^<\/[bcdefghijklmn]/ HeaderLine /^<[abcdefghijklmopqrstuvwxyzn]/ EndLine /^<\/[abcdefghijklmopqrstuvwxyzn]/ HeaderLine /^<[abcdefghijklmnopqrstuvwxyz]/ EndLine /^</[abcdefghijklmnopqrstuvwxyz]/ Right now I still have a problem because many of my bessages start with <n. I think this is a bug in the module. Can you confirm so I can open a ticket? Thanks   Fyi, this is a duplicate of http://stackoverflow.com/questions/27429234/which-headerline-and-endline-for-multiline-xml-with-different-elements