LS_140907 created
jd01 created
Hi guys! I really someone can help because I think I have tested all the things I could think of to make it work...
Ok, so we have those logs:
'[2018-10-11T12:06:47,434][DEBUG][o.e.a.s.TransportSearchAction] [master01] [245674] Failed to execute fetch phase org.elasticsearch.transport.RemoteTransportException: [hot08][10.10.30.168:9300][indices:data/read/search[phase/fetch/id]] Caused by: org.elasticsearch.search.SearchContextMissingException: No search context found for id [245674] at org.elasticsearch.search.SearchService.findContext(SearchService.java:520) ~[elasticsearch-6.2.4.jar:6.2.4] at org.elasticsearch.search.SearchService.executeFetchPhase(SearchService.java:487) ~[elasticsearch-6.2.4.jar:6.2.4] at org.elasticsearch.action.search.SearchTransportService$11.messageReceived(SearchTransportService.java:440) ~[elasticsearch-6.2.4.jar:6.2.4] at org.elasticsearch.action.search.SearchTransportService$11.messageReceived(SearchTransportService.java:437) ~[elasticsearch-6.2.4.jar:6.2.4] at org.elasticsearch.xpack.security.transport.SecurityServerTransportInterceptor$ProfileSecuredRequestHandler$1.doRun(SecurityServerTransportInterceptor.java:258) ~[?:?] at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-6.2.4.jar:6.2.4] at org.elasticsearch.common.util.concurrent.EsExecutors$1.execute(EsExecutors.java:135) ~[elasticsearch-6.2.4.jar:6.2.4] at org.elasticsearch.xpack.security.transport.SecurityServerTransportInterceptor$ProfileSecuredRequestHandler.lambda$messageReceived$0(SecurityServerTransportInterceptor.java:307) ~[?:?] at org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:60) ~[elasticsearch-6.2.4.jar:6.2.4] at org.elasticsearch.xpack.security.transport.ServerTransportFilter$NodeProfile.lambda$inbound$2(ServerTransportFilter.java:166) ~[?:?] at org.elasticsearch.xpack.security.authz.AuthorizationUtils$AsyncAuthorizer.maybeRun(AuthorizationUtils.java:183) ~[?:?] at org.elasticsearch.xpack.security.authz.AuthorizationUtils$AsyncAuthorizer.setRunAsRoles(AuthorizationUtils.java:177) ~[?:?] at org.elasticsearch.xpack.security.authz.AuthorizationUtils$AsyncAuthorizer.authorize(AuthorizationUtils.java:165) ~[?:?] at org.elasticsearch.xpack.security.transport.ServerTransportFilter$NodeProfile.lambda$inbound$3(ServerTransportFilter.java:168) ~[?:?] at org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:60) ~[elasticsearch-6.2.4.jar:6.2.4] at org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.lambda$authenticateAsync$2(AuthenticationService.java:184) ~[?:?] at org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.lambda$lookForExistingAuthentication$4(AuthenticationService.java:217) ~[?:?] at org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.lookForExistingAuthentication(AuthenticationService.java:228) ~[?:?] at org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.authenticateAsync(AuthenticationService.java:182) ~[?:?] at org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.access$000(AuthenticationService.java:143) ~[?:?] at org.elasticsearch.xpack.security.authc.AuthenticationService.authenticate(AuthenticationService.java:113) ~[?:?] at org.elasticsearch.xpack.security.transport.ServerTransportFilter$NodeProfile.inbound(ServerTransportFilter.java:142) ~[?:?] at org.elasticsearch.xpack.security.transport.SecurityServerTransportInterceptor$ProfileSecuredRequestHandler.messageReceived(SecurityServerTransportInterceptor.java:314) ~[?:?] at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:66) ~[elasticsearch-6.2.4.jar:6.2.4] at org.elasticsearch.transport.TcpTransport$RequestHandler.doRun(TcpTransport.java:1555) ~[elasticsearch-6.2.4.jar:6.2.4] at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:672) ~[elasticsearch-6.2.4.jar:6.2.4] at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-6.2.4.jar:6.2.4] at org.elasticsearch.common.util.concurrent.TimedRunnable.doRun(TimedRunnable.java:41) ~[elasticsearch-6.2.4.jar:6.2.4] at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-6.2.4.jar:6.2.4] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) ~[?:1.8.0_181] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) ~[?:1.8.0_181] at java.lang.Thread.run(Thread.java:748) [?:1.8.0_181]'
I really need only the first 3 lines, and is possible, make only one at the end.
I have tried the following config:
<Extension multi> Module xm_multiline HeaderLine /^[\d{0,4}-\d{0,2}-\d{0,2}\D\d\d{0,2}:\d{0,2}:\d{0,2}\D\d{0,4}]*/ EndLine /^.+(at)/ </Extension>
<Input elastic-log> InputType multi Module im_file File "/var/log/elasticsearch/mega.log.test" </Input>
<Output file> Module om_file File '/tmp/output' </Output>
The the output file keep giving me all the lines instead of the first 3 that I expect...
I have tested my regular expressions and I know they are working so.. why I can't have my first 3 lines!!!!???? :)
Any help will be very appreciated. R.
romainp created
Hello,
I would like to obtain information in alert.log multiple lines for an event but without success. Below, my configuration :
<Extension multiline>
Module xm_multiline
HeaderLine /^\w\w\w\s\w\w\w\s\d\d\s\d\d:\d\d:\d\d\s\d\d\d\d/
</Extension>
<Extension syslog>
Module xm_syslog
</Extension>
<Input aix_syslog>
Module im_file
File "/var/log/syslog.out*"
Exec to_syslog_bsd();
</Input>
<Input oracle_log_DBA102XX>
Module im_file
InputType multiline
File "/oradata/DBA102XX/admin/trace/bdump/alert_DBA102XX.log"
Exec to_syslog_bsd();
Exec $SourceName = "DBA102XX";
</Input>
<Output fluent_aix>
Module om_udp
Host 172.31.12.104
Port 5139
</Output>
<Output fluent_oracle>
Module om_udp
Host 172.31.12.104
Port 5127
</Output>
########################################
# Routes #
########################################
<Route 1>
Path aix_syslog => fluent_aix
</Route>
<Route 2>
Path oracle_log_DBA102XX => fluent_oracle
</Route>
Result Fluentd side :
7 Jun 2016 23:21:21 oracle.user.notice {"host":"xmgpoc01","ident":"Tue","message":"21:21 DFT 2016"}
7 Jun 2016 23:21:21 oracle.user.notice {"host":"xmgpoc01","ident":"Thread","message":"1 advanced to log sequence 66 (LGWR switch)"}
7 Jun 2016 23:21:21 oracle.user.notice {"host":"xmgpoc01","ident":"","message":"/oradata/DBA102XX/oraredo01/LOG3A_DBA102XX.ORA"}
7 Jun 2016 23:21:21 oracle.user.notice {"host":"xmgpoc01","ident":"","message":"/oradata/DBA102XX/oraredo02/LOG3B_DBA102XX.ORA"}
This line "Exec $SourceName = "DBA102XX";" doesn't works, i dont know why.
Thank you for your help.
gfouere created
I am trying to use the multlog module in order to start ingesting a custom log:
I have the following regex: \^(\d{2}|\d).(\d{2}|\d).(\d{4})\s(\d\d|\d):(\d\d|\d):(\d\d|\d)\s(AM|PM).\[(.*)\](.*)
This works in a regex test; however I cannot get it to work with the log file that looks something like this
9/10/2015 11:29:16 AM [0-3-1-SecondaryPortStatus.cs-17] GetStatus for IP: 192.168.0.231 on port: 5016 9/10/2015 11:29:16 AM [0-3-1-SecondaryPortStatus.cs-47] <TRANSACTION> <FUNCTION_TYPE>SECONDARYPORT</FUNCTION_TYPE> <COMMAND>STATUS</COMMAND> <MAC_LABEL>P_061</MAC_LABEL> <MAC>az4FMuLbvrPz720bBeKWz3c+zBh6MsKVo4nJEW96B04=</MAC> <COUNTER>217</COUNTER> </TRANSACTION> 9/10/2015 11:29:16 AM [0-3-1-SecondaryPortStatus.cs-57] <RESPONSE> <RESPONSE_TEXT>Operation SUCCESSFUL</RESPONSE_TEXT> <RESULT>OK</RESULT> <RESULT_CODE>-1</RESULT_CODE> <TERMINATION_STATUS>SUCCESS</TERMINATION_STATUS> <COUNTER>217</COUNTER> <SECONDARY_DATA>10</SECONDARY_DATA> <SERIAL_NUMBER>285498613</SERIAL_NUMBER> </RESPONSE> 9/10/2015 11:29:16 AM [0-1-1-LandingPage.xaml.cs-49] POS opened
However when running the nxlog.conf for this I am getting the following error
2015-09-15 08:00:43 ERROR couldn't parse expression at line 12, character 13 in C:\Program Files (x86)\nxlog\conf\nxlog.conf; invalid character: '\'
I am unsure what i need to do in order to get this correct; does anyone have any insight or resources I should further explore. Is there a REGEX specific doc for NXLOG?
chris.ried created
Hi,
I am trying to parse a log4net file into json.
Here's my sample log4net:
----------------
2015-01-27 01:06:18,859 [7] ERROR Web.Cms.Content.Base.Taxonomy.TaxonomyDetectionProvider [(null)] - Get taxonomy Type Failed for Tools
2015-01-27 06:34:31,051 [26] ERROR www.Status404 [(null)] - ErrorId: 20150127_102b01c6-3208-48c5-8c8b-ae4f92cf2b20
UserAgent: Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/39.0.2171.99 Safari/537.36
HostAddress: 192.168.10.2
RequestUrl: /ErrorPages/404.aspx
MachineName: QA01
Raw Url:/undefined/
Referrer: http://qa1.www.something.com/toolset.aspx
2015-01-27 06:34:33,270 [26] DEBUG Web.Caching.Core.CacheManagerBase [(null)] - Custom CacheProvider:Web.Caching.Core.AppFabricCacheManager,Web.Caching.Core Disabled
Now I am using xm_multiline to capture each log entries.
----------------
<Extension multiline>
Module xm_multiline
HeaderLine /^\d{4}\-\d{2}\-\d{2} \d{2}\:\d{2}\:\d{2},\d{3}/
EndLine /\r?\n\r?\n^\d{4}\-\d{2}\-\d{2} \d{2}\:\d{2}\:\d{2},\d{3}/
</Extension>
I use a regex to capture the timestamp as the header then I use a regex to capture twice newline then the next timestamp as endline. However it still treat the second and last entry as ONE log entry.
Here's the output:
----------------
{
"EventReceivedTime":"2015-01-27 01:06:35",
"SourceModuleName":"log4net",
"SourceModuleType":"im_file",
"time":"2015-01-27 01:06:18,859",
"thread":"7",
"level":"ERROR",
"logger":"Web.Cms.Content.Base.Taxonomy.TaxonomyDetectionProvider",
"ndc":"(null)",
"message":"Get taxonomy Type Failed for Tools"
}{
"EventReceivedTime":"2015-01-27 06:34:35",
"SourceModuleName":"log4net",
"SourceModuleType":"im_file",
"time":"2015-01-27 06:34:31,051",
"thread":"26",
"level":"ERROR",
"logger":"www.Status404",
"ndc":"(null)",
"message":" ErrorId: 20150127_102b01c6-3208-48c5-8c8b-ae4f92cf2b20\r\n UserAgent: Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/39.0.2171.99
Safari/537.36\r\n HostAddress: 192.168.10.2\r\n RequestUrl: /ErrorPages/404.aspx\r\n MachineName: QA01\r\n
Raw Url:/undefined/\r\n Referrer: http://qa1.www.something.com/toolset.aspx\r\n\r\n2015-01-27 06:34:33,270 [26] DEBUG Web.Caching.Core.CacheManagerBase [(null)] - Custom CacheProvider:Web.Caching.Core.AppFabricCacheManager,Web.Caching.Core Disabled"
}
I used this to produce that output:
----------------
Exec if $raw_event =~ /^(\d{4}\-\d{2}\-\d{2} \d{2}\:\d{2}\:\d{2},\d{3}) \[(\S+)\] (\S+) (\S+) \[(\S+)\] \- (.*)/s \
{ \
$time = $1; \
$thread = $2; \
$level = $3; \
$logger = $4; \
$ndc = $5; \
$message = $6; \
to_json(); \
} \
else \
{ \
drop(); \
}
I've also tried to tweak it by using this to avoid the combining the last two entries as one. However I am not able to get the last entry anymore.
----------------
Exec if $raw_event =~ /^(\d{4}\-\d{2}\-\d{2} \d{2}\:\d{2}\:\d{2},\d{3}) \[(\S+)\] (\S+) (\S+) \[(\S+)\] \- ([\s\S]*?)(\r?\n\r?\n|$)/ \
{ \
$time = $1; \
$thread = $2; \
$level = $3; \
$logger = $4; \
$ndc = $5; \
$message = $6; \
to_json(); \
} \
else \
{ \
drop(); \
}
mvf.right created
Hi,
I'm trying to use the xm_multiline module with nxlog to forward content of a logfile to logstash The log contains different xml elements which are properly indented (opening and closing elements are located at the start of the line) . E.g.
<data
version="x"
xmlns:bla="http://www.example.com/bla">
<val:InfoSet>
...
...
...
</val:InfoSet>
</data>
<message ...>
<ns>bla</ns>
...
...
</message>
Because the elements have different names, I can only use < and </ to find the start and end line. I was hoping a filter like this should be enough to select the correct lines:
HeaderLine /^</
EndLine /^<//
But somehow nxlog gets confused with the / in the regex pattern. I also tried escaping which dindn't help. More testing showed that it needs at least one letter. I tried to specify all letters via regex but that didn't work:
HeaderLine /^<[a-z]/
Only way that seems to work is to specify all letters in the square braket (with the exception of the lettern, which breaks).
HeaderLine /^<[abcdefghijklmopqrstuvwxyz]/ (left out n)
Here all my test results.
These lines worked:
HeaderLine /^<m/
EndLine /^</m/
HeaderLine /^<m/
EndLine /^<\/m/
HeaderLine /^<[abcdefghijklm]/
EndLine /^<\/[abcdefghijklm]/
HeaderLine /^<[abcdefghijklmo]/
EndLine /^<\/[abcdefghijklmo]/
HeaderLine /^<[abcdefghijklmopqrstuvwxyz]/ (left out n)
EndLine /^<\/[abcdefghijklmopqrstuvwxyz]/
HeaderLine /^<[abcdefghijklmopqrstuvwxyz]/ (left out n + not escaped
EndLine /^</[abcdefghijklmopqrstuvwxyz]/
These lines didn't work:
HeaderLine /^</
EndLine /^</m/
HeaderLine /^<[a-z]/
EndLine /^</m/
HeaderLine /^<\w/
EndLine /^</m/
HeaderLine /^<[abcdefghijklmn]/
EndLine /^<\/[abcdefghijklmn]/
HeaderLine /^<[bcdefghijklmn]/
EndLine /^<\/[bcdefghijklmn]/
HeaderLine /^<[abcdefghijklmopqrstuvwxyzn]/
EndLine /^<\/[abcdefghijklmopqrstuvwxyzn]/
HeaderLine /^<[abcdefghijklmnopqrstuvwxyz]/
EndLine /^</[abcdefghijklmnopqrstuvwxyz]/
Right now I still have a problem because many of my bessages start with <n. I think this is a bug in the module. Can you confirm so I can open a ticket? Thanks
Fyi, this is a duplicate of http://stackoverflow.com/questions/27429234/which-headerline-and-endline-for-multiline-xml-with-different-elements
pgs created
