Issues developing Regex Patterns

Tags: regex | logging

#1 pabloe2021

Hi, I am trying to create a regex for parsing log lines from an application. The issue is that the regex works fine in other applications, regex makers, etc. But when used by NXLog it won't find any matches, so I'm afraid there may be some NXLog specific regex syntax. Before I deconstruct this large regex and restart my service repeatedly, I wanted to ask if there is anything immediately obvious that is wrong... or is there a way to create these patterns in a reliable way? I saw some documentation about NXLog manager being able to create patterns, but I don't have access to this tool at the moment.

/\[.*?\] (\d+\s\w+\s\d+\s\d+\:\d+\:\d+\,\d+)\s(\S+)\s+(\S+)\s+\W+BusinessApplication:(.+?(?=\|))\|Component:(.+?(?=\|))\|Service:(.+?(?=\|))\|Operation:(.+?(?=\|))\|HttpMethod:(.+?(?=\|))\|Version:(.+?(?=\|))\|Client:(.+?(?=\|))\|ResponseTime:(.+?(?=\|))\|HttpStatus:(.+?(?=\|))\|Status:(.+?(?=\|))\|Severity:(.+?(?=\|))\|StatusDescription:(.+?(?=\|))\|MessageID:(.+?(?=\|))\|PE:(.+?(?=\|))\|CorrelationID:(.+?(?=\|))\|RelativeURI:(.+?(?=\|))\|Region:(.+?(?=\\n))\\n","stream":"(.+?(?="))","time":"(.+?(?="))\"}/

sample log line

{"log":"[http-nio-8080-exec-5] 10 Sep 2021 22:59:16,420 INFO PerfLog [{}]: BusinessApplication:NA|Component:NA|Service:Account Search|Operation:NA|HttpMethod:POST|Version:1|Client:enterpriseapi-2e900c67f3b948a09b0209306c64aa47|ResponseTime:1132|HttpStatus:200|Status:0|Severity:INFO|StatusDescription:SUCCESS|MessageID:MONARCH-afbd568f-e4b7-4a52-9150-26c730077c8e|PE:2718deb1806c4d6fa54efd4bf10a1abf|CorrelationID:MONARCH-2e387962-ec0c-4a11-90b3-7ea834dda252|RelativeURI:/yyyyyy-44444-333/private/25886/auto/accounts/search|Region:prod-west\n","stream":"stdout","time":"2021-09-10T22:59:16.421453207Z"}

Thank you

#2 rafDeactivated Nxlog ✓
#1 pabloe2021
Hi, I am trying to create a regex for parsing log lines from an application. The issue is that the regex works fine in other applications, regex makers, etc. But when used by NXLog it won't find any matches, so I'm afraid there may be some NXLog specific regex syntax. Before I deconstruct this large regex and restart my service repeatedly, I wanted to ask if there is anything immediately obvious that is wrong... or is there a way to create these patterns in a reliable way? I saw some documentation about NXLog manager being able to create patterns, but I don't have access to this tool at the moment. /\[.*?\] (\d+\s\w+\s\d+\s\d+\:\d+\:\d+\,\d+)\s(\S+)\s+(\S+)\s+\W+BusinessApplication:(.+?(?=\|))\|Component:(.+?(?=\|))\|Service:(.+?(?=\|))\|Operation:(.+?(?=\|))\|HttpMethod:(.+?(?=\|))\|Version:(.+?(?=\|))\|Client:(.+?(?=\|))\|ResponseTime:(.+?(?=\|))\|HttpStatus:(.+?(?=\|))\|Status:(.+?(?=\|))\|Severity:(.+?(?=\|))\|StatusDescription:(.+?(?=\|))\|MessageID:(.+?(?=\|))\|PE:(.+?(?=\|))\|CorrelationID:(.+?(?=\|))\|RelativeURI:(.+?(?=\|))\|Region:(.+?(?=\\n))\\n","stream":"(.+?(?="))","time":"(.+?(?="))\"}/ sample log line {"log":"[http-nio-8080-exec-5] 10 Sep 2021 22:59:16,420 INFO PerfLog [{}]: BusinessApplication:NA|Component:NA|Service:Account Search|Operation:NA|HttpMethod:POST|Version:1|Client:enterpriseapi-2e900c67f3b948a09b0209306c64aa47|ResponseTime:1132|HttpStatus:200|Status:0|Severity:INFO|StatusDescription:SUCCESS|MessageID:MONARCH-afbd568f-e4b7-4a52-9150-26c730077c8e|PE:2718deb1806c4d6fa54efd4bf10a1abf|CorrelationID:MONARCH-2e387962-ec0c-4a11-90b3-7ea834dda252|RelativeURI:/yyyyyy-44444-333/private/25886/auto/accounts/search|Region:prod-west\n","stream":"stdout","time":"2021-09-10T22:59:16.421453207Z"} Thank you

Hey,

Without getting to deep with your rather complex regex - do you use any tool like https://regex101.com? It really helps with building regular expressions :)

Best regards,
Raf