Hi, I am trying to create a regex for parsing log lines from an application. The issue is that the regex works fine in other applications, regex makers, etc. But when used by NXLog it won't find any matches, so I'm afraid there may be some NXLog specific regex syntax. Before I deconstruct this large regex and restart my service repeatedly, I wanted to ask if there is anything immediately obvious that is wrong... or is there a way to create these patterns in a reliable way? I saw some documentation about NXLog manager being able to create patterns, but I don't have access to this tool at the moment. /\[.*?\] (\d+\s\w+\s\d+\s\d+\:\d+\:\d+\,\d+)\s(\S+)\s+(\S+)\s+\W+BusinessApplication:(.+?(?=\|))\|Component:(.+?(?=\|))\|Service:(.+?(?=\|))\|Operation:(.+?(?=\|))\|HttpMethod:(.+?(?=\|))\|Version:(.+?(?=\|))\|Client:(.+?(?=\|))\|ResponseTime:(.+?(?=\|))\|HttpStatus:(.+?(?=\|))\|Status:(.+?(?=\|))\|Severity:(.+?(?=\|))\|StatusDescription:(.+?(?=\|))\|MessageID:(.+?(?=\|))\|PE:(.+?(?=\|))\|CorrelationID:(.+?(?=\|))\|RelativeURI:(.+?(?=\|))\|Region:(.+?(?=\\n))\\n","stream":"(.+?(?="))","time":"(.+?(?="))\"}/ sample log line {"log":"[http-nio-8080-exec-5] 10 Sep 2021 22:59:16,420 INFO PerfLog [{}]: BusinessApplication:NA|Component:NA|Service:Account Search|Operation:NA|HttpMethod:POST|Version:1|Client:enterpriseapi-2e900c67f3b948a09b0209306c64aa47|ResponseTime:1132|HttpStatus:200|Status:0|Severity:INFO|StatusDescription:SUCCESS|MessageID:MONARCH-afbd568f-e4b7-4a52-9150-26c730077c8e|PE:2718deb1806c4d6fa54efd4bf10a1abf|CorrelationID:MONARCH-2e387962-ec0c-4a11-90b3-7ea834dda252|RelativeURI:/yyyyyy-44444-333/private/25886/auto/accounts/search|Region:prod-west\n","stream":"stdout","time":"2021-09-10T22:59:16.421453207Z"} Thank you

Hi, I’m trying to use regex in nxlog. My current configuration is to save firewall logs to a file .txt using the $Sender value to create the file name. ....... <Input *****> Module im_tcp Host 0.0.0.0 Port 1001 <Exec> if $raw_event =~ /LEEF/ parse_leef(); else parse_syslog(); </Exec> </Input> ....... <Output > define OUT_DIR %LOGDIR2%/ Module om_file File "%OUT_DIR%/" + $Sender + ".txt" <Schedule> Every 3600 sec <Exec> if ->file_size() > 0M { set_var('newfile', file_name() + strftime(now(), '_%Y%m%d%H%M%S') + '.log'); rotate_to(get_var('newfile')); exec_async('C:/Program Files/GnuWin32/bin/bzip2.exe', 'E:// *.log'); } </Exec> </Schedule> </Output> ......... This is the Log: <13>Sep 4 16:07:23 Firewall: LEEF:1.0|FORCEPOINT|Firewall|1.1.1|Connection_Discarded|src=122.1.1.1 EventReceivedTime=2019-09-04 16:07:23 SourceModuleName=****** SourceModuleType=im_tcp LEEFVersion=<1> LEEF:0.0 Vendor=FORCEPOINT vSrcName=Firewall Version=1.1.1 EventID=Connection_Discarded devTimeFormat=MMM dd yyyy HH:mm:ss devTime=2019-09-04 16:07:23 proto=1 dstPort=80 srcPort=53438 dst=192.1.1.1 sender=services.fw.mi01.custom.cloud node 1 action=Discard the system sets the value of $Sender like this: $Sender = services.fw.mi01.custom.cloud node 1 action=Discard.txt but I need instead the system to set $Sender this way, only up to "node 1": $Sender = services.fw.mi01.custom.cloud node 1.txt I thought about using a regex to extrapolate the value I need, but it doesn’t work. this one: <Exec> if $Sender =~ /(?<=sender=).[^\t]+/g; $Sender = $1 </Exec> Can I do this thing? If so, what should I do? Thank you Antonio

I'm trying to get a PatternDB working correctly, and it looks like I'm getting some fields but not all of them. There's only one pattern that's actually generating extra fields, and even it is dropping the first field (ParsedDate). Not sure what's going on here... Config file (via file inclusion): <Extension json>     Module      xm_json </Extension> <Extension syslog>     Module      xm_syslog </Extension> <Input vg_tsw_client>   Module     im_file   File       "C:\Program Files (x86)\Steam\steamapps\common\The Secret World\ClientLog.txt"   Exec       if not ($raw_event =~ /Scaleform\.TSWACT/) drop();   Exec         parse_syslog(); </Input> <Input vg_tsw_combat>   Module     im_file   File       "C:\Program Files (x86)\Steam\steamapps\common\The Secret World\CombatLog-*.txt"   Exec       if ($raw_event =~ /Sprinting [VI]+/) drop();   Exec         parse_syslog(); </Input> <Processor vg_tsw_pattern>     Module    pm_pattern     PatternFile %ROOT%\conf\SecretWorld\patterndb.xml </Processor> <Output vg_tsw_testfile>   Module     om_file   File       "C:\\ProgramData\\nxlogs\\vg-tsw-logs.log"   Exec       to_json(); </Output> <Route vg_tsw_route>   Path       vg_tsw_client, vg_tsw_combat => vg_tsw_pattern => vg_tsw_testfile </Route> Pattern DB: <?xml version='1.0' encoding='UTF-8'?> <patterndb>  <created>2010-01-01 01:02:03</created>  <version>42</version>   <group>   <name>tswCombat</name>   <id>50284624</id>   <matchfield>     <name>SourceModuleName</name>     <type>exact</type>     <value>vg_tsw_combat</value>   </matchfield>   <pattern>     <id>1000</id>     <name>basic combat swing</name>     <matchfield>      <name>Message</name>      <type>regexp</type>         <!-- [00:00:28] (Critical) Solomon County Cop's Spray and Pray hits (Normal) Ravenous Horde for 522 physical damage. (Normal) -->      <value>^\[([^\]]+)\] ((?:\(Critical\) |\(Normal\) )?)(.+?'s|Your) (.+?) hits \((Normal|Glancing)\) (.*?) for (\d+) (physical|magical) damage. \((Normal|Penetrated|Blocked)\)</value>      <capturedfield>       <name>ParsedTime</name>       <type>datetime</type>      </capturedfield>      <capturedfield>       <name>CriticalHit</name>       <type>string</type>      </capturedfield>      <capturedfield>         <name>AttackerName</name>         <type>string</type>      </capturedfield>      <capturedfield>         <name>AttackName</name>         <type>string</type>      </capturedfield>      <capturedfield>         <name>Glancing</name>         <type>string</type>      </capturedfield>      <capturedfield>         <name>VictimName</name>         <type>string</type>      </capturedfield>      <capturedfield>         <name>Damage</name>         <type>integer</type>      </capturedfield>      <capturedfield>         <name>DamageType</name>         <type>string</type>      </capturedfield>      <capturedfield>         <name>BlockOrPen</name>         <type>string</type>      </capturedfield>     </matchfield>     <set>      <field>        <name>type</name>        <value>Swing</value>        <type>string</type>      </field>     </set>   </pattern> </group> <group>   <name>tswClient</name>   <id>50284625</id>   <matchfield>     <name>SourceModuleName</name>     <type>exact</type>     <value>vg_tsw_client</value>   </matchfield>     <pattern>     <id>2000</id>     <name>tswact load plugin</name>     <matchfield>      <name>Message</name>      <type>regexp</type>         <!-- [2017-02-10 05:47:07Z #3886] [ID:0] ERROR: Scaleform.TSWACT - TSWACT Loaded for |Sheriban| -->      <value>^\[([0-9-:]+)Z #\d+\] \[ID:\d+\] ERROR: Scaleform.TSWACT - TSWACT Loaded for - \|(\w+)\|</value>      <capturedfield>       <name>ParsedTime</name>       <type>string</type>      </capturedfield>      <capturedfield>       <name>PlayerName</name>       <type>string</type>      </capturedfield>     </matchfield>     <set>      <field>        <name>type</name>        <value>TswactLoaded</value>        <type>string</type>      </field>     </set>   </pattern>     <pattern>     <id>2001</id>     <name>tswact load playfield</name>     <matchfield>      <name>Message</name>      <type>regexp</type>         <!-- [2017-02-10 05:47:07Z #3886] [ID:0] ERROR: Scaleform.TSWACT - Playfield - |Kingsmouth Town| -->      <value>^\[([0-9-:]+)Z #\d+\] \[ID:\d+\] ERROR: Scaleform.TSWACT - Playfield - \|(\w+)\|</value>      <capturedfield>       <name>ParsedTime</name>       <type>datetime</type>      </capturedfield>      <capturedfield>       <name>ZoneName</name>       <type>string</type>      </capturedfield>     </matchfield>     <set>      <field>        <name>type</name>        <value>SetZoneName</value>        <type>string</type>      </field>     </set>   </pattern>   <pattern>     <id>2002</id>     <name>tswact enter combat</name>     <matchfield>      <name>Message</name>      <type>regexp</type>         <!-- [2017-02-10 05:00:22Z #10910] [ID:0] ERROR: Scaleform.TSWACT - Enter combat - |Sheriban|Buffs:Sprinting VI:Elemental Force:Third Degree :World Domination| -->      <value>^\[([0-9-:]+)Z #\d+\] \[ID:\d+\] ERROR: Scaleform.TSWACT - Enter combat - \|(\w+)\|</value>      <capturedfield>       <name>ParsedTime</name>       <type>datetime</type>      </capturedfield>      <capturedfield>       <name>PlayerName</name>       <type>string</type>      </capturedfield>     </matchfield>     <set>      <field>        <name>type</name>        <value>EnterCombat</value>        <type>string</type>      </field>     </set>     <exec>       $TestField = 'testValue';     </exec>   </pattern> </group> </patterndb> Some of the output I'm getting: {"EventReceivedTime":"2017-02-10 11:45:00","SourceModuleName":"vg_tsw_combat","SourceModuleType":"im_file","SyslogFacilityValue":1,"SyslogFacility":"USER","SyslogSeverityValue":5,"SyslogSeverity":"NOTICE","SeverityValue":2,"Severity":"INFO","EventTime":"2017-02-10 11:45:00","Hostname":"shepard","Message":"[11:45:00] Your Pop Shot hits (Normal) Undead Islander for 1437 physical damage. (Normal)","CriticalHit":"","AttackerName":"Your","AttackName":"Pop Shot","Glancing":"Normal","VictimName":"Undead Islander","Damage":1437,"DamageType":"physical","BlockOrPen":"Normal","PatternID":1000,"PatternName":"basic combat swing","type":"Swing"} {"EventReceivedTime":"2017-02-10 11:45:00","SourceModuleName":"vg_tsw_combat","SourceModuleType":"im_file","SyslogFacilityValue":1,"SyslogFacility":"USER","SyslogSeverityValue":5,"SyslogSeverity":"NOTICE","SeverityValue":2,"Severity":"INFO","EventTime":"2017-02-10 11:45:00","Hostname":"shepard","Message":"[11:45:00] (Critical) Your Pop Shot hits (Normal) Undead Islander for 2965 physical damage. (Penetrated)","CriticalHit":"(Critical) ","AttackerName":"Your","AttackName":"Pop Shot","Glancing":"Normal","VictimName":"Undead Islander","Damage":2965,"DamageType":"physical","BlockOrPen":"Penetrated","PatternID":1000,"PatternName":"basic combat swing","type":"Swing"} {"EventReceivedTime":"2017-02-10 11:45:00","SourceModuleName":"vg_tsw_combat","SourceModuleType":"im_file","SyslogFacilityValue":1,"SyslogFacility":"USER","SyslogSeverityValue":5,"SyslogSeverity":"NOTICE","SeverityValue":2,"Severity":"INFO","EventTime":"2017-02-10 11:45:00","Hostname":"shepard","Message":"[11:45:00] You gain buff Live Wire"} {"EventReceivedTime":"2017-02-10 11:45:01","SourceModuleName":"vg_tsw_client","SourceModuleType":"im_file","SyslogFacilityValue":1,"SyslogFacility":"USER","SyslogSeverityValue":5,"SyslogSeverity":"NOTICE","SeverityValue":2,"Severity":"INFO","EventTime":"2017-02-10 11:45:01","Hostname":"shepard","Message":"[2017-02-10 16:45:01Z #18498] [ID:0] ERROR: Scaleform.TSWACT - Out of combat - |Sheriban|"} {"EventReceivedTime":"2017-02-10 11:45:10","SourceModuleName":"vg_tsw_combat","SourceModuleType":"im_file","SyslogFacilityValue":1,"SyslogFacility":"USER","SyslogSeverityValue":5,"SyslogSeverity":"NOTICE","SeverityValue":2,"Severity":"INFO","EventTime":"2017-02-10 11:45:10","Hostname":"shepard","Message":"[11:45:10] Buff Live Wire terminated."} Some of the vg_tsw_combat input file: [11:45:00] Your One in the Chamber hits (Normal) Undead Islander for 231 physical damage. (Normal) [11:45:00] Buff Sudden Return terminated on Undead Islander. [11:45:00] Buff One in the Chamber terminated on Undead Islander. [11:45:00] You gained 146 XP. [11:45:00] Undead Islander died. [11:45:00] Your Sudden Return hits (Normal) Undead Islander for 259 physical damage. (Normal) [11:45:00] Your Pop Shot hits (Normal) Undead Islander for 2045 physical damage. (Penetrated) [11:45:00] Your Pop Shot hits (Normal) Undead Islander for 2175 physical damage. (Penetrated) [11:45:00] Your Pop Shot hits (Normal) Undead Islander for 1437 physical damage. (Normal) [11:45:00] (Critical) Your Pop Shot hits (Normal) Undead Islander for 2965 physical damage. (Penetrated) [11:45:00] You gain buff Live Wire [11:45:02] You start using Sprinting VI. [11:45:03] You gain buff Sprinting VI [11:45:03] You successfully used Sprinting VI. [11:45:10] Buff Live Wire terminated. Some of the vg_tsw_client input: [2017-02-10 16:33:43Z #6790] [ID:0] ERROR: Scaleform.TSWACT - TSWACT Loaded for |Sheriban| [2017-02-10 16:33:43Z #6790] [ID:0] ERROR: Scaleform.TSWACT - Playfield - |The Savage Coast| [2017-02-10 16:34:12Z #7313] [ID:0] ERROR: Scaleform.TSWACT - Enter combat - |Sheriban|Buffs:World Domination| [2017-02-10 16:34:14Z #7373] [ID:0] ERROR: Scaleform.TSWACT - Out of combat - |Sheriban| [2017-02-10 16:39:06Z #10609] [ID:0] ERROR: MagicCommand - Trying to prepone the execute timeline to the pass. Spell:7760057 [2017-02-10 16:39:06Z #10624] [ID:0] ERROR: Scaleform.TSWACT - Enter combat - |Sheriban|Buffs:Elemental Force:World Domination| [2017-02-10 16:39:08Z #10655] [ID:0] ERROR: Scaleform.TSWACT - Out of combat - |Sheriban| [2017-02-10 16:44:58Z #18330] [ID:0] ERROR: MagicCommand - Trying to prepone the execute timeline to the pass. Spell:7760057 [2017-02-10 16:44:59Z #18388] [ID:0] ERROR: Scaleform.TSWACT - Enter combat - |Sheriban|Buffs:Elemental Force:World Domination| [2017-02-10 16:45:01Z #18498] [ID:0] ERROR: Scaleform.TSWACT - Out of combat - |Sheriban| Any ideas?

I am trying to use the multlog module in order to start ingesting a custom log: I have the following regex: \^(\d{2}|\d).(\d{2}|\d).(\d{4})\s(\d\d|\d):(\d\d|\d):(\d\d|\d)\s(AM|PM).\[(.*)\](.*) This works in a regex test; however I cannot get it to work with the log file that looks something like this 9/10/2015 11:29:16 AM [0-3-1-SecondaryPortStatus.cs-17] GetStatus for IP: 192.168.0.231 on port: 5016 9/10/2015 11:29:16 AM [0-3-1-SecondaryPortStatus.cs-47] <TRANSACTION> <FUNCTION_TYPE>SECONDARYPORT</FUNCTION_TYPE> <COMMAND>STATUS</COMMAND> <MAC_LABEL>P_061</MAC_LABEL> <MAC>az4FMuLbvrPz720bBeKWz3c+zBh6MsKVo4nJEW96B04=</MAC> <COUNTER>217</COUNTER> </TRANSACTION> 9/10/2015 11:29:16 AM [0-3-1-SecondaryPortStatus.cs-57] <RESPONSE> <RESPONSE_TEXT>Operation SUCCESSFUL</RESPONSE_TEXT> <RESULT>OK</RESULT> <RESULT_CODE>-1</RESULT_CODE> <TERMINATION_STATUS>SUCCESS</TERMINATION_STATUS> <COUNTER>217</COUNTER> <SECONDARY_DATA>10</SECONDARY_DATA> <SERIAL_NUMBER>285498613</SERIAL_NUMBER> </RESPONSE> 9/10/2015 11:29:16 AM [0-1-1-LandingPage.xaml.cs-49] POS opened However when running the nxlog.conf for this I am getting the following error  2015-09-15 08:00:43 ERROR couldn't parse expression at line 12, character 13 in C:\Program Files (x86)\nxlog\conf\nxlog.conf; invalid character: '\' I am unsure what i need to do in order to get this correct; does anyone have any insight or resources I should further explore. Is there a REGEX specific doc for NXLOG? 

Hello, I'm quite new to nxlog, so forgive me if my question is  trivial but I'm having hard time to get the values I extract from my  logs using exec and a regex. I have a very large stash of old windows logs in text file (in multiline format), what I want to do is use nxlog to load them in graylog, but I want to format the log in a different way. It works perfectly when I do not use the the exec and the regexp, but it fails with the message: 2015-08-31 12:12:42 ERROR invalid keyword: $timestamp at C:\Program Files (x86)\nxlog\conf\nxlog.conf:36 The regex works when I test it using http://www.regexr.com/ The error seems in the way I'm trying to assign/write the variables matched by the regex   My nxlog.conf is like the below: ## This is a sample configuration file. See the nxlog reference manual about the ## configuration options. It should be installed locally and is also available ## online at http://nxlog.org/nxlog-docs/en/nxlog-reference-manual.html ## Please set the ROOT to the folder your nxlog was installed into, ## otherwise it will not start. #define ROOT C:\Program Files\nxlog define ROOT C:\Program Files (x86)\nxlog Moduledir %ROOT%\modules CacheDir %ROOT%\data Pidfile %ROOT%\data\nxlog.pid SpoolDir %ROOT%\data LogFile %ROOT%\data\nxlog.log <Extension gelf>     Module       xm_gelf </Extension> <Extension multiline>     Module      xm_multiline     HeaderLine  /^{/     EndLine     /^}/ </Extension> <Input in>         Module  im_file         File    "C:\\tmp\\\\example-log.txt"         SavePos  TRUE         Recursive TRUE         InputType       multiline         exec if $raw_event =~ /(?:\{"([0-9]+?), ([0-9]+?), "(.+?)", "(.+?)", "(?:.+?)","(.+?)\n(?:[a-zA-Z0-9:,"\n\r\s\f\t\-\{\}\.\(\)]+?)New\sLogon:\s*(?:.+?)\n\s*Account\sName:\s*(.+?)\n(?:[a-zA-Z0-9:,"\n\r\s\f\t\-\{\}\.\(\)]+?)Source\sNetwork\sAddress:\s*([0-9\.]{7,15})\n(?:[a-zA-Z0-9:,"\n\r\s\f\t\-\{\}\.\(\)]+?)"\})/g;            $timestamp = $1;      $event = $2;      $status = $3;      $type = $4;      $short = $5;      $user = $6;      $source = $7;       </Input> #<Output out> #    Module      om_tcp #    Host        192.168.1.15 #    Port        12201 #    OutputType  GELF_TCP #</Output> <Output out>     Module    om_file     File    'C:\\tmp\\output' </Output> <Route 1>     Path        in => out </Route>    Could someone kindly help me out on this? I tried to read the nxlog manual but I need to confess that I could not understand what I'm doing wrong  Basically I just would the output to be composed by the raw message and the few fields I match with the regex   Thanks very much!   Mark

Hi, I am trying to parse a log4net file into json. Here's my sample log4net: ---------------- 2015-01-27 01:06:18,859 [7] ERROR Web.Cms.Content.Base.Taxonomy.TaxonomyDetectionProvider [(null)] - Get taxonomy Type Failed for Tools 2015-01-27 06:34:31,051 [26] ERROR www.Status404 [(null)] - ErrorId: 20150127_102b01c6-3208-48c5-8c8b-ae4f92cf2b20     UserAgent: Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/39.0.2171.99 Safari/537.36     HostAddress: 192.168.10.2     RequestUrl: /ErrorPages/404.aspx     MachineName: QA01     Raw Url:/undefined/     Referrer: http://qa1.www.something.com/toolset.aspx 2015-01-27 06:34:33,270 [26] DEBUG Web.Caching.Core.CacheManagerBase [(null)] - Custom CacheProvider:Web.Caching.Core.AppFabricCacheManager,Web.Caching.Core Disabled   Now I am using xm_multiline to capture each log entries. ---------------- <Extension multiline>     Module        xm_multiline     HeaderLine    /^\d{4}\-\d{2}\-\d{2} \d{2}\:\d{2}\:\d{2},\d{3}/     EndLine        /\r?\n\r?\n^\d{4}\-\d{2}\-\d{2} \d{2}\:\d{2}\:\d{2},\d{3}/ </Extension> I use a regex to capture the timestamp as the header then I use a regex to capture twice newline then the next timestamp as endline. However it still treat the second and last entry as ONE log entry. Here's the output: ---------------- {     "EventReceivedTime":"2015-01-27 01:06:35",   "SourceModuleName":"log4net",   "SourceModuleType":"im_file",   "time":"2015-01-27 01:06:18,859",   "thread":"7",   "level":"ERROR",   "logger":"Web.Cms.Content.Base.Taxonomy.TaxonomyDetectionProvider",   "ndc":"(null)",   "message":"Get taxonomy Type Failed for Tools"}{     "EventReceivedTime":"2015-01-27 06:34:35",   "SourceModuleName":"log4net",   "SourceModuleType":"im_file",   "time":"2015-01-27 06:34:31,051",   "thread":"26",   "level":"ERROR",   "logger":"www.Status404",   "ndc":"(null)",   "message":"  ErrorId: 20150127_102b01c6-3208-48c5-8c8b-ae4f92cf2b20\r\n  UserAgent: Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/39.0.2171.99  Safari/537.36\r\n  HostAddress: 192.168.10.2\r\n  RequestUrl: /ErrorPages/404.aspx\r\n  MachineName: QA01\r\n   Raw Url:/undefined/\r\n  Referrer: http://qa1.www.something.com/toolset.aspx\r\n\r\n2015-01-27 06:34:33,270 [26] DEBUG Web.Caching.Core.CacheManagerBase [(null)] - Custom CacheProvider:Web.Caching.Core.AppFabricCacheManager,Web.Caching.Core Disabled"} I used this to produce that output: ---------------- Exec        if $raw_event =~ /^(\d{4}\-\d{2}\-\d{2} \d{2}\:\d{2}\:\d{2},\d{3}) \[(\S+)\] (\S+) (\S+) \[(\S+)\] \- (.*)/s \                 { \                     $time = $1; \                     $thread = $2; \                     $level = $3; \                     $logger = $4; \                     $ndc = $5; \                     $message = $6; \                     to_json(); \                 } \                 else \                 { \                     drop(); \                 }     I've also tried to tweak it by using this to avoid the combining the last two entries as one. However I am not able to get the last entry anymore. ---------------- Exec        if $raw_event =~ /^(\d{4}\-\d{2}\-\d{2} \d{2}\:\d{2}\:\d{2},\d{3}) \[(\S+)\] (\S+) (\S+) \[(\S+)\] \- ([\s\S]*?)(\r?\n\r?\n|$)/ \                 { \                     $time = $1; \                     $thread = $2; \                     $level = $3; \                     $logger = $4; \                     $ndc = $5; \                     $message = $6; \                     to_json(); \                 } \                 else \                 { \                     drop(); \                 }​