Pattern DB appears to drop fields

Tags: regex | pm_pattern

#1 progssilb

I'm trying to get a PatternDB working correctly, and it looks like I'm getting some fields but not all of them. There's only one pattern that's actually generating extra fields, and even it is dropping the first field (ParsedDate). Not sure what's going on here...


Config file (via file inclusion):

<Extension json>
    Module      xm_json
</Extension>

<Extension syslog>
    Module      xm_syslog
</Extension>

<Input vg_tsw_client>
  Module     im_file
  File       "C:\Program Files (x86)\Steam\steamapps\common\The Secret World\ClientLog.txt"
  Exec       if not ($raw_event =~ /Scaleform\.TSWACT/) drop();
  Exec         parse_syslog();
</Input>

<Input vg_tsw_combat>
  Module     im_file
  File       "C:\Program Files (x86)\Steam\steamapps\common\The Secret World\CombatLog-*.txt"
  Exec       if ($raw_event =~ /Sprinting [VI]+/) drop();
  Exec         parse_syslog();
</Input>

<Processor vg_tsw_pattern>
    Module    pm_pattern
    PatternFile %ROOT%\conf\SecretWorld\patterndb.xml
</Processor>

<Output vg_tsw_testfile>
  Module     om_file
  File       "C:\\ProgramData\\nxlogs\\vg-tsw-logs.log"
  Exec       to_json();
</Output>

<Route vg_tsw_route>
  Path       vg_tsw_client, vg_tsw_combat => vg_tsw_pattern => vg_tsw_testfile
</Route>


Pattern DB:

<?xml version='1.0' encoding='UTF-8'?>
<patterndb>
 <created>2010-01-01 01:02:03</created>
 <version>42</version>
 
<group>
  <name>tswCombat</name>
  <id>50284624</id>
  <matchfield>
    <name>SourceModuleName</name>
    <type>exact</type>
    <value>vg_tsw_combat</value>
  </matchfield>

  <pattern>
    <id>1000</id>
    <name>basic combat swing</name>

    <matchfield>
     <name>Message</name>
     <type>regexp</type>
        <!-- [00:00:28] (Critical) Solomon County Cop's Spray and Pray hits (Normal) Ravenous Horde for 522 physical damage. (Normal) -->
     <value>^\[([^\]]+)\] ((?:\(Critical\) |\(Normal\) )?)(.+?'s|Your) (.+?) hits \((Normal|Glancing)\) (.*?) for (\d+) (physical|magical) damage. \((Normal|Penetrated|Blocked)\)</value>
     <capturedfield>
      <name>ParsedTime</name>
      <type>datetime</type>
     </capturedfield>
     <capturedfield>
      <name>CriticalHit</name>
      <type>string</type>
     </capturedfield>
     <capturedfield>
        <name>AttackerName</name>
        <type>string</type>
     </capturedfield>
     <capturedfield>
        <name>AttackName</name>
        <type>string</type>
     </capturedfield>
     <capturedfield>
        <name>Glancing</name>
        <type>string</type>
     </capturedfield>
     <capturedfield>
        <name>VictimName</name>
        <type>string</type>
     </capturedfield>
     <capturedfield>
        <name>Damage</name>
        <type>integer</type>
     </capturedfield>
     <capturedfield>
        <name>DamageType</name>
        <type>string</type>
     </capturedfield>
     <capturedfield>
        <name>BlockOrPen</name>
        <type>string</type>
     </capturedfield>
    </matchfield>

    <set>
     <field>
       <name>type</name>
       <value>Swing</value>
       <type>string</type>
     </field>
    </set>
  </pattern>
</group>

<group>
  <name>tswClient</name>
  <id>50284625</id>
  <matchfield>
    <name>SourceModuleName</name>
    <type>exact</type>
    <value>vg_tsw_client</value>
  </matchfield>
 
  <pattern>
    <id>2000</id>
    <name>tswact load plugin</name>

    <matchfield>
     <name>Message</name>
     <type>regexp</type>
        <!-- [2017-02-10 05:47:07Z #3886] [ID:0] ERROR: Scaleform.TSWACT - TSWACT Loaded for |Sheriban| -->
     <value>^\[([0-9-:]+)Z #\d+\] \[ID:\d+\] ERROR: Scaleform.TSWACT - TSWACT Loaded for - \|(\w+)\|</value>
     <capturedfield>
      <name>ParsedTime</name>
      <type>string</type>
     </capturedfield>
     <capturedfield>
      <name>PlayerName</name>
      <type>string</type>
     </capturedfield>
    </matchfield>

    <set>
     <field>
       <name>type</name>
       <value>TswactLoaded</value>
       <type>string</type>
     </field>
    </set>
  </pattern>
 
  <pattern>
    <id>2001</id>
    <name>tswact load playfield</name>

    <matchfield>
     <name>Message</name>
     <type>regexp</type>
        <!-- [2017-02-10 05:47:07Z #3886] [ID:0] ERROR: Scaleform.TSWACT - Playfield - |Kingsmouth Town| -->
     <value>^\[([0-9-:]+)Z #\d+\] \[ID:\d+\] ERROR: Scaleform.TSWACT - Playfield - \|(\w+)\|</value>
     <capturedfield>
      <name>ParsedTime</name>
      <type>datetime</type>
     </capturedfield>
     <capturedfield>
      <name>ZoneName</name>
      <type>string</type>
     </capturedfield>
    </matchfield>

    <set>
     <field>
       <name>type</name>
       <value>SetZoneName</value>
       <type>string</type>
     </field>
    </set>
  </pattern>

  <pattern>
    <id>2002</id>
    <name>tswact enter combat</name>

    <matchfield>
     <name>Message</name>
     <type>regexp</type>
        <!-- [2017-02-10 05:00:22Z #10910] [ID:0] ERROR: Scaleform.TSWACT - Enter combat - |Sheriban|Buffs:Sprinting VI:Elemental Force:Third Degree :World Domination| -->
     <value>^\[([0-9-:]+)Z #\d+\] \[ID:\d+\] ERROR: Scaleform.TSWACT - Enter combat - \|(\w+)\|</value>
     <capturedfield>
      <name>ParsedTime</name>
      <type>datetime</type>
     </capturedfield>
     <capturedfield>
      <name>PlayerName</name>
      <type>string</type>
     </capturedfield>
    </matchfield>

    <set>
     <field>
       <name>type</name>
       <value>EnterCombat</value>
       <type>string</type>
     </field>
    </set>
    <exec>
      $TestField = 'testValue';
    </exec>
  </pattern>
</group>

</patterndb>


Some of the output I'm getting:

{"EventReceivedTime":"2017-02-10 11:45:00","SourceModuleName":"vg_tsw_combat","SourceModuleType":"im_file","SyslogFacilityValue":1,"SyslogFacility":"USER","SyslogSeverityValue":5,"SyslogSeverity":"NOTICE","SeverityValue":2,"Severity":"INFO","EventTime":"2017-02-10 11:45:00","Hostname":"shepard","Message":"[11:45:00] Your Pop Shot hits (Normal) Undead Islander for 1437 physical damage. (Normal)","CriticalHit":"","AttackerName":"Your","AttackName":"Pop Shot","Glancing":"Normal","VictimName":"Undead Islander","Damage":1437,"DamageType":"physical","BlockOrPen":"Normal","PatternID":1000,"PatternName":"basic combat swing","type":"Swing"}
{"EventReceivedTime":"2017-02-10 11:45:00","SourceModuleName":"vg_tsw_combat","SourceModuleType":"im_file","SyslogFacilityValue":1,"SyslogFacility":"USER","SyslogSeverityValue":5,"SyslogSeverity":"NOTICE","SeverityValue":2,"Severity":"INFO","EventTime":"2017-02-10 11:45:00","Hostname":"shepard","Message":"[11:45:00] (Critical) Your Pop Shot hits (Normal) Undead Islander for 2965 physical damage. (Penetrated)","CriticalHit":"(Critical) ","AttackerName":"Your","AttackName":"Pop Shot","Glancing":"Normal","VictimName":"Undead Islander","Damage":2965,"DamageType":"physical","BlockOrPen":"Penetrated","PatternID":1000,"PatternName":"basic combat swing","type":"Swing"}
{"EventReceivedTime":"2017-02-10 11:45:00","SourceModuleName":"vg_tsw_combat","SourceModuleType":"im_file","SyslogFacilityValue":1,"SyslogFacility":"USER","SyslogSeverityValue":5,"SyslogSeverity":"NOTICE","SeverityValue":2,"Severity":"INFO","EventTime":"2017-02-10 11:45:00","Hostname":"shepard","Message":"[11:45:00] You gain buff Live Wire"}
{"EventReceivedTime":"2017-02-10 11:45:01","SourceModuleName":"vg_tsw_client","SourceModuleType":"im_file","SyslogFacilityValue":1,"SyslogFacility":"USER","SyslogSeverityValue":5,"SyslogSeverity":"NOTICE","SeverityValue":2,"Severity":"INFO","EventTime":"2017-02-10 11:45:01","Hostname":"shepard","Message":"[2017-02-10 16:45:01Z #18498] [ID:0] ERROR: Scaleform.TSWACT - Out of combat - |Sheriban|"}
{"EventReceivedTime":"2017-02-10 11:45:10","SourceModuleName":"vg_tsw_combat","SourceModuleType":"im_file","SyslogFacilityValue":1,"SyslogFacility":"USER","SyslogSeverityValue":5,"SyslogSeverity":"NOTICE","SeverityValue":2,"Severity":"INFO","EventTime":"2017-02-10 11:45:10","Hostname":"shepard","Message":"[11:45:10] Buff Live Wire terminated."}


Some of the vg_tsw_combat input file:

[11:45:00] Your One in the Chamber hits (Normal) Undead Islander for 231 physical damage. (Normal)
[11:45:00] Buff Sudden Return terminated on Undead Islander.
[11:45:00] Buff One in the Chamber terminated on Undead Islander.
[11:45:00] You gained 146 XP.
[11:45:00] Undead Islander died.
[11:45:00] Your Sudden Return hits (Normal) Undead Islander for 259 physical damage. (Normal)
[11:45:00] Your Pop Shot hits (Normal) Undead Islander for 2045 physical damage. (Penetrated)
[11:45:00] Your Pop Shot hits (Normal) Undead Islander for 2175 physical damage. (Penetrated)
[11:45:00] Your Pop Shot hits (Normal) Undead Islander for 1437 physical damage. (Normal)
[11:45:00] (Critical) Your Pop Shot hits (Normal) Undead Islander for 2965 physical damage. (Penetrated)
[11:45:00] You gain buff Live Wire
[11:45:02] You start using Sprinting VI.
[11:45:03] You gain buff Sprinting VI
[11:45:03] You successfully used Sprinting VI.
[11:45:10] Buff Live Wire terminated.


Some of the vg_tsw_client input:

[2017-02-10 16:33:43Z #6790] [ID:0] ERROR: Scaleform.TSWACT - TSWACT Loaded for |Sheriban|
[2017-02-10 16:33:43Z #6790] [ID:0] ERROR: Scaleform.TSWACT - Playfield - |The Savage Coast|
[2017-02-10 16:34:12Z #7313] [ID:0] ERROR: Scaleform.TSWACT - Enter combat - |Sheriban|Buffs:World Domination|
[2017-02-10 16:34:14Z #7373] [ID:0] ERROR: Scaleform.TSWACT - Out of combat - |Sheriban|
[2017-02-10 16:39:06Z #10609] [ID:0] ERROR: MagicCommand - Trying to prepone the execute timeline to the pass. Spell:7760057
[2017-02-10 16:39:06Z #10624] [ID:0] ERROR: Scaleform.TSWACT - Enter combat - |Sheriban|Buffs:Elemental Force:World Domination|
[2017-02-10 16:39:08Z #10655] [ID:0] ERROR: Scaleform.TSWACT - Out of combat - |Sheriban|
[2017-02-10 16:44:58Z #18330] [ID:0] ERROR: MagicCommand - Trying to prepone the execute timeline to the pass. Spell:7760057
[2017-02-10 16:44:59Z #18388] [ID:0] ERROR: Scaleform.TSWACT - Enter combat - |Sheriban|Buffs:Elemental Force:World Domination|
[2017-02-10 16:45:01Z #18498] [ID:0] ERROR: Scaleform.TSWACT - Out of combat - |Sheriban|


Any ideas?

#2 progssilb
#1 progssilb
I'm trying to get a PatternDB working correctly, and it looks like I'm getting some fields but not all of them. There's only one pattern that's actually generating extra fields, and even it is dropping the first field (ParsedDate). Not sure what's going on here... Config file (via file inclusion): <Extension json>     Module      xm_json </Extension> <Extension syslog>     Module      xm_syslog </Extension> <Input vg_tsw_client>   Module     im_file   File       "C:\Program Files (x86)\Steam\steamapps\common\The Secret World\ClientLog.txt"   Exec       if not ($raw_event =~ /Scaleform\.TSWACT/) drop();   Exec         parse_syslog(); </Input> <Input vg_tsw_combat>   Module     im_file   File       "C:\Program Files (x86)\Steam\steamapps\common\The Secret World\CombatLog-*.txt"   Exec       if ($raw_event =~ /Sprinting [VI]+/) drop();   Exec         parse_syslog(); </Input> <Processor vg_tsw_pattern>     Module    pm_pattern     PatternFile %ROOT%\conf\SecretWorld\patterndb.xml </Processor> <Output vg_tsw_testfile>   Module     om_file   File       "C:\\ProgramData\\nxlogs\\vg-tsw-logs.log"   Exec       to_json(); </Output> <Route vg_tsw_route>   Path       vg_tsw_client, vg_tsw_combat => vg_tsw_pattern => vg_tsw_testfile </Route> Pattern DB: <?xml version='1.0' encoding='UTF-8'?> <patterndb>  <created>2010-01-01 01:02:03</created>  <version>42</version>   <group>   <name>tswCombat</name>   <id>50284624</id>   <matchfield>     <name>SourceModuleName</name>     <type>exact</type>     <value>vg_tsw_combat</value>   </matchfield>   <pattern>     <id>1000</id>     <name>basic combat swing</name>     <matchfield>      <name>Message</name>      <type>regexp</type>         <!-- [00:00:28] (Critical) Solomon County Cop's Spray and Pray hits (Normal) Ravenous Horde for 522 physical damage. (Normal) -->      <value>^\[([^\]]+)\] ((?:\(Critical\) |\(Normal\) )?)(.+?'s|Your) (.+?) hits \((Normal|Glancing)\) (.*?) for (\d+) (physical|magical) damage. \((Normal|Penetrated|Blocked)\)</value>      <capturedfield>       <name>ParsedTime</name>       <type>datetime</type>      </capturedfield>      <capturedfield>       <name>CriticalHit</name>       <type>string</type>      </capturedfield>      <capturedfield>         <name>AttackerName</name>         <type>string</type>      </capturedfield>      <capturedfield>         <name>AttackName</name>         <type>string</type>      </capturedfield>      <capturedfield>         <name>Glancing</name>         <type>string</type>      </capturedfield>      <capturedfield>         <name>VictimName</name>         <type>string</type>      </capturedfield>      <capturedfield>         <name>Damage</name>         <type>integer</type>      </capturedfield>      <capturedfield>         <name>DamageType</name>         <type>string</type>      </capturedfield>      <capturedfield>         <name>BlockOrPen</name>         <type>string</type>      </capturedfield>     </matchfield>     <set>      <field>        <name>type</name>        <value>Swing</value>        <type>string</type>      </field>     </set>   </pattern> </group> <group>   <name>tswClient</name>   <id>50284625</id>   <matchfield>     <name>SourceModuleName</name>     <type>exact</type>     <value>vg_tsw_client</value>   </matchfield>     <pattern>     <id>2000</id>     <name>tswact load plugin</name>     <matchfield>      <name>Message</name>      <type>regexp</type>         <!-- [2017-02-10 05:47:07Z #3886] [ID:0] ERROR: Scaleform.TSWACT - TSWACT Loaded for |Sheriban| -->      <value>^\[([0-9-:]+)Z #\d+\] \[ID:\d+\] ERROR: Scaleform.TSWACT - TSWACT Loaded for - \|(\w+)\|</value>      <capturedfield>       <name>ParsedTime</name>       <type>string</type>      </capturedfield>      <capturedfield>       <name>PlayerName</name>       <type>string</type>      </capturedfield>     </matchfield>     <set>      <field>        <name>type</name>        <value>TswactLoaded</value>        <type>string</type>      </field>     </set>   </pattern>     <pattern>     <id>2001</id>     <name>tswact load playfield</name>     <matchfield>      <name>Message</name>      <type>regexp</type>         <!-- [2017-02-10 05:47:07Z #3886] [ID:0] ERROR: Scaleform.TSWACT - Playfield - |Kingsmouth Town| -->      <value>^\[([0-9-:]+)Z #\d+\] \[ID:\d+\] ERROR: Scaleform.TSWACT - Playfield - \|(\w+)\|</value>      <capturedfield>       <name>ParsedTime</name>       <type>datetime</type>      </capturedfield>      <capturedfield>       <name>ZoneName</name>       <type>string</type>      </capturedfield>     </matchfield>     <set>      <field>        <name>type</name>        <value>SetZoneName</value>        <type>string</type>      </field>     </set>   </pattern>   <pattern>     <id>2002</id>     <name>tswact enter combat</name>     <matchfield>      <name>Message</name>      <type>regexp</type>         <!-- [2017-02-10 05:00:22Z #10910] [ID:0] ERROR: Scaleform.TSWACT - Enter combat - |Sheriban|Buffs:Sprinting VI:Elemental Force:Third Degree :World Domination| -->      <value>^\[([0-9-:]+)Z #\d+\] \[ID:\d+\] ERROR: Scaleform.TSWACT - Enter combat - \|(\w+)\|</value>      <capturedfield>       <name>ParsedTime</name>       <type>datetime</type>      </capturedfield>      <capturedfield>       <name>PlayerName</name>       <type>string</type>      </capturedfield>     </matchfield>     <set>      <field>        <name>type</name>        <value>EnterCombat</value>        <type>string</type>      </field>     </set>     <exec>       $TestField = 'testValue';     </exec>   </pattern> </group> </patterndb> Some of the output I'm getting: {"EventReceivedTime":"2017-02-10 11:45:00","SourceModuleName":"vg_tsw_combat","SourceModuleType":"im_file","SyslogFacilityValue":1,"SyslogFacility":"USER","SyslogSeverityValue":5,"SyslogSeverity":"NOTICE","SeverityValue":2,"Severity":"INFO","EventTime":"2017-02-10 11:45:00","Hostname":"shepard","Message":"[11:45:00] Your Pop Shot hits (Normal) Undead Islander for 1437 physical damage. (Normal)","CriticalHit":"","AttackerName":"Your","AttackName":"Pop Shot","Glancing":"Normal","VictimName":"Undead Islander","Damage":1437,"DamageType":"physical","BlockOrPen":"Normal","PatternID":1000,"PatternName":"basic combat swing","type":"Swing"} {"EventReceivedTime":"2017-02-10 11:45:00","SourceModuleName":"vg_tsw_combat","SourceModuleType":"im_file","SyslogFacilityValue":1,"SyslogFacility":"USER","SyslogSeverityValue":5,"SyslogSeverity":"NOTICE","SeverityValue":2,"Severity":"INFO","EventTime":"2017-02-10 11:45:00","Hostname":"shepard","Message":"[11:45:00] (Critical) Your Pop Shot hits (Normal) Undead Islander for 2965 physical damage. (Penetrated)","CriticalHit":"(Critical) ","AttackerName":"Your","AttackName":"Pop Shot","Glancing":"Normal","VictimName":"Undead Islander","Damage":2965,"DamageType":"physical","BlockOrPen":"Penetrated","PatternID":1000,"PatternName":"basic combat swing","type":"Swing"} {"EventReceivedTime":"2017-02-10 11:45:00","SourceModuleName":"vg_tsw_combat","SourceModuleType":"im_file","SyslogFacilityValue":1,"SyslogFacility":"USER","SyslogSeverityValue":5,"SyslogSeverity":"NOTICE","SeverityValue":2,"Severity":"INFO","EventTime":"2017-02-10 11:45:00","Hostname":"shepard","Message":"[11:45:00] You gain buff Live Wire"} {"EventReceivedTime":"2017-02-10 11:45:01","SourceModuleName":"vg_tsw_client","SourceModuleType":"im_file","SyslogFacilityValue":1,"SyslogFacility":"USER","SyslogSeverityValue":5,"SyslogSeverity":"NOTICE","SeverityValue":2,"Severity":"INFO","EventTime":"2017-02-10 11:45:01","Hostname":"shepard","Message":"[2017-02-10 16:45:01Z #18498] [ID:0] ERROR: Scaleform.TSWACT - Out of combat - |Sheriban|"} {"EventReceivedTime":"2017-02-10 11:45:10","SourceModuleName":"vg_tsw_combat","SourceModuleType":"im_file","SyslogFacilityValue":1,"SyslogFacility":"USER","SyslogSeverityValue":5,"SyslogSeverity":"NOTICE","SeverityValue":2,"Severity":"INFO","EventTime":"2017-02-10 11:45:10","Hostname":"shepard","Message":"[11:45:10] Buff Live Wire terminated."} Some of the vg_tsw_combat input file: [11:45:00] Your One in the Chamber hits (Normal) Undead Islander for 231 physical damage. (Normal) [11:45:00] Buff Sudden Return terminated on Undead Islander. [11:45:00] Buff One in the Chamber terminated on Undead Islander. [11:45:00] You gained 146 XP. [11:45:00] Undead Islander died. [11:45:00] Your Sudden Return hits (Normal) Undead Islander for 259 physical damage. (Normal) [11:45:00] Your Pop Shot hits (Normal) Undead Islander for 2045 physical damage. (Penetrated) [11:45:00] Your Pop Shot hits (Normal) Undead Islander for 2175 physical damage. (Penetrated) [11:45:00] Your Pop Shot hits (Normal) Undead Islander for 1437 physical damage. (Normal) [11:45:00] (Critical) Your Pop Shot hits (Normal) Undead Islander for 2965 physical damage. (Penetrated) [11:45:00] You gain buff Live Wire [11:45:02] You start using Sprinting VI. [11:45:03] You gain buff Sprinting VI [11:45:03] You successfully used Sprinting VI. [11:45:10] Buff Live Wire terminated. Some of the vg_tsw_client input: [2017-02-10 16:33:43Z #6790] [ID:0] ERROR: Scaleform.TSWACT - TSWACT Loaded for |Sheriban| [2017-02-10 16:33:43Z #6790] [ID:0] ERROR: Scaleform.TSWACT - Playfield - |The Savage Coast| [2017-02-10 16:34:12Z #7313] [ID:0] ERROR: Scaleform.TSWACT - Enter combat - |Sheriban|Buffs:World Domination| [2017-02-10 16:34:14Z #7373] [ID:0] ERROR: Scaleform.TSWACT - Out of combat - |Sheriban| [2017-02-10 16:39:06Z #10609] [ID:0] ERROR: MagicCommand - Trying to prepone the execute timeline to the pass. Spell:7760057 [2017-02-10 16:39:06Z #10624] [ID:0] ERROR: Scaleform.TSWACT - Enter combat - |Sheriban|Buffs:Elemental Force:World Domination| [2017-02-10 16:39:08Z #10655] [ID:0] ERROR: Scaleform.TSWACT - Out of combat - |Sheriban| [2017-02-10 16:44:58Z #18330] [ID:0] ERROR: MagicCommand - Trying to prepone the execute timeline to the pass. Spell:7760057 [2017-02-10 16:44:59Z #18388] [ID:0] ERROR: Scaleform.TSWACT - Enter combat - |Sheriban|Buffs:Elemental Force:World Domination| [2017-02-10 16:45:01Z #18498] [ID:0] ERROR: Scaleform.TSWACT - Out of combat - |Sheriban| Any ideas?

OK, I tried formatting this with indents and horizontal lines, but neither of those stuck. Sorry.