Duplicate Detection Rules

After attending the PTP PowerBase training in August 2018, we (Lisa & Yongho) are making a series of changes to deal with duplicates in PowerBase. After doing the initial configuration in 2012, we didn't revise PowerBase's duplicate detection default settings, and now realized that it was too strict for some situations, and too loose for other cases. 

We are improving the duplicate rules, which will reduce the number of duplicates. We will also be addressing existing duplicates in bulk in the coming months. Below are details:

From September 7, 2018, the following automatic rules will apply:

  • For every entry made into PowerBase, either through staff, online contribution, petition etc, if the following rules match, the entry will be considered a duplicate and be merged together automatically:
    • If:
      • Last name is same, and
      • First 3 letters of first name is same, and
      • either 
        • phone is same, or
        • street address is same, or
          • (for entries made by staff, it will only compare the first 9 letters of the street address, to take into account the different ways in which people enter apartment numbers)
        • email is same
    • then the entry will be considered a duplicate, and be merged together with the already existing entry. Meaning that if a person donates and it matches through the above rules with an existing contact, the donation will be recorded for the existing person, not as a new entry.
  • For staff entries, staff will see a warning sign on the top right corner of the screen, prompting them to consider using the system-detected original entry. Staff can override the warning and create the new entry by selecting "
    • However, due to a technicality, most of the ways in which staff enter information - New Contact Form, Service Entry Form, Voter Registration Form - are not considered staff entry and are instead treated as a non-staff form of entry. (So the "first 9 letter of address" special rule doesn't apply, and there is no warning sign)
  • What the rules looked like before: for people, it only checked for name and email - meaning it did not check at all into people with no email. And for organizations, it only compared the first 10 letters of the organization's name, which is a very loose rule resulting in too many false positives. We addressed this by making the organization name check in full (the entire org name must match)

Reference

Projects: