A fuzzy match score tells you how similar two strings are. It does not tell you whether they refer to the same real-world entity. "IBM" and "IBM India" have a reasonably high string similarity. They are not the same entity in most business contexts. A confidence scorer bridges this gap: it combines multiple match signals into a single calibrated probability that the candidate pair is a true match.
This post covers how to build that scorer from first principles, what features to use, and how to calibrate it so that a 0.9 confidence score actually corresponds to a ~90% precision rate in practice.
Why a Single String Similarity Score Is Not Enough
String similarity functions (Jaro-Winkler, edit distance, cosine over character n-grams) are good at catching name variations within a single language and convention. They have systematic blind spots:
- Abbreviations: "General Electric" vs. "GE" has low string similarity but is often the same entity
- Name transpositions: "Mehta & Associates" vs. "Associates, Mehta" has moderate similarity but is the same firm
- Noisy tokens: "LLC", "Inc", "Ltd" create similarity inflation between unrelated companies that share these suffixes
- Short names: common short names ("Atlas", "Apex") have high similarity to each other but are often completely different businesses
A single string similarity threshold produces too many false positives in the short-name case and too many false negatives in the abbreviation case. A confidence scorer that incorporates non-name signals handles both better.
Feature Construction
For a pair of entity records (A, B), compute the following feature vector:
| Feature | How to compute | Range |
|---|---|---|
name_jaro |
Jaro-Winkler similarity on normalized names | 0.0-1.0 |
name_token_sort |
Token sort ratio (handles word order variation) | 0-100 |
domain_match |
1.0 if email domain matches, 0.0 otherwise | 0.0 or 1.0 |
address_similarity |
Jaro-Winkler on normalized street address | 0.0-1.0 |
phone_match |
Exact match on normalized E.164 phone | 0.0 or 1.0 |
type_suffix_stripped |
Name similarity after stripping Inc/Ltd/LLC/Corp | 0.0-1.0 |
Name normalization before computing any similarity: lowercase, strip punctuation, expand common abbreviations if you have a lookup table (St. to Street, Ave. to Avenue, Co. to Company).
import re
SUFFIXES_TO_STRIP = {"inc", "incorporated", "ltd", "limited",
"llc", "corp", "corporation", "co", "company"}
def normalize_entity_name(name: str) -> str:
name = name.lower()
name = re.sub(r'[^\w\s]', '', name)
tokens = [t for t in name.split() if t not in SUFFIXES_TO_STRIP]
return ' '.join(tokens).strip()The type_suffix_stripped feature specifically addresses the inflated similarity between "Acme Corp" and "Other Corp" that appears when both share the "Corp" token.
Training the Scorer
The scorer is a binary classifier trained on labeled pairs. You need a ground-truth labeled dataset of (entity_a, entity_b, label) where label is 1 for true matches and 0 for non-matches. Creating this dataset requires human review of a sample of candidate pairs.
The class imbalance challenge: in a typical B2B company dataset, true matches are a small fraction of all candidate pairs after blocking. A 10,000-entity dataset with 5% duplicates has approximately 500 true match pairs. A naive classifier that labels everything as non-match gets 95% accuracy while being useless.
Handle this with stratified sampling during training and by evaluating on precision-recall metrics rather than accuracy. For entity resolution, precision typically matters more than recall: a false positive (merging two actually different entities) causes real damage downstream, while a false negative (missing a duplicate) can sometimes be caught on a later pass.
from sklearn.ensemble import GradientBoostingClassifier
from sklearn.model_selection import cross_val_predict
from sklearn.calibration import CalibratedClassifierCV
base_clf = GradientBoostingClassifier(n_estimators=100, max_depth=4)
calibrated_clf = CalibratedClassifierCV(base_clf, method='isotonic', cv=5)
calibrated_clf.fit(X_train, y_train)
proba = calibrated_clf.predict_proba(X_test)[:, 1]The calibration step is important. Without calibration, the probability outputs from a gradient boosting classifier are not well-calibrated: a raw score of 0.8 does not mean the pair is a true match 80% of the time. Isotonic regression calibration fits a monotone function from predicted probabilities to empirical probabilities using cross-validation. After calibration, the probability output approximates the true precision at each threshold.
Setting Thresholds
With a calibrated scorer, you can set thresholds based on your data team's tolerance for false positives and false negatives. A practical three-tier threshold structure:
| Confidence range | Action | Rationale |
|---|---|---|
| 0.95 and above | Auto-merge canonical record | Very high precision in this band |
| 0.70 to 0.94 | Queue for human review | Majority are true matches, some borderline cases |
| Below 0.70 | Treat as non-match | Below this band, false positive rate climbs quickly |
The specific cutoffs should be tuned on your data. Plot a precision-recall curve for your validation set and find the threshold at which precision first drops below your tolerance. The auto-merge threshold should be set where your validation precision is above 98%. Below that, human review is cheaper than undoing incorrect merges.
Feature Weights Are Data-Dependent
One thing we found when building the initial HyperNorm entity resolution engine: feature importance varies significantly between data domains. For a B2B CRM dataset, domain match is a very strong signal because each company typically has one email domain. For a consumer database, it's weaker because many individuals share employer domains. For an address-heavy logistics dataset, address similarity carries more weight.
This means a model trained on one client's data should not be deployed as-is on another client's data without at least a validation pass. The feature construction stays the same; the weights the classifier learns need to reflect the actual signal strength in the specific dataset. We retrain on a per-connector basis for this reason, using the client's labeled sample as the training seed rather than a generic pre-trained model.
The Limits of Automated Confidence Scoring
Confidence scoring works well for the high-confidence and clear non-match tails of the distribution. The middle band (0.60 to 0.85 in practice) often contains structurally ambiguous pairs that no classifier can resolve with high confidence from the available features.
Example: "Atlas Logistics" and "Atlas Freight Solutions" with no domain or address overlap. These are plausibly the same company (rebrand or acquisition) or plausibly two different companies in the same vertical. The name similarity is moderate, the supporting features are absent, and the score will land in the middle band regardless of how good your classifier is.
For these cases, the scorer's output should be treated as a routing signal to a human reviewer with context, not as a final decision. Building a feedback loop where reviewer decisions on mid-band pairs feed back into future training rounds gradually improves the classifier's coverage of ambiguous cases as it accumulates labeled examples from your specific data domain.
