Cleaning vendor and customer master data with AI

Duplicate and mismatched master records get cleaned by treating deduplication as entity resolution rather than string matching: block the records into candidate groups, score every pair across name, address, tax ID and bank details together, then apply survivorship rules to pick one surviving record. The model proposes merges. A reviewer approves the ones above your false-positive budget, and the rest go straight through.

Bad vendor master data does not announce itself. It sits quietly until the same supplier, entered twice under slightly different names, receives two payments for one invoice. Or until three-way match fails because the PO points at “Acme Ltd” and the invoice arrives from “Acme Limited,” and a clerk keys around it. The messy table itself is cheap. The duplicate payment is not, and neither are the hours someone loses reconciling a total that three systems report three different ways.

Why the duplicates exist in the first place

Master data degrades through normal use. A supplier gets onboarded by procurement, then again by accounts payable when the first record cannot be found. A company changes its trading name after an acquisition and both versions live on. Someone pastes an address with a trailing space. None of these are dramatic, and that is the problem: each one looks fine on its own screen.

The patterns worth naming, because they drive how you match:

• Near-duplicate names: “Acme Ltd,” “Acme Limited,” “ACME LTD.” Same entity, three strings.
• Distinct entities that look identical: two subsidiaries at one head-office address, with different tax IDs and different bank accounts. Merging these is worse than leaving them apart.
• Transposed or partial identifiers: a VAT number with a digit swapped, an IBAN captured to the wrong length.
• Stale versus current: an old registered address alongside the new one, both technically correct at different points in time.

That last case is why point-in-time correctness matters even here. If you are matching an invoice dated in March against a master record whose address changed in May, the March-era attributes are the ones that should agree. Collapsing every version into one “golden” row throws away the history that reconciliation and audit later need.

The entity-resolution approach

String similarity on one field is not enough, and comparing every record against every other record does not scale. A million-vendor table is half a trillion pairs. The method has three stages.

Blocking comes first. You partition the records into candidate groups so that only plausibly-matching records are ever compared. A block key might be the first four characters of a normalised name plus the postal code, or a phonetic encoding of the name, or the tax ID prefix. Good blocking cuts the comparison space by orders of magnitude while keeping true matches inside the same block. If a real duplicate pair never lands in a shared block, no downstream model can recover it, so blocking recall is the number to watch.

Scoring comes second. Within each block, every candidate pair gets a similarity vector: edit distance on the name, token overlap on the address, exact-or-not on tax ID, exact-or-not on bank account, distance between registration dates. A model (often gradient-boosted trees, sometimes a fine-tuned encoder for the text fields) turns that vector into a match probability. The point of scoring across fields together is that no single field decides. A shared bank account plus a similar name is a strong match even when the addresses differ. A shared address with different tax IDs and different bank accounts is a strong non-match, which is exactly the subsidiary case you must not merge.

Survivorship comes third. Once a cluster of records is judged to be the same entity, you decide which values survive into the merged record: the most recent verified bank account, the longest complete address, the tax ID that validates against the registry. This is where lineage earns its keep. Every surviving field should carry a pointer back to the source record it came from, so an auditor can ask why the merged vendor has this IBAN and get an answer.

Setting the threshold, and living with it

There is no threshold that catches every duplicate and merges nothing it shouldn’t. You are trading two error types, and in payments they are not symmetric. A missed duplicate leaves two records that might cause a double payment later. A wrong merge fuses two real entities and can route one company’s money to another’s account. The second is worse, and your false-positive budget should reflect that.

So set the operating point deliberately:

• Build an eval set of labelled pairs that includes real matches, real non-matches, and the hard subsidiary cases, and measure precision and recall on it, not on a demo.
• Pick a high-confidence band that auto-merges straight through, a low band that auto-rejects, and a middle band that a human reviews. The width of that middle band is a cost decision, not a technical one.
• Route every auto-merge and every reviewer decision into an audit trail, so a merge can be explained and, if it was wrong, unwound.

Then watch for drift. A new source system, a different onboarding form, a change in how one region formats addresses: any of these can shift the score distribution and quietly move records across your thresholds. Re-scoring against the eval set on a schedule, and again before quarter-end when the volume spikes, tells you whether the operating point still holds. Master data never reaches a finished state. It is a table that keeps drifting while people enter suppliers the way people do, and the resolution pipeline is what holds it in place.