Master data and entity resolution

Master data is the small set of business entities that many systems must refer to consistently: customers, patients, suppliers, products, sites, assets, employees. It is the noun layer of the organisation. It changes slowly, it is referenced everywhere, and when two systems disagree about it every report built on either one becomes suspect.

Transactional data records events. A payment, a meter read, a hospital admission, an order line. Each row is a thing that happened at a time, and almost every row points at one or more master data entities. Transactional data is high volume, append-heavy and rarely disputed on its own. It becomes disputed when the entity it points at turns out to be ambiguous, because then the aggregate is ambiguous too.

Reference data is the controlled list of permitted values used to classify the other two: country codes, currency codes, tariff types, meter point classifications, diagnosis codes. Reference data has a publisher, a version and a change announcement. It looks trivial until two systems hold different vintages of the same list, at which point every join between them is quietly wrong in a way that produces plausible numbers rather than obvious errors.

The practical test is what breaks when the value changes. Change a transaction and you have corrected one event. Change a reference code and you have reclassified a population. Change a master data record and you have changed who the organisation thinks it is dealing with, which is why master data changes need an owner, an audit trail and a rule about who is allowed to make them.

There is a definitional step that comes before any of this, and skipping it is what makes the later work impossible. The word customer has to mean one thing. Is a customer a person, a household, a legal entity, a billing account or a supply point? An energy retailer that treats the customer as a supply point will report two customers for a couple who hold gas and electricity separately. One that treats the customer as a person will report one. Neither is wrong, but only one of them can be the answer the board is given, and the choice has consequences for consent, for arrears handling and for how a complaint is traced. The entity definition is a governance artefact, and it should be written down and approved before anyone builds a matching pipeline to enforce it.

Entity resolution is the work of deciding which records, within or across datasets, describe the same real thing. Inside a single well-designed database this is not a problem: aprimary key settles it by construction. The problem appears at the seams, where systems were built at different times by different teams for different purposes and none of them was asked to share a key.

Public bodies almost never get a shared identifier. There is no single citizen number that runs across justice, health, education and welfare, and the deliberate absence of one is a privacy design choice rather than an oversight. Commercial organisations are not much better off: a customer acquired through an acquisition, a broker, a marketplace and a shop front arrives four times with four internal identifiers and no way to reconcile them.

The fields that remain are all imperfect in specific ways.

• Names vary lawfully. Marriage, transliteration, initials, double-barrelled surnames, cultural ordering of given and family names, and the simple fact that call handlers type what they hear.
• Addresses vary structurally. Flat 3, 12 Marsh Court and 12A Marsh Court may be the same door or two different ones. Building names come and go. Postcodes are re-sectored.
• Dates of birth are entered wrongly. Day and month are transposed, and default values such as the first of January cluster suspiciously.
• Contact details are shared and reused. A household shares a landline. A carer uses one email address for four people. A mobile number is recycled by the network after a year.

So the honest position is that any decision about whether two records are the same is uncertain, and the only defensible thing to do with an uncertain decision is to attach a number to it and state the threshold at which you act. That is what makes this a statistics problem rather than a data cleansing chore.

Linkage pipelines converge on five steps, and the reason is arithmetic rather than fashion. Two datasets of one million records each generate a million million candidate pairs. No amount of hardware makes that comparison sensible, so the chain is built to discard almost all of it cheaply and to spend real effort only on the small remainder that could plausibly match.

1. Standardise. Bring every field to one representation before anything is compared. Upper-case and trim names, expand or strip titles, parse addresses into components, normalise postcodes, convert dates to one format, strip punctuation from telephone numbers. Standardisation is unglamorous and it removes more spurious mismatches than any clever algorithm later in the chain.
2. Block. Only compare pairs that agree on something cheap and reasonably discriminating: the same postcode district, the same year of birth, the same first three letters of the surname. Blocking is the step that makes the problem tractable. It is also the step that silently loses true matches, because a pair that disagrees on the blocking field is never compared at all. Serious implementations use several blocking rules in combination so that a record has more than one chance to meet its twin.
3. Compare. For each surviving candidate pair, score each field on a small scale rather than as a simple equal or not equal. Exact match, near match by edit distance or phonetic code, same value in transposed positions, both missing, one missing, clear disagreement.
4. Score. Combine the field-level comparisons into one number for the pair, weighting each field by how much its agreement or disagreement actually tells you.
5. Decide. Compare the score with a threshold. Above it, link. Below it, leave separate. In between, send to a human.

Blocking deserves more attention than it usually gets, because it is the only step whose errors are invisible. A pair that blocking never generated cannot be scored, cannot appear in the review queue and will never show up in a precision measurement, since precision only ever looks at the pairs the model did consider. The way to see the damage is to measure recall against a labelled sample, or at minimum to run a second blocking pass on a deliberately different field and count how many new true pairs it surfaces. If a second pass on year of birth finds matches that the postcode pass missed, that is a direct estimate of what a single-rule design would have thrown away.

Notice that the chain has three outcomes, not two. The middle band is not a failure of the method; it is the method being honest about the pairs where the evidence genuinely does not settle the question.

Probabilistic record linkage turns the pattern of agreement across several fields into a probability that the pair refers to the same entity. The framework that underpins almost every implementation was set out by Ivan Fellegi and Alan Sunter in 1969, and the intuition is simpler than the algebra.

For each comparison field, ask two questions. First, if these two records really are the same person, how often would this field agree? Call that the match probability. It is below one because data is dirty: people move, names are mistyped, dates of birth are entered wrongly. Second, if these two records are different people drawn at random, how often would this field agree anyway? Call that the chance probability. It is above zero because coincidence exists, and for some fields it is not small at all.

The weight that a field contributes is the ratio between those two numbers. A field where agreement is likely for a true pair and unlikely by chance carries a great deal of evidence. A field where agreement happens all the time by chance carries almost none. Agreement on a rare surname is powerful. Agreement on a common forename is nearly worthless. Agreement on gender in a two-value field is worth so little that including it can be actively misleading, because a long run of weak agreements can accumulate into a score that looks like evidence and is not.

Disagreement carries evidence in the other direction, and it is weighted the same way. Two records that disagree on date of birth are pushed apart hard, because true pairs rarely disagree there. Two records that disagree on telephone number are pushed apart only gently, because people change numbers all the time. This is why probabilistic matching beats deterministic rules: a rule says a field matches or it does not, whereas a weight says how much that fact should move your belief.

Adding the weights across all the compared fields gives the match weight for the pair, which converts to a probability. The parameters themselves are usually not hand-set. They are estimated from the data using expectation-maximisation, an iterative procedure that alternates between guessing which pairs are matches and re-estimating the field probabilities from those guesses until the estimates stop moving.

Once every candidate pair has a score, someone has to say how high the score must be before the pair is linked automatically. That choice is a trade between two errors that are not equally costly, and the relative cost depends entirely on what the linked data is used for.

A false match joins two different people into one record. In a marketing list that means one wasted mailing. In a clinical or a criminal justice context it can mean one person carrying another person's history, which is a harm that may take years to surface and is very hard to undo. A missed match leaves one person as two records. In research it biases every count downward. In service delivery it means the vulnerable customer flag sits on the record nobody is looking at.

So the threshold belongs to whoever owns the consequence, not to whoever wrote the pipeline. A data stewardwho can describe what a wrong link does to a person is the right person to set it, and the setting should be written down alongside the reason. The middle band that goes to human review is the same decision in a different form: it says which range of evidence is worth a person's time.

Two operational consequences follow. The linkage output should record the score and the version of the model that produced it, so a link can be explained later. And merges must be reversible, because the review that overturns a link may arrive months after the link was made, quite possibly as a complaint from the person the link was about.

One further complication surfaces as soon as you move from pairs to people. Linkage produces pairwise decisions, but an entity is a group. Record A links to record B, and record B links to record C, yet A and C may score below the threshold against each other. Treating the links as a graph and taking the connected components resolves that, but it also means a single weak link can chain two otherwise unrelated groups into one large cluster. Implementations guard against this by requiring a minimum density inside a cluster before accepting it, or by breaking the weakest links until the clusters are plausible, and by capping cluster size so that a runaway group is flagged for review rather than merged. A cluster containing four hundred people is not a very popular customer; it is a blocking or scoring defect announcing itself.

A golden record is the consolidated record an organisation agrees to treat as the version of an entity, assembled from the duplicates that resolution has matched together. Survivorship is the set of rules that decides, field by field, which source value wins. Several systems hold different values for the same handful of fields, and one named rule per field decides which value survives into the golden record and which is retained only as history.

Survivorship rules are chosen per field, not per source, and the common ones are:

• System of record. One named system is authoritative for this field by policy. The billing platform owns the legal name because that is the name on the contract.
• Most recently updated. The freshest value wins. Sensible for a mobile number, dangerous for a legal name, because recency is not the same as correctness.
• Most complete. The value with the most populated sub-components wins. Useful for addresses, where one source has the flat number and another does not.
• Highest trust tier. Sources are ranked by how they were captured. A value verified against an external register outranks one typed into a web form.
• Most restrictive. Reserved for consent and marketing preference. If any source records an objection, the objection survives.

That last rule is different in kind from the others. Applying most recently updated to a consent flag will eventually resurrect a withdrawn consent from a stale system, and the person on the receiving end will experience that as being ignored. Consent state should be treated as one-way: an objection propagates into the golden record and never falls out of it because another source is quiet.

The golden record must also keep its own history. Recording which source supplied each surviving value, and when, is what makes the record explainable. An organisation that cannot say why the golden record holds the address it holds has not built an authority. It has built one more version of the customer with better branding.

Implementations of master data management sit on a spectrum, and the position you choose is mostly a statement about how much you are willing to change the systems that already exist.

• Registry. The hub stores only the matched identifiers and the cross-references between systems. Values stay where they are and are assembled on demand. Cheap and unintrusive, and it changes nothing about the source systems, but it gives you a read-only view and no enforcement.
• Consolidation. Source values are copied into the hub, matched and survived into golden records there. Reporting and analytics read the hub. The sources are untouched and carry on diverging, so the hub is a point-in-time consolidation refreshed on a schedule.
• Coexistence. As consolidation, but the golden values are published back to the source systems, so the estate converges over time. This is where most organisations land, and it is the first style that requires the source system owners to agree to accept a value they did not create.
• Centralised authoring. The hub is the place where master data is created and edited, and the sources subscribe. It gives the strongest guarantee and demands the most change, because a business process that used to create a customer in the billing system now has to create one somewhere else.

The technical differences between these are smaller than they look. The real difference is political: each step along the spectrum takes a decision that a source system owner used to make and moves it somewhere central. That is why master data programmes fail on governance far more often than on matching accuracy, and why the operating model question of who may create, amend and merge an entity should be settled before a tool is selected.

Splink is an open-source Python package built by analysts at the Ministry of Justice. Its own documentation describes it as a package for probabilistic record linkage, or entity resolution, that lets you deduplicate and link records from datasets without unique identifiers, and states that its core linkage algorithm is based on the Fellegi-Sunter model of record linkage with customisations to improve accuracy.

It matters for three reasons beyond the code. First, it is a government department publishing its statistical method rather than describing it, which means the weighting and the thresholds can be inspected and argued with. Second, it originated in a real constraint that most organisations share: the Ministry of Justice operates several administrative justice systems with no consistent person identifier running between them, and the Data First programme needed to follow individuals through those systems for research. Third, GOV.UK reports that the department has used it on record linkage problems of up to around fifteen million records with a runtime of less than an hour. That is a stated ceiling rather than a typical job, but it puts the technique inside the reach of an ordinary analytics team rather than a specialist vendor engagement.

Splink also ships the diagnostics that make a linkage defensible: charts of the estimated match weights per field, comparisons of the model against labelled data, and cluster views that show which records ended up joined into one entity. Those outputs are the evidence you present when someone challenges a link, and their absence is what makes a hand-rolled matching script hard to defend.