Orchestrate

Report an issue

Something wrong, out of date or unclear? Tell us. Corrections are prioritized over new topics. You don't need to sign in.

0 / 2000

Don't include personal or confidential company details. Reports are reviewed with AI assistance.

Master data quality as AI readiness: duplicates, golden records and agents

Why duplicate and inconsistent master data breaks AI agents, how matching and golden records fix it, and what SAP MDG and SAP Reltio offer.

Updated Nov 16, 2026Foundational 9 minDeep 40 min
Foundational layer · 9 min read

The 60-second version

An AI agent answers a question about a customer, a supplier or a product by looking the record up. If the business has two records for one customer, the agent finds one and reports half the truth. If a field is blank, it guesses or stops. If one product's weight is in grams and another's in kilograms, it multiplies the wrong number with total confidence.

People work around bad master data every day without noticing: they know the second Brennock account is "the same people". An agent doesn't know. So master data quality is not a background hygiene task for AI. It decides whether the agent's answers are right.

The fix has a name: entity resolution. Find the records that describe the same real thing (matching), combine them into one trusted version (a golden record), remember which source record belongs to which golden record (the cross-reference), and send the fixes back to the systems that own the data. SAP offers this in SAP Master Data Governance (MDG) and in SAP Reltio, which SAP called generally available in SAP Business Data Cloud on 29 October 2026.

This topic shows what breaks, how the fix works, and how to prove the cleanup helped: not with a quality score alone, but with the questions your agents must answer correctly.

Why it matters to the business

A split customer is a wrong credit decision. At Calder Pumpworks, our fictional pump maker, a sales clerk created a second customer record for Brennock Water Utility during a rush order. Brennock's open orders now sit on two numbers. An agent asked "how much credit headroom does Brennock have?" reads the first record, misses 8,400.00 USD of orders on the second, and reports more headroom than Brennock really has.

A duplicate supplier is a payment risk. A buyer created ORRIN MOTORWORKS INC next to the existing Orrin Motor Works supplier, without payment terms. An invoice agent that picks the newer record can't work out the due date. Worse, duplicate suppliers are a classic route to paying the same invoice twice.

Mixed units are silent. A service clerk created a second seal kit with its weight in grams. A freight agent that reads the number without the unit plans 48 tonnes of seal kits instead of 48 kilograms.

Agents scale the error. A person makes this mistake once and notices. An agent makes it on every run, and its answers look just as confident when they are wrong. That is why data teams list master data among the first readiness checks, as covered in Is your SAP data ready for AI?.

The cost sits in the cleanup, not the software. Matching tools find candidates quickly. The expensive part is people deciding the unclear cases, moving open documents off duplicates, and filling facts no system holds. Budget steward time, not just licenses.

How SAP does it

As of November 2026, the SAP pages opened for this topic describe three routes. Check each against your contract and release.

Offering What it does for duplicates Where it runs
SAP Master Data Governance (MDG), consolidation Loads records from several sources, checks and standardizes them, finds duplicates, lets people review the matches, and calculates a best record before activating it (SAP Learning) MDG on SAP S/4HANA; a cloud-ready mode covers business partners (SAP Learning)
SAP MDG, central governance New and changed records go through a governed request, so duplicates are stopped at creation. Joule in MDG lets users search, display and change business partner records in plain language (SAP product page) MDG on S/4HANA, including Cloud Private Edition
SAP Reltio in SAP Business Data Cloud AI-based entity resolution that matches and reconciles records across SAP and non-SAP systems into unified profiles, published as governed data products for Joule and Joule agents. SAP says it is "now generally available in SAP BDC" as of 29 October 2026 (SAP News) SAP Business Data Cloud

SAP's May 2026 article describes Reltio as giving Joule agents "a golden record system of context". It names MDG a core component of SAP Business Data Cloud, governing master data and policies (SAP News, 13 May 2026). How MDG and Reltio divide the work in one landscape is not yet spelled out on a page we could open, so ask SAP before you design around both.

A cleanup on one page

Every master data cleanup for AI runs the same five steps. The tools differ; the decisions don't.

flowchart LR
  A[Standardize<br/>names, units, codes] --> B[Match<br/>find candidate pairs]
  B --> C{Clear case?}
  C -- yes --> D[Merge into<br/>golden record]
  C -- unclear --> E[Data steward<br/>decides]
  E --> D
  D --> F[Fix the source<br/>block, link, fill]
  F --> G[Re-test the<br/>agent questions]
Step Who What you get What can go wrong
Standardize Data team, with rules agreed by owners One spelling, one unit, one code list Over-cleaning: "Systems" and "Services" become alike
Match Matching tool Pairs with a score and a reason Too low a bar merges different companies
Review Data steward from the business A merge or keep-apart decision, with a reason A queue nobody works, so duplicates stay
Merge Matching tool, under agreed rules A golden record and a cross-reference Rules that let a weak source overwrite a strong one
Fix the source Master data team, system owners Duplicates blocked, links stored, gaps filled Fixing a copy only; the next extract brings the errors back
Re-test AI team with the business The agent's questions answered correctly Measuring only the quality score, not the outcome

At Calder, a cleanup of 21 customer and supplier records from S/4HANA, the CRM and the supplier portal, plus 7 products, moved the agent's test questions from 2 of 8 correct to 8 of 8. Those numbers come from a small made-up sample in this topic's hands-on part. They show the method, not a result you should expect from a real system.

Questions to ask

  1. Which questions will our agents answer from master data, and which fields does each one read?
  2. How many duplicates do we have today, and how do we know? Who decides an unclear match?
  3. Which system wins for each field: S/4HANA, the CRM, the supplier portal? Who agreed that, and is it written down?
  4. When two records merge, what happens to the open orders, invoices and payments on the duplicate?
  5. Do we keep a cross-reference from every source record to the golden record, and can our agents use it?
  6. Are duplicates stopped at creation now, or will they grow back after the cleanup?
  7. For SAP Reltio or MDG: which edition, which domains, what does it cost in our contract, and what does SAP say about using both?
  8. How will we show the cleanup helped: which agent questions, before and after?

Common misconceptions

  • "The model will figure out that two records are the same." It may, sometimes. You can't audit "sometimes". Matching belongs in a governed step with a cross-reference, not in a prompt.
  • "A high data quality score means the agent is safe." A score averages fields. One split customer with large open orders can sink an agent while the score looks fine.
  • "Merge everything that looks alike." Over-merging is worse than duplicates: two companies' orders, credit and bank details end up on one record. A lower bar finds more matches and more wrong ones.
  • "Cleanup is a one-time project." New records arrive every day. Without checks at creation, duplicates return within months.
  • "Fill the gaps with the AI." A model can suggest a missing weight or tax number. It must not invent one. Missing facts go to their owner.
  • "The golden record replaces the source systems." It usually doesn't. The sources keep running; the golden record and the cross-reference connect them, and fixes still go back to each source.

Key terms

All course terms, A to Z, in the glossary

  • Master data: Long-lived records that many transactions use, such as business partners (customers, suppliers) and products.
  • Entity resolution: Finding the records that describe the same real-world thing and linking or merging them.
  • Duplicate: Two or more records in one system for the same real thing.
  • Matching: Comparing records to find likely duplicates, usually with a score and a reason.
  • Blocking: Comparing records only within groups such as the same country, to save time and avoid nonsense matches.
  • Data steward: A business person who decides unclear matches and owns the quality of a data domain.
  • Golden record: The single trusted version of an entity built from all matching records; SAP's MDG calls it the best record.
  • Survivorship: The rules that decide which source's value wins for each field of the golden record.
  • Cross-reference: The stored link from each source record to its golden record; MDG calls it key mapping, Reltio calls the source links crosswalks.
  • Over-merging: Wrongly combining records of different real things.

Check yourself

Pick one answer for each question. The explanation appears after you choose.

1Calder has two customer records for Brennock, each with open orders. What does an agent that checks Brennock's credit headroom most likely do?

Answer: C. A lookup finds one record, so the orders on the other are missed and the headroom looks bigger. Nothing in the data tells the agent the second record exists. That is why duplicates break agents silently.

2Why is a data quality score alone a weak test of whether the cleanup made the agent safe?

Answer: D. Averages hide the one record that matters. Measure the outcome too: the questions your agents must answer, checked before and after the cleanup.

3A team proposes to lower the matching bar so the tool merges more records automatically. What is the main risk?

Answer: D. A lower bar finds more matches, including wrong ones. Over-merging is harder to undo than a duplicate, which is why unclear pairs go to a data steward.

4Which question best tells you whether duplicates will come back after the cleanup?

Answer: C. Without a check at creation, new duplicates arrive every day. A cleanup without that check is a one-time effect.

5As of 29 October 2026, what does SAP say about SAP Reltio?

Answer: C. SAP's 29 October 2026 article says SAP Reltio is "now generally available in SAP BDC" and describes cross-system entity resolution. SAP doesn't say it replaces MDG, so ask how the two fit in your landscape.

6A golden record says Orrin's payment terms are NT45, but the duplicate supplier in S/4HANA still has none. What should happen?

Answer: C. Transactions still run in S/4HANA, so a duplicate there can still get orders and invoices. Fixes go back through the people who own the data, not straight into tables.

7What is the job of the cross-reference in a cleanup?

Answer: A. The cross-reference (key mapping in MDG, crosswalks in Reltio) is what lets an invoice from the supplier portal find the right SAP supplier number. Without it, the golden record is an island.
Deep layer · 40 min read

Mental model

Master data is the agent's join key. Almost every agent question starts with "find the business partner" or "find the product", then follows links to orders, invoices, stock and prices. If the first lookup lands on the wrong record, or on one of two, every later step is correct arithmetic on the wrong rows. No prompt and no model can reason its way out of a wrong join, because nothing in the data says the join was wrong.

Three defects cause most of the damage:

  1. Split identity: one real customer, supplier or product under two numbers, or in two systems with no link. The agent sees part of the history.
  2. Missing facts: a blank payment term, tax number or weight. The agent guesses, stops, or picks another record that has the field.
  3. Mixed meaning: the same field in different units, code lists or spellings. The agent compares numbers that aren't comparable.

Entity resolution fixes the first, standardization the third, and survivorship plus owners fix the second as far as any source holds the fact. The rest of this layer turns that into code, then maps it onto SAP MDG and SAP Reltio.

How it works

Why agents are more fragile than people

A buyer who sees ORRIN MOTORWORKS INC next to Orrin Motor Works knows they are the same supplier. A tool that an agent calls does what its code says: an exact name lookup returns nothing, a tax-number lookup returns whichever record the database lists first, a "most recent" rule returns the duplicate created last week. Each tool is reasonable on its own. Together, on bad master data, they produce confident wrong answers, and an agent repeats them on every run.

There is a second effect. Agents look records up by natural language: "the Brennock order", "seal kit for the ES-200". Retrieval over names and descriptions, as in Semantic search over SAP master data, returns both duplicates with near-equal scores. The agent then has to pick, and picking is a guess.

Standardize, match, review, merge

flowchart LR
  S1[(S/4HANA<br/>CP10)] --> ST[Standardize]
  S2[(CRM)] --> ST
  S3[(Supplier portal)] --> ST
  ST --> BL[Block by kind<br/>and country]
  BL --> SC[Score pairs]
  SC -->|high, cross-system| AU[Auto-match]
  SC -->|middle, or two SAP records| RV[Steward review]
  SC -->|low| NO[No match]
  AU --> CL[Clusters]
  RV -->|merge| CL
  CL --> GR[Golden record<br/>+ lineage]
  GR --> XR[(Cross-reference)]
  GR --> PR[Proposed fixes<br/>for owners]

Standardize first, because matching compares text. Lowercase, strip accents and punctuation, drop legal forms (Inc., Ltd., GmbH), expand agreed abbreviations ("Util." to "Utility"), and convert units to one base (grams to kilograms). Keep the original values; standardized values are for comparing, not for writing back.

Block before you compare. Comparing every record with every other grows with the square of the record count and produces noise. Compare only within a block: the same kind (customer or supplier) and the same country is a common start. The cost is that a duplicate with a wrong country is never found, so check blocking keys for blanks first.

Score each pair. Strong identifiers come first: two records with the same tax number in the same country are almost certainly the same legal entity. Without one, compare standardized names (this topic uses Python's difflib ratio, from 0 to 1) and use other fields as evidence: a matching postal code raises confidence, a different one lowers it. One guard matters a lot: if the names contain different numbers ("Bearing 6310" and "Bearing 6312"), never match, however similar the rest is.

Band the scores. Above a high bar, match automatically. In a middle band, send the pair to a person. Below it, do nothing. Two extra rules from practice: a pair of records inside the same ERP goes to review even with a perfect name score, because merging them means moving open documents; and the bar is a business decision, written down, not a tuning knob.

Cluster the accepted pairs. Matches are transitive: if A matches B and B matches C, all three are one entity. A union-find structure (each record points to a group leader) builds the clusters. Transitivity is also a risk: one wrong link joins two whole clusters, which is why middle-band pairs need a person.

Survivorship: which value wins

A cluster has several values for each field. Survivorship rules pick one per field and record where it came from:

Rule Picks Good for
Source system priority The value from the most trusted source that has one Legal and commercial fields: name, tax number, payment terms (S/4HANA first)
Recency The most recently changed value Contact details kept current in the CRM or portal
Completeness The most complete record or field Filling gaps when no source is clearly better
Frequency The value most sources agree on Many weak sources, no clear owner

The surviving record's number matters as much as its values. Keep the oldest SAP number as the golden record's identifier, because open documents, contracts and reports already point at it. New numbers break every link.

Record lineage for each field: which source won. When a steward asks "why does the golden record say NT45?", the answer is a source record and a rule, not "the algorithm".

Hub, registry and fixing the source

There are three common ways to hold the result:

  • Registry: keep only the cross-reference. Source systems keep their records; consumers (agents, reports) look up the golden identifier. Cheapest, and enough for many AI uses.
  • Consolidation: build golden records in a hub for analytics and AI, and send fixes back to the sources as proposals. This is what MDG's consolidation and SAP Reltio's unified profiles do in their own ways.
  • Central governance: create and change records only through the hub, so duplicates are stopped at creation. The most control and the most process change.

Whatever the style, fixes go back to the systems that run transactions. A duplicate customer that stays unblocked in S/4HANA can still receive an order tomorrow.

Measure the outcome, not only the data

A quality score (completeness, validity, duplicate rate), as built in Is your SAP data ready for AI?, tells you how the data looks. It doesn't tell you whether the agent now answers correctly. Add an agent test set: the questions your agents must answer from master data, each with the right answer written down by the business. Run it before and after the cleanup. It catches what scores miss, including over-merging, which can make a duplicate rate look better while the answers get worse. This is the same idea as the evaluation sets in Evaluating agents, applied to data.

Build it yourself: clean Calder's partners and products, and test the agent

You will run one script that takes 21 customer and supplier records from three Calder systems (S/4HANA, the CRM and the supplier portal) and 7 products, standardizes them, finds duplicates, applies a data steward's decisions, builds golden records with lineage, and writes the fixes it proposes. Then it asks eight agent questions twice: once with a lookup tool on the raw records, once on the golden records. You will also run it without the steward and with a careless matching bar, and watch what breaks.

Before you start: complete Set up your computer for this course and Set up for Unit 7. They give you the orchestrate-course folder, its .venv and the unit07 folder. This walkthrough needs no new library: the script uses only modules built into Python.

flowchart LR
  D[21 partners, 7 products<br/>made-up, in the script] --> M[Standardize and match]
  M --> R[Steward decisions]
  R --> G[Golden records<br/>+ cross-reference]
  G --> T[Agent test:<br/>before vs after]
  G --> P[proposed_changes.csv]

What you need

  • The course folder, .venv and unit07 folder from the two setup topics.
  • About 35 minutes. No account, no key and no internet. Free.
  • Python 3.10 or newer.

The made-up data: Calder Pumpworks is a fictional company. The records are typed into the script, so there is no randomness: every run prints the same numbers. S/4HANA customers 10000001 Brennock, 10000002 Hallam, 10000021 Ostervale and 10000031 Pellmark, plus a duplicate 10000061 "Brennock Water Util." created for a rush order; suppliers 20000001 Orrin, 20000003 Ridgeline, 20000005 Larkspur, 20000006 Tallis, 20000014 Brayford and a duplicate 20000061 "ORRIN MOTORWORKS INC" without payment terms; CRM accounts A-0101 to A-0131; supplier portal registrations SP-0007 to SP-0021; products including a duplicate seal kit CP-SK200-X with its weight in grams. Postal codes, tax numbers (FX-...), e-mail addresses (.example domains), account and registration numbers and open order 1000501 are course labels.

Step 1: Open the course folder and turn on the environment

  1. Open VS Code, choose File > Open Folder and open orchestrate-course.

  2. Open a terminal: Terminal > New Terminal.

  3. Turn on the virtual environment if the prompt doesn't start with (.venv):

    • Windows (PowerShell):

      .venv\Scripts\Activate.ps1
    • macOS / Linux:

      source .venv/bin/activate
  4. Make sure the unit07 folder exists (this does no harm if it's there):

    • Windows (PowerShell):

      New-Item -ItemType Directory -Force unit07
    • macOS / Linux:

      mkdir -p unit07

Step 2: Save the script

  1. In VS Code's file list, right-click unit07, choose New File and name it master_data_cleanup.py.
  2. Paste the code below and save (Ctrl+S, or Cmd+S on Mac).
"""Master data cleanup for AI readiness: match, merge and measure (Calder Pumpworks, fictional).

Run:  python unit07/master_data_cleanup.py            # full run with the steward's decisions
      python unit07/master_data_cleanup.py --no-review # leave the review queue undecided
      python unit07/master_data_cleanup.py --auto-only 0.80  # lower the auto-merge bar (unsafe)

Built-in modules only. No randomness: every run prints the same numbers.
Nothing here writes to SAP. The script proposes changes; people approve them.
"""
import argparse
import csv
import re
import unicodedata
from difflib import SequenceMatcher
from pathlib import Path

OUT = Path(__file__).resolve().parent / "cleanup_out"

# ------------------------------------------------ business partners from three sources
# source: S4 = CP10's S/4HANA system, CRM = the sales CRM, PORTAL = the supplier portal.
# All names, numbers, postal codes, tax numbers and addresses are fictional course labels.
FIELDS = ["source", "id", "kind", "name", "country", "postal", "tax", "terms", "email", "changed"]
PARTNERS = [
    ("S4", "10000001", "C", "Brennock Water Utility", "US", "60990", "FX-1001", "NT30", "purchasing@brennock-water.example", "2026-09-14"),
    ("S4", "10000002", "C", "Hallam Fluid Systems", "US", "61350", "FX-1002", "NT45", "orders@hallam-fluid.example", "2026-10-02"),
    ("S4", "10000021", "C", "Ostervale Municipal Water", "US", "61604", "FX-1021", "NT30", "", "2026-11-03"),
    ("S4", "10000031", "C", "Pellmark Industrial Supply", "US", "60440", "FX-1031", "NT30", "", "2026-11-03"),
    ("S4", "10000061", "C", "Brennock Water Util.", "US", "60990", "", "", "", "2026-10-21"),
    ("S4", "20000001", "S", "Orrin Motor Works", "US", "61108", "FX-2001", "NT45", "", "2026-08-10"),
    ("S4", "20000003", "S", "Ridgeline Bearing Supply", "US", "60123", "FX-2003", "NT45", "", "2026-07-22"),
    ("S4", "20000005", "S", "Larkspur Freight Lines", "US", "61109", "FX-2005", "NT30", "", "2026-06-30"),
    ("S4", "20000006", "S", "Tallis Road Freight", "US", "61032", "FX-2006", "NT30", "", "2026-06-30"),
    ("S4", "20000014", "S", "Brayford Process Fluids Inc.", "US", "60431", "FX-2014", "NT30", "", "2026-10-12"),
    ("S4", "20000061", "S", "ORRIN MOTORWORKS INC", "US", "61108", "FX-2001", "", "", "2026-10-30"),
    ("CRM", "A-0101", "C", "Brennock Water Utility District", "US", "60990", "", "", "m.adler@brennock-water.example", "2026-11-10"),
    ("CRM", "A-0102", "C", "Hallam Fluid Systems, Inc.", "US", "61350", "FX-1002", "", "orders@hallam-fluid.example", "2026-09-02"),
    ("CRM", "A-0117", "C", "Ostervale Municipal Water", "US", "61604", "", "", "procurement@ostervale-water.example", "2026-11-04"),
    ("CRM", "A-0124", "C", "Hallam Fluid Systems Canada Ltd.", "CA", "N7T", "", "", "", "2026-10-15"),
    ("CRM", "A-0131", "C", "Pellmark Industrial Services", "US", "60440", "", "", "", "2026-10-28"),
    ("PORTAL", "SP-0007", "S", "Orrin Motor Works, Inc.", "US", "61108", "FX-2001", "", "ar@orrin-motor.example", "2026-11-05"),
    ("PORTAL", "SP-0012", "S", "Ridgeline Bearing Supply Europe GmbH", "DE", "86150", "FX-2008", "", "", "2026-09-30"),
    ("PORTAL", "SP-0015", "S", "Ridgeline Bearing Supply Co.", "US", "60123", "", "", "billing@ridgeline-bearing.example", "2026-10-19"),
    ("PORTAL", "SP-0019", "S", "Tallis Road Freight", "US", "61032", "FX-2006", "", "invoices@tallis-road.example", "2026-11-02"),
    ("PORTAL", "SP-0021", "S", "Larkspur Freight Line", "US", "61109", "", "", "", "2026-10-07"),
]

# ------------------------------------------------ products in CP10 (S/4HANA only)
PFIELDS = ["id", "text", "unit", "weight", "wunit", "changed"]
PRODUCTS = [
    ("CP-ES200", "End-suction pump ES-200", "EA", "180", "KG", "2026-03-02"),
    ("CP-ES300", "End-suction pump ES-300", "EA", "260", "KG", "2026-03-02"),
    ("CP-SK200", "Seal kit ES-200", "EA", "0.8", "KG", "2026-03-02"),
    ("CP-SK200-X", "SEAL KIT ES200", "EA", "800", "G", "2026-10-26"),
    ("CP-IMP200", "Impeller, ES-200", "EA", "", "", "2026-03-02"),
    ("CP-BRG6310", "Bearing 6310", "EA", "", "", "2026-03-02"),
    ("CP-BRG6312", "Bearing 6312", "EA", "", "", "2026-05-11"),
]
TO_KG = {"KG": 1.0, "G": 0.001, "LB": 0.45359237}

SOURCE_RANK = {"S4": 0, "PORTAL": 1, "CRM": 2}   # for legal and commercial fields
AUTO, REVIEW = 0.95, 0.75                          # name-score bands (course values)
LEGAL_FORMS = {"inc", "ltd", "llc", "gmbh", "ag", "co", "corp", "company"}
ABBREVIATIONS = {"util": "utility", "sys": "systems", "motorworks": "motor works"}

# The data steward's decisions on the review queue (pair of record keys -> merge or apart).
STEWARD = {
    frozenset({"S4:10000001", "S4:10000061"}): ("merge", "created for a rush order on 2026-10-21 without a tax number"),
    frozenset({"S4:10000001", "CRM:A-0101"}): ("merge", "same postal code and buyer domain; CRM adds 'District'"),
    frozenset({"S4:10000031", "CRM:A-0131"}): ("apart", "different company: Pellmark Industrial Services is a service firm"),
}

# Open documents the agents look at (course values).
OPEN_ORDERS = {"10000001": [("1000011", 67200.00)], "10000061": [("1000501", 8400.00)],
               "10000002": [("1000014", 5700.00)]}


# ------------------------------------------------ standardize
def norm_name(name):
    text = unicodedata.normalize("NFKD", name).encode("ascii", "ignore").decode().lower()
    text = re.sub(r"[^a-z0-9 ]", " ", text)
    words = []
    for w in text.split():
        w = ABBREVIATIONS.get(w, w)
        if w not in LEGAL_FORMS:
            words.append(w)
    return " ".join(" ".join(words).split())


def norm_text(text):
    text = text.lower().replace("-", "")
    text = re.sub(r"[^a-z0-9 ]", " ", text)
    return " ".join(text.split())


def numbers(text):
    return set(re.findall(r"\d+", text))


def key(rec):
    return f"{rec['source']}:{rec['id']}"


# ------------------------------------------------ match
def score_pair(a, b):
    """Return (score, reason) for two partners, or None when they must never match."""
    if a["kind"] != b["kind"] or a["country"] != b["country"]:
        return None                                   # blocking: compare within kind and country
    if a["source"] == "S4" and b["source"] == "S4" and a["tax"] and b["tax"] and a["tax"] != b["tax"]:
        return None
    if a["tax"] and a["tax"] == b["tax"]:
        return 1.0, "same tax number"
    na, nb = norm_name(a["name"]), norm_name(b["name"])
    if numbers(na) != numbers(nb):
        return None
    s = SequenceMatcher(None, na, nb).ratio()
    if a["postal"] != b["postal"]:
        s -= 0.10                                     # same name, other postal code: weaker
    return round(s, 2), "similar name" + ("" if a["postal"] != b["postal"] else ", same postal code")


def match(partners, auto_bar, use_steward):
    auto, queue = [], []
    for i, a in enumerate(partners):
        for b in partners[i + 1:]:
            r = score_pair(a, b)
            if not r:
                continue
            s, why = r
            both_sap = a["source"] == b["source"] == "S4"
            if s >= auto_bar and not (both_sap and why != "same tax number"):
                auto.append((key(a), key(b), s, why))
            elif s >= REVIEW or (both_sap and s >= auto_bar):
                decision = STEWARD.get(frozenset({key(a), key(b)})) if use_steward else None
                queue.append((key(a), key(b), s, why, decision))
    return auto, queue


def clusters(keys, pairs):
    parent = {k: k for k in keys}

    def find(k):
        while parent[k] != k:
            parent[k] = parent[parent[k]]
            k = parent[k]
        return k

    for a, b in pairs:
        parent[find(a)] = find(b)
    groups = {}
    for k in keys:
        groups.setdefault(find(k), []).append(k)
    return list(groups.values())


# ------------------------------------------------ merge (survivorship)
def survive(members):
    """Build one golden record. Returns (golden, lineage)."""
    by_rank = sorted(members, key=lambda r: (SOURCE_RANK[r["source"]], r["changed"]))
    s4 = [r for r in members if r["source"] == "S4"]
    anchor = min(s4, key=lambda r: r["id"]) if s4 else by_rank[0]   # oldest SAP number survives
    gold, lineage = {"id": anchor["id"] if s4 else "NEW:" + anchor["id"], "kind": anchor["kind"]}, {}
    for f in ["name", "country", "postal", "tax", "terms"]:       # source system rule
        order = [anchor] + [r for r in by_rank if r is not anchor]
        win = next((r for r in order if r[f]), None)
        gold[f] = win[f] if win else ""
        lineage[f] = key(win) if win else "-"
    win = max((r for r in members if r["email"]), key=lambda r: r["changed"], default=None)  # recency rule
    gold["email"] = win["email"] if win else ""
    lineage["email"] = key(win) if win else "-"
    gold["sources"] = sorted(key(r) for r in members)
    return gold, lineage


# ------------------------------------------------ products
def clean_products(products):
    for p in products:
        if p["weight"]:
            p["kg"] = round(float(p["weight"]) * TO_KG[p["wunit"]], 3)
        else:
            p["kg"] = None
    pairs = []
    for i, a in enumerate(products):
        for b in products[i + 1:]:
            ta, tb = norm_text(a["text"]), norm_text(b["text"])
            if numbers(ta) != numbers(tb):
                continue
            s = SequenceMatcher(None, ta, tb).ratio()
            if s >= AUTO:
                pairs.append((a["id"], b["id"], round(s, 2)))
    return pairs


# ------------------------------------------------ the agent test
def naive_by_name(partners, name, kind="C"):
    """What a tool without a golden record does: exact name lookup in S/4HANA."""
    return [p["id"] for p in partners if p["source"] == "S4" and p["kind"] == kind and p["name"] == name]


def naive_supplier_by_tax(partners, tax):
    hits = [p for p in partners if p["source"] == "S4" and p["kind"] == "S" and p["tax"] == tax]
    return max(hits, key=lambda p: p["changed"]) if hits else None   # the most recently changed one


def agent_tests(partners, products, golden, xref):
    exp = lambda ids: sum(v for i in ids for _, v in OPEN_ORDERS.get(i, []))
    tests = []
    # T1 open order value for Brennock
    before = exp(naive_by_name(partners, "Brennock Water Utility"))
    g = next(x for x in golden if x["id"] == "10000001")
    after = exp([s.split(":")[1] for s in g["sources"] if s.startswith("S4:")])
    tests.append(("T1", "Brennock's open order value", f"{before:,.2f}", f"{after:,.2f}", "75,600.00"))
    # T2 credit headroom for Brennock
    tests.append(("T2", "Brennock's credit headroom", f"{250000 - before:,.2f}", f"{250000 - after:,.2f}", "174,400.00"))
    # T3 payment terms for an Orrin invoice that arrived through the portal (tax FX-2001)
    nb = naive_supplier_by_tax(partners, "FX-2001")
    ga = next(x for x in golden if "PORTAL:SP-0007" in x["sources"])
    tests.append(("T3", "Payment terms for Orrin's portal invoice", nb["terms"] or "(blank)", ga["terms"] or "(blank)", "NT45"))
    # T4 which supplier number to pay Ridgeline's portal invoice to
    before = naive_by_name(partners, "Ridgeline Bearing Supply Co.", "S")  # no exact name in S/4
    ga = next(x for x in golden if "PORTAL:SP-0015" in x["sources"])
    tests.append(("T4", "Supplier number for Ridgeline's portal invoice", before[0] if before else "(not found)", ga["id"], "20000003"))
    # T5 which product is 'seal kit ES-200'
    hits = [p["id"] for p in products if norm_text(p["text"]) == "seal kit es200"]
    gp = [p["id"] for p in products if not p.get("merged_into") and norm_text(p["text"]) == "seal kit es200"]
    tests.append(("T5", "Product for 'seal kit ES-200'", " or ".join(hits), " or ".join(gp), "CP-SK200"))
    # T6 shipping weight of 60 seal kits ordered under the duplicate number
    raw = next(p for p in products if p["id"] == "CP-SK200-X")
    tests.append(("T6", "Weight of 60 seal kits (CP-SK200-X), kg", f"{60 * float(raw['weight']):,.1f}", f"{60 * raw['kg']:,.1f}", "48.0"))
    # T8 a CRM account that is a different company must stay unlinked (catches over-merging)
    before = naive_by_name(partners, "Pellmark Industrial Services")
    after = xref["CRM:A-0131"]
    tests.append(("T8", "SAP customer for CRM account A-0131", before[0] if before else "(none)",
                  "(none)" if after.startswith("NEW:") else after, "(none)"))
    # T7 Hallam's open order value (clean already: a control question)
    before = exp(naive_by_name(partners, "Hallam Fluid Systems"))
    tests.append(("T7", "Hallam's open order value", f"{before:,.2f}", f"{before:,.2f}", "5,700.00"))
    return sorted(tests)


# ------------------------------------------------ measures
def share(n, d):
    return f"{100 * n / d:.0f}%" if d else "-"


def main():
    ap = argparse.ArgumentParser(description=__doc__.splitlines()[0])
    ap.add_argument("--no-review", action="store_true", help="leave the review queue undecided")
    ap.add_argument("--auto-only", type=float, default=AUTO, help="auto-merge name score (default 0.95)")
    args = ap.parse_args()

    partners = [dict(zip(FIELDS, row)) for row in PARTNERS]
    products = [dict(zip(PFIELDS, row)) for row in PRODUCTS]
    keys = [key(p) for p in partners]
    by_key = {key(p): p for p in partners}

    # 1. match
    auto, queue = match(partners, args.auto_only, not args.no_review)
    merges = [(a, b) for a, b, *_ in auto] + [(a, b) for a, b, _, _, d in queue if d and d[0] == "merge"]
    groups = clusters(keys, merges)

    # 2. merge
    golden, lineage = [], {}
    for g in groups:
        gold, lin = survive([by_key[k] for k in g])
        golden.append(gold)
        lineage[gold["id"]] = lin
    golden.sort(key=lambda g: g["id"])
    xref = {k: g["id"] for g in golden for k in g["sources"]}

    pairs = clean_products(products)
    for a, b, _ in pairs:
        older, newer = sorted([a, b])                 # the original number survives
        next(p for p in products if p["id"] == newer)["merged_into"] = older

    # 3. measure
    s4 = [p for p in partners if p["source"] == "S4"]
    s4_pairs = [(a, b) for a, b, *_ in auto + queue if a.startswith("S4:") and b.startswith("S4:")]
    decided_apart = {frozenset({a, b}) for a, b, *_, d in queue if d and d[0] == "apart"}
    dupes_before = len({max(a, b) for a, b in s4_pairs})
    dupes_after = len({max(a, b) for a, b in s4_pairs
                       if xref[a] != xref[b] and frozenset({a, b}) not in decided_apart})
    s4_complete = sum(bool(p["tax"] and p["terms"]) for p in s4)
    golden_sap = [g for g in golden if not g["id"].startswith("NEW:")]
    g_complete = sum(bool(g["tax"] and g["terms"]) for g in golden_sap)
    unlinked = sum(1 for p in partners if p["source"] != "S4" and xref[key(p)].startswith("NEW:"))
    p_live = [p for p in products if not p.get("merged_into")]
    tests = agent_tests(partners, products, golden, xref)
    ok_before = sum(t[2] == t[4] for t in tests)
    ok_after = sum(t[3] == t[4] for t in tests)

    print("Master data cleanup, Calder Pumpworks (fictional data, as of 2026-11-13)\n")
    print(f"Matching: {len(auto)} automatic merges (score >= {args.auto_only:.2f}), "
          f"{len(queue)} pairs for a data steward (score {REVIEW:.2f} or more)")
    for a, b, s, why in auto:
        print(f"  auto    {a:<15} = {b:<15} {s:.2f}  {why}")
    for a, b, s, why, d in queue:
        if d:
            status = f"steward: {d[0]} ({d[1]})"
        elif xref[a] == xref[b]:
            status = "already linked through other matches"
        else:
            status = "waiting for a steward"
        print(f"  review  {a:<15} ? {b:<15} {s:.2f}  {why}; {status}")
    print()
    print("Golden records (field: winning source):")
    for g in golden:
        if len(g["sources"]) < 2:
            continue
        lin = lineage[g["id"]]
        print(f"  {g['id']:<12} {g['name']:<28} terms {g['terms'] or '-':<5} tax {g['tax'] or '-':<8} from {', '.join(g['sources'])}")
        print(f"               name {lin['name']}, terms {lin['terms']}, tax {lin['tax']}, email {lin['email']}")
    print()
    print(f"{'Measure':<46}{'Before':>10}{'After':>10}")
    rows = [
        ("Business partner records in S/4HANA", len(s4), len(golden_sap)),
        ("Suspected duplicate S/4HANA partners", dupes_before, dupes_after),
        ("S/4HANA partners with tax number and terms", share(s4_complete, len(s4)), share(g_complete, len(golden_sap))),
        ("CRM and portal records not linked to S/4HANA", sum(p['source'] != 'S4' for p in partners), unlinked),
        ("Duplicate products", len(pairs), 0),
        ("Products with a gross weight", share(sum(bool(p['weight']) for p in products), len(products)),
         share(sum(p['kg'] is not None for p in p_live), len(p_live))),
        ("Weights stored in kg", share(sum(p['wunit'] == 'KG' for p in products), sum(bool(p['weight']) for p in products)),
         "100%"),
        ("Agent test questions answered correctly", f"{ok_before}/{len(tests)}", f"{ok_after}/{len(tests)}"),
    ]
    for name, b, a in rows:
        print(f"{name:<46}{str(b):>10}{str(a):>10}")
    print()
    print("Agent test (a lookup tool on raw records vs. on golden records):")
    for tid, q, b, a, want in tests:
        mark = lambda v: "ok" if v == want else "WRONG"
        print(f"  {tid} {q:<45} before {b:<22} {mark(b):<6} after {a:<12} {mark(a)}")
    print()

    # 4. proposals for people, never applied here
    OUT.mkdir(exist_ok=True)
    with open(OUT / "proposed_changes.csv", "w", newline="", encoding="utf-8") as f:
        w = csv.writer(f)
        w.writerow(["object", "number", "change", "detail", "owner"])
        for g in golden:
            for k in g["sources"]:
                src, num = k.split(":")
                if src == "S4" and num != g["id"]:
                    w.writerow(["business partner", num, "block and mark as replaced",
                                f"replaced by {g['id']}; move open documents first", "Master data team"])
                if src != "S4" and not g["id"].startswith("NEW:"):
                    w.writerow([src.lower() + " record", num, "link to SAP number", g["id"], "Integration team"])
            anchor = by_key.get("S4:" + g["id"])
            if anchor:
                for f_ in ["terms", "tax"]:
                    if g[f_] and not anchor[f_]:
                        w.writerow(["business partner", g["id"], f"fill {f_}", g[f_], "Master data team"])
        for p in products:
            if p.get("merged_into"):
                w.writerow(["product", p["id"], "flag for deletion", f"duplicate of {p['merged_into']}", "Product data owner"])
            elif p["kg"] is None:
                w.writerow(["product", p["id"], "fill gross weight", "ask the owner: no source has it", "Product data owner"])
    with open(OUT / "golden_records.csv", "w", newline="", encoding="utf-8") as f:
        w = csv.writer(f)
        w.writerow(["golden_id", "kind", "name", "country", "postal", "tax", "terms", "email", "sources"])
        for g in golden:
            w.writerow([g["id"], g["kind"], g["name"], g["country"], g["postal"], g["tax"], g["terms"], g["email"], " ".join(g["sources"])])
    with open(OUT / "cross_reference.csv", "w", newline="", encoding="utf-8") as f:
        w = csv.writer(f)
        w.writerow(["source_record", "golden_id"])
        for k in keys:
            w.writerow([k, xref[k]])
    n = sum(1 for _ in open(OUT / "proposed_changes.csv", encoding="utf-8")) - 1
    waiting = sum(1 for a, b, *_, d in queue if not d and xref[a] != xref[b])
    print(f"Wrote {n} proposed changes, golden records and the cross-reference to {OUT.name}/.")
    print("Nothing was changed in SAP: each proposal needs its owner's approval."
          + (f" {waiting} review pairs still wait for a steward." if waiting else ""))


if __name__ == "__main__":
    main()

Step 3: Run the cleanup

  1. Run the script from the course folder:

    • Windows (PowerShell):

      python unit07\master_data_cleanup.py
    • macOS / Linux:

      python unit07/master_data_cleanup.py
  2. You should see this (the same on every run):

Master data cleanup, Calder Pumpworks (fictional data, as of 2026-11-13)

Matching: 8 automatic merges (score >= 0.95), 4 pairs for a data steward (score 0.75 or more)
  auto    S4:10000002     = CRM:A-0102      1.00  same tax number
  auto    S4:10000021     = CRM:A-0117      1.00  similar name, same postal code
  auto    S4:20000001     = S4:20000061     1.00  same tax number
  auto    S4:20000001     = PORTAL:SP-0007  1.00  same tax number
  auto    S4:20000003     = PORTAL:SP-0015  1.00  similar name, same postal code
  auto    S4:20000005     = PORTAL:SP-0021  0.98  similar name, same postal code
  auto    S4:20000006     = PORTAL:SP-0019  1.00  same tax number
  auto    S4:20000061     = PORTAL:SP-0007  1.00  same tax number
  review  S4:10000001     ? S4:10000061     1.00  similar name, same postal code; steward: merge (created for a rush order on 2026-10-21 without a tax number)
  review  S4:10000001     ? CRM:A-0101      0.83  similar name, same postal code; steward: merge (same postal code and buyer domain; CRM adds 'District')
  review  S4:10000031     ? CRM:A-0131      0.78  similar name, same postal code; steward: apart (different company: Pellmark Industrial Services is a service firm)
  review  S4:10000061     ? CRM:A-0101      0.83  similar name, same postal code; already linked through other matches

Golden records (field: winning source):
  10000001     Brennock Water Utility       terms NT30  tax FX-1001  from CRM:A-0101, S4:10000001, S4:10000061
               name S4:10000001, terms S4:10000001, tax S4:10000001, email CRM:A-0101
  10000002     Hallam Fluid Systems         terms NT45  tax FX-1002  from CRM:A-0102, S4:10000002
               name S4:10000002, terms S4:10000002, tax S4:10000002, email S4:10000002
  10000021     Ostervale Municipal Water    terms NT30  tax FX-1021  from CRM:A-0117, S4:10000021
               name S4:10000021, terms S4:10000021, tax S4:10000021, email CRM:A-0117
  20000001     Orrin Motor Works            terms NT45  tax FX-2001  from PORTAL:SP-0007, S4:20000001, S4:20000061
               name S4:20000001, terms S4:20000001, tax S4:20000001, email PORTAL:SP-0007
  20000003     Ridgeline Bearing Supply     terms NT45  tax FX-2003  from PORTAL:SP-0015, S4:20000003
               name S4:20000003, terms S4:20000003, tax S4:20000003, email PORTAL:SP-0015
  20000005     Larkspur Freight Lines       terms NT30  tax FX-2005  from PORTAL:SP-0021, S4:20000005
               name S4:20000005, terms S4:20000005, tax S4:20000005, email -
  20000006     Tallis Road Freight          terms NT30  tax FX-2006  from PORTAL:SP-0019, S4:20000006
               name S4:20000006, terms S4:20000006, tax S4:20000006, email PORTAL:SP-0019

Measure                                           Before     After
Business partner records in S/4HANA                   11         9
Suspected duplicate S/4HANA partners                   2         0
S/4HANA partners with tax number and terms           82%      100%
CRM and portal records not linked to S/4HANA          10         3
Duplicate products                                     1         0
Products with a gross weight                         57%       50%
Weights stored in kg                                 75%      100%
Agent test questions answered correctly              2/8       8/8

Agent test (a lookup tool on raw records vs. on golden records):
  T1 Brennock's open order value                   before 67,200.00              WRONG  after 75,600.00    ok
  T2 Brennock's credit headroom                    before 182,800.00             WRONG  after 174,400.00   ok
  T3 Payment terms for Orrin's portal invoice      before (blank)                WRONG  after NT45         ok
  T4 Supplier number for Ridgeline's portal invoice before (not found)            WRONG  after 20000003     ok
  T5 Product for 'seal kit ES-200'                 before CP-SK200 or CP-SK200-X WRONG  after CP-SK200     ok
  T6 Weight of 60 seal kits (CP-SK200-X), kg       before 48,000.0               WRONG  after 48.0         ok
  T7 Hallam's open order value                     before 5,700.00               ok     after 5,700.00     ok
  T8 SAP customer for CRM account A-0131           before (none)                 ok     after (none)       ok

Wrote 13 proposed changes, golden records and the cross-reference to cleanup_out/.
Nothing was changed in SAP: each proposal needs its owner's approval.
  1. Read the matching block from the top:

    • auto lines are pairs with a score of 0.95 or more. Five rest on the same tax number; three on a near-identical name and the same postal code.
    • 20000001 = 20000061 is two S/4HANA records matched automatically because they share a tax number. A same-tax pair inside one system is strong evidence, but the fix (blocking 20000061) still waits for the master data team in the proposals file.
    • 10000001 ? 10000061 scores 1.00 after "Util." is expanded, yet it goes to review, because both records live in S/4HANA. The steward merges it and writes why.
    • 10000031 ? A-0131 scores 0.78. The steward keeps them apart: Pellmark Industrial Services is another company.
    • The last review pair needs no decision: both records already joined Brennock's cluster through other matches.
  2. Read the golden records. Each shows its sources, then which source won each field. Brennock's e-mail comes from the CRM (the recency rule); its name, terms and tax number come from 10000001 (the source priority rule, S/4HANA first, oldest number as anchor).

Step 4: Read the before-and-after table and the agent test

  1. In the table, three changes matter most:
    • Suspected duplicate S/4HANA partners falls from 2 to 0, and the partners with a tax number and terms rise from 82% to 100%, because the two incomplete duplicates fold into complete records.
    • Products with a gross weight falls from 57% to 50%. Removing the duplicate seal kit removed one weighed record, and the impeller and both bearings still have no weight. Cleanup can't create facts no source holds; the proposals send those three to the product data owner.
    • Agent test questions rise from 2 of 8 to 8 of 8.
  2. Read the agent test line by line. Each "before" answer is what a plain lookup tool returns on the raw records:
    • T1, T2: an exact name lookup finds 10000001 only, so Brennock's open order value misses the 8,400.00 USD on 10000061 and the credit headroom is too high.
    • T3: a lookup by tax number returns the most recently changed Orrin record: the duplicate without payment terms.
    • T4: the portal's "Ridgeline Bearing Supply Co." has no exact match in S/4HANA, so the invoice can't be routed.
    • T5, T6: the description search returns two seal kits, and the duplicate's weight of 800 is read as kilograms: 48,000 kg for 60 kits instead of 48.
    • T7, T8: control questions that were already right. T8 checks that the CRM's Pellmark Industrial Services stays unlinked; it exists to catch over-merging.
  3. Open unit07/cleanup_out/proposed_changes.csv in Excel or VS Code. It has 13 rows: block and mark 10000061 and 20000061 as replaced (after moving their open documents), link seven CRM and portal records to SAP numbers, flag CP-SK200-X for deletion, and ask for three missing weights. The script changed nothing in SAP.

Step 5: Leave the review queue undecided

  1. Run:

    • Windows (PowerShell):

      python unit07\master_data_cleanup.py --no-review
    • macOS / Linux:

      python unit07/master_data_cleanup.py --no-review
  2. Compare the table with Step 4:

Measure                                           Before     After
Business partner records in S/4HANA                   11        10
Suspected duplicate S/4HANA partners                   2         1
S/4HANA partners with tax number and terms           82%       90%
CRM and portal records not linked to S/4HANA          10         4
Duplicate products                                     1         0
Products with a gross weight                         57%       50%
Weights stored in kg                                 75%      100%
Agent test questions answered correctly              2/8       6/8

Agent test (a lookup tool on raw records vs. on golden records):
  T1 Brennock's open order value                   before 67,200.00              WRONG  after 67,200.00    WRONG
  T2 Brennock's credit headroom                    before 182,800.00             WRONG  after 182,800.00   WRONG
  T3 Payment terms for Orrin's portal invoice      before (blank)                WRONG  after NT45         ok
  T4 Supplier number for Ridgeline's portal invoice before (not found)            WRONG  after 20000003     ok
  T5 Product for 'seal kit ES-200'                 before CP-SK200 or CP-SK200-X WRONG  after CP-SK200     ok
  T6 Weight of 60 seal kits (CP-SK200-X), kg       before 48,000.0               WRONG  after 48.0         ok
  T7 Hallam's open order value                     before 5,700.00               ok     after 5,700.00     ok
  T8 SAP customer for CRM account A-0131           before (none)                 ok     after (none)       ok
  1. Brennock's duplicate stays split, so T1 and T2 are wrong again, and four pairs wait for a steward. Automatic matching did most of the work, but the cases that matter most for credit were the unclear ones. A cleanup without staffed review stops short of its goal.

Step 6: Lower the bar and watch it over-merge

  1. Run with the automatic bar lowered to 0.75:

    • Windows (PowerShell):

      python unit07\master_data_cleanup.py --auto-only 0.75
    • macOS / Linux:

      python unit07/master_data_cleanup.py --auto-only 0.75
  2. Look at the table and the last test line:

Measure                                           Before     After
Business partner records in S/4HANA                   11         9
Suspected duplicate S/4HANA partners                   2         0
S/4HANA partners with tax number and terms           82%      100%
CRM and portal records not linked to S/4HANA          10         2
Duplicate products                                     1         0
Products with a gross weight                         57%       50%
Weights stored in kg                                 75%      100%
Agent test questions answered correctly              2/8       7/8
  T8 SAP customer for CRM account A-0131           before (none)                 ok     after 10000031     WRONG
  1. Pellmark Industrial Supply and Pellmark Industrial Services are now one record. The duplicate count and the "not linked" count both look better than in Step 4, yet T8 fails: an opportunity for the service firm would now be booked against Pellmark Industrial Supply's credit and terms. Only the outcome test shows it.
  2. Run the default again (Step 3) so the output files match the steward's decisions.

Step 7: Save your work in Git

From the course folder, the same on Windows and macOS/Linux:

git add unit07/master_data_cleanup.py unit07/cleanup_out
git commit -m "Unit 7: master data cleanup with agent test"

The data here is made up, so committing the outputs is fine. Never commit real customer or supplier extracts.

What each part of the script does

Part What it does
PARTNERS, PRODUCTS The made-up records from S/4HANA, the CRM and the supplier portal
norm_name, norm_text, numbers Standardize names and descriptions; extract numbers for the "different numbers never match" guard
score_pair Blocking by kind and country, tax number first, then name similarity with a postal-code adjustment
match Sorts pairs into automatic, review (middle band, or two SAP records matched on name) and no match; attaches the steward's decision
clusters Union-find: joins accepted pairs into groups, including transitive links
survive Builds the golden record: oldest SAP number as anchor, source priority for legal and commercial fields, recency for e-mail, lineage per field
clean_products Converts weights to kilograms and finds duplicate products
agent_tests Eight questions answered by a plain lookup on raw records and on golden records, against written-down answers
End of main Writes proposed_changes.csv, golden_records.csv and cross_reference.csv; changes nothing in SAP

If something goes wrong

What you see What it means What to do
python is not recognized, or command not found Python isn't on the path, or the environment is off Turn on .venv (Step 1); on macOS or Linux try python3. See the course setup topic
can't open file ... master_data_cleanup.py You're in the wrong folder, or the file name differs Run from orchestrate-course; check the file is unit07/master_data_cleanup.py
ModuleNotFoundError Not expected: the script uses only built-in modules Check you pasted the whole script, starting with the """ line
IndentationError or SyntaxError Part of the script was lost when pasting Paste the whole code block again into an empty file
PermissionError on a .csv file Excel has the file open Close it in Excel, then run again
Numbers differ from this page The script was edited, or you passed other options Compare with the code above; run without options
Network or proxy errors Not expected: nothing here uses the internet or needs a key Check you're running this script, not one that calls the SAP sandbox

The SAP way

MDG consolidation on SAP S/4HANA

As of November 2026, SAP's MDG course describes two kinds of consolidation (SAP Learning):

  • Consolidation of source records loads records from other systems and runs them through a process template: data load (CSV or XLSX files, a SOAP service for business partners, or ETL tools), initial check, standardize, match, match review, calculate best record, validate and activate.
  • Consolidation of active records finds duplicates among records already active in MDG, with the steps selection, match, review, best record, review, validate and activate. SAP names two strategies: Improve best record, which links each duplicate to the best record with an "is replaced by" relationship and can set a flag such as a central block on it through a customer enhancement, and Remove duplicates, which sets an archiving or deletion flag and moves the duplicate's key mappings to the best record. Scheduled runs are available for business partners and products.

The parts map onto this topic's script:

Script MDG consolidation
norm_name, unit conversion Standardize step: BRFplus, or for business partners SAP HANA smart data quality or SAP Data Services
score_pair and the bands Match step: SAP HANA fuzzy search, smart data quality or Data Services for business partners, simple database matching for materials; match groups above an upper threshold are approved automatically
STEWARD Match review: approve or reject match groups, split records
survive Calculate best record: table-level rules by source system reliability, recency or BRFplus, and field-level completeness
cross_reference.csv Key mapping between the hub and client systems
proposed_changes.csv Activate with a change request for records that need correction, or the active-record strategies above

SAP's cloud-ready mode course describes the same flow for business partners and their relationships: the Import Master Data app, an initial check with tabular correction, optional address validation (which needs a subscription to SAP Data Quality Management, microservices for location data), matching on name, address, identification numbers and tax numbers with a review screen, best record calculation into a golden record, then validation and activation, with changes visible in the Track Changes app (SAP Learning).

To stop new duplicates, central governance routes creation and change through governed requests. SAP's product page for MDG on Cloud Private Edition also describes Joule in MDG: searching, displaying, submitting or changing business partner records in natural language and checking the status of governance processes (SAP product page). The data quality rules and evaluations that measure the result are covered in Is your SAP data ready for AI?.

SAP Reltio in SAP Business Data Cloud

SAP's 29 October 2026 article states that, with the acquisition complete, SAP Reltio is "now generally available in SAP BDC" (SAP News). It describes AI-based entity resolution that matches and reconciles records across SAP and third-party systems into unified views of customers, suppliers and products, and trusted profiles published as governed data products that give Joule and Joule agents connected business context. The article doesn't mention MDG; SAP's May 2026 article names MDG a core component of SAP Business Data Cloud, governing master data and policies (SAP News, 13 May 2026).

Reltio's own documentation describes the mechanics in terms close to this topic's:

  • Matching: match rules per entity type, built from comparison operators such as Exact, ExactOrNull and Fuzzy, with match tokens to find candidates quickly. A pair can lead to no action, a merge, or a queue for data steward review (Reltio docs). Machine-learning matching is also offered.
  • Merge and the golden record: a merge aggregates the contributing profiles' crosswalks (links to source records), attributes, relationships and interactions. The golden record is a view built on the fly from the operational values that survivorship selects, and cleanse, match and merge are designed to run in real time (Reltio docs). A potential match can also be recorded as a relationship instead of a merge.
  • Survivorship: rule types include Recency, Source System, Frequency, Aggregation, OldestValue, MinValue and MaxValue (Reltio docs). The script's survive uses the first two.

Reltio's documentation describes the Reltio platform; check which features SAP Reltio in BDC includes in your edition.

Sending fixes back through released APIs

Fixes reach S/4HANA through governed processes or released APIs, not table updates. The SAP Cloud SDK's generated models (version 2.1.0, inspected in this run) show the fields involved: the business partner service API_BUSINESS_PARTNER has BusinessPartnerIsBlocked and IsMarkedForArchiving on A_BusinessPartner, and the product service API_PRODUCT_SRV has GrossWeight, WeightUnit and IsMarkedForDeletion on A_Product. Check them on the SAP Business Accelerator Hub for your release. Before blocking a duplicate, move or finish its open orders, invoices and payments; that is a business decision per document, not an API call.

Licensing notes

MDG, SAP Data Quality Management and SAP Business Data Cloud are licensed separately, and the pages opened for this topic give no prices. SAP's pages don't say whether SAP Reltio is part of a BDC subscription or priced on its own. Ask your SAP account team, and get the answer in writing before you plan around it.

Build vs. SAP

Situation Use Why
One agent pilot, a few thousand records, days not months A script like this one, plus a steward Sizes the problem and proves the agent gain before any license
Duplicates inside S/4HANA, MDG already licensed MDG consolidation of active records Best record, key mapping and blocking stay inside SAP's governance
Records from several SAP and non-SAP systems, agents in Joule or BDC SAP Reltio in BDC, or another MDM tool Cross-system entity resolution at scale, published as data products
Stop duplicates at creation MDG central governance, or a duplicate check in the creation process A cleanup without prevention decays
ECC records before an S/4HANA move Cleanup as a migration workstream Duplicates block the business partner conversion anyway
Agents only need to find the right record A registry: the cross-reference and a lookup tool Cheapest; the golden values can wait

Architecture view

Where it sits. At Calder (fictional), customers live in S/4HANA (CP10) and the CRM, suppliers in S/4HANA and the supplier portal, and CP20 keeps its own in ECC until the July 2027 rollout. A matching job reads all three, writes the cross-reference and golden records to a store the agents read, and sends proposals to the master data team.

flowchart LR
  S4[S/4HANA CP10] -->|released APIs| MJ[Matching job]
  CRM[CRM] --> MJ
  SP[Supplier portal] --> MJ
  MJ --> XR[(Cross-reference<br/>+ golden records)]
  MJ --> Q[Steward queue]
  Q --> MJ
  XR --> AG[Agent lookup tools]
  MJ --> PR[Proposals] --> MDT[Master data team] -->|governed change| S4

Non-functional requirements that decide the design

Requirement Typical target If missed
Match precision Wrong merges near zero on a labeled sample; unclear pairs reviewed Over-merging mixes companies' credit, terms and bank data
Freshness New records matched within a day, before agents use them A new duplicate splits a customer again for days
Review backlog Queue worked within an agreed time, owners named Unclear cases, often the costly ones, stay split
Lineage Every golden value traceable to a source record and a rule Stewards and auditors can't explain an agent's answer
Security Read-only extraction; personal data only where the purpose needs it The golden store becomes a copy of everyone's contact data

Integration and identity. The matching job reads with a technical user limited to the partner and product fields it needs, through released APIs or data products, as in CDS extraction pipelines for AI. Agents read the golden store with the end user's authorizations applied, as in Grounding on SAP data with authorizations; a cross-reference must not let a user see a customer their roles hide. Nothing in this design writes to S/4HANA: changes go through the master data team's governed process.

The decision to record (in the ADR form from Writing a solution design doc):

  • Context: Calder's agents look up partners and products across S/4HANA, the CRM and the supplier portal; duplicates already cause wrong credit and payment answers. CP20 joins CP10's system in July 2027, and the master data decision for that move is to screen by script now and re-evaluate MDG after the migration.
  • Options: (a) MDG consolidation now; (b) SAP Reltio in BDC now; (c) a scheduled matching job with steward review and a cross-reference that agents use, with fixes sent to the master data team; (d) let each agent tool handle name variants itself.
  • Decision: (c) until the rollout, with the agent test set as the acceptance check; evaluate (a) and (b) together after July 2027, when CP20's records are in the same system.
  • Consequences: Matching rules live in code and need an owner. Duplicates are found, not prevented. Steward time must be staffed. The agent test set becomes part of every release.

What a design review will ask

  1. Which agent questions depend on master data, and where is the test set with its expected answers?
  2. Who decides unclear matches, how fast, and how are decisions stored and reused?
  3. Which source wins for each field, and who signed that off?
  4. How are open documents on a duplicate handled before it is blocked?
  5. Can any component write to SAP directly? If yes, why?

Production concerns

  • Security and SAP authorizations. Bank details, payment terms and tax numbers are sensitive fields: never let a survivorship rule overwrite them without approval, and keep changing them separate from paying, as covered in Roles and authorization design. A recency rule that copies a new bank account from a supplier portal is a fraud path.
  • Evaluation. Keep two test sets: labeled pairs (same or different, from steward decisions) to measure match precision and recall, and the agent questions with written answers. Run both on every rule change.
  • Personal data. Contact names and e-mails in the CRM are personal data. Keep only what the agent's purpose needs, and check the golden store's retention with the data protection officer.
  • Cost. Matching compute is cheap; steward hours, moving open documents and filling missing facts are not. Estimate per review pair and per open document.
  • Operations. Run matching on a schedule and on creation events, track the review backlog and the agent test score as KPIs, and alert when a new duplicate appears for a customer with open orders.
  • Clean core. Read through released APIs, CDS views or data products; write through governed processes. No custom table updates.
  • Undo. Keep enough lineage to split a wrong merge. MDG's match review can split records, and Reltio supports potential matches as relationships, but a merge already pushed to S/4HANA is hard to reverse.

Pitfalls

  • Matching before standardizing. "ORRIN MOTORWORKS INC" and "Orrin Motor Works" look different until case, legal forms and spacing are handled.
  • Blocking on a blank field. A record with no country never meets its twin. Profile the blocking keys first.
  • Trusting transitivity. One wrong link joins two clusters. Review middle-band pairs, and look at large clusters.
  • Letting a weak source win. A CRM name or a portal address overwriting S/4HANA's without a rule and an owner.
  • Renumbering the survivor. A new golden number breaks every open document, contract and report that points at the old one.
  • Celebrating the duplicate rate. Over-merging improves it. Test the outcome.
  • Fixing the copy only. If the duplicate stays unblocked in S/4HANA, the next order lands on it.
  • Asking the model to fill gaps. A suggested weight or tax number is a proposal for an owner, never a value.

Exercise

Add a new defect, extend the agent test, and prove the fix.

  1. Copy unit07/master_data_cleanup.py to unit07/my_cleanup.py.
  2. In PARTNERS, add a supplier portal record: ("PORTAL", "SP-0024", "S", "Brayford Process Fluids", "US", "60431", "", "", "ap@brayford-fluids.example", "2026-11-09").
  3. In agent_tests, add a test T9 "Supplier number for Brayford's portal invoice" that looks up SP-0024 by exact name in S/4HANA for "before" and uses xref["PORTAL:SP-0024"] for "after", with the expected answer 20000014. Copy the pattern of T4.
  4. Run python unit07/my_cleanup.py (Windows: python unit07\my_cleanup.py). Check where SP-0024 lands: auto, review or no match, and why.
  5. Add one CRM record of your own that must not match anything, plus a test that checks it stays unlinked, like T8.
  6. Run with --auto-only 0.75 and note which of your tests fail.
  7. Write three lines at the top of the file as a comment: the automatic bar you recommend, the review bar, and the evidence from steps 4 to 6.

Done when my_cleanup.py runs without errors, prints 10 agent tests, T9 is wrong before and right after the cleanup, your keep-apart test passes at the default bar, and your comment names a bar backed by the runs. You will reuse the test set when you evaluate agents end to end in Evaluating agents and when you scope a data cleanup in Scoping and estimating AI engagements.

Check yourself

Pick one answer for each question. The explanation appears after you choose.

1Why can't a well-written prompt make an agent safe on duplicate master data?

Answer: A. The failure is a wrong join: the tool returns a valid record and the arithmetic on it is correct. Nothing signals that a second record exists, so the fix belongs in the data and the cross-reference, not in the prompt.

2In the script, 10000001 and 10000061 score 1.00 on name, yet the pair goes to review. Why?

Answer: A. Two records inside the ERP carry orders and invoices, so a person decides even when the name is a perfect match. Cross-system pairs at the same score are linked automatically, because linking a CRM account changes nothing in SAP.

3After the cleanup, "Products with a gross weight" falls from 57% to 50%. What does that tell you?

Answer: A. The duplicate had a weight, and the impeller and bearings never did. Cleanup can't create facts no source holds, so the proposals ask the product data owner. It is also a reminder that scores can move the wrong way while the outcome improves.

4With --auto-only 0.75, the duplicate count and unlinked count look better than at 0.95. What does test T8 reveal?

Answer: D. The lower bar merged Pellmark Industrial Services into Pellmark Industrial Supply. Over-merging improves both counts, which is why an outcome test with a keep-apart case belongs in every run.

5Which survivorship rule should decide a supplier's bank details in the golden record?

Answer: A. Copying a new bank account from a portal by recency is a classic fraud path. Sensitive fields need a governed change with separate duties, whatever the matching tool proposes.

6In MDG consolidation of active records, what does the Improve best record strategy do with a duplicate?

Answer: C. SAP's MDG course describes an "is replaced by" relationship plus an optional flag such as a central block. The Remove duplicates strategy is the one that sets an archiving or deletion flag and moves key mappings.

7What does SAP's 29 October 2026 article say SAP Reltio gives Joule and Joule agents?

Answer: D. The article describes trusted profiles published as governed data products that give Joule and Joule agents connected business context. It doesn't say Reltio replaces MDG or writes to S/4HANA, and pricing isn't stated.

8A new supplier record arrives in the portal and scores 0.83 against an S/4HANA supplier with the same postal code. What should the matching job do?

Answer: B. 0.83 is in the middle band, where both duplicates and different companies occur, as the Brennock and Pellmark cases show. A person decides, and the decision is stored so the next run reuses it.

Sources

Sign in to track your progress

We'll email you a one-time sign-in link. No password needed.

or

Tell us a little about you

Optional, every field. It helps us pitch answers to your questions at the right level and decide which topics to write next. It is never shown publicly, and you can change or clear it anytime from the account menu.

SAP areas you work in