Orchestrate

Report an issue

Something wrong, out of date or unclear? Tell us. Corrections are prioritized over new topics. You don't need to sign in.

0 / 2000

Don't include personal or confidential company details. Reports are reviewed with AI assistance.

AI-accelerated delivery on SAP programs

Where LLMs speed up an SAP program, from fit-gap to hypercare, where a person must review, and how to measure the time saved honestly.

Updated Nov 3, 2026Foundational 8 minDeep 38 min
Foundational layer · 8 min read

The 60-second version

An SAP program produces a mountain of documents: workshop notes, requirement lists, functional specs, interface mappings, test cases, migration mappings, cutover runbooks and, after go-live, thousands of error messages. Most of that work is reading and drafting. Large language models (LLMs) draft quickly.

So AI can make delivery faster, but only in one way: a person writes less and checks more. The saving is real only when checking a draft is faster than writing it, and when the check catches what the AI got wrong.

That gives leaders three rules. Use AI where the draft can be checked against something: the workshop notes, a template, an answer key. Make a named person sign everything that reaches the system or the business. And measure the saving as hours before and after on comparable work, including the review time, not as a vendor's percentage.

Why it matters to the business

Take the course's running example. The fictional Calder Pumpworks is moving its German company CP20 from ECC into CP10's S/4HANA system, with go-live planned for 5 July 2027 (Data migration strategy). Before then, the program must:

  • run fit-to-standard workshops and turn their notes into a requirement backlog (Fit-to-standard and extension choices);
  • write functional specs for every gap that becomes configuration or an extension;
  • specify and test interfaces such as I10, distributor purchase orders by EDI (The integration lead's toolkit);
  • map and validate migration data through several mock loads;
  • write a cutover runbook for the weekend of 2 to 4 July 2027;
  • run hypercare, the weeks after go-live when the project team fixes problems fast.

All of these are consultant days, and consultant days drive the budget. For example, if a spec takes six hours by hand and 90 minutes to review and fix as an AI draft, each spec frees most of a day. Across a program with dozens of specs, that adds up to weeks, but only if the review really takes 90 minutes.

The risk is just as concrete:

  • Confident gaps. An AI-drafted spec can look complete and still miss the one error case the business cares about, such as a duplicate order.
  • Invented rules. A draft can state a check nobody asked for, or map a field the workshop ruled out.
  • Hidden review cost. If reviewers rewrite most of each draft, the saving disappears, and nobody notices unless review time is measured.
  • Data leaving the project. Workshop notes and error logs hold customer names, prices and sometimes personal data.

How SAP does it

As of November 2026, SAP offers AI for several delivery jobs. Check each one for your contract, release and region.

  • Requirements from workshops. SAP lists a requirement generation AI feature for SAP Cloud ALM, SAP's application lifecycle management tool. It drafts business requirements from Fit-to-Standard workshop transcripts into a requirements template and suggests solution proposals from SAP Activate and SAP Best Practices content. Consultants "review, refine, and validate" the result before saving (sap.com). The page shows both a flat-fee metric and consumption-based pricing, so ask for the terms in writing.
  • Long documents. Cloud ALM's AI-assisted document summary was generally available as of SAP's Q2 2026 release highlights. Users edit the summary before keeping it (SAP News, July 2026).
  • Answers for consultants. Joule for Consultants answers configuration, design and code questions grounded in SAP Notes and knowledge base articles. SAP's product page says it needs AI Units (sap.com). SAP's July 2026 highlights add grounding on a company's own methods, policies and templates.
  • Interface mappings. SAP Integration Suite's Integration Advisor offers "AI-powered mapping proposals" for message mappings (sap.com).
  • Migration. SAP's AI for custom code, data, testing and configuration in migrations is covered in AI in S/4HANA migration projects.
  • Hypercare monitoring. Cloud ALM's Job & Automation Monitoring rates job runs and their exceptions (SAP Support). Its Integration & Exception Monitoring collects IDoc and web service errors and can raise alerts (SAP Support). The pages we read describe no AI triage, so grouping and explaining errors with AI is today something a team builds itself.

Where AI helps, job by job

Job What AI drafts Safe when A person must review How to measure the saving
Fit-gap from workshop notes Requirement list, first fit or gap call per item Every item quotes the notes it came from Every fit or gap call; the product owner accepts the backlog Hours from workshop to signed backlog, per workshop
Functional specs First draft in the program's template The template fixes the sections; gaps are marked open Every rule and every number; the business owner signs Hours per spec, drafting plus review, against past specs of the same size
Interface mappings and test cases Mapping rows, error cases, test cases Scored against a hand-written answer key or a reviewed spec Each mapping row before build; test data and expected results Review and fix minutes against a hand-written baseline; defects found later in testing
Data migration mapping and validation Value mappings, duplicate candidates, validation rules Every proposal is checked by a mock load and reconciliation Every merge, exclusion and mapping; the data owner signs Defects per mock load; hours per object
Hypercare error triage Groups of similar errors, likely cause, suggested owner Deterministic grouping first; data masked; read-only Every fix and every reprocessing; AI never reprocesses Time from error to owner; backlog per day
Cutover runbooks Task list from templates, dependencies, timings Built from the program's own plan and rehearsals Every task, owner and timing; the go/no-go call is never drafted Rehearsal findings; tasks changed after the dress rehearsal

Questions to ask

Ask your partner, your program lead or SAP:

  1. Which delivery jobs use AI on this program, and which tool for each? Get the status (generally available, early adopter, planned) and the terms in writing.
  2. What does each draft get checked against? The notes, a template, an answer key, a reviewed earlier spec?
  3. Who signs each AI-drafted artifact, and where is that recorded?
  4. How is the saving measured? Hours before and after on comparable work, with review time included. Ask to see the log.
  5. What data goes to the model? Workshop notes and error logs can hold names, prices and personal data. Which provider, which region, which terms?
  6. What is AI never allowed to do? Reprocess messages, change configuration, release transports, make the go/no-go call.
  7. What happens to quality? Count defects found in testing and hypercare in AI-drafted work against hand-written work.

Common misconceptions

  • "The draft looks complete, so it is." Fluent drafts hide gaps. Only a check against the notes or an answer key shows what is missing.
  • "AI saves 70% of the effort." A vendor or demo number is not your number. Measure drafting plus review plus fixing on your own work.
  • "Grounded means correct." A draft can quote the notes and still be wrong, for example by mapping a field the notes say is out of scope.
  • "Hypercare can be automated with an agent." AI can group errors and suggest owners. People fix master data and reprocess messages.
  • "We'll measure the saving at the end." Without a baseline from before, there is nothing to compare against.

Key terms

All course terms, A to Z, in the glossary

  • Fit-to-standard workshop: an SAP Activate workshop where the business sees the standard process and records the gaps (delta requirements).
  • Functional spec: the document that says what a configuration change, extension or interface must do, in business terms.
  • Answer key: a hand-written, reviewed version of an artifact used to score AI drafts of the same kind.
  • Grounding: tying each item in a draft to a quote from its source, so a reviewer can check it.
  • Coverage: the share of the answer key's items a draft contains.
  • Hypercare: the support period right after go-live, when the project team resolves problems quickly.
  • Cutover runbook: the hour-by-hour task list for the switch from the old system to the new one.
  • Baseline: the time the work took before AI, on comparable items, measured the same way.

Check yourself

Pick one answer for each question. The explanation appears after you choose.

1When does AI actually speed up delivery work such as writing a spec?

Answer: B. AI moves effort from writing to checking. If review and fixes take as long as writing, the saving disappears, which is why review time must be measured.

2A partner claims their AI cuts spec effort by 70%. What should you ask for first?

Answer: D. Only a baseline from comparable work, with drafting, review and fixes counted, shows your saving. A demo or another client's number says nothing about your program.

3Which delivery job should AI never do on its own, even in hypercare?

Answer: B. Reprocessing changes business data, so the error owner decides and acts. AI can group, summarize and suggest, which are read-only.

4An AI-drafted spec quotes the workshop notes for every row. What can still be wrong?

Answer: C. Grounding shows where an item came from, not that it is right. A quote about Incoterms being out of scope can still be used to map Incoterms.

5SAP Cloud ALM's requirement generation drafts requirements from workshop transcripts. Who decides what is saved?

Answer: A. SAP's page says consultants review, refine and validate the generated requirements before saving them. The product owner then accepts the backlog as usual.

6Why does the program need a baseline before it starts using AI for specs?

Answer: B. The saving is the difference between hours before and after on comparable work. Without the before number there is nothing to compare, only a feeling.
Deep layer · 38 min read

Mental model

Treat every AI-assisted delivery job as draft, check, sign.

  • Draft: a model turns a source (workshop notes, a template, an error log) into a structured artifact.
  • Check: automatic checks run before a person looks: is every item grounded in the source, does the draft cover what an answer key expects, does it break any rule the source states?
  • Sign: a named person reviews, fixes and signs. The log records their minutes.

The time saved is then simple arithmetic: baseline minutes minus review and fix minutes, on comparable artifacts. Two properties decide whether a job is a good fit:

  1. Checkable. Is there something to check the draft against? Notes, a template, an answer key, a mock load, a rehearsal. If not, the reviewer checks against their memory, which is slow and unreliable.
  2. Contained. If a wrong item slips through, how far does it get? A wrong requirement is caught in the design workshop. A wrong mapping row is caught in string testing. A wrong reprocessing posts a sales order.

Checkable and contained: let AI draft freely. Hard to check or far-reaching: AI drafts smaller pieces, or only summarizes, and a person decides each item.

How it works

flowchart LR
  S[Source<br/>notes, template, log] --> D[Model drafts<br/>structured JSON]
  D --> G[Grounding check<br/>quote in source?]
  G --> C[Coverage check<br/>against answer key]
  C --> X[Rule check<br/>what the source rules out]
  X --> R{Reviewer<br/>fixes and signs}
  R --> A[Signed artifact]
  R --> L[Log: minutes,<br/>coverage, flags]

The automatic checks do not replace the review. They point the reviewer at the items most likely to be wrong, so review time goes where it matters.

Structured drafts, not prose

Ask the model for JSON in a fixed shape (rows, error cases, test cases), as Structured outputs and function calling shows. A structure can be counted and compared; prose can't. It also maps straight into the program's template.

Grounding with quotes

Anthropic's guidance for reducing hallucinations includes letting the model say it doesn't know, using direct quotes for grounding, and verifying with citations (Claude Docs). For delivery work this becomes a rule: every item carries an evidence quote copied word for word from the source, and anything the source leaves open goes into a list of open questions. A script then checks that each quote really appears in the source. An item whose quote isn't there was probably invented.

Grounding is necessary but not enough. A model can quote "Someone asked about Incoterms" and still map Incoterms, although the next words say it is out of scope. That is why a rule check follows.

Answer keys

An answer key is a hand-written, reviewed version of the same kind of artifact. For interfaces, Calder already has one: the I10 spec written in Unit 5B. Score each draft against it: which mapping rows, error cases and test cases it covers, fully or partly. Keep one key per artifact type and grow it as reviewers find new misses. This is the evaluation discipline of LLM evaluation fundamentals, applied to delivery documents.

The six jobs in more detail

Fit-gap from workshop notes. SAP Activate's Explore phase runs fit-to-standard workshops that collect delta requirements and create the initial backlog, then drives sign-off (SAP Learning). A model can turn notes or a transcript into a requirement list with a first fit or gap call per item. Review each call: the fit-gap decision is where custom code gets approved, and Fit-to-standard and extension choices shows what each wrong "gap" costs.

Functional specs. Give the model the program's template, the requirement and its workshop quotes. It fills sections and marks the rest open. Numbers (tolerances, limits, volumes) and rules are where drafts go wrong; review each one against its source.

Interface mappings and test cases. The build below does exactly this. Integration Advisor's mapping proposals are SAP's tool for the message-mapping part. Test cases come from the mapping and the error table: each row needs a valid and an invalid case, each error a test.

Data migration mapping and validation. AI proposes value mappings and duplicate candidates; the mock load and the reconciliation decide. Use the method in Data migration strategy and SAP's migration tooling in AI in S/4HANA migration projects; this topic doesn't repeat them.

Hypercare error triage. Group errors with plain code first (same interface, same message text, same key), then let a model explain each group and suggest an owner, with numbers masked before any text leaves. The integration lead's toolkit builds that grouping. The model never reprocesses anything.

Cutover runbooks. A model can draft tasks, owners and dependencies from the program's plan and templates. SAP Activate plans cutover in Realize and runs it on the last weekend before go-live. Rehearsals check the runbook, not the reviewer's reading. The go/no-go decision is a meeting of people with the facts, never a drafted output.

Measuring the time saved

  1. Baseline first. Before using AI, record hours per artifact on a handful of comparable items (specs of similar size, interfaces of similar complexity).
  2. Log every AI-assisted item: drafting tool, coverage score, number of flagged items, review minutes, fix minutes, reviewer.
  3. Compare like with like. A simple interface against a simple interface.
  4. Track quality too: defects found in testing or hypercare on AI-drafted against hand-written artifacts. A saving that comes back as defects is not a saving.

SAP's July 2026 highlights quote one customer seeing unit testing 15 to 20% faster with Joule generating test cases (SAP News). Treat such numbers as one customer's report, not a forecast for your program.

Build it yourself: draft and score the I10 pack

Before you start: complete Set up your computer for this course and Set up for Unit 14. This walkthrough doesn't repeat those steps.

You will take the notes from a workshop on Calder's interface I10 (distributor purchase orders by EDI), let an AI draft the spec's mapping, error cases and test cases, and score the draft against a hand-written answer key based on the I10 spec from The integration lead's toolkit. Then you log your review time and see the time saved. The point is to see what the checks catch, and what only a reviewer can.

flowchart LR
  N[notes_I10.md] --> D[draft<br/>sample, model or file]
  D --> J[draft_I10.json]
  K[key_I10.json<br/>answer key] --> S[score]
  J --> S
  N --> S
  S --> L[log<br/>your minutes]
  L --> R[report<br/>time saved]

What you need

  • The course folder orchestrate-course with its .venv, from the setup topics.
  • For the free path (--sample): nothing else. A recorded draft stands in for the model.
  • For the model path (--llm, optional): the ANTHROPIC_API_KEY in your .env from Unit 1, and the anthropic and python-dotenv libraries already in your requirements.txt. One call per draft, a small per-request charge.
  • Time: about 40 minutes. Cost: free on the sample path.

Unit 14 adds no lines to requirements.txt. All notes, keys, numbers and minutes are made up for Calder, a fictional company. Nothing connects to an SAP system.

Step 1: Open your course folder and turn on the virtual environment

  1. Open VS Code, choose File > Open Folder, and open orchestrate-course.

  2. Open a terminal: Terminal > New Terminal.

  3. Turn on the virtual environment:

    • Windows (PowerShell):

      .venv\Scripts\Activate.ps1
    • macOS / Linux:

      source .venv/bin/activate
  4. You should see (.venv) at the start of the prompt. The unit14 folder exists from Set up for Unit 14. If it doesn't, create it with mkdir unit14 (the same command works in both terminals).

Run every command from the course folder, not from inside unit14. On macOS or Linux, if python isn't found before .venv is active, use python3.

Step 2: Create the script

  1. In VS Code's Explorer, right-click unit14 and choose New File. Name it delivery_pack.py.
  2. Paste in the code below and save.
"""Draft a spec and test-case pack for Calder's interface I10 with AI, then score it.

Calder Pumpworks is fictional. Notes, answer key, rules and minutes are course values.

Commands (run from your course folder):
  python unit14/delivery_pack.py init               write the workshop notes and the answer key
  python unit14/delivery_pack.py draft --sample     use a recorded draft (free, no account)
  python unit14/delivery_pack.py draft --llm        a real model drafts the pack (small charge)
  python unit14/delivery_pack.py draft --file my_draft.json   score your own draft
  python unit14/delivery_pack.py score              compare the draft with the answer key
  python unit14/delivery_pack.py log --reviewer "Your Name" --review-minutes 50 --fix-minutes 40
  python unit14/delivery_pack.py report             time saved per pack, from the log

Nothing here connects to an SAP system. The draft is never accepted by this script:
a person reviews it, fixes it and signs the spec, and the log records that work.
"""
import argparse
import json
import os
import re
import sys
from datetime import datetime, timezone
from pathlib import Path

HERE = Path(__file__).resolve().parent
WORK = HERE / "delivery"
NOTES = WORK / "notes_I10.md"
KEY = WORK / "key_I10.json"
DRAFT = WORK / "draft_I10.json"
SCORE = WORK / "score_I10.json"
LOG = WORK / "delivery_log.jsonl"

# Course baseline: what writing this pack by hand took on comparable interfaces.
BASELINE_MINUTES = 360

NOTES_TEXT = """\
# Workshop notes: I10 distributor purchase orders (Calder Pumpworks, fictional)

Workshop on 2026-10-27. Present: customer service lead, integration lead, Hallam's EDI contact.
Course notes, typed during the meeting. Not reviewed.

- Distributors send POs by EDI to Calder's EDI address. One message = one PO, one or more lines.
- Hallam Fluid Systems is the first distributor. The trading partner profile says Hallam = 10000002.
  If we don't know the sender, reject it and tell the integration team. Never trust a customer number in the message.
- Ship-to: distributors send a location code. Sales master data keeps the cross-reference.
  Blank ship-to means same as sold-to.
- Their PO number goes into the customer reference. Customer service wants a check:
  same PO number for the same sold-to twice means a duplicate, no second order. Customer service owns duplicates.
- PO date goes into the customer reference date.
- Line numbers: line 1 becomes item 10, line 2 item 20, so line x 10.
- Hallam sends its own part numbers. HFS-SK-200 is our CP-SK200 (seal kit).
  Unknown part number -> message stops in S/4HANA, customer service adds the cross-reference and reprocesses.
- Units: they send EA or PCE, we use PC. Unknown unit -> integration team extends the code list.
- Requested date goes into the requested delivery date. A date in the past is kept and flagged for customer service.
- Price: they send a unit price but S/4HANA determines the price. Do not map it. A price dispute is a sales issue.
- Sales area is always sales org CPUS, channel 20, division PU (from the partner agreement).
- Someone asked about Incoterms. Not in scope for now; maybe later with the order change interface.
- Order changes and cancellations are a separate interface. The confirmation back to the distributor stays on I04.
- If S/4HANA is down: automatic retry, 4 tries, backoff 0.5 s doubling to 4 s, then alert the integration team.
- Credit block on a created order is NOT an interface error. Credit management works it in the normal worklist.
- Volumes: about 120 messages a day, peak 30 an hour. The business notices after 1 hour.
- Test data: HFS-PO-7781 (60 CP-SK200) is the happy path. HFS-PO-7790 for the S/4HANA-down case.
  XYZ-PO-0001 is an unknown sender. Volume test: 60 messages in one hour, twice the peak.
- Technical user posts with a role limited to creating sales orders in sales org CPUS.
"""

# The answer key, written by hand from the I10 spec in Unit 5B. Each item lists the words
# that must appear for a draft item to count (any one of "find", all of "need").
KEY_DATA = {
    "interface": "I10",
    "mapping": [
        {"id": "M01", "target": "Sold-to party", "find": ["sold-to", "sold to"], "need": ["10000002"]},
        {"id": "M02", "target": "Ship-to party", "find": ["ship-to", "ship to"], "need": ["sold-to"]},
        {"id": "M03", "target": "Customer reference", "find": ["customer reference", "customer po"], "need": []},
        {"id": "M04", "target": "Customer reference date", "find": ["reference date", "po date"], "need": []},
        {"id": "M05", "target": "Item number", "find": ["item number", "item"], "need": ["10"]},
        {"id": "M06", "target": "Product", "find": ["product", "material"], "need": ["CP-SK200"]},
        {"id": "M07", "target": "Order quantity and unit", "find": ["unit", "quantity"], "need": ["EA", "PCE", "PC"]},
        {"id": "M08", "target": "Requested delivery date", "find": ["requested delivery date", "delivery date"], "need": []},
        {"id": "M09", "target": "Sales area", "find": ["sales area", "sales org"], "need": ["CPUS", "20", "PU"]},
    ],
    "errors": [
        {"id": "E01", "what": "Unknown sender", "find": ["unknown sender", "sender not", "partner not"], "owner": "integration"},
        {"id": "E02", "what": "Unknown part number", "find": ["part number", "unknown part", "product not"], "owner": "customer service"},
        {"id": "E03", "what": "Duplicate customer PO", "find": ["duplicate"], "owner": "customer service"},
        {"id": "E04", "what": "Unknown unit of measure", "find": ["unit of measure", "unknown unit"], "owner": "integration"},
        {"id": "E05", "what": "S/4HANA not reachable", "find": ["down", "not reachable", "unavailable"], "owner": "integration"},
    ],
    "tests": [
        {"id": "T-HAPPY", "level": "string", "find": ["HFS-PO-7781"]},
        {"id": "T-DOWN", "level": "string", "find": ["HFS-PO-7790"]},
        {"id": "T-DUP", "level": "string", "find": ["duplicate", "twice", "again"]},
        {"id": "T-SENDER", "level": "string", "find": ["XYZ-PO-0001"]},
        {"id": "T-E2E", "level": "end-to-end", "find": ["invoice", "delivery"]},
        {"id": "T-REPROC", "level": "end-to-end", "find": ["reprocess"]},
        {"id": "T-VOLUME", "level": "volume", "find": ["60 messages", "60 msgs", "twice the peak"]},
    ],
    "levels": ["unit", "string", "end-to-end", "volume"],
    # Things the notes rule out. A draft that does them contradicts the notes.
    "must_not": [
        {"id": "X01", "section": "mapping", "find": ["price"], "why": "S/4HANA determines the price; the notes say do not map it"},
        {"id": "X02", "section": "mapping", "find": ["incoterm"], "why": "Incoterms are out of scope for now"},
        {"id": "X03", "section": "errors", "find": ["credit"], "why": "A credit block is not an interface error"},
    ],
}

# A recorded draft, kept so the walkthrough runs without an account. It has the kinds of
# mistakes model drafts make: a missed case, a ruled-out field, a made-up rule, a wrong owner.
SAMPLE_DRAFT = {
    "interface": "I10",
    "drafted_by": "recorded sample (no model call)",
    "mapping": [
        {"source": "Sender identifier", "target": "Sold-to party",
         "rule": "Look up the trading partner profile: Hallam = 10000002",
         "evidence": "The trading partner profile says Hallam = 10000002."},
        {"source": "Purchase order number", "target": "Customer reference",
         "rule": "Copy", "evidence": "Their PO number goes into the customer reference."},
        {"source": "Purchase order date", "target": "Customer reference date",
         "rule": "Copy", "evidence": "PO date goes into the customer reference date."},
        {"source": "Line number", "target": "Item number", "rule": "Line x 10",
         "evidence": "so line x 10"},
        {"source": "Distributor part number", "target": "Product",
         "rule": "Cross-reference HFS-SK-200 -> CP-SK200",
         "evidence": "HFS-SK-200 is our CP-SK200 (seal kit)."},
        {"source": "Quantity and unit", "target": "Order quantity and unit", "rule": "EA -> PC",
         "evidence": "they send EA or PCE, we use PC."},
        {"source": "Requested date", "target": "Requested delivery date", "rule": "Copy",
         "evidence": "Requested date goes into the requested delivery date."},
        {"source": "Unit price", "target": "Net price", "rule": "Copy the distributor's unit price",
         "evidence": "they send a unit price"},
        {"source": "Incoterms", "target": "Incoterms", "rule": "Copy",
         "evidence": "Someone asked about Incoterms."},
        {"source": "(constant)", "target": "Sales area", "rule": "CPUS / 20 / PU",
         "evidence": "Sales area is always sales org CPUS, channel 20, division PU"},
    ],
    "errors": [
        {"code": "E01", "error": "Unknown sender", "owner": "Integration team",
         "next": "Reject and alert", "evidence": "If we don't know the sender, reject it and tell the integration team."},
        {"code": "E02", "error": "Unknown part number", "owner": "Customer service",
         "next": "Add the cross-reference and reprocess",
         "evidence": "customer service adds the cross-reference and reprocesses."},
        {"code": "E03", "error": "Unknown unit of measure", "owner": "Integration team",
         "next": "Extend the code list", "evidence": "Unknown unit -> integration team extends the code list."},
        {"code": "E04", "error": "S/4HANA down", "owner": "Integration team",
         "next": "Retry 4 times, then alert", "evidence": "automatic retry, 4 tries, backoff 0.5 s doubling to 4 s"},
        {"code": "E05", "error": "Credit block on the order", "owner": "Integration team",
         "next": "Retry automatically after one hour", "evidence": "Credit block on a created order"},
        {"code": "E06", "error": "Invalid tax code", "owner": "Integration team",
         "next": "Correct the tax code and reprocess", "evidence": "Tax codes are checked by the interface."},
    ],
    "tests": [
        {"level": "unit", "case": "Each mapping row with a valid value", "data": "Rows 1-10",
         "expected": "Target field filled", "evidence": "line 1 becomes item 10"},
        {"level": "string", "case": "Happy path", "data": "HFS-PO-7781",
         "expected": "One sales order for 60 CP-SK200", "evidence": "HFS-PO-7781 (60 CP-SK200) is the happy path."},
        {"level": "string", "case": "S/4HANA down", "data": "HFS-PO-7790",
         "expected": "Retries, then one order", "evidence": "HFS-PO-7790 for the S/4HANA-down case."},
        {"level": "string", "case": "Unknown sender", "data": "XYZ-PO-0001",
         "expected": "Rejected, alert to the integration team", "evidence": "XYZ-PO-0001 is an unknown sender."},
        {"level": "end-to-end", "case": "Distributor order to invoice", "data": "HFS-PO-7781",
         "expected": "Order, delivery, invoice, confirmation on I04",
         "evidence": "The confirmation back to the distributor stays on I04."},
    ],
    "open_questions": ["Which IDoc basic type and segments does the EDI provider deliver?"],
}

SYSTEM = """You draft interface specifications for an SAP S/4HANA program.
Use only the workshop notes you are given. Rules:
- Every mapping row, error and test case carries an "evidence" field: a short quote copied
  word for word from the notes. If the notes don't support an item, leave the item out.
- Anything the notes leave open goes into "open_questions", not into the mapping.
- Do not map fields the notes say are out of scope or must not be mapped.
- Reply with JSON only, in this shape:
{"interface": "I10",
 "mapping": [{"source": "", "target": "", "rule": "", "evidence": ""}],
 "errors": [{"code": "E01", "error": "", "owner": "", "next": "", "evidence": ""}],
 "tests": [{"level": "unit|string|end-to-end|volume", "case": "", "data": "", "expected": "", "evidence": ""}],
 "open_questions": [""]}"""


def norm(text: str) -> str:
    return re.sub(r"\s+", " ", str(text)).strip().lower()


def has_any(text: str, words) -> bool:
    t = norm(text)
    return any(norm(w) in t for w in words)


def has_all(text: str, words) -> bool:
    t = norm(text)
    return all(norm(w) in t for w in words)


def need(path: Path, hint: str):
    if not path.exists():
        sys.exit(f"{path.relative_to(HERE.parent)} not found. Run first: {hint}")
    return path.read_text(encoding="utf-8")


# ---------- drafting ----------

def llm_draft(model: str, notes: str) -> dict:
    try:
        from dotenv import load_dotenv
        from anthropic import Anthropic  # imported here so --sample works without the library
    except ImportError:
        sys.exit("The anthropic or python-dotenv library is missing. Run: pip install -r requirements.txt")
    load_dotenv()
    if not os.environ.get("ANTHROPIC_API_KEY"):
        sys.exit("ANTHROPIC_API_KEY is not set in .env. Add it, or use --sample.")
    try:
        reply = Anthropic().messages.create(
            model=model, max_tokens=4000, system=SYSTEM,
            messages=[{"role": "user", "content": f"<notes>\n{notes}</notes>\n\nDraft the I10 pack."}])
    except Exception as err:  # wrong key, wrong model name, network blocked
        sys.exit(f"The model call failed: {type(err).__name__}: {err}")
    text = "".join(b.text for b in reply.content if b.type == "text")
    try:
        data = json.loads(text[text.index("{"):text.rindex("}") + 1])
    except ValueError as err:
        sys.exit(f"The model's reply could not be read as JSON ({err}). Run draft again.")
    data["drafted_by"] = f"model {model}"
    return data


def cmd_init(args) -> None:
    if WORK.exists() and any(WORK.iterdir()) and not args.force:
        sys.exit("unit14/delivery already exists. Add --force to start over (deletes the draft, score and log).")
    WORK.mkdir(parents=True, exist_ok=True)
    for old in WORK.glob("*"):
        old.unlink()
    NOTES.write_text(NOTES_TEXT, encoding="utf-8")
    KEY.write_text(json.dumps(KEY_DATA, indent=1) + "\n", encoding="utf-8")
    print("Wrote unit14/delivery/notes_I10.md (workshop notes) and key_I10.json (answer key)")
    print("Next: python unit14/delivery_pack.py draft --sample")


def cmd_draft(args) -> None:
    notes = need(NOTES, "python unit14/delivery_pack.py init")
    if args.llm:
        data = llm_draft(args.model, notes)
    elif args.file:
        try:
            data = json.loads(Path(args.file).read_text(encoding="utf-8"))
        except (OSError, ValueError) as err:
            sys.exit(f"Could not read {args.file} as JSON: {err}")
        data.setdefault("drafted_by", f"file {args.file}")
    else:
        data = SAMPLE_DRAFT
    for part in ("mapping", "errors", "tests"):
        if not isinstance(data.get(part), list):
            sys.exit(f"The draft has no '{part}' list. Check the JSON shape in the script's SYSTEM text.")
    DRAFT.write_text(json.dumps(data, indent=1) + "\n", encoding="utf-8")
    print(f"Wrote unit14/delivery/draft_I10.json ({data['drafted_by']}): "
          f"{len(data['mapping'])} mapping rows, {len(data['errors'])} errors, {len(data['tests'])} tests")
    print("Next: python unit14/delivery_pack.py score")


# ---------- scoring ----------

def item_text(item: dict) -> str:
    return " ".join(str(v) for k, v in item.items() if k != "evidence")


def grounded(item: dict, notes: str) -> bool:
    quote = norm(item.get("evidence", ""))
    return len(quote) >= 8 and quote in norm(notes)


def score(draft: dict, key: dict, notes: str) -> dict:
    out = {"mapping": [], "errors": [], "tests": [], "levels": [], "contradictions": [], "ungrounded": []}

    for k in key["mapping"]:
        rows = [r for r in draft["mapping"] if has_any(r.get("target", ""), k["find"])]
        if not rows:
            out["mapping"].append((k["id"], k["target"], "MISSING", ""))
        elif any(has_all(item_text(r), k["need"]) for r in rows):
            out["mapping"].append((k["id"], k["target"], "MATCH", ""))
        else:
            text = norm(item_text(rows[0]))
            lacks = [w for w in k["need"] if norm(w) not in text]
            out["mapping"].append((k["id"], k["target"], "PARTIAL", "rule lacks " + ", ".join(lacks)))

    for k in key["errors"]:
        rows = [r for r in draft["errors"] if has_any(r.get("error", ""), k["find"])]
        if not rows:
            out["errors"].append((k["id"], k["what"], "MISSING", ""))
        elif any(k["owner"] in norm(r.get("owner", "")) for r in rows):
            out["errors"].append((k["id"], k["what"], "MATCH", ""))
        else:
            out["errors"].append((k["id"], k["what"], "PARTIAL", "wrong owner: " + rows[0].get("owner", "")))

    for k in key["tests"]:
        hit = any(has_any(item_text(t), k["find"]) for t in draft["tests"])
        out["tests"].append((k["id"], k["level"], "MATCH" if hit else "MISSING", ""))
    levels = {norm(t.get("level", "")) for t in draft["tests"]}
    out["levels"] = [lvl for lvl in key["levels"] if lvl not in levels]

    for x in key["must_not"]:
        for r in draft[x["section"]]:
            field = r.get("target", "") if x["section"] == "mapping" else r.get("error", "")
            if has_any(field, x["find"]):
                out["contradictions"].append((x["id"], f"{x['section']}: {field}", x["why"]))

    for part in ("mapping", "errors", "tests"):
        for i, r in enumerate(draft[part], 1):
            if not grounded(r, notes):
                label = r.get("target") or r.get("error") or r.get("case") or f"item {i}"
                out["ungrounded"].append((part, label, r.get("evidence", "")))
    return out


def summary(out: dict) -> dict:
    def pct(rows):
        hit = sum(1 for r in rows if r[2] == "MATCH") + 0.5 * sum(1 for r in rows if r[2] == "PARTIAL")
        return round(100 * hit / len(rows)) if rows else 0
    s = {"mapping_pct": pct(out["mapping"]), "errors_pct": pct(out["errors"]), "tests_pct": pct(out["tests"]),
         "missing_levels": len(out["levels"]), "contradictions": len(out["contradictions"]),
         "ungrounded": len(out["ungrounded"])}
    s["coverage_pct"] = round((s["mapping_pct"] + s["errors_pct"] + s["tests_pct"]) / 3)
    blockers = s["contradictions"] + s["ungrounded"]
    s["verdict"] = ("REDRAFT" if s["coverage_pct"] < 60
                    else "REVIEW, FIX THE FLAGGED ITEMS FIRST" if blockers or s["missing_levels"]
                    else "REVIEW")
    return s


def cmd_score(args) -> None:
    notes = need(NOTES, "python unit14/delivery_pack.py init")
    key = json.loads(need(KEY, "python unit14/delivery_pack.py init"))
    draft = json.loads(need(DRAFT, "python unit14/delivery_pack.py draft --sample"))
    out = score(draft, key, notes)
    s = summary(out)
    print(f"Draft: {draft.get('drafted_by', '?')}\n")
    for part, title in (("mapping", "Mapping rows"), ("errors", "Error cases"), ("tests", "Test cases")):
        print(f"{title} against the answer key: {s[part + '_pct']}%")
        for iid, what, status, note in out[part]:
            print(f"  {status:8} {iid:9} {what}" + (f"  ({note})" if note else ""))
    print(f"\nTest levels missing: {', '.join(out['levels']) or 'none'}")
    print(f"\nContradicts the notes: {len(out['contradictions'])}")
    for iid, where, why in out["contradictions"]:
        print(f"  {iid}  {where}: {why}")
    print(f"\nNot grounded in the notes (evidence quote not found): {len(out['ungrounded'])}")
    for part, label, quote in out["ungrounded"]:
        print(f"  {part}: {label}  quote: \"{quote}\"")
    print(f"\nCoverage {s['coverage_pct']}%  ->  {s['verdict']}")
    SCORE.write_text(json.dumps({"draft": draft.get("drafted_by"), "summary": s, "detail": out}, indent=1) + "\n",
                     encoding="utf-8")


# ---------- time saved ----------

def cmd_log(args) -> None:
    if not SCORE.exists():
        sys.exit("No score yet. Run: python unit14/delivery_pack.py score")
    if not args.reviewer.strip():
        sys.exit("Name the reviewer: --reviewer \"Your Name\"")
    sc = json.loads(SCORE.read_text(encoding="utf-8"))
    entry = {"at": datetime.now(timezone.utc).isoformat(timespec="seconds"), "artifact": "I10 spec and test pack",
             "draft": sc["draft"], "coverage_pct": sc["summary"]["coverage_pct"],
             "flagged": sc["summary"]["contradictions"] + sc["summary"]["ungrounded"],
             "reviewer": args.reviewer.strip(), "review_minutes": args.review_minutes,
             "fix_minutes": args.fix_minutes, "baseline_minutes": BASELINE_MINUTES}
    with LOG.open("a", encoding="utf-8") as f:
        f.write(json.dumps(entry) + "\n")
    print(f"Logged: {entry['reviewer']} reviewed for {args.review_minutes} min and fixed for {args.fix_minutes} min.")


def cmd_report(args) -> None:
    if not LOG.exists():
        sys.exit("No log yet. Run: python unit14/delivery_pack.py log --reviewer \"Your Name\" ...")
    rows = [json.loads(line) for line in LOG.read_text(encoding="utf-8").splitlines() if line.strip()]
    print(f"{'Artifact':24} {'Draft':34} {'Cov.':>5} {'Flag':>5} {'Hand':>6} {'AI+rev':>7} {'Saved':>7}")
    total_base = total_ai = 0
    for r in rows:
        spent = r["review_minutes"] + r["fix_minutes"]
        total_base += r["baseline_minutes"]
        total_ai += spent
        print(f"{r['artifact']:24} {r['draft'][:34]:34} {r['coverage_pct']:>4}% {r['flagged']:>5} "
              f"{r['baseline_minutes']:>5}m {spent:>6}m {r['baseline_minutes'] - spent:>6}m")
    saved = total_base - total_ai
    print(f"\n{len(rows)} pack(s): {total_base} min by hand, {total_ai} min with AI and review, "
          f"{saved} min saved ({round(100 * saved / total_base)}%).")
    print("Course baseline and your own minutes. Compare like with like before you claim a saving.")


def main() -> None:
    p = argparse.ArgumentParser(description="AI-drafted spec and test pack for Calder's I10, scored against a key")
    sub = p.add_subparsers(dest="cmd", required=True)
    s = sub.add_parser("init")
    s.add_argument("--force", action="store_true")
    s = sub.add_parser("draft")
    g = s.add_mutually_exclusive_group()
    g.add_argument("--sample", action="store_true", help="recorded draft (default)")
    g.add_argument("--llm", action="store_true", help="a real model drafts the pack")
    g.add_argument("--file", help="your own draft as JSON")
    s.add_argument("--model", default="claude-haiku-5-5", help="Anthropic model ID for --llm")
    sub.add_parser("score")
    s = sub.add_parser("log")
    s.add_argument("--reviewer", required=True)
    s.add_argument("--review-minutes", type=int, required=True)
    s.add_argument("--fix-minutes", type=int, required=True)
    sub.add_parser("report")
    args = p.parse_args()
    {"init": cmd_init, "draft": cmd_draft, "score": cmd_score, "log": cmd_log, "report": cmd_report}[args.cmd](args)


if __name__ == "__main__":
    main()

Step 3: Write the notes and the answer key

python unit14/delivery_pack.py init
Wrote unit14/delivery/notes_I10.md (workshop notes) and key_I10.json (answer key)
Next: python unit14/delivery_pack.py draft --sample

Open unit14/delivery/notes_I10.md. These are the kind of notes a consultant types during a workshop: short, unpolished, with a side remark about Incoterms. Then open key_I10.json. It lists the 9 mapping rows, 5 interface error cases and 7 test cases the I10 spec must contain, plus three things the notes rule out. The key's codes (E01 to E05, M01 to M09) are its own labels; a draft may number things differently.

Step 4: Get a draft

python unit14/delivery_pack.py draft --sample
Wrote unit14/delivery/draft_I10.json (recorded sample (no model call)): 10 mapping rows, 6 errors, 5 tests
Next: python unit14/delivery_pack.py score

The sample is a recorded draft with the mistakes model drafts typically make. Open draft_I10.json and read it first, before scoring. Try to spot the problems yourself; then see what the checks found.

Step 5: Score the draft

python unit14/delivery_pack.py score
Draft: recorded sample (no model call)

Mapping rows against the answer key: 83%
  MATCH    M01       Sold-to party
  MISSING  M02       Ship-to party
  MATCH    M03       Customer reference
  MATCH    M04       Customer reference date
  MATCH    M05       Item number
  MATCH    M06       Product
  PARTIAL  M07       Order quantity and unit  (rule lacks PCE)
  MATCH    M08       Requested delivery date
  MATCH    M09       Sales area
Error cases against the answer key: 80%
  MATCH    E01       Unknown sender
  MATCH    E02       Unknown part number
  MISSING  E03       Duplicate customer PO
  MATCH    E04       Unknown unit of measure
  MATCH    E05       S/4HANA not reachable
Test cases against the answer key: 57%
  MATCH    T-HAPPY   string
  MATCH    T-DOWN    string
  MISSING  T-DUP     string
  MATCH    T-SENDER  string
  MATCH    T-E2E     end-to-end
  MISSING  T-REPROC  end-to-end
  MISSING  T-VOLUME  volume

Test levels missing: volume

Contradicts the notes: 3
  X01  mapping: Net price: S/4HANA determines the price; the notes say do not map it
  X02  mapping: Incoterms: Incoterms are out of scope for now
  X03  errors: Credit block on the order: A credit block is not an interface error

Not grounded in the notes (evidence quote not found): 1
  errors: Invalid tax code  quote: "Tax codes are checked by the interface."

Coverage 73%  ->  REVIEW, FIX THE FLAGGED ITEMS FIRST

Read the result as a reviewer would:

  • The draft looks good and isn't. Most rows match, but it misses the duplicate check that customer service asked for, and with it the duplicate test. Those are the gaps that cause double orders in production.
  • Two rows contradict the notes even though both quote them: the price and Incoterms rows. Grounding alone wouldn't catch these.
  • One error is invented. Nobody mentioned tax codes; the quote isn't in the notes.
  • One owner is wrong. The draft sends credit blocks to the integration team with an automatic retry. In the notes, a credit block isn't an interface error at all.

A coverage below 60% would mean "redraft": fixing would take longer than writing. Here the verdict is to review, starting with the four flagged items.

Step 6: Log your review time

Imagine you reviewed the draft for 50 minutes and fixed it for 40. Log that:

python unit14/delivery_pack.py log --reviewer "Your Name" --review-minutes 50 --fix-minutes 40
python unit14/delivery_pack.py report
Logged: Your Name reviewed for 50 min and fixed for 40 min.
Artifact                 Draft                               Cov.  Flag   Hand  AI+rev   Saved
I10 spec and test pack   recorded sample (no model call)      73%     4   360m     90m    270m

1 pack(s): 360 min by hand, 90 min with AI and review, 270 min saved (75%).
Course baseline and your own minutes. Compare like with like before you claim a saving.

Step 7: Try a real model (optional, small charge)

python unit14/delivery_pack.py draft --llm
python unit14/delivery_pack.py score

The default model is claude-haiku-5-5; add --model <id> for another model your key supports. The system prompt asks for a quote per item and open questions for anything unclear. Results vary from run to run: some runs miss the duplicate case, some add the price. Run it three times and compare the scores. That spread is why one good demo proves nothing.

Step 8: Save your work in Git

git add unit14/delivery_pack.py unit14/delivery
git commit -m "Unit 14: AI-drafted I10 spec pack scored against an answer key"

Check with git status first: you must not see .env.

How the code works

Part of the script What it does
NOTES_TEXT The I10 workshop notes, the only source the draft may use
KEY_DATA The answer key: mapping rows, error cases, test cases and test levels the spec needs, with the words that must appear; must_not lists what the notes rule out
SAMPLE_DRAFT A recorded draft with typical mistakes, so the walkthrough runs without an account
SYSTEM, llm_draft() The prompt for --llm: JSON only, an evidence quote per item, open questions for gaps
grounded() Checks that each item's evidence quote (at least 8 characters) appears word for word in the notes
score() Matches draft items to key items as MATCH, PARTIAL or MISSING; finds contradictions and ungrounded items
summary() Turns the matches into percentages (a partial counts half) and a verdict: redraft, review with flags, or review
cmd_log(), cmd_report() Record a named reviewer's minutes next to the score; compute minutes saved against the baseline

Two design choices to notice. The script never accepts a draft: the verdict says what to review, and a person signs the spec. And the log records review time next to the draft's quality, so a program can see whether better drafts really cost less review.

If something goes wrong

What you see What it means What to do
python is not recognized, or command not found Python isn't installed, or the terminal can't find it Windows: repeat Unit 1, Step 1, then open a new terminal. macOS/Linux: use python3 until .venv is active
unit14/delivery/notes_I10.md not found You skipped Step 3 Run python unit14/delivery_pack.py init
unit14/delivery already exists. Add --force The folder is there from an earlier run Add --force; it deletes the draft, score and log
No score yet You ran log before score Run score first
The draft has no 'mapping' list Your own JSON file has another shape Copy draft_I10.json and edit it, keeping the field names
ANTHROPIC_API_KEY is not set in .env The key line is missing Add it as in Set up your computer, or use --sample
The anthropic or python-dotenv library is missing The libraries aren't installed in .venv Run pip install -r requirements.txt with .venv active
The model call failed: AuthenticationError The key is wrong or revoked Create a new key in the Claude Console and update .env
The model call failed: APIConnectionError A network or proxy blocks the call Try another network, or ask IT to allow api.anthropic.com
could not be read as JSON The model answered with text around or instead of JSON Run draft --llm again

The SAP way

As of November 2026, these SAP offerings cover parts of AI-assisted delivery. Several changed during 2026; check each for your contract and region.

  • SAP Cloud ALM, requirement generation. Drafts business requirements from Fit-to-Standard workshop transcripts into a predefined requirements template and suggests solution proposals using SAP Activate and SAP Best Practices content; consultants review before saving (sap.com). The page shows a flat-fee sales metric next to "Pricing based on consumption"; confirm which applies.
  • SAP Cloud ALM, AI-assisted document summary. Generally available per SAP's Q2 2026 highlights; the summary is edited and applied as a section of the document (SAP News).
  • Joule for Consultants. SAP Learning describes help by role: configuration guidance and documentation structure for functional consultants, root cause analysis and code explanation for technical consultants, design validation for architects, and expected deliverables per SAP Activate phase for project managers (SAP Learning). The product page says AI Units are currently required (sap.com). The Q2 2026 enhancements add custom knowledge grounding through SAP AI Core, so answers can use the program's own templates and policies.
  • Integration Advisor. AI-powered mapping proposals for message mappings, from a crowdsourced machine learning approach (sap.com). Use it for the field-level mapping of B2B and A2A messages; the spec's business rules, owners and tests still come from the workshop.
  • SAP Cloud ALM operations for hypercare. Integration & Exception Monitoring covers IDoc and web service errors, with optional payload collection, and raises alerts by inbox, email or operations flow (SAP Support). Job & Automation Monitoring rates job executions and their exceptions (SAP Support). Neither page describes AI triage; the error data they collect is what a triage step would read.
  • Migration assistants and test automation: see AI in S/4HANA migration projects.

Licensing notes. Joule for Consultants runs on AI Units. Cloud ALM's requirement generation page shows mixed pricing statements. Your own pipeline costs model tokens, through a direct provider or SAP's generative AI hub, plus review time. In every case, the review time is the larger cost.

Build vs. SAP

Situation Use Why
Fit-to-standard documented in Cloud ALM Cloud ALM requirement generation Drafts into the program's requirement template where the backlog lives
Consultants need answers grounded in SAP Notes and SAP's documentation Joule for Consultants Grounded in SAP knowledge; can add the program's own templates
B2B or A2A message mappings in Integration Suite Integration Advisor mapping proposals Built into the mapping tool; proposals at field level
Specs and test packs in the program's own template, scored against your answer keys Your own pipeline, like the script, on an approved model Full control of the template, the checks and the log
Hypercare triage across SAP and non-SAP interfaces Cloud ALM monitoring for collection, plus your own grouping and AI explanation Monitoring collects and alerts; no AI triage is described
Anything that changes data or configuration Neither drafts it into the system People act; AI drafts and explains

Architecture view

Where it sits. The pipeline lives beside the program's tools, not inside the SAP systems. Sources come from Cloud ALM (requirements, documents), the shared drive (notes, templates) and monitoring (error data). Drafts go back as documents for review.

flowchart LR
  W[Workshop notes,<br/>templates] --> P[Delivery pipeline<br/>draft + checks + log]
  CA[Cloud ALM<br/>requirements, monitoring] --> P
  P <--> M[Approved model<br/>via AI gateway]
  P --> R[Reviewer]
  R --> CA
  R --> T[Specs and tests<br/>in the program's repository]
  T --> S4[S4D build<br/>by people]

Non-functional requirements that decide the design:

Requirement Typical target What breaks if missed
Data protection Notes and logs masked before they reach a model; approved provider and region only Customer or personal data leaves the program's terms
Traceability Every signed artifact links to its draft, score, reviewer and minutes No evidence for the saving or for audits
Repeatability Same prompt, key and model version per artifact type; versions logged Scores can't be compared across weeks
Review capacity Drafts produced only as fast as reviewers can check them Review queue grows; drafts get waved through
Cost Tokens per artifact tracked next to minutes saved Savings claimed but never shown

Integration and identity. Model calls go through the program's AI gateway with the program's own key, as in AI gateways for model and agent traffic. The pipeline reads exports and documents; it has no user in S4D, S4Q or S4P. Reviewers sign in their own name in Cloud ALM or the program's repository.

The decision to record (short ADR form, as in Writing a solution design doc):

  • Context: Calder's CP20 rollout needs specs and test packs for each changed interface and gap. The program wants AI drafts.
  • Options: A, AI drafts the full spec, reviewers read it. B, AI drafts mapping, errors and tests as JSON; scripts check grounding, coverage against an answer key and the notes' rules; reviewers fix and sign; minutes are logged. C, no AI.
  • Decision: B. Checks point reviewers at likely errors, and the log shows the saving.
  • Consequences: one answer key per artifact type must be written and kept current; reviewers log minutes; saving claims cite the log.

What a design review will ask:

  1. What does each draft get checked against, and who keeps the answer keys current?
  2. Which data goes to which model, and how is it masked?
  3. Who signs each artifact, and where is the signature?
  4. How is the saving measured, and against which baseline?
  5. What may the pipeline never do?

Production concerns

Security. Workshop notes hold prices, customer names and sometimes personal data; error logs hold order numbers and addresses. Mask before sending, use an approved provider and region, and follow Data security and PII. Notes, emails and error texts can also carry instructions; treat them as data, as in Prompt injection and tool poisoning. The pipeline holds no SAP authorizations.

Evaluation. Keep answer keys per artifact type and score every draft. Track coverage, flags, review minutes and later defects. Rerun the scores when you change the prompt or the model; Building an evaluation harness shows how to make that routine.

Cost. Tokens are cheap next to consultant time; review time is the real cost. Stop drafting an artifact type when its review plus fix time approaches the baseline.

Operations. Version the prompt, the key and the model per artifact type. Keep the log with the program's test evidence until after go-live.

Clean core. AI-drafted specs can invent extensions as easily as rules. Every "gap" in a draft goes through the same fit-to-standard decision as a hand-written one.

Pitfalls

  • Scoring nothing. Drafts reviewed by feel. Without a key or a check, review is slow and misses gaps.
  • Grounding treated as truth. A quote shows the source, not that the item is right.
  • No baseline. Savings claimed without "before" hours.
  • Review time unlogged. The saving looks large because the review time is invisible.
  • Drafting faster than reviewing. A queue of drafts invites rubber-stamping.
  • AI in the action path. A triage bot that reprocesses messages or edits master data.
  • One demo as proof. Model output varies; score several runs.

Exercise

Fix the sample draft until it passes, then grow the answer key with a case it misses. The log you produce goes into your capstone's delivery evidence.

Before you start: finish Build it yourself above.

  1. Copy the draft: in VS Code's Explorer, right-click unit14/delivery/draft_I10.json, choose Copy, then right-click unit14 and choose Paste. Rename the copy to draft_fixed.json.
  2. In draft_fixed.json, fix every finding from Step 5: add a ship-to row, add PCE to the unit rule, remove the price, Incoterms, credit block and tax code items, and add a duplicate PO error owned by customer service. Add tests for a duplicate message, for error-fix-reprocess, and a volume test with 60 messages in one hour. Every item needs an evidence quote copied from notes_I10.md. Change drafted_by to fixed by Your Name.
  3. Run python unit14/delivery_pack.py draft --file unit14/draft_fixed.json, then python unit14/delivery_pack.py score. Repeat until it ends Coverage 100% -> REVIEW, with 0 contradictions and 0 ungrounded items.
  4. The notes say a requested date in the past is kept and flagged for customer service, but the key has no test for it. Open unit14/delivery/key_I10.json and add to the tests list: {"id": "T-PASTDATE", "level": "unit", "find": ["past"]}.
  5. Score again. T-PASTDATE shows MISSING. Add a unit test for it to draft_fixed.json, load it with draft --file and score again.
  6. Write down how many minutes steps 2 to 5 took you. Don't log them as a second pack: it is the same artifact, and a second log line would count its baseline twice.
  7. Write three sentences in unit14/delivery/NOTES.md: which finding the checks would have missed without the key, why the past-date case belongs in the key, and how your minutes compare with the 90 logged in Step 6.
  8. Commit: git add unit14 and git commit -m "Unit 14: fixed I10 pack and extended answer key".

Done when score prints Coverage 100% -> REVIEW with T-PASTDATE as MATCH, 0 contradictions and 0 ungrounded items, and NOTES.md is committed.

Check yourself

Pick one answer for each question. The explanation appears after you choose.

1In the draft, check, sign model, what do the automatic checks do?

Answer: C. The checks find ungrounded items, gaps against the key and contradictions, so review time goes where errors are likely. A person still reviews, fixes and signs every artifact.

2How does grounded() decide whether a draft item is grounded?

Answer: C. Each item carries an evidence quote, and the script checks that the quote appears in the notes. A missing quote, like the tax code one, usually means the item was invented.

3The sample draft maps Incoterms and quotes the notes correctly. Why is it still flagged?

Answer: D. Grounding shows where an item came from, not that it is right. The must_not list encodes what the notes rule out, which catches a grounded but wrong item.

4A draft scores 45% coverage against the answer key. What does the script's verdict mean?

Answer: A. Below 60% coverage the verdict is REDRAFT: too much is missing for review and fixes to beat writing. Above it, the reviewer starts with the flagged items.

5Your team logs 30 review minutes per AI-drafted spec but no fix minutes, and claims a 90% saving. What would you do?

Answer: A. The saving is baseline minus review and fix minutes on comparable work. Leaving out fix time or the baseline makes the number look better than it is.

6In hypercare, a model has grouped 40 I10 errors into three families. What should happen next?

Answer: A. AI may group, explain and suggest owners, which is read-only. Fixing master data and reprocessing change business data, so the error owner does them.

7Why does the architecture route model calls through the program's AI gateway and give the pipeline no SAP user?

Answer: D. The gateway applies the program's keys, limits and logging to every call. With no SAP user, the pipeline can only draft; people build and change the systems.

Sources

  • SAP Cloud ALM, requirement generation (sap.com AI feature page) — Opened 2026-11-03. Generates business requirements from Fit-to-Standard workshop transcripts into a predefined customer requirements template; suggests solution proposals using SAP Activate and SAP Best Practices content; consultants review, refine and validate before saving; pricing section shows both a flat fee metric and 'pricing based on consumption'
  • SAP Business AI: Release Highlights Q2 2026 (SAP News, 20 July 2026) — Opened 2026-11-03. SAP Cloud ALM AI-assisted document summary generally available (summary edited and applied as a section); Joule for Consultants enhancements: custom knowledge grounding on an organization's own methodologies, policies and templates via SAP AI Core, Expert workspace; one customer reports 15-20% faster unit testing with Joule generating test cases
  • SAP Joule for Consultants (sap.com AI feature page) — Opened 2026-11-03. Explains purpose, business logic and structure of ABAP code; guidance on design decisions; grounded in SAP Notes and Knowledge Base Articles; AI Units currently required, 'Activate with AI Units'
  • Delivering Role-Specific Value (SAP Learning, Introducing SAP Joule for Consultants) — Opened 2026-11-03. Functional consultants: configuration guidance, integration options, structuring documentation; technical consultants: root cause analysis, explaining inherited ABAP; enterprise architects: validating designs for clean core; project managers: expected deliverables per SAP Activate phase
  • SAP Integration Suite features (sap.com) — Opened 2026-11-03. Integration Advisor: 'AI-powered mapping proposals' for message-mapping artifacts, crowdsourced machine learning approach; Cloud Integration: AI-assisted web interface, crowdsourced message-mapping recommendations
  • Analyzing each phase of SAP Activate (SAP Learning) — Opened 2026-11-03. Explore: fit-to-standard workshops collect delta requirements and create the initial backlog, sign-off; Realize: sprints, string and integration testing, cutover planning; Deploy: cutover on the last weekend before go-live, hypercare after go-live
  • Job & Automation Monitoring (SAP Support Portal, SAP Cloud ALM expert portal) — Opened 2026-11-03. Collects job executions and exceptions and rates each execution; job types include SAP Application Jobs, ABAP jobs, BW process chains, SAP Build Process Automation jobs; lists SAP S/4HANA and SAP Business Suite among supported solutions; no AI features described
  • Integration & Exception Monitoring setup (SAP Support Portal, SAP Cloud ALM expert portal) — Opened 2026-11-03. Monitoring categories per managed service or system; IDoc and Web Services with optional payload collection; events can raise an alert in the alert inbox, an email or an operations flow; no AI features described
  • Reduce hallucinations (Claude Docs) — Opened 2026-11-03. Basic strategies: allow the model to say it doesn't know, use direct quotes for factual grounding, verify with citations; advanced: chain-of-thought verification, best-of-N, iterative refinement, external knowledge restriction

Sign in to track your progress

We'll email you a one-time sign-in link. No password needed.

or

Tell us a little about you

Optional, every field. It helps us pitch answers to your questions at the right level and decide which topics to write next. It is never shown publicly, and you can change or clear it anytime from the account menu.

SAP areas you work in