Rapid prototyping: one day, one week, two weeks

#The 60-second version

When an AI idea passes the opportunity assessment, the usual next step is a large program: a budget, a vendor, months of build. Most of the risk in that program could have been found in the first few days.

Rapid prototyping buys the answer in three short, fixed timeboxes, each ending in a decision:

1. One day: prove the data and the idea. Can we get the data the solution needs? On 20 real cases, does the output help an expert?
2. One week: a usable prototype with real users. Do a handful of the people who do the work use it, come back to it, and keep most of what it produces?
3. Two weeks: a pilot with evaluation and a decision. On a fixed test set and in daily work, measured against people working without it, does it move the metric enough, safely, at an acceptable cost?

Each timebox ends at a gate with three possible answers: GO to the next timebox, ITERATE once with a named fix, or STOP. The criteria are written down before the work starts. At most a month later, the sponsor has evidence instead of a forecast.

#Why it matters to the business

A gated prototype protects money and credibility.

• It moves the expensive surprises to the cheap end. Missing data, users who don't trust the output, or a time saving that only exists in a demo are all found in days, not after a six-month build.
• It makes stopping normal. A STOP at day one costs one day. A STOP after a program launch costs a budget, a reputation and often the next good idea.
• It turns the business case into measurements. The business case needs real numbers for quality, time saved, adoption and run cost. The pilot produces them.
• It keeps scope honest. Each timebox has one question to answer. Anything else waits.

Take the course's order-to-cash example. Credit analysts spend about 17 minutes on each blocked sales order, mostly gathering context. The frame's target is 10 minutes, with the analyst still deciding every release. On day one, the team pulls a sample of blocked orders and has two analysts rate drafted context notes for 20 of them. In week one, five analysts use a simple prototype on their real queue. In weeks two and three, the assistant is scored on a fixed test set, and analysts with it are timed against analysts without it. By then the sponsor knows whether the 10 minutes is real.

#How SAP does it

SAP describes the same arc, in its own words. As of October 2026:

• SAP Learning's path for generative AI use cases runs from ideation and feasibility to a prototype or proof of concept, then productization, deployment and monitoring. For the prototype it points to the generative AI hub playground, often with sample data, to find "a working combination of data, prompt, and model". Productization uses the SAP AI SDK. Monitoring collects user feedback such as thumbs up or down.
• SAP's evaluation guidance recommends a fixed suite of known inputs with expected outputs, expert review of high-stakes outputs, in-app feedback, and a gradual rollout that starts with a small share of traffic.
• SAP AppHaus offers the workshops before the build. Its Business AI Design Workshop takes one day on site or two and a half days virtually, and maps the future scenario step by step, including the data each step needs and where it comes from. AppHaus's prototyping guide asks teams to prepare their validation questions in advance and to keep prototypes cheap.
• SAP BTP trial accounts give an individual 90 days for experiments. SAP calls the trial a non-commercial, time-limited offering, and deletes its data at the end. A pilot with a customer's real data and users belongs in the customer's own account.

None of these sets the gates for you. Choosing the criteria, and holding the line on them, is the forward deployed engineer's job with the sponsor.

#The three timeboxes at a glance

The one-day slot can also be a "Wizard of Oz" test, where a person secretly writes the outputs that the AI would produce. It tells you whether users want the output before anyone builds the AI.

Two rules hold across all three:

• Write the gate criteria before the timebox starts, and have the sponsor agree to them. Criteria written afterwards tend to match whatever happened.
• ITERATE happens once per gate. A second failure at the same gate is a STOP. Without that rule, a prototype becomes a program by drift.

#Questions to ask

Your team or partner:

• What exactly will we know at the end of day one, week one and the pilot? What would make each one a STOP?
• Who are the five users in week one, and do they do this work every day?
• Against what will we compare in the pilot: the same people last month, or people working without the tool at the same time?
• How many test cases are in the evaluation set, and who decided the right answers?
• What data leaves the SAP system during the prototype, and who approved that?
• Does anything the prototype does change SAP data? Where does a person approve it?

Your vendor:

• Can we run this on a sample of our own data in a day, before signing a larger contract?
• Which of your reference numbers came from a pilot with a comparison group?

Yourself, as sponsor:

• Am I ready to accept a STOP at any gate, and to say so publicly?
• If the pilot reaches only part of the target, what will I do?

#Common misconceptions

• "A demo proves it works." A demo shows the best case, chosen by the builder. A gate checks many cases chosen in advance, judged by the people who do the work.
• "A prototype is a small version of the final product." It is an experiment to answer one question. Most of its code should be thrown away.
• "Two weeks is too short to learn anything." It is long enough to time real work with and without the tool. If it isn't, the metric or the volume is the problem.
• "If the pilot misses the target, the work was wasted." The pilot measured the real effect. That number is what the sponsor needed, even when it says STOP.
• "We can adjust the criteria once we see the results." Then the gate decides nothing. Any change to criteria is a new, openly agreed plan.

#Key terms

All course terms, A to Z, in the glossary

• Timebox: a fixed period of work whose length doesn't move; the scope does.
• Gate: a decision point at the end of a timebox, with criteria agreed in advance.
• GO / ITERATE / STOP: the three gate outcomes: continue, repeat once with a named fix, or end.
• Pilot: real users doing real work with the tool for a set period, measured.
• Control group: people doing the same work without the tool during the pilot, for comparison.
• Evaluation set: fixed test cases with agreed right answers, scored the same way after every change.
• Wizard of Oz test: a test where a person secretly produces the system's outputs, to learn whether users want them before building.
• Fill rate: the share of records where a needed field actually holds a value.

#Check yourself

#Mental model

A prototype is an experiment that buys one answer. The gate is where you read the answer.

Every AI idea carries a stack of risks, and they get more expensive to test the further down the stack you go:

Test the cheapest risks first, and stop as soon as one of them is fatal. That is the whole method. Everything below is about making each answer trustworthy: deciding the criteria in advance, collecting the right evidence, and holding the line at the gate.

It is the same ladder logic as the opportunity assessment, applied in time instead of in machinery. The assessment asks "what is the least machinery that meets the frame?". Prototyping asks "what is the least work that tells us whether to continue?".

#How it works

flowchart LR
  A[PROTOTYPE decision] --> D1[Day 1: data and idea]
  D1 --> G1{Gate 1}
  G1 -->|GO| W1[Week 1: real users]
  W1 --> G2{Gate 2}
  G2 -->|GO| P[Weeks 2-3: pilot]
  P --> G3{Gate 3}
  G3 -->|GO| B[Business case and production plan]
  G1 & G2 & G3 -->|ITERATE once| R[Same timebox again]
  G1 & G2 & G3 -->|STOP| S[Record lessons, park or drop]

#Before day 1: write the plan

The prototype starts from one problem the opportunity assessment marked PROTOTYPE, with its signed frame: actor, unit, baseline, target and constraints from Problem framing. The plan adds three things:

1. The evidence each gate will read, as files. If you can't name the file, you don't yet know how you will decide.
2. The criteria, as numbers. "Users like it" is not a criterion. "At least three of five users worked with it on two or more days" is.
3. The sponsor's agreement, before the work starts. AppHaus's prototyping guide makes the same point in design terms: prepare your validation questions in advance, so you know what to build.

#Day 1: prove the data and the idea

Two questions, both answered with real records.

Is the data there? Pull a sample of the cases the solution will handle: a few dozen blocked orders from a sandbox, a test system or an agreed extract. Count the fill rate of each field the solution needs. A field that is empty in half the records is a STOP for any design that depends on it, whatever the model. Note optional fields too: they often explain where outputs go wrong.

Does the idea help? Produce outputs for about 20 cases and have one or two experts rate each one. Use three ratings: useful as is, edit (right facts, needs a light change), or wrong (a fact is wrong or missing in a way that matters). SAP Learning describes the same step for generative AI: try models and prompts on sample data in the generative AI hub playground until you have "a working combination of data, prompt, and model".

These 20 rated cases are the seed of your evaluation set. A post on evaluating LLM systems in SAP Community's technology blog by SAP recommends starting with a small, representative labeled set, about 30 examples in its case, that includes failures. Keep every case the experts marked wrong.

If building even a rough version takes more than a day, run a Wizard of Oz test instead: an expert writes the outputs by hand, and users react to them. Nielsen Norman Group describes it as early validation before building expensive technology such as generative AI. Two cautions from the same article: debrief participants afterwards, and have an engineer check that the hand-written outputs are realistic for the system you could build.

#Week 1: a usable prototype with real users

Now the question is behavior, not quality. Give about five people who do the work every day a tool they can open on their real queue. A notebook is not a tool. A small web page or command-line script that reads a case and shows a draft is enough; Building an AI API shows one way.

The week has a well-known model in GV's five-day design sprint: map the problem, sketch, decide, build a prototype on day four with a "fake it" attitude, and test it with customers on day five. For an AI prototype, keep the spirit and shift the weight: build in the first day or two, then spend the rest of the week watching real use.

Why five? Nielsen Norman Group's usability research found that about five users reveal most problems in a test round, and recommends several small rounds over one large one. If the work is done by distinct groups, such as credit analysts and sales reps, test three or four people from each.

Log every use: date, user, case, minutes, and what happened to the output: accepted, edited or rejected. Add an issue field for anything that went wrong, marked critical if it could have led to a wrong business decision. The gate reads three things from the log:

• Return use. Did people come back on a second day without being chased? First-day use measures curiosity. Second-day use measures value.
• Acceptance. What share of outputs were kept, with or without edits?
• Critical issues. Any at all usually means ITERATE: fix, then retest with the same users.

Don't read the minutes column as a time saving yet. In week one, users are learning the tool and you are changing it daily.

#Weeks 2 and 3: the pilot

The pilot answers the sponsor's question: does it move the metric, safely, at an acceptable cost? It needs three kinds of evidence.

A fixed evaluation set. Grow the day-1 cases to around 40 or more, covering the slices that matter: credit blocks, overdue customers, new customers, cases with missing notes. Agree the right answer for each with the experts. Score every version of the prototype against the same set. SAP Learning calls this regression testing: known inputs with their expected outputs, rerun after each change. Building an evaluation harness builds the machinery. Mark any failure that could cause a wrong release or a wrong customer message as critical.

A comparison group. Time the same work with and without the tool, in the same weeks. Split users or cases into an assisted group and a control group. Without the control group, a quiet week or a new colleague gets credited to the AI. SAP's evaluation guidance recommends A/B testing and gradual rollouts that start with a small share of traffic; the pilot is the small, measured version of that. Defining success metrics with the business covers how to choose and guard the metric.

Run cost. Log the model cost per case. Token economics shows how to estimate it before the pilot; the log shows what it really was.

Use the median, not the mean, for handling times. A few cases that take an hour shouldn't decide the result.

#The gate decision

Each gate gives one of three outcomes, decided by rules agreed in advance:

Two details make the rules work.

• Floors. Some failures are fatal at once. If a required field is filled in only a third of records, another day won't create the data. A floor, set lower than the criterion, marks that line: below it, the decision is STOP even in round one.
• One iteration per gate. The second failure is a STOP. Without this, "one more week" repeats until the prototype is a program with no decision behind it.

A STOP against the target is not always the end of the idea. If the pilot shows a real but smaller gain, say 3.5 minutes saved per order instead of 7, the sponsor may decide that is worth having. That is a new frame with a new target, agreed openly and recorded, not a quiet change to the old gate.

#Build it yourself: a prototype gate checker

You will build a command-line tool that reads a prototype plan and the evidence files from each timebox, checks each gate, and writes a one-page decision memo for the sponsor. It works on made-up files for the blocked sales orders example, so you can see every outcome without an account.

Before you start: complete Set up your computer for this course and Set up for Unit 14. They create your orchestrate-course folder, its .venv and the unit14 folder. This script uses only Python's built-in modules, so there is nothing new to install. It doesn't call a model: the evidence files stand in for what your prototype and your users would produce.

flowchart LR
  S[sample] --> P[(plan.txt)]
  S --> E[(evidence CSV files)]
  P --> G[gate 1, 2, 3]
  E --> G
  G --> M[memo: decision.md]

#What you need

• Your course folder from the setup topics.
• About 40 minutes.
• No accounts and no cost. Every file the sample writes is made up.

#Step 1: Open your course folder

1. Open VS Code, choose File > Open Folder, and open orchestrate-course.

2. Open a terminal: Terminal > New Terminal.

3. If the prompt doesn't start with (.venv), turn it on:

   • Windows (PowerShell):

     Copy

     .venv\Scripts\Activate.ps1

   • macOS / Linux:

     Copy

     source .venv/bin/activate

4. Check that the unit14 folder exists:

   • Windows (PowerShell):

     Copy

     dir unit14

   • macOS / Linux:

     Copy

     ls unit14

   If you see an error saying the folder doesn't exist, create it with mkdir unit14.

#Step 2: Create the script

1. In VS Code, right-click the unit14 folder, choose New File, name it prototype_gates.py, paste the code below and save.

"""Prototype gates: decide GO, ITERATE or STOP at the end of each prototype timebox.

Timeboxes: 1 day (prove the data and the idea), 1 week (a usable prototype with
real users), 2 weeks (a pilot with evaluation and a decision).

Commands:
  sample   write a made-up plan and evidence files for the blocked sales orders example
  plan     print the three timeboxes, their evidence files and their criteria
  gate N   check gate 1, 2 or 3 against its evidence and print the decision
  memo     run the gates in order, stop at the first that isn't GO, write a decision memo

Uses only Python's built-in modules. No account, no network.
"""
import argparse
import csv
import statistics
import sys
from pathlib import Path

HERE = Path(__file__).resolve().parent
DEFAULT_PLAN = HERE / "prototype" / "plan.txt"

# ---------------------------------------------------------------------------
# 1. Made-up sample files. Column names are simplified on purpose; they are not
#    SAP field names. Every number is invented for the exercise.
# ---------------------------------------------------------------------------
PLAN = """# Prototype plan for one PROTOTYPE decision from the opportunity assessment.
# One FIELD: value per line. Lines starting with # are ignored. Paths are relative to this file.
NAME: Blocked sales orders, context assistant
ASSESSMENT: ../assessment/blocked_orders.txt
SPONSOR: Head of credit management
BASELINE_MIN: 17 | measured, shadowing of 8 orders
TARGET_MIN: 10

# Gate 1, end of day 1: prove the data and the idea
G1_DATA: orders.csv
G1_REQUIRED: order, customer, net_value, block_note, credit_limit, open_items
G1_OPTIONAL: rep_note
G1_REVIEW: day1_review.csv
G1_ROUND: 1
G1_FILL_MIN: 0.95
G1_FILL_FLOOR: 0.50
G1_USABLE_MIN: 0.70
G1_WRONG_MAX: 0.10

# Gate 2, end of week 1: a usable prototype with real users
G2_LOG: week_log.csv
G2_ROUND: 1
G2_ACTIVE_DAYS: 2
G2_ACTIVE_MIN: 3
G2_ACTIVE_FLOOR: 1
G2_ACCEPT_MIN: 0.60
G2_CRITICAL_MAX: 0

# Gate 3, end of week 3: a pilot with evaluation and a decision
G3_EVAL: pilot_eval.csv
G3_LOG: pilot_log.csv
G3_ROUND: 1
G3_PASS_MIN: 0.85
G3_CRITICAL_MAX: 0
G3_TARGET_SHARE_MIN: 0.70
G3_COST_MAX: 0.05 | model cost per assisted order, in your currency
"""

CUSTOMERS = ["10023", "10051", "10077", "10088", "10102", "10140"]


def _orders():
    rows = [["order", "customer", "net_value", "currency", "block_note", "credit_limit", "open_items", "rep_note"]]
    notes = ["Blocked by the credit check.", "Blocked by the credit check.", "Credit limit exceeded after new order.",
             "Blocked by the credit check. Payment overdue."]
    for i in range(24):
        customer = CUSTOMERS[i % len(CUSTOMERS)]
        limit = "" if i == 17 else str(20000 + 5000 * (i % 6))
        rep = "" if i % 4 == 3 else f"Rep: customer expects payment run on day {5 + i % 20}."
        rows.append([str(4800 + i), customer, f"{450 + 137 * i:.2f}", "EUR", notes[i % 4], limit,
                     str(18000 + 4100 * (i % 7)), rep])
    return rows


def _review(wrong_cases):
    rows = [["order", "rating", "note"]]
    for i in range(20):
        order = str(4800 + i)
        if i in wrong_cases:
            rows.append([order, "wrong", "Draft states a payment date that is not in the data"])
        elif i % 3 == 0:
            rows.append([order, "edit", "Right facts, too long"])
        else:
            rows.append([order, "useful", ""])
    return rows


def _week_log():
    rows = [["date", "user", "order", "minutes", "outcome", "issue"]]
    users = {"ana": 5, "ben": 4, "chen": 3, "dina": 1, "eli": 2}
    n = 0
    for user, days in users.items():
        for day in range(days):
            for k in range(2):
                n += 1
                outcome = "rejected" if n % 5 == 0 else ("edited" if n % 3 == 0 else "accepted")
                issue = "minor" if n % 7 == 0 else "none"
                rows.append([f"2026-10-{12 + day:02d}", user, str(5000 + n), str(9 + n % 6), outcome, issue])
    return rows


def _pilot_eval():
    rows = [["case", "slice", "result", "critical"]]
    slices = ["credit", "credit", "overdue", "new_customer", "missing_rep_note"]
    for i in range(40):
        fail = i in (6, 19, 27, 33, 38)
        rows.append([f"e{i + 1:02d}", slices[i % 5], "fail" if fail else "pass", "no"])
    return rows


def _pilot_log():
    rows = [["order", "user", "group", "minutes", "cost"]]
    for i in range(60):
        assisted = i % 3 != 0
        minutes = (10 + i % 5) if assisted else (14 + i % 6)
        cost = "0.031" if assisted else "0"
        rows.append([str(5200 + i), ["ana", "ben", "chen", "dina", "eli"][i % 5],
                     "assisted" if assisted else "control", str(minutes), cost])
    return rows


SAMPLE_CSV = {
    "orders.csv": _orders,
    "day1_review.csv": lambda: _review({3, 7, 11, 15}),
    "day1_review_round2.csv": lambda: _review({11}),
    "week_log.csv": _week_log,
    "pilot_eval.csv": _pilot_eval,
    "pilot_log.csv": _pilot_log,
}

# ---------------------------------------------------------------------------
# 2. Reading the plan and the evidence
# ---------------------------------------------------------------------------

def show(path):
    try:
        return str(Path(path).resolve().relative_to(Path.cwd().resolve()))
    except ValueError:
        return str(path)


def read_plan(path):
    plan = {}
    for raw in Path(path).read_text(encoding="utf-8").splitlines():
        line = raw.strip()
        if line and not line.startswith("#") and ":" in line:
            name, value = line.split(":", 1)
            plan[name.strip().upper()] = value.split("|")[0].strip()
    return plan


def num(plan, name):
    try:
        return float(plan[name])
    except KeyError:
        raise SystemExit(f"ERROR  the plan has no {name} line")
    except ValueError:
        raise SystemExit(f"ERROR  {name} must be a number, not '{plan[name]}'")


def read_rows(base, plan, name):
    if name not in plan:
        raise SystemExit(f"ERROR  the plan has no {name} line")
    path = base / plan[name]
    if not path.exists():
        raise SystemExit(f"ERROR  evidence file not found: {show(path)} (from {name})")
    with path.open(newline="", encoding="utf-8") as f:
        rows = list(csv.DictReader(f))
    if not rows:
        raise SystemExit(f"ERROR  {show(path)} has a header but no rows")
    return rows, path


def need(rows, path, columns):
    missing = [c for c in columns if c not in rows[0]]
    if missing:
        raise SystemExit(f"ERROR  {show(path)} is missing columns: {', '.join(missing)}")


def share(count, total):
    return count / total if total else 0.0

# ---------------------------------------------------------------------------
# 3. The gates. Each returns a list of checks: (ok, hard_fail, text).
#    ok: True if met, False if not, None for information only.
#    hard_fail: so far off that more time won't help.
# ---------------------------------------------------------------------------

def gate1(plan, base):
    checks = []
    rows, path = read_rows(base, plan, "G1_DATA")
    required = [c.strip() for c in plan.get("G1_REQUIRED", "").split(",") if c.strip()]
    need(rows, path, required)
    fill_min, floor = num(plan, "G1_FILL_MIN"), num(plan, "G1_FILL_FLOOR")
    for column in required:
        fill = share(sum(1 for r in rows if r[column].strip()), len(rows))
        checks.append((fill >= fill_min, fill < floor,
                       f"data: {column} filled in {fill:.0%} of {len(rows)} rows (need {fill_min:.0%})"))
    optional = [c.strip() for c in plan.get("G1_OPTIONAL", "").split(",") if c.strip()]
    need(rows, path, optional)
    for column in optional:
        fill = share(sum(1 for r in rows if r[column].strip()), len(rows))
        checks.append((None, False, f"data: {column} filled in {fill:.0%} of {len(rows)} rows (optional)"))
    review, rpath = read_rows(base, plan, "G1_REVIEW")
    need(review, rpath, ["rating"])
    ratings = [r["rating"].strip().lower() for r in review]
    bad = sorted({x for x in ratings if x not in ("useful", "edit", "wrong")})
    if bad:
        raise SystemExit(f"ERROR  {show(rpath)}: rating must be useful, edit or wrong, not {', '.join(bad)}")
    usable = share(ratings.count("useful") + ratings.count("edit"), len(ratings))
    wrong = share(ratings.count("wrong"), len(ratings))
    checks.append((usable >= num(plan, "G1_USABLE_MIN"), False,
                   f"idea: {usable:.0%} of {len(ratings)} drafts useful or fixable by a light edit "
                   f"(need {num(plan, 'G1_USABLE_MIN'):.0%})"))
    checks.append((wrong <= num(plan, "G1_WRONG_MAX"), False,
                   f"idea: {wrong:.0%} of drafts wrong (allow {num(plan, 'G1_WRONG_MAX'):.0%})"))
    return checks


def gate2(plan, base):
    rows, path = read_rows(base, plan, "G2_LOG")
    need(rows, path, ["date", "user", "outcome", "issue"])
    days = {}
    for r in rows:
        days.setdefault(r["user"], set()).add(r["date"])
    active = sum(1 for d in days.values() if len(d) >= num(plan, "G2_ACTIVE_DAYS"))
    outcomes = [r["outcome"].strip().lower() for r in rows]
    accept = share(outcomes.count("accepted") + outcomes.count("edited"), len(outcomes))
    critical = sum(1 for r in rows if r["issue"].strip().lower() == "critical")
    return [
        (active >= num(plan, "G2_ACTIVE_MIN"), active <= num(plan, "G2_ACTIVE_FLOOR"),
         f"use: {active} of {len(days)} users worked with it on {num(plan, 'G2_ACTIVE_DAYS'):g}+ days "
         f"(need {num(plan, 'G2_ACTIVE_MIN'):g})"),
        (accept >= num(plan, "G2_ACCEPT_MIN"), False,
         f"use: {accept:.0%} of {len(outcomes)} drafts accepted or edited, not rejected "
         f"(need {num(plan, 'G2_ACCEPT_MIN'):.0%})"),
        (critical <= num(plan, "G2_CRITICAL_MAX"), False,
         f"safety: {critical} critical issues reported (allow {num(plan, 'G2_CRITICAL_MAX'):g})"),
    ]


def gate3(plan, base):
    evals, epath = read_rows(base, plan, "G3_EVAL")
    need(evals, epath, ["result", "critical"])
    passed = share(sum(1 for r in evals if r["result"].strip().lower() == "pass"), len(evals))
    critical = sum(1 for r in evals if r["critical"].strip().lower() == "yes")
    log, lpath = read_rows(base, plan, "G3_LOG")
    need(log, lpath, ["group", "minutes", "cost"])
    assisted = [float(r["minutes"]) for r in log if r["group"] == "assisted"]
    control = [float(r["minutes"]) for r in log if r["group"] == "control"]
    if not assisted or not control:
        raise SystemExit(f"ERROR  {show(lpath)} needs rows in both groups: assisted and control")
    bench, now = statistics.median(control), statistics.median(assisted)
    target = num(plan, "TARGET_MIN")
    reached = share(bench - now, bench - target) if bench > target else 0.0
    costs = [float(r["cost"]) for r in log if r["group"] == "assisted"]
    cost = statistics.mean(costs)
    return [
        (passed >= num(plan, "G3_PASS_MIN"), False,
         f"quality: {passed:.0%} of {len(evals)} evaluation cases pass (need {num(plan, 'G3_PASS_MIN'):.0%})"),
        (critical <= num(plan, "G3_CRITICAL_MAX"), False,
         f"safety: {critical} critical failures (allow {num(plan, 'G3_CRITICAL_MAX'):g})"),
        (reached >= num(plan, "G3_TARGET_SHARE_MIN"), False,
         f"value: median {now:g} min assisted vs {bench:g} min control ({len(assisted)} vs {len(control)} orders); "
         f"{reached:.0%} of the way to {target:g} min (need {num(plan, 'G3_TARGET_SHARE_MIN'):.0%})"),
        (cost <= num(plan, "G3_COST_MAX"), False,
         f"cost: {cost:.3f} per assisted order (allow {num(plan, 'G3_COST_MAX'):.3f})"),
    ]


GATES = {1: ("end of day 1", gate1, "Start the one-week prototype with real users"),
         2: ("end of week 1", gate2, "Start the two-week pilot with an evaluation set and a control group"),
         3: ("end of the pilot", gate3, "Write the business case and the production plan")}


def decide(plan, number, checks):
    """GO if every check passes. STOP on a hard fail, or on any fail in round 2.
    Otherwise ITERATE: one more timebox of the same length, aimed at the failed checks."""
    round_no = num(plan, f"G{number}_ROUND")
    failed = [text for ok, _, text in checks if ok is False]
    hard = [text for _, hard_fail, text in checks if hard_fail]
    if not failed:
        return "GO", GATES[number][2]
    if hard:
        return "STOP", "Too far from the criteria for more time to help: " + hard[0]
    if round_no >= 2:
        return "STOP", "Second round and still failing; record what was learned and stop"
    return "ITERATE", "One more timebox of the same length, aimed only at: " + failed[0]


def label(ok, hard):
    if ok is None:
        return "INFO"
    return "PASS" if ok else ("FAIL!" if hard else "FAIL")


def run_gate(plan, base, number, quiet=False):
    title, func, _ = GATES[number]
    checks = func(plan, base)
    decision, step = decide(plan, number, checks)
    if not quiet:
        print(f"== Gate {number}, {title} (round {num(plan, f'G{number}_ROUND'):g})")
        for ok, hard, text in checks:
            print(f"  {label(ok, hard):<5}  {text}")
        print(f"  -> {decision}: {step}\n")
    return title, checks, decision, step

# ---------------------------------------------------------------------------
# 4. Commands
# ---------------------------------------------------------------------------

def cmd_sample(args):
    base = Path(args.dir)
    base.mkdir(parents=True, exist_ok=True)
    names = ["plan.txt"] + list(SAMPLE_CSV)
    if any((base / n).exists() for n in names) and not args.force:
        print(f"Sample files already exist in {show(base)}. Add --force to overwrite them.")
        return 1
    (base / "plan.txt").write_text(PLAN, encoding="utf-8")
    for name, make in SAMPLE_CSV.items():
        with (base / name).open("w", newline="", encoding="utf-8") as f:
            csv.writer(f).writerows(make())
    print(f"Wrote {len(names)} files to {show(base)}: " + ", ".join(names))
    print("All of them are made up. Next: python unit14/prototype_gates.py plan")
    return 0


def cmd_plan(args):
    plan = read_plan(args.plan)
    print(f"{plan.get('NAME', '(no NAME)')}  |  sponsor: {plan.get('SPONSOR', '(none)')}")
    print(f"Baseline {plan.get('BASELINE_MIN')} min, target {plan.get('TARGET_MIN')} min per case\n")
    rows = [("Day 1", "Prove the data and the idea", plan.get("G1_DATA", "") + ", " + plan.get("G1_REVIEW", ""),
             "required fields filled, drafts usable, few wrong"),
            ("Week 1", "Usable prototype with real users", plan.get("G2_LOG", ""),
             "users come back, accept drafts, no critical issue"),
            ("Weeks 2-3", "Pilot with evaluation and a decision",
             plan.get("G3_EVAL", "") + ", " + plan.get("G3_LOG", ""),
             "quality, safety, value against a control group, cost")]
    for when, goal, files, criteria in rows:
        print(f"{when:<10} {goal}\n{'':<10} evidence: {files}\n{'':<10} gate: {criteria}\n")
    return 0


def cmd_gate(args):
    plan = read_plan(args.plan)
    run_gate(plan, Path(args.plan).resolve().parent, args.number)
    return 0


def cmd_memo(args):
    plan = read_plan(args.plan)
    base = Path(args.plan).resolve().parent
    md = [f"# Prototype decision: {plan.get('NAME', '')}", "",
          f"Sponsor: {plan.get('SPONSOR', '')}. Baseline {plan.get('BASELINE_MIN')} min, "
          f"target {plan.get('TARGET_MIN')} min per case. All evidence files are listed in plan.txt.", ""]
    final = "GO"
    for number in (1, 2, 3):
        title, checks, decision, step = run_gate(plan, base, number)
        md += [f"## Gate {number}, {title} (round {num(plan, f'G{number}_ROUND'):g})", ""]
        md += [f"- {label(ok, hard)}: {text}" for ok, hard, text in checks]
        md += ["", f"**Decision: {decision}.** {step}", ""]
        if decision != "GO":
            final = decision
            md += [f"Later gates not run: gate {number} must pass first.", ""]
            break
    md += ["## Recommendation", "",
           {"GO": "Proceed to the business case and a production plan.",
            "ITERATE": "Run one more timebox aimed at the failed checks, then decide again.",
            "STOP": "Stop. Keep the evidence and the lessons; reassess if the named condition changes."}[final],
           "", "Decided by: ____________________  Date: __________", ""]
    out = Path(args.out) if args.out else base / "decision.md"
    out.write_text("\n".join(md), encoding="utf-8")
    print(f"Wrote {show(out)}")
    return 0


def main(argv=None):
    p = argparse.ArgumentParser(description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter)
    sub = p.add_subparsers(dest="cmd", required=True)
    s = sub.add_parser("sample", help="write made-up plan and evidence files")
    s.add_argument("--dir", default=str(DEFAULT_PLAN.parent), help="folder (default: unit14/prototype)")
    s.add_argument("--force", action="store_true", help="overwrite existing sample files")
    for name, text in [("plan", "print the timeboxes"), ("gate", "check one gate"), ("memo", "write decision.md")]:
        c = sub.add_parser(name, help=text)
        if name == "gate":
            c.add_argument("number", type=int, choices=[1, 2, 3])
        c.add_argument("--plan", default=str(DEFAULT_PLAN), help="plan file (default: unit14/prototype/plan.txt)")
        if name == "memo":
            c.add_argument("--out", help="where to write the memo (default: decision.md next to the plan)")
    args = p.parse_args(argv)
    if args.cmd != "sample" and not Path(args.plan).exists():
        print(f"ERROR  plan not found: {show(args.plan)}. Run the sample command first.")
        return 1
    handler = {"sample": cmd_sample, "plan": cmd_plan, "gate": cmd_gate, "memo": cmd_memo}[args.cmd]
    return handler(args) or 0


if __name__ == "__main__":
    sys.exit(main())

#Step 3: Write the sample plan and evidence

1. Run:

   Copy

   python unit14/prototype_gates.py sample

2. You should see:

   Copy

   Wrote 7 files to unit14/prototype: plan.txt, orders.csv, day1_review.csv, day1_review_round2.csv, week_log.csv, pilot_eval.csv, pilot_log.csv
   All of them are made up. Next: python unit14/prototype_gates.py plan

3. Open unit14/prototype/plan.txt. Each line is one setting, in the same FIELD: value style as the assessment files. Text after a | is a note for people.

   Lines	What they hold
   NAME, ASSESSMENT, SPONSOR	What is being prototyped, the assessment it came from, and who decides
   BASELINE_MIN, TARGET_MIN	Minutes per case today and the target, from the frame
   G1_...	Day 1: the data sample, its required and optional columns, the expert ratings, and the criteria
   G2_...	Week 1: the usage log and the criteria for return use, acceptance and critical issues
   G3_...	The pilot: the evaluation results, the timing and cost log, and the criteria
   G1_ROUND, G2_ROUND, G3_ROUND	Which attempt this is at each gate: 1 or 2
   ..._FLOOR	Below this, the gate says STOP at once

4. Open the CSV files in VS Code to see the evidence. For example, week_log.csv starts like this:

   Copy

   date,user,order,minutes,outcome,issue
   2026-10-12,ana,5001,10,accepted,none
   2026-10-12,ana,5002,11,accepted,none
   2026-10-13,ana,5003,12,edited,none

   The column names in orders.csv are simplified on purpose; they are not SAP field names. In a real prototype the sample would come from an SAP API, as in Calling your first SAP API, or from an extract the data owner approved.

If you see Sample files already exist, you ran this before. Add --force to overwrite them.

#Step 4: Print the plan

1. Run:

   Copy

   python unit14/prototype_gates.py plan

2. You should see the three timeboxes:

   Copy

   Blocked sales orders, context assistant  |  sponsor: Head of credit management
   Baseline 17 min, target 10 min per case
   
   Day 1      Prove the data and the idea
              evidence: orders.csv, day1_review.csv
              gate: required fields filled, drafts usable, few wrong
   
   Week 1     Usable prototype with real users
              evidence: week_log.csv
              gate: users come back, accept drafts, no critical issue
   
   Weeks 2-3  Pilot with evaluation and a decision
              evidence: pilot_eval.csv, pilot_log.csv
              gate: quality, safety, value against a control group, cost

This is the page the sponsor agrees to before day one.

#Step 5: Check gate 1

1. Run:

   Copy

   python unit14/prototype_gates.py gate 1

2. You should see:

   Copy

   == Gate 1, end of day 1 (round 1)
     PASS   data: order filled in 100% of 24 rows (need 95%)
     PASS   data: customer filled in 100% of 24 rows (need 95%)
     PASS   data: net_value filled in 100% of 24 rows (need 95%)
     PASS   data: block_note filled in 100% of 24 rows (need 95%)
     PASS   data: credit_limit filled in 96% of 24 rows (need 95%)
     PASS   data: open_items filled in 100% of 24 rows (need 95%)
     INFO   data: rep_note filled in 75% of 24 rows (optional)
     PASS   idea: 80% of 20 drafts useful or fixable by a light edit (need 70%)
     FAIL   idea: 20% of drafts wrong (allow 10%)
     -> ITERATE: One more timebox of the same length, aimed only at: idea: 20% of drafts wrong (allow 10%)

3. Open unit14/prototype/day1_review.csv and find the four rows rated wrong: orders 4803, 4807, 4811 and 4815. Then find the same orders in orders.csv. All four have an empty rep_note. The drafts invented a payment date when the sales rep's note was missing.

That is a typical day-1 finding: the data check and the expert ratings together point at the cause. The fix for the next round is narrow: tell the model to say "no note from the sales rep" when the field is empty, and rate 20 drafts again.

#Step 6: Record round 2 of gate 1

Suppose you made that fix and the experts rated 20 new drafts. The sample already holds their ratings in day1_review_round2.csv.

1. Open unit14/prototype/plan.txt in VS Code.

2. Change the line G1_REVIEW: day1_review.csv to:

   Copy

   G1_REVIEW: day1_review_round2.csv

3. Change the line G1_ROUND: 1 to:

   Copy

   G1_ROUND: 2

4. Save the file and run gate 1 again:

   Copy

   python unit14/prototype_gates.py gate 1

5. The last lines should be:

   Copy

     PASS   idea: 95% of 20 drafts useful or fixable by a light edit (need 70%)
     PASS   idea: 5% of drafts wrong (allow 10%)
     -> GO: Start the one-week prototype with real users

Keep the old review file. The memo and your Git history should show that gate 1 needed a second round, and why.

#Step 7: Check gates 2 and 3

1. Run:

   Copy

   python unit14/prototype_gates.py gate 2

2. You should see:

   Copy

   == Gate 2, end of week 1 (round 1)
     PASS   use: 4 of 5 users worked with it on 2+ days (need 3)
     PASS   use: 80% of 30 drafts accepted or edited, not rejected (need 60%)
     PASS   safety: 0 critical issues reported (allow 0)
     -> GO: Start the two-week pilot with an evaluation set and a control group

3. Run:

   Copy

   python unit14/prototype_gates.py gate 3

4. You should see:

   Copy

   == Gate 3, end of the pilot (round 1)
     PASS   quality: 88% of 40 evaluation cases pass (need 85%)
     PASS   safety: 0 critical failures (allow 0)
     FAIL   value: median 12 min assisted vs 15.5 min control (40 vs 20 orders); 64% of the way to 10 min (need 70%)
     PASS   cost: 0.031 per assisted order (allow 0.050)
     -> ITERATE: One more timebox of the same length, aimed only at: value: median 12 min assisted vs 15.5 min control (40 vs 20 orders); 64% of the way to 10 min (need 70%)

Read the value line carefully. The control group took a median of 15.5 minutes, not the 17 from the baseline: the work itself changed between discovery and the pilot. The script measures progress from the control group, so the AI only gets credit for its own effect. From 15.5 to 12 minutes is 3.5 of the 5.5 minutes to the target, or 64%. Quality, safety and cost passed. The pilot shows a real gain that falls short of the agreed bar.

#Step 8: Write the decision memo

1. Run:

   Copy

   python unit14/prototype_gates.py memo

2. The terminal prints each gate again and ends with:

   Copy

   Wrote unit14/prototype/decision.md

3. Open unit14/prototype/decision.md. It lists every check for gates 1 to 3, the decision at each, a recommendation and a line for the sponsor's signature. The memo stops at the first gate that isn't GO, so a later gate is never reported on evidence that shouldn't exist yet.

#Step 9: See a STOP

1. In plan.txt, change G3_ROUND: 1 to G3_ROUND: 2 and save. This pretends the second pilot round gave the same numbers.

2. Run:

   Copy

   python unit14/prototype_gates.py gate 3

3. The last line should be:

   Copy

     -> STOP: Second round and still failing; record what was learned and stop

4. Change G3_ROUND back to 1 and save, so your memo matches the evidence you actually have.

The rule says STOP. The sponsor may still decide that 3.5 minutes on 6,000 orders a month is worth having. If so, that is a new frame with a new target, written down, not an edit to the old criteria.

#How the code works

#Use your own prototype

1. Copy unit14/prototype/plan.txt to a new folder, for example unit14/my_prototype/plan.txt.

2. Change NAME, SPONSOR, the baseline, the target and the criteria to fit your problem. Write them before you collect any evidence.

3. Put your own evidence files next to it, with the same columns as the samples.

4. Run any command with --plan:

   Copy

   python unit14/prototype_gates.py gate 1 --plan unit14/my_prototype/plan.txt

#If something goes wrong

#The SAP way

Each timebox maps onto SAP tools. Names and terms below are as of October 2026, from the sources listed.

#Day 1: data and a first output

• Data. For learning, the SAP Business Accelerator Hub offers a sandbox for trying its APIs, as in Calling your first SAP API. For a customer, the day-1 sample comes from a test system or an extract the data owner approves. Data security and PII covers what may leave the system.
• Outputs. SAP Learning places the proof of concept in the generative AI hub playground, comparing models and prompts on sample data. Learners can use the 30-day generative AI hub trial from Set up for Unit 5; a customer prototype uses the customer's own SAP AI Core instance.
• Workshops. If the scenario itself is still unclear, run an AppHaus Business AI Design Workshop first. Its to-be scenario names the data and source for every step, which is exactly the day-1 data list. Its Capability Idea Napkin asks what a user would need in order to trust the result; that answer becomes a day-1 rating criterion.

#Week 1: something users can open

• A small app. A side-by-side app on SAP BTP, as in CAP and side-by-side extensions and Deploying AI apps on BTP, calling the model through the generative AI hub and orchestration. SAP Learning names the SAP AI SDK for connecting an application to the generative AI hub.
• Feedback. AppHaus offers a Feedback Grid for capturing user impressions; SAP Learning describes in-app thumbs up or down. Both complement the usage log; neither replaces it.

#The pilot

• Evaluation. SAP Learning's evaluation lesson lists regression suites with ground truth, expert review, LLM-as-judge and A/B testing. The evaluation harness from Unit 8 runs the fixed set.
• Observability and cost. Log every call with its case, user, latency and token cost, as in Observability for AI systems.
• Where it runs. The SAP BTP trial lasts 90 days, needs a login within every 30 days, and deletes all data at the end. SAP calls it a non-commercial, time-limited offering. That suits a learner's version of this topic. A pilot with a customer's users and data runs in the customer's own BTP account, under an Enterprise Agreement or Pay-As-You-Go.

#What SAP doesn't prescribe

None of the SAP sources opened for this topic set day, week and fortnight timeboxes or numeric gate criteria. SAP Learning's stages and AppHaus's phases describe the order of work, not how long each step takes. The timeboxes and thresholds here are this course's starting points.

#Build vs. SAP

A practical mix: an AppHaus-style workshop for agreement on the scenario, the playground for day one, and your own plan, logs and gate memo kept in Git so the decision can be checked later.

#Production concerns

A prototype isn't production, but it touches production data and real users. Treat these as gate conditions, not afterthoughts.

• Data handling. Know before day one what data leaves the SAP system, where it goes and who approved it. Use the smallest sample that answers the question. Personal data needs data protection involved from the start.
• Authorizations. Even in week one, the tool should read data with the user's own authorizations, not a broad technical user. A prototype built on broad access fails its security review later, and teaches users to expect data they shouldn't see. Agent permissions and SAP authorizations covers the patterns.
• No changes to SAP data. A prototype drafts; a person decides. Anything that would change SAP data needs explicit approval, enforced in code. Most prototypes shouldn't write to SAP at all.
• Evaluation integrity. Freeze the evaluation set before the pilot. Adding easy cases during the pilot raises the score without improving the tool.
• Cost. Log cost per case from day one. A pilot that passes on quality but costs more per case than the time it saves fails the business case.
• Clean core. Build side by side, on released APIs. A prototype that needs a modification in the core is a red flag for production.
• Exit. Plan what happens on STOP: switch off access, delete extracts and logs that hold customer data, and tell the users what was learned.

#Pitfalls

• Criteria written after the evidence. The gate then decides nothing. Write them first, get them agreed, keep them in Git.
• Demo cases instead of a sample. Experts rating cases the builder picked is a demo, not a test. Sample the cases.
• Polishing in week one. Screens and integration before anyone has used the tool are effort spent on a question nobody asked yet.
• No control group. Comparing pilot timings with an old baseline credits the AI with everything else that changed.
• Counting first-day use. Curiosity is not adoption. Look for second-day use without reminders.
• ITERATE without a target. "Improve it" is not an iteration plan. Name the failed criterion and aim only at it.
• Endless iteration. One ITERATE per gate. The second failure is a STOP.
• Letting the prototype become production. Prototype code skips security, error handling and operations on purpose. After GO at gate 3, plan the production build properly.

#Exercise

Write the prototype plan for your own problem, the one your opportunity assessment marked PROTOTYPE, and run it through the gate checker.

1. Create the folder unit14/my_prototype and copy unit14/prototype/plan.txt into it.

2. In the copy, set G1_REVIEW: day1_review.csv and G1_ROUND: 1, then set NAME, ASSESSMENT (the path to your assessment file, if you have one), SPONSOR, BASELINE_MIN and TARGET_MIN from your frame. If you have no frame of your own, keep the blocked sales orders numbers.

3. Change at least two criteria to fit your problem, and add a # comment line above each saying why. Do this before step 4.

4. Create a day1_review.csv next to the plan with at least 10 cases you rated yourself, using useful, edit or wrong. Copy orders.csv, week_log.csv, pilot_eval.csv and pilot_log.csv from the sample folder for the other gates, or make your own.

5. Run the gates and the memo:

   Copy

   python unit14/prototype_gates.py gate 1 --plan unit14/my_prototype/plan.txt
   python unit14/prototype_gates.py memo --plan unit14/my_prototype/plan.txt

6. Under the recommendation in unit14/my_prototype/decision.md, add two sentences: what your day-1 ratings taught you, and what would make you STOP at gate 1.

7. Save your work:

   Copy

   git add unit14/prototype_gates.py unit14/prototype/ unit14/my_prototype/
   git commit -m "Unit 14: prototype plan, gate checker and decision memo"

Done when gate 1 --plan unit14/my_prototype/plan.txt prints a decision with no ERROR lines, the plan has a comment explaining each criterion you changed, and decision.md ends with your two sentences. Keep the plan: the capstone at the end of Unit 14 starts from it.

#Check yourself