Evaluating in production: shadow mode, A/B tests and user feedback

#The 60-second version

Before launch you test an AI assistant on a fixed set of cases. That is offline evaluation, and earlier Unit 8 topics built it. It tells you the assistant works on the cases you thought of.

Online evaluation tells you whether it works on the cases your users actually bring. You measure it in production, in three steps that get riskier as they go:

1. Shadow mode. The assistant runs on real work, but nobody sees its output. You compare what it suggested with what people decided.
2. Staged rollout. A small group sees the suggestions. You compare them with colleagues who don't, in the same weeks.
3. Full rollout, with the same signals still recorded.

The most useful signals are not the thumbs-up buttons. They are what people do: accept the suggestion, edit it, or override it. Every override is a free lesson, and the worst ones become new test cases.

#Why it matters to the business

Take the blocked sales order assistant used throughout this course. Credit analysts get a drafted note for each order on credit hold, with a suggested next step: release, reject or escalate.

The offline test set had a few dozen cases and the assistant passed. Then real orders arrive: customers with several blocks at once, block codes nobody put in the test set, new analysts with different habits. Three things can go wrong without online evaluation:

• Quiet errors. The assistant suggests "release" on orders the analyst would have kept blocked. If analysts trust it, a risky order ships and a bad debt follows.
• False confidence. A dashboard shows "80% of suggestions accepted". Nobody asks whether 80% is good, or whether people accepted because the suggestion was right or because it was there.
• Wasted feedback. Analysts override the assistant dozens of times a week. Nobody turns those overrides into test cases, so the next prompt change repeats the same mistakes.

Shadow mode costs almost nothing in risk: users don't see the output, so it can't mislead them. It costs some model calls and a few weeks of patience. In return you get real accuracy numbers before anyone relies on the assistant.

#How SAP does it

As of October 2026, SAP's own training treats evaluation as continuing after launch. SAP Learning's lesson on evaluating and testing an LLM use case lists user feedback (thumbs, star ratings, comments), A/B tests across user groups, gradual rollout to a small share of traffic, monitoring, and expert review. Its ideation-to-productization lesson says building the logging and feedback mechanisms is part of the builder's job.

The pieces SAP provides:

• The real decision, in SAP. For credit blocks, SAP Learning describes the Manage Documented Credit Decisions app: analysts review blocked orders, then release, reject with a rejection reason and note, or recheck. That decision is the ground truth your shadow mode compares against (SAP Learning).
• Recording AI traffic. SAP AI Core's inference observability in the generative AI hub records requests, responses, tokens and latency when you switch it on per request. You can post feedback, such as a star rating, to a recorded inference. A tutorial dated April 2026 describes it; it needs the extended SAP AI Core plan (SAP Developers).
• Versioned prompts. The generative AI hub prompt registry keeps prompt templates with a version and history, and supports rollback. That lets two versions run side by side during a rollout (SAP Learning).
• Feedback design. SAP's Fiori design guidelines (v26.4) include an AI User Feedback pattern: thumbs up or down, optional reason categories and free text (SAP design system).

What SAP does not decide for you: which signals count as success, how large the rollout group is, or when to widen it. Those are your team's rules, agreed before the rollout.

#The rollout ladder

An A/B test is not always worth it. If the decision is "roll out or not" and the assistant only drafts notes a person checks, a staged rollout with good logging is often enough. A proper A/B test is worth its cost when the decision is large, such as letting the assistant release orders below a limit without review. The success metrics topic shows how to size such a pilot.

#Questions to ask

• What does the assistant's suggestion get compared with in shadow mode, and where does that real decision come from in SAP?
• Which signals do we log: accept, edit, override, thumbs? Who can read those logs?
• How many decisions do we need before a rate means anything? How long will that take at our volume?
• Who is in the rollout group, and who decided? Is a comparable group without the assistant measured in the same weeks?
• What would make us stop or roll back? Is that written down before the rollout starts?
• If acceptance is high, how do we know people aren't just following the suggestion?
• Who reviews overrides each week, and how do the worst ones become test cases?
• Do we need a full A/B test for this decision, or is a staged rollout enough?

#Common misconceptions

• "Thumbs-up rates tell us quality." Few people click thumbs, and unhappy users click more. What people do with the suggestion is a stronger, larger signal.
• "High acceptance means the assistant is right." People sometimes follow a suggestion because it is there. Compare with shadow mode and check accepted cases by hand.
• "Shadow mode is wasted money, since users don't see it." It is the only way to measure accuracy on real work with no risk to the process.
• "Two weeks of rollout is enough." Time doesn't decide; the number of decisions does. A small group may need months to show a small difference.
• "Every change needs an A/B test." A/B tests are slow and costly. Use them for large decisions; use offline tests and staged rollout for the rest.
• "Overrides are the analysts' problem." They are the assistant's best test cases. Ignoring them wastes the most useful data you have.

#Key terms

All course terms, A to Z, in the glossary

• Offline evaluation: testing on a fixed set of cases before release.
• Online evaluation: measuring quality on real work after release.
• Shadow mode: the AI runs on real inputs but its output is only logged, never shown.
• Staged rollout (canary): a small group gets the change first and is compared with the rest in the same period.
• A/B test: users or cases are split at random between two versions to measure a difference.
• Implicit signal: feedback read from behavior, such as accepting, editing or overriding a suggestion.
• Explicit feedback: feedback the user gives on purpose, such as a thumb or a comment.
• Override: the user decides something different from what the AI suggested.
• Confidence interval: a range that likely holds the true rate, given how many cases you have.
• Automation bias: following a system's suggestion so readily that its errors go unnoticed.

#Check yourself

#Mental model

Offline evaluation asks: does it work on the cases we chose? Online evaluation asks: does it work on the cases we get? The second question can only be answered with production logs, and each rollout step trades a little more risk for a better answer.

Every logged decision is a small experiment with one outcome: the person agreed with the AI, edited its output, or overrode it. A rate over many such experiments is a measurement with an error bar. Treat it like one.

flowchart LR
  O[Offline suite<br/>chosen cases] --> S[Shadow mode<br/>logged, hidden]
  S --> C[Staged rollout<br/>small group sees it]
  C --> F[Full rollout]
  S -. overrides .-> O
  C -. overrides .-> O
  F -. overrides .-> O

The dotted arrows are the point. Production finds cases the suite missed, and the worst of them flow back into the suite. Over time, the offline suite looks more like real work, and offline evaluation gets more trustworthy. LLM evaluation fundamentals introduced offline and online evaluation; this topic builds the online half.

#How it works

#Shadow mode: compare with the real decision

In shadow mode, the application calls the model exactly as it would in production, logs the output, and shows nothing. Amazon SageMaker's documentation describes the same idea for model endpoints: a shadow variant receives a copy of the requests, and its responses are logged for comparison, not returned to the caller.

For an assistant that suggests an action, the comparison is simple: the suggestion against what the person actually did. For the blocked-order assistant, the real decision lives in SAP: release, reject with a reason, or recheck in the Manage Documented Credit Decisions app (as SAP Learning describes it). Join your log to that decision by sales order number.

Three design rules:

• Log the version. Every record carries the prompt version, model and configuration. Without it, you can't tell which change caused what.
• Compare by slice. An overall rate hides the weak spots. Split by order type, block type, amount band and confidence, the same slices as your offline suite in Building an evaluation harness.
• Don't call it accuracy. Agreement with an analyst is not correctness: analysts make mistakes too. It is the best cheap proxy you have. Disagreements are where you look, not automatic failures.

#Staged rollout: a canary with a control

When shadow numbers look good, a small group starts to see the suggestions. Google's SRE workbook describes canarying as a partial, time-limited deployment compared against a control, and notes it is effectively an A/B test. Its key warning: time is one of the biggest sources of change in observed metrics, so compare the canary with a control in the same time window, not with last month.

The workbook also names the size trade-off. A larger canary gives a stronger signal but exposes more users to a bad change. Start small, with clear signals, and widen in stages.

For an assistant, the canary is a group of users. Keep the group stable: if people move in and out, you can't compare.

#Implicit and explicit signals

Once users see the output, they create two kinds of signal:

SAP's AI User Feedback pattern offers optional reason categories and free-text detail, requires a Submit tap for negative feedback, and warns against asking repeatedly about the same experience. Langfuse's guidance treats edited drafts as an implicit signal worth storing, with the edit itself as the comment.

#Rates with error bars

"38 of 48 accepted" is 79%. But with 48 decisions, the true rate could easily be 66% or 88%. A confidence interval says how wide that uncertainty is.

The textbook interval (the rate plus or minus 1.96 standard errors) behaves badly with small samples or rates near 0% or 100%. NIST's reference manual notes its weaker coverage and lists the Wilson score interval among the recommended alternatives, especially for small samples. The script below uses Wilson.

Rules of thumb that follow from it:

• About 60 decisions give a rate near 80% within roughly ±10 points.
• About 250 decisions give it within roughly ±5 points.
• Comparing two groups to detect a 5-point difference needs around 900 decisions in each group, at the usual 95% confidence and 80% power.

That last number is why most teams can't A/B test small changes in a credit department. At 80 blocked orders a day, 1,800 decisions is more than a month of working days, with every order in the test.

#From overrides to test cases

Not every override deserves a test case. Rank them by how much the miss would cost:

1. Direction. "Assistant said release, analyst rejected" can cost money; "assistant said escalate, analyst released" costs time.
2. Confidence. A highly confident wrong suggestion is worse than a hesitant one.
3. Value. A large order matters more.
4. Variety. Take a few per slice so one pattern doesn't fill the suite.

Then a person reviews each candidate. Langfuse's guide makes two points worth copying: review a sample before adding anything, and take the expected answer from the reviewer, never from the output the user rejected. Keep a link from each new case to the log record it came from, and keep production-derived cases marked as such.

#Reporting to the business

Report like this, not with a single percentage:

#Build it yourself: evaluate a week of shadow-mode logs

You will generate a made-up working week of logs for the blocked-order assistant: four days of shadow mode, with a small staged rollout from Wednesday. Then you will compute agreement, edit and override rates with 95% intervals, size an A/B test, and promote the worst overrides into the Unit 8 test suite.

Before you start: complete Set up your computer for this course and Set up for Unit 8. Steps 7 and 8 add cases to the suite from Building an evaluation harness; do that topic first if you want to run them. This script uses only Python's built-in modules, so there is nothing new to install.

flowchart LR
  A[simulate<br/>a week of logs] --> B[report<br/>rates and intervals]
  B --> C[size<br/>how many decisions]
  B --> D[promote<br/>rank overrides]
  D --> E[review<br/>by a person]
  E --> F[apply<br/>add to the suite]
  F --> G[harness run]

#What you need

• Your course folder orchestrate-course with its .venv, from the setup topics.
• About 45 minutes.
• No accounts and no cost. There is no model call: the log is generated by the script.
• For Steps 7 and 8 only: unit08/harness.py and its suite from Building an evaluation harness.

#Step 1: Open your course folder

1. Open VS Code, choose File > Open Folder, and open orchestrate-course.

2. Open a terminal: Terminal > New Terminal.

3. If the prompt doesn't start with (.venv), turn the virtual environment on:

   • Windows (PowerShell):

     Copy

     .venv\Scripts\Activate.ps1

   • macOS / Linux:

     Copy

     source .venv/bin/activate

4. Make sure the unit08/data folder exists:

   • Windows (PowerShell):

     Copy

     New-Item -ItemType Directory -Force unit08\data

   • macOS / Linux:

     Copy

     mkdir -p unit08/data

#Step 2: Save the script

1. In VS Code, right-click the unit08 folder, choose New File, name it online_eval.py, paste the code below and save.

"""Unit 8: evaluate the blocked-order assistant in production, from shadow-mode logs.

    python unit08/online_eval.py simulate                      # write a made-up working week of logs
    python unit08/online_eval.py report                        # agreement, edit and override rates with 95% intervals
    python unit08/online_eval.py promote                       # list the worst overrides as candidate test cases
    python unit08/online_eval.py promote --apply c01,c02       # add reviewed candidates to the Unit 8 suite
    python unit08/online_eval.py size --rate 0.80 --margin 0.05   # decisions needed for a +/-5 point interval
    python unit08/online_eval.py size --ab 0.80 0.85              # decisions per group for an A/B test

Only Python's built-in modules. The logs are synthetic: they show the method, not results from an SAP system.
"""
import argparse
import json
import math
import random
import sys
from collections import Counter
from datetime import date, datetime, timedelta, timezone
from pathlib import Path

HERE = Path(__file__).resolve().parent
LOG = HERE / "data" / "shadow_log.jsonl"
CANDIDATES = HERE / "data" / "promotion_candidates.jsonl"
SUITE = HERE / "data" / "triage_suite.jsonl"  # the suite from "Building an evaluation harness"
REPORT = HERE / "results" / "online_report.md"
Z95 = 1.959964  # 95% two-sided

# ---------------------------------------------------------------- the simulated week (made-up, SAP-shaped)

START = date(2026, 10, 12)                   # a Monday
ANALYSTS = [f"AN{i:02d}" for i in range(1, 9)]
ASSIST_GROUP = {"AN01", "AN02"}              # the staged-rollout group: they see the notes from ROLLOUT_DAY on
ROLLOUT_DAY = 3                              # day 1 is Monday; days 3 to 5 have the rollout
ACTIONS = ["release", "reject", "escalate"]

# slice: share of orders, chance the assistant's suggestion is right, mix of the right actions
SLICES = {
    "credit":   (0.40, 0.86, {"release": 0.60, "reject": 0.20, "escalate": 0.20}),
    "billing":  (0.15, 0.84, {"release": 0.70, "reject": 0.10, "escalate": 0.20}),
    "delivery": (0.20, 0.83, {"release": 0.65, "reject": 0.10, "escalate": 0.25}),
    "multi":    (0.10, 0.58, {"release": 0.40, "reject": 0.20, "escalate": 0.40}),
    "no-block": (0.15, 0.90, {"release": 0.50, "reject": 0.05, "escalate": 0.45}),
}
ANALYST_RIGHT = 0.96      # analysts pick the right action this often on their own
FOLLOWS_WRONG_NOTE = 0.30  # when they see a wrong suggestion, they follow it this often (automation bias)
REASONS = {"release": "Checked exposure and open items; fine to release.",
           "reject": "Customer has overdue items; keep the order blocked.",
           "escalate": "Needs confirmation from sales before a decision."}


def pick(rng: random.Random, weights: dict) -> str:
    return rng.choices(list(weights), weights=list(weights.values()))[0]


def order_fields(rng: random.Random, number: int, slice_name: str) -> dict:
    """Header fields shaped like Unit 1's sales order. The codes are placeholders, not SAP meanings."""
    delivery = billing = credit = ""
    if slice_name == "credit":
        delivery, credit = "01", "B"
    elif slice_name == "billing":
        billing = rng.choice(["02", "04"])
    elif slice_name == "delivery":
        delivery = rng.choice(["03", "05"])
    elif slice_name == "multi":
        delivery, billing = "03", "02"
        credit = rng.choice(["", "B"])
    amount = min(rng.lognormvariate(9.0, 1.0), 250000.0)
    return {"SalesOrder": str(9200000 + number), "SoldToParty": f"CUST-{rng.choice('ABCDEFGHIJKLMNOPQRST')}",
            "TotalNetAmount": f"{amount:.2f}", "TransactionCurrency": "USD", "TotalCreditCheckStatus": credit,
            "DeliveryBlockReason": delivery, "HeaderBillingBlockReason": billing}


def simulate_event(rng: random.Random, n: int, day: int) -> dict:
    slice_name = pick(rng, {s: v[0] for s, v in SLICES.items()})
    _, ai_right, mix = SLICES[slice_name]
    right = pick(rng, mix)                                   # what a careful analyst should decide
    ai_ok = rng.random() < ai_right
    if ai_ok:
        ai_action = right
    elif slice_name == "credit" and right == "reject" and rng.random() < 0.8:
        ai_action = "release"                                # the costly mistake: release a risky order
    else:
        ai_action = rng.choice([a for a in ACTIONS if a != right])
    confidence = pick(rng, {"high": 0.55, "medium": 0.35, "low": 0.10} if ai_ok
                      else {"high": 0.30, "medium": 0.40, "low": 0.30})
    analyst = rng.choice(ANALYSTS)
    shown = day >= ROLLOUT_DAY and analyst in ASSIST_GROUP
    if shown and not ai_ok and rng.random() < FOLLOWS_WRONG_NOTE:
        human = ai_action                                    # followed the note although it was wrong
    elif rng.random() < ANALYST_RIGHT:
        human = right
    else:
        human = rng.choice([a for a in ACTIONS if a != right])
    edited = thumb = None
    if shown:
        if human == ai_action:
            edited = rng.random() < (0.45 if confidence == "low" else 0.20)
        if rng.random() < 0.12:                              # few users ever click a thumb
            thumb = "down" if rng.random() < (0.70 if human != ai_action else 0.15) else "up"
    when = datetime(START.year, START.month, START.day, 8, tzinfo=timezone.utc) \
        + timedelta(days=day - 1, minutes=rng.randint(0, 9 * 60))
    return {"event_id": f"e{n:04d}", "ts": when.isoformat().replace("+00:00", "Z"), "day": day,
            "analyst": analyst, "prompt_version": "v1", "mode": "assist" if shown else "shadow",
            "slice": slice_name, "order": order_fields(rng, n, slice_name),
            "ai": {"action": ai_action, "confidence": confidence},
            "human": {"action": human, "reason": REASONS[human]},
            "ui": {"shown": shown, "note_edited": edited, "thumb": thumb}}


def cmd_simulate(args) -> None:
    rng = random.Random(args.seed)
    events, n = [], 0
    for day in range(1, args.days + 1):
        for _ in range(args.per_day):
            n += 1
            events.append(simulate_event(rng, n, day))
    events.sort(key=lambda e: e["ts"])
    LOG.parent.mkdir(parents=True, exist_ok=True)
    LOG.write_text("".join(json.dumps(e) + "\n" for e in events), encoding="utf-8")
    modes = Counter(e["mode"] for e in events)
    print(f"Wrote    {LOG.relative_to(HERE.parent).as_posix()}  ({len(events)} decisions, {args.days} days, seed {args.seed})")
    print(f"         shadow {modes['shadow']}, assist {modes['assist']} "
          f"(analysts {', '.join(sorted(ASSIST_GROUP))} see notes from day {ROLLOUT_DAY})")
    print("Next: python unit08/online_eval.py report")


# ---------------------------------------------------------------- statistics


def wilson(k: int, n: int, z: float = Z95) -> tuple:
    """Wilson score interval for k successes out of n (NIST/SEMATECH formula)."""
    if n == 0:
        return (0.0, 1.0)
    p = k / n
    centre = p + z * z / (2 * n)
    spread = z * math.sqrt(p * (1 - p) / n + z * z / (4 * n * n))
    return ((centre - spread) / (1 + z * z / n), (centre + spread) / (1 + z * z / n))


def pooled_z(k1: int, n1: int, k2: int, n2: int) -> float:
    """Two-proportion z statistic with the pooled rate. |z| above 1.96 means a clear difference at 95%."""
    pooled = (k1 + k2) / (n1 + n2)
    se = math.sqrt(pooled * (1 - pooled) * (1 / n1 + 1 / n2))
    return 0.0 if se == 0 else (k1 / n1 - k2 / n2) / se


def rate_line(label: str, k: int, n: int) -> str:
    if n == 0:
        return f"  {label:<22}{'no data':>12}"
    low, high = wilson(k, n)
    return f"  {label:<22}{k:>4} of {n:<4}{100 * k / n:6.1f}%   95% interval {100 * low:5.1f}% to {100 * high:5.1f}%"


# ---------------------------------------------------------------- report


def read_log() -> list:
    if not LOG.exists():
        sys.exit("No log yet. Run: python unit08/online_eval.py simulate")
    return [json.loads(line) for line in LOG.read_text(encoding="utf-8").splitlines() if line.strip()]


def agrees(e: dict) -> bool:
    return e["human"]["action"] == e["ai"]["action"]


def cmd_report(args) -> None:
    events = read_log()
    days = sorted({e["day"] for e in events})
    first = START + timedelta(days=days[0] - 1)
    last = START + timedelta(days=days[-1] - 1)
    out = [f"Online evaluation: blocked-order assistant, {first:%d %b} to {last:%d %b %Y} (SYNTHETIC data)",
           f"Logged decisions: {len(events)}", ""]

    shadow = [e for e in events if e["mode"] == "shadow"]
    out.append("1. Shadow mode: would the analyst have taken the suggested action? (notes hidden)")
    out.append(rate_line("agreement", sum(map(agrees, shadow)), len(shadow)))
    out.append("   by slice:")
    for s in SLICES:
        group = [e for e in shadow if e["slice"] == s]
        out.append(rate_line(f"  {s}", sum(map(agrees, group)), len(group)))
    out.append("   by the assistant's confidence:")
    for c in ("high", "medium", "low"):
        group = [e for e in shadow if e["ai"]["confidence"] == c]
        out.append(rate_line(f"  {c}", sum(map(agrees, group)), len(group)))

    rollout = [e for e in events if e["day"] >= ROLLOUT_DAY]
    assist = [e for e in rollout if e["mode"] == "assist"]
    same_days = [e for e in rollout if e["mode"] == "shadow"]
    out += ["", f"2. Staged rollout, days {ROLLOUT_DAY} to {days[-1]}: analysts who see the note vs. those who don't"]
    if assist:
        unchanged = sum(1 for e in assist if agrees(e) and not e["ui"]["note_edited"])
        edited = sum(1 for e in assist if agrees(e) and e["ui"]["note_edited"])
        out.append(rate_line("assist: accepted as is", unchanged, len(assist)))
        out.append(rate_line("assist: edited", edited, len(assist)))
        out.append(rate_line("assist: overridden", len(assist) - unchanged - edited, len(assist)))
        k1, k2 = sum(map(agrees, assist)), sum(map(agrees, same_days))
        out.append(rate_line("assist: agreement", k1, len(assist)))
        out.append(rate_line("shadow: agreement", k2, len(same_days)))
        z = pooled_z(k1, len(assist), k2, len(same_days))
        verdict = "a clear difference at 95%" if abs(z) > Z95 else "not a clear difference at 95%"
        out.append(f"  difference {100 * (k1 / len(assist) - k2 / len(same_days)):+.1f} points, z = {z:.2f}: {verdict}")
        if k1 / len(assist) > k2 / len(same_days):
            out.append("  Higher agreement when the note is shown can mean people follow it, right or wrong.")
            out.append("  Check a sample of accepted cases by hand before you call it an improvement.")
        out.append(f"  Is {len(assist)} enough? Try: python unit08/online_eval.py size --ab "
                   f"{k2 / len(same_days):.2f} {min(k2 / len(same_days) + 0.05, 0.99):.2f}")
    else:
        out.append("  no assist events in the log")

    shown = [e for e in events if e["ui"]["shown"]]
    thumbs = [e for e in shown if e["ui"]["thumb"]]
    out += ["", "3. Explicit feedback (thumbs) on shown notes"]
    out.append(rate_line("notes with a thumb", len(thumbs), len(shown)))
    out.append(rate_line("thumbs down", sum(1 for e in thumbs if e["ui"]["thumb"] == "down"), len(thumbs)))

    overrides = [e for e in events if not agrees(e)]
    out += ["", f"4. Overrides by direction ({len(overrides)} in total)"]
    for (ai, human), count in Counter((e["ai"]["action"], e["human"]["action"]) for e in overrides).most_common():
        flag = "   <- costly: a risky order would be released" if (ai, human) == ("release", "reject") else ""
        out.append(f"  assistant {ai:<8} -> analyst {human:<8}{count:>4}{flag}")
    out += ["", "Next: python unit08/online_eval.py promote"]

    text = "\n".join(out)
    print(text)
    REPORT.parent.mkdir(parents=True, exist_ok=True)
    REPORT.write_text("```text\n" + text + "\n```\n", encoding="utf-8")
    print(f"\nReport saved to {REPORT.relative_to(HERE.parent).as_posix()}")


# ---------------------------------------------------------------- promote overrides into the test suite


def badness(e: dict) -> tuple:
    """Score an override: the costlier and more confident the miss, the higher. Uses only logged fields."""
    score, why = 1.0, ["analyst overrode the suggestion"]
    if (e["ai"]["action"], e["human"]["action"]) == ("release", "reject"):
        score += 3
        why.append("assistant said release, analyst kept it blocked")
    if e["ai"]["confidence"] == "high":
        score += 2
        why.append("assistant was highly confident")
    amount = float(e["order"]["TotalNetAmount"])
    if amount >= 10000:
        score += min(math.log10(amount / 10000) * 2, 3)
        why.append(f"order value {amount:,.0f} USD")
    if e["ui"]["thumb"] == "down":
        score += 1
        why.append("thumbs down")
    return round(score, 2), "; ".join(why)


def cmd_promote(args) -> None:
    if args.apply:
        return apply_candidates(args.apply)
    overrides = [e for e in read_log() if not agrees(e)]
    ranked = sorted(overrides, key=lambda e: (-badness(e)[0], e["event_id"]))
    chosen, per_slice = [], Counter()
    for e in ranked:                                          # keep variety: at most 2 per slice
        if per_slice[e["slice"]] < 2:
            chosen.append(e)
            per_slice[e["slice"]] += 1
        if len(chosen) == args.top:
            break
    rows = []
    for i, e in enumerate(chosen, 1):
        score, why = badness(e)
        risky = (e["ai"]["action"], e["human"]["action"]) == ("release", "reject")
        rows.append({"candidate": f"c{i:02d}", "score": score, "why": why,
                     "case": {"id": f"prod-{e['event_id']}", "slice": e["slice"], "must_pass": risky,
                              "order": e["order"], "expected_action": e["human"]["action"],
                              "source": f"shadow log {e['ts'][:10]}, event {e['event_id']}, analyst {e['analyst']}",
                              "assistant_said": f"{e['ai']['action']} ({e['ai']['confidence']})",
                              "analyst_reason": e["human"]["reason"]}})
    CANDIDATES.write_text("".join(json.dumps(r) + "\n" for r in rows), encoding="utf-8")
    print(f"{len(overrides)} overrides in the log. Top {len(rows)} candidates for the test suite:\n")
    print(f"  {'cand':<5}{'score':>6}  {'case id':<11}{'slice':<10}{'said':<17}{'analyst':<10}must_pass")
    for r in rows:
        c = r["case"]
        print(f"  {r['candidate']:<5}{r['score']:>6}  {c['id']:<11}{c['slice']:<10}{c['assistant_said']:<17}"
              f"{c['expected_action']:<10}{str(c['must_pass']).lower()}")
    print(f"\nWrote    {CANDIDATES.relative_to(HERE.parent).as_posix()}")
    print("Review each one with a credit expert. Then add the ones you agree with, for example:")
    print("  python unit08/online_eval.py promote --apply c01,c02")


def apply_candidates(ids: str) -> None:
    if not CANDIDATES.exists():
        sys.exit("No candidates yet. Run: python unit08/online_eval.py promote")
    if not SUITE.exists():
        sys.exit("No suite at unit08/data/triage_suite.jsonl. Run: python unit08/harness.py init")
    wanted = [x.strip() for x in ids.split(",") if x.strip()]
    rows = {r["candidate"]: r for r in map(json.loads, CANDIDATES.read_text(encoding="utf-8").splitlines())}
    unknown = [w for w in wanted if w not in rows]
    if unknown:
        sys.exit(f"Unknown candidate(s): {', '.join(unknown)}. See unit08/data/promotion_candidates.jsonl")
    existing = {json.loads(line)["id"] for line in SUITE.read_text(encoding="utf-8").splitlines() if line.strip()}
    added = []
    with SUITE.open("a", encoding="utf-8") as f:
        for w in wanted:
            case = rows[w]["case"]
            if case["id"] in existing:
                print(f"Skipped  {case['id']} (already in the suite)")
                continue
            f.write(json.dumps(case) + "\n")
            existing.add(case["id"])
            added.append(case["id"])
            print(f"Added    {case['id']} ({case['slice']}, must_pass {str(case['must_pass']).lower()})")
    print(f"\nSuite now has {len(existing)} cases. Next: python unit08/harness.py run")


# ---------------------------------------------------------------- sample sizes in plain terms


def binomial(rng: random.Random, n: int, p: float) -> int:
    if hasattr(rng, "binomialvariate"):                       # Python 3.12 and later
        return rng.binomialvariate(n, p)
    return sum(rng.random() < p for _ in range(n))


def cmd_size(args) -> None:
    if args.ab:
        p1, p2 = args.ab
        if not (0 < p1 < 1 and 0 < p2 < 1) or p1 == p2:
            sys.exit("Give two different rates between 0 and 1, for example: --ab 0.80 0.85")
        z_beta = {0.8: 0.841621, 0.9: 1.281552}[args.power]
        bar = (p1 + p2) / 2
        n = math.ceil((Z95 * math.sqrt(2 * bar * (1 - bar)) + z_beta * math.sqrt(p1 * (1 - p1) + p2 * (1 - p2))) ** 2
                      / (p1 - p2) ** 2)
        rng = random.Random(args.seed)
        hits = sum(abs(pooled_z(binomial(rng, n, p2), n, binomial(rng, n, p1), n)) > Z95 for _ in range(args.trials))
        print(f"To tell {100 * p1:.0f}% from {100 * p2:.0f}% with {100 * args.power:.0f}% power at 95%:")
        print(f"  about {n} decisions in EACH group ({2 * n} in total)")
        print(f"  check: in {args.trials} simulated A/B tests of that size, {100 * hits / args.trials:.1f}% found the difference")
        return
    p, m = args.rate, args.margin
    if not (0 < p < 1 and 0 < m < 0.5):
        sys.exit("Give --rate between 0 and 1 and --margin between 0 and 0.5, for example: --rate 0.80 --margin 0.05")
    n = 10
    while True:
        low, high = wilson(round(p * n), n)
        if (high - low) / 2 <= m:
            break
        n += 1
    print(f"To report a rate near {100 * p:.0f}% within about +/-{100 * m:.0f} points (95% interval):")
    print(f"  about {n} logged decisions")
    for size in (50, 100, 250, 1000):
        low, high = wilson(round(p * size), size)
        print(f"  with {size:>4}: {100 * low:5.1f}% to {100 * high:5.1f}%")


def main() -> None:
    parser = argparse.ArgumentParser(description="Online evaluation from shadow-mode logs (Unit 8).")
    sub = parser.add_subparsers(dest="command", required=True)
    p = sub.add_parser("simulate", help="write a made-up week of shadow-mode logs")
    p.add_argument("--days", type=int, default=5)
    p.add_argument("--per-day", type=int, default=80)
    p.add_argument("--seed", type=int, default=42)
    p.set_defaults(func=cmd_simulate)
    p = sub.add_parser("report", help="rates with 95%% intervals")
    p.set_defaults(func=cmd_report)
    p = sub.add_parser("promote", help="turn the worst overrides into candidate test cases")
    p.add_argument("--top", type=int, default=6)
    p.add_argument("--apply", help="comma-separated candidate ids to add to the suite, e.g. c01,c02")
    p.set_defaults(func=cmd_promote)
    p = sub.add_parser("size", help="how many decisions you need")
    p.add_argument("--rate", type=float, default=0.80)
    p.add_argument("--margin", type=float, default=0.05)
    p.add_argument("--ab", type=float, nargs=2, metavar=("CURRENT", "HOPED"))
    p.add_argument("--power", type=float, choices=[0.8, 0.9], default=0.8)
    p.add_argument("--trials", type=int, default=2000)
    p.add_argument("--seed", type=int, default=7)
    p.set_defaults(func=cmd_size)
    args = parser.parse_args()
    args.func(args)


if __name__ == "__main__":
    main()

2. Check that Python can read it:

   Copy

   python unit08/online_eval.py --help

   You should see usage: online_eval.py [-h] {simulate,report,promote,size} .... If you see can't open file, you are not in the orchestrate-course folder, or the file has another name.

#Step 3: Generate a week of shadow-mode logs

1. Run:

   Copy

   python unit08/online_eval.py simulate

2. You should see:

   Copy

   Wrote    unit08/data/shadow_log.jsonl  (400 decisions, 5 days, seed 42)
            shadow 352, assist 48 (analysts AN01, AN02 see notes from day 3)
   Next: python unit08/online_eval.py report

3. Open unit08/data/shadow_log.jsonl in VS Code. Each line is one decision. One line, spread out for reading:

   Copy

   {"event_id": "e0002", "ts": "2026-10-12T08:06:00Z", "day": 1, "analyst": "AN05",
    "prompt_version": "v1", "mode": "shadow", "slice": "credit",
    "order": {"SalesOrder": "9200002", "SoldToParty": "CUST-N", "TotalNetAmount": "13743.85",
              "TransactionCurrency": "USD", "TotalCreditCheckStatus": "B",
              "DeliveryBlockReason": "01", "HeaderBillingBlockReason": ""},
    "ai": {"action": "reject", "confidence": "high"},
    "human": {"action": "reject", "reason": "Customer has overdue items; keep the order blocked."},
    "ui": {"shown": false, "note_edited": null, "thumb": null}}

   The order fields have the same shape as the suite in the harness topic. The codes are placeholders, not SAP meanings; block codes differ by system. ai is what the assistant suggested, human is what the analyst decided, and ui says whether the analyst saw the note.

How the week is made: 80 orders a day for five days, starting Monday 12 October 2026, with a fixed seed of 42. Each order gets a slice, a "right" action, and an assistant suggestion that is right with a chance set per slice (86% for credit blocks, 58% for orders with several blocks). Analysts pick the right action 96% of the time on their own. In the rollout group, analysts follow a wrong suggestion 30% of the time: that is the automation bias the report tries to detect. All of this is in the SLICES table and the constants above simulate_event, so you can change it.

#Step 4: Read the report

1. Run:

   Copy

   python unit08/online_eval.py report

2. You should see this (the last line names the saved file):

   Copy

   Online evaluation: blocked-order assistant, 12 Oct to 16 Oct 2026 (SYNTHETIC data)
   Logged decisions: 400
   
   1. Shadow mode: would the analyst have taken the suggested action? (notes hidden)
     agreement              283 of 352   80.4%   95% interval  75.9% to  84.2%
      by slice:
       credit               135 of 159   84.9%   95% interval  78.5% to  89.6%
       billing               40 of 50    80.0%   95% interval  67.0% to  88.8%
       delivery              49 of 63    77.8%   95% interval  66.1% to  86.3%
       multi                 13 of 28    46.4%   95% interval  29.5% to  64.2%
       no-block              46 of 52    88.5%   95% interval  77.0% to  94.6%
      by the assistant's confidence:
       high                 161 of 185   87.0%   95% interval  81.4% to  91.1%
       medium                97 of 125   77.6%   95% interval  69.5% to  84.0%
       low                   25 of 42    59.5%   95% interval  44.5% to  73.0%
   
   2. Staged rollout, days 3 to 5: analysts who see the note vs. those who don't
     assist: accepted as is  28 of 48    58.3%   95% interval  44.3% to  71.2%
     assist: edited          10 of 48    20.8%   95% interval  11.7% to  34.3%
     assist: overridden      10 of 48    20.8%   95% interval  11.7% to  34.3%
     assist: agreement       38 of 48    79.2%   95% interval  65.7% to  88.3%
     shadow: agreement      153 of 192   79.7%   95% interval  73.4% to  84.8%
     difference -0.5 points, z = -0.08: not a clear difference at 95%
     Is 48 enough? Try: python unit08/online_eval.py size --ab 0.80 0.85
   
   3. Explicit feedback (thumbs) on shown notes
     notes with a thumb       7 of 48    14.6%   95% interval   7.2% to  27.2%
     thumbs down              2 of 7     28.6%   95% interval   8.2% to  64.1%
   
   4. Overrides by direction (79 in total)
     assistant escalate -> analyst release   26
     assistant reject   -> analyst release   20
     assistant release  -> analyst reject    12   <- costly: a risky order would be released
     assistant reject   -> analyst escalate  10
     assistant escalate -> analyst reject     6
     assistant release  -> analyst escalate   5
   
   Next: python unit08/online_eval.py promote
   
   Report saved to unit08/results/online_report.md

3. Read it section by section:

   • Section 1. Agreement in shadow mode is 80%, with an interval of 76% to 84%. The weak slice stands out: orders with several blocks agree only 46% of the time, on 28 decisions. Agreement also falls with the assistant's confidence, which is what you want to see: its "high" label means something.
   • Section 2. In the rollout group, 58% of notes were accepted as is, 21% edited and 21% overridden. Agreement is about the same as in shadow mode for the same days. The script built in a 30% chance of following a wrong note, yet the report can't see it. With 48 decisions, the interval is 23 points wide. That is the sample-size problem, live.
   • Section 3. Only 7 of 48 shown notes got a thumb. A thumbs-down rate of 29% on 7 votes has an interval from 8% to 64%: almost no information. The edits and overrides in Section 2 tell you far more.
   • Section 4. Twelve times the assistant said release and the analyst kept the order blocked. Those are the misses that cost money, and the first place to look.

#Step 5: Size the next stage

1. Ask how many decisions you need to report one rate within ±5 points:

   Copy

   python unit08/online_eval.py size --rate 0.80 --margin 0.05

   Copy

   To report a rate near 80% within about +/-5 points (95% interval):
     about 245 logged decisions
     with   50:  67.0% to  88.8%
     with  100:  71.1% to  86.7%
     with  250:  74.6% to  84.5%
     with 1000:  77.4% to  82.4%

2. Ask how many decisions an A/B test needs to tell 80% from 85% agreement, the command Section 2 suggested:

   Copy

   python unit08/online_eval.py size --ab 0.80 0.85

   Copy

   To tell 80% from 85% with 80% power at 95%:
     about 906 decisions in EACH group (1812 in total)
     check: in 2000 simulated A/B tests of that size, 80.2% found the difference

   The first number comes from the standard formula for comparing two rates. The check line runs 2,000 fake A/B tests of that size and counts how often the difference showed. On Python 3.12 or later you should see 80.2%; earlier versions use a slower method and give a slightly different number, within about 2 points.

3. Compare with the rollout group: 48 decisions in three days, 16 a day. At that pace, 906 would take about 57 working days, close to three months. The decision for the business is not "run the A/B test"; it is "widen the rollout group, or accept a rougher answer".

#Step 6: Rank the overrides

1. Run:

   Copy

   python unit08/online_eval.py promote

2. You should see:

   Copy

   79 overrides in the log. Top 6 candidates for the test suite:
   
     cand  score  case id    slice     said             analyst   must_pass
     c01     7.0  prod-e0263 multi     release (high)   reject    true
     c02    6.78  prod-e0320 delivery  release (high)   reject    true
     c03    6.42  prod-e0009 delivery  release (high)   reject    true
     c04     6.0  prod-e0045 no-block  release (high)   reject    true
     c05    5.12  prod-e0278 multi     release (medium) reject    true
     c06    4.75  prod-e0168 credit    escalate (high)  release   false
   
   Wrote    unit08/data/promotion_candidates.jsonl
   Review each one with a credit expert. Then add the ones you agree with, for example:
     python unit08/online_eval.py promote --apply c01,c02

3. Open unit08/data/promotion_candidates.jsonl. Each line has a why field (the reasons for its score) and a case in the suite's format, with three extra fields: expected_action (what the analyst decided), source (which log line it came from) and analyst_reason.

4. Review them as a credit expert would. For each, ask: was the analyst right? Is this a pattern, or a one-off? In a real project, this review is a short weekly meeting with someone from the credit team.

#Step 7: Add the reviewed cases to the suite

1. If you haven't built the harness, run python unit08/harness.py init first (from Building an evaluation harness). It writes the 12-case suite.

2. Add the three candidates you agreed with:

   Copy

   python unit08/online_eval.py promote --apply c01,c02,c03

   Copy

   Added    prod-e0263 (multi, must_pass true)
   Added    prod-e0320 (delivery, must_pass true)
   Added    prod-e0009 (delivery, must_pass true)
   
   Suite now has 15 cases. Next: python unit08/harness.py run

   Running the same command again prints Skipped ... (already in the suite) for each, so a case is never added twice.

#Step 8: Run the harness on the grown suite

1. Run:

   Copy

   python unit08/harness.py run

2. The first lines say Running 15 cases. The table has the same slices as before, now with the production cases included:

   Copy

   scorer               all   billing    credit  delivery     multi  no-block
   valid_format        1.00      1.00      1.00      1.00      1.00      1.00
   covers_facts        1.00      1.00      1.00      1.00      1.00      1.00
   no_code_guessing    1.00      1.00      1.00      1.00      1.00      1.00
   no_outside_orders   0.93      1.00      1.00      1.00      0.67      1.00
   
   Failing cases: s10 (no_outside_orders)

   Your run ID and timestamps differ. The new cases pass, and that is worth understanding: the harness's four scorers check the note (format, facts, no code guessing, no outside orders). None of them checks the suggested action. The cases now carry expected_action, ready for a scorer that does. Until that scorer exists, the overrides are in the suite but not yet tested for the thing that went wrong.

3. If you saved a baseline in the harness topic, python unit08/harness.py compare still passes: the new cases pass, so the scores barely move. Save a new baseline so later runs are compared on the grown suite: python unit08/harness.py baseline.

#Step 9: Save your work in Git

1. The log and report are rebuilt by the script, so keep them out of Git. Open .gitignore, add these lines at the end and save:

   Copy

   unit08/data/shadow_log.jsonl
   unit08/data/promotion_candidates.jsonl
   unit08/results/online_report.md

2. Commit the script and the grown suite:

   Copy

   git add .gitignore unit08/online_eval.py unit08/data/triage_suite.jsonl
   git commit -m "Unit 8: online evaluation and production test cases"

#How the code works

#If something goes wrong

No keys are used in this lab. When you connect real logs later, keep any keys in .env, never in the script.

#The SAP way

As of October 2026, the SAP pieces line up with the stages like this. Recheck status before a project depends on them.

How the lab maps onto it, as a sketch (needs SAP AI Core with the extended plan, set up in Set up for Unit 5; don't run it now):

1. Call the orchestration deployment with the observability headers the tutorial names, and label each call with the prompt version, the rollout group and the sales order.
2. Record the analyst's final action from the credit decision, and post accept, edit or override back as feedback on that inference.
3. Export the records for the week into the same JSON Lines shape as shadow_log.jsonl, and run report and promote on them unchanged.

#Build vs. SAP

#Production concerns

• Personal and customer data. Logs hold customer numbers, amounts and analysts' names. Apply the same access rules as the credit decisions, keep only the fields you need, and set a retention period. See Data security and PII.
• Analyst privacy and works councils. Per-analyst override rates can look like performance monitoring. Report by group or slice, and agree with HR and, where one exists, the works council what is measured.
• SAP authorizations. The evaluation job reads credit decisions. Give it a technical user with display access only, scoped to the credit control areas in the rollout.
• Who sees the assistant. Rollout groups belong in a role or a configuration table that is changed through a request, not in code.
• Cost. Shadow mode pays for every model call while saving no time. Budget it: calls per day times price per call, from token economics.
• Comparability. Keep the rollout group stable, compare in the same weeks, and log the version on every record.
• Stopping rules. Agree the rollback and widen rules before the rollout. The exercise below builds one.
• Clean core. Everything here runs outside S/4HANA. It reads decisions through released APIs or extracts and writes nothing back.

#Pitfalls

• Calling agreement "accuracy". The analyst can be wrong. Disagreements are where you look, not automatic failures.
• Comparing with last month. Volume, season and staffing change. Compare groups in the same weeks.
• Reading a rate without its interval. 79% on 48 decisions is a range of more than 20 points.
• Trusting high acceptance after launch. Check it against shadow mode and audit accepted cases.
• Counting thumbs as the main signal. They are rare and skewed. Behavior first.
• Adding every override to the suite. The suite fills with noise. Rank, review, then add a few.
• Using the rejected output as the expected answer. The expert's correction is the expected answer.
• Peeking every day and stopping when it looks good. Decide the size and the rule first; checking repeatedly and stopping at the first good day overstates the result.

#Exercise

Add a rollout gate: a command that says whether to widen, hold or roll back the rollout, with an exit code a pipeline can read. Shipping changes with gates like this is the subject of Shipping AI changes safely.

1. Open unit08/online_eval.py.

2. Directly above the line def main() -> None:, paste this function, with one blank line before def main:

   Copy

   def cmd_gate(args) -> None:
       """Decide the next rollout step from the assist group's numbers. Agree the limits before the rollout."""
       assist = [e for e in read_log() if e["mode"] == "assist"]
       n = len(assist)
       overridden = sum(1 for e in assist if not agrees(e))
       low, high = wilson(overridden, n)
       print(f"Assist group: {n} decisions, override rate {100 * overridden / max(n, 1):.1f}% "
             f"(95% interval {100 * low:.1f}% to {100 * high:.1f}%)")
       if n and low > args.max_override:
           print(f"ROLL BACK: even the low end of the interval is above {100 * args.max_override:.0f}%")
           sys.exit(1)
       if n < args.min_n:
           print(f"HOLD: fewer than {args.min_n} decisions; keep the rollout at its current size")
           sys.exit(2)
       print("WIDEN: enough decisions, and the override rate is within the limit")

3. In main(), directly above the line args = parser.parse_args(), paste:

   Copy

       p = sub.add_parser("gate", help="widen, hold or roll back the rollout")
       p.add_argument("--max-override", type=float, default=0.30)
       p.add_argument("--min-n", type=int, default=245)
       p.set_defaults(func=cmd_gate)

4. Save, then run the gate with its defaults:

   Copy

   python unit08/online_eval.py gate

   Copy

   Assist group: 48 decisions, override rate 20.8% (95% interval 11.7% to 34.3%)
   HOLD: fewer than 245 decisions; keep the rollout at its current size

5. Try the other two outcomes: python unit08/online_eval.py gate --min-n 40 should print WIDEN, and python unit08/online_eval.py gate --max-override 0.10 should print ROLL BACK.

6. Check the exit codes. On macOS/Linux run echo $? after each command; in PowerShell run $LASTEXITCODE. You should see 2 for HOLD, 0 for WIDEN and 1 for ROLL BACK.

7. Write two sentences in unit08/rollout_rules.md: why the minimum is 245 (Step 5) and who agreed the 30% limit. Commit both files.

Done when: gate prints HOLD, WIDEN and ROLL BACK for the three commands in Steps 4 and 5, with exit codes 2, 0 and 1, and rollout_rules.md is committed.

#Check yourself