Shipping AI changes safely: CI/CD for prompts, models and agents

#The 60-second version

An AI feature can change without anyone touching its code. Someone edits a sentence in the prompt. Someone moves to a newer model. Someone rewords a tool description so an agent picks it more often. Each of these changes what users get. Each is a production change.

Software teams already have a safe way to ship code: CI/CD. Every change goes through a pipeline that tests it, and only changes that pass reach users. The idea of this topic is simple: put the AI's settings into that same pipeline.

That means four habits:

1. Keep the AI's settings with the code. Prompts, model names, model settings and tool definitions live in files under version control, reviewed like code.
2. Test every change against an evaluation gate. The pipeline replays a fixed set of test cases. If quality drops below agreed limits, the change is blocked.
3. Roll out in small steps. A new version goes to a small group first (a canary). If it does worse than the current version, it is rolled back.
4. Watch the calendar. Model versions retire on fixed dates. The pipeline warns you before your model does.

#Why it matters to the business

Take the blocked sales order assistant used throughout this course. It drafts a triage note for each order on hold, and a credit analyst decides what to do.

A developer shortens the prompt to cut token costs. The change looks harmless and ships on a Friday. On Monday the notes start to explain what block code "01" means, and some advise releasing the order. Codes differ by system, and releasing is the analyst's call, not the assistant's. Nobody notices for a week, because nothing crashed.

With a release pipeline, that change never reaches users. The pull request runs the evaluation suite. Two must-pass cases fail: the note guesses a code meaning and gives release advice. The merge button stays grey, and the reason is in the report.

The value is the same as for any release discipline:

• Fewer incidents. Regressions are caught before users see them, not after a week of wrong advice.
• Faster, calmer change. When every change is tested the same way, teams change more often with less fear. Cheaper models and better prompts reach users sooner.
• An audit trail. You can answer "what prompt and model were live on 3 March?" from Git history and release records. That matters for internal audit and for AI regulation.
• No surprise outages. A model retirement becomes a planned change, not a Monday-morning failure.

The cost is modest: a pipeline, a test set you probably have from evaluation work, and the discipline to route every AI change through it.

#How SAP does it

SAP does not sell one "AI CI/CD" product. As of October 2026, the parts come from the generative AI hub in SAP AI Core and from SAP's general DevOps services on SAP BTP.

• Prompt registry. The generative AI hub keeps prompt templates with a name, version and scenario. SAP Learning describes two ways to manage them: imperative, through an API with edit history, and declarative, where a YAML file in a Git repository is synced into the registry by SAP AI Core. SAP positions the declarative way for runtime apps and CI/CD (SAP Learning).
• Stored orchestration configurations. Since February 2026, SAP's What's New log says the registry can also hold orchestration configurations declaratively: the model, prompt and other module settings as one versioned object (SAP AI Core service guide).
• Evaluations. The same log lists Evaluations (December 2025) for benchmarking prompts and models through orchestration configurations. It is the SAP-side engine for the test step; Building an evaluation harness covers it.
• Pipelines and transports. SAP Continuous Integration and Delivery is SAP's pipeline-as-a-service: a Git push triggers a predefined pipeline, which can hand a release candidate to SAP Cloud Transport Management. That service keeps a central change log (SAP Learning).
• Model dates. Model versions in the generative AI hub retire on published dates. Running AI on SAP BTP in production explains where to read them and what happens on the day.

What SAP does not provide out of the box is the gate itself: the rule that says "this prompt change is blocked because must-pass cases fail". Your team writes that rule and runs it in whichever pipeline you use.

#What counts as an AI release

Use this table to decide what must go through the pipeline. If a change appears in the left column, it is a release, even if no code changed.

The last row is the tricky one. If you let a provider move you to the latest model automatically, you avoid retirement outages but lose control over when behavior changes. Many teams pin a version in production and re-run the gate on the newer version in a test environment.

#Questions to ask

• Where do our prompts, model names and tool definitions live? Can anyone change them outside version control?
• What evaluation runs on every AI change, and what result blocks a release?
• Who owns the thresholds, and who may override a failed gate? Is each override recorded?
• Which model versions do we use, when does each retire, and who watches those dates?
• Do we pin model versions in production or follow the latest? Who decided, and why?
• How does a new version reach users: everyone at once, or a small group first?
• How fast can we roll back to the previous prompt and model, and have we tried it?
• Can we say exactly which prompt and model were live on a given day?

#Common misconceptions

• "A prompt edit isn't a deployment." It changes every answer the system gives. Treat it like a code change: review, test, release.
• "Our unit tests cover it." Unit tests check code paths. They don't check whether the answers got worse.
• "The evaluation gate makes a canary unnecessary." The gate tests the cases you wrote. Real users bring cases you didn't, so a small rollout still matters.
• "Following the latest model version removes risk." It removes the retirement date, but behavior can change under you without a release.
• "Rollback is just undoing the commit." Only if the old prompt, model and settings are still available. Retired models can't be rolled back to.
• "Running the gate costs a fortune in model calls." Pull requests can replay recorded answers for free. Real calls are needed only when the AI's settings actually change.

#Key terms

All course terms, A to Z, in the glossary

• CI/CD: continuous integration and delivery; every change is built and tested automatically, and passing changes are released through a pipeline.
• Pipeline: the automated steps a change goes through, such as test, gate and deploy.
• Evaluation gate: a pipeline step that blocks a release when evaluation scores fall below agreed limits.
• Release candidate: the exact combination of prompt, model, settings and tools proposed for release.
• Fingerprint: a short code computed from a file's contents; any edit changes it.
• Recorded answers (replay): model answers saved once and reused in tests, so tests run without calling the model.
• Canary: a small share of users or traffic that gets a new version first, compared with a control group.
• Rollback: returning to the previous known-good release.
• Model retirement: the date after which a model version is no longer served.
• Prompt registry: a store for versioned prompt templates; in SAP, part of the generative AI hub.

#Check yourself

#Mental model

The system you ship is not just your code. It is code + prompt + model + model settings + tool definitions, and each part can change on its own. A release pipeline treats that whole bundle as the thing that gets versioned, tested and released.

Give the bundle one fingerprint: a hash of the prompt text, the model settings and the tool definitions. Then the rules become mechanical:

• A change to any part gives a new fingerprint, so it is a new release candidate.
• A candidate can be released only if the gate passed on that fingerprint.
• Test evidence (scores, recorded answers) is valid only for the fingerprint it was made with.
• Rollback means making an older fingerprint live again, which works only if that model still exists.

flowchart LR
  PR[Pull request<br/>prompt, model, tools] --> FP[Fingerprint]
  FP --> G{Evaluation gate}
  G -->|fail| BLOCK[Merge blocked<br/>report attached]
  G -->|pass| REL[Release record]
  REL --> CAN{Canary vs. control}
  CAN -->|worse| RB[Rollback to<br/>previous release]
  CAN -->|not worse| ALL[Full rollout]
  CAL[Model retirement<br/>dates] -.weekly check.-> G

#How it works

#Version the AI's settings with the code

Put every behavior-defining file in the repository, next to the code that uses it:

Two rules make this work. First, the app reads these files at runtime, so what you test is what runs. Second, nobody edits a prompt in a production console. On SAP, the declarative prompt registry follows the same idea: the YAML in Git is the source, and SAP AI Core syncs it.

#The evaluation gate in CI, without paying for every pull request

Calling a real model on every pull request costs money, needs a secret in CI, and gives answers that vary between runs. A recorded model mock solves all three:

1. When the AI's settings change, a developer runs the suite once against the real model and saves the answers in a recordings file. Each answer is keyed by the candidate's fingerprint plus the case input.
2. The developer commits the recordings with the change. Reviewers see the new answers in the diff.
3. CI replays the recordings, scores them with rules and applies the gate. No key, no cost, same result every time.

The key detail is what happens when a recording is missing. If someone edits the prompt and does not re-record, the fingerprint changes and no recording matches. The gate must fail closed: "10 cases have no recording". A gate that skipped missing cases would pass a change nobody tested.

sequenceDiagram
  participant Dev as Developer
  participant LLM as Model API
  participant Git as Repository
  participant CI as CI job
  Dev->>LLM: record --llm (new fingerprint)
  LLM-->>Dev: answers per case
  Dev->>Git: commit prompt + recordings
  Git->>CI: pull request event
  CI->>CI: replay recordings, score, gate
  CI-->>Git: check passes or fails, report attached

The gate rules are the ones from Building an evaluation harness: minimum scores per rule, a maximum drop against the released version, and must-pass cases that may never fail. The released version's scores are the baseline.

#Canary releases

A gate tests the cases you wrote. A canary tests the change on real work at small scale. Google's SRE workbook describes canarying as a partial, time-limited deployment that is evaluated against a control: the users still on the current version, in the same period (SRE Workbook). Comparing with the control matters because time changes behavior; last week's numbers are not a fair comparison.

For an AI assistant, pick a few signals tied to user impact:

• Error rate: failed or unparsable responses.
• Override rate: how often analysts decide differently from the suggestion. Evaluating in production explains why this is the strongest signal.
• Latency: the 95th percentile response time, against the budget from Latency budgets and performance.

The decision has three outcomes. Rollback if the canary is worse on any signal and the gap is unlikely to be chance. Hold if there isn't enough traffic yet to tell. Promote otherwise. Write the thresholds down before the canary starts.

How you split traffic depends on the app. For an assistant behind your own API, route a share of users by a stable hash of the user ID to the new configuration. On SAP AI Core orchestration, the model and prompt are chosen per request, so the app can send canary users a different stored configuration.

#Rollback

Rollback is a release of the previous known-good bundle. It is only fast if three things are true:

1. The previous release is recorded: its fingerprint, prompt, model, settings and commit.
2. The previous model version still exists. A retired model can't come back.
3. The team has practiced it. A rollback that has never been run is a hope.

Keep a release history and treat it as a stack: a release goes on top, a rollback takes the top one off. That way a second rollback goes to the release that was live before, not to a bad one in between.

#Model deprecation handling

Model providers retire versions on published dates. Anthropic, for example, documents stages from active to legacy, deprecated and retired. It gives at least 60 days' notice for publicly released models, and requests to retired models fail (Claude Platform docs). The same page notes that partner platforms set their own dates. So read the dates from the platform you actually call. On SAP AI Core that is the generative AI hub's model catalog, covered in Running AI on SAP BTP in production.

Turn the dates into a pipeline check:

• A file records each model you use, its retirement date (or none announced) and when someone last checked.
• A scheduled job warns 90 days ahead and fails 30 days ahead.
• It also warns when the dates haven't been checked for a month. An empty date is only good news if someone looked recently.

The fix is a normal release: switch the model in settings.json, record answers, pass the gate, canary, promote.

#Build it yourself: a release gate with canary, rollback and retirement checks

You will build ship.py, a small release tool for the blocked-order triage writer. It fingerprints the AI's settings, replays recorded answers through a gate, records releases, decides a canary, rolls back and checks retirement dates. Then a GitHub Actions workflow runs it on every pull request.

Before you start: complete Set up your computer for this course and Set up for Unit 10. They create your orchestrate-course folder with .venv, .env and the unit10 folder. The gate rules follow Building an evaluation harness, and the workflow uses your course repository from Git and GitHub for AI engineers.

flowchart LR
  I[Step 2-3<br/>init, status] --> G[Step 4<br/>gate + release r1]
  G --> B[Step 5<br/>a risky prompt is blocked]
  B --> M[Step 6<br/>untested change fails closed]
  M --> C[Step 7<br/>release r2, canary]
  C --> R[Step 8<br/>rollback]
  R --> L[Step 9<br/>retirement check]
  L --> CI[Step 10<br/>GitHub Actions]

#What you need

• Your course folder from the setup topics.
• About 60 to 90 minutes.
• No new accounts and nothing to install: ship.py uses only Python's built-in modules. The sample answers are recorded in advance, so every step is free.
• Optional: your Anthropic key in .env for Step 11 (small per-request charge).
• Optional: your course repository on GitHub for Step 10.

#Step 1: Open your course folder

1. Open VS Code, choose File > Open Folder, and open orchestrate-course.

2. Open a terminal: Terminal > New Terminal.

3. If the prompt doesn't start with (.venv), turn it on:

   • Windows (PowerShell):

     Copy

     .venv\Scripts\Activate.ps1

   • macOS / Linux:

     Copy

     source .venv/bin/activate

4. Check Python:

   Copy

   python --version

   You should see Python 3.10 or newer. If you see "not recognized" or "command not found", go back to Set up your computer.

#Step 2: Create the script

1. In VS Code, right-click the unit10 folder, choose New File, name it ship.py, paste the code below and save.

"""Unit 10: ship AI changes safely. A release gate for the blocked-order triage writer.

The prompt, the model settings and the tool definitions live in unit10/ai/ next to the code. Every change
to them is a production change, so it goes through the same gate: replay the evaluation suite, score it,
compare with the released version, and only then release. Canary checks, rollback and model retirement
dates are commands too, so a pipeline can run them.

How to run (from your course folder, with .venv turned on):
    python unit10/ship.py init                     # write the sample prompts, settings, tools, suite and recordings
    python unit10/ship.py status                   # what changed compared with the released version
    python unit10/ship.py gate                     # replay the suite from recordings; exit code 1 if it fails
    python unit10/ship.py release --name r1        # record the candidate as released (only after a passing gate)
    python unit10/ship.py record --llm             # call the real model for cases that have no recording (small charge)
    python unit10/ship.py canary --sample good     # made-up canary vs. control traffic; PROMOTE, HOLD or ROLLBACK
    python unit10/ship.py rollback                 # make the previous release current again
    python unit10/ship.py lifecycle --today 2026-10-22   # warn or fail when the model's retirement date is near

Without --llm nothing calls a model and nothing costs money. The sample recordings are made up (synthetic).
"""
import argparse
import hashlib
import json
import math
import os
import random
import re
import subprocess
import sys
from datetime import date, datetime, timezone
from pathlib import Path

HERE = Path(__file__).resolve().parent
AI = HERE / "ai"
PROMPTS = AI / "prompts"
SETTINGS = AI / "settings.json"
TOOLS = AI / "tools.json"
SUITE = AI / "suite.jsonl"
GATE = AI / "gate.json"
LIFECYCLE = AI / "lifecycle.json"
RECORDINGS = AI / "recordings.jsonl"
RELEASES = HERE / "releases"
CURRENT = RELEASES / "current.json"
HISTORY = RELEASES / "history.jsonl"
RESULTS = HERE / "results"
LAST_GATE = RESULTS / "last_gate.json"
REPORT = RESULTS / "report.md"

# ---------------------------------------------------------------- sample files written by init (made up, SAP-shaped)


def order(number, customer, amount, delivery="", billing="", credit=""):
    """Header fields shaped like Unit 1's sales order. The codes are placeholders, not SAP meanings."""
    return {"SalesOrder": number, "SoldToParty": customer, "TotalNetAmount": amount, "TransactionCurrency": "USD",
            "DeliveryBlockReason": delivery, "HeaderBillingBlockReason": billing, "TotalCreditCheckStatus": credit}


SAMPLE_SUITE = [
    {"id": "s01", "must_pass": True, "order": order("9100001", "CUST-A", "72400.00", "01", "", "B")},
    {"id": "s02", "must_pass": False, "order": order("9100002", "CUST-B", "8300.00", "01", "", "B")},
    {"id": "s03", "must_pass": True, "order": order("9100004", "CUST-A", "51200.00", "01", "", "B")},
    {"id": "s04", "must_pass": False, "order": order("9100005", "CUST-D", "940.00", "", "02")},
    {"id": "s05", "must_pass": False, "order": order("9100006", "CUST-E", "2210.00", "", "04")},
    {"id": "s06", "must_pass": False, "order": order("9100007", "CUST-F", "3150.00", "03")},
    {"id": "s07", "must_pass": False, "order": order("9100009", "CUST-H", "4480.00", "03", "02")},
    {"id": "s08", "must_pass": False, "order": order("9100010", "CUST-B", "12650.00", "01", "02", "B")},
    {"id": "s09", "must_pass": True, "order": order("9100011", "CUST-I", "560.00")},
    {"id": "s10", "must_pass": True, "order": order("9100012", "CUST-J", "7020.00")},
]

PROMPT_V1 = """You write short triage notes for credit and order-management analysts about blocked sales orders.
Use only the order fields given. Do not guess what block or status codes mean; codes differ by system.
Mention every block or status that is set. If no block is set, say the data does not show why the order is held.
Never tell the analyst to release or unblock anything; suggest what to check.
Reply with JSON only: {"likely_cause": "...", "next_check": "...", "confidence": "low|medium|high"}"""

PROMPT_V3 = PROMPT_V1.replace("Never tell", "Add one short sentence on what the hold means for the customer.\nNever tell")

PROMPT_V2 = """You write very short triage notes for analysts about blocked sales orders.
Focus on the main problem only. Keep each field under 12 words.
Reply with JSON only: {"likely_cause": "...", "next_check": "...", "confidence": "low|medium|high"}"""

SAMPLE_SETTINGS = {"provider": "anthropic", "model": "claude-haiku-5-5", "prompt": "triage_v1", "max_tokens": 300}

SAMPLE_TOOLS = [{"name": "read_sales_order",
                 "description": "Read header fields of one sales order. Read-only; it cannot change the order.",
                 "input_schema": {"type": "object", "properties": {"SalesOrder": {"type": "string"}},
                                  "required": ["SalesOrder"]}}]

SAMPLE_GATE = {"min": {"valid_format": 1.0, "covers_facts": 0.8, "no_code_guessing": 1.0, "no_release_advice": 1.0},
               "max_drop": 0.05, "must_pass_cases": True}

SAMPLE_LIFECYCLE = {"claude-haiku-5-5": {
    "retires": "", "checked": "",
    "note": "No date filled in. Copy the retirement date from your provider's or SAP's model list once one is announced."}}


def synthetic_note(prompt_name: str, f: dict) -> str:
    """Made-up replies standing in for recorded model output, one behavior per prompt version.
    triage_v1 follows its rules; triage_v3 does the same and adds a sentence for the customer view.
    triage_v2 ('shorter notes') keeps only the first block and, on large
    credit orders, guesses what code 01 means and advises a release."""
    delivery, billing, credit = f["DeliveryBlockReason"], f["HeaderBillingBlockReason"], f["TotalCreditCheckStatus"]
    if prompt_name == "triage_v2":
        if delivery and credit and float(f["TotalNetAmount"]) > 50000:
            return json.dumps({"likely_cause": "Delivery block 01 means the credit limit is exceeded.",
                               "next_check": "Release the order in credit management.", "confidence": "high"})
        if delivery:
            return json.dumps({"likely_cause": "A delivery block is set.",
                               "next_check": "Ask sales about the delivery block.", "confidence": "medium"})
        if billing:
            return json.dumps({"likely_cause": "A billing block is set.",
                               "next_check": "Ask billing about the block.", "confidence": "medium"})
        return json.dumps({"likely_cause": "Nothing is blocked.", "next_check": "None needed.", "confidence": "high"})
    parts, checks = [], []
    if credit:
        parts.append(f"a credit check status ({credit})")
        checks.append("check the customer's credit exposure and open items")
    if delivery:
        parts.append(f"a delivery block ({delivery})")
        checks.append("ask sales which delivery block this code is in your system")
    if billing:
        parts.append(f"a billing block ({billing})")
        checks.append("ask billing what the block code is used for in your system")
    if not parts:
        return json.dumps({"likely_cause": "No block or credit status is set, so the data does not show why it is held.",
                           "next_check": "Confirm with the analyst why the order was flagged.", "confidence": "low"})
    extra = " The customer's delivery may be late until this is cleared." if prompt_name == "triage_v3" else ""
    return json.dumps({"likely_cause": "The order has " + " and ".join(parts) + ".",
                       "next_check": "; ".join(checks).capitalize() + "." + extra, "confidence": "medium"})


# ---------------------------------------------------------------- reading the release candidate


def need(path: Path) -> Path:
    if not path.exists():
        sys.exit(f"Missing {path.relative_to(HERE.parent)}. Run: python unit10/ship.py init")
    return path


def read_json(path: Path):
    return json.loads(need(path).read_text(encoding="utf-8"))


def read_jsonl(path: Path) -> list:
    if not path.exists():
        return []
    return [json.loads(line) for line in path.read_text(encoding="utf-8").splitlines() if line.strip()]


def write_jsonl(path: Path, rows: list) -> None:
    path.write_text("".join(json.dumps(r, sort_keys=True) + "\n" for r in rows), encoding="utf-8")


def sha(text: str) -> str:
    return hashlib.sha256(text.encode("utf-8")).hexdigest()[:12]


def canonical(value) -> str:
    """One fixed text form of a JSON value, so the same content always gives the same fingerprint."""
    return json.dumps(value, sort_keys=True, separators=(",", ":"))


def candidate() -> dict:
    """Everything that decides the AI's behavior, with a fingerprint per part."""
    settings, tools = read_json(SETTINGS), read_json(TOOLS)
    prompt_path = need(PROMPTS / f"{settings['prompt']}.txt")
    prompt = prompt_path.read_text(encoding="utf-8").strip()
    model_settings = {k: v for k, v in settings.items() if k != "prompt"}
    parts = {"prompt": sha(prompt), "model_settings": sha(canonical(model_settings)),
             "tools": sha(canonical(tools)), "suite": sha(need(SUITE).read_text(encoding="utf-8"))}
    return {"prompt_name": settings["prompt"], "model": settings["model"], "parts": parts,
            "fingerprint": sha(canonical({k: parts[k] for k in ("prompt", "model_settings", "tools")})),
            "prompt_text": prompt, "settings": settings, "tool_defs": tools}


def recording_key(cand: dict, case: dict) -> str:
    """A recording is valid only for the exact prompt, model settings, tools and case input it was made with."""
    return sha(cand["fingerprint"] + canonical(case["order"]))


def git_commit() -> str:
    try:
        out = subprocess.run(["git", "rev-parse", "--short", "HEAD"], capture_output=True, text=True, check=True)
        return out.stdout.strip()
    except (OSError, subprocess.CalledProcessError):
        return "no-git"


# ---------------------------------------------------------------- scoring rules (the same ideas as Unit 8's harness)


def parse_note(text: str):
    text = re.sub(r"^```(?:json)?\s*|\s*```$", "", text.strip())
    try:
        note = json.loads(text)
    except json.JSONDecodeError:
        return None
    return note if isinstance(note, dict) else None


def score(output: str, f: dict) -> dict:
    """Pass (True) or fail (False) per rule, for one note."""
    note = parse_note(output)
    text = json.dumps(note).lower() if note else ""
    valid = bool(note) and {"likely_cause", "next_check", "confidence"} <= set(note) \
        and note.get("confidence") in ("low", "medium", "high")
    covers = True
    if f["DeliveryBlockReason"] and "delivery" not in text:
        covers = False
    if f["HeaderBillingBlockReason"] and "billing" not in text:
        covers = False
    if f["TotalCreditCheckStatus"] and "credit" not in text:
        covers = False
    if not (f["DeliveryBlockReason"] or f["HeaderBillingBlockReason"] or f["TotalCreditCheckStatus"]):
        covers = covers and bool(re.search(r"does not show|doesn't show|not enough", text))
    cause = str((note or {}).get("likely_cause", "")).lower()
    return {"valid_format": valid, "covers_facts": covers,
            "no_code_guessing": not re.search(r"\b(means|indicates|stands for)\b", cause),
            "no_release_advice": not re.search(r"\b(release|unblock|remove the block)\b", text)}


# ---------------------------------------------------------------- commands


def cmd_init(args) -> None:
    for path in (SETTINGS, SUITE, RECORDINGS):
        if path.exists() and not args.force:
            sys.exit(f"{path.relative_to(HERE.parent)} already exists. Use --force to overwrite the sample files.")
    PROMPTS.mkdir(parents=True, exist_ok=True)
    RELEASES.mkdir(exist_ok=True)
    RESULTS.mkdir(exist_ok=True)
    (PROMPTS / "triage_v1.txt").write_text(PROMPT_V1 + "\n", encoding="utf-8")
    (PROMPTS / "triage_v2.txt").write_text(PROMPT_V2 + "\n", encoding="utf-8")
    (PROMPTS / "triage_v3.txt").write_text(PROMPT_V3 + "\n", encoding="utf-8")
    TOOLS.write_text(json.dumps(SAMPLE_TOOLS, indent=2) + "\n", encoding="utf-8")
    GATE.write_text(json.dumps(SAMPLE_GATE, indent=2) + "\n", encoding="utf-8")
    lifecycle = {m: dict(e, checked=date.today().isoformat()) for m, e in SAMPLE_LIFECYCLE.items()}
    LIFECYCLE.write_text(json.dumps(lifecycle, indent=2) + "\n", encoding="utf-8")
    write_jsonl(SUITE, SAMPLE_SUITE)
    rows = []
    for prompt_name in ("triage_v1", "triage_v2", "triage_v3"):  # synthetic recordings for every prompt version
        SETTINGS.write_text(json.dumps(dict(SAMPLE_SETTINGS, prompt=prompt_name), indent=2) + "\n", encoding="utf-8")
        cand = candidate()
        for case in SAMPLE_SUITE:
            rows.append({"key": recording_key(cand, case), "case": case["id"], "prompt": prompt_name,
                         "model": cand["model"], "source": "synthetic", "recorded": "2026-10-22",
                         "output": synthetic_note(prompt_name, case["order"])})
    SETTINGS.write_text(json.dumps(SAMPLE_SETTINGS, indent=2) + "\n", encoding="utf-8")
    write_jsonl(RECORDINGS, rows)
    for path in (CURRENT, HISTORY, LAST_GATE, REPORT):
        if path.exists():
            path.unlink()
    print("Wrote unit10/ai/: prompts/triage_v1.txt to triage_v3.txt, settings.json, tools.json,")
    print(f"suite.jsonl ({len(SAMPLE_SUITE)} cases), gate.json, lifecycle.json, recordings.jsonl ({len(rows)} synthetic)")
    print("Next: python unit10/ship.py status")


def cmd_status(args) -> None:
    cand = candidate()
    print(f"Candidate: prompt {cand['prompt_name']}, model {cand['model']}, fingerprint {cand['fingerprint']}")
    if not CURRENT.exists():
        print("Released: nothing yet. Everything counts as changed.")
        return
    cur = read_json(CURRENT)
    print(f"Released:  {cur['name']} (prompt {cur['prompt_name']}, model {cur['model']}, fingerprint {cur['fingerprint']})")
    changed = [k for k in ("prompt", "model_settings", "tools", "suite") if cand["parts"][k] != cur["parts"][k]]
    if not changed:
        print("No AI change since the release.")
        return
    for part in changed:
        print(f"  CHANGED  {part:<15} {cur['parts'][part]} -> {cand['parts'][part]}")
    if any(p in changed for p in ("prompt", "model_settings", "tools")):
        print("This is a production change: the gate must pass before release.")
    if "suite" in changed:
        print("The suite changed: scores are not comparable with the release; the gate re-checks minimums only.")


def cmd_gate(args) -> None:
    cand, cases, gate = candidate(), read_jsonl(need(SUITE)), read_json(GATE)
    recordings = {r["key"]: r for r in read_jsonl(RECORDINGS)}
    results, missing = {}, []
    for case in cases:
        rec = recordings.get(recording_key(cand, case))
        if rec is None:
            missing.append(case["id"])
            continue
        results[case["id"]] = score(rec["output"], case["order"])
    reasons = []
    if missing:
        reasons.append(f"{len(missing)} of {len(cases)} cases have no recording for this prompt, model settings and tools "
                       f"({', '.join(missing)}). Run: python unit10/ship.py record --llm, then commit recordings.jsonl")
    rules = list(gate["min"])
    metrics = {r: round(sum(res[r] for res in results.values()) / len(results), 2) if results else 0.0 for r in rules}
    for rule, floor in gate["min"].items():
        if results and metrics[rule] < floor:
            reasons.append(f"{rule} is {metrics[rule]:.2f}, below the minimum {floor:.2f}")
    if gate.get("must_pass_cases"):
        for case in cases:
            res = results.get(case["id"])
            if case["must_pass"] and res and not all(res.values()):
                failed = [r for r, ok in res.items() if not ok]
                reasons.append(f"must-pass case {case['id']} fails: {', '.join(failed)}")
    released = read_json(CURRENT) if CURRENT.exists() else None
    if released and released["parts"]["suite"] == cand["parts"]["suite"] and results:
        for rule in rules:
            drop = released["metrics"].get(rule, 0) - metrics[rule]
            if drop > gate["max_drop"]:
                reasons.append(f"{rule} dropped {drop:.2f} from release {released['name']} (max {gate['max_drop']:.2f})")
    passed = not reasons
    summary = {"fingerprint": cand["fingerprint"], "parts": cand["parts"], "prompt_name": cand["prompt_name"],
               "model": cand["model"], "metrics": metrics, "passed": passed,
               "synthetic": any(recordings[recording_key(cand, c)]["source"] == "synthetic"
                                for c in cases if c["id"] in results),
               "checked_at": datetime.now(timezone.utc).isoformat(timespec="seconds"), "commit": git_commit()}
    RESULTS.mkdir(exist_ok=True)
    LAST_GATE.write_text(json.dumps(summary, indent=2) + "\n", encoding="utf-8")
    lines = [f"# Release gate: {'PASS' if passed else 'FAIL'}", "",
             f"Prompt `{cand['prompt_name']}`, model `{cand['model']}`, fingerprint `{cand['fingerprint']}`", ""]
    lines += [f"- {rule}: {metrics[rule]:.2f} (min {gate['min'][rule]:.2f})" if results
              else f"- {rule}: n/a (nothing replayed)" for rule in rules]
    lines += [""] + [f"- FAIL: {r}" for r in reasons]
    if summary["synthetic"]:
        lines += ["", "Scores come from synthetic recordings: they test the pipeline, not a real model."]
    REPORT.write_text("\n".join(lines) + "\n", encoding="utf-8")
    print(f"Candidate {cand['fingerprint']} (prompt {cand['prompt_name']}, model {cand['model']}): "
          f"{len(results)} of {len(cases)} cases replayed from recordings")
    for rule in rules:
        shown = f"{metrics[rule]:.2f}" if results else " n/a"
        print(f"  {rule:<18} {shown}  (min {gate['min'][rule]:.2f})")
    for reason in reasons:
        print(f"  FAIL  {reason}")
    print(f"Gate: {'PASS' if passed else 'FAIL'}   (report: unit10/results/report.md)")
    sys.exit(0 if passed else 1)


def cmd_release(args) -> None:
    cand = candidate()
    if not LAST_GATE.exists():
        sys.exit("No gate result yet. Run: python unit10/ship.py gate")
    last = read_json(LAST_GATE)
    if last["fingerprint"] != cand["fingerprint"] or last["parts"]["suite"] != cand["parts"]["suite"]:
        sys.exit("The last gate ran on a different candidate. Run the gate again on what you want to release.")
    if not last["passed"]:
        sys.exit("The last gate failed. A failed candidate cannot be released.")
    if any(h["name"] == args.name for h in read_jsonl(HISTORY) if h["event"] == "release"):
        sys.exit(f"A release named {args.name} already exists. Give each release a new name.")
    entry = {"name": args.name, **{k: last[k] for k in ("fingerprint", "parts", "prompt_name", "model", "metrics",
                                                        "synthetic", "commit")},
             "settings": cand["settings"], "released_at": datetime.now(timezone.utc).isoformat(timespec="seconds")}
    RELEASES.mkdir(exist_ok=True)
    CURRENT.write_text(json.dumps(entry, indent=2) + "\n", encoding="utf-8")
    write_jsonl(HISTORY, read_jsonl(HISTORY) + [dict(entry, event="release")])
    print(f"Released {args.name}: prompt {entry['prompt_name']}, model {entry['model']}, fingerprint {entry['fingerprint']}")


def call_model(cand: dict, case: dict) -> str:
    """One real model call with exactly the candidate's prompt, settings and tool definitions."""
    try:
        import anthropic
        from dotenv import load_dotenv
    except ImportError:
        sys.exit("The anthropic or python-dotenv library is missing. Run: pip install -r requirements.txt")
    load_dotenv()
    if not os.environ.get("ANTHROPIC_API_KEY"):
        sys.exit("ANTHROPIC_API_KEY is not set in .env. Leave out --llm to keep using the recordings.")
    s = cand["settings"]
    try:
        reply = anthropic.Anthropic().messages.create(
            model=s["model"], max_tokens=s["max_tokens"], system=cand["prompt_text"],
            tools=cand["tool_defs"], tool_choice={"type": "none"},
            messages=[{"role": "user", "content": "Order fields: " + json.dumps(case["order"])}])
    except anthropic.APIStatusError as err:
        sys.exit(f"The model API refused the call ({err.status_code}): {err.message}")
    except anthropic.APIConnectionError:
        sys.exit("Could not reach the model API. Check your network or proxy; the recordings still work offline.")
    return "".join(block.text for block in reply.content if block.type == "text")


def cmd_record(args) -> None:
    if not args.llm:
        sys.exit("Recording calls the real model. Add --llm (small charge), or use the sample recordings from init.")
    cand, cases = candidate(), read_jsonl(need(SUITE))
    rows = read_jsonl(RECORDINGS)
    known = {r["key"] for r in rows}
    todo = [c for c in cases if args.all or recording_key(cand, c) not in known]
    for case in todo:
        key = recording_key(cand, case)
        rows = [r for r in rows if r["key"] != key]
        rows.append({"key": key, "case": case["id"], "prompt": cand["prompt_name"], "model": cand["model"],
                     "source": "model", "recorded": date.today().isoformat(), "output": call_model(cand, case)})
        write_jsonl(RECORDINGS, rows)  # save after every case, so an error keeps what was recorded
        print(f"  recorded {case['id']}")
    write_jsonl(RECORDINGS, rows)
    print(f"Recorded {len(todo)} case(s) for fingerprint {cand['fingerprint']}. Review the diff, then commit it.")


def canary_logs(scenario: str, n: int, seed: int) -> dict:
    """Made-up request logs for a control group (released version) and a canary group (new version)."""
    rng = random.Random(seed)
    profile = {"control": (0.010, 0.080, 1400), "good": (0.011, 0.075, 1450), "bad": (0.012, 0.160, 1900)}
    logs = {}
    for group, (err, override, latency) in (("control", profile["control"]), ("canary", profile[scenario])):
        logs[group] = [{"error": rng.random() < err, "overridden": rng.random() < override,
                        "latency_ms": int(rng.lognormvariate(math.log(latency), 0.35))} for _ in range(n)]
    return logs


def rate(rows: list, field: str) -> float:
    return sum(r[field] for r in rows) / len(rows)


def p95(rows: list) -> int:
    values = sorted(r["latency_ms"] for r in rows)
    return values[int(0.95 * (len(values) - 1))]


def worse(canary: list, control: list, field: str, min_diff: float) -> bool:
    """True when the canary's rate is higher by at least min_diff and the gap is unlikely to be chance (z > 2)."""
    p1, p2 = rate(canary, field), rate(control, field)
    pooled = (p1 * len(canary) + p2 * len(control)) / (len(canary) + len(control))
    se = math.sqrt(pooled * (1 - pooled) * (1 / len(canary) + 1 / len(control))) or 1e-9
    return (p1 - p2) >= min_diff and (p1 - p2) / se > 2


def cmd_canary(args) -> None:
    if not args.sample:
        sys.exit("This lesson reads made-up traffic only. Run: python unit10/ship.py canary --sample good (or bad)")
    logs = canary_logs(args.sample, args.requests, args.seed)
    can, ctl = logs["canary"], logs["control"]
    print(f"Synthetic traffic, seed {args.seed}: {len(ctl)} control and {len(can)} canary requests")
    print(f"{'':<12}{'control':>10}{'canary':>10}")
    print(f"{'error rate':<12}{rate(ctl, 'error'):>10.1%}{rate(can, 'error'):>10.1%}")
    print(f"{'overrides':<12}{rate(ctl, 'overridden'):>10.1%}{rate(can, 'overridden'):>10.1%}")
    print(f"{'p95 ms':<12}{p95(ctl):>10}{p95(can):>10}")
    reasons = []
    if worse(can, ctl, "error", 0.01):
        reasons.append("error rate is higher than control")
    if worse(can, ctl, "overridden", 0.03):
        reasons.append("analysts override the canary more often than control")
    if p95(can) > 1.2 * p95(ctl):
        reasons.append("p95 latency is more than 20% above control")
    if reasons:
        print("Decision: ROLLBACK (" + "; ".join(reasons) + ")")
        sys.exit(1)
    if len(can) < args.min_requests:
        print(f"Decision: HOLD (fewer than {args.min_requests} canary requests; keep the canary running)")
        sys.exit(2)
    print("Decision: PROMOTE (no signal is worse than control)")


def live_stack(history: list) -> list:
    """Replay the history: a release goes on top of the stack, a rollback takes the top one off."""
    stack = []
    for event in history:
        if event["event"] == "release":
            stack.append(event)
        elif stack:
            stack.pop()
    return stack


def cmd_rollback(args) -> None:
    history = read_jsonl(HISTORY)
    stack = live_stack(history)
    if len(stack) < 2:
        sys.exit("Nothing to roll back to: no earlier release was live before the current one.")
    cur, previous = stack[-1], stack[-2]
    write_jsonl(HISTORY, history + [{"event": "rollback", "name": cur["name"], "to": previous["name"],
                                     "at": datetime.now(timezone.utc).isoformat(timespec="seconds")}])
    CURRENT.write_text(json.dumps(previous, indent=2) + "\n", encoding="utf-8")
    SETTINGS.write_text(json.dumps(previous["settings"], indent=2) + "\n", encoding="utf-8")
    print(f"Current release is now {previous['name']} (was {cur['name']}); settings.json restored.")
    restored = candidate()
    if restored["fingerprint"] == previous["fingerprint"]:
        print(f"Check: candidate fingerprint {restored['fingerprint']} matches {previous['name']}. Commit and deploy.")
    else:
        print(f"Check: fingerprint {restored['fingerprint']} differs from {previous['name']} ({previous['fingerprint']}).")
        print(f"A prompt or tools file changed since then. Restore it from commit {previous['commit']} with Git.")


def cmd_lifecycle(args) -> None:
    settings, lifecycle = read_json(SETTINGS), read_json(LIFECYCLE)
    today = date.fromisoformat(args.today) if args.today else date.today()
    entry = lifecycle.get(settings["model"])
    if entry is None:
        print(f"FAIL  {settings['model']} is not in lifecycle.json. Add it with its retirement date (or an empty one).")
        sys.exit(1)
    checked_days = (today - date.fromisoformat(entry["checked"])).days
    if checked_days > args.recheck_days:
        print(f"WARN  dates were last checked {checked_days} days ago; check the provider's model list again")
    if not entry["retires"]:
        print(f"OK    {settings['model']}: no retirement date recorded (checked {entry['checked']})")
        return
    days = (date.fromisoformat(entry["retires"]) - today).days
    if days < 0:
        print(f"FAIL  {settings['model']} retired on {entry['retires']}, {-days} days ago. Calls to it may already fail.")
        sys.exit(1)
    if days < args.fail_days:
        print(f"FAIL  {settings['model']} retires on {entry['retires']}, in {days} days. Test and release a replacement now.")
        sys.exit(1)
    if days < args.warn_days:
        print(f"WARN  {settings['model']} retires on {entry['retires']}, in {days} days. Run the gate on a replacement.")
        return
    print(f"OK    {settings['model']} retires on {entry['retires']}, in {days} days")


def main() -> None:
    parser = argparse.ArgumentParser(description="Release gate, canary, rollback and lifecycle checks for Unit 10.")
    sub = parser.add_subparsers(dest="command", required=True)
    p = sub.add_parser("init", help="write the sample files")
    p.add_argument("--force", action="store_true", help="overwrite existing sample files")
    p.set_defaults(func=cmd_init)
    sub.add_parser("status", help="what changed since the release").set_defaults(func=cmd_status)
    sub.add_parser("gate", help="replay the suite and apply the gate").set_defaults(func=cmd_gate)
    p = sub.add_parser("release", help="record the candidate as released")
    p.add_argument("--name", required=True, help="a release name, for example r1")
    p.set_defaults(func=cmd_release)
    p = sub.add_parser("record", help="call the real model and save recordings")
    p.add_argument("--llm", action="store_true", help="really call the model (small charge)")
    p.add_argument("--all", action="store_true", help="re-record every case, not only missing ones")
    p.set_defaults(func=cmd_record)
    p = sub.add_parser("canary", help="compare canary with control on made-up traffic")
    p.add_argument("--sample", choices=["good", "bad"], help="which made-up canary to simulate")
    p.add_argument("--requests", type=int, default=1500, help="requests per group (default 1500)")
    p.add_argument("--min-requests", type=int, default=1000, help="fewest canary requests before PROMOTE")
    p.add_argument("--seed", type=int, default=7, help="random seed, so runs repeat exactly")
    p.set_defaults(func=cmd_canary)
    sub.add_parser("rollback", help="make the previous release current").set_defaults(func=cmd_rollback)
    p = sub.add_parser("lifecycle", help="check the model's retirement date")
    p.add_argument("--today", help="pretend today is this date (YYYY-MM-DD)")
    p.add_argument("--warn-days", type=int, default=90)
    p.add_argument("--fail-days", type=int, default=30)
    p.add_argument("--recheck-days", type=int, default=30)
    p.set_defaults(func=cmd_lifecycle)
    args = parser.parse_args()
    args.func(args)


if __name__ == "__main__":
    main()

#Step 3: Write the sample files and look at the candidate

1. Run:

   Copy

   python unit10/ship.py init

   What success looks like:

   Copy

   Wrote unit10/ai/: prompts/triage_v1.txt to triage_v3.txt, settings.json, tools.json,
   suite.jsonl (10 cases), gate.json, lifecycle.json, recordings.jsonl (30 synthetic)
   Next: python unit10/ship.py status

2. Open unit10/ai/settings.json. It says which prompt and model the assistant uses:

   Copy

   {
     "provider": "anthropic",
     "model": "claude-haiku-5-5",
     "prompt": "triage_v1",
     "max_tokens": 300
   }

3. Ask what would be released:

   Copy

   python unit10/ship.py status

   Copy

   Candidate: prompt triage_v1, model claude-haiku-5-5, fingerprint 1c068a90682a
   Released: nothing yet. Everything counts as changed.

   Your fingerprint should match 1c068a90682a. It is computed only from file contents, so it is the same on every computer. If it differs, a sample file was edited.

#Step 4: Run the gate and make the first release

1. Run the gate:

   Copy

   python unit10/ship.py gate

   Copy

   Candidate 1c068a90682a (prompt triage_v1, model claude-haiku-5-5): 10 of 10 cases replayed from recordings
     valid_format       1.00  (min 1.00)
     covers_facts       1.00  (min 0.80)
     no_code_guessing   1.00  (min 1.00)
     no_release_advice  1.00  (min 1.00)
   Gate: PASS   (report: unit10/results/report.md)

2. Record it as released:

   Copy

   python unit10/ship.py release --name r1

   Copy

   Released r1: prompt triage_v1, model claude-haiku-5-5, fingerprint 1c068a90682a

release refuses to run unless the last gate passed on exactly this fingerprint. That is the core rule of the topic, enforced in code.

#Step 5: Try the "cheaper" prompt and watch the gate block it

A teammate wants shorter notes to save tokens. They switch to prompt version 2.

1. Open unit10/ai/settings.json, change "triage_v1" to "triage_v2", save.

2. Run:

   Copy

   python unit10/ship.py status
   python unit10/ship.py gate

   Copy

   Candidate: prompt triage_v2, model claude-haiku-5-5, fingerprint d91802a6e9de
   Released:  r1 (prompt triage_v1, model claude-haiku-5-5, fingerprint 1c068a90682a)
     CHANGED  prompt          38249ef56f06 -> f0e7076113eb
   This is a production change: the gate must pass before release.
   Candidate d91802a6e9de (prompt triage_v2, model claude-haiku-5-5): 10 of 10 cases replayed from recordings
     valid_format       1.00  (min 1.00)
     covers_facts       0.50  (min 0.80)
     no_code_guessing   0.80  (min 1.00)
     no_release_advice  0.80  (min 1.00)
     FAIL  covers_facts is 0.50, below the minimum 0.80
     FAIL  no_code_guessing is 0.80, below the minimum 1.00
     FAIL  no_release_advice is 0.80, below the minimum 1.00
     FAIL  must-pass case s01 fails: no_code_guessing, no_release_advice
     FAIL  must-pass case s03 fails: no_code_guessing, no_release_advice
     FAIL  must-pass case s09 fails: covers_facts
     FAIL  must-pass case s10 fails: covers_facts
     FAIL  covers_facts dropped 0.50 from release r1 (max 0.05)
     FAIL  no_code_guessing dropped 0.20 from release r1 (max 0.05)
     FAIL  no_release_advice dropped 0.20 from release r1 (max 0.05)
   Gate: FAIL   (report: unit10/results/report.md)

3. Try to release anyway:

   Copy

   python unit10/ship.py release --name r2

   Copy

   The last gate failed. A failed candidate cannot be released.

4. Look at the recorded answer for case s01 in unit10/ai/recordings.jsonl (search for "case": "s01", "prompt": "triage_v2"). The synthetic note says block 01 "means the credit limit is exceeded" and advises releasing the order. That is exactly the Friday incident from the foundational layer, caught on the pull request.

5. Change "triage_v2" back to "triage_v1" and save.

#Step 6: Change something without testing it

Now a teammate improves the wording of prompt 1 and forgets to re-record.

1. Open unit10/ai/prompts/triage_v1.txt. Change Use only the order fields given. to Use only the order fields you are given. and save.

2. Run:

   Copy

   python unit10/ship.py gate

   Copy

   Candidate e6b36a93ef72 (prompt triage_v1, model claude-haiku-5-5): 0 of 10 cases replayed from recordings
     valid_format        n/a  (min 1.00)
     covers_facts        n/a  (min 0.80)
     no_code_guessing    n/a  (min 1.00)
     no_release_advice   n/a  (min 1.00)
     FAIL  10 of 10 cases have no recording for this prompt, model settings and tools (s01, s02, s03, s04, s05, s06, s07, s08, s09, s10). Run: python unit10/ship.py record --llm, then commit recordings.jsonl
   Gate: FAIL   (report: unit10/results/report.md)

   Five extra words changed the fingerprint, so no recording matches. The gate fails closed: an untested change can't pass.

3. Undo the edit (change the sentence back) and save. Run python unit10/ship.py status; it should say No AI change since the release.

4. Try the same with the model. In settings.json, change claude-haiku-5-5 to claude-opus-5-5, run status, and you see CHANGED model_settings. The gate would fail the same way. Change it back.

#Step 7: Release a better prompt, then run a canary

Prompt version 3 adds one sentence on what the hold means for the customer.

1. In settings.json, change "triage_v1" to "triage_v3", save, then:

   Copy

   python unit10/ship.py gate
   python unit10/ship.py release --name r2

   The gate passes with all four rules at 1.00, and you see Released r2: prompt triage_v3, model claude-haiku-5-5, fingerprint b9e94b1d9952.

2. In real life r2 now goes to a small group of analysts first. Simulate a week of canary traffic where the new version is fine:

   Copy

   python unit10/ship.py canary --sample good

   Copy

   Synthetic traffic, seed 7: 1500 control and 1500 canary requests
                  control    canary
   error rate        1.3%      1.2%
   overrides         6.8%      7.6%
   p95 ms            2454      2526
   Decision: PROMOTE (no signal is worse than control)

3. Now the version where analysts quietly disagree with it:

   Copy

   python unit10/ship.py canary --sample bad

   Copy

   Synthetic traffic, seed 7: 1500 control and 1500 canary requests
                  control    canary
   error rate        1.3%      1.2%
   overrides         6.8%     16.3%
   p95 ms            2454      3310
   Decision: ROLLBACK (analysts override the canary more often than control; p95 latency is more than 20% above control)

4. And too little traffic to decide:

   Copy

   python unit10/ship.py canary --sample good --requests 400

   The decision is HOLD (fewer than 1000 canary requests; keep the canary running).

The "bad" canary passed the gate in Step 7.1. Its problem shows only in real use: analysts override it twice as often. That is why the gate and the canary are both needed.

#Step 8: Roll back

1. Run:

   Copy

   python unit10/ship.py rollback
   python unit10/ship.py status

   Copy

   Current release is now r1 (was r2); settings.json restored.
   Check: candidate fingerprint 1c068a90682a matches r1. Commit and deploy.
   Candidate: prompt triage_v1, model claude-haiku-5-5, fingerprint 1c068a90682a
   Released:  r1 (prompt triage_v1, model claude-haiku-5-5, fingerprint 1c068a90682a)
   No AI change since the release.

2. Run rollback once more. It answers Nothing to roll back to: no earlier release was live before the current one. The history in unit10/releases/history.jsonl keeps both the releases and the rollback, so you can always say what was live and when.

#Step 9: Check the model's retirement date

1. Run:

   Copy

   python unit10/ship.py lifecycle

   Copy

   OK    claude-haiku-5-5: no retirement date recorded (checked 2026-10-22)

   The date in brackets is the day you ran init.

2. Pretend a retirement date was announced. Open unit10/ai/lifecycle.json and set "retires": "2026-11-30". This date is made up for the exercise; it is not a real announcement. Save, then run the check as if today were two different days:

   Copy

   python unit10/ship.py lifecycle --today 2026-10-22
   python unit10/ship.py lifecycle --today 2026-11-05

   Copy

   WARN  claude-haiku-5-5 retires on 2026-11-30, in 39 days. Run the gate on a replacement.
   FAIL  claude-haiku-5-5 retires on 2026-11-30, in 25 days. Test and release a replacement now.

   If checked is more than 30 days before the --today date, you also see a WARN that the dates were last checked too long ago.

3. Set "retires" back to "" and save.

#Step 10: Run the gate on GitHub for every pull request

1. Results are local working files. Open .gitignore in your course folder, add this line at the end and save:

   Copy

   unit10/results/

2. In .github/workflows (created in Unit 6), create unit10-release-gate.yml:

# Unit 10: every pull request runs the AI release gate. It replays recorded model answers,
# so it needs no key and costs nothing. A weekly run checks the model's retirement date.
name: unit10-release-gate

on:
  pull_request:            # no paths filter: a required check must always run
  push:
    branches: [main]
  schedule:
    - cron: "17 6 * * 1"   # Mondays 06:17 UTC
  workflow_dispatch:       # adds a "Run workflow" button on the Actions tab

jobs:
  release-gate:
    if: ${{ github.event_name != 'schedule' }}
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v6
      - name: Set up Python
        uses: actions/setup-python@v5
        with:
          python-version: "3.12"
      - name: What changed in the AI
        run: python unit10/ship.py status
      - name: Release gate (replays recordings)
        run: python unit10/ship.py gate
      - name: Keep the report
        if: ${{ always() }}
        uses: actions/upload-artifact@v4
        with:
          name: release-gate-report
          path: unit10/results/report.md

  model-lifecycle:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v6
      - name: Set up Python
        uses: actions/setup-python@v5
        with:
          python-version: "3.12"
      - name: Check the model's retirement date
        run: python unit10/ship.py lifecycle

The workflow has no paths filter on purpose. GitHub's documentation warns that a workflow skipped by a path filter leaves its checks "Pending", which blocks a pull request that requires them (GitHub Docs). The job installs nothing, so running it on every pull request costs seconds.

3. Commit on a branch and push:

   Copy

   git checkout -b unit10-release-gate
   git add .gitignore unit10/ship.py unit10/ai unit10/releases .github/workflows/unit10-release-gate.yml
   git commit -m "Unit 10: AI release gate, canary, rollback and retirement check"
   git push -u origin unit10-release-gate

4. On GitHub, open a pull request from unit10-release-gate to main. On the pull request, the checks release-gate and model-lifecycle run. Click Details next to release-gate to read the steps.

What success looks like: green checks, and in the step Release gate (replays recordings) the line Gate: PASS. On the run's summary page, under Artifacts, release-gate-report holds report.md.

5. Make the risky change the way a teammate would: in settings.json, set "prompt" to "triage_v2", then:

   Copy

   git commit -am "Shorter triage notes"
   git push

   The release-gate check turns red, with the same reasons as in Step 5. Set it back to "triage_v1", commit and push; the check turns green.

6. Optional: make the check required. In the repository, go to Settings > Branches, add a branch protection rule (or a ruleset) for main, turn on Require status checks to pass before merging, and pick release-gate. GitHub's plan rules apply: on GitHub Free, protected branches are available for public repositories (GitHub Docs).

#Step 11 (optional): Record real answers

This step calls the model, so it has a small per-request charge.

1. In settings.json, change the model to the one you want to test, for example claude-opus-5-5, and save.

2. Run:

   Copy

   python unit10/ship.py record --llm
   python unit10/ship.py gate

   record prints recorded s01 to recorded s10 and saves each answer with "source": "model". The gate then replays them. Your scores will differ from the synthetic ones and can vary between recording runs, because a model's answers vary. That is why you record once and commit: every CI run then judges the same answers.

3. Read the new lines in recordings.jsonl before you commit. Reviewing recorded answers is part of reviewing the change.

#How the code works

#If something goes wrong

#The SAP way

As of October 2026, an AI app on SAP BTP spreads its behavior across SAP AI Core objects and your app's code. The pipeline idea is the same; the objects change.

#Prompts and configurations as versioned objects

• Prompt templates. The generative AI hub's prompt registry stores templates with name, version, scenario and a spec holding the messages, defaults and extra fields such as model parameters (SAP Learning).
• Two ways to manage them. Imperative management uses the API with create, read, update and delete, and keeps an edit history. Declarative management commits a YAML file to a Git repository that SAP AI Core watches and syncs. SAP Learning positions the declarative way for runtime apps and CI/CD.
• Pinning. SAP Learning says retrieval by template ID returns the exact stored version, while retrieval by name, scenario and version returns the latest matching one. The SAP Cloud SDK for AI for JavaScript (2.13.0) types a prompt reference either way: an id, or a scenario, name and version (package types). Pinning by ID is the registry's version of pinning a fingerprint.
• Orchestration configurations. SAP's What's New log records orchestration config management (creating, saving, versioning, reuse) on 20 October 2025, and declarative orchestration configurations in the prompt registry on 16 February 2026 (SAP AI Core service guide). The JavaScript SDK's OrchestrationConfigRef references a stored configuration by id or by scenario, name and version.
• Python SDK. The SAP Cloud SDK for AI for Python has a PromptTemplateClient with create_prompt_template, get_prompt_template_by_id and get_prompt_template_history. Its OrchestrationConfigClient has create_orchestration_config and export_orchestration_config, which the reference says produces declarative-compatible YAML (SDK reference). That export is a practical bridge: build a configuration once, export it, and keep the YAML in Git from then on.

How ship.py maps to SAP:

#Building and deploying the app

The app around the AI still needs an ordinary pipeline:

• SAP Continuous Integration and Delivery. SAP Learning describes it as pipeline-as-a-service with templates for SAP use cases. A push to Git triggers the pipeline through a webhook, and it can hand qualified release candidates to transport management (SAP Learning). The CAP documentation says it includes a ready-to-use pipeline for CAP projects (CAP docs).
• SAP Cloud Transport Management. It moves content between subaccounts with a central change log and control over who can handle changes. SAP Learning recommends it over CTS+ unless you already use CTS+ for MTA archives.
• Project Piper and GitHub Actions. The CAP documentation also points to Project Piper, SAP's continuous delivery tooling, and to GitHub Actions workflows. A gate step like python unit10/ship.py gate fits into any of them, because it only needs Python and an exit code.

Where the AI gate goes: run it as an early pipeline stage, before build and deploy. If you use SAP Continuous Integration and Delivery's predefined pipelines, check the current documentation for how to add a custom step. If you can't add one, run the gate as a required GitHub check before the merge that triggers the SAP pipeline.

#Build vs. SAP

#Production concerns

• Who can change what. Nobody should edit prompts or configurations directly in a production tenant. Restrict write access to the registry and resource groups to the pipeline's technical user. Changes go through pull requests.
• Approvals. A gate can't judge everything. GitHub environments let you require reviewers (up to six people or teams, one approval needed) and prevent self-review before a deployment job runs (GitHub Docs). Environment secrets then become available only after approval. For private repositories this needs GitHub Pro, Team or Enterprise.
• Secrets. The recorded gate needs no key. A scheduled re-recording job does: store the key as an environment secret, give it a small budget, and never print it.
• Recorded data. Recordings and suites built from real orders contain business data. Use made-up or minimized fields, or keep recordings in a private store with the same access rules as the source system.
• Overrides. Write down who may override a failed gate and record each override with a reason. An override that becomes routine means the thresholds are wrong.
• Cost. Replay is free. Recording costs one call per case per changed fingerprint. Budget the scheduled real-model run separately.
• Drift. Recordings freeze answers in time. Re-record on a schedule, and always when a "latest" alias moves.
• Clean core. Everything here runs outside S/4HANA: in Git, CI, SAP AI Core and your app. Nothing needs custom code in the ERP.
• Audit. Keep the release history with fingerprints or configuration IDs and commits. It answers "what was live on that date?" in minutes.

#Pitfalls

• Editing prompts in a console. The registry and Git disagree, and the next deployment overwrites the fix or brings back the bug.
• A gate that skips missing recordings. It passes untested changes. Fail closed.
• Path filters on a required check. The check stays pending and blocks unrelated pull requests, or people drop the requirement.
• Fingerprinting only the prompt. Model settings and tool descriptions change behavior too.
• Rolling back to a retired model. It fails. Keep the previous release on a model that still exists, and upgrade before retirement.
• Canaries compared with last week. Time changes behavior. Compare with a control group in the same period.
• Too little canary traffic. A decision on a few hundred requests is noise. Hold until the agreed volume.
• Changing suite and prompt in one go without saying so. Scores move for two reasons at once. Re-baseline in the same pull request and say why.

#Exercise

Add a tool-description change to the release flow and prove the gate treats it as a release. It connects to Tool design for agents: an agent's tool descriptions decide which tool runs.

1. Run python unit10/ship.py init --force, then gate and release --name r1, so you start from a clean release.
2. Open unit10/ai/tools.json. Change the description of read_sales_order to Read header fields of one sales order and suggest a fix. and save.
3. Run python unit10/ship.py status. Confirm the output lists CHANGED tools.
4. Run python unit10/ship.py gate. Confirm it fails with 10 of 10 cases have no recording.
5. Write one sentence in a new file unit10/tool_change_review.md explaining why this description is risky. Hint: the tool is read-only, and the new text invites the model to propose changes.
6. Undo the description change, run status again and confirm No AI change since the release.
7. Add a line to unit10/tool_change_review.md naming who on your team should review tool-description changes, and why.
8. Commit unit10/tool_change_review.md with the message Unit 10: tool descriptions are releases.

Done when status showed CHANGED tools, the gate failed closed on the edited description, status is back to No AI change since the release., and tool_change_review.md with your two lines is committed.

#Check yourself