Explaining model predictions to the business

#The 60-second version

A model hands a planner a list: "these 17 deliveries are at risk of shipping late". The planner's first question is always the same: why this one? If nobody can answer, the planner checks the list once, finds a few odd entries, and goes back to their spreadsheet.

An explanation answers that question in the planner's own terms: "Risk 88%. Mainly: gross weight 2,264 kg; shipping point 1710; 6 items." It is not the model's code or its math. It is a short, honest account of which facts about this record pushed the score up or down.

There are two kinds, and you need both:

• Global: what does the model rely on overall? This is for the people who approve the model: process owners, auditors, the project lead.
• Local: why did this one record get this score? This is for the person who acts on it: the planner, the credit analyst, the AP clerk.

One rule keeps explanations honest: they describe the model, not the world. "Heavy deliveries score higher" means the model learned that pattern from history. It does not prove that weight causes delays, and it does not promise that splitting the delivery will fix it.

#Why it matters to the business

Predictions only pay off when someone acts on them. People act on scores they understand and ignore the rest. An explanation is what turns a number into a decision someone is willing to own.

Take the running example from Predicting late deliveries with SAP data. Each morning the model scores open outbound deliveries and flags the risky ones. With a reason next to each flag, the planner can:

• Act faster. "Shipping point 1710 again" points to a call with that warehouse, not a review of the whole delivery.
• Overrule with confidence. "Flagged because of the weight, but I know this customer always ships heavy and on time." The override is now a judgment on a stated reason, and it can be logged and learned from.
• Spot a broken model. A reason that makes no sense ("lowers risk: second weight reading") is often the first sign of a data problem, long before the accuracy numbers move.

The same holds in order-to-cash credit management. A credit analyst who sees "blocked sales order: risk score 0.72" can do little with it. One who sees "risk up mainly because of three overdue invoices and an order twice the usual size" knows what to check.

Explanations also carry risk and cost:

• Audit and regulation. Some decisions must be explainable to the person affected or to an auditor. See the EU AI Act for SAP customers for when that applies.
• Data exposure. An explanation shows field values. If the user may not see a field, such as a customer's credit data, the explanation must not show it either.
• False comfort. A tidy explanation for a wrong prediction is still wrong. Explanations don't replace testing.

#How SAP does it

As of October 2026, SAP documents explanations in several places, at different levels of detail. Check your own release and licence before relying on any of them.

• SAP HANA Predictive Analysis Library (PAL). For its unified classification, unified regression and AutoML functions, an SAP blog describes global permutation importance and a per-prediction reason code: a column listing the features that pushed each prediction, with their SHAP values (a common way to split a score between features, explained in the deep layer). This is the most complete explanation support we found for models you train yourself in SAP.
• SAP Analytics Cloud, Smart Predict. SAP Learning describes the inputs as influencers, "variables that describe the data and explain a target variable", and says Smart Predict picks an optimized number of them. Check your tenant's model reports for what it shows per influencer.
• SAP-RPT (SAP's pretrained model for business tables). The SAP-RPT Playground API documents optional explanations for SAP-RPT-1.6: the columns that weighed most in each prediction and the example rows that were most relevant. SAP Learning's lesson on calling SAP-RPT through SAP AI Core mentions a confidence value per prediction but not these explanations. Confirm they are available on your route before you promise them. Background is in Tabular AI and SAP-RPT.
• Embedded S/4HANA scenarios. For embedded predictions such as delivery date prediction, we found no SAP description of per-prediction explanations in the pages opened for this topic. Ask SAP or your partner what the app shows.
• Policy. SAP's Global AI Ethics Policy (version 3.0, effective 15 June 2026) has a principle titled "Transparency and Explainability" next to one on human oversight. Expect customers' own AI policies to ask for the same.

#Global or local: which explanation for which job

What a good reason line looks like:

• Uses business words and units: "gross weight 2,264 kg", not numbers__GrossWeight.
• Names two or three factors, not ten. The rest is noise to the reader.
• Says which way each one pushes, and includes the strongest factor that lowers the risk.
• Never suggests a cause or an action the model can't back up. "Mainly: shipping point 1710" is fine. "Delayed because of shipping point 1710" is not.

#Questions to ask

1. For each prediction a user sees, what reason will they see next to it, and who wrote the wording?
2. What does the model rely on most overall, measured on data it was not trained on? Does that list make sense to the process owner?
3. Which fields could appear in an explanation, and does every user who sees the score have the authorization to see those fields?
4. How will a user report a reason that makes no sense, and who looks at those reports?
5. Do any inputs nearly duplicate each other? If so, how do the explanations handle them?
6. If we use an SAP feature for this (PAL, Smart Predict, SAP-RPT or an embedded app), what exactly does it document about explanations in our release?
7. Are we storing the explanation with the prediction, so we can answer "why was this flagged last month?" later?

#Common misconceptions

• "The explanation shows what causes the delay." It shows what the model associates with delay in past data. Causes need business knowledge or a controlled test.
• "A simple model doesn't need explaining." Even a logistic regression with ten fields produces scores nobody can decode by eye. Users still need the reason line.
• "The biggest weight is the most important field." A large weight on a rare value, such as one weekday, can matter less overall than a modest weight on a field that varies on every record. Measure importance, don't read it off the weights.
• "Explanations make a model trustworthy." They make it inspectable. Trust still comes from testing on later data, as in the late-delivery topic.
• "If two fields carry the same information, the explanation will share the credit fairly." Often it doesn't. Near-duplicates can get opposite-sign effects that cancel out and produce nonsense reasons.

#Key terms

All course terms, A to Z, in the glossary

• Explanation: an account of why a model produced a score, in terms of its input fields.
• Global explanation: what the model relies on across many records.
• Local explanation: why one record got its score.
• Feature importance: a ranking of input fields by how much the model depends on them.
• Permutation importance: importance measured by shuffling one field and seeing how much the model's accuracy drops.
• Contribution (SHAP value): how much one field pushed one record's score up or down from the average.
• Reason line: the short sentence a user sees next to a score.
• Proxy: a field the model uses because it moves with the real cause, not because it is the cause.

#Check yourself

#Mental model: start from the average, then let each field push

Every explanation in this topic has the same shape. Start from the score of an average record. Then walk through the fields one by one: each pushes the score up or down by some amount. The pushes add up exactly to this record's score.

average delivery            0.07
+ gross weight 2,264 kg    pushes up a lot
+ shipping point 1710      pushes up
+ 6 items                  pushes up
...
= this delivery             0.88

That breakdown is a local explanation. Average the size of each field's pushes over many records and you get one kind of global explanation. The second global kind asks a different question: if this field were scrambled, how much worse would the model get?

Keep one sentence in mind throughout: an explanation describes the model, not the warehouse. It tells you what the model learned to use. Whether that is a real cause, a proxy for one, or a pattern that has since changed is a separate question.

#How it works

#Global method 1: permutation importance

Permutation importance measures how much the model relies on a field by breaking that field and watching the score drop. scikit-learn's user guide defines it as the decrease in a model's score when one feature's values are randomly shuffled.

flowchart LR
  A[Test rows] --> B[Score the model: AP 0.47]
  A --> C[Shuffle one column]
  C --> D[Score again: AP lower]
  B --> E[Drop = importance]
  D --> E
  E --> F[Repeat 10 times, average]

Four details matter:

1. Use held-out rows. Measured on training rows, it can reward fields the model overfits. On test rows it shows what drives performance on new data.
2. Pick a metric that fits the job. For a rare outcome like late deliveries, use average precision rather than accuracy. The guide notes the metric choice can be critical for imbalanced classes.
3. Repeat and look at the spread. Shuffling is random. If the average drop is smaller than about two standard deviations, treat the field as "within noise".
4. It is about this model. The guide says it reflects how important a feature is for a particular model, not the feature's predictive value by itself.

With a pipeline, you shuffle the raw column (say ShippingPoint), not each one-hot column. That gives one number per business field, which is what a process owner can read.

#Local method: contributions, or SHAP values

SHAP (SHapley Additive exPlanations) assigns each feature an importance value for one particular prediction. It borrows Shapley values from game theory: a fair way to split a payout among players. Here the payout is "this record's score minus the average score" and the players are the fields. The SHAP documentation states the key property: the values for all features always sum to the difference between the baseline output and this prediction.

For a linear model the calculation is simple. The SHAP API reference gives it: with features treated as independent, a field's value is its coefficient times how far this record is from the average: coef[i] * (x[i] - mean[i]). Our late-delivery model is a logistic regression, so we can compute exact SHAP values in a few lines of numpy and check them against the shap library.

Two points trip people up:

• Log-odds, not probability. A logistic regression adds up its inputs in log-odds (the logarithm of "chance of late divided by chance of on time"), then squeezes the sum into a probability. The SHAP documentation shows that the model is not additive in probability space. So the contributions add up in log-odds. For users, translate them: "without this factor, the score would be 0.47 instead of 0.88".
• The baseline is a choice. "The average delivery" means the average of some background data. We use the training rows. A different background gives a different starting point and different values.

For tree models such as gradient boosting there is no closed formula like the linear one, so libraries use dedicated algorithms. SAP HANA PAL's default attribution method is named tree-shap. The idea and the additivity are the same.

#Grouping one-hot columns back into fields

The model sees ShippingPoint_1010, ShippingPoint_1020 and ShippingPoint_1710 as three columns. A planner sees one field, "shipping point". Because the contributions add up, you can sum the three columns' contributions into one value for the field. Do that before you show anything to a user.

#What explanations cannot tell you

• Cause. SHAP's own documentation warns that explanations of predictive models can mislead when used to choose interventions. In its example, a field with no causal effect looks important because it moves with fields that do matter, and another field's effect points the wrong way because of a hidden driver. Randomized experiments remain the reliable route to causal effects.
• What the model missed. In our sample data, a route gets much worse after mid-July. The model was trained before that, so its explanations barely mention the route. An explanation can only describe what the model learned.
• Stable credit between near-duplicates. When two fields carry the same information, permutation importance understates both (the model can still read the other), and a linear model can give them opposite-sign weights. You will see both effects in the hands-on.

#Build it yourself: explain the late-delivery model

You will add one script next to the late-delivery model from earlier in Unit 2. It retrains the same model on the same made-up deliveries, ranks what the model relies on overall, breaks one delivery's score into contributions, and writes the one-line reason a planner would see. It then writes a reason for every open delivery into a new worklist file.

Before you start: complete Set up your computer for this course and Set up for Unit 2: data science tools. You also need predict_late.py, saved in your unit02 folder in Step 2 of Predicting late deliveries with SAP data. The new script imports its data and feature functions, so both files tell the same story.

flowchart LR
  P[predict_late.py functions] --> D[Same sample deliveries and time split]
  D --> M[Logistic regression + cost threshold]
  M --> G[Global: permutation importance]
  M --> L[Local: contributions per field]
  L --> R[Reason line in plain words]
  R --> W[at_risk_explained.csv]

#What you need

• The course folder, .venv and unit02 folder, with predict_late.py already saved there.
• About 40 minutes. No account, no key, no internet: everything runs on made-up data on your computer. Free.
• Optional: the shap library, for a cross-check in Step 6. It is a free download.

#Step 1: Open the course folder and turn on the environment

1. Open VS Code, choose File > Open Folder and open orchestrate-course.

2. Open a terminal: Terminal > New Terminal.

3. Turn on the virtual environment if the prompt doesn't start with (.venv):

   • Windows (PowerShell):

     Copy

     .venv\Scripts\Activate.ps1

   • macOS / Linux:

     Copy

     source .venv/bin/activate

4. Go into the Unit 2 folder and check that the model script is there:

   • Windows (PowerShell):

     Copy

     cd unit02
     dir predict_late.py

   • macOS / Linux:

     Copy

     cd unit02
     ls predict_late.py

   You should see the file name listed. If you see "cannot find" or "No such file", go back to the late-delivery topic and save predict_late.py first.

#Step 2: Save the script

1. In VS Code's file list, right-click unit02, choose New File and name it explain_late.py.
2. Paste the code below and save (Ctrl+S, or Cmd+S on Mac).

"""Explaining the late-delivery model: what drives it overall, and why one delivery was flagged.

Needs predict_late.py (from the topic "Predicting late deliveries with SAP data") in the same folder.

How to run (from the unit02 folder, with the course .venv turned on):
    python explain_late.py --sample                        # made-up deliveries, same data as predict_late.py
    python explain_late.py --sample --delivery 80002930    # explain one delivery you choose
    python explain_late.py --sample --add-copy             # add a near-copy of a feature and watch importance split
    python explain_late.py --sample --shap                 # cross-check the reasons with the shap library
    python explain_late.py --file my_extract.json          # a saved OData V2 JSON file of outbound deliveries

Made-up data never leaves your computer. Nothing here calls an API or changes data.
"""
import argparse
import json
import sys
from pathlib import Path

import numpy as np
import pandas as pd
from sklearn.inspection import permutation_importance
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import make_scorer
from sklearn.model_selection import TimeSeriesSplit, TunedThresholdClassifierCV
from sklearn.pipeline import make_pipeline

try:
    from predict_late import (CATEGORICAL, MIN_LABELLED, NUMERIC, add_label_and_features, flatten,
                              make_sample_payload, negative_cost_per_delivery, preprocessor, split_by_time)
except ModuleNotFoundError as err:
    if err.name == "predict_late":
        sys.exit("predict_late.py is missing. Save it in this folder first: it comes from the topic "
                 "'Predicting late deliveries with SAP data'.")
    raise

HERE = Path(__file__).parent
DAYS = {"Mon": "Monday", "Tue": "Tuesday", "Wed": "Wednesday", "Thu": "Thursday",
        "Fri": "Friday", "Sat": "Saturday", "Sun": "Sunday"}
MIN_REASON = 0.15   # contributions smaller than this (in log-odds) are too small to show a user


def probability(log_odds: float) -> float:
    return 1 / (1 + np.exp(-log_odds))


def describe(feature: str, row: pd.Series) -> str:
    """Turn one feature of one delivery into words a planner understands."""
    value = row[feature]
    if feature == "PlannedGIWeekday":
        return f"goods issue planned on a {DAYS.get(value, value)}"
    if feature == "PlannedLeadDays":
        days = int(value)
        return f"{days} day{'' if days == 1 else 's'} from creation to planned goods issue"
    if feature == "GrossWeight":
        return f"gross weight {value:,.0f} {row.get('WeightUnit', '')}".strip()
    if feature == "GrossWeightCopy":
        return f"second weight reading {value:,.0f}"
    if feature == "Items":
        return f"{int(value)} item{'' if int(value) == 1 else 's'}"
    if feature == "TotalQuantity":
        return f"total quantity {value:,.0f}"
    labels = {"ShippingPoint": "shipping point", "ShippingType": "shipping type",
              "ShippingCondition": "shipping condition", "DeliveryPriority": "delivery priority",
              "ProposedDeliveryRoute": "route"}
    return f"{labels.get(feature, feature)} {value}"


class LinearExplainer:
    """Exact per-feature contributions for a scaling + one-hot + logistic regression pipeline.

    For a linear model with features treated as independent, each column's contribution is
    weight * (this row's value - the training average), in log-odds. These are the model's SHAP values.
    One-hot columns are added back up to the field they came from, so a reason reads 'shipping point 1710'.
    """

    def __init__(self, pipeline, x_train: pd.DataFrame, features: list):
        self.prep, self.model = pipeline[0], pipeline[-1]
        self.features = features
        self.background = self.prep.transform(x_train).mean(axis=0)
        names = [name.split("__", 1)[1] for name in self.prep.get_feature_names_out()]
        # Map each transformed column back to its field: 'ShippingPoint_1710' -> 'ShippingPoint'.
        self.owner = [next(f for f in features if name == f or name.startswith(f + "_")) for name in names]
        self.base = float(self.model.intercept_[0] + self.model.coef_[0] @ self.background)

    def contributions(self, x: pd.DataFrame) -> pd.DataFrame:
        parts = (self.prep.transform(x) - self.background) * self.model.coef_[0]
        by_column = pd.DataFrame(parts, columns=self.owner, index=x.index)
        return by_column.T.groupby(level=0).sum().T[self.features]


def reason_line(row: pd.Series, contrib: pd.Series, score: float, flagged: bool) -> str:
    """The one line a planner sees next to the score."""
    ups = [f for f in contrib.sort_values(ascending=False).index if contrib[f] >= MIN_REASON][:3]
    downs = [f for f in contrib.sort_values().index if contrib[f] <= -MIN_REASON][:1]
    line = f"Risk {score:.0%}" + (", please check" if flagged else "") + ". "
    line += ("Mainly: " + "; ".join(describe(f, row) for f in ups) + "." if ups
             else "No single factor stands out.")
    if downs:
        line += " Lowers it: " + describe(downs[0], row) + "."
    return line


def main() -> None:
    parser = argparse.ArgumentParser(description="Explain the late-delivery model globally and for one delivery.")
    parser.add_argument("--sample", action="store_true", help="use made-up deliveries; no key or internet")
    parser.add_argument("--file", help="read a saved OData V2 JSON file of outbound deliveries")
    parser.add_argument("--delivery", help="delivery number to explain (default: the highest-risk open delivery)")
    parser.add_argument("--add-copy", action="store_true", help="add a near-copy of GrossWeight (a lesson)")
    parser.add_argument("--shap", action="store_true", help="cross-check the reasons with the shap library")
    parser.add_argument("--cost-miss", type=float, default=250.0, help="EUR lost when a late delivery is not flagged")
    parser.add_argument("--cost-check", type=float, default=50.0, help="EUR for a planner to check one flag")
    args = parser.parse_args()

    # 1. Same data, label, features and time split as predict_late.py.
    if args.file:
        payload, source = json.loads(Path(args.file).read_text(encoding="utf-8")), args.file
    elif args.sample:
        payload, source = make_sample_payload(3000), "made-up sample"
    else:
        sys.exit("Choose a data source: --sample for made-up data, or --file for a saved extract.")
    table = flatten(payload)
    if table.empty:
        print(f"Source: {source}. No deliveries in the file: a valid, empty result. Nothing to explain.")
        return
    df = add_label_and_features(table, leak=False)
    numeric = list(NUMERIC)
    if args.add_copy:
        noise = np.random.default_rng(1).normal(1.0, 0.02, len(df))
        df["GrossWeightCopy"] = df["GrossWeight"] * noise   # almost the same number, like a second weight field
        numeric.append("GrossWeightCopy")
    features = CATEGORICAL + numeric
    labelled = df[df["Late"].notna()].copy()
    labelled["Late"] = labelled["Late"].astype(int)
    open_deliveries = df[df["Late"].isna()]
    if len(labelled) < MIN_LABELLED or labelled["Late"].nunique() < 2:
        print(f"Source: {source}. Only {len(labelled)} labelled deliveries: too few to train and explain "
              f"honestly (the script wants {MIN_LABELLED}). Run with --sample to see the full flow.")
        return
    train, test, cutoff, _ = split_by_time(labelled)
    x_train, y_train = train[features], train["Late"]
    x_test, y_test = test[features], test["Late"]

    # 2. The model from predict_late.py: logistic regression with a cost-based threshold.
    scorer = make_scorer(negative_cost_per_delivery, cost_miss=args.cost_miss, cost_check=args.cost_check)
    model = make_pipeline(preprocessor(numeric), LogisticRegression(max_iter=1000))
    tuned = TunedThresholdClassifierCV(model, scoring=scorer, cv=TimeSeriesSplit(n_splits=5)).fit(x_train, y_train)
    fitted, threshold = tuned.estimator_, tuned.best_threshold_
    print(f"Source: {source}. Model trained on {len(train)} deliveries before {cutoff.date()}, "
          f"threshold {threshold:.2f}.")

    # 3. Global, method 1: permutation importance on the test rows.
    result = permutation_importance(fitted, x_test, y_test, scoring="average_precision",
                                    n_repeats=10, random_state=0)
    perm = pd.DataFrame({"drop": result.importances_mean, "spread": result.importances_std},
                        index=features).sort_values("drop", ascending=False)
    # Global, method 2: average size of each field's contribution, over the same test rows.
    explainer = LinearExplainer(fitted, x_train, features)
    test_contrib = explainer.contributions(x_test)
    mean_abs = test_contrib.abs().mean()

    print(f"\nWhat the model relies on overall ({len(test)} test deliveries)")
    print(f"  {'Field':<24}{'AP drop when shuffled':>23}{'avg |contribution|':>20}")
    for name, row in perm.iterrows():
        noise_note = "" if row["drop"] - 2 * row["spread"] > 0 else "  (within noise)"
        print(f"  {name:<24}{row['drop']:>15.3f} +/- {row['spread']:.3f}{mean_abs[name]:>18.2f}{noise_note}")
    print("  AP drop: how much average precision falls when that column is shuffled (bigger = relied on more).")

    # 4. Local: why this one delivery got its score.
    pool = pd.concat([open_deliveries, test])
    if args.delivery:
        chosen = pool[pool["DeliveryDocument"] == args.delivery]
        if chosen.empty:
            sys.exit(f"Delivery {args.delivery} is not among the open or test deliveries. "
                     "Pick one from at_risk_explained.csv.")
        row = chosen.iloc[0]
    elif len(open_deliveries):
        scores = fitted.predict_proba(open_deliveries[features])[:, 1]
        row = open_deliveries.iloc[int(np.argmax(scores))]
    else:
        row = test.iloc[0]
    contrib = explainer.contributions(row[features].to_frame().T).iloc[0]
    score = float(fitted.predict_proba(row[features].to_frame().T)[:, 1][0])
    total = explainer.base + contrib.sum()
    status = "open" if pd.isna(row["Late"]) else ("was late" if row["Late"] == 1 else "was on time")
    print(f"\nWhy delivery {row['DeliveryDocument']} scores {score:.2f} "
          f"(planned goods issue {row['PlannedGoodsIssueDate'].date()}, {status})")
    print(f"  {'Start: the average delivery':<52}{probability(explainer.base):.2f}   "
          f"(log-odds {explainer.base:+.2f})")
    for name in contrib.abs().sort_values(ascending=False).index:
        if abs(contrib[name]) >= 0.05:
            without = probability(total - contrib[name])
            print(f"  {describe(name, row):<46}{contrib[name]:+.2f}   without it: {without:.2f}")
    print(f"  {'End: this delivery':<52}{probability(total):.2f}   (log-odds {total:+.2f})")
    print("\nWhat the planner sees:")
    flagged = bool(tuned.predict(row[features].to_frame().T)[0] == 1)
    print("  " + reason_line(row, contrib, score, flagged))

    # 5. Optional cross-check with the shap library (pip install shap).
    if args.shap:
        try:
            import shap
        except ModuleNotFoundError:
            sys.exit("The shap library is missing. Run: pip install shap  (or leave out --shap).")
        background = fitted[0].transform(x_train)
        lin = shap.LinearExplainer(fitted[-1], shap.maskers.Independent(background, max_samples=len(background)))
        values = lin(fitted[0].transform(row[features].to_frame().T)).values[0]
        grouped = pd.Series(values, index=explainer.owner).groupby(level=0).sum()[features]
        print(f"\nshap {shap.__version__} cross-check: largest difference from our contributions "
              f"{(grouped - contrib).abs().max():.4f} (0 means identical).")

    # 6. A reason line for every open delivery, next to its score.
    if len(open_deliveries):
        x_open = open_deliveries[features]
        scores = fitted.predict_proba(x_open)[:, 1]
        open_contrib = explainer.contributions(x_open)
        out = open_deliveries[["DeliveryDocument", "PlannedGoodsIssueDate"]].copy()
        out["PlannedGoodsIssueDate"] = out["PlannedGoodsIssueDate"].dt.date
        out["Score"] = scores.round(2)
        flags = tuned.predict(x_open) == 1   # the same rule predict_late.py uses
        out["Flag"] = np.where(flags, "CHECK", "")
        out["Reason"] = [reason_line(open_deliveries.loc[i], open_contrib.loc[i], s, f)
                         for i, s, f in zip(open_deliveries.index, scores, flags)]
        out = out.sort_values("Score", ascending=False)
        out.to_csv(HERE / "at_risk_explained.csv", index=False)
        print(f"\nWrote at_risk_explained.csv: {len(out)} open deliveries, "
              f"{(out['Flag'] == 'CHECK').sum()} flagged, each with its reason line.")


if __name__ == "__main__":
    main()

#Step 3: Run it

In the terminal, still inside unit02, run:

python explain_late.py --sample

It takes about five seconds. What success looks like:

Source: made-up sample. Model trained on 2119 deliveries before 2026-06-22, threshold 0.20.

What the model relies on overall (727 test deliveries)
  Field                     AP drop when shuffled  avg |contribution|
  PlannedLeadDays                   0.130 +/- 0.014              0.61
  ShippingPoint                     0.059 +/- 0.012              0.24
  GrossWeight                       0.040 +/- 0.010              0.37
  ProposedDeliveryRoute             0.036 +/- 0.007              0.15
  PlannedGIWeekday                  0.022 +/- 0.018              0.34  (within noise)
  ShippingCondition                 0.021 +/- 0.012              0.26  (within noise)
  Items                             0.020 +/- 0.009              0.28
  DeliveryPriority                  0.003 +/- 0.003              0.10  (within noise)
  ShippingType                     -0.001 +/- 0.002              0.03  (within noise)
  TotalQuantity                    -0.006 +/- 0.005              0.17  (within noise)
  AP drop: how much average precision falls when that column is shuffled (bigger = relied on more).

Why delivery 80002918 scores 0.88 (planned goods issue 2026-09-20, open)
  Start: the average delivery                         0.07   (log-odds -2.55)
  gross weight 2,264 KG                         +2.08   without it: 0.47
  shipping point 1710                           +0.59   without it: 0.80
  6 items                                       +0.54   without it: 0.81
  total quantity 156                            +0.48   without it: 0.82
  route R17102                                  +0.41   without it: 0.83
  shipping condition 01                         +0.17   without it: 0.86
  5 days from creation to planned goods issue   +0.11   without it: 0.87
  goods issue planned on a Sunday               +0.07   without it: 0.87
  delivery priority 02                          +0.06   without it: 0.87
  End: this delivery                                  0.88   (log-odds +1.98)

What the planner sees:
  Risk 88%, please check. Mainly: gross weight 2,264 KG; shipping point 1710; 6 items.

Wrote at_risk_explained.csv: 108 open deliveries, 17 flagged, each with its reason line.

Your numbers should match, because the sample data comes from a fixed random seed. A different last decimal is fine; library versions can cause that. A new file, at_risk_explained.csv, appears in unit02.

#Step 4: Read the global block

1. Lead time comes first. Shuffling PlannedLeadDays drops average precision by 0.130, far more than any other field. The model leans hardest on how many days the warehouse has between creation and planned goods issue.
2. The weekday is "within noise". In the late-delivery topic, the largest logistic regression weight was for a Friday goods issue (+0.83). Here the weekday field's drop, 0.022, is smaller than two times its spread. A big weight on one value of a field (Fridays) is not the same as the whole field mattering. Its average contribution, 0.34, is still sizeable: it moves individual scores, but scrambling it hardly hurts the ranking.
3. Total quantity gets no credit from shuffling. Its drop is slightly negative, yet its average contribution is 0.17. In this data, total quantity moves closely with the number of items and the weight. Shuffle it, and the model still reads the same signal from the other two. That is the correlated-features warning from the scikit-learn guide, live.
4. Two numbers, two questions. The first column asks "how much worse does the model get without this field?". The second asks "how much does this field move scores?". Show the process owner both, and explain the difference in one line.

#Step 5: Read the local block and the reason line

1. The start. The average training delivery scores 0.07 (log-odds -2.55).
2. The pushes. For delivery 80002918, the weight adds +2.08 in log-odds. The "without it" column translates that: remove the weight's push and the score would be 0.47 instead of 0.88. Shipping point 1710, six items and the total quantity add smaller pushes.
3. The end. The pushes add up to log-odds +1.98, which is the score 0.88. Nothing is left over; that is the additivity property.
4. The reason line. The script keeps the three largest pushes upward, if each is at least 0.15, and the largest push downward. Here nothing pushes down by 0.15 or more, so there is no "Lowers it" part.
5. What is missing. In the made-up data, route R17102 got much worse from mid-July 2026. This delivery is on that route, but the route adds only +0.41: the model was trained on deliveries before 22 June and never saw the change. The explanation is faithful to the model. The model is out of date. That is a job for drift monitoring, not for explanations.

Now open at_risk_explained.csv in VS Code. Each open delivery has its score, the CHECK flag and its reason. Read the top ten as a planner would. Do the reasons help you decide where to look first?

#Step 6 (optional): Cross-check with the shap library

The script computes contributions with its own few lines of numpy. To prove they are the real SHAP values, compare them with the shap library.

1. Install it into the course environment:

   Copy

   pip install shap

2. Run with the cross-check:

   Copy

   python explain_late.py --sample --shap

3. What success looks like: the same output as Step 3, plus one line before the last one:

   Copy

   shap 0.54.1 cross-check: largest difference from our contributions 0.0000 (0 means identical).

   Your version number may differ. A difference of 0.0000 means our formula and the library agree for every field. If pip install shap fails on your Python version, skip this step; nothing else depends on it.

#Step 7: Add a near-duplicate on purpose

Real SAP extracts often hold the same fact twice: gross weight on the header and summed from the items, or two quantity fields in different units. See what that does to explanations.

python explain_late.py --sample --add-copy

This adds GrossWeightCopy: the weight again with a 2% random wobble. The local block now reads:

Why delivery 80002918 scores 0.87 (planned goods issue 2026-09-20, open)
  Start: the average delivery                         0.07   (log-odds -2.55)
  gross weight 2,264 KG                         +3.09   without it: 0.24
  second weight reading 2,317                   -1.05   without it: 0.95
  shipping point 1710                           +0.59   without it: 0.79
  6 items                                       +0.54   without it: 0.80
  total quantity 156                            +0.48   without it: 0.81
  route R17102                                  +0.41   without it: 0.82
  shipping condition 01                         +0.17   without it: 0.85
  5 days from creation to planned goods issue   +0.11   without it: 0.86
  goods issue planned on a Sunday               +0.07   without it: 0.87
  delivery priority 02                          +0.06   without it: 0.87

1. The score barely moved: 0.87 instead of 0.88.
2. The reasons did. The weight now pushes +3.09, and the "second weight reading" pushes -1.05. The model learned to use the gap between two almost identical numbers.
3. The reason line is now nonsense: "Lowers it: second weight reading 2,317". No planner should ever see that.
4. The global block changed too. GrossWeight's drop rises from 0.040 to 0.098, because the model now leans on it harder to cancel the copy.

The fix is upstream: keep one field per business fact before you train. Explanations are often where you discover the duplicate.

#Step 8: Save your work in Git

From the course folder:

cd ..
git add unit02/explain_late.py
git commit -m "Explain the late-delivery model: importance and reason lines"

The CSV is an output you can recreate, so you don't need to commit it.

#What each part of the script does

#If something goes wrong

#The SAP way

What SAP documents, as of October 2026. Each item names the source we read; check it against your release before you build on it.

#SAP HANA Predictive Analysis Library: reason codes

An SAP Community blog by SAP describes explainability for PAL's Unified Classification, Unified Regression and AutoML, used through SQL or the hana_ml Python client:

• Global: permutation_importance=True in fit(), with permutation_evaluation_metric, permutation_n_repeats and permutation_seed. Results land in importance_. Tree models (random decision trees, HGBT) also give tree-based importance.
• Local: predict() and score() return a REASON_CODE column in JSON. Each entry has attr (the feature), val (its SHAP value) and pct (its share). top_k_attributions limits how many are returned. attribution_method takes 'tree-shap' (the default), 'saabas' (faster, for large data) or 'no'.
• Charts: a ShapleyExplainer gives summary plots across records and a force plot for one prediction.

Which model types and data splits each option needs differs by function and version, so check the hana-ml reference for yours before relying on it.

This is the same pair of ideas you built: permutation importance for the model, SHAP values for each record. The difference is where it runs: in SAP HANA, next to the data, so records don't leave the database.

#SAP Analytics Cloud: Smart Predict

SAP Learning describes Smart Predict's inputs as influencers, "variables that describe the data and explain a target variable", and says it selects an optimized number of them during training. That is a global view aimed at business analysts. The lesson we read doesn't say how individual predictions are explained; check the model reports in your tenant.

#SAP-RPT: explanations from attention

The SAP-RPT Playground API documents an optional explanations object for SAP-RPT-1.6:

• top_column_scores (1 to 20, default 4): the columns with the most weight in each prediction.
• top_relevant_context_rows (1 to 20, default 3): the indexes of the example rows that were most relevant.

The documentation says both are "derived from the model's internal attention representations". That is a different technique from SHAP: it shows where the model looked, not a split of the score that adds up. It also warns that results "can behave unintuitively" on very small datasets or with highly correlated columns, the same trap as Step 7. SAP Learning's lesson on calling sap-rpt-1.6 and sap-rpt-1.6-large through SAP AI Core mentions a confidence value between 0.0 and 1.0 per prediction, but not these explanations. Confirm the explanations object on the route you will use.

#Embedded scenarios in S/4HANA

For embedded predictions, such as delivery delay or supplier delivery date prediction, we found no SAP description of per-prediction explanations in the pages read for this topic. Before promising a planner "the app will tell you why", check the app's documentation for your release, or ask SAP.

#Policy

SAP's Global AI Ethics Policy (version 3.0, effective 15 June 2026) lists "Transparency and Explainability" and "Human Oversight and Determination" among its principles. Only the headings were readable to us; read the policy text before quoting it to a customer.

#Build vs. SAP

#Production concerns

• Authorizations. A reason line shows field values. Generate it under the same authorization checks as the score, and drop any field the user may not see. If a credit field drives the score but the user can't see credit data, say "credit factors" or nothing, and agree that wording with data protection.
• Store the explanation with the prediction. Log the score, threshold, model version and the contributions you showed. Recomputing later with a newer model gives a different answer to "why was this flagged in March?".
• Evaluate the explanations, not just the model. Before go-live, have a planner rate 20 reason lines as "makes sense", "surprising but plausible" or "nonsense". Nonsense reasons point to data problems (Step 7) or proxies.
• Stability. Retrain on a new month, and compare the global ranking. A field jumping from first to eighth needs an answer before users see new reasons.
• Cost. Linear contributions cost nothing. For tree models, SHAP runs once per scored record; at millions of records per day that is real compute. PAL's 'saabas' option exists for that reason, according to the SAP blog.
• Wording governance. Keep the describe rules in version control and review changes like code. A reworded reason can change user behaviour as much as a new model.
• Clean core. This explanation layer runs side by side with S/4HANA, on extracted data, and changes nothing in the system. Keep it there.
• Regulation. If the use case is in scope of the EU AI Act, explanation and human oversight are part of the obligations, not extras. Governance for SAP programmes is covered in AI governance for SAP programs.

#Pitfalls

• Reading importance off the weights. Scaled weights, one-hot columns and rare values make raw weights misleading. Measure instead.
• Explaining on training rows. Importance on training data rewards overfitting. Use the test rows.
• Showing log-odds to users. "+2.08" means nothing to a planner. Translate to "without it, 0.47".
• Ten reasons per line. Users read two or three. Set a minimum size and a maximum count.
• Causal language. "Late because of shipping point 1710" invites the wrong action. Write "Mainly:".
• Ignoring duplicates. Near-identical fields produce cancelling, nonsensical reasons.
• Trusting an explanation of an outdated model. The reasons can be faithful and still miss what changed, as with route R17102.

#Exercise: reason codes and a planner review

You will make the reasons countable and get a first human judgment on them. Both feed Evaluating in production, where overrides and comments become test cases.

1. Open explain_late.py. Below MIN_REASON, add a dictionary that maps each field to a short code:

   Copy

   CODES = {"GrossWeight": "WEIGHT", "PlannedLeadDays": "LEAD_TIME", "ShippingPoint": "SHIP_POINT",
            "Items": "ITEMS", "TotalQuantity": "QUANTITY", "ProposedDeliveryRoute": "ROUTE",
            "PlannedGIWeekday": "WEEKDAY", "ShippingCondition": "SHIP_COND",
            "DeliveryPriority": "PRIORITY", "ShippingType": "SHIP_TYPE", "GrossWeightCopy": "WEIGHT_2"}

2. In the worklist block, after the out["Reason"] = ... lines, add a column with the codes of the upward reasons, separated by |:

   Copy

   out["ReasonCodes"] = ["|".join([CODES[f] for f in open_contrib.loc[i].sort_values(ascending=False).index
                                   if open_contrib.loc[i, f] >= MIN_REASON][:3])
                         for i in open_deliveries.index]

3. Save and run python explain_late.py --sample. Open at_risk_explained.csv and check the new column.

4. Count the codes across the flagged deliveries. Which reason appears most often?

5. Create unit02/explanation_review.md. Copy in the five highest-scoring deliveries with their reason lines. For each, write one of "makes sense", "surprising but plausible" or "nonsense", and one sentence on why. If you can, ask someone who works in logistics to fill in this column instead of you.

6. Commit both files.

Done when: at_risk_explained.csv has a ReasonCodes column such as WEIGHT|SHIP_POINT|ITEMS for delivery 80002918, and explanation_review.md holds five rated reasons, committed to Git.

#Going further

For readers who want the math behind SHAP and the limits of the method.

Shapley values. Take a set of fields N and a value function v(S): the model's expected output when only the fields in S are known. The Shapley value of field i is its marginal contribution v(S ∪ {i}) − v(S), averaged over every order in which fields could be added. Written out, the weight of each subset S not containing i is |S|! (|N| − |S| − 1)! / |N|!. That averaging is what makes the split fair in game-theory terms. It is also why exact computation is expensive: the number of subsets doubles with every field.

Why it sums up. Averaged over orders, the marginal contributions telescope: in any one order, they add up to v(N) − v(∅), the prediction minus the baseline. The average of sums is the sum of averages. That is the additivity you saw in Step 5.

The linear case. If the model is linear and fields are treated as independent, v(S) is the intercept plus the weights times known values plus the weights times background means for the rest. Every marginal contribution of field i is then wᵢ (xᵢ − E[xᵢ]), whatever the order, so the average is that too. This is the formula in the SHAP API reference and in our LinearExplainer.

Interventional or conditional. "Treated as independent" is a choice. The interventional view replaces unknown fields with background values regardless of the known ones; it credits only fields the model actually uses. The conditional view respects correlations, so a field the model ignores can still get credit through a correlated one. shap.LinearExplainer documents both options. Neither is "right"; they answer different questions. Interventional values explain the model; conditional values explain the model on realistic data.

Not causal. Both views describe associations in the model. The SHAP documentation's retention example shows a proxy getting credit and a confounded effect flipping sign. Causal questions ("would splitting the delivery cut the risk?") need a design that controls for confounders, or an experiment.

Primary sources:

• A Unified Approach to Interpreting Model Predictions (Lundberg and Lee): the paper that framed SHAP as a family of additive attribution methods with a unique solution.
• shap.LinearExplainer: the linear formula and the interventional versus correlation-dependent options.
• Be careful when interpreting predictive models in search of causal insights: a worked example of explanations misleading policy choices, with alternatives.
• Permutation feature importance: the algorithm, and its behaviour with correlated features.

#Check yourself