Classifying text: LLMs, embeddings or a trained model?

#The 60-second version

A lot of enterprise AI is sorting. An email arrives and someone decides: is it an order question, a billing dispute, a complaint? A maintenance notification needs a damage code. A supplier message needs the right buyer. That decision is text classification: read a piece of text, pick one label from a fixed list.

There are three common ways to automate it:

• Ask an LLM. Write the labels and a few examples into a prompt. It works on day one, with almost no labeled history, but every item costs a model call.
• Embeddings plus a simple classifier. Turn each text into numbers that capture its meaning, then train a small, cheap model on your labeled history.
• A trained model. Train a small model directly on your own labeled history. It is fast and nearly free to run, but it only knows what the history taught it.

None wins everywhere. The right choice depends on how much labeled history you have, how many items a day you process, how fast new wording appears, and what a wrong label costs. In all three, the low-confidence items should go to a person.

#Why it matters to the business

Take Calder Pumpworks, the fictional pump maker this course uses. Its customer service inbox gets order status questions, delivery changes, invoice disputes, quote requests, complaints and a lot of noise. Each kind belongs in a different queue: the order desk, accounts receivable, inside sales, service and quality.

When routing is slow or wrong, the cost shows up in SAP. A disputed invoice such as 90000002 sits in the order desk's queue for days, so collections calls the customer about money that is already in dispute. A leaking-pump complaint waits behind status questions, so the warranty clock runs. A quote request goes unanswered and the distributor buys elsewhere.

Automated routing pays when three things hold:

• Volume. Hundreds of items a day, so minutes per item add up.
• Stable labels. The queues are known and don't change every month.
• A safe fallback. Items the system is unsure about go to a person, so a wrong guess costs a little time, not a lost customer.

The approach decides the cost. An LLM call per email is small, but it is paid on every item, every day. A trained model costs almost nothing per item, but someone has to label history and retrain it when the business changes. The same trade-off applies to maintenance notifications, supplier emails and service tickets.

#How SAP does it

SAP ships classification inside some applications and gives you platforms to build your own. As of October 2026:

• Built into SAP Service Cloud Version 2. Case categorization proposes a case category. SAP says it learns from past cases and your category catalog, and fills the field once the model is trained and activated and confidence thresholds are met. Case type determination reads an incoming email's subject and description and sets the case type when the email becomes a case. Both are listed as AI features that currently need AI Units, the entitlement SAP uses for many AI features; prices are on request. (case categorization, case type determination)
• LLM prompts through the generative AI hub in SAP AI Core. Its orchestration service combines a prompt template with optional data masking, content filtering, translation and grounding. The generative AI hub needs the SAP AI Core extended service plan. (SAP tutorial)
• Trained text classifiers in SAP HANA Cloud. The SAP HANA Python client (hana_ml) wraps text classification functions of the Predictive Analysis Library (PAL), including one that trains a model and predicts with it, plus TF-IDF and an embeddings function. (hana_ml reference)
• Your own model on SAP AI Core. AI Core runs training pipelines as batch jobs and serves trained models as web services. (SAP documentation)

If an SAP application already classifies the object you care about, start there. Embedded AI across SAP applications and buy vs. build cover that decision in general.

#The three approaches side by side

A common pattern combines them: a trained or embedding model handles the bulk, an LLM handles the items the model is unsure about, and a person handles what is left.

#Questions to ask

1. What does a wrong label cost, and what does a slow one cost? That sets the confidence threshold.
2. How much correctly labeled history do we have? Ask for a sample. Catch-all labels such as "General" mean the history needs cleaning.
3. How many items a day, and how fast must each be routed? Volume and latency decide whether a model call per item is affordable.
4. What share goes to a person? Ask for the threshold and the measured accuracy above it, on items the model never saw.
5. How is it measured over time? Ask how corrections by agents are captured and fed back.
6. Does an SAP application already do this? If the object lives in SAP Service Cloud, ask about its built-in categorization first.
7. What happens to personal data in the text? Emails carry names and addresses; ask where they go and whether they are masked.

#Common misconceptions

• "An LLM is always more accurate." On frequent, well-labeled categories, a cheap trained model often matches it. The LLM's edge is wording the history never showed.
• "We have no labeled data." Most SAP systems have years of tickets, notifications and cases with categories. They need cleaning, not inventing.
• "Accuracy is one number." A model can be 95% right overall and miss most of a rare but costly label. Look at each label.
• "Automation means no people." Good routing sends the unsure items to a person and learns from their corrections.
• "Set it up once." Products, customers and wording change. Plan for monitoring and retraining.

#Key terms

All course terms, A to Z, in the glossary

• Text classification: choosing one label from a fixed list for a piece of text.
• Label: one of the allowed answers, such as invoice_dispute.
• Labeled data: past items with the correct label, used to train or test.
• Zero-shot / few-shot: asking an LLM with no examples / a few examples in the prompt.
• Embedding: a list of numbers representing a text's meaning; similar texts get similar numbers.
• Confidence threshold: the minimum confidence for automatic routing; below it, a person decides.
• Macro F1: a score that weighs every label equally, so rare labels count.
• AI Units: the entitlement SAP uses to license many of its AI features.

#Check yourself

#Mental model

Classification is a decision with a fallback, and the three approaches differ in where the knowledge lives.

• In an LLM prompt, the knowledge lives in the model's pre-training plus a few examples you send with every call. You pay for that knowledge on every item.
• With embeddings plus a classifier, a pre-trained model turns text into meaning-vectors once per item, and a tiny model learns where your labels sit in that space from your history.
• In a trained model, all of the knowledge comes from your history. It costs almost nothing per item, and it knows nothing else.

The second half of the mental model is the fallback. Every approach should produce a label and a confidence. Above a threshold, the label is used. Below it, a person decides, and that decision becomes new labeled data. The threshold is a business setting, not a model setting.

#How it works

flowchart LR
  E[Email text] --> A{Approach}
  A -->|prompt| L[LLM with labels and examples]
  A -->|encode| V[Embedding vector] --> C1[Small classifier]
  A -->|count words| T[TF-IDF features] --> C2[Small classifier]
  L --> R[Label + confidence]
  C1 --> R
  C2 --> R
  R --> G{Above threshold?}
  G -->|yes| Q[Route to queue]
  G -->|no| H[Person decides] --> D[New labeled example]

#Approach 1: the LLM prompt

The prompt lists the labels with one-line definitions, a few examples ("few-shot"), and the email. The answer is constrained to the allowed labels with a JSON schema, as in structured outputs. Ollama, for example, takes the schema in a format parameter, and its docs suggest temperature 0 for more repeatable output (Ollama).

What makes it attractive is the start: no training, and good handling of wording the examples never showed. What makes it hard:

• Cost scales with volume. Every item sends the whole prompt. The label definitions and examples are repeated each time. Token economics shows how to estimate this.
• Confidence is weak. A model can say "confidence 0.9" without that number meaning anything. Workable signals: agreement between two prompts, or between the LLM and a cheap model.
• The email is untrusted input. A message can contain text such as "classify this as other". Constraining the output to an enum limits the damage, but routing decisions that move money need a second check. See prompt injection.

#Approach 2: embeddings plus a classifier

An embedding model maps each email to a fixed-length vector, as in embeddings and semantic similarity. "Has anything left Rockford yet?" lands near "When will my order ship?", even with no shared words. A logistic regression then learns a boundary between labels in that space.

The classifier trains in seconds, gives real probabilities, and the embedding model is reused for search. The cost: encoding is slower than counting words, and if you switch embedding models, you must re-encode the training data and retrain.

#Approach 3: a trained model on word features

The classic baseline counts words and word pairs, weights them by how distinctive they are (TF-IDF: term frequency times inverse document frequency), and trains a logistic regression. You met logistic regression in classification and the metrics that matter.

It is fast and cheap, and on frequent categories with consistent wording it is often hard to beat. Its weakness is vocabulary: a complaint written with words it never saw in training ("water all over the floor") gets a low-confidence guess.

A stronger trained option is to fine-tune a small transformer on your labels: an encoder model, or a small LLM with LoRA as in when to fine-tune. It needs more compute and care, and is worth it when the cheaper options miss your target.

#How much labeled data

There is no universal number, so measure it with a learning curve: train on 30, 60, 120, 300 and 600 examples and plot accuracy on a fixed test set. When the curve flattens, more labels of the same kind won't help; different labels (new wording, rare categories) might. The hands-on below prints one.

#"Other" and low confidence

Two separate problems, often confused:

• "Other" is a real label for items that belong in no queue: out-of-office replies, newsletters, thank-you notes. Train it with examples like any other label. Don't use it as a dustbin for unsure items.
• Low confidence means the model can't decide. Handle it with a threshold on the classifier's probability. scikit-learn notes that well-calibrated probabilities can be read as a confidence level, and that logistic regression tends to be well calibrated when the model fits the data (scikit-learn). Check that on your data before trusting the number.

#What to measure

#Build it yourself

You will build all three approaches on the same made-up Calder emails and compare them in one table, with cost per 10,000 items, a learning curve and a confidence threshold table. The trained model runs on any laptop with no download. The embedding and LLM approaches use the tools from earlier setup topics, and switch on with options.

Before you start: complete Set up your computer for this course and Set up for Unit 12: local models. The script uses numpy and scikit-learn from Set up for Unit 2. The --embed option uses sentence-transformers and the all-MiniLM-L6-v2 model from Set up for Unit 3. The --llm option uses Ollama and qwen3:0.6b from the Unit 12 setup. The default run needs no model, account or key.

flowchart LR
  D[Made-up emails<br/>600 train, 300 test] --> T[Trained model]
  D --> E[Embeddings + classifier<br/>--embed]
  D --> L[Local LLM<br/>--llm]
  T --> R[Comparison table<br/>+ clf_report.json]
  E --> R
  L --> R

#What you need

• Your course folder orchestrate-course with its .venv.
• 25 to 40 minutes. The --llm run takes a few minutes more on a laptop CPU.
• Cost: free. Everything runs on your computer with made-up data.
• Optional: the Unit 3 and Unit 12 setups, for --embed and --llm.

#Step 1: Open your course folder

1. Open VS Code, choose File > Open Folder, and open orchestrate-course.

2. Open a terminal with Terminal > New Terminal. Turn on the virtual environment if the prompt doesn't start with (.venv):

   Windows (PowerShell):

   Copy

   .venv\Scripts\Activate.ps1

   macOS or Linux:

   Copy

   source .venv/bin/activate

3. Check that the unit12 folder exists. If it doesn't, create it:

   Copy

   mkdir unit12

4. Check that scikit-learn is installed:

   Copy

   python -c "import sklearn; print(sklearn.__version__)"

   You should see a version number, such as 1.9.1. If you see ModuleNotFoundError, add scikit-learn==1.9.1 and numpy==2.5.4 to requirements.txt and run pip install -r requirements.txt.

Run every command in this topic from the course folder, not from inside unit12.

#Step 2: Save the script

1. In VS Code, choose File > New File, and save it as unit12/classify_compare.py.
2. Paste this code and save.

"""Compare three ways to route customer emails at Calder Pumpworks (fictional).

Approaches, all on the same made-up labeled emails:
  trained  TF-IDF word counts + logistic regression, trained here (always runs)
  embed    all-MiniLM-L6-v2 sentence embeddings + logistic regression (--embed)
  llm      few-shot prompt to qwen3:0.6b in Ollama, JSON answer (--llm)

Usage (from the course folder):
  python unit12/classify_compare.py                  # trained model only, no download
  python unit12/classify_compare.py --embed          # add embeddings (Unit 3 setup)
  python unit12/classify_compare.py --llm --limit 60 # add the local LLM (Unit 12 setup)
"""

import argparse
import json
import random
import sys
import time
from pathlib import Path

import numpy as np
from sklearn.feature_extraction.text import TfidfVectorizer
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import accuracy_score, f1_score
from sklearn.pipeline import make_pipeline

SEED = 42
HERE = Path(__file__).resolve().parent
DATA_DIR = HERE / "clf_data"
REPORT = HERE / "clf_report.json"

LABELS = ["order_status", "delivery_change", "invoice_dispute",
          "quote_request", "complaint_return", "other"]
SHARE = [0.30, 0.15, 0.18, 0.15, 0.12, 0.10]   # how often each label occurs

# Templates per label. "common" phrasings appear in training and test;
# "unseen" phrasings appear only in the test set, to mimic new wording.
TEMPLATES = {
    "order_status": {
        "common": [
            "Hi, can you tell me the status of order {order}? We need the {model} pumps soon.",
            "Where is our order {order}? No shipping confirmation yet.",
            "Please confirm when order {order} will ship. Thanks.",
            "Any update on delivery for PO {cpo}? Our site team is waiting.",
            "Could you send the tracking number for order {order}?",
        ],
        "unseen": [
            "Our crew is standing by for the {model} units on {cpo}. Are they on a truck yet?",
            "Has anything left Rockford for {cpo}, or is it still in the plant?",
        ],
    },
    "delivery_change": {
        "common": [
            "Please move the delivery of order {order} to {date}.",
            "We need to change the quantity on order {order} to {qty} pumps.",
            "Can you ship order {order} to our second site instead? New address follows.",
            "Please hold order {order} until {date}, the site is not ready.",
            "Change request for PO {cpo}: split into two deliveries please.",
        ],
        "unseen": [
            "Our install slipped two weeks, so {cpo} should arrive later than planned.",
            "Only {qty} of the {model} pumps on {cpo} are needed now, the rest next quarter.",
        ],
    },
    "invoice_dispute": {
        "common": [
            "Invoice {inv} shows the wrong price for the {model}. Please correct it.",
            "We were billed twice for order {order}, see invoice {inv}.",
            "Invoice {inv} does not match our PO {cpo}. We paid the agreed amount.",
            "Please send a credit note for invoice {inv}, freight was charged in error.",
            "Our payment for invoice {inv} was short because of the price difference.",
        ],
        "unseen": [
            "Accounts payable flagged {inv}: the unit rate is above our contract.",
            "Why is there a fuel surcharge on {inv}? It was never agreed.",
        ],
    },
    "quote_request": {
        "common": [
            "Please quote {qty} {model} pumps with delivery to our plant.",
            "Can we get a price for {qty} seal kits CP-SK200?",
            "Request for quotation: {qty} impellers for {model}, lead time please.",
            "What is your price and lead time for an engineered pump like the EP-900?",
            "Please send your current price list for spare parts.",
        ],
        "unseen": [
            "We are budgeting next year's spares. What would {qty} seal kits cost us?",
            "Ballpark figure for {qty} {model} units, landed at our site?",
        ],
    },
    "complaint_return": {
        "common": [
            "The {model} pump from order {order} is leaking at the seal.",
            "Two seal kits arrived damaged. We want to return them.",
            "Pump serial {serial} failed after one week. Is this under warranty?",
            "The impeller on our {model} is cracked. Please advise on a return.",
            "Wrong part delivered on order {order}, we need a return label.",
        ],
        "unseen": [
            "Unit {serial} trips the motor every morning since it went in.",
            "There is water all over the floor under the new {model}.",
        ],
    },
    "other": {
        "common": [
            "I am out of the office until {date} with limited access to email.",
            "Subscribe to our newsletter for the latest offers.",
            "Thank you, received.",
            "Please update our contact person to the new purchasing manager.",
            "We are hiring maintenance technicians, see our careers page.",
        ],
        "unseen": [
            "Season's greetings from all of us at the utility.",
            "Reminder: our offices close early on Friday.",
        ],
    },
}

GREETINGS = ["", "Hello, ", "Good morning, ", "Hi team, ", "Dear Calder, "]
CLOSINGS = ["", " Regards, Brennock Water Utility", " Thanks, Hallam Fluid Systems",
            " Best, purchasing", " Kind regards"]


def fill(template, rng):
    """Put made-up but Calder-shaped values into a template."""
    return template.format(
        order=rng.choice(["1000001", "1000004", "1000006", "1000011"]),
        cpo=rng.choice(["BWU-PO-5602", "HFS-PO-7781", "HFS-PO-7790", "BWU-PO-5633"]),
        inv=rng.choice(["90000002", "90000003"]),
        model=rng.choice(["ES-200", "ES-300"]),
        qty=rng.choice([2, 4, 8, 16, 20, 60]),
        date=rng.choice(["2026-11-13", "2026-11-30", "2026-12-04"]),
        serial=rng.choice(["ES300-0188", "ES200-0412"]),
    )


def make_email(label, kind, rng):
    text = rng.choice(GREETINGS) + fill(rng.choice(TEMPLATES[label][kind]), rng)
    return text + rng.choice(CLOSINGS)


def make_split(n, unseen_share, rng, noise=0.0):
    rows = []
    for label, share in zip(LABELS, SHARE):
        for _ in range(round(n * share)):
            kind = "unseen" if rng.random() < unseen_share else "common"
            row = {"text": make_email(label, kind, rng), "label": label, "phrasing": kind}
            if rng.random() < noise:       # a few labels are wrong, as in real history
                row["label"] = rng.choice([x for x in LABELS if x != label])
            rows.append(row)
    rng.shuffle(rows)
    return rows


def write_jsonl(path, rows):
    path.parent.mkdir(parents=True, exist_ok=True)
    with path.open("w", encoding="utf-8") as f:
        for row in rows:
            f.write(json.dumps(row) + "\n")


def score(name, labeled, test, predicted, seconds, cost_10k):
    gold = [r["label"] for r in test]
    unseen = [i for i, r in enumerate(test) if r["phrasing"] == "unseen"]
    return {
        "approach": name,
        "labeled_examples": labeled,
        "accuracy": round(float(accuracy_score(gold, predicted)), 3),
        "macro_f1": round(float(f1_score(gold, predicted, average="macro", zero_division=0)), 3),
        "unseen_accuracy": (round(float(np.mean([predicted[i] == gold[i] for i in unseen])), 3)
                            if unseen else None),
        "ms_per_item": round(1000 * seconds / len(test), 2),
        "cost_per_10k_usd": cost_10k,
        "items_scored": len(test),
    }


def train_tfidf(train):
    model = make_pipeline(
        TfidfVectorizer(ngram_range=(1, 2), min_df=1, sublinear_tf=True),
        LogisticRegression(max_iter=2000, C=10.0),
    )
    model.fit([r["text"] for r in train], [r["label"] for r in train])
    return model


def run_trained(train, test):
    model = train_tfidf(train)
    start = time.perf_counter()
    predicted = list(model.predict([r["text"] for r in test]))
    seconds = time.perf_counter() - start
    return model, score("trained (TF-IDF + logistic regression)", len(train),
                        test, predicted, seconds, 0.0)


def run_embed(train, test):
    try:
        from sentence_transformers import SentenceTransformer
    except ImportError:
        sys.exit("sentence-transformers isn't installed. See Set up for Unit 3.")
    print("Loading sentence-transformers/all-MiniLM-L6-v2 (downloads once)...")
    encoder = SentenceTransformer("sentence-transformers/all-MiniLM-L6-v2")
    x_train = encoder.encode([r["text"] for r in train], normalize_embeddings=True)
    clf = LogisticRegression(max_iter=2000, C=10.0)
    clf.fit(x_train, [r["label"] for r in train])
    start = time.perf_counter()
    x_test = encoder.encode([r["text"] for r in test], normalize_embeddings=True)
    predicted = list(clf.predict(x_test))
    seconds = time.perf_counter() - start
    return score("embeddings (MiniLM + logistic regression)", len(train),
                 test, predicted, seconds, 0.0)


def build_prompt(text, examples):
    lines = ["Route this customer email to one queue. Queues:",
             "order_status (where is my order), delivery_change (change date, quantity "
             "or address), invoice_dispute (billing problem), quote_request (price or "
             "lead time), complaint_return (defect, damage, warranty, return), "
             "other (anything else).", "", "Examples:"]
    for ex in examples:
        lines.append(f'Email: "{ex["text"]}" -> {ex["label"]}')
    lines += ["", f'Email: "{text}"', 'Answer as JSON: {"label": "<queue>"}']
    return "\n".join(lines)


def pick_examples(train):
    """One training email per label, chosen with the fixed seed."""
    rng = random.Random(SEED)
    return [rng.choice([r for r in train if r["label"] == lab]) for lab in LABELS]


def llm_cost_10k(test, examples, price_in, price_out):
    """Estimate API cost per 10,000 emails from prompt size (about 4 characters per token)."""
    avg_chars = np.mean([len(build_prompt(r["text"], examples)) for r in test])
    tokens_in, tokens_out = avg_chars / 4, 12
    per_item = (tokens_in * price_in + tokens_out * price_out) / 1_000_000
    return round(per_item * 10_000, 2), round(tokens_in)


def run_llm(test, examples, model_name, cost):
    try:
        import ollama
    except ImportError:
        sys.exit("The ollama library isn't installed. See Set up for Unit 12.")
    schema = {"type": "object", "properties": {"label": {"type": "string", "enum": LABELS}},
              "required": ["label"]}
    predicted, start = [], time.perf_counter()
    for i, row in enumerate(test, 1):
        try:
            reply = ollama.chat(model=model_name, think=False, format=schema,
                                options={"temperature": 0, "seed": SEED},
                                messages=[{"role": "user",
                                           "content": build_prompt(row["text"], examples)}])
            label = json.loads(reply.message.content).get("label", "other")
        except ollama.ResponseError as error:
            sys.exit(f"Ollama error: {error}. Did you run: ollama pull {model_name}?")
        except ConnectionError:
            sys.exit("Can't reach Ollama. Start the Ollama app, then try again.")
        except json.JSONDecodeError:
            label = "other"
        predicted.append(label if label in LABELS else "other")
        if i % 20 == 0:
            print(f"  LLM: {i}/{len(test)} emails")
    seconds = time.perf_counter() - start
    return score(f"LLM few-shot ({model_name})", len(examples), test, predicted, seconds, cost)


def learning_curve(train, test):
    rows = []
    for n in [30, 60, 120, 300, len(train)]:
        model = train_tfidf(train[:n])
        predicted = model.predict([r["text"] for r in test])
        rows.append((n, accuracy_score([r["label"] for r in test], predicted)))
    return rows


def threshold_table(model, test):
    proba = model.predict_proba([r["text"] for r in test])
    best = proba.argmax(axis=1)
    confidence = proba.max(axis=1)
    gold = np.array([r["label"] for r in test])
    guess = model.classes_[best]
    rows = []
    for t in [0.0, 0.5, 0.6, 0.7, 0.8]:
        auto = confidence >= t
        acc = float((guess[auto] == gold[auto]).mean()) if auto.any() else 0.0
        rows.append((t, float(auto.mean()), acc))
    return rows


def main():
    parser = argparse.ArgumentParser(description=__doc__,
                                     formatter_class=argparse.RawDescriptionHelpFormatter)
    parser.add_argument("--embed", action="store_true", help="add the embeddings approach")
    parser.add_argument("--llm", action="store_true", help="add the local LLM approach")
    parser.add_argument("--model", default="qwen3:0.6b", help="Ollama model for --llm")
    parser.add_argument("--limit", type=int, default=60, help="test emails sent to the LLM")
    parser.add_argument("--train-size", type=int, default=600)
    parser.add_argument("--price-in", type=float, default=0.50,
                        help="USD per million input tokens (course value, not a real price)")
    parser.add_argument("--price-out", type=float, default=2.00,
                        help="USD per million output tokens (course value, not a real price)")
    args = parser.parse_args()

    # Separate generators, so changing the training set never changes the test set.
    train = make_split(args.train_size, unseen_share=0.0, rng=random.Random(SEED), noise=0.03)
    test = make_split(300, unseen_share=0.2, rng=random.Random(SEED + 1))
    write_jsonl(DATA_DIR / "train.jsonl", train)
    write_jsonl(DATA_DIR / "test.jsonl", test)
    print(f"Wrote {len(train)} training and {len(test)} test emails to {DATA_DIR}")

    model, trained = run_trained(train, test)
    results = [trained]
    if args.embed:
        results.append(run_embed(train, test))

    examples = pick_examples(train)
    cost, tokens_in = llm_cost_10k(test, examples, args.price_in, args.price_out)
    if args.llm:
        results.append(run_llm(test[:args.limit], examples, args.model, cost))

    print("\nApproach                                   labels  acc   F1    unseen  ms/item")
    for r in results:
        unseen = "n/a" if r["unseen_accuracy"] is None else f"{r['unseen_accuracy']:.3f}"
        print(f"{r['approach']:<42} {r['labeled_examples']:>6}  {r['accuracy']:.3f} "
              f"{r['macro_f1']:.3f}  {unseen:<6}  {r['ms_per_item']:>7}")
    print(f"\nLLM prompt size: about {tokens_in} input tokens per email.")
    print(f"LLM API cost per 10,000 emails at the course prices: {cost:.2f} USD "
          f"(change with --price-in and --price-out).")
    print("Trained and embedding models: no per-call charge; you pay for the machine.")

    print("\nLearning curve, trained model (labeled emails -> accuracy):")
    for n, acc in learning_curve(train, test):
        print(f"  {n:>4} -> {acc:.3f}")

    print("\nConfidence threshold, trained model (send the rest to a person):")
    print("  threshold  auto-routed  accuracy on auto-routed")
    table = threshold_table(model, test)
    for t, share, acc in table:
        print(f"  {t:>9.1f}  {share:>10.0%}  {acc:>10.3f}")

    REPORT.write_text(json.dumps({"seed": SEED, "results": results,
                                  "thresholds": table}, indent=2), encoding="utf-8")
    print(f"\nSaved {REPORT}")


if __name__ == "__main__":
    main()

#Step 3: Run the trained model

python unit12/classify_compare.py

It takes a few seconds. What success looks like:

Wrote 600 training and 300 test emails to /Users/you/orchestrate-course/unit12/clf_data

Approach                                   labels  acc   F1    unseen  ms/item
trained (TF-IDF + logistic regression)        600  0.887 0.885  0.404      0.02

LLM prompt size: about 284 input tokens per email.
LLM API cost per 10,000 emails at the course prices: 1.66 USD (change with --price-in and --price-out).
Trained and embedding models: no per-call charge; you pay for the machine.

Learning curve, trained model (labeled emails -> accuracy):
    30 -> 0.640
    60 -> 0.787
   120 -> 0.860
   300 -> 0.880
   600 -> 0.887

Confidence threshold, trained model (send the rest to a person):
  threshold  auto-routed  accuracy on auto-routed
        0.0        100%       0.887
        0.5         82%       0.996
        0.6         81%       1.000
        0.7         80%       1.000
        0.8         77%       1.000

Saved /Users/you/orchestrate-course/unit12/clf_report.json

How to read it:

• Overall accuracy is 0.887, but accuracy on unseen phrasing is only 0.404. The trained model is near perfect on wording it has seen and weak on wording it hasn't. That gap is the most important number in the table.
• The learning curve climbs steeply to about 120 labeled emails, then flattens. More labels with the same wording add little; labels with new wording would.
• The threshold table is what you would run in production. At 0.5, about 82% of emails are routed automatically at 0.996 accuracy, and the other 18%, mostly the unseen phrasings, go to a person.
• The LLM cost line works even without Ollama: the script measures the prompt it would send (about 284 tokens) and multiplies by the prices you give it. The defaults, 0.50 and 2.00 USD per million input and output tokens, are course values, not a real price list. Look up your provider's prices and pass them with --price-in and --price-out.

#Step 4: Add embeddings

If you did the Unit 3 setup:

python unit12/classify_compare.py --embed

The first line says Loading sentence-transformers/all-MiniLM-L6-v2 (downloads once).... A second row appears in the table: embeddings (MiniLM + logistic regression). Compare its unseen column with the trained model's. Encoding 900 emails takes from a few seconds to a minute on a laptop CPU, so its ms/item is much higher than the trained model's.

#Step 5: Add the local LLM

If you did the Unit 12 setup, make sure the Ollama app is running, then:

python unit12/classify_compare.py --llm --limit 60

The script sends the first 60 test emails to qwen3:0.6b, one at a time, and prints progress every 20. A third row appears, scored on those 60 emails only (its labels column says 6: one example per label in the prompt). With temperature 0 and a fixed seed the answers should repeat between runs on the same computer, but they can differ between Ollama versions and machines.

A 0.6-billion-parameter model is small, so expect it to be weaker than the trained model on familiar wording. The point is the method: the same test set, the same metrics, one table. To try a bigger model you have pulled, pass --model, for example --model qwen3:4b.

#Step 6: Save your work

1. Check what Git sees:

   Copy

   git status

   You should see unit12/classify_compare.py, unit12/clf_data/ and unit12/clf_report.json.

2. Save them:

   Copy

   git add unit12/classify_compare.py unit12/clf_data unit12/clf_report.json
   git commit -m "Unit 12: compare three text classifiers"

The emails in clf_data/ are reused by the routing project in the portfolio projects.

#How the code works

#If something goes wrong

#The SAP way

The same three options exist on SAP's stack. As of October 2026:

Use the classifier in the application. SAP Service Cloud Version 2 lists case categorization, which proposes the case category once the model is trained and activated and confidence thresholds are met, and case type determination, which reads an email's subject and description and sets the case type during email-to-case. Both currently need AI Units (case categorization, case type determination). The pages don't say which model type is used, so test them on your own cases rather than assuming LLM-like handling of new wording.

LLM prompts through the generative AI hub. In SAP AI Core, the orchestration service runs your prompt template against a chosen model, and can add data masking (useful for names in emails), content filtering and translation. It requires the SAP AI Core extended plan (SAP tutorial). The generative AI hub topic shows the calls.

Trained models where the data is. If the history is already in SAP HANA Cloud, the hana_ml client exposes PAL text classification: text_classification, and TextClassificationWithModel, which trains a model and predicts with it, plus TFIDF and PALEmbeddings (hana_ml reference). The data doesn't leave the database. Check your HANA version: the reference notes that some functions differ between on-premise and HANA Cloud.

Your own model on SAP AI Core. AI Core runs training as a batch pipeline (Argo Workflows) and serves the model as a web service (KServe), from your own Docker images (SAP documentation). This is the home for an embeddings classifier or a fine-tuned encoder.

#Build vs. SAP

#Architecture view

Where it sits. For Calder, emails reach a mailbox or SAP Service Cloud Version 2. A classification service, in SAP Service Cloud itself or on SAP BTP, returns a label and confidence. The case or ticket is created in the right queue; for order-related labels, a later step reads the sales order from CP10's S/4HANA system through a released API.

flowchart LR
  M[Mailbox] --> SC[Service Cloud<br/>email-to-case]
  SC --> CL[Classifier<br/>built-in or on BTP]
  CL -->|label, confidence| SC
  SC -->|order lookup| S4[S/4HANA CP10<br/>released API]
  CL -.->|corrections| DS[Labeled history]
  DS -.->|retrain| CL

Non-functional requirements.

Integration and identity. Email routing runs as a technical user; nobody's personal authorizations are involved in the label. Authorizations matter in the next step: reading an order or invoice uses a communication user with read-only access to the released APIs it needs. Agent corrections flow back as labeled data, so the case history must keep both the proposed and the final label.

The decision to record (short ADR form from the design-doc topic):

• Context: about 400 emails a day (course value), six queues, two years of cases with categories, 30% of them "General".
• Options: A built-in case categorization; B LLM prompt per email; C trained model with an LLM for low-confidence items; D fine-tuned encoder.
• Decision: A if the categories fit after cleaning; otherwise C.
• Consequences: history must be cleaned first; a person owns the low-confidence queue; retraining is scheduled quarterly or when a label changes.

What a design review will ask.

1. What accuracy did you measure, per label, on unseen emails, and at what threshold?
2. Where does email text go, and is personal data masked before an external model?
3. What is the cost per month at peak volume, and who pays when a mailbox is added?
4. How are corrections captured, and who approves a retrain?
5. What happens when the classifier is down: does mail queue or route to a default?

#Production concerns

• Security. Emails are untrusted input. Constrain LLM output to the label list, never let the classifier trigger SAP changes directly, and treat a label as a routing suggestion. Human approval covers the patterns for actions.
• Personal data. Mask names, addresses and bank details before text leaves your landscape, as in data security and PII.
• Evaluation. Keep a fixed test set with recent emails and rare labels. Score per label and on new wording, before and after every change. Online evaluation covers measuring in production.
• Calibration. Check that "0.8 confidence" means about 80% right on your data before choosing a threshold.
• Cost. Estimate per item from measured prompt size and your provider's real prices. For a cascade, the LLM cost is the price times the share below the threshold.
• Operations. Log the text hash, label, confidence, model version and final label. Alert when the auto-routed share drops: new wording has arrived.
• Clean core. Keep the classifier outside the S/4HANA core: in the application's built-in feature or a BTP service, writing back through released APIs only.

#Pitfalls

• Testing on training wording. Template-like history gives near-perfect scores that collapse on new emails. Hold out recent months, or new customers.
• Using "other" as a dustbin. Keep "other" for real non-queue mail and use the threshold for unsure items.
• Trusting old categories. Catch-all and wrong labels in history teach the model to be wrong. Clean a sample first and measure label quality.
• Comparing on different test sets. Score all approaches on the same items, or the table means nothing.
• Asking an LLM for a confidence number. Use model agreement or a calibrated classifier instead.
• Forgetting the long tail. Macro F1 and per-label recall show the rare labels that accuracy hides.

#Exercise

Feed the reviewed emails back. In production, the emails a person routes become new training data. Simulate it.

1. Open unit12/classify_compare.py and find this line in main():

   Copy

   train = make_split(args.train_size, unseen_share=0.0, rng=random.Random(SEED), noise=0.03)

2. Change unseen_share=0.0 to unseen_share=0.1. Now 10% of training emails use the wording the test calls unseen, as if reviewers had labeled such emails.

3. Save and run python unit12/classify_compare.py.

4. Compare the unseen column and the threshold table with your earlier run.

5. Write two sentences in unit12/notes.md: what changed, and what it tells you about where to spend labeling effort.

6. Change the value back to 0.0 and run once more so your saved report matches the walkthrough, then commit notes.md.

Done when the run with unseen_share=0.1 shows unseen accuracy above 0.95, and unit12/notes.md explains why labeling new wording helped more than adding more of the same.

#Going further

For readers who want to go past a word-count baseline without paying for an LLM call per item.

• SetFit (Tunstall et al., 2022). Fine-tunes a Sentence Transformer on pairs of examples (same label or not), then trains a classification head on the tuned embeddings. The authors report accuracy comparable to parameter-efficient fine-tuning and pattern-based methods, about an order of magnitude faster to train, with no prompts (arXiv 2209.11055). It sits between approach 2 and a full fine-tune, and suits tasks with few labels per class.
• Calibration curves. Before you set a threshold, plot predicted confidence against observed accuracy. scikit-learn's guide explains calibration curves and when a classifier's probabilities can be read as confidence (scikit-learn).
• Constrained decoding. JSON schemas with an enum keep an LLM's answer inside your label list. The Ollama docs show the pattern and also suggest putting the schema in the prompt (Ollama).

A cascade is the natural next experiment: route with the trained model above a threshold, send only the rest to the LLM, and compute the blended accuracy and cost per 10,000 emails.

#Check yourself