Orchestrate

Report an issue

Something wrong, out of date or unclear? Tell us. Corrections are prioritized over new topics. You don't need to sign in.

0 / 2000

Don't include personal or confidential company details. Reports are reviewed with AI assistance.

CDS extraction pipelines for AI: delta, keys, deletions and time travel

How extraction-enabled CDS views feed an AI feature table with deltas, what SAP's API policy allows, and how to keep history so training data is honest.

Updated Nov 13, 2026Foundational 8 minDeep 40 min
Foundational layer · 8 min read

The 60-second version

A model that predicts late deliveries, an agent that ranks blocked orders, a forecast for spare parts: all of them learn from a copy of SAP data, not from the live system. Something has to make that copy and keep it current. In S/4HANA, the standard way is CDS-based extraction: SAP's data model is built from core data services (CDS) views, and views marked for extraction can be read in bulk by a data platform.

Copying everything every night is slow and expensive, so pipelines copy only what changed since the last run. That is called a delta. Deltas are where pipelines quietly go wrong: a record changed at the wrong moment is skipped, a deleted order stays in the copy forever, or the copy keeps only today's version of each record, so the model trains on facts that weren't known when the prediction would have been made.

This topic explains how SAP's extraction works, what SAP's 2026 API policy allows, and the three checks every AI data pipeline needs: nothing missed, nothing deleted left behind, and history kept so the past can be replayed.

Why it matters to the business

A model is only as honest as its copy. At Calder Pumpworks, our fictional pump maker, a late-delivery model reads three months of outbound deliveries from CP10's S/4HANA system. If the copy misses the goods issues posted by the evening batch job, six in ten shipped deliveries look "still open". The planners get a worklist full of shipments that left days ago, and they stop trusting it within a week.

Deleted records cost money twice. A delivery created by mistake and deleted in S/4HANA should leave the copy too. If it doesn't, the model scores it every day, and someone phones a customer about a shipment that never existed.

History is a feature, not a luxury. The planned goods issue date on a delivery often moves after it is created. If the copy only keeps the latest date, a training run "knows" the new date at the moment the prediction should have been made. Test scores look better than live results, and the business case is built on a number that won't repeat.

Rules decide the route, not only technology. As of SAP's API policy of 27 April 2026, large-scale extraction is allowed only through an architecture SAP endorses, and one popular interface (ODP-RFC) is classed as unpermitted for customer and partner tools (SAP API policy overview). A pipeline built on the wrong interface may stop working, or breach the contract, regardless of how good the model is.

How SAP does it

As of November 2026, the SAP pages opened for this topic describe these pieces. Check each against your own release.

Piece What it does Where it applies
Extraction-enabled CDS views A view annotated for extraction becomes a source a data platform can read fully and then in deltas (SAP Learning) S/4HANA on-premise and cloud; in the cloud through a communication scenario over HTTP
Operational data provisioning (ODP) SAP's framework that serves full and delta loads from ABAP systems, with a delta queue on the source (SAP Learning) Behind BW/4HANA, Datasphere and other SAP consumers
Replication flows in SAP Datasphere Copy released extraction views with an initial load and then deltas, at an interval you set (SAP Learning) Datasphere, also inside SAP Business Data Cloud
Data products SAP-managed, ready-made data sets shared without copying (zero copy), which the API policy names as an endorsed pattern SAP Business Data Cloud; see SAP data on a lakehouse

A sensible order, consistent with the endorsed patterns in SAP's API policy: first, is there a data product for this data? If not, is there an extraction-enabled view SAP has released? Only then build something of your own.

The three checks every pipeline needs

Check What goes wrong without it At Calder Ask for
Nothing missed Changes that commit late fall between two runs The evening goods-issue job's postings vanish from the copy A safety interval (overlap) on timestamp deltas, or change data capture
Deletions handled Deleted records stay in the copy forever Deliveries created by mistake are scored every day Deletion detection, or a delta method that sends deletions
History kept Training sees today's values for past moments A moved goods issue date leaks the answer into the features Every version kept with its valid-from time, so the past can be replayed ("time travel")

A fourth check makes the first three visible: a reconciliation that compares the copy with the source after each run (counts, keys, a sample of values) and stops training when they don't match.

Questions to ask

  • Is there an SAP data product for this data in Business Data Cloud? If yes, why are we extracting?
  • Which released, extraction-enabled CDS view will we read, and does SAP list it for our release?
  • Which interface carries the data? Does it fit SAP's API policy, and who confirmed that with SAP in writing?
  • How does the pipeline find changes: a timestamp delta (with what safety interval) or change data capture?
  • How are deletions and archiving handled?
  • Do we keep every version of a record, or only the latest? Can we rebuild last month's training set exactly?
  • What reconciliation runs after each load, and who is alerted when it fails?
  • What load does extraction put on the production system at month end?

Common misconceptions

  • "A nightly full copy is the safe option." It is simple, but it is slow on large tables, loads the source system, and still loses history unless you keep every night's copy.
  • "Deltas catch everything that changed." A timestamp delta only sees what its field records. Late commits, untouched timestamps and deletions all need explicit handling.
  • "Any interface that returns data is fine to use." SAP's 2026 API policy restricts large-scale extraction to endorsed architectures and names ODP-RFC as unpermitted for customer and partner use.
  • "The latest version of each record is the truth for training." It is the truth for today. A model must learn from what was known at prediction time, which needs history.
  • "Once the pipeline runs, the job is done." Without a reconciliation, a pipeline can fail silently for weeks while the model keeps scoring.

Key terms

All course terms, A to Z, in the glossary

  • CDS view: a core data services view, SAP's way to define a data model on top of database tables.
  • Extraction-enabled view: a CDS view annotated so a data platform can read it in full and in deltas.
  • ODP (operational data provisioning): SAP's framework that serves full and delta extraction from ABAP systems.
  • Delta: only the records that changed since the last run.
  • Safety interval: an overlap between delta runs so records committed late are not skipped.
  • Change data capture (CDC): logging every insert, update and delete as it happens, so the pipeline reads changes instead of guessing them from a timestamp.
  • Reconciliation: comparing the copy with the source to prove nothing is missing or extra.
  • Time travel: reading data as it was at a past moment, which needs every version kept.
  • Feature table: the table a model learns from, one row per case, with only facts known at prediction time.

Check yourself

Pick one answer for each question. The explanation appears after you choose.

1Why does a late-delivery model need more than a copy of today's delivery records?

Answer: B. A planned goods issue date can move after a delivery is created. If training reads only today's value, it sees information that wasn't known when the prediction would have been made, and test scores flatter the model.

2The data team proposes reading S/4HANA with ODP-RFC from a third-party tool. What should a leader ask first?

Answer: D. SAP's API policy of April 2026 classes ODP-RFC as unpermitted for customer and partner use, with an opt-out only until 31 December 2026. The route must be settled before any performance comparison matters.

3Calder's copy shows six in ten shipped deliveries as still open. What is the most likely cause?

Answer: C. The evening goods-issue job stamps its records before midnight but commits after the midnight run has read. Without a safety interval, the next run starts after those timestamps and never sees them.

4What does a reconciliation after each load protect against?

Answer: C. The reconciliation compares the copy with the source, so missed changes, leftover deletions and different values show up at once. Without it, the pipeline can be wrong for weeks before anyone notices the worklist.

5Before funding a new extraction pipeline, what should a program lead check first?

Answer: C. SAP's API policy names zero-copy data sharing as an endorsed pattern, and a data product saves building and running a pipeline. Extraction is for data no data product covers.

6A functional consultant adds a mass-update program to the delivery process. What should the test plan add?

Answer: D. Timestamp deltas only see records whose delta field moved. A program that changes records without setting it creates changes the pipeline never extracts.
Deep layer · 40 min read

Mental model

An extraction pipeline for AI has one job: let you answer "what did the business know at moment T?" for any T in the past, from a copy you can trust.

Three ideas carry that job:

Idea The question it answers What breaks without it
Delta What changed since the last run? Full copies every night: slow, heavy on the source, still no history
Key Which record does this change belong to, and is it still there? Duplicates, and deleted records that never leave
Version What did this record look like at moment T? Training reads today's values for past moments: leakage
flowchart LR
  S[(S/4HANA tables)] --> V[Extraction-enabled<br/>CDS view]
  V -->|full, then deltas| P[Pipeline run]
  P -->|merge by key,<br/>keep versions| H[(History table)]
  H -->|as of T| F[Feature table]
  H -->|compare| R{Reconcile<br/>with source}
  F --> M[Model]

The rest of this layer is about each arrow: how SAP serves the deltas, how a pipeline merges them, how history turns into features, and how to prove the copy is right.

How it works

Extraction-enabled CDS views

S/4HANA's data model is a virtual data model of CDS views on top of the database tables, as Reporting and analytics architecture describes. A view becomes a source for bulk extraction when it carries @Analytics.dataExtraction.enabled: true. SAP's BW/4HANA training says this generates an operational data provider, in the ODP context it calls ABAP_CDS, reached over RFC or HTTP (SAP Learning). In BW/4HANA the matching source system type is ODP_CDS, and some SAP pages use that name for the context as well.

ODP (operational data provisioning) is the framework underneath. A provider offers delta queues; subscribers order changes from them through subscriptions. With S/4HANA as the provider, the operational delta queue sits on S/4HANA, is monitored in transaction ODQMON, and keeps its data for a set period so a subscriber can retrieve records again (SAP Learning).

Which views can you use? On S/4HANA Cloud, SAP's training says the connection runs through an HTTP/SOAP communication scenario and the CDS route is the strategic interface; standard CDS extraction views were rolled out from Cloud release 1905 (SAP Learning). Datasphere's replication flows list C1-released views with the extraction annotation under the CDS_EXTRACTION container (SAP Learning). Treat "released and extraction-enabled" as the rule, and look up the view for your object and release in your system rather than guessing a name. SAP's example of a delivered extraction view is C_SALESDOCUMENTITEMDEX_1 for sales document items.

Two ways to find changes

SAP's pages describe two delta methods for CDS views.

Generic (timestamp) delta. You name one field, a UTC timestamp (preferred) or a date, with @Analytics.dataExtraction.delta.byElement.name. Each run selects records whose field is later than the last run's upper limit (SAP Learning). Three details decide whether it is correct:

  • Safety interval. A record can carry a timestamp from before a run but commit after it. @Analytics.dataExtraction.delta.byElement.maxDelayInSeconds (default 1,800 seconds) makes each run reach back that far, so records in the window are selected in two consecutive runs; ODP stores keys and hashes so unchanged ones aren't sent twice. SAP's training warns that a value set too low can miss records permanently.
  • Deletions. A deleted row simply disappears from the view, so a timestamp can't report it. Deletion detection stores every extracted key and sends a delete for keys that vanish. SAP limits it to small data sets (roughly under a million records) such as master data and texts, and ignoreDeletionAfterDays can ignore old deletions, for example from archiving. SAP's training pages spell the annotation both detectDeletedRecords (in the full annotation path) and detectDeletions; check the exact name in your release.
  • Empty or calculated fields. Records with an empty delta field are extracted only by the initial load. A calculated delta field can cause severe performance problems, so use a persisted field, and fill it when the record is created, not only on its first change (SAP Community).

Change data capture (CDC). Database triggers, built on SAP Landscape Transformation (SLT) technology, log every insert, update and delete on the underlying tables (SAP Learning). For a view on one table, @Analytics.dataExtraction.delta.changeDataCapture.automatic: true is enough. For a view with joins, changeDataCapture.mapping names which tables to log and how their keys map to the view's fields, with one main table and only left-outer-to-one joins. A deletion arrives as a record holding only the key and a deletion indicator, and only when the main table's record is deleted (SAP Learning). SAP's training dates CDC from S/4HANA Cloud 1905 and on-premise 1909 FP01, with DHCDCMON to monitor it.

Generic timestamp delta Change data capture
Finds changes by The delta field's value Triggers on the tables
Late commits Need a safety interval Logged at commit, no window needed
Deletions Optional key comparison, small data only Key-only delete records
Changes that don't touch the delta field Missed Caught, if the table is mapped
Cost Reads the view's recent rows Trigger and logging overhead on every write
Trap Too short a safety interval Mapping a table that changes often triggers deltas for many records

That last trap is why SAP's own C_SALESDOCUMENTITEMDEX_1 deliberately leaves the company code and sales organization tables out of its mapping: a change there would otherwise send every related sales record again.

Keys, merges and history

Whatever the method, a delta package is a list of record images keyed by the view's key. The pipeline merges by key:

  1. If the key is new, insert it.
  2. If the key exists and the image differs, close the current version and add the new one.
  3. If the image is the same, do nothing. This makes a run safe to repeat.
  4. If a delete arrives, close the current version and add a "deleted" marker.

Keeping closed versions instead of overwriting them is what the data warehouse world calls a type 2 slowly changing dimension. For AI it has a plainer name: it's how you get time travel. "The feature table as of 31 August" means: for each key, the versions that had begun by then, as the pipeline had loaded them by then.

Two limits are worth saying out loud. A delta run reads the record as it looks at extraction time, so two changes between runs reach your history as one version: history is only as fine as your run frequency. And the history starts when the pipeline starts. Older changes are gone unless the source keeps them (change documents, for example) and you extract those too.

From history to features

A feature table has one row per case, holding only what was known at the prediction moment, as Predicting late deliveries with SAP data explains. With history kept, "known at delivery creation" becomes a query: take each delivery's first version for the features, and its latest version for the outcome. Without history, you only have the latest version, and any field that moved after creation leaks.

At Calder, the planned goods issue date moves on about one delivery in sixteen. In the simulated data, 25% of shipped deliveries left after the date planned at creation, but only 19% after the latest plan, because planners moved the date to match reality. A model trained on the latest record learns a different, easier question.

Rules: what SAP allows

How the data leaves S/4HANA is also a contract question. As of the SAP API policy dated 27 April 2026, SAP's overview deck says large-scale data extraction or scraping, and integration with autonomous AI agents, is allowed only through an endorsed SAP architecture; customer-namespace OData services and CDS views remain permitted; and ODP-RFC is so far the only interface classed as unpermitted, with a security patch shipped from 9 June and a temporary opt-out until 31 December 2026 (SAP API policy overview). The SAP Note behind the ODP-RFC restriction, 3255746 (version 11, 21 April 2026), limits the RFC modules of the ODP data replication API to transfers between SAP applications and leaves the RFC protocol, table and CDS extraction and BAPIs usable; SAP points customers to Business Data Cloud and to OData (SAPinsider). The deck names zero-copy data sharing as an endorsed data pattern.

The policy text is short on detail for any one pipeline. Write down which interface you use, which views, how much and how often, and get SAP's confirmation in writing.

Build it yourself: a delta pipeline into a feature table

You will run one script that simulates three months of Calder deliveries changing inside S/4HANA, extracts them every six hours the way a timestamp delta or CDC would, merges them into a history table with every version kept, checks the copy against the source, and writes a feature table that the late-delivery predictor from Unit 2 reads. Then you will break it on purpose: no safety interval, no deletion detection, today's values instead of history.

Before you start: complete Set up your computer for this course and Set up for Unit 7, which give you the orchestrate-course folder, its .venv and the unit07 folder. Step 6 also uses unit02/predict_late.py from Predicting late deliveries with SAP data and the Unit 2 libraries; the rest needs only Python itself.

flowchart LR
  SIM[Simulated S/4HANA:<br/>deliveries change] --> EX[Extract every 6 h:<br/>generic delta or CDC]
  EX --> DB[(SQLite history table)]
  DB --> REC[Reconcile with source]
  DB --> FT[Feature table as of a date]
  FT --> J[deliveries_creation.json]
  J --> PL[unit02/predict_late.py]

What you need

  • The course folder, .venv and unit07 folder from the setup topics.
  • About 40 minutes. No SAP account, no key and no internet: everything is simulated on your computer. Free.
  • Python 3.10 or newer. The pipeline uses only built-in modules (sqlite3, json, hashlib, random).
  • For Step 6 only: unit02/predict_late.py and the Unit 2 libraries (pandas, scikit-learn 1.5 or newer).

The made-up data: deliveries numbered from 82000001, created on weekdays from 1 July to 30 September 2026 at Calder's Rockford shipping point RK01, for Brennock (10000001), Hallam (10000002), Ostervale (10000021), Pellmark (10000031) and other made-up customers, with pumps CP-ES200 and CP-ES300 and seal kits CP-SK200. Routes RK0001 to RK0003 and all codes are course labels, not SAP values. The random seed is fixed at 42, so your numbers match the ones below.

Step 1: Open the course folder and turn on the environment

  1. Open VS Code, choose File > Open Folder and open orchestrate-course.

  2. Open a terminal: Terminal > New Terminal.

  3. Turn on the virtual environment if the prompt doesn't start with (.venv):

    • Windows (PowerShell):

      .venv\Scripts\Activate.ps1
    • macOS / Linux:

      source .venv/bin/activate
  4. Make sure the unit07 folder exists (the Unit 7 setup creates it; this does no harm if it's there):

    • Windows (PowerShell):

      New-Item -ItemType Directory -Force unit07
    • macOS / Linux:

      mkdir -p unit07

Step 2: Save the script

  1. In VS Code's file list, right-click unit07, choose New File and name it cds_delta_pipeline.py.
  2. Paste the code below and save (Ctrl+S, or Cmd+S on Mac).
"""CDS extraction pipelines for AI: a delta pipeline from Calder's deliveries into a feature table.

How to run (from the unit07 folder, with the course .venv turned on):
    python cds_delta_pipeline.py                          # generic timestamp delta, 1,800 s safety interval
    python cds_delta_pipeline.py --safety 0               # no safety interval: watch late commits go missing
    python cds_delta_pipeline.py --detect-deletions       # also compare keys, so deleted deliveries leave
    python cds_delta_pipeline.py --mode cdc               # change data capture: every insert, update and delete
    python cds_delta_pipeline.py --as-of 2026-08-31       # time travel: the feature table as it looked that day
    python cds_delta_pipeline.py --snapshot current       # export today's records instead of the creation snapshot
    python cds_delta_pipeline.py --show 82000045          # every version the target keeps for one delivery

Everything is simulated on your computer with made-up Calder Pumpworks data (fictional). No SAP system,
key or internet is used. Field names follow SAP's outbound delivery API so unit02/predict_late.py can read
the export. Built-in Python modules only.
"""
import argparse
import hashlib
import json
import random
import sqlite3
from datetime import date, datetime, timedelta, timezone
from pathlib import Path

HERE = Path(__file__).parent
SEED = 42
START, END = date(2026, 7, 1), date(2026, 9, 30)            # deliveries are created in this window
RUN_EVERY_HOURS = 6                                          # extraction at 00:00, 06:00, 12:00, 18:00 UTC
LAST_RUN = datetime(2026, 10, 1, 6, 0, tzinfo=timezone.utc)  # the made-up "now"
GI_JOB_START = (23, 45)                                      # evening goods-issue job, UTC (18:45 in Rockford)

# Course labels, not SAP standard values.
CUSTOMERS = {"10000001": "RK0001", "10000002": "RK0002", "10000021": "RK0001", "10000031": "RK0003"}
OTHER_CUSTOMERS = [f"100000{n}" for n in list(range(22, 29)) + list(range(32, 37))]
MATERIALS = {"CP-ES200": (180.0, 1, 6), "CP-ES300": (260.0, 1, 4), "CP-SK200": (0.8, 10, 60)}
FIELDS = ["DeliveryDocument", "ShippingPoint", "ShippingType", "ShippingCondition", "DeliveryPriority",
          "ProposedDeliveryRoute", "ShipToParty", "CreationDate", "PlannedGoodsIssueDate",
          "ActualGoodsMovementDate", "HeaderGrossWeight", "HeaderWeightUnit", "Items", "LastChangeDateTime"]


# ---------------------------------------------------------------- 1. The source system (simulated)

def at(day: date, hour: int, minute: int = 0, second: int = 0) -> datetime:
    return datetime(day.year, day.month, day.day, hour, minute, second, tzinfo=timezone.utc)


def next_workday(day: date) -> date:
    while day.weekday() >= 5:
        day += timedelta(days=1)
    return day


def simulate_source(seed: int = SEED) -> list:
    """Return every change as (change_time, commit_time, op, key, record or None), in commit order.

    change_time is what the record's LastChangeDateTime says. commit_time is when another session can
    see it. The evening goods-issue job stamps records at 23:45 but commits once, 20 to 35 minutes later.
    """
    rng = random.Random(seed)
    events, number, day = [], 82000001, START
    while day <= END:
        if day.weekday() < 5:
            for _ in range(rng.randint(8, 14)):
                key = str(number)
                number += 1
                created = at(day, rng.randint(13, 21), rng.randint(0, 59), rng.randint(0, 59))
                customer = rng.choice(list(CUSTOMERS) + OTHER_CUSTOMERS)
                route = CUSTOMERS.get(customer, rng.choice(["RK0001", "RK0003"]))
                items, weight = [], 0.0
                for material in rng.sample(list(MATERIALS), rng.randint(1, 3)):
                    unit_kg, low, high = MATERIALS[material]
                    qty = rng.randint(low, high)
                    items.append({"Material": material, "ActualDeliveryQuantity": f"{qty:.3f}"})
                    weight += qty * unit_kg + (25.0 if material != "CP-SK200" else 1.5)
                lead = rng.randint(1, 8)
                planned = next_workday(day + timedelta(days=lead))
                shipping_type = rng.choices(["01", "02"], [0.8, 0.2])[0]
                priority = rng.choices(["02", "01"], [0.85, 0.15])[0]
                # A made-up rule plus noise decides how late goods issue is.
                risk = (-1.3 + 0.0009 * weight - 0.25 * lead + 0.35 * len(items)
                        + (1.1 if route == "RK0002" else 0.0) + (0.7 if planned.weekday() == 4 else 0.0)
                        + (-0.9 if priority == "01" else 0.0)
                        + (1.4 if route == "RK0002" and day >= date(2026, 8, 17) else 0.0)
                        + rng.gauss(0, 1.0))
                delay = (1 + int(rng.expovariate(0.7))) if risk > 0 else 0
                actual = next_workday(planned + timedelta(days=delay))
                record = {
                    "DeliveryDocument": key, "ShippingPoint": "RK01", "ShippingType": shipping_type,
                    "ShippingCondition": "01", "DeliveryPriority": priority, "ProposedDeliveryRoute": route,
                    "ShipToParty": customer, "CreationDate": day.isoformat(),
                    "PlannedGoodsIssueDate": planned.isoformat(), "ActualGoodsMovementDate": None,
                    "HeaderGrossWeight": f"{weight:.3f}", "HeaderWeightUnit": "KG",
                    "Items": items, "LastChangeDateTime": created.isoformat(),
                }
                events.append((created, created + timedelta(seconds=rng.randint(0, 3)), "insert", key, dict(record)))

                if rng.random() < 0.03:                                   # created by mistake, deleted later
                    gone = created + timedelta(hours=rng.randint(2, 30))
                    events.append((gone, gone + timedelta(seconds=1), "delete", key, None))
                    continue
                if delay >= 2 and rng.random() < 0.6:                     # the planner moves the date
                    when = at(next_workday(planned - timedelta(days=1)), rng.randint(13, 21), rng.randint(0, 59))
                    if when > created:
                        record = dict(record, PlannedGoodsIssueDate=actual.isoformat(),
                                      LastChangeDateTime=when.isoformat())
                        events.append((when, when + timedelta(seconds=2), "update", key, dict(record)))
                if actual <= END:                                         # goods issue posted
                    if rng.random() < 0.6:
                        posted = at(actual, *GI_JOB_START) + timedelta(seconds=rng.randint(0, 299))
                        commit = at(actual, 23, 59) + timedelta(minutes=rng.randint(21, 36))
                    else:
                        posted = at(actual, rng.randint(13, 21), rng.randint(0, 59))
                        commit = posted + timedelta(seconds=rng.randint(0, 5))
                    record = dict(record, ActualGoodsMovementDate=actual.isoformat(),
                                  LastChangeDateTime=posted.isoformat())
                    events.append((posted, commit, "update", key, dict(record)))
        day += timedelta(days=1)
    return sorted(events, key=lambda e: (e[1], e[3]))


def source_state(events: list, now: datetime) -> dict:
    """What a SELECT on the extraction view returns at `now`: the latest committed image of each key."""
    state = {}
    for _, commit, op, key, record in events:
        if commit > now:
            break
        if op == "delete":
            state.pop(key, None)
        else:
            state[key] = record
    return state


def image_hash(record: dict) -> str:
    return hashlib.sha256(json.dumps(record, sort_keys=True).encode()).hexdigest()[:16]


# ---------------------------------------------------------------- 2. Two ways to read changes

class GenericDelta:
    """Timestamp delta, like @Analytics.dataExtraction.delta.byElement on LastChangeDateTime.

    Each run selects records with low < LastChangeDateTime <= now, where low is the previous upper limit
    minus the safety interval. Records in the overlap that were already sent unchanged are skipped,
    using stored keys and hashes. Optional deletion detection compares all keys with what was sent.
    """
    def __init__(self, safety_seconds: int, detect_deletions: bool):
        self.safety = timedelta(seconds=safety_seconds)
        self.detect_deletions = detect_deletions
        self.high = None                     # upper limit of the previous run
        self.window = {}                     # key -> (timestamp, hash) sent inside the safety window
        self.known_keys = set()              # every key sent and not yet deleted
        self.skipped = 0

    def run(self, events: list, now: datetime) -> list:
        view = source_state(events, now)
        if self.high is None:                                   # initial load: everything
            package = [("upsert", rec) for rec in view.values()]
        else:
            low = self.high - self.safety
            package = []
            for key, rec in view.items():
                ts = datetime.fromisoformat(rec["LastChangeDateTime"])
                if not (low < ts <= now):
                    continue
                if ts <= self.high and self.window.get(key) == (ts, image_hash(rec)):
                    self.skipped += 1                            # seen in the overlap, unchanged
                    continue
                package.append(("upsert", rec))
        for _, rec in package:
            ts = datetime.fromisoformat(rec["LastChangeDateTime"])
            self.window[rec["DeliveryDocument"]] = (ts, image_hash(rec))
            self.known_keys.add(rec["DeliveryDocument"])
        if self.detect_deletions:                               # costs a full read of every key, every run
            for key in sorted(self.known_keys - set(view)):
                package.append(("delete", {"DeliveryDocument": key}))
                self.known_keys.discard(key)
        self.high = now
        self.window = {k: v for k, v in self.window.items() if v[0] > now - self.safety}
        return package


class ChangeDataCapture:
    """Change data capture, like @Analytics.dataExtraction.delta.changeDataCapture.

    Database triggers log the key of every insert, update and delete. Each run reads the log entries
    committed since the last run, then sends the current image of each key, or a key-only delete image.
    """
    def __init__(self):
        self.position = 0                    # how far into the change log the last run read
        self.skipped = 0

    def run(self, events: list, now: datetime) -> list:
        view = source_state(events, now)
        changed = []
        while self.position < len(events) and events[self.position][1] <= now:
            key = events[self.position][3]
            if key not in changed:
                changed.append(key)
            self.position += 1
        return [("upsert", view[k]) if k in view else ("delete", {"DeliveryDocument": k}) for k in changed]


# ---------------------------------------------------------------- 3. The target: history with time travel

SCHEMA = """
CREATE TABLE delivery_history (
    DeliveryDocument TEXT NOT NULL,
    valid_from TEXT NOT NULL,          -- when the source says this version began (or when a delete was seen)
    valid_to TEXT,                     -- NULL while it is the current version
    loaded_at TEXT NOT NULL,           -- the extraction run that brought it
    is_deleted INTEGER NOT NULL DEFAULT 0,
    image_hash TEXT NOT NULL,
    record TEXT NOT NULL,
    PRIMARY KEY (DeliveryDocument, valid_from, loaded_at)
);
CREATE TABLE extraction_runs (run_at TEXT PRIMARY KEY, upserts INTEGER, deletes INTEGER);
"""


def apply_package(db: sqlite3.Connection, package: list, run_at: datetime) -> None:
    """Merge one delta package by key: close the current version, add the new one. Idempotent."""
    upserts = deletes = 0
    for op, rec in package:
        key = rec["DeliveryDocument"]
        current = db.execute("SELECT valid_from, image_hash, is_deleted FROM delivery_history "
                             "WHERE DeliveryDocument = ? AND valid_to IS NULL", (key,)).fetchone()
        if op == "delete":
            if current is None or current[2]:
                continue
            valid_from, new_hash, deleted = run_at.isoformat(), "deleted", 1
            deletes += 1
        else:
            valid_from, new_hash, deleted = rec["LastChangeDateTime"], image_hash(rec), 0
            if current and current[1] == new_hash:
                continue                                         # same image again: nothing to do
            upserts += 1
        if current:
            db.execute("UPDATE delivery_history SET valid_to = ? WHERE DeliveryDocument = ? AND valid_to IS NULL",
                       (valid_from, key))
        db.execute("INSERT INTO delivery_history VALUES (?, ?, NULL, ?, ?, ?, ?)",
                   (key, valid_from, run_at.isoformat(), deleted, new_hash, json.dumps(rec)))
    db.execute("INSERT INTO extraction_runs VALUES (?, ?, ?)", (run_at.isoformat(), upserts, deletes))
    db.commit()


def versions_as_of(db: sqlite3.Connection, as_of: datetime) -> dict:
    """Time travel: every key's versions that had begun by `as_of`, oldest first, as the target knew them."""
    rows = db.execute("SELECT DeliveryDocument, valid_from, is_deleted, record FROM delivery_history "
                      "WHERE valid_from <= ? AND loaded_at <= ? ORDER BY DeliveryDocument, valid_from",
                      (as_of.isoformat(), as_of.isoformat())).fetchall()
    out = {}
    for key, _, deleted, record in rows:
        out.setdefault(key, []).append(None if deleted else json.loads(record))
    return out


# ---------------------------------------------------------------- 4. Feature table and export

def odata_date(iso_day):
    if not iso_day:
        return None
    d = date.fromisoformat(iso_day)
    return f"/Date({int(datetime(d.year, d.month, d.day, tzinfo=timezone.utc).timestamp() * 1000)})/"


def feature_rows(versions: dict, snapshot: str, as_of: datetime) -> list:
    """One row per delivery that still exists: features from the first version (or the latest),
    the label field from the latest version, but only if goods issue was known by `as_of`."""
    rows = []
    for key, history in versions.items():
        latest = history[-1]
        if latest is None:                                       # deleted in the source
            continue
        first = next(v for v in history if v is not None)
        base = dict(first if snapshot == "creation" else latest)
        gi = latest["ActualGoodsMovementDate"]
        base["ActualGoodsMovementDate"] = gi if gi and date.fromisoformat(gi) <= as_of.date() else None
        rows.append(base)
    return rows


def export_odata(rows: list, path: Path) -> None:
    """Write the rows in the OData V2 JSON shape that unit02/predict_late.py --file reads."""
    results = []
    for r in rows:
        results.append({
            **{f: r[f] for f in FIELDS if f not in ("Items", "LastChangeDateTime", "CreationDate",
                                                     "PlannedGoodsIssueDate", "ActualGoodsMovementDate")},
            "CreationDate": odata_date(r["CreationDate"]),
            "PlannedGoodsIssueDate": odata_date(r["PlannedGoodsIssueDate"]),
            "ActualGoodsMovementDate": odata_date(r["ActualGoodsMovementDate"]),
            "to_DeliveryDocumentItem": {"results": [
                {"DeliveryDocument": r["DeliveryDocument"], "DeliveryDocumentItem": f"{(i + 1) * 10:06d}", **item}
                for i, item in enumerate(r["Items"])]},
        })
    path.write_text(json.dumps({"d": {"results": results}}, indent=1), encoding="utf-8")


def late_share(rows: list, plan_field_rows: list) -> float:
    """Share of labelled rows whose goods issue came after the plan in the given rows."""
    plan = {r["DeliveryDocument"]: r["PlannedGoodsIssueDate"] for r in plan_field_rows}
    labelled = [r for r in rows if r["ActualGoodsMovementDate"]]
    late = sum(r["ActualGoodsMovementDate"] > plan[r["DeliveryDocument"]] for r in labelled)
    return late / max(len(labelled), 1)


# ---------------------------------------------------------------- 5. Run it

def main() -> None:
    parser = argparse.ArgumentParser(description="Simulated CDS delta extraction into a feature table.")
    parser.add_argument("--mode", choices=["generic", "cdc"], default="generic", help="how changes are read")
    parser.add_argument("--safety", type=int, default=1800, help="safety interval in seconds (generic mode)")
    parser.add_argument("--detect-deletions", action="store_true", help="compare keys to find deletions")
    parser.add_argument("--as-of", help="time travel: build the feature table as known at the end of this day")
    parser.add_argument("--snapshot", choices=["creation", "current"], default="creation",
                        help="features from the first version (creation) or the latest one (current)")
    parser.add_argument("--show", help="print every stored version of one delivery number")
    parser.add_argument("--seed", type=int, default=SEED, help="random seed for the made-up data")
    args = parser.parse_args()

    events = simulate_source(args.seed)
    extractor = (ChangeDataCapture() if args.mode == "cdc"
                 else GenericDelta(args.safety, args.detect_deletions))
    db_path = HERE / f"calder_features_{args.mode}.db"
    db_path.unlink(missing_ok=True)
    db = sqlite3.connect(db_path)
    db.executescript(SCHEMA)

    run_at, runs, sent = at(START, 0), 0, 0
    while run_at <= LAST_RUN:
        package = extractor.run(events, run_at)
        apply_package(db, package, run_at)
        runs, sent = runs + 1, sent + len(package)
        run_at += timedelta(hours=RUN_EVERY_HOURS)

    label = ("change data capture" if args.mode == "cdc" else
             f"generic timestamp delta, safety interval {args.safety:,} s, deletions "
             + ("detected" if args.detect_deletions else "not detected"))
    created = sum(1 for e in events if e[2] == "insert")
    deleted = sum(1 for e in events if e[2] == "delete")
    late_commits = sum(1 for e in events if (e[1] - e[0]).total_seconds() > 60)
    print(f"Calder delivery extraction ({label})")
    print(f"  Source: {created:,} deliveries created {START} to {END}, {len(events):,} changes, "
          f"{deleted} deleted, {late_commits} committed over a minute after their timestamp")
    print(f"  {runs} extraction runs every {RUN_EVERY_HOURS} h, {sent:,} records sent"
          + (f", {extractor.skipped:,} skipped as already sent" if args.mode == "generic" else ""))

    # Reconciliation: compare the target's current versions with the source at the last run.
    truth = source_state(events, LAST_RUN)
    target = {k: v for k, v in ((k, h[-1]) for k, h in versions_as_of(db, LAST_RUN).items()) if v is not None}
    phantoms = sorted(set(target) - set(truth))
    missing = sorted(set(truth) - set(target))
    stale = sorted(k for k in set(truth) & set(target) if image_hash(truth[k]) != image_hash(target[k]))
    print(f"\nReconciliation with the source at {LAST_RUN:%Y-%m-%d %H:%M} UTC")
    print(f"  Deliveries: source {len(truth):,}, target {len(target):,}")
    print(f"  Missing in target: {len(missing)}   still in target but deleted in source: {len(phantoms)}"
          f"   different image: {len(stale)}")
    if stale:
        k = stale[0]
        print(f"  Example {k}: source goods issue {truth[k]['ActualGoodsMovementDate']}, "
              f"target {target[k]['ActualGoodsMovementDate']} (changed {truth[k]['LastChangeDateTime'][:16]})")
    print("  Result: " + ("MATCH" if not (phantoms or missing or stale) else "MISMATCH, fix before training"))

    # Feature table, as known at the end of the --as-of day (default: the last run).
    as_of = (datetime.fromisoformat(args.as_of).replace(tzinfo=timezone.utc) + timedelta(days=1, seconds=-1)
             if args.as_of else LAST_RUN)
    versions = versions_as_of(db, as_of)
    rows = feature_rows(versions, args.snapshot, as_of)
    current_rows = feature_rows(versions, "current", as_of)
    moved = sum(1 for h in versions.values() if h[-1] is not None and len({v["PlannedGoodsIssueDate"]
                                                                            for v in h if v}) > 1)
    labelled = sum(1 for r in rows if r["ActualGoodsMovementDate"])
    print(f"\nFeature table as known at {as_of:%Y-%m-%d %H:%M} UTC ({args.snapshot} snapshot)")
    print(f"  {len(rows):,} deliveries: {labelled:,} with goods issue (labelled), {len(rows) - labelled:,} open")
    print(f"  Planned goods issue date moved after creation: {moved}")
    print(f"  Late against the plan at creation: {late_share(rows, feature_rows(versions, 'creation', as_of)):.0%}"
          f"   against the latest plan: {late_share(rows, current_rows):.0%}")
    if args.show:
        print(f"\nVersions of delivery {args.show} in the target (valid from, loaded by run, planned GI, actual GI)")
        rows_shown = db.execute("SELECT valid_from, valid_to, loaded_at, is_deleted, record FROM delivery_history "
                                "WHERE DeliveryDocument = ? ORDER BY valid_from", (args.show,)).fetchall()
        for valid_from, valid_to, loaded_at, deleted, record in rows_shown:
            rec = json.loads(record)
            what = "deleted" if deleted else f"{rec['PlannedGoodsIssueDate']}  {rec['ActualGoodsMovementDate'] or '-'}"
            print(f"  {valid_from[:16]}  {loaded_at[:16]}  {what}" + ("" if valid_to else "   (current)"))
        if not rows_shown:
            print("  None: that number was never extracted. Delivery numbers here start at 82000001.")
    out = HERE / ("deliveries_" + args.snapshot + (f"_{args.as_of}" if args.as_of else "") + ".json")
    export_odata(rows, out)
    print(f"Wrote {out.name} and {db_path.name}")


if __name__ == "__main__":
    main()

Step 3: Run the default pipeline

Run, from the orchestrate-course folder (the same command on Windows, macOS and Linux):

python unit07/cds_delta_pipeline.py

You should see:

Calder delivery extraction (generic timestamp delta, safety interval 1,800 s, deletions not detected)
  Source: 721 deliveries created 2026-07-01 to 2026-09-30, 1,447 changes, 19 deleted, 400 committed over a minute after their timestamp
  370 extraction runs every 6 h, 1,425 records sent, 55 skipped as already sent

Reconciliation with the source at 2026-10-01 06:00 UTC
  Deliveries: source 702, target 720
  Missing in target: 0   still in target but deleted in source: 18   different image: 0
  Result: MISMATCH, fix before training

Feature table as known at 2026-10-01 06:00 UTC (creation snapshot)
  720 deliveries: 661 with goods issue (labelled), 59 open
  Planned goods issue date moved after creation: 44
  Late against the plan at creation: 25%   against the latest plan: 19%
Wrote deliveries_creation.json and calder_features_generic.db

Read it line by line:

  • 400 committed over a minute after their timestamp. These are the evening goods-issue job's postings: stamped from 23:45 UTC, committed after the midnight run. The 1,800-second safety interval caught every one of them (different image: 0).
  • 55 skipped as already sent. Records in the overlap window that hadn't changed. That's the key-and-hash check doing its job.
  • 18 still in target but deleted in source. Timestamp deltas can't see deletions. These phantoms show up as "open" deliveries in the feature table. (19 were deleted; one was created and deleted between two runs, so it never reached the copy.)

The reconciliation says MISMATCH, which is correct: don't train on this copy yet.

Step 4: Break it, then fix it

  1. Remove the safety interval:

    python unit07/cds_delta_pipeline.py --safety 0
    Reconciliation with the source at 2026-10-01 06:00 UTC
      Deliveries: source 702, target 720
      Missing in target: 0   still in target but deleted in source: 18   different image: 400
      Example 82000002: source goods issue 2026-07-08, target None (changed 2026-07-08T23:48)
      Result: MISMATCH, fix before training
    
    Feature table as known at 2026-10-01 06:00 UTC (creation snapshot)
      720 deliveries: 261 with goods issue (labelled), 459 open

    All 400 late commits are lost for good: each later run starts after their timestamps. Six in ten shipped deliveries now look open. This is the failure the business section described.

  2. Turn on deletion detection, with the safety interval back:

    python unit07/cds_delta_pipeline.py --detect-deletions

    The reconciliation now reads source 702, target 702, all zeros, Result: MATCH, and the feature table has 702 deliveries, 41 of them open.

  3. Try change data capture instead:

    python unit07/cds_delta_pipeline.py --mode cdc
    Calder delivery extraction (change data capture)
      Source: 721 deliveries created 2026-07-01 to 2026-09-30, 1,447 changes, 19 deleted, 400 committed over a minute after their timestamp
      370 extraction runs every 6 h, 1,444 records sent
    
    Reconciliation with the source at 2026-10-01 06:00 UTC
      Deliveries: source 702, target 702
      Missing in target: 0   still in target but deleted in source: 0   different image: 0
      Result: MATCH

    CDC reads a log of committed changes, so it needs no safety window and sees deletions without comparing keys. It writes its own database, calder_features_cdc.db.

Step 5: Travel in time

  1. Look at one delivery's history:

    python unit07/cds_delta_pipeline.py --detect-deletions --show 82000045

    The last lines:

    Versions of delivery 82000045 in the target (valid from, loaded by run, planned GI, actual GI)
      2026-07-06T21:29  2026-07-07T00:00  2026-07-09  -
      2026-07-08T13:35  2026-07-08T18:00  2026-07-15  -
      2026-07-15T23:46  2026-07-16T06:00  2026-07-15  2026-07-15   (current)

    Created on 6 July with goods issue planned for 9 July, moved to 15 July on 8 July, shipped on 15 July by the evening job (picked up by the 06:00 run thanks to the overlap). Against the plan at creation it was late; against the latest plan it was on time.

  2. Build the feature table as it looked at the end of August:

    python unit07/cds_delta_pipeline.py --detect-deletions --as-of 2026-08-31
    Feature table as known at 2026-08-31 23:59 UTC (creation snapshot)
      457 deliveries: 399 with goods issue (labelled), 58 open
      Planned goods issue date moved after creation: 24
      Late against the plan at creation: 23%   against the latest plan: 18%
    Wrote deliveries_creation_2026-08-31.json and calder_features_generic.db

    This is the training set a run on 31 August would have used, rebuilt exactly. You can repeat a past training run, or audit why the model said what it said on a given day.

Step 6: Feed the late-delivery predictor

  1. Rebuild the clean copy, then hand the feature table to the Unit 2 predictor:

    python unit07/cds_delta_pipeline.py --detect-deletions
    python unit02/predict_late.py --file unit07/deliveries_creation.json
  2. The predictor reads the file like any OData V2 extract. The first lines:

    Source: unit07/deliveries_creation.json. 702 deliveries read.
      With a goods issue date (labelled): 661. Still open: 41.
      25% of labelled deliveries had goods issue after the planned date.

    It then trains, tests on later deliveries and writes unit02/at_risk_deliveries.csv. On this data it reported a test ROC AUC of 0.80, with weights pointing at Friday goods issues and route RK0002 (Hallam's long lane). Model scores can differ slightly between scikit-learn versions.

  3. Now try the snapshot of today's records:

    python unit07/cds_delta_pipeline.py --detect-deletions --snapshot current
    python unit02/predict_late.py --file unit07/deliveries_current.json

    The late share drops to 19%: the moved dates make deliveries look on time. Neither file is "wrong", but they answer different questions. Write in your metric contract which plan "late" is measured against, and build features only from the creation snapshot.

Step 7: Save your work in Git

git add unit07/cds_delta_pipeline.py
git commit -m "Unit 7: CDS delta pipeline into a feature table"

The database and JSON files are rebuilt on every run; you don't need to commit them.

What each part of the script does

Part What it does
simulate_source Makes three months of delivery changes (create, move date, goods issue, delete), each with the time it is stamped and the time it commits
source_state What a read of the extraction view returns at a moment: only committed changes
GenericDelta Timestamp delta: reads from the last upper limit minus the safety interval, skips unchanged records already sent, optionally compares keys to find deletions
ChangeDataCapture Reads the log of committed changes since the last run; sends the current image or a key-only delete
apply_package Merges by key into delivery_history, closing old versions; repeating a run changes nothing
versions_as_of Time travel: the versions that had begun and had been loaded by a given moment
feature_rows Features from the first version (or latest, with --snapshot current), goods issue from the latest
export_odata Writes the OData V2 JSON shape that predict_late.py --file reads
Reconciliation in main Compares the target's current versions with the source: missing, phantom and different records

If something goes wrong

What you see What it means What to do
python is not recognized, or command not found Python isn't on the path, or the environment is off Turn on .venv (Step 1); on macOS or Linux try python3. See the course setup topic
can't open file ... cds_delta_pipeline.py You're in the wrong folder, or the file name differs Run from orchestrate-course; check the file is unit07/cds_delta_pipeline.py
PermissionError on the .db file (Windows) Another program has the database open Close any database viewer, then run again
ModuleNotFoundError: No module named 'pandas' in Step 6 The Unit 2 libraries aren't in this environment pip install -r requirements.txt from the course folder
No such file for unit02/predict_late.py You haven't done the Unit 2 topic yet Do Steps 1 to 5 now; Step 6 can wait
Numbers differ from this page A different seed, or an edited script Run without --seed; compare your script with the one above
Network or proxy errors Not expected: nothing here uses the internet Check you're running this script, not one that calls the sandbox

The SAP way

Choosing the route

As of November 2026, these are the routes the SAP pages opened for this topic describe, in the order to consider them:

  1. A data product in SAP Business Data Cloud. If SAP already publishes the data as a product, share it without a copy. The API policy deck lists zero-copy sharing among endorsed patterns. SAP data on a lakehouse covers data products and sharing them with other platforms.
  2. A replication flow in SAP Datasphere from a released, extraction-enabled CDS view (container CDS_EXTRACTION), with load type Initial and Delta at an interval you set in the Data Integration Monitor (SAP Learning). The same page lists Initial Only and Delta Only, and a SQL_SERVICE container for CDS view entities exposed through the ABAP SQL service. When the target is HANA Data Lake Files, data lands in a buffer and needs a merge task before it shows.
  3. BW/4HANA reading the same views through ODP, if your data warehouse is already there (SAP Learning).
  4. A non-SAP tool. Check its interface against the API policy. The ODP-RFC route is restricted, and the opt-out ends on 31 December 2026; SAPinsider reports that SAP points to Business Data Cloud and OData for external data access (SAPinsider). Confirm any such route with SAP in writing.

Polling a transactional OData API such as API_OUTBOUND_DELIVERY_SRV row by row is fine for a small sandbox exercise and for reading single records. It is not a bulk extraction route: the API policy deck lists rate limits and quotas among its controls, and it allows large-scale extraction only through an endorsed SAP architecture.

Making your own view extraction-enabled

On S/4HANA on-premise and Private Edition, a developer can add the extraction annotations to a custom CDS view in the customer namespace. A sketch, using the annotation names from SAP's pages:

// Sketch: needs an S/4HANA system and ABAP Development Tools; not runnable here.
@Analytics.dataCategory: #FACT
@Analytics.dataExtraction.enabled: true
@Analytics.dataExtraction.delta.changeDataCapture.automatic: true
define view entity ZI_CalderDeliveryFeatures
  as select from ... // a single-table projection, so automatic CDC applies

For a view with joins, use changeDataCapture.mapping instead of automatic, and map only the tables whose changes should really send the record again. For a generic delta, use delta.byElement.name on a persisted UTC timestamp and keep maxDelayInSeconds at least at the default unless you have measured your longest commit delay.

Monitoring

ODQMON shows queues, subscriptions, requests and units of the delta queue; DHCDCMON monitors CDC; RODPS_REPL_TEST tests a replication from the source side (SAP Learning). In Datasphere, the Data Integration Monitor shows replication flow runs.

Licensing notes

The SAP pages opened for this topic don't state prices for extraction itself. Replication flows to non-SAP targets, Business Data Cloud and data products are subscription questions: ask SAP what your contract includes, and record the answer next to the ADR below.

Build vs. SAP

Need Build your own SAP route Choose SAP when
Data already published as a data product Extract it anyway Share the data product Almost always: no pipeline to run
Delta copy of a released view Your own extractor Replication flow, Initial and Delta The view is released and extraction-enabled
History of every version Merge-by-key with versions, as in the script Keep versions in the target (Datasphere, a lakehouse table format) Always keep history somewhere; the source delta only sends current images
Data no released view covers A custom view plus your pipeline Custom extraction-enabled view read by a replication flow Your landscape allows custom CDS views, and SAP has confirmed the route
Reconciliation Count and key checks after each run Monitor run status in Datasphere or ODQMON Neither replaces a business reconciliation; build that either way

Architecture view

Where it sits. For Calder, the source is CP10's S/4HANA system on RISE. A replication flow in Datasphere (inside Business Data Cloud, per Calder's reporting decision in Reporting and analytics architecture) reads a released delivery extraction view, lands a history table, and a transformation builds the feature table the model reads. CP20's ECC data joins only after the July 2027 rollout.

flowchart LR
  subgraph S4[CP10 S/4HANA]
    T[(Delivery tables)] --> CV[Released extraction view]
  end
  CV -->|replication flow,<br/>initial and delta| H[(History table<br/>Datasphere)]
  H --> FT[(Feature table)]
  FT --> ML[Model training<br/>and scoring]
  H --> RC[Reconciliation job]
  RC -->|alert| OPS[Data owner]

Non-functional requirements.

Requirement Typical target What breaks if missed
Completeness Reconciliation matches after every run Missed goods issues; the worklist shows shipped deliveries
Freshness Delta at least as often as scoring (here every 6 h) The model scores yesterday's picture
Source load Delta only; no full loads during month-end close Order entry slows at the busiest time
History retention Every version for at least the training window plus audit period Past training runs can't be rebuilt or explained
Security Feature table has its own access control in the target Anyone with platform access reads delivery data the source restricted

Integration and identity. On S/4HANA Cloud the extraction runs over a communication scenario, a system-to-system connection, not as a person (SAP Learning). S/4HANA's authorizations therefore don't follow the data: the target must apply its own rules, as Grounding on SAP data with authorizations explains. No events are needed; the replication flow's delta is the integration.

The decision to record.

  • Context: The late-delivery model needs three months of delivery history with versions, refreshed every six hours, from CP10. For this course, assume no data product covers the fields the model needs; check yours.
  • Options: A, replication flow from a released extraction view into Datasphere, with history kept in the target. B, a third-party tool over ODP-RFC. C, paging the transactional OData API every six hours. D, a custom view with CDC.
  • Decision: A. B is classed as unpermitted under SAP Note 3255746, and the temporary opt-out ends on 31 December 2026. C is not meant for bulk extraction and misses deletions. D only if no released view covers a field, with SAP's confirmation.
  • Consequences: History is kept in Datasphere, not the source. A reconciliation job runs after each delta and blocks retraining on mismatch. The route and volumes are confirmed with SAP in writing.

What a design review will ask.

  1. Is there a data product for this, and if so, why a pipeline?
  2. Which view, which delta method, and what safety interval, measured against which commit delays?
  3. How do deletions and archiving reach the copy?
  4. Can you rebuild last month's training set exactly?
  5. Who is alerted when reconciliation fails, and what stops?

Production concerns

  • Security and SAP authorizations. The extraction user reads everything in the view. Restrict the view to the fields the model needs, keep personal data out of feature tables unless the use case needs it, and apply access control in the target. Data security and PII covers masking.
  • Policy. Record the interface, views, volume and frequency, and SAP's written confirmation that the route fits the API policy. Revisit it when the policy changes.
  • Evaluation. Train only on snapshots as of a past date (Step 5). Keep each training set's as-of date with the model version so results can be repeated.
  • Cost. Deltas, not full loads. Run frequency drives both freshness and source load; match it to how often the model scores.
  • Operations. Alert on reconciliation mismatch, on a run that sends zero records on a working day, and on a delta queue that grows without being read. Plan a re-initialization (a fresh initial load) for when the delta breaks, and know how long it takes.
  • Clean core. Prefer released views. A custom extraction view in the customer namespace is permitted under the policy, but it is code you maintain through upgrades.

Pitfalls

  • Safety interval sized to the edge. The evening job stamps records from 23:45 and commits them after the midnight run. A 900-second window reaches back exactly to 23:45 and misses records stamped on that second; SAP's 1,800-second default has room to spare. Measure your longest gap between stamp and commit, then add margin.
  • Fields changed without the delta field. A mass update or custom program that doesn't set the last-changed timestamp produces changes a timestamp delta never sees. CDC catches them, if the table is mapped.
  • Deletion detection on big tables. SAP limits it to small data sets. For transaction data, use CDC or a separate deletion feed.
  • Archiving read as deletion. Archived records disappear from the view. Use ignoreDeletionAfterDays or a CDC setup that tells archive from delete, and tell the data team before archiving runs.
  • Overwriting instead of versioning. The copy looks correct today and can't answer "what did we know then?" Training on it leaks.
  • Trusting the run status. "Finished" means the run ended, not that the copy is complete. Reconcile.
  • Building on a route about to close. ODP-RFC opt-outs end on 31 December 2026.

Exercise

Size the safety interval and write the extraction contract.

  1. Run the pipeline with deletion detection and safety intervals of 600, 900, 901 and 1,200 seconds:

    python unit07/cds_delta_pipeline.py --detect-deletions --safety 600
    python unit07/cds_delta_pipeline.py --detect-deletions --safety 900
    python unit07/cds_delta_pipeline.py --detect-deletions --safety 901
    python unit07/cds_delta_pipeline.py --detect-deletions --safety 1200
  2. Note the different image count for each. Find in simulate_source when the evening job stamps its records and when the midnight run reads. Explain in one sentence why 900 seconds misses only a few records and 901 misses none.

  3. Explain in one more sentence why you would still choose 1,800 seconds.

  4. Create unit07/extraction_contract.md with these headings, filled in for Calder's delivery pipeline: Source view and release, Route and API policy confirmation, Delta method and safety interval, Deletions and archiving, Run frequency, History kept and for how long, Reconciliation and who is alerted, Snapshot used for features.

  5. Commit both:

    git add unit07/extraction_contract.md
    git commit -m "Unit 7: extraction contract for the delivery pipeline"

Done when your results show a mismatch at 600 and 900 seconds and a match at 901 and 1,200, your two sentences explain why, and extraction_contract.md fills all eight headings. The contract becomes the data section of the late delivery predictor project.

Check yourself

Pick one answer for each question. The explanation appears after you choose.

1A generic delta runs at 00:00 with no safety interval. A record stamped 23:47 commits at 00:20. What happens?

Answer: B. The 00:00 run can't see an uncommitted record, and the next run selects only timestamps after 00:00. Without an overlap, the record falls between runs for good, which is why SAP's training warns that too low a value can miss records permanently.

2What does maxDelayInSeconds change in an ODP timestamp delta?

Answer: B. It sets the safety interval: each run selects records from the last upper limit minus that many seconds. ODP stores keys and hashes so unchanged records in the overlap aren't sent again. The default is 1,800 seconds.

3The pipeline's reconciliation shows 18 deliveries in the target that the source no longer has. Which setup most likely produced this?

Answer: D. A deleted row disappears from the view, so a timestamp can't report it. Without deletion detection the copy keeps phantoms. CDC sends key-only delete records, and too short a safety interval loses changes rather than keeping deleted rows.

4Why does the feature export take features from each delivery's first version but the goods issue date from its latest?

Answer: C. The prediction moment is delivery creation, so features come from the version that existed then. The outcome is only known later, from the latest version. Using the latest planned date as a feature leaks the planner's later knowledge.

5Your custom CDS view joins delivery items to the company code table and uses automatic change data capture. Deltas are huge. What would you change?

Answer: C. Automatic CDC is for single-table views; joins need a mapping. Mapping only tables whose changes should resend a record avoids a company code change triggering every related item, as SAP did for C_SALESDOCUMENTITEMDEX_1.

6A third-party tool reads S/4HANA with ODP-RFC today. It is November 2026. What would you do first?

Answer: D. SAP classes ODP-RFC as unpermitted for customer and partner use, with an opt-out only until 31 December 2026. Wrapping it doesn't change the interface, and transactional APIs aren't a bulk extraction route; the replacement needs SAP's confirmation.

7What does keeping every version of each record in the target let you do that the source delta alone cannot?

Answer: D. Versions with valid-from and load times let you query the copy as of any past moment, as --as-of does. History starts when the pipeline starts, and changes between runs collapse into one image, so it can't recover those.

8Which extraction user design fits a feature table built for a planner worklist?

Answer: B. Extraction runs as a technical communication user, so S/4HANA's authorizations don't travel with the data. The target must enforce who may read the feature table and the worklist built from it.

Sources

Sign in to track your progress

We'll email you a one-time sign-in link. No password needed.

or

Tell us a little about you

Optional, every field. It helps us pitch answers to your questions at the right level and decide which topics to write next. It is never shown publicly, and you can change or clear it anytime from the account menu.

SAP areas you work in