Skip to main content

SAP Datasphere

Overview​

The SAP Datasphere connector extracts metadata from SAP Datasphere — SAP's cloud-native data warehouse platform (the successor to SAP Data Warehouse Cloud). It reads the SAP-supported REST/OData v4 Catalog and Consumption APIs (authenticated via XSUAA OAuth), so no JDBC driver or SQL dialect is required.

DataHub ingests Spaces as containers and Views, Analytic Models, and Local Tables as datasets, with column schema read from the per-asset OData EDMX ($metadata) surface. Optional features include table- and column-level lineage (include_lineage), SAP CDS semantic annotations emitted as tags, federated Remote Tables routed to their native storage platform (so they merge with native warehouse connectors), and stateful stale-entity removal.

Concept Mapping​

SAP Datasphere conceptDataHub entityNotes
SpaceContainer2-tier Space → object model (no folder layer)
ViewDataset (subtype View)Emits viewProperties; SQL-editor views use viewLanguage="SQL", else CSN
Analytic ModelDataset (subtype Analytic Model)Assets with supportsAnalyticalQueries: true; star-schema lineage
Local Table (base table)Dataset (subtype Local Table)Discovered when include_local_tables: true; schema from per-table CSN
Remote Table (federated)Dataset on its storage platformRouted via connection_to_platform_map / platform_type_defaults
ColumnSchema fieldTypes from OData EDMX
CDS semantic annotation (Dimension, Measure)TagEmitted when emit_sap_semantics_as_tags: true

Module sap-datasphere​

Testing

Important Capabilities​

CapabilityStatusNotes
Asset Containers✅Spaces emitted as containers.
Column-level Lineage✅Column-level lineage from CSN columns[] expressions. Enable via include_lineage: true.
Descriptions✅Field descriptions from EDMX Common.Label annotations.
Detect Deleted Entities✅Soft-delete via stateful ingestion stale-entity removal.
Extract Tags✅CDS semantic annotations (Dimension/Measure/Calendar/Currency/Unit/DimensionType) emitted as DataHub tags on the relevant schema fields and datasets. Toggle via emit_sap_semantics_as_tags (default True).
Platform Instance✅Per-connection platform_instance via connection_to_platform_map.
Schema Metadata✅Columns from OData EDMX.
Table-Level Lineage✅Table-level lineage from CSN query refs and @remote.source annotations. Enable via include_lineage: true.
Test Connection✅Validates OAuth credentials and tenant URL.

Overview​

The connector emits assets on two platforms, depending on whether an asset is managed (lives in Datasphere) or federated (lives in an external system that Datasphere accesses via a Remote Table).

Managed assets → sap-datasphere platform​

Views, Analytical Models, and Local Tables — the objects you create inside Datasphere — emit on the sap-datasphere platform. Their URN shape:

urn:li:dataset:(urn:li:dataPlatform:sap-datasphere, <space>.<asset>, ENV)

(prefixed with <platform_instance>. if you set the connector's top-level platform_instance).

Datasphere-specific subtypes (Local Table, View, Analytic Model), CDS-annotation tags, and Space hierarchy are all emitted on top of these URNs.

Federated Remote Tables → storage platform​

A Datasphere Remote Table that federates from e.g. Snowflake emits on the Snowflake platform — its URN matches what DataHub's native Snowflake connector emits for the same physical table. Lineage joins automatically: a downstream Datasphere View's UpstreamLineage points at the same snowflake: URN that the Snowflake connector ingests, with no Siblings configuration needed.

Federated routing is driven by connection_to_platform_map (per connection name) and platform_type_defaults (per typeId fallback). See the recipe for examples.

Two per-connection knobs matter for URN stitching with the native connector:

  • convert_urns_to_lowercase (per connection; defaults to true) — set it to false when the sibling native connector preserves source case, so the URNs match. Needed for BigQuery (project.dataset.MyTable) and for a HANA connector left at its uppercase default. This is independent of the connector's top-level convert_urns_to_lowercase, which governs managed sap-datasphere assets.
  • database (per connection) — a leading name segment prepended ahead of the schema/dataset the flow reports. Chiefly the BigQuery GCP project, which the Datasphere API never exposes: with database: my-gcp-project, a replication-flow target in dataset staging becomes my-gcp-project.staging.<table>. Set it on the per-connection entry (keyed by connection name), not on platform_type_defaults, since different connections of the same type can point at different projects.

typeId matching in platform_type_defaults is case-insensitive (BIGQUERY, BigQuery, and bigquery all match the same entry).

Built-in typeId routing​

The following SAP Datasphere connection typeIds ship with built-in platform defaults (verified against a live tenant). You can override any of them in your recipe under platform_type_defaults.

Datasphere typeIdDataHub platformCommon usage
HANAhanaHANA on-prem / external HANA Cloud federated as a Remote Table source
MSSQLmssqlSQL Server federation
S3s3S3 buckets federated as remote tables
GCSgcsGoogle Cloud Storage federation
ABAPabapSAP ABAP system extraction
SAPS4HANACLOUDs4hanaSAP S/4HANA Cloud federation
SAPBWMODELTRANSFERbwSAP BW analytical model transfer (matches the SAC connector's bw)
BIGQUERYbigqueryGoogle BigQuery replication-flow target / federated remote tables

Other typeIds (Snowflake, Kafka, Salesforce, ...) default to enabled: false with a warning — opt in by adding them to platform_type_defaults in your recipe. The connector reports each unmapped-typeId asset once via report.assets_skipped_unknown_typeid.

Local Tables (base tables)​

By default the connector emits only Datasphere assets exposed for OData consumption (views and analytical models). Base tables — which lineage edges typically point at — are not in that surface. Set include_local_tables: true to ALSO discover them via the supported /dwaas-core/api/v1/spaces/X/localtables endpoint (the same endpoint the official datasphere CLI uses; SAP-blessed, no policy caveat).

Local Tables emit on the sap-datasphere platform (same as Views and Analytical Models) so phantom-lineage edges from consuming views resolve to real entities. Their column schema is read from the per-table CSN (/dwaas-core/api/v1/spaces/X/localtables/Y) when available, enabling column-level lineage edges between a View and its base table; if the CSN is unavailable the table is still emitted as a schema-less stub. Each Local Table has subtype Local Table and parents directly to its Space container — the connector uses a 2-tier Space → object model, with no separate folder layer.

Prerequisites​

Connection​

Set base_url to your SAP Datasphere tenant URL (e.g. https://yourtenant.eu10.hcs.cloud.sap). The previous field name tenant_url remains accepted as a deprecated alias; new recipes should use base_url.

Authentication​

The connector supports three authentication methods (in priority order):

  1. Raw bearer token (token) — For local development. Obtain from your browser's DevTools after logging into Datasphere.
  2. OAuth refresh token (refresh_token) — Authorization code flow. Compatible with credentials created for Atlan. Requires client_id too.
  3. XSUAA client credentials (client_id + client_secret) — Recommended for production. Requires a Technical User OAuth Client created under System → Administration → App Integration by a DW Administrator.

Creating an OAuth Client (Client Credentials)​

  1. Log into your SAP Datasphere tenant as DW Administrator.
  2. Navigate to System → Administration → App Integration.
  3. Click Add a New OAuth Client → Purpose: API Access.
  4. Note the generated client_id, client_secret, and the OAuth Token URL (this is your xsuaa_url).

Space membership​

The ingestion principal must be a member of every Datasphere space you want to ingest.

Both the consumption catalog and the dwaas-core APIs only return spaces that the ingestion OAuth principal (the user behind a refresh token, or the technical user behind a client-credentials OAuth client) is a member of. A space the principal is not a member of returns HTTP 403 and is silently skipped — the connector logs a report warning "Not a member of SAP Datasphere space" and moves on.

Add the principal as a member of each target space:

  1. Open Space Management.
  2. Select the space.
  3. Under Members, add the ingestion user / OAuth client.

Install the Plugin​

pip install 'acryl-datahub[sap-datasphere]'

Starter Recipe​

Check out the following recipe to get started with ingestion! See below for full configuration options.

For general pointers on writing and running a recipe, see our main recipe guide.

# Example recipe for SAP Datasphere ingestion into DataHub.
#
# Managed Datasphere assets (Views, Analytical Models, Local Tables) emit on the
# `sap-datasphere` platform. Federated Remote Tables emit on the storage platform
# they federate from (snowflake / s3 / bigquery / hana-on-prem / ...) so they
# merge automatically with native warehouse-connector URNs for downstream
# lineage join purposes.
#
# THREE AUTHENTICATION OPTIONS are supported; pick exactly one:
#   1) Raw bearer token (simplest, but expires; good for ad-hoc runs)
#   2) Refresh-token grant: refresh_token + client_id + xsuaa_url
#   3) Client-credentials grant: client_id + client_secret + xsuaa_url
#
# PREREQUISITE — SPACE MEMBERSHIP: the ingestion principal (the user / OAuth
# client behind the credentials above) must be a MEMBER of every Datasphere
# space you want to ingest (Space Management -> space -> Members). Spaces the
# principal is not a member of return HTTP 403 and are silently skipped (logged
# as a `report` warning). This is the #1 reason DataHub captures fewer objects
# than a human sees in the Data Builder.
#
source:
  type: sap-datasphere
  config:
    # The SAP Datasphere tenant URL. The legacy field name `tenant_url` is still
    # accepted (deprecated alias) for backwards compatibility.
    base_url: "https://yourtenant.eu10.hcs.cloud.sap"

    # Option 1: raw bearer token from the UI.
    token: "${SAP_DATASPHERE_TOKEN}"

    # Option 2: refresh-token flow.
    # refresh_token: "${SAP_DATASPHERE_REFRESH_TOKEN}"
    # client_id: "${SAP_DATASPHERE_CLIENT_ID}"
    # xsuaa_url: "https://yourtenant.authentication.eu10.hana.ondemand.com"

    # Option 3: client-credentials flow.
    # client_id: "${SAP_DATASPHERE_CLIENT_ID}"
    # client_secret: "${SAP_DATASPHERE_CLIENT_SECRET}"
    # xsuaa_url: "https://yourtenant.authentication.eu10.hana.ondemand.com"

    env: "PROD"

    # ─────────────────────────────────────────────────────────────────────────
    # Platform mapping — federated Remote Tables only.
    #
    # Managed Datasphere assets (Views, Analytical Models, Local Tables) always
    # emit on the `sap-datasphere` platform and do NOT use these maps.
    #
    # `connection_to_platform_map` is keyed by the Datasphere CONNECTION NAME
    # (visible in the Datasphere UI's Connections page) for federated Remote
    # Tables.
    #
    # `platform_type_defaults` is a fallback keyed by Datasphere connection
    # TYPE-ID (e.g., HANA, SNOWFLAKE, S3) — applied when a federated asset's
    # connection isn't explicitly listed in `connection_to_platform_map`. The
    # connector ships with built-in defaults for HANA, MSSQL, S3, GCS, ABAP,
    # SAPS4HANACLOUD, and SAPBWMODELTRANSFER; others default to
    # `enabled: false` until you opt them in.
    # ─────────────────────────────────────────────────────────────────────────
    # Managed assets (Views, Analytical Models, Local Tables in Datasphere's own
    # HANA Cloud) always emit on the `sap-datasphere` platform — no entry needed.
    #
    # Populate connection_to_platform_map ONLY for federated Remote Tables so
    # their URNs match what your native warehouse connectors emit.
    #
    # To skip managed-asset ingestion entirely, set `_managed: {enabled: false}`
    # — that's the only field honored on the synthetic `_managed` key.
    connection_to_platform_map:
      # Example: federated Snowflake source
      # SNOWFLAKE_PROD:
      #   platform: snowflake
      #   platform_instance: "acct_xyz_warehouse_prod"
      #   env: "PROD"
      #   enabled: true

      # Example: federated S3 — disable to skip these assets
      # CLOUD_STORAGE:
      #   platform: s3
      #   enabled: false

      # Example: federated BigQuery. The GCP project never appears in the
      # Datasphere connection/flow payloads, so supply it via `database`, and
      # keep source case so the URN stitches to the BigQuery connector
      # (project.dataset.MyTable). `convert_urns_to_lowercase` here is
      # per-connection and independent of the top-level flag below.
      # BIGQUERY_PROD:
      #   platform: bigquery
      #   database: my-gcp-project
      #   convert_urns_to_lowercase: false
      #   enabled: true

    # Override platform defaults for typeIds not in the built-in table:
    # platform_type_defaults:
    #   SNOWFLAKE:
    #     platform: snowflake
    #   BIGQUERY:
    #     platform: bigquery

    # ─────────────────────────────────────────────────────────────────────────
    # Lineage extraction (table-level + column-level).
    #
    # Uses the SAP-supported per-object-type endpoint under
    # `/dwaas-core/api/v1/spaces/X/{views,analyticmodels}/Y` (same surface the
    # official `datasphere` CLI uses). All other connector functionality
    # (catalog discovery, EDMX schema, connection routing) is also on
    # SAP-supported APIs. Enabling include_lineage:
    #   - Gives you full table-level + column-level lineage from each
    #     view's SELECT FROM and SELECT columns clauses
    #   - Adds one extra HTTP call per asset (assets where the CSN fetch
    #     fails are surfaced via `report.assets_csn_fetch_failed` and
    #     emitted without lineage rather than aborting ingestion)
    #
    # The legacy field name `include_table_lineage` is still accepted (deprecated
    # alias) for backwards compatibility with older recipes.
    # ─────────────────────────────────────────────────────────────────────────
    include_lineage: false

    # Filters
    space_pattern:
      allow:
        - "^PROD_.*"
      deny:
        - "^TEMP_.*"

    asset_pattern:
      deny:
        - "^STAGING_.*"

    expose_for_consumption_only: false

    # HTTP behavior
    request_timeout_sec: 30
    max_retries: 3

    # Stateful ingestion — enables soft-deletion of removed assets/spaces.
    stateful_ingestion:
      enabled: true

sink:
  type: datahub-rest
  config:
    server: "${DATAHUB_GMS_URL}"
    token: "${DATAHUB_GMS_TOKEN}"

# ─────────────────────────────────────────────────────────────────────────────
# SAP semantic annotations surfaced by this connector
# ─────────────────────────────────────────────────────────────────────────────
# These are emitted as customProperties / description suffixes on the
# dataset URN the connector emits for each asset (sap-datasphere for managed
# assets; the federated storage platform for Remote Tables).
#
# ENTITY-LEVEL (on dataset customProperties):
#   sap_is_dimension   = "true"   if EDMX has Analytics.Dimension
#   sap_is_measure     = "true"   if Analytics.Measure
#   sap_dimension_type = "Time"   from Analytics.dimensionType (enum prefix stripped)
#   sap_datasphere_space = "<space-id>"
#   sap_datasphere_asset = "<asset-name>"
#
# FIELD-LEVEL (in field description brackets):
#   sap_calendar_type = "date" | "year" | "quarter" | "month" | "week" | "yearmonth"
#   sap_semantic      = "currency" | "unit"

Config Details​

Note that a . is used to denote nested fields in the YAML recipe.

FieldDescription
base_url ✅
string
SAP Datasphere tenant URL, e.g. https://foo.eu10.hcs.cloud.sap
asset_batch_size
integer
How many assets are submitted to the parallel executor at a time. The connector processes each space's assets in batches of this size so memory stays bounded regardless of how many assets a single space holds (the executor otherwise submits every task up front). Larger = slightly better parallelism overlap at higher memory; the default keeps peak memory low while preserving effectively full parallelism (batch size >> max_workers_assets). Only relevant when max_workers_assets > 1.
Default: 5000
client_id
One of string, null
XSUAA OAuth client ID for client credentials flow.
Default: None
client_secret
One of string(password), null
XSUAA OAuth client secret for client credentials flow.
Default: None
convert_urns_to_lowercase
boolean
Whether to lowercase emitted dataset and container URNs. The original case is preserved in display names and custom properties. Defaults to True to match the convention used by Snowflake and other DataHub sources so lineage merges case-insensitively across connectors.
Default: True
emit_sap_semantics_as_tags
boolean
If True (default), emit SAP CDS semantic annotations (@Analytics.Dimension/Measure, @Common.IsCalendar, @Common.IsCurrency/IsUnit, @Analytics.DimensionType) as DataHub tags on the relevant schema fields or datasets. Tags are searchable/filterable in DataHub Search and render as colored badges in the UI. The existing `sap_` custom properties are emitted in parallel (additive) for backward compat — no breaking change. Set to False to suppress tag emission entirely.
Default: True
expose_for_consumption_only
boolean
If True, only emit datasets that have an assetRelationalMetadataUrl set (i.e., assets exposed for consumption via the SAP Datasphere consumption API). Assets without an exposure URL are counted in assets_filtered. Useful when you want DataHub to only show consumable views.
Default: False
include_data_flows
boolean
If True, discover SAP Datasphere Data Flows and emit each as its own DataFlow + DataJob (parented under the space container) with its source/target tables as input/output datasets and column-level lineage from the producer's attributeMappings. Data Flows are the primary lineage source for Local Tables populated by an ETL flow. Uses the supported /dwaas-core/api/v1/spaces/X/dataflows endpoint (one list call per space + one read call per flow).
Default: False
include_lineage
boolean
Extract lineage from each asset's CSN. When enabled, the connector emits table-level upstream references (from SELECT FROM clauses and @remote.source annotations) AND column-level FineGrainedLineage (walking SELECT columns[] expressions). Uses the SAP-supported per-object-type endpoint under /dwaas-core/api/v1/spaces/X/{views,analyticmodels}/Y (same surface the official datasphere CLI uses); assets where CSN fetch fails are logged to report.assets_csn_fetch_failed and emitted without lineage. Cost: one extra HTTP call per asset.
Default: False
include_local_tables
boolean
If True, ALSO discover Datasphere Local Tables (base tables not exposed for OData consumption) via the supported /dwaas-core/api/v1/spaces/X/localtables endpoint. These tables typically appear only as upstream lineage targets of views. Emitting them as Datasets closes the phantom-lineage gap. Their column schema is read from the per-table CSN when available (enabling column-level lineage to the base table); otherwise they are emitted as schema-less stubs carrying platform + subtype + container membership. Off by default to keep ingestion minimal; turn on when you want full catalog coverage.
Default: False
include_remote_tables
boolean
If True, discover federated Remote Tables and emit each as a Dataset with upstream lineage to its external source object (parsed from the CSN @DataWarehouse.remote.connection / @DataWarehouse.remote.entity annotations). The external upstream is routed via connection_to_platform_map / platform_type_defaults; unmapped connections leave the remote table without upstream lineage (reported under remote_table_source_unresolved).
Default: False
include_replication_flows
boolean
If True, discover SAP Datasphere Replication Flows and emit each as a DataJob with its source/target objects as input/output datasets and per-task column-level lineage. Replication Flows move data between two external systems; their source/target objects are routed to DataHub platforms via connection_to_platform_map / platform_type_defaults (objects on unmapped connections are skipped and reported under flow_endpoints_unresolved).
Default: False
include_task_chains
boolean
EXPERIMENTAL. If True, discover Task Chains and emit each as a DataJob. No live Task Chain payload was available to reverse-engineer the member/reference grammar, so chains are currently surfaced as IO-less jobs (their presence + subtype only) without lineage edges. Enable to catalogue the objects; please report payload samples so lineage can be added.
Default: False
include_transformation_flows
boolean
EXPERIMENTAL. If True, discover Transformation Flows and emit them as DataJobs. This path is parsed with the Data Flow process-graph reader but has NOT been verified against a live Transformation Flow payload, so lineage may be incomplete. Enable only if you can validate the output; please report payload samples so the parser can be hardened.
Default: False
include_view_definitions
boolean
Whether to emit each View's / Analytic Model's definition as the DataHub viewProperties aspect (shown in the View Definition tab). SQL-editor views emit their raw SQL with viewLanguage 'SQL'; graphical/modeled views emit the Core Schema Notation (CSN/CQN) query tree with viewLanguage 'CSN'. Requires fetching the CSN, which is also fetched when include_lineage is enabled.
Default: True
incremental_lineage
boolean
When enabled, emits lineage as incremental to existing lineage already in DataHub. When disabled, re-states lineage on each run.
Default: False
max_retries
integer
Maximum number of retries for transient HTTP errors (5xx, connection errors) when calling SAP Datasphere APIs. Uses exponential backoff.
Default: 3
max_workers_assets
integer
Maximum number of parallel workers for per-asset metadata fetch (EDMX schema + CSN lineage). At 1M assets this is the dominant performance lever. Default 10 gives ~10x speedup vs serial. Set to 1 to disable parallelism.
Default: 10
platform_instance
One of string, null
The instance of the platform that all assets produced by this recipe belong to. This should be unique within the platform. See https://docs.datahub.com/docs/platform-instances/ for more details.
Default: None
refresh_token
One of string(password), null
OAuth refresh token (authorization code flow). Takes priority over client credentials.
Default: None
request_timeout_sec
integer
Per-request timeout (seconds) for HTTP calls to the SAP Datasphere catalog and consumption APIs. Increase if you see frequent timeouts on large EDMX documents.
Default: 30
resolve_external_urns_via_graph
boolean
If True, reconcile external flow endpoint URNs (replication/data-flow sources and targets routed to another platform such as BigQuery) against the real physical URNs already in DataHub, using the graph on the configured sink. SAP's flow API reports logical names (lower-cased leaf, and sometimes without a source-added prefix like w01_cds_) that need not match the physical warehouse table, so the raw edge can dangle. When enabled, each endpoint is matched by its case-insensitive leaf within the same platform/platform_instance/env and dataset/schema; an unambiguous hit is rewritten to the real URN so the lineage stitches, and misses fall back to the original candidate. Requires a DataHub graph (REST sink or datahub_api configured); no-op otherwise. Cost: one scoped search per distinct external dataset/schema (cached per run).
Default: False
token
One of string(password), null
Raw bearer token for dev/testing. Takes priority over all OAuth flows.
Default: None
xsuaa_url
One of string, null
XSUAA token endpoint base URL. Auto-derived from base_url if not set.
Default: None
env
string
The environment that all assets produced by this connector belong to
Default: PROD
asset_pattern
AllowDenyPattern
A class to store allow deny regexes
asset_pattern.ignoreCase
One of boolean, null
Whether to ignore case sensitivity during pattern matching.
Default: True
column_pattern
AllowDenyPattern
A class to store allow deny regexes
column_pattern.ignoreCase
One of boolean, null
Whether to ignore case sensitivity during pattern matching.
Default: True
connection_to_platform_map
map(str,ConnectionPlatformConfig)
How to project a SAP Datasphere connection (or the tenant's own managed storage)
onto a DataHub platform; one is resolved per asset to build its DatasetUrn.
connection_to_platform_map.key.platform ❓
string
The DataHub platform name (e.g. 'hana', 'snowflake', 's3'). Required.
connection_to_platform_map.key.convert_urns_to_lowercase
boolean
Whether to lowercase external URNs (federated Remote Tables and replication-flow endpoints) routed to this platform. Defaults to True to match the case-insensitive lineage convention Snowflake and most DataHub sources follow. Set to False to preserve source case when the sibling native connector does, so the URNs stitch — e.g. BigQuery (project.dataset.MyTable) or a HANA connector left at its default (uppercase catalog identifiers). Independent of the connector's top-level convert_urns_to_lowercase, which governs managed sap-datasphere assets.
Default: True
connection_to_platform_map.key.database
One of string, null
Optional leading name segment prepended to external URNs routed here, ahead of the schema/dataset the Datasphere flow reports. Use it to supply a qualifier the Datasphere API omits — chiefly the BigQuery GCP project, which never appears in the connections/flow payloads. With database: my-gcp-project, a replication-flow object in dataset staging becomes my-gcp-project.staging.<table> so it stitches to the BigQuery connector. Because different connections of the same type can point at different projects, set this on a per-connection entry in connection_to_platform_map (keyed by the Datasphere connection name) rather than on a platform_type_defaults entry.
Default: None
connection_to_platform_map.key.enabled
boolean
If False, assets routed to this connection are skipped (counted under assets_skipped_disabled in the ingestion report).
Default: True
connection_to_platform_map.key.platform_instance
One of string, null
Optional DataHub platform_instance. Use this to align with the platform_instance a separate connector (e.g. the DataHub Snowflake connector) emits for the same physical system.
Default: None
connection_to_platform_map.key.env
One of string, null
Optional environment override (PROD/DEV/STAGING/...). Falls back to the connector's top-level env when None. Must be a valid DataHub FabricType.
Default: None
platform_type_defaults
map(str,ConnectionPlatformConfig)
How to project a SAP Datasphere connection (or the tenant's own managed storage)
onto a DataHub platform; one is resolved per asset to build its DatasetUrn.
platform_type_defaults.key.platform ❓
string
The DataHub platform name (e.g. 'hana', 'snowflake', 's3'). Required.
platform_type_defaults.key.convert_urns_to_lowercase
boolean
Whether to lowercase external URNs (federated Remote Tables and replication-flow endpoints) routed to this platform. Defaults to True to match the case-insensitive lineage convention Snowflake and most DataHub sources follow. Set to False to preserve source case when the sibling native connector does, so the URNs stitch — e.g. BigQuery (project.dataset.MyTable) or a HANA connector left at its default (uppercase catalog identifiers). Independent of the connector's top-level convert_urns_to_lowercase, which governs managed sap-datasphere assets.
Default: True
platform_type_defaults.key.database
One of string, null
Optional leading name segment prepended to external URNs routed here, ahead of the schema/dataset the Datasphere flow reports. Use it to supply a qualifier the Datasphere API omits — chiefly the BigQuery GCP project, which never appears in the connections/flow payloads. With database: my-gcp-project, a replication-flow object in dataset staging becomes my-gcp-project.staging.<table> so it stitches to the BigQuery connector. Because different connections of the same type can point at different projects, set this on a per-connection entry in connection_to_platform_map (keyed by the Datasphere connection name) rather than on a platform_type_defaults entry.
Default: None
platform_type_defaults.key.enabled
boolean
If False, assets routed to this connection are skipped (counted under assets_skipped_disabled in the ingestion report).
Default: True
platform_type_defaults.key.platform_instance
One of string, null
Optional DataHub platform_instance. Use this to align with the platform_instance a separate connector (e.g. the DataHub Snowflake connector) emits for the same physical system.
Default: None
platform_type_defaults.key.env
One of string, null
Optional environment override (PROD/DEV/STAGING/...). Falls back to the connector's top-level env when None. Must be a valid DataHub FabricType.
Default: None
space_pattern
AllowDenyPattern
A class to store allow deny regexes
space_pattern.ignoreCase
One of boolean, null
Whether to ignore case sensitivity during pattern matching.
Default: True
stateful_ingestion
One of StatefulStaleMetadataRemovalConfig, null
Stateful Ingestion Config with stale metadata removal (soft-deletes spaces/assets that disappear between runs).
Default: None
stateful_ingestion.enabled
boolean
Whether or not to enable stateful ingest. Default: True if a pipeline_name is set and either a datahub-rest sink or datahub_api is specified, otherwise False
Default: False
stateful_ingestion.fail_safe_threshold
number
Prevents large amount of soft deletes & the state from committing from accidental changes to the source configuration if the relative change percent in entities compared to the previous state is above the 'fail_safe_threshold'.
Default: 75.0
stateful_ingestion.remove_stale_metadata
boolean
Soft-deletes the entities present in the last successful run but missing in the current run with stateful_ingestion enabled.
Default: True

Capabilities​

Lineage extraction​

When include_lineage: true is set on the recipe, the connector emits both table-level and column-level lineage by walking the CSN (Core Schema Notation) returned by the supported per-object-type endpoint under /dwaas-core/api/v1/spaces/{space}/{views,analyticmodels}/{name} — the same surface the official datasphere CLI uses.

Table-level lineage is derived from the asset's query.SELECT.from clause (direct refs, joins, and inline subqueries) plus @remote.source annotations on federated remote tables. For Analytic Models, the star-schema fact and dimension sources come from the business layer, which is authoritative over the query's FROM. UNION / INTERSECT / EXCEPT (query.SET) branches are flattened so every branch's FROM contributes an upstream. CDS associations / compositions (cds.Association / cds.Composition elements) are treated as upstreams only when actually referenced — projected in SELECT / columns or used in a join/on — so unused navigation links don't create phantom edges.

Column-level lineage walks query.SELECT.columns[] and emits one FineGrainedLineage entry per downstream column with at least one resolvable upstream column. Supported expression shapes:

CSN shapeExampleResult
Direct ref{ref: [AMOUNT]}IDENTITY transformation
Aliased ref{ref: [BASE, X], as: y}RENAME
Aggregate function{func: SUM, args: [{ref: [X]}], as: total}AGGREGATE
Arithmetic / xpr{xpr: [{ref: [A]}, "+", {ref: [B]}], as: sum}EXPRESSION
Function{func: UPPER, args: [{ref: [X]}], as: u}TRANSFORMATION
Association nav{ref: [_Customer, NAME], as: cust}Attributed to the association's target entity
UNION / SET{SET: {op: union, args: [SELECT, SELECT]}}Output column merges upstreams from every branch
Literal{val: 5, as: const}No lineage emitted (no upstream column)

Unresolvable refs (e.g., SELECT * star expansion, or columns referring to unknown aliases) are skipped and logged to report.column_lineage_unresolved.

Lineage extraction adds one HTTP call per asset (the per-object-type CSN fetch). At 1M assets with max_workers_assets=10, this adds ~3 hours to the total run time. The column-level walker itself is essentially free (in-memory tree walk over an already-fetched JSON document).

Flow and Remote Table lineage (ETL)​

The query-derived lineage above captures how views are computed, but not how the tables they read are populated. Several opt-in flags surface the design-time objects that move and transform data, so an otherwise phantom upstream Local Table becomes a navigable node with real upstream lineage:

  • include_data_flows: true — emits each Data Flow as its own DataFlow (pipeline) with a single DataJob, parented under the space container, with its source tables as inputs, its target tables as outputs, and column-level lineage from the flow's column mappings. Column lineage is attributed only when a target has a single input source.
  • include_replication_flows: true — emits each Replication Flow as its own DataFlow + DataJob wiring its source object(s) to target object(s), with per-task column mappings. Source/target systems are routed to a DataHub platform via connection_to_platform_map / platform_type_defaults (using the flow's own connectionType when the connection isn't in the space's connections list).
  • include_remote_tables: true — emits federated Remote Tables as datasets with upstream lineage to their external source object (parsed from the CSN @DataWarehouse.remote.* annotations and routed via the connection maps).
  • include_task_chains: true — experimental. Task Chains are discovered against a live tenant and emitted as IO-less DataFlow + DataJob pairs (their presence + subtype), but their internal member/reference grammar has not been reverse-engineered, so no lineage edges are produced. Enable to catalogue the objects; please share a payload sample if you need lineage.
  • include_transformation_flows: true — experimental. Parsed with the Data Flow process-graph reader, but not yet verified against a live Transformation Flow payload, so lineage may be incomplete. Enable only if you can validate the output.

Endpoints or sources that don't resolve to a platform are counted in report.flow_endpoints_unresolved and report.remote_table_source_unresolved rather than being silently dropped. Each flag adds one list call per space plus one read call per object.

View definitions​

With include_view_definitions: true (default), a View's or Analytic Model's definition is emitted as the viewProperties aspect — viewLanguage="SQL" for SQL-editor views (the raw SQL the modeler wrote) and viewLanguage="CSN" for graphical/modeled views (the CSN/CQN query tree).

Analytic Models​

Assets with supportsAnalyticalQueries: true are cataloged as the Analytic Model subtype. Column schema for exposed assets is read from the OData EDMX relational-metadata URL.

Tags (SAP CDS semantic annotations)​

With emit_sap_semantics_as_tags: true (default), the connector emits DataHub tags for the SAP CDS semantic annotations it parses from EDMX, so they're searchable and filterable in DataHub Search and render as colored badges in the UI:

Tag URNSource annotationWhere attached
urn:li:tag:Dimension@Analytics.Dimensiondataset (entity-level) or column
urn:li:tag:Measure@Analytics.Measuredataset (entity-level) or column
urn:li:tag:sap:semantic:currency@Common.IsCurrencycolumn
urn:li:tag:sap:semantic:unit@Common.IsUnitcolumn
urn:li:tag:sap:calendar:{year,month,quarter,week,date,yearmonth}@Common.IsCalendar*column
urn:li:tag:sap:dimension_type:{Time,Customer,...}@Analytics.DimensionTypedataset

Dimension and Measure URNs are flat (no namespace) so they collide with the equivalent tags from other DataHub connectors (Looker, Snowflake semantic views, Mode, ThoughtSpot, ...). Customers get cross-connector pivot on a single "Measure" tag. SAP-specific concepts are namespaced under sap: to avoid collision.

The existing sap_* custom properties (sap_is_dimension, sap_calendar_type, sap_semantic, sap_dimension_type, ...) continue to be emitted on every asset for backward compat — tag emission is purely additive. The same annotations also appear in field descriptions with [sap_calendar_type=date]-style suffixes. Set emit_sap_semantics_as_tags: false to suppress tag emission entirely.

API support​

All endpoints called by this connector are SAP-supported public APIs:

  • /api/v1/datasphere/consumption/catalog/... — asset discovery (Catalog API)
  • /api/v1/datasphere/consumption/relational/.../$metadata — EDMX schema
  • /api/v1/datasphere/spaces/X/connections — Connections API
  • /dwaas-core/api/v1/spaces/X/{views,analyticmodels,localtables,remotetables}/{name} — per-object-type CSN read (with Accept: application/vnd.sap.datasphere.object.content+json), the same surface the official datasphere CLI uses
  • /dwaas-core/api/v1/spaces/X/{dataflows,replicationflows,transformationflows,taskchains}/{name} — per-flow design-time definitions (only called when the matching include_* flag is set)

Previous releases of this connector used the /deepsea/repository/ endpoint for lineage extraction, which SAP marks as internal-use-only in KBA #3517441. That dependency was removed in this release — include_lineage: true is now a fully supported feature.

The catalog service caps a page at 500 records (both the default and the documented maximum). The connector paginates by following the server's @odata.nextLink cursor (falling back to advancing its offset by the number of records actually returned), so spaces with more than 500 exposed assets are fully captured.

Performance and scale​

The dominant cost is HTTP calls, not memory. For each exposed asset the connector makes roughly 2 HTTP calls — one CSN definition fetch (per-object type endpoint) plus one EDMX $metadata schema fetch — on top of paginated catalog listing. That works out to ~2N HTTP calls for N assets (e.g. ~2,000,000 calls for 1M datasets), so expect long runtimes at very large scale.

Levers to reduce cost and scope:

  • space_pattern / asset_pattern — filtering is applied before the per-asset HTTP calls, so filtered-out assets are nearly free. Scope the run to the spaces / assets you actually need.
  • expose_for_consumption_only: true — skip assets that have no consumption exposure URL. By default the connector catalogs all assets in each Space regardless of whether they are exposed for OData consumption.
  • include_view_definitions: false AND include_lineage: false — together these skip the per-asset CSN fetch, roughly halving the HTTP calls for users who only need catalog + schema (no lineage / view definitions).
  • max_workers_assets — default 10; raise toward 64 if the tenant tolerates the request rate. EDMX schema and CSN lineage are fetched per-asset (the SAP Datasphere API has no batch metadata endpoint), so this is the main parallelism lever for large catalogs.

Memory is bounded regardless of catalog size:

  • Workunits stream out with backpressure (the connector never buffers the whole run).
  • The ingestion report uses capped LossyLists, so per-run diagnostics don't grow unbounded.
  • Each space's assets are processed in batches of asset_batch_size (default 5000) before being submitted to the parallel executor, so peak memory does not grow with the number of assets in a single space.

Limitations​

Catalog business-enrichment metadata is not extracted​

SAP Datasphere's Catalog & Marketplace application surfaces governance metadata that this connector does not ingest:

  • Responsible Team, Business Contact Person, Purpose
  • User-defined Tags and related Glossary Terms / KPIs
  • Published By / created-by / published-on (user-level ownership & audit)

These fields are not exposed by any public, OAuth-accessible SAP API. They are served only by the Catalog's internal UI backend under /deepsea/catalog/ and /deepsea/repository/, which:

  1. Rejects OAuth. Called with an XSUAA Bearer token (the connector's auth), /deepsea/... returns an HTML login page, not JSON — it only accepts interactive browser session cookies.
  2. Is an internal UI endpoint, not a supported REST API (its OData service names are explicitly marked ...private...), so it may change without notice.

The supported public surface the connector uses (/api/v1/datasphere/consumption/catalog/...) returns an asset's technical name, business name (label), description, space, consumption URLs, and capability flags — but none of the governance-enrichment fields above. There is therefore no supported path to ingest them today. Support will be added if and when SAP publishes a public Catalog metadata API.

Note: the connector does extract the SAP CDS semantic annotations that are available via the supported EDMX/CSN surface (Dimension/Measure, currency/unit, calendar, dimension type) — see Capabilities → Tags. The gap is specifically the Catalog-app governance fields, not all business metadata.

Schema and type limitations​

These follow from what the supported consumption surface exposes:

  • Geospatial and binary types are reported as strings. Per SAP's "OData Annotation Limitations", the consumption $metadata (EDMX) overwrites several CDS types to Edm.String: cds.hana.ST_GEOMETRY, cds.hana.ST_POINT, cds.Binary, cds.LargeBinary, cds.hana.BINARY, cds.LargeString, cds.UUID. Because the connector reads column types from this EDMX surface, such columns appear as string types in DataHub — the original geo/binary/UUID type cannot be recovered from the consumption API.
  • View input parameters are not enumerated. The connector records whether an asset has parameters (the sap_* flag derived from the catalog's hasParameters) and surfaces Analytic Model variables (as the sap_variables custom property), but it does not yet emit the individual input-parameter definitions of a view as schema/metadata.

Lineage limitations​

  • Scalar subqueries in column expressions ARE supported (resolved against the subquery's own FROM clause). Correlated subqueries that reference outer-scope aliases are NOT — their refs land in report.column_lineage_unresolved.
  • Cross-Space upstream resolution: a reference that is already Space-qualified (e.g. OTHER_SPACE.OBJECT) is detected and used as-is, so it is no longer double-prefixed into a phantom URN. It is still routed to the downstream's storage platform (managed cross-Space refs stay on sap-datasphere), and column matching assumes the upstream schema mirrors the downstream.
  • SELECT * star expansion is not unrolled — those columns are listed in report.column_lineage_unresolved.
  • A SELECT FROM T AS t with unqualified column refs (e.g. {ref: [X]}) does NOT auto-resolve through the alias; users either need to qualify the ref (t.X) or remove the explicit AS alias.

Stateful ingestion with large catalogs​

The connector uses DataHub's standard StaleEntityRemovalHandler for soft-delete detection. This handler keeps a checkpoint of every emitted URN in a single MCP that is sent to GMS at the end of each run.

At default GMS / Kafka payload limits (1 MB Kafka message, 16 MB GMS REST), the checkpoint can hold approximately 80,000 dataset URNs before exceeding limits. If your Datasphere catalog has more than ~50,000 datasets we recommend:

  • Disable stateful_ingestion (set enabled: false) and use manual soft-delete cleanup via the DataHub UI / CLI.
  • Or, partition the ingestion by space_pattern so each run stays under the ceiling.

The connector emits a report.warning when emitted entity count crosses 50,000 during a single run (via _check_scale_warning).

Troubleshooting​

Fewer objects than the Data Builder shows​

This is the #1 reason DataHub captures fewer objects than the Data Builder shows: a human browsing the Data Builder sees spaces they personally belong to, while the ingestion principal only sees the spaces it belongs to. If a space is missing from DataHub, check the report warnings for the "Not a member of SAP Datasphere space" message and add the principal to that space (see Prerequisites → Space membership).

HTTP 429 throttling​

SAP Datasphere rate-limits the OData/consumption APIs to ~300 requests per user per minute (per the OData API documentation); exceeding it returns HTTP 429 with a Retry-After header. The connector's HTTP layer already retries 429 with exponential backoff (honoring Retry-After, bounded by max_retries), but setting max_workers_assets too high can sustain throttling and slow the run — tune it down if you see repeated 429 retries in the logs.

Local testing with a mock Datasphere (no tenant required)​

You can exercise the full connector path — including the OAuth handshake — without a real SAP tenant, using the bundled standalone mock server. It serves real recorded tenant responses (catalog spaces/assets/connections, per-view CSN, EDMX, local tables) plus a permissive /oauth/token endpoint, so the connector's OAuth cold-start, data fetch, and lineage all run end-to-end.

Start the mock server (it prints a ready-to-run recipe and the command to run):

# Push into a running DataHub:
python tests/integration/sap_datasphere/mock_datasphere_server.py --sink http://localhost:8080

# Or omit --sink for a file sink (writes to /tmp/sap_datasphere_mock_out.json):
python tests/integration/sap_datasphere/mock_datasphere_server.py

Then, in another terminal, run ingestion against the generated recipe:

datahub ingest -c /tmp/sap_datasphere_mock_recipe.yml

mock_datasphere_server.py needs pytest-httpserver (a test dependency). If it isn't importable, install it with pip install pytest-httpserver or install this package's test extra.

UI-ingestion caveat: the DataHub UI's ingestion executor runs in a container, so localhost / 127.0.0.1 in the recipe won't reach a mock server running on the host. Bind the server to all interfaces with --host 0.0.0.0 and point the recipe's base_url / xsuaa_url at host.docker.internal:18000 (Docker Desktop) or the host's LAN IP instead.

Code Coordinates​

  • Class Name: datahub.ingestion.source.sap_datasphere.source.SapDatasphereSource
  • Browse on GitHub
Questions?

If you've got any questions on configuring ingestion for SAP Datasphere, feel free to ping us on our Slack.

💡 Contributing to this documentation

This page is auto-generated from the underlying source code. To make changes, please edit the relevant source files in the metadata-ingestion directory.

Tip: For quick typo fixes or documentation updates, you can click the ✏️ Edit icon directly in the GitHub UI to open a Pull Request. For larger changes and PR naming conventions, please refer to our Contributing Guide.