Skip to main content

v2.1.0

Release Changelog​

v2.1.0​

Release Availability Date​

30-Jul-2026

  • CLI/SDK: 1.6.0.16
  • Remote Executor: v2.1.0-cloud, v2.0.4-cloud, v2.0.3-cloud, v2.0.2-cloud, v2.0.1-cloud, v1.1.4-cloud
  • On-Prem Versions:
    • Helm: 1.6.233
    • API Gateway: v0.7.3

Breaking Changes​

  • Orchestration plugins default to async emit. The Airflow, Dagster, Prefect, and Great Expectations plugins now default to ASYNC. Async emit() no longer raises on a rejected write and gives no read-after-write guarantee. To restore the previous behavior, set emit_mode: SYNC_PRIMARY (per-plugin config; [datahub] emit_mode = SYNC_PRIMARY in airflow.cfg). The Python SDK, CLI, and DataHub Actions defaults are unchanged.
  • Built-in column-level classifier removed. The DataHubClassifier and its acryl-datahub-classify dependency have been removed from acryl-datahub. The classification framework is retained but no classifier is registered by default. Recipes that set classification.enabled: true without registering a replacement now fail fast at startup. Remove the classification block, register a custom classifier, or pin the last release that ships the built-in classifier.
  • Spark lineage bundles OpenLineage 1.50. OpenLineage's built-in dataset-name trimmers strip common partition directories from filesystem and object-store dataset names by default. The plugin disables them automatically when DataHub path trimming (path_spec_list or file_partition_regexp) is configured. Recipes with neither configured pick up the auto-stripping, so dataset URNs for partitioned paths change and edges to the old URNs can orphan. Configure path trimming, or set spark.datahub.metadata.dataset.openLineageTrimmersEnabled=false to restore the pre-1.50 naming. Separately, file_partition_regexp now also applies to bare-namespace file/dbfs datasets.
  • Power BI and Mode column-level lineage preserves original casing. These connectors no longer lowercase upstream column names, so previously-missing column-level edges will render after re-ingestion. If you set convert_column_urns_to_lowercase: true on an upstream SQL source as a workaround, remove it so both sides preserve casing.
  • BigQuery table statistics source changed. Row count, size, and last-altered now come from INFORMATION_SCHEMA.PARTITIONS instead of the undocumented __TABLES__. PARTITIONS covers base tables only, so empty tables, external tables, views, and snapshots lose their lastModified timestamp (and snapshots lose row/size custom properties). Set use_legacy_table_stats: true to restore the previous behavior.
  • MongoDB can be ingested as AWS DocumentDB. A new opt-in platform: documentdb (with hostingEnvironment: AWS_DOCUMENTDB) emits entities under a new documentdb data platform. The default is unchanged; enabling it changes dataset and container URNs, and stale mongodb entities must be cleaned up.
  • Snowplow enrichment DataJob URNs are now name-based. The connector moved to the supported enrichments endpoint, which no longer returns per-enrichment UUIDs, so enrichment DataJob URNs derive from the enrichment name. Old UUID-based DataJobs are re-created; clean up stale entities.
  • Teradata database-container URN casing. When convert_urns_to_lowercase: true (non-default), the connector now also lower-cases database-container URNs, not just datasets, so containers and their datasets match. Deployments on the default (false) are unaffected.
  • Ingestion framework: workunit processor API. Workunit-processor helper functions have been removed in favor of WorkunitProcessor classes, and several processors were renamed. Custom code that called the helper functions directly must migrate to the processor-class API.
  • Dataset semantic search is no longer implicit. Enabling semantic search with the default entity set no longer also bridges datasets; set ELASTICSEARCH_SEMANTIC_SEARCH_ENTITIES=document,dataset on GMS and the system-update job to keep dataset semantic search after upgrade.
  • Logical parent links require Edit Entity on both datasets. Linking a physical dataset to a logical parent now requires Edit Entity on both the child and the proposed parent. Clearing a parent requires Edit Entity on the child only. Grant the privilege or update policies accordingly.
  • GMS usage-metric semantics (opt-in). With USAGE_AGGREGATION_ENABLED=true, datahub_request_count is emitted from the in-memory aggregation flush to Micrometer/Prometheus (not per-request) and the actor tag becomes actor_class. New byte and active-identity metrics are added. Aggregation is off by default; legacy JMX metrics are unaffected.
  • Executor coordinator env flags. DATAHUB_EXECUTOR_MONITORS_ENABLED and DATAHUB_EXECUTOR_TASKS_ENABLED are now hard opt-outs that skip subsystem wiring and heavy imports, not fetcher-only toggles. Use DATAHUB_EXECUTOR_INGESTION_PIPELINE_ENABLED to disable the Kafka / actions pipeline.
  • Subscriptions inherit notification defaults dynamically. GraphQL callers that omit notificationConfig (or send it without notificationSettings) when creating a subscription now leave it configured to use the actor's current notification defaults at delivery time, instead of creating a subscription with no sinks. Send notificationSettings.sinkTypes: [] explicitly for an intentional no-sink subscription.
  • Removed REQUEST_MINIMAL_SLACK_PERMISSIONS. Its role is now served by DATAHUB_SLACK_SERVER_SIDE_HISTORY_ENABLED, which marks the Slack history scopes as optional in the install screen so admins can deselect them per install. Remove the old variable if set.
  • Lineage scroll endpoint contract. POST /openapi/v3/lineage/scroll now takes a single urns list (replacing the separate source/destination/edge filters); direction accepts UPSTREAM/DOWNSTREAM. No known callers, so no migration is expected.

New Feature Highlights​

  • Metrics and Semantic Models — new first-class Metric and SemanticModel entities, with dedicated entity pages, a metrics home page, sidebar navigation, and autocomplete, so business metrics and the semantic models that define them are cataloged, searchable, and connected to lineage alongside the datasets they derive from.

  • Scoped MCP servers — create and manage purpose-built Model Context Protocol (MCP) servers from the UI, each exposing a curated set of tools, configurable instructions and optional data asset scoping via Views, to enable you to build domain-specific agents with access to specific data & tools. Available in Context Platform private beta by default.

  • Agent, Service, and Repository catalog entities — catalog AI agents, the skills they adopt and tools they invoke, the services those tools belong to (REST APIs, MCP servers), and the source-code repositories behind them as first-class entities that participate in lineage, with auto-registration available for LangChain and Google ADK agents. See the Agent Registry (API tutorial) and Service Catalog guides.

  • Bi-Directional Context Document Sync for GitHub - import context documents from one or more GitHub repositories, and sync changes made on DataHub back as GitHub PRs. Read more here.

  • Expanded internationalization — multi-language support can be enabled upon request. New locales: French (fr), Italian (it), Norwegian Bokmål (nb), Swedish (sv), and Hungarian (hu) (all Beta). When enabled, DataHub follows the browser locale where a translation exists.

  • New ingestion sources — Cube (semantic layer), AWS Kinesis Data Streams and Amazon Data Firehose, MicroStrategy, and Open Data Contract Standard (ODCS).

  • Data Product Lineage - Data products now appear directly in lineage, making it easier to understand how business-level data entities connect across your ecosystem.

  • All changes in https://github.com/datahub-project/datahub/releases/tag/v1.6.0

Other Product Changes​

  • Ask DataHub is generally available — the Beta label is gone, and the chat agent now pauses for human-in-the-loop approval before applying any metadata edit on the web, so a human can confirm important changes before they are made.
  • Improved lineage graph controls — added a redraw control, an empty-graph nudge, and a modernized control layout for the lineage visualization.
  • Glossary Node Domains — domains can now be attached to Glossary Nodes, and the Manage Domains sidebar reaches parity with the documents experience.
  • Logical Models in the UI — create logical models, link them to physical datasets, map columns, and edit their schema entirely from the UI, in addition to the existing API and SDK paths. Enabled by default.
  • Home View All for asset collections — asset-collection modules on the home page now include a View All action that opens the full filtered result set for that module.
  • Support bundle download and configurable log shipping (self-managed) — self-managed deployments can opt into shipping service logs to a Loki-compatible aggregator via LOG_AGGREGATOR_ENDPOINT, which also enables support-bundle log collection without relying on the Kubernetes API.
  • Lineage graph UX refresh — a redraw control, an empty-graph nudge, and modernized on-canvas controls make the lineage graph easier to read and navigate.
  • Document search scoped to the active View — document search now respects the active View, so results stay within the working set the user has selected.
  • Redesigned assertion notification emails. Assertion notification emails have a refreshed layout and more useful details about the assertion and the failure that's easier to scan. The email body also now includes an alert id that can be used for deduplicating alerts in your inbox or through external system integrations.
  • Subscriptions follow their asset's lifecycle. When an asset you're subscribed to is soft-deleted, its subscriptions are deactivated automatically, so you stop receiving alerts for an asset that no longer exists; restoring the asset brings them back. A background reconciliation keeps subscriptions consistent with their target's status even if an update is missed.
  • Snowflake Column Casing Adds support for case-sensitive column identifiers in Snowflake assertion queries through an opt-in configuration flag: DATAHUB_EXECUTOR_SNOWFLAKE_QUOTE_COLUMNS. Turn this flag on if your column names are case sensitive in Snowflake.
  • Smart-assertion inference improvements on by default. Volume-assertion preprocessing (DATAHUB_USE_OBSERVE_MODELS) and delta-space prediction bounds (DATAHUB_EXECUTOR_ENABLE_DELTA_BOUNDS) are now enabled by default, improving anomaly-detection accuracy out of the box.

Ingestion​

New ingestion sources:

  • Cube — new semantic-layer connector.
  • AWS Kinesis — new source for Kinesis Data Streams and Amazon Data Firehose.
  • MicroStrategy — new BI connector.
  • Open Data Contract Standard (ODCS) — new source, registered as its own data platform.

Connector improvements:

  • Teradata — profiling now respects profile_table_size_limit (default 5 GB, fail-open when size is unknown); lineage fetch batch size is configurable; view definitions can be offloaded to a disk-backed store; multi-row query reconstruction is capped to protect memory; permission errors during size checks are categorized; and zero-row lineage fetches surface warnings.
  • Databricks Unity Catalog — Lakehouse Federation (foreign catalog) support; usage, operations, and query entities derived from system.query.history via the shared SQL parsing aggregator (window up to 365 days); zero-value usage stamped on idle windows; and fixed ML model ingestion controls (include_ml_models, an enforced ml_model_max_results cap, and MLModel URNs honoring the recipe env).
  • Redshift — per-query popularity statistics (queryUsageStatistics) on Query entities; optional column-level usage; and table_pattern now applied to SQL-parsing-derived lineage and usage. The synthetic per-user userEmail in usage stats is no longer populated (user attribution via CorpUser URN is unchanged).
  • BigQuery — table statistics now sourced from INFORMATION_SCHEMA.PARTITIONS (see Breaking Changes); usage.* window and formatting fields consolidated to top-level equivalents (see Deprecations).
  • Power BI — Hive ODBC sources now resolve to the hive data platform (instead of hadoop), so Hive lineage parses correctly and aligns with the Hive connector.
  • Spark — Apache Spark 4.x support (Scala 2.13 build); OpenLineage upgraded to 1.50 with full shadow-jar shading so the agent coexists with environments that ship their own OpenLineage (for example Amazon EMR 7.12+ / SageMaker Unified Studio); and file_partition_regexp now applies to bare-namespace file/dbfs datasets.
  • Glue — view classification and view lineage extraction.
  • Matillion — foldered container hierarchy, environment-scoped lineage, run history at pipeline and component levels, corrected external console links, and unpublished-pipeline handling.
  • ClickHouse — usage statistics migrated to the shared SQL parsing aggregator.
  • MySQL — usage and query lineage via performance_schema.
  • Confluence — folders ingested as documents to preserve hierarchy.
  • Amazon S3 — catalog object-store folders without ingesting the underlying files via emit_folders_only.
  • Kafka Connect — Iceberg Sink Connector and Snowflake Streaming Sink Connector support.
  • Dataplex — ingestion source UI forms for the V2 experience.

Ingestion infrastructure:

  • Async emit by default for orchestration plugins — Airflow, Dagster, Prefect, and Great Expectations default to async emit to reduce load at high volume (see Breaking Changes).
  • Column-level classifier removed — the built-in DataHubClassifier and its dependency are gone; the classification framework remains (see Breaking Changes).
  • Workunit processor API — helper functions replaced by WorkunitProcessor classes (see Breaking Changes).
  • mTLS client authentication — the CLI and SDK can present a client certificate on every outbound HTTPS path via DATAHUB_CLIENT_CERT_PATH (and optionally DATAHUB_CLIENT_KEY_PATH). Fully backwards compatible.
  • Environment-based OAuth — DATAHUB_AUTH_TYPE selects OAuth for the CLI, default sink, and env-resolved emitters, backed by reusable OAuth token providers for the Python clients.
  • Opt-in Kafka default sink for managed ingestion
  • Escape hatch for pure-Python SQLGlot - DATAHUB_SQLGLOT_DISABLE_C
  • Temp-table usage stats fixed — per-query popularity statistics for queries that load through temporary/staging tables (the common ELT/dbt pattern) now land on the composite Query that lineage references, across every connector on the shared SQL parsing aggregator.

Executor:

  • Ingestion-log garbage collector — opt-in in-process cleanup of per-execution log directories older than a retention window, with a size-cap safety net (DATAHUB_EXECUTOR_LOG_GC_ENABLED, default false).
  • Stored-procedure CALL statements — custom SQL assertions can optionally allow a CALL statement (DATAHUB_EXECUTOR_ALLOW_CALL_STATEMENTS, default false).

Security / Dependencies​

  • Spring Security and commons-configuration2: bumped to address Java CVEs.
  • starlette: bumped to 1.3.1 to address CVE-2026-48710, CVE-2026-48817, CVE-2026-48818, and CVE-2026-54283.
  • redshift-connector: minimum bumped to 7.45.5-equivalent floor (>=2.1.14) to address CVE-2026-8838.
  • joserfc and soupsieve: CVE floors raised for the executor and integrations service.
  • Netty: bumped to 4.2.16.Final to address CVE-2026-55831, CVE-2026-55833, CVE-2026-56745, CVE-2026-55851, CVE-2026-56816, CVE-2026-56817, CVE-2026-56820, CVE-2026-56821, CVE-2026-56822, and CVE-2026-59901.
  • Jetty: bumped to 12.1.10 to address CVE-2026-10050.
  • pyasn1: bumped to 0.6.4 to address CVE-2026-59885 and CVE-2026-59886.
  • gitpython: bumped to 3.1.52+ to address GHSA-2f96-g7mh-g2hx, GHSA-956x-8gvw-wg5v, GHSA-v396-v7q4-x2qj, and GHSA-rwj8-pgh3-r573.
  • mcp: bumped to 1.27.2 to address CVE-2026-52869 and CVE-2026-52870.
  • pillow: bumped to 12.3.0 to address 10 HIGH-severity CVEs (CVE-2026-54058, CVE-2026-54059, CVE-2026-54060, CVE-2026-55379, CVE-2026-55380, CVE-2026-59197, CVE-2026-59199, CVE-2026-59200, CVE-2026-59204, CVE-2026-59205).

Bug Fixes​

  • Fixed keyed-array patch rebase dropping the map key when a patch created a new element through a deeper path (for example adding a field-level tag or term to a schema field with no existing editable metadata), which previously failed with an HTTP 422 - fieldPath is required.
  • Raised the JSON property-name length limit on metadata deserialization so deeply-nested struct field paths (for example dbt column-level lineage carried in patches) no longer fail with a name-length exception.
  • Fixed oversized queryProperties aspects failing ingestion runs by measuring serialized size correctly; oversized statements are now truncated with a warning instead of failing the run.
  • Improved Ask DataHub and chat reliability, including prompt-cache friendliness and delegation-routing latency.
  • Improved server-side performance for glossary root loading, entity-registry schema scans, and assertion-assignment GraphQL batching.
  • Fixed Slack and Microsoft Teams settings pages that could show a connected integration as disconnected.
  • Fixed retention policy application slowing down (and risking skipped rows) at scale by replacing OFFSET-based batch deletion with keyset pagination.
  • Fixed incremental reindex (zero-downtime upgrade) silently skipping the alias-swap retry after a doc-count-mismatch failure, which could leave an index alias pinned to stale data.
  • Fixed Ask DataHub and the Slack/Teams bots rendering raw tool-call JSON as progress narration, which could also freeze progress updates mid-turn.
  • Fixed a dependency conflict that blocked Google ADK agent auto-registration.
  • Fixed the EQUAL TO operator crashing the Validations tab and assertion description for volume assertions.
  • Fixed Metrics and Semantic Model lineage module controls and entity-edge hydration.
  • Fixed Remote Executor pool APIs returning a server error (and aborting executor startup) when a pool's isEmbedded flag was unset.
  • Fixed the workflow picker modal returning no results when the selected platform's matches weren't in the first page of unfiltered results.
  • Fixed the Analytics usage/credits dashboard failing to render and corrected its commit and credit consumption categories.

Environment Variables​

  • DATAHUB_AI_TOOL_APPROVAL_ENABLED (default true): gates the AI chat's human-in-the-loop approval of mutating tools on the web UI. Set false to auto-apply edits as before. Re-read per chat turn.
  • CHAT_SUGGESTIONS_ENABLED (default true): gates generation of Ask DataHub follow-up suggestions.
  • SQL_SKETCH_SYNTHESIS_MODEL (default: a Bedrock Sonnet profile): model for SQL sketch/advice synthesis; accepts bedrock/, openai/, or google_vertexai/ prefixes.
  • I18N_ENABLED: enables multi-language translation. On DataHub Cloud, enable upon request; in OSS the default is true (set false for English-only).
  • USAGE_AGGREGATION_ENABLED (default false): enables in-memory request/byte aggregation exported to Micrometer/Prometheus on flush.
  • USAGE_AGGREGATION_INCLUDE_AGENT_NAME (default false): when usage aggregation is on, also roll up by client agent_name. Enable only when finer client attribution is required — values are client-controlled and can increase metric cardinality.
  • ENTITY_COUNT_METRICS_ENABLED (default true): enables periodic Micrometer/Prometheus gauges for key-aspect entity counts (datahub.entity.count).
  • ENTITY_COUNT_METRICS_UPDATE_INTERVAL_SECONDS (default 3600) / ENTITY_COUNT_METRICS_INITIAL_DELAY_SECONDS (default 60) / ENTITY_COUNT_METRICS_SKIP_CACHE (default false): refresh cadence and whether each refresh bypasses the key-aspect count cache.
  • EBEAN_READ_POOL_ENABLED (default false) / EBEAN_READ_POOL_URL: optional Ebean read pool for entity-aspect reads (omit URL for same-DB split-pool; set URL to a replica for offload). See Primary storage read pool.
  • CASSANDRA_READ_POOL_ENABLED (default false) / CASSANDRA_READ_POOL_HOSTS: optional Cassandra read session for entity-aspect reads.
  • ENTITY_GRAPH_CACHE_ENABLED (default true on GMS): enables the GMS entity graph cache (Hazelcast snapshots). Set false when Hazelcast is unavailable or on non-GMS pods.
  • ENTITY_GRAPH_CACHE_CONFIG_FILE_ENABLED / ENTITY_GRAPH_CACHE_CONFIG_FILE / ENTITY_GRAPH_CACHE_CONFIG_JSON: overlay graph definitions and bounds; see GMS Entity Graph Cache.
  • DATAHUB_USE_INFERENCE_V2 (default false): routes smart-assertion training to the V2 trainer. Requires DATAHUB_USE_OBSERVE_MODELS=true.
  • DATAHUB_USE_OBSERVE_MODELS (default true): gates whether observe-models may be loaded by the V1 preprocessor; prerequisite for DATAHUB_USE_INFERENCE_V2.
  • DATAHUB_EXECUTOR_ENABLE_DELTA_BOUNDS (default true): gates whether the V1 trainer emits delta-space prediction bounds.
  • DATAHUB_EXECUTOR_LOG_GC_ENABLED (default false): enables the in-process ingestion-log garbage collector (retention and size-cap tunables via the related DATAHUB_EXECUTOR_LOG_GC_* variables).
  • DATAHUB_EXECUTOR_ALLOW_CALL_STATEMENTS (default false): allows a stored-procedure CALL statement in custom SQL assertions. Requires an executor restart.
  • DATAHUB_CLIENT_CERT_PATH / DATAHUB_CLIENT_KEY_PATH: enable mTLS client authentication for CLI/SDK outbound HTTPS.
  • DATAHUB_AUTH_TYPE: selects OAuth for the CLI, default sink, and env-resolved emitters.
  • CORP_USER_SIBLINGS_ENABLED: enables the daily user-siblings de-duplication source.
  • AUTH_GMS_SESSION_COOKIE_NAME (default SESSION): name of the GMS session cookie the frontend invalidates on OAuth2 logout.
  • INGESTION_MAX_SERIALIZED_NAME_LENGTH (default 16 MB): tunable limit for JSON property-name length on metadata deserialization.
  • NOTIFICATION_LOGO_URL / NOTIFICATION_FOOTER_TEXT: customize the logo and footer on SMTP notification emails.
  • LOG_AGGREGATOR_ENDPOINT (opt-in): ships each service's logs to a separately-deployed log aggregator and enables the support bundle's log collector.

Deprecations​

  • BigQuery usage.* window fields deprecated. usage.start_time, usage.end_time, usage.bucket_duration, and usage.max_query_duration are deprecated in favor of their top-level equivalents. Previously the usage-scoped fields were silently ignored; they now take effect (with a warning), which can widen the ingested window and increase query cost. Move them to the top level of the recipe; setting the same field in both places is now an error.