Skip to main content

DataHub Dagster Plugin

DataHub Dagster Plugin

Automatic lineage and run metadata from Dagster into DataHub — captures asset definitions, job runs, and dataset-level lineage with no manual instrumentation.

What you can do​

  • Capture asset lineage — automatically extract upstream/downstream relationships between Dagster assets and external datasets
  • Track run history — record job execution status and task-level outcomes in DataHub
  • Map Dagster assets to DataHub entities — assets appear as datasets in your DataHub catalog
  • Works with any DataHub deployment — self-hosted or DataHub Cloud

Installation​

pip install acryl-datahub-dagster-plugin

Quickstart​

Add the DataHub sensor to your Dagster project:

from datahub.ingestion.graph.config import DatahubClientConfig
from datahub_dagster_plugin.sensors.datahub_sensors import (
    DatahubDagsterSourceConfig,
    make_datahub_sensor,
)

config = DatahubDagsterSourceConfig(
    datahub_client_config=DatahubClientConfig(server="http://localhost:8080"),
    dagster_url="http://localhost:3000",
)
datahub_sensor = make_datahub_sensor(config=config)

Register it alongside your jobs and assets in your Definitions:

from dagster import Definitions

defs = Definitions(
    assets=[...],
    jobs=[...],
    sensors=[datahub_sensor],
)

Once the sensor is running, every job run will emit lineage and status metadata to DataHub automatically.