Skip to main content

SQLMesh

Overview​

SQLMesh is an open-source DataOps framework for building and operating SQL transformation pipelines. It manages model versioning, virtual environments, incremental execution, and data quality audits across warehouses including Snowflake, BigQuery, Databricks, DuckDB, and others.

DataHub ingests SQLMesh model metadata—schema, lineage, column-level lineage, descriptions, and data quality audits—and links each model to its corresponding warehouse view as a sibling entity. This follows the same pattern as the dbt connector: the SQLMesh entity owns model definitions and lineage while the warehouse connector contributes runtime metadata such as query history, profiling, and usage. DataHub merges both views in the UI automatically.

Concept Mapping​

SQLMesh conceptDataHub entity / aspect
ModelDataset (urn:li:dataPlatform:sqlmesh,...)
Model depends_onUpstreamLineage (coarse-grained)
Column dependenciesFineGrainedLineage (column-level lineage)
Model descriptionDatasetProperties.description
Column descriptionsSchemaMetadata field descriptions
Model tagsGlobalTags
Model ownerOwnership
Audit definitionAssertion entity (AssertionInfo)
Audit run resultAssertionRunEvent (pass / fail)
Warehouse viewSibling dataset on the target warehouse platform
Database / SchemaContainer hierarchy

Module sqlmesh​

Incubating

Important Capabilities​

CapabilityStatusNotes
Descriptions✅Supported when model descriptions are defined.
Detect Deleted Entities✅Enabled by default via stateful ingestion.
Schema Metadata✅Enabled by default.
Table-Level Lineage✅Enabled by default.

Overview​

The sqlmesh source plugin reads SQLMesh project metadata directly from the SQLMesh Python API—no running SQLMesh server is required. It emits Dataset entities on the urn:li:dataPlatform:sqlmesh platform and links each one to its corresponding warehouse view (Snowflake, BigQuery, DuckDB, etc.) as a sibling, so DataHub merges both into a single unified view in the UI.

Prerequisites​

  • Python 3.9 or later
  • The sqlmesh Python package installed in the ingestion environment: pip install 'acryl-datahub[sqlmesh]'
  • Read access to the SQLMesh project directory (config files and model SQL)
  • If using a remote gateway (Snowflake, BigQuery, etc.), valid gateway credentials in the SQLMesh project config—the source plugin loads the SQLMesh context which opens a connection to resolve model metadata

Concepts and setup​

The rest of this section covers the concepts and setup you need before configuring a recipe.

Project location: local, S3, or Git​

project_path accepts three kinds of location, so the SQLMesh project does not have to be checked out next to the ingestion process:

  • Local directory (default) — a filesystem path, e.g. project_path: /opt/sqlmesh_project.
  • S3 prefix — project_path: s3://my-bucket/sqlmesh_project with an aws_connection block for credentials. The entire prefix is downloaded to a temporary directory for the run, so the whole project tree (config, models/, audits/, macros/, seeds) must live under that prefix.
  • Git repository — a git_info block shallow-clones the repo (authenticated with an SSH deploy key); project_path is then interpreted relative to the checkout (., the default, is the repo root; use a subdirectory such as sqlmesh/ when the project isn't at the root).
# S3
source:
  type: sqlmesh
  config:
    project_path: s3://my-bucket/sqlmesh_project
    aws_connection:
      aws_region: us-east-1
      # aws_access_key_id / aws_secret_access_key, aws_role, or an instance profile

# Git
source:
  type: sqlmesh
  config:
    git_info:
      repo: https://github.com/my-org/my-sqlmesh-repo
      branch: main
      deploy_key_file: /secrets/sqlmesh_deploy_key
    project_path: sqlmesh # relative to the repo root

Fingerprint table abstraction​

SQLMesh internally manages model versions through fingerprint tables—versioned physical tables with hashes in their names (e.g., schema.model__47716296). These are implementation details that track snapshots and enable efficient incremental processing. The connector automatically abstracts away this complexity: fingerprint tables are never exposed in DataHub.

Instead, users see only the logical views (clean published model names like schema.model) and the physical source tables that feed into them. This keeps the data lineage graph clean and focused on business semantics, not internal versioning mechanics.

Sibling stitching​

For sibling URNs to stitch correctly, the sqlmesh connector and your warehouse connector must agree on the dataset name. Key config options:

  • target_platform_instance — must match the platform_instance in your warehouse recipe exactly.
  • default_catalog — required when SQLMesh model names are two-part (schema.model) but your warehouse connector emits three-part URNs (catalog.schema.table).
  • convert_urns_to_lowercase — auto-enabled for Snowflake; set manually for other case-folding warehouses.

Column-level lineage​

SQLMesh parses all SQL through SQLGlot, so column-level lineage is available natively for all SQL models without any additional configuration. Python models (ibis, pandas) do not have column-level lineage available.

Data quality audits​

SQLMesh audit definitions (not_null, unique_values, unique_combination_of_columns, number_of_rows, forall, accepted_range, accepted_values) are emitted as typed DataHub Assertion entities and appear on the Validation tab of each dataset.

To surface pass/fail run results on the Validation tab, produce a JSON results file and point audit_results_path at it. The expected format is described in the config reference below.

Multi-gateway projects​

SQLMesh projects can declare multiple gateways targeting different warehouses (e.g. staging models on BigQuery, curated models on Snowflake). The connector reads ctx.engine_adapters to discover every gateway and routes each model's sibling URN to the correct warehouse platform automatically.

Configuration:

  • Top-level target_platform / target_platform_instance / default_catalog continue to apply to the project's default gateway (default_gateway: in the SQLMesh config).
  • gateway_overrides: provides per-gateway values for any non-default gateway. Each entry takes the same four fields, all optional. Anything omitted is auto-detected from the gateway's connection config or falls back to the project-level default.

Example multi-gateway recipe:

source:
  type: sqlmesh
  config:
    project_path: /path/to/sqlmesh_project
    target_platform: snowflake # for the default gateway
    target_platform_instance: prod_snowflake
    default_catalog: analytics
    gateway_overrides:
      bigquery_lake:
        target_platform: bigquery
        target_platform_instance: prod_bigquery
        default_catalog: lake-prod
      duckdb_sandbox:
        target_platform: duckdb
        target_platform_instance: dev_duckdb

For a single-gateway project just omit gateway_overrides — behaviour is unchanged.

Authentication and secrets​

Two kinds of credentials matter for an ingest run:

CredentialWhere it livesHow to provide it
DataHub GMS tokenDataHub sink config${DATAHUB_TOKEN} interpolated in the recipe
Tobiko Cloud token (Enterprise only)This connector's configtobiko_cloud_token: ${TOBIKO_TOKEN} or tobiko_cloud_token_file: ${SECRETS_DIR}/tobiko
Warehouse credentials (Snowflake, etc.)SQLMesh project's config.yaml or config.pySQLMesh's own env-var interpolation — password: ${SNOWFLAKE_PASSWORD} etc.

The two patterns the connector itself supports:

  1. Env-var inline — every config field is interpolated by DataHub's YAML loader, so any ${ENV_VAR} reference works. Use this for tokens that change rarely.

  2. Env-var-specified file path — for tokens that rotate (Kubernetes projected secret volumes etc.) point a *_file field at the mount path:

    source:
      type: sqlmesh
      config:
        tobiko_cloud_token_file: ${SECRETS_DIR}/tobiko-cloud-token

    File reads go through a 60-second TTL cache so the connector picks up rotated tokens within one cache window without restarting the ingestion process.

For multi-gateway projects, every gateway in the SQLMesh project must have working credentials at ingest time — SQLMesh opens a connection per gateway when loading the Context. Set those credentials via env vars in the SQLMesh project's config.yaml:

gateways:
  snowflake_prod:
    connection:
      type: snowflake
      user: ${SNOWFLAKE_USER}
      password: ${SNOWFLAKE_PASSWORD}
      account: ${SNOWFLAKE_ACCOUNT}
  bigquery_lake:
    connection:
      type: bigquery
      method: service-account
      keyfile: ${BIGQUERY_KEYFILE}

The connector inherits whatever credentials SQLMesh resolves; it never reads warehouse creds directly. If a gateway can't open its connection the corresponding models still get URN routing (sibling stitching uses metadata only), but volume / freshness signals that need ctx.engine_adapter will skip for that gateway and the report shows has_warehouse_query_access: False.

Warehouse permissions​

The connector inherits whatever SQLMesh's gateway has. The minimum the SQLMesh user / service account needs at ingestion time:

  • Read on the SQLMesh state schema (sqlmesh__*, or the configured state_connection schema). Used to read snapshot.updated_ts for freshness and snapshot.table_name() for the authoritative fingerprint name. Without this the connector still emits assertion definitions but skips the freshness OperationAspect and the volume row count.
  • SELECT on the fingerprint tables themselves (<catalog>.sqlmesh__<schema>.*). Used for the SELECT COUNT(*) query that populates DatasetProfile.rowCount for the volume assertion baseline.
  • No write privileges needed — the connector is read-only.

If you're using a Tobiko Cloud deployment the state-store read is fulfilled by the cloud API token; the warehouse read still applies.

The SQLMesh project directory is required​

Unlike state and warehouse access (which the connector can degrade around), the SQLMesh project files themselves are mandatory. SQLMesh's Python API loads everything — model SQL, audits, macros, the gateway config — from a directory; there is no API path that returns model metadata without it. If the connector can't read config.yaml / config.py at project_path, ingestion fails fast.

This affects deployment in a few ways depending on where the project lives:

  • Same repo as the warehouse / dbt project: trivial — the project is in the working tree, point project_path at it.
  • Separate repo: clone or sync the SQLMesh project alongside the recipe at ingest time. In CI this is typically a second actions/checkout for the SQLMesh repo. In Kubernetes a git-sync sidecar (or initContainer that does a shallow clone) is the standard pattern, both for keeping the project current and for avoiding a full image rebuild on every commit.
  • Tobiko Cloud-managed projects: the source files still live in the user's git repo even when state is in Tobiko Cloud — Tobiko Cloud doesn't host the SQL. Same patterns apply.
  • Project doesn't fit on the ingestion host (very large repos): the connector only reads the files; no SQLMesh-side execution happens during ingest. A sparse-checkout limited to the SQLMesh project subtree is sufficient.

What the connector reads from the project at ingestion time:

  • config.yaml / config.py — for the gateway list and SQLMesh defaults
  • models/**/*.sql and models/**/*.py — model definitions, audits, lineage
  • audits/** and macros/** — referenced by models
  • external_models.yaml — declared external sources (Category 2)

Optional / read separately from a configured path:

  • audit_results_path JSON — produced externally by sqlmesh audit --output or your own tooling; outside project_path

What works with vs without state-store access​

The connector probes three capabilities once at Context load (has_state, has_warehouse_query, has_graph) and surfaces them in the report. Different emissions depend on different probes.

Happy path — all three available:

EmissionDepends on
Dataset entities + schema + lineagenothing (just project files)
Sibling URN routingnothing
Assertion definitions (audit only)nothing
Assertion run events (audit pass/fail)audit_results_path JSON file (no probes needed)
Volume DatasetProfile.rowCounthas_state (for authoritative fingerprint name) + has_warehouse_query (for COUNT)
Pipeline OperationAspect.lastUpdatedTimestamphas_state (for snapshot.updated_ts)

Minimal path — state store unavailable (has_state: false):

The connector still emits the full metadata model — datasets, schema, column-level lineage, audits, assertion definitions, siblings, containers, ownership, tags. The audit assertions are CUSTOM (SQLMesh runs them, DataHub records the definition and the pass/fail run events from audit_results_path), so they populate the Validation tab regardless of state availability. What you lose:

  • OperationAspect for fingerprint rebuild timestamps — DataHub doesn't see "when did SQLMesh last apply this model".
  • DatasetProfile.rowCount from this connector — your warehouse connector still profiles the underlying table, so volume baselines for Cloud Monitor flow through the warehouse connector instead.

Reasons you'd land in this state:

  • Tobiko Cloud without a token: the connector falls back to a local DuckDB stub for state; has_state is technically True against the stub but useless. Set tobiko_cloud_token to recover.
  • State schema permissions the SQLMesh role doesn't have.
  • Fresh project that has never run plan/apply: the state schema exists but is empty; nothing to skip into.

Minimal recipe that explicitly opts out of state-dependent emissions (useful for environments where the state store is off-limits and you want a clean ingest report without the "skipped" warnings):

source:
  type: sqlmesh
  config:
    project_path: /path/to/sqlmesh_project
    gateway: snowflake_prod
    target_platform_instance: prod_snowflake
    default_catalog: ANALYTICS
    sqlmesh_platform_instance: prod
    include_column_lineage: true
    convert_urns_to_lowercase: true
    # OperationAspect + DatasetProfile are emitted when state/warehouse are reachable.
    # Create freshness/volume monitors in DataHub against those timeseries.
    stateful_ingestion:
      enabled: true

What this gives you: complete dataset metadata, the audit Validation tab, and Operation/Profile timeseries for the monitors you create. Strictly less than the happy path, but still the bulk of the value.

Real-world configuration examples​

End-to-end Snowflake setup, the most common case:

# In your SQLMesh project's config.yaml
gateways:
  snowflake_prod:
    connection:
      type: snowflake
      user: ${SNOWFLAKE_USER}
      password: ${SNOWFLAKE_PASSWORD}
      account: ${SNOWFLAKE_ACCOUNT}
      warehouse: COMPUTE_WH
      role: SQLMESH_ROLE
      database: ANALYTICS # the catalog SQLMesh writes to
default_gateway: snowflake_prod
model_defaults:
  dialect: snowflake
# DataHub ingestion recipe
source:
  type: sqlmesh
  config:
    project_path: /path/to/sqlmesh_project
    gateway: snowflake_prod
    # target_platform auto-detects to "snowflake" from the gateway dialect
    target_platform_instance: prod_snowflake # must match Snowflake connector
    default_catalog: ANALYTICS # set when SQLMesh model names are 2-part
    environment: prod # SQLMesh environment to ingest from
    sqlmesh_platform_instance: prod # namespace for urn:li:dataPlatform:sqlmesh
    include_column_lineage: true
    convert_urns_to_lowercase: true # Snowflake auto-folds — required for stitching
    stateful_ingestion:
      enabled: true
      remove_stale_metadata: true

sink:
  type: datahub-rest
  config:
    server: ${DATAHUB_GMS_URL}
    token: ${DATAHUB_TOKEN}

Env vars expected at runtime: SNOWFLAKE_USER, SNOWFLAKE_PASSWORD, SNOWFLAKE_ACCOUNT, DATAHUB_GMS_URL, DATAHUB_TOKEN.

BigQuery service-account JSON​
# SQLMesh config.yaml
gateways:
  bigquery_lake:
    connection:
      type: bigquery
      method: service-account
      keyfile: ${GOOGLE_APPLICATION_CREDENTIALS} # path to JSON key
      project: my-gcp-project
default_gateway: bigquery_lake
model_defaults:
  dialect: bigquery

GOOGLE_APPLICATION_CREDENTIALS may point at a path mounted by a secret manager (GCP Secret Manager, K8s projected secret, etc.). The file is read by the BigQuery client library directly; the connector itself doesn't touch it.

Tobiko Cloud Enterprise​
source:
  type: sqlmesh
  config:
    project_path: /path/to/sqlmesh_project
    gateway: snowflake_prod # must be set so tobikodata knows which gateway
    # Either inline:
    tobiko_cloud_token: ${TOBIKO_CLOUD_TOKEN}
    # OR file-based (preferred for K8s with rotated tokens):
    tobiko_cloud_token_file: ${SECRETS_DIR}/tobiko-cloud-token
    # Only needed when the project's config.py doesn't already declare it
    # tobiko_cloud_url: https://my-org.tobiko.cloud

Without a token, the connector falls back to a local DuckDB stub for state — model definitions still load from project files, but snapshot.updated_ts and other state-derived signals are unavailable (has_state_store_access: False in the report).

Running in Kubernetes​

The connector ships no special K8s requirements. Common setup:

  • Mount the SQLMesh project as a configMap or git-sync sidecar.
  • Mount warehouse credentials and the Tobiko Cloud token via projected secret volumes under e.g. /var/run/secrets/. Point tobiko_cloud_token_file at the mount path. The 60-second TTL cache means token rotation is picked up within one cache window without pod restart.
  • Set SECRETS_DIR so the env-var-interpolation pattern shown above works without hardcoding paths.
  • Run the connector as a CronJob or via Airflow / Dagster / GitHub Actions on whatever cadence makes sense for your refresh rate (commonly once per day, aligned with the SQLMesh apply cron).
Running in CI/CD (GitHub Actions example)​
- name: Ingest SQLMesh metadata to DataHub
  env:
    SNOWFLAKE_USER: ${{ secrets.SNOWFLAKE_USER }}
    SNOWFLAKE_PASSWORD: ${{ secrets.SNOWFLAKE_PASSWORD }}
    SNOWFLAKE_ACCOUNT: ${{ secrets.SNOWFLAKE_ACCOUNT }}
    DATAHUB_GMS_URL: ${{ secrets.DATAHUB_GMS_URL }}
    DATAHUB_TOKEN: ${{ secrets.DATAHUB_TOKEN }}
  run: |
    pip install 'acryl-datahub[sqlmesh]'
    datahub ingest -c recipe.yml

Run this step after your sqlmesh plan / sqlmesh apply step in the same workflow so the freshness OperationAspect reflects the just-completed rebuild, and so the optional audit_results_path JSON (if you produce one with sqlmesh audit --output) shows the latest pass/fail.

Install the Plugin​

pip install 'acryl-datahub[sqlmesh]'

Starter Recipe​

Check out the following recipe to get started with ingestion! See below for full configuration options.

For general pointers on writing and running a recipe, see our main recipe guide.

source:
  type: sqlmesh
  config:
    project_path: /path/to/sqlmesh_project
    # gateway: my_gateway  # defaults to the project's default gateway
    # Tobiko Cloud projects: point at your cloud state store. The token is only
    # needed for CI/service accounts; SSO users authenticate via `tcloud auth login`.
    # tobiko_cloud_url: https://cloud.tobikodata.com/sqlmesh/<org>/<project>/
    # tobiko_cloud_token: "${TCLOUD_TOKEN}"
    # target_platform: snowflake  # auto-detected from gateway connection type
    # target_platform_instance: prod_snowflake  # must match warehouse connector
    # default_catalog: analytics  # required for 2-part model names + 3-part warehouse URNs
    environment: prod # SQLMesh environment to ingest from
    include_column_lineage: true
    stateful_ingestion:
      enabled: true
      remove_stale_metadata: true

sink:
  type: datahub-rest
  config:
    server: "http://localhost:8080"
    # token is optional for a local unauthenticated DataHub; uncomment for
    # authenticated / hosted (DataHub Cloud) instances.
    # token: "${DATAHUB_TOKEN}"

Config Details​

Note that a . is used to denote nested fields in the YAML recipe.

FieldDescription
audit_results_path
One of string, null
Path to a JSON file containing SQLMesh audit pass/fail results. When set, the connector emits AssertionRunEvent aspects for each result, making pass/fail status visible on the DataHub Data Quality tab. The file must exist at ingestion time; results with no matching assertion definition are silently skipped.

Expected JSON format::

{
"metadata": {"generated_at": "2024-01-01T00:00:00Z"},
"results": [
{
"model": "myschema.orders",
"audit": "not_null",
"columns": ["order_id"],
"status": "pass",
"failing_rows": 0
}
]
}

Valid status values: pass, fail, skip.
Default: None
convert_column_urns_to_lowercase
One of boolean, null
Force column names in field URNs to lowercase. Defaults to the same value as convert_urns_to_lowercase when not set. Set explicitly when column name casing in your warehouse connector differs from the dataset URN casing (e.g. Snowflake uppercases column names).
Default: None
convert_urns_to_lowercase
boolean
Whether to convert dataset urns to lowercase. This value is part of each dataset's URN identity, so it must stay fixed for the life of a deployment. Changing it after data has been ingested re-keys every dataset (e.g. MyDb.MyTable becomes mydb.mytable); with stateful ingestion enabled the old-cased URNs are then soft-deleted as stale while the new-cased ones are created, producing duplicate or orphaned entities. Pick one value before the first run and leave it unchanged.
Default: False
default_catalog
One of string, null
Default catalog (database) to prepend to model names that are only two-part (schema.model). Required for sibling URN stitching when your warehouse connector emits three-part URNs (catalog.schema.table) but SQLMesh model names omit the catalog. Example: set to 'analytics' so that 'star.dim_developer' becomes 'analytics.star.dim_developer', matching what the Snowflake connector emits.
Default: None
detect_stale_fingerprints
boolean
When enabled, detect SQLMesh fingerprint tables that haven't been regenerated recently (no plan/apply runs). Use this to monitor if SQLMesh transformations are running on their expected schedules. Reads snapshot timestamps from the SQLMesh state store; silently skipped when state is unreachable. When a fingerprint is stale, a custom property 'sqlmesh.fingerprint_stale' is added to the dataset.
Default: False
emit_incidents_on_failure
boolean
Emit a DataHub Incident entity (urn:li:incident:…) every time _emit_audit_run_events reads a "fail" result from the audit_results_path JSON file. The incident links back to the assertion via IncidentSource(type=ASSERTION_FAILURE, sourceUrn=<assertion>) so the Incidents tab on the dataset shows the failure history. Standard DataHub entity — works regardless of edition. Cloud additionally adds Slack threading and triage ML on top. Re-emitting the same incident is idempotent because the URN is derived from a hash of (assertion_urn, run_id).
Default: True
environment
string
SQLMesh environment to ingest from (e.g. prod, dev).
Default: prod
fingerprint_staleness_threshold_hours
integer
Number of hours before a fingerprint table is considered stale. Only used when detect_stale_fingerprints=True. A fingerprint that hasn't been updated (via plan/apply) within this many hours will be flagged as stale. Default: 48 hours (2 days).
Default: 48
gateway
One of string, null
SQLMesh gateway name. Defaults to the project's default gateway.
Default: None
include_column_lineage
boolean
Emit column-level lineage derived from SQLMesh's SQL parsing (via SQLGlot). Available for all SQL models natively — no separate parsing step needed. Disable for very large projects where per-column analysis is too slow.
Default: True
include_database_name
boolean
Whether to include the database/catalog component in warehouse sibling URNs. Set to false for platforms like Athena that omit the catalog from their URNs. When false, 'analytics.star.dim_developer' becomes 'star.dim_developer' in the warehouse URN.
Default: True
include_lineage
boolean
Emit model-to-model lineage derived from SQLMesh DAG dependencies. Disable if lineage is managed by another connector or not needed.
Default: True
include_model_properties
boolean
Emit dataset properties (description, custom properties) for each model. Disable to ingest schema and lineage only.
Default: True
include_schema
boolean
Emit column schema metadata for each model. Disable to reduce ingestion volume when schema is already captured by a warehouse connector.
Default: True
incremental_lineage
boolean
Use patch/incremental lineage mode for non-SQLMesh entities (e.g. external warehouse tables referenced in lineage). When enabled, the plugin adds lineage edges without overwriting edges the warehouse connector previously discovered. Must match the warehouse connector's incremental_lineage setting.
Default: True
owner_extraction_pattern
One of string, null
Regex pattern to extract the owner identity from the SQLMesh model owner field. The first capture group is used as the owner. Example: (.*)@.* extracts the username from an email address. When not set, the owner field value is used as-is.
Default: None
platform_instance
One of string, null
The instance of the platform that all assets produced by this recipe belong to. This should be unique within the platform. See https://docs.datahub.com/docs/platform-instances/ for more details.
Default: None
preview_urns
boolean
Before emitting metadata, print a sample of generated sqlmesh URNs and expected warehouse sibling URNs side-by-side to the log. Helps validate URN stitching before a full run. Set to true for a dry-run style check, or use --dry-run on the CLI.
Default: False
preview_urns_sample_size
integer
Number of sample models to include in the URN preview output.
Default: 10
project_path
string
Location of the SQLMesh project. One of: a local directory path; an s3://bucket/prefix pointing at the project tree (requires aws_connection); or — when git_info is set — a path relative to the cloned repository (., the default, is the repo root).
Default: .
skip_external_models_in_lineage
boolean
When false (default), declared external models (defined in external_models.yaml) appear as SQLMesh 'Source' entities in the lineage graph. When true, lineage from managed models points directly to the warehouse URN for external models — skipping the SQLMesh entity. Produces a cleaner graph if external models are already well-represented by the warehouse connector.
Default: False
sqlmesh_is_primary_sibling
boolean
When true (default), the SQLMesh entity is the primary sibling — its name, description, and lineage take precedence in the merged UI view. The warehouse entity contributes runtime metadata (tags, query history, profiling, usage). Matches dbt's dbt_is_primary_sibling=true default. Set to false if your warehouse entity carries authoritative documentation.
Default: True
sqlmesh_platform_instance
One of string, null
Platform instance for the sqlmesh entities themselves. Use this to namespace the urn:li:dataPlatform:sqlmesh entities and avoid collisions when multiple SQLMesh projects write to the same warehouse.
Default: None
tag_prefix
string
Prefix prepended to SQLMesh model tags when creating DataHub tags. Example: a model tag 'pii' becomes DataHub tag 'sqlmesh:pii'. Set to empty string to use tags as-is.
Default: sqlmesh:
target_platform
One of string, null
Warehouse platform SQLMesh writes to (e.g. snowflake, bigquery, databricks). Auto-detected from the gateway connection type if not set — only specify this when auto-detection produces the wrong value. Must match the platform used in your warehouse connector recipe so that sibling URNs stitch correctly.
Default: None
target_platform_instance
One of string, null
Platform instance for the target warehouse. Must exactly match the platform_instance configured in your warehouse connector recipe so that sibling URNs stitch correctly.
Default: None
tobiko_cloud_token
One of string(password), null
Tobiko Cloud API token. Set this when the SQLMesh project is configured against Tobiko Cloud (an EnterpriseConfig with a cloud state connection) and DataHub should read from the real cloud state store. Mutually exclusive with tobiko_cloud_token_file. When neither is set, DataHub falls back to a local DuckDB stub so Context init succeeds without creds — model definitions still come from the project files, but anything that depends on remote state (snapshot history, environment promotions) is unavailable. Requires gateway to be set; the gateway name determines which SQLMESH__GATEWAYS__<gw>__STATE_CONNECTION__* variables get populated for tobikodata to read.
Default: None
tobiko_cloud_token_file
One of string, null
Path to a file containing the Tobiko Cloud API token (single line). Re-read with a 60-second cache TTL so projected Kubernetes secret mounts pick up rotated tokens without a process restart. Mutually exclusive with tobiko_cloud_token.
Default: None
tobiko_cloud_url
One of string, null
Tobiko Cloud state-store URL. Only needed when the project's config.py does not already declare it on its cloud state connection. Used for both static-token and SSO auth, so it must be https:// whenever it is set (credentials/state travel over it).
Default: None
env
string
The environment that all assets produced by this connector belong to
Default: PROD
aws_connection
One of AwsConnectionConfig, null
AWS connection details for loading the project from an s3:// project_path. Required whenever project_path is an S3 URI. The entire prefix is downloaded to a temp directory for the run.
Default: None
aws_connection.aws_access_key_id
One of string, null
AWS access key ID. Can be auto-detected, see the AWS boto3 docs for details.
Default: None
aws_connection.aws_advanced_config
object
Advanced AWS configuration options. These are passed directly to botocore.config.Config.
aws_connection.aws_endpoint_url
One of string, null
The AWS service endpoint. This is normally constructed automatically, but can be overridden here.
Default: None
aws_connection.aws_profile
One of string, null
The named profile to use from AWS credentials. Falls back to default profile if not specified and no access keys provided. Profiles are configured in ~/.aws/credentials or ~/.aws/config.
Default: None
aws_connection.aws_proxy
One of string, null
A set of proxy configs to use with AWS. See the botocore.config docs for details.
Default: None
aws_connection.aws_region
One of string, null
AWS region code.
Default: None
aws_connection.aws_retry_mode
Enum
One of: "legacy", "standard", "adaptive"
Default: standard
aws_connection.aws_retry_num
integer
Number of times to retry failed AWS requests. See the botocore.retry docs for details.
Default: 5
aws_connection.aws_secret_access_key
One of string(password), null
AWS secret access key. Can be auto-detected, see the AWS boto3 docs for details.
Default: None
aws_connection.aws_session_token
One of string(password), null
AWS session token. Can be auto-detected, see the AWS boto3 docs for details.
Default: None
aws_connection.read_timeout
number
The timeout for reading from the connection (in seconds).
Default: 60
aws_connection.aws_role
One of string, array, null
AWS roles to assume. If using the string format, the role ARN can be specified directly. If using the object format, the role can be specified in the RoleArn field and additional available arguments are the same as boto3's STS.Client.assume_role.
Default: None
aws_connection.aws_role.union
One of string, AwsAssumeRoleConfig
aws_connection.aws_role.union.RoleArn ❓
string
ARN of the role to assume.
aws_connection.aws_role.union.ExternalId
One of string, null
External ID to use when assuming the role.
Default: None
gateway_overrides
map(str,GatewayOverride)
Per-gateway overrides for warehouse-URN construction.

SQLMesh projects can declare multiple gateways (e.g. Snowflake for some
models, BigQuery for others). The top-level target_platform /
target_platform_instance / default_catalog apply to the default
gateway; gateway_overrides lets you set per-gateway values for the
others. Anything left None falls back to auto-detection from
ctx.engine_adapters[gateway].dialect.
gateway_overrides.key.target_platform
One of string, null
Warehouse platform for this gateway. Auto-detected from the gateway connection type if not set.
Default: None
gateway_overrides.key.convert_urns_to_lowercase
One of boolean, null
Force-lowercase URNs for this gateway. Defaults to the project-level value, or True for Snowflake.
Default: None
gateway_overrides.key.default_catalog
One of string, null
default_catalog for this gateway. Prepended to 2-part model names to build 3-part warehouse URNs.
Default: None
gateway_overrides.key.target_platform_instance
One of string, null
platform_instance for this gateway. Must match the warehouse connector's platform_instance for sibling URN stitching.
Default: None
git_info
One of GitInfo, null
Git repository to shallow-clone (authenticated with an SSH deploy key) and load the SQLMesh project from. When set, project_path is interpreted relative to the checkout (e.g. project_path: sqlmesh/ for a project in a repo subdirectory).
Default: None
git_info.repo ❓
string
Name of your Git repo e.g. https://github.com/datahub-project/datahub or https://gitlab.com/gitlab-org/gitlab. If organization/repo is provided, we assume it is a GitHub repo.
git_info.branch
string
Branch on which your files live by default. Typically main or master. This can also be a commit hash.
Default: main
git_info.clone_timeout
One of integer, null
Timeout in seconds for git clone operations. Set to None to disable the timeout.
Default: 300
git_info.deploy_key
One of string(password), null
A private key that contains an ssh key that has been configured as a deploy key for this repository. See deploy_key_file if you want to use a file that contains this key.
Default: None
git_info.deploy_key_file
One of string(file-path), null
A private key file that contains an ssh key that has been configured as a deploy key for this repository. Use a file where possible, else see deploy_key for a config field that accepts a raw string. We expect the key not have a passphrase.
Default: None
git_info.repo_ssh_locator
One of string, null
The url to call git clone on. We infer this for github and gitlab repos, but it is required for other hosts.
Default: None
git_info.url_subdir
One of string, null
Prefix to prepend when generating URLs for files - useful when files are in a subdirectory. Only affects URL generation, not git operations.
Default: None
git_info.url_template
One of string, null
Template for generating a URL to a file in the repo e.g. '{repo_url}/blob/{branch}/{file_path}'. We can infer this for GitHub and GitLab repos, and it is otherwise required.It supports the following variables: {repo_url}, {branch}, {file_path}
Default: None
model_kind_filter
One of array, null
Filter which model kinds to ingest. When set, only models whose kind matches one of the listed values are processed. Valid values: FULL, INCREMENTAL_BY_TIME_RANGE, INCREMENTAL_BY_UNIQUE_KEY, INCREMENTAL_BY_PARTITION, SCD_TYPE_2_BY_TIME, SCD_TYPE_2_BY_COLUMN, VIEW, SEED, EXTERNAL, EMBEDDED. Default: all kinds.
Default: None
model_kind_filter.string
string
model_name_pattern
AllowDenyPattern
A class to store allow deny regexes.

Patterns are matched against the start of the string only, not the entire
string - a pattern does not need to match to the end to be considered a match.
For example, the pattern "prod" matches "prod", "prod_east", and "production".
To require an exact match, anchor your pattern explicitly, e.g. "^prod$".
model_name_pattern.ignoreCase
One of boolean, null
Whether to ignore case sensitivity during pattern matching.
Default: True
stateful_ingestion
One of StatefulStaleMetadataRemovalConfig, null
Stateful Ingestion Config
Default: None
stateful_ingestion.enabled
boolean
Whether or not to enable stateful ingest. Default: True if a pipeline_name is set and either a datahub-rest sink or datahub_api is specified, otherwise False
Default: False
stateful_ingestion.fail_safe_threshold
number
Prevents large amount of soft deletes & the state from committing from accidental changes to the source configuration if the relative change percent in entities compared to the previous state is above the 'fail_safe_threshold'.
Default: 75.0
stateful_ingestion.remove_stale_metadata
boolean
Soft-deletes the entities present in the last successful run but missing in the current run with stateful_ingestion enabled.
Default: True

Capabilities​

Supported model kinds​

All SQLMesh model kinds are supported: FULL, INCREMENTAL_BY_TIME_RANGE, INCREMENTAL_BY_UNIQUE_KEY, INCREMENTAL_BY_PARTITION, SCD_TYPE_2, VIEW, SEED, EXTERNAL, and EMBEDDED. Each maps to a DataHub dataset subtype (Model, Seed, Source, or Embedded).

Data quality assertions​

SQLMesh audits become DataHub CUSTOM assertions attached to the SQLMesh logical URN (sibling stitching surfaces them on the warehouse sibling in the UI):

  • Each built-in audit (not_null, unique_values, unique_combination_of_columns, number_of_rows, forall, accepted_range, accepted_values) and any unrecognised audit name becomes an AssertionType.CUSTOM assertion with customAssertion.type = "SQLMesh".
  • Useful semantics (scope / operator / aggregation / fields / kwargs) are carried as custom properties. The source plugin does not invent a SQL statement for DataHub to evaluate — SQLMesh executes the audits.
  • Pass/fail history comes from an external audit_results_path JSON (sqlmesh audit --output). Failures can emit Incidents when emit_incidents_on_failure is enabled.

Freshness and volume signals​

The source plugin does not emit FRESHNESS or VOLUME assertion definitions (there is no Cloud path that auto-attaches a monitor to an ingested assertion). Instead it emits the timeseries signals users can point monitors at:

  • OperationAspect with customOperationType=SQLMESH_FINGERPRINT_REBUILD from snapshot.updated_ts (when the SQLMesh state store is reachable)
  • DatasetProfile.rowCount from a warehouse COUNT(*) against the physical fingerprint table (when the gateway engine adapter is reachable)

sqlmesh.cron / interval_unit are still ingested as dataset custom properties for reference; they are not mapped into a freshness SLA.

Stateful ingestion​

When stateful_ingestion.enabled: true, the source plugin tracks emitted URNs across runs and soft-deletes entities that have been removed from the SQLMesh project.

Limitations​

  • Python models: Python-based SQLMesh models (ibis, pandas) are supported for schema extraction when columns_to_types is defined, but column-level lineage is not available.
  • Audit run results: Pass/fail status on the Validation tab requires an external JSON results file (audit_results_path). The source plugin does not execute audits itself at ingestion time.
  • Sibling merging: Sibling stitching requires the warehouse source plugin to be running and producing URNs that match this source plugin's output. Verify URN alignment using preview_urns: true before full ingestion. The warehouse sibling edge is patched (not overwritten) so an existing dbt sibling relationship is preserved.

Troubleshooting​

URNs do not stitch with warehouse entities​

Run with preview_urns: true to print a sample of generated sqlmesh and warehouse URN pairs before emitting. Compare these against the URNs your warehouse source plugin produces. Common causes:

  • target_platform_instance mismatch — must be identical in both recipes
  • Missing default_catalog — needed when model names are two-part but warehouse URNs are three-part
  • Case mismatch — enable convert_urns_to_lowercase: true for Snowflake and other case-folding warehouses

Context fails to load​

If ingestion logs Could not initialize SQLMesh context, check:

  • The project_path points to a valid SQLMesh project directory (contains a config.yaml or config.py)
  • The specified gateway name matches a gateway defined in the project config
  • All Python dependencies for Python models are installed in the ingestion environment (for example ibis-framework for ibis models)

DuckDB "cannot open file" error​

If you see IO Error: Cannot open file "...": No such file or directory when using a DuckDB gateway, your config.yaml contains a relative database: path (e.g. db/myproject.db). SQLMesh resolves this path against the working directory of the process that loads it, not against project_path.

Run the source plugin from the SQLMesh project directory:

cd /path/to/sqlmesh_project
datahub ingest -c recipe.yml

Or switch to an absolute path in your SQLMesh config.yaml:

gateways:
  local:
    connection:
      type: duckdb
      database: /absolute/path/to/myproject.db

Code Coordinates​

  • Class Name: datahub.ingestion.source.sqlmesh.sqlmesh_source.SqlmeshSource
  • Browse on GitHub
Questions?

If you've got any questions on configuring ingestion for SQLMesh, feel free to ping us on our Slack.

💡 Contributing to this documentation

This page is auto-generated from the underlying source code. To make changes, please edit the relevant source files in the metadata-ingestion directory.

Tip: For quick typo fixes or documentation updates, you can click the ✏️ Edit icon directly in the GitHub UI to open a Pull Request. For larger changes and PR naming conventions, please refer to our Contributing Guide.