# dbt
There are 2 sources that provide integration with dbt

<table>
<tr><td>Source Module</td><td>Documentation</td></tr><tr>
<td>

`dbt`

</td>
<td>


 [Read more...](#module-dbt)


</td>
</tr>
<tr>
<td>

`dbt-cloud`

</td>
<td>


 [Read more...](#module-dbt-cloud)


</td>
</tr>
</table>


## Overview

dbt is a data platform used to store and query analytical or operational data. Learn more in the [official dbt documentation](https://www.getdbt.com/).

The DataHub integration for dbt covers core metadata entities such as datasets/tables/views, schema fields, and containers. It also captures table- and column-level lineage and stateful deletion detection.

:::info Run both dbt and data warehouse ingestion for lineage

1. You must **run ingestion for both dbt and your data warehouse** (target platform). They can be run in any order.
2. It generates column lineage between the `dbt` nodes (e.g. when a model/snapshot depends on a dbt source or ephemeral model) as well as lineage between the `dbt` nodes and the underlying target platform nodes (e.g. BigQuery Table -> dbt source, dbt model -> BigQuery table/view).
3. It automatically generates "sibling" relationships between the dbt nodes and the target / data warehouse nodes. These nodes will show up in the UI with both platform logos.
4. We also support automated actions (like add a tag, term or owner) based on properties defined in dbt meta.

:::

## Concept Mapping

| Source Concept | DataHub Concept                                                        | Notes                   |
| -------------- | ---------------------------------------------------------------------- | ----------------------- |
| Source         | [Dataset](../../metamodel/entities/dataset.md)                         | Subtype `Source`        |
| Seed           | [Dataset](../../metamodel/entities/dataset.md)                         | Subtype `Seed`          |
| Model          | [Dataset](../../metamodel/entities/dataset.md)                         | Subtype `Model`         |
| Snapshot       | [Dataset](../../metamodel/entities/dataset.md)                         | Subtype `Snapshot`      |
| Semantic View  | [Dataset](../../metamodel/entities/dataset.md)                         | Subtype `Semantic View` |
| Test           | [Assertion](../../metamodel/entities/assertion.md)                     |                         |
| Test Result    | [Assertion Run Result](../../metamodel/entities/assertion.md)          |                         |
| Model Runs     | [DataProcessInstance](../../metamodel/entities/dataProcessInstance.md) |                         |


## Module `dbt`
![GA](https://img.shields.io/badge/support%20status-GA-brightgreen)


### Important Capabilities
| Capability | Status | Notes |
| ---------- | ------ | ----- |
| Column-level Lineage | ✅ | Enabled by default, configure using `include_column_lineage`. |
| [Detect Deleted Entities](../../../../metadata-ingestion/docs/dev_guides/stateful.md#stale-entity-removal) | ✅ | Enabled by default via stateful ingestion. |
| Table-Level Lineage | ✅ | Enabled by default. |
| Test Connection | ✅ | Enabled by default. |

### Overview

The `dbt` module ingests metadata from Dbt into DataHub. It is intended for production ingestion workflows and module-specific capabilities are documented below.

#### Dataset Statistics from Catalog

When you provide a `catalog.json` file generated by `dbt docs generate`, DataHub will automatically extract and display table statistics in the Stats tab. This includes:

- **Row Count**: Number of rows in the table (from `num_rows` stat)
- **Size in Bytes**: Approximate table size (from `num_bytes` stat)
- **Column Count**: Number of columns

These statistics are emitted as `DatasetProfile` aspects to DataHub, which powers the Stats tab in the UI.

**Requirements:**

- Statistics are only available for materialized tables (not views or ephemeral models)
- The underlying data warehouse must support table statistics (BigQuery, Snowflake, Redshift, etc.)
- The catalog.json must be generated after the models are built

**Config Options:**

```yaml
source:
  type: dbt
  config:
    manifest_path: target/manifest.json
    catalog_path: target/catalog.json
    # Control catalog stats emission
    entities_enabled:
      catalog_stats: "YES" # set to "NO" to disable
```

:::note Statistics Timestamp

The timestamp on the DatasetProfile uses the catalog's `generated_at` time when available, ensuring the stats reflect when the catalog was generated rather than when ingestion ran.

:::

#### Query Entities from dbt Meta

DataHub can ingest Query entities from the `meta.queries` field in your dbt models. This allows you to document "blessed" or commonly-used query patterns directly in dbt and surface them in DataHub's Queries tab for easy discovery and reuse by your team.

**Config Options:**

```yaml
source:
  type: dbt
  config:
    manifest_path: target/manifest.json
    # Control Query entity emission (default: YES)
    entities_enabled:
      queries: "NO" # or "YES" (default), "ONLY"
    # Limit queries per model (default: 100, set 0 for unlimited)
    max_queries_per_model: 100
```

:::note Integration with Warehouse Query Ingestion

If you're also using warehouse query ingestion (e.g., Snowflake usage, BigQuery audit logs), dbt-emitted queries will coexist with warehouse-discovered queries in the Queries tab. They're differentiated by source: dbt queries have `source: MANUAL` while warehouse queries typically have `source: SYSTEM`.

:::

##### How to Configure

The `meta.queries` field is defined in your dbt model's properties file (e.g., `schema.yml`, `models.yml`, or any `.yml` file in your dbt project). When you run `dbt docs generate` or `dbt compile`, this metadata is included in the `manifest.json` file, which DataHub then ingests.

**Add queries to your model's `meta` field in your dbt properties file:**

```yaml
# models/schema.yml or models/customers.yml
version: 2

models:
  - name: customers
    description: "Customer dimension table"
    meta:
      queries:
        - name: "Active customers (30d)"
          description: "Customers active in the last 30 days"
          sql: |
            SELECT *
            FROM {{ ref('customers') }}
            WHERE active = true
              AND last_seen > CURRENT_DATE - INTERVAL '30 days'
          tags: ["production", "analytics"]
          terms: ["CustomerData", "Engagement"]

        - name: "Revenue by customer"
          description: "Total revenue aggregated by customer"
          sql: |
            SELECT
              customer_id,
              SUM(amount) as total_revenue
            FROM {{ ref('customers') }}
            GROUP BY customer_id
          tags: ["finance", "reporting"]
```

**Then generate your dbt artifacts:**

```sh
dbt docs generate
# This creates/updates target/manifest.json with the meta.queries data
```

**Finally, run DataHub ingestion:**

```sh
datahub ingest -c your_dbt_recipe.yml
# DataHub reads manifest.json and creates Query entities
```

##### Field Reference

Each query in the `queries` list supports the following fields:

| Field         | Required | Type            | Description                                                    |
| ------------- | -------- | --------------- | -------------------------------------------------------------- |
| `name`        | ✅ Yes   | string          | Unique name for the query                                      |
| `sql`         | ✅ Yes   | string          | SQL statement for the query                                    |
| `description` | ❌ No    | string          | Human-readable description                                     |
| `tags`        | ❌ No    | list of strings | Tags for categorization (stored in customProperties)           |
| `terms`       | ❌ No    | list of strings | Glossary terms for classification (stored in customProperties) |

##### How It Works

1. **dbt Configuration**: You define `queries` in the `meta` field of your dbt model properties
2. **Manifest Generation**: When you run `dbt docs generate`, the `meta.queries` data is included in `manifest.json`
3. **DataHub Ingestion**: DataHub reads the manifest.json and extracts the `meta.queries` field
4. **Query Entity Creation**: Each query in `meta.queries` becomes a Query entity in DataHub
5. **URN Generation**: Query URN is generated as `urn:li:query:{dbt_unique_id}_{sanitized_query_name}` (e.g., `urn:li:query:model.my_project.customers_Active_customers_30d_`)
6. **Dataset Linking**: Queries are linked to the **target platform** dataset (e.g., Snowflake, Postgres) via QuerySubjects aspect, so they appear where analysts actually query
7. **UI Visibility**: Queries appear in the "Queries" tab of the target platform dataset in DataHub UI

##### Technical Details

- **Actor**: All queries are attributed to the `dbt_executor` actor
- **Timestamps**: Uses manifest `generated_at` for reproducibility; falls back to current time if unavailable
- **Custom Properties**: Tags/terms stored in `customProperties` (see [Known Limitations](#known-limitations) below)
- **SQL Truncation**: SQL exceeding 1MB is truncated with "..." suffix
- **URN Sanitization**: Consecutive special characters collapsed into single underscore (`[^a-zA-Z0-9_\-\.]+` → `_`)

##### Error Handling

| Scenario                           | Behavior                                                                             |
| ---------------------------------- | ------------------------------------------------------------------------------------ |
| `meta.queries` not a list          | Skipped with WARNING log                                                             |
| Query missing `name` or `sql`      | Skipped, all validation errors shown in log and `queries_failed_list`                |
| Duplicate query names              | Duplicate skipped, first definition wins (WARNING)                                   |
| Invalid `description` (not string) | Field ignored with WARNING log                                                       |
| Invalid `tags`/`terms` (not list)  | Field ignored with WARNING log                                                       |
| Empty values in tags/terms list    | Filtered out automatically                                                           |
| Manifest timestamp unparseable     | Falls back to current time with WARNING; tracked in `query_timestamps_fallback_used` |
| Queries on ephemeral model         | Skipped with WARNING (ephemeral models don't exist in target platform)               |
| Exceeds `max_queries_per_model`    | Only first N processed (configurable, default 100), WARNING logged                   |

All validation errors are logged at WARNING level and tracked in the ingestion report.

##### Example Output in DataHub

After ingestion, you'll see:

- Query entities in DataHub with name, description, and SQL statement
- Queries linked to the target platform dataset (visible in the dataset's "Queries" tab)
- Tags and terms visible in custom properties
- Creation/modification timestamps from dbt manifest
- Queries attributed to `dbt_executor` actor

##### Use Cases

- **Blessed Query Patterns**: Document approved query patterns for common analytics use cases
- **Query Templates**: Provide reusable query templates for team members
- **Best Practices**: Share optimized queries that follow your organization's standards
- **Self-Service Analytics**: Enable analysts to discover and reuse proven queries

##### Integration with Other dbt Features

The `meta.queries` feature works alongside other dbt metadata capabilities in DataHub:

| Feature                      | Purpose                                          | Config Key                 |
| ---------------------------- | ------------------------------------------------ | -------------------------- |
| **meta.queries**             | Define Query entities for discovery              | `entities_enabled.queries` |
| **meta_mapping**             | Map dbt meta fields to DataHub tags/terms/owners | `meta_mapping`             |
| **column_meta_mapping**      | Map column-level meta to DataHub aspects         | `column_meta_mapping`      |
| **owner_extraction_pattern** | Extract owners from meta fields                  | `owner_extraction_pattern` |
| **tag_prefix**               | Prefix for auto-generated tags                   | `tag_prefix`               |

**Example combining features:**

```yaml
source:
  type: dbt
  config:
    manifest_path: target/manifest.json
    catalog_path: target/catalog.json
    target_platform: snowflake

    # Enable query entities from meta.queries
    entities_enabled:
      queries: "YES"
    max_queries_per_model: 100

    # Map other meta fields to DataHub aspects
    meta_mapping:
      business_owner:
        match: ".*"
        operation: "add_owner"
        config:
          owner_type: user
      data_tier:
        match: ".*"
        operation: "add_tag"

    # Extract owners from specific meta patterns
    enable_meta_mapping: true
```

##### Known Limitations

1. **Tags/Terms in Custom Properties**: Query entities don't currently support native `GlobalTags` or `GlossaryTerms` aspects. Tags and terms are stored as comma-separated strings in `customProperties`. This means:

   - Cannot filter queries by tags in the DataHub UI search
   - Cannot apply tag-based governance policies to queries
   - Tags/terms appear as plain text in the Properties tab, not as clickable links

2. **No SQL Validation Against Model**: The `sql` field in `meta.queries` is not validated against the model it's defined on. You could define `sql: "SELECT * FROM products"` under the `customers` model. DataHub trusts that users define meaningful queries. Consider documenting your team's conventions for query definitions.

3. **URN Collision on Similar Names**: Query names are sanitized for URN generation. Names like `"Revenue (USD)"` and `"Revenue [USD]"` both become `Revenue_USD_`, causing a collision (second one is skipped with a warning). Use distinct, alphanumeric query names to avoid this.

4. **Ephemeral Models Not Supported**: Queries defined on ephemeral models (`materialized: ephemeral`) are skipped because ephemeral models don't exist as physical tables in the target platform. Queries are linked to target platform datasets, so there's no dataset to link to.

:::tip Choosing Between meta.queries and meta_mapping

- Use **meta.queries** for defining reusable SQL query patterns that should appear in the Queries tab
- Use **meta_mapping** for mapping arbitrary meta fields to DataHub tags, terms, and owners
- Both features can be used together - they operate on different parts of the meta object

:::

#### Remote File Access (S3 and GCS)

The dbt connector supports reading artifact files directly from cloud storage. All path fields (`manifest_path`, `catalog_path`, `sources_path`, `run_results_paths`) accept S3 (`s3://`) and GCS (`gs://`) URIs in addition to local paths.

**For S3**, provide `aws_connection` with your AWS credentials:

```yaml
source:
  type: dbt
  config:
    manifest_path: "s3://my-bucket/dbt/target/manifest.json"
    aws_connection:
      aws_access_key_id: "${AWS_ACCESS_KEY_ID}"
      aws_secret_access_key: "${AWS_SECRET_ACCESS_KEY}"
```

**For GCS**, provide `gcs_connection` with [HMAC keys](https://cloud.google.com/storage/docs/authentication/hmackeys):

```yaml
source:
  type: dbt
  config:
    manifest_path: "gs://my-bucket/dbt/target/manifest.json"
    catalog_path: "gs://my-bucket/dbt/target/catalog.json"
    target_platform: bigquery
    gcs_connection:
      credential:
        hmac_access_id: "${GCS_HMAC_ACCESS_ID}"
        hmac_access_secret: "${GCS_HMAC_ACCESS_SECRET}"
```

To create HMAC keys, see the [GCS HMAC key documentation](https://cloud.google.com/storage/docs/authentication/managing-hmackeys).

### Prerequisites

The artifacts used by this source are:

- [dbt manifest file](https://docs.getdbt.com/reference/artifacts/manifest-json) — **required**
  - Models, sources, seeds, snapshots, tests, exposures, semantic models, and lineage.
  - The manifest is the source of truth for which nodes and tests are active. Tests disabled in the manifest (via `enabled: false` or the `--exclude` flag) are excluded from DataHub even if they appear in `run_results.json`.
- [dbt catalog file](https://docs.getdbt.com/reference/artifacts/catalog-json) — optional but recommended
  - Column schemas and table statistics. Generate it with `dbt docs generate`.
  - dbt does not record schema data for ephemeral models, so DataHub shows ephemeral models in lineage but without an associated schema.
  - If not provided, DataHub falls back to basic column information from the manifest, or from a warehouse sibling already in DataHub (e.g. Snowflake, BigQuery).
- [dbt sources file](https://docs.getdbt.com/reference/artifacts/sources-json) — optional
  - Source freshness results.
  - If not provided, last-modified fields are not populated (there is no ingestion-time fallback).
- [dbt run_results file(s)](https://docs.getdbt.com/reference/artifacts/run-results-json) — optional
  - Outcomes of a dbt run, e.g. `dbt test` results and model execution timing.
  - Multiple files and glob patterns are supported. If not provided, test results and model performance are not populated.

The table below summarizes the DataHub entities and aspects produced from each artifact. Several of these outputs have dedicated sections on this page (dataset statistics, query entities, exposures, semantic models, and `meta_mapping`).

| Artifact           | DataHub entity · aspect                                                       | What it captures                                                                                                                |
| ------------------ | ----------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------- |
| `manifest.json`    | Dataset · DatasetProperties                                                   | Name, description, and dbt metadata (materialization, package, file path, unique id, dbt version, adapter) as custom properties |
|                    | Dataset · SubTypes                                                            | dbt resource type: `Model`, `Source`, `Seed`, or `Snapshot`                                                                     |
|                    | Dataset · UpstreamLineage                                                     | Table lineage from `depends_on.nodes`; column-level lineage from parsed SQL when available                                      |
|                    | Dataset · ViewProperties                                                      | Raw and compiled SQL                                                                                                            |
|                    | Dataset · Ownership, GlobalTags, GlossaryTerms, Domains, StructuredProperties | Owners and tags from `meta`/`config`; terms, domains, and structured properties via `meta_mapping`/`column_meta_mapping`        |
|                    | Assertion · AssertionInfo                                                     | Test definitions and parameters                                                                                                 |
|                    | Dashboard                                                                     | dbt exposures (dashboards, notebooks, ML models, applications) with upstream lineage                                            |
|                    | Dataset (Semantic Model)                                                      | dbt semantic models (dbt 1.6+): entities, dimensions, and measures                                                              |
|                    | Query                                                                         | Queries defined in a model's `meta.queries`                                                                                     |
| `catalog.json`     | Dataset · SchemaMetadata                                                      | Column names, types, comments, and descriptions                                                                                 |
|                    | Dataset · DatasetProfile                                                      | Table statistics: row count, size, and column count                                                                             |
| `sources.json`     | Dataset · SchemaMetadata (`lastModified`)                                     | Source freshness timestamp (`max_loaded_at`)                                                                                    |
|                    | Assertion                                                                     | Source freshness checks, as freshness assertions                                                                                |
| `run_results.json` | Assertion · AssertionRunEvent                                                 | Test run status, timing, failure count, and failure messages                                                                    |
|                    | DataProcessInstance                                                           | Model run performance: status, start/end times, and run id (dbt Core only)                                                      |

**Recommended workflow for dbt build and DataHub ingestion:**

```sh
dbt source snapshot-freshness
dbt build
cp target/run_results.json target/run_results_backup.json
dbt docs generate
cp target/run_results_backup.json target/run_results.json

# Run datahub ingestion, pointing at the files in the target/ directory
```

The necessary artifact files will then appear in the `target/` directory of your dbt project.

We also have guides on handling more complex dbt orchestration techniques and multi-project setups below.

:::note Entity is in manifest but missing from catalog

This warning usually appears when the catalog.json file was not generated by a `dbt docs generate` command.
Most other dbt commands generate a partial catalog file, which may impact the completeness of the metadata in ingested into DataHub.

Following the above workflow should ensure that the catalog file is generated correctly.

:::


### Install the Plugin
```shell
pip install 'acryl-datahub[dbt]'
```

### Starter Recipe
Check out the following recipe to get started with ingestion! See [below](#config-details) for full configuration options.


For general pointers on writing and running a recipe, see our [main recipe guide](../../../../metadata-ingestion/README.md#recipes).
```yaml
source:
  type: "dbt"
  config:
    # Coordinates
    # To use this as-is, set the environment variable DBT_PROJECT_ROOT to the root folder of your dbt project
    manifest_path: "${DBT_PROJECT_ROOT}/target/manifest_file.json"
    catalog_path: "${DBT_PROJECT_ROOT}/target/catalog_file.json"
    sources_path: "${DBT_PROJECT_ROOT}/target/sources_file.json" # optional for freshness
    run_results_paths:
      - "${DBT_PROJECT_ROOT}/target/run_results.json" # optional for recording dbt test results after running dbt test
      # Glob patterns are supported for S3, GCS, and local paths, e.g.:
      # - "s3://my-bucket/dbt/run_results/*/run_results.json"
      # - "gs://my-bucket/dbt/run_results/*/run_results.json"
      # - "${DBT_PROJECT_ROOT}/run_results/*/run_results.json"

    # Options
    target_platform: "my_target_platform_id" # e.g. bigquery/postgres/etc.
    # convert_urns_to_lowercase: false  # optional: set to false for case-sensitive platforms like BigQuery to preserve original casing (default: true)

    # For S3 file access:
    # aws_connection:
    #   aws_access_key_id: "${AWS_ACCESS_KEY_ID}"
    #   aws_secret_access_key: "${AWS_SECRET_ACCESS_KEY}"

    # For GCS file access (using HMAC keys):
    # gcs_connection:
    #   credential:
    #     hmac_access_id: "${GCS_HMAC_ACCESS_ID}"
    #     hmac_access_secret: "${GCS_HMAC_ACCESS_SECRET}"

# sink configs

```

### Config Details

                
#### Options


Note that a `.` is used to denote nested fields in the YAML recipe.


<div className='config-table'>

| Field | Description |
|:--- |:--- |
| <div className="path-line"><span className="path-main">manifest_path</span>&nbsp;<abbr title="Required">✅</abbr></div> <div className="type-name-line"><span className="type-name">string</span></div> | Path to dbt manifest JSON. See https://docs.getdbt.com/reference/artifacts/manifest-json. This can be a local file or a URI.  |
| <div className="path-line"><span className="path-main">target_platform</span>&nbsp;<abbr title="Required">✅</abbr></div> <div className="type-name-line"><span className="type-name">string</span></div> | The platform that dbt is loading onto. (e.g. bigquery / redshift / postgres etc.)  |
| <div className="path-line"><span className="path-main">catalog_path</span></div> <div className="type-name-line"><span className="type-name">One of string, null</span></div> | Path to dbt catalog JSON. See https://docs.getdbt.com/reference/artifacts/catalog-json. This file is optional, but highly recommended. Without it, some metadata like column info will be incomplete or missing. This can be a local file or a URI. <div className="default-line default-line-with-docs">Default: <span className="default-value">None</span></div> |
| <div className="path-line"><span className="path-main">column_meta_mapping</span></div> <div className="type-name-line"><span className="type-name">object</span></div> | mapping rules that will be executed against dbt column meta properties. Refer to the section below on dbt meta automated mappings. <div className="default-line default-line-with-docs">Default: <span className="default-value">&#123;&#125;</span></div> |
| <div className="path-line"><span className="path-main">convert_column_urns_to_lowercase</span></div> <div className="type-name-line"><span className="type-name">boolean</span></div> | When enabled, converts column URNs to lowercase to ensure cross-platform compatibility. If `target_platform` is Snowflake, the default is True. <div className="default-line default-line-with-docs">Default: <span className="default-value">False</span></div> |
| <div className="path-line"><span className="path-main">convert_urns_to_lowercase</span></div> <div className="type-name-line"><span className="type-name">boolean</span></div> | Whether to convert dataset urns to lowercase. Default True to match historical dbt behavior. Set to False for case-sensitive platforms like BigQuery if you need to preserve original identifier casing in URNs. <div className="default-line default-line-with-docs">Default: <span className="default-value">True</span></div> |
| <div className="path-line"><span className="path-main">dbt_is_primary_sibling</span></div> <div className="type-name-line"><span className="type-name">boolean</span></div> | Experimental: Controls sibling relationship primary designation between dbt entities and target platform entities. When True (default), dbt entities are primary and target platform entities are secondary. When False, target platform entities are primary and dbt entities are secondary. Uses aspect patches for precise control. Requires DataHub server 1.3.0+. <div className="default-line default-line-with-docs">Default: <span className="default-value">True</span></div> |
| <div className="path-line"><span className="path-main">drop_duplicate_sources</span></div> <div className="type-name-line"><span className="type-name">boolean</span></div> | When enabled, drops sources that have the same name in the target platform as a model. This ensures that lineage is generated reliably, but will lose any documentation associated only with the source. <div className="default-line default-line-with-docs">Default: <span className="default-value">True</span></div> |
| <div className="path-line"><span className="path-main">emit_target_platform_display_name</span></div> <div className="type-name-line"><span className="type-name">boolean</span></div> | Set a display name on target-platform entities that the warehouse connector has not ingested. Those entities have no datasetProperties, so the UI falls back to the urn and shows the full dotted path (instance.database.schema.table) rather than just the table name. Enabling this patches datasetProperties.name with the table name, matching how the warehouse connector's own entities are labelled. Has no effect unless both `target_platform_instance` is set and `emit_target_platform_instance_aspects` is enabled - a warning is logged if set without them. <div className="default-line default-line-with-docs">Default: <span className="default-value">True</span></div> |
| <div className="path-line"><span className="path-main">emit_target_platform_instance_aspects</span></div> <div className="type-name-line"><span className="type-name">boolean</span></div> | When target_platform_instance is set, emit dataPlatformInstance and browsePathsV2 aspects for target-platform sibling entities so they are correctly grouped under their platform instance in browse and filters. Browse paths written by the warehouse connector are never overwritten. <div className="default-line default-line-with-docs">Default: <span className="default-value">True</span></div> |
| <div className="path-line"><span className="path-main">enable_meta_mapping</span></div> <div className="type-name-line"><span className="type-name">boolean</span></div> | When enabled, applies the mappings that are defined through the meta_mapping directives. <div className="default-line default-line-with-docs">Default: <span className="default-value">True</span></div> |
| <div className="path-line"><span className="path-main">enable_owner_extraction</span></div> <div className="type-name-line"><span className="type-name">boolean</span></div> | When enabled, ownership info will be extracted from the dbt meta <div className="default-line default-line-with-docs">Default: <span className="default-value">True</span></div> |
| <div className="path-line"><span className="path-main">enable_query_tag_mapping</span></div> <div className="type-name-line"><span className="type-name">boolean</span></div> | When enabled, applies the mappings that are defined through the `query_tag_mapping` directives. <div className="default-line default-line-with-docs">Default: <span className="default-value">True</span></div> |
| <div className="path-line"><span className="path-main">include_column_lineage</span></div> <div className="type-name-line"><span className="type-name">boolean</span></div> | When enabled, column-level lineage will be extracted from the dbt node definition. Requires `infer_dbt_schemas` to be enabled. If you run into issues where the column name casing does not match up with properly, providing a datahub_api or using the rest sink will improve accuracy. <div className="default-line default-line-with-docs">Default: <span className="default-value">True</span></div> |
| <div className="path-line"><span className="path-main">include_compiled_code</span></div> <div className="type-name-line"><span className="type-name">boolean</span></div> | When enabled, includes the compiled code in the emitted metadata. <div className="default-line default-line-with-docs">Default: <span className="default-value">True</span></div> |
| <div className="path-line"><span className="path-main">include_database_name</span></div> <div className="type-name-line"><span className="type-name">boolean</span></div> | Whether to add database name to the table urn. Set to False to skip it for engines like AWS Athena where it's not required. <div className="default-line default-line-with-docs">Default: <span className="default-value">True</span></div> |
| <div className="path-line"><span className="path-main">include_env_in_assertion_guid</span></div> <div className="type-name-line"><span className="type-name">boolean</span></div> | Prior to version 0.9.4.2, the assertion GUIDs did not include the environment. If you're using multiple dbt ingestion that are only distinguished by env, then you should set this flag to True. <div className="default-line default-line-with-docs">Default: <span className="default-value">False</span></div> |
| <div className="path-line"><span className="path-main">incremental_lineage</span></div> <div className="type-name-line"><span className="type-name">boolean</span></div> | When enabled, emits incremental/patch lineage for non-dbt entities. When disabled, re-states lineage on each run. This would also require enabling 'incremental_lineage' in the counterpart warehouse ingestion (_e.g._ BigQuery, Redshift, etc). <div className="default-line default-line-with-docs">Default: <span className="default-value">True</span></div> |
| <div className="path-line"><span className="path-main">infer_dbt_schemas</span></div> <div className="type-name-line"><span className="type-name">boolean</span></div> | When enabled, schemas will be inferred from the dbt node definition. <div className="default-line default-line-with-docs">Default: <span className="default-value">True</span></div> |
| <div className="path-line"><span className="path-main">max_queries_per_model</span></div> <div className="type-name-line"><span className="type-name">integer</span></div> | Maximum number of Query entities to emit per dbt model. Prevents metadata explosion from malformed manifests. Set to 0 for unlimited. <div className="default-line default-line-with-docs">Default: <span className="default-value">100</span></div> |
| <div className="path-line"><span className="path-main">meta_mapping</span></div> <div className="type-name-line"><span className="type-name">object</span></div> | mapping rules that will be executed against dbt meta properties. Refer to the section below on dbt meta automated mappings. <div className="default-line default-line-with-docs">Default: <span className="default-value">&#123;&#125;</span></div> |
| <div className="path-line"><span className="path-main">only_include_if_in_catalog</span></div> <div className="type-name-line"><span className="type-name">boolean</span></div> | [experimental] If true, only include nodes that are also present in the catalog file. This is useful if you only want to include models that have been built by the associated run. <div className="default-line default-line-with-docs">Default: <span className="default-value">False</span></div> |
| <div className="path-line"><span className="path-main">owner_extraction_pattern</span></div> <div className="type-name-line"><span className="type-name">One of string, null</span></div> | Regex string to extract owner from the dbt node using the `(?P<name>...) syntax` of the [match object](https://docs.python.org/3/library/re.html#match-objects), where the group name must be `owner`. Examples: (1)`r"(?P<owner>(.*)): (\w+) (\w+)"` will extract `jdoe` as the owner from `"jdoe: John Doe"` (2) `r"@(?P<owner>(.*))"` will extract `alice` as the owner from `"@alice"`. <div className="default-line default-line-with-docs">Default: <span className="default-value">None</span></div> |
| <div className="path-line"><span className="path-main">platform_instance</span></div> <div className="type-name-line"><span className="type-name">One of string, null</span></div> | The instance of the platform that all assets produced by this recipe belong to. This should be unique within the platform. See https://docs.datahub.com/docs/platform-instances/ for more details. <div className="default-line default-line-with-docs">Default: <span className="default-value">None</span></div> |
| <div className="path-line"><span className="path-main">prefer_sql_parser_lineage</span></div> <div className="type-name-line"><span className="type-name">boolean</span></div> | Normally we use dbt's metadata to generate table lineage. When enabled, we prefer results from the SQL parser when generating lineage instead. This can be useful when dbt models reference tables directly, instead of using the ref() macro. This requires that `skip_sources_in_lineage` is enabled. <div className="default-line default-line-with-docs">Default: <span className="default-value">False</span></div> |
| <div className="path-line"><span className="path-main">query_tag_mapping</span></div> <div className="type-name-line"><span className="type-name">object</span></div> | mapping rules that will be executed against dbt query_tag meta properties. Refer to the section below on dbt meta automated mappings. <div className="default-line default-line-with-docs">Default: <span className="default-value">&#123;&#125;</span></div> |
| <div className="path-line"><span className="path-main">skip_missing_upstreams_in_lineage</span></div> <div className="type-name-line"><span className="type-name">boolean</span></div> | When enabled, upstream datasets that do not already exist in DataHub are excluded from lineage, preventing dangling graph edges from appearing in the lineage UI. Typically used together with `skip_sources_in_lineage` and `entities_enabled.sources: NO`. Important caveats: (1) if dbt is ingested before its upstream source systems, those lineage edges will be silently omitted until dbt is re-ingested after the upstreams are present; (2) adds one graph.exists() round-trip per unique upstream URN per run (cached within the run); (3) soft-deleted upstream entities are treated as present. Requires a DataHub graph connection. <div className="default-line default-line-with-docs">Default: <span className="default-value">False</span></div> |
| <div className="path-line"><span className="path-main">skip_sources_in_lineage</span></div> <div className="type-name-line"><span className="type-name">boolean</span></div> | [Experimental] When enabled, dbt sources will not be included in the lineage graph. Requires that `entities_enabled.sources` is set to `NO`. This is mainly useful when you have multiple, interdependent dbt projects.  <div className="default-line default-line-with-docs">Default: <span className="default-value">False</span></div> |
| <div className="path-line"><span className="path-main">sources_path</span></div> <div className="type-name-line"><span className="type-name">One of string, null</span></div> | Path to dbt sources JSON. See https://docs.getdbt.com/reference/artifacts/sources-json. If not specified, last-modified fields will not be populated. This can be a local file or a URI. <div className="default-line default-line-with-docs">Default: <span className="default-value">None</span></div> |
| <div className="path-line"><span className="path-main">strip_user_ids_from_email</span></div> <div className="type-name-line"><span className="type-name">boolean</span></div> | Whether or not to strip email id while adding owners using dbt meta actions. <div className="default-line default-line-with-docs">Default: <span className="default-value">False</span></div> |
| <div className="path-line"><span className="path-main">tag_prefix</span></div> <div className="type-name-line"><span className="type-name">string</span></div> | Prefix added to tags during ingestion. <div className="default-line default-line-with-docs">Default: <span className="default-value">dbt:</span></div> |
| <div className="path-line"><span className="path-main">target_platform_instance</span></div> <div className="type-name-line"><span className="type-name">One of string, null</span></div> | The platform instance for the platform that dbt is operating on. Use this if you have multiple instances of the same platform (e.g. redshift) and need to distinguish between them. <div className="default-line default-line-with-docs">Default: <span className="default-value">None</span></div> |
| <div className="path-line"><span className="path-main">test_warnings_are_errors</span></div> <div className="type-name-line"><span className="type-name">boolean</span></div> | When enabled, dbt test warnings will be treated as failures (emitted as ``AssertionResult.type = FAILURE`` with ``severity = LOW``). The default will change to ``true`` in a future release once assertion result consumers can filter by severity; set ``true`` today to adopt the forthcoming behavior. <div className="default-line default-line-with-docs">Default: <span className="default-value">False</span></div> |
| <div className="path-line"><span className="path-main">use_identifiers</span></div> <div className="type-name-line"><span className="type-name">boolean</span></div> | Use model identifier instead of model name if defined (if not, default to model name). <div className="default-line default-line-with-docs">Default: <span className="default-value">False</span></div> |
| <div className="path-line"><span className="path-main">write_semantics</span></div> <div className="type-name-line"><span className="type-name">string</span></div> | Whether the new tags, terms and owners to be added will override the existing ones added only by this source or not. Value for this config can be "PATCH" or "OVERRIDE" <div className="default-line default-line-with-docs">Default: <span className="default-value">PATCH</span></div> |
| <div className="path-line"><span className="path-main">env</span></div> <div className="type-name-line"><span className="type-name">string</span></div> | Environment to use in namespace when constructing URNs. <div className="default-line default-line-with-docs">Default: <span className="default-value">PROD</span></div> |
| <div className="path-line"><span className="path-main">aws_connection</span></div> <div className="type-name-line"><span className="type-name">One of AwsConnectionConfig, null</span></div> | When fetching manifest files from s3, configuration for aws connection details <div className="default-line default-line-with-docs">Default: <span className="default-value">None</span></div> |
| <div className="path-line"><span className="path-prefix">aws_connection.</span><span className="path-main">aws_access_key_id</span></div> <div className="type-name-line"><span className="type-name">One of string, null</span></div> | AWS access key ID. Can be auto-detected, see [the AWS boto3 docs](https://boto3.amazonaws.com/v1/documentation/api/latest/guide/credentials.html) for details. <div className="default-line default-line-with-docs">Default: <span className="default-value">None</span></div> |
| <div className="path-line"><span className="path-prefix">aws_connection.</span><span className="path-main">aws_advanced_config</span></div> <div className="type-name-line"><span className="type-name">object</span></div> | Advanced AWS configuration options. These are passed directly to [botocore.config.Config](https://botocore.amazonaws.com/v1/documentation/api/latest/reference/config.html).  |
| <div className="path-line"><span className="path-prefix">aws_connection.</span><span className="path-main">aws_endpoint_url</span></div> <div className="type-name-line"><span className="type-name">One of string, null</span></div> | The AWS service endpoint. This is normally [constructed automatically](https://boto3.amazonaws.com/v1/documentation/api/latest/reference/core/session.html), but can be overridden here. <div className="default-line default-line-with-docs">Default: <span className="default-value">None</span></div> |
| <div className="path-line"><span className="path-prefix">aws_connection.</span><span className="path-main">aws_profile</span></div> <div className="type-name-line"><span className="type-name">One of string, null</span></div> | The [named profile](https://docs.aws.amazon.com/cli/latest/userguide/cli-configure-profiles.html) to use from AWS credentials. Falls back to default profile if not specified and no access keys provided. Profiles are configured in ~/.aws/credentials or ~/.aws/config. <div className="default-line default-line-with-docs">Default: <span className="default-value">None</span></div> |
| <div className="path-line"><span className="path-prefix">aws_connection.</span><span className="path-main">aws_proxy</span></div> <div className="type-name-line"><span className="type-name">One of string, null</span></div> | A set of proxy configs to use with AWS. See the [botocore.config](https://botocore.amazonaws.com/v1/documentation/api/latest/reference/config.html) docs for details. <div className="default-line default-line-with-docs">Default: <span className="default-value">None</span></div> |
| <div className="path-line"><span className="path-prefix">aws_connection.</span><span className="path-main">aws_region</span></div> <div className="type-name-line"><span className="type-name">One of string, null</span></div> | AWS region code. <div className="default-line default-line-with-docs">Default: <span className="default-value">None</span></div> |
| <div className="path-line"><span className="path-prefix">aws_connection.</span><span className="path-main">aws_retry_mode</span></div> <div className="type-name-line"><span className="type-name">Enum</span></div> | One of: "legacy", "standard", "adaptive" <div className="default-line default-line-with-docs">Default: <span className="default-value">standard</span></div> |
| <div className="path-line"><span className="path-prefix">aws_connection.</span><span className="path-main">aws_retry_num</span></div> <div className="type-name-line"><span className="type-name">integer</span></div> | Number of times to retry failed AWS requests. See the [botocore.retry](https://boto3.amazonaws.com/v1/documentation/api/latest/guide/retries.html) docs for details. <div className="default-line default-line-with-docs">Default: <span className="default-value">5</span></div> |
| <div className="path-line"><span className="path-prefix">aws_connection.</span><span className="path-main">aws_secret_access_key</span></div> <div className="type-name-line"><span className="type-name">One of string(password), null</span></div> | AWS secret access key. Can be auto-detected, see [the AWS boto3 docs](https://boto3.amazonaws.com/v1/documentation/api/latest/guide/credentials.html) for details. <div className="default-line default-line-with-docs">Default: <span className="default-value">None</span></div> |
| <div className="path-line"><span className="path-prefix">aws_connection.</span><span className="path-main">aws_session_token</span></div> <div className="type-name-line"><span className="type-name">One of string(password), null</span></div> | AWS session token. Can be auto-detected, see [the AWS boto3 docs](https://boto3.amazonaws.com/v1/documentation/api/latest/guide/credentials.html) for details. <div className="default-line default-line-with-docs">Default: <span className="default-value">None</span></div> |
| <div className="path-line"><span className="path-prefix">aws_connection.</span><span className="path-main">read_timeout</span></div> <div className="type-name-line"><span className="type-name">number</span></div> | The timeout for reading from the connection (in seconds). <div className="default-line default-line-with-docs">Default: <span className="default-value">60</span></div> |
| <div className="path-line"><span className="path-prefix">aws_connection.</span><span className="path-main">aws_role</span></div> <div className="type-name-line"><span className="type-name">One of string, array, null</span></div> | AWS roles to assume. If using the string format, the role ARN can be specified directly. If using the object format, the role can be specified in the RoleArn field and additional available arguments are the same as [boto3's STS.Client.assume_role](https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/sts.html?highlight=assume_role#STS.Client.assume_role). <div className="default-line default-line-with-docs">Default: <span className="default-value">None</span></div> |
| <div className="path-line"><span className="path-prefix">aws_connection.aws_role.</span><span className="path-main">union</span></div> <div className="type-name-line"><span className="type-name">One of string, AwsAssumeRoleConfig</span></div> |   |
| <div className="path-line"><span className="path-prefix">aws_connection.aws_role.union.</span><span className="path-main">RoleArn</span>&nbsp;<abbr title="Required if union is set">❓</abbr></div> <div className="type-name-line"><span className="type-name">string</span></div> | ARN of the role to assume.  |
| <div className="path-line"><span className="path-prefix">aws_connection.aws_role.union.</span><span className="path-main">ExternalId</span></div> <div className="type-name-line"><span className="type-name">One of string, null</span></div> | External ID to use when assuming the role. <div className="default-line default-line-with-docs">Default: <span className="default-value">None</span></div> |
| <div className="path-line"><span className="path-main">entities_enabled</span></div> <div className="type-name-line"><span className="type-name">DBTEntitiesEnabled</span></div> | Controls which dbt entities are going to be emitted by this source  |
| <div className="path-line"><span className="path-prefix">entities_enabled.</span><span className="path-main">catalog_stats</span></div> <div className="type-name-line"><span className="type-name">Enum</span></div> | One of: "YES", "NO", "ONLY"  |
| <div className="path-line"><span className="path-prefix">entities_enabled.</span><span className="path-main">exposures</span></div> <div className="type-name-line"><span className="type-name">Enum</span></div> | One of: "YES", "NO", "ONLY"  |
| <div className="path-line"><span className="path-prefix">entities_enabled.</span><span className="path-main">model_performance</span></div> <div className="type-name-line"><span className="type-name">Enum</span></div> | One of: "YES", "NO", "ONLY"  |
| <div className="path-line"><span className="path-prefix">entities_enabled.</span><span className="path-main">models</span></div> <div className="type-name-line"><span className="type-name">Enum</span></div> | One of: "YES", "NO", "ONLY"  |
| <div className="path-line"><span className="path-prefix">entities_enabled.</span><span className="path-main">queries</span></div> <div className="type-name-line"><span className="type-name">Enum</span></div> | One of: "YES", "NO", "ONLY"  |
| <div className="path-line"><span className="path-prefix">entities_enabled.</span><span className="path-main">seeds</span></div> <div className="type-name-line"><span className="type-name">Enum</span></div> | One of: "YES", "NO", "ONLY"  |
| <div className="path-line"><span className="path-prefix">entities_enabled.</span><span className="path-main">semantic_models</span></div> <div className="type-name-line"><span className="type-name">Enum</span></div> | One of: "YES", "NO", "ONLY"  |
| <div className="path-line"><span className="path-prefix">entities_enabled.</span><span className="path-main">snapshots</span></div> <div className="type-name-line"><span className="type-name">Enum</span></div> | One of: "YES", "NO", "ONLY"  |
| <div className="path-line"><span className="path-prefix">entities_enabled.</span><span className="path-main">sources</span></div> <div className="type-name-line"><span className="type-name">Enum</span></div> | One of: "YES", "NO", "ONLY"  |
| <div className="path-line"><span className="path-prefix">entities_enabled.</span><span className="path-main">test_definitions</span></div> <div className="type-name-line"><span className="type-name">Enum</span></div> | One of: "YES", "NO", "ONLY"  |
| <div className="path-line"><span className="path-prefix">entities_enabled.</span><span className="path-main">test_results</span></div> <div className="type-name-line"><span className="type-name">Enum</span></div> | One of: "YES", "NO", "ONLY"  |
| <div className="path-line"><span className="path-main">gcs_connection</span></div> <div className="type-name-line"><span className="type-name">One of GCSConnectionConfig, null</span></div> | When fetching manifest files from gs://, GCS connection using HMAC credentials. See https://cloud.google.com/storage/docs/authentication/hmackeys <div className="default-line default-line-with-docs">Default: <span className="default-value">None</span></div> |
| <div className="path-line"><span className="path-prefix">gcs_connection.</span><span className="path-main">credential</span>&nbsp;<abbr title="Required if gcs_connection is set">❓</abbr></div> <div className="type-name-line"><span className="type-name">HMACKey</span></div> |   |
| <div className="path-line"><span className="path-prefix">gcs_connection.credential.</span><span className="path-main">hmac_access_id</span>&nbsp;<abbr title="Required if credential is set">❓</abbr></div> <div className="type-name-line"><span className="type-name">string</span></div> | Access ID  |
| <div className="path-line"><span className="path-prefix">gcs_connection.credential.</span><span className="path-main">hmac_access_secret</span>&nbsp;<abbr title="Required if credential is set">❓</abbr></div> <div className="type-name-line"><span className="type-name">string(password)</span></div> | Secret  |
| <div className="path-line"><span className="path-prefix">gcs_connection.</span><span className="path-main">endpoint_url</span></div> <div className="type-name-line"><span className="type-name">string</span></div> | GCS S3-compatible endpoint URL. Useful for testing with local S3-compatible servers. <div className="default-line default-line-with-docs">Default: <span className="default-value">https://storage.googleapis.com</span></div> |
| <div className="path-line"><span className="path-main">git_info</span></div> <div className="type-name-line"><span className="type-name">One of GitReference, null</span></div> | Reference to your git location to enable easy navigation from DataHub to your dbt files. <div className="default-line default-line-with-docs">Default: <span className="default-value">None</span></div> |
| <div className="path-line"><span className="path-prefix">git_info.</span><span className="path-main">repo</span>&nbsp;<abbr title="Required if git_info is set">❓</abbr></div> <div className="type-name-line"><span className="type-name">string</span></div> | Name of your Git repo e.g. https://github.com/datahub-project/datahub or https://gitlab.com/gitlab-org/gitlab. If organization/repo is provided, we assume it is a GitHub repo.  |
| <div className="path-line"><span className="path-prefix">git_info.</span><span className="path-main">branch</span></div> <div className="type-name-line"><span className="type-name">string</span></div> | Branch on which your files live by default. Typically main or master. This can also be a commit hash. <div className="default-line default-line-with-docs">Default: <span className="default-value">main</span></div> |
| <div className="path-line"><span className="path-prefix">git_info.</span><span className="path-main">url_subdir</span></div> <div className="type-name-line"><span className="type-name">One of string, null</span></div> | Prefix to prepend when generating URLs for files - useful when files are in a subdirectory. Only affects URL generation, not git operations. <div className="default-line default-line-with-docs">Default: <span className="default-value">None</span></div> |
| <div className="path-line"><span className="path-prefix">git_info.</span><span className="path-main">url_template</span></div> <div className="type-name-line"><span className="type-name">One of string, null</span></div> | Template for generating a URL to a file in the repo e.g. '{repo_url}/blob/{branch}/{file_path}'. We can infer this for GitHub and GitLab repos, and it is otherwise required.It supports the following variables: {repo_url}, {branch}, {file_path} <div className="default-line default-line-with-docs">Default: <span className="default-value">None</span></div> |
| <div className="path-line"><span className="path-main">materialized_node_pattern</span></div> <div className="type-name-line"><span className="type-name">MaterializedNodePatternConfig</span></div> | Configuration for filtering materialized nodes based on their physical location  |
| <div className="path-line"><span className="path-prefix">materialized_node_pattern.</span><span className="path-main">database_pattern</span></div> <div className="type-name-line"><span className="type-name">AllowDenyPattern</span></div> | A class to store allow deny regexes. <br />  <br /> Patterns are matched against the start of the string only, not the entire <br /> string - a pattern does not need to match to the end to be considered a match. <br /> For example, the pattern "prod" matches "prod", "prod_east", and "production". <br /> To require an exact match, anchor your pattern explicitly, e.g. "^prod$".  |
| <div className="path-line"><span className="path-prefix">materialized_node_pattern.database_pattern.</span><span className="path-main">ignoreCase</span></div> <div className="type-name-line"><span className="type-name">One of boolean, null</span></div> | Whether to ignore case sensitivity during pattern matching. <div className="default-line default-line-with-docs">Default: <span className="default-value">True</span></div> |
| <div className="path-line"><span className="path-prefix">materialized_node_pattern.</span><span className="path-main">schema_pattern</span></div> <div className="type-name-line"><span className="type-name">AllowDenyPattern</span></div> | A class to store allow deny regexes. <br />  <br /> Patterns are matched against the start of the string only, not the entire <br /> string - a pattern does not need to match to the end to be considered a match. <br /> For example, the pattern "prod" matches "prod", "prod_east", and "production". <br /> To require an exact match, anchor your pattern explicitly, e.g. "^prod$".  |
| <div className="path-line"><span className="path-prefix">materialized_node_pattern.schema_pattern.</span><span className="path-main">ignoreCase</span></div> <div className="type-name-line"><span className="type-name">One of boolean, null</span></div> | Whether to ignore case sensitivity during pattern matching. <div className="default-line default-line-with-docs">Default: <span className="default-value">True</span></div> |
| <div className="path-line"><span className="path-prefix">materialized_node_pattern.</span><span className="path-main">table_pattern</span></div> <div className="type-name-line"><span className="type-name">AllowDenyPattern</span></div> | A class to store allow deny regexes. <br />  <br /> Patterns are matched against the start of the string only, not the entire <br /> string - a pattern does not need to match to the end to be considered a match. <br /> For example, the pattern "prod" matches "prod", "prod_east", and "production". <br /> To require an exact match, anchor your pattern explicitly, e.g. "^prod$".  |
| <div className="path-line"><span className="path-prefix">materialized_node_pattern.table_pattern.</span><span className="path-main">ignoreCase</span></div> <div className="type-name-line"><span className="type-name">One of boolean, null</span></div> | Whether to ignore case sensitivity during pattern matching. <div className="default-line default-line-with-docs">Default: <span className="default-value">True</span></div> |
| <div className="path-line"><span className="path-main">node_name_pattern</span></div> <div className="type-name-line"><span className="type-name">AllowDenyPattern</span></div> | A class to store allow deny regexes. <br />  <br /> Patterns are matched against the start of the string only, not the entire <br /> string - a pattern does not need to match to the end to be considered a match. <br /> For example, the pattern "prod" matches "prod", "prod_east", and "production". <br /> To require an exact match, anchor your pattern explicitly, e.g. "^prod$".  |
| <div className="path-line"><span className="path-prefix">node_name_pattern.</span><span className="path-main">ignoreCase</span></div> <div className="type-name-line"><span className="type-name">One of boolean, null</span></div> | Whether to ignore case sensitivity during pattern matching. <div className="default-line default-line-with-docs">Default: <span className="default-value">True</span></div> |
| <div className="path-line"><span className="path-main">run_results_paths</span></div> <div className="type-name-line"><span className="type-name">array</span></div> | Path to output of dbt test run as run_results files in JSON format. If not specified, test execution results and model performance metadata will not be populated in DataHub. If invoking dbt multiple times, you can provide paths to multiple run result files. Glob patterns are supported for S3, GCS, and local paths (e.g. 's3://bucket/results/*/run_results.json', 'gs://bucket/results/*/run_results.json', or '/path/to/results/*/run_results.json'). See https://docs.getdbt.com/reference/artifacts/run-results-json. <div className="default-line default-line-with-docs">Default: <span className="default-value">&#91;&#93;</span></div> |
| <div className="path-line"><span className="path-prefix">run_results_paths.</span><span className="path-main">string</span></div> <div className="type-name-line"><span className="type-name">string</span></div> |   |
| <div className="path-line"><span className="path-main">stateful_ingestion</span></div> <div className="type-name-line"><span className="type-name">One of StatefulStaleMetadataRemovalConfig, null</span></div> | DBT Stateful Ingestion Config. <div className="default-line default-line-with-docs">Default: <span className="default-value">None</span></div> |
| <div className="path-line"><span className="path-prefix">stateful_ingestion.</span><span className="path-main">enabled</span></div> <div className="type-name-line"><span className="type-name">boolean</span></div> | Whether or not to enable stateful ingest. Default: True if a pipeline_name is set and either a datahub-rest sink or `datahub_api` is specified, otherwise False <div className="default-line default-line-with-docs">Default: <span className="default-value">False</span></div> |
| <div className="path-line"><span className="path-prefix">stateful_ingestion.</span><span className="path-main">fail_safe_threshold</span></div> <div className="type-name-line"><span className="type-name">number</span></div> | Prevents large amount of soft deletes & the state from committing from accidental changes to the source configuration if the relative change percent in entities compared to the previous state is above the 'fail_safe_threshold'. <div className="default-line default-line-with-docs">Default: <span className="default-value">75.0</span></div> |
| <div className="path-line"><span className="path-prefix">stateful_ingestion.</span><span className="path-main">remove_stale_metadata</span></div> <div className="type-name-line"><span className="type-name">boolean</span></div> | Soft-deletes the entities present in the last successful run but missing in the current run with stateful_ingestion enabled. <div className="default-line default-line-with-docs">Default: <span className="default-value">True</span></div> |

</div>




#### Schema


The [JSONSchema](https://json-schema.org/) for this configuration is inlined below.


```javascript
{
  "$defs": {
    "AllowDenyPattern": {
      "additionalProperties": false,
      "description": "A class to store allow deny regexes.\n\nPatterns are matched against the start of the string only, not the entire\nstring - a pattern does not need to match to the end to be considered a match.\nFor example, the pattern \"prod\" matches \"prod\", \"prod_east\", and \"production\".\nTo require an exact match, anchor your pattern explicitly, e.g. \"^prod$\".",
      "properties": {
        "allow": {
          "default": [
            ".*"
          ],
          "description": "List of regex patterns to include in ingestion. Patterns match from the start of the string only, not the entire string - anchor with '^...$' for an exact match, e.g. '^prod$'.",
          "items": {
            "type": "string"
          },
          "title": "Allow",
          "type": "array"
        },
        "deny": {
          "default": [],
          "description": "List of regex patterns to exclude from ingestion. Patterns match from the start of the string only, not the entire string - anchor with '^...$' for an exact match, e.g. '^prod$'.",
          "items": {
            "type": "string"
          },
          "title": "Deny",
          "type": "array"
        },
        "ignoreCase": {
          "anyOf": [
            {
              "type": "boolean"
            },
            {
              "type": "null"
            }
          ],
          "default": true,
          "description": "Whether to ignore case sensitivity during pattern matching.",
          "title": "Ignorecase"
        }
      },
      "title": "AllowDenyPattern",
      "type": "object"
    },
    "AwsAssumeRoleConfig": {
      "additionalProperties": true,
      "properties": {
        "RoleArn": {
          "description": "ARN of the role to assume.",
          "title": "Rolearn",
          "type": "string"
        },
        "ExternalId": {
          "anyOf": [
            {
              "type": "string"
            },
            {
              "type": "null"
            }
          ],
          "default": null,
          "description": "External ID to use when assuming the role.",
          "title": "Externalid"
        }
      },
      "required": [
        "RoleArn"
      ],
      "title": "AwsAssumeRoleConfig",
      "type": "object"
    },
    "AwsConnectionConfig": {
      "additionalProperties": false,
      "description": "Common AWS credentials config.\n\nCurrently used by:\n    - Glue source\n    - SageMaker source\n    - dbt source",
      "properties": {
        "aws_access_key_id": {
          "anyOf": [
            {
              "type": "string"
            },
            {
              "type": "null"
            }
          ],
          "default": null,
          "description": "AWS access key ID. Can be auto-detected, see [the AWS boto3 docs](https://boto3.amazonaws.com/v1/documentation/api/latest/guide/credentials.html) for details.",
          "title": "Aws Access Key Id"
        },
        "aws_secret_access_key": {
          "anyOf": [
            {
              "format": "password",
              "type": "string",
              "writeOnly": true
            },
            {
              "type": "null"
            }
          ],
          "default": null,
          "description": "AWS secret access key. Can be auto-detected, see [the AWS boto3 docs](https://boto3.amazonaws.com/v1/documentation/api/latest/guide/credentials.html) for details.",
          "title": "Aws Secret Access Key"
        },
        "aws_session_token": {
          "anyOf": [
            {
              "format": "password",
              "type": "string",
              "writeOnly": true
            },
            {
              "type": "null"
            }
          ],
          "default": null,
          "description": "AWS session token. Can be auto-detected, see [the AWS boto3 docs](https://boto3.amazonaws.com/v1/documentation/api/latest/guide/credentials.html) for details.",
          "title": "Aws Session Token"
        },
        "aws_role": {
          "anyOf": [
            {
              "type": "string"
            },
            {
              "items": {
                "anyOf": [
                  {
                    "type": "string"
                  },
                  {
                    "$ref": "#/$defs/AwsAssumeRoleConfig"
                  }
                ]
              },
              "type": "array"
            },
            {
              "type": "null"
            }
          ],
          "default": null,
          "description": "AWS roles to assume. If using the string format, the role ARN can be specified directly. If using the object format, the role can be specified in the RoleArn field and additional available arguments are the same as [boto3's STS.Client.assume_role](https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/sts.html?highlight=assume_role#STS.Client.assume_role).",
          "title": "Aws Role"
        },
        "aws_profile": {
          "anyOf": [
            {
              "type": "string"
            },
            {
              "type": "null"
            }
          ],
          "default": null,
          "description": "The [named profile](https://docs.aws.amazon.com/cli/latest/userguide/cli-configure-profiles.html) to use from AWS credentials. Falls back to default profile if not specified and no access keys provided. Profiles are configured in ~/.aws/credentials or ~/.aws/config.",
          "title": "Aws Profile"
        },
        "aws_region": {
          "anyOf": [
            {
              "type": "string"
            },
            {
              "type": "null"
            }
          ],
          "default": null,
          "description": "AWS region code.",
          "title": "Aws Region"
        },
        "aws_endpoint_url": {
          "anyOf": [
            {
              "type": "string"
            },
            {
              "type": "null"
            }
          ],
          "default": null,
          "description": "The AWS service endpoint. This is normally [constructed automatically](https://boto3.amazonaws.com/v1/documentation/api/latest/reference/core/session.html), but can be overridden here.",
          "title": "Aws Endpoint Url"
        },
        "aws_proxy": {
          "anyOf": [
            {
              "additionalProperties": {
                "type": "string"
              },
              "type": "object"
            },
            {
              "type": "null"
            }
          ],
          "default": null,
          "description": "A set of proxy configs to use with AWS. See the [botocore.config](https://botocore.amazonaws.com/v1/documentation/api/latest/reference/config.html) docs for details.",
          "title": "Aws Proxy"
        },
        "aws_retry_num": {
          "default": 5,
          "description": "Number of times to retry failed AWS requests. See the [botocore.retry](https://boto3.amazonaws.com/v1/documentation/api/latest/guide/retries.html) docs for details.",
          "title": "Aws Retry Num",
          "type": "integer"
        },
        "aws_retry_mode": {
          "default": "standard",
          "description": "Retry mode to use for failed AWS requests. See the [botocore.retry](https://boto3.amazonaws.com/v1/documentation/api/latest/guide/retries.html) docs for details.",
          "enum": [
            "legacy",
            "standard",
            "adaptive"
          ],
          "title": "Aws Retry Mode",
          "type": "string"
        },
        "read_timeout": {
          "default": 60,
          "description": "The timeout for reading from the connection (in seconds).",
          "title": "Read Timeout",
          "type": "number"
        },
        "aws_advanced_config": {
          "additionalProperties": true,
          "description": "Advanced AWS configuration options. These are passed directly to [botocore.config.Config](https://botocore.amazonaws.com/v1/documentation/api/latest/reference/config.html).",
          "title": "Aws Advanced Config",
          "type": "object"
        }
      },
      "title": "AwsConnectionConfig",
      "type": "object"
    },
    "DBTEntitiesEnabled": {
      "additionalProperties": false,
      "description": "Controls which dbt entities are going to be emitted by this source",
      "properties": {
        "models": {
          "$ref": "#/$defs/EmitDirective",
          "default": "YES",
          "description": "Emit metadata for dbt models when set to Yes or Only"
        },
        "sources": {
          "$ref": "#/$defs/EmitDirective",
          "default": "YES",
          "description": "Emit metadata for dbt sources when set to Yes or Only"
        },
        "seeds": {
          "$ref": "#/$defs/EmitDirective",
          "default": "YES",
          "description": "Emit metadata for dbt seeds when set to Yes or Only"
        },
        "snapshots": {
          "$ref": "#/$defs/EmitDirective",
          "default": "YES",
          "description": "Emit metadata for dbt snapshots when set to Yes or Only"
        },
        "test_definitions": {
          "$ref": "#/$defs/EmitDirective",
          "default": "YES",
          "description": "Emit metadata for test definitions when enabled when set to Yes or Only"
        },
        "test_results": {
          "$ref": "#/$defs/EmitDirective",
          "default": "YES",
          "description": "Emit metadata for test results when set to Yes or Only"
        },
        "model_performance": {
          "$ref": "#/$defs/EmitDirective",
          "default": "YES",
          "description": "Emit model performance metadata when set to Yes or Only. Only supported with dbt core."
        },
        "exposures": {
          "$ref": "#/$defs/EmitDirective",
          "default": "YES",
          "description": "Emit metadata for dbt exposures when set to Yes or Only. Exposures represent downstream consumers like dashboards, notebooks, or applications."
        },
        "semantic_models": {
          "$ref": "#/$defs/EmitDirective",
          "default": "YES",
          "description": "Emit metadata for dbt semantic models when set to Yes or Only. Semantic models define entities, dimensions, and measures for the dbt semantic layer (dbt 1.6+)."
        },
        "queries": {
          "$ref": "#/$defs/EmitDirective",
          "default": "YES",
          "description": "Emit Query entities from meta.queries field when set to Yes or Only."
        },
        "catalog_stats": {
          "$ref": "#/$defs/EmitDirective",
          "default": "YES",
          "description": "Emit DatasetProfile aspects with row counts and size from catalog.json stats when set to Yes. Requires catalog.json to be generated by `dbt docs generate`."
        }
      },
      "title": "DBTEntitiesEnabled",
      "type": "object"
    },
    "EmitDirective": {
      "description": "A holder for directives for emission for specific types of entities",
      "enum": [
        "YES",
        "NO",
        "ONLY"
      ],
      "title": "EmitDirective",
      "type": "string"
    },
    "GCSConnectionConfig": {
      "additionalProperties": false,
      "description": "GCS connection using HMAC keys, accessed via the S3-compatible XML API.",
      "properties": {
        "credential": {
          "$ref": "#/$defs/HMACKey",
          "description": "GCS HMAC credentials. See https://cloud.google.com/storage/docs/authentication/hmackeys"
        },
        "endpoint_url": {
          "default": "https://storage.googleapis.com",
          "description": "GCS S3-compatible endpoint URL. Useful for testing with local S3-compatible servers.",
          "title": "Endpoint Url",
          "type": "string"
        }
      },
      "required": [
        "credential"
      ],
      "title": "GCSConnectionConfig",
      "type": "object"
    },
    "GitReference": {
      "additionalProperties": false,
      "description": "Reference to a hosted Git repository. Used to generate \"view source\" links.",
      "properties": {
        "repo": {
          "description": "Name of your Git repo e.g. https://github.com/datahub-project/datahub or https://gitlab.com/gitlab-org/gitlab. If organization/repo is provided, we assume it is a GitHub repo.",
          "title": "Repo",
          "type": "string"
        },
        "branch": {
          "default": "main",
          "description": "Branch on which your files live by default. Typically main or master. This can also be a commit hash.",
          "title": "Branch",
          "type": "string"
        },
        "url_subdir": {
          "anyOf": [
            {
              "type": "string"
            },
            {
              "type": "null"
            }
          ],
          "default": null,
          "description": "Prefix to prepend when generating URLs for files - useful when files are in a subdirectory. Only affects URL generation, not git operations.",
          "title": "Url Subdir"
        },
        "url_template": {
          "anyOf": [
            {
              "type": "string"
            },
            {
              "type": "null"
            }
          ],
          "default": null,
          "description": "Template for generating a URL to a file in the repo e.g. '{repo_url}/blob/{branch}/{file_path}'. We can infer this for GitHub and GitLab repos, and it is otherwise required.It supports the following variables: {repo_url}, {branch}, {file_path}",
          "title": "Url Template"
        }
      },
      "required": [
        "repo"
      ],
      "title": "GitReference",
      "type": "object"
    },
    "HMACKey": {
      "additionalProperties": false,
      "properties": {
        "hmac_access_id": {
          "description": "Access ID",
          "title": "Hmac Access Id",
          "type": "string"
        },
        "hmac_access_secret": {
          "description": "Secret",
          "format": "password",
          "title": "Hmac Access Secret",
          "type": "string",
          "writeOnly": true
        }
      },
      "required": [
        "hmac_access_id",
        "hmac_access_secret"
      ],
      "title": "HMACKey",
      "type": "object"
    },
    "MaterializedNodePatternConfig": {
      "additionalProperties": false,
      "description": "Configuration for filtering materialized nodes based on their physical location",
      "properties": {
        "database_pattern": {
          "$ref": "#/$defs/AllowDenyPattern",
          "default": {
            "allow": [
              ".*"
            ],
            "deny": [],
            "ignoreCase": true
          },
          "description": "Regex patterns for database names to filter materialized nodes."
        },
        "schema_pattern": {
          "$ref": "#/$defs/AllowDenyPattern",
          "default": {
            "allow": [
              ".*"
            ],
            "deny": [],
            "ignoreCase": true
          },
          "description": "Regex patterns for schema names in format '{database}.{schema}' to filter materialized nodes."
        },
        "table_pattern": {
          "$ref": "#/$defs/AllowDenyPattern",
          "default": {
            "allow": [
              ".*"
            ],
            "deny": [],
            "ignoreCase": true
          },
          "description": "Regex patterns for table/view names in format '{database}.{schema}.{table}' to filter materialized nodes."
        }
      },
      "title": "MaterializedNodePatternConfig",
      "type": "object"
    },
    "StatefulStaleMetadataRemovalConfig": {
      "additionalProperties": false,
      "description": "Base specialized config for Stateful Ingestion with stale metadata removal capability.",
      "properties": {
        "enabled": {
          "default": false,
          "description": "Whether or not to enable stateful ingest. Default: True if a pipeline_name is set and either a datahub-rest sink or `datahub_api` is specified, otherwise False",
          "title": "Enabled",
          "type": "boolean"
        },
        "remove_stale_metadata": {
          "default": true,
          "description": "Soft-deletes the entities present in the last successful run but missing in the current run with stateful_ingestion enabled.",
          "title": "Remove Stale Metadata",
          "type": "boolean"
        },
        "fail_safe_threshold": {
          "default": 75.0,
          "description": "Prevents large amount of soft deletes & the state from committing from accidental changes to the source configuration if the relative change percent in entities compared to the previous state is above the 'fail_safe_threshold'.",
          "maximum": 100.0,
          "minimum": 0.0,
          "title": "Fail Safe Threshold",
          "type": "number"
        }
      },
      "title": "StatefulStaleMetadataRemovalConfig",
      "type": "object"
    }
  },
  "additionalProperties": false,
  "properties": {
    "convert_urns_to_lowercase": {
      "default": true,
      "description": "Whether to convert dataset urns to lowercase. Default True to match historical dbt behavior. Set to False for case-sensitive platforms like BigQuery if you need to preserve original identifier casing in URNs.",
      "title": "Convert Urns To Lowercase",
      "type": "boolean"
    },
    "incremental_lineage": {
      "default": true,
      "description": "When enabled, emits incremental/patch lineage for non-dbt entities. When disabled, re-states lineage on each run. This would also require enabling 'incremental_lineage' in the counterpart warehouse ingestion (_e.g._ BigQuery, Redshift, etc).",
      "title": "Incremental Lineage",
      "type": "boolean"
    },
    "env": {
      "default": "PROD",
      "description": "Environment to use in namespace when constructing URNs.",
      "title": "Env",
      "type": "string"
    },
    "platform_instance": {
      "anyOf": [
        {
          "type": "string"
        },
        {
          "type": "null"
        }
      ],
      "default": null,
      "description": "The instance of the platform that all assets produced by this recipe belong to. This should be unique within the platform. See https://docs.datahub.com/docs/platform-instances/ for more details.",
      "title": "Platform Instance"
    },
    "stateful_ingestion": {
      "anyOf": [
        {
          "$ref": "#/$defs/StatefulStaleMetadataRemovalConfig"
        },
        {
          "type": "null"
        }
      ],
      "default": null,
      "description": "DBT Stateful Ingestion Config."
    },
    "target_platform": {
      "description": "The platform that dbt is loading onto. (e.g. bigquery / redshift / postgres etc.)",
      "title": "Target Platform",
      "type": "string"
    },
    "target_platform_instance": {
      "anyOf": [
        {
          "type": "string"
        },
        {
          "type": "null"
        }
      ],
      "default": null,
      "description": "The platform instance for the platform that dbt is operating on. Use this if you have multiple instances of the same platform (e.g. redshift) and need to distinguish between them.",
      "title": "Target Platform Instance"
    },
    "emit_target_platform_instance_aspects": {
      "default": true,
      "description": "When target_platform_instance is set, emit dataPlatformInstance and browsePathsV2 aspects for target-platform sibling entities so they are correctly grouped under their platform instance in browse and filters. Browse paths written by the warehouse connector are never overwritten.",
      "title": "Emit Target Platform Instance Aspects",
      "type": "boolean"
    },
    "emit_target_platform_display_name": {
      "default": true,
      "description": "Set a display name on target-platform entities that the warehouse connector has not ingested. Those entities have no datasetProperties, so the UI falls back to the urn and shows the full dotted path (instance.database.schema.table) rather than just the table name. Enabling this patches datasetProperties.name with the table name, matching how the warehouse connector's own entities are labelled. Has no effect unless both `target_platform_instance` is set and `emit_target_platform_instance_aspects` is enabled - a warning is logged if set without them.",
      "title": "Emit Target Platform Display Name",
      "type": "boolean"
    },
    "use_identifiers": {
      "default": false,
      "description": "Use model identifier instead of model name if defined (if not, default to model name).",
      "title": "Use Identifiers",
      "type": "boolean"
    },
    "entities_enabled": {
      "$ref": "#/$defs/DBTEntitiesEnabled",
      "default": {
        "models": "YES",
        "sources": "YES",
        "seeds": "YES",
        "snapshots": "YES",
        "test_definitions": "YES",
        "test_results": "YES",
        "model_performance": "YES",
        "exposures": "YES",
        "semantic_models": "YES",
        "queries": "YES",
        "catalog_stats": "YES"
      },
      "description": "Controls for enabling / disabling metadata emission for different dbt entities (models, test definitions, test results, etc.)"
    },
    "prefer_sql_parser_lineage": {
      "default": false,
      "description": "Normally we use dbt's metadata to generate table lineage. When enabled, we prefer results from the SQL parser when generating lineage instead. This can be useful when dbt models reference tables directly, instead of using the ref() macro. This requires that `skip_sources_in_lineage` is enabled.",
      "title": "Prefer Sql Parser Lineage",
      "type": "boolean"
    },
    "skip_sources_in_lineage": {
      "default": false,
      "description": "[Experimental] When enabled, dbt sources will not be included in the lineage graph. Requires that `entities_enabled.sources` is set to `NO`. This is mainly useful when you have multiple, interdependent dbt projects. ",
      "title": "Skip Sources In Lineage",
      "type": "boolean"
    },
    "skip_missing_upstreams_in_lineage": {
      "default": false,
      "description": "When enabled, upstream datasets that do not already exist in DataHub are excluded from lineage, preventing dangling graph edges from appearing in the lineage UI. Typically used together with `skip_sources_in_lineage` and `entities_enabled.sources: NO`. Important caveats: (1) if dbt is ingested before its upstream source systems, those lineage edges will be silently omitted until dbt is re-ingested after the upstreams are present; (2) adds one graph.exists() round-trip per unique upstream URN per run (cached within the run); (3) soft-deleted upstream entities are treated as present. Requires a DataHub graph connection.",
      "title": "Skip Missing Upstreams In Lineage",
      "type": "boolean"
    },
    "tag_prefix": {
      "default": "dbt:",
      "description": "Prefix added to tags during ingestion.",
      "title": "Tag Prefix",
      "type": "string"
    },
    "node_name_pattern": {
      "$ref": "#/$defs/AllowDenyPattern",
      "default": {
        "allow": [
          ".*"
        ],
        "deny": [],
        "ignoreCase": true
      },
      "description": "regex patterns for dbt model names to filter in ingestion."
    },
    "materialized_node_pattern": {
      "$ref": "#/$defs/MaterializedNodePatternConfig",
      "default": {
        "database_pattern": {
          "allow": [
            ".*"
          ],
          "deny": [],
          "ignoreCase": true
        },
        "schema_pattern": {
          "allow": [
            ".*"
          ],
          "deny": [],
          "ignoreCase": true
        },
        "table_pattern": {
          "allow": [
            ".*"
          ],
          "deny": [],
          "ignoreCase": true
        }
      },
      "description": "Advanced filtering for materialized nodes based on their physical database location. Provides fine-grained control over database.schema.table patterns for catalog consistency."
    },
    "meta_mapping": {
      "additionalProperties": true,
      "default": {},
      "description": "mapping rules that will be executed against dbt meta properties. Refer to the section below on dbt meta automated mappings.",
      "title": "Meta Mapping",
      "type": "object"
    },
    "column_meta_mapping": {
      "additionalProperties": true,
      "default": {},
      "description": "mapping rules that will be executed against dbt column meta properties. Refer to the section below on dbt meta automated mappings.",
      "title": "Column Meta Mapping",
      "type": "object"
    },
    "enable_meta_mapping": {
      "default": true,
      "description": "When enabled, applies the mappings that are defined through the meta_mapping directives.",
      "title": "Enable Meta Mapping",
      "type": "boolean"
    },
    "query_tag_mapping": {
      "additionalProperties": true,
      "default": {},
      "description": "mapping rules that will be executed against dbt query_tag meta properties. Refer to the section below on dbt meta automated mappings.",
      "title": "Query Tag Mapping",
      "type": "object"
    },
    "enable_query_tag_mapping": {
      "default": true,
      "description": "When enabled, applies the mappings that are defined through the `query_tag_mapping` directives.",
      "title": "Enable Query Tag Mapping",
      "type": "boolean"
    },
    "write_semantics": {
      "default": "PATCH",
      "description": "Whether the new tags, terms and owners to be added will override the existing ones added only by this source or not. Value for this config can be \"PATCH\" or \"OVERRIDE\"",
      "title": "Write Semantics",
      "type": "string"
    },
    "strip_user_ids_from_email": {
      "default": false,
      "description": "Whether or not to strip email id while adding owners using dbt meta actions.",
      "title": "Strip User Ids From Email",
      "type": "boolean"
    },
    "enable_owner_extraction": {
      "default": true,
      "description": "When enabled, ownership info will be extracted from the dbt meta",
      "title": "Enable Owner Extraction",
      "type": "boolean"
    },
    "max_queries_per_model": {
      "default": 100,
      "description": "Maximum number of Query entities to emit per dbt model. Prevents metadata explosion from malformed manifests. Set to 0 for unlimited.",
      "minimum": 0,
      "title": "Max Queries Per Model",
      "type": "integer"
    },
    "owner_extraction_pattern": {
      "anyOf": [
        {
          "type": "string"
        },
        {
          "type": "null"
        }
      ],
      "default": null,
      "description": "Regex string to extract owner from the dbt node using the `(?P<name>...) syntax` of the [match object](https://docs.python.org/3/library/re.html#match-objects), where the group name must be `owner`. Examples: (1)`r\"(?P<owner>(.*)): (\\w+) (\\w+)\"` will extract `jdoe` as the owner from `\"jdoe: John Doe\"` (2) `r\"@(?P<owner>(.*))\"` will extract `alice` as the owner from `\"@alice\"`.",
      "title": "Owner Extraction Pattern"
    },
    "include_env_in_assertion_guid": {
      "default": false,
      "description": "Prior to version 0.9.4.2, the assertion GUIDs did not include the environment. If you're using multiple dbt ingestion that are only distinguished by env, then you should set this flag to True.",
      "title": "Include Env In Assertion Guid",
      "type": "boolean"
    },
    "convert_column_urns_to_lowercase": {
      "default": false,
      "description": "When enabled, converts column URNs to lowercase to ensure cross-platform compatibility. If `target_platform` is Snowflake, the default is True.",
      "title": "Convert Column Urns To Lowercase",
      "type": "boolean"
    },
    "test_warnings_are_errors": {
      "default": false,
      "description": "When enabled, dbt test warnings will be treated as failures (emitted as ``AssertionResult.type = FAILURE`` with ``severity = LOW``). The default will change to ``true`` in a future release once assertion result consumers can filter by severity; set ``true`` today to adopt the forthcoming behavior.",
      "title": "Test Warnings Are Errors",
      "type": "boolean"
    },
    "infer_dbt_schemas": {
      "default": true,
      "description": "When enabled, schemas will be inferred from the dbt node definition.",
      "title": "Infer Dbt Schemas",
      "type": "boolean"
    },
    "include_column_lineage": {
      "default": true,
      "description": "When enabled, column-level lineage will be extracted from the dbt node definition. Requires `infer_dbt_schemas` to be enabled. If you run into issues where the column name casing does not match up with properly, providing a datahub_api or using the rest sink will improve accuracy.",
      "title": "Include Column Lineage",
      "type": "boolean"
    },
    "include_compiled_code": {
      "default": true,
      "description": "When enabled, includes the compiled code in the emitted metadata.",
      "title": "Include Compiled Code",
      "type": "boolean"
    },
    "include_database_name": {
      "default": true,
      "description": "Whether to add database name to the table urn. Set to False to skip it for engines like AWS Athena where it's not required.",
      "title": "Include Database Name",
      "type": "boolean"
    },
    "dbt_is_primary_sibling": {
      "default": true,
      "description": "Experimental: Controls sibling relationship primary designation between dbt entities and target platform entities. When True (default), dbt entities are primary and target platform entities are secondary. When False, target platform entities are primary and dbt entities are secondary. Uses aspect patches for precise control. Requires DataHub server 1.3.0+.",
      "title": "Dbt Is Primary Sibling",
      "type": "boolean"
    },
    "drop_duplicate_sources": {
      "default": true,
      "description": "When enabled, drops sources that have the same name in the target platform as a model. This ensures that lineage is generated reliably, but will lose any documentation associated only with the source.",
      "title": "Drop Duplicate Sources",
      "type": "boolean"
    },
    "manifest_path": {
      "description": "Path to dbt manifest JSON. See https://docs.getdbt.com/reference/artifacts/manifest-json. This can be a local file or a URI.",
      "title": "Manifest Path",
      "type": "string"
    },
    "catalog_path": {
      "anyOf": [
        {
          "type": "string"
        },
        {
          "type": "null"
        }
      ],
      "default": null,
      "description": "Path to dbt catalog JSON. See https://docs.getdbt.com/reference/artifacts/catalog-json. This file is optional, but highly recommended. Without it, some metadata like column info will be incomplete or missing. This can be a local file or a URI.",
      "title": "Catalog Path"
    },
    "sources_path": {
      "anyOf": [
        {
          "type": "string"
        },
        {
          "type": "null"
        }
      ],
      "default": null,
      "description": "Path to dbt sources JSON. See https://docs.getdbt.com/reference/artifacts/sources-json. If not specified, last-modified fields will not be populated. This can be a local file or a URI.",
      "title": "Sources Path"
    },
    "run_results_paths": {
      "default": [],
      "description": "Path to output of dbt test run as run_results files in JSON format. If not specified, test execution results and model performance metadata will not be populated in DataHub. If invoking dbt multiple times, you can provide paths to multiple run result files. Glob patterns are supported for S3, GCS, and local paths (e.g. 's3://bucket/results/*/run_results.json', 'gs://bucket/results/*/run_results.json', or '/path/to/results/*/run_results.json'). See https://docs.getdbt.com/reference/artifacts/run-results-json.",
      "items": {
        "type": "string"
      },
      "title": "Run Results Paths",
      "type": "array"
    },
    "only_include_if_in_catalog": {
      "default": false,
      "description": "[experimental] If true, only include nodes that are also present in the catalog file. This is useful if you only want to include models that have been built by the associated run.",
      "title": "Only Include If In Catalog",
      "type": "boolean"
    },
    "aws_connection": {
      "anyOf": [
        {
          "$ref": "#/$defs/AwsConnectionConfig"
        },
        {
          "type": "null"
        }
      ],
      "default": null,
      "description": "When fetching manifest files from s3, configuration for aws connection details"
    },
    "gcs_connection": {
      "anyOf": [
        {
          "$ref": "#/$defs/GCSConnectionConfig"
        },
        {
          "type": "null"
        }
      ],
      "default": null,
      "description": "When fetching manifest files from gs://, GCS connection using HMAC credentials. See https://cloud.google.com/storage/docs/authentication/hmackeys"
    },
    "git_info": {
      "anyOf": [
        {
          "$ref": "#/$defs/GitReference"
        },
        {
          "type": "null"
        }
      ],
      "default": null,
      "description": "Reference to your git location to enable easy navigation from DataHub to your dbt files."
    }
  },
  "required": [
    "target_platform",
    "manifest_path"
  ],
  "title": "DBTCoreConfig",
  "type": "object"
}
```





### Capabilities

Use the **Important Capabilities** table above as the source of truth for supported features and whether additional configuration is required.

#### dbt meta automated mappings

dbt allows authors to define meta properties for datasets. Checkout this link to know more - [dbt meta](https://docs.getdbt.com/reference/resource-configs/meta). Our dbt source allows users to define
actions such as add a tag, term or owner. For example if a dbt model has a meta config `"has_pii": True`, we can define an action
that evaluates if the property is set to true and add, lets say, a `pii` tag.
To leverage this feature we require users to define mappings as part of the recipe. The following section describes how you can build these mappings. Listed below is a `meta_mapping` and `column_meta_mapping` section that among other things, looks for keys like `business_owner` and adds owners that are listed there.

```yaml
meta_mapping:
  business_owner:
    match: ".*"
    operation: "add_owner"
    config:
      owner_type: user
      owner_category: BUSINESS_OWNER
  has_pii:
    match: True
    operation: "add_tag"
    config:
      tag: "has_pii_test"
  int_property:
    match: 1
    operation: "add_tag"
    config:
      tag: "int_meta_property"
  double_property:
    match: 2.5
    operation: "add_term"
    config:
      term: "double_meta_property"
  data_governance.team_owner:
    match: "Finance"
    operation: "add_term"
    config:
      term: "Finance_test"
  terms_list:
    match: ".*"
    operation: "add_terms"
    config:
      separator: ","
  documentation_link:
    match: "(?:https?)?\:\/\/\w*[^#]*"
    operation: "add_doc_link"
    config:
      link: {{ $match }}
      description: "Documentation Link"
  business_domain:
    match: ".*"
    operation: "add_domain"
    config:
      domain: "{{ $match }}"
  data_load_frequency:
    match: ".*"
    operation: "add_structured_property"
    config:
      structured_property_urn: "urn:li:structuredProperty:io.acme.data_load_frequency"
column_meta_mapping:
  terms_list:
    match: ".*"
    operation: "add_terms"
    config:
      separator: ","
  is_sensitive:
    match: True
    operation: "add_tag"
    config:
      tag: "sensitive"
  gdpr.pii:
    match: true
    operation: "add_tag"
    config:
      tag: "pii"
  classification:
    match: ".*"
    operation: "add_structured_property"
    config:
      structured_property_urn: "urn:li:structuredProperty:io.acme.classification"
```

We support the following operations:

1. add_tag - Requires `tag` property in config.
2. add_term - Requires `term` property in config.
3. add_terms - Accepts an optional `separator` property in config.
4. add_owner - Requires `owner_type` property in config which can be either `user` or `group`. Optionally accepts the `owner_category` config property which can be set to either a [custom ownership type](../../../../docs/ownership/ownership-types.md) urn like `urn:li:ownershipType:architect` or one of `['TECHNICAL_OWNER', 'BUSINESS_OWNER', 'DATA_STEWARD', 'DATAOWNER'` (defaults to `DATAOWNER`).

   - The `owner_type` property will be ignored if the owner is a fully qualified urn.
   - You can use commas to specify multiple owners - e.g. `business_owner: "jane,john,urn:li:corpGroup:data-team"`.

5. add_doc_link - Requires `link` and `description` properties in config. Upon ingestion run, this will overwrite current links in the institutional knowledge section with this new link. The anchor text is defined here in the meta_mappings as `description`.
6. add_domain - Adds the dataset to a DataHub Domain. The `domain` config value can be a short ID (e.g. `Marketing`) or a fully-qualified URN (`urn:li:domain:Marketing`). Supports `{{ $match }}` substitution.
7. add_structured_property - Assigns a value to a [DataHub Structured Property](../../../../docs/features/feature-guides/properties/overview.md). Required config: `structured_property_urn` (full URN or qualified name). Optional config: `value` (literal or `{{ $match }}` template; defaults to the raw meta value, preserving numeric types) and `value_type` (`string` or `number`). Multiple rules targeting the same property URN have their values aggregated into a single assignment. The structured property itself must be defined in DataHub before ingestion runs — this operation only assigns values, it does not create the property definition.

Note:

1. The dbt `meta_mapping` config works at the model level, while the `column_meta_mapping` config works at the column level. The `add_owner` operation is not supported at the column level. The `add_structured_property` operation is supported at both levels — at the column level it produces a `structuredProperties` aspect attached to each matching `schemaField` URN.
2. For string meta properties we support regex matching.
3. **List support**: YAML lists are now supported in meta properties. Each item in the list that matches the regex pattern will be processed.

With regex matching, you can also use the matched value to customize how you populate the tag, term or owner fields. Here are a few advanced examples:

##### Data Tier - Bronze, Silver, Gold

If your meta section looks like this:

```yaml
meta:
  data_tier: Bronze # chosen from [Bronze,Gold,Silver]
```

and you wanted to attach a glossary term like `urn:li:glossaryTerm:Bronze` for all the models that have this value in the meta section attached to them, the following meta_mapping section would achieve that outcome:

```yaml
meta_mapping:
  data_tier:
    match: "Bronze|Silver|Gold"
    operation: "add_term"
    config:
      term: "{{ $match }}"
```

to match any data_tier of Bronze, Silver or Gold and maps it to a glossary term with the same name.

##### Case Numbers - create tags

If your meta section looks like this:

```yaml
meta:
  case: PLT-4678 # internal Case Number
```

and you want to generate tags that look like `case_4678` from this, you can use the following meta_mapping section:

```yaml
meta_mapping:
  case:
    match: "PLT-(.*)"
    operation: "add_tag"
    config:
      tag: "case_{{ $match }}"
```

##### Nested meta properties

If your meta section has nested properties and looks like this:

```yaml
meta:
  data_governance:
    team_owner: "Finance"
```

and you want attach term Finance_test in case of data_governance.team_owner is set to Finance, you can use the following meta_mapping section:

```yaml
meta_mapping:
  data_governance.team_owner:
    match: "Finance"
    operation: "add_term"
    config:
      term: "Finance_test"
```

Note: nested meta properties mapping is supported also for column_meta_mapping

##### Stripping out leading @ sign

You can also match specific groups within the value to extract subsets of the matched value. e.g. if you have a meta section that looks like this:

```yaml
meta:
  owner: "@finance-team"
  business_owner: "@janet"
```

and you want to mark the finance-team as a group that owns the dataset (skipping the leading @ sign), while marking janet as an individual user (again, skipping the leading @ sign) that owns the dataset, you can use the following meta-mapping section.

```yaml
meta_mapping:
  owner:
    match: "^@(.*)"
    operation: "add_owner"
    config:
      owner_type: group
  business_owner:
    match: "^@(?P<owner>(.*))"
    operation: "add_owner"
    config:
      owner_type: user
      owner_category: BUSINESS_OWNER
```

In the examples above, we show two ways of writing the matching regexes. In the first one, `^@(.*)` the first matching group (a.k.a. match.group(1)) is automatically inferred. In the second example, `^@(?P<owner>(.*))`, we use a named matching group (called owner, since we are matching an owner) to capture the string we want to provide to the ownership urn.

##### Working with Lists

YAML lists are fully supported in dbt meta properties. Each item in the list is evaluated against the match pattern, and only matching items are processed.

```yaml
meta:
  owners:
    - alice@company.com
    - bob@company.com
    - contractor@external.com
```

```yaml
meta_mapping:
  owners:
    match: ".*@company.com"
    operation: "add_owner"
    config:
      owner_type: user
```

This will add `alice@company.com` and `bob@company.com` as owners (matching `.*@company.com`) but skip `contractor@external.com` (doesn't match the pattern).

#### dbt query_tag automated mappings

This works similarly as the dbt meta mapping but for the query tags

We support the below actions -

1. add_tag - Requires `tag` property in config.

The below example set as global tag the query tag `tag` key's value.

```json
"query_tag_mapping":
{
   "tag":
      "match": ".*"
      "operation": "add_tag"
      "config":
        "tag": "{{ $match }}"
}
```

#### Integrating with dbt test

To integrate with dbt tests, the `dbt` source needs access to the `run_results.json` file generated after a `dbt test` or `dbt build` execution. Typically, this is written to the `target` directory. A common pattern you can follow is:

1. Run `dbt build`
2. Copy the `target/run_results.json` file to a separate location. This is important, because otherwise subsequent `dbt` commands will overwrite the run results.
3. Run `dbt docs generate` to generate the `manifest.json` and `catalog.json` files
4. The dbt source makes use of the manifest, catalog, and run results file, and hence will need to be moved to a location accessible to the `dbt` source (e.g. s3 or local file system). In the ingestion recipe, the `run_results_paths` config must be set to the location of the `run_results.json` file from the `dbt build` or `dbt test` run.

The connector will produce the following things:

- Assertion definitions that are attached to the dataset (or datasets)
- Results from running the tests attached to the timeline of the dataset

##### How dbt test statuses map to DataHub assertion results

Each dbt test result is translated into a DataHub `AssertionResult` with a `type` (SUCCESS, FAILURE, or ERROR) and, on failures, an optional `severity` (LOW, MEDIUM, HIGH).

DataHub distinguishes **the test produced a verdict** (SUCCESS / FAILURE) from **the test could not run** (ERROR). This lets you separate data-quality regressions from infrastructure or SQL-compile problems downstream.

The mapping for regular dbt tests:

| dbt status      | Meaning                                                                             | `AssertionResult.type` | `AssertionResult.severity` |
| --------------- | ----------------------------------------------------------------------------------- | ---------------------- | -------------------------- |
| `pass`          | Test ran, no failing rows                                                           | `SUCCESS`              | —                          |
| `warn`          | Test ran, found failing rows, configured with `severity: warn`                      | see below              | `LOW` (if emitted)         |
| `fail`          | Test ran, found failing rows, configured with `severity: error` (the default)       | `FAILURE`              | `HIGH`                     |
| `error`         | Test invocation failed to compile/start (SQL error, missing ref, permissions, etc.) | `ERROR`                | —                          |
| `runtime error` | Test started running and then the warehouse raised an exception                     | `ERROR`                | —                          |

The `warn` row depends on the `test_warnings_are_errors` config:

- `test_warnings_are_errors: false` (default): `warn` → `SUCCESS`. Honors dbt's author-declared "soft concern" semantics.
- `test_warnings_are_errors: true`: `warn` → `FAILURE` with `severity: LOW`. Stricter — every failing test row counts as a failure, but soft ones are tagged for downstream filtering.

:::note Default will change in a future release

`test_warnings_are_errors` is planned to default to `true` once assertion result consumers (UI, saved searches, alerting) can filter by severity. At that point, `warn` will always emit as `FAILURE` with `severity: LOW`, and the flag itself will be deprecated. Set `test_warnings_are_errors: true` today to adopt the forthcoming behavior.

:::

For dbt source freshness checks, the semantics of `error` differ: `error` means the `error_after` threshold was exceeded, i.e. the check ran and the source is considered stale. The freshness mapping:

| dbt freshness status | Meaning                              | `AssertionResult.type`               | `AssertionResult.severity` |
| -------------------- | ------------------------------------ | ------------------------------------ | -------------------------- |
| `pass`               | Source is fresh                      | `SUCCESS`                            | —                          |
| `warn`               | `warn_after` threshold exceeded      | see `test_warnings_are_errors` above | `LOW` (if emitted)         |
| `error`              | `error_after` threshold exceeded     | `FAILURE`                            | `HIGH`                     |
| `runtime error`      | Freshness check itself failed to run | `ERROR`                              | —                          |

:::note Missing test results?

The most common reason for missing test results is that the `run_results.json` with the test result information is getting overwritten by a subsequent `dbt` command. We recommend copying the `run_results.json` file before running other `dbt` commands.

```sh
dbt source snapshot-freshness
dbt build
cp target/run_results.json target/run_results_backup.json
dbt docs generate
cp target/run_results_backup.json target/run_results.json
```

:::

#### Glob patterns for run_results_paths

If your dbt setup produces many `run_results.json` files (e.g. one per Airflow DAG task or retry), you can use glob patterns instead of listing every file explicitly. This works for both S3 URIs and local paths.

```yaml
source:
  type: dbt
  config:
    manifest_path: "s3://my-bucket/dbt/target/manifest.json"
    catalog_path: "s3://my-bucket/dbt/target/catalog.json"
    target_platform: postgres
    run_results_paths:
      - "s3://my-bucket/dbt/run_results/*/*/*.json"
    aws_connection: {}
```

```yaml
# Local paths also support glob patterns
source:
  type: dbt
  config:
    manifest_path: /dbt/target/manifest.json
    catalog_path: /dbt/target/catalog.json
    target_platform: postgres
    run_results_paths:
      - "/dbt/run_results/*/run_results.json"
```

Supported wildcard characters: `*` (any characters within a path segment), `?` (single character), `[...]` (character set). If a glob pattern matches zero files, a warning is emitted in the ingestion report. The expanded file list is also recorded in the report for debugging.

:::warning Avoid globbing over historical run result files
Glob patterns that match timestamped or versioned files (e.g. `run_results/2024-01-01/run_results.json`) will cause DataHub to re-process old test failures on every ingestion cycle, triggering repeated alerts. Instead, write results to a **stable path that is overwritten on each run** and glob across jobs, not history:

```yaml
# ✅ Glob across tasks, stable path overwritten each run
- "s3://my-bucket/run_results_latest/my_dag/*/run_results.json"

# ❌ Glob across timestamps — re-processes historical failures every cycle
- "s3://my-bucket/run_results/*/2024-*/run_results.json"
```

:::

:::note S3 IAM permissions for glob patterns
When using glob patterns with S3 paths, the IAM role or user must have **`s3:ListBucket`** permission on the bucket in addition to `s3:GetObject`. Without `s3:ListBucket`, the glob expansion will fail with an `AccessDenied` error. Explicit (non-glob) S3 paths only require `s3:GetObject`.
:::

##### View of dbt tests for a dataset

![test view](https://raw.githubusercontent.com/datahub-project/static-assets/main/imgs/dbt-tests-view.png)

##### Viewing the SQL for a dbt test

![test logic view](https://raw.githubusercontent.com/datahub-project/static-assets/main/imgs/dbt-test-logic-view.png)

##### Viewing timeline for a failed dbt test

![test view](https://raw.githubusercontent.com/datahub-project/static-assets/main/imgs/dbt-tests-failure-view.png)

##### Separating test result emission from other metadata emission

You can segregate emission of test results from the emission of other dbt metadata using the `entities_enabled` config flag.
The following recipe shows you how to emit only test results.

```yaml
source:
  type: dbt
  config:
    manifest_path: _path_to_manifest_json
    catalog_path: _path_to_catalog_json
    run_results_paths:
      - _path_to_run_results_json
    target_platform: postgres
    entities_enabled:
      test_results: Only
```

Similarly, the following recipe shows you how to emit everything (i.e. models, sources, seeds, test definitions) but not test results:

```yaml
source:
  type: dbt
  config:
    manifest_path: _path_to_manifest_json
    catalog_path: _path_to_catalog_json
    run_results_paths:
      - _path_to_run_results_json
    target_platform: postgres
    entities_enabled:
      test_results: No
```

:::note Tests not showing up in the Assertion UI?

Double check you are using the same `job_id` for your `test_results: Only` and `test_results: No` recipes. Otherwise you might end up reporting `tests_results` for tests that haven't had their `test_definitions` ingested yet. This can lead to orphaned assertions that do not show up under your dataset.

If you choose not to share a `job_id`, you should adjust your recipe to also ingest the `test_definitions`

```yaml
entities_enabled:
  models: No
  sources: No
  seeds: No
  snapshots: No
  test_definitions: Yes
  test_results: Yes
```

:::

#### Multiple dbt projects

In more complex dbt setups, you may have multiple dbt projects, where models from one project are used as sources in another project.
DataHub supports this setup natively.

Each dbt project should have its own dbt ingestion recipe, and the `platform_instance` field in the recipe should be set to the dbt project name.

For example, if you have two dbt projects `analytics` and `data_mart`, you would have two ingestion recipes.
If you have models in the `data_mart` project that are used as sources in the `analytics` project, the lineage will be automatically captured.

```yaml
# Analytics dbt project
source:
  type: dbt
  config:
    platform_instance: analytics
    target_platform: postgres
    manifest_path: analytics/target/manifest.json
    catalog_path: analytics/target/catalog.json
    # ... other configs
```

```yaml
# Data Mart dbt project
source:
  type: dbt
  config:
    platform_instance: data_mart
    target_platform: postgres
    manifest_path: data_mart/target/manifest.json
    catalog_path: data_mart/target/catalog.json
    # ... other configs
```

If you have models that have tons of sources from other projects listed in the "Composed Of" section, it may also make sense to hide sources.

#### Reducing "composed of" sprawl by hiding sources

When many dbt projects use a single table as a source, the "Composed Of" relationships can become very large and difficult to navigate
and extra source nodes can clutter the lineage graph.

This is particularly useful for multi-project setups, but can be useful in single-project setups as well.

The benefit is that your entire dbt estate becomes much easier to navigate, and the borders between projects less noticeable.
The downside is that we will not pick up any documentation or meta mappings applied to dbt sources.

To enable this, set `entities_enabled.sources: No` and `skip_sources_in_lineage: true` in your dbt source config:

```yaml
source:
  type: dbt
  config:
    platform_instance: analytics
    target_platform: postgres
    manifest_path: analytics/target/manifest.json
    catalog_path: analytics/target/catalog.json
    # ... other configs
    entities_enabled:
      sources: No
    skip_sources_in_lineage: true
```

[Experimental] It's also possible to use `skip_sources_in_lineage: true` without disabling sources entirely. If you do this, sources will not participate in the lineage graph - they'll have upstreams but no downstreams. However, they will still contribute to docs, tags, etc to the warehouse entity.

#### Semantic Views

DataHub can ingest dbt models that have been materialized as `semantic_view` objects, a pattern used to define a semantic layer directly in warehouses like Snowflake.

##### What are Materialized Semantic Views?

A materialized [semantic view](https://docs.snowflake.com/en/user-guide/views-semantic/overview) is a dbt model (a `.sql` file) that uses the `materialized='semantic_view'` configuration via the [dbt_semantic_view package](https://github.com/Snowflake-Labs/dbt_semantic_view). This creates a `SEMANTIC VIEW` object in Snowflake, containing a rich set of metadata including dimensions and metrics.

When you define a dbt model as a semantic view:

```sql
-- models/sales_analytics.sql
{{ config(
    materialized='semantic_view'
) }}

TABLES (
    OrdersTable AS {{ source('coffee_shop_source', 'ORDERS') }}
)
DIMENSIONS (
    OrdersTable.CUSTOMER_ID AS CUSTOMER_ID
)
METRICS (
    OrdersTable.GROSS_REVENUE AS SUM(ORDER_TOTAL)
)
```

DataHub will:

1. Create a dataset with the subtype `Semantic View`.
2. Create sibling relationships to the underlying Snowflake `SEMANTIC VIEW` object.
3. Extract column-level lineage from the semantic view's DDL.

##### Configuration

Semantic views are dbt models with `materialized='semantic_view'`. They are emitted by default along with other models when `entities_enabled.models: Yes` (the default).

##### How Semantic Views Appear in DataHub

- **Subtype**: Datasets are tagged with the subtype `Semantic View`.
- **Lineage**: Upstream lineage to the source dbt model is created, with column-level lineage where available.

##### Column-Level Lineage for Snowflake Semantic Views

For dbt models materialized as semantic views in Snowflake, DataHub can extract column-level lineage from the compiled DDL. This requires:

1. Using Snowflake as the `target_platform`.
2. Having the `compiled_code` for the model available in the dbt manifest or dbt Cloud API.

If these conditions are not met, warnings will appear in the ingestion report:

| Condition               | Warning                                 |
| ----------------------- | --------------------------------------- |
| Non-Snowflake adapter   | `Semantic View CLL Unsupported Adapter` |
| Missing `compiled_code` | `Semantic View Missing compiled_code`   |
| Empty CLL results       | `Semantic View CLL Empty`               |
| Parsing failure         | `Semantic View CLL Parsing Failed`      |

> **Note: Limitations**
>
> Column-level lineage is currently only supported for Snowflake semantic views, as it relies on parsing the Snowflake-specific DDL.

#### Exposures

DataHub supports ingesting [dbt exposures](https://docs.getdbt.com/docs/build/exposures) - downstream consumers of your dbt models such as dashboards, notebooks, ML models, and applications.

##### What are dbt Exposures?

Exposures define how your dbt models are used downstream. They help you understand the full picture of your data ecosystem by documenting:

- **Dashboards** - BI tools like Looker, Tableau, or Metabase
- **Notebooks** - Jupyter notebooks or other analysis tools
- **ML Models** - Machine learning pipelines consuming your data
- **Applications** - Apps or services that use your data
- **Analysis** - Ad-hoc analysis or reports

##### How Exposures Map to DataHub

dbt exposures are ingested as **Dashboard** entities in DataHub, with the exposure type preserved as a subtype:

| dbt Exposure Type | DataHub SubType |
| ----------------- | --------------- |
| `dashboard`       | Dashboard       |
| `notebook`        | Notebook        |
| `ml`              | ML Model        |
| `application`     | Application     |
| `analysis`        | Analysis        |

##### Configuration

Exposures are enabled by default. You can control their emission using the `entities_enabled.exposures` config:

```yaml
source:
  type: dbt
  config:
    manifest_path: _path_to_manifest_json
    catalog_path: _path_to_catalog_json
    target_platform: postgres
    entities_enabled:
      exposures: Yes # Default - emit exposures
      # exposures: No  # Disable exposure ingestion
      # exposures: Only  # Only emit exposures, skip other entities
```

##### Lineage

Exposures automatically create lineage relationships to their upstream dbt models. The `depends_on` field in your exposure definition determines which models appear as upstreams:

```yaml
# models/exposures.yml
exposures:
  - name: weekly_metrics_dashboard
    type: dashboard
    owner:
      email: analytics@company.com
    depends_on:
      - ref('orders')
      - ref('customers')
    url: https://bi.company.com/dashboards/weekly-metrics
```

This creates lineage: `orders` → `weekly_metrics_dashboard` and `customers` → `weekly_metrics_dashboard`.

##### Owner Resolution

Owner resolution for exposures is **the same** as for other dbt assets (models, sources, etc.). The `enable_owner_extraction` config applies to all of them: when `true` (default), ownership is extracted; when `false`, no ownership aspect is emitted for any dbt asset, including exposures.

For exposures, the owner is read from the exposure definition:

- **`owner.email`** (recommended): used as the user URN (e.g., `analytics@company.com` → `urn:li:corpuser:analytics@company.com`)
- **`owner.name`** (fallback): converted to lowercase with underscores (e.g., `"John Doe"` → `urn:li:corpuser:john_doe`)

The same configs apply as elsewhere: `strip_user_ids_from_email` strips the domain from email-based owners when set.

:::note
When only `owner.name` is provided (without email), the generated URN may not match existing users in DataHub. We recommend providing `owner.email` for accurate user matching.
:::

##### Example Output

An exposure like:

```yaml
exposures:
  - name: weekly_metrics_dashboard
    type: dashboard
    description: Weekly business metrics for leadership
    owner:
      email: analytics@company.com
    maturity: high
    url: https://bi.company.com/dashboards/123
    depends_on:
      - ref('orders')
```

Will create a Dashboard entity in DataHub with:

- **Title**: `weekly_metrics_dashboard`
- **Description**: `Weekly business metrics for leadership`
- **SubType**: `Dashboard`, `Notebook`, `ML Model`, `Application`, `Analysis`
- **Owner**: `urn:li:corpuser:analytics@company.com`
- **External URL**: `https://bi.company.com/dashboards/123`
- **Upstream Lineage**: Link to the `orders` dbt model
- **Custom Properties**: `exposure_type`, `maturity`, `dbt_unique_id`

### Limitations

Module behavior is constrained by source APIs, permissions, and metadata exposed by the platform. Refer to capability notes for unsupported or conditional features.

### Troubleshooting

If ingestion fails, validate credentials, permissions, connectivity, and scope filters first. Then review ingestion logs for source-specific errors and adjust configuration accordingly.


### Code Coordinates
- Class Name: `datahub.ingestion.source.dbt.dbt_core.DBTCoreSource`
- Browse on [GitHub](https://github.com/datahub-project/datahub/blob/master/metadata-ingestion/src/datahub/ingestion/source/dbt/dbt_core.py)



## Module `dbt-cloud`
![GA](https://img.shields.io/badge/support%20status-GA-brightgreen)


### Important Capabilities
| Capability | Status | Notes |
| ---------- | ------ | ----- |
| Column-level Lineage | ✅ | Enabled by default, configure using `include_column_lineage`. |
| [Detect Deleted Entities](../../../../metadata-ingestion/docs/dev_guides/stateful.md#stale-entity-removal) | ✅ | Enabled by default via stateful ingestion. |
| Table-Level Lineage | ✅ | Enabled by default. |
| Test Connection | ✅ | Enabled by default. |

### Overview

The `dbt-cloud` module ingests metadata from Dbt into DataHub. It is intended for production ingestion workflows and module-specific capabilities are documented below.

### Prerequisites

Before running ingestion, ensure network connectivity to the source, valid authentication credentials, and read permissions for metadata APIs required by this module.

#### Setup

Extracts dbt metadata from dbt Cloud APIs.

Create a [service account token](https://docs.getdbt.com/docs/dbt-cloud-apis/service-tokens) with "Metadata Only" permission (read-only).

##### Operating Modes

The dbt Cloud source supports two modes of operation:

##### 1. Explicit Mode (Default)

Specify a single dbt Cloud job to ingest metadata from. The job must have "Generate docs on run" enabled and should process all/most models (otherwise multiple job ingestion may be required).

To get the required IDs, go to the job details page (this is the one with the "Run History" table), and look at the URL.
It should look something like this: https://cloud.getdbt.com/next/deploy/107298/projects/175705/jobs/148094.
In this example, the account ID is 107298, the project ID is 175705, and the job ID is 148094.

##### 2. Auto-Discovery Mode

Automatically discovers and ingests metadata from all eligible jobs in a dbt Cloud project. This mode:

- Discovers all jobs in the specified project's **production environment only**
- Filters to jobs with **"Generate docs on run" enabled** (`generate_docs=True`)
- Always uses the **latest run** for each job (ignores `run_id` configuration)
- Supports optional regex-based filtering to include/exclude specific job IDs
- Ingests metadata from multiple jobs in a single run

**When to use auto-discovery:**

- You have multiple dbt Cloud jobs in a project and want to ingest all of them
- You want to automatically pick up new jobs without updating configuration

**Requirements:**

- Jobs must be in the production environment
- Jobs must have "Generate docs on run" enabled


### Install the Plugin
```shell
pip install 'acryl-datahub[dbt-cloud]'
```

### Starter Recipe
Check out the following recipe to get started with ingestion! See [below](#config-details) for full configuration options.


For general pointers on writing and running a recipe, see our [main recipe guide](../../../../metadata-ingestion/README.md#recipes).
```yaml
source:
  type: "dbt-cloud"
  config:
    token: ${DBT_CLOUD_TOKEN}

    # In the URL https://cloud.getdbt.com/next/deploy/107298/projects/175705/jobs/148094,
    # 107298 is the account_id, 175705 is the project_id, and 148094 is the job_id

    account_id: "${DBT_ACCOUNT_ID}" # set to your dbt cloud account id
    project_id: "${DBT_PROJECT_ID}" # set to your dbt cloud project id

    # Mode 1: Explicit Mode (specify a single job)
    job_id: "${DBT_JOB_ID}" # set to your dbt cloud job id
    run_id: # optional: set to a specific dbt cloud run id. Defaults to the latest run

    # Mode 2: Auto-Discovery Mode (automatically discover all eligible jobs)
    # Uncomment the section below to enable auto-discovery
    # Note: When auto_discovery is enabled, job_id can be omitted (will be ignored if provided)
    # and run_id is ignored (always uses the latest run)
    # auto_discovery:
    #   enabled: true
    #   job_id_pattern: # optional
    #     allow:
    #       - ".*"  # regex pattern to include jobs (default: include all)
    #     # deny:
    #     #   - "test.*"  # optional: regex pattern to exclude specific jobs

    target_platform: "${TARGET_PLATFORM_ID}" # e.g. bigquery/postgres/snowflake/etc.
    # convert_urns_to_lowercase: false  # optional: set to false for case-sensitive platforms like BigQuery to preserve original casing (default: true)

# sink configs

```

### Config Details

                
#### Options


Note that a `.` is used to denote nested fields in the YAML recipe.


<div className='config-table'>

| Field | Description |
|:--- |:--- |
| <div className="path-line"><span className="path-main">account_id</span>&nbsp;<abbr title="Required">✅</abbr></div> <div className="type-name-line"><span className="type-name">integer</span></div> | The DBT Cloud account ID to use.  |
| <div className="path-line"><span className="path-main">project_id</span>&nbsp;<abbr title="Required">✅</abbr></div> <div className="type-name-line"><span className="type-name">integer</span></div> | The dbt Cloud project ID to use.  |
| <div className="path-line"><span className="path-main">target_platform</span>&nbsp;<abbr title="Required">✅</abbr></div> <div className="type-name-line"><span className="type-name">string</span></div> | The platform that dbt is loading onto. (e.g. bigquery / redshift / postgres etc.)  |
| <div className="path-line"><span className="path-main">token</span>&nbsp;<abbr title="Required">✅</abbr></div> <div className="type-name-line"><span className="type-name">string(password)</span></div> | The API token to use to authenticate with DBT Cloud.  |
| <div className="path-line"><span className="path-main">access_url</span></div> <div className="type-name-line"><span className="type-name">string</span></div> | The base URL of the dbt Cloud instance to use. This should be the URL you use to access the dbt Cloud UI. It should include the scheme (http/https) and not include a trailing slash. See the access url for your dbt Cloud region here: https://docs.getdbt.com/docs/cloud/about-cloud/regions-ip-addresses <div className="default-line default-line-with-docs">Default: <span className="default-value">https://cloud.getdbt.com</span></div> |
| <div className="path-line"><span className="path-main">column_meta_mapping</span></div> <div className="type-name-line"><span className="type-name">object</span></div> | mapping rules that will be executed against dbt column meta properties. Refer to the section below on dbt meta automated mappings. <div className="default-line default-line-with-docs">Default: <span className="default-value">&#123;&#125;</span></div> |
| <div className="path-line"><span className="path-main">convert_column_urns_to_lowercase</span></div> <div className="type-name-line"><span className="type-name">boolean</span></div> | When enabled, converts column URNs to lowercase to ensure cross-platform compatibility. If `target_platform` is Snowflake, the default is True. <div className="default-line default-line-with-docs">Default: <span className="default-value">False</span></div> |
| <div className="path-line"><span className="path-main">convert_urns_to_lowercase</span></div> <div className="type-name-line"><span className="type-name">boolean</span></div> | Whether to convert dataset urns to lowercase. Default True to match historical dbt behavior. Set to False for case-sensitive platforms like BigQuery if you need to preserve original identifier casing in URNs. <div className="default-line default-line-with-docs">Default: <span className="default-value">True</span></div> |
| <div className="path-line"><span className="path-main">dbt_is_primary_sibling</span></div> <div className="type-name-line"><span className="type-name">boolean</span></div> | Experimental: Controls sibling relationship primary designation between dbt entities and target platform entities. When True (default), dbt entities are primary and target platform entities are secondary. When False, target platform entities are primary and dbt entities are secondary. Uses aspect patches for precise control. Requires DataHub server 1.3.0+. <div className="default-line default-line-with-docs">Default: <span className="default-value">True</span></div> |
| <div className="path-line"><span className="path-main">drop_duplicate_sources</span></div> <div className="type-name-line"><span className="type-name">boolean</span></div> | When enabled, drops sources that have the same name in the target platform as a model. This ensures that lineage is generated reliably, but will lose any documentation associated only with the source. <div className="default-line default-line-with-docs">Default: <span className="default-value">True</span></div> |
| <div className="path-line"><span className="path-main">emit_target_platform_display_name</span></div> <div className="type-name-line"><span className="type-name">boolean</span></div> | Set a display name on target-platform entities that the warehouse connector has not ingested. Those entities have no datasetProperties, so the UI falls back to the urn and shows the full dotted path (instance.database.schema.table) rather than just the table name. Enabling this patches datasetProperties.name with the table name, matching how the warehouse connector's own entities are labelled. Has no effect unless both `target_platform_instance` is set and `emit_target_platform_instance_aspects` is enabled - a warning is logged if set without them. <div className="default-line default-line-with-docs">Default: <span className="default-value">True</span></div> |
| <div className="path-line"><span className="path-main">emit_target_platform_instance_aspects</span></div> <div className="type-name-line"><span className="type-name">boolean</span></div> | When target_platform_instance is set, emit dataPlatformInstance and browsePathsV2 aspects for target-platform sibling entities so they are correctly grouped under their platform instance in browse and filters. Browse paths written by the warehouse connector are never overwritten. <div className="default-line default-line-with-docs">Default: <span className="default-value">True</span></div> |
| <div className="path-line"><span className="path-main">enable_meta_mapping</span></div> <div className="type-name-line"><span className="type-name">boolean</span></div> | When enabled, applies the mappings that are defined through the meta_mapping directives. <div className="default-line default-line-with-docs">Default: <span className="default-value">True</span></div> |
| <div className="path-line"><span className="path-main">enable_owner_extraction</span></div> <div className="type-name-line"><span className="type-name">boolean</span></div> | When enabled, ownership info will be extracted from the dbt meta <div className="default-line default-line-with-docs">Default: <span className="default-value">True</span></div> |
| <div className="path-line"><span className="path-main">enable_query_tag_mapping</span></div> <div className="type-name-line"><span className="type-name">boolean</span></div> | When enabled, applies the mappings that are defined through the `query_tag_mapping` directives. <div className="default-line default-line-with-docs">Default: <span className="default-value">True</span></div> |
| <div className="path-line"><span className="path-main">external_url_mode</span></div> <div className="type-name-line"><span className="type-name">Enum</span></div> | One of: "explore", "ide" <div className="default-line default-line-with-docs">Default: <span className="default-value">explore</span></div> |
| <div className="path-line"><span className="path-main">include_column_lineage</span></div> <div className="type-name-line"><span className="type-name">boolean</span></div> | When enabled, column-level lineage will be extracted from the dbt node definition. Requires `infer_dbt_schemas` to be enabled. If you run into issues where the column name casing does not match up with properly, providing a datahub_api or using the rest sink will improve accuracy. <div className="default-line default-line-with-docs">Default: <span className="default-value">True</span></div> |
| <div className="path-line"><span className="path-main">include_compiled_code</span></div> <div className="type-name-line"><span className="type-name">boolean</span></div> | When enabled, includes the compiled code in the emitted metadata. <div className="default-line default-line-with-docs">Default: <span className="default-value">True</span></div> |
| <div className="path-line"><span className="path-main">include_database_name</span></div> <div className="type-name-line"><span className="type-name">boolean</span></div> | Whether to add database name to the table urn. Set to False to skip it for engines like AWS Athena where it's not required. <div className="default-line default-line-with-docs">Default: <span className="default-value">True</span></div> |
| <div className="path-line"><span className="path-main">include_env_in_assertion_guid</span></div> <div className="type-name-line"><span className="type-name">boolean</span></div> | Prior to version 0.9.4.2, the assertion GUIDs did not include the environment. If you're using multiple dbt ingestion that are only distinguished by env, then you should set this flag to True. <div className="default-line default-line-with-docs">Default: <span className="default-value">False</span></div> |
| <div className="path-line"><span className="path-main">incremental_lineage</span></div> <div className="type-name-line"><span className="type-name">boolean</span></div> | When enabled, emits incremental/patch lineage for non-dbt entities. When disabled, re-states lineage on each run. This would also require enabling 'incremental_lineage' in the counterpart warehouse ingestion (_e.g._ BigQuery, Redshift, etc). <div className="default-line default-line-with-docs">Default: <span className="default-value">True</span></div> |
| <div className="path-line"><span className="path-main">infer_dbt_schemas</span></div> <div className="type-name-line"><span className="type-name">boolean</span></div> | When enabled, schemas will be inferred from the dbt node definition. <div className="default-line default-line-with-docs">Default: <span className="default-value">True</span></div> |
| <div className="path-line"><span className="path-main">job_id</span></div> <div className="type-name-line"><span className="type-name">One of integer, null</span></div> | The ID of the job to ingest metadata from. Required in explicit mode (when auto_discovery is disabled). <div className="default-line default-line-with-docs">Default: <span className="default-value">None</span></div> |
| <div className="path-line"><span className="path-main">max_queries_per_model</span></div> <div className="type-name-line"><span className="type-name">integer</span></div> | Maximum number of Query entities to emit per dbt model. Prevents metadata explosion from malformed manifests. Set to 0 for unlimited. <div className="default-line default-line-with-docs">Default: <span className="default-value">100</span></div> |
| <div className="path-line"><span className="path-main">meta_mapping</span></div> <div className="type-name-line"><span className="type-name">object</span></div> | mapping rules that will be executed against dbt meta properties. Refer to the section below on dbt meta automated mappings. <div className="default-line default-line-with-docs">Default: <span className="default-value">&#123;&#125;</span></div> |
| <div className="path-line"><span className="path-main">metadata_endpoint</span></div> <div className="type-name-line"><span className="type-name">string</span></div> | The dbt Cloud metadata API endpoint. If not provided, we will try to infer it from the access_url. <div className="default-line default-line-with-docs">Default: <span className="default-value">https://metadata.cloud.getdbt.com/graphql</span></div> |
| <div className="path-line"><span className="path-main">owner_extraction_pattern</span></div> <div className="type-name-line"><span className="type-name">One of string, null</span></div> | Regex string to extract owner from the dbt node using the `(?P<name>...) syntax` of the [match object](https://docs.python.org/3/library/re.html#match-objects), where the group name must be `owner`. Examples: (1)`r"(?P<owner>(.*)): (\w+) (\w+)"` will extract `jdoe` as the owner from `"jdoe: John Doe"` (2) `r"@(?P<owner>(.*))"` will extract `alice` as the owner from `"@alice"`. <div className="default-line default-line-with-docs">Default: <span className="default-value">None</span></div> |
| <div className="path-line"><span className="path-main">platform_instance</span></div> <div className="type-name-line"><span className="type-name">One of string, null</span></div> | The instance of the platform that all assets produced by this recipe belong to. This should be unique within the platform. See https://docs.datahub.com/docs/platform-instances/ for more details. <div className="default-line default-line-with-docs">Default: <span className="default-value">None</span></div> |
| <div className="path-line"><span className="path-main">prefer_sql_parser_lineage</span></div> <div className="type-name-line"><span className="type-name">boolean</span></div> | Normally we use dbt's metadata to generate table lineage. When enabled, we prefer results from the SQL parser when generating lineage instead. This can be useful when dbt models reference tables directly, instead of using the ref() macro. This requires that `skip_sources_in_lineage` is enabled. <div className="default-line default-line-with-docs">Default: <span className="default-value">False</span></div> |
| <div className="path-line"><span className="path-main">query_tag_mapping</span></div> <div className="type-name-line"><span className="type-name">object</span></div> | mapping rules that will be executed against dbt query_tag meta properties. Refer to the section below on dbt meta automated mappings. <div className="default-line default-line-with-docs">Default: <span className="default-value">&#123;&#125;</span></div> |
| <div className="path-line"><span className="path-main">run_id</span></div> <div className="type-name-line"><span className="type-name">One of integer, null</span></div> | The ID of the run to ingest metadata from. If not specified, defaults to the latest run. In auto-discovery mode, always uses the latest run for each job. <div className="default-line default-line-with-docs">Default: <span className="default-value">None</span></div> |
| <div className="path-line"><span className="path-main">skip_missing_upstreams_in_lineage</span></div> <div className="type-name-line"><span className="type-name">boolean</span></div> | When enabled, upstream datasets that do not already exist in DataHub are excluded from lineage, preventing dangling graph edges from appearing in the lineage UI. Typically used together with `skip_sources_in_lineage` and `entities_enabled.sources: NO`. Important caveats: (1) if dbt is ingested before its upstream source systems, those lineage edges will be silently omitted until dbt is re-ingested after the upstreams are present; (2) adds one graph.exists() round-trip per unique upstream URN per run (cached within the run); (3) soft-deleted upstream entities are treated as present. Requires a DataHub graph connection. <div className="default-line default-line-with-docs">Default: <span className="default-value">False</span></div> |
| <div className="path-line"><span className="path-main">skip_sources_in_lineage</span></div> <div className="type-name-line"><span className="type-name">boolean</span></div> | [Experimental] When enabled, dbt sources will not be included in the lineage graph. Requires that `entities_enabled.sources` is set to `NO`. This is mainly useful when you have multiple, interdependent dbt projects.  <div className="default-line default-line-with-docs">Default: <span className="default-value">False</span></div> |
| <div className="path-line"><span className="path-main">strip_user_ids_from_email</span></div> <div className="type-name-line"><span className="type-name">boolean</span></div> | Whether or not to strip email id while adding owners using dbt meta actions. <div className="default-line default-line-with-docs">Default: <span className="default-value">False</span></div> |
| <div className="path-line"><span className="path-main">tag_prefix</span></div> <div className="type-name-line"><span className="type-name">string</span></div> | Prefix added to tags during ingestion. <div className="default-line default-line-with-docs">Default: <span className="default-value">dbt:</span></div> |
| <div className="path-line"><span className="path-main">target_platform_instance</span></div> <div className="type-name-line"><span className="type-name">One of string, null</span></div> | The platform instance for the platform that dbt is operating on. Use this if you have multiple instances of the same platform (e.g. redshift) and need to distinguish between them. <div className="default-line default-line-with-docs">Default: <span className="default-value">None</span></div> |
| <div className="path-line"><span className="path-main">test_warnings_are_errors</span></div> <div className="type-name-line"><span className="type-name">boolean</span></div> | When enabled, dbt test warnings will be treated as failures (emitted as ``AssertionResult.type = FAILURE`` with ``severity = LOW``). The default will change to ``true`` in a future release once assertion result consumers can filter by severity; set ``true`` today to adopt the forthcoming behavior. <div className="default-line default-line-with-docs">Default: <span className="default-value">False</span></div> |
| <div className="path-line"><span className="path-main">use_identifiers</span></div> <div className="type-name-line"><span className="type-name">boolean</span></div> | Use model identifier instead of model name if defined (if not, default to model name). <div className="default-line default-line-with-docs">Default: <span className="default-value">False</span></div> |
| <div className="path-line"><span className="path-main">write_semantics</span></div> <div className="type-name-line"><span className="type-name">string</span></div> | Whether the new tags, terms and owners to be added will override the existing ones added only by this source or not. Value for this config can be "PATCH" or "OVERRIDE" <div className="default-line default-line-with-docs">Default: <span className="default-value">PATCH</span></div> |
| <div className="path-line"><span className="path-main">env</span></div> <div className="type-name-line"><span className="type-name">string</span></div> | Environment to use in namespace when constructing URNs. <div className="default-line default-line-with-docs">Default: <span className="default-value">PROD</span></div> |
| <div className="path-line"><span className="path-main">auto_discovery</span></div> <div className="type-name-line"><span className="type-name">One of AutoDiscoveryConfig, null</span></div> | Auto-discovery configuration. When enabled, automatically discovers jobs for the specified project. <div className="default-line default-line-with-docs">Default: <span className="default-value">None</span></div> |
| <div className="path-line"><span className="path-prefix">auto_discovery.</span><span className="path-main">enabled</span></div> <div className="type-name-line"><span className="type-name">boolean</span></div> | Enable/disable auto-discovery mode. When enabled, discovers production jobs for the specified project. <div className="default-line default-line-with-docs">Default: <span className="default-value">False</span></div> |
| <div className="path-line"><span className="path-prefix">auto_discovery.</span><span className="path-main">require_generate_docs</span></div> <div className="type-name-line"><span className="type-name">boolean</span></div> | If True, only ingest jobs that have 'Generate docs on run' enabled in dbt Cloud. If False (default), ingest all production jobs regardless of the generate_docs setting. <div className="default-line default-line-with-docs">Default: <span className="default-value">False</span></div> |
| <div className="path-line"><span className="path-prefix">auto_discovery.</span><span className="path-main">job_id_pattern</span></div> <div className="type-name-line"><span className="type-name">AllowDenyPattern</span></div> | A class to store allow deny regexes. <br />  <br /> Patterns are matched against the start of the string only, not the entire <br /> string - a pattern does not need to match to the end to be considered a match. <br /> For example, the pattern "prod" matches "prod", "prod_east", and "production". <br /> To require an exact match, anchor your pattern explicitly, e.g. "^prod$".  |
| <div className="path-line"><span className="path-prefix">auto_discovery.job_id_pattern.</span><span className="path-main">ignoreCase</span></div> <div className="type-name-line"><span className="type-name">One of boolean, null</span></div> | Whether to ignore case sensitivity during pattern matching. <div className="default-line default-line-with-docs">Default: <span className="default-value">True</span></div> |
| <div className="path-line"><span className="path-main">entities_enabled</span></div> <div className="type-name-line"><span className="type-name">DBTEntitiesEnabled</span></div> | Controls which dbt entities are going to be emitted by this source  |
| <div className="path-line"><span className="path-prefix">entities_enabled.</span><span className="path-main">catalog_stats</span></div> <div className="type-name-line"><span className="type-name">Enum</span></div> | One of: "YES", "NO", "ONLY"  |
| <div className="path-line"><span className="path-prefix">entities_enabled.</span><span className="path-main">exposures</span></div> <div className="type-name-line"><span className="type-name">Enum</span></div> | One of: "YES", "NO", "ONLY"  |
| <div className="path-line"><span className="path-prefix">entities_enabled.</span><span className="path-main">model_performance</span></div> <div className="type-name-line"><span className="type-name">Enum</span></div> | One of: "YES", "NO", "ONLY"  |
| <div className="path-line"><span className="path-prefix">entities_enabled.</span><span className="path-main">models</span></div> <div className="type-name-line"><span className="type-name">Enum</span></div> | One of: "YES", "NO", "ONLY"  |
| <div className="path-line"><span className="path-prefix">entities_enabled.</span><span className="path-main">queries</span></div> <div className="type-name-line"><span className="type-name">Enum</span></div> | One of: "YES", "NO", "ONLY"  |
| <div className="path-line"><span className="path-prefix">entities_enabled.</span><span className="path-main">seeds</span></div> <div className="type-name-line"><span className="type-name">Enum</span></div> | One of: "YES", "NO", "ONLY"  |
| <div className="path-line"><span className="path-prefix">entities_enabled.</span><span className="path-main">semantic_models</span></div> <div className="type-name-line"><span className="type-name">Enum</span></div> | One of: "YES", "NO", "ONLY"  |
| <div className="path-line"><span className="path-prefix">entities_enabled.</span><span className="path-main">snapshots</span></div> <div className="type-name-line"><span className="type-name">Enum</span></div> | One of: "YES", "NO", "ONLY"  |
| <div className="path-line"><span className="path-prefix">entities_enabled.</span><span className="path-main">sources</span></div> <div className="type-name-line"><span className="type-name">Enum</span></div> | One of: "YES", "NO", "ONLY"  |
| <div className="path-line"><span className="path-prefix">entities_enabled.</span><span className="path-main">test_definitions</span></div> <div className="type-name-line"><span className="type-name">Enum</span></div> | One of: "YES", "NO", "ONLY"  |
| <div className="path-line"><span className="path-prefix">entities_enabled.</span><span className="path-main">test_results</span></div> <div className="type-name-line"><span className="type-name">Enum</span></div> | One of: "YES", "NO", "ONLY"  |
| <div className="path-line"><span className="path-main">materialized_node_pattern</span></div> <div className="type-name-line"><span className="type-name">MaterializedNodePatternConfig</span></div> | Configuration for filtering materialized nodes based on their physical location  |
| <div className="path-line"><span className="path-prefix">materialized_node_pattern.</span><span className="path-main">database_pattern</span></div> <div className="type-name-line"><span className="type-name">AllowDenyPattern</span></div> | A class to store allow deny regexes. <br />  <br /> Patterns are matched against the start of the string only, not the entire <br /> string - a pattern does not need to match to the end to be considered a match. <br /> For example, the pattern "prod" matches "prod", "prod_east", and "production". <br /> To require an exact match, anchor your pattern explicitly, e.g. "^prod$".  |
| <div className="path-line"><span className="path-prefix">materialized_node_pattern.database_pattern.</span><span className="path-main">ignoreCase</span></div> <div className="type-name-line"><span className="type-name">One of boolean, null</span></div> | Whether to ignore case sensitivity during pattern matching. <div className="default-line default-line-with-docs">Default: <span className="default-value">True</span></div> |
| <div className="path-line"><span className="path-prefix">materialized_node_pattern.</span><span className="path-main">schema_pattern</span></div> <div className="type-name-line"><span className="type-name">AllowDenyPattern</span></div> | A class to store allow deny regexes. <br />  <br /> Patterns are matched against the start of the string only, not the entire <br /> string - a pattern does not need to match to the end to be considered a match. <br /> For example, the pattern "prod" matches "prod", "prod_east", and "production". <br /> To require an exact match, anchor your pattern explicitly, e.g. "^prod$".  |
| <div className="path-line"><span className="path-prefix">materialized_node_pattern.schema_pattern.</span><span className="path-main">ignoreCase</span></div> <div className="type-name-line"><span className="type-name">One of boolean, null</span></div> | Whether to ignore case sensitivity during pattern matching. <div className="default-line default-line-with-docs">Default: <span className="default-value">True</span></div> |
| <div className="path-line"><span className="path-prefix">materialized_node_pattern.</span><span className="path-main">table_pattern</span></div> <div className="type-name-line"><span className="type-name">AllowDenyPattern</span></div> | A class to store allow deny regexes. <br />  <br /> Patterns are matched against the start of the string only, not the entire <br /> string - a pattern does not need to match to the end to be considered a match. <br /> For example, the pattern "prod" matches "prod", "prod_east", and "production". <br /> To require an exact match, anchor your pattern explicitly, e.g. "^prod$".  |
| <div className="path-line"><span className="path-prefix">materialized_node_pattern.table_pattern.</span><span className="path-main">ignoreCase</span></div> <div className="type-name-line"><span className="type-name">One of boolean, null</span></div> | Whether to ignore case sensitivity during pattern matching. <div className="default-line default-line-with-docs">Default: <span className="default-value">True</span></div> |
| <div className="path-line"><span className="path-main">node_name_pattern</span></div> <div className="type-name-line"><span className="type-name">AllowDenyPattern</span></div> | A class to store allow deny regexes. <br />  <br /> Patterns are matched against the start of the string only, not the entire <br /> string - a pattern does not need to match to the end to be considered a match. <br /> For example, the pattern "prod" matches "prod", "prod_east", and "production". <br /> To require an exact match, anchor your pattern explicitly, e.g. "^prod$".  |
| <div className="path-line"><span className="path-prefix">node_name_pattern.</span><span className="path-main">ignoreCase</span></div> <div className="type-name-line"><span className="type-name">One of boolean, null</span></div> | Whether to ignore case sensitivity during pattern matching. <div className="default-line default-line-with-docs">Default: <span className="default-value">True</span></div> |
| <div className="path-line"><span className="path-main">stateful_ingestion</span></div> <div className="type-name-line"><span className="type-name">One of StatefulStaleMetadataRemovalConfig, null</span></div> | DBT Stateful Ingestion Config. <div className="default-line default-line-with-docs">Default: <span className="default-value">None</span></div> |
| <div className="path-line"><span className="path-prefix">stateful_ingestion.</span><span className="path-main">enabled</span></div> <div className="type-name-line"><span className="type-name">boolean</span></div> | Whether or not to enable stateful ingest. Default: True if a pipeline_name is set and either a datahub-rest sink or `datahub_api` is specified, otherwise False <div className="default-line default-line-with-docs">Default: <span className="default-value">False</span></div> |
| <div className="path-line"><span className="path-prefix">stateful_ingestion.</span><span className="path-main">fail_safe_threshold</span></div> <div className="type-name-line"><span className="type-name">number</span></div> | Prevents large amount of soft deletes & the state from committing from accidental changes to the source configuration if the relative change percent in entities compared to the previous state is above the 'fail_safe_threshold'. <div className="default-line default-line-with-docs">Default: <span className="default-value">75.0</span></div> |
| <div className="path-line"><span className="path-prefix">stateful_ingestion.</span><span className="path-main">remove_stale_metadata</span></div> <div className="type-name-line"><span className="type-name">boolean</span></div> | Soft-deletes the entities present in the last successful run but missing in the current run with stateful_ingestion enabled. <div className="default-line default-line-with-docs">Default: <span className="default-value">True</span></div> |

</div>




#### Schema


The [JSONSchema](https://json-schema.org/) for this configuration is inlined below.


```javascript
{
  "$defs": {
    "AllowDenyPattern": {
      "additionalProperties": false,
      "description": "A class to store allow deny regexes.\n\nPatterns are matched against the start of the string only, not the entire\nstring - a pattern does not need to match to the end to be considered a match.\nFor example, the pattern \"prod\" matches \"prod\", \"prod_east\", and \"production\".\nTo require an exact match, anchor your pattern explicitly, e.g. \"^prod$\".",
      "properties": {
        "allow": {
          "default": [
            ".*"
          ],
          "description": "List of regex patterns to include in ingestion. Patterns match from the start of the string only, not the entire string - anchor with '^...$' for an exact match, e.g. '^prod$'.",
          "items": {
            "type": "string"
          },
          "title": "Allow",
          "type": "array"
        },
        "deny": {
          "default": [],
          "description": "List of regex patterns to exclude from ingestion. Patterns match from the start of the string only, not the entire string - anchor with '^...$' for an exact match, e.g. '^prod$'.",
          "items": {
            "type": "string"
          },
          "title": "Deny",
          "type": "array"
        },
        "ignoreCase": {
          "anyOf": [
            {
              "type": "boolean"
            },
            {
              "type": "null"
            }
          ],
          "default": true,
          "description": "Whether to ignore case sensitivity during pattern matching.",
          "title": "Ignorecase"
        }
      },
      "title": "AllowDenyPattern",
      "type": "object"
    },
    "AutoDiscoveryConfig": {
      "additionalProperties": false,
      "description": "Configuration for auto-discovery mode that automatically discovers jobs for a project.\nRef: DBT Jobs: http://docs.getdbt.com/docs/deploy/jobs\nTODO: The configuration is oraganised this way to allow for future expansion to project discovery at account level.",
      "properties": {
        "enabled": {
          "default": false,
          "description": "Enable/disable auto-discovery mode. When enabled, discovers production jobs for the specified project.",
          "title": "Enabled",
          "type": "boolean"
        },
        "require_generate_docs": {
          "default": false,
          "description": "If True, only ingest jobs that have 'Generate docs on run' enabled in dbt Cloud. If False (default), ingest all production jobs regardless of the generate_docs setting.",
          "title": "Require Generate Docs",
          "type": "boolean"
        },
        "job_id_pattern": {
          "$ref": "#/$defs/AllowDenyPattern",
          "description": "Regex patterns to filter jobs by job_id when auto-discovering."
        }
      },
      "title": "AutoDiscoveryConfig",
      "type": "object"
    },
    "DBTEntitiesEnabled": {
      "additionalProperties": false,
      "description": "Controls which dbt entities are going to be emitted by this source",
      "properties": {
        "models": {
          "$ref": "#/$defs/EmitDirective",
          "default": "YES",
          "description": "Emit metadata for dbt models when set to Yes or Only"
        },
        "sources": {
          "$ref": "#/$defs/EmitDirective",
          "default": "YES",
          "description": "Emit metadata for dbt sources when set to Yes or Only"
        },
        "seeds": {
          "$ref": "#/$defs/EmitDirective",
          "default": "YES",
          "description": "Emit metadata for dbt seeds when set to Yes or Only"
        },
        "snapshots": {
          "$ref": "#/$defs/EmitDirective",
          "default": "YES",
          "description": "Emit metadata for dbt snapshots when set to Yes or Only"
        },
        "test_definitions": {
          "$ref": "#/$defs/EmitDirective",
          "default": "YES",
          "description": "Emit metadata for test definitions when enabled when set to Yes or Only"
        },
        "test_results": {
          "$ref": "#/$defs/EmitDirective",
          "default": "YES",
          "description": "Emit metadata for test results when set to Yes or Only"
        },
        "model_performance": {
          "$ref": "#/$defs/EmitDirective",
          "default": "YES",
          "description": "Emit model performance metadata when set to Yes or Only. Only supported with dbt core."
        },
        "exposures": {
          "$ref": "#/$defs/EmitDirective",
          "default": "YES",
          "description": "Emit metadata for dbt exposures when set to Yes or Only. Exposures represent downstream consumers like dashboards, notebooks, or applications."
        },
        "semantic_models": {
          "$ref": "#/$defs/EmitDirective",
          "default": "YES",
          "description": "Emit metadata for dbt semantic models when set to Yes or Only. Semantic models define entities, dimensions, and measures for the dbt semantic layer (dbt 1.6+)."
        },
        "queries": {
          "$ref": "#/$defs/EmitDirective",
          "default": "YES",
          "description": "Emit Query entities from meta.queries field when set to Yes or Only."
        },
        "catalog_stats": {
          "$ref": "#/$defs/EmitDirective",
          "default": "YES",
          "description": "Emit DatasetProfile aspects with row counts and size from catalog.json stats when set to Yes. Requires catalog.json to be generated by `dbt docs generate`."
        }
      },
      "title": "DBTEntitiesEnabled",
      "type": "object"
    },
    "EmitDirective": {
      "description": "A holder for directives for emission for specific types of entities",
      "enum": [
        "YES",
        "NO",
        "ONLY"
      ],
      "title": "EmitDirective",
      "type": "string"
    },
    "MaterializedNodePatternConfig": {
      "additionalProperties": false,
      "description": "Configuration for filtering materialized nodes based on their physical location",
      "properties": {
        "database_pattern": {
          "$ref": "#/$defs/AllowDenyPattern",
          "default": {
            "allow": [
              ".*"
            ],
            "deny": [],
            "ignoreCase": true
          },
          "description": "Regex patterns for database names to filter materialized nodes."
        },
        "schema_pattern": {
          "$ref": "#/$defs/AllowDenyPattern",
          "default": {
            "allow": [
              ".*"
            ],
            "deny": [],
            "ignoreCase": true
          },
          "description": "Regex patterns for schema names in format '{database}.{schema}' to filter materialized nodes."
        },
        "table_pattern": {
          "$ref": "#/$defs/AllowDenyPattern",
          "default": {
            "allow": [
              ".*"
            ],
            "deny": [],
            "ignoreCase": true
          },
          "description": "Regex patterns for table/view names in format '{database}.{schema}.{table}' to filter materialized nodes."
        }
      },
      "title": "MaterializedNodePatternConfig",
      "type": "object"
    },
    "StatefulStaleMetadataRemovalConfig": {
      "additionalProperties": false,
      "description": "Base specialized config for Stateful Ingestion with stale metadata removal capability.",
      "properties": {
        "enabled": {
          "default": false,
          "description": "Whether or not to enable stateful ingest. Default: True if a pipeline_name is set and either a datahub-rest sink or `datahub_api` is specified, otherwise False",
          "title": "Enabled",
          "type": "boolean"
        },
        "remove_stale_metadata": {
          "default": true,
          "description": "Soft-deletes the entities present in the last successful run but missing in the current run with stateful_ingestion enabled.",
          "title": "Remove Stale Metadata",
          "type": "boolean"
        },
        "fail_safe_threshold": {
          "default": 75.0,
          "description": "Prevents large amount of soft deletes & the state from committing from accidental changes to the source configuration if the relative change percent in entities compared to the previous state is above the 'fail_safe_threshold'.",
          "maximum": 100.0,
          "minimum": 0.0,
          "title": "Fail Safe Threshold",
          "type": "number"
        }
      },
      "title": "StatefulStaleMetadataRemovalConfig",
      "type": "object"
    }
  },
  "additionalProperties": false,
  "properties": {
    "convert_urns_to_lowercase": {
      "default": true,
      "description": "Whether to convert dataset urns to lowercase. Default True to match historical dbt behavior. Set to False for case-sensitive platforms like BigQuery if you need to preserve original identifier casing in URNs.",
      "title": "Convert Urns To Lowercase",
      "type": "boolean"
    },
    "incremental_lineage": {
      "default": true,
      "description": "When enabled, emits incremental/patch lineage for non-dbt entities. When disabled, re-states lineage on each run. This would also require enabling 'incremental_lineage' in the counterpart warehouse ingestion (_e.g._ BigQuery, Redshift, etc).",
      "title": "Incremental Lineage",
      "type": "boolean"
    },
    "env": {
      "default": "PROD",
      "description": "Environment to use in namespace when constructing URNs.",
      "title": "Env",
      "type": "string"
    },
    "platform_instance": {
      "anyOf": [
        {
          "type": "string"
        },
        {
          "type": "null"
        }
      ],
      "default": null,
      "description": "The instance of the platform that all assets produced by this recipe belong to. This should be unique within the platform. See https://docs.datahub.com/docs/platform-instances/ for more details.",
      "title": "Platform Instance"
    },
    "stateful_ingestion": {
      "anyOf": [
        {
          "$ref": "#/$defs/StatefulStaleMetadataRemovalConfig"
        },
        {
          "type": "null"
        }
      ],
      "default": null,
      "description": "DBT Stateful Ingestion Config."
    },
    "target_platform": {
      "description": "The platform that dbt is loading onto. (e.g. bigquery / redshift / postgres etc.)",
      "title": "Target Platform",
      "type": "string"
    },
    "target_platform_instance": {
      "anyOf": [
        {
          "type": "string"
        },
        {
          "type": "null"
        }
      ],
      "default": null,
      "description": "The platform instance for the platform that dbt is operating on. Use this if you have multiple instances of the same platform (e.g. redshift) and need to distinguish between them.",
      "title": "Target Platform Instance"
    },
    "emit_target_platform_instance_aspects": {
      "default": true,
      "description": "When target_platform_instance is set, emit dataPlatformInstance and browsePathsV2 aspects for target-platform sibling entities so they are correctly grouped under their platform instance in browse and filters. Browse paths written by the warehouse connector are never overwritten.",
      "title": "Emit Target Platform Instance Aspects",
      "type": "boolean"
    },
    "emit_target_platform_display_name": {
      "default": true,
      "description": "Set a display name on target-platform entities that the warehouse connector has not ingested. Those entities have no datasetProperties, so the UI falls back to the urn and shows the full dotted path (instance.database.schema.table) rather than just the table name. Enabling this patches datasetProperties.name with the table name, matching how the warehouse connector's own entities are labelled. Has no effect unless both `target_platform_instance` is set and `emit_target_platform_instance_aspects` is enabled - a warning is logged if set without them.",
      "title": "Emit Target Platform Display Name",
      "type": "boolean"
    },
    "use_identifiers": {
      "default": false,
      "description": "Use model identifier instead of model name if defined (if not, default to model name).",
      "title": "Use Identifiers",
      "type": "boolean"
    },
    "entities_enabled": {
      "$ref": "#/$defs/DBTEntitiesEnabled",
      "default": {
        "models": "YES",
        "sources": "YES",
        "seeds": "YES",
        "snapshots": "YES",
        "test_definitions": "YES",
        "test_results": "YES",
        "model_performance": "YES",
        "exposures": "YES",
        "semantic_models": "YES",
        "queries": "YES",
        "catalog_stats": "YES"
      },
      "description": "Controls for enabling / disabling metadata emission for different dbt entities (models, test definitions, test results, etc.)"
    },
    "prefer_sql_parser_lineage": {
      "default": false,
      "description": "Normally we use dbt's metadata to generate table lineage. When enabled, we prefer results from the SQL parser when generating lineage instead. This can be useful when dbt models reference tables directly, instead of using the ref() macro. This requires that `skip_sources_in_lineage` is enabled.",
      "title": "Prefer Sql Parser Lineage",
      "type": "boolean"
    },
    "skip_sources_in_lineage": {
      "default": false,
      "description": "[Experimental] When enabled, dbt sources will not be included in the lineage graph. Requires that `entities_enabled.sources` is set to `NO`. This is mainly useful when you have multiple, interdependent dbt projects. ",
      "title": "Skip Sources In Lineage",
      "type": "boolean"
    },
    "skip_missing_upstreams_in_lineage": {
      "default": false,
      "description": "When enabled, upstream datasets that do not already exist in DataHub are excluded from lineage, preventing dangling graph edges from appearing in the lineage UI. Typically used together with `skip_sources_in_lineage` and `entities_enabled.sources: NO`. Important caveats: (1) if dbt is ingested before its upstream source systems, those lineage edges will be silently omitted until dbt is re-ingested after the upstreams are present; (2) adds one graph.exists() round-trip per unique upstream URN per run (cached within the run); (3) soft-deleted upstream entities are treated as present. Requires a DataHub graph connection.",
      "title": "Skip Missing Upstreams In Lineage",
      "type": "boolean"
    },
    "tag_prefix": {
      "default": "dbt:",
      "description": "Prefix added to tags during ingestion.",
      "title": "Tag Prefix",
      "type": "string"
    },
    "node_name_pattern": {
      "$ref": "#/$defs/AllowDenyPattern",
      "default": {
        "allow": [
          ".*"
        ],
        "deny": [],
        "ignoreCase": true
      },
      "description": "regex patterns for dbt model names to filter in ingestion."
    },
    "materialized_node_pattern": {
      "$ref": "#/$defs/MaterializedNodePatternConfig",
      "default": {
        "database_pattern": {
          "allow": [
            ".*"
          ],
          "deny": [],
          "ignoreCase": true
        },
        "schema_pattern": {
          "allow": [
            ".*"
          ],
          "deny": [],
          "ignoreCase": true
        },
        "table_pattern": {
          "allow": [
            ".*"
          ],
          "deny": [],
          "ignoreCase": true
        }
      },
      "description": "Advanced filtering for materialized nodes based on their physical database location. Provides fine-grained control over database.schema.table patterns for catalog consistency."
    },
    "meta_mapping": {
      "additionalProperties": true,
      "default": {},
      "description": "mapping rules that will be executed against dbt meta properties. Refer to the section below on dbt meta automated mappings.",
      "title": "Meta Mapping",
      "type": "object"
    },
    "column_meta_mapping": {
      "additionalProperties": true,
      "default": {},
      "description": "mapping rules that will be executed against dbt column meta properties. Refer to the section below on dbt meta automated mappings.",
      "title": "Column Meta Mapping",
      "type": "object"
    },
    "enable_meta_mapping": {
      "default": true,
      "description": "When enabled, applies the mappings that are defined through the meta_mapping directives.",
      "title": "Enable Meta Mapping",
      "type": "boolean"
    },
    "query_tag_mapping": {
      "additionalProperties": true,
      "default": {},
      "description": "mapping rules that will be executed against dbt query_tag meta properties. Refer to the section below on dbt meta automated mappings.",
      "title": "Query Tag Mapping",
      "type": "object"
    },
    "enable_query_tag_mapping": {
      "default": true,
      "description": "When enabled, applies the mappings that are defined through the `query_tag_mapping` directives.",
      "title": "Enable Query Tag Mapping",
      "type": "boolean"
    },
    "write_semantics": {
      "default": "PATCH",
      "description": "Whether the new tags, terms and owners to be added will override the existing ones added only by this source or not. Value for this config can be \"PATCH\" or \"OVERRIDE\"",
      "title": "Write Semantics",
      "type": "string"
    },
    "strip_user_ids_from_email": {
      "default": false,
      "description": "Whether or not to strip email id while adding owners using dbt meta actions.",
      "title": "Strip User Ids From Email",
      "type": "boolean"
    },
    "enable_owner_extraction": {
      "default": true,
      "description": "When enabled, ownership info will be extracted from the dbt meta",
      "title": "Enable Owner Extraction",
      "type": "boolean"
    },
    "max_queries_per_model": {
      "default": 100,
      "description": "Maximum number of Query entities to emit per dbt model. Prevents metadata explosion from malformed manifests. Set to 0 for unlimited.",
      "minimum": 0,
      "title": "Max Queries Per Model",
      "type": "integer"
    },
    "owner_extraction_pattern": {
      "anyOf": [
        {
          "type": "string"
        },
        {
          "type": "null"
        }
      ],
      "default": null,
      "description": "Regex string to extract owner from the dbt node using the `(?P<name>...) syntax` of the [match object](https://docs.python.org/3/library/re.html#match-objects), where the group name must be `owner`. Examples: (1)`r\"(?P<owner>(.*)): (\\w+) (\\w+)\"` will extract `jdoe` as the owner from `\"jdoe: John Doe\"` (2) `r\"@(?P<owner>(.*))\"` will extract `alice` as the owner from `\"@alice\"`.",
      "title": "Owner Extraction Pattern"
    },
    "include_env_in_assertion_guid": {
      "default": false,
      "description": "Prior to version 0.9.4.2, the assertion GUIDs did not include the environment. If you're using multiple dbt ingestion that are only distinguished by env, then you should set this flag to True.",
      "title": "Include Env In Assertion Guid",
      "type": "boolean"
    },
    "convert_column_urns_to_lowercase": {
      "default": false,
      "description": "When enabled, converts column URNs to lowercase to ensure cross-platform compatibility. If `target_platform` is Snowflake, the default is True.",
      "title": "Convert Column Urns To Lowercase",
      "type": "boolean"
    },
    "test_warnings_are_errors": {
      "default": false,
      "description": "When enabled, dbt test warnings will be treated as failures (emitted as ``AssertionResult.type = FAILURE`` with ``severity = LOW``). The default will change to ``true`` in a future release once assertion result consumers can filter by severity; set ``true`` today to adopt the forthcoming behavior.",
      "title": "Test Warnings Are Errors",
      "type": "boolean"
    },
    "infer_dbt_schemas": {
      "default": true,
      "description": "When enabled, schemas will be inferred from the dbt node definition.",
      "title": "Infer Dbt Schemas",
      "type": "boolean"
    },
    "include_column_lineage": {
      "default": true,
      "description": "When enabled, column-level lineage will be extracted from the dbt node definition. Requires `infer_dbt_schemas` to be enabled. If you run into issues where the column name casing does not match up with properly, providing a datahub_api or using the rest sink will improve accuracy.",
      "title": "Include Column Lineage",
      "type": "boolean"
    },
    "include_compiled_code": {
      "default": true,
      "description": "When enabled, includes the compiled code in the emitted metadata.",
      "title": "Include Compiled Code",
      "type": "boolean"
    },
    "include_database_name": {
      "default": true,
      "description": "Whether to add database name to the table urn. Set to False to skip it for engines like AWS Athena where it's not required.",
      "title": "Include Database Name",
      "type": "boolean"
    },
    "dbt_is_primary_sibling": {
      "default": true,
      "description": "Experimental: Controls sibling relationship primary designation between dbt entities and target platform entities. When True (default), dbt entities are primary and target platform entities are secondary. When False, target platform entities are primary and dbt entities are secondary. Uses aspect patches for precise control. Requires DataHub server 1.3.0+.",
      "title": "Dbt Is Primary Sibling",
      "type": "boolean"
    },
    "drop_duplicate_sources": {
      "default": true,
      "description": "When enabled, drops sources that have the same name in the target platform as a model. This ensures that lineage is generated reliably, but will lose any documentation associated only with the source.",
      "title": "Drop Duplicate Sources",
      "type": "boolean"
    },
    "access_url": {
      "default": "https://cloud.getdbt.com",
      "description": "The base URL of the dbt Cloud instance to use. This should be the URL you use to access the dbt Cloud UI. It should include the scheme (http/https) and not include a trailing slash. See the access url for your dbt Cloud region here: https://docs.getdbt.com/docs/cloud/about-cloud/regions-ip-addresses",
      "title": "Access Url",
      "type": "string"
    },
    "metadata_endpoint": {
      "default": "https://metadata.cloud.getdbt.com/graphql",
      "description": "The dbt Cloud metadata API endpoint. If not provided, we will try to infer it from the access_url.",
      "title": "Metadata Endpoint",
      "type": "string"
    },
    "token": {
      "description": "The API token to use to authenticate with DBT Cloud.",
      "format": "password",
      "title": "Token",
      "type": "string",
      "writeOnly": true
    },
    "account_id": {
      "description": "The DBT Cloud account ID to use.",
      "title": "Account Id",
      "type": "integer"
    },
    "project_id": {
      "description": "The dbt Cloud project ID to use.",
      "title": "Project Id",
      "type": "integer"
    },
    "job_id": {
      "anyOf": [
        {
          "type": "integer"
        },
        {
          "type": "null"
        }
      ],
      "default": null,
      "description": "The ID of the job to ingest metadata from. Required in explicit mode (when auto_discovery is disabled).",
      "title": "Job Id"
    },
    "run_id": {
      "anyOf": [
        {
          "type": "integer"
        },
        {
          "type": "null"
        }
      ],
      "default": null,
      "description": "The ID of the run to ingest metadata from. If not specified, defaults to the latest run. In auto-discovery mode, always uses the latest run for each job.",
      "title": "Run Id"
    },
    "auto_discovery": {
      "anyOf": [
        {
          "$ref": "#/$defs/AutoDiscoveryConfig"
        },
        {
          "type": "null"
        }
      ],
      "default": null,
      "description": "Auto-discovery configuration. When enabled, automatically discovers jobs for the specified project."
    },
    "external_url_mode": {
      "default": "explore",
      "description": "Where should the \"View in dbt\" link point to - either the \"Explore\" UI or the dbt Cloud IDE",
      "enum": [
        "explore",
        "ide"
      ],
      "title": "External Url Mode",
      "type": "string"
    }
  },
  "required": [
    "target_platform",
    "token",
    "account_id",
    "project_id"
  ],
  "title": "DBTCloudConfig",
  "type": "object"
}
```





### Capabilities

Use the **Important Capabilities** table above as the source of truth for supported features and whether additional configuration is required.

### Limitations

Module behavior is constrained by source APIs, permissions, and metadata exposed by the platform. Refer to capability notes for unsupported or conditional features.

### Troubleshooting

If ingestion fails, validate credentials, permissions, connectivity, and scope filters first. Then review ingestion logs for source-specific errors and adjust configuration accordingly.


### Code Coordinates
- Class Name: `datahub.ingestion.source.dbt.dbt_cloud.DBTCloudSource`
- Browse on [GitHub](https://github.com/datahub-project/datahub/blob/master/metadata-ingestion/src/datahub/ingestion/source/dbt/dbt_cloud.py)


:::tip Questions?

If you've got any questions on configuring ingestion for dbt, feel free to ping us on [our Slack](https://datahub.com/slack).
:::



:::note 💡 **Contributing to this documentation**
This page is auto-generated from the underlying source code. To make changes, please edit the relevant source files in the [metadata-ingestion](https://github.com/datahub-project/datahub/tree/master/metadata-ingestion) directory. 

**Tip:** For quick typo fixes or documentation updates, you can click the ✏️ **Edit** icon directly in the GitHub UI to open a Pull Request. For larger changes and PR naming conventions, please refer to our [Contributing Guide](/docs/contributing).
:::