Skip to main content

Dremio

Overview​

Dremio is a DataHub utility or metadata-focused integration.

The DataHub integration for Dremio covers metadata entities and operational objects relevant to this connector. It also captures table- and column-level lineage, usage statistics, data profiling, ownership, and stateful deletion detection.

Concept Mapping​

Source ConceptDataHub ConceptNotes
Physical Dataset/TableDatasetSubtype: Table
Virtual Dataset/ViewsDatasetSubtype: View
SpacesContainerMapped to DataHub’s Container aspect. Subtype: Space
FoldersContainerMapped as a Container in DataHub. Subtype: Folder
SourcesContainerRepresented as a Container in DataHub. Subtype: Source

Module dremio​

Certified

Important Capabilities​

CapabilityStatusNotes
Asset Containers✅Enabled by default. Supported for types - Dremio Space, Dremio Source.
Column-level Lineage✅Extract column-level lineage. Supported for types - Table.
Data Profiling✅Optionally enabled via configuration.
Dataset Usage✅Enabled by default to get usage stats.
Descriptions✅Enabled by default.
Detect Deleted Entities✅Enabled by default via stateful ingestion.
Domains✅Supported via the domain config field.
Extract Ownership✅Enabled by default.
Operation Capture✅Optionally enabled via include_query_lineage; generated from Dremio job history.
Platform Instance✅Enabled by default.
Table-Level Lineage✅Enabled by default. Supported for types - Table.

Overview​

Dremio is a data lakehouse platform that provides SQL query capabilities across diverse data sources without copying data. The DataHub integration connects directly to Dremio via REST API and system table queries to discover and catalog your data assets.

This module captures physical datasets (tables), virtual datasets (views), spaces, folders, and data sources as containers. Optional capabilities include column-level lineage from view definitions, query-based lineage from job history, data profiling, and stateful deletion of removed entities.

Prerequisites​

You need a running Dremio instance with API access, a user account with appropriate permissions, and network connectivity between DataHub and Dremio.

Authentication​

Generate a Personal Access Token for programmatic access:

  • Log in to your Dremio instance
  • Navigate to your user profile (top-right corner)
  • Select Generate API Token to create a Personal Access Token
  • Save the token securely — it is only displayed once

Required Permissions​

Your user account needs the following Dremio privileges:

Container Access (Required for container discovery)

-- Grant access to view and use sources and spaces
GRANT READ METADATA ON SOURCE <source_name> TO USER <username>
GRANT READ METADATA ON SPACE <space_name> TO USER <username>

-- Or grant at system level for all containers
GRANT READ METADATA ON SYSTEM TO USER <username>

System Tables Access (Required for metadata extraction)

-- Grant access to system tables for dataset metadata
GRANT SELECT ON SYSTEM TO USER <username>

Dataset Access (Optional — only needed for data profiling)

-- For data profiling (only if profiling is enabled)
GRANT SELECT ON SOURCE <source_name> TO USER <username>
GRANT SELECT ON SPACE <space_name> TO USER <username>

Query Lineage Access (Optional — only if include_query_lineage: true)

-- Required for query lineage extraction
GRANT VIEW JOB HISTORY ON SYSTEM TO USER <username>

What each permission does:

  • READ METADATA: Access to view containers and their metadata via API calls
  • SELECT ON SYSTEM: Access to system tables (SYS.*, INFORMATION_SCHEMA.*) for dataset metadata
  • SELECT ON SOURCE/SPACE: Read actual data for profiling (optional)
  • VIEW JOB HISTORY: Access query history tables for lineage (optional)

External Data Source Verification​

If your Dremio instance connects to external data sources (AWS S3, databases, etc.), ensure that:

  • Dremio has proper credentials configured for those sources
  • The DataHub user can access datasets from those external sources
  • Network connectivity exists between Dremio and external sources

Install the Plugin​

pip install 'acryl-datahub[dremio]'

Starter Recipe​

Check out the following recipe to get started with ingestion! See below for full configuration options.

For general pointers on writing and running a recipe, see our main recipe guide.

source:
  type: dremio
  config:
    # Coordinates
    hostname: localhost
    port: 9047
    tls: true

    # Credentials — Personal Access Token (recommended)
    # The user must have READ METADATA and SELECT ON SYSTEM privileges.
    authentication_method: PAT
    password: pass
    # OR basic auth:
    # authentication_method: password
    # username: user
    # password: pass

    # For Dremio Cloud instances
    # is_dremio_cloud: True
    # dremio_cloud_project_id: <project_id>

    # Extract ownership information
    ingest_owner: true

    # --- Lineage ---

    # Enable query-based lineage (reads SYS.JOBS_RECENT)
    include_query_lineage: true

    # Emit lineage as PATCH rather than full-replace, preserving any
    # lineage edges added manually in the DataHub UI between runs.
    # Defaults to false; set true if your GMS supports patch aspects.
    incremental_lineage: false

    # --- Incremental properties ---

    # Emit dataset properties as PATCH rather than full-replace, so
    # descriptions and tags set in the DataHub UI are not overwritten on
    # subsequent runs. Particularly useful when Dremio acts as a semantic
    # layer on top of raw sources that are re-documented in DataHub.
    incremental_properties: false

    # --- Stateful ingestion (checkpointing) ---

    # Stateful ingestion records a checkpoint after each run so that
    # subsequent runs can skip already-processed data and remove stale entities.
    stateful_ingestion:
      enabled: true

    # Advance the query lineage extraction window to start from the end of
    # the previous run, avoiding re-processing job history already ingested.
    # Requires stateful_ingestion to be configured.
    enable_stateful_time_window: false

    # --- Profiling ---

    profiling:
      enabled: false
      # Skip re-profiling tables that DataHub profiled within this many days.
      # Dremio has no table modification timestamps, so this is compared against
      # the last-profiled time rather than last-modified time.
      # Requires stateful_ingestion to be enabled.
      # profile_if_updated_since_days: 1

    # --- Filtering ---

    # Optional: map Dremio sources to external platforms for cross-source lineage
    source_mappings:
      - platform: s3
        source_name: samples

    # Optional: register Dremio source types not yet in the built-in map
    # (e.g. Dremio ARP connectors). Scoped to this recipe.
    # source_type_mappings:
    #   MYNEWCONNECTOR:        # Dremio source type (case-insensitive)
    #     platform: myplatform # DataHub platform name to emit
    #     category: database   # optional: "database" or "file_object_storage"

    # Optional: restrict ingestion to specific schemas or datasets
    schema_pattern:
      allow:
        - "<source_name>.*"
    dataset_pattern:
      allow:
        - "<source_name>.<schema>.*"

sink:
  # sink configs

Config Details​

Note that a . is used to denote nested fields in the YAML recipe.

FieldDescription
authentication_method
One of string, null
Authentication method: 'password' or 'PAT' (Personal Access Token)
Default: PAT
bucket_duration
Enum
One of: "DAY", "HOUR"
disable_certificate_verification
One of boolean, null
Disable TLS certificate verification
Default: False
domain
One of string, null
Domain for all source objects.
Default: None
dremio_cloud_project_id
One of string, null
ID of Dremio Cloud Project. Found in Project Settings in the Dremio Cloud UI
Default: None
dremio_cloud_region
Enum
One of: "US", "EU"
Default: US
enable_stateful_time_window
boolean
Enable stateful time window tracking for query lineage/usage extraction. When enabled, subsequent runs will skip time windows already fully processed, avoiding redundant API calls. Requires stateful_ingestion to be configured.
Default: False
end_time
string(date-time)
Latest date of lineage/usage to consider. Default: Current time in UTC
hostname
One of string, null
Hostname or IP Address of the Dremio server
Default: None
include_query_lineage
boolean
Whether to include query-based lineage information.
Default: False
include_system_tables
boolean
Whether to include system tables and schemas (INFORMATION_SCHEMA, SYS) in ingestion.
Default: True
incremental_lineage
boolean
When enabled, lineage aspects are emitted as PATCH operations rather than full overwrites. This preserves any lineage edges that were manually added in DataHub between runs. Disable if you want each run to fully replace lineage. PATCH emission requires a GMS that supports patch aspects.
Default: False
incremental_properties
boolean
When enabled, emits dataset properties as incremental to existing dataset properties in DataHub. When disabled, re-states dataset properties on each run.
Default: False
ingest_owner
boolean
Ingest Owner from source. This will override Owner info entered from UI
Default: True
is_dremio_cloud
boolean
Whether this is a Dremio Cloud instance
Default: False
max_workers
integer
Number of worker threads to use for parallel processing
Default: 20
password
One of string(password), null
Dremio password or Personal Access Token
Default: None
path_to_certificates
string
Path to SSL certificates
Default: /vercel/path0/metadata-ingestion/venv/lib/python3....
platform_instance
One of string, null
The instance of the platform that all assets produced by this recipe belong to. This should be unique within the platform. See https://docs.datahub.com/docs/platform-instances/ for more details.
Default: None
port
integer
Port of the Dremio REST API
Default: 9047
start_time
string(date-time)
Earliest date of lineage/usage to consider. Default: Last full day in UTC (or hour, depending on bucket_duration). You can also specify relative time with respect to end_time such as '-7 days' Or '-7d'.
Default: None
tls
boolean
Whether the Dremio REST API port is encrypted
Default: True
username
One of string, null
Dremio username
Default: None
env
string
The environment that all assets produced by this connector belong to
Default: PROD
dataset_pattern
AllowDenyPattern
A class to store allow deny regexes
dataset_pattern.ignoreCase
One of boolean, null
Whether to ignore case sensitivity during pattern matching.
Default: True
profile_pattern
AllowDenyPattern
A class to store allow deny regexes
profile_pattern.ignoreCase
One of boolean, null
Whether to ignore case sensitivity during pattern matching.
Default: True
schema_pattern
AllowDenyPattern
A class to store allow deny regexes
schema_pattern.ignoreCase
One of boolean, null
Whether to ignore case sensitivity during pattern matching.
Default: True
source_mappings
One of array, null
Mappings from Dremio sources to DataHub platforms and datasets.
Default: None
source_mappings.DremioSourceMapping
DremioSourceMapping
source_mappings.DremioSourceMapping.platform ❓
string
Source connection made by Dremio (e.g. S3, Snowflake)
source_mappings.DremioSourceMapping.source_name ❓
string
Alias of platform in Dremio connection
source_mappings.DremioSourceMapping.platform_instance
One of string, null
The instance of the platform that all assets produced by this recipe belong to. This should be unique within the platform. See https://docs.datahub.com/docs/platform-instances/ for more details.
Default: None
source_mappings.DremioSourceMapping.env
string
The environment that all assets produced by this connector belong to
Default: PROD
source_type_mappings
map(str,DremioSourceTypeOverride)
source_type_mappings.key.platform ❓
string
DataHub platform name to emit for this Dremio source type (e.g. kafka, iceberg, snowflake).
source_type_mappings.key.category
One of Enum, null
Whether the Dremio source uses dot-notation database/schema paths (database) or slash-notation file paths (file_object_storage). Defaults to unknown.
Default: None
profiling
ProfileConfig
profiling.catch_exceptions
boolean
Default: True
profiling.enabled
boolean
Whether profiling should be done.
Default: False
profiling.field_sample_values_limit
integer
Upper limit for number of sample values to collect for all columns.
Default: 20
profiling.include_field_distinct_count
boolean
Whether to profile for the number of distinct values for each column.
Default: True
profiling.include_field_distinct_value_frequencies
boolean
Whether to profile for distinct value frequencies.
Default: False
profiling.include_field_histogram
boolean
Whether to profile for the histogram for numeric fields.
Default: False
profiling.include_field_max_value
boolean
Whether to profile for the max value of numeric columns.
Default: True
profiling.include_field_mean_value
boolean
Whether to profile for the mean value of numeric columns.
Default: True
profiling.include_field_min_value
boolean
Whether to profile for the min value of numeric columns.
Default: True
profiling.include_field_null_count
boolean
Whether to profile for the number of nulls for each column.
Default: True
profiling.include_field_quantiles
boolean
Whether to profile for the quantiles of numeric columns.
Default: False
profiling.include_field_sample_values
boolean
Whether to profile for the sample values for all columns.
Default: True
profiling.include_field_stddev_value
boolean
Whether to profile for the standard deviation of numeric columns.
Default: True
profiling.limit
One of integer, null
Max number of documents to profile. By default, profiles all documents.
Default: None
profiling.max_number_of_fields_to_profile
One of integer, null
A positive integer that specifies the maximum number of columns to profile for any table. None implies all columns. The cost of profiling goes up significantly as the number of columns to profile goes up.
Default: None
profiling.max_workers
integer
Number of worker threads to use for profiling. Set to 1 to disable.
Default: 20
profiling.method
Enum
One of: "ge", "sqlalchemy"
Default: sqlalchemy
profiling.offset
One of integer, null
Offset in documents to profile. By default, uses no offset.
Default: None
profiling.partition_datetime
One of string(date-time), null
If specified, profile only the partition which matches this datetime. If not specified, profile the latest partition. Only Bigquery supports this.
Default: None
profiling.partition_profiling_enabled
boolean
Whether to profile partitioned tables. Only BigQuery and Aws Athena supports this. If enabled, latest partition data is used for profiling.
Default: True
profiling.profile_external_tables
boolean
Whether to profile external tables. Only Snowflake and Redshift supports this.
Default: False
profiling.profile_if_updated_since_days
One of number, null
Profile table only if it has been updated since these many number of days. If set to null, no constraint of last modified time for tables to profile. Supported in Snowflake, BigQuery, and Dremio. Note: for Dremio this compares against DataHub's last-profiled timestamp (Dremio exposes no table modification time), so it controls profile frequency rather than reacting to upstream change.
Default: None
profiling.profile_nested_fields
boolean
Whether to profile complex types like structs, arrays and maps.
Default: False
profiling.profile_table_level_only
boolean
Whether to perform profiling at table-level only, or include column-level profiling as well.
Default: False
profiling.profile_table_row_count_estimate_only
boolean
Use an approximate query for row count. This will be much faster but slightly less accurate. Only supported for Postgres and MySQL.
Default: False
profiling.profile_table_row_limit
One of integer, null
Profile tables only if their row count is less than specified count. If set to null, no limit on the row count of tables to profile. Supported only in Snowflake, BigQuery. Supported for Oracle based on gathered stats.
Default: 5000000
profiling.profile_table_size_limit
One of integer, null
Profile tables only if their size is less than specified GBs. If set to null, no limit on the size of tables to profile. Supported only in Snowflake, BigQuery and Databricks. Supported for Oracle based on calculated size from gathered stats.
Default: 5
profiling.query_combiner_enabled
boolean
This feature is still experimental and can be disabled if it causes issues. Reduces the total number of queries issued and speeds up profiling by dynamically combining SQL queries where possible.
Default: True
profiling.query_timeout
integer
Time before cancelling Dremio profiling query
Default: 300
profiling.report_dropped_profiles
boolean
Whether to report datasets or dataset columns which were not profiled. Set to True for debugging purposes.
Default: False
profiling.sample_size
integer
Number of rows to be sampled from table for column level profiling.Applicable only if use_sampling is set to True.
Default: 10000
profiling.turn_off_expensive_profiling_metrics
boolean
Whether to turn off expensive profiling or not. This turns off profiling for quantiles, distinct_value_frequencies, histogram & sample_values. This also limits maximum number of fields being profiled to 10.
Default: False
profiling.use_sampling
boolean
Whether to profile column level stats on sample of table. Only BigQuery and Snowflake support this. If enabled, profiling is done on rows sampled from table. Sampling is not done for smaller tables.
Default: True
profiling.operation_config
OperationConfig
profiling.operation_config.lower_freq_profile_enabled
boolean
Whether to do profiling at lower freq or not. This does not do any scheduling just adds additional checks to when not to run profiling.
Default: False
profiling.operation_config.profile_date_of_month
One of integer, null
Number between 1 to 31 for date of month (both inclusive). If not specified, defaults to Nothing and this field does not take affect.
Default: None
profiling.operation_config.profile_day_of_week
One of integer, null
Number between 0 to 6 for day of week (both inclusive). 0 is Monday and 6 is Sunday. If not specified, defaults to Nothing and this field does not take affect.
Default: None
profiling.tags_to_ignore_sampling
One of array, null
Fixed list of tags to ignore sampling. If not specified, tables will be sampled based on use_sampling.
Default: None
profiling.tags_to_ignore_sampling.string
string
stateful_ingestion
One of StatefulIngestionConfig, null
Stateful Ingestion Config
Default: None
stateful_ingestion.enabled
boolean
Whether or not to enable stateful ingest. Default: True if a pipeline_name is set and either a datahub-rest sink or datahub_api is specified, otherwise False
Default: False

Capabilities​

Use the Important Capabilities table above as the source of truth for supported features and whether additional configuration is required.

Stateful Ingestion​

Enabling stateful_ingestion unlocks three incremental capabilities that reduce API load on repeated runs:

FeatureConfig keyWhat it does
Stale entity removalstateful_ingestion.enabled: trueRemoves entities from DataHub that no longer exist in Dremio
Time-window deduplicationenable_stateful_time_window: trueAdvances the query lineage start time to the previous run's end time, so SYS.JOBS_RECENT is never re-processed
Incremental profilingprofiling.profile_if_updated_since_daysSkips re-profiling tables that DataHub profiled within the configured window (compared against last-profiled time, since Dremio has no table modification timestamps)
stateful_ingestion:
  enabled: true

enable_stateful_time_window: true # only process new job history each run

profiling:
  enabled: true
  profile_if_updated_since_days: 1 # re-profile at most once per day

Preserving Manual Edits Between Runs​

By default, each ingestion run overwrites the full DatasetProperties aspect, which resets any descriptions or custom properties edited in the DataHub UI. Set incremental_properties: true to emit properties as PATCH operations instead, so only the fields the connector knows about are updated and your manual edits are preserved.

This is particularly useful for Dremio, which often acts as a semantic layer on top of raw sources that teams re-document in DataHub.

incremental_properties: true

Set incremental_lineage: true to emit lineage as PATCH operations, so manually-curated lineage edges added in the DataHub UI are not removed on the next run. Defaults to false (full-overwrite) for consistency with the standard IncrementalLineageConfigMixin used by other connectors; PATCH emission requires a GMS that supports patch aspects.

Limitations​

Module behavior is constrained by source APIs, permissions, and metadata exposed by the platform. Refer to capability notes for unsupported or conditional features.

Troubleshooting​

If ingestion fails, validate credentials, permissions, connectivity, and scope filters first. Then review ingestion logs for source-specific errors and adjust configuration accordingly.

Code Coordinates​

  • Class Name: datahub.ingestion.source.dremio.dremio_source.DremioSource
  • Browse on GitHub
Questions?

If you've got any questions on configuring ingestion for Dremio, feel free to ping us on our Slack.

💡 Contributing to this documentation

This page is auto-generated from the underlying source code. To make changes, please edit the relevant source files in the metadata-ingestion directory.

Tip: For quick typo fixes or documentation updates, you can click the ✏️ Edit icon directly in the GitHub UI to open a Pull Request. For larger changes and PR naming conventions, please refer to our Contributing Guide.