`. Default: True
|
| max_query_duration
string(duration)
| Correction to pad start_time and end_time with. For handling the case where the read happens within our time range but the query completion event is delayed and happens after the configured end time. Default: PT15M
|
| max_threads_dataset_parallelism
integer
| Number of worker threads to use to parallelize BigQuery Dataset Metadata Extraction. Set to 1 to disable. Default: 20
|
| number_of_datasets_process_in_batch_if_profiling_enabled
integer
| Number of partitioned table queried in batch when getting metadata. This is a low level config property which should be touched with care. This restriction is needed because we query partitions system view which throws error if we try to touch too many tables. Default: 1000
|
| options
object
| Any options specified here will be passed to [SQLAlchemy.create_engine](https://docs.sqlalchemy.org/en/14/core/engines.html#sqlalchemy.create_engine) as kwargs. |
| platform_instance
One of string, null
| The instance of the platform that all assets produced by this recipe belong to. This should be unique within the platform. See https://docs.datahub.com/docs/platform-instances/ for more details. Default: None
|
| project_on_behalf
One of string, null
| [Advanced] The BigQuery project in which queries are executed. Will be passed when creating a job. If not passed, falls back to the project associated with the service account. Default: None
|
| rate_limit
boolean
| Should we rate limit requests made to API. Default: False
|
| region_qualifiers_auto_discovery
boolean
| When True, automatically extends `region_qualifiers` with any BigQuery regions detected from dataset locations during schema ingestion. Defaults to False to avoid unexpected query cost increases. Set to True if your project has datasets in regions beyond `region-us` and `region-eu`. Default: False
|
| requests_per_min
integer
| Used to control number of API calls made per min. Only used when `rate_limit` is set to `True`. Default: 60
|
| scheme
string
| Default: bigquery
|
| sharded_table_pattern
string
| The regex pattern to match sharded tables and group as one table. This is a very low level config parameter, only change if you know what you are doing, Default: ((.+\D)[_$]?)?(\d\d\d\d(?:0[1-9]|1[0-2])(?:0[1-9]|...
|
| start_time
One of string(date-time), string
| Earliest date of lineage/usage to consider. Default: Last full day in UTC (or hour, depending on `bucket_duration`). You can also specify relative time with respect to end_time such as '-7 days' Or '-7d'. Default: None
|
| temp_table_dataset_prefix
string
| If you are creating temp tables in a dataset with a particular prefix you can use this config to set the prefix for the dataset. This is to support workflows from before bigquery's introduction of temp tables. By default we use `_` because of datasets that begin with an underscore are hidden by default https://cloud.google.com/bigquery/docs/datasets#dataset-naming. Default: _
|
| upstream_lineage_in_report
boolean
| Useful for debugging lineage information. Set to True to see the raw lineage created internally. Only works with legacy approach (`use_queries_v2: False`). Default: False
|
| use_date_sharded_audit_log_tables
boolean
| Whether to read date sharded tables or time partitioned tables when extracting usage from exported audit logs. Default: False
|
| use_exported_bigquery_audit_metadata
boolean
| When configured, use BigQueryAuditMetadata in bigquery_audit_metadata_datasets to compute lineage information. Default: False
|
| use_file_backed_cache
boolean
| Whether to use a file backed cache for the view definitions. Default: True
|
| use_legacy_table_stats
boolean
| Source table row count, size, and last-altered time from the undocumented `__TABLES__` Legacy SQL construct instead of the supported `INFORMATION_SCHEMA.PARTITIONS` view. `__TABLES__` additionally covers views and snapshots: views keep their `lastModified` timestamp, and snapshots keep their `lastModified` timestamp along with the `rows_count` and `size_in_bytes` custom properties. `__TABLES__` is undocumented and unsupported by Google, so it may change or stop working without notice. Leave `False` to use `PARTITIONS`, which is dataset-scoped but covers base tables only. Default: False
|
| use_queries_v2
boolean
| If enabled, uses the new queries extractor to extract queries from bigquery. Default: True
|
| use_tables_list_query_v2
boolean
| List tables using an improved query that extracts partitions and last modified timestamps more accurately. Requires the ability to read table data. Automatically enabled when profiling is enabled. Default: False
|
| env
string
| The environment that all assets produced by this connector belong to Default: PROD
|
| bigquery_audit_metadata_datasets
One of array, null
| A list of datasets that contain a table named cloudaudit_googleapis_com_data_access which contain BigQuery audit logs, specifically, those containing BigQueryAuditMetadata. It is recommended that the project of the dataset is also specified, for example, projectA.datasetB. Default: None
|
| bigquery_audit_metadata_datasets.string
string
| |
| capture_dataset_label_as_tag
One of boolean, AllowDenyPattern
| Capture BigQuery dataset labels as DataHub tag Default: False
|
| capture_dataset_label_as_tag.ignoreCase
One of boolean, null
| Whether to ignore case sensitivity during pattern matching. Default: True
|
| capture_dataset_label_as_tag.allow
array
| List of regex patterns to include in ingestion. Patterns match from the start of the string only, not the entire string - anchor with '^...$' for an exact match, e.g. '^prod$'. Default: ['.*']
|
| capture_dataset_label_as_tag.allow.string
string
| |
| capture_dataset_label_as_tag.deny
array
| List of regex patterns to exclude from ingestion. Patterns match from the start of the string only, not the entire string - anchor with '^...$' for an exact match, e.g. '^prod$'. Default: []
|
| capture_dataset_label_as_tag.deny.string
string
| |
| capture_table_label_as_tag
One of boolean, AllowDenyPattern
| Capture BigQuery table labels as DataHub tag Default: False
|
| capture_table_label_as_tag.ignoreCase
One of boolean, null
| Whether to ignore case sensitivity during pattern matching. Default: True
|
| capture_table_label_as_tag.allow
array
| List of regex patterns to include in ingestion. Patterns match from the start of the string only, not the entire string - anchor with '^...$' for an exact match, e.g. '^prod$'. Default: ['.*']
|
| capture_table_label_as_tag.allow.string
string
| |
| capture_table_label_as_tag.deny
array
| List of regex patterns to exclude from ingestion. Patterns match from the start of the string only, not the entire string - anchor with '^...$' for an exact match, e.g. '^prod$'. Default: []
|
| capture_table_label_as_tag.deny.string
string
| |
| capture_view_label_as_tag
One of boolean, AllowDenyPattern
| Capture BigQuery view labels as DataHub tag Default: False
|
| capture_view_label_as_tag.ignoreCase
One of boolean, null
| Whether to ignore case sensitivity during pattern matching. Default: True
|
| capture_view_label_as_tag.allow
array
| List of regex patterns to include in ingestion. Patterns match from the start of the string only, not the entire string - anchor with '^...$' for an exact match, e.g. '^prod$'. Default: ['.*']
|
| capture_view_label_as_tag.allow.string
string
| |
| capture_view_label_as_tag.deny
array
| List of regex patterns to exclude from ingestion. Patterns match from the start of the string only, not the entire string - anchor with '^...$' for an exact match, e.g. '^prod$'. Default: []
|
| capture_view_label_as_tag.deny.string
string
| |
| credential
One of GCPCredential, null
| BigQuery service account credential. Required when auth_type is 'service_account' unless Application Default Credentials are available (e.g. running on GCE/GKE). Default: None
|
| credential.client_email ❓ string
| Client email |
| string
| Client Id |
| string(password)
| Private key in a form of '-----BEGIN PRIVATE KEY-----\nprivate-key\n-----END PRIVATE KEY-----\n' |
| credential.private_key_id ❓ string
| Private key id |
| credential.auth_provider_x509_cert_url
string
| Auth provider x509 certificate url Default: https://www.googleapis.com/oauth2/v1/certs
|
| credential.auth_uri
string
| Authentication uri Default: https://accounts.google.com/o/oauth2/auth
|
| credential.client_x509_cert_url
One of string, null
| If not set it will be default to https://www.googleapis.com/robot/v1/metadata/x509/client_email Default: None
|
| credential.project_id
One of string, null
| Project id to set the credentials Default: None
|
| credential.token_uri
string
| Token uri Default: https://oauth2.googleapis.com/token
|
| credential.type
string
| Authentication type Default: service_account
|
| dataset_pattern
AllowDenyPattern
| A class to store allow deny regexes.
Patterns are matched against the start of the string only, not the entire
string - a pattern does not need to match to the end to be considered a match.
For example, the pattern "prod" matches "prod", "prod_east", and "production".
To require an exact match, anchor your pattern explicitly, e.g. "^prod$". |
| dataset_pattern.ignoreCase
One of boolean, null
| Whether to ignore case sensitivity during pattern matching. Default: True
|
| domain
map(str,AllowDenyPattern)
| A class to store allow deny regexes.
Patterns are matched against the start of the string only, not the entire
string - a pattern does not need to match to the end to be considered a match.
For example, the pattern "prod" matches "prod", "prod_east", and "production".
To require an exact match, anchor your pattern explicitly, e.g. "^prod$". |
| domain.`key`.allow
array
| List of regex patterns to include in ingestion. Patterns match from the start of the string only, not the entire string - anchor with '^...$' for an exact match, e.g. '^prod$'. Default: ['.*']
|
| domain.`key`.allow.string
string
| |
| domain.`key`.ignoreCase
One of boolean, null
| Whether to ignore case sensitivity during pattern matching. Default: True
|
| domain.`key`.deny
array
| List of regex patterns to exclude from ingestion. Patterns match from the start of the string only, not the entire string - anchor with '^...$' for an exact match, e.g. '^prod$'. Default: []
|
| domain.`key`.deny.string
string
| |
| gcs_lineage_config
GcsLineageProviderConfig
| Any source that produces gcs lineage from/to Datasets should inherit this class. |
| gcs_lineage_config.ignore_non_path_spec_path
boolean
| Ignore paths that are not match in path_specs. It only applies if path_specs are specified. Default: False
|
| gcs_lineage_config.strip_urls
boolean
| Strip filename from gcs url. It only applies if path_specs are not specified. Default: True
|
| gcs_lineage_config.path_specs
array
| List of PathSpec. See below the details about PathSpec Default: []
|
| gcs_lineage_config.path_specs.PathSpec
PathSpec
| |
| gcs_lineage_config.path_specs.PathSpec.include ❓ string
| Path to table. Name variable `{table}` is used to mark the folder with dataset. In absence of `{table}`, file level dataset will be created. Check below examples for more details. |
| gcs_lineage_config.path_specs.PathSpec.allow_double_stars
boolean
| Allow double stars in the include path. This can affect performance significantly if enabled Default: False
|
| gcs_lineage_config.path_specs.PathSpec.autodetect_partitions
boolean
| Autodetect partition(s) from the path. If set to true, it will autodetect partition key/value if the folder format is {partition_key}={partition_value} for example `year=2024` Default: True
|
| gcs_lineage_config.path_specs.PathSpec.default_extension
One of string, null
| For files without extension it will assume the specified file type. If it is not set the files without extensions will be skipped. Default: None
|
| gcs_lineage_config.path_specs.PathSpec.emit_folders_only
boolean
| If set, this path_spec emits each matching storage folder as a DataHub Container and creates no dataset entities. Folder depth is set by the number of wildcard levels in `include` (e.g. `.../*/` one level deep, `.../*/*/` two levels). `exclude` and `include_hidden_folders` still apply to the folder walk; file/dataset fields (`file_types`, `default_extension`, `table_name`, `tables_filter_pattern`) are rejected since no datasets are produced. `include` must end at a folder and must not contain `{table}` or `**`. Default: False
|
| gcs_lineage_config.path_specs.PathSpec.enable_compression
boolean
| Enable or disable processing compressed files. Currently .gz and .bz files are supported. Default: True
|
| gcs_lineage_config.path_specs.PathSpec.include_hidden_folders
boolean
| Include hidden folders in the traversal (folders starting with . or _ Default: False
|
| gcs_lineage_config.path_specs.PathSpec.sample_files
boolean
| Not listing all the files but only taking a handful amount of sample file to infer the schema. File count and file size calculation will be disabled. This can affect performance significantly if enabled Default: True
|
| gcs_lineage_config.path_specs.PathSpec.table_name
One of string, null
| Display name of the dataset.Combination of named variables from include path and strings Default: None
|
| gcs_lineage_config.path_specs.PathSpec.traversal_method
Enum
| One of: "ALL", "MIN_MAX", "MAX" |
| gcs_lineage_config.path_specs.PathSpec.exclude
One of array, null
| list of paths in glob pattern which will be excluded while scanning for the datasets Default: []
|
| gcs_lineage_config.path_specs.PathSpec.exclude.string
string
| |
| gcs_lineage_config.path_specs.PathSpec.file_types
array
| Files with extenstions specified here (subset of default value) only will be scanned to create dataset. Other files will be omitted. Default: ['csv', 'tsv', 'json', 'jsonl', 'parquet', 'avro']
|
| gcs_lineage_config.path_specs.PathSpec.file_types.string
string
| |
| gcs_lineage_config.path_specs.PathSpec.tables_filter_pattern
AllowDenyPattern
| A class to store allow deny regexes.
Patterns are matched against the start of the string only, not the entire
string - a pattern does not need to match to the end to be considered a match.
For example, the pattern "prod" matches "prod", "prod_east", and "production".
To require an exact match, anchor your pattern explicitly, e.g. "^prod$". |
| gcs_lineage_config.path_specs.PathSpec.tables_filter_pattern.ignoreCase
One of boolean, null
| Whether to ignore case sensitivity during pattern matching. Default: True
|
| profile_pattern
AllowDenyPattern
| A class to store allow deny regexes.
Patterns are matched against the start of the string only, not the entire
string - a pattern does not need to match to the end to be considered a match.
For example, the pattern "prod" matches "prod", "prod_east", and "production".
To require an exact match, anchor your pattern explicitly, e.g. "^prod$". |
| profile_pattern.ignoreCase
One of boolean, null
| Whether to ignore case sensitivity during pattern matching. Default: True
|
| project_id_pattern
AllowDenyPattern
| A class to store allow deny regexes.
Patterns are matched against the start of the string only, not the entire
string - a pattern does not need to match to the end to be considered a match.
For example, the pattern "prod" matches "prod", "prod_east", and "production".
To require an exact match, anchor your pattern explicitly, e.g. "^prod$". |
| project_id_pattern.ignoreCase
One of boolean, null
| Whether to ignore case sensitivity during pattern matching. Default: True
|
| project_ids
array
| Ingests specified project_ids. Use this property if you want to specify what projects to ingest or don't want to give project resourcemanager.projects.list to your service account. Overrides `project_id_pattern`. |
| project_ids.string
string
| |
| project_labels
array
| Ingests projects with the specified labels. Set value in the format of `key:value`. Use this property to define which projects to ingest basedon project-level labels. If project_ids or project_id is set, this configuration has no effect. The ingestion process filters projects by label first, and then applies the project_id_pattern. |
| project_labels.string
string
| |
| pushdown_allow_usernames
array
| List of user email patterns using SQL LIKE syntax (e.g., 'analyst_%@company.com') which WILL be considered for lineage/usage/queries extraction. Uses case-insensitive LIKE for server-side filtering (e.g., 'analyst_%' matches 'Analyst_John@company.com'). Only applicable if `use_queries_v2` is enabled. If not specified, all users not in deny list are included. Default: []
|
| pushdown_allow_usernames.string
string
| |
| pushdown_deny_usernames
array
| List of user email patterns using SQL LIKE syntax (e.g., 'bot_%', '%@%.iam.gserviceaccount.com') which will NOT be considered for lineage/usage/queries extraction. Uses case-insensitive LIKE for server-side filtering (e.g., 'bot_%' matches 'Bot_User@example.com'). This is primarily useful for improving performance by filtering out users with extremely high query volumes. Only applicable if `use_queries_v2` is enabled. Default: []
|
| pushdown_deny_usernames.string
string
| |
| region_qualifiers
array
| BigQuery regions to be scanned for bigquery jobs when using `use_queries_v2`. See [this](https://cloud.google.com/bigquery/docs/information-schema-jobs#scope_and_syntax) for details. |
| region_qualifiers.string
string
| |
| table_pattern
AllowDenyPattern
| A class to store allow deny regexes.
Patterns are matched against the start of the string only, not the entire
string - a pattern does not need to match to the end to be considered a match.
For example, the pattern "prod" matches "prod", "prod_east", and "production".
To require an exact match, anchor your pattern explicitly, e.g. "^prod$". |
| table_pattern.ignoreCase
One of boolean, null
| Whether to ignore case sensitivity during pattern matching. Default: True
|
| table_snapshot_pattern
AllowDenyPattern
| A class to store allow deny regexes.
Patterns are matched against the start of the string only, not the entire
string - a pattern does not need to match to the end to be considered a match.
For example, the pattern "prod" matches "prod", "prod_east", and "production".
To require an exact match, anchor your pattern explicitly, e.g. "^prod$". |
| table_snapshot_pattern.ignoreCase
One of boolean, null
| Whether to ignore case sensitivity during pattern matching. Default: True
|
| usage
BigQueryUsageConfig
| |
| usage.apply_view_usage_to_tables
boolean
| Whether to apply view's usage to its base tables. If set to False, uses sql parser and applies usage to views / tables mentioned in the query. If set to True, usage is applied to base tables only. Only applied with the legacy extraction path (`use_queries_v2: False`); ignored under queries-v2. Default: False
|
| usage.bucket_duration
Enum
| One of: "DAY", "HOUR" |
| usage.end_time
string(date-time)
| Latest date of lineage/usage to consider. Default: Current time in UTC **Deprecated**: set the top-level `end_time` instead - it governs lineage, usage, and operations together. |
| usage.format_sql_queries
boolean
| Whether to format sql queries Default: False
|
| usage.include_operational_stats
boolean
| Whether to display operational stats. Default: True
|
| usage.include_read_operational_stats
boolean
| Whether to report read operational stats. Experimental. Only applied with the legacy extraction path (`use_queries_v2: False`); ignored under queries-v2. Default: False
|
| usage.include_top_n_queries
boolean
| Whether to ingest the top_n_queries. Default: True
|
| usage.max_query_duration
string(duration)
| Correction to pad start_time and end_time with. For handling the case where the read happens within our time range but the query completion event is delayed and happens after the configured end time. **Deprecated**: set the top-level `max_query_duration` instead. Note it only takes effect with the legacy extraction path (`use_queries_v2: False`). Default: PT15M
|
| usage.start_time
string(date-time)
| Earliest date of lineage/usage to consider. Default: Last full day in UTC (or hour, depending on `bucket_duration`). You can also specify relative time with respect to end_time such as '-7 days' Or '-7d'. **Deprecated**: set the top-level `start_time` instead - it governs lineage, usage, and operations together. Default: None
|
| usage.top_n_queries
integer
| Number of top queries to save to each table. Default: 10
|
| usage.user_email_pattern
AllowDenyPattern
| A class to store allow deny regexes.
Patterns are matched against the start of the string only, not the entire
string - a pattern does not need to match to the end to be considered a match.
For example, the pattern "prod" matches "prod", "prod_east", and "production".
To require an exact match, anchor your pattern explicitly, e.g. "^prod$". |
| usage.user_email_pattern.ignoreCase
One of boolean, null
| Whether to ignore case sensitivity during pattern matching. Default: True
|
| view_pattern
AllowDenyPattern
| A class to store allow deny regexes.
Patterns are matched against the start of the string only, not the entire
string - a pattern does not need to match to the end to be considered a match.
For example, the pattern "prod" matches "prod", "prod_east", and "production".
To require an exact match, anchor your pattern explicitly, e.g. "^prod$". |
| view_pattern.ignoreCase
One of boolean, null
| Whether to ignore case sensitivity during pattern matching. Default: True
|
| classification
ClassificationConfig
| |
| classification.enabled
boolean
| Whether classification should be used to auto-detect glossary terms Default: False
|
| classification.info_type_to_term
map(str,string)
| |
| classification.max_workers
integer
| Number of worker processes to use for classification. Set to 1 to disable. Default: 4
|
| classification.sample_size
integer
| Number of sample values used for classification. Default: 100
|
| classification.classifiers
array
| Classifiers to use to auto-detect glossary terms. If more than one classifier, infotype predictions from the classifier defined later in sequence take precedance. Default: [{'type': 'datahub', 'config': None}]
|
| classification.classifiers.DynamicTypedClassifierConfig
DynamicTypedClassifierConfig
| |
| classification.classifiers.DynamicTypedClassifierConfig.type ❓ string
| The type of the classifier to use. The built-in `datahub` classifier has been removed; register a custom classifier and reference its type here. |
| classification.classifiers.DynamicTypedClassifierConfig.config
One of object, null
| The configuration required for initializing the classifier. If not specified, uses defaults for classifer type. Default: None
|
| classification.column_pattern
AllowDenyPattern
| A class to store allow deny regexes.
Patterns are matched against the start of the string only, not the entire
string - a pattern does not need to match to the end to be considered a match.
For example, the pattern "prod" matches "prod", "prod_east", and "production".
To require an exact match, anchor your pattern explicitly, e.g. "^prod$". |
| classification.column_pattern.ignoreCase
One of boolean, null
| Whether to ignore case sensitivity during pattern matching. Default: True
|
| classification.table_pattern
AllowDenyPattern
| A class to store allow deny regexes.
Patterns are matched against the start of the string only, not the entire
string - a pattern does not need to match to the end to be considered a match.
For example, the pattern "prod" matches "prod", "prod_east", and "production".
To require an exact match, anchor your pattern explicitly, e.g. "^prod$". |
| classification.table_pattern.ignoreCase
One of boolean, null
| Whether to ignore case sensitivity during pattern matching. Default: True
|
| profiling
GEProfilingConfig
| |
| profiling.catch_exceptions
boolean
| Default: True
|
| profiling.enabled
boolean
| Whether profiling should be done. Default: False
|
| profiling.field_sample_values_limit
integer
| Upper limit for number of sample values to collect for all columns. Default: 20
|
| profiling.include_field_distinct_count
boolean
| Whether to profile for the number of distinct values for each column. Default: True
|
| profiling.include_field_distinct_value_frequencies
boolean
| Whether to profile for distinct value frequencies. Default: False
|
| profiling.include_field_histogram
boolean
| Whether to profile for the histogram for numeric fields. Default: False
|
| profiling.include_field_max_value
boolean
| Whether to profile for the max value of numeric columns. Default: True
|
| profiling.include_field_mean_value
boolean
| Whether to profile for the mean value of numeric columns. Default: True
|
| profiling.include_field_median_value
boolean
| Whether to profile for the median value of numeric columns. Default: True
|
| profiling.include_field_min_value
boolean
| Whether to profile for the min value of numeric columns. Default: True
|
| profiling.include_field_null_count
boolean
| Whether to profile for the number of nulls for each column. Default: True
|
| profiling.include_field_quantiles
boolean
| Whether to profile for the quantiles of numeric columns. Default: False
|
| profiling.include_field_sample_values
boolean
| Whether to profile for the sample values for all columns. Default: True
|
| profiling.include_field_stddev_value
boolean
| Whether to profile for the standard deviation of numeric columns. Default: True
|
| profiling.limit
One of integer, null
| Max number of documents to profile. By default, profiles all documents. Default: None
|
| profiling.max_distinct_per_statement
integer
| Only used when `query_combiner_flatten_enabled` is on. Maximum COUNT(DISTINCT) columns allowed in one flattened statement. Each one builds a distinct-value tree in server memory, so merging too many trades a scan problem for a memory problem. The default is a starting point, not a measured optimum — raise it if your server has headroom and unique counts dominate profiling time, lower it if profiling causes memory pressure. Default: 5
|
| profiling.max_number_of_fields_to_profile
One of integer, null
| A positive integer that specifies the maximum number of columns to profile for any table. `None` implies all columns. The cost of profiling goes up significantly as the number of columns to profile goes up. Default: None
|
| profiling.max_workers
integer
| Number of worker threads to use for profiling. Set to 1 to disable. Default: 20
|
| profiling.nested_field_max_depth
integer
| Maximum recursion depth when flattening nested JSON structures during profiling. Lower values prevent recursion errors but may truncate deeply nested data. Applies to connectors that process dynamic JSON content (e.g., Kafka, MongoDB, Elasticsearch). Default: 10
|
| profiling.offset
One of integer, null
| Offset in documents to profile. By default, uses no offset. Default: None
|
| profiling.partition_datetime
One of string(date-time), null
| If specified, profile only the partition which matches this datetime. If not specified, profile the latest partition. Only Bigquery supports this. Default: None
|
| profiling.partition_profiling_enabled
boolean
| Whether to profile partitioned tables. Only BigQuery and Aws Athena supports this. If enabled, latest partition data is used for profiling. Default: True
|
| profiling.profile_external_tables
boolean
| Whether to profile external tables. Only Snowflake and Redshift supports this. Default: False
|
| profiling.profile_if_updated_since_days
One of number, null
| Profile table only if it has been updated since these many number of days. If set to `null`, no constraint of last modified time for tables to profile. Supported in `Snowflake`, `BigQuery`, and `Dremio`. Note: for Dremio this compares against DataHub's last-profiled timestamp (Dremio exposes no table modification time), so it controls profile frequency rather than reacting to upstream change. Default: None
|
| profiling.profile_nested_fields
boolean
| Whether to profile complex types like structs, arrays and maps. Default: False
|
| profiling.profile_table_level_only
boolean
| Whether to perform profiling at table-level only, or include column-level profiling as well. Default: False
|
| profiling.profile_table_row_count_estimate_only
boolean
| Use an approximate query for row count. This will be much faster but slightly less accurate. Only supported for Postgres and MySQL. Default: False
|
| profiling.profile_table_row_limit
One of integer, null
| Profile tables only if their row count is less than specified count. If set to `null`, no limit on the row count of tables to profile. Supported only in `Snowflake`, `BigQuery`. Supported for `Oracle` based on gathered stats. Default: 5000000
|
| profiling.profile_table_size_limit
One of integer, null
| Profile tables only if their size is less than specified GBs. If set to `null`, no limit on the size of tables to profile. Supported in `Snowflake`, `BigQuery`, `Databricks`, `Oracle`, and `Teradata`. `Oracle` uses calculated size from gathered stats. `Teradata` uses DBC space accounting. Default: 5
|
| profiling.profiling_isolation_level
One of Enum, null
| Isolation level for the profiling connection. Defaults to unset, so the connection keeps the driver default and one transaction spans the whole table profile. Set AUTOCOMMIT if profiling is holding Postgres idle-in-transaction (blocking VACUUM) or pinning an InnoDB read view and growing the undo log on MySQL; each profiling SELECT then runs on its own. Under AUTOCOMMIT or READ_COMMITTED, metrics come from different snapshots, so a profile can be internally inconsistent on a concurrently-written table (e.g. uniqueCount > rowCount). On MySQL InnoDB, REPEATABLE_READ keeps a consistent snapshot; on Postgres, READ COMMITTED already takes a fresh snapshot per statement, so AUTOCOMMIT loses no consistency there. Default: None
|
| profiling.query_combiner_enabled
boolean
| *This feature is still experimental and can be disabled if it causes issues.* Reduces the total number of queries issued and speeds up profiling by dynamically combining SQL queries where possible. Default: True
|
| profiling.query_combiner_flatten_enabled
boolean
| Flattens same-shape aggregate queries into one flat SELECT per FROM group to reduce full table scans on row stores (e.g. MySQL). Requires `query_combiner_enabled`; has no effect on its own. Off by default. COUNT(DISTINCT) columns are capped per statement to bound server memory. Default: False
|
| profiling.report_dropped_profiles
boolean
| Whether to report datasets or dataset columns which were not profiled. Set to `True` for debugging purposes. Default: False
|
| profiling.sample_size
integer
| Number of rows to be sampled from table for column level profiling.Applicable only if `use_sampling` is set to True. Default: 10000
|
| profiling.turn_off_expensive_profiling_metrics
boolean
| Whether to turn off expensive profiling or not. This turns off profiling for quantiles, distinct_value_frequencies, histogram & sample_values. This also limits maximum number of fields being profiled to 10. Default: False
|
| profiling.use_sampling
boolean
| Whether to profile column level stats on sample of table. Only BigQuery and Snowflake support this. If enabled, profiling is done on rows sampled from table. Sampling is not done for smaller tables. Default: True
|
| profiling.operation_config
OperationConfig
| |
| profiling.operation_config.lower_freq_profile_enabled
boolean
| Whether to do profiling at lower freq or not. This does not do any scheduling just adds additional checks to when not to run profiling. Default: False
|
| profiling.operation_config.profile_date_of_month
One of integer, null
| Number between 1 to 31 for date of month (both inclusive). If not specified, defaults to Nothing and this field does not take affect. Default: None
|
| profiling.operation_config.profile_day_of_week
One of integer, null
| Number between 0 to 6 for day of week (both inclusive). 0 is Monday and 6 is Sunday. If not specified, defaults to Nothing and this field does not take affect. Default: None
|
| profiling.tags_to_ignore_sampling
One of array, null
| Fixed list of tags to ignore sampling. Each entry may be a full tag URN (e.g. `urn:li:tag:my_tag`) or just the tag name (e.g. `my_tag`). If not specified, tables will be sampled based on `use_sampling`. Default: None
|
| profiling.tags_to_ignore_sampling.string
string
| |
| stateful_ingestion
One of StatefulStaleMetadataRemovalConfig, null
| Default: None
|
| stateful_ingestion.enabled
boolean
| Whether or not to enable stateful ingest. Default: True if a pipeline_name is set and either a datahub-rest sink or `datahub_api` is specified, otherwise False Default: False
|
| stateful_ingestion.fail_safe_threshold
number
| Prevents large amount of soft deletes & the state from committing from accidental changes to the source configuration if the relative change percent in entities compared to the previous state is above the 'fail_safe_threshold'. Default: 75.0
|
| stateful_ingestion.remove_stale_metadata
boolean
| Soft-deletes the entities present in the last successful run but missing in the current run with stateful_ingestion enabled. Default: True
|
