# Oracle ## Overview Oracle is a data platform used to store and query analytical or operational data. Learn more in the [official Oracle documentation](https://www.oracle.com/database/). The DataHub integration for Oracle covers core metadata entities such as datasets/tables/views, schema fields, and containers. It also captures table- and column-level lineage, usage statistics, and stateful deletion detection. ## Concept Mapping While the specific concept mapping is still pending, this shows the generic concept mapping in DataHub. | Source Concept | DataHub Concept | Notes | | -------------------------------------------------------- | ---------------------------- | ---------------------------------------------------------------- | | Platform/account/project scope | Platform Instance, Container | Organizes assets within the platform context. | | Core technical asset (for example table/view/topic/file) | Dataset | Primary ingested technical asset. | | Schema fields / columns | SchemaField | Included when schema extraction is supported. | | Ownership and collaboration principals | CorpUser, CorpGroup | Emitted by modules that support ownership and identity metadata. | | Dependencies and processing relationships | Lineage edges | Available when lineage extraction is supported and enabled. | ## Module `oracle` ![GA](https://img.shields.io/badge/support%20status-GA-brightgreen) ### Important Capabilities | Capability | Status | Notes | | ---------- | ------ | ----- | | Asset Containers | ✅ | Enabled by default. Supported for types - Database, Schema. | | Column-level Lineage | ✅ | Enabled by default to get lineage for stored procedures via `include_lineage` and for views via `include_view_column_lineage`. Supported for types - Stored Procedure, View. | | Dataset Usage | ✅ | Optionally enabled via `include_query_usage` to extract from V$SQL, or via `include_usage_stats` for view/procedure lineage. | | Descriptions | ✅ | Enabled by default. | | [Detect Deleted Entities](../../../../metadata-ingestion/docs/dev_guides/stateful.md#stale-entity-removal) | ✅ | Enabled by default via stateful ingestion. | | [Domains](../../../domains.md) | ✅ | Enabled by default. | | [Operation Capture](../../../api/tutorials/operations.md) | ✅ | Optionally enabled via `include_query_usage` and `include_operational_stats`. | | Schema Metadata | ✅ | Enabled by default. | | Table-Level Lineage | ✅ | Enabled by default to get lineage for stored procedures via `include_lineage` and for views via `include_view_lineage`. Supported for types - Stored Procedure, View. | | Test Connection | ✅ | Enabled by default. | ### Overview The `oracle` module ingests metadata from Oracle into DataHub. It is intended for production ingestion workflows and module-specific capabilities are documented below. The Oracle source extracts metadata from Oracle databases, including: - **Tables and Views**: Standard relational tables and views with column information, constraints, and comments - **Stored Procedures**: Functions, procedures, and packages with source code, arguments, and dependency tracking - **Materialized Views**: Materialized views with proper lineage and refresh information - **Lineage**: Automatic lineage generation from stored procedure definitions and materialized view queries via SQL parsing - **Usage Statistics**: Query execution statistics and table access patterns (when audit data is available) - **Operations**: Data modification events (CREATE, INSERT, UPDATE, DELETE) from audit trail data The connector uses the `python-oracledb` driver and supports both thin mode (default, no Oracle client required) and thick mode (requires Oracle client installation). As a SQL-based service, the Oracle integration is also supported by our SQL profiler for table and column statistics. ### Prerequisites #### Data Dictionary Mode/Views Oracle supports two extraction modes via the `data_dictionary_mode` option: - **`ALL` (default)**: Queries `ALL_*` views — extracts only objects accessible to the ingestion user - **`DBA`**: Queries `DBA_*` views — extracts all schema objects in the database (requires elevated privileges) The following table contains a brief description of what each data dictionary view is used for: | Data Dictionary View | What's it used for? | | -------------------------------------------------- | ----------------------------------------------------- | | `ALL_TABLES` or `DBA_TABLES` | Get list of all relational tables in the database | | `ALL_VIEWS` or `DBA_VIEWS` | Get list of all views in the database | | `ALL_TAB_COMMENTS` or `DBA_TAB_COMMENTS` | Get comments on tables and views | | `ALL_TAB_COLS` or `DBA_TAB_COLS` | Get description of the columns of tables and views | | `ALL_COL_COMMENTS` or `DBA_COL_COMMENTS` | Get comments on the columns of tables and views | | `ALL_TAB_IDENTITY_COLS` or `DBA_TAB_IDENTITY_COLS` | Get table identity columns | | `ALL_CONSTRAINTS` or `DBA_CONSTRAINTS` | Get constraint definitions on tables | | `ALL_CONS_COLUMNS` or `DBA_CONS_COLUMNS` | Get list of columns that are specified in constraints | | `ALL_USERS` or `DBA_USERS` | Get all schema names | | `ALL_OBJECTS` or `DBA_OBJECTS` | Get stored procedures, functions, and packages | | `ALL_SOURCE` or `DBA_SOURCE` | Get source code for stored procedures and functions | | `ALL_ARGUMENTS` or `DBA_ARGUMENTS` | Get arguments for stored procedures and functions | | `ALL_DEPENDENCIES` or `DBA_DEPENDENCIES` | Get dependency information for database objects | | `ALL_MVIEWS` or `DBA_MVIEWS` | Get materialized views and their definitions | #### Data Dictionary Views accessible information and required privileges - **`ALL_` views**: Accessible with standard user privileges — shows only objects the user can access - **`DBA_` views**: Requires `SYSDBA`, `SELECT ANY DICTIONARY` privilege, or `SELECT_CATALOG_ROLE` role #### Required Permissions The following permissions are required based on features used. **Object Access** The ingestion user must have SELECT privileges on the tables and views in each schema you want to catalog. Without these grants, `ALL_*` views will not return metadata for those objects. ```sql -- Option 1: Grant access to all tables (simplest) GRANT SELECT ANY TABLE TO datahub_user; -- Option 2: Grant access per schema BEGIN FOR t IN (SELECT table_name FROM dba_tables WHERE owner = '') LOOP EXECUTE IMMEDIATE 'GRANT SELECT ON .' || t.table_name || ' TO datahub_user'; END LOOP; END; / ``` If using `data_dictionary_mode: DBA`, object-level grants are not required. Instead, grant one of the following: ```sql GRANT SELECT_CATALOG_ROLE TO datahub_user; -- OR GRANT SELECT ANY DICTIONARY TO datahub_user; ``` **Data Dictionary Views (Tables & Views)** ```sql -- Using data_dictionary_mode: ALL (default) GRANT SELECT ON ALL_TABLES TO datahub_user; GRANT SELECT ON ALL_TAB_COLS TO datahub_user; GRANT SELECT ON ALL_TAB_COMMENTS TO datahub_user; GRANT SELECT ON ALL_COL_COMMENTS TO datahub_user; GRANT SELECT ON ALL_VIEWS TO datahub_user; GRANT SELECT ON ALL_CONSTRAINTS TO datahub_user; GRANT SELECT ON ALL_CONS_COLUMNS TO datahub_user; -- Using data_dictionary_mode: DBA (elevated permissions) GRANT SELECT ON DBA_TABLES TO datahub_user; GRANT SELECT ON DBA_TAB_COLS TO datahub_user; GRANT SELECT ON DBA_TAB_COMMENTS TO datahub_user; GRANT SELECT ON DBA_COL_COMMENTS TO datahub_user; GRANT SELECT ON DBA_VIEWS TO datahub_user; GRANT SELECT ON DBA_CONSTRAINTS TO datahub_user; GRANT SELECT ON DBA_CONS_COLUMNS TO datahub_user; ``` **Stored Procedures (enabled by default)** ```sql -- For ALL mode GRANT SELECT ON ALL_OBJECTS TO datahub_user; GRANT SELECT ON ALL_SOURCE TO datahub_user; GRANT SELECT ON ALL_ARGUMENTS TO datahub_user; GRANT SELECT ON ALL_DEPENDENCIES TO datahub_user; -- For DBA mode GRANT SELECT ON DBA_OBJECTS TO datahub_user; GRANT SELECT ON DBA_SOURCE TO datahub_user; GRANT SELECT ON DBA_ARGUMENTS TO datahub_user; GRANT SELECT ON DBA_DEPENDENCIES TO datahub_user; ``` **Materialized Views (enabled by default)** ```sql -- For ALL mode GRANT SELECT ON ALL_MVIEWS TO datahub_user; -- For DBA mode GRANT SELECT ON DBA_MVIEWS TO datahub_user; ``` **Query Usage from V$SQL (optional)** For extracting real query usage patterns, enable `include_query_usage: true` and grant: ```sql GRANT SELECT ON V_$SQL TO datahub_user; -- OR GRANT SELECT_CATALOG_ROLE TO datahub_user; ``` **Database Name Resolution** ```sql GRANT SELECT ON V_$DATABASE TO datahub_user; ``` #### Multitenant (CDB/PDB) Environments In Oracle multitenant architectures (including Exadata), user data lives inside Pluggable Databases (PDBs). The Container Database root (CDB$ROOT) only contains system schemas (SYS, SYSTEM, XDB). To ingest from a PDB, use `service_name` in your recipe config to connect directly to the PDB rather than the CDB: ```yaml source: type: oracle config: host_port: "oracle-host:1521" service_name: "" ``` Grants must be applied inside each PDB, not at the CDB level: ```sql ALTER SESSION SET CONTAINER = ; GRANT SELECT ANY TABLE TO datahub_user; -- (plus dictionary view grants listed above) ``` If you need to ingest from multiple PDBs, create a separate ingestion source for each PDB service name. ### Install the Plugin ```shell pip install 'acryl-datahub[oracle]' ``` ### Starter Recipe Check out the following recipe to get started with ingestion! See [below](#config-details) for full configuration options. For general pointers on writing and running a recipe, see our [main recipe guide](../../../../metadata-ingestion/README.md#recipes). ```yaml source: type: oracle config: # Coordinates host_port: localhost:1521 database: dbname # Credentials username: user password: pass # Options service_name: svc # omit database if using this option # Data Dictionary Mode data_dictionary_mode: "ALL" # or "DBA" for full database access # Stored Procedures include_stored_procedures: true procedure_pattern: allow: - "SCHEMA.*" # Include all procedures from SCHEMA deny: - "SYS.*" # Exclude system procedures # Materialized Views include_materialized_views: true # Usage and Operations (requires audit data or query logs) include_usage_stats: true include_operational_stats: true # Oracle Client Configuration (optional) enable_thick_mode: false # Set to true to use Oracle thick client # thick_mode_lib_dir: "/path/to/oracle/client" # Required on Mac/Windows if thick mode enabled sink: # sink configs ``` ### Config Details #### Options Note that a `.` is used to denote nested fields in the YAML recipe.
| Field | Description | |:--- |:--- | |
host_port ✅
string
| host URL | |
add_database_name_to_urn
One of boolean, null
| Include database name in URNs. Default is False (schema.table format). Set to True for database.schema.table format when ingesting from multiple Oracle databases.
Default: False
| |
bucket_duration
Enum
| One of: "DAY", "HOUR" | |
convert_urns_to_lowercase
boolean
| Whether to convert dataset urns to lowercase. This value is part of each dataset's URN identity, so it must stay fixed for the life of a deployment. Changing it after data has been ingested re-keys every dataset (e.g. `MyDb.MyTable` becomes `mydb.mytable`); with stateful ingestion enabled the old-cased URNs are then soft-deleted as stale while the new-cased ones are created, producing duplicate or orphaned entities. Pick one value before the first run and leave it unchanged.
Default: False
| |
data_dictionary_mode
Enum
| One of: "ALL", "DBA" | |
database
One of string, null
| If using, omit `service_name`.
Default: None
| |
enable_thick_mode
One of boolean, null
| Connection defaults to thin mode. Set to True to enable thick mode.
Default: False
| |
end_time
string(date-time)
| Latest date of lineage/usage to consider. Default: Current time in UTC | |
format_sql_queries
boolean
| Whether to format sql queries
Default: False
| |
include_lineage
boolean
| Enable lineage extraction for stored procedures. When enabled, SQL code in procedures/functions is parsed to extract table dependencies.
Default: True
| |
include_materialized_views
boolean
| Include materialized views in ingestion. Requires access to DBA_MVIEWS or ALL_MVIEWS. If permission errors occur, you can disable this feature or grant the required permissions.
Default: True
| |
include_operational_stats
boolean
| Generate operation statistics from audit trail data (CREATE, INSERT, UPDATE, DELETE operations).
Default: False
| |
include_query_usage
boolean
| Extract actual query usage from V$SQL for usage statistics. Requires SELECT privilege on V$SQL (or SELECT_CATALOG_ROLE). When enabled, usage statistics will be based on actual queries executed in Oracle.
Default: False
| |
include_read_operational_stats
boolean
| Whether to report read operational stats. Experimental.
Default: False
| |
include_stored_procedures
boolean
| Include ingest of stored procedures, functions, and packages. Requires access to DBA_PROCEDURES or ALL_PROCEDURES.
Default: True
| |
include_table_location_lineage
boolean
| If the source supports it, include table lineage to the underlying storage location.
Default: True
| |
include_tables
boolean
| Whether tables should be ingested.
Default: True
| |
include_top_n_queries
boolean
| Whether to ingest the top_n_queries.
Default: True
| |
include_usage_stats
boolean
| Generate usage statistics via SQL aggregator. Requires observed queries to be processed.
Default: False
| |
include_view_column_lineage
boolean
| Populates column-level lineage for view->view and table->view lineage using DataHub's sql parser. Requires `include_view_lineage` to be enabled.
Default: True
| |
include_view_lineage
boolean
| Populates view->view and table->view lineage using DataHub's sql parser.
Default: True
| |
include_views
boolean
| Whether views should be ingested.
Default: True
| |
incremental_lineage
boolean
| When enabled, emits lineage as incremental to existing lineage already in DataHub. When disabled, re-states lineage on each run.
Default: False
| |
lazy_schema_resolver
boolean
| If enabled, skips the upfront bulk fetch of all known schemas from DataHub when resolving lineage. Useful on large DataHub instances where the bulk fetch causes memory or performance issues.
Default: False
| |
max_queries_to_extract
integer
| Maximum number of queries to extract from V$SQL for usage statistics. Queries are ordered by recency (most recent first).
Default: 1000
| |
options
object
| Any options specified here will be passed to [SQLAlchemy.create_engine](https://docs.sqlalchemy.org/en/14/core/engines.html#sqlalchemy.create_engine) as kwargs. To set connection arguments in the URL, specify them under `connect_args`. | |
password
One of string(password), null
| password
Default: None
| |
platform_instance
One of string, null
| The instance of the platform that all assets produced by this recipe belong to. This should be unique within the platform. See https://docs.datahub.com/docs/platform-instances/ for more details.
Default: None
| |
scheme
string
| Will be set automatically to default value.
Default: oracle
| |
service_name
One of string, null
| Oracle service name. If using, omit `database`.
Default: None
| |
sqlalchemy_uri
One of string, null
| URI of database to connect to. See https://docs.sqlalchemy.org/en/14/core/engines.html#database-urls. Takes precedence over other connection parameters.
Default: None
| |
start_time
One of string(date-time), string
| Earliest date of lineage/usage to consider. Default: Last full day in UTC (or hour, depending on `bucket_duration`). You can also specify relative time with respect to end_time such as '-7 days' Or '-7d'.
Default: None
| |
thick_mode_lib_dir
One of string, null
| Path to the directory containing the Oracle Instant Client libraries. Required on Windows and Mac when enable_thick_mode is true. Optional on Linux: when set, the connector preloads the client libraries from this directory before initializing python-oracledb, which makes thick mode work without needing ldconfig or LD_LIBRARY_PATH to be set (see https://github.com/oracle/python-oracledb/issues/578). When unset on Linux, the standard ldconfig / LD_LIBRARY_PATH search is used.
Default: None
| |
top_n_queries
integer
| Number of top queries to save to each table.
Default: 10
| |
urn_db_name
One of string, null
| Override the database name used in URN construction. Only relevant when add_database_name_to_urn is true and service_name is used (i.e. database is not set). The connector auto-detects the name by querying sys_context('USERENV','CON_NAME') (the PDB name in multitenant setups) with a fallback to DB_NAME. Set this explicitly if auto-detection returns the wrong value — for example if your service_name does not route directly to the target PDB. Do not set this alongside database; only one should be used.
Default: None
| |
use_file_backed_cache
boolean
| Whether to use a file backed cache for the view definitions.
Default: True
| |
username
One of string, null
| username
Default: None
| |
env
string
| The environment that all assets produced by this connector belong to
Default: PROD
| |
domain
map(str,AllowDenyPattern)
| A class to store allow deny regexes.

Patterns are matched against the start of the string only, not the entire
string - a pattern does not need to match to the end to be considered a match.
For example, the pattern "prod" matches "prod", "prod_east", and "production".
To require an exact match, anchor your pattern explicitly, e.g. "^prod$". | |
domain.`key`.allow
array
| List of regex patterns to include in ingestion. Patterns match from the start of the string only, not the entire string - anchor with '^...$' for an exact match, e.g. '^prod$'.
Default: ['.*']
| |
domain.`key`.allow.string
string
| | |
domain.`key`.ignoreCase
One of boolean, null
| Whether to ignore case sensitivity during pattern matching.
Default: True
| |
domain.`key`.deny
array
| List of regex patterns to exclude from ingestion. Patterns match from the start of the string only, not the entire string - anchor with '^...$' for an exact match, e.g. '^prod$'.
Default: []
| |
domain.`key`.deny.string
string
| | |
procedure_pattern
AllowDenyPattern
| A class to store allow deny regexes.

Patterns are matched against the start of the string only, not the entire
string - a pattern does not need to match to the end to be considered a match.
For example, the pattern "prod" matches "prod", "prod_east", and "production".
To require an exact match, anchor your pattern explicitly, e.g. "^prod$". | |
procedure_pattern.ignoreCase
One of boolean, null
| Whether to ignore case sensitivity during pattern matching.
Default: True
| |
profile_pattern
AllowDenyPattern
| A class to store allow deny regexes.

Patterns are matched against the start of the string only, not the entire
string - a pattern does not need to match to the end to be considered a match.
For example, the pattern "prod" matches "prod", "prod_east", and "production".
To require an exact match, anchor your pattern explicitly, e.g. "^prod$". | |
profile_pattern.ignoreCase
One of boolean, null
| Whether to ignore case sensitivity during pattern matching.
Default: True
| |
query_exclude_patterns
One of array, null
| Regex patterns for SQL statements to exclude from usage statistics. e.g., ['^SELECT.*FROM SYS\..*', '^BEGIN.*END;'] to exclude system queries and PL/SQL blocks.
Default: None
| |
query_exclude_patterns.string
string
| | |
schema_pattern
AllowDenyPattern
| A class to store allow deny regexes.

Patterns are matched against the start of the string only, not the entire
string - a pattern does not need to match to the end to be considered a match.
For example, the pattern "prod" matches "prod", "prod_east", and "production".
To require an exact match, anchor your pattern explicitly, e.g. "^prod$". | |
schema_pattern.ignoreCase
One of boolean, null
| Whether to ignore case sensitivity during pattern matching.
Default: True
| |
table_pattern
AllowDenyPattern
| A class to store allow deny regexes.

Patterns are matched against the start of the string only, not the entire
string - a pattern does not need to match to the end to be considered a match.
For example, the pattern "prod" matches "prod", "prod_east", and "production".
To require an exact match, anchor your pattern explicitly, e.g. "^prod$". | |
table_pattern.ignoreCase
One of boolean, null
| Whether to ignore case sensitivity during pattern matching.
Default: True
| |
user_email_pattern
AllowDenyPattern
| A class to store allow deny regexes.

Patterns are matched against the start of the string only, not the entire
string - a pattern does not need to match to the end to be considered a match.
For example, the pattern "prod" matches "prod", "prod_east", and "production".
To require an exact match, anchor your pattern explicitly, e.g. "^prod$". | |
user_email_pattern.ignoreCase
One of boolean, null
| Whether to ignore case sensitivity during pattern matching.
Default: True
| |
view_pattern
AllowDenyPattern
| A class to store allow deny regexes.

Patterns are matched against the start of the string only, not the entire
string - a pattern does not need to match to the end to be considered a match.
For example, the pattern "prod" matches "prod", "prod_east", and "production".
To require an exact match, anchor your pattern explicitly, e.g. "^prod$". | |
view_pattern.ignoreCase
One of boolean, null
| Whether to ignore case sensitivity during pattern matching.
Default: True
| |
classification
ClassificationConfig
| | |
classification.enabled
boolean
| Whether classification should be used to auto-detect glossary terms
Default: False
| |
classification.info_type_to_term
map(str,string)
| | |
classification.max_workers
integer
| Number of worker processes to use for classification. Set to 1 to disable.
Default: 4
| |
classification.sample_size
integer
| Number of sample values used for classification.
Default: 100
| |
classification.classifiers
array
| Classifiers to use to auto-detect glossary terms. If more than one classifier, infotype predictions from the classifier defined later in sequence take precedance.
Default: [{'type': 'datahub', 'config': None}]
| |
classification.classifiers.DynamicTypedClassifierConfig
DynamicTypedClassifierConfig
| | |
classification.classifiers.DynamicTypedClassifierConfig.type ❓
string
| The type of the classifier to use. The built-in `datahub` classifier has been removed; register a custom classifier and reference its type here. | |
classification.classifiers.DynamicTypedClassifierConfig.config
One of object, null
| The configuration required for initializing the classifier. If not specified, uses defaults for classifer type.
Default: None
| |
classification.column_pattern
AllowDenyPattern
| A class to store allow deny regexes.

Patterns are matched against the start of the string only, not the entire
string - a pattern does not need to match to the end to be considered a match.
For example, the pattern "prod" matches "prod", "prod_east", and "production".
To require an exact match, anchor your pattern explicitly, e.g. "^prod$". | |
classification.column_pattern.ignoreCase
One of boolean, null
| Whether to ignore case sensitivity during pattern matching.
Default: True
| |
classification.table_pattern
AllowDenyPattern
| A class to store allow deny regexes.

Patterns are matched against the start of the string only, not the entire
string - a pattern does not need to match to the end to be considered a match.
For example, the pattern "prod" matches "prod", "prod_east", and "production".
To require an exact match, anchor your pattern explicitly, e.g. "^prod$". | |
classification.table_pattern.ignoreCase
One of boolean, null
| Whether to ignore case sensitivity during pattern matching.
Default: True
| |
profiling
GEProfilingConfig
| | |
profiling.catch_exceptions
boolean
|
Default: True
| |
profiling.enabled
boolean
| Whether profiling should be done.
Default: False
| |
profiling.field_sample_values_limit
integer
| Upper limit for number of sample values to collect for all columns.
Default: 20
| |
profiling.include_field_distinct_count
boolean
| Whether to profile for the number of distinct values for each column.
Default: True
| |
profiling.include_field_distinct_value_frequencies
boolean
| Whether to profile for distinct value frequencies.
Default: False
| |
profiling.include_field_histogram
boolean
| Whether to profile for the histogram for numeric fields.
Default: False
| |
profiling.include_field_max_value
boolean
| Whether to profile for the max value of numeric columns.
Default: True
| |
profiling.include_field_mean_value
boolean
| Whether to profile for the mean value of numeric columns.
Default: True
| |
profiling.include_field_median_value
boolean
| Whether to profile for the median value of numeric columns.
Default: True
| |
profiling.include_field_min_value
boolean
| Whether to profile for the min value of numeric columns.
Default: True
| |
profiling.include_field_null_count
boolean
| Whether to profile for the number of nulls for each column.
Default: True
| |
profiling.include_field_quantiles
boolean
| Whether to profile for the quantiles of numeric columns.
Default: False
| |
profiling.include_field_sample_values
boolean
| Whether to profile for the sample values for all columns.
Default: True
| |
profiling.include_field_stddev_value
boolean
| Whether to profile for the standard deviation of numeric columns.
Default: True
| |
profiling.limit
One of integer, null
| Max number of documents to profile. By default, profiles all documents.
Default: None
| |
profiling.max_distinct_per_statement
integer
| Only used when `query_combiner_flatten_enabled` is on. Maximum COUNT(DISTINCT) columns allowed in one flattened statement. Each one builds a distinct-value tree in server memory, so merging too many trades a scan problem for a memory problem. The default is a starting point, not a measured optimum — raise it if your server has headroom and unique counts dominate profiling time, lower it if profiling causes memory pressure.
Default: 5
| |
profiling.max_number_of_fields_to_profile
One of integer, null
| A positive integer that specifies the maximum number of columns to profile for any table. `None` implies all columns. The cost of profiling goes up significantly as the number of columns to profile goes up.
Default: None
| |
profiling.max_workers
integer
| Number of worker threads to use for profiling. Set to 1 to disable.
Default: 20
| |
profiling.nested_field_max_depth
integer
| Maximum recursion depth when flattening nested JSON structures during profiling. Lower values prevent recursion errors but may truncate deeply nested data. Applies to connectors that process dynamic JSON content (e.g., Kafka, MongoDB, Elasticsearch).
Default: 10
| |
profiling.offset
One of integer, null
| Offset in documents to profile. By default, uses no offset.
Default: None
| |
profiling.partition_datetime
One of string(date-time), null
| If specified, profile only the partition which matches this datetime. If not specified, profile the latest partition. Only Bigquery supports this.
Default: None
| |
profiling.partition_profiling_enabled
boolean
| Whether to profile partitioned tables. Only BigQuery and Aws Athena supports this. If enabled, latest partition data is used for profiling.
Default: True
| |
profiling.profile_external_tables
boolean
| Whether to profile external tables. Only Snowflake and Redshift supports this.
Default: False
| |
profiling.profile_if_updated_since_days
One of number, null
| Profile table only if it has been updated since these many number of days. If set to `null`, no constraint of last modified time for tables to profile. Supported in `Snowflake`, `BigQuery`, and `Dremio`. Note: for Dremio this compares against DataHub's last-profiled timestamp (Dremio exposes no table modification time), so it controls profile frequency rather than reacting to upstream change.
Default: None
| |
profiling.profile_nested_fields
boolean
| Whether to profile complex types like structs, arrays and maps.
Default: False
| |
profiling.profile_table_level_only
boolean
| Whether to perform profiling at table-level only, or include column-level profiling as well.
Default: False
| |
profiling.profile_table_row_count_estimate_only
boolean
| Use an approximate query for row count. This will be much faster but slightly less accurate. Only supported for Postgres and MySQL.
Default: False
| |
profiling.profile_table_row_limit
One of integer, null
| Profile tables only if their row count is less than specified count. If set to `null`, no limit on the row count of tables to profile. Supported only in `Snowflake`, `BigQuery`. Supported for `Oracle` based on gathered stats.
Default: 5000000
| |
profiling.profile_table_size_limit
One of integer, null
| Profile tables only if their size is less than specified GBs. If set to `null`, no limit on the size of tables to profile. Supported in `Snowflake`, `BigQuery`, `Databricks`, `Oracle`, and `Teradata`. `Oracle` uses calculated size from gathered stats. `Teradata` uses DBC space accounting.
Default: 5
| |
profiling.profiling_isolation_level
One of Enum, null
| Isolation level for the profiling connection. Defaults to unset, so the connection keeps the driver default and one transaction spans the whole table profile. Set AUTOCOMMIT if profiling is holding Postgres idle-in-transaction (blocking VACUUM) or pinning an InnoDB read view and growing the undo log on MySQL; each profiling SELECT then runs on its own. Under AUTOCOMMIT or READ_COMMITTED, metrics come from different snapshots, so a profile can be internally inconsistent on a concurrently-written table (e.g. uniqueCount > rowCount). On MySQL InnoDB, REPEATABLE_READ keeps a consistent snapshot; on Postgres, READ COMMITTED already takes a fresh snapshot per statement, so AUTOCOMMIT loses no consistency there.
Default: None
| |
profiling.query_combiner_enabled
boolean
| *This feature is still experimental and can be disabled if it causes issues.* Reduces the total number of queries issued and speeds up profiling by dynamically combining SQL queries where possible.
Default: True
| |
profiling.query_combiner_flatten_enabled
boolean
| Flattens same-shape aggregate queries into one flat SELECT per FROM group to reduce full table scans on row stores (e.g. MySQL). Requires `query_combiner_enabled`; has no effect on its own. Off by default. COUNT(DISTINCT) columns are capped per statement to bound server memory.
Default: False
| |
profiling.report_dropped_profiles
boolean
| Whether to report datasets or dataset columns which were not profiled. Set to `True` for debugging purposes.
Default: False
| |
profiling.sample_size
integer
| Number of rows to be sampled from table for column level profiling.Applicable only if `use_sampling` is set to True.
Default: 10000
| |
profiling.turn_off_expensive_profiling_metrics
boolean
| Whether to turn off expensive profiling or not. This turns off profiling for quantiles, distinct_value_frequencies, histogram & sample_values. This also limits maximum number of fields being profiled to 10.
Default: False
| |
profiling.use_sampling
boolean
| Whether to profile column level stats on sample of table. Only BigQuery and Snowflake support this. If enabled, profiling is done on rows sampled from table. Sampling is not done for smaller tables.
Default: True
| |
profiling.operation_config
OperationConfig
| | |
profiling.operation_config.lower_freq_profile_enabled
boolean
| Whether to do profiling at lower freq or not. This does not do any scheduling just adds additional checks to when not to run profiling.
Default: False
| |
profiling.operation_config.profile_date_of_month
One of integer, null
| Number between 1 to 31 for date of month (both inclusive). If not specified, defaults to Nothing and this field does not take affect.
Default: None
| |
profiling.operation_config.profile_day_of_week
One of integer, null
| Number between 0 to 6 for day of week (both inclusive). 0 is Monday and 6 is Sunday. If not specified, defaults to Nothing and this field does not take affect.
Default: None
| |
profiling.tags_to_ignore_sampling
One of array, null
| Fixed list of tags to ignore sampling. Each entry may be a full tag URN (e.g. `urn:li:tag:my_tag`) or just the tag name (e.g. `my_tag`). If not specified, tables will be sampled based on `use_sampling`.
Default: None
| |
profiling.tags_to_ignore_sampling.string
string
| | |
stateful_ingestion
One of StatefulStaleMetadataRemovalConfig, null
|
Default: None
| |
stateful_ingestion.enabled
boolean
| Whether or not to enable stateful ingest. Default: True if a pipeline_name is set and either a datahub-rest sink or `datahub_api` is specified, otherwise False
Default: False
| |
stateful_ingestion.fail_safe_threshold
number
| Prevents large amount of soft deletes & the state from committing from accidental changes to the source configuration if the relative change percent in entities compared to the previous state is above the 'fail_safe_threshold'.
Default: 75.0
| |
stateful_ingestion.remove_stale_metadata
boolean
| Soft-deletes the entities present in the last successful run but missing in the current run with stateful_ingestion enabled.
Default: True
|
#### Schema The [JSONSchema](https://json-schema.org/) for this configuration is inlined below. ```javascript { "$defs": { "AllowDenyPattern": { "additionalProperties": false, "description": "A class to store allow deny regexes.\n\nPatterns are matched against the start of the string only, not the entire\nstring - a pattern does not need to match to the end to be considered a match.\nFor example, the pattern \"prod\" matches \"prod\", \"prod_east\", and \"production\".\nTo require an exact match, anchor your pattern explicitly, e.g. \"^prod$\".", "properties": { "allow": { "default": [ ".*" ], "description": "List of regex patterns to include in ingestion. Patterns match from the start of the string only, not the entire string - anchor with '^...$' for an exact match, e.g. '^prod$'.", "items": { "type": "string" }, "title": "Allow", "type": "array" }, "deny": { "default": [], "description": "List of regex patterns to exclude from ingestion. Patterns match from the start of the string only, not the entire string - anchor with '^...$' for an exact match, e.g. '^prod$'.", "items": { "type": "string" }, "title": "Deny", "type": "array" }, "ignoreCase": { "anyOf": [ { "type": "boolean" }, { "type": "null" } ], "default": true, "description": "Whether to ignore case sensitivity during pattern matching.", "title": "Ignorecase" } }, "title": "AllowDenyPattern", "type": "object" }, "BucketDuration": { "enum": [ "DAY", "HOUR" ], "title": "BucketDuration", "type": "string" }, "ClassificationConfig": { "additionalProperties": false, "properties": { "enabled": { "default": false, "description": "Whether classification should be used to auto-detect glossary terms", "title": "Enabled", "type": "boolean" }, "sample_size": { "default": 100, "description": "Number of sample values used for classification.", "title": "Sample Size", "type": "integer" }, "max_workers": { "default": 4, "description": "Number of worker processes to use for classification. Set to 1 to disable.", "title": "Max Workers", "type": "integer" }, "table_pattern": { "$ref": "#/$defs/AllowDenyPattern", "default": { "allow": [ ".*" ], "deny": [], "ignoreCase": true }, "description": "Regex patterns to filter tables for classification. This is used in combination with other patterns in parent config. Specify regex to match the entire table name in `database.schema.table` format. e.g. to match all tables starting with customer in Customer database and public schema, use the regex 'Customer.public.customer.*'" }, "column_pattern": { "$ref": "#/$defs/AllowDenyPattern", "default": { "allow": [ ".*" ], "deny": [], "ignoreCase": true }, "description": "Regex patterns to filter columns for classification. This is used in combination with other patterns in parent config. Specify regex to match the column name in `database.schema.table.column` format." }, "info_type_to_term": { "additionalProperties": { "type": "string" }, "default": {}, "description": "Optional mapping to provide glossary term identifier for info type", "title": "Info Type To Term", "type": "object" }, "classifiers": { "default": [ { "type": "datahub", "config": null } ], "description": "Classifiers to use to auto-detect glossary terms. If more than one classifier, infotype predictions from the classifier defined later in sequence take precedance.", "items": { "$ref": "#/$defs/DynamicTypedClassifierConfig" }, "title": "Classifiers", "type": "array" } }, "title": "ClassificationConfig", "type": "object" }, "DataDictionaryMode": { "description": "Oracle data dictionary access modes.", "enum": [ "ALL", "DBA" ], "title": "DataDictionaryMode", "type": "string" }, "DynamicTypedClassifierConfig": { "additionalProperties": false, "properties": { "type": { "description": "The type of the classifier to use. The built-in `datahub` classifier has been removed; register a custom classifier and reference its type here.", "title": "Type", "type": "string" }, "config": { "anyOf": [ {}, { "type": "null" } ], "default": null, "description": "The configuration required for initializing the classifier. If not specified, uses defaults for classifer type.", "title": "Config" } }, "required": [ "type" ], "title": "DynamicTypedClassifierConfig", "type": "object" }, "GEProfilingConfig": { "additionalProperties": false, "properties": { "enabled": { "default": false, "description": "Whether profiling should be done.", "title": "Enabled", "type": "boolean" }, "operation_config": { "$ref": "#/$defs/OperationConfig", "description": "Experimental feature. To specify operation configs." }, "limit": { "anyOf": [ { "type": "integer" }, { "type": "null" } ], "default": null, "description": "Max number of documents to profile. By default, profiles all documents.", "title": "Limit" }, "offset": { "anyOf": [ { "type": "integer" }, { "type": "null" } ], "default": null, "description": "Offset in documents to profile. By default, uses no offset.", "title": "Offset" }, "profile_table_level_only": { "default": false, "description": "Whether to perform profiling at table-level only, or include column-level profiling as well.", "title": "Profile Table Level Only", "type": "boolean" }, "include_field_null_count": { "default": true, "description": "Whether to profile for the number of nulls for each column.", "title": "Include Field Null Count", "type": "boolean" }, "include_field_distinct_count": { "default": true, "description": "Whether to profile for the number of distinct values for each column.", "title": "Include Field Distinct Count", "type": "boolean" }, "include_field_min_value": { "default": true, "description": "Whether to profile for the min value of numeric columns.", "title": "Include Field Min Value", "type": "boolean" }, "include_field_max_value": { "default": true, "description": "Whether to profile for the max value of numeric columns.", "title": "Include Field Max Value", "type": "boolean" }, "include_field_mean_value": { "default": true, "description": "Whether to profile for the mean value of numeric columns.", "title": "Include Field Mean Value", "type": "boolean" }, "include_field_median_value": { "default": true, "description": "Whether to profile for the median value of numeric columns.", "title": "Include Field Median Value", "type": "boolean" }, "include_field_stddev_value": { "default": true, "description": "Whether to profile for the standard deviation of numeric columns.", "title": "Include Field Stddev Value", "type": "boolean" }, "include_field_quantiles": { "default": false, "description": "Whether to profile for the quantiles of numeric columns.", "title": "Include Field Quantiles", "type": "boolean" }, "include_field_distinct_value_frequencies": { "default": false, "description": "Whether to profile for distinct value frequencies.", "title": "Include Field Distinct Value Frequencies", "type": "boolean" }, "include_field_histogram": { "default": false, "description": "Whether to profile for the histogram for numeric fields.", "title": "Include Field Histogram", "type": "boolean" }, "include_field_sample_values": { "default": true, "description": "Whether to profile for the sample values for all columns.", "title": "Include Field Sample Values", "type": "boolean" }, "max_workers": { "default": 20, "description": "Number of worker threads to use for profiling. Set to 1 to disable.", "title": "Max Workers", "type": "integer" }, "report_dropped_profiles": { "default": false, "description": "Whether to report datasets or dataset columns which were not profiled. Set to `True` for debugging purposes.", "title": "Report Dropped Profiles", "type": "boolean" }, "turn_off_expensive_profiling_metrics": { "default": false, "description": "Whether to turn off expensive profiling or not. This turns off profiling for quantiles, distinct_value_frequencies, histogram & sample_values. This also limits maximum number of fields being profiled to 10.", "title": "Turn Off Expensive Profiling Metrics", "type": "boolean" }, "field_sample_values_limit": { "default": 20, "description": "Upper limit for number of sample values to collect for all columns.", "title": "Field Sample Values Limit", "type": "integer" }, "max_number_of_fields_to_profile": { "anyOf": [ { "exclusiveMinimum": 0, "type": "integer" }, { "type": "null" } ], "default": null, "description": "A positive integer that specifies the maximum number of columns to profile for any table. `None` implies all columns. The cost of profiling goes up significantly as the number of columns to profile goes up.", "title": "Max Number Of Fields To Profile" }, "profile_if_updated_since_days": { "anyOf": [ { "exclusiveMinimum": 0, "type": "number" }, { "type": "null" } ], "default": null, "description": "Profile table only if it has been updated since these many number of days. If set to `null`, no constraint of last modified time for tables to profile. Supported in `Snowflake`, `BigQuery`, and `Dremio`. Note: for Dremio this compares against DataHub's last-profiled timestamp (Dremio exposes no table modification time), so it controls profile frequency rather than reacting to upstream change.", "schema_extra": { "supported_sources": [ "snowflake", "bigquery", "dremio" ] }, "title": "Profile If Updated Since Days" }, "profile_table_size_limit": { "anyOf": [ { "type": "integer" }, { "type": "null" } ], "default": 5, "description": "Profile tables only if their size is less than specified GBs. If set to `null`, no limit on the size of tables to profile. Supported in `Snowflake`, `BigQuery`, `Databricks`, `Oracle`, and `Teradata`. `Oracle` uses calculated size from gathered stats. `Teradata` uses DBC space accounting.", "schema_extra": { "supported_sources": [ "snowflake", "bigquery", "unity-catalog", "oracle", "teradata" ] }, "title": "Profile Table Size Limit" }, "profile_table_row_limit": { "anyOf": [ { "type": "integer" }, { "type": "null" } ], "default": 5000000, "description": "Profile tables only if their row count is less than specified count. If set to `null`, no limit on the row count of tables to profile. Supported only in `Snowflake`, `BigQuery`. Supported for `Oracle` based on gathered stats.", "schema_extra": { "supported_sources": [ "snowflake", "bigquery", "oracle" ] }, "title": "Profile Table Row Limit" }, "profile_table_row_count_estimate_only": { "default": false, "description": "Use an approximate query for row count. This will be much faster but slightly less accurate. Only supported for Postgres and MySQL. ", "schema_extra": { "supported_sources": [ "postgres", "mysql" ] }, "title": "Profile Table Row Count Estimate Only", "type": "boolean" }, "query_combiner_enabled": { "default": true, "description": "*This feature is still experimental and can be disabled if it causes issues.* Reduces the total number of queries issued and speeds up profiling by dynamically combining SQL queries where possible.", "title": "Query Combiner Enabled", "type": "boolean" }, "query_combiner_flatten_enabled": { "default": false, "description": "Flattens same-shape aggregate queries into one flat SELECT per FROM group to reduce full table scans on row stores (e.g. MySQL). Requires `query_combiner_enabled`; has no effect on its own. Off by default. COUNT(DISTINCT) columns are capped per statement to bound server memory.", "title": "Query Combiner Flatten Enabled", "type": "boolean" }, "max_distinct_per_statement": { "default": 5, "description": "Only used when `query_combiner_flatten_enabled` is on. Maximum COUNT(DISTINCT) columns allowed in one flattened statement. Each one builds a distinct-value tree in server memory, so merging too many trades a scan problem for a memory problem. The default is a starting point, not a measured optimum \u2014 raise it if your server has headroom and unique counts dominate profiling time, lower it if profiling causes memory pressure.", "exclusiveMinimum": 0, "title": "Max Distinct Per Statement", "type": "integer" }, "catch_exceptions": { "default": true, "description": "", "title": "Catch Exceptions", "type": "boolean" }, "profiling_isolation_level": { "anyOf": [ { "$ref": "#/$defs/ProfilingIsolationLevel" }, { "type": "null" } ], "default": null, "description": "Isolation level for the profiling connection. Defaults to unset, so the connection keeps the driver default and one transaction spans the whole table profile. Set AUTOCOMMIT if profiling is holding Postgres idle-in-transaction (blocking VACUUM) or pinning an InnoDB read view and growing the undo log on MySQL; each profiling SELECT then runs on its own. Under AUTOCOMMIT or READ_COMMITTED, metrics come from different snapshots, so a profile can be internally inconsistent on a concurrently-written table (e.g. uniqueCount > rowCount). On MySQL InnoDB, REPEATABLE_READ keeps a consistent snapshot; on Postgres, READ COMMITTED already takes a fresh snapshot per statement, so AUTOCOMMIT loses no consistency there.", "schema_extra": { "supported_sources": [ "mysql", "postgres" ] } }, "partition_profiling_enabled": { "default": true, "description": "Whether to profile partitioned tables. Only BigQuery and Aws Athena supports this. If enabled, latest partition data is used for profiling.", "schema_extra": { "supported_sources": [ "athena", "bigquery" ] }, "title": "Partition Profiling Enabled", "type": "boolean" }, "partition_datetime": { "anyOf": [ { "format": "date-time", "type": "string" }, { "type": "null" } ], "default": null, "description": "If specified, profile only the partition which matches this datetime. If not specified, profile the latest partition. Only Bigquery supports this.", "schema_extra": { "supported_sources": [ "bigquery" ] }, "title": "Partition Datetime" }, "use_sampling": { "default": true, "description": "Whether to profile column level stats on sample of table. Only BigQuery and Snowflake support this. If enabled, profiling is done on rows sampled from table. Sampling is not done for smaller tables. ", "schema_extra": { "supported_sources": [ "bigquery", "snowflake" ] }, "title": "Use Sampling", "type": "boolean" }, "sample_size": { "default": 10000, "description": "Number of rows to be sampled from table for column level profiling.Applicable only if `use_sampling` is set to True.", "schema_extra": { "supported_sources": [ "bigquery", "snowflake" ] }, "title": "Sample Size", "type": "integer" }, "profile_external_tables": { "default": false, "description": "Whether to profile external tables. Only Snowflake and Redshift supports this.", "schema_extra": { "supported_sources": [ "redshift", "snowflake" ] }, "title": "Profile External Tables", "type": "boolean" }, "tags_to_ignore_sampling": { "anyOf": [ { "items": { "type": "string" }, "type": "array" }, { "type": "null" } ], "default": null, "description": "Fixed list of tags to ignore sampling. Each entry may be a full tag URN (e.g. `urn:li:tag:my_tag`) or just the tag name (e.g. `my_tag`). If not specified, tables will be sampled based on `use_sampling`.", "title": "Tags To Ignore Sampling" }, "profile_nested_fields": { "default": false, "description": "Whether to profile complex types like structs, arrays and maps. ", "title": "Profile Nested Fields", "type": "boolean" }, "nested_field_max_depth": { "default": 10, "description": "Maximum recursion depth when flattening nested JSON structures during profiling. Lower values prevent recursion errors but may truncate deeply nested data. Applies to connectors that process dynamic JSON content (e.g., Kafka, MongoDB, Elasticsearch).", "exclusiveMinimum": 0, "title": "Nested Field Max Depth", "type": "integer" } }, "title": "GEProfilingConfig", "type": "object" }, "OperationConfig": { "additionalProperties": false, "properties": { "lower_freq_profile_enabled": { "default": false, "description": "Whether to do profiling at lower freq or not. This does not do any scheduling just adds additional checks to when not to run profiling.", "title": "Lower Freq Profile Enabled", "type": "boolean" }, "profile_day_of_week": { "anyOf": [ { "type": "integer" }, { "type": "null" } ], "default": null, "description": "Number between 0 to 6 for day of week (both inclusive). 0 is Monday and 6 is Sunday. If not specified, defaults to Nothing and this field does not take affect.", "title": "Profile Day Of Week" }, "profile_date_of_month": { "anyOf": [ { "type": "integer" }, { "type": "null" } ], "default": null, "description": "Number between 1 to 31 for date of month (both inclusive). If not specified, defaults to Nothing and this field does not take affect.", "title": "Profile Date Of Month" } }, "title": "OperationConfig", "type": "object" }, "ProfilingIsolationLevel": { "enum": [ "AUTOCOMMIT", "READ COMMITTED", "REPEATABLE READ", "SERIALIZABLE" ], "title": "ProfilingIsolationLevel", "type": "string" }, "StatefulStaleMetadataRemovalConfig": { "additionalProperties": false, "description": "Base specialized config for Stateful Ingestion with stale metadata removal capability.", "properties": { "enabled": { "default": false, "description": "Whether or not to enable stateful ingest. Default: True if a pipeline_name is set and either a datahub-rest sink or `datahub_api` is specified, otherwise False", "title": "Enabled", "type": "boolean" }, "remove_stale_metadata": { "default": true, "description": "Soft-deletes the entities present in the last successful run but missing in the current run with stateful_ingestion enabled.", "title": "Remove Stale Metadata", "type": "boolean" }, "fail_safe_threshold": { "default": 75.0, "description": "Prevents large amount of soft deletes & the state from committing from accidental changes to the source configuration if the relative change percent in entities compared to the previous state is above the 'fail_safe_threshold'.", "maximum": 100.0, "minimum": 0.0, "title": "Fail Safe Threshold", "type": "number" } }, "title": "StatefulStaleMetadataRemovalConfig", "type": "object" } }, "additionalProperties": false, "properties": { "bucket_duration": { "$ref": "#/$defs/BucketDuration", "default": "DAY", "description": "Size of the time window to aggregate usage stats." }, "end_time": { "description": "Latest date of lineage/usage to consider. Default: Current time in UTC", "format": "date-time", "title": "End Time", "type": "string" }, "start_time": { "anyOf": [ { "format": "date-time", "type": "string" }, { "pattern": "^\\s*-\\s*\\d+(\\.\\d+)?\\s*(ms|milliseconds?|s|secs?|seconds?|m|mins?|minutes?|h|hrs?|hours?|d|days?|w|weeks?|y|years?)\\s*$", "type": "string" } ], "default": null, "description": "Earliest date of lineage/usage to consider. Default: Last full day in UTC (or hour, depending on `bucket_duration`). You can also specify relative time with respect to end_time such as '-7 days' Or '-7d'.", "title": "Start Time" }, "top_n_queries": { "default": 10, "description": "Number of top queries to save to each table.", "exclusiveMinimum": 0, "title": "Top N Queries", "type": "integer" }, "user_email_pattern": { "$ref": "#/$defs/AllowDenyPattern", "default": { "allow": [ ".*" ], "deny": [], "ignoreCase": true }, "description": "regex patterns for user emails to filter in usage." }, "include_operational_stats": { "default": false, "description": "Generate operation statistics from audit trail data (CREATE, INSERT, UPDATE, DELETE operations).", "title": "Include Operational Stats", "type": "boolean" }, "include_read_operational_stats": { "default": false, "description": "Whether to report read operational stats. Experimental.", "title": "Include Read Operational Stats", "type": "boolean" }, "format_sql_queries": { "default": false, "description": "Whether to format sql queries", "title": "Format Sql Queries", "type": "boolean" }, "include_top_n_queries": { "default": true, "description": "Whether to ingest the top_n_queries.", "title": "Include Top N Queries", "type": "boolean" }, "schema_pattern": { "$ref": "#/$defs/AllowDenyPattern", "default": { "allow": [ ".*" ], "deny": [], "ignoreCase": true }, "description": "Regex patterns for schemas to filter in ingestion. Specify regex to only match the schema name. e.g. to match all tables in schema analytics, use the regex 'analytics'" }, "table_pattern": { "$ref": "#/$defs/AllowDenyPattern", "default": { "allow": [ ".*" ], "deny": [], "ignoreCase": true }, "description": "Regex patterns for tables to filter in ingestion. Specify regex to match the entire table name in database.schema.table format. e.g. to match all tables starting with customer in Customer database and public schema, use the regex 'Customer.public.customer.*'" }, "view_pattern": { "$ref": "#/$defs/AllowDenyPattern", "default": { "allow": [ ".*" ], "deny": [], "ignoreCase": true }, "description": "Regex patterns for views to filter in ingestion. Note: Defaults to table_pattern if not specified. Specify regex to match the entire view name in database.schema.view format. e.g. to match all views starting with customer in Customer database and public schema, use the regex 'Customer.public.customer.*'" }, "classification": { "$ref": "#/$defs/ClassificationConfig", "default": { "enabled": false, "sample_size": 100, "max_workers": 4, "table_pattern": { "allow": [ ".*" ], "deny": [], "ignoreCase": true }, "column_pattern": { "allow": [ ".*" ], "deny": [], "ignoreCase": true }, "info_type_to_term": {}, "classifiers": [ { "config": null, "type": "datahub" } ] }, "description": "For details, refer to [Classification](../../../../metadata-ingestion/docs/dev_guides/classification.md)." }, "incremental_lineage": { "default": false, "description": "When enabled, emits lineage as incremental to existing lineage already in DataHub. When disabled, re-states lineage on each run.", "title": "Incremental Lineage", "type": "boolean" }, "convert_urns_to_lowercase": { "default": false, "description": "Whether to convert dataset urns to lowercase. This value is part of each dataset's URN identity, so it must stay fixed for the life of a deployment. Changing it after data has been ingested re-keys every dataset (e.g. `MyDb.MyTable` becomes `mydb.mytable`); with stateful ingestion enabled the old-cased URNs are then soft-deleted as stale while the new-cased ones are created, producing duplicate or orphaned entities. Pick one value before the first run and leave it unchanged.", "title": "Convert Urns To Lowercase", "type": "boolean" }, "env": { "default": "PROD", "description": "The environment that all assets produced by this connector belong to", "title": "Env", "type": "string" }, "platform_instance": { "anyOf": [ { "type": "string" }, { "type": "null" } ], "default": null, "description": "The instance of the platform that all assets produced by this recipe belong to. This should be unique within the platform. See https://docs.datahub.com/docs/platform-instances/ for more details.", "title": "Platform Instance" }, "stateful_ingestion": { "anyOf": [ { "$ref": "#/$defs/StatefulStaleMetadataRemovalConfig" }, { "type": "null" } ], "default": null }, "options": { "additionalProperties": true, "description": "Any options specified here will be passed to [SQLAlchemy.create_engine](https://docs.sqlalchemy.org/en/14/core/engines.html#sqlalchemy.create_engine) as kwargs. To set connection arguments in the URL, specify them under `connect_args`.", "title": "Options", "type": "object" }, "profile_pattern": { "$ref": "#/$defs/AllowDenyPattern", "default": { "allow": [ ".*" ], "deny": [], "ignoreCase": true }, "description": "Regex patterns to filter tables (or specific columns) for profiling during ingestion. Note that only tables allowed by the `table_pattern` will be considered." }, "domain": { "additionalProperties": { "$ref": "#/$defs/AllowDenyPattern" }, "default": {}, "description": "Attach domains to databases, schemas or tables during ingestion using regex patterns. Domain key can be a guid like *urn:li:domain:ec428203-ce86-4db3-985d-5a8ee6df32ba* or a string like \"Marketing\".) If you provide strings, then datahub will attempt to resolve this name to a guid, and will error out if this fails. There can be multiple domain keys specified.", "title": "Domain", "type": "object" }, "include_views": { "default": true, "description": "Whether views should be ingested.", "title": "Include Views", "type": "boolean" }, "include_tables": { "default": true, "description": "Whether tables should be ingested.", "title": "Include Tables", "type": "boolean" }, "include_table_location_lineage": { "default": true, "description": "If the source supports it, include table lineage to the underlying storage location.", "title": "Include Table Location Lineage", "type": "boolean" }, "include_view_lineage": { "default": true, "description": "Populates view->view and table->view lineage using DataHub's sql parser.", "title": "Include View Lineage", "type": "boolean" }, "include_view_column_lineage": { "default": true, "description": "Populates column-level lineage for view->view and table->view lineage using DataHub's sql parser. Requires `include_view_lineage` to be enabled.", "title": "Include View Column Lineage", "type": "boolean" }, "use_file_backed_cache": { "default": true, "description": "Whether to use a file backed cache for the view definitions.", "title": "Use File Backed Cache", "type": "boolean" }, "profiling": { "$ref": "#/$defs/GEProfilingConfig", "default": { "enabled": false, "operation_config": { "lower_freq_profile_enabled": false, "profile_date_of_month": null, "profile_day_of_week": null }, "limit": null, "offset": null, "profile_table_level_only": false, "include_field_null_count": true, "include_field_distinct_count": true, "include_field_min_value": true, "include_field_max_value": true, "include_field_mean_value": true, "include_field_median_value": true, "include_field_stddev_value": true, "include_field_quantiles": false, "include_field_distinct_value_frequencies": false, "include_field_histogram": false, "include_field_sample_values": true, "max_workers": 20, "report_dropped_profiles": false, "turn_off_expensive_profiling_metrics": false, "field_sample_values_limit": 20, "max_number_of_fields_to_profile": null, "profile_if_updated_since_days": null, "profile_table_size_limit": 5, "profile_table_row_limit": 5000000, "profile_table_row_count_estimate_only": false, "query_combiner_enabled": true, "query_combiner_flatten_enabled": false, "max_distinct_per_statement": 5, "catch_exceptions": true, "profiling_isolation_level": null, "partition_profiling_enabled": true, "partition_datetime": null, "use_sampling": true, "sample_size": 10000, "profile_external_tables": false, "tags_to_ignore_sampling": null, "profile_nested_fields": false, "nested_field_max_depth": 10 } }, "username": { "anyOf": [ { "type": "string" }, { "type": "null" } ], "default": null, "description": "username", "title": "Username" }, "password": { "anyOf": [ { "format": "password", "type": "string", "writeOnly": true }, { "type": "null" } ], "default": null, "description": "password", "title": "Password" }, "host_port": { "description": "host URL", "title": "Host Port", "type": "string" }, "database": { "anyOf": [ { "type": "string" }, { "type": "null" } ], "default": null, "description": "If using, omit `service_name`.", "title": "Database" }, "scheme": { "default": "oracle", "description": "Will be set automatically to default value.", "title": "Scheme", "type": "string" }, "sqlalchemy_uri": { "anyOf": [ { "type": "string" }, { "type": "null" } ], "default": null, "description": "URI of database to connect to. See https://docs.sqlalchemy.org/en/14/core/engines.html#database-urls. Takes precedence over other connection parameters.", "title": "Sqlalchemy Uri" }, "service_name": { "anyOf": [ { "type": "string" }, { "type": "null" } ], "default": null, "description": "Oracle service name. If using, omit `database`.", "title": "Service Name" }, "urn_db_name": { "anyOf": [ { "type": "string" }, { "type": "null" } ], "default": null, "description": "Override the database name used in URN construction. Only relevant when add_database_name_to_urn is true and service_name is used (i.e. database is not set). The connector auto-detects the name by querying sys_context('USERENV','CON_NAME') (the PDB name in multitenant setups) with a fallback to DB_NAME. Set this explicitly if auto-detection returns the wrong value \u2014 for example if your service_name does not route directly to the target PDB. Do not set this alongside database; only one should be used.", "title": "Urn Db Name" }, "add_database_name_to_urn": { "anyOf": [ { "type": "boolean" }, { "type": "null" } ], "default": false, "description": "Include database name in URNs. Default is False (schema.table format). Set to True for database.schema.table format when ingesting from multiple Oracle databases.", "title": "Add Database Name To Urn" }, "data_dictionary_mode": { "$ref": "#/$defs/DataDictionaryMode", "default": "ALL", "description": "The data dictionary views mode, to extract information about schema objects ('ALL' and 'DBA' views are supported). (https://docs.oracle.com/cd/E11882_01/nav/catalog_views.htm)" }, "enable_thick_mode": { "anyOf": [ { "type": "boolean" }, { "type": "null" } ], "default": false, "description": "Connection defaults to thin mode. Set to True to enable thick mode.", "title": "Enable Thick Mode" }, "thick_mode_lib_dir": { "anyOf": [ { "type": "string" }, { "type": "null" } ], "default": null, "description": "Path to the directory containing the Oracle Instant Client libraries. Required on Windows and Mac when enable_thick_mode is true. Optional on Linux: when set, the connector preloads the client libraries from this directory before initializing python-oracledb, which makes thick mode work without needing ldconfig or LD_LIBRARY_PATH to be set (see https://github.com/oracle/python-oracledb/issues/578). When unset on Linux, the standard ldconfig / LD_LIBRARY_PATH search is used.", "title": "Thick Mode Lib Dir" }, "include_stored_procedures": { "default": true, "description": "Include ingest of stored procedures, functions, and packages. Requires access to DBA_PROCEDURES or ALL_PROCEDURES.", "title": "Include Stored Procedures", "type": "boolean" }, "procedure_pattern": { "$ref": "#/$defs/AllowDenyPattern", "default": { "allow": [ ".*" ], "deny": [], "ignoreCase": true }, "description": "Regex patterns for stored procedures to filter in ingestion. Specify regex to match the entire procedure name in database.schema.procedure_name format. e.g. to match all procedures starting with customer in HR schema, use the regex 'ORCL.HR.CUSTOMER.*'" }, "include_lineage": { "default": true, "description": "Enable lineage extraction for stored procedures. When enabled, SQL code in procedures/functions is parsed to extract table dependencies.", "title": "Include Lineage", "type": "boolean" }, "include_materialized_views": { "default": true, "description": "Include materialized views in ingestion. Requires access to DBA_MVIEWS or ALL_MVIEWS. If permission errors occur, you can disable this feature or grant the required permissions.", "title": "Include Materialized Views", "type": "boolean" }, "include_usage_stats": { "default": false, "description": "Generate usage statistics via SQL aggregator. Requires observed queries to be processed.", "title": "Include Usage Stats", "type": "boolean" }, "lazy_schema_resolver": { "default": false, "description": "If enabled, skips the upfront bulk fetch of all known schemas from DataHub when resolving lineage. Useful on large DataHub instances where the bulk fetch causes memory or performance issues.", "title": "Lazy Schema Resolver", "type": "boolean" }, "include_query_usage": { "default": false, "description": "Extract actual query usage from V$SQL for usage statistics. Requires SELECT privilege on V$SQL (or SELECT_CATALOG_ROLE). When enabled, usage statistics will be based on actual queries executed in Oracle.", "title": "Include Query Usage", "type": "boolean" }, "max_queries_to_extract": { "default": 1000, "description": "Maximum number of queries to extract from V$SQL for usage statistics. Queries are ordered by recency (most recent first).", "title": "Max Queries To Extract", "type": "integer" }, "query_exclude_patterns": { "anyOf": [ { "items": { "type": "string" }, "type": "array" }, { "type": "null" } ], "default": null, "description": "Regex patterns for SQL statements to exclude from usage statistics. e.g., ['^SELECT.*FROM SYS\\..*', '^BEGIN.*END;'] to exclude system queries and PL/SQL blocks.", "title": "Query Exclude Patterns" } }, "required": [ "host_port" ], "title": "OracleConfig", "type": "object" } ``` ### Capabilities Use the **Important Capabilities** table above as the source of truth for supported features and whether additional configuration is required. #### URN Format Configuration By default, Oracle URNs are formatted as `schema.table` (e.g., `HR.EMPLOYEES`). **When using `service_name` (recommended):** URNs are always `schema.table`, regardless of configuration. **When using `database` in config:** You can optionally include the database name in URNs by setting `add_database_name_to_urn: true`. This is useful when ingesting from multiple Oracle databases into the same DataHub instance. **URN Format Examples:** - **With `service_name` (any `add_database_name_to_urn` value)**: `urn:li:dataset:(urn:li:dataPlatform:oracle,schema.table,PROD)` - Example: `urn:li:dataset:(urn:li:dataPlatform:oracle,hr.employees,PROD)` - **With `database` in config + `add_database_name_to_urn: true`**: `urn:li:dataset:(urn:li:dataPlatform:oracle,database.schema.table,PROD)` - Example: `urn:li:dataset:(urn:li:dataPlatform:oracle,orcl.hr.employees,PROD)` **Important Notes:** - All assets (tables, views, stored procedures) will use the same URN format for consistent lineage - Once you've ingested with a particular setting, changing it will create new URNs and break existing lineage - The `add_database_name_to_urn` flag only has an effect when you specify `database` in your config (not when using `service_name`) #### Stored Procedures and Functions The Oracle connector ingests both stored procedures and functions as DataJob entities with distinct subtypes: - **Stored Procedures** (object_type = `PROCEDURE`, `PACKAGE`): - Appear with subtype "Stored Procedure" - **Functions** (object_type = `FUNCTION`): - Appear with subtype "Function" **Container Hierarchy:** Stored procedures and functions are organized together in a hierarchy matching tables and views: ``` Database (e.g., ORCL) └── Schema (e.g., HR) ├── Tables and Views └── stored_procedures (Flow container for all procedures and functions) ├── Individual procedures └── Individual functions ``` **Container URN Example:** - All procedures and functions: `urn:li:dataFlow:(oracle,hr.stored_procedures,PROD)` **Benefits:** - Filter by subtype in the DataHub UI to distinguish functions from procedures - Functions and procedures are organized together in the same container (consistent with PostgreSQL, MySQL, and Snowflake) - Consistent browse path hierarchy with tables and views - Full lineage extraction from SQL definitions, including cross-type lineage (e.g., a procedure calling a function) **Note:** Functions return values and are typically used in queries, while procedures perform operations. Both are captured with their distinct subtypes for identification. ### Limitations Module behavior is constrained by source APIs, permissions, and metadata exposed by the platform. Refer to capability notes for unsupported or conditional features. ### Troubleshooting If ingestion fails, validate credentials, permissions, connectivity, and scope filters first. Then review ingestion logs for source-specific errors and adjust configuration accordingly. ### Code Coordinates - Class Name: `datahub.ingestion.source.sql.oracle.OracleSource` - Browse on [GitHub](https://github.com/datahub-project/datahub/blob/master/metadata-ingestion/src/datahub/ingestion/source/sql/oracle.py) :::tip Questions? If you've got any questions on configuring ingestion for Oracle, feel free to ping us on [our Slack](https://datahub.com/slack). ::: :::note 💡 **Contributing to this documentation** This page is auto-generated from the underlying source code. To make changes, please edit the relevant source files in the [metadata-ingestion](https://github.com/datahub-project/datahub/tree/master/metadata-ingestion) directory. **Tip:** For quick typo fixes or documentation updates, you can click the ✏️ **Edit** icon directly in the GitHub UI to open a Pull Request. For larger changes and PR naming conventions, please refer to our [Contributing Guide](/docs/contributing). :::