| Field | Description |
|:--- |:--- |
|
bucket_uri
One of string, null
| Bucket URI used in your project
Default: None
|
|
discover_regions
boolean
| If true, discover available Vertex AI regions per project and scan all.
Default: False
|
|
extract_execution_metrics
boolean
| Extract hyperparameters and metrics from ML Metadata Executions. Useful for training jobs that don't use Experiments but log to ML Metadata.
Default: True
|
|
include_evaluations
boolean
| Ingest model evaluations and evaluation metrics.
Default: True
|
|
include_experiments
boolean
| Ingest experiments and experiment runs.
Default: True
|
|
include_models
boolean
| Ingest models and model versions from the registry.
Default: True
|
|
include_pipelines
boolean
| Ingest pipelines and tasks.
Default: True
|
|
include_training_jobs
boolean
| Ingest training jobs and related run events.
Default: True
|
|
incremental_lineage
boolean
| When enabled, emits lineage as incremental to existing lineage already in DataHub. When disabled, re-states lineage on each run.
Default: False
|
|
max_evaluations_per_model
One of integer, null
| Maximum evaluations per model. Default: 10, Max: 100. Set to None for unlimited (not recommended).
Default: 10
|
|
max_experiments
One of integer, null
| Maximum number of experiments to ingest. Experiments are ordered by update_time descending (most recently updated first). Default: 1000, Max: 10000. Set to None for unlimited (not recommended).
Default: 1000
|
|
max_models
One of integer, null
| Maximum number of models to ingest. Models are ordered by update_time descending (most recently updated first). Default: 10000, Max: 50000. Set to None for unlimited (not recommended).
Default: 10000
|
|
max_runs_per_experiment
One of integer, null
| Maximum experiment runs per experiment. Runs are ordered by update_time descending (most recently updated first). Default: 100, Max: 1000. Set to None for unlimited (not recommended).
Default: 100
|
|
max_training_jobs_per_type
One of integer, null
| Maximum training jobs per type (CustomJob, AutoML, etc.). Jobs are ordered by update_time descending (most recently updated first). Default: 1000, Max: 10000. Set to None for unlimited (not recommended).
Default: 1000
|
|
ml_metadata_max_execution_search_limit
integer
| Maximum number of ML Metadata executions to retrieve when searching for a training job. Ordered by LAST_UPDATE_TIME descending. Lower this if ingestion is slow or timing out.
Default: 500
|
|
normalize_external_dataset_paths
boolean
| Strip partition segments from external dataset paths (GCS/S3/ABS) to create stable dataset URNs. When enabled, 'gs://bucket/data/year=2024/month=01/' becomes 'gs://bucket/data/'. Partition-level information is captured via DataProcessInstance. Default is False for backward compatibility. Will default to True in a future major version.
Default: False
|
|
platform_instance
One of string, null
| The instance of the platform that all assets produced by this recipe belong to. This should be unique within the platform. See https://docs.datahub.com/docs/platform-instances/ for more details.
Default: None
|
|
rate_limit
boolean
| Slow down ingestion to avoid hitting Vertex AI API quota limits. Enable if you see '429 Quota Exceeded' errors during ingestion.
Default: False
|
|
region
One of string, null
| [deprecated] Single Vertex AI region. Prefer 'regions' or 'discover_regions'.
Default: None
|
|
requests_per_min
integer
| Max API requests per minute when rate_limit is enabled. 600 is Google's quota ceiling for resource management requests per project per region (see https://cloud.google.com/vertex-ai/docs/quotas). Lower this only if you share quota with other workloads running in the same project and region.
Default: 600
|
|
use_ml_metadata_for_lineage
boolean
| Extract lineage from Vertex AI ML Metadata API for CustomJob and other training jobs. This enables input dataset → training job → output model lineage for non-AutoML jobs.
Default: True
|
|
vertexai_url
One of string, null
| VertexUI URI
Default: https://console.cloud.google.com/vertex-ai
|
|
env
string
| The environment that all assets produced by this connector belong to
Default: PROD
|
|
credential
One of GCPCredential, null
| GCP credential information
Default: None
|
|
credential.client_email ❓ string
| Client email |
|
string
| Client Id |
|
string(password)
| Private key in a form of '-----BEGIN PRIVATE KEY-----\nprivate-key\n-----END PRIVATE KEY-----\n' |
|
credential.private_key_id ❓ string
| Private key id |
|
credential.auth_provider_x509_cert_url
string
| Auth provider x509 certificate url
Default: https://www.googleapis.com/oauth2/v1/certs
|
|
credential.auth_uri
string
| Authentication uri
Default: https://accounts.google.com/o/oauth2/auth
|
|
credential.client_x509_cert_url
One of string, null
| If not set it will be default to https://www.googleapis.com/robot/v1/metadata/x509/client_email
Default: None
|
|
credential.project_id
One of string, null
| Project id to set the credentials
Default: None
|
|
credential.token_uri
string
| Token uri
Default: https://oauth2.googleapis.com/token
|
|
credential.type
string
| Authentication type
Default: service_account
|
|
experiment_name_pattern
AllowDenyPattern
| A class to store allow deny regexes.
Patterns are matched against the start of the string only, not the entire
string - a pattern does not need to match to the end to be considered a match.
For example, the pattern "prod" matches "prod", "prod_east", and "production".
To require an exact match, anchor your pattern explicitly, e.g. "^prod$". |
|
experiment_name_pattern.ignoreCase
One of boolean, null
| Whether to ignore case sensitivity during pattern matching.
Default: True
|
|
model_name_pattern
AllowDenyPattern
| A class to store allow deny regexes.
Patterns are matched against the start of the string only, not the entire
string - a pattern does not need to match to the end to be considered a match.
For example, the pattern "prod" matches "prod", "prod_east", and "production".
To require an exact match, anchor your pattern explicitly, e.g. "^prod$". |
|
model_name_pattern.ignoreCase
One of boolean, null
| Whether to ignore case sensitivity during pattern matching.
Default: True
|
|
partition_pattern_rules
array
| Regex patterns to identify and strip partition segments from paths. Applied when normalize_external_dataset_paths is enabled. Patterns are applied in order.
Default: ['/[^/]+=([^/]+)', '/dt=\\d{4}-\\d{2}-\\d{2}', '/\...
|
|
partition_pattern_rules.string
string
| |
|
platform_instance_map
map(str,PlatformDetail)
| Platform instance configuration for external datasets referenced in Vertex AI lineage. |
|
platform_instance_map.`key`.platform_instance
One of string, null
| Platform instance for URN generation. If not set, no platform instance in URN.
Default: None
|
|
platform_instance_map.`key`.convert_urns_to_lowercase
boolean
| Convert dataset names to lowercase. Set to true for Snowflake (which defaults to lowercase URNs). Leave false for GCS, BigQuery, S3, and ABS.
Default: False
|
|
platform_instance_map.`key`.env
string
| Environment for all assets from this platform
Default: PROD
|
|
project_id_pattern
AllowDenyPattern
| A class to store allow deny regexes.
Patterns are matched against the start of the string only, not the entire
string - a pattern does not need to match to the end to be considered a match.
For example, the pattern "prod" matches "prod", "prod_east", and "production".
To require an exact match, anchor your pattern explicitly, e.g. "^prod$". |
|
project_id_pattern.ignoreCase
One of boolean, null
| Whether to ignore case sensitivity during pattern matching.
Default: True
|
|
project_ids
array
| Ingest specified GCP project ids. Overrides project_id_pattern. |
|
project_ids.string
string
| |
|
project_labels
array
| Ingest projects with these labels (key:value). Applied before project_id_pattern. |
|
project_labels.string
string
| |
|
regions
array
| List of Vertex AI regions to scan. If empty and discover_regions is false, falls back to 'region'. |
|
regions.string
string
| |
|
training_job_type_pattern
AllowDenyPattern
| A class to store allow deny regexes.
Patterns are matched against the start of the string only, not the entire
string - a pattern does not need to match to the end to be considered a match.
For example, the pattern "prod" matches "prod", "prod_east", and "production".
To require an exact match, anchor your pattern explicitly, e.g. "^prod$". |
|
training_job_type_pattern.ignoreCase
One of boolean, null
| Whether to ignore case sensitivity during pattern matching.
Default: True
|
|
stateful_ingestion
One of StatefulStaleMetadataRemovalConfig, null
| Stateful ingestion configuration for tracking and removing stale metadata.
Default: None
|
|
stateful_ingestion.enabled
boolean
| Whether or not to enable stateful ingest. Default: True if a pipeline_name is set and either a datahub-rest sink or `datahub_api` is specified, otherwise False
Default: False
|
|
stateful_ingestion.fail_safe_threshold
number
| Prevents large amount of soft deletes & the state from committing from accidental changes to the source configuration if the relative change percent in entities compared to the previous state is above the 'fail_safe_threshold'.
Default: 75.0
|
|
stateful_ingestion.remove_stale_metadata
boolean
| Soft-deletes the entities present in the last successful run but missing in the current run with stateful_ingestion enabled.
Default: True
|
