# DataFlow A DataFlow represents a high-level data processing pipeline or workflow orchestrated by systems like Apache Airflow, Azkaban, Prefect, Dagster, or similar workflow management platforms. DataFlows serve as containers for related DataJobs, representing the overall execution context and organization of data processing tasks. ## Identity DataFlows are uniquely identified by three components: - **Orchestrator**: The workflow management platform that executes the flow (e.g., `airflow`, `azkaban`, `prefect`, `dagster`) - **Flow ID**: A unique identifier for the workflow within the orchestrator (typically the DAG name, pipeline name, or workflow identifier) - **Cluster**: The execution environment or cluster where the flow runs (e.g., `prod`, `staging`, `dev`) The URN structure follows this pattern: ``` urn:li:dataFlow:(,,) ``` ### URN Examples **Apache Airflow DAG in production:** ``` urn:li:dataFlow:(airflow,daily_sales_pipeline,prod) ``` **Prefect flow in staging:** ``` urn:li:dataFlow:(prefect,customer_analytics,staging) ``` **Azkaban workflow in development:** ``` urn:li:dataFlow:(azkaban,data_quality_checks,dev) ``` ## Important Capabilities ### Core DataFlow Information DataFlows maintain essential metadata about the pipeline through the `dataFlowInfo` aspect: - **Name**: The display name of the flow (may differ from flow_id) - **Description**: Detailed explanation of what the flow does and its purpose - **Project**: Optional namespace or project the flow belongs to - **Created/Modified Timestamps**: When the flow was created and last modified in the source system - **External URL**: Link to the flow in the orchestration platform's UI - **Custom Properties**: Key-value pairs for additional platform-specific metadata - **Environment**: The fabric type (PROD, DEV, STAGING, etc.) ### Editable Properties The `editableDataFlowProperties` aspect allows users to modify certain properties through the DataHub UI without interfering with ingestion from source systems: - **Description**: User-edited documentation that takes precedence over ingested descriptions This separation ensures that edits made in DataHub are preserved and not overwritten by subsequent ingestion runs. ### Version Information The `versionInfo` aspect tracks versioning details for the flow: - **Version**: An identifier like a git commit hash or md5 hash - **Version Type**: The type of version identifier being used (e.g., "git", "md5") This is particularly useful for tracking changes to pipeline code and correlating pipeline versions with their execution history. ### Relationship with DataJobs DataFlows act as parent entities for DataJobs. Each DataJob's identity includes a reference to its parent DataFlow through the `flow` field in the DataJobKey. This creates a hierarchical relationship: ``` DataFlow (Pipeline) └─ DataJob (Task 1) └─ DataJob (Task 2) └─ DataJob (Task 3) ``` This structure mirrors how workflow orchestrators organize tasks within DAGs or pipelines. ### Incidents Tracking The `incidentsSummary` aspect provides visibility into data quality or operational issues: - **Active Incidents**: Currently unresolved incidents affecting this flow - **Resolved Incidents**: Historical record of incidents that have been addressed - **Incident Details**: Type, priority, creation time, and resolution time for each incident This enables DataHub to serve as a centralized incident management system for data pipelines. ## Code Examples ### Creating a DataFlow **Python SDK: Create a basic DataFlow** ```python # Inlined from /metadata-ingestion/examples/library/dataflow_create.py from datahub.metadata.urns import TagUrn from datahub.sdk import DataFlow, DataHubClient client = DataHubClient.from_env() dataflow = DataFlow( name="example_dataflow", platform="airflow", description="airflow pipeline for production", tags=[TagUrn(name="production"), TagUrn(name="data_engineering")], ) client.entities.upsert(dataflow) ``` ### Creating a DataFlow with Comprehensive Metadata **Python SDK: Create a DataFlow with owners, tags, and custom properties** ```python # Inlined from /metadata-ingestion/examples/library/dataflow_comprehensive.py # metadata-ingestion/examples/library/dataflow_comprehensive.py from datetime import datetime, timezone from datahub.metadata.urns import CorpGroupUrn, CorpUserUrn, GlossaryTermUrn, TagUrn from datahub.sdk import DataFlow, DataHubClient client = DataHubClient.from_env() # Create a DataFlow with comprehensive metadata dataflow = DataFlow( platform="airflow", name="daily_sales_aggregation", display_name="Daily Sales Aggregation Pipeline", platform_instance="PROD-US-EAST", env="PROD", description="Aggregates daily sales data from multiple sources and updates reporting tables", external_url="https://airflow.company.com/dags/daily_sales_aggregation", custom_properties={ "team": "analytics", "schedule": "0 2 * * *", "sla_hours": "4", "priority": "high", }, created=datetime(2024, 1, 15, tzinfo=timezone.utc), last_modified=datetime.now(timezone.utc), subtype="ETL", owners=[ CorpUserUrn("jdoe"), CorpGroupUrn("data-engineering"), ], tags=[ TagUrn(name="production"), TagUrn(name="sales"), TagUrn(name="critical"), ], terms=[ GlossaryTermUrn("Classification.Confidential"), ], domain="urn:li:domain:sales", ) # Upsert the DataFlow client.entities.upsert(dataflow) print(f"Created DataFlow: {dataflow.urn}") print(f"Display Name: {dataflow.display_name}") print(f"Description: {dataflow.description}") print(f"External URL: {dataflow.external_url}") print(f"Custom Properties: {dataflow.custom_properties}") ``` ### Reading DataFlow Metadata **Python SDK: Read a DataFlow and access its properties** ```python # Inlined from /metadata-ingestion/examples/library/dataflow_read.py from datahub.sdk import DataFlowUrn, DataHubClient client = DataHubClient.from_env() # Or get this from the UI (share -> copy urn) and use DataFlowUrn.from_string(...) dataflow_urn = DataFlowUrn("airflow", "example_dataflow_id") dataflow_entity = client.entities.get(dataflow_urn) print("DataFlow name:", dataflow_entity.name) print("DataFlow platform:", dataflow_entity.platform) print("DataFlow description:", dataflow_entity.description) ``` ### Adding Tags and Terms to a DataFlow **Python SDK: Add tags and glossary terms to a DataFlow** ```python # Inlined from /metadata-ingestion/examples/library/dataflow_add_tags_terms.py # metadata-ingestion/examples/library/dataflow_add_tags_terms.py from datahub.metadata.urns import DataFlowUrn, GlossaryTermUrn, TagUrn from datahub.sdk import DataHubClient client = DataHubClient.from_env() # Get the existing DataFlow dataflow_urn = DataFlowUrn("airflow", "daily_sales_pipeline", "prod") dataflow = client.entities.get(dataflow_urn) # Add tags dataflow.add_tag(TagUrn(name="pii")) dataflow.add_tag(TagUrn(name="quarterly-review")) # Add glossary terms dataflow.add_term(GlossaryTermUrn("DataQuality.Validated")) dataflow.add_term(GlossaryTermUrn("BusinessCritical.Revenue")) # Save changes client.entities.upsert(dataflow) print(f"Updated DataFlow: {dataflow.urn}") print(f"Tags: {[str(tag) for tag in dataflow.tags] if dataflow.tags else []}") print(f"Terms: {[str(term) for term in dataflow.terms] if dataflow.terms else []}") ``` ### Adding Ownership to a DataFlow **Python SDK: Set owners for a DataFlow** ```python # Inlined from /metadata-ingestion/examples/library/dataflow_add_ownership.py # metadata-ingestion/examples/library/dataflow_add_ownership.py from datahub.metadata.urns import CorpGroupUrn, CorpUserUrn, DataFlowUrn from datahub.sdk import DataHubClient client = DataHubClient.from_env() # Get the existing DataFlow dataflow_urn = DataFlowUrn("airflow", "daily_sales_pipeline", "prod") dataflow = client.entities.get(dataflow_urn) # Add individual owners dataflow.add_owner((CorpUserUrn("alice"), "DATAOWNER")) dataflow.add_owner((CorpUserUrn("bob"), "DEVELOPER")) # Add group owner dataflow.add_owner((CorpGroupUrn("analytics-team"), "DATAOWNER")) # Save changes client.entities.upsert(dataflow) print(f"Updated DataFlow: {dataflow.urn}") print(f"Owners: {dataflow.owners}") ``` ### Querying DataFlow via REST API DataFlows can be queried using the standard DataHub REST APIs: **REST API: Fetch a DataFlow entity** ```bash # Get a complete DataFlow snapshot curl 'http://localhost:8080/entities/urn%3Ali%3AdataFlow%3A(airflow,daily_sales_pipeline,prod)' ``` Response includes all aspects: ```json { "urn": "urn:li:dataFlow:(airflow,daily_sales_pipeline,prod)", "aspects": { "dataFlowKey": { "orchestrator": "airflow", "flowId": "daily_sales_pipeline", "cluster": "prod" }, "dataFlowInfo": { "name": "Daily Sales Pipeline", "description": "Processes daily sales data and updates aggregates", "project": "analytics", "externalUrl": "https://airflow.company.com/dags/daily_sales_pipeline" }, "ownership": { ... }, "globalTags": { ... } } } ``` ### Creating DataFlow with DataJobs **Python SDK: Create a DataFlow with associated DataJobs** ```python # Inlined from /metadata-ingestion/examples/library/datajob_create_full.py from datahub.metadata.urns import DatasetUrn, TagUrn from datahub.sdk import DataFlow, DataHubClient, DataJob client = DataHubClient.from_env() # datajob will inherit the platform and platform instance from the flow dataflow = DataFlow( platform="airflow", name="example_dag", platform_instance="PROD", description="example dataflow", tags=[TagUrn(name="tag1"), TagUrn(name="tag2")], ) datajob = DataJob( name="example_datajob", flow=dataflow, inlets=[ DatasetUrn(platform="hdfs", name="dataset1", env="PROD"), ], outlets=[ DatasetUrn(platform="hdfs", name="dataset2", env="PROD"), ], ) client.entities.upsert(dataflow) client.entities.upsert(datajob) ``` ## Integration Points ### Relationship with DataJobs DataFlows have a parent-child relationship with DataJobs through the `IsPartOf` relationship. This is the primary integration point: ```python from datahub.sdk import DataFlow, DataJob, DataHubClient # DataJob automatically links to its parent DataFlow flow = DataFlow(platform="airflow", name="my_dag") job = DataJob(name="extract_data", flow=flow) ``` ### Relationship with Datasets While DataFlows don't directly reference datasets, their child DataJobs establish lineage relationships with datasets through: - **Inlets**: Input datasets consumed by jobs - **Outlets**: Output datasets produced by jobs This creates indirect lineage from DataFlows to datasets through their constituent jobs. ### Orchestrator Platform Integration Common orchestrators that produce DataFlow entities: - **Apache Airflow**: Each DAG becomes a DataFlow, tasks become DataJobs - **Prefect**: Flows are DataFlows, tasks are DataJobs - **Dagster**: Jobs/Pipelines are DataFlows, ops/solids are DataJobs - **Azkaban**: Flows are DataFlows, jobs are DataJobs - **Fivetran**: Connectors are represented as DataFlows with sync operations as DataJobs ### DataProcessInstance Tracking DataFlow executions can be tracked using DataProcessInstance entities, which record: - Start and end times of flow runs - Success/failure status - Input and output datasets for specific runs This enables tracking of pipeline run history and troubleshooting failures. ### GraphQL Integration The DataFlow entity is exposed through DataHub's GraphQL API with full support for: - Querying flow metadata - Browsing flows by orchestrator and cluster - Searching flows by name, description, and project - Updating flow properties (ownership, tags, terms, etc.) Key GraphQL resolvers: - `dataFlow`: Fetch a single DataFlow by URN - `searchAcrossEntities`: Search for DataFlows with filters - `updateDataFlow`: Modify DataFlow properties ## Notable Exceptions ### Cluster vs Environment While DataFlows use the `cluster` field in their URN for identification, they also have an `env` field in the `dataFlowInfo` aspect. These serve different purposes: - **Cluster**: Part of the identity, typically matches the environment but can represent specific deployment instances - **Environment (env)**: A semantic indicator of the fabric type (PROD, DEV, STAGING) that's searchable and filterable In most cases, these should align (e.g., cluster="prod" and env="PROD"), but they can diverge when multiple production clusters exist or when representing complex deployment topologies. ### Platform vs Orchestrator Throughout the codebase, you'll see both terms used: - In the URN structure and key aspect, the field is called `orchestrator` - In Python SDK and some APIs, it's referred to as `platform` These are synonymous and refer to the same concept: the workflow management system executing the flow. ### Legacy API vs SDK Two APIs exist for creating DataFlows: 1. **Legacy API** (`datahub.api.entities.datajob.DataFlow`): Uses the older emitter pattern 2. **Modern SDK** (`datahub.sdk.DataFlow`): Preferred approach with cleaner interfaces New code should use the modern SDK (imported from `datahub.sdk`), though both are maintained for backward compatibility. ### DataFlow vs DataPipeline In some contexts, you might see references to "data pipelines" in documentation or UI. These are informal terms that refer to DataFlows. The formal entity type in the metadata model is `dataFlow`, not `dataPipeline`. ## Technical Reference Guide The sections above provide an overview of how to use this entity. The following sections provide detailed technical information about how metadata is stored and represented in DataHub. **Aspects** are the individual pieces of metadata that can be attached to an entity. Each aspect contains specific information (like ownership, tags, or properties) and is stored as a separate record, allowing for flexible and incremental metadata updates. **Relationships** show how this entity connects to other entities in the metadata graph. These connections are derived from the fields within each aspect and form the foundation of DataHub's knowledge graph. ### Reading the Field Tables Each aspect's field table includes an **Annotations** column that provides additional metadata about how fields are used: - **⚠️ Deprecated**: This field is deprecated and may be removed in a future version. Check the description for the recommended alternative - **Searchable**: This field is indexed and can be searched in DataHub's search interface - **Searchable (fieldname)**: When the field name in parentheses is shown, it indicates the field is indexed under a different name in the search index. For example, `dashboardTool` is indexed as `tool` - **→ RelationshipName**: This field creates a relationship to another entity. The arrow indicates this field contains a reference (URN) to another entity, and the name indicates the type of relationship (e.g., `→ Contains`, `→ OwnedBy`) Fields with complex types (like `Edge`, `AuditStamp`) link to their definitions in the [Common Types](#common-types) section below. ### Aspects #### dataFlowKey Python class: `DataFlowKeyClass` Key for a Data Flow #### Fields | Field | Type | Required | Description | Annotations | |-------|------|----------|-------------|-------------| | orchestrator | string | ✓ | Workflow manager like azkaban, airflow which orchestrates the flow | Searchable | | flowId | string | ✓ | Unique Identifier of the data flow | Searchable | | cluster | string | ✓ | Cluster where the flow is executed | Searchable | #### Raw Schema ```javascript { "type": "record", "Aspect": { "name": "dataFlowKey" }, "name": "DataFlowKey", "namespace": "com.linkedin.metadata.key", "fields": [ { "Searchable": { "fieldType": "TEXT_PARTIAL" }, "type": "string", "name": "orchestrator", "doc": "Workflow manager like azkaban, airflow which orchestrates the flow" }, { "Searchable": { "enableAutocomplete": true, "fieldType": "WORD_GRAM" }, "type": "string", "name": "flowId", "doc": "Unique Identifier of the data flow" }, { "Searchable": { "fieldType": "TEXT_PARTIAL" }, "type": "string", "name": "cluster", "doc": "Cluster where the flow is executed" } ], "doc": "Key for a Data Flow" } ``` #### dataFlowInfo Python class: `DataFlowInfoClass` Information about a Data processing flow #### Fields | Field | Type | Required | Description | Annotations | |-------|------|----------|-------------|-------------| | customProperties | map | ✓ | Custom property bag. | Searchable | | externalUrl | string | | URL where the reference exist | Searchable | | name | string | ✓ | Flow name | Searchable | | description | string | | Flow description | Searchable | | project | string | | Optional project/namespace associated with the flow | Searchable | | created | [TimeStamp](#timestamp) | | A timestamp documenting when the asset was created in the source Data Platform (not on DataHub) | Searchable | | lastModified | [TimeStamp](#timestamp) | | A timestamp documenting when the asset was last modified in the source Data Platform (not on Data... | Searchable | | env | FabricType | | Environment for this flow | Searchable | #### Raw Schema ```javascript { "type": "record", "Aspect": { "name": "dataFlowInfo" }, "name": "DataFlowInfo", "namespace": "com.linkedin.datajob", "fields": [ { "Searchable": { "/*": { "fieldType": "TEXT", "queryByDefault": true } }, "type": { "type": "map", "values": "string" }, "name": "customProperties", "default": {}, "doc": "Custom property bag." }, { "Searchable": { "fieldType": "KEYWORD" }, "java": { "class": "com.linkedin.common.url.Url", "coercerClass": "com.linkedin.common.url.UrlCoercer" }, "type": [ "null", "string" ], "name": "externalUrl", "default": null, "doc": "URL where the reference exist" }, { "Searchable": { "boostScore": 10.0, "enableAutocomplete": true, "fieldNameAliases": [ "_entityName" ], "fieldType": "WORD_GRAM", "searchLabel": "entityName", "searchTier": 1 }, "type": "string", "name": "name", "doc": "Flow name" }, { "Searchable": { "fieldType": "TEXT", "hasValuesFieldName": "hasDescription", "sanitizeRichText": true, "searchTier": 2 }, "type": [ "null", "string" ], "name": "description", "default": null, "doc": "Flow description" }, { "Searchable": { "fieldType": "TEXT_PARTIAL", "queryByDefault": false, "searchTier": 3 }, "type": [ "null", "string" ], "name": "project", "default": null, "doc": "Optional project/namespace associated with the flow" }, { "Searchable": { "/time": { "fieldName": "createdAt", "fieldType": "DATETIME", "searchLabel": "createdAt" } }, "type": [ "null", { "type": "record", "name": "TimeStamp", "namespace": "com.linkedin.common", "fields": [ { "type": "long", "name": "time", "doc": "When did the event occur" }, { "java": { "class": "com.linkedin.common.urn.Urn" }, "type": [ "null", "string" ], "name": "actor", "default": null, "doc": "Optional: The actor urn involved in the event." } ], "doc": "A standard event timestamp" } ], "name": "created", "default": null, "doc": "A timestamp documenting when the asset was created in the source Data Platform (not on DataHub)" }, { "Searchable": { "/time": { "fieldName": "lastModifiedAt", "fieldType": "DATETIME", "searchLabel": "lastModifiedAt" } }, "type": [ "null", "com.linkedin.common.TimeStamp" ], "name": "lastModified", "default": null, "doc": "A timestamp documenting when the asset was last modified in the source Data Platform (not on DataHub)" }, { "Searchable": { "addToFilters": true, "fieldType": "KEYWORD", "filterNameOverride": "Environment", "queryByDefault": false }, "type": [ "null", { "type": "enum", "symbolDocs": { "CERT": "Designates certification fabrics", "CORP": "Designates corporation fabrics", "DEV": "Designates development fabrics", "EI": "Designates early-integration fabrics", "NON_PROD": "Designates non-production fabrics", "PRD": "Alternative Prod spelling", "PRE": "Designates pre-production fabrics", "PROD": "Designates production fabrics", "QA": "Designates quality assurance fabrics", "RVW": "Designates review fabrics", "SANDBOX": "Designates sandbox fabrics", "SBX": "Alternative spelling for sandbox", "SIT": "System Integration Testing", "STG": "Designates staging fabrics", "TEST": "Designates testing fabrics", "TST": "Alternative Test spelling", "UAT": "Designates user acceptance testing fabrics" }, "name": "FabricType", "namespace": "com.linkedin.common", "symbols": [ "DEV", "TEST", "QA", "UAT", "EI", "PRE", "STG", "NON_PROD", "PROD", "CORP", "RVW", "PRD", "TST", "SIT", "SBX", "SANDBOX", "CERT" ], "doc": "Fabric group type" } ], "name": "env", "default": null, "doc": "Environment for this flow" } ], "doc": "Information about a Data processing flow" } ``` #### editableDataFlowProperties Python class: `EditableDataFlowPropertiesClass` Stores editable changes made to properties. This separates changes made from ingestion pipelines and edits in the UI to avoid accidental overwrites of user-provided data by ingestion pipelines #### Fields | Field | Type | Required | Description | Annotations | |-------|------|----------|-------------|-------------| | created | [AuditStamp](#auditstamp) | ✓ | An AuditStamp corresponding to the creation of this resource/association/sub-resource. A value of... | | | lastModified | [AuditStamp](#auditstamp) | ✓ | An AuditStamp corresponding to the last modification of this resource/association/sub-resource. I... | | | deleted | [AuditStamp](#auditstamp) | | An AuditStamp corresponding to the deletion of this resource/association/sub-resource. Logically,... | | | description | string | | Edited documentation of the data flow | Searchable (editedDescription) | #### Raw Schema ```javascript { "type": "record", "Aspect": { "name": "editableDataFlowProperties" }, "name": "EditableDataFlowProperties", "namespace": "com.linkedin.datajob", "fields": [ { "type": { "type": "record", "name": "AuditStamp", "namespace": "com.linkedin.common", "fields": [ { "type": "long", "name": "time", "doc": "When did the resource/association/sub-resource move into the specific lifecycle stage represented by this AuditEvent." }, { "java": { "class": "com.linkedin.common.urn.Urn" }, "type": "string", "name": "actor", "doc": "The entity (e.g. a member URN) which will be credited for moving the resource/association/sub-resource into the specific lifecycle stage. It is also the one used to authorize the change." }, { "java": { "class": "com.linkedin.common.urn.Urn" }, "type": [ "null", "string" ], "name": "impersonator", "default": null, "doc": "The entity (e.g. a service URN) which performs the change on behalf of the Actor and must be authorized to act as the Actor." }, { "type": [ "null", "string" ], "name": "message", "default": null, "doc": "Additional context around how DataHub was informed of the particular change. For example: was the change created by an automated process, or manually." } ], "doc": "Data captured on a resource/association/sub-resource level giving insight into when that resource/association/sub-resource moved into a particular lifecycle stage, and who acted to move it into that specific lifecycle stage." }, "name": "created", "default": { "actor": "urn:li:corpuser:unknown", "impersonator": null, "time": 0, "message": null }, "doc": "An AuditStamp corresponding to the creation of this resource/association/sub-resource. A value of 0 for time indicates missing data." }, { "type": "com.linkedin.common.AuditStamp", "name": "lastModified", "default": { "actor": "urn:li:corpuser:unknown", "impersonator": null, "time": 0, "message": null }, "doc": "An AuditStamp corresponding to the last modification of this resource/association/sub-resource. If no modification has happened since creation, lastModified should be the same as created. A value of 0 for time indicates missing data." }, { "type": [ "null", "com.linkedin.common.AuditStamp" ], "name": "deleted", "default": null, "doc": "An AuditStamp corresponding to the deletion of this resource/association/sub-resource. Logically, deleted MUST have a later timestamp than creation. It may or may not have the same time as lastModified depending upon the resource/association/sub-resource semantics." }, { "Searchable": { "fieldName": "editedDescription", "fieldType": "TEXT", "sanitizeRichText": true, "searchTier": 2 }, "type": [ "null", "string" ], "name": "description", "default": null, "doc": "Edited documentation of the data flow" } ], "doc": "Stores editable changes made to properties. This separates changes made from\ningestion pipelines and edits in the UI to avoid accidental overwrites of user-provided data by ingestion pipelines" } ``` #### ownership Python class: `OwnershipClass` Ownership information of an entity. #### Fields | Field | Type | Required | Description | Annotations | |-------|------|----------|-------------|-------------| | owners | Owner[] | ✓ | List of owners of the entity. | | | ownerTypes | map | | Ownership type to Owners map, populated via mutation hook. | Searchable | | lastModified | [AuditStamp](#auditstamp) | ✓ | Audit stamp containing who last modified the record and when. A value of 0 in the time field indi... | | #### Raw Schema ```javascript { "type": "record", "Aspect": { "name": "ownership" }, "name": "Ownership", "namespace": "com.linkedin.common", "fields": [ { "type": { "type": "array", "items": { "type": "record", "name": "Owner", "namespace": "com.linkedin.common", "fields": [ { "Relationship": { "entityTypes": [ "corpuser", "corpGroup" ], "name": "OwnedBy" }, "Searchable": { "addToFilters": true, "fieldName": "owners", "fieldType": "URN", "filterNameOverride": "Owned By", "hasValuesFieldName": "hasOwners", "queryByDefault": false, "searchTier": 2 }, "java": { "class": "com.linkedin.common.urn.Urn" }, "type": "string", "name": "owner", "doc": "Owner URN, e.g. urn:li:corpuser:ldap, urn:li:corpGroup:group_name, and urn:li:multiProduct:mp_name\n(Caveat: only corpuser is currently supported in the frontend.)" }, { "deprecated": true, "type": { "type": "enum", "symbolDocs": { "BUSINESS_OWNER": "A person or group who is responsible for logical, or business related, aspects of the asset.", "CONSUMER": "A person, group, or service that consumes the data\nDeprecated! Use TECHNICAL_OWNER or BUSINESS_OWNER instead.", "CUSTOM": "Set when ownership type is unknown or a when new one is specified as an ownership type entity for which we have no\nenum value for. This is used for backwards compatibility", "DATAOWNER": "A person or group that is owning the data\nDeprecated! Use TECHNICAL_OWNER instead.", "DATA_STEWARD": "A steward, expert, or delegate responsible for the asset.", "DELEGATE": "A person or a group that overseas the operation, e.g. a DBA or SRE.\nDeprecated! Use TECHNICAL_OWNER instead.", "DEVELOPER": "A person or group that is in charge of developing the code\nDeprecated! Use TECHNICAL_OWNER instead.", "NONE": "No specific type associated to the owner.", "PRODUCER": "A person, group, or service that produces/generates the data\nDeprecated! Use TECHNICAL_OWNER instead.", "STAKEHOLDER": "A person or a group that has direct business interest\nDeprecated! Use TECHNICAL_OWNER, BUSINESS_OWNER, or STEWARD instead.", "TECHNICAL_OWNER": "person or group who is responsible for technical aspects of the asset." }, "deprecatedSymbols": { "CONSUMER": true, "DATAOWNER": true, "DELEGATE": true, "DEVELOPER": true, "PRODUCER": true, "STAKEHOLDER": true }, "name": "OwnershipType", "namespace": "com.linkedin.common", "symbols": [ "CUSTOM", "TECHNICAL_OWNER", "BUSINESS_OWNER", "DATA_STEWARD", "NONE", "DEVELOPER", "DATAOWNER", "DELEGATE", "PRODUCER", "CONSUMER", "STAKEHOLDER" ], "doc": "Asset owner types" }, "name": "type", "doc": "The type of the ownership" }, { "Relationship": { "entityTypes": [ "ownershipType" ], "name": "ownershipType" }, "java": { "class": "com.linkedin.common.urn.Urn" }, "type": [ "null", "string" ], "name": "typeUrn", "default": null, "doc": "The type of the ownership\nUrn of type O" }, { "type": [ "null", { "type": "record", "name": "OwnershipSource", "namespace": "com.linkedin.common", "fields": [ { "type": { "type": "enum", "symbolDocs": { "AUDIT": "Auditing system or audit logs", "DATABASE": "Database, e.g. GRANTS table", "FILE_SYSTEM": "File system, e.g. file/directory owner", "ISSUE_TRACKING_SYSTEM": "Issue tracking system, e.g. Jira", "MANUAL": "Manually provided by a user", "OTHER": "Other sources", "SERVICE": "Other ownership-like service, e.g. Nuage, ACL service etc", "SOURCE_CONTROL": "SCM system, e.g. GIT, SVN" }, "name": "OwnershipSourceType", "namespace": "com.linkedin.common", "symbols": [ "AUDIT", "DATABASE", "FILE_SYSTEM", "ISSUE_TRACKING_SYSTEM", "MANUAL", "SERVICE", "SOURCE_CONTROL", "OTHER" ] }, "name": "type", "doc": "The type of the source" }, { "type": [ "null", "string" ], "name": "url", "default": null, "doc": "A reference URL for the source" } ], "doc": "Source/provider of the ownership information" } ], "name": "source", "default": null, "doc": "Source information for the ownership" }, { "Searchable": { "/actor": { "fieldName": "ownerAttributionActors", "fieldType": "URN", "queryByDefault": false }, "/source": { "fieldName": "ownerAttributionSources", "fieldType": "URN", "queryByDefault": false }, "/time": { "fieldName": "ownerAttributionDates", "fieldType": "DATETIME", "queryByDefault": false } }, "type": [ "null", { "type": "record", "name": "MetadataAttribution", "namespace": "com.linkedin.common", "fields": [ { "type": "long", "name": "time", "doc": "When this metadata was updated." }, { "java": { "class": "com.linkedin.common.urn.Urn" }, "type": "string", "name": "actor", "doc": "The entity (e.g. a member URN) responsible for applying the assocated metadata. This can\neither be a user (in case of UI edits) or the datahub system for automation." }, { "java": { "class": "com.linkedin.common.urn.Urn" }, "type": [ "null", "string" ], "name": "source", "default": null, "doc": "The DataHub source responsible for applying the associated metadata. This will only be filled out\nwhen a DataHub source is responsible. This includes the specific metadata test urn, the automation urn." }, { "type": { "type": "map", "values": "string" }, "name": "sourceDetail", "default": {}, "doc": "The details associated with why this metadata was applied. For example, this could include\nthe actual regex rule, sql statement, ingestion pipeline ID, etc." } ], "doc": "Information about who, why, and how this metadata was applied" } ], "name": "attribution", "default": null, "doc": "Information about who, why, and how this metadata was applied" } ], "doc": "Ownership information" } }, "name": "owners", "doc": "List of owners of the entity." }, { "Searchable": { "/$key": { "fieldType": "MAP_ARRAY", "queryByDefault": false } }, "type": [ { "type": "map", "values": { "type": "array", "items": "string" } }, "null" ], "name": "ownerTypes", "default": {}, "doc": "Ownership type to Owners map, populated via mutation hook." }, { "type": { "type": "record", "name": "AuditStamp", "namespace": "com.linkedin.common", "fields": [ { "type": "long", "name": "time", "doc": "When did the resource/association/sub-resource move into the specific lifecycle stage represented by this AuditEvent." }, { "java": { "class": "com.linkedin.common.urn.Urn" }, "type": "string", "name": "actor", "doc": "The entity (e.g. a member URN) which will be credited for moving the resource/association/sub-resource into the specific lifecycle stage. It is also the one used to authorize the change." }, { "java": { "class": "com.linkedin.common.urn.Urn" }, "type": [ "null", "string" ], "name": "impersonator", "default": null, "doc": "The entity (e.g. a service URN) which performs the change on behalf of the Actor and must be authorized to act as the Actor." }, { "type": [ "null", "string" ], "name": "message", "default": null, "doc": "Additional context around how DataHub was informed of the particular change. For example: was the change created by an automated process, or manually." } ], "doc": "Data captured on a resource/association/sub-resource level giving insight into when that resource/association/sub-resource moved into a particular lifecycle stage, and who acted to move it into that specific lifecycle stage." }, "name": "lastModified", "default": { "actor": "urn:li:corpuser:unknown", "impersonator": null, "time": 0, "message": null }, "doc": "Audit stamp containing who last modified the record and when. A value of 0 in the time field indicates missing data." } ], "doc": "Ownership information of an entity." } ``` #### status Python class: `StatusClass` The lifecycle status metadata of an entity, e.g. dataset, metric, feature, etc. This aspect is used to represent soft deletes conventionally. #### Fields | Field | Type | Required | Description | Annotations | |-------|------|----------|-------------|-------------| | removed | boolean | ✓ | Whether the entity has been removed (soft-deleted). Kept for backward compatibility. When lifecyc... | Searchable | | lifecycleStage | string | | The lifecycle stage of the entity, referencing a lifecycleStageType entity. When null, the entity... | Searchable | | lifecycleLastUpdated | [AuditStamp](#auditstamp) | | Attribution for the lifecycle stage transition ��� who moved the entity into its current stage an... | | #### Raw Schema ```javascript { "type": "record", "Aspect": { "name": "status" }, "name": "Status", "namespace": "com.linkedin.common", "fields": [ { "Searchable": { "fieldType": "BOOLEAN" }, "type": "boolean", "name": "removed", "default": false, "doc": "Whether the entity has been removed (soft-deleted).\nKept for backward compatibility. When lifecycleStage is set to a stage\nwith hideInSearch=true, this field is NOT automatically synced \ufffd\ufffd\ufffd the\nsearch layer uses lifecycleStage settings directly." }, { "Searchable": { "fieldType": "KEYWORD", "queryByDefault": false }, "java": { "class": "com.linkedin.common.urn.Urn" }, "type": [ "null", "string" ], "name": "lifecycleStage", "default": null, "doc": "The lifecycle stage of the entity, referencing a lifecycleStageType entity.\nWhen null, the entity is in its default active state (visible in search).\nWhen set, the referenced lifecycle stage's settings determine behavior\n(e.g., hideInSearch=true excludes the entity from default search results).\n\nUsers can override default filtering by explicitly filtering on this field." }, { "type": [ "null", { "type": "record", "name": "AuditStamp", "namespace": "com.linkedin.common", "fields": [ { "type": "long", "name": "time", "doc": "When did the resource/association/sub-resource move into the specific lifecycle stage represented by this AuditEvent." }, { "java": { "class": "com.linkedin.common.urn.Urn" }, "type": "string", "name": "actor", "doc": "The entity (e.g. a member URN) which will be credited for moving the resource/association/sub-resource into the specific lifecycle stage. It is also the one used to authorize the change." }, { "java": { "class": "com.linkedin.common.urn.Urn" }, "type": [ "null", "string" ], "name": "impersonator", "default": null, "doc": "The entity (e.g. a service URN) which performs the change on behalf of the Actor and must be authorized to act as the Actor." }, { "type": [ "null", "string" ], "name": "message", "default": null, "doc": "Additional context around how DataHub was informed of the particular change. For example: was the change created by an automated process, or manually." } ], "doc": "Data captured on a resource/association/sub-resource level giving insight into when that resource/association/sub-resource moved into a particular lifecycle stage, and who acted to move it into that specific lifecycle stage." } ], "name": "lifecycleLastUpdated", "default": null, "doc": "Attribution for the lifecycle stage transition \ufffd\ufffd\ufffd who moved the entity\ninto its current stage and when. Populated automatically by the\nsetLifecycleStage mutation; should be set by any code path that\nwrites the lifecycleStage field." } ], "doc": "The lifecycle status metadata of an entity, e.g. dataset, metric, feature, etc.\nThis aspect is used to represent soft deletes conventionally." } ``` #### globalTags Python class: `GlobalTagsClass` Tag aspect used for applying tags to an entity #### Fields | Field | Type | Required | Description | Annotations | |-------|------|----------|-------------|-------------| | tags | TagAssociation[] | ✓ | Tags associated with a given entity | Searchable, → TaggedWith | #### Raw Schema ```javascript { "type": "record", "Aspect": { "name": "globalTags" }, "name": "GlobalTags", "namespace": "com.linkedin.common", "fields": [ { "Relationship": { "/*/tag": { "entityTypes": [ "tag" ], "name": "TaggedWith" } }, "Searchable": { "/*/tag": { "addToFilters": true, "boostScore": 0.5, "fieldName": "tags", "fieldType": "URN", "filterNameOverride": "Tagged With", "hasValuesFieldName": "hasTags", "queryByDefault": true, "searchTier": 2 } }, "type": { "type": "array", "items": { "type": "record", "name": "TagAssociation", "namespace": "com.linkedin.common", "fields": [ { "java": { "class": "com.linkedin.common.urn.TagUrn" }, "type": "string", "name": "tag", "doc": "Urn of the applied tag" }, { "type": [ "null", "string" ], "name": "context", "default": null, "doc": "Additional context about the association" }, { "Searchable": { "/actor": { "fieldName": "tagAttributionActors", "fieldType": "URN", "queryByDefault": false }, "/source": { "fieldName": "tagAttributionSources", "fieldType": "URN", "queryByDefault": false }, "/time": { "fieldName": "tagAttributionDates", "fieldType": "DATETIME", "queryByDefault": false } }, "type": [ "null", { "type": "record", "name": "MetadataAttribution", "namespace": "com.linkedin.common", "fields": [ { "type": "long", "name": "time", "doc": "When this metadata was updated." }, { "java": { "class": "com.linkedin.common.urn.Urn" }, "type": "string", "name": "actor", "doc": "The entity (e.g. a member URN) responsible for applying the assocated metadata. This can\neither be a user (in case of UI edits) or the datahub system for automation." }, { "java": { "class": "com.linkedin.common.urn.Urn" }, "type": [ "null", "string" ], "name": "source", "default": null, "doc": "The DataHub source responsible for applying the associated metadata. This will only be filled out\nwhen a DataHub source is responsible. This includes the specific metadata test urn, the automation urn." }, { "type": { "type": "map", "values": "string" }, "name": "sourceDetail", "default": {}, "doc": "The details associated with why this metadata was applied. For example, this could include\nthe actual regex rule, sql statement, ingestion pipeline ID, etc." } ], "doc": "Information about who, why, and how this metadata was applied" } ], "name": "attribution", "default": null, "doc": "Information about who, why, and how this metadata was applied" } ], "doc": "Properties of an applied tag. For now, just an Urn. In the future we can extend this with other properties, e.g.\npropagation parameters." } }, "name": "tags", "doc": "Tags associated with a given entity" } ], "doc": "Tag aspect used for applying tags to an entity" } ``` #### browsePaths Python class: `BrowsePathsClass` Shared aspect containing Browse Paths to be indexed for an entity. #### Fields | Field | Type | Required | Description | Annotations | |-------|------|----------|-------------|-------------| | paths | string[] | ✓ | A list of valid browse paths for the entity. Browse paths are expected to be forward slash-separ... | Searchable | #### Raw Schema ```javascript { "type": "record", "Aspect": { "name": "browsePaths" }, "name": "BrowsePaths", "namespace": "com.linkedin.common", "fields": [ { "Searchable": { "/*": { "fieldName": "browsePaths", "fieldType": "BROWSE_PATH" } }, "type": { "type": "array", "items": "string" }, "name": "paths", "doc": "A list of valid browse paths for the entity.\n\nBrowse paths are expected to be forward slash-separated strings. For example: 'prod/snowflake/datasetName'" } ], "doc": "Shared aspect containing Browse Paths to be indexed for an entity." } ``` #### glossaryTerms Python class: `GlossaryTermsClass` Related business terms information #### Fields | Field | Type | Required | Description | Annotations | |-------|------|----------|-------------|-------------| | terms | GlossaryTermAssociation[] | ✓ | The related business terms | | | auditStamp | [AuditStamp](#auditstamp) | ✓ | Audit stamp containing who reported the related business term | | #### Raw Schema ```javascript { "type": "record", "Aspect": { "name": "glossaryTerms" }, "name": "GlossaryTerms", "namespace": "com.linkedin.common", "fields": [ { "type": { "type": "array", "items": { "type": "record", "name": "GlossaryTermAssociation", "namespace": "com.linkedin.common", "fields": [ { "Relationship": { "entityTypes": [ "glossaryTerm" ], "name": "TermedWith" }, "Searchable": { "addToFilters": true, "fieldName": "glossaryTerms", "fieldType": "URN", "filterNameOverride": "Glossary Term", "hasValuesFieldName": "hasGlossaryTerms", "includeSystemModifiedAt": true, "systemModifiedAtFieldName": "termsModifiedAt" }, "java": { "class": "com.linkedin.common.urn.GlossaryTermUrn" }, "type": "string", "name": "urn", "doc": "Urn of the applied glossary term" }, { "java": { "class": "com.linkedin.common.urn.Urn" }, "type": [ "null", "string" ], "name": "actor", "default": null, "doc": "The user URN which will be credited for adding associating this term to the entity" }, { "type": [ "null", "string" ], "name": "context", "default": null, "doc": "Additional context about the association" }, { "Searchable": { "/actor": { "fieldName": "termAttributionActors", "fieldType": "URN", "queryByDefault": false }, "/source": { "fieldName": "termAttributionSources", "fieldType": "URN", "queryByDefault": false }, "/time": { "fieldName": "termAttributionDates", "fieldType": "DATETIME", "queryByDefault": false } }, "type": [ "null", { "type": "record", "name": "MetadataAttribution", "namespace": "com.linkedin.common", "fields": [ { "type": "long", "name": "time", "doc": "When this metadata was updated." }, { "java": { "class": "com.linkedin.common.urn.Urn" }, "type": "string", "name": "actor", "doc": "The entity (e.g. a member URN) responsible for applying the assocated metadata. This can\neither be a user (in case of UI edits) or the datahub system for automation." }, { "java": { "class": "com.linkedin.common.urn.Urn" }, "type": [ "null", "string" ], "name": "source", "default": null, "doc": "The DataHub source responsible for applying the associated metadata. This will only be filled out\nwhen a DataHub source is responsible. This includes the specific metadata test urn, the automation urn." }, { "type": { "type": "map", "values": "string" }, "name": "sourceDetail", "default": {}, "doc": "The details associated with why this metadata was applied. For example, this could include\nthe actual regex rule, sql statement, ingestion pipeline ID, etc." } ], "doc": "Information about who, why, and how this metadata was applied" } ], "name": "attribution", "default": null, "doc": "Information about who, why, and how this metadata was applied" } ], "doc": "Properties of an applied glossary term." } }, "name": "terms", "doc": "The related business terms" }, { "type": { "type": "record", "name": "AuditStamp", "namespace": "com.linkedin.common", "fields": [ { "type": "long", "name": "time", "doc": "When did the resource/association/sub-resource move into the specific lifecycle stage represented by this AuditEvent." }, { "java": { "class": "com.linkedin.common.urn.Urn" }, "type": "string", "name": "actor", "doc": "The entity (e.g. a member URN) which will be credited for moving the resource/association/sub-resource into the specific lifecycle stage. It is also the one used to authorize the change." }, { "java": { "class": "com.linkedin.common.urn.Urn" }, "type": [ "null", "string" ], "name": "impersonator", "default": null, "doc": "The entity (e.g. a service URN) which performs the change on behalf of the Actor and must be authorized to act as the Actor." }, { "type": [ "null", "string" ], "name": "message", "default": null, "doc": "Additional context around how DataHub was informed of the particular change. For example: was the change created by an automated process, or manually." } ], "doc": "Data captured on a resource/association/sub-resource level giving insight into when that resource/association/sub-resource moved into a particular lifecycle stage, and who acted to move it into that specific lifecycle stage." }, "name": "auditStamp", "doc": "Audit stamp containing who reported the related business term" } ], "doc": "Related business terms information" } ``` #### institutionalMemory Python class: `InstitutionalMemoryClass` Institutional memory of an entity. This is a way to link to relevant documentation and provide description of the documentation. Institutional or tribal knowledge is very important for users to leverage the entity. #### Fields | Field | Type | Required | Description | Annotations | |-------|------|----------|-------------|-------------| | elements | InstitutionalMemoryMetadata[] | ✓ | List of records that represent institutional memory of an entity. Each record consists of a link,... | | #### Raw Schema ```javascript { "type": "record", "Aspect": { "name": "institutionalMemory" }, "name": "InstitutionalMemory", "namespace": "com.linkedin.common", "fields": [ { "type": { "type": "array", "items": { "type": "record", "name": "InstitutionalMemoryMetadata", "namespace": "com.linkedin.common", "fields": [ { "java": { "class": "com.linkedin.common.url.Url", "coercerClass": "com.linkedin.common.url.UrlCoercer" }, "type": "string", "name": "url", "doc": "Link to an engineering design document or a wiki page." }, { "type": "string", "name": "description", "doc": "Description of the link." }, { "type": { "type": "record", "name": "AuditStamp", "namespace": "com.linkedin.common", "fields": [ { "type": "long", "name": "time", "doc": "When did the resource/association/sub-resource move into the specific lifecycle stage represented by this AuditEvent." }, { "java": { "class": "com.linkedin.common.urn.Urn" }, "type": "string", "name": "actor", "doc": "The entity (e.g. a member URN) which will be credited for moving the resource/association/sub-resource into the specific lifecycle stage. It is also the one used to authorize the change." }, { "java": { "class": "com.linkedin.common.urn.Urn" }, "type": [ "null", "string" ], "name": "impersonator", "default": null, "doc": "The entity (e.g. a service URN) which performs the change on behalf of the Actor and must be authorized to act as the Actor." }, { "type": [ "null", "string" ], "name": "message", "default": null, "doc": "Additional context around how DataHub was informed of the particular change. For example: was the change created by an automated process, or manually." } ], "doc": "Data captured on a resource/association/sub-resource level giving insight into when that resource/association/sub-resource moved into a particular lifecycle stage, and who acted to move it into that specific lifecycle stage." }, "name": "createStamp", "doc": "Audit stamp associated with creation of this record" }, { "type": [ "null", "com.linkedin.common.AuditStamp" ], "name": "updateStamp", "default": null, "doc": "Audit stamp associated with updation of this record" }, { "type": [ "null", { "type": "record", "name": "InstitutionalMemoryMetadataSettings", "namespace": "com.linkedin.common", "fields": [ { "type": "boolean", "name": "showInAssetPreview", "default": false, "doc": "Show record in asset preview like on entity header and search previews" } ], "doc": "Settings related to a record of InstitutionalMemoryMetadata" } ], "name": "settings", "default": null, "doc": "Settings for this record" } ], "doc": "Metadata corresponding to a record of institutional memory." } }, "name": "elements", "doc": "List of records that represent institutional memory of an entity. Each record consists of a link, description, creator and timestamps associated with that record." } ], "doc": "Institutional memory of an entity. This is a way to link to relevant documentation and provide description of the documentation. Institutional or tribal knowledge is very important for users to leverage the entity." } ``` #### dataPlatformInstance Python class: `DataPlatformInstanceClass` The specific instance of the data platform that this entity belongs to #### Fields | Field | Type | Required | Description | Annotations | |-------|------|----------|-------------|-------------| | platform | string | ✓ | Data Platform | Searchable | | instance | string | | Instance of the data platform (e.g. db instance) | Searchable (platformInstance) | #### Raw Schema ```javascript { "type": "record", "Aspect": { "name": "dataPlatformInstance" }, "name": "DataPlatformInstance", "namespace": "com.linkedin.common", "fields": [ { "Searchable": { "addToFilters": true, "fieldType": "URN", "filterNameOverride": "Platform" }, "java": { "class": "com.linkedin.common.urn.Urn" }, "type": "string", "name": "platform", "doc": "Data Platform" }, { "Searchable": { "addToFilters": true, "fieldName": "platformInstance", "fieldType": "URN", "filterNameOverride": "Platform Instance" }, "java": { "class": "com.linkedin.common.urn.Urn" }, "type": [ "null", "string" ], "name": "instance", "default": null, "doc": "Instance of the data platform (e.g. db instance)" } ], "doc": "The specific instance of the data platform that this entity belongs to" } ``` #### browsePathsV2 Python class: `BrowsePathsV2Class` Shared aspect containing a Browse Path to be indexed for an entity. #### Fields | Field | Type | Required | Description | Annotations | |-------|------|----------|-------------|-------------| | path | BrowsePathEntry[] | ✓ | A valid browse path for the entity. This field is provided by DataHub by default. This aspect is ... | Searchable | #### Raw Schema ```javascript { "type": "record", "Aspect": { "name": "browsePathsV2" }, "name": "BrowsePathsV2", "namespace": "com.linkedin.common", "fields": [ { "Searchable": { "/*/id": { "fieldName": "browsePathV2", "fieldType": "BROWSE_PATH_V2" } }, "type": { "type": "array", "items": { "type": "record", "name": "BrowsePathEntry", "namespace": "com.linkedin.common", "fields": [ { "type": "string", "name": "id", "doc": "The ID of the browse path entry. This is what gets stored in the index.\nIf there's an urn associated with this entry, id and urn will be the same" }, { "java": { "class": "com.linkedin.common.urn.Urn" }, "type": [ "null", "string" ], "name": "urn", "default": null, "doc": "Optional urn pointing to some entity in DataHub" } ], "doc": "Represents a single level in an entity's browsePathV2" } }, "name": "path", "doc": "A valid browse path for the entity. This field is provided by DataHub by default.\nThis aspect is a newer version of browsePaths where we can encode more information in the path.\nThis path is also based on containers for a given entity if it has containers.\n\nThis is stored in elasticsearch as unit-separator delimited strings and only includes platform specific folders or containers.\nThese paths should not include high level info captured elsewhere ie. Platform and Environment." } ], "doc": "Shared aspect containing a Browse Path to be indexed for an entity." } ``` #### domains Python class: `DomainsClass` Links from an Asset to its Domains #### Fields | Field | Type | Required | Description | Annotations | |-------|------|----------|-------------|-------------| | domains | string[] | ✓ | The Domains attached to an Asset | Searchable, → AssociatedWith | | domainAssociations | DomainAssociation[] | | Additional per-domain association metadata such as attribution and propagation source. A superset... | | #### Raw Schema ```javascript { "type": "record", "Aspect": { "name": "domains", "schemaVersion": 2 }, "name": "Domains", "namespace": "com.linkedin.domain", "fields": [ { "Relationship": { "/*": { "entityTypes": [ "domain" ], "name": "AssociatedWith" } }, "Searchable": { "/*": { "addToFilters": true, "fieldName": "domains", "fieldType": "URN", "filterNameOverride": "Domain", "hasValuesFieldName": "hasDomain" } }, "type": { "type": "array", "items": "string" }, "name": "domains", "doc": "The Domains attached to an Asset" }, { "type": [ "null", { "type": "array", "items": { "type": "record", "name": "DomainAssociation", "namespace": "com.linkedin.domain", "fields": [ { "java": { "class": "com.linkedin.common.urn.Urn" }, "type": "string", "name": "domain", "doc": "Urn of the associated domain. Corresponds to an entry in the parallel domains array." }, { "type": [ "null", "string" ], "name": "context", "default": null, "doc": "Additional context about the association" }, { "Searchable": { "/actor": { "fieldName": "domainAttributionActors", "fieldType": "URN", "queryByDefault": false }, "/source": { "fieldName": "domainAttributionSources", "fieldType": "URN", "queryByDefault": false }, "/time": { "fieldName": "domainAttributionDates", "fieldType": "DATETIME", "queryByDefault": false } }, "type": [ "null", { "type": "record", "name": "MetadataAttribution", "namespace": "com.linkedin.common", "fields": [ { "type": "long", "name": "time", "doc": "When this metadata was updated." }, { "java": { "class": "com.linkedin.common.urn.Urn" }, "type": "string", "name": "actor", "doc": "The entity (e.g. a member URN) responsible for applying the assocated metadata. This can\neither be a user (in case of UI edits) or the datahub system for automation." }, { "java": { "class": "com.linkedin.common.urn.Urn" }, "type": [ "null", "string" ], "name": "source", "default": null, "doc": "The DataHub source responsible for applying the associated metadata. This will only be filled out\nwhen a DataHub source is responsible. This includes the specific metadata test urn, the automation urn." }, { "type": { "type": "map", "values": "string" }, "name": "sourceDetail", "default": {}, "doc": "The details associated with why this metadata was applied. For example, this could include\nthe actual regex rule, sql statement, ingestion pipeline ID, etc." } ], "doc": "Information about who, why, and how this metadata was applied" } ], "name": "attribution", "default": null, "doc": "Information about who, why, and how this domain was applied.\nsourceDetail may carry flags such as 'propagated'='true' when set via glossary tree propagation." } ], "doc": "Properties of an applied domain association." } } ], "name": "domainAssociations", "default": null, "doc": "Additional per-domain association metadata such as attribution and propagation source.\nA superset of the domains field; entries correspond by domain URN.\nInitial migration handled by the DomainsMigrationMutator;\nthe two fields are kept in sync via the DomainsSyncMutationHook." } ], "doc": "Links from an Asset to its Domains" } ``` #### dataProducts Python class: `DataProductsClass` Links from an Asset to the Data Products it belongs to. This aspect is a denormalized, search-only mirror of the authoritative membership stored on the Data Product side (dataProductProperties.assets). It is maintained by DataProductAssetsSideEffect so that assets can be filtered and faceted by Data Product in normal search, the same way domains and applications work. Unlike those aspects there is intentionally no @Relationship here: the authoritative graph edge (DataProductContains) already exists from the Data Product side, and a second reverse edge would double-count. #### Fields | Field | Type | Required | Description | Annotations | |-------|------|----------|-------------|-------------| | dataProducts | DataProductAssociation[] | ✓ | The Data Products this asset belongs to, including per-membership fields such as outputPort. | Searchable | #### Raw Schema ```javascript { "type": "record", "Aspect": { "name": "dataProducts", "schemaVersion": 1 }, "name": "DataProducts", "namespace": "com.linkedin.dataproduct", "fields": [ { "Searchable": { "/*/destinationUrn": { "addToFilters": true, "fieldName": "dataProduct", "fieldType": "URN", "filterNameOverride": "Data Product", "hasValuesFieldName": "hasDataProduct" }, "/*/outputPort": { "addToFilters": true, "fieldName": "isDataProductOutputPort", "fieldType": "BOOLEAN", "filterNameOverride": "Data Product Output Port", "queryByDefault": false } }, "type": { "type": "array", "items": { "type": "record", "name": "DataProductAssociation", "namespace": "com.linkedin.dataproduct", "fields": [ { "java": { "class": "com.linkedin.common.urn.Urn" }, "type": [ "null", "string" ], "name": "sourceUrn", "default": null, "doc": "Urn of the source of this relationship edge.\nIf not specified, assumed to be the entity that this aspect belongs to." }, { "java": { "class": "com.linkedin.common.urn.Urn" }, "type": "string", "name": "destinationUrn", "doc": "Urn of the destination of this relationship edge." }, { "type": [ "null", { "type": "record", "name": "AuditStamp", "namespace": "com.linkedin.common", "fields": [ { "type": "long", "name": "time", "doc": "When did the resource/association/sub-resource move into the specific lifecycle stage represented by this AuditEvent." }, { "java": { "class": "com.linkedin.common.urn.Urn" }, "type": "string", "name": "actor", "doc": "The entity (e.g. a member URN) which will be credited for moving the resource/association/sub-resource into the specific lifecycle stage. It is also the one used to authorize the change." }, { "java": { "class": "com.linkedin.common.urn.Urn" }, "type": [ "null", "string" ], "name": "impersonator", "default": null, "doc": "The entity (e.g. a service URN) which performs the change on behalf of the Actor and must be authorized to act as the Actor." }, { "type": [ "null", "string" ], "name": "message", "default": null, "doc": "Additional context around how DataHub was informed of the particular change. For example: was the change created by an automated process, or manually." } ], "doc": "Data captured on a resource/association/sub-resource level giving insight into when that resource/association/sub-resource moved into a particular lifecycle stage, and who acted to move it into that specific lifecycle stage." } ], "name": "created", "default": null, "doc": "Audit stamp containing who created this relationship edge and when" }, { "type": [ "null", "com.linkedin.common.AuditStamp" ], "name": "lastModified", "default": null, "doc": "Audit stamp containing who last modified this relationship edge and when" }, { "type": [ "null", { "type": "map", "values": "string" } ], "name": "properties", "default": null, "doc": "A generic properties bag that allows us to store specific information on this graph edge." }, { "type": "boolean", "name": "outputPort", "default": false, "doc": "If set to true, this asset is an output port of the Data Product." } ], "doc": "Represents an association of assets to a Data Product." } }, "name": "dataProducts", "doc": "The Data Products this asset belongs to, including per-membership fields such as outputPort." } ], "doc": "Links from an Asset to the Data Products it belongs to.\n\nThis aspect is a denormalized, search-only mirror of the authoritative membership stored on the\nData Product side (dataProductProperties.assets). It is maintained by DataProductAssetsSideEffect\nso that assets can be filtered and faceted by Data Product in normal search, the same way domains\nand applications work. Unlike those aspects there is intentionally no @Relationship here: the\nauthoritative graph edge (DataProductContains) already exists from the Data Product side, and a\nsecond reverse edge would double-count." } ``` #### applications Python class: `ApplicationsClass` Links from an Asset to its Applications #### Fields | Field | Type | Required | Description | Annotations | |-------|------|----------|-------------|-------------| | applications | string[] | ✓ | The Applications attached to an Asset | Searchable, → AssociatedWith | #### Raw Schema ```javascript { "type": "record", "Aspect": { "name": "applications" }, "name": "Applications", "namespace": "com.linkedin.application", "fields": [ { "Relationship": { "/*": { "entityTypes": [ "application" ], "name": "AssociatedWith" } }, "Searchable": { "/*": { "addToFilters": true, "fieldName": "applications", "fieldType": "URN", "filterNameOverride": "Application", "hasValuesFieldName": "hasApplication" } }, "type": { "type": "array", "items": "string" }, "name": "applications", "doc": "The Applications attached to an Asset" } ], "doc": "Links from an Asset to its Applications" } ``` #### deprecation Python class: `DeprecationClass` Deprecation status of an entity #### Fields | Field | Type | Required | Description | Annotations | |-------|------|----------|-------------|-------------| | deprecated | boolean | ✓ | Whether the entity is deprecated. | Searchable | | decommissionTime | long | | The time user plan to decommission this entity. | | | note | string | ✓ | Additional information about the entity deprecation plan, such as the wiki, doc, RB. | | | actor | string | ✓ | The user URN which will be credited for modifying this deprecation content. | | | replacement | string | | | | #### Raw Schema ```javascript { "type": "record", "Aspect": { "name": "deprecation" }, "name": "Deprecation", "namespace": "com.linkedin.common", "fields": [ { "Searchable": { "addToFilters": true, "fieldType": "BOOLEAN", "filterNameOverride": "Deprecated", "weightsPerFieldValue": { "true": 0.5 } }, "type": "boolean", "name": "deprecated", "doc": "Whether the entity is deprecated." }, { "type": [ "null", "long" ], "name": "decommissionTime", "default": null, "doc": "The time user plan to decommission this entity." }, { "type": "string", "name": "note", "doc": "Additional information about the entity deprecation plan, such as the wiki, doc, RB." }, { "java": { "class": "com.linkedin.common.urn.Urn" }, "type": "string", "name": "actor", "doc": "The user URN which will be credited for modifying this deprecation content." }, { "java": { "class": "com.linkedin.common.urn.Urn" }, "type": [ "null", "string" ], "name": "replacement", "default": null } ], "doc": "Deprecation status of an entity" } ``` #### versionInfo Python class: `VersionInfoClass` Information about a Data processing job #### Fields | Field | Type | Required | Description | Annotations | |-------|------|----------|-------------|-------------| | customProperties | map | ✓ | Custom property bag. | Searchable | | externalUrl | string | | URL where the reference exist | Searchable | | version | string | ✓ | The version which can indentify a job version like a commit hash or md5 hash | | | versionType | string | ✓ | The type of the version like git hash or md5 hash | | #### Raw Schema ```javascript { "type": "record", "Aspect": { "name": "versionInfo" }, "name": "VersionInfo", "namespace": "com.linkedin.datajob", "fields": [ { "Searchable": { "/*": { "fieldType": "TEXT", "queryByDefault": true } }, "type": { "type": "map", "values": "string" }, "name": "customProperties", "default": {}, "doc": "Custom property bag." }, { "Searchable": { "fieldType": "KEYWORD" }, "java": { "class": "com.linkedin.common.url.Url", "coercerClass": "com.linkedin.common.url.UrlCoercer" }, "type": [ "null", "string" ], "name": "externalUrl", "default": null, "doc": "URL where the reference exist" }, { "type": "string", "name": "version", "doc": "The version which can indentify a job version like a commit hash or md5 hash" }, { "type": "string", "name": "versionType", "doc": "The type of the version like git hash or md5 hash" } ], "doc": "Information about a Data processing job" } ``` #### container Python class: `ContainerClass` Link from an asset to its parent container #### Fields | Field | Type | Required | Description | Annotations | |-------|------|----------|-------------|-------------| | container | string | ✓ | The parent container of an asset | Searchable, → IsPartOf | #### Raw Schema ```javascript { "type": "record", "Aspect": { "name": "container" }, "name": "Container", "namespace": "com.linkedin.container", "fields": [ { "Relationship": { "entityTypes": [ "container" ], "name": "IsPartOf" }, "Searchable": { "addToFilters": true, "fieldName": "container", "fieldType": "URN", "filterNameOverride": "Container", "hasValuesFieldName": "hasContainer" }, "java": { "class": "com.linkedin.common.urn.Urn" }, "type": "string", "name": "container", "doc": "The parent container of an asset" } ], "doc": "Link from an asset to its parent container" } ``` #### structuredProperties Python class: `StructuredPropertiesClass` Properties about an entity governed by StructuredPropertyDefinition #### Fields | Field | Type | Required | Description | Annotations | |-------|------|----------|-------------|-------------| | properties | StructuredPropertyValueAssignment[] | ✓ | Custom property bag. | | #### Raw Schema ```javascript { "type": "record", "Aspect": { "name": "structuredProperties" }, "name": "StructuredProperties", "namespace": "com.linkedin.structured", "fields": [ { "type": { "type": "array", "items": { "type": "record", "name": "StructuredPropertyValueAssignment", "namespace": "com.linkedin.structured", "fields": [ { "java": { "class": "com.linkedin.common.urn.Urn" }, "type": "string", "name": "propertyUrn", "doc": "The property that is being assigned a value." }, { "type": { "type": "array", "items": [ "string", "double" ] }, "name": "values", "doc": "The value assigned to the property." }, { "type": [ "null", { "type": "record", "name": "AuditStamp", "namespace": "com.linkedin.common", "fields": [ { "type": "long", "name": "time", "doc": "When did the resource/association/sub-resource move into the specific lifecycle stage represented by this AuditEvent." }, { "java": { "class": "com.linkedin.common.urn.Urn" }, "type": "string", "name": "actor", "doc": "The entity (e.g. a member URN) which will be credited for moving the resource/association/sub-resource into the specific lifecycle stage. It is also the one used to authorize the change." }, { "java": { "class": "com.linkedin.common.urn.Urn" }, "type": [ "null", "string" ], "name": "impersonator", "default": null, "doc": "The entity (e.g. a service URN) which performs the change on behalf of the Actor and must be authorized to act as the Actor." }, { "type": [ "null", "string" ], "name": "message", "default": null, "doc": "Additional context around how DataHub was informed of the particular change. For example: was the change created by an automated process, or manually." } ], "doc": "Data captured on a resource/association/sub-resource level giving insight into when that resource/association/sub-resource moved into a particular lifecycle stage, and who acted to move it into that specific lifecycle stage." } ], "name": "created", "default": null, "doc": "Audit stamp containing who created this relationship edge and when" }, { "type": [ "null", "com.linkedin.common.AuditStamp" ], "name": "lastModified", "default": null, "doc": "Audit stamp containing who last modified this relationship edge and when" }, { "Searchable": { "/actor": { "fieldName": "structuredPropertyAttributionActors", "fieldType": "URN", "queryByDefault": false }, "/source": { "fieldName": "structuredPropertyAttributionSources", "fieldType": "URN", "queryByDefault": false }, "/time": { "fieldName": "structuredPropertyAttributionDates", "fieldType": "DATETIME", "queryByDefault": false } }, "type": [ "null", { "type": "record", "name": "MetadataAttribution", "namespace": "com.linkedin.common", "fields": [ { "type": "long", "name": "time", "doc": "When this metadata was updated." }, { "java": { "class": "com.linkedin.common.urn.Urn" }, "type": "string", "name": "actor", "doc": "The entity (e.g. a member URN) responsible for applying the assocated metadata. This can\neither be a user (in case of UI edits) or the datahub system for automation." }, { "java": { "class": "com.linkedin.common.urn.Urn" }, "type": [ "null", "string" ], "name": "source", "default": null, "doc": "The DataHub source responsible for applying the associated metadata. This will only be filled out\nwhen a DataHub source is responsible. This includes the specific metadata test urn, the automation urn." }, { "type": { "type": "map", "values": "string" }, "name": "sourceDetail", "default": {}, "doc": "The details associated with why this metadata was applied. For example, this could include\nthe actual regex rule, sql statement, ingestion pipeline ID, etc." } ], "doc": "Information about who, why, and how this metadata was applied" } ], "name": "attribution", "default": null, "doc": "Information about who, why, and how this metadata was applied" } ] } }, "name": "properties", "doc": "Custom property bag." } ], "doc": "Properties about an entity governed by StructuredPropertyDefinition" } ``` #### incidentsSummary Python class: `IncidentsSummaryClass` Summary related incidents on an entity. #### Fields | Field | Type | Required | Description | Annotations | |-------|------|----------|-------------|-------------| | resolvedIncidents | string[] | ✓ | Resolved incidents for an asset Deprecated! Use the richer resolvedIncidentsDetails instead. | ⚠️ Deprecated | | activeIncidents | string[] | ✓ | Active incidents for an asset Deprecated! Use the richer activeIncidentsDetails instead. | ⚠️ Deprecated | | resolvedIncidentDetails | [IncidentSummaryDetails](#incidentsummarydetails)[] | ✓ | Summary details about the set of resolved incidents | Searchable, → ResolvedIncidents | | activeIncidentDetails | [IncidentSummaryDetails](#incidentsummarydetails)[] | ✓ | Summary details about the set of active incidents | Searchable, → ActiveIncidents | #### Raw Schema ```javascript { "type": "record", "Aspect": { "name": "incidentsSummary" }, "name": "IncidentsSummary", "namespace": "com.linkedin.common", "fields": [ { "deprecated": true, "type": { "type": "array", "items": "string" }, "name": "resolvedIncidents", "default": [], "doc": "Resolved incidents for an asset\nDeprecated! Use the richer resolvedIncidentsDetails instead." }, { "deprecated": true, "type": { "type": "array", "items": "string" }, "name": "activeIncidents", "default": [], "doc": "Active incidents for an asset\nDeprecated! Use the richer activeIncidentsDetails instead." }, { "Relationship": { "/*/urn": { "entityTypes": [ "incident" ], "name": "ResolvedIncidents" } }, "Searchable": { "/*/createdAt": { "fieldName": "resolvedIncidentCreatedTimes", "fieldType": "DATETIME" }, "/*/priority": { "fieldName": "resolvedIncidentPriorities", "fieldType": "COUNT" }, "/*/resolvedAt": { "fieldName": "resolvedIncidentResolvedTimes", "fieldType": "DATETIME" }, "/*/type": { "fieldName": "resolvedIncidentTypes", "fieldType": "KEYWORD" }, "/*/urn": { "fieldName": "resolvedIncidents", "fieldType": "URN", "hasValuesFieldName": "hasResolvedIncidents", "numValuesFieldName": "numResolvedIncidents", "queryByDefault": false } }, "type": { "type": "array", "items": { "type": "record", "name": "IncidentSummaryDetails", "namespace": "com.linkedin.common", "fields": [ { "java": { "class": "com.linkedin.common.urn.Urn" }, "type": "string", "name": "urn", "doc": "The urn of the incident" }, { "type": "string", "name": "type", "doc": "The type of an incident" }, { "type": "long", "name": "createdAt", "doc": "The time at which the incident was raised in milliseconds since epoch." }, { "type": [ "null", "long" ], "name": "resolvedAt", "default": null, "doc": "The time at which the incident was marked as resolved in milliseconds since epoch. Null if the incident is still active." }, { "type": [ "null", "int" ], "name": "priority", "default": null, "doc": "The priority of the incident" } ], "doc": "Summary statistics about incidents on an entity." } }, "name": "resolvedIncidentDetails", "default": [], "doc": "Summary details about the set of resolved incidents" }, { "Relationship": { "/*/urn": { "entityTypes": [ "incident" ], "name": "ActiveIncidents" } }, "Searchable": { "/*/createdAt": { "fieldName": "activeIncidentCreatedTimes", "fieldType": "DATETIME" }, "/*/priority": { "fieldName": "activeIncidentPriorities", "fieldType": "COUNT" }, "/*/type": { "fieldName": "activeIncidentTypes", "fieldType": "KEYWORD" }, "/*/urn": { "addHasValuesToFilters": true, "fieldName": "activeIncidents", "fieldType": "URN", "hasValuesFieldName": "hasActiveIncidents", "numValuesFieldName": "numActiveIncidents", "queryByDefault": false } }, "type": { "type": "array", "items": "com.linkedin.common.IncidentSummaryDetails" }, "name": "activeIncidentDetails", "default": [], "doc": "Summary details about the set of active incidents" } ], "doc": "Summary related incidents on an entity." } ``` #### forms Python class: `FormsClass` Forms that are assigned to this entity to be filled out #### Fields | Field | Type | Required | Description | Annotations | |-------|------|----------|-------------|-------------| | incompleteForms | [FormAssociation](#formassociation)[] | ✓ | All incomplete forms assigned to the entity. | Searchable | | completedForms | [FormAssociation](#formassociation)[] | ✓ | All complete forms assigned to the entity. | Searchable | | verifications | FormVerificationAssociation[] | ✓ | Verifications that have been applied to the entity via completed forms. | Searchable | #### Raw Schema ```javascript { "type": "record", "Aspect": { "name": "forms" }, "name": "Forms", "namespace": "com.linkedin.common", "fields": [ { "Searchable": { "/*/completedPrompts/*/id": { "fieldName": "incompleteFormsCompletedPromptIds", "fieldType": "KEYWORD", "queryByDefault": false }, "/*/completedPrompts/*/lastModified/time": { "fieldName": "incompleteFormsCompletedPromptResponseTimes", "fieldType": "DATETIME", "queryByDefault": false }, "/*/incompletePrompts/*/id": { "fieldName": "incompleteFormsIncompletePromptIds", "fieldType": "KEYWORD", "queryByDefault": false }, "/*/urn": { "fieldName": "incompleteForms", "fieldType": "URN", "queryByDefault": false } }, "type": { "type": "array", "items": { "type": "record", "name": "FormAssociation", "namespace": "com.linkedin.common", "fields": [ { "java": { "class": "com.linkedin.common.urn.Urn" }, "type": "string", "name": "urn", "doc": "Urn of the applied form" }, { "type": { "type": "array", "items": { "type": "record", "name": "FormPromptAssociation", "namespace": "com.linkedin.common", "fields": [ { "type": "string", "name": "id", "doc": "The id for the prompt. This must be GLOBALLY UNIQUE." }, { "type": { "type": "record", "name": "AuditStamp", "namespace": "com.linkedin.common", "fields": [ { "type": "long", "name": "time", "doc": "When did the resource/association/sub-resource move into the specific lifecycle stage represented by this AuditEvent." }, { "java": { "class": "com.linkedin.common.urn.Urn" }, "type": "string", "name": "actor", "doc": "The entity (e.g. a member URN) which will be credited for moving the resource/association/sub-resource into the specific lifecycle stage. It is also the one used to authorize the change." }, { "java": { "class": "com.linkedin.common.urn.Urn" }, "type": [ "null", "string" ], "name": "impersonator", "default": null, "doc": "The entity (e.g. a service URN) which performs the change on behalf of the Actor and must be authorized to act as the Actor." }, { "type": [ "null", "string" ], "name": "message", "default": null, "doc": "Additional context around how DataHub was informed of the particular change. For example: was the change created by an automated process, or manually." } ], "doc": "Data captured on a resource/association/sub-resource level giving insight into when that resource/association/sub-resource moved into a particular lifecycle stage, and who acted to move it into that specific lifecycle stage." }, "name": "lastModified", "doc": "The last time this prompt was touched for the entity (set, unset)" }, { "type": [ "null", { "type": "record", "name": "FormPromptFieldAssociations", "namespace": "com.linkedin.common", "fields": [ { "type": [ "null", { "type": "array", "items": { "type": "record", "name": "FieldFormPromptAssociation", "namespace": "com.linkedin.common", "fields": [ { "type": "string", "name": "fieldPath", "doc": "The field path on a schema field." }, { "type": "com.linkedin.common.AuditStamp", "name": "lastModified", "doc": "The last time this prompt was touched for the field on the entity (set, unset)" } ], "doc": "Information about the status of a particular prompt for a specific schema field\non an entity." } } ], "name": "completedFieldPrompts", "default": null, "doc": "A list of field-level prompt associations that are not yet complete for this form." }, { "type": [ "null", { "type": "array", "items": "com.linkedin.common.FieldFormPromptAssociation" } ], "name": "incompleteFieldPrompts", "default": null, "doc": "A list of field-level prompt associations that are complete for this form." } ], "doc": "Information about the field-level prompt associations on a top-level prompt association." } ], "name": "fieldAssociations", "default": null, "doc": "Optional information about the field-level prompt associations." } ], "doc": "Information about the status of a particular prompt.\nNote that this is where we can add additional information about individual responses:\nactor, timestamp, and the response itself." } }, "name": "incompletePrompts", "default": [], "doc": "A list of prompts that are not yet complete for this form." }, { "type": { "type": "array", "items": "com.linkedin.common.FormPromptAssociation" }, "name": "completedPrompts", "default": [], "doc": "A list of prompts that have been completed for this form." } ], "doc": "Properties of an applied form." } }, "name": "incompleteForms", "doc": "All incomplete forms assigned to the entity." }, { "Searchable": { "/*/completedPrompts/*/id": { "fieldName": "completedFormsCompletedPromptIds", "fieldType": "KEYWORD", "queryByDefault": false }, "/*/completedPrompts/*/lastModified/time": { "fieldName": "completedFormsCompletedPromptResponseTimes", "fieldType": "DATETIME", "queryByDefault": false }, "/*/incompletePrompts/*/id": { "fieldName": "completedFormsIncompletePromptIds", "fieldType": "KEYWORD", "queryByDefault": false }, "/*/urn": { "fieldName": "completedForms", "fieldType": "URN", "queryByDefault": false } }, "type": { "type": "array", "items": "com.linkedin.common.FormAssociation" }, "name": "completedForms", "doc": "All complete forms assigned to the entity." }, { "Searchable": { "/*/form": { "fieldName": "verifiedForms", "fieldType": "URN", "queryByDefault": false } }, "type": { "type": "array", "items": { "type": "record", "name": "FormVerificationAssociation", "namespace": "com.linkedin.common", "fields": [ { "java": { "class": "com.linkedin.common.urn.Urn" }, "type": "string", "name": "form", "doc": "The urn of the form that granted this verification." }, { "type": [ "null", "com.linkedin.common.AuditStamp" ], "name": "lastModified", "default": null, "doc": "An audit stamp capturing who and when verification was applied for this form." } ], "doc": "An association between a verification and an entity that has been granted\nvia completion of one or more forms of type 'VERIFICATION'." } }, "name": "verifications", "default": [], "doc": "Verifications that have been applied to the entity via completed forms." } ], "doc": "Forms that are assigned to this entity to be filled out" } ``` #### subTypes Python class: `SubTypesClass` Sub Types. Use this aspect to specialize a generic Entity e.g. Making a Dataset also be a View or also be a LookerExplore #### Fields | Field | Type | Required | Description | Annotations | |-------|------|----------|-------------|-------------| | typeNames | string[] | ✓ | The names of the specific types. | Searchable | #### Raw Schema ```javascript { "type": "record", "Aspect": { "name": "subTypes" }, "name": "SubTypes", "namespace": "com.linkedin.common", "fields": [ { "Searchable": { "/*": { "addToFilters": true, "fieldType": "KEYWORD", "filterNameOverride": "Sub Type", "queryByDefault": false } }, "type": { "type": "array", "items": "string" }, "name": "typeNames", "doc": "The names of the specific types." } ], "doc": "Sub Types. Use this aspect to specialize a generic Entity\ne.g. Making a Dataset also be a View or also be a LookerExplore" } ``` #### testResults Python class: `TestResultsClass` Information about a Test Result #### Fields | Field | Type | Required | Description | Annotations | |-------|------|----------|-------------|-------------| | failing | [TestResult](#testresult)[] | ✓ | Results that are failing | Searchable, → IsFailing | | passing | [TestResult](#testresult)[] | ✓ | Results that are passing | Searchable, → IsPassing | #### Raw Schema ```javascript { "type": "record", "Aspect": { "name": "testResults" }, "name": "TestResults", "namespace": "com.linkedin.test", "fields": [ { "Relationship": { "/*/test": { "entityTypes": [ "test" ], "name": "IsFailing" } }, "Searchable": { "/*/test": { "fieldName": "failingTests", "fieldType": "URN", "hasValuesFieldName": "hasFailingTests", "queryByDefault": false } }, "type": { "type": "array", "items": { "type": "record", "name": "TestResult", "namespace": "com.linkedin.test", "fields": [ { "java": { "class": "com.linkedin.common.urn.Urn" }, "type": "string", "name": "test", "doc": "The urn of the test" }, { "type": { "type": "enum", "symbolDocs": { "FAILURE": " The Test Failed", "SUCCESS": " The Test Succeeded" }, "name": "TestResultType", "namespace": "com.linkedin.test", "symbols": [ "SUCCESS", "FAILURE" ] }, "name": "type", "doc": "The type of the result" }, { "type": [ "null", "string" ], "name": "testDefinitionMd5", "default": null, "doc": "The md5 of the test definition that was used to compute this result.\nSee TestInfo.testDefinition.md5 for more information." }, { "type": [ "null", { "type": "record", "name": "AuditStamp", "namespace": "com.linkedin.common", "fields": [ { "type": "long", "name": "time", "doc": "When did the resource/association/sub-resource move into the specific lifecycle stage represented by this AuditEvent." }, { "java": { "class": "com.linkedin.common.urn.Urn" }, "type": "string", "name": "actor", "doc": "The entity (e.g. a member URN) which will be credited for moving the resource/association/sub-resource into the specific lifecycle stage. It is also the one used to authorize the change." }, { "java": { "class": "com.linkedin.common.urn.Urn" }, "type": [ "null", "string" ], "name": "impersonator", "default": null, "doc": "The entity (e.g. a service URN) which performs the change on behalf of the Actor and must be authorized to act as the Actor." }, { "type": [ "null", "string" ], "name": "message", "default": null, "doc": "Additional context around how DataHub was informed of the particular change. For example: was the change created by an automated process, or manually." } ], "doc": "Data captured on a resource/association/sub-resource level giving insight into when that resource/association/sub-resource moved into a particular lifecycle stage, and who acted to move it into that specific lifecycle stage." } ], "name": "lastComputed", "default": null, "doc": "The audit stamp of when the result was computed, including the actor who computed it." } ], "doc": "Information about a Test Result" } }, "name": "failing", "doc": "Results that are failing" }, { "Relationship": { "/*/test": { "entityTypes": [ "test" ], "name": "IsPassing" } }, "Searchable": { "/*/test": { "fieldName": "passingTests", "fieldType": "URN", "hasValuesFieldName": "hasPassingTests", "queryByDefault": false } }, "type": { "type": "array", "items": "com.linkedin.test.TestResult" }, "name": "passing", "doc": "Results that are passing" } ], "doc": "Information about a Test Result" } ``` #### documentation Python class: `DocumentationClass` Aspect used for storing all applicable documentations on assets. This aspect supports multiple documentations from different sources. There is an implicit assumption that there is only one documentation per source. For example, if there are two documentations from the same source, the latest one will overwrite the previous one. If there are two documentations from different sources, both will be stored. Future evolution considerations: The first entity that uses this aspect is Schema Field. We will expand this aspect to other entities eventually. The values of the documentation are not currently searchable. This will be changed once this aspect develops opinion on which documentation entry is the authoritative one. Ensuring that there is only one documentation per source is a business rule that is not enforced by the aspect yet. This will currently be enforced by the application that uses this aspect. We will eventually enforce this rule in the aspect using AspectMutators. #### Fields | Field | Type | Required | Description | Annotations | |-------|------|----------|-------------|-------------| | documentations | DocumentationAssociation[] | ✓ | Documentations associated with this asset. We could be receiving docs from different sources | | #### Raw Schema ```javascript { "type": "record", "Aspect": { "name": "documentation" }, "name": "Documentation", "namespace": "com.linkedin.common", "fields": [ { "type": { "type": "array", "items": { "type": "record", "name": "DocumentationAssociation", "namespace": "com.linkedin.common", "fields": [ { "type": "string", "name": "documentation", "doc": "Description of this asset" }, { "Searchable": { "/actor": { "fieldName": "documentationAttributionActors", "fieldType": "URN", "queryByDefault": false }, "/source": { "fieldName": "documentationAttributionSources", "fieldType": "URN", "queryByDefault": false }, "/time": { "fieldName": "documentationAttributionDates", "fieldType": "DATETIME", "queryByDefault": false } }, "type": [ "null", { "type": "record", "name": "MetadataAttribution", "namespace": "com.linkedin.common", "fields": [ { "type": "long", "name": "time", "doc": "When this metadata was updated." }, { "java": { "class": "com.linkedin.common.urn.Urn" }, "type": "string", "name": "actor", "doc": "The entity (e.g. a member URN) responsible for applying the assocated metadata. This can\neither be a user (in case of UI edits) or the datahub system for automation." }, { "java": { "class": "com.linkedin.common.urn.Urn" }, "type": [ "null", "string" ], "name": "source", "default": null, "doc": "The DataHub source responsible for applying the associated metadata. This will only be filled out\nwhen a DataHub source is responsible. This includes the specific metadata test urn, the automation urn." }, { "type": { "type": "map", "values": "string" }, "name": "sourceDetail", "default": {}, "doc": "The details associated with why this metadata was applied. For example, this could include\nthe actual regex rule, sql statement, ingestion pipeline ID, etc." } ], "doc": "Information about who, why, and how this metadata was applied" } ], "name": "attribution", "default": null, "doc": "Information about who, why, and how this metadata was applied" } ], "doc": "Properties of applied documentation including the attribution of the doc" } }, "name": "documentations", "doc": "Documentations associated with this asset. We could be receiving docs from different sources" } ], "doc": "Aspect used for storing all applicable documentations on assets.\nThis aspect supports multiple documentations from different sources.\nThere is an implicit assumption that there is only one documentation per\n source.\nFor example, if there are two documentations from the same source, the\n latest one will overwrite the previous one.\nIf there are two documentations from different sources, both will be\n stored.\nFuture evolution considerations:\nThe first entity that uses this aspect is Schema Field. We will expand this\n aspect to other entities eventually.\nThe values of the documentation are not currently searchable. This will be\n changed once this aspect develops opinion on which documentation entry is\n the authoritative one.\nEnsuring that there is only one documentation per source is a business\n rule that is not enforced by the aspect yet. This will currently be enforced by the\n application that uses this aspect. We will eventually enforce this rule in\n the aspect using AspectMutators." } ``` ### Common Types These types are used across multiple aspects in this entity. #### AuditStamp Data captured on a resource/association/sub-resource level giving insight into when that resource/association/sub-resource moved into a particular lifecycle stage, and who acted to move it into that specific lifecycle stage. **Fields:** - `time` (long): When did the resource/association/sub-resource move into the specific lifecyc... - `actor` (string): The entity (e.g. a member URN) which will be credited for moving the resource... - `impersonator` (string?): The entity (e.g. a service URN) which performs the change on behalf of the Ac... - `message` (string?): Additional context around how DataHub was informed of the particular change. ... #### FormAssociation Properties of an applied form. **Fields:** - `urn` (string): Urn of the applied form - `incompletePrompts` (FormPromptAssociation[]): A list of prompts that are not yet complete for this form. - `completedPrompts` (FormPromptAssociation[]): A list of prompts that have been completed for this form. #### IncidentSummaryDetails Summary statistics about incidents on an entity. **Fields:** - `urn` (string): The urn of the incident - `type` (string): The type of an incident - `createdAt` (long): The time at which the incident was raised in milliseconds since epoch. - `resolvedAt` (long?): The time at which the incident was marked as resolved in milliseconds since e... - `priority` (int?): The priority of the incident #### TestResult Information about a Test Result **Fields:** - `test` (string): The urn of the test - `type` (TestResultType): The type of the result - `testDefinitionMd5` (string?): The md5 of the test definition that was used to compute this result. See Test... - `lastComputed` (AuditStamp?): The audit stamp of when the result was computed, including the actor who comp... #### TimeStamp A standard event timestamp **Fields:** - `time` (long): When did the event occur - `actor` (string?): Optional: The actor urn involved in the event. ### Relationships #### Outgoing These are the relationships stored in this entity's aspects - OwnedBy - Corpuser via `ownership.owners.owner` - CorpGroup via `ownership.owners.owner` - ownershipType - OwnershipType via `ownership.owners.typeUrn` - TaggedWith - Tag via `globalTags.tags` - TermedWith - GlossaryTerm via `glossaryTerms.terms.urn` - AssociatedWith - Domain via `domains.domains` - Application via `applications.applications` - IsPartOf - Container via `container.container` - ResolvedIncidents - Incident via `incidentsSummary.resolvedIncidentDetails` - ActiveIncidents - Incident via `incidentsSummary.activeIncidentDetails` - IsFailing - Test via `testResults.failing` - IsPassing - Test via `testResults.passing` #### Incoming These are the relationships stored in other entity's aspects - IsPartOf - DataJob via `dataJobKey.flow` ### [Global Metadata Model](https://github.com/datahub-project/static-assets/raw/main/imgs/datahub-metadata-model.png) ![Global Graph](https://github.com/datahub-project/static-assets/raw/main/imgs/datahub-metadata-model.png) :::note 💡 **Contributing to this documentation** This page is auto-generated from the underlying source code. To make changes, please edit the relevant source files in the [metadata-models](https://github.com/datahub-project/datahub/tree/master/metadata-models) directory. **Tip:** For quick typo fixes or documentation updates, you can click the ✏️ **Edit** icon directly in the GitHub UI to open a Pull Request. For larger changes and PR naming conventions, please refer to our [Contributing Guide](/docs/contributing). :::