Skip to main content

Hightouch

Overview​

Hightouch is a Reverse ETL platform that syncs data from your data warehouse to business tools such as CRMs, marketing automation platforms, and analytics applications. Learn more in the official Hightouch documentation.

The DataHub integration for Hightouch covers sources, models, syncs, destinations, and sync run execution history. It captures coarse-grained and column-level lineage from sources through models to destinations, and supports stateful deletion detection.

Concept Mapping​

Hightouch ObjectDataHub EntityDescription
SourceDatasetSource database/warehouse (referenced as input)
ModelDatasetSQL query or transformation (optional, platform: "hightouch")
SyncData JobData pipeline that moves data from model to destination
DestinationDatasetTarget system (referenced as output)
Sync RunData Process InstanceExecution instance with statistics
WorkspacePlatform InstanceHightouch workspace (optional grouping)

Module hightouch​

Alpha

Important Capabilities​

CapabilityStatusNotes
Asset Containers✅Enabled by default, one per workspace.
Column-level Lineage✅Enabled by default, from parsed model SQL and sync field mappings.
Detect Deleted Entities✅Enabled by default via stateful ingestion.
Extract Ownership✅Enabled by default via extract_owners.
Extract Tags✅Enabled by default.
Platform Instance✅Enabled by default.
Schema Metadata✅Enabled by default for models.
Table-Level Lineage✅Enabled by default, from warehouse tables through models and syncs to destinations.
Test Connection✅Enabled by default.

Overview​

The hightouch module ingests metadata from Hightouch into DataHub. It is intended for production ingestion workflows and module-specific capabilities are documented below.

Prerequisites​

1. Generate a Hightouch API Key​

  1. Log in to your Hightouch account
  2. Navigate to Settings → API Keys
  3. Click Create API Key
  4. Give your key a descriptive name (e.g., "DataHub Integration")
  5. Select the appropriate permissions:
    • Read access to: Sources, Models, Syncs, Destinations, Sync Runs
  6. Copy the generated API key (you won't be able to see it again)
  7. Store the API key securely — you'll use it in the DataHub recipe configuration

2. Required Permissions​

The API key needs read access to:

  • Sources
  • Models
  • Syncs
  • Destinations
  • Sync Runs (for execution history)

Install the Plugin​

pip install 'acryl-datahub[hightouch]'

Starter Recipe​

Check out the following recipe to get started with ingestion! See below for full configuration options.

For general pointers on writing and running a recipe, see our main recipe guide.

source:
  type: hightouch
  config:
    api_config:
      api_key: "${HIGHTOUCH_API_KEY}"

    env: "PROD"
    platform_instance: "hightouch-prod"

    emit_models_as_datasets: true

    include_sync_runs: true
    max_sync_runs_per_sync: 10

    stateful_ingestion:
      enabled: true
      remove_stale_metadata: true

sink:
  type: datahub-rest
  config:
    server: "http://localhost:8080"

Config Details​

Note that a . is used to denote nested fields in the YAML recipe.

FieldDescription
api_config ✅
HightouchAPIConfig
api_config.api_key ❓
string(password)
Hightouch API key (a workspace API key with read access) used as a Bearer token on every request.
api_config.base_url
string
Base URL of the Hightouch REST API. Override for EU or self-hosted deployments. A trailing slash is stripped automatically.
Default: https://api.hightouch.com/api/v1
api_config.request_timeout_sec
integer
Per-request timeout in seconds for calls to the Hightouch API. Increase this if large workspaces time out on listing endpoints.
Default: 30
emit_models_as_datasets
boolean
Emit Hightouch models as datasets on the hightouch platform, with lineage from the warehouse tables they read. When disabled, syncs read directly from those warehouse tables.
Default: True
extract_owners
boolean
Set the user who created a model or sync as its technical owner, using their email address as the DataHub user id.
Default: True
include_contracts
boolean
Ingest Hightouch Event Contracts. Each event becomes a dataset with its JSON Schema, a DataContract holding a schema assertion, and lineage from the contract's event sources. Event Contracts are not available on every Hightouch plan.
Default: False
include_sibling_relationships
boolean
Make each table model a sibling of the warehouse table it reads, with the Hightouch model as the primary sibling. The warehouse table only gets the back-link when it already exists in DataHub.
Default: True
include_sync_runs
boolean
Ingest each sync's recent runs as DataProcessInstances.
Default: True
max_sync_runs_per_sync
integer
Maximum number of runs to ingest per sync. Must be at least 1; set include_sync_runs: false to skip runs.
Default: 10
parse_model_sql
boolean
Parse raw SQL models to derive table and column lineage from the tables they query.
Default: True
platform_instance
One of string, null
The instance of the platform that all assets produced by this recipe belong to. This should be unique within the platform. See https://docs.datahub.com/docs/platform-instances/ for more details.
Default: None
env
string
The environment that all assets produced by this connector belong to
Default: PROD
contract_patterns
AllowDenyPattern
A class to store allow deny regexes.

Patterns are matched against the start of the string only, not the entire
string - a pattern does not need to match to the end to be considered a match.
For example, the pattern "prod" matches "prod", "prod_east", and "production".
To require an exact match, anchor your pattern explicitly, e.g. "^prod$".
contract_patterns.ignoreCase
One of boolean, null
Whether to ignore case sensitivity during pattern matching.
Default: True
contract_patterns.allow
array
List of regex patterns to include in ingestion. Patterns match from the start of the string only, not the entire string - anchor with '^...$' for an exact match, e.g. '^prod$'.
Default: ['.*']
contract_patterns.allow.string
string
contract_patterns.deny
array
List of regex patterns to exclude from ingestion. Patterns match from the start of the string only, not the entire string - anchor with '^...$' for an exact match, e.g. '^prod$'.
Default: []
contract_patterns.deny.string
string
destination_to_platform_instance
map(str,PlatformDetail)
destination_to_platform_instance.key.platform
One of string, null
DataHub platform of the source or destination. Detected from the Hightouch type when unset.
Default: None
destination_to_platform_instance.key.convert_urns_to_lowercase
One of boolean, null
Lowercase table names, including database and schema, so they match the platform connector's URNs. Defaults to true for Snowflake, whose connector lowercases URNs by default, and false otherwise.
Default: None
destination_to_platform_instance.key.database
One of string, null
Database used to qualify table names. Defaults to the database in the Hightouch source configuration.
Default: None
destination_to_platform_instance.key.include_schema_in_urn
boolean
Include the schema in dataset URNs. Turn off for platforms whose connector emits two-part database.table names.
Default: True
destination_to_platform_instance.key.platform_instance
One of string, null
Platform instance used by the connector that ingests this source or destination.
Default: None
destination_to_platform_instance.key.env
One of string, null
Environment of the source or destination datasets. When unset, the connector-level env is used.
Default: None
model_patterns
AllowDenyPattern
A class to store allow deny regexes.

Patterns are matched against the start of the string only, not the entire
string - a pattern does not need to match to the end to be considered a match.
For example, the pattern "prod" matches "prod", "prod_east", and "production".
To require an exact match, anchor your pattern explicitly, e.g. "^prod$".
model_patterns.ignoreCase
One of boolean, null
Whether to ignore case sensitivity during pattern matching.
Default: True
model_patterns.allow
array
List of regex patterns to include in ingestion. Patterns match from the start of the string only, not the entire string - anchor with '^...$' for an exact match, e.g. '^prod$'.
Default: ['.*']
model_patterns.allow.string
string
model_patterns.deny
array
List of regex patterns to exclude from ingestion. Patterns match from the start of the string only, not the entire string - anchor with '^...$' for an exact match, e.g. '^prod$'.
Default: []
model_patterns.deny.string
string
sources_to_platform_instance
map(str,PlatformDetail)
sources_to_platform_instance.key.platform
One of string, null
DataHub platform of the source or destination. Detected from the Hightouch type when unset.
Default: None
sources_to_platform_instance.key.convert_urns_to_lowercase
One of boolean, null
Lowercase table names, including database and schema, so they match the platform connector's URNs. Defaults to true for Snowflake, whose connector lowercases URNs by default, and false otherwise.
Default: None
sources_to_platform_instance.key.database
One of string, null
Database used to qualify table names. Defaults to the database in the Hightouch source configuration.
Default: None
sources_to_platform_instance.key.include_schema_in_urn
boolean
Include the schema in dataset URNs. Turn off for platforms whose connector emits two-part database.table names.
Default: True
sources_to_platform_instance.key.platform_instance
One of string, null
Platform instance used by the connector that ingests this source or destination.
Default: None
sources_to_platform_instance.key.env
One of string, null
Environment of the source or destination datasets. When unset, the connector-level env is used.
Default: None
sync_patterns
AllowDenyPattern
A class to store allow deny regexes.

Patterns are matched against the start of the string only, not the entire
string - a pattern does not need to match to the end to be considered a match.
For example, the pattern "prod" matches "prod", "prod_east", and "production".
To require an exact match, anchor your pattern explicitly, e.g. "^prod$".
sync_patterns.ignoreCase
One of boolean, null
Whether to ignore case sensitivity during pattern matching.
Default: True
sync_patterns.allow
array
List of regex patterns to include in ingestion. Patterns match from the start of the string only, not the entire string - anchor with '^...$' for an exact match, e.g. '^prod$'.
Default: ['.*']
sync_patterns.allow.string
string
sync_patterns.deny
array
List of regex patterns to exclude from ingestion. Patterns match from the start of the string only, not the entire string - anchor with '^...$' for an exact match, e.g. '^prod$'.
Default: []
sync_patterns.deny.string
string
stateful_ingestion
One of StatefulStaleMetadataRemovalConfig, null
Hightouch Stateful Ingestion Config.
Default: None
stateful_ingestion.enabled
boolean
Whether or not to enable stateful ingest. Default: True if a pipeline_name is set and either a datahub-rest sink or datahub_api is specified, otherwise False
Default: False
stateful_ingestion.fail_safe_threshold
number
Prevents large amount of soft deletes & the state from committing from accidental changes to the source configuration if the relative change percent in entities compared to the previous state is above the 'fail_safe_threshold'.
Default: 75.0
stateful_ingestion.remove_stale_metadata
boolean
Soft-deletes the entities present in the last successful run but missing in the current run with stateful_ingestion enabled.
Default: True

Capabilities​

This connector extracts the following metadata from Hightouch:

  • Workspaces — One container per Hightouch workspace, holding its models and syncs
  • Sources — Database and warehouse connections (as DataHub Dataset references)
  • Models — SQL queries and transformations (optionally as DataHub Datasets)
  • Syncs — One DataFlow per workspace, with a DataJob per sync from its model to its destination
  • Destinations — Target systems like Salesforce, HubSpot, etc. (as DataHub Dataset references)
  • Sync Runs — Execution history with detailed statistics (as DataHub DataProcessInstances)
  • Lineage — Complete data flow from sources through models and syncs to destinations
  • Column-level Lineage — Field mappings between source and destination systems
  • Ownership — The user who created a model or sync, as its technical owner
  • Tags — Model and sync tags, as key:value tags
  • Event Contracts — Event schema definitions emitted as Datasets, each with a DataHub DataContract housing a schema assertion, plus lineage from the contract's Event Sources

Model Ingestion Options​

You can configure how Hightouch models are represented in DataHub:

  • Option 1 (Recommended): Set emit_models_as_datasets: true

    • Models appear as separate Dataset entities in DataHub with platform "hightouch"
    • Lineage flows: Source Tables → Hightouch Models → Syncs → Destination Tables
    • Provides visibility into intermediate transformations

  • Option 2: Set emit_models_as_datasets: false

    • Models are not created as separate entities
    • Lineage flows directly: Source Tables → Syncs → Destination Tables
    • Simpler lineage graph, but less visibility into transformations

Lineage​

The connector extracts comprehensive lineage:

  1. Coarse-grained Lineage: Table-to-table relationships showing data flow
  2. Fine-grained Lineage: Column-to-column mappings extracted from sync field mappings
  3. Multi-hop Lineage: Complete path from source databases through models to destinations

SQL parsing is supported for raw_sql models, automatically extracting upstream table dependencies and creating column-level lineage. The model's schema is the query's output columns. When the upstream tables are already in DataHub, their names are matched with the casing they were ingested with. Column casing is normalized to match upstream table schemas (e.g., Snowflake lowercasing). For best results:

  • Ensure upstream source tables are already ingested into DataHub
  • Configure sources_to_platform_instance for accurate URN generation
  • Enable a DataHub graph connection for schema resolution

Sibling Relationships​

For table models, which expose a warehouse table as is, you can establish sibling relationships between the Hightouch model and that table. Raw SQL models get lineage to the tables they read but no sibling, because a query over a table is a different asset from the table. dbt models also get lineage but no sibling, because the dbt connector already makes its dbt node the table's primary sibling:

include_sibling_relationships: true # default: true

When enabled, the Hightouch model is marked as the primary sibling and the upstream source table is linked as a secondary sibling. Sibling aspects are only emitted on the upstream table if it already exists in DataHub, preventing the creation of "ghost" entities. Requires sources_to_platform_instance configuration for proper URN matching.

Event Contracts​

Hightouch Event Contracts define the schemas used to validate incoming events. Event Contracts are not available on every Hightouch plan, so they are off by default. With include_contracts: true, the connector ingests them as:

  • one Dataset per event, whose schema is the event's JSON Schema and whose custom properties capture the enforcement rules (on_schema_violation, on_undeclared_fields, on_undeclared_schema),
  • a DataContract on each event dataset, housing a schema assertion derived from the JSON Schema,
  • lineage from the contract's Event Sources (emitted as datasets) to each event dataset.
include_contracts: true # default: false
contract_patterns:
  allow:
    - ".*"

The Hightouch API does not expose contract validation history, so the assertions are emitted as definitions only (no pass/fail run results).

Platform Instance Configuration​

If you have multiple instances of source or destination systems, configure platform instances to generate correct lineage edges:

sources_to_platform_instance:
  # Key is the Hightouch source ID (visible in Hightouch URL)
  "12345":
    platform: "snowflake"
    platform_instance: "prod-snowflake"
    env: "PROD"
    database: "analytics"
    # Lowercase upstream table names to match the source connector's URNs.
    # Defaults to true for Snowflake and false otherwise.
    convert_urns_to_lowercase: true

destination_to_platform_instance:
  # Key is the Hightouch destination ID
  "dest_123":
    platform: "salesforce"
    platform_instance: "prod-salesforce"
    env: "PROD"

To find source and destination IDs, navigate to the resource in the Hightouch UI and check the URL (https://app.hightouch.com/[workspace]/sources/[source_id]), or query the Hightouch API:

curl -H "Authorization: Bearer YOUR_API_KEY" https://api.hightouch.com/api/v1/sources

Filtering​

Control which syncs and models are ingested using regex patterns. sync_patterns match each sync's slug, and model_patterns match each model's display name:

sync_patterns:
  allow:
    - "prod-.*"
  deny:
    - "test-.*"
    - ".*-staging"

model_patterns:
  allow:
    - "customer-.*"
  deny:
    - ".*-draft"

Sync Run History​

Limit the number of historical sync runs ingested:

include_sync_runs: true
max_sync_runs_per_sync: 10 # last 10 runs per sync

Set include_sync_runs: false to skip sync run history entirely. A run that has not finished (queued, querying, processing or reporting) is recorded with its start only.

Supported Source and Destination Types​

This connector supports 30+ data source types and 50+ destination types. Platform detection works automatically for major sources including Snowflake, BigQuery, Redshift, Databricks, PostgreSQL, MySQL, SQL Server, and common SaaS destinations. For sources or destinations not automatically recognized, configure the mapping manually using sources_to_platform_instance and destination_to_platform_instance. For the complete list of supported integrations, refer to the Hightouch documentation.

Limitations​

  • Ownership: The creator of a model or sync becomes its technical owner, with their email address as the DataHub user id. Set extract_owners: false if your DataHub users are keyed differently.
  • Destination lineage: Lineage to a destination is carried by the sync's DataJob. Nothing is written onto destination datasets, so their own connector's lineage is left as it is. A sync whose configuration names no destination object the connector recognises has no outlet, and is listed under unresolved_destinations in the report.
  • Model schemas: The API returns no column list for a model. A table or dbt model takes its schema from the table it reads, when that table is already in DataHub, and a raw SQL model from its query's output columns. Otherwise the schema is the union of the columns its syncs reference, without types.
  • API Rate Limits: The Hightouch API has rate limits. The connector implements retry logic with exponential backoff, but very large workspaces with thousands of syncs may need to be ingested in batches.
  • Platform Detection: Automatic platform mapping works for major databases and SaaS tools. Custom or less common source/destination types may need manual platform mapping via sources_to_platform_instance or destination_to_platform_instance.
  • Complex SQL Queries: SQL parsing works best for simple queries. Complex multi-table JOINs, CTEs, and subqueries may not produce complete column-level lineage.

Troubleshooting​

If ingestion fails, validate your API key, permissions, and connectivity first. Then review ingestion logs for source-specific errors and adjust configuration accordingly.

Authentication Errors​

If you see 401 Unauthorized, verify your API key is correct and has not expired. Generate a new key from Settings → API Keys in the Hightouch UI if needed.

Missing Lineage​

If lineage is not appearing correctly:

  1. Verify source and destination IDs are configured in sources_to_platform_instance / destination_to_platform_instance
  2. Check that emit_models_as_datasets is set to your preferred mode
  3. Ensure source and destination datasets already exist in DataHub (or ingest them separately)

Failed Runs and Stale Metadata​

A sync or model that fails to process, Event Contracts that cannot be read (other than an API key without access to them), an entity the API returns in an unexpected shape, or a list cut off at the pagination limit fails the run. That keeps stale-metadata removal from soft-deleting entities that still exist in Hightouch but were missing from the run. A single bad entity therefore marks the whole run as failed on purpose, even though everything else was ingested; the report names the entity.

A sync whose model could not be emitted reads directly from the model's warehouse tables instead, so its lineage never points at a model that does not exist.

Rate Limiting​

If you see 429 Too Many Requests, the connector will automatically retry with backoff. For large workspaces, consider using sync_patterns / model_patterns to reduce scope, or reducing max_sync_runs_per_sync to limit API calls.

Code Coordinates​

  • Class Name: datahub.ingestion.source.hightouch.hightouch.HightouchSource
  • Browse on GitHub
Questions?

If you've got any questions on configuring ingestion for Hightouch, feel free to ping us on our Slack.

💡 Contributing to this documentation

This page is auto-generated from the underlying source code. To make changes, please edit the relevant source files in the metadata-ingestion directory.

Tip: For quick typo fixes or documentation updates, you can click the ✏️ Edit icon directly in the GitHub UI to open a Pull Request. For larger changes and PR naming conventions, please refer to our Contributing Guide.