Skip to main content

DataHub Recipe CLI - Agent Context

Best practices for AI agents building and validating DataHub ingestion recipes using the datahub recipe command group.

Purpose​

Build or fix an ingestion recipe without ever handling a resolved secret value.

Credential Boundary — What You Can and Cannot See​

This interface is built so a resolved secret value never reaches you. The guarantee holds only if you follow these rules — the boundary is a shared responsibility.

What the CLI guarantees: Secrets are resolved from ${ENV_VAR} references inside the CLI process, never in your context. All datahub recipe output is redacted — on success and on error, on stdout and stderr, for top-level and nested secret fields. So you may run probe and test-connection freely and will only ever see redacted results.

What you MUST NOT do (these break the boundary):

  • Never inline a secret value into a recipe. Always use ${ENV_VAR} references. If you see a plaintext secret in a recipe, you have already been exposed to it — do not copy it; recommend switching that field to ${ENV_VAR} and rerun validate.
  • Never set, read, or print a secret value yourself. Do not export SECRET=<value>, do not echo $SECRET, do not run env, and do not read files that contain literal secret values (e.g. an .env file holding real credentials). The operator sets the environment variables out of band before your session; the CLI inherits them without your involvement.
  • Never obtain credentials outside datahub recipe. Redaction lives in this command group only. Running any other command that prints a resolved credential (a raw driver call, an ingestion run with verbose logging, etc.) can leak it into your context.

Inline secrets: a recipe with a plaintext secret is still probeable, and validate will warn about it — but the boundary is already broken for that recipe because reading the file exposed the value. Treat the warning as a prompt to externalize the secret to ${ENV_VAR}.

Redaction can over-mask. Any string equal to a secret is blanked, so if a password happens to match a real identifier (a database also named datahub), that identifier reports as *** everywhere. probe filter cannot judge a name it cannot see, so it refuses a --name or --parent holding *** (exit 2), and --from-run skips masked entries with a warning. Do not guess the real name; ask the operator, or change the secret so it no longer equals an identifier.

Workflow​

  1. Describe — config fields and capabilities, no connection

    datahub recipe describe <source_type>

  2. Scaffold — a starter recipe: required fields, and secrets as ${ENV_VAR} references

    datahub recipe scaffold <source_type> > recipe.yml

    Pattern fields are deliberately absent. Writing one at all replaces the connector's own default, both halves — a scaffolded allow-all would switch off Snowflake's built-in deny of SNOWFLAKE_SAMPLE_DATA in database_pattern, or Kafka's of ^_.* in topic_patterns. The field is named after the connector's own vocabulary, so read it off describe rather than assuming database_pattern. Add a pattern only when you mean to narrow something, and use describe to see what the default already denies.

  3. Edit — modify the recipe locally, keeping secrets as ${ENV_VAR}.

  4. Validate — schema check, plus warnings for inline secrets

    datahub recipe validate recipe.yml

    It validates the recipe as resolved, so ${ENV_VAR} in an integer or boolean field is fine; an unresolvable reference is reported by name. The plaintext-secret warning covers nested config too, including free-form dicts like Kafka's consumer_config.

    References resolve exactly as datahub ingest resolves them: ${X}, a $X that starts the value, ${X:-default} and ${X:=default}; $$ is kept as written. Two differences matter:

    • A dotted name such as ${gms.server} is not a path into ~/.datahubenv. It reads as ${gms} and becomes an empty string; validate warns. Use ${DATAHUB_GMS_URL} and ${DATAHUB_GMS_TOKEN}.
    • A reference that takes part of a value (${X:0:8}, ${X:8}, ${#X}) is refused (exit 2): only whole values can be masked. Reference the whole value.

  5. Explore — see what is actually in the source (below).

  6. Check filters — see what your patterns would keep (below).

  7. Test connection

    datahub recipe test-connection --recipe recipe.yml

Exploring a source​

First check that the source can be probed: recipe describe lists Probe among its capabilities when it can. Without it, probe methods lists nothing and probe run is refused.

Two ways in, depending on the connector. Always start with probe methods, which is connection-free and tells you what this source offers:

datahub recipe probe methods --recipe recipe.yml

Each entry carries a command, its params, and a description taken from the method's own docstring. Parameters imply the nesting: a tables(schema) command sits under whatever lists schemas, and columns(schema, table) under that. Call one with:

datahub recipe probe run columns --recipe recipe.yml --schema public --table orders

Per-object commands (columns, foreign_keys, ...) accept only a schema and a table, view or materialized view that the source's own listings return, spelled as listed. Anything else is refused (exit 2), with the listed spelling as a hint when only the case differs.

SQL sources list their objects with containers, tables and views — prefer these over writing a catalog query. They come from the connector's own inspector, so tables excludes views (which views returns, judged by view_pattern rather than table_pattern), and each reports the schema it looked in, so probe filter needs no --parent from you:

datahub recipe probe run containers --recipe recipe.yml
datahub recipe probe run tables --recipe recipe.yml --schema public

containers reports kind as Schema or Database depending on the source — two-tier sources like MySQL filter their databases with database_pattern, not schema_pattern. Use the kind it reports rather than assuming.

sql is the escape hatch for a question the listings do not answer. It is an ordinary command in the list above — nothing special to remember:

datahub recipe probe run sql --recipe recipe.yml --limit 50 \
  --query "SELECT table_name FROM information_schema.tables WHERE table_schema = 'public'"

Only single SELECT statements over catalog metadata are permitted — the framework checks the query before the connector sees it. information_schema is allowed as a whole schema; everything else is allowed relation by relation, because "it lives in a catalog schema" does not mean "it is metadata": pg_catalog.pg_stats holds sampled values out of your user columns, and pg_stat_statements holds query text with literals in it. Anything not listed is refused before the database sees it, with exit code 2 and a message naming the reference that failed and what this source does permit:

{
  "error": "'pg_catalog.pg_stats' is outside the catalog metadata this probe may read; this source permits schemas ['information_schema'] and 21 individually listed relations"
}

Refusals you should expect, and not try to work around: user tables, unqualified table names, multiple statements, non-SELECT statements, vendor-specific functions (pg_read_file, dblink, load_file), and any comment or optimizer hint (--, /* */, /*+ */), since some engines run text inside comments. A query the database itself rejects as wrong (SQLSTATE class 42: a misspelled column, a missing relation, no grant) also exits 2: fix the query. A write hidden inside a query is refused too, however read-only the outer statement looks:

WITH x AS (DELETE FROM t RETURNING id) SELECT id FROM x

That is a data-modifying CTE: the write is inside the query, so checking only the outer statement type would pass it.

BigQuery addresses catalog views as <dataset>.INFORMATION_SCHEMA.TABLES, which is understood.

Every probe result is the same envelope:

{ "source_type": "...", "command": "...", "params": {}, "kind": "Table",
  "parent_path": [], "result": ..., "truncated": false,
  "warnings": [], "failures": [] }

result is where the shapes differ. A typed listing puts its list there ("result": ["orders", "customers"]); only sql nests a table, "result": {"columns": [...], "rows": [[...]], "truncated": false}, with positional rows. So read result first and do not expect columns/rows from a typed command.

truncated is top level on every result, typed listings as well as sql — read it before concluding you have seen everything. --limit is clamped to 1000 however large a value you pass, and a command that takes no --limit is capped at the same 1000, so narrow the query or the schema rather than raising the limit when a listing comes back truncated.

failures non-empty means the result is NOT a complete answer, and the command exits non-zero: "could not read" and "nothing there" are different, and this is how they are told apart.

Some sources expose an api command, a read passthrough to their own API, for questions no typed command answers:

datahub recipe probe run api --recipe recipe.yml --path /spaces/sp1/reports

Only GET, and only paths the connector lists; anything else exits 2. Prefer probe run where a getter exists — it returns the names patterns are matched against, whereas a raw record leaves you guessing which field that is.

Both passthroughs can be switched off for a deployment (DATAHUB_PROBE_DISABLE_RAW_ACCESS). The refusal tells you what is left for this connector, because it varies: most have typed commands to fall back on, but Snowflake exposes sql as its only command, so switching it off withholds its probe entirely. BigQuery also declares projects, which takes no caller-supplied query, so switching it off narrows BigQuery's probe rather than withholding it. Read what the message names and do not look for another route to raw access.

A deployment can also switch off probe run entirely (DATAHUB_PROBE_DISABLED). Then every probe run command is refused — typed listings as well as the passthroughs — and the commands that need no connection keep working: recipe describe, recipe scaffold, recipe validate, probe methods and probe filter. recipe test-connection is not covered by this switch: it still connects, because it checks the recipe rather than reading the source. You can still read what a connector supports, write a recipe and check its filters against names you already have; you cannot read anything from the source. Both refusals exit 2, which means the command was the wrong one to ask for: change what you are asking for rather than retrying it.

Which levels a source has, and in what order​

describe tells you which config fields gate a hierarchy level. Read filters on each pattern field — a level-bearing pattern reports the kind it filters, and one that filters something outside the hierarchy reports null:

datahub recipe describe postgres
database_pattern   filters: "Database"
schema_pattern     filters: "Schema"
table_pattern      filters: "Table"
view_pattern       filters: "View"
procedure_pattern  filters: null      <- a real filter, not a level
profile_pattern    filters: null
user_email_pattern filters: null

This matters more than it looks: Snowflake has fifteen pattern fields and only four are levels. Guessing from the name would have you editing procedure_pattern to fix a missing table.

Containment order is not machine-readable — Database and Schema as names do not say which contains which. Use this table, and note that a source's levels are a subset of it:

familyorder
Postgres, MSSQL, SnowflakeDatabase → Schema → Table/View → Column
BigQueryProject → Dataset → Table/View → Column
MySQL and other two-tier SQLDatabase → Table/View → Column
Redshift, most other SQLSchema → Table/View → Column
KafkaTopic
ModeSpace → Report → Query, and Space → Dataset

You rarely need the order to diagnose something. Each probe filter --kind X answer is correct on its own, so to find out why an object was skipped you can check its own kind first and then walk outwards — table, then schema, then database — stopping at the first included: false. Order matters only if you want to enumerate a source top-down.

Checking what your filters would do​

This is the step most worth not skipping, because you cannot work it out yourself from the names alone.

datahub recipe probe filter --recipe recipe.yml \
  --kind Table --parent public \
  --name orders --name users --name audit_log_v2

AllowDenyPattern matches with a start-anchored regex against the identifier ingestion uses — which is usually not the bare name. MySQL matches schema.table, Postgres db.schema.table, Druid the bare name. So a pattern like ^orders.* looks precise and matches nothing, because the string being tested is public.orders.

Measured against a real MySQL instance, four of six plausible patterns gave the wrong answer when reasoned about from bare names. One allow: ^INNODB.* suggested 31 tables would be kept; ingestion kept 0. Patterns starting with .* happen to agree; anchored ones do not.

Every result reports the target it was matched against — read it, because that is what explains a surprising verdict:

{
  "pattern_field": "table_pattern",
  "results": [
    {
      "name": "audit_log_v2",
      "target": "public.audit_log_v2",
      "included": false,
      "excluded_by": "table_pattern"
    }
  ]
}

pattern_field names the field that actually decided, which is not always the one named after the kind — MySQL copies table_pattern into view_pattern, so a view is decided by the latter. Edit the field the output names.

excluded_by is not always a pattern. A recipe can switch a whole kind off, and that overrules any pattern:

{ "name": "some_view", "included": false, "excluded_by": "include_views" }

Seeing a config flag there means editing include_views: true, not the pattern — the pattern never got a say.

filtering says why pattern_field is null, which the null alone cannot:

valuemeaning
by_patterna field decided; pattern_field names it
by_rulerules that are not a pattern decide (path_specs, say); excluded_by says which
unfilteredthe source declares nothing filters this level — every name is included
unresolvedno field could be found and none was declared absent

pattern_field can be a dotted path (filter_config.entries.pattern) when the pattern sits in a nested block; edit that path in the recipe. A by_rule kind ignores --try-allow/--try-deny with a warning: edit the rules in the recipe to test a change.

unresolved is the one to act on: it usually means the kind is wrong for this source rather than that the source filters nothing. The warning beside it lists the kinds the source does declare — a two-tier source like MySQL has no Schema level, so ask about Database.

Test a pattern before committing to it with --try-allow / --try-deny. Each replaces only its own half of the recipe's pattern — passing --try-deny alone keeps the recipe's allow list, so you are testing the deny you asked about rather than an allow-all you did not. The tried field in the output echoes both halves as they were applied:

datahub recipe probe filter --recipe recipe.yml --kind Table --parent public \
  --name orders --name users --try-allow '^public\.ord.*'

probe filter needs no connection: it judges names you already have. To judge a whole listing, save it with probe run ... --report-to run.json and pass --from-run run.json instead of --name: the kind, the parent and each record's facts come from the file unless you pass --kind or --parent. A file that is not a probe run listing, or over 50 MB, is refused (exit 2).

Boundary Rules​

  • Probe output is metadata only — names, types, DDL, constraints, counts. Never table rows or message payloads. If you need row data, you are outside this tool's purpose.
  • Identity columns are withheld even from permitted relations. A catalog view can be worth reading for its shape and still name a person: ACCOUNT_USAGE.ACCESS_HISTORY.USER_NAME comes back as ***. The column is kept so you can see it exists; the value is not returned. That is a deliberate withholding, not a failure, and not something to work around.
  • The scope check on the sql command narrows what a query can reach; it is not a security boundary. Recommend the recipe authenticate as a read-only role scoped to catalog metadata.
  • An empty result with warnings means part of it could not be read — never "this is empty". Read warnings before concluding a source is empty.

Error Handling​

Branch on exit codes, not on message text. The code says who has to act:

codemeaning
0success
1a defect in DataHub or the connector, or a missing Python package (the error names it): not fixable by you
2your input was wrong — bad recipe or URL, bad parameter, a name the source does not list, a refused query
3the source could not be reached or read, or recorded a failure (failures is non-empty)

Do not retry a 2 unchanged; change what you ask for. A 3 may be transient, or the credential may lack a grant.

An error from a driver, an SDK or the connector shows its class and at most one code, never its text, which is where connection strings and query literals leak from: 'tables' failed (ProgrammingError; SQLSTATE 42P01). The operator can see the full text by running the same command with DATAHUB_PROBE_VERBOSE_LOGS=1; never set it yourself, since it also turns log scrubbing off. stderr carries the connector's log lines (INFO and above, scrubbed, tracebacks dropped) before the final {"error": ...} line; read that line, not the whole stream.

When the source cannot be opened because of the network, the label is followed by a reason and a fixed hint: opening source 'mysql' failed (OperationalError; errno 2003): ConnectionRefused - .... The reason comes from the exception's type, never its text. Act on it before touching credentials:

reasonwhat to do
HostNotResolvedfix the host's spelling, or ask the user about DNS or VPN
ConnectionRefusedcheck the port, and that the server is running
Timeoutask the user about a firewall, allowlist or private network; retrying will not help
HostUnreachablethe host is on a network this machine cannot route to; ask the user
TlsVerifyFailedfix the recipe's CA or TLS settings

No reason means the network was not the cause, or the driver did not say (Postgres never does).

Common Recipes​

# Inspect a connector's config fields
datahub recipe describe snowflake

# Generate a starter recipe, then edit it (keep secrets as ${ENV_VAR})
datahub recipe scaffold snowflake > recipe.yml
datahub recipe validate recipe.yml

# See what this source offers
datahub recipe probe methods --recipe recipe.yml

# Explore: catalog query (SQL sources) or a typed command
datahub recipe probe run sql --recipe recipe.yml \
  --query "SELECT schema_name FROM information_schema.schemata"
datahub recipe probe run columns --recipe recipe.yml --schema public --table orders

# Would my filters keep these?
datahub recipe probe filter --recipe recipe.yml --kind Table --parent public \
  --name orders --name users

# Verify connectivity
datahub recipe test-connection --recipe recipe.yml