Skip to main content

Informix

Overview​

IBM Informix is a relational database management system used for transactional and analytical workloads. Learn more in the official Informix documentation.

The DataHub integration for Informix covers tables, views, schema fields, and containers (database/schema hierarchy). It also extracts foreign-key relationships, table- and column-level lineage for views, ownership, and approximate row counts, and supports stateful deletion detection. It connects via JDBC, so provisioning the IBM Informix JDBC driver is required before ingestion can run.

Concept Mapping​

Source ConceptDataHub ConceptNotes
DatabaseContainerTop-level container.
Owner (schema)ContainerNested under the database container.
Table / ViewDatasettabtype 'T'/'V' in systables.
ColumnSchemaFieldMapped from syscolumns.coltype.
ownerOwnership (DATAOWNER)The creating database user, applied to schemas and datasets.
Foreign keyForeignKeyConstraintFrom sysconstraints / sysreferences; tables only.
View textUpstream lineagesysviews.viewtext, parsed for table- and column-level lineage.
nrowsDataset profileApproximate row count from systables, not a row scan.

Module informix​

Testing

Important Capabilities​

CapabilityStatusNotes
Asset Containers✅Enabled by default.
Column-level Lineage✅Column-level view lineage. Supported for types - View.
Data Profiling✅Row counts only, via systables.nrows.
Detect Deleted Entities✅Enabled by default via stateful ingestion.
Domains✅Supported via the domain config field.
Extract Ownership✅Schema/table/view owner from systables.owner, via the include_ownership config field.
Platform Instance✅Enabled by default.
Schema Metadata✅Enabled by default.
Table-Level Lineage✅View lineage. Supported for types - View.

Overview​

The informix module ingests metadata from IBM Informix into DataHub by querying the Informix system catalog over JDBC. Module-specific capabilities are documented below.

Prerequisites​

JDBC Driver​

This source connects through the IBM Informix JDBC driver, which is not bundled with acryl-datahub because it is proprietary. Provide it in one of two ways:

  1. Bring your own jars (recommended for air-gapped or license-restricted environments): set driver_jar_paths to explicit paths for the com.ibm.informix:jdbc jar and the org.mongodb:bson jar it depends on. No download is attempted when this is set.
  2. Auto-download: set accept_ibm_jdbc_license: true to have the source download and checksum-verify the driver from Maven Central at runtime, caching it under ~/.datahub/jars/informix (override with driver_cache_dir). Pin exact versions with jdbc_driver_version / bson_version if needed.
caution

Auto-download requires accepting the IBM Informix JDBC Driver Software License Agreement. Setting accept_ibm_jdbc_license: true is your confirmation that you accept these terms; DataHub does not redistribute the driver.

Permissions​

The connecting user needs SELECT on the Informix system catalog to enumerate metadata:

  • systables, syscolumns — tables, views, and columns.
  • sysconstraints, sysindexes, sysreferences — primary- and foreign-key detection.
  • sysviews — view definitions (used for view lineage).

No access to user table data is required; row counts are approximate values read from the system catalog (systables.nrows), not row scans or sampling.

Install the Plugin​

pip install 'acryl-datahub[informix]'

Starter Recipe​

Check out the following recipe to get started with ingestion! See below for full configuration options.

For general pointers on writing and running a recipe, see our main recipe guide.

source:
  type: informix
  config:
    host_port: "localhost:9088"
    server: "informix"
    database: "mydb"
    username: "informix"
    password: "${INFORMIX_PASSWORD}"
    accept_ibm_jdbc_license: true

sink:
  type: datahub-rest
  config:
    server: "http://localhost:8080"

Config Details​

Note that a . is used to denote nested fields in the YAML recipe.

FieldDescription
database ✅
string
Informix database to ingest from.
server ✅
string
Informix server name (INFORMIXSERVER).
accept_ibm_jdbc_license
boolean
Set true to allow auto-downloading the proprietary IBM Informix JDBC driver from Maven Central under the IBM Informix JDBC Software License Agreement. Ignored when driver_jar_paths is set.
Default: False
bson_version
string
Pinned org.mongodb:bson version to download.
Default: 4.11.1
convert_urns_to_lowercase
boolean
Whether to convert dataset urns to lowercase. This value is part of each dataset's URN identity, so it must stay fixed for the life of a deployment. Changing it after data has been ingested re-keys every dataset (e.g. MyDb.MyTable becomes mydb.mytable); with stateful ingestion enabled the old-cased URNs are then soft-deleted as stale while the new-cased ones are created, producing duplicate or orphaned entities. Pick one value before the first run and leave it unchanged.
Default: False
driver_cache_dir
One of string, null
Directory to cache downloaded jars. Defaults to ~/.datahub/jars/informix.
Default: None
extra_props
string
Extra JDBC properties appended to the connection URL, e.g. 'DB_LOCALE=en_US.utf8;CLIENT_LOCALE=en_US.utf8'.
Default:
host_port
string
Informix host and port.
Default: localhost:9088
include_foreign_keys
boolean
Extract foreign-key relationships from sysconstraints/sysreferences.
Default: True
include_ownership
boolean
Emit ownership for schemas, tables and views from systables.owner. Informix records the owning database user rather than a person, so this produces a corpuser URN for that account (e.g. 'informix'), not an individual's identity.
Default: True
include_row_counts
boolean
Emit approximate row counts from systables.nrows.
Default: True
include_tables
boolean
Whether tables should be ingested.
Default: True
include_view_lineage
boolean
Extract table- and column-level lineage for views by parsing their SQL definitions.
Default: True
include_views
boolean
Whether views should be ingested.
Default: True
jdbc_driver_version
string
Pinned com.ibm.informix:jdbc version to download.
Default: 4.50.10
password
One of string(password), null
Login password.
Default: None
platform_instance
One of string, null
The instance of the platform that all assets produced by this recipe belong to. This should be unique within the platform. See https://docs.datahub.com/docs/platform-instances/ for more details.
Default: None
username
One of string, null
Login user.
Default: None
env
string
The environment that all assets produced by this connector belong to
Default: PROD
domain
map(str,AllowDenyPattern)
A class to store allow deny regexes.

Patterns are matched against the start of the string only, not the entire
string - a pattern does not need to match to the end to be considered a match.
For example, the pattern "prod" matches "prod", "prod_east", and "production".
To require an exact match, anchor your pattern explicitly, e.g. "^prod$".
domain.key.allow
array
List of regex patterns to include in ingestion. Patterns match from the start of the string only, not the entire string - anchor with '^...$' for an exact match, e.g. '^prod$'.
Default: ['.*']
domain.key.allow.string
string
domain.key.ignoreCase
One of boolean, null
Whether to ignore case sensitivity during pattern matching.
Default: True
domain.key.deny
array
List of regex patterns to exclude from ingestion. Patterns match from the start of the string only, not the entire string - anchor with '^...$' for an exact match, e.g. '^prod$'.
Default: []
domain.key.deny.string
string
driver_jar_paths
One of array, null
Explicit paths to the Informix JDBC jar and org.mongodb bson jar. If set, no download is attempted (use for air-gapped environments).
Default: None
driver_jar_paths.string
string
schema_pattern
AllowDenyPattern
A class to store allow deny regexes.

Patterns are matched against the start of the string only, not the entire
string - a pattern does not need to match to the end to be considered a match.
For example, the pattern "prod" matches "prod", "prod_east", and "production".
To require an exact match, anchor your pattern explicitly, e.g. "^prod$".
schema_pattern.ignoreCase
One of boolean, null
Whether to ignore case sensitivity during pattern matching.
Default: True
table_pattern
AllowDenyPattern
A class to store allow deny regexes.

Patterns are matched against the start of the string only, not the entire
string - a pattern does not need to match to the end to be considered a match.
For example, the pattern "prod" matches "prod", "prod_east", and "production".
To require an exact match, anchor your pattern explicitly, e.g. "^prod$".
table_pattern.ignoreCase
One of boolean, null
Whether to ignore case sensitivity during pattern matching.
Default: True
view_pattern
AllowDenyPattern
A class to store allow deny regexes.

Patterns are matched against the start of the string only, not the entire
string - a pattern does not need to match to the end to be considered a match.
For example, the pattern "prod" matches "prod", "prod_east", and "production".
To require an exact match, anchor your pattern explicitly, e.g. "^prod$".
view_pattern.ignoreCase
One of boolean, null
Whether to ignore case sensitivity during pattern matching.
Default: True
stateful_ingestion
One of StatefulStaleMetadataRemovalConfig, null
Stateful ingestion / stale-entity removal config.
Default: None
stateful_ingestion.enabled
boolean
Whether or not to enable stateful ingest. Default: True if a pipeline_name is set and either a datahub-rest sink or datahub_api is specified, otherwise False
Default: False
stateful_ingestion.fail_safe_threshold
number
Prevents large amount of soft deletes & the state from committing from accidental changes to the source configuration if the relative change percent in entities compared to the previous state is above the 'fail_safe_threshold'.
Default: 75.0
stateful_ingestion.remove_stale_metadata
boolean
Soft-deletes the entities present in the last successful run but missing in the current run with stateful_ingestion enabled.
Default: True

Capabilities​

Use the Important Capabilities table above as the source of truth for supported features and whether additional configuration is required. This module:

  • Emits a database → owner (schema) container hierarchy, with tables and views as datasets underneath.
  • Emits schema fields with native types (including length, e.g. VARCHAR(100)), nullability, and primary-key flags.
  • Extracts foreign-key relationships from sysconstraints / sysreferences, for tables (include_foreign_keys).
  • Extracts table- and column-level lineage for views by parsing sysviews.viewtext (include_view_lineage).
  • Emits viewProperties (the stored view SQL) for every view whose definition is readable.
  • Emits approximate row counts from systables.nrows, for tables (include_row_counts).
  • Assigns ownership from systables.owner (include_ownership).
  • Supports stateful ingestion with stale-entity (deletion) detection.

Ownership​

Ownership is taken from systables.owner, which Informix populates with the database user that created the object. Each schema, table and view is assigned that user as a DATAOWNER.

Two things to be aware of before relying on this:

  • Informix records a database account, not a person or team. Objects created by an administrative account all come back owned by that account (commonly informix), so the resulting owner is an identity in the DataHub sense but not necessarily a useful point of contact.
  • The owner name is also the schema name — in Informix the two are the same concept — so a schema container is owned by the user it is named after.

Set include_ownership: false to skip emitting it.

Limitations​

This module does not support:

  • Column profiling — no row sampling, null counts, or other column-level statistics; only approximate row counts from systables.nrows are emitted.
  • Stored procedures — SPL routines are not ingested as DataJobs.
  • Usage / query-log lineage — view lineage is derived only from parsing view SQL definitions, not from query logs or runtime usage.

View lineage and Informix-specific SQL​

View lineage is produced by parsing sysviews.viewtext with sqlglot. sqlglot has no Informix dialect, so the postgres dialect is used: Informix normalizes stored view text into a qualified, aliased, comma-join form that postgres parses correctly for the common case.

Views whose stored text retains Informix-specific syntax — MATCHES / NOT MATCHES, FIRST / SKIP, native OUTER joins, or DATETIME ... YEAR TO DAY — will not parse and get no lineage. This is per-view and non-fatal: the rest of the run is unaffected, and each failure is counted as view_lineage_failures in the ingestion report.

A view can also resolve at the table level while its column lineage fails to parse. In that case table-level lineage is still emitted, the shortfall is reported as a warning, and it is counted as view_column_lineage_failures.

Composite foreign keys​

Informix's catalog exposes a constraint's child and parent key columns as two independent 16-slot sysindexes column lists, so a composite foreign key comes back as a cross product rather than as ordered column pairs. Single-column foreign keys are always exact. For composite keys the columns are paired best-effort and a warning is reported, since the catalog does not record the pairing order.

If the two lists come back with different lengths — which happens when a constraint is backed by a wider pre-existing index — the pairing is ambiguous, so that constraint is skipped rather than emitted misaligned. Skipped constraints are counted as foreign_keys_dropped_mismatched in the ingestion report.

Extended type mapping​

syscolumns.coltype cannot identify an extended type on its own — LVARCHAR is type 40, and BOOLEAN, BLOB and CLOB all share type 41 — so the column's extended_id is resolved against sysxtdtypes to recover the real type name.

LVARCHAR, BOOLEAN, BLOB and CLOB map to their DataHub equivalents. A DISTINCT type reports its own name (for example MONEY_USD) and takes its DataHub type from the built-in it was defined over — read from coltype for an ordinary built-in, and from sysxtdtypes.source when it was defined over LVARCHAR, BOOLEAN, BLOB or CLOB, which coltype cannot express. A DISTINCT type defined over another DISTINCT type maps to a null type. A named ROW type maps to a record.

User-defined opaque types (JSON, BSON, time series, and spatial types such as ST_Geometry) have no DataHub equivalent and still map to a null type, but the native type name is reported rather than a placeholder. Each one is counted as a warning in the ingestion report.

Troubleshooting​

If ingestion fails, first confirm the JDBC driver is resolvable (see Prerequisites) and that the connecting user has SELECT on the system catalog tables. Then review ingestion logs for connection or query errors.

Code Coordinates​

  • Class Name: datahub.ingestion.source.informix.source.InformixSource
  • Browse on GitHub
Questions?

If you've got any questions on configuring ingestion for Informix, feel free to ping us on our Slack.

💡 Contributing to this documentation

This page is auto-generated from the underlying source code. To make changes, please edit the relevant source files in the metadata-ingestion directory.

Tip: For quick typo fixes or documentation updates, you can click the ✏️ Edit icon directly in the GitHub UI to open a Pull Request. For larger changes and PR naming conventions, please refer to our Contributing Guide.