NiFi

Overview

Apache NiFi is a streaming or integration platform. Learn more in the official Apache NiFi documentation.

The DataHub integration for Apache NiFi covers streaming/integration entities such as topics, connectors, pipelines, or jobs. It also captures table-level lineage and stateful deletion detection.

Concept Mapping

Source Concept	DataHub Concept	Notes
`"Nifi"`	Data Platform
Nifi flow	Data Flow
Nifi Ingress / Egress Processor	Data Job
Nifi Remote Port	Data Job
Nifi Port with remote connections	Dataset
Nifi Process Group	Container	Subtype `Process Group`

Module `nifi`

Important Capabilities

Capability	Status	Notes
Detect Deleted Entities	✅	Enabled by default via stateful ingestion.
Table-Level Lineage	✅	Supported. See docs for limitations.

Overview

The nifi module ingests metadata from Nifi into DataHub. It is intended for production ingestion workflows and module-specific capabilities are documented below.

Prerequisites

Before running ingestion, ensure network connectivity to the source, valid authentication credentials, and read permissions for metadata APIs required by this module.

Access Policies

This connector requires following access policies to be set in Nifi for ingestion user.

Global Access Policies

Policy	Privilege	Resource	Action
view the UI	Allows users to view the UI	`/flow`	R
query provenance	Allows users to submit a Provenance Search and request Event Lineage	`/provenance`	R

Component level Access Policies (required to be set on root process group)

Policy	Privilege	Resource	Action
view the component	Allows users to view component configuration details	`/<component-type>/<component-UUID>`	R
view the data	Allows users to view metadata and content for this component in flowfile queues in outbound connections and through provenance events	`/data/<component-type>/<component-UUID>`	R
view provenance	Allows users to view provenance events generated by this component	`/provenance-data/<component-type>/<component-UUID>`	R

Authentication

This connector supports following authentication mechanisms

Single User Authentication (`auth: SINGLE_USER`)

Connector will pass this username and password as used on Nifi Login Page over /access/token REST endpoint. This mode also works when Kerberos login identity provider is set up for Nifi.

Client Certificates Authentication (`auth: CLIENT_CERT`)

Connector will use client_cert_file(required) and client_key_file(optional), client_key_password(optional) for mutual TLS authentication.

Kerberos Authentication via SPNEGO (`auth: Kerberos`)

If nifi has been configured to use Kerberos SPNEGO, connector will pass user’s Kerberos ticket to nifi over /access/kerberos REST endpoint. It is assumed that user's Kerberos ticket is already present on the machine on which ingestion runs. This is usually done by installing krb5-user and then running kinit for user.

sudo apt install krb5-user
kinit user@REALM

Basic Authentication (`auth: BASIC_AUTH`)

Connector will use HTTPBasicAuth with username and password.

No Authentication (`auth: NO_AUTH`)

This is useful for testing purposes.

Install the Plugin

pip install 'acryl-datahub[nifi]'

Starter Recipe

Check out the following recipe to get started with ingestion! See below for full configuration options.

For general pointers on writing and running a recipe, see our main recipe guide.

source:
  type: "nifi"
  config:
    # Coordinates
    site_url: "https://localhost:8443/nifi/"

    # Credentials
    auth: SINGLE_USER
    username: admin
    password: password

sink:
  # sink configs

Config Details

Options
Schema

Note that a . is used to denote nested fields in the YAML recipe.

Field	Description
site_url ✅ string	URL for Nifi, ending with /nifi/. e.g. https://mynifi.domain/nifi/
auth Enum	One of: "NO_AUTH", "SINGLE_USER", "CLIENT_CERT", "KERBEROS", "BASIC_AUTH"
ca_file One of boolean, string, null	Path to PEM file containing certs for the root CA(s) for the NiFi.Set to False to disable SSL verification. Default: None
client_cert_file One of string, null	Path to PEM file containing the public certificates for the user/client identity, must be set for auth = "CLIENT_CERT" Default: None
client_key_file One of string, null	Path to PEM file containing the client’s secret key Default: None
client_key_password One of string(password), null	The password to decrypt the client_key_file Default: None
emit_process_group_as_container boolean	Whether to emit Nifi process groups as container entities. Default: False
incremental_lineage boolean	When enabled, emits incremental/patch lineage for Nifi processors. When disabled, re-states lineage on each run. Default: True
password One of string(password), null	Nifi password, must be set for auth = "SINGLE_USER" Default: None
provenance_days integer	time window to analyze provenance events for external datasets Default: 7
site_name string	Site name to identify this site with, useful when using input and output ports receiving remote connections Default: default
site_url_to_site_name map(str,string)
username One of string, null	Nifi username, must be set for auth = "SINGLE_USER" Default: None
env string	The environment that all assets produced by this connector belong to Default: PROD
process_group_pattern AllowDenyPattern	A class to store allow deny regexes. Patterns are matched against the start of the string only, not the entire string - a pattern does not need to match to the end to be considered a match. For example, the pattern "prod" matches "prod", "prod_east", and "production". To require an exact match, anchor your pattern explicitly, e.g. "^prod$".
process_group_pattern.ignoreCase One of boolean, null	Whether to ignore case sensitivity during pattern matching. Default: True
stateful_ingestion One of StatefulIngestionConfig, null	Stateful Ingestion Config Default: None
stateful_ingestion.enabled boolean	Whether or not to enable stateful ingest. Default: True if a pipeline_name is set and either a datahub-rest sink or `datahub_api` is specified, otherwise False Default: False

The JSONSchema for this configuration is inlined below.

{
  "$defs": {
    "AllowDenyPattern": {
      "additionalProperties": false,
      "description": "A class to store allow deny regexes.\n\nPatterns are matched against the start of the string only, not the entire\nstring - a pattern does not need to match to the end to be considered a match.\nFor example, the pattern \"prod\" matches \"prod\", \"prod_east\", and \"production\".\nTo require an exact match, anchor your pattern explicitly, e.g. \"^prod$\".",
      "properties": {
        "allow": {
          "default": [
            ".*"
          ],
          "description": "List of regex patterns to include in ingestion. Patterns match from the start of the string only, not the entire string - anchor with '^...$' for an exact match, e.g. '^prod$'.",
          "items": {
            "type": "string"
          },
          "title": "Allow",
          "type": "array"
        },
        "deny": {
          "default": [],
          "description": "List of regex patterns to exclude from ingestion. Patterns match from the start of the string only, not the entire string - anchor with '^...$' for an exact match, e.g. '^prod$'.",
          "items": {
            "type": "string"
          },
          "title": "Deny",
          "type": "array"
        },
        "ignoreCase": {
          "anyOf": [
            {
              "type": "boolean"
            },
            {
              "type": "null"
            }
          ],
          "default": true,
          "description": "Whether to ignore case sensitivity during pattern matching.",
          "title": "Ignorecase"
        }
      },
      "title": "AllowDenyPattern",
      "type": "object"
    },
    "NifiAuthType": {
      "enum": [
        "NO_AUTH",
        "SINGLE_USER",
        "CLIENT_CERT",
        "KERBEROS",
        "BASIC_AUTH"
      ],
      "title": "NifiAuthType",
      "type": "string"
    },
    "StatefulIngestionConfig": {
      "additionalProperties": false,
      "description": "Basic Stateful Ingestion Specific Configuration for any source.",
      "properties": {
        "enabled": {
          "default": false,
          "description": "Whether or not to enable stateful ingest. Default: True if a pipeline_name is set and either a datahub-rest sink or `datahub_api` is specified, otherwise False",
          "title": "Enabled",
          "type": "boolean"
        }
      },
      "title": "StatefulIngestionConfig",
      "type": "object"
    }
  },
  "additionalProperties": false,
  "properties": {
    "env": {
      "default": "PROD",
      "description": "The environment that all assets produced by this connector belong to",
      "title": "Env",
      "type": "string"
    },
    "stateful_ingestion": {
      "anyOf": [
        {
          "$ref": "#/$defs/StatefulIngestionConfig"
        },
        {
          "type": "null"
        }
      ],
      "default": null,
      "description": "Stateful Ingestion Config"
    },
    "site_url": {
      "description": "URL for Nifi, ending with /nifi/. e.g. https://mynifi.domain/nifi/",
      "title": "Site Url",
      "type": "string"
    },
    "auth": {
      "$ref": "#/$defs/NifiAuthType",
      "default": "NO_AUTH",
      "description": "Nifi authentication. must be one of : NO_AUTH, SINGLE_USER, CLIENT_CERT, KERBEROS"
    },
    "provenance_days": {
      "default": 7,
      "description": "time window to analyze provenance events for external datasets",
      "title": "Provenance Days",
      "type": "integer"
    },
    "process_group_pattern": {
      "$ref": "#/$defs/AllowDenyPattern",
      "default": {
        "allow": [
          ".*"
        ],
        "deny": [],
        "ignoreCase": true
      },
      "description": "regex patterns for filtering process groups"
    },
    "site_name": {
      "default": "default",
      "description": "Site name to identify this site with, useful when using input and output ports receiving remote connections",
      "title": "Site Name",
      "type": "string"
    },
    "site_url_to_site_name": {
      "additionalProperties": {
        "type": "string"
      },
      "default": {},
      "description": "Lookup to find site_name for site_url ending with /nifi/, required if using remote process groups in nifi flow",
      "title": "Site Url To Site Name",
      "type": "object"
    },
    "username": {
      "anyOf": [
        {
          "type": "string"
        },
        {
          "type": "null"
        }
      ],
      "default": null,
      "description": "Nifi username, must be set for auth = \"SINGLE_USER\"",
      "title": "Username"
    },
    "password": {
      "anyOf": [
        {
          "format": "password",
          "type": "string",
          "writeOnly": true
        },
        {
          "type": "null"
        }
      ],
      "default": null,
      "description": "Nifi password, must be set for auth = \"SINGLE_USER\"",
      "title": "Password"
    },
    "client_cert_file": {
      "anyOf": [
        {
          "type": "string"
        },
        {
          "type": "null"
        }
      ],
      "default": null,
      "description": "Path to PEM file containing the public certificates for the user/client identity, must be set for auth = \"CLIENT_CERT\"",
      "title": "Client Cert File"
    },
    "client_key_file": {
      "anyOf": [
        {
          "type": "string"
        },
        {
          "type": "null"
        }
      ],
      "default": null,
      "description": "Path to PEM file containing the client\u2019s secret key",
      "title": "Client Key File"
    },
    "client_key_password": {
      "anyOf": [
        {
          "format": "password",
          "type": "string",
          "writeOnly": true
        },
        {
          "type": "null"
        }
      ],
      "default": null,
      "description": "The password to decrypt the client_key_file",
      "title": "Client Key Password"
    },
    "ca_file": {
      "anyOf": [
        {
          "type": "boolean"
        },
        {
          "type": "string"
        },
        {
          "type": "null"
        }
      ],
      "default": null,
      "description": "Path to PEM file containing certs for the root CA(s) for the NiFi.Set to False to disable SSL verification.",
      "title": "Ca File"
    },
    "emit_process_group_as_container": {
      "default": false,
      "description": "Whether to emit Nifi process groups as container entities.",
      "title": "Emit Process Group As Container",
      "type": "boolean"
    },
    "incremental_lineage": {
      "default": true,
      "description": "When enabled, emits incremental/patch lineage for Nifi processors. When disabled, re-states lineage on each run.",
      "title": "Incremental Lineage",
      "type": "boolean"
    }
  },
  "required": [
    "site_url"
  ],
  "title": "NifiSourceConfig",
  "type": "object"
}

Capabilities

Use the Important Capabilities table above as the source of truth for supported features and whether additional configuration is required.

Limitations

Module behavior is constrained by source APIs, permissions, and metadata exposed by the platform. Refer to capability notes for unsupported or conditional features.

Lineage extraction analyzes provenance events. Verify your NiFi provenance retention period and run ingestion frequently enough to capture events before they expire.
Limited ingress/egress processors are supported
- S3: ListS3, FetchS3Object, PutS3Object
- SFTP: ListSFTP, FetchSFTP, GetSFTP, PutSFTP

Troubleshooting

If ingestion fails, validate credentials, permissions, connectivity, and scope filters first. Then review ingestion logs for source-specific errors and adjust configuration accordingly.

Code Coordinates

Class Name: datahub.ingestion.source.nifi.NifiSource
Browse on GitHub

Questions?

If you've got any questions on configuring ingestion for NiFi, feel free to ping us on our Slack.

💡 Contributing to this documentation

This page is auto-generated from the underlying source code. To make changes, please edit the relevant source files in the metadata-ingestion directory.

Tip: For quick typo fixes or documentation updates, you can click the ✏️ Edit icon directly in the GitHub UI to open a Pull Request. For larger changes and PR naming conventions, please refer to our Contributing Guide.

NiFi

Overview​

Concept Mapping​

Module nifi​

Important Capabilities​

Overview​

Prerequisites​

Access Policies​

Global Access Policies​

Component level Access Policies (required to be set on root process group)​

Authentication​

Single User Authentication (auth: SINGLE_USER)​

Client Certificates Authentication (auth: CLIENT_CERT)​

Kerberos Authentication via SPNEGO (auth: Kerberos)​

Basic Authentication (auth: BASIC_AUTH)​

No Authentication (auth: NO_AUTH)​

Install the Plugin​

Starter Recipe​

Config Details​

Capabilities​

Limitations​

Troubleshooting​

Code Coordinates​

Overview

Concept Mapping

Module `nifi`

Important Capabilities

Overview

Prerequisites

Access Policies

Global Access Policies

Component level Access Policies (required to be set on root process group)

Authentication

Single User Authentication (`auth: SINGLE_USER`)

Client Certificates Authentication (`auth: CLIENT_CERT`)

Kerberos Authentication via SPNEGO (`auth: Kerberos`)

Basic Authentication (`auth: BASIC_AUTH`)

No Authentication (`auth: NO_AUTH`)

Install the Plugin

Starter Recipe

Config Details

Capabilities

Limitations

Troubleshooting

Code Coordinates