GitLab
Overview
GitLab hosts source files, documentation, and knowledge bases in git repositories. Learn more in the official GitLab documentation.
The DataHub gitlab-documents integration ingests markdown and text files from a GitLab project on GitLab.com or self-managed GitLab as Document entities. Folder structure in the project is preserved as parent-child document relationships in DataHub.
Concept Mapping
| Source Concept | DataHub Concept | Notes |
|---|---|---|
| Project folder | Document | Optional folder documents for navigation. |
| Markdown / text file | Document | Native (editable) or external (read-only) depending on config. |
| Project + branch | Custom metadata | Stored on documents for traceability. |
Module gitlab-documents
Important Capabilities
| Capability | Status | Notes |
|---|---|---|
| Detect Deleted Entities | ✅ | Enabled by default via stateful ingestion. |
| Platform Instance | ✅ | Enabled by default. |
| Test Connection | ✅ | Enabled by default. |
Overview
The gitlab-documents source ingests files from a GitLab project branch using the GitLab REST API (v4). It works with GitLab.com and self-managed GitLab, and is designed for scheduled, repeatable imports of documentation into DataHub, such as context documents nested under a parent folder.
Prerequisites
GitLab access token
Create a personal, project, or group access token with the read_api scope. The read_repository scope is not enough, because the source reads the branch and repository tree through the API. Project and group access tokens need at least the Reporter role.
Store the token in a DataHub secret and reference it from your ingestion recipe.
Project access
Set repository to the project's full path, including any subgroups (group/subgroup/project), or paste the project URL. For self-managed GitLab, set base_url to the instance URL, or use a project URL and the instance is inferred from it.
Supported imports
- Import
.md,.txt, and other configured extensions - Preserve folder hierarchy as parent-child documents
- Optionally attach imported trees under a parent document URN
- Import as native (editable) or external (read-only) documents
- Schedule recurring syncs via ingestion sources
Install the Plugin
pip install 'acryl-datahub[gitlab-documents]'
Starter Recipe
Check out the following recipe to get started with ingestion! See below for full configuration options.
For general pointers on writing and running a recipe, see our main recipe guide.
source:
type: gitlab-documents
config:
gitlab_token: "${GITLAB_TOKEN}"
# Omit base_url for GitLab.com.
base_url: https://gitlab.example.com
repository: acme/handbook
branch: main
path_prefix: docs
file_extensions:
- .md
- .txt
parent_document_urn: null
create_repo_root_document: true
max_files: 500
document_import_mode: NATIVE
show_in_global_context: true
stateful_ingestion:
enabled: true
sink:
type: datahub-rest
config:
server: "http://localhost:8080"
Config Details
- Options
- Schema
Note that a . is used to denote nested fields in the YAML recipe.
| Field | Description |
|---|---|
gitlab_token ✅ string(password) | GitLab personal, project, or group access token with the read_api scope. Project and group tokens need at least the Reporter role. |
repository ✅ string | Project to ingest, as its full path ('group/subgroup/project') or URL ('https://gitlab.com/group/project'). |
base_url string | GitLab instance URL. Set this for self-managed GitLab, for example 'https://gitlab.example.com'. Inferred from repository when that is a URL. Default: https://gitlab.com |
branch string | Branch to read files from. Default: main |
create_repo_root_document boolean | When True, create a folder document named after the repository and nest imported files beneath it. Default: True |
document_import_mode Enum | One of: "NATIVE", "EXTERNAL" |
enable_okf boolean | Opt in to importing OKF concepts inside discovered or configured bundles. Requires a server that supports the okfProperties aspect; enabling this on an unsupported server fails the run. Disabled by default to import every file verbatim without OKF metadata. Default: False |
include_organization_in_browse_path boolean | Include the repository owner (GitHub organization, GitLab group path) as the top-most entry in the browse path, above the repository name. Default: False |
include_repository_in_browse_path boolean | Include the repository name as a top-level entry in the browse path. When a repository root document is created, that document already provides the repository entry; this adds it for configurations that skip the root document or nest under a configured parent document. Default: True |
max_files integer | Maximum number of matching files to import per run. Additional matches are skipped with a report warning. Default: 500 |
parent_document_urn One of string, null | Optional parent document URN. Top-level imported items are nested beneath this document while preserving the repository folder hierarchy. Default: None |
path_prefix string | Only ingest files under this path (e.g. 'docs'). Default: |
platform_instance One of string, null | The instance of the platform that all assets produced by this recipe belong to. This should be unique within the platform. See https://docs.datahub.com/docs/platform-instances/ for more details. Default: None |
show_in_global_context boolean | Whether imported documents appear in global search and navigation. Default: True |
file_extensions array | File extensions to include (include the leading dot). |
file_extensions.string string | |
okf_bundle_roots array | Additional bundle roots relative to the repository, for bundles without a versioned index.md. Use an empty string for the repository root. Directories with an index.md declaring okf_version are also discovered. |
okf_bundle_roots.string string | |
stateful_ingestion One of StatefulStaleMetadataRemovalConfig, null | Stateful Ingestion Config Default: None |
stateful_ingestion.enabled boolean | Whether or not to enable stateful ingest. Default: True if a pipeline_name is set and either a datahub-rest sink or datahub_api is specified, otherwise False Default: False |
stateful_ingestion.fail_safe_threshold number | Prevents large amount of soft deletes & the state from committing from accidental changes to the source configuration if the relative change percent in entities compared to the previous state is above the 'fail_safe_threshold'. Default: 75.0 |
stateful_ingestion.remove_stale_metadata boolean | Soft-deletes the entities present in the last successful run but missing in the current run with stateful_ingestion enabled. Default: True |
The JSONSchema for this configuration is inlined below.
{
"$defs": {
"DocumentImportMode": {
"description": "Whether ingested documents are native (editable) or external (read-only references).",
"enum": [
"NATIVE",
"EXTERNAL"
],
"title": "DocumentImportMode",
"type": "string"
},
"StatefulStaleMetadataRemovalConfig": {
"additionalProperties": false,
"description": "Base specialized config for Stateful Ingestion with stale metadata removal capability.",
"properties": {
"enabled": {
"default": false,
"description": "Whether or not to enable stateful ingest. Default: True if a pipeline_name is set and either a datahub-rest sink or `datahub_api` is specified, otherwise False",
"title": "Enabled",
"type": "boolean"
},
"remove_stale_metadata": {
"default": true,
"description": "Soft-deletes the entities present in the last successful run but missing in the current run with stateful_ingestion enabled.",
"title": "Remove Stale Metadata",
"type": "boolean"
},
"fail_safe_threshold": {
"default": 75.0,
"description": "Prevents large amount of soft deletes & the state from committing from accidental changes to the source configuration if the relative change percent in entities compared to the previous state is above the 'fail_safe_threshold'.",
"maximum": 100.0,
"minimum": 0.0,
"title": "Fail Safe Threshold",
"type": "number"
}
},
"title": "StatefulStaleMetadataRemovalConfig",
"type": "object"
}
},
"additionalProperties": false,
"description": "Configuration for ingesting markdown and text documents from a GitLab project.",
"properties": {
"platform_instance": {
"anyOf": [
{
"type": "string"
},
{
"type": "null"
}
],
"default": null,
"description": "The instance of the platform that all assets produced by this recipe belong to. This should be unique within the platform. See https://docs.datahub.com/docs/platform-instances/ for more details.",
"title": "Platform Instance"
},
"stateful_ingestion": {
"anyOf": [
{
"$ref": "#/$defs/StatefulStaleMetadataRemovalConfig"
},
{
"type": "null"
}
],
"default": null,
"description": "Stateful Ingestion Config"
},
"repository": {
"description": "Project to ingest, as its full path ('group/subgroup/project') or URL ('https://gitlab.com/group/project').",
"title": "Repository",
"type": "string"
},
"branch": {
"default": "main",
"description": "Branch to read files from.",
"title": "Branch",
"type": "string"
},
"path_prefix": {
"default": "",
"description": "Only ingest files under this path (e.g. 'docs').",
"title": "Path Prefix",
"type": "string"
},
"file_extensions": {
"description": "File extensions to include (include the leading dot).",
"items": {
"type": "string"
},
"title": "File Extensions",
"type": "array"
},
"max_files": {
"default": 500,
"description": "Maximum number of matching files to import per run. Additional matches are skipped with a report warning.",
"minimum": 1,
"title": "Max Files",
"type": "integer"
},
"create_repo_root_document": {
"default": true,
"description": "When True, create a folder document named after the repository and nest imported files beneath it.",
"title": "Create Repo Root Document",
"type": "boolean"
},
"parent_document_urn": {
"anyOf": [
{
"type": "string"
},
{
"type": "null"
}
],
"default": null,
"description": "Optional parent document URN. Top-level imported items are nested beneath this document while preserving the repository folder hierarchy.",
"title": "Parent Document Urn"
},
"document_import_mode": {
"$ref": "#/$defs/DocumentImportMode",
"default": "NATIVE",
"description": "NATIVE imports editable documents in DataHub. EXTERNAL imports read-only references that link back to the repository."
},
"show_in_global_context": {
"default": true,
"description": "Whether imported documents appear in global search and navigation.",
"title": "Show In Global Context",
"type": "boolean"
},
"enable_okf": {
"default": false,
"description": "Opt in to importing OKF concepts inside discovered or configured bundles. Requires a server that supports the okfProperties aspect; enabling this on an unsupported server fails the run. Disabled by default to import every file verbatim without OKF metadata.",
"title": "Enable Okf",
"type": "boolean"
},
"okf_bundle_roots": {
"description": "Additional bundle roots relative to the repository, for bundles without a versioned index.md. Use an empty string for the repository root. Directories with an index.md declaring okf_version are also discovered.",
"items": {
"type": "string"
},
"title": "Okf Bundle Roots",
"type": "array"
},
"include_repository_in_browse_path": {
"default": true,
"description": "Include the repository name as a top-level entry in the browse path. When a repository root document is created, that document already provides the repository entry; this adds it for configurations that skip the root document or nest under a configured parent document.",
"title": "Include Repository In Browse Path",
"type": "boolean"
},
"include_organization_in_browse_path": {
"default": false,
"description": "Include the repository owner (GitHub organization, GitLab group path) as the top-most entry in the browse path, above the repository name.",
"title": "Include Organization In Browse Path",
"type": "boolean"
},
"gitlab_token": {
"description": "GitLab personal, project, or group access token with the read_api scope. Project and group tokens need at least the Reporter role.",
"format": "password",
"title": "Gitlab Token",
"type": "string",
"writeOnly": true
},
"base_url": {
"default": "https://gitlab.com",
"description": "GitLab instance URL. Set this for self-managed GitLab, for example 'https://gitlab.example.com'. Inferred from repository when that is a URL.",
"title": "Base Url",
"type": "string"
}
},
"required": [
"repository",
"gitlab_token"
],
"title": "GitLabDocumentsSourceConfig",
"type": "object"
}
Capabilities
Context documents from self-managed GitLab
source:
type: gitlab-documents
config:
gitlab_token: "${GITLAB_TOKEN}"
base_url: https://gitlab.example.com
repository: data-platform/analytics/handbook
branch: main
path_prefix: docs
parent_document_urn: "urn:li:document:context-handbook"
document_import_mode: NATIVE
stateful_ingestion:
enabled: true
sink:
type: datahub-rest
config:
server: "http://localhost:8080"
Read-only external references
source:
type: gitlab-documents
config:
gitlab_token: "${GITLAB_TOKEN}"
repository: https://gitlab.com/example-group/analytics
document_import_mode: EXTERNAL
stateful_ingestion:
enabled: true
Documents link back to the file in GitLab (<project URL>/-/blob/<branch>/<path>), and folder documents link to the folder (/-/tree/).
Shared behaviour with github-documents
This source shares its import logic with the GitHub documents source, so hashing, unchanged-file skipping, stateful soft-deletion, and Open Knowledge Format bundles (enable_okf: true) work the same way. The differences are:
- Custom properties use the
gitlab_prefix:gitlab_repo,gitlab_branch,gitlab_file_path,gitlab_directory_path,gitlab_commit_sha, andgitlab_blob_sha. - The OKF
bundle.locationis the project URL, such ashttps://gitlab.com/example-group/analytics. - With
include_organization_in_browse_path: true, the browse path starts with the full group path (data-platform/analytics).
Several GitLab instances or overlapping project paths
Document IDs are built from the project path. Two sources whose projects produce the same IDs overwrite each other's documents unless each sets a different platform_instance. This happens in two cases:
- The same project path on two GitLab instances, such as gitlab.com and a self-managed server. Give each instance its own name, such as
gitlab-comandself-managed. - Project paths that differ only by
.and/, such asdata/platform.docsanddata/platform/docs. IDs join path segments with., so these two produce the same IDs even on one server. Give the two sources different instance names, such asself-managed-platform-docsandself-managed-platform. One name for the whole server is not enough here.
source:
type: gitlab-documents
config:
gitlab_token: "${GITLAB_TOKEN}"
base_url: https://gitlab.example.com
repository: data/platform.docs
platform_instance: self-managed
The instance is part of every document ID, and it becomes the documents' data platform instance. Without it, the data platform instance is the project path. Instance names cannot be blank or contain ., ,, ( or ); surrounding spaces are removed. Changing platform_instance on an existing source creates new documents; with stateful ingestion enabled, the old ones are soft-deleted.
The source resolves the branch head once and reads the tree and files at that commit, so all documents in a run come from one snapshot.
Limitations
- Maximum file size is 1 MB per file. Larger files are skipped with a
Repository file not readwarning. - Requires network access from the ingestion executor to the GitLab instance.
- Binary formats are not parsed; use text/markdown files.
- Imports are capped by
max_files(default 500) per run. - Document IDs are built from the project path. Without
platform_instance, two sources whose projects produce the same IDs overwrite each other's documents. See Several GitLab instances or overlapping project paths. - Changes made in DataHub are not written back to GitLab.
Troubleshooting
- 404 from GitLab: GitLab answers 404, not 403, for private projects the token cannot see. Check the project path (including subgroups), the branch name, and that the token's user or bot is a project member.
- 401 / 403 from GitLab: Verify the token has not expired and has the
read_apiscope. Project and group tokens need at least the Reporter role. - Wrong instance: When
repositoryis a URL andbase_urlis also set, the URL must be on that instance. - No files imported: Check
path_prefixandfile_extensionsfilters. Repository file not readwarnings: The file could not be downloaded, was too large, or was not text. A file that failed to download keeps its previous import, so a temporary GitLab error does not remove the document.Repository tree truncatedwarning: GitLab returned a full page of the project tree without pagination headers, and a follow-up request showed more entries exist. This usually means a proxy stripped the headers. If the follow-up request failed, the warning's context includes the API error. The run imports what it listed and skips stale document removal, so nothing is deleted. Let the executor reach the GitLab API directly.- Redirect errors: The source does not follow redirects, so the token is never sent to another host. A redirect usually means a proxy or SSO login in front of the API, or a
base_urlthat is not the address serving the API. Setbase_urlto that address. - Non-JSON or HTML response errors: A proxy or SSO login page answered instead of the GitLab API. Let the ingestion executor reach the API directly with the token.
- Documents not removed after a GitLab delete: Ensure
stateful_ingestion.enabled: trueand the pipeline has a graph connection for checkpoint storage.
Code Coordinates
- Class Name:
datahub.ingestion.source.gitlab_documents.gitlab_documents_source.GitLabDocumentsSource - Browse on GitHub
If you've got any questions on configuring ingestion for GitLab, feel free to ping us on our Slack.
This page is auto-generated from the underlying source code. To make changes, please edit the relevant source files in the metadata-ingestion directory.
Tip: For quick typo fixes or documentation updates, you can click the ✏️ Edit icon directly in the GitHub UI to open a Pull Request. For larger changes and PR naming conventions, please refer to our Contributing Guide.