Skip to main content

GitLab

Overview​

GitLab hosts source files, documentation, and knowledge bases in git repositories. Learn more in the official GitLab documentation.

The DataHub gitlab-documents integration ingests markdown and text files from a GitLab project on GitLab.com or self-managed GitLab as Document entities. Folder structure in the project is preserved as parent-child document relationships in DataHub.

Concept Mapping​

Source ConceptDataHub ConceptNotes
Project folderDocumentOptional folder documents for navigation.
Markdown / text fileDocumentNative (editable) or external (read-only) depending on config.
Project + branchCustom metadataStored on documents for traceability.

Module gitlab-documents​

Alpha

Important Capabilities​

CapabilityStatusNotes
Detect Deleted Entities✅Enabled by default via stateful ingestion.
Platform Instance✅Enabled by default.
Test Connection✅Enabled by default.

Overview​

The gitlab-documents source ingests files from a GitLab project branch using the GitLab REST API (v4). It works with GitLab.com and self-managed GitLab, and is designed for scheduled, repeatable imports of documentation into DataHub, such as context documents nested under a parent folder.

Prerequisites​

GitLab access token​

Create a personal, project, or group access token with the read_api scope. The read_repository scope is not enough, because the source reads the branch and repository tree through the API. Project and group access tokens need at least the Reporter role.

Store the token in a DataHub secret and reference it from your ingestion recipe.

Project access​

Set repository to the project's full path, including any subgroups (group/subgroup/project), or paste the project URL. For self-managed GitLab, set base_url to the instance URL, or use a project URL and the instance is inferred from it.

Supported imports​

  • Import .md, .txt, and other configured extensions
  • Preserve folder hierarchy as parent-child documents
  • Optionally attach imported trees under a parent document URN
  • Import as native (editable) or external (read-only) documents
  • Schedule recurring syncs via ingestion sources

Install the Plugin​

pip install 'acryl-datahub[gitlab-documents]'

Starter Recipe​

Check out the following recipe to get started with ingestion! See below for full configuration options.

For general pointers on writing and running a recipe, see our main recipe guide.

source:
type: gitlab-documents
config:
gitlab_token: "${GITLAB_TOKEN}"
# Omit base_url for GitLab.com.
base_url: https://gitlab.example.com
repository: acme/handbook
branch: main
path_prefix: docs
file_extensions:
- .md
- .txt
parent_document_urn: null
create_repo_root_document: true
max_files: 500
document_import_mode: NATIVE
show_in_global_context: true
stateful_ingestion:
enabled: true

sink:
type: datahub-rest
config:
server: "http://localhost:8080"

Config Details​

Note that a . is used to denote nested fields in the YAML recipe.

FieldDescription
gitlab_token ✅
string(password)
GitLab personal, project, or group access token with the read_api scope. Project and group tokens need at least the Reporter role.
repository ✅
string
Project to ingest, as its full path ('group/subgroup/project') or URL ('https://gitlab.com/group/project').
base_url
string
GitLab instance URL. Set this for self-managed GitLab, for example 'https://gitlab.example.com'. Inferred from repository when that is a URL.
branch
string
Branch to read files from.
Default: main
create_repo_root_document
boolean
When True, create a folder document named after the repository and nest imported files beneath it.
Default: True
document_import_mode
Enum
One of: "NATIVE", "EXTERNAL"
enable_okf
boolean
Opt in to importing OKF concepts inside discovered or configured bundles. Requires a server that supports the okfProperties aspect; enabling this on an unsupported server fails the run. Disabled by default to import every file verbatim without OKF metadata.
Default: False
include_organization_in_browse_path
boolean
Include the repository owner (GitHub organization, GitLab group path) as the top-most entry in the browse path, above the repository name.
Default: False
include_repository_in_browse_path
boolean
Include the repository name as a top-level entry in the browse path. When a repository root document is created, that document already provides the repository entry; this adds it for configurations that skip the root document or nest under a configured parent document.
Default: True
max_files
integer
Maximum number of matching files to import per run. Additional matches are skipped with a report warning.
Default: 500
parent_document_urn
One of string, null
Optional parent document URN. Top-level imported items are nested beneath this document while preserving the repository folder hierarchy.
Default: None
path_prefix
string
Only ingest files under this path (e.g. 'docs').
Default:
platform_instance
One of string, null
The instance of the platform that all assets produced by this recipe belong to. This should be unique within the platform. See https://docs.datahub.com/docs/platform-instances/ for more details.
Default: None
show_in_global_context
boolean
Whether imported documents appear in global search and navigation.
Default: True
file_extensions
array
File extensions to include (include the leading dot).
file_extensions.string
string
okf_bundle_roots
array
Additional bundle roots relative to the repository, for bundles without a versioned index.md. Use an empty string for the repository root. Directories with an index.md declaring okf_version are also discovered.
okf_bundle_roots.string
string
stateful_ingestion
One of StatefulStaleMetadataRemovalConfig, null
Stateful Ingestion Config
Default: None
stateful_ingestion.enabled
boolean
Whether or not to enable stateful ingest. Default: True if a pipeline_name is set and either a datahub-rest sink or datahub_api is specified, otherwise False
Default: False
stateful_ingestion.fail_safe_threshold
number
Prevents large amount of soft deletes & the state from committing from accidental changes to the source configuration if the relative change percent in entities compared to the previous state is above the 'fail_safe_threshold'.
Default: 75.0
stateful_ingestion.remove_stale_metadata
boolean
Soft-deletes the entities present in the last successful run but missing in the current run with stateful_ingestion enabled.
Default: True

Capabilities​

Context documents from self-managed GitLab​

source:
type: gitlab-documents
config:
gitlab_token: "${GITLAB_TOKEN}"
base_url: https://gitlab.example.com
repository: data-platform/analytics/handbook
branch: main
path_prefix: docs
parent_document_urn: "urn:li:document:context-handbook"
document_import_mode: NATIVE
stateful_ingestion:
enabled: true

sink:
type: datahub-rest
config:
server: "http://localhost:8080"

Read-only external references​

source:
type: gitlab-documents
config:
gitlab_token: "${GITLAB_TOKEN}"
repository: https://gitlab.com/example-group/analytics
document_import_mode: EXTERNAL
stateful_ingestion:
enabled: true

Documents link back to the file in GitLab (<project URL>/-/blob/<branch>/<path>), and folder documents link to the folder (/-/tree/).

Shared behaviour with github-documents​

This source shares its import logic with the GitHub documents source, so hashing, unchanged-file skipping, stateful soft-deletion, and Open Knowledge Format bundles (enable_okf: true) work the same way. The differences are:

  • Custom properties use the gitlab_ prefix: gitlab_repo, gitlab_branch, gitlab_file_path, gitlab_directory_path, gitlab_commit_sha, and gitlab_blob_sha.
  • The OKF bundle.location is the project URL, such as https://gitlab.com/example-group/analytics.
  • With include_organization_in_browse_path: true, the browse path starts with the full group path (data-platform/analytics).

Several GitLab instances or overlapping project paths​

Document IDs are built from the project path. Two sources whose projects produce the same IDs overwrite each other's documents unless each sets a different platform_instance. This happens in two cases:

  • The same project path on two GitLab instances, such as gitlab.com and a self-managed server. Give each instance its own name, such as gitlab-com and self-managed.
  • Project paths that differ only by . and /, such as data/platform.docs and data/platform/docs. IDs join path segments with ., so these two produce the same IDs even on one server. Give the two sources different instance names, such as self-managed-platform-docs and self-managed-platform. One name for the whole server is not enough here.
source:
type: gitlab-documents
config:
gitlab_token: "${GITLAB_TOKEN}"
base_url: https://gitlab.example.com
repository: data/platform.docs
platform_instance: self-managed

The instance is part of every document ID, and it becomes the documents' data platform instance. Without it, the data platform instance is the project path. Instance names cannot be blank or contain ., ,, ( or ); surrounding spaces are removed. Changing platform_instance on an existing source creates new documents; with stateful ingestion enabled, the old ones are soft-deleted.

The source resolves the branch head once and reads the tree and files at that commit, so all documents in a run come from one snapshot.

Limitations​

  • Maximum file size is 1 MB per file. Larger files are skipped with a Repository file not read warning.
  • Requires network access from the ingestion executor to the GitLab instance.
  • Binary formats are not parsed; use text/markdown files.
  • Imports are capped by max_files (default 500) per run.
  • Document IDs are built from the project path. Without platform_instance, two sources whose projects produce the same IDs overwrite each other's documents. See Several GitLab instances or overlapping project paths.
  • Changes made in DataHub are not written back to GitLab.

Troubleshooting​

  • 404 from GitLab: GitLab answers 404, not 403, for private projects the token cannot see. Check the project path (including subgroups), the branch name, and that the token's user or bot is a project member.
  • 401 / 403 from GitLab: Verify the token has not expired and has the read_api scope. Project and group tokens need at least the Reporter role.
  • Wrong instance: When repository is a URL and base_url is also set, the URL must be on that instance.
  • No files imported: Check path_prefix and file_extensions filters.
  • Repository file not read warnings: The file could not be downloaded, was too large, or was not text. A file that failed to download keeps its previous import, so a temporary GitLab error does not remove the document.
  • Repository tree truncated warning: GitLab returned a full page of the project tree without pagination headers, and a follow-up request showed more entries exist. This usually means a proxy stripped the headers. If the follow-up request failed, the warning's context includes the API error. The run imports what it listed and skips stale document removal, so nothing is deleted. Let the executor reach the GitLab API directly.
  • Redirect errors: The source does not follow redirects, so the token is never sent to another host. A redirect usually means a proxy or SSO login in front of the API, or a base_url that is not the address serving the API. Set base_url to that address.
  • Non-JSON or HTML response errors: A proxy or SSO login page answered instead of the GitLab API. Let the ingestion executor reach the API directly with the token.
  • Documents not removed after a GitLab delete: Ensure stateful_ingestion.enabled: true and the pipeline has a graph connection for checkpoint storage.

Code Coordinates​

  • Class Name: datahub.ingestion.source.gitlab_documents.gitlab_documents_source.GitLabDocumentsSource
  • Browse on GitHub
Questions?

If you've got any questions on configuring ingestion for GitLab, feel free to ping us on our Slack.

💡 Contributing to this documentation

This page is auto-generated from the underlying source code. To make changes, please edit the relevant source files in the metadata-ingestion directory.

Tip: For quick typo fixes or documentation updates, you can click the ✏️ Edit icon directly in the GitHub UI to open a Pull Request. For larger changes and PR naming conventions, please refer to our Contributing Guide.