Skip to main content

Google Drive

Overview​

Google Drive is Google Workspace's file storage and collaboration service, and Google Docs is its document editor. Learn more in the official Google Drive documentation.

The DataHub integration for Google Drive ingests Google Docs from the folders and shared drives you choose as DataHub Document entities. It keeps the folder hierarchy, owners, and links back to Drive, generates semantic-search embeddings inline, and supports stateful deletion detection.

Concept Mapping​

Source ConceptDataHub ConceptNotes
Configured folder/shared driveDocument (subtype Folder)Emitted when create_root_documents is enabled (the default).
FolderDocument (subtype Folder)Empty body; parent of the documents and folders it contains.
Google DocDocumentBody is the Doc exported as Markdown; with the default EXTERNAL import mode, links back to Drive via its external URL.
File ownerCorpUser (owner)Keyed by email address. Files in shared drives have no owners.
Folder pathBrowse path, parent documentEvery ancestor folder is a clickable browse-path entry; with create_root_documents: false, the root is a plain label.
Google WorkspacePlatform instance (optional)Set platform_instance to distinguish workspaces; URNs use globally unique file IDs.

Module google-drive​

Alpha

Important Capabilities​

CapabilityStatusNotes
Detect Deleted Entities✅Enabled via stateful ingestion.
Extract Ownership✅Enabled by default via extract_owners.
Platform Instance✅Enabled by default.
Test Connection✅Enabled by default.

Overview​

The google-drive module walks the folders and shared drives listed in root_folder_ids, exports every Google Doc it finds as Markdown, and ingests each one as a DataHub Document. Folders become Documents with the Folder subtype, so the Drive hierarchy is preserved for navigation.

This source ingests Google Docs only. Sheets, Slides, PDFs, and other file types in the same folders are ignored, as are shortcuts and files in the trash.

Prerequisites​

1. Enable the Google Drive API​

In the Google Cloud project that owns your service account, enable the Google Drive API (APIs & Services → Library → Google Drive API).

2. Create a service account​

Create a service account and a JSON key for it. The connector takes the key's fields under credential, the same shape the BigQuery source uses.

If you omit credential, the connector uses Application Default Credentials. Those credentials must carry the https://www.googleapis.com/auth/drive.readonly scope; the cloud-platform scope alone does not grant Drive access. On Google Cloud, attach the service account to the VM or workload and grant it folder access as described below.

To test locally with your own Google account instead of a service account key:

  1. In your Google Cloud project, configure the OAuth consent screen (APIs & Services → OAuth consent screen). If your account is in a Google Workspace organization, choose Internal.

  2. Create an OAuth client ID of type Desktop app (APIs & Services → Credentials) and download its JSON file. The gcloud CLI's built-in OAuth client cannot request the Drive scope, so you need your own.

  3. Log in with that client:

    gcloud auth application-default login \
      --client-id-file=<path-to-oauth-client.json> \
      --scopes=https://www.googleapis.com/auth/drive.readonly,https://www.googleapis.com/auth/cloud-platform

The connector then reads Drive as you, so it sees the folders your account can open. delegated_user is not supported with these credentials.

3. Grant access to the folders​

Choose one of the following:

  • Share the folders with the service account. Share each root folder (or shared drive) with the service account's client_email as a Viewer. The connector then sees exactly what is shared with it.
  • Use domain-wide delegation. In the Google Workspace Admin console (Security → API controls → Domain-wide delegation), authorize the service account's client ID for the scope https://www.googleapis.com/auth/drive.readonly. Then set delegated_user to a Workspace user who can read the folders, and the connector reads Drive as that user.

The connector only requests the read-only Drive scope.

caution

DataHub does not mirror Google Drive permissions. Anyone who can read a Document in DataHub can read its text, so choose root folders whose audience matches your DataHub audience.

4. Find the folder IDs​

Open the folder in Drive and copy the ID from the URL (https://drive.google.com/drive/folders/<folder-id>). You can also paste the full folder URL into root_folder_ids. For a shared drive, use the ID from the shared drive's URL.

5. Embedding provider (optional)​

By default the connector loads the embedding configuration from your DataHub server and generates semantic-search embeddings for every Doc. The machine that runs ingestion needs credentials for that embedding provider (for example, AWS credentials for Amazon Bedrock). Without them, Docs are ingested without embeddings and are exported again on every run until embedding succeeds. See Semantic Search Configuration for details.

6. DataHub permissions​

The DataHub access token used by the ingestion recipe must be allowed to read, create, and edit Document entities. The source reads Documents it ingested earlier to skip Docs that have not changed; without read access, every Doc is exported again on every run.

7. Python version​

The google-drive plugin requires Python 3.11 or later.

Install the Plugin​

pip install 'acryl-datahub[google-drive]'

Starter Recipe​

Check out the following recipe to get started with ingestion! See below for full configuration options.

For general pointers on writing and running a recipe, see our main recipe guide.

source:
  type: google-drive
  config:
    # Service account key fields. Omit `credential` to use Application Default Credentials.
    credential:
      project_id: my-project
      private_key_id: "${GDRIVE_PRIVATE_KEY_ID}"
      private_key: "${GDRIVE_PRIVATE_KEY}"
      client_email: datahub-ingest@my-project.iam.gserviceaccount.com
      client_id: "123456789012345678901"

    # Optional: read Drive as this Workspace user (requires domain-wide delegation).
    # delegated_user: docs-reader@example.com

    # Folder IDs, shared drive IDs, or folder URLs.
    root_folder_ids:
      - "https://drive.google.com/drive/folders/1AbCdEfGhIjKlMnOpQrStUv"

    folder_pattern:
      deny:
        - ".*/Archive$"
    document_pattern:
      deny:
        - "^Copy of "

    stateful_ingestion:
      enabled: true
      remove_stale_metadata: true

sink:
  # sink configs

Config Details​

Note that a . is used to denote nested fields in the YAML recipe.

FieldDescription
root_folder_ids ✅
array
Google Drive folder IDs, shared drive IDs, or Drive folder URLs to ingest. Google Docs under each root are ingested, recursively by default.
root_folder_ids.string
string
continue_on_failure
boolean
Whether to keep ingesting other documents when one document fails. Set to false to abort the run on the first per-document error.
Default: True
create_root_documents
boolean
Emit a folder document for each configured root. When false, items directly inside a root attach to parent_document_urn (or become top-level documents).
Default: True
delegated_user
One of string, null
Email of a Google Workspace user to impersonate via domain-wide delegation. Without it, the service account only sees files that are explicitly shared with its client_email. Requires credential.
Default: None
document_import_mode
Enum
One of: "NATIVE", "EXTERNAL"
extract_owners
boolean
Map Google Drive file owners to DataHub owners (users keyed by email address). Files in shared drives have no owners.
Default: True
max_documents
integer
Maximum number of documents to embed per ingestion run. The run fails once this limit is reached. Set to 0 or -1 to disable the limit.
Default: 10000
max_text_length
integer
Skip Google Docs whose exported text (after images are replaced with placeholders) is longer than this many characters. A skipped document that was ingested earlier is soft-deleted by stale metadata removal. 0 disables the limit.
Default: 1000000
min_text_length
integer
Skip Google Docs whose exported text is shorter than this many characters. 0 keeps every non-empty document.
Default: 0
parent_document_urn
One of string, null
Optional URN of an existing DataHub document to place the imported folder tree under.
Default: None
platform_instance
One of string, null
Optional identifier for this Google Workspace, attached to every ingested document. Document URNs do not depend on it because Drive file IDs are globally unique.
Default: None
recursive
boolean
Whether to walk subfolders. When false, only Google Docs directly inside each root are ingested and no subfolder documents are emitted.
Default: True
skip_empty_documents
boolean
Skip Google Docs whose exported text is empty.
Default: True
chunking
ChunkingConfig
Chunking strategy configuration.
chunking.combine_text_under_n_chars
integer
Combine chunks smaller than this size
Default: 100
chunking.max_characters
integer
Maximum characters per chunk
Default: 500
chunking.max_chunks_per_document
integer
Maximum number of chunks embedded per document. A document that produces more chunks is truncated to the first N so its semanticContent aspect stays under the Kafka producer size limit; the dropped chunks are reported as a warning, not a failure.
Default: 100
chunking.overlap
integer
Character overlap between chunks
Default: 0
chunking.strategy
Enum
One of: "basic", "by_title"
Default: by_title
credential
One of GCPCredential, null
Service account key used to call the Google Drive API. If omitted, Application Default Credentials are used.
Default: None
credential.client_email ❓
string
Client email
credential.client_id ❓
string
Client Id
credential.private_key ❓
string(password)
Private key in a form of '-----BEGIN PRIVATE KEY-----\nprivate-key\n-----END PRIVATE KEY-----\n'
credential.private_key_id ❓
string
Private key id
credential.auth_provider_x509_cert_url
string
Auth provider x509 certificate url
Default: https://www.googleapis.com/oauth2/v1/certs
credential.auth_uri
string
Authentication uri
Default: https://accounts.google.com/o/oauth2/auth
credential.client_x509_cert_url
One of string, null
If not set it will be default to https://www.googleapis.com/robot/v1/metadata/x509/client_email
Default: None
credential.project_id
One of string, null
Project id to set the credentials
Default: None
credential.token_uri
string
Token uri
Default: https://oauth2.googleapis.com/token
credential.type
string
Authentication type
Default: service_account
document_pattern
AllowDenyPattern
A class to store allow deny regexes.

Patterns are matched against the start of the string only, not the entire
string - a pattern does not need to match to the end to be considered a match.
For example, the pattern "prod" matches "prod", "prod_east", and "production".
To require an exact match, anchor your pattern explicitly, e.g. "^prod$".
document_pattern.ignoreCase
One of boolean, null
Whether to ignore case sensitivity during pattern matching.
Default: True
embedding
EmbeddingConfig
Embedding generation configuration.

Default behavior: Fetches configuration from DataHub server automatically.
Override behavior: Validates local config against server when explicitly set.
embedding.allow_local_embedding_config
boolean
BREAK-GLASS: Allow local config without server validation. NOT RECOMMENDED - may break semantic search.
Default: False
embedding.api_key
One of string(password), null
API key for Cohere (not needed for Bedrock with IAM roles)
Default: None
embedding.aws_region
One of string, null
AWS region for Bedrock. If not set, loads from server.
Default: None
embedding.batch_size
integer
Batch size for embedding API calls
Default: 25
embedding.documents_per_minute
integer
Maximum number of documents to embed per minute when rate_limit is enabled.
Default: 300
embedding.endpoint
One of string, null
Endpoint URL for local embedding server (e.g., http://localhost:11434/v1/embeddings). Only used when provider='local'. Falls back to LOCAL_EMBEDDING_ENDPOINT env var.
Default: None
embedding.input_type
One of string, null
Input type for Cohere embeddings
Default: search_document
embedding.model
One of string, null
Model name. If not set, loads from server.
Default: None
embedding.model_embedding_key
One of string, null
Storage key for embeddings (e.g., 'cohere_embed_v3'). Required if overriding server config. If not set, loads from server.
Default: None
embedding.onnx_model_dir
One of string, null
Directory containing model.onnx (or model_quantized.onnx) and tokenizer.json. Only used when provider='onnx'. Falls back to ONNX_EMBEDDING_MODEL_DIR env var. Must be the same model the GMS query-side provider uses, or semantic search will not match.
Default: None
embedding.onnx_pooling
string
Pooling strategy for provider='onnx': 'cls' (first-token, default) or 'mean' (attention-masked mean). Must match the GMS provider's pooling for query/doc vector parity. When config is loaded from the server, falls back to the ONNX_EMBEDDING_POOLING env var (the same variable GMS reads), since the server API does not expose pooling.
Default: cls
embedding.provider
One of Enum, null
Embedding provider. 'local' calls a locally-running OpenAI-compatible server (e.g. Ollama). 'vertex_ai' uses GCP. 'onnx' runs a local ONNX model in-process (matches the GMS built-in provider). 'classical' is a deterministic hashed-feature embedding with no external service (matches the GMS built-in provider). If not set, loads from server.
Default: None
embedding.rate_limit
boolean
Enable rate limiting for embedding API calls.
Default: True
embedding.request_timeout
number
Per-call HTTP timeout in seconds for embedding providers.
Default: 60.0
embedding.vertex_location
One of string, null
GCP region for the Vertex AI endpoint (e.g., 'us-east1', 'us-central1'). Matches the Java server default.
Default: us-east1
embedding.vertex_project_id
One of string, null
GCP project ID for Vertex AI. Required when provider is 'vertex_ai' and using local config. If not set, loads from server.
Default: None
folder_pattern
AllowDenyPattern
A class to store allow deny regexes.

Patterns are matched against the start of the string only, not the entire
string - a pattern does not need to match to the end to be considered a match.
For example, the pattern "prod" matches "prod", "prod_east", and "production".
To require an exact match, anchor your pattern explicitly, e.g. "^prod$".
folder_pattern.ignoreCase
One of boolean, null
Whether to ignore case sensitivity during pattern matching.
Default: True
stateful_ingestion
One of StatefulStaleMetadataRemovalConfig, null
Stateful Ingestion Config
Default: None
stateful_ingestion.enabled
boolean
Whether or not to enable stateful ingest. Default: True if a pipeline_name is set and either a datahub-rest sink or datahub_api is specified, otherwise False
Default: False
stateful_ingestion.fail_safe_threshold
number
Prevents large amount of soft deletes & the state from committing from accidental changes to the source configuration if the relative change percent in entities compared to the previous state is above the 'fail_safe_threshold'.
Default: 75.0
stateful_ingestion.remove_stale_metadata
boolean
Soft-deletes the entities present in the last successful run but missing in the current run with stateful_ingestion enabled.
Default: True

Capabilities​

Common Use Cases​

Ingest a shared drive under an existing DataHub document​
source:
  type: google-drive
  config:
    root_folder_ids:
      - "0AXyZsharedDriveId00"
    parent_document_urn: "urn:li:document:team-handbook"
Ingest editable copies instead of read-only references​

document_import_mode: NATIVE imports Docs as editable DataHub documents. The default, EXTERNAL, imports read-only references that link back to Google Drive.

Ingest only the top level of a folder​
source:
  type: google-drive
  config:
    root_folder_ids:
      - "1AbCdEfGhIjKlMnOpQrStUv"
    recursive: false

Filtering​

  • folder_pattern regexes are matched against the folder path, starting with the root folder's name (for example Handbook/Engineering/Archive). A denied folder is never listed, so nothing beneath it is ingested. Configured roots are never filtered.
  • document_pattern regexes are matched against Doc titles.

How Change Detection Works​

On each run the connector lists every configured folder, then compares each Doc's Drive version, title, parent folder, and the connector settings that shape its output with what DataHub already stores. Unchanged Docs are not exported again, which keeps re-runs cheap. Docs that were deleted, trashed, moved out of scope, or newly filtered out are soft-deleted when stateful ingestion is enabled.

Limitations​

  • Only Google Docs are ingested. Sheets, Slides, PDFs, uploaded files, and shortcuts are skipped.
  • Images are not ingested. Each embedded image is replaced with a text placeholder such as [Image: <alt text>].
  • Docs with multiple tabs are ingested as a single Document. Each tab begins with its title as a heading.
  • Comments, suggestions, and some smart chips are not part of Google's Markdown export.
  • Google Drive refuses to export Docs larger than 10 MB. Those Docs are skipped with a warning.
  • Docs whose text is longer than max_text_length characters (default 1,000,000) are skipped with a warning. If such a Doc was ingested earlier and stateful ingestion is enabled, it is soft-deleted. Set max_text_length: 0 to remove the limit.
  • A run stops with a failure once max_documents Docs (default 10,000) have been embedded. Raise the limit, or set it to 0 to remove it, for larger folders.
  • Permissions are not mirrored. Anyone who can read the Document in DataHub can read its text, so choose roots whose audience matches your DataHub audience.
  • Files in shared drives have no owners, so they are ingested without ownership.
  • Drive owners are added alongside any owners you assign in DataHub. When a Doc's Drive owner changes, only the owner this source added is removed.
  • Each run lists every folder in scope. Unchanged Docs are not exported again, and unchanged folders are not written again.
  • Re-ingesting a changed Doc or folder replaces its related assets and related documents, so links you add to them in DataHub are not kept.
  • Do not point two recipes at overlapping folders. With stateful ingestion enabled, one recipe can soft-delete Documents that the other still ingests.

Troubleshooting​

A root folder is reported as not accessible​

Either the ID is wrong or the source cannot see the folder.

  • Check that the ID or URL in root_folder_ids points to a folder or shared drive, not a file.
  • Share the folder with the account the source authenticates as: the service account's client_email when you set credential, or the Application Default Credentials account otherwise. Alternatively, set delegated_user to a user who can open it.

Run datahub check source-connection to see which roots fail.

When any root or folder fails, the run reports a failure and stale-entity removal is skipped for that run, so nothing is soft-deleted because of a partial listing.

Docs are missing​

  • Check documents_filtered, folders_filtered, documents_skipped_empty, documents_skipped_too_long, documents_export_failed, documents_unparseable, documents_failed, and shortcuts_skipped in the ingestion report. failed_documents lists the Docs that could not be ingested; their existing copies in DataHub are kept.
  • Without delegated_user, the service account only sees files shared with it. Files that a folder's owner can see are not automatically visible to it.
  • Docs in the trash are never ingested.

Embeddings are missing for some Docs​

If embedding generation fails, the Doc is still ingested and the failure is reported as a warning. The Doc is retried on the next run. documents_embedding_deferred counts these Docs, and num_documents_truncated_oversized and num_documents_dropped_oversized count Docs whose embeddings were truncated or dropped for size.

Every run exports every Doc again​

Unchanged Docs are skipped by comparing them with what DataHub already stores. If change_detection_lookup_failures is non-zero, the ingestion token could not read Documents back from DataHub; grant it read access to Document entities.

Code Coordinates​

  • Class Name: datahub.ingestion.source.google_drive.google_drive_source.GoogleDriveSource
  • Browse on GitHub
Questions?

If you've got any questions on configuring ingestion for Google Drive, feel free to ping us on our Slack.

💡 Contributing to this documentation

This page is auto-generated from the underlying source code. To make changes, please edit the relevant source files in the metadata-ingestion directory.

Tip: For quick typo fixes or documentation updates, you can click the ✏️ Edit icon directly in the GitHub UI to open a Pull Request. For larger changes and PR naming conventions, please refer to our Contributing Guide.