Skip to main content

Grafana

Overview​

Grafana is a business intelligence and analytics platform. Learn more in the official Grafana documentation.

The DataHub integration for Grafana covers BI entities such as dashboards, charts, datasets, and related ownership context. Depending on module capabilities, it can also capture features such as lineage, usage, profiling, ownership, tags, and stateful deletion detection.

Concept Mapping​

Source ConceptDataHub ConceptNotes
"grafana"Data Platform
Grafana FolderContainerSubtype Folder
Grafana DashboardContainerSubtype Dashboard
Grafana Panel/VisualizationChartVarious types mapped based on panel type (e.g., graph → LINE, pie → PIE)
Grafana Data SourceDatasetCreated for each panel's data source
Dashboard OwnerCorp UserDashboard creator assigned as TECHNICAL_OWNER; email suffix removal configurable via remove_email_suffix
Dashboard TagsTagSupports both simple tags and key:value tags

Module grafana​

Certified

Important Capabilities​

CapabilityStatusNotes
Column-level Lineage✅Enabled by default.
Detect Deleted Entities✅Enabled by default.
Extract Ownership✅Enabled by default.
Extract Tags✅Enabled by default.
Platform Instance✅Enabled by default.
Table-Level Lineage✅Enabled by default.

Overview​

The grafana module ingests metadata from Grafana into DataHub. It is intended for production ingestion workflows and module-specific capabilities are documented below.

Compatibility​

Supports any Grafana instance accessible via API. Extracts column-level lineage from parseable SQL queries in data sources.

For optimal SQL lineage extraction:

  • Configure database/schema information in data source connection settings
  • Set connection_to_platform_map to match your data sources

Extracted Metadata Scope​

The connector extracts metadata from Grafana APIs with support for:

  • Folder and dashboard container hierarchy
  • Panel and visualization entities (chart modeling)
  • Data source references for dataset linking
  • Dashboard ownership and tags
  • Optional table/column lineage from parseable SQL-based panels

Prerequisites​

The Grafana source supports two extraction modes based on your permission level:

Enhanced Mode (Default)​

For full metadata extraction including lineage, containers, and detailed panel information:

  1. A running Grafana instance
  2. A service account token with Admin permissions to:
    • Read dashboards and folders
    • Access data source configurations
    • View user information
    • Access detailed dashboard metadata
    • Read panel configurations and transformations

Basic Mode (Limited Permissions)​

For users with limited permissions who only need basic dashboard metadata:

  1. A running Grafana instance
  2. A service account token with Viewer permissions to:
    • Read dashboards (via /api/search endpoint)
    • Basic dashboard metadata access

To enable basic mode, set basic_mode: true in your configuration. This provides backwards compatibility with the original simple connector behavior.

Note: Basic mode extracts only dashboard entities without folder hierarchy, panel details, lineage information, or schema metadata. It's recommended to use enhanced mode when possible for complete metadata extraction.

Configuration Examples​

Enhanced Mode (Default):

source:
  type: grafana
  config:
    url: "https://grafana.company.com"
    service_account_token: "your_admin_token"
    # basic_mode: false  # Default - full extraction

Basic Mode (Limited Permissions):

source:
  type: grafana
  config:
    url: "https://grafana.company.com"
    service_account_token: "your_viewer_token"
    basic_mode: true # Enable basic mode for limited permissions

Install the Plugin​

pip install 'acryl-datahub[grafana]'

Starter Recipe​

Check out the following recipe to get started with ingestion! See below for full configuration options.

For general pointers on writing and running a recipe, see our main recipe guide.

source:
  type: grafana
  config:
    # Coordinates
    platform_instance: production # optional
    env: PROD # optional
    url: https://grafana.company.com
    service_account_token: ${GRAFANA_SERVICE_ACCOUNT_TOKEN}

    # SSL verification for HTTPS connections
    verify_ssl: true # optional, default is true

    # Ownership configuration
    ingest_owners: true # optional, default is true - extract dashboard ownership
    remove_email_suffix: true # optional, default is true - remove email suffix like @acryl.io

    # Source type mapping for lineage
    connection_to_platform_map:
      postgres:
        platform: postgres
        database: grafana # optional
        database_schema: grafana # optional
        platform_instance: database_2 # optional
        env: PROD # optional
      mysql_uid_1: # Grafana datasource UID
        platform: mysql
        platform_instance: database_1 # optional
        database: my_database # optional
sink:
  # sink configs

Config Details​

Note that a . is used to denote nested fields in the YAML recipe.

FieldDescription
service_account_token ✅
string(password)
Service account token for Grafana
url ✅
string
Grafana URL in the format http://your-grafana-instance with no trailing slash
basic_mode
boolean
Enable basic extraction mode for users with limited permissions. In basic mode, only dashboard metadata is extracted without detailed panel information, lineage, or folder hierarchy. This requires only basic dashboard read permissions.
Default: False
include_column_lineage
boolean
Whether to extract column-level lineage from SQL queries. Only applicable when include_lineage is enabled.
Default: True
include_lineage
boolean
Whether to extract lineage between charts and data sources. When enabled, the source will parse SQL queries and datasource configurations to build lineage relationships.
Default: True
ingest_owners
boolean
Whether to ingest dashboard ownership information
Default: True
ingest_tags
boolean
Whether to ingest dashboard and chart tags
Default: True
page_size
integer
Number of items to fetch per API call when paginating through folders and dashboards
Default: 100
platform_instance
One of string, null
The instance of the platform that all assets produced by this recipe belong to. This should be unique within the platform. See https://docs.datahub.com/docs/platform-instances/ for more details.
Default: None
platform_instance_map
One of string, null
A holder for platform -> platform_instance mappings to generate correct dataset urns
Default: None
remove_email_suffix
boolean
Remove Grafana user email suffix for example, @acryl.io, when assigning ownership.
Default: True
skip_text_panels
boolean
Whether to skip text panels during ingestion. Text panels don't contain data visualizations and may not be relevant for data lineage.
Default: False
verify_ssl
boolean
Whether to verify SSL certificates when connecting to Grafana
Default: True
env
string
The environment that all assets produced by this connector belong to
Default: PROD
connection_to_platform_map
map(str,PlatformConnectionConfig)
Platform connection configuration for mapping Grafana datasources to their actual platforms.
connection_to_platform_map.key.platform_instance
One of string, null
The instance of the platform that all assets produced by this recipe belong to. This should be unique within the platform. See https://docs.datahub.com/docs/platform-instances/ for more details.
Default: None
connection_to_platform_map.key.platform ❓
string
The platform name (e.g., 'postgres', 'mysql', 'snowflake')
connection_to_platform_map.key.database
One of string, null
Default database name
Default: None
connection_to_platform_map.key.database_schema
One of string, null
Default schema name
Default: None
connection_to_platform_map.key.env
string
The environment that all assets produced by this connector belong to
Default: PROD
dashboard_pattern
AllowDenyPattern
A class to store allow deny regexes
dashboard_pattern.ignoreCase
One of boolean, null
Whether to ignore case sensitivity during pattern matching.
Default: True
folder_pattern
AllowDenyPattern
A class to store allow deny regexes
folder_pattern.ignoreCase
One of boolean, null
Whether to ignore case sensitivity during pattern matching.
Default: True
stateful_ingestion
One of StatefulStaleMetadataRemovalConfig, null
Stateful ingestion configuration
Default: None
stateful_ingestion.enabled
boolean
Whether or not to enable stateful ingest. Default: True if a pipeline_name is set and either a datahub-rest sink or datahub_api is specified, otherwise False
Default: False
stateful_ingestion.fail_safe_threshold
number
Prevents large amount of soft deletes & the state from committing from accidental changes to the source configuration if the relative change percent in entities compared to the previous state is above the 'fail_safe_threshold'.
Default: 75.0
stateful_ingestion.remove_stale_metadata
boolean
Soft-deletes the entities present in the last successful run but missing in the current run with stateful_ingestion enabled.
Default: True

Capabilities​

Use the Important Capabilities table above as the source of truth for supported features and whether additional configuration is required.

Lineage​

The Grafana source can extract lineage information between charts and their data sources. You can control lineage extraction using these configuration options:

source:
  type: grafana
  config:
    url: "https://grafana.company.com"
    service_account_token: "your_token"

    # Lineage extraction (default: true)
    include_lineage: true

    # Column-level lineage from SQL queries (default: true)
    # Only applicable when include_lineage is true
    include_column_lineage: true

    # Platform mappings for lineage extraction
    connection_to_platform_map:
      postgres_datasource_uid:
        platform: postgres
        platform_instance: my_postgres
        env: PROD
        database: analytics
        database_schema: public

Lineage Features:

  • Dataset-level lineage: Links charts to their underlying data sources
  • Column-level lineage: Extracts field-to-field relationships from SQL queries
  • Platform mapping: Maps Grafana data sources to their actual platforms for accurate lineage
  • SQL parsing: Supports parsing of SQL queries for detailed lineage extraction

Performance Note: Lineage extraction can be disabled (include_lineage: false) to improve ingestion performance when lineage information is not needed.

Ownership​

The Grafana source extracts dashboard ownership from the dashboard creator and assigns them as a Technical Owner.

source:
  type: grafana
  config:
    url: "https://grafana.company.com"
    service_account_token: "your_token"

    # Ownership extraction (default: true)
    ingest_owners: true

    # Email suffix removal like @acryl.io (default: true)
    remove_email_suffix: true

Ownership Features:

  • Technical Owner assignment: Dashboard creators are automatically assigned as Technical Owners
  • Email suffix control: Configure how user email addresses are converted to DataHub user URNs via remove_email_suffix
  • Disable ownership: Set ingest_owners: false to skip ownership extraction entirely

Limitations​

Module behavior is constrained by source APIs, permissions, and metadata exposed by the platform. Refer to capability notes for unsupported or conditional features.

Troubleshooting​

If ingestion fails, validate credentials, permissions, connectivity, and scope filters first. Then review ingestion logs for source-specific errors and adjust configuration accordingly.

Code Coordinates​

  • Class Name: datahub.ingestion.source.grafana.grafana_source.GrafanaSource
  • Browse on GitHub
Questions?

If you've got any questions on configuring ingestion for Grafana, feel free to ping us on our Slack.

💡 Contributing to this documentation

This page is auto-generated from the underlying source code. To make changes, please edit the relevant source files in the metadata-ingestion directory.

Tip: For quick typo fixes or documentation updates, you can click the ✏️ Edit icon directly in the GitHub UI to open a Pull Request. For larger changes and PR naming conventions, please refer to our Contributing Guide.