Skip to main content
Version: Next

Grafana

Certified

Important Capabilities​

CapabilityStatusNotes
Column-level Lineage✅Enabled by default.
Detect Deleted Entities✅Enabled by default.
Extract Ownership✅Enabled by default.
Extract Tags✅Enabled by default.
Platform Instance✅Enabled by default.
Table-Level Lineage✅Enabled by default.

This plugin extracts metadata from Grafana and ingests it into DataHub. It connects to Grafana's API to extract metadata about dashboards, charts, and data sources. The following types of metadata are extracted:

  • Container Entities:

    • Folders: Top-level organizational units in Grafana
    • Dashboards: Collections of panels and charts
    • The full container hierarchy is preserved (Folders -> Dashboards -> Charts/Datasets)

  • Charts and Visualizations:

    • All panel types (graphs, tables, stat panels, etc.)
    • Chart configuration and properties
    • Links to the original Grafana UI
    • Custom properties including panel types and data source information
    • Input fields and schema information when available

  • Data Sources and Datasets:

    • Physical datasets representing Grafana's data sources
    • Dataset schema information extracted from queries and panel configurations
    • Support for various data source types (SQL, Prometheus, etc.)
    • Custom properties including data source type and configuration

  • Lineage Information:

    • Dataset-level lineage showing relationships between:
      • Source data systems and Grafana datasets
      • Grafana datasets and charts
    • Column-level lineage for SQL-based data sources
    • Support for external source systems through configurable platform mappings

  • Tags and Ownership:

    • Dashboard and chart tags
    • Ownership information derived from:
      • Dashboard creators
      • Technical owners based on dashboard UIDs
      • Custom ownership assignments

The source supports the following capabilities:

  • Platform instance support for multi-Grafana deployments
  • Stateful ingestion with support for soft-deletes
  • Fine-grained lineage at both dataset and column levels
  • Automated tag extraction
  • Support for both HTTP and HTTPS connections with optional SSL verification

Concept Mapping​

Source ConceptDataHub ConceptNotes
"grafana"Data Platform
Grafana FolderContainerSubtype Folder
Grafana DashboardContainerSubtype Dashboard
Grafana Panel/VisualizationChartVarious types mapped based on panel type (e.g., graph → LINE, pie → PIE)
Grafana Data SourceDatasetCreated for each panel's data source
Dashboard OwnerCorp UserDerived from dashboard UID and creator
Dashboard TagsTagSupports both simple tags and key:value tags

Compatibility​

The connector supports extracting metadata from any Grafana instance accessible via API. For SQL-based data sources, column-level lineage can be extracted when the queries are parseable. The connector supports various panel types and their transformations, and can work with both standalone Grafana instances and those integrated with other platforms.

For optimal lineage extraction from SQL-based data sources:

  • Database/schema information should be properly configured in the connection settings
  • The platform mapping (connection_to_platform_map) should be configured to match your data sources

Prerequisites:​

The Grafana source supports two extraction modes based on your permission level:

Enhanced Mode (Default)​

For full metadata extraction including lineage, containers, and detailed panel information:

  1. A running Grafana instance
  2. A service account token with Admin permissions to:
    • Read dashboards and folders
    • Access data source configurations
    • View user information
    • Access detailed dashboard metadata
    • Read panel configurations and transformations

Basic Mode (Limited Permissions)​

For users with limited permissions who only need basic dashboard metadata:

  1. A running Grafana instance
  2. A service account token with Viewer permissions to:
    • Read dashboards (via /api/search endpoint)
    • Basic dashboard metadata access

To enable basic mode, set basic_mode: true in your configuration. This provides backwards compatibility with the original simple connector behavior.

Note: Basic mode extracts only dashboard entities without folder hierarchy, panel details, lineage information, or schema metadata. It's recommended to use enhanced mode when possible for complete metadata extraction.

Configuration Examples​

Enhanced Mode (Default):

source:
  type: grafana
  config:
    url: "https://grafana.company.com"
    service_account_token: "your_admin_token"
    # basic_mode: false  # Default - full extraction

Basic Mode (Limited Permissions):

source:
  type: grafana
  config:
    url: "https://grafana.company.com"
    service_account_token: "your_viewer_token"
    basic_mode: true # Enable basic mode for limited permissions

Lineage Configuration​

The Grafana source can extract lineage information between charts and their data sources. You can control lineage extraction using these configuration options:

source:
  type: grafana
  config:
    url: "https://grafana.company.com"
    service_account_token: "your_token"

    # Lineage extraction (default: true)
    include_lineage: true

    # Column-level lineage from SQL queries (default: true)
    # Only applicable when include_lineage is true
    include_column_lineage: true

    # Platform mappings for lineage extraction
    connection_to_platform_map:
      postgres_datasource_uid:
        platform: postgres
        platform_instance: my_postgres
        env: PROD
        database: analytics
        database_schema: public

Lineage Features:

  • Dataset-level lineage: Links charts to their underlying data sources
  • Column-level lineage: Extracts field-to-field relationships from SQL queries
  • Platform mapping: Maps Grafana data sources to their actual platforms for accurate lineage
  • SQL parsing: Supports parsing of SQL queries for detailed lineage extraction

Performance Note: Lineage extraction can be disabled (include_lineage: false) to improve ingestion performance when lineage information is not needed.

CLI based Ingestion​

Starter Recipe​

Check out the following recipe to get started with ingestion! See below for full configuration options.

For general pointers on writing and running a recipe, see our main recipe guide.

source:
  type: grafana
  config:
    # Coordinates
    platform_instance: production # optional
    env: PROD # optional
    url: https://grafana.company.com
    service_account_token: ${GRAFANA_SERVICE_ACCOUNT_TOKEN}

    # SSL verification for HTTPS connections
    verify_ssl: true # optional, default is true

    # Source type mapping for lineage
    connection_to_platform_map:
      postgres:
        platform: postgres
        database: grafana # optional
        database_schema: grafana # optional
        platform_instance: database_2 # optional
        env: PROD # optional
      mysql_uid_1: # Grafana datasource UID
        platform: mysql
        platform_instance: database_1 # optional
        database: my_database # optional
sink:
  # sink configs

Config Details​

Note that a . is used to denote nested fields in the YAML recipe.

FieldDescription
service_account_token ✅
string(password)
Service account token for Grafana
url ✅
string
Grafana URL in the format http://your-grafana-instance with no trailing slash
basic_mode
boolean
Enable basic extraction mode for users with limited permissions. In basic mode, only dashboard metadata is extracted without detailed panel information, lineage, or folder hierarchy. This requires only basic dashboard read permissions.
Default: False
dashboard_pattern
AllowDenyPattern
Regex pattern to filter dashboards for ingestion
Default: {'allow': ['.*'], 'deny': [], 'ignoreCase': True}
folder_pattern
AllowDenyPattern
Regex pattern to filter folders for ingestion
Default: {'allow': ['.*'], 'deny': [], 'ignoreCase': True}
include_column_lineage
boolean
Whether to extract column-level lineage from SQL queries. Only applicable when include_lineage is enabled.
Default: True
include_lineage
boolean
Whether to extract lineage between charts and data sources. When enabled, the source will parse SQL queries and datasource configurations to build lineage relationships.
Default: True
ingest_owners
boolean
Whether to ingest dashboard ownership information
Default: True
ingest_tags
boolean
Whether to ingest dashboard and chart tags
Default: True
page_size
integer
Number of items to fetch per API call when paginating through folders and dashboards
Default: 100
platform_instance
string
The instance of the platform that all assets produced by this recipe belong to. This should be unique within the platform. See https://docs.datahub.com/docs/platform-instances/ for more details.
platform_instance_map
map(str,string)
verify_ssl
boolean
Whether to verify SSL certificates when connecting to Grafana
Default: True
env
string
The environment that all assets produced by this connector belong to
Default: PROD
connection_to_platform_map
map(str,PlatformConnectionConfig)
Platform connection configuration for mapping Grafana datasources to their actual platforms.
connection_to_platform_map.key.platform_instance
string
The instance of the platform that all assets produced by this recipe belong to. This should be unique within the platform. See https://docs.datahub.com/docs/platform-instances/ for more details.
connection_to_platform_map.key.platform ❓
string
The platform name (e.g., 'postgres', 'mysql', 'snowflake')
connection_to_platform_map.key.database
string
Default database name
connection_to_platform_map.key.database_schema
string
Default schema name
connection_to_platform_map.key.env
string
The environment that all assets produced by this connector belong to
Default: PROD
stateful_ingestion
StatefulIngestionConfig
Stateful Ingestion Config
stateful_ingestion.enabled
boolean
Whether or not to enable stateful ingest. Default: True if a pipeline_name is set and either a datahub-rest sink or datahub_api is specified, otherwise False
Default: False

Code Coordinates​

  • Class Name: datahub.ingestion.source.grafana.grafana_source.GrafanaSource
  • Browse on GitHub

Questions

If you've got any questions on configuring ingestion for Grafana, feel free to ping us on our Slack.