Grafana
Important Capabilities
Capability | Status | Notes |
---|---|---|
Column-level Lineage | ✅ | Enabled by default. |
Detect Deleted Entities | ✅ | Enabled by default. |
Extract Ownership | ✅ | Enabled by default. |
Extract Tags | ✅ | Enabled by default. |
Platform Instance | ✅ | Enabled by default. |
Table-Level Lineage | ✅ | Enabled by default. |
This plugin extracts metadata from Grafana and ingests it into DataHub. It connects to Grafana's API to extract metadata about dashboards, charts, and data sources. The following types of metadata are extracted:
Container Entities:
- Folders: Top-level organizational units in Grafana
- Dashboards: Collections of panels and charts
- The full container hierarchy is preserved (Folders -> Dashboards -> Charts/Datasets)
Charts and Visualizations:
- All panel types (graphs, tables, stat panels, etc.)
- Chart configuration and properties
- Links to the original Grafana UI
- Custom properties including panel types and data source information
- Input fields and schema information when available
Data Sources and Datasets:
- Physical datasets representing Grafana's data sources
- Dataset schema information extracted from queries and panel configurations
- Support for various data source types (SQL, Prometheus, etc.)
- Custom properties including data source type and configuration
Lineage Information:
- Dataset-level lineage showing relationships between:
- Source data systems and Grafana datasets
- Grafana datasets and charts
- Column-level lineage for SQL-based data sources
- Support for external source systems through configurable platform mappings
- Dataset-level lineage showing relationships between:
Tags and Ownership:
- Dashboard and chart tags
- Ownership information derived from:
- Dashboard creators
- Technical owners based on dashboard UIDs
- Custom ownership assignments
The source supports the following capabilities:
- Platform instance support for multi-Grafana deployments
- Stateful ingestion with support for soft-deletes
- Fine-grained lineage at both dataset and column levels
- Automated tag extraction
- Support for both HTTP and HTTPS connections with optional SSL verification
Concept Mapping
Source Concept | DataHub Concept | Notes |
---|---|---|
"grafana" | Data Platform | |
Grafana Folder | Container | Subtype Folder |
Grafana Dashboard | Container | Subtype Dashboard |
Grafana Panel/Visualization | Chart | Various types mapped based on panel type (e.g., graph → LINE, pie → PIE) |
Grafana Data Source | Dataset | Created for each panel's data source |
Dashboard Owner | Corp User | Derived from dashboard UID and creator |
Dashboard Tags | Tag | Supports both simple tags and key:value tags |
Compatibility
The connector supports extracting metadata from any Grafana instance accessible via API. For SQL-based data sources, column-level lineage can be extracted when the queries are parseable. The connector supports various panel types and their transformations, and can work with both standalone Grafana instances and those integrated with other platforms.
For optimal lineage extraction from SQL-based data sources:
- Database/schema information should be properly configured in the connection settings
- The platform mapping (
connection_to_platform_map
) should be configured to match your data sources
Prerequisites:
The Grafana source supports two extraction modes based on your permission level:
Enhanced Mode (Default)
For full metadata extraction including lineage, containers, and detailed panel information:
- A running Grafana instance
- A service account token with Admin permissions to:
- Read dashboards and folders
- Access data source configurations
- View user information
- Access detailed dashboard metadata
- Read panel configurations and transformations
Basic Mode (Limited Permissions)
For users with limited permissions who only need basic dashboard metadata:
- A running Grafana instance
- A service account token with Viewer permissions to:
- Read dashboards (via
/api/search
endpoint) - Basic dashboard metadata access
- Read dashboards (via
To enable basic mode, set basic_mode: true
in your configuration. This provides backwards compatibility with the original simple connector behavior.
Note: Basic mode extracts only dashboard entities without folder hierarchy, panel details, lineage information, or schema metadata. It's recommended to use enhanced mode when possible for complete metadata extraction.
Configuration Examples
Enhanced Mode (Default):
source:
type: grafana
config:
url: "https://grafana.company.com"
service_account_token: "your_admin_token"
# basic_mode: false # Default - full extraction
Basic Mode (Limited Permissions):
source:
type: grafana
config:
url: "https://grafana.company.com"
service_account_token: "your_viewer_token"
basic_mode: true # Enable basic mode for limited permissions
Lineage Configuration
The Grafana source can extract lineage information between charts and their data sources. You can control lineage extraction using these configuration options:
source:
type: grafana
config:
url: "https://grafana.company.com"
service_account_token: "your_token"
# Lineage extraction (default: true)
include_lineage: true
# Column-level lineage from SQL queries (default: true)
# Only applicable when include_lineage is true
include_column_lineage: true
# Platform mappings for lineage extraction
connection_to_platform_map:
postgres_datasource_uid:
platform: postgres
platform_instance: my_postgres
env: PROD
database: analytics
database_schema: public
Lineage Features:
- Dataset-level lineage: Links charts to their underlying data sources
- Column-level lineage: Extracts field-to-field relationships from SQL queries
- Platform mapping: Maps Grafana data sources to their actual platforms for accurate lineage
- SQL parsing: Supports parsing of SQL queries for detailed lineage extraction
Performance Note: Lineage extraction can be disabled (include_lineage: false
) to improve ingestion performance when lineage information is not needed.
CLI based Ingestion
Starter Recipe
Check out the following recipe to get started with ingestion! See below for full configuration options.
For general pointers on writing and running a recipe, see our main recipe guide.
source:
type: grafana
config:
# Coordinates
platform_instance: production # optional
env: PROD # optional
url: https://grafana.company.com
service_account_token: ${GRAFANA_SERVICE_ACCOUNT_TOKEN}
# SSL verification for HTTPS connections
verify_ssl: true # optional, default is true
# Source type mapping for lineage
connection_to_platform_map:
postgres:
platform: postgres
database: grafana # optional
database_schema: grafana # optional
platform_instance: database_2 # optional
env: PROD # optional
mysql_uid_1: # Grafana datasource UID
platform: mysql
platform_instance: database_1 # optional
database: my_database # optional
sink:
# sink configs
Config Details
- Options
- Schema
Note that a .
is used to denote nested fields in the YAML recipe.
Field | Description |
---|---|
service_account_token ✅ string(password) | Service account token for Grafana |
url ✅ string | Grafana URL in the format http://your-grafana-instance with no trailing slash |
basic_mode boolean | Enable basic extraction mode for users with limited permissions. In basic mode, only dashboard metadata is extracted without detailed panel information, lineage, or folder hierarchy. This requires only basic dashboard read permissions. Default: False |
dashboard_pattern AllowDenyPattern | Regex pattern to filter dashboards for ingestion Default: {'allow': ['.*'], 'deny': [], 'ignoreCase': True} |
folder_pattern AllowDenyPattern | Regex pattern to filter folders for ingestion Default: {'allow': ['.*'], 'deny': [], 'ignoreCase': True} |
include_column_lineage boolean | Whether to extract column-level lineage from SQL queries. Only applicable when include_lineage is enabled. Default: True |
include_lineage boolean | Whether to extract lineage between charts and data sources. When enabled, the source will parse SQL queries and datasource configurations to build lineage relationships. Default: True |
ingest_owners boolean | Whether to ingest dashboard ownership information Default: True |
ingest_tags boolean | Whether to ingest dashboard and chart tags Default: True |
page_size integer | Number of items to fetch per API call when paginating through folders and dashboards Default: 100 |
platform_instance string | The instance of the platform that all assets produced by this recipe belong to. This should be unique within the platform. See https://docs.datahub.com/docs/platform-instances/ for more details. |
platform_instance_map map(str,string) | |
verify_ssl boolean | Whether to verify SSL certificates when connecting to Grafana Default: True |
env string | The environment that all assets produced by this connector belong to Default: PROD |
connection_to_platform_map map(str,PlatformConnectionConfig) | Platform connection configuration for mapping Grafana datasources to their actual platforms. |
connection_to_platform_map. key .platform_instancestring | The instance of the platform that all assets produced by this recipe belong to. This should be unique within the platform. See https://docs.datahub.com/docs/platform-instances/ for more details. |
connection_to_platform_map. key .platform ❓string | The platform name (e.g., 'postgres', 'mysql', 'snowflake') |
connection_to_platform_map. key .databasestring | Default database name |
connection_to_platform_map. key .database_schemastring | Default schema name |
connection_to_platform_map. key .envstring | The environment that all assets produced by this connector belong to Default: PROD |
stateful_ingestion StatefulIngestionConfig | Stateful Ingestion Config |
stateful_ingestion.enabled boolean | Whether or not to enable stateful ingest. Default: True if a pipeline_name is set and either a datahub-rest sink or datahub_api is specified, otherwise False Default: False |
The JSONSchema for this configuration is inlined below.
{
"title": "GrafanaSourceConfig",
"description": "Configuration for Grafana source",
"type": "object",
"properties": {
"env": {
"title": "Env",
"description": "The environment that all assets produced by this connector belong to",
"default": "PROD",
"type": "string"
},
"platform_instance": {
"title": "Platform Instance",
"description": "The instance of the platform that all assets produced by this recipe belong to. This should be unique within the platform. See https://docs.datahub.com/docs/platform-instances/ for more details.",
"type": "string"
},
"stateful_ingestion": {
"title": "Stateful Ingestion",
"description": "Stateful Ingestion Config",
"allOf": [
{
"$ref": "#/definitions/StatefulIngestionConfig"
}
]
},
"platform_instance_map": {
"title": "Platform Instance Map",
"description": "A holder for platform -> platform_instance mappings to generate correct dataset urns",
"type": "object",
"additionalProperties": {
"type": "string"
}
},
"url": {
"title": "Url",
"description": "Grafana URL in the format http://your-grafana-instance with no trailing slash",
"type": "string"
},
"service_account_token": {
"title": "Service Account Token",
"description": "Service account token for Grafana",
"type": "string",
"writeOnly": true,
"format": "password"
},
"verify_ssl": {
"title": "Verify Ssl",
"description": "Whether to verify SSL certificates when connecting to Grafana",
"default": true,
"type": "boolean"
},
"page_size": {
"title": "Page Size",
"description": "Number of items to fetch per API call when paginating through folders and dashboards",
"default": 100,
"type": "integer"
},
"basic_mode": {
"title": "Basic Mode",
"description": "Enable basic extraction mode for users with limited permissions. In basic mode, only dashboard metadata is extracted without detailed panel information, lineage, or folder hierarchy. This requires only basic dashboard read permissions.",
"default": false,
"type": "boolean"
},
"dashboard_pattern": {
"title": "Dashboard Pattern",
"description": "Regex pattern to filter dashboards for ingestion",
"default": {
"allow": [
".*"
],
"deny": [],
"ignoreCase": true
},
"allOf": [
{
"$ref": "#/definitions/AllowDenyPattern"
}
]
},
"folder_pattern": {
"title": "Folder Pattern",
"description": "Regex pattern to filter folders for ingestion",
"default": {
"allow": [
".*"
],
"deny": [],
"ignoreCase": true
},
"allOf": [
{
"$ref": "#/definitions/AllowDenyPattern"
}
]
},
"ingest_tags": {
"title": "Ingest Tags",
"description": "Whether to ingest dashboard and chart tags",
"default": true,
"type": "boolean"
},
"ingest_owners": {
"title": "Ingest Owners",
"description": "Whether to ingest dashboard ownership information",
"default": true,
"type": "boolean"
},
"include_lineage": {
"title": "Include Lineage",
"description": "Whether to extract lineage between charts and data sources. When enabled, the source will parse SQL queries and datasource configurations to build lineage relationships.",
"default": true,
"type": "boolean"
},
"include_column_lineage": {
"title": "Include Column Lineage",
"description": "Whether to extract column-level lineage from SQL queries. Only applicable when include_lineage is enabled.",
"default": true,
"type": "boolean"
},
"connection_to_platform_map": {
"title": "Connection To Platform Map",
"description": "Map of Grafana datasource types/UIDs to platform connection configs for lineage extraction",
"type": "object",
"additionalProperties": {
"$ref": "#/definitions/PlatformConnectionConfig"
}
}
},
"required": [
"url",
"service_account_token"
],
"additionalProperties": false,
"definitions": {
"DynamicTypedStateProviderConfig": {
"title": "DynamicTypedStateProviderConfig",
"type": "object",
"properties": {
"type": {
"title": "Type",
"description": "The type of the state provider to use. For DataHub use `datahub`",
"type": "string"
},
"config": {
"title": "Config",
"description": "The configuration required for initializing the state provider. Default: The datahub_api config if set at pipeline level. Otherwise, the default DatahubClientConfig. See the defaults (https://github.com/datahub-project/datahub/blob/master/metadata-ingestion/src/datahub/ingestion/graph/client.py#L19).",
"default": {},
"type": "object"
}
},
"required": [
"type"
],
"additionalProperties": false
},
"StatefulIngestionConfig": {
"title": "StatefulIngestionConfig",
"description": "Basic Stateful Ingestion Specific Configuration for any source.",
"type": "object",
"properties": {
"enabled": {
"title": "Enabled",
"description": "Whether or not to enable stateful ingest. Default: True if a pipeline_name is set and either a datahub-rest sink or `datahub_api` is specified, otherwise False",
"default": false,
"type": "boolean"
}
},
"additionalProperties": false
},
"AllowDenyPattern": {
"title": "AllowDenyPattern",
"description": "A class to store allow deny regexes",
"type": "object",
"properties": {
"allow": {
"title": "Allow",
"description": "List of regex patterns to include in ingestion",
"default": [
".*"
],
"type": "array",
"items": {
"type": "string"
}
},
"deny": {
"title": "Deny",
"description": "List of regex patterns to exclude from ingestion.",
"default": [],
"type": "array",
"items": {
"type": "string"
}
},
"ignoreCase": {
"title": "Ignorecase",
"description": "Whether to ignore case sensitivity during pattern matching.",
"default": true,
"type": "boolean"
}
},
"additionalProperties": false
},
"PlatformConnectionConfig": {
"title": "PlatformConnectionConfig",
"description": "Platform connection configuration for mapping Grafana datasources to their actual platforms.",
"type": "object",
"properties": {
"platform_instance": {
"title": "Platform Instance",
"description": "The instance of the platform that all assets produced by this recipe belong to. This should be unique within the platform. See https://docs.datahub.com/docs/platform-instances/ for more details.",
"type": "string"
},
"env": {
"title": "Env",
"description": "The environment that all assets produced by this connector belong to",
"default": "PROD",
"type": "string"
},
"platform": {
"title": "Platform",
"description": "The platform name (e.g., 'postgres', 'mysql', 'snowflake')",
"type": "string"
},
"database": {
"title": "Database",
"description": "Default database name",
"type": "string"
},
"database_schema": {
"title": "Database Schema",
"description": "Default schema name",
"type": "string"
}
},
"required": [
"platform"
],
"additionalProperties": false
}
}
}
Code Coordinates
- Class Name:
datahub.ingestion.source.grafana.grafana_source.GrafanaSource
- Browse on GitHub
Questions
If you've got any questions on configuring ingestion for Grafana, feel free to ping us on our Slack.