Skip to main content

Langfuse

Overview​

Langfuse is an open-source LLM observability and evaluation platform. Applications send LLM traces to Langfuse via its SDKs or OpenTelemetry, and Langfuse records the resulting Traces, Observations (spans/generations/events), Scores, and versioned Prompts, all scoped to a Project.

The DataHub integration for Langfuse catalogs a Langfuse project's versioned Prompts as first-class, stale-tracked entities, and surfaces recent Traces and their LLM-call ("generation") Observations as execution records within a configurable rolling time window, with Scores attached as metrics. Datasets, Dataset Runs/Experiments, and Dataset↔Trace lineage are not covered by this connector version.

Concept Mapping​

Source ConceptDataHub ConceptNotes
"langfuse"Data Platform
Langfuse ProjectContainerSubtype Langfuse Project
Langfuse TraceDataProcessInstanceSubtype Langfuse Trace; retrieved within the configured rolling time window, not fully enumerated
Langfuse Observation (type generation)DataProcessInstanceSubtype Langfuse Generation; nested under its parent Trace via parentInstance; input lineage to the Prompt version it used
Langfuse Observation (type span/event)(not emitted)Filtered out on the Langfuse server; only root and generation Observations are retrieved
Langfuse ScoreMLMetric on the target Trace/GenerationAttached to Trace- and Observation-level scores only; Session- and Dataset-Run-level scores are dropped (reported)
Langfuse Session / UserCustom properties on the TraceNot emitted as separate entities in this version
Langfuse Prompt (per version)DatasetSubtype Langfuse Prompt; versions of the same prompt are linked via a shared Version Set

Module langfuse​

Alpha

Important Capabilities​

CapabilityStatusNotes
Asset Containers✅Enabled by default (Project container).
Detect Deleted Entities✅Enabled for Prompts only (fully enumerated each run). NOT enabled for Traces/Generations, which are retrieved through a rolling time window and are excluded from stale-entity tracking to avoid incorrectly soft-deleting entities that simply aged out of the window.
Table-Level Lineage✅Enabled by default: Generation -> Prompt version lineage via DataProcessInstanceInput.

Overview​

The langfuse module ingests metadata from a single Langfuse project into DataHub.

Compatibility​

This connector targets the Langfuse Public API v2/v3 surface (/api/public/v2/observations, /api/public/v2/prompts, /api/public/v3/scores). It requires self-hosted Langfuse v4 or newer, or Langfuse Cloud. Some self-hosted v4 deployments run in "events_only" mode, which disables the legacy, unversioned /api/public/traces-style endpoints entirely; this connector never uses those endpoints, so it works correctly against both events_only and non-events_only v4 deployments. Older self-hosted deployments (pre-v4) that only expose the legacy v1 API are not supported.

Extracted Metadata Scope​

The connector extracts, on every run:

  • The Project as a container
  • All Prompts and every version of each Prompt, linked via a shared Version Set

The connector extracts, within a configurable rolling time window (default: the last 7 days):

  • Traces, and their generation-type Observations (LLM calls)
  • Scores attached to those Traces/Observations

Prerequisites​

  1. A running Langfuse instance (self-hosted v4+, or Langfuse Cloud).
  2. A Langfuse Public Key and Secret Key for the project you want to ingest. Create these in the Langfuse UI under Project Settings > API Keys. The key pair is scoped to exactly one project; ingesting multiple projects requires one recipe per project.

No special role beyond the default API key permissions is required — the API key's Basic Auth credentials already scope every request to the correct project.

Install the Plugin​

pip install 'acryl-datahub[langfuse]'

Starter Recipe​

Check out the following recipe to get started with ingestion! See below for full configuration options.

For general pointers on writing and running a recipe, see our main recipe guide.

source:
  type: langfuse
  config:
    # Coordinates
    platform_instance: production # optional
    env: PROD # optional

    connection:
      host: http://localhost:3000 # or your self-hosted URL, e.g. http://localhost:3000
      public_key: ${LANGFUSE_PUBLIC_KEY}
      secret_key: ${LANGFUSE_SECRET_KEY}

    # Rolling time window for Trace/Observation/Score retrieval (default: last 7 days)
    window:
      start_time: "-7d" # optional, also accepts an absolute ISO 8601 datetime
      # end_time: "2026-01-01T00:00:00Z"  # optional, defaults to now

    # Feature toggles
    include_traces: true # optional, default true - extract Traces and generation Observations
    include_scores: true # optional, default true - attach Scores as MLMetric; requires include_traces
    include_sessions: true # optional, default true - record session/user IDs as custom properties
    include_prompts: true # optional, default true - extract Prompts as versioned Datasets

    # Filtering
    trace_name_pattern:
      deny:
        - "^healthcheck.*" # example: exclude internal healthcheck traces
    prompt_pattern:
      allow:
        - ".*" # default: allow all prompts
sink:
  # sink configs

Config Details​

Note that a . is used to denote nested fields in the YAML recipe.

FieldDescription
connection ✅
LangfuseConnectionConfig
Connection configuration for a single Langfuse project.

Langfuse Basic Auth resolves to exactly one project per key pair, so one
recipe ingests exactly one project. Users with multiple Langfuse projects
run one recipe per project.
connection.host ❓
string
Langfuse base URL, e.g. 'http://localhost:3000' for a self-hosted instance, or the Langfuse Cloud region URL (e.g. 'https://cloud.langfuse.com'). No trailing slash.
connection.public_key ❓
string
Langfuse project Public Key (Basic Auth username). Find it in Project Settings > API Keys.
connection.secret_key ❓
string(password)
Langfuse project Secret Key (Basic Auth password). Find it in Project Settings > API Keys. Only shown once at creation time in the Langfuse UI.
include_prompts
boolean
Extract Prompts as versioned Dataset entities, one per prompt version, linked together via a shared VersionSet.
Default: True
include_scores
boolean
Attach Langfuse Scores as MLMetric entries on the Trace or Generation they were recorded against. Session- and dataset-run-level scores are not attached in this version (Sessions and Dataset Runs are not emitted as entities) and are dropped with a reported count. Requires include_traces=True.
Default: True
include_sessions
boolean
Record each Trace's Langfuse sessionId and userId as customProperties on the Trace's DataProcessInstance. Sessions and Users are not emitted as separate DataHub entities in this version.
Default: True
include_traces
boolean
Extract Traces, and their 'generation'-type Observations, within the configured time window as DataProcessInstance entities nested under a Project container. Non-generation Observations (plain spans/events) are summarized as counts on the parent Trace rather than emitted individually.
Default: True
page_size
integer
Number of items to fetch per API call when paginating through observations, scores, and prompts. Langfuse's v2/v3 APIs cap this at 100-1000 depending on endpoint; values above the endpoint's cap are rejected by the Langfuse server.
Default: 50
platform_instance
One of string, null
The instance of the platform that all assets produced by this recipe belong to. This should be unique within the platform. See https://docs.datahub.com/docs/platform-instances/ for more details.
Default: None
env
string
The environment that all assets produced by this connector belong to
Default: PROD
prompt_pattern
AllowDenyPattern
A class to store allow deny regexes.

Patterns are matched against the start of the string only, not the entire
string - a pattern does not need to match to the end to be considered a match.
For example, the pattern "prod" matches "prod", "prod_east", and "production".
To require an exact match, anchor your pattern explicitly, e.g. "^prod$".
prompt_pattern.ignoreCase
One of boolean, null
Whether to ignore case sensitivity during pattern matching.
Default: True
trace_name_pattern
AllowDenyPattern
A class to store allow deny regexes.

Patterns are matched against the start of the string only, not the entire
string - a pattern does not need to match to the end to be considered a match.
For example, the pattern "prod" matches "prod", "prod_east", and "production".
To require an exact match, anchor your pattern explicitly, e.g. "^prod$".
trace_name_pattern.ignoreCase
One of boolean, null
Whether to ignore case sensitivity during pattern matching.
Default: True
window
BaseTimeWindowConfig
window.bucket_duration
Enum
One of: "DAY", "HOUR"
window.end_time
string(date-time)
Latest date of lineage/usage to consider. Default: Current time in UTC
window.start_time
One of string(date-time), string
Earliest date of lineage/usage to consider. Default: Last full day in UTC (or hour, depending on bucket_duration). You can also specify relative time with respect to end_time such as '-7 days' Or '-7d'.
Default: None
stateful_ingestion
One of StatefulStaleMetadataRemovalConfig, null
Stateful ingestion configuration for stale-entity soft deletion. Applies only to Prompt entities, which are fully enumerated on every run. It does NOT apply to Trace/Generation entities: those are retrieved through the rolling time window above, so a naive full-history comparison would incorrectly flag entities that simply aged out of the window as deleted. Trace/Generation workunits are emitted with is_primary_source=False specifically to exclude them from this comparison.
Default: None
stateful_ingestion.enabled
boolean
Whether or not to enable stateful ingest. Default: True if a pipeline_name is set and either a datahub-rest sink or datahub_api is specified, otherwise False
Default: False
stateful_ingestion.fail_safe_threshold
number
Prevents large amount of soft deletes & the state from committing from accidental changes to the source configuration if the relative change percent in entities compared to the previous state is above the 'fail_safe_threshold'.
Default: 75.0
stateful_ingestion.remove_stale_metadata
boolean
Soft-deletes the entities present in the last successful run but missing in the current run with stateful_ingestion enabled.
Default: True

Capabilities​

Use the Important Capabilities table above as the source of truth for supported features and whether additional configuration is required.

Deletion Detection​

Deletion detection is enabled, but scoped narrowly and intentionally:

  • Prompts are fully enumerated on every run, so stale-entity soft deletion is safe and correctly removes Prompt versions that no longer exist in Langfuse.
  • Traces and Generations are excluded from deletion detection. They are retrieved through the rolling time window above, not fully enumerated, so comparing "entities seen this run" against "entities seen last run" would incorrectly flag Traces/Generations that simply aged out of the window as deleted. This connector never soft-deletes a Trace or Generation, even if it is later removed from Langfuse.

Scores​

Langfuse Scores are attached as MLMetric entries on the Trace or Generation they were recorded against. The original Langfuse dataType (NUMERIC, BOOLEAN, CATEGORICAL, etc.) is preserved in the metric's description, since DataHub's MLMetric.value field is a plain string.

Session-level and Dataset-Run-level Scores have no corresponding DataHub entity in this connector version and are dropped; the ingestion report includes a count of dropped scores.

Prompts​

Every version of every Prompt is ingested as its own Dataset entity, with all versions of the same Prompt name linked together via a shared Version Set (the same mechanism DataHub uses for versioned ML models). Prompt labels (e.g. production, latest) become version aliases.

Limitations​

  • Datasets, Dataset Items, Dataset Runs/Experiments, and Dataset↔Trace lineage are not extracted. Only Traces, Generations, Scores, and Prompts are in scope for this connector version.
  • Only generation-type Observations are emitted as individual entities. Only root and generation Observations are retrieved (filtered on the Langfuse server); every other Observation type — span, event, agent, tool, chain, retriever, embedding, guardrail, etc. — is not ingested, to bound API volume and entity cardinality for high-volume traces. In practice, applications instrumented primarily with agent/tool-call frameworks may have zero generation-type Observations, in which case Traces from that application will show zero nested Observation entities even though real LLM-call-shaped work happened underneath.
  • Sessions and Users are not emitted as separate entities. Their IDs are recorded as custom properties on the Trace.
  • No lineage to conventional catalogued datasets. Langfuse does not expose which upstream tables or warehouses an LLM application actually read from or wrote to.
  • A Trace whose root Observation started before the configured window, but which has a generation Observation that started inside it, is ingested partially. Its generations are emitted and linked to the Trace, but the Trace's own properties, run events, and Scores are not emitted from that run, so a full Trace record written by an earlier run is never overwritten. The ingestion report counts these as partial_traces.

Troubleshooting​

If ingestion fails, first verify:

  • The configured connection.host is reachable and correct (no trailing slash).
  • connection.public_key / connection.secret_key are valid for the target project — test with datahub ingest -c recipe.yml --test-source-connection.
  • The target Langfuse deployment is self-hosted v4+ or Langfuse Cloud. If the connection test reports a v2/v3 API error, the deployment may be running an unsupported, older API generation.

Then review ingestion logs for source-specific warnings (e.g., a specific Prompt version failing to fetch) and adjust trace_name_pattern / prompt_pattern or the time window as needed.

Code Coordinates​

  • Class Name: datahub.ingestion.source.langfuse.langfuse.LangfuseSource
  • Browse on GitHub
Questions?

If you've got any questions on configuring ingestion for Langfuse, feel free to ping us on our Slack.

💡 Contributing to this documentation

This page is auto-generated from the underlying source code. To make changes, please edit the relevant source files in the metadata-ingestion directory.

Tip: For quick typo fixes or documentation updates, you can click the ✏️ Edit icon directly in the GitHub UI to open a Pull Request. For larger changes and PR naming conventions, please refer to our Contributing Guide.