Primary storage read pool

GMS can maintain a second connection pool (Ebean) or Cassandra session (read pool) for entity-aspect reads, while all writes and locking reads stay on the primary pool. This is scoped to the entity aspect DAO only — not Elasticsearch, Kafka consumers, pgQueue, or upgrade jobs.

For the full list of environment variables, see Environment Variables. For metrics, see Monitoring — Primary storage read pool.

When to use it

Goal	Recommended mode
Isolate read connection usage from write pool (same database)	Split-pool — same URL/hosts, two pools
Offload reads to a physical read replica	Replica — set EBEAN_READ_POOL_URL or CASSANDRA_READ_POOL_HOSTS to the replica endpoint
Entire GMS instance must not write metadata	Use DATAHUB_READ_ONLY=true (separate from read pool; see below)

Split-pool does not avoid replica lag (there is no replica). It only separates connection limits and can reduce contention on the primary pool.

Replica mode also isolates in-process entity cache keys by read preference so primary and replica reads do not share the same cache entry.

What is routed

Routing is driven by OperationContext and PrimaryStorageResolver:

Operation	Pool
Writes, transactions, FOR UPDATE reads	PRIMARY (always)
batchGet, getAspect, getAspectsInRange, timeline reads (forUpdate=false)	READ when enabled and registered
opContext == null or READ pool not registered	PRIMARY (safe fallback)

Consumers and jobs that construct their own OperationContext should use ReadPreference.PRIMARY when read-your-writes matters (for example ingestion paths).

Replica lag is environment-specific. Aurora readers are often milliseconds behind; a Cloud SQL replica can fall minutes behind. These paths always read primary, so they do not depend on that lag:

• Stateful access-token revocation. The revocation flag lives in the Hazelcast map datahubAccessTokenRevoked (5-minute TTL, no near cache). Create and revoke publish to every GMS, and the next read sees them, without changing near-cache invalidation for other maps. A cache miss reloads exists from primary, so a replica miss is never stored as "revoked". The map is not the entity-client cache.
• Aspect validation (AspectsBatchImpl.build) and pre-commit checks.
• Read-modify-write before a mutation (EntityUtils.getAspectFromEntity and the structured-property and secret update reads that build the next write).

Ordinary browse and GET aspect reads still use the replica when ReadPreference is READ. They can be stale by however far behind the replica is. A cached miss expires after 5 seconds even when the hit TTL is longer, so a replica miss of structuredPropertyKey does not stick for its 1-day hit TTL. "Entities not searchable after ingest" is Elasticsearch indexing delay, not this read pool.

On GMS the embedded Hazelcast node starts for token revocation even when the entity graph cache is off.

Ebean (MySQL / PostgreSQL)

Enable on GMS when entityService.impl is ebean (default):

export EBEAN_READ_POOL_ENABLED=true
# Optional — omit for split-pool (defaults to primary URL):
# export EBEAN_READ_POOL_URL=jdbc:postgresql://reader.example.com:5432/datahub

Spring maps ebean.readPool.* in application.yaml to the variables above.

The read pool datasource is configured with JDBC read-only connections (readOnly=true, autoCommit=true). Mutations attempted on that pool fail at the driver level.

Cassandra

Enable on GMS when entityService.impl=cassandra:

export CASSANDRA_READ_POOL_ENABLED=true
# Optional — omit for split-pool (defaults to primary hosts):
# export CASSANDRA_READ_POOL_HOSTS=replica-host1,replica-host2

Unlike EBean, Cassandra has no driver-level read-only session mode. Routing uses a separate CqlSession (split-pool or replica contact points). Use Cassandra roles with SELECT-only grants on replica credentials when you need database-enforced read access.

Interaction with DATAHUB_READ_ONLY

DATAHUB_READ_ONLY=true disables writes on entity aspect DAOs and does not register the read pool even if *_READ_POOL_ENABLED=true. Do not enable the read pool expecting it to replace read-only mode — they solve different problems.

After changing read pool or read-only settings, restart GMS (for example scripts/dev/datahub-dev.sh env restart).

Observability

GMS emits Micrometer counters when metrics are enabled:

• primary_storage_target_used — tags: target (PRIMARY | READ), store (ebean |cassandra), forUpdate
• primary_storage_read_fallback_to_primary — incremented when the context prefers READ but no read pool is registered

Verification

Integration tests (Testcontainers, split-pool, no separate replica container):

• PrimaryStorageReadPoolPostgresIT — PostgreSQL + Ebean
• PrimaryStorageReadPoolCassandraIT — Cassandra

Run:

./gradlew :metadata-io:test \
  --tests com.linkedin.metadata.entity.storage.PrimaryStorageReadPoolPostgresIT \
  --tests com.linkedin.metadata.entity.cassandra.PrimaryStorageReadPoolCassandraIT

Out of scope

The following continue to use the primary Ebean Database or Cassandra session only:

• MCE/MAE consumer jobs with their own Ebean configuration
• pgQueue and SQL upgrade paths
• datahub-upgrade restore/load jobs
• Listing, streaming, and migration DAO APIs