Skip to main content

Aerospike

Overview​

Aerospike is a high-performance, distributed NoSQL key-value database designed for real-time applications. Learn more in the official Aerospike documentation.

The DataHub integration for Aerospike extracts namespaces as containers and sets as datasets, including schema inference from sampled records. It also supports platform instance mapping, stateful ingestion for stale entity removal, and optional XDR (Cross-Datacenter Replication) metadata.

Concept Mapping​

Source ConceptDataHub ConceptNotes
NamespaceContainerTop-level grouping of sets within a cluster.
SetDatasetPrimary ingested entity, analogous to a table.
Record binsSchemaFieldInferred from sampled records via schema probing.
ClusterPlatform InstanceOptional; use when ingesting multiple clusters.

Module aerospike​

Testing

Important Capabilities​

CapabilityStatusNotes
Detect Deleted Entities✅Enabled by default via stateful ingestion.
Platform Instance✅Enabled by default.
Schema Metadata✅Enabled by default.

Overview​

The aerospike module ingests metadata from Aerospike clusters into DataHub. It connects to an Aerospike node, discovers namespaces and sets, and infers schema by sampling records.

Prerequisites​

Before running ingestion, ensure you have:

  1. Network connectivity to at least one Aerospike node on its service port (default 3000).
  2. Authentication credentials (username/password) if the cluster has security enabled.
  3. Read permissions on the namespaces and sets you want to ingest.

Authentication Modes​

Aerospike supports three authentication modes:

  • AUTH_INTERNAL (default) — standard username/password authentication.
  • AUTH_EXTERNAL — external authentication (e.g., LDAP) over TLS.
  • AUTH_EXTERNAL_INSECURE — external authentication without TLS.

Schema Inference​

Schema is inferred by sampling records from each set. You can control:

  • infer_schema_depth — how many nesting levels to traverse (default 1, use -1 for all levels, 0 to skip).
  • schema_sampling_size — number of records to sample per set (default 1000, use null for full scan).
  • max_schema_size — cap on schema fields emitted (default 300).

Install the Plugin​

pip install 'acryl-datahub[aerospike]'

Starter Recipe​

Check out the following recipe to get started with ingestion! See below for full configuration options.

For general pointers on writing and running a recipe, see our main recipe guide.

source:
  type: aerospike
  config:
    # Coordinates
    hosts:
      - - host1
        - 3000
      - - host2
        - 3000

    # Credentials
    username: user
    password: "${AEROSPIKE_PASSWORD}"
    # auth_mode: AUTH_INTERNAL  # AUTH_INTERNAL (default), AUTH_EXTERNAL, AUTH_EXTERNAL_INSECURE

    # TLS (optional)
    # tls_enabled: false
    # tls_cafile: /path/to/ca.crt
    # tls_capath: /path/to/ca/dir

    # Platform instance (optional) — use when ingesting multiple Aerospike clusters
    # platform_instance: prod-cluster

    # Namespace and set filtering (regex allow/deny patterns)
    # namespace_pattern:
    #   allow:
    #     - "^prod.*"
    #   deny:
    #     - "^test.*"
    # set_pattern:
    #   allow:
    #     - "^events.*"

    # Schema inference
    # infer_schema_depth: 1     # Depth of nested fields to infer. -1 = all levels, 0 = skip inference.
    # schema_sampling_size: 1000  # Number of records to sample per set. null = scan entire set.
    # max_schema_size: 300        # Maximum number of fields to include in the schema.
    # ignore_empty_sets: false  # Skip sets with zero records.

    # Performance
    # records_per_second: 0   # Rate limit for Aerospike queries. 0 = no limit.
    # schema_query_timeout_ms: null  # Socket timeout in ms for schema inference queries. null = client default.
    # login_timeout_ms: null  # Login timeout in ms. null = Aerospike client default.

    # XDR (Cross-Datacenter Replication) metadata
    # include_xdr: false  # Annotate sets with the DCs they are replicated to.

    # Stateful ingestion — removes stale entities from DataHub when sets are dropped
    # stateful_ingestion:
    #   enabled: true


sink:
  # sink configs

Config Details​

Note that a . is used to denote nested fields in the YAML recipe.

FieldDescription
auth_mode
One of Enum, null
The authentication mode with the server.
Default: 0
ignore_empty_sets
boolean
Ignore empty sets in the schema inference.
Default: False
include_xdr
boolean
Include XDR information in the dataset properties.
Default: False
inferSchemaDepth
integer
The depth of nested fields to infer schema. If set to -1, infer schema at all levels. If set to 0, does not infer the schema. Default is 1.
Default: 1
login_timeout_ms
One of integer, null
Login timeout in milliseconds. Default None, using the default value of the Aerospike Python client.
Default: None
maxSchemaSize
One of integer, null
Maximum number of fields to include in the schema.
Default: 300
password
One of string(password), null
Aerospike password.
Default: None
platform_instance
One of string, null
The instance of the platform that all assets produced by this recipe belong to. This should be unique within the platform. See https://docs.datahub.com/docs/platform-instances/ for more details.
Default: None
records_per_second
integer
Number of records per second for Aerospike query. Default is 0, which means no limit.
Default: 0
schemaSamplingSize
One of integer, null
Number of documents to use when inferring schema. If set to null, all documents will be scanned.
Default: 1000
schema_query_timeout_ms
One of integer, null
Socket timeout in milliseconds for schema inference queries. Default None uses the Aerospike client default.
Default: None
tls_cafile
One of string, null
Path to the CA certificate file.
Default: None
tls_capath
One of string, null
Path to the CA certificate directory.
Default: None
tls_enabled
boolean
Whether to use TLS for the connection.
Default: False
username
One of string, null
Aerospike username.
Default: None
env
string
The environment that all assets produced by this connector belong to
Default: PROD
hosts
array
Aerospike hosts list.
Default: [['localhost', 3000]]
hosts.array
array
hosts.array.object
object
namespace_pattern
AllowDenyPattern
A class to store allow deny regexes
namespace_pattern.ignoreCase
One of boolean, null
Whether to ignore case sensitivity during pattern matching.
Default: True
set_pattern
AllowDenyPattern
A class to store allow deny regexes
set_pattern.ignoreCase
One of boolean, null
Whether to ignore case sensitivity during pattern matching.
Default: True
stateful_ingestion
One of StatefulStaleMetadataRemovalConfig, null
Default: None
stateful_ingestion.enabled
boolean
Whether or not to enable stateful ingest. Default: True if a pipeline_name is set and either a datahub-rest sink or datahub_api is specified, otherwise False
Default: False
stateful_ingestion.fail_safe_threshold
number
Prevents large amount of soft deletes & the state from committing from accidental changes to the source configuration if the relative change percent in entities compared to the previous state is above the 'fail_safe_threshold'.
Default: 75.0
stateful_ingestion.remove_stale_metadata
boolean
Soft-deletes the entities present in the last successful run but missing in the current run with stateful_ingestion enabled.
Default: True

Capabilities​

  • Schema inference — bin names and types are inferred by sampling records. Nested maps and lists are traversed up to the configured depth.
  • Namespace containers — namespaces are represented as DataHub containers, with sets organized under them.
  • XDR metadata — when include_xdr is enabled, sets are annotated with the data centers they replicate to.
  • Stateful ingestion — enables automatic removal of datasets from DataHub when the corresponding sets are dropped from Aerospike.

Limitations​

  • Schema inference relies on sampling and may not capture all bin names if the data is sparse or heterogeneous.
  • Aerospike does not store schema definitions; all type information is derived from record values at ingestion time.
  • The primary key (PK) field is always reported as a string since Aerospike returns digest-based keys.

Troubleshooting​

Connection Refused​

Ensure the Aerospike node is reachable on the configured host and port. Verify firewall rules and that the service port is correct.

Empty Schema​

If sets appear with no schema fields, verify that infer_schema_depth is not set to 0 and that schema_sampling_size is large enough to capture representative records.

Code Coordinates​

  • Class Name: datahub.ingestion.source.aerospike.AerospikeSource
  • Browse on GitHub
Questions?

If you've got any questions on configuring ingestion for Aerospike, feel free to ping us on our Slack.

💡 Contributing to this documentation

This page is auto-generated from the underlying source code. To make changes, please edit the relevant source files in the metadata-ingestion directory.

Tip: For quick typo fixes or documentation updates, you can click the ✏️ Edit icon directly in the GitHub UI to open a Pull Request. For larger changes and PR naming conventions, please refer to our Contributing Guide.