meltano-state-backend-rocksdb

This is a Meltano extension that provides a RocksDB state backend using the RocksDict library.

Installation

This package needs to be installed in the same Python environment as Meltano.

From GitHub

With pipx

pipx install meltano
pipx inject meltano 'meltano-state-backend-rocksdb @ git+https://github.com/reservoir-data/meltano-state-backend-rocksdb.git'

With uv

uv tool install --with 'meltano-state-backend-rocksdb @ git+https://github.com/reservoir-data/meltano-state-backend-rocksdb.git' meltano

From PyPI

This package is not yet available on PyPI.

Configuration

meltano.yml

state_backend:
  uri: rocksdb://${MELTANO_SYS_DIR_ROOT}/state
  rocksdb:
    write_buffer_size: 0x2000000  # 32MB

Environment Variables

• MELTANO_STATE_BACKEND_URI: The URI of the RocksDB state backend.
• MELTANO_STATE_BACKEND_ROCKSDB_WRITE_BUFFER_SIZE: The RocksDB write buffer size in bytes.