Multi-Storage Client

The Multi-Storage Client (MSC) is a unified high-performance Python client for object and file stores such as AWS S3, Azure Blob Storage, Google Cloud Storage (GCS), NVIDIA AIStore, Oracle Cloud Infrastructure (OCI) Object Storage, POSIX file systems, and more.

It provides a generic interface to interact with objects and files across various storage services. This lets you spend less time learning each storage service's unique interface and lets you change where data is stored without having to change how your code accesses it.

See the documentation to get started.

Layout

Important landmarks:

Key:
🤖 = Generated

.
│   # GitHub templates and pipelines.
├── .github/
│   └── ...
│
│   # GitLab templates and pipelines.
├── .gitlab/
│   └── ...
│
│   # Release notes.
├── .release_notes/
│   └── ...
│
│   # Python package build outputs.
├── dist/ 🤖
│   └── ...
│
│   # Python documentation configuration.
├── docs/
│   │
│   │   # Python documentation build outputs.
│   ├── dist/ 🤖
│   │   └── ...
│   │
│   │   # Python documentation source.
│   ├── src/
│   │   └── ...
│   │
│   │   # Python documentation configuration.
│   └── conf.py
│
│   # Python package source.
├── src/
│   └── ...
│
│   # Python package test source.
├── tests/
│   │
│   │   # Unit tests.
│   ├── unit/
│   │   └── ...
│   │
│   │   # Integration tests.
│   ├── integration/
│   │   └── ...
│   │
│   │   # End-to-end (E2E) tests.
│   └── e2e/
│       └── ...
│
│   # GitLab pipeline entrypoint.
├── .gitlab-ci.yml
│
│   # Integration test containers.
├── docker-compose.yml
│
│   # Reproducible shell configuration.
├── flake.nix
├── flake.lock 🤖
│
│   # Build recipes.
├── justfile
│
│   # Python package configuration.
├── pyproject.toml
└── poetry.lock 🤖

Tools

Nix

Nix is a package manager and build system centered around reproducibility.

For us, Nix's most useful feature is its ability to create reproducible + isolated CLI shells on the same machine which use different versions of the same package (e.g. Java 17 and 21). Shell configurations can be encapsulated in Nix configurations which can be shared across multiple computers.

The best way to install Nix is with the Determinate Nix Installer (guide).

Once installed, running nix develop in a directory with a flake.nix will create a nested Bash shell defined by the flake.

If you're on a network with lots of GitHub traffic, you may get a rate limiting error. To work around this, you can either switch networks (e.g. turn off VPN) or add a GitHub personal access token (classic) to your /etc/nix/nix.conf (system) or ~/.config/nix/nix.conf (user).

# https://nixos.org/manual/nix/stable/command-ref/conf-file
access-tokens = github.com=ghp_{rest of token}

direnv

direnv (🍺) is a shell extension which can automatically load and unload environment variables when you enter or leave a specific directory.

It can automatically load and unload a Nix environment when we enter and leave a project directory.

Unlike nix develop which drops you in a nested Bash shell, direnv extracts the environment variables from the nested Bash shell into your current shell (e.g. Bash, Zsh, Fish).

Follow the installation instructions on its website.

Editor Plugins

Plugins to add editor support for direnv. Note that these won't automatically reload the environment after you change Nix flakes unlike direnv itself so you need to manually trigger a reload.

• Sublime Text
  • Direnv (caveat)
• JetBrains IDEs
  • Direnv integration (JetBrains feature request)
• Visual Studio Code
  • direnv (caveat)
• Vim
  • direnv.vim

Developing

Common recipes are provided as Just recipes. To list them, run:

just

Building the Package

To do a full release build (runs static analysis + unit tests), run:

just build

If you want to use a specific Python binary such as Python 3.9, run:

just python-binary=python3.9 build

Running Tests

The project includes unit, integration, and end-to-end (E2E) tests. In most cases, you'll only run the unit and integration tests.

Unit Tests

Unit tests verify the functionality of individual components:

poetry run pytest tests/unit/

Integration Tests

Integration tests verify interactions between components and local storage services:

just start-storage-systems

just run-integration-tests

just stop-storage-systems

If you want to use a specific Python binary such as Python 3.9, run:

just python-binary=python3.9 run-integration-tests

Notes

Updating Flake Locks

The flake.lock file locks the inputs (e.g. the Nixpkgs revision) used to evaluate flake.nix files. To update the inputs (e.g. to get newer packages in a later Nixpkgs revision), you'll need to update your flake.lock file.

# Update flake.lock.
nix flake update