Test improvement: Use moto to simulate S3

[jeking3]

Using moto for unit tests and local integration tests against a simulated S3 back-end allows for build-time sanity checks of code correctness in the face of changes. This can improve test coverage and improve deliverable quality.

[commiterate]

Is there an advantage to using Moto over the existing setup which points S3-based providers at a local MinIO?

We use this emulation approach for S3 API based (with MinIO), GCS (with fake-gcs-server), and Azure (with Azurite) storage providers in unit tests.

https://github.com/NVIDIA/multi-storage-client/blob/491d9e999faf199dcec69cdcdf0ce53a69b7d328/tests/test_multistorageclient/unit/utils/tempdatastore.py

multi-storage-client/tests/test_multistorageclient/unit/providers/test_storage_providers.py

Lines 31 to 44 in 491d9e9

	@pytest.mark.parametrize(
	argnames=["temp_data_store_type", "with_cache"],
	argvalues=[
	# Test all store types without cache
	[tempdatastore.TemporaryPOSIXDirectory, False],
	[tempdatastore.TemporaryAWSS3Bucket, False],
	[tempdatastore.TemporaryAzureBlobStorageContainer, False],
	[tempdatastore.TemporaryGoogleCloudStorageBucket, False],
	[tempdatastore.TemporaryGoogleCloudStorageS3Bucket, False],
	[tempdatastore.TemporarySwiftStackBucket, False],
	# Test only one store type with cache enabled
	[tempdatastore.TemporaryAWSS3Bucket, True],
	],
	)

[jeking3]

Is there an advantage to using Moto over the existing setup which points S3-based providers at a local MinIO?

Yes and no.

1. Unit testing: yes. You can python-mock the S3 client with moto using a decorator on a test. Wicked cool. No minio required at all here, and you can force error code paths that would be exceedingly hard to do with minio.
2. Integration: sort of. minio does not implement the latest S3 APIs like flexible (trailing) checksums so if you are using them, minio won't help, but it's a viable alternative for "legacy" S3 emulation.
3. Depending on your build and test farm limitations, it might be hard to spin up minio alongside the code being tested, but moto is a test dependency so it's always there and firing off a moto_server is trivial.

Just another tool in the toolbox to consider. Should allow for better testing of error code paths, but good to see some actual live testing is used too.

[commiterate]

For 1 and 2, we'll keep Moto in mind for recreating specific fault paths or AWS S3-specific features that MinIO doesn't replicate. The latter can also be covered in our E2E tests which run against real object stores. We have to rely on those anyways for object stores without local emulators (e.g. OCI object storage).

The MSC generally relies on the underlying cloud provider SDK for handling service-specific errors (e.g. for retries). Most errors are bubbled up as is since there may not be a uniform way (or at least a sensible option) to handle all of them otherwise.

Not Found errors are the exception. Those we translate to Python's native FileNotFoundError and are pretty trivial to force in testing (just specify some non-existent key).

---

For 3, portable shells with the entire toolchain is fortunately something we don't have to worry about since we use Nix (see the CONTRIBUTING.md's Tools section).

The storage emulators as well as the entire Python + Rust toolchain (including the macOS SDKs to cross-compile macOS binaries on Linux) are provided by our Nix shell.

multi-storage-client/flake.nix

Lines 88 to 91 in 491d9e9

	# Storage systems.
	azurite
	fake-gcs-server
	minio

For both local development and CI, we spin up the same Nix shell and run the same build recipes to ensure the workflow is uniform for everyone.

These shells are reproducible by default and act as overlays on top of your current shell (if you use direnv in conjunction. Otherwise it's a Bash shell subprocess). This lets you retain access to your personal programs of choice on your machine (unlike OCI containers which firewall off the rest of the system). We also try to make CI environments have almost nothing else to ensure a bare-minimum Nix shell can still build the project.

A sample workflow GIF is shown here: #10 (comment)

Python packages are still sourced through a Python package manager (uv in our case). We basically use Nix to provide idiomatic tools, then use the idiomatic tools for what they're good at.