Allow alembic to pull migration files from a remote source

[jarviliam]

Would it be possible to have alembic read migration files from a remote source like S3 and run those files against a database?

In my current organization we use alembic to run and generate migrations which has been working very well. Our current workflow goes along the lines of building a docker image containing alembic and the alembic migration files on every migration change, and then running that docker image. When looking at other migration tools such as golang's golang-migrate, I noticed it provides a functionality to read migrations from a remote source. I think this would be a good improvement/alternative in the way we approach migrations in my current organization as instead of having to build images every migration, we could just push these migration files to remote S3 bucket and read it from there (with some sort of folder versioning to prevent misreading).

I'm not too sure of the complexity it takes to add a feature like this in alembic, nor if there is already a similar way of doing the above in alembic. I personally wouldn't mind looking into what it takes to add a feature like this and maybe attempting it but I just wanted to hear everyone's feedback on such a feature first.

As always, thank you for your continued work on Alembic and SQLAlchemy!

[zzzeek]

definitely not something we'd build in to alembic itself, but you can do this yourself by simply reading the files from S3 into a temp directory, then running alembic on those files like any other migration files. You have complete control over how the Alembic config gets set up and runs commands, see https://alembic.sqlalchemy.org/en/latest/api/commands.html for a quick example how to programmatically set up config + run commands.

[CaselIT]

You can also mount a s3 (or other blob storage) in that docker image using the cli provided by the service provider. This way you may also just keep a static alembic config