Devising a release process for packages with native extension modules

[davesque]

I'm trying to figure some things out about automating the release process for the blake2b project as well as for any project we have in the future which includes a native extension module for which wheels need to be built.

Here's a bit of background. The release process for packages with pre-compiled extension modules is a bit complex. You need to build a source distribution (*.tar.gz), which can done on any platform and can be used to compile the extension from source. But to make things really nice, you also need to create pre-compiled distributions (wheels or *.whl files) for the specific versions of python and specific platforms (windows, linux, etc.) that you support. MacOS wheels need to be built on macOS, linux wheels on linux, windows wheels on windows, etc. I've thought of a couple of approaches to automating this:

1. Include a collection of Vagrantfiles in your project that define Vagrant VMs for build environments and scripts that execute inside of them. Then the building would all be done locally.

• Advantages: More generic, more control over process, PyPI creds never leave your machine
• Disadvantages: More local setup for building and releasing, Vagrant boxes are annoyingly large (e.g. macOS box is ~10GB), maintenance of Vagrant base boxes is complicated

2. You could build the wheels on CI and have the CI jobs also publish them to PyPI.

• Advantages: More hands-off, no local setup required
• Disadvantages: Build process is more coupled with a particular CI platform, PyPI API token needs to live in env var on CI

So what are everyone's thoughts on this? Which one of those two options seems better? Are there any other completely different approaches that I'm not considering?

Updates:

• Added a note about the size of Vagrant boxes

[carver]

• PyPI creds need to live in env vars on CI

This makes me very uncomfortable. I'd rather take some extra time on a heavy local release process.

[davesque]

This makes me very uncomfortable. I'd rather take some extra time on a heavy local release process.

Yep, I can relate. Although I just discovered that PyPI has support for API tokens for publishing packages. So you don't need to upload creds that someone could use to lock you out of your account. It's also pretty typical in other settings that CI has the power to deploy code. So I've been thinking for a while that there might be some utility in avoiding having to always manually cut releases. Especially as the number of packages we maintain grows over time.

But I'm not set on the idea. Just wanted to discuss it.

[davesque]

I suppose actually that, in most other situations where CI manages deploys (such as with a cloud service provider), the damage may be somewhat more reversible since bad deploys can just be rolled back. If someone compromises one of our API tokens for PyPI releases, it may be harder to undo the damage done by that since, for a time, compromised packages will be downloaded and used by people and we have much less control over that.

[davesque]

Now that I'm getting into trying to use Vagrant boxes, it's a bit annoying. The boxes are huge, for one thing (~10GB for macOS, maybe the same for Windows), and it seems unwise to just depend on already existing boxes available on Vagrant Cloud since none of them seem to be from official sources. Those boxes could be running any code and it would be much harder to audit them than for a Docker image. So then we need to figure out how to build Vagrant base boxes and deal with all the maintenance overhead that comes along with that.

We would do all of that if we thought it was likely that a PyPI API token on Circle CI could get compromised. So I think we need to get a clearer picture of how much of a risk that actually is. The main way that I think a PyPI API token could get compromised (other than if CircleCI itself was hacked) is if someone accidentally enabled the "make env vars available to forked builds" option. How likely do we think that is to happen? Also, if Circle CI itself was compromised, it seems like we'd have a bit of time to revoke existing PyPI API tokens before anyone used them improperly. We might have less time if someone compromised Circle CI specifically to get to us, but that seems unlikely.

Anyway, just some more food for thought.

[lithp]

Another option is to to use CI to build the wheels (and add them as build artifacts on github), but still have a manual step where those wheels are uploaded to PyPi.

It also might be possible to use mingw to cross-compile for windows from a linux box? That route might be more trouble than it's worth.

[davesque]

@lithp Both excellent suggestions.

[davesque]

@lithp I think the easiest thing to do right now would be to proceed with your suggestion of storing CI build artifacts. Maybe I can somehow write a script that downloads the latest artifacts then uploads them to PyPI.

[cburgdorf]

think the easiest thing to do right now would be to proceed with your suggestion of storing CI build artifacts. Maybe I can somehow write a script that downloads the latest artifacts then uploads them to PyPI.

This sounds like good strategy to me 👍

[carver]

This is a more difficult attack vector, but a compromised CI could build an arbitrary wheel. One option for double-checking this is to build the one wheel locally, and compare it to the one that CI built. As long as they match, we can just trust that the other distros were built correctly. (Frankly, this is more important for bug-catching than for security, I expect)

If you get really paranoid, you imagine that an infiltrator could leave one package as untouched and corrupts the others. Since packages will be released by devs with either linux or mac, that helps probabalistically catch the fault, and reduce the effectiveness of that approach.

[lithp]

This is a more difficult attack vector, but a compromised CI could build an arbitrary wheel. One option for double-checking this is to build the one wheel locally, and compare it to the one that CI built. As long as they match, we can just trust that the other distros were built correctly. (Frankly, this is more important for bug-catching than for security, I expect)

Does Rust support deterministic builds? This might be harder than it sounds! But I agree this is a vulnerability. It'll be a tradeoff between the hassle of running everything locally and the danger of letting someone else's computers be involved in the process. It feels like it makes sense to prioritize making releases easier for now, and to make them safer once our libraries become important and worth attacking.

[pipermerriam]

In the beginning I'd advocate for something simple and in our control like provisioning cloud machines and doing the builds on those. OSX + linux seem like requirements. Before we should think about doing windows I think we need to get AppVeyor working in CI so that our tests are running against windows.

I'm keen to avoid CI based builds for the attack vector reasons that have been pointed out.

[davesque]

@pipermerriam

I already got builds for blake2b-py working on Appveyor. For blake2b-py, as of today, we're doing:

• Linux builds on Circle CI. MacOS and Windows builds could come later on Circle CI but they're only available for paid plans. I know we've got one but it kinda locks us into Circle CI if we depend on it which isn't great.
• Windows builds on Appveyor.
• MacOS and Linux builds on Travis CI. I think Travis might also work for Windows builds (they support them, but I haven't set them up yet).

I suppose, if we could get both Circle CI and Travis CI doing all three build kinds, then we could compare the builds from both of those sources. Someone would have to compromise both platforms to compromise our build process. @lithp seemed to have some concerns that builds might not be deterministic but my intuition is that they should be.

Otherwise, I think the only other alternative is to do local builds with Vagrant VMs (for macOS and Windows; Linux can just use Docker). We can do it, it'll just mean a bit more work. And we'll have to build and maintain our own Vagrant base boxes.

[davesque]

Actually, it seems HashiCorp (the guys who made Vagrant) also have a pretty nice tool called Packer for building Vagrant boxes. It might make maintaining custom boxes a lot easier. Am spending the day playing around with it.

[pipermerriam]

It would also be ok for us to pay $$ if theres anyone who offers this as a service.