Builds wouldn't be so horrendously slow and brittle if they didn't have to download Rust from scratch, try to cache it etc.
Surely there are Docker images somewhere from a reputable source that are just the manylinux Python images + a recent Rust toolchain?