Investigate `cuda.bindings`'s long build time on linux-aarch64

[leofang]

Based on observing the CI behavior, I noticed recently that it is consistently almost 2x slower on linux-aarch64 (13-14 mins) than on linux-64 (7-8 mins). FWIW it builds blazing fast on win-64 (~4 mins), so linux-aarch64 is always the last to finish building. This can be worsen by the fact that we have fewer Arm64 runners than x86-64 runners, so if the queue is full the wait time is really really long.