cache 404s and purge cache on upload

[arianvp]

Is your feature request related to a problem? Please describe.

We are currently not caching 404s. So they are quite slow as they need to reach out to us-east-1 S3 bucket.

Maybe this is by accident as the VCL code does have sections named cache-is-404

However if we want to cache 404s we need to be able to purge them on channel bumps.

Describe the solution you'd like

We can create an S3 Event Notification that triggers a Lambda every time an object is PUT into the bucket. The lambda then simply does a curl -XPURGE https://cache.nixos.org/<key>

Describe alternatives you've considered

Have some script in Hydra do this

Additional context

We do around 12 million PUTs per month to S3
Lambda costs $0.20 per million requests. So this would cost $2.40 per month

https://aws.amazon.com/lambda/pricing/

[arianvp]

Because the S3 notification happens asynchronously after upload, this means we lose read-after-write consistency for https://cache.nixos.org which @zimbatm raises might be an issue for our hydra builders as it's very likely that after one builder completes a build another builder immediately wants to read the built object

IDK if the remote build protocol handles this but if the builder can't substitute a path with builders-use-substitutes doesn't the remote build protocol send the missing path from the coordinator to the builder? I know that builders will always send back paths to the coordinator after building so the coordinator should have any path that builder A just built and builder B needs

[vcunat]

On Hydra.nixos.org the builders don't pull from the cache, I believe. Hydra will send them all the dependencies for each individual build step. EDIT: and I don't think we need this kind of consistency for anything else, too. I expect it will be just a short delay anyway.

[vcunat]

But please, note that channel bump is way too late for a purge. That's often days after the build happens, and channels don't get everything that we need, so I'd rather have 1h cache without any purge than long cache and purging this late.

[zimbatm]

That also maps with my understanding @arianvp . AFAIK the builders respect the builders-use-substitutes setting, so most dependencies get pulled from S3. This is important because it reduces the amount of bandwidth over the SSH connections between hydra and the builders. If we lose read-after-write consistency it will fallback to sending dependencies from hydra.

[nixos-discourse]

This pull request has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/2025-s3-sponsorship-more-resources-for-a-sustainable-nix/67019/1