increase cache capacity

[zowoq]

Currently we have a sponsored 1TB Cachix cache, recently it may only take 1-3 weeks before the LRU paths are evicted.

I think self hosting a cache on a https://www.hetzner.com/dedicated-rootserver/sx65 for ~113€/month is the most cost effective option and it also avoids the bill shock that we may encounter with using hosted storage.

[Mic92]

This would give us with raidz2: 44000 GB - also we shouldn't fully max out the size to avoid heavy zfs fragmentation

https://www.hetzner.com/storage/object-storage/

would cost 199 Euro for the same amount of storage but presumably high avaibility but also extra costs for traffic.

I think currently cachix is also set up to be somewhat fast around the world - But this something you can probably validate better than me. I don't think hetzner's object storage has any point of presents outside their data center.. Potentially some CDN could help - for the NixOS foundation we have this fastly deal.

[Mic92]

If we combine backblaze s3 with cloudflare, we can get a 6 Dollar
/ TB redundant high-available binary cache. That sound cheaper than paying 113 Euro for a single machine that we cannot reboot without downtime of the cache.

[zowoq]

Yeah, it's cheaper if we don't need a big increase in cache capacity e.g. 5-10TB. I had been working on the assumption we'd want to store much, much more than that, e.g. enough to cover >6 months of cuda packages so we had a release cycle and overlap cached. Was also looking at snix CA store to stretch the space even further. But we don't need to worry about that sort of scale now, I don't think we'll even need to upgrade beyond our current capacity for a while.

Tangential to this issue we could do a 1TB cache for gha with odic anyway, I'll probably look at that next month.

backblaze s3 with cloudflare

I've seen the niks3 comparisons, have you looked at bunny.net for cdn and/or storage?

[zowoq]

For gha I guess we'll probably want to do something like https://github.com/cachix/cachix-action that sets up the cache, tokens, post-build-hook so users can just add a line to their config and it works OOTB (e.g. - uses: nix-community/push-to-cache@main).