Consider making revision.txt uncachable by our CDN

[stevejalim]

Description

For Bedrock (and Springfield) we serve a text file whose content is just the hash of the deployed git commit.

However, this file is being cached by our CDN for six hours

cache-control: max-age=21600, public

This makes it unwise to rely on revision.txt.

---

Success Criteria

• revision.txt is no longer cached at the CDN level - or anywhere

[janbrasna]

The tricky part might be it's actually a real, existing file written out at build time:

bedrock/Dockerfile

Line 148 in ac3bf83

RUN echo "${GIT_SHA}" > ./root_files/revision.txt

Is this served with https://whitenoise.readthedocs.io/en/stable/django.html#WHITENOISE_ROOT that may have some own defaults for these un-versioned files, like the root_files?

(Passing that through a callback for each file and modifying the header for revision.txt could work https://whitenoise.readthedocs.io/en/stable/django.html#WHITENOISE_ADD_HEADERS_FUNCTION but sounds like a lot of exercise to target a single exception to the caching rules)

[stevejalim]

Another angle might be doing something with Fastly to force no-cache; private or something similar at that point

[stevejalim]

Then again, skipping the CDN for revision.txt does increase the scope of punishing the origin servers. We could just ensure any tooling that uses revision.txt points at the origins...

[janbrasna]

a) If cache disposition is feasible to get tweaked, even a 60sec cache with the shielding globally should cover for any "punishing". (This is BTW true for e.g. 404s as well, just a few mins to cover "DoS"-level concerns, but don't hold onto it for too long?)

b) If automation needs to grab a current revision.txt even from the "public" hosts, it can set relevant cache disposition headers on the request to skip any intermediate content cached and get a result delivered as PASS (Fastly does honor that and is happy to get a fresh content for you.)

[stevejalim]

a) If cache disposition is feasible to get tweaked, even a 60sec cache with the shielding globally should cover for any "punishing". (This is BTW true for e.g. 404s as well, just a few mins to cover "DoS"-level concerns, but don't hold onto it for too long?)

This is true and part of a bigger conversation with SRE that can tack onto the 503s we've been seeing as a separate point

b) If automation needs to grab a current revision.txt even from the "public" hosts, it can set relevant cache disposition headers on the request to skip any intermediate content cached and get a result delivered as PASS (Fastly does honor that and is happy to get a fresh content for you.)

Agreed, this would work. However,
a) it adds complexity to the tests (counterpoint: we make a handy reusable thing that drops in to any tests)
b) if there's a way to bypass the cacheing via request headers (Cache-Control: no-cache) then anything could just bypass the CDN and slam the origins. (Counterpoint: we'd have combine it with an allowlist of trusted source hosts. Counter-counterpoint 😆: this may not be easy when many of our tests are run from GHAs - we can't allowlist all of the GHA runners (or their proxies/gateways))

[janbrasna]

"if there's a way to bypass the cacheing via request headers (Cache-Control: […]) then anything could just bypass the CDN and slam the origins"

That's how the Varnish setup currently works. Maybe that can be tweaked to "ignore standards and RFCs" and slap a hard-coded override in the VCL… but currently the CDN behaves like a good citizen, observing the cache disposition headers as instructed. (With the "anything could just bypass CDN" above being an available option, yes. As is hitting the origins right away, that doesn't take too much of tribal knowledge if that's more valuable than a few minutes of human time to figure it out.)

There definitely needs to be a WAF–level protection for the scenarios you have in mind, as even on the CDN with unmodified headers, there are endpoints that still just PASS given their private~esque nature, so to make use of the concern of yours just on its own, I'd need to hit only those content negotiation URLs to scale up or flood the origins at my will… so there has to be more "bad behavior" layers than just the edge CDN, to help with performance/availability and cost control for any bad actor — so even with MISS MISS or PASS, there's still the ngWAF doing its job around the shielding POP location and keeping the patterns reasonable.

[janbrasna]

But more on topic, it looks like a URL pattern can be clickops'd to override the settings?

https://www.fastly.com/documentation/guides/full-site-delivery/caching/temporarily-disabling-caching/ "Disabling caching at the individual URL level"

So I guess revision.txt on public hosts set to anything between 60–300sec sounds still good enough?