Consider shorter CDN cache for 404 results #16840
Description
Description
404 results are currently cached for too long. (Apparently not just the usual 6/10hr limit as is often in the app, but it either caches based on the mime/content, or the path(? … or does the upstream UploadServer set its own TTL?), so e.g. media or their 404 XML response get cached as a static document for day(s?))
Since it does not exactly serve as an "attack" protection given one can invent an infinite number of nonexistent endpoints to hit, this is also more in a WAF territory to filter out bad actors, than to cache nonexistent URL requests for days+(?)
From cache perspective, anything more than a few minutes for e.g. the media 404 doesn't seem that valuable, expecting the 404 XML hits served from the GCS bucket being cheap enough to not impact its traffic cost.
There seem to be VCLs to change things around based on the upstream status:
https://www.fastly.com/documentation/solutions/examples/overriding-caching-defaults-based-on-a-backend-response/
The only downside I can think of in short TTL is the extra egress cost from the shielding POP, having to serve those to the consumer POPs, as that POP-to-POP traffic is also charged, so these 404s would rack up the extra hits from BFI every few mins — hopefully the 404s are small enough in transit to not be an issue.
Success Criteria
- Consider different rules for /media on GCS vs. other paths on GCP GKE?
- Consider the cost of increased 404 XML responses served from GCS