Website caching issue

[paulirwin]

The Lucene.NET website has a server-side caching issue where, after rolling out an update, the returned etag value was the same, which caused browsers that have the previously-cached contents to not get the update. More details below.

Originally posted by @paulirwin in #1198 (comment)

I found the problem (but not the cause, yet) of the caching issue: the etag returned has not changed between the old and new sites.

I fortunately had another browser instance still open that had the old cached version. Doing a normal load of the page was sending If-None-Match with etag value "24f2-6428c36d87de9-gzip". (If you're unfamiliar, If-None-Match sends a previously returned etag value, to check if it matches. If it doesn't match, the server should return 200 OK with the updated contents along with a new etag header to update its cached value, and if it matches, that means the cached version is up to date so the server returns 304 Not Modified with an empty body.)

The server returned with a 304 Not Modified and also included the etag response header with the same value, which is interpreted by the browser as "the server confirmed that my cached version is correct". So this is not a client-side caching issue, this is actually the server saying that the cached version is correct.

I then looked at what was happening in a browser that had the updated HTML/etc, and it was sending and receiving the exact same header value. So something is wrong on the server to where it's returning the same etag value even if the code has changed.

Note that the server does not send either an Expires header nor Cache-Control header with max-age, so effectively this cache may never expire since, according to the server, the etag value is exactly the same as before.

I am not sure why the server is returning the same etag value as before, as that is usually produced as effectively a hash of the contents of that page/resource, which should have changed with the updated HTML. If we are producing that etag and hardcoding it somewhere, we should fix that. Or perhaps this is an issue with the server infrastructure?

[NightOwl888]

Just a thought - maybe we have some control over this by modifying the .htaccess file?

[paulirwin]

I found the cause, and I'm embarrassed I didn't think of it sooner. The reason why it returned the same etag is... that the index.html code didn't change. The top links are loaded via JavaScript after the page loads.

This is also the reason why the Whimsy crawler can't see the links. It doesn't execute JS on the page to add them to the DOM, it just looks in the HTML.

We need to statically render the links in the top bar for the homepage, if possible.

Edit: Alternatively, we could additionally add the links to the footer, statically, which would cause the cache to bust as well as meet the Whimsy requirement.

[zka26]

Right! I should have thought of that, since I always had to check the changes using hard refresh.

Today, I tried to include the links in the html itself and it worked for the site, I will go ahead and try it for the docs along with it's search bar as well and open a follow-up PR to the previous PR and to this issue. (And will include the other fixes for the event page, breadcrumb and browser-related problem, etc)

But let me know, which option do you prefer: included statically into the navbar or into the footer.

[NightOwl888]

@zka26 - I don't have much of a preference as to where the links are in the HTML, but docfx might. I asked ChatGPT how this might work within the docfx build and it suggested adding a new template file:

https://chatgpt.com/share/690cb2a4-42c8-8005-ba83-0b40773f29a7

I don't recall the exact reason, but we are using full URL paths in the final output (at least for some links). This means that when building locally, the paths are different than when we deploy the site live.

With that in mind, I like option 3 for putting the links in the configuration where docfx can process them (or our scripts can pre-process them) when building the site. Ideally, this would all happen through docfx, but it has historically been hard to figure out how to make it work right at times and we eventually just give up and use Powershell to patch any behavior we cannot make work.

We really should fix the navbars so other crawlers can index them, as well. Ideally we would ensure the entire navbar is output as HTML when docfx does a build, not just the ASF links.