Description

Originally posted by @janbrasna in #15007 (comment):

"The offered languages, esp. from the bot/baseline root page, should all be working. I've found some in prod that link to 404 — e.g. Tadjik (Тоҷикӣ) /tg is being offered there, but not working." ~~~ (resolved since; now Sicilian /scn can be used to repro)

Steps to reproduce

Open https://www.mozilla.org/ with Accept-Language header empty to get to the locale selection page (i.e. this from crawlers PoV), and check all the links work.

(Alternative hreflangs meta for / home follows the same pattern, and differs for the list of hreflangs for /en home — TODO: investigate why the active locales are assessed with different results here… Generated sitemaps seem unaffected by this.)

Expected result

Only working locales linked.

Actual result

https://www.mozilla.org/ach › default fallback
https://www.mozilla.org/lo › /en-US/lo/ 404
https://www.mozilla.org/mk › default fallback
https://www.mozilla.org/km › default fallback
https://www.mozilla.org/tg › /en-US/tg/ 404
https://www.mozilla.org/tl › default fallback
https://www.mozilla.org/scn/ › /en-US/scn/ 404

and notably some active and working locales (Armenian /hy, Gujarati /gu) missing from the list.

Environment

Overrides with googlebot UA string, empty Accept-Locale etc.