[SPIKE] Check all pages' accessibility

[domoscargin]

Popped this together to check how long it'd take to run accessibility tests on ALL our pages. Just a hacky function to get all directories with an index.html file, then running an axe validation test on each of those.

Takes around 80 seconds on my machine, but open for ways to speed it up.

There's a few failures flagged, mostly to do with our heading order.

If taken forward, the code would need to be cleaned up and optimised.

Edit: looks like the tests took about 2m 45s.

[netlify]

✅ You can preview this change here:

Name	Link
🔨 Latest commit	91ca1ac
🔍 Latest deploy log	https://app.netlify.com/sites/govuk-design-system-preview/deploys/66f438fbf8599100085723a6
😎 Deploy Preview	https://deploy-preview-2957--govuk-design-system-preview.netlify.app
📱 Preview on mobile	Toggle QR Code... Use your smartphone camera to open QR code link.

---

To edit notification comments on pull requests, go to your Netlify site configuration.

[domoscargin]

Had a brief look at it.concurrent.each, but at first glance this doesn't play nicely with Puppeteer. Maybe something with promise.all?

[colinrotherham]

Hey @domoscargin I've rebased this one, seems a shame to leave it

I've brought over the Axe rules from alphagov/govuk-frontend@ebb9e4e added in:

• Switch Accessibility tests to @axe-core/puppeteer govuk-frontend#3522

Locally I'm seeing Puppeteer (with Axe) disconnect after a while but let's hope it's resolved by:

• axe-core-npm v4.8.0 Release dequelabs/axe-core-npm#827
• puppeteer: Investigate if Puppeteer headless: new and unloaded iframes are still an issue dequelabs/axe-core-npm#828

[colinrotherham]

@domoscargin Just rebased this to see if the @axe-core/puppeteer update fixed the disconnect issue

Looking good locally

[domoscargin]

Darn, failed here.

[colinrotherham]

Darn, failed here.

How frustrating 😭

Worth trying a separate await browser.newPage() per test (just pushed)

Otherwise, let's check again when more next major updates come out 🤦‍♂️

[colinrotherham]

Another Puppeteer update so rebasing this again

[colinrotherham]

Had some more Axe and Puppeteer updates in #3468 so giving this one a rebase

[colinrotherham]

@domoscargin Looks like it's all working 🙌

Status check failures are now accessibility report issues, as intended

[domoscargin]

I've reduced failures on this PR down to zero BUT only by disabling rules across the board.

Just wanted to get this down to zero and allow us to think about how to address each of the special cases (which are all documented in __tests__/accessibility-audit.test.js).

The rules that cause failures unless disabled:

[
  'region',
  'color-contrast-enhanced',
  'aria-allowed-attr',
  'target-size',
  'aria-allowed-role'
]

[domoscargin]

Failures notes

aria-allowed-role

This flags our use of role="region" on the <video> element, which doesn't allow that role.

This was done on purpose, but unclear if it's still relevant.

target-size

Bit of a weird one, this. It's flagging the "Open this example in a new tab" links, but only on certain examples:

#table-second-example in /components/table/index.html
#select-second-example in /components/select/default/index.html
#panel-second-example in /components/panel/default/index.html
#details-second-example in /components/details/default/index.html

There are plenty more examples with the second-example suffix, but maybe there's something about these that's confusing Axe? When I check with the web version, I don't get the same error.

It's also flagging:

button[value="yes"] (Accept analytics cookies) in /styles/page-template/custom/code/index.html
.govuk-header__service-name (Service name) in /styles/page-template/custom/code/index.html
.govuk-header__navigation-item--active > .govuk-header__link[href="#"] (Navigation item 1, 2 & 3) in /styles/page-template/custom/code/index.html
.govuk-footer__inline-list-item:nth-child(1) > .govuk-footer__link[href="#"] (Footer links) in /styles/page-template/custom/code/index.html

Again, I can't recreate this on the web.

aria-allowed-attr

This is flagging a bunch of conditional radios on the Equality Information examples. It doesn't like them having aria-expanded="false". Again, I can't recreate this on the web, and there are other examples using conditionals that don't get flagged.

color-contrast-enhanced

This flags 180+ issues, but is a AAA rule and I think we can potentially ignore it for now in our automated testing and deal with the important cases manually.

region

"All page content should be contained by landmarks" - this flags over 100 issues, but we already ignore a few instances separately.

@selfthinker If you're ever bored, feel free to have a gander at these and have a reckon (this isn't currently prioritised work).

[owenatgov]

I'm looking at this as part of https://github.com/alphagov/design-system-team-internal/issues/912 (internal repo) as it helps us with auditing the site for at the very least issues we've had reported to us on heading structure. I'm going to rebase it so that this is repeatable and referable but as part of this I'm going to drop 2161784 as this is where the majorty of conflicts are being reported from and we resolved this as part of #4054

As a personal take, whilst we were nervous about how long this took to run per test run, I think it'd be valuable to just have and run manually as an ocassional auditing tool.

[selfthinker]

I just looked at the failures.

target-size: That behaviour is indeed weird. I can confirm none of those failures exist and are not flagged by the Axe browser extension. I wonder if this check is simply buggy. It might be worth checking once a year or so if it's been fixed and enabling the rule again when it was.

color-contrast-enhanced: I agree this check makes sense to disable.

aria-allowed-attr: This seems generally like a sensible check. The browser extension does flag it for me as well. Is there a way to only disable it for certain components? It would be a shame to lose the check for other components than conditionally revealing radios.

region: Is "All page content should be contained by landmarks" the only thing this one checks? If it does, I agree we should disable it. If not, same with the above, would it be possible to only disable it for certain components? I see on the list of exceptions that certain areas like .app-phase-banner and .govuk-skip-link are also excluded. Similarly here, it would be a shame to lose those being checked as a whole just for this single reason.

aria-allowed-role: Changing the video role to a region is quite iffy, so definitely a good find by the tool. Although I can see from the relevant PR that there was a good amount of testing, even if this does not do anything weird at the moment, chances that it might do in the future are quite high. I would suggest to solve this differently.

[selfthinker]

aria-allowed-role: Changing the video role to a region is quite iffy, so definitely a good find by the tool. Although I can see from the relevant PR that there was a good amount of testing, even if this does not do anything weird at the moment, chances that it might do in the future are quite high. I would suggest to solve this differently.

@owenatgov reminded me today to add any potential solution. Here is what I would do:
Don't add the role=region and aria-labelledby to the <video> element but to the <div> surrounding it.