Measure the accessibility of our content #2935

benchristel · 2025-09-30T23:39:24Z

This PR adds a new perseus-a11y-metrics package. It includes a script that
downloads a snapshot of Khan Academy's content and evaluates the accessibility
of each exercise.

It is still a work in progress!

Issue: LEMS-3261

Test plan:

packages/perseus-a11y-metrics/src/main.ts

should output a report that looks like this:

EXERCISE ACCESSIBILITY STATS:
-----------------------------
full:         7424 exercises
limited:      1706 exercises
inaccessible: 2469 exercises
-----------------------------
total:        11599 exercises

Note: you need gcloud and jq installed.

…ent snapshot file

…eded to read assessment item data

github-actions · 2025-09-30T23:40:36Z

🗄️ Schema Change: No Changes ✅

packages/perseus-a11y-metrics/src/parse-snapshot.ts

packages/perseus-a11y-metrics/src/content-repository.ts

benchristel · 2025-10-01T16:48:38Z

packages/perseus-a11y-metrics/src/main.ts

+import {ContentRepository} from "./content-repository";
+import {getPublishedContentVersion} from "./content-version";
+
+async function main() {


We could print a nice error message here if gcloud or jq isn't installed.

…oaded

…re the data directory wasn't created

…ge.json

…tput locations

catandthemachines · 2025-10-10T18:21:02Z

packages/perseus-a11y-metrics/README.md

Yay!! Thank you for writing this!

catandthemachines · 2025-10-23T23:34:05Z

packages/perseus-a11y-metrics/src/main.ts

+    // bucket is supposed to be for.
+    // TODO(benchristel): should the extension for the data file be .ndjson?
+    // See: https://github.com/ndjson/ndjson-spec?tab=readme-ov-file#33-mediatype-and-file-extensions
+    await gcloudStorage.cp(


Adding a note here, this script works when run once on a new object, but when you attempt to overwrite an existing object you will experience 404 errors.

Added a bug here detailing this phenomenon, will also add a TODO here in case anyone wants to run this script before it is fixed.
https://khanacademy.atlassian.net/browse/LEMS-3674

benchristel added 11 commits September 29, 2025 13:11

[benc/a11y-metrics] Add main.ts script that downloads the latest cont…

2a601eb

…ent snapshot file

[benc/a11y-metrics] Use stdio: inherit and log errors

c1d413d

[benc/a11y-metrics] Extract Command class

3ce5c44

[benc/a11y-metrics] Extract command constructor function

d5d334a

[benc/a11y-metrics] Extract Command interface

7adcfb0

[benc/a11y-metrics] Extract exercises from the snapshot with jq

82eca13

[benc/a11y-metrics] Parse exercise JSON

23a1034

[benc/a11y-metrics] Prove we can get information from the snapshot ne…

414c5c4

…eded to read assessment item data

[benc/a11y-metrics] Fix TODO

f2183f1

[benc/a11y-metrics] Working proof-of-concept

0fc636d

[benc/a11y-metrics] Fix lint

7110b38

benchristel self-assigned this Sep 30, 2025

benchristel added the olc-5.0.853e0 label Sep 30, 2025

benchristel requested a review from catandthemachines September 30, 2025 23:39

github-actions bot added the item-splitting-change label Sep 30, 2025

benchristel commented Sep 30, 2025

View reviewed changes

packages/perseus-a11y-metrics/src/parse-snapshot.ts Outdated Show resolved Hide resolved

benchristel commented Sep 30, 2025

View reviewed changes

packages/perseus-a11y-metrics/src/parse-snapshot.ts Show resolved Hide resolved

benchristel commented Oct 1, 2025

View reviewed changes

packages/perseus-a11y-metrics/src/content-repository.ts Outdated Show resolved Hide resolved

benchristel commented Oct 1, 2025

View reviewed changes

benchristel added 10 commits October 1, 2025 10:49

[benc/a11y-metrics] Remove unused fields from schemas

b91f872

[benc/a11y-metrics] Rename variable

2769d08

[benc/a11y-metrics] Fix errors when content snapshot is not yet downl…

c4159e9

…oaded

[benc/a11y-metrics] Remove unused schemas

9dc497d

[benc/a11y-metrics] Resolve TODO

0fd477c

[benc/a11y-metrics] Remove gzipInFlightAll - it only works for uploads

44754e1

[benc/a11y-metrics] Add tests for ContentRepository and fix a bug whe…

7774a91

…re the data directory wasn't created

[benc/a11y-metrics] Add test for getAssessmentItems

7d25f0c

[benc/a11y-metrics] Account for isContextInaccessible

51b36db

[benc/a11y-metrics] Remove references to nonexistent files from packa…

40e558b

…ge.json

benchristel added 14 commits October 8, 2025 14:30

[benc/a11y-metrics] Move parsing to its own file

6cb0a3a

[benc/a11y-metrics] Move parsers to their own directory

8be05f5

[benc/a11y-metrics] Document GcsContentJsonRepository

be78701

[benc/a11y-metrics] Rename repository stuff

7208d07

[benc/a11y-metrics] Prune old downloads of content data

d0c4d35

[benc/a11y-metrics] Upload file

e91f035

[benc/a11y-metrics] Add README.md

dd374da

[benc/a11y-metrics] Shorten type name so lines don't wrap

54988ce

[benc/a11y-metrics] Rename AssessmentItemList to ExerciseContent

776f740

[benc/a11y-metrics] Remove redundant interface

3a89120

[benc/a11y-metrics] Extract variable

abd0bb1

[benc/a11y-metrics] Move comment

887d2d3

[benc/a11y-metrics] Swap argument order for consistency

6dc79cf

[benc/a11y-metrics] Remove date from output file path and document ou…

873498b

…tput locations

catandthemachines reviewed Oct 10, 2025

View reviewed changes

packages/perseus-a11y-metrics/README.md

Copy link

Member

catandthemachines Oct 10, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yay!! Thank you for writing this!

benchristel added 3 commits October 10, 2025 11:28

[benc/a11y-metrics] Update comment

86b8825

[benc/a11y-metrics] Add source field to package.json so cypress can run

0185c42

[benc/a11y-metrics] Undo changes to vite.config.ts

05cf793

github-actions bot added the item-splitting-change label Oct 10, 2025

[benc/a11y-metrics] Update copyright year

847e395

github-actions bot added schema-change Attached to PRs when we detect Perseus Schema changes in it and removed schema-change Attached to PRs when we detect Perseus Schema changes in it item-splitting-change labels Oct 10, 2025

catandthemachines and others added 2 commits October 10, 2025 12:57

[benc/a11y-metrics] Exposing exerciseId to data.

df5b529

[benc/a11y-metrics] Document future work in README.md

3eba7a0

benchristel marked this pull request as ready for review October 10, 2025 21:25

benchristel requested a review from catandthemachines October 10, 2025 21:26

[benc/a11y-metrics] Updating readme with permissions requirements.

88ac370

catandthemachines reviewed Oct 23, 2025

View reviewed changes

[benc/a11y-metrics] Adding a note on the bug discovered.

9ae8aed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Measure the accessibility of our content #2935

Measure the accessibility of our content #2935

Uh oh!

benchristel commented Sep 30, 2025

Uh oh!

github-actions bot commented Sep 30, 2025 •

edited

Loading

Uh oh!

Uh oh!

Uh oh!

Uh oh!

benchristel Oct 1, 2025

Uh oh!

catandthemachines Oct 10, 2025

Uh oh!

catandthemachines Oct 23, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Measure the accessibility of our content #2935

Are you sure you want to change the base?

Measure the accessibility of our content #2935

Uh oh!

Conversation

benchristel commented Sep 30, 2025

Test plan:

Uh oh!

github-actions bot commented Sep 30, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

🗄️ Schema Change: No Changes ✅

Uh oh!

Uh oh!

Uh oh!

Uh oh!

benchristel Oct 1, 2025

Choose a reason for hiding this comment

Uh oh!

catandthemachines Oct 10, 2025

Choose a reason for hiding this comment

Uh oh!

catandthemachines Oct 23, 2025

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

github-actions bot commented Sep 30, 2025 •

edited

Loading