Initial accessibility support in Compose

[Leedwon]

Description

This PR adds minimal accessibility support to Vico charts. It introduces a contentDescription parameter, allowing meaningful descriptions to be passed for data points. These descriptions will be read by TalkBack, improving the chart experience for screen reader users.

Note: this doesn’t make the entire chart accessible, full support should be introduced in separate PRs. If preferred, I can retarget this PR to a feature branch instead of master, but it should already provide some accessibility improvements on its own.

How to Review This PR:
The suggested way to review this PR is by going commit-by-commit. I’ve left comments in places where I saw potential for improvement or ran into limitations. I hope that your deep knowledge of the library can help improve or fix some of these areas.

PR Scope

• Chart data points can now include a content description that will be announced by TalkBack, making them more accessible to screen reader users.

Out of scope

• Chart legend accessibility
• Chart markers accessibility
• Chart decorations accessibility
• Automatically scrolling to chart elements that are out of screen bounds as accessibility focus moves.
  This should be handled in a future PR, as it requires implementing BringIntoViewRequester together with BringIntoViewResponder and connecting it to VicoScrollState. The main challenge here is the lack of a reliable callback that tells us when an element is focused by accessibility services unfortunately, .onFocusChanged { } doesn’t get triggered during accessibility navigation. To keep this PR smaller, I’m leaving this out for now.

Demos

In each demo, TalkBack is enabled and I’m using left/right swipe gestures (default accessibility navigation method), to move between elements with semantics. You’ll notice that chart data points with contentDescription are now announced.

Previously, the entire chart was skipped, and accessibility focus jumped directly to previous/next arrows.

Please watch the demos with sound on to hear the TalkBack announcements in action.

Chart Type	Accessibility Demo
Column Chart	column_chart_demo.mp4
Line Chart	line_chart_demo.mp4
Candlestick Chart	candlestick_demo.mp4

Known issues

This section describes known issues currently affecting chart accessibility. These can be addressed in this PR if you consider them blockers, or in future PRs. In most cases, I ran into some issues while trying to fix them, which is why they’re currently left as-is.

Combo charts

In combo charts, some data points may prevent others from being announced. For example, in the demo video, you can see that for x = 8 and x = 9, the column data points are not announced. This happens because the line points are positioned higher than the columns, and the accessibility system assumes the columns are not visible (and thus not focusable).

combo_chart_issues.mp4

Negative values

The highlighter for negative values starts from the bottom instead of the top. It might look a bit weird visually, but it’s not a problem for TalkBack.

negative_values_issues.mp4

Clickable charts with markers

When charts contain markers (making them clickable), the accessibility focus might jump to the touched position. This only happens when the user swipes back and forth through accessibility elements while swiping on the chart itself. If the user swipes just under or over the chart, focus moves correctly between data points.. This can be seen i.e on the Electric car sales chart.

Resources

https://eevis.codes/blog/2023-07-24/more-accessible-graphs-with-jetpack-compose-part-1-adding-content/

[Leedwon]

We need to update targets after calling chart.prepare(), since it looks like they're updated during that call - before it, targets were empty.

Let me know if I got that right, or if there's a better place to do this and it works differently under the hood.

[Gowsky]

targets are updated in CartesianChart.draw call. Also, it seems that we don't need targets MutableState property. The targets can be passed directly to AccessibilityHighlighter from CartesianChart.

[Leedwon]

Without targets MutableState the AccessibilityHighlighter is never recomposed with the updated targets. I initially used the approach that you mention - passing targets directly from CartesianChart, but it only passes initial targets which are empty list. This can be tested if you replace targets = targets with targets = chart.allTargets. I'll focus on other improvements first and we can improve this once we have the Provider approach in place

[Gowsky]

Are you sure that it's still the case? I've run a quick check and passing chart.allTargets directly to AccessibilityHighlighter worked fine in the sample app.

[Leedwon]

You're right this is no longer needed - I switched to passing chart.allTargets directly in f11217a. Thanks for checking it.

[Gowsky]

Hello, @Leedwon. Thanks for your contribution. Below are some general comments. We'll address your comments individually shortly.

1. Feature parity across modules is required, so while this can be merged into a feature branch, support for views and Compose Multiplatform will be needed for it to be released.
2. A more flexible and cohesive way of letting consumers specify content descriptions would be a a provider-style interface (similar to DefaultCartesianMarker.ValueFormatter).
3. Rather than treating each target separately, we should group them by x value, so the user selects a given x value and hears a description of all of the associated entries. This is how CartesianMarker works, as well as how accessibility is implemented in the article you've shared.
4. The Highlighter bounds should be corrected so that they don't extend past the CartesianLayer area vertically. Considering point 2, the y coordinates of the top and bottom edges of the Highlighter should match those of the top and bottom edges of the CartesianLayer area.

[Leedwon]

Hello, @Leedwon. Thanks for your contribution. Below are some general comments. We'll address your comments individually shortly.

1. Feature parity across modules is required, so while this can be merged into a feature branch, support for views and Compose Multiplatform will be needed for it to be released.
2. A more flexible and cohesive way of letting consumers specify content descriptions would be a a provider-style interface (similar to DefaultCartesianMarker.ValueFormatter).
3. Rather than treating each target separately, we should group them by x value, so the user selects a given x value and hears a description of all of the associated entries. This is how CartesianMarker works, as well as how accessibility is implemented in the article you've shared.
4. The Highlighter bounds should be corrected so that they don't extend past the CartesianLayer area vertically. Considering point 2, the y coordinates of the top and bottom edges of the Highlighter should match those of the top and bottom edges of the CartesianLayer area.

Hi @Gowsky, thanks for your reply and the suggested ideas. If feature parity is required, I think it would be better to retarget this to the feature branch, to keep this PR focused on the compose part (could you please create one?). Once the compose part is merged, we’ll also have a clearer idea of what basic accessibility for Vico should look like, so we can port it to Views and Compose Multiplatform. I’ll take a look at the other points soon and try to apply your suggestions.

[patrickmichalik]

Hello, and thanks for the PR! I’ve created a feature branch and made it the target.

[Leedwon]

Hello, @Leedwon. Thanks for your contribution. Below are some general comments. We'll address your comments individually shortly.

1. Feature parity across modules is required, so while this can be merged into a feature branch, support for views and Compose Multiplatform will be needed for it to be released.
2. A more flexible and cohesive way of letting consumers specify content descriptions would be a a provider-style interface (similar to DefaultCartesianMarker.ValueFormatter).
3. Rather than treating each target separately, we should group them by x value, so the user selects a given x value and hears a description of all of the associated entries. This is how CartesianMarker works, as well as how accessibility is implemented in the article you've shared.
4. The Highlighter bounds should be corrected so that they don't extend past the CartesianLayer area vertically. Considering point 2, the y coordinates of the top and bottom edges of the Highlighter should match those of the top and bottom edges of the CartesianLayer area.

Hi @Gowsky, before I start improving the code in this PR I wanted to clarify couple of things:

2. Could you help me understand how you envision the API for this provider? I think the tricky part is that accessibility usually requires more context than formatting alone, and I’m not sure how we could provide that.
   An example of this is the AITestScores chart - we have three datasets tied to a single x value: "Image recognition," "Nuanced-language interpretation," and "Programming." Ideally, the screen reader would say something like:
   "Year 2019: Image recognition: 9.52, Year 2019: Nuanced-language interpretation: -100, ..."
   If we had an API like public fun getContentDescription(target: CartesianMarker.Target): CharSequence?, I’m wondering whether it would be possible to know if a given Target corresponds to "Image recognition," "Nuanced-language interpretation," or "Programming." That context would need to be provided somehow.
   Passing contentDescription during entry creation has the benefit of having that context available upfront, but I also get that you'd prefer to keep the Entries API lean. I can rework this to use a provider-style interface, but I’d really appreciate a starting point or example to better understand how we can pass along that metadata. Also, would this interface belong in the com.patrykandpatrick.vico.multiplatform.cartesian.marker package, or some other place would be better for it?
3. I think there are multiple valid ways to announce chart data. The article presents one approach, but I wouldn’t take it as the definitive method. In that example, they have the luxury of merging the content description and formatting the x-axis label together with the values - something that’s not so straightforward in general-purpose library. For instance, iOS charts handle this differently - their screen reader goes through all the values in one dataset first, then proceeds to the next dataset. Here’s a demo of chart accessibility on iOS (with sound on):

ios_demo.mov

3. That being said, I agree that grouping content descriptions by x values is a good option, but I just wanted to share some alternative approaches as well. The only issue I see with grouping by x value is that announcing the x value only once per group might be challenging. To do that we’d likely need to support specifying content descriptions for both x and y values separately. For example, in a candlestick chart, instead of announcing just “twelve hour,” we might want to announce “midnight” - so content description is not simple a formatted x axis value. The solution I’d propose is:

• Group all content descriptions by x values
• Merge them so they’re read as one semantic entry (and also highlighted only once)
• Avoid introducing a separate interface for providing the x value’s content description - instead, it would be the consumer’s responsibility to provide the full content description via the provider we discussed in point 2.

For example, in the AITestScores case, we should provide content descriptions like:

• "Year 2019, Image recognition: 9.52"
• "Year 2019, Nuanced-language interpretation: -100"

It’s not as smart (since “Year 2019” is repeated), but it’s much simpler from a usage standpoint, and it avoids having two separate providers for content descriptions. WDYT?

4. Good point - I’ll improve that. I guess I should use the height of the CartesianLayer rather than the whole Canvas, right?

[Gowsky]

The provider's function wouldn't receive a single CartesianMarker.Target but rather a list of them, just like DefaultCartesianMarker.ValueFormatter.format. This addresses both concerns.

Regarding point 2, you can use the index of the CartesianMarker.Target in the list, usually in combination with extras (such as a list of series labels). Extras are accessed via CartesianMeasuringContext.extraStore, so the provider's function should also have a context parameter (once again, like DefaultCartesianMarker.ValueFormatter.format).

We also plan on adding an index property to CartesianLayer.Entry. This will make the indices easier to access (without the need for forEach or similar), and more importantly, it'll make it easier to link entries to series in cases where there exist x values for which not all series have y values. In the latter case, the index of the CartesianMarker.Target in the list can't be used directly, since there are fewer CartesianMarker.Targets than series. Currently, for such setups, an extra that helps map a CartesianMarker.Target index for a given x value to the index of the corresponding series must be used. CartesianLayer.Entry.index would remove the need for this. Now might be a good time to add this property, so feel free to do so if you're interested.

Regarding point 3, yes, there should be a single provider, but it's not just that x and y would be handled together. Instead, since the function would receive a CartesianMarker.Target list with all currently highlighted CartesianMarker.Targets, the entire text spoken by TalkBack for a single focus event would be returned from a single invocation, as one string. As a result, there's no issue with the repetition of x values. The provider can return any string at all, with or without the x value repeated before every y value. For the example you've given, the function could return e.g. "Year 2019. Image recognition: 9.52. Nuanced-language interpretation: -100."

Also, would this interface belong in the com.patrykandpatrick.vico.multiplatform.cartesian.marker package, or some other place would be better for it?

This can be ignored for now. It would be optimal to move things around a bit more here, converting CartesianMarker.Target to something more general (since it's now no longer specific to CartesianMarker). However, this may be challenging to achieve in a backward-compatible manner. Let's focus on it when everything else is done.

I guess I should use the height of the CartesianLayer rather than the whole Canvas, right?

Yes. You can use CartesianDrawingContext.layerBounds.

[Leedwon]

I'm not sure if we have other way to pass drawingContext to AccessibilityHighlighter.

[Gowsky]

We can improve this and avoid using MutableState which causes excessive recompositions.

The CartesianDrawingContext instance creation needs to made right before the Box composable. The canvas instance won't be available at this point. Thus, the canvas parameter of CartesianDrawingContext function can be changed into a parameter with a default value. For the default value, we can use a new private val emptyCanvas = Canvas() in the CartesianDrawingContext.kt file. For the chart to be drawn to the correct Canvas, the following lines need to be updated like so:

- chart.draw(context)
- measuringContext.reset()
+ context.withCanvas(canvas) {
+   chart.draw(context)
+   measuringContext.reset()
+ }

[Leedwon]

Done b411a30 - I'm not sure if we need rememberCartesianDrawingContext given that android.graphics.Canvas is probably not considered stable by compose, but I introduced it to follow the existing convention. Stability could also be checked and if needed compose_compiler_config could be added but this is out of scope of this PR

[Leedwon]

I wasn't sure if it should be added to CartesianChart or to CartesianChartHost. I picked the former, because it is not only compose-specific, but a core class, so the provider can be used in views and multiplatform as well

[Leedwon]

I've added several fixup! commits to revert changes related to adding contentDescription to chart entries, which are no longer relevant.

I'm keeping these commits for now to maintain visibility during review, but once the PR is approved, I'll rebase and squash the fixups to produce a clean final history.

[Leedwon]

I wanted to use val legendLabels = context.extraStore[LegendLabelKey] instead of legendLabels = data.keys to showcase the extraStore usage, as you suggested. However, for some reason, the extraStore available here doesn't contain that key, which leads to app crashes.

I briefly tried to debug it, and it looks like the key is present in the extraStore from measuringContext when creating the legend, but it's missing from the drawingContext in this part of the code. The same thing happens with the marker ValueFormatter - the key is missing during the format call.

Could you help me understand why these extraStores differ, and how we could ensure that the key is available in this context?

[Gowsky]

Could you help me understand why these extraStores differ

ExtraStore defined in MeasuringContext is used for internal drawing data. The ExtraStore, which has consumer-provided extras, is accessible via CartesianMeasuringContext.model.extraStore.

[Leedwon]

Thanks for the context, I changed it to use model.extraStore

[Leedwon]

Hi @Gowsky,

I added changes that use the Provider approach that you suggested.

Couple of things to note:

• Highlighters now span the full height of the CartesianLayer area. This change was necessary because I discovered that, without it, TalkBack could read chart entries out of order. It prioritizes top-to-bottom reading, so if an entry toward the end had a higher canvasY than one at the start, it would be read first. Additionally, entries with y-values close to 0 were sometimes skipped entirely. This update not only improves accessibility behavior but also simplifies the implementation.
• I've removed changes that are no longer relevant in this PR via fixup! commits. This keeps the PR review-friendly (no force pushes with dropped commits), and once it's approved, I’ll rebase and squash the fixups to leave a clean, final commit history.
• I ran into some trouble using extraStore for content descriptions. I’d appreciate your help here, more info can be found in this comment: Initial accessibility support in Compose #1069 (comment)

Demo:

vico_demo.mp4

[Leedwon]

Hi @Gowsky I've addressed latest comments, could you please take a look once you find some time

[Gowsky]

Hi @Leedwon. Yes, I will when I have the time.

[Leedwon]

Hi @Gowsky, just following up - any chance you'll have time to take a look in near future? Sorry to nudge, but this is a blocker for some accessibility work on our side.

[Gowsky]

There is one more important thing to add. CartesianLayerModel.Entry has no seriesIndex property which is required for ContentDescriptionProvider to label values properly. The content description for AI test scores chart uses "Image recognition" legend label for every top-most point at a given x value.

[Gowsky]

We can improve this and avoid using MutableState which causes excessive recompositions.

The CartesianDrawingContext instance creation needs to made right before the Box composable. The canvas instance won't be available at this point. Thus, the canvas parameter of CartesianDrawingContext function can be changed into a parameter with a default value. For the default value, we can use a new private val emptyCanvas = Canvas() in the CartesianDrawingContext.kt file. For the chart to be drawn to the correct Canvas, the following lines need to be updated like so:

- chart.draw(context)
- measuringContext.reset()
+ context.withCanvas(canvas) {
+   chart.draw(context)
+   measuringContext.reset()
+ }

[Gowsky]

Are you sure that it's still the case? I've run a quick check and passing chart.allTargets directly to AccessibilityHighlighter worked fine in the sample app.

[Gowsky]

This should be wrapped with a remember block.

Suggested change

	val accessibilityManager =
	context.getSystemService(Context.ACCESSIBILITY_SERVICE) as? AccessibilityManager
	val accessibilityManager =
	remember { context.getSystemService(Context.ACCESSIBILITY_SERVICE) as? AccessibilityManager }

[Leedwon]

Done 0559fcf

[Gowsky]

graphicsLayer has a better performance than offset and works well here.

Suggested change

	Modifier.offset(x = canvasX.pxToDp() - width / 2, y = canvasTopY.pxToDp())
	Modifier.graphicsLayer {
	translationX = canvasX - width.toPx() / 2
	translationY = canvasTopY
	}

[Leedwon]

Done 80711b6

[Gowsky]

The accessibility services expect String, thus returning a CharSequence here most likely isn't what we're looking for. Any extra information stored in CharSequence subclasses will be lost.

[Leedwon]

Done 06608ff

[Gowsky]

This should be moved to a separate file. It isn't used by DefaultCartesianMarker and isn't really related to CartesianMarker in general.

[Leedwon]

Done fa5bb1a

[Leedwon]

Hi @Gowsky, I’ve rebased and squashed all fixup commits to keep the history clean. From my side, the PR is ready - unless you see anything that needs changing, I think we can go ahead and merge it.

[Gowsky]

Thanks, @Leedwon. We need to wait for @patrickmichalik's review before merging it.

[Leedwon]

Hi @patrickmichalik 👋, is there a chance you could take a look sometime soon? Just asking to get an idea of when it might be merged 🙏

[Leedwon]

Hi @patrickmichalik 👋 Just wanted to check in on this - totally understand if things are busy, but I’d appreciate any thoughts or an idea of when this might get a look. Thanks so much! 🙏