Screen reader speech interruptions

[jugglinmike]

The channel for speech delivery is limited to one message at any given time (if not technically, then by convention--many users prefer not to hear multiple annunciations simultaneously). Screen readers may nonetheless send speech instructions to the operating system faster than the voice can actually render them.

In some cases, the speech instructions are queued so that the AT user eventually perceives them all. This can occur with ARIA live regions defined using the markup aria-live="polite". In other cases, the rendering of a speech instruction may be prematurely halted so that a subsequent instruction may be rendered immediately. Interruptions like this can occur with ARIA live regions defined using the markup aria-live="assertive", and they can also occur more routinely when users quickly traverse documents.

Currently, the AT Driver protocol does not include any information about the interaction between speech events. Protocol users therefore have a limited understanding of the actual user experience of rapidly-produced information.

As we learn more of the nuance behind screen reader implementations, we feel a growing need to address this deficiency. Specifically, we have recently observed interruptions occurring in unexpected contexts. Some screen readers appear to occasionally "revise" speech instructions, emitting assertive instructions in such rapid succession that no human user would ever perceive them.

The pervasive and unpredictable nature of this phenomena undermines the utility of the protocol, so we'd like to learn more about the implementation concerns and extend the protocol to accommodate them.

[css-meeting-bot]

The ARIA-AT Community Group just discussed Screen Reader Speech Interruptions.

The full IRC log of that discussion

<ChrisCuellar> topic: Screen Reader Speech Interruptions
<ChrisCuellar> github: https://github.com//issues/94
<ChrisCuellar> jugglinmike: We've always known that a screen reader makes decisions in rendering text when there's multiple concurrent messages. ARIA-live being polite or assertive is an example. But it's also common when navigating any web page. Right now, AT-driver is ignorant of those interruptions. This isn't new to us, we anticipated that this would someday be important for us. I'm bringing it up today really because of the work that Bocoup has been
<ChrisCuellar> doing to improve the consistency of the ARIA-AT automation.
<ChrisCuellar> ... In our observations, there appear to be cases where we our tests catch multiple messages that don't seem to be rendered to a human user of the AT.
<ChrisCuellar> ... So right now AT-driver may be reflecting what the AT is doing in a way that isn't meaningful to a human user.
<ChrisCuellar> Matt_King: We saw an instance of this recently with NVDA. The bot recorded double speech but most human users couldn't detect the duplication.
<ChrisCuellar> ... So the speed of interruption matters. It's possible that the AT renders parts of the utterance before it's interrupted. What's the most accurate way of recording this? Is it impossible to predict when this actually affects human users?
<ChrisCuellar> ... I'm actually wondering if what we're really highlighting here is more that we discovered a limitation of relying so heavily on human review in the ARIA-AT testing protocols.
<ChrisCuellar> jugglinmike: It's hard to verify what human testers report.
<ChrisCuellar> ... I'm also thinking about other possible users of AT-driver and what their goals might be? I'm also thinking about the case of aria-live, where it seems like this problem would come up a lot more.
<ChrisCuellar> Matt_King: AT-driver might not be able to detect product regressions.
<ChrisCuellar> jugglinmike: Incoming timestamps and priority levels of utterances might be helpful.
<ChrisCuellar> Matt_King: The new aria-notify API does have timestamps.
<ChrisCuellar> ... at least this API has this concept of priority levels. I don't know if this exists generally in AT's however. Probably not.
<ChrisCuellar> ... I wonder if priority levels should be treated as a settings thing?
<ChrisCuellar> ... Some utterances are the results of commands, others are from events (like aria-live). Some web apps like Google Docs heavily rely on aria-live to report what's happening.
<ChrisCuellar> ... One of the inherent limitations of the AT-driver concept is that we don't know whether or not the speech that is presented through the driver is in response to a command or some other event.
<ChrisCuellar> jugglinmike: That's correct. Both JAWS and NVDA aren't technically capable of linking utterances with specific user interactions.
<ChrisCuellar> ... Maybe the timestamp could be inferred from an AT-driver client. It's true that we can't know how interruptions are handled in the wild, but maybe we can make some assumptions based on what we know about human cognition. Maybe we can test for interruptions and make assertions about what's expected in terms of space between utterances?
<ChrisCuellar> Matt_King: That would be hard to do.
<ChrisCuellar> mfairchild: We can probably safely assume if the timestamps between two utterances is less than 50 milliseconds we could infer an interruption (unless the messages get queued).
<ChrisCuellar> ... For the issues we're running into? Are the problems really obvious?
<ChrisCuellar> jugglinmike: Good question. I don't have the data about timestamps yet. Maybe that would be a good first step though. Capture timestamps and filter out messages that come in with too short of an interval between the timestamps.
<ChrisCuellar> Matt_King: Yes, we can capture all of the utterances with their timestamps and then making decisions later about how to calculate the reliability of the captured utterances.
<ChrisCuellar> jugglinmike: It's starting to feel like storing this data as strings is not sufficient. We might need to talk about the data model at some point. This is connected to the issues with normalization that we had recently.
<ChrisCuellar> ... I'm glad where we got to today. This seems like something we can implement for the macOS voice and it's probably a low-lift for Vispero and NVDA. Do any APG patterns actually use aria-live?
<ChrisCuellar> Matt_King: We can look at the index by property in the APG for that info.
<ChrisCuellar> ... There's 5 patterns. Alert, 2 different Carousel patterns and 2 Date/Timepicker patterns. And Alert uses "assertive".
<ChrisCuellar> ... But the phrase spoken is really short and it's only spoken once, so it's not useful for testing interruptions.
<ChrisCuellar> jugglinmike: This sounds like a test coverage issue.
<ChrisCuellar> ... Would this fit into our roadmap at all?
<ChrisCuellar> Matt_King: I would love to have people submit test plans for aria-live specifically at some point in the future to address coverage gaps like this. APG doesn't cover everything, especially more atomic tests.
<ChrisCuellar> jugglinmike: Just to speculate on changing the Alert test to test sequences of utterances, what would that test like that?
<ChrisCuellar> Matt_King: As an aside, the other option for aria-live used to be "rude," but that became "assertive".
<ChrisCuellar> ... But back to the question, we would just need to test how utterances are rendered with sequential longer strings of text. It would be very different from the current APG example.
<ChrisCuellar> jugglinmike: How would an ARIA-AT human tester capture the output?
<ChrisCuellar> Matt_King: I
<ChrisCuellar> ... I'm actually really curious how the ATs would perform now with a test like this? I really wonder what AT vendors would think of this?
<ChrisCuellar> jugglinmike: I'm guessing that the AT doesn't know whether or not an interruption happens, but on the level of platform API's it seems possible that the screen reader could be more aware about interruptions.
<ChrisCuellar> Matt_King: It's complex to think about how all of this makes its way to the actual speech synthesizer.
<jugglinmike> scribe+ jugglinmike
<jugglinmike> ChrisCuellar: It would be great to have an AT vendor here!
<jugglinmike> ChrisCuellar: We're trying to specify this, after all. How would we want this to actually work? From the perspective of a human end-user
<jugglinmike> ChrisCuellar: It seems like interruptions should never happen unless there was an aria-live=assertive setting
<jugglinmike> Matt_King: By default, speech is always getting interrupted, and that's necessary
<jugglinmike> Matt_King: Imagine you are going through a list of files in Windows Explorer. You only need to hear the first two or three syllables to determine whether to continue to the next item or not
<jugglinmike> Matt_King: If you had to listen to every item queued up in full, you would never get anything done!
<jugglinmike> Matt_King: In a similar vein, a lot of keys are echoed. Screen readers can typically keep up with typing, but not always
<jugglinmike> Matt_King: So by default, speech is interrupted by subsequent key presses
<jugglinmike> Matt_King: And each AT even has an explicit interrupt key
<jugglinmike> ChrisCuellar: Is there any kind of user setting? Can you change the default interruption threshold?
<jugglinmike> Matt_King: Normally, you can only either turn them on or off
<jugglinmike> Matt_King: I don't know anyone who has turned off interruptions
<jugglinmike> Matt_King: You could also just "ride" the control key. That would clear the buffer after every command in NVDA. I'm not sure what would happen in VoiceOver
<jugglinmike> Matt_King: I don't know if VoiceOver even has the ability to turn off interruptions, come to think of it
<ChrisCuellar> scribe+
<ChrisCuellar> jugglinmike: So for next steps, we'll immediately specify a timestamp for speech events. We also want to explore the feasibility of value for capturing "assertiveness" or priority levels for utterances. We can also immediately extend our current consistency reports to see how tracking timestamps at different thresholds and attempting to filter for interruptions possibly improves consistency.
<ChrisCuellar> Matt_King: How long does our test runner currently wait between sending sequential commands?
<ChrisCuellar> jugglinmike: It waits 1 second after navigation and waits 5 seconds after each key press.
<ChrisCuellar> Matt_King: So we wouldn't be in danger of losing or filtering out valid speech. Ok.
<ChrisCuellar> jugglinmike: We won't make any changes to the ARIA-AT app or the bots until we collect this info from our consistency reports first.
<ChrisCuellar> That's it for the day!
<ChrisCuellar> Zakim, end the meeting