Repository navigation
Correlating user intents with speech responses #90
Description
We've designed the protocol to decouple user intents and AT responses. The remote end responds to commands to perform an intent with a generic "success" or an "error". Separately (and at any moment), the remote end notifies clients of AT responses (e.g. speech).
For example, if the "press keys" user intent completes successfully, the remote end responds with a generic "success" message. If a screen reader is being automated, and the simulated key press triggers some kind of vocalization, the remote end must eventually emit a "captured output" event which is not correlated to the key press.
This de-coupled design does not necessarily match all end-user's expectations. Screen readers typically respond to key presses with some kind of vocalization, after all, so some users may expect a correlation. Clients effectively have to determine cause-and-effect on their own based solely on timing.
The protocol's current design was informed by a number of considerations:
- our earliest implementations of the protocol were technically unable to correlate user intents and speech events
- we know that an evented system is necessary to fully encompass AT behavior given the potential for unpredictable output from, e.g. ARIA live regions
- we suspect that a strong correlation may be infeasible in some ATs due to architectural constraints
However, we believe there is still time to modify this design, so we would like to invite folks to reconsider it with us, today.
- For users: does shouldering the burden of cause-and-effect fundamentally undermine any of your use cases?
- For implementers: would it be onerous for your AT to respond to user intents (e.g. key presses) with the response they triggered?
Activity
jscholes commentedon Jul 13, 2025
I view this as a non-starter for NVDA.
When an input action directly triggers output, e.g. pressing NVDA+F12 to speak the current time, there is no traceability of that causality through the system. That is, it is not possible to work backwards from the output and determine that it was generated by some particular input, nor to work forwards from some particular input and work out that it will generate some particular output.
In purely technical terms, it might be possible to inspect the call stack. But while it might tell you that the gesture handler for NVDA+F12 called
ui.messageorspeech.speech.speak, it doesn't necessarily follow that any output was generated, that the output was spoken without interruption, that the output wasn't modified by another part of the system en route to the synthesiser, etc. Meanwhile, patching a screen reader with code that is constantly inspecting its call stacks at runtime would likely result in a performance hit at best, and instability/unrepresentative results at worst.A more important point is that in NVDA's case, many user input actions do not directly generate any output. Consider the common case of a user moving the system focus with the Tab key:
Even in cases where NVDA has its own internal "hacks" for handling certain user intents, these are implementation details and subject to change at any point.
Finally, I think the conceptual assertions encoded in this proposal are fundamentally false. It cannot be guaranteed:
v-tab-vispero commentedon Jul 14, 2025
Regarding JAWS, we've discussed it internally before and concluded that generally there is no good way to match a speech output to a triggering action.
jugglinmike commentedon Jul 14, 2025
During today's "automation subgroup" meeting, we reflected on the perspective shared here and determined that the correlation described above is not only technically infeasible but also fundamentally misguided. In the interest of surfacing the design constraint to audiences who lack the experience of the experts in this discussion, I agreed to draft a succinct non-normative note for the spec. We'll track that in issue gh-91.
Thanks to all for helping us reaffirm the current design and (ultimately) reduce confusion for the spec's readers!