Attempt to track if the user is using keyboard navigation

[maribethb]

Check for duplicates

• I have searched for similar issues before opening a new one.

Problem

It has become more clear through user testing that we need to track whether or not the user is actively "using" keyboard navigation. For users who exclusively use the mouse, showing the keyboard navigation cursor is likely to be confusing and distracting. For users who exclusively use the keyboard, not showing the cursor is confusing as they will lose track of where their actions will take place.

Until now, we've been acting as if using the keyboard to initiate an action means you're using keyboard navigation, and we show the cursor after that action.

However, consider the user who primarily uses the mouse, but uses the "delete" key to delete a block. They've never previously interacted with keyboard navigation, and now all of a sudden they have a flashing cursor on their workspace that they don't know what to do with.

What we don't want is to have the user have to activate a "keyboard navigation mode." Keyboard nav should always work but be invisible if it's not being used.

Request

See if we can come up with a heuristic to determine the user is using keyboard navigation -- not just that they've used a keyboard shortcut like delete or copy/paste. I think some keyboard actions like using tab to navigate between the page and blockly, using t to open the toolbox, using the arrow keys to navigate the workspace, are all "keyboard navigation" behaviors while delete, enter to commit a field change, cut/copy/paste are all "keyboard shortcut" behaviors

Open questions:

• Can we determine that you've focused Blockly via tab vs via clicking?
• We have at least an idea of things you could do that mean you want to start using keyboard navigation. How would we know if they are no longer using it? e.g. they accidentally pressed the up arrow and now they can see a cursor. Does using the mouse immediately take them out of keyboard nav mode?
• What is the full list of behaviors that mean you're using keyboard navigation?
• What if we get it wrong? Is there a way the user can manually opt in to see a cursor?

If we had such a heuristic, we could query some method that tracks it when we're doing a "keyboard shortcut" behavior and use it to determine whether or not to show the cursor.

Alternatives considered

Another way of framing this is is the idea of "only show the cursor after an action if it was already visible." That would be a great heuristic for things like pressing the delete key. But how do you ever get a cursor to show up in the first place then? You would need some list of actions that always show the cursor... like using t to open the toolbox, using the arrow keys to navigate the workspace, etc. You end up still needing a list of actions that ultimately signify that the user is using keyboard navigation, so this framing has all the same open questions as above.

---

Instead of a method to track if we're using keyboard navigation, we could instead have all actions update the location of the cursor (e.g. when a block is deleted, move the cursor to the workspace or to the previous connection or whatever) but not toggle the visibility of the cursor (e.g. deleting a block does not change the visibility). Instead, keyboard-navigation-specific actions would always trigger the cursor to visible, and mouse actions would trigger it to not be visible (assuming the cursor appearance is distinct from the block selection). This is probably simpler, if the cursor visibility is indeed distinct from the idea of a block being selected.

Additional context

No response

[maribethb]

Talked over with Rachel today and here's a brain dump of what we discussed/decided

Basics

An item that is focused is the same thing as the item pointed at by the cursor saying "current node location"
An item that is selected is not necessarily the same thing that is focused (especially in multiselect mode)

All selectable items that are currently focused are also selected (i.e. you cannot be active-focused on a block while it is not selected). (this implies that focusing a selectable item should select it which is not really related to this issue but i'm not sure if it's true today or who should be responsible for doing that, I guess the cursor)

Selected items are always visualized. That is, if an item is selected, it always has some visual indicator of such.
Focused items are not always visualized. It depends on the item. Some items should only be visually focused if the user is using keyboard navigation, like a block connection. But others are always visualized regardless of mouse/keyboard use [note that the examples we came up with during our chat I think don't actually apply, but they theoretically could]

There are cases where something is both selected and focused. How do we design showing both of these cases without causing conflicting styles?

focus is styled with css, selection is currently styled with svg, though there is some desire to move this to css, but that's outside the scope of this issue.

if both were style in css you could write good enough descriptors to say "apply this style to things that are selected and focused" and "apply this style to things that are selected but not focused" and/or your styles could just be redundant (e.g. if they both change the background color to blue then there is no conflict). But today you don't want to necessarily style something in css if it's already being styled in svg.

The only "selectable" things in blockly at the moment are blocks and bubbles (including workspace comments). So we can just special case these two things. i.e. if the css classes contain "blocklyActiveFocus" and "blocklyBlock" then don't add any css, because you know it's already being styled with svg.

What manages the focus state?

the FocusManager keeps track of the currently focused node. the cursor might tell the focusmanager when it changes, but the source of truth is the focus manager. the focus manager adds/removes css classes to items when they gain or lose active or passive focus.

What manages the selected state?

Today, only one thing can be selected at a time and it's just set through essentially a global variable in the form of Blockly.common.setSelected(). The selection is updated in a variety of ways, including by context menu items (e.g. duplicating a block), the object itself's event handlers (e.g. workspace comments select themselves in a gesture handler), etc. Objects that implement the ISelectable interface have select and unselect methods that are called by Blockly.common.setSelected() and it is in these methods that styling is done.

In the future we might want the cursor to track which item(s) are selected, but that is outside the scope of this issue

Sooo what do we do about turning on/off focus visualization?

1. Add enableFullCursorVisualization and disableFullCursorVisualization methods to Blockly (name definitely up for debate).
   • Where this should live is up for debate. It doesn't really make sense for it to live in the cursor, because the cursor is primarily concerned with a workspace, and this is a global state (if you're using keyboard nav in one workspace you're using it in all).
   • Doesn't make sense to live in the workspace either for the same reason
   • It could go in FocusManager but it doesn't necessarily make sense that the focus manager cares about if you're using keyboard navigation or not... it's related but this doesn't feel right.
   • It could go in our normal catchall, Blockly.common but Rachel really really doesn't want to dump new things there
   • It could go in a new namespace like Blockly.KeyboardNavigation or Blockly.Accessibility or something.
2. This method would add/remove a class to the root level of the page like blocklyFullCursor or whatever we decide based on (1)
3. It is up to the individual components that write their own styles to take this class into account. For example, a connection could have the css that makes it look focused be in a selector like .blocklyFullCursor > .blocklyActiveFocus.blocklyConnection instead of only selecting .blocklyActiveFocus. But elements that always are styled differently can ignore this class. This gives flexibility to custom components and also allows applications to decide differently which items are always/sometimes shown.
4. Certain keyboard actions (like T for toolbox, I for insert, using the arrow keys to navigate between stacks, etc.) should call enableFullCursorVisualization and certain mouse actions (like selecting anything with the mouse or dragging a block) should call disableFullCursorVisualization. Ambiguous actions (like pressing the delete key) should do neither.
5. Actions that change focus should always update the cursor location / currently focused node regardless of the setting.

[BenHenning]

@maribethb the basics section makes sense to me with one caveat: we should be a bit careful when referring to something as focused since there are really several distinct things that could mean focused:

• DOM focus (might have Blockly active focus).
• Blockly active focus (always has DOM focus).
• Blockly passive focus (never has DOM focus).

Not to mention that, from a user perspective, selection might be considered 'focus' as well.

Regarding the design dilemma, it occurs to me that we will need something in Core that's generally responsible for managing keyboard navigation. Since we only care about the cursor during keyboard navigation, perhaps we could introduce a KeyboardNavigationController class? This seems like a reasonable place to move the long-term routing logic from the plugin, and also a reasonable place to own enabling/disabling cursor visualization.

[microbit-matt-hillsdon]

I have tried for a very simple version of this idea in the plugin while looking at active/passive styling on google/blockly-keyboard-experimentation#511 (demo). This could be enough?