Idea to improve the impact of our work as language leaders

[jlpouffier]

Morning @home-assistant/language-leaders !
(I'm JLo, I work at Nabu Casa as product manager for Home Assistant)

I am currently thinking about a solution with @matthiasdebaat and would like your feedback before going further.

Problem(s)

I'll present two problems, only one of these is focused on Language leaders but the solution I'll explain further down solves both

• As a user, I do not know what to do when Assist fails to understand my intent
• As a language leader, I do not know what sentences are missing, and where my users struggle

Solution

The first problem is partly solved with our better error handling that will soon be available in the next release of home assistant.
This:

    no_intent: "Sorry, I couldn't understand that"
    no_area: "Sorry, I am not aware of any area called {{ area }}"
    no_domain: "Sorry, I am not aware of any {{ domain }} in the {{ area }} area"
    no_device_class: "Sorry, I am not aware of any {{ device_class }} in the {{ area }} area"
    no_entity: "Sorry, I am not aware of any device or entity called {{ entity }}"
    handle_error: "An unexpected error occurred while handling the intent"

This means that a user will have bits of context about how to solve a particular issue.

A dummy example:
If you say "Turn off the lights in the bathroom" and assist replies "Sorry, I am not aware of any lights in the bathroom area", it's a clue on how to solve this: You need to assign lights in the bathroom area.
It's already a great start.

I am thinking about going one step further:

I'd like to provide a summary of all the sentences that failed on a user system (Somewhere ... We don't know where yet, we are brainstorming with @matthiasdebaat)

This list of unmatched sentences would be split into two parts:

Solvable by the user

Here the idea is pretty simple, it's to create resolutions flows tailored to a specific error that can be solved directly by the user.

A few examples to illustrate:

• You said "Turn off the lights in the spaceship", but I do not know any area called spaceship
  • Resolution flow: Here are your areas, pick one and I will add spaceship as an alias
• You said "Open the curtains in the living room", I am aware of the living room area and you do have curtains ... but none of them are assigned to the living room
  • Resolution flow: Here are your curtains, pick the ones you want to assign to the living room area and I'll do it.

Not solvable by the user

This is the part where I need your feedback.
I see all the sentences said by our users that currently do not match any intent as very actionable data.
And I would like to use this data in a way.

Of course, while keeping the privacy DNA in mind, we cannot just "push" this data somewhere without user consent - That's not us.

What I am thinking about is to

• List all sentences that a user said that we failed to match
• Provide a simple button to push this sentence to the relevant language leader so he/she is aware that a user expects this sentence to work

Pretty simple.

We'll need to be pretty clear in the wording and UX behind this button, I don't see this button as a "Report bug" that requires a fix at some point in time.
I see it more as a data point that a user I willing to share with you: "I was expecting this sentence to work, and it does not" (End of the story)

Pros

• You'll get clear signals of what's not working in your language. Some of these sentences will be able to be added to current intent (like a missing common sentence to turn on a light in dutch that the leader just did not think about)
• We (all of us) will get clear signals on the most important missing intents (Clean a room? Set the position of a cover? Who knows today ... )

Cons

• Spam spam spam. We need to find a solution that does not spam you. I'd like to avoid having one GitHub issue per button click, I'd like to find a way to consolidate this into a table somewhere so that you can act on it at your own pace.
• Non-actionnalbe data, I foresee a lot of failed intents due to mismatched speech-to-text (A very common one is "Closed the garage door" instead of "Close the garage door")

What do you think?

Would you like to know to top 10 failing sentences in your language to work on them?
Do you think the solution solves the problem?
Feel free to discuss here ☺️

Thanks a lot!

JLo

[LaStrada]

Would you like to know to top 10 failing sentences in your language to work on them?

This would definitely help! Here in Norway we have a lot of dialects, and people write differently (and with typos). It's challenging for me to keep track of things that don't work. Although we all understand what's written, certain words can be spelled differently depending on which part of the country you live in.

I've noticed people complaining in forums, facebook etc, and I've encouraged some of them to either make a PR themselves, or for simple things, I've fixed it myself. The key here is that's much easier with input/feedback from the users, and it would be nice to have an official feedback channel for this.

[TheFes]

I fully agree we should give more feedback to the users, and help them in solving the issues. But, like you already said, we should prevent bringing a lot of workload to the language leaders.

We as language leaders can provide translations, and tweak the sentences for supported intents so that as many ways to ask eg turn on a light are supported. However we can not add support for new intents, because this requires changes we are unable to make. We all understand the YAML structure and the way the intents are set up, but to support an intent changes to the intent processing and HA Core are needed, which most of will probably be unable to do.

For example, from the start of the whole intents repo, over a year ago, there have been intents created to set the temperature of a climate entity. But this is not supported yet. So, basically our task as language leader to provide translations is done. However if a user will try to do this, he will get the no_intent or handle_error message, and might submit an issue to us as language leaders.
This will mean we will have a lot of work in answering issues we are unable to solve.

On the other end I do agree it would be good if we, as language leaders, would have an feedback channel for cases where described by @LaStrada

In any case I think it's really good that there will be more focus on this, and that the users will get better understanding of why something is not working. The suggestion about getting the top 10 lists of failing sentences does sound like a good middle ground. It will also give some more insight in which domains or intents should get focus to get implemented. For example, I can imagine a lot of people try to control a media player now, which is not working.

[jlpouffier]

Yes in fact the solution solves yet another problem that is not for you to solve, which could be phrased as:
"As the home assistant organization, we do not know what to focus on next for the intent support"

The "list of non-matching intent" will contain two types of sentences

• Sentences that could be added to existing intents, making them stronger, you can help here
• Sentences that require a new intent, and these could be used as signals to prioritize them on core side

So we really need to find a way not to overload you, that is definitely not the aim

[tetele]

In the tool that @jlpouffier proposed, I'd add a dropdown labeled "What are you trying to do?", with options for each supported intent and a last option called "Something else" or something, which would then notify core codeowners instead of language leaders.

[jlpouffier]

Ouuuh. Nice

[tetele]

This is an excellent sum-up with very good suggestions. It does, however, open some new questions.

How can you tell if something is solvable by the user or not?

A sentence may not be supported or it may have to do with an unexposed entity, a miss-spelled alias, a lack of assignment to an area for an entity or other reasons. Can we surely determine if any sentence falls under the user-fixable category?

Obviously, sentence formats differ between languages. Can we actually figure out what the user was trying to achieve?

I'll provide an example in English. Say someone has a Kitchen area and a Kitchen lights light, then says turn on the kitchen lights.

Context	Result	Explanations
Kitchen lights is exposed and assigned to the Kitchen area	Kitchen lights turns on	N/A
Kitchen lights is not exposed, but assigned to the Kitchen area	all exposed lights in the Kitchen area turn on	the sentence matched is that regarding the area, not the name
Kitchen lights is exposed, but not assigned to the Kitchen area	Kitchen lights turns on	the sentence matches the name
Kitchen lights is not exposed and not assigned to the Kitchen area	all exposed lights in the Kitchen area turn on	the sentence matched is that regarding the area, not the name
Kitchen lights is not exposed and no other lights exist in the Kitchen area	Mismatch error for area, mismatch error for name (?)	no sentence matches

There may be other cases, but a single pair of overlapping sentences (turn on [the] {area} {device_class} and turn on [the] {name}) generates at least those above.

My suggestion

I love the "Send this sentence to your language leader" functionality. Obviously, we need to check for duplicates to eliminate spam and to make sure there is a language leader to handle the issue, but otherwise it's great!

However, I'd not draw the line between user-solvable and non-user-solvable. I'd let the user try to debug a sentence, provide assessments based on all defined sentences (including custom sentences and sentence triggers, btw, since they can surely alter the behavior). Watch out for the fact that sentence triggers don't have an assigned language (yet, at least).

The tool could list all possible matching sentences (again, there could be multiple sentences, as I listed in the table above) along with the causes that prevent matching and any steps the user might take to help with matching and retry logic. If all steps are exhausted without any favorable result or if there are no match candidates (sentences), only then I'd allow the users to report the issue upstream, to language leaders.

For the mistmatched STT, Mike's already been working on some "similar sentence" mechanism that, when applied to phonemes instead of letters (see the discussion in the comments), could greatly aid in such cases.

[jlpouffier]

Ok I love the idea of having multiple resolution paths based on name, area or others.
Noted!
Thx @tetele!

[Uriziel01]

In general I'm not against any of the proposed solution, maybe not a great fan of using the "Assist failed because <***>" mechanism as a solution from language leader's perspective , sure it's usable to self-debug things from the user's perspective, this still very much CAN or CAN NOT be reproduced on my end, it's a lottery.

If it's intended only for user needs, yeah this is fantastic for the end user, I strongly agree.

But from a language leader's perspective it's way more valuable to have some sort of debug data sent to github/issue board/whatever (given user did consent to this), even just as a templates issue where we can discuss it and/or triage it, maybe get some more details from this or other users as well.
Of course it would still needs to contain more than a basic error code/reason. If that would be implemented I would be more than happy to help our users with the issues there are having because I would be able to at least reproduce then and didn't had to resolve top ure guesswork.

[felixschndr]

Provide a simple button to push this sentence to the relevant language leader so he/she is aware that a user expects this sentence to work

(As a user) I would totally be fine with that. Some thoughts:

• Show a list of all unrecognized sentences and let the user deselect all the ones he does not want to share.
• Maybe even share the sentences automatic as an opt-in part of Analytics?