Can't comprehend these validation and test errors for Malayalam language.

[arunshekher]

I am trying to pass validation and tests for intents translated to Malayalam language (ISO 639-1: ml, Name in native script: മലയാളം).

I have this in sentences/ml/homeassistant_HassTurnOff.yaml :

language: ml
intents:
  HassTurnOff:
    data:
      - sentences:
          - "ഓഫ് ആക്കൂ <name>"
          - "<name> ഓഫ് ആക്കൂ"

I have this in test/ml/homeassistant_HassTurnOff.yaml:

language: ml
tests:
- sentences:
    - "ഓഫ് ആക്കൂ bedroom lamp"
    - "ഓഫ് ആക്കൂ ഈ bedroom lamp"
    - "bedroom lamp ഓഫ് ആക്കൂ"
    - "ആക്കൂ bedroom lamp ഓഫ്"
  intent:
    name: HassTurnOff
    slots:
      name: "light.bedroom_lamp"

When I run python3 -m script.intentfest validate --language ml I get this error:

Validation failed

Language: ml
 - sentences/ml/homeassistant_HassTurnOn.yaml: invalid format: Sentences should only contain words and matching syntax. They should not contain punctuation. @ data['intents']['HassTurnOn']['data'][0]['sentences'][0]. Got 'ഓൺ ആക്കൂ <name>'
Sentences should only contain words and matching syntax. They should not contain punctuation. @ data['intents']['HassTurnOn']['data'][0]['sentences'][1]. Got '<name> ഓൺ ആക്കൂ'
 - sentences/ml/homeassistant_HassTurnOff.yaml: invalid format: Sentences should only contain words and matching syntax. They should not contain punctuation. @ data['intents']['HassTurnOff']['data'][0]['sentences'][0]. Got 'ഓഫ് ആക്കൂ <name>'
Sentences should only contain words and matching syntax. They should not contain punctuation. @ data['intents']['HassTurnOff']['data'][0]['sentences'][1]. Got '<name> ഓഫ് ആക്കൂ'

When pytest tests --language ml -k homeassistant_HassTurnOff is ran I get these errors:

(venv) ➜  intents git:(ml_IN-intents) ✗ pytest tests --language ml -k homeassistant_HassTurnOff
========================================================== test session starts ===========================================================
platform darwin -- Python 3.10.6, pytest-7.2.0, pluggy-1.0.0
rootdir: /Users/<user>/Code/Python/intents
collected 23 items / 21 deselected / 2 selected

tests/test_language_intents.py F                                                                                                   [ 50%]
tests/test_language_sentences.py F                                                                                                 [100%]

================================================================ FAILURES ================================================================
___________________________________________________ test_homeassistant_HassTurnOff[ml] ___________________________________________________

intent_schemas = {'HassClimateGetTemperature': {'description': 'Gets the current temperature of a climate device or entity', 'domain': ...: 'Name of color', 'required': False}, 'name': {'description': 'Name of a device or entity', 'required': False}}}, ...}
language_sentences_yaml = {'_common.yaml': {'expansion_rules': {}, 'intents': {}, 'language': 'ml', 'lists': {}, ...}, 'climate_HassClimateGetTe...'}, 'cover_HassCloseCover.yaml': {'intents': {'HassCloseCover': {'data': [{'sentences': []}]}}, 'language': 'ml'}, ...}
language_sentences_common = Intents(language='ml', intents={}, slot_lists={'area': TextSlotList(values=[]), 'name': TextSlotList(values=[])}, expansion_rules={}, skip_words=[])

    def test_func(
        intent_schemas: dict[str, Any],
        language_sentences_yaml: dict[str, Any],
        language_sentences_common: Intents,
    ) -> None:
>       do_test_language_sentences(
            test_file.name,
            intent_schemas,
            language_sentences_yaml,
            language_sentences_common,
        )

tests/test_language_intents.py:165:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
tests/test_language_intents.py:73: in do_test_language_sentences
    _verify(
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

expression = RuleReference(rule_name='name')
intents = Intents(language='ml', intents={'HassTurnOff': Intent(name='HassTurnOff', data=[IntentData(sentences=[Sentence(items=[...)])}, slot_lists={'area': TextSlotList(values=[]), 'name': TextSlotList(values=[])}, expansion_rules={}, skip_words=[])
intent_name = 'HassTurnOff'
slot_schema = {'area': {'description': 'Name of an area', 'required': False}, 'device_class': {'description': 'Device class of devic...ces/entities in an area', 'required': False}, 'name': {'description': 'Name of a device or entity', 'required': False}}
visited_rules = set(), found_slots = set()

    def _verify(
        expression: Expression,
        intents: Intents,
        intent_name: str,
        slot_schema: dict[str, Any],
        visited_rules: set[str],
        found_slots: set[str],
    ) -> None:
        if isinstance(expression, ListReference):
            list_ref: ListReference = expression

            # Ensure list exists
            assert (
                list_ref.list_name in intents.slot_lists
            ), f"Missing slot list: {{{list_ref.list_name}}}. Are you missing a 'lists' entry in _common.yaml?"

            # Ensure slot is part of intent
            assert (
                list_ref.slot_name in slot_schema
            ), f"Intent {intent_name} does not support slot '{list_ref.slot_name}'. See intents.yaml for supported slots"

            # Track slots for combination check
            found_slots.add(list_ref.slot_name)
        elif isinstance(expression, RuleReference):
            rule_ref: RuleReference = expression
>           assert (
                rule_ref.rule_name in intents.expansion_rules
            ), f"Missing expansion rule: <{rule_ref.rule_name}>. Are you missing an 'expansion_rules' entry in _common.yaml?"
E           AssertionError: Missing expansion rule: <name>. Are you missing an 'expansion_rules' entry in _common.yaml?
E           assert 'name' in {}
E            +  where 'name' = RuleReference(rule_name='name').rule_name
E            +  and   {} = Intents(language='ml', intents={'HassTurnOff': Intent(name='HassTurnOff', data=[IntentData(sentences=[Sentence(items=[...)])}, slot_lists={'area': TextSlotList(values=[]), 'name': TextSlotList(values=[])}, expansion_rules={}, skip_words=[]).expansion_rules

tests/test_language_intents.py:127: AssertionError
___________________________________________________ test_homeassistant_HassTurnOff[ml] ___________________________________________________

language = 'ml'
slot_lists = {'area': TextSlotList(values=[TextSlotValue(text_in=Sentence(items=[TextChunk(text='kitchen')], type=<SequenceType.GRO...s=[TextChunk(text='ceiling fan')], type=<SequenceType.GROUP: 'group'>, text='Ceiling Fan'), value_out='fan.ceiling')])}
language_sentences = Intents(language='ml', intents={'HassTurnOn': Intent(name='HassTurnOn', data=[IntentData(sentences=[Sentence(items=[Te...)])}, slot_lists={'area': TextSlotList(values=[]), 'name': TextSlotList(values=[])}, expansion_rules={}, skip_words=[])

    def test_func(
        language: str,
        slot_lists: dict[str, SlotList],
        language_sentences: Intents,
    ) -> None:
>       do_test_language_sentences_file(
            language, test_file.stem, slot_lists, language_sentences
        )

tests/test_language_sentences.py:74:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
tests/test_language_sentences.py:51: in do_test_language_sentences_file
    result = recognize(sentence, language_sentences, slot_lists=slot_lists)
venv/lib/python3.10/site-packages/hassil/recognize.py:132: in recognize
    for sentence_context in sentence_contexts:
venv/lib/python3.10/site-packages/hassil/recognize.py:201: in _match_and_skip
    for match_context in match_expression(context, *args, *kwargs):
venv/lib/python3.10/site-packages/hassil/recognize.py:257: in match_expression
    group_contexts = [
venv/lib/python3.10/site-packages/hassil/recognize.py:257: in <listcomp>
    group_contexts = [
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

context = MatchContext(text='ഓഫ് ആക്കൂ bedroom lamp', text_index=0, slot_lists={'area': TextSlotList(values=[TextSlotValue(text_...Type.GROUP: 'group'>, text='Ceiling Fan'), value_out='fan.ceiling')])}, expansion_rules={}, skip_words=[], entities=[])
expression = RuleReference(rule_name='name')

    def match_expression(
        context: MatchContext, expression: Expression
    ) -> Iterable[MatchContext]:
        """Yield matching contexts for an expresion"""
        if isinstance(expression, TextChunk):
            chunk: TextChunk = expression
            if chunk.is_empty:
                # Skip template chunk
                yield context

            context_text_left = context.text
            chunk_text_left = chunk.text.lstrip()

            while chunk_text_left.strip():
                prefix = commonprefix((context_text_left, chunk_text_left))

                if prefix:
                    # Texts share a common prefix
                    context_text_left = context_text_left[len(prefix) :]
                    chunk_text_left = chunk_text_left[len(prefix) :]
                else:
                    # Try to skip over "skip words" or non-words
                    can_skip, context_text_left = _skip(
                        context_text_left, context.skip_words
                    )

                    if not can_skip:
                        break

            if not chunk_text_left.strip():
                # Successful match for chunk
                yield dataclasses.replace(context, text=context_text_left)

        elif isinstance(expression, Sequence):
            seq: Sequence = expression
            if seq.type == SequenceType.ALTERNATIVE:
                # Any may match (words | in | alternative)
                for item in seq.items:
                    yield from match_expression(context, item)

            elif seq.type == SequenceType.GROUP:
                # All must match (words in group)
                # NOTE: [optional] = (optional | )
                if seq.items:
                    group_contexts = [context]
                    for item in seq.items:
                        # Next step
                        group_contexts = [
                            item_context
                            for group_context in group_contexts
                            for item_context in match_expression(group_context, item)
                        ]
                        if not group_contexts:
                            break

                    for group_context in group_contexts:
                        yield group_context
            else:
                raise ValueError(f"Unexpected sequence type: {seq}")

        elif isinstance(expression, ListReference):
            # {list}
            list_ref: ListReference = expression
            if (not context.slot_lists) or (list_ref.list_name not in context.slot_lists):
                raise MissingListError(f"Missing slot list {{{list_ref.list_name}}}")

            slot_list = context.slot_lists[list_ref.list_name]
            if isinstance(slot_list, TextSlotList):
                text_list: TextSlotList = slot_list

                if context.text:
                    # Any value may match
                    for slot_value in text_list.values:
                        value_contexts = match_expression(
                            dataclasses.replace(context), slot_value.text_in
                        )

                        for value_context in value_contexts:
                            value_context.entities.append(
                                MatchEntity(
                                    name=list_ref.slot_name, value=slot_value.value_out
                                )
                            )
                            yield value_context

            elif isinstance(slot_list, RangeSlotList):
                # List that represents a number range.
                # Numbers must currently be digits ("1" not "one").
                range_list: RangeSlotList = slot_list
                if context.text:
                    number_match = NUMBER_START.match(context.text)
                    if number_match is not None:
                        number_text = number_match[1]
                        word_number = int(number_text)
                        if range_list.step == 1:
                            # Unit step
                            in_range = range_list.start <= word_number <= range_list.stop
                        else:
                            # Non-unit step
                            in_range = word_number in range(
                                range_list.start, range_list.stop + 1, range_list.step
                            )

                        if in_range:
                            context.entities.append(
                                MatchEntity(name=list_ref.slot_name, value=word_number)
                            )

                            yield dataclasses.replace(
                                context, text=context.text[len(number_text) :].lstrip()
                            )

            else:
                raise ValueError(f"Unexpected slot list type: {slot_list}")

        elif isinstance(expression, RuleReference):
            # <rule>
            rule_ref: RuleReference = expression
            if (not context.expansion_rules) or (
                rule_ref.rule_name not in context.expansion_rules
            ):
>               raise MissingRuleError(f"Missing expansion rule <{rule_ref.rule_name}>")
E               hassil.recognize.MissingRuleError: Missing expansion rule <name>

venv/lib/python3.10/site-packages/hassil/recognize.py:331: MissingRuleError
======================================================== short test summary info =========================================================
FAILED tests/test_language_intents.py::test_homeassistant_HassTurnOff[ml] - AssertionError: Missing expansion rule: <name>. Are you missing an 'expansion_rules' entry in _common.yaml?
FAILED tests/test_language_sentences.py::test_homeassistant_HassTurnOff[ml] - hassil.recognize.MissingRuleError: Missing expansion rule <name>
==================================================== 2 failed, 21 deselected in 0.66s ====================================================