How to search for square brackets and underscores?

[h0lg]

I'm struggling to find a working query for the expression [ __ ] ([\u00A0__\u00A0] in JSON representation).
The error message I get is Expected at least one query part to be parsed. Same for the query \[.
Querying for __ yields no results, even though it should.

Am I missing something, is this a bug or a query DSL corner case that needs special escaping?

[mikegoatly]

Hi @h0lg - always nice to hear from you!

TL;DR:

Try setting SplitOnPunctuation to false.

---

By default LIFTI will treat some characters as "split" characters:

public virtual bool IsSplitCharacter(char character)
{
    return
        // Split when the character is well known as a Unicode separator or control character
        char.IsSeparator(character) || char.IsControl(character) || (
            (
                // Split if we are splitting on punctuation and the character is a punctuation character
                (this.Options.SplitOnPunctuation == true && char.IsPunctuation(character)) ||
                // Or the character is in the list of additional split characters
                this.additionalSplitChars.Contains(character)
            )
            // Unless the character is an ignored characters
            && this.ignoreChars.Contains(character) == false
       );
}

In your expression:

• \u00A0 is a separator
• _, [ and ] are all punctuation characters

So depending on the configuration of your index, it's possible that none of that text will be indexed as they will all be considered split characters.

During query tokenization, search terms are also fed through the index's tokenizer in order to split text appropriately. In your case, the query tokenizer is complaining that although you gave it some text to search for, once everything was split up, no search terms were found.

So if you need to index and search on puncuation characters like [ and _, you'll need to set SplitOnPunctuation to false when defining your index.

I hope that helps!

[h0lg]

Hey @mikegoatly :) Thank you once more for your quick and competent support!

I've updated my index tokenizer config (for YouTube subtitles) with .SplitOnPunctuation(false).SplitOnCharacters(',', '.', '?', '!', '"') and now searches return the expected results.

.WithDefaultTokenization(o => o.AccentInsensitive().CaseInsensitive() // allows for easier matches
    /* only split on custom punctuation chars to enable searching for placeholders and compound words
     * like "ever-lasting" or "can't" or "[Laughter]" or "[ __ ]", see
     * https://mikegoatly.github.io/lifti/docs/index-construction/withdefaulttokenization/#splitonpunctuationbool
     * https://research.google/blog/adding-sound-effect-information-to-youtube-captions/
     * https://support.google.com/youtube/answer/6373554?hl=en#zippy=%2Cpotentially-inappropriate-words-in-automatic-captions */
    .SplitOnPunctuation(false).SplitOnCharacters(',', '.', '?', '!', '"'))