Missing property to indicate needed grouping?

[NSoiffer]

Summary

Some intents are formed from linear notations and some from 2d notations. That information is lost when the intent is used and so the need to verbally group the intent (such as would be done for "fraction x plus 1 over y end fraction") is lost.

The Problem

Intent is typically layered on top of the presentation. Consider $(a+b) ⨯ c$. A likely mrow version of the intent would be:

   <math>
    <mrow intent="cross-product:infix($x, $y)">
      <mrow arg="x">
        <mo>(</mo>
        <mrow>
          <mi>a</mi>
          <mo>+</mo>
          <mi>b</mi>
        </mrow>
        <mo>)</mo>
      </mrow>
      <mo>⨯</mo>
      <mi arg="y">c</mi>
    </mrow>
   </math>

Because the presentation requires the parens and because they are likely to be part of the arguments to the intent, speech will naturally work because they will be spoken and listeners will understand what the arguments to the cross product are.

However, a \binom macro, An author/generating software would understand how that is laid out and would know that no parens are needed assuming it generates a 2D layout such as $\binom{n+k}{j+1}$. The corresponding concept is binomial-coefficient:infix and the MathML would be:

<math>
  <mrow intent="binomial-coefficient:infix($a, $b)">
    <mo>(</mo>
    </mrow>
    <mfrac linethickness="0">
      <mrow arg='a'>
        <mi>n</mi>
        <mo>+</mo>
        <mi>k</mi>
      </mrow>
      <mrow arg='b'>
        <mi>j</mi>
        <mo>+</mo>
        <mn>1</mn>
      </mrow>
    </mfrac>
    <mo>)</mo>
  </mrow>
</math>

Because this is a core concept, MathCAT knows that "binomial-coefficient" should be spoken as "choose", so "a choose b" or with more complicated args from the example "binomial coefficient n + k choose j + 1 end binomial coefficient". But for open concepts, there is nothing to say grouping around the construct is needed. MathCAT could decide to add parens around the arguments, or more accurately, it could add start xxx/end xxx as it does for fraction. But what tells MathCAT that this needs to happen for xxx:infix but not cross-product:infix?

Potential Solution

One potential solution is to say that AT needs to figure this out. In a simple case like above, the AT can look at the argument and if is non-trivial and not bracketed, look at the ancestors up to the element with the intent. If there is ancestor that has 2D layout (maybe not counting the base of msub, munder, etc.). If any of the arguments has a 2D ancestor, then bracketing words "start binomial ... choose ... end binomial" potentially are used.

I'm not sure this always works though with reference args. For example, what if the intent were $f($x, $y)? Is the stopping point still the intent? Is there a more complicated example where it isn't the element with the intent but some descendent?

Another potential solution is to add a property where the author says whether this requires grouping/bracketing. In a discussion with @davidcarlisle, he suggested a :grouped or :fenced property. If we introduced a similar property, is AT constrained to use it if present and assume its absence indicates that it should not be grouped?

[davidcarlisle]

Just to expand a bit on @NSoiffer's final comment the (half formed) suggestion was that

<mrow intent=":fenced">...</mrow>

would be an invisible mfenced analogous to invisibletimes that in visual layout would be a normal mrow but gave a hint that AT might (or should?) voice the start and end.

I suspect we don't want to say that if this intent is not there that the mrow should not be announced, as in many cases AT is probably better placed to pick a suitable reading verbosity than the author.

[dginev]

I don't understand the problem exactly. An intent function application fun(arg1,..., argn) has implied groups for both the outer expression, as well as groups for each of the arguments.

Whether those should be vocalized in practice likely depends on AT speech style preferences, as well as the size of the subexpressions.

The :infix property doesn't really make that inference any harder, an AT could choose to voice out for plus:infix(a,b,c)

"start subexpression" a plus b plus c "end subexpression"

Without any additional markup hints.

[NSoiffer]

It would be an awful experience to hear "start subexpression ... end subexpression" around each operator. Consider $ax+b=0$: If someone used intents for times, plus, and equals and AT didn't know about them, you would hear "start equals start plus start times a x end times plus b end plus equals 0 end equals".

If someone needed to do grouping with a linear form, they would use parens and rely on people's knowledge of "PEMDAS": $(a+1)x+b = c$ would be spoken with the parens so that you know the $a+1$ is the argument of times. But for fractions, one doesn't write those parens. fun:infix(arg1, arg2) carries no information to distinguish those two forms. That leaves AT (maybe) using a heuristic to figure out whether to use "start fun... end fun" or not.

Note: I wrote "(maybe)" because maybe there is an algorithm to determine this. I don't know if my suggestion of searching upwards in the presentation to mark whether bracketing words might be needed is a heuristic or works all the time.

[dginev]

Yes, but you didn't answer my actual point above - an :infix intent functional expression already carries all needed grouping information, as the arguments are clearly separated, as is the outer context.

I agree that the AT needs to judiciously make the choice when to vocalize grouping indicators. But I think that is the AT's job, more so than the author's ? In fact if :fenced was a necessity, I would question if we made some fundamental design mistake in how we formulated the intent functional syntax - it was supposed to be able to avoid low-level speech overrides.

I suspect any subexpression of non-trivial size, let's say 10 subterms for sake of example, will always need delimiting grouping speech, be it in a fraction context, an infix context, or just a large summand in plus.

The one case where :fenced may appear useful is in partial annotation over "legacy" subtrees. Your (a+1) example seems to assume that (a+1) is raw presentation MathML with no further intent, when you'd need to specify it is fenced. But if there was an intent="plus:infix(a,1)", I believe all grouping information is already marked up, even if AT had no prior knowledge of "plus".

[davidcarlisle]

I hadn't planned on suggesting :fenced it just came up in chat with Neil immediately prior to this issue to a wider audience. Partly I think it's good if there is a way to make explicit a default heuristic, just as a system can often infer multiplcation from juxtaposition , &invisibletimes; allows that system (or the author) to make that inference explicit so downstream applications don't need to.

Perhaps such a property may be useful here.

as for whether it's actually needed other that for explicitly exposing system behaviour I'm, not sure. Even if Neil can come up with a heuristic in mathcat that covers all reasonable cases, other systems may not, and an author may want to give a stronger hint that some group should be spoken and such an intent might give a simple system-neutral way to give such a hint.

[NSoiffer]

Hmm. This problem seems to be more subtle to others than I think it is. Let me try again. Below are two bits of MathML with intents on them:

<mfrac intent='foo:infix($arg1, $arg2)>
   <mrow arg='arg1'> <mi>a>/mi> <mo>+</mo> <mn>1</mn> </mrow>
   <mi arg='arg2'>x</mi>
</mfrac>

<mrow intent='bar:infix($arg1, $arg2)>
   <mrow arg='arg1'>
      <mo>(</mo>
     <mrow> <mi>a>/mi> <mo>+</mo> <mn>1</mn> </mrow>
      <mo>)</mo>
   </mrow>
   <mi arg='arg2'>x</mi>
</mrow>

Without some sort of indicator, maybe the listener knows "foo" binds less or more tightly than "plus" in "a plus 1 foo x" or maybe they don't. There is no way for AT to know this at the moment. So AT probably should add "start foo"/"end foo" to be clear, but that isn't needed if the listener is aware of precedences, such as "foo" being "equals". If the mfrac was in another mrow, some sort of grouping would almost certainly be needed because the $a+1$ is not grouped. Can AT always know this when the intent is complicated? That's the main thrust of my initial post/comment.

For the "bar" example, parens group the first arg, so AT doesn't need to add grouping if it stands alone. However, if there is an mrow around the bar, we are back to the same situation as with foo: AT probably should add "start bar"/"end bar" to be clear, but that isn't needed if the listener is aware of precedences, such as "bar" being "equals".

Although I have phrased this as "if the listener is aware that...", in fact, the author should know their audience and so I could have written "if author knows that...". In particular, they should know whether the speech needs to be grouped or not. Hence, if there is not a way for AT to know what is needed, intent should provide a way for the author to express this. As with all features of intent, if AT knows this is not relevant for its audience, it can ignore this. An example of this is that for braille, for the most part, intent is ignored because most braille described the layout, not the meaning.

[dginev]

Can AT always know this when the intent is complicated? That's the main thrust of my initial post/comment.

In times of doubt, adding explicit grouping seems like a good decision in complex expressions. So a property with the opposite effect could even be more useful, as with :terse.

To me this is more fundamental than operator precedence. As I mentioned above it is also about basic "legibility" of the text. A large argument (say 10 subterms, such as with a large numerator of a fraction) needs some grouping indicators, or the reader will lose track of which level of the expression they are at.

[brucemiller]

This is somewhat like operator precedence, but more along the lines of AT babbling long enough in the numerator that the listener gets confused before the denominator comes. But it seems to me to be the AT's task to figure that out, and probably depends a lot on the listener as well. I'd think this would be impractical for authors and generators to decide if and where such adjustments were needed.

[MurrayIII]

FWIW, the UnicodeMath expression (\choose → ⒞) (𝑛+𝑘)⒞(𝑛−𝑘)=0 looks like

and is spoken as "open n + k close choose open n minus k close = 0". UnicodeMath encloses compound numerators, denominators, subscripts, superscripts, etc., in parentheses. The outermost parentheses are discarded on building up to 2D. But (𝑛+𝑘)/(𝑛−𝑘)=0 is spoken as "fraction n + k over n minus k , end fraction = 0". Somehow "binomial coefficient n + k over n - k, end binomial coefficient seems too wordy.

[dginev]

seems too wordy.

Typical of toy examples with brief arguments. You can also use the "choose" connector word for multinomials, and those get quite large. See for example equation (2.8) and similar in arXiv:0804.3841

https://arxiv.org/pdf/0804.3841

https://ar5iv.labs.arxiv.org/html/0804.3841#S2.E8

[MurrayIII]

Agree that for large arguments, it'd be better to use the "binomial coefficient ... over ... end binomial coefficient". It's not hard for AT to make this choice.

[davidcarlisle]

Perhaps this would be less needed if we accepted #524

The example above

<mfrac intent='foo:infix($arg1, $arg2)>
   <mrow arg='arg1' intent=":argname"> <mi>a</mi> <mo>+</mo> <mn>1</mn> </mrow>
   <mi arg='arg2'  intent=":argname">x</mi>
</mfrac>

begin arg1 x + 1 end arg1 foo begin arg2 x end arg2

<mfrac intent='foo:infix($arg1, $arg2)>
   <mrow arg='arg1' intent=":argname"> <mi>a</mi> <mo>+</mo> <mn>1</mn> </mrow>
   <mi arg='arg2'  >x</mi>
</mfrac>

begin arg1 x + 1 end arg1 foo x