[RFC] Specifying how terminals process Unicode text

[kovidgoyal]

Hi all,

Following on from the text-sizing work in #8226 I have decided to specify the exact algorithm terminals should use to split Unicode text into cells and implement it in kitty. It is based on the Unicode specification's Grapheme segmentation rules but in addition also specifies things particular to terminals not covered in the Unicode spec. It fixes various long standing issues such as #3810 (emoji with zwj) and #8433 (Korean text).

The specification is here. Feel free to read it and comment. There might well be differing opinions on the parts not covered by the Unicode spec. I am open to suggestions for modification.

From kitty users, I would appreciate if some of you can run nightly and report if there are any issues. It's possible you will have issues if you use ZWJ based emoji in your workflows, as the width kitty assigns to these has changed, and terminal programs may use a different width than the correct one.

In master, there is also a kitten that can be run easily to test a terminal's compliance with the spec. It uses grapheme test data from the Unicode consortium. Run it as:

kitten __width_test__

Here are results of running it on various terminals.

Terminal name	Number of tests failed
kitty (master)	0
kitty 0.41.1	45
wezterm 5046fc22	179
foot 1.21.0	186
konsole 24.12.3	280
iTerm2 3.5.13	289
gnome-term 3.56.0	317
kitty-master+tmux-3.5	347
xterm 397	371
Apple terminal 2.14	479

And finally, we have ghostty 1.1.3, on which the test kitten failed to run because ghostty returned way more cursor position reports than were expected, something badly broken there. I did happen to look at its code as it claims to do grapheme segmentation, and it doesn't implement the segmentation algorithm correctly anyway.

[kovidgoyal]

Closing since there were zero comments in a month and kitty 0.42.0 has been released with the new code anyway. If anyone points out serious concerns with the spec I am still open to revisions, but the bar is now higher.

[Wukuyon]

I think the proposal looks great. I have one general comment.

If you haven't already, you should look into participating in the Unicode Text Terminal Working Group and asking for feedback of this proposal there.

The Text Terminal WG is an official Unicode Working Group that formed in 2023 to standardize terminal character widths, signaling between TUI applications and terminals, complex scripts in terminals, and defining what makes a font "text terminal compatible". The Unicode Text Terminal WG is also closely related to ISO WG21 SG16, the C++ committee's Unicode study group, because the Text Terminal WG's current chairperson Fraser Gordon is also in SG16.

The Unicode Text Terminal WG seems like exactly the kind of place that should be aware of the great de facto terminal standards you've been making, like this one and #8226.

The WG got off to a slow start because the working group's original chairperson became unavailable, but they appointed a new chairperson and seem to have been more active in the past year. Their most recent report is from last November (L2/24-264), where they say they also have been 'working to define algorithms for consistent and correct display of‬‭ complex (shaped, bidirectional and/or “wide”) text in text terminals and other applications‬‭ that display text on a fixed-width grid.'

I don't know how someone would join the Unicode Text Terminal WG, but they apparently meet monthly. I know that @ctrlcctrlv has participated in it (see L2/23-153 and L2/23-194). I think it would be a good idea to make your voice and proposals heard there too, in the future, if you haven't tried already.

That's all and thank you for your work.

[kovidgoyal]

IIRC I did email them at @ctrlcctrlv 's urging a few (couple?) years ago, and never heard back. In any case if they are interested in my input, I am quite easy to contact and happy to participate.

[00-kat]

https://sw.kovidgoyal.net/kitty/text-sizing-protocol/#gseg states the following:

Warning

This algorithm is under public discussion in #8533. Until that issue is closed, it is subject to change based on feedback from the community. Additionally, in the future if the Unicode standard changes in ways that affect this algorithm, it will be updated. Currently the algorithm is based on Unicode version 16.

This issue is closed though, so shouldn't the first part of that warning be removed now?

[kovidgoyal]

Sure, done.

[Wukuyon]

I have some more feedback about the algorithm for splitting text into cells. I apologize that this is coming after you closed the issue, but I think it's important enough.

First, let me say again: the Kitty text sizing protocol is an excellent design. Like I said in ghostty-org/ghostty#5563 (comment), I think that giving power to TUI apps to define expected text widths has the potential to be one of the most important innovations in the entire TUI ecosystem. Inconsistencies between terminal emulators and TUI applications over character widths have been a persistent problem. A widely adopted standard like this would greatly improve interoperability. Kudos for the hard work.

This feedback is in response to neovim/neovim#34733 (comment). I will argue that some combining marks should behave as spacing marks, and furthermore some grapheme extenders should behave as spacing marks. I will cite normative clauses from the Unicode Standard where possible.

The algorithm assumes that "extended grapheme clusters take the width of their initial character". In other words, it assumes that "all non-initial characters in a cluster are nonspacing marks" that have zero width.

This assumption works well for many cases:

• Combining character sequences with nonspacing marks (D53).
• Emoji sequences.
• Hangul syllables.

But the assumption does not hold for all extended grapheme clusters.

1. Spacing marks (General Category Mc):
   • There are two subtypes of combining characters: nonspacing marks (D53) and spacing marks (D55).
     • Nonspacing marks include all characters with the General Category of Nonspacing Mark (Mn) or Enclosing Mark (Me) (D53).
     • All combining characters that are not nonspacing marks are spacing marks (D55), i.e., all characters with the General Category of Spacing Mark (Mc) (D52).
   • Rule UAX29-GB9a prevents grapheme cluster breaks before SpacingMark characters, which include characters with a General Category of Spacing_Mark (Mc).
   • UAX44, Table 12 describes Spacing_Mark (Mc) as "a spacing combining mark (positive advance width)". This implies that Spacing_Mark (Mc) characters should each take up a positive width, even though they are combining characters.
   • Furthermore, D55 in the Core Specification defines spacing marks (any Spacing_Mark/Mc character) and says, "In general, the behavior of spacing marks does not differ greatly from that of base characters."
   • In other words, according to D55, spacing marks should take up positive advance width, like base characters do (D51).
   • Consider the two examples from D55: U+093F DEVANAGARI VOWEL SIGN I ( ि) and U+0BCA TAMIL VOWEL SIGN O (◌ொ).
   • When you combine ि into कि or खि or ◌ொ into டொ or ணொ, each of those is a single extended grapheme cluster. That's useful for line breaking and text selection.
   • But putting each of those clusters in a single halfwidth terminal cells each is probably a bad default. Imagine shrinking कि, खि, டொ, or even ணொ into a single halfwidth terminal cell each. They would become unreadable to any reader.
   • This is why the Unicode Standard defines ि and ◌ொ as spacing marks, which the Standard says have "positive advance width", like base characters.
   • And so, terminals should treat them as such and give them positive advance width, unlike nonspacing marks.
   • (Arguably, டொ and ணொ should take up three cells, since ◌ொ is a split spacing combining mark. See Core Specification, Section 4.3.1, "Split Class Zero Combining Marks". It's worth considering making spacing combining marks that have Indic_Positional_Category=Left_And_Right, like ◌ொ, have an advance width of two in terminals.)
2. Grapheme extenders that are also base characters (U+FF9E & U+FF9F):
   • Rule UAX29-GB9 prevents grapheme cluster breaks before Extend characters, which include characters with the Grapheme_Extend property.
   • D59 defines grapheme extenders (characters with Grapheme_Extend) to include:
     • All nonspacing marks.
     • Some spacing marks that are the second parts of two-part combining marks.
     • The letters U+FF9E HALFWIDTH KATAKANA VOICED SOUND MARK and U+FF9F HALFWIDTH KATAKANA SEMI-VOICED SOUND MARK.
   • Those last two halfwidth katakana letters are special among the grapheme extenders. They are letters (General Category Letter/L), and therefore they are base characters (D51).
     • All base characters are "independent graphic characters" that can form combining character sequences (D51).
     • Why are U+FF9E and U+FF9F, among all base characters, uniquely classed as Grapheme_Extend? L2/03-106 explains: they have compatibility decompositions to nonspacing marks. But compatibility decompositions do not imply identical behavior (D66); unlike their compatibility decompositions, U+FF9E and U+FF9F are still spacing base characters. Unicode added them to Other_Grapheme_Extend mainly to prevent awkward line breaks in Japanese text, not because they should be zero-width.
     • As base characters, U+FF9E and U+FF9F also should take up advance width and occupy cells.
     • (Notably, the UTC was considering making U+FF9E and U+FF9F combining marks, rather than letters. But Japan's representatives vetoed this and insisted on making them base characters instead (L2/06-246). But even Japan let them become combining marks, if they were also spacing marks, then they should still have positive advance width and take up their own terminal cells.)

My proposed adjustments to the standard:

• Update the splitting algorithm so that Step 6 uses a profile of UAX29-C1-1 with the following modifications:
  1. Always break before SpacingMark characters (i.e., remove rule UAX29-GB9a).
  2. Also, always break before U+FF9E..U+FF9F (because they are base characters).
• Update step 4 in the width algorithm:
  • Make it define only Nonspacing_Mark (Mn) and Enclosing_Mark (Me) characters as zero width, and make it exclude Spacing_Mark (Mc) characters, as per D55 and UAX44.
  • Maybe add a new step saying that Spacing_Mark (Mc) characters that have Indic_Positional_Category=Left_And_Right have width two, as per Section 4.3.1.

I also want to post about this new standard to the Unicode public mailing list. (You can post it there yourself if you prefer.) Guidance from the Unicode gurus like Ken Whistler, Mark Davis, or Asmus Freytag (or the Text Terminal Working Group, if it's still active) would be really valuable.

In summary, I hopefully made a good case that there are several combining (or grapheme-extending) marks that should not behave like nonspacing marks:

• Spacing marks (General Category Mc) are combining marks, but the Standard says they have positive advance width, like base characters (D55, UAX44). Collapsing combining character sequences with spacing marks, like कि or டொ, into halfwidth cells by default makes them unreadable and goes against the Standard.
• U+FF9E and U+FF9F are uniquely both grapheme extenders but also letters (D51). As base characters, they should occupy space, as the Standard says, regardless of their status as grapheme extenders.
• For these two reasons, UAX29-C1-1 alone is not sufficient for segmenting text among terminal cells. The algorithm should break before spacing marks and the special grapheme-extending katakana letters.

With all that said, I'm still a big fan of this protocol and I support it. Let me know what you think of the suggestions, and let me know whether you're okay with me posting the protocol on the Unicode public mailing list. I'm cheering for this proposal.