Please include the actual ranges mapped to, not just the bit-operator-code to get there

Currently I find the spec easy to read for implementing it, but very bad to read for understanding what the actual effect is in code point space. Particularly, I have trouble figuring out the actual ranges in the code point space the invalid surrogates are mapped to, and I find this suboptimal given Unicode already has a lot of confusing ranges.

The problem IMHO is in particular this section, or the lack of concrete ranges given afterwards:

4. Potentially ill-formed UTF-16

A sequence of 16-bit code units is potentially ill-formed UTF-16 if it is intended to be interpreted as UTF-16, but is not necessarily well-formed in UTF-16. It effectively encodes a sequence of code points that do not contain any surrogate code point pair.

Note: Like UTF-16, potentially ill-formed UTF-16 can not represent a surrogate code point pair since the corresponding surrogate 16-bit code unit pair would instead represent a supplementary code point. Unlike well-formed UTF-16, it might contain isolated surrogate code points.

Any sequence of 16-bit code units has an interpretation as potentially ill-formed UTF-16.

WTF-16 is sometimes used as a shorter name for potentially ill-formed UTF-16, especially in the context of systems were originally designed for UCS-2 and later upgraded to UTF-16 but never enforced well-formedness, either by neglect or because of backward-compatibility constraints.

Don't get me wrong, this formal definition is nice, but this is only followed up by actual encoding steps that are on the other extreme end and way too practical, filled with bit transform ops that don't make it obvious what ranges are actually used.

What I would have expected in the 4. Potentially ill-formed UTF-16 section is something like this addition (possibly incorrect, this is my best guess & what I would have liked to have properly spelled out):

previous stuff removed -> see next comment for revised, better suggestion

Ah, it seems the surrogates are just translated as-is to the same code point? Maybe this text would do then:

4.0.1 Concept of Decoding Ill-formed UTF-16 to Code Points "The WTF-8 Way"

Before the detailed decoding algorithm below, this section will first explain on a code point level how to read Ill-formed UTF-16 for WTF-8 use.

• All 16-bit values in range 0x0000 to 0xFFFF excluding surrogates in 0xD800 to 0xDFFF can be interpreted as code points as-is. This is how valid UTF-16 is always parsed.

• All surrogates in range 0xD800 to 0xDFFF which are valid surrogate sequences << link this to the part that explains valid surrogates need to be translated as such to get the appropriate code points that exceed the regular UTF-16 range (U+10000 and higher). This is also how valid UTF-16 is always parsed.

• For invalid surrogate pairs or orphaned surrogates in 0xD800 to 0xDFFF, each single surrogate value is directly translated to the according U+D800 to U+DFFF code points. This derivates from standard UTF-16 handling, where these invalid surrogates would be rejected: these could points would usually not be used after being done decoding from UTF-16, but now we use them to retain invalid surrogates.

This results in the final code point range being the full U+000000 - U+10FFFF (at the time of writing) as usual, with the code point range U+D800 t U+DFFF inside that used for keeping the invalid surrogates. This way, we can reinserted them later when converting to WTF-8.

[SimonSapin]

I find this suboptimal given Unicode already has a lot of confusing ranges.

I’m sorry, but brushing aside this particular aspect of Unicode as confusing is not gonna work here since dealing with it is what WTF-8 is all about.

I’d suggest studying the definitions of these terms, or the equivalent ones at https://unicode.org/glossary/:

• Unicode code point
• Unicode scalar value
• Surrogate code point
• Surrogate 16-bit code unit

I have trouble figuring out the actual ranges in the code point space the invalid surrogates are mapped to

Hopefully the definitions and names of surrogate code points v.s. surrogate 16-bit code units should strongly suggest that they map to each other. The formal definition is in the decode from potentially ill-formed UTF-16 to code points algorithm, the step relevant to your question is:

Otherwise, append to result a code point of value U.

Note that there is no arithmetic involved in that step, only a "type conversion".

@SimonSapin what is wrong with adding the text I suggested?

My suggestion was some clarifications to add, just to clarify this.

Edit: just to explain better:

The formal definition is in the decode from potentially ill-formed UTF-16 to code points algorithm

This one I find personally(!) not that readable as a start because it is too hands on, too bit-shifty. It just feels unnecessarily super-verbose instead of explaining the basic mapping first, which would be useful for people who don't want to just copy out the bit-shift code without understanding. That is why I suggested the above text as an additional entry point, not a replacement - which I think would actually encourage reading & understanding the specs more properly, instead of "brushing aside" as you suggest.

I was in no way trying to suggest anything that is already there is wrongly defined, or should be removed, or would be formally incomplete, or should be rewritten. I'm sorry if it sounded like that.

(edit 3: revised comment a little, was all over the place. my bad)

[SimonSapin]

My previous comment was dismissive, I’m sorry.

I feel that the text you propose does not belong as-is because it would be somewhat redundant with existing text. There’s probably some higher-level explanation that could be added instead. But this document is something I made back in 2014, unless there are bugs I’m not all that interested in spending time polishing it.

Thanks for the response! Here is why I think this is useful despite "just" repeating info:

If you look at the document structure: You got 1.+2. Background & Motivation (very good! and very in-depth too, I liked those), then 3. has very in-depth terminology which was also very useful. Now, I would expect an overview over the algorithm in plain English: but now, in section 4, we first get4. Potentially ill-formed UTF-16 (important & relevant, but not how WTF-8 actually works itself!). Summed up, super extensive info all around on everything but what transforming to WTF-8 looks like, both on a concrete but also on a transformation function level.

Next: 4.1 Encoding. Super in-depth steps, bit shifts, exact instructions. And I know the problem, but what is the exact solution we're after? I don't know at all, I just know the rough sets between which we are moving but not how or why exactly. So I have trouble piecing things together: what is moving where and why? What is the exact target state we're shooting for?

The addition attempts to fix this: a new overview what-does-it text, right before encode/decode super detail steps what-does-it text in 4.1, so that people actually head into the latter with an idea of what they are about to see. Is that really so redundant, even if it technically speaking "just" summarizes the 16-bit to codepoint transformation in a more conceptual way first, if that is what helps readers understand the super detailed steps?

I get not wanting to be too redundant, but I feel like you could try to ease people into it at least a little more. 🤷‍♀️ (or am I just not smart enough to read it? I feel like my addition would really help though)

I’m not all that interested in spending time polishing it.

That is a shame, because I got to it days ago and it's still very relevant. Windows and it's weird UTF-16-but-also-any-16-bit filesystem...

Edit: updated the suggested text a few comments above some more

[SimonSapin]

Out of curiosity: where would WTF-8 be relevant to you?

For example it is used inside the Rust standard library as the memory of the OsStr and OsString types (used for filename and such strings from the operating system). But it’s an implementation details, users of Rust don’t need to care about the details.

@SimonSapin I might be writing a programming language runtime from scratch soon where I need to deal with this. Also I got curious about that one library I use, icculus' common PhysFS filesystem library, and indeed that one appears to break with invalid UTF-16 on windows (collapses all invalid chars into one "broken" char, making the mapping irreversible). So I mentioned WTF-8 in a mailing list post to the developers pointing this out.

I mean obviously it's not relevant to people using higher level runtimes and libs, but I do tend to write the occasional lower level backend code 🤷‍♀️ and there's always people who will need to keep doing that