Feature: add character/word/line index attributes to the text geometry

[lojjic]

Custom shaders applied to Text instances could enable some really nice animation effects. For many of these, you'd want to treat each character/glyph independently, and for that you need something in the shader telling you which character is currently being rendered.

Theoretically gl_InstanceID could be used for this, since we use instancing for the glyph quads. And this kinda-sorta works: https://codesandbox.io/s/zealous-water-m8lzq?file=/src/index.js -- but gl_InstanceID is only available in WebGL2, and it appears to be broken in ANGLE implementations when used within functions other than void main. So it's not realistically usable for this right now.

Instead, we could add our own instance attribute, something like attribute float charIndex;, which just holds an incrementing character index. Custom shaders could then make use of that.

I'd probably want to make it an opt-in feature, something like textmesh.includeCharIndexInShader = true, just to avoid creating that extra attribute array if it isn't needed.

[lojjic]

Also maybe: wordIndex, lineIndex ...?

[gsimone]

word and line would be enable per line and per word animation, much like https://greensock.com/splittext/ (which we can consider a good example of what this change would enable)

Am I right assuming this would also allow per-token (char, word, line) changes in the fragment shader?

[lojjic]

Am I right assuming this would also allow per-token (char, word, line) changes in the fragment shader?

You'd have to pass it from vertex to fragment as a varying, but yes. :)

[lojjic]

Down the rabbit hole we go...

I can think of uses for all the following indices, plus a total count for each one:

• charIndex, totalChars
• wordIndex, totalWords
• lineIndex, totalLines
• charInWordIndex, totalCharsInWord
• charInLineIndex, totalCharsInLine
• wordInLineIndex, totalWordsInLine

We'd want to make all of these opt-in and do some smart packing to minimize the number of new glsl attributes we're introducing. I'll have to think more about the API for this.

[gsimone]

we could let users pick ALL of them with flags, so

{
split: { words: true, characters: true, lines: true }
}

do some smart packing

Mind expanding on this? I'm curious 👐

[lojjic]

I like that idea of letting users choose the split(s).

By packing I just mean we don't want to add 12 new attribute float foo declarations or we'll hit the limit (I think it's 16 attributes total in webgl), but we could pack those into a maximum of 3 new attribute vec4 foo declarations, and then unpack them to nicer-looking float variables within the shader.

[gsimone]

Oh yeah, makes a lot of sense, thank you!

[lojjic]

Should the charIndex include whitespaces in its incrementing? Or just increment for visible glyphs? I'm leaning toward just visible glyphs.

[gsimone]

Counting whitespaces would create weird staggers when animating by index, so I'd also lean toward visible

[lojjic]

I've got lots of cleanup to do, but I have an initial POC of the above counts. Here are examples using the various index/total pairs to change color in a fragment shader:

charIndex / totalChars:

charInWordIndex / totalCharsInWord:

charInLineIndex / totalCharsInLine:

wordIndex / totalWords: (looks very similar to charIndex/totalChars in this example but there's a subtle difference)

wordInLineIndex / totalWordsInLine:

lineIndex / totalLines:

I managed to get all the data into a max of 3 uniforms + 2 attributes, which is pretty good. I still want to make them optional.

[gsimone]

Amazing! Let me know when you have a test version ready 😄

[FunMiles]

I was looking for doing a feature that this could address. I have a bunch of 3 letter labels to put around a sphere. I am making them always face the viewer and make adjustments so that on the screen they are always the same size, no matter the scaling applied in the view matrix or modeling matrix.
Right now, I create one Text object per label and apply a transformation.
I was thinking I could get a much better performance by packing all the labels in a single text and have an attribute to have the position on the sphere of each label to do the correct transformations in the vertex shader.
@lojjic can you document how to create custom shaders? I think adding the attribute should not be difficult as the existing method setAttribute of BufferGeometry should do.

The other option I was considering would be instanced rendering, since all my labels all have 3 letters. But that would require WebGL 2 for sure and might be too complex to be worth it, at least at the start.

[lojjic]

@FunMiles I think you're right, this may facilitate that sort of thing. If I'm understanding it correctly, optimization could involve two parts:

1. Moving the rotation/scaling logic from CPU-side JS to GPU-side vertex shader logic
2. That plus combining multiple pieces of text into a single draw call

For 1, take a look at this comment, that may actually be sufficient for your needs already. That's also a good demonstration of how to apply a custom shader, basically just assigning it as the material. I've used Troika's createDerivedMaterial utility for it there, but you could similarly pass any ShaderMaterial or a material with its own onBeforeCompile modifications.

For 2, I think you're right you could render a single Text and then use some attribute to displace individual characters around your sphere. The new char/word index attributes described in this issue may be of some help, but I'm not sure they're even necessary. You could probably just encode those displacements into a new InstancedBufferAttribute, where each of its vectors holds the displacement for one character (the Text's geometry is a simple quad that is instanced for each glyph, so additional InstancedBufferAttributes will be stepped through for each glyph.)

I'm interested to see if you have any luck doing this!

[FunMiles]

@lojjic You understood correctly. And the code you linked does 90% of what I need for (1). Your demo eerily matches my use case.
If I may ask you to correct my understanding, here is how I see what you are doing:

• mvPosition is initialized with the reference point position in the model-view space. I presume that reference point to be the anchor point. Is that right?
• position is the offset from that reference point and I presume it only has x and y non-zero coordinates. (which makes me think you can skip computing the z-component of the scale?) (PS: Looking at the instanced rendering vertex shader code, it seems one can rotate the text, in which case the z component won't be zero)
• The scale in each direction is recovered on the assumption that the 3x3 block only contains a composition of scaling and rotations.
• The scaled position is added, and if the z-component is zero, this contribution keeps the position in a plane parallel to the projection plane.

The changes for me to this code for just doing (1) would be mostly in the scaling and a slight coordinate shift so that the text stays outside of the sphere I have.

You did understand 2 as well. However I am a bit confused when reading the code. I had not realized that the text rendering is already instance rendering. What confuses me is that you do have GlyphsGeometry deriving from InstancedBufferAttribute and yet Text is a subclass of Mesh and not of InstancedMesh. I guess I have to dig deeper into Three.js.
Otherwise, given that each character is instanced, I would need just an attribute with a vector to offset position for each glyph instance. Is that right?

[lojjic]

I've had to step away from this for a bit, but here's a quick status update:

The PR #109 feels pretty solid in terms of collecting and exposing the various counts. I want to make them opt-in but otherwise I'm happy with where it's at.

However, I have a strong hunch that shader-based animations are going to require not only these new counts, but also maybe access to some other data like:

• The current glyph's quad bounds
• Overall block bounds
• Font metric info like baseline/ascender/descender which can't be inferred from the quad
• Other?

Some of these are already technically present in the shader, but if users are going to depend on them then they'll need to be exposed with friendly names and documented as a reliable contract.

If anyone has time to play with that PR branch and try implementing some shader animations, and let me know what pieces of info are missing, that would be a big help.

[ffdead]

@lojjic Just want to express my interest in this feature 😺

[canadaduane]

I resolved some merge conflicts with latest master here: https://github.com/canadaduane/troika/tree/char-indices

I haven't had a chance to test it yet, but intend to in the next few days.

[FunMiles]

I've been away from this discussion for a long while. However I was just reminded that I want to get back to looking at it.
At the end of this post is an example of something I've done using createDerivedMaterial.
However, right now, there's one mesh per label and when the number of labels becomes very large, the rendering gets a big jerky on low end phones. I am assuming it is due to the CPU calls for each label. To address that, I would like to replace the multiple meshes with a single one with all the label and use an improved vertex shader to do the rest.

I think that, as @lojjic mentioned, I would need some bounds on the words. Or at least, for my use, the center of each word.
Any other suggestion on how to do it otherwise or on how to get that center?

https://flightlog-beta.vercel.app/circles?route=BRU-HER-ATH-IST-AMM-CAI-TUN-CMN-RAK-MAD-LIS-ZRH-GVA-NCE-CDG-DUS-FRA-MUC-LHR-LCY-LGW-DUB-HBA-SYD-MEL-CNS-KTM-DEL-BLR-COK-MAA-CMB-MLE-HKG-HND-NRT-KIX-ICN-GMP-MNL-DPS-JOG-CGK-SIN-KUL-PEN-LGK-BKK-SGN-HAN-LPQ-CNX-DOH-AUH-DXB-JNB-CPT-LVI-ZNZ-JRO-DAR-SSH-BUD-OTP-IPC-SCL-PMC-GRU-EZE-GIG-BSB-LIM-LPB-BOG-PTY-SJO-SJU-CUN-ACA-MEX-MTY-JFK-BOS-PHL-DCA-IAD-BWI-CLT-ATL-MSP-MIA-DFW-DTW-ORD-IAH-COS-DEN-PHX-SAN-LAX-SJC-SFO-SEA-PDX-YVR-YYZ-YUL-LGA-EWR-ASP-BNE-AKL-CHC-ZQN-VIE-SJJ-VCE-MXP-FCO-BCN-PEK-PVG-HNL-OGG-KOA-HIJ-RGN-PBH-TIP-AMS-CPH-SLC-CVG-GSP-TPE-ULN-PNH-VTE-ABQ-BUF-MCO-HKT-MDL-