Add option to use UTF-8 glyphs

[DoomHammer]

As progress on #200 and #244 seems to have stalled, here is the code, I've used to enable UTF-8 glyphs on an HUB75 RGB LED Panel with Adafruit-GFX-Library.

To test it, I've created a UTF-8 enabled .h font based on the open Hack font using ./fontconvert ~/Downloads/ttf/Hack-Regular.ttf 4 380 > hack-regular-4.h (380 is the codepoint of the last letter used in Polish writing, you may need to substitute it if you plan to use other characters).

I'm also tagging @idea--list and @Bodmer as the original authors of the code used here.

[edo44]

Very nice

Just it doesn't work with symbols above code point 10175

This is the font I used for testing
https://fonts.google.com/noto/specimen/Noto+Emoji

[DoomHammer]

@edo44 can you share which command did you use to generate the font and the code to display it? Would be easier to test what might be the problem this way.

Thanks!

[edo44]

@edo44 can you share which command did you use to generate the font and the code to display it? Would be easier to test what might be the problem this way.

Thanks!

fontconvert.exe NotoEmoji-Medium.ttf 12 128512 128591 > notoemoji-medium12.h

#include <SPI.h>
#include <Wire.h>
#include <Adafruit_GFX.h>
#include <Adafruit_SSD1306.h>
#include <Fonts/notoemoji-medium12.h>

#define SCREEN_WIDTH 128 // OLED display width, in pixels
#define SCREEN_HEIGHT 64 // OLED display height, in pixels

/*TwoWire I2CBME = TwoWire(0);
I2CBME.begin(22, 18, 400000);*/

// Declaration for an SSD1306 display connected to I2C (SDA, SCL pins)
// The pins for I2C are defined by the Wire-library. 
// On an arduino UNO:       A4(SDA), A5(SCL)
// On an arduino MEGA 2560: 20(SDA), 21(SCL)
// On an arduino LEONARDO:   2(SDA),  3(SCL), ...
#define OLED_RESET     -1 // Reset pin # (or -1 if sharing Arduino reset pin)
#define SCREEN_ADDRESS 0x3C ///< See datasheet for Address; 0x3D for 128x64, 0x3C for 128x32
Adafruit_SSD1306 display(SCREEN_WIDTH, SCREEN_HEIGHT, &Wire, OLED_RESET);

void setup() {
//  Serial.begin(115200);
//  pinMode(22, INPUT_PULLUP);
//  pinMode(18, INPUT_PULLUP);
//  Wire.begin(22, 18);
  // SSD1306_SWITCHCAPVCC = generate display voltage from 3.3V internally
  if(!display.begin(SSD1306_SWITCHCAPVCC, SCREEN_ADDRESS)) {
    Serial.println(F("SSD1306 allocation failed"));
    for(;;); // Don't proceed, loop forever
  }
  display.clearDisplay();
  display.setTextColor(1);
  display.setCursor(0,48);
  display.setFont(&NotoEmoji_Medium12pt8b);
  display.print("😀😁😂");
  display.display();
}

void loop() {
}

[mcer12]

This is a great work! The fact that there's been an UTF-8 fork for 4 years but there's no UTF-8 support in the official library is very sad. Every related issue raised here seems to result in another frozen pull request or goes nowhere. We're in the year 2023 when ESP32 or RP2040 board is cheaper than Arduino Nano. I am just now making a library for a custom display where I'd like to use extended latin but now I'm not sure if I should use the old dusty fork or this official library and hope it gets implemented one day... Hopefully this gets merged! :)

[mcer12]

@edo44 can you share which command did you use to generate the font and the code to display it? Would be easier to test what might be the problem this way.
Thanks!

fontconvert.exe NotoEmoji-Medium.ttf 12 128512 128591 > notoemoji-medium12.h

#include <SPI.h>
#include <Wire.h>
#include <Adafruit_GFX.h>
#include <Adafruit_SSD1306.h>
#include <Fonts/notoemoji-medium12.h>

#define SCREEN_WIDTH 128 // OLED display width, in pixels
#define SCREEN_HEIGHT 64 // OLED display height, in pixels

/*TwoWire I2CBME = TwoWire(0);
I2CBME.begin(22, 18, 400000);*/

// Declaration for an SSD1306 display connected to I2C (SDA, SCL pins)
// The pins for I2C are defined by the Wire-library. 
// On an arduino UNO:       A4(SDA), A5(SCL)
// On an arduino MEGA 2560: 20(SDA), 21(SCL)
// On an arduino LEONARDO:   2(SDA),  3(SCL), ...
#define OLED_RESET     -1 // Reset pin # (or -1 if sharing Arduino reset pin)
#define SCREEN_ADDRESS 0x3C ///< See datasheet for Address; 0x3D for 128x64, 0x3C for 128x32
Adafruit_SSD1306 display(SCREEN_WIDTH, SCREEN_HEIGHT, &Wire, OLED_RESET);

void setup() {
//  Serial.begin(115200);
//  pinMode(22, INPUT_PULLUP);
//  pinMode(18, INPUT_PULLUP);
//  Wire.begin(22, 18);
  // SSD1306_SWITCHCAPVCC = generate display voltage from 3.3V internally
  if(!display.begin(SSD1306_SWITCHCAPVCC, SCREEN_ADDRESS)) {
    Serial.println(F("SSD1306 allocation failed"));
    for(;;); // Don't proceed, loop forever
  }
  display.clearDisplay();
  display.setTextColor(1);
  display.setCursor(0,48);
  display.setFont(&NotoEmoji_Medium12pt8b);
  display.print("😀😁😂");
  display.display();
}

void loop() {
}

Emojis in arduino code is a sight in and of itself :D

[edo44]

@edo44 can you share which command did you use to generate the font and the code to display it? Would be easier to test what might be the problem this way.
Thanks!

fontconvert.exe NotoEmoji-Medium.ttf 12 128512 128591 > notoemoji-medium12.h

#include <SPI.h>
#include <Wire.h>
#include <Adafruit_GFX.h>
#include <Adafruit_SSD1306.h>
#include <Fonts/notoemoji-medium12.h>

#define SCREEN_WIDTH 128 // OLED display width, in pixels
#define SCREEN_HEIGHT 64 // OLED display height, in pixels

/*TwoWire I2CBME = TwoWire(0);
I2CBME.begin(22, 18, 400000);*/

// Declaration for an SSD1306 display connected to I2C (SDA, SCL pins)
// The pins for I2C are defined by the Wire-library. 
// On an arduino UNO:       A4(SDA), A5(SCL)
// On an arduino MEGA 2560: 20(SDA), 21(SCL)
// On an arduino LEONARDO:   2(SDA),  3(SCL), ...
#define OLED_RESET     -1 // Reset pin # (or -1 if sharing Arduino reset pin)
#define SCREEN_ADDRESS 0x3C ///< See datasheet for Address; 0x3D for 128x64, 0x3C for 128x32
Adafruit_SSD1306 display(SCREEN_WIDTH, SCREEN_HEIGHT, &Wire, OLED_RESET);

void setup() {
//  Serial.begin(115200);
//  pinMode(22, INPUT_PULLUP);
//  pinMode(18, INPUT_PULLUP);
//  Wire.begin(22, 18);
  // SSD1306_SWITCHCAPVCC = generate display voltage from 3.3V internally
  if(!display.begin(SSD1306_SWITCHCAPVCC, SCREEN_ADDRESS)) {
    Serial.println(F("SSD1306 allocation failed"));
    for(;;); // Don't proceed, loop forever
  }
  display.clearDisplay();
  display.setTextColor(1);
  display.setCursor(0,48);
  display.setFont(&NotoEmoji_Medium12pt8b);
  display.print("😀😁😂");
  display.display();
}

void loop() {
}

Emojis in arduino code is a sight in and of itself :D

It's 2023 after all ;)

[mcer12]

@edo44 I finally got to start working with this fork and what's not mentioned (or maybe I'm blind) is that you have to enable UTF8 support with:
display.utf8(true);
Also @DoomHammer enabling/disabling it doesn't seem to affect program size which is probably the point? :) At least for me with ESP8266 the program size is the same.

[DoomHammer]

You're right @edo44 . Perhaps I should add a simple example of how this works.

[edo44]

I enabled utf8 in my library, yep.

By the way I now understand why emoji can't be shown, the utf decoder it's 8-16 bit while emoji are stored in 3 bytes.

[DoomHammer]

Ah, good catch, haven't thought about that :D

[Crapy]

Would be interesting to edit getTextBounds as well to work with custom unicode fonts

[BillyDonahue]

I had this idea to do some backwards-compatible refactoring of the GFX base class so that the entire text engine becomes a pluggable interface similar to how Font* pointers are installed today.

I actually got it working and it makes the GFX library's footprint smaller! This is because it doesn't then have to link the code for text engines that it doesn't need (like the currently mandatory "legacy" font"). It would also allow the use of arbitrary embedded font libraries and their formats. I should probably find it and put it up for others to use! It would be a solution to all of these unicode troubles that people keep bringing up, and make the GFX library more compact at the same time.

[Bodmer]

I doubt if refactoring the library will be accepted given the Adafruit "prime directive" constraint. The library purpose is mainly a display sales enabler so there is little increased profit incentive for any significant improvement due to the increased support/documentation/example load on Adafruit support staff.

I think the way forward is to create independant compatible font rendering support libraries. This approach then has zero impact on the graphics library.

As an example there is already an excellent compatible library available which can render TrueType fonts on a TFT screen. This has been developed by takkaO.

I have created a branch with some bug fixes here. I added "flicker free" background rendering and more efficient line based rendering(rather than pixel by pixel). That library provides access to compact font files, with fully scaleable anti-aliased glyphs. Left, middle and right justified text can also be printed to the screen. I have added TFT_eSPI specific examples to the OpenFontRender library and tested on RP2040 and ESP32 processors, the ESP8266 does not have sufficient RAM due to the glyph render complexity. Here is a demo screen where a single 12kbyte font file binary was used to render fully anti-aliased glyphs of gradually increasing size on a 320x480 TFT screen:

https://i.imgur.com/bKkilIb.png

This is a complex library but the interface principles are quite simple. The graphics library just needs to support drawPixel and draw horizontal line of a given length.

[DoomHammer]

@Bodmer my use case is with LED matrix displays, so I don't think the suggested approach would work there.

[Bodmer]

@DoomHammer, it will work. The approach has been used by this library for a long time.
https://github.com/olikraus/U8g2_for_Adafruit_GFX

Incidentally that library supports extended fonts.

[BillyDonahue]

@Bodmer, the "prime directive" you're referring to is:

The PRIME DIRECTIVE is to maintain backward compatibility with existing Arduino sketches

I would not break backward compatibility. The API of a Adafruit_GFX class would stay the same.

But Adafruit_GFX can be internally composed of a core AND a parameterized text system as an implementation detail.

New code can, OPTIONALLY, make a variant GFX object that uses the same Adafruit_GFX::NoText core object, but attaches a different text system to it or none at all if it's an application without text. Then you don't pay for the memory for rendering engine you aren't using, which is kind of brutal. There's no option to cut out the builtin default small font bitmaps or rendering code currently.

We've similarly found ways to removed the memory for the splash bitmaps in the past. AVR footprint is very important. This gives much more room on the Arduino Uno for other app features.

This is comparable to what setFont does today for the bitmaps, just handing off more responsibility to a helper object.

I think it would make the core GFX code smaller and more organized by taking text APIs out of the main class. It would also offload text engine extension requests and make GFX easier to support. All those third party text renderers can just be have an Adafruit_GFX::TextEngine wrapper and integration should much easier than it is now. You can print text to a GFX object and not have to be aware of whether there's a u8g2 or whatever doing the rendering.

Anyway talk is cheap and I should shut up and put up my code, right? :)
Hopefully I'll find time for it again and we'll see how it goes.

[DoomHammer]

@evaherrada is this a viable PR or is there anything that requires changing?

[DoomHammer]

@DoomHammer, it will work. The approach has been used by this library for a long time. https://github.com/olikraus/U8g2_for_Adafruit_GFX

Incidentally that library supports extended fonts.

I've tried that library @Bodmer together with RGB Matrix Panel but it doesn't seem to display anything.

The (simplified) code looks like this:

  matrix.begin();

  u8g2_for_adafruit_gfx.begin(matrix);
  u8g2_for_adafruit_gfx.setCursor(10, 10);
  u8g2_for_adafruit_gfx.setFont(u8g2_font_helvR14_tf);
  u8g2_for_adafruit_gfx.print(F("Oo"));

[DoomHammer]

Nevermind, I needed to set the foreground color for this to work...

[DoomHammer]

@PaintYourDragon @ladyada @tyeth any input on that?

Is it decent? Is it total crap? Is it useful?

[PaintYourDragon]

I’m not entirely opposed to this, but I don’t make The Big Decisions. I think the idea is to steer folks toward CircuitPython, which has much better text and font support all around.

My concern with the implementation is that Good and Proper Failsafe UTF-8 Support would require some kind of sparse array implementation for the glyph list. As written, this only works because the max glyph used is in the 300s and the resulting array fits in flash…but someone could easily see “UTF-8 support” and assume “oh cool I can finally use Linear B Syllable B008 A” and that’s really not the case because the glyph index is way out there in space. Along with each feature comes a certain support footprint.

If you need this in your own projects, I’d suggest just keeping a fork of GFX and syncing up with any updates.

[DoomHammer]

The way I started with this PR is because I tried to achieve the same with CircuitPython but for whatever reason didn't get enough memory. I could either have UTF-8 font or WiFi but not both.

It's possible I've done something wrong, but it made me switch to C/C++ and I finished the task at hand so I'm happy about it.

As for proper UTF-8 support: you're definitely right.

[TadeuszKarpinski]

I tested it and it works. Good job

[tyeth]

Sorry for my lack of comment/input.

I have no real say here either, both as a less seasoned embedded developer and not a direct employee. I am currently contracting for Adafruit and tasked with releasing already merged work every week. There is a plan to tackle older PRs where an obvious non-breaking change/improvement exists, Ladyada will be suggesting which each week to me, but probably this won't be included as I get the impression it would affect older boards with less memory along with not providing universal/full UTF8 support.

There appears to also be a very good alternative solution linked by bodmer https://github.com/takkaO/OpenFontRender and Bodmer's fork with extra fixes.

Personally I used Bodmer's stuff in the past gratefully, and in Circuitpython compiled my own custom fonts to get any glpyhs I needed (degree symbol and micro for microgram)... Where there's a will, there's a way!

However don't lose faith that this is useful, it is as it solved the original requirement, and got the work out in the public domain for others to discover more easily. It helped at least a couple of people already, along with aiding/triggering the discussion about UTF-8 and signposting alternative solutions.

So thanks basically.