Perform font fallback #6926

nulano · 2023-02-02T17:00:55Z

Add a new type, ImageFont.FreeTypeFontFamily(font1, font2, ..., layout_engine=layout_engine), that can be used with ImageDraw.text*(...) functions performing font fallback. Font fallback is done per cluster with Raqm layout (similar to Chromium) and per codepoint with basic layout.

This PR is far from complete, several TODOs:

Font families have only a minimal API so far, e.g. retrieving metrics or setting font variations should be supported
Maybe add a wrapper similar to ImageFont.truetype(...), perhaps ImageFont.truetype_family(...)?
Lots of tests
Documentation

I would like to get some feedback, both on the API and the implementation, before working on the TODOs above.
A dev build for Windows is available from the artifact here: https://github.com/nulano/Pillow/actions/runs/8583137967

A few examples (click to expand):

All examples use this helper block:

from PIL import Image, ImageDraw, ImageFont

im = Image.new("RGBA", (500, 200), "white")
draw = ImageDraw.Draw(im)
def line(y, string, font, name, **kwargs):
  draw.text((10, y), name, fill="black", font=font, **kwargs)
  draw.text((300, y), string, fill="black", font=font, **kwargs)

example()

im.show()

Combining Latin, symbols, and an emoji:

def example():
  s = "smile ⊗ 😀"

  times = ImageFont.truetype("times.ttf", 24)
  segoe_ui_emoji = ImageFont.truetype("seguiemj.ttf", 24)
  segoe_ui_symbol = ImageFont.truetype("seguisym.ttf", 24)
  family = ImageFont.FreeTypeFontFamily(times, segoe_ui_emoji, segoe_ui_symbol)

  line(30, s, times, "Times New Roman", anchor="ls", embedded_color=True)
  line(80, s, segoe_ui_emoji, "Segoe UI Emoji", anchor="ls", embedded_color=True)
  line(130, s, segoe_ui_symbol, "Segoe UI Symbol", anchor="ls", embedded_color=True)
  line(180, s, family, "Font Family", anchor="ls", embedded_color=True)

Combining Arabic, Greek, Latin, and a symbol:

def example():
  s = "ية↦α,abc"

  scriptin = ImageFont.truetype(r"C:\Users\Nulano\AppData\Local\Microsoft\Windows\Fonts\SCRIPTIN.ttf", 24)
  segoe_ui = ImageFont.truetype("segoeui.ttf", 24)
  segoe_ui_symbol = ImageFont.truetype("seguisym.ttf", 24)
  family = ImageFont.FreeTypeFontFamily(scriptin, segoe_ui, segoe_ui_symbol)

  line(30, s, scriptin, "Scriptina", direction="ltr", anchor="ls")
  line(80, s, segoe_ui, "Segoe UI", direction="ltr", anchor="ls")
  line(130, s, segoe_ui_symbol, "Segoe UI Symbol", direction="ltr", anchor="ls")
  line(180, s, family, "Font Family", direction="ltr", anchor="ls")

Combining characters are treated as part of a single cluster (with Raqm layout):

def example():
  import unicodedata

  s = " ̌,ῶ,ω̃,ώ,ώ, ́,á,č,č"
  for c in s:
    print(unicodedata.name(c))

  le = ImageFont.Layout.RAQM  # or ImageFont.Layout.BASIC
  scriptin = ImageFont.truetype(r"C:\Users\Nulano\AppData\Local\Microsoft\Windows\Fonts\SCRIPTIN.ttf", 24, layout_engine=le)
  dubai = ImageFont.truetype(r"DUBAI-REGULAR.TTF", 24, layout_engine=le)
  gentium = ImageFont.truetype(r"C:\Users\Nulano\AppData\Local\Microsoft\Windows\Fonts\GentiumPlus-Regular.ttf", 24, layout_engine=le)
  family = ImageFont.FreeTypeFontFamily(scriptin, dubai, gentium, layout_engine=le)

  line(30, s, scriptin, "Scriptina", anchor="ls")
  line(80, s, dubai, "Dubai", anchor="ls")
  line(130, s, gentium, "GentiumPlus", anchor="ls")
  line(180, s, family, "Font Family", anchor="ls")

Raqm layout:

Basic layout:

The string s contains:

SPACE
COMBINING CARON
COMMA
GREEK SMALL LETTER OMEGA WITH PERISPOMENI
COMMA
GREEK SMALL LETTER OMEGA
COMBINING TILDE
COMMA
GREEK SMALL LETTER OMEGA WITH TONOS
COMMA
GREEK SMALL LETTER OMEGA
COMBINING ACUTE ACCENT
COMMA
SPACE
COMBINING ACUTE ACCENT
COMMA
LATIN SMALL LETTER A
COMBINING ACUTE ACCENT
COMMA
LATIN SMALL LETTER C WITH CARON
COMMA
LATIN SMALL LETTER C
COMBINING CARON

Yay295 · 2023-02-02T18:34:03Z

src/_imagingft.c

            switch (anchor[1]) {
                case 'a':  // ascender
-                    y_anchor = PIXEL(self->face->size->metrics.ascender);
+                    y_anchor = PIXEL(family->faces[0]->size->metrics.ascender);
                    break;
                case 't':  // top
                    y_anchor = y_max;
                    break;
                case 'm':  // middle (ascender + descender) / 2
                    y_anchor = PIXEL(
-                        (self->face->size->metrics.ascender +
-                         self->face->size->metrics.descender) /
+                        (family->faces[0]->size->metrics.ascender +
+                         family->faces[0]->size->metrics.descender) /
                        2);
                    break;
                case 's':  // horizontal baseline


I think here you would need to get the ascenders and descenders of every font used on this line, not just the first one, and use the greatest value.

Perhaps. I don't think looking just at the fonts used in a given line would be good - that could create an inconsistent layout for the text_multiline functions.

This should use either the first font's metrics or the greatest value across all fonts in a family object, and it should be consistent with family.getmetrics() (not yet implemented).

I've just tested what MS Edge (i.e. Chromium) does - it uses the metrics of all fonts up to the last one that is in use for a given line, even if it causes uneven line spacing.

For example, with three fonts, a paragraph that only uses the second font will have line spacing computed from the first two fonts. If one word in the paragraph then uses the third font, that line will include the third font's line spacing, but the other lines are unaffected.

This is incompatible with how Pillow currently calculates line spacing for multiline text (see #6469 (comment)), so I think I'll just use the maximum over all fonts (i.e. assume that if a user creates a FontFamily object they actually want to use all of the fonts), and leave fixing it for #1646 (+document the limitation).

radarhere · 2023-02-18T07:21:21Z

src/_imagingft.c

            }
-        }
+            /* prefer first font's missing glyph if no font support this codepoint */


Suggested change

/* prefer first font's missing glyph if no font support this codepoint */

/* prefer first font's missing glyph if no font supports this codepoint */

radarhere · 2023-02-18T07:22:35Z

src/_imagingft.c

@@ -209,6 +222,177 @@ getfont(PyObject *self_, PyObject *args, PyObject *kw) {
    return (PyObject *)self;
 }

+static PyObject *
+getfamily(PyObject *self_, PyObject *args, PyObject *kw) {
+    /* create a font family object from a list of file names and a sizes (in pixels) */


Suggested change

/* create a font family object from a list of file names and a sizes (in pixels) */

/* create a font family object from a list of file names and sizes (in pixels) */

src/_imagingft.c

radarhere · 2023-06-07T20:48:37Z

src/_imagingft.c

+                (*glyph_info)[i].x_offset = 0;
+                (*glyph_info)[i].y_offset = 0;
+
+                /* This has been broken and had no effect for many years now...


Could you expand on this?

#5191 (comment)

nissansz · 2023-07-16T12:22:18Z

How to download the module or the build with below module?

AttributeError: module 'PIL.ImageFont' has no attribute 'FreeTypeFontFamily'

nulano · 2023-07-16T17:54:00Z

You can use a source install from the branch https://github.com/nulano/Pillow/archive/refs/heads/font-fallback.zip using the installation instructions: https://pillow.readthedocs.io/en/stable/installation.html#building-from-source

The previous Windows dev build has expired so I've re-run the CI for this branch here: https://github.com/nulano/Pillow/actions/runs/5568846705
You can download the dist-font-fallback-build.zip artifact from there and install the relevant wheel for your version of Python.
(I've had to mark the test step as ignored because some crash test files seem to have been moved since.)

nissansz · 2023-07-16T22:02:50Z

Thank you.
I tried. It works now.
Will it be updated to new pillow version 10.0 too?

nissansz · 2023-09-30T01:59:36Z

If I want to use below method to draw text for each char. is it slower than fontfamily?

Iterate all chacters in a string,
judge each character to see whether it is in the char list of a desire font,
if not in the list, specify the backup font to draw the character, then continue

nissansz · 2023-09-30T02:23:20Z

If a character is not in char list of a font, it can be displayed by a backup font.

But some fonts are strange, the backupfont did not work too.

mic-user · 2024-02-01T13:46:20Z

Hi. Is there any progress?

aclark4life · 2024-02-01T17:27:10Z

Hi. Is there any progress?

@Mitchell-kw-Lee Looks like this is still a WIP, are you able to test the branch and report results? That may help …

nulano · 2024-02-01T17:56:35Z

This needs a pretty large rebase before it can cleanly merge with the main branch.

Currently, I see #6926 (comment) as the biggest unresolved issue with this PR. I think I have an idea that could work, but I've not had time to work on this.

However, if you are able to test this PR as is (even though it is made for an older version of Pillow) and report whether it works / causes issues, it could perhaps be helpful.

mic-user · 2024-02-02T12:29:55Z

@nulano @aclark4life Errr, actually have no idea that the test process. Zero knowledge of git cowork test and manual installation stuff. And there is running environment that may impact own work.
Well...seems some most of part works as the post at #4808 (comment).
So, my suggestion is that, if possible, release current code base to public,
and see and wait the bug report(include from me) that may helpful variety of sample code by report can check widely.
Holding and being abandoned the prior effort/code are so shame which have potential..... sorry, just giving an idea is all i can do for now.

nulano · 2024-02-02T17:05:40Z

As I wrote above, this branch has a non-trivial merge conflict I haven't had time to resolve, even though I would like to get to it at some point.

If you'd like (keep in mind this is based on Pillow from a year ago), I can re-run the CI to generate new Windows wheels for easier installation. If you are on Linux/macOS, you'll need to figure out source installation (hint: https://pillow.readthedocs.io/en/stable/installation.html#building-from-source)

mic-user · 2024-02-02T23:35:41Z

@nulano Ah? You are going to give me aa package that can be installed by a command 'python -m pip install pillow==TESTVERSION --upgrade --user' or so? Then lets do it. I'll test it by applying piilow code in my code. And I can do feedback.(I'm running on Windows 11 64bit BUT Python 3.11.7- 32BIT, beware please.)
Before we proceed, I need the syntax example to do that set a custom single font path for main usage and I will have another font for fallbacking(It should be #4808 (comment), right?). And those are Korean(part of CJK) and an open source symbol font. Send me the installation code

nulano · 2024-02-03T08:58:51Z

I've rerun the CI: https://github.com/nulano/Pillow/actions/runs/7765749734
You can download dist-font-fallback-build from the list of artifacts at the bottom, unzip it, and install with python -m pip install Pillow-9.5.0.dev0-cp311-cp311-win32.whl --user.

Example usage is at the top of this page (click on A few examples (click to expand) in the top comment) or #4808 (comment).

mic-user · 2024-02-03T09:06:23Z

@nulano okay, after fix version conflict, I have got the installation on my system now. BUT will comback early next week. Give me sometime.

mic-user · 2024-02-04T02:54:36Z

@nulano When I call the FreeTypeFontFamily I got the error <class 'AttributeError'>'FreeTypeFontFamily' object has no attribute 'getsize'. (Fortunately, there is no 'No FreeTypeFontFamily method found' stuff.
Do I need to do something prerequsition part like ...remove some library or so? What should I do for now?

radarhere · 2024-02-04T05:17:30Z

FreeTypeFontFamily doesn't have getsize() - but that's not a method that needs to be added later, it's a method that has been transitioned out of handling fonts in the latest Pillow versions. I suggest you use getbbox() instead. You can read more at https://pillow.readthedocs.io/en/stable/deprecations.html#font-size-and-offset-methods

mic-user · 2024-02-04T05:54:36Z

@radarhere I do not catch. Or you don't catch the whole situation.
The error message occured from the method FreeTypeFontFamily that something happended inside.

mic-user · 2024-02-04T05:56:28Z

The error above came from the line
_font_family = ImageFont.FreeTypeFontFamily(_font, _backup_font) <<-- HERE

mic-user · 2024-02-04T11:52:20Z

@nulano Questions.

Is there the 'Someone' already on it? or?
'Wait' mean.....how long?

aclark4life · 2024-02-04T12:46:49Z

@nulano Questions.

* Is there the 'Someone' already on it? or?

* 'Wait' mean.....how long?

@Mitchell-kw-Lee You can follow related issues and provide comments, questions and feedback just as you did here to find the answers to those questions. Pillow is released quarterly, and your fix may or may not be in the next release, based on the answers to those questions. I suspect we already know most of the involved parties from the discussion in this thread, and I don't think we can give any meaningful answer in this case to the the question of how long you'll have to wait for a new feature, developed in large part by volunteers, or underpaid developers. Sometimes we can answer! But I don't see anything definitive here yet … more than @nulano has already provided at least.

nissansz · 2024-04-05T11:18:43Z

fontfamily (font1 (include some chars)+font2(include most chars)) cannot show some characters

font1: see attachment.

font2: Arial Unicode MS.ttf or any other fonts
str = 'をも資资儲储議议歷历權权個个TextInAaBbCcDdEeFfGgHhIi family PILLOW'

ballpen.zip

    # 写入字体效果的文字
    image = Image.new('RGB', (image_width, image_height), color='white')
    draw = ImageDraw.Draw(image)

     font0 = ImageFont.truetype(font_file, size=image_height-2)
     font1 = ImageFont.truetype(r'C:/F/fonts/tianshiyanti2.0.ttf', size=image_height-2)

        font_family = ImageFont.FreeTypeFontFamily(font0, font1)
        draw.text((0, 0), text+'   family PILLOW', fill='black', font=font_family)

nulano · 2024-04-06T17:34:31Z

Your code is incomplete. What is font_file? What is C:/F/fonts/tianshiyanti2.0.ttf? What is text? Please provide a complete example. See https://stackoverflow.com/help/minimal-reproducible-example

…-missing glyph in a single cluster

… instead of last font's missing glyph

nulano · 2024-04-06T19:21:09Z

@mic-user I have rebased this PR to include Pillow 10.2.0 changes, multiline text seems to work now.
You should be able to get a built wheel from https://github.com/nulano/Pillow/actions/runs/8583137967

nissansz · 2024-04-06T22:22:59Z

@mic-user I have rebased this PR to include Pillow 10.3.0 changes, multiline text seems to work now. You should be able to get a built wheel from https://github.com/nulano/Pillow/actions/runs/8583137967

Cannot see download link for windows. Does this wheel 10.3.0 support fontfamily?

nissansz · 2024-04-06T22:29:08Z

fontfamily (font1 (include some chars)+font2(include most chars)) cannot show some characters

font_file0: ballpen.zip.

font_file1: tianshiyanti2.0.ttf.zip

str = 'をも資资儲储議议歷历權权個个TextInAaBbCcDdEeFfGgHhIi family PILLOW'

image = Image.new('RGB', (image_width=1000, image_height=25), color='white')
draw = ImageDraw.Draw(image)

font0 = ImageFont.truetype(font_file0, size=20)
font1 = ImageFont.truetype(font_file1, size=20)

font_family = ImageFont.FreeTypeFontFamily(font0, font1)
draw.text((0, 0), textstr, fill='black', font=font_family)

radarhere · 2024-04-06T22:45:03Z

Cannot see download link for windows. Does this wheel 10.3.0 support fontfamily?

If you scroll down on that page, you will see dist-windows-x86, dist-windows-ARM64 and dist-windows-AMD64.

Yes, those wheels would support FreeTypeFontFamily.

nissansz · 2024-04-06T22:55:13Z

Cannot see download link for windows. Does this wheel 10.3.0 support fontfamily?

If you scroll down on that page, you will see dist-windows-x86, dist-windows-ARM64 and dist-windows-AMD64.

Yes, those wheels would support FreeTypeFontFamily.

I can download from your above link, but I don't see such link on list. How to find above links?
Any new function for fontfamily in new version?

What is difference between [dist-windows-x86] and [dist-windows-AMD64]?

(https://github.com/nulano/Pillow/actions/runs/8583137967/artifacts/1391023653).

nulano · 2024-04-06T23:02:19Z

What is difference between [dist-windows-x86] and [dist-windows-AMD64]?

dist-windows-x86 is 32-bit x86
dist-windows-AMD64 is 64-bit x86.
dist-windows-ARM64 is 64-bit ARM.

Any new function for fontfamily in new version?

No, except multiline text seems to work now.

I can download from your above link, but I don't see such link on list. How to find above links?

Go to https://github.com/nulano/Pillow/actions/runs/8583137967 and scroll down:

nissansz · 2024-04-06T23:05:18Z

What is difference between [dist-windows-x86] and [dist-windows-AMD64]?

dist-windows-x86 is 32-bit x86 dist-windows-AMD64 is 64-bit x86. dist-windows-ARM64 is 64-bit ARM.

Any new function for fontfamily in new version?

No, except multiline text seems to work now.

I can download from your above link, but I don't see such link on list. How to find above links?

Go to https://github.com/nulano/Pillow/actions/runs/8583137967 and scroll down:

Saw it. Thank you.
How to use multiple lines for font family?

There is still below problem from yesterday.
fontfamily (font1 (include some chars)+font2(include most chars)) cannot show some characters

font_file0: ballpen.zip.

font_file1: tianshiyanti2.0.ttf.zip

str = 'をも資资儲储議议歷历權权個个TextInAaBbCcDdEeFfGgHhIi family PILLOW'

image = Image.new('RGB', (image_width=1000, image_height=25), color='white')
draw = ImageDraw.Draw(image)

font0 = ImageFont.truetype(font_file0, size=20)
font1 = ImageFont.truetype(font_file1, size=20)

font_family = ImageFont.FreeTypeFontFamily(font0, font1)
draw.text((0, 0), textstr, fill='black', font=font_family)

nulano · 2024-04-06T23:12:32Z

How to use multiple lines for font family?

Same as for regular text:

draw.text((0, 0), "line1\nline2", font=family, ...)

There is still below problem from yesterday.

I'll take a look at it when I have the time.

nissansz · 2024-04-06T23:14:37Z

How to use multiple lines for font family?

Same as for regular text:
draw.text((0, 0), "line1\nline2", font=family, ...)
There is still below problem from yesterday.

I'll take a look at it when I have the time.

I wonder whether there is any function to import html to pillow canvas, like rendering a web page.
Write html tables, etc.

radarhere · 2024-04-07T01:09:48Z

I wonder whether there is any function to import html to pillow canvas, like rendering a web page.
Write html tables, etc.

This is off-topic for this discussion of multiple fonts, but no, Pillow doesn't have any functionality for rendering HTML to an image.

PaulOlteanu · 2024-04-18T20:41:47Z

Would it be possible to get a getmask method on FreeTypeFontFamily so that it could be used with TransposedFont?

nulano · 2024-04-21T18:32:40Z

Would it be possible to get a getmask method on FreeTypeFontFamily so that it could be used with TransposedFont?

Sure, I didn't bother for a WIP PR when I wasn't sure it would be useful, but it's quite simple (literally just copy-pasted from FreeTypeFont 😄).

You should be able to download wheels from this workflow in a few hours: https://github.com/nulano/Pillow/actions/runs/8774721560

nulano mentioned this pull request Feb 2, 2023

Backup font for missing characters when drawing text #4808

Open

Yay295 reviewed Feb 2, 2023

View reviewed changes

nulano force-pushed the font-fallback branch from 8c4c0d6 to 2b0b5b3 Compare February 8, 2023 02:47

radarhere reviewed Feb 18, 2023

View reviewed changes

rahmcoff mentioned this pull request Feb 19, 2023

Support Emoji in /profile PADD Text jp00p/AGIMUS#386

Open

radarhere mentioned this pull request May 21, 2023

Corrected error code #7177

Merged

radarhere reviewed Jun 7, 2023

View reviewed changes

src/_imagingft.c Outdated Show resolved Hide resolved

radarhere reviewed Jun 7, 2023

View reviewed changes

radarhere added the Needs Rebase label Jun 13, 2023

This was referenced Jul 17, 2023

FreeType error when drawing text #7290

Closed

Is it possible to generate or simulate italic text for any font other than an original italic font? #7292

Closed

radarhere mentioned this pull request Sep 30, 2023

Fontfamily cannot show some characters as browser effect #7429

Closed

Minnozz mentioned this pull request Feb 27, 2024

Preview Card doesn't display non-latin characters properly bookwyrm-social/bookwyrm#1828

Closed

nulano mentioned this pull request Apr 5, 2024

fontfamily (font1 (include some chars)+font2(include most chars)) cannot show some characters #7946

Closed

nulano added 8 commits April 6, 2024 19:51

(WIP) support font-family style fallback with basic text layout

06f4afb

merge single font and font family code paths

d29c94f

font fallback for Raqm layout

cf0d2d6

C89 / Lint / Raqm<0.9.0 fixes

79bf466

fix Raqm fallback algorithm when a missing glyph is followed by a non…

2298afc

…-missing glyph in a single cluster

fix Raqm fallback for first codepoint, use first font's missing glyph…

cd3d3b4

… instead of last font's missing glyph

fix face reference for missing glyphs with Raqm layout

b46432c

consider all fonts when computing font family metrics

0af68e8

nulano force-pushed the font-fallback branch from 2b0b5b3 to 0af68e8 Compare April 6, 2024 18:40

nulano added Font Rendering and removed Needs Rebase labels Apr 6, 2024

FreeTypeFontFamily: add getmask() method

5a83821

	/* prefer first font's missing glyph if no font support this codepoint */
	/* prefer first font's missing glyph if no font supports this codepoint */

	/* create a font family object from a list of file names and a sizes (in pixels) */
	/* create a font family object from a list of file names and sizes (in pixels) */

Perform font fallback #6926

Are you sure you want to change the base?

Perform font fallback #6926

Conversation

nulano commented Feb 2, 2023 • edited

Yay295 Feb 2, 2023

Choose a reason for hiding this comment

nulano Feb 2, 2023

Choose a reason for hiding this comment

nulano Feb 8, 2023

Choose a reason for hiding this comment

radarhere Feb 18, 2023

Choose a reason for hiding this comment

radarhere Feb 18, 2023

Choose a reason for hiding this comment

radarhere Jun 7, 2023

Choose a reason for hiding this comment

nulano Jun 13, 2023

Choose a reason for hiding this comment

nissansz commented Jul 16, 2023

nulano commented Jul 16, 2023

nissansz commented Jul 16, 2023 • edited

nissansz commented Sep 30, 2023

nissansz commented Sep 30, 2023 • edited

mic-user commented Feb 1, 2024

aclark4life commented Feb 1, 2024 • edited

nulano commented Feb 1, 2024

mic-user commented Feb 2, 2024 • edited

nulano commented Feb 2, 2024

mic-user commented Feb 2, 2024 • edited

nulano commented Feb 3, 2024

mic-user commented Feb 3, 2024 • edited

mic-user commented Feb 4, 2024

radarhere commented Feb 4, 2024

mic-user commented Feb 4, 2024

mic-user commented Feb 4, 2024

mic-user commented Feb 4, 2024

aclark4life commented Feb 4, 2024

nissansz commented Apr 5, 2024

nulano commented Apr 6, 2024

nulano commented Apr 6, 2024 • edited

nissansz commented Apr 6, 2024

nissansz commented Apr 6, 2024 • edited by radarhere

radarhere commented Apr 6, 2024

nissansz commented Apr 6, 2024 • edited

nulano commented Apr 6, 2024

nissansz commented Apr 6, 2024

nulano commented Apr 6, 2024

nissansz commented Apr 6, 2024

radarhere commented Apr 7, 2024

PaulOlteanu commented Apr 18, 2024

nulano commented Apr 21, 2024

nulano commented Feb 2, 2023 •

edited

nissansz commented Jul 16, 2023 •

edited

nissansz commented Sep 30, 2023 •

edited

aclark4life commented Feb 1, 2024 •

edited

mic-user commented Feb 2, 2024 •

edited

mic-user commented Feb 2, 2024 •

edited

mic-user commented Feb 3, 2024 •

edited

nulano commented Apr 6, 2024 •

edited

nissansz commented Apr 6, 2024 •

edited by radarhere

nissansz commented Apr 6, 2024 •

edited