Making fontmake faster

We want to drastically speed up fontmake: https://github.com/googlei18n/fontmake/issues/367
Other than switching booleanOperations to something vector-based and fast (https://github.com/typemytype/booleanOperations/issues/40), the other big offenders are basically 1. MutatorMath / fontMath, and general defcon / read/write UFO.  For example, when James replaces defcon use with tiny objects, he saw about 20% speedup:
https://github.com/googlei18n/ufo2ft/commit/2e65007df653515762ec95888f9c29b5137dec28
Cosimo has also reports that using lxml module instead of xml can significantly speed up UFO reading.
Since https://github.com/unified-font-object/ufoLib is essentially a reference implementation, I like to propose keeping it that way, with validators and converters, but add a trimmed-down version in fontTools.ufoLib that is optimized for speed.  I suggest the API for this be strictly a subset of the upstream ufoLib, such that users can use one or the other based on their need.
I also like to add fs module support to it, as we need that for build-system experimentations.  Which brings us to the topic of the UFO4 branch: https://github.com/unified-font-object/ufoLib/tree/ufo4  In general, I suggest we avoid these big revolutionary major version bumps that take forever to roll out and cause dependecy hell and instead work on an evolutionary UFO that changes slowly over time in master branch.
Please discuss.
@typesupply @typemytype @anthrotype @justvanrossum @LettError @jenskutilek @madig @brawer

Just a small clarification: lxml is faster at writing rather than reading than ElementTree, because the writer is written in C (libxml2)

Would it make more sense that fontmake has a trimmed down implementation (like your tiny objects example), than putting it into fontTools? My strong concern here is that by putting it into fontTools, that the ufoLib implementation becomes sidelined. It's important for interoperability to have design tools use the same, more strictly checked, ufoLib. I assume that this isn't as much of a concern with fontmake (and, if it knows that things are getting strictly checked in design tools, it'll help it make correct assumptions about data), so it seems better put there than fontTools.

Would it make more sense that fontmake has a trimmed down implementation (like your tiny objects example), than putting it into fontTools?

Well, then all kinds of scripts would have to import fontmake.  We don't want fontmake to be a library, and we know we want to rewrite it at some point.  This is NOT something only for fontmake.  Any kind of batch processing can use a faster ufoLib / Defcon, which are designed more towards the font-editor usecase.

My strong concern here is that by putting it into fontTools, that the ufoLib implementation becomes sidelined. It's important for interoperability to have design tools use the same, more strictly checked, ufoLib. I assume that this isn't as much of a concern with fontmake (and, if it knows that things are getting strictly checked in design tools, it'll help it make correct assumptions about data), so it seems better put there than fontTools.

That would essentially make this yet another module, and I don't think @anthrotype is rushing to volunteer maintaining that :).
I also think fontTools really should be able to open UFOs directly.  See https://github.com/fonttools/fonttools/issues/814  When we do that, we probably want to rewrite ufo2ft, properly, and put it in fontTools as well.  Plus going from binaries to UFO.  At that point, UFO can become yet another input/output format for ttx, etc.

why keep both? if the only difference between a slimmed down version and ufoLib are converters (between different ufo versions) and validators, then I would propose to make ufoLib more modular so users can add an request whatever they prefer: converting, validation, speed, ...
All the speed improvements you are proposing are also welcome in ufoLib (fe lxml if available) and everyone would then benefit from your time and resources instead of only fontmake...
And this has already been researched see https://github.com/unified-font-object/ufoLib/pull/28 and https://github.com/unified-font-object/ufoLib/issues/27

@behdad Thanks for bringing this up. This is a very important discussion and I'm glad that it has started. Here are my (slightly disjointed) thoughts...

Since https://github.com/unified-font-object/ufoLib is essentially a reference implementation, I like to propose keeping it that way, with validators and converters, but add a trimmed-down version in fontTools.ufoLib that is optimized for speed. I suggest the API for this be strictly a subset of the upstream ufoLib, such that users can use one or the other based on their need.

From my perspective, I'd be very happy if these optimizations went into ufoLib. I'm really wary of dividing the finite resources we have as a type tool developer community on duplicate code bases. We just have so much code to maintain as is that I'd hate to see us split ourselves even more just for some optimizations. If fontTools doesn't want to have a dependency on ufoLib, I think I'd be okay with ufoLib moving under the fontTools umbrella. (This is something that I just though of. I'd need to give it a bit more thought.)
I've been thinking about removing the vast majority of the validation code from ufoLib. I wrote that stuff in parallel with the UFO 3 spec as a check and balance on my thoughts in each. The validator is VERY zealous about balking at even the tiniest thing. That was fine for my theoretical work, but it practice it is having some very unintended consequences. I'm starting to think that it should only do basic sanity checking so that it's not propagating bogus data into callers. That will remove a huge chunk of code and possibly speed things up.

I also like to add fs module support to it, as we need that for build-system experimentations. Which brings us to the topic of the UFO4 branch: https://github.com/unified-font-object/ufoLib/tree/ufo4 In general, I suggest we avoid these big revolutionary major version bumps that take forever to roll out and cause dependecy hell and instead work on an evolutionary UFO that changes slowly over time in master branch.

I agree. As far as UFO spec changes go, yeah, that's been the post UFO 3 plan. UFO 4 isn't a huge change. The only reason that it's called "4" instead of "3.1" is that the formatVersion has been defined as an int since UFO 1. We tried to write UFO 3 so that it can grow much more organically. The hard part is GLIF since it has defined element names. Introducing new elements leads to nasty backwards compatibility issues.
As far as that UFO 4 branch goes... My goal was to get that written and merge it into master quickly. I never meant for it to linger for so long. I felt like it was done, but it got stuck in the approval process and I never had time to keep working towards getting it merged. I was really happy with it and seriously considered merging it into master without approval, but I thought I'd get in trouble for doing that. 😄

Cosimo has also reports that using lxml module instead of xml can significantly speed up UFO reading.

I looked at LXML in the past for reading, but didn't see any dramatic speed differences (as @anthrotype notes). I never looked at it for writing because for my use the vast majority of UFO writes are atomic instead of everything at once. If you want ufoLib to use LXML if available, I'm totally fine with that.

For example, when James replaces defcon use with tiny objects, he saw about 20% speedup:

Yeah. defcon creates a deep tree of objects. I made it more shallow for quicker loading in the last major revision, but as soon as the data is touched the object tree is expanded. If the goal is to create UFOs quickly without much manipulation before save, defcon is probably not the best fit. FWIW, fontMath has a tiny object structure and it's a huge speed improvement. It's even shallower than what you have in James' sketch. Contour objects are just tuples from what I remember. It may be a good thing to look at when working out a set of hyper fast objects.

This brings up an interesting issue: Point Pens. These are deeply linked to ufoLib and vice-versa. As such, the base implementation of AbstractPointPen and the other basic bits and bobs would need to go to FontTools. I'm fine with that as well.

I also feel that maintaining two implementations of ufoLib is quite wasteful. +1 on integrations with fontTools, +1 on making it more versatile and fast, +1 in factoring out ufo validation, +1 on fs support, etc etc.
Probably much fontPens should then also migrate to fonttools.

Ok, thank you all.  How about:

Let's try to get the fs work Tal did merged: https://github.com/unified-font-object/ufoLib/pull/26
Remove any low-hanging old cruft,
Move it as is to fontTools.ufoLib
Start adding experimental tiny objects and port fontmake to it and rinse and repeat on speeding up.

For pens, I think we can move them to fontTools one by one as we need to.  That's what we have been doing anyway.

ok, and then also let's also be brave and break the taboo that says fonttools can't have dependencies :)
I want to require pyfilesystem2.

ok, and then let's also be brave and break the taboo that says fonttools can't have dependencies :)
I want to require pyfilesystem2.

As long as it's pip-installable, I don't mind.  Also, since we are in Python, lazy-enough dependencies are fine.
We still need to figure-out about circular dependencies for certain functionalities, like a module in fonttools importing glyphsLib.

it certainly is pip-installable (https://pypi.python.org/pypi/fs)

since we are in Python, lazy-enough dependencies are fine

sure, though we already have a long list of optional and un-declared dependencies in various fonttools modules (see README) and I'm sure it's very confusing for users and downstream maintainers to keep track of all of them, given that our setup.py says we have no dependencies at all. I think some linux distros still think we require numpy...
I think the decision to require fs module in setup.py depends on how "core" this fontTools.ufoLib will be: i.e. if we deem it extra like the pyftinspect script, or if we place it on the same standing as ttLib or varLib.
I just don't want to reinvent the wheel and have to maintain it too.

Start adding experimental tiny objects and port fontmake to it and rinse and repeat on speeding up.

So a separate implementation within the reference implementation? :)

So a separate implementation within the reference implementation? :)

I wouldn't take the "reference implementation" designation on ufoLib too seriously. When I wrote that bit of documentation (and the strict validation code that is mentioned above) I was just coming off of writing a bunch of very official code for the WOFF specification. I was still speaking RFC 2119 (W3C dialect) and that leaked into the UFO specification. It sounds much more Very Official than it actually is.

Start adding experimental tiny objects and port fontmake to it and rinse and repeat on speeding up

If you need help with this, please let me know. I've written a lot of implementations of these structures over the years and I'd be happy to pass along info about some of the things that I tried and learned.

Start adding experimental tiny objects and port fontmake to it and rinse and repeat on speeding up.
So a separate implementation within the reference implementation? :)

Oh those will be defcon replacement objects, not ufoLibs.  ufoLibs looks pretty lean, I don't think we'll be rewriting a lot of it, just cleaning up / improving.

@adrientetar's thoughts: https://gist.github.com/adrientetar/2c515f2e304bfc57c9d37c7e39b586c7

@adrientetar's thoughts: https://gist.github.com/adrientetar/2c515f2e304bfc57c9d37c7e39b586c7

FWIW, I agree with most of the points raised. Nice writeup Adrien!

I dumbly tested the first statement of Ardien:

there is no with ... except ... statement, oke it can be embedded into a `try ... with ... except...
my test shows that is either quicker with os.path.exist as pre test when none of the paths exists
my test shows that is equally fast with os.path.exists as are test when there is a mix of existing and non existing paths
my test shows that is equally fast with os.path.exists as are test when all paths exists
over test: the old implementation is faster
weirdly its 4-5 times faster in python2???

please correct me if Im wrong https://gist.github.com/typemytype/eb69d0a4b6acd92241af9d97e0bb3baa
you can test this side by side in the new DrawBot with py3 support!!!

The difference between py2 and py3 becomes less if you open the files in binary mode (avoids unicode conversion). Still, py3 is about twice as slow in the "all exists" case.

Re: @adrientetar's thoughts:

it seems most overhead is in defcon, not ufoLib
then why propose to rewrite ufoLib from scratch?
on the one hand complaining about defcon's overhead
on the other hand proposing a complex COW system that will use more memory
what is the main goal of the operation? To make fontmake faster? To cater better to Adrian's taste in object design for his font editor? Those two things should not interfere with each other. I'm ok with fonttools making font building workflows more efficient, I'm not ok with fonttools catering to specific needs of client software. That's why defcon is separate and things like it should be separate [from fonttools].

So +1 to:

a super minimal lean object model so fontmake-like things don't need to depend on defcon

And -1 to:

a new object model designed for font editors (even if it would be much better than defcon)

The idea is to have something like TTFont but for UFO objects. The one thing that really needs support is custom classes, such that an app backend can be written of top of the fontTools objects. (And as I say it doesn't have to be dynamic like in defcon, so it's zero-cost.)

on the other hand proposing a complex COW system that will use more memory

I'm not sure what you mean by that. Hash tables and trees have more overhead but they make certain operations faster. Once structural optimizations are exhausted speedups are often a matter of speed/memory tradeoff.
Anyway it's just an idea, I haven't explored whether it makes sense to use here. It's always good to think outside of well-worn paths.

then why propose to rewrite ufoLib from scratch?

I think it can be made a lot more compact, that plus the fact that Cosimo wants to use a different XML lib makes it basically a rewrite. Anyway it's playing semantics, the end results matters not the means (it's up to the one writing it).

my test shows that is equally fast with os.path.exists as are test when all paths exists

That's surprising since pre-check amounts to extra syscalls.
Anyway, note my correctness argument. I think it's more important than speed.

Added a section about "Single file UFO".

The idea is to have something like TTFont but for UFO objects.

That is not a bad idea per se.

The one thing that really needs support is custom classes, such that an app backend can be written of top of the fontTools objects. (And as I say it doesn't have to be dynamic like in defcon, so it's zero-cost.)

This makes no sense to me at all. How can fonttools even pretend to know what font editors may need in terms of data structures?
Anyway. Perhaps it's more useful to try to define a standard minimal object API. "Objects conforming to x, y, and z will interoperate nicely with tools a, b and c".

Thanks Adrien!
I agree with many of the points you raised, especially the need for a "fast path" between reader and object initialization and avoid unnecessary copies after the initial read.
The goal is to make building or batch processing fonts faster, without unnecessary features that are more meant for interactive editors.
What I would really love to see is a UFO reader/writer that's as fast as if it were written in pure C -- and we can achieve that speed with Cython, without giving up on all the niceties that Python offers. Even simply cythonizing a pure python code as is into an equivalent C code (i.e. one that calls the very same C API that the Python interpreter uses) gives you from 20 to 50 % for free, by just removing the interpretater step. Then as you add static typing to variables the gains can boost to 10 times and more depending on the task at hand.
Lxml is also written in Cython, thus provides a Cython/C interface which could be called from other Cython modules, which makes building parsers/writers built on top of it even faster than it already is from pure Python.
I would like this fast UFO reader/writer to be able to deserialize/serialize to/from primitive built-in data types like dict, lists and tuples. These have the advantage of being compatible with other serialization libraries like json, yaml, pickle, etc. Then from this unstructured data I would like to be able to build classes each representing specific data structures with certain combinations of fields and valid values.
Now.. there is a very cool library which I'd love to use in one of my projects but I'm wary of proposing it because of the usual "dependency scare". It's called attrs, and it allows to declaratively define the attributes of a class and to remove much of the boilerplate involved in defining classes. It's hugely popular, well maintained and used my lots of projects -- so much so that there's now a [PEP](https://www.python.org/dev/peps/pep-0557/ modelled after it, but that will only be available for future python versions, while attrs work with any python 2 or 3.
I haven't given as much thought as Adrien yet, but you can see that what I have in mind goes more in the direction of a from-scratch rewrite rather than an incremental optimization.
I'm not totally convinced about the idea of immutable data structures, but it may be just me not being much familiar with functional programming techniques, so I'd willing to learn more on that.

define a standard minimal object API. "Objects conforming to x, y, and z will interoperate nicely with tools a, b and c".

that is basically fontParts

that is basically fontParts

Yes, but no. fontParts is a large API that is for convenient scripting in (perhaps) a font editor. I'm rather thinking of something small, with only simple operations and attributes.

yeah, I meant a subset of fontParts tailored for command-line batch processing tools.

Yes, but really a very small, well documented subset. I think that can be extremely useful.

How can fonttools even pretend to know what font editors may need in terms of data structures?

The basic data structures are always the same (for instance lib is a dict), font editors "only" need extra methods, attributes and functionality (e.g. notifications) on top of those.

How can fonttools even pretend to know what font editors may need in terms of data structures?

The basic data structures are always the same (for instance lib is a dict), font editors "only" need extra methods, attributes and functionality (e.g. notifications) on top of those.

For those trivial things you don't even need a base implementation. But how do you store your outline data internally? That highly depends on how you use that data. A base implementation doesn't help there at all.
I think we should take about interfaces and protocols, and not try to provide a base implementation that wants to cater all. Fonttools is a library, not a framework.

Yes, but really a very small, well documented subset.

ufoLib requires only a very small API in the objects it writes to and reads from. It's pretty basic stuff and it may even be documented in the doc strings. It also doesn't require any objects below the glyph level, so you can make a very fast, low-overhead glyph object by just implementing the PointPen API directly in an object that represents a glyph. Info, kerning, groups, etc. require even a smaller API so those are even simpler. Further, if transport to/from fontMath is necessary, that should also just work or be pretty close to just working and it will be fast. The requirements for read/write and basic data manipulation are quite small.
defcon implements a huge number of framework features (observer pattern, smart caching, deferred loading, serialization, etc.) because it's intended to be the basis of interactive applications. It's primary goal is not fast full-file read/write so trying. (I'm not sure why defcon is even being discussed here, honestly. It's quite a different thing from ufoLib.)

I'm not sure why defcon is even being discussed here, honestly

it's because that's what we happen to use in fontmake (basically as a robofab replacement) because it was already there and does the job.
ok so you're saying we shouldn't rely on it for non-interactive use but rather make our own stripped-down UFO objects library, which is what we would like fontTools.ufoLib to be, in addition to the raw reading/writing stuff.
I'm ok to just fold ufoLib in fontTools as is at least at the beginning, and then later if I find the time I'll experiment with lxml and cython.
Then, we will also add a set of shallow objects to store the data, which will have a minimal subset of the behavior provided by fontParts, just enough to read the UFO data and build a TTFont from it, or to get data from Glyphs.app source file and populate objects that can be saved as UFOs.

I thought that the plan @behdad had made sense: bring ufoLib into fontTools, removing a lot of the validation when doing so, and then build a tiny objects defcon-like thing for fontTools.
It seems that the major gain in speed for fontmake is going to be in not using defcon, not in super optimizing ufoLib, which, as @behdad said, is pretty lean as is. lxml may make it faster, but it seems to make more sense —and would be faster— to first focus on tinyObjectsDefconLikeThing and see what speed that gains fontmake, and then look at lxml and any further ufoLib optimization. Then if more speed is needed, optimize tinyObjects more. In other words, stage it out like @behdad suggested.
I would imagine that fontTools tinyObjectsThing would have a defcon-like api, so one could swap in/out as needed (again, as @behdad said above), but would strip away all the things that slow defcon down use on the command line (as @adrientetar and @justvanrossum detailed and discussed above). Such a thing would be useful to everyone.
I am wary of a complete rewrite of ufoLib for a couple of reasons: the first being that it's been well tested in production and is reliable. The second is that if it's re-written just to use lxml, that adds another dependency. If going to lxml, I'd rather see it be the first choice, with a fallback to cElementTree (standard library).

sounds good to me
note that lxml now ships pre-compiled binaries for all python versions and all platforms (linux, mac and windows), so it would just be sufficient to specify it in setup.py as installation requirement and pip would magically install it as a dependency without needing any C compiler or pre-installed native library:
https://pypi.python.org/pypi/lxml/4.1.1
And lxml also provides the same ElementTree API so can be used interchangeably with the standard library module.

@adrientetar can I ask what your plan is for the current ufolib branch? It looks like you are re-writing the reader and writer there.

Yes, well it's very similar to the current one. Actually I'm working from this branch: https://github.com/adrientetar/fonttools/tree/ufoLib seems I pushed to fontTools at some point.

I have to admit that I was a bit surprised to see this, as I thought that a consensus had been reached to merge the current ufoLib to fonttools. However, I'm not opposed to this branch, but I want to be sure that we're not going to be introducing breaking changes to everyone's workflow.
With an eye to that, I am curious about: Will this support python 2.7? UFO 1 and 2? What is your plan for testing the new reader/writer to be sure it behaves like the current well-tested reader/writer? Does the api match defcon so that one or the other can be used when appropriate (I assume so, but haven't gone through to check everything)?

It'll support 2.7 if it's to be part of fontTools and UFO2 (1 isn't hard to support either iirc). Still working on ti.

It'll support 2.7 if it's to be part of fontTools and UFO2 (1 isn't hard to support either iirc). Still working on ti.
👍 1

Can you also include fs module please?  We can have a stub fs impl if needed.

I have to admit that I was a bit surprised to see this, as I thought that a consensus had been reached to merge the current ufoLib to fonttools.

If he's working on it to show what can be achieved, I don't see a problem with that.  We can take a look at both and decide how to merge them.

@behdad agree, sorry, I was just taken by surprise.

He's a maverick :D.

Pyfilesystem author here. Happy to lend a hand if you need it.

Adrien, any update? Can you report how your experience went? I like to move forward with this as we like to do some fontmake speedup hacking around Robothon, and that depends on integrating ufoLib into fonttools. Thanks.

My branch: https://github.com/adrientetar/fonttools/tree/ufoLib
can read/write ufo3 (you could also serialize to a different format, e.g. with cattrs, since the classes are data-driven). I haven't benchmarked it yet.

I ran a benchmark of opening Roboto-Regular.ufo, setting all points to zero and saving to another file (here) on Windows it takes 77s with ufoLib/defcon and 58s with fontTools so 25% faster.

I ran a benchmark of opening Roboto-Regular.ufo, setting all points to zero and saving to another file (here) on Windows it takes 77s with ufoLib/defcon and 58s with fontTools so 25% faster.

Thanks. Looks like in both cases over 50s is spent in writing out the results. We should look into optimizing that. But it also means that your branch should be, like, 10x, faster than defcon for reading only. Can you measure that?

Does yours use lxml for writing?

Can you measure that?

If I comment out the font.save(..) line I get 14s for defcon and 4.5s for fontTools.

Does yours use lxml for writing?

Yes reading and writing.

Can you measure that?
If I comment out the font.save(..) line I get 14s for defcon and 4.5s for fontTools.

Nice.

Does yours use lxml for writing?

Yes reading and writing.

Any idea why writing is so much slower than reading?

Any idea why writing is so much slower than reading?

I would say writing 100s of small files is slow (5.5s spent in io.open), we also spend 6.5s building the set of existing names every time a glyph is written, that could be alleviated by caching it (I have a TODO about it in the code).
Beyond that I don't see many easy optimizations, rewriting plistlib to use lxml would probably be another speedup of the same kind (but that'd take some dev time), otherwise I think it could be far faster to write a JSON dump of the classes in a text file rather than the UFO directory.

Ok, I convinced myself that we should fold https://github.com/fonttools/ufoLib2 as fontTools.ufoLib.
I want to start using it in ufo2ft, glyphsLib, fontmake, but I don't want to support an additional package.
Since it requires a few third-party libraries (lxml, attrs, cattrs), I'm gonna make it into a so-called "extra" feature named "ufo".
So if one wishes to install fonttools with UFO support, one needs to do pip install fonttools[ufo]. The additional dependencies are automatically installed only if the "[ufo]" specifier is added to the package name in pip's command line (or setup.py or requirements.txt of the dependent libraries or applications).
For more info on extras, see:
http://setuptools.readthedocs.io/en/latest/setuptools.html#id15
https://hynek.me/articles/conditional-python-dependencies/
By default, fonttools will stay dependency-free as the other maintainers like it. If one doesn't need to use fontTools.ufoLib, one can simply not import it and everything else will work.
Note this new ufoLib2 only supports UFO3 for now. We can add UFO2 support later. Although I must say that's not at the top of my priorities and the time I can devote to this is limited, given all my the other commitments.
If I don't hear objections from others, I'll start working on merging the repos later today.

@anthrotype I've not been following ufolib2, it's actually the first time i've seen it: can you comment on what was ported from ufolib, how it's been tested to conform to the spec and how ufolib behaves?

hey @benkiel. I opened that repo a month ago, it's basically @adrientetar "fonttools-ufolib" branch, backported to work on py27, plus some fixups. I did not publicize it much as I first wanted to make sure it worked for my immediate use case, which is reading UFO sources (quickly) to produce font binaries.
This is not meant to replace ufoLib as the reference UFO implementation, at least not in this initial form.
It's more like a minimalist implementation of a UFO reader/writer, plus some shallow classes to manipulate the data, which is intended to be as lean and fast. In general, the API is borrowed from ufoLib and defcon, so that other tools built on the latter can use it as drop-in replacement if they wish.
About the tests, I know there aren't much on the ufoLib2 repo itself (besides the ones on the plistlib module). It is indirectly tested through the ufo2ft test suite, which runs with both defcon and ufoLib2 Font objects. Of course, I'm planning to add more unit tests over time.
We can mark this new submodule as work-in-progress and subject to change, both in the README and NEWS files.

Please don't confuse my questions with resistance to this

of course, I understand :)

I'm just trying to get a handle on what is new, what's changed

sure, we need to document this.

@anthrotype Understood. I wasn't checking the general fontTools account, so I didn't see the new repo.
It would be great to see how this API differs from the ufoLib API, there was consensus about having a lightweight UFOLib like thing in FontParts that used the same API as ufoLib; it would be great if this could be that thing, and it sounds like it is/could be.
I'll spend more time looking at this when I get the chance, I've only had a quick look when I saw the fontTools PR for merging this in.
+1 for marking it as a work-in-progress in the README and NEWS files.

@typemytype thank you for taking a look.
Maybe we should explore again the possibility of integrating the current ufoLib reader/writer in fontTools, and work together on defining a standard UFO object interface or protocol like @justvanrossum has suggested, plus maybe a minimalist base implementation that fonttools' own scripts can use. Then, clients would implement these objects as they see fit and pass them to the UFO reader/writer classes.
What I'd would also like is an option to turn off validation, at least upon reading; being able to (optionally) use the faster lxml instead of the built-in elementtree.

What I would also like is an option to turn off validation, at least upon reading;

+1

and being able to (optionally) use the faster lxml instead of the built-in elementtree.

Would be nice, if the speedup is significant compared to the amount of intrusion to the code base.

here is my quick tests: https://gist.github.com/typemytype/d338ebdc84ccc0a0be534cf6667797e4
and a full object implementation :)
comparing the current ufoLib and fontTools.ufoLib
test loading into font object
__main__ 4.59287758305436
fontTools.ufoLib.objects.font 4.336412827018648
defcon.objects.font 8.892666688014288
done

Maybe we should explore again the possibility of integrating the current ufoLib reader/writer in fontTools

I'm 100% onboard with this.

work together on defining a standard UFO object interface or protocol like @justvanrossum has suggested, plus maybe a minimalist base implementation that fonttools' own scripts can use. Then, clients would implement these objects as they see fit and pass them to the UFO reader/writer classes.

Yeah. This protocol can be very simple and Pythonic. The UFO 1 and UFO 2 data protocols were built as we needed them, so there wasn't a Big Plan. When we added the new stuff from UFO 3, I tried to use standard object types (primarily dicts and lists) and make things more easily updatable as new things are added.

What I would also like is an option to turn off validation, at least upon reading;

+1
My hunch is that this will offer a big speedup. I'd also suggest that the validation should be off by default. I added the validation as a way of ensuring that receivers, such as defcon, would be free to assume that they wouldn't be getting bogus data. In the case of scripts in a controlled development environment with known data quality, this validation is not necessary.

and being able to (optionally) use the faster lxml instead of the built-in elementtree

I'm cool with this. I switched to cElementTree from our homemade tree builder because ET was there, worked, was faster, and was code that we didn't have to maintain. If LXML speeds it up without introducing complexity, go for it.

Here is a first swing at giving ufoLib the option to turn off validation: https://github.com/unified-font-object/ufoLib/pull/149

Err, now in the validation_optional branch of ufoLib

FWIW, I just tried to drop lxml in as a replacement, it isn't going to be as simple as from lxml import etree as ElementTree, but may not be too bad.

We (@benkiel and I, with feedback from others) have a branch of ufoLib in which:

lxml is used for reading and writing
validation is optional

Here is a script that can be used to compare times and dump a profile. This is what I get on my machine:
Speed reading...
ufoLib: validate=True 0.856248
ufoLib: validate=False 0.772182
fontTools: validate=None 0.701451

Speed writing...
ufoLib: validate=True 1.190951
ufoLib: validate=False 1.155090
fontTools: validate=None 0.980606

Our ufoLib modifications are still a tiny bit slower than fontTools.ufoLib, but we're handling UFO 1-3, edge cases, and cleaning up some known bad UFO data. Still, we made it run about twice as fast as the current master.
The code is still a little sloppy, but we figured it is at a good point to see if you have any suggestions or comments. If not, we'll marge this into ufoLib's master.

My understanding is that FontTools ufoLib is a replacement of both ufoLib and defcon, so I think a fair comparison should include defcon somehow (I vaguely remember that defcon speed was a significant issue).

My understanding is that FontTools ufoLib is a replacement of both ufoLib and defcon, so I think a fair comparison should include defcon somehow (I vaguely remember that defcon speed was a significant issue).

(See @anthrotype's comment.)
defcon and ufoLib are completely different things. The speed issues that have been raised in the past are almost entirely related to the overhead of defcon's internal notification system. Replacing defcon makes sense if you are looking to write lots of fonts at the fastest speed possible. A very minimal set of objects can be hyper fast and I absolutely think these should be a part of fontTools if you want to support UFO.
However, this doesn't mean that ufoLib needs to be thrown away. It's stable, well-established code that handles all versions of UFO and knows about the various real-world edge cases. The validation it can perform is extremely helpful when debugging and as a way of ensuring data reliability. The point of this work was to try to speed up the existing ufoLib by removing the validation requirement (which, as it turns out isn't much of a speed issue 😉) and switching to lxml. We were able to do this without sacrificing anything and the speedup is near 200% in my tests.
If you don't want fontTools to have a defcon replacement without having a UFO reader/writer, I am comfortable with transferring ufoLib to fontTools. I'm also cool with updating the API to be more modern/considered/less-ad-hoc or whatever you want to call it.

Alright, I just reread this long conversation to refresh my memory, and it looks like the consensus was that we can move the current ufoLib reader/writer to fontTools (as fontTools.ufoLib), and we then work on making a shallow defcon-like API to hold/manipulate the UFO data.
If we are still all happy with this plan, I think I can start making the move tomorrow.
Please let me know
cc @typesupply @benkiel @typemytype @LettError @justvanrossum @behdad etc.

Happy with that, that's why we made the UFOLib changes that we did (validation optional, lxml). I think somewhere in this are some gists that point to ways of making shallow objects (@typesupply has done this in the past). The other low hanging fruit is finishing the fs bit for single file UFOs, perhaps this first, then that?

The other low hanging fruit is finishing the fs bit for single file UFOs, perhaps this first, then that?

OK, I finally managed to do that (it wasn't very "low hanging" to be honest.. ;).
ufoLib (master branch) now supports reading/writing from UFOZ (zip) files, as well as any other filesystems supported by pyfilesystem2.
Tomorrow I'll start preparing for moving ufoLib to `fontTools.ufoLib.

@anthrotype sorry! But thanks for doing so!

@anthrotype Is there a branch or something that I can look at to see the file system changes? I'll give it a once over.

It’s in master 🙃
Thanks for taking a look

ufoLib is BSD-3-Clause whereas fontTools is MIT. Both are permissive licenses and thus are compatible with each other:
https://softwareengineering.stackexchange.com/a/278142
Shall I keep the separate ufoLib/LICENCE.txt file with its own copyright notice under Lib/fontTools/ufoLib directory?
Or shall we re-licence ufoLib under MIT, have a single LICENSE file and add @LettError and @typesupply to the FontTools copyright notice (which currently only has @justvanrossum)?
Please advise, thanks

alternatively I can copy the current ufoLib/LICENSE.txt to a file named "LICENSE.ufoLib" in the root of the fontTools repository, similarly to how we have a "LICENSE.external" for the test font files licensed unde SIL Open Font License.

We're fine with relicensing to MIT.

We're fine with relicensing to MIT.

thanks tal. shouldn't we contact all the contributors and ask if they are ok with it?
/cc @davelab6

actually, BSD and MIT are basically the same, I don't think we need to bother.

I mean, I don't think we need to bother asking everyone.

shouldn't we contact all the contributors and ask if they are ok with it?

I asked on our Slack. I can ask them to chime in here if you'd like.

No, I'm ok if you guys are ok.

Not a lawyer, but I believe the transition from 3-clause BSD to MIT removes the BSD promotion clause:
"Neither the name of the The RoboFab Developers nor the names of its contributors may be used to endorse or promote products derived from this software without specific prior written permission."
MIT allows you to promote away with anyone's name in derived projects. If you intend to use contributor names in derived software here, it may be worth having the individuals who you list provide permission since that is the clause that you are removing from the upstream license. Short of that, I don't believe that there are significant changes.

convert to MIT license upstream?

does it make any differences? the "upstream" will become this.
Anyway, yes I can do that in ufolib repo before merging the git histories.

Perhaps it is worth making the change to MIT upstream with whatever ufoLib project side permissions are felt to be necessary by the upstream copyright holders before the git commit history reflects the merge here?  Seems to be the most legit way to do it and preserves the history of the decision before the merge.   This leaves an open documentation trail should anyone raise an issue down the road (unlikely I know, but licenses are licenses...)

Should this be closed now?

I'd like to revamp my

I'd like to revamp my efforts to have that "super minimal lean [UFO] object model" (as Just called it) in fontTools.ufoLib.objects that I started last year in #1335 and didn't complete.
In the meantime the separate ufoLib2 project has matured and we have been using it in ufo2ft and fontmake, so I think the time is ripe to riconsider this.
Before I start working on the merge, I'd like to check if there any concerns/objections/feedback to this.

Could you refresh my memory about what this is/does? I looked at the links but wasn't sure what to look at specifically. I don't have any objections to whatever this is because I don't have time for discussing. I'm just curious. Thanks!

ufoLib2 is meant to be a thin representation of the Unified Font Object (UFO) version 3 data model, intended for programmatic manipulation and fast batch processing of UFOs. It resembles the defcon library, but does without notifications, the layout engine and other support classes. Where useful and possible, ufoLib2 tries to be API-compatible with defcon.

A pared down defcon for batch processing if you will, without code that is mainly useful for design applications. Think conversion of formats, reading UFOs for compilation with fontmake, etc.
I wouldn't merge it just yet, there are various things in the issue tracker that need to be dealt with or dismissed first.

there are various things in the issue tracker that need to be dealt with or dismissed first.

Any thing that would make it harder to fix these issues here after merging? (issues can be migrated as well, if this is the concern).

Hm, well, you never know if an innocuous looking issue results in comparatively major effort to fix it, maybe breaking APIs in the process. No one cares if that happens in ufoLib2 :)
Oh, and I need to finish up the WOFF info support at some point.

Hm, well, you never know if an innocuous looking issue results in comparatively major effort to fix it, maybe breaking APIs in the process. No one cares if that happens in ufoLib2 :)

That is bound to happen to any piece of software, sooner or later. You will always have the choice between breaking API or not breaking it and living with the consequences of either decision, or no software library will be ever released. If it is good for use in ufo2ft it should be good here IMHO.

Am I correct when I summarize that ufoLib2 really is two things?

a UFO reader/writer library as an alternative to fontTools.ufoLib
an simple object library partially compatible with defcon objects

a UFO reader/writer library as an alternative to fontTools.ufoLib

That part was dropped in favor of fontTools.ufoLib already.

@khaledhosny maybe, but I'm a perfectionist and prefer to clean my dirty laundry before showing it to the public :)
@justvanrossum ufoLib2 uses ufoLib to read, write, validate. It started out as a complete replacement (hence the number) but the scope was later reduced. Point two is true. See https://ufolib2.readthedocs.io/en/latest/explanations.html.

Got it, thanks! It must've been the name that confused me :)

(Regarding replacing ufoLib, I think engineering time would be better spent on refactoring it -- e.g. we found that ufoLib's data validation is often tied to the functions that do the actual work, so if you start to write out all the files and there's a data error somewhere deep down your UFO, you are left with a UFO on disk in an undefined state. One could write off-target and rename once done and validated, but there's got to be a better way)

No objections, makes sense to be here in fontTools. I would like to see any differences between the defcon api and fonttools.ufolib.objects clearly documented so that folks don't get confused and shoot themselves in the foot if they are expecting one behavior and don't get it, but other than that, it seems a good idea.
@madig we dug into speed a bunch; i've not looked deeply into norad, but do you know if the speed difference is due to a faster xml parser in rust, not having validation (we put in an option to turn off validation into ufoLib as requested to speed things up, not sure if fontmake is using that or not), or something else? I.E. is this a Python vs. Rust issue or a code issue? Curious.

I have not yet looked into where the speed difference comes from, but it may well be using a compiled to machine code language vs. Python. From my profiling efforts, a large amount of time in ufoLib is spent simply reading/writing stuff from disk.

Edit: (Also, read/write performance could maybe be optimized. I was astonished that https://github.com/linebender/norad is able to read and write UFOs with thousands of glyphs in sub-second time both ways. On Windows!)

Interesting! I think ufoLib uses the Expat parser underneath several wrappers. I wonder if Rust is using something different?
As Ben said, we dug into speed a lot. I spent an agonizing amount of time looking for bottlenecks and eked out everything I could. The validation and conversion, mostly the GLIF validation and version conversion, is a huge expense. Another is the expense of setting up defcon's object notification system during load. At one point, there was some redundant loading in ufoLib as a result of lazy partial parsing (for example: it would skim GLIFs for <unicode> elements to create a CMAP-like dict, but the loaded text would be discarded. Then, a fraction of a second later, the whole GLIF would be loaded. The IO expense added up.). Anyway, I've written various defcon-lite objects at different times and they have always been significantly faster. It'll be very interesting to see how ufoLib.objects compares (my hunch is that it will be significantly faster than defcon). These faster objects will be very useful for quick in and out operations as opposed to the longer, multi-step operations that defcon was designed to facilitate.

In my understanding the bulk of loading time will be spent in creating Python objects.  I don't think there's much we can do there without going to custom arrays (of points etc, like we do in glyf table already) and reaching out to C for populating such object maybe.

At any rate, I think implementing a quick objects scheme and measuring should be the first step.

FWIW, I made a benchmarking repo a while ago: https://github.com/madig/test-generate-instances.
Eagerly loading the NotoSans-Regular UFO takes ~10.8s on my machine with and ~9.4s without validation.
Here's a py-spy generated profile trace (functions are aggregated, no line number information): speedscope.zip. Can be viewed on https://www.speedscope.app/.

Import loading is the purple-ish iceberg to the right. 83% of time is spent in and below readGlyph.

FWIW, I made a benchmarking repo a while ago: https://github.com/madig/test-generate-instances.
Eagerly loading the NotoSans-Regular UFO takes ~10.8s on my machine with and ~9.4s without validation.
Here's a py-spy generated profile trace (functions are aggregated, no line number information): speedscope.zip. Can be viewed on https://www.speedscope.app/.

Import loading is the purple-ish iceberg to the right. 83% of time is spent in and below readGlyph.

I presume that Norad can be used for ufo2ft etc. fine, but would need some clever engineering.

 Total drop-in replacement with fast loading.

@simoncozens interesting! Probably better to file an issue at ufoLib2 repo to continue this discussion.

I'm still confused about this. Is this a proposal for ufoLib or ufoLib2? Can ufoLib2 please be renamed to something that doesn't make it seem like a new version of ufoLib? This is seriously confusing for users.

ufoLib2 originally was intended to one day become part of a proposed fontTools.ufoLib.objects (streamlined defcon-like api), hence the name. Not sure if we still want to do that after all. If you guys feel the name is confusing and want to change it, I don't feel too strongly about it (besides the chore of updating all the import names of maby subprojects). It was meant to be temporary, but then you know how these things go.. Maybe the oxidation can be the chance to find a better alternative name FWIW

create a bunch of ufoLib2 objects in Rust, and hand a ufoLib2.Font back to Python

Can Python classes be instantiated in parallel so we can go as fast as possible before handing back to CPython? 🤔

Python classes be instantiated in parallel

remember the GIL

ufoLib2 originally was intended to one day become part of a proposed fontTools.ufoLib.objects (streamlined defcon-like api), hence the name. Not sure if we still want to do that after all. If you guys feel the name is confusing and want to change it, I don't feel too strongly about it (besides the chore of updating all the import names of many subprojects). It was meant to be temporary, but then you know how these things go.. Maybe the oxidation can be the chance to find a better alternative name FWIW

So what keeps us from moving it to fontTools.ufoLib.objects now, and keep refining?

well, for one thing, adding rust to the mix would break the rule that fonttools is pure-python (see the other thread about cython). I think we can experiment with a separate package first.

My plan at this stage would be a separate module which just oxidises the load step. ufoLib2 can try-import this module and use it to speed up load if found, using lxml if not.

ufoLib2 originally was intended to one day become part of a proposed fontTools.ufoLib.objects (streamlined defcon-like api), hence the name.

Yeah, I know the history. 😉  I get emails fairly regularly from people who are wondering why ufoLib2 is so different from ufoLib. I then explain the history and "This is not a UFO reader/writer directly. It's something like defcon, but without the intra-object communication to speed up operations that don't need that. The response is always, "Why is it called ufoLib2 then? That's confusing."
@behdad

So what keeps us from moving it to fontTools.ufoLib.objects now, and keep refining?

I agree. defcon wasn't built for rapid read-change-write-done operations at large scale, but there is clearly a need for something that does that. Fast objects would be super useful. We could add a wrapper over at fontParts for it so that it has a convenient API for scripting situations.

Yes, currently we use defcon for the fontParts fontshell, but that's overkill for the command line.

My plan at this stage would be a separate module which just oxidises the load step. ufoLib2 can try-import this module and use it to speed up load if found, using lxml if not.

I once had the idea of writing a fontTools.ufoLib in Rust, using norad as the backend, but quickly stumbled over the impedance mismatch between the two: norad is currently written to just load everything up front and then you do whatever you want, whereas ufoLib acts as a load-and-write-through layer, where libraries on top must load and write each individual piece of data and expect results to be immediately written to disk. Something on top of norad would hand over all data at once in a big batch. If you want to mime the ufoLib API with norad, you either do a considerable rework or just handle pieces like glif loading. Then you may not be able to use rayon.

and expect results to be immediately written to disk

I'm not sure that's true.
>>> from ufoLib2 import Font
>>> f1 = Font("Taronga-Regular.ufo")
>>> f1.info.familyName
'Taronga'
>>> f1.info.familyName = "ImmediateWrite"
>>> f2 = Font("Taronga-Regular.ufo")
>>> f2.info.familyName
'Taronga'

Would there be any significant speed gain from using Python asyncio or the aiofile package?

@simoncozens I meant fontTools.ufoLib, not ufoLib2 😄 See how e.g. https://github.com/fonttools/fonttools/blob/4.21.0/Lib/fontTools/ufoLib/init.py#L1195 writes data out immediately.

See also https://github.com/simoncozens/iondrive , https://github.com/fonttools/ufoLib2/pull/136

The iondrive idea (create ufoLib2 objects using norad) does speed up loading quite a bit, but it seems like it only gives a negligible speed increase with fontmake. (I imagine for tens of thousands of glyphs it would be worthwhile.) A full norad wrap might be faster.
But it looks like it depends on usage patterns. Comparing against my own variable font toolchain (testing with Nunito):
fontmake without iondrive: 18.92s
fontmake with iondrive: 18.93s
babelfont without iondrive: 11.69s
babelfont with iondrive: 8.186s

I'm happy to talk about possible patterns for exposing norad through python in a way that works with existing python API conventions, which I don't think should be too hard (although norad has nowhere near the level of polish or real-world use as any of the existing python libs, and I'm sure we would run into a bunch of differences in behaviour, which I'm also happy to help address as they come up)

@cmyr It would be really useful if you could have a look at https://github.com/simoncozens/boulmer and help me understand how to return the objects to Python as mutable references. Raph said something about wrapping them all in  Arc<Mutex<>>s or something, but we're hitting the limits of my Rust ability.

Can we please disambiguate this conversation from ufoLib vs. ufoLib2? Like the conversation above between @madig and @simoncozens, it is unclear which thing is being discussed. A new issue if this is about ufoLib or a new issue in the ufoLib2 repository would be helpful.

It's all about how to speed up fontmake, and we are exploring many different ways to do that.

@cmyr It would be really useful if you could have a look at https://github.com/simoncozens/boulmer and help me understand how to return the objects to Python as mutable references. Raph said something about wrapping them all in  Arc<Mutex<>>s or something, but we're hitting the limits of my Rust ability.

sure, I can take a look at this in a little bit.

@simoncozens can you use https://github.com/benfred/py-spy to profile your fontmake runs? To see where the time is spent. See the bottom of https://github.com/madig/test-generate-instances#ufo-font-instance-generation-benchmarking

@simoncozens can you use https://github.com/benfred/py-spy to profile your fontmake runs? To see where the time is spent. See the bottom of https://github.com/madig/test-generate-instances#ufo-font-instance-generation-benchmarking

Yeah I'm curious as well. It's been a while since I've seen a measurement.

Here's the profile of building Noto Serif as a variable font.
Headline figures:
12% of time in ufo2ft preprocessor (11% loading glifs)
8% compiling outlines
13% compiling opentype features
15% saving interpolatable master TTF files
6% varLib.merge
32% computing gvar
5% saving

This tells me that speeding up the UFO library is cool but will not really solve the fontmake problems. Here’s some benchmarks I just did building an 82 master variable font:
fontmake = 393s
fontmake + iondrive = 388s
babelfont = 81s
babelfont + iondrive = 63s

(Tip: generate speedscope files and go to www.speedscope.app)
I think the "easy" big wins, trying to go as parallel as possible, may be

Faster UFO loading (more for general purpose UFO processing than fontmake, maybe)
Faster variation (gvar and UFO interpolation)

I’m surprised that’s where you’re focusing. You could save at least 34% (compiling a feature file per UFO, writing out master TTFs per UFO, merging) by going variable-first.
Don’t think why fontmake is slow. Think why Babelfont is fast. ;-)

Just throwing this out there... Is the problem that UFO format itself (with its idea of many, many, many small files) is not a good fit for your compiling needs? If so, would a different file structure built to your specific needs do more to improve things than trying to skim tiny bits of performance here and there with increasingly complex solutions? We didn't design UFO for rapid read -> build binary of enormous fonts in mind (UFO was designed 18 years ago! :), so it's understandable that it is hitting a wall in a scenario for which it was not designed. It seems to be the common denominator in almost all of the speed issues being discussed.

I'm not saying this to be facetious or cheeky. I reread this thread to see where it all started and all of it goes back to UFO. UFO is really good at being a flexible and reliable design time format. That's what it was intended for. The info in this thread points to it being inefficient as a format for compile time use for large fonts being compiled repeatedly in an environment where speed is a massively important thing. I think back to when we were trying to speed up defcon for the same purpose and we reached a limit because defcon was designed for a different task entirely so there was only so much speed to be gained. The lightweight objects in ufoLib2 were the result a rethink of the base needs from the "what do we need to do?" perspective instead of the "what already exists that we can get away with using?" perspective. Perhaps it would be a good exercise to look at the format from that perspective. Maybe UFO is the best option or maybe there needs to be a format designed for performance intensive compile situations.

I used to think the UFO format was the speed problem, but look at the benchmarks above. It isn’t. Loading UFO files accounts for 10% of the time and once that’s done then you’re in Python object land. Using a faster UFO loader helps a little, but most of the work is nothing to do with UFO. There are other reasons why UFO isn’t necessarily wise as a variable font interchange format, but speed isn’t one of them.

Out of curiosity, why is this specific issue still open here? Shouldn’t fontmake host it?

You could save at least 34% (compiling a feature file per UFO, writing out master TTFs per UFO, merging) by going variable-first.

Oh, yes. I need to test this approach again with fT 4.21.x and I think it's a forward looking workflow. Eagerly awaiting the hb-subset instancer! Still, sometimes you need interpolation :)

The info in this thread points to it being inefficient as a format for compile time use for large fonts being compiled repeatedly in an environment where speed is a massively important thing.

Not entirely sure about this, I think UFO is fine except for all the things we would like to see in a v4; it feels more like a pipeline-is-written-in-Python problem. Just think of all the steps that could happen in parallel but currently happen serially. I think that's where a lot of performance is lying dormant still.

it feels more like a pipeline-is-written-in-Python problem

When is this a problem? Always? Under a specific set of conditions? A specific font or fonts?
Again, I'm not being facetious. I work on the design side and the speed problems are largely negligible there. Speeding up the most expensive operation results in saving a type designer mere seconds of time per day. At this point, most of the expense is in the concept of how a designer interacts with a tool, not the raw performance of the tool. I'm genuinely curious to know where performance is such an issue that putting an optimization underneath Python is under discussion.

The most pressing issue that I see daily at work, on the engineering and help desk/fire brigade side, is that compilation takes so much time, measured in minutes, when you just want to quickly compile a bunch of UFOs. The most brutal example where even our designers lose their cool is our flagship font with 18 masters and gradual addition of new scripts, where:

import into and export from a certain design program available on Windows literally takes multiple minutes (okay, probably not a Python problem), and the export needs to be copied elsewhere by script (Python) because the design program is being stupid about it
the full package includes variable fonts and interpolated instances, processed and partly WOFF2 compressed in multiple steps. Last I tried, the entire process (parallelized by the build system) takes > 10 minutes and consumes 10+ GB of RAM on not too shabby a machine
even doing a subset just to check your work can take 5 minutes (less if you cut out interpolation), so we had to put in build step subdivisions to do as little as possible while still outputting something the designer can test somewhere

This is equivalent to wanting to quickly check a code change but having to wait minutes for the compiler to finish. Now multiply this for a day's worth of work :)
The product owner of Noto once told me a full serial rebuild of the repo takes > 24h.

This is equivalent to wanting to quickly check a code change but having to wait minutes for the compiler to finish. Now multiply this for a day's worth of work :)

Yeah. That would be annoying. This doesn't sound like a problem that can be solved with incremental improvements of a few seconds here and there.

Random thought: if you make a .ttx file of one of the compiled fonts, then time the recompile from the .ttx file, is the compile time fast/slow/other?

You mean how quick a TTX roundtrip is? 3s forth, 6s back on my private machine. You mean one should try to generate TTX dumps instead? :D

That's for the font that takes minutes from UFO!?
Here's what I'm thinking: a background process for the designer that incrementally updates the relevant portions of a TTX with each save (there are hooks for this in RoboFont, not sure about elsewhere, or you could use asyncio like FontGoggles does). When the designer goes to compile the binary, the process is actually TTX -> binary, not UFO -> binary. There'd be the matter of making sure the TTX still reflects the UFO data, but that's doable.

Keep in mind that compiling the constituting 18 masters with 1000 glyphs each (and soon more) and merging them into a VF does take time, it's even worse when it has to be done serially because Python makes it hard to do in parallel.
I suppose incremental compilation could help indeed, but ufo2ft and all underlying libs would have to be adapted away from batch compiler mode I suppose...

does take time

Of course, but does a full compile have to happen 100% of the time? I don't know many designers who are changing every outline, all their features, all their info, all their kerning, etc. in the same save session. Plus, if the TTX is can be guaranteed to be in sync with the UFO source it could be stored as a compile source. This makes UFO the design source and TTX the compile source.

I suppose incremental compilation could help indeed

Yep. This is why ufoLib and defcon use lazy read/write.

but ufo2ft and all underlying libs would have to be adapted away from batch compiler mode...

Hm. I haven't looked at the ufo2ft source code, but I know defcon and ufoLib handle incremental write really well.

pipeline-is-written-in-Python problem.

Babelfont is written in Python and it's six times faster than fontmake. Don't blame Python for this.

I just experimented with having Babelfont calculate the gvar entries in parallel using multiprocessing. Even though this is an embarrassingly parallel task (each glyph can be computed independently), doing it in parallel is much slower because of the GIL and all the pickling/unpickling required. Sometimes worse is better...
Really, the answer is very simple. fontmake is slow because, to generate a 16-master variable font, it compiles 16 GSUB/GPOS tables including parsing (largely identical) feature code; it generates, compiles and saves 17 full (and largely identical) TTF files; and so on.
babelfont is faster because to produce one variable font, it produces one variable font.

doing it in parallel is much slower because of the GIL and all the pickling/unpickling required

FWIW: doing things in parallel in a single process doesn't help because of the GIL. Doing things in parallel in separate processes does help (because this avoids the GIL), but then you need to pickle/unpickle. You can't blame both bottlenecks at the same time :)

Ah! Maybe I want to try multithreading rather than multiprocessing.

Ah! Maybe I want to try multithreading rather than multiprocessing.

It's the other way around: multithreading is single-process, no real parallelism due to the GIL, and there's significant thread switching overhead. Multiprocessing delegates to separate processes, and involves implicit pickling. I've had more success with explicit external processes (via the subprocess module) than with the multiprocessing module.

Hmm. It strikes me that rather than trying to parallelise gvar creation, it may be more fruitful to try vectorising it with numpy.

it may be more fruitful to try vectorising it with numpy

That's what I do in FontGoggles and can be very effective. In addition to that I wrote a tiny bit of C code that writes directly from the array data to a native bezier path, so there's no inner loop in Python that needs to juggle lots of numbers.
Parallelizing may still be effective if you can avoid excessive pickling/unpickling by sending and receiving "simple data". Eg. send ufo paths and glyph names to the subprocess, receive raw gvar glyph data back.

it may be more fruitful to try vectorising it with numpy

That's what I do in FontGoggles and can be very effective.

Ho hum.
babelfont + iondrive: 8.86s
babelfont + iondrive + numpy gvar: 9.21s
Babelfont's gvar implementation is already pretty well-optimised, but at the cost of debuggability (lots of nested zip(*thing) loops to do matrix transformation), so at least the numpy version is much easier to understand.
Admittedly that's doing the vectorization per-glyph. I'm sure there's a clever way to build a coordinate ndarray for the entire font at once, but it's too clever for me to find.

Maybe use https://github.com/plasma-umass/scalene to see if you're accidentally spending all your time copying objects around, see https://github.com/plasma-umass/scalene#output.

babelfont + iondrive: 8.86s
babelfont + iondrive + numpy gvar: 9.21s

Going in and out of numpy is relatively expensive, esp. with floats. So if you only do a small number of vector operations, the from/to Python conversion may cost more than the vector operation saves. In FG I never convert back to Python, thanks to the C code, and I never need to convert from Python again after the delta's have been set up. It's optimized for interaction, and perhaps these ideas simply don't translate well to a one-shot pipeline.

Admittedly that's doing the vectorization per-glyph. I'm sure there's a clever way to build a coordinate ndarray for the entire font at once, but it's too clever for me to find.

Not worth it.

Great discussion. I have input on various aspects. Different goals justify different actions at this point. I'll try to group them logically:
Main reason why fontmake is slow is this: it was developed with "correctness first" in mind with limited resources. In my view it was going to be rewritten at some point.
Based on Simon's flamegraph in https://github.com/fonttools/fonttools/issues/1095#issuecomment-797755327, below I identify low-hanging fruit that can be sped up with relative ease in the existing fontmake, while addressing the "to Python or not Python" and "Babelfont's 5x faster" questions implicitly. In a subsequent comment I'll share some ideas that I had back then for what a rewrite might look like.
For the sake of discussion, I like to name three different use-cases for fontmake:

Incremental build: Ie. "edit, build, test, repeat" loop.
Fat build: build a full set of binaries from one source. Eg. variable-ttf; full set of ttf and otf instances; perhaps subsetted versions of the character set, etc.
Narrow build: Just build one output binary from one source. Eg. build a variable-ttf.

The rest of this comment is mostly involved with speeding up narrow builds, as in "build me a variable font".
Low-hanging fruit
For reference, here's the flame-graph:

I think we can make it 2x faster by trimming fat from the following:

The middle of the graph (the wide column with _writeTable etc) suggests that individual master binaries are generated; indeed, two columns to its right, the mention of "expand" suggests that then the binaries are being loaded again.  That's unnecessary. We just need to build Python TTFont but no need to compile it before passing to varLib to make a varfont.  Removing that unnecessary save (and fixing up any breakage from it) should save 1 babelfont of work.

Another ~1 babelfont of time is spend in iup_delta_optimize. There are multiple lessens here:

babelfont doesn't necessarily do everything fontmake is doing. So it's expected that it's faster.
Python is fine if it's used for scripting, orchestrating, but is VERY inefficient for implementing algorithms that do lots of work per each byte / coordinate, etc. And iup_delta_optimize is one such algorithm. It's a simple dynamic-programming. The runtime is well-understood. It would be probably over 1000x faster if done in C.

Note: This is not the only time we have hit that. I have implemented optimal dynamic-programming-based glyf outline compilation as well, but it was too slow and the savings negligible, so it's disabled and the default greedy implementation used: https://github.com/fonttools/fonttools/blob/2687c12c2e9dc372e3f35ee5e7318c20c1519c26/Lib/fontTools/ttLib/tables/_g_l_y_f.py#L819

iup_delta_optimize has two parts: simple cases are handled first (entire contour shifted, etc; no actual interpolation involved); then the full algorithm. The full algorithm's gains are extremely tiny: maybe a dozen bytes in a mid-sized font. So perhaps if it's very slow and gains are very small, maybe it shouldn't be enabled by default. If we had optimization levels like compilers do, this should be enabled by -O3 or -Os/-Oz, but not -O2 which would be the recommended default.
I have a feeling that the slow part can very easily be Cythonized though. Definitely worth exploring.

There's no point trying to speed up gvar generation using fancy facilities.  The gvar is already generated very optimally since entire GlyphCoordinates objects are passed to VariationModel. Now, the GlyphCoordinates uses an array.array for storage; this was done when I removed NumPy. As a result, the __add__/__mul__ etc of it are implemented in Python. They are still fast, but making those happen in C would give us significant speedup in gvar generation. Try Cythonizing those? Ideally maybe share this code / work with fontTools.misc.vector.

Currently OTTableWriter keeps a list of "items" that are concatenated at the end. The items are typically bytes() objects that are 2 or 4 bytes long. That's insane overhead to use a Python object to store 2 bytes. Now I'm thinking that switching OTTableWriter to use an array.array internally as a bytes-builder makes more sense.  The offsets and other calculated ints need more bookkeeping to know their position inside the table; that can be done easily. I have a feeling that this can make GSUB/GPOS table compilation many times faster.

Tangent
The GSUB/GPOS compiler in fonttools is REALLY slow if overflows happen. I started working on fixing that in the 99proof branch back in 2018:
https://github.com/fonttools/fonttools/tree/99proof
I talked about it extensively at Robothon 2018, the so-called "99 Proof Smal Batch Distillery":
https://vimeo.com/330981972
But never finished it.
Later in 2018 / 2019, I wrote a design document for the HarfBuzz Subsetter, building on top of those ideas but suggesting a different approach, called "Faster Horse Freezer":
https://goo.gl/bHvnTn
but I never implemented that. Fortunately, @garretrieger implemented most of the reordering ideas from 99proof into a hb-repacker module, which we are landing in HarfBuzz today:
https://github.com/harfbuzz/harfbuzz/pull/2857
The code is very isolated from the rest of HarfBuzz Subsetter. It basically takes a graph of objects and links, and tries to produce an ordering that wouldn't overflow. It shouldn't be too much work to pass the OTTableWriter graph to C and call the repacker on it...

Another easy way to parallelize things is using multiple Python processes. Here's a simple way to speed up fontmake 2x or more that way:
Multi-processing
One process loads the project then spawns multiple processors to divide work. These processes load and compile things and send back to the orchestrator (or other processes) using pipes, sending very simple data that can be sent almost verbatim (eg. bytes()). Here's a simple task breakdown:

One process to load glyphs of each master UFO. Sends GlyphCoordinates or similar fast object using Pickle or possibly custom (eg. FlatBuffers) to the consuming process,

One process receives GlyphCoordinates from all masters and builds glyf and gvar. If the font is single-mastered, the extra process separation won't happen.

One process to load .fea files / kerning and compile feature files. This might become  dependent on .glyphs loader for anchors though. In that case, the .glyphs loader process might need to send them here, causing an unpleasant time dependency, but probably still fruitful. This process compiles GSUB/GPOS/GDEF binaries and send back.

The orchestrator process itself can build the misc small tables (head, etc) while waiting for others.

The GSUB/GPOS compiler in fonttools is REALLY slow if overflows happen.

This is another good reason not to run it 32 times.
Any tweaks that are still O(number of masters) aren’t really interesting.

+1 for multiprocessing. It's amazing how much of a modern processor's power sits idle while waiting for that 1 Python process. I've implemented multiprocessing in a couple of places in my custom scripts (woff generation, kern pair filtering), and even on my 10-year-old 4-core machine the difference is huge.

babelfont doesn't necessarily do everything fontmake is doing. So it's expected that it's faster.

I added IUP now.

I did some testing around multiprocessing at https://github.com/madig/test-generate-instances and implemented it for Cascadia Code. It's about running compilation in parallel instead of parallelizing subtasks. It can yield significant speed-ups, but e.g. integrating it into our build system yielded not as much. So, it's tricky.

Here's a tracing of fontmake -m NotoSans-MM.designspace -o variable from my repo above for more detaield analysis: fontmake_speedscope.txt. Load it up in www.speedscope.app to browse it.
There's some interesting stuff in there, did you know that computing glyph bounds takes up 9% of the runtime, presumably because it is called so often (total time >> self time), on par with _iup_contour_optimize_dp?

Any tweaks that are still O(number of masters) aren’t really interesting. (And that particularly includes the needless conversion of Glyphs masters to UFO masters to TTF masters before actually doing anything with the data.)

I wouldn't say that. Up to TTFont masters is well justified. But they should then be merged without going to binary and back.

I did some testing around multiprocessing at https://github.com/madig/test-generate-instances and implemented it for Cascadia Code. It's about running compilation in parallel instead of parallelizing subtasks. It can yield significant speed-ups, but e.g. integrating it into our build system yielded not as much. So, it's tricky.

Right. Another cheap trick is, in fontmake itself, fork for generating each instance.

Here's a tracing of fontmake -m NotoSans-MM.designspace -o variable from my repo above for more detailed analysis: fontmake_speedscope.txt. Load it up in www.speedscope.app to browse it.

Awesome. Thanks.

There's some interesting stuff in there, did you know that computing glyph bounds takes up 9% of the runtime,

Well, most of it will go away if we stop compiling the masters to binary.

presumably because it is called so often (total time >> self time), on par with _iup_contour_optimize_dp?

Now with your more detailed analysis I can zoom in and see that more than half the time in iup_delta_optimize is also spend loading ("expand") glyf outlines. Which, again, if we didn't save the masters to binary wouldn't happen to begin with.
So, really, someone should just see why we are compiling masters to binaries. Remove that, get fresh flamegraph then we see where we are. Right now it's obvious that's the most wasteful thing happening.

Great. I started looking into it in https://github.com/googlefonts/ufo2ft/issues/485 and promptly got flooded with obscure test suite errors when I made PostProcessor stop reloading every font that gets passed in. It seems to me to be an easy way to clean up the data that gets passed in. Ugh.

...got flooded with obscure test suite errors when I made PostProcessor stop reloading every font that gets passed in...

That's the curse of a language like Python: that you can't compile-time enforce immutable types / instances. :(

Basically, babelfont is doing to fontmake what Chrome did to Firefox and clang to gcc: give them a kick in the posterior to get their things together and run :)).

Well, maybe. But I'm not doing babelfont directly to compete with fontmake. In writing Flux (layout editor), I need to be able to address font source files in a format-independent way, and I need to be able to create and emit layout rules that vary across a font's design space.
The first part of that is not really in scope fontmake, and the second part - variable layout rules - you are never going to be able to do while you have a split/merge workflow. That's the real reason I need this stuff.

and the second part - variable layout rules - you are never going to be able to do while you have a split/merge workflow. That's the real reason I need this stuff.

The only reason fontmake was done the way it was done is that there was no alternatives available on the timescale needed. Indeed, as you see, adding variations to .fea file made zero progress between 2016 and now that you picked it up. Back in 2016 I couldn't take on upsetting Adobe, etc. So we just made fontmake take sources in existing formats. Neither the input format, nor the architecture are set in stone.

I did some optimizations in GlyphCoordinates: https://github.com/fonttools/fonttools/pull/2261
@madig can you profile with that change?
I think I can extract some juice from optimizing TupleVariation as well. But after that we should go back to the charts to see where to dig next.

FontToold 4.22.0: fontmake.txt
Your branch: fontmake2.txt
No appreciable diffs in wall time.

So there's good news and bad news :).
The good news is that I made gvar compilation 40% faster in https://github.com/fonttools/fonttools/pull/2261
The bad news is that this probably made babelfont much faster than it made fontmake faster. :))

Should we start assessing how much work babelfont would need to have all the features we need?

Please see https://github.com/simoncozens/babelfont/issues/14
Even if you use Babelfont for development and fontmake for production it’ll speed up  your iterations.

I opened an issue to optimize IUP mechanism:
https://github.com/fonttools/fonttools/issues/2264
I've also separately suggested to turn off the full IUP in fontmake:
https://github.com/googlefonts/fontmake/issues/759

(I just watched https://www.youtube.com/watch?v=r-TLSBdHe1A and now want this)

(I just watched https://www.youtube.com/watch?v=r-TLSBdHe1A and now want this)

Thanks for sharing.
I also use cProfile module, which gives you exact number each function is called. That helps giving me spots that might be easy to optimize..

I also use cProfile module, which gives you exact number each function is called. That helps giving me spots that might be easy to optimize..

Here's something weird. In calling varLib to build NotoSans varfont from interpolatable-masters, we call ttFont.__getitem__() over 200,000 times...

Use py-spy, make a speedscope file and on www.speedscope.app select "Sandwich" mode and click on the appropriate getitem and see where the calls are coming from :dancers:

Also consider accidental quadraticity like in https://github.com/adobe-type-tools/psautohint/pull/289, which can easily cause 200k calls :D.

Also consider accidental quadraticity like in adobe-type-tools/psautohint#289, which can easily cause 200k calls :D.

My personal favorite:
https://github.com/fonttools/fonttools/commit/85be2e0a9773acec3c6d14c345b1fd94ab3aa5c3

Use py-spy, make a speedscope file and on www.speedscope.app select "Sandwich" mode and click on the appropriate getitem and see where the calls are coming from

BTW; speedscope app works on my Chrome Mac; but Chrome Linux shows me a blank page. Have you experienced that by any chance? I haven't got to figure out what's going on.

Huh. Works for me (Chrome and Firefox)? Maybe uBlock blocking more than expected? :D BTW you can use Ctrl+F to search the sandwich list for names.

Here's something weird. In calling varLib to build NotoSans varfont from interpolatable-masters, we call ttFont.__getitem__() over 200,000 times...

Is from varLib's main loop:
        for glyph in font.getGlyphOrder():

                isComposite = glyf[glyph].isComposite()

                allData = [
                        m["glyf"].getCoordinatesAndControls(glyph, m, defaultVerticalOrigin=baseAscent)
                        for m in master_ttfs
                ]

I'll probably rewrite to remove use of getCoordinatesAndControls and append phantom points in one go instead of fetching those per glyph.

good catch!

Also prepare a list of (m, m[“glyf”]) before the hot loop.

Also prepare a list of (m, m[“glyf”]) before the hot loop.

Yeah, that was my initial instinct. But it's inside the getCoordinatesAndControls fetching the phantom points that is really wasteful. So I'm going to make it inside-out.

Yeah, that was my initial instinct. But it's inside the getCoordinatesAndControls fetching the phantom points that is really wasteful. So I'm going to make it inside-out.

https://github.com/fonttools/fonttools/pull/2266

Random showerthought this morning: Imagine we have a fast, efficient Rust library for reading, manipulating and writing TTF files. Imagine also that we write a set of small, single-purpose command-line utilities which filter TTF files and manipulate them in a single, specific way. Basically I want you to imagine that loading and saving TTF is essentially a zero-cost activity.
Now, here's the question: what elements of the existing Python toolchain could we offload to these fast single-purpose Rust filters on an incremental basis, without requiring a major rewrite of everything? What can we do in a mixed Python/Rust environment? Quite a lot, including many of the things we have been madly trying to optimize here:

Load TTF, rename glyphs to production, save TTF.
Load TTF, IUP gvar table, save TTF.
Load TTF, decompose components, save TTF.
Load TTF, remove overlaps, save TTF.
Load TTF, generate STAT table, save TTF.
...

These are nice, distinct, small, and hence hopefully easily achievable projects which we can drop in as replacements to the existing pipeline.
Of course, if we end up loading the result of a filter back into fontTools again, then adding an external filter step is not going to be exactly a zero-cost activity, but I wonder if we can get enough of the work done in these postprocessing filters that after doing an initial conversion to TTF we don't need to go back into Python and do all the finalization in Rust filters.
And even if we do need to go back to Python and this is slower in the short term, it has the advantage of moving us in a good direction for the long term.

I fully support rewriting fontTools or at least ttLib in Rust. However, don't the listed tasks require that you manipulate the ttLib object graph from Python? Rewriting that in Rust would therefore require an interop layer between the rusty OpenType data structures and the code that tries to fiddle with it? I'd be worried that this requires an elaborate glue layer akin to what @cmyr is experimenting with for norad.

I’m envisioning stand-alone command line filter utilities, no Python at all.

@thundernixon gave a little video presentation a few days ago about some of his UFO workflows, and one of the questions I came away with was whether or not we were asking UFO to do too much, in being both a file format during the design process (a problem it feels fairly well suited to) as well as the de facto standard for compilation and post-processing pipelines (where it feels like an awkward fit). It made me wonder if there might be space for a separate (binary) interchange format, that was designed specifically for this second task? Forgive me if this has been discussed previously..

a separate (binary) interchange format ... designed specifically for post-processing pipelines

from time to time I entertain that idea as well. Imagine using protobuf or capnproto for source font formats.

@cmyr yes, Tal said something to the same effect above ( reference here if helpful: https://github.com/fonttools/fonttools/issues/1095#issuecomment-798566863 —yes, this thread is really long!)

from time to time I entertain that idea as well. Imagine using protobuf or capnproto for source font formats.

It might be useful to experiment with .ttx as the format between UFO and binary. TTX may not be the fastest possible way, but the toolchain for going UFO -> .ttx and .ttx -> binary exist so concept this can be tested without building much. The way I've thought of doing this is something like this: When a batch of UFO changes are declared "complete" , the .ttx is opened as a TTFont. The changed data in the UFO is loaded into the TTFont. The TTFont is then saved as .ttx. When binary compilation is needed, the .ttx is used.

from time to time I entertain that idea as well. Imagine using protobuf or capnproto for source font formats.

I've been thinking along those lines for a while as well. I've come to the conclusion that FlatBuffers is the right format for this, and in some future as the font format itself possibly. Check it out:
https://google.github.io/flatbuffers/flatbuffers_internals.html

I made a little prototype command-line filter in Rust to demonstrate the sort of thing I'm talking about above - it doesn't quite do everything yet, but what it does is:

Load and parse a font, deserialising the tables into structs.
Rewrites the glyph strings in post table, renaming the glyphs according to the Unicode values in the cmap table (i.e. rename the glyphs to production names)
Serialize the font back to disk.

Remember I said "I want you to imagine that loading and saving TTF is essentially a zero-cost activity"?
time ./target/release/ttf-rename-glyphs Sofia-NoProd.ttf Sofia-Prod.ttf
./target/release/ttf-rename-glyphs Sofia-NoProd.ttf Sofia-Prod.ttf  0.01s user 0.00s system 90% cpu 0.011 total

So.... some goood newwws... Just running varLib on pypy3 instead of python3 (CPython) makes it some 40% faster....
I haven't tried with full fontmake. But that's what @madig should look into. pypy jit-compiles stuff. When I looked into it years ago, I was looking to speedup subsetting, and since each subset operation was shortlived (<100ms), that wasn't enough for pypy's jit magic to kick into action. But with fontmake, it totally does.

Cool, thanks @simoncozens (impressive!) and @behdad 😀 I will experiment with the stuff when I get to it.

This is the kind of thing I'm thinking of (using Sofia Sans, 940 glyphs):
$ time ttf-remove-overlap < Sofia-NoProd.ttf  | ttf-rename-glyphs | ttf-fix-checksum > Sophia-Prod.ttf
ttf-remove-overlap < Sofia-NoProd.ttf  0.03s user 0.00s system 92% cpu 0.031 total
ttf-rename-glyphs  0.01s user 0.00s system 31% cpu 0.042 total
ttf-fix-checksum > Sophia-Prod.ttf  0.01s user 0.00s system 23% cpu 0.051 total

This is not the kind of thing I'm thinking of, and it's pretty much just a toy, but I kind of hand all the components handy so I did it anyway:
$ time ./target/release/ufo2ttf NotoSans-Regular.ufo NotoSans-Regular.ttf
./target/release/ufo2ttf  NotoSans-Regular.ttf  0.33s user 0.08s system 99% cpu 0.413 total

Send PRs, not issues. :-)

So.... some goood newwws... Just running varLib on pypy3 instead of python3 (CPython) makes it some 40% faster....

Meh. Running it on full fontmake is like 40% slower...

I guess we have to pivot to Rust now :crab:

Yeah I'm convinced we're wasting our time trying to optimize Python....

I still think there are things to optimize in Python-land, i.e. the way features are put into a VF is, uh, suboptimal. Once I get some time again, I'll continue to follow the profiler.

Sure. This is what we have right now and will take a while to completely replace it. I'll also continue optimizing IUP and GSUB/GPOS packing.

I'll have fonttools-rs handling gvar very soon. Then we can do IUP in there.

I'll have fonttools-rs handling gvar very soon. Then we can do IUP in there.

Count me in :)).

Meh. Running it on full fontmake is like 40% slower...

Could it be that some C extension is wreaking havoc? I understand that the use of C extensions can be (very) slow in PyPy. Could it be lxml? Just a wild guess.
Re: fonttools-rs: that's some awesome results already! Looking forward to these new building blocks.

@simoncozens @behdad It would be worth thinking about how fonttools and fonttools-rs would relate. Would fonttools-rs become fonttools? If so, what effort would be spent documenting how to incorporate the rust version into python workflows. Would fonttools-rs become a dependency of fonttools and handle some things, but not all things? If so, what should the name of fonttools-rs be, as having two things called fonttools is super confusing for folks that don't track everything.
I know this is an experiment at this point, and I don't want to slow that down. But, thinking about this now would be helpful. The last thing that one wants is to have two projects doing similar things that then don't get attention because attention is split between the two.
This is basically a reminder that there are lots of folks who use fonttools who are primarily designers. They dabble in code to get things done, but it can be mysterious and confusing at times to them. That isn't a group to be dismissed, but embraced when thinking about shifts. How can a change be made seamless for them? Doing so has the side benefit of making it even easier for those who are in Vi/Emacs all day every day.

Could it be that some C extension is wreaking havoc? I understand that the use of C extensions can be (very) slow in PyPy. Could it be lxml? Just a wild guess.

Could it be. Cosimo suggested the same. I read the following article recently so was hoping that it wouldn't be that bad:
https://morepypy.blogspot.com/2018/09/inside-cpyext-why-emulating-cpython-c.html?m=1

It would be worth thinking about how fonttools and fonttools-rs would relate. Would fonttools-rs become fonttools? If so, what effort would be spent documenting how to incorporate the rust version into python workflows. Would fonttools-rs become a dependency of fonttools and handle some things, but not all things?

OK, here's kinda my vision for fonttools-in-rust:
FontTools-in-Python is really important as a way for designers to manipulate font files in custom ways. As you mention, designers know Python, they want to write simple scripts, and they can. That's great, and I don't want to change that. So fontTools is definitely part of the future.
However, one major use case of fontTools is not custom, designer-generated, code - it's basically running fontmake, which is what this issue is about. What I'd like to do is:

in the short-term, see if any parts of the fontmake process (or any other commonly repeated tasks) can be rewritten in Rust to make them faster. e.g. renaming glyphs to production, IUP gvar, remove overlaps, etc. Once those pipeline utilities are in place, we can get fontmake to call them instead of implementing those bit of the process itself.
in the long term, turn the whole of fontmake into Rust.

Of course to do that, we need a Rust library which reads, manipulates and writes OT files, which is basically "fonttools equivalent" - but I'm doing this not because I want to replace fonttools per se, but because I want to replace fontmake.

Running time fontmake -m myfont.designspace -o variable on a ~1000 glyph, 15 master designspace gives me the following:
CPython 3.8
real	0m31.321s
user	0m30.656s
sys	0m0.621s

PyPy 3.7
real	0m15.511s
user	0m14.796s
sys	0m0.701s

This is the set of dependencies that I installed manually (I use pip install --no-deps XXX until fontmake worked):
attrs-20.3.0
cu2qu-1.6.7
fontMath-0.6.0
fontmake-2.4.0
fonttools-4.22.0
fs-2.4.13
pip-21.0.1
pytz-2021.1
setuptools-47.1.0
six-1.15.0
typing_extensions-<IP_ADDRESS>
ufo2ft-2.21.0
ufoLib2-0.11.1
wheel-0.36.2

I reran on NotoSans-MM, this time from designspace.
CPython 3.9.4:
real	0m51.467s
user	0m49.850s
sys	0m1.547s

and PyPy ~3.7:
real	1m7.141s
user	1m4.308s
sys	0m2.501s

I did notice however that the step that says:
INFO:ufo2ft:Pre-processing glyphs

takes ~25s in CPython and ~50s in PyPy. I looked into that step, it looks like that's actually when the UFO glyphs are loaded. So yes, this might be lxml PyPy overhead.

FWIW, I'm getting 40% to 50% speedup when running fontmake -g NotoSans-MM.glyphs -o variable with pypy.
(I can't get skia-pathops to work in pypy, probably something messed up in my py2 (!) environment, so I haven't been able to compare static font building with remove overlaps.)
Adding lxml to my env indeed makes the whole thing a lot slower (close to CPython speed, but still a little faster), even when running from .glyphs.
I wonder how much the JIT warmup costs. If it's significant, a longer running fontmake server process could save more.

FWIW2: I'm running a big CJK pipeline (of which fontmake is only a part): it takes about 19 minutes with CPython, and a little over 5 minutes with pypy3.

Probably moving processing of multiple fonts into Py would help. In cases like Noto, many fonts are built: many sources are read, and many output formats are built. In the current setup, for each font a new Py process gets started, so with PyPy, the JIT optimization gets lost. If multiple files are processed in the same process, then only the 1st font would be slower and then the JIT-compiled code would run faster on subsequent fonts.

There could be a YAML file that holds the tasks for multiple fonts. So all args for all conversions would be in one YAML, fontmake would be called with the YAML and then all would work in t same run.

I can't get skia-pathops to work in pypy, probably something messed up in my py2 (!) environment,

I can't get skia-pathops to work in pypy, probably something messed up in my py2 (!) environment,

we don't distribute wheels for pypy3 (last time I checked there was no portable wheel tag that worked, maybe things have changed since), so it's probably trying to build skia-pathops from source, and skia requires python2 to be on $PATH (it's gn build tool only works with python2.7), maybe you don't have that on your local machine.

so it's probably trying to build skia-pathops from source, and skia requires python2

Yes. It was some SSL related thing that I didn't care to debug. For a CI project elswhere I built Linux wheels: https://github.com/justvanrossum/skia-pathops-pypy-wheels/releases/tag/v0.1.0 Perhaps I should try to add macOS to the matrix there. But similar to lxml, it's disproportionally expensive to run pathops under pypy, although it was still a net win in my case.

still a net win in my case

yeah, skia-pathops in general is much faster than booleanOperations (pyclipper).
Another thing you can try is forcing to install the pure-python version of cu2qu. Right now when you pip install cu2qu, since there is no pypy binary wheel on PyPI, it attemps to build one from the .zip source, bootstrapping cython and building the extension module from source. But you can tell it not to use Cython by exporting the environment variable CU2QU_WITH_CYTHON=0 before (re)installing cu2qu. I noticed that static font builds of NotoSans went down from 37 to 32 seconds after I instaled cu2qu in pure-python mode; the variable font build also got faster by 4.4 seconds with pure cu2qu

(Maybe I should detect in cu2qu's setup.py if we are building cu2qu on pypy, and make it default to the pure-python version)

note, if you installed cu2qu normally once, pip may have build and cached a binary wheel, so the next time you pip install cu2qu it will attempt to use that. You can force it to build from source (so that it reads the CU2QU_WITH_CYTHON=0) by passing the pip install option --no-binary=cu2qu:
$ CU2QU_WITH_CYTHON=0 pip install --no-binary=cu2qu cu2qu

yeah, skia-pathops in general is much faster than booleanOperations (pyclipper).

No, I was already using pathops, I meant the switch to pypy :)

(Maybe I should detect in cu2qu's setup.py if we are building cu2qu on pypy, and make it default to the pure-python version)

I tried to remove cu2qu's *.so by renaming it but I couldn't measure any difference. I possibly messed up something.

I tried to remove cu2qu's *.so by renaming it but I couldn't measure any difference.

strange, I can see reproducible 10 to 12% speed gains when switching to pure-python cu2qu

still a net win in my case

yeah, skia-pathops in general is much faster than booleanOperations (pyclipper).

Last time I checked, there was a huge timesink somewhere in bOps, where it converts curves to something pyclipper can handle I think. It shows up as a huge bar in a profiler. bOps can probably be sped up significantly if that bottleneck is removed.

strange, I can see reproducible 10 to 12% speed gains when switching to pure-python cu2qu

I must've been doing something wrong before. I'm seeing good improvements with that now, too. Thanks for the instructions how to configure that!

I have now got gvar generation working (very crudely) in fonttools-rs.
fontmake -m NotoSans/NotoSans-MM.designspace -o variable  66.60s user 2.03s system 95% cpu 1:11.82 total
../target/release/fonticulus NotoSans-MM.designspace NotoSans.ttf  4.19s user 1.12s system 390% cpu 1.361 total

Still some bugs to fix, notably applying the avar maps, and it perform any IUP/shared point optimisations on the gvar table. Can potentially be optimised further by processing the glyphs in parallel, but it looks like UFO loading is the slowest part of the operation.

https://github.com/facebookincubator/cinder
"Instagram's internal performance-oriented production version of CPython 3.8"

https://github.com/facebookincubator/cinder
"Instagram's internal performance-oriented production version of CPython 3.8"

Oh VERY interesting!

Still some bugs to fix, notably applying the avar maps, and it doesn't perform any IUP or shared point optimisations on the gvar table. Can potentially be optimised further by processing the glyphs in parallel, but it looks like UFO loading is the slowest part of the operation.

I fixed a bunch of gvar bugs, applied avar maps, and implemented IUP optimisation. I also changed a .iter into .par_iter, and now the UFO loading is done in parallel.
If anyone's still keeping score, I just built an 81 master, 1049 glyph font in 3.10 seconds.
As far as fontmake is concerned, there are now a bunch of utilities in fonttools-cli which could be used to speed up fontmake/ufo2ft's post processing:

ttf-flatten-components replaces the FlattenComponents filter
ttf-optimize-gvar performs IUP optimisation on an unoptimized gvar table
ttf-rename-glyphs quickly renames glyphs to production names

I tried to get more UFO loading performance out of Rust again: https://github.com/madig/readwrite-ufo-glif (experimental ufoLib2 PR at https://github.com/fonttools/ufoLib2/pull/147)
It uses a variation of what Simon tried before, loading glyphs or entire layers in norad, serializing the data to Python dictionaries and passing them back to Python land, where ufoLib2 re-instantiates them again directly, without going through ufoLib or pens.
It works and speeds up the process, but there remain friction losses. I described my findings in https://github.com/madig/readwrite-ufo-glif/issues/2.
Cosimo suggested trying to reduce the cross-over performance tax by circumventing PyO3 and serializing the data to JSON or pickle it or do something else where you don't need to go through Python's facilities on the Rust side. I'll see.

https://github.com/be-fonts/better-ergonomics-spec/issues/2

I have now got gvar generation working (very crudely) in fonttools-rs.

How does this work?

Same way as anywhere else: convert curves to quadratic, compute the deltas, stuff them in a gvar table.

Closing this sentiissue.