Reviving TTX and FontTools

Did you know...?

LWN.net is a subscriber-supported publication; we rely on subscribers to keep the entire operation going. Please help out by buying a subscription and keeping LWN on the net.

At ATypI 2014 in Barcelona, Google's Behdad Esfahbod gave a report on the work he has been doing to resurrect and reinvigorate the development of a long-dormant free-software project used for font manipulation. In addition to bug fixes and optimizations, he has added several new tools to the original set, and the code is now in use not only at Google, but by several third-party applications as well.

The project in question is known to most users as TTX/FontTools; FontTools is the name of the underlying Python library, while TTX is the most commonly used tool from the included suite of utilities. Esfahbod said he had been "working on a secret project" to revive FontTools for the past few years; he has now made the project public and is inviting users and contributors to get involved. Google started using it internally for font-production work in 2013, at which point he began adapting it and adding new features. His version is available on GitHub, and there is a discussion forum on Google Groups.

By way of background, he outlined the history of FontTools. The project was started in 1999 by Just van Rossum, but in a few years work on it had slowed down to "maintenance mode." Development virtually ceased in 2004, although the code lived on at its SourceForge project page and has continued to be packaged by multiple Linux distributions. The better-known TTX component is a command-line utility that converts font files to XML and vice-versa, since editing and processing XML is considerably easier than performing the same tasks on binary OpenType files. FontTools is the library that powers TTX, he said, but it can do much, much more.

The usefulness of TTX comes from the fact that its XML representation is extremely close to the internal format of OpenType tables. Thus, it is easy to convert a font through TTX and back again, getting a stable, usable output font without losing any information. "I'll make a bet," he said, "no matter what you use to create a font, if you round trip it through TTX, it will be functionally the same." It can look inside of a font file and extract data that other tools cannot. In other words, he said, "it expresses what you want to do with a font much more compactly" than other tools do.

Practical matters

FontTools's XML compiler/decompiler functionality, Esfahbod said, can also be used to optimize fonts—reducing redundancies, for example. The XML it produces is also more compact than that of other conversion tools, so operations are quite fast—fast enough for batch operations on the hundreds of fonts used in the Google Fonts service, for example. In addition, FontTools works with both Python 2 and Python 3, it has no external dependencies, and it introduces minimal abstractions, making it simple to learn.

He then discussed a series of practical usage examples. FontTools can read Type 1, TrueType, and Compact Font Format (CFF) fonts (as well as several less-common formats). It can be used to quickly figure out what a font file includes, such as character coverage, hinting, ligatures, and kerning. That makes it possible to "diff" font versions in a meaningful way, as well as to analyze and generate reports (particularly on large batches of fonts). It can also be used as the final step in producing fonts—for example, adding name-table entries and hints, or even subsetting (such as generating a Japanese-only font file) and merging font files. As a library rather than a graphical application, he said, FontTools is also one of the only options for doing server-side manipulation or reporting tasks, and its XML serialization feature can turn binary font files into a version-control-friendly text representation.

Another major use case for FontTools is "hotfixing" fonts, he said. Converting to XML and back makes it easy to regularize glyph names, fix encoding problems, or add missing characters—for example, he said that Google uses FontTools to make sure that its fonts are not missing the non-breaking space character, which is easy to overlook. As font engineers, he said, there are a number of such things that type designers do not need to worry about, "but are fun for those of us who enjoy writing scripts."

New work

Initially, Esfahbod said, his work on FontTools involved implementing longstanding bug fixes and feature requests that had gone unaddressed in the SourceForge project. He then moved on to ensuring that the XML format was stable, that conversion produced predictable output, and to optimizing the output to be as compact as possible (which he said was vital for Google's web-font service). He then focused on speed improvements, telling the audience that "Python can be slow—if you're not careful about it."

Eventually, though, he began adding new features to the library, and ultimately new utilities as well. Among the highlights are support for Web Open Font Format (WOFF) files and many additional OpenType tables—including bitmap tables and all of the competing chromatic font formats: SVG glyphs and the various formats promoted by Google, Apple, and Microsoft.

There are three new utilities available, he said: pyftsubset, pyftmerge, and pyftinspect. pyftsubset is for generating a subset of an existing font—only certain character blocks, for example, or even a minimal set of glyphs for use in a particular document. It supports both TrueType and CFF glyphs, and it optimizes file output size. pyftmerge is the opposite; it combines multiple fonts into a single file. At the moment it is a "very early prototype," he said, supporting just TrueType. It includes logic to handle conflicting characters and hinting, but in a naive manner at the moment: it preserves the contents of the first input file given. pyftinspect is a previewer, showing the font's table contents in a pop-up window.

In addition to Google Fonts, Esfahbod said that there were already several other users of his relaunched FontTools code. They include several font foundries, Adobe's AFDKO, the free-software font build-service tool Font Bakery, and even Adam Twardoch, one of the main developers at the proprietary font-design-software company FontLab. There are many more features still on the to-do list, Esfahbod said, including support for more input formats, a "lint" tool, better merging options, and better optimization. He would also like to add entirely new functionality, such as the ability to convert between TrueType and CFF, and the ability to optimize TrueType outlines.

There is more work already in development. An intern has been working for the past few months on a CFF subroutiner (which reduces file size by eliminating redundancy) that is expected to be released as free software soon. Esfahbod is also working with a faculty and student team at the University of Waterloo on code to analyze TrueType's bytecode language.

Impact

The font-development community is heavily invested in Python; it is the preferred scripting interface to essentially all font-design applications. But there are relatively few projects that provide a reusable library. In fact, the only other such open-source library that implements a large feature set is Robofab, which is developed primarily to serve as the scripting interface to the commercial font editor Robofont. Nevertheless, there are a lot of small free-software utilities and scripts out there that use Robofab.

But FontTools and Robofab are not direct competitors; they serve two very different purposes. Robofab is a library for working on font design: glyph drawing, spacing, kerning, and so on. The primitives it defines are limited to what is required for these tasks (points, segments, contours, glyphs, etc), and it is designed to work with the Unified Font Object (UFO) drawing format. FontTools, on the other hand, is designed to operate on already built font files: .TTFs, .OTFs, .WOFFs, etc. Its abstractions are not about the drawing of letters and numerals, but about the (arguably) more tedious process of verifying and manipulating all of the tables and embedded features required to make a font work. The fact that it has survived in most major Linux distributions for a decade without real maintainership is a testament to the demand for such a library.

Esfahbod's report on FontTools was also interesting in light of Adobe's subsequent release of AFDKO as an open-source project. Both of these projects attest to the fact that the font-development community is quickly learning the value of sharing tools. Whereas in past years much of the "open source discussion" at ATypI and similar events was focused on the design of open-font licensing, this year open source was more prevalent in conversation—but in a more subtle fashion.

In fact, the first talk of the week was on the subject of what commercial font-makers can learn from open source, and was given by Thomas Phinney, who works at FontLab. He cited advantages that people already enmeshed in FOSS will find familiar: the value of collaboration, the freedom to specialize on certain tasks by not reinventing the wheel for every task, the benefit to everyone of sharing information, and so on. Open source is still only one of multiple development models within the industry, but it has never looked healthier.