UFO 4 (and 3)

[Tal Leming]

Hi Everyone,

Last week I presented my current thoughts about UFO 4. Video of the presentation can be seen here:

https://www.youtube.com/watch?v=wHf8M6u8Hjc&t=13506

My talk starts around 3:45:00. The basic gist was this:

- We need to put hints into UFO. This is not easy for a lot of reasons. The only thing that I can come up with is to put compiled data for both PS and TT hints in. This isn't ideal, but high-level representations of that data are going to be a long-term project at best.

- I have some thoughts on Adobe's PS hint proposal and I'll post those later. We'll probably do something that is conceptually very similar to what Read has proposed.

- TT hints are an especially massive pile of pain. Seriously. Ouch.

- I have a couple of ideas about some other tiny enhancements.

With the UFO 3 spec, we developed a backwards compatible way of storing new data. That can be used for the data I described above, so those things don't have to wait on UFO 4.

UFO 4 is just getting started and it is likely to be a very big change. We're starting to run into problems with the complexity of UFOs (fonts are getting bigger, bigger, bigger every second of every day) with various new technologies. We're going to start researching different file structures. At this point, I don't want to add any other new features to UFO 4 other than a new file structure. UFO 3 introduced a lot of new things for developers to support and I want to take a break from that. We're in a pretty good place right now in terms of what the UFO contains.

I'll be posting more extensive thoughts, proposals and plans for each of these in the near future as my time permits.

We had a meeting on Saturday morning with around 20 attendees (TWENTY!!!) and we discussed potential file structures and a plan for researching these them. Volunteers stepped up to help with researching this as well as researching some of the complexity of the TT hint issues mentioned above. (I am extremely grateful for this. Thanks everyone!) The minutes of the meeting are below if you want to see what we talked about, courtesy of Mr. Frank Griesshammer.

The UFO 4 research is going to be happening here:

https://github.com/unified-font-object/ufo4Research

Some initial notes and research links are there. Work is underway. Much, much more info about this will be coming just as soon as I get the base implementation of the test suite done.

That's it for now.

Thanks,

Tal

####

UFO spec meeting 2015/03/07

# Potential file structures for UFO:

## zip

seems good except for couple of bad things

potential version control problems

performance with 1000s of files?

alternate suggestions:

a compressed W3C XML exists, but it is binary

KML Google format (diffs?)

## long XML file

single document

UFO contains custom-defined XML + plist

There are capabilities for parsing XML and switching to plist

No images inside file?

“Students move files around like crazy”

Performance question?

Georg is pre-caching XML for writing big files, so the writing is fast.

How to deal with 100K glyphs XML file?

`iterparse` in elemettree

parsing could become performance drain

## SQlite

documented on sqlite site:

“pile-of-files format”

“wrapped pile-of-files format”

Everything could be stored as key-value dict:

fontinfo.plist: XML

Version control inside database?

Does SQLite record history? No.

git has a flag to set for specific repos to enable diffing for sqlite files.

pretty widely supported

many sqlite editors

does/doesn’t pass human-readable test

If one file/entry in the database goes corrupt, does the whole database go corrupt? Apparently not.

Not very prone to database corruption bc crash

Python has sqlite support out-of-the box

“simple db service Amazon is very cheap”

## shallower package

Version control?

ideas:

- Take glyphs directory and fold into one single XML file.

- sub-group glyphs with max number of entries == 500

not passing the “easy to implement” criteria

How to specify where the glyphs are supposed to go?

“Head hurt”

semantics of version control get lost in bucket structure?

Have to establish rules about ordering in big XML file.

====

Sounds like database is going to solve most problems except version control.

Export to “classic” UFO for version control?

Does sqlite change structure without changing content?

====

what about postgreSQL?

noSQL databases?

mongoDB?

couchDB?

====

JH: UFO on windows?

Add a single file that serves as an index?

UFOs just crashing Windows file server

“Dropbox” problem

====

# misc

Local storage is generally not an issue

Local networks? Writing UFOs to local server takes long time.

Dropbox test via human Python API called Guido

Need test suite that writes UFO3

What kind of benchmarks need to be established?

### Conclusion:

Form some kind of working group without phone conferences, quagmire

Tal needs help with testing specific variations of implementations.

Test suite needs to be accessible to foundries who can run it and test it on their systems.

Build small implementations of every idea and compare them.

### Next step:

Start profiling some fonts we can use for testing.

Tal will keep working on UFO reader/writer.

Create Full test suite

Create list for things to be tested

5 things to test per field: max min average below above

Tal will write core library.

====

# Hinting

Rob (MS): Storing compiled data in source format is not a good idea.

Tal agrees, but is dead set on storing compiled data because at this point we need to store something and that's the only common thing between editors.

JH: VTT to work with names?

Could open source compiler code?

Could open source spec?

VTT storing is a different thing than Adobe .fea standard

because it needs to be interpreted, not obvious from just reading it.

VTT should/could be de-facto standard.

PS hints:

Read’s implementation is more low-level, would be preferable high-level.

Tal to figure conversion into plist format from Read’s implementation.

Good enough.

[Dave Crossland]

On 11 March 2015 at 19:10, Tal Leming <t...@typesupply.com> wrote:

alternate suggestions:

a compressed W3C XML exists, but it is binary

KML Google format (diffs?)

If binary formats are under consideration, I highly recommend considering https://github.com/sandstorm-io/capnproto

[Dave Crossland]

On Wednesday, 11 March 2015 21:40:07 UTC+8, Tal Leming wrote:

Sounds like database is going to solve most problems except version control.

Export to “classic” UFO for version control?

Does sqlite change structure without changing content?

Reading about SQLite, I found out about http://www.fossil-scm.org - I wonder that a font.ufo.fossil file might meet all the needs that a new file structure is meant to, while making it exceedingly backwards compatible, like if .ufoz was blessed as ufov4

> I highly recommend considering https://github.com/sandstorm-io/capnproto

https://www.sqlite.org/appfileformat.html makes a good case against developing new application file formats based on something like capn proto, and using sqlite instead.

[Dave Crossland]

On 11 March 2015 at 21:40, Tal Leming <t...@typesupply.com> wrote:

I don't want to add any other new features to UFO 4 other than a new file structure. UFO 3 introduced a lot of new things for developers to support and I want to take a break from that.

I think this is a good idea.

However, I wonder about when and where the appropriate forum is for discussing new features beyond the file structure - since there's a few things that one can reasonably expect from a UFOv5, such as color font support.

While UFOv3's layers provide a path to the 'layered color' fonts from Microsoft, the 'true color' fonts from Adobe and Mozilla use SVG for glyph data within the SFNT container, and it would be nice for the next GLYF format to support that.