The UFO spec process

[Adam Twardoch (List)]

> On 13 Mar 2015, at 09:24, Adam Twardoch (List) <list...@twardoch.com> wrote:
>
>
>> On 13 Mar 2015, at 07:25, Tal Leming <t...@typesupply.com> wrote:
>>
>> 1. Does this cross over into the spec defining application behavior? I don't know, but it is something that we should think about. We *try* to keep the spec from dictating anything other than read, write and "don't discard without alerting the user". There are some interpretation and translation guidelines, but those are mostly to ensure data consistency.
>
> Well, given that the foundation of UFO is very much linked to FontLab Studio 5 (most data structures present in FLS 5 translate to UFO in some way) and AFDKO (most things that makeOTF expects as input are in UFO), I don't think you can really "unlink" UFO and app behavior. UFO seems also to be linked to defcon/Robofab. Given that RoboFont is a "UFO editor", and there also seems to be a certain reluctance in introducing things to UFO that would be difficult for Frederik to implement. I wonder how much this situation is compliant with item #2 of the UFO Design Philosophy for me.

Hello Tal,

In your Robothon talk, you made a comment that went along the lines of "I'm not sure when I should recommend app developers that it's OK to switch to UFO 3".

From what I've understood, you're inclined to wait until one particular implementation of UFO (defcon/Robofab/RoboFont) implements UFO 3, and then it would be "all right" for other implementations (FontLab, Glyphs, FontForge, the ufo.js/Metapolator team) to switch to UFO 3. Am I right or did I (very possibly) not understand you correctly?

That is, of course, "apps dictating spec behavior" or "some apps dictating other apps behavior", in a way. Now, again, I'm not saying that this is a bad thing per se. But it has consequences. For example, I wouldn't mind if one particular implementation of UFO is declared the "reference implementation", and other implementers were encouraged to consider the combination of both the spec *and* the reference implementation as the authoritative source.

I realize the effort that needs to be put into upgrading defcon, Robofab, RoboFont etc. to UFO 3. I now even understand that there may be some fundamental challenges (have to do with the Pen protocol somehow?). And I'm eternally grateful for your efforts that have lead to creating Robofab and your continuous support of the "FontLab world", with, sadly, only very modest help from FontLab's side. We're now in the most fortunate situation where we have RoboFont, Glyphs, FontForge, Superpolator, Metrics Machine, roundingUFO, AFDKO, ufo.js, Metapolator, defcon, Robofab, the upcoming "VIctoria" series of FontLab apps, as well as a number of internal apps made by some font development companies, which all use and support UFO to some extent.

For example, it seems to be that right now, neither RoboFont nor AFDKO support either the UFO 2 spec or the UFO 3 spec.

So there is no reference implementation of neither, and possibly all implementations are non-conformant with the spec. (I cannot quite tell how good the compatibility of Glyphs or FontForge with the UFO spec. I'd be very grateful if people who have experience with this would speak up!)

At FontLab, I'd much rather just support UFO 3 as is, or UFO 2 as is, without any app-specific emulation behavior. Or support the reference implementation. I know how messy it is to support all the under- or non-specified aspects of OpenType already, and I'd really like to avoid going through the same ordeal with UFO. :)

I can think of several possible ways to improve the situation:

1. Adopt a "dotted" UFO versioning. Some people already refer to the "flavor" of UFO that RoboFont outputs (UFO 2 with elements of UFO 3) as "UFO 2.5". Perhaps we should just call it that, formalize it? We might switch to "dotted releases" of the spec, and perhaps update the spec more frequently. I would be glad to assume a somewhat more active role in it, as others might.

2. With RawGit, we could create a fully browsable "development" version of any of the UFO major versions, as in:
https://rawgit.com/unified-font-object/ufo-spec/master/source/versions/ufo3/fontinfo.html
https://rawgit.com/unified-font-object/ufo-spec/master/source/versions/ufo3/glyphs.html

Or perhaps we could convert the current HTML spec to Markdown and use Github's "gh-pages" branch so the spec exists on something like http://ufo-spec.github.io ?

3. We could have a concept of "contributions" or "addenda" or "extensions" to the spec, perhaps hosted in the same Github place, where people could document their de-facto extensions or proposed notations for things like hinting, so implementers might now "faster" what other implementers have in store.

4. Declare RoboFont, or defcon, or Robofab, or something else, a reference implementation, and recommend to the implementers that they consider the spec and the reference implementation as a whole.

Those are all non-exclusive points, and merely ideas.

I have the feeling that the UFO world is "speeding up" at this point, which is great, because it shows that UFO is gaining wide adoption. But that also presents challenges. I'd love to find a method where the burden of decision-making and spec maintenance would not rest entirely on you, Tal. I'm not sure how open you are to changing the process, but if you were, I think there would be more people now than ever before who'd be willing to help you out, myself included, so in the end, you don't need to decide yourself "when it's OK for app developers to switch to version X". It must be a huge responsibility, and I'm very grateful that you have assumed it.

But if you think that I'm trying to fix something that is not actually at all broken, and that the current process is fine, please let me know. I may be missing some points! :)

In general, please let us know how we can help in any way. I'd like to see UFO becoming more community-driven, and I'll try to help in any way I can.

Best,
Adam

[Dave Crossland]

Hi

I recommend www.semver.org for versioning

Cheers
Dave

[Tal Leming]

On Mar 13, 2015, at 1:17 AM, Adam Twardoch (List) <list...@twardoch.com> wrote:

In your Robothon talk, you made a comment that went along the lines of "I'm not sure when I should recommend app developers that it's OK to switch to UFO 3".

From what I've understood, you're inclined to wait until one particular implementation of UFO (defcon/Robofab/RoboFont) implements UFO 3, and then it would be "all right" for other implementations (FontLab, Glyphs, FontForge, the ufo.js/Metapolator team) to switch to UFO 3. Am I right or did I (very possibly) not understand you correctly?

I don't think I said this. Years ago, right before UFO 3 was finished, you asked me what I thought TransType should support. Since UFO 3 was in development, I said UFO 2. My memory is fuzzy on the timeline. Anyway, no, I think if something is trying to support UFO, UFO 3 is the way to go. Supporting UFO 2 and UFO 1 is pretty easy if UFO 3 is supported. I wrote conversion instructions in the spec. The need for those conversion instructions is unfortunate, but times change and data has to adapt.

I realize the effort that needs to be put into upgrading defcon, Robofab, RoboFont etc. to UFO 3.

ufoLib and defcon support UFO 3. RoboFab does not because I got bogged down trying to figure out what to do about UFO 3 support in FontLab. We may end up rewriting the whole thing. RoboFont, I don't know where UFO 3 support stands, but I'm pretty sure that Frederik is mad at me about something in the UFO 3 spec. (Hi Frederik!) MetricsMachine and Prepolator have support for UFO 3 in beta versions, but I got sidetracked writing completely new versions of those tools. (I think you know how that goes.)

I now even understand that there may be some fundamental challenges (have to do with the Pen protocol somehow?).

As far as defcon/ufoLib/RoboFab go, the changes that were the most complex were the new layer things. That was hard. Pens needed to adapt to handle identifiers, but that was more of a goof in our implementation than an actual problem. (Specifically, the base class of the PointPen and sub-classes should have had an open ended **kwargs at the end to handle future additions to the protocol. All fixed in the UFO 3 version of the code.)

For example, it seems to be that right now, neither RoboFont nor AFDKO support either the UFO 2 spec or the UFO 3 spec.

I know that the FDK reads/writes unencoded <data> elements in the glyph lib.

I don't think RoboFont has any non-compliant data in the UFO. I believe that it may write an images directory, which is against the spec but the UFO 2 spec wasn't so clear about that so it's technically not in violation. I don't consideration these to constitute a custom variation. Maybe there are other things that I don't know about though.

Or, do you mean the lib extensions that they use? Those are all perfectly within the spec and are exactly what the lib was built for.

1. Adopt a "dotted" UFO versioning. Some people already refer to the "flavor" of UFO that RoboFont outputs (UFO 2 with elements of UFO 3) as "UFO 2.5". Perhaps we should just call it that, formalize it? We might switch to "dotted releases" of the spec, and perhaps update the spec more frequently. I would be glad to assume a somewhat more active role in it, as others might.

Frederik, if you are reading this, is there anything in the way that RoboFont writes UFO that is in violation of the spec?

3. We could have a concept of "contributions" or "addenda" or "extensions" to the spec, perhaps hosted in the same Github place, where people could document their de-facto extensions or proposed notations for things like hinting, so implementers might now "faster" what other implementers have in store. 

I have given this a great deal of thought. What I would like to see is everyone who has a significant amount of custom data going into (the legal places within) the UFO, document it somewhere. The private.com.example lib key idea that I presented at Robothon relates to this. If you aren't privatizing the data structure, it would be ideal that it be documented. Where? I don't know. It can't be a part of the spec, but maybe there could be a collection of these things somewhere.

4. Declare RoboFont, or defcon, or Robofab, or something else, a reference implementation, and recommend to the implementers that they consider the spec and the reference implementation as a whole.

Erik and I have talked about something along these lines. When we worked on the UFO 3 version of RoboFab's ufoLib (which handles reading and writing UFOs) we broke it apart from RoboFab. We have considered moving ufoLib to the UFO spec and calling it a reference implementation. We haven't done that because of time and because I didn't want the "THIS IS IN PYTHON!!!!?!" swats. Parts of it are already there, for example the identifier generator and the various algorithm example implementations. I think the whole thing will move eventually. The code needs a thorough review first. It's here:

https://github.com/robofab-developers/robofab/tree/ufo3k/Lib/ufoLib

I have the feeling that the UFO world is "speeding up" at this point, which is great, because it shows that UFO is gaining wide adoption. But that also presents challenges. I'd love to find a method where the burden of decision-making and spec maintenance would not rest entirely on you, Tal. I'm not sure how open you are to changing the process, but if you were, I think there would be more people now than ever before who'd be willing to help you out, myself included, so in the end, you don't need to decide yourself "when it's OK for app developers to switch to version X". It must be a huge responsibility, and I'm very grateful that you have assumed it.

We talked about this briefly at the UFO meeting after Robothon. There is more work than I have time for. We need a real test suite with lots of edge cases. That's a big project. UFO needs to be much better with large fonts. That's another big project. Hints need to be supported. That's yet another big project. I can't do all of this myself, so I'm at the point where I need help. I have only 1/4th baked ideas about this, but here goes...

I don't want to turn this into a consensus driven specification. That always ends up as a mess. I have a lot of respect for the W3C and I'm a very proud former Invited Expert. But, the consensus driven process used there can too easily be subverted or go off the rails into theoretical-use-case-so-we-should-support-it-land. I think someone needs to be in charge so that there is a definitive voice and vision. If that's me, that's fine. If it's me, Erik and Just, that's fine. If I get the feeling that someone other than me should be saying yes or no to things, I'll gladly step aside and go draw some fonts.

There would be a general discussion area (this list) where we talk about things as a group. I'd like to have some sort of structure where there could be small, focused groups that work on specific problems that the larger group identifies as needing a solution. This group would consist of people who actually need the problem to be solved, understand how to solve it and can write a proposal for the solution along with example code. This small group would check in with the big group and eventually propose something. The big group would review, the proposal would be revised, (usual design process goes here) and eventually whoever has the final authority/gets the ultimate blame would say yes or no. Then a timeline for release would be developed and we'd go from there. I'm vaguely familiar with Python's PEP process and I think it is something like this.

I don't know how formal this should be. I don't want conference calls. I want it to be small and focused but include as many experts as needed. I want the process to be open so that there are no accusations of private agreements.

At the meeting, this all sort of happened organically. A few people are working on hinting issues. A handful of people want to work on the UFO 4 practicality tests. Someone volunteered to help with making the spec a little easier for me to edit. I'm not good with delegating tasks or organizational structures, so if anyone has ideas about how to do this, I'm all ears.

Thanks,

Tal

[Frederik Berlaen]

A side note on RoboFonts ufo2 implementation of ufo3 features like layers, images, guides, ... (ufo3 was back then in an early draft state)

For a very, very, very, short period of time RoboFont produced ufo2+ files, that was before it was ever released and before it was even called RoboFont (for historic reasons: the app was called first Doodle). This never saw daylight from the dungeons of the precious early beta testers.

Since the release RoboFont produces valid ufo2 files. Guides are stored both in the glyph.lib as in the font.lib. Images are stored in the glyph.lib as a hard path to the image file. There is an additional file added next to the ufo, not inside a ufo, if a user wishes to collect them during save, this is custom to RoboFont and an easy way to share both ufo data as image data. Layers are stored in glyph.lib as valid plist. Point labels are stored in the name tag. Before RoboFont did so, I tested this on all my ufos both on my HD as my production/mastering machine, non of those ufos had points with data in the name tag. 

RoboFont has always been very careful in writing valid ufo files. This has been a struggle as it is easy to violation the spec.

 but I'm pretty sure that Frederik is mad at me about something in the UFO 3 spec. (Hi Frederik!) MetricsMachine and Prepolator have support for UFO 3 in beta versions, but I got sidetracked writing completely new versions of those tools. (I think you know how that goes.)

not mad, I just have to play the developer, the webmaster, the support dude, the teacher and the dad, I think you know how that goes :)

PS: UFO3 is coming... no date, no promise, but it will be there...

gr Frederik

www.typemytype.com

--
You received this message because you are subscribed to the Google Groups "Unified Font Object Specification" group.
To unsubscribe from this group and stop receiving emails from it, send an email to ufo-spec+u...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Adam Twardoch (List)]

Frederik,

many thanks for your detailed reply. I am not that much of an expert in RoboFont, so I relied on some more experienced users' feedback who told me the term "UFO 2.5".

> Point labels are stored in the name tag. Before RoboFont did so, I tested this on all my ufos both on my HD as my production/mastering machine, non of those ufos had points with data in the name tag.

This is the issue which I noticed myself because the FontLab TTH functionality as implemented in vfb2ufo relies on point names. So my question is: does any of RoboFont's internal functionality rely on any point names? And specifically, that the point name are treated as, in fact, a "list" that is comma- or space-separated? Will anything "break" if the user in an external app changes a point name?

One thing that I've been thinking a lot (and the reason why I proposed formalizing this "references" concept is that all UFO implementers may run into the following situation:

There are some points, guidelines, anchors or other objects in UFO. They have identifiers, names and possibly tags.

In the glyph.lib, there may be structures that one app or plugin writes, and another may not "understand".

Now, imagine that another app opens one such UFO, and then the user modifies some stuff. Let's say changes some points or guidelines, adds some, removes some, moves some, whatever. That app may have a concept of identifiers identical to UFO 3, but may have a different concept. So it may need to, say, change all or some identifiers, tags or names.

While not ideal, it would still be good if such an app had a "basic understanding" of the custom glyph.lib.

In a way, the custom glyph.lib may define both some internal custom logic *and* some relationships to standardized structures. The TTH structure is a perfect example (let's assume it remains custom). In the TTH structures, you have portions which another app may not understand at all (such as "links", "attachments", "interpolations", "deltas", whatever), but you also have references to standardized objects (points, guides etc.).

So the non-understanding app could "understand" half of the custom glyph.lib entry (the references) but not understand another half (the logic). If that app needs to "rename" or "re-identify" some objects, it should "know" which pieces of the custom glyph.lib to "rename" as well, without destroying the logic. Essentially, it'd be great if we establish some minimal procedures for apps to interact with unknown data in such a way that, whenever possible, that data is not rendered useless.

> PS: UFO3 is coming... no date, no promise, but it will be there...

That's understandable. UFO 3 is a big change. I keep my fingers crossed! :)

Best,
Adam

[Frederik Berlaen]

Adam

> I am not that much of an expert in RoboFont,

I understand :)

> so I relied on some more experienced users' feedback who told me the term "UFO 2.5".

don't, if you can ask the one and only :)

Im using point labels and unique identifiers, so does TTH (correct my if Im wrong)
This is extremely powerfull and opens tons of functionallities and opportunities to alter data and connect data.

So far RoboFont is the only one using this and no internal logic relies on this, this data belongs to the user (as all the data that is written back into the UFO)
Storing point ID and labels in point.name is just a temporally solution, when UFO3 gets implemented this will not be an issue.

vfb2ufo can add whatever in the point.name but tools like TTH and some other external extensions (that Im aware off) will complain or not function as it should be. Not so nice...

Tools that are writing data in custom lib keys should decide if they want to open a small spec how to data should be formatted. If that data is widely used it can receive a public* tag from Tal, similar to "public.glyphOrder" and "public.markColor"


So far RoboFont stores point labels and unique ID in the point.name tag as a "," comma separated list, an unique ID is a label starting with an "*" asterisk, the order is not important, spaces aren't allowed (they are removed silently)

example
name="label1,serif,top,*A511C5E"

if you consider to store labels and identifiers in UFO2, I would advise to follow this tiny spec, it has proven itself bullet proof the last 4 years

> I keep my fingers crossed! :)

I dont understand how you are still able to type!



gr Frederik
www.typemytype.com

[Adam Twardoch (List)]

> On 18 Mar 2015, at 15:37, Frederik Berlaen <fred...@typemytype.com> wrote:
>
>> so I relied on some more experienced users' feedback who told me the term "UFO 2.5".
>
> don't, if you can ask the one and only :)

Good point :)

> Im using point labels and unique identifiers, so does TTH (correct my if Im wrong)

Yes, this is how we have done it (in co-operation with Jeremy) for the first TTH implementation (the one that has to hide inside base64 for reasons why all know now). So we followed your tiny spec, and will continue as long as we do UFO 2.

>> I keep my fingers crossed! :)
>
> I dont understand how you are still able to type!

Dictation software, man! :) This is why what I send to this list comes out so odd. I forgot to install the "de-assholization" plugin for my Dragon Naturally Speaking app. ;)

A.

[Tal Leming]

> On Mar 18, 2015, at 9:09 AM, Adam Twardoch (List) <list...@twardoch.com> wrote:
>
> Now, imagine that another app opens one such UFO, and then the user modifies some stuff. Let's say changes some points or guidelines, adds some, removes some, moves some, whatever. That app may have a concept of identifiers identical to UFO 3, but may have a different concept. So it may need to, say, change all or some identifiers, tags or names.

For it to be in compliance with the spec, it must not change the identifiers. For example, here is what is said in the guideline spec:

>> However, once an identifier has been assigned to a guideline it must not be unnecessarily removed or changed. Identifiers may be changed in incoming guidelines during editing operations such as "paste," but they should be maintained unless a duplicate identifier will be created. The identifier value must be unique within all identifiers (including identifiers for elements other than guidelines) in the glyph that the guideline belongs to but it is not required to be unique among the identifiers assigned in other glyphs or in fontinfo.plist.

The reason that the spec is strict about this is that Tool A doesn't need to know about Tool B's custom data structures. Tool A can manipulate the official elements as it sees fit so long as it does not arbitrarily change object identifiers without a good reason. (There *are* good reasons sometimes.) If Tool B's custom data is very specific to a particular state of an official element, Tool B should have some way of checking the validity of the data between Tool B sessions. This is similar to what I have been saying about hinting. If we store hint data in the lib, either in a high-level or low-level representation, we will need to store some sort of fingerprint. Perhaps it would be good for the spec to define a suggested fingerprinting algorithm with differing degrees of specificity so that everyone isn't always reinventing this wheel.

> So the non-understanding app could "understand" half of the custom glyph.lib entry (the references) but not understand another half (the logic). If that app needs to "rename" or "re-identify" some objects, it should "know" which pieces of the custom glyph.lib to "rename" as well, without destroying the logic. Essentially, it'd be great if we establish some minimal procedures for apps to interact with unknown data in such a way that, whenever possible, that data is not rendered useless.

This gets to be really complex really fast. If we had such a thing, tools would have to iterate over the entire lib, and its entire depth, and manipulate relevant values as needed. This could be a very expensive operation to perform in something like a point drag. It's far more efficient for tools to do their own special checking as needed.

Another reason that we didn't go this way is that tools would have to trust each other to update each other's data. In the example above, Tool B would have to assume that Tool A did the right thing. We all know what happens when software starts assuming things about data validity, so Tool B would necessarily have to validate the data it stored before. At that point, it needs to have a fingerprint to reference, so we're back to verifying everything anyway.

Make sense?

Thanks,
Tal