Bug#864082: fontconfig: please make the cache files reproducible

[Chris Lamb]

Source: fontconfig
Version: 2.12.1-0.1
Severity: wishlist
Tags: patch
User: reproduci...@lists.alioth.debian.org
Usertags: timestamps
X-Debbugs-Cc: reproduc...@lists.alioth.debian.org

Hi,

Whilst working on the Reproducible Builds effort [0], we noticed that
fontconfig generates non-reproducible cache files under
/var/cache/fontconfig.

This is because fontconfig embeds the mtime of each font directory
in a "checksum" member of a "_FcCache" struct. This is so that it
can identify which cache files remain valid and/or require
regeneration.

We therefore "clamp" the mtimes of font directories to SOURCE_DATE_EPOCH
prior to calling fc-cache to avoid these non-deterministic values
appearing in the files themselves. This is safe as we now force
regeneration in subsequent fc-cache calls with -f.

(We can't just replace the checksum value with SOURCE_DATE_EPOCH as it
will result in fontconfig believing the cache to be outdated, defeating
the entire point of generating them in the first place.)

This work was sponsored by Tails[1].

Patch attached.


[0] https://reproducible-builds.org/
[1] https://tails.boum.org/


Regards,

--
,''`.
: :' : Chris Lamb
`. `'` la...@debian.org / chris-lamb.co.uk
`-

[Chris Lamb]

Chris Lamb wrote:

> Patch attached.

A better patch is in-progress on the upstream mailing list, also
attached here.

[Chris Lamb]

Chris Lamb wrote:

> A better patch is in-progress on the upstream mailing list, also
> attached here.

It should eventually appear here, I assume I'm stuck in a
moderation queue or similar:

https://lists.freedesktop.org/archives/fontconfig/2018-May/thread.html

[Keith Packard]

Chris Lamb <la...@debian.org> writes:


> + source_date_epoch = getenv("SOURCE_DATE_EPOCH");

Could this work as a build-time value in the library instead of a
run-time environment variable?

--
-keith

[Chris Lamb]

Hi Keith,

> > + source_date_epoch = getenv("SOURCE_DATE_EPOCH");
>
> Could this work as a build-time value in the library instead of a
> run-time environment variable?

Unfortunately not. Imagine the situation where we are installing
font packages in a chroot that will eventually end up as, for
example, an .ISO: in this case, we are running fc-cache at runtime
(in Debian's case, via the dpkg trigger).

[Keith Packard]

Chris Lamb <la...@debian.org> writes:

> Hi Keith,
>
>> > + source_date_epoch = getenv("SOURCE_DATE_EPOCH");
>>
>> Could this work as a build-time value in the library instead of a
>> run-time environment variable?
>
> Unfortunately not. Imagine the situation where we are installing
> font packages in a chroot that will eventually end up as, for
> example, an .ISO: in this case, we are running fc-cache at runtime
> (in Debian's case, via the dpkg trigger).

Thanks for the explanation.

I think it would be useful for me to understand when and where the cache
files end up being part of a build product and then figuring out what
the right solution is in each case, rather than an environment variable
kludge of this nature.

For instance, in the case described above, the ISO is read-only in use,
and so the cache file contents *cannot* be out of date, and should
always be used with no need to even check the timestamps on
directories. I can imagine a special flag to fc-cache that would
mark the cache files for this use. I feel that this would solve the
problem in a better way.

--
-keith

[Chris Lamb]

Hi Keith,

> I think it would be useful for me to understand when and where the cache
> files end up being part of a build product

The idea is that two entirely separate builds of an essentially-
vanilla Debian-based ISO end up being bit-for-bit identical.

Currently that is not occurring as, for example:

/var/cache/fontconfig/04aabc0a78ac019cf9454389977116d2-le64.cache-7

.. in a "first" build has different contents to the one in a
second varying based on the mtime of the source directories (which
were naturally created at different times).

> I can imagine a special flag to fc-cache that would mark the cache
> files for this use. I feel that this would solve the problem in a
> better way.

As outlined above, the problem is more that the cache files
themselves are varying on each creation. It is also unclear how
this special flag would be activated via, essentially, a call to
"apt install fontpkg". :)

[Chris Lamb]

forwarded 864082 https://lists.freedesktop.org/archives/fontconfig/2018-May/006271.html
thanks

This is now *really* on the upstream mailing list... :)

https://lists.freedesktop.org/archives/fontconfig/2018-May/006271.html

[Chris Lamb]

Hi,

> fontconfig: please make the cache files reproducible

This was merged into the upstream Git repository - would it be
possible to make another Debian release with this change? :)


Best wishes,

[Chris Lamb]

Chris Lamb wrote:

> This was merged into the upstream Git repository - would it be
> possible to make another Debian release with this change? :)

Gentle ping on this? :)

[Chris Lamb]

Chris Lamb wrote:

> This was merged into the upstream Git repository - would it be
> possible to make another Debian release with this change? :)

Gentle ping on this? :) Would love to see this Tails-related work
in Debian!


Best wishes,

[Chris Lamb]

Dear Keith,

> > Gentle ping on this? :) Would love to see this Tails-related work
> > in Debian!
>

> I was stalling for an upstream release with this patch; it looks like
> that shouldn't be more than a month or two from now.

Heh, it seems like fontconfig is perpetually in this state alas. ;)

> Any particular reason for urgency here?

Not especially but it's always nice to reduce the package differences
between in Tails and Debian; the philosophy is very much in that
direction rather than "just" throwing patches over the wall and hoping
they land.

[Keith Packard]

Chris Lamb <la...@debian.org> writes:

> Chris Lamb wrote:
>
>> This was merged into the upstream Git repository - would it be
>> possible to make another Debian release with this change? :)
>
> Gentle ping on this? :) Would love to see this Tails-related work
> in Debian!

I was stalling for an upstream release with this patch; it looks like

that shouldn't be more than a month or two from now. Any particular
reason for urgency here?

--
-keith

[Chris Lamb]

Dear Keith,

> > Gentle ping on this? :) Would love to see this Tails-related work
> > in Debian!
>
> I was stalling for an upstream release with this patch; it looks like
> that shouldn't be more than a month or two from now. Any particular
> reason for urgency here?

Somewhat; it's currently causing buster-based builds for Tails to be
unreproducible.

This means that the in-depth results are being ignored (hey "it's
already unreproducible!") and thus other issues may pile up, causing
scheduling/fixing issues later on.


Best wishes,

[Sven Joachim]

On 2018-05-18 08:42 +0100, Chris Lamb wrote:

> Hi,
>
>> fontconfig: please make the cache files reproducible
>
> This was merged into the upstream Git repository - would it be
> possible to make another Debian release with this change? :)

It seems that 2.13.1-1 was that release, can you confirm that?

Just flying by - apparently the new fontconfig upstream release has
fixed several bugs in the BTS, but the Debian changelog did not mention
any.

Cheers,
Sven

[Chris Lamb]

Hi Sven,

> It seems that 2.13.1-1 was that release, can you confirm that?

Thanks for chiming in.

Given that this version does not contain the string SOURCE_DATE_EPOCH,
this was probably another release and/or this should remain open.

[Sven Joachim]

On 2018-09-26 19:24 +0100, Chris Lamb wrote:

> Hi Sven,
>
>> It seems that 2.13.1-1 was that release, can you confirm that?
>
> Thanks for chiming in.
>
> Given that this version does not contain the string SOURCE_DATE_EPOCH,

It does:

,----
| $ grep -n SOURCE_DATE_EPOCH **/*
| doc/fontconfig-user.html:1549:>SOURCE_DATE_EPOCH</I
| doc/fontconfig-user.html:1556:>SOURCE_DATE_EPOCH</TT
| doc/fontconfig-user.html:1571:>SOURCE_DATE_EPOCH</A
| doc/fontconfig-user.sgml:806:<emphasis>SOURCE_DATE_EPOCH</emphasis>
| doc/fontconfig-user.sgml:807:is used to ensure <literal>fc-cache(1)</literal> generates files in a deterministic manner in order to support reproducible builds. When set to a numeric representation of UNIX timestamp, fontconfig will prefer this value over using the modification timestamps of the input files in order to identify which cache files require regeneration. If <literal>SOURCE_DATE_EPOCH</literal> is not set (or is newer than the mtime of the directory), the existing behaviour is unchanged.
| doc/fontconfig-user.sgml:812:fc-cat(1), fc-cache(1), fc-list(1), fc-match(1), fc-query(1), <ulink url="https://reproducible-builds.org/specs/source-date-epoch/">SOURCE_DATE_EPOCH</ulink>.
| doc/fontconfig-user.txt:790: SOURCE_DATE_EPOCH is used to ensure fc-cache(1) generates files in a
| doc/fontconfig-user.txt:794: to identify which cache files require regeneration. If SOURCE_DATE_EPOCH
| doc/fontconfig-user.txt:801: [3]SOURCE_DATE_EPOCH.
| doc/fonts-conf.5:756:\fBSOURCE_DATE_EPOCH\fR
| doc/fonts-conf.5:757:is used to ensure fc-cache(1) generates files in a deterministic manner in order to support reproducible builds. When set to a numeric representation of UNIX timestamp, fontconfig will prefer this value over using the modification timestamps of the input files in order to identify which cache files require regeneration. If SOURCE_DATE_EPOCH is not set (or is newer than the mtime of the directory), the existing behaviour is unchanged.
| doc/fonts-conf.5:760:fc-cat(1), fc-cache(1), fc-list(1), fc-match(1), fc-query(1), SOURCE_DATE_EPOCH <URL:https://reproducible-builds.org/specs/source-date-epoch/>\&.
| src/fccache.c:1079: source_date_epoch = getenv("SOURCE_DATE_EPOCH");
| src/fccache.c:1086: "Fontconfig: SOURCE_DATE_EPOCH invalid\n");
| src/fccache.c:1090: "Fontconfig: SOURCE_DATE_EPOCH: strtoull: %s: %llu\n",
| src/fccache.c:1094: "Fontconfig: SOURCE_DATE_EPOCH has trailing garbage\n");
| src/fccache.c:1097: "Fontconfig: SOURCE_DATE_EPOCH must be <= %lu but saw: %llu\n",
| src/fccache.c:1112: if (getenv("SOURCE_DATE_EPOCH"))
`----

Did you look at the git repository on salsa.debian.org? It only
contains the debian/ directory, so SOURCE_DATE_EPOCH does indeed not
show up there…

Cheers,
Sven

[Chris Lamb]

Hi Sven,

> It does:

Mea culpa; I was looking at 2.13.0-1, not 2.13.1-1. Thanks!

[Chris Lamb]

Hi Johannes,

> Unfortunately, fontconfig still installs unreproducibly. Try this:
>
> $ sudo SOURCE_DATE_EPOCH=1540078640 debootstrap --variant=minbase --
> include=fontconfig unstable debian-fontconfig1
> [...]
> $ sudo SOURCE_DATE_EPOCH=1540078640 debootstrap --variant=minbase --
> include=fontconfig unstable debian-fontconfig2

On this. Indeed, I can reproduce it:

$ diffoscope --markdown=- debian-fontconfig*/var/cache/fontconfig

# Comparing debian-fontconfig1/var/cache/fontconfig & debian-fontconfig2/var/cache/fontconfig

## file list

@@ -1,5 +1,5 @@
+7fd806a4-197a-4989-8a34-2c49019d041b-le64.cache-7
+95c367ca-9c9b-47d7-9625-c03688da4239-le64.cache-7
CACHEDIR.TAG
-a4fcff53-9cdb-4103-baea-3115d0f9e21e-le64.cache-7
-afd762ff-b72e-4c3f-98f5-19b3b7cf7f95-le64.cache-7
-c8c796f6-9945-4521-bb11-2ad6a193bcf2-le64.cache-7
-ee218622-3364-4921-aaae-6e7d011e7c5e-le64.cache-7
+cb3a236e-83c1-49d4-92f9-a44aa67ef71c-le64.cache-7
+dc12f21a-6ea1-4373-b9e4-cfc7bd8165f7-le64.cache-7

## stat {}

@@ -1,8 +1,8 @@

Size: 4096 Blocks: 8 IO Block: 4096 directory
Links: 2
Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root)

-Modify: 2018-10-29 15:00:44.915312284 +0000
+Modify: 2018-10-29 15:02:00.475850377 +0000

Birth: -

## Comparing debian-fontconfig1/var/cache/fontconfig/CACHEDIR.TAG & debian-fontconfig2/var/cache/fontconfig/CACHEDIR.TAG

### stat {}

@@ -1,8 +1,8 @@

Size: 200 Blocks: 8 IO Block: 4096 regular file
Links: 1
Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root)

-Modify: 2018-10-29 15:00:44.915312284 +0000
+Modify: 2018-10-29 15:02:00.475850377 +0000

Birth: -


Best wishes,

[Chris Lamb]

forwarded 864082 https://lists.freedesktop.org/archives/fontconfig/2018-October/006374.html
thanks

Chris Lamb wrote:

> On this. Indeed, I can reproduce it […]

Can also confirm that SOURCE_DATE_EPOCH is available in the
postinst and to fc-cache itself;locally added some printf(3)
statements to FcDirChecksum.

> @@ -1,5 +1,5 @@
> +7fd806a4-197a-4989-8a34-2c49019d041b-le64.cache-7
> +95c367ca-9c9b-47d7-9625-c03688da4239-le64.cache-7
> CACHEDIR.TAG
> -a4fcff53-9cdb-4103-baea-3115d0f9e21e-le64.cache-7
> -afd762ff-b72e-4c3f-98f5-19b3b7cf7f95-le64.cache-7
> -c8c796f6-9945-4521-bb11-2ad6a193bcf2-le64.cache-7
> -ee218622-3364-4921-aaae-6e7d011e7c5e-le64.cache-7
> +cb3a236e-83c1-49d4-92f9-a44aa67ef71c-le64.cache-7
> +dc12f21a-6ea1-4373-b9e4-cfc7bd8165f7-le64.cache-7

These are uuid's generated by uuid(3)'s uuid_generate_random.

I've sent a patch upstream to:

https://lists.freedesktop.org/archives/fontconfig/2018-October/006374.html

[Chris Lamb]

[Adding 864...@bugs.debian.org to CC]

Dear fontconfig maintainers,

I've just spent a coffee-or-two unpicking this to get the latest status
and to load the history back into my brain.

As a bit of background, I'm working on the Reproducible Builds
effort and fontconfig — in its usual usage, or at least in Debian
at the time — generated unreproducible cache files.

This was due to it using the timestamps of each directory in the
`checksum` member of the `_FcCache` struct. This is so that it can

identify which cache files remain valid and/or require regeneration

(or similar logic).

So therefore in June 2017 I sent an initial patch:

https://lists.freedesktop.org/archives/fontconfig/2017-June/005948.html

… which, after some (private?) discussion regarding the implementation, resulted in:

https://lists.freedesktop.org/archives/fontconfig/2018-May/006285.html

… and that was merged after some further round-trips in f098adac54:

https://lists.freedesktop.org/archives/fontconfig/2018-May/006289.html

… which was released as part of:

$ git tag --contains f098adac54 | head -n1
2.13.1

So far, so good. However, Johannes Schauer then reported that
fontconfig "still" installs unreproduciby:

https://bugs.debian.org/864082#101

… so I prepared a new patch:

https://lists.freedesktop.org/archives/fontconfig/2018-October/006374.html

… and that was "soft NACK'd" in the sense that Keith mentions:

> I've dug into this a bit more and I think an architectural change in the
> cache files made last year is probably not what we want.

— https://lists.freedesktop.org/archives/fontconfig/2018-October/006376.html

(I am now inferring that it was this "architectural change"
resulted in the regression Johannes reported, rather than the bug
being incomplete from the beginning.)

Anyway, the upshot from my proposal was that some larger/different
changes are/were "requested" instead.

Behdad Esfahbod also chimed in with:

> I don't like the new mechanism either, but I think it was added to resolve
> bind-mounted font dirs

— https://lists.freedesktop.org/archives/fontconfig/2018-October/006381.html

… in the context of Flatpak apps. Keith then addressed all this
with a branch which he published here:

https://gitlab.freedesktop.org/keithp/fontconfig

… the most salient commit being (I think?):

https://gitlab.freedesktop.org/keithp/fontconfig/commit/a04751b2e624d034becec7588159ef2f9a8dfc1b

Since then, I don't believe there has been any review of this
branch both in the sense of the code itself but also in terms of
the architectural changes that it implies. I might be able to help
on the former front but without knowing the "lore" of Fontconfig I
simply cannot comment on the latter parts.

Anyway, I'd love to get this resolved once and for all ideally get
it into Debian buster which is about to start "freezing" very
soon.

What would be the best way for me to help here? Can I entreat Keith
to merge his branch? I can put some cycles onto this issue if that is
of some assistance.


Best wishes,

--
Chris Lamb
chris-lamb.co.uk / @lolamby

[Akira TAGOH]

Thank you for reminding and sorry for late response on this. I was on
long holidays.

On Fri, Jan 4, 2019 at 9:29 PM Chris Lamb <ch...@chris-lamb.co.uk> wrote:
> Since then, I don't believe there has been any review of this
> branch both in the sense of the code itself but also in terms of
> the architectural changes that it implies. I might be able to help
> on the former front but without knowing the "lore" of Fontconfig I
> simply cannot comment on the latter parts.

As of the discussion on the list, Keith's changes doesn't address the
original purpose - allow sharing caches on bind-mounts in flatpaks.
particularly for the case where flatpaks uses same location in
sandbox. this is the reason why it can't be merged into master.
So just reverting the change that removes .uuid file only when a
directory has empty is done in master so far. if you have .uuid file
prior to run fc-cache, your issue could be worked around at this
moment. for more details, you can check what Alex Larsson said on this
list.

>
> Anyway, I'd love to get this resolved once and for all ideally get
> it into Debian buster which is about to start "freezing" very
> soon.
>
> What would be the best way for me to help here? Can I entreat Keith
> to merge his branch? I can put some cycles onto this issue if that is
> of some assistance.
>
>
> Best wishes,
>
> --
> Chris Lamb
> chris-lamb.co.uk / @lolamby

> _______________________________________________
> Fontconfig mailing list
> Fontc...@lists.freedesktop.org
> https://lists.freedesktop.org/mailman/listinfo/fontconfig



--
Akira TAGOH

[Keith Packard]

Akira TAGOH <ak...@tagoh.org> writes:

> As of the discussion on the list, Keith's changes doesn't address the
> original purpose - allow sharing caches on bind-mounts in flatpaks.
> particularly for the case where flatpaks uses same location in
> sandbox.

I'm probably forgetting a bunch of context here, but I think the problem
was that flatpaks have /run/host/fonts pointing to the "real"
/usr/share/fonts and then a separate /usr/share/fonts of their own with
a small set of fonts, and so you end up with collisions in the cache
file namespace as both directories end up generating the same cache file
name.

> this is the reason why it can't be merged into master. So just
> reverting the change that removes .uuid file only when a directory has
> empty is done in master so far. if you have .uuid file prior to run
> fc-cache, your issue could be worked around at this moment. for more
> details, you can check what Alex Larsson said on this list.

Making the build reproducible means having all content generated
deterministically based only on the source package and toolchain. The
current UUID files are generated randomly making them
non-deterministic.

For the font cache, making it reproducible requires that the keys
mapping directories to cache filenames be the same each time the cache
is built. This means we cannot use the current randomly generated UUID
values and also have a reproducible system.

I considered whether we might provide a mechanism to generate UUID
values deterministically for purposes of packaging. However, this would
mean that we couldn't use these same packages when creating a flatpak as
the deterministic UUID values would collide if those same packages were
used in the outer system.

Without deterministic UUID values, I'm left with the feeling that
our only available solutions involve changing how flatpaks reference
fonts.

If we agree that a solution to this involves changing the flatpak
mechanism, I'd like to suggest that the most straightforward fix for the
overall system would be to expose the external fonts using the external
path names -- bind mounting the external /usr/share/fonts as
/usr/share/fonts within the flatpak, and creating a new
/usr/share/fonts-minimal (or whatever) to hold the fonts provided by the
flatpak itself. With this change, we can simply delete the UUID code
from fontconfig and go back to using global font paths as keys to the
font cache database.

I'd love to hear about alternative ideas which might lead to solutions
that make builds involving fontconfig reproducible. I'd be happy to take
even vague hints at this point; all I've got at this point are a
collection of dead ends.

(Also, if I've missed or forgotten something relevant, please let me
know; I've re-read a lot of stuff while writing this, but surely
something escaped my notice).

--
-keith

[Akira TAGOH]

On Thu, Jan 10, 2019 at 4:29 AM Keith Packard <kei...@keithp.com> wrote:
> I'm probably forgetting a bunch of context here, but I think the problem
> was that flatpaks have /run/host/fonts pointing to the "real"
> /usr/share/fonts and then a separate /usr/share/fonts of their own with
> a small set of fonts, and so you end up with collisions in the cache
> file namespace as both directories end up generating the same cache file
> name.

That is correct.

> Making the build reproducible means having all content generated
> deterministically based only on the source package and toolchain. The
> current UUID files are generated randomly making them
> non-deterministic.

Hm, I see.

> For the font cache, making it reproducible requires that the keys
> mapping directories to cache filenames be the same each time the cache
> is built. This means we cannot use the current randomly generated UUID
> values and also have a reproducible system.

Right.

> I considered whether we might provide a mechanism to generate UUID
> values deterministically for purposes of packaging. However, this would
> mean that we couldn't use these same packages when creating a flatpak as
> the deterministic UUID values would collide if those same packages were
> used in the outer system.
>
> Without deterministic UUID values, I'm left with the feeling that
> our only available solutions involve changing how flatpaks reference
> fonts.
>
> If we agree that a solution to this involves changing the flatpak
> mechanism, I'd like to suggest that the most straightforward fix for the
> overall system would be to expose the external fonts using the external
> path names -- bind mounting the external /usr/share/fonts as
> /usr/share/fonts within the flatpak, and creating a new
> /usr/share/fonts-minimal (or whatever) to hold the fonts provided by the
> flatpak itself. With this change, we can simply delete the UUID code
> from fontconfig and go back to using global font paths as keys to the
> font cache database.

Indeed, that would be able to accomplish both with the minimal efforts
for us at least. though they might came up with this but they didn't
do it that way. so there might be some reason why they didn't do so.
packaging issue perhaps?

> I'd love to hear about alternative ideas which might lead to solutions
> that make builds involving fontconfig reproducible. I'd be happy to take
> even vague hints at this point; all I've got at this point are a
> collection of dead ends.

I can't figure out completely but, fontconfig may needs to deal with
different namespaces in a cache filename to avoid a collision between
host and sandboxes. dunno if we may see different state in the future
but it might be represented as a depth in a filename to make it
different like 0:<md5>-le64.cache-<version> for host and
1:<md5>-le64.cache-<version> for a sandbox. we could increase a depth
for a child in sandbox as needed, anyway.
We can mix up caches that is located at the same place then. the last
missing piece would be to map them to the right place. flatpaks should
knows where they mounted directories to. they can create a map table
with proper parent depth and current depth I think.

I may be missing something so this might not work though...

> (Also, if I've missed or forgotten something relevant, please let me
> know; I've re-read a lot of stuff while writing this, but surely
> something escaped my notice).
>
> --
> -keith

--
Akira TAGOH

[Alexander Larsson]

On Wed, Jan 9, 2019 at 8:29 PM Keith Packard <kei...@keithp.com> wrote:
>
> Akira TAGOH <ak...@tagoh.org> writes:
>
> > As of the discussion on the list, Keith's changes doesn't address the
> > original purpose - allow sharing caches on bind-mounts in flatpaks.
> > particularly for the case where flatpaks uses same location in
> > sandbox.
>
> I'm probably forgetting a bunch of context here, but I think the problem
> was that flatpaks have /run/host/fonts pointing to the "real"
> /usr/share/fonts and then a separate /usr/share/fonts of their own with
> a small set of fonts, and so you end up with collisions in the cache
> file namespace as both directories end up generating the same cache file
> name.

Yes, and there is two aspects of this problems with this:

A sandbox has something in /usr/share/fonts that isn't the same as
what is on the host, but the cache is shared, so we pick up host cache
for the sandbox directory (or the reverse), causing misrendering of
fonts and whatnot.

A version of the host /usr/share/fonts is available to the sandbox
under a different pathname, causing the host cache to not be used and
thus have to be regenerated, which is slow first time the app is
started.

I'd like to repeat that this is not really flatpak specific as such.
The issue can happen in multiple cases like nfs mounts, multi-boot
systems, docker containers, etc.

Also, the focus on /usr/share/fonts here is mainly for illustration,
the same issue could happen with any other path. For instance,
separate containers with shared $HOME would have its font path
(whatever it is as long as it is the same) conflict between the
containers.

Flatpak has an additional weakness here, which is that we don't store
mtimes (to maximize content sharing abilities the mtime is not part of
the content addressing). This means the mtime can't be used to detect
a stale cache, so we use the uuid to detect such changes.

> Making the build reproducible means having all content generated
> deterministically based only on the source package and toolchain. The
> current UUID files are generated randomly making them
> non-deterministic.
>
> For the font cache, making it reproducible requires that the keys
> mapping directories to cache filenames be the same each time the cache
> is built. This means we cannot use the current randomly generated UUID
> values and also have a reproducible system.

I think this misrepresents what the UUID is for. The UUID represents
a uniqe identifier for the *location*, not the contents, and the goal
is to make it independent on how you found the directory. If you add
fonts to that directory, then you're supposed to keep the UUID because
you want to regenerate the same cache for the new content.

I realize this is not what the reproducible builds project wants, but
it is what the UUID was added for.

> I considered whether we might provide a mechanism to generate UUID
> values deterministically for purposes of packaging. However, this would
> mean that we couldn't use these same packages when creating a flatpak as
> the deterministic UUID values would collide if those same packages were
> used in the outer system.

I think this furthers the misunderstanding from above, but lets
continue this idea.

If the UUID really *was* content addressed, then it would change each
time some font was added, and old font caches would become stale (and
reaped via some other way like mtimes). In this case the fact that
caches between the sandbox and the hosts collide is not even a
problem, since the cached data is identical and could be shared. The
problem is rather that the font directory is mutable, and if it
changes without immediately updating the uuid you run into issues.

> Without deterministic UUID values, I'm left with the feeling that
> our only available solutions involve changing how flatpaks reference
> fonts.
>
> If we agree that a solution to this involves changing the flatpak
> mechanism, I'd like to suggest that the most straightforward fix for the
> overall system would be to expose the external fonts using the external
> path names -- bind mounting the external /usr/share/fonts as
> /usr/share/fonts within the flatpak, and creating a new
> /usr/share/fonts-minimal (or whatever) to hold the fonts provided by the
> flatpak itself. With this change, we can simply delete the UUID code
> from fontconfig and go back to using global font paths as keys to the
> font cache database.

I'm willing to make *some* changes to flatpak, but I'm not sure this
is the right approach. First of all it just looks at a tiny subset of
the problem (only flatpak, and only one directory). Secondly, it is
likely to run into issues having non-standard paths. For example, the
fedora flatpak runtime is created from the standard fedora rpms, so it
will have to be tweaked post install, and its possible that some code
hard-coded /usr/share/fonts/some-specific-font which is in the app but
not the host..

Also, we'll be guaranteeing that caches for /usr/share/fonts and
/usr/share/fonts-minimal don't conflict, but there is no guarantee
that different versions of /usr/share/fonts-minimal don't conflict. In
flatpak we set XDG_CACHE_HOME separate for each app, so this doesn't
happen cross-apps like it would with e.g. docker. However it will
cause flatpak to fail to detect an update of /usr/share/fonts-minimal
due to the mtime issue. I can imagine changing flatpak to modify the
mtime of the fonts directory after install (since only files are
hardlink-shared between apps, not directories), but this would cause
problems for all outstanding flatpak deployments that don't do this.

As a side point, flatpak also maps /run/host/user-fonts, which points
to the ~/.fonts or ~/.local/share/fonts directory even when the app
doesn't have regular homedir access. It would similarly have to be
changed to use the real path if the above approach is used.

> I'd love to hear about alternative ideas which might lead to solutions
> that make builds involving fontconfig reproducible. I'd be happy to take
> even vague hints at this point; all I've got at this point are a
> collection of dead ends.

Here is my proposal:

Make the uuid *generation* optional and manual. Then, when we create
the flatpak runtime we run fc-cache --make-uuid (or something) to
generate the uuid files. Then fontconfig would never confuse the
sandboxed /usr/share/fonts with any other, and since we would get a
new uuid each time we regenerated the runtime it would correctly pick
up stale caches when we update the runtime (even with no mtime
change).

This would make the default installation of fontconfig reproducible,
and it would solve the first problem (don't mix up sandboxed and host
font dirs). It would also let you opt-in to the uuid in other cases
where it makes sense. For instance, you could have a uuid file on a
NFS share or USB drive font dir, so that any caches for it will always
be the same no matter how it happens to be mounted.

We still wouldn't have a way to reuse host caches which were mounted
in a different way, but if we assume all conflicting directories use
uuids (like they would in the flatpak case), then we could solve this
in a pretty simple way by a config file saying "treat all instances of
/run/host/fonts as /usr/share/fonts", and I could make flatpak
generate such a file.

[Keith Packard]

Akira TAGOH <ak...@tagoh.org> writes:

> Indeed, that would be able to accomplish both with the minimal efforts
> for us at least. though they might came up with this but they didn't
> do it that way. so there might be some reason why they didn't do so.
> packaging issue perhaps?

flatpak appears to change many pathnames used to access host files. I
don't know if there are other subsystems affected by these changes,
perhaps we'll see more in the future though.

> I can't figure out completely but, fontconfig may needs to deal with
> different namespaces in a cache filename to avoid a collision between
> host and sandboxes. dunno if we may see different state in the future
> but it might be represented as a depth in a filename to make it
> different like 0:<md5>-le64.cache-<version> for host and
> 1:<md5>-le64.cache-<version> for a sandbox.
>
> we could increase a depth
> for a child in sandbox as needed, anyway.
> We can mix up caches that is located at the same place then. the last
> missing piece would be to map them to the right place. flatpaks should
> knows where they mounted directories to. they can create a map table
> with proper parent depth and current depth I think.

That would require customizing the contents of a flatpak on install, or
perhaps this could be done when the flatpak was run?

> I may be missing something so this might not work though...

This seems to extend the change I proposed; which provides an
indirection between the actual filename and the font config cache
database key. Instead of just mapping sandbox names to external names
(which results in collisions), we also add some 'salt' to the sandbox
names to perturb the generated key for internal paths.

Let's look at some examples:

sandbox prefix cache prefix
-------------- ----------
/run/host/fonts /usr/share/fonts
/run/host/user-fonts /home/keithp/.fonts
/ sandbox-depth-1/

'sandbox-depth-1' is the "salt" added to the cache keys for paths within
the sandbox to ensure they do not collide with cache keys for paths
outside of the sandbox.

You'd add all mounted file systems to this list so that fonts found
anywhere outside the sandbox would generate cache keys using the names
from outside the sandbox. If you ran another sandbox *inside* this
sandbox, you'd have another level of indirection:

/run/host/fonts sandbox-depth-1/usr/share/fonts
/ sandbox-depth-1/

(assuming that the sandbox didn't manage to mount the "real" system
fonts inside the sandbox somewhere).

This configuration file would be generated by flatpak at runtime.

--
-keith

[Keith Packard]

Alexander Larsson <alexande...@gmail.com> writes:

> I'd like to repeat that this is not really flatpak specific as such.
> The issue can happen in multiple cases like nfs mounts, multi-boot
> systems, docker containers, etc.

Sure, any place where path names are not the same would cause the same
issue. I think your other examples are unlikely to exhibit this in
practice though -- when NFS is used for system-level sharing, the normal
configuration tooling goes to extreme measures to ensure that pathnames
are the same across all systems.

Given that no-one has ever complained about this issue until flatpak
came along, I'd like to suggest that flatpak is somewhat unique in its
requirements here. Which means that we should feel free to find
solutions involving changes within flatpak, instead of attempting to fix
this solely within fontconfig. I'm not yet certain that we couldn't come
up with a fix that didn't affect flatpak, but we've got some strong
indications that this isn't possible.

> Flatpak has an additional weakness here, which is that we don't store
> mtimes (to maximize content sharing abilities the mtime is not part of
> the content addressing). This means the mtime can't be used to detect
> a stale cache, so we use the uuid to detect such changes.

I don't understand this -- the only way UUID could be used to trigger a
stale cache is by being missing. Otherwise, I thought mtimes were still
the only method used to know if the cache file was out of date?

> I think this misrepresents what the UUID is for. The UUID represents
> a uniqe identifier for the *location*, not the contents, and the goal
> is to make it independent on how you found the directory. If you add
> fonts to that directory, then you're supposed to keep the UUID because
> you want to regenerate the same cache for the new content.

I may have mis-represented the UUID idea in my email, but please accept
that I do understand that they are designed to be a path-independent
identifier for each directory.

The problem is not the UUID file itself, but in how the key it contains
is generated. To generate a reproducible UUID file, you have to have a
deterministic algorithm using only data in the source files and
toolchain along with the font directory path name. But that means that
/usr/share/fonts will necessarily have the same key on all systems.

So we end with conflicting requirements -- reproducibility requires that
the font cache key for '/usr/share/fonts' be the same everywhere, but
flatpak requires that they differ so that the host /usr/share/fonts have
a different cache key than the flatpak /usr/share/fonts.

> I realize this is not what the reproducible builds project wants, but
> it is what the UUID was added for.

The UUID files also violate one of the principle design goals I adopted
for fontconfig when cache files were moved to a separate directory --
never touch the font directories.

I also fought with fontconfig for about a week when the release
including them was installed on my machine as firefox would spin
whenever it found a directory with no fonts. At the time, I felt injured
by this change.

I add this here just to let you know that I am biased against any design
including UUID files and would prefer a solution which eliminates them.

> If the UUID really *was* content addressed, then it would change each
> time some font was added, and old font caches would become stale (and
> reaped via some other way like mtimes). In this case the fact that
> caches between the sandbox and the hosts collide is not even a
> problem, since the cached data is identical and could be shared. The
> problem is rather that the font directory is mutable, and if it
> changes without immediately updating the uuid you run into issues.

Hrm. Could some combination of mtime checking and content addressing
work? Consider a .uuid file generated from a hash of the directory
contents. If the .uuid file is older than the directory, you could
regenerate it reasonably quickly by hashing the directory contents. That
would be faster than re-scanning all of the fonts in the directory at
least.

I think this would assume that font file names were globally unique,
which may not be reasonable.

It also ignores the question of reaping stale cache files; presumably
fc-cache could do that somehow...

> I'm willing to make *some* changes to flatpak, but I'm not sure this
> is the right approach. First of all it just looks at a tiny subset of
> the problem (only flatpak, and only one directory).

Changing flatpak to *always* mount host directories at the same place in
the container should solve the problem for all directories.

> Secondly, it is likely to run into issues having non-standard
> paths. For example, the fedora flatpak runtime is created from the
> standard fedora rpms, so it will have to be tweaked post install, and
> its possible that some code hard-coded
> /usr/share/fonts/some-specific-font which is in the app but not the
> host..

This is not an entirely theoretical issue -- I've run into the
ruby-prawn-icon package in the last week. That ships a selection of
fonts and accesses them via hard-coded paths. "Fortunately", the default
location for those is not within /usr/share/fonts, but I could be
convinced that these fonts should be installed in /usr/share/fonts to
match distribution policy/conventions.

I know -- 'union mounts' to the rescue! (not a serious suggestion)

> Also, we'll be guaranteeing that caches for /usr/share/fonts and
> /usr/share/fonts-minimal don't conflict, but there is no guarantee
> that different versions of /usr/share/fonts-minimal don't conflict.

I don't understand this -- the cache for /usr/share/fonts-minimalive inside the flatpak environment, and should be per-flatpak?

> Here is my proposal:
>
> Make the uuid *generation* optional and manual. Then, when we create
> the flatpak runtime we run fc-cache --make-uuid (or something) to
> generate the uuid files. Then fontconfig would never confuse the
> sandboxed /usr/share/fonts with any other, and since we would get a
> new uuid each time we regenerated the runtime it would correctly pick
> up stale caches when we update the runtime (even with no mtime
> change).

Hrm. This is a tempting solution -- normal users would never see .uuid
files at all.

However, it means that new directories created within the flatpak while
the system is running would not get .uuid files and might then have
cache names which collide with the outer system.

How about making it a font configuration per-'dir' option instead? This
way, uuid files would be automatically added to all 'internal'
directories and never to external ones.

And users could add this when adding references to external font
repositories.

> This would make the default installation of fontconfig reproducible,
> and it would solve the first problem (don't mix up sandboxed and host
> font dirs). It would also let you opt-in to the uuid in other cases
> where it makes sense. For instance, you could have a uuid file on a
> NFS share or USB drive font dir, so that any caches for it will always
> be the same no matter how it happens to be mounted.

It sounds like a good direction for discussion at least.

> We still wouldn't have a way to reuse host caches which were mounted
> in a different way, but if we assume all conflicting directories use
> uuids (like they would in the flatpak case), then we could solve this
> in a pretty simple way by a config file saying "treat all instances of
> /run/host/fonts as /usr/share/fonts", and I could make flatpak
> generate such a file.

I've already got a patch series which solves this problem -- you can map
paths to cache keys on a per-'dir' element basis.

Here's an alternative proposal:

Add a per 'dir' element 'salt' value, which is stirred into the
path name when generating the cache key. You'd generate this
randomly when the flatpak was created so that all cache keys
would not collide with entries using a different (or absent)
salt value.

With this, and my path->key mapping series, we would be able to access
the existing cache files for external fonts (via the mapping mechanism), as
well as avoid collisions between internal and external font paths within
the cache. And we wouldn't have .uuid files (see above).

I still don't understand how UUID files help with the missing mtime
issue though; if you could explain that in a bit more detail, that would
help me, and perhaps expose a weakness with my alternative proposal.

--
-keith

[Alexander Larsson]

On Thu, Jan 10, 2019 at 9:41 PM Keith Packard <kei...@keithp.com> wrote:
>
> Alexander Larsson <alexande...@gmail.com> writes:
>
> > I'd like to repeat that this is not really flatpak specific as such.
> > The issue can happen in multiple cases like nfs mounts, multi-boot
> > systems, docker containers, etc.
>
> Sure, any place where path names are not the same would cause the same
> issue. I think your other examples are unlikely to exhibit this in
> practice though -- when NFS is used for system-level sharing, the normal
> configuration tooling goes to extreme measures to ensure that pathnames
> are the same across all systems.

Yeah, practice seems to indicate that this is not presently a problem.
But as the world moves more and more towards containerization in many
places and other tools like snappy also target the desktop it is imho
important to at least consider it.

> > Flatpak has an additional weakness here, which is that we don't store
> > mtimes (to maximize content sharing abilities the mtime is not part of
> > the content addressing). This means the mtime can't be used to detect
> > a stale cache, so we use the uuid to detect such changes.
>
> I don't understand this -- the only way UUID could be used to trigger a
> stale cache is by being missing. Otherwise, I thought mtimes were still
> the only method used to know if the cache file was out of date?

Lemme describe the expected mode of operation for fontconfig in a flatpak:

First of all, /usr comes from the runtime, and it is immutable. We
only ever modify the runtime by creating a new version of it in a
separate place and pointing new sandbox instances at the new
directory, and /usr is mounted read-only in the sandbox. Since all the
files in the runtime are maintained by ostree they all have a mtime of
0.

The runtime contains /usr/share/fonts with uuid files. These uuid
files are re-generated each time the runtime is rebuilt, so an updated
runtime will get a new uuid. It also contains a pre-generated font
caches for the fonts in the runtime based on the uuid.

In the sandbox $XDG_CACHE_HOME is set to ~/.var/app/$APPID/cache, and
this is the first writable cache directory in the fontconfig config,
so all caches will be per-app. However, if you update the app (say
from 1.0 to 2.0) the same cache dir will be used, just not between two
completely different apps).

In the normal case we will never need to generate any caches for the
runtime fonts. However, in weird cases (maybe someone pulled in a
fontconfig that has a different font cache format or whatever) the
cache generated will be based on the uuid, so if you update the
runtime you will get a new uuid and the old cache will not be used (if
we did we could not detect it was stale using the mtime).

Host fonts are mounted in /run/host/[user-]fonts, and host caches in
/run/host/[user-]fonts-cache, and the runtime has a config snippet to
add these. If the host has uuid for the dirs, then we can pick up the
host caches from the host cache dirs. If not, then the app will on the
first run re-generate the caches for /run/host/fonts and store it in
the per-app cache dir.

These directories are just bind-mounts of the host stuff, so normally
they have proper mtimes and we can detect stale caches fine, its just
slow the first time you start the app. However, on an ostree-using
base system (such as fedora silverblue) the mtime issue may actually
cause us to use stale caches here.

> > I realize this is not what the reproducible builds project wants, but
> > it is what the UUID was added for.
>
> The UUID files also violate one of the principle design goals I adopted
> for fontconfig when cache files were moved to a separate directory --
> never touch the font directories.

I think an optional approach fits with this. I.e. when it makes sense,
allow the font dirs to carry some identity, but don't touch them by
default.

> I also fought with fontconfig for about a week when the release
> including them was installed on my machine as firefox would spin
> whenever it found a directory with no fonts. At the time, I felt injured
> by this change.

Hmm, why was it doing that though? Doesn't seem like it would have to.

> Hrm. Could some combination of mtime checking and content addressing
> work? Consider a .uuid file generated from a hash of the directory
> contents. If the .uuid file is older than the directory, you could
> regenerate it reasonably quickly by hashing the directory contents. That
> would be faster than re-scanning all of the fonts in the directory at
> least.

You can't trust directory mtimes in this way. A file in the directory
can be updated without modifying the directory mtime. That is only
modified when you create or remove files.

> > I'm willing to make *some* changes to flatpak, but I'm not sure this
> > is the right approach. First of all it just looks at a tiny subset of
> > the problem (only flatpak, and only one directory).
>
> Changing flatpak to *always* mount host directories at the same place in
> the container should solve the problem for all directories.

There are two issues with this.

First of all, the runtime is like a chroot, and its supposed to look
like a "regular" system, but now certain paths are "reserved" and has
to be handled specially. I could accept this for a limited set of
paths though, although all existing flatpak runtimes would have to be
modified.

However, the second problem is that it puts demands on the *host*, as
it now has to match the layout of the runtimes so the pathnames can
match identically. The distro may store fonts in a different location,
and as long as you configure flatpak for your distro with
--with-system-fonts-dir= then the current setup will work. But if we
enforce it to /usr/share/fonts that will break. I'm not sure if this
is currently a practical issue for existing distros. I know distros
put the font caches in different places, but i'm not sure about fonts.
Maybe "weird" setups like nix will run into issues. You might know
this better.

> > Also, we'll be guaranteeing that caches for /usr/share/fonts and
> > /usr/share/fonts-minimal don't conflict, but there is no guarantee
> > that different versions of /usr/share/fonts-minimal don't conflict.
>

> I don't understand this -- the cache for /usr/share/fonts-minimal live inside the flatpak environment, and should be per-flatpak?

Yes, but if you update the runtime and /usr/share/fonts-minimal
changed in the new version (but has same mtime), then the stale cache
file in the users homedir will still be used.

> > Here is my proposal:
> >
> > Make the uuid *generation* optional and manual. Then, when we create
> > the flatpak runtime we run fc-cache --make-uuid (or something) to
> > generate the uuid files. Then fontconfig would never confuse the
> > sandboxed /usr/share/fonts with any other, and since we would get a
> > new uuid each time we regenerated the runtime it would correctly pick
> > up stale caches when we update the runtime (even with no mtime
> > change).
>
> Hrm. This is a tempting solution -- normal users would never see .uuid
> files at all.
>
> However, it means that new directories created within the flatpak while
> the system is running would not get .uuid files and might then have
> cache names which collide with the outer system.

In practice, this is not an issue, because the flatpak directories are
immutable.

> How about making it a font configuration per-'dir' option instead? This
> way, uuid files would be automatically added to all 'internal'
> directories and never to external ones.

Initially I imagined uuid files would work somewhat like this. I.e.
you put a .uuid file in the top /usr/share/fonts directory, and all
the subdirectories would hash based on uuid + relative path. This way
you don't have to have all these uuid files all over the place.
But, as you say we could do that but inject the uuid file from the
outside in a config file. Then all we have to do is rewrite such a
config file in flatpak.

However, flatpak will never parse the entire xml fontconfig file
format (which isn't even really stable over time), so such a config
would have to be external in a simpler config format.
For example, we could have a /etc/fonts/uuids file which is a simple list like:

/usr/share/fonts b81b806a-fb12-4a31-b458-181b1be0ec23

And then flatpak could read this and generate one for the sandbox that said:

/run/host/fonts b81b806a-fb12-4a31-b458-181b1be0ec23

This, in combination with the use of uuid + relative path would allow
the sandbox to use host caches.

However, the /etc/fonts/uuids file would still not be reproducible, so
i'm not sure this is better than using uuids for the sandboxed dirs
only, and then mapping the paths for picking up the host caches.

> > We still wouldn't have a way to reuse host caches which were mounted
> > in a different way, but if we assume all conflicting directories use
> > uuids (like they would in the flatpak case), then we could solve this
> > in a pretty simple way by a config file saying "treat all instances of
> > /run/host/fonts as /usr/share/fonts", and I could make flatpak
> > generate such a file.
>
> I've already got a patch series which solves this problem -- you can map
> paths to cache keys on a per-'dir' element basis.

Currently the runtime contains a static conf.d snippet that says:
<dir>/run/host/fonts</dir>
<dir>/run/host/user-fonts</dir>

We would have to turn that into a dynamic snippet. But that would be a
problem for pre-existing flatpak binaries which doesn't do this.
Could we instead have a way to modify a previously added dir element?
Or maybe it could handle duplicate <dir> tags such that they keep the
original order, but update the mapping? Then we can have both the
static snippet an a dynamic one for later flatpak versions.

> Here's an alternative proposal:
>
> Add a per 'dir' element 'salt' value, which is stirred into the
> path name when generating the cache key. You'd generate this
> randomly when the flatpak was created so that all cache keys
> would not collide with entries using a different (or absent)
> salt value.
>
> With this, and my path->key mapping series, we would be able to access
> the existing cache files for external fonts (via the mapping mechanism), as
> well as avoid collisions between internal and external font paths within
> the cache. And we wouldn't have .uuid files (see above).

I like this in theory. But i don't want to rewrite the entire runtime
font config xml file (its just to fragile). I much prefer if such a
salt could be added by just dropping a file somewhere. I.e. a
fonts.conf snippet that tweaked the salt of a previously defined dir
element.

So, counter proposal:

Flatpak generates at startup a file like this in /run/host/fontconf.xml

<cache-as path="/usr/share/fonts">/run/host/fonts></cache-as>
<cache-as path="/home/alex/.fonts">/run/host/user-fonts></cache-as>

In the runtime we create at build-time a /etc/fonts/conf.d/ file:

<salt id="randomdata">/usr/share/fonts</salt>
# Duplicate this with static versions for old flatpak versions
<cache-as path="/usr/share/fonts">/run/host/fonts></cache-as>
# This will (if it exists) override the above with the live values
<include ignore_missing="yes">/run/host/fontconf.xml</include>

We are basically in full control of existing flatpak runtimes, so this
could easily be added to all of them an propagated out. However, the
new flatpak version that creates the fontconf.xml file will not reach
distros in a while, so the above duplicates the "most likely to be
right" value for /usr/share/fonts for older flatpak versions to pick
up.

[Alexander Larsson]

On Fri, Jan 11, 2019 at 12:17 PM Akira TAGOH <ak...@tagoh.org> wrote:

>
> On Thu, Jan 10, 2019 at 7:54 PM Alexander Larsson
> <alexande...@gmail.com> wrote:
> > Here is my proposal:
> >
> > Make the uuid *generation* optional and manual. Then, when we create
> > the flatpak runtime we run fc-cache --make-uuid (or something) to
> > generate the uuid files. Then fontconfig would never confuse the
> > sandboxed /usr/share/fonts with any other, and since we would get a
> > new uuid each time we regenerated the runtime it would correctly pick
> > up stale caches when we update the runtime (even with no mtime
> > change).
>

> This would requires the root privilege to create uuid file on
> directories where root own though, are you going to have setuid to
> flatpak?
> or can this be done without it?

This would be done when creating the runtime, and at that time we have
write rights to /usr. Its not actually owned by root by that time
though, we build the image as some other user.

[Akira TAGOH]

On Thu, Jan 10, 2019 at 7:54 PM Alexander Larsson
<alexande...@gmail.com> wrote:

> Here is my proposal:
>
> Make the uuid *generation* optional and manual. Then, when we create
> the flatpak runtime we run fc-cache --make-uuid (or something) to
> generate the uuid files. Then fontconfig would never confuse the
> sandboxed /usr/share/fonts with any other, and since we would get a
> new uuid each time we regenerated the runtime it would correctly pick
> up stale caches when we update the runtime (even with no mtime
> change).

This would requires the root privilege to create uuid file on
directories where root own though, are you going to have setuid to
flatpak?
or can this be done without it?

>

> This would make the default installation of fontconfig reproducible,
> and it would solve the first problem (don't mix up sandboxed and host
> font dirs). It would also let you opt-in to the uuid in other cases
> where it makes sense. For instance, you could have a uuid file on a
> NFS share or USB drive font dir, so that any caches for it will always
> be the same no matter how it happens to be mounted.
>
> We still wouldn't have a way to reuse host caches which were mounted
> in a different way, but if we assume all conflicting directories use
> uuids (like they would in the flatpak case), then we could solve this
> in a pretty simple way by a config file saying "treat all instances of
> /run/host/fonts as /usr/share/fonts", and I could make flatpak
> generate such a file.

--
Akira TAGOH

[Akira TAGOH]

On Fri, Jan 11, 2019 at 7:06 PM Alexander Larsson
<alexande...@gmail.com> wrote:
> We would have to turn that into a dynamic snippet. But that would be a
> problem for pre-existing flatpak binaries which doesn't do this.
> Could we instead have a way to modify a previously added dir element?
> Or maybe it could handle duplicate <dir> tags such that they keep the
> original order, but update the mapping? Then we can have both the
> static snippet an a dynamic one for later flatpak versions.

There was an idea in the previous discussion to have a new element to
reset <dir> and split up <dir> into a file. that may helps you to have
an own desired font directories list instead of modifying entire
fonts.conf dynamically. this can be put into /etc/fonts/conf.d. so it
can be activated no matter how apps has own modified fonts.conf as
long as it contains a line to read config files from conf.d.
This would makes easier to have a map.

--
Akira TAGOH

[Keith Packard]

Alexander Larsson <alexande...@gmail.com> writes:

>> I also fought with fontconfig for about a week when the release
>> including them was installed on my machine as firefox would spin
>> whenever it found a directory with no fonts. At the time, I felt injured
>> by this change.
>
> Hmm, why was it doing that though? Doesn't seem like it would have to.

A .uuid file was added and removed to every directory in the font tree
which contained no fonts or sub-directories; (a) the directory mtime was
changed, causing the system to re-scan the fonts and then
re-create/re-delete the .uuid file (goto a). Somehow this would
eventually stablize (I'm not entirely sure how). It was a 'surprising'
problem that happened only once in a while and it took several days of
searching to locate as the only symptom was that firefox would hang for
'a while' and then start working. Once I ran strace on the process while
hung, it was pretty easy to track down, but having fontconfig affecting
directory mtimes was not what I expected...

> You can't trust directory mtimes in this way. A file in the directory
> can be updated without modifying the directory mtime. That is only
> modified when you create or remove files.

Fontconfig already assumes that all 'interesting' changes result in
directory mtime changes -- it doesn't expect files to be changed in
place. If you have a system doing this, fontconfig will fail.

> However, the second problem is that it puts demands on the *host*, as
> it now has to match the layout of the runtimes so the pathnames can
> match identically.

Well, I was thinking that the runtime would dynamically adapt to the
host environment and change the configuration inside to match the
host.

Is it that flatpak treats fonts very specially, or are there other bits
of host data injected into the flatpak runtime in a similar fashion?

Essentially, I was imagining the flatpak runtime system would discover
the set of fontconfig directories exposed by the host and inject those
into the runtime using matching paths. Right now, you inject two fixed
paths (/usr/share/fonts and ~/.fonts), but if we really want to deal
with systems that put fonts in other places, then presumably flatpak
will need to adapt anyways.

Once you've discovered where fonts are being stored in the host, having
those get mounted in a matching path in the runtime doesn't seem like a
huge step to me. It's a fairly simple matter of changing the mount
target path and injecting an xml fragment file into /etc/fonts/conf.d

> Maybe "weird" setups like nix will run into issues. You might know
> this better.

Good point; as new distros start experimenting with different filesystem
layouts, we will need to be more cautious about assuming fixed paths
of any kind.

> Yes, but if you update the runtime and /usr/share/fonts-minimal
> changed in the new version (but has same mtime), then the stale cache
> file in the users homedir will still be used.

Ah. Thanks for explaining this. It seems like the only way this could
happen is if the cache file within the flatpak was stale and a
replacement generated and written to the user's homedir (as the only
writable location available).

I think that's just a bug in the flatpak generation -- the cache file
within the flatpak should always be up-to-date as (I assume) the
fontconfig library provided in the flatpak would have been used to
generate that cache file. And, a future flatpak shipped without that bug
would have a correct cache file, which would presumably be used in
preference to the one stored in the home directory?

> Initially I imagined uuid files would work somewhat like this. I.e.
> you put a .uuid file in the top /usr/share/fonts directory, and all
> the subdirectories would hash based on uuid + relative path.

That sounds like the same as my 'salt' idea, except you're storing it in
the font directories rather than the font configuration.

> However, flatpak will never parse the entire xml fontconfig file
> format (which isn't even really stable over time), so such a config
> would have to be external in a simpler config format.

The fontconfig xml format is quite stable and is designed to be
manipulated by tools that do not understand the full contents, hence
using XML and providing a suitable DTD.

However, in practice it's become far easier to use XML snippets instead,
so let's figure out how to do that instead.

> For example, we could have a /etc/fonts/uuids file which is a simple list like:
>
> /usr/share/fonts b81b806a-fb12-4a31-b458-181b1be0ec23
>
> And then flatpak could read this and generate one for the sandbox that said:
>
> /run/host/fonts b81b806a-fb12-4a31-b458-181b1be0ec23

Yeah, moving the 'salt' out of the font directory and into the font
configuration seems like a good direction here. And placing it in a
separate file (although probably in XML format) seems like a better idea
than merging it into the existing <dir> elements.

> However, the /etc/fonts/uuids file would still not be reproducible, so
> i'm not sure this is better than using uuids for the sandboxed dirs
> only, and then mapping the paths for picking up the host caches.

Right, the 'host' installation cannot include any UUID values to
ensure that builds are reproducible. That doesn't invalidate this
approach though; we can add UUID values for any directories within the
sandbox and those will not collide with the host entries.

> Currently the runtime contains a static conf.d snippet that says:
> <dir>/run/host/fonts</dir>
> <dir>/run/host/user-fonts</dir>
>
> We would have to turn that into a dynamic snippet. But that would be a
> problem for pre-existing flatpak binaries which doesn't do this.

Any existing flatpaks will presumably include an existing fontconfig
which will not use font caches for the host which will not have .uuid
files in each directory anyways. As long as those existing flatpaks
"work" by re-generating the cache at first start, that seems fine to
me. Re-create those flatpaks with new fontconfig bits and things will
work better.

> Flatpak generates at startup a file like this in /run/host/fontconf.xml
>
> <cache-as path="/usr/share/fonts">/run/host/fonts></cache-as>
> <cache-as path="/home/alex/.fonts">/run/host/user-fonts></cache-as>
>
> In the runtime we create at build-time a /etc/fonts/conf.d/ file:
>
> <salt id="randomdata">/usr/share/fonts</salt>
> # Duplicate this with static versions for old flatpak versions
> <cache-as path="/usr/share/fonts">/run/host/fonts></cache-as>
> # This will (if it exists) override the above with the live values
> <include ignore_missing="yes">/run/host/fontconf.xml</include>

I think you'll want two separate files -- the salt is "constant" for the
flatpak, while the cache-as values depend on the host
environment. Otherwise, this looks good to me.

If we can convince you to mount the host font paths inside the runtime
where they had been in the host, then we don't even need that part. But
I can easily see where you'd end up with a pile of magic conditionals to
avoid having some critical part of the flatpak file system smashed by
fonts. Plus, we've already got that part of the solution mostly
implemented, and we all know how horrible it is to throw away code (just
kidding).

--
-keith

[Alexander Larsson]

On Fri, Jan 11, 2019 at 11:07 PM Keith Packard <kei...@keithp.com> wrote:
>
> Alexander Larsson <alexande...@gmail.com> writes:
>

> A .uuid file was added and removed to every directory in the font tree
> which contained no fonts or sub-directories; (a) the directory mtime was
> changed, causing the system to re-scan the fonts and then
> re-create/re-delete the .uuid file (goto a). Somehow this would
> eventually stablize (I'm not entirely sure how). It was a 'surprising'
> problem that happened only once in a while and it took several days of
> searching to locate as the only symptom was that firefox would hang for
> 'a while' and then start working. Once I ran strace on the process while
> hung, it was pretty easy to track down, but having fontconfig affecting
> directory mtimes was not what I expected...

Ugh. Sorry about that!

> > However, the second problem is that it puts demands on the *host*, as
> > it now has to match the layout of the runtimes so the pathnames can
> > match identically.
>
> Well, I was thinking that the runtime would dynamically adapt to the
> host environment and change the configuration inside to match the
> host.
>
> Is it that flatpak treats fonts very specially, or are there other bits
> of host data injected into the flatpak runtime in a similar fashion?

The point of any kind of container system is that you can have a
container filesystem that is guaranteed (to some 99% extent anyway) to
be the same wherever you run it. This way any kind of testing and
hardcoding you do on your system will be valid on the users system,
elimiting a whole form of bugs.

There are also technical issues with doing this. First of all, the
runtime (which is basically a directory that gets mounted on /usr have
no real requirements, its up to whoever create it to decide how and
what goes in there. So, say the host stores fonts in /usr/data/fonts
or whatever here might not be a corresponding directory in the runtime
to bind-mount on top of. And if there is there is no guarantee that
its not used for something else.

Basically, flatpak lets the runtime have full control of /usr, and
anything special (i.e. not direct mappings of directories the sandbox
is supposed to have access too) that we inject in the sandbox is in
/run.

There are a bunch of stuff we inject in the sandbox other than fonts,
but not a lot. First of all there are various unix domain sockets for
things like dbus, X11, etc. Then fonts, icon themes, timezone config,
resolv config, and the host /usr (as /run/host/usr) if the sandbox has
full filesystem access.

> Essentially, I was imagining the flatpak runtime system would discover
> the set of fontconfig directories exposed by the host and inject those
> into the runtime using matching paths. Right now, you inject two fixed
> paths (/usr/share/fonts and ~/.fonts), but if we really want to deal
> with systems that put fonts in other places, then presumably flatpak
> will need to adapt anyways.

They are fixed, but at flatpak build time, so a "weird" system could
configure flatpak to pick them up elsewhere, while exposing them in a
standardized location in the sandbox. True, this only allows two
directories, but I don't think this is really a huge issue in
practice.

> > Maybe "weird" setups like nix will run into issues. You might know
> > this better.
>
> Good point; as new distros start experimenting with different filesystem
> layouts, we will need to be more cautious about assuming fixed paths
> of any kind.

Same with experimenting with the layout of the runtime itself.

> > Yes, but if you update the runtime and /usr/share/fonts-minimal
> > changed in the new version (but has same mtime), then the stale cache
> > file in the users homedir will still be used.
>
> Ah. Thanks for explaining this. It seems like the only way this could
> happen is if the cache file within the flatpak was stale and a
> replacement generated and written to the user's homedir (as the only
> writable location available).

Yes:ish. It should not *normally* happen. But you may run into it in
uncommon situations like e.g. chrome using a statically linked version
of fontconfig that has a different fontconfig cache format.

> > However, flatpak will never parse the entire xml fontconfig file
> > format (which isn't even really stable over time), so such a config
> > would have to be external in a simpler config format.
>
> The fontconfig xml format is quite stable and is designed to be
> manipulated by tools that do not understand the full contents, hence
> using XML and providing a suitable DTD.

We recently ran into issues with fontconfig xml parsing errors in chrome
when using config files from a newer host fontconfig that were not
parsable by the statically linked chrome copy, so it is not perfect.

Additionally there is a question of forward compat. If flatpak was to
generate a "new" xml option that older fontconfigs were using, then we
would not support existing runtimes with older fontconfigs in it.
So, what we need is to be able to create snippets that get included by
the runtime config, added to the runtime at the same time as the
fontconfig in the runtime is updated to support it. These snippets can
be xml of course, but its different than parsing the entire fontconfig
xml config file in the runtime and understanding+modifying that.

> > Currently the runtime contains a static conf.d snippet that says:
> > <dir>/run/host/fonts</dir>
> > <dir>/run/host/user-fonts</dir>
> >
> > We would have to turn that into a dynamic snippet. But that would be a
> > problem for pre-existing flatpak binaries which doesn't do this.
>
> Any existing flatpaks will presumably include an existing fontconfig
> which will not use font caches for the host which will not have .uuid
> files in each directory anyways. As long as those existing flatpaks
> "work" by re-generating the cache at first start, that seems fine to
> me. Re-create those flatpaks with new fontconfig bits and things will
> work better.

I agree with this, but the point was that we can't just modify the
fontconfig to be dynamic always. We need to design it such that
whatever flatpak generates is optional for the runtime to pick up when
it is able to handle it.

> > Flatpak generates at startup a file like this in /run/host/fontconf.xml
> >
> > <cache-as path="/usr/share/fonts">/run/host/fonts></cache-as>
> > <cache-as path="/home/alex/.fonts">/run/host/user-fonts></cache-as>
> >
> > In the runtime we create at build-time a /etc/fonts/conf.d/ file:
> >
> > <salt id="randomdata">/usr/share/fonts</salt>
> > # Duplicate this with static versions for old flatpak versions
> > <cache-as path="/usr/share/fonts">/run/host/fonts></cache-as>
> > # This will (if it exists) override the above with the live values
> > <include ignore_missing="yes">/run/host/fontconf.xml</include>
>
> I think you'll want two separate files -- the salt is "constant" for the
> flatpak, while the cache-as values depend on the host
> environment. Otherwise, this looks good to me.

Yeah, there will be two files. One static in the runtime
(/etc/fonts/conf.d/50-flatpak.xml), and one generated by flatpak
(/run/host/fontconf.xml).

The idea would be that if you're using flatpak 1.0 (current stable)
then there will be no dynamic /run/host/fontconf.xml generated, while
in 1.2 (soon to be released) there will be one. So, to support 1.0
somewhat we hardcode the /run/host/fonts=/usr/share/fonts config in
the static file, and on >= 1.2 we override that in the dynamic to the
correct location. Then we still get to reuse the host caches in old
flatpaks in the "normal" font path case.

Of course, we could also rely on "old flatpk == old fontconfig",
meaning the host has uuid files... That might be enough to handle old
installations.

[Keith Packard]

Alexander Larsson <alexande...@gmail.com> writes:

> Ugh. Sorry about that!

Thanks. Bugs happen, fortunately I was able to track this one down and
fix it (we all love free software!)

> Yes:ish. It should not *normally* happen. But you may run into it in
> uncommon situations like e.g. chrome using a statically linked version
> of fontconfig that has a different fontconfig cache format.

Hrm. I don't get the sense that we've got a solution to this problem
yet. Given that the contents of the flatpak cannot change on the fly,
would it be sufficient to generate a new 'salt' value for the flatpak
each time it is changed? It might be sufficient to use the name of the
flatpak including a version as this salt (that has the advantage of
making the flatpak reproducible, which you probably want to encourage).

> I agree with this, but the point was that we can't just modify the
> fontconfig to be dynamic always. We need to design it such that
> whatever flatpak generates is optional for the runtime to pick up when
> it is able to handle it.

Sounds like being able to handle an arbitrary host fonts directory
mounted as /run/host/fonts will really help avoid future issues. And
that needs to be in the fontconfig used inside the flatpak, which means
we can't wait for a system which has fonts in a different place and plan
on fixing it in the host.

> Yeah, there will be two files. One static in the runtime
> (/etc/fonts/conf.d/50-flatpak.xml), and one generated by flatpak
> (/run/host/fontconf.xml).

Sounds like we've got a plan for this part -- fix my mapping code to use
config bits separate from the <dir> elements, then add a 'salt'
mechanism in the config bits that stirs in some random data when
generating the cache keys for specific directory trees.

Let's figure out how we should handle the stale flatpak fonts cache
issue. Once we've settled that, we can go implement the whole mess and
get a new fontconfig release made in time for debian freeze.

--
-keith

[Alexander Larsson]

On Sun, Jan 13, 2019 at 7:21 PM Keith Packard <kei...@keithp.com> wrote:
> > Yes:ish. It should not *normally* happen. But you may run into it in
> > uncommon situations like e.g. chrome using a statically linked version
> > of fontconfig that has a different fontconfig cache format.
>
> Hrm. I don't get the sense that we've got a solution to this problem
> yet. Given that the contents of the flatpak cannot change on the fly,
> would it be sufficient to generate a new 'salt' value for the flatpak
> each time it is changed? It might be sufficient to use the name of the
> flatpak including a version as this salt (that has the advantage of
> making the flatpak reproducible, which you probably want to encourage).

The plan was to generate a new random salt each build. We *do* in
general want to make flatpak builds as reproducible as possible,
because the magic of ostree means identical files are deduplicated on
disk and not downloaded on updates. However, if fixed set of small
files change each build that is not a huge problem.

So, in such a setup we're ok as long as the statically linked
fontconfig is new enough to respect the salt config.

However, we could run into issues if e.g. chrome statically links to
an older fontconfig. Such fonconfig versions would ignore the salt,
and miss the runtime-shipped caches, instead picking up the host
caches. It seems in that case the host cache would have a mismatching
mtime though, and we'd generate a new one, plus we'd regenerate the
host cache for the remapped dir. So this should work, at some cost in
performance.

> > Yeah, there will be two files. One static in the runtime
> > (/etc/fonts/conf.d/50-flatpak.xml), and one generated by flatpak
> > (/run/host/fontconf.xml).
>
> Sounds like we've got a plan for this part -- fix my mapping code to use
> config bits separate from the <dir> elements, then add a 'salt'
> mechanism in the config bits that stirs in some random data when
> generating the cache keys for specific directory trees.
>
> Let's figure out how we should handle the stale flatpak fonts cache
> issue. Once we've settled that, we can go implement the whole mess and
> get a new fontconfig release made in time for debian freeze.

Sounds good to me. Actually, i'm planning a flatpak stable (1.2)
release in the next few weeks, and I would like to add support for the
flatpak-side of this to the release. So, if we can just decide the
syntax for the dir remapping bits now I can add support for that. Then
we can do the runtime part when there is a fontconfig release that
supports it.

[Akira TAGOH]

On Sun, Jan 13, 2019 at 8:53 PM Alexander Larsson
<alexande...@gmail.com> wrote:
> Yes:ish. It should not *normally* happen. But you may run into it in
> uncommon situations like e.g. chrome using a statically linked version
> of fontconfig that has a different fontconfig cache format.
>
> > > However, flatpak will never parse the entire xml fontconfig file
> > > format (which isn't even really stable over time), so such a config
> > > would have to be external in a simpler config format.
> >
> > The fontconfig xml format is quite stable and is designed to be
> > manipulated by tools that do not understand the full contents, hence
> > using XML and providing a suitable DTD.
>
> We recently ran into issues with fontconfig xml parsing errors in chrome
> when using config files from a newer host fontconfig that were not
> parsable by the statically linked chrome copy, so it is not perfect.

Right. that is really a pain in the neck. not adding new syntax is
hard to improve and grow. that said, just ignoring unknown syntax
would makes harder to find an error. we may need to think about
measures for that like checking a version of config and library say,
but anyway.

--
Akira TAGOH

[Alexander Larsson]

On Sun, Jan 13, 2019 at 7:21 PM Keith Packard <kei...@keithp.com> wrote:
>
> Alexander Larsson <alexande...@gmail.com> writes:

> > Yeah, there will be two files. One static in the runtime
> > (/etc/fonts/conf.d/50-flatpak.xml), and one generated by flatpak
> > (/run/host/fontconf.xml).
>
> Sounds like we've got a plan for this part -- fix my mapping code to use
> config bits separate from the <dir> elements, then add a 'salt'
> mechanism in the config bits that stirs in some random data when
> generating the cache keys for specific directory trees.

Whats the status of this. Can we come up with an agreed upon format
for doing the dir remapping (outside of the dir nodes) soon? I'm about
to do a flatpak 1.2 release and would like to preempt things by
generating the dir mapping dynamic snippet so we can use it when we
update fontconfig in the runtime. (Note: We need not yet finalize the
salt stuff, because that will not be in the dynamic flatpak-generated
part of this.)

[Akira TAGOH]

Keith,

I'm fine with it. do you have a time to work on it? or already started working?
If not, please let me know.

--
Akira TAGOH

[Keith Packard]

Akira TAGOH <ak...@tagoh.org> writes:

> Keith,
>
> I'm fine with it. do you have a time to work on it? or already started working?
> If not, please let me know.

I've been at a conference all week, but hope to have time next
week. Would be happy to see others get this started, or collaborate in
any way. If I do get started, I'll be pushing stuff frequently and
posting mail so I don't block others.

Would be awesome to get this done in the next couple of weeks...

--
-keith

[Alexander Larsson]

On Wed, Jan 23, 2019 at 9:49 PM Keith Packard <kei...@keithp.com> wrote:
>
> Akira TAGOH <ak...@tagoh.org> writes:
>
> > Keith,
> >
> > I'm fine with it. do you have a time to work on it? or already started working?
> > If not, please let me know.
>
> I've been at a conference all week, but hope to have time next
> week. Would be happy to see others get this started, or collaborate in
> any way. If I do get started, I'll be pushing stuff frequently and
> posting mail so I don't block others.

So, what i really want to happen immediately (ideally this week) is to
decide on a format for the directory renaming configuration, because I
want to get that into the next stable flatpak release.
To expedite this I just made up some shit and made a flatpak PR for
generating that format: https://github.com/flatpak/flatpak/pull/2635

As an example, this generates on my system:

$ ./flatpak run --command=sh org.gnome.gedit
[📦 org.gnome.gedit flatpak]$ cat /run/host/font-dirs.xml
<?xml version="1.0"?>
<!DOCTYPE fontconfig SYSTEM "fonts.dtd">
<fontconfig>
<remap-dir real-path="/usr/share/fonts">/run/host/fonts</remap-dir>
<remap-dir real-path="/home/alex/.fonts">/run/host/user-fonts</remap-dir>
</fontconfig>

Is this format acceptable? Its mostly about naming the nodes and the
attributes, so its basically trivial. If you want i can rename things
or change orders, but I'd really just like an Ack on something.

[Keith Packard]

Alexander Larsson <alexande...@gmail.com> writes:

$ cat /run/host/font-dirs.xml
> <?xml version="1.0"?>
> <!DOCTYPE fontconfig SYSTEM "fonts.dtd">
> <fontconfig>
> <remap-dir real-path="/usr/share/fonts">/run/host/fonts</remap-dir>
> <remap-dir real-path="/home/alex/.fonts">/run/host/user-fonts</remap-dir>
> </fontconfig>
>
> Is this format acceptable? Its mostly about naming the nodes and the
> attributes, so its basically trivial. If you want i can rename things
> or change orders, but I'd really just like an Ack on something.

Format looks OK.

I think we might bike-shed on the names here a bit -- 'real-path' is
pretty ambiguous as both paths are 'real', one is just the file system
path and the other is the cache path. I like using the file system path
as the contents of the element and the cache path as the property, but
perhaps the property name could be 'cache-path' instead?

--
-keith

[Alexander Larsson]

Or maybe 'cache-as' ?

[Alexander Larsson]

On Thu, Jan 24, 2019 at 12:02 PM Alexander Larsson
<alexande...@gmail.com> wrote:

> Actually, i think cache-path fits better with the "remap-dir" node
> name. I'll change the PR.

https://github.com/flatpak/flatpak/pull/2635#issuecomment-457158368

[Alexander Larsson]

[Akira TAGOH]

On Thu, Jan 24, 2019 at 7:54 PM Keith Packard <kei...@keithp.com> wrote:
> I think we might bike-shed on the names here a bit -- 'real-path' is
> pretty ambiguous as both paths are 'real', one is just the file system
> path and the other is the cache path. I like using the file system path
> as the contents of the element and the cache path as the property, but
> perhaps the property name could be 'cache-path' instead?

Hmm, that looks not intuitive to me. in fact both are also a font
path. "cache path" are the sort of /var/cache/fontconfig or so. how
about "host-path"? although one can see it is remapping a path to
somewhere but still missing how this actually works in fontconfig.
i.e. to determine a cache filename to read.

[Akira TAGOH]

On Thu, Jan 24, 2019 at 8:20 PM Akira TAGOH <ak...@tagoh.org> wrote:
> Hmm, that looks not intuitive to me. in fact both are also a font
> path. "cache path" are the sort of /var/cache/fontconfig or so. how
> about "host-path"? although one can see it is remapping a path to
> somewhere but still missing how this actually works in fontconfig.
> i.e. to determine a cache filename to read.

Actually there are no way to guess how cache filenames are determined
without looking at source code. dir element is to add a font path. and
"remap-" dir is to remap a font path to somewhere. that is consistent
enough at this point. I revoke the last sentence where is starting
from "although".

--
Akira TAGOH

[Alexander Larsson]

I agree that cache path is wrong as we have something else by that name.

host-path kinda hardcodes the sandbox case for rewriting though. Maybe
"remapped-path"?

[Akira TAGOH]

On Thu, Jan 24, 2019 at 8:55 PM Alexander Larsson
<alexande...@gmail.com> wrote:
> I agree that cache path is wrong as we have something else by that name.
>
> host-path kinda hardcodes the sandbox case for rewriting though. Maybe
> "remapped-path"?

"remap" looks redundant to the element name. simply "to" or "as" as
you proposed that earlier?

--
Akira TAGOH

[Alexander Larsson]

as-path=... ?

[Akira TAGOH]

On Thu, Jan 24, 2019 at 9:43 PM Alexander Larsson

<alexande...@gmail.com> wrote:
>
> On Thu, Jan 24, 2019 at 1:18 PM Akira TAGOH <ak...@tagoh.org> wrote:
> >
> > On Thu, Jan 24, 2019 at 8:55 PM Alexander Larsson
> > <alexande...@gmail.com> wrote:
> > > I agree that cache path is wrong as we have something else by that name.
> > >
> > > host-path kinda hardcodes the sandbox case for rewriting though. Maybe
> > > "remapped-path"?
> >
> > "remap" looks redundant to the element name. simply "to" or "as" as
> > you proposed that earlier?
>
> as-path=... ?

That sounds good to me.

--
Akira TAGOH

[Alexander Larsson]

On Fri, Jan 25, 2019 at 5:07 AM Akira TAGOH <ak...@tagoh.org> wrote:
>
>
> >
> > as-path=... ?
>
> That sounds good to me.

This sounds good to me to. I updated the PR:
https://github.com/flatpak/flatpak/pull/2635#issuecomment-457482239

Can I get an ack on this format?

[Akira TAGOH]

Sure. I started to implement it based on Keith's branch.

--
Akira TAGOH

[Keith Packard]

Akira TAGOH <ak...@tagoh.org> writes:

> Sure. I started to implement it based on Keith's branch.

Awesome. I'll be back home "tomorrow" and able to spend some time on
this next week.

--
-keith

[Akira TAGOH]

Hi,

We are still missing a piece of a salt to deal with a directory name
separately where possibly have different fonts in sandbox etc. my tree
based on Keith's previous implementation works and passed test cases
except this salt thing:

https://gitlab.freedesktop.org/tagoh/fontconfig/commits/flatpak-rework

So have we got a consensus on letting flatpak provide a separate
config file contained a salt?

--
Akira TAGOH

[Keith Packard]

Akira TAGOH <ak...@tagoh.org> writes:

> Hi,
>
> We are still missing a piece of a salt to deal with a directory name
> separately where possibly have different fonts in sandbox etc. my tree
> based on Keith's previous implementation works and passed test cases
> except this salt thing:
>
> https://gitlab.freedesktop.org/tagoh/fontconfig/commits/flatpak-rework

Awesome, thanks for getting this going. I'm digging out from being away
from the office for a week...

> So have we got a consensus on letting flatpak provide a separate
> config file contained a salt?

Yes, that was the plan. Alexander suggested the following syntax:

<salt id="randomdata">/usr/share/fonts</salt>

I think this will work, although it seems a bit fragile. In particular,
if the host has salt for some directories, those will be defined
relative to the host paths, not the flatpak paths.

Do we need to process the 'salt' elements and 'remap-dir' elements in
order and remap old salt elements as remap-dir elements get loaded? That
also seems fragile to me.

Perhaps some command that the flatpak could run to generate host salt
values so that it could remap them into new salt elements using the
mapped paths?

Alternatively, we could just assume that only flatpak will use the salt
mechanism and leave this for a future enhancement?

--
-keith

[Akira TAGOH]

On Wed, Jan 30, 2019 at 5:02 AM Keith Packard <kei...@keithp.com> wrote:
> Yes, that was the plan. Alexander suggested the following syntax:
>
> <salt id="randomdata">/usr/share/fonts</salt>
>
> I think this will work, although it seems a bit fragile. In particular,
> if the host has salt for some directories, those will be defined
> relative to the host paths, not the flatpak paths.

Right. though this syntax looks primitive. it might manages to be done
if one writes everything for what they want. but if we can have easier
way - which can be done with minimal effort and inheritance for others
from host - that may be better.

> Do we need to process the 'salt' elements and 'remap-dir' elements in
> order and remap old salt elements as remap-dir elements get loaded? That
> also seems fragile to me.

Hm, to deal with more complicated cases, I guess we may need to have
one global salt to affect everything and a path-specific salt for
remapped path. for flatpak case, they want to have a global salt to
change a salt in sandbox (for /usr/share/fonts in sandbox etc) and set
salt from host in 'remap-dir' to build cache filenames on host (for
/run/host/fonts and so on).
This would avoid collision between one and origins. and assuming that
flatpaks can load config from host too, we could have:

10-salt.conf (from host):
<salt id="default"/>

50-flatpak.conf (sandbox specific):
<remap-dir as-path="/usr/share/fonts">/run/host/fonts</remap-dir>
<salt id="randomdata"/>
<dir>/usr/share/fonts</dir>

First salt element affects to 'remap-dir' and second one overrides it
for paths and change a salt in sandbox.
To make things easier, we may also want to export all of dir elements
from fonts.conf to the separate file. flatpak can replace it with
50-flatpak.conf in this case. or the file operation isn't desirable,
let's implement dir-reset element or something like that.

>
> Perhaps some command that the flatpak could run to generate host salt
> values so that it could remap them into new salt elements using the
> mapped paths?
>
> Alternatively, we could just assume that only flatpak will use the salt
> mechanism and leave this for a future enhancement?
>
> --
> -keith

--
Akira TAGOH

[Keith Packard]

Akira TAGOH <ak...@tagoh.org> writes:

> Hm, to deal with more complicated cases, I guess we may need to have
> one global salt to affect everything and a path-specific salt for
> remapped path.

Or, more likely, no salt at all for the outermost layer as it doesn't
really add anything here.

> for flatpak case, they want to have a global salt to
> change a salt in sandbox (for /usr/share/fonts in sandbox etc) and set
> salt from host in 'remap-dir' to build cache filenames on host (for
> /run/host/fonts and so on).

We may want a command line tool that extracts data from the config
for use by flatpak in building the dynamic configuration, including
things like salt values per directory. Yeah, that might be made to
work with flatpak essentially manually overriding the salt configuration
so that it uses the flatpak-relative names.

> This would avoid collision between one and origins. and assuming that
> flatpaks can load config from host too, we could have:
>
> 10-salt.conf (from host):
> <salt id="default"/>

I'd leave this out and not have salt in the host.

> 50-flatpak.conf (sandbox specific):
> <remap-dir as-path="/usr/share/fonts">/run/host/fonts</remap-dir>
> <salt id="randomdata"/>
> <dir>/usr/share/fonts</dir>

The salt here would need to have CDATA for the target directories, and I
think flatpack wants to split the dynamic from static config bits.

Dynamic (built at runtime):

<remap-dir as-path="/usr/share/fonts" salt="">/run/host/fonts</remap-dir>

Static (built in the flatpak):

<dir salt="salt">/usr/share/fonts</dir>

>
> First salt element affects to 'remap-dir' and second one overrides it
> for paths and change a salt in sandbox.

I think we can put that into the remap-dir element as both of those
are built at runtime?

> To make things easier, we may also want to export all of dir elements
> from fonts.conf to the separate file. flatpak can replace it with
> 50-flatpak.conf in this case. or the file operation isn't desirable,
> let's implement dir-reset element or something like that.

I think a dir-reset makes a lot of sense so that the flatpak can control
the set of font paths used. Building a command-line tool that flatpak
can use to discover the relevant fontconfig information seems like a
useful improvement; as I recall, flatpak is currently assuming
/usr/share/fonts and ~/.fonts are used on the host.

--
-keith

[Akira TAGOH]

On Thu, Jan 31, 2019 at 8:07 AM Keith Packard <kei...@keithp.com> wrote:
> We may want a command line tool that extracts data from the config
> for use by flatpak in building the dynamic configuration, including
> things like salt values per directory. Yeah, that might be made to
> work with flatpak essentially manually overriding the salt configuration
> so that it uses the flatpak-relative names.

Aha. yeah, that's a good idea.

> > This would avoid collision between one and origins. and assuming that
> > flatpaks can load config from host too, we could have:
> >
> > 10-salt.conf (from host):
> > <salt id="default"/>
>
> I'd leave this out and not have salt in the host.

Sure. though implicit thing may breaks something easily like own salt
affects 'remap-dir' unexpectedly. that should be documented carefully.
or should we have salt attribute in remap-dir and dir elements
instead? that would be obvious.

> The salt here would need to have CDATA for the target directories, and I
> think flatpack wants to split the dynamic from static config bits.
>
> Dynamic (built at runtime):
>
> <remap-dir as-path="/usr/share/fonts" salt="">/run/host/fonts</remap-dir>
>
> Static (built in the flatpak):
>
> <dir salt="salt">/usr/share/fonts</dir>

Yep, that looks good to me.

> I think a dir-reset makes a lot of sense so that the flatpak can control
> the set of font paths used. Building a command-line tool that flatpak
> can use to discover the relevant fontconfig information seems like a
> useful improvement; as I recall, flatpak is currently assuming
> /usr/share/fonts and ~/.fonts are used on the host.
>
> --
> -keith

--
Akira TAGOH

[Keith Packard]

Akira TAGOH <ak...@tagoh.org> writes:

> Yep, that looks good to me.

I don't time this week to hack on the code, and it looks like you're
doing great anyways; let me know if you need help in any way with the
implementation work.

--
-keith

[Alexander Larsson]

On Tue, Jan 29, 2019 at 9:02 PM Keith Packard <kei...@keithp.com> wrote:
>
> Akira TAGOH <ak...@tagoh.org> writes:

> Do we need to process the 'salt' elements and 'remap-dir' elements in
> order and remap old salt elements as remap-dir elements get loaded? That
> also seems fragile to me.
>
> Perhaps some command that the flatpak could run to generate host salt
> values so that it could remap them into new salt elements using the
> mapped paths?
>
> Alternatively, we could just assume that only flatpak will use the salt
> mechanism and leave this for a future enhancement?

I think in practice salt on the host-side of something like this will
only be there if we're using multiple levels of containers, which all
pass down the outermost fonts. I don't think this is going to be very
common at all, so maybe just ignoring it for now is fine?

[Alexander Larsson]

On Wed, Jan 30, 2019 at 10:33 AM Akira TAGOH <ak...@tagoh.org> wrote:
>
> On Wed, Jan 30, 2019 at 5:02 AM Keith Packard <kei...@keithp.com> wrote:
> > Yes, that was the plan. Alexander suggested the following syntax:
> >
> > <salt id="randomdata">/usr/share/fonts</salt>
> >
> > I think this will work, although it seems a bit fragile. In particular,
> > if the host has salt for some directories, those will be defined
> > relative to the host paths, not the flatpak paths.
>
> Right. though this syntax looks primitive. it might manages to be done
> if one writes everything for what they want. but if we can have easier
> way - which can be done with minimal effort and inheritance for others
> from host - that may be better.
>
> > Do we need to process the 'salt' elements and 'remap-dir' elements in
> > order and remap old salt elements as remap-dir elements get loaded? That
> > also seems fragile to me.
>
> Hm, to deal with more complicated cases, I guess we may need to have
> one global salt to affect everything and a path-specific salt for
> remapped path. for flatpak case, they want to have a global salt to
> change a salt in sandbox (for /usr/share/fonts in sandbox etc) and set
> salt from host in 'remap-dir' to build cache filenames on host (for
> /run/host/fonts and so on).
> This would avoid collision between one and origins. and assuming that
> flatpaks can load config from host too, we could have:

We don't want a global salt for everything in the container. In
reality things are more complicated than that. For example, an app may
bundle fonts, which will be in like /app/share/fonts, in addition to
the runtime fonts in /usr/share/fonts. These come from different
places and may individually be different in a different (or updated)
app, so the directories need to have different salts.

Also, it is quite possible that some host font directory is *not*
remapped, but still visible to the app. For example /opt/fonts for an
app that has filesystem access. If for whatever reason fontconfig
looks at this directory it should not apply any salt for it.

[Alexander Larsson]

On Thu, Jan 31, 2019 at 7:14 AM Akira TAGOH <ak...@tagoh.org> wrote:
>
> On Thu, Jan 31, 2019 at 8:07 AM Keith Packard <kei...@keithp.com> wrote:
>
> > > This would avoid collision between one and origins. and assuming that
> > > flatpaks can load config from host too, we could have:
> > >
> > > 10-salt.conf (from host):
> > > <salt id="default"/>
> >
> > I'd leave this out and not have salt in the host.
>
> Sure. though implicit thing may breaks something easily like own salt
> affects 'remap-dir' unexpectedly. that should be documented carefully.
> or should we have salt attribute in remap-dir and dir elements
> instead? that would be obvious.

As I said in an earlier email, it needs to be in the individual dir
elements, because a global salt is not right.

[Keith Packard]

Alexander Larsson <alexande...@gmail.com> writes:

> As I said in an earlier email, it needs to be in the individual dir
> elements, because a global salt is not right.

Do you want it in the <dir> elements directly? That would be more
straightforward in many ways and could avoid troubles with separate salt
declarations that take effect more broadly than one directory.

So, one file (generated at flatpak creation time) with

<dir salt="random bits">/usr/share/fonts</dir>
<dir>/run/host/fonts</dir>

and another (generated at runtime) with

<remap-dir as-path="/usr/share/fonts">/run/host/fonts</remap-dir>

Presumably you will mask all host configured font paths somehow? Maybe
you need to be able to inherit the 'salt' value from the host (if set)?
If so, we could have:

<remap-dir as-path="/usr/share/fonts" salt="host salt for /usr/share/fonts">/run/host/fonts</remap-dir>

--
-keith

[Keith Packard]

Alexander Larsson <alexande...@gmail.com> writes:

> We don't want a global salt for everything in the container.

I guess I wonder why not? Salt + dir inside the container will always be
unique. The place where you want to have different salt is for
directories mapped from the host; I think those will always be in
remap-dir clauses, if we have salt there, that should work?

> In
> reality things are more complicated than that. For example, an app may
> bundle fonts, which will be in like /app/share/fonts, in addition to
> the runtime fonts in /usr/share/fonts. These come from different
> places and may individually be different in a different (or updated)
> app, so the directories need to have different salts.

If building the flatpak generates the font caches, then per-flatpak salt
would make those correct.

> Also, it is quite possible that some host font directory is *not*
> remapped, but still visible to the app. For example /opt/fonts for an
> app that has filesystem access. If for whatever reason fontconfig
> looks at this directory it should not apply any salt for it.

Oh, so some host directories may be visible unmapped and unknown to the
flatpak? In that case, we'll need to enumerate all flatpak visible font
directories separately.

I think we need a complete enumeration of the cases; I keep seeing more
options...

--
-keith

[Alexander Larsson]

On Thu, Jan 31, 2019 at 9:40 AM Keith Packard <kei...@keithp.com> wrote:
>
> Alexander Larsson <alexande...@gmail.com> writes:
>
> > We don't want a global salt for everything in the container.
>
> I guess I wonder why not? Salt + dir inside the container will always be
> unique. The place where you want to have different salt is for
> directories mapped from the host; I think those will always be in
> remap-dir clauses, if we have salt there, that should work?

It will be unique per runtime, as the default salt comes from the
runtime generated file in the sandbox.
But, if multiple apps use the same runtime they will always use the
same salt for its /app/share/fonts, even though it could differ, which
seems risky.
But worse, if the runtime changes the default salt, then the caches
built into the app will be for the wrong salt until the app is
rebuilt.

> > In
> > reality things are more complicated than that. For example, an app may
> > bundle fonts, which will be in like /app/share/fonts, in addition to
> > the runtime fonts in /usr/share/fonts. These come from different
> > places and may individually be different in a different (or updated)
> > app, so the directories need to have different salts.
>
> If building the flatpak generates the font caches, then per-flatpak salt
> would make those correct.

Only as long as the runtime salt doesn't change. But its also a
robustness issue, if for whatever reason (cache version, etc) a cache
is generated for the app dir I want that to map to a uniqe cache id
rather than one shared between every app using that runtime.

> > Also, it is quite possible that some host font directory is *not*
> > remapped, but still visible to the app. For example /opt/fonts for an
> > app that has filesystem access. If for whatever reason fontconfig
> > looks at this directory it should not apply any salt for it.
>
> Oh, so some host directories may be visible unmapped and unknown to the
> flatpak? In that case, we'll need to enumerate all flatpak visible font
> directories separately.

What happens today is that if you host has fonts somewhere, say
/opt/fonts, and the app has --filesystem=host access, then /opt will
be visible as-is in the sandbox (along with essentially everything but
/usr and /app).

These directories are not (currently) added to the fontconfig setup in
the sandbox, so it will not be automatically picked up. But that
doesn't mean fontconfig in the sandbox can *never* see it. For
example, the app could call FcConfigAppFontAddDir() if it has a
per-app font dir option or something.

We *could* make flatpak enumerate *all* configured host font
directories and set up snippets for them. However, currently we don't,
and I don't see a huge need for this.

[Akira TAGOH]

Yeah, I agree with it. Having a salt in dir and remap-dir would
flexibly works I think.
Though, given that there are no salt in host, we just need to have a
salt for dirs inside sandbox only. so salt shouldn't be needed for
remap-dir in this case.

Also if host dirs are available as is on sandbox like Alex concerned,
we could simply have:

<dir>/opt/fonts</dir>

[Alexander Larsson]

On Thu, Jan 31, 2019 at 9:54 AM Akira TAGOH <ak...@tagoh.org> wrote:
>
> Also if host dirs are available as is on sandbox like Alex concerned,
> we could simply have:
>
> <dir>/opt/fonts</dir>

Yeah, assuming there is no global default salt that messes this up.

[Akira TAGOH]

I've done implementation for salt:
https://gitlab.freedesktop.org/tagoh/fontconfig/commits/flatpak-rework

That should works as expected.
For summary to support this on flatpak side, All of flatpaks needs to have:

<dir salt="something">/usr/share/fonts</dir>

and more if there are any fonts directories on sandbox where has same
directory names on host.
or may be better having a salt regardless of it to make them
differenciated to host perhaps. I'll leave it to you, Alex.

No salt should be needed for remap-dir so far.

Ah, and if you want to manage fonts directories in a single file and
don't want to modify fonts.conf, you can put <reset-dirs/> prior to
re-define own fonts directories with dir elements.

Please test it.

--
Akira TAGOH

[Keith Packard]

Alexander Larsson <alexande...@gmail.com> writes:

> Yeah, assuming there is no global default salt that messes this up.

It sounds like having 'salt' values in dir and remap-dir elements is what
we want then -- no need for separate salt elements.

--
-keith

[Akira TAGOH]

No comments on this (yet) so just opened a merge request here:
https://gitlab.freedesktop.org/fontconfig/fontconfig/merge_requests/30

If anyone can review, that would be appreciated.

--
Akira TAGOH

[Akira TAGOH]

Hi Alex,

Have you tried new implementation yet? I believe it should be
reflected our discussion though, I want to see some comment from you
because flatpak is only customer for salt thing so far. if it works, I
can commit it to master and make a release for them.

TIA,

--
Akira TAGOH

[Chris Lamb]

Chris Lamb wrote:

> [..]

Friendly ping on this?


Regards,

--
,''`.
: :' : Chris Lamb
`. `'` la...@debian.org / chris-lamb.co.uk
`-

[Johannes Schauer Marin Rodrigues]

Dear maintainers,

On Sun, 13 Sep 2020 08:13:06 -0000 "Chris Lamb" <la...@debian.org> wrote:
> Friendly ping on this?

I'd like to send another ping about this.

This bug is affecting my package mmdebstrap so I'd love to see it fixed.

Thanks!

cheers, josch

[Andres Pavez]

Hello,
I would like to send another gentle ping about this.

This bug is affecting the deployment upgrade of a current reproducible iso.

Thank you very much,
--
Andrés Pavez

[Johannes Schauer Marin Rodrigues]

Hi,

since this bug has not seen maintainer action for several months now and
because fontconfig is one of the last remaining bits that make a
Priority:Standard chroot unreproducible, I'd like to propose to NMU fontconfig
with the attached patch.

Essentially, I took the patch by Chris Lamb and modified it so that
uuid_generate_sha1() will be used if SOURCE_DATE_EPOCH is set. The original
behaviour is retained without SOURCE_DATE_EPOCH being set. This should be the
solution with zero impact on normal installations while making those
installations that care (like reproducible chroot or image creation)
reproducible.

If I don't get a NACK from you on this, I plan to upload to DELAYED/10 in two
weeks.

Thanks!

cheers, josch

[Johannes Schauer Marin Rodrigues]

Hi,

Quoting Johannes Schauer Marin Rodrigues (2021-12-28 09:38:51)

I just uploaded fontcontig to DELAYED/10 with the attached debdiff.

Thanks!

cheers, josch

[Roland Clobus]

Sorry about the noise,

The long mail was sent only to control@... and therefore invisible.

With kind regards,
Roland Clobus


-------- Forwarded Message --------
Subject: Improved patch
Date: Fri, 28 Jan 2022 18:08:28 +0100
From: Roland Clobus <rcl...@rclobus.nl>
To: con...@bugs.debian.org

reopen 864082 =
thanks

Hello maintainers,

Thanks for releasing a new version of fontconfig with the patch for the
reproducible cache files
(0001-Ensure-cache-checksums-are-determinstic.patch).

Unfortunately, I see side-effects of the applied patch (as shown in the
live-build images [1])

* Potential out-ouf-bounds-read issue: The function uuid_generate_sha1
is called with an incorrect second argument.

The second argument must be guaranteed to be of length 16 (or
longer), which is the size of a uuid.
E.g. /root/.fonts is only 12 bytes, which means that some random
bytes at the end of the string will be used for the sha1 sum.

The updated patch uses the null namespace as the basis for the sha1 sum.

... or should I have use one of the predefine namespaces instead?

* The patch adds new compiler warnings. I've added some casts to remove
compiler warnings

* There is a second scenario: initramfs with fonts:
plymouth-set-default-theme tribar --rebuild-initrd
or
update-initramfs -k all -u

The value for 'target' is contains a random part:
/var/tmp/mkinitramfs_ijJP8d//usr/share/fonts

This path is created by the plymouth hook in initramfs which uses
'fc-cache -s -y TEMPDIR'

The fonts in the ramdisk can be listed with:
zless /initrd.img | cpio --list --quiet | grep fontconfig | grep cache-7

For regular invocations of fc-cache, the '-y' argument is not used
and then 'target' and 'dir' are identical. The attached patch uses 'dir'
instead of 'target' and then the cache of the embedded fonts in the
ramdisk is reproducible as well.

Attached you'll find the patch that fixes all three issues mentioned above.

With kind regards,
Roland Clobus

[1] https://jenkins.debian.net/view/live/

[Johannes Schauer Marin Rodrigues]

Hi Roland,

Quoting Roland Clobus (2022-01-28 18:22:17)
> Sorry about the noise,

as far as I'm concerned, you made no noise but you made a very valuable
contribution instead. Thank you! :)

You completely rewrote the patch. I think it makes sense to replace Chris'
authorship with yours.

Chris, do you agree?

Would you be willing to submit an updated patch containing the name and email
of your choice and a commit message that explains your change? What you wrote
above is a good explanation I think.

Thanks!

cheers, josch

[Roland Clobus]

Hello Johannes,

On 28/01/2022 20:22, Johannes Schauer Marin Rodrigues wrote:
> Would you be willing to submit an updated patch containing the name and email
> of your choice and a commit message that explains your change? What you wrote
> above is a good explanation I think.

Attached is the updated patch.

A commit message would be:
<quote>
This patch changes the behaviour of the cache directory filename
calculation to be based on the "source" directory name, rather than
being entirely random if the SOURCE_DATE_EPOCH[1] environment variable
was determined to be present via getenv(3).

The two main scenarios are covered by this patch:
1) Invocation of 'fc-cache' as a postinst step
2) Invocation of 'update-initramfs' as a postinst step where the
initial ramdisk contains a font (e.g. when the plymouth hook calls
'fc-cache -s -y TEMPDIR')
</quote>

or much shorter:

<quote>
Generate deterministic cache directory filenames if SOURCE_DATE_EPOCH is
set.
</quote>