Bug#1029269: gio-2.0.pc references the wrong glib-compile-schemas again

[Helmut Grohne]

Package: libglib2.0-dev
Version: 2.74.4-1
File: /usr/lib/x86_64-linux-gnu/pkgconfig/gio-2.0.pc
X-Debbugs-Cc: sm...@debian.org
User: debian...@lists.debian.org
Usertags: ftcbfs
Control: affects -1 + src:ibus-unikey

Hi Simon et al,

gio-2.0.pc is broken again. This is a revival of #982213. Let me recap.

We figured that glib-compile-schemas is an architecture-independent
tool. Yet, glib installs it for multiple architectures (and I don't
quite recall why) with /usr/bin/glib-compile-schemas being a symlink to
the native one. Originally, gio-2.0.pc referenced this. Then, a Debian
patch made it reference the one from libglib2.0-0, which happened to
break cross compilation of stuff. Then I opened #982213 and we agreed to
revert that patch. Now the revert has been reverted, but it seems it was
reverted upstream. Can we revert that again? Would you be able to
provide more context here than I have?

Thanks in advance

Helmut

[Simon McVittie]

On Fri, 20 Jan 2023 at 13:10:49 +0100, Helmut Grohne wrote:
> We figured that glib-compile-schemas is an architecture-independent
> tool. Yet, glib installs it for multiple architectures (and I don't
> quite recall why) with /usr/bin/glib-compile-schemas being a symlink to
> the native one.

So that this is written down somewhere: this is because libglib2.0-0 needs
to run (some version of) g-c-s via its triggers. The naive implementation
of this would be to split out g-c-s into its own binary package, but
as well as being a bit of a waste of space (g-c-s is only 64K), that
would introduce a circular dependency, because glib-compile-schemas has
DT_NEEDED on libgio-2.0.so.0 and others:

for triggers
libglib2.0-0 ---------------> glib-compile-schemas
M-A: same <--------------- M-A: foreign
via dpkg-shlibdeps

Splitting up libglib2.0-0 into its separate SONAMEs with their
Policy-compliant package names would not help here, because it's
libgio-2.0.so.0 that makes most sense to "own" the trigger for schemas,
but glib-compile-schemas depends on libgio-2.0.so.0. So the same circular
dependency would exist for libgio-2.0-0.

> Originally, gio-2.0.pc referenced this. Then, a Debian
> patch made it reference the one from libglib2.0-0, which happened to
> break cross compilation of stuff. Then I opened #982213 and we agreed to
> revert that patch. Now the revert has been reverted, but it seems it was
> reverted upstream. Can we revert that again? Would you be able to
> provide more context here than I have?

The upstream change here was 0c087d12 "gio: Optionally install trigger
executables to architecture-specific paths", new in 2.73.3 and written
by me, in order to upstream some of our Debian-specific changes as a
solution to new upstream problems.

Because you have argued that glib-compile-schemas is an
architecture-independent interface, I think if other packages want
to run glib-compile-schemas *at build-time*, and they want to be
cross-compilable, then they ought to be looking for a build-architecture
tool, similar to the way dbus-binding-tool was traditionally handled.

In Meson this would be

dependency(
'gio-2.0',
native: true,
).get_variable(
pkgconfig: 'glib_compile_schemas',
default_value: 'glib-compile-schemas',
)

or maybe

find_program('glib-compile-schemas', native: true)

or some combination of the two (where "native: true" is the part that
gets you a build-architecture dependency lookup, instead of the default
host-architecture).

(One reason why this is annoying to set up is that the upstream build
system of a package that depends on glib-compile-schemas is trying to do
two orthogonal things: it's trying to support having a non-default version
of the tools in a non-default location outside PATH for jhbuild-like
use-cases (which are uninteresting for Debian, but upstreams want to
support this), and it's also trying to deal with cross-compilation. It's
difficult to get both right at the same time without having something like
Meson's dependency(native: true) in your build system.)

I don't think it's really reasonable to expect that querying a
variable from a host-architecture pkg-config file will give you a
build-architecture executable - how would that work in general, when the
host-architecture pkg-config file is common to all build architectures,
and generally specifies its tools by their absolute path? For instance, in
traditional (pre-multiarch) cross-compilation, each architecture of GLib
would have installed its own /usr/${tuple}/bin/glib-compile-schemas, and
querying the ARM pkg-config file would always give you an ARM executable,
never an x86 executable.

However, I'm aware that many build systems don't have as principled
a dividing line between build- and host-architecture dependencies
as Meson does; and for Debian specifically, given the organization
of our GLib packaging, it would be harmless to go back to having
glib_compile_schemas=${bindir}/glib-compile-schemas on all architectures,
which would have the side-effect of it not mattering which architecture
you ask for the appropriate path.

Because this is not a recent regression, I'd prefer to
let the recently-uploaded GLib 2.74.5 migrate to testing
before fixing this, but I agree it would be pragmatic to use
glib_compile_schemas=${bindir}/glib-compile-schemas in bookworm.

This would likely be unappealing upstream, particularly in the short
term, because it relies on the
${bindir}/glib-compile-schemas -> ${libdir}/glib-2.0/glib-compile-schemas
symlink that is created for the primary architecture, but the upstream
build system doesn't guarantee to create that symlink (in particular, it
doesn't do that if the version of Meson is too old for install_symlink()).

It might become upstreamable once their Meson requirement goes up, but
really I think it should be up to the GLib-dependent package (in this
case ibus-unikey) to ask for what it needs, which is a build-architecture
interface.

I also think that it would probably be better if this dependent package
was not running glib-compile-schemas during cross .deb builds at all.
There are two main use-cases for glib-compile-schemas:

- To summarize ${datadir}/glib-2.0/schemas/*.xml into
${datadir}/glib-2.0/schemas/gschemas.compiled.
For a shared directory like /usr/share or when installing into a
$DESTDIR, this is not an individual package's job - the individual
package does not "own" /usr/share/glib-2.0/schemas/gschemas.compiled,
should be letting GLib handle this via triggers, and certainly should
not be creating a $DESTDIR/usr/share/glib-2.0/schemas/gschemas.compiled
that will overwrite the one from the system.

- To summarize locally-provided schemas so that they can be used for
unit tests or similar, via GSETTINGS_SCHEMA_DIR or XDG_DATA_DIRS.
This is fine for native builds, but should probably not be done for
cross-builds, because in a "true" cross-build (without qemu-user or
some similar way to run cross-binaries), you are not going to be able
to run the tests anyway.

A cross-build with a DESTDIR doesn't seem like it should be doing either
of those?

smcv

[Helmut Grohne]

Hi Simon,

thank you very much for your quick and detailed answer! Let me point out
though that this is not urgent nor is there an expectation to fix this
in bookworm.

On Fri, Jan 20, 2023 at 05:01:44PM +0000, Simon McVittie wrote:
> So that this is written down somewhere: this is because libglib2.0-0 needs
> to run (some version of) g-c-s via its triggers. The naive implementation
> of this would be to split out g-c-s into its own binary package, but
> as well as being a bit of a waste of space (g-c-s is only 64K), that
> would introduce a circular dependency, because glib-compile-schemas has
> DT_NEEDED on libgio-2.0.so.0 and others:
>
> for triggers
> libglib2.0-0 ---------------> glib-compile-schemas
> M-A: same <--------------- M-A: foreign
> via dpkg-shlibdeps
>
> Splitting up libglib2.0-0 into its separate SONAMEs with their
> Policy-compliant package names would not help here, because it's
> libgio-2.0.so.0 that makes most sense to "own" the trigger for schemas,
> but glib-compile-schemas depends on libgio-2.0.so.0. So the same circular
> dependency would exist for libgio-2.0-0.

Is such a circular dependency actually a problem? We do have such cycles
in a few places. A popular example is libdevmapper1.02.1 <-> dmsetup.
apt can handle such cycles. It breaks the configure loop in one way or
another. Either it decides to configure libglib2.0-0 first. It would run
its postinst, then libglib2.0-bin's postinst (noop) and then invoke
triggers. All seems fine. Or it would configure in the other order and
run triggers last as well. In this case, it may run glib-compile-schemas
before having run ldconfig after unpacking libglib2.0-0. That is a
little slower, but should also work. I fail to see what could break
here.

> Because you have argued that glib-compile-schemas is an
> architecture-independent interface, I think if other packages want
> to run glib-compile-schemas *at build-time*, and they want to be
> cross-compilable, then they ought to be looking for a build-architecture
> tool, similar to the way dbus-binding-tool was traditionally handled.

Outside the meson ecosystem, dealing with two pkg-config is very rare.
When looking beyond Debian, the expectation typically is a different
one: When cross building, you are expected to build packages (such as
glib2.0) specifically for a combination of build and host architecture
such that the one .pc file references the mixture of tools. It is only
meson that tries to change this picture with limited success.

> or some combination of the two (where "native: true" is the part that
> gets you a build-architecture dependency lookup, instead of the default
> host-architecture).

This is difficult to achieve with autoconf and next to impossible to
achieve with CMake. As it happens, ibus-unikey uses CMake.

> I don't think it's really reasonable to expect that querying a
> variable from a host-architecture pkg-config file will give you a
> build-architecture executable - how would that work in general, when the
> host-architecture pkg-config file is common to all build architectures,
> and generally specifies its tools by their absolute path? For instance, in
> traditional (pre-multiarch) cross-compilation, each architecture of GLib
> would have installed its own /usr/${tuple}/bin/glib-compile-schemas, and
> querying the ARM pkg-config file would always give you an ARM executable,
> never an x86 executable.

The way this works in general (outside Debian) is that you build your
toolchain specifically for your combination of build and host
architecture. That's how basically all qt cross toolchains (except
Debian's work). Of course this approach produces O(n^2) toolchains for n
architectures and for Debian that's very much not what we want. But it
happens to be what many upstreams expect.

> However, I'm aware that many build systems don't have as principled
> a dividing line between build- and host-architecture dependencies
> as Meson does; and for Debian specifically, given the organization
> of our GLib packaging, it would be harmless to go back to having
> glib_compile_schemas=${bindir}/glib-compile-schemas on all architectures,
> which would have the side-effect of it not mattering which architecture
> you ask for the appropriate path.

Yeah. I don't think we have to upstream this change. Carrying such a
difference as a Debian patch (even post-build) would seem reasonable to
me.

> Because this is not a recent regression, I'd prefer to
> let the recently-uploaded GLib 2.74.5 migrate to testing
> before fixing this, but I agree it would be pragmatic to use
> glib_compile_schemas=${bindir}/glib-compile-schemas in bookworm.

There is little urgency from my point of view. I'm looking for a
sustainable long-term solution here.

> This would likely be unappealing upstream, particularly in the short
> term, because it relies on the
> ${bindir}/glib-compile-schemas -> ${libdir}/glib-2.0/glib-compile-schemas
> symlink that is created for the primary architecture, but the upstream
> build system doesn't guarantee to create that symlink (in particular, it
> doesn't do that if the version of Meson is too old for install_symlink()).

I concur.

> It might become upstreamable once their Meson requirement goes up, but
> really I think it should be up to the GLib-dependent package (in this
> case ibus-unikey) to ask for what it needs, which is a build-architecture
> interface.

Thus far CMake has completely refused to even recognize the concept of a
build architecture. I think you are asking for a major refactoring of
CMake or for lots of projects abandoning CMake entirely here. That
request looks infeasible to me.

> I also think that it would probably be better if this dependent package
> was not running glib-compile-schemas during cross .deb builds at all.
> There are two main use-cases for glib-compile-schemas:
>
> - To summarize ${datadir}/glib-2.0/schemas/*.xml into
> ${datadir}/glib-2.0/schemas/gschemas.compiled.
> For a shared directory like /usr/share or when installing into a
> $DESTDIR, this is not an individual package's job - the individual
> package does not "own" /usr/share/glib-2.0/schemas/gschemas.compiled,
> should be letting GLib handle this via triggers, and certainly should
> not be creating a $DESTDIR/usr/share/glib-2.0/schemas/gschemas.compiled
> that will overwrite the one from the system.
>
> - To summarize locally-provided schemas so that they can be used for
> unit tests or similar, via GSETTINGS_SCHEMA_DIR or XDG_DATA_DIRS.
> This is fine for native builds, but should probably not be done for
> cross-builds, because in a "true" cross-build (without qemu-user or
> some similar way to run cross-binaries), you are not going to be able
> to run the tests anyway.
>
> A cross-build with a DESTDIR doesn't seem like it should be doing either
> of those?

I had not questions the use of glib-compile-schemas. Thank you for doing
it. Your arguments look reasonable to me. As far as I can see,
ibus-unikey does not install a .compiled schema, so the call to
glib-compile-schemas is probably redundant and questionable. Your
request to disable such calls seems very reasonable to me. Would you
prefer me trying to generally remove calls to glib-compile-schemas over
changing glib2.0?

After all of this insightful conversation, I see three ways of moving
forward:

a. Accept the dependency cycle and make the per-architecture paths
reference the one in /usr/bin.
b. Change gio-2.0.pc to reference the /usr/bin one.
c. Change upstream project to avoid unnecessary calls to
glib-compile-schemas.

Do you concur that these are the options on the table? Which of them do
you prefer? I admit that c looks appealing as it means least work for
you and some work for me while a or b shifts that burden to you. ;)

Looking forward to your next reply.

Helmut

[Simon McVittie]

On Fri, 20 Jan 2023 at 20:45:06 +0100, Helmut Grohne wrote:
> I had not questions the use of glib-compile-schemas. Thank you for doing
> it. Your arguments look reasonable to me. As far as I can see,
> ibus-unikey does not install a .compiled schema, so the call to
> glib-compile-schemas is probably redundant and questionable.

ibus-unikey does install
/usr/share/glib-2.0/schemas/org.freedesktop.ibus.engine.unikey.gschema.xml,
so in the absence of triggers, yes it would have to run
glib-compile-schemas. The function of g-c-s is to collect all the
*.gschema.xml (and possibly some less common files describing enums),
and summarize them in a single mmap'able binary file that doesn't require
spending time, malloc/free cycles and readdir calls on parsing multiple
XML files at runtime.

However, in a well-integrated distribution, dpkg triggers or equivalent
should be responsible for doing that - not individual packages.

A pattern I've often seen in GNOME is to run glib-compile-schemas as a
post-install hook if and only if $DESTDIR is empty or unset, so that when
compiling a dpkg/rpm/etc. package with something like

./configure --prefix=/usr # or CMake/Meson equivalent
DESTDIR=$(pwd)/debian/tmp make install

we wouldn't run glib-compile-schemas (on the assumption that the package
manager will take responsibility), and would maybe show a warning "you
need to run glib-compile-schemas later" instead; but when doing a local
installation that is installed directly into place with something like

./configure --prefix=$HOME/jhbuild # or CMake/Meson equivalent
make install

or

./configure --prefix=/opt/gnome # or CMake/Meson equivalent
make
sudo make install

each package would re-run glib-compile-schemas for convenience. This
mirrors how ldconfig tends to work, and I think this would be the ideal
thing for ibus-unikey to do as well.

Looking at
https://sources.debian.org/src/ibus-unikey/0.7.0~beta1-1/cmake/GSettings.cmake/
it seems that the build should be specifying -DGSETTINGS_COMPILE=OFF in
debian/rules, as a way to let the dpkg triggers take responsibility for
doing this later.

The other common reason to run glib-compile-schemas is so that unit tests
can find schemas in a temporary ($builddir) location at build-time, and
I think this should only be done if the build-time tests are actually going
to be run. In Debian terms, looking at
https://sources.debian.org/src/ibus-unikey/0.7.0~beta1-1/cmake/GSettings.cmake/
I think the nocheck build profile should be specifying
-DGSETTINGS_COMPILE_IN_PLACE=OFF (or maybe this particular package should
just have -DGSETTINGS_COMPILE_IN_PLACE=OFF unconditionally, I don't know
whether it even has build-time tests that want the schemas).

> Your
> request to disable such calls seems very reasonable to me. Would you
> prefer me trying to generally remove calls to glib-compile-schemas over
> changing glib2.0?

In the medium to long term, yes I would.

> After all of this insightful conversation, I see three ways of moving
> forward:
>
> a. Accept the dependency cycle and make the per-architecture paths
> reference the one in /usr/bin.
> b. Change gio-2.0.pc to reference the /usr/bin one.
> c. Change upstream project to avoid unnecessary calls to
> glib-compile-schemas.
>
> Do you concur that these are the options on the table? Which of them do
> you prefer? I admit that c looks appealing as it means least work for
> you and some work for me while a or b shifts that burden to you. ;)

I'd be OK with doing (b) as Debian downstream delta in the short term if
it would help, but in the medium to long term, I think the right way is
(c) (combined with Meson "native: true" or equivalent tool discovery if
feasible, but I realise that's probably only viable in Meson-world).

> > for triggers
> > libglib2.0-0 ---------------> glib-compile-schemas
> > M-A: same <--------------- M-A: foreign
> > via dpkg-shlibdeps
>

> Is such a circular dependency actually a problem?

I would say yes if it can be avoided. Any time apt breaks a cycle, it's
doing something that's out-of-spec for the guarantees that we normally
use to reason about what is safe to assume.

Circular dependencies also make it harder to clean up unused packages,
because each package in the circle prevents the next one round the circle
from being diagnosed as unused...

> A popular example is libdevmapper1.02.1 <-> dmsetup

... and as it happens, I ran into this exact example last week while
cleaning up cruft that had accumulated in one of the development chroots I
use for my day job (I think it got there via a Recommends chain involving
cryptsetup and systemd, but removing those was not sufficient to hint
aptitude into also removing the dmsetup/libdevmapper cycle).

> > Because you have argued that glib-compile-schemas is an
> > architecture-independent interface, I think if other packages want
> > to run glib-compile-schemas *at build-time*, and they want to be
> > cross-compilable, then they ought to be looking for a build-architecture
> > tool, similar to the way dbus-binding-tool was traditionally handled.
>
> Outside the meson ecosystem, dealing with two pkg-config is very rare.
> When looking beyond Debian, the expectation typically is a different
> one: When cross building, you are expected to build packages (such as
> glib2.0) specifically for a combination of build and host architecture
> such that the one .pc file references the mixture of tools.

How would you go about building GLib like that? Let's say for the sake
of a concrete example that your cross-compilation is the popular "ARM on
x86" scenario (host=ARM, build=x86). I can't think of any way you could
configure GLib to produce a pkg-config file that lists an ARM shared
library and an x86 glib-compile-schemas, either in its current Meson
build system or in its older Autotools build - the only way I can see
would be to rely on either the Debian-specific separation of one source
package into multiple binary packages, or installing in just the right
order so that one build overwrites another.

If we were still in Autotools-world, the way I would always have done
this would be to force use of a specific x86 version of compile-time
executable tools, overriding what would ordinarily have been picked up
from pkg-config, like for example if glib-compile-schemas is the only
native tool you needed, you might invoke:

./configure --build=x86_64-etc --host=arm-etc \
PKG_CONFIG=arm-etc-pkg-config \
GLIB_COMPILE_SCHEMAS=/usr/bin/glib-compile-schemas

As long as the dependent package uses either AC_CHECK_PROG or
PKG_CHECK_VAR to find glib-compile-schemas, it's straightforward to do
that, because both macros always let you specify the "result variable"
as an AC_ARG_VAR configure parameter.

Or in CMake you'd have to hope the dependent project offers an appropriate
-D option, either to specify the right glib-compile-schemas or to turn
off unnecessary use of it.

smcv