Google Groups no longer supports new Usenet posts or subscriptions. Historical content remains viewable.
Dismiss
Wine MinGW system libraries
90 views
Skip to first unread message
Zebediah Figura
Sep 4, 2021, 9:40:07 PM9/4/21
to
Hello all,
I'm a contributor to the Wine project. To summarize the following mail,
Wine needs special versions of some of its normal dependencies, such as
libfreetype and libgnutls, built using the MinGW cross-compiler, and I'm
sending out a mail to major distributions in order to get some feedback
from our packagers on how these should be built and packaged.
For a long time Wine has built all of its Win32 libraries (DLLs and
EXEs) as ELF binaries. For various reasons related to application
compatibility, we have started building our binaries as PE instead,
using the MinGW cross-compiler. It is our intent to expand this to some
of our dependencies as well. The list of dependencies that we intend to
build using MinGW is not quite fixed yet, but we expect it to include
and be mostly limited to the following:
* libvkd3d
* libFAudio
* libgnutls
* zlib (currently included via manual source import)
* libmpg123
* libgsm
* libpng
* libjpeg-turbo
* libtiff
* libfreetype
* liblcms2
* jxrlib
and dependencies of the above packages (not including CRT dependencies,
which Wine provides).
There is currently some internal discussion about how these dependencies
should be built and linked. There are essentially three questions I see
that need to be resolved, and while these resolutions have a significant
impact on the Wine building and development process, they also have an
impact on distributions, and accordingly I'd like to get input from our
packagers to ensure that their considerations are accurately taken into
account.
(1) Should we build via source import, or link statically, or dynamically?
Source imports are dispreferred by Debian [1], on the grounds that they
cause duplication of libraries on disk and in memory, and make it harder
to handle security updates. They also make building and bisecting
harder. Static libraries don't seem to be expressly discouraged, but
share most of the same downsides (see also [2]).
Note however that if they are linked dynamically, we need to make sure
that we load our packages instead of MinGW builds of open-source
libraries with applications ship with. There's some internal discussion
about whether this is possible while using "stock" builds of MinGW
libraries, but, due to the way the Win32 loader works, we will probably
need to compile each library, and its dependencies, with a separate,
wine-specific name, e.g. "libwinefreetype-6.dll" and
"libwineharfbuzz.dll". For a detailed explantion see footnote [3]. Note
that all we actually need to change is the name; we don't need to patch
the source.
Accordingly, although static linking and source imports are generally
disprefered, it may quite likely be preferable in our case. We don't get
the benefits of on-disk deduplication, since Wine is essentially the
only piece of software which needs these libraries.
(2) If we use dynamic libraries, should dependencies be included in the
main wine package, or packaged separately?
This is mostly a question for packagers, although it also relates to (3).
I expect that Debian will want to answer "packaged separately" here, on
the grounds that this lets them update (say) Wine's libgnutls
separately, and in sync with ELF libgnutls, if some security fix is
needed. There is a snag, though: we need libraries to be copied into the
prefix (there's some internal effort to allow using something like
symlinks instead, but this hard and not done yet). Normally we perform
this copy every time Wine is updated, but if Wine and its dependencies
aren't updated on the same schedule, we may end up loading an old
version of a dependency in the prefix.
(3) If dependencies are packaged separately, should Wine build them as
part of its build tree (e.g. using submodules), or find and link
(statically or dynamically) to existing binaries?
Linking to existing binaries is generally preferable: it avoids
duplication on disk; it reduces compile times when compiling a single
package from source (especially the first time). However, we aren't
going to benefit from on-disk duplication. And, most importantly, unlike
with ELF dependencies, there is no standardized way to locate MinGW
libraries—especially if it comes to Wine-specific libraries. We would
need a way for Wine's configure script to find these packages—and
ideally find them automatically, or else fall back to a submodule-based
approach.
If we rely on distributions to provide our dependencies, the best idea I
have here would be something like a x86_64-w64-mingw32-pkg-config
(Fedora actually already ships this, but I think Fedora is the only
one). And if we use shared libraries rather than static, things get
worse: we need to know the exact path of each library and its
dependencies at runtime so that we can copy (or symlink) them into a
user's WINEPREFIX. For ELF this is the job of ld.so. For MinGW there is
no standardized install location across distributions.
For what it's worth, the current proposed solution (which has the
support of the Wine maintainer) involves source imports and submodules.
There's probably room for changing our approach even after things are
committed, but I'd still like to get early feedback from distributions,
and make sure that their interests are accurately represented, before we
commit.
ἔρρωσθε,
Zebediah
(she/her)
[1]
https://www.debian.org/doc/debian-policy/ch-source.html#embedded-code-copies
[2] https://wiki.debian.org/StaticLinking
[3] The basic problem is that applications can and often do ship with PE
builds of cross-platform libraries. These libraries can be ahead of
Wine's system libraries, behind them, or even built with custom patches.
Accordingly we really don't want to load "our" freetype in place of
"their" freetype, or "theirs" in place of "ours". But because of the way
the Win32 loader works, you can generally only have one DLL of a given
name loaded in a process, and further attempts to dlopen() [as it were]
"libfreetype-6.dll" will return the handle to the already loaded
library, potentially breaking either Wine or the application. There
*may* be ways we can hack around this internally, but it's not clear
that it's feasible yet, and so it's probably best to assume that we'll
need special builds of dynamic libraries.
I'm a contributor to the Wine project. To summarize the following mail,
Wine needs special versions of some of its normal dependencies, such as
libfreetype and libgnutls, built using the MinGW cross-compiler, and I'm
sending out a mail to major distributions in order to get some feedback
from our packagers on how these should be built and packaged.
For a long time Wine has built all of its Win32 libraries (DLLs and
EXEs) as ELF binaries. For various reasons related to application
compatibility, we have started building our binaries as PE instead,
using the MinGW cross-compiler. It is our intent to expand this to some
of our dependencies as well. The list of dependencies that we intend to
build using MinGW is not quite fixed yet, but we expect it to include
and be mostly limited to the following:
* libvkd3d
* libFAudio
* libgnutls
* zlib (currently included via manual source import)
* libmpg123
* libgsm
* libpng
* libjpeg-turbo
* libtiff
* libfreetype
* liblcms2
* jxrlib
and dependencies of the above packages (not including CRT dependencies,
which Wine provides).
There is currently some internal discussion about how these dependencies
should be built and linked. There are essentially three questions I see
that need to be resolved, and while these resolutions have a significant
impact on the Wine building and development process, they also have an
impact on distributions, and accordingly I'd like to get input from our
packagers to ensure that their considerations are accurately taken into
account.
(1) Should we build via source import, or link statically, or dynamically?
Source imports are dispreferred by Debian [1], on the grounds that they
cause duplication of libraries on disk and in memory, and make it harder
to handle security updates. They also make building and bisecting
harder. Static libraries don't seem to be expressly discouraged, but
share most of the same downsides (see also [2]).
Note however that if they are linked dynamically, we need to make sure
that we load our packages instead of MinGW builds of open-source
libraries with applications ship with. There's some internal discussion
about whether this is possible while using "stock" builds of MinGW
libraries, but, due to the way the Win32 loader works, we will probably
need to compile each library, and its dependencies, with a separate,
wine-specific name, e.g. "libwinefreetype-6.dll" and
"libwineharfbuzz.dll". For a detailed explantion see footnote [3]. Note
that all we actually need to change is the name; we don't need to patch
the source.
Accordingly, although static linking and source imports are generally
disprefered, it may quite likely be preferable in our case. We don't get
the benefits of on-disk deduplication, since Wine is essentially the
only piece of software which needs these libraries.
(2) If we use dynamic libraries, should dependencies be included in the
main wine package, or packaged separately?
This is mostly a question for packagers, although it also relates to (3).
I expect that Debian will want to answer "packaged separately" here, on
the grounds that this lets them update (say) Wine's libgnutls
separately, and in sync with ELF libgnutls, if some security fix is
needed. There is a snag, though: we need libraries to be copied into the
prefix (there's some internal effort to allow using something like
symlinks instead, but this hard and not done yet). Normally we perform
this copy every time Wine is updated, but if Wine and its dependencies
aren't updated on the same schedule, we may end up loading an old
version of a dependency in the prefix.
(3) If dependencies are packaged separately, should Wine build them as
part of its build tree (e.g. using submodules), or find and link
(statically or dynamically) to existing binaries?
Linking to existing binaries is generally preferable: it avoids
duplication on disk; it reduces compile times when compiling a single
package from source (especially the first time). However, we aren't
going to benefit from on-disk duplication. And, most importantly, unlike
with ELF dependencies, there is no standardized way to locate MinGW
libraries—especially if it comes to Wine-specific libraries. We would
need a way for Wine's configure script to find these packages—and
ideally find them automatically, or else fall back to a submodule-based
approach.
If we rely on distributions to provide our dependencies, the best idea I
have here would be something like a x86_64-w64-mingw32-pkg-config
(Fedora actually already ships this, but I think Fedora is the only
one). And if we use shared libraries rather than static, things get
worse: we need to know the exact path of each library and its
dependencies at runtime so that we can copy (or symlink) them into a
user's WINEPREFIX. For ELF this is the job of ld.so. For MinGW there is
no standardized install location across distributions.
For what it's worth, the current proposed solution (which has the
support of the Wine maintainer) involves source imports and submodules.
There's probably room for changing our approach even after things are
committed, but I'd still like to get early feedback from distributions,
and make sure that their interests are accurately represented, before we
commit.
ἔρρωσθε,
Zebediah
(she/her)
[1]
https://www.debian.org/doc/debian-policy/ch-source.html#embedded-code-copies
[2] https://wiki.debian.org/StaticLinking
[3] The basic problem is that applications can and often do ship with PE
builds of cross-platform libraries. These libraries can be ahead of
Wine's system libraries, behind them, or even built with custom patches.
Accordingly we really don't want to load "our" freetype in place of
"their" freetype, or "theirs" in place of "ours". But because of the way
the Win32 loader works, you can generally only have one DLL of a given
name loaded in a process, and further attempts to dlopen() [as it were]
"libfreetype-6.dll" will return the handle to the already loaded
library, potentially breaking either Wine or the application. There
*may* be ways we can hack around this internally, but it's not clear
that it's feasible yet, and so it's probably best to assume that we'll
need special builds of dynamic libraries.
Bastien ROUCARIES
Sep 5, 2021, 3:20:03 AM9/5/21
to
Improve dpkg to support partial arch. I volonteer to implement none arch but i am waiting from guillem here.
Partial arch is the solution hère.
I can help also here
Stephen Kitt
Sep 5, 2021, 12:20:04 PM9/5/21
to
Hi Zebediah,
On Sat, 4 Sep 2021 20:17:53 -0500, Zebediah Figura <zfi...@codeweavers.com>
wrote:
Assuming Debian provides the dependencies (which is currently true only for
zlib), this could be handled in packaging by providing symlinks, couldn’t it?
Not in the Wine prefixes, but elsewhere.
(The Wine team also maintains libvkd3d and libFaudio, so we can take care of
those at least.)
> Accordingly, although static linking and source imports are generally
> disprefered, it may quite likely be preferable in our case. We don't get
> the benefits of on-disk deduplication, since Wine is essentially the
> only piece of software which needs these libraries.
>
> (2) If we use dynamic libraries, should dependencies be included in the
> main wine package, or packaged separately?
>
> This is mostly a question for packagers, although it also relates to (3).
>
> I expect that Debian will want to answer "packaged separately" here, on
> the grounds that this lets them update (say) Wine's libgnutls
> separately, and in sync with ELF libgnutls, if some security fix is
> needed. There is a snag, though: we need libraries to be copied into the
> prefix (there's some internal effort to allow using something like
> symlinks instead, but this hard and not done yet). Normally we perform
> this copy every time Wine is updated, but if Wine and its dependencies
> aren't updated on the same schedule, we may end up loading an old
> version of a dependency in the prefix.
Debian packaging doesn’t touch anything in users’ home directories, so this
would have to be handled in Wine itself, perhaps in a similar fashion to
existing provisions for Gecko and Mono.
> (3) If dependencies are packaged separately, should Wine build them as
> part of its build tree (e.g. using submodules), or find and link
> (statically or dynamically) to existing binaries?
>
> Linking to existing binaries is generally preferable: it avoids
> duplication on disk; it reduces compile times when compiling a single
> package from source (especially the first time). However, we aren't
> going to benefit from on-disk duplication. And, most importantly, unlike
> with ELF dependencies, there is no standardized way to locate MinGW
> libraries—especially if it comes to Wine-specific libraries. We would
> need a way for Wine's configure script to find these packages—and
> ideally find them automatically, or else fall back to a submodule-based
> approach.
>
> If we rely on distributions to provide our dependencies, the best idea I
> have here would be something like a x86_64-w64-mingw32-pkg-config
> (Fedora actually already ships this, but I think Fedora is the only
> one). And if we use shared libraries rather than static, things get
> worse: we need to know the exact path of each library and its
> dependencies at runtime so that we can copy (or symlink) them into a
> user's WINEPREFIX. For ELF this is the job of ld.so. For MinGW there is
> no standardized install location across distributions.
This also works in Debian:
$ sudo apt install libz-mingw-w64-dev mingw-w64-tools
$ x86_64-w64-mingw32-pkg-config --libs zlib
-L/usr/x86_64-w64-mingw32/lib -lz
> For what it's worth, the current proposed solution (which has the
> support of the Wine maintainer) involves source imports and submodules.
> There's probably room for changing our approach even after things are
> committed, but I'd still like to get early feedback from distributions,
> and make sure that their interests are accurately represented, before we
> commit.
Realistically, I think this is the best approach for now. As Debian adds
support for PE libraries, we can replace the source imports in the Wine
source code; this is done in many other Debian packages for projects which
vendor dependencies.
Regards,
Stephen
On Sat, 4 Sep 2021 20:17:53 -0500, Zebediah Figura <zfi...@codeweavers.com>
wrote:
zlib), this could be handled in packaging by providing symlinks, couldn’t it?
Not in the Wine prefixes, but elsewhere.
(The Wine team also maintains libvkd3d and libFaudio, so we can take care of
those at least.)
> Accordingly, although static linking and source imports are generally
> disprefered, it may quite likely be preferable in our case. We don't get
> the benefits of on-disk deduplication, since Wine is essentially the
> only piece of software which needs these libraries.
>
> (2) If we use dynamic libraries, should dependencies be included in the
> main wine package, or packaged separately?
>
> This is mostly a question for packagers, although it also relates to (3).
>
> I expect that Debian will want to answer "packaged separately" here, on
> the grounds that this lets them update (say) Wine's libgnutls
> separately, and in sync with ELF libgnutls, if some security fix is
> needed. There is a snag, though: we need libraries to be copied into the
> prefix (there's some internal effort to allow using something like
> symlinks instead, but this hard and not done yet). Normally we perform
> this copy every time Wine is updated, but if Wine and its dependencies
> aren't updated on the same schedule, we may end up loading an old
> version of a dependency in the prefix.
would have to be handled in Wine itself, perhaps in a similar fashion to
existing provisions for Gecko and Mono.
> (3) If dependencies are packaged separately, should Wine build them as
> part of its build tree (e.g. using submodules), or find and link
> (statically or dynamically) to existing binaries?
>
> Linking to existing binaries is generally preferable: it avoids
> duplication on disk; it reduces compile times when compiling a single
> package from source (especially the first time). However, we aren't
> going to benefit from on-disk duplication. And, most importantly, unlike
> with ELF dependencies, there is no standardized way to locate MinGW
> libraries—especially if it comes to Wine-specific libraries. We would
> need a way for Wine's configure script to find these packages—and
> ideally find them automatically, or else fall back to a submodule-based
> approach.
>
> If we rely on distributions to provide our dependencies, the best idea I
> have here would be something like a x86_64-w64-mingw32-pkg-config
> (Fedora actually already ships this, but I think Fedora is the only
> one). And if we use shared libraries rather than static, things get
> worse: we need to know the exact path of each library and its
> dependencies at runtime so that we can copy (or symlink) them into a
> user's WINEPREFIX. For ELF this is the job of ld.so. For MinGW there is
> no standardized install location across distributions.
$ sudo apt install libz-mingw-w64-dev mingw-w64-tools
$ x86_64-w64-mingw32-pkg-config --libs zlib
-L/usr/x86_64-w64-mingw32/lib -lz
> For what it's worth, the current proposed solution (which has the
> support of the Wine maintainer) involves source imports and submodules.
> There's probably room for changing our approach even after things are
> committed, but I'd still like to get early feedback from distributions,
> and make sure that their interests are accurately represented, before we
> commit.
support for PE libraries, we can replace the source imports in the Wine
source code; this is done in many other Debian packages for projects which
vendor dependencies.
Regards,
Stephen
Stephen Kitt
Sep 5, 2021, 12:30:03 PM9/5/21
to
Hi Bastien,
On Sun, 5 Sep 2021 08:53:49 +0200, Bastien ROUCARIES
<roucarie...@gmail.com> wrote:
> Le dim. 5 sept. 2021 à 03:34, Zebediah Figura <zfi...@codeweavers.com> a
> écrit :
Thanks for stepping up!
For MinGW-w64 dependencies, an additional requirement is figuring out a
solution for https://bugs.debian.org/606825;
https://wiki.debian.org/Teams/Dpkg/FAQ#Q._Can_we_add_support_for_new_dpkg_architectures.3F
gives additional context.
Regards,
Stephen
On Sun, 5 Sep 2021 08:53:49 +0200, Bastien ROUCARIES
<roucarie...@gmail.com> wrote:
> Le dim. 5 sept. 2021 à 03:34, Zebediah Figura <zfi...@codeweavers.com> a
> écrit :
For MinGW-w64 dependencies, an additional requirement is figuring out a
solution for https://bugs.debian.org/606825;
https://wiki.debian.org/Teams/Dpkg/FAQ#Q._Can_we_add_support_for_new_dpkg_architectures.3F
gives additional context.
Regards,
Stephen
Zebediah Figura
Sep 5, 2021, 1:20:05 PM9/5/21
to
Copying/symlinking libfreetype-1.dll to libwinefreetype-1.dll is easy.
The problem is that libwinefreetype-1.dll is still going to link to
libharfbuzz-1.dll, but we need it to link to libwineharfbuzz-1.dll.
Note that's the wrong path for dynamic libraries, though; those go in
/usr/x86_64-w64-mingw32/bin/. We'll need a way to find those. Maybe
hardcoding a list won't be too painful, though...
Stephen Kitt
Sep 6, 2021, 3:10:04 AM9/6/21
to
On Sun, 5 Sep 2021 12:14:47 -0500, Zebediah Figura <zfi...@codeweavers.com>
wrote:
Ah yes, I hadn’t thought it through. So really Wine needs its own parallel
ecosystem of DLLs in any case, which effectively means building them along
with Wine (from Wine’s perspective, ignoring distribution constraints and
preferences).
> > This also works in Debian:
> >
> > $ sudo apt install libz-mingw-w64-dev mingw-w64-tools
> > $ x86_64-w64-mingw32-pkg-config --libs zlib
> > -L/usr/x86_64-w64-mingw32/lib -lz
>
> Ah, cool, I looked for that and somehow missed it.
>
> Note that's the wrong path for dynamic libraries, though; those go in
> /usr/x86_64-w64-mingw32/bin/. We'll need a way to find those. Maybe
> hardcoding a list won't be too painful, though...
In Debian they go in /usr/x86_64-w64-mingw32/lib currently, which is why
the .pc file points there. But as you point out, that doesn’t help at
runtime.
Also, having pkg-config support doesn’t really help with a parallel set of
DLLs, does it?
> >> For what it's worth, the current proposed solution (which has the
> >> support of the Wine maintainer) involves source imports and submodules.
> >> There's probably room for changing our approach even after things are
> >> committed, but I'd still like to get early feedback from distributions,
> >> and make sure that their interests are accurately represented, before we
> >> commit.
> >
> > Realistically, I think this is the best approach for now. As Debian adds
> > support for PE libraries, we can replace the source imports in the Wine
> > source code; this is done in many other Debian packages for projects which
> > vendor dependencies.
I still think this is true. With requirement for a Wine-specific set of DLLs,
improving the situation in Debian would involve supporting source
build-dependencies, i.e. being able to say that a given package wants access
to the source code of another package. That’s something that’s been brought
up previously, and is worked around by providing binary packages containing
package source code (e.g. binutils-source, gcc-11-source etc.), but isn’t
really upstream’s concern, in that it’s a Debian implementation detail.
Going back to your original three questions, I think that the best approach
for you as upstream is to focus on providing a complete set of source code
(including dependencies) which works, and to make it friendlier to
distributions, make the build process capable of handling alternative
locations for the dependencies’ source code or even build artifacts. (This
has a number of knock-on effects — in particular, you should ensure that the
upstream source code for all your dependencies works with Wine, i.e. that
Wine doesn’t require Wine-specific patches to any of its dependencies.)
Given how varied MinGW-w64 handling is in different distributions, pushing
things further risks making it easier for one distribution and harder for the
others...
Regards,
Stephen
wrote:
ecosystem of DLLs in any case, which effectively means building them along
with Wine (from Wine’s perspective, ignoring distribution constraints and
preferences).
> > This also works in Debian:
> >
> > $ sudo apt install libz-mingw-w64-dev mingw-w64-tools
> > $ x86_64-w64-mingw32-pkg-config --libs zlib
> > -L/usr/x86_64-w64-mingw32/lib -lz
>
> Ah, cool, I looked for that and somehow missed it.
>
> Note that's the wrong path for dynamic libraries, though; those go in
> /usr/x86_64-w64-mingw32/bin/. We'll need a way to find those. Maybe
> hardcoding a list won't be too painful, though...
the .pc file points there. But as you point out, that doesn’t help at
runtime.
Also, having pkg-config support doesn’t really help with a parallel set of
DLLs, does it?
> >> For what it's worth, the current proposed solution (which has the
> >> support of the Wine maintainer) involves source imports and submodules.
> >> There's probably room for changing our approach even after things are
> >> committed, but I'd still like to get early feedback from distributions,
> >> and make sure that their interests are accurately represented, before we
> >> commit.
> >
> > Realistically, I think this is the best approach for now. As Debian adds
> > support for PE libraries, we can replace the source imports in the Wine
> > source code; this is done in many other Debian packages for projects which
> > vendor dependencies.
improving the situation in Debian would involve supporting source
build-dependencies, i.e. being able to say that a given package wants access
to the source code of another package. That’s something that’s been brought
up previously, and is worked around by providing binary packages containing
package source code (e.g. binutils-source, gcc-11-source etc.), but isn’t
really upstream’s concern, in that it’s a Debian implementation detail.
Going back to your original three questions, I think that the best approach
for you as upstream is to focus on providing a complete set of source code
(including dependencies) which works, and to make it friendlier to
distributions, make the build process capable of handling alternative
locations for the dependencies’ source code or even build artifacts. (This
has a number of knock-on effects — in particular, you should ensure that the
upstream source code for all your dependencies works with Wine, i.e. that
Wine doesn’t require Wine-specific patches to any of its dependencies.)
Given how varied MinGW-w64 handling is in different distributions, pushing
things further risks making it easier for one distribution and harder for the
others...
Regards,
Stephen
Bastien ROUCARIES
Sep 6, 2021, 4:20:03 AM9/6/21
to
your own library.
Moreover using a custom loader will decrease the patching burden for
debian side, so a win win.
Bastien
Zebediah Figura
Sep 6, 2021, 12:40:03 PM9/6/21
to
libwinefreetype, and to find it you do `pkg-config --cflags --libs
libwinefreetype`", but that does strike me as more than a little janky.
It's probably worth pointing out that:
(1) if we use static linking, we should be able to use distribution
libraries unmodified. Of course, static linking comes with its own
downsides...
(2) Like I mentioned or hinted at in my original email, it *may* be
possible to use distribution dynamic libraries unmodified. It's not
clear that it can be done without breaking compatibility, but if it can,
would that change anything?
ἔρρωσθε,
Zebediah
Bastien ROUCARIES
Sep 6, 2021, 3:00:05 PM9/6/21
to
Yes i really prefer (2) it means we could use multiarch easilly....
Please try to document thé blocking point.
ἔρρωσθε,
Zebediah
Zebediah Figura
Sep 6, 2021, 6:00:04 PM9/6/21
to
I'll copy it here for reference.
The basic problem is that applications can and often do ship with PE
builds of cross-platform libraries. These libraries can be ahead of
Wine's system libraries, behind them, or even built with custom patches.
Accordingly we really don't want to load "our" freetype in place of
"their" freetype, or "theirs" in place of "ours". But because of the way
the Win32 loader works, you can generally only have one DLL of a given
name loaded in a process, and further attempts to dlopen() [as it were]
"libfreetype-6.dll" will return the handle to the already loaded
library, potentially breaking either Wine or the application.
different name (or by not using dynamic linking at all.)
Failing that, we need a way to determine whether a library is a system
dependency—and thus can *only* be loaded from the system DLL path—or an
application dependency—and thus should *never* be loaded from the system
DLL path. I think this can be done, but it gets tricky, and it's not
clear that there aren't going to be subtle problems from having two
different DLLs with the same name loaded in the same process. Especially
keeping in mind that Windows programs do absolutely insane things when
it comes to depending on OS internals.
Paul Wise
Sep 6, 2021, 8:50:03 PM9/6/21
to
On Mon, Sep 6, 2021 at 9:54 PM Zebediah Figura wrote:
> The basic problem is that applications can and often do ship with PE
> builds of cross-platform libraries. These libraries can be ahead of
> Wine's system libraries, behind them, or even built with custom patches.
> Accordingly we really don't want to load "our" freetype in place of
> "their" freetype, or "theirs" in place of "ours". But because of the way
> the Win32 loader works, you can generally only have one DLL of a given
> name loaded in a process, and further attempts to dlopen() [as it were]
> "libfreetype-6.dll" will return the handle to the already loaded
> library, potentially breaking either Wine or the application.
I don't know the details here, but my immediate thought when reading
> The basic problem is that applications can and often do ship with PE
> builds of cross-platform libraries. These libraries can be ahead of
> Wine's system libraries, behind them, or even built with custom patches.
> Accordingly we really don't want to load "our" freetype in place of
> "their" freetype, or "theirs" in place of "ours". But because of the way
> the Win32 loader works, you can generally only have one DLL of a given
> name loaded in a process, and further attempts to dlopen() [as it were]
> "libfreetype-6.dll" will return the handle to the already loaded
> library, potentially breaking either Wine or the application.
this is that you need some kind of namespace. I then found that linker
namespaces are a thing, perhaps they would provide the solution for
you. It sounds like the OpenGL shim loader solution listed on the
glibc wiki might work for your use-case. Or perhaps the LD_AUDIT
feature would also work.
https://www.sourceware.org/glibc/wiki/LinkerNamespaces
--
bye,
pabs
https://wiki.debian.org/PaulWise
Bastien Roucariès
Sep 7, 2021, 6:40:03 AM9/7/21
to
and will allow to be distrib friendly.
Moreover it will be best if marking a library as wine system one could be done
post build. We will prefer to not modify upstream (like libpng) source.
It is easier for us to apply small patch to upstream, or better in your case
to add a post link linker script or even some compiler flag that will add some
notes section to dll in order to mark it as use namespace system, or so on.
The problem on windows side is well known as dll hell
Renaming of the lib could be done post install. Moreover a crazy idea why not
renaming system dll instead of libpng-2.0.0.dll as libpng-2.0.0.sll (note the
version is manadaroty) Therefore we could use upstream source. Then add a
small linker script that will rewrite the depends of libpng-2.0.0.sll to
.sll file (aka binary sed s/.dll/.sll/g).
then add a small wraper (shim) like said pab that load the .sll library, the
best pratice will be a command tool that take the name of the sll and create
magically the dll
Therefore from distrib point of side we only changes the way we build the
package, it could be automated, it is scalable and it does not need to patch
package by package. Only to add some linker script magic
Bastien
Zebediah Figura
Sep 7, 2021, 11:50:03 AM9/7/21
to
system library; I think we can come up with a reliable heuristic for
that, without even needing linker namespaces or anything.
The outstanding problem seems to be more about potentially breaking
applications because they see two identically named DLLs loaded in the
same process. Applications can and do trawl the internal loader state,
although the Win32 loader also exposes some APIs so they don't even have to.
It may also be that the aforementioned heuristic is too hacky to be
accepted upstream into Wine.
Bastien ROUCARIES
Sep 7, 2021, 1:30:03 PM9/7/21
to
I disagree.
I do not propose to change the loader, i propose to change the link or post link stage of system library. You could rename the lib and change the dépend to new name after build. So you will have an unique name per system lib without changing the way you build these lib.
It is équivalent to use a namespace resolved at late build time.
Bastien ROUCARIES
Sep 7, 2021, 2:20:03 PM9/7/21
to
Zebediah Figura
Sep 7, 2021, 5:20:03 PM9/7/21
to
patching at runtime, e.g. when copying the dependency into the prefix.
It probably won't be easy, but it might be more feasible than modifying
the loader. I'll look into it...
Paul Wise
Sep 7, 2021, 7:10:03 PM9/7/21
to
On Tue, 2021-09-07 at 10:48 -0500, Zebediah Figura wrote:
> The outstanding problem seems to be more about potentially breaking
> applications because they see two identically named DLLs loaded in the
> same process. Applications can and do trawl the internal loader state,
> although the Win32 loader also exposes some APIs so they don't even have to.
I might be wrong but I got the impression that runtime linker
> The outstanding problem seems to be more about potentially breaking
> applications because they see two identically named DLLs loaded in the
> same process. Applications can and do trawl the internal loader state,
> although the Win32 loader also exposes some APIs so they don't even have to.
namespaces & LD_AUDIT are designed to solve this exact issue.
Bastien ROUCARIES
Sep 7, 2021, 7:30:04 PM9/7/21
to
messy stuff with lib loader...
see dll hell for instance.
Spec was wrong for the beginning for PE. And wine does not use the ELF
loader but a homemade PE in order to be bug to bug compatible with
windows.
In the long term implementing a LT_AUDIT in the wine loader will be
worthless, but in the short term elfpatch like approach could work.
Python use it.
Bastien
Bastien ROUCARIES
Sep 7, 2021, 7:50:04 PM9/7/21
to
Le mar. 7 sept. 2021 à 21:16, Zebediah Figura
Yes sort of obcopy patching under linux it is called patchelf...
If it work, it could be latter done in memory like paul wise suggest
implementing namespace and LT_AUDIT (but it is a long term goal)
1. capture library loading
2. do the patching at loading time (using the same code then patchelf
for pe replacing file read/write by mmap operation)
And voila
bastien
If it work, it could be latter done in memory like paul wise suggest
implementing namespace and LT_AUDIT (but it is a long term goal)
1. capture library loading
2. do the patching at loading time (using the same code then patchelf
for pe replacing file read/write by mmap operation)
And voila
bastien
Bastien ROUCARIES
Sep 8, 2021, 3:50:03 AM9/8/21
to
Hi,
Adding smcv to the thread.
Le mar. 7 sept. 2021 à 23:25, Bastien ROUCARIES
<roucarie...@gmail.com> a écrit :
>
Simon, do you think you could implement a version of libcapasule for PE object ?
Bastien
> bastien
Adding smcv to the thread.
Le mar. 7 sept. 2021 à 23:25, Bastien ROUCARIES
<roucarie...@gmail.com> a écrit :
>
Bastien
> bastien
Simon McVittie
Sep 8, 2021, 4:20:03 AM9/8/21
to
On Wed, 08 Sep 2021 at 07:31:59 +0000, Bastien ROUCARIES wrote:
> Simon, do you think you could implement a version of libcapasule for PE object ?
Given that libcapsule is very glibc- and ELF-specific, doesn't work
> Simon, do you think you could implement a version of libcapasule for PE object ?
properly without new glibc feature work that isn't in Debian yet (I've
lost track of whether/when it got merged upstream, it might be in 2.34)
and wasn't written by me, the answer is likely to be no.
As far as I understand it, the PE loader used for Wine is part of Wine,
so it has total control over the libraries that it loads and how it loads
them. This means that if Wine developers (the experts on this codebase)
have decided a libcapsule-like approach isn't feasible or isn't desirable,
they are probably right.
More generally, if Wine developers wanted Wine's dependency libraries
to be in one version of reality, and Windows DLLs to be in a different
version of reality, then there's a straightforward way to do that:
use ELF .so dependency libraries, like older versions of Wine did, and
don't give them a representation in the PE layer. Presumably they have
put in the work to move from ELF to PE dependency libraries because it
has some compelling advantage, rather than creating work for themselves
for no reason!
smcv
Zebediah Figura
Sep 8, 2021, 11:50:03 AM9/8/21
to
The basic reason is that Wine is essentially trying to split itself into
"PE" and "Unix" parts. The idea is that all of the "user" code is built
in PE format, and only calls into unix code by making a sort of fake
syscall. The "kernel" code is built in ELF format.
There are a few basic reasons for doing this:
* Many applications, mostly anti-cheat and anti-tamper engines, depend
on our libraries actually being in PE format on disk, and matching that
in memory. We used to have "fake" DLLs for this, but they weren't
verisimilar enough.
* Some of the important hosts for Wine have dropped support for 32-bit
libraries or are going to drop it. Mac is the obvious example here, but
many commercial Linux distributions also want to drop support. By
limiting the surface through which code can transition between PE and
Unix code, it becomes feasible to do 32-to-64 translation, where
previously this was quite infeasible.
* There's some demand for running Win32 debuggers under wine. These have
always worked more than a little tenuously, but a couple of them are
causing problems when they try to break in and unwind from Unix code. By
acting like we're making actual syscalls, things work much better.
Of the several dozen modules we have that currently call into Unix
libraries, about half of them aren't going to use PE dependencies. Most
of these have already been fully split into PE and Unix parts, including
things like the socket layer and winegstreamer. This raises the
question: why not just call into Unix for the other half as well? There
are a few reasons:
* Making fake syscalls is, like real syscalls, not cheap. It involves a
full context switch into and out of Unix code.
* Making callbacks from Unix code is difficult, and not cheap either.
For some libraries this doesn't matter, but for others it really does.
* Writing the wrappers between Unix and PE code involves some nontrivial
work in itself. The smaller the interface is, the easier things are for us.
Paul Wise
Sep 8, 2021, 9:20:13 PM9/8/21
to
On 9/8/21 3:13 AM, Simon McVittie wrote:
> As far as I understand it, the PE loader used for Wine is part of Wine,
> so it has total control over the libraries that it loads and how it loads
> them. This means that if Wine developers (the experts on this codebase)
> have decided a libcapsule-like approach isn't feasible or isn't desirable,
> they are probably right.
Could the PE loader add support for some sort of namespaces and then be
> As far as I understand it, the PE loader used for Wine is part of Wine,
> so it has total control over the libraries that it loads and how it loads
> them. This means that if Wine developers (the experts on this codebase)
> have decided a libcapsule-like approach isn't feasible or isn't desirable,
> they are probably right.
told to load Wine dependencies within a Wine namespace?
Does the Microsoft Windows loader support anything like this?
Recompiling every PE library with special flags or renaming every PE
library with a Wine prefix seems a bit tedious if just the PE loader
and anything that requests Wine deps can use a namespace/capsule/etc.
On Wed, 2021-09-08 at 10:42 -0500, Zebediah Figura wrote:
> The basic reason is that Wine is essentially trying to split itself into
> "PE" and "Unix" parts. The idea is that all of the "user" code is built
> in PE format, and only calls into unix code by making a sort of fake
> syscall. The "kernel" code is built in ELF format.
>
> There are a few basic reasons for doing this:
Zebediah Figura
Sep 9, 2021, 12:50:03 AM9/9/21
to
On 9/8/21 8:12 PM, Paul Wise wrote:
> On 9/8/21 3:13 AM, Simon McVittie wrote:
>> As far as I understand it, the PE loader used for Wine is part of Wine,
>> so it has total control over the libraries that it loads and how it loads
>> them. This means that if Wine developers (the experts on this codebase)
>> have decided a libcapsule-like approach isn't feasible or isn't desirable,
>> they are probably right.
>
> Could the PE loader add support for some sort of namespaces and then be
> told to load Wine dependencies within a Wine namespace?
>
> Does the Microsoft Windows loader support anything like this?
>
> Recompiling every PE library with special flags or renaming every PE
> library with a Wine prefix seems a bit tedious if just the PE loader
> and anything that requests Wine deps can use a namespace/capsule/etc.
>
The closest thing that Windows has is SxS, which is meant to solve I
> On 9/8/21 3:13 AM, Simon McVittie wrote:
>> As far as I understand it, the PE loader used for Wine is part of Wine,
>> so it has total control over the libraries that it loads and how it loads
>> them. This means that if Wine developers (the experts on this codebase)
>> have decided a libcapsule-like approach isn't feasible or isn't desirable,
>> they are probably right.
>
> Could the PE loader add support for some sort of namespaces and then be
> told to load Wine dependencies within a Wine namespace?
>
> Does the Microsoft Windows loader support anything like this?
>
> Recompiling every PE library with special flags or renaming every PE
> library with a Wine prefix seems a bit tedious if just the PE loader
> and anything that requests Wine deps can use a namespace/capsule/etc.
>
think some of the same problems (well, it's mainly meant to solve the
versioning problem that Microsoft dug themselves into). The problem is
that I believe it requires the namespace to be effectively hard-coded
into the library in question, in the form of a manifest resource.
Unfortunately, while thinking about the answer to this question, I
realized another snag, which I think really does make using
identically-named dynamic libraries impossible: if system library A
loads system library B dynamically, i.e. does the equivalent of a
dlopen(), we have no way of knowing which namespace to apply to it.
[AFAICT dlmopen() has this problem too, and just isn't meant for that
use case.] Even rewriting the import table doesn't help here.
Paul Wise
Sep 9, 2021, 1:20:04 AM9/9/21
to
On Wed, 2021-09-08 at 23:47 -0500, Zebediah Figura wrote:
> Unfortunately, while thinking about the answer to this question, I
> realized another snag, which I think really does make using
> identically-named dynamic libraries impossible: if system library A
> loads system library B dynamically, i.e. does the equivalent of a
> dlopen(), we have no way of knowing which namespace to apply to it.
> [AFAICT dlmopen() has this problem too, and just isn't meant for that
> use case.] Even rewriting the import table doesn't help here.
While loading library A into a namespace Foo, the loader could set a
> Unfortunately, while thinking about the answer to this question, I
> realized another snag, which I think really does make using
> identically-named dynamic libraries impossible: if system library A
> loads system library B dynamically, i.e. does the equivalent of a
> dlopen(), we have no way of knowing which namespace to apply to it.
> [AFAICT dlmopen() has this problem too, and just isn't meant for that
> use case.] Even rewriting the import table doesn't help here.
process global variable containing the current namespace to load into,
and then have the dlopen equivalent use namespace Foo in preference to
the default namespace for the process. Then when library B is loaded it
will go into namespace Foo. Once loading library B and A has completed
the loader could revert to using the default namespace.
Zebediah Figura
Sep 9, 2021, 1:40:03 AM9/9/21
to
B in its constructor routines. In fact, if it's loading library B
dynamically, it's probably not doing that.
Paul Wise
Sep 9, 2021, 1:50:03 AM9/9/21
to
On Thu, 2021-09-09 at 00:39 -0500, Zebediah Figura wrote:
> Right, but we don't have any guarantee that library A will load library
> B in its constructor routines. In fact, if it's loading library B
> dynamically, it's probably not doing that.
Can the loader tell which library asked it to load a library?
> Right, but we don't have any guarantee that library A will load library
> B in its constructor routines. In fact, if it's loading library B
> dynamically, it's probably not doing that.
Then the namespace could be inherited from the parent library.
Zebediah Figura
Sep 9, 2021, 2:10:02 AM9/9/21
to
On 9/9/21 12:45 AM, Paul Wise wrote:
> On Thu, 2021-09-09 at 00:39 -0500, Zebediah Figura wrote:
>
>> Right, but we don't have any guarantee that library A will load library
>> B in its constructor routines. In fact, if it's loading library B
>> dynamically, it's probably not doing that.
>
> Can the loader tell which library asked it to load a library?
> Then the namespace could be inherited from the parent library.
>
Unfortunately, no. We have no way of knowing the caller.
> On Thu, 2021-09-09 at 00:39 -0500, Zebediah Figura wrote:
>
>> Right, but we don't have any guarantee that library A will load library
>> B in its constructor routines. In fact, if it's loading library B
>> dynamically, it's probably not doing that.
>
> Can the loader tell which library asked it to load a library?
> Then the namespace could be inherited from the parent library.
>
Paul Wise
Sep 9, 2021, 3:40:03 AM9/9/21
to
On Thu, 2021-09-09 at 00:59 -0500, Zebediah Figura wrote:
> Unfortunately, no. We have no way of knowing the caller.
Can the PE loading mechanism do something like inject a fake dlopen
> Unfortunately, no. We have no way of knowing the caller.
function available only in the Wine namespace that just passes the Wine
namespace to the dlmopen function? Or the same but for each library?
Bastien ROUCARIES
Sep 10, 2021, 5:50:03 AM9/10/21
to
Le jeu. 9 sept. 2021 à 07:32, Paul Wise <pa...@debian.org> a écrit :
>
> On Thu, 2021-09-09 at 00:59 -0500, Zebediah Figura wrote:
>
> > Unfortunately, no. We have no way of knowing the caller.
>
> Can the PE loading mechanism do something like inject a fake dlopen
> function available only in the Wine namespace that just passes the Wine
> namespace to the dlmopen function? Or the same but for each library?
The problem is that windows apps particularly games try to check if
>
> On Thu, 2021-09-09 at 00:59 -0500, Zebediah Figura wrote:
>
> > Unfortunately, no. We have no way of knowing the caller.
>
> Can the PE loading mechanism do something like inject a fake dlopen
> function available only in the Wine namespace that just passes the Wine
> namespace to the dlmopen function? Or the same but for each library?
mapped ram exec pages are from dll from disk and not modified in
memory.
Thus if we create namespace, this crappy software will stop to work
except if we create or symlink a library with an unique name that the
wine applications could fetch
Note that windows use for libraryname only the basename not the
fullpatch (zebediah could you confirm)
I am proposing (for simplicity I call the problematic lib libpng that
depends on libz)
1. build like usual a libpng and libz. libpng depend on linbz
2. copy libpng to libwinepng and libz to libwinez
3 use a new patchpe command (like the patchelf command) to patch the
depend of libwinepng
So globally a post link time namespace (it will be better on debian
side than to recompile and pass flags to library).
You propose:
1. build like usual a libpng and libz. libpng
2 create symlink libwinepng -> libpng, liwinez->libz
3. instruct the loader to load libwinepng privatly using namespace
approach and do at run time the patchpe work
I believe that your approach is technically superior but could be done
on the top of my first approach and a second step.
Zebediah could you please give use feedback from wine team ?
If patchpe is workable (and I firmly believe it is), you could add a
task to see if paul wise solution is workable and report here.
Next step, will be to refine your proposal. For instance i really
dislike the libpng libwinepng mapping I will prefer a mapping given as
perlre
and something that encode the wine ABI like s/^lib/lib-wineinternaluseABI1.0/g
It will allow to create pkgconfig file for wine version of lib automatically
Bastien
Paul Wise
Sep 11, 2021, 2:10:04 AM9/11/21
to
Disclaimer: I know precisely zero of the details here nor
if the PE loader can support any of the below features.
On Fri, 2021-09-10 at 09:23 +0000, Bastien ROUCARIES wrote:
> The problem is that windows apps particularly games try to check if
> mapped ram exec pages are from dll from disk and not modified in
> memory.
The fake dlopen could presumably be provided by an on-disk DLL?
The PE loader would just load the fake dlopen DLL first, like
glibc does when you set the LD_PRELOAD environment variable.
> You propose:
I'm not proposing any on-disk symlinks, renaming DLLs or any other
changes to any PE DLLs, just loading into memory like dlmopen.
The solution I'm thinking of looks something like this:
src:libpng -> libpng16-16:$winarch -> /usr/lib/$wintriplet/libpng16.dll
(requires Debian to support cross-only, partial and Windows arch types)
PE loader sets up in-memory Wine namespace.
PE loader loads into Wine namespace: /usr/lib/$windowstriplet/wine/fake-dlopen.dll
PE loader loads Wine dependencies into Wine namespace.
Some of the Wine dependencies call the Windows equivalent of dlopen,
which resolves to the fake dlopen equivalent within the namespace,
which calls the Windows equivalent of dlmopen("Wine", dll), which makes
the PE loader load those libraries into the Wine namespace.
PE loader finishes loading Wine dependencies and proceeds loading the
PE file and its dependencies into the default namespace.
Windows apps check mapped RAM pages and they show up correctly.
The main issue would be that apps could check they run in Wine.
OTOH the PE loader could also just hide the pages somehow?
if the PE loader can support any of the below features.
On Fri, 2021-09-10 at 09:23 +0000, Bastien ROUCARIES wrote:
> The problem is that windows apps particularly games try to check if
> mapped ram exec pages are from dll from disk and not modified in
> memory.
The PE loader would just load the fake dlopen DLL first, like
glibc does when you set the LD_PRELOAD environment variable.
> You propose:
I'm not proposing any on-disk symlinks, renaming DLLs or any other
changes to any PE DLLs, just loading into memory like dlmopen.
The solution I'm thinking of looks something like this:
src:libpng -> libpng16-16:$winarch -> /usr/lib/$wintriplet/libpng16.dll
(requires Debian to support cross-only, partial and Windows arch types)
PE loader sets up in-memory Wine namespace.
PE loader loads into Wine namespace: /usr/lib/$windowstriplet/wine/fake-dlopen.dll
PE loader loads Wine dependencies into Wine namespace.
Some of the Wine dependencies call the Windows equivalent of dlopen,
which resolves to the fake dlopen equivalent within the namespace,
which calls the Windows equivalent of dlmopen("Wine", dll), which makes
the PE loader load those libraries into the Wine namespace.
PE loader finishes loading Wine dependencies and proceeds loading the
PE file and its dependencies into the default namespace.
Windows apps check mapped RAM pages and they show up correctly.
The main issue would be that apps could check they run in Wine.
OTOH the PE loader could also just hide the pages somehow?
Adrian Bunk
Sep 11, 2021, 11:40:02 AM9/11/21
to
On Sat, Sep 04, 2021 at 08:17:53PM -0500, Zebediah Figura wrote:
> Hello all,
>...
> Accordingly, although static linking and source imports are generally
> disprefered, it may quite likely be preferable in our case. We don't get the
> benefits of on-disk deduplication, since Wine is essentially the only piece
> of software which needs these libraries.
>...
How are distributions supposed to provide security support for Wine?
As an example, Debian 10 that is security supported by Debian
until next summer ships Wine 4.0.
The libgnutls in Debian 10 has already been patched several times
by Debian for CVE fixes.
Having to patch several different versions of the same library
in different packages multiplies the effort required to provide
security support for a library.
cu
Adrian
> Hello all,
> Accordingly, although static linking and source imports are generally
> disprefered, it may quite likely be preferable in our case. We don't get the
> benefits of on-disk deduplication, since Wine is essentially the only piece
> of software which needs these libraries.
>...
How are distributions supposed to provide security support for Wine?
As an example, Debian 10 that is security supported by Debian
until next summer ships Wine 4.0.
The libgnutls in Debian 10 has already been patched several times
by Debian for CVE fixes.
Having to patch several different versions of the same library
in different packages multiplies the effort required to provide
security support for a library.
cu
Adrian
Bastien ROUCARIES
Sep 11, 2021, 2:10:03 PM9/11/21
to
Yes what why Paul and myself ask for partial arch. No static linking no embeded lib but we need answer from upstream.
Note that windows application that use old library are not a concern from the debian si de
Bastien
cu
Adrian
Adrian Bunk
Sep 12, 2021, 4:00:03 AM9/12/21
to
On Sun, Sep 05, 2021 at 08:53:49AM +0200, Bastien ROUCARIES wrote:
>...
> Improve dpkg to support partial arch. I volonteer to implement none arch
> but i am waiting from guillem here.
>...
There is also plenty of infrastructure on the buildd, archive and
release team sides that would likely need changes for supporting
anything like that.
A multilib based approach might be more realistic for bookworm.
What benefits would a "none arch" even bring compared to building
binary-all packages?
The ability to binNMU is the only one that comes into my mind.
cu
Adrian
>...
> Improve dpkg to support partial arch. I volonteer to implement none arch
> but i am waiting from guillem here.
>...
There is also plenty of infrastructure on the buildd, archive and
release team sides that would likely need changes for supporting
anything like that.
A multilib based approach might be more realistic for bookworm.
What benefits would a "none arch" even bring compared to building
binary-all packages?
The ability to binNMU is the only one that comes into my mind.
cu
Adrian
Bastien ROUCARIES
Sep 12, 2021, 9:40:03 AM9/12/21
to
So we build three arch-all package with paterning name.
>
> cu
> Adrian
Adrian Bunk
Sep 12, 2021, 10:10:03 AM9/12/21
to
What release architecture would "none" packages be in the Debian
archive, and on what buildds would they be built?
Debian buildds are not cross-compiling packages for a different Debian
architecture, a partial architecture would still require DSA-maintained
buildds doing native building on this partial architecture.
cu
Adrian
Bastien ROUCARIES
Sep 12, 2021, 10:20:03 AM9/12/21
to
https://wiki.debian.org/Teams/Dpkg/Spec/FreestandingArches
They are a full color gradiant between:
- freestanding arches pure cross compile without any depends except arch:all
- partial cross built arch
- partial arch
- full arch
I believe the first step to get partial cross built arch is to begin
by freestanding arch.
Bastien
> cu
> Adrian
Adrian Bunk
Sep 12, 2021, 10:40:03 AM9/12/21
to
On Sun, Sep 12, 2021 at 01:57:44PM +0000, Bastien ROUCARIES wrote:
>
> I think you misunderstand:
> https://wiki.debian.org/Teams/Dpkg/Spec/FreestandingArches
>
> They are a full color gradiant between:
> - freestanding arches pure cross compile without any depends except arch:all
> - partial cross built arch
> - partial arch
> - full arch
>
> I believe the first step to get partial cross built arch is to begin
> by freestanding arch.
I believe this is mostly irrelevant for the discussion about Wine.
>
> I think you misunderstand:
> https://wiki.debian.org/Teams/Dpkg/Spec/FreestandingArches
>
> They are a full color gradiant between:
> - freestanding arches pure cross compile without any depends except arch:all
> - partial cross built arch
> - partial arch
> - full arch
>
> I believe the first step to get partial cross built arch is to begin
> by freestanding arch.
Making Wine in Debian depend on such lofty ideas that might be available
very far in the future (if ever) would imply no Wine in the next Debian
releases.
Debian 12 will be released mid-2023, which is only 2 years away.
Debian 13 will be released mid-2025, which is only 4 years away.
If your partial cross built arch is ever fully implemented and supported
by Debian it would be a good idea to migrate Wine to using that.
But a realistic solution for Wine in Debian 12 has to be multilib,
not multiarch.
> Bastien
cu
Adrian
Bastien ROUCARIES
Sep 12, 2021, 11:20:03 AM9/12/21
to
multiarch. And It was since 2012 that multiarch exists. Maybe time to
speed up
bastien
>
> > Bastien
>
> cu
> Adrian
Bastien ROUCARIES
Sep 12, 2021, 11:20:03 AM9/12/21
to
https://svn.python.org/projects/python/trunk/PC/dl_nt.c
Windows already implemented the name space called activation context.
Need only windows XP or better but wine could offer the API
Adrian Bunk
Sep 12, 2021, 11:30:03 AM9/12/21
to
Debian maintainers of Wine in a direction where Wine would be taken
hostage for some lofty idea you have.
> bastien
cu
Adrian
Stephen Kitt
Sep 12, 2021, 11:40:04 AM9/12/21
to
for the new architectures without changes (ideally). libz is currently built
for MinGW-w64 in an “Architecture: all” package, but it’s a separate source
package; various other libraries are built with specific support in their
source packages. While one could imagine adding support to all the appropriate
source packages to build similar “Architecture: all” packages, that would
require convincing all the relevant maintainers, and it would end up tying
the testing migrations to MinGW-w64...
In this scenario the solution wouldn’t be a “none” arch, but a Windows arch,
if we can ever resolve https://bugs.debian.org/606825
The buildd situation isn’t necessarily that much of an obstacle: it seems to
me we could have “Windows” buildds which are really Linux amd64 systems, that
cross-build for Windows.
Regards,
Stephen
Adrian Bunk
Sep 12, 2021, 12:40:02 PM9/12/21
to
On Sun, Sep 12, 2021 at 05:31:41PM +0200, Stephen Kitt wrote:
>...
the ftp team and the release team.
It would also require many software changes.
How would for example the dependencies of wine:amd64 be fulfilled?
Just supporting that package dependencies might only be fulfilled by
also using packages from a different architecture would require changes
in many places, like for example the testing migration scripts that
ensure installability after migration.
> and it would end up tying the testing migrations to MinGW-w64...
If this partial arch (and Wine) should be part of Debian releases,
then testing migrations would have to be tied to it in any case.
>...
building packages on the buildds, there is likely a lot of work and
bugfixing ahead for you for getting that working.
> Regards,
>
> Stephen
cu
Adrian
>...
> While one could imagine adding support to all the appropriate
> source packages to build similar “Architecture: all” packages, that would
> require convincing all the relevant maintainers,
Adding a new release architecture (partial or not) requires convincing
> source packages to build similar “Architecture: all” packages, that would
> require convincing all the relevant maintainers,
the ftp team and the release team.
It would also require many software changes.
How would for example the dependencies of wine:amd64 be fulfilled?
Just supporting that package dependencies might only be fulfilled by
also using packages from a different architecture would require changes
in many places, like for example the testing migration scripts that
ensure installability after migration.
> and it would end up tying the testing migrations to MinGW-w64...
then testing migrations would have to be tied to it in any case.
>...
> The buildd situation isn’t necessarily that much of an obstacle: it seems to
> me we could have “Windows” buildds which are really Linux amd64 systems, that
> cross-build for Windows.
The first obstacle is that if you want to be the first who does cross
> me we could have “Windows” buildds which are really Linux amd64 systems, that
> cross-build for Windows.
building packages on the buildds, there is likely a lot of work and
bugfixing ahead for you for getting that working.
> Regards,
>
> Stephen
cu
Adrian
Stephen Kitt
Sep 12, 2021, 1:10:03 PM9/12/21
to
On Sun, 12 Sep 2021 19:20:16 +0300, Adrian Bunk <bu...@debian.org> wrote:
> On Sun, Sep 12, 2021 at 05:31:41PM +0200, Stephen Kitt wrote:
> >...
> > While one could imagine adding support to all the appropriate
> > source packages to build similar “Architecture: all” packages, that would
> > require convincing all the relevant maintainers,
>
> Adding a new release architecture (partial or not) requires convincing
> the ftp team and the release team.
>
> It would also require many software changes.
Yes, it would. However, it requires convincing a well-defined set of people
> On Sun, Sep 12, 2021 at 05:31:41PM +0200, Stephen Kitt wrote:
> >...
> > While one could imagine adding support to all the appropriate
> > source packages to build similar “Architecture: all” packages, that would
> > require convincing all the relevant maintainers,
>
> Adding a new release architecture (partial or not) requires convincing
> the ftp team and the release team.
>
> It would also require many software changes.
*once*. If we handle Windows dependencies using “Architecture: all” packages,
we would either end up with something like Fedora’s MinGW-w64 SIG (with a
complete set of independent packages), or we would end up having to convince
a huge set of package maintainers over and over again.
> How would for example the dependencies of wine:amd64 be fulfilled?
>
> Just supporting that package dependencies might only be fulfilled by
> also using packages from a different architecture would require changes
> in many places, like for example the testing migration scripts that
> ensure installability after migration.
but it’s useful for more than just Wine, and it can be done once for all the
interested architectures.
> > and it would end up tying the testing migrations to MinGW-w64...
>
> If this partial arch (and Wine) should be part of Debian releases,
> then testing migrations would have to be tied to it in any case.
source packages.)
> >...
> > The buildd situation isn’t necessarily that much of an obstacle: it seems
> > to me we could have “Windows” buildds which are really Linux amd64
> > systems, that cross-build for Windows.
>
> The first obstacle is that if you want to be the first who does cross
> building packages on the buildds, there is likely a lot of work and
> bugfixing ahead for you for getting that working.
Fixing this would reduce the burden on all the maintainers who currently
handle cross-built packages in the archive, so overall I think it would be a
net win.
Regards,
Stephen
Adrian Bunk
Sep 12, 2021, 2:20:03 PM9/12/21
to
On Sun, Sep 12, 2021 at 07:03:54PM +0200, Stephen Kitt wrote:
> On Sun, 12 Sep 2021 19:20:16 +0300, Adrian Bunk <bu...@debian.org> wrote:
> > On Sun, Sep 12, 2021 at 05:31:41PM +0200, Stephen Kitt wrote:
> > >...
> > > While one could imagine adding support to all the appropriate
> > > source packages to build similar “Architecture: all” packages, that would
> > > require convincing all the relevant maintainers,
> >
> > Adding a new release architecture (partial or not) requires convincing
> > the ftp team and the release team.
> >
> > It would also require many software changes.
>
> Yes, it would. However, it requires convincing a well-defined set of people
> *once*. If we handle Windows dependencies using “Architecture: all” packages,
> we would either end up with something like Fedora’s MinGW-w64 SIG (with a
> complete set of independent packages), or we would end up having to convince
> a huge set of package maintainers over and over again.
Based on the package list provided in the initial email in this thread
> On Sun, 12 Sep 2021 19:20:16 +0300, Adrian Bunk <bu...@debian.org> wrote:
> > On Sun, Sep 12, 2021 at 05:31:41PM +0200, Stephen Kitt wrote:
> > >...
> > > While one could imagine adding support to all the appropriate
> > > source packages to build similar “Architecture: all” packages, that would
> > > require convincing all the relevant maintainers,
> >
> > Adding a new release architecture (partial or not) requires convincing
> > the ftp team and the release team.
> >
> > It would also require many software changes.
>
> Yes, it would. However, it requires convincing a well-defined set of people
> *once*. If we handle Windows dependencies using “Architecture: all” packages,
> we would either end up with something like Fedora’s MinGW-w64 SIG (with a
> complete set of independent packages), or we would end up having to convince
> a huge set of package maintainers over and over again.
(and guessing the amount of transitive dependencies) I would have guessed
that there are perhaps ~ 30 packages that would have to be touched once.
And after that things will just continue working, just like with udebs
which are in some ways similar to what is being discussed here.
> > How would for example the dependencies of wine:amd64 be fulfilled?
> >
> > Just supporting that package dependencies might only be fulfilled by
> > also using packages from a different architecture would require changes
> > in many places, like for example the testing migration scripts that
> > ensure installability after migration.
>
> In the same way as the none-arch packages would be. Yes, it’s a lot of work,
> but it’s useful for more than just Wine, and it can be done once for all the
> interested architectures.
fully supported in one release before it can be used by packages in the
next release, if for example any of dpkg/apt/aptitude/... in the
previous stable gives an error when seeing dependencies to another
architecture.
>...
> Regards,
>
> Stephen
cu
Adrian
Helmut Grohne
Sep 13, 2021, 11:50:03 AM9/13/21
to
I believe I can speak with my "main cross building porter" hat on.
On Sun, Sep 12, 2021 at 01:57:44PM +0000, Bastien ROUCARIES wrote:
unrealistic. wine can quite simply use -$arch-cross:all packages like
very other bare metal target and switch to our pipe dream once it is
ready (if ever).
cross building is very far off becoming the thing you want it to be. It
presently has a bus factor of around 1.5. Around 59% of source packages
are cross-satisfiable. Around 78% of cross-satisfiable packages are
cross buildable. In other words: less than 50% of packages are cross
buildable. Beyond that, the cross porters cannot keep up with the
current rate of regressions. Fractions are falling.
To make cross building a viable thing we'd need at least three more
porters determined to make it happen. Talk doesn't make it happen.
Patches do. Let me know if you want to help. I'll get you started.
Helmut
On Sun, Sep 12, 2021 at 01:57:44PM +0000, Bastien ROUCARIES wrote:
> They are a full color gradiant between:
> - freestanding arches pure cross compile without any depends except arch:all
> - partial cross built arch
> - partial arch
> - full arch
>
> I believe the first step to get partial cross built arch is to begin
> by freestanding arch.
Certainly not. As much as I would love to see this happen, it is very
> - freestanding arches pure cross compile without any depends except arch:all
> - partial cross built arch
> - partial arch
> - full arch
>
> I believe the first step to get partial cross built arch is to begin
> by freestanding arch.
unrealistic. wine can quite simply use -$arch-cross:all packages like
very other bare metal target and switch to our pipe dream once it is
ready (if ever).
cross building is very far off becoming the thing you want it to be. It
presently has a bus factor of around 1.5. Around 59% of source packages
are cross-satisfiable. Around 78% of cross-satisfiable packages are
cross buildable. In other words: less than 50% of packages are cross
buildable. Beyond that, the cross porters cannot keep up with the
current rate of regressions. Fractions are falling.
To make cross building a viable thing we'd need at least three more
porters determined to make it happen. Talk doesn't make it happen.
Patches do. Let me know if you want to help. I'll get you started.
Helmut
Stephen Kitt
Sep 13, 2021, 2:20:04 PM9/13/21
to
On Mon, 13 Sep 2021 17:39:25 +0200, Helmut Grohne <hel...@subdivi.de> wrote:
> I believe I can speak with my "main cross building porter" hat on.
>
> On Sun, Sep 12, 2021 at 01:57:44PM +0000, Bastien ROUCARIES wrote:
> > They are a full color gradiant between:
> > - freestanding arches pure cross compile without any depends except
> > arch:all
> > - partial cross built arch
> > - partial arch
> > - full arch
> >
> > I believe the first step to get partial cross built arch is to begin
> > by freestanding arch.
>
> Certainly not. As much as I would love to see this happen, it is very
> unrealistic. wine can quite simply use -$arch-cross:all packages like
> very other bare metal target and switch to our pipe dream once it is
> ready (if ever).
And even that assumes that we can get agreement on what $arch should be for
> I believe I can speak with my "main cross building porter" hat on.
>
> On Sun, Sep 12, 2021 at 01:57:44PM +0000, Bastien ROUCARIES wrote:
> > They are a full color gradiant between:
> > - freestanding arches pure cross compile without any depends except
> > arch:all
> > - partial cross built arch
> > - partial arch
> > - full arch
> >
> > I believe the first step to get partial cross built arch is to begin
> > by freestanding arch.
>
> Certainly not. As much as I would love to see this happen, it is very
> unrealistic. wine can quite simply use -$arch-cross:all packages like
> very other bare metal target and switch to our pipe dream once it is
> ready (if ever).
Windows targets :-(.
> cross building is very far off becoming the thing you want it to be. It
> presently has a bus factor of around 1.5. Around 59% of source packages
> are cross-satisfiable. Around 78% of cross-satisfiable packages are
> cross buildable. In other words: less than 50% of packages are cross
> buildable. Beyond that, the cross porters cannot keep up with the
> current rate of regressions. Fractions are falling.
ecosystem) we don’t need all that many source packages to be
cross-satisfiable for the whole endeavour to be useful...
The regressions are significant though: if packages can’t stay
cross-satisfiable for Debian cross-targets, there’s little hope they can stay
cross-satisfiable for Windows! This means that separate source packages are
probably the only viable option.
> To make cross building a viable thing we'd need at least three more
> porters determined to make it happen. Talk doesn't make it happen.
> Patches do. Let me know if you want to help. I'll get you started.
http://crossqa.debian.net and figure out what’s going wrong (and hope that
pulling the threads doesn’t reveal a monster...)?
Regards,
Stephen
Stephen Kitt
Sep 13, 2021, 2:30:04 PM9/13/21
to
On Sun, 12 Sep 2021 20:54:56 +0300, Adrian Bunk <bu...@debian.org> wrote:
> On Sun, Sep 12, 2021 at 07:03:54PM +0200, Stephen Kitt wrote:
> > On Sun, 12 Sep 2021 19:20:16 +0300, Adrian Bunk <bu...@debian.org> wrote:
> > > On Sun, Sep 12, 2021 at 05:31:41PM +0200, Stephen Kitt wrote:
> > > >...
> > > > While one could imagine adding support to all the appropriate
> > > > source packages to build similar “Architecture: all” packages, that
> > > > would require convincing all the relevant maintainers,
> > >
> > > Adding a new release architecture (partial or not) requires convincing
> > > the ftp team and the release team.
> > >
> > > It would also require many software changes.
> >
> > Yes, it would. However, it requires convincing a well-defined set of
> > people *once*. If we handle Windows dependencies using “Architecture:
> > all” packages, we would either end up with something like Fedora’s
> > MinGW-w64 SIG (with a complete set of independent packages), or we would
> > end up having to convince a huge set of package maintainers over and over
> > again.
>
> Based on the package list provided in the initial email in this thread
> (and guessing the amount of transitive dependencies) I would have guessed
> that there are perhaps ~ 30 packages that would have to be touched once.
... which means ~30 maintainers who need convincing individually. My
> On Sun, Sep 12, 2021 at 07:03:54PM +0200, Stephen Kitt wrote:
> > On Sun, 12 Sep 2021 19:20:16 +0300, Adrian Bunk <bu...@debian.org> wrote:
> > > On Sun, Sep 12, 2021 at 05:31:41PM +0200, Stephen Kitt wrote:
> > > >...
> > > > While one could imagine adding support to all the appropriate
> > > > source packages to build similar “Architecture: all” packages, that
> > > > would require convincing all the relevant maintainers,
> > >
> > > Adding a new release architecture (partial or not) requires convincing
> > > the ftp team and the release team.
> > >
> > > It would also require many software changes.
> >
> > Yes, it would. However, it requires convincing a well-defined set of
> > people *once*. If we handle Windows dependencies using “Architecture:
> > all” packages, we would either end up with something like Fedora’s
> > MinGW-w64 SIG (with a complete set of independent packages), or we would
> > end up having to convince a huge set of package maintainers over and over
> > again.
>
> Based on the package list provided in the initial email in this thread
> (and guessing the amount of transitive dependencies) I would have guessed
> that there are perhaps ~ 30 packages that would have to be touched once.
experience so far is that maintainers tend not to be interested (justifiably)
in Windows cross-builds.
> And after that things will just continue working, just like with udebs
> which are in some ways similar to what is being discussed here.
tends to require a bit more effort unfortunately.
> > > How would for example the dependencies of wine:amd64 be fulfilled?
> > >
> > > Just supporting that package dependencies might only be fulfilled by
> > > also using packages from a different architecture would require changes
> > > in many places, like for example the testing migration scripts that
> > > ensure installability after migration.
> >
> > In the same way as the none-arch packages would be. Yes, it’s a lot of
> > work, but it’s useful for more than just Wine, and it can be done once
> > for all the interested architectures.
>
> It is also the kind of change where it might be required that it is
> fully supported in one release before it can be used by packages in the
> next release, if for example any of dpkg/apt/aptitude/... in the
> previous stable gives an error when seeing dependencies to another
> architecture.
Regards,
Stephen
Helmut Grohne
Sep 13, 2021, 4:10:03 PM9/13/21
to
On Mon, Sep 13, 2021 at 08:19:31PM +0200, Stephen Kitt wrote:
> Is the process documented anywhere? Or does one simply pick a failure from
> http://crossqa.debian.net and figure out what’s going wrong (and hope that
> pulling the threads doesn’t reveal a monster...)?
That describes the process quite well. In particular, crossqa.d.n links
> Is the process documented anywhere? Or does one simply pick a failure from
> http://crossqa.debian.net and figure out what’s going wrong (and hope that
> pulling the threads doesn’t reveal a monster...)?
to existing bug reports to avoid duplication of work. A major lack
(beyond people doing the work) is domain-specific knowledge of the
various archive sections. So once you locate a monster, don't stop
there, but contact debian...@lists.debian.org or
#debian-bootstrap@oftc and let us combine your domain-specific knowledge
with some cross compilation experience to create an original solution.
So yeah, start with those packages that you know well.
Check http://crossqa.debian.net/src/<sourcepackagename> for your pet
packages or follow the "cross" link in the right sidebar at
tracker.d.o. Also (thanks to IOhannes zmölnig) enable cross building in
your salsa-ci pipeline.
In any case though, don't let wine wait for cross building to become a
thing.
Helmut
Adrian Bunk
Sep 13, 2021, 4:30:02 PM9/13/21
to
On Mon, Sep 13, 2021 at 08:19:31PM +0200, Stephen Kitt wrote:
>...
> For Wine (and even a wider MinGW-w64
> ecosystem) we don’t need all that many source packages to be
> cross-satisfiable for the whole endeavour to be useful...
But you would still need to create and maintain the whole
> ecosystem) we don’t need all that many source packages to be
> cross-satisfiable for the whole endeavour to be useful...
infrastructure.
The larger picture is that there are at least 3 big topics:
- cross-architecture dependencies
- partial release architectures
- cross-building a release architecture
Every single of these topics would be a huge amount of work.
If you need all 3 as basis for a solution for Wine,
then that's simply not realistic in the foreseeable future.
> The regressions are significant though: if packages can’t stay
> cross-satisfiable for Debian cross-targets, there’s little hope they can stay
> cross-satisfiable for Windows! This means that separate source packages are
> probably the only viable option.
Separate source packages don't bring any real benefits compared to
just bundling and building everything in src:wine.
Either would likely imply that Wine in Debian comes without security
support, and the Release Notes stating that Wine should only be used
for trusted local contents.
> Regards,
>
> Stephen
cu
Adrian
Bastien ROUCARIES
Sep 13, 2021, 5:00:03 PM9/13/21
to
Le lun. 13 sept. 2021 à 20:24, Adrian Bunk <bu...@debian.org> a écrit :
>
> On Mon, Sep 13, 2021 at 08:19:31PM +0200, Stephen Kitt wrote:
> >...
> > For Wine (and even a wider MinGW-w64
> > ecosystem) we don’t need all that many source packages to be
> > cross-satisfiable for the whole endeavour to be useful...
>
> But you would still need to create and maintain the whole
> infrastructure.
>
> The larger picture is that there are at least 3 big topics:
> - cross-architecture dependencies
> - partial release architectures
> - cross-building a release architecture
>
> Every single of these topics would be a huge amount of work.
>
> If you need all 3 as basis for a solution for Wine,
> then that's simply not realistic in the foreseeable future.
>
> > The regressions are significant though: if packages can’t stay
> > cross-satisfiable for Debian cross-targets, there’s little hope they can stay
> > cross-satisfiable for Windows! This means that separate source packages are
> > probably the only viable option.
> >...
>
> Separate source packages don't bring any real benefits compared to
> just bundling and building everything in src:wine.
No please do not do that. Vendoring is bad and I fight on the
>
> On Mon, Sep 13, 2021 at 08:19:31PM +0200, Stephen Kitt wrote:
> >...
> > For Wine (and even a wider MinGW-w64
> > ecosystem) we don’t need all that many source packages to be
> > cross-satisfiable for the whole endeavour to be useful...
>
> But you would still need to create and maintain the whole
> infrastructure.
>
> The larger picture is that there are at least 3 big topics:
> - cross-architecture dependencies
> - partial release architectures
> - cross-building a release architecture
>
> Every single of these topics would be a huge amount of work.
>
> If you need all 3 as basis for a solution for Wine,
> then that's simply not realistic in the foreseeable future.
>
> > The regressions are significant though: if packages can’t stay
> > cross-satisfiable for Debian cross-targets, there’s little hope they can stay
> > cross-satisfiable for Windows! This means that separate source packages are
> > probably the only viable option.
> >...
>
> Separate source packages don't bring any real benefits compared to
> just bundling and building everything in src:wine.
javascript side since four years, against it.
Adrian Bunk
Sep 13, 2021, 5:40:03 PM9/13/21
to
On Mon, Sep 13, 2021 at 08:38:52PM +0000, Bastien ROUCARIES wrote:
> Le lun. 13 sept. 2021 à 20:24, Adrian Bunk <bu...@debian.org> a écrit :
> > On Mon, Sep 13, 2021 at 08:19:31PM +0200, Stephen Kitt wrote:
>...
> Le lun. 13 sept. 2021 à 20:24, Adrian Bunk <bu...@debian.org> a écrit :
> > On Mon, Sep 13, 2021 at 08:19:31PM +0200, Stephen Kitt wrote:
>...
> > > The regressions are significant though: if packages can’t stay
> > > cross-satisfiable for Debian cross-targets, there’s little hope they can stay
> > > cross-satisfiable for Windows! This means that separate source packages are
> > > probably the only viable option.
> > >...
> >
> > Separate source packages don't bring any real benefits compared to
> > just bundling and building everything in src:wine.
>
> No please do not do that. Vendoring is bad and I fight on the
> javascript side since four years, against it.
>...
> > > cross-satisfiable for Debian cross-targets, there’s little hope they can stay
> > > cross-satisfiable for Windows! This means that separate source packages are
> > > probably the only viable option.
> > >...
> >
> > Separate source packages don't bring any real benefits compared to
> > just bundling and building everything in src:wine.
>
> No please do not do that. Vendoring is bad and I fight on the
> javascript side since four years, against it.
Separate source packages that duplicate existing sources are as much
vendoring as doing the same inside src:wine.
There are only two realistic options for Wine:
1. building multilib packages from the regular sources
of the libraries in the archive, or
2. vendoring
Noone seems to be willing to do 1., which leaves 2. as only option.
cu
Adrian
Zebediah Figura
Apr 16, 2022, 8:30:03 PM4/16/22
to
Hello all,
Since this conversation several months ago I've been working with the
Wine maintainer on implementing a solution upstream that is compatible
with our requirements and the pretty much universal desire by packagers
to avoid system library imports. I believe I've found a solution that
will work for now for everyone, although it has raised some questions
about standardization of MinGW packages—see the "Open Questions" section
of this mail for more details.
== Summary of upstream progress ==
Currently Wine bundles the source of several libraries. At the same
time, as was discussed we've implemented a system upstream by which
external shared libraries can be linked to and loaded (while keeping
them in a separate "namespace" so that they don't conflict with
identically named application DLLs.) There are still some concerns on
the Wine side that this will break applications in subtle ways, but
unfortunately there has been almost no testing of this feature yet.
The libraries which we import are as follows:
* libFAudio
* libgsm
* libjpeg
* jxrlib
* liblcms2
* libmpg123
* libpng
* libtiff
* libvkd3d
* libxml2
* libxslt
* zlib
This list is probably going to remain stable. Despite what was discussed
earlier I do *not* forsee us importing libOSMesa (it turns out that it
actually makes more sense on the Win32 "kernel" side). An import of
libfreetype or libgnutls is also unlikely at this point (we've found it
more worthwhile to write thunks for both).
We currently do *not* support external static linking. The basic reason
for this is that Wine builds against its own CRT rather than MinGW's,
and we cannot safely statically link to a module built with a different CRT.
== Technical details ==
In order to use system shared libraries, Wine needs to know where to
find them at runtime. This is done with a configure argument:
configure --with-system-dllpath=/first/directory/:/second/directory/
Note that this may need to include the location of libgcc_s and other
dependencies, which is in a different location on Debian.
Separately, for each argument, we need to know the compiler and linker
flags used to find the library. This also can be done with configure
arguments, much like for native libraries:
configure GSM_PE_CFLAGS="-Iwhatever" GSM_PE_LIBS="-Lwhatever -lgsm"
If MinGW pkg-config is present, and --with-system-dllpath is specified,
Wine will automatically use it to detect the compiler and linker flags
for all of the above packages supporting pkg-config, which in particular
excludes libgsm and jxrlib. Thus in practice it should not be necessary
to hand-code most CFLAGS and LIBS variables.
== Open Question(s) ==
Currently it's necessary to use a configure argument to specify where to
find the runtime path. This is fine for distributions, which is where
most people are going to be getting Wine anyway. It's a bit unfortunate
for anyone self-compiling Wine, though, and it'd be nice if there was
some way to avoid it. Unfortunately, there are two things standing in
our way.
The first problem is that the location of runtime DLLs varies wildly
between distributions, and there's no common independent way to detect
it. We could potentially hardcode a few "guesses" at the runtime path
into Wine's configure script, but that brings us to the second problem:
there's no way to verify the presence of runtime DLLs. We *are* the
loader and lower-level APIs and would have to bootstrap ourselves first,
and this is pretty much infeasible. Verifying by name doesn't really
work either, since the name isn't really predictable.
Hence "standardize the location of runtime DLLs across distributions"
isn't really a feasible solution either (or at least not a complete
one), as nice as it would be—we have no way to know whether a
distribution is post-standardization or not.
Rather, I think the best solution is to have some tool which, if present
or successful, lists one or more directories where libraries can be
found. I'm not sure what the most sensible thing here is:
* pkg-config would be an obvious tool to extend, but would presumably
require modifying .pc files for all relevant packages upstream. The
location also seems to be consistent across all packages within each
distribution I was able to test, so there probably isn't much point in
making it package-specific. [For the packages we care about, cmake and
libtool both put things in /bin by default, and zlib apparently has no
built-in cross-compilation support.]
* Introducing an analogue of ld.so.conf seems like a good idea. It would
allow for per-package extra directories, and is relatively easy to
parse, although ideally it would be even easier than that (i.e. if
someone else could deal with the "include" part, that would be great).
ἔρρωσθε,
Zeb Figura
Since this conversation several months ago I've been working with the
Wine maintainer on implementing a solution upstream that is compatible
with our requirements and the pretty much universal desire by packagers
to avoid system library imports. I believe I've found a solution that
will work for now for everyone, although it has raised some questions
about standardization of MinGW packages—see the "Open Questions" section
of this mail for more details.
== Summary of upstream progress ==
Currently Wine bundles the source of several libraries. At the same
time, as was discussed we've implemented a system upstream by which
external shared libraries can be linked to and loaded (while keeping
them in a separate "namespace" so that they don't conflict with
identically named application DLLs.) There are still some concerns on
the Wine side that this will break applications in subtle ways, but
unfortunately there has been almost no testing of this feature yet.
The libraries which we import are as follows:
* libFAudio
* libgsm
* libjpeg
* jxrlib
* liblcms2
* libmpg123
* libpng
* libtiff
* libvkd3d
* libxml2
* libxslt
* zlib
This list is probably going to remain stable. Despite what was discussed
earlier I do *not* forsee us importing libOSMesa (it turns out that it
actually makes more sense on the Win32 "kernel" side). An import of
libfreetype or libgnutls is also unlikely at this point (we've found it
more worthwhile to write thunks for both).
We currently do *not* support external static linking. The basic reason
for this is that Wine builds against its own CRT rather than MinGW's,
and we cannot safely statically link to a module built with a different CRT.
== Technical details ==
In order to use system shared libraries, Wine needs to know where to
find them at runtime. This is done with a configure argument:
configure --with-system-dllpath=/first/directory/:/second/directory/
Note that this may need to include the location of libgcc_s and other
dependencies, which is in a different location on Debian.
Separately, for each argument, we need to know the compiler and linker
flags used to find the library. This also can be done with configure
arguments, much like for native libraries:
configure GSM_PE_CFLAGS="-Iwhatever" GSM_PE_LIBS="-Lwhatever -lgsm"
If MinGW pkg-config is present, and --with-system-dllpath is specified,
Wine will automatically use it to detect the compiler and linker flags
for all of the above packages supporting pkg-config, which in particular
excludes libgsm and jxrlib. Thus in practice it should not be necessary
to hand-code most CFLAGS and LIBS variables.
== Open Question(s) ==
Currently it's necessary to use a configure argument to specify where to
find the runtime path. This is fine for distributions, which is where
most people are going to be getting Wine anyway. It's a bit unfortunate
for anyone self-compiling Wine, though, and it'd be nice if there was
some way to avoid it. Unfortunately, there are two things standing in
our way.
The first problem is that the location of runtime DLLs varies wildly
between distributions, and there's no common independent way to detect
it. We could potentially hardcode a few "guesses" at the runtime path
into Wine's configure script, but that brings us to the second problem:
there's no way to verify the presence of runtime DLLs. We *are* the
loader and lower-level APIs and would have to bootstrap ourselves first,
and this is pretty much infeasible. Verifying by name doesn't really
work either, since the name isn't really predictable.
Hence "standardize the location of runtime DLLs across distributions"
isn't really a feasible solution either (or at least not a complete
one), as nice as it would be—we have no way to know whether a
distribution is post-standardization or not.
Rather, I think the best solution is to have some tool which, if present
or successful, lists one or more directories where libraries can be
found. I'm not sure what the most sensible thing here is:
* pkg-config would be an obvious tool to extend, but would presumably
require modifying .pc files for all relevant packages upstream. The
location also seems to be consistent across all packages within each
distribution I was able to test, so there probably isn't much point in
making it package-specific. [For the packages we care about, cmake and
libtool both put things in /bin by default, and zlib apparently has no
built-in cross-compilation support.]
* Introducing an analogue of ld.so.conf seems like a good idea. It would
allow for per-package extra directories, and is relatively easy to
parse, although ideally it would be even easier than that (i.e. if
someone else could deal with the "include" part, that would be great).
ἔρρωσθε,
Zeb Figura
0 new messages