Bug#845498: /usr/bin/fpc-3.0.0: Provide cross-compilers

[Ben Longbons]

Package: fp-compiler-3.0.0
Version: 3.0.0+dfsg-9
Severity: wishlist
File: /usr/bin/fpc-3.0.0

Dear Maintainer,

According to `fpc -help`,
-P<x> Set target CPU (arm,avr,i386,jvm,m68k,mips,mipsel,powerpc,powerpc64,sparc,x86_64)

However, if I try any of those besides the current CPU, I get:

Error: ppc386 can't be executed, error message: Failed to execute "ppc386", error code: 127

(note: while, multiarch would work for x86_64 -> i386, it won't work for
anything else, so you really do need a native ppc386 binary, etc)

You'll probably have to also tell it to use ${triple}-ld, etc.


-- System Information:
Debian Release: stretch/sid
APT prefers testing-debug
APT policy: (600, 'testing-debug'), (600, 'testing'), (500, 'unstable-debug'), (500, 'unstable'), (1, 'experimental')
Architecture: amd64 (x86_64)
Foreign Architectures: i386, x32

Kernel: Linux 4.7.0-1-amd64 (SMP w/4 CPU cores)
Locale: LANG=en_US.UTF-8, LC_CTYPE=en_US.UTF-8 (charmap=UTF-8)
Shell: /bin/sh linked to /bin/dash
Init: systemd (via /run/systemd/system)

Versions of packages fp-compiler-3.0.0 depends on:
ii binutils 2.27.51.20161108-1
ii debconf [debconf-2.0] 1.5.59
ii fp-units-rtl-3.0.0 3.0.0+dfsg-9
ii libc6 2.24-5

Versions of packages fp-compiler-3.0.0 recommends:
ii fp-utils-3.0.0 3.0.0+dfsg-9

Versions of packages fp-compiler-3.0.0 suggests:
pn fp-docs-3.0.0 <none>
pn mingw32-binutils <none>

-- debconf information excluded

[Abou Al Montacir]

Dear Ben,

Thanks for reporting this bug. We are aware of this limitation and we have a plan for fixing this, but unfortunately we lack man power to put this in place.

One of my own goals is to make it possible to compile Android applications as easy as installing a virtual package that polls everything, which is much more ambitious than compiling for other architectures using multi-arch.

For now you can use multi-arch to install fp-compiler and RTL packages and then just use a shell script to call call the right FPC using qemu while calling the script as expected by fpc binary. Of course this a workaround and native cross compiler would be preferable, but it should work until someone dos package the cross compilers.

-- 
Cheers,
Abou Al Montacir

[Ben Longbons]

On Mon, Nov 28, 2016 at 12:19 AM, Abou Al Montacir
<abou.al...@sfr.fr> wrote:
> For now you can use multi-arch to install fp-compiler

No, you can't (that was the first thing I thought of):
fp-compiler:i386 depends on binutils:i386 rather than
binutils-i586-linux-gnu, and binutils:i386 isn't multiarch
installable. You'd have to do a full cross chroot, currently.

But once the dependency part is fixed, /etc/fpc.cfg can `#INCLUDE
/etc/fpc/$FPCTARGET.cfg` and put `-e/usr/i586-linux-gnu/bin
-Fl/usr/lib/i386-linux-gnu -Fl/lib/i386-linux-gnu -Fl /usr/lib32
-Fl/lib32` in there (each of those .cfg files will have to be manually
written/installed based on the Debian arch (of the fp-compiler
package), the Debian triple (subdir of /usr/lib), the legacy libdir
(/lib32 - actually not sure if this is necessary or not anymore), the
GNU triple (of binutils), and the FPC target (for choice of the
filename)). Incidentally, managing /etc/fpc.cfg through
update-alternatives is a waste since it could be implemented as just
`#INCLUDE /etc/fpc-$fpcversion.cfg` (though since jessie has 2.6.4, an
appropriate upgrade path would need to be written).

The above is fairly trivial and will get you a multiarch "cross"
compiler, with /etc/ ready for real (non-multiarch fp-compiler (I
*think* the libc6-*-cross packages are only needed because of ld.so
conflicting. but fp-units-* are actually multiarch safe already,
they're just not marked as such - they put all their files in
/usr/lib/fpc/$fpcversion/{units,fpmkinst}/$FPCTARGET/ already)) ones.
Then it's just a SMOC to actually build real cross-compilers binary
packages from the Debian source package.

I should probably write a tool to hack-up what I've described above by
using `apt-get download` and extracting/modifying the .deb files.
Maybe test it on Jessie since it has backports to test multiple
*versions* too.

> then just use a shell script to call call the right FPC using qemu

The shell script is unnecessary if you install qemu-user-binfmt (or
qemu-user-static, which `Provides:` it).

-Ben

[Abou Al Montacir]

Hi Ben,

Wow, I'm impressed with your analysis, you really hit many points that are real problem for going multiarch.

The dependency on linker package could be fixed easily as you said.

For the /etc/fpc.cfg, this could be solved by adding liens like:

# path to the gcclib

#ifdef cpui386

-Fl/usr/lib/gcc/x86_64-linux-gnu/6/32

#endif

#ifdef cpux86_64

-Fl/usr/lib/gcc/x86_64-linux-gnu/6

#endif

The file itself is installed by an arch independent package polled by all fp-compiler packages. This way we have a platform config file

Instead of writing a tool to hack all this, I'd propose you submit patches and join the maintain team.

[Paul Gevers]

Control: tags 845504 pending patch

Hi,

On 29-11-16 20:55, Abou Al Montacir wrote:
> The dependency on linker package could be fixed easily as you said.

But how should we do this in reality?
Depends: binutils | binutils-aarch64-linux-gnu |
binutils-alpha-linux-gnu | .... etc?

> For the /etc/fpc.cfg, this could be solved by adding liens like:
>
> # path to the gcclib
>
> #ifdef cpui386
>
> -Fl/usr/lib/gcc/x86_64-linux-gnu/6/32
>
> #endif
>
> #ifdef cpux86_64
>
> -Fl/usr/lib/gcc/x86_64-linux-gnu/6
>
> #endif

Is this instead of what we now add in fp-compiler.postinst? (By the way,
shouldn't we be using ucf for that there?)

> The file itself is installed by an arch independent package polled by
> all fp-compiler packages. This way we have a platform config file

You mean arch-dependent, right? fp-compiler-3.0.0 is now creating and
installing that. Should we revise this then?

> Instead of writing a tool to hack all this, I'd propose you submit
> patches and join the maintain team.

Yes, you are welcome.

@Ben, do you think the multi-arch hinter on the tracker is correct
(action needed section of: https://tracker.debian.org/pkg/fpc)

Paul
PS: I'll upload soon for the other bug and stuff that is already
pending. Would be nice if we could get further on this bug as well.

[Paul Gevers]

Hi,

On 29-11-16 02:25, Ben Longbons wrote:
> On Mon, Nov 28, 2016 at 12:19 AM, Abou Al Montacir
> <abou.al...@sfr.fr> wrote:
>> For now you can use multi-arch to install fp-compiler
>
> No, you can't (that was the first thing I thought of):
> fp-compiler:i386 depends on binutils:i386 rather than
> binutils-i586-linux-gnu, and binutils:i386 isn't multiarch
> installable. You'd have to do a full cross chroot, currently.

Diving into this a little deeper. binutils-i586-linux-gnu is not
available in stretch, it seems most binutils-* packages are now build by
binutils, except the i586 and x86-64 variants, which are build by
cross-binutils. So before we go that route, we should probably first as
for builds of those packages. I was thinking of something like:

Depends: binutils | binutils-aarch64-linux-gnu [ppc64] |
binutils-i586-linux-gnu [i386] |

Would that work to fix this issue?

Paul

[Ben Longbons]

Okay, I've got it *almost* working:
https://gist.github.com/o11c/cf98115ba716ebdd1dc2cc75b290f321

I'm still getting errors from update-alternatives in postinst, but I
*think* everything else is right - at least, things that weren't
completely wrong before (there are a lot of those).

I have SUCCESSFULLY produced 32-bit binaries on a 64-bit host, but
that doesn't prove the /etc/ stuff is right. Need to add testcases,
and also force linking with libc.

Hopefully this answers the various questions people have had, e.g.
about what we need to do about binutils.

There are a LOT of TODOs, referring to various bugs (or things) in
various packages (or things), in the script. I suggest you guys look
at them.

Oh, and I haven't tested phase3 with anything besides --dist=stretch.

I will be updating the gist, so remember to `git pull` before.

On Tue, Nov 29, 2016 at 11:55 AM, Abou Al Montacir
<abou.al...@sfr.fr> wrote:
> Instead of writing a tool to hack all this, I'd propose you submit patches
> and join the maintain team.

I'd rather pull my own teeth out with a rusty spoon. Debian packaging
is beyond all doubts the worst I've ever played with - nearly entirely
due to the "in-source" rather than "out-of-source" nature of the
packaging (which is also extremely hostile to upstreams).

Compare e.g. The Fedora documentation [1] - it is *far* simpler for
newcomers, and this is *not* a matter of documentation. (I used to
mention Gentoo since that was my first, but that always makes people
get distracted by the totally-irrelevant fact that the Gentoo project
doesn't *ship* the binary packages). For that matter, even NixOS
packages[2], which live in an utterly alien environment, are easier to
get into than Debian packages.

[1]: https://docs.fedoraproject.org/en-US/Fedora_Draft_Documentation/0.1/html/Packagers_Guide/sect-Packagers_Guide-Creating_a_Basic_Spec_File.html
[2]: https://nixos.org/nixpkgs/manual/

[Ben Longbons]

I got it completely working now! I did have to repack
binutils-{i586,x86-64}-linux-gnu though.

Tested that I can generate both i386 and aarch64 binaries, solely by
specifying `-P`. Still haven't actually tested linking with libc, for
that we'll need to do something nasty about /lib32/ (probably just
some ifdefs based on the *host* platform - need to extend the arch
tables for that)

For packaging, you'll almost certainly want to split up the
fp-compiler bits into 2-5 packages
(fp-compiler-{bin,driver,config,config-$fpc_target}). Note that the
new /etc/fpc/debian.cfg must be installed from the *unversioned*
package - which will require a "backwards" dependency
(fp-compiler-config-3.0.0 depends on fp-compiler-config-common).

Updated the GIST at the same URL:

https://gist.github.com/o11c/cf98115ba716ebdd1dc2cc75b290f321

[Paul Gevers]

Hi Ben

On 03-12-16 03:39, Ben Longbons wrote:
> I got it completely working now! I did have to repack
> binutils-{i586,x86-64}-linux-gnu though.
>
> Tested that I can generate both i386 and aarch64 binaries, solely by
> specifying `-P`. Still haven't actually tested linking with libc, for
> that we'll need to do something nasty about /lib32/ (probably just
> some ifdefs based on the *host* platform - need to extend the arch
> tables for that)

I am trying to understand you shell script but it contains lots and lots
of overhead which doesn't make reading it easy (the interesting stuff
only starts at line 725). Just to make sure I am not completely
misunderstanding, you are trying to use binutils-<gnu-triplet> package
on the arch of that triplet, right? But e.g. binutils-aarch64-linux-gnu
doesn't exist on amd64. So am I not reading the script correctly or how
is that supposed to work?

> Note that the
> new /etc/fpc/debian.cfg must be installed from the *unversioned*
> package - which will require a "backwards" dependency
> (fp-compiler-config-3.0.0 depends on fp-compiler-config-common).

Can you explain where this requirement comes from? If really required,
then we'll have to figure out an other solution, because circular
dependencies are a problem.

Paul

[Ben Longbons]

On Fri, Dec 9, 2016 at 1:46 PM, Paul Gevers <elb...@debian.org> wrote:
> I am trying to understand you shell script

You may find it easier to just run it and inspect the resulting
`.deb`s, then refer to the script only when you want to see where a
specific path/package is handled.

> Just to make sure I am not completely
> misunderstanding, you are trying to use binutils-<gnu-triplet> package
> on the arch of that triplet, right? But e.g. binutils-aarch64-linux-gnu
> doesn't exist on amd64.

Er, yes it does:
https://packages.debian.org/search?keywords=binutils-aarch64-linux-gnu

The *only* packages that are "missing" (actually just outdated and
with wrong headers) are the two x86 arches that are still from
src:cross-binutils rather than src:binutils, for which I filed bug
#846628.

>> Note that the
>> new /etc/fpc/debian.cfg must be installed from the *unversioned*
>> package - which will require a "backwards" dependency
>> (fp-compiler-config-3.0.0 depends on fp-compiler-config-common).
>
> Can you explain where this requirement comes from? If really required,
> then we'll have to figure out an other solution, because circular
> dependencies are a problem.

Background: managing /etc/fpc.cfg via update-alternatives is
fundamentally wrong, because it is the file read by *all* installed
versions of fpc. In order for each FPC to use its *own* fpc.cfg, they
all need to be conditionally included from a *single* fpc.cfg. Since
jessie shipped with packages that manage /etc/fpc.cfg via
update-alternatives, the symlink must still be managed by it for the
stretch upgrade (for the stretch -> buster upgrade this will no longer
be the case).

Fix: Regardless of version, make /etc/fpc.cfg a symlink to
/etc/fpc/debian.cfg file, which then includes the files specific to
the arch and version.

So the hard dependencies will be:

fp-compiler:$a -> fp-compiler-$v:$a
fp-compiler-$v:$a -> fp-compiler-config-$v:all,
fp-compiler-driver-$v(:$a, but Multi-Arch:foreign)
fp-compiler-config-$v:all -> fp-compiler-config-common:all

(Additionally, fp-compiler-driver-$v should have a backwards
Recommends: fp-compiler-$v:$a and a Description to match, so that
people finding it via `apt-file` (including `command-not-found`) will
get the right thing.)

There is no circular dependency - only the -common package has a
versioned -> nonversioned dependency. And the contents will be:

fp-compiler:$a : empty
fp-compiler-$v:$a : executable /usr/lib/fpc/$v/ppc$a
fp-compiler-driver-$v:$a : executable /usr/lib/fpc/$v/fpc, symlink
/usr/bin/fpc-$v and sometimes (via update-alternatives) /usr/bin/fpc
fp-compiler-config-$v:all : /etc/fpc/$v/{mkcfg,local}.cfg
fp-compiler-config-common:all : /etc/fpc/{debian,local}.cfg, *all*
/etc/fpc/$a/{target,local}.cfg and (via update-alternatives)
/etc/fpc.cfg

It would be possible to put the /etc/fpc/$a/{target,local}.cfg files
in yet *another* package, but IMO there's no value in it. (They are
unversioned so that can't go in fp-compiler:$a which might not be
installed if just fp-compiler-$v:$a is).

While it would *currently* be possible to make fp-compiler-config-$v
architecture-specific (since multi-arch allows overwriting files as
long as they are identical), that would prove to be a mistake once
"real" cross-compiler packages are added. By avoiding relying on this,
it becomes easier to transition from *:$a to *-$a packages in future.

All `Architecture: all` packages should be `Multi-Arch: foreign`.
All `Architecture: $a` packages should be `Multi-Arch: same` *except*
`fp-compiler-driver{,-$v}`, `fp-utils{,-$v}`, and `fp-ide{,-$v}` which
should be `Multi-Arch: foreign` since they only make sense on the
host. (The future fp-compiler{,-$v}-$a packages will also be
`Multi-Arch: foreign`).

-Ben

[Abou Al Montacir]

Hi Ben & All,

On Fri, 2016-12-09 at 20:40 -0800, Ben Longbons wrote:

On Fri, Dec 9, 2016 at 1:46 PM, Paul Gevers <elb...@debian.org> wrote:

I am trying to understand you shell scrip

You may find it easier to just run it and inspect the resulting
`.deb`s, then refer to the script only when you want to see where a
specific path/package is handled.

Sorry but this is not the way we accept contributions. We have a big responsibility against our users to completely ensure the packages we ship are as safe. as possible This means that we have the duty to inspect all changes coming from new contributers. Indeed, a big data damage could result on some bad change that may appear on some system due to some corner cases. Please don't understand this as if I'm not saying you have bad intension. I'm just saying that what Paul is doing is the way any Debian mentors shall proceed according to the DSC.

...

Note that the
new /etc/fpc/debian.cfg must be installed from the *unversioned*
package - which will require a "backwards" dependency
(fp-compiler-config-3.0.0 depends on fp-compiler-config-common).


Can you explain where this requirement comes from? If really required,
then we'll have to figure out an other solution, because circular
dependencies are a problem.


Background: managing /etc/fpc.cfg via update-alternatives is
fundamentally wrong, because it is the file read by *all* installed

I agree with you here even if when I implemented that, the include statement was not present in FPC. That was very long time ago you know!

versions of fpc. In order for each FPC to use its *own* fpc.cfg, they
all need to be conditionally included from a *single* fpc.cfg. Since

This is a better solution indeed

jessie shipped with packages that manage /etc/fpc.cfg via
update-alternatives, the symlink must still be managed by it for the
stretch upgrade (for the stretch -> buster upgrade this will no longer
be the case).

Looks good proposal

Fix: Regardless of version, make /etc/fpc.cfg a symlink to
/etc/fpc/debian.cfg file, which then includes the files specific to
the arch and version.

No sure we need this /etc/fpc/debian.cfg if we remove the alternatives in pre-installation step, but why not. Maybe better call it /etc/fpc/fpc.cfg

So the hard dependencies will be:

fp-compiler:$a -> fp-compiler-$v:$a
fp-compiler-$v:$a -> fp-compiler-config-$v:all,
fp-compiler-driver-$v(:$a, but Multi-Arch:foreign)
fp-compiler-config-$v:all -> fp-compiler-config-common:all

(Additionally, fp-compiler-driver-$v should have a backwards
Recommends: fp-compiler-$v:$a and a Description to match, so that
people finding it via `apt-file` (including `command-not-found`) will
get the right thing.)

There is no circular dependency - only the -common package has a
versioned -> nonversioned dependency. And the contents will be:

fp-compiler:$a : empty
fp-compiler-$v:$a : executable /usr/lib/fpc/$v/ppc$a
fp-compiler-driver-$v:$a : executable /usr/lib/fpc/$v/fpc, symlink
/usr/bin/fpc-$v and sometimes (via update-alternatives) /usr/bin/fpc
fp-compiler-config-$v:all : /etc/fpc/$v/{mkcfg,local}.cfg
fp-compiler-config-common:all : /etc/fpc/{debian,local}.cfg, *all*
/etc/fpc/$a/{target,local}.cfg and (via update-alternatives)
/etc/fpc.cfg

I don't understand why do you need a new package. The fpc configuration file does not belong to any package, it is created on the fly upon installation.

Can you please explain more your point?

It would be possible to put the /etc/fpc/$a/{target,local}.cfg files
in yet *another* package, but IMO there's no value in it. (They are
unversioned so that can't go in fp-compiler:$a which might not be
installed if just fp-compiler-$v:$a is).

That does not make too much sense.

While it would *currently* be possible to make fp-compiler-config-$v
architecture-specific (since multi-arch allows overwriting files as
long as they are identical), that would prove to be a mistake once
"real" cross-compiler packages are added. By avoiding relying on this,
it becomes easier to transition from *:$a to *-$a packages in future.

I'm not fan of overwriting file.

All `Architecture: all` packages should be `Multi-Arch: foreign`.
All `Architecture: $a` packages should be `Multi-Arch: same` *except*
`fp-compiler-driver{,-$v}`, `fp-utils{,-$v}`, and `fp-ide{,-$v}` which
should be `Multi-Arch: foreign` since they only make sense on the
host. (The future fp-compiler{,-$v}-$a packages will also be
`Multi-Arch: foreign`).

OK on that

[Abou Al Montacir]

Hi Paul and All,

On Wed, 2016-11-30 at 22:29 +0100, Paul Gevers wrote:

Control: tags 845504 pending patch

Hi,

On 29-11-16 20:55, Abou Al Montacir wrote:

The dependency on linker package could be fixed easily as you said.
But how should we do this in reality?
Depends:  binutils | binutils-aarch64-linux-gnu |
binutils-alpha-linux-gnu | .... etc?

What about adding a new package fpc-cross that adds support for cross compiling with fpc by pulling all these linkers or maybe add a set of fpc-cross-${arch}. This way if I want to enable cross compiling for armel on amd64 then I would install fpc-cross-armel

#ifdef cpui386

-Fl/usr/lib/gcc/x86_64-linux-gnu/6/32

#endif

#ifdef cpux86_64

-Fl/usr/lib/gcc/x86_64-linux-gnu/6

#endif
Is this instead of what we now add in fp-compiler.postinst? (By the way,
shouldn't we be using ucf for that there?)

I think there is a better solution. For now we have this at the end of /etc/fpc.cfg:

# Third party units should be installe in a, multi-arch compatible location.
# Units should be installed in /usr/lib/$fpctarget-gnu/fp-units-2.6.2/$pkg/.
# Ech fp-units package should install a configuration file called $pkg.cfg in
#CFGDIR /etc/fpc-$fpcversion.cfg.d/$fpctarget 

This means that every file there will be included.

If we place the regular config file there and we surround it by #ifdef ${arch} then we can fix this without having all arch configs in the same file

If this is not clear I explain: we replace fpc.cfg to become a regular file that contains only #CFGDIR directive. Each package will install its config file while paying attention to surround all its config with the correct #ifdef. This allows the config to be extended automatically when one installs a foreign compiler.

The file itself is installed by an arch independent package polled by
all fp-compiler packages. This way we have a platform config file
You mean arch-dependent, right? fp-compiler-3.0.0 is now creating and
installing that. Should we revise this then?

Yes, sorry for this lapsus

Instead of writing a tool to hack all this, I'd propose you submit
patches and join the maintain team.
Yes, you are welcome.

@Ben, do you think the multi-arch hinter on the tracker is correct
(action needed section of: https://tracker.debian.org/pkg/fpc)

Paul
PS: I'll upload soon for the other bug and stuff that is already
pending. Would be nice if we could get further on this bug as well.

Thanks for the hard work you are doing.

[Abou Al Montacir]

Hi Paul, and All,

I've just pushed a commit [1] that I hope will improve the situation of MA
support.

In this commit I've moved units from ${LIB_DIR}/units/${FPCTARGET}-${FPCARCH}
to ${LIB_DIR}/units/${FPCTARGET}-${FPCARCH}/'$${PACKAGE_NAME}'

where 
LIB_DIR=/usr/lib/${DEB_PACKAGE_NAME}/${DEB_UPSTREAM_MAIN_VERSION}
FPCTARGET=$(CPU_TARGET)-$(OS_TARGET)
FPCARCH=$(shell dpkg-architecture -qDEB_BUILD_ARCH_ABI)

This makes each architecture installs its units in a separate directory.

I've also added a patch against fpcmkcfg to change /etc/fpc.cfg to look for the
units in the right place depending on the cpu/os tragets and the used ARCH/ABI.

I did not upload this as I did not test it extensively and I fear this is a
quite big change that may require manual upload for many arches is not tested
extensively before it is uploaded. I'd prefer to have it in experimental first
also in order to avoid annoyance to our users.

Please let me know what do you think about it.

[1]: https://anonscm.debian.org/cgit/pkg-pascal/fpc.git/commit/?id=a5bfc6ba0fa81
c9e1d627d9314078cb1ddb3d329

[peter green]

Going to experimental before unstable with aggressive changes certainly makes sense. IIRC experimental buildds only use build-depends from experimental when required to satisfy version constraints, so by changing version constraints one can choose whether to build a new version in experimental with the previous version from experimental or with the known-good version from unstable.

However I am skeptical as to whether such an aggressive change is really needed. IMO given the relatively small scale of this problem it makes more sense to leave most architectures alone and only deviate from upstream where we actually need to do so.

I just ran a quick check and currently the only architectures with a conflict are armel and armhf. It seems likely ppc64el would also have a conflict but IMO we can cross that bridge when we come to it.

[Abou Al Montacir]

Hi Peter and All,

On Tue, 2017-11-14 at 23:32 +0000, peter green wrote:

Going to experimental before unstable with aggressive changes certainly makes sense. IIRC experimental buildds only use build-depends from experimental when required to satisfy version constraints, so by changing version constraints one can choose whether to build a new version in experimental with the previous version from experimental or with the known-good version from unstable.

Yes, I think this is the way to go in order to avoid the need of re-uploading binary packages in cas anything goes wrong.

However I am skeptical as to whether such an aggressive change is really needed. IMO given the relatively small scale of this problem it makes more sense to leave most architectures alone and only deviate from upstream where we actually need to do so.

In fact, the biggest interest of MA for FPC is to build for arm and especially to cross build for Android. So I fear that this is not a tiny goal. Otherwise we drop MA support which will be a pity.

I just ran a quick check and currently the only architectures with a conflict are armel and armhf. It seems likely ppc64el would also have a conflict but IMO we can cross that bridge when we come to it.

This is indeed the mail problem. People are asking for better MA support to be able to use FPC for their phone development and here arm architectures are important.

Upstream solves this by using a different base dir. We can for instance replace

/usr/lib/${DEB_PACKAGE_NAME}/${DEB_UPSTREAM_MAIN_VERSION}
by
/usr/lib/${FPCTARGET}-${FPCARCH}/${DEB_PACKAGE_NAME}/${DEB_UPSTREAM_MAIN_VERSION}
This is probably less intrusive, but have 2 levels of ${FPCTARGET}

[Paul Gevers]

Hi Abou,

On 14-11-17 23:21, Abou Al Montacir wrote:
> I did not upload this as I did not test it extensively and I fear this is a
> quite big change that may require manual upload for many arches is not tested
> extensively before it is uploaded. I'd prefer to have it in experimental first
> also in order to avoid annoyance to our users.
>
> Please let me know what do you think about it.

I am perfectly fine with you experimenting in experimental.

I have a question regarding the patch, as you change generated *.inc
files, can't you generate those in the rules file instead of in the
patch? My experience is that if upstream isn't going to carry the patch
it is hard to update the patch for files like for that every new
upstream release. Also, I am not sure if we aren't already rebuilding
all generated *.inc files in our current build process. I have stripped
major blocks for our patches in the past, because they were totally
unneeded as the file was (or could be) regenerated.

Paul

PS: I would have appreciated it when you would have committed this on a
separate (experimental) branch.

[Abou Al Montacir]

Hi Paul,

On Wed, 2017-11-15 at 21:01 +0100, Paul Gevers wrote:

I am perfectly fine with you experimenting in experimental.

I have a question regarding the patch, as you change generated *.inc
files, can't you generate those in the rules file instead of in the
patch?

That's a very good question and I've asked myself why not regenerate the .inc file? The issue is that if we do then we end with a possible source change that is not carried in a patch and this does not please to dpkg-source (or some other dpkg-* tools I don't remember well anymore). So either we strip this file from original tar and force regenerating it or we patch it. I'm open to both and know how to regenerate this file.

 My experience is that if upstream isn't going to carry the patch
it is hard to update the patch for files like for that every new
upstream release.

Yes this is obvious, but updating the patch can be as simple as: apply the patch to real source file and then regenerate inc file and then update the patch (kind of quilt commit)

Also, I am not sure if we aren't already rebuilding
all generated *.inc files in our current build process. I have stripped
major blocks for our patches in the past, because they were totally
unneeded as the file was (or could be) regenerated.

We are rebuilding some of them.

I think many of them are automatically regenerated by upstream make files.

Paul

PS: I would have appreciated it when you would have committed this on a
separate (experimental) branch.

Sorry I didn't think about it. we can cherry-pick to experimental and then push -f to master~, but seems our git config refuse forced update.

Do you have a clue other than git revert? otherwise it is also fine.

[Abou Al Montacir]

Hi All,

On Wed, 2017-11-15 at 15:33 +0100, Abou Al Montacir wrote:

...

Upstream solves this by using a different base dir. We can for instance replace

/usr/lib/${DEB_PACKAGE_NAME}/${DEB_UPSTREAM_MAIN_VERSION}
by
/usr/lib/${FPCTARGET}-${FPCARCH}/${DEB_PACKAGE_NAME}/${DEB_UPSTREAM_MAIN_VERSION}
This is probably less intrusive, but have 2 levels of ${FPCTARGET}

I've uploaded this WE to experimental a solution that puts files in

/usr/lib/${DEB_PACKAGE_NAME}/${DEB_UPSTREAM_MAIN_VERSION}/units/${FPCTARGET}-${DEB_ABI}/${FPC_PACKAGE_NAME}

For example:

/usr/lib/fpc/3.0.4/units/i386-linux-base/fpmkunit

However this location of unit files produces the following Lintian errors:

https://lintian.debian.org/tags/arch-dependent-file-not-in-arch-specific-directory.html

I suppose that I need to change it again to be

/usr/lib/${DEB_TARGET_GNU_TYPE}/${DEB_PACKAGE_NAME}/${DEB_UPSTREAM_MAIN_VERSION}/units/${FPC_PACKAGE_NAME}

So for example

/usr/lib/i386-linux-gnu/fpc/3.0.4/units/fpmkunit

It is now quite easy to do this after my recent changes.

Is that OK or do we need to do it other way, maybe in a simpler manner?

[Paul Gevers]

Hi Abou,

On 06-12-17 10:26, Abou Al Montacir wrote:
> So for example
> /usr/lib/i386-linux-gnu/fpc/3.0.4/units/fpmkunit
> It is now quite easy to do this after my recent changes.
>
> Is that OK or do we need to do it other way, maybe in a simpler manner?

For whatever it is worth, that sounds correct to me.

Paul

[Abou Al Montacir]

Hi Paul,

This is now done, we now have unit packages co-installable. However, There is still an issue with fp-compiler package. It contains some executables that are installed in /usr/bin while is is MA same according to this report.

As far as I can see fpc-3.0.4, fpcmkcfg-3.0.4, and fpcres-3.0.4 are tools and should be moved to fp-utils. In this case I'd make fp-compiler recommend fp-utils.

Then there is grab_vcsa-3.0.4 which is a tool that is specific to video unit, and that shall be installed by fp-units-base, but this is not my preference as this just moves the problem to that package. I would rather move grab_vcsa-3.0.4 to fp-utils, and make the fp-units-base recommend fp-utils (maybe not needed as it recommends fp-compiler).

What do you think about this proposal?

[Paul Gevers]

Hi Abou,

On 11-12-17 10:20, Abou Al Montacir wrote:
> This is now done, we now have unit packages co-installable. However,
> There is still an issue with fp-compiler package. It contains some
> executables that are installed in /usr/bin while is is MA same according
> to this report

> <https://lintian.debian.org/maintainer/pkg-pasc...@lists.alioth.debian.org.html#fpc>.

>
> As far as I can see fpc-3.0.4, fpcmkcfg-3.0.4, and fpcres-3.0.4 are
> tools and should be moved to fp-utils. In this case I'd make fp-compiler
> recommend fp-utils.
> Then there is grab_vcsa-3.0.4 which is a tool that is specific to video
> unit, and that shall be installed by fp-units-base, but this is not my
> preference as this just moves the problem to that package. I would
> rather move grab_vcsa-3.0.4 to fp-utils, and make the fp-units-base
> recommend fp-utils (maybe not needed as it recommends fp-compiler).
>
> What do you think about this proposal?

I don't see obvious flaws in your proposal so I think it is OK.

But I think you didn't fix this in your upload to unstable, so Lintian
is now complaining loudly about all this, as is the MA hinter:
https://lintian.debian.org/maintainer/pkg-pasc...@lists.alioth.debian.org.html#fpc
and
https://tracker.debian.org/pkg/fpc

Just curious, why did you upload to unstable without fixing these issues
first?

Paul

[Abou Al Montacir]

Hi Paul

On Fri, 2017-12-15 at 09:26 +0100, Paul Gevers wrote:

Just curious, why did you upload to unstable without fixing these issues
first?

As I explained in my previous message, this will need to take a decision on either moving files to other packages or to put them in /usr/bin/<triplet>.

I don't generally like big changes in one upload, so I preferred an intermediate step that moves only unit files.

Also Lasarus new version is being released and I'd like the the new units path get used in the new release. However on that one I've missed that you already uploaded it few days ago!

[Paul Gevers]

Hi Abou,

On 15-12-17 09:26, Paul Gevers wrote:
> But I think you didn't fix this in your upload to unstable, so Lintian
> is now complaining loudly about all this, as is the MA hinter:
> https://lintian.debian.org/maintainer/pkg-pasc...@lists.alioth.debian.org.html#fpc
> and
> https://tracker.debian.org/pkg/fpc

On top of this, the autopkgtest now seems broken (both in Debian and
Ubuntu):
https://ci.debian.net/packages/f/fpc/unstable/amd64/
http://autopkgtest.ubuntu.com/packages/f/fpc (they were green in bionic)

Paul

[Abou Al Montacir]

Hi Paul,

On top of this, the autopkgtest now seems broken (both in Debian and
Ubuntu):
https://ci.debian.net/packages/f/fpc/unstable/amd64/
http://autopkgtest.ubuntu.com/packages/f/fpc (they were green in bionic)

This looks like a make file was not generated.

# Make files are now generated.
touch makefiles-stamp
make: Entering directory '/tmp/autopkgtest-lxc.tockqviv/downtmp/build.dcy/src/fpcsrc/tests'
make: *** No rule to make target 'create_c_objects'.  Stop.
make: Leaving directory '/tmp/autopkgtest-lxc.tockqviv/downtmp/build.dcy/src/fpcsrc/tests'

I was not aware of this, but I suppose that the fix is easy, I just need to call fpcmake for that direcotry.

[Helmut Grohne]

Hi,

I'm coming late to the party and I only understand a fraction of what
you wrote, but the parts I do understand make a lot of sense.

On Mon, Nov 28, 2016 at 05:25:14PM -0800, Ben Longbons wrote:
> On Mon, Nov 28, 2016 at 12:19 AM, Abou Al Montacir
> <abou.al...@sfr.fr> wrote:
> > For now you can use multi-arch to install fp-compiler
>
> No, you can't (that was the first thing I thought of):
> fp-compiler:i386 depends on binutils:i386 rather than
> binutils-i586-linux-gnu, and binutils:i386 isn't multiarch
> installable. You'd have to do a full cross chroot, currently.

I've read the whole discussion and I'm getting the impression that there
are two distinct issues that are conflated inside this bug.

One one hand, it would be nice to be able to install a foreign
fp-compiler to be able to call an emulated (via qemu) compiler. On the
other hand it would be nice to be able to install a cross compiler. And
then there is a third issue not otherwise mentioned, which is cross
building the compiler itself.

All of them are nice and they interact somewhat with one another, but it
helps to tell these issues apart.

> But once the dependency part is fixed, /etc/fpc.cfg can `#INCLUDE
> /etc/fpc/$FPCTARGET.cfg` and put `-e/usr/i586-linux-gnu/bin
> -Fl/usr/lib/i386-linux-gnu -Fl/lib/i386-linux-gnu -Fl /usr/lib32
> -Fl/lib32` in there (each of those .cfg files will have to be manually
> written/installed based on the Debian arch (of the fp-compiler
> package), the Debian triple (subdir of /usr/lib), the legacy libdir
> (/lib32 - actually not sure if this is necessary or not anymore), the
> GNU triple (of binutils), and the FPC target (for choice of the
> filename)). Incidentally, managing /etc/fpc.cfg through
> update-alternatives is a waste since it could be implemented as just
> `#INCLUDE /etc/fpc-$fpcversion.cfg` (though since jessie has 2.6.4, an
> appropriate upgrade path would need to be written).

This is the part where I only understand very little. The thing I do
understand is that there is a fpc.cfg that influences how the compiler
behaves and given the "right" fpc.cfg it can mostly act as a cross
compiler.

As far as I understand though, there is a major piece missing in the
Debian package to make this reality. I've run fpc hello.pas under strace
and I observe that it calls out to ppcx64. If I were to cross compile
for armel, it would likely call into ppcarm, but there is no ppcarm
binary in any :amd64 package. So regardles how we configure it, we'll
need more binaries to make this happen. Please tell if you think this is
wrong.

> The above is fairly trivial and will get you a multiarch "cross"
> compiler, with /etc/ ready for real (non-multiarch fp-compiler (I
> *think* the libc6-*-cross packages are only needed because of ld.so
> conflicting. but fp-units-* are actually multiarch safe already,
> they're just not marked as such - they put all their files in
> /usr/lib/fpc/$fpcversion/{units,fpmkinst}/$FPCTARGET/ already)) ones.
> Then it's just a SMOC to actually build real cross-compilers binary
> packages from the Debian source package.

I'm not sure yet how you solve the missing binaries, but I'm all for
having cross compilers in the archive.

The initial point was switching the binutils dependency to
binutils-$triplet. Since binutils 2.30-6, we have binutils-for-host and
that means that we always have a binutils-$triplet:$arch where arch and
triplet match. So if you are ok with ignoring oldstable, you can now
depend on the triplet variant without issues.

However, I think that doing so is technically wrong. When I straced fpc,
I saw that it calls into ld.bfd. binutils-x86-64-linux-gnu does not
contain ld.bfd, so the dependency is too weak for what fpc uses. In
order for a binutils-$triplet to be a correct dependency, we'd have to
somehow tell fpc to always invoke triplet-prefixed tools such as
x86_64-linux-gnu-ld.bfd.

And given the pointer to fpc.cfg, I looked into it and stumbled into the
-XP option. It is conditional to cross compiling, but given Debian's
packaging of binutils, it is also relevant to foreign installation.
We'll need to teach fpc to always use the -XP option even natively. Once
that is done, we can replace the binutils dependency with
"binutils-$triplet" or if you require backwards compatibility with
stretch "binutils-$triplet | binutils". Once doing these two bits,
foreign installation should become possible.

The next part was fpc as a cross compiler. I think that one of the first
pieces to getting there is ensuring that all the compiler backend
binaries (e.g. ppcx86 or ppcarm) are built for all architectures. This
does not happen today. Any ideas on how to get there?

A separate problem (at least for me) is figuring out what the API to
cross building with fpc is. What is a builder expected to do to build
natively or to cross build?

How to build sources for some architecture that can be run (aka build
archtecture)?
* fpc answer: Run plain fpc.
* gcc answer: Run plain gcc. In future, this requires a dependency on
gcc-for-build.
* clang answer: Run plain clang.

How to build sources for architecture $x with triplet $y?
* fpc answer: ???
* gcc answer: Run $y-gcc. This requires a dependency on
gcc-$y. In future, a dependency on gcc-for-host:$x will also suffice.
* clang answer: Run clang -target $y.

Can someone fill in the fpc answer? I kinda suspect that it is "fpc
-P$z" for some $z that can be computed from $x or $y and the o11c gist
has such a mapping.

Depending on the answer, the packaging layout may need to change. So
I'll stop here.

Hope this helps

Helmut

[Abou Al Montacir]

Hi Helmut,

On Tue, 2021-01-12 at 22:05 +0100, Helmut Grohne wrote:

Hi,

I'm coming late to the party and I only understand a fraction of what

you wrote, but the parts I do understand make a lot of sense.

Better late than never. I'm surprised by the analysis you made, it is quite deep and correct.

On Mon, Nov 28, 2016 at 05:25:14PM -0800, Ben Longbons wrote:

On Mon, Nov 28, 2016 at 12:19 AM, Abou Al Montacir

<

abou.al...@sfr.fr

> wrote:

For now you can use multi-arch to install fp-compiler

No, you can't (that was the first thing I thought of):

fp-compiler:i386 depends on binutils:i386 rather than

binutils-i586-linux-gnu, and binutils:i386 isn't multiarch

installable. You'd have to do a full cross chroot, currently.

I've read the whole discussion and I'm getting the impression that there

are two distinct issues that are conflated inside this bug.

Maybe even three as you wrote below.

One one hand, it would be nice to be able to install a foreign

fp-compiler to be able to call an emulated (via qemu) compiler. On the

This was the idea. But not only, as the foreign RTL should be usable by both a foreign or a native cross compiler.

other hand it would be nice to be able to install a cross compiler. And

This will require more work on the packaging and I should admit that the current bandwidth of the team is so low that I fear it will be very difficult. Of course if someone, with enough free time, volunteers I'll be very happy to help. But I really have very little time for FPC currently.

then there is a third issue not otherwise mentioned, which is cross

building the compiler itself.

For me this is not very useful for users, but only for bootstrapping a new target. We can ignore it I think.

All of them are nice and they interact somewhat with one another, but it

helps to tell these issues apart.

Maybe a good start is to split this ticket into several ones, each dealing with a unique feature.

But once the dependency part is fixed, /etc/fpc.cfg can `#INCLUDE

/etc/fpc/$FPCTARGET.cfg` and put `-e/usr/i586-linux-gnu/bin

-Fl/usr/lib/i386-linux-gnu -Fl/lib/i386-linux-gnu -Fl /usr/lib32

-Fl/lib32` in there (each of those .cfg files will have to be manually

written/installed based on the Debian arch (of the fp-compiler

package), the Debian triple (subdir of /usr/lib), the legacy libdir

(/lib32 - actually not sure if this is necessary or not anymore), the

GNU triple (of binutils), and the FPC target (for choice of the

filename)). Incidentally, managing /etc/fpc.cfg through

update-alternatives is a waste since it could be implemented as just

`#INCLUDE /etc/fpc-$fpcversion.cfg` (though since jessie has 2.6.4, an

appropriate upgrade path would need to be written).

This is the part where I only understand very little. The thing I do

understand is that there is a fpc.cfg that influences how the compiler

behaves and given the "right" fpc.cfg it can mostly act as a cross

compiler.

fpc.cfg is a file that gives the path for the RTL (Run Time Library). On Debian we added a new feature for the compiler to support a #INCLUDE directive. That one can be used for architecture specific directories. The paths issue is not only for RTL, but also for the system libraries and the linker configuration. So a deep analysis is needed. Not sure I have the time to do it for now. But If done, I can help pointing to the code where we can hack to fix something either in compiler or in the build system.

As far as I understand though, there is a major piece missing in the

Debian package to make this reality. I've run fpc hello.pas under strace

and I observe that it calls out to ppcx64. If I were to cross compile

for armel, it would likely call into ppcarm, but there is no ppcarm

binary in any :amd64 package. So regardles how we configure it, we'll

need more binaries to make this happen. Please tell if you think this is

wrong.

fpc is a pure wrapper that calls an architecture compiler called ppc$TARGET. ppcx64 is native compiler in x86_64, but allows to compile for all system that run on this architecture (windows, FreeBSD, ...) as long as related RTL is available.

If you want to compile for arm, then it will call ppcarm, which is an arm architecture compiler. It can be a cross compiler, but it can be a shell script that calls the native foreign ppcarm with quemu. This needs to be further analyzed for pros and cons. The same ppcarm should also be able to compile for all arm related OSes like Android ...

However there is still a problem to be solved as we call ppcarm the compiler for armel and armhf. So there will be a need to hack this in fpc code and send the patch to upstream.

The above is fairly trivial and will get you a multiarch "cross"

compiler, with /etc/ ready for real (non-multiarch fp-compiler (I

*think* the libc6-*-cross packages are only needed because of ld.so

conflicting. but fp-units-* are actually multiarch safe already,

they're just not marked as such - they put all their files in

/usr/lib/fpc/$fpcversion/{units,fpmkinst}/$FPCTARGET/ already)) ones.

Then it's just a SMOC to actually build real cross-compilers binary

packages from the Debian source package.

I'm not sure yet how you solve the missing binaries, but I'm all for

having cross compilers in the archive.

The missing binaries can be first emulated by qemu with foreign compilers, and later with a native cross compiler it ever needed.

The initial point was switching the binutils dependency to

binutils-$triplet. Since binutils 2.30-6, we have binutils-for-host and

that means that we always have a binutils-$triplet:$arch where arch and

triplet match. So if you are ok with ignoring oldstable, you can now

depend on the triplet variant without issues.

I don't think old-stable is important anymore. However the real issue, as you say later, is that we can not use ls, but ld.bfd.

However, I think that doing so is technically wrong. When I straced fpc,

I saw that it calls into ld.bfd. binutils-x86-64-linux-gnu does not

contain ld.bfd, so the dependency is too weak for what fpc uses. In

order for a binutils-$triplet to be a correct dependency, we'd have to

somehow tell fpc to always invoke triplet-prefixed tools such as

x86_64-linux-gnu-ld.bfd.

We already patch the compiler to call ld.bfd instead of ld, so we can change the name as you wish.

And given the pointer to fpc.cfg, I looked into it and stumbled into the

-XP option. It is conditional to cross compiling, but given Debian's

packaging of binutils, it is also relevant to foreign installation.

We'll need to teach fpc to always use the -XP option even natively. Once

that is done, we can replace the binutils dependency with

"binutils-$triplet" or if you require backwards compatibility with

stretch "binutils-$triplet | binutils". Once doing these two bits,

foreign installation should become possible.

I should admit this is new for me, but looks feasible. I can help patching the compiler code to force using -XP as required.

The next part was fpc as a cross compiler. I think that one of the first

pieces to getting there is ensuring that all the compiler backend

binaries (e.g. ppcx86 or ppcarm) are built for all architectures. This

does not happen today. Any ideas on how to get there?

You need to compile the source package with special flags to generate a given compiler.

So this will be as if you build several separate compilers.

I would advise to make the cross compilers as separate source package that relies on FPC source. Otherwise it will be too complicated to package any new version. But it is feasible.

A separate problem (at least for me) is figuring out what the API to

cross building with fpc is. What is a builder expected to do to build

natively or to cross build?

How to build sources for some architecture that can be run (aka build

archtecture)?

 * fpc answer: Run plain fpc.

 * gcc answer: Run plain gcc. In future, this requires a dependency on

   gcc-for-build.

 * clang answer: Run plain clang.

How to build sources for architecture $x with triplet $y?

 * fpc answer: ???

 * gcc answer: Run $y-gcc. This requires a dependency on

   gcc-$y. In future, a dependency on gcc-for-host:$x will also suffice.

 * clang answer: Run clang -target $y.

Can someone fill in the fpc answer? I kinda suspect that it is "fpc

-P$z" for some $z that can be computed from $x or $y and the o11c gist

has such a mapping.

The answer is ver easy; fpc -P$z where $z is the suffix of the arch (cross)compiler. The suffixes are defined by upstream, but the rules file manages to detect them for each dpkg arch using a mapping table. We cal also add a helper program that returns fpc arch from dpkg arch.

Depending on the answer, the packaging layout may need to change. So

I'll stop here.

Hope this helps

Helmut

Yes It helped a lot. thanks a lot and don't hesitate to correct me if I misunderstood some of your comments.

[Helmut Grohne]

Hi,

On Thu, Jan 21, 2021 at 08:30:10PM +0100, Abou Al Montacir wrote:
> We already patch the compiler to call ld.bfd instead of ld, so we can change the
> name as you wish.

Can you implement just this part and poke me once that has hit unstable?
I can send a patch for the next step then.

> You need to compile the source package with special flags to generate a given
> compiler.
> So this will be as if you build several separate compilers.
>
> I would advise to make the cross compilers as separate source package that
> relies on FPC source. Otherwise it will be too complicated to package any new
> version. But it is feasible.

Both models are used in Debian and I've gained some experience with
both. For instance, binutils uses the "lump everything
together"-approach. And gcc is split into several source packages
(gcc-10, gcc-10-cross, gcc-10-cross-ports). In my experience the cross
packages go out of sync way too often and I strongly prefer the combined
approach. Why do you think that is getting too complicated?

The way I see things is that rather than building fpc once, the Debian
package would build it a number of times. The build time would go up
significantly, which is a disadvantage, but testing time can be reduced
using build profiles. I think I can help contain this approach.

> The answer is ver easy; fpc -P$z where $z is the suffix of the arch
> (cross)compiler. The suffixes are defined by upstream, but the rules file
> manages to detect them for each dpkg arch using a mapping table. We cal also add
> a helper program that returns fpc arch from dpkg arch.

This is a mix of clang and gcc approaches. Like clang, the target is
specified using a command line switch. Like gcc, we need one binary
executable for every combination of build architecture and host
architecture (simplified, it's a bit less). As such, I think we should
provide packages fpc-${DEB_HOST_ARCH} that makes fpc -P$mapped_arch
work. In a first iteration, we'd just do the "diagonal" of the matrix
(i.e. no cross tools, just working with emulated compilers). The rest
can be filled in later on demand. Cross building fpc itself strongly
depends on a filled matrix, so that part comes last.

Sounds like we do have an implementation sketch now. :)

1. Make fpc call into triplet-ld.bfd instead of ld.bfd.
2. Change the binutils dependency to binutils-triplet | binutils.
-> Foreign installation of an emulated fpc should work.
3. Provide suitable fpc-arch packages from fpc.
4. Add fpc-for-host and fpc-for-build metapackages.
5. Add cross compilers.
-> Cross building with fpc should work.
6. Make fpc cross buildable.
-> Cross building fpc should work.

The devil is in the detail of course.

Helmut

[Abou Al Montacir]

Hi Helmut,

On Fri, 2021-01-22 at 10:06 +0100, Helmut Grohne wrote:

We already patch the compiler to call ld.bfd instead of ld, so we can change the

name as you wish.

Can you implement just this part and poke me once that has hit unstable?

I can send a patch for the next step then.

I did not really patch this, but while looking for such a way to patch, I discovered a nice command line option -XP.

This ca be used as follows:

fpc -XPx86_64-linux-gnu- myprogram

fpc -Pi386 -XPi386-linux-gnu- myprogram

It should be possible to hack the default value of this CLO so that we can make it point to the right triplet.

Does this fit your need?