Cross-compile worked, cross-install not so much ...

[Patrick M. Hausen]

Hi all,

getting into FreeBSD ARM64 I tried to compile a current system on a fast AMD64 VM and
now I am somewhat stuck - no help using search engines and the like.

1st step:

# checkout main aka 14-CURRENT
cd /usr/src
make -j 8 TARGET=arm64 TARGET_ARCH=aarch64 buildworld buildkernel

That worked like a charm and I seem to have a well populated /usr/obj/usr/src/arm64.aarch64.

2nd step:

I then mounted /usr/src and /usr/obj on my Raspberry Pi via NFS. The binaries
in /usr/obj are the correct architecture and run perfectly well:

root@pi8:~ # file /usr/obj/usr/src/arm64.aarch64/bin/sh/sh
/usr/obj/usr/src/arm64.aarch64/bin/sh/sh: ELF 64-bit LSB pie executable, ARM aarch64, version 1 (FreeBSD), dynamically linked, interpreter /libexec/ld-elf.so.1, for FreeBSD 14.0 (1400057), FreeBSD-style, not stripped
root@pi8:~ # /usr/obj/usr/src/arm64.aarch64/bin/sh/sh
root@pi8:~ # ^D

3rd step:

root@pi8:/usr/src # make TARGET=arm64 TARGET_ARCH=aarch64 installkernel
--------------------------------------------------------------
>>> Install check kernel
--------------------------------------------------------------
--------------------------------------------------------------
>>> Installing kernel GENERIC on Mon Apr 25 21:03:58 CEST 2022
--------------------------------------------------------------
cd /usr/obj/usr/src/arm64.aarch64/sys/GENERIC; MACHINE_ARCH=aarch64 MACHINE=arm64 CPUTYPE= CC="cc -target aarch64-unknown-freebsd14.0 --sysroot=/usr/obj/usr/src/arm64.aarch64/tmp -B/usr/obj/usr/src/arm64.aarch64/tmp/usr/bin" CXX="c++ -target aarch64-unknown-freebsd14.0 --sysroot=/usr/obj/usr/src/arm64.aarch64/tmp -B/usr/obj/usr/src/arm64.aarch64/tmp/usr/bin" CPP="cpp -target aarch64-unknown-freebsd14.0 --sysroot=/usr/obj/usr/src/arm64.aarch64/tmp -B/usr/obj/usr/src/arm64.aarch64/tmp/usr/bin" AS="as" AR="ar" ELFCTL="elfctl" LD="ld" LLVM_LINK="" NM=nm OBJCOPY="objcopy" RANLIB=ranlib STRINGS= SIZE="size" STRIPBIN="strip" PATH=/usr/obj/usr/src/arm64.aarch64/tmp/bin:/usr/obj/usr/src/arm64.aarch64/tmp/usr/sbin:/usr/obj/usr/src/arm64.aarch64/tmp/usr/bin:/usr/obj/usr/src/arm64.aarch64/tmp/legacy/usr/sbin:/usr/obj/usr/src/arm64.aarch64/tmp/legacy/usr/bin:/usr/obj/usr/src/arm64.aarch64/tmp/legacy/bin:/usr/obj/usr/src/arm64.aarch64/tmp/legacy/usr/libexec::/sbin:/bin:/usr/sbin:/usr/bin make KERNEL=kernel install
/bin/sh: make: Exec format error
*** Error code 126

Stop.
make[1]: stopped in /usr/src
*** Error code 1

Stop.
make: stopped in /usr/src

So via that complicated PATH setting it seems to run: /usr/obj/usr/src/arm64.aarch64/tmp/legacy/usr/bin/make

Which surprisingly is an amd64 binary:

root@pi8:/usr/src # file /usr/obj/usr/src/arm64.aarch64/tmp/legacy/usr/bin/make
/usr/obj/usr/src/arm64.aarch64/tmp/legacy/usr/bin/make: ELF 64-bit LSB pie executable, x86-64, version 1 (FreeBSD), dynamically linked, interpreter /libexec/ld-elf.so.1, for FreeBSD 14.0 (1400057), FreeBSD-style, stripped


So what did I do wrong?

Thanks for all help,
Patrick

[Brooks Davis]

Cross install is not supported. As you have seen, certain tools are
bootstrapped on the build host and used during the install process. You
might be able to get away with nuking
/usr/obj/usr/src/arm64.aarch64/tmp/legacy (or maybe tmp) and then
running `make toolchain` to build native versions of those tools.

-- Brooks

[Patrick M. Hausen]

Hi,

> Am 25.04.2022 um 21:18 schrieb Brooks Davis <bro...@freebsd.org>:
> Cross install is not supported. As you have seen, certain tools are
> bootstrapped on the build host and used during the install process. You
> might be able to get away with nuking
> /usr/obj/usr/src/arm64.aarch64/tmp/legacy (or maybe tmp) and then
> running `make toolchain` to build native versions of those tools.

that comes as a big surprise and disappointment. What is the point of cross-compiling, then?
How to update a small slow embedded platform?

I tried your suggestion - unfortunately no worky:

cd /usr/src/tools/build; make DIRPRFX=tools/build/ DESTDIR=/usr/obj/usr/src/arm64.aarch64/tmp/legacy host-symlinks
Linking host tools into /usr/obj/usr/src/arm64.aarch64/tmp/legacy/bin
cp: chflags: /usr/obj/usr/src/arm64.aarch64/tmp/legacy/bin/basename: Operation not supported
*** Error code 1

So I will probably need to checkout and compile on the Pi. What are typical
build times on a CM3+? Plus I am going to wear down the builtin eMMC much faster.

Kind regards and thanks,
Patrick

[Mehmet Erol Sanliturk]

On Mon, Apr 25, 2022 at 10:26 PM Patrick M. Hausen <p...@hausen.com> wrote:

Hi,

> Am 25.04.2022 um 21:18 schrieb Brooks Davis <bro...@freebsd.org>:
> Cross install is not supported.  As you have seen, certain tools are
> bootstrapped on the build host and used during the install process.  You
> might be able to get away with nuking
> /usr/obj/usr/src/arm64.aarch64/tmp/legacy (or maybe tmp) and then
> running `make toolchain` to build native versions of those tools.

that comes as a big surprise and disappointment. What is the point of cross-compiling, then?
How to update a small slow embedded platform?

You can cross compile a program and then use it

on a related system .

Please think this is a contribution . In that way , piece by piece

you may construct another system .

You are right : Being able to construct an installable system is a

good idea . When it is not available as a whole , having

a partial capability is a good step .

Mehmet Erol Sanliturk

[Warner Losh]

On Mon, Apr 25, 2022 at 1:26 PM Patrick M. Hausen <p...@hausen.com> wrote:

Hi,

> Am 25.04.2022 um 21:18 schrieb Brooks Davis <bro...@freebsd.org>:
> Cross install is not supported.  As you have seen, certain tools are
> bootstrapped on the build host and used during the install process.  You
> might be able to get away with nuking
> /usr/obj/usr/src/arm64.aarch64/tmp/legacy (or maybe tmp) and then
> running `make toolchain` to build native versions of those tools.

that comes as a big surprise and disappointment. What is the point of cross-compiling, then?
How to update a small slow embedded platform?

Cross installing is supported. Native installing of a cross build world isn't.

When I've had to do this in the past, I've just mounted the embedded system on my

beefy server under /mumble (allowing root on the embedded system for NFS) and did

a sudo make installworld ... DESTDIR=/mumble. Though I've forgotten the gymnastics

to do etcupdate/mergemaster this way.

Warner

[Brooks Davis]

On Mon, Apr 25, 2022 at 09:26:04PM +0200, Patrick M. Hausen wrote:
> Hi,
>
> > Am 25.04.2022 um 21:18 schrieb Brooks Davis <bro...@freebsd.org>:
> > Cross install is not supported. As you have seen, certain tools are
> > bootstrapped on the build host and used during the install process. You
> > might be able to get away with nuking
> > /usr/obj/usr/src/arm64.aarch64/tmp/legacy (or maybe tmp) and then
> > running `make toolchain` to build native versions of those tools.
>
> that comes as a big surprise and disappointment. What is the point of cross-compiling, then?
> How to update a small slow embedded platform?

The traditional answer is: build a new image and install it as a
whole-sale replacement for the device image or in a nanobsd system
with two boot partitions.

If someone was motivated they could presumably work things out so we could
cross compile the boostrap tools, but that would be a fair bit of work.

It would probably be easier to figure out how to process the METALOG
file in a sysroot to extract the files you want to update and install
them in the right order (the order mostly doesn't matter until it really
does and everything explodes as you crossthread rtld/libc/libthr). If I
had time, this is the path I would pursue.

> I tried your suggestion - unfortunately no worky:
>
> cd /usr/src/tools/build; make DIRPRFX=tools/build/ DESTDIR=/usr/obj/usr/src/arm64.aarch64/tmp/legacy host-symlinks
> Linking host tools into /usr/obj/usr/src/arm64.aarch64/tmp/legacy/bin
> cp: chflags: /usr/obj/usr/src/arm64.aarch64/tmp/legacy/bin/basename: Operation not supported
> *** Error code 1

You might try editing tools/build/Makefile and dropping -p from
_COPY_HOST_TOOL or just tweaking the code to take the !FreeBSD branch.
I'm not fully convinced that cp -p should fail if file flags can't be
manipulated, but apparently it does.

-- Brooks

[Paul Mather]

On Apr 25, 2022, at 3:26 PM, Patrick M. Hausen <p...@hausen.com> wrote:

> Hi,
>
>> Am 25.04.2022 um 21:18 schrieb Brooks Davis <bro...@freebsd.org>:
>> Cross install is not supported. As you have seen, certain tools are
>> bootstrapped on the build host and used during the install process. You
>> might be able to get away with nuking
>> /usr/obj/usr/src/arm64.aarch64/tmp/legacy (or maybe tmp) and then
>> running `make toolchain` to build native versions of those tools.
>
> that comes as a big surprise and disappointment. What is the point of cross-compiling, then?
> How to update a small slow embedded platform?

At least as far back as July 2016 I kept my FreeBSD/arm systems updated by cross-building on a FreeBSD/amd64 system and using PkgBase to update the OS on the FreeBSD/arm systems:

https://lists.freebsd.org/pipermail/freebsd-arm/2016-July/014444.html

This worked well for me, at least until I stopped running FreeBSD on my ARM systems. I don't know whether it still works.

Cheers,

Paul.

[Patrick M. Hausen]

Hi all,

> Am 25.04.2022 um 21:54 schrieb Warner Losh <i...@bsdimp.com>:
> Cross installing is supported. Native installing of a cross build world isn't.
>
> When I've had to do this in the past, I've just mounted the embedded system on my
> beefy server under /mumble (allowing root on the embedded system for NFS) and did
> a sudo make installworld ... DESTDIR=/mumble. Though I've forgotten the gymnastics
> to do etcupdate/mergemaster this way.

I'll try that tonight, thanks. How do you get chflags to work over NFS?

Kind regards,
Patrick

[Patrick M. Hausen]

Hi all,

I just threw a bit of hardware at the problem for now.

In addition to the seven-node TuringPi that I would like to run with
FreeBSD after I cannot make Raspbian run stable even when
completely idle I have another actively cooled single CM3+ system
(the "pi8" you saw in my first post)

So I connected a USB powered SSD and compile on the Pi now.
System is CPU bound, so no bottleneck because of USB:

--------
CPU: 95.4% user, 0.0% nice, 4.5% system, 0.1% interrupt, 0.0% idle
Mem: 395M Active, 95M Inact, 384K Laundry, 224M Wired, 97M Buf, 187M Free
Swap: 4096M Total, 27M Used, 4069M Free

PID USERNAME THR PRI NICE SIZE RES STATE C TIME WCPU COMMAND
39861 root 1 100 0 248M 162M CPU0 0 0:55 96.39% c++
39867 root 1 86 0 124M 54M CPU1 1 0:06 96.34% c++
39865 root 1 87 0 124M 56M RUN 2 0:07 91.54% c++
39863 root 1 99 0 224M 138M RUN 3 0:45 83.17% c++
--------

I'd rather run 13.1 and use freebsd-update, but 13 does not support the CM3(+).
That was in fact the first not so pleasant surprise - thought, ARM64 was tier 1, now?
Anyway, if -current works for now, there will be a 14.0-RELEASE, eventually.

Thanks for your help, folks.
Patrick

[Patrick M. Hausen]

Hi all,

> Am 26.04.2022 um 17:47 schrieb bob prohaska <fb...@www.zefox.net>:
> If the result is unsatisfactory, self-hosting isn't impossible. I've been
> doing it for a few years now, albeit with much help from the list. On a
> Pi3 running aarch64 memory and swap are a constraint. I'd suggest 4 GB
> of swap and -j2 or -j3, perhaps increasing to -j4 as you see how things
> go. If you can split the swap across devices it helps some. Useful
> /boot/loader.conf tweaks include
>
> vm.pageout_oom_seq="4096"
> vm.pfault_oom_attempts="120"
> vm.pfault_oom_wait="20"
>
> Mark Millard made me aware of these parameters over the list.

without any additional tuning but with an SSD connected via USB
and 4GB swap on that I was able to compile with -j4 and a mostly CPU
bound system.

--------------------------------------------------------------
>>> World build completed on Thu Apr 28 10:30:53 CEST 2022
>>> World built in 155832 seconds, ncpu: 4, make -j4
--------------------------------------------------------------
--------------------------------------------------------------
>>> Kernel build for GENERIC completed on Thu Apr 28 13:11:37 CEST 2022
>>> Kernel(s) GENERIC built in 9643 seconds, ncpu: 4, make -j4
--------------------------------------------------------------

Thanks everyone for your valuable hints. Guess I will subscribe to
-arm, since there are some more rough edges compared to "just put a
Debian or Ubuntu image on it".

And then I wonder what workload I can put on a seven-node FreeBSD
cluster, since it won't be k8s, obviously. Let's start with Ceph, I guess.

Kind regards
Patrick

[Mark Millard]

Patrick M. Hausen <pmh_at_hausen.com> wrote on
Date: Sun, 1 May 2022 17:29:27 +0200 :

> > Am 26.04.2022 um 17:47 schrieb bob prohaska <fbsd_at_www.zefox.net>:
> > If the result is unsatisfactory, self-hosting isn't impossible. I've been
> > doing it for a few years now, albeit with much help from the list. On a
> > Pi3 running aarch64 memory and swap are a constraint. I'd suggest 4 GB
> > of swap and -j2 or -j3, perhaps increasing to -j4 as you see how things
> > go. If you can split the swap across devices it helps some. Useful
> > /boot/loader.conf tweaks include
> >
> > vm.pageout_oom_seq="4096"
> > vm.pfault_oom_attempts="120"
> > vm.pfault_oom_wait="20"
> >
> > Mark Millard made me aware of these parameters over the list.

And going back farther: Mark Johnston made us both
aware of vm.pageout_oom_seq and its use back when
you were having problems with the system killing
processes. He had provided some investigative patches
that we used as well. THat is part of how Mark J.
determiened that vm.pageout_oom_seq use was
appropraite. Konstantin Belousov als corrected my
mistaken mental model relative to FreeBSD
swapping/paging back in that time frame --and that
feeds into what vm.pageout_oom_seq controls.

In recent enough 13.1-??? , stable/13 , and main :

Console messages with:

was killed: failed to reclaim memory

are tied to what vm.pageout_oom_seq controls: the
number of tries to reclaim the targeted amount of
free RAM before initiating kills to do so.

Console messages with:

was killed: a thread waited too long to allocate a page

are tied to the combination of vm.pfault_oom_attempts
and vm.pfault_oom_wait used, which, together result in
an overall time frame (multiply the two) before such
kills start.

There can be messages with:

was killed: out of swap space

which is somewhat of a misnomer: the out-of-space is
actually in one or both of a couple of related kernel
data structures for managing the swap space, not the
swap partition content itself. As near as I can tell,
this type of failure is rare.

But I'll note that in FreeBSD versions before the
messages were added for "failed to reclaim memory"
and "waited too long to allocate a page", the
messaging always said words about "out of swap space"
for all 3 types of contexts: rather misleading.

> without any additional tuning but with an SSD connected via USB
> and 4GB swap on that I was able to compile with -j4 and a mostly CPU
> bound system.
>
> --------------------------------------------------------------
> >>> World build completed on Thu Apr 28 10:30:53 CEST 2022
> >>> World built in 155832 seconds, ncpu: 4, make -j4
> --------------------------------------------------------------
> --------------------------------------------------------------
> >>> Kernel build for GENERIC completed on Thu Apr 28 13:11:37 CEST 2022
> >>> Kernel(s) GENERIC built in 9643 seconds, ncpu: 4, make -j4
> --------------------------------------------------------------

So: a little under 46 hours, if I calculated right.

I'll note that in some past experiments on some types
of RPi*'s, using -j3 actually took less overall time
than -j4 when deliberately repeating from-scratch
builds. The differences were not all that large as I
remember. But, if -j3 and -j4 end up with even similar
time frames, then -j3 has the additional advantage in
limited-RAM contexts of not being as likely to have
resource problems. So -j3 could be appropriate.

I do not know if -j3 would work out better for you or not.
I'm just noting that it may be worth experimenting with.

> Thanks everyone for your valuable hints. Guess I will subscribe to
> -arm, since there are some more rough edges compared to "just put a
> Debian or Ubuntu image on it".
>
> And then I wonder what workload I can put on a seven-node FreeBSD
> cluster, since it won't be k8s, obviously. Let's start with Ceph, I guess.

===
Mark Millard
marklmi at yahoo.com