Re: [Multiarch] armhf on arm64 is not working

[Reco]

Hi.

On Mon, 28 Aug 2017 22:50:17 +0200
Adam Cécile <adam....@hitec.lu> wrote:

> Hello,
>
>
> I'm trying to set up armhf on arm64 stretch but it does not work:
>
>
> file /tmp/bash/bin/bash
>
> /tmp/bash/bin/bash: ELF 32-bit LSB executable, ARM, EABI5 version 1
> (SYSV), dynamically linked, interpreter /lib/ld-linux-armhf.so.3, for
> GNU/Linux 3.2.0, BuildID[sha1]=e3aedef3dd05738b73e7756ddb9be88dd0f9a8ce,
> stripped
>
>
> ldd /tmp/bash/bin/bash
>
> not a dynamic executable

That's supposed to happen. Try 'objdump -p' instead.


> /tmp/bash/bin/bash
>
> -bash: /tmp/bash/bin/bash: cannot execute binary file: Exec format error

And that is supposed to happen too.


> Any idea about what's wrong ? Everything looks ok to me. Is it possible
> to be kernel related ?

In a way that's kernel related.

What multiarch is - it's an ability to co-install libraries from
different architectures (for cross-compilation usually).
What multiarch is not - it's not an ability to use foreign architecture
libraries to run foreign architecture binaries.

The reason being - an executable binary contains CPU instructions that
are suited to CPU of designated architecture only. In your example it's
armv5. Your CPU is amd64 and its unable to interpret those - it can only
understand i386 and amd64 instruction sets.

Due to this Linux kernel (any reasonable OS kernel that I dealt with, in
fact) refuses to acknowledge that foreign architecture binaries could be
executed at all. Hence 'Exec format error' from the direct execution
and 'not a dynamic executable' from ldd (which is special form of
binary execution anyway).


There's a way around this limitation, and it involves CPU instruction
translation via qemu-user. Assuming that you have 'qemu-user-static'
installed, this should work:

qemu-arm-static /tmp/bash/bin/bash

Reco

[Sven Hartge]

Reco <recov...@gmail.com> wrote:

> The reason being - an executable binary contains CPU instructions that
> are suited to CPU of designated architecture only. In your example it's
> armv5. Your CPU is amd64 and its unable to interpret those - it can only
> understand i386 and amd64 instruction sets.

No, he is talking about ARM64, the 64bit variant von ARM.

S°

--
Sigmentation fault. Core dumped.

[Reco]

Hi.

On Mon, 28 Aug 2017 23:33:50 +0200
Hitec Adam Cecile <adam....@hitec.lu> wrote:

> Hi,
>
> I'm actually running arm64, not amd64 ;-)

My eyes are failing me then ☺.

Can you provide your /proc/cpuinfo then please?

Reco

[Reco]

Hi.

On Tue, Aug 29, 2017 at 08:49:40AM +0200, Adam Cecile wrote:
> > > I'm actually running arm64, not amd64 ;-)
> > My eyes are failing me then ☺.
> >
> > Can you provide your /proc/cpuinfo then please?
>

> Sure:
>
> processor : 0
> BogoMIPS : 200.00
> Features : fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics
> CPU implementer : 0x43
> CPU architecture: 8
> CPU variant : 0x1
> CPU part : 0x0a1
> CPU revision : 1

Ok, I got armhf binary running on ARM64, but the only CPU I have access
to has 'CPU implementer' 0x41 and a different set of features. Unsure if
it matters.

So, I'm proposing to start with something more simple - statically
linked binaries.

Get yourself busybox-static package from armhf and armel. Unpack it with
'dpkg -x', see if either of them works.

Reco

[Christian Seiler]

Hi,

32bit ARM compatibility is optional according to the specification,
and if your CPU doesn't support it, you won't be able to natively
run armhf executables. This is in contrast to x86, where all[*]
64bit x86 CPUs also support running old 32bit programs.

From what I've read it appears to be that the vast majority of chips
currently sold do have 32bit ARM compatibility - except for one, the
Cavium Thunder X. And if you look at your /proc/cpuinfo output:

Am 2017-08-29 08:49, schrieb Adam Cecile:
> CPU implementer : 0x43
> CPU architecture: 8
> CPU variant : 0x1
> CPU part : 0x0a1
> CPU revision : 1

Well, yeah, that's the one.

See also:
https://askubuntu.com/questions/928249/how-to-run-armhf-executables-on-an-arm64-system

So it appears you're out of luck on your hardware: your CPU
simply doesn't support running armhf executables. You could run them
emulated in qemu-user-static, but that's probably not what you want,
because that's going to be _really_ slow.

Regards,
Christian

[*] There were very, very few exceptions, and nothing marketed as a
general purpose CPU.

[Christian Seiler]

Hi,

Am 29. August 2017 17:04:29 MESZ schrieb Adam Cecile <adam....@hitec.lu>:
>I was not aware of this optional 32 bit compatibility. That kinda
>sucks.

Yeah, especially since you had the misfortune of getting the one chip that is sold that doesn't support it.

>Actually, I'm already using qemu with cowbuilder (I mean, a lot) but my
>biggest problem is not the slowness but the broken thread
>implementation.

Yeah, that is a problem indeed. And one that will take a long time to fix, because threading is already hard enough when you don't need to juggle hardware architectures with different memory coherency models...

>If you have any hint about online cheap arm server, let me know...

Does it have to be a server? The raspberry pi 3 has an ARMv8 chip with 32bit compat mode. Granted, it's not the fastest, doesn't have that much RAM, and if you boot it in 64bit mode some peripherals don't work yet, but for a pure compile/build box...? Especially if you're kind of OK with the speed of qemu?

Regards,
Christian

[Christian Seiler]

Hi there,

On 08/29/2017 06:07 PM, Adam Cecile wrote:
> Could be an alternative indeed, but what about the speed compared to
> my quad-core i5 with qemu ?

I haven't actually tried that specific comparison, but form my
experience a Pi tends to be a tiny bit faster in pure CPU
performance than qemu on Intel. (But not much.) The RPi3 is
also a quad core, so that is similar. YMMV may vary depending
on the precise workload though.

That said:

- It has only 1 GiB of RAM. That might be a problem.
- It doesn't have as much cache as an Intel Core CPU, so if
you have workloads that require a lot of memory access,
that'll probably offset any small advantages in pure CPU
performance.
- I/O is quite slow. If you compile large things my guess is
that just because of I/O it'll take longer on the Pi than
with qemu.

OTOH, it's cheap, so even if it's not the right thing in the
end you're not going to waste a ton of money. You could also
first buy just the board and power supply and only buy a case
and other accessories once you've verified that it's sufficient
for your use case.

Then again, there are also other ARM boards in a similar price
range out there, which might suit your use case better. But I
really am not an expert here, I've just played around with the
Pi a bit in the past...

Regards,
Christian