Linux "native" Virtualization: ARM64 vs x86_64

[Robert Heller]

I presently have an older AMD Quad Core processor on a ATX motherboard in a
tower case. I am considering upgrading the motherboard & processor. Right now
I am using Linux "native" Virtualization (libvirt and QEMU) for several x86_64
and i686 installs of Linux with 64-bit Ubuntu 18.04 on the "bare metal". The
VMs are mostly used as 32-bit and 64-bit build hosts and in one case a special
version to run a closed source program that won't run on Ubuntu 18.04 -- it
requires a bleeding edge version of Ubuntu 18.04 (not a LTS version).

I am considering getting an ARM64 based ATX motherboard to swap into my tower
case. I have several questions:

1) I know that the x86_64 processors have hardware support for virtualization,
allowing VMs to run at full hardware processor speed. Do the ARM64 processors
have this sort of support? If not what sort of Virtualization options are
available for ARM64 Linux? I specifically would want CLI user interface and
with support for raw disk volumes (such as LVM volumes) for the VM disks, not
special container files (harder to intelligently backup).

2) I've never bothered to run non x86 VMs on my x86_64 machine (I have
physical ARM7 and ARM8 processors/OSs that I use as build boxes). Is it
possible to run x86_64 VMs on an ARM64 system? I'm guessing it would have to
be with software emulation.

(If I replace my AMD ATX motherboard with an ARM64, I would end up with a LAN
with all ARM-based Linux boxen, so the only way to build for x86 would be with
VMs.)

I want to move to ARM because ARM processor are cheaper, use less power, and
look to be getting effectively faster than anything Intel or AMD is going to
be putting out (unless either / both start making ARM processors).

--
Robert Heller -- Cell: 413-658-7953 GV: 978-633-5364
Deepwoods Software -- Custom Software Services
http://www.deepsoft.com/ -- Linux Administration Services
hel...@deepsoft.com -- Webhosting Services

[Jason Evans]

On Sat, 30 Oct 2021 23:21:01 -0500, Robert Heller wrote:

> 2) I've never bothered to run non x86 VMs on my x86_64 machine (I have
> physical ARM7 and ARM8 processors/OSs that I use as build boxes). Is it
> possible to run x86_64 VMs on an ARM64 system? I'm guessing it would
> have to be with software emulation.
>
> (If I replace my AMD ATX motherboard with an ARM64, I would end up with
> a LAN with all ARM-based Linux boxen, so the only way to build for x86
> would be with VMs.)

I run openSUSE Tumbleweed on my Raspberry Pis. Right now I can use QEMU/
Libvirt to emulate ARM VMs. Considerably slower, I can also emulator x86
32 and 64-bit machines.

I would also suggest that you look into using LXD. While it is container
based, you can keep all of your virtual file systems in real Linux
directories which are easy to back up. LXD creates more general system
containers which are meant to function similar to VMs as opposed to
Podman/Docker which containers that are meant to be one application per
container.

I won't say that I'm an expert in virtualization or containers, but it is
what I focus on in my work and both are a part of my daily workflow.

[Anssi Saari]

Robert Heller <hel...@deepsoft.com> writes:

> I am considering getting an ARM64 based ATX motherboard to swap into my tower
> case.

I don't know much about virtualization but if you know of ARM based ATX
motherboards, I'd certainly like to know.

[Aragorn]

On 01.11.2021 at 09:44, Anssi Saari scribbled:

RISC-V will probably be getting there sooner. They're already working
on it. ;)

--
With respect,
= Aragorn =

[Anssi Saari]

Yes, in fact SiFive has their Unmatched Mini-ITX board coming out or may
have shipped some already. Currently not available though.

But it's pretty expensive and I don't believe they're there with
performance for everyday use. They themselves call it a development
platform.

[The Natural Philosopher]

I meant to comment yesterday - yes there are ARM MoBos out there, but
they seem to be optimised for low power server farms and are not cheap
at all.

Many servers dont need the MIPS - just RAM. And SSDS.

Obviously virtualisation on a target with a *different* instruction set
will require the equivalent of a microcode instruction set emulator.

I would expect an X86 emulating ARM to be as crap as an ARM emulating X86...




--
Climate is what you expect but weather is what you get.
Mark Twain

[Pascal Hambourg]

Le 03/11/2021 à 11:36, The Natural Philosopher a écrit :
> I would expect an X86 emulating ARM to be as crap as an ARM emulating
> X86...

I may be wrong, but I would expect RISC emulating CISC to perform better
than CISC emulating RISC.

[Grant Taylor]

On 11/3/21 1:34 PM, Pascal Hambourg wrote:
> I would expect RISC emulating CISC to perform better than CISC emulating
> RISC.

Didn't AMD K6 processors get a performance over Intel counterparts by
doing exactly that, be a high performance RISK core emulating the Intel
x86 CISC instruction set?



--
Grant. . . .
unix || die

[Robert Riches]

If they did, they were a year late to the party. The Pentium Pro
processor was RISC on the inside:

https://duckduckgo.com/?q=%22pentium+pro%22+risc+uop&t=h_&ia=web

One of the hits on that search says the Pentium Pro processor was
released in 1996. One of the hits on this search says the K6 was
released in 1997:

https://duckduckgo.com/?q=amd+k6&t=h_&ia=web

--
Robert Riches
spamt...@jacob21819.net
(Yes, that is one of my email addresses.)

[Rich]

Which is also how every P6 and derivative Intel CPU since the Pentium
Pro introduced in 1995 operates:

https://en.wikipedia.org/wiki/P6_(microarchitecture)#From_Pentium_Pro_to_Pentium_III

"P6 processors dynamically translate IA-32 instructions into
sequences of buffered RISC-like micro-operations,"

[The Natural Philosopher]

That is in fact and interesting question, since CISC itself is RISC
emulating CISC in microcode, half the time.

The point being whether or not the microcode inside the processor can
execute faster than the RISC instruction in the main bus or cache of a
RISC chip.

To date Intel beats an ARM on pure speed. Just not on power consumption
IIRC.

I cant see either of them performing satisfactorily in a mode they were
not designed for however, though my money might be on intel emulating
ARM, only because they are more powerful to start with.

--
"When a true genius appears in the world, you may know him by this sign,
that the dunces are all in confederacy against him."

Jonathan Swift.

[Robert Heller]

At Thu, 4 Nov 2021 11:03:24 +0000 The Natural Philosopher <t...@invalid.invalid> wrote:

>
> On 03/11/2021 19:34, Pascal Hambourg wrote:

> > Le 03/11/2021 à 11:36, The Natural Philosopher a écrit :

> >> I would expect an X86 emulating ARM to be as crap as an ARM emulating
> >> X86...
> >
> > I may be wrong, but I would expect RISC emulating CISC to perform better
> > than CISC emulating RISC.
>
> That is in fact and interesting question, since CISC itself is RISC
> emulating CISC in microcode, half the time.
>
> The point being whether or not the microcode inside the processor can
> execute faster than the RISC instruction in the main bus or cache of a
> RISC chip.
>
> To date Intel beats an ARM on pure speed. Just not on power consumption
> IIRC.

The Apple M1 is said to faster than Intel. And lower power...

>
> I cant see either of them performing satisfactorily in a mode they were
> not designed for however, though my money might be on intel emulating
> ARM, only because they are more powerful to start with.

I am not really interested performance -- My only interest in emulating x86 on
aarch64 (64-bit ARM) is as a "build box" -- if it takes hours to compile, so
what...

[The Natural Philosopher]

On 04/11/2021 12:07, Robert Heller wrote:

> I am not really interested performance -- My only interest in emulating x86 on
> aarch64 (64-bit ARM) is as a "build box" -- if it takes hours to compile, so
> what...
>

Ahh..ok. Have fun writing an ARM RISC=>x86 CISC interpreter...


--
Outside of a dog, a book is a man's best friend. Inside of a dog it's
too dark to read.

Groucho Marx

[Robert Heller]

At Thu, 4 Nov 2021 15:54:42 +0000 The Natural Philosopher <t...@invalid.invalid> wrote:

>
> On 04/11/2021 12:07, Robert Heller wrote:
>
> > I am not really interested performance -- My only interest in emulating x86 on
> > aarch64 (64-bit ARM) is as a "build box" -- if it takes hours to compile, so
> > what...
> >
> Ahh..ok. Have fun writing an ARM RISC=>x86 CISC interpreter...

I'm actually hoping a x86_64 emulator already exists for ARM. I guess *worst
case* I could build a x86_64 cross-compiler hosted on ARM.

[Pascal Hambourg]

Le 04/11/2021 à 16:54, The Natural Philosopher a écrit :
> On 04/11/2021 12:07, Robert Heller wrote:
>
>> I am not really interested performance -- My only interest in emulating x86 on
>> aarch64 (64-bit ARM) is as a "build box" -- if it takes hours to compile, so
>> what...

Do you really need an x86 platform to compile for x86 target ? What
about cross-compilers ?

> Ahh..ok. Have fun writing an ARM RISC=>x86 CISC interpreter...

Doesn't QEMU already do this ?

[John-Paul Stewart]

On 2021-11-04 2:41 p.m., Robert Heller wrote:
>
> I'm actually hoping a x86_64 emulator already exists for ARM. I guess *worst
> case* I could build a x86_64 cross-compiler hosted on ARM.

I believe QEMU can do full-system emulation of x86_64 on ARM hosts. But
their documentation isn't as good as it used to be, so I'm finding it
difficult to locate that info to verify it.

[Pascal Hambourg]

<https://packages.debian.org/stable/qemu-system-arm>
"This package provides the full system emulation binaries to emulate the
following arm hardware: aarch64 arm."

What is that for if not emulating ARM on x86 and other architectures ?

[Pascal Hambourg]

Oops ! you meant the other way around.

<https://packages.debian.org/en/stable/qemu-system-x86>

" This package provides the full system emulation binaries to emulate

the following x86 hardware: i386 x86_64."
Available for arm64, armel (ARM little-endian), armhf and others.

[The Natural Philosopher]

On 04/11/2021 18:41, Robert Heller wrote:
> At Thu, 4 Nov 2021 15:54:42 +0000 The Natural Philosopher <t...@invalid.invalid> wrote:
>
>>
>> On 04/11/2021 12:07, Robert Heller wrote:
>>
>>> I am not really interested performance -- My only interest in emulating x86 on
>>> aarch64 (64-bit ARM) is as a "build box" -- if it takes hours to compile, so
>>> what...
>>>
>> Ahh..ok. Have fun writing an ARM RISC=>x86 CISC interpreter...
>
> I'm actually hoping a x86_64 emulator already exists for ARM. I guess *worst
> case* I could build a x86_64 cross-compiler hosted on ARM.

Oh I think that already exists

[The Natural Philosopher]

On 04/11/2021 19:25, The Natural Philosopher wrote:
> On 04/11/2021 18:41, Robert Heller wrote:
>> At Thu, 4 Nov 2021 15:54:42 +0000 The Natural Philosopher
>> <t...@invalid.invalid> wrote:
>>
>>>
>>> On 04/11/2021 12:07, Robert Heller wrote:
>>>
>>>> I am not really interested performance -- My only interest in
>>>> emulating x86 on
>>>> aarch64 (64-bit ARM) is as a "build box" -- if it takes hours to
>>>> compile, so
>>>> what...
>>>>
>>> Ahh..ok. Have fun writing an ARM RISC=>x86 CISC interpreter...
>>
>> I'm actually hoping a x86_64 emulator already exists for ARM.  I guess
>> *worst
>> case* I could build a x86_64 cross-compiler hosted on ARM.
>
> Oh I think that already exists

gcc surely runs on a PI and will compile to a linux target on X86?



--
"Corbyn talks about equality, justice, opportunity, health care, peace,
community, compassion, investment, security, housing...."
"What kind of person is not interested in those things?"

"Jeremy Corbyn?"

[John-Paul Stewart]

That's exactly what I was thinking, too!

[Robert Heller]

At Thu, 4 Nov 2021 19:52:54 +0100 Pascal Hambourg <pas...@plouf.fr.eu.org> wrote:

>
> Le 04/11/2021 à 16:54, The Natural Philosopher a écrit :

> > On 04/11/2021 12:07, Robert Heller wrote:
> >
> >> I am not really interested performance -- My only interest in emulating x86 on
> >> aarch64 (64-bit ARM) is as a "build box" -- if it takes hours to compile, so
> >> what...
>
> Do you really need an x86 platform to compile for x86 target ? What
> about cross-compilers ?

I might want to do basic tests to be sure the binaries actually run under x86
Linuxes

>
> > Ahh..ok. Have fun writing an ARM RISC=>x86 CISC interpreter...
>
> Doesn't QEMU already do this ?
>
>
>

[Pascal Hambourg]

Le 05/11/2021 à 00:24, Robert Heller a écrit :
> At Thu, 4 Nov 2021 19:52:54 +0100 Pascal Hambourg <pas...@plouf.fr.eu.org> wrote:
>
>>> On 04/11/2021 12:07, Robert Heller wrote:
>>>
>>>> I am not really interested performance -- My only interest in emulating x86 on
>>>> aarch64 (64-bit ARM) is as a "build box" -- if it takes hours to compile, so
>>>> what...
>>
>> Do you really need an x86 platform to compile for x86 target ? What
>> about cross-compilers ?
>
> I might want to do basic tests to be sure the binaries actually run under x86
> Linuxes

It seems that QEMU can transparently run foreign binaries (qemu-user).
So you could build them efficiently with a cross-compiler and test them
with QEMU.

[The Natural Philosopher]

When I tried to boot DOS on QEMU on a power PC I gave up after an hour...

--
"A point of view can be a dangerous luxury when substituted for insight
and understanding".

Marshall McLuhan

[Anssi Saari]

Robert Heller <hel...@deepsoft.com> writes:

> I'm actually hoping a x86_64 emulator already exists for ARM. I guess *worst
> case* I could build a x86_64 cross-compiler hosted on ARM.

Why is that the worst case? I've done cross compilation to MIPS and ARM
a couple of times but not recently. Cross compilation was fine, mostly
the problems I had were some software packages using build systems like
the common Autotools in a way that was not compatible with cross
compilation and required some fiddling. Or even a lot of fiddling.

[The Natural Philosopher]

I cant understand why, if you want an x86 target, you don't just grab a
second-hand WINPC and bang Linux on it, and cross mount a source
directory and compile on it using the command line - no need even for
screen and keyboard

My Pi code was all prototyped on a fast compiling x86 machine prior to
mounting the source directory on the pi and then typing "make clean; make;"
Cross compiling works, but is a lot easier to have the right CPU and
libraries handy for a target to compile and test upon.

--
When plunder becomes a way of life for a group of men in a society, over
the course of time they create for themselves a legal system that
authorizes it and a moral code that glorifies it.

Frédéric Bastiat

[Robert Heller]

Cross compiling to a target *OS*, as opposed to a micro-controller (which has
no bare-metal O/S), can be "interesting", since the build environment is not
quite the same as the target. And compiling gcc/g++ cross-compiler can take a
long time (well building an embeded ARMv7 target on an ARMv7 linux host took
like 72 hours and pushed close to the 1G RAM on a Pi3). And yes, the
Autotools are fiddly when cross building (at least for a MS-Windows target).
My build infrastructure is not really cleanly set up from cross-building to a
Linux target on a Linux host -- I suspect some fiddling with the
autoconf/aclocal/automake stuff would be needed, esp. since the
cross-compiling test assumes MS-Windows target on a Linux host.

[Robert Heller]

At Fri, 5 Nov 2021 14:15:12 +0000 The Natural Philosopher <t...@invalid.invalid> wrote:

>
> On 05/11/2021 12:09, Anssi Saari wrote:
> > Robert Heller <hel...@deepsoft.com> writes:
> >
> >> I'm actually hoping a x86_64 emulator already exists for ARM. I guess *worst
> >> case* I could build a x86_64 cross-compiler hosted on ARM.
> >
> > Why is that the worst case? I've done cross compilation to MIPS and ARM
> > a couple of times but not recently. Cross compilation was fine, mostly
> > the problems I had were some software packages using build systems like
> > the common Autotools in a way that was not compatible with cross
> > compilation and required some fiddling. Or even a lot of fiddling.
> >
> I cant understand why, if you want an x86 target, you don't just grab a
> second-hand WINPC and bang Linux on it, and cross mount a source
> directory and compile on it using the command line - no need even for
> screen and keyboard

It is another piece of hardware to maintain... (I do everything from the
command line and all of my build boxen are headless.) I am also getting fed
up with Intel and AMD processors and corresponding motherboard chipsets and I
really at the end of the line with laptops.

>
> My Pi code was all prototyped on a fast compiling x86 machine prior to
> mounting the source directory on the pi and then typing "make clean; make;"
> Cross compiling works, but is a lot easier to have the right CPU and
> libraries handy for a target to compile and test upon.
>

--