Nearly painless RISC-V userspace testing with qemu + Fedora userspace + docker
534 views
Skip to first unread message
Stefan O'Rear
Aug 31, 2016, 11:11:07 PM8/31/16
to RISC-V SW Dev, Richard W.M. Jones, Alex Suykov
# Easy and hygenic RISC-V userspace tesing on Linux
So I was playing around yesterday with [Alex Suykov's QEMU fork][Q] and [Richard
W.M. Jones' Fedora bootstrapping efforts][F] and noticed that they go really
well together. Using binfmt_misc to invoke qemu, I can set up a chroot running
a RISC-V Fedora userland, including *non-cross* gcc compilers.
[Q]: https://github.com/arsv/riscv-qemu
[F]: https://github.com/rwmjones/fedora-riscv
Automating the process with Docker (a very fancy chroot manager for Linux with
additional sandboxing facilities), you can now run a sandboxed RISC-V Linux
user environment with just two commands. I've pushed the necessary image to
[Docker hub][I], although it's also surprisingly simple to rebuild (below).
[I]: https://hub.docker.com/r/sorear/fedora-riscv-wip/
## The two-command way
1. Set up binfmt_misc to pass RISC-V ELF binaries to `/qemu-riscv64` in the
current filesystem root. This configuration is _not_ scoped to the sandbox,
so it will affect the host system; since you don't have a file named
`/qemu-riscv64` on your host system, the only effect of this is that trying
to run a RISC-V binary may result in a spurious ENOENT instead of ENOEXEC.
There are no spaces or newlines in the argument to echo.
Z='\x00\x00\x00\x00\x00\x00\x00'; echo ':riscv64:M::\x7f\x45\x4c\x46'\
$Z$Z'\xf3\x00:\xff\xff\xff\xff'$Z$Z'\xff\xff:/qemu-riscv64:' >
/proc/sys/fs/binfmt_misc/register
2. Use docker to download the chroot environment and run a shell. The chroot
contains a `/qemu-riscv64` binary, so RISC-V executables invoked within it
will be emulated automatically; furthermore, the emulator is inside the
sandbox. This is roughly a 600MB download.
The qemu binary is compiled for amd64; if you are on a different
architecture, you will need to build your own image (below).
docker run -it sorear/fedora-riscv-wip:interp
## Known limitations
* I'm using an incomplete old snapshot of the
bootstrapping process, so there are relatively few tools.
* Job control doesn't work (control-Z does nothing).
* Control-C at the shell doesn't clear the current command line(?)
* Many commands, including `git` and `make`, will segfault immediately if run
from the context of a `docker exec` but work properly from `docker run`.
No idea why.
* There's something wrong with library search paths and gcc doesn't work unless
I set `LD_LIBRARY_PATH=/lib64`; this workaround is included in the image.
## I want to build the image myself.
Say you don't want me in your chain of custody, or you're not on amd64.
1. Download the Fedora stage 3 root image and convert it into a docker image.
wget http://oirase.annexia.org/riscv/stage3-disk.img.xz
unxz stage3-disk.img.xz
sudo mount -o loop stage3-disk.img.xz stage3-disk
cd stage3-disk
sudo tar -c . | docker import -c 'CMD /bin/bash' - \
sorear/fedora-riscv-wip:pristine
cd ..
sudo umount stage3-disk
2. Download and compile a statically linked riscv64-linux-user qemu.
git clone https://github.com/arsv/riscv-qemu
cd riscv-qemu
sudo dnf install glibc-static glib2-static libstdc++-static
./configure --prefix=~/opt/riscv-qu --target-list=riscv64-linux-user \
--static
make
cd ..
3. Create an image which combines the two.
mkdir addinterp
cd addinterp
cp ../riscv-qemu/riscv64-linux-user/qemu-riscv64 .
cat >Dockerfile <<END
FROM sorear/fedora-riscv-wip:pristine
COPY qemu-riscv64 .
ENV LD_LIBRARY_PATH=/lib64
END
docker build -t sorear/fedora-riscv-wip:interp .
cd ..
## I don't want Docker.
The previous steps can be easily adapted to build a directory tree instead of
an image, which you can then invoke with a chroot. This results in much weaker
sandboxing, though.
## I'm not using Linux.
You will need a Linux VM for this project. Docker has downloadable VM tools
in the command line package for macOS and Windows; I haven't tried them.
I believe those tools do not give you easy shell access to the VM outside of a
container. The binfmt_misc setup command cannot be run in a normal container
because it's a privileged operation (it affects all containers), but it should
be possible to run it in a container started with `--privileged`, e.g.
docker run --rm -it --privileged alpine /bin/sh
So I was playing around yesterday with [Alex Suykov's QEMU fork][Q] and [Richard
W.M. Jones' Fedora bootstrapping efforts][F] and noticed that they go really
well together. Using binfmt_misc to invoke qemu, I can set up a chroot running
a RISC-V Fedora userland, including *non-cross* gcc compilers.
[Q]: https://github.com/arsv/riscv-qemu
[F]: https://github.com/rwmjones/fedora-riscv
Automating the process with Docker (a very fancy chroot manager for Linux with
additional sandboxing facilities), you can now run a sandboxed RISC-V Linux
user environment with just two commands. I've pushed the necessary image to
[Docker hub][I], although it's also surprisingly simple to rebuild (below).
[I]: https://hub.docker.com/r/sorear/fedora-riscv-wip/
## The two-command way
1. Set up binfmt_misc to pass RISC-V ELF binaries to `/qemu-riscv64` in the
current filesystem root. This configuration is _not_ scoped to the sandbox,
so it will affect the host system; since you don't have a file named
`/qemu-riscv64` on your host system, the only effect of this is that trying
to run a RISC-V binary may result in a spurious ENOENT instead of ENOEXEC.
There are no spaces or newlines in the argument to echo.
Z='\x00\x00\x00\x00\x00\x00\x00'; echo ':riscv64:M::\x7f\x45\x4c\x46'\
$Z$Z'\xf3\x00:\xff\xff\xff\xff'$Z$Z'\xff\xff:/qemu-riscv64:' >
/proc/sys/fs/binfmt_misc/register
2. Use docker to download the chroot environment and run a shell. The chroot
contains a `/qemu-riscv64` binary, so RISC-V executables invoked within it
will be emulated automatically; furthermore, the emulator is inside the
sandbox. This is roughly a 600MB download.
The qemu binary is compiled for amd64; if you are on a different
architecture, you will need to build your own image (below).
docker run -it sorear/fedora-riscv-wip:interp
## Known limitations
* I'm using an incomplete old snapshot of the
bootstrapping process, so there are relatively few tools.
* Job control doesn't work (control-Z does nothing).
* Control-C at the shell doesn't clear the current command line(?)
* Many commands, including `git` and `make`, will segfault immediately if run
from the context of a `docker exec` but work properly from `docker run`.
No idea why.
* There's something wrong with library search paths and gcc doesn't work unless
I set `LD_LIBRARY_PATH=/lib64`; this workaround is included in the image.
## I want to build the image myself.
Say you don't want me in your chain of custody, or you're not on amd64.
1. Download the Fedora stage 3 root image and convert it into a docker image.
wget http://oirase.annexia.org/riscv/stage3-disk.img.xz
unxz stage3-disk.img.xz
sudo mount -o loop stage3-disk.img.xz stage3-disk
cd stage3-disk
sudo tar -c . | docker import -c 'CMD /bin/bash' - \
sorear/fedora-riscv-wip:pristine
cd ..
sudo umount stage3-disk
2. Download and compile a statically linked riscv64-linux-user qemu.
git clone https://github.com/arsv/riscv-qemu
cd riscv-qemu
sudo dnf install glibc-static glib2-static libstdc++-static
./configure --prefix=~/opt/riscv-qu --target-list=riscv64-linux-user \
--static
make
cd ..
3. Create an image which combines the two.
mkdir addinterp
cd addinterp
cp ../riscv-qemu/riscv64-linux-user/qemu-riscv64 .
cat >Dockerfile <<END
FROM sorear/fedora-riscv-wip:pristine
COPY qemu-riscv64 .
ENV LD_LIBRARY_PATH=/lib64
END
docker build -t sorear/fedora-riscv-wip:interp .
cd ..
## I don't want Docker.
The previous steps can be easily adapted to build a directory tree instead of
an image, which you can then invoke with a chroot. This results in much weaker
sandboxing, though.
## I'm not using Linux.
You will need a Linux VM for this project. Docker has downloadable VM tools
in the command line package for macOS and Windows; I haven't tried them.
I believe those tools do not give you easy shell access to the VM outside of a
container. The binfmt_misc setup command cannot be run in a normal container
because it's a privileged operation (it affects all containers), but it should
be possible to run it in a container started with `--privileged`, e.g.
docker run --rm -it --privileged alpine /bin/sh
Richard W.M. Jones
Sep 1, 2016, 3:47:09 AM9/1/16
to Stefan O'Rear, RISC-V SW Dev, Alex Suykov
On Wed, Aug 31, 2016 at 08:11:05PM -0700, Stefan O'Rear wrote:
> # Easy and hygenic RISC-V userspace tesing on Linux
>
> So I was playing around yesterday with [Alex Suykov's QEMU fork][Q] and [Richard
> W.M. Jones' Fedora bootstrapping efforts][F] and noticed that they go really
> well together. Using binfmt_misc to invoke qemu, I can set up a chroot running
> a RISC-V Fedora userland,
I should just note that the stage3 image is not really a "Fedora
> # Easy and hygenic RISC-V userspace tesing on Linux
>
> So I was playing around yesterday with [Alex Suykov's QEMU fork][Q] and [Richard
> W.M. Jones' Fedora bootstrapping efforts][F] and noticed that they go really
> well together. Using binfmt_misc to invoke qemu, I can set up a chroot running
> a RISC-V Fedora userland,
userland". Stage3 is mostly built out of cross-compiled packages from
an x86 host. However soon we're going to build a stage4 image cleanly
from RPMs. (None of this invalidates anything you write below!)
--
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
Read my programming and virtualization blog: http://rwmj.wordpress.com
Fedora Windows cross-compiler. Compile Windows programs, test, and
build Windows installers. Over 100 libraries supported.
http://fedoraproject.org/wiki/MinGW
Andrew Waterman
Sep 1, 2016, 4:36:18 AM9/1/16
to Richard W.M. Jones, Stefan O'Rear, RISC-V SW Dev, Alex Suykov
This is great stuff, y'all. I'm looking forward to making good use of it.
On Thursday, September 1, 2016, Richard W.M. Jones <rjo...@redhat.com> wrote:
On Thursday, September 1, 2016, Richard W.M. Jones <rjo...@redhat.com> wrote:
On Wed, Aug 31, 2016 at 08:11:05PM -0700, Stefan O'Rear wrote:
> # Easy and hygenic RISC-V userspace tesing on Linux
>
> So I was playing around yesterday with [Alex Suykov's QEMU fork][Q] and [Richard
> W.M. Jones' Fedora bootstrapping efforts][F] and noticed that they go really
> well together. Using binfmt_misc to invoke qemu, I can set up a chroot running
> a RISC-V Fedora userland,
I should just note that the stage3 image is not really a "Fedora
userland". Stage3 is mostly built out of cross-compiled packages from
an x86 host. However soon we're going to build a stage4 image cleanly
from RPMs. (None of this invalidates anything you write below!)
Even better!
--
You received this message because you are subscribed to the Google Groups "RISC-V SW Dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to sw-dev+un...@groups.riscv.org.
To post to this group, send email to sw-...@groups.riscv.org.
Visit this group at https://groups.google.com/a/groups.riscv.org/group/sw-dev/.
To view this discussion on the web visit https://groups.google.com/a/groups.riscv.org/d/msgid/sw-dev/20160901074705.GA26315%40redhat.com.
Stefan O'Rear
Sep 1, 2016, 4:47:36 AM9/1/16
to Richard W.M. Jones, RISC-V SW Dev, Alex Suykov
On Thu, Sep 1, 2016 at 12:47 AM, Richard W.M. Jones <rjo...@redhat.com> wrote:
> On Wed, Aug 31, 2016 at 08:11:05PM -0700, Stefan O'Rear wrote:
>> # Easy and hygenic RISC-V userspace tesing on Linux
>>
>> So I was playing around yesterday with [Alex Suykov's QEMU fork][Q] and [Richard
>> W.M. Jones' Fedora bootstrapping efforts][F] and noticed that they go really
>> well together. Using binfmt_misc to invoke qemu, I can set up a chroot running
>> a RISC-V Fedora userland,
>
> I should just note that the stage3 image is not really a "Fedora
> userland". Stage3 is mostly built out of cross-compiled packages from
> an x86 host. However soon we're going to build a stage4 image cleanly
> from RPMs. (None of this invalidates anything you write below!)
Ah, I see. Do I have this right: in the image I'm using, /bin/bash is
> On Wed, Aug 31, 2016 at 08:11:05PM -0700, Stefan O'Rear wrote:
>> # Easy and hygenic RISC-V userspace tesing on Linux
>>
>> So I was playing around yesterday with [Alex Suykov's QEMU fork][Q] and [Richard
>> W.M. Jones' Fedora bootstrapping efforts][F] and noticed that they go really
>> well together. Using binfmt_misc to invoke qemu, I can set up a chroot running
>> a RISC-V Fedora userland,
>
> I should just note that the stage3 image is not really a "Fedora
> userland". Stage3 is mostly built out of cross-compiled packages from
> an x86 host. However soon we're going to build a stage4 image cleanly
> from RPMs. (None of this invalidates anything you write below!)
cross-compiled, /rpmbuild/RPMS/riscv64/bash-4.3.43-1.fc24.riscv64.rpm
is compiled on the emulator (using an older version of the emulator
with HTIF disk support)?
Can I be useful here?
-s
Richard W.M. Jones
Sep 1, 2016, 5:15:20 AM9/1/16
to Stefan O'Rear, RISC-V SW Dev, Alex Suykov
Eventually -- once we have a consistent self-referential set of RPMs
-- we will be able to build a clean disk image from those RPMs, which
is what I'm calling stage4.
Jamey Hicks
Sep 1, 2016, 4:05:08 PM9/1/16
to Richard W.M. Jones, Stefan O'Rear, RISC-V SW Dev, Alex Suykov
This is very cool!
But I'm a little confused which non-cross gcc compilers you would want to use? In another project we did something very similar to cross-build RPMs using qemu-user but we generally used the cross compilers rather than emulating the compilers in order to make the builds as fast as possible.
--
You received this message because you are subscribed to the Google Groups "RISC-V SW Dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to sw-dev+un...@groups.riscv.org.
To post to this group, send email to sw-...@groups.riscv.org.
Visit this group at https://groups.google.com/a/groups.riscv.org/group/sw-dev/.
To view this discussion on the web visit https://groups.google.com/a/groups.riscv.org/d/msgid/sw-dev/20160901091516.GB26315%40redhat.com.
Richard W.M. Jones
Sep 1, 2016, 4:26:18 PM9/1/16
to Jamey Hicks, Stefan O'Rear, RISC-V SW Dev, Alex Suykov
On Thu, Sep 01, 2016 at 08:04:55PM +0000, Jamey Hicks wrote:
> This is very cool!
>
> But I'm a little confused which non-cross gcc compilers you would want to
> use? In another project we did something very similar to cross-build RPMs
> using qemu-user but we generally used the cross compilers rather than
> emulating the compilers in order to make the builds as fast as possible.
Fedora is all built self hosting. Nothing significant in Fedora is
> This is very cool!
>
> But I'm a little confused which non-cross gcc compilers you would want to
> use? In another project we did something very similar to cross-build RPMs
> using qemu-user but we generally used the cross compilers rather than
> emulating the compilers in order to make the builds as fast as possible.
cross-compiled.
To have a Fedora/RISC-V therefore we need to build everything within
the RISC-V environment, currently inside qemu-system-riscv, and
eventually on real hardware.
For a similar reason I'm at least suspicious of using qemu-user, since
the thunking down to the x86 kernel is bound to create enough
differences that the RPMs wouldn't be built correctly.
It turns out that with a sufficiently fast Xeon server, qemu is
reasonably fast enough to build RPMs. (Be nicer when TCG is
multithreaded in qemu 2.7!)
Stefan O'Rear
Sep 1, 2016, 10:28:01 PM9/1/16
to Richard W.M. Jones, Jamey Hicks, RISC-V SW Dev, Alex Suykov
On Thu, Sep 1, 2016 at 1:26 PM, Richard W.M. Jones <rjo...@redhat.com> wrote:
> For a similar reason I'm at least suspicious of using qemu-user, since
> the thunking down to the x86 kernel is bound to create enough
> differences that the RPMs wouldn't be built correctly.
Sure. So far it seems to work pretty well except when it doesn't. It
> For a similar reason I'm at least suspicious of using qemu-user, since
> the thunking down to the x86 kernel is bound to create enough
> differences that the RPMs wouldn't be built correctly.
is nice for iteration to be able to just share folders, though; I
would be interested in helping to get virtio-{scsi,net,9p} supported
for riscv system emulation, but I'm not sure where to start.
Tangent: Is there any consideration for "reproducible builds" in this
process? Will it be possible to base the integrity of the riscv
compilers on the integrity of (say) the amd64 compilers, or will any
such chain of trust necessarily stop with the rpms you are building
now?
-s
Richard W.M. Jones
Sep 2, 2016, 3:28:26 AM9/2/16
to Stefan O'Rear, Jamey Hicks, RISC-V SW Dev, Alex Suykov
On Thu, Sep 01, 2016 at 07:27:58PM -0700, Stefan O'Rear wrote:
> Tangent: Is there any consideration for "reproducible builds" in this
> process? Will it be possible to base the integrity of the riscv
> compilers on the integrity of (say) the amd64 compilers, or will any
> such chain of trust necessarily stop with the rpms you are building
> now?
There is no consideration of reproducible builds.
> Tangent: Is there any consideration for "reproducible builds" in this
> process? Will it be possible to base the integrity of the riscv
> compilers on the integrity of (say) the amd64 compilers, or will any
> such chain of trust necessarily stop with the rpms you are building
> now?
Rich.
--
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
Read my programming and virtualization blog: http://rwmj.wordpress.com
bindings from many languages. http://libguestfs.org
Reply all
Reply to author
Forward
0 new messages