How to build qubes components without privileged commands?

[Paras Chetal]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256

Hello!

I am currently working on integrating the various qubes components
with oss-fuzz (as part of my GSoC project). That requires the
components to be built inside a docker container. I have successfully
been able to write the Dockerfile (the travis builds helped) and with
a few tweaks, am able to build the individual components locally
through the qubes-builder running inside the container.

However, the regular build process requires the containers to be run
with the "--privileged" option because the qubes-builder uses commands
like mount while interacting with the chroot. Running the containers
in privileged mode doesn't seem to be an option in oss-fuzz [1].

@jpo gave some suggestions:

1. Using proot [2] instead of chroot and mount.
I still have to try it out. Basically that would involve changing all
the chroot commands with "proot -r" and the mount commands with "proot
- -b", right? Also, we'll have to take care of handling umount while
cleaning the build.


2. Ignoring the mounts and see what breaks:
I removed all the mount instructions from the qubes-builder (and the
builder plugin) to test this. The dependencies for the component do
not get installed and so it doesn't get built. This is the error I get:

Curl error (37): Couldn't read a file:// file for
file:///etc/pki/rpm-gpg/RPM-GPG-KEY-qubes-3-primary [Couldn't open
file /etc/pki/rpm-gpg/RPM-GPG-KEY-qubes-3-primary]
The downloaded packages were saved in cache until the next successful
transaction.
You can remove cached packages by executing 'dnf clean packages'.
/home/user/qubes-builder/qubes-src/builder-fedora/Makefile.fedora:106:
recipe for target 'dist-build-dep' failed
make[2]: *** [dist-build-dep] Error 1
- --> build failed!
Makefile.generic:147: recipe for target 'packages' failed
make[1]: *** [packages] Error 1
Makefile:209: recipe for target 'linux-utils-vm' failed
make: *** [linux-utils-vm] Error 1

Is it possible to circumvent the mounts somehow?


3. Drop the chroot entirely.
If it is possible to build the components directly, I think it would
be best considering what I know of the oss-fuzz environment.



@marmarek, how do you think we should go ahead with this?



Regards,
Paras Chetal


[1]: https://github.com/google/oss-fuzz/blob/master/infra/helper.py#L215
[2]: https://github.com/proot-me/PRoot
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2

iQEcBAEBCAAGBQJZMJ32AAoJEA4SQJU2s0ILkmoIAJs4Xwx2lNn7fyMuM6q+ZDY8
0FQSUZTAd6718bapjVJsphWgpc2mRp45PMcwLWGdBjTH1gvqYktLdXx6ilZIo/kP
vew1WMvP2EyJlqUsZikdsxeqrF0SfM+6xpdU/aMMUo1C49YO5YKoCwWov9nF3RHg
pm+Bpoa2yZ/VOGeqCUDl7N44HBWfSFPF9CtVzuCBjfPWxi+WWMijCwaxuasdkUI6
v+oNeHj+2q4ARAn0qbR38ABAlW/W2ohMURoNZMgURbKmq/4R2hQ5EG5C2TgNRAOC
N+mpmV75lzphofjOCpP0owuJC7Fb8YBGB9bNGKXXZ6IWI3N5dcYdoodK39TBgPI=
=AINe
-----END PGP SIGNATURE-----

[Marek Marczykowski-Górecki]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256

On Fri, Jun 02, 2017 at 04:36:56AM +0530, Paras Chetal wrote:
> Hello!

Hi!

> I am currently working on integrating the various qubes components
> with oss-fuzz (as part of my GSoC project). That requires the
> components to be built inside a docker container. I have successfully
> been able to write the Dockerfile (the travis builds helped) and with
> a few tweaks, am able to build the individual components locally
> through the qubes-builder running inside the container.
>
> However, the regular build process requires the containers to be run
> with the "--privileged" option because the qubes-builder uses commands
> like mount while interacting with the chroot. Running the containers
> in privileged mode doesn't seem to be an option in oss-fuzz [1].
>
> @jpo gave some suggestions:
>
> 1. Using proot [2] instead of chroot and mount.
> I still have to try it out. Basically that would involve changing all
> the chroot commands with "proot -r" and the mount commands with "proot

> -b", right? Also, we'll have to take care of handling umount while
> cleaning the build.

I don't know proot, but tried fakechroot in the past (similar but using
LD_PRELOAD instead of ptrace). I gave up after few iterations of
workarounds...

> 2. Ignoring the mounts and see what breaks:
> I removed all the mount instructions from the qubes-builder (and the
> builder plugin) to test this. The dependencies for the component do
> not get installed and so it doesn't get built. This is the error I get:
>
> Curl error (37): Couldn't read a file:// file for
> file:///etc/pki/rpm-gpg/RPM-GPG-KEY-qubes-3-primary [Couldn't open
> file /etc/pki/rpm-gpg/RPM-GPG-KEY-qubes-3-primary]
> The downloaded packages were saved in cache until the next successful
> transaction.
> You can remove cached packages by executing 'dnf clean packages'.
> /home/user/qubes-builder/qubes-src/builder-fedora/Makefile.fedora:106:
> recipe for target 'dist-build-dep' failed
> make[2]: *** [dist-build-dep] Error 1

> --> build failed!
> Makefile.generic:147: recipe for target 'packages' failed
> make[1]: *** [packages] Error 1
> Makefile:209: recipe for target 'linux-utils-vm' failed
> make: *** [linux-utils-vm] Error 1
>
> Is it possible to circumvent the mounts somehow?

Probably, but not sure if worth it - see below.

> 3. Drop the chroot entirely.
> If it is possible to build the components directly, I think it would
> be best considering what I know of the oss-fuzz environment.

choot is used only to have uniform build environment there
(appropriate distribution version) to build a _package_, including linking
with the right libraries etc. Regardless of host distribution.
Actually Docker could be used for this, instead of plain chroot, but at
the time we've written qubes-builder, Docker didn't existed yet.

Since packaging and moving results to other systems probably doesn't
matter in your case, you can drop the whole thing. And use standard
build commands - in most cases "make all" is enough - as long as you
have build dependencies installed. You can also use existing packaging
scripts (call dpkg-buildpackage or rpm-build -bb rpm_spec/*.spec).
Some components require BACKEND_VMM=xen env variable at build time.

As for build dependencies - you can install them from appropriate
repository (yum.qubes-os.org or deb.qubes-os.org).

- --
Best Regards,
Marek Marczykowski-Górecki
Invisible Things Lab
A: Because it messes up the order in which people normally read text.
Q: Why is top-posting such a bad thing?

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2

iQEcBAEBCAAGBQJZMKRcAAoJENuP0xzK19csPQ8H/2oQ3fvK1+C9XwxuHdb3AO9r
4miQcuFJ19Sz74CRFNelmjICdjLn67RgKTWSZ5PhTguL9BfJeDYkrZ5uBXNRaVHH
25FwmXbWiRvKsI2PL5Jy3IRq3x4TPCJlLmCsdmliSQvRT/2gyiMGf4KwQnE8WftA
ropZRXMbxZwUJvEF+UYk2RGiL1Jba3LGQ/z6Gpvuoh0KPdDaHjEfiqRjUq2owsGs
sfJ4raCGsgNbCXwyro7nRyUlFPTG0xW+s9HEL/Fx41QgzbuW0Dmi553KxkQfmexK
6B85+c6lYI/W71K1soHde/ezHbdqY/hHEGhYOjD0DbqxbVx6ecFlKBZllgrV9ao=
=pVxH
-----END PGP SIGNATURE-----

[Paras Chetal]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256

Alright, I will go ahead with building the components directly then.
If I run into any issues I'll ask here. Thank you :)


Regards,
Paras Chetal

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2

iQEcBAEBCAAGBQJZMKzTAAoJEA4SQJU2s0ILqGIH+gI4CtxBlU85MYnHQKqRwjLV
p/Zp7wYpa/Qm1MbCuwFFY7sKXnfZG8bUwwksFreqtfH3TGBHe/IE5E0+zALCuiXh
4UiDnBFgdZW9hhIG1Bo1JigC5kzwYdmIHYLJ9OHF+VuGNLcMZB8yo7vD58AOTHJQ
MM77nMmArM7zRJEoTFwTydCTH1EzRhYAiXSPkWZbSoUOkdsZ4MPMSy+ZLJPYQHRA
SjzbrwpRdnkGy8sOpDtTh1GnGf1FxTZy02ELF1ScmPbYSDkdnfc1QZMdms4EOxAb
taY1UXXsO915LP+ei8IN9JX00ja8OAE2MeldhC+B5gd81kcUA85sfnUSjuCMeGI=
=Marn
-----END PGP SIGNATURE-----