Testing the new Qubes automatic build system

[Joanna Rutkowska]

Hi List,

We have recently created a fully automated build system for Qubes, that
downloads, builds and packages all the Qubes components, and finally
should spit out a ready-to-use installation ISO.

In order to use it one should use an rpm-based distro, like Fedora :)
and should ensure the following packages are installed:
* git
* createrepo
* rpm-build
* make

Unusually one can install those packages by just issuing:

sudo yum install git createrepo rpm-build make

The build system creates build environments in chroots and so no other
packages are needed on the host. All files created by the build system
are contained within the qubes-builder directory. The full build
requires some 25GB of free space, so keep that in mind when deciding
where to place this directory.

One additional useful requirement is that 'sudo root' work without any
prompt, which is default on most distros (e.g. 'sudo bash' brings you
the root shell without asking for any password). This is important as
the builder needs to switch to root and then back to user several times
during the build process.

So, to build Qubes one would do:

# Import the Qubes master key
gpg --recv-keys 0x36879494

# Verify its fingerprint, set as 'trusted'.
# This is described here:
# https://wiki.qubes-os.org/trac/wiki/VerifyingSignatures

wget http://keys.qubes-os.org/keys/qubes-developers-keys.asc
gpg --import qubes-developers-keys.asc

git clone git://git.qubes-os.org/joanna/qubes-builder.git qubes-builder
cd qubes-builder

cp builder.conf.default builder.conf
# edit the builder.conf file and set the following variables:
# (make sure to leave no spaces around '=' sign!)
# GIT_SUBDIR="joanna"
# NO_SIGN="1"

# And now to build all Qubes rpms (this will take a few hours):

make qubes

# ... and then to build the ISO

make iso

And this should produce a shiny new ISO (currently this would be for
Qubes Beta 3 pre1)

But the problem is that the resulting ISO is incorrectly built -- it
doesn't include any of the Qubes rpms for some reason. Perhaps somebody,
who has access to some fast computing servers, and who has some
experience with building Fedora-like distros, would be willing to test
this build system and help us finding and fixing this problem?

Thanks!
joanna.

[Laszlo Zrubecz]

On 11 November 2011 16:04, Joanna Rutkowska

<joa...@invisiblethingslab.com> wrote:
> We have recently created a fully automated build system for Qubes, that
> downloads, builds and packages all the Qubes components, and finally
> should spit out a ready-to-use installation ISO.
>
> In order to use it one should use an rpm-based distro, like Fedora :)
> and should ensure the following packages are installed:

This system is building binaries for the dom0 only, or the default VM
templates too?

The operating system 'under' the building system will affect the
generated binaries? Or it's just provide the building tools like gcc?

I can use huge processing resources for that kind of job, the only
problem is that I really don't know too much about building rpm based
systems... :o

Apart from that:
I see you like fedora pretty much, but this is really a good choice
for a dom0 system?

I mean, fedora is like a testing release of RedHat, and it has a too
small lifetime. For now f16 is released, and dom0 is based on f13 if
I'm correct...

Not would be better for that job a source based distro like gentoo?
These distros already has a realy working and tested build systems
exactly for this job: building customized binaries.

--
Zrubi

[Joanna Rutkowska]

On 11/11/11 19:11, Laszlo Zrubecz wrote:
> On 11 November 2011 16:04, Joanna Rutkowska
> <joa...@invisiblethingslab.com> wrote:
>> We have recently created a fully automated build system for Qubes, that
>> downloads, builds and packages all the Qubes components, and finally
>> should spit out a ready-to-use installation ISO.
>>
>> In order to use it one should use an rpm-based distro, like Fedora :)
>> and should ensure the following packages are installed:
>
> This system is building binaries for the dom0 only, or the default VM
> templates too?
>

Everything.

> The operating system 'under' the building system will affect the
> generated binaries? Or it's just provide the building tools like gcc?
>

It creates build enviroments (F13 for Dom0 stuff and F14 for template)
in chroot's, downloading all the necessary packages from fedora.

> I can use huge processing resources for that kind of job, the only
> problem is that I really don't know too much about building rpm based
> systems... :o
>
> Apart from that:
> I see you like fedora pretty much, but this is really a good choice
> for a dom0 system?
>
> I mean, fedora is like a testing release of RedHat, and it has a too
> small lifetime. For now f16 is released, and dom0 is based on f13 if
> I'm correct...
>
> Not would be better for that job a source based distro like gentoo?

Does it have a longer lifetime? How long is each version supported (in
terms of new xorg drivers being updated)?

> These distros already has a realy working and tested build systems
> exactly for this job: building customized binaries.
>

Most distro have dedicated build systems, it's just that Qubes is not a
Linux distro, and so we needed our own :) At least so I thought.

joanna.

[Laszlo Zrubecz]

On 11 November 2011 19:18, Joanna Rutkowska

<joa...@invisiblethingslab.com> wrote:
> On 11/11/11 19:11, Laszlo Zrubecz wrote:
>> On 11 November 2011 16:04, Joanna Rutkowska
>> <joa...@invisiblethingslab.com> wrote:
>>> We have recently created a fully automated build system for Qubes, that
>>> downloads, builds and packages all the Qubes components, and finally
>>> should spit out a ready-to-use installation ISO.
>>>
>>> In order to use it one should use an rpm-based distro, like Fedora :)
>>> and should ensure the following packages are installed:
>>
>> This system is building binaries for the dom0 only, or the default VM
>> templates too?
>>
>
> Everything.

cool, I will try it soon.

> Does it have a longer lifetime? How long is each version supported (in
> terms of new xorg drivers being updated)?

Sure it does.
Because you are building the whole thing from source, no binary
dependencies, and no such thing like 'backport' ever. Yo can make it
as a rolling release and the release time and frequency is always
depends only on you.

> Most distro have dedicated build systems, it's just that Qubes is not a
> Linux distro, and so we needed our own :) At least so I thought.

As I see:
gentoo (portage) is an exellent build system. With this, you can
build everything you really want.

If we are using a ready made binary distro, at the end we will have to
rebuild all the packages, and have to backport all the patches witch
are not ever released for an outdated distro like f13

We are building customized linux systems, (base OS and applications
running in jail) for firewall systems for 10 years now... We tried a
lots of things during this time, to keep them up to date, secure, and
customized by our own patches... but we always end up with gentoo and
portage.

--
Zrubi

[7v5w7go9ub0o]

On 11/11/11 13:45, Laszlo Zrubecz wrote:

>
> As I see: gentoo (portage) is an exellent build system. With this,
> you can build everything you really want.
>
> If we are using a ready made binary distro, at the end we will have
> to rebuild all the packages, and have to backport all the patches
> witch are not ever released for an outdated distro like f13
>
> We are building customized linux systems, (base OS and applications
> running in jail) for firewall systems for 10 years now... We tried a
> lots of things during this time, to keep them up to date, secure,
> and customized by our own patches... but we always end up with gentoo
> and portage.

FWIW, I installed my Gentoo-Hardened desktop about 6 years ago. Am
presently running gcc 4.5.3 default, xorg-x11-7.4, xorg-server-1.10.4,
kernels 3.0.8 hardened, and 3.1.0 un-hardened.

I periodically run portage to find out what's been updated, and
painlessly keep the box current. Where necessary, portage will rebuild
programs that depend upon the dynamic loading of any specific modules
that have been updated and replaced by newer versions - based upon
instructions from the package "maintainer" and/or based upon a "scan" of
installed packages.

[Daniel Cegiełka]

I agree with opinions that portage would be the best solution.
Portage has built-in git support (git.eclass), so you can download and
build packages directly from qubes repo.

http://devmanual.gentoo.org/eclass-reference/git.eclass/index.html

and example of how to build packages directly from the git repo:

http://gentoo-overlays.zugaina.org/xarthisius/portage/dev-lang/path64/path64-9999.ebuild

regards,
daniel

[Joanna Rutkowska]

We might consider changing the dist used for Dom0, as well as moving all
the GUI into a separate domain, but not earlier then after 1.0 release.

For now we need to get the current build system to work. And it seems to
be mostly working (i.e. rpms are generated fine), only the iso
generation seems to be failing (which strange, because we almost didn't
change anything there, and it worked fine previously...).

joanna.

[Daniel Cegiełka]

> We might consider changing the dist used for Dom0, as well as moving all
> the GUI into a separate domain, but not earlier then after 1.0 release.
>
> For now we need to get the current build system to work. And it seems to
> be mostly working (i.e. rpms are generated fine), only the iso
> generation seems to be failing (which strange, because we almost didn't
> change anything there, and it worked fine previously...).
>
> joanna.

http://openbuildservice.org/

http://en.opensuse.org/Portal:Build_Service

daniel :)

[Joanna Rutkowska]

Using 3rd part servers to build Qubes is not something we would consider
in a foreseeable future, because this would require that we 1) trust all
the stuff at the hosting provider that have access to the server, and 2)
trust that the server cannot be easily compromised e.g. via hypothetical
SSH bug, or 3) through an attack coming from a compromised client
machine of a user that was granted access to the build server.

Our goal is that it should be possible for anybody to build Qubes
without any 3rd party build server. This is how our build system works.
Specifically, the version that will released via our website will be
build on a local machine at my apartment and then signed by my keys.

Can we get back to the main topic, which is how to make ISO generation
to work? :)

joanna.

[Daniel Cegiełka]

W dniu 11 listopada 2011 22:17 użytkownik Joanna Rutkowska
<joa...@invisiblethingslab.com> napisał:

Portage still looks best if you want to do everything on your own
infrastructure. Portage is also very well documented.

http://en.gentoo-wiki.com/wiki/Build_Your_Own_LiveCD_or_LiveDVD

But I'm not objective.. I always had an allergy to rpm :)

daniel.

>
> joanna.
>
>

[Laszlo Zrubecz]

On 11 November 2011 16:04, Joanna Rutkowska
<joa...@invisiblethingslab.com> wrote:

> In order to use it one should use an rpm-based distro, like Fedora :)
> and should ensure the following packages are installed:

It seems this will work only under fedora... Just tried under CentOS
6.0, and got this error:

+ echo '-> Installing package groups...'
-> Installing package groups...
+ yum install -c /root/qubes-builder/yum.conf -y -x kernel
--disablerepo=updates --installroot=/root/qubes-builder/fc13 gnupg git
gcc python make rpm-build MAKEDEV glibc-devel redhat-rpm-config
Loaded plugins: fastestmirror
qubes-builder-pkgs
| 1.3
kB 00:00 ...
qubes-builder-pkgs/primary
| 191
B 00:00 ...
Error: Cannot retrieve repository metadata (repomd.xml) for
repository: fedora. Please verify its path and try again
make: *** [xen] Error 1

Tomorrow I will try under f16? f15? f14? - which should I pick?
but I'm really don't feel good when I have to install a full desktop
system for some rpm building work :P

Ps: Sorry for hijacking the thread with that portage thing ;)

--
Zrubi

[Laszlo Zrubecz]

On 12 November 2011 02:53, Laszlo Zrubecz <laszlo....@gmail.com> wrote:
> On 11 November 2011 16:04, Joanna Rutkowska
> <joa...@invisiblethingslab.com> wrote:
>> In order to use it one should use an rpm-based distro, like Fedora :)
>> and should ensure the following packages are installed:
>
> It seems this will work only under fedora... Just tried under CentOS
> 6.0, and got this error:
>
> + echo '-> Installing package groups...'
> -> Installing package groups...
> + yum install -c /root/qubes-builder/yum.conf -y -x kernel
> --disablerepo=updates --installroot=/root/qubes-builder/fc13 gnupg git
> gcc python make rpm-build MAKEDEV glibc-devel redhat-rpm-config
> Loaded plugins: fastestmirror
> qubes-builder-pkgs
>                                                                | 1.3
> kB     00:00 ...
> qubes-builder-pkgs/primary
>                                                                |  191
> B     00:00 ...
> Error: Cannot retrieve repository metadata (repomd.xml) for
> repository: fedora. Please verify its path and try again
> make: *** [xen] Error 1

My mistake, I just had to tell yum about my proxy....

--
Zrubi

[Laszlo Zrubecz]

Now, the building process stopped after a lots of try:

RPM build errors:
File not found by glob:
/home/user/rpmbuild/BUILDROOT/xen-4.1.1-6.qubes.x86_64/usr/share/man/man1/xm.1*
File not found by glob:
/home/user/rpmbuild/BUILDROOT/xen-4.1.1-6.qubes.x86_64/usr/share/man/man5/xend-config.sxp.5*
File not found by glob:
/home/user/rpmbuild/BUILDROOT/xen-4.1.1-6.qubes.x86_64/usr/share/man/man5/xmdomain.cfg.5*
make: *** [rpms] Error 1
make: *** [xen] Error 2

And some errors what I was able to resolv before:

- lots of missing groups under the jail, but only this stopped the
process for some reason:
no floppy group in /root/qubes-builder/fc13/etc/group

and at some point: `useradd -u 0 -m user` -> `useradd -o -u 0 -m user`
if you wanna make more 0 uid....

But this building process is a pain :(
- Not parallel at all, so my the huge cumputing power is not really
help. the slowest thing is writing the output to the terminal ;)
- at every error, I have to start from the begginning? - no way to
make it fully finish... with some of the errors, when I run the
command again and again it was goes further till the last error I
got....

---
Zrubi

[Joanna Rutkowska]

On 11/12/11 03:52, Laszlo Zrubecz wrote:
> Now, the building process stopped after a lots of try:
>
> RPM build errors:
> File not found by glob:
> /home/user/rpmbuild/BUILDROOT/xen-4.1.1-6.qubes.x86_64/usr/share/man/man1/xm.1*
> File not found by glob:
> /home/user/rpmbuild/BUILDROOT/xen-4.1.1-6.qubes.x86_64/usr/share/man/man5/xend-config.sxp.5*
> File not found by glob:
> /home/user/rpmbuild/BUILDROOT/xen-4.1.1-6.qubes.x86_64/usr/share/man/man5/xmdomain.cfg.5*
> make: *** [rpms] Error 1
> make: *** [xen] Error 2
>
>
>

Very strange. I tested this build on several distros: F13, F14, and even
on Amazon AMI Linux and it always builds fine...

> And some errors what I was able to resolv before:
>
> - lots of missing groups under the jail, but only this stopped the
> process for some reason:
> no floppy group in /root/qubes-builder/fc13/etc/group
>

Very strange, never observed. Perhaps some problems with networking on
your side?

> and at some point: `useradd -u 0 -m user` -> `useradd -o -u 0 -m user`
> if you wanna make more 0 uid....
>

Do we really create it on the host?

>
> But this building process is a pain :(
> - Not parallel at all, so my the huge cumputing power is not really
> help. the slowest thing is writing the output to the terminal ;)

Building of many components is parallel, e.g. kernel, xen, kde, all
build in parallel.

Is there an easy way to instruct make to build two targets in parallel,
so e.g. if I have:

qubes: component_1 component_2

That make build those two in parallel? On the other hand, given the fact
that most time consuming components (kernel, xen, kde) do build fully
parallel, further paralleling those components might not be desired. One
component that is a bottleneck is template, but this is because of rpm
being single-threaded.

> - at every error, I have to start from the begginning?

No, you can manually specify which components to build. Seeing that the
'qubes' target is defined as:

qubes: get-sources xen core kernel gui template kde-dom0 installer
qubes-manager dom0-updates

if you get an error building, say, kde-dom0, then you can resume the
process by typing:

make kde-dom0 installer qubes-manager dom0-updates

But then again, it is very strange that the build process fails on your
system...

> - no way to
> make it fully finish... with some of the errors, when I run the
> command again and again it was goes further till the last error I
> got....
>
>

Strange. Can you locate the actual error messages?

Thanks,
joanna.

[Radoslaw Szkodzinski]

2011/11/11 Daniel Cegiełka <daniel....@gmail.com>:

> Portage still looks best if you want to do everything on your own
> infrastructure. Portage is also very well documented.
>
> http://en.gentoo-wiki.com/wiki/Build_Your_Own_LiveCD_or_LiveDVD
>
> But I'm not objective.. I always had an allergy to rpm :)
>
> daniel.

Silly cellphone ate half of my message.

I was to write that there's a possible another build system that's
based on portage.
Sabayon Linux Entropy infrastructure, coming with a dedicated build
server/client as well.
It supports delta transfers, GPG signatures and multiple repositories,
has nice command line and graphical interfaces.
Base Sabayon packages are not GPG-signed, but do have strong hash
checksums (SHA256), while the tree state, which contains those
checksums, is signed with a GPG key.

https://www.sabayon.org/

It's binary-based, yet it's possible to build anything from source
using Portage as well as merge installed packages both ways. (entropy
<-> portage)
In fact, only syncing from portage to entropy is manual, resulting in
a special package repository.
Building packages is simple and integrated well - both local and
remote builds use the same entropy-build-server infrastructure.

The distribution is released as a both nightly "spins" and rarer major
releases - both are installable livecds and livedvds. The development
model is a rolling-release similar to Gentoo, but a bit slower and
more tested. Their spins support incremental overlay modification. but
they also have an Anaconda installer and it works pretty well.
The developers cooperate tightly with Gentoo parent distributions.

--
Radosław Szkodziński

[Joanna Rutkowska]

On 11/12/11 14:42, Laszlo Zrubecz wrote:

> On 12 November 2011 14:11, Laszlo Zrubecz <laszlo....@gmail.com> wrote:
>> I will start is over again from scratch and doing 'make tasks'
>> individually and try to log the output, to idendtify the all the
>> errors I got...
>
> actualli this file is not accessible at all:
> http://www.kernel.org/pub/software/utils/pciutils/pciutils-2.2.9.tar.bz2
>
> Just found it on the main site:
> ftp://atrey.karlin.mff.cuni.cz/pub/linux/pci/pciutils-2.2.9.tar.gz
> but not signed... so I just had to remove that prart from teh makefile...
>
>
> but then I got another error what I can't resolv:
> kdemultimedia-4.4.5.tar.bz2: OK
> make[2]: Leaving directory
> `/root/qubes-builder/qubes-src/kde-dom0/kdemultimedia'
> make[1]: Leaving directory `/root/qubes-builder/qubes-src/kde-dom0'
> Initialized empty Git repository in
> /root/qubes-builder/qubes-src/antievilmaid/.git/
> fatal: The remote end hung up unexpectedly
> make: *** [get-sources] Error 128
>
> Not even know where to search for the error codes...
>

Ah, I see you didn't set GIT_SUBDIR="joanna"...

[ARAI Shun-ichi]

Hi,

I tried it and met difficulty.
I did like this.

> git clone git://git.qubes-os.org/joanna/qubes-builder.git qubes-builder
> cd qubes-builder
>
> cp builder.conf.default builder.conf
> # edit the builder.conf file and set the following variables:
> # (make sure to leave no spaces around '=' sign!)
> # GIT_SUBDIR="joanna"
> # NO_SIGN="1"

The diff is...

$ diff builder.conf.default builder.conf
11a12
> GIT_SUBDIR="joanna"
37a39
> NO_SIGN="1"

And I typed...

> make qubes

Then it result many errors (at "template" target?).

Transaction Check Error:
file /usr/share/locale/af/LC_MESSAGES/gnome-media-2.0.mo conflicts between attempted installs of gnome-media-libs-2.32.0-2.fc14.i686 and gnome-media-libs-2.32.0-2.fc14.x86_64
file /usr/share/locale/am/LC_MESSAGES/gnome-media-2.0.mo conflicts between attempted installs of gnome-media-libs-2.32.0-2.fc14.i686 and gnome-media-libs-2.32.0-2.fc14.x86_64
(snip)
file /usr/share/locale/zu/LC_MESSAGES/gdm.mo conflicts between attempted installs of gdm-1:2.32.1-2.fc14.i686 and gdm-1:2.32.1-2.fc14.x86_64
file /var/lib/gdm/.gconf.mandatory/%gconf-tree.xml conflicts between attempted installs of gdm-1:2.32.1-2.fc14.i686 and gdm-1:2.32.1-2.fc14.x86_64

Error Summary
-------------

Is there any advise?

[Marek Marczykowski]

Ah, you did it on multilib system.
Before make:
export YUM_OPTS=--set=multilib_policy=best

--
Pozdrawiam / Best Regards,
Marek Marczykowski
Invisible Things Lab

[ARAI Shun-ichi]

<4EC14902...@invisiblethingslab.com>;
Marek Marczykowski <marm...@invisiblethingslab.com> wrote
as Subject "Re: [qubes-devel] Testing the new Qubes automatic build system":

>>> make qubes
>>
>> Then it result many errors (at "template" target?).
>>
>> Transaction Check Error:
>> file /usr/share/locale/af/LC_MESSAGES/gnome-media-2.0.mo conflicts between attempted installs of gnome-media-libs-2.32.0-2.fc14.i686 and gnome-media-libs-2.32.0-2.fc14.x86_64
>> file /usr/share/locale/am/LC_MESSAGES/gnome-media-2.0.mo conflicts between attempted installs of gnome-media-libs-2.32.0-2.fc14.i686 and gnome-media-libs-2.32.0-2.fc14.x86_64
>> (snip)
>> file /usr/share/locale/zu/LC_MESSAGES/gdm.mo conflicts between attempted installs of gdm-1:2.32.1-2.fc14.i686 and gdm-1:2.32.1-2.fc14.x86_64
>> file /var/lib/gdm/.gconf.mandatory/%gconf-tree.xml conflicts between attempted installs of gdm-1:2.32.1-2.fc14.i686 and gdm-1:2.32.1-2.fc14.x86_64
>>
>> Error Summary
>> -------------
>>
>> Is there any advise?
>
> Ah, you did it on multilib system.
> Before make:
> export YUM_OPTS=--set=multilib_policy=best

The problem was resolved.
I tried following steps.

1. export YUM_OPTS=--set=multilib_policy=best before make
This causes same errors.

2. modify qubes-builder/yum.conf
Added this parameter.
multilib_policy=best
This causes same errors.

3. modify qubes-builder/qubes-src/template-builder/yum.conf
Added this parameter, and with original qubes-builder/yum.conf.
multilib_policy=best

Then "make template" succeeded.

Thanks!

[Laszlo Zrubecz]

On 11 November 2011 16:04, Joanna Rutkowska
<joa...@invisiblethingslab.com> wrote:

> make iso

Finaly just got it.
'make iso' was successfully run, I get an iso called:
iso/Qubes-R1-Beta3-pre1-x86_64-DVD.iso

How should I check it if it's wrong or not?

make is process wasn't too verbose, soo how shoud I know what should
happen during the process?

--
Zrubi