Which parts of qubes-builder are guaranteed to work/supported?

[Achim Patzner]

Hi!

Is it worth creating issues if certain parts of the Builder tools do not work (e. g. template-local-centos7 or template-local-fc29+xfce) which would be creating things not in the Qubes distribution?

Achim

[Frédéric Pierret (fepitre)]

What do you mean by 'parts'? The build of CentOS 7 and Fedora 29 with XFCE flavor works well. I did a build last week for both of them. Can you be more precise please?

Frédéric

[Achim Patzner]

Right now I'm not even getting to centos-7:

make get-sources get-sources-extra qubes-vm is stopping at

-> Installing core RPM packages...
error: Failed dependencies:
glibc = 2.28-9.fc29 is needed by
glibc-all-langpacks-2.28-9.fc29.x86_64
glibc-common = 2.28-9.fc29 is needed by
glibc-all-langpacks-2.28-9.fc29.x86_64
make[1]: ***
[/home/user/qubes-builder/qubes-src/builder-rpm/Makefile-leg
acy.rpmbuilder:35:
/home/user/qubes-builder/chroot-fc29/home/user/.prepared_bas
e] Error 1
make: *** [Makefile:217: vmm-xen-vm] Error 1

[unman]

I would say ALL parts are "intended to work" rather than "guaranteed to work".
Anything that is included in builder *should* work - if it doesn't, it
may be an issue, but I'd suggest raising it here first before heading
over to github. You've already encountered this.
(I exclude live iso build which hasnt worked as written for years.)

[Frédéric Pierret (fepitre)]

You are trying to build CentOS 7. I guess your DISTS_VM=centos7 ? I saw Fedora 29 related packages. Can you provide your builder.conf please ? (e.g. using gist).

[Achim Patzner]

Am Donnerstag, den 03.01.2019, 00:02 +0000 schrieb unman:
> On Wed, Jan 02, 2019 at 09:13:40PM +0100, Achim Patzner wrote:
> > Is it worth creating issues if certain parts of the Builder tools do
> > not work (e. g. template-local-centos7 or template-local-fc29+xfce)
> > which would be creating things not in the Qubes distribution?

> I would say ALL parts are "intended to work" rather than "guaranteed to work".

Let me rephrase it: Should bugs be reported to qubes-issues if the
affected item is not relevant for the generation of distribution-
related stuff? After all just reading it will bind resources that are
probably needed somewhere else.

> Anything that is included in builder *should* work - if it doesn't, it
> may be an issue, but I'd suggest raising it here first before heading
> over to github. You've already encountered this.

In that case a good example would be that make -j n (with n >=4) builds
of the targe qubes-vm are failing, right? (Which would also fix
rebuilding targets that have been finished already, too).

> (I exclude live iso build which hasnt worked as written for years.)

Maybe it should be removed...


Achim

[Robert Rettig]

> Right now I'm not even getting to centos-7:
>
> make get-sources get-sources-extra qubes-vm is stopping at
>
> -> Installing core RPM packages...
> error: Failed dependencies:
> glibc = 2.28-9.fc29 is needed by
> glibc-all-langpacks-2.28-9.fc29.x86_64
> glibc-common = 2.28-9.fc29 is needed by
> glibc-all-langpacks-2.28-9.fc29.x86_64
> make[1]: ***
> [/home/user/qubes-builder/qubes-src/builder-rpm/Makefile-legacy.rpmbuilder:35:

> /home/user/qubes-builder/chroot-fc29/home/user/.prepared_base] Error 1

> make: *** [Makefile:217: vmm-xen-vm] Error 1

Different environment. Started with a generic/fedora29 box (see https://app.vagrantup.com/generic/boxes/fedora29 )

Got same error but different reason
https://pastebin.com/raw/Efi5JQKU

```
E: Failed to fetch https://deb.debian.org/debian/pool/main/r/reprepro/reprepro_4.16.0-1_amd64.deb GnuTLS recv error (-54): Error in the pull function.

E: Unable to fetch some archives, maybe run apt-get update or try with --fix-missing?
make[1]: *** [/home/vagrant/qubes-builder/qubes-src/builder-debian/Makefile.debian:176: /home/vagrant/qubes-builder/chroot-jessie/home/user/.prepared_base] Error 100
```

How can I resume the broken build?

[unman]

Obviously you have had some network issue, so downloads have failed for
jessie.(Why Jessie? Latest Whonix is based on stretch.)
It doesnt look as if you built much (anything) so you should be able to
just start the build again.
I recommend breaking the build down to separate distros, rather than
building all at once. Also, you can use make qubes-vm and make template
as separate step.

If you use a caching proxy upstream from the build device then this
helps to mitigate the pain, and also dramatically speeds up template
updates. Since you are downloading to update templates anyway, why
download again to build (vice versa)

[Robert Rettig]

The build especially the gcc takes a lot of time. The network issue happened somehow at night.
Therefore I asked how to resume and retry the build without building gcc again ... .
Normally I would assume if a component like gcc is build upfront that it will be used from later components.

To make it more clear what I did.
Essentially I followed the rabbit in the documentation from
https://www.qubes-os.org/faq/#how-do-i-build-qubes-from-sources
to https://www.qubes-os.org/doc/qubes-builder/
to https://www.qubes-os.org/doc/qubes-r3-building/ .
Maybe that needs some cleanup and pick up your recommendation "breaking the build down to separate distros"?

I choose `qubes` meta target as it should "build all required components in correct order".
"List of components is configured in builder.conf. You can also check the current value at the end of make help, or using make build-info."
For me that shows up
'gcc vmm-xen core-libvirt core-vchan-xen core-qubesdb linux-utils python-cffi python-xcffib python-sphinx python-pillow python-quamash python-objgraph python-hid python-u2flib-host core-admin core-admin-client core-admin-addon-whonix core-admin-linux core-agent-linux intel-microcode linux-firmware linux-kernel artwork gui-common gui-daemon gui-agent-linux gui-agent-xen-hvm-stubdom vmm-xen-stubdom-linux app-linux-split-gpg app-linux-tor app-thunderbird app-linux-pdf-converter app-linux-img-converter app-linux-input-proxy app-linux-usb-proxy app-linux-snapd-helper app-shutdown-idle app-yubikey app-u2f mgmt-salt mgmt-salt-base mgmt-salt-base-topd mgmt-salt-base-config mgmt-salt-base-overrides mgmt-salt-dom0-qvm mgmt-salt-dom0-virtual-machines mgmt-salt-dom0-update infrastructure meta-packages dbus manager desktop-linux-common desktop-linux-kde desktop-linux-xfce4 desktop-linux-i3 desktop-linux-awesome desktop-linux-manager linux-dom0-updates linux-pvgrub2 linux-gbulb linux-scrypt linux-template-builder installer-qubes-os linux-yum linux-deb antievilmaid xscreensaver builder builder-rpm builder-debian template-whonix'

As I wrote https://groups.google.com/d/msgid/qubes-users/AM0PR04MB590621E2E36C66F18DBBB4B8D2B30%40AM0PR04MB5906.eurprd04.prod.outlook.com?utm_medium=email&utm_source=footer
I would like to rebuild a component which includes my BIOS SLIC information. The changes should be part of `vmm-xen` component.

And yes I would like to create the whole ISO with just those changes ... if possible.
Currently a fresh install with the original Qubes ISO is not possible for me. I have some trouble to install from USB device (dd'ed the ISO to flash) and I even with external USB DVD.
Installer works as expected without errors but after reboot the pc, it show black screen no errors and is haltet (only power switch holding down >=4s is working).

[unman]

If you check in builder.conf you'll see the components list - comment
out those that are already built as you go, so you arent rebuilding them
over and over, and the build should continue from where you left off and
use the already built components.
There is a long standing open issue about strengthening the build
process. I confess that using a caching proxy (and having a stable
net connection) helps no end.
If you are only rebuilding one component, then there's no reason why you
cant use Qubes packages for the rest - either strip them from the CD,
or pull from repo, and just 'make iso'.

[Foppe de Haan]

OT, but could you briefly clarify the benefit of running the xfce flavor over the generic f29 template?

[Frédéric Pierret]

Sure. I globally don't like gnome 3 tools/desktop. I prefer to use XFCE
tools/desktop directly in my AppVM. It's simple, flexible and easily
customizable. It's really a matter of taste. They are still some minor
UX tuning to do like default tray-icon of power management and
dnfdragora appearing but it's working fine and I'm using these flavors
since a very long time. So if there is also some feedback from other
users, it will help me to fix UX bugs I have not seen yet.

Frédéric