Building Qubes from source, strange error.

[lok...@gmail.com]

I'm trying to build Qubes from source, but when following the instructions (https://www.qubes-os.org/doc/qubes-builder/), the build fails with the message below. Does anyone have any advice?

Regards,
Elias

make[1]: Leaving directory '/home/emartenson/qubes-builder'
make[1]: Entering directory '/home/emartenson/qubes-builder'
-> Building core-admin-linux (rpm_spec/core-dom0-linux.spec) for fc23 dom0 (logfile: build-logs/core-admin-linux-dom0-fc23.log)
--> Done:
qubes-src/core-admin-linux/pkgs/fc23/x86_64/qubes-core-dom0-linux-3.2.12-1.fc23.x86_64.rpm
qubes-src/core-admin-linux/pkgs/fc23/x86_64/qubes-core-dom0-linux-debuginfo-3.2.12-1.fc23.x86_64.rpm
qubes-src/core-admin-linux/pkgs/fc23/x86_64/qubes-core-dom0-linux-kernel-install-3.2.12-1.fc23.x86_64.rpm
-> Building core-admin-linux (rpm_spec/core-dom0-vaio-fixes.spec) for fc23 dom0 (logfile: build-logs/core-admin-linux-dom0-fc23.log)
--> Done:
qubes-src/core-admin-linux/pkgs/fc23/x86_64/qubes-core-dom0-vaio-fixes-1.6.1-1.fc23.x86_64.rpm
make[1]: Leaving directory '/home/emartenson/qubes-builder'
make[1]: Entering directory '/home/emartenson/qubes-builder'
make[1]: Leaving directory '/home/emartenson/qubes-builder'
-> Building core-agent-linux (rpm_spec/core-vm.spec) for fc23 vm (logfile: build-logs/core-agent-linux-vm-fc23.log)
--> Done:
qubes-src/core-agent-linux/pkgs/fc23/x86_64/python2-dnf-plugins-qubes-hooks-3.2.16-1.fc23.x86_64.rpm
qubes-src/core-agent-linux/pkgs/fc23/x86_64/python3-dnf-plugins-qubes-hooks-3.2.16-1.fc23.x86_64.rpm
qubes-src/core-agent-linux/pkgs/fc23/x86_64/qubes-core-vm-3.2.16-1.fc23.x86_64.rpm
qubes-src/core-agent-linux/pkgs/fc23/x86_64/qubes-core-vm-debuginfo-3.2.16-1.fc23.x86_64.rpm
qubes-src/core-agent-linux/pkgs/fc23/x86_64/qubes-core-vm-systemd-3.2.16-1.fc23.x86_64.rpm
qubes-src/core-agent-linux/pkgs/fc23/x86_64/qubes-core-vm-sysvinit-3.2.16-1.fc23.x86_64.rpm
-> Building core-agent-linux (rpm_spec/core-vm-doc.spec) for fc23 vm (logfile: build-logs/core-agent-linux-vm-fc23.log)
--> Done:
qubes-src/core-agent-linux/pkgs/fc23/noarch/qubes-core-vm-doc-3.2.16-1.noarch.rpm
make[1]: Entering directory '/home/emartenson/qubes-builder'
-> Building linux-kernel (kernel.spec) for fc23 dom0 (logfile: build-logs/linux-kernel-dom0-fc23.log)
--> Done:
qubes-src/linux-kernel/pkgs/fc23/x86_64/kernel-4.4.55-11.pvops.qubes.x86_64.rpm
qubes-src/linux-kernel/pkgs/fc23/x86_64/kernel-devel-4.4.55-11.pvops.qubes.x86_64.rpm
qubes-src/linux-kernel/pkgs/fc23/x86_64/kernel-qubes-vm-4.4.55-11.pvops.qubes.x86_64.rpm
make[1]: Leaving directory '/home/emartenson/qubes-builder'
make[1]: Entering directory '/home/emartenson/qubes-builder'
-> Building artwork (qubes-artwork.spec) for fc23 dom0 (logfile: build-logs/artwork-dom0-fc23.log)
--> build failed!
make[2]: Entering directory '/home/emartenson/qubes-builder'
/home/emartenson/qubes-builder/qubes-src/builder-fedora//update-local-repo.sh fc23
sudo BACKEND_VMM=xen chroot /home/emartenson/qubes-builder/chroot-fc23 sh -c 'cd /home/user/qubes-src/artwork; dnf --refresh -y update'
Dependencies resolved.
Nothing to do.
Complete!
sudo BACKEND_VMM=xen chroot /home/emartenson/qubes-builder/chroot-fc23 sh -c 'cd /home/user/qubes-src/artwork; dnf builddep --allowerasing --best -y qubes-artwork.spec'
Package ImageMagick-6.9.2.7-1.fc23.x86_64 is already installed, skipping.
Package netpbm-progs-10.75.99-1.fc23.x86_64 is already installed, skipping.
No matching package to install: 'qubes-utils >= 3.9.0'
Not all dependencies satisfied
Error: Some packages could not be found.
/home/emartenson/qubes-builder/qubes-src/builder-fedora/Makefile.fedora:106: recipe for target 'dist-build-dep' failed
make[2]: *** [dist-build-dep] Error 1
make[2]: Leaving directory '/home/emartenson/qubes-builder'
Makefile.generic:147: recipe for target 'packages' failed
make[1]: *** [packages] Error 1
make[1]: Leaving directory '/home/emartenson/qubes-builder'
Makefile:221: recipe for target 'artwork-dom0' failed
make: *** [artwork-dom0] Error 1

[Unman]

How did you get your builder.conf file? The build order is important
and it looks as if you are trying to build artwork before you have
built qubes-utils.

You can either comment out artwork for the moment, build the rest and
then build artwork or rejig the order in builder.conf, and try the build
again.

unman

[Elias Mårtenson]

On 23 Mar 2017 00:00, "Unman" <un...@thirdeyesecurity.org> wrote:

How did you get your builder.conf file? The build order is important
and it looks as if you are trying to build artwork before you have
built qubes-utils.

You can either comment out artwork for the moment, build the rest and
then build artwork or rejig the order in builder.conf, and try the build
again.

I got the builder.conf file by copying it from examples/, as was explained in the documentation that I linked to. 

All I really did after setting up the gpg keys was to run make get-sources, followed by make qubes. 

If things were built in the wrong order, there must be a problem with the makefile, since it was all run as part of make qubes from a clean source directory. 

Or, I have missed something obvious, which wouldn't be too surprising. 

Regards, 

Elias 

[Unman]

Strange because the builder.conf there has linux-utils as entry before
artwork.
Did you succesfully build linux-utils?

[Elias Mårtenson]

On 23 Mar 2017 00:19, "Unman" <un...@thirdeyesecurity.org> wrote:

Strange because the builder.conf there has linux-utils as entry before
artwork.
Did you succesfully build linux-utils? 

I'm not sure. I made the mistake of not capturing output to a file. Is there an easy way to check? 

Regards, 

Elias 

[Unman]

There should be build logs in the build-logs directory :-)

[Opal Raava]

There is a version issue here. The current qubes repo produces a qubes-utils-3.2.3 rpm (qubes-utils is built by the component 'linux-utils' which is indeed built before artwork gets built.
But somehow the artwork .spec file requires qubes-utils >= 3.9.0 (that arwork spec file is qubes-builder/qubes-src/artwork-qubes-artwork.spec)

So I changed that to >= 3.2.3 and started a fresh build. I'll post my results because I'm not sure if this fixes anything.

[Opal Raava]

Sorry typo in the artwork spec filename, it should be qubes-builder/qubes-src/artwork/qubes-artwork.spec

[Unman]

That build-requires has been the same since December 2015, so it's
surprising this hasnt come up before. Let us know how you get on - what
branch are you building?

[Opal Raava]

Ok that build did not work. I'm getting the following error:

../../bin/mkpadlock.py -s 36 -c 0xcc0000 devices/appvm-red.png
Traceback (most recent call last):
File "../../bin/mkpadlock.py", line 25, in <module>
import qubesimgconverter
ImportError: No module named qubesimgconverter
../Makefile.common:39: recipe for target 'devices/appvm-red.png' failed
make[2]: *** [devices/appvm-red.png] Error 1
make[2]: Leaving directory '/home/user/qubes-src/artwork/icons/36x36'
Makefile:4: recipe for target 'all' failed
make[1]: *** [all] Error 1

Which tells us that we probably really do need version 3.9.0 of qubes-tools.

[lok...@gmail.com]

On Thursday, 23 March 2017 02:45:53 UTC+8, Opal Raava wrote:

> Ok that build did not work. I'm getting the following error:
>
> ../../bin/mkpadlock.py -s 36 -c 0xcc0000 devices/appvm-red.png
> Traceback (most recent call last):
> File "../../bin/mkpadlock.py", line 25, in <module>
> import qubesimgconverter
> ImportError: No module named qubesimgconverter
> ../Makefile.common:39: recipe for target 'devices/appvm-red.png' failed
> make[2]: *** [devices/appvm-red.png] Error 1
> make[2]: Leaving directory '/home/user/qubes-src/artwork/icons/36x36'
> Makefile:4: recipe for target 'all' failed
> make[1]: *** [all] Error 1
>
> Which tells us that we probably really do need version 3.9.0 of qubes-tools.

Where can that version be found? Looking at the latest master of qubes-linux-utils, it only contains version 3.2.3:

https://github.com/QubesOS/qubes-linux-utils/blob/master/version

Regards,
Elias

[Opal Raava]

Yes, and that is what surprises me as well. The relevant source has not been touched in a while. I've looked, but I don't know why the build breaks.

All I know for a fact is what you said, it asks for 3.9.0 and delivers 3.2.3 at the master branch. And that the relevant source files all have not been changed recently.

[lok...@gmail.com]

On Thursday, 23 March 2017 18:23:30 UTC+8, Opal Raava wrote:
> On Thursday, March 23, 2017 at 3:27:58 AM UTC+1, lok...@gmail.com wrote:
> > On Thursday, 23 March 2017 02:45:53 UTC+8, Opal Raava wrote:
> >
> > > Which tells us that we probably really do need version 3.9.0 of qubes-tools.
> >
> > Where can that version be found? Looking at the latest master of qubes-linux-utils, it only contains version 3.2.3:
> >
> > https://github.com/QubesOS/qubes-linux-utils/blob/master/version

> Yes, and that is what surprises me as well. The relevant source has not been touched in a while. I've looked, but I don't know why the build breaks.

>
> All I know for a fact is what you said, it asks for 3.9.0 and delivers 3.2.3 at the master branch. And that the relevant source files all have not been changed recently.

The spec file was updated from 2.0.10 to 3.9.0 as of commit 9780a95a0ad20b63f2da401f108f2dcb3511554b. The interesting thing is that the commit was on the 17'th December 2015. This is more than half a year before the most recent ISO images were built.

I really have no idea what the cause for this could be.

Regards,
Elias

[Unman]

Finally I understand - you're trying to build a full build of master -
I just dont think this works at the moment - you could *try* following
my suggestion of NOT building artwork at all - just comment it out.

Lots of the devel work on v4 (that's now in master) is in the devs repos
- you can see the mysterious 3.9 here:
https://github.com/woju/qubes-linux-utils/tree/core3-devel

So I think you have to cherry pick from a number of different repos to
get something like a working v4 if that's what you want.
At least from marmarek and woju.
I dont have scope to try this.

(You can just build some components as you like - e.g just the manager.)

It IS possible to build templates from master, and they seem to work
fine with 3.2 :although marek has said that things might break, I haven't
seen this as yet.)

unman

[Opal Raava]

On Thursday, March 23, 2017 at 12:11:51 PM UTC+1, Unman wrote:

Ah okay, so all we have to do is specify 'releng3.2' into github or something to build the R3.2 branch?

[Opal Raava]

Ah, problem solved. In the web build documentation it says to:

cp example-configs/qubes-os-master.conf builder.conf

if you instead do:

cp example-configs/qubes-os-r3.2.conf

you get a better branch. I'm currently trying to build that

[Opal Raava]

I have now a new shiny ISO, so this works. Isn't it a bug in the website documentation that they recommend building from the master branch?

[lok...@gmail.com]

On Friday, 24 March 2017 19:34:47 UTC+8, Opal Raava wrote:

> > Ah, problem solved. In the web build documentation it says to:
> >
> > cp example-configs/qubes-os-master.conf builder.conf
> >
> > if you instead do:
> >
> > cp example-configs/qubes-os-r3.2.conf
> >
> > you get a better branch. I'm currently trying to build that
>
> I have now a new shiny ISO, so this works. Isn't it a bug in the website documentation that they recommend building from the master branch?

I'm building it myself at the moment. I'm wondering if what I'll be getting is something identical to the release image that you can download?

My goal with all of this is to build an image that contains the most recent release in the image, so that I might be able to escape the Catch-22 situation where I need the network in order to download the update so I can use the network.

[Reg Tiangha]

On 2017-03-25 8:43 AM, lok...@gmail.com
wrote:

Let us know if this works. It'd be great to build an ISO on-demand that
could pull in the latest kernel in order to support installations on
newer hardware. Although now I'm curious if this could also be used to
re-base R3.2 dom0 on say FC24 or FC25 or anything else that already has
Qubes template support. I get that I'd then be responsible for compiling
Qubes-specific dom0 updates on my own and manually installing them
whenever they become available, but it'd be nice to see if an updated X
stack as well would help with some graphical glitches I've noticed pop
up every once in a while (nothing a reboot doesn't solve, though).

[Opal Raava]

The build procedure does not give an identical iso:

$ sha256sum ../Qubes-R3.2-x86_64.iso Qubes-DVD-x86_64-20170324.iso
6b998045a513dcdd45c1c6e61ace4f1b4e7eff799f381dccb9eb0170c80f678a ../Qubes-R3.2-x86_64.iso
b51b6aa9e468eacf279b6932d4a1b615495ae38fe458b92143be98ddd57c84ac Qubes-DVD-x86_64-20170324.iso

[Reg Tiangha]

It probably wouldn't because when generating the iso, it'll pull in the
latest versions of the packages available (I believe; correct me if I'm
wrong) rather than the ones that were shipped on the original iso. So
the included kernel will be the latest 4.4 version available (4.4.55
rather than 4.4.14), all of the FC23 packages will be their final
versions (since FC23 is now EOL), the various templates will include the
latest versions of their own packages as well, etc.

I rather like this approach since you can spin isos on demand to have
the newest packages (assuming it works; it failed for me yesterday
trying to symlink to a non-existent Qubes package, but I didn't have
time to troubleshoot it so I gave up), but I get the security concern as
well. I suppose it'd be nice to have a "create a CD based on the one
available on the website" option, but I think it would require
hardcoding all the package versions to freeze them at what they were at
release time or something like that? I'm not sure if that's worth it.

[Unman]

Yes, you're quite right: you can see this by running 'make
switch-branch', and looking at the output. You will see that many of the
Qubes tools are from master.

The build process itself is designed to check and validate the source,
so this mitigates the security concern you speak of.

[Andrew David Wong]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA512

They won't be identical until we achieve reproducible builds:

https://github.com/QubesOS/qubes-issues/issues/816

- --
Andrew David Wong (Axon)
Community Manager, Qubes OS
https://www.qubes-os.org
-----BEGIN PGP SIGNATURE-----

iQIcBAEBCgAGBQJY2OdBAAoJENtN07w5UDAw1cEQAKuysm2aU/IDv3IdpK05CBzh
+GTwKPAjlXzaiE9cCJxvjPBC9ONjIEqVcpL7B2jXfsay1zx7281hFpKKBEyAGC90
gb+//aJSQ97TdeTC/D1pOo6IS5pSzxJylU+GIBNV29bqvwdb3FmljznBHAANt6HM
lN8pJIHl1TNcNbre7rJIMdpHCaxXGoV/z7UCMxp9dy4m/eU5+7KBWTQ9QajhEs5d
DAwtB8OfO0Uoe7Ycmd64Ge/Cd21I/vlYJkymCPZ6YCfLByFTDV1ihi5nwctoPfZG
fOMRcZAE17OByIeOA+NniYk6sHDLi3dpG15vkXboq3QiTyREAOwhSLQEHpM62U1J
ZWamyZn7P7MivURL5wzI+Ohxyxa2VNzR24/3DfsZTmYqDumxAGMBoJkTDsWf8wz7
nzL3MEk8TnKYFc/ABMotySVaLxxYYywTa0CHMIg9I7gtweZ9h99XOnU+1H7xT/gO
AejXuRbB9ZbZG4+zob6u+FMc5JZga38KSLSZYi8A2jkYf7VoiwVpS+VJUwE9O5Q5
cnVHSCKI92IC0sRg7ZWP8wKrn5GzKLQySQrzDMNoJbThpidMEKFgPfc7MJ66vVj0
1WlkZkgo2dPGcrGbRfnP/ErRDhKUGeipYr+o38pwxJdyBfLRZYocvbl+ke7HFye1
bzIYXzk4qniZmAKQu21t
=WTNY
-----END PGP SIGNATURE-----

[Elias Mårtenson]

On 27 March 2017 at 18:19, Andrew David Wong <a...@qubes-os.org> wrote:

They won't be identical until we achieve reproducible builds:

https://github.com/QubesOS/qubes-issues/issues/816

The builds are significantly different. The main difference that I noted was that while the image I can download from the Qubes site is approximately 4 GB in size, the one I built from source is 2.3 GB.

Also, using the home-built image, the graphics is incredibly slow, taking several seconds to repaint the screen on the laptop on which I am doing my tests. The official image doesn't have that issue. These two facts alone suggests to me that there are some major components missing from the image that I built.

Finally, I have to report that I can't get the network running on the Dell Latitude 7480 even with a home-built image. Thus, I have to conclude that until a new release (3.2.1 or 4.0), this machine can't be used with Qubes. This is sad, since hardware-wise it'd be a perfect machine for the purpose.

Regards,

Elias