qubes-builder issues
178 views
Skip to first unread message
Jon
Mar 7, 2016, 10:28:33 PM3/7/16
to qubes-devel
Hi,
I have a few issues using qubes-builder, I'm wondering if I'm alone or not:
1) After fetching sources and building deps, making gui-daemon without having built anything else fails.
Error: No Package found for qubes-core-libs
Error: No Package found for qubes-core-libs-devel >= 1.6.1
Error: No Package found for qubes-gui-common-devel
Error: No Package found for qubes-libvchan-xen-devel
Makefile.fedora:91: recipe for target 'dist-build-dep' failed
2) To work around this I build everything using 'make qubes' instead. But python-yaml is needed for 'make qubes' otherwise mgmt-salt fails to build. install-deps doesn't include it and the web site docs don't mention it.
3) If you don't pass VERBOSE=1 to on the make command line (or put it in builder.conf), then things like 'make build-info' won't output anything. That's a bit confusing.
4) Attempting to build in parallel with -j breaks rather badly even after fixing submake calls, I'm guessing this isn't supported?
5) If you have a build error, fix it and re-run the top level make command, at least some of the targets will be rebuilt even though they haven't changed. That must be a bug?
6) dnf actions seem to take forever while building (a known issue with a patch that Fedora hasn't taken yet).
I feel like I must be doing something wrong, it can't be this painful to build for everyone. Am I missing a better way to build and test components?
Cheers
Jon
I have a few issues using qubes-builder, I'm wondering if I'm alone or not:
1) After fetching sources and building deps, making gui-daemon without having built anything else fails.
Error: No Package found for qubes-core-libs
Error: No Package found for qubes-core-libs-devel >= 1.6.1
Error: No Package found for qubes-gui-common-devel
Error: No Package found for qubes-libvchan-xen-devel
Makefile.fedora:91: recipe for target 'dist-build-dep' failed
2) To work around this I build everything using 'make qubes' instead. But python-yaml is needed for 'make qubes' otherwise mgmt-salt fails to build. install-deps doesn't include it and the web site docs don't mention it.
3) If you don't pass VERBOSE=1 to on the make command line (or put it in builder.conf), then things like 'make build-info' won't output anything. That's a bit confusing.
4) Attempting to build in parallel with -j breaks rather badly even after fixing submake calls, I'm guessing this isn't supported?
5) If you have a build error, fix it and re-run the top level make command, at least some of the targets will be rebuilt even though they haven't changed. That must be a bug?
6) dnf actions seem to take forever while building (a known issue with a patch that Fedora hasn't taken yet).
I feel like I must be doing something wrong, it can't be this painful to build for everyone. Am I missing a better way to build and test components?
Cheers
Jon
Marek Marczykowski-Górecki
Mar 8, 2016, 5:46:09 AM3/8/16
to Jon, qubes-devel
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256
On Mon, Mar 07, 2016 at 07:28:33PM -0800, Jon wrote:
> Hi,
>
> I have a few issues using qubes-builder, I'm wondering if I'm alone or not:
>
> 1) After fetching sources and building deps, making gui-daemon without
> having built anything else fails.
>
> Error: No Package found for qubes-core-libs
> Error: No Package found for qubes-core-libs-devel >= 1.6.1
> Error: No Package found for qubes-gui-common-devel
> Error: No Package found for qubes-libvchan-xen-devel
> Makefile.fedora:91: recipe for target 'dist-build-dep' failed
Yes, gui-daemon needs some other components as build dependency.
Theoretically it is possible to download them from yum.qubes-os.org, but
it isn't enabled by default and I haven't tried this for a while.
We should document how to do that...
https://github.com/QubesOS/qubes-issues/issues/1820
> 2) To work around this I build everything using 'make qubes' instead. But
> python-yaml is needed for 'make qubes' otherwise mgmt-salt fails to build.
> install-deps doesn't include it and the web site docs don't mention it.
https://github.com/QubesOS/qubes-issues/issues/1822
> 3) If you don't pass VERBOSE=1 to on the make command line (or put it in
> builder.conf), then things like 'make build-info' won't output anything.
> That's a bit confusing.
'make build-info' is called before 'make qubes', so the idea was to not
output that big table with VERBOSE=0. But indeed executing 'make
build-info' directly should show the output always.
https://github.com/QubesOS/qubes-issues/issues/1821
> 4) Attempting to build in parallel with -j breaks rather badly even after
> fixing submake calls, I'm guessing this isn't supported?
Indeed, it isn't. But building some components internally is using -j,
where it makes sense (libvirt, xen, etc).
Probably we should somehow block it at builder level, or at least add
some warning...
> 5) If you have a build error, fix it and re-run the top level make command,
> at least some of the targets will be rebuilt even though they haven't
> changed. That must be a bug?
I'd call it a missing feature. It's really hard to guess what should be
rebuild at qubes-builder abstraction layer. Each component can be built
for multiple target distributions. There are several files used only on
some of them (no sense to rebuild rpm package when something is changed
only in debian/ subdir). You may need to rebuild some component just to
link with some newer version of library, even when nothing have changed
in the component itself. Also builder itself have no direct knowledge
what output files (packages) will be produced for given distribution
(sometimes it is tricky - for example you can have spec file with rpm
package definition, but the package will be built only for some target
architectures and x86_64 isn't one of them).
Because all of this, we've decided to go a simple way: when you call
make gui-daemon, it rebuilds gui-daemon, unconditionally.
If you want to resume after fixing some problem, you can call `make ...
list of components to build ...`. You can get full list easily from
`make help` (at the end) - just copy&paste starting with appropriate place.
> 6) dnf actions seem to take forever while building (a known issue with a
> patch that Fedora hasn't taken yet).
>
>
> I feel like I must be doing something wrong, it can't be this painful to
> build for everyone. Am I missing a better way to build and test components?
I'm afraid building only a single component is really that painful...
When you build the whole template, or even full ISO, preparing
configuration with `./setup` script it is slightly easier.
After initially building all the components, you can easily build any of
them separately.
- --
Best Regards,
Marek Marczykowski-Górecki
Invisible Things Lab
A: Because it messes up the order in which people normally read text.
Q: Why is top-posting such a bad thing?
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2
iQEcBAEBCAAGBQJW3q1nAAoJENuP0xzK19csMksIAJrtZ1vlkfAzl8/2GbKNOQbc
buVa+8PD5ueTsclgHXMYGA8Uk2mWiPc+kqGJY/i7YrROlRY9R7roqMtgs6eA78OW
Gz+sGKO+lixSabnGi0ALScjjAnudxe9aka8Z3WpjNdmgcEznKzb9oYikjgc3DLMZ
NCqGjomcUBlPlDodSYdS+wTdV/yNTXWG1REv3aNATedXIWqF+xhnv6OQu6Y28nNo
1UdthrIitqbdYrJ3MeXnzQWE1WiI6+FPAr6tC18Sv094hwfqgVEE1H6N2MzZ+kS/
E4AIAZYa5xBvhDMX7tvp3lshHd8njb+hD8ABYuvMuKkEsllvZ1QBoAGi238Tzus=
=0r9s
-----END PGP SIGNATURE-----
Hash: SHA256
On Mon, Mar 07, 2016 at 07:28:33PM -0800, Jon wrote:
> Hi,
>
> I have a few issues using qubes-builder, I'm wondering if I'm alone or not:
>
> 1) After fetching sources and building deps, making gui-daemon without
> having built anything else fails.
>
> Error: No Package found for qubes-core-libs
> Error: No Package found for qubes-core-libs-devel >= 1.6.1
> Error: No Package found for qubes-gui-common-devel
> Error: No Package found for qubes-libvchan-xen-devel
> Makefile.fedora:91: recipe for target 'dist-build-dep' failed
Theoretically it is possible to download them from yum.qubes-os.org, but
it isn't enabled by default and I haven't tried this for a while.
We should document how to do that...
https://github.com/QubesOS/qubes-issues/issues/1820
> 2) To work around this I build everything using 'make qubes' instead. But
> python-yaml is needed for 'make qubes' otherwise mgmt-salt fails to build.
> install-deps doesn't include it and the web site docs don't mention it.
> 3) If you don't pass VERBOSE=1 to on the make command line (or put it in
> builder.conf), then things like 'make build-info' won't output anything.
> That's a bit confusing.
output that big table with VERBOSE=0. But indeed executing 'make
build-info' directly should show the output always.
https://github.com/QubesOS/qubes-issues/issues/1821
> 4) Attempting to build in parallel with -j breaks rather badly even after
> fixing submake calls, I'm guessing this isn't supported?
where it makes sense (libvirt, xen, etc).
Probably we should somehow block it at builder level, or at least add
some warning...
> 5) If you have a build error, fix it and re-run the top level make command,
> at least some of the targets will be rebuilt even though they haven't
> changed. That must be a bug?
rebuild at qubes-builder abstraction layer. Each component can be built
for multiple target distributions. There are several files used only on
some of them (no sense to rebuild rpm package when something is changed
only in debian/ subdir). You may need to rebuild some component just to
link with some newer version of library, even when nothing have changed
in the component itself. Also builder itself have no direct knowledge
what output files (packages) will be produced for given distribution
(sometimes it is tricky - for example you can have spec file with rpm
package definition, but the package will be built only for some target
architectures and x86_64 isn't one of them).
Because all of this, we've decided to go a simple way: when you call
make gui-daemon, it rebuilds gui-daemon, unconditionally.
If you want to resume after fixing some problem, you can call `make ...
list of components to build ...`. You can get full list easily from
`make help` (at the end) - just copy&paste starting with appropriate place.
> 6) dnf actions seem to take forever while building (a known issue with a
> patch that Fedora hasn't taken yet).
>
>
> I feel like I must be doing something wrong, it can't be this painful to
> build for everyone. Am I missing a better way to build and test components?
When you build the whole template, or even full ISO, preparing
configuration with `./setup` script it is slightly easier.
After initially building all the components, you can easily build any of
them separately.
- --
Best Regards,
Marek Marczykowski-Górecki
Invisible Things Lab
A: Because it messes up the order in which people normally read text.
Q: Why is top-posting such a bad thing?
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2
iQEcBAEBCAAGBQJW3q1nAAoJENuP0xzK19csMksIAJrtZ1vlkfAzl8/2GbKNOQbc
buVa+8PD5ueTsclgHXMYGA8Uk2mWiPc+kqGJY/i7YrROlRY9R7roqMtgs6eA78OW
Gz+sGKO+lixSabnGi0ALScjjAnudxe9aka8Z3WpjNdmgcEznKzb9oYikjgc3DLMZ
NCqGjomcUBlPlDodSYdS+wTdV/yNTXWG1REv3aNATedXIWqF+xhnv6OQu6Y28nNo
1UdthrIitqbdYrJ3MeXnzQWE1WiI6+FPAr6tC18Sv094hwfqgVEE1H6N2MzZ+kS/
E4AIAZYa5xBvhDMX7tvp3lshHd8njb+hD8ABYuvMuKkEsllvZ1QBoAGi238Tzus=
=0r9s
-----END PGP SIGNATURE-----
Jon
Mar 8, 2016, 6:19:03 AM3/8/16
to qubes-devel, linu...@gmail.com
On Tuesday, March 8, 2016 at 11:46:09 PM UTC+13, Marek Marczykowski-Górecki wrote:
> <snipped>
Thanks for raising those issues.
> <snipped>
Thanks for raising those issues.
>Probably we should somehow block it at builder level, or at least add
>some warning...
As I get time I will submit some changes that may make this a little clearer, thanks.
> It's really hard to guess what should be rebuild at qubes-builder abstraction layer.
> Because all of this, we've decided to go a simple way: when you call
> make gui-daemon, it rebuilds gui-daemon, unconditionally.
> make gui-daemon, it rebuilds gui-daemon, unconditionally.
Yeah expressing the exact dependencies is difficult in most cases, and as you noted its easy to miss dependencies or include false ones.
> If you want to resume after fixing some problem, you can call `make ...
> list of components to build ...`. You can get full list easily from
> `make help` (at the end) - just copy&paste starting with appropriate place.
Yes - I realised after investigating further that the components are listed in the makefile in dependency order. So in theory I can pass the list up to the component I want to build if building from scratch, or from the failed component onwards for subsequent builds, to avoid rebuilding stuff. I'm used to the ordering of dependencies being worked out by make implicitly which isn't the case at the top level.
> I'm afraid building only a single component is really that painful...
> After initially building all the components, you can easily build any of
them separately.
them separately.
I have a fully built tree now (for fc20/fc23 at least). As you say, after running through once, building a single component becomes pretty painless. I guess most devs are mostly working with a populated build tree!
Thanks for your comments,
Cheers
Jon
Thanks for your comments,
Cheers
Jon
Chris Laprise
Apr 22, 2016, 11:13:26 PM4/22/16
to Marek Marczykowski-Górecki, Jon, qubes-devel
I'm trying to follow the builder instructions also, but getting blocked
by this error:
$ make qubes
ERROR: call 'make install-deps' to install missing dependencies
Makefile:189: recipe for target 'check-depend.0' failed
make: *** [check-depend.0] Error 1
Install-deps installs python-yaml, but that doesn't resolve the problem.
Chris
by this error:
$ make qubes
ERROR: call 'make install-deps' to install missing dependencies
Makefile:189: recipe for target 'check-depend.0' failed
make: *** [check-depend.0] Error 1
Install-deps installs python-yaml, but that doesn't resolve the problem.
Chris
Marek Marczykowski-Górecki
Apr 23, 2016, 4:33:28 PM4/23/16
to Chris Laprise, Jon, qubes-devel
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256
Hash: SHA256
package is missing.
- --
Best Regards,
Marek Marczykowski-Górecki
Invisible Things Lab
A: Because it messes up the order in which people normally read text.
Q: Why is top-posting such a bad thing?
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2
HFEfOMuh6/i33H7s+FAJExKCgx/Rg21WLK71esWxN0wNY2yUZilJ4WtWPFJ554dW
kC4blCXTcyDOc79kEO/+rrlUsGDnzhkf6jJhcUGl5b+YKXI7GMBNo1ZhkBJI3Opq
3JWclcBE+P6wrPUwdy+GfaDY5EXFLvUxcK0DHtRlsR+MphSQqh3Qn2vi8qqG9Vn7
7vQ8f+PKrbkYsvDCNPgSY4pF599Oa4PCkv0EMyw1yTjHkxDRaZWVJ5r9Kr78VS2+
dZBPDgKu+XklJU+I4qZA/KzOUf+5RC1ARhyRcBccsvyP/6xKl5lb+4d/s7xES8Q=
=TMEh
-----END PGP SIGNATURE-----
Chris Laprise
Apr 24, 2016, 6:41:31 PM4/24/16
to Marek Marczykowski-Górecki, Jon, qubes-devel
On 04/23/2016 04:33 PM, Marek Marczykowski-Górecki wrote:
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA256
>
> On Fri, Apr 22, 2016 at 11:13:20PM -0400, Chris Laprise wrote:
>> I'm trying to follow the builder instructions also, but getting blocked by
>> this error:
>>
>> $ make qubes
>> ERROR: call 'make install-deps' to install missing dependencies
>> Makefile:189: recipe for target 'check-depend.0' failed
>> make: *** [check-depend.0] Error 1
>>
>>
>> Install-deps installs python-yaml, but that doesn't resolve the problem.
> Increase verbosity (VERBOSE=2 in builder.conf). Then you'll see what
> package is missing.
>
$ make qubes
================================================================================
B U I L D I N F O
Items in red indicate it was automatically generated by configuration
file(s)
Items in white indicate it was automatically removed by configuration
file(s)
================================================================================
DISTS_VM:
fc23,
DISTS_ALL:
fc20, fc23,
DIST_DOM0:
fc20,
BUILDER_PLUGINS:
builder-fedora, mgmt-salt,
COMPONENTS:
vmm-xen, core-libvirt, core-vchan-xen, core-qubesdb,
linux-utils, core-admin, core-admin-linux, core-agent-linux,
linux-kernel, artwork, gui-common, gui-daemon,
gui-agent-linux, gui-agent-xen-hvm-stubdom, app-linux-split-gpg,
app-linux-tor, app-thunderbird, app-linux-pdf-converter,
app-linux-img-converter, app-linux-input-proxy, mgmt-salt,
mgmt-salt-base, mgmt-salt-base-topd, mgmt-salt-base-config,
mgmt-salt-base-overrides, mgmt-salt-dom0-qvm,
mgmt-salt-dom0-virtual-machines, mgmt-salt-dom0-update,
linux-template-builder, desktop-linux-kde, desktop-linux-xfce4,
manager, linux-dom0-updates, linux-pvgrub2,
installer-qubes-os, linux-yum, vmm-xen-windows-pvdrivers,
antievilmaid, builder, builder-fedora,
GIT_REPOS:
qubes-src/vmm-xen, qubes-src/core-libvirt,
qubes-src/core-vchan-xen, qubes-src/core-qubesdb,
qubes-src/linux-utils, qubes-src/core-admin,
qubes-src/core-admin-linux,
qubes-src/core-agent-linux, qubes-src/linux-kernel, qubes-src/artwork,
qubes-src/gui-common, qubes-src/gui-daemon, qubes-src/gui-agent-linux,
qubes-src/gui-agent-xen-hvm-stubdom, qubes-src/app-linux-split-gpg,
qubes-src/app-linux-tor, qubes-src/app-thunderbird,
qubes-src/app-linux-pdf-converter, qubes-src/app-linux-img-converter,
qubes-src/app-linux-input-proxy, qubes-src/mgmt-salt,
qubes-src/mgmt-salt-base, qubes-src/mgmt-salt-base-topd,
qubes-src/mgmt-salt-base-config, qubes-src/mgmt-salt-base-overrides,
qubes-src/mgmt-salt-dom0-qvm,
qubes-src/mgmt-salt-dom0-virtual-machines,
qubes-src/mgmt-salt-dom0-update, qubes-src/linux-template-builder,
qubes-src/desktop-linux-kde, qubes-src/desktop-linux-xfce4,
qubes-src/manager, qubes-src/linux-dom0-updates,
qubes-src/linux-pvgrub2, qubes-src/installer-qubes-os,
qubes-src/linux-yum, qubes-src/vmm-xen-windows-pvdrivers,
qubes-src/antievilmaid, qubes-src/builder-fedora, .,
TEMPLATE:
TEMPLATE_FLAVOR_DIR:
TEMPLATE_ALIAS:
TEMPLATE_LABEL:
fc23:fedora-23, :, fc20:fedora-20 fc21:fedora-21
fc22:fedora-22
Currently installed dependencies:
git-2.5.5-1.fc23.x86_64
rpmdevtools-8.6-2.fc23.noarch
rpm-build-4.13.0-0.rc1.12.fc23.x86_64
createrepo-0.10.3-3.fc21.noarch
python-sh-1.11-1.fc23.noarch
package python-yaml is not installed
Makefile:191: recipe for target 'check-depend.2' failed
make: *** [check-depend.2] Error 1
$ sudo yum install python-yaml
Yum command has been deprecated, redirecting to '/usr/bin/dnf install
python-yaml'.
See 'man dnf' and 'man yum2dnf' for more information.
To transfer transaction metadata from yum to DNF, run:
'dnf install python-dnf-plugins-extras-migrate && dnf-2 migrate'
Last metadata expiration check: 0:02:31 ago on Sun Apr 24 18:33:45 2016.
Package PyYAML-3.11-11.fc23.x86_64 is already installed, skipping.
Dependencies resolved.
Nothing to do.
Sending application list and icons to dom0
Complete!
Chris Laprise
Apr 25, 2016, 6:46:30 PM4/25/16
to Marek Marczykowski-Górecki, Jon, qubes-devel
got the same result.
I think the problem may stem from the fact that yum/dnf references
'python-yaml' when installing, but then the package name becomes 'PyYAML'.
Chris
Marek Marczykowski-Górecki
Apr 25, 2016, 6:54:52 PM4/25/16
to Chris Laprise, Jon, qubes-devel
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256
Hash: SHA256
get-sources).
- --
Best Regards,
Marek Marczykowski-Górecki
Invisible Things Lab
A: Because it messes up the order in which people normally read text.
Q: Why is top-posting such a bad thing?
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2
ux1bKPDEcY1JEvDdtLcWaSRtg7/BJmWN/Mygtrq9yewVWynzBfZSUxqwkJEJ6avO
yuGSXj5xygq40P5TXec/jFO2CrIiPHPaQiImVXWeP68ulY2QW+FHm/YPjvncaLgL
ncn14fjRLFawpwCVuYQLDKTyeT3djl9QnysxIq9XzqJ5v4K9RKXFnVLjFMBKm80X
EvDQs+Lso9bgf1qF0GURXbRNHW4AeoSeGtex5lwD/MikN9u6yGKNjly8k2jv/3ve
katEMpkwfBuqA4HChi/mjdj2U3jvq5WiCXiHk9uGkJAHww208Nic7CHBl800/qg=
=7pqn
-----END PGP SIGNATURE-----
Chris Laprise
Apr 25, 2016, 7:09:50 PM4/25/16
to Marek Marczykowski-Górecki, Jon, qubes-devel
Problem still exists. Should I be working from a different branch than
the default?
Chris
the default?
Chris
Reply all
Reply to author
Forward
0 new messages