Makefile pkgs-for-template question

[Patrick Schleizer]

Hi!

I made various changes [1] to qubes-template-whonix. My goal is to leave
building and installation of the qubes-whonix package to Whonix's build
script (Whonix 12) as opposed to Qubes Builder (Whonix 11). This helps
to greatly simplify qubes-template-whonix, because it needs to build
fewer components.

(The qubes-whonix package is supposed to pull dependencies such as
qubes-core-agent from Qubes' local APT repository.)

Currently I am running installQubesRepo from 02_install_groups_pre.sh
[2] before Whonix's build script gets executed by Qubes Builder with the
intent to have Qubes' local APT repository available to Whonix's build
script within the chroot.

During running of apt-get with the chroot, there is an issue.

W: Failed to fetch
file:/tmp/qubes_repo/dists/jessie/main/binary-amd64/Packages File not found

Inside the chroot, folder /tmp/qubes_repo/ exists, but is empty. Folder
/home/user/qubes-builder/qubes-src/linux-template-builder/pkgs-for-template/
is still empty.

So the Makefile build step that creates pkgs-for-template has not run at
that stage.

Can I have pkgs-for-template available at time of
02_install_groups_pre.sh's run or do you have other suggestions on how
to solve this?

Cheers,
Patrick

[1] https://github.com/adrelanos/qubes-template-whonix/commits/master
[2]
https://github.com/adrelanos/qubes-template-whonix/blob/master/whonix-gateway/02_install_groups_pre.sh

[nrgaway]

On 12 August 2015 at 23:04, Patrick Schleizer <patrick-ma...@whonix.org> wrote:

Hi!

I made various changes [1] to qubes-template-whonix. My goal is to leave
building and installation of the qubes-whonix package to Whonix's build
script (Whonix 12) as opposed to Qubes Builder (Whonix 11). This helps
to greatly simplify qubes-template-whonix, because it needs to build
fewer components.

(The qubes-whonix package is supposed to pull dependencies such as
qubes-core-agent from Qubes' local APT repository.)

Currently I am running installQubesRepo from 02_install_groups_pre.sh
[2] before Whonix's build script gets executed by Qubes Builder with the
intent to have Qubes' local APT repository available to Whonix's build
script within the chroot.

During running of apt-get with the chroot, there is an issue.

W: Failed to fetch
file:/tmp/qubes_repo/dists/jessie/main/binary-amd64/Packages  File not found

Inside the chroot, folder /tmp/qubes_repo/ exists, but is empty. Folder
/home/user/qubes-builder/qubes-src/linux-template-builder/pkgs-for-template/
is still empty.

So the Makefile build step that creates pkgs-for-template has not run at
that stage.

When you run `make template` the packages are copied over to the `linux-template-builder` in the `pkgs-for-template` directory.  04_install_qubes.sh has code to mount the 'repo', which calls a function in `distribution.sh`:

installQubesRepo
aptUpdate
buildStep "${0}" "${DIST}"
installPackages packages_qubes.list
uninstallQubesRepo

`template-whonix/whonix-gateway/04_install_qubes_post.sh` also has code to access repo:

installQubesRepo
aptInstall qubes-whonix
uninstallQubesRepo

Note, that Qubes packages are purposely installed in the 04* step and not 02* step.  02 makes sure complete system is set up and 04 installs Qubes.  This allows one to be able to rebuild Qubes packages and create a new image and not having to re-build the base system again.  This is key for development purposes as it saves a ton of build time.

You will also need to make sure it can be built within a DispVM :)

Can I have pkgs-for-template available at time of
02_install_groups_pre.sh's run or do you have other suggestions on how
to solve this?

Add this to `02_install_groups_pre.sh`:

installQubesRepo
aptInstall <package name(s)> --or-- installPackages <packages_name.list>
uninstallQubesRepo

[Patrick Schleizer]

nrgaway:

I am at that point.

When I build the jessie template first, then folder
/home/user/qubes-builder/qubes-src/linux-template-builder/pkgs-for-template/
will be populated. When building the Whonix template after that,
installQubesRepo works and Qubes' local APT repository is available to
Whonix's build script.

But... When I 'make mostlyclean' (or something lesser) and then only
build the Whonix template, then folder
/home/user/qubes-builder/qubes-src/linux-template-builder/pkgs-for-template/
will not be populated at time of 02_install_groups_pre.sh. The build
will fail.

My question remains, how can I have
/home/user/qubes-builder/qubes-src/linux-template-builder/pkgs-for-template/
populated at time of 02_install_groups_pre.sh?

The same question in other words...

> Add this to `02_install_groups_pre.sh`:
> installQubesRepo

> aptInstall qubes-core-agent
> uninstallQubesRepo

That wouldn't work -
/home/user/qubes-builder/qubes-src/linux-template-builder/pkgs-for-template/
is not populated with packages at time of 02_install_groups_pre.sh.
installQubesRepo doesn't create the repository. It also does not check
if
/home/user/qubes-builder/qubes-src/linux-template-builder/pkgs-for-template/
is actually populated.

Chers,
Patrick

[nrgaway]

It should be.  How are you making the template.  You mentioned doing a make mostly clean in qubes-builder I assume, since that's the only place I implemented it.  That will remove built packages and only keeps the actual downloaded repos compared to clean-all which removes qubes-src directory too.

You would need to:

make qubes-vm

make template

The files get bound over during the `make template` update-repo target stage; before any of the template scripts are run and would be available, if the packages are actually built, at any stage of the script build process.

I used to install qubes-whonix in 02...pre, but removed it had depend on qubes and therefore could not be installed at that point. Guess you will also need to remove that depend.

[Marek Marczykowski-Górecki]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256

Exactly - 02* steps shouldn't access any locally build packages
(pkgs-for-template directory) - only packages of upstream distribution.

This is to ensure that when you rebuild the template, it will surely use
packages just built, even if you've just rebuild then without bumping
the version. This is critical for development builds, otherwise it will
require several hours to test every single typo fixed.

Basically steps are:
00* - some preparation, no target image exists at this point
01* - install base system - basically to be able to execute package
manager by chrooting into that system
02* - install the distribution (according to selected template flavor),
using external repositories only
(virtual) 03* - image prepared this way is copied here
04* - add locally built packages - in plain Qubes template this means
"add locally built qubes packages", but in qubes-whonix template this also means "add
locally built whonix packages" (maybe "04_install_qubes.sh" isn't the
best name now)
09* - cleanup - remove temporary files, reduce image size etc

This workflow ensures that every time you build the template, you're
sure that the right (local) packages are installed, instead of testing
some stale version.
So I think the Whonix packages installation should be in 04*, not in
02*.

> > You will also need to make sure it can be built within a DispVM :)
> >
> >
> >> Can I have pkgs-for-template available at time of
> >> 02_install_groups_pre.sh's run or do you have other suggestions on how
> >> to solve this?
> >>
> >
> > Add this to `02_install_groups_pre.sh`:
> > installQubesRepo
> > aptInstall <package name(s)> --or-- installPackages <packages_name.list>
> > uninstallQubesRepo
> >
>
> I am at that point.
>
> When I build the jessie template first, then folder
> /home/user/qubes-builder/qubes-src/linux-template-builder/pkgs-for-template/
> will be populated. When building the Whonix template after that,
> installQubesRepo works and Qubes' local APT repository is available to
> Whonix's build script.
>
> But... When I 'make mostlyclean' (or something lesser) and then only
> build the Whonix template, then folder
> /home/user/qubes-builder/qubes-src/linux-template-builder/pkgs-for-template/
> will not be populated at time of 02_install_groups_pre.sh. The build
> will fail.
>
> My question remains, how can I have
> /home/user/qubes-builder/qubes-src/linux-template-builder/pkgs-for-template/
> populated at time of 02_install_groups_pre.sh?

Actually it should be there (but not bind-mounted into /tmp/qubes_repo
inside chroot)...

It is populated at the start of "make template" based on selected
`COMPONENTS` - you should get build error if any of those components
haven't been built (which is probably the case after make mostlyclean),
not empty pkgs-for-template.

> The same question in other words...
>
> > Add this to `02_install_groups_pre.sh`:
> > installQubesRepo
> > aptInstall qubes-core-agent
> > uninstallQubesRepo
>
> That wouldn't work -
> /home/user/qubes-builder/qubes-src/linux-template-builder/pkgs-for-template/
> is not populated with packages at time of 02_install_groups_pre.sh.
> installQubesRepo doesn't create the repository. It also does not check
> if
> /home/user/qubes-builder/qubes-src/linux-template-builder/pkgs-for-template/
> is actually populated.
>
> Chers,
> Patrick
>

- --
Best Regards,
Marek Marczykowski-Górecki
Invisible Things Lab
A: Because it messes up the order in which people normally read text.
Q: Why is top-posting such a bad thing?
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1

iQEcBAEBCAAGBQJV3SnrAAoJENuP0xzK19csZM8H/iyj3NMgFk6HhQQOznwXde2F
6aKvESqcpjGeQ/5VhprhtZ6AHt/MkxsI/qrBnMzj1Tm3zOtW4778Iwu2KQLHrOcE
Xbd3f6woCy4HU6TkJkZr7TwxnVkkczrLkEGhFQB3qhihfd3n4jDh+N4A6h6nvWud
QIxmMewz+7mABA2IRD793zyAtybNtxrZrQGsUM7rM4LE1G/1LUchv1HFdPPq5Xs9
oEn+p5odqIa3KZQIQ4nHrjRQkf2+v+nilg0h8gKIUZhPE8/YV6FPJTyhfPKbsZCz
9DqHbJBVsU8/6/nzfxFm2W40PMDQHW0V3137vwY8YXDK12B5M9ypM7XouQWb7BY=
=wWhr
-----END PGP SIGNATURE-----

[Patrick Schleizer]

Marek Marczykowski-Górecki:
> ********* *BEGIN ENCRYPTED or SIGNED PART* *********

Created https://phabricator.whonix.org/T402 for it.

Cheers,
Patrick