Re: [GSoC] Qubes Live USB

[Frédéric Pierret]

Hello,

On 2020-02-26 21:26, Sai Tarun Inaganti wrote:
> Dear Mr. Pierret,
>
> I am an undergraduate student who is interested in the project "Qubes Live USB". I have some experience in managing initial ramdisks. I also have some knowledge of systemd and a bit of other init systems. Can you guide me on how to set the project up and give me further details?

Do you have Qubes OS installed somewhere already? For testing it is enough to install on an external USB disk (although it will be slow). If you don't have already, this is definitely something to start with. Generally, for new development, you'll want Qubes R4.1, but for just playing with it stable (R4.0) is also fine. R4.1 installation image you can download from CI here:

- https://openqa.qubes-os.org/tests/6159#downloads

Also I'd recommend trying to build some of the packages with some minor change, to familiarize yourself with the build system:

- https://www.qubes-os.org/doc/qubes-builder/
- https://www.qubes-os.org/doc/qubes-iso-building/

and especially the source of component https://github.com/QubesOS/qubes-installer-qubes-os which will take care of generating the ISO file.

> Thank you,
> Sai Tarun Inaganti

Best regards,
Frédéric Pierret

[saitaruninaganti]

When I try to build the image, the program takes a long time at the "make qubes" step without generating any output or exiting. I am not sure if it is my computer or the program. Can you help me?

[Frédéric Pierret]

On 2020-03-11 19:03, saitaruninaganti wrote:
> When I try to build the image, the program takes a long time at the "make qubes" step without generating any output or exiting. I am not sure if it is my computer or the program. Can you help me?

You are building all Qubes dom0 + templates which takes many hours...I suggest you to add 'VERBOSE=2' and 'DEBUG=1' in your builder.conf for enabling verbosity in the console directly.

Also, take a look to "https://github.com/QubesOS/qubes-release-configs/blob/master/R4.1/qubes-os-iso-full-online.conf" this is a 'builder.conf' which is used for building only ISO with pre-built packages and templates from Qubes mirror. This is mostly a good starting point on creating live USB.

Best,

[altaway]

Thank you.

I now understand how these builder templates work slightly better. So is it possible to change the distribution dom0 is based on by simply changing DIST_DOM0 (for example, from fc31 to buster)?

Second, I'm facing an issue about not being able to locate the kickstart file which is added using the variable INSTALLER_KICKSTART. I made sure the said file is present in the given location. A quick search showed this is a common problem among redhat based distributions. But the solutions people have provided aren't complete. One more thing is that I've noticed that some of the mounted sources are not being unmounted once the program (makefile) quits on error. Could this be a reason?

On Thursday, February 27, 2020 at 2:13:47 PM UTC+5:30, Frédéric Pierret wrote:

[Frédéric Pierret]

On 2020-03-17 15:07, altaway wrote:
> Thank you.
>
> I now understand how these builder templates work slightly better. So is it possible to change the distribution dom0 is based on by simply changing DIST_DOM0 (for example, from fc31 to buster)?

No. Only Fedora 31 is supported for master branch (Fedora 25 for R4.0).

>
> Second, I'm facing an issue about not being able to locate the kickstart file which is added using the variable INSTALLER_KICKSTART. I made sure the said file

It's mostly certain it's unrelated. INSTALLER_KICKSTART is a Qubes specific variable defining the __chroot__ path of the kickstart file. Default one is located in installer-qubes-os component. Have a look to https://github.com/QubesOS/qubes-release-configs/blob/master/R4.1/qubes-os-iso-full-online.conf. The path '/home/user/qubes-src' in INSTALLER_KICKSTART refers to chroot filesystem, not local one. So for testing/modifications etc., you can modify local sources where you cloned the qubes-builder, e.g: /home/user/work-qubes/qubes-builder/qubes-src/installer-qubes-os/. Then, all of local /home/user/work-qubes/qubes-builder/qubes-src/installer-qubes-os/ will be copied into /home/user/qubes-src/installer-qubes-os/.

is present in the given location. A quick search showed this is a common problem among redhat based distributions. But the solutions people have provided aren't complete. One more thing is that I've noticed that some of the mounted sources are not being unmounted once the program (makefile) quits on error. Could this be a reason?

What do you mean not being unmounted? For each 'make' of any Qubes component, local sources are copied and replace those into chroot.

>
> On Thursday, February 27, 2020 at 2:13:47 PM UTC+5:30, Frédéric Pierret wrote:
>
> Hello,
>
> On 2020-02-26 21:26, Sai Tarun Inaganti wrote:
> > Dear Mr. Pierret,
> >
> > I am an undergraduate student who is interested in the project "Qubes Live USB". I have some experience in managing initial ramdisks. I also have some knowledge of systemd and a bit of other init systems. Can you guide me on how to set the project up and give me further details?
>
> Do you have Qubes OS installed somewhere already? For testing it is enough to install on an external USB disk (although it will be slow). If you don't have already, this is definitely something to start with. Generally, for new development, you'll want Qubes R4.1, but for just playing with it stable (R4.0) is also fine. R4.1 installation image you can download from CI here:
>

> - https://openqa.qubes-os.org/tests/6159#downloads <https://openqa.qubes-os.org/tests/6159#downloads>

>
> Also I'd recommend trying to build some of the packages with some minor change, to familiarize yourself with the build system:
>

> - https://www.qubes-os.org/doc/qubes-builder/ <https://www.qubes-os.org/doc/qubes-builder/>
> - https://www.qubes-os.org/doc/qubes-iso-building/ <https://www.qubes-os.org/doc/qubes-iso-building/>
>
> and especially the source of component https://github.com/QubesOS/qubes-installer-qubes-os <https://github.com/QubesOS/qubes-installer-qubes-os> which will take care of generating the ISO file.

>
> > Thank you,
> > Sai Tarun Inaganti
>
> Best regards,
> Frédéric Pierret
>

> --
> You received this message because you are subscribed to the Google Groups "qubes-devel" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to qubes-devel...@googlegroups.com <mailto:qubes-devel...@googlegroups.com>.
> To view this discussion on the web visit https://groups.google.com/d/msgid/qubes-devel/e605fc13-ffa7-43e1-8c3d-90425587d7f9%40googlegroups.com <https://groups.google.com/d/msgid/qubes-devel/e605fc13-ffa7-43e1-8c3d-90425587d7f9%40googlegroups.com?utm_medium=email&utm_source=footer>.

[altaway]

chroot-dom0-fc31

chroot-vm-archlinux

chroot-vm-buster

chroot-vm-centos7

chroot-vm-fc31

These chroot directories for example stayed mounted even after exiting. I'll redo the build and check it. Do I have to clean everything in case of an error or when I make a change in builder.conf?

On Thursday, February 27, 2020 at 2:13:47 PM UTC+5:30, Frédéric Pierret wrote:

[altaway]

I'm almost there. I got the mount issue resolved. At the "make iso" step, I get two errors

1. The first one's related to a dnf update within the chroot directory: 

"Error: Failed to download metadata for repo 'qubes-builder-dom0-current': Cannot download repomd.xml: Cannot download repodata/repomd.xml: All mirrors were tried"

2. The second one's related to the kickstart file:

"pykickstart.errors.KickstartParseError: The following problem occurred on line 7 of the kickstart file:

unrecognized arguments: --gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-fedora-31-primary"

But the kickstart only has six lines. Do I need to install any python module?

Putting the errors aside, I have noticed there is build support for debian as well. Also, the documentation mentions that CI uses ubuntu for final deployment. In addition to debian, I would like to add support for archlinux, which I think doesn't take too much. At the moment, I am using both fedora 31 and arch (I have a qubes install available as well for testing something). I personally think it is a lot simpler to run and resolve something on archlinux than it is on any of the deb or rpm based distributions. What do you say?

[Marek Marczykowski-Górecki]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256

On Fri, Mar 20, 2020 at 04:15:43PM -0700, altaway wrote:
> I'm almost there. I got the mount issue resolved. At the "make iso" step, I
> get two errors
>
> 1. The first one's related to a dnf update within the chroot directory:
> "Error: Failed to download metadata for repo 'qubes-builder-dom0-current':
> Cannot download repomd.xml: Cannot download repodata/repomd.xml: All
> mirrors were tried"

How your builder.conf file looks like? Especially those entries:
- DIST_DOM0
- USE_QUBES_VERISON_REPO
- RELEASE

> 2. The second one's related to the kickstart file:
> "pykickstart.errors.KickstartParseError: The following problem occurred on
> line 7 of the kickstart file:
> unrecognized arguments:
> --gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-fedora-31-primary"

This looks like you don't use our patched pykickstart package. Did you
built it? This actually may be related to the first issue - dnf can't
find the repository so it downloads package from elsewhere.
How have you dealt with "mount issue"? Those mounted directories are
needed...

> But the kickstart only has six lines. Do I need to install any python
> module?
>
> Putting the errors aside, I have noticed there is build support for debian
> as well. Also, the documentation mentions that CI uses ubuntu for final
> deployment. In addition to debian, I would like to add support for
> archlinux, which I think doesn't take too much. At the moment, I am using
> both fedora 31 and arch (I have a qubes install available as well for
> testing something). I personally think it is a lot simpler to run and
> resolve something on archlinux than it is on any of the deb or rpm based
> distributions. What do you say?

This all already exists, take a look here:
https://www.qubes-os.org/doc/templates/

- --
Best Regards,
Marek Marczykowski-Górecki
Invisible Things Lab
A: Because it messes up the order in which people normally read text.
Q: Why is top-posting such a bad thing?
-----BEGIN PGP SIGNATURE-----

iQEzBAEBCAAdFiEEhrpukzGPukRmQqkK24/THMrX1ywFAl51UFwACgkQ24/THMrX
1yy2bwf/Yn259PKBYrO+8W8n2w0dI6gUg2VYwwU4K3gvjdPRS+0MV6NHSxfxHtch
8beE1cnIo5Ca87oyKmVIbLR0UkkKf32DNgHEgMGn+SgscTPVxgEDOkr6VkCGZ0WK
D1I+2NS/ggP2Q20EDE59eipu7quRVX4xyAyEHdrZmbqNbW5Pt0B7ySjuFQ1EsQUj
fYQEQonWfRrzxLy2ngwMqZdic2NuYes4X1ZdrbjvdqKhftv9JqWIQlwzPntF7HR5
rPJ+K6GlT0nkRfYn9riW01OGeCbp7liK5eUUbRnJOUTz+qHzf86f7gUfjVDsVEG6
1bugYGDLjKyhiFsg7nKr/aURu9qDKw==
=Z8Ei
-----END PGP SIGNATURE-----

[altaway]

Hello Marek, 

How your builder.conf file looks like? Especially those entries:
 - DIST_DOM0
 - USE_QUBES_VERISON_REPO
 - RELEASE

DIST_DOM0 ?= fc31

USE_QUBES_REPO_VERSION = $(RELEASE)

RELEASE ?= 4.1

I used this: https://github.com/QubesOS/qubes-release-configs/blob/master/R4.1/qubes-os-iso-full-online.conf 

This looks like you don't use our patched pykickstart package.Did you

built it?

No, I followed the guide. Is that what the COMPONENTS variable is for? I did not iniclude pykickstart in COMPONENTS before. I was trying to buld the minimal version to save time. Maybe that's the reason. If I add pykickstart to COMPONENTS and then run "make iso" (after running "make qubes"), it says there is no rule to make target 'iso.copy-rpms.pykickstart'. There is no mention of pykickstart in the makefile. 

 

This actually may be related to the first issue - dnf can't
find the repository so it downloads package from elsewhere.
How have you dealt with "mount issue"? Those mounted directories are
needed...

 Yes, "make clean-chroot" did exactly what I needed.

 

> But the kickstart only has six lines. Do I need to install any python
> module?
>
> Putting the errors aside, I have noticed there is build support for debian
> as well. Also, the documentation mentions that CI uses ubuntu for final
> deployment. In addition to debian, I would like to add support for
> archlinux, which I think doesn't take too much. At the moment, I am using
> both fedora 31 and arch (I have a qubes install available as well for
> testing something). I personally think it is a lot simpler to run and
> resolve something on archlinux than it is on any of the deb or rpm based
> distributions. What do you say?

This all already exists, take a look here:
https://www.qubes-os.org/doc/templates/

I was talking about the makefile. For example, there is a rule named install-deps.rpm (RPM based) and one named install-deps.dpkg (debian based), but there isn't one named install-deps.pacman (archlinux based) or something like that. Maybe you didn't include it for a reason.

 

[Marek Marczykowski-Górecki]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256

On Sat, Mar 21, 2020 at 02:09:22PM -0700, altaway wrote:
> Hello Marek,
>
> How your builder.conf file looks like? Especially those entries:
> > - DIST_DOM0
> > - USE_QUBES_VERISON_REPO
> > - RELEASE
> >
>
> DIST_DOM0 ?= fc31
> USE_QUBES_REPO_VERSION = $(RELEASE)

I would also add "USE_QUBES_REPO_TESTING = 1", as many packages for R4.1
are still in testing repository only.

> RELEASE ?= 4.1
>
> I used this:
> https://github.com/QubesOS/qubes-release-configs/blob/master/R4.1/qubes-os-iso-full-online.conf
>
>
> This looks like you don't use our patched pykickstart package.Did you
>
> built it?
>
> No, I followed the guide. Is that what the COMPONENTS variable is for?

Yes.

> I
> did not iniclude pykickstart in COMPONENTS before. I was trying to buld the
> minimal version to save time. Maybe that's the reason. If I add pykickstart
> to COMPONENTS and then run "make iso" (after running "make qubes"), it says
> there is no rule to make target 'iso.copy-rpms.pykickstart'. There is no
> mention of pykickstart in the makefile.

After modifying COMPONENTS, you need to execute 'make get-sources'
again, to fetch it.

> > This actually may be related to the first issue - dnf can't
> > find the repository so it downloads package from elsewhere.
> > How have you dealt with "mount issue"? Those mounted directories are
> > needed...
> >
> Yes, "make clean-chroot" did exactly what I needed.
>
>
> > > But the kickstart only has six lines. Do I need to install any python
> > > module?
> > >
> > > Putting the errors aside, I have noticed there is build support for
> > debian
> > > as well. Also, the documentation mentions that CI uses ubuntu for final
> > > deployment. In addition to debian, I would like to add support for
> > > archlinux, which I think doesn't take too much. At the moment, I am using
> > > both fedora 31 and arch (I have a qubes install available as well for
> > > testing something). I personally think it is a lot simpler to run and
> > > resolve something on archlinux than it is on any of the deb or rpm based
> > > distributions. What do you say?
> >
> > This all already exists, take a look here:
> > https://www.qubes-os.org/doc/templates/
> >
> I was talking about the makefile. For example, there is a rule named
> install-deps.rpm (RPM based) and one named install-deps.dpkg (debian
> based), but there isn't one named install-deps.pacman (archlinux based) or
> something like that. Maybe you didn't include it for a reason.

Ah, I see. That makes sense to add. But in fact that part is just a
sanity check - if basic build tools are installed. Starting with empty
one for Arch and simply installing required packages[1] by hand should
be enough to get it working. In theory. In practice, there will be
probably some subtle issues to fix, like stuff being in /usr/bin,
instead of /usr/sbin.

[1] https://www.qubes-os.org/doc/qubes-builder/

- --
Best Regards,
Marek Marczykowski-Górecki
Invisible Things Lab
A: Because it messes up the order in which people normally read text.
Q: Why is top-posting such a bad thing?
-----BEGIN PGP SIGNATURE-----

iQEzBAEBCAAdFiEEhrpukzGPukRmQqkK24/THMrX1ywFAl58GgwACgkQ24/THMrX
1yzJHwf/Ws5EIoYPhVb68BtGsbdcY1U0g0DnqJnMIU9wCLMuhWqqlEPVy2qFvp+Z
ZC4Cf9UCtJrkEdN7haGWmeVKvWwaK79sgdZ/sp3ZFVzbgjnJDTVRUEqfCNqc/urL
AOLLN3MbOOlBqi1Fx3xCVR3LYaRHxm92zfwIbV4jIO4EddRlYFxvRGALj3DXt/g8
Gy2EfoSdozP0kx9aubxhcNZ223BHM4NoXq133utD6PzLQlBtRAI+uhdOQGEvUmZC
hk6EDM9/mG4x0fBOdmytYpdcvA1mvtVuATu4vjmKm4bjJW2K1kZg14VVmioLMHdY
hWqQbpN4XfPc6Mx8x6hDqG2p4f0/tw==
=9p9S
-----END PGP SIGNATURE-----

[altaway]

Thanks Marek.

One more thing, I need to tweak the rpm configuration a bit to prevent delays caused errors. I read that the rpm mirrors refresh every 12 hours and sometimes (pretty often with whatever mirror my fedora install chooses for some reason) the metadata gets corrupted. When that happens, all dnf functions related to the mirrors (update, for example) fail and the only way out is to wait until the mirrors have refreshed themselves. There are a few tweaks which help with the issue. 

1. Set zchunk=false in /etc/dnf/dnf.conf. This disables integrity verification and so the zchunk errors won't arise.

2. Change the default mirrors in /etc/yum.repos.d/. The corrupt mirror's url can be replaced with a working one.

Well, for this to work while running "make iso", I need to do the above in chroot-dom0-fc31 as well (while using https://github.com/QubesOS/qubes-release-configs/blob/master/R4.1/qubes-os-iso-full-online.conf). However, chroot-dom0-fc31 shows only var/. There is no other directory like etc/ or usr/. Can you tell me where etc/ is located so I can resolve these errors?

I am also going to set up ci/cd in travis or gitlab using an fc31 docker container in case the local build process gives too many issues like this (no net-neutrality in my place, dnf reposyncs take a long time). 

Also, for the GSoC proposal, is there anything else I need to know?

[Marek Marczykowski-Górecki]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256

On Fri, Mar 27, 2020 at 11:27:37PM -0700, altaway wrote:
> Thanks Marek.
>
> One more thing, I need to tweak the rpm configuration a bit to prevent
> delays caused errors. I read that the rpm mirrors refresh every 12 hours
> and sometimes (pretty often with whatever mirror my fedora install chooses
> for some reason) the metadata gets corrupted. When that happens, all dnf
> functions related to the mirrors (update, for example) fail and the only
> way out is to wait until the mirrors have refreshed themselves. There are a
> few tweaks which help with the issue.

If you set specific mirror with baseurl option, this could happen. But
the default uses metalink, which allows dnf to choose the right (up to
date) mirror dynamically. Have you changed that?

> 1. Set zchunk=false in /etc/dnf/dnf.conf. This disables integrity
> verification and so the zchunk errors won't arise.

You may want something like this:
https://github.com/QubesOS/qubes-builder-rpm/commit/dcae08a35d7b4777f0b807debebc48794673f552
I have spent significant amount of time debugging this. Reported here:
https://bugzilla.opensuse.org/1160702

> 2. Change the default mirrors in /etc/yum.repos.d/. The corrupt mirror's
> url can be replaced with a working one.
>
> Well, for this to work while running "make iso", I need to do the above in
> chroot-dom0-fc31 as well (while using
> https://github.com/QubesOS/qubes-release-configs/blob/master/R4.1/qubes-os-iso-full-online.conf

> <https://www.google.com/url?q=https%3A%2F%2Fgithub.com%2FQubesOS%2Fqubes-release-configs%2Fblob%2Fmaster%2FR4.1%2Fqubes-os-iso-full-online.conf&sa=D&sntz=1&usg=AFQjCNHxL8Qds4SvBgIldkEqH8AjM90NrA>).

> However, chroot-dom0-fc31 shows only var/. There is no other directory like
> etc/ or usr/. Can you tell me where etc/ is located so I can resolve these
> errors?

This looks like chroot-dom0-fc31 not initialized yet. Take a look at the
commit I've linked above.

> I am also going to set up ci/cd in travis or gitlab using an fc31 docker
> container in case the local build process gives too many issues like this
> (no net-neutrality in my place, dnf reposyncs take a long time).

May be a good idea. Take a look at the .travis.yml and configs use there
in installer-qubes-os repo. Travis have time limit (1h or less) and ISO
build is just below that limit.

> Also, for the GSoC proposal, is there anything else I need to know?

Basically this:
https://www.qubes-os.org/gsoc/#student-proposal-guidelines

- --
Best Regards,
Marek Marczykowski-Górecki
Invisible Things Lab
A: Because it messes up the order in which people normally read text.
Q: Why is top-posting such a bad thing?
-----BEGIN PGP SIGNATURE-----

iQEzBAEBCAAdFiEEhrpukzGPukRmQqkK24/THMrX1ywFAl6BagYACgkQ24/THMrX
1yxIBgf/XHC4aGrRiNt03nYZIL5hxUzwl0u0exVUWxOKsyConFazeybt+plgJjDL
psyiZ6/hw6oq6RKl7/tziMbIVJXqpzMA04fq02LCLMwmfmtJQIdJQxzs8wbIge9D
pV78aTkOMYzsjghYSmj36ON28Q4UZpRo9cR0D5bmVWGjog3ZY7MYxh92ET7BFwqq
Ly+m7gVcvGXrmkne7WCNOXAVCnXrb4u3MYy5vGVKkJgk5GXcvcRsS5w4euzK71pO
tBtj5MN2DCm3VjSrnUfZzqSOOO4aDaQX/xgH/yTHU9gEMYL1EZ+uNFoQpWWdkgGm
8XXGp1GlxrXAKobWoFjvydzu2IX8qg==
=gfGD
-----END PGP SIGNATURE-----

[altaway]

I have (almost) succeeded in set up the docker build. 

Personally, I prefer gitlab ci over travis as I feel it is more intuitive and there are a few added advantages (for example, you can host and use custom docker images without having to dockerhub or having to provide your credentials when wanting to push docker images). Also, gitlab doesn't have the hard limits that travis has. So I attempted to get it to work here before moving to travis. All the configuration is the same as before (full-iso-online, with the few improvements you have suggested). The progress can be found at https://gitlab.com/altaway/qubes-builder/pipelines.

I got a few issues which I (mostly) solved and would like to put them here (for future reference if not some other reason).

1. "make get-sources" failed. When I looked through the rule, I found that the sources were being shallow-cloned instead of full-cloned (65-89 in https://github.com/QubesOS/qubes-builder/blob/master/scripts/get-sources). I am not sure if this is because of gitlab or if this is a configuration issue in the project. So I explicitly put the rule so that the sources be full-cloned (13 in https://gitlab.com/altaway/qubes-builder/-/blob/testing/.gitlab-ci.yml). And I think there is a mistake in the if-else logic in the get-sources script(65-89 mentioned before). The option unshallow belongs only to "git-fetch" and not "git-clone". I have not gone thoroughly through that block, so I am not exactly sure how to exactly correct it. Can you look into it?

2. The second issue is with "make check.rpm". The rule runs  "which rpm" (214 in https://github.com/QubesOS/qubes-builder/blob/master/Makefile). Now, this command is a builtin in zsh (which I use daily), but has an explicit executable for bash (which is the default for the images served by the docker hub). Now, the "which" package isn't installed by default in many docker images. So the "check.rpm" rule fails, not because "rpm" is not present, but because "which" is not present, in many standard dockerfiles including fedora:latest. I think it is better to change "which rpm" to something like "rpm --version". Solved using a custom image with "which" installed.

3. It seems the project must be run as an unprivileged user who in the sudo / wheel group. Docker containers get the root user by default. Solved using a custom image which an added user with appropriate privileges.

4. The fourth issue is a bit strange (https://gitlab.com/altaway/qubes-builder/-/jobs/491648744). This is the only thing preventing me from proceeding forward. No satisfactory answers on the internet. Have you ever encountered this before?

All-in-all, the basic docker image from docker hub (fedora:latest) has issues, so I made my own image (https://gitlab.com/altaway/docker-fedora) with the purpose that it should closely resemble an offline fc31 installation (where qubes-project is built).

In the meantime, I will try building with the configuration used in the .travis.yml and see how it works.