templates-update-repos, local mirrors and CentOS

[Frédéric Pierret (fepitre)]

Hi,

From our discussion with Marek (https://github.com/QubesOS/qubes-linux-utils/pull/20#issuecomment-339944344), the process of updating templates to repos is manual contrary to the others (current, current-testing etc.).

I wrote some rules for :
- qubes-builder (https://github.com/fepitre/qubes-builder/commit/051a0f462b7c5a69936045d8f07b8d27a761213d)
- linux-template-builder (https://github.com/fepitre/qubes-linux-template-builder/commit/01f2f9d8672b545d504a53c11d2a3cb9dd074206)
and updated the syncing script in qubes-linux-yum (https://github.com/fepitre/qubes-linux-yum/commit/9f529a548c2551d3e0b5fa871bb7aa40a702c368).

I'm already using it to automatize the process of uploading the templates to test them. We could probably do better and your comments are welcome.

Moreover, I'm using local mirrors (I have a very bad internet bandwidth...) for building Fedora and CentOS and instead of always changing the URL in the yum-bootstrap and chroot for builder-fedora and builder-centos, I wrote a script to add the possibility in the builder.conf to set the local URLs of the baseurl. I should upload it tomorrow.

The template for CentOS7 is not yet available for r4.0. The problem is that there is not always the python3 packages for CentOS needed by Qubes tools, and we have to construct specific components for the missing dependencies.

Best,

[Marek Marczykowski-Górecki]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256

On Sat, Oct 28, 2017 at 03:32:05PM -0700, Frédéric Pierret (fepitre) wrote:
> Hi,
>
> From our discussion with Marek
> (https://github.com/QubesOS/qubes-linux-utils/pull/20#issuecomment-339944344),
> the process of updating templates to repos is manual contrary to the others
> (current, current-testing etc.).
>
> I wrote some rules for :
> - qubes-builder
> (https://github.com/fepitre/qubes-builder/commit/051a0f462b7c5a69936045d8f07b8d27a761213d)
> - linux-template-builder
> (https://github.com/fepitre/qubes-linux-template-builder/commit/01f2f9d8672b545d504a53c11d2a3cb9dd074206)
> and updated the syncing script in qubes-linux-yum
> (https://github.com/fepitre/qubes-linux-yum/commit/9f529a548c2551d3e0b5fa871bb7aa40a702c368).
>
> I'm already using it to automatize the process of uploading the templates
> to test them. We could probably do better and your comments are welcome.

I think generic update-repo-* targets could be used for that. Some work
will be needed there, but IMO that would be cleaner. See details in
comments on that commits.

> Moreover, I'm using local mirrors (I have a very bad internet bandwidth...)
> for building Fedora and CentOS and instead of always changing the URL in
> the yum-bootstrap and chroot for builder-fedora and builder-centos, I wrote
> a script to add the possibility in the builder.conf to set the local URLs
> of the baseurl. I should upload it tomorrow.

Have you tried using caching proxy? See here:
https://github.com/QubesOS/qubes-builder/blob/master/doc/Configuration.md#repo_proxy

> The template for CentOS7 is not yet available for r4.0. The problem is that
> there is not always the python3 packages for CentOS needed by Qubes tools,
> and we have to construct specific components for the missing dependencies.

- --
Best Regards,
Marek Marczykowski-Górecki
Invisible Things Lab
A: Because it messes up the order in which people normally read text.
Q: Why is top-posting such a bad thing?
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2

iQEcBAEBCAAGBQJZ84OHAAoJENuP0xzK19csMSAH/jLGbqeLJqkVedUgKustTd8B
Jlh7BWd9YJOhztwZOj4YN7yrbRZu2Dz+ROPXoJtE7v4gEENi6w2j3sg+CaQ6hiCd
42df75bkMPV5M2Igc/lN802bRrrJeyMsjGZL1laPKjlnLOkmpZrPwLdAS3Y1/6Jv
q1qAWsDzHQ8LvxnE617RtKLrZkRE+zq+xIaSP4v5iENcm62pG0QizwLgOR+rc3Ki
gwovpzps5GSgdV3rFxBgmHAYLTsGngZRAHyJ+r3f+Ry9ACDJdhGTENiZRgsrIb8A
SY2gOYZO3HLSGfKG6spjjgzvLF7HCGqQsWfEVpTtUr1FFGEBNRbbLOC0AnreYKw=
=exSN
-----END PGP SIGNATURE-----

[Frédéric Pierret (fepitre)]

No I have not tried caching proxy. In fact, I have local mirrors for Fedora and CentOS for all my computers at home, so at first it was more simple to change the baseurl for the builders. So I don't know what is the best solution for that, creating a proxy service or add the possibility directly in the builders to use local mirrors, especially if I'm the only one to use that solution.

[Frédéric Pierret (fepitre)]

Hi,
I uploaded a new version following your comments Marek. I used dynamic targets instead of a loop over DISTS_VM:
https://github.com/fepitre/qubes-builder/commit/f7e0c94f6f9c4a6f309ebf385c2c3f9a55eb4d8d
https://github.com/fepitre/qubes-linux-template-builder/commit/aeed2f1fc8f72fa45601e0e364cff64933b292bd

I added a test condition to isolate the targets update-repos-template-* from update-repos-*. I could have done the same for internal-update-repos-* but I kept the choice of having internal-templates-update-repos-*. What do you think?

[Marek Marczykowski-Górecki]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256

On Sun, Oct 29, 2017 at 12:48:13PM -0700, Frédéric Pierret (fepitre) wrote:
> Hi,
> I uploaded a new version following your comments Marek. I used dynamic
> targets instead of a loop over DISTS_VM:
> https://github.com/fepitre/qubes-builder/commit/f7e0c94f6f9c4a6f309ebf385c2c3f9a55eb4d8d
> https://github.com/fepitre/qubes-linux-template-builder/commit/aeed2f1fc8f72fa45601e0e364cff64933b292bd

Now it is better :)
Could you create pull requests from this?

> I added a test condition to isolate the targets update-repos-template-*
> from update-repos-*. I could have done the same for internal-update-repos-*
> but I kept the choice of having internal-templates-update-repos-*. What do
> you think?

That condition make it impossible to call both update-repo-templates-*
and other update-repo-* targets in one make call. But IMO it isn't a
problem.

- --
Best Regards,
Marek Marczykowski-Górecki
Invisible Things Lab
A: Because it messes up the order in which people normally read text.
Q: Why is top-posting such a bad thing?
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2

iQEcBAEBCAAGBQJZ9Mb+AAoJENuP0xzK19csiywH/0mgGndgwGobC8g8CC3KKIiO
Bfciwtq8zFnNOeMnmSkhlnjKZhVvr2xGPkbg0kFJuQfYqCw9lwCxNuxnuJPlRjh+
qqLTZ30e24sMORPvpn4nAUcDyvI7WaIWAbwg8bAlO9kaGiBuXYLSn8BUYr1r2xDq
+E43s+wHgGtSP86klDzE73CJoXnkAEo9/oVyYotA9CJhCCbA3WRecVpcXt1vwrps
YaTx2KhJIRJ82r6YRKiPwx8jyogBqkHpnjM6yzuct9NVssdJ+8MdYjPy92SmOeYg
76sD3ShnsD+/HfsBvwXB7KWdnWN1J3QV8wwvdJAvTwlVT0rbjHMjESDMPzzrnQU=
=CFko
-----END PGP SIGNATURE-----

[Frédéric Pierret (fepitre)]

Le dimanche 29 octobre 2017 22:45:38 UTC+1, Marek Marczykowski-Górecki a écrit :

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256

On Sun, Oct 29, 2017 at 12:48:13PM -0700, Frédéric Pierret (fepitre) wrote:
> Hi,
> I uploaded a new version following your comments Marek. I used dynamic
> targets instead of a loop over DISTS_VM:
> https://github.com/fepitre/qubes-builder/commit/f7e0c94f6f9c4a6f309ebf385c2c3f9a55eb4d8d
> https://github.com/fepitre/qubes-linux-template-builder/commit/aeed2f1fc8f72fa45601e0e364cff64933b292bd

Now it is better :)
Could you create pull requests from this?

Done

> I added a test condition to isolate the targets update-repos-template-*
> from update-repos-*. I could have done the same for internal-update-repos-*
> but I kept the choice of having internal-templates-update-repos-*. What do
> you think?

That condition make it impossible to call both update-repo-templates-*
and other update-repo-* targets in one make call. But IMO it isn't a
problem.

I tried a: "make update-repo-current-testing update-repo-templates-itl" and it works. It seems that the targets are properly parsed.

[Marek Marczykowski-Górecki]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256

On Sun, Oct 29, 2017 at 03:59:34PM -0700, Frédéric Pierret (fepitre) wrote:
>
>
> Le dimanche 29 octobre 2017 22:45:38 UTC+1, Marek Marczykowski-Górecki a
> écrit :
> >
> > -----BEGIN PGP SIGNED MESSAGE-----
> > Hash: SHA256
> >
> > On Sun, Oct 29, 2017 at 12:48:13PM -0700, Frédéric Pierret (fepitre)
> > wrote:
> > > Hi,
> > > I uploaded a new version following your comments Marek. I used dynamic
> > > targets instead of a loop over DISTS_VM:
> > >
> > https://github.com/fepitre/qubes-builder/commit/f7e0c94f6f9c4a6f309ebf385c2c3f9a55eb4d8d
> > >
> > https://github.com/fepitre/qubes-linux-template-builder/commit/aeed2f1fc8f72fa45601e0e364cff64933b292bd
> >
> > Now it is better :)
> > Could you create pull requests from this?
> >
> Done

Thanks, all 3 merged.

> > > I added a test condition to isolate the targets update-repos-template-*
> > > from update-repos-*. I could have done the same for
> > internal-update-repos-*
> > > but I kept the choice of having internal-templates-update-repos-*. What
> > do
> > > you think?
> >
> > That condition make it impossible to call both update-repo-templates-*
> > and other update-repo-* targets in one make call. But IMO it isn't a
> > problem.
> >
> I tried a: "make update-repo-current-testing update-repo-templates-itl" and
> it works. It seems that the targets are properly parsed.

Interesting, will try later.

iQEcBAEBCAAGBQJZ9OptAAoJENuP0xzK19csVGoH+wdUFBmZDMpszgP4iLXj8n1Y
evdyIBIy2aqOG5oTb6qGvb4JCzVvo3RXbRea0jNflzcOFcfJfZIFprmYqOSPdYg2
Iwntbv8+mi3/IdHNZM+Qu8OH6iPUuKy3GD8guIp8d7c7NbGStv0GMQxa+Zn4mVt/
MCp6YrlnnoH6xhpzH7QJz5FNdME+vH1a+8zbCaROYNMi0MFRtpAPj11Yps9DVR7v
MkqrBJrpRKYKKBpQ/w9GED89KO0nrp3wOvlQ8lzEvdOdg8f60Jeuord1DRypGJ0C
ORkjq8ug4ugeG3E5UTQSskBplQ9fziooIjzEeYUQxCvmbKXlnplQNBvXfuGjyGE=
=Ea1R
-----END PGP SIGNATURE-----