Stable Template Build Versions

[WhonixQubes]

Hi,

In documenting the template source build process for Whonix, I'm
wondering a few things...



1.

Does the QubesBuilder support building specific historical stable
versions of the templates?

For example, it seems like tag "jm_8f6b2d98" of "qubes-builder" was
maybe used for building the last 2.1.8 Whonix template release?

Is there a straightforward way with QubesBuilder to build using all the
same code of the 2.1.8 Whonix stable template release?

Or does QubesBuilder not support historical build versions and, rather,
just uses whatever the most current commit of the supporting "qubes-"
repos during the build process?



2.

More general:

At any given time, is there a consistent source for building "guaranteed
will build" current versions of the QubesBuilder items (templates, ISO,
etc)?

As an example:

The "QubesOS/qubes-builder" repo currently has 671 commits at "ac2e132".

The "marmarek/qubes-builder" repo currently has 686 commits at
"fbe87bc".

Along with other updates, it looks in-between these, in the "marmarek"
repo, Jason has refactored the Whonix template build scripts to fit the
new QubesBuilder configuration. Meaning that the "QubesOS" repo does not
include these changes.

With these repo differences, if someone was looking to build Whonix
templates using the main "QubesOS" repo -- right now -- would they even
build?

Again, generally speaking...

At any given time, is there a consistent source for building "guaranteed
will build" current versions of the QubesBuilder items (templates, ISO,
etc)?



Thanks!


WhonixQubes

[Marek Marczykowski-Górecki]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

You can (ab)use BRANCH_* settings by pointing it at specific commit/tag
instead of real branch. It requires you to collect the values though...
We create R* tag on every component when releasing new major version -
this tag can be used to get all the sources from the moment of release.
But in case of templates (especially community-supported ones) there is
no such tag, and in case of Whonix there cannot be one, because we do
not control all the repositories.
However I can collect tag names when building new templates and provide
such config somewhere (example-configs directory? website?). You can try
to do that for already released templates, based on commit timestamps...

Generally, content of QubesOS github account should be considered as
stable, at least in terms of Qubes components.

- --
Best Regards,
Marek Marczykowski-Górecki
Invisible Things Lab
A: Because it messes up the order in which people normally read text.
Q: Why is top-posting such a bad thing?
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1

iQEcBAEBAgAGBQJVNHBYAAoJENuP0xzK19csI+kIAJlIqpMR8IAVBCPlmm+0aMb0
WC+j1Q18dJrnPHIC1ecLROrHwzHCBpPQXmFAJH5HIxv6TEzHXpHrqDLZOof4KfKK
QorB8Cv5PSvgR5mhjQG0PEq5z8cbtLAyr3hawEI2ZHHr8XMQYC48IIzye+ATtdbr
UATAkUYjyEbWxV6OoYulQtlpkVyWivsOHylS8LU1vYpDXAmD6T1GjRngtcD22+pt
MgSX+I6TcVDAnbJnKmTvPxIviME4kdjmfaFJYmPcfHKOcZK6KViiKsTUW/GjiOBa
Iq5FFW+4UT8bc/C3dpLd0RxBe0G2C8bmO9ajhFmdKtcl3XSjL1RIVOjUajP6Kgw=
=BT/3
-----END PGP SIGNATURE-----

[Jason M]

On Sunday, 19 April 2015 22:50:51 UTC-4, WhonixQubes wrote:

Hi,

In documenting the template source build process for Whonix, I'm
wondering a few things...

That sounds like a great idea.  You may want to checkout the new setup application which I designed for the purpose of making it easier to build; especially templates.  With the new qubes-builder layout, one currently would need to run setup, then get-sources, then re-run setup again to be able to choose whonix.  at some point I may add in the ability to be able to grab the whonix (BUILDER_PLUGINS) directly from setup to prevent having to run it twice.

I suppose it could also be interesting to add historical tags to the setup configuration file to allow selecting some of those historical builds, although some of the history may have been lost due to refactoring into new packages.  In regards to whonix historical builds, we would need to also need the tags of whonix related repos as well.  I am thinking that this data would belong in the qubes-template-whonix package using an ini type configuration file unless the powers to be would approve a yaml format which would mean the data would be directly importable as a dictionary :)

Here is an example of an .ini formatted file that could be used.  Any other ideas?  The setup program could then use that data to display a list of historical versions to build and then create the proper configuration file to use the tags provided.
[Whonix 9.6-6]
COMPONENT = VERSION_TAG
vmm-xen = 51d80f3e1b1946f477d324b15a53af44b7ed3c94
core-libvirt = ...
core-vchan-xen = ...
core-libvirt = ...
core-vchan-xen = ...

We may want to consider full URL to components incase they change locations?

At this point I do not consider it worthwhile to be able to build historical builds of Whonix due to its experimental nature.  I think it will be worthwhile for the next (and possibly last) 9.6 release since it has many bug fixes.  That release will immediately be superseded by Whonix 10.

[WhonixQubes]

On 2015-04-20 3:19 am, Marek Marczykowski-Górecki wrote:
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
>

> On Mon, Apr 20, 2015 at 02:50:49AM +0000, WhonixQubes wrote:
>> Hi,
>>
>> In documenting the template source build process for Whonix, I'm
>> wondering a
>> few things...
>>
>>
>>

> You can (ab)use BRANCH_* settings by pointing it at specific commit/tag
> instead of real branch. It requires you to collect the values though...
> We create R* tag on every component when releasing new major version -
> this tag can be used to get all the sources from the moment of release.

Great. Thank you.

> But in case of templates (especially community-supported ones) there is
> no such tag, and in case of Whonix there cannot be one, because we do
> not control all the repositories.

Isn't this taken care of by the general Whonix repository version tag?

Like how the "Makefile" in "qubes-template-whonix" references the Whonix
9.6 tag...

export BRANCH_Whonix = 9.6

Along with a few other Whonix packages that need more current
versions...

export BRANCH_qubes_whonix = 9.6.2
export BRANCH_whonix_setup_wizard = 0.5-1
export BRANCH_python_guimessages = 0.3-1

I thought each general tag version of the Whonix build script associates
historically specific individual packages of Whonix. So that, for
example, Whonix 9.6 will always build with the same Whonix 9.6
associated package versions from when it was originally released?

So aren't the Qubes tags and the general Whonix tags from QubesBuilder
enough, where we can assume the Whonix build script handles the
historical Whonix package tags?

> However I can collect tag names when building new templates and provide
> such config somewhere (example-configs directory? website?). You can
> try
> to do that for already released templates, based on commit
> timestamps...

Yes. This would certainly help.

Jason's followup proposal looks interesting.

Maybe a script that collects the tags and puts them into a structured
data file that can be programmatically built from later by the "setup"
script would be worthwhile.


For cross-referencing past already released builds... Are past RPM
builds and Git commits set to Poland CET/CEST timezone?

> Generally, content of QubesOS github account should be considered as
> stable, at least in terms of Qubes components.
>
> - --
> Best Regards,
> Marek Marczykowski-Górecki
> Invisible Things Lab
> A: Because it messes up the order in which people normally read text.
> Q: Why is top-posting such a bad thing?
> -----BEGIN PGP SIGNATURE-----
> Version: GnuPG v1
>
> iQEcBAEBAgAGBQJVNHBYAAoJENuP0xzK19csI+kIAJlIqpMR8IAVBCPlmm+0aMb0
> WC+j1Q18dJrnPHIC1ecLROrHwzHCBpPQXmFAJH5HIxv6TEzHXpHrqDLZOof4KfKK
> QorB8Cv5PSvgR5mhjQG0PEq5z8cbtLAyr3hawEI2ZHHr8XMQYC48IIzye+ATtdbr
> UATAkUYjyEbWxV6OoYulQtlpkVyWivsOHylS8LU1vYpDXAmD6T1GjRngtcD22+pt
> MgSX+I6TcVDAnbJnKmTvPxIviME4kdjmfaFJYmPcfHKOcZK6KViiKsTUW/GjiOBa
> Iq5FFW+4UT8bc/C3dpLd0RxBe0G2C8bmO9ajhFmdKtcl3XSjL1RIVOjUajP6Kgw=
> =BT/3
> -----END PGP SIGNATURE-----

On 2015-04-20 5:50 am, Jason M wrote:
> On Sunday, 19 April 2015 22:50:51 UTC-4, WhonixQubes wrote:
>>
>> Hi,
>>
>> In documenting the template source build process for Whonix, I'm
>> wondering a few things...
>
>
> That sounds like a great idea.

Great.

> You may want to checkout the new setup
> application which I designed for the purpose of making it easier to
> build;
> especially templates.

I have been looking at your work. Very nice. :)

> With the new qubes-builder layout, one currently
> would need to run setup, then get-sources, then re-run setup again to
> be
> able to choose whonix. at some point I may add in the ability to be
> able
> to grab the whonix (BUILDER_PLUGINS) directly from setup to prevent
> having
> to run it twice.

Thanks for the tip!

> I suppose it could also be interesting to add historical tags to the
> setup
> configuration file to allow selecting some of those historical builds,
> although some of the history may have been lost due to refactoring into
> new
> packages.

The main concern is with people being able to match the current release
build with a source build.

As the QubesOS repo gets updated, the binary release and people's source
builds will get out of sync -- many different source build versions
(with potential breaks that'd be tougher to support) based on offering
public build instructions -- unless we include this option to use the
same Qubes package tags to sync with the current binary version.

Historical versions before the current binary release being supported by
the current repo version wouldn't be a true necessity, IMO.

But, supporting a tag snapshot from at least the current rolling binary
release would be nice going forward.

> In regards to whonix historical builds, we would need to also
> need the tags of whonix related repos as well.

COPYING FROM ABOVE > ADDRESSING SAME ISSUE HERE:

Isn't this taken care of by the general Whonix repository version tag?

Like how the "Makefile" in "qubes-template-whonix" references the Whonix
9.6 tag...

export BRANCH_Whonix = 9.6

Along with a few other Whonix packages that need more current
versions...

export BRANCH_qubes_whonix = 9.6.2
export BRANCH_whonix_setup_wizard = 0.5-1
export BRANCH_python_guimessages = 0.3-1

I thought each general tag version of the Whonix build script associates
historically specific individual packages of Whonix. So that, for
example, Whonix 9.6 will always build with the same Whonix 9.6
associated package versions from when it was originally released?

So aren't the Qubes tags and the general Whonix tags from QubesBuilder
enough, where we can assume the Whonix build script handles the
historical Whonix package tags?

> I am thinking that this
> data would belong in the qubes-template-whonix package using an ini
> type
> configuration file unless the powers to be would approve a yaml format
> which would mean the data would be directly importable as a dictionary
> :)
>
> Here is an example of an .ini formatted file that could be used. Any
> other
> ideas?
> The setup program could then use that data to display a list of
> historical versions to build and then create the proper configuration
> file
> to use the tags provided.
> [Whonix 9.6-6]
> COMPONENT = VERSION_TAG
> vmm-xen = 51d80f3e1b1946f477d324b15a53af44b7ed3c94
> core-libvirt = ...
> core-vchan-xen = ...
> core-libvirt = ...
> core-vchan-xen = ...
>

This sounds good.

Would there be any good future looking formatting considerations for
incorporating this into a management stack, Salt or something, for
initiating QubesBuilder in an AppVM from a central domain?

Salt makes use of some YAML, right?

> We may want to consider full URL to components incase they change
> locations?

Sure thing.

> At this point I do not consider it worthwhile to be able to build
> historical builds of Whonix due to its experimental nature. I think it
> will be worthwhile for the next (and possibly last) 9.6 release since
> it
> has many bug fixes. That release will immediately be superseded by
> Whonix
> 10.
>
>

Yup. I agree with that.

That can be collected and done manually if people truly truly wanted to.

WhonixQubes

[Marek Marczykowski-Górecki]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

IMO there is no need for new config file format, builder.conf allow to
express all of that. You can simply have ready to use config files to
build particular template version. This would be better in terms of
stability - the plain file will be guaranteed to not change, but
setup-generated one could change in the future if some feature is added
to the setup script.

Additionally if you're using DispVM to build template, it needs just a
single config file.

- --
Best Regards,
Marek Marczykowski-Górecki
Invisible Things Lab
A: Because it messes up the order in which people normally read text.
Q: Why is top-posting such a bad thing?
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1

iQEcBAEBAgAGBQJVNPSHAAoJENuP0xzK19csWmsIAIq6JoPaBjCln865sF8Ab/8h
U4DW8puwr0ZFzo3MgXYdTD+ObRS6TOjIoMO6hfN+orQHEoSysXeSyntk3Y6HV8kx
LOiAWZ61kG2STFjkaUdzd+LqndUYlxmTiR+hXf7I7LOerbJtk+s699qxfDyiuHYd
I6574kHcz55gXz5S6ldk9HUkHYkFF23hNQSAIFmLiX0fo8J5EdgP05JA35ZZwx7V
jcIGACxAK0GzyGupUL+NGu7JtamirQTmBqucz/RX/YbQu0GdNq8DrjBsZKjbYjnG
XR2fegZIMGKC/GTTEWVJfJE+eSzHmVNSxaoVnqXJadrYMbkTy2Dp0/kTbkZsEcI=
=+r6I
-----END PGP SIGNATURE-----

[Marek Marczykowski-Górecki]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

[pasting my previous reply here, to have it all in one place]

I wouldn't assume so. For example recently some scripts were moved to
separate package and while still work with the same Whonix version, the
produced template is slightly different. If you want to reproduce
(almost) exactly the same build, it needs to use the same version of
builder plugins.

> >However I can collect tag names when building new templates and provide
> >such config somewhere (example-configs directory? website?). You can try
> >to do that for already released templates, based on commit timestamps...
>
>
> Yes. This would certainly help.
>
> Jason's followup proposal looks interesting.
>
> Maybe a script that collects the tags and puts them into a structured data
> file that can be programmatically built from later by the "setup" script
> would be worthwhile.
>
>
> For cross-referencing past already released builds... Are past RPM builds
> and Git commits set to Poland CET/CEST timezone?

All of those contains timezone information, so can be easily compared.
But yes - most timestamps are in Poland CET/CEST timezone (with
exception to Jason's).

> >Generally, content of QubesOS github account should be considered as
> >stable, at least in terms of Qubes components.
>
>

> On 2015-04-20 5:50 am, Jason M wrote:
> >On Sunday, 19 April 2015 22:50:51 UTC-4, WhonixQubes wrote:
> >>
> >>Hi,
> >>
> >>In documenting the template source build process for Whonix, I'm
> >>wondering a few things...
> >
> >
> >That sounds like a great idea.
>
>
> Great.
>
>
> >You may want to checkout the new setup
> >application which I designed for the purpose of making it easier to build;
> >especially templates.
>
>
> I have been looking at your work. Very nice. :)
>
>
> >With the new qubes-builder layout, one currently
> >would need to run setup, then get-sources, then re-run setup again to be
> >able to choose whonix. at some point I may add in the ability to be able
> >to grab the whonix (BUILDER_PLUGINS) directly from setup to prevent having
> >to run it twice.
>
>
> Thanks for the tip!
>
>
> >I suppose it could also be interesting to add historical tags to the setup
> >configuration file to allow selecting some of those historical builds,
> >although some of the history may have been lost due to refactoring into
> >new
> >packages.
>
>
> The main concern is with people being able to match the current release
> build with a source build.

That would be even more powerful if we'd have reproducible builds
implemented...

> As the QubesOS repo gets updated, the binary release and people's source
> builds will get out of sync -- many different source build versions (with
> potential breaks that'd be tougher to support) based on offering public
> build instructions -- unless we include this option to use the same Qubes
> package tags to sync with the current binary version.
>
> Historical versions before the current binary release being supported by the
> current repo version wouldn't be a true necessity, IMO.
>
> But, supporting a tag snapshot from at least the current rolling binary
> release would be nice going forward.
>
>
> >In regards to whonix historical builds, we would need to also
> >need the tags of whonix related repos as well.
>
>
> COPYING FROM ABOVE > ADDRESSING SAME ISSUE HERE:
>
> Isn't this taken care of by the general Whonix repository version tag?
>
> Like how the "Makefile" in "qubes-template-whonix" references the Whonix 9.6
> tag...
>
> export BRANCH_Whonix = 9.6
>
> Along with a few other Whonix packages that need more current versions...
>
> export BRANCH_qubes_whonix = 9.6.2
> export BRANCH_whonix_setup_wizard = 0.5-1
> export BRANCH_python_guimessages = 0.3-1
>
> I thought each general tag version of the Whonix build script associates
> historically specific individual packages of Whonix. So that, for example,
> Whonix 9.6 will always build with the same Whonix 9.6 associated package
> versions from when it was originally released?

Actually this looks to be a branch name, at least in case of
qubes-whonix package. But the main Whonix repository indeed uses a tag
name.

> So aren't the Qubes tags and the general Whonix tags from QubesBuilder
> enough, where we can assume the Whonix build script handles the historical
> Whonix package tags?
>
>
> >I am thinking that this
> >data would belong in the qubes-template-whonix package using an ini type
> >configuration file unless the powers to be would approve a yaml format
> >which would mean the data would be directly importable as a dictionary :)
> >
> >Here is an example of an .ini formatted file that could be used. Any
> >other
> >ideas?
> >The setup program could then use that data to display a list of
> >historical versions to build and then create the proper configuration file
> >to use the tags provided.
> >[Whonix 9.6-6]
> >COMPONENT = VERSION_TAG
> >vmm-xen = 51d80f3e1b1946f477d324b15a53af44b7ed3c94
> >core-libvirt = ...
> >core-vchan-xen = ...
> >core-libvirt = ...
> >core-vchan-xen = ...
> >
>
> This sounds good.
>

IMO there is no need for new config file format, builder.conf allow to
express all of that. You can simply have ready to use config files to
build particular template version. This would be better in terms of
stability - the plain file will be guaranteed to not change, but
setup-generated one could change in the future if some feature is added
to the setup script.

Additionally if you're using DispVM to build template, it needs just a
single config file.

- --
Best Regards,
Marek Marczykowski-Górecki
Invisible Things Lab
A: Because it messes up the order in which people normally read text.
Q: Why is top-posting such a bad thing?
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1

iQEcBAEBAgAGBQJVNPgpAAoJENuP0xzK19cs2zoH/iITVWWSpArDDANMOD9bK5u4
TMwdHvhsllF6HP2lRaNUYW+f6IhQP5Nd8lh0Zgadv3TkqgztfDJXdDb4cUMksF//
1TDcyY0IHx47Q2TsCgOm2XzRfi/Bvh430o7/GiUSO6Hl8S3+BSSO/2C/wsMuP4XV
EefoX+fbDAZTZxraCvRM25hxLWId4G75tgdb8oT4TogQ+Z1Or5vUWXlxnXjF1Ndt
UOK5Dc4/cjEi/4N1ZsQH/KomWxQ+LLx02+STGS7s5gLMp7htqf/2XbqsFlekr1Q1
KDk2HuMZPNOjKiTxtgS0/0aA3wRi9s37jkK4uypuQlTSbReZ6cDeO5MbXJwF0/w=
=zSLY
-----END PGP SIGNATURE-----

[WhonixQubes]

Okay, then going forward can the commit/tag version of the Qubes plugin
repos *also* be collected at time of binary builds and later reused by
QubesBuilder BRANCH_ settings to achieve a fixed versioning of the Qubes
+ Whonix build code?

For Whonix templates, I don't ultimately mind if older past historical
versions get broken. Rather, I'm primarily looking to maintain a working
and user-friendly source build guide that generally syncs to wherever
the latest ITL binary Whonix release is at.


FYI: Whonix project uses Git Submodules in its repo, so that includes a
fixed historical build state for all of its code repos with each Whonix
release tag.

More info: https://www.whonix.org/forum/index.php/topic,1187.0.html

[Marek Marczykowski-Górecki]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Yes, that's totally possible.
The only problem currently is the main qubes-builder repository - it
will not exactly use BRANCH_builder setting(*). But since more and more
code is moved to the plugins, it shouldn't be a problem.

(*) make get-sources calls 'git merge', not 'git reset', so it will not
rollback to some past tag.

> For Whonix templates, I don't ultimately mind if older past historical
> versions get broken. Rather, I'm primarily looking to maintain a working and
> user-friendly source build guide that generally syncs to wherever the latest
> ITL binary Whonix release is at.
>
>
> FYI: Whonix project uses Git Submodules in its repo, so that includes a
> fixed historical build state for all of its code repos with each Whonix
> release tag.
>
> More info: https://www.whonix.org/forum/index.php/topic,1187.0.html

- --
Best Regards,
Marek Marczykowski-Górecki
Invisible Things Lab
A: Because it messes up the order in which people normally read text.
Q: Why is top-posting such a bad thing?
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1

iQEcBAEBAgAGBQJVPo0GAAoJENuP0xzK19csWMMIAI2gJFS/kjSeSxtbnLUXw1o7
51CCQVialiabUMFpfAKhEjLnfol7KCq0utBEgXko62iXyCkzDh+v/M10yIpBmSyt
PN9J17FlcLyASNqLCPHxakaTzlp8lbQh+lcbSK4CDpVYeEh8wAZ71Ma6DPiTwDmT
NLzr/10Zjgfh5RRWWihW1LFDQ0Ns2YtrdMot+Dp5O3MmY6RrEneGk3kgFOXuQ9+7
rQJDzSNc1x1O9HbpAMrB0lsBTnVVz3ERGkMthVCS2rwufdshGDE88OVoeycXNI54
6zlmwlq+Vd+B0c4letKyOLcBMBSvK+BItZA0/JrxKuiW5sreQ50c5X+7mkLcI4I=
=iimV
-----END PGP SIGNATURE-----

[nrgaway]

Patrick was also looking into some other solutions other than `submodules`.  I am not a huge fan of them myself as they tend to cause many headaches.  You can ask Patrick or do some google search on the topic. :)