Qubes 4.0 development status update
1,932 views
Skip to first unread message
Marek Marczykowski-Górecki
Dec 4, 2016, 9:50:44 PM12/4/16
to qubes-devel
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256
Hi,
Most of the features of Qubes 4.0 are done, but there are few more things to do
be fully useable. Below is (not so) quick summary.
0. (Mostly) rewrite of Qubes core management scripts.
Over time, our codebase grew significantly, and the focus was on delivering
features and refining overall architecture. This allowed for steady growth of
Qubes OS as both the system and the community. However less thought was given to
code quality and practices. Also, the code documentation was nonexistent.
So, we've decided to rewrite this part, having modularization and extensibility
in mind when designing new code.
The work have started before Qubes 3.0 release by Wojtek and initially was
scheduled to land there (hence the name "core3" still used internally in some
places). But finalizing it turned out to be much more time consuming than
expected and in consequence this task was deferred to Qubes 4.0.
Today I can say that the core part of it is done. Every major functionality
is already (re-)implemented[5]. And thanks to the new, much easier to extend
code, we've already implemented few cool new features:
- multiple DispVM support[6]
- new storage API[7], allowing instant VM cloning, keeping arbitrary number of
volume revisions (somehow extended qvm-revert-template-changes feature) and
possibly more in the future
- improved firewall internal interface[8], allowing more rules, not interfering
with other firewall tools[9]
- and many more
1. Moving away from PV[1].
Initially we wanted to move to PVHv2 (aka HVMlite) domains, which are just like
HVM, but without qemu. But unfortunately it turned out PVHv2 is still far from
complete[2], so we've decided to go with "normal" HVM, with qemu in (still PV)
stubdomain and move to PVHv2 when it will be complete. But for Qubes 4.0
release, update qemu to more recent version - instead of the ancient,
unmaintained fork of it, from early Xen days. Mostly to improve
interoperability (see [4] for example problem) and some missing features in old
version (direct kernel boot, just to name one). Unfortunately only that old
qemu fork is supported in upstream Xen stubdomain and updating it there isn't a
priority to the Xen project[3] (the message is year old but nothing really changed
since then). So, we've decided to do it ourself, going Linux-based stubdomain
way (see [3] for other options).
This task is handled by HW42, the current status is that the majority of work is
done (it's bootable, most of things just work), but still have issues. For
example PCI passthrough doesn't work yet, which is essential for NetVM...
The current code referenced below still use PV domains by default.
2. Qubes manager
Since the internal API have changed, Qubes Manager also require adjustments.
And since the code quality degraded here too and its overall design was quite
poor[10], we've decided to throw it away and write new one, improving its
interface at the same time[11]. While there was some progress on this, it
turned out to be too much work to finish it without delaying Qubes 4.0 even
more. So, we've fall back to simpler idea: replace its main window (which was
the most horrible part) with a new one[12], as nice applet, then mostly reuse other
parts of the old manager code (like settings window, backup/restore wizards etc).
This task is handled by Kalkin. The current state is that most of frontend work
is done, but it still needs to be plugged into new backend API. See [12] for details.
3. VM Admin API
While working on connecting GUI parts and new command line tools, it turned out
it will be easier if we'd implement administration API[13] right now - not
delay it until Qubes 4.1. We've designed it the way to allow later expose part
of it to semi-trusted management VM - for example allowing it to manage
_subset_ of VMs. For now, it is mostly to have just one entity (qubesd - qubes
daemon) which handle reads and writes to /var/lib/qubes/qubes.xml and avoid all
the problems with access synchronization.
This task is handled by Wojtek. The current state is protocol design is done,
but it wasn't implemented yet. The current code use racy method of accessing
qubes.xml and may occasionally result in failure on some VM modification. We've
added an band aid to detect such situation and fail the operation, instead of
breaking qubes.xml. This is just temporary solution, until said administration
API got implemented.
4. Summary, the code
Besides the above listed things, it is possible to compile and run what's
already done for Qubes 4.0. You can use qubes-builder[14] with the config
linked below. In addition to the above list, there are plenty of
minor issues, some of them may result in data loss. So, at this point I would
not recommend using it for anything important. I think it's fair to name it
"technology preview 2".
There are also still a couple of rough edges during installation/first run. For
example "LVM thin" storage should be used, but currently it needs to be
selected manually (using custom partitioning option). And depending on the disk
layout, it may require creating those partitions manually.
There is also no package repository yet and most of components are still
versioned with 3.2.x. This means no easy update path from such built system.
The only option to migrate later to the official build is to reinstall the system.
The linked configuration use Fedora 23 as a base for dom0 (which will
probably stay the same in final version), and also Fedora 23 as the only
template. But templates will be updated later to more recent version. And
additionally, templates from Qubes 3.2 should be mostly compatible - some
feature new in Qubes 4.0 may not work, but all the basic things should be fine.
Builder configuration:
https://gist.github.com/marmarek/3e59944a890c440157c4a596f6f38b6a
$ sha256sum builder-r4.0.conf
0c0814608b07dc215b2640eb09d0ce07f4eee2ddddb6f9646a7cbdf867c1e4c6 builder-r4.0.conf
[1] https://github.com/QubesOS/qubes-issues/issues/2185
[2] http://markmail.org/message/pndwc3igjjt5p5qe
[3] http://markmail.org/message/fhor7ojwkfdopkyz
[4] https://github.com/QubesOS/qubes-issues/issues/1943
[5] https://github.com/QubesOS/qubes-issues/issues/1825
[6] https://github.com/QubesOS/qubes-issues/issues/866
[7] https://github.com/QubesOS/qubes-issues/issues/1842
[8] https://github.com/QubesOS/qubes-issues/issues/1815
[9] https://github.com/QubesOS/qubes-issues/issues/974
[10] https://github.com/QubesOS/qubes-issues/issues/1288
[11] https://github.com/QubesOS/qubes-issues/issues/1870
[12] https://github.com/QubesOS/qubes-issues/issues/2132
[13] https://github.com/QubesOS/qubes-issues/issues/853
[14] https://www.qubes-os.org/doc/qubes-builder/
- --
Best Regards,
Marek Marczykowski-Górecki
Invisible Things Lab
A: Because it messes up the order in which people normally read text.
Q: Why is top-posting such a bad thing?
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2
iQEcBAEBCAAGBQJYRNX/AAoJENuP0xzK19cso0EH/1hoAemx3WnnbUhxQMLEr0zU
3PuO1ENY3rXHSI/UVhn3N4XmjLRLV9jXlaR7E8b139jztyisrxL4dHrfE3rGPaIH
IsoV3UPo1+rySOBraAPt8Lrae77hMPnWoelVOEZzz1bgVcqKxAIALEb22CGm9KAp
A9EPKOrqANgBpwtbOhEwThB4CfHjaFbvCHljbgL5HipZh29i//dfZ4bsSJ7U218o
6+8yzX8HV4B/VFzqknpWtt+xKiccPEQMZdt97ExOCBn6Dq+TjYt0i9x9KucB7C/J
l5FggNS8E55DaJsajQkktziU9gRCVT9fDFdulk+jd4ktKc2FGWMJvnIu0m0NCWQ=
=cg+8
-----END PGP SIGNATURE-----
Hash: SHA256
Hi,
Most of the features of Qubes 4.0 are done, but there are few more things to do
be fully useable. Below is (not so) quick summary.
0. (Mostly) rewrite of Qubes core management scripts.
Over time, our codebase grew significantly, and the focus was on delivering
features and refining overall architecture. This allowed for steady growth of
Qubes OS as both the system and the community. However less thought was given to
code quality and practices. Also, the code documentation was nonexistent.
So, we've decided to rewrite this part, having modularization and extensibility
in mind when designing new code.
The work have started before Qubes 3.0 release by Wojtek and initially was
scheduled to land there (hence the name "core3" still used internally in some
places). But finalizing it turned out to be much more time consuming than
expected and in consequence this task was deferred to Qubes 4.0.
Today I can say that the core part of it is done. Every major functionality
is already (re-)implemented[5]. And thanks to the new, much easier to extend
code, we've already implemented few cool new features:
- multiple DispVM support[6]
- new storage API[7], allowing instant VM cloning, keeping arbitrary number of
volume revisions (somehow extended qvm-revert-template-changes feature) and
possibly more in the future
- improved firewall internal interface[8], allowing more rules, not interfering
with other firewall tools[9]
- and many more
1. Moving away from PV[1].
Initially we wanted to move to PVHv2 (aka HVMlite) domains, which are just like
HVM, but without qemu. But unfortunately it turned out PVHv2 is still far from
complete[2], so we've decided to go with "normal" HVM, with qemu in (still PV)
stubdomain and move to PVHv2 when it will be complete. But for Qubes 4.0
release, update qemu to more recent version - instead of the ancient,
unmaintained fork of it, from early Xen days. Mostly to improve
interoperability (see [4] for example problem) and some missing features in old
version (direct kernel boot, just to name one). Unfortunately only that old
qemu fork is supported in upstream Xen stubdomain and updating it there isn't a
priority to the Xen project[3] (the message is year old but nothing really changed
since then). So, we've decided to do it ourself, going Linux-based stubdomain
way (see [3] for other options).
This task is handled by HW42, the current status is that the majority of work is
done (it's bootable, most of things just work), but still have issues. For
example PCI passthrough doesn't work yet, which is essential for NetVM...
The current code referenced below still use PV domains by default.
2. Qubes manager
Since the internal API have changed, Qubes Manager also require adjustments.
And since the code quality degraded here too and its overall design was quite
poor[10], we've decided to throw it away and write new one, improving its
interface at the same time[11]. While there was some progress on this, it
turned out to be too much work to finish it without delaying Qubes 4.0 even
more. So, we've fall back to simpler idea: replace its main window (which was
the most horrible part) with a new one[12], as nice applet, then mostly reuse other
parts of the old manager code (like settings window, backup/restore wizards etc).
This task is handled by Kalkin. The current state is that most of frontend work
is done, but it still needs to be plugged into new backend API. See [12] for details.
3. VM Admin API
While working on connecting GUI parts and new command line tools, it turned out
it will be easier if we'd implement administration API[13] right now - not
delay it until Qubes 4.1. We've designed it the way to allow later expose part
of it to semi-trusted management VM - for example allowing it to manage
_subset_ of VMs. For now, it is mostly to have just one entity (qubesd - qubes
daemon) which handle reads and writes to /var/lib/qubes/qubes.xml and avoid all
the problems with access synchronization.
This task is handled by Wojtek. The current state is protocol design is done,
but it wasn't implemented yet. The current code use racy method of accessing
qubes.xml and may occasionally result in failure on some VM modification. We've
added an band aid to detect such situation and fail the operation, instead of
breaking qubes.xml. This is just temporary solution, until said administration
API got implemented.
4. Summary, the code
Besides the above listed things, it is possible to compile and run what's
already done for Qubes 4.0. You can use qubes-builder[14] with the config
linked below. In addition to the above list, there are plenty of
minor issues, some of them may result in data loss. So, at this point I would
not recommend using it for anything important. I think it's fair to name it
"technology preview 2".
There are also still a couple of rough edges during installation/first run. For
example "LVM thin" storage should be used, but currently it needs to be
selected manually (using custom partitioning option). And depending on the disk
layout, it may require creating those partitions manually.
There is also no package repository yet and most of components are still
versioned with 3.2.x. This means no easy update path from such built system.
The only option to migrate later to the official build is to reinstall the system.
The linked configuration use Fedora 23 as a base for dom0 (which will
probably stay the same in final version), and also Fedora 23 as the only
template. But templates will be updated later to more recent version. And
additionally, templates from Qubes 3.2 should be mostly compatible - some
feature new in Qubes 4.0 may not work, but all the basic things should be fine.
Builder configuration:
https://gist.github.com/marmarek/3e59944a890c440157c4a596f6f38b6a
$ sha256sum builder-r4.0.conf
0c0814608b07dc215b2640eb09d0ce07f4eee2ddddb6f9646a7cbdf867c1e4c6 builder-r4.0.conf
[1] https://github.com/QubesOS/qubes-issues/issues/2185
[2] http://markmail.org/message/pndwc3igjjt5p5qe
[3] http://markmail.org/message/fhor7ojwkfdopkyz
[4] https://github.com/QubesOS/qubes-issues/issues/1943
[5] https://github.com/QubesOS/qubes-issues/issues/1825
[6] https://github.com/QubesOS/qubes-issues/issues/866
[7] https://github.com/QubesOS/qubes-issues/issues/1842
[8] https://github.com/QubesOS/qubes-issues/issues/1815
[9] https://github.com/QubesOS/qubes-issues/issues/974
[10] https://github.com/QubesOS/qubes-issues/issues/1288
[11] https://github.com/QubesOS/qubes-issues/issues/1870
[12] https://github.com/QubesOS/qubes-issues/issues/2132
[13] https://github.com/QubesOS/qubes-issues/issues/853
[14] https://www.qubes-os.org/doc/qubes-builder/
- --
Best Regards,
Marek Marczykowski-Górecki
Invisible Things Lab
A: Because it messes up the order in which people normally read text.
Q: Why is top-posting such a bad thing?
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2
iQEcBAEBCAAGBQJYRNX/AAoJENuP0xzK19cso0EH/1hoAemx3WnnbUhxQMLEr0zU
3PuO1ENY3rXHSI/UVhn3N4XmjLRLV9jXlaR7E8b139jztyisrxL4dHrfE3rGPaIH
IsoV3UPo1+rySOBraAPt8Lrae77hMPnWoelVOEZzz1bgVcqKxAIALEb22CGm9KAp
A9EPKOrqANgBpwtbOhEwThB4CfHjaFbvCHljbgL5HipZh29i//dfZ4bsSJ7U218o
6+8yzX8HV4B/VFzqknpWtt+xKiccPEQMZdt97ExOCBn6Dq+TjYt0i9x9KucB7C/J
l5FggNS8E55DaJsajQkktziU9gRCVT9fDFdulk+jd4ktKc2FGWMJvnIu0m0NCWQ=
=cg+8
-----END PGP SIGNATURE-----
Manuel Amador (Rudd-O)
Dec 4, 2016, 10:55:51 PM12/4/16
to qubes...@googlegroups.com
On 12/05/2016 02:50 AM, Marek Marczykowski-Górecki wrote:
>
> There are also still a couple of rough edges during installation/first
> run. For
> example "LVM thin" storage should be used, but currently it needs to be
> selected manually (using custom partitioning option). And depending on
> the disk
> layout, it may require creating those partitions manually.
>
Honestly, it would be so much better if Qubes OS did NOT use LVM, and
>
> There are also still a couple of rough edges during installation/first
> run. For
> example "LVM thin" storage should be used, but currently it needs to be
> selected manually (using custom partitioning option). And depending on
> the disk
> layout, it may require creating those partitions manually.
>
used btrfs or ZFS instead. LVM has no protection whatsoever against
data corruption. I hope the APIs are modular such that there is an
implementable interface for storage backends which does not assume LVM
is the only thing that underlies the system. As a person using Qubes OS
on top of ZFS, I would find myself in quite some trouble if that was not
the case.
--
Rudd-O
http://rudd-o.com/
Vít Šesták
Dec 5, 2016, 2:19:41 AM12/5/16
to qubes-devel
I am interested about the implications of using HVMs. I know about the introduced need of some CPU features, some changes under-the-hood and potentiially better compatibility. But I am interested in some others:
1. What will change about performance and memory-usage characteristics? I believe there will be just small (and maybe even positive) performance difference for CPU/RAM-related tasks. The difference for I/O will probably be rather low if any. There might be some differences for PCI latency, but I am unsure about them. Start of a VM will probably take slightly longer, because of the need of the stubdom. RAM usage will be also somewhat (how much?) higher because of the stubdom.
2. Security implications. When attacker has a QEMU 0day, HVM fails as a counter-measure against PV-related vulnerabilities. In such case, the attack scenario is even larger (both PV-only and HVM-only Xen vulnerabilities can be used). I remember you mentioned an idea about killing the stubdom in an early phase of the boot (probably even before mounting /rw), which would somehow mitigate some (most?) attacks. As a nice side-effect, it would also lower the memory usage. What's the current state of this countermeasure?
Regards,
Vít Šesták 'v6ak'
1. What will change about performance and memory-usage characteristics? I believe there will be just small (and maybe even positive) performance difference for CPU/RAM-related tasks. The difference for I/O will probably be rather low if any. There might be some differences for PCI latency, but I am unsure about them. Start of a VM will probably take slightly longer, because of the need of the stubdom. RAM usage will be also somewhat (how much?) higher because of the stubdom.
2. Security implications. When attacker has a QEMU 0day, HVM fails as a counter-measure against PV-related vulnerabilities. In such case, the attack scenario is even larger (both PV-only and HVM-only Xen vulnerabilities can be used). I remember you mentioned an idea about killing the stubdom in an early phase of the boot (probably even before mounting /rw), which would somehow mitigate some (most?) attacks. As a nice side-effect, it would also lower the memory usage. What's the current state of this countermeasure?
Regards,
Vít Šesták 'v6ak'
Marek Marczykowski-Górecki
Dec 5, 2016, 5:12:34 AM12/5/16
to Vít Šesták, qubes-devel
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256
Hash: SHA256
On Sun, Dec 04, 2016 at 11:19:41PM -0800, Vít Šesták wrote:
> I am interested about the implications of using HVMs. I know about the introduced need of some CPU features, some changes under-the-hood and potentiially better compatibility. But I am interested in some others:
>
> 1. What will change about performance and memory-usage
> characteristics? I believe there will be just small (and maybe even
> positive) performance difference for CPU/RAM-related tasks. The
> difference for I/O will probably be rather low if any. There might be
> some differences for PCI latency, but I am unsure about them. Start of
> a VM will probably take slightly longer, because of the need of the
> stubdom. RAM usage will be also somewhat (how much?) higher because of
> the stubdom.
Your intuition is right. As for memory usage, stubdomain use about 50MB,
> I am interested about the implications of using HVMs. I know about the introduced need of some CPU features, some changes under-the-hood and potentiially better compatibility. But I am interested in some others:
>
> 1. What will change about performance and memory-usage
> characteristics? I believe there will be just small (and maybe even
> positive) performance difference for CPU/RAM-related tasks. The
> difference for I/O will probably be rather low if any. There might be
> some differences for PCI latency, but I am unsure about them. Start of
> a VM will probably take slightly longer, because of the need of the
> stubdom. RAM usage will be also somewhat (how much?) higher because of
> the stubdom.
so not that much...
> 2. Security implications. When attacker has a QEMU 0day, HVM fails as
> a counter-measure against PV-related vulnerabilities. In such case,
> the attack scenario is even larger (both PV-only and HVM-only Xen
> vulnerabilities can be used).
things are unavoidable.
> I remember you mentioned an idea about
> killing the stubdom in an early phase of the boot (probably even
> before mounting /rw), which would somehow mitigate some (most?)
> attacks. As a nice side-effect, it would also lower the memory usage.
> What's the current state of this countermeasure?
- --
Best Regards,
Marek Marczykowski-Górecki
Invisible Things Lab
A: Because it messes up the order in which people normally read text.
Q: Why is top-posting such a bad thing?
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2
C3aNNKFQSHh3QqP6TQaGksmB1dOlGH5hGW606U+iA83K/oDBJ1BlegdO5HzY5SYh
JWAoMK9/nRpw5bCoYkJtOmFpzBcI3YCIV0SfWu80kEj9Ihszg6qOBmzo/og7eUtl
9RSO5OWYI9Jso1o8bxVvcUdiSr8M+GR1rc5bBQlyza5GwGV/SOXXOMWPekDijggh
aUXZTq8aiVsZ65QsnXn3OjsJp3ptftsPWzxpWwMrimNNsn0D+6U8syUC7epNBnCI
4m/77bjBOo0Xfq5HEzCCfwMndrm++GqBpr2grCPkEFE6HSaLuoa2oNlphXk9Ls0=
=i2ru
-----END PGP SIGNATURE-----
Marek Marczykowski-Górecki
Dec 5, 2016, 5:14:28 AM12/5/16
to Manuel Amador (Rudd-O), qubes...@googlegroups.com
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256
Hash: SHA256
linked github issue (will be moved to qubes-doc later). Also adding
fs-based module should be quite easy, because we have also 'file'
storage module, for handling file based VM images. Should be easy to
build btrfs or zfs module based on it.
- --
Best Regards,
Marek Marczykowski-Górecki
Invisible Things Lab
A: Because it messes up the order in which people normally read text.
Q: Why is top-posting such a bad thing?
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2
fLKI6tHS3OwlMbmHj2vzd4PBBRkL+MCGenNoedyR8i0wZQ8m93G+tZtH9l8V101j
LI5YrZ3P+yFSEuHsL3d0c/6DWIEoZHp2THHW0nuIs3CovxfoeXVPx8B8QZa1zJ6f
QJcrRLqDeVgErbk+m5/pL/T8nCoOWy1Sk5AMvghCHZktsZD0/247zP86g5dkTW4Y
NquxudIJxRCXx6q4pxksGD3m86GFnU6lZBiuxSIMZOzH85OsV+8dVqfulArkoFY/
yg4LpEex2URz19oSamKD/kGd8FYHQy/DrnFHYAMZIATtzLFTjIfnOpLpasExZBk=
=QbjE
-----END PGP SIGNATURE-----
Vít Šesták
Dec 8, 2016, 1:39:29 AM12/8/16
to qubes-devel
Thanks for the responses. Knowing the RAM requirements is important for choosing new hardware.
Regards,
Vít Šesták 'v6ak'
Regards,
Vít Šesták 'v6ak'
Reply all
Reply to author
Forward
0 new messages