Today Xen.org has announced a critical bug in the Xen hypervisor [1],
that allows for PV DomU->hypervisor escalation. The bug has been found,
as usual, by Rafal Wojtczuk, Qubes co-author, and is a beautiful
demonstration of incorrect design decisions made by Intel, specifically
in the behavior of the SYSRET instruction (AMD processors are not
vulnerable).
So, while the bug has been patched in software by adding some extra
checks to the hypervisor (Xen.org just released a patch), one should
still consider it a CPU-level bug. The issue also affects other systems,
not just Xen (more details in the original advisory). Congrats to Rafal
for coming unprecedentedly close to "The Holly Grail" of system-level
exploitation!
Patching
----------
We have uploaded the patched Xen packages and users of current Qubes
Beta 3 should upgrade immediately, by running (in Dom0 console):
sudo qubes-dom0-updates
... which should bring Xen v4.1.2-4 packages that are immune to the
attack. Please reboot your systems afterwards.
> Today Xen.org has announced a critical bug in the Xen hypervisor [1],
> that allows for PV DomU->hypervisor escalation. The bug has been found,
> as usual, by Rafal Wojtczuk, Qubes co-author, and is a beautiful
> demonstration of incorrect design decisions made by Intel, specifically
> in the behavior of the SYSRET instruction (AMD processors are not
> vulnerable).
> So, while the bug has been patched in software by adding some extra
> checks to the hypervisor (Xen.org just released a patch), one should
> still consider it a CPU-level bug. The issue also affects other systems,
> not just Xen (more details in the original advisory). Congrats to Rafal
> for coming unprecedentedly close to "The Holly Grail" of system-level
> exploitation!
> Patching
> ----------
> We have uploaded the patched Xen packages and users of current Qubes
> Beta 3 should upgrade immediately, by running (in Dom0 console):
> sudo qubes-dom0-updates
> ... which should bring Xen v4.1.2-4 packages that are immune to the
> attack. Please reboot your systems afterwards.
> On 12 June 2012 14:13, Joanna Rutkowska <joa...@invisiblethingslab.com> wrote:
>> > Patching
>> > ----------
>> > We have uploaded the patched Xen packages and users of current Qubes
>> > Beta 3 should upgrade immediately, by running (in Dom0 console):
>> > sudo qubes-dom0-updates
> Am I missed something?:
> "bash: qubes-dom0-updates: command not found"
Ah, indeed, qubes-dom0-update has been introduced after Beta3 (and is
available only in the current-testing branch), so, yes, please use
qvm-dom0-update instead...
> Instead I have qvm-dom0-update - but it is said: "No updates avaliable"
Hm... that's strange. What qubes-* packages versions do you have?
> On 06/12/12 19:21, Zrubecz Laszlo wrote:
>> On 12 June 2012 14:13, Joanna Rutkowska <joa...@invisiblethingslab.com> wrote:
>>>> Patching
>>>> ----------
>>>> We have uploaded the patched Xen packages and users of current Qubes
>>>> Beta 3 should upgrade immediately, by running (in Dom0 console):
>>>> sudo qubes-dom0-updates
>> Am I missed something?:
>> "bash: qubes-dom0-updates: command not found"
> Ah, indeed, qubes-dom0-update has been introduced after Beta3 (and is
> available only in the current-testing branch), so, yes, please use
> qvm-dom0-update instead...
>> Instead I have qvm-dom0-update - but it is said: "No updates avaliable"
Perhaps you have cached metadata from some time before publishing fix. You can
try:
qvm-dom0-update --clean
Or just wait some time (for qubes-dom0-current repo, metadata_expire is set to
7 days...).
-- Best Regards / Pozdrawiam,
Marek Marczykowski
Invisible Things Lab
<marma...@invisiblethingslab.com> wrote:
> Perhaps you have cached metadata from some time before publishing fix. You can
> try:
> qvm-dom0-update --clean
> On 12 June 2012 22:31, Marek Marczykowski
> <marma...@invisiblethingslab.com> wrote:
>> Perhaps you have cached metadata from some time before publishing fix. You can
>> try:
>> qvm-dom0-update --clean
> It is not helped... still: "No updates avaliable"
Ah, in the meantime our release signing key expired (in March), we now use new
key. Updates signatures are verified while fetched to dom0, so because you
haven't new key, verification fails.
You need to update qubes-release package first (signed still by old key) and
then the rest of updates:
On 13 June 2012 09:18, Zrubecz Laszlo <m...@zrubi.hu> wrote:
> On 13 June 2012 02:34, Marek Marczykowski
> <marma...@invisiblethingslab.com> wrote:
>> sudo qvm-dom0-update --clean qubes-release
>> sudo qvm-dom0-update --clean
> Now that's working... Update was successfull :)
Somthing wrong... after Updated my system already got a full system
freeze 3 times now :(
I had 2 kind of freeze:
* instant hard reset
No logs, no error messages just a hard reset and qubes reboots...
* screen freeze
I can see my last 'screenshot' but my system not respondst to any kind
of input - just the power button (4 sec and hard off)
No logs, no error messages.
all these are random, and I can't reproduce it... Ther was no problem
at all befor the upgrade...
> On 13 June 2012 09:18, Zrubecz Laszlo <m...@zrubi.hu> wrote:
>> > On 13 June 2012 02:34, Marek Marczykowski
>> > <marma...@invisiblethingslab.com> wrote:
>>> >> sudo qvm-dom0-update --clean qubes-release
>>> >> sudo qvm-dom0-update --clean
>> > Now that's working... Update was successfull :)
> Somthing wrong... after Updated my system already got a full system
> freeze 3 times now :(
> I had 2 kind of freeze:
> * instant hard reset
> No logs, no error messages just a hard reset and qubes reboots...
> * screen freeze
> I can see my last 'screenshot' but my system not respondst to any kind
> of input - just the power button (4 sec and hard off)
> No logs, no error messages.
> all these are random, and I can't reproduce it... Ther was no problem
> at all befor the upgrade...
> Any idea how can I solve this?
And which packages the update brought to your system? Perhaps also some
new Dom0 kernel(s)? Are you sure you still use the same Dom0 kernel as
before?
On 14 June 2012 13:46, Joanna Rutkowska <joa...@invisiblethingslab.com> wrote:
> And which packages the update brought to your system? Perhaps also some
> new Dom0 kernel(s)? Are you sure you still use the same Dom0 kernel as
> before?
not sure... but also not sure which packages contains the dom0 kernel...
These packages was updated:
Jun 13 07:39:22 Updated: qubes-release-1-3.noarch
Jun 13 07:41:14 Updated: 1000:xen-hypervisor-4.1.2-4.qubes.x86_64
Jun 13 07:41:15 Updated: 1000:xen-licenses-4.1.2-4.qubes.x86_64
Jun 13 07:41:17 Updated: 1000:xen-libs-4.1.2-4.qubes.x86_64
Jun 13 07:41:21 Updated: 1000:xen-runtime-4.1.2-4.qubes.x86_64
Jun 13 07:41:23 Updated: 1000:xen-4.1.2-4.qubes.x86_64
> On 13 June 2012 09:18, Zrubecz Laszlo <m...@zrubi.hu> wrote:
>> > On 13 June 2012 02:34, Marek Marczykowski
>> > <marma...@invisiblethingslab.com> wrote:
>>> >> sudo qvm-dom0-update --clean qubes-release
>>> >> sudo qvm-dom0-update --clean
>> > Now that's working... Update was successfull :)
> Somthing wrong... after Updated my system already got a full system
> freeze 3 times now :(
> I had 2 kind of freeze:
> * instant hard reset
> No logs, no error messages just a hard reset and qubes reboots...
> * screen freeze
> I can see my last 'screenshot' but my system not respondst to any kind
> of input - just the power button (4 sec and hard off)
> No logs, no error messages.
> all these are random, and I can't reproduce it... Ther was no problem
> at all befor the upgrade...
> Any idea how can I solve this?
Perhaps the problems were introduced by this commit:
If that doesn't help, then the problem is most likely caused by switch
from Xen 4.1.1 (that was default in Beta 3) and 4.1.2 that we switched
to later, as can be seen in the repo. In that case, well... you're out
of luck, and you would need to debug the issue with the help of
xen-devel people... :/
> On 14 June 2012 14:57, Joanna Rutkowska <joa...@invisiblethingslab.com> wrote:
>> In order to rule this out, can you please try to downgrade your xen
>> packages to 4.1.2-3? See the wiki[1] for howto on downgrading.
> Just did it...
> now waiting for the crash - hopefully this wil not happen :)
You can also compare dom0 kernel output from different xen versions (original
4.1.1, new 4.1.2 and old 4.1.2) - perhaps it differs somehow in active kernel
features. You can find it in /var/log/dmesg (current), dmesg.old (previous)
and /var/log/messages (all together).
-- Best Regards / Pozdrawiam,
Marek Marczykowski
Invisible Things Lab
On 14 June 2012 14:57, Joanna Rutkowska <joa...@invisiblethingslab.com>wrote:
> If that doesn't help, then the problem is most likely caused by switch
> from Xen 4.1.1 (that was default in Beta 3) and 4.1.2 that we switched
> to later, as can be seen in the repo. In that case, well... you're out
> of luck, and you would need to debug the issue with the help of
> xen-devel people... :/
Well, It is happened again with 4.1.2-3 as well :(
Now reverted back to 4.1.1 to be sure this was the reason... Unfortunatelly
i do not have time for such debugging right now :(
On 14 June 2012 09:59, Marek Marczykowski
<marma...@invisiblethingslab.com>wrote:
> You can also compare dom0 kernel output from different xen versions
> (original
> 4.1.1, new 4.1.2 and old 4.1.2) - perhaps it differs somehow in active
> kernel
> features. You can find it in /var/log/dmesg (current), dmesg.old (previous)
> and /var/log/messages (all together).
Just diffed the 3 xen version dmesg's - but no significant diff beside the
memory mappings...
