virsh edit issue

[Patrick Schleizer]

Hi!

Using Qubes Q3 RC1...

Is 'virsh edit' supported in Qubes?

Can you try it please? Just 'sudo virsh edit some-vm-name'. Then try to
modify a minor detail. vcpu number or name or so. virsh won't accept the
changes. Complain with a validation error.

Why do I need it? [1]

Can this be fixed?

Cheers,
Patrick

[1] I am trying to create instructions for Qubes-Whonix how users can
disable clocksource=xen. [a] [b]

This is in context of Qubes-Whonix ticket 'make sure Qubes-Whonix has no
access to clocksource=xen'. [2]

[2] https://phabricator.whonix.org/T389

[a] https://www.whonix.org/wiki/Dev/Qubes#Instructions
[b]
Install a comfortable editor (optional!).

sudo qubes-dom0-update kate

Edit your VM settings. (Feel free to drop 'EDITOR=kate' and/or to use an
editor of your choice.)

sudo EDITOR=kate virsh edit whonix-ws

Find the following block.

<clock offset='utc' adjustment='reset'>
<timer name='tsc' mode='native'/>
</clock>

Replace it with the following.

<clock offset='utc'>
<timer name='rtc' tickpolicy='catchup' track='guest'/>
<timer name='xen' present='no'/>
</clock>

[Marek Marczykowski-Górecki]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256

On Wed, Aug 05, 2015 at 06:31:40PM +0000, Patrick Schleizer wrote:
> Hi!
>
> Using Qubes Q3 RC1...
>
> Is 'virsh edit' supported in Qubes?
>
> Can you try it please? Just 'sudo virsh edit some-vm-name'. Then try to
> modify a minor detail. vcpu number or name or so. virsh won't accept the
> changes. Complain with a validation error.
>
> Why do I need it? [1]
>
> Can this be fixed?

There is indeed some problem with the schema, I'll look into it.
But for now you can skip validation by using `virsh edit
- --skip-validation`. Then you'll learn that 'xen' clocksource name isn't
valid...

Take a look here:
http://libvirt.org/formatdomain.html

There is no clocksource named 'xen' in libvirt domain config. For libxl
supported values are: 'tsc', 'hpet'.

Anyway if I change anything there, VM hangs at startup, the last kernel
message is "Installing Xen timer for CPU 0". If I add "clocksource=tsc"
kernel option, it doesn't work, because "Override clocksource tsc is not
HRT compatible. Cannot swithc while in HRT/NOHZ mode". It works when I
disable both nohz (nohz=off) and HRT (highres=off). But still it is set
after initially using 'xen' clocksource.

Additionally 'tsc' clocksource doesn't work when system is going to
sleep. After suspend "Clocksource tsc unstable (delta = -4375642021 ns)"
is logged and 'tsc' clocksource is no longer available (according to
`/sys/devices/system/clocksource/clocksource0/available_clocksource`).
Unless I disable both NOHZ and HRT; then 'tsc' is still available, even
after suspend.

But disabling NOHZ will have major impact on performance and power
usage. Simple test reveals that idle VM with nohz=on uses ~0.2% CPU, but
the same idle VM with with nohz=off uses over 4% CPU.

So, the question is... how bad for privacy is clocksource 'xen'? It
doesn't prevent Whonix from using sdwdate... Just makes quite complex
attack a little easier - the attacker needs:
- compromise _both_ VMs on the same system (so already have some
suspicious about user identification)
- getting access to raw xen clocksource reads (ok, this would be
probably easy)

Especially using clocksource 'xen' doesn't prevent each VM having
slightly different system time (used by all the applications), so no
direct leaks here.

- --
Best Regards,
Marek Marczykowski-Górecki
Invisible Things Lab
A: Because it messes up the order in which people normally read text.
Q: Why is top-posting such a bad thing?
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1

iQEcBAEBCAAGBQJVwn3IAAoJENuP0xzK19csLX4H/2wtEXHriHbnap5mo6qtcLXi
9N+JbahxYtGHCL1T/F+93td5lTxi3BvbthLfKgjZZkTCrOqFh5tu09a1J3HhFSL3
VGp5GTKUf+eXG+B5xfzIEwi9SD5TmyfWOhJCLnZTISjkQwLBZDbPbzRVllEtpQ/N
y5v60VAUiwrXCIIjvPbUJ5vznCq/+6rWbiIk6DBkRr5gWLqJe+hGKVmnWcyCOY4e
KiAnUEBaW3beSXNIrzxVySCouqDJz7EqFi0ZFNa0/TsiSijBPV3TcHLnbh/J5xp/
vRmdb8MtupL6R8xB2OvAGEeiYAaG+a8JdrIsfr+oA57RxofYkPtC1FmhW7xTxrU=
=Bxyj
-----END PGP SIGNATURE-----

[Marek Marczykowski-Górecki]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256

On Wed, Aug 05, 2015 at 11:19:04PM +0200, Marek Marczykowski-Górecki wrote:
> On Wed, Aug 05, 2015 at 06:31:40PM +0000, Patrick Schleizer wrote:
> > Hi!
> >
> > Using Qubes Q3 RC1...
> >
> > Is 'virsh edit' supported in Qubes?
> >
> > Can you try it please? Just 'sudo virsh edit some-vm-name'. Then try to
> > modify a minor detail. vcpu number or name or so. virsh won't accept the
> > changes. Complain with a validation error.
> >
> > Why do I need it? [1]
> >
> > Can this be fixed?
>
> There is indeed some problem with the schema, I'll look into it.
> But for now you can skip validation by using `virsh edit

> --skip-validation`. Then you'll learn that 'xen' clocksource name isn't
> valid...

BTW You'll not achieve anything with virsh edit anyway. Libvirt domain
is redefined at each startup based on VM configuration. You can start a
VM with custom libvirt config with qvm-start --custom-config option.
This would of course ignore all the VM settings.
This is how I achieved results below.

iQEcBAEBCAAGBQJVwoE5AAoJENuP0xzK19csC4cH/jjphjzqAacYrpsteu3PmhzS
7WsdeIX4jpPophLj2/8vzhrDizqj+LR1X+DXDvGeHnH4lslpltF8jHfRLGdHyH5F
tXvnJoPUPPAxHDZc5M/j5+3cXx2u8sNlH+yk1rpw6uSAAHWtekyO07U5Owqx5YvQ
VQKPC27vGLo3+MQGg9cpZPkDqqvxg4wn7F7pTX4IiGp7mxUnU6FztDLlmWq1RRG6
ZdFu2GF2AnWieRdR2uVOR3m757NnErB7xSE+myXdYgyhIkiT9QkT4UZvBm4lNpug
AoAF5IYVIyKJ5NsiyK90rtdQdtbdVNTd9sQTXXillz+rJ10wlOs3vs4LEfWTygk=
=b9yH
-----END PGP SIGNATURE-----

[Patrick Schleizer]

Marek Marczykowski-Górecki:

> On Wed, Aug 05, 2015 at 06:31:40PM +0000, Patrick Schleizer wrote:

> So, the question is... how bad for privacy is clocksource 'xen'? It
> doesn't prevent Whonix from using sdwdate... Just makes quite complex
> attack a little easier - the attacker needs:
> - compromise _both_ VMs on the same system (so already have some
> suspicious about user identification)

No need to compromise multiple VMs.

In most cases - hard to avoid all cases - any passive ISP level
adversary watching traffic can read the clients local clock. It's leaked
on many levels. For example TCP timestamps. Also outgoing NTP queries
leak it. See this list of local clock leaks. [1]

The second step for an attacker is to compromise a
Qubes-Whonix-Workstation. Then read xen clocksource. Correlate those
times. Attack succeeded.

> Especially using clocksource 'xen' doesn't prevent each VM having
> slightly different system time (used by all the applications), so no
> direct leaks here.

Yes. It's orthogonal. Still very much worth fixing.

Cheers,
Patrick

[1] https://www.whonix.org/wiki/Dev/TimeSync#Local_Clock_Leaks

[Marek Marczykowski-Górecki]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256

On Wed, Aug 05, 2015 at 09:58:56PM +0000, Patrick Schleizer wrote:
> Marek Marczykowski-Górecki:
> > On Wed, Aug 05, 2015 at 06:31:40PM +0000, Patrick Schleizer wrote:
>
> > So, the question is... how bad for privacy is clocksource 'xen'? It
> > doesn't prevent Whonix from using sdwdate... Just makes quite complex
> > attack a little easier - the attacker needs:
> > - compromise _both_ VMs on the same system (so already have some
> > suspicious about user identification)
>
> No need to compromise multiple VMs.
>
> In most cases - hard to avoid all cases - any passive ISP level
> adversary watching traffic can read the clients local clock. It's leaked
> on many levels. For example TCP timestamps. Also outgoing NTP queries
> leak it. See this list of local clock leaks. [1]

Ok, but this doesn't leak host time, only VM system time, which can be
different for every VM, regardless of kernel clocksource used. In Whinix
case it would be whonix-gw system time, right? It shouldn't be possible
to get system time of whonix-ws VM just by watching traffic at ISP
level.

> The second step for an attacker is to compromise a
> Qubes-Whonix-Workstation. Then read xen clocksource. Correlate those
> times. Attack succeeded.

I don't see how this helps correlation with other VMs (including
whonix-gw). You still don't know what is the clocksource vs system time
offsets in different VMs.

Or are you talking about tracking a single VM across its different exit
nodes? In such a case ANY whonix-ws compromise would be enough. But if
the user sufficiently divided activities across VMs, it shouldn't help
with correlating multiple identifies, especially with the real one.

> > Especially using clocksource 'xen' doesn't prevent each VM having
> > slightly different system time (used by all the applications), so no
> > direct leaks here.
>
> Yes. It's orthogonal. Still very much worth fixing.
>
> Cheers,
> Patrick
>
> [1] https://www.whonix.org/wiki/Dev/TimeSync#Local_Clock_Leaks

- --
Best Regards,
Marek Marczykowski-Górecki
Invisible Things Lab
A: Because it messes up the order in which people normally read text.
Q: Why is top-posting such a bad thing?
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1

iQEcBAEBCAAGBQJVwp4aAAoJENuP0xzK19csXGEH/25Zu2wcv5F+oMZV24eqE2pr
lgzFQ8d3z79B8FXkI0L/X6UG77NyYxEG8sw0xDVQ1Ht2abOC6X0EMIlqnIa9M+pB
Qlsd1yPMTMf8GDqAjXuKpCOLR6HBEsFnX220t1nGQUGlEFCpd1A0ap6lMOQJC7T3
3xoKXG5uXvbVRepubr6Vp7QcirBY5DCQL7UJxdOekFwmHfZ1nmk6MpkuIZv4CEf/
nU8+fuUI9nvZpQBe/J1UNewFZ/JQWBtIv1lVFp9tFx9ro44N//UoPajNMxNZc4SR
Dk8AZPzkC9Gyyr7rjiRyq55UlZGvZ4jNF8uZhKYgt4RPjih6Ou6zOE2YWMdDhA0=
=HSck
-----END PGP SIGNATURE-----

[Patrick Schleizer]

Marek Marczykowski-Górecki:
> On Wed, Aug 05, 2015 at 09:58:56PM +0000, Patrick Schleizer wrote:
>> Marek Marczykowski-Górecki:
>>> On Wed, Aug 05, 2015 at 06:31:40PM +0000, Patrick Schleizer
>>> wrote:
>
>>> So, the question is... how bad for privacy is clocksource 'xen'?
>>> It doesn't prevent Whonix from using sdwdate... Just makes quite
>>> complex attack a little easier - the attacker needs: - compromise
>>> _both_ VMs on the same system (so already have some suspicious
>>> about user identification)
>
>> No need to compromise multiple VMs.
>
>> In most cases - hard to avoid all cases - any passive ISP level
>> adversary watching traffic can read the clients local clock. It's
>> leaked on many levels. For example TCP timestamps. Also outgoing
>> NTP queries leak it. See this list of local clock leaks. [1]
>
> Ok, but this doesn't leak host time, only VM system time, which can
> be different for every VM, regardless of kernel clocksource used. In

> Whonix case it would be whonix-gw system time, right? It shouldn't be

> possible to get system time of whonix-ws VM just by watching traffic
> at ISP level.

Yes, not dom0 host time. Only VM system time. But those times a very,
very similar. At most one second difference. Or less. In dom0, see:
date ; qvm-run --pass-io sys-net date

whonix-ws VM's system time could indeed not be obtained by watching
traffic at ISP level.

The threat model is here, that whonix-ws gets compromised up to local
root code execution. Then the attacker uses the local exploit, reads
whonix-ws's clocksource xen.

I imagine clocksource xen is similar to kvmclock...

Quote https://rwmj.wordpress.com/tag/kvmclock/
> [...] kvmclock or KVM pvclock lets guests read the host’s wall clock time.
> It’s really very simple: the guest sets aside a page of its RAM and
> asks the host to write time into that page (using an MSR). The host
> writes a structure containing the current time to this page — in
> theory the host updates this page constantly, [...]

The host's clock in Qubes case is dom0. No networking. But since as far
I know, dom0 gets its time from the ClockVM, which is currently net-vm,
which uses NTP... And NTP leaks the local clock to ISP level observers.
So [approximate] dom0's clock is known to ISP level observers.

My assumption here is, that clocksource xen equals [or very similar]
'dom0 `date`'. Also based on [1].

In summary: locally compromised whonix-ws time leak through clocksource
xen + ISP level observer time leak through NTP leak in net-vm ->
correlate times attack succeeds.

>> The second step for an attacker is to compromise a
>> Qubes-Whonix-Workstation. Then read xen clocksource. Correlate
>> those times. Attack succeeded.
>
> I don't see how this helps correlation with other VMs (including
> whonix-gw). You still don't know what is the clocksource vs system
> time offsets in different VMs.

Do you know how to read the current value of clocksource xen as a [root]
user?

> Or are you talking about tracking a single VM across its different
> exit nodes?

No.

Cheers,
Patrick

[1] Re: [Xen-users] clocksource xen documentation?
http://lists.xen.org/archives/html/xen-users/2015-08/msg00019.html

[Marek Marczykowski-Górecki]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256

Ok. So we need to make sure that all of whonix-gw, whonix-ws, dom0 uses
slightly different time, right? Even if whonix-ws would be compromised,
and the attacker gets access to dom0 time, it wouldn't give much hint
about whonix-gw time.

> The threat model is here, that whonix-ws gets compromised up to local
> root code execution. Then the attacker uses the local exploit, reads
> whonix-ws's clocksource xen.
>
> I imagine clocksource xen is similar to kvmclock...
>
> Quote https://rwmj.wordpress.com/tag/kvmclock/
> > [...] kvmclock or KVM pvclock lets guests read the host’s wall clock time.
> > It’s really very simple: the guest sets aside a page of its RAM and
> > asks the host to write time into that page (using an MSR). The host
> > writes a structure containing the current time to this page — in
> > theory the host updates this page constantly, [...]
>
> The host's clock in Qubes case is dom0. No networking. But since as far
> I know, dom0 gets its time from the ClockVM, which is currently net-vm,
> which uses NTP... And NTP leaks the local clock to ISP level observers.
> So [approximate] dom0's clock is known to ISP level observers.

The point is "approximate". ISP level observers can get some
approximation of whonix-gw VM (or probably any non-anon VMs). If that's
slightly different from dom0 time, it shouldn't be a big problem.
Given the amount of tor users, it needs to be very accurate to give
meaningful correlation. If most of systems have time properly
synchronized (say, +- 10s) and our "slightly" is also somewhere about
this range, this wouldn't give much correlation. I.e. it would be really
hard to distinguish if it is the same system (and the time difference is
because whonix time sync approach), or this is another system.

This all means that we should implement Whonix time sync approach also
for non-Whonix VMs, or at least dom0. So dom0 clock would be slightly
different than all the timestamps leaked from clearnet VMs.

> My assumption here is, that clocksource xen equals [or very similar]
> 'dom0 `date`'. Also based on [1].

Actually based on that response, it doesn't provide dom0 time, just
"ticks" counter, from system boot.

The other interface ("wallclock interface") would provide such time,
though. And it isn't configurable. But not sure if it is used anywhere
in Linux by default (probably besides initial system time set).

> In summary: locally compromised whonix-ws time leak through clocksource
> xen + ISP level observer time leak through NTP leak in net-vm ->
> correlate times attack succeeds.
>
> >> The second step for an attacker is to compromise a
> >> Qubes-Whonix-Workstation. Then read xen clocksource. Correlate
> >> those times. Attack succeeded.
> >
> > I don't see how this helps correlation with other VMs (including
> > whonix-gw). You still don't know what is the clocksource vs system
> > time offsets in different VMs.
>
> Do you know how to read the current value of clocksource xen as a [root]
> user?

No, I don't know any tool for that.
Surely doable (but hard) using debugger + /dev/kmem or /dev/mem.

- From dom0 you can get some information from 'xl debug-key s', then
access hypervisor log using 'xl dmesg'.

- --
Best Regards,
Marek Marczykowski-Górecki
Invisible Things Lab
A: Because it messes up the order in which people normally read text.
Q: Why is top-posting such a bad thing?
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1

iQEcBAEBCAAGBQJVw2uUAAoJENuP0xzK19csYv8IAJa2Y3GUAA1pmxPUYv895zJi
cWE+FZ7uOXY2tsQbp9eFbYhKioWunLHu7EJBP/yK98VA9kBAaKQk9PbK6P1a8ino
IVwSTXDSWayHVM4aMsbGCyTn7jDrObbYi6Ai0iI1S4jCSFbgULAwbXJYTmkmXG1q
CcCJnyJAGtAIlo3gz1Aa7F+6JvwpXXJNlVKNpvRbaUFc1ZpqnY7x3Ns20tcSI5ng
cwoFy7jpx+flYgiORnCW9eD2c4kUzycgegLGZOqu2OF0V6jPtcScT3g8N9d7DM9Y
XxszVbRibc6rybW0z7pvk4RkktSc9lJyrLgX1NtbKwSmzywCbafclXb2HDSGNHw=
=m7Wy
-----END PGP SIGNATURE-----

[Unman]

On Thu, Aug 06, 2015 at 04:13:40PM +0200, Marek Marczykowski-G??recki wrote:
> On Thu, Aug 06, 2015 at 12:53:26PM +0000, Patrick Schleizer wrote:

> > Marek Marczykowski-G??recki:

> > > On Wed, Aug 05, 2015 at 09:58:56PM +0000, Patrick Schleizer wrote:

> > >> Marek Marczykowski-G??recki:

> > > [...] kvmclock or KVM pvclock lets guests read the host???s wall clock time.
> > > It???s really very simple: the guest sets aside a page of its RAM and

> > > asks the host to write time into that page (using an MSR). The host

> > > writes a structure containing the current time to this page ??? in

> > > theory the host updates this page constantly, [...]
> >
> > The host's clock in Qubes case is dom0. No networking. But since as far
> > I know, dom0 gets its time from the ClockVM, which is currently net-vm,
> > which uses NTP... And NTP leaks the local clock to ISP level observers.
> > So [approximate] dom0's clock is known to ISP level observers.
>
> The point is "approximate". ISP level observers can get some
> approximation of whonix-gw VM (or probably any non-anon VMs). If that's
> slightly different from dom0 time, it shouldn't be a big problem.
> Given the amount of tor users, it needs to be very accurate to give
> meaningful correlation. If most of systems have time properly
> synchronized (say, +- 10s) and our "slightly" is also somewhere about
> this range, this wouldn't give much correlation. I.e. it would be really
> hard to distinguish if it is the same system (and the time difference is
> because whonix time sync approach), or this is another system.
>
> This all means that we should implement Whonix time sync approach also
> for non-Whonix VMs, or at least dom0. So dom0 clock would be slightly
> different than all the timestamps leaked from clearnet VMs.
>

There's a lot going on in this thread.

I'm somewhat concerned by that last comment. Putting on my corporate hat
it will be a really hard sell if the VMs dont have the same time. I'm
thinking here of trying to pitch qubes to lawyers for example. Most
corporates spend a lot of time trying to get their machines with the
same time, so I cant see them readily accepting a system where the
individual parts are all at different times.

So imo if qubes is to make an impact it's important not to lose the
synced times.

If the comment was restricted to qubes-whonix, then fair enough. But
it will restrict qubes-whonix takeup if installing the whonix
packages breaks the synced times for non-Whonix VMs.

unman

[Patrick Schleizer]

Unman:

I agree. dom0 and non-anonymous VM's clocks should be as synced, correct
and secure as those can be.

But those should differ from the clocks within Whonix VMs. Fortunately,
we don't have a hard incompatibility here. We can spoof Whonix VMs
initial time offset. [0]

KVM has:
<clock offset='variable' adjustment='123456' basis='utc'>

VirtualBox has biostimeoffset.

Hopefully Xen / Qubes can do this as well.

Cheers,
Patrick

[0]
https://www.whonix.org/wiki/Advanced_Security_Guide#Spoof_the_Initial_Virtual_Hardware_Clock_Offset

[Manuel Amador (Rudd-O)]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256

On 08/05/2015 02:58 PM, Patrick Schleizer wrote:
> No need to compromise multiple VMs. > > In most cases - hard to avoid all cases - any passive ISP level >
adversary watching traffic can read the clients local clock. It's leaked
> on many levels. For example TCP timestamps. Also outgoing NTP queries
> leak it. See this list of local clock leaks. [1] > > The second step
for an attacker is to compromise a > Qubes-Whonix-Workstation. Then read
xen clocksource. Correlate those > times. Attack succeeded. >

What is the significance of such an attack?

- --
Rudd-O
http://rudd-o.com/
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2

iQIcBAEBCAAGBQJVzuZAAAoJEFmZwbV7vYQ2wHgP+wYdrhSlD+sdNN00uT4A4E0L
/+VUV6i7EdYNEY+JwGmBG8LOJXAlxjmxlOAU9j1vAn8J5V4/tqO6WBHo+DtRFWJb
VKw7QuiruuzFBb5f0zQfu/hH6lriIiYNoUWqLllRM9/xJTWeu1m4vCRGKZic3NeY
Eqp9JTEgvdcVImBVftbwSXv844IqwPxzXi859KHvaLeY+g7hJbkSNJsFFNmCWkM/
kHTcnwWud40jzVi2hD5spqH354wv6K5S4TCV1zlFslJAa6+gSyY6pOkGskL8Z7EA
Gxi1ww0Nf8Nro8Kz5o40NdiYPRjUVvOkH08QzE2yQtMAMSWfO8QI3T2eVU1x3CEr
kDFiORv9oUgu5GBIcpYn4ZGfjnD5TZnhFW5zm5ORQRz9o6wYhMMLzKLGFCpn6ViW
FOy85PSBIhTfkdmZfwi5N+6rYE1mTKr0o3M8ZbeqWpgkDVcF1H+dEp5mxbp1pdgt
4gmOr+Eo17x2En+3iHFKAxZgXAGCyrsSl3ZZ2pHVVt7LMlHCi3rhy0RhG7Pn6a5t
YgiekCsQpaKVbv5eGYTHNscQzJ91/vjWJhE87HYNVD794EB2r6IH6hdIZZ8VeuM+
aYeG+hJ0+OUTpo6GIKLhg9EUk8BBkrgmY5X5BUu0ha29oW4qv82I5cHxXT34WG14
rt9eYJLJf05rGfkoa4Td
=796B
-----END PGP SIGNATURE-----

[Patrick Schleizer]

Manuel Amador (Rudd-O):

>
> On 08/05/2015 02:58 PM, Patrick Schleizer wrote:
>> No need to compromise multiple VMs. > > In most cases - hard to avoid all cases - any passive ISP level >
> adversary watching traffic can read the clients local clock. It's leaked
>> on many levels. For example TCP timestamps. Also outgoing NTP queries
>> leak it. See this list of local clock leaks. [1] > > The second step
> for an attacker is to compromise a > Qubes-Whonix-Workstation. Then read
> xen clocksource. Correlate those > times. Attack succeeded. >
>
> What is the significance of such an attack?

The context is anonymity.

The significance is deanonymization.

Cheers,
Patrick

[Patrick Schleizer]

Marek Marczykowski-Górecki:

Yes.

> Even if whonix-ws would be compromised,
> and the attacker gets access to dom0 time, it wouldn't give much hint
> about whonix-gw time.

Yes, but more importantly, not give any hints about net-vm, personal vm
etc. time. [Because those easily leak the system time.]

>> The threat model is here, that whonix-ws gets compromised up to local
>> root code execution. Then the attacker uses the local exploit, reads
>> whonix-ws's clocksource xen.
>
>> I imagine clocksource xen is similar to kvmclock...
>
>> Quote https://rwmj.wordpress.com/tag/kvmclock/
>>> [...] kvmclock or KVM pvclock lets guests read the host’s wall clock time.
>>> It’s really very simple: the guest sets aside a page of its RAM and
>>> asks the host to write time into that page (using an MSR). The host
>>> writes a structure containing the current time to this page — in
>>> theory the host updates this page constantly, [...]
>
>> The host's clock in Qubes case is dom0. No networking. But since as far
>> I know, dom0 gets its time from the ClockVM, which is currently net-vm,
>> which uses NTP... And NTP leaks the local clock to ISP level observers.
>> So [approximate] dom0's clock is known to ISP level observers.
>
> The point is "approximate". ISP level observers can get some
> approximation of whonix-gw VM (or probably any non-anon VMs). If that's
> slightly different from dom0 time, it shouldn't be a big problem.
> Given the amount of tor users, it needs to be very accurate to give
> meaningful correlation. If most of systems have time properly
> synchronized (say, +- 10s) and our "slightly" is also somewhere about
> this range, this wouldn't give much correlation. I.e. it would be really
> hard to distinguish if it is the same system (and the time difference is
> because whonix time sync approach), or this is another system.

In anonymity we don't just care about about correlations. Also about
estimations. Anonymity set reduction. Partitioning users is already a
success. Because it may be combined with other guesses. No direct proof
is required. From some countries, there are just a few 100 users or
less. If there is already a lead by country, then combined with these
time related issues, things begin to really matter.

> This all means that we should implement Whonix time sync approach also
> for non-Whonix VMs, or at least dom0. So dom0 clock would be slightly
> different than all the timestamps leaked from clearnet VMs.

That would work. But as unman pointed out, that may not a solution
applied by default for everyone. Below I describe an alternative "spoof
the initial virtual hardware clock offset" that will hopefully work out.

>> My assumption here is, that clocksource xen equals [or very similar]
>> 'dom0 `date`'. Also based on [1].
>
> Actually based on that response, it doesn't provide dom0 time, just
> "ticks" counter, from system boot.

Hm. If it's just ticks, then what is clocksource xen's difference from
clocksource tsc?

Can you please check it's just "ticks" from the Xen code?

http://lxr.free-electrons.com/source/arch/x86/xen/time.c#L155

struct pvclock_vcpu_time_info contains both tsc_timestamp and system_time:

http://lxr.free-electrons.com/source/arch/arm/include/asm/xen/interface.h#L63

> The other interface ("wallclock interface") would provide such time,
> though. And it isn't configurable. But not sure if it is used anywhere
> in Linux by default (probably besides initial system time set).

Yes. Must be so. Otherwise with "ticks" alone the VM had no chance to
figure out the current time. And this is a problem, because we don't
want the VM to know dom0's time.

Rather than modifying dom0's clock, we could perhaps spoof the initial
virtual hardware clock offset? It's possible for KVM, as per:
https://libvirt.org/formatdomain.html#elementsTime

Something like this:

<clock offset='variable' adjustment='123456' basis='utc'>

That's also why I wanted to experiment with virsh xml edits to see if it
works with Xen as well.

>> In summary: locally compromised whonix-ws time leak through clocksource
>> xen + ISP level observer time leak through NTP leak in net-vm ->
>> correlate times attack succeeds.
>
>>>> The second step for an attacker is to compromise a
>>>> Qubes-Whonix-Workstation. Then read xen clocksource. Correlate
>>>> those times. Attack succeeded.
>>>
>>> I don't see how this helps correlation with other VMs (including
>>> whonix-gw). You still don't know what is the clocksource vs system
>>> time offsets in different VMs.
>
>> Do you know how to read the current value of clocksource xen as a [root]
>> user?
>
> No, I don't know any tool for that.
> Surely doable (but hard) using debugger + /dev/kmem or /dev/mem.

I asked on the xen-users list.
'How to read clocksource xen as a [root] user?':
http://lists.xen.org/archives/html/xen-users/2015-08/msg00020.html
Paraphrased answer: "no, not easily possible".

Cheers,
Patrick

[Marek Marczykowski-Górecki]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256

On Wed, Feb 17, 2016 at 10:53:03PM +0000, Patrick Schleizer wrote:
> Marek Marczykowski-Górecki:

> > Anyway if I change anything there, VM hangs at startup, the last kernel
> > message is "Installing Xen timer for CPU 0". If I add "clocksource=tsc"
> > kernel option, it doesn't work, because "Override clocksource tsc is not
> > HRT compatible. Cannot swithc while in HRT/NOHZ mode". It works when I
> > disable both nohz (nohz=off) and HRT (highres=off). But still it is set
> > after initially using 'xen' clocksource.
> >
> > Additionally 'tsc' clocksource doesn't work when system is going to
> > sleep. After suspend "Clocksource tsc unstable (delta = -4375642021 ns)"
> > is logged and 'tsc' clocksource is no longer available (according to
> > `/sys/devices/system/clocksource/clocksource0/available_clocksource`).
> > Unless I disable both NOHZ and HRT; then 'tsc' is still available, even
> > after suspend.
> >
> > But disabling NOHZ will have major impact on performance and power
> > usage. Simple test reveals that idle VM with nohz=on uses ~0.2% CPU, but
> > the same idle VM with with nohz=off uses over 4% CPU.
>

> qvm-prefs -s temp kernelopts "nopat clocksource=tsc nohz=off highres=off"
>
> Still,
>
> cat /sys/devices/system/clocksource/clocksource0/current_clocksource
>
> is xen, not tsc.
>
> cat /proc/cmdline
>
> shows, that the kernel really has been booted with these kernel options.
>
> How to set clocksource from xen to tsc?

Simply writing to
/sys/devices/system/clocksource/clocksource0/current_clocksource ?

But still VM will be able to switch back to 'xen'.

- --
Best Regards,
Marek Marczykowski-Górecki
Invisible Things Lab
A: Because it messes up the order in which people normally read text.
Q: Why is top-posting such a bad thing?
-----BEGIN PGP SIGNATURE-----

Version: GnuPG v2

iQEcBAEBCAAGBQJWxPxxAAoJENuP0xzK19csv5AH/A/k8XaiZ/ZJUEo2ZdcXsbYt
BRvsypNkhUO08jYJ1ZtJfcOYMQvxQd9c0YFFwBdHlubcOHdEQPA3m/e7Z0a9H/CF
86nqxCpOIIu3aNAl9ELv9asor6+E/7p8T/BrjImidYcsLE16GmM6TyUILy+CFVjZ
YO10galwqNiJxzXAiNTHyZoLHgbCCtg4fVcixMD38ZC5bRcDoceRU+M34FI6aBxA
ziREzEq5+fP2JSnxH8+rwoK7bSFlYIQatnmpDlPKH28MEiMHpyo2LeSqlHzg3fZx
slpYvBusvapf7OCvaDyGhit1QwQCPXIoSMiANIkJnq495CKrCP2GoUx5gKjfZOg=
=NfPD
-----END PGP SIGNATURE-----

[Patrick Schleizer]

Marek Marczykowski-Górecki:

> Anyway if I change anything there, VM hangs at startup, the last kernel
> message is "Installing Xen timer for CPU 0". If I add "clocksource=tsc"
> kernel option, it doesn't work, because "Override clocksource tsc is not
> HRT compatible. Cannot swithc while in HRT/NOHZ mode". It works when I
> disable both nohz (nohz=off) and HRT (highres=off). But still it is set
> after initially using 'xen' clocksource.
>
> Additionally 'tsc' clocksource doesn't work when system is going to
> sleep. After suspend "Clocksource tsc unstable (delta = -4375642021 ns)"
> is logged and 'tsc' clocksource is no longer available (according to
> `/sys/devices/system/clocksource/clocksource0/available_clocksource`).
> Unless I disable both NOHZ and HRT; then 'tsc' is still available, even
> after suspend.
>
> But disabling NOHZ will have major impact on performance and power
> usage. Simple test reveals that idle VM with nohz=on uses ~0.2% CPU, but
> the same idle VM with with nohz=off uses over 4% CPU.

qvm-prefs -s temp kernelopts "nopat clocksource=tsc nohz=off highres=off"

Still,

cat /sys/devices/system/clocksource/clocksource0/current_clocksource

is xen, not tsc.

cat /proc/cmdline

shows, that the kernel really has been booted with these kernel options.

How to set clocksource from xen to tsc?

Cheers,
Patrick

[Patrick Schleizer]

Marek Marczykowski-Górecki:

That's working. And a short test with suspend (pause) and resume it
seems to be stable also.

> But still VM will be able to switch back to 'xen'.

Right, however just switching to tsc alone is also worthwhile as this
should help making time correlation attacks harder. It should make the
VM clock more independent. The clock should be slightly differently
ticking then, right? Therefore making measurements at the remote
(javascript time leaks etc) less useful.

Totally removing clocksource xen would require solving
https://phabricator.whonix.org/T389 which would probably mean, "re-add
the independent_clocksource feature to xen", which requires knowledge of
C and writing a xen patch. Is it realistic for Qubes to do that some day?

Cheers,
Patrick

[Marek Marczykowski-Górecki]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256

Ok, IMHO worth doing that from some Whonix-specific startup script.

> Totally removing clocksource xen would require solving
> https://phabricator.whonix.org/T389 which would probably mean, "re-add
> the independent_clocksource feature to xen", which requires knowledge of
> C and writing a xen patch. Is it realistic for Qubes to do that some day?

I'm afraid it isn't. At least not with current quite small team.

- --
Best Regards,
Marek Marczykowski-Górecki
Invisible Things Lab
A: Because it messes up the order in which people normally read text.
Q: Why is top-posting such a bad thing?
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2

iQEcBAEBCAAGBQJWyLCXAAoJENuP0xzK19cs/GMIAIAQYWeO0he7ZdlHgWjbZ3Fx
wKa+uLh0siS8BkQEDSa97gMD4qdXXqPgrxhsA++xllbklpwTVHLG3yoI/ZGk09Zk
wm6f60UN7pVJ+jQPaSGbzBHfA5f4g1GSCflMBIF6V6jhipMF09h0HDCj839C7KCb
jfkb746o/XQ4wyaThfsJJfpJEtp3RvY83kPNJ1joLWhg8oWHD+MdXd79z1Jx1Uw5
t8AKbzrjuFeiLpPKV8ROaU5WIUFJVZSA3d3cFgGHt7Ca+8lRlghfCWV0rVpvfj5q
lEOqOgen7lCsMS6MNlL1vPkOYRfjIr4+TDQC8KEVTktLRkzoSDn3Q95SFFQPjXM=
=KdpC
-----END PGP SIGNATURE-----

[Patrick Schleizer]

Marek Marczykowski-Górecki:

Experimented with this. Will unfortunately probably not work well. Weird
kernel message:

[22559.779482] timekeeping watchdog: Marking clocksource 'tsc' as
unstable, because the skew is too large:
[22559.779494] 'xen' wd_now: 2745b95736a1 wd_last: 27448e384015 mask:
ffffffffffffffff
[22559.779498] 'tsc' cs_now: 15e1920f6 cs_last: 2fa7ae81384a mask:
ffffffffffffffff

user@temp:~$ cat
/sys/devices/system/clocksource/clocksource0/available_clocksource
xen

user@temp:~$ cat
/sys/devices/system/clocksource/clocksource0/current_clocksource
xen

Therefore not implemented.

Cheers,
Patrick

[Marek Marczykowski-Górecki]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256

Did you get this after resume from suspend? Maybe related to
https://github.com/QubesOS/qubes-issues/issues/1764 ?

> user@temp:~$ cat
> /sys/devices/system/clocksource/clocksource0/available_clocksource
> xen
>
> user@temp:~$ cat
> /sys/devices/system/clocksource/clocksource0/current_clocksource
> xen
>
> Therefore not implemented.
>
> Cheers,
> Patrick

- --
Best Regards,
Marek Marczykowski-Górecki
Invisible Things Lab
A: Because it messes up the order in which people normally read text.
Q: Why is top-posting such a bad thing?
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2

iQEcBAEBCAAGBQJWz3GXAAoJENuP0xzK19csxHoH/3pO4jLku267iWdTO3hjQLr8
O9eSg7W69fUmHtOc0Krib7eGaX6I1vIgoZNxcDzxJyqA1MWZYd5JLQYgjmjyL67g
8j/CD3ETMNqHP6fjle00ZsDU7HunNP7WGcWwdRl6zjwo28PwsQ0/I6P3yOLUTkVq
xVuagKDGs1BIaOIOFF7WtgJ9XKEit0PD1ZLbHWwxmmWBgQk5WUsfZVHVdsFJw3Pt
fGw1TwQ2jJ2I4gwW0N8C1k8FgvFaeeTHLCSwtkH1A7cTZ+mZGICeWG9+3NLYeaHo
lO5DZe6lxGry4qFRSzsa66LN6LjpB7at53qoTRp6UDkcwWtUPCIEDsDyK5CJtKU=
=DxFN
-----END PGP SIGNATURE-----

[Marek Marczykowski-Górecki]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256

On Thu, Feb 25, 2016 at 10:26:46PM +0100, Marek Marczykowski-Górecki wrote:
> On Thu, Feb 25, 2016 at 09:22:11PM +0000, Patrick Schleizer wrote:
> > Marek Marczykowski-Górecki:
> > > On Sat, Feb 20, 2016 at 06:12:25PM +0000, Patrick Schleizer wrote:
> > >> Marek Marczykowski-Górecki:
> > >>> On Wed, Feb 17, 2016 at 10:53:03PM +0000, Patrick Schleizer wrote:
> > >>>> Marek Marczykowski-Górecki:

> > >>> But still VM will be able to switch back to 'xen'.
> > >
> > >> Right, however just switching to tsc alone is also worthwhile as this
> > >> should help making time correlation attacks harder. It should make the
> > >> VM clock more independent. The clock should be slightly differently
> > >> ticking then, right? Therefore making measurements at the remote
> > >> (javascript time leaks etc) less useful.
> > >
> > > Ok, IMHO worth doing that from some Whonix-specific startup script.
> >
> > Experimented with this. Will unfortunately probably not work well. Weird
> > kernel message:
> >
> > [22559.779482] timekeeping watchdog: Marking clocksource 'tsc' as
> > unstable, because the skew is too large:
> > [22559.779494] 'xen' wd_now: 2745b95736a1 wd_last: 27448e384015 mask:
> > ffffffffffffffff
> > [22559.779498] 'tsc' cs_now: 15e1920f6 cs_last: 2fa7ae81384a mask:
> > ffffffffffffffff
>
> Did you get this after resume from suspend? Maybe related to
> https://github.com/QubesOS/qubes-issues/issues/1764 ?

It still isn't clear to me if clocksource 'xen' provides absolute
Xen/dom0 timestamp value to the VM. But from the userspace point of view, it
looks like only some counter. Also clocksource 'xen' seems to be a
better choice for solving system suspend issue.

Simple experiment shows that clocksource 'tsc' after host suspend is
totally broken - when VM has is set, VM's clock is frozen after resume
(until clocksource switched to 'xen' - either manually, or automatically
as you've seen above).

But, even with clocksource 'xen', every VM clock seems to be independent
- - if the time was desynchronized before suspend, it is still such after
resume, with the same offset. With an exception, that it seems like the
VM time was frozen for the time of system sleep.

Not sure if such abstraction is implemented in Xen (which would be
good), or in Linux (which could be bypassed by an attacker with
kernel-level access).

And back to the suspend issue - is it possible for Whonix gateway (and
workstation) to obtain time for itself, without dom0 cooperation? So
only some qrexec call to notify that suspend happened would be enough.
But since tor can't connect with large clock skew (or it can?), I guess
not.

In the later case, dom0 will need to provide slightly randomized time to
the VM. How large that difference needs to be to prevent time
correlations? bootclockrandomization[1] seems to use +/- 180s
Currently dom0 (in qvm-sync-clock -> qubes.SetDateTime) gives time with
precision up to a second (no milliseconds/nanoseconds). I guess it is
too precise.

Do you have an idea how this should be implemented? I guess there should
be dom0 qrexec service like qubes.GetRandomizedTime, which would give
such randomized time. This service could be called in response to
"system resume" notification from dom0 (as you've already written
somewhere).

BTW why 0-5 is excluded in bootclockrandomization? I think this leaks
some data. For example if attacker have already some selected set of
data to correlate (like "all Tor users in area X"), he/she can easily
further narrow the search by eliminating those with time almost in sync
(+/-5sec). Given popularity of NTP, which (when works) is pretty good at
keeping the time in sync, this allows to easily eliminate all such
users, leaving probably only those using Whonix, or maybe even more
precise correlation...

[1]
https://github.com/Whonix/bootclockrandomization/blob/master/usr/share/bootclockrandomization/start

- --
Best Regards,
Marek Marczykowski-Górecki
Invisible Things Lab
A: Because it messes up the order in which people normally read text.
Q: Why is top-posting such a bad thing?
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2

iQEcBAEBCAAGBQJW1GJcAAoJENuP0xzK19csOdkIAJIJLG2YPjSkCvMggIGFx1Xc
wUmPMnGYxu59rP63qzKcEzf3k5L7IgZbEW3HlZtdWD/GQVYrbJggfZy8gA4v9buS
20AyT6Sg1VnrkLDU8UzF87fGID7zOrxxTnADn+nIINKKb4F6iniHkWaJhA77w2Ne
Cgxq/GFEXBf3okCKcJwY254wXmdPLLe2AugKzR3YuLX9rbFbjHh1g4L4SzFl5w5/
fvPSTUDxDfcHpRWHBQdw4XtXnaIYNG0LV4G4f9TzVatJF7Z441GFiUznbdDqRztW
HAYKN8rQOvcODwGAb3kIuSiseP3HbFAp4Y4ss8o6yZ6rdMNSU9cKJJMkB8L0py0=
=aDr7
-----END PGP SIGNATURE-----

[Patrick Schleizer]

Marek Marczykowski-Górecki:

> Simple experiment shows that clocksource 'tsc' after host suspend is
> totally broken - when VM has is set, VM's clock is frozen after resume
> (until clocksource switched to 'xen' - either manually, or automatically
> as you've seen above).

Yes. It's a dead end. Without xen patches there is no way forward on
passing the advanced threat model.

> And back to the suspend issue - is it possible for Whonix gateway (and
> workstation) to obtain time for itself, without dom0 cooperation?

There is no sane way to fix bigger offsets.

[For the insane way, see [1].]

> But since tor can't connect with large clock skew (or it can?)

It can't.

> In the later case, dom0 will need to provide slightly randomized time to
> the VM. How large that difference needs to be to prevent time
> correlations?

I would say "same as bootclockrandomization". ;) (Even if we conclude to
change that.)

> bootclockrandomization[1] seems to use +/- 180s
> Currently dom0 (in qvm-sync-clock -> qubes.SetDateTime) gives time with
> precision up to a second (no milliseconds/nanoseconds). I guess it is
> too precise.

Yes.

> Do you have an idea how this should be implemented? I guess there should
> be dom0 qrexec service like qubes.GetRandomizedTime, which would give
> such randomized time. This service could be called in response to
> "system resume" notification from dom0 (as you've already written
> somewhere).

Sounds good.

> BTW why 0-5 is excluded in bootclockrandomization?

To make sure there will be a randomization as opposed to none or a too
small randomization.

> I think this leaks
> some data. For example if attacker have already some selected set of
> data to correlate (like "all Tor users in area X"), he/she can easily
> further narrow the search by eliminating those with time almost in sync
> (+/-5sec). Given popularity of NTP, which (when works) is pretty good at
> keeping the time in sync, this allows to easily eliminate all such
> users, leaving probably only those using Whonix, or maybe even more
> precise correlation...

Good point. Will be discussing this in Whonix forums.

bootclockrandomization always moving clock plus or 5 seconds

https://forums.whonix.org/t/bootclockrandomization-always-moving-clock-plus-or-5-seconds/2200

Cheers,
Patrick

[1] https://www.whonix.org/wiki/Dev/TimeSync#Tor_Consensus_Method