Window content get trashed due to missing memory pinning (#1028)

[HW42]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA512

Hi,

as described in #1028 the window content of VM windows get sometimes
trashed.

The problem is that the pages with the windows content need to be pinned
to a physical page when being mapped with xc_map_foreign_pages. To
archive this the current code calls mlock(). Unfortunately this only
ensures that the page doesn't get swapped out but the kernel can still
move the page.

- From [0]:
> A page that has been locked into memory with a call like mlock() is
> required to always be physically present in the system's RAM. [...].
> But there is nothing that requires a locked page to always be present
> in the same place; the kernel is free to move a locked page if the
> need arises.

So when a VM window looks corrupted the VM kernel has moved the page and
it has now a new physical address and in dom0 you now see the new
content of the old physical page.

If you want to monitor the effect you can use the attached script. It
shows the mapping of virtual addresses to physical addresses of a given
memory range for a given process. So call strace on Xorg to get the
mlock() call and pass those arguments to the script. Now either wait
until you have "luck" or provoke it by
'echo 1 > /proc/sys/vm/compact_memory'

Unfortunately the patch from [0] didn't got into the kernel, so there
is no easy way to pin a page similar to mlock().

One workaround is to disable memory compactation of unevictable (i.e.
mlocked) pages:

echo 0 > /proc/sys/vm/compact_unevictable_allowed

This has two downsides:

a) While I didn't observed the effect anymore with this setting I'm not
sure if this prevents all cases in which the kernel moves pages. (A
bit grepping suggest that it ignores this flag when allocating
hugepages but I'm not sure about this.)

b) Given that VMs in Qubes are often low on memory I'm a bit uneasy to
change memory management options without knowing the side effects.

Properly pinning is not that easy since there seems to be no way to flag
a page as unmovable. The common way seems to be to call
get_user_pages() to increment the reference counter and thereby pin it.
The problem is that then the u2mfn module needs to be changed
significantly to track allocation and deallocation since else the pages
never get freed.

If somebody (who is familiar with the linux kernel) knows another
solution this would be great.

#2185 mentions that you want do get rid of u2mfn anyway. How does the
new solution should work?

Unless somebody knows a better way to lock pages the best thing for R3.2
is probably to use the above mentioned work-around and implement a
better solution in R4.

HW42

[0]: https://lwn.net/Articles/600502/
-----BEGIN PGP SIGNATURE-----

iQIcBAEBCgAGBQJXmk+pAAoJEOSsySeKZGgWfgsQAKtHepI4CTnjtDx4xLoZlYAq
l3kdbtU7JEWERVnGZyAx8jwE5lL0olJQmf43eXID5bjuBFgXucYFKWsQFYC2rzjJ
EIuRuVdk1FjyXQc5T72Vtj1xJtXL07PByKgCao3GeL0iCaJDCZAR0w9Yb1l5uTsP
LAyR5GUmLHzqlv1S/qrmrCVCflENzWG309BdiPL6X203UaS8yOGgo5SY4MTiz9dL
1jPDwam/eH0Sguh7nBYJqhek5E3Dz1Vh7c2x7vOZXOgrxdU4NpOwFVcYEgY3m/9D
3ZobtaF+QDXU8VDjpapNLOXwL7IzKcI+lMj7lxIRiXU9CcfaiVWQsraWlHv6JYix
myetT04F7zTEu2YX+/9izn9EQ7u0y40K2/5918m8pRdWZjPJKtked0Z1R7+xOgpZ
+ZUn6hbHwZT8PzxtW3TKWmiHBMdjmlSFDNnqXcM+T+LV7JKwxf6/QKg9mYtqiAKI
mtr/BjXvPesfsIVY5ArHFf4DDjdX4Inuip1EC9EC0Scp4RSmZqX/9lcUUfexkr1w
yCVxATHi2wk5y22KNd62pVwxsIbaskEMVfHDbzrtD/VXPgLjGipyZogdxAIX6mWM
bC7FiZOD3FNyp49TH6KdNeM8mcjoXgj2CFZ++qTRoRsrtJaWV6jbDjaUsVydPauE
7diqcF+NyrSwT2meuMG/
=7zde
-----END PGP SIGNATURE-----

[Marek Marczykowski-Górecki]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256

On Thu, Jul 28, 2016 at 06:32:00PM +0000, HW42 wrote:
> Hi,
>
> as described in #1028 the window content of VM windows get sometimes
> trashed.
>
> The problem is that the pages with the windows content need to be pinned
> to a physical page when being mapped with xc_map_foreign_pages. To
> archive this the current code calls mlock(). Unfortunately this only
> ensures that the page doesn't get swapped out but the kernel can still
> move the page.
>

> From [0]:
> > A page that has been locked into memory with a call like mlock() is
> > required to always be physically present in the system's RAM. [...].
> > But there is nothing that requires a locked page to always be present
> > in the same place; the kernel is free to move a locked page if the
> > need arises.
>
> So when a VM window looks corrupted the VM kernel has moved the page and
> it has now a new physical address and in dom0 you now see the new
> content of the old physical page.

Wow, thanks for finding this out!

> If you want to monitor the effect you can use the attached script. It
> shows the mapping of virtual addresses to physical addresses of a given
> memory range for a given process. So call strace on Xorg to get the
> mlock() call and pass those arguments to the script. Now either wait
> until you have "luck" or provoke it by
> 'echo 1 > /proc/sys/vm/compact_memory'
>
> Unfortunately the patch from [0] didn't got into the kernel, so there
> is no easy way to pin a page similar to mlock().
>
> One workaround is to disable memory compactation of unevictable (i.e.
> mlocked) pages:
>
> echo 0 > /proc/sys/vm/compact_unevictable_allowed
>
> This has two downsides:
>
> a) While I didn't observed the effect anymore with this setting I'm not
> sure if this prevents all cases in which the kernel moves pages. (A
> bit grepping suggest that it ignores this flag when allocating
> hugepages but I'm not sure about this.)
>
> b) Given that VMs in Qubes are often low on memory I'm a bit uneasy to
> change memory management options without knowing the side effects.

I think we can live with those downsides for now.

> Properly pinning is not that easy since there seems to be no way to flag
> a page as unmovable. The common way seems to be to call
> get_user_pages() to increment the reference counter and thereby pin it.
> The problem is that then the u2mfn module needs to be changed
> significantly to track allocation and deallocation since else the pages
> never get freed.
>
> If somebody (who is familiar with the linux kernel) knows another
> solution this would be great.
>
> #2185 mentions that you want do get rid of u2mfn anyway. How does the
> new solution should work?

u2mfn module is needed to get page mfn, because otherwise kernel doesn't
provide this information to user space. But in HVM/PVH, it should be
enough to look at /proc/PID/pagemap. For PV this file contains pfn
("guest-specific pseudo-physical frame number"), which is not what we
need. So this problem will still apply.

So this problem will still apply. Proper solution would require
switching from xc_map_foreign_pages, to grant tables. But since this
require pages to be initially allocated using grant tables API, it
require substantial changes to video driver or even patching X server
itself.
But on the other hand, it should make this issue go away. And also will
be probably required for separate GUI domain.

> Unless somebody knows a better way to lock pages the best thing for R3.2
> is probably to use the above mentioned work-around and implement a
> better solution in R4.
>
> HW42
>
> [0]: https://lwn.net/Articles/600502/
>

- --

Best Regards,
Marek Marczykowski-Górecki
Invisible Things Lab
A: Because it messes up the order in which people normally read text.
Q: Why is top-posting such a bad thing?
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2

iQEcBAEBCAAGBQJXmnPpAAoJENuP0xzK19csPR4H/0X+rCCECxQN8dU7qpozLxM+
3ElTkj8eV4ZSDhVYLt8cfVYUXnHgcLiYBIqspukwlQU8IO1aUrH4MaSdKtqr4YvU
Iie9N99ilfD6V0Z4fesfF0O9BTCeryOOYAJWwBTHmqu/b75ZtiRoMTfTvvNupTO4
ukYtQA68WLqeDTRsop0CoWAJIxYPdL6HFL9+zy2gsBOfO4J9c4dvGMDpJZzyqDjd
UF4SLZ1q3zLPYl71lj3PbGg377I6X4m90FeiKmEvqyd4LsehQ2feBhGdTwx9CDUl
Bux56Z+s569nbpZWLvG4OP4K2aAGMSxClS7ILDQDOK2XiNuDJ2ZCompsYBu66UU=
=xyrZ
-----END PGP SIGNATURE-----

[Chris Laprise]

As an alternative, would it be possible to detect eviction events and
then tell the guest to refresh the window (as manually refreshing the
window is already a sort of workaround to the display garbage problem).

Chris

[Marek Marczykowski-Górecki]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256

As far I understand, no. At least not easily (without additional kernel
modification/module).

- --
Best Regards,
Marek Marczykowski-Górecki
Invisible Things Lab
A: Because it messes up the order in which people normally read text.
Q: Why is top-posting such a bad thing?
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2

iQEcBAEBCAAGBQJXmpykAAoJENuP0xzK19cs9sMH/2w1o/lm/7pIe6kS2ZMXM6hP
K8caCD5OGtIi5eXloULkFgFfQSEyDXbGYJh9kJm2NwP9mX+BhCdPENZLFTQzrqZG
UsRFQSAZiKws5FYxWGh/w92vtoyLuAl6QUnmT9M/ci9q9BnCyuKT6FGlUL8OaErr
RVRUgqsuvkuFRV8GmJl6822YUo7wdbNERaAysx3wQzT4C6NkRBBtiGBun7jxpViB
9QsCnjdanwCQ/QAPAzJ4qAaqhwY4HWSTb0X0IV9mBrw6jEoFc5O+dTiHbT2fhJV8
4Zrk4ykhJHyYtdL0KYBgnog1ykPi/XKY/wn+ioBCfelFB4mgl3ExAAApgf4oR7E=
=+Nzv
-----END PGP SIGNATURE-----