Memory deduplication for VM instances?
232 views
Skip to first unread message
Navid Zamani
May 15, 2014, 12:23:35 PM5/15/14
to qubes...@googlegroups.com
Hello. I wonder if Qubes offers memory deduplication for VM
instances, to save RAM.
I planned on designing a similar system, but my plan went something like this: (Simplified)
START VM:
boot template instance.
Instance mounts generic required modules read-only for disk deduplication and updateability.
Pause instance.
Optional: Write instance image to disk and allow unloading from RAM.
Make sure deduplication with other VMs happens in memory too.
START BINARY:
Clone associated template VM with memory deduplication. (Takes zero time and memory.)
Unpause new VM instance.
Mount binary’s image inside the instance.
Execute binary from its image inside the instance.
This would make it possible to use barely any RAM at all, and still allow writing with copy-on-write.
Especially in cases where the Dom0 OS participates in the memory deduplication, both for RAM and disk.
And it would work even better when using a large swap space in Dom0 to allow more templates to exist.
As far as I can tell from the infos on the website, Qubes seems to only do disk deduplication, and hence takes quite a lot of RAM.
Is it possible to add that, or does it already exist?
I planned on designing a similar system, but my plan went something like this: (Simplified)
START VM:
boot template instance.
Instance mounts generic required modules read-only for disk deduplication and updateability.
Pause instance.
Optional: Write instance image to disk and allow unloading from RAM.
Make sure deduplication with other VMs happens in memory too.
START BINARY:
Clone associated template VM with memory deduplication. (Takes zero time and memory.)
Unpause new VM instance.
Mount binary’s image inside the instance.
Execute binary from its image inside the instance.
This would make it possible to use barely any RAM at all, and still allow writing with copy-on-write.
Especially in cases where the Dom0 OS participates in the memory deduplication, both for RAM and disk.
And it would work even better when using a large swap space in Dom0 to allow more templates to exist.
As far as I can tell from the infos on the website, Qubes seems to only do disk deduplication, and hence takes quite a lot of RAM.
Is it possible to add that, or does it already exist?
Navid
Joanna Rutkowska
May 19, 2014, 6:48:08 AM5/19/14
to Navid Zamani, qubes...@googlegroups.com
On 05/15/14 18:23, Navid Zamani wrote:
> Hello. I wonder if Qubes offers memory deduplication for VM instances,
> to save RAM.
>
No. Although, IIRC, Xen has a thing called tmem that offered this
> Hello. I wonder if Qubes offers memory deduplication for VM instances,
> to save RAM.
>
mechanism, albeit only for Linux PV guests (which is the reason we
didn't use it).
> I planned on designing a similar system, but my plan went something like
> this: (Simplified)
>
> START VM:
> boot template instance.
> Instance mounts generic required modules read-only for disk
> deduplication and updateability.
> Optional: Write instance image to disk and allow unloading from RAM.
> *Make sure deduplication with other VMs happens in memory too.*
>
> START BINARY:
> *Clone associated template VM with memory deduplication.* (Takes zero
> time and memory.)
> /Unpause new VM instance./
> Mount binary’s image inside the instance.
> Execute binary from its image inside the instance.
>
> This would make it possible to use barely any RAM at all, and still
> allow writing with copy-on-write.
>
> Especially in cases where the Dom0 OS participates in the memory
> deduplication, both for RAM and disk.
> And it would work even better when using a large swap space in Dom0 to
> allow more templates to exist.
>
I don't quite follow your description, please consider describing it in
> Execute binary from its image inside the instance.
>
> This would make it possible to use barely any RAM at all, and still
> allow writing with copy-on-write.
>
> Especially in cases where the Dom0 OS participates in the memory
> deduplication, both for RAM and disk.
> And it would work even better when using a large swap space in Dom0 to
> allow more templates to exist.
>
a more clear way. Specifically indicate who does what and who is who.
E.g. what does it mean "instance mounts"? Instance of a VM or a
Template? How does it "mount"? Does the template exposed a block device
to the "instance"? Etc.
> As far as I can tell from the infos on the website, Qubes seems to
> only do disk deduplication, and hence takes quite a lot of RAM.
>
> Is it possible to add that, or does it already exist?
>
>
Andrew Sorensen
May 19, 2014, 12:57:24 PM5/19/14
to Joanna Rutkowska, Navid Zamani, qubes...@googlegroups.com
On 05/19/2014 03:48 AM, Joanna Rutkowska wrote:
> On 05/15/14 18:23, Navid Zamani wrote:
>> Hello. I wonder if Qubes offers memory deduplication for VM instances,
>> to save RAM.
>>
>
> No. Although, IIRC, Xen has a thing called tmem that offered this
> mechanism, albeit only for Linux PV guests (which is the reason we
> didn't use it).
>
What about the potential for exposing a side channel through memory
> On 05/15/14 18:23, Navid Zamani wrote:
>> Hello. I wonder if Qubes offers memory deduplication for VM instances,
>> to save RAM.
>>
>
> No. Although, IIRC, Xen has a thing called tmem that offered this
> mechanism, albeit only for Linux PV guests (which is the reason we
> didn't use it).
>
deduplication?
https://eprint.iacr.org/2014/248.pdf
cprise
May 19, 2014, 2:55:32 PM5/19/14
to qubes...@googlegroups.com, Joanna Rutkowska, Navid Zamani
On Monday, May 19, 2014 12:57:24 PM UTC-4, Andrew Sorensen wrote:
On 05/19/2014 03:48 AM, Joanna Rutkowska wrote:
> On 05/15/14 18:23, Navid Zamani wrote:
>> Hello. I wonder if Qubes offers memory deduplication for VM instances,
>> to save RAM.
>>
>
> No. Although, IIRC, Xen has a thing called tmem that offered this
> mechanism, albeit only for Linux PV guests (which is the reason we
> didn't use it).
>
What about the potential for exposing a side channel through memory
deduplication?
https://eprint.iacr.org/2014/248.pdf
Its interesting reading about those deduplication schemes. But I would have thought that any such scheme would be limited to immutable executable pages only. Implementing this for data seems really odd.
It would be great if deduplication could be based on the process of mapping executables into virtual memory. So vm A being the first to use libxyz.so would cause memory to be allocated for that library. But when vm B comes along and tries to load libxyz.so from the same disk volume (i.e. template) then the vmm layer would use no additional physical space. IOW, treat all non-self-modifying code loaded from the same template like they were Linux shared objects-- but across vms.
The amount of information leaked from executables-only sharing should be very limited; they could tell which executables you were using in other vms and that's about it.
Andrew Sorensen
May 19, 2014, 3:07:16 PM5/19/14
to cprise, qubes...@googlegroups.com, Joanna Rutkowska, Navid Zamani
On 05/19/2014 11:55 AM, cprise wrote:
>
>
> On Monday, May 19, 2014 12:57:24 PM UTC-4, Andrew Sorensen wrote:
>
> On 05/19/2014 03:48 AM, Joanna Rutkowska wrote:
> > On 05/15/14 18:23, Navid Zamani wrote:
> >> Hello. I wonder if Qubes offers memory deduplication for VM
> instances,
> >> to save RAM.
> >>
> >
> > No. Although, IIRC, Xen has a thing called tmem that offered this
> > mechanism, albeit only for Linux PV guests (which is the reason we
> > didn't use it).
> >
>
> What about the potential for exposing a side channel through memory
> deduplication?
>
> https://eprint.iacr.org/2014/248.pdf
> <https://eprint.iacr.org/2014/248.pdf>
>
>
> Its interesting reading about those deduplication schemes. But I would
> have thought that any such scheme would be limited to immutable
> executable pages only. Implementing this for data seems really odd.
Actually the attack does not require deduplication of the program's
>
>
> On Monday, May 19, 2014 12:57:24 PM UTC-4, Andrew Sorensen wrote:
>
> On 05/19/2014 03:48 AM, Joanna Rutkowska wrote:
> > On 05/15/14 18:23, Navid Zamani wrote:
> >> Hello. I wonder if Qubes offers memory deduplication for VM
> instances,
> >> to save RAM.
> >>
> >
> > No. Although, IIRC, Xen has a thing called tmem that offered this
> > mechanism, albeit only for Linux PV guests (which is the reason we
> > didn't use it).
> >
>
> What about the potential for exposing a side channel through memory
> deduplication?
>
> https://eprint.iacr.org/2014/248.pdf
> <https://eprint.iacr.org/2014/248.pdf>
>
>
> Its interesting reading about those deduplication schemes. But I would
> have thought that any such scheme would be limited to immutable
> executable pages only. Implementing this for data seems really odd.
data, just the instructions. The attack is carried out on the CPU's
instruction cache (as described in the flush+reload paper).
Eric Shelton
Jun 18, 2014, 3:07:13 PM6/18/14
to qubes...@googlegroups.com, navid....@gmail.com
On Monday, May 19, 2014 6:48:08 AM UTC-4, joanna wrote:
On 05/15/14 18:23, Navid Zamani wrote:
> Hello. I wonder if Qubes offers memory deduplication for VM instances,
> to save RAM.
>
No. Although, IIRC, Xen has a thing called tmem that offered this
mechanism, albeit only for Linux PV guests (which is the reason we
didn't use it).
... snip ...
> As far as I can tell from the infos on the website, Qubes seems to
> only do disk deduplication, and hence takes quite a lot of RAM.
>
> Is it possible to add that, or does it already exist?
>
>
joanna.
FWIW, after working on a system constrained to 4GB of RAM, I also raised the deduplication issue last year:
However, the discussion in this group did not lead to much, probably because there is no real public use of dedup. I found better information via xen-devel:
Basically, Xen does have a facility supporting what you are asking about. However, it may be limited to HVM domains. tools/memshr in the Xen source provides an example of its use. The core facility provided by Xen allows you to point out 2 identical pages to the hypervisor; if they indeed are identical, the pages will be "shared" in the Xen parlance (other, or at least similar terms you'll see include "deduplication", "dedup", and "merging"). Similar facilities are provided by KVM, although they seem to be more mature and actually in production use (a focus on PV over HVM in Xen may be responsible for less happening under Xen).
The Xen hypervisor itself does not provide a mechanism for identifying candidates for merging - some other program must run which, hopefully without introducing too much overhead, identifies duplicate pages. There are many papers and presentations online which discuss various approaches to this. Although some papers suggest zero-page merging is a big and fairly easy win, this mostly seems true shortly after a domain is started. However, over time most OSes will assign free pages to the buffer cache, which will tend to populate them with non-sharable data. This behavior (maximizing use of available RAM) is ingrained into memory managers, and nearly impossible to change directly. However, ballooning can counteract it, by having a process running in a domain that sucks up a significant fraction of free memory rather than allowing it to be "wasted" by the buffer cache. Luckily, page sharing can coexist with ballooning, and serve to share any zero pages not taken up by ballooning, and more importantly allow sharing of other data (which ballooning will never be able to provide). Another confounding factor is that there are differences in how operating systems treat freed pages: Win7 zeroes out pages shortly after they are freed (helpful for sharing), whereas Linux leaves their contents untouched when freed (not so helpful).
There is a group in China that has been working on a page scanner for KVM. More recently, it appears they have ported these efforts to Xen as well:
However, although the KVM codebase is available, only binaries are available for Xen at this time.
Best,
Eric
Reply all
Reply to author
Forward
0 new messages