Launching speed of disposable VMs 15-18sec

[799]

Hello, 

I am just wondering if there is a way to speed up the start of disposable VMs.

On my W540 with an Intel Core i7-4900MQ with 4 Cores @ 2.8GHz / 32 GB RAM / 512GB SSD and having only sys-net / sys-firewall running the first boot of a disposable VM takes 18sec, later starts take 15sec.

There is not much difference compared to my less powerful X230.

Are these normal launch times (~15sec) for a DispVM?

What could be done to accelerate this to get below 10sec?

[799]

[Yuraeitha]

For comparison, it took me 33,49 seconds (stopwatch), on my Intel M-5Y10c @ 800Mhz processor, SSD, 8GB RAM, to open a picture in a dispVM. There was sufficient RAM free to open the dispVM unrestricted. It looks like if enough RAM is available, that it might be depending on CPU speed? But it could also be SSD speed (yours is a SATA SSD right? not a NVMe one?), but such a big difference between two non-NVMe doesn't seem right, so it's probably the CPU making a difference here. Question is though, is the CPU still the bottleneck once at current-day available high-speed CPU's? I.e. what point does SSD's or RAM speed become the bottleneck to opening up dispVM's fast?

So there must be two approaches?
- Faster CPU
- Semi-pre-loaded dispVM's (I believe I saw a topic to pre-loaded dispVM's with a similar headline in Qubes mail-list or github, once a very long time ago. Maybe it can be found again).

I suppose picking between PV/HVM/PVH might make a difference too? If I understand it right, PVH is the fastest one atm?

[Yuraeitha]

On Friday, March 9, 2018 at 11:11:09 PM UTC+1, [ 799 ] wrote:

I did a benchmark comparison (not overly accurate, but it might give some pointers).

Your CPU 9061 rating. Single Thread Rating: 2084. Margin for error: Low.
No of Cores: 4 (2 logical cores per physical).
https://www.cpubenchmark.net/cpu.php?cpu=Intel+Core+i7-4900MQ+%40+2.80GHz
15 seconds to open dispVM.

My CPU 2820 rating, Single Thread Rating: 1051. Margin for error: Low.
No of Cores: 2 (2 logical cores per physical).
https://www.cpubenchmark.net/cpu.php?cpu=Intel+Core+M-5Y10c+%40+0.80GHz&id=2464
33,49 seconds to open picture in dispVM.

So I got one real core, and one logical core to run my AppVM's, while you essentially have double up, 2 cores, 2 logical (unless you modified default Qubes CPU core layout to AppVM's of course).

If you notice the value rating pr. core, yours is rated 2048, while mine is rated 1051. That's a roughly double up difference in performance.

What I wonder about, what would happen if you assigned the 3'rd physical+logical core to your AppVM usecases, while preserving the last 1 physical+logical core in dom0? Would it give you 75% performance in AppVM's instead of just 50% performance?

I have no idea if this is even possible to do in Qubes without breaking Qubes, but I believe it should be possible though, question is probably if it is a good idea to do that? But at least performance wise, dom0 doesn't need all that much juice.

[799]

On 10 March 2018 at 00:03, Yuraeitha <yura...@gmail.com> wrote:

[...]

I did a benchmark comparison (not overly accurate, but it might give some pointers).

Your CPU 9061 rating. Single Thread Rating: 2084. Margin for error: Low.
No of Cores: 4 (2 logical cores per physical).
https://www.cpubenchmark.net/cpu.php?cpu=Intel+Core+i7-4900MQ+%40+2.80GHz
15 seconds to open dispVM.

My CPU 2820 rating, Single Thread Rating: 1051. Margin for error: Low.
No of Cores: 2 (2 logical cores per physical).
https://www.cpubenchmark.net/cpu.php?cpu=Intel+Core+M-5Y10c+%40+0.80GHz&id=2464
33,49 seconds to open picture in dispVM.

So I got one real core, and one logical core to run my AppVM's, while you essentially have double up, 2 cores, 2 logical (unless you modified default Qubes CPU core layout to AppVM's of course).

If you notice the value rating pr. core, yours is rated 2048, while mine is rated 1051. That's a roughly double up difference in performance.

What I wonder about, what would happen if you assigned the 3'rd physical+logical core to your AppVM usecases, while preserving the last 1 physical+logical core in dom0? Would it give you 75% performance in AppVM's instead of just 50% performance?

Start = click on launch DispVM Firefox

Stop = default website starts to load in Firefox

Mode = PVH 4GB RAM, 1vCPUs, Memory Balancing enabled
1st start = 19 seconds

2nd start = 19 seconds

Mode = PVH 4GB RAM, 2vCPUs, Memory Balancing enabled
1st start = 16 seconds

2nd start = 16 seconds

Increase vCPUs:

Mode = PVH, 4GB RAM, 4vCPUs, Memory Balancing enabled
1st start = 15 seconds

2nd start = 16 seconds

Disable Memory Belancing,

Mode = PVH, 4GB RAM, 4vCPUs, Memory Balancing disabled
1st start = 12 seconds

2nd start = 13 seconds

Mode = PVH, 8GB RAM, 6vCPUs, Memory Balancing disabled
1st start = 13 seconds

2nd start = 13 seconds

I did the same tests with Virt-Mode = HVM

Mode = HVM 4GB RAM, 2vCPUs, Memory Balancing enabled
1st start = 22 seconds

2nd start = 21  seconds

Mode = HVM 4GB RAM, 2vCPUs, Memory Balancing disabled
1st start = 17 seconds

2nd start = 18 seconds

Summary:

Adding more RAM or vCPUs doesn't change much, it seems that disabling memory balance brings a small performance improvement.

Using Virt-Mode PVH seems to deliver the best performance

===

Prebuild -> fast delivery

It is a very interesting idea would to have prebooted disposable VMs available. which gets activated as soon as I need a disposable VM.

As I have enougfh ressources I don't care if my RAM is eaten up or I loose a bit CPU performance.

If disposable VM opens within a few seconds, I can change my workflow and open my documents in a disposable VM.

[799]

[MirrorWay]

You can reduce the start time to almost zero by using an already-running, named DIspVM, see marmarek's post in https://github.com/QubesOS/qubes-issues/issues/2801.

You can set a cron job that ensures they shutdown at least once per day.

[799]

Hello,

Am 10.03.2018 1:10 vorm. schrieb "'MirrorWay' via qubes-users" <qubes...@googlegroups.com>:

You can reduce the start time to almost zero by using an already-running, named DIspVM, see marmarek's post in https://github.com/QubesOS/qubes-issues/issues/2801.

That sounds very interesting.

I have looked at the link, but didn't figure out what to do, to get faster DispVM boot up times.

What do I need to do?

You can set a cron job that ensures they shutdown at least once per day.

Why? The DispVM should be shutdown after I close the window.

[799]

[Yuraeitha]

nice CPU/virt_mode/memory benchmarks! That was a really interesting read.

btw I think what Mirrorway meant is if it automatically shutting down are to make up for down-time, for example if you don't use dispVM's for a full day or longer, then it'll shutdown on it's own and start again. Thereby, I believe, you prevent any potential internet based attacks by reloading fresh and clean system-files from the template. It seems like a pretty cool idea, it automates everything even if not using the computer for a period of time. Maybe make it more frequent, say once every 3 or 6 hours?

btw I found this too just now
https://github.com/QubesOS/qubes-issues/issues/2253

It seems even if the dispVM is shutdown, it can be made much faster too with a savefile. But if I understood it right, as Marek write in the first post they lack the manpower to get a savefile working for Qubes 4. But Qubes 3.2. has one, as it can be seen in Marek's time comparison in his second post.

but whoa, in Qubes 3.2. a savefile makes a difference from 25,5 seconds to 4 seconds, on this particular hardware. Considering this is a completely shutdown dispVM, that is some pretty impressive speed differences.

I wonder what would happen on Qubes 4 with PVH mode, savefile enabled, powerful CPU with a really fast NVMe SSD and RAM, all cores assigned but one which is kept in dom0, just how fast would it be then? In theory it should be less than 4 seconds at least if Marek's number on Qubes 3.2. can be seen as a maximum here for hardware, which is hardware that right now isn't the fastest available. What speeds would be possible here? Once a savefile is made for Qubes 4, this could probably be possible.

[MirrorWay]

Unlike regular dispvms, the lifetime of a named dispVMs is not tied to an app, you have to shutdown manually. Like regular dispvms, named dispVMs forget all changes to private storage after shutdown.

To create a named dispVM called "disp-untrusted" that is based on the "untrusted" VM:

$ qvm-prefs untrusted template_for_dispvms True

$ qvm-create --class DispVM --template untrusted -l red disp-untrusted

Your new named dispvm doesn't appear in the menu, so you'll need to rely on CLI to manipulate it:

$ qubes-vm-settings disp-untrusted

$ qvm-run disp-untrusted firefox

$ qvm-shutdown disp-untrusted

At this point you can verify that changes to /home DO NOT persist. You can also make sys-net, sys-usb disposable.

marmarek has some more posts about this.

I personally think this feature should be better advertised (e.g. added to Create Qubes VM).

[Unman]

No, it wont be - what Marek suggests is creating a qubes with a
disposableVM as its template. Then you can start this, and open term or
firefox in it straight away. But that qube stays running until you
actually close it.

Just qvm-create a qube with a dvm as the template.

[Yuraeitha]

On Saturday, March 10, 2018 at 1:21:25 AM UTC+1, [ 799 ] wrote:

or maybe I should go catch some sleep, I completely miss-read the savefile thing in Marek's post. It seems the savefile is what actually generates the long bootup's of dispVM's, contrary to what I thought at first. So by not using a pagefile, it becomes faster. But Qubes 4 doesn't even use page-files it seems.

Qubestion then is, what is it that is lacking development/manpower. I'm too tired to think about that right now though, I just need to correct my mistake here at least.

[Yuraeitha]

the speed people reply around here is indeed scary sometimes, I didn't even chance to try correct my self after realizing I had read it all wrong before you posted a few seconds before me ^^; but I appreciate the explanation, it does make more sense now by reading your interpretation. I didn't catch the use of a dispVM as a template. That does however seem very fascinating.

[799]

On 10 March 2018 at 01:48, 'MirrorWay' via qubes-users <qubes...@googlegroups.com> wrote:

Unlike regular dispvms, the lifetime of a named dispVMs is not tied to an app, you have to shutdown manually. Like regular dispvms, named dispVMs forget all changes to private storage after shutdown.

To create a named dispVM called "disp-untrusted" that is based on the "untrusted" VM:

$ qvm-prefs untrusted template_for_dispvms True

$ qvm-create --class DispVM --template untrusted -l red disp-untrusted

Your new named dispvm doesn't appear in the menu, so you'll need to rely on CLI to manipulate it:

$ qubes-vm-settings disp-untrusted

$ qvm-run disp-untrusted firefox

$ qvm-shutdown disp-untrusted

but this VM is just one (1) VM that will be reset (including the home directory) on each reboot, as such I can't start two of those VMs which are separated from each other (like real disposable VMs)?

$ qvm-prefs untrusted template_for_dispvms True

I can't run this command. It seems something is wrong here

I'm running Qubes 4rc5 and if I enter:

 qubes-prefs --get

I can't see a property template_for_dispvms

[799]

[Yuraeitha]

@Mirrorway

Agreed, there is sometimes some very interesting information/use-cases about/for Qubes, which is either lost in long detailed guides as a quick remark, or not documented at all. It's understandable that the developers are busy though, but it'd be interesting if we one day can get these interesting use-cases of Qubes highlighted more.

[MirrorWay]

On March 10, 2018 1:02 AM, 799 <one7...@gmail.com> wrote:

On 10 March 2018 at 01:48, 'MirrorWay' via qubes-users <qubes...@googlegroups.com> wrote:

Unlike regular dispvms, the lifetime of a named dispVMs is not tied to an app, you have to shutdown manually. Like regular dispvms, named dispVMs forget all changes to private storage after shutdown.

To create a named dispVM called "disp-untrusted" that is based on the "untrusted" VM:

$ qvm-prefs untrusted template_for_dispvms True

$ qvm-create --class DispVM --template untrusted -l red disp-untrusted

Your new named dispvm doesn't appear in the menu, so you'll need to rely on CLI to manipulate it:

$ qubes-vm-settings disp-untrusted

$ qvm-run disp-untrusted firefox

$ qvm-shutdown disp-untrusted

but this VM is just one (1) VM that will be reset (including the home directory) on each reboot, as such I can't start two of those VMs which are separated from each other (like real disposable VMs)?

Right, they would not be separated. You'd just send both documents to the same running VM instance.

$ qvm-prefs untrusted template_for_dispvms True

I can't run this command. It seems something is wrong here

I'm running Qubes 4rc5 and if I enter:

 qubes-prefs --get

I can't see a property template_for_dispvms

qvm-prefs, not qubes-prefs

[799]

[hopkins...@gmail.com]

>32 GB RAM. launch times (~15-19 sec)

This was the reason why i left Qubes OS. I cant coupe with hours starting vm-s. 3.2 version were faster.

[Yuraeitha]

On Saturday, March 10, 2018 at 8:21:02 AM UTC+1, hopkins...@gmail.com wrote:
> >32 GB RAM. launch times (~15-19 sec)
>
> This was the reason why i left Qubes OS. I cant coupe with hours starting vm-s. 3.2 version were faster.

well not really slow, you might just have had a bad setup and slow hardware. I don't think it's fair to blame Qubes for that though, you make it sound like it's their fault that you're on slow hardware with a bad version/settings layout. But maybe that's not your intention though.

I also hope you realize that "lots, and lots of RAM" doesn't just automatically equal fast start-ups, your specific cherry picking of quote in relation to your words, seem to imply just that.

Also why were you on Qubes if you didn't care about security? Leaving Qubes for something minor like this, seems to point that you don't care about security. So what made you come to Qubes in the first place?

Not to be a butthead or anything, I just think your rash comment is uncalled for.

[799]

Hello,

On 10 March 2018 at 17:14, Yuraeitha <yura...@gmail.com> wrote:

On Saturday, March 10, 2018 at 8:21:02 AM UTC+1, hopkins...@gmail.com wrote:
> >32 GB RAM.  launch times (~15-19 sec)
>
> This was the reason why i left Qubes OS. I cant coupe with hours starting vm-s. 3.2 version were faster.

I came up with an idea to accelerate the start of an application in a Disposable VM to ~2 seconds.

The idea came to my mind when I thought about how we accelerate stateless virtual desktops for our customers who are running VMware virtual desktop (Horizon View),

We can build desktops that only exist as long as the user is logged in, after logout the desktop is destroyed and he gets a complete new desktop (build from an image) on the next login.

To make it possible that users have a good user experience and don't have to wait during logon, when their desktop is provisioned we are prebuilding desktops, so that a certain amount of desktops is always available.

If a new users takes a desktop, the system is automatically reprovisioning a new one, for the next user.

I took the same idea to accelerate the launch of applications within disposable VMs.

I have one or more disposable AppVMs available that can be used to launch an application within it.

If the application ends the DispVM will be killed and for the next application another disposable AppVM will be provisioned in advance.

The downside is that those started Disposable VMs may use some ressources but as long as they are not running any calculation, the overhead shouldn't be to big.

This approach is a workarround not a very smart solution, but it works.

Please don't be to hard judging the current state of the work, as I am missing some scripting skills to make it do a first alpha version.

But someone with some more skills might be able to fill in the gaps.

This is how it works:

1) Launch a new disposable VM with nothing more than an empty xterm window. This is only to have something like a container, to start the application in

2) move the xterm window to the last desktop (default Qubes installation has 4 desktops (=desktop overview pager left of the clock in the menubar)

3) If you need to open a disposable Application, start this application in the already running disposable VM

    add a qvm-kill/qvm-shutdown after the command launch

4) provision a new disposable AppVM in the background which will be used when the next disposable Application must be started.

This is far away from being perfect but it would be good enough for me, I've run the commands manually and proved that something like this can work, but I am missing some more scripting skills.

I hope someone can support me, filling the missing gaps.

=====

Quick'n dirty notes, playing arround with the above idea / needs to be polished.

# Create list of all open windows = Window-List-1
wmctrl -l | gawk '{ print $1 }'

# Show all Running DispVMs = List-DVMs-1
qvm-ls | grep DispVM


# Create a new AppVM and open an xterm window in it
# This will open up a new xterm window in the current window
qvm-run -q -a --service --dispvm=fedora-26-dvm -- qubes.StartApp+xterm &


# Show all Running DispVMs = List-DVMs-2
qvm-ls | grep DispVM


# TODO: commands to get the name of the newly created DispVM
# This DispVM will be called "Newest-DispVM"


# Create a new list of all open windows = Window-List-2
wmctrl -l | gawk '{ print $1 }'


# TODO:
# Commands to find out which is the new window ID from the xterm-window
# The DispVM-window-ID = Window-List-2 - Windows-List-1
# we call this window here DispVM-xterm-Window-ID


# List of all available desktops
# First desktop = 0
wmctrl -d |  gawk '{ print $1 }'
# TODO: Get the greatest number from this list = LastDesktop


# move the new xterm window to the last desktop, so that is it out of the way
wmctrl -i -r $DispVM-xterm-Window-ID -t $LastDesktop


# If a new DisposableVM is needed, the following steps need to be done:

1) $Current-DispVM = Newest-DispVM

2) Prepare another DispVM in the background (!)
   Create a new DispVM  by opening an xterm session and
   move it to the last desktop (see above)
   This DVM will be the "Newest-DispVM

3) Run the application in Current-DispVM
   and kill the DispVM when the command has been ended
   qvm-run $Current-DispVM <COMMAND> && qvm-kill $Current-DispVM
   this will also kill the xterm window on the last desktop

What do you think?

Can someone can tell me the neccessary command to find out what is the Windows-ID of the xterm window and the DisposableVM-Name after launching a new App-VM?

[799]

[MirrorWay]

You can probably simplify this by basing it on named dispvms.

That way you don't have to keep an xterm open somewhere, nor do you need to extract the dispvm name from Xwindows.

Just restart the dispvm after you close the app.

For example, assuming disp-untrusted is already running:

$ qvm-run -p disp-untrusted firefox ; qvm-shutdown disp-untrusted ; qvm-start disp-untrusted

-p above causes the qvm-run to block until you close firefox. Then it restarts the named dispvm, which stays running until the next launch request.

Maybe you can replace qvm-shutdown with qvm-kill to make it faster.

[799]

Hello,

On 10 March 2018 at 22:52, 'MirrorWay' via qubes-users <qubes...@googlegroups.com> wrote:

You can probably simplify this by basing it on named dispvms.

That way you don't have to keep an xterm open somewhere, nor do you need to extract the dispvm name from Xwindows.

Just restart the dispvm after you close the app.

For example, assuming disp-untrusted is already running:

$ qvm-run -p disp-untrusted firefox ; qvm-shutdown disp-untrusted ; qvm-start disp-untrusted

-p above causes the qvm-run to block until you close firefox. Then it restarts the named dispvm, which stays running until the next launch request.

I've tested your suggestion, unfortunately this will not work like a normal disposable VM.

I have downloaded an HTML-page in the disp-untrusted VM and when it gets closed and started the next time, the file is still there.

This means it doesn't behave like a real disposable VM.

[799]

[Unman]

Change your template - base it off a -dvm, and it *will* work like a 3.2
disposableVM.
You can supplement this by cycling through a number of named
disposableVMs, so you can at least open a few at once.

[MirrorWay]

> > I've tested your suggestion, unfortunately this will not work like a normal
> >
> > disposable VM.
> >
> > I have downloaded an HTML-page in the disp-untrusted VM and when it gets
> >
> > closed and started the next time, the file is still there.
> >
> > This means it doesn't behave like a real disposable VM.

This is strange, did you manually restart disp-untrusted?

Check that `qvm-prefs disp-untrusted klass` says DispVM.

> Change your template - base it off a -dvm, and itwill work like a 3.2

Named dispvms should work with any appvm with template_for_dispvms set, though.

[799]

Hello,

On 11 March 2018 at 01:49, 'MirrorWay' via qubes-users <qubes...@googlegroups.com> wrote:

This is strange, did you manually restart disp-untrusted?

Check that `qvm-prefs disp-untrusted class` says DispVM.

> Change your template - base it off a -dvm, and itwill work like a 3.2
Named dispvms should work with any appvm with template_for_dispvms set, though.

sorry, but I still can't follow, I have already build a custom dvm, but it had the -class=AppVM

It seems that there are different disposable VMs now?

I have to read: https://www.qubes-os.org/doc/dispvm/

From my own Qubes Installation notes (Howto create an iwn disposable VM template):

# Create a new Disposable App-VM which is based on a custom template t-fedora-26

qvm-create --template t-fedora-26 --label red --property template_for_dispvms=True --class=AppVM my-dvm

 

# TEST: Start an application in this dvm

qvm-run --dispvm=my-dvm xterm

 

# Fix menu entry from Domain: my-dvm to Disposable: my-dvm

# https://groups.google.com/forum/#!msg/qubes-users/gfBfqTNzUIg/sbPp-pyiCAAJ

# https://github.com/QubesOS/qubes-issues/issues/1339#issuecomment-338813581

qvm-features vmname appmenus-dispvm 1

qvm-sync-appmenus --regenerate-only my-dvm

 

# Change the Disp-VM from an AppVM (here: my-untrusted)

qvm-prefs --set my-untrusted default_dispvm my-dvm

 

# Try to start something from this AppVM in a disposable VM

qvm-run --auto my-untrusted 'qvm-open-in-dvm https:/google.de'

# This should start a new dispvm which is based on your dvm-App

# Check the template on which the dispvm is based on in dom0

qvm-ls | grep disp

[799]