Shaving N seconds off VM startup

147 views
Skip to first unread message

Chris Laprise

unread,
Apr 13, 2018, 1:20:00 PM4/13/18
to qubes-users
I've done some experimenting to get my Debian VMs to boot faster. So far
I've reduced the start time significantly by disabling these services in
the template:

apt-daily.service
apt-daily.timer
apt-daily-upgrade.service
apt-daily-upgrade.timer
pppd-dns
lvm2-monitor

Disabling the last two may have consequences, e.g. if you use VMs to
access LVM storage. But that lvm2-monitor does consume a whopping 4+
seconds according to systemd-analyze. YMMV.

And note that my criteria for picking these is just a cursory glance at
unit start times.

Ultimately, a good solution may be getting some of these units to start
10-20 seconds later. I think that makes sense in the case of
lvm2-monitor. Some other time-consuming services like qubes-update-check
already start later and don't seem to impact VM start times and
responsiveness.

FWIW, Ubuntu has announced that boot times have worsened a lot and
they'll make an effort to reduce them (again). Not sure to what extent
that reflects on Debian.

--

Chris Laprise, tas...@posteo.net
https://github.com/tasket
https://twitter.com/ttaskett
PGP: BEE2 20C5 356E 764A 73EB 4AB3 1DC4 D106 F07F 1886

799

unread,
Apr 13, 2018, 7:50:56 PM4/13/18
to Chris Laprise, qubes-users
Hello Chris,

Thanks for the info, very interesting.

Chris Laprise <tas...@posteo.net> schrieb am Fr., 13. Apr. 2018, 19:20:
I've done some experimenting to get my Debian VMs to boot faster. So far
I've reduced the start time significantly by disabling these services in
the template:
(...).

Can you post some more I formation what you have done to measure startup time?
Additionally I like to know if you can do the same for Fedora based VMs, as most of the AppVMs from me are Fedora based.

Regards 

[799]


Chris Laprise

unread,
Apr 15, 2018, 1:16:16 AM4/15/18
to 799, qubes-users
On 04/13/2018 07:50 PM, 799 wrote:
> Hello Chris,
>
> Thanks for the info, very interesting.
>
> Chris Laprise <tas...@posteo.net <mailto:tas...@posteo.net>> schrieb am
> Fr., 13. Apr. 2018, 19:20:
>
> I've done some experimenting to get my Debian VMs to boot faster. So
> far
> I've reduced the start time significantly by disabling these
> services in
> the template:
> (...).
>
>
> Can you post some more I formation what you have done to measure startup
> time?
> Additionally I like to know if you can do the same for Fedora based VMs,
> as most of the AppVMs from me are Fedora based.

I don't normally use Fedora, but it should be possible. Use tools like
'systemd-analyze blame' as your guide.

I'll try to post some timings tomorrow.

Another service that shows up with significant time is wpa_supplicant.
You can have it start only for network VMs by creating
/lib/systemd/system/wpa_supplicant.service.d/20_netvms with the following


[Unit]
ConditionPathExists=/var/run/qubes/this-is-netvm

john

unread,
Apr 15, 2018, 6:13:56 PM4/15/18
to qubes...@googlegroups.com
On 04/13/18 07:19, Chris Laprise wrote:
> I've done some experimenting to get my Debian VMs to boot faster. So far
> I've reduced the start time significantly by disabling these services in
> the template:
>
> apt-daily.service
> apt-daily.timer
> apt-daily-upgrade.service
> apt-daily-upgrade.timer
> pppd-dns
> lvm2-monitor
>
> Disabling the last two may have consequences, e.g. if you use VMs to
> access LVM storage. But that lvm2-monitor does consume a whopping 4+
> seconds according to systemd-analyze. YMMV.
>
> And note that my criteria for picking these is just a cursory glance at
> unit start times.
>
> Ultimately, a good solution may be getting some of these units to start
> 10-20 seconds later. I think that makes sense in the case of
> lvm2-monitor. Some other time-consuming services like qubes-update-check
> already start later and don't seem to impact VM start times and
> responsiveness.
>
> FWIW, Ubuntu has announced that boot times have worsened a lot and
> they'll make an effort to reduce them (again). Not sure to what extent
> that reflects on Debian.
>

Could be my imagination but in my ASRock Z170 UEFI (as another post on
qubes-user suggested)

*turning off the Intel speed stepping

seems to have fixed everything:

(I've no idea pros and cons of what this feature is; though I seem to
have a EFI install, I turned back some sub-settings in the UEFI EFI
choices to "legacy", no idea what those were/are either)

boot time, Fed-26 VM starts , qvm-run etc;

Issue I seem to have is qvm-shutdown <appvm> often completes and has
failed (maybe there is a timeout for shutdown as well as start-up ?); so
I'm having to qvm-kill most/many of the AppVMs ...... :)

[799]

unread,
Jun 2, 2018, 11:46:15 AM6/2/18
to qubes-users
Hello,

On 04/13 01:19, Chris Laprise wrote:
> I've done some experimenting to get my Debian VMs to boot faster. So far
> I've reduced the start time significantly by disabling these services in the
> template:
>
> apt-daily.service
> apt-daily.timer
> apt-daily-upgrade.service
> apt-daily-upgrade.timer
> pppd-dns
> lvm2-monitor
>
> Disabling the last two may have consequences, e.g. if you use VMs to access
> LVM storage. But that lvm2-monitor does consume a whopping 4+ seconds
> according to systemd-analyze. YMMV.
> [...]

I've made some further research and tweaked my multimedia template which is the only debian-based template I am running.
I the template:

user@t-multimedia:~$ systemd-analyze
Startup finished in 3.903s (kernel) + 4.980s (userspace) = 8.883s

looking at the time the services need during boot:

user@t-multimedia:~$ systemd-analyze blame
6.105s qubes-update-check.service
3.646s lvm2-monitor.service
3.585s dev-xvda3.device
964ms keyboard-setup.service
688ms qubes-misc-post.service
631ms dev-xvdd.device
[...]

as Chris suggested I have disabled some services:

user@t-multimedia:~$ sudo systemctl disable lvm2-monitor
user@t-multimedia:~$ sudo systemctl disable qubes-update-check
user@t-multimedia:~$ sudo systemctl disable apt-daily.service
user@t-multimedia:~$ sudo systemctl disable apt-daily.timer
user@t-multimedia:~$ sudo systemctl disable apt-daily-upgrade.service
user@t-multimedia:~$ sudo systemctl disable pppd-dns

I rebooted the template and made another measurement:

user@t-multimedia:~$ systemd-analyze
Startup finished in 4.535s (kernel) + 2.165s (userspace) = 6.701s

so basically I reduced runtime from 8.8 to 6.7 seconds

Looking at the services it seems that I was unable to disable qubes-update-check:

user@t-multimedia:~$ systemd-analyze blame
7.913s qubes-update-check.service
3.623s dev-xvda3.device
1.008s keyboard-setup.service
652ms dev-xvdd.device
502ms networking.service


QUESTION:
How can I disable qubes-update-check.service and what are the consequences doing so?


I have also removed cups and lvm2 as I don't need it in the VM.

ser@t-multimedia:~$ sudo apt-get remove lvm2 cups

Do you have any other ideas to reduce startup time?
I am also interested in improving bootup speed for my fedora-based App-VMs

[799]

donoban

unread,
Jun 2, 2018, 11:58:10 AM6/2/18
to qubes...@googlegroups.com
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256

On 06/02/18 17:45, [799] wrote:> QUESTION:
> How can I disable qubes-update-check.service and what are the
> consequences doing so?
>

Did you disable it on the AppVM or the template? If first the change
was probably not permanent. The consequence is that you will stop
receiving updates notifications if there is no AppVM running it for
that template.

This service should not delay the start up process since any other
service waits for it.

Disabling lvm-monitor seems nice because it delays 'local-fs-pre.target'
.
-----BEGIN PGP SIGNATURE-----

iQIzBAEBCAAdFiEEznLCgPSfWTT+LPrmFBMQ2OPtCKUFAlsSvogACgkQFBMQ2OPt
CKWMMxAAkHmpIGOZy4O9W2LUCINFccfqIBLC/gwTLjbvZ46EWt5B7/hj3qMvyhqj
pWGGiNM59NdUOCPaoS7UMFH/t7kjezFq1vAG/B2Jcv0Hy1CaRUFz4aBAijYt3Psi
X09t5pZ77uyoG4CjnEKgDgJGzWTHCddP58KzpBib7A5+Hut3CFALGxqsTW+cgxFj
+9ef/GMjX0A9VoQIGZylSHzMT83mHwNnibMWe4obCWrs2DoXV+qQDdG8asQPhKGe
nvXgE1O3dt6NAP6GMgYd9FsQ8m4QoTnXZVZdplM3TL1NoAaAxDQaIJudaXyDjOd4
vsoyyf5LgMIAUFXhMorGtzkcEomDGM8LfT90BysUxaIjRiRyLt/z4tMqOznEiShC
fxDJFpkfR60/GWPK39zZ9JgBQc+fR9eNh/n6ELQ9KuTsMapw1cuf07jNY+q5u6/K
TklxP9rDhhhm19whbQ6nD2uCNFKC9PNvjG4BPoQCP4/RbYPDk4Tfdf28vbc080ci
GrC1Q5fkPX4PWUphZMNKZjalnJAIDQmihxMQgOPJMmviLmQc6P1LmDXlzv0s1694
gcErBhCoeqo34b3Bf3JytVyMzXAU316K99JqxYFnJZQo/FbHU3+wKA/6zXXnoDu3
3me3+hNjlTv135W2a57ilN9T473W6dZASpMDdQBjO1PABKTvFeU=
=Dqx9
-----END PGP SIGNATURE-----

donoban

unread,
Jun 2, 2018, 12:00:51 PM6/2/18
to qubes...@googlegroups.com
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256

On 06/02/18 17:58, donoban wrote:
> Did you disable it on the AppVM or the template? If first the
> change was probably not permanent. The consequence is that you will
> stop receiving updates notifications if there is no AppVM running
> it for that template.


I re-read you and saw it was a custom template. I think that you
should disable the timer that fires the 'qubes-update-check' service. Tr
y:

# systemctl disable qubes-update-check.timer

-----BEGIN PGP SIGNATURE-----

iQIzBAEBCAAdFiEEznLCgPSfWTT+LPrmFBMQ2OPtCKUFAlsSvywACgkQFBMQ2OPt
CKVdhw/9G4i61Dgh8bAYL3qakGwFfpaECVNKjWGqMP41QAvFORdIO+DDB3fnjv53
Uhn1NOh1iTs5ldrbRUoQ5+BGimAkQK0g2sJ2KirJ9v3VM5HoRtPZBIXrzqZG7r2K
BU9eGaygE72ph80BgRkWIt3cCzdVIB3o8pLHDwpEVjHmzPm2vFA2MZWb/rsNflLT
ZBlLYoqnpn1CtoSXSHzhyttREVE4udIg6SvlSXOekBwipZaqczTJA7PXoA6V54Td
8issfBBujwHy4u79hKzNWHA2qCHaR/7U27UHSjNPYRiTNPv6hAekOkCkYsXjkY4x
hkVW8qUN9kMFjIt/bIgqLpCrd0vkt9XL8CB5ZjvNtP8KCR+Ilybg1QS+jMBOEUD3
lqhnsVAloHqd6O5Tlui5RshBeAFVn9ueYm6/4T7EiIZZDH8xLBWvaAo+WTLAwGo2
SjbT7jx9MLVZZ2MQAnLVGOYmwuGKVhiDYgnvBPu3+Gh0YboEnShdV9Jf+435pDwM
Z7KQFoY4DrRi3Dd9GFxGWzBFuxZ4Ig8BNPha0yWcsLWToJ9rggAb//ZRE1slppjh
7Vr9Pjjt8jfMn1+U2/MKbrJ1njLoUzU3uIBeTAtYXcPALSHjoB3Z2x3UanZZ82zj
9+AuftAxMae4vAKVCN6A8ANvGgS95oRrZ5OKQCm5mURaqpnm0os=
=ev8S
-----END PGP SIGNATURE-----

donoban

unread,
Jun 2, 2018, 12:11:59 PM6/2/18
to qubes...@googlegroups.com
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256

On 06/02/18 17:58, donoban wrote:
> On 06/02/18 17:45, [799] wrote:> QUESTION:
>> How can I disable qubes-update-check.service and what are the
>> consequences doing so?
>
> This service should not delay the start up process since any other
> service waits for it.
>

Well, maybe until 'multi-user.target' is achieved the started command
is not fired, but standard icons use service mode for run apps so
maybe them are started before. I am not sure.
-----BEGIN PGP SIGNATURE-----

iQIzBAEBCAAdFiEEznLCgPSfWTT+LPrmFBMQ2OPtCKUFAlsSwcYACgkQFBMQ2OPt
CKVQLw/7Bi5h2UUXocgkr05FIfjwHPEEck6gYznpLhpfBBuuCY+nIeVaaE6nOW2Q
WVhGd7+cX9QMC9P9CmlscB5dSCG7I8GT7IBA7ADJEW7dnXHrRw/UZIC6XE9IxWdi
ZnLaYUXb9RWX+yUbhe2B2MR+w96YRrlXI4CE2x2lhz/l7fHu7arJzxzFQMH09MeF
IPRTyNgFvnhtKWbSLLLtaO4ewNuUzzhh+XNn1NAafEZws3svvJoH/F9Qv0CYsa+v
LFiyxehq1/ecKfKu1hQDNu87w7YkyodRR0HNVnF0I46Su+CGaGm7CW5CvOhOGMGa
P45D6lsugW1p8KEfKpJgFEoJ1TPvwuReWq9pGd9MSHrQwaZ91Mfobi2FcYkU0Ma0
96ug5/9etgw6czo6/OyLqFWBiCOydt5Y644pqxFpuCDerzBxt/hjR6hEareU7dmC
+1Ml+WnunWPFYdF8hud5XZkDBUq0fqW8KUXDDkRJitfDsR9M0NRl0jcP9Oz6M/d2
p/m7dDcSVrSHg+UVw2GnFyn7XjJ1EhizT7uFjTpRo2e1D+2dD/+E8qnoM6CMqHLn
xHVIhaEQNCJ/L/siHiSn3zR7dR/XrXuudBYre14azBlSsVFfPVt20ZoKV5xkd/dd
bjnoo/GywPxaAp+1tmzrUUXj5lA4keiDVf+1CiJEk74ZGKILugs=
=TWix
-----END PGP SIGNATURE-----

Ivan Mitev

unread,
Jun 2, 2018, 1:02:33 PM6/2/18
to qubes...@googlegroups.com
Hey,
If you look at the service and timer definitions
(/usr/lib/systemd/system/qubes-update-check.{service,timer}) you'll see
that they're started only if the
/var/run/qubes-service/qubes-update-check file exists. In theory this
file is created if you have a `qubes-update-check` service defined for
the VM ; see the output of `qvm-service vmname` (or the 'services' tab
in the vm qubes gui)

You should also be able to disable updates globally with `qubes-prefs
check_updates_vm` or with the qubes global settings gui.

The consequences are that you won't have a "VM needs update"
notification in the gui. If debian updates flow like fedora and new
updates are available daily, the feature is pretty much unneeded.


> I have also removed cups and lvm2 as I don't need it in the VM.
>
> ser@t-multimedia:~$ sudo apt-get remove lvm2 cups
>
> Do you have any other ideas to reduce startup time?

yes :) - remove (almost) everything from /etc/xdg/autostart. I'm sure
there's a prettier way but that works for me. Some .desktop files will
get re-installed with time when updating the VM, remove them again (I
have a script for that btw, that is run automatically when I update my
templateVMs).

Exceptions:

- qubes-pulseaudio.desktop: if you want to have sound. Otherwise you'll
have to run `start-pulseaudio-with-vchan` manually before opening a
browser or some multimedia files.

- nm-applet.desktop: if you want to have the applet started. The problem
is that it'll start in *every* VMs, not only sys-net (the applet is then
hidden by qubes-show-hide-nm-applet.desktop). I got rid of it too and
I'm starting nm-applet only in sys-net (with
`/home/user/.config/autostart/nm-applet.desktop` ; starting it from
/rw/config/rc.local won't work because X isn't started yet when rc.local
is ran).

- qubes-icon-sender.desktop: if you want to have pretty icons (I don't)

- qubes-input-trigger.desktop




> I am also interested in improving bootup speed for my fedora-based App-VMs

It'd be great if you could write a guide with your findings (I can help
too).

>
> [799]
>

Cheers
Ivan

Chris Laprise

unread,
Jun 3, 2018, 2:35:54 PM6/3/18
to qubes...@googlegroups.com, donoban
On 06/02/2018 12:00 PM, donoban wrote:
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA256
>
> On 06/02/18 17:58, donoban wrote:
>> Did you disable it on the AppVM or the template? If first the
>> change was probably not permanent. The consequence is that you will
>> stop receiving updates notifications if there is no AppVM running
>> it for that template.
>
>
> I re-read you and saw it was a custom template. I think that you
> should disable the timer that fires the 'qubes-update-check' service. Tr
> y:
>
> # systemctl disable qubes-update-check.timer

You probably want to leave this one enabled if you want to be notified
by Qubes of available updates.

It's on a timer so that it won't run during boot, and it doesn't affect
VM start times. It shows up in 'blame' output... after a substantial delay.

799

unread,
Jun 3, 2018, 5:00:49 PM6/3/18
to Chris Laprise, qubes...@googlegroups.com, donoban
Hello Chris,

Chris Laprise <tas...@posteo.net> schrieb am So., 3. Juni 2018, 20:35:
On 06/02/2018 12:00 PM, donoban wrote:
> [...]

> I re-read you and saw it was a custom template. I think that you
> should disable the timer that fires the 'qubes-update-check' service. Tr
> y:
>
> # systemctl disable qubes-update-check.timer

You probably want to leave this one enabled if you want to be notified
by Qubes of available updates.

It's on a timer so that it won't run during boot, and it doesn't affect
VM start times. It shows up in 'blame' output... after a substantial delay.

I am mostly interested in performance.
If I can do anything to make my appVMs start faster I will do it.
For me the feature, that Qubes can tell me if there are updated available is not that interesting.
If I am on offline mode, I don't care about updates and if I am online my templates get checked at least every day once.
As such if you have any idea to further reduce AppVM startup I am interested.
Most important to me is speeding up the launch of a disposable VM, as I would like to open email attachments in disposable VMs.

[799]
Reply all
Reply to author
Forward
0 new messages