Re: [qubes-devel] NetVM very slow at starting up (or not starting up at all!)

[Andrew David Wong]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA512

(Replying on qubes-users, which is the appropriate list for threads
like this one. Please see: https://www.qubes-os.org/mailing-lists/)

On 2017-01-31 06:39, Dario Faggioli wrote:
> Hello,
>
> I've been here a couple of times already, trying out Qubes. :-)
>
> So, here's the thing. I download ISO and install Qubes 3.2. Boot
> into it and everything works. I do some customization, mostly in
> the form of installing new packages in templates, and all still
> looks fine. Then I reboot, and for some reason the NetVM has become
> either very very slow, or impossible to startup it properly.

This sounds like it could be a hardware-specific problem. Have you
tried Qubes on any other hardware?

> The specific cause seems a bit random too. One time, it was
> qubes-db which was not starting inside it (neither automatically,
> nor if I tried manually). Another time I got a libvirt error saying
> that the PCI ids of the network card was already assigned to
> sys-net! :-O

That's strange. When you said "the NetVM" above, I thought you were
referring to sys-net (the default NetVM). Are we talking about more
than one VM here?

> And yet another one, it seemed like VM build was crashing very
> very early (and I did not find any log containing the latest
> startup attempt).
>
> I'm reinstalling, right now (I'm a little bit on a hurry to get it
> running :-P), so I can't provide much logs (and even if I still had
> the system, it'd be hard to get the logs out without _any_
> networking!).

USB flash drive?

> Note that I've already tried re-installing and starting from
> scratch a couple of times. At every attempt, all is ok until I
> touch something withing the default fedora-23 template. :-(

This suggests that you might be doing something wrong when you
manipulate the template. Perhaps you could provide a precise list of
the steps you perform that affect the template?

> I do Xen development as my daily job, so I should be able to look
> into and debug things, if you give me an hint

CCing Marek, Master of Hints. :)

- --
Andrew David Wong (Axon)
Community Manager, Qubes OS
https://www.qubes-os.org
-----BEGIN PGP SIGNATURE-----

iQIcBAEBCgAGBQJYkVJpAAoJENtN07w5UDAwiTUP/0J0QZDE78MWHo26VYkvrKaX
ffnmfgI3FRl6twwquqpfCBrT2FY1CGVW8EIpWw/zU0AduyO0JpB5uWIp7bTFh5Dt
cO9jgpEVRBEBTz/oQUpo+fC1eWrbfdimG86Cz7TUI9YQg7d7HXSRgBl/AhrfBN1g
mMOE33H2uvj3WGAVJI/3ctr3er2canY8QC8quLX0TzdPq3IAM9+BiHhIZzUhj3ty
MYmOO4VVAY4ENIawpQm0ksxQHRwnPB4v0HAHJOA0wtbq3jqtT59HT7r20QaCz/Uh
jTsJrRuj7yN0SgBdqp7fXJXhk76EplHsjpaLsODPLP9tsz/TDpIG9Q9cQd5uKq7g
UhY11mU7rsXyfDEGHelwLWp4HmNxQWIkDxx/njFgmiXB7DA8OxuRRvzLkuT784LJ
iVHnIr7neLSXMuv98bfesNkluQ14TusRUvqPhx30cHIA0f5k4e8Rjvo75c5tK1mB
IsScT5dMw/fBkOea6585Uc7bQODwMBFkFG00esdD2Yj8dYyuaw9Zre7kOiEdjKzL
YC2TzqqY3IMPXqMYOtR0CS4onXHZS/0jClXocbr+1uUBLEFOFj+4UIJTMt3DiecZ
Gq2r2u6/Bu/fQn8zTKM7Dqaeh8Ro0H4HdnOeNLk58uk0tWu+FV5hIB98N+el38/h
G9yaMLtCSskrMDL5iAK9
=Mit7
-----END PGP SIGNATURE-----

[Dario Faggioli]

On Tue, 2017-01-31 at 19:13 -0800, Andrew David Wong wrote:
> (Replying on qubes-users, which is the appropriate list for threads
> like this one. Please see: https://www.qubes-os.org/mailing-lists/)
>

Yeah, well, it did looked like a bug to me, that's why I posted on
devel. Anyway...

> On 2017-01-31 06:39, Dario Faggioli wrote:
> > The specific cause seems a bit random too. One time, it was
> > qubes-db which was not starting inside it (neither automatically,
> > nor if I tried manually). Another time I got a libvirt error saying
> > that the PCI ids of the network card was already assigned to
> > sys-net! :-O
>
> That's strange. When you said "the NetVM" above, I thought you were
> referring to sys-net (the default NetVM). Are we talking about more
> than one VM here?
>

Yes, it's sys-net.

> > Note that I've already tried re-installing and starting from
> > scratch a couple of times. At every attempt, all is ok until I
> > touch something withing the default fedora-23 template. :-(
>
> This suggests that you might be doing something wrong when you
> manipulate the template. Perhaps you could provide a precise list of
> the steps you perform that affect the template?
>

It turned out to be something similar/related to this:
https://github.com/QubesOS/qubes-issues/issues/1449

I.e., too few (contiguous?) memory to start a VM with PCI devices
passed through to it.

This is a not so new laptop, with 8GB RAM. As I said, some of the time,
even without modifying anything, sys-net and sys-usb just can't get
started. Others, they do.

If I balloon down dom0 before trying to starting them, like this:

xl mem-set 0 1500

everything works. So far, instead of touching swiotlb, I've worked it
around by passing "dom0_mem=1536,max:4096".

With this, system VMs start at boot, and this is enough for me right
now.

Thanks and Regards,
Dario
--
<<This happens because I choose it to happen!>> (Raistlin Majere)
-----------------------------------------------------------------
Dario Faggioli, Ph.D, http://about.me/dario.faggioli
Senior Software Engineer, Citrix Systems R&D Ltd., Cambridge (UK)

[Chris Laprise]

On 02/03/2017 10:15 AM, Dario Faggioli wrote:
>
> It turned out to be something similar/related to this:
> https://github.com/QubesOS/qubes-issues/issues/1449
>
> I.e., too few (contiguous?) memory to start a VM with PCI devices
> passed through to it.
>
> This is a not so new laptop, with 8GB RAM. As I said, some of the time,
> even without modifying anything, sys-net and sys-usb just can't get
> started. Others, they do.
>
> If I balloon down dom0 before trying to starting them, like this:
>
> xl mem-set 0 1500
>
> everything works. So far, instead of touching swiotlb, I've worked it
> around by passing "dom0_mem=1536,max:4096".
>
> With this, system VMs start at boot, and this is enough for me right
> now.
>
> Thanks and Regards,
> Dario

Hi,

With 8GB RAM in my system also, I don't encounter the sys-net issue
you're experiencing.

Maybe not all RAM is getting recognized at boot. Long ago, I had a
low-RAM problem when I started using tboot (part of AEM) and a
workaround was required. Did you check total_memory from 'xl info'?

Chris