Google Groups no longer supports new Usenet posts or subscriptions. Historical content remains viewable.
Dismiss

Bug#994870: Memory allocation problem for VM after xen security update

311 views
Skip to first unread message

H.-R. Oberhage

unread,
Sep 22, 2021, 6:00:03 AM9/22/21
to
Package: xen-system-amd64
Version: 4.14.3-1~deb11u1

After applying the buster security update to xen, my VM won't start
any longer, complaining about a memory allocation error.

Switching back to the previous version 4.14.2+25-gb6a8c4f72d-2 lets
the VM start (again,) normally.

/var/log/libvirt/libxl/libxl-driver.log:
2021-09-21 14:01:44.645+0000: xc: panic: xc_dom_boot.c:120:
xc_dom_boot_mem_init: can't allocate low memory for domain: Out of
memory
2021-09-21 14:01:44.653+0000: libxl: libxl_dom.c:593:libxl__build_dom:
xc_dom_boot_mem_init failed: Die Operation wird nicht unterstützt
[means: the operation is not supported]
2021-09-21 14:01:44.662+0000: libxl:
libxl_create.c:1576:domcreate_rebuild_done: Domain 1:cannot (re-)build
domain: -3

The error is triggered, regardless if there was a boot-parameter
"dom0_mem=1024M:max=2048M" set or not.
/etc/xen/xl.conf was unaltered, i.e. 'autoballoon' was implicitely set
to "auto".

I am "on" Buster, kernel 5.10.0-8-amd64 (5.10.46-4), all relevant fixes
included.

Best regards,
Ruediger Oberhage
--
Dr. H.-R. Oberhage
Mail: Univ. Duisburg-Essen E-Mail: ober...@Uni-Duisburg-Essen.DE
Fakultaet fuer Physik rued...@Theo.Physik.Uni-Duisburg-Essen.DE

Hans van Kranenburg

unread,
Sep 22, 2021, 3:10:03 PM9/22/21
to
Hi Ruediger,

On 9/22/21 11:37 AM, H.-R. Oberhage wrote:
> Package: xen-system-amd64
> Version: 4.14.3-1~deb11u1
>
> After applying the buster security update to xen, my VM won't start
> any longer, complaining about a memory allocation error.

Can you confirm that this is a virtual machine that tries to boot a
32-bit kernel as PV type?

The error message you are seeing is not particularly helpful, but it is
most likely related to this.

The fact that with this package update 32-bit PV guests fail to start is
indeed a regression problem, which is quite inconvenient for you, right now.

At this point I would really recommend to not wait for a fix to arrive
which makes it start again, but change your VM to use a 64-bit kernel.

Let me know if you need help or run into problems while making this change.

Running 32-bit PV at all is already 'on life support' upstream for quite
a while now, and it also not under security support any more.

In the long run, I'd suggest working towards having 64-bit guests in PVH
mode, since that's one of the best options we have these days.

If there's a reason you really cannot switch to a 64-bit kernel or move
the functionality of this virtual machine to a new fully 64 bit system,
switching the virtualization type from PV to HVM would also be an option.

> Switching back to the previous version 4.14.2+25-gb6a8c4f72d-2 lets
> the VM start (again,) normally.
>
> /var/log/libvirt/libxl/libxl-driver.log:
> 2021-09-21 14:01:44.645+0000: xc: panic: xc_dom_boot.c:120:
> xc_dom_boot_mem_init: can't allocate low memory for domain: Out of
> memory
> 2021-09-21 14:01:44.653+0000: libxl: libxl_dom.c:593:libxl__build_dom:
> xc_dom_boot_mem_init failed: Die Operation wird nicht unterstützt
> [means: the operation is not supported]
> 2021-09-21 14:01:44.662+0000: libxl:
> libxl_create.c:1576:domcreate_rebuild_done: Domain 1:cannot (re-)build
> domain: -3
>
> The error is triggered, regardless if there was a boot-parameter
> "dom0_mem=1024M:max=2048M" set or not.
> /etc/xen/xl.conf was unaltered, i.e. 'autoballoon' was implicitely set
> to "auto".
>
> I am "on" Buster, kernel 5.10.0-8-amd64 (5.10.46-4), all relevant fixes
> included.

Apologies for the inconvenience,

Hans

Hans van Kranenburg

unread,
Sep 24, 2021, 9:40:02 AM9/24/21
to
Hi!,

Please don't (accidentally) drop the debian bug email from the recipient
list. This information might also be useful for others later.

On 9/23/21 1:47 AM, H.-R. Oberhage wrote:
> Good evening Hans,
>
> On 22.09.2021 20:54, Hans van Kranenburg wrote:
>> Hi Ruediger,
>>
>> On 9/22/21 11:37 AM, H.-R. Oberhage wrote:
>>> Package: xen-system-amd64
>>> Version: 4.14.3-1~deb11u1
>>>
>>> After applying the buster security update to xen, my VM won't start
>>> any longer, complaining about a memory allocation error.
>>
>> Can you confirm that this is a virtual machine that tries to boot a
>> 32-bit kernel as PV type?
>
> yes, your assumption ...
>
>> The error message you are seeing is not particularly helpful, but it is
>> most likely related to this.
>
> ... is correct.
>
>> The fact that with this package update 32-bit PV guests fail to start
>> is
>> indeed a regression problem, which is quite inconvenient for you, right
>> now.
>
> Ok, then I will put the Xen-package on "hold" for now.
>
>> At this point I would really recommend to not wait for a fix to arrive
>> which makes it start again, but change your VM to use a 64-bit kernel.
>
> It really is a shame, that 32-bit isn't supported properly any longer.
> The address- and data-overhead in 64-bit machines only using a 32-bit
> address- and data-space is considerable.
>
> I already experienced, "bullseye" not supporting a dom0-Kernel for the
> i686-pae architecture any longer :-(. A shame that it doesn't come with
> a kernel before 5.9, which would still allow this.
>
>> Let me know if you need help or run into problems while making this
>> change.
>
> Would you know of a "simple" way to convert/clone a 32-bit VM to a
> work-alike 64-bit one? One has to replace all the .debs for this, after
> all.

The smallest amount of work to initially get your VM going again is to
only install the 64 bit kernel and keep running a 32 bit user land.

The process to fully change from a 32 to 64 system (in place) is called
'cross grading'. I found instructions at
https://wiki.debian.org/CrossGrading

I never did this myself, though.

>> Running 32-bit PV at all is already 'on life support' upstream for
>> quite
>> a while now, and it also not under security support any more.
>
> Well it's a Debian "stretch" one, so it's just working for now :-).

One of the main reasons why it's so problematic to keep around is that
in the 32 bit PV case, there are no possibilities to implement fixes for
all the speculative vulnerabilities that have been very much in the news
in the last years.

More about this: https://xenbits.xen.org/xsa/advisory-370.html

>> In the long run, I'd suggest working towards having 64-bit guests in
>> PVH
>> mode, since that's one of the best options we have these days.
>
> Thanks, I'll consider this for any newer VMs.
> Are 64-bit PV VMs automatically "moved" to or executed as PVH?
> I would even be willing to edit the .xml/.cfg-file manually.
> I see "bullseye's" virt-manager/libvirt offering only choices for
> "xen (fullvirt)", "xen (paravirt)", or xen", when creating a new
> VM.

It should be as simple as changing type="pv" to type="pvh" in the config
file. In Debian, using PVH this is possible since Buster. Also, using
the xen variant of grub2 (grub-xen and grub-xen-host) is possible.

More info:
https://wiki.xenproject.org/wiki/Understanding_the_Virtualization_Spectrum

Have fun,
Hans

Andy Smith

unread,
Sep 24, 2021, 10:10:03 AM9/24/21
to
Hello,

On Fri, Sep 24, 2021 at 03:33:21PM +0200, Hans van Kranenburg wrote:
> The smallest amount of work to initially get your VM going again is to
> only install the 64 bit kernel and keep running a 32 bit user land.

Yep, I have a few 32-bit PV domUs that I run as 64-bit just the
kernel, works great.

> The process to fully change from a 32 to 64 system (in place) is called
> 'cross grading'. I found instructions at
> https://wiki.debian.org/CrossGrading
>
> I never did this myself, though.

I've done it lots of times but for any non-trivial host you have to
pretty much be a Debian expert to know how to handle the many
problems that dpkg will encounter. It takes longer than
reinstalling and a mistake breaks everything. Not recommended.

> It should be as simple as changing type="pv" to type="pvh" in the config
> file. In Debian, using PVH this is possible since Buster. Also, using
> the xen variant of grub2 (grub-xen and grub-xen-host) is possible.

I can confirm that I have a lot of Debian stretch domUs running PVH
but you do need to install backports kernel in the domU. The 4.9.x
stretch kernelis too old. You need the 4.19.x one from
stretch-backports. They boot fine for me with pvhgrub.

Cheers,
Andy

H.-R. Oberhage

unread,
Sep 24, 2021, 10:40:02 AM9/24/21
to
Hello,

thank you for your kind and helpful answers.

No including the "bugs" e-mail address in my first reply
was my mistake, I just didn't notice, that there was more than
one address. Shouldn't happen again :-).

On 24.09.2021 15:50, Andy Smith wrote:
> Hello,
>
> On Fri, Sep 24, 2021 at 03:33:21PM +0200, Hans van Kranenburg wrote:
>> The smallest amount of work to initially get your VM going again is to
>> only install the 64 bit kernel and keep running a 32 bit user land.
>
> Yep, I have a few 32-bit PV domUs that I run as 64-bit just the
> kernel, works great.

That information helps me a lot!

>> The process to fully change from a 32 to 64 system (in place) is
>> called
>> 'cross grading'. I found instructions at
>> https://wiki.debian.org/CrossGrading
>>
>> I never did this myself, though.
>
> I've done it lots of times but for any non-trivial host you have to
> pretty much be a Debian expert to know how to handle the many
> problems that dpkg will encounter. It takes longer than
> reinstalling and a mistake breaks everything. Not recommended.

I also did it with a few machines some time ago, but I remembered
how cumbersome it was. That's why I asked about a "simple" method :-).
I won't do it (cross grading), if not forced to do so.

>> It should be as simple as changing type="pv" to type="pvh" in the
>> config
>> file. In Debian, using PVH this is possible since Buster. Also, using
>> the xen variant of grub2 (grub-xen and grub-xen-host) is possible.
>
> I can confirm that I have a lot of Debian stretch domUs running PVH
> but you do need to install backports kernel in the domU. The 4.9.x
> stretch kernelis too old. You need the 4.19.x one from
> stretch-backports. They boot fine for me with pvhgrub.

This, too, helps me a lot. Thank you. All of the machines in question
here, are on buster or newer - although by upgrading. So no stress
here.

> Cheers,
> Andy

Thanks to both of you, Hans and Andy,
Ruediger

Alexander Dahl

unread,
Sep 29, 2021, 6:20:04 PM9/29/21
to
Hello,

Am 22.09.21 um 20:54 schrieb Hans van Kranenburg:
> Hi Ruediger,
>
> On 9/22/21 11:37 AM, H.-R. Oberhage wrote:
>> Package: xen-system-amd64
>> Version: 4.14.3-1~deb11u1
>>
>> After applying the buster security update to xen, my VM won't start
>> any longer, complaining about a memory allocation error.
>
> Can you confirm that this is a virtual machine that tries to boot a
> 32-bit kernel as PV type?

I can confirm this for three of my virtual machines.

> The error message you are seeing is not particularly helpful, but it is
> most likely related to this.
>
> The fact that with this package update 32-bit PV guests fail to start is
> indeed a regression problem, which is quite inconvenient for you, right now.

And for me. :-/

> At this point I would really recommend to not wait for a fix to arrive
> which makes it start again, but change your VM to use a 64-bit kernel.

How?

> Let me know if you need help or run into problems while making this change.
>
> Running 32-bit PV at all is already 'on life support' upstream for quite
> a while now, and it also not under security support any more.

Ack.

FWIW, Debian 10 VMs with 32 bit running with PVH work fine. My important
VM is still Debian 9 however due to a software I can not simply upgrade.

> In the long run, I'd suggest working towards having 64-bit guests in PVH
> mode, since that's one of the best options we have these days.

Ack.

> If there's a reason you really cannot switch to a 64-bit kernel or move
> the functionality of this virtual machine to a new fully 64 bit system,
> switching the virtualization type from PV to HVM would also be an option.

Thanks for your support.

Greets
Alex

>> Switching back to the previous version 4.14.2+25-gb6a8c4f72d-2 lets
>> the VM start (again,) normally.
>>
>> /var/log/libvirt/libxl/libxl-driver.log:
>> 2021-09-21 14:01:44.645+0000: xc: panic: xc_dom_boot.c:120:
>> xc_dom_boot_mem_init: can't allocate low memory for domain: Out of
>> memory
>> 2021-09-21 14:01:44.653+0000: libxl: libxl_dom.c:593:libxl__build_dom:
>> xc_dom_boot_mem_init failed: Die Operation wird nicht unterstützt
>> [means: the operation is not supported]
>> 2021-09-21 14:01:44.662+0000: libxl:
>> libxl_create.c:1576:domcreate_rebuild_done: Domain 1:cannot (re-)build
>> domain: -3
>>
>> The error is triggered, regardless if there was a boot-parameter
>> "dom0_mem=1024M:max=2048M" set or not.
>> /etc/xen/xl.conf was unaltered, i.e. 'autoballoon' was implicitely set
>> to "auto".
>>
>> I am "on" Buster, kernel 5.10.0-8-amd64 (5.10.46-4), all relevant fixes
>> included.
>
> Apologies for the inconvenience,
>
> Hans
>
> _______________________________________________
> Pkg-xen-devel mailing list
> Pkg-xe...@alioth-lists.debian.net
> https://alioth-lists.debian.net/cgi-bin/mailman/listinfo/pkg-xen-devel
>

Andy Smith

unread,
Sep 29, 2021, 6:50:03 PM9/29/21
to
Hi Alex,

On Thu, Sep 30, 2021 at 12:10:32AM +0200, Alexander Dahl wrote:
> Am 22.09.21 um 20:54 schrieb Hans van Kranenburg:
> > At this point I would really recommend to not wait for a fix to arrive
> > which makes it start again, but change your VM to use a 64-bit kernel.
>
> How?

This was answered in earlier comments on this bug; please see:

https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=994870#15
https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=994870#20

The brief summary is, "start out like a crossgrade, but only do the
kernel". Very simple and quite safe.

You haven't said how you boot your guest though (show us your
/etc/xen/guest.cfg file). If it's pvgrub, that has a 32-bit and a
64-bit version so you'll need to change those as well. If it's
pygrub you probably don't need to do anything, though pygrub has its
own issues outside the scope of this bug.

> FWIW, Debian 10 VMs with 32 bit running with PVH work fine. My important VM
> is still Debian 9 however due to a software I can not simply upgrade.

I've found PVH needs at least 4.19 guest kernel to work, which can
be achieved in Debian 9 (stretch) today by using kernel from
stretch-backports, so perhaps that is an option for you.

Cheers,
Andy

Hans van Kranenburg

unread,
Sep 30, 2021, 7:20:03 AM9/30/21
to
Hi!
You can certainly do that and then run PVH.

Since stretch-backports is not used any more since stretch became
oldoldstable, new 4.19 backports kernels for Stretch are released
through the security updates channel. Be aware of this.

https://lists.debian.org/debian-lts-announce/2020/08/msg00019.html

Latest in stretch-backports (frozen) is 4.19.118, and stretch security
is now at 4.19.194. So double check you end up following the right one.

Hans

Alexander Dahl

unread,
Oct 2, 2021, 2:20:04 PM10/2/21
to
Hei hei,
This actually works. I'm running 4.19 i686 kernel in the stretch VM
now with PVH, at least for the Debian stretch VM (I had to permanently
disable some old OpenWRT VMs, where I get no updates anymore). Was a
little tricky to install it, because I had to install that kernel
without the vm being able to start, but it worked like this:

- mount the root filesystem of the vm in the host, e.g. to /mnt
- bind mount /dev, /sys to /mnt/dev and /mnt/sys
- mount procfs to /mnt/proc
- mount a tmpfs to /mnt/run and create /run/lock
- chroot into /mnt
- install the needed kernel with apt
- leave chroot, umount the things from above
- change domU config to PVH
- when in grub, edit the cmdline and change root= if it was changed by
update-grub (might have been changed to the mount point from the
chroot)
- in the now booted system, run update-grub again

> Since stretch-backports is not used any more since stretch became
> oldoldstable, new 4.19 backports kernels for Stretch are released
> through the security updates channel. Be aware of this.
>
> https://lists.debian.org/debian-lts-announce/2020/08/msg00019.html
>
> Latest in stretch-backports (frozen) is 4.19.118, and stretch security
> is now at 4.19.194. So double check you end up following the right one.

Thanks for all your hints. I really have to migrate my virtual
machines. :-/

Greets
Alex

--
/"\ ASCII RIBBON | »With the first link, the chain is forged. The first
\ / CAMPAIGN | speech censured, the first thought forbidden, the
X AGAINST | first freedom denied, chains us all irrevocably.«
/ \ HTML MAIL | (Jean-Luc Picard, quoting Judge Aaron Satie)
signature.asc
0 new messages