Xen VM experiments fail because of disk space with Ubuntu 22 image

[ff...@nyu.edu]

Hello,

I really appreciate the bigger root filesystem on the new standard images! But, I noticed that in experiments with Xen VMs, the resource mapper doesn't give me sufficient physical resources for the larger image size.

For example, if I use the "default" small-lan profile to ask for 3 Ubuntu 22 VMs on an m510 (which won't fit; max 2 VMs with this size image will fit on an m510):

the experiment fails because one of the VMs fails, and the log shows that it is because it runs out of disk space for the third VM - 

mysystem: \'lvcreate -Zy -L 67373108k -n ms0828vm-2 -i1 xen-vg\'
  Rounding up size to full physical extent 64.25 GiB
  Volume group "xen-vg" has insufficient free space (14836 extents): 16449 required.
Command failed: 1280 - \'lvcreate -Zy -L 67373108k -n ms0828vm-2 -i1 xen-vg\'
mysystem: \'lvcreate -V 67373108k -n ms0828vm-2 --thinpool xen-vg/disk-pool\'
  Pool disk-pool not found in Volume group xen-vg.
Command failed: 1280 - \'lvcreate -V 67373108k -n ms0828vm-2 --thinpool xen-vg/disk-pool\'
createLV: could not create 67373108k LV ms0828vm-2
libvnode_xen: could not create disk for ms0828vm-2
TIMESTAMP: 09/27/2023 11:17:44.031092 failed xen vnodeCreate(): *** /usr/libexec/emulab/mkvnode.pl:
    libvnode_xen: could not clone image+emulab-ops-emulab-ops-UBUNTU22-64-X86-2

*** /usr/libexec/emulab/mkvnode.pl:
    vnodeCreate failed: *** /usr/libexec/emulab/mkvnode.pl:
    libvnode_xen: could not clone image+emulab-ops-emulab-ops-UBUNTU22-64-X86-2

TIMESTAMP: 11:17:44.043751 vnodesetup (7357) exiting with status 512

More observations - 

• if I use the small-lan profile to ask for 12 Ubuntu 20 VMs on an m510, it will correctly map to 2 physical servers (one m510 can fit max 10x the Ubuntu 20 disk image).
• If I ask for 12 Ubuntu 22 VMs on an m510, it will still try to put them on 2 physical servers, even though it needs more to accommodate the larger image... and 8/12 of the VMs fail, since only the first 2 on each server have enough disk space.

I have a big education project that involves networks of many VMs with Ubuntu 22 image, I think my students are experiencing many failed experiments because of this.

Thanks!

Fraida

[Leigh Stoller]

> I really appreciate the bigger root filesystem on the new standard images! But, I noticed that in experiments with Xen VMs, the resource mapper doesn't give me sufficient physical resources for the larger image size.
>
> For example, if I use the "default" small-lan profile to ask for 3 Ubuntu 22 VMs on an m510 (which won't fit; max 2 VMs with this size image will fit on an m510):

Hi. The m510s have very small disks and date back to a time when our
standard disk image was 16GB root partition. As you have noticed, we
have shifted to 64GB root partitions, making the m510s even more
obsolete. :-)

Here is a profile of mine that you can use as a guide to modify your
own profile.

https://www.cloudlab.us/show-profile.php?uuid=adff18fe-5aa9-11ed-994d-e4434b2381fc

Note that there are a couple of knobs for controlling how many VMs are
put on each physical host. You can incorporate those into your own profile
as needed.

Leigh

[ff...@nyu.edu]

Hi, I thought that the resource mapper was supposed to automatically determine what physical resources are required for a Xen VM experiment? is that not the case?

My experiments don't have specific requirements about which hardware type to use or how many VMs are mapped to a physical server - I was letting the resource mapper take care of that, to use whatever is available. The problem is that sometimes the resource mapper would assign experiments to physical hardware that didn't support the correct number of VMs.

[Leigh Stoller]

> Hi, I thought that the resource mapper was supposed to automatically determine what physical resources are required for a Xen VM experiment? is that not the case?

It takes many things into account, but disk space for XEN VMs is not
one of them.

> My experiments don't have specific requirements about which hardware type to use or how many VMs are mapped to a physical server - I was letting the resource mapper take care of that, to use whatever is available.

It is somewhat rare for users to not specify the node type, since any
measured results can be vastly different, depending on what you get. But
there are definite use cases where any node type will suffice. In this
case it is making things more difficult for you. Sorry about that!

Leigh

[ff...@nyu.edu]

I see. What is your recommendation for this use case - 

• education (intro-level) - need to minimize friction in getting access to resources. I definitely could not expect students to parse https://www.cloudlab.us/resinfo.php on their own to figure out what hardware type to ask for.
• large class - so I am trying to minimize resource usage. I don't want each profile to be hard-coded to a specific cluster and hardware type, since then it would probably not be able to accommodate my entire class. It also seems wasteful (these experiments would be fine running on less-popular hardware types, as long as it is given enough of them... I am trying to be a "good CloudLab citizen" and not monopolize the more capable hardware types when not really needed.)

Is there any way to give the resource mapper "hints" without having to either (A) make the user pick a hardware type at instantiation time, or (B) specify a single hardware type in the experiment profile? 

For example, tell the resource mapper "this experiment needs either 2 m510s or one c220g5 or..." and then let it pick whatever is available.

[ff...@nyu.edu]

I think I had the impression from this statement in the CloudLab manual - 

> Finally, when you allocate virtual nodes, you can specify the amount of CPU and RAM (and, for Xen VMs, virtual disk space) each node will be allocated. CloudLab’s resource assigner will not oversubscribe these quantities.

that the resource mapper would take disk space into account.

[Leigh Stoller]

> > Finally, when you allocate virtual nodes, you can specify the amount of CPU and RAM (and, for Xen VMs, virtual disk space) each node will be allocated. CloudLab’s resource assigner will not oversubscribe these quantities.
>
> that the resource mapper would take disk space into account.

Sorry, that statement is in error, the manual needs to be fixed. The
mapper can be changed to take it into account, but that is not something
that will happen in the near future. I will though make a note of it
for when I have time to work on it.

I am still thinking about your previous message.

Leigh

[Leigh Stoller]

> • large class - so I am trying to minimize resource usage. I don't want each profile to be hard-coded to a specific cluster and hardware type, since then it would probably not be able to accommodate my entire class. It also seems wasteful (these experiments would be fine running on less-popular hardware types, as long as it is given enough of them... I am trying to be a "good CloudLab citizen" and not monopolize the more capable hardware types when not really needed.)

> Is there any way to give the resource mapper "hints" without having to either (A) make the user pick a hardware type at instantiation time, or (B) specify a single hardware type in the experiment profile?

Hi. Just a heads up that I am hoping to install changes to the resource mapper so
that it does not oversubscribe disk space. Most likely tomorrow (Monday) at the
Utah cluster (where the m510s are) for testing in your project. I will let you
know when I get it installed.

Thanks
Leigh

[ff...@nyu.edu]

Sounds great, thank you!

[Leigh Stoller]

> Hi. Just a heads up that I am hoping to install changes to the resource mapper so
> that it does not oversubscribe disk space. Most likely tomorrow (Monday) at the
> Utah cluster (where the m510s are) for testing in your project. I will let you
> know when I get it installed.

And these fixes are now installed and appear to work. Let us know if you
see any problems.

Thanks
Leigh