Small-disk-size images for experiments with many VMs

[ff...@nyu.edu]

Hi CloudLab team!

For the following use case - education, large class size, experiments involving a moderate number of VMs - we have been sticking to the Ubuntu 20 image because of its much smaller disk size than the newer Ubuntu images. 

For this use case the disk space is often the constraint for the resource mapper, so it is our attempt to be a "good CloudLab citizen" and avoid monopolizing more resources than needed. 

Now that Ubuntu 20 is EOL, I was wondering if it would make sense to maintain a parallel "small disk size" version of the latest Ubuntu image? instead of getting stuck with a deprecated OS forever.

[Mike Hibler]

Sorry for the delay in responding, we wanted to talk about this at our
weekly meeting.

In the short term, do what you need to do. If that involves using more
physical machines, then go for it. Don't redesign your assignments or
stick with an old OS because of this. We have some 200 newer general purpose
nodes (c6620s at Utah, r6615 at Clemson) that each have ~1.6TB of fast disk
which you can use.

We don't really have the resources to maintain multiple versions of the
standard images, so medium term the solution might be to give you a "one off"
smaller image that you can update and maintain as a custom image.

The longer term solution will be to adopt a more cloud image style where
there is a small filesystem whose size can be more easily customized,
rather than using the small number of fixed layouts we do today.

> --
> You received this message because you are subscribed to the Google Groups
> "cloudlab-users" group.
> To unsubscribe from this group and stop receiving emails from it, send an email
> to cloudlab-user...@googlegroups.com.
> To view this discussion visit https://groups.google.com/d/msgid/cloudlab-users/
> 5d619c94-e014-4216-bf25-9d2e579f0c30n%40googlegroups.com.

[ff...@nyu.edu]

Hi Mike, thanks for your response and thanks for bringing it up with your team.

I wanted to test the r6615s at Clemson, but it looks like Xen VMs are broken at Clemson.

• I tried to instantiate https://www.cloudlab.us/instantiate.php?project=nyunetworks&profile=education&refspec=refs/heads/learning_switch_22 which brings up 5 Ubuntu 22 VMs, for one hour, and it failed with " No available physical nodes of type ppc64le-vm found".
• I also tried the "official" small-lan profile at Clemson. With 1 VM, it failed with "Experiment setup on the Cloudlab Clemson cluster failed: The following nodes failed to setup: clnode332". 
• With 4 nodes in a LAN, the "official" small-lan profile failed at Clemson with "ERROR: mapper: Cannot mix trivial_ok|emulated with * bw *** Could not create vtop for [Experiment: nyunetworks-PG0/test-4vm-clem] "

(Xen VMs are also broken at APT right now, and this is a recurring issue, see e.g. here and here. VMs are broken on APT more often than not.)

[Mike Hibler]

You will probably need to use the XEN416-64-STD image on those physical
nodes, which is not currently the default for a vnode host.

> 8178aaa9-a6ba-4950-9e7d-7f8fdd67a265n%40googlegroups.com.

[ff...@nyu.edu]

Sorry, I'm not sure what you mean by that. I don't see any reference to  "XEN416-64-STD" anywhere in this mailing list or in the docs, and I don't see any relevant option in the official "small-lan" profile.

Currently the way my "education" profiles bring up Xen VMs is just by something like this: https://github.com/teaching-on-testbeds/hello-cloudlab/blob/main/profile.py

I don't specify a physical node type, "exclusive" property, or anything else, because we don't care, want to use whatever is available, and don't have the sophistication to figure out what is compatible with what - we want to leave it up to the resource mapper. 

[Mike Hibler]

Awful magnanimous of me to offer you a couple hundred nodes that don't work...
:-)

Looks like the Xen 4.16 image is not official yet. Let me sort this out
and get back to you.

On Fri, Aug 22, 2025 at 09:06:58AM -0600, Mike Hibler wrote:
> You will probably need to use the XEN416-64-STD image on those physical
> nodes, which is not currently the default for a vnode host.
>
> On Fri, Aug 22, 2025 at 07:52:40AM -0700, ff...@nyu.edu wrote:
> > Hi Mike, thanks for your response and thanks for bringing it up with your team.
> >

> > I wanted to test the??r6615s at Clemson, but it looks like Xen VMs are broken at
> > Clemson.
> >
> > ??? I tried to instantiate??https://www.cloudlab.us/instantiate.php?project=

> > nyunetworks&profile=education&refspec=refs/heads/learning_switch_22 which
> > brings up 5 Ubuntu 22 VMs, for one hour, and it failed with " No available
> > physical nodes of type ppc64le-vm found".

> > ??? I also tried the "official" small-lan profile at Clemson. With 1 VM, it

> > failed with "Experiment setup on the Cloudlab Clemson cluster failed: The

> > following nodes failed to setup: clnode332".??
> > ??? With 4 nodes in a LAN, the "official" small-lan profile failed at Clemson
> > with "ERROR: mapper: Cannot mix trivial_ok|emulated with * bw ***??Could not

> > create vtop for [Experiment: nyunetworks-PG0/test-4vm-clem] "
> >
> > (Xen VMs are also broken at APT right now, and this is a recurring issue, see
> > e.g. here and here. VMs are broken on APT more often than not.)

> > On Thursday, August 21, 2025 at 5:42:38???PM UTC-4 Mike Hibler wrote:
> >
> > Sorry for the delay in responding, we wanted to talk about this at our
> > weekly meeting.
> >
> > In the short term, do what you need to do. If that involves using more
> > physical machines, then go for it. Don't redesign your assignments or
> > stick with an old OS because of this. We have some 200 newer general
> > purpose
> > nodes (c6620s at Utah, r6615 at Clemson) that each have ~1.6TB of fast disk
> > which you can use.
> >
> > We don't really have the resources to maintain multiple versions of the
> > standard images, so medium term the solution might be to give you a "one
> > off"
> > smaller image that you can update and maintain as a custom image.
> >
> > The longer term solution will be to adopt a more cloud image style where
> > there is a small filesystem whose size can be more easily customized,
> > rather than using the small number of fixed layouts we do today.
> >
> > On Wed, Aug 20, 2025 at 09:02:32AM -0700, ff...@nyu.edu wrote:
> > > Hi CloudLab team!
> > >
> > > For the following use case - education, large class size, experiments
> > involving
> > > a moderate number of VMs - we have been sticking to the Ubuntu 20 image
> > because

> > > of its much smaller disk size than the newer Ubuntu images.??

> > >
> > > For this use case the disk space is often the constraint for the resource
> > > mapper, so it is our attempt to be a "good CloudLab citizen" and avoid

> > > monopolizing more resources than needed.??

> > >
> > > Now that Ubuntu 20 is EOL, I was wondering if it would make sense to
> > maintain a
> > > parallel "small disk size" version of the latest Ubuntu image? instead of
> > > getting stuck with a deprecated OS forever.
> > >
> > > --
> > > You received this message because you are subscribed to the Google Groups
> > > "cloudlab-users" group.
> > > To unsubscribe from this group and stop receiving emails from it, send an
> > email
> > > to cloudlab-user...@googlegroups.com.
> > > To view this discussion visit https://groups.google.com/d/msgid/
> > cloudlab-users/
> > > 5d619c94-e014-4216-bf25-9d2e579f0c30n%40googlegroups.com.
> >
> >
> > --
> > You received this message because you are subscribed to the Google Groups
> > "cloudlab-users" group.
> > To unsubscribe from this group and stop receiving emails from it, send an email
> > to cloudlab-user...@googlegroups.com.
> > To view this discussion visit https://groups.google.com/d/msgid/cloudlab-users/
> > 8178aaa9-a6ba-4950-9e7d-7f8fdd67a265n%40googlegroups.com.
>
> --
> You received this message because you are subscribed to the Google Groups "cloudlab-users" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to cloudlab-user...@googlegroups.com.

> To view this discussion visit https://groups.google.com/d/msgid/cloudlab-users/20250822150658.GD21701%40flux.utah.edu.