Setting up heterogeneous links across VMs

[Ertza Warraich]

Hi all, 

I want to setup an experiment with heterogeneous links between different nodes, maybe through inter-region or inter-instance communication? For example VM1-VM2 connection can be 100G, while VM1-VM9 would be 10G. 

What is the best way to achieve this? Note that I want to use the same interface for all communication so I cannot use a different interface of VM1 for VM1-VM2 and a different one for VM1-VM9. 

I am not sure the best way to get this, maybe there is a way to make subnets and have faster connectivity within subnets and then slower across subnets and just use a single type of region and instance? Also, I do need clusters/VMs with GPUs in them for my specific testing, but I am not sure what would be the best/easiest way to achieve this setup.

Any suggestions or help in this regard would be greatly appreciated!

Thank you. 

Best,

Ertza

[Ertza Warraich]

Hi still waiting on any suggestions for this please.

Thank you. 

[Mike Hibler]

We need to clarify some terminology to figure out exactly what you want.

First, Cloudlab primarily allocates bare-metal machines when you create an
instance of a profile (an "experiment"). You can also automatically allocate
virtual machines on top of the bare metal, but that is not as widely used.
When people want VMs or containers, they generally allocate one or more bare
metal machines and setup their own orchestration mechanism, rather than use
the Cloudlab basic, unoptimized setup of VMs on a Xen physical node. So do you
want actual virtual machines when you say "VM" or do you want physical nodes?

Second, what do you mean by "region" and "instance" here?

As far as doing QoS or other shaping on links, we do not provide good
mechanisms for that. From our Emulab origins, we still support "shaped" links
but it is really only for much lower bandwidth links (i.e., 1Gbps).

What Cloudlab can best provide you with are physical servers running Linux,
interconnected via 10-100Gbps physical links (or multiple vlans over physical
links) in arbitrary topologies. You can build your own infrastructure on top
of that base.

I don't know if this answers any of your questions...

> --
> You received this message because you are subscribed to the Google Groups
> "cloudlab-users" group.
> To unsubscribe from this group and stop receiving emails from it, send an email
> to cloudlab-user...@googlegroups.com.
> To view this discussion visit https://groups.google.com/d/msgid/cloudlab-users/
> CAD%2BhOztLUC2macw5qX%2B_%3DHuHC5fCSi16JQ%2BU6TKWBaWH9Yjd0g%40mail.gmail.com.

[Ertza Warraich]

Hi there, thank you for getting back on this. I'll go over the points one by one: 

1. I don't need VMs or virtual machines, physical machines are fine. But the question I asked has no bearing on if I get a physical machine or virtual machine, its about the networking that connects them so I don't know how all this is relevant. 

2. a) By region I mean Utah, Clemson, and other physical regions. What is the terminology I should use for them? Can machines be connected across what I am calling regions? For example two machines from a cluster in Utah talking to two machines in Clemson? 

b) By instance I am referring to a machine. Usually in cloud terminology a VM or a machine is called an instance. 

3. With these multiple vlans and arbitrary topologies, can I achieve the notion of groups that I asked about? I'll try to re-explain, imagine two subnets, each has multiple machines and they're all connected with a switch over 100G links, and then a router connects the two subnets over a 10G link. I think it's pretty standard what I'm asking for and networking 101 but I don't know if it's possible to do it on cloudlab and hence my question. 

Yeah it doesn't answer my question so far but I'm hoping I'm using the right terminology this time around.

To view this discussion visit https://groups.google.com/d/msgid/cloudlab-users/20260126200321.GG39017%40flux.utah.edu.

[Mike Hibler]

We have hundreds of active users of all skill levels and expectations.
So I find it works best to make sure I understand what is wanted while
proactively offering potential answers and clarifications. I was not intending
to be condescending.

On Tue, Jan 27, 2026 at 03:20:54PM -0500, Ertza Warraich wrote:
> Hi there, thank you for getting back on this. I'll go over the points one by
> one: 
>
> 1. I don't need VMs or virtual machines, physical machines are fine. But the
> question I asked has no bearing on if I get a physical machine or virtual
> machine, its about the networking that connects them so I don't know how all
> this is relevant. 
>

You said "VM" specifically (as opposed to "servers" or "machines" or
"instances"), so I wanted to know if that was really what you wanted. It
matters as to the scale of networks you can create and the performance and
variety of link speeds you could potentially expect.

> 2. a) By region I mean Utah, Clemson, and other physical regions. What is the
> terminology I should use for them? Can machines be connected across what I am
> calling regions? For example two machines from a cluster in Utah talking to two
> machines in Clemson? 

These we call "sites". We also sometimes call them "clusters" but that can be
ambiguous. And yes, you can create cross-site network links/lans in your
profile, though you cannot get guaranteed bandwidth on these links.

> b) By instance I am referring to a machine. Usually in cloud terminology a VM
> or a machine is called an instance. 
>

We refer to them as "nodes". "Instance" in Cloudlab is usually used
interchangeably with "experiment" which is roughly your "group"--a collection
of nodes and networks. You instantiate a profile to get an experiment and you
can have multiple instances of the same profile (multiple experiments).

> 3. With these multiple vlans and arbitrary topologies, can I achieve the notion
> of groups that I asked about? I'll try to re-explain, imagine two subnets, each
> has multiple machines and they're all connected with a switch over 100G links,
> and then a router connects the two subnets over a 10G link. I think it's pretty
> standard what I'm asking for and networking 101 but I don't know if it's
> possible to do it on cloudlab and hence my question. 
>

Yes, you can for example setup a "networking 101" dumbbell topology such that
one lan (subnet) is at the Clemson site and the other at the Utah site, with
one member of each lan acting as a router to interconnect the two sides.

What you cannot readily do is specify different link speeds for different nodes.
You select nodes by their "node type" and each node type has specific
characteristics (see https://www.cloudlab.us/portal-hardware.php). Some node
types have multiple interfaces with different speeds (e.g., 25Gb and 100Gb
NICs). But others only have a single available interface or multiple at the
same speed.

> Yeah it doesn't answer my question so far but I'm hoping I'm using the right
> terminology this time around.
>

What I suggest is that you look at the manual and example profiles if you have
not already. Both are linked on the main page. Then try out some topologies at
a single site (leave the cross-site case for later).

> cloudlab-users/20260126200321.GG39017%40flux.utah.edu.

>
> --
> You received this message because you are subscribed to the Google Groups
> "cloudlab-users" group.
> To unsubscribe from this group and stop receiving emails from it, send an email
> to cloudlab-user...@googlegroups.com.
> To view this discussion visit https://groups.google.com/d/msgid/cloudlab-users/

> CAD%2BhOzuBkf-vS9PWpLrb_ntfVJfrxTCGePb86BEMhm8UDDr%2B_A%40mail.gmail.com.