User-Controlled Switches and Configurable Layer-1 Topologies Now Available

[Robert P Ricci]

We are happy to announce two new features available at the Utah
CloudlLab site: the ability for users to fully control Ethernet
switches, and the ability to directly 'wire' together ports on the xl170
nodes and user-controlled switches at layer 1.

User-allocated switches are treated similarly to the way we treat
servers: the switches appear as nodes in your topology, and you 'wire'
them to PCs and each other using point-to-point layer-1 links. When one
of these switches is allocated to you, you are the exclusive user, just
like you are for a raw PC, and you have ssh access with full
administrative control. This means that you are free to enable and
disable features, tweak parameters, reconfigure as will, etc.  You will
be given the switches in a 'clean' state (we do little configuration on
them), and can reload and reboot them like you would do with a server.

We currently have six Dell S4048 switches (type 'dell-s4048') available,
and will follow up with two Mellanox SN2410 switches (type
'mlnx-sn2410') in the future. Each switch will have 16 to 48 ports
available to connect to.

You can give this a try with the following profile:

    https://cloudlab.us/p/PortalProfiles/layer1-sw-2pcs

It is also possible to directly 'wire' two xl170 nodes together in this
manner: you can only connect two NICs, but the one-way latency between
them will be only 28ns, and they are effectively connected with a simple
wire: no Ethernet frame processing in the middle. A demonstration of
this capability can be found at:

    https://cloudlab.us/p/PortalProfiles/layer1-2pcs

Note that all layer-1 links run at 10 Gbps.

This is a new feature, and we still have more planned: interconnection
between the user-allocated switches at 40Gbps, loading of custom switch
operating systems, 'taps' for high-performance packet capture, and a
deployment at Wisconsin. So if you use the feature, please keep in touch
with so that we can learn how well it's working for you!

You can find the user manual for the Dell switches here:

    http://downloads.dell.com/manuals/all-products/esuprt_ser_stor_net/esuprt_networking/esuprt_net_fxd_prt_swtchs/force10-s4048-on_white-papers1_en-us.pdf

... and for the Mellanox switches here:

    http://www.mellanox.com/related-docs/prod_management_software/MLNX-OS_ETH_v3_6_3508_UM.pdf

[PS: We are still finishing running all of the wires, but we have enough available for people to starting trying it out.]

[Lucas Nussbaum]

On Monday, April 23, 2018 at 6:35:09 PM UTC+2, Robert P Ricci wrote:

We are happy to announce two new features available at the Utah
CloudlLab site: the ability for users to fully control Ethernet
switches, and the ability to directly 'wire' together ports on the xl170
nodes and user-controlled switches at layer 1.

Very nice! I'm curious: are you going to allow changing the the OS from

the Dell OS, to third party ones (such as Cumulus Linux), or to Dell's

new OS 10? I looked into that some time ago and it looked possible

(using the ONIE protocol instead of PXE etc.)

I'm not sure any of the alternatives really has a licensing scheme that

is compatible with use on CloudLab though. I know Dell open sourced

their OS 10 in an "Open Edition" but it's not entirely clear if this

covers everything needed to run on a switch.

--

| Lucas Nussbaum                 Grid'5000 Technical Director |

| lucas.n...@loria.fr                      LORIA / RESIST |

| http://members.loria.fr/lnussbaum/        +33 3 54 95 86 19 |

[Robert P Ricci]

Yes, it is our intention to allow people to switch OSes - we specifically picked the models that we did because they had available open source OSes with reasonably good looking SDKs. In the case of Mellanox, that's Open Net Linux, and in the case of the Dells, it's OPX. And yes, to the best of our knowledge, it's like to take some work on the part of the user to get the functionality they need on top of those bare-bones OSes.

However: doing so has turned out to be trickier than we had anticipated, so we do not have a current ETA. As it turns out, images for the Dell switches are spread across 7 different partitions, and for Mellanox, 12! How much we prioritize (or don't) this feature will be in large part a function of how much people tell us they need it. :)

On Mon, Apr 23, 2018 at 11:21 PM Lucas Nussbaum <lucas.n...@loria.fr> wrote:

On 23/04/18 at 16:34 +0000, Robert P Ricci wrote:
> We are happy to announce two new features available at the Utah
> CloudlLab site: the ability for users to fully control Ethernet
> switches, and the ability to directly 'wire' together ports on the xl170
> nodes and user-controlled switches at layer 1.

[Kunal Mahajan]

This is great. Exactly what I was looking for. 

Is the feature of interconnection between the user-allocated switches at 40Gbps available?

[Leigh Stoller]

at 8:42 AM, Kunal Mahajan <kunalma...@gmail.com> wrote:

> This is great. Exactly what I was looking for.
>
> Is the feature of interconnection between the user-allocated switches at 40Gbps available?

Hi, we have 10G only on the user allocatable switches … sorry.

Leigh

[Mike Hibler]

The plan is to directly (not through the layer-1 switch) interconnect them
at 40 (Dell) or 100Gb (Mellanox) in a fixed topology. But we have not
reached that point.

> >> | lucas.n...@loria.fr <javascript:> LORIA / RESIST |

> >> | http://members.loria.fr/lnussbaum/ +33 3 54 95 86 19 |
> >>
> >
>

> --
> You received this message because you are subscribed to the Google Groups "cloudlab-users" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to cloudlab-user...@googlegroups.com.
> To post to this group, send email to cloudla...@googlegroups.com.
> To view this discussion on the web visit https://groups.google.com/d/msgid/cloudlab-users/9053e66d-6311-4ff1-b1ae-3b3339d87a5a%40googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.

[Zain Ruan]

Dear cloudlab staff,

The user-controlled switch is very useful. Mike mentioned that the ultimate goal is to use a fixed topology to achieve 40/100 Gbps interconnection bandwidth. Just curious if there's any update or plan on this.

Best,

Zain