Networking issues on Utah c6620 nodes

[Leonid Kondrashov]

Hello,

I'm experiencing issues accessing my experiments from my local machine over the control network: everything is very slow (< 1 MB/s). This makes it inconvenient to work remotely. Experiments affected: https://www.cloudlab.us/status.php?uuid=af7b7faa-5c55-4390-aac1-097da051cca8, https://www.cloudlab.us/status.php?uuid=46a80240-e231-4fe7-bd10-57b9b1991f0c.

This is what I see in the iperf3 output when checking the bandwidth to one of the nodes:

Connecting to host er111.utah.cloudlab.us, port 5201
[  5] local <local_IP> port 36330 connected to 128.110.220.111 port 5201
[ ID] Interval           Transfer     Bitrate         Retr  Cwnd
[  5]   0.00-1.00   sec  0.00 Bytes  0.00 bits/sec    6   8.48 KBytes
[  5]   1.00-2.00   sec   128 KBytes  1.05 Mbits/sec    5   5.66 KBytes
[  5]   2.00-3.00   sec  0.00 Bytes  0.00 bits/sec    3   4.24 KBytes
[  5]   3.00-4.00   sec  0.00 Bytes  0.00 bits/sec    3   2.83 KBytes
[  5]   4.00-5.00   sec  0.00 Bytes  0.00 bits/sec    2   2.83 KBytes
[  5]   5.00-6.00   sec  0.00 Bytes  0.00 bits/sec    3   1.41 KBytes
[  5]   6.00-7.00   sec  0.00 Bytes  0.00 bits/sec    2   1.41 KBytes
[  5]   7.00-8.00   sec  0.00 Bytes  0.00 bits/sec    1   2.83 KBytes
[  5]   8.00-9.00   sec  0.00 Bytes  0.00 bits/sec    1   4.24 KBytes
[  5]   9.00-10.00  sec  0.00 Bytes  0.00 bits/sec    2   2.83 KBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.00  sec   128 KBytes   105 Kbits/sec   28             sender
[  5]   0.00-10.20  sec   128 KBytes   103 Kbits/sec                  receiver

iperf Done.

Same result if trying to access it from the Wisconsin cluster and my personal VPS over the control network (overall, Asia, Europe, and US regions tested). The experiment network and connections from other experiments in the same region over the control network are fine (25gbps as advertised).

Regards,

Leonid

[Aleksander Maricq]

Hi Leonid,

Apologies for the delay in response here.  In case you didn't see the message on the portal, we have indeed been seeing low bandwidth and high packet-loss at CloudLab Utah, and have been working on trying to track this problem down this morning.  At this point we are confident it is upstream of us, so we are working with our upstream network provider to figure out what's going on.  We'll send an update once things are back to normal.  Sorry for the inconvenience here.

Best,

 - Aleks

[Aleksander Maricq]

The issue has been identified and fixed, we should be back on the air now.