Long packet gaps in c6620 cluster

[john.ou...@gmail.com]

I'm starting a new thread to discuss the performance problems I am experiencing on the c6620 cluster for this experiment:

https://www.cloudlab.us/status.php?uuid=79312f89-0e45-4585-ab0d-ef160ca7768f

which has 15 nodes on one of the c6620 switches and 35 nodes on the other. The top-level problem I am seeing is significantly worse tail latency in network-intensive workloads than I measured before the c6620 reconfiguration (where the nodes were all on one switch).

I initially thought that priority queues were not properly configured on the switches, but additional measurements showed that this is not the problem.

In addition, I do not believe that oversubscription of the TOR uplinks is the problem, for reasons I'll present shortly, and the additional latency of passing through the core switch is insignificant compared to the delays I'm seeing.

There seems to be something more fundamental going wrong with one or more of the switches.

What I am seeing is significant gaps (more than 500 usecs in some cases) where a node A on one switch receives *zero* packets from any node on the other switch, even though dozens of packets have been transmitted by the NICs of multiple nodes on the other switch. During this time, other nodes on A's switch receive lots of packets from a variety of nodes on the other switch, and A receives lots of packets from other nodes on its switch. This suggests that the problem isn't link bandwidth; it's something about the node (but the problem occurs on a variety of nodes at different times).

I'm wondering if something is going wrong with the switch routing tables, causing a routing entry for A to be lost in the core switch, resulting in an expensive recalculation, during which time no packets can be forwarded to A? However, I have also noticed that once A starts receiving packets again, it doesn't get a flood: packets gradually start trickling in, and bandwidth ramps up over 50-100 usecs to drain the backlog; this doesn't seem consistent with a cache refill operation, after which I would expect an immediate ramp-up.

Unfortunately my knowledge of switches and routers is pretty limited....

I will need to end my 50-node experiment at the end of this week; does anyone have time to work on this with me this week, while we can still run experiments to reproduce the problem?

-John-

[John Ousterhout]

TL;DR I think I know what the problem is; can you give me switch console access so I can make the configuration changes to fix it?

I have spent the last day gathering a bunch more measurements and having a long (and very interesting!) conversation with Claude about the performance problem I'm seeing. Together I think we've figured out the problem:

* I suspect that the uplinks between the leaf switches and the core switches are not configured to classify packets on ingress. As a result, DSCP is ignored on all of these packets and all the packets passing from one switch to another end up in priority queue 0 on the downlink egress ports.

* When a node experiences high incast, it can go a long time without receiving any packets from queue 0. This starves all packets arriving from the other switch, regardless of their intended priority.

Thus I think we need to configure classification on the switch uplinks.

Can you give me console access to both the leaf switches (as you have done in the past) so I can configure this and see if it fixes my problems? If it does, I will send you patches to make this permanent, as I did for the original patches to enable priority queues. You can create the ssh tunnels on node0 of this experiment:

https://www.cloudlab.us/status.php?uuid=79312f89-0e45-4585-ab0d-ef160ca7768f

Also, it would be great if you could save the switch configuration before giving me access, in the (hopefully very unlikely) event that I mess things up.

Also, can you tell me which 4 ports on each switch correspond to the uplinks?

-John-

--
You received this message because you are subscribed to the Google Groups "cloudlab-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to cloudlab-user...@googlegroups.com.
To view this discussion visit https://groups.google.com/d/msgid/cloudlab-users/71e330dc-523b-43be-a005-06095c6ca59cn%40googlegroups.com.

[John Ousterhout]

A bit more followup on this for Mike and Aleks:

When we configured the c6620 for priority queues in November of last year, Aleks requested that the uplink ports *not* be configured, so the patches I created did not configure them.

That was fine when all of the nodes were on a single switch, but now that the nodes are split across two switches the uplink ports also need to be configured. They only need partial configuration: they need to classify incoming packets so they can be distributed among priority queues, but they don't need to have priority queues themselves for packets being transmitted upwards. Homa packets will not get any special priority when being transmitted upwards, so Homa's use of priorities should have no adverse consequences on other experiments running concurrently on other nodes (though they will probably impact Homa if they congest the oversubscribed uplinks).

-John-

[Aleksander Maricq]

Glad you were able to sleuth this out!  We'll set up access for you tomorrow, hopefully this does the trick.