Reservation exists, but nodes not available

[john.ou...@gmail.com]

I have had a reservation in effect for almost a week, and I had an experiment running that used all of the xl170 nodes available with the reservation (7). I stopped that experiment and attempted to start a new 7-node experiment in order to get a different kernel image, but now I get errors saying there are only 6 nodes available. I know it can take a while for nodes to free after an experiment terminates, but it has now been almost an hour since the experiment ended. Furthermore, when I look at the resource availability Web page (https://www.cloudlab.us/resinfo.php) it shows 7 xl170 nodes available. Why is it that I still can't start an experiment?

-John-

[John Ousterhout]

I have now been able to start an experiment with 5 nodes (for some reason the number of available nodes dropped from 6 to 5 since the time I sent my last email):

https://www.cloudlab.us/status.php?uuid=273a9eb1-28cb-470f-9b3a-38df0cd35cef

Can you make the other 2 nodes in my reservation available and let me know so I can restart with all 7 nodes? Thanks.

-John-

On Tue, Mar 10, 2026 at 3:17 PM john.ou...@gmail.com <john.ou...@gmail.com> wrote:

I have had a reservation in effect for almost a week, and I had an experiment running that used all of the xl170 nodes available with the reservation (7). I stopped that experiment and attempted to start a new 7-node experiment in order to get a different kernel image, but now I get errors saying there are only 6 nodes available. I know it can take a while for nodes to free after an experiment terminates, but it has now been almost an hour since the experiment ended. Furthermore, when I look at the resource availability Web page (https://www.cloudlab.us/resinfo.php) it shows 7 xl170 nodes available. Why is it that I still can't start an experiment?

-John-

--
You received this message because you are subscribed to the Google Groups "cloudlab-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to cloudlab-user...@googlegroups.com.
To view this discussion visit https://groups.google.com/d/msgid/cloudlab-users/6c7377e1-eeb0-49fe-af7a-bc090f0042ban%40googlegroups.com.

[Mike Hibler]

In about 5 minutes, try starting up a 2 node experiment (e.g., with the
"small-lan" profile) to make sure there are 2 available. If that works, then
you can terminate it and your 5 node experiment and reinstantiate with 7.

The problem is that we got in a state where the reservation system thought
there were nodes free but they are actually stuck in an experiment that did
not fully terminate due to a switch problem. Cleanly getting rid of that
experiment is going to take some time.

> CAGXJAmzgbJ6pYFcoK5Rbs%2B8U6qYKKAMuznUkDxx9zjMFHA8hvA%40mail.gmail.com.

[John Ousterhout]

That fixed it; thanks!

-John-

To view this discussion visit https://groups.google.com/d/msgid/cloudlab-users/20260311011616.GG59795%40flux.utah.edu.