Unable To Powercycle c6525-100g

[Simon Bertron]

I have a running experiment and when I tried to powercycle all of the nodes some of them came back up but some of them seem to be stuck (for an hour or more) in the PXEBOOTING state.

I tried restarting the entire experiment but in both instances of the experiment this happened (though it was different machines that got stuck each time). Any help in debugging this would be appreciated.

Here is a link to the experiment status page:

https://www.cloudlab.us/status.php?uuid=795a1151-9d0a-4efb-8ae3-1940a00c8cc0#

[Mike Hibler]

I am looking at this, please don't do anything further.

> --
> You received this message because you are subscribed to the Google Groups
> "cloudlab-users" group.
> To unsubscribe from this group and stop receiving emails from it, send an email
> to cloudlab-user...@googlegroups.com.
> To view this discussion visit https://groups.google.com/d/msgid/cloudlab-users/
> 285fdbd7-5300-48e7-b4c1-a5f8167ca76cn%40googlegroups.com.

[Mike Hibler]

Were you just instantiating the experiment and things went wrong?
Or were you reloading the OS on nodes?

The one node I am looking at has no OS on either disk.

> To view this discussion visit https://groups.google.com/d/msgid/cloudlab-users/arGOCjyInMgl48Gv%40emulab.net.

[Simon Bertron]

That is interesting. I did not attempt to reload the OS. I was initially
powercycling to try to resolve something that had gone wrong and then in
the second experiment I powercycled just to see if it would work. The
machines all booted okay when I first created the experiments and I
successfully ssh-ed into all of the machines I tried to ssh into.

I try to perform the same actions on every node so it would be odd if I
managed to overwrite the OS on only some of the machines. That being
said, I am reformatting some of the block devices so it isn't impossible
that I broke something. Although the script I am using has worked before.

I have noticed that these machines always have one disk with a single
partition and one disk with 4 partitions. I have been assuming that the
unpartitioned disk and the fourth partition of the partitioned disk are
both safe to reformat. Is that true?

On 9/21/26 16:25, Mike Hibler wrote:
> [You don't often get email from mi...@emulab.net. Learn why this is important at https://aka.ms/LearnAboutSenderIdentification ]

> You received this message because you are subscribed to a topic in the Google Groups "cloudlab-users" group.
> To unsubscribe from this topic, visit https://groups.google.com/d/topic/cloudlab-users/xqhfkfYdK2w/unsubscribe.
> To unsubscribe from this group and all its topics, send an email to cloudlab-user...@googlegroups.com.
> To view this discussion visit https://groups.google.com/d/msgid/cloudlab-users/arGSrLHYgtUH0Ncf%40emulab.net.

[Mike Hibler]

I reloaded the disk on those nodes and everything is up again.

Note that Linux has non-deterministic naming of disk devices, so what
is "nvm0n1" on one boot might be something else on the next boot. So
make sure you use something other than the name to identify which one
has no partitions and which one has four.

Note also that the boot disk partitioning changed between Ubuntu 20
and 22. Where the root filesystem is in partition 1 in older images
(e.g., U20), it is in partition 3 in newer images (e.g., U22).

If you want to gather all the unused disk space into a filesystem,
then you can use the "local dataset" mechanism through your profile
and let our scripts figure it out. See the storage section of the manual:
https://docs.cloudlab.us/advanced-storage.html#(part._advanced-storage)

> To view this discussion visit https://groups.google.com/d/msgid/cloudlab-users/d04fe8e0-4532-4d6e-b138-61a3390ff577%40cornell.edu.

[Simon Bertron]

Thank you!

I check for the existence of the fourth partition in my script to
determine which disk is partitioned (i.e. `if [[ -e "/dev/nvme0n1p4" ]];`).

I didn't know about the "local dataset" mechanism, that is a nice
feature! Unfortunately, I only want a filesystem on the big partition, I
want to treat the unpartitioned device as a block device.

I am using exclusively U20 images so as long as the root filesystem is
consistently on partition 1 in those images I don't think that is the
culprit.

> To view this discussion visit https://groups.google.com/d/msgid/cloudlab-users/arGsbz3Y3uwm2vBB%40emulab.net.

[Mike Hibler]

The boot disk is not always nvme0n1, sometimes it will be nvme1n1 depending
on how Linux names the devices. So do not assume that nvme0n1 is always the
boot disk.

For local datasets, you can create one without a filesystem, but it will still
group all the extra storage into an LVM volume group. Which is probably not
what you want.

> To view this discussion visit https://groups.google.com/d/msgid/cloudlab-users/ebdaed74-cfc3-46f8-bf68-71d01be63651%40cornell.edu.