Re: pc176 unavailable despite no active experiment allocation

[Mike Hibler]

Sorry for the delay. I have freed up pc176 (it was in an odd state) and
it should be available again. Thanks for pointing this out.

On Sun, Apr 05, 2026 at 12:27:24AM -0400, Sandeep Bal wrote:
> Hello CloudLab Support and CloudLab Users,
>
> I am writing to report an issue with node pc176.
>
> pc176 appears to be unavailable even though it is not currently being used by
> anyone in an experiment. I was advised to report this so the team can look into
> it and fix the issue.
>
> For context, I was also told that pc177 may have a hardware issue and therefore
> may not be allocatable, but the main issue I am reporting here is that pc176 is
> unavailable despite not being allocated to any active experiment.
>
> My CloudLab user ID: sandman7
> Project name: PRATE
> Account email: sb...@umass.edu 
>
> Please let me know if you need any additional details from my side.
>
> Best,
> Sandeep
>

[Mike Hibler]

Got stuck in the same place. I freed it again so you can use it while I figure
out why it is winding up where it does.

On Mon, Apr 06, 2026 at 05:52:06PM -0400, Sandeep Bal wrote:
> Hi Mike,
>
> Sorry for bothering you again, but I am facing issues with the pc176 again. 
>
> I was able to get the pc176 experiment started. I was able to get some power
> data from the node, but wasn't able to talk to the VCK5000 through xbutil or
> any other mechanism that have already worked on other nodes.But then when I
> tried to terminate the experiment so that I could try to get it later and test
> if I am doing something wrong, I am not even being able to terminate the node.
> It is staying in a cancelled limbo, but nothing is happening. Could you please
> kindly look into this? I have attached a screenshot from my side as a reference
> for you as well.
>
> Hoping to hear from you soon. Thanks in advance.
>
> Regards,
> Sandeep Bal
> 32991085
>
> image.png
>
> On Mon, Apr 6, 2026 at 12:00 PM Sandeep Bal <sb...@umass.edu> wrote:
>
> Hello Mike,
>
> Hope you’re well.
>
> Thanks for your support. Have a great day ahead!
>
> Regards,
> Sandeep Bal

[Mike Hibler]

The node is getting hung trying to reset the FPGA on experiment termination.
I tried manually running the FPGA reset script but it show a bunch of
warnings (errors?) and eventually hangs.

I am going to turn this over to Suranga now. The node is in the "hwdown"
experiment. I don't know anything about the FPGA reset path, that was probably
set up by Hakan or Leigh (now retired).

On Mon, Apr 06, 2026 at 06:47:14PM -0400, Sandeep Bal wrote:
> Just wanted to say that it is still in that weird cancelled limbo on my side. I
> still can't get the experiment.
>
> image.png

[Handagala, Suranga]

Hi all, I disabled the reset logic in the tcl script. @Mike, can you free up pc176 again? It shouldn't give us any trouble this time since the reset script won’t interact with the FPGA. We should be able to manually reset the FPGA if required.

> --
> You received this message because you are subscribed to the Google Groups "cloudlab-users" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to cloudlab-user...@googlegroups.com.
> To view this discussion visit https://groups.google.com/d/msgid/cloudlab-users/20260407150516.GB4973%40flux.utah.edu.

[Mike Hibler]

I had already disabled the DB feature that causes it to invoke the script,
just so that I could swap it out.

I thought you might want to check and make sure the FGPA is working correctly
before we go further. But if you are happy with it, I will go ahead and
release the node!

On Tue, Apr 07, 2026 at 05:36:16PM +0000, Handagala, Suranga wrote:
> Hi all, I disabled the reset logic in the tcl script. @Mike, can you free up pc176 again? It shouldn't give us any trouble this time since the reset script won???t interact with the FPGA. We should be able to manually reset the FPGA if required.

> >>> Hope you???re well.

> >>>
> >>> Thanks for your support. Have a great day ahead!
> >>>
> >>> Regards,
> >>> Sandeep Bal
> >>>

> >>> On Mon, Apr 6, 2026 at 11:46???AM Mike Hibler <mi...@flux.utah.edu>

[Mike Hibler]

Okay Sandeep, pc176 has been freed. Should be available in a couple of minutes.

[Handagala, Suranga]

Thanks for freeing up that node, Mike. I ran three back-to-back experiments with the node and found no issue. The FPGA had already reset to the factory image, which likely happened before it hung up earlier. @Sandeep, you are all set to allocate it in a new experiment for those base power measurements.

To view this discussion visit https://nam12.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgroups.google.com%2Fd%2Fmsgid%2Fcloudlab-users%2F20260407150516.GB4973%2540flux.utah.edu&data=05%7C02%7Cs.handagala%40northeastern.edu%7C5a1bd4dd64c544e6901908de94dddc93%7Ca8eec281aaa34daeac9b9a398b9215e7%7C0%7C0%7C639111877781287433%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=NDgsggJJfL3KtcsQVDP%2BeGlXHtdYffc3MCF%2Bo%2Bi4A4k%3D&reserved=0.