Server Stuck in Changing Status and Unable to use Created Disk Image

[Adithya Anand]

Hi Powder Community 

I recently created a disk image of my current experiment so I can use it in the future with a different profile. After creating the image I ran into an issue where my server has been stuck in the changing status for over 24 hours, I talked about this in office hours and was recommended to make a post about it. 

In the meantime I figured I would replicate the same experiment using the created disk image but identical profile types (single node running ubuntu 22.04 or any ubuntu) are not allowing me to provide a disk image. Is there a way to provide a disk image on these profiles or do I have to chose a profile that allows me to provide a disk image in the parameterize step. 

As another option, I do plan on creating a profile eventually. Is it possible to use a disk image when creating a profile?

[David M Johnson]

On 9/24/24 07:44, Adithya Anand wrote:
> Hi Powder Community
>
> I recently created a disk image of my current experiment so I can use it
> in the future with a different profile. After creating the image I ran
> into an issue where my server has been stuck in the changing status for
> over 24 hours, I talked about this in office hours and was recommended
> to make a post about it.

> Screenshot 2024-09-24 093949.png

Hi. Most likely some package was installed that broke reboot. What
things did you do (e.g. installed packages, changing OS configuration,
etc) before you captured your image? Did you try to reboot and make
sure everything was as desired, before you captured the image?

The experiment status page allows you to view the node's serial console
log, in the per-node action menu. This could give you useful
information as to where boot is hanging. If you don't understand the
problem, please point us to your experiment without terminating it, and
hopefully we can help before it expires.

> In the meantime I figured I would replicate the same experiment using
> the created disk image but identical profile types (single node running
> ubuntu 22.04 or any ubuntu) are not allowing me to provide a disk image.
> Is there a way to provide a disk image on these profiles or do I have to
> chose a profile that allows me to provide a disk image in the
> parameterize step.
>

> As another option, I do plan on creating a profile eventually. Is it
> possible to use a disk image when creating a profile?

Yes, if you create a custom image, one easy way to use it is to create
your own profile. We recommend you create your profile using a geni-lib
python script. Please take a look at the documentation, especially this
example profile:
https://docs.powderwireless.net/geni-lib.html#%28part._geni-lib-example-os-install-scripts%29
. It shows you how to request a single node with a specific disk image
and boot-time configuration scripts that run at experiment startup.

David

[Adithya Anand]

I think the issue might be with how I moved the files when I was creating the disk image. I moved all the directories I downloaded into 'usr/local'. I am thinking this as the profile that I have created based on the disk image is also not working and stuck in the changing state. Is there an issue with how I moved the files in order to create that disk image, basically is there anything wrong with moving all the folders into 'usr/local'. I am going to retry the entire experiment on a new node, but this time start all the downloads from the /opt location. 

I see the 'portal log' and 'logs'. In the 'portal log' I see the following image, maybe this could be causing the error. I am not exactly sure why it would though, as the server was at least booting up before. 

https://www.powderwireless.net/spewlogfile.php?logfile=235ed3139bc40e45ae07de8d5a22b7870d4c4296

This is a link to the full portal log, im not sure if it is accessible. 

[Adithya Anand]

This is the log file I got from the failed experiment based on my disk image

https://www.emulab.net/spewlogfile.php3?logfile=cd75c6d126ef7ef5011dbb8f5b3a61c4ad87e62e

On Tuesday, September 24, 2024 at 7:30:58 PM UTC-4 john...@flux.utah.edu wrote:

[Mike Hibler]

As David said, this is almost certainly the consequences of some package
you loaded on the system before you made the image. It should not have
anything to do with the imaging process or where you put your data. To
verify this, do as David suggested and load up your packages and then try
to reboot the node *before* you take a snapshot.

The console for pc489 (from the logfile you give below) shows that the
node was imaged and started to boot Linux. It then hung on a startup job
and eventually timed out and dropped to the console login prompt showing
the hostname as "localhost". This generally means that the control network
did not get correctly configured.

> --
> You received this message because you are subscribed to the Google Groups
> "Powder Users" group.
> To unsubscribe from this group and stop receiving emails from it, send an email
> to powder-users...@googlegroups.com.
> To view this discussion on the web visit https://groups.google.com/d/msgid/
> powder-users/ec396d31-b672-42a8-a55f-eed363183cafn%40googlegroups.com.

[Adithya Anand]

How would I " load up your packages and then try to reboot the node *before* you take a snapshot". I dont think this would mean me reinstalling the packages.  

[David M Johnson]

We were suggesting that you recreate your disk image from scratch,
installing packages and performing customizations --- but before
capturing the disk image, this time, try rebooting the node first, and
make sure it boots post-changes.

David

> https://groups.google.com/d/msgid/ <https://groups.google.com/d/msgid/>
> >
> powder-users/ec396d31-b672-42a8-a55f-eed363183cafn%40googlegroups.com <http://40googlegroups.com>.

>
> --
> You received this message because you are subscribed to the Google
> Groups "Powder Users" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an email to powder-users...@googlegroups.com

> <mailto:powder-users...@googlegroups.com>.

> To view this discussion on the web visit

> https://groups.google.com/d/msgid/powder-users/a51e9363-dacb-4cef-848a-f2be02af8a44n%40googlegroups.com <https://groups.google.com/d/msgid/powder-users/a51e9363-dacb-4cef-848a-f2be02af8a44n%40googlegroups.com?utm_medium=email&utm_source=footer>.