Cannot SSH into my nodes

[Emil Abbasov]

Hi,

For the last few days, I am not able to SSH into my nodes, either from my computer or from the CloudLab page itself (the "shell" option in the menu). It says:

Unable to connect to c220g2-010820.wisc.cloudlab.us:22

So, I reboot and it works for about 10min or so, and then the same thing. What is going on?

https://www.cloudlab.us/status.php?uuid=bf43bbc4-c8e9-11ed-9f39-e4434b2381fc

Emil

[Leigh Stoller]

Hi. See this message (and the whole thread).

https://groups.google.com/g/cloudlab-users/c/B6rNj7Vhltk/m/rwkHf_kwAgAJ

There is a good chance that the NetworkManager got installed and is
interfering with the Cloudlab control network setup.

Leigh

[Emil Abbasov]

Tried all this. Not that issue, I think. Getting this:

root@server:/# systemctl disable NetworkManager
Unit /etc/systemd/system/NetworkManager.service is masked, ignoring.
root@server:/# sudo ln -s /dev/null /etc/systemd/system/NetworkManager.service
ln: failed to create symbolic link '/etc/systemd/system/NetworkManager.service': File exists
 

[Leigh Stoller]

HI. Did you put your nodes into recovery mode? If so, you are not
actually operating on the disk, /etc is on a memory based filesystem.

More info here: https://gitlab.flux.utah.edu/emulab/emulab-devel/-/wikis/faq/Using-the-Testbed/Using-the-Recovery-MFS

Leigh

[Emil Abbasov]

Yes, I followed the guidelines. Still getting the above message (NetworkManager is masked, ignroing). 

In fact, the nodes are still in the recovery mode. 

Emil

[David M Johnson]

Since the nodes were still in recovery mode, I took a look at the last
few boots on c220g2-010831 (e.g. `journalctl -b-1 -x`) after following
the instructions in the wiki entry Leigh pointed you to, to mount the
disk and chroot inside. Each log ends with stuff like

Apr 11 15:03:23 gnome-shell[1493]: Screen lock is locked down, not locking
Apr 11 15:03:23 systemd[1]: Reached target Sleep.

I see you are doing stuff with changing CPU settings; maybe you are
(unintentionally?) setting up automatic suspend along with it?

Either way, better check the node's console logs next time this happens
and see if the system has suspended, and go from there.

David

> https://gitlab.flux.utah.edu/emulab/emulab-devel/-/wikis/faq/Using-the-Testbed/Using-the-Recovery-MFS <https://gitlab.flux.utah.edu/emulab/emulab-devel/-/wikis/faq/Using-the-Testbed/Using-the-Recovery-MFS>

>
> Leigh
>
> > On Apr 11, 2023, at 2:55 PM, Emil Abbasov <emi...@gmail.com> wrote:
> >
> > root@server:/# systemctl disable NetworkManager
> > Unit /etc/systemd/system/NetworkManager.service is masked, ignoring.
> > root@server:/# sudo ln -s /dev/null
> /etc/systemd/system/NetworkManager.service
> > ln: failed to create symbolic link
> '/etc/systemd/system/NetworkManager.service': File exists
>

> --
> You received this message because you are subscribed to the Google
> Groups "cloudlab-users" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an email to cloudlab-user...@googlegroups.com
> <mailto:cloudlab-user...@googlegroups.com>.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/cloudlab-users/ae7856de-fa6c-443f-8fff-8c2c53403a60n%40googlegroups.com <https://groups.google.com/d/msgid/cloudlab-users/ae7856de-fa6c-443f-8fff-8c2c53403a60n%40googlegroups.com?utm_medium=email&utm_source=footer>.