Clemson Node clgpu001 (c4130) becomes unresponsive

[Umakant Kulkarni]

Hello,

I'm running following experiment: https://www.cloudlab.us/status.php?uuid=2e7e795e-b3d8-11ec-b318-e4434b2381fc on clemson cloudlab server with node clgpu001 of type c4130.

I'm running some machine learning models with Tesla GPUs. But, while the experiment is in progress, the node becomes unresponsive and I get exit out of ssh session. Only way to resume is to restart the node from cloudlab portal. FYI, on cloudlab portal, the node is still shown in "Ready/Green" state. Currently, the node is in the same state in case you want to take a look at the logs.

The same program runs fine on other node (ibm8335). So, I assume issue not related to the underlying program.

Is there any issue with this node or GPUs?

Thanks,

Umakant 

[Leigh Stoller]

> I'm running some machine learning models with Tesla GPUs. But, while the experiment is in progress, the node becomes unresponsive and I get exit out of ssh session. Only way to resume is to restart the node from cloudlab portal. FYI, on cloudlab portal, the node is still shown in "Ready/Green" state. Currently, the node is in the same state in case you want to take a look at the logs.

Hi. We will look at the node today, please leave it as is. Thanks!

Leigh

[Umakant Kulkarni]

Sure, its in that state.

Thanks, again,

Umakant

[Umakant Kulkarni]

Hi,

Did you get a chance to look at this issue?

-Umakant

[Leigh Stoller]

> Did you get a chance to look at this issue?

Hi. This should be resolved as of a few minutes ago. Some faulty
hardware on the node was replaced. Let us know if you have further
problems.

Leigh

[Umakant Kulkarni]

Oh, Thank you so much!

Let me try the experiment again.

Regards,

Umakant

[Umakant Kulkarni]

Hi, 

I'm still facing the same issue even while simply installing the packages.

[Leigh Stoller]

> I'm still facing the same issue even while simply installing the packages.

OK, we are looking at it again. Sorry about that.

Leigh

[Umakant Kulkarni]

No worries and thank you again!

[Umakant Kulkarni]

Hi,

Did you get a chance to look at this issue?

-Umakant

[Leigh Stoller]

>
> Did you get a chance to look at this issue?

Hi. Unfortunately we are not going to be able to fix this node today,
we are going to have to take it offline for diagnostics. Can you please
terminate your experiment so we can have control of it back.

Thanks!
Leigh

[Umakant Kulkarni]

Sure, I've terminated my experiment!

Thanks,

Umakant

[Rajesh Shashi Kumar]

Hi,

Was there a resolution on this thread?

I'm facing the same issue on the following c4130 node in the Wisconsin cluster from the past few days. Machine becomes unresponsive after every 15-20 minutes. A reboot from the dashboard is necessary to bring it to usable state.

ID: node-0
Node: c4130-110133
Type: c4130
Cluster: Wisc

Thanks,
Rajesh Shashi Kumar

[Umakant Kulkarni]

I just resolved this issue by making only one GPU visible to CUDA.

Added following line to ~/.bashrc:

export CUDA_VISIBLE_DEVICES=0

where 0 is the GPU device #

[Rajesh Shashi Kumar]

I found that this issue does not exist on Ubuntu 18.04 but only on Ubunti 20.04. Latest CUDA in both cases.

Used the following before CUDA install to work around the NetworkManager issue
# Ubuntu 16.04
sudo ln -s /dev/null /etc/systemd/system/NetworkManager.service

Thanks,
Rajesh Shashi Kumar