One node not connected to rest in my experiment

[Ertza Warraich]

Hi, I created an experiment on Clemson with 8 nodes, out of the 8 the node 3 does not seem to be connected to the rest as I cannot ping between them, rest all are connected.

I am using the NIC eno33 as I want to use the 25G link and set so in the profile.

This is my experiment link: 

https://www.cloudlab.us/status.php?uuid=6992ded1-4210-11ef-9f39-e4434b2381fc 

Can anyone take a look and suggest something please. 

[Ertza Warraich]

I re-created the experiment and similarly this time node 7 is not connected.

New experiment's link:
https://www.cloudlab.us/status.php?uuid=a9025d61-423c-11ef-9f39-e4434b2381fc

[Sam Pyankov]

Why do I receive those emails?

Sincerely,

Sam Pyankov

spya...@uci.edu

--
You received this message because you are subscribed to the Google Groups "cloudlab-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to cloudlab-user...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/cloudlab-users/a184eea1-ea70-46fa-a237-dda0ae454a66n%40googlegroups.com.

[Mike Hibler]

Maybe something wrong with the interface or cable. I will have the Clemson
folks check it out. FYI, at least a couple of your nodes are constantly
spitting out:
-----
[ 5138.736663] NVRM: GPU 0000:21:00.0 is already bound to nouveau.
[ 5138.736671] NVRM: GPU 0000:e2:00.0 is already bound to nouveau.
[ 5138.736692] NVRM: The NVIDIA probe routine was not called for 2 device(s).
[ 5138.736693] NVRM: This can occur when another driver was loaded and
NVRM: obtained ownership of the NVIDIA device(s).
[ 5138.736693] NVRM: Try unloading the conflicting kernel module (and/or
NVRM: reconfigure your kernel without the conflicting
NVRM: driver(s)), then try loading the NVIDIA kernel module
NVRM: again.
[ 5138.736694] NVRM: No NVIDIA devices probed.
[ 5138.736894] nvidia-nvlink: Unregistered Nvlink Core, major device number 509
----
to "dmesg" and the console. Shouldn't have anything to do with the network
problem.

[Ricardo Guimarães]

I have the same issue sometimes, but with an image I have in the Utah cluster.

I reported it in this conv: https://groups.google.com/g/cloudlab-users/c/7H7ImNsvZyc/m/gS-OWGgXBAAJ.

I have not made any changes to the Mellanox driver and neither explicitly made anything with "bpfilter". I was using this image for a long time and suddenly I started having this issue.

If possible, I would like to share the solution so I can run my large experiments. 

Thankfully,

Ricardo.

[Leigh Stoller]

> I re-created the experiment and similarly this time node 7 is not connected.
>
> New experiment's link:
> https://www.cloudlab.us/status.php?uuid=a9025d61-423c-11ef-9f39-e4434b2381fc

Hi. Take a look and let us know if clgpu014 has its link now.

Thanks!
Leigh

[Ertza Warraich]

Perfect, working now!

Thanks so much. 

--
You received this message because you are subscribed to the Google Groups "cloudlab-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to cloudlab-user...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/cloudlab-users/0D59869A-8BB0-4317-B495-D1D0A5C2284A%40gmail.com.