Ping failed with private IP address, but worked with public IP address

[timch...@gmail.com]

Hi,

I have an experiment(AMD-Cluster) with 8 d6515 nodes based on the default small-lan profile.

I cannot ping to any other nodes with private IP address (10.10.1.xxx) on each node in the cluster. But I can ping or ssh login with public IP address (128.110.155.xxx).

The details on node0:

enp1s0f0  Link encap:Ethernet  HWaddr b0:26:28:74:d4:d0

          inet addr:128.110.155.73  Bcast:128.110.155.255  Mask:255.255.252.0

          inet6 addr: fe80::b226:28ff:fe74:d4d0/64 Scope:Link

          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1

          RX packets:16673 errors:0 dropped:0 overruns:0 frame:0

          TX packets:1821 errors:0 dropped:0 overruns:0 carrier:0

          collisions:0 txqueuelen:1000

          RX bytes:1429441 (1.4 MB)  TX bytes:297135 (297.1 KB)

enp65s0f0 Link encap:Ethernet  HWaddr 1c:34:da:41:cb:5c

          inet addr:10.10.1.3  Bcast:10.10.1.255  Mask:255.255.255.0

          inet6 addr: fe80::1e34:daff:fe41:cb5c/64 Scope:Link

          UP BROADCAST MULTICAST  MTU:9000  Metric:1

          RX packets:0 errors:0 dropped:0 overruns:0 frame:0

          TX packets:3 errors:0 dropped:0 overruns:0 carrier:0

          collisions:0 txqueuelen:1000

          RX bytes:0 (0.0 B)  TX bytes:266 (266.0 B)

Ping on node0 with private IP address:

root@node0:/users/fzhou# ping node1

PING node1-link-1 (10.10.1.2) 56(84) bytes of data.

From node0-link-1 (10.10.1.1) icmp_seq=1 Destination Host Unreachable

From node0-link-1 (10.10.1.1) icmp_seq=2 Destination Host Unreachable

From node0-link-1 (10.10.1.1) icmp_seq=3 Destination Host Unreachable

From node0-link-1 (10.10.1.1) icmp_seq=4 Destination Host Unreachable

^C

--- node1-link-1 ping statistics ---

6 packets transmitted, 0 received, +4 errors, 100% packet loss, time 5062ms

Any suggestions?

Best,

Fang

[Leigh Stoller]

at 11:39 PM, timch...@gmail.com <timch...@gmail.com> wrote:

> I have an experiment(AMD-Cluster) with 8 d6515 nodes based on the default small-lan profile.
>
> I cannot ping to any other nodes with private IP address (10.10.1.xxx) on each node in the cluster. But I can ping or ssh login with public IP address (128.110.155.xxx).

Hi. I see this experiment started on Feb 2. When did the network stop working?

Thanks
Leigh

[timch...@gmail.com]

Hi Leigh,

I did some tests locally and am not sure when the network does not work.

But I just stopped the experiment and started a new experiment. I find nodes are still not reachable with private IP address.

I'm using LAN profile with Ubuntu 16.04 .

fzhou@node0:~$ ping node1

PING node1-link-1 (10.10.1.2) 56(84) bytes of data.

From node0-link-1 (10.10.1.1) icmp_seq=1 Destination Host Unreachable

From node0-link-1 (10.10.1.1) icmp_seq=2 Destination Host Unreachable

From node0-link-1 (10.10.1.1) icmp_seq=3 Destination Host Unreachable

^C

--- node1-link-1 ping statistics ---

4 packets transmitted, 0 received, +3 errors, 100% packet loss, time 3014ms

Best,

Fang

[Mike Hibler]

Ubuntu 16 is too old to recognize the NICs on the d6515s. You should use
Ubuntu 18 or 20 instead.

> --
> You received this message because you are subscribed to the Google Groups
> "cloudlab-users" group.
> To unsubscribe from this group and stop receiving emails from it, send an email
> to cloudlab-user...@googlegroups.com.
> To view this discussion on the web visit https://groups.google.com/d/msgid/
> cloudlab-users/45eb43ac-45cb-41d7-97c8-bc6a20a548a1n%40googlegroups.com.

[timch...@gmail.com]

Hi Mike,

I see. It makes sense. Thank you so much.

Best,

Fang

[timch...@gmail.com]

I try to create experiments based on Ubuntu 18 and 20. However, nodes fail to connect with each other using private IP address in both cases.

I checked the kernel message on one node and found: "[   22.119790] bnxt_en 0000:01:00.1 ens1f1np1: NIC Link is Down". 

Any suggestions?

[Leigh Stoller]

at 11:07 AM, timch...@gmail.com <timch...@gmail.com> wrote:

> I try to create experiments based on Ubuntu 18 and 20. However, nodes fail to connect with each other using private IP address in both cases.
>
> I checked the kernel message on one node and found: "[ 22.119790] bnxt_en 0000:01:00.1 ens1f1np1: NIC Link is Down”.

Hi. This is fixed now …

Leigh