Suspected experiment-network issue on er010 in experiment mrashid2-296178

[Md Hasanur Rashid]

Dear Concerned,

I’m seeing what looks like an experiment-network issue on one node in my Utah experiment and wanted to ask for help checking it.

Experiment details:

• Experiment: mrashid2-296178
• Profile: hpc_lustre_2_15_5
• Project: DIRR
• Affected node: er010.utah.cloudlab.us

Suspected issue:

• The experiment-network interface on er010 appears to be down or disconnected.
• The node is reachable over normal SSH, but its 10.10.1.x interface cannot communicate with other nodes in the experiment network.

Why I believe this:

• On er010, ens1f0 has the expected 10.10.1.7/24 IP, but shows NO-CARRIER, state DOWN, and Link detected: no.
• er010 cannot ping 10.10.1.1.
• A healthy peer (er114) also cannot ping 10.10.1.7.
• Neighbor/ARP resolution is incomplete on both sides.

Commands and outputs:

On er010:

ssh mras...@er010.utah.cloudlab.us 'sudo -i bash -lc " ip -4 addr show ens1f0 ip link show ens1f0 ethtool ens1f0 2>/dev/null | egrep -i \"Speed|Duplex|Link detected\" ping -c 2 10.10.1.1 ip neigh show dev ens1f0 "'

Output:

2: ens1f0: <NO-CARRIER,BROADCAST,MULTICAST,UP> ... state DOWN ... inet 10.10.1.7/24 ... Speed: Unknown! Duplex: Unknown! (255) Link detected: no PING 10.10.1.1 ... From 10.10.1.7 Destination Host Unreachable 10.10.1.1 FAILED

From er114:

ssh mras...@er114.utah.cloudlab.us 'sudo -i bash -lc " ping -c 2 10.10.1.7 arp -n | grep 10.10.1.7 || true "'

Output:

PING 10.10.1.7 ... From 10.10.1.1 Destination Host Unreachable 10.10.1.7 (incomplete) ens1f0

Why this seems administrator-side:

• The node itself is reachable via SSH, so it is not fully down.
• The issue seems isolated to the experiment-network port/interface.
• Since the interface has no carrier and peers cannot resolve each other at all, this looks more like a provisioning, wiring, switch-port, or NIC-level problem than something I can fix from within the OS.

Kindly look into this issue at your earliest convenience. Let me know if you need any further information.

Best regards,
Hasan

[Mike Hibler]

The port was down on the switch when I first looked. After a couple of
disable/enable cycles on the switch, the link came up.

Note that we do not support CentOS Stream 8 any longer, so this is not
something we are going to diagnose further. I suspect an older driver
or kernel problem.

> --
> You received this message because you are subscribed to the Google Groups
> "cloudlab-users" group.
> To unsubscribe from this group and stop receiving emails from it, send an email
> to cloudlab-user...@googlegroups.com.
> To view this discussion visit https://groups.google.com/d/msgid/cloudlab-users/
> 643d7307-0906-4aba-85ab-3427f110075an%40googlegroups.com.