Issue of Private IPs

[Haoyu Li]

Dear CloudLab Team,

We're having another issue in an experiment of one m510 and two d750 nodes (`rubble-optane` in project `lsm-rep`). After initiating the experiment, the network configuration looks strange. For example, node-0 has private IP 10.10.1.2, and node-1 has 10.10.1.1. Moreover, after configuring the private keys on each node with the script in Section 11.5.4 of the user manual, each node will connect to itself when trying to ssh to another. It's a fresh experiment initiated from profile `lsm-rep/test`. The only thing we did is the private key configuration.

Would you mind taking a look? Thank you very much!

Best,

Haoyu

[Leigh Stoller]

> We're having another issue in an experiment of one m510 and two d750 nodes (`rubble-optane` in project `lsm-rep`). After initiating the experiment, the network configuration looks strange. For example, node-0 has private IP 10.10.1.2, and node-1 has 10.10.1.1. Moreover, after configuring the private keys on each node with the script in Section 11.5.4 of the user manual, each node will connect to itself when trying to ssh to another. It's a fresh experiment initiated from profile `lsm-rep/test`. The only thing we did is the private key configuration.

In a case like this, we would need to see your ssh command line, using
the -v option. Append the debug output as well.

Thanks
Leigh

[Haoyu Li]

No problem. I’m pasting ssh -v debug output below.

On node-0:
haoyuli@node-0:~$ ifconfig
eno1: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
 inet 128.110.217.133 netmask 255.255.248.0 broadcast 128.110.223.255
 inet6 fe80::1658:d0ff:fe58:7f22 prefixlen 64 scopeid 0x20<link>
 ether 14:58:d0:58:7f:22 txqueuelen 1000 (Ethernet)
 RX packets 49333 bytes 3106011 (3.1 MB)
 RX errors 0 dropped 0 overruns 0 frame 0
 TX packets 1006 bytes 111013 (111.0 KB)
 TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0

eno1d1: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
 inet 10.10.1.2 netmask 255.255.255.0 broadcast 10.10.1.255
 inet6 fe80::1658:d0ff:fe58:7f23 prefixlen 64 scopeid 0x20<link>
 ether 14:58:d0:58:7f:23 txqueuelen 1000 (Ethernet)
 RX packets 0 bytes 0 (0.0 B)
 RX errors 0 dropped 0 overruns 0 frame 0
 TX packets 23 bytes 1526 (1.5 KB)
 TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0

lo: flags=73<UP,LOOPBACK,RUNNING> mtu 65536
 inet 127.0.0.1 netmask 255.0.0.0
 inet6 ::1 prefixlen 128 scopeid 0x10<host>
 loop txqueuelen 1000 (Local Loopback)
 RX packets 185 bytes 19590 (19.5 KB)
 RX errors 0 dropped 0 overruns 0 frame 0
 TX packets 185 bytes 19590 (19.5 KB)
 TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
haoyuli@node-0:~$ ssh -v node-1
OpenSSH_8.2p1 Ubuntu-4ubuntu0.2, OpenSSL 1.1.1f 31 Mar 2020
debug1: Reading configuration data /etc/ssh/ssh_config
debug1: /etc/ssh/ssh_config line 19: include /etc/ssh/ssh_config.d/*.conf matched no files
debug1: /etc/ssh/ssh_config line 21: Applying options for *
debug1: Connecting to node-1 [10.10.1.1] port 22.
debug1: connect to address 10.10.1.1 port 22: No route to host
ssh: connect to host node-1 port 22: No route to host
haoyuli@node-0:~$ ssh -v 10.10.1.1
OpenSSH_8.2p1 Ubuntu-4ubuntu0.2, OpenSSL 1.1.1f 31 Mar 2020
debug1: Reading configuration data /etc/ssh/ssh_config
debug1: /etc/ssh/ssh_config line 19: include /etc/ssh/ssh_config.d/*.conf matched no files
debug1: /etc/ssh/ssh_config line 21: Applying options for *
debug1: Connecting to 10.10.1.1 [10.10.1.1] port 22.
debug1: connect to address 10.10.1.1 port 22: No route to host
ssh: connect to host 10.10.1.1 port 22: No route to host

On node-1:
haoyuli@node-1:~$ ifconfig
eno12399np0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
 inet 128.110.219.210 netmask 255.255.248.0 broadcast 128.110.223.255
 inet6 fe80::5e6f:69ff:fe63:ffd0 prefixlen 64 scopeid 0x20<link>
 ether 5c:6f:69:63:ff:d0 txqueuelen 1000 (Ethernet)
 RX packets 60184 bytes 3848139 (3.8 MB)
 RX errors 0 dropped 1840 overruns 0 frame 0
 TX packets 2580 bytes 329524 (329.5 KB)
 TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0

lo: flags=73<UP,LOOPBACK,RUNNING> mtu 9000
 inet 10.10.1.1 netmask 255.255.255.0
 inet6 ::1 prefixlen 128 scopeid 0x10<host>
 loop txqueuelen 1000 (Local Loopback)
 RX packets 782 bytes 104468 (104.4 KB)
 RX errors 0 dropped 0 overruns 0 frame 0
 TX packets 782 bytes 104468 (104.4 KB)
 TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
haoyuli@node-1:~$ ssh -v node-0
OpenSSH_8.2p1 Ubuntu-4ubuntu0.2, OpenSSL 1.1.1f 31 Mar 2020
debug1: Reading configuration data /etc/ssh/ssh_config
debug1: /etc/ssh/ssh_config line 19: include /etc/ssh/ssh_config.d/*.conf matched no files
debug1: /etc/ssh/ssh_config line 21: Applying options for *
debug1: Connecting to node-0 [10.10.1.2] port 22.
debug1: Connection established.
debug1: identity file /users/haoyuli/.ssh/id_rsa type 0
debug1: identity file /users/haoyuli/.ssh/id_rsa-cert type -1
debug1: identity file /users/haoyuli/.ssh/id_dsa type -1
debug1: identity file /users/haoyuli/.ssh/id_dsa-cert type -1
debug1: identity file /users/haoyuli/.ssh/id_ecdsa type -1
debug1: identity file /users/haoyuli/.ssh/id_ecdsa-cert type -1
debug1: identity file /users/haoyuli/.ssh/id_ecdsa_sk type -1
debug1: identity file /users/haoyuli/.ssh/id_ecdsa_sk-cert type -1
debug1: identity file /users/haoyuli/.ssh/id_ed25519 type -1
debug1: identity file /users/haoyuli/.ssh/id_ed25519-cert type -1
debug1: identity file /users/haoyuli/.ssh/id_ed25519_sk type -1
debug1: identity file /users/haoyuli/.ssh/id_ed25519_sk-cert type -1
debug1: identity file /users/haoyuli/.ssh/id_xmss type -1
debug1: identity file /users/haoyuli/.ssh/id_xmss-cert type -1
debug1: Local version string SSH-2.0-OpenSSH_8.2p1 Ubuntu-4ubuntu0.2
debug1: Remote protocol version 2.0, remote software version OpenSSH_8.2p1 Ubuntu-4ubuntu0.2
debug1: match: OpenSSH_8.2p1 Ubuntu-4ubuntu0.2 pat OpenSSH* compat 0x04000000
debug1: Authenticating to node-0:22 as 'haoyuli'
debug1: SSH2_MSG_KEXINIT sent
debug1: SSH2_MSG_KEXINIT received
debug1: kex: algorithm: curve25519-sha256
debug1: kex: host key algorithm: ecdsa-sha2-nistp256
debug1: kex: server->client cipher: chacha20...@openssh.com MAC: <implicit> compression: none
debug1: kex: client->server cipher: chacha20...@openssh.com MAC: <implicit> compression: none
debug1: expecting SSH2_MSG_KEX_ECDH_REPLY
debug1: Server host key: ecdsa-sha2-nistp256 SHA256:bB4TBZMIEhrtJN2mb/Mzn8fvSRUBUC+UamxqkhCU7HA
debug1: Host 'node-0' is known and matches the ECDSA host key.
debug1: Found key in /users/haoyuli/.ssh/known_hosts:1
debug1: rekey out after 134217728 blocks
debug1: SSH2_MSG_NEWKEYS sent
debug1: expecting SSH2_MSG_NEWKEYS
debug1: SSH2_MSG_NEWKEYS received
debug1: rekey in after 134217728 blocks
debug1: Will attempt key: /users/haoyuli/.ssh/id_rsa RSA SHA256:bZCEH9CD/wfdI1h+Wkf163JBULTfMWZ2cocRSpL90S4
debug1: Will attempt key: /users/haoyuli/.ssh/id_dsa 
debug1: Will attempt key: /users/haoyuli/.ssh/id_ecdsa 
debug1: Will attempt key: /users/haoyuli/.ssh/id_ecdsa_sk 
debug1: Will attempt key: /users/haoyuli/.ssh/id_ed25519 
debug1: Will attempt key: /users/haoyuli/.ssh/id_ed25519_sk 
debug1: Will attempt key: /users/haoyuli/.ssh/id_xmss 
debug1: SSH2_MSG_EXT_INFO received
debug1: kex_input_ext_info: server-sig-algs=<ssh-ed25519,sk-ssh-...@openssh.com,ssh-rsa,rsa-sha2-256,rsa-sha2-512,ssh-dss,ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521,sk-ecdsa-sh...@openssh.com>
debug1: SSH2_MSG_SERVICE_ACCEPT received
debug1: Authentications that can continue: publickey
debug1: Next authentication method: publickey
debug1: Offering public key: /users/haoyuli/.ssh/id_rsa RSA SHA256:bZCEH9CD/wfdI1h+Wkf163JBULTfMWZ2cocRSpL90S4
debug1: Server accepts key: /users/haoyuli/.ssh/id_rsa RSA SHA256:bZCEH9CD/wfdI1h+Wkf163JBULTfMWZ2cocRSpL90S4
debug1: Authentication succeeded (publickey).
Authenticated to node-0 ([10.10.1.2]:22).
debug1: channel 0: new [client-session]
debug1: Requesting no-more-...@openssh.com
debug1: Entering interactive session.
debug1: pledge: network
debug1: client_input_global_request: rtype hostk...@openssh.com want_reply 0
debug1: Remote: /users/haoyuli/.ssh/authorized_keys:9: key options: agent-forwarding port-forwarding pty user-rc x11-forwarding
debug1: Remote: /users/haoyuli/.ssh/authorized_keys:9: key options: agent-forwarding port-forwarding pty user-rc x11-forwarding
debug1: Sending environment.
debug1: Sending env LANG = en_US.UTF-8
Welcome to Ubuntu 20.04 LTS (GNU/Linux 5.4.0-100-generic x86_64)

 * Documentation: https://help.ubuntu.com
 * Management: https://landscape.canonical.com
 * Support: https://ubuntu.com/advantage

 * Super-optimized for small spaces - read how we shrank the memory
 footprint of MicroK8s to make it the smallest full K8s around.

https://ubuntu.com/blog/microk8s-memory-optimisation
New release '22.04.1 LTS' available.
Run 'do-release-upgrade' to upgrade to it.

Last login: Tue Jan 3 11:30:54 2023 from 10.10.1.1
haoyuli@node-1:~$ exit
logout
debug1: client_input_channel_req: channel 0 rtype exit-status reply 0
debug1: client_input_channel_req: channel 0 rtype e...@openssh.com reply 0
debug1: channel 0: free: client-session, nchannels 1
Connection to node-0 closed.
Transferred: sent 3288, received 3724 bytes, in 2.8 seconds
Bytes per second: sent 1174.4, received 1330.1
debug1: Exit status 0
haoyuli@node-1:~$ ssh -v 10.10.1.2
OpenSSH_8.2p1 Ubuntu-4ubuntu0.2, OpenSSL 1.1.1f 31 Mar 2020
debug1: Reading configuration data /etc/ssh/ssh_config
debug1: /etc/ssh/ssh_config line 19: include /etc/ssh/ssh_config.d/*.conf matched no files
debug1: /etc/ssh/ssh_config line 21: Applying options for *
debug1: Connecting to 10.10.1.2 [10.10.1.2] port 22.
debug1: Connection established.
debug1: identity file /users/haoyuli/.ssh/id_rsa type 0
debug1: identity file /users/haoyuli/.ssh/id_rsa-cert type -1
debug1: identity file /users/haoyuli/.ssh/id_dsa type -1
debug1: identity file /users/haoyuli/.ssh/id_dsa-cert type -1
debug1: identity file /users/haoyuli/.ssh/id_ecdsa type -1
debug1: identity file /users/haoyuli/.ssh/id_ecdsa-cert type -1
debug1: identity file /users/haoyuli/.ssh/id_ecdsa_sk type -1
debug1: identity file /users/haoyuli/.ssh/id_ecdsa_sk-cert type -1
debug1: identity file /users/haoyuli/.ssh/id_ed25519 type -1
debug1: identity file /users/haoyuli/.ssh/id_ed25519-cert type -1
debug1: identity file /users/haoyuli/.ssh/id_ed25519_sk type -1
debug1: identity file /users/haoyuli/.ssh/id_ed25519_sk-cert type -1
debug1: identity file /users/haoyuli/.ssh/id_xmss type -1
debug1: identity file /users/haoyuli/.ssh/id_xmss-cert type -1
debug1: Local version string SSH-2.0-OpenSSH_8.2p1 Ubuntu-4ubuntu0.2
debug1: Remote protocol version 2.0, remote software version OpenSSH_8.2p1 Ubuntu-4ubuntu0.2
debug1: match: OpenSSH_8.2p1 Ubuntu-4ubuntu0.2 pat OpenSSH* compat 0x04000000
debug1: Authenticating to 10.10.1.2:22 as 'haoyuli'
debug1: SSH2_MSG_KEXINIT sent
debug1: SSH2_MSG_KEXINIT received
debug1: kex: algorithm: curve25519-sha256
debug1: kex: host key algorithm: ecdsa-sha2-nistp256
debug1: kex: server->client cipher: chacha20...@openssh.com MAC: <implicit> compression: none
debug1: kex: client->server cipher: chacha20...@openssh.com MAC: <implicit> compression: none
debug1: expecting SSH2_MSG_KEX_ECDH_REPLY
debug1: Server host key: ecdsa-sha2-nistp256 SHA256:bB4TBZMIEhrtJN2mb/Mzn8fvSRUBUC+UamxqkhCU7HA
debug1: Host '10.10.1.2' is known and matches the ECDSA host key.
debug1: Found key in /users/haoyuli/.ssh/known_hosts:2
debug1: rekey out after 134217728 blocks
debug1: SSH2_MSG_NEWKEYS sent
debug1: expecting SSH2_MSG_NEWKEYS
debug1: SSH2_MSG_NEWKEYS received
debug1: rekey in after 134217728 blocks
debug1: Will attempt key: /users/haoyuli/.ssh/id_rsa RSA SHA256:bZCEH9CD/wfdI1h+Wkf163JBULTfMWZ2cocRSpL90S4
debug1: Will attempt key: /users/haoyuli/.ssh/id_dsa 
debug1: Will attempt key: /users/haoyuli/.ssh/id_ecdsa 
debug1: Will attempt key: /users/haoyuli/.ssh/id_ecdsa_sk 
debug1: Will attempt key: /users/haoyuli/.ssh/id_ed25519 
debug1: Will attempt key: /users/haoyuli/.ssh/id_ed25519_sk 
debug1: Will attempt key: /users/haoyuli/.ssh/id_xmss 
debug1: SSH2_MSG_EXT_INFO received
debug1: kex_input_ext_info: server-sig-algs=<ssh-ed25519,sk-ssh-...@openssh.com,ssh-rsa,rsa-sha2-256,rsa-sha2-512,ssh-dss,ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521,sk-ecdsa-sh...@openssh.com>
debug1: SSH2_MSG_SERVICE_ACCEPT received
debug1: Authentications that can continue: publickey
debug1: Next authentication method: publickey
debug1: Offering public key: /users/haoyuli/.ssh/id_rsa RSA SHA256:bZCEH9CD/wfdI1h+Wkf163JBULTfMWZ2cocRSpL90S4
debug1: Server accepts key: /users/haoyuli/.ssh/id_rsa RSA SHA256:bZCEH9CD/wfdI1h+Wkf163JBULTfMWZ2cocRSpL90S4
debug1: Authentication succeeded (publickey).
Authenticated to 10.10.1.2 ([10.10.1.2]:22).
debug1: channel 0: new [client-session]
debug1: Requesting no-more-...@openssh.com
debug1: Entering interactive session.
debug1: pledge: network
debug1: client_input_global_request: rtype hostk...@openssh.com want_reply 0
debug1: Remote: /users/haoyuli/.ssh/authorized_keys:9: key options: agent-forwarding port-forwarding pty user-rc x11-forwarding
debug1: Remote: /users/haoyuli/.ssh/authorized_keys:9: key options: agent-forwarding port-forwarding pty user-rc x11-forwarding
debug1: Sending environment.
debug1: Sending env LANG = en_US.UTF-8
Welcome to Ubuntu 20.04 LTS (GNU/Linux 5.4.0-100-generic x86_64)

 * Documentation: https://help.ubuntu.com
 * Management: https://landscape.canonical.com
 * Support: https://ubuntu.com/advantage

 * Super-optimized for small spaces - read how we shrank the memory
 footprint of MicroK8s to make it the smallest full K8s around.

https://ubuntu.com/blog/microk8s-memory-optimisation
New release '22.04.1 LTS' available.
Run 'do-release-upgrade' to upgrade to it.

Last login: Tue Jan 3 11:30:57 2023 from 10.10.1.1
haoyuli@node-1:~$ exit
logout
debug1: client_input_channel_req: channel 0 rtype exit-status reply 0
debug1: client_input_channel_req: channel 0 rtype e...@openssh.com reply 0
debug1: channel 0: free: client-session, nchannels 1
Connection to 10.10.1.2 closed.
Transferred: sent 3288, received 3792 bytes, in 2.5 seconds
Bytes per second: sent 1315.4, received 1517.0
debug1: Exit status 0

I would like to provide any further information as well. Thanks!

Best,

Haoyu

--
You received this message because you are subscribed to a topic in the Google Groups "cloudlab-users" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/cloudlab-users/yhbtjL3CMjk/unsubscribe.
To unsubscribe from this group and all its topics, send an email to cloudlab-user...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/cloudlab-users/2DE0268D-E42E-4666-812F-21E3C1384D1D%40gmail.com.

[Mike Hibler]

You will have to terminate and reinstantiate. We had bad state in the DB
that led you to get your LAN mapped to non-existent NICs on the d750s.
This in turn led to the experiment subnet (10.10.1.0/24) being assigned
to the loopback interface on the flex nodes. Hilarity ensued.

The DB should be fixed now.

> You received this message because you are subscribed to the Google Groups
> "cloudlab-users" group.
> To unsubscribe from this group and stop receiving emails from it, send an email

> to cloudlab-user...@googlegroups.com.
> To view this discussion on the web visit https://groups.google.com/d/msgid/

> cloudlab-users/958fd387-1709-49ce-9b01-606af8b75482%40Spark.