How do I fix a [Errno 101] Network is unreachable error?

[Idan Dekel]

Hello!

I have a VM running on GCE, and a local CentOS 7 VM running on VirtualBox.

I'm trying to create an IAP tunnel from my local VM to the GCE one, and I'm getting the following error:

OSError: [Errno 101] Network is unreachable

This is the command I'm using:

gcloud compute start-iap-tunnel --project=[redacted] --zone=[redacted] --verbosity=debug instance-1 22

Using the same command I'm able to create the tunnel from my Windows 10 machine to the GCE VM, and a colleague of mine managed to create the tunnel on his CentOS VM.

I'm also able to SSH from my local VM into the GCE VM using its external IP.

So it doesn't look like it's an issue with the GCE VM's configuration, or with the connection between the VMs (pings are working just fine).

I'd appreciate any ideas you have in solving this. Thanks!

This is the complete output I'm getting after enabling debug mode:

DEBUG: Running [gcloud.compute.start-iap-tunnel] with arguments: [--project: "[redacted]", --verbosity: "debug", --zone: "[redacted]", INSTANCE_NAME: "instance-1", INSTANCE_PORT: "22"                              ]

DEBUG: Starting new HTTPS connection (1): compute.googleapis.com:443

DEBUG: https://compute.googleapis.com:443 "POST /batch/compute/v1 HTTP/1.1" 200 None

Picking local unused port [21324].

Testing if tunnel connection works.

DEBUG: credentials type for _GetAccessTokenCallback is [<googlecloudsdk.core.credentials.google_auth_credentials.UserCredWithReauth object at 0x7f38c7fa0e20>].

DEBUG: Making request: POST https://oauth2.googleapis.com/token

DEBUG: Starting new HTTPS connection (1): oauth2.googleapis.com:443

DEBUG: https://oauth2.googleapis.com:443 "POST /token HTTP/1.1" 200 None

INFO: Connecting with URL ['wss://tunnel.cloudproxy.app/v4/connect?project=[redacted]&zone=[redacted]&instance=instance-1&interface=nic0&port=22']

DEBUG: Error during WebSocket processing.

Traceback (most recent call last):

  File "/usr/lib64/google-cloud-sdk/lib/third_party/websocket/_app.py", line 248, in run_forever

    self.sock.connect(

  File "/usr/lib64/google-cloud-sdk/lib/third_party/websocket/_core.py", line 222, in connect

    self.sock, addrs = connect(url, self.sock_opt, proxy_info(**options),

  File "/usr/lib64/google-cloud-sdk/lib/third_party/websocket/_http.py", line 120, in connect

    sock = _open_socket(addrinfo_list, options.sockopt, options.timeout)

  File "/usr/lib64/google-cloud-sdk/lib/third_party/websocket/_http.py", line 189, in _open_socket

    raise error

  File "/usr/lib64/google-cloud-sdk/lib/third_party/websocket/_http.py", line 172, in _open_socket

    sock.connect(address)

OSError: [Errno 101] Network is unreachable

INFO: Error during WebSocket processing:

OSError: [Errno 101] Network is unreachable

DEBUG: (gcloud.compute.start-iap-tunnel) While checking if a connection can be made: Error while connecting [[Errno 101] Network is unreachable].

Traceback (most recent call last):

  File "/usr/lib64/google-cloud-sdk/lib/googlecloudsdk/command_lib/compute/iap_tunnel.py", line 723, in Run

    self._TestConnection()

  File "/usr/lib64/google-cloud-sdk/lib/googlecloudsdk/command_lib/compute/iap_tunnel.py", line 750, in _TestConnection

    websocket_conn = self._InitiateWebSocketConnection(

  File "/usr/lib64/google-cloud-sdk/lib/googlecloudsdk/command_lib/compute/iap_tunnel.py", line 644, in _InitiateWebSocketConnection

    new_websocket.InitiateConnection()

  File "/usr/lib64/google-cloud-sdk/lib/googlecloudsdk/api_lib/compute/iap_tunnel_websocket.py", line 144, in InitiateConnection

    self._WaitForOpenOrRaiseError()

  File "/usr/lib64/google-cloud-sdk/lib/googlecloudsdk/api_lib/compute/iap_tunnel_websocket.py", line 393, in _WaitForOpenOrRaiseError

    raise ConnectionCreationError(error_msg)

googlecloudsdk.api_lib.compute.iap_tunnel_websocket.ConnectionCreationError: Error while connecting [[Errno 101] Network is unreachable].

During handling of the above exception, another exception occurred:

Traceback (most recent call last):

  File "/usr/lib64/google-cloud-sdk/lib/googlecloudsdk/calliope/cli.py", line 987, in Execute

    resources = calliope_command.Run(cli=self, args=args)

  File "/usr/lib64/google-cloud-sdk/lib/googlecloudsdk/calliope/backend.py", line 809, in Run

    resources = command_instance.Run(args)

  File "/usr/lib64/google-cloud-sdk/lib/surface/compute/start_iap_tunnel.py", line 160, in Run

    iap_tunnel_helper.Run()

  File "/usr/lib64/google-cloud-sdk/lib/googlecloudsdk/command_lib/compute/iap_tunnel.py", line 725, in Run

    raise iap_tunnel_websocket.ConnectionCreationError(

googlecloudsdk.api_lib.compute.iap_tunnel_websocket.ConnectionCreationError: While checking if a connection can be made: Error while connecting [[Errno 101] Network is unreachable].

ERROR: (gcloud.compute.start-iap-tunnel) While checking if a connection can be made: Error while connecting [[Errno 101] Network is unreachable].

[Adebisi Ibirogba]

The command you used above is for connecting to windows instances using rdp. However here is the command for the one for connecting to Linux via ssh.

They are different commands

[Idan Dekel]

Thanks for the pointer. I somehow missed that this was RDP-specific 🤦

I tried running the 'gcloud compute ssh' command with the --tunnel-through-iap flag on my local VM and my Windows machine.

On Windows the connection worked smoothly, but the CentOS VM fails with similar output to the above:

INFO: Connecting with URL ['wss://tunnel.cloudproxy.app/v4/connect?project=[redacted]&zone=[redacted]&instance=instance-1&interface=nic0&port=22']

DEBUG: Error during WebSocket processing.

Traceback (most recent call last):

  File "/usr/lib64/google-cloud-sdk/lib/third_party/websocket/_app.py", line 248, in run_forever

    self.sock.connect(

  File "/usr/lib64/google-cloud-sdk/lib/third_party/websocket/_core.py", line 222, in connect

    self.sock, addrs = connect(url, self.sock_opt, proxy_info(**options),

  File "/usr/lib64/google-cloud-sdk/lib/third_party/websocket/_http.py", line 120, in connect

    sock = _open_socket(addrinfo_list, options.sockopt, options.timeout)

  File "/usr/lib64/google-cloud-sdk/lib/third_party/websocket/_http.py", line 189, in _open_socket

    raise error

  File "/usr/lib64/google-cloud-sdk/lib/third_party/websocket/_http.py", line 172, in _open_socket

    sock.connect(address)

OSError: [Errno 101] Network is unreachable

INFO: Error during WebSocket processing:

OSError: [Errno 101] Network is unreachable

INFO: Client closed connection from [stdin].

DEBUG: (gcloud.compute.start-iap-tunnel) Error while connecting [[Errno 101] Network is unreachable].

Traceback (most recent call last):

  File "/usr/lib64/google-cloud-sdk/lib/googlecloudsdk/calliope/cli.py", line 987, in Execute

    resources = calliope_command.Run(cli=self, args=args)

  File "/usr/lib64/google-cloud-sdk/lib/googlecloudsdk/calliope/backend.py", line 809, in Run

    resources = command_instance.Run(args)

  File "/usr/lib64/google-cloud-sdk/lib/surface/compute/start_iap_tunnel.py", line 160, in Run

    iap_tunnel_helper.Run()

  File "/usr/lib64/google-cloud-sdk/lib/googlecloudsdk/command_lib/compute/iap_tunnel.py", line 847, in Run

    self._RunReceiveLocalData(_StdinSocket(), 'stdin', user_agent)

  File "/usr/lib64/google-cloud-sdk/lib/googlecloudsdk/command_lib/compute/iap_tunnel.py", line 673, in _RunReceiveLocalData

    websocket_conn = self._InitiateWebSocketConnection(

  File "/usr/lib64/google-cloud-sdk/lib/googlecloudsdk/command_lib/compute/iap_tunnel.py", line 644, in _InitiateWebSocketConnection

    new_websocket.InitiateConnection()

  File "/usr/lib64/google-cloud-sdk/lib/googlecloudsdk/api_lib/compute/iap_tunnel_websocket.py", line 144, in InitiateConnection

    self._WaitForOpenOrRaiseError()

  File "/usr/lib64/google-cloud-sdk/lib/googlecloudsdk/api_lib/compute/iap_tunnel_websocket.py", line 393, in _WaitForOpenOrRaiseError

    raise ConnectionCreationError(error_msg)

googlecloudsdk.api_lib.compute.iap_tunnel_websocket.ConnectionCreationError: Error while connecting [[Errno 101] Network is unreachable].

ERROR: (gcloud.compute.start-iap-tunnel) Error while connecting [[Errno 101] Network is unreachable].

ssh_exchange_identification: Connection closed by remote host

DEBUG: (gcloud.compute.ssh) [/usr/bin/ssh] exited with return code [255].

Traceback (most recent call last):

  File "/usr/lib64/google-cloud-sdk/lib/googlecloudsdk/calliope/cli.py", line 987, in Execute

    resources = calliope_command.Run(cli=self, args=args)

  File "/usr/lib64/google-cloud-sdk/lib/googlecloudsdk/calliope/backend.py", line 809, in Run

    resources = command_instance.Run(args)

  File "/usr/lib64/google-cloud-sdk/lib/surface/compute/ssh.py", line 382, in Run

    raise e

  File "/usr/lib64/google-cloud-sdk/lib/surface/compute/ssh.py", line 376, in Run

    return_code = cmd.Run(

  File "/usr/lib64/google-cloud-sdk/lib/googlecloudsdk/command_lib/util/ssh/ssh.py", line 1404, in Run

    raise CommandError(args[0], return_code=status)

googlecloudsdk.command_lib.util.ssh.ssh.CommandError: [/usr/bin/ssh] exited with return code [255].

ERROR: (gcloud.compute.ssh) [/usr/bin/ssh] exited with return code [255].

[Ahmad P - Cloud Platform Support]

Hello,

I suspect the firewall rules.

Please check these rules that allow IAP to connect to your VM instances:

-Allows ingress traffic from the IP range 35.235.240.0/20. This range contains all IP addresses that IAP uses for TCP forwarding

- Allows connections to all ports that you want to be accessible by using IAP TCP forwarding, for example, port 22 for SSH and port 3389 for RDP.

[Idan Dekel]

Hi Ahmad, thanks for your reply.

I'm afraid that the required firewall rules are all present, for the correct IP range and expected ports (TCP 22 and 3389).

[Antonio Gomes]

It seems like a network issue, would be good to know if it's in the VirtualBox VM side or if it could be in the GCE side.

I suggest you update Cloud SDK [1] gcloud components list.

Then it would help to perform some tests in order to determine in which side is the issue. You could create a new Linux instance in GCE and also a VirtualBox VM for testing purposes where you could open the network and ports, and check if the original CentOS 7 VM running on VirtualBox is able to connect to the new Linux (Ubuntu) GCE VM, or if a new fresh Linux (Ubuntu) VM running on VirtualBox in able to connect to the original GCE VM. You could also give permissions to your colleague so he could test to create a tunnel from his CentOS VM running on VirtualBox into your project GCE VM.

Please check the console IAP in SSH and TCP Resources tab that the config is ok.

Remember also to follow the IAP TCP forwarding guide carefully.

[Idan Dekel]

Thanks for your reply Antonio,

In an attempt to single-out the point of failure, I ran some more tests - 

I tried logging-in from my local VM into an instance on a different project, and got the same result (SSH works fine, "Network is unreachable" error when using IAP)

I also asked some colleagues to log-into both GCE VMs (The original one and the new one) from their Linux VMs, and they all succeeded in SSHing using IAP.

So, my assumption is that whatever the issue is, it's probably in my local VM, rather that the GCE one.

When looking at the IAP config tab, I do get shown the following warning for each instance:

Not enough access to resources

You might not be allowing all traffic from IAP to your VM. Add the following firewall rule to correct this issue.

Source IP range       35.235.240.0/20

Allowed protocols   tcp

However, I made sure that this rule is already applied to my firewall (see screengrab: https://photos.app.goo.gl/k5YEdUbLRor1ggkF9), so I'm not sure what triggers this warning, nor how to fix it.

Moreover, I assume that the fact that my colleagues were able to SSH using IAP into these instances means that the firewall rule is indeed configured correctly.

[Idan Dekel]

I think I managed to fix this.

One of my colleagues suggested it has to do with my system preferring IPV6 over IPV4 (which is a problem since our office network doesn't handle IPV6 correctly), so I followed this StackExchange answer to make sure my VM prefers IPV4:

https://unix.stackexchange.com/questions/428736/how-to-make-my-system-centos-7-use-primarily-ipv4-if-its-available

And so far, looks like that fixed my issue.