How do I fix a [Errno 101] Network is unreachable error?

91 views
Skip to first unread message

Idan Dekel

unread,
Oct 25, 2021, 2:44:12 PM10/25/21
to gce-discussion
Hello!

I have a VM running on GCE, and a local CentOS 7 VM running on VirtualBox.
I'm trying to create an IAP tunnel from my local VM to the GCE one, and I'm getting the following error:

OSError: [Errno 101] Network is unreachable

This is the command I'm using:

gcloud compute start-iap-tunnel --project=[redacted] --zone=[redacted] --verbosity=debug instance-1 22

Using the same command I'm able to create the tunnel from my Windows 10 machine to the GCE VM, and a colleague of mine managed to create the tunnel on his CentOS VM.
I'm also able to SSH from my local VM into the GCE VM using its external IP.

So it doesn't look like it's an issue with the GCE VM's configuration, or with the connection between the VMs (pings are working just fine).

I'd appreciate any ideas you have in solving this. Thanks!

This is the complete output I'm getting after enabling debug mode:

DEBUG: Running [gcloud.compute.start-iap-tunnel] with arguments: [--project: "[redacted]", --verbosity: "debug", --zone: "[redacted]", INSTANCE_NAME: "instance-1", INSTANCE_PORT: "22"                              ]
DEBUG: Starting new HTTPS connection (1): compute.googleapis.com:443
DEBUG: https://compute.googleapis.com:443 "POST /batch/compute/v1 HTTP/1.1" 200 None
Picking local unused port [21324].
Testing if tunnel connection works.
DEBUG: credentials type for _GetAccessTokenCallback is [<googlecloudsdk.core.credentials.google_auth_credentials.UserCredWithReauth object at 0x7f38c7fa0e20>].
DEBUG: Making request: POST https://oauth2.googleapis.com/token
DEBUG: Starting new HTTPS connection (1): oauth2.googleapis.com:443
DEBUG: https://oauth2.googleapis.com:443 "POST /token HTTP/1.1" 200 None
INFO: Connecting with URL ['wss://tunnel.cloudproxy.app/v4/connect?project=[redacted]&zone=[redacted]&instance=instance-1&interface=nic0&port=22']
DEBUG: Error during WebSocket processing.
Traceback (most recent call last):
  File "/usr/lib64/google-cloud-sdk/lib/third_party/websocket/_app.py", line 248, in run_forever
    self.sock.connect(
  File "/usr/lib64/google-cloud-sdk/lib/third_party/websocket/_core.py", line 222, in connect
    self.sock, addrs = connect(url, self.sock_opt, proxy_info(**options),
  File "/usr/lib64/google-cloud-sdk/lib/third_party/websocket/_http.py", line 120, in connect
    sock = _open_socket(addrinfo_list, options.sockopt, options.timeout)
  File "/usr/lib64/google-cloud-sdk/lib/third_party/websocket/_http.py", line 189, in _open_socket
    raise error
  File "/usr/lib64/google-cloud-sdk/lib/third_party/websocket/_http.py", line 172, in _open_socket
    sock.connect(address)
OSError: [Errno 101] Network is unreachable
INFO: Error during WebSocket processing:
OSError: [Errno 101] Network is unreachable

DEBUG: (gcloud.compute.start-iap-tunnel) While checking if a connection can be made: Error while connecting [[Errno 101] Network is unreachable].
Traceback (most recent call last):
  File "/usr/lib64/google-cloud-sdk/lib/googlecloudsdk/command_lib/compute/iap_tunnel.py", line 723, in Run
    self._TestConnection()
  File "/usr/lib64/google-cloud-sdk/lib/googlecloudsdk/command_lib/compute/iap_tunnel.py", line 750, in _TestConnection
    websocket_conn = self._InitiateWebSocketConnection(
  File "/usr/lib64/google-cloud-sdk/lib/googlecloudsdk/command_lib/compute/iap_tunnel.py", line 644, in _InitiateWebSocketConnection
    new_websocket.InitiateConnection()
  File "/usr/lib64/google-cloud-sdk/lib/googlecloudsdk/api_lib/compute/iap_tunnel_websocket.py", line 144, in InitiateConnection
    self._WaitForOpenOrRaiseError()
  File "/usr/lib64/google-cloud-sdk/lib/googlecloudsdk/api_lib/compute/iap_tunnel_websocket.py", line 393, in _WaitForOpenOrRaiseError
    raise ConnectionCreationError(error_msg)
googlecloudsdk.api_lib.compute.iap_tunnel_websocket.ConnectionCreationError: Error while connecting [[Errno 101] Network is unreachable].

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/lib64/google-cloud-sdk/lib/googlecloudsdk/calliope/cli.py", line 987, in Execute
    resources = calliope_command.Run(cli=self, args=args)
  File "/usr/lib64/google-cloud-sdk/lib/googlecloudsdk/calliope/backend.py", line 809, in Run
    resources = command_instance.Run(args)
  File "/usr/lib64/google-cloud-sdk/lib/surface/compute/start_iap_tunnel.py", line 160, in Run
    iap_tunnel_helper.Run()
  File "/usr/lib64/google-cloud-sdk/lib/googlecloudsdk/command_lib/compute/iap_tunnel.py", line 725, in Run
    raise iap_tunnel_websocket.ConnectionCreationError(
googlecloudsdk.api_lib.compute.iap_tunnel_websocket.ConnectionCreationError: While checking if a connection can be made: Error while connecting [[Errno 101] Network is unreachable].
ERROR: (gcloud.compute.start-iap-tunnel) While checking if a connection can be made: Error while connecting [[Errno 101] Network is unreachable].

Adebisi Ibirogba

unread,
Oct 25, 2021, 4:15:02 PM10/25/21
to gce-discussion
The command you used above is for connecting to windows instances using rdp. However here is the command for the one for connecting to Linux via ssh.

They are different commands

Idan Dekel

unread,
Oct 26, 2021, 7:02:46 AM10/26/21
to gce-discussion
Thanks for the pointer. I somehow missed that this was RDP-specific 🤦

I tried running the 'gcloud compute ssh' command with the --tunnel-through-iap flag on my local VM and my Windows machine.
On Windows the connection worked smoothly, but the CentOS VM fails with similar output to the above:

INFO: Connecting with URL ['wss://tunnel.cloudproxy.app/v4/connect?project=[redacted]&zone=[redacted]&instance=instance-1&interface=nic0&port=22']
DEBUG: Error during WebSocket processing.
Traceback (most recent call last):
  File "/usr/lib64/google-cloud-sdk/lib/third_party/websocket/_app.py", line 248, in run_forever
    self.sock.connect(
  File "/usr/lib64/google-cloud-sdk/lib/third_party/websocket/_core.py", line 222, in connect
    self.sock, addrs = connect(url, self.sock_opt, proxy_info(**options),
  File "/usr/lib64/google-cloud-sdk/lib/third_party/websocket/_http.py", line 120, in connect
    sock = _open_socket(addrinfo_list, options.sockopt, options.timeout)
  File "/usr/lib64/google-cloud-sdk/lib/third_party/websocket/_http.py", line 189, in _open_socket
    raise error
  File "/usr/lib64/google-cloud-sdk/lib/third_party/websocket/_http.py", line 172, in _open_socket
    sock.connect(address)
OSError: [Errno 101] Network is unreachable
INFO: Error during WebSocket processing:
OSError: [Errno 101] Network is unreachable

INFO: Client closed connection from [stdin].
DEBUG: (gcloud.compute.start-iap-tunnel) Error while connecting [[Errno 101] Network is unreachable].
Traceback (most recent call last):
  File "/usr/lib64/google-cloud-sdk/lib/googlecloudsdk/calliope/cli.py", line 987, in Execute
    resources = calliope_command.Run(cli=self, args=args)
  File "/usr/lib64/google-cloud-sdk/lib/googlecloudsdk/calliope/backend.py", line 809, in Run
    resources = command_instance.Run(args)
  File "/usr/lib64/google-cloud-sdk/lib/surface/compute/start_iap_tunnel.py", line 160, in Run
    iap_tunnel_helper.Run()
  File "/usr/lib64/google-cloud-sdk/lib/googlecloudsdk/command_lib/compute/iap_tunnel.py", line 847, in Run
    self._RunReceiveLocalData(_StdinSocket(), 'stdin', user_agent)
  File "/usr/lib64/google-cloud-sdk/lib/googlecloudsdk/command_lib/compute/iap_tunnel.py", line 673, in _RunReceiveLocalData
    websocket_conn = self._InitiateWebSocketConnection(
  File "/usr/lib64/google-cloud-sdk/lib/googlecloudsdk/command_lib/compute/iap_tunnel.py", line 644, in _InitiateWebSocketConnection
    new_websocket.InitiateConnection()
  File "/usr/lib64/google-cloud-sdk/lib/googlecloudsdk/api_lib/compute/iap_tunnel_websocket.py", line 144, in InitiateConnection
    self._WaitForOpenOrRaiseError()
  File "/usr/lib64/google-cloud-sdk/lib/googlecloudsdk/api_lib/compute/iap_tunnel_websocket.py", line 393, in _WaitForOpenOrRaiseError
    raise ConnectionCreationError(error_msg)
googlecloudsdk.api_lib.compute.iap_tunnel_websocket.ConnectionCreationError: Error while connecting [[Errno 101] Network is unreachable].
ERROR: (gcloud.compute.start-iap-tunnel) Error while connecting [[Errno 101] Network is unreachable].
ssh_exchange_identification: Connection closed by remote host
DEBUG: (gcloud.compute.ssh) [/usr/bin/ssh] exited with return code [255].
Traceback (most recent call last):
  File "/usr/lib64/google-cloud-sdk/lib/googlecloudsdk/calliope/cli.py", line 987, in Execute
    resources = calliope_command.Run(cli=self, args=args)
  File "/usr/lib64/google-cloud-sdk/lib/googlecloudsdk/calliope/backend.py", line 809, in Run
    resources = command_instance.Run(args)
  File "/usr/lib64/google-cloud-sdk/lib/surface/compute/ssh.py", line 382, in Run
    raise e
  File "/usr/lib64/google-cloud-sdk/lib/surface/compute/ssh.py", line 376, in Run
    return_code = cmd.Run(
  File "/usr/lib64/google-cloud-sdk/lib/googlecloudsdk/command_lib/util/ssh/ssh.py", line 1404, in Run
    raise CommandError(args[0], return_code=status)
googlecloudsdk.command_lib.util.ssh.ssh.CommandError: [/usr/bin/ssh] exited with return code [255].
ERROR: (gcloud.compute.ssh) [/usr/bin/ssh] exited with return code [255].

Ahmad P - Cloud Platform Support

unread,
Oct 26, 2021, 10:46:12 AM10/26/21
to gce-discussion

Hello,


I suspect the firewall rules.


Please check these rules that allow IAP to connect to your VM instances:


-Allows ingress traffic from the IP range 35.235.240.0/20. This range contains all IP addresses that IAP uses for TCP forwarding


- Allows connections to all ports that you want to be accessible by using IAP TCP forwarding, for example, port 22 for SSH and port 3389 for RDP.

Idan Dekel

unread,
Oct 26, 2021, 12:49:48 PM10/26/21
to gce-discussion
Hi Ahmad, thanks for your reply.

I'm afraid that the required firewall rules are all present, for the correct IP range and expected ports (TCP 22 and 3389).

Antonio Gomes

unread,
Oct 28, 2021, 7:16:59 AM10/28/21
to gce-discussion
It seems like a network issue, would be good to know if it's in the VirtualBox VM side or if it could be in the GCE side.

I suggest you update Cloud SDK [1] gcloud components list.

Then it would help to perform some tests in order to determine in which side is the issue. You could create a new Linux instance in GCE and also a VirtualBox VM for testing purposes where you could open the network and ports, and check if the original CentOS 7 VM running on VirtualBox is able to connect to the new Linux (Ubuntu) GCE VM, or if a new fresh Linux (Ubuntu) VM running on VirtualBox in able to connect to the original GCE VM. You could also give permissions to your colleague so he could test to create a tunnel from his CentOS VM running on VirtualBox into your project GCE VM.

Please check the console IAP in SSH and TCP Resources tab that the config is ok.

Remember also to follow the IAP TCP forwarding guide carefully.

Idan Dekel

unread,
Oct 29, 2021, 4:12:00 PM10/29/21
to gce-discussion
Thanks for your reply Antonio,

In an attempt to single-out the point of failure, I ran some more tests - 
I tried logging-in from my local VM into an instance on a different project, and got the same result (SSH works fine, "Network is unreachable" error when using IAP)
I also asked some colleagues to log-into both GCE VMs (The original one and the new one) from their Linux VMs, and they all succeeded in SSHing using IAP.

So, my assumption is that whatever the issue is, it's probably in my local VM, rather that the GCE one.

When looking at the IAP config tab, I do get shown the following warning for each instance:

Not enough access to resources

You might not be allowing all traffic from IAP to your VM. Add the following firewall rule to correct this issue.
Source IP range       35.235.240.0/20
Allowed protocols   tcp

However, I made sure that this rule is already applied to my firewall (see screengrab: https://photos.app.goo.gl/k5YEdUbLRor1ggkF9), so I'm not sure what triggers this warning, nor how to fix it.
Moreover, I assume that the fact that my colleagues were able to SSH using IAP into these instances means that the firewall rule is indeed configured correctly.

Idan Dekel

unread,
Nov 9, 2021, 12:16:04 PM11/9/21
to gce-discussion
I think I managed to fix this.

One of my colleagues suggested it has to do with my system preferring IPV6 over IPV4 (which is a problem since our office network doesn't handle IPV6 correctly), so I followed this StackExchange answer to make sure my VM prefers IPV4:

And so far, looks like that fixed my issue.

Reply all
Reply to author
Forward
0 new messages