Intermittent SSH timeouts GCP -> AWS

[Johan Andersson]

Hello all,

I have a VM running on GCP and one VM running on AWS. i have tried on many VMs on AWS and on GCP so i doubt there is anything wrong with the VMs.

On the VM on GCP  I have a program that needs to SSH to the VM on AWS., but it times out from time to time and the latency goes up the roof;  From around 1 sec to up to minutes. 

Do you know if GCP does rate limiting on the outgoing traffic? 
Is there anything that can be set in any network settings?

Please see details below:

I constructed a simple script that I run on the GCP VM, which executes a simple command on the remote AWS server:

i=1; while true; do echo $i; i=`expr $i + 1` ; time ssh -i x s9s@NNN.NNN.NNN.NNN "sudo cat /etc/mtab > /dev/null"; done

Outliers in bold below.
real    0m1.198s
user    0m0.009s
sys     0m0.005s
38
real    0m0.982s
user    0m0.013s
sys     0m0.000s
39
real    0m0.969s
user    0m0.009s
sys     0m0.005s
40
real    0m32.442s
user    0m0.000s
sys     0m0.014s
41
real    1m5.566s
user    0m0.015s
sys     0m0.001s
42
real    0m1.156s
user    0m0.015s
sys     0m0.008s
43
real    0m1.277s
user    0m0.014s
sys     0m0.000s
44
real    0m16.474s

user    0m0.010s
sys     0m0.004s
45

real    0m0.958s
user    0m0.010s
sys     0m0.004s
46
real    0m0.981s
user    0m0.009s
sys     0m0.005s

Thanks for your input.

BR
johan

[Bruno (Google Cloud Support)]

Hello ~ Yes there is some limitations on egress bandwith which are:

• 7 Gbps total for all egress flows to external IP addresses
• 3 Gbps per individual egress flow to an external IP address

You can find more details in this documentation [1] including the different ingress bandwith based on machine types.
Its also possible to increase this bandwith with certain types of CPU [2] but not sure its very helpful in your use case.


[1] https://cloud.google.com/compute/docs/network-bandwidth
[2] https://cloud.google.com/compute/docs/networking/configure-vm-with-high-bandwidth-configuration

[Johan Andersson]

I should also add that when I run this from my local computer, then I do not have these timeouts .

Thanks for any input.
BR
johan

[Kamran Shaghaghi]

Hello Johan

According to this documentation[1] real time is from start to finish of the call or the process and not necessarily the time out of the ssh if the process is successful.

in your case as it is intermittent it may related to the network connection also and you can verify the network performances between your VMs on both sides [2], [3] (documents are not backed by google)

and also these documentation may help [4] & [5]. you can find lots on information about "How to Increase SSH Connection Timeout in Linux" or "Keep SSH session alive" in case you may need.

and also i encourage you to have a look at this google documentation [6].

hope it can help 

[1]: https://stackoverflow.com/questions/556405/what-do-real-user-and-sys-mean-in-the-output-of-time1#:~:text=Real%20refers%20to%20actual%20elapsed,I%2FO%20to%20complete).
[2]: https://stackoverflow.com/questions/2813843/possible-reasons-for-timeout-when-trying-to-access-ec2-instance

[3]: https://aws.amazon.com/premiumsupport/knowledge-center/network-issue-vpc-onprem-ig/

[4]: https://cloud.google.com/blog/products/networking/your-top-network-performance-problems-and-how-to-fix-them

[5]: https://cloud.google.com/compute/docs/troubleshooting/troubleshooting-networking

[6]: https://cloud.google.com/sdk/gcloud/reference/compute/config-ssh