How to solve the recent extreme high latency in China?

[Christopher Cui]

Recently, It seems to be strange that latency may possibly reach up to 600ms. With that figure, it is possibly out of service.

How to solve this problem? 

[Carlos (Cloud Platform Support)]

Hi Christopher,

Can you provide some additional details on the delay you are experiencing? i.e

Where is your instance deployed? What is the destination IP outside Google?

Is this an intermittent issue?

Does an MTR show packet loss? When did the issue started?

[Kai Bolay]

On Wed, Aug 2, 2017 at 4:47 PM, 'Carlos (Cloud Platform Support)' via gce-discussion <gce-dis...@googlegroups.com> wrote:

Does an MTR show packet loss? When did the issue started?

I think you mean mtr, not MTR ;-)

[Eric Aarts]

Hello,

 

We run GCP instances from Europa, US and one nearby China; zone asia-east1-c.

 

With our own scripts from a Europe europe-west1-d instance we check the availability and latency of the production instances and (almost) never experience issues with the connection of the China instances.

 

But our customers in China (in and nearby Shanghai) complain some every three months, for some hours up to one or two days the connection to our ERP instance is very, very slow. In the Apache2 server logs those hours/days we see an increase of broken connections. After some time the connection speeds up automatically again without interference of us.

 

We suspect the ‘international’ internet in Shanghai/China causes the delay, as every occasion we investigated is temporarily and then also other international sites have delay, while not by us China mainland hosted sites do not show latency.

 

Regards, Eric

[Christopher Cui]

Hello,

Thanks for your reply.

My instance is deployed at asia-east-a. My destination IP address is 175.30.86.19 provided by a Chinese ISP, China Telecom. 

Things were just gone wrong these days. And the packet loss rate is up to nearly 20%.

Currently, I am wondering whether it is due to (1) the possible changes of the IP routing from my instance deployed in Taiwan to my client or (2) the more complicated networking environment in mainland China recently.

[Eric Aarts]

Hello Christopher,

 

I guess – or considering our 4 year experience, even bet – due to the complicated networking environment in mainland China.

 

I just checked the webserver logs of our instance. From Aug. 26 - 28 the logs show more than normal broken connections. Fortunately, from Aug. 31 the situation normalized again. Meanwhile, we did not change anything.

 

We have an escape plan and a thought should the connection remain bad for a longer period of time:

(1) From the China customer connect to the instance via VPN; this solved issues in the past.

(2) Migrate the China instance e.g. to Alibaba Cloud, hosted in China mainland and with ICP License Support. I do not prefer this 2^nd option.

 

Regards, Eric

--
© 2017 Google Inc. 1600 Amphitheatre Parkway, Mountain View, CA 94043
 
Email preferences: You received this email because you signed up for the Google Compute Engine Discussion Google Group (gce-discussion@googlegroups.com) to participate in discussions with other members of the Google Compute Engine community and the Google Compute Engine Team.
---
You received this message because you are subscribed to a topic in the Google Groups "gce-discussion" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/gce-discussion/ieln7kHSsPc/unsubscribe.
To unsubscribe from this group and all its topics, send an email to gce-discussion+unsubscribe@googlegroups.com.
To post to this group, send email to gce-discussion@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/gce-discussion/7e59ce38-cbf6-48f0-9720-84a8462f8fde%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

[Carlos (Cloud Platform Support)]

@Kay Thanks for pointing that out, I have corrected the link.

@Eric and @Christopher

Intermittent issues are harder to troubleshoot since the information has to be caught while they are happening. If this takes place for two or more consecutive days, you do have some time to grab it.

To isolate the issue I would suggest the following:

1) Installing Stackdriver Apache plugin. It can show some application metrics that can help to discard the problem is not related to the server. You will also need to confirm that your server is not running out of resources when you experience the anomaly.

2) Capturing the MTR will let you know if there are packets being lost on one particular network node.
      a) Log the time and date the MTR is taken
      b) Log the MTR from the server to Shanghai and in the other direction (Shanghai to the server)
      c) If possible log MTR from other locations to the server.

I would also suggest having those logs when the issue is not present so it can be compared.  I believe that will help you to isolate the problem.

[Eric Aarts]

Hello Carlos,

 

This week was one with terrible connection from China to GCP.

 

We are sure our instances and the Google Singapore data center are not causing the impossibility and/or slowness to connect to from China Mainland. We monitor our instances every minute 24/7 from Europe and have no issues at all connecting, while the instances were unreachable from Shanghai.

 

This week the connection from China Mainland to our Asia instances again was very, very bad. In fact in was impossible to connect from China for >50 hours, only this week, confirmed by our customers and by the Apache access log files. To be sure, we also tried ourselves, connecting with TeamViewer from PC`s of Shanghai customers and noticed also other Google instances from US and Europe could not be connected to from China. Other international websites could be opened, sadly not our Google instances.

 

Last Thursday, out of China connection was only possible from 02:00 to 04:00 CET, on Friday only from 11:30 CET and now the situation seems to be normalized again.

 

As the bad or no connection to Google data centers from China Mainland last months seem to occur more often and have a longer duration, out of precaution we decided setting up Amazon instances, also in Singapore.

Last Friday we did checking of connection both to Google and Amazon instances from Shanghai to Singapore. Our Google instances seemed dead, to the AWS instance connection was no problem.

 

I know connect to international websites from China can be slow from time to time. Could the Google situation in China – the blocking of all Google Services by China – cause our problems? As the IP-addresses GCP issue, relate easily to Google.

 

AWS Singapore will be a temporarily solution, for now I strongly believe an ultimate solution will be moving to (e.g. Alibaba Cloud) data centers in China Mainland, with a Chinese ICP Filing.

 

Or do you have different thoughts?

 

Regards, Eric

[Eric Aarts]

An update after 6 weeks and after migrating to Alibaba Cloud in China mainland.

 

Also the Amazon EC2 instance became slower and slower. Latency was high and the instance could not be connected to for some hours per week. No difference thus between GCP and AWS.

 

In co-operation with a customer we started the Chinese ICP-filing process, necessary to be allowed to host a website in China mainland. Took some weeks.

We did an instance setup (copy of those on GCP and AWS) on Alibaba Shanghai region. You need a verified personal identification (for me after 3 attempts a EU passport was okay) to be able to create an instance in China, access by domain blocks until the ICP is registered.

 

The Chinese users experience a huge improvement, a boost in speed and connectivity.

All GCP, AWS and the Alibaba instance have an equal setup and user-load.

 

From China mainland to data centers outside (GCP or AWS; Singapore, Taiwan and Tokyo area) the connection worsened from June.

Conclusion so far for me: if you really need a working website in China – without VPN or leased internet lines – you really need China mainland hosting.