Long latency/timeouts to Google Cloud Storage since 7/29/2016

Shaun Murphy

unread,

Aug 1, 2016, 3:35:56 PM8/1/16

to gce-discussion

Last week was saw a remarkable increase in latency and frequent timeouts connecting to one of our google cloud storage-enabled projects. We wrote a small commandline app that simply lists our buckets and we're seeing about 5% of the requests have a socket timeout and 20% of the requests have greater than 3s latency. The rest of the requests are under 200ms.

Any idea what happened or recommendations to fix?

Here is the MTR :

mtr -4 -n -r -c 100 www.googleapis.com

Start: Mon Aug 1 14:54:59 2016

HOST: KALI_SNDR_BLOCK Loss% Snt Last Avg Best Wrst StDev

1.|-- 192.168.110.1 0.0% 100 0.3 0.3 0.3 0.5 0.0

2.|-- 192.168.0.1 0.0% 100 0.9 2.0 0.9 50.9 5.4

3.|-- 10.123.160.1 0.0% 100 9.8 12.5 8.5 28.1 3.3

4.|-- 72.31.218.188 0.0% 100 9.7 12.9 8.9 33.9 3.8

5.|-- 72.31.194.168 0.0% 100 12.9 16.0 11.2 42.0 4.2

6.|-- 72.31.220.228 0.0% 100 12.9 15.9 10.9 33.8 4.1

7.|-- 66.109.6.98 0.0% 100 28.9 19.4 11.3 41.3 5.0

8.|-- 66.109.1.72 0.0% 100 28.3 28.1 20.8 54.3 5.5

9.|-- 205.197.180.41 0.0% 100 20.2 24.1 19.5 46.0 4.4

10.|-- 207.88.13.52 0.0% 100 26.6 24.2 19.9 49.4 4.4

11.|-- 207.88.13.49 0.0% 100 24.8 25.0 19.7 36.0 3.8

12.|-- 216.156.108.114 0.0% 100 21.0 27.0 19.8 63.5 6.5

13.|-- 72.14.233.56 0.0% 100 42.8 39.0 34.9 50.4 3.2

14.|-- 209.85.142.138 71.0% 100 45.0 41.2 36.1 49.4 3.2

15.|-- 216.239.48.146 0.0% 100 35.1 38.6 34.4 55.4 3.5

16.|-- 216.239.57.225 0.0% 100 36.0 50.9 34.9 88.2 19.3

17.|-- ??? 100.0 100 0.0 0.0 0.0 0.0 0.0

18.|-- 209.85.232.95 0.0% 100 36.0 38.8 33.9 61.0 4.5

Our test program (code snippet at the end) shows some of the successful connections but with 1+second latency

1: Getting list of buckets took 22079 ms

2: Getting list of buckets took 180 ms

3: Getting list of buckets took 143 ms

4: Getting list of buckets took 159 ms

...

104: Getting list of buckets took 129 ms

105: Getting list of buckets took 2510 ms

106: Getting list of buckets took 131 ms

Our test program uses the java com.google.cloud client to connect and lists the buckets with a time for each

try {

Storage storage = StorageOptions.builder()

.projectId(GCS_PROJECT_ID)

.authCredentials(AuthCredentials.createForJson(new FileInputStream(GCS_JSON_KEY)))

.build()

.service();

final int numberOfSamples = 1000;

IntStream.rangeClosed(1, numberOfSamples).forEach(iteration -> {

long startBucketListingTime = System.currentTimeMillis();

Iterator<Bucket> bucketIterator = storage.list().iterateAll();

System.out.println(iteration + ": Getting list of buckets took " + (System.currentTimeMillis() - startBucketListingTime) + " ms");

});

} catch (IOException ex) {

java.util.logging.Logger.getLogger(Application.class.getName()).log(Level.SEVERE, null, ex);

}

Faizan (Google Cloud Support)

unread,

Aug 1, 2016, 7:21:50 PM8/1/16

to gce-discussion

Hello Shaun,

Can you use perfdiag command to run performance diagnostic tests for a given bucket and post the results on this thread ?

Looking forward to your response.

Faizan

Shaun Murphy

unread,

Aug 1, 2016, 11:04:13 PM8/1/16

to gce-discussion

I emailed the original output json to gs-team@goo.... Attached redacted output

We imaged a brand new machine so we would have a pristine gcloud environment but it took several attempts to gcloud init due to various timeouts (seen in the prior MTR output) - Note we have no proxy on this test bench, nat to wan

Welcome! This command will take you through the configuration of gcloud.

Your current configuration has been set to: [default]

Network diagnostic detects and fixes local network connection issues.
Checking network connection...done.
ERROR: Reachability Check failed.
Cannot reach https://accounts.google.com (ResponseNotReady)
Cannot reach https://www.googleapis.com/auth/cloud-platform (ResponseNotReady)
Network connection problems may be due to proxy or firewall settings.

Do you have a network proxy you would like to set in gcloud (Y/n)?

output.json

Shaun Murphy

unread,

Aug 2, 2016, 9:27:15 PM8/2/16

to gce-discussion

Any additional thoughts on this? It seemed to get a bit better on average today but we're still having some timeouts and double-digit latency

Faizan (Google Cloud Support)

unread,

Aug 3, 2016, 5:15:05 PM8/3/16

to gce-discussion

Hello Shaun,

I believe you ran this test from your local machine. Is it possible to run the same test from a GCE instance just to confirm if the issue is related to the network or the service itself.

Looking forward to your response.

Faizan

Shaun Murphy

unread,

Aug 3, 2016, 8:32:13 PM8/3/16

to gce-discussion

This high latency is for google cloud storage, not gce

Paul Nash

unread,

Aug 4, 2016, 2:01:49 AM8/4/16

to Shaun Murphy, gce-discussion

Hi Shaun,

I think what Faizan was suggesting is, if you can run a similar test from a VM to GCS, that side-steps all (most) the networking between you and us. This would help us (and you) determine if there is something in the network path that is causing the issue.

--
© 2016 Google Inc. 1600 Amphitheatre Parkway, Mountain View, CA 94043

Email preferences: You received this email because you signed up for the Google Compute Engine Discussion Google Group (gce-dis...@googlegroups.com) to participate in discussions with other members of the Google Compute Engine community and the Google Compute Engine Team.
---
You received this message because you are subscribed to the Google Groups "gce-discussion" group.
To unsubscribe from this group and stop receiving emails from it, send an email to gce-discussio...@googlegroups.com.
To post to this group, send email to gce-dis...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/gce-discussion/dba7c43b-cef7-462b-93e8-48b3a23f924b%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--

Paul R. Nash |

Product Manager, Compute Engine |

paul...@google.com |

206-876-1620

Reply all

Reply to author

Forward