Why are cloud endpoints so slow ?

[Diego Duclos]

I've done some (non extensive) tests on google appengine,

and my response times vary from anywhere between 100ms and 5000ms when directly sending http requests to a cloud endpoints.

Regardless of the actual response time, the google cloud console always shows a processing time of around 50ms, which, while also somewhat long-ish, is much more reasonable.

For the 100ms requests, I can safely know that the other 50ms are just regular latency, but I have no idea where the cloud endpoint could be spending 4.5 seconds at, and the logs show nothing useful at all.

Does anyone have some guidance for me regarding to this ? 5 seconds is unacceptable slow and makes them completely unusable.

[Robert King]

Don't get me wrong - I absolutely love cloud endpoints - they speed up my development time and simplify my code significantly.
Having said that, I'd really like to see some clarification from google. Are endpoints intended to be high performance? I haven't once seen mentioned in any google documentation that endpoints are low latency?  I've often been waiting 5-20 seconds for calls such as /_ah/api/discovery/v1/apis/archivedash/v1/rpc?fields=methods%2F*%2Fid&pp=0.
even on apps that have little traffic, tiny payloads and no rpc calls. One of the new systems i'm building is using endpoints but i'll have to switch away from endpoints ASAP if I can't get some reassurance. Also I don't have time to wait "a couple of months" to see if they get faster. I'd also be interested to know how efficient python / go / java / php endpoints are at encoding & decoding different sized payloads with json or protobuff protocols. (Will probably have to generate these statistics myself & present some graphs etc - although I'm assuming google would have already performance tested their own product?)

cheers

[Robert King]

also might be worth noting I'm using CORS on multiple app engine modules etc. perhaps it's something to do with preflight requests? http://monsur.hossa.in/2012/09/07/thoughts-on-the-cors-preflight-cache.html

[Jeff Schnitzer]

The long delays sound like loading requests. Are they not specifically marked as such in the logs? Check the old legacy log system too.

Loading requests are the achilles heel of appengine, especially for low-traffic apps. The only solution so far seems to be "use Go".

Jeff

--
You received this message because you are subscribed to the Google Groups "Google App Engine" group.
To unsubscribe from this group and stop receiving emails from it, send an email to google-appengi...@googlegroups.com.
To post to this group, send email to google-a...@googlegroups.com.
Visit this group at http://groups.google.com/group/google-appengine.
For more options, visit https://groups.google.com/d/optout.

[Diego Duclos]

These are not loading requests unfortunatly, The first request was ignored here, that one usually takes around 10 seconds (That's with Go, funnily enough)

[alex]

Could be related to this thread? 

https://groups.google.com/forum/#!topic/google-appengine/D1b_ZC4pKww

[Jun Yang]

On Thursday, May 29, 2014 3:43:14 PM UTC-7, alex wrote:

Could be related to this thread? 

https://groups.google.com/forum/#!topic/google-appengine/D1b_ZC4pKww

Hi Alex,

Compared to ProtoRPC, Endpoints does involve an extra hop. The thread was from a long time ago. Do you want to update the current numbers and how you measure it (end to end?)?

On Sunday, 25 May 2014 09:53:15 UTC+1, Robert King wrote:

Don't get me wrong - I absolutely love cloud endpoints - they speed up my development time and simplify my code significantly.

HI Robert,

Glad to hear that!

 

Having said that, I'd really like to see some clarification from google. Are endpoints intended to be high performance? I haven't once seen mentioned in any google documentation that endpoints are low latency?  

 

I've often been waiting 5-20 seconds for calls such as /_ah/api/discovery/v1/apis/archivedash/v1/rpc?fields=methods%2F*%2Fid&pp=0.
even on apps that have little traffic,

Our infrastructure handles many many APIs. And thus some parts are loaded only on demand. Currently we don't load an API everywhere when we see the first request (we should and we are working on it). If you warm your API using ~50 requests, you should see fast responses from then on.

 

tiny payloads and no rpc calls. One of the new systems i'm building is using endpoints but i'll have to switch away from endpoints ASAP if I can't get some reassurance. Also I don't have time to wait "a couple of months" to see if they get faster. I'd also be interested to know how efficient python / go / java / php endpoints are at encoding & decoding different sized payloads with json or protobuff protocols. (Will probably have to generate these statistics myself & present some graphs etc - although I'm assuming google would have already performance tested their own product?)

cheers

On Sunday, 25 May 2014 08:29:48 UTC+12, Diego Duclos wrote:

I've done some (non extensive) tests on google appengine,

and my response times vary from anywhere between 100ms and 5000ms when directly sending http requests to a cloud endpoints.

Regardless of the actual response time, the google cloud console always shows a processing time of around 50ms, which, while also somewhat long-ish, is much more reasonable.

For the 100ms requests, I can safely know that the other 50ms are just regular latency, but I have no idea where the cloud endpoint could be spending 4.5 seconds at, and the logs show nothing useful at all.

Does anyone have some guidance for me regarding to this ? 5 seconds is unacceptable slow and makes them completely unusable.

HI Diego,

Do you have the URL of your request? I'd like to see if you are seeing the same issue as Robert.

Thanks all for your feedback!

Jun

[Diego Duclos]

Hello Jun,

The endpoint is sitting at https://login-dot-psg-delta.appspot.com/_ah/api/version/v1/info

[Vinny P]

On Fri, Jun 6, 2014 at 3:14 PM, Jun Yang <jy...@google.com> wrote:

Our infrastructure handles many many APIs. And thus some parts are loaded only on demand. Currently we don't load an API everywhere when we see the first request (we should and we are working on it). If you warm your API using ~50 requests, you should see fast responses from then on.

Hi Jun, thanks for stepping in and handling some questions.

You noted in your post that the whole API isn't loaded; if we want to warm up the whole API service, should the 50 warmup requests be split among multiple API endpoints or to a single endpoint? 

Thanks. 

 

-----------------

-Vinny P

Technology & Media Advisor

Chicago, IL

App Engine Code Samples: http://www.learntogoogleit.com

[Jun Yang (杨骏)]

On Sun, Jun 8, 2014 at 12:20 AM, Vinny P <vinn...@gmail.com> wrote:

On Fri, Jun 6, 2014 at 3:14 PM, Jun Yang <jy...@google.com> wrote:

Our infrastructure handles many many APIs. And thus some parts are loaded only on demand. Currently we don't load an API everywhere when we see the first request (we should and we are working on it). If you warm your API using ~50 requests, you should see fast responses from then on.

Hi Jun, thanks for stepping in and handling some questions.

You noted in your post that the whole API isn't loaded; if we want to warm up the whole API service, should the 50 warmup requests be split among multiple API endpoints or to a single endpoint? 

50 requests to any API on the same host, e.g. 2-dot-guestbook.appspot.com or guestbook.appspot.com.

Thanks. 

 

-----------------

-Vinny P

Technology & Media Advisor

Chicago, IL

App Engine Code Samples: http://www.learntogoogleit.com

--
You received this message because you are subscribed to a topic in the Google Groups "Google App Engine" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/google-appengine/3XGJFaosX9s/unsubscribe.
To unsubscribe from this group and all its topics, send an email to google-appengi...@googlegroups.com.

[Jun Yang (杨骏)]

On Fri, Jun 6, 2014 at 11:07 PM, Diego Duclos <diego....@palmstonegames.com> wrote:

Hello Jun,

The endpoint is sitting at https://login-dot-psg-delta.appspot.com/_ah/api/version/v1/info

I put this request in a loop. Now response times range from over 100ms to over 200ms. Looks like the same issue of warm-up.

[Thomas Schranz]

We are also having some performance problems with our cloud endpoints at the moment and will spend some time to improve the response time for some endpoints.

Do you have something like 'appstats' planned for cloud endpoints or is there a way to get a breakdown re what takes how long (datastore queries, memcache, other API calls, …)?

It seems like right now if we want to get the advantages from google cloud endpoints we need to give up the performance visibility/insight that we get from 'good old' app engine request handlers or am I missing something?

(PS: we're on the python 2.7 runtime if that matters)

(Sorry for taking this thread a bit off-topic)

[Jun Yang (杨骏)]

On Wed, Jun 11, 2014 at 9:11 AM, Thomas Schranz <tho...@blossom.io> wrote:

We are also having some performance problems with our cloud endpoints at the moment and will spend some time to improve the response time for some endpoints.

Are you saying you are trying to improve the response time of your app (that is also the API backend)?

Do you have something like 'appstats' planned for cloud endpoints or is there a way to get a breakdown re what takes how long (datastore queries, memcache, other API calls, …)?

Yes.

It seems like right now if we want to get the advantages from google cloud endpoints we need to give up the performance visibility/insight that we get from 'good old' app engine request handlers or am I missing something?

Endpoints has 2 layers in its architecture: the API frontend (a proxy) and your API backend on App Engine. The second half is a regular App Engine app and all the existing tools still work there. What we are adding is the visibility into the first half.

(PS: we're on the python 2.7 runtime if that matters)

(Sorry for taking this thread a bit off-topic)

On Saturday, May 24, 2014 10:29:48 PM UTC+2, Diego Duclos wrote:

I've done some (non extensive) tests on google appengine,

and my response times vary from anywhere between 100ms and 5000ms when directly sending http requests to a cloud endpoints.

Regardless of the actual response time, the google cloud console always shows a processing time of around 50ms, which, while also somewhat long-ish, is much more reasonable.

For the 100ms requests, I can safely know that the other 50ms are just regular latency, but I have no idea where the cloud endpoint could be spending 4.5 seconds at, and the logs show nothing useful at all.

Does anyone have some guidance for me regarding to this ? 5 seconds is unacceptable slow and makes them completely unusable.

--

[Thomas Schranz]

Thanks a lot for your clarification Jun.

I did not find anything in the documentation regarding how to hook appstats up with cloud endpoints (python)

(also see this unanswered stackoverflow question: http://stackoverflow.com/questions/17353467/how-to-get-appstats-data-for-a-cloud-endpoint-method)

Looking at https://code.google.com/p/googleappengine/source/browse/trunk/python/google/appengine/ext/appstats/recording.py

it seems like we might be able to somehow call/integrate appstats recording manually but if you have a pointer

(ideally a code example like you have for WSGI apps or django apps would be incredible).

For some reason we thought cloud endpoints are incompatible with appstats, but if we can use appstats this would rock.

[Robert King]

Hi Jun,
Thanks so much for answering these questions - it's very helpful.
What does the additional frontend API layer do? What's the performance impact?

[Jun Yang (杨骏)]

On Thu, Jun 12, 2014 at 1:38 PM, Robert King <kingrob...@gmail.com> wrote:

Hi Jun,
Thanks so much for answering these questions - it's very helpful.
What does the additional frontend API layer do?

It turns your simple backend into a full blown Google API.

What's the performance impact?

Up to tens of milliseconds typically.

[J Jones]

Not sure if Google Endpoints are serious or not.  My appengine instance is warm and serving http request very quickly.  However, my endpoints are ranging from 3 seconds to 9 seconds... the endpoint just returns a hardcoded object so there's 0 happening there.  What a joke.

[Jeff Schnitzer]

I don't really understand the point of Cloud Endpoints. Adding an RPC layer that is platform-dependent? Why? This seems a major step backwards from Hessian, which has been around forever.

Hell, if you just want Android RPC (ie Java<->Java), try this:

https://github.com/stickfigure/trivet

I don't know why it wouldn't work on Android, but somebody please try it out and let me know. It's the simplest RPC protocol you could possibly imagine: It simply serializes the request and response and passes it back and forth over HTTP. Seriously, less than 250 lines of code. And it "does the right thing" with exceptions, unlike JRMP.

Jeff

--

You received this message because you are subscribed to the Google Groups "Google App Engine" group.

To unsubscribe from this group and stop receiving emails from it, send an email to google-appengi...@googlegroups.com.

[Fabio Sgrò]

We have the same problems with Endpoints, latency is a huge problem on new deploy; it slows down if you use your application and if you reuse always the same deployed version.

It seems that Google team has some problem caching configuration and discovery process, after you use your endpoints (after 50/100 calls) latency decrease a lot.

In daily use our endpoints now respond in 100/300 ms

[Dan Ciruli]

On Sat, Nov 15, 2014 at 8:28 AM, J Jones <jona...@planetjones.co.uk> wrote:

Not sure if Google Endpoints are serious or not.  My appengine instance is warm and serving http request very quickly.  However, my endpoints are ranging from 3 seconds to 9 seconds... the endpoint just returns a hardcoded object so there's 0 happening there.  What a joke.

J Jones -

Please see some notes from earlier in the thread. With our current architecture, "warming up" means not just warming up your AppEngine back end but (possibly numerous) API proxy. It may take up to 50 RPCs to ensure that your API is "warm" in proxy layer (across all instances). Try running a quick for loop to see if this improves the performance.

And: yes, we are working on an improvement to this warmup behavior. 

[Thomas Wiradikusuma]

Hi guys,

Any news on this? It's almost a year since last discussion here and Google Cloud Endpoints is still very slow.

Regarding suggestion to "hit the rpc 50x to warm it up", how do you know "when"? I mean, "hit rpc from time to time, and when it's slow, hit it 50x" doesn't sound like a good solution for me.

[Nick (Cloud Platform Support)]

Hi Thomas,

As this thread is quite old, it might be worth starting a new one which addresses any specific performance concerns you've noticed, along with data to demonstrate it. As Dan had said almost a year ago, improvements were in the works, and it's not certain that the same advice applies today.

Best wishes,

Nick