app engine performance tuning

[DragonFist]

Latency for my application seems to be very high even though I've optimized it significantly.

Here is an appstats sample for one of the requests:

(4) 2013-01-16 08:33:19.750 "GET /" 200 real=3154ms api=0ms overhead=12ms (39 RPCs, cost=0, billed_ops=[])
images.GetUrlBase   18  0   
memcache.Get            10  0   
datastore_v3.Get    9   0   
datastore_v3.RunQuery   1   0   
user.CreateLogoutURL    1   0

Seems images.GetUrlBase is one of the main culprit as well as memcache (?). Currently I have blob being fetched and passed to get_serving_url, with variable image sizes throughout the application.

Any recommendation on performance improvements? thanks

[DragonFist]

 

figured out the main problem is in fact the get_serving_url, because its fetching the blob file and passing in the key each time. any ideas on caching methods for this particular situation?

[DragonFist]

This seems ridiculous that only a few images would incur so much overhead. 5+ seconds to load a page with 8-10 images via get_serving_url?

[Aleksei Rovenski]

how about caching the url? calling ger_serving_url once when you create the image blob and save it where you keep blob key
 
среда, 16 января 2013 г., 20:11:20 UTC+2 пользователь Joker321 написал:

[Jesse]

I never use get_serving_url on user-facing requests--it's slow and it sometimes craps out and needs a retry.  Use it a task queue once and save the resulting URL onto your model.  Then just manipulate the URL as needed (a property would make this uber simple).

[Igor Kharin]

I agree with Jesse--get_serving_url() isn't designed for online URL generation, it's slow and raises deadlines reagularly. It is mostly surprising it worked for you that way so far e.g. my actual code:

for tries in xrange(1, 4):
    try:
        url = images.get_serving_url(key)
        break
    except:
        logging.warning(...)
else:
    logging.error(...)

--
You received this message because you are subscribed to the Google Groups "Google App Engine" group.

To view this discussion on the web visit https://groups.google.com/d/msg/google-appengine/-/pA887tjIKPgJ.

To post to this group, send email to google-a...@googlegroups.com.
To unsubscribe from this group, send email to google-appengi...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.

[Joker321]

Thanks for the feedback.

I solved the issue by caching the get_serving_url for subsequent requests.

something like this (not exact code):

def image(self, size=0):
 try:
    not_found '/images/not_found.jpg'
    image_url = memcache.get('some_key:' + self.id + ':' + str(size))
    if not image_url:
     if self.image:
      blob_key = self.image.blob_key
      image_url = get_serving_url(blob_key, size)
      memcache.set('some_key:' + self.id + ':' + str(size), image_url)
     else:
      image_url = not_found
 except:
  image_url = not_found
  #logging error
 return image_url

[Joker321]

This has decreased the latency significantly from 5-9 seconds now to 300-800 ms

[Joker321]

Hi,

I'm not sure what you mean get_serving_url isn't "designed for online URL generation".

If you mean by calling get_serving_url directly then I agree, but otherwise its exactly what its designed for.

[Joker321]

task queue. why?

that would be overcomplicated.

[Joker321]

yes, thanks for the feedback. I went with this solution and it helped significantly.