images.GetUrlBase deadline problems since yesterday morning

[blackpawn]

I figured this would just clear up after awhile but since it's been
over 24 hours I thought I better post so it can get checked out. For
a day now I'm getting tons of these errors:

The API call images.GetUrlBase() took too long to respond and was
cancelled

That API is regularly taking over 5 seconds and getting aborted and
before Thursday morning had no problems. Meanwhile the system status
page says latencies are fine. O_o Has there been some change to
Images or Blobstore that requires us to update our servers or is this
a temporary problem?

[pamela]

I am also experiencing an increased number of these (although I also
have more traffic going to my app generally, so the increase in errors
may be proportional). Either way, I call this function on a deferred
task, and the most recent task has retried 15 times trying to call
GetUrlBase. Is there anything I can do?
(My app id is everyday-app, if that is useful).

I see this issue is also logged here:
http://code.google.com/p/googleappengine/issues/detail?id=6771

[Amy Unruh]

On Tue, Jan 24, 2012 at 1:19 PM, pamela <pame...@eatdifferent.com> wrote:

I am also experiencing an increased number of these (although I also
have more traffic going to my app generally, so the increase in errors
may be proportional). Either way, I call this function on a deferred
task, and the most recent task has retried 15 times trying to call
GetUrlBase. Is there anything I can do?
(My app id is everyday-app, if that is useful).

I see this issue is also logged here:
http://code.google.com/p/googleappengine/issues/detail?id=6771

So that we can gather as much info as possible, if anyone else has been seeing a large number of GetUrlBase timeouts, can you add a note to the issue above (or post to this thread)?

 

On Jan 20, 2:36 pm, blackpawn <pharmapsycho...@gmail.com> wrote:
> I figured this would just clear up after awhile but since it's been
> over 24 hours I thought I better post so it can get checked out.  For
> a day now I'm getting tons of these errors:
>
> The API call images.GetUrlBase() took too long to respond and was
> cancelled
>
> That API is regularly taking over 5 seconds and getting aborted and
> before Thursday morning had no problems.  Meanwhile the system status
> page says latencies are fine.  O_o  Has there been some change to
> Images or Blobstore that requires us to update our servers or is this
> a temporary problem?

--
You received this message because you are subscribed to the Google Groups "Google App Engine" group.
To post to this group, send email to google-a...@googlegroups.com.
To unsubscribe from this group, send email to google-appengi...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.

[blackpawn]

After a week this is still killing my app.  :(  Would love any extra info I can get on the issue or things I can do to help track it down or work around it.

[Pamela Fox]

Here's how I am currently dealing with it, if it helps:

1) I upload blobs using the deferred queue, and save the blob_key to an entity. I try to save the URLs in the entity via get_serving_url, but if it fails, I set them to a designated empty URL (e.g. "blank.png")

2) On a cron job (run every 20 minutes), I find all the entities with empty URLs, and attempt to save the URLs then. Eventually I get the URLs for them.

- pamela

On Wed, Jan 25, 2012 at 12:41 PM, blackpawn <pharmap...@gmail.com> wrote:

After a week this is still killing my app.  :(  Would love any extra info I can get on the issue or things I can do to help track it down or work around it.

--

You received this message because you are subscribed to the Google Groups "Google App Engine" group.

To view this discussion on the web visit https://groups.google.com/d/msg/google-appengine/-/9-YOHmh9BdIJ.

[Stuart Langley]

FYI I've extended the deadline from 5 sec to 15 sec. This change will be available in the 1.6.2 release. 

It would be great to get some feedback on any improvements that are noticed with this chance once 1.6.2 is available.

[blackpawn]

this doesn't seem to have made a difference, i've updated to 1.6.2 and still see this time out with 5 second requests.  :/

[Brandon Wirtz]

I think there are still performance issues. Wide spread. I’m seeing it on several apps, higher than average errors, longer latencies and higher than normal CPU usage.

 

Brandon Wirtz BlackWaterOps: President / Lead Mercenary
Work: 510-992-6548 Toll Free: 866-400-4536 IM: dra...@gmail.com (Google Talk) Skype: drakegreene YouTube: BlackWaterOpsDotCom BlackWater Ops Cloud On A String Mastermind Group

--

You received this message because you are subscribed to the Google Groups "Google App Engine" group.

To view this discussion on the web visit https://groups.google.com/d/msg/google-appengine/-/KnNwuimimi8J.

[blackpawn]

thanks for sharing your solution pamela.  i have some image processing that happens in task queues and those retry and retry and retry until they finally get a serving URL.  i'm still reworking parts that the users should be able to expect happen immediately to handle some notice that the changes will be applied eventually or something.  (rotating images for example)

[Ubaldo Huerta]

I'm observing the issue right now.

The API call images.GetUrlBase() took too long to respond and was cancelled.
Traceback (most recent call last):
  File "/base/data/home/apps/s~yagruma-site/433.356791994696982198/handlers/project.py", line 177, in get
    project.set_image(image_blob)
  File "/base/data/home/apps/s~yagruma-site/433.356791994696982198/models/__init__.py", line 214, in set_image
    self.lowres_image_url = get_serving_url(lowres_image_blob_key)
  File "/base/python_runtime/python_lib/versions/1/google/appengine/api/images/__init__.py", line 1273, in get_serving_url
    response)
  File "/base/python_runtime/python_lib/versions/1/google/appengine/api/apiproxy_stub_map.py", line 94, in MakeSyncCall
    return stubmap.MakeSyncCall(service, call, request, response)
  File "/base/python_runtime/python_lib/versions/1/google/appengine/api/apiproxy_stub_map.py", line 308, in MakeSyncCall
    rpc.CheckSuccess()
  File "/base/python_runtime/python_lib/versions/1/google/appengine/api/apiproxy_rpc.py", line 133, in CheckSuccess
    raise self.exception
DeadlineExceededError: The API call images.GetUrlBase() took too long to respond and was cancelled.

[Andreas]

got a few of them too.

--
You received this message because you are subscribed to the Google Groups "Google App Engine" group.

To view this discussion on the web visit https://groups.google.com/d/msg/google-appengine/-/i83Hg7yJsiIJ.

[Amy Unruh]

For those of you seeing this deadline issue, do you see any pattern w.r.t. it occurring with large images?  Approximately how large?  

Are you running 1.6.2?

[Joseph Letness]

Hi Amy, I'm running 1.6.2 and I've tried with both a 4k PNG and a 500k
JPEG and both hit the deadline exceeded error after 5 sec. Earlier in
this thread, Stuart Langley stated that the deadline for
images.GetUrlBase() had been increased to 15 sec but that does not
seem to be the case.

Also, I'm encountering this on two apps, one python 2.5 and the other
2.7

Thanks.

--Joe

On Feb 14, 2:05 pm, Amy Unruh <amyu+gro...@google.com> wrote:
> For those of you seeing this deadline issue, do you see any pattern w.r.t.
> it occurring with large images?  Approximately how large?
> Are you running 1.6.2?
>
>
>
>
>
>
>
> On Wed, Feb 15, 2012 at 1:17 AM, Andreas <a.schmi...@gmail.com> wrote:
> > got a few of them too.
>
> > On Feb 14, 2012, at 7:46 AM, Ubaldo Huerta wrote:
>
> > I'm observing the issue right now.
>
> > The API call images.GetUrlBase() took too long to respond and was cancelled.
> > Traceback (most recent call last):

> >   File "/base/data/home/apps/s~yagruma-site/433.356791994696982198/handlers/projec t.py", line 177, in get
> >     project.set_image(image_blob)
> >   File "/base/data/home/apps/s~yagruma-site/433.356791994696982198/models/__init__ .py", line 214, in set_image
> >     self.lowres_image_url = get_serving_url(lowres_image_blob_key)
> >   File "/base/python_runtime/python_lib/versions/1/google/appengine/api/images/__i nit__.py", line 1273, in get_serving_url
> >     response)
> >   File "/base/python_runtime/python_lib/versions/1/google/appengine/api/apiproxy_s tub_map.py", line 94, in MakeSyncCall

> >     return stubmap.MakeSyncCall(service, call, request, response)

> >   File "/base/python_runtime/python_lib/versions/1/google/appengine/api/apiproxy_s tub_map.py", line 308, in MakeSyncCall
> >     rpc.CheckSuccess()
> >   File "/base/python_runtime/python_lib/versions/1/google/appengine/api/apiproxy_r pc.py", line 133, in CheckSuccess

[blackpawn]

can't say i've noticed any difference with size but all my images are
pretty consistent sizes (~1MB png and converted to ~100KB jpg)
getting a serving URL for them (both png and jpg versions) has
randomly failed with time out for four weeks now. so the error log for
my app has been filled with these as every few minutes an image is
uploaded and the serving URL is attempted and a good percentage of the
time fails. the app catches the exception and queues a task to
retry. usually the first retry works but sometimes it'll retry half a
dozen times before succeeding. the 1.6.2 update didn't seem to have
any impact and it still times out after 5 seconds. it'd be nice to
have this fixed as soon as possible or to have some billing change
since the past month i've been paying more as it requires all these
instances to keep retrying this API. :-/

hope this info helps! would love all the info and updates possible on
this bug... the Images service is fantastic with the easy serving of
different scales and cropping options and all that.

blackpawn

On Feb 14, 12:05 pm, Amy Unruh <amyu+gro...@google.com> wrote:
> For those of you seeing this deadline issue, do you see any pattern w.r.t.
> it occurring with large images?  Approximately how large?
> Are you running 1.6.2?
>
>
>
>
>
>
>
> On Wed, Feb 15, 2012 at 1:17 AM, Andreas <a.schmi...@gmail.com> wrote:
> > got a few of them too.
>
> > On Feb 14, 2012, at 7:46 AM, Ubaldo Huerta wrote:
>
> > I'm observing the issue right now.
>
> > The API call images.GetUrlBase() took too long to respond and was cancelled.
> > Traceback (most recent call last):

> >   File "/base/data/home/apps/s~yagruma-site/433.356791994696982198/handlers/projec t.py", line 177, in get
> >     project.set_image(image_blob)
> >   File "/base/data/home/apps/s~yagruma-site/433.356791994696982198/models/__init__ .py", line 214, in set_image
> >     self.lowres_image_url = get_serving_url(lowres_image_blob_key)
> >   File "/base/python_runtime/python_lib/versions/1/google/appengine/api/images/__i nit__.py", line 1273, in get_serving_url
> >     response)
> >   File "/base/python_runtime/python_lib/versions/1/google/appengine/api/apiproxy_s tub_map.py", line 94, in MakeSyncCall

> >     return stubmap.MakeSyncCall(service, call, request, response)

> >   File "/base/python_runtime/python_lib/versions/1/google/appengine/api/apiproxy_s tub_map.py", line 308, in MakeSyncCall
> >     rpc.CheckSuccess()
> >   File "/base/python_runtime/python_lib/versions/1/google/appengine/api/apiproxy_r pc.py", line 133, in CheckSuccess

[Stuart Langley]

Does you app typically call get_serving_url on the same set of images over an over, or do you typically only call it once per blob?

Just trying to understand you application pattern and how you use this API.

[Joseph Letness]

Hi Stuart, the pattern that I use involves calling get_serving_url
once, without size or crop arguments, and then storing the result as a
string in the entity that references the blob_info.

Thanks for any help getting this resolved.

[Stuart Langley]

Got it - thanks.

[jon]

I'm using 1.6.1.1 and it's happening right now. Every attempt to
upload an image fails consistently. The images are jpg, relatively
small, only about 37K in size.

On Feb 15, 7:05 am, Amy Unruh <amyu+gro...@google.com> wrote:
> For those of you seeing this deadline issue, do you see any pattern w.r.t.
> it occurring with large images?  Approximately how large?
> Are you running 1.6.2?
>
>
>
>
>
>
>
> On Wed, Feb 15, 2012 at 1:17 AM, Andreas <a.schmi...@gmail.com> wrote:
> > got a few of them too.
>
> > On Feb 14, 2012, at 7:46 AM, Ubaldo Huerta wrote:
>
> > I'm observing the issue right now.
>
> > The API call images.GetUrlBase() took too long to respond and was cancelled.
> > Traceback (most recent call last):

> >   File "/base/data/home/apps/s~yagruma-site/433.356791994696982198/handlers/projec t.py", line 177, in get
> >     project.set_image(image_blob)
> >   File "/base/data/home/apps/s~yagruma-site/433.356791994696982198/models/__init__ .py", line 214, in set_image
> >     self.lowres_image_url = get_serving_url(lowres_image_blob_key)
> >   File "/base/python_runtime/python_lib/versions/1/google/appengine/api/images/__i nit__.py", line 1273, in get_serving_url
> >     response)
> >   File "/base/python_runtime/python_lib/versions/1/google/appengine/api/apiproxy_s tub_map.py", line 94, in MakeSyncCall

> >     return stubmap.MakeSyncCall(service, call, request, response)

> >   File "/base/python_runtime/python_lib/versions/1/google/appengine/api/apiproxy_s tub_map.py", line 308, in MakeSyncCall
> >     rpc.CheckSuccess()
> >   File "/base/python_runtime/python_lib/versions/1/google/appengine/api/apiproxy_r pc.py", line 133, in CheckSuccess

[Stuart Langley]

Same question as before, are you calling get_serving_url many times for the same blob_key or are you only calling it once and storing the value?

[jon]

I use Java and we call getServingUrl() twice for the same key.
Afterwards the stored value will be used. Note that this error
happened on the first getServiceUrl() call.

Interesting to note that I made the problem go away by redeploying.

[Stuart Langley]

Can you send me your app-id?

Thanks

[Guy Armitage]

We're experiencing the same issue, with varied images sizes and types.

We are also getServingUrl() twice on a given key. Our app id is
zealous-co (www.zealous.co / http://8.zealous-co.appspot.com/)

[Stuart Langley]

Please see the comment I added to the issue in the tracker.

http://code.google.com/p/googleappengine/issues/detail?id=6771

[Richard Spillane]

I've been experiencing this exact same problem, but for the go client.  

When I upload a small image (<1M), then call getServingUrl on the blob key, everything works fine, but when I upload a larger image (8M was what I tried), I consistently got timeout error when calling image.getServingUrl.  

[Vinny P]

On Sun, Aug 11, 2013 at 6:39 PM, Richard Spillane <necr...@gmail.com> wrote:

I've been experiencing this exact same problem, but for the go client.  

When I upload a small image (<1M), then call getServingUrl on the blob key, everything works fine, but when I upload a larger image (8M was what I tried), I consistently got timeout error when calling image.getServingUrl.  
 

 

 

How soon are you calling getServingUrl after you write/upload the image? As the issue you linked stated, the blobstore needs some time to process the image before it can provide a serving URL.

 

 

-----------------

-Vinny P

Technology & Media Advisor

Chicago, IL

App Engine Code Samples: http://www.learntogoogleit.com

 

[Richard Spillane]

Hi Vinny, so what is the best practice in this case? Our website has to wait for the image to become available. So, should we poll somehow (repeatedly call getServingUrl())? Is there a way to be notified when getServingUrl() will no longer fail (some kind of completion callback we need to fill out)? What is the best way to wait for getServingUrl() to succeed?

--
You received this message because you are subscribed to a topic in the Google Groups "Google App Engine" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/google-appengine/Fj2MLH-wJIg/unsubscribe.
To unsubscribe from this group and all its topics, send an email to google-appengi...@googlegroups.com.

To post to this group, send email to google-a...@googlegroups.com.

Visit this group at http://groups.google.com/group/google-appengine.
For more options, visit https://groups.google.com/groups/opt_out.

--
Dr. Richard P. Spillane
necr...@gmail.com

[Vinny P]

On Sun, Aug 18, 2013 at 5:44 PM, Richard Spillane <necr...@gmail.com> wrote:

Our website has to wait for the image to become available. So, should we poll somehow (repeatedly call getServingUrl())? Is there a way to be notified when getServingUrl() will no longer fail (some kind of completion callback we need to fill out)? What is the best way to wait for getServingUrl() to succeed?

 

 

For a previous project I did, I scheduled a task to repeatedly attempt to call getServingUrl(). If the task failed, the Task Queue service automatically retried the task until it succeeded.

 

As for "best way", it really doesn't matter what option you choose, you just need to find a way to buy time until GAE processes your uploaded image.

[Richard Spillane]

Thanks for the advice Vinny, much appreciated.

--
You received this message because you are subscribed to a topic in the Google Groups "Google App Engine" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/google-appengine/Fj2MLH-wJIg/unsubscribe.
To unsubscribe from this group and all its topics, send an email to google-appengi...@googlegroups.com.
To post to this group, send email to google-a...@googlegroups.com.
Visit this group at http://groups.google.com/group/google-appengine.
For more options, visit https://groups.google.com/groups/opt_out.

[Kaan Soral]

Let me haunt this thread, instead of creating a new one:

DeadlineExceededError: The API call images.GetUrlBase() took too long to respond and was cancelled.

My app received these errors today, during a time period, for the first time

It's pretty devastating losing an uploaded image, and potentially a user because of errors like these

Any ideas? Will it re-occur, should I improve my upload routines?

Thanks in Advance

[Amandeep Singh]

Facing the same problem recently thought of a ppossible solution,
The google.appengine.api.images has a method create_rpc.The create_rpc take deadline as param. Hence, one cloud initiate an RPC with greater deadline value then the default 5sec and pass that to images.get_serving_url method.

images.get_serving_url(blob_key, secure_url=True, rpc=create_rpc(deadline=15))

Haven't deployed this yet, hoping this might solve the problem else would need to revert to loop to try getting url n number of times.

On Saturday, 21 January 2012 04:06:15 UTC+5:30, blackpawn wrote:

I figured this would just clear up after awhile but since it's been
over 24 hours I thought I better post so it can get checked out.  For
a day now I'm getting tons of these errors:

The API call images.GetUrlBase() took too long to respond and was
cancelled

That API is regularly taking over 5 seconds and getting aborted and
before Thursday morning had no problems.  Meanwhile the system status
page says latencies are fine.  O_o  Has there been some change to
Images or Blobstore that requires us to update our servers or is this
a temporary problem?