Clean way to code around timeouts?

[Bemmu]

I decided to finally do something about the Timeout exceptions
littering my log.

I read somewhere on this forum that I am supposed to code around data
store accesses to try things out several times in case of timeouts. Is
this still necessary? Why won't the methods just do that internally?

This is my first attempt to handle a timeout situation, is there any
nicer way to code this?

retries = 3
retry = True
while retry:
try:
retry = False
recipient.put()
except:
retries -= 1
if retries > 0:
retry = True
logging.info("recipient.put() failed, retrying")
else:
logging.error("failed even after trying recipient.put() three
times")

[Alkis Evlogimenos ('Αλκης Ευλογημένος)]

You can make it into a decorator which will make it easier for your functions to code. I use this:

def retry_on_timeout(retries=3, secs=1):

  """A decorator to retry a given function performing db operations."""

  def _decorator(func):

    def _wrapper(*args, **kwds):

      tries = 0

      while True:

        try:

          tries += 1

          return func(*args, **kwds)

        except db.Timeout, e:

          logging.debug(e)

          if tries > retries:

            raise e

          else:

            wait_secs = secs * tries ** 2

            logging.warning("Retrying function %r in %d secs" % (func, wait_secs))

            time.sleep(wait_secs)

    return _wrapper

  return _decorator

--

Alkis

[Bemmu]

Seems super useful, thanks a lot!

[Devel63]

Can you give an example as to how this is used? I understand the
purpose, I'm just a little hazy on the calling syntax.

On Aug 24, 5:25 am, Alkis Evlogimenos ('Αλκης Ευλογημένος)

[Alkis Evlogimenos ('Αλκης Ευλογημένος)]

@retry_on_timeout(retries=10, secs=0.2)

def some_idempotent_function():

  # do stuff

If you do not give retries or secs it defaults to 3 retries with 1 sec starting delay. The delay is exponential, it doubles after each retry.

2009/8/25 Devel63 <danst...@gmail.com>

--

Alkis

[Devel63]

Got it, thanks.

On Aug 25, 11:50 am, Alkis Evlogimenos ('Αλκης Ευλογημένος)

<evlogime...@gmail.com> wrote:
> @retry_on_timeout(retries=10, secs=0.2)
> def some_idempotent_function():  # do stuff
>
> If you do not give retries or secs it defaults to 3 retries with 1 sec
> starting delay. The delay is exponential, it doubles after each retry.
>

> 2009/8/25 Devel63 <danstic...@gmail.com>

[NealWalters]

Does the decorator work in Python 2.5.2 with GAE?
Thanks,
Neal

[Mike Wesner]

I saw this lower level way to handle Timeouts. Seems like the best
way to handle it. No need to decorate or litter your code with retry
stuff.

http://appengine-cookbook.appspot.com/recipe/autoretry-datastore-timeouts/

[johntray]

Has anyone reported problems using the autoretry cookbook method? I
added a log message to check that the wrapper is getting installed,
and I also added a log message in the retry loop. I see the log
message that the wrapper is installed, however, I do *not* see any
retry-log messages despite getting datastore timeouts.

For what it is worth, I also used the development server to verify
that the wrapper is indeed getting called, but of course, I don't
experience datastore timeouts there.

Has anyone else had problems with this code?
john

> http://appengine-cookbook.appspot.com/recipe/autoretry-datastore-time...

[Robin B]

You are absolutely correct that the code is not handling db.Timeout
exceptions correctly. When I wrote the recipe, I could not simulate
Timeouts on the production datastore, so I wrote some code to randomly
raise db.Timeouts and tested that the exponential backoff worked
well. Today, after load testing with 'ab' to hammer a test app in
production, I could still not simulate a db.Timeout, but I could
produce some TransactionFailedErrors and instead attempted to catch
and retry those. It turns out apiproxy_stub_map.MakeSyncCall does not
raise db.Timeout or db.TransactionFailedError, it raises a lower level
Exception. The recipe is now updated in the cookbook to catch the
correct low level exception.

http://appengine-cookbook.appspot.com/recipe/autoretry-datastore-timeouts

Under load, I can see the retries in my log output. If you are able
to consistently produce db.Timeouts in production, please let me know
that it is working for you too.

Thanks for the helpful feedback,

Robin

[johntray]

Great -- we'll definitely include the update in our next production
server push (this weekend) and I'll report back next week.

Thanks,
john

On Oct 21, 2:50 pm, Robin B <robi...@gmail.com> wrote:
> You are absolutely correct that the code is not handling db.Timeout
> exceptions correctly.  When I wrote the recipe, I could not simulate
> Timeouts on the production datastore, so I wrote some code to randomly
> raise db.Timeouts and tested that the exponential backoff worked
> well.  Today, after load testing with 'ab' to hammer a test app in
> production, I could still not simulate a db.Timeout, but I could
> produce some TransactionFailedErrors and instead attempted to catch
> and retry those.  It turns out apiproxy_stub_map.MakeSyncCall does not
> raise db.Timeout or db.TransactionFailedError, it raises a lower level
> Exception.  The recipe is now updated in the cookbook to catch the
> correct low level exception.
>

> http://appengine-cookbook.appspot.com/recipe/autoretry-datastore-time...