Cross-request tasklet deadlock / set_result cannot be called twice (ndb)

[Andreas Blixt]

Hello, I have a standard GAE project on Python 2.7 using Flask for handling web requests. The project has threadsafe: yes configured.

I'm seeing an issue where two requests that are very close in timing failing with two separate errors. Since these errors/requests come in pairs often, I believe they're related.

One error will say "Deadlock waiting for <Future ...>" with an arbitrary stack trace, the other will be the following set_result cannot be called twice:

Traceback (most recent call last):

  File "/base/data/home/apps/s~appname/2017-07-26-002.402945478053763657/lib/appname/flask_extras.py", line 91, in wrap

    result = func(*args, **kwargs)

  File "/base/data/home/apps/s~appname/2017-07-26-002.402945478053763657/appname/auth.py", line 242, in wrap

    result = func(session, *args, **kwargs)

  File "/base/data/home/apps/s~appname/2017-07-26-002.402945478053763657/appname/apps/api.py", line 695, in post_content_content_id_votes

    _wait_all(futures)

  File "/base/data/home/runtimes/python27/python27_lib/versions/1/google/appengine/ext/ndb/tasklets.py", line 1087, in synctasklet_wrapper

    return taskletfunc(*args, **kwds).get_result()

  File "/base/data/home/runtimes/python27/python27_lib/versions/1/google/appengine/ext/ndb/tasklets.py", line 383, in get_result

    self.check_success()

  File "/base/data/home/runtimes/python27/python27_lib/versions/1/google/appengine/ext/ndb/tasklets.py", line 378, in check_success

    self.wait()

  File "/base/data/home/runtimes/python27/python27_lib/versions/1/google/appengine/ext/ndb/tasklets.py", line 362, in wait

    if not ev.run1():

  File "/base/data/home/runtimes/python27/python27_lib/versions/1/google/appengine/ext/ndb/eventloop.py", line 268, in run1

    delay = self.run0()

  File "/base/data/home/runtimes/python27/python27_lib/versions/1/google/appengine/ext/ndb/eventloop.py", line 230, in run0

    callback(*args, **kwds)

  File "/base/data/home/runtimes/python27/python27_lib/versions/1/google/appengine/ext/ndb/tasklets.py", line 533, in _on_future_completion

    self._help_tasklet_along(ns, ds_conn, gen, val)

  File "/base/data/home/runtimes/python27/python27_lib/versions/1/google/appengine/ext/ndb/tasklets.py", line 444, in _help_tasklet_along

    self.set_result(result)

  File "/base/data/home/runtimes/python27/python27_lib/versions/1/google/appengine/ext/ndb/tasklets.py", line 322, in set_result

    raise RuntimeError('Result cannot be set twice.')

RuntimeError: Result cannot be set twice.

Is there any obvious reasons why this would happen? Could it be the treadsafe configuration?

As far as I've been able to confirm, I don't call get_result() or perform any synchronous ndb calls in any of the crashing functions, and I wait for all Futures to complete before returning from the function (that said, would ndb.toplevel(...) have any effect in this case?)

Appreciate any help!

[Yannick (Cloud Platform Support)]

Hello, while it is not certain to be the case it appears likely that has to do with threading. As this is a specific technical question the best way to get it answered would be to post it on Stack Overflow using one of the tags monitored by our community technical support team. Be sure to include the code you believe this error originates from and the full error traces. Once that's done you should post a link to it here so other posters can help answer it.