Transaction Failed Exception from a non-transactional .put()

[Joshua Smith]

This is a weird one.

I have a cron that rolls up some data every few minutes.  Nothing too complicated:

class CollectStatsHandler(webapp.RequestHandler):

  def get(self):

    nodes = NodeModel.gql("WHERE online = :1 and procs > :2", True, 0).fetch(1000)

    sum = 0

    for n in nodes:

      sum = sum + n.procs

    now = datetime.datetime.now().timetuple()

    hour = datetime.datetime(now[0], now[1], now[2], now[3])

    hourly = HourlyStatsModel.gql("WHERE hour = :1", hour).get()

    if not hourly:

      hourly = HourlyStatsModel()

      hourly.hour = hour

      hourly.stats = str(sum)

    else:

      hourly.stats = hourly.stats + "," + str(sum)

    hourly.put()    

I'm running on M/S, so every now and then I'll get timeouts or whatever, but that's OK because this is really just for my internal "app health" dashboard.

Anyway, yesterday I got this exception:

google.appengine.api.datastore_errors.TransactionFailedError@/base/python_runtime/python_lib/versions/1/google/appengine/datastore/datastore_rpc.py:1074 (at least 2 occurrences)

Handler:	main.py
URL:	http://mesonstreaming.appspot.com/admin/collectStats
Stacktrace:	Traceback (most recent call last): File "/base/python_runtime/python_lib/versions/1/google/appengine/ext/webapp/__init__.py", line 700, in __call__ handler.get(*groups) File "/base/data/home/apps/mesonstreaming/51.352130110004909428/main.py", line 1677, in get hourly.put() File "/base/python_runtime/python_lib/versions/1/google/appengine/ext/db/__init__.py", line 1006, in put return datastore.Put(self._entity, config=config) File "/base/python_runtime/python_lib/versions/1/google/appengine/api/datastore.py", line 467, in Put return PutAsync(entities, **kwargs).get_result() File "/base/python_runtime/python_lib/versions/1/google/appengine/datastore/datastore_rpc.py", line 658, in get_result results = self.__rpcs[0].get_result() File "/base/python_runtime/python_lib/versions/1/google/appengine/api/apiproxy_stub_map.py", line 592, in get_result return self.__get_result_hook(self) File "/base/python_runtime/python_lib/versions/1/google/appengine/datastore/datastore_rpc.py", line 1385, in __put_hook self.check_rpc_success(rpc) File "/base/python_runtime/python_lib/versions/1/google/appengine/datastore/datastore_rpc.py", line 1074, in check_rpc_success raise _ToDatastoreError(err) TransactionFailedError: too much contention on these datastore entities. please try again.

So we have a transaction failed error on a put which is NOT a transaction.

And we have too much contention on an entity which is written once every 5 minutes by a cron.

Huh?

-Joshua

[Ikai Lan (Google)]

Yeah, that's probably not a good thing. I'm betting the wrong error message was surfaced. Are you still seeing this?

--

Ikai Lan 
Developer Programs Engineer, Google App Engine

plus.ikailan.com | twitter.com/ikai

--
You received this message because you are subscribed to the Google Groups "Google App Engine" group.
To post to this group, send email to google-a...@googlegroups.com.
To unsubscribe from this group, send email to google-appengi...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.

[Joshua Smith]

I've seen it once more since the initial report, but not recently.

[Kartik Domadiya]

We are facing the same issue thought we have non-transnational. Any ideas ?

Here is the stacktrace:

  File "/base/data/home/runtimes/python27/python27_lib/versions/1/google/appengine/ext/db/__init__.py", line 1077, in put
    return datastore.Put(self._entity, **kwargs)
  File "/base/data/home/runtimes/python27/python27_lib/versions/1/google/appengine/api/datastore.py", line 605, in Put
    return PutAsync(entities, **kwargs).get_result()
  File "/base/data/home/runtimes/python27/python27_lib/versions/1/google/appengine/api/apiproxy_stub_map.py", line 613, in get_result
    return self.__get_result_hook(self)
  File "/base/data/home/runtimes/python27/python27_lib/versions/1/google/appengine/datastore/datastore_rpc.py", line 1881, in __put_hook
    self.check_rpc_success(rpc)
  File "/base/data/home/runtimes/python27/python27_lib/versions/1/google/appengine/datastore/datastore_rpc.py", line 1373, in check_rpc_success
    raise _ToDatastoreError(err)
TransactionFailedError: too much contention on these datastore entities. please try again.

[Ryan (Cloud Platform Support)]

Salutations Kartik,

Contention happens when you are doing to many writes per second. This is a great article on avoiding it. If you give some more details as to what you are doing I might be able to give some more specific advice.

[Karl MacMillan]

That error is about write limits on entities within the same entity group - it actually doesn’t have anything to do with a transaction.

The old datastore documentation has a better explanation of this: https://cloud.google.com/appengine/docs/python/datastore/structuring_for_strong_consistency

You can also read https://cloud.google.com/appengine/articles/scaling/contention?hl=en.

I don’t quite know what to say about the cron job every 5 minutes other than since the write limitation is per entity group it could be that writes are happening in other places in you app that are pushing you over the limit.

For myself - I’ve just structured my data so that almost nothing is within an entity group and most of the time I’m fetching by key. For those times when I need to query the nature of my app makes it fine for the results to be eventually consistent.

Karl

-- 
You received this message because you are subscribed to the Google Groups "Google App Engine" group.

To unsubscribe from this group and stop receiving emails from it, send an email to google-appengi...@googlegroups.com.

To post to this group, send email to google-a...@googlegroups.com.

Visit this group at http://groups.google.com/group/google-appengine.
To view this discussion on the web visit https://groups.google.com/d/msgid/google-appengine/474d3b75-c056-4af1-a8ae-cc6a1f81cb53%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.