Concurrent transaction exception

60 views
Skip to first unread message

Anton Leonov

unread,
Jun 25, 2015, 6:10:12 AM6/25/15
to appscale_...@googlegroups.com
I'm working on installing our App Engine application in China.
The first step is installing the application in AppScale environment.
It had been deployed successfully and I have started test it.
After few successful tests I started to get "Concurrent transaction exception on put" (also on delete and get). After first fail all tests are failing even if it was passed successfully before.
It looks like cross group ndb transaction (or something else) breaks something around locks for transactions.

So I have tests A and tests B. Tests A are passing successfully when I just up appscale. Then I'm trying to run tests B, and it fails. After that tests A fails too.
After first fail I get "Concurrent transaction exception on put" (also on delete and get) on most of get/put/delete operations with ndb even if it is out of ndb transaction.

Error in my application logs looks like:
ERROR    2015-06-25 07:45:16,864 datastore_distributed.py:358] Concurrent transaction exception on delete.
WARNING  2015-06-25 07:45:16,865 tasklets.py:409] suspended generator _delete_tasklet(context.py:363) raised TransactionFailedError(Concurrent transaction exception on delete.)
WARNING  2015-06-25 07:45:16,866 tasklets.py:409] suspended generator delete(context.py:844) raised TransactionFailedError(Concurrent transaction exception on delete.)
CRITICAL 2015-06-25 07:45:16,867 logger.py:105]  Unexpected exception. Traceback (most recent call last):
  File "/root/appscale/AppServer/lib/webapp2-2.5.2/webapp2.py", line 570, in dispatch
    return method(*args, **kwargs)
  File "/var/apps/toi-proving-ground/app/server/security/core.py", line 43, in check_right
    return func(*args, **kwargs)
  File "/var/apps/toi-proving-ground/app/server/integration_tests_utils/namespace_cleaner.py", line 52, in post
    ndb.delete_multi(to_delete)
  File "/root/appscale/AppServer/google/appengine/ext/ndb/model.py", line 3902, in delete_multi
    for future in delete_multi_async(keys, **ctx_options)]
  File "/root/appscale/AppServer/google/appengine/ext/ndb/tasklets.py", line 325, in get_result
    self.check_success()
  File "/root/appscale/AppServer/google/appengine/ext/ndb/tasklets.py", line 368, in _help_tasklet_along
    value = gen.throw(exc.__class__, exc, tb)
  File "/root/appscale/AppServer/google/appengine/ext/ndb/context.py", line 844, in delete
    yield self._delete_batcher.add(key, options)
  File "/root/appscale/AppServer/google/appengine/ext/ndb/tasklets.py", line 368, in _help_tasklet_along
    value = gen.throw(exc.__class__, exc, tb)
  File "/root/appscale/AppServer/google/appengine/ext/ndb/context.py", line 363, in _delete_tasklet
    yield self._conn.async_delete(options, datastore_keys)
  File "/root/appscale/AppServer/google/appengine/ext/ndb/tasklets.py", line 454, in _on_rpc_completion
    result = rpc.get_result()
  File "/root/appscale/AppServer/google/appengine/api/apiproxy_stub_map.py", line 615, in get_result
    return self.__get_result_hook(self)
  File "/root/appscale/AppServer/google/appengine/datastore/datastore_rpc.py", line 1643, in __delete_hook
    self.check_rpc_success(rpc)
  File "/root/appscale/AppServer/google/appengine/datastore/datastore_rpc.py", line 1224, in check_rpc_success
    raise _ToDatastoreError(err)
TransactionFailedError: Concurrent transaction exception on delete.


Error in datastore logs:
WARNING:root:Concurrent transaction exception for app id toi-proving-ground with info acquire_additional_lock: There is already another transaction using /appscale/apps/toi-proving-ground/locks/toi-proving-ground%00%00Company%3Aaleonov-test%01 lock
WARNING:root:Trying again to acquire lockinfo acquire_additional_lock: There is already another transaction using /appscale/apps/toi-proving-ground/locks/toi-proving-ground%00%00Company%3Aaleonov-test%01 lock with retry #2
ERROR:root:Lock /appscale/apps/toi-proving-ground/locks/toi-proving-ground%00%00Company%3Aaleonov-test%01 in use by /appscale/apps/toi-proving-ground/txids/tx0000001299
WARNING:root:Concurrent transaction exception for app id toi-proving-ground with info acquire_additional_lock: There is already another transaction using /appscale/apps/toi-proving-ground/locks/toi-proving-ground%00%00Company%3Aaleonov-test%01 lock
WARNING:root:Trying again to acquire lockinfo acquire_additional_lock: There is already another transaction using /appscale/apps/toi-proving-ground/locks/toi-proving-ground%00%00Company%3Aaleonov-test%01 lock with retry #1
ERROR:root:Lock /appscale/apps/toi-proving-ground/locks/toi-proving-ground%00%00Company%3Aaleonov-test%01 in use by /appscale/apps/toi-proving-ground/txids/tx0000001299
WARNING:root:Concurrent transaction exception for app id toi-proving-ground with info acquire_additional_lock: There is already another transaction using /appscale/apps/toi-proving-ground/locks/toi-proving-ground%00%00Company%3Aaleonov-test%01 lock
ERROR:root:Notify failed transaction removing lock: /appscale/apps/toi-proving-ground/txids/tx0000001306
ERROR:root:Concurrent transaction exception for app id toi-proving-ground, info acquire_additional_lock: There is already another transaction using /appscale/apps/toi-proving-ground/locks/toi-proving-ground%00%00Company%3Aaleonov-test%01 lock


To make tests A passed successfully I have to shut down appscale and up it again.
All these tests are Ok in GAE environment.

Thank You.
Anton

Anton Leonov

unread,
Jun 25, 2015, 6:55:43 AM6/25/15
to appscale_...@googlegroups.com
I'm sorry. I was wrong.
Nothing is broken in datastore. It fails independently of previous actions with datastore. It fails when I'm using ndb.transaction.
Please ignore the nonsense which I wrote about tests A and B and about restart of appscale.

But problem does exist. Code with ndb.transaction which works fine on GAE, fails in AppScale environment.

How I have to work with datastore transaction in AppScale environment?

Navraj S. Chohan

unread,
Jun 25, 2015, 9:11:50 AM6/25/15
to appscale_...@googlegroups.com

Hey there,
Local datastore expert here. The difference between the two is that AppScale uses pessimistic locking and the GAE uses optimistic concurrency control.
The API itself is almost 100% compatible minus the test that fail because of this issue. Maybe AppScale should be more optimistic also. ;)
-Raj

--
You received this message because you are subscribed to the Google Groups "AppScale Community" group.
To unsubscribe from this group and stop receiving emails from it, send an email to appscale_commun...@googlegroups.com.
To post to this group, send email to appscale_...@googlegroups.com.
Visit this group at http://groups.google.com/group/appscale_community.
For more options, visit https://groups.google.com/d/optout.

Anton Leonov

unread,
Jun 25, 2015, 10:40:51 AM6/25/15
to appscale_...@googlegroups.com
Thank You, Raj!

Considering that my application doesn't want to work with pessimist, it would be good if AppScale (or Cassandra) were more optimistic=)

Navraj S. Chohan

unread,
Jun 25, 2015, 10:27:10 PM6/25/15
to appscale_...@googlegroups.com

The team at AppScale is always optimistic :)

--

Anton Leonov

unread,
Jun 26, 2015, 9:27:05 AM6/26/15
to appscale_...@googlegroups.com
Raj, could You explain me, what exactly is cause of Concurrent transaction exception?
It have to be simple: many transactions are trying to read or write entities from one entity group => error.
In the optimistic strategy checking of collisions is made on the commit stage.
In the pessimistic strategy it is made on any read or write operation.

But I get this error in code that is OUT OF TRANSACTION when I call ndb.delete_multi(keys).
Maybe transaction is created inside delete_multi.. but it shouldn't..

I can't understand what is happening there..

Navraj Chohan

unread,
Jul 5, 2015, 7:11:45 AM7/5/15
to appscale_...@googlegroups.com

Sorry I didn't get to this earlier. When I can figure it out I'll get back to you.
-Raj

--

Graziano Obertelli

unread,
Jul 5, 2015, 1:04:05 PM7/5/15
to appscale_...@googlegroups.com
In this case the error was because on a specific error condition (user
raised exception) the path wouldn't release a lock, thus creating the
error or multiple path trying to acquire the same transaction lock
(well the same entity). The fix is already in master and we we are
running long term regression testing for it.

cheers
graziano
> > > > > > > > -test%01 lock with retry #1
> > > > > > > > ERROR:root:Lock /appscale/apps/toi-proving
> > > > > > > > -ground/locks/toi-proving-ground%00%00Company%3Aaleonov
> > > > > > > > -test%01 in use by /appscale/apps/toi-proving
> > > > > > > > -ground/txids/tx0000001299
> > > > > > > > WARNING:root:Concurrent transaction exception for app
> > > > > > > > id toi-proving-ground with info
> > > > > > > > acquire_additional_lock: There is already another
> > > > > > > > transaction using /appscale/apps/toi-proving
> > > > > > > > -ground/locks/toi-proving-ground%00%00Company%3Aaleonov
> > > > > > > > -test%01 lock
> > > > > > > > ERROR:root:Notify failed transaction removing lock:
> > > > > > > > /appscale/apps/toi-proving-ground/txids/tx0000001306
> > > > > > > > ERROR:root:Concurrent transaction exception for app id
> > > > > > > > toi-proving-ground, info acquire_additional_lock: There
> > > > > > > > is already another transaction using /appscale/apps/toi
> > > > > > > > -proving-ground/locks/toi-proving
> > > > > > > > -ground%00%00Company%3Aaleonov-test%01 lock
> > > > > > > >
> > > > > > > To make tests A passed successfully I have to shut down
> > > > > > > appscale and up it again.
> > > > > > > All these tests are Ok in GAE environment.
> > > > > > >
> > > > > > > Thank You.
> > > > > > > Anton
> > > > > > >
--
Graziano Obertelli
AppScale Systems, Inc.


Reply all
Reply to author
Forward
0 new messages