java.util.ConcurrentModificationException on committing a transaction with only task queue entries

[FastGeert]

Hi, 

last night our application suffered from java.util.ConcurrentModificationException on committing a transaction with only task queue entries.

The transaction only had to persist 5 task queue entries, no additional changes needed to be committed in the transaction.

After plunging through the documentation I conclude that this must be a bug in App Engine.

Any thoughts on this ?

[Michael Hermus]

I have just experienced this same issue. I submit 4 tasks to 4 different queues as a part of a single transaction. If multiple requests come in rapid succession, the following exception is raised upon commit:

 

java.util.ConcurrentModificationException: too much contention on these datastore entities. please try again.
	at com.google.appengine.api.datastore.DatastoreApiHelper.translateError(DatastoreApiHelper.java:39)
	at com.google.appengine.api.datastore.DatastoreApiHelper$1.convertException(DatastoreApiHelper.java:76)
	at com.google.appengine.api.utils.FutureWrapper.get(FutureWrapper.java:106)
	at com.google.appengine.api.utils.FutureWrapper.get(FutureWrapper.java:90)
	at com.google.appengine.api.utils.FutureWrapper.get(FutureWrapper.java:90)
	at com.googlecode.objectify.cache.TriggerFuture.get(TriggerFuture.java:99)
	at com.googlecode.objectify.util.FutureHelper.quietGet(FutureHelper.java:26)
	at com.googlecode.objectify.cache.TransactionWrapper.commit(TransactionWrapper.java:44)

 

I would really like to know if this is a bug (I really hope so) or a limitiation of the Task Queue service.

[Michael Hermus]

As an expirement, I replaced my use of the Objectify transaction with the normal low level datastore API transaction; the problem appears to have gone away. There is really no need to use the Objecty wrapper, since there are no actual Entities involved. If the issue comes back up subsequent to this change, I will post an update.

[Jeff Schnitzer]

ConcurentModifcationException happens whenever you have transaction
collisions. You will often have transaction collisions if you modify
the same piece of data from multiple tasks.

There are two possible situations here:

1) "It happens" - sometimes transactions collide. The solution is
that you should always make transactions idempotent and you should
always retry them. If you're using Objectify4, doing an
ofy.transact(new Work() { ... }) will do the retries for you.

2) You may need to rethink the way data flows in your app. If you
have a bunch of tasks trying to modify the same piece of data, you're
going to get a lot of collisions and collisions slow things down.
Figure out how you can serialize the transactions or combine them.

Jeff

> --
> You received this message because you are subscribed to the Google Groups
> "Google App Engine" group.
> To view this discussion on the web visit
> https://groups.google.com/d/msg/google-appengine/-/oNr1a1W0shwJ.
>
> To post to this group, send email to google-a...@googlegroups.com.
> To unsubscribe from this group, send email to
> google-appengi...@googlegroups.com.
> For more options, visit this group at
> http://groups.google.com/group/google-appengine?hl=en.

[Geert Audenaert]

Hi Jef,

please read the topic.

--
Geert Audenaert

Next-generation communication at http://rogerthatmessenger.com

Email: ge...@mobicage.com
GSM: +32 474 20 15 41
Phone: +32 9 324 25 64
Fax: +32 9 324 25 65

MOBICAGE NV
Antwerpsesteenweg 19
9080 Lochristi
Belgium

Scan to connect with me on Rogerthat Messenger.

[Michael Hermus]

Jeff,

There are absolutely no entities being modified or created, so there is no possibility of data store contention (based on my current understanding), hence the topic. The only operations during the transaction are Task Queue inserts. As I said, there is really no reason to use Objectify under those circumstances so its not really a problem, just something to be aware of.

Regards,
Mike

[Jeff Schnitzer]

Ah, sorry that wasn't clear to me.

I'm not sure how the presence of Objectify in this chain could make
any difference. There's nothing special about the way Objectify
starts or commits a transaction. Was this behavior consistent?

Jeff

> --
> You received this message because you are subscribed to the Google Groups
> "Google App Engine" group.
> To view this discussion on the web visit

> https://groups.google.com/d/msg/google-appengine/-/YEVYcmRii_gJ.

[Geert Audenaert]

I experienced it once, and changed my code to solve the problem differently.

Did not have the luxury to investigate it further.

Geert.

[Dao Linh Viet]

[Dao Linh Viet]

> To post to this group, send email to google-appengine@googlegroups.com.

> To unsubscribe from this group, send email to

> google-appengine+unsubscribe@googlegroups.com.

[Dao Linh Viet]

[Michael Hermus]

Jeff,

I haven't seen the problem since I made the change, and it seemed consistent under certain circumstances before that point. However, I have relatively few data points, so I wouldn't call it conclusive. If I have time, I will switch the transaction back to Objectify and see if that causes the issue to crop up again.

Mike

[Michael Hermus]

I just saw the issue pop up again, so it appears that (as you correctly predicted) Objectify had nothing to do with the issue.

Sigh... I am not sure how to approach a resolution for this, because my app makes significant use of this pattern. Do I perhaps need to store at least one entity with each transaction in order to avoid this?

[David Hardwick]

We can say unequivocally that under high load, if you have a transaction around a task enqueue without any database calls, you will get the ConcurrentModificationException error.

This is how the Pipeline library is setup for FINALIZE_JOB.

If you need to implement this pattern, then I would create a non-sense database call to see if that will work...we didn't need to make a database call so we took it out of the transaction, but the code that did have database operations and task enqueues within the transaction did not have this error.