Understanding java.util.ConcurrentModificationException: too much contention on these datastore entities. please try again.

[Louise Elmose Hedegaard]

Hi,

In my app I have the following logic (pseudoCode):

List<NonSentMails> nonSentMails= getNonSentMails();//1) datastore read

List<NonSentMails> sendNow = new ArrayList<NonSentMails>(); 

for each nonSentMail in nonSentMails

   if (sendNow(nonSentMail)){

       sendNow.add(nonSentMail);

   }

for each nonSentMail in sendNow

    SentStatus sentStatus = sendMailViaRestApi();

    create maillog in datastore with sent status//2) datastore write - new entry

    nonSentMail.setSentStatus(sentStatus);

update(sendNow) //3) datastore write

When comitting the above I get:

java.util.ConcurrentModificationException: too much contention on these datastore entities. please try again.

This is a problem, as the operation is not idempotent - when I send a mail via a REST API the mail is sent immediately, and I want the status of my maillog and NonSentMail objects to reflect this.

I do not understand why this error occurs. 

There are only three operations on the datastore:

1) The initial read of non sent mails

2) The creation of new mail log entries

3) The update of the status of non sent mails

2) should never lead to ConcurrentModificationException as far as I understand it. 1) and 3) operates on the same entities, but 1) only reads, so I cannot see any problem here either.

Furthermore there is currently only one user of my app, so there should be no other threads accessing the data.

Note that the elements in sendNow might have the same parents, but again I do not believe this should be a problem?

Can you please help me understand why the above code can lead to a ConcurrentModificationException?

Thanks,

-Lull

[Adam (Cloud Platform Support)]

A great article that explains this is 'Timeouts due to write contention': 

Writes to a single entity group are serialized by the App Engine datastore, and thus there's a limit on how quickly you can update one entity group. In general, this works out to somewhere between 1 and 5 updates per second.

Writing to the same entity twice successively is enough to cause a ConcurrentModificationException. Strategies to avoid this are discussed in 'Avoiding datastore contention'.  

[Jeff Schnitzer]

That does not sound correct.

If you have a linear order of operations in the same transaction (or entirely without transactions), you should never see concurrent modification exception. Timeouts are a different matter.

To the OP: Where are you transaction boundaries? Are you accidentally starting a new transaction “inside” an outer transaction and modifying the same entity as the outer transaction? That will guarantee CME every time.

Jeff

--
You received this message because you are subscribed to the Google Groups "Google App Engine" group.
To unsubscribe from this group and stop receiving emails from it, send an email to google-appengine+unsubscribe@googlegroups.com.
To post to this group, send email to google-appengine@googlegroups.com.
Visit this group at https://groups.google.com/group/google-appengine.

To view this discussion on the web visit https://groups.google.com/d/msgid/google-appengine/aec47937-549f-4de2-899d-b59da5163ad8%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

[Louise Elmose Hedegaard]

Hi Jeff,

What do you mean by "linear order of operations"?

I do not get any timeouts.

I begin the transaction before the pseudo code above, and attempt to commit the transaction after the pseudo code - it is the commit that causes the ConcurrentModification operation.

I am quite sure I do not start any nested transactions, as my logic only begins/commits/rollback transactions when a servlet is invoked.

I did try starting a nested transaction for the part of the pseudo code that sends mail to the external system and updates statusses accordingly, but that did not work either, so I went back to the original code with only one transaction.

Thanks,

-Louise

To unsubscribe from this group and stop receiving emails from it, send an email to google-appengi...@googlegroups.com.
To post to this group, send email to google-a...@googlegroups.com.

Visit this group at https://groups.google.com/group/google-appengine.

[Louise Elmose Hedegaard]

Hi Adam,

Why do you mention timeout - ConcurrentModificationException is not related to timeouts is it?

Thanks,

-Louise

[Jeff Schnitzer]

When you load a key in a datastore transaction, that EG is enlisted in the transaction. Any change to that EG by any other process in the system will cause your commit to rollback with CME. Even if your transaction is "read-only”.

When I said “linear” I mean that if you have a quiet datastore (no other activity) and you start a transaction and you perform a series of operations and you commit the transaction, you should not see CME. 

If you see CME erratically, then you have contention in your system.

If you see CME consistently 100% of the time and you aren’t under heavy load, then you have a bug in your transaction logic.

Jeff

On Tue, Mar 28, 2017 at 6:29 AM, Louise Elmose Hedegaard <louise...@gmail.com> wrote:

Hi,

I stumbled upon an instance in the log where the sendNow list is empty - but still the operation fails with ConcurrentModificationException.

How is that possible - it is only the initial read 1) operation which is executed on the datastore.

Thanks,

-Louise

--
You received this message because you are subscribed to the Google Groups "Google App Engine" group.

To unsubscribe from this group and stop receiving emails from it, send an email to google-appengine+unsubscribe@googlegroups.com.
To post to this group, send email to google-appengine@googlegroups.com.

Visit this group at https://groups.google.com/group/google-appengine.

To view this discussion on the web visit https://groups.google.com/d/msgid/google-appengine/02ac6e5d-e70d-4da8-9f51-2ac68aefc3d7%40googlegroups.com.

[Adam (Cloud Platform Support)]

Yes, actually you can ignore my post as I misread your original issue.

[Louise Elmose Hedegaard]

Hi Jeff,

I have CME erratically - so I guess you are saying I should take a look at the look for the CME runs, and check whether the same EG's are accessed by other operations?

It does seem that there might be a clash with other operations when the CME occurs.

I am not sure about the timing though - there is a updateOrder operation which fails with CME at 14:59:46.684 and the send mail fails with CME at 15:01:06.583...
When you say "Any change to that EG" it is the entire EG you mean, or only rows with the same id in the EG?

I need to ensure that my transaction does not fail, as it will result in duplicate emails being sent if it does.

There are other operations working on the same EG which comes randomly as they are caused by a webhook in another application.

What is the best way to ensure that these webhook operations do not cause the send mail operation to get a CME?

I can only think of very hacked and ugly solutions - e.g. to have a lock/switch which is on when the send mail operation is running, on only executing webhook operations when the lock/switch is off.

Thanks,

-Louise

To unsubscribe from this group and stop receiving emails from it, send an email to google-appengi...@googlegroups.com.
To post to this group, send email to google-a...@googlegroups.com.

Visit this group at https://groups.google.com/group/google-appengine.

[Jeff Schnitzer]

There may be clock skew in the cluster; 15s is a lot but you can’t assume that log entry timestamps are exact.

You should send email by enqueueing a transactional task. Do not put non-idempotent remote calls in your transactions (ideally, don’t put any remote calls in transactions). 

For example, let’s say you want to email a receipt on purchase. Create a deferred SendReceiptEmailTask that holds the purchase record key; it looks up the record, formulates the email body, and makes the final call to the email service. You can enlist this in your transaction that creates the purchase record. This also has the benefit of moving all that work outside of your transaction and shortening the critical section.

There is a very small risk that tasks may execute twice, but it’s small enough and harmless enough to ignore. I’ve sent millions of emails this way with no complaints. And you don’t really have a choice anyway - I haven’t found any email sending companies that allow you to specify an idempotency key or some other way of guaranteeing once-and-only-once behavior.

Cheers,

Jeff

To unsubscribe from this group and stop receiving emails from it, send an email to google-appengine+unsubscribe@googlegroups.com.
To post to this group, send email to google-appengine@googlegroups.com.

Visit this group at https://groups.google.com/group/google-appengine.

To view this discussion on the web visit https://groups.google.com/d/msgid/google-appengine/ea889f39-59c1-4493-ab7b-f4db8d25b2a8%40googlegroups.com.

[Louise Elmose Hedegaard]

Hi Jeff,

I am not sure I understand how putting the code in a transactional task helps me.

The simplified code is:

Mail mailToSend = getMail();

SentStatus sentStatus = sendMailToExternalMailSystem(mailToSend);

updateMailStatusInDatastore(mailToSend, sentStatus)

The problem is that the datastore write "updateMailStatusInDatastore" might fail due to CME, and at this point the mail has already been sent to the external mail system.

So even though I move this code to a transactional task, a failure in the datastore write will still lead me to a scenario where the mail has been sent, but the mail status has not been updated properly in my app.

Thanks,

-Louise

[Jeff Schnitzer]

You want:

Mail mailToSend = getMail();

runInTransaction(() -> {

   enqueueSendMailTask(mailToSend);

   updateMailStatusInDatastore(mailToSend, SENT);

});

Email is fundamentally asynchronous; you don’t get a “real” response when sending it. What you’re interpreting as a status code is just the fact that some queueing system somewhere accepted it. Just put it in your task queue and (as with all queues) make sure you don’t end up with a bunch of broken tasks.

Emails go from queue to queue to queue to queue. Adding one more queue is not going to hurt anything and the transaction will make your application a lot more reliable.

Jeff

To unsubscribe from this group and stop receiving emails from it, send an email to google-appengine+unsubscribe@googlegroups.com.
To post to this group, send email to google-appengine@googlegroups.com.

Visit this group at https://groups.google.com/group/google-appengine.

To view this discussion on the web visit https://groups.google.com/d/msgid/google-appengine/b546a386-0365-4749-9f64-faee69adbe72%40googlegroups.com.