Error to create a domain application - operating on too many entity groups in a single transaction

[nobu.tamba]

Hi,

We face the following error while creating landrush domain applications for the same domain name more than 20 times.

  Error: "too many entity groups in a single transaction",  triggered in Datastore API.

Findings:

- The error may be related to the following limitation

   - All the data accessed by a transaction must be contained in at most 25 entity groups.

- The domain application entity in DomainBase is created as a root entity.

- The domain application index holds the domain application IDs as a reference.

- The error is fired while saveCommitLog after processing the domain <create> command

- In the saveCommitLog, the following step is the last step in the stack trace

     CommitLogBucket bucket = loadBucket(info.bucketKey);

  * info is google.registry.model.ofy.TransactionInfo which provides the information 

    about save/delete operations performed in transaction. 

    bucketKey is the key for Bucket shard to under which commit log will be stored, chosen at random (in production).

 

- The following method loads domain applications for validation to prohibit creating a landrush application in

 LANDRUSH (but not in SUNRUSH) if there is exactly one sunrise application for the same name.

     google.registry.flows.domain.DomainApplicationCreateFlow#verifyDomainCreateIsAllowed

Questions:

Q1. Does the domain registry service touch the domain application entities referenced in domain application index? 

Q2. Does it count reading the domain applications as a entity group in the same transaction?

Q3. What does the other thing cause the 'too many entity groups' error potentially?

Full Stack Trace

google.registry.flows.EppController handleEppCommand: Unexpected failure java.lang.IllegalArgumentException: operating on too many entity groups in a single transaction.

   at com.google.appengine.api.datastore.DatastoreApiHelper.translateError(DatastoreApiHelper.java:54)

   at com.google.appengine.api.datastore.DatastoreApiHelper$1.convertException(DatastoreApiHelper.java:129)

   at com.google.appengine.api.utils.FutureWrapper.get(FutureWrapper.java:97)

   at com.google.appengine.api.datastore.AsyncDatastoreServiceImpl$5.getFutureWithOptionalTimeout(AsyncDatastoreServiceImpl.java:292)

   at com.google.appengine.api.datastore.AsyncDatastoreServiceImpl$5.aggregate(AsyncDatastoreServiceImpl.java:264)

   at com.google.appengine.api.datastore.AsyncDatastoreServiceImpl$5.get(AsyncDatastoreServiceImpl.java:235)

   at com.google.appengine.api.datastore.AsyncDatastoreServiceImpl$5.get(AsyncDatastoreServiceImpl.java:224)

   at com.google.appengine.api.datastore.FutureHelper$TxnAwareFuture.get(FutureHelper.java:182)

   at com.google.appengine.api.utils.FutureWrapper.get(FutureWrapper.java:89)

   at com.google.appengine.api.utils.FutureWrapper.get(FutureWrapper.java:89)

   at com.googlecode.objectify.impl.ResultAdapter.now(ResultAdapter.java:34)

   at com.googlecode.objectify.impl.Round$2.now(Round.java:135)

   at com.googlecode.objectify.impl.Round$2.now(Round.java:132)

   at com.googlecode.objectify.impl.LoadEngine$1.nowUncached(LoadEngine.java:172)

   at com.googlecode.objectify.impl.LoadEngine$1.nowUncached(LoadEngine.java:164)

   at com.googlecode.objectify.util.ResultCache.now(ResultCache.java:30)

   at com.googlecode.objectify.impl.Round$1.nowUncached(Round.java:73)

   at com.googlecode.objectify.util.ResultCache.now(ResultCache.java:30)

   at com.googlecode.objectify.LoadResult.now(LoadResult.java:25)

   at google.registry.model.ofy.CommitLogBucket.loadBucket(CommitLogBucket.java:128)

   at google.registry.model.ofy.CommitLoggedWork.saveCommitLog(CommitLoggedWork.java:134)

   at google.registry.model.ofy.CommitLoggedWork.vrun(CommitLoggedWork.java:121)

   at com.googlecode.objectify.VoidWork.run(VoidWork.java:14)

   at com.googlecode.objectify.VoidWork.run(VoidWork.java:11)

   at com.googlecode.objectify.impl.TransactorNo.transactOnce(TransactorNo.java:118)

   at com.googlecode.objectify.impl.TransactorNo.transactNew(TransactorNo.java:95)

   at com.googlecode.objectify.impl.ObjectifyImpl.transactNew(ObjectifyImpl.java:193)

   at com.googlecode.objectify.impl.ObjectifyImpl.transactNew(ObjectifyImpl.java:185)

   at google.registry.model.ofy.Ofy.transactCommitLoggedWork(Ofy.java:225)

   at google.registry.model.ofy.Ofy.transactNew(Ofy.java:210)

   at google.registry.model.ofy.Ofy.transact(Ofy.java:203)

   at google.registry.flows.FlowRunner.run(FlowRunner.java:101)

   at google.registry.flows.EppController.handleEppCommand(EppController.java:82)

   at google.registry.flows.EppServletUtils.handleEppCommandAndWriteResponse(EppServletUtils.java:49)

   at google.registry.flows.EppTlsServlet.doPost(EppTlsServlet.java:80)

   at javax.servlet.http.HttpServlet.service(HttpServlet.java:637)

   at javax.servlet.http.HttpServlet.service(HttpServlet.java:717)

   at org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:511)

   at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1166)

   at google.registry.flows.EppOauthFilter.doFilter(EppOauthFilter.java:62)

   at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1157)

   at google.registry.model.ofy.OfyFilter.doFilter(OfyFilter.java:33)

   at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1157)

   at com.googlecode.objectify.cache.AsyncCacheFilter.doFilter(AsyncCacheFilter.java:59)

   at com.googlecode.objectify.ObjectifyFilter.doFilter(ObjectifyFilter.java:49)

   at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1157)

   at com.google.apphosting.utils.servlet.ParseBlobUploadFilter.doFilter(ParseBlobUploadFilter.java:125)

   at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1157)

   at com.google.apphosting.runtime.jetty.SaveSessionFilter.doFilter(SaveSessionFilter.java:37)

   at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1157)

   at com.google.apphosting.utils.servlet.JdbcMySqlConnectionCleanupFilter.doFilter(JdbcMySqlConnectionCleanupFilter.java:60)

   at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1157)

   at com.google.apphosting.utils.servlet.TransactionCleanupFilter.doFilter(TransactionCleanupFilter.java:50)

   at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1157)

   at org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:388)

   at org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)

   at org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182)

   at org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:765)

   at org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:418)

   at com.google.apphosting.runtime.jetty.AppVersionHandlerMap.handle(AppVersionHandlerMap.java:260)

   at org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)

   at org.mortbay.jetty.Server.handle(Server.java:326)

   at org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542)

   at org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:923)

   at com.google.apphosting.runtime.jetty.RpcRequestParser.parseAvailable(RpcRequestParser.java:78)

   at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404)

   at com.google.apphosting.runtime.jetty.JettyServletEngineAdapter.serviceRequest(JettyServletEngineAdapter.java:148)

   at com.google.apphosting.runtime.JavaRuntime$RequestRunnable.run(JavaRuntime.java:469)

   at com.google.tracing.TraceContext$TraceContextRunnable.runInContext(TraceContext.java:439)

   at com.google.tracing.TraceContext$TraceContextRunnable$1.run(TraceContext.java:446)

   at com.google.tracing.CurrentContext.runInContext(CurrentContext.java:256)

   at com.google.tracing.TraceContext$AbstractTraceContextCallback.runInInheritedContextNoUnref(TraceContext.java:310)

   at com.google.tracing.TraceContext$AbstractTraceContextCallback.runInInheritedContext(TraceContext.java:302)

   at com.google.tracing.TraceContext$TraceContextRunnable.run(TraceContext.java:443)

   at com.google.apphosting.runtime.ThreadGroupPool$PoolEntry.run(ThreadGroupPool.java:235)

   at java.lang.Thread.run(Thread.java:745)

Thanks,

Nobu

[Nick Felt]

Interesting, I think this is a real issue that we didn't anticipate.  I'm guessing that to fix this we'll have to somehow store the data than answers this query ("how many active sunrise applications are there for this domain name") directly on the DomainApplicationIndex.  It's a bit tricky since we'd need to maintain that even if applications are deleted (or possibly if they're rejected? not sure).  But it should be possible, by just keeping a separate set of DomainApplication keys that we add/remove elements from as appropriate.

Some further comments inline.

On Wed, Jul 6, 2016 at 3:54 PM nobu.tamba <nobu....@dev9.com> wrote:

Hi,

We face the following error while creating landrush domain applications for the same domain name more than 20 times.

  Error: "too many entity groups in a single transaction",  triggered in Datastore API.

Findings:

- The error may be related to the following limitation

   - All the data accessed by a transaction must be contained in at most 25 entity groups.

- The domain application entity in DomainBase is created as a root entity.

- The domain application index holds the domain application IDs as a reference.

- The error is fired while saveCommitLog after processing the domain <create> command

- In the saveCommitLog, the following step is the last step in the stack trace

     CommitLogBucket bucket = loadBucket(info.bucketKey);

  * info is google.registry.model.ofy.TransactionInfo which provides the information 

    about save/delete operations performed in transaction. 

    bucketKey is the key for Bucket shard to under which commit log will be stored, chosen at random (in production).

I think the fact that it's coming from saveCommitLog() is mostly coincidence - when saving a commit log, we enroll one additional entity group in the transaction (the one for the commit log bucket that the commit log gets saved under).  It's probably just the case that the prior part of the transaction had touched exactly 25 entity groups, and that the commit log bucket was the attempted 26th entity group.

This would roughly match having about 20 existing applications plus a few contacts and hosts = 25 or so.

 

 

- The following method loads domain applications for validation to prohibit creating a landrush application in

 LANDRUSH (but not in SUNRUSH) if there is exactly one sunrise application for the same name.

     google.registry.flows.domain.DomainApplicationCreateFlow#verifyDomainCreateIsAllowed

Questions:

Q1. Does the domain registry service touch the domain application entities referenced in domain application index? 

 

It reads them, yes.  It happens in DomainApplicationIndex.loadActiveApplicationsByDomainName() which is invoked from that section of DomainApplicationCreateFlow:

https://github.com/google/domain-registry/blob/c458c058017e3cf8a4ee72a2ef87ee321a3310e3/java/google/registry/model/index/DomainApplicationIndex.java#L92

Q2. Does it count reading the domain applications as a entity group in the same transaction?

Yes, by default everything in a mutating flow happens within a single transaction.

Q3. What does the other thing cause the 'too many entity groups' error potentially?

I think you correctly identified the issue; I don't think there is anything else that triggers this error.  It's thrown only when you attempt to read or write more than 25 EGs within a single transaction.

--
NOTE: This is a public mailing list for users of the domain-registry project.
---
You received this message because you are subscribed to the Google Groups "domain-registry-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to domain-registry-...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/domain-registry-users/a14d853d-5993-406c-8945-2dfa0fcb19f2%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[hans.ridder]

So should we file an issue on github? I think it would take a deeper understanding for us to come up with a PR for it, and while it's important, I believe we have higher priorities at the moment.

-h

Some further comments inline.

To unsubscribe from this group and stop receiving emails from it, send an email to domain-registry-users+unsub...@googlegroups.com.

[Nick Felt]

Yeah, let's file a github issue - I already filed an internal issue just for tracking, but it'd be good to have one that you can see as well.

The fix will be a little complicated since it would require a data migration to store new data on DomainApplicationIndex to support this use case, as well as changes to all the flows that touch DomainApplicationIndex.  We've done stuff like this before and could probably take this on since it's a core registry issue, but if it's not urgent for you I'm guessing we wouldn't get to it for a while.

To unsubscribe from this group and stop receiving emails from it, send an email to domain-registry-...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/domain-registry-users/a14d853d-5993-406c-8945-2dfa0fcb19f2%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

NOTE: This is a public mailing list for users of the domain-registry project.
---
You received this message because you are subscribed to the Google Groups "domain-registry-users" group.

To unsubscribe from this group and stop receiving emails from it, send an email to domain-registry-...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/domain-registry-users/760403c5-8885-45df-93cf-1e94110fb219%40googlegroups.com.

[Chris Cowherd]

From our point of view, this is a low priority as within our timeframe, we don't anticipate going through any Sunrise phases.  Also in practice, we don't see very many applications for the same trademarked term.  The issue should be noted though for the next gTLD round.

Perhaps at a minimum add a comment with a reference to the issue in GitHub for others that may review the code.

To unsubscribe from this group and stop receiving emails from it, send an email to domain-registry-users+unsub...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/domain-registry-users/a14d853d-5993-406c-8945-2dfa0fcb19f2%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
NOTE: This is a public mailing list for users of the domain-registry project.
---
You received this message because you are subscribed to the Google Groups "domain-registry-users" group.

To unsubscribe from this group and stop receiving emails from it, send an email to domain-registry-users+unsub...@googlegroups.com.

[Nick Felt]

Sounds good to me.  Nobu or Hans, want to file an issue?

To unsubscribe from this group and stop receiving emails from it, send an email to domain-registry-...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/domain-registry-users/a14d853d-5993-406c-8945-2dfa0fcb19f2%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
NOTE: This is a public mailing list for users of the domain-registry project.
---
You received this message because you are subscribed to the Google Groups "domain-registry-users" group.

To unsubscribe from this group and stop receiving emails from it, send an email to domain-registry-...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/domain-registry-users/760403c5-8885-45df-93cf-1e94110fb219%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

NOTE: This is a public mailing list for users of the domain-registry project.
---
You received this message because you are subscribed to the Google Groups "domain-registry-users" group.

To unsubscribe from this group and stop receiving emails from it, send an email to domain-registry-...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/domain-registry-users/627b070e-7680-41ae-99d2-577422bd1fa1%40googlegroups.com.