Entity groups

[Louise Elmose Hedegaard]

Hi,

I have the following datastore entity setup:

- Shop is the root element

- Shop is parent of Settings and Product

- Product is parent of Variant, Option and Image

According to the definition of entity groups "A root entity and all of its descendants belong to the same entity group" I assume I have one entity group, as the Shop entity has all the other entities as descendants.

But if that is correct, why do I get this error:

operating on too many entity groups in a single transaction.

The documentation states that "All the data accessed by a transaction must be contained in at most 25 entity groups", so with only one entity group there should be no problem.

Or is this restriction related to the number of rows in the tables? So if a query on e.g. Product returns more than 25 rows, then the error occurs?

Maybe my problem is related to the fact that I use cross transactions (withXG(true)) except in Shop queries, as you cannot create root queries on transactions.

(but again - why should it be necessary to use cross transactions, when I only have one entity group??)

Any input would be appreciated.

Thanks,

-Louise

[Alex (Cloud Platform Support)]

Hi Louise,

You’re right, as stated per the Transactions and Entity Groups documentation, there is a 25 entity groups limit for a single data transaction. However, according to your description this error should not happen and there should be no need of using Cross-Group (XG) Transactions either. Still, it is possible that this error is triggered by something else as the way your entities were defined.

Can you provide a Minimal Working Example of your code including the transaction and a snippet of how you defined your entities? In fact, this would help to better visualize the root cause of your error and provide you with answer on how to solve it.

Regards,

Alex

[Louise Elmose Hedegaard]

Hi Alex,

Thank you for answering.

Here's how I create the entities:

Entity shopEntity = new Entity(ShopDBFields.SHOP_TABLE_NAME);
shopEntity.setProperty(ShopDBFields.CREATED, new Date());

....
datastore.put(shopEntity);

Entity productEntity = new Entity(ProductDBFields.PRODUCT_TABLE_NAME, KeyFactory.createKey(ShopDBFields.SHOP_TABLE_NAME, shopId));
productEntity.setProperty(ProductDBFields.TITLE, product.getTitle());
...
datastore.put(transaction, productEntity);
(same principle for Settings)


Entity variantEntity = new Entity(VariantDBFields.VARIANT_TABLE_NAME, KeyFactory.createKey(ProductDBFields.PRODUCT_TABLE_NAME, productId));
variantEntity.setProperty(VariantDBFields.TITLE, variant.getTitle());
...
datastore.put(transaction, variantEntity);
(same principle for Option and Image)

The flow which eventually leads to the exception (the exact code is different, but simplified below):

DatastoreService datastore = DatastoreServiceFactory.getDatastoreService();
TransactionOptions options = TransactionOptions.Builder.withXG(true);
Transaction transaction = datastore.beginTransaction(options);

try {
     Entity entity = datastore.get(KeyFactory.createKey(ShopDBFields.SHOP_TABLE_NAME, id));
     Shop shop = createShop(entity);
     <get list of products from web shop API - for each product:>
          Entity productEntity = new Entity(ProductDBFields.PRODUCT_TABLE_NAME, KeyFactory.createKey(ShopDBFields.SHOP_TABLE_NAME, shopId));
          productEntity.setProperty(ProductDBFields.TITLE, product.getTitle());
          ...
          datastore.put(transaction, productEntity);
         <for each variant for the given product:> 
               Entity variantEntity = new Entity(VariantDBFields.VARIANT_TABLE_NAME, KeyFactory.createKey(ProductDBFields.PRODUCT_TABLE_NAME, productId));
               variantEntity.setProperty(VariantDBFields.TITLE, variant.getTitle());
               ...
               datastore.put(transaction, variantEntity); <EXCEPTION - VariantDao.java:45>
    logger.info("Initiating commit of transaction");
    if (transaction.isActive()){
        transaction.commit();
        logger.info("Transaction comittet");
    }else{
        logger.warning("Transaction was not active - transaction was not comittet");
    }
}
finally {
    if (transaction.isActive()){
        transaction.rollback();
        logger.info("Transaction rolled back as transaction was still active");
    }else{
        logger.info("Transaction was not active - transaction was not rolledback");
    }
}


The stack trace:
/initializeProducts
java.lang.IllegalArgumentException: operating on too many entity groups in a single transaction.
	at com.google.appengine.api.datastore.DatastoreApiHelper.translateError(DatastoreApiHelper.java:50)
	at com.google.appengine.api.datastore.DatastoreApiHelper$1.convertException(DatastoreApiHelper.java:121)
	at com.google.appengine.api.utils.FutureWrapper.get(FutureWrapper.java:97)
	at com.google.appengine.api.datastore.Batcher$ReorderingMultiFuture.get(Batcher.java:131)
	at com.google.appengine.api.datastore.FutureHelper$TxnAwareFuture.get(FutureHelper.java:182)
	at com.google.appengine.api.utils.FutureWrapper.get(FutureWrapper.java:89)
	at com.google.appengine.api.datastore.FutureHelper.getInternal(FutureHelper.java:76)
	at com.google.appengine.api.datastore.FutureHelper.quietGet(FutureHelper.java:36)
	at com.google.appengine.api.datastore.DatastoreServiceImpl.put(DatastoreServiceImpl.java:61)
	at dk.elmose.apps.shopify.stocksniffer.dao.VariantDao.create(VariantDao.java:45)

[Louise Elmose Hedegaard]

My application obviously does not agree that I have only one entity group.

When I try to insert more than 25 products, I get the error "operating on too many entity groups in a single transaction.".

When I do not define a XG transaction, I get the error "java.lang.IllegalArgumentException: cross-group transaction need to be explicitly specified, see TransactionOptions.Builder.withXG".

If anybody can provide me with some ideas or hints as to why the above defintions of the entities constitute more than one entity group, I would appreciate it.

Thanks,

-Louise

[Louise Elmose Hedegaard]

Maybe I figured it out myself.

I assumed that definition of parent entity was transitive, i.e. if the parent of an image is a product and the parent of the product is a shop, then shop is ancestor of image, but that is not the case.

So instead of:

Entity variantEntity = new Entity(VariantDBFields.VARIANT_TABLE_NAME, KeyFactory.createKey(ProductDBFields.PRODUCT_TABLE_NAME, productId));
variantEntity.setProperty(VariantDBFields.TITLE, variant.getTitle());
...
datastore.put(transaction, variantEntity);

I should do:

Entity variantEntity = new Entity(VariantDBFields.VARIANT_TABLE_NAME, KeyFactory.createKey(KeyFactory.createKey(ShopDBFields.SHOP_TABLE_NAME, shopId), ProductDBFields.PRODUCT_TABLE_NAME, productId));

variantEntity.setProperty(VariantDBFields.TITLE, variant.getTitle());
...
datastore.put(transaction, variantEntity);

-Louise

[Alex (Cloud Platform Support)]

Thanks for posting your code sample and your solution. In fact you got it right, Datastore Entity ancestor paths [1] are designed to allow entities to optionally be assigned parents. Meaning that upon creation of a datastore entity, the ancestor path of this entity has to be specified explicitly in order to be part of the same entity group.

If possible, it would be great if you could let the community know whether your proposed solution worked for you.

Thanks,

Alex

[1] https://cloud.google.com/appengine/docs/java/datastore/entities#Java_Ancestor_paths