java.lang.IllegalArgumentException: operating on too many entity groups in a single transaction.

[Louise Elmose Hedegaard]

I have a datastore with a kind "shop", a kind "order" and a kind "transaction".

One shop has many orders, and one order has many transactions.

The parent of a transaction is an order, the parent of an order is a shop.

I want to delete all entries in the transactions table:

    public void cleanUp() {

        Query query = new Query(TransactionDBFields.TRANSACTION_TABLE_NAME);

        query.setKeysOnly();

        query.setFilter(

                new Query.FilterPredicate(

                        TransactionDBFields.CREATED_AT,

                        Query.FilterOperator.LESS_THAN_OR_EQUAL,

                        new Date())

        );

        PreparedQuery preparedQuery = datastore.prepare(query);

        Iterable<Entity> entities = preparedQuery.asIterable(FetchOptions.Builder.withLimit(5));

        TransactionOptions options = TransactionOptions.Builder.withXG(true);

        Transaction txn = datastore.beginTransaction(options);

        for (Entity en : entities) {

            datastore.delete(txn, en.getKey());

        }

        txn.commit();

    }

If I change the limit to anything larger than 5 I get the error:

java.lang.IllegalArgumentException: operating on too many entity groups in a single transaction.

- apparently as GAE does not allow operating on more than 5 entity groups in a single transaction. 

Is there a way to delete more than 5 entities at a time? executing the above code continuously to delete all transaction entities does not seem reasonable, nor when considering performance.

-Louise

[Chad Vincent]

I can think of three ways:

1) Iterate over the query results and delete each without the transaction.
2) Make a separate transaction for every group of 5 keys.
3) Use the MapReduce library and a MapOnlyMapper to walk all the keys.

[Chad Vincent]

Oh!  A fourth...

Walk over all the Orders instead of the transactions, and do an ancestor query to get all the transactions for each.  Then each should be a single entity group.

[Nick (Cloud Platform Support)]

According to the documentation on XG-Transactions, you should be able to operate on up to 25 entity groups in a cross-group transaction. It does mention that it will fail if another transaction running at the same time modifies any of those entity groups. This might be happening, although only you would know your system to be able to say definitively yes or no.

If that isn't happening, I think it would be worth posting this, along with simple code snippets allowing reproduction of the behaviour, in the Public Issue Tracker. We do regular monitoring of the Public Issue Trackers and the issue would get looked-at relatively quickly.

Best wishes,

Nick

[V.B.]

Hi Nick,

   Is that value (the 25 limit) available as an environment variable, i.e. to avoid hard coding it in the rare instance when the value is needed?

[Nick (Cloud Platform Support)]

Hey V.B,

For now it appears this will need to be simply included as a constant in your application. You can update the constant if in future you notice that the value has changed. These changes are announced in the release notes for the SDK.

Cheers,

Nick
Cloud Platform Community Support