Reading existing entity from DB

[Louise Elmose Hedegaard]

Hi,

I have an issue with too many entities being created in my GAE datastore. 

I have an unique id on an entity (which is not the key in this case), and there should thus only be one entity in the datastore per unique id.

When an entity is updated or created by the user, I check whether the entity already exists in the DB, and only create a new entity, if there is not already an existing entity with a matching id. If there is an existing entity, I update the parameters of the entity.

This logic should ensure that there is only one entry/entity per unique id, but somehow several entities with the same id are created in the DB - I do not understand how/why!

When I look at this page https://cloud.google.com/appengine/docs/java/datastore/entities#Java_Updating_an_entity is says "You can use a transaction to test whether an entity with a given key exists before creating one.".

Do I really need to use transactions to validate whether there is an existing entity with a given id, or does that only apply when you use the key as primary key??

Thanks,

-Louise

[Nick]

Yes.

Generating unique sequential ids is a hard problem in this kind of distributed system.

The cause of your issue is probably eventual consistency, but you probably will not be able to get strong consistency without crippling your datastore throughput.

If you really want sequential ids, I recommend using a sequence in cloudsql to generate them when you need one. This won't be perfect (some numbers may get skipped) but it won't create duplicates.

[timh]

Having a unique id that is only a property and not the key of the entity makes it hard.

Are you generating the ID - is it sequential, or just unique (ie an email address) if the later, you could reliably use a combination of transactions and a child entity with a key of the unique id, and the ancestors key).

You don't even have store any data.

This can be reliable as you are able to use get operations in a transaction.

Anything entity you fetch by a query (unless it's an ancestor query) will be liable to eventual consistancy issues.

T

[Louise Elmose Hedegaard]

Hi guys,

Thank you for your answers.

I do not think I have explained my problem properly though when I read your answers.

I have a table "order" where I save different information about an order. The unique id of the order comes from an external system, and I want to save this unique id for the order.

When an order is updated in the external system, I want to update the GAE datastore accordingly. 

For this reason I first check whether the order that was updated in the external system already exists in my datastore. If it already exists I update it, if it does not already exist, I create the order in the datastore.

My problem is that I use this:

datastore.prepare(query).asSingleEntity();

which fails, as there are more than one entity with the unique id.

I do not understand how several entities can be created with the same unique id, when my logic says:

Entity entity = getEntity(uniqueId);

if (entity == null){

  createEntity();

}else{

  updateEntity();

}

It is as if the getEntity check does not detect that there already is an entity with the unique id.

It does seem that this is caused by a timing/concurrency issue - the external system sends two updates approximately at the same time, but shouldn't the datastore be able to handle this?

My first attempt of getEntity was:

Query query = new Query(OrderDBFields.ORDER_TABLE_NAME).setFilter(new Query.FilterPredicate(OrderDBFields.ID, Query.FilterOperator.EQUAL, id));

        return datastore.prepare(query).asSingleEntity();

My second (confused attempt) is:

        Transaction txn = datastore.beginTransaction();

        Key orderKey = KeyFactory.createKey(OrderDBFields.ORDER_TABLE_NAME, id);

        Entity orderEntity = null;

        try {

            orderEntity = datastore.get(orderKey);

        } catch (EntityNotFoundException e) {

            orderEntity = null;

        }

        txn.commit();

        return orderEntity;

Any input is appreciated,

Thanks

-Louise

[Karl MacMillan]

On Jan 28, 2016, at 7:22 AM, Louise Elmose Hedegaard <louise...@gmail.com> wrote:

Hi guys,

Thank you for your answers.

I do not think I have explained my problem properly though when I read your answers.

I have a table "order" where I save different information about an order. The unique id of the order comes from an external system, and I want to save this unique id for the order.

When an order is updated in the external system, I want to update the GAE datastore accordingly. 

For this reason I first check whether the order that was updated in the external system already exists in my datastore. If it already exists I update it, if it does not already exist, I create the order in the datastore.

My problem is that I use this:

datastore.prepare(query).asSingleEntity();

which fails, as there are more than one entity with the unique id.

I do not understand how several entities can be created with the same unique id, when my logic says:

In order for this to work as you expect you need strong consistency, which typically means a) have your entities be a part of the same entity group and b) do the query inside of a transaction.

As others have pointed out it’s much, much easier to simply use the id as they key and fetch by key. Though you still need to do the logic below inside of a transaction.

Karl

--
You received this message because you are subscribed to the Google Groups "Google App Engine" group.
To unsubscribe from this group and stop receiving emails from it, send an email to google-appengi...@googlegroups.com.
To post to this group, send email to google-a...@googlegroups.com.
Visit this group at https://groups.google.com/group/google-appengine.
To view this discussion on the web visit https://groups.google.com/d/msgid/google-appengine/c774c62a-98d8-467d-b97a-c5e817644cc5%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Nick]

I think you've been given some good pointers in the right direction, I'll just add that you need to read and understand the eventual consistency, transactional and cross entity group (XG) transaction docs for the datastore.

My guess is based on your question you haven't wrapped your head around it, and as a result your data models and subsequently data are going to get broken. Fixing them after go live will be hard, so you need to get a lot closer now.

Once you've got the basic concepts down, it should become clearer how to model this, and why this is happening.

[Louise Elmose Hedegaard]

Ok, thank you for your input!

-Louise

Den onsdag den 27. januar 2016 kl. 12.39.54 UTC+1 skrev Louise Elmose Hedegaard: