Queries in Transactions, Ancestor Queries

[Jake]

Hey all,

I'm a bit confused about transactions and queries with Objectify and
would love some advice/clarification.

I had originally done this:

Objectify ofy = ObjectifyService.beginTransaction();
Prompt prompt = null;

try {

Query<Prompt> q = ofy.query(Prompt.class);
/* filters for 'book', 'type' */
prompt = q.get();

// If no prompt exists for this book, create one.
if (prompt == null) {
prompt = new Prompt(book, type);
ofy.put(prompt);
}

ofy.getTxn().commit();

} finally {
/* rollbacks */
}

However, that was giving me an error (probably well known) "Only
Ancestor Queries are allowed inside of transactions." I presume this
means I will never be able to do a root entity query inside a
transaction? I could use some clarification on this.

In the meantime, I found deep in a python forum the comment "Do your
query first, then do your error checking inside the transaction."

So, I've swapped to this:

Query<Prompt> q = ObjectifyService.begin().query(Prompt.class);
/* filters for 'book', 'type' */
Prompt prompt = q.get();

Objectify ofy = ObjectifyService.beginTransaction();

try {
// If no prompt exists for this book, create one.
if (prompt == null) {
prompt = new Prompt(book, type);
ofy.put(prompt);
}

ofy.getTxn().commit();

} finally {
/* rollbacks */
}

So, is that really the proper way to do it? It looks weird to me
since I'm creating two different Objectify objects, one with a
transaction and one without. Is there any way to take a non-
transactional Objectify object, do some queries, and then say, "I
would like to start a transaction, now." Also, doesn't this defeat
the entire point of the transaction? I'm trying to avoid creating
Prompt objects with the same properties and it seems like it could
happen in this new scenario. As far as ancestors go, I'm not using
any relationships - just root objects and id references throughout.

Thanks in advance for the help!

Jake

[Scott Hernandez]

You don't need a transaction. 

Transactions are primarily useful when you are changing more than one entity, that have the same common root entity as a ancestor.

--
To unsubscribe, reply using "remove me" as the subject.

[Jake]

Hello,

Ok - since I don't have that scenario, it looks as if I would never
need transactions. I confess, I'm not a huge datastore expert (GAE,
SQL, otherwise), but what prevents this from happening:

User 1 triggers the above function. Query indicates that no <Prompt>
exists.
User 2 triggers the above function at the same time as User 1. Query
indicates that no <Prompt> exists.
User 1 creates a <Prompt> with certain features - Objectify assigns a
unique Long id.
User 2 creates a <Prompt> with same features - Objectify assigns a
unique Long id.

Two <Prompt> objects, when I only want one that is global to the
system - it should simply be created by the first person who
encounters it.

Thanks!

Jake

On Apr 2, 4:20 pm, Scott Hernandez <scotthernan...@gmail.com> wrote:
> You don't need a transaction.
>
> Transactions are primarily useful when you are changing more than one
> entity, that have the same common root entity as a ancestor.
>

[Scott Hernandez]

Please read this: http://code.google.com/p/objectify-appengine/wiki/Concepts#Transactions

Nothing prevents that from happening; there is no unique constraint in the datastore. If you want that you will have to have a single object used to do locking, you would have write that yourself. Most likely what you want is to use memcache to create a lock and then check that before persisting your entity.

You should search the list for Version(ed). There is some sample code for this kind of thing.

[Jake]

Hello,

Thanks! I had read that document, but I didn't see how it applied. I
guess the point is that the term "transaction" as it relates to GAE
doesn't imply concepts related to locking - just about establishing
some consistency about where data is read from (?).

Jake

On Apr 2, 4:40 pm, Scott Hernandez <scotthernan...@gmail.com> wrote:
> Please read this:http://code.google.com/p/objectify-appengine/wiki/Concepts#Transactions
>
> <http://code.google.com/p/objectify-appengine/wiki/Concepts#Transactions>Nothing
> prevents that from happening; there is no unique constraint in the
> datastore. If you want that you will have to have a single object used to do
> locking, you would have write that yourself. Most likely what you want is to
> use memcache to create a lock and then check that before persisting your
> entity.
>
> You should search the list for Version(ed). There is some sample code for
> this kind of thing.
>

[Scott Hernandez]

Actually, the opposite. Transactions are more about writing a group of entities, than reads. 

All reads (except those in the same entity group, in the current transaction) are done outside a transaction. That is why you can only do an ancestor based query in the transaction; because it is guaranteed to be in the same entity group.

The take-away here is that you probably don't need to use transactions. You should avoid them for many reasons.

[Jeff Schnitzer]

I don't wholly agree with Scott here. Transactions are for single
entities, but you have to use them carefully.

Jake, you basically understand the problem. How do you prevent there
from being a Prompt object before you create it?

The basic idea is sound:

* Start a transaction
* Try to load the entity
* If not found, put() the entity
* Commit the transaction
* If the transaction committed, all is good

The problem is not that you can't run a transactional query on root
entities. You can - just make up a "fake" parent key and use it as
the parent. The key doesn't have to point at a real entity.

The problem is that this still won't give you a uniqueness constraint.

The only way to enforce a uniqueness constraint in the datastore is
with a single-entity transaction on an entity with a natural primary
key. There is some discussion of that here:

http://groups.google.com/group/google-appengine-java/browse_thread/thread/998289f853f5a688/21baa42132388497

The "easy way" to make this work is if you make the PK of your Prompt
class a natural key, possibly a String combination of the components
that go about making it unique. This can be ugly and messy. Another
alternative is to make an extra class with this String PK that does
nothing but enforce uniqueness of your Prompt classes.

Does this make sense? I can elaborate.

Jeff

[Dan]

Not wanting to get into the theory of transactions because theory
hurts my head, but maybe this would work:

query prompt
if count = 0
create prompt
query prompt
if count > 1
delete prompt (the one you created)
end
end

[Jake]

Hey all,

Jeff, your comments make sense, but I just want to confirm since I
skipped the Database course in undergrad :) A natural key is, instead
of an auto-generated Long, a string that perhaps represents this
entity in a human-readable form (String key = "book-4-type-8"). That
way, even if two functions try to create the same Prompt object,
they'll have the same key and they'll just overwrite. That's fine, I
suppose, but - despite the severe warnings of the Objectify crowd - I
am trying to keep some consistency in my code since a few of these
components will be used outside of App Engine. In such a case, I'm
really hoping to keep auto-generated Long ids in play.

I believe Dan's method of deleting a recently created object could
potentially work, but it sounds inefficient. Granted, each of these
objects will be created once-and-only-once so perhaps the extra
overhead isn't so bad. However, many of these objects will be created
throughout the life of the program - potentially several dozen in a
single page request. The question is whether that's better than doing
some sort of synchronized key pool that grants the ability to write to
the datastore.

My last question - I was using "transactions" with JDO. Was that
enforcing uniqueness in a similar implementation? Did I lose
something, in this case, by switching to Objectify? The speed will
make up for it, so I'm happy, but I just want to be sure.

Thanks!

Jake

[Jeff Schnitzer]

Comments inline:

On Mon, Apr 5, 2010 at 5:52 AM, Jake <jbroo...@cast.org> wrote:
> Hey all,
>
> Jeff, your comments make sense, but I just want to confirm since I
> skipped the Database course in undergrad :)  A natural key is, instead
> of an auto-generated Long, a string that perhaps represents this
> entity in a human-readable form (String key = "book-4-type-8").  That
> way, even if two functions try to create the same Prompt object,
> they'll have the same key and they'll just overwrite.  That's fine, I
> suppose, but - despite the severe warnings of the Objectify crowd - I
> am trying to keep some consistency in my code since a few of these
> components will be used outside of App Engine.  In such a case, I'm
> really hoping to keep auto-generated Long ids in play.

You basically have the idea - natural keys are keys that exist
"naturally" in the world and are not autogenerated. They can be
String (example, email) or they can be numeric (example, Facebook ID).

Every RDBMS in the world supports the use of natural keys. Most
object stores do as well. If you decide to switch away from GAE,
you'll find this aspect translates just fine.

> I believe Dan's method of deleting a recently created object could
> potentially work, but it sounds inefficient.  Granted, each of these
> objects will be created once-and-only-once so perhaps the extra
> overhead isn't so bad.  However, many of these objects will be created
> throughout the life of the program - potentially several dozen in a
> single page request.  The question is whether that's better than doing
> some sort of synchronized key pool that grants the ability to write to
> the datastore.

You must be somewhat careful using Dan's method because the datastore
could fail during the delete, leaving corrupt data in your DB. It
happens pretty rarely, but sometimes the datastore goes haywire and a
high percentage of calls will fail. It hasn't happened in a major way
since they rolled out the automatic retries, but it *will* happen
again at some point. When that happens, how do you scavenge the
duplicate records?

Maybe by writing the record twice, adding a "committed" field.

Given that you might be querying on these fields and therefore the bad
entity written to the DB could be a big problem, I still think you're
better off making the exclusion check before you create your entity.
Here's the least invasive way to do it:

Create an entity with nothing but String id like this:

class PromptExclusion {
@Id String unique; // will be "bookid-typeid"
}

Now, when you create a Prompt object, use a PromptExclusion to lock
the values first:

* start a transaction
* get the appropriate PromptExclusion
* if exists, throw error
* create new PromptExclusion with the unique bookid-typeid
* commit transaction
* if commit succeeds, you're ok

This code is actually quite portable because if you switch to an RDBMS
(or any other datastore that has builtin uniqueness constraints), all
you need to do is delete all this code. Put it in a self-contained
enforceUniqueness(long bookId, long typeId) method and then turn it
into a no-op.

> My last question - I was using "transactions" with JDO.  Was that
> enforcing uniqueness in a similar implementation?  Did I lose
> something, in this case, by switching to Objectify?  The speed will
> make up for it, so I'm happy, but I just want to be sure.

Nope, JDO wasn't helping you. JDO transactions and Objectify
transactions are the same low-level datastore transactions with all
the same limitations.

There really is only one good way to enforce uniqueness in the GAE
datastore, and that's with single-entity transactions on an entity
with a natural primary key. It's an unfortunate limitation.

Jeff