are numeric ids of deleted entities reused?

[jeremy]

are numeric ids of deleted entities reused? i really hope not.

[ryan]

On Dec 20, 7:09 pm, jeremy <jeremy.a...@gmail.com> wrote:
> are numeric ids of deleted entities reused? i really hope not.

good question. no, they're not.

you're probably already on top of this, but just for the record, ids
themselves aren't unique across entities. only an entity's full path
is guaranteed to be unique. ids and key names are only guaranteed to
be unique among entities of the same kind that have the same parent
(or that are root entities). more in
http://code.google.com/appengine/docs/datastore/keysandentitygroups.html#Paths_and_Key_Uniqueness
.

[Thomas Johansson]

Hi ryan,

Is this to be interpreted as getting the following, in order, more or
less? (where the ID/Key is in square brackets)

/Image[42]
/Image[127841]

/Thread[10]/Post[42]
/Thread[10]/Post[127841]

E.g. the ID is ever increasing, but only for the current fully
qualified path? Can we ever experience inserting a record that will
have a lower ID than the previous?

Thanks,
Thomas

> (or that are root entities). more inhttp://code.google.com/appengine/docs/datastore/keysandentitygroups.h...
> .

[Thomas Johansson]

Forgot to ask; How does the key fit into the hierarchy, compared to
the key_name & id?

[Alexander Kojevnikov]

> Can we ever experience inserting a record that will
> have a lower ID than the previous?

You can, at least that's what I'm seeing in my app. As far as I
understand, the IDs are pre-allocated in batches, and if your app
actually runs on different boxes, they will use IDs from different
batches.

[Andy Freeman]

> Can we ever experience inserting a record that will
> have a lower ID than the previous?

Yes.

"An application should not rely on numeric IDs being assigned in
increasing order with the order of entity creation. This is generally
the case, but not guaranteed."

from http://code.google.com/appengine/docs/datastore/keysandentitygroups.html#Paths_and_Key_Uniqueness

> > .- Hide quoted text -
>
> - Show quoted text -

[Andy Freeman]

> good question. no, they're not.

> you're probably already on top of this, but just for the record, ids
> themselves aren't unique across entities. only an entity's full path
> is guaranteed to be unique. ids and key names are only guaranteed to
> be unique among entities of the same kind that have the same parent
> (or that are root entities).

That's a curious comment. It suggests that ids may be reused if
there's another difference in the path.

However, combining that with the statement that there's no reuse for a
given path is interesting because there's no obvious place for the
datastore to keep track of what ids it has used.

Yes, one can store an indicator of what ids have been used (such as
the last id used) with an entity group, but what happens to that
indicator when the entity group goes away? If it is destroyed and
then recreated when the entity group is recreated, how does the id
allocator avoid previously used ids?

> (or that are root entities). more inhttp://code.google.com/appengine/docs/datastore/keysandentitygroups.h...
> .

[ryan]

On Dec 20, 11:14 pm, Thomas Johansson <prenc...@gmail.com> wrote:
>
> Is this to be interpreted as getting the following, in order, more or
> less? (where the ID/Key is in square brackets)
>
> /Image[42]
> /Image[127841]
>
> /Thread[10]/Post[42]
> /Thread[10]/Post[127841]
>
> E.g. the ID is ever increasing, but only for the current fully
> qualified path?

correct.

> Can we ever experience inserting a record that will have a lower ID than the
> previous?

the others beat me to this answer. :P if the rest of the path is
identical, then it's uncommon, but definitely possible.

> Forgot to ask; How does the key fit into the hierarchy, compared to
> the key_name & id?

i'm not sure i understand this question. a key is composed of app id
and the full path to the entity. key names and ids are components of
paths. feel free to follow up if that doesn't answer the question.

On Dec 23, 10:44 am, Andy Freeman <ana...@earthlink.net> wrote:

> That's a curious comment. It suggests that ids may be reused if
> there's another difference in the path.

correct. for example, each of these keys is valid and points to a
distinct entity:

/Post[1]
/Post[1]/Post[1]
/Post[1]/Post[1]/Post[1]
/Post[1]/Thread[1]
/Thread[1]
/Thread[1]/Post[1]
/Thread[1]/Post[1]/Thread[1]/Post[1]
/Thread[1]/Post[1]/Post[1]/Thread[1]

> However, combining that with the statement that there's no reuse for a
> given path is interesting because there's no obvious place for the
> datastore to keep track of what ids it has used.

under the covers, each entity group has its own id namespace that's
used for non-root entities. root entities have their own per-app id
namespace. both namespaces are independent of kind and path
hierarchies (apart from the root entities, since they define entity
groups).

that means that in practice, every id-based root entity has an id
that's across all other root entities, and every id-based non-root
entity has an id that's unique across all other entities in that
entity group.

this is all just implementation details, though! we don't have any
plans to change id allocation right now, and it's unlikely that we
will, but the only hard uniqueness guarantee we provide is that the
full path is unique.

the upshot of all of this is that you generally want to use full keys
whenever possible. bare ids or key names are generally only
interesting or useful when the rest of the path is held constant,
including the kind of the entity in question. that's the only way to
guarantee the key names or ids will be unique.

> Yes, one can store an indicator of what ids have been used (such as
> the last id used) with an entity group, but what happens to that
> indicator when the entity group goes away? If it is destroyed and
> then recreated when the entity group is recreated, how does the id
> allocator avoid previously used ids?

good question! per-entity-group id namespaces currently live forever.
if all of the entities in a given entity group disappear, and then the
entity group later reappears, ids for new entities in that group will
not be reused.

this is another implementation detail, though, not a hard guarantee!

[Andy Freeman]

> good question! per-entity-group id namespaces currently live forever.
> if all of the entities in a given entity group disappear, and then the
> entity group later reappears, ids for new entities in that group will
> not be reused.

If I never allocate ids within a given entity-group namespace, does
anything for that namespace live forever (after all entities in said
group have been deleted)?

I ask because I'm creating lots of entity groups with multiple
entities where every below-root entity has an explicit name. (In
other words, I'm not creating any entities that have ids within these
groups.) These entity groups don't last very long and I never reuse a
given entity group name (the root class name and its unique name).

The total number of such groups that have entities at any given time
is fairly modest, but the total number that has existed over the life
of the application can be fairly large. Thus, if any information
about these groups "live[s] forever", I may have a problem.

> this is another implementation detail, though, not a hard guarantee!

Thanks for the information. I'm clearly surprised that any
information "live[s] forever" as I assumed that most applications that
use entity groups would not reuse their namespaces.

[ryan]

On Dec 31 2008, 6:49 am, Andy Freeman <ana...@earthlink.net> wrote:
>
> If I never allocate ids within a given entity-group namespace, does
> anything for that namespace live forever (after all entities in said
> group have been deleted)?

...

> The total number of such groups that have entities at any given time
> is fairly modest, but the total number that has existed over the life
> of the application can be fairly large. Thus, if any information
> about these groups "live[s] forever", I may have a problem.

ah, i understand the motivation now. if you never create any id-based
entities, then no, there's nothing in the entity group that lives
forever.

[Andy Freeman]

I think that this "id counts for an entity-group namespace live
forever" property should be documented because I suspect that many
uses of entity-groups will involve single-use names.

The only reason that I'm not running into it is that I prematurely
optimized to a single db.put, which forced me to use explicit key_name
s. If I'd gone with multiple db.put s in a transaction, which would
have been somewhat simpler, I'd have let the system assign ids within
my "use once" entity-groups.

Of course, someone will want a way to purge these counts.

[Robin B]

Ryan,

These 2 statements appear to contradict:

"...in practice, every id-based root entity has an id
that's across all other root entities..."

"...bare ids or key names are generally only

interesting or useful when the rest of the path is held constant,
including the kind of the entity in question. that's the only way to
guarantee the key names or ids will be unique"

The first statement says, in practice, the id alone is unique across
kinds for root-entities, the second statement says the kind+id is
needed for uniqueness across root-entities. Could you please clarify?

When you say 'in practice' does that mean, in the current
implementation, all root entities regardless of kind each will have a
unique id, but that it is not guaranteed in the future?

Robin



On Jan 5, 7:23 pm, Andy Freeman <ana...@earthlink.net> wrote:
> I think that this "idcounts for an entity-group namespace live

[ryan]

hi robin!

On Jan 7, 10:49 am, Robin B <robi...@gmail.com> wrote:

> The first statement says, in practice, the id alone is unique across
> kinds for root-entities, the second statement says the kind+id is
> needed for uniqueness across root-entities. Could you please clarify?

yes, as you guessed, the difference is between what we explicitly
guarantee and how things actually work under the covers.

as i mentioned earlier, the only hard guarantee is that the full path
is unique. in practice, right now, ids themselves are actually unique
within each entity group (across kinds), and separately, across all
root entities (also across kinds). we don't have any plans to change
that in the near term, but it's still just an implementation detail,
not a hard guarantee.