Very large entity ids on 1.7.6 dev server

[Mahron]

hi,

I just installed the new sdk and the scattered ids allocation is
giving me trouble. It is generating very large ids like :
5261330264675581952.

I am communicating with the server with a flash client which can not
handle such large numbers.

The problem can be resolved by forcing sequential allocation.

I would however like to know if such large numbers are likely to
appear in production, as it would create problems and require to
handle ids as strings ?

[Alex Burgel]

On Thursday, March 21, 2013 5:10:30 AM UTC-4, Jesse Rohwer wrote:

We are aware of this issue and will have it fixed before we release scattered ids in production.

Does this mean that scattered ids will become the default id generation strategy in production? How does that affect allocate_ids which generates ids in a sequence? 

[D X]

I'd like to star this one, this essentially killed all my client side javascript.

On Thursday, March 21, 2013 5:10:30 AM UTC-4, Jesse Rohwer wrote:

We are aware of this issue and will have it fixed before we release scattered ids in production.

Thanks for reporting it.

[Jeff Schnitzer]

I also want to make sure this isn't dropped on us automatically in
production. This will have user-facing effects since
(crockford-encoded) ids show up in our URLs - and users put these URLs
in print materials, so Very Large Ids are a problem.

Jeff

> --
> You received this message because you are subscribed to the Google Groups
> "Google App Engine" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to google-appengi...@googlegroups.com.
> To post to this group, send email to google-a...@googlegroups.com.
> Visit this group at http://groups.google.com/group/google-appengine?hl=en.
> For more options, visit https://groups.google.com/groups/opt_out.
>
>

[Vinny P]

On Wednesday, March 20, 2013 11:13:32 AM UTC-5, Mahron wrote:

I just installed the new sdk and the scattered ids allocation is 
giving me trouble. It is generating very large ids like : 
5261330264675581952. 

Star and +1 on this. I'd like a notice before this is placed into production.

On Thursday, March 21, 2013 4:10:30 AM UTC-5, Jesse Rohwer wrote:

We are aware of this issue and will have it fixed before we release scattered ids in production.

 

Just to be clear, what is Google's intended fix on this? Are you capping the ID at Javascript max number, turning the number into a string. or something else?

-Vinny P

[Mahron]

Ok, so strings it is. Upgrade complete !

[Jesse Rohwer]

Hi all,

To address the questions on this thread in order:

1. While we will be moving to scattered ids in production soon, we're going to limit the size of ids to the max integer that a 64-bit float can represent exactly, which will address the problem with floating-point representations. This will also be changed in the 1.7.7 SDKs.

The motivation for scattered ids is that sequential ids cause hot spots in bigtable and GAE MR, which leads to significant performance limitations.

More info on scattered ids:

Scattered auto ids in 1.7.5 SDK

See the section "Specifying the automatic id allocation policy":

https://developers.google.com/appengine/docs/Python/tools/devserver#Using_the_Datastore

https://developers.google.com/appengine/docs/java/tools/devserver#Using_the_Datastore

2. The current allocate_ids() functions will continue to allocate sequential ids. So, manually allocated sequential ids will still be available.

3. We're sorry for the inconvenience this caused with the 1.7.6 dev appserver, but we're glad we caught this early!

4. If you need small ids, you won't be able to rely on automatic ids for this. One option would be to use the allocate_ids() functions to manually allocate ids.

Thanks for your feedback!

[Jeff Schnitzer]

Will there be any problems intermingling allocated and automatic ids?
Right now I can switch back and forth between explicitly allocated and
automatically generated ids. Will this continue to be the case?

Will there be an allocator method to generate a scattered id?

Jeff

[Jesse Rohwer]

Manually allocated ids will not conflict with automatic ids. Yes, you will be able to manually allocate scattered ids as well.

[stevep]

Quick question -- just for my clarity, and I am sorry if this was already clear. Once this change is implemented in production (this is not an SDK question), if prior to writing an new entity, I get one entity id value using allocate_ids(), is this going to be an integer value of reasonable size consistent with today's automatically generated IDs? Or will allocate_ids('mymodel', 1) yield one of the new "bigint" scattered number IDs? TIA -stevep

On Tuesday, March 26, 2013 6:11:38 PM UTC-7, Jeff Schnitzer wrote:

[Brandon Thomson]

On Tuesday, March 26, 2013 7:33:00 PM UTC-4, Jesse Rohwer wrote:

1. While we will be moving to scattered ids in production soon 

4. If you need small ids, you won't be able to rely on automatic ids for this. One option would be to use the allocate_ids() functions to manually allocate ids.

Definitely not happy about this. Will require a lot of effort on our end to mitigate. 

[Jesse Rohwer]

Hi all,

After hearing from many of you that this would be a painful change, we've decided that when we switch to scattered auto ids we will provide a temporary option to continue sequential auto ids in production.

We will still be deprecating sequential auto ids, for the performance reasons described above; but you will have time to prepare, and an interim period during which you can opt-out.

Thanks again for your feedback.

[Jeff Schnitzer]

This is a good idea. I've gone through my code and moved to manual
allocation in the places that matter... but if this changed by
surprise it would have caused a lot of user grief.

Jeff