Upcoming Change to Datastore Auto IDs

[Chris Ramsdale]

Hello App Engine folks,

In the upcoming 1.8.1 release, the Datastore default auto ID policy in production will switch to scattered IDs to improve performance. This change will take effect for all versions of your app uploaded with the 1.8.1 SDK.

You can preview the new behavior in the dev server, where scattered auto IDs are the default. These IDs are large, well-distributed integers, but are guaranteed to be small enough to be completely represented as 64-bit floats. If you still need legacy ids for some entities (e.g. because you want smaller numbers for user-facing ids), we recommend you use the allocateIds() API, which will continue to behave as before. You can also override the default auto id policy by setting the new auto_id_policy option in your app.yaml/appengine-web.xml to legacy, but please note that this option will be deprecated in a future release and will eventually be removed.

If you have any questions don't hesitate to follow-up here or drop me a line directly.

-- Chris

Product Manager, Google App Engine

[Ray]

In short: if any part of your applications use 32 bit integers (which is still the default for many environments) to store any IDs of entities, your application will break once this change is deployed.

[timh]

But not python.  It seems to cause the javascript front ends the most woe.

T

[Vinny P]

On Friday, May 24, 2013 7:04:08 PM UTC-5, Chris Ramsdale wrote:

In the upcoming 1.8.1 release, the Datastore default auto ID policy in production will switch to scattered IDs to improve performance. 

Out of curiosity, what does "improve performance" mean? Faster datastore puts? Less variable times? Cheaper infrastructure costs?

-----------------

-Vinny P

Technology & Media Advisor

Chicago, IL

My Go side project: http://invalidmail.com/

[timh]

From my understanding sequential IDs, keys etc  mean that data is written into the same area (tablet).  By scattering ID's they can more distributed data and better concurrency.

Maybe not the best description but thats my guess,   Sequential values have been discussed in the past with regards to indexes as well.

T

[Chris Ramsdale]

timh@'s response is correct.  by scattering the IDs we avoid overloading a Bigtable tablet and thus the associated latency.

-- Chris

--
You received this message because you are subscribed to the Google Groups "Google App Engine" group.
To unsubscribe from this group and stop receiving emails from it, send an email to google-appengi...@googlegroups.com.
To post to this group, send email to google-a...@googlegroups.com.
Visit this group at http://groups.google.com/group/google-appengine?hl=en.
For more options, visit https://groups.google.com/groups/opt_out.
 
 

[Jesse Rohwer]

Also, in response to Ray and Timh regarding integer data types: although the new auto ids will be too large to store in 32-bit integers, javascript code which encodes ids as 64-bit floats will still function. The new auto ids are designed not to exceed the maximum integer exactly representable by an IEEE standard double precision floating point.

[jon]

Also, in response to Ray and Timh regarding integer data types: although the new auto ids will be too large to store in 32-bit integers, javascript code which encodes ids as 64-bit floats will still function. The new auto ids are designed not to exceed the maximum integer exactly representable by an IEEE standard double precision floating point.

I think I know what the highlighted bit means, but I don't know what that maximum integer value is. I use Java and I declare my IDs as Long. Is my app safe? (I'm sure other Java devs have the same question)
 

[Adi Ben-Ari]

my Java app uses Integer and just broke!

[Alfred Fuller]

A Java long is a 64-bit integer and the correct primitive to store an ID in.

--

[Jeff Schnitzer]

On Thu, Jun 13, 2013 at 5:12 AM, Adi Ben-Ari <adi.b...@gmail.com> wrote:

my Java app uses Integer and just broke!

That was never safe. You should replace all Integer ids with Longs in your code.

Jeff

[Nico Verwer]

On Saturday, May 25, 2013 1:53:33 PM UTC+2, timh wrote:

But not python.  It seems to cause the javascript front ends the most woe.

This is important to note if your application has a front-end using javascript.

It means that any id greater than 9007199254740992 will break, if you try to represent it as a javascript number. See this question on stackoverflow.

Strings will of course work. They do not support number arithmetic, but you should not need to do that on id's.

It is a shame that Google has not warned users of the appengine about this potential hazard.Of course, as appengine developers we know what we are doing, but it would show that Google cares about its customers. However, maybe there is nothing to worry about:

In the announcement it says: "These IDs are large, well-distributed integers, but are guaranteed to be small enough to be completely represented as 64-bit floats." The javascript specification says: "... all the positive and negative integers whose magnitude is no greater than 2^53 are representable in the Number type". Now, 2^53 is the largest exact integral value, so this might mean that things will work well if you represent id's as javascript numbers. Can someone from Google confirm this?

[Nico Verwer]

I should have said: 2^53 is the largest exact integral value representable as a 64-bits integer.

[Alfred Fuller]

2^53 is the largest exact integer value representable by a 64-bit float

scattered ids are guaranteed to be representable by a 64-bit float which means they are less than or equal to 2^53.

On Fri, Jun 14, 2013 at 11:12 AM, Nico Verwer <nve...@gmail.com> wrote:

I should have said: 2^53 is the largest exact integral value representable as a 64-bits integer.

--
You received this message because you are subscribed to the Google Groups "Google App Engine" group.
To unsubscribe from this group and stop receiving emails from it, send an email to google-appengi...@googlegroups.com.
To post to this group, send email to google-a...@googlegroups.com.

Visit this group at http://groups.google.com/group/google-appengine.

[Nico Verwer]

On Friday, June 14, 2013 8:14:43 PM UTC+2, Alfred Fuller wrote:

2^53 is the largest exact integer value representable by a 64-bit float

scattered ids are guaranteed to be representable by a 64-bit float which means they are less than or equal to 2^53.

Thanks a lot for answering this, Alfred. It saves me some worries and potential headaches.