Datastore OFFSET is WORTHLESS

[James Gilliam]

I discovered that GAE charges you for skipping entities in Datastore with OFFSET.  So, if you want to retrieve 1 record at offset 10, you pay for reading 11.

This makes OFFSET expensive and WORTHLESS ...

Offset should never be mentioned in the documentation without a footnote that says you can't afford to use it.

Kinda pissed about this.

[Thiago Catoto]

Hi,

I guess a work around is using cursors, I guess thats why in the GAE performance track on Google I/O they say "never use offset, use cursor instead"

Cheers

--
You received this message because you are subscribed to the Google Groups "Google App Engine" group.
To view this discussion on the web visit https://groups.google.com/d/msg/google-appengine/-/JJAn3Ju-rD4J.
To post to this group, send email to google-a...@googlegroups.com.
To unsubscribe from this group, send email to google-appengi...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.

[Jeff Schnitzer]

It's not a secret. This behavior is clearly documented, and _every_
time offsets come up in discussion Google says "don't use offsets, use
cursors" - because that's what you should do.

Yes, offset is expensive and rarely useful. Like the docs say, don't
use it unless you really know what you are doing. You shouldn't be
pissed at anyone but yourself for not paying attention. The datastore
is not an RDBMS.

Jeff

[James Gilliam]

I do use cursors a lot but had no idea that offset is so expensive ... IT IS ONLY MENTIONED ONCE IN THE DOCUMENTATION

My point is WHY HAVE OFFSET AT ALL ?   

IT IS WORTHLESS JUST AS I SAID IN MY ORIGINAL POST.

IF YOU WANT OFFSET, JUST READ ALL THE RECORDS -- COST THE SAME.

[Jeff Schnitzer]

I'd say the huge "Note" is pretty blatantly obvious:

https://img.skitch.com/20121009-xxm3tru195tjysm1sj913umj1t.jpg

Offset has its uses. But annoying people who don't read
documentation, blame other people for their mistakes, and post in all
caps can figure it out for themselves.

Jeff

[James Gilliam]

You hurl insults while asserting offset has some usefulness without any support. So, give me an example of how it is anything but mathematically pathological.

[Takashi Matsuo]

Hi James,

First of all, you are right. You should always use cursors whenever possible.

We used to offer only 'offset' for going back and forth within query results before 1.3.1 release.

http://googleappengine.blogspot.jp/2010/02/app-engine-sdk-131-including-major.html

Thus there is a lot of code with the 'offset' still running in the production for a historical reason. It is hard for us to just kill this feature.

However, as you guys noticed, please use cursors whenever you can.

If you want us to improve our documentation, maybe you can file a bug on our issue tracker with more concret suggestions rather than ranting with the caps.

Thanks,

-- Takashi

--

You received this message because you are subscribed to the Google Groups "Google App Engine" group.

To view this discussion on the web visit https://groups.google.com/d/msg/google-appengine/-/Kd0I90rUVLcJ.

To post to this group, send email to google-a...@googlegroups.com.
To unsubscribe from this group, send email to google-appengi...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.

--
Takashi Matsuo | Developers Advocate | tma...@google.com

[Vinny P]

Cursors are nice, but not necessarily useful in all situations. I have code that uses both offsets and cursors in the same application, because I need the flexibility of offsets occasionally. In the cases where cursors can be used, I use cursors for speed. The GAE documentation comments on the limits to cursor queries here: https://developers.google.com/appengine/docs/java/datastore/queries#Limitations_of_Cursors 

In particular, cursor queries do not update themselves if a new entity matching the query is added between cursor operations, or if an existing entity is updated to no longer fit the cursor query (which is a big problem if your data changes quickly, as mine does).

Perhaps you could share examples of your code/datasets, and we could find ways to speed it up/reduce its costs?

-Vinny P

[James Gilliam]

The main point I am making is offsets are useless because reading all records from 0 to offset costs the same as reading a a record from +offset.  If GAE didn't charge for the records being skipped, it would be useful to use offsets. I suspect the only reason they charge for skipping records is because they cannot locate the nth record without reading them all.  This suggests a defect in their design that has nothing to do with entity database.

I realize that getting GAE to fix this issue is impossible.  However, the list of GAE issues continues to mount.  For example, not getting bounces on mail has been an outstanding issue for three years even though the documentation clearly says it is supported.  Another example is the horrible cost for secure domains and offering a 'cheaper' alternative that only works in limited situations.  I am just getting tired of the gae issues and they clearly don't care.

On Monday, October 8, 2012 11:07:45 AM UTC-7, James Gilliam wrote:

[Jeff Schnitzer]

On Tue, Oct 9, 2012 at 10:40 AM, James Gilliam <jimgi...@gmail.com> wrote:
> The main point I am making is offsets are useless because reading all
> records from 0 to offset costs the same as reading a a record from +offset.
> If GAE didn't charge for the records being skipped, it would be useful to
> use offsets. I suspect the only reason they charge for skipping records is
> because they cannot locate the nth record without reading them all. This
> suggests a defect in their design that has nothing to do with entity
> database.

This is a defect in pretty much all database designs. What do you
think MySQL does when you specify an offset? It runs the query and
walks through the result set until it has skipped N results. It's
only fast in MySQL if you have all the necessary indexes in RAM; GAE
is designed for very large datasets so you can't make this assumption.
This is one quirk of GAE; it's not very convenient for small
datasets. But your app won't suddenly collapse under load when the
index grows beyond an arbitrary point.

You could make an argument that GAE offsets should be small ops
instead of read ops. But you can easily make this happen yourself -
issue a keys-only query and batch get the results. Performance of
keys-only + batch-get is fairly similar to normal query performance.

Jeff

[stevep]

Google is a bit schizophrenic right now IMHO. Its pricing is enterprise, yet it still desires the startup developers' halo (appealing to their desire for high-productivity). So GAE product managers have been riding the backs of startup developers who chafe at the enterprise-level pricing while waiting for enterprise CIOs to lose their fear of GAE's PaaS lock-in. Recent lowering for SSL monthly costs (and posts by new GAE PMs) suggests some internal debate. You may or may not be in the final product's pricing schema. Unfortunately I believe these are the pains we suffer as a very intelligent set of engineers gain an understanding of value-added services (as opposed to Amazon who makes no mistakes re: providing value-added services). Remember: Google is a great adwords/adsense company trying to expand to value-added services. GAE is pretty new for them.

[Richard Watson]

Enterprise pricing and GAE aren't in the same universe, unless we're talking about enterprises that run out of the back of a truck. "Enterprise" is where they ask for your phone number, and then get an expensive salesperson to figure out how much you value the product, and then charge you that. This is free to start, 9 bucks a month to enable billing, mostly usage-based and very few customers will ever have a monthly GAE cost that equals that of hiring their next employee.

Compare Oracle, SQL Server or SAP to GAE, just for some perspective. GAE is exceptionally startup-friendly, unless you have a startup that will never have revenue.

[Matija]

Hey,

is there any news about backwards cursor? Two years ago they were 'presented' in next gen queries google io presentation. We did get some features but not backwards cursors.

Matija

[stevep]

I did say 'schizophrenic'. $500/month for premier support. $100 for full SSL (since lowered). Maybe I was wrong re: saying enterprise. Clearly they want high value-add developers which is  a good thing IMHO. If you were around when they went to instance billing from cpu usage billing, you will recall some devs who were using absurd amounts of resources for the amount they paid (e.g. very low cpu charges whilst consuming astronomical non-cpu resources). Didn't mean much with my prior post - just that I believe we are in a transitional stage.

[Alfred Fuller]

Backwards paging is currently supported, you can see this ndb example of how to do it: https://developers.google.com/appengine/docs/python/ndb/queries#cursors

To view this discussion on the web visit https://groups.google.com/d/msg/google-appengine/-/Ww6zx2oKOxMJ.

[Matija]

Also in Java API?

[Alfred Fuller]

Yes:

http://code.google.com/p/googleappengine/source/browse/trunk/java/src/main/com/google/appengine/api/datastore/Query.java#505

http://code.google.com/p/googleappengine/source/browse/trunk/java/src/main/com/google/appengine/api/datastore/Cursor.java#87

To view this discussion on the web visit https://groups.google.com/d/msg/google-appengine/-/JMm4_mlQZTQJ.

[Matija]

Holly shit... I am still like in 2010 XD 

Tnx.