what's up with COUNT?

[Philip]

Hello

Has anybody managed to overcome the count() limitation? I really need
to be able to query the EXACT number of records within datastore based
on different criteria (really no way to store counters within data
store itself) and I expect that number to be well over 1k.

I think this is somewhat ironic that with all the powerful features
that datastore provides we still don't have a simple record
aggregation functionality. Or am I mistaking?

Thank you

[joh...@easypublisher.com]

On 5/28/08, Philip <philip....@gmail.com> wrote:
>
> Hello
>
> Has anybody managed to overcome the count() limitation? I really need
> to be able to query the EXACT number of records within datastore based
> on different criteria (really no way to store counters within data
> store itself) and I expect that number to be well over 1k.

Have a look at vrypan counter, I think he is on to something.
http://groups.google.com/group/google-appengine/browse_thread/thread/f2ce195dc989aa2d

My own idea is to utilities the fact (or at least my assumption of a fact) that
the instance run in a single thread, to cache the counter in RAM instead of
writing it to the database. It's a bit less secure/accurate (you can
potentially loose
cache size if counts if the instance server explodes etc). But it doesn't
stress the data layer as much.
I guess it all depends on usage and requirement which is the best solution.

> I think this is somewhat ironic that with all the powerful features
> that datastore provides we still don't have a simple record
> aggregation functionality. Or am I mistaking?

Yes. It is a deliberate limitation, or sacrifice you will, to be able to
scale massively. If there was a way to eat the cake and have it, I think
Google would have walked it :-D

Someone said that app-engine scales horizontal not vertical.
I think to work with and get the most of app-engine, it is essential to learn
and accept that fact.
It is a struggle to get to term with a whole new set of design principles
that goes against all prior training and knowledge.
But for me, that's what makes it fun :-D

Best regards,
Johan

--
Johan Carlsson
Colliberty Easy Publisher
http://www.easypublisher.com

[NerdyTommy]

> Someone said that app-engine scales horizontal not vertical.
> I think to work with and get the most of app-engine, it is essential to learn
> and accept that fact.
> It is a struggle to get to term with a whole new set of design principles
> that goes against all prior training and knowledge.
> But for me, that's what makes it fun :-D

I totally agree with your point Johan. It is kind of fun to learn new
approach to design and application programming. I've lots of
experience from both Java and relational database worlds and it's very
interesting to me how to build truly scalable application using Python
and that odd BigTable thingy :)

Cheers,
--
NyTy

[Filip]

Johan,

I guess you mean the GAE memcache api? Just to point out to new
readers that global python variables do not work across calls because
the physical machine might change.

Filip

On 28 mei, 07:33, joh...@easypublisher.com wrote:

> On 5/28/08, Philip <philip.nuzh...@gmail.com> wrote:
>
>
>
> > Hello
>
> > Has anybody managed to overcome the count() limitation? I really need
> > to be able to query the EXACT number of records within datastore based
> > on different criteria (really no way to store counters within data
> > store itself) and I expect that number to be well over 1k.
>

> Have a look at vrypan counter, I think he is on to something.http://groups.google.com/group/google-appengine/browse_thread/thread/...

[Filip]

The idea is to count, sum, etc at put() time.
By the way, it was mentioned at Google IO that the built-in count()
actually consumes lots of resources, and that you should not use it.

Filip.