How are your apps going with the new pricing policy?

[Guillaume Laforge]

Hi all,

I wanted to know how your respective apps were doing, with the recent switch to the new GAE pricing policy.

How was your experience?

Personally, I've got three somewhat busy apps:

- the Gaelyk site itself: http://gaelyk.appspot.com/

- my blog: http://glaforge.appspot.com/

- the Groovy Web Console: http://groovyconsole.appspot.com/

The two key aspects that are bothersome are: the instance hours and the datastore reads.

That's usually those two limits that we tend to hit -- at least, that was the case for me.

Despite the policy changes, the Gaelyk website wasn't affected, and was staying under the quotas.

It's essentially a pretty static website, although the pages are templates (hence dynamic), but since the pages are cached, through the routes.groovy configuration, and that there's no datastore access, everything is fine here.

For my blog, I was a bit below the datastore reads level, but the frontend instance hours were more problematic.

The fact of moving the max idle instance knobs and latency knobs to their opposite, helped me a lot.

And although the website is up and running 24h/24, GAE doesn't spin up too many instances with those knob settings, and I'm now under the 28 hours limit. So all is fine for my blog too.

And the biggest problem was the Groovy Web Console... 

For the frontend instance hours, I'm always at the limit, but with the knobs to the opposite, it's okay, but really short...

For the datastore reads, that was the most painful aspect, as the Google bots and friends are doing a ton of reads when harvesting all the Groovy scripts that are saved.

I must confess the application wasn't very well written, so for example, I was doing three queries (basically the same one) to list the latest scripts published, the scripts by authors, the scripts by tags. Three queries which were essentially the same, more or less, that were totalling 5 read operations each (on op for the query + 2 * two ops because of the indexed columns I was sorting / filtering on).

So I had like 15 queries or so just to build up the pages that you see when you click on the link "recent scripts".

Despite my pages were cached (using just one read operation), it happened often that pages were not in the cache, and that my app needed those 15 operations.

So I had to optimize my application. Instead of doing three different queries, I ended up doing just one, and then I apply some sorting / filtering / grouping in memory with the results I got from the query. So I ended up with just 5 operations instead of 15.

Then, I ended up putting the results of the queries... in memcache, so instead of 5 ops, most of the time I ended up getting only one read operation.

In the end, I reduced my datastore reads by a factor of 15 or so.

And with those changes, I only consume about 40% of the datastore read operations quota!

Speaking of caching... with Vladimir in particular, and also a bit with the Objectify lead, we discussed some ideas of how to cache the results of get()'s and datastore.execute()'s in memcache, so that you would not have to deal with that caching trick yourself.

I still haven't totally made up my mind on how this should look like or how it should be implemented, but I think it might be a key feature of the next release.

Anyhow... I wanted to know what has been your experience with the pricing change?

Did you have to optimize your apps, if yes, what solutions did you find?

I'm looking forward to your feedback.

--
Guillaume Laforge
Groovy Project Manager
SpringSource, a division of VMware

Blog: http://glaforge.appspot.com/

Twitter: @glaforge

Google+: http://gplus.to/glaforge

[Youri]

Hello,
I'm using GAE for my nabaztag, there is only my rabbit hitting the
server every 2 seconds.
I have a very low usage for my app (and plenty of ways to improve) but
so far no issues with it. I'm almost using all the daily quota if my
rabbit is up all day.

Youri,

On Nov 25, 9:32 am, Guillaume Laforge <glafo...@gmail.com> wrote:
> Hi all,
>
> I wanted to know how your respective apps were doing, with the recent
> switch to the new GAE pricing policy.
> How was your experience?
>
> Personally, I've got three somewhat busy apps:
> - the Gaelyk site itself:http://gaelyk.appspot.com/
> - my blog:http://glaforge.appspot.com/
> - the Groovy Web Console:http://groovyconsole.appspot.com/
>

> *The two key aspects that are bothersome are: the instance hours and the
> datastore reads.*

> That's usually those two limits that we tend to hit -- at least, that was
> the case for me.
>
> Despite the policy changes, the Gaelyk website wasn't affected, and was
> staying under the quotas.
> It's essentially a pretty static website, although the pages are templates

> (hence dynamic), but since the *pages are cached, through the routes.groovy
> configuration*, and that there's no datastore access, everything is fine

> here.
>
> For my blog, I was a bit below the datastore reads level, but the frontend
> instance hours were more problematic.

> The fact of *moving the max idle instance knobs and latency knobs to their
> opposite, helped me a lot*.

> And although the website is up and running 24h/24, GAE doesn't spin up too
> many instances with those knob settings, and I'm now under the 28 hours
> limit. So all is fine for my blog too.
>
> And the biggest problem was the Groovy Web Console...
> For the frontend instance hours, I'm always at the limit, but with the
> knobs to the opposite, it's okay, but really short...
> For the datastore reads, that was the most painful aspect, as the Google
> bots and friends are doing a ton of reads when harvesting all the Groovy
> scripts that are saved.
> I must confess the application wasn't very well written, so for example, I
> was doing three queries (basically the same one) to list the latest scripts
> published, the scripts by authors, the scripts by tags. Three queries which
> were essentially the same, more or less, that were totalling 5 read
> operations each (on op for the query + 2 * two ops because of the indexed
> columns I was sorting / filtering on).
> So I had like 15 queries or so just to build up the pages that you see when
> you click on the link "recent scripts".
> Despite my pages were cached (using just one read operation), it happened
> often that pages were not in the cache, and that my app needed those 15
> operations.
> So I had to optimize my application. Instead of doing three different
> queries, I ended up doing just one, and then I apply some sorting /
> filtering / grouping in memory with the results I got from the query. So I
> ended up with just 5 operations instead of 15.

> Then, I ended up *putting the results of the queries... in memcache*, so

> instead of 5 ops, most of the time I ended up getting only one read
> operation.
> In the end, I reduced my datastore reads by a factor of 15 or so.
> And with those changes, I only consume about 40% of the datastore read
> operations quota!
>
> Speaking of caching... with Vladimir in particular, and also a bit with the
> Objectify lead, we discussed some ideas of how to cache the results of
> get()'s and datastore.execute()'s in memcache, so that you would not have
> to deal with that caching trick yourself.
> I still haven't totally made up my mind on how this should look like or how
> it should be implemented, but I think it might be a key feature of the next
> release.
>
> Anyhow... I wanted to know what has been your experience with the pricing
> change?
> Did you have to optimize your apps, if yes, what solutions did you find?
>
> I'm looking forward to your feedback.
>
> --
> Guillaume Laforge
> Groovy Project Manager
> SpringSource, a division of VMware
>
> Blog:http://glaforge.appspot.com/

> Twitter: @glaforge <http://twitter.com/glaforge>
> Google+:http://gplus.to/glaforge

[tronda]

> I'm using GAE for my nabaztag, there is only my rabbit hitting the
> server every 2 seconds.

I also have a Nabaztag. Have you written a server with Gaelyk? Is this
something you consider open sourcing. I'm very interested in getting
my Nabaztag up and running again and if the server infrastructure runs
with Gaelyk, this would be a perfect project for me to contribute to.

If you are interested please email me: trond dot andersen at gmail dot
com