Datastore operations are seriously expensive

[Kenneth]

I've been monitoring my app today for cost and the instances are so far fairly ok.  However I'm getting absolutely hammered on the datastore operations. (yes, this was in the billing preview)

I found this gem with seems to have been updated since I last looked at it.  This may be old news to people but I just discovered it.

http://code.google.com/appengine/kb/postpreviewpricing.html#operations_charged_for

I was under the impression that if you did an entity get by key it was a small operation, apparently not.

If you put a new entity with 4 indexed properties that's actually 10 writes!

If you update those 4 properties 18 writes!

If you delete that entity 10 writes again!

If you have a composite index you're screwed even more.

I understand how the datastore works and all but that's pretty crazy.  Every time I look at a property and think maybe I won't index that one I think actually no, I can see how I might need that sometime.

Damn.

[roberto.cr]

It's sad how we find ourselves thinking about not implementing stuff,
because of the pricing...

The new pricing Model had to be implemented, we can all agree on
that... but the pricing itself is too high!

Does Google servers cost that much more than Amazon's? I'm sure that's
not the case.

On Nov 7, 2:48 pm, Kenneth <kennet...@aladdinschools.com> wrote:
> I've been monitoring my app today for cost and the instances are so far
> fairly ok.  However I'm getting absolutely hammered on the datastore
> operations. (yes, this was in the billing preview)
>
> I found this gem with seems to have been updated since I last looked at it.
>  This may be old news to people but I just discovered it.
>

> http://code.google.com/appengine/kb/postpreviewpricing.html#operation...

[Gerald Tan]

Maybe Google Cloud SQL will better suit your dataset.

[Kenneth]

I hear what you're saying, but at that rate I'm pretty much rewriting my application plus cost isn't even known.  And I like the datastore.

The problem as always is how the app was written.  Users can update one field at a time on a data grid and save it back via ajax, so that's seriously not efficient from a cost pov if they're doing a few fields.  The model they're writing to has more than 100 indexes (property indexes both directions + composite indexes).  So I need to look at de-indexing fields and removing composite indexes given some of the datastore goodies that have come on stream in the past year.  There's also a problem with another model with 3 large key lists that's written a lot.  I don't know how the cost on that is even calculated tbh.

The fundamental problem is I lack numbers on how much each request is costing me so I don't know where to focus my efforts.  It's mostly guesswork and intuition.  There could be a big elephant I'm missing.

I'd really love to pre-purchase datastore operations in the same vein as instances.

[Rishi Arora]

After managing my instances, I had a similar sticker shock with data store usage.  However, the app stats utility, even though its really built around the old pricing model, was invaluable in telling me what my heavy hitters were in terms of data store ops.  I implemented caching for my most important routinely accessed data, and that helped a great deal.  Also important was to completely move off less critical data from the data store to external storage.  This increased CPU and instance time for me, because I implemented it using URL fetch, but in the end, saved a lot of costs.

--
You received this message because you are subscribed to the Google Groups "Google App Engine" group.

To view this discussion on the web visit https://groups.google.com/d/msg/google-appengine/-/Wgyduglk9KMJ.

To post to this group, send email to google-a...@googlegroups.com.
To unsubscribe from this group, send email to google-appengi...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.

[vlad]

link does not work

[Jeff Schnitzer]

FWIW, your app would have been a problem under the old pricing model
as well. The only difference WRT the datastore is that instead of
being charged "datastore operations" you were charged a magic number
of "api_cpu_ms" per operation. The pricing model is pretty much the
same, it's just more expensive.

If you're churning through your datastore ops because you have 100
indexes, you would have churned through your CPU quota before. The
problem is you're trying to update too many indexes. The only
solution is "don't".

BTW you only incur costs for index changes; if you re-put an entity
without changing the index, you don't get charged for an index update.

Jeff

[Alfred Fuller]

Here is an article that can help you reduce the # of composite indexes you use in some cases:

http://code.google.com/appengine/articles/indexselection.html

[Volker Thiel]

> If you're churning through your datastore ops because you have 100
> indexes, you would have churned through your CPU quota before.  The
> problem is you're trying to update too many indexes.  The only
> solution is "don't".

Not true. I had an application that I optimized in CPU usage and it
was well below the limits every day. I only had two entities with a
few indexes each (three and four respectively). The latter is updated
frequently (several times a day). I (currently) have about 4000
entities which required approximately 20 write operations on each
update. This was no problem under the old billing model!
I threw out some of the indexes but on the cost of usability. Now I'm
down to 8 write ops for a new entity and 6 for an update. I got those
figures from the dev environment, but what I'd really like to see is
read and write ops in the production logs, since the new billing isn't
focused around CPU cycles anymore. My two cents.

Volker

[djidjadji]

The links is a redirect to

http://code.google.com/appengine/docs/kb/billing.html

But it should be (I think)

http://code.google.com/appengine/kb/billing.html

[Jeff Schnitzer]

I don't mean to argue that the price hasn't gone up... just that it bills more or less the same way as before (which hid datastore ops in "api_cpu_ms").  They just tripled (or more) the price for these ops.  Sounds like your app is in the margin of what was cheap before and expensive now.

Read and write ops in the production logs would be rad.  Is there an issue I can star?

Jeff