AWS equivalence of GAE datastore

[will]

I'm dismayed by the huge price increase of the new GAE pricing plans,
and in serious consideration of migrating to AWS. Does anyone know
what's the closest equivalence of GAE Datastore in AWS? Is it
SimpleDB or something else? AWS releases something new often and I've
kept up with it.

[Tom Carchrae]

MongoDB and http://code.google.com/p/morphia/

Morphia is not equivalent, but has a similar API to Objectify.  Search this group for my previous noob questions on it.

Tom

[Jon Stevens]

Depending on what you are doing, I think you'll also find that the final pricing of AWS won't be that much better. Just imagine how much it is going to 'cost' you when your database goes down in the middle of the night and you have to wake up and fix it. ;-)

jon

On Thu, Sep 1, 2011 at 1:01 AM, will <willi...@gmail.com> wrote:

[William Wong]

I hope AWS SimpleDB won't need me to babysit it.  That's why I ask for AWS's equivalence.

[Jon Stevens]

Simpledb is *nothing* like the datastore.

Missplet on my ipone

[William Wong]

What are the major differences?

Admittedly I haven't looked at SimpleDB for a while, but from what I remember Datastore and SimpleDB have similar underlying architecture and similar premise: HA, non-relational, transactional, shard-based, etc.

Anyway I'm not looking for a flamefest on SimpleDB vs Datastore.  I'm just looking at options to get out of GAE.

[Mat Jaggard]

Have you seen appscale? http://appscale.cs.ucsb.edu/

On Sep 1, 8:43 pm, William Wong <williamw...@gmail.com> wrote:
> What are the major differences?
>
> Admittedly I haven't looked at SimpleDB for a while, but from what I
> remember Datastore and SimpleDB have similar underlying architecture and
> similar premise: HA, non-relational, transactional, shard-based, etc.
>
> Anyway I'm not looking for a flamefest on SimpleDB vs Datastore.  I'm just
> looking at options to get out of GAE.
>
>
>
>
>
>
>
> On Thu, Sep 1, 2011 at 12:23 PM, Jon Stevens <latch...@gmail.com> wrote:
> > Simpledb is *nothing* like the datastore.
>
> > Missplet on my ipone
>

> > On Sep 1, 2011, at 10:44 AM, William Wong <williamw...@gmail.com> wrote:
>
> > I hope AWS SimpleDB won't need me to babysit it.  That's why I ask for
> > AWS's equivalence.
>

> > On Thu, Sep 1, 2011 at 10:08 AM, Jon Stevens < <latch...@gmail.com>
> > latch...@gmail.com> wrote:
>
> >> Depending on what you are doing, I think you'll also find that the final
> >> pricing of AWS won't be that much better. Just imagine how much it is going
> >> to 'cost' you when your database goes down in the middle of the night and
> >> you have to wake up and fix it. ;-)
>
> >> jon
>

[Jon Stevens]

This is the page which describes the limits. If you already understand the limits of the Datastore, you will quickly come to the conclusion that these are two different products.

http://docs.amazonwebservices.com/AmazonSimpleDB/latest/DeveloperGuide/index.html?SDBLimits.html

It isn't a flame fest. It is that they are two different products. End of story.

jon

[Didier Durand]

Hi,

Would do same proposal as Mat above: AppScale.

regards

didier

On Sep 2, 12:27 am, Jon Stevens <latch...@gmail.com> wrote:
> This is the page which describes the limits. If you already understand the
> limits of the Datastore, you will quickly come to the conclusion that these
> are two different products.
>

> http://docs.amazonwebservices.com/AmazonSimpleDB/latest/DeveloperGuid...

>
> It isn't a flame fest. It is that they are two different products. End of
> story.
>
> jon
>

> On Thu, Sep 1, 2011 at 12:43 PM, William Wong <williamw...@gmail.com> wrote:
>
> > What are the major differences?
>
> > Admittedly I haven't looked at SimpleDB for a while, but from what I
> > remember Datastore and SimpleDB have similar underlying architecture and
> > similar premise: HA, non-relational, transactional, shard-based, etc.
>
> > Anyway I'm not looking for a flamefest on SimpleDB vs Datastore.  I'm just
> > looking at options to get out of GAE.
>

> > On Thu, Sep 1, 2011 at 12:23 PM, Jon Stevens <latch...@gmail.com> wrote:
>
> >> Simpledb is *nothing* like the datastore.
>
> >> Missplet on my ipone
>

> >> On Sep 1, 2011, at 10:44 AM, William Wong <williamw...@gmail.com> wrote:
>
> >> I hope AWS SimpleDB won't need me to babysit it.  That's why I ask for
> >> AWS's equivalence.
>

> >> On Thu, Sep 1, 2011 at 10:08 AM, Jon Stevens < <latch...@gmail.com>
> >> latch...@gmail.com> wrote:
>
> >>> Depending on what you are doing, I think you'll also find that the final
> >>> pricing of AWS won't be that much better. Just imagine how much it is going
> >>> to 'cost' you when your database goes down in the middle of the night and
> >>> you have to wake up and fix it. ;-)
>
> >>> jon
>

[Jeff Schnitzer]

If you make heavy use of the datastore, my guess is that your easiest
path to an alternative host will involve MongoDB and Morphia.

You can try moving to Elastic Beanstalk, SimpleDB, and SQS but this
feels like a lot of work for a platform you will have to do a lot of
management for. AWS' offerings are not nearly as well integrated as
GAE. I did some experimentation there and didn't like what I found.

Maybe one of the other cloud providers will be better - check out
RedHat's new offering. Or maybe VMware's. Neither will be
deploy-and-forget like GAE.

It's certainly possible that you have an app that is not sustainable
with GAE's new pricing, but I would look closely. With multithreading
a single Java appserver instance can today handle 10 simultaneous
requests, and this number will probably go up as they refine the
service. So instance pricing is probably not as painful as it sounds.
The bigger question is datastore call pricing, which you may be able
to optimize with memcaching. It's certainly going to be more
expensive than running your own Mongo instances on linode or whatnot,
but GAE provides HRD and automatic infinite scaling. Deploy and
forget.

I'm somewhat worried about the pricing as well, but for the time being
the benefits of GAE are simply too compelling. That said, I don't use
backends even though I'd like to because they're too expensive. But
the costs of the rest of the system do not seem outrageous for most
applications.

Jeff

[William Wong]

Jeff,

Thanks for the detail reply.  It has good points and good opinions.

In my case, I don't use datastore extensively.  It's kind of restrictive and slow as a database to begin with.  A good portion of the development time was getting around its quirks, which fortunately limited its usage in my case.  BTW Objectify is great in lessen the pain in dealing with datastore.  I was hoping Objectify can be used to wrap other datastores in other places.

Good luck.  Hope Google change this position to make it better.

[ZeroCool]

I agree with Jeff.
The typical price rise for big apps (100qps) will be 100%, which means
you will pay $2 with the new pricing for the same resources consumed
of $1 as for now.
But at the same time, the effortless management keeps me away from
other options.
It's a pain for sure, but we have to move on.

[ZeroCool]

I can manage to cut the CPU Time by 50% with the slide bars in
Application settings.
But that's all I can do, as the main cost comes from datastore reads/
writes while memcache is already heavily used in my app.
Oh I miss the old days. :(

Resource Used Free Billable Charge
CPU Time:
$0.10/CPU hour 422.90 6.50 416.40 $41.64
Bandwidth Out:
$0.12/GByte 9.81 1.00 8.81 $1.06
Bandwidth In:
$0.10/GByte 4.14 1.00 3.14 $0.32
Stored Data:
$0.005/GByte-day 0.00 1.00 0.00 $0.00
Recipients Emailed:
$0.10/1000 Emails 0.10 2.00 0.00 $0.00
High Replication Storage:
$0.008/GByte-day 220.81 0.50 220.31 $1.77
Backend Usage:
Prices $0.00 $0.72 $0.00 $0.00
Always On:
$0.30/Day No - - $0.00
Total: $44.79
Estimated Charges Under New Pricing
The charges below are estimates of what you would be paying once App
Engine's new pricing model goes live. The amounts shown below are for
your information only, they are not being charged and therefore do not
affect your balance.

If you would like to optmize your application to reduce your costs in
the future, make sure to read our Optimization Article. If you have
any additional questions or concerns, please contact us at:
appengine_up...@google.com.
Frontend Instance Hour costs reflect a 50% price reduction active
until November 20th, 2011.
Resource Used Free Billable Charge
Frontend Instance Hours:
$0.04/Hour 352.71 24.00 328.71 $13.15
Backend Instance Hours:
$0.08/Hour 0.00 9.00 0.00 $0.00
Datastore Storage:
$0.008/GByte-day 220.81 1.00 219.81 $1.76
Blobstore Storage:
$0.0057/GByte-day 0.00 5.00 0.00 $0.00
Datastore Writes:
$1.00/Million Ops 26.39 0.05 26.34 $26.34
Datastore Reads:
$0.70/Million Ops 85.26 0.05 85.21 $59.65
Small Datastore Operations:
$0.10/Million Ops 75.41 0.05 75.36 $7.54
Bandwidth In:
$0.10/GByte 4.14 1.00 3.14 $0.32
Bandwidth Out:
$0.15/GByte 9.81 1.00 8.81 $1.33
Emails:
$0.01/100 Messages 0.91 1.00 0.00 $0.00
XMPP Stanzas:
$0.01/1000 Stanzas 0.00 1.00 0.00 $0.00
Opened Channels:
$0.01/100 Opens 0.00 1.00 0.00 $0.00
Total*: (before clipping to daily budget) $110.09

[Natalie]

Jeff,

You can try moving to Elastic Beanstalk, SimpleDB, and SQS but this
feels like a lot of work for a platform you will have to do a lot of
management for.  AWS' offerings are not nearly as well integrated as
GAE.  I did some experimentation there and didn't like what I found.

Do you mind talking a bit more about your experience with Elastic Beanstalk?

I know you do not recommend migrating to there but I really wish you could tell me why. Due to the nature of our apps (long average request (500ms), write heavy), we are looking at alternative options like Beanstalk that seems to offer some level of auto management to Java app. What can we expect from AWS Beanstalk?

Thank you.

[Jon Stevens]

I don't think he is recommending that you don't migrate there. He is just saying it is a platform that you'd have to do work managing. It is up to you to factor the 'costs' involved with each platform into how you decide to implement and deploy it.

AppEngine = Zero Management

AWS = Management

jon

[Jeff Schnitzer]

On Sun, Sep 4, 2011 at 7:53 AM, Natalie <nataliegl...@gmail.com> wrote:
>

> Do you mind talking a bit more about your experience with Elastic Beanstalk?
> I know you do not recommend migrating to there but I really wish you could
> tell me why. Due to the nature of our apps (long average request (500ms),
> write heavy), we are looking at alternative options like Beanstalk that
> seems to offer some level of auto management to Java app. What can we expect
> from AWS Beanstalk?

I am hardly an expert. I tried setting up the match hive for
Similarity on EB as an experiment. I found:

* The documentation is quite poor by GAE standards.
* The community forums are pretty much a wasteland. Doesn't look
like many people are using it.
* The tools were clumsy. Nothing so elegant as the "deploy" button in eclipse.
* Long waits while a server starts or restarts. Really, unacceptably
long - multiple minutes.
* Old versions of some java libraries (don't remember which ones
offhand, probably apache ones like httpclient) in the deployment
profile conflicted with ones I was trying to use in my WAR.
* I couldn't find *any* way to control java.util.logging output. I
don't think there is one. Seriously, WTF?

It was a dreadful experience, and sent me scurrying back to
rackspacecloud. EB felt entirely half-baked. Maybe it will get
better, but I doubt it. In AWS, everything is a hypervisor virtual
machine. For application serving, GAE's approach of "everything is a
JVM" is vastly superior - a JVM starts up in seconds and consumes less
resources.

Seriously, if I had to ditch GAE, I'd probably go to some PaaS
provider that offers MongoDB/Morphia. Most likely RedHat's new
offering, but that's mainly to get positioned for using Gavin's new
language when it's ready. Either that or I'd go to Joylent and start
using Node/CoffeeScript on the server side. Depends on the project.

Unfortunately there really isn't anything else like GAE on the market.
All the other PaaS (if you can really call it that) providers seem to
be adopting the philosophy of "we'll give you any piece you want to
manage!" rather than "we just give you an API and you can forget about
what's underneath".

Jeff

[pac]

Found one open source project

http://code.google.com/p/appscale/

unfortunately does not support all java api

http://appscale.cs.ucsb.edu/appengine.html


On Sep 5, 6:06 am, Jeff Schnitzer <j...@infohazard.org> wrote:

[Jeff Schnitzer]

I'm finally back from my trip (all arms, legs, and wheels attached)
and catching up on the pricing shitstorm that hit the GAE mailing
lists. Wow. Some comments:

Everyone's app is going to be different, but I notice the loudest
complaints come from Python users. We Java developers have
<threadsafe>true</threadsafe> which should provide some significant
insulation from per-instance pricing; it *should* (given a properly
behaving scheduler) make per-instance pricing equivalent to CPU usage.
That is to say, GAE should only spin up a new instance of your app
when you've already maxed out a CPU of the old instance. The
scheduler might have some bugs still, and it may not be smart enough
yet (ie max concurrency might be temporarily hardcoded at 10), but I
have faith that these issues will be resolved.

So, don't freak out over per-instance pricing. Just make sure you
have threadsafe=true and file bugs if the scheduler doesn't prune idle
instances fast enough (or spin up new instances at the appropriate
time). I haven't spent time tuning the new knobs yet, but my guess is
that we shouldn't need to set high max latency because a single
instance can handle 10 (possibly more) concurrent requests.

The big change that may affect Java developers is the new datastore
pricing. This will vary wildly by application. The solution is to
optimize the app in the same way you were doing so before - optimize
queries, use memcache where possible. I don't have a good feeling for
what this price increase will generally entail because I can't filter
out all the per-instance complaints from Python users on the mailing
lists. However, you aren't paying for a DBA so appengine is still a
win for most apps.

Philosophically, here's what I see happening:

Appengine (probably among many other departments at Google) now needs
to pay for itself. This means that one of Google's previous
advantages (nearly infinite resources to throw at projects) is now a
liability - we customers must now bear the brunt of all the
inefficiencies built into the system because cost was previously a
non-issue. Google has little experience building lean products, so I
expect that unsubsidized GAE will *always* be significantly more
expensive than "other providers".

So look at your own business model. Is it like Google's? Are you
building a product whereby the actual server time represents a tiny
percentage of your net value? Then GAE is still a good bet. On the
other hand, are you building a product that competes on price, where
you charge small premium for an enhancement to basic computing
services? Examples: CDN-in-a-box, Dropbox. If so, then GAE is a
terrible idea. Because Google is not an efficient company, the
baseline cost will always be too high.

I hope this doesn't discourage most of you from using Appengine. It
hasn't discouraged me. As Jon mentioned, we just started a new
company developing a new product and we're choosing GAE as a platform
despite the price increase. For a small bootstrapped startup, the
benefits are just too compelling.

Jeff

[Jeff Schnitzer]

On Wed, Sep 7, 2011 at 10:38 AM, Jeff Schnitzer <je...@infohazard.org> wrote:
> I'm finally back from my trip (all arms, legs, and wheels attached)

Oh, this means I will roll a release of Objectify 3.0.1 tonight.

Jeff

[Jon Stevens]

I did. Heroku is a neat idea, but the summary is that it is even more expensive than AWS (and AppEngine).

"A single dyno can only process one HTTP request or job from a job queue at a time."

http://devcenter.heroku.com/articles/dynos

And the pricing _really_ reflects that. So, as soon as you want say... 10 concurrency (ie: a single java appengine instance), you are looking at $323 a month and that is *just* for http processing. What happens when you are slammed and want 100 concurrency? Oh wait, the pricing slider on the website doesn't go that high... 

Factoring in postgres or another database solution and the side costs start to multiply quickly. 

The fact is that there really isn't a comparable product to AppEngine. The front end isn't a big deal to me. That has been commoditized out and I'm an expert in that area. It isn't hard to reliably solve that problem. 

What *is* missing from any other product is the HR datastore. You can run many instances of Mongo, but as soon as one of them has a problem in any way, that means a pager is going off (assuming your monitoring is setup and working... which is an exercise in itself) and you're scrambling to figure out why it is broken. I'm willing to allocate a significant portion of money to not having to deal with those issues.

Also, the fact that I'll never have to think about a database connection string again is amazing.

Ikai, you guys really have a winner product on your hands. I'm sure I'll complain about the pricing of it at some point, but in my eyes, the benefits outweigh the IT management issues and the pricing.

jon

On Wed, Sep 7, 2011 at 11:18 AM, Ikai Lan <ik...@google.com> wrote:

Jeff, did you ever look at Heroku with Java? I think that's probably one of the closest products to App Engine. I haven't had time to work with it myself, though I fiddled around with a very early version (Rails/MySQL) and was generally impressed with what I saw. Both that team and ours seem to have the same rough set of ideas about what a platform as a service should do.

In terms of businesses, I fully agree. There's another thread on the main groups where someone is asking if Google has any tips for people trying to bootstrap a business because App Engine is no longer, well, free. I really don't. I have a lot of respect for entrepreneurs because bootstrapping is a hell of a lot harder than it sounds. That applies to starting a business in general.

--

Ikai

[Ikai Lan]

Jeff, did you ever look at Heroku with Java? I think that's probably one of the closest products to App Engine. I haven't had time to work with it myself, though I fiddled around with a very early version (Rails/MySQL) and was generally impressed with what I saw. Both that team and ours seem to have the same rough set of ideas about what a platform as a service should do.

In terms of businesses, I fully agree. There's another thread on the main groups where someone is asking if Google has any tips for people trying to bootstrap a business because App Engine is no longer, well, free. I really don't. I have a lot of respect for entrepreneurs because bootstrapping is a hell of a lot harder than it sounds. That applies to starting a business in general.

--

Ikai

On Sun, Sep 4, 2011 at 10:06 PM, Jeff Schnitzer <je...@infohazard.org> wrote:

[Jeff Schnitzer]

On Wed, Sep 7, 2011 at 11:45 AM, Jon Stevens <latc...@gmail.com> wrote:
> I did. Heroku is a neat idea, but the summary is that it is even more
> expensive than AWS (and AppEngine).
> "A single dyno can only process one HTTP request or job from a job queue at
> a time."
> http://devcenter.heroku.com/articles/dynos

It looks like that doc may be slightly out of date. This page
suggests there is a second stack (herokuapp.com instead of heroku.com)
which allows multiple concurrent requests:

http://devcenter.heroku.com/articles/http-routing

But yeah, the shining gem in the world of Appengine is the HRD.
Everything else is can be replicated in other services - maybe with
some effort, but it's possible for a small team. The HRD changes
everything.

Jeff

[Ikai Lan]

(I'm not a spy!)

I've passed on your thoughts to the team. Thanks for the feedback!

By the way, if either of you gentlemen want to come out tonight to meet the team at the Thirsty Bear, a beer will be reserved for you:

https://docs.google.com/a/google.com/spreadsheet/viewform?formkey=dEpzUWdiMU9ZLVlSd0NwSjlHaTFtQVE6MQ

This invite also extends to anyone else on this list in San Francisco who might want to come out and talk about App Engine coming out of preview.

--

Ikai

[Jeff Schnitzer]

Cool, Jon and I will be there!

Jeff

[Mat Jaggard]

Wish I could come - it's a little bit too far from the UK :-(

On Sep 8, 9:28 pm, Jeff Schnitzer <j...@infohazard.org> wrote:
> Cool, Jon and I will be there!
>
> Jeff
>
>
>
>
>
>
>
> On Thu, Sep 8, 2011 at 11:46 AM, Ikai Lan <i...@google.com> wrote:
> > (I'm not a spy!)
> > I've passed on your thoughts to the team. Thanks for the feedback!
> > By the way, if either of you gentlemen want to come out tonight to meet the
> > team at the Thirsty Bear, a beer will be reserved for you:

> >https://docs.google.com/a/google.com/spreadsheet/viewform?formkey=dEp...

>
> > This invite also extends to anyone else on this list in San Francisco who
> > might want to come out and talk about App Engine coming out of preview.
> > --
> > Ikai
>

> > On Wed, Sep 7, 2011 at 12:16 PM, Jeff Schnitzer <j...@infohazard.org> wrote:

[Scott Hernandez]

I like reserved beer. I will be there.

[Ikai Lan]

I forgot to mention I am probably not going to be able to make it out tonight. You guys see me too much anyway.

--

Ikai

[Thiago Rossi]

I have an idea for an iPhone/Android app and I want to store the data using MongoDB. I'm still learning and new to this. I don't know what's the best way to link the iPhone with MongoDB? I mean, I need to create an API/WebService/frontend, like a website, right? I know the cheapest hosting are PHP/MySQL, and not sure there's a way to use MongoDB… I was using GAE, but because of the prices I want to migrate and I need to take care sometimes I can have 1000 users accessing at the same time or maybe more.

[Jeff Schnitzer]

Perhaps you should ask this on a MongoDB list?

Jeff

[Thiago Rossi]

Oh, sorry, I was just reading the thread (that started from a user unhappy with the new prices of GAE) and forgot I was on the AppEngine list. LOL Sorry!

[Tom Carchrae]

Anyone looking for the menu, this is a good article: http://kkovacs.eu/cassandra-vs-mongodb-vs-couchdb-vs-redis

Would be great if people have any similar links that point to best DB for job X.

Tom

[Thiago Rossi]

Great!