Sharp increase in instance count starting two days ago!

[Waleed Abdulla]

Two days ago, the number of instances assigned to my app started going up considerably and surpassed 1200 instance at some point (see graph). The number of billed instances went up as well, but not a significant hike like the total non-billed instances. Anyone know why this is happening? I'm on Py2.5 & master/slave datastore. 

Waleed

[Brandon Wirtz]

Dear Google App Engine Patron,

This is an automated response to your posting.

Thank you for your inquiry about why your GAE experience sucks.  It appears from your posting that you are running on Master Slave.  We regret to inform you that any issues you have are related to this decision which you made early on in your work with GAE and you will continue to regret this decision until you move to High Replication.  If you are unwilling, or unable to move off of Master / Slave we make the following suggestions:

 

Make a Funny Custom Error page with lots of pop-ups so that you can monetize your downtime.

Mark scheduled downtimes on the calendar in advance and claim they are religious holidays (Google is God, and if God is resting so should you)

Modify your code to work in a read only, mode so that maintenance has minimal impact.

Put Migration to High Replication on your roadmap, schedule a vacation during the month the migration is scheduled to take place so it is someone else's problem.

 

In the unlikely event that you still believe the Master Slave offers the best choice for your application we might suggest you visit a psychiatrist, or a neurologist as clearly your brain is also experiencing some sort of malfunction.  We are aware that Master Slave sounds much sexier than High Replication. We are also aware that replication is what threatened the SyFy Stargate:SG1 Universe, but we remind you that was a work of fiction.

 

Have a great day, and we thank you for your continued patronage.

 

Google App Engine Volunteer Support (We are in no way affiliated with Google, and we don't really volunteer any support)

 

PS

Did you like that marketing thing Google did where they pre-select High Replication, then on the Master Slave description imply that Master Slave will cost 1/3 as much, but with downtime and performance issues it actually costs more? Yeah, we wish we thought of it too. We'd do similar things with our marketing but Google probably patented it.

 

[Jeff Schnitzer]

Ah hah! Now we know who has been hogging the cluster! It's YOUR FAULT!

Jeff
;-)

> --
> You received this message because you are subscribed to the Google Groups
> "Google App Engine" group.
> To post to this group, send email to google-a...@googlegroups.com.
> To unsubscribe from this group, send email to
> google-appengi...@googlegroups.com.
> For more options, visit this group at
> http://groups.google.com/group/google-appengine?hl=en.

[Brandon Wirtz]

More seriously…

 

Waleed has one of those apps that I believe the concistency model makes MS the “right” choice for. Because eventual is not as instant as you might want.

That said, I think MS seems to be a lot more temperamental in terms of how fast it performs and how the scheduler responds to conditions.

 

1200 is a crap ton, and while I realize the SLA doesn’t cover MS. This seems like a “Billing” error kind of thing that Google should take some responsibility for.

 

 

[IPv6]

sheer brilliance ^))

[Jeff Schnitzer]

To repeat that important point:

The problem with the current (new) billing model is that when the
service provider screws up, the customer pays. In the long run this
can only alienate the customer. How many times do I want to call my
cellphone provider and tell them that they screwed up my bill, even if
they apologize and give me a credit? Having actually gone through
exactly this scenario, the answer is "twice" before I change
providers.

Google has a significant reserve of goodwill with me, so it would take
a lot more than a few billing issues to make me choose another
platform. But I can't imagine that too many other people feel the
same way.

The solution to this *seems* pretty straightforward - Google should
"stop the clock" when executing internal RPC calls. We're already
paying for datastore operations. If you need to change what a
datastore operation costs to make it revenue-neutral, so be it. But
we've gone a step backwards - the whole point of moving to
bill-by-datastore-ops was to make pricing more transparent, yet what
we've actually produced is "bill by datastore ops plus a random
additional amount of instance hours depending on how sick the
datastore happens to be right now". We were better off with
api_cpu_ms, at least that was consistent.

Actually, when you think about it, charging instance hours only makes
sense for single-threaded apps. In multithreaded apps, concurrency is
dependent on CPU usage, so charging by the megacycle really does make
sense. Really, single-threaded GAE needs a totally different billing
model than multi-threaded GAE.

Jeff

[Ugorji Nwoke]

Hear, hear.

This is a clear case of moral hazard. The incentives to optimize and reduce latency are backwards, since more efficiency leads directly to reduced revenue for Google. We just have to trust that Google wouldn't screw us?

Like Jeff says, there's a large amount of goodwill, but there has to be some give from Google's part.

> To post to this group, send email to google-appengine@googlegroups.com.

> To unsubscribe from this group, send email to

> google-appengine+unsubscribe@googlegroups.com.

[Waleed Abdulla]

Haha @Brandon & Jeff. Thanks for the chuckle :)  Believe it or not, I do check for read-only mode and I have a lot of code dedicated to recovering from GAE errors. True story.

I plan to switch to HRD at some point, but I fear it might be a major hassle because we have a lot of data and entity types, and we use the blobstore as well. Also, it worries me that if something goes wrong after I flip the switch, I have no way to contact anyone to get help. 

Good point about the reserve of goodwill, Jeff. I've definitely been tapping into my goodwill savings recently. 

Waleed

To view this discussion on the web visit https://groups.google.com/d/msg/google-appengine/-/8j0IhBVMbV4J.

To post to this group, send email to google-a...@googlegroups.com.
To unsubscribe from this group, send email to google-appengi...@googlegroups.com.