instances dying is a permanent states for datastore apps?

[Rafael Sanches]

Hello,

I have the application "yumyumlabs" and all of my users are complaining about "slowness" or broken connections.

To my surprise I went to the appengine console to see that my instances were dying out of nothing. No releases were made on my side and this seems a new behavior. This happening for a week already and I have tried everything.

I was hopping that this was temporary, but as it turns out it's not. This is the issue: http://code.google.com/p/googleappengine/issues/detail?id=8781

Can anyone tell me if this will be a new permanent state? All the datastore applications will continue to be this slow? 

I must take actions ASAP. The current service quality is unbearable. 

thanks

rafael

[timh]

Do you have billing enabled ?

idle instance can be shut down at any point where their count is above min idle.  What sort of settings are you running ?

I have a low traffic site with billing usually 1 or 2 reserved instances (depending on the time of the week.)  All content is dynamic (though cached by memcache) and not seeing any problems at all.

T

[Rafael]

Yes, I have a lot of traffic. Out of nothing the instances die and new ones come up. Check out this screenshot: http://cl.ly/image/3e1W1a1d0V3A

Please, see that some instances are up for more than an hour and a half. Although, many new ones are spawned. 

You're running on HDR? The may be related to datastore.

thanks

rafa

--
You received this message because you are subscribed to the Google Groups "Google App Engine" group.
To unsubscribe from this group and stop receiving emails from it, send an email to google-appengi...@googlegroups.com.
To post to this group, send email to google-a...@googlegroups.com.
Visit this group at http://groups.google.com/group/google-appengine?hl=en.
For more options, visit https://groups.google.com/groups/opt_out.
 
 

[stevep]

Quite some time ago I started a thread about instances for my paid app shutting down quickly (almost immediately). Same issue I believe. Perhaps it is working differently now, but I would be hesitant to conclude that this will stop occurring once billing is activated. It seems quite variable, so I am not suggesting timh is wrong. Perhaps it is a function of how busy other apps are in one's slice of GAE's virtualized pie.  As noted, this question was posted a good time ago, and has been followed by many similar questions asked by many devs affected by this. I would also be hesitant to suggest this pattern will quit repeating or that we will be given any insights about it from G. I think the last big thread on this (and this subject has engendered MUCHO discussion) ended with someone concluding that Java can not be used for low-volume GAE apps given this issue. I am Python with fast start times, so can not offer insights when spin up times are slow, but do sympathize.

[Vinny P]

Can you screenshot an AppStats profile view of a sample request?

-Vinny

[Jeff Schnitzer]

As an experiment, try getting rid of the resident instances and go
with Automatic - Automatic. It's been a while since I last touched
those buttons, but I had very bad results whenever I configure a lower
bound.

Jeff

[timh]

Hi

I am using HRD and usually my resident instances are many days old, however today I have one resident instance 2 hours old, 1 dynamic instance 2 hours old, and a dynamic instance 1.5 days old ;-)

T

[Rafael]

You are living the dream my brother. Check this out: http://cl.ly/image/3F0m2c3T1b3D

It's a screenshot from right now. Please, notice that there's two brave instances that are 1:40 hours old, although we have over 15 that are dying randomly. 

The best thing is that I am paying a lot for those 15 randomy dying. Also, my 10 million user app is quickly dying just like the instances.. it's beautiful. 

Alright, 3 days without sleeping. I officially tried everything possible on my side. Now it's just pray for google to fix it.

[timh]

Are you running M/S ?

I just had a look at a different app I have access to on M/S (has large numbers of users) and it appears to be cycling instances fast, and long startup times.

T

[Rafael]

YES! I'm on the master/slave. 

Google putting a message at the top of the appengine site saying that it's deprecated isn't an excuse for extreme reduction of performance.

Migrating to HDR will never happen to me. This is frustrating. 

[Cesium]

You read the previous posts on this right?

You are complaining about instances lasting an hour?

I'd sacrifice my prize unicorn to get that!
When service goes pear shaped, instances last seconds!

Yes. This behavior will change.
No. You can't predict when the change will occur.
No. There's nothing (effective) you can do.
No. You won't get a response from G.
Yes. You will get used to shoveling unicorn poo.
Yes. It looks like little piles of rainbows.

David

[Barry Hunter]

On Wed, Mar 13, 2013 at 9:36 AM, Rafael <mufu...@gmail.com> wrote:

YES! I'm on the master/slave. 

Google putting a message at the top of the appengine site saying that it's deprecated isn't an excuse for extreme reduction of performance.

The depreciation pretty much is Google admitting the can't (and/or won't) fix M/S. They KNOW it's bad, and want you to stop using it. 

 

M/S is a sinking ship. Google arent allocating much resources to bailing it out. They've moved on to brighter and better things. 

Migrating to HDR will never happen to me.

Then you have pretty much resigned yourself to issues. Why are you pointing a loaded gun at your own foot?

 By continuing to use M/S you making yourself and Google look bad. 

[johnP]

Barry -

If you look through this message board and at the issue tracker, you'll see that many people with HRD are also affected by this issue.  

As often is the case, the big problem is not the crime - it's the cover-up.  Everyone here can understand technical issues, if people are working to fix them.  The non-response from Google is what's freaking people out.

[johnP]

Or even if Google says that there is not a technical issue; that there have been internal tweaks to the scheduler that make instances that (in my case) would happily serve for days or weeks to now serve for less than 10-15 minutes (during a good part of the day); and that we should do a, b, and c to maximize performance; that would also provide some clarity.

[Barry Hunter]

I didnt say there where not problems with HRD. 

Just that its expected there will be problems with M/S. Google have pretty much said they WONT even try to be fixing them. 

Any resources Google have working on the issue, will be fixing it (ultimately) within the HRD codebase. They probably wont back-port it to M/S. M/S is like a live grenade just waiting to explode, they wont touch it. 

Alas its just in Googles culture to be tight-lipped about upcoming changes (and that includes bug fixes) - so as you say we are just all running around like headless chickens, while Google hopefully has someone calmly sitting there working on the issue (we just dont know about it). One day just to push it into production with little fanfare. 

Google's timescale on fixing issues, is also almost certainly orders of magnitude slower than most people would like. 

--

[johnP]

That's you projecting your hopes, that someone is calmly fixing the problem.  

But one of the attributes of competent leadership is effective communication.  When customers see that effective communication is lacking, they a symptom of poor leadership. If appengine has poor leadership, this is an existential threat to everyone who has bet their future on this platform.

[Rafael]

This is my biggest concern. I don't care about technical problems, I care about shipping high quality products. That was the reason why we picked appengine 2 years ago. 

The problem is that the service isn't working well and we can't predict for how long. We need to have any insight on how long this instability will last, so we can play ahead. 

We have serialized blobs everywhere across our application. We're paying the price for "clever optimizations". I believe it would be less work to move out of appengine rather than moving to HDR. 

[Barry Hunter]

 I believe it would be less work to move out of appengine rather than moving to HDR. 

You will have to do one or the other soon anyway. M/S will be turned off.  

[Rafael]

Barry,

Where have you read that it's going to be turned off? 

All google says is: "This application is using the Master/Slave datastore which is deprecated! All customers should migrate to the High-Replication datastore which offers greater stability and is resilient to data center failure. To learn more about migration see High Replication Datastore Migration."

That message and nowhere in the documentation it says that it's going to be turned off. It just says it offers greater stability. It doesn't mention that M/S will be worst than in the past either. 

thanks

rafael

On Wed, Mar 13, 2013 at 12:20 PM, Barry Hunter <barryb...@gmail.com> wrote:

 I believe it would be less work to move out of appengine rather than moving to HDR. 

You will have to do one or the other soon anyway. M/S will be turned off.  

--

[stevep]

I am uncertain about GAE given recent history, but have hope. Certainly given MS is deprecated, we cannot hold G. accountable for issues there. They **clearly** recommend HR, so MS app are 100% caveat emptor. However, the SO and Google Groups threads recently have been consumed with ire not limited to MS. A while ago, Jeff S. (whose posts I value the most along with Drake) posited that G. would have fixed the issues had they been able to do so. Going way back (when I started with GAE) billing was based on CPU milliseconds consumed. My guess is they thought this was highly disruptive. Then after beta, we got billed for 15 minute instance slices which was a big change from beta. This suggest that the initial assumptions were wrong (sh*t happens sometimes even if you are the brightest bulb in the pack). Overall, I think Jeff is right. There are some problems that can not be fixed given GAE's initial architecture. However, there may be some big changes coming with the App>Server architecture that only trusted devs (me excluded) get to enjoy. As I read the limited docs about what this entails, I wonder if this will be a fundamental change for GAE. It appears to fix most of my complaints, so I will be happy.

[Stuart Langley]

Rafael:

Here's the notice: http://googleappengine.blogspot.com.au/2012/04/masterslave-datastore-thanks-for-all.html

TOS for deprecation: https://developers.google.com/appengine/terms#Deprecation

M/S will be turned off 20th April, 2015.

[Rafael]

Stuart, 

The service and it's issues are getting worst and worst as we speak: http://cl.ly/image/2t1y2O2S2U1h (68 dying instances, just starting up by coincidence)

Can you please give us a word about the issues? Is this going to be happening on a constant basis? Is it only M/S affected?

By 2015 I hope to be out. Spending $40k a year on a service that I could host for $3k isn't in my plans for long. 

Please, respond. I must know if I need to migrate right now or it can wait more time. Migrating to HDR or outside isn't in my plans at all and will kill my startup.

thanks!

rafa

[Alpha Scorpii]

seeing from your screen capture that the memory usage seems so high... perhaps an new instance is started when there is no memory left to serve a new request?

[Alpha Scorpii]

by the way... i wonder why you are spending 40k a year....it doesn't seems your site got that much visitor

[Rafael]

my app has 10mi downloads. 

it's not much traffic and i could pay $5k if it was out of appengine. I would say the new billing is a bit off for high volume stuff.

On Fri, Mar 15, 2013 at 12:14 AM, Alpha Scorpii <hip8...@gmail.com> wrote:

by the way... i wonder why you are spending 40k a year....it doesn't seems your site got that much visitor

[John V Denley]

My Application has been charged TEN TIMES as much every week since mid february. I have changed NOTHING and I have NO extra customers and NO extra traffic.... everything has been stable apart from the fact that my CPU times have dramatically increased since we started having these instance warm up/hard deadline exceeded issues...

Below are my charges, as you can see, a fairly stable $2(ish) per week before 11th Feb and since then SIGNIFICANTLY higher charges, and NOTHING has changed at my end!!

2013-03-25 12:55:21	Charge Cleared	($25.74)	$0.00
2013-03-18 16:11:03	Charge Cleared	($19.67)	$0.00
2013-03-11 13:01:14	Charge Cleared	($6.85)	$0.00
2013-03-04 13:58:43	Charge Cleared	($6.94)	$0.00
2013-02-25 13:01:18	Charge Cleared	($13.16)	$0.00
2013-02-18 13:00:10	Charge Cleared	($9.06)	$0.00
2013-02-11 13:02:06	Charge Cleared	($20.42)	$0.00
2013-02-04 12:49:39	Charge Cleared	($2.30)	$0.00
2013-01-28 12:48:57	Charge Cleared	($2.10)	$0.00
2013-01-21 12:52:44	Charge Cleared	($2.10)	$0.00
2013-01-14 12:48:24	Charge Cleared	($2.10)	$0.00
2013-01-07 12:50:48	Charge Cleared	($2.10)	$0.00
2012-12-31 12:59:44	Charge Cleared	($2.10)	$0.00
2012-12-24 13:02:54	Charge Cleared	($2.10)	$0.00
2012-12-17 13:05:47	Charge Cleared	($2.10)	$0.00
2012-12-10 13:01:34	Charge Cleared	($2.10)	$0.00
2012-12-03 15:09:00	Charge Cleared	($2.64)	$0.00
2012-11-26 12:59:38	Charge Cleared	($2.10)	$0.00
2012-11-19 13:08:04	Charge Cleared	($2.10)	$0.00
2012-11-12 13:01:26	Charge Cleared	($2.10)	$0.00

[Rafael]

Hi John,

Migrating my app to HDR has solved most of the issues! I would highly recommend to migrate yours ASAP. 

The whole migration process was waaay easier than expected. Due to the fact that the old blobKeys are now alias to the new ones. 

thanks

rafa

--

[John V Denley]

Thanks Rafa (is this what you have been doing since you vanished from the tennis scene!! LOL )

I was actually trying to migrate to HRD yesterday, the only trouble is I wanted to migrate my current M/S appid ("idebanet") over to an new appid ("diarybooker") that I already own, the problem is that "diarybooker" is also a M/S datastore, but I am just not using it yet at all, I want to change that to an HRD datastore and THEN migrate, but there does not seem to be any mechanism available to do this. I already have "diarybooker-hrd" but really it would be nice to be able to use diarybooker.appspot.com given that my actual domain is www.DiaryBooker.com

If you or anyone reading this can help me either switch my MS "diarybooker" appid to HRD or allow me to delete "diarybooker" and recreate it as an HRD.

I could even live with aliasing diarybooker AND idebanet over to diarybooker-hrd if thats the way I need to go, but I cant even find anyway to create my own permanent alias manually!

PS as an aside: The minimum "support" contract with Google is $500 per month for App Engine, which I find frankly insane, surely there is a market there for use smaller (startup?) businesses to be able to get a minimal amount of support without having to pay that kind of money!! I dont even mind if email replies are "48 hours" and SLA is 1 week, but it just seems madness that its either NOTHING (and frankly almost zero actual support) or $500 per month!

--
You received this message because you are subscribed to a topic in the Google Groups "Google App Engine" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/google-appengine/8wBjYXIhYyg/unsubscribe?hl=en.
To unsubscribe from this group and all its topics, send an email to google-appengi...@googlegroups.com.

[Cesium]

Dearest Rafael,

You are living in a dream world if you think migrating to HRD will stop instances from dying prematurely.

This is not a criticism! Purposeful rejection of reality will save you!

Do not apply reason to the instance scheduler! Here, we call it a "faith-based" scheduler.

Have you not read any of the other posts on this issue???

Is it not definitive proof of the validity of my faith that the almighty G has not spoken to us about this problem?

(Should have posted this on Easter.)

David

[Carl Schroeder]

Ever since I ported our app to Go, we have had no trouble with the scheduler aggressively recycling dynamic Java instances (because we have none).

This may be due to the Go runtime depending on Mermaids instead of Unicorns and Skittles like the Java stack.

[Jeff Schnitzer]

FWIW, it's probably unwise to feel so strongly about your appid. You have your own domain, there's no reason for anyone to see the *.appspot.com. Just use diarybooker-hrd.

There may actually be some advantage in an obscure appid; unless you've built in precautions or have unguessable version names, a malicious user could spin up old versions of your app which might, for example, create entities with an obsolete structure.

Jeff

[John V Denley]

Good idea Jeff, never considered that.

The only real reason I wanted to use it, is because there have been times in the past when I have needed to direct my customers direct to the appspot url and its always a bit embarrassing to send them to my old idebanet account as its just confusing! one time it was to allow one of my customers to trial a new bit of functionality, but thinking back on it now it might have been better to use a URL redirect such as http://test.DiaryBooker.com, which I have just noticed DOES seem to go somewhere, I cant remember setting that up... i wonder where it goes ;)