High latency in HRD app. HELP!

[Nicanor Babula]

appid: domodentweb2

datastore: High Replication

normal latencies: 50-200ms

today's latencies: 1000-60000ms

Dear Appengine Team,

Google appengine is a wonderful product and gives us satisfaction seeing our app solving our customers problems every day. Thanks to appengine, we provide to our customers a web-based solution as fast as a desktop. Our customers are our fans. They were complimenting us for the app. Unfortunately, the dream seems to be over, as our app started encountering high latencies out of nothing. The first time we thought it was an isolated case (bad things happen to anybody, don't they?). As of today, we are encountering AGAIN high latencies. I think is the third or fourth time in less than 40 days. At a closer look, instances seem to either get restarted or disappear from the instances page for a few minutes. The same code version that worked fine yesterday, today is slow. Customers are upset and are calling us, saying that the app is so slow, it cannot be used. In fact the user experience is poor right now. The app is very slow. We don't know what to tell them anymore, since this happened again recently. Soon we'll start losing customers.

What can we do in this situation? 

Do we have to migrate to Premier account in order to have low latencies? Or the resources are the same for everybody?

If so, what do we have to do in order to migrate?

Please, Google Appengine Team, let us know something.

Thanks and keep up the good work,

Cristian Nicanor Babula 

and the Domodent team

[Mike Wesner]

Hopefully someone on the Google team can take a look for you.  I have a few questions and things for you to try.

1. How young are your instances?  are you seeing instance churn?

2. Can you post your app settings, max idle, how is your traffic? spikey?

3. Do you see the same issues on other appids?  (do you have a test instance setup somewhere for QA or staging?)

You might just be in a bad spot somewhere or it could be a legitimate issue based on your settings and some part of your application causing churn or other issues.

We have seen some of the best performance on HRD in a long time over the last 2 weeks or so.  

Mike

[Nicanor Babula]

Hi Mike, 

Thanks for your answer. 

1) The ages of the instances span from 8 minutes to 30 minutes. What do you mean by instance churn?

2) 

Idle instances: 1 - 6

Pending latency: Automatic - Automatic

Traffic: spikey (I don't think it's relevant, since yesterday was as spikey as today, but latency was good)

3) Not relevant, since I don't have test instances with billing enabled.

On Wed, Apr 11, 2012 at 3:53 PM, Mike Wesner <mike....@webfilings.com> wrote:

Hopefully someone on the Google team can take a look for you.  I have a few questions and things for you to try.

1. How young are your instances?  are you seeing instance churn?

2. Can you post your app settings, max idle, how is your traffic? spikey?

3. Do you see the same issues on other appids?  (do you have a test instance setup somewhere for QA or staging?)

You might just be in a bad spot somewhere or it could be a legitimate issue based on your settings and some part of your application causing churn or other issues.

Is there such thing as a bad spot? As for parts of the app generating problems: I don't think it is relevant as well, because the same code worked well yesterday, last week and so on.

We have seen some of the best performance on HRD in a long time over the last 2 weeks or so.  

Mike

--
Nicanor Cristian Babula

[Nicanor Babula]

Now the app is fine. Until next time...

--
Nicanor Cristian Babula

[Jeff Schnitzer]

On Wed, Apr 11, 2012 at 11:30 AM, Nicanor Babula
<nicanor...@gmail.com> wrote:
> Hi Mike,
>
> Thanks for your answer.
> 1) The ages of the instances span from 8 minutes to 30 minutes. What do you
> mean by instance churn?
> 2)
> Idle instances: 1 - 6
> Pending latency: Automatic - Automatic

I presume you mean you have Min Idle Instances set to 1 and Max Idle
Instances set to 6.

Set these to Auto-Auto. It's been a couple months since I last
touched these settings, but every time I did I got a crappy UX with
lots of user-facing loading requests. Since going to Auto-Auto our
app has been consistently snappy.

Jeff

[Mike Wesner]

Your app will run well if you use auto for max idle instances, but you should not have to do that and it is expensive.  When things are running like they should you can tune it based on your traffic.  Ours is very smooth so we can safely set it to a fairly low setting and it runs well.  We leave some room for an odd process to spike our average latency (and thus spin up instances).  That could be your issue though.  Depending on how much traffic and how much it varies, 6 idle instances might not be enough to handle the ramp up and it's choking out your users.  (not enough instances for the traffic spikes)

What I meant by churn is lots of instances dying (from memory limits, errors or unknown infrastructure issues) or shutting down due to your max idle instances being too low for your ramp up/down rates and then getting started again.  

[Nicanor Babula]

Thanks for your answer. 

Yes, we have been experiencing instance churn. I suppose your app is related to webfilings.com. Have you ever experienced problems like this one? Your app is under a premier account? You are free not to answer the questions. I am asking because it is the first time I go in production with an appengine app and would be glad to learn from others who made it...

Cheers.

--
You received this message because you are subscribed to the Google Groups "Google App Engine" group.

To view this discussion on the web visit https://groups.google.com/d/msg/google-appengine/-/p3BNM_Fs4bUJ.

To post to this group, send email to google-a...@googlegroups.com.
To unsubscribe from this group, send email to google-appengi...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.

--
Nicanor Cristian Babula