Too many instances being launched!

[Edward Sentongo]

Hello, 

Since February 29, too many instances are being launched to service few requests.

I have not updated any app whatsoever. Front end requests are normal or even slightly less that previous.

Example: 

- at one time 2 instances of v1.9.34 and 4 instances of v1.9.33 were launched. Yes, it release versions v1.9.33 and v1.9.34 are being toggled at will!

- one instanced serviced 700 request.

- the other 2 serviced 10 - 20 requests

- the rest serviced 1 or 2 requests. 

What I have done:

- I shutdown the instances serving less than 10 requests. They were immediately restarted. Why should 3 instances service 1 simple request be restarted as soon as I shut them down?

- I disabled the affected apps and re-enable them. This didn't help anything.

* There's no options in my app-settings to limit the number of (idle) instances

Consequences:

- My costs have been increasing. In the last 3 days, I have been billed what I normally use in a week. Is there a possibility of a refund? Is there a 'trick' I am missing here?

[Edward Sentongo]

It just go worse. In the 2 billed days of this month (March), most apps have been billed what they normally cost in 2 weeks!
Is this going to be fixed any soon?

[Edward Sentongo]

In the photos. One app with a daily limit of $2. Today is Thursday March 3 2016 (The day is not over) yet the billing is already $7.87!!!
If I divide that by 3 days, I get $2.62. If I divide it by 2 days I get $3.94. 

Is App Engine broken that they now have to charge me way above my limit? This is happening on all my apps.

Even last month's bill (February) jumped from $95 to $119 out of the blue. I hadn't take a screenshot so I have no proof. But please consider those 2 pictures. Something is really wrong.

On Thursday, March 3, 2016 at 2:33:39 PM UTC+1, Edward Sentongo wrote:

[Edward Sentongo]

Here is another app with a daily limit of $2. Today, Thursday 3rd of March, it's already $8.37 even before day 3 of March ends.
Either you divide that by 2 days or 3 days, the daily limit is already exceeded. Something is very wrong here. Every other app I host on App Engine is reporting these crazy numbers yet the daily requests haven't even increase.

It's not fair to pay for phantom usage. That's why I always set the daily limit to $2. To avoid weird situations.

On Thursday, March 3, 2016 at 2:33:39 PM UTC+1, Edward Sentongo wrote:

[Nick (Cloud Platform Support)]

Hey Edward,

It sounds as though some of your instances were migrating to the newest version of the runtime (1.9.34), and it's possible that the new version caused the instances to spin up in strange ways. This should never affect your billing or cause you to go over the daily budget, so you should contact Billing to get that sorted out immediately. I wish you the best of luck in contacting them.

As far as the reason for the instances restarting once killed, this could have something to do with the autoscaler's perception of your request load, and the configuration you gave in your app.yaml, but it could also be a behaviour of the version of the 1.9.34 runtime which some of you instances were using. You can feel free to ask about this in the case that you open with billing.

This is about all the advice I can give you from looking at the graphs and reading your post. Again, I wish you the best of luck in getting a good resolution to your issues here.

Sincerely,

Nick
Cloud Platform Community Support

[Edward Sentongo]

Hello Nick,

Thank you for your reply. I submitted the issue. There was no option to submit screenshots.

"Thank you for contacting us. One of our support agents will process it shortly, usually within 2-3 business days. We can not guarantee a response time. In the meantime, please see our help center to see if your question may have already been answered there."

Hopefully my issue will be solved immediately because around this time is when payments are deduct from me for google app engine usage.

Cheers,

Edward

[Nick (Cloud Platform Support)]

Good to hear, I wish you the best in your interaction with Billing support. Feel free to update this thread if you have any useful information for other users.

[Edward Sentongo]

Hello Nick (and to whoever it may concern):

1. March 3, 2016: 

  - I reported this issue here and provided screenshots showing the aggressive creation of instances.

  - I clearly stated that is was abnormal because front-end requests from my users hadn't changed positively/negatively to have caused the issue whatsoever.

  - I clearly stated that I hadn't edited my code either.

  - I clearly stated the continous toggle between v1.9.33 and v1.9.34.

  - Nick advised me to contact billing on this issue. Stating that it couldn't have affected my billing.

2. March 4th, 2016: 

  - I sent the issue to Billing department twice. I provided screenshots. As usual, I never heard from them.

  - I was being charged for frontend instances. Data store and othere were all less than $0.30 cents daily and still are. 

  - App Engine was supposed to fail on all frontend requests above my set billing but it didn't. 

3. March 8th, 2016:

  - Having not heard from billing, I filed the issue here: https://code.google.com/p/googleappengine/issues/detail?id=12798 (now flagged private and only visible to those with Editing permission)

4. March 9th, 2016:

  - I was requested for more information and specification which I promptly provided.

5. March 15th, 2016:

  - The number of instances started settling down to the normal 2 or 3 BUT they were still being killed and spawned. Thus costing me more money. 

  - I was advised to explicitly limit the maximum number of instances that can be alive. This doesn't help the damage cause in the last week of February and first 10 days of March.

  What is the use of limit max_instances to 1 or 2, yet they'll be recreated almost every 20 minutes on continued traffic? I also know the 15 minute rule of idle instances, so please, let no one bring it up because I monitor front-end requests as well.

  - I was promised contact from Billing department as well. As usual, I never heard from the Billing department.

6. March 21st, 2016:

  - My issue was declared 'Invalid' due to lack of information on how to reproduce the issue. Thus it wasn't deemed as the fault of App Engine for anything that happened.

7. March 22nd, 2016:

  - I sent screenshots I had provided on this thread. 

  - I clearly showed that the math didn't work on over 4 projects. The sum was way above the maximum daily limit. This was a result of App Engine Frontend Instances, not DataStore, not anything extra I use outside App Engine.

  - I was told that it wasn't frontend end instances but extra use of other resources outside App Engine. I don't do this, incase by billing is requested, I can provide it showing that frontend instances are the ones incurring those costs.

Notes:

- I am providing information here as a reference because the issue tracker thread will most likely be killed.

- I have been requested to export billing (https://support.google.com/cloud/answer/6293835?hl=en) which is done starting from the moment I requested. This will not help for the last week of February and first week of March when limits were exceeded and numerous instances were being spawned.

- Billing of busy December 2015 (31 days) was:                 $159

- Billing of January 2016 (31 days) was:                              $139

- Billing of February 2016 (29 days) was:                            $169.Problem started last week of February.

- Billing of March 2016 (1st - 19th, 19 days finalized) is:     $196.

On the billing history it is clearly stating App Engine Frontend Instances are the ones costing anything more than $1. December is supposed to be the most expensive month due to increased user calls to frontend api. DataStore and others are always below $1. It is at FrontEnd Instances where I always & must be billed the most.

So which extra spending am I doing outside this?

- I have provided my monthly billing summary (February and March, with January for reference purposes) in .csv to the issue tracker.

- I have also enabled auto daily export of billing. But this is almost useless because most damage was done in the last week of February and first 2 weeks of March.

[joshuamo]

Hi Edward,

First, thank you for this very detailed account of your interactions around this issue; it will be very helpful in debugging our processes to better resolve issues like these.  You've given a very clear impression of the anxiety induced here, and that's exactly what we want to reduce.  I'm also escalating this particular issue up our leadership chain, as well.

In terms of product, the difficulty in understanding the breakdown of charges over several projects seems to be the core issue here.  If that was more clear, it would be much easier to point to a cause  We do provide an optional service to export your billing data to CSV[1], which can then be ingested and analyzed however you prefer.  This won't provide a look at data before being enabled, but it will give more concrete data moving ahead.

I'll be jumping onto the linked issue with some more specific questions so we can follow up internally on the billing cases you mentioned, as well as to discuss specifics to your projects around the discrepancy seen.

[1] https://support.google.com/cloud/answer/6293835

Cheers,

Josh

Google Cloud Platform Support

[Edward Sentongo]

Hello Josh, thank you for your attention to this matter.

May I quote you for certainty:

 "In terms of product, the difficulty in understanding the breakdown of charges over several projects seems to be the core issue here."

I have been a paying customer on App Engine for more than a year now. All billing I received specifically pointed out the resources I used: Front-End instances, Datastore, bandwidth, etc.

When an app uses up all its resources allocated to it within 24 hours, App Engine stops serving any requests to/from that app.

During those hectic days, without prior influence from my side or users (I checked the logs and frontend requests), daily limits were maxed out and even exceeded on several apps.

It was about Front-End Instances. I noticed and made screenshots but also started disabling apps to try save the day (yup, StackOverflow) and also noticed the toggle between v1.9.33 and v1.9.34. Instances serving only 2 requests and immediately restarting yet traffic was continuos (I know about the 15 minutes idle time please!).

I am now repeating myself, not good for this thread.

Point is: If the billing says "App Engine Frontend instances" and an app shows 6 instances running, which instances shutdown and immediately restart within short times, and I notice billing has exceeded daily limits. Can't I conclude that those constantly toggling (and re-spawning) instances caused the massive billing? 

Anyhow, I hope your team comes finds out what went wrong in order to avoid this happening again.

[Edward Sentongo]

I have updated this issue https://code.google.com/p/googleappengine/issues/detail?id=12798 with evidence based on screenshots and exported billing analysis as was requested.

There are still inconsistencies with the frontend instances in some apps (thus, NOT all apps).

On Thursday, March 3, 2016 at 2:33:39 PM UTC+1, Edward Sentongo wrote: