did App Engine change instance scheduler algorithm in the past month?

[Tapir]

From the "Usage History" page, I found that the instance hours of all my apps increase much.
Comparing to previous months, there are no big traffic changes for my apps.

One of my app has only 200 pageviews each day, generally the free quota is enough to cover the usage,
and this app is charged about 2 cent every month.
But in April, there are about 1 dollar accumulated to be charged.

My another app has about 2000 pageviews each day.
Before April, this app will use about 26-29 CPU hours every day, and will be charged about 1 dollar every month.
But in April, the CPU hours increased to 35+ hours in several day, and there are about 10 dollars accumulated to be charged.

So it looks each app will be charged 10 times than before.
However, I didn't find any big traffic changes for my apps.
Has the App Engine change instance scheduler algorithm changed?

[Jordan (Cloud Platform Support)]

You should investigate how many instances each of your applications used in March and compare that with the amount in April. If you changed your applications' scaling settings it would have directly affected instance count, and in turn the amount of instance hours you use. 

It is also recommended to make sure you are stopping previous versions of your applications in the Cloud Console, as older versions can still be serving with instances.

- Note that Google Groups is reserved for general product discussions and not for technical support. It is recommended to open a free Billing Support case for any billing related inquiries. 

[Randee Frankel]

I am a Technology Integration Specialist for the School District of Philadelphia, but I do not understand what you are saying I should check. This is the first time I have tried to set this up and I have no other cloud apps. 

The page with the directions I am following directed me to this forum for assistance. 

If you get any auth-related issues while trying to deploy, please see the Google App Engine community forum for help.

If I am not in the correct place for technical support, then where do I go? There was no other reference to support on the page. 

--
You received this message because you are subscribed to the Google Groups "Google App Engine" group.
To unsubscribe from this group and stop receiving emails from it, send an email to google-appengine+unsubscribe@googlegroups.com.
To post to this group, send email to google-appengine@googlegroups.com.
Visit this group at https://groups.google.com/group/google-appengine.
To view this discussion on the web visit https://groups.google.com/d/msgid/google-appengine/8f11c1c8-9934-4692-93dc-2847d71afa9d%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

Randee Frankel

Technology Program Specialist

School District of Philadelphia

Educational Technology

Apple Teacher

Any questions about the Student Information System (SIS)?

Please call the I.T. Help Desk at 215-400-5555 or email at help...@philasd.org.

Visit sishelp.philasd.org for training resources.

Click here to tell us how we are doing!

The date for the 2018 Technology Expo and Competition is Friday, March 23, 2018. Registration for this year's Tech Expo and Competition is now open. To register your students and learn more, please visit our website at https://tinyurl.com/phila-tec.

[Jordan (Cloud Platform Support)]

Randee it looks as though you are attempting to seek support for a separate issue. For free technical support you can use the supported Cloud tags on Stack Exchange. For all other support options you can view the Google Cloud Support page. 

[Randee Frankel]

Thank you for letting me know. The error actually says contact the owner of the file and tells me what to tell them. Seeing as how that owner is Google I am not sure how to do that. I think this may be more trouble than it is worth. I do not think it is something I will share with my school district or train them to use. Thanks again.

On Tue, May 1, 2018 at 12:30 PM, 'Jordan (Cloud Platform Support)' via Google App Engine <google-a...@googlegroups.com> wrote:

Randee it looks as though you are attempting to seek support for a separate issue. For free technical support you can use the supported Cloud tags on Stack Exchange. For all other support options you can view the Google Cloud Support page. 

--

You received this message because you are subscribed to the Google Groups "Google App Engine" group.
To unsubscribe from this group and stop receiving emails from it, send an email to google-appengine+unsubscribe@googlegroups.com.
To post to this group, send email to google-appengine@googlegroups.com.
Visit this group at https://groups.google.com/group/google-appengine.

To view this discussion on the web visit https://groups.google.com/d/msgid/google-appengine/163a4e28-f74a-44b0-a386-861a0b385767%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

[Kyle Finley]

Tapir & Jordan,

For what it's worth, I've notice the same issue. I spoke with billing in the beginning of April, but at the time they weren't aware of the issue, and since it was only a few dollars I didn't pursue it further. If your interesting in investigating, here is some information that may be useful.

- The issue seems to have started on March 6th

- For me, it affected 3 low traffic apps (services).

- The apps had previously managed with one instance, and had never or very rarely gone over the 28hrs free quota

- No recent setting or code changes. Months or years since last updated

- All apps where using the Go runtime

May be related:

- There was another billing issue that affect 2 of the 3 around the same time. The issue involved instance hours being tracked incorrectly (attributed to the wrong day). Billing proactively notified me, via email, and issued a refund for that issue. According to billing that issue was isolated to one day, but I saw uncharacteristically high quota usage for multiple days that month.

- There is a UI bug in the quota table view where quota is reported as $0 daily while still incurring charges for the day (report to billing).

I suspect the issue is still ongoing. I made changes to 2 of the 3 to eliminate instance usage, but the third still seems to be requiring increased instance. Since the scheduler behavior is somewhat opaque it's difficult to determine if this is a scheduler issue or caused by increased traffic, however.

Sorry if this isn't the correct place to post this, I just wanted to give you an additional datapoint in case it's helpful. 

[Tapir]

On Tuesday, May 1, 2018 at 12:04:04 PM UTC-4, Jordan (Cloud Platform Support) wrote:

You should investigate how many instances each of your applications used in March and compare that with the amount in April. If you changed your applications' scaling settings it would have directly affected instance count, and in turn the amount of instance hours you use. 

I haven't changed the settings for years.
 

It is also recommended to make sure you are stopping previous versions of your applications in the Cloud Console, as older versions can still be serving with instances.

I really stopped the old version instances when each time I updated my app.

And I have impression that old running instances will not be charged if it doesn't handle any requests any more.
 

[Tapir]

On Tuesday, May 1, 2018 at 2:03:04 PM UTC-4, Kyle Finley wrote:

Tapir & Jordan,

For what it's worth, I've notice the same issue. I spoke with billing in the beginning of April, but at the time they weren't aware of the issue, and since it was only a few dollars I didn't pursue it further. If your interesting in investigating, here is some information that may be useful.

- The issue seems to have started on March 6th

- For me, it affected 3 low traffic apps (services).

- The apps had previously managed with one instance, and had never or very rarely gone over the 28hrs free quota

- No recent setting or code changes. Months or years since last updated

- All apps where using the Go runtime

Similar phenomenons.
Yes, the charge for March is about 3 times as previous months.
I contacts Google support, but they said it may caused by a bug and will refund me.

One of my apps is Java runtime.
 

May be related:

- There was another billing issue that affect 2 of the 3 around the same time. The issue involved instance hours being tracked incorrectly (attributed to the wrong day). Billing proactively notified me, via email, and issued a refund for that issue. According to billing that issue was isolated to one day, but I saw uncharacteristically high quota usage for multiple days that month.

- There is a UI bug in the quota table view where quota is reported as $0 daily while still incurring charges for the day (report to billing).

Yes, I also ever observed this UI bug.
 

[Steren Giannini]

Hi,

I am a product manager on the App Engine team.

We indeed rolled out a new scheduler over the last months. This happened gradually across all regions and all apps, and we closely monitored the rollout using performance and billing metrics. Within a few weeks, we will be publishing this change in the App Engine release notes, as well as new documentation and a blog post.

Our tests show that it delivers better scaling performance and more efficient resource consumption, and on average lower costs for our users.

But as you pointed out, for very small applications, it can happen that the cost increases.

We introduced new automatic scaling parameters (in the automatic_scaling section of app.yaml or appengine-web.xml) to control the scaling behavior. I invite you to tweaks those parameters for your application:

- "target_cpu_utilization": represents the CPU utilization ratio threshold used to determine if the number of instances should be scaled up or down. This is probably the parameter you should tweak to optimize between performance and costs. It take values between 0 and 1, 0.6 being the default. A higher value is likely to reduce the number of instances scheduled, but might have an impact on your application tail latency. 

- "target_throughput_utilization":  Specify a value from 0.5 to 0.95. Used with max_concurrent_requests to specify when a new instance is started due to concurrent requests. When the number of concurrent requests reaches a value equal to max_concurrent_requests times target_throughput_utilization, the scheduler starts a new instance.

- "max_instances": allows you to cap the total number of instances to be scheduled. You can use this new parameter as a safeguard. This new parameter is still rolling out and might not be available today.

We will publish very soon these paramters in our documentation

I hope this helps.

Steren

Product Manager

Google Cloud Platform

[Attila-Mihaly Balazs]

Hi Steren,

Could you please confirm that this is right? You introduced the "max_instances" parameter for the "automatic_scaling" section of the app.yaml for AppEngine *standard*? (because the docs are not yet update).

If so, thank you!! This solves a potential DoS issue for us where too many frontend instances could exhaust the connection limit to CloudSQL

Attila

[Steren Giannini]

This is correct, we introduced the "max_instances" parameter for the "automatic_scaling" section of the app.yaml for AppEngine *standard*.

This change might still not be rolled out to your app, but feel free to give it a try.

--

You received this message because you are subscribed to the Google Groups "Google App Engine" group.

To unsubscribe from this group and stop receiving emails from it, send an email to google-appengi...@googlegroups.com.
To post to this group, send email to google-a...@googlegroups.com.

Visit this group at https://groups.google.com/group/google-appengine.

To view this discussion on the web visit https://groups.google.com/d/msgid/google-appengine/727c9ea3-ad57-425c-9609-1946ada2bc47%40googlegroups.com.

[Nickolas Daskalou]

Our bills increased 30-40% in March and April compared with previous months, e.g. for one of our projects:

We were definitely surprised by the big increase and planned on investigating soon. The new scheduler could be the culprit.

Who can we ping at Google to look into this further?

Nick

On 8 May 2018 at 14:26, 'Steren Giannini' via Google App Engine <google-a...@googlegroups.com> wrote:

This is correct, we introduced the "max_instances" parameter for the "automatic_scaling" section of the app.yaml for AppEngine *standard*.

This change might still not be rolled out to your app, but feel free to give it a try.

On Mon, May 7, 2018 at 9:09 PM Attila-Mihaly Balazs <dify...@gmail.com> wrote:

Hi Steren,

Could you please confirm that this is right? You introduced the "max_instances" parameter for the "automatic_scaling" section of the app.yaml for AppEngine *standard*? (because the docs are not yet update).

If so, thank you!! This solves a potential DoS issue for us where too many frontend instances could exhaust the connection limit to CloudSQL

Attila

--
You received this message because you are subscribed to the Google Groups "Google App Engine" group.

To unsubscribe from this group and stop receiving emails from it, send an email to google-appengine+unsubscribe@googlegroups.com.
To post to this group, send email to google-appengine@googlegroups.com.

Visit this group at https://groups.google.com/group/google-appengine.
To view this discussion on the web visit https://groups.google.com/d/msgid/google-appengine/727c9ea3-ad57-425c-9609-1946ada2bc47%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

You received this message because you are subscribed to the Google Groups "Google App Engine" group.

To unsubscribe from this group and stop receiving emails from it, send an email to google-appengine+unsubscribe@googlegroups.com.
To post to this group, send email to google-appengine@googlegroups.com.

Visit this group at https://groups.google.com/group/google-appengine.

To view this discussion on the web visit https://groups.google.com/d/msgid/google-appengine/CANHKq5FxUv28HBcPnx4qL_mfiXKxuCOMvq%2BvhTtskzs227zCPw%40mail.gmail.com.

[Jordan (Cloud Platform Support)]

As previously mentioned, Google Groups is reserved for general product discussions and not for billing support. It is therefore recommended to open a free Billing Support case for any billing related inquiries. 

[Vitaly Bogomolov]

Hi, Steren.

What is default value for new "target_throughput_utilization" parameter? 

WBR, Vitaly.

[Jordan (Cloud Platform Support)]

I believe "target_throughput_utilization" is defaulted to the value of '0.6'. But as Steren mentioned, all of this information will be made available in the official documentation once the release has completed (as these properties might not even be available currently).