Anyone written their own automatic scaling (flexible or standard)?
52 views
Skip to first unread message
Evan Jones
May 2, 2016, 9:51:42 AM5/2/16
to Google App Engine
We are considering implementing our own automatic scaling policy by periodically monitoring our service and calling the SetNumModules() API that App Engine provides. We want to work around the bug that when you deploy to an automatically scaled Flexible Environment service, the current policy switches your production load on to your minimum number of instances, rather than scaling up the new version to match the old version (issue filed):
Before we embark on this bug workaround adventure: has anyone successfully written their own automatic scaling policy, for any reason? Was it easy or hard? Are there any bugs for us to be aware of?
Thanks!
Nick (Cloud Platform Support)
May 2, 2016, 4:04:01 PM5/2/16
to Google App Engine
Hey Evan,
I can't necessarily relate any experience with writing a custom auto-scaler, but I have some pieces of advice which might help with broad overview of possibilities.
One service to look into leveraging is Cloud Monitoring, which allows you to monitor all kinds of metrics related to your application, from CPU usage to request latency (using custom metrics). You can use webhook notifications to notify your control instance(s) to spin up / down instances according to upper / lower bounds on relevant metrics. There's also the possibility of using the Compute Engine Autoscaler to manage the instances by grouping them, which also has support for use of custom metrics.
Overall, the more granular your desired control of scaling, the closer you get to writing entirely custom control logic in an instance, although there are several scaling / monitoring solutions which exist already. Finally, your Feature Request will be responded-to shortly and if implemented, would make much of this superfluous, but you might quite enjoy having the sort of granular control that you described coding. The main problems to be solved are accurate monitoring / projection of available resources vs resources required, and the technical question of how to implement control of scaling in line with that.
I hope this has helped turn some wheels, let me know if you have any further questions, and hopefully others with more practical experience could chime in?
Regards,
Nick
Cloud Platform Community Support
I can't necessarily relate any experience with writing a custom auto-scaler, but I have some pieces of advice which might help with broad overview of possibilities.
One service to look into leveraging is Cloud Monitoring, which allows you to monitor all kinds of metrics related to your application, from CPU usage to request latency (using custom metrics). You can use webhook notifications to notify your control instance(s) to spin up / down instances according to upper / lower bounds on relevant metrics. There's also the possibility of using the Compute Engine Autoscaler to manage the instances by grouping them, which also has support for use of custom metrics.
Overall, the more granular your desired control of scaling, the closer you get to writing entirely custom control logic in an instance, although there are several scaling / monitoring solutions which exist already. Finally, your Feature Request will be responded-to shortly and if implemented, would make much of this superfluous, but you might quite enjoy having the sort of granular control that you described coding. The main problems to be solved are accurate monitoring / projection of available resources vs resources required, and the technical question of how to implement control of scaling in line with that.
I hope this has helped turn some wheels, let me know if you have any further questions, and hopefully others with more practical experience could chime in?
Regards,
Nick
Cloud Platform Community Support
Reply all
Reply to author
Forward
0 new messages