Scaling - Can we disable the automatic termination of an instance?

[Brandon Newport]

Is there a way to disable the automatic termination of an Instance with Scaling function in the Farm Role settings.

We currently have a Community version of Scalr deployed, and connected to OpenStack.  By default when an instance goes into ERROR state in openstack.  Scalr will terminate the existing instance and deploy a new one.  Is there a way to disable this or ignore the instance in error state?  We have tested the manual Scaling, but it will still terminate the instance and wait for you to redeploy the new instance.

We are gradually working our way to using scaling the way it is designed but we are just not there yet.

Any help here would be appreciated.

Brandon

[Brandon Newport]

We are using this version of Scalr, Version 5.11.22.

[Marc O'Brien]

Hi Brandon,

This is expected behavior as Scalr implements a Replace vs a Repair methodology to resource management.  You may wish to merge "scalr.openstack.action_on_missing_server = ignore" in to your scalr-server.rb to prevent Scalr from terminating instances that are marked as "Missing" but there is not currently a method to completely prevent termination of resources that enter an "Error" state.  This is intentional, but we do understand your perspective/situation and have considered this previously.  We are exploring possible changes to this behavior in future versions of Enterprise Scalr, but this is currently a fundamental behavior of our Desired State Engine and is not something that can be disabled.

Many thanks,
Wm. Marc O'Brien
Scalr Technical Support

[Marc O'Brien]

Hi Brandon,

To clarify, the ignore on missing setting will not impact servers in Error state so this does not directly apply to this case as you stated these instances are in Error state.  Could you provide some additional details on why you would not want to terminate servers in Error state?  Do you have a specific use case for this?

Many thanks,
Wm. Marc O'Brien
Scalr Technical Support

[Brandon Newport]

We had an incident a few weeks ago with a Farm that was deployed.  It lost its DHCP address for some reason and while we were troubleshooting, we attempted a live migration from one host to another in Openstack, when this happened it forced the instance into Error state, which terminated the instance.

Since then we have been testing Scalr actions by forcing instances into error state in OpenStack.

However when we were testing today we forced a Hypervisor to shut  down.  Which did nothing to the Instance, until the Hypervisor was powered on, changed the Instance to shut off state.  This forced the Replace.

I hope this helped answer your question.

Brandon

[Marc O'Brien]

Hi Brandon,

Thank you for the details.  This helps us to better understand our customer's needs and prioritize future development efforts.

By the way, welcome to the Scalr Open Source community! We're excited to have you join us.  Please feel free to post any issues or questions you have moving forward and we will do our best to lend a hand.

Cheers,

Wm. Marc O'Brien
Scalr Technical Support

[Enrico Kern]

Hello all,

was there something implemented already? we have the same issues with the latest open source scalr. Sometimes on hypervisor reboot openstack reports error state or cant boot up vms for whatever reasons and then scalr packs the hammer and terminates the instance. Would love to only allow manual termination in whatever state.

[Enrico Kern]

ok nvm. i patched CloudPoller.php myself todo that. If anyone is interessted. just add a continue; from line 347 on in /opt/scalr-server/embedded/scalr/app/src/Scalr/System/Zmq/Cron/Task/CloudPoller.php

 example:

                        // If openstack server is in ERROR state we need more details

                        if ($openstackErrorState) {

                            try {

                                $info = $p->GetServerExtendedInformation($DBServer);

                                $status = empty($info['Status']) ? false : $info['Status'];

                            } catch (Exception $e) {}

                            // added by Enrico. ignore error (i hope)

                            $this->getLogger()->error("openstack error and i want to terminate it... oh wait i better let it alone");

                            continue;

[Brandon Newport]

I tested this patch in our environment and it worked.  Scalr ignored the openstack error state.

Thanks for the fix, this was a big help.

[Enrico Kern]

just dont forget todo it again with the next OSS update (if the on_error config parameter doesnt get in for whatever reasons). :)