Need a practical approach to shut down Cloud Foundry at night

[maartens...@opencredo.com]

Folks,

Our client has the requirement to shutdown their Cloud Foundry platform at night in order to save money (in the production environment as well). One option we looked at was to:

In the evening:

1. Shut down all CF jobs in a sensible order (“bosh stop <job>”)
2. Power off all the VMs via BOSH (“bosh stop <job> —hard”)

In the morning:

3. Start the VMs up one by one
4. Start the jobs in a sensible order (if they hadn’t already started when the VMs powered up)

Unfortunately, step 2 causes BOSH to try to refill the resource pool to replace the VM it has just powered down. This resulted in the deployment becoming unstable and the solution had to be abandoned.

Does anyone know of a good approach to fulfill the requirement? Our next option is to do step 2 and 3 on the IAAS layer rather than via BOSH, but we haven’t had a chance to test it out yet.

Regards,
Maartens

[James Bayer]

which IaaS is this? just thinking through this in my head, as long as the BOSH Resurrector is turned off, then you should be able to stop the VMs in the IaaS layer (make sure that the Health Monitor isn't configured to alert you to death).

when starting back up, i think a "bosh cck" will go through things in a similar order as a deployment. one thing i'm not certain about is whether the cck will handle the persistent disk management correctly if the previous VMs are still around.

To unsubscribe from this group and stop receiving emails from it, send an email to bosh-users+...@cloudfoundry.org.

--

Thank you,

James Bayer

[Maartens Lourens]

Hi James,

It's VCloud.

Thanks, we will give bosh cck a try as well. When we tried it with the previous proposed solution the result was not as expected, but it's worth another try.

Regards,

Maartens

[Greg Oehmen]

I believe James has a valid point about persistent disk handling.  The BOSH team has a story currently in-flight [0] to resolve this issue.  The use case for the story pertains more to a casandra scenario where one node in a cluster is going to be bounced.  Rather than redistributing ~1tb (or whatever) of data back out to the remaining nodes in the cluster, the persistent disk will be detached and reattached after the bounce.  The difference here with Maarten's ask is the length of time difference of the 'bounce' - few minutes vs many hours, but it should be the right path.

[0] https://www.pivotaltracker.com/story/show/67329942

Best

Greg

Greg Oehmen

Cloud Foundry Product Manager - Bosh

Pivotal

[co...@cloudcredo.com]

Hi Maartens,

We've used a night manifest and a day manifest with some clients before. A cron-like scheduler then does a simple `bosh deployment {day,night}-scale.yml; bosh deploy` at the right time.

This takes care of the ordering, and is the cleanest way to achieve the desired solution. Maintaining state correctly on persistent disks will be the biggest problem - reducing instances of data nodes to 0 is probably not a good idea in most cases.

Colin

[Maartens Lourens]

Thanks Colin,
That's an interesting option we might give a try in future. For present purposes a simple shutdown might prove to be enough for us.

Maartens