Simple redis service built on bosh ($ bosh create redis)

[Dr Nic Williams]

I need a simple way to create and delete (and ultimate upgrade) dedicate services - redis, postgresql, etc. In this post I introduce a simple CLI for creating a Redis server, getting the URI for the service, binding it to your Cloud Foundry app, and later deleting it.

Here is an example scenario for installing the plugin, login to bosh (ask your sysadmin for credentials), uploading the redis release to bosh, and then creating and deleting as many redis servers as you and your team need.

$ gem install bosh_cli "~> 1.5.0.pre" --source https://s3.amazonaws.com/bosh-jenkins-gems/ 
$ gem install bosh_cli_plugin_redis

$ bosh prepare redis
$ bosh create redis
$ bosh show redis uri
redis://:c1da04...@0.redis.default.demoredis.microbosh:6379/0
$ cf set-env myapp REDIS_URI redis://:c1da04...@0.redis.default.redis-123.microbosh:6379/0

$ cf unset-env myapp REDIS_URI
$ bosh delete redis




The blog post with known limitations & how-tos is at http://starkandwayne.com/articles/2013/07/08/simple-redis-service/

The source for the CLI plugin is at https://github.com/drnic/bosh_cli_plugin_redis

The source for running redis upon bosh is https://github.com/cloudfoundry-community/redis-boshrelease

Does this UX seem nice to you?

--

Dr Nic Williams

Stark & Wayne LLC - consultancy for Cloud Foundry users

http://drnicwilliams.com

http://starkandwayne.com

cell +1 (415) 860-2185

twitter @drnic

[Dr Nic Williams]

If you try this out and dig around the code, you'll notice there is no "git clone" for the redis bosh release. All the bits necessary to upload the bosh release are stored in the rubygem that you download above.

[ruben....@innovationfactory.eu]

why not use services for this?

---

Have an innovative day

Innovation Factory   De Lairessestraat 180   1075 HM Amsterdam   +31 20 7787008   www.innovationfactory.eu

Disclaimer
The information transmitted is intended only for the person or entity to which it is addressed and may contain confidential and/or privileged material. Any use of, or taking of any action in reliance upon, this information by persons or entities other than the intended recipient is prohibited. If you received this in error, please contact the sender and delete the material from any computer. Innovation Factory does not accept liability for any errors, viruses or omissions in the contents of this message, which may arise as a result of e-mail transmission. No employee or agent is authorized to conclude any binding agreement on behalf of Innovation Factory with another party by email.

[Dr Nic Williams]

Good question Ruben.

Ultimately I want to use the services API again. I decided to release the bosh part of it asap.

​

Perhaps a service gateway could do the job that this plugin does - to provision a service, delete a service and change a service (no API for this at the moment afaik).

​

I wanted to experiment with pushing behaviour down towards bosh (it has scaling and disk management and snapshots and backups).

I also wanted to experiment with how to make a bosh release easier to consume (redis-boshrelease).

​

I also wanted to try out the "bosh diff" temp laying system; and is what it's pros/cons are compared to building up a hash and to_yaml'ing it (like I did in bosh-cloudfoundry). 

​

I don't understand warden yet, how to debug it, and how it's being wielded by vcap-services-base.

​

I am uncomfortable with over investing my time (and clients' money) in the existing services that Pivotal (the main investor in contributions to CF and the only people with commit access) have recently executed their promise to "un-support".

They have started internal discussions on a v3 API. From chats with them, their plans for v3 services do not go into the realms of functionality that I think are necessary (using bosh as part of the service; not just to deploy the running pieces), so I wanted to start experimenting towards the services UX that I think we need (something user facing like Heroku Postgres).

​

I also want to keep the implementation of the core service (Redis in this case) isolated from the orchestration and management (the bosh plugin). I found it very hard to understand the relationships between vcap-services-base, cf-services-release, warden and the actual postgresql service itself. I could not just drop it all into cf-vagrant-installer and develop on it. It was all tangled up together.

​

Finally, I just needed to get a Redis UX that worked for a client asap. They can't run apps on their shiny CF deployment that need Redis if there isn't a way to get Redis. So I first deployed Redis-boshrelease manually; then wrote this nice bosh plugin to simplify it over the weekend.

​

Next I might wrote another plugin to simplify cf-release v2 to replace bosh-cloudfoundry (and hopefully the AWS-VPC-only one-size only bootstrapper we have now). Oooh yeah.

--
Dr Nic Williams
Stark & Wayne LLC - the consultancy for Cloud Foundry
http://starkandwayne.com
+1 415 860 2185
twitter: drnic

[ruben....@innovationfactory.eu]

Thanks for the detailed explanation. 

So the idea is to have a dedicated instance per service?

Isn't this the opposit of what cloud foundry tries to do?.

Maybe there are some usecases in which cloudfoundry is used to horizontally scale 1 application which has to handle a big load.

I could see why you would want to use this approach. But this tool would than also need to support clustering of the service.

However for the plugin aproach would not be usefull in a multi tenant setup.

I'm also not yet sure of the create a bosh plugin for every thing approach.

It feels really limited in its use. By adding an extra layer of complexity you scare of potential contributors.

Also manifests are really deployment specific and there for not a good candidate for automation.

Its however great that you are experimenting with services because they definitely need some love.

Now that pivotal has gone the closed source marketplace route.

Kind regards,

Ruben 

[Dr Nic Williams]

The idea of that plugin was to make deploying the redis-boshrelease super easy. The bosh release happens to only support one redis-server per instance. The v1 release of it also only supports single VM redis. Martin did start (finish?) support for sentinal; but there isn't a v2 final release with that in it yet; so therefore the CLI can't support it either.

Once the vcap-services releases are working, I'd love to help create a CLI for people to use those releases too (and bundle up the final release themselves so they don't have to "git clone" to use bosh releases).

bosh_cli_plugin_redis was a two day spike over a weekend whilst I was supposed to be on holidays. No tests & no reusable classes.

The next iteration was bosh-cloudfoundry. It takes the ideas and starts to flesh out reusable classes, project structure (like templates/v132/aws/medium/deployment_file.yml.erb). Its all in one project; but hopefully its moving towards a framework project that can make it easy for create a CLI (with versioned/CPI-specific templates) for any bosh release.

Then I can rewrite bosh_cli_plugin_redis again :)

I have to disagree with the idea that there is an "extra layer of complexity" here that's bad - the model in my head is that of rubygems vs ruby projects. A ruby project is a repo with ruby code in it. I can optionally export one or more .gems (rails & bosh both release lots of .gem instead of a single .gem for the whole project).

But we'd agree that rubygems has been NET good for the ruby universe. Versioning, easy installation of libraries, easy installation of executables (even works on Windows), etc. You can use a rubygem without having to "git clone" the original project. You can use known working versions (with known bugs) rather than have master/HEAD work/break/work/break in different ways each week.

Currently bosh & a bosh release are relatively unusable. For example, I don't think there is a single human being on the planet that could go to https://github.com/cloudfoundry/cf-release#readme and deploy Cloud Foundry. So currently without an extra layer upon these constructs I don't see a bright future for bosh releases. Something needs to make them easier to "use" for people who just want to use the service/system they are describing.

If I build a few examples perhaps then other people will start to have different (better?) opinions of how this could look and feel. At the moment there are so few of us in this community that we don't have a healthy set of diverse opinions & contributions (IMO). I build stuff, Pivotal builds stuff, you build stuff. Hopefully later in the year we can look across it and figure out what's the most awesome user experience for the next 10,000 bosh users.

Everyone will just know you go:

$ bosh list public releases

$ bosh install release cloudfoundry/cf

Installing plugin...

Installing release into bosh...

$ bosh create cf --size production --dns apps.megacorp.com --ip 1.2.3.4,1.2.3.5

...

And Cloud Foundry (or whataver system) will have been introduced into their company.

To quote the late, great Senator Palpatine, "and then we'll have peace".

Nic