Making ironfan-serengeti work with ironfan-homebase

[Ryan Grenz]

Hi all,

Firstly, thanks for all your efforts so far with Serengeti, its great to see VMware on board and developing alongside infochimps on what looks like a great framework. 

I am looking forward to seeing the fruits of your efforts merged into Infochimps' ironfan (assuming thats whats going to happen in the future?)

In the meantime, I've been tasked with trying to get the latest ironfan-homebase working with the ironfan serengeti gem, so I can demonstrate something working end to end (ie. all the way into vsphere) to my devops colleagues here.

I built the fog and cloud_manager gems, which were dependencies of the serengeti fork of ironfan.

However, when I modify the Gemfile of the ironfan-homebase to depend on the ironfan serengeti gem I'm met with Gem dependency issues with json and chef.

I'm trying to go thru all the Gemfiles to find appropriate workable dependencies for those 2 gems, but am finding it a bit tricky so far.

My question is - how progressed is the migration work from the serengeti fork of ironfan v3 into the main ironfan v4 repo so far? I read on a blog post (http://blog.infochimps.com/2012/11/02/announcing_ironfan_v4/) that its something that VMware/Infochimps are intending to do.

The reason I ask is I'm unsure whether its worth just waiting for the work to be completed, or if there is any way I might be able to help towards its completion?

Thanks,

Ryan

[Jesse Hu]

Hi Ryan,

VMware's Serengeti Ironfan is not fully compatible with the cookbooks in Infochimps' ironfan-homebase (https://github.com/infochimps-labs/ironfan-pantry). If you want to provision a Hadoop (with Pig and Hive), Zookeeper, HBase cluster,  please use Serengeti; If you want to provision other cluster (like Cassendra, etc), you can try to copy the cookbooks in ironfan-homebase into Serengeti Server, upload the cookbooks, and follow Serengeti guide to provision any cluster you want.

Currently we don't have a concrete plan to migrate our Ironfan V3 based code to Ironfan V4, as well as Serengeti Ironfan cookbooks.

Thanks,
Hui Hu

在 2013年1月9日星期三UTC+8上午5时48分20秒，Ryan Grenz写道：

[Ryan Grenz]

Ahh ok maybe I misread the blog post - so Infochimps are not currently intending to pick up your vsphere work into their ironfan v4, but the other way around (although no concrete plans yet)?

Yes, the goal for me is to proof of concept using Ironfan to deploy Cloud Foundry and other clustered services (eg. mysql/cassandra) to multiple environments in vsphere or vagrant for local testing.

I did look at BOSH a couple of months ago, but felt it was a bit immature at the time, and the Chef approach seemed much more appropriate for us.

Ironfan on the other hand seems very much like what I've been looking for.

I will take a look at how Serengeti Server can work for us in the way that you suggested and see how I get on that way.

Thanks,

Ryan

[Jesse Hu]

I think Serengeti is fit for your requirement since you want to deploy the cluster in vSphere.

ironfan-homebase cookbooks and Serengeti cookbooks have the following significant difference :

1) the cluster discovery cookbook in Serengeti is  cluster_service_discovery https://github.com/vmware-serengeti/serengeti-pantry/tree/master/cookbooks/cluster_service_discovery,  while it's silverware in Ironfan-homebase https://github.com/infochimps-labs/ironfan-pantry/tree/master/cookbooks/silverware .  cluster_service_discovery is the older version of silverware, and they have very similar API. We have done a few enhancement based on original cluster_service_discovery.   If mysql/cassendra cookbook uses silverware, you can try to replace the call to silverware API with similar cluster_service_discovery API.

2) some ironfan-homebase cookbooks is written for EC2 (ubuntu only), while Serengeti cookbook uses CentOS 5.6. So you might need to modify the cookbook to make it work for CentOS.

Please let us know if you have any question when deploying mysql/cassendra cluster in Serengeti.  We had done a prototype of using Serengeti to deploy a  Servlet based Java app in Tomcat and nginx.

Thanks,
Hui Hu

在 2013年1月9日星期三UTC+8下午10时44分14秒，Ryan Grenz写道：

[fl...@infochimps.com]

On Wednesday, January 9, 2013 8:44:14 AM UTC-6, Ryan Grenz wrote:

Ahh ok maybe I misread the blog post - so Infochimps are not currently intending to pick up your vsphere work into their ironfan v4, but the other way around (although no concrete plans yet)?

We refactored the cloud provider interface to be pluggable in v4 to make integrations with eg VSphere or Cloud Foundry easier, but unfortunately until now neither side has had the roadmap bandwidth to re-merge the trees. 

However! It looks like we'll be able to throw some bandwidth at targeting VSphere in the next month, assuming we can find a good path towards re-integrating the trees. So no concrete yet, but concrete trucks and burly-looking folks with shovels have shown up.

The Hadoop recipes in v4 are /significantly/ improved, and now support for auto-wired monitoring, logging and so forth; HBase, Hive, Hue, Pig (and soon Impala!) are production-hard.

 

Yes, the goal for me is to proof of concept using Ironfan to deploy Cloud Foundry and other clustered services (eg. mysql/cassandra) to multiple environments in vsphere or vagrant for local testing.

I did look at BOSH a couple of months ago, but felt it was a bit immature at the time, and the Chef approach seemed much more appropriate for us.

Ironfan on the other hand seems very much like what I've been looking for.

I'd love to chat further about how to collaborate; I will be in touch separately. 

flip

[fl...@infochimps.com]

One more piece of good news: the ironfan-pantry cookbooks work on CentOS, Ubuntu and in many cases on OSX.

flip

[Jesse Hu]

Hi Flip,  Thanks for the valuable feedback from Infochimps side.

在 2013年1月11日星期五UTC+8下午3时47分24秒，fl...@infochimps.com写道：

[Ryan Grenz]

Hi Flip - nice to hear from you and thanks for your reply.

Encouraging news indeed :)

I'd also really like to see the ability to reproduce/test clusters locally again; ie. Vagrant or Virtualbox, although I believe this is something you're reintroducing in ironfan v4 soon anyway? For now, I've been trying an earlier version of Ironfan with an earlier ironfan-homebase, to see if I can get Vagrant support working.

My Ruby experience isn't lengthy but I'm getting there, and am interested in trying to help wherever I can if you wouldn't mind helping direct me a little bit. I could also blog about my experiences, and help update any documentation if it would help? 

Sorry if this is slipping off topic for the serengeti-dev list - Flip, I couldn't find an ironfan-dev list .. do you have anywhere else to continue Ironfan specific discussions?

Look forward to hearing from you.

Ryan

[Nathaniel Eliot]

Ryan:

For further discussion of Ironfan itself, please join to https://groups.google.com/a/infochimps.com/forum/?fromgroups#!forum/ironfan-users (ironfa...@infochimps.com).

(Following up on more Ironfan specific questions in private email.)

Cheers,

Nathan