Cluster Chef backports from Infochimps production branch

[Philip (flip) Kromer]

Cookbooks now track between master and version_2:

I pulled non-conflicting changes from master into version_2 and version_2 into master. 

Here are the few remaining cookbooks that are different between the branches: 

• hadoop_cluster,hbase,hive,cloudera_hue. At some point in the future the versions in master will replace those in version_2. We will try to ensure stability for people using version_2 copies of those cookbooks.
• big_package: these are probably better in version_2 and I'll sort them out at some point.
• elasticsearch: one uses cluster discovery, one uses explicit seeds. You're working with these Nathan, go with the one you think makes more sense.
• redis -- see following email
• zookeeper -- see following email
• cookbooks/java, cookbooks/ubuntu -- the version_2 cookbook forces sun java, which is GOOD. openjdk is an infestation. I didn't pull this in to master, because it should be done thoughtfully.

Here are the changes I made to each branch. To version_2:

• new to the version_2 branch, pulled in wholesale from master: apache2, graphite, nagios, nodejs, ntp, python, statds
• fixes to cassandra and thrift from Mike Heffner.
• Pulled in from the production versions in master, patches applied cleanly: flume, zookeeper, pig, jruby

To master:

• backported fixes from the opscode-master for cookbooks/{aws,nfs,runit,thrift}
• nuked some bad symlinks
• fixes to the cassandra recipes
• fix to nfs recipe for NFS servers under ubuntu maverick
• new to the master branch, pulled in wholesale from version_2: Rstats, motd/...node_name
• a one-line nitpick in hbase.

site-cookbooks vs cookbooks:

I pulled apache2, ntp, python into *site-cookbooks* because that's where it was in master. I think many of these are unmolested copies from the opscode-cookbooks versions -- Nathan, would you please relocate them into cookbooks and put only the overrides if any in site-cookbooks?

On Sat, May 14, 2011 at 3:51 PM, Philip (flip) Kromer <fl...@infochimps.com> wrote:

Sounds good.

Would rather use standardized cluster_chef, not clusterchef and ditch the prefix:

node[:cluster_chef][:cluster]

node[:cluster_chef][:facet]
node[:cluster_chef][:index]

On Sat, May 14, 2011 at 9:01 AM, Chris Howe <how...@infochimps.com> wrote:

Various of the requirement for cluster chef make point toward having a way to glean from the chef node data exactly which nodes belong to a particular cluster (and facet).  This is easy enough to do if you iterate through all of the nodes and filter on the cluster_name and cluster_role attributes:

chris@basqueseed:/usr/lib/ruby/gems/1.8/gems/chef-0.9.14/lib/chef/knife$ knife exec -E 'nodes.all {|n| next unless n.cluster_name == "greeneggs"; puts n.node_name}'

greeneggs-delta-0

greeneggs-delta-1

greeneggs-delta-2

greeneggs-delta-3

greeneggs-delta-4

greeneggs-beta

greeneggs-gamma-0

greeneggs-gamma-1

greeneggs-delta-5

greeneggs-gamma-2

greeneggs-delta-6

greeneggs-alpha

chris@basqueseed:/usr/lib/ruby/gems/1.8/gems/chef-0.9.14/lib/chef/knife$ 

Iterating through all of the nodes is a little slow, so I would really like to use searching instead. The problem with this is that for searching, chef indexes all of the nested attributes as if they were top level attributes, making it easy to search for values buried deep within the node structure. This feature frustrates using search as a cluster discovery because of some unfortunate choices of other configuration environments:

chris@basqueseed:~/.ssh$ knife exec -E 'nodes.search("cluster_name:greeneggs") {|n| puts n.node_name}' | sort

chimpmark-master-0

chimpmark-slave-0

chimpmark-slave-1

chimpmark-slave-10

chimpmark-slave-11

chimpmark-slave-12

chimpmark-slave-13

chimpmark-slave-14

chimpmark-slave-15

chimpmark-slave-16

chimpmark-slave-17

chimpmark-slave-18

chimpmark-slave-2

chimpmark-slave-3

chimpmark-slave-4

chimpmark-slave-5

chimpmark-slave-6

chimpmark-slave-7

chimpmark-slave-8

chimpmark-slave-9

goldencap-nikko-0

goldencap-twscraper-0

goldencap-twscraper-1

goldencap-twstream-0

greeneggs-alpha

greeneggs-beta

greeneggs-delta-0

greeneggs-delta-1

greeneggs-delta-2

greeneggs-delta-3

greeneggs-delta-4

greeneggs-delta-5

greeneggs-delta-6

greeneggs-gamma-0

greeneggs-gamma-1

greeneggs-gamma-2

What happened? goldencap and chimpmark are configured to talk to the hbase residing on greeneggs by the hbase.cluster_name attribute. Maybe this was a poor choice of attributes to use to indicate which hbase cluster to talk to (it was my choice, actually). But it brings up a general problem with chef search and attribute collisions that we need to be aware of.

Another feature of chef search is that the node attributes are also indexed by the full path. A nodes node.hbase.cluster_name attribute gets indexed as "cluster_name:<value>" and "hbase_cluster_name:value":

chris@basqueseed:~/.ssh$ knife exec -E 'nodes.search("hbase_cluster_name:greeneggs") {|n| puts "#{n.node_name} #{n.cluster_name} #{n.hbase.cluster_name}" }' | sort

chimpmark-master-0 chimpmark greeneggs

chimpmark-slave-0 chimpmark greeneggs

chimpmark-slave-10 chimpmark greeneggs

chimpmark-slave-11 chimpmark greeneggs

chimpmark-slave-12 chimpmark greeneggs

chimpmark-slave-13 chimpmark greeneggs

chimpmark-slave-14 chimpmark greeneggs

chimpmark-slave-15 chimpmark greeneggs

chimpmark-slave-16 chimpmark greeneggs

chimpmark-slave-17 chimpmark greeneggs

chimpmark-slave-18 chimpmark greeneggs

chimpmark-slave-1 chimpmark greeneggs

chimpmark-slave-2 chimpmark greeneggs

chimpmark-slave-3 chimpmark greeneggs

chimpmark-slave-4 chimpmark greeneggs

chimpmark-slave-5 chimpmark greeneggs

chimpmark-slave-6 chimpmark greeneggs

chimpmark-slave-7 chimpmark greeneggs

chimpmark-slave-8 chimpmark greeneggs

chimpmark-slave-9 chimpmark greeneggs

goldencap-nikko-0 goldencap greeneggs

goldencap-twscraper-0 goldencap greeneggs

goldencap-twscraper-1 goldencap greeneggs

goldencap-twstream-0 goldencap greeneggs

greeneggs-alpha greeneggs greeneggs

greeneggs-beta greeneggs greeneggs

greeneggs-delta-0 greeneggs greeneggs

greeneggs-delta-1 greeneggs greeneggs

greeneggs-delta-2 greeneggs greeneggs

greeneggs-delta-3 greeneggs greeneggs

greeneggs-delta-4 greeneggs greeneggs

greeneggs-delta-5 greeneggs greeneggs

greeneggs-delta-6 greeneggs greeneggs

greeneggs-gamma-0 greeneggs greeneggs

greeneggs-gamma-1 greeneggs greeneggs

greeneggs-gamma-2 greeneggs greeneggs

I propose that cluster chef should mark nodes in an unequivocal way so that it can search for them and get the answer it expects. Basically, this means adding a top level "clusterchef" attribute that contains  cluster_name, cluster_facet, and cluster_facet_index. The top level "clusterchef" attribute gives us a unique namespace that we can use for the chef search interface. (note that the top level cluster_name attribute will still be there.)

--Chris  

--
infochimps.com - discover data

--
infochimps.com - discover data

[Philip (flip) Kromer]

I'm posting these as issues on the github repo -- please join the discussion there.

 https://github.com/infochimps/cluster_chef/issues

flip