Sharing information between systems

[Joe Topjian]

Here's a slide deck discussing various ways of sharing information between systems. Chef, Puppet, and etcd/confd are used as examples:

http://www.slideshare.net/m_richardson/node-collaboration-sharing-information-between-your-systems

This is an area that I'm very interested in at the moment. While we've had a lot of success with Puppet, I'm finding that it's difficult to use in environments that change frequently. Slide #43 describes the exact problems that I'm seeing.

The more I dig into this problem, the more I am realizing that there are, in fact, two layers of configuration management:

* The non-volatile layer: this contains configuration such as usernames, passwords, database names, website names, etc.

* The volatile layer: this has hostnames, IP addresses, cluster membership, etc

The slide deck uses etcd as an example of a volatile layer solution, but there are other emerging projects: 

* SmartStack

* Consul

The commonality between these projects is that they provide a "volatile layer" made up of an in-memory, distributed, key-value database that is frequently updated and polled. I think this is an awesome idea, but I have not fully accepted the fact that to use this layer effectively, I must give up all control of it and let the infrastructure itself populate and update it. Perhaps it's just my subconscious trying to never build a Skynet.

I've spent a few hours with Consul and I feel it has a lot of potential. The services that it provides are very complementary to traditional configuration management, but in a way that makes it feel as if it's one year down the road, waving to Puppet and saying "I'll be here when you catch up".

Joe

--

Joe Topjian

Systems Architect

Cybera Inc.

www.cybera.ca

Cybera is a not-for-profit organization that works to spur and support innovation, for the economic benefit of Alberta, through the use of cyberinfrastructure.