Combining our experience into a larger, common module repo
Michael DeHaan
So I was talking with several folks on IRC this morning, and we came up
with an idea.
One of the strengths of Puppet is it has a very large community with
tons of systems administration experience. This is huge. I'd like to
unite that experience more closely, so that this power is immediately
available and obvious to new and existing users. Currently we have a
large collection of repos, some containing one module, some containing
many, but they are fragmented:
http://www.reductivelabs.com/trac/puppet/wiki/PuppetModules
What I'd like to do initially is start getting these together in one
giant curated repo, hosted on our github space, that makes it easier for
all Puppet users to contribute to.
Now this immediately starts the question of what other things people
will like to make this easier. Package management capability.
Metadata. Standards. New language features. Whoa, horsie!
I'm thinking let's avoid going there right now, and see what we can
accomplish for the current installed versions of Puppet, and in the
process of doing this, we'll see what we actually need and have a
framework in which to test them. I'm sure this will point to all
sorts of questions about how cross-OS modules should be, what metadata
is required, what the interoperability challenges may be, etc. Though,
even short term, it will provide a really good reference full of
examples for new and existing Puppet users to go to. And by sharing,
we can make sure the modules become the best they can possibly be.
I think we need to start small, and identify some basic concepts we need
a collection of namespaced modules
to have, in order to work together well. If this takes off, we may
want to create a seperate list for development of the common modules --
TBD -- but we could use puppet-users for now.
This way, by having all the repos in one place, and one common place to
talk about them, it would be easier for everyone to contribute --
whether or not they had a github account -- and we can commonly work on
codifying our own best practices and tools.
Existing folks are welcome to contain their own repos, though hopefully
we'll see a trend of more folks just creating github "forks" of the main
branch set, from which we can respond to merge requests.
Thoughts?
--Michael
James Turnbull
Hash: SHA1
On 3/02/10 3:27 AM, Michael DeHaan wrote:
> Hi List!
>
> So I was talking with several folks on IRC this morning, and we came up
> with an idea.
>
> One of the strengths of Puppet is it has a very large community with
> tons of systems administration experience. This is huge. I'd like to
> unite that experience more closely, so that this power is immediately
> available and obvious to new and existing users. Currently we have a
> large collection of repos, some containing one module, some containing
> many, but they are fragmented:
Mike
I'd love to see this get off the ground. There have been a couple
of attempts at it - and you can see some background at:
http://reductivelabs.com/trac/puppet/wiki/CommonModules
http://reductivelabs.com/trac/puppet/wiki/ModuleStandards
> What I'd like to do initially is start getting these together in one
> giant curated repo, hosted on our github space, that makes it easier for
> all Puppet users to contribute to.
Sounds great - we had a play with a couple of ways to do this -
mostly around git submodules (to allow you to check out individual
modules from a git repo). I'd personally recommend not going down
that path - I had issues. :)
> I think we need to start small, and identify some basic concepts we need
> a collection of namespaced modules
> to have, in order to work together well. If this takes off, we may
> want to create a seperate list for development of the common modules --
> TBD -- but we could use puppet-users for now.
I think the primary issue here is to have some common rules for
handling certain scenarios:
1. Cross-platform support
2. Naming standard
3. Module construction
I personally don't think these are hard problems to solve and I
think it we put together two or three straw man variants as a
talking point that'd get us further ahead than talking about it.
Regards
James Turnbull
- --
Author of:
* Pro Linux System Administration (http://tinyurl.com/linuxadmin)
* Pulling Strings with Puppet (http://tinyurl.com/pupbook)
* Pro Nagios 2.0 (http://tinyurl.com/pronagios)
* Hardening Linux (http://tinyurl.com/hardeninglinux)
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.7 (Darwin)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/
iQEVAwUBS2ipniFa/lDkFHAyAQLuKAgAvMWAhzS3idQrWZ+tVabOmSR1TfNoRBcR
HqdIhMkO9tpfV4GKxwgQOcZuD72sIZ5Kil6+WBIx6B/27jXMtPKwLHxc+aT/2Ylh
FulUZ9ig0wWdBHiStEMLIW4VjnUM4MEK7lDf4NXz4wi56jdEnXTxQ1O+KGIQG+cg
Ui6MvQeDw8TIxhHeJ0EeQ5OJ01MiiL1/3cSqAFcasB6mrDLP/24xvkopp95mkajT
N8cnyCtZbX/oj5U9HMNbqDS9C+XhMAuCpyrCyItuAVzoqdY9njLZe4VbUsAIRkIp
uXdTkaXZEH5DbheP0ujSsydkJ6GpevTqg2JzeqOSRvlzmvQQBgS1aQ==
=uJCK
-----END PGP SIGNATURE-----
Julian Simpson
> of attempts at it - and you can see some background at:
+1. I'd love to see better platform support for the modules that we have.
J.
--
Julian Simpson
Software Build and Deployment
http://www.build-doctor.com
http://twitter.com/builddoctor
Michael DeHaan
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
>
> On 3/02/10 3:27 AM, Michael DeHaan wrote:
>
>> Hi List!
>>
>> So I was talking with several folks on IRC this morning, and we came up
>> with an idea.
>>
>> One of the strengths of Puppet is it has a very large community with
>> tons of systems administration experience. This is huge. I'd like to
>> unite that experience more closely, so that this power is immediately
>> available and obvious to new and existing users. Currently we have a
>> large collection of repos, some containing one module, some containing
>> many, but they are fragmented:
>>
>
> Mike
>
> I'd love to see this get off the ground. There have been a couple
> of attempts at it - and you can see some background at:
>
> http://reductivelabs.com/trac/puppet/wiki/CommonModules
> http://reductivelabs.com/trac/puppet/wiki/ModuleStandards
>
>
Yep, read these (for benefit of the list, we were talking on IRC, I
don't actually read that fast!) ... seems not
be insurmountable.
Even a tiny bit of pain getting this off the ground seems hugely worth it.
>> What I'd like to do initially is start getting these together in one
>> giant curated repo, hosted on our github space, that makes it easier for
>> all Puppet users to contribute to.
>>
>
> Sounds great - we had a play with a couple of ways to do this -
> mostly around git submodules (to allow you to check out individual
> modules from a git repo). I'd personally recommend not going down
> that path - I had issues. :)
>
Can we start by grafting together everyone's modules and trying to
namespace them?
Git subtree merge preserves attribution nicely... so everyone still gets
their credits. The paths
will then need to be merged around some. Temporarily, this might
mean prefixing modules
the paths of their creators if we have conflicts.
If we need to make changes to them, there will be some testing work to
be done as well, which we can use help with. So this may mean
identifying which ones out of which repos are important for starters and
trying them out ... and moving more into the "curated" repo later ...
basically keeping a "to be processed" directory seperate from the main
so we can more easily identify what we need to transmogrify? We'll
start small with some good examples.
>
>> I think we need to start small, and identify some basic concepts we need
>> a collection of namespaced modules
>> to have, in order to work together well. If this takes off, we may
>> want to create a seperate list for development of the common modules --
>> TBD -- but we could use puppet-users for now.
>>
>
> I think the primary issue here is to have some common rules for
> handling certain scenarios:
>
> 1. Cross-platform support
> 2. Naming standard
> 3. Module construction
>
> I personally don't think these are hard problems to solve and I
> think it we put together two or three straw man variants as a
> talking point that'd get us further ahead than talking about it.
>
Indeed. I'd rather try it and see what comes crashing down. We need
to be aware of what the problems could be though, no doubt.
For our initial attempt, perhaps we should aim for mostly Debian+Red Hat
cross platformness with modules
supporting both? That seems reasonable and achieves a good starting
base. Of course there may be a lot of work getting to that...
I need to check for any explict licensing in there, of course. We'd
also need to identify
what parts of them did things "weird" and try to homogenize them a bit,
I'd suspect.
I don't mind doing the initial driving and getting the repo set up.
More collaborators on this would be good, and I think once we have this,
it would be easier for more people to contribute than what we have now.
That all being said, I'll be travelling a bit, so this may appear in
pieces. Help would be outstanding.
If we commit to having something done a bit more quickly rather than
trying to get it perfect, and perhaps we get further this time, and over
time we can clean up what's there. Folks should know if they do a
checkout in these early days of such an effort, the modules may not
remain compatible and they should suspect a "git rebase" to break
them. Over time, we can move that towards being more consistent.
So say we all?
--Michael
James Turnbull
Hash: SHA1
On 3/02/10 9:55 AM, Michael DeHaan wrote:
> Can we start by grafting together everyone's modules and trying to
> namespace them?
Sounds good. Puppet module collection owners? Alessandro? David?
Others?
> Git subtree merge preserves attribution nicely... so everyone still gets
> their credits. The paths
> will then need to be merged around some. Temporarily, this might
> mean prefixing modules
> the paths of their creators if we have conflicts.
+1
> For our initial attempt, perhaps we should aim for mostly Debian+Red Hat
> cross platformness with modules
> supporting both? That seems reasonable and achieves a good starting
> base. Of course there may be a lot of work getting to that...
+1 - naming, documentation also good.
> I need to check for any explict licensing in there, of course. We'd
> also need to identify
> what parts of them did things "weird" and try to homogenize them a bit,
> I'd suspect.
GPL is my preference for this sort of thing.
> I don't mind doing the initial driving and getting the repo set up.
> More collaborators on this would be good, and I think once we have this,
> it would be easier for more people to contribute than what we have now.
> That all being said, I'll be travelling a bit, so this may appear in
> pieces. Help would be outstanding.
In my copious spare time I am happy to help.
> If we commit to having something done a bit more quickly rather than
> trying to get it perfect, and perhaps we get further this time, and over
> time we can clean up what's there. Folks should know if they do a
> checkout in these early days of such an effort, the modules may not
> remain compatible and they should suspect a "git rebase" to break
> them. Over time, we can move that towards being more consistent.
>
> So say we all?
I say anyway.
Regards
James Turnbull
- --
Author of:
* Pro Linux System Administration (http://tinyurl.com/linuxadmin)
* Pulling Strings with Puppet (http://tinyurl.com/pupbook)
* Pro Nagios 2.0 (http://tinyurl.com/pronagios)
* Hardening Linux (http://tinyurl.com/hardeninglinux)
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.7 (Darwin)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/
iQEVAwUBS2iw/CFa/lDkFHAyAQLvNwgA1v2gLACO7ush7C5jT3wzhGJZvuPi8Rkm
zPx5VsLNUJ5VQRChWQOASF011z4M5pcgPMJQrmj8iNWrTFnAwH7KP9akW85wOG6x
X7310ydbMfuggS2iRXn+Kq2eoprpAR40OnR1sW+gd2l7dKUhhYt9nyhAyfmSUF13
jIjtEqJWeKFoXo9Sqzkke8A/iPjpiwNxO3+a6QCB/LTg1W/bCvv3YCgFAF9J+4L3
7Wgyt+7BCUEXK2ui18OlvcgY2gjvzcRKrCKOjnn//s9fMYN9GzPV+11Q83wLpSf5
NtmOs3hcjcnNiu1W1BxeW28slfpGcw1EmQCH6i9oVID/bEOP+FwH/w==
=UN0Q
-----END PGP SIGNATURE-----
Julian Simpson
>> namespace them?
Has there been any discussion of bundling some common modules in the
puppet distribution? It might help people get up to speed faster.
ja...@lovedthanlost.net
>>> Can we start by grafting together everyone's modules and trying to
>>> namespace them?
>
> Has there been any discussion of bundling some common modules in the
> puppet distribution? It might help people get up to speed faster.
>
Something like a rake task (or perhaps in the Dashboard).
$ rake pull apache
That clones a repository containing the common Apache module.
Regards
James Turnbull
Michael DeHaan
> On 3 February 2010 21:04, Julian Simpson <simpso...@gmail.com> wrote:
>>>>
>>>> Can we start by grafting together everyone's modules and trying to
>>>> namespace them?
>>
>> Has there been any discussion of bundling some common modules in the
>> puppet distribution? It might help people get up to speed faster.
>>
>
> I don't think we'd want to bundle modules into core - let's keep that lean
> and mean
Yes. It won't be in the core.
- I do think it'd be good to bundle the capability of pull common
> modules into a Puppet installation.
>
> Something like a rake task (or perhaps in the Dashboard).
>
> $ rake pull apache
We'll do something to make it easy, but I don't like the idea of
exposing rake (a Ruby build tool) as part of our tool suite -- we
don't want to be very Ruby centric, and also Rake isn't really great
at option parsing and becoming an application, should we want to grow
into that. We should probably have command line tools for stuff
like this, however simple, but we'll see when we get there.
Let's get this going and then see what software we needs as it goes.
No need to design up-front on the list. For purposes of full
disclosure, I don't expect to start this for several weeks, so look
for something then, and then we can start looking at the repo and
seeing what it needs and what folks think we should change.
I want to roll up some of Teyo's modules best practices into the start
of this, and then we'll have something to hang new work on.
--Michael
Al @ Lab42
I'm sorry to have missed the list in the last days, and I take the
occasion to reopen this thread it in order to avoid that also this one
fades forgotten as various other threads, in the past, about modules
standards, naming conventions, metadata and interoperability.
I'm glad that there's new drive, directlty from RL, on this topic and
I'm definitively willing to contribute in some way.
So, even if this might not be the right place and time, I'd like to
throw in some points.
I've just recently started to extend multiOS support in my modules and
I'm still trying to figure out the best way to handle that but one
thing is sure: a way to routinely test modules behavious on different
OS and different Puppet versions is necessary.
I wonder (or, better, ask to James) if Puppet continuos integration
( http://reductivelabs.com/trac/puppet/wiki/Development/PuppetContinuousIntegration
) relates only to Puppet building and its tests or can (will) be
somehow applied on Puppet runs on a set of Puppet Modules.
Besides this, I think that big isses in interoperability about
different sets of modules raise in the use of custom types, different
approaches to the same problem, and different ways to handle cross-
modules functions, such has monitoring, backup, firewall management
and so on, that generally work well only inside the same set of
modules.
I like, and have played a bit with, the idea of using "wrappers" with
(erm... ) standard naming conventions to manage common activities.
Something like:
A "wrapper" define called "Config" to manage files' inline
modifications using different methods ( something like:
http://www.example42.com/browser/common/manifests/defines/config.pp )
A wrapper module/define called "Monitor" to manage monitoring of nodes
(and services/ports/file systems...) using different monitoring tools
(according to custom needs) so that in your (standard) module you just
define what you want to Monitor with a standard naming convention,
using then the Monitor method you prefer ( some embrional attempts
here: http://www.example42.com/browser/monitor/manifests/init.pp )
Similar wrappers can be used for Backup or Firewall and so on. I Love
the idea to have, for example, an Apache module where is wrtitten: I
want to Backup /var/www/html (or whatever) AND make the Firewall open
input port 80 without caring what backup system I use and how I manage
my firewalls (that is done and customized in the backup and firewall
modules...)
Dunno if I made the point.
Last but not least, I would like to have a shared modules environment
where I can choose the module I want, for the same application, among
alternatives.
I'm costantly looking at how others do theirs modules and I always
learn a lot from them (DavidS, Camptocamp, Vulcane's PuppetManaged to
name a few), still most of the times, when I wanted to import and use
modules made by others, I found myself rewriting them in order to
follow my own way of thinking, my knowledge (or lack of) and style to
approach the same problem.
This is probably the same for eveybody, so I don't think there should
be just ONE approach in a module for an application, but a base
structure, and different alternatives/defines/subclasses to choose
from.
But hey, I know, I'm putting too much inside this pot, big results are
achieved in small steps, so Michael and RL, drive the path, do as
you've said ("no need to design up-front on the list") and tell us
what we can do.
Alessandro
Telmo X
Light travels faster than sound. This is why some people appear bright until you hear them speak
--
You received this message because you are subscribed to the Google Groups "Puppet Users" group.
To post to this group, send email to puppet...@googlegroups.com.
To unsubscribe from this group, send email to puppet-users...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en.
Michael DeHaan
> This thread is music for my hears...
> I'm sorry to have missed the list in the last days, and I take the
> occasion to reopen this thread it in order to avoid that also this one
> fades forgotten as various other threads, in the past, about modules
> standards, naming conventions, metadata and interoperability.
> I'm glad that there's new drive, directlty from RL, on this topic and
> I'm definitively willing to contribute in some way.
>
> So, even if this might not be the right place and time, I'd like to
> throw in some points.
>
> I've just recently started to extend multiOS support in my modules and
> I'm still trying to figure out the best way to handle that but one
> thing is sure: a way to routinely test modules behavious on different
> OS and different Puppet versions is necessary.
> I wonder (or, better, ask to James) if Puppet continuos integration
> ( http://reductivelabs.com/trac/puppet/wiki/Development/PuppetContinuousIntegration
> ) relates only to Puppet building and its tests or can (will) be
> somehow applied on Puppet runs on a set of Puppet Modules.
We were talking a bit about how to do this for a more integration
based test farm.
Seems like it is not wise to just use puppet to check puppet there,
and this could get complex.
Yeah, I understand it.
Everyone's infrastructure is managed slightly differently, so trying
to make something for everyone is difficult :)
It will be a challenge, no doubt! I very much like the idea of
trying to abstract out the "what you are doing" (monitoring) with the
"how you are monitoring" (ex: nagios).
I need to dig closely through example42 and the others more in the
near future to gather up a list of the ideas we want to incorporate.
>
> Last but not least, I would like to have a shared modules environment
> where I can choose the module I want, for the same application, among
> alternatives.
> I'm costantly looking at how others do theirs modules and I always
> learn a lot from them (DavidS, Camptocamp, Vulcane's PuppetManaged to
> name a few), still most of the times, when I wanted to import and use
> modules made by others, I found myself rewriting them in order to
> follow my own way of thinking, my knowledge (or lack of) and style to
> approach the same problem.
> This is probably the same for eveybody, so I don't think there should
> be just ONE approach in a module for an application, but a base
> structure, and different alternatives/defines/subclasses to choose
> from.
Hmm.... I understand the thinking.
I think what I want to build here is a "mostly acceptable to 80% of
everyone" way to do it out of the box, as a 'best practices' type of
repo (actually this is looking more and more like a set of repos) with
a way for people to also host/fork there own repos once we start to
add tooling around this. The central 'main' repo will be curated
and reviewed, so all the modules follow the same kinds of ideas and
work well together .... though it's up to everyone to kind of do a
trial by fire at arriving at that. These other sets of repos should
be easily attachable with the common tools though. TBD?
I'd prefer we don't get too idiomatic if possible so the examples can
also be instructive.
Here's an example -- One thing that came up in training was an idiom
for copying a file to temp, checking it with visudo, and copying it to
the intended location if the check passed. For instance, in that
case, it makes sense to write a "ValidatedFile" type and put it in the
common module repo. In other words, we are not only creating
a shared resource of modules, but we're also trying to make something
that serves as self-documenting tips-and-examples at the same time,
and something
that is easy to contribute to.
FWIW, we're mostly calling the common module "Forge" at the moment, so
I'll probably start referring to it as that. It is shorter :)
>
> But hey, I know, I'm putting too much inside this pot, big results are
> achieved in small steps, so Michael and RL, drive the path, do as
> you've said ("no need to design up-front on the list") and tell us
> what we can do.
Yeah, I apologize for not having immediate cycles to pound away on
this -- but look for something to start moving in March or so.
I think most likely we'll just set up a new account on github for
these and each module can be a repo, and we do some (very very
minimal) ruby tooling to make this be usable at first until we see
what we need.
>
> Alessandro