How to manage puppet modules? git submodules? hg subrepositories? own solution?

[Malte Janduda]

Hi folks,

we are looking for a way to manage our puppet modules in a convenient way. At the moment we have about 30 modules which are versionized with git or mercurial. Most of them are self-made, some are from github.

We thought it would be a good idea to use mercurial subrepositories to manage the versions of those modules. Well, it is not. Everytime you push your commits to the server all subrepositories are checked for updates. It takes minutes to push your changes! And if you have to merge something you do not really know what happens. You just try every command which appears in your mind - multiple times and in different order :D

It happened three times that every developer had to delete the whole module tree and check it out again because something was broken.

git submodules do not seem to be fun either.

I have looked for an alternative to submodules/subrepositories and found librarian-puppet (https://github.com/rodjek/librarian-puppet). Unfortunately it does not really work good for me. A lot of useless error messages if something is configured wrong. This is pretty important for us. All developers should be able to use the versioning tool. It has to be easy to use. Further the librarian-puppet does not seem to be backed by a healthy community. Just every few months a commit and a lot of open issues and pull-requests.

I have hacked some Python code to realize something similar: dyps - https://github.com/MalteJ/dyps

You have a yaml file which defines your modules: name, source, version and DVCS system (git/hg):

- dep: mysql

  source: https://github.com/puppetlabs/puppetlabs-mysql.git

  type: git

  version: af4b8bff0ad59f21211be2a98d31588066c41e0c

All modules are downloaded to the folder "dependencies" when executing "dyps run".

You can decide if you want to get a specific changeset at the next run or if you want to get the latest changes - just discard the version property or execute "dyps release your-module-name". If you want to keep the current version execute "dyps keep your-module". You can update one or all versionized modules via "dyps update your-module" or "dyps update-all". The modules without a defined versioned are updated at every dyps run.

I have added an import function for mercurial subrepository .hgsub files. Feel free to extend it with an importGitSubmodules function ;-)

What do you think about this approach?
How are you managing your modules/dependencies?

Best,

Malte

[Benjamin Priestman]

I've tried using https://github.com/azimux/externals (referenced in Version Control with Git) which kind of works, but is a bit buggy and I've yet to come up with an easily understood workflow.

[Mason Turner]

We are still using svn, but looking on adapt this to git next year.

This is a simplified view of our svn repo:

- nodes

-- dev

--- trunk

--- tags

---- 20121220-121212

-- prod

--- trunk

--- tags

- modules

-- apache

--- trunk

--- tags

---- 20121220-121212

-- something_else

--- trunk

--- tags

---- 20121220-121212

We then have a utility (link_puppet) that:

* takes a module, tag and environment as command line options 

* Updates the svn sandbox in /opt/puppet/svn/modules/$module

* links /opt/puppet/svn/modules/$module/tags/$tag to /etc/puppet/environments/$env/modules/$module

It also validates the module before linking, logs who did what when. Pretty handy.

— Mason Turner

--
You received this message because you are subscribed to the Google Groups "Puppet Users" group.
To view this discussion on the web visit https://groups.google.com/d/msg/puppet-users/-/VZjQExuTnvAJ.
To post to this group, send email to puppet...@googlegroups.com.
To unsubscribe from this group, send email to puppet-users...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en.

[Nikola Petrov]

On Thu, Dec 20, 2012 at 01:28:04PM -0500, Mason Turner wrote:
> We are still using svn, but looking on adapt this to git next year.
>
> This is a simplified view of our svn repo:
>
> - nodes
> -- dev
> --- trunk
> --- tags
> ---- 20121220-121212
> -- prod
> --- trunk
> --- tags
> - modules
> -- apache
> --- trunk
> --- tags
> ---- 20121220-121212
> -- something_else
> --- trunk
> --- tags
> ---- 20121220-121212
>
> We then have a utility (link_puppet) that:

Can you please post the link_puppet utility. I will be happy if we can
burrow some code and not start from scratch on almost the same setup(we
are doing it manually at the moment)

Best, Nikola