Module repository organization

[Johan Sunnerstig]

Hello,

I'm just wondering how people with a decent collection of modules are organizing their modules and repositories?

We currently have one big repository with everything in it, and three branches, basically stuff moves alpha -> beta -> production. This is nice and simple in a way, but has some obvious and not so obvious downsides.

For one updating a module means going to the "next" branch and doing a checkout of that specific module from the previous branch so as to not just yank in every change made in the previous branch. This along with having just one big commit history for everything makes the git history pretty much useless.

And we're now in the process of implementing Jenkins for CI, the lack of granularity is making this a bit of a headache(though this could very well be because we're complete newbies at Jenkins ;) . For example having a post-receive hook triggering a build in Jenkins, with one big repo this will trigger rebuilds of every module since they all subscribe to the same repo even though they're separate jobs.

So, how are you people who have actually done this for a while managed this little problem? Separate repos per module? Something else we haven't thought of?

Regards

Johan

[Jeremy T. Bouse]

Johan,

I actually have 2 unique workflows between work and personal puppet
environments. In both cases they utilize Jenkins CI and git repos to
manage but in different ways. That said it appears that our workflow for
work is going to be moving more towards how I currently manage my own
puppet environment.

I will leave open for discussion if this is considered a best practice
within the community but this is how I'm managing it and I keep it out
there for anyone to make use of as starting point. I have my module
repository available at https://github.com/UGNS/standard-modules and I'm
currently running with puppet 3.5.1 though I'm working to update it for
3.6.0 and using directory environments which it's nearly ready for as is.

I make use of r10k to deploy so my repo contains the manifests/
sub-directory along with the Puppetfile which r10k uses to deploy out to
the modules/ sub-directory. I plan to obviously add an environment.conf
for 3.6.0+ directory environment support. With r10k every module is
either pulled from the Forge or a git repo. This allows the modules to
be developed independently of what is deployed out for a given environment.

With r10k each branch is itself a separate environment. Currently I
operate with 3 branches (development, staging & production) which my
Jenkins CI jobs handle. I have the GitHub repo setup using Amazon SQS
which Jenkins is setup as a consumer and triggers the job runs when
commits are pushed to the repo. The job uses librarian-puppet locally on
the Jenkins host to process the Puppetfile and verify that it doesn't
attempt to install any other dependencies than what is defined in the
Puppetfile itself. Upon success it triggers a job to execute r10k over
an SSH connection to the puppet master followed by triggered job to
execute an 'mco puppet runall <CONCURRENCY> -F environment=<BRANCH>' to
force execution on all hosts within that environment.

Successful job runs can then be approved for promotion which runs the
triggered job to merge from one branch to the other (development ->
staging, staging -> production) and promoted to production I tag it with
the date/time stamp.

I still have some thoughts on how better to improve what I have but
haven't had time to develop them yet. I'd like to make the process more
fluid within Jenkins so it would allow for feature branches to use as
test environments that could then be merged in rather than having a
fairly strict configuration that only supports the 3 existing branches
and ignores everything else. I'm also looking at the option of using mco
itself to trigger the r10k deployment rather than having to do so over
an SSH connection. Along with the fact the system can't easily stand
itself up without being done in stages currently.

[David Schmitt]

Hi,

thank you Jeremy for sharing your setup. That looks very nice indeed!

I do have a few questions, which might help further the understanding of
r10k deployments.

* How is r10k used with local vagrant development?

* How do you manage local patches to puppet-forge modules?


Some notes from my side:

* jenkins with multiple branches: depending on the complexity of your
jobs, and non-functional requirements, you might consider using
jenkins-job-builder to create separate jobs for each branch, or use
one of the more built-in mechnisms like the GIT_BRANCH environment
variable or job parameters (when triggering externally).

* For bootstrapping, you might want to look into the kafo tool
https://github.com/theforeman/kafo
That is being developed by the foreman guis and can use puppet
modules and manifests to bootstrap/install systems.


Regards, David

[Jeremy T. Bouse]

On 05/18/2014 07:47 AM, David Schmitt wrote:
> Hi,
>
> thank you Jeremy for sharing your setup. That looks very nice indeed!
>
> I do have a few questions, which might help further the understanding of
> r10k deployments.
>
> * How is r10k used with local vagrant development?
>

I guess I would have to better understand in what capacity you are
thinking with local vagrant development. All r10k is doing is checking
out the repo to a cached directory then deploying the branches out as
environment directories under the specified basedir. Likewise the
Puppetfile itself is also understood by librarian-puppet which can
install the modules specified into a modules/ sub-directory and is
actually part of my local workflow to test and ensure I have
dependencies met when including a new module since r10k doesn't do the
dependency check itself and simply installs only what's specified in the
Puppetfile.

> * How do you manage local patches to puppet-forge modules?
>

I approach modifications as I do as a Debian Developer, and if I need
to patch a forge module I've forked it, fixed it and submitted a pull
request back to the author. The Puppetfile format allows for both
specifying the forge module or a git repo and with r10k > 1.1.3 it
handles switching between a forge module declaration and a git repo so
until the patch is accepted I can point the module at the modified git repo.

>
> Some notes from my side:
>
> * jenkins with multiple branches: depending on the complexity of your
> jobs, and non-functional requirements, you might consider using
> jenkins-job-builder to create separate jobs for each branch, or use
> one of the more built-in mechnisms like the GIT_BRANCH environment
> variable or job parameters (when triggering externally).
>

I'll have to take a look at jenkins-job-builder and see if it can't
provide the solution I'm looking for. My job that triggers the r10k
deployment is using GIT_BRANCH to decide which environment to update.
The trick is in the promotion trigger as I currently have to pass the
next branch to promote to and the actual job itself for development &
staging which have this promotion trigger are locked by specifying the
"Branches to build" within the job. The production branch deploy job is
almost identical with the exception it uses the git publisher to tag the
run on success and it doesn't have the promotions.

> * For bootstrapping, you might want to look into the kafo tool
> https://github.com/theforeman/kafo
> That is being developed by the foreman guis and can use puppet
> modules and manifests to bootstrap/install systems.
>

As I haven't used theforeman I haven't looked much at the other
projects coming out of it. I'll have to take a look at this and see.
>
> Regards, David

[David Schmitt]

On 2014-05-18 17:14, Jeremy T. Bouse wrote:
> On 05/18/2014 07:47 AM, David Schmitt wrote:
>> Hi,
>>
>> thank you Jeremy for sharing your setup. That looks very nice indeed!
>>
>> I do have a few questions, which might help further the understanding of
>> r10k deployments.
>>
>> * How is r10k used with local vagrant development?
>>
> I guess I would have to better understand in what capacity you are
> thinking with local vagrant development. All r10k is doing is checking
> out the repo to a cached directory then deploying the branches out as
> environment directories under the specified basedir. Likewise the
> Puppetfile itself is also understood by librarian-puppet which can
> install the modules specified into a modules/ sub-directory and is
> actually part of my local workflow to test and ensure I have
> dependencies met when including a new module since r10k doesn't do the
> dependency check itself and simply installs only what's specified in the
> Puppetfile.

So the answer I was looking for seems to be "I'm using puppet-librarian
for local development."?

>> * How do you manage local patches to puppet-forge modules?
>>
> I approach modifications as I do as a Debian Developer, and if I need
> to patch a forge module I've forked it, fixed it and submitted a pull
> request back to the author. The Puppetfile format allows for both
> specifying the forge module or a git repo and with r10k > 1.1.3 it
> handles switching between a forge module declaration and a git repo so
> until the patch is accepted I can point the module at the modified git repo.

Seems to work out well for you, as I haven't seen any local forks in
your Puppetfile.

>> Some notes from my side:
>>
>> * jenkins with multiple branches: depending on the complexity of your
>> jobs, and non-functional requirements, you might consider using
>> jenkins-job-builder to create separate jobs for each branch, or use
>> one of the more built-in mechnisms like the GIT_BRANCH environment
>> variable or job parameters (when triggering externally).
>>
> I'll have to take a look at jenkins-job-builder and see if it can't
> provide the solution I'm looking for. My job that triggers the r10k
> deployment is using GIT_BRANCH to decide which environment to update.
> The trick is in the promotion trigger as I currently have to pass the
> next branch to promote to and the actual job itself for development &
> staging which have this promotion trigger are locked by specifying the
> "Branches to build" within the job. The production branch deploy job is
> almost identical with the exception it uses the git publisher to tag the
> run on success and it doesn't have the promotions.

Wouldn't everything except "development" promote (i.e. integrate) to
"development" first, and then "development" is released as "production"?
OTOH that seems excessive when a primary use case is fast-forwards of
development to the developer's HEAD, after testing it somewhere.



Regards, David

[Johan Sunnerstig]

Thanks for the discussion and suggestions, 

I'll have some reading to look forwards to it appears. In particular I'll look into Job Builder since that seems like it could slot into what we're building now with minimal effort(we're "a little" short on time at the moment). We're using R10k currently but without any granularity to speak of, the problem in general is that our module collection is a mix of things from the forge, some modified and some not, and we have quite a few home built modules as well, so granular control at a module level is a bit complicated.

Regards

Johan