Updating salt exec modules via git hooks

[Steve Hajducko]

We have several teams that work on different execution modules in their own repositories.  I'd like to come up with a way to get those to minions whenever the modules are updated.

The idea I have at the moment is to create a runner that executes based off a reactor.  The reactor is listening for an event via a git hook.  When git gets a push to the master branch, it sends an event to the reactor, which would then execute the runner.

The runner would pull/refresh the repository, then move the module itself over to /srv/salt/_modules, then issue a saltutil.sync_modules.  I haven't seen a way to specify custom sync directories for syncing execution modules - it appears that '_modules' is the only allowed directory.  Also because we have multiple teams working on multiple repos, I can't just force them all into one repo ( although we thought about doing that and having a maintainer of the 'Salt Modules' repo that everyone just sends PRs for and have that sync to /srv/salt/_modules.

Does anyone have a good methodology updating their minions with new execution modules that are stored in git?

Thanks

[Thomas Phipps]

switch to gitfs instead of using /srv/salt. 

by using gitfs you can have a _modules directory in each repo and to the minions it will look like one _modules directory. 

this also removes the need for writing a runner to handle this. as you can just have a git post commit hook hit a salt webhook that triggers a reactor. the reactor can then trigger an orchestration state rather like this one https://github.com/whytewolf/salt-phase0-orch/blob/master/orch/sys/salt/update.sls that updates the fileserver backend, [in my case updates the pillar backend, as well as mine_functions i store in pillar] and pushes any modules to minions [and master]

gits documentation: https://docs.saltstack.com/en/latest/topics/tutorials/gitfs.html

--
You received this message because you are subscribed to the Google Groups "Salt-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to salt-users+...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/salt-users/c00c1e37-683c-4efb-a7fc-097bd6f60cb2%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Steve Hajducko]

We do use gitfs.  I wasn't aware that salt would look for a _modules in each directory though - I thought it was only the _modules in the file root.

To view this discussion on the web visit https://groups.google.com/d/msgid/salt-users/CAPaX09hHeuefT5RBFQ8mvft%3DmeFpD6x4K9mcMC1kojtjDdkJpQ%40mail.gmail.com.

[Steve Hajducko]

A quick test of this ( at least on 2015.8.8-2 ) didn't seem to work.

We mount each git repo to a team directory, for example:

gitfs_remotes:

  - ssh://g...@github.acme.com/web-team.git:

    - mountpoint: teams/web

  - ssh://g...@github.acme.com/db-team.git:

    - mountpoint: teams/db

I created a _modules directory in one of the repos, verified it had synced via a cp.list_master but doing a sync_utils shows that the module did not sync.  We don't use a separate file_roots for each git repo.  Just a basic:

file_roots:

  base:

    - /srv/salt

So each team accesses their scripts via something like salt://srv/salt/teams/web/myscript.sh

Is there some extra bit I'm missing to get this to work?

[Thomas Phipps]

ahh you are using a mountpoint that is the problem. 

that causes everything in the repo to be mounted under that virtual directory including the _modules directory. but salt is looking for the _modules directory at salt://_modules

also file_roots has no connection to gitfs. it is used for the local filesystem only to be pulled into the virtual filesystem that salt uses. 

also if the mount point is teams/web the salt url would be salt://teams/web/myscript.sh

if you have things showing up as salt://srv/salt/* then you have some other config doing that. 

honestly. your teams should be using a more flat setup for the salt setup. but include more depth in their repos. take a look at some of the formula repos out there. like https://github.com/saltstack-formulas/ckan-formula these are built to be run directly from gitfs if needed. and that formula has a state module included.

To view this discussion on the web visit https://groups.google.com/d/msgid/salt-users/CA%2BNJt%3DUS5dd%2BQ%2B5xcCAAm9jDHVBdCzJWq9UFFftAVZZjmKaPjA%40mail.gmail.com.

[Steve Hajducko]

Yea sorry - I mistyped the salt:// example - we don't have the srv/salt in the uri

So you're saying that instead of mounting each team to a directory, to effectively mount them all at the root of the file root. Then have them create the depth inside of their actual repo.

The only downside here would seem to be that the teams need to be sure that they are not clobbering each other's namespace and overwriting each other's scripts.

I'm unsure at this point if we could make that sort of change but I'll take it under consideration.

I'm guessing I'll have to fall back to the previous solution idea for now unless there's another way of doing this with the mountpoints.

To view this discussion on the web visit https://groups.google.com/d/msgid/salt-users/CAPaX09j6JcXkVpXjUzeJfv-MhsEhPMWM7y1HxtOb8PB%3D2OGbfQ%40mail.gmail.com.

[Thomas Phipps]

you might be right. i can think of a nasty hacks to do it. but as clean as either a centralized repo or

having the mounts twice. one with the mountpoint and another with a root setting that points to a subdirectory that holds the _modules directory. 

another work around would be each team having 2 repos one for the modules one everything else. with the modules ones being at the root. 

thing to consider with all of these solutions is that for each repo added to gitfs it is going to take more time to do filesystem.update 

this is kind of why the centralized repo with PR's makes sense. it might be more work, but it encourages auditing. and in the end it is a better representation of what salt actually has. 

as is they are going to have to avoid clobbering of the _modules directory anyway. no matter what solution you end up going with. [even if it worked perfectly all the modules sync system does is install the modules into the cache under a file system directory. so the clobbering would happen there]

To view this discussion on the web visit https://groups.google.com/d/msgid/salt-users/CA%2BNJt%3DVCmWQ8mbW3pveOpwAkJShT7rhw6dNGGPH%3DV2MGJHEAhw%40mail.gmail.com.