Pillar top.sls from GIT subfolder question...

[Jeff]

Howdy all,

I'm trying to add my pillar top.sls file to a subfolder in a GIT repository so that when I commit a change to top.sls in GIT, salt-master automagically picks it up.

I've added the following file structure to my GIT repository:

./salt-pillars:   

├───.git

└───_top_files

    ├───gin

    └───ops

In /etc/salt/master file I have added:

ext_pillar:

  - git:

    - master g...@gitlab.mycompany.com:mygroup/salt-pillars.git:

      - root: _top_files/gin

(Note: I have additional pillar_roots defined for the local filesystem.)  If I remove the top.sls from the file system and restart salt-master, the logs show it is reading the top.sls file from GIT:

2017-04-14 16:56:12,561 [salt.template    ][PROFILE ][28568] Time (in seconds) to render '/var/cache/salt/master/git_pillar/e93f46b268c1490f881ba65f80bb31a562d9aa643482e86485582a57984db661/_top_files/gin/top.sls' using 'yaml' renderer: 0.0197930335999

But when I try to list the pillar items for a minion node:

salt 'filereputationservice-acc01*' pillar.items pillarenv=acc

it doesn't show anything:

filereputationservice-acc01:

    ----------

If I put the file system top.sls back, pillar data returns.  It doesn't seem to be treating the GIT file as a top file.  Should this even work?  What am I missing?  

Thanks!

[Jeff]

Just checking back to see if anyone can offer some tips as to how to get this to work.  Thanks!

[Pier B.]

Hi Jeff,

Just in case, I know I am late, but there is not that many questions on which I can be of any help, BUT I do use ext_pillar based on gitfs and it does work. The difference is that I store my top file under the root dir, not inside a root sub directory, have you tried to move your top file from inside to straight under your root dir ? Just to be sure it works that way, as it should.

There is a close relation between the tops file gitfs_root declaration and the pillarenv your are using when launching the pillar.get in your example. I don't get all the details (actually mostly non of them) but I think that could be related somehow, might be a good direction to search .

using the parameter "top_file_merging_strategy: same" parameter is recommended , you may want to try this.

well I realize that I am not of a big help here either :)

Just in case here is may setup :

├── ROOT1
│   ├── common
│   ├── lan
│   ├── orch
│   ├── README_FIRST.txt
│   ├── resource
│   ├── role
│   └── top.sls
├── ROOT2
│   ├── common
│   ├── lan
│   ├── orch
│   ├── README_FIRST.txt
│   ├── resource
│   ├── role
│   └── top.sls

and the corresponding master config excerpt :

## Pillar git config
ext_pillar:
  - git:
    - master git@xxxxxxxxx/Salt_Pillar.git:
      # Pillar SLS files will be read from the 'testBed' subdirectory in this repository (equivalent to "gitfs_root" param for states)
      - root: ROOT2
      - pubkey: /root/.ssh/id_rsa.pub
      - privkey: /root/.ssh/id_rsa

Hope that'd give you some directions ...

PierB

[Pier B.]

Oh and if you want your commits to automagically become available to the master you'll need to add some hook on your git server and a reactor to your master (https://docs.saltstack.com/en/latest/topics/tutorials/gitfs.html#refreshing-gitfs-upon-push). Otherwise you have to wait for ~60 seconds in order to get the latest commit from your master (seems like state.apply get the latest anyway, or am I wrong ?)

[Jeff]

Thanks!  I'll try to explain in excruciating detail what I'm doing and why I'm doing it and maybe get to a better solution.

We are implementing "Continuous Deployment" (CD) using SaltStack to provision the minions, with ThoughtWorks GoCD to handle the CD pipeline workflow and Jenkins to do the CI part (build, unit test, SonarQube analysis, GoCD Pipeline trigger).  A typical GoCD pipeline looks like the following:

     

An overview of what each pipeline does is as follows (some detail omitted):

• ratingtool
• Triggered by Jenkins
• Takes as input from Jenkins the following build information (among other things)
• Nexus Artifact URL (WAR, DEB, JAR_With_Deps)
• Nexus Arifact SHA1
• etc.
• Saves build information for downstream pipelines as a properties file
• RatingTool_Prep_Materials
• Retrieves property file from upstream pipeline 'ratingtool'
• Checks out salt_pillars from GIT containing salt state templates
• Generates pillar states from templates containing the new build URL/SHA1
• Commits change back to git
• Generates salt pillar file/folder structure expected by SaltMaster
• Saves pillars for all environments
• RatingTool_DEV (Developer Environment)
• Retrieve pillar from upstream pipeline "RatingTool_Prep_Materials"
• Copy the DEV environment pillars to salt master
• Initiate salt highstate for DEV minions via salt-api
• Run Automated Test Suite
• RatingTool_ACC (Acceptance Test Environment)
• Retrieve pillar from upstream pipeline "RatingTool_Prep_Materials"
• Copy the ACC environment pillars to salt master
• Initiate salt highstate for ACC minions via salt-api
• Run Automated Test Suite
• <Repeat Until Production environment is reached>

Even though our pillars are stored in GIT, the Salt Master does NOT directly use GIT for pillars.  

Issue #1 - We initially started with GIT using branches for each environment.  This became a nightmare to maintain.  With some of our projects, we will eventually have 100+ environments across 40 data centers for our cloud service in production.  Each application for each environment requires unique settings/configuration specific to that environment.  Some pillars are shared but keeping some files in the master branch and some in environment-specific branches for 100+ environments is PAINFUL to maintain.

Issue #2 - Idempotency in each environment for each application... or how to control the progression/promotion of new builds into each environment in to prevent a HighState from deploying a build that wasn't ready for that environment.  For example, if the current state has:

DEV = v1.3.6 | ACC = v1.3.2 | BTA = v1.2.9 | PRD-DC5 = v1.2.2

If DEV validation passes, build v1.3.6 can then be promoted to ACC and the state will become:

DEV = v1.3.6 | ACC = v1.3.6 | BTA = v1.2.9 | PRD-DC5 = v1.2.2

Then a new build might get triggered changing the state to:

DEV = v1.3.7 | ACC = v1.3.6 | BTA = v1.2.9 | PRD-DC5 = v1.2.2

Issue #3 - How to minimize duplication of pillar state files for a given application as well as keep all application-specific files together?   For a given application, many of the pillar settings are the same.   This also relates to making it so that when a DEV or OPS engineer makes changes to the settings, we minimize the number of files and file locations that get touched.

Current solution:

Our pillars files are in the GIT 'master' branch using the following file structure:

├   ...  

├───ratingtool

│   ├───acc

│   │       custom.sls

│   │       globals.yml

│   │       salt-api_request.json

│   │

│   ├───bta

│   │       custom.sls

│   │       globals.yml

│   │       salt-api_request.json

│   │

│   ├───dev

│   │       custom.sls

│   │       globals.yml

│   │       salt-api_request.json

│   │

│   ├───prd-dc5

│   │       custom.sls

│   │       globals.yml

│   │       salt-api_request.json

│   │

│   └───shared

│           init.sls

│           war_file.sls           //from Template by "prep_materials" pipeline

│           war_file.sls.template  //Template 

├   ...  

│

└top.sls

This allows all application-specific pillars to be located within one subfolder per-application.

Additionally to minimize duplication of pillar files, common files were moved to a 'shared' folder to be used in each environment.  In the example above, the "RatingTool_Prep_Materials" pipeline phase transforms the GIT folder structure to the following structure compatible with the salt-master layout:

...

└───pillar

   ├───acc

   │       custom.sls

   │       globals.yml

   │       init.sls

   │       salt-api_request.json

   │       war_file.sls

   │       war_file.sls.template

   ├───bta

   │       custom.sls

   │       globals.yml

   │       init.sls

   │       salt-api_request.json

   │       war_file.sls

   │       war_file.sls.template

   ├───dev

   │       custom.sls

   │       globals.yml

   │       init.sls

   │       salt-api_request.json

   │       war_file.sls

   │       war_file.sls.template

   └───prd-dc5

           custom.sls

           globals.yml

           init.sls

           salt-api_request.json

           war_file.sls

           war_file.sls.template

This structure is NOT saved to GIT, but managed by the GoCD pipeline as an artifact of the pipeline process.  The following pillar file structure exists on the salt master:

/srv/pillar

├── acc

│   ├── ratingtool

├── bta

│   └── ratingtool

├── dev

│   └── ratingtool

├── int

│   └── ratingtool

├── prd-dc5

│   └── ratingtool

└── base

     └── top.sls

When the pipeline 'RatingTool_DEV' runs, it copies files from pillar/dev/** for that build to the salt master at /srv/pillar/dev/ratingtool and then calls state.highstate. The salt-master now has the updated pillar containing the new application URL/SHA1.  

The file copy process is cumbersome and I'd like to find a pure GIT mechanism.  

The first step to a GIT solution (and the reason I started this thread) is to try and get top.sls (for all environments) to be pulled from GIT since I don't have an automated process in place to update top.sls.  Since we have different salt masters, I split top.sls into separate files based on the salt master using it.  Each master now has an ext_pillar entry for pulling top:

#/etc/salt/master from DEV salt-master

ext_pillar:

  - git:

    - master git@gitlab:/salt-pillars.git:

      - env: base

      - root: _top_files/dev

#/etc/salt/master from OPS salt-master

ext_pillar:

  - git:

    - master git@gitlab:/salt-pillars.git:

      - env: base

      - root: _top_files/ops

The salt-master seems to be reading the top file, but not applying it to any environment except 'base' which seems counter-intuitive since the top.sls file in the 'base' pillar_root works for all environments.

I'd also like to use a GIT-only solution for managing build promotion to the various environments, but I don't have a clear picture as to what might work without adding a lot of complexity.

Sorry this was long but I'm open to all input.

-Jeff

--
You received this message because you are subscribed to the Google Groups "Salt-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to salt-users+unsubscribe@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/salt-users/fc762e2f-ec83-438e-b0f8-83e10f80bbbb%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

[Pier B.]

Wow ! I won't be of any help here sorry mate ! You process indeed looks complex and I do understand you very easily when you say that you want to use git only or salt only, mixing tools this way is always a real pain... I am trying to avoid this as much as possible, and luckily enough we started a brand new infra / CI  (almost) from since we use salt....

You may wanna give the merging option I told you about previously , just in case.

I first thought about using multiple branches for our 3 envs and 4 DC's (I feel so little now :) ) but I rapidly find out that'd be to hard for me to maintain, so I switched to one repo, one branche and 4 gitfs_root definition, but I must confess I had to use a rsync script to deploy changes from test env to prod env so this is far from perfect.

Good luck.


Pier B