States and pillar validation / non-regression test

[Pier B.]

Hi,

Please excuse the noobiness of the following as I am no dev, and not yet aware of code review and so on....

I found that many times after I commit and push to our company gitlab the following provisioning / setup of a test server finds syntax error and so on.

I am sure you guys already check this basics (correct syntax / basic code review) and am wondering how you handle these.

we are using git as backend fs, abd the dev team already has a CI flow using selenium / Jenkins / maven / nexus, in case you got tips regarding this exisiting tools.


Thanks !

[Pier B.]

Answering my own question ? Yep sure ! (as usual :) )
Well this rainy week-end got me some time to think about and find "a" way to achieve my goals, here is a rough overview of my flow :



I still need to improve this but this is a good start, and it works, which is not the worst case ;)

the first step : Salt_linting is using the

sudo /usr/bin/salt-call state.highstate --retcode-passthrough mocked=True

to validate states end pillar basic syntax.
Then I just tend to follow my orchestration state but running them not from orchestrate runner as I want to get the output and exit status....
All these comands are launched using the remote ssh execution jenkins plugin, not the classiest way but it does work so it is not that bad after all.

Fell free to ask for any details.

[viq]

Check out test-kitchen http://kitchen.ci with kitchen-salt plug-in.

On 11 Nov 2017 7:37 p.m., "Pier B." <bell...@gmail.com> wrote:

Hi,

Please excuse the noobiness of the following as I am no dev, and not yet aware of code review and so on....

I found that many times after I commit and push to our company gitlab the following provisioning / setup of a test server finds syntax error and so on.

I am sure you guys already check this basics (correct syntax / basic code review) and am wondering how you handle these.

...

[Pier B.]

Checking it right now, I saw many occurences of kitchen during my initial ressearch on this topic but this looked too much for me at this time, I really need to reconsider I guess.

[Christian McHugh]

Kitchen salt will spin up a test environment for you, but it is still on you to define the tests and validations.

We already use Jenkins and docker images for CI test running in our environment. So we've started doing the following scripted Jenkinsfile (it's a big rough and in progress, but should give you the idea):

import groovy.json.JsonSlurper

node("cs-prod-saltstack"){
    
    stage("Checkout"){
        sh "mkdir -p /srv/salt"
        dir('/srv/salt'){
            checkout scm
        }
    }

    stage("Lint"){
        sh "salt-run fileserver.update && salt-run git_pillar.update && salt '*' saltutil.sync_all"
        sh "salt '*' state.show_lowstate" // execution exits if command failure
        sh "salt '*' state.highstate mock=True" // execution exits if command failure
    }

    stage("Execute highstate"){
        // Executing Highstate apply
        try {
            sh "docker exec -t test-salt_minion salt-call state.highstate -l debug --retcode-passthrough --no-color"
        } catch (Exception err) {
            currentBuild.result = "FAILURE"
        }
    }

    stage ("Testing") {
        def jsonResp = sh(script: 'docker exec -t test-salt_minion salt-call saltcheck.run_highstate_tests --output=json -l quiet --no-color', returnStdout: true).trim()
        echo jsonResp
        def jsonout = new groovy.json.JsonSlurper().parseText(jsonResp)
        try {
            assert jsonout.local[13]."TEST RESULTS".Failed == 0
            currentBuild.result = 'SUCCESS'
        } catch (AssertionError e) {
            currentBuild.result = 'FAILURE'
            office365ConnectorSend(message: "${JOB_NAME} - Salt State validation Problem!\n", status: "${currentBuild.result}, took ${currentBuild.durationString}.", webhookUrl: <someurl>)
        }
    }
}

That gives us a lint test with state.show_lowstate and state.highstate mock=True. From there we then kick off a new docker image to run a real highstate and saltcheck.run_highstate_tests check. I've really liked saltcheck so far as it allows you to define tests using a syntax very much like writing salt states. It gives you the capabilities of jinja (salt renders), pillar lookups, and tests with salt execution modules. 

Here's a quick example to validate that the openssh package is installed and at the latest version. 

{% from "openssh/map.jinja" import openssh with context %}

openssh-server-latest:
  module_and_function: pkg.latest_version
  args:
    - {{ openssh.server }}
  assertion: assertEmpty

If we were to write the same check in serverspec, we'd need to hardcode the package name. However, his test can use the same map.jinja containing that info as used by the state. 

Hopefully that helps!