How to automate the verification of salt command output

[Bubunia Patra]

Hi all,

I want to run a salt command from the salt master targetting towards minions. I want to automate the verification of salt command output to verify if the output returned is correct or not? 

Can anyone suggest me how I can automate such usecase with some examples? I am thinking of using python modules + salt to achieve this. Can someone suggest me some better way for achieving this with some examples so that I can tryout and learn quickly?

Regards

Pradeep

[Amse Master]

The simplest checking I know of is using "onfail" to see if a state run failed, then using that to do something.  For example:

Something that might fail:
  cmd.run:
    - name: /bin/some_command -arg1 -arg2

Output a message on failure:
  test.show_notification:
    - text: some_command failed!
    - onfail:
      - cmd: Something that might fail

This uses "onfail" to tie the second state to the first one.

However this only checks to see if the cmd.run command failed with a non-zero return code. 

If you want to check for specific text in the command output, the only way I can think of would be to use a helper script that would check the output and then return 0 or 1.  For example, assuming the helper script is located on the fileserver root:

helper_script.sh

#!/bin/bash

/bin/some_command -arg1 -arg2 | grep "text that indicates success"

This would return grep's exit code (0 if text found, 1 if not).

Then the first state above would change to:

Something that might fail:
  cmd.script:
    - source: salt://helper_script.sh

Admittedly this is a hacky solution.  I'm using it in one case right now and would love to know if there's a more self-contained way of doing it.

[Bubunia Patra]

There is another way to do this as below:

https://docs.saltstack.com/en/latest/topics/tutorials/writing_tests.html

I have not dig more to it though, If anyone have used this concept can throw some light on pro and cons of it.

[Christian McHugh]

Recently we've been using the saltcheck module to handle this use case. I've recently submitted a bunch of fixes and documentation improvements to make it work even better for our needs, so future releases should just work, but for now I'd suggest backporting the saltcheck module to your environment.

Saltcheck allows you to write tests for your states in a very similar yaml (or supported renderer of choice) syntax, including having access to the same pillars and grains. 

For example, if you had a salt state which managed linux users where the data comes from a pillar, you could use:

# /srv/salt/users/init.sls

{% for user, uid in pillar.get('users', {}).items() %}

{{user}}:

  user.present:

    - uid: {{uid}}

{% endfor %}

You could then create a saltcheck-tests directory in the same location as your states, and create a users.tst file inside with the content:

# /srv/salt/users/saltcheck-tests/init.tst

{% for user, uid in pillar.get('users', {}).items() %}

check-user-{{ user }}:

  module_and_function: user.info

  args:

    - {{ user }}

  assertion: assertEqual

  expected-return: {{ uid }}

  assertion_section: uid

{% endfor %}

(This example relies on functionality submitted in https://github.com/saltstack/salt/pull/49395, so it might take a few more days for it to be merged in to develop)

If your state was run with:

salt <minion-id> state.sls users

The saltcheck could be run as:

salt <minion-id> saltcheck.run_state_tests users

The cool thing is that it uses the standard salt modules to perform all the checks, so you have all the power of salt behind your tests (it's just checking the output of user.info in this case). This example validates all users defined in the pillar have been added, so it works for 1 or 100 just as easily. This also allows you to output the results as you would any other salt command, so you can use the standard human text output to view green or red success messages, or output as json for further processing.

We started using saltcheck recently to validate application deployments to hadoop clusters using the hadoop.dfs and s3 modules. Where our QA team would spend time manually checking files in various locations, this task is now accomplished in seconds with greater reliability.

Saltcheck validations can then be combined with test kitchen or Jenkins pipelines for CI, but it's also a simple to use but super powerful tool for validating state results.

Hopefully that helps!

[Jay Farschman]

Christian,

Thanks for this.  I got a bit caught up in a project that went a little sideways for the last week, but thanks. I'm hoping to get my team introduced to saltcheck tests this next quarter if not sooner.  Seems to solve some problems.

--
You received this message because you are subscribed to the Google Groups "Salt-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to salt-users+...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/salt-users/53da1681-6de5-4a75-b39a-4a83923a5898%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

	Jay Farschman Site Reliability Engineer jfars...@pingidentity.com
Connect with us:

CONFIDENTIALITY NOTICE: This email may contain confidential and privileged material for the sole use of the intended recipient(s). Any review, use, distribution or disclosure by others is strictly prohibited.  If you have received this communication in error, please notify the sender immediately by e-mail and delete the message and any file attachments from your computer. Thank you.