how to get detailed state context on state failure during orchestration

[Travis Mehlinger]

Hi, I've done a fair bit of digging on this and can't find a good solution.

I have a number of states that perform the various operations for deploying an application. I apply those states using orchestration, with a state the looks more or less like this:

# extra_pillar is something like: 

#  {"package": "<package name>", "environment": "<environment>"}

deploy_package:

  salt.state:

    - tgt: 'roles:deploy_node'

    - tgt_type: 'grain

    - sls:

      - deploy.package

do_migrations:

  salt.state:

    - tgt: 'roles:deploy_node'

    - tgt_type: 'grain

    - sls:

      - deploy.migrations

    - pillar: {{ extra_pillar }}

    - require:

      - salt: deploy_package

deploy_application:

  salt.state:

    - tgt: 'G@roles:app and G@environment:some_env'

    - tgt_type: compound

    - sls:

      - deploy.application

    - pillar: {{ extra_pillar }}

    - require:

      - salt: do_migrations

deploy_tasks:

  salt.state:

    - tgt: 'G@roles:app and G@environment:some_env'

    - tgt_type: compound

    - sls:

      - deploy.tasks

    - pillar: {{ extra_pillar }}

    - require:

      - salt: deploy_application

So now, my challenge: for each of these states I can add an onchanges requisite to send whatever notifications may be useful to let me know that a deploy successfully finished, and I don't really need any more detail than what I'm supplying to pillar when I fire off orchestration. However, I *don't* know how to get the level of detail I would like to send notifications when a state fails to apply. I can of course add some onfail requisites, but all I know how to do is fire off some notification saying that the state failed.

What I would like is to gather the output of the state run to munge details whenever I handle a failure and use some of that information in my notifications. I feel like I could accomplish this with a saltmod.runner state to gather job output, but without knowing the job ID at the time the state run fails, I don't have a reliable way to actually get the right job.

As I'm writing this, it occurs to me that I could use the reactor to solve this problem. Would this be a better approach, or am I on the right track solving this within orchestration?

Thanks!

[Colton Myers]

Reactor is probably the right path.

That said, I wonder if you could write a custom state module which was designed to use with orchestrate, and then inside of that module use the __running__ dictionary to fetch the results of previous states and pass that data into your subsequent state runs…

Not 100% sure it would work, because I’m not sure if the __running__ dictionary works the same in orchestration.

Anyway, rambling aside, I would head in the reactor direction.

–
Colton Myers

--
You received this message because you are subscribed to the Google Groups "Salt-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to salt-users+...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Travis Mehlinger]

Thanks for confirming what I was thinking. That's actually the route I ended up going, though I ended up using a custom runner instead of state modules. My runner more or less duplicates the baked-in orchestrate runner, processes the return data, then fires events that get handled by other reactor SLSes.

I've made a gist with a stripped-down version of the runner I wrote and a reactor SLS that invokes it. Perhaps it will be useful to someone else. :)

https://gist.github.com/tmehlinger/ffac546eff5cce33bb8b0aa82cff8070

[Colton Myers]

Nice! Thanks for posting!

--  

Colton Myers