Support for async execution in StackStorm - Mistral workflow

[amit.ko...@gmail.com]

Hi,

I am using Mistral to create workflows as part of StackStorm automation. In certain use cases, I want to create a custom action using python runner. The action is long running (e.g. it may take 5 mins to return the result back). Is it possible to create this using async pattern? Is there any help available to create this? May be a sample code will help.

I checked the mistral_http action but could not use it because
1. My actual execution will be a python code and not a http service
2. Even while testing, the mistral_http didn't send any Mistral-Callback-URL as part of the HTTP header. I could see other headers like Mistral-Workflow-Execution-Id, Mistral-Action-Execution-Id

Any help would be appreciated. Just FYI, I am currently using 1.3.2.16 version of ST2 (upgrade is planned)

Thanks,
Amit

[Tomaz Muraus]

Hello,

Right not we don't include native support for asynchronous actions.

You can achieve a similar end-result (with some limitations) by having long running Python runner actions. You simply need to make sure that the default value of "timeout" action parameter is high enough (higher than the longest time your action can run).

The actual StackStorm platform uses green threads and non-blocking I/O which means that many running Python actions are not a big deal - they don't block and shouldn't consume a lot of resources.

Only limitation of this approach is HA - if the action runner dies or similar during the action execution, the execution won't resume when the runner comes back online. That's (usually) not a big deal, especially in distributed system where you need to design actions to be atomic and as idempotent as possible so you can re-run them at any time.

Having said that, I do see some value in supporting asynchronous actions, but that's something we need to discuss some more to see if it's a good future fit.

Personally, my main problem with this approach is that it might encourage users to not follow best practices for designing actions for a distributed system. This can be fine if user is aware of the limitations and what they are doing, but in the end, StackStorm is a distributed system and there is no way around some things.

[Tomaz Muraus]

After posting my first reply above, I remembered there is another fairly common approach you can use.

You can use event-driven approach (hey, that's what StackStorm is all about!). This means potentially splitting your large workflow in multiple smaller ones. Then inside each workflow you have a long-running action which, when it's done, emits an event (trigger, can also be webhook) which then triggers a run of another workflow (via rule) and so on.

This event-driven approach is very powerful, but it also has some disadvantages compared to "one big workflow" approach. For one, it's usually harder to reason about it (you have many small interconnected pieces, similar to micro-services) and debug it.

On Tuesday, June 28, 2016 at 8:25:59 AM UTC+2, Amit Kocharekar wrote:

[Amit Kocharekar]

Hi,

Thanks for your suggestions. The even driven approach sounds good, but instead of triggering a different workflow on response, can I resume from the task (or one after that) which started the long running action? I was just reading about mistral and came across this link: https://wiki.openstack.org/wiki/Mistral/Long_Running_Business_Process. Is something like this possible to implement in ST2-Mistral?

[Dmitri Zimine]

Hi Amit, 

we have began working on a relevant feature, “Ask”, I am cc-ing Matt who’s working on the design,

while what you’ve asked is not directly mapping to what Matt is thinking, 

we’d like to know your feedback, AND Matt may look at seeing how your request may fit in his proposed design.

Connecting you two to pick conversation on slack, github, or email.

Cheers, Dmitri.

--
You received this message because you are subscribed to the Google Groups "StackStorm" group.
To unsubscribe from this group and stop receiving emails from it, send an email to stackstorm+...@googlegroups.com.
To post to this group, send email to stack...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/stackstorm/f0903811-768a-42e0-90f1-524749519a1c%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Matt Oswalt]

Hello Amit – as Dmitri said, I’m working on a new feature that might be interesting to you. Let me describe it to you briefly and we take it from there. Keep in mind this is still a WIP idea and may change, but what we’re thinking is that we’ll be able to add a new return status from an Action, like “paused” or “awaits” that signifies that additional data is needed. This could be a simple binary approve/reject from a manager, or it could be a request for additional data to proceed, such as a second factor for authentication – something you simply can’t bake in to the workflow.

 

So, we’d likely introduce a new action, something like “st2.ask” that returns “awaits” by default, and only returns a success status if this data checks out (someone has approved). The API will have a new method that external entities like the CLI can provide the needed data and re-run this execution. Provided it checks out, the rest of the workflow continues normally.

 

All that said, this isn’t really an async execution solution, it just pauses for more information. So, while this may not tackle your use case directly, I am still interested in your thoughts. It does seem to map fairly well to the Mistral link you shared.