Long running machine tasks

[Alton Alexander]

First of all, great project! I am an analytics expert I'm already seeing tons of use cases.

Many times my machine tasks involve a longer running process and require the computing resources of a distributed network. I'd like to be able to involve these longer running machine tasks before and after human tasks such that the input and output establish a part of the entire workflow.

Starting a machine task that kicks off a remote job isn't a problem and your demos go a long way to implementing a them in a custom workflow. However I'd like to know the best practices for the human task that immediately follows a long running machine task.

A couple of scenarios to consider:

1. I don't want the either human to experience any delay waiting for the machine task to load or be distracted by the process running as long as it is running correctly.
2. Since the long running machine task will create dependencies for the subsequent human task, can the machine task be marked as in process and unavailable for the next human to pick up until it truly has completed?
3. can a group of users be pushed a notice via email that the task is complete and the next step needs to be done.
4. what is the best practice for changing the status on the long running task to needing review if it has taken longer to complete than expected. Should this be like a cron job that is set at the same time the machine task starts, if it confirms the task is running too long does it change the status on the machine task, or the leading human task?

Thanks again.
Alton

[Daniel Haas]

Hi Alton,

Welcome to the list--you're asking some great questions!

Long-running machine tasks are definitely something we want to support well, and we already have existing support for many of the scenarios you describe. Let me start by overviewing existing functionality, then describe how it can support your use cases.

Orchestra supports any task scheduler that exposes a python function with a certain signature (see https://github.com/unlimitedlabs/orchestra/blob/master/orchestra/utils/machine_step_scheduler.py if you want to dig into the code) that we can call to schedule a new task when it becomes available. The only requirement for the schedulers is that they call `orchestra.machine_tasks.execute` on whatever machine they wish to compute the machine function. By default, Orchestra uses a simple `SynchronousMachineStepScheduler` that just runs machine tasks on the web server machine as they become available. However, we also include an `AsynchronousMachineStepScheduler` that uses Amazon Web Service's SQS (https://aws.amazon.com/sqs/) to queue tasks, and Elastic Beanstalk worker machines (https://aws.amazon.com/elasticbeanstalk/) to process them.

This makes it easy to provision many workers to service a task queue as work becomes available. It's not hard to imagine a setup where the Orchestra queue worker is actually a Spark master that takes new tasks and processes them on a distributed cluster.

Your other questions seem more oriented around how to handle the hand-off between human and machine tasks:

1. I don't want the either human to experience any delay waiting for the machine task to load or be distracted by the process running as long as it is running correctly.

Workflows are specified as a declarative graph of steps, where each step's creation and completion can depend on the completion of one or more other steps. If a human step is running in parallel to a machine step (and you are using the asynchronous task scheduler), there will be no interaction between the two or competition for computing resources.

2. Since the long running machine task will create dependencies for the subsequent human task, can the machine task be marked as in process and unavailable for the next human to pick up until it truly has completed?

By default, workflow step dependencies do exactly what you're asking for here. Simply specify that the next human step 'depends on' the machine step, and the human task won't be available until the machine step finishes.

3. can a group of users be pushed a notice via email that the task is complete and the next step needs to be done.

Orchestra has a built-in notification system (https://github.com/unlimitedlabs/orchestra/blob/master/orchestra/utils/notifications.py for implementation) that by default sends users an email when they are assigned to a task. Task assignment happens according to 'policies' that are evaluated when a task's dependencies are completed. By default, Orchestra is pull-based, so users would have to log in to the interface and ask for a new task in order to have it assigned. We also support an assignment policy that auto-assigns based on previous steps, i.e., the worker who completed a previous task can be assigned to complete a subsequent one. You could imagine your workflow having a human completing a step, then auto-assigning them to the step that occurs after an intervening machine step, which would accomplish your goals. We’re also looking at adding new assignment policies, and adding your own is likely a small number of lines of python.

4. what is the best practice for changing the status on the long running task to needing review if it has taken longer to complete than expected. Should this be like a cron job that is set at the same time the machine task starts, if it confirms the task is running too long does it change the status on the machine task, or the leading human task?

This is an awesome question, and I don't think we have an answer for it yet! For now, we rely on external systems for job monitoring, since most work queues have their own reporting and debugging built in. If you have ideas for integrating that with Orchestra, let us know.

Sorry for the insanely long email, and I hope this has been informative. Feel free to continue this thread, or chat with us on gitter.im/unlimitedlabs/orchestra for more quick back-and-forth.
-- Daniel

--
You received this message because you are subscribed to the Google Groups "Orchestra" group.
To unsubscribe from this group and stop receiving emails from it, send an email to orchestra-dev...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/orchestra-devel/1ed2df89-a5f9-45da-88b4-d9d214a9e388%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.