makeflow of PBS Pro workflow

[Willem Marais]

Hi With the current implementation of makeflow of PBS Pro, is it possible configure makeflow such that it 1. submits a makeflow job to a node that has X CPUs, 2. the makeflow job is a collection of X commands, 3. and the makeflow job executes the X commands on the node with X CPUs? This might seem like a weird question. But, the way the UCAR cheyenne cluster operates, is that when you submit a job in the PBS Pro queue, you reserve a node with 36 CPUs and not a node with a single CPU. If the current implementation of makeflow does not support his non-standard workflow, do you think it is appropriate for makeflow to support such a non-standard workflow? If yes, then I would be happy to try to adapt makeflow to support such a non-standard workflow if I can receive some guidance, and depending how much effort it will require. Willem

[Ben Tovar]

Willem,

There are some immediate alternatives:

1) If you don't expect a lot of concurrent jobs (e.g. less than 36) you could submit the makeflow as a pbs job using -Tlocal as the batch system. In that way makeflow executes in the compute node and all its rules execute locally in that node.

2) With some little changes perhaps your makeflow can be adapted to use the Work Queue (-Twq) batch system. The idea here is that the pbs jobs are work queue workers (one worker per node) that connect back to makeflow. Makeflow then sends rules to the workers available. Some changes may be needed, as Work Queue does not assume a shared filesystem, so you may have to be more explicit about the pathnames, and also about the resources each rule uses (e.g. you may have to put a CORES=1 at the top of your makeflow so that each rule uses one core. otherwise one rule would use the whole 36 cores in a worker).

3) Finally, if your makeflow has a natural hierarchy, you may be able to define submakeflows, with each submakeflow executing in a whole node. For this you would need to use the JX syntax for makeflow, as explained here:

https://cctools.readthedocs.io/en/latest/jx/jx-tutorial

Please let us know if any of these solutions would be suitable for you, and we can explore more in detail how to adapt them to your workflows.

Ben

On Wed, Dec 8, 2021 at 3:24 PM Willem Marais <wil...@jmarais.com> wrote:

Hi With the current implementation of makeflow of PBS Pro, is it possible configure makeflow such that it 1. submits a makeflow job to a node that has X CPUs, 2. the makeflow job is a collection of X commands, 3. and the makeflow job executes the X commands on the node with X CPUs? This might seem like a weird question. But, the way the UCAR cheyenne cluster operates, is that when you submit a job in the PBS Pro queue, you reserve a node with 36 CPUs and not a node with a single CPU. If the current implementation of makeflow does not support his non-standard workflow, do you think it is appropriate for makeflow to support such a non-standard workflow? If yes, then I would be happy to try to adapt makeflow to support such a non-standard workflow if I can receive some guidance, and depending how much effort it will require. Willem

--
You received this message because you are subscribed to the Google Groups "Cooperative Computing Tools" group.
To unsubscribe from this group and stop receiving emails from it, send an email to cctools-nd+...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/cctools-nd/9d6ca44b-82b8-479c-a83d-b3fb08e13ea5n%40googlegroups.com.

[Willem Marais]

Hi Ben

Thank you for the very fast reply. Option 2 seems the best option since it provides the maximum flexibility for my needs. Option 3 seems also like a viable solution, though I don't see any documentation in JX about submakeflows - would it be something that has to be implemented?

Overall, I have a preference for option 2. 

Willem

You received this message because you are subscribed to a topic in the Google Groups "Cooperative Computing Tools" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/cctools-nd/sPNNNbAyQ8o/unsubscribe.
To unsubscribe from this group and all its topics, send an email to cctools-nd+...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/cctools-nd/CAMik99U576TNoPdViyOiHxUQBzNUMGaConiiyUnP2Z6w9h1Cnw%40mail.gmail.com.

[Ben Tovar]

Willem,

I used the incorrect terminology. In JX submakeflows are called nested workflows.

For Work Queue, it may be easier if you try the example that we show here:

https://cctools.readthedocs.io/en/latest/makeflow/#using-work-queue

(The example.makeflow is the workflow in this section https://cctools.readthedocs.io/en/latest/makeflow/#basic-usage).

That example shows the main points to take into account when running a makeflow with Work Queue. Work Queue was designed for use cases very similar to your current situation, so hopefully it will be useful for you. We will be happy to answer any questions you may have!

Ben

To view this discussion on the web visit https://groups.google.com/d/msgid/cctools-nd/CAKsVgOu8HSZSt1eyE%2BOGXpp6wtudLcPq2%3DxECujrynx3d9VFsQ%40mail.gmail.com.

[Willem Marais]

Thank you Ben. All these resources are super useful! If I get stuck, I'll reply to this thread.

Willem

To view this discussion on the web visit https://groups.google.com/d/msgid/cctools-nd/CAMik99VgxdabJiKn-HJshusMtg9aPjDbjvoWrPTu%3DyvUL8Lm6g%40mail.gmail.com.