tell Shake the # of threads of an `cmd_`?

[zesen...@gmail.com]

Hi List,

There is already `shakeThreads` option that controls the maximal number of threads that can be run at the same time. While this is certainly useful, I shall note that there are some `cmd_`s that's already utilizing threading - say, `cmd_ prog -j 5` where `prog` is a C program using pthread.

An enhancement to the existing threading management would be therefore to make Shake aware of the number of the threads a cmd_ will be using. Say shake knows there are 10 threads running, and the configured "shakeThread" is 16, and there is a runnable cmd_ which is marked to be triple-threaded. Shake can then invoke the cmd_, and update the count to 13(instead of 11 as of now).

AFAIK, the only way there can be more than one thread in a shake action is by cmd_. The problem is that the current thread management in shake is action-based. So I guess the cmd_ has to be altered to block in case the threads run out.

Best,
Zesen

[Neil Mitchell]

Hi Zesen,

Yep, all sounds very interesting. I've raised a bug here to track
progress towards that feature:
https://github.com/ndmitchell/shake/issues/603

I think in most cases you can control the -j passed on to commands, so
you might want the primitive:

speculativelyGrabUpToNThreads :: Int -> (Int -> Action a) -> Action a

Which runs an action with upto N threads, telling you how many it got.
That way, if the build system is fully utilising all cores, you build
with 1 thread, and if it's mostly idle, you build with more.

Thanks, Neil