Benchmarking rule

[Luke Hoersten]

Hey guys,

One of my Shake rules runs benchmarks. I'd like Shake to run as multi-threaded as possible except when running benchmarks. What's the proper way to configure something like this?

Thanks,

Luke

[Neil Mitchell]

Hi Luke,

There's nothing that can do that at the moment, but I'm not averse to
doing so (Bake has a mechanism for doing exactly this operation).

There is a single function which has the flavour of roughly what you
want: https://hackage.haskell.org/package/shake-0.15/docs/Development-Shake.html#v:unsafeExtraThread

That function lets you use an extra thread without it being accounted
to you. In your case you want basically the inverse, to be considered
to be using a thread even though you aren't. You can do that with
resources by making every rule depend on a particular resource, but
that's nasty. Instead, I think the way to go is have a "more threads"
resource, so you could write:

phony "benchmark" $ withResource resourceThreads n $ ...

And that would acquire the necessary threads, one by one, and once it
had them all would do the action. Does that sound like it would work
for you?

Thanks, Neil

[Luke Hoersten]

I'm not sure how the monadic dependency tree is represented internally but what I had in mind was some directive similar to "alwaysRerun" except like "runInSequence" or something that would force an articulation point in the dependency tree. Basically force it to be a rule run all by itself.

But your idea is something I hadn't considered. Basically we're talking about a thread "wait" of all the Shake rule threads before running. Your model seems to fit into preexisting paradigms.

Any idea how this could be implemented?

Thanks,

Luke

[Neil Mitchell]

Hi Luke,

> I'm not sure how the monadic dependency tree is represented internally but
> what I had in mind was some directive similar to "alwaysRerun" except like
> "runInSequence" or something that would force an articulation point in the
> dependency tree. Basically force it to be a rule run all by itself.

Yes, Shake doesn't really have that notion, and there's no obvious way
to run it.

> But your idea is something I hadn't considered. Basically we're talking
> about a thread "wait" of all the Shake rule threads before running. Your
> model seems to fit into preexisting paradigms.
>
> Any idea how this could be implemented?

It needs to be implemented in the Core of Shake. Something using the
Pool mechanism to reserve threads, then when you have all the ones you
want, doing something else, but I suspect the details are a bit
fiddly. If the idea works for you, I'll raise a ticket.

Thanks, Neil

[Luke Hoersten]

Works for me. Do you want me to file the ticket or you as you can probably describe the details better?

Thanks,

Luke

[Neil Mitchell]

Sure, I've created the ticket at https://github.com/ndmitchell/shake/issues/229

Thanks, Neil

[Luke Hoersten]

Looks good. Thanks Neil.