Add -l N option to limit the load average.

[Nicolas Desprès]

Hi,

I have pushed a pull request
(https://github.com/martine/ninja/pull/274) which get rejected for a
fairly good reason by Evan because he think "we'll hit the "this is a
bug" case in build.cc."

In my opinion, this won't happen as I have commented on the pull
request thread. I am convinced the condition is robust enough to
prevent such a case to occur.

Anyway, I really need this feature, so I am posting here to have more
opinions. I'm trying to convinced Evan that my patch work :-). I would
be glade if some of you could give it a try and to tell me if you
reach this "this is a bug" case. Of course, I am open to any
suggestion to make the condition testing more robust if it has to.

Unfortunately, this is a Unix only feature right now because, I did
not know any easy way to implement it on Windows yet. But, I did leave
the place to write the Windows part in the patch.

Thanks in advance,
-Nico

[Evan Martin]

On Sat, Apr 28, 2012 at 8:18 AM, Nicolas Desprès
<nicolas...@gmail.com> wrote:
> I have pushed a pull request
> (https://github.com/martine/ninja/pull/274) which get rejected for a
> fairly good reason by Evan because he think "we'll hit the "this is a
> bug" case in build.cc."
>
> In my opinion, this won't happen as I have commented on the pull
> request thread. I am convinced the condition is robust enough to
> prevent such a case to occur.
>
> Anyway, I really need this feature, so I am posting here to have more
> opinions. I'm trying to convinced Evan that my patch work :-). I would
> be glade if some of you could give it a try and to tell me if you
> reach this "this is a bug" case. Of course, I am open to any
> suggestion to make the condition testing more robust if it has to.

Hm, I find your answer pretty convincing. Perhaps we should just
merge it and see what happens. :)

I wonder, in make's case, does passing -l imply -j as well? (That is,
does specifying a load limit implicitly lift the maximum parallel
process limit?)

[Nicolas Desprès]

On Sat, Apr 28, 2012 at 7:54 PM, Evan Martin <mar...@danga.com> wrote:
> On Sat, Apr 28, 2012 at 8:18 AM, Nicolas Desprès
> <nicolas...@gmail.com> wrote:
>> I have pushed a pull request
>> (https://github.com/martine/ninja/pull/274) which get rejected for a
>> fairly good reason by Evan because he think "we'll hit the "this is a
>> bug" case in build.cc."
>>
>> In my opinion, this won't happen as I have commented on the pull
>> request thread. I am convinced the condition is robust enough to
>> prevent such a case to occur.
>>
>> Anyway, I really need this feature, so I am posting here to have more
>> opinions. I'm trying to convinced Evan that my patch work :-). I would
>> be glade if some of you could give it a try and to tell me if you
>> reach this "this is a bug" case. Of course, I am open to any
>> suggestion to make the condition testing more robust if it has to.
>
> Hm, I find your answer pretty convincing.  Perhaps we should just
> merge it and see what happens. :)

I would love so :-)

>
> I wonder, in make's case, does passing -l imply -j as well?  (That is,
> does specifying a load limit implicitly lift the maximum parallel
> process limit?)

The answer is no. They are two independent options. -l does not imply -j.

From the GNU make manual
(http://www.gnu.org/software/make/manual/make.html#Parallel):

------------------------------------------------------------------------------------------------
When the system is heavily loaded, you will probably want to run fewer
jobs than when it is lightly loaded. You can use the ‘-l’ option to
tell make to limit the number of jobs to run at once, based on the
load average. The ‘-l’ or ‘--max-load’ option is followed by a
floating-point number. For example,

-l 2.5
will not let make start more than one job if the load average is above
2.5. The ‘-l’ option with no following number removes the load limit,
if one was given with a previous ‘-l’ option.

More precisely, when make goes to start up a job, and it already has
at least one job running, it checks the current load average; if it is
not lower than the limit given with ‘-l’, make waits until the load
average goes below that limit, or until all the other jobs finish.

By default, there is no load limit.
------------------------------------------------------------------------------------------------

So the behavior, we currently have is similar to GNU make's behavior
=> you always have *at least one job* running. If you want to tweak a
little bit further the resources consumption of ninja, you can still
use nice(1).

Example:
- If you start: ninja -j 1 -l 100000.0 => you will always have one running job
- If you start: ninja -j 10000 -l 0.1 => you will (most probably...)
always have only one running job

The goal of -l is to limit the number of job but not to stop ninja and
make it waits until the load average goes down. So to make it short:
you will always have at least one job running.

Cheers,
-Nico

[Jeremy Apthorp]

. o O ( I wonder if you could send SIGTSTP/SIGCONT to further throttle the CPU load... )