Setting 'make -jN' in makeprg
John Beckett
automatically detect the number of processors and adjust the
'-j' make option. This is out of my league (although I have seen
a friend build the Linux kernel using -j16, although I think he
gave that up because of the confusing error messages), but I
suggested the following replacement:
let &makeprg = 'make -j'.system('echo -n "$(echo $(grep -c ^processor /proc/cpuinfo)+1 | bc)"')
Any thoughts on whether that is correct and useful? The tip is:
http://vim.wikia.com/wiki/Auto-detect_number_of_cores_for_parallel_built
John
Marc Weber
> a friend build the Linux kernel using -j16, although I think he
> gave that up because of the confusing error messages)
one error message. With that solution in mind you understand soon that
setting -j option in .vimrc is broken because you need *two* commands:
1) compile till hitting any error
2) compile with -j1 to find that one error without confusing error
messages
Its not perfect but fastest in many cases.
If you have a look at vim-addon-actions you can bind make commands to
keys multiple times easily - and while doing so add -jX options.
Using vim-addon-local-vimrc for each project is another easy solution:
define those two mappings once and continue working
Marc Weber
Christian Brabandt
On So, 15 Apr 2012, John Beckett wrote:
> A new tip suggests a vimrc command to set makeprg so it will
> automatically detect the number of processors and adjust the
> '-j' make option. This is out of my league (although I have seen
> a friend build the Linux kernel using -j16, although I think he
> gave that up because of the confusing error messages), but I
> suggested the following replacement:
>
> let &makeprg = 'make -j'.system('echo -n "$(echo $(grep -c ^processor /proc/cpuinfo)+1 | bc)"')
Your command looks strange, why are you adding 1 there? I think, better
is something like:
let &makeprg = 'make -j'.system('grep -c ^processor /proc/cpuinfo')
regards,
Christian
--
Eine Ostk�ste ist eine K�ste die westlich des Wassers liegt
- daher der Name..., Folge 185
Ein Ostufer ist ein Ufer das �stlich des Wassers liegt.
- daher der Name..., Folge 213
John Beckett
>> let &makeprg = 'make -j'.system('echo -n "$(echo
>> $(grep -c ^processor /proc/cpuinfo)+1 | bc)"')
>
> Your command looks strange, why are you adding 1 there? I
> think, better is something like:
>
> let &makeprg = 'make -j'.system('grep -c ^processor /proc/cpuinfo')
Huh, you think my command is strange: you should see the
original tip (link in previous message).
I was just adding 1 per the original idea, which is the belief
that if you have N processors you should pass N+1 to -j so there
is always a process waiting to be run. Google shows varying
opinions on that (one rather convincing one I glanced at
suggested that such fiddling doesn't do anything significant).
I don't know the consequences, but a defect in your command
above is that makeprg ends up with a NUL (^@) at the end, due
to the newline output by grep (seen with ':set makeprg?').
John
Christian Brabandt
Ok, I never cared much about it the number of jobs to run in paralel
with make, but I heard, that you should use as many jobs as
processors/cores you have. One page I googled, even suggested to use
1.5 times the number of processors.
In any case, this one should be slightly better:
:let &makeprg = 'make -j'.system('grep -c ^processor /proc/cpuinfo')[:-2]
and if you want to add one (or perform other calculations), simply
adjust this to (There is no need to do the processing in shell and
fork another couple of processes, since vim can do it perfectly well
itself):
:let &makeprg = 'make -j'.(system('grep -c ^processor /proc/cpuinfo')[:-2]+1)
regards,
Christian
John Beckett
> and if you want to add one (or perform other calculations),
> simply adjust this to (There is no need to do the processing
> in shell and fork another couple of processes, since vim can
> do it perfectly well itself):
>
> :let &makeprg = 'make -j'.(system('grep -c ^processor
> /proc/cpuinfo')[:-2]+1)
Of course! Very nice. Amazingly, it can be further shortened
because the trailing newline is ignored when adding 1 (so you
can omit "[:-2]" from the above.
I used this to update the tip:
http://vim.wikia.com/wiki/Auto-detect_number_of_cores_for_parallel_built
It suffers a defect that on a single-processor system it will
use 'make -j2' which is probably not helpful. Fixing that would
require a function that detracts from the elegance.
John
Ben Fritz
>
> Ok, I never cared much about it the number of jobs to run in paralel
> with make, but I heard, that you should use as many jobs as
> processors/cores you have. One page I googled, even suggested to use
> 1.5 times the number of processors.
>
I recently helped redo our make process at work, which included using parallel builds, and doing it through Eclipse.
Eclipse has a "use optimal number of jobs" setting, but all this does is pass the -j flag with NO argument to make. Apparently this actually spawns ALL jobs which can be parallelized, all at once when make begins. Somehow on my and several coworkers' machines, this works just fine, and gives equivalent performance to the fastest time given from experimenting with various explicit numbers of jobs (which happened to be 1.5 times the number of processors for us...we all have 4-core machines). But...not passing an explicit number of jobs fails miserably on other machines, which more predictably run out of memory and/or process IDs and the build crashes.
This discussion prompted me to investigate further, coming up with 'psrinfo' on Solaris and the NUMBER_OF_PROCESSORS environment variable on Windows which can be used in the same way as the tip being discussed here.
I wonder if we want to modify the tip to use 1.5*CPUs instead of 1+CPUs? This would also prevent us from starting multiple jobs on a single-processor system, if we round down or use integer math. It should be fairly easy for a user who decides they want a different number of processes to modify the arithmetic.
Christian Brabandt
On Mo, 16 Apr 2012, Ben Fritz wrote:
> Eclipse has a "use optimal number of jobs" setting, but all this does
> is pass the -j flag with NO argument to make. Apparently this actually
> spawns ALL jobs which can be parallelized, all at once when make
> begins. Somehow on my and several coworkers' machines, this works just
> fine, and gives equivalent performance to the fastest time given from
> experimenting with various explicit numbers of jobs (which happened to
> be 1.5 times the number of processors for us...we all have 4-core
> machines). But...not passing an explicit number of jobs fails
> miserably on other machines, which more predictably run out of memory
> and/or process IDs and the build crashes.
,----[ http://www.gnu.org/software/make/manual/make.txt ]-
| If the `-j' option is followed by an integer, this is the number of
| recipes to execute at once; this is called the number of "job slots".
| If there is nothing looking like an integer after the `-j' option,
| there is no limit on the number of job slots. The default number of job
| slots is one, which means serial execution (one thing at a time).
`----
Well, "no limit on the number of job slots" doesn't sound right, in my
opinion.
regards,
Christian
--
Benjamin Fritz
Yeah, it doesn't sound like it should work at all. Somehow though it
works wonderfully on my 4-core machine, with equal performance to
using 6 processes. Experiments trying 1-8 processes showed 6 as the
magic number for my 4-core machine, and I had two coworkers verify it
on theirs. It must be some magic in the scheduler on the 64-bit
Windows XP system we all have, because another coworker with an older
dual-core crashed 'make' every time he tried running without an
explicit number of jobs.
By the way, Eclipse showing "Optimal number of jobs" for the option
name is even considered a bug:
https://bugs.eclipse.org/bugs/show_bug.cgi?id=259768
I'm certainly not advocating using -j with no explicit number. I like
the idea of this number being determined for me, though. I meant to
give more weight to the "1.5 * number of CPUs seemed to work better
for me" but re-reading what I wrote, I failed in that regard.
Quincy Bowers
You could just deal with this in make by creating targets that re-run make with the arguments you want. Then you don't have to change Vim settings to alter the way make runs. It's probably not an important distinction but doing it in the Makefile lets other developers benefit from the change as well.
Example:
# Makefile
parallel:
make -jX all
serial:
make -j1 all
# The original make target
all:
...
Quincy
Paul
>let &makeprg = 'make -j'.system('echo -n "$(echo $(grep -c ^processor /proc/cpuinfo)+1 | bc)"')
>
>Any thoughts on whether that is correct and useful? The tip is:
Having vim calculate it in such a fashion sounds inefficient to me. I use the MAKEOPTS environment variable. Have if calculated once in your .bash_profile equivalent, if you must.
--
.
Ben Fritz
I think the idea would be to have it "just work" on pretty much any system you might use the same config on.
Is MAKEOPTS standard? I currently only compile with multiple processes at work, using a cygwin-based cross-compiler on Windows, or on a Solaris server. Neither make seems to pay any attention to MAKEOPTS. I found this, where someone claims it is Gentoo-specific.