kernel compile: make -j...

[Grant]

No use having an i7? TL;DR just some kernel compile times, speed
up memory from 1333MHz to 1600MHz near the bottom, cold vs hot cache.

Was it worth AU$1200? See comparison with 'pooh' at end, below.

i7 box will be described on

<http://bugsplatter.id.au/kernel/boxen/phobos/>

when I get find a round tuit!

grant@phobos:~/linux/linux-3.14.8a$ make clean; time make -j20
...
Kernel: arch/x86/boot/bzImage is ready (#4)

real 1m15.555s
user 8m56.670s
sys 0m26.852s

grant@phobos:~/linux/linux-3.14.8a$ make clean; time make -j200
...
Kernel: arch/x86/boot/bzImage is ready (#5)

real 1m16.098s
user 9m2.275s
sys 0m27.114s

grant@phobos:~/linux/linux-3.14.8a$ make clean; time make -j2000
...
Kernel: arch/x86/boot/bzImage is ready (#6)

real 3m29.682s
user 8m33.174s
sys 0m30.361s

This one above bogged down in a swap storm... Load average peaked in
the high 500s though.

grant@phobos:~/linux/linux-3.14.8a$ make clean; time make -j9
...
Kernel: arch/x86/boot/bzImage is ready (#7)

real 1m16.585s
user 8m45.602s
sys 0m26.377s

grant@phobos:~/linux/linux-3.14.8a$ make clean; time make -j50
...
Kernel: arch/x86/boot/bzImage is ready (#8)

real 1m16.511s
user 9m4.475s
sys 0m27.346s

Reboot, change memory to 1600MHz, cold cache:

grant@phobos:~/linux/linux-3.14.8a$ make clean; time make -j50
...
Kernel: arch/x86/boot/bzImage is ready (#9)

real 1m14.848s
user 8m28.461s
sys 0m25.524s

grant@phobos:~/linux/linux-3.14.8a$ make clean; time make -j50
...
Kernel: arch/x86/boot/bzImage is ready (#10)

real 1m11.643s
user 8m28.573s
sys 0m25.459s

grant@phobos:~/linux/linux-3.14.8a$ make clean; time make -j20
...
Kernel: arch/x86/boot/bzImage is ready (#11)

real 1m10.871s
user 8m22.802s
sys 0m25.030s

grant@phobos:~/linux/linux-3.14.8a$ make clean; time make -j10
...
Kernel: arch/x86/boot/bzImage is ready (#12)

real 1m10.915s
user 8m17.234s
sys 0m24.801s

For comparison, first run:
grant@pooh:~/linux/linux-3.14.5a$ make clean; time make -j10
...
Kernel: arch/x86/boot/bzImage is ready (#100)

real 3m55.915s
user 12m55.730s
sys 0m40.130s

grant@pooh:~/linux/linux-3.14.5a$ make clean; time make -j10
...
Kernel: arch/x86/boot/bzImage is ready (#101)

real 3m52.125s
user 12m57.300s
sys 0m39.060s

Grant.

[Floyd L. Davidson]

Grant <o...@grrr.id.au> wrote:
>grant@phobos:~/linux/linux-3.14.8a$ make clean; time make -j20

Any significant number for a -j argument greater than
the number of CPUs seems to be a waste, and just adds to
the amount of overhead rather than to parallel
processing.

Typically I use a shell script to determine how many
processors a machine has, and adjust the -j argument
with that.

makeopts="-j$(expr $(grep -i '^processor' /proc/cpuinfo | wc -l))"
make "${makeopts}"

With a Linux kernel though, usually I do something like this,
customizing it for each machine. This is for an 8 CPU box:

make -j8 clean
make -j3 MAKE='make -j4' bzImage || {echo "make bzImage failed"; exit 1;}

The -j3 determines how many subdirectories are worked on at one time,
and the 'make -j4' is for how many compiles are done at one time
in each subdirectory.

The target is to add up to more than the number of CPU's, but not
too much more.

--
Floyd L. Davidson http://www.apaflo.com/
Ukpeagvik (Barrow, Alaska) fl...@apaflo.com

[Grant]

One guideline I read years ago was number of processors +1, but I noticed
that a 'make -j10' left a small amount of idle time shown in top. Mainly
I was playing with having an i7-4790 ;) Kernel compiles used to take hours!

The <http://bugsplatter.id.au/kernel/boxen/phobos/> page is up now.

Cheers,
Grant.

[Henrik Carlqvist]

On Fri, 20 Jun 2014 12:02:18 +1000, Grant wrote:
> One guideline I read years ago was number of processors +1,

That was my first thought when I saw your benchmarks starting with -j 20
and then increasing the number of jobs with longer compile times as a
result.

> but I noticed that a 'make -j10' left a small amount of idle
> time shown in top.

As you have now seen that -j20 is slightly faster than both -j10 and
-j50, what about -j15 and -j25? Or maybe -j19 or -j21? A binary search
for the optimal number of jobs would be interesting.

regards Henrik
--
The address in the header is only to prevent spam. My real address is:
hc351(at)poolhem.se Examples of addresses which go to spammers:
root@localhost postmaster@localhost

[Martin]

On 06/20/2014 12:25 AM, Grant wrote:

> grant@phobos:~/linux/linux-3.14.8a$ make clean; time make -j2000
> ...
> Kernel: arch/x86/boot/bzImage is ready (#6)
>
> real 3m29.682s
> user 8m33.174s
> sys 0m30.361s

just to show off the qualities of the BFS cpu scheduler (in addition I
use the BFQ I/O scheduler):

$ make clean ; time make -j20
...

Kernel: arch/x86/boot/bzImage is ready (#3)

real 1m25.980s
user 9m33.738s
sys 0m27.682s

$ make clean ; time make -j200
...

Kernel: arch/x86/boot/bzImage is ready (#4)

real 1m26.689s
user 9m34.717s
sys 0m27.210s

$ make clean ; time make -j2000
...

Kernel: arch/x86/boot/bzImage is ready (#5)

real 1m31.043s
user 9m31.585s
sys 0m37.191s

Also, since I run my xterms in scheduling class IDLEPRIO, I can use
interactive programs without even noticing there is a compilation run
going on in the background (only the cpu fan gives it away).

[Jerry Peters]

There's also the getconf command. This is from my kernel build script:
MKOPTS="-j$(($(getconf _NPROCESSORS_ONLN) + 1))"

It's a GNU utility, but neither man nor info has anything about it.

~$ getconf --help
Usage: getconf [-v SPEC] VAR
or: getconf [-v SPEC] PATH_VAR PATH

Get the configuration value for variable VAR, or for variable
PATH_VAR
for path PATH. If SPEC is given, give values for compilation
environment SPEC.

For bug reporting instructions, please see:
<http://www.gnu.org/software/libc/bugs.html>.

[Grant]

On Fri, 20 Jun 2014 19:16:50 +0200, Martin <m...@abc.invalid> wrote:

>On 06/20/2014 12:25 AM, Grant wrote:
>

...

>Also, since I run my xterms in scheduling class IDLEPRIO, I can use
>interactive programs without even noticing there is a compilation run
>going on in the background (only the cpu fan gives it away).

I run these sessions in PuTTY terminals from my main windows box
(the old one now), as the new box is over on another table in a
temporary hookup. There's extra processing cost in sending terminal
info over link for two consoles, the other console shows 'top'.

I built a silent PC, cannot hear it across the room when it is compiling
a kernel ;)

Yesterday I was doing burnin testing with IntelBurnTest, the fans full
on simply emit a soft pink windy noise.

I'm very happy with the Nanoxia Deep Silence PWM 120mm fans.

Grant.

[Martin]

On 06/21/2014 01:00 AM, Grant wrote:
> I built a silent PC, cannot hear it across the room when it is compiling
> a kernel ;)
>
> Yesterday I was doing burnin testing with IntelBurnTest, the fans full
> on simply emit a soft pink windy noise.
>
> I'm very happy with the Nanoxia Deep Silence PWM 120mm fans.
>
> Grant.
>

I'm a great fan of silent systems too, but when creating 800% cpu load
(on 8 virtual cores) at over 120 W TDP (due to OC), I'd expect the
cooling system to become noticable...

The task of removing 120 W continuously and keeping the system
continuously stable in summer temperatures is by no means trivial. For
instance, I have built air ducts to direct outside air into the cpu fans,

[Grant]

It's winter here, and I have airco for summer, no more worries about
keeping computers cool on 40'C plus days...

Got three 120MM fans, two on the front, one at rear. Power supply is
bottom mount with separate airflow from the main box.

I'm (totally) deaf in right ear, so I will place computer to my right
when it's finished. At least these modern beasties not emit that whine
Intel cursed us with on the P4 CPUs, along with the various noisy AMDs
I had to put up with over the years.

Grant.