[gentoo-user] emerge times blown out

[Adam Carter]

I have three systems (all ~arch) and the emerge times have blown out on all of them across all packages. Worst example appears to be;

    Fri Dec 23 13:11:44 2022 >>> net-libs/webkit-gtk-2.38.3-r410
       merge time: 37 minutes and 8 seconds.

     Fri Dec 23 13:43:08 2022 >>> net-libs/webkit-gtk-2.38.3
       merge time: 31 minutes and 24 seconds.

     Sat Feb  4 21:16:40 2023 >>> net-libs/webkit-gtk-2.38.4-r410
       merge time: 6 hours, 53 minutes and 28 seconds.

     Sun Feb  5 04:17:12 2023 >>> net-libs/webkit-gtk-2.38.4
       merge time: 7 hours and 32 seconds.

Is anyone else seeing this?

[Dale]

A lot of this will obviously depend on CPU speed, number of
cores/threads, amount of memory and also, is it compiling on its own or
are other packages also being compiled and taking up CPU time or other
things using CPU time as well.  Here, AMD CPU at 4GHz with 8 cores. 
32GBs of memory and most packages built on tmpfs, except large
packages.  This is a recent few examples. 


     Sat Aug  6 06:34:31 2022 >>> net-libs/webkit-gtk-2.36.5-r1
       merge time: 1 hour, 39 minutes and 13 seconds.

     Sun Aug 21 16:04:23 2022 >>> net-libs/webkit-gtk-2.36.6
       merge time: 3 hours, 41 minutes and 44 seconds.

     Sat Sep  3 22:29:36 2022 >>> net-libs/webkit-gtk-2.36.7
       merge time: 2 hours, 41 minutes and 7 seconds.



I left out binary emerges and such.  Oldest at top, newest at bottom.  I
suspect for the middle and most likely the bottom one, there was another
package compiling as well.  It's possible the top one was either on its
own part or all of the time. 

It's really hard to compare these things.  Even if our rigs are close in
speed/power/whatever, there can still be a lot of things that affect the
compile time.  I run KDE, watch TV from computer, have torrent software
that runs basically 24/7 plus other stuff running.  If you for example
use Fluxbox and have little else running, even with the same CPU and
memory, compile times will vary. 

Does that info help? 

Dale

:-)  :-) 

[Adam Carter]

Does that info help? 

My reason for asking is that i'm seeing this across multiple systems, 2 AMD, 1 Intel, who's configuration hasn't really changed and while there is some variance there has been a step change late December / early January. Another example

    Sat Nov 26 14:34:50 2022 >>> sys-apps/systemd-252.2
       merge time: 2 minutes and 19 seconds.

     Sat Dec 10 20:59:29 2022 >>> sys-apps/systemd-252.3
       merge time: 1 minute and 54 seconds.

     Wed Dec 14 13:56:52 2022 >>> sys-apps/systemd-252.3
       merge time: 2 minutes and 56 seconds.

     Wed Dec 21 20:08:36 2022 >>> sys-apps/systemd-252.4
       merge time: 3 minutes and 7 seconds.

     Tue Jan  3 22:29:43 2023 >>> sys-apps/systemd-252.4
       merge time: 12 minutes and 42 seconds.

     Thu Jan 12 14:56:32 2023 >>> sys-apps/systemd-252.4-r1
       merge time: 22 minutes and 12 seconds.

     Sat Jan 21 12:00:06 2023 >>> sys-apps/systemd-252.4-r1
       merge time: 12 minutes and 3 seconds.

     Mon Jan 30 15:41:44 2023 >>> sys-apps/systemd-252.5
       merge time: 21 minutes and 45 seconds.

     Fri Feb 17 21:18:21 2023 >>> sys-apps/systemd-252.6
       merge time: 22 minutes and 18 seconds.

[Dale]

The ones I listed before also jumped in compile times.  As I said tho, I don't know if other things compiling affected that time.  Still, it does seem to have increased but I remember when I was on my old single core rig with just a few GBs of memory.  As time goes by, software gets bigger therefore takes longer to compile.  Yours from the 4th to the 6th in the list sure does increase.  That's 8 to 10 times longer roughly.  That's a large difference.  A true test, emerge something interesting all by itself.  See what it comes closest to, the old times or the newer and longer times. 

I suspect this is changes in features of software and could even be related to gcc or some other tool compiling uses.  It is a interesting jump.  I don't think you are alone in this.  Maybe someone else will post their info.  For those interested, genlop -t <package name> is how to get this info. 

BTW, I don't use systemd so I can't list mine.  ;-)

Dale

:-)  :-)

[Alan J. Wylie]

All my three systems are fine

AMD FX(tm)-8350 Eight-Core Processor, 16GiB

Sun Oct 2 13:07:33 2022 >>> sys-apps/systemd-251.4
merge time: 3 minutes and 35 seconds.

Fri Nov 25 08:06:10 2022 >>> sys-apps/systemd-251.7
merge time: 3 minutes and 24 seconds.

Sat Dec 10 10:11:16 2022 >>> sys-apps/systemd-251.8
merge time: 3 minutes and 19 seconds.

Tue Jan 3 08:43:49 2023 >>> sys-apps/systemd-252.4
merge time: 3 minutes and 22 seconds.

Sat Jan 28 06:28:16 2023 >>> sys-apps/systemd-252.4-r1
merge time: 5 minutes and 24 seconds.

AMD FX(tm)-4300 Quad-Core Processor, 16GiB

Sun Oct 2 13:42:48 2022 >>> sys-apps/systemd-251.4
merge time: 5 minutes and 32 seconds.

Fri Nov 25 08:11:27 2022 >>> sys-apps/systemd-251.7
merge time: 5 minutes and 40 seconds.

Sat Dec 10 12:31:38 2022 >>> sys-apps/systemd-251.8
merge time: 5 minutes and 36 seconds.

Tue Jan 3 08:47:13 2023 >>> sys-apps/systemd-252.4
merge time: 5 minutes and 36 seconds.

Sat Jan 28 06:28:27 2023 >>> sys-apps/systemd-252.4-r1
merge time: 6 minutes and 17 seconds.

Intel(R) Atom(TM) CPU N2800 @ 1.86GHz, 4GiB, not systemd

Wed Sep 28 08:32:50 2022 >>> net-dns/bind-9.16.33
merge time: 15 minutes and 28 seconds.

Fri Jan 20 07:50:41 2023 >>> net-dns/bind-9.16.36
merge time: 16 minutes and 23 seconds.

Thu Feb 16 07:33:46 2023 >>> net-dns/bind-9.16.37
merge time: 17 minutes and 9 seconds.

--
Alan J. Wylie https://www.wylie.me.uk/

Dance like no-one's watching. / Encrypt like everyone is.
Security is inversely proportional to convenience

[Stefan Schmiedl]

Samstag, 18. Februar 2023 01:49:

When I had something like this about two years ago, the culprit was me
setting the cpufreq thingie in the kernel config to powersave or whatever
it's called ... running emerge with 800 MHz instead of 3.2 GHz produced
effects similar to yours.

s. 
 

[Grant Edwards]

The same thing happened to me recently. Somehow, I broke the load/temp
based CPU clock scaling on one of my machines, and it suddenly took 4X
as long to build things as my other machines. It was running at the
lowest clock speed possible all the time. Unfortunately, I don't
remember exactly what I did to fix it...

--
Grant

[Stefan Schmiedl]

Samstag, 18. Februar 2023 19:05:

 
>
>On 2023-02-18, Stefan Schmiedl <s...@xss.de> wrote:
>>
>>Samstag, 18. Februar 2023 01:49:
>>>
>>>I have three systems (all ~arch) and the emerge times have blown out on all of them across all packages.
>>

>>When I had something like this about two years ago, the culprit was me
>>setting the cpufreq thingie in the kernel config to powersave or whatever
>>it's called ... running emerge with 800 MHz instead of 3.2 GHz produced
>>effects similar to yours.

>
>The same thing happened to me recently. Somehow, I broke the load/temp
>based CPU clock scaling on one of my machines, and it suddenly took 4X
>as long to build things as my other machines. It was running at the
>lowest clock speed possible all the time. Unfortunately, I don't
>remember exactly what I did to fix it...

CONFIG_CPU_FREQ_GOV_PERFORMANCE
CONFIG_CPU_FREQ_GOV_POWERSAVE
CONFIG_CPU_FREQ_GOV_USERSPACE
CONFIG_CPU_FREQ_GOV_ONDEMAND
CONFIG_CPU_FREQ_GOV_CONSERVATIVE
CONFIG_CPU_FREQ_GOV_SCHEDUTIL

Back when I finally found it, all of these were set to "n" but for POWERSAVE which was set to "y".
I rebuilt the kernel to use ONDEMAND and things were back to normal.

s.

[Adam Carter]

Thanks everyone for your suggestions. I've checked the frequencies of the cores and they are scaling properly:

cpu MHz         : 4024.653
cpu MHz         : 4024.678
cpu MHz         : 4024.639
cpu MHz         : 4024.605

cpu MHz         : 4024.643

etc

Will continue to pursue these lines of thought.

[Alexander Puchmayr]

Could it be that some kind of Spectre mitigation is active? I just read about
some massive performance problems in Kernel 5.19+ on Skylake CPUs. Stable
gentoo kernel was upgraded to 6.1 recently, which could also be affected by
this problem.
See https://lkml.iu.edu/hypermail/linux/kernel/2209.1/02248.html

Try turning the mitigations off or revert to a 5.15.x kernel.

Alex

[Adam Carter]

Could it be that some kind of Spectre mitigation is active? I just read about
some massive performance problems in Kernel 5.19+ on Skylake CPUs. Stable
gentoo kernel was upgraded to 6.1 recently, which could also be affected by
this problem.
See https://lkml.iu.edu/hypermail/linux/kernel/2209.1/02248.html

Try turning the mitigations off or revert to a 5.15.x kernel.

I have one skylake and its now running 6.2 with retbleed=stuff. The others are AMD.

I will try 5.15.x. I have .configs from that time so it will rule kernel stuff in/out.