CONFIG_PREEMPT_DYNAMIC=y?

[Johannes Schauer Marin Rodrigues]

(please CC me, I'm not subscribed)

Hi,

I'm wondering whether it'd make sense for the Debian kernel to be built with
CONFIG_PREEMPT_DYNAMIC=y. Sorry if I have missed existing discussion on that
topic but I haven't found any.

Context: I'm maintaining a fork of the Debian linux kernel (thank you so much
for your amazing work!) with a couple of patches and config options on top so
that the Debian kernel can work on the imx8mq, imx8mplus, bananapi a311d and
ls1028 SoM for the MNT Reform open source laptop. A user suggested that we also
enable PREEMPT_DYNAMIC for our custom kernel builds:

https://source.mnt.re/reform/reform-debian-packages/-/merge_requests/18

I do not have enough insight to make this choice and was wondering whether it
would make sense for the Debian kernel to be built with that option enabled?
And if not: why is it not done? As far as I understand its documentation, the
option is meant for distros to enable it so that users can choose at boot-time
instead of at compile time preempt=none, voluntary or full.

Thanks!

cheers, josch

[Diederik de Haas]

On 8 October 2023 18:20:20 CEST Johannes Schauer Marin Rodrigues wrote:
> I'm wondering whether it'd make sense for the Debian kernel to be built with
> CONFIG_PREEMPT_DYNAMIC=y.

Thanks for raising this, as the current situation appears a bit 'weird'.

======= PC (amd64) ======
$ uname -a
Linux bagend 6.5.0-1-amd64 #1 SMP PREEMPT_DYNAMIC Debian 6.5.3-1 (2023-09-13) x86_64 GNU/Linux

$ grep PREEMPT_DYNAMIC /boot/config-6.5.0-1-amd64
CONFIG_PREEMPT_DYNAMIC=y
CONFIG_HAVE_PREEMPT_DYNAMIC=y
CONFIG_HAVE_PREEMPT_DYNAMIC_CALL=y
========================

======= SBC (arm64) ======
$ uname -a
Linux cs21 6.1.0-13-arm64 #1 SMP Debian 6.1.55-1 (2023-09-29) aarch64 GNU/Linux

$ grep PREEMPT_DYNAMIC /boot/config-6.1.0-13-arm64
# CONFIG_PREEMPT_DYNAMIC is not set
CONFIG_HAVE_PREEMPT_DYNAMIC=y
CONFIG_HAVE_PREEMPT_DYNAMIC_KEY=y
=========================
(grep output is the same on a 6.6 kernel I use on another arm64 SBC)

In the Debian kernel it's not explicitly enabled and it's having a different
effect on amd64 compared to arm64.
In any case, I think it would be useful if this setting was explicitly set or
unset, or the above situation been made explicit ... with a nice commit message
explaining why those settings were made.

Cheers,
Diederik

[Emanuele Rocca]

Hi,

On 2023-10-08 07:17, Diederik de Haas wrote:
> $ grep PREEMPT_DYNAMIC /boot/config-6.5.0-1-amd64
> CONFIG_PREEMPT_DYNAMIC=y
> CONFIG_HAVE_PREEMPT_DYNAMIC=y
> CONFIG_HAVE_PREEMPT_DYNAMIC_CALL=y

[...]

> $ grep PREEMPT_DYNAMIC /boot/config-6.1.0-13-arm64
> # CONFIG_PREEMPT_DYNAMIC is not set
> CONFIG_HAVE_PREEMPT_DYNAMIC=y
> CONFIG_HAVE_PREEMPT_DYNAMIC_KEY=y

CONFIG_PREEMPT_DYNAMIC is set to 'y' by default on amd64 due to
HAVE_PREEMPT_DYNAMIC_CALL being 'y', see:
https://sources.debian.org/src/linux/6.5.6-1/kernel/Kconfig.preempt/?hl=101#L101

arm64 does not have PREEMPT_DYNAMIC_CALL, this is why PREEMPT_DYNAMIC is
not set by default there. However, there's another feature which is
functionally equivalent, PREEMPT_DYNAMIC_KEY. While _CALL and _KEY are
equivalent, in theory _KEY may have more overhead. See:
https://lore.kernel.org/lkml/20220214165216.223...@arm.com/#t

SUSE and Fedora both seem to have CONFIG_HAVE_PREEMPT_DYNAMIC=y on
arm64, see:
https://github.com/SUSE/kernel-source/blob/SLE15-SP6/config/arm64/default#L123
https://src.fedoraproject.org/rpms/kernel/blob/f34/f/kernel.spec#_4458

Setting CONFIG_HAVE_PREEMPT_DYNAMIC=y on arm64 is likely safe for us
too, but we may want to run some benchmark first to see if there are any
noticeable slowdowns.

[Johannes Schauer Marin Rodrigues]

Hi,

Quoting Emanuele Rocca (2023-10-10 12:10:07)

> Setting CONFIG_HAVE_PREEMPT_DYNAMIC=y on arm64 is likely safe for us too, but
> we may want to run some benchmark first to see if there are any noticeable
> slowdowns.

I have a couple of arm64 boards here that I can run benchmarks on. What would
you recommend to run to test the effect of CONFIG_HAVE_PREEMPT_DYNAMIC=y?

Thanks!

cheers, josch

[Emanuele Rocca]

Hello Johannes,

I'm not sure what the best benchmark would be, but maybe you could start
with something generic like timing a kernel build with and without
PREEMPT_DYNAMIC and throwing 'hey' at your favorite web server?
https://github.com/rakyll/hey

[Diederik de Haas]

On Tuesday, 10 October 2023 12:10:07 CEST Emanuele Rocca wrote:
> CONFIG_PREEMPT_DYNAMIC is set to 'y' by default on amd64 due to
> HAVE_PREEMPT_DYNAMIC_CALL being 'y', see:
> https://sources.debian.org/src/linux/6.5.6-1/kernel/Kconfig.preempt/?hl=101#
> L101
>
> arm64 does not have PREEMPT_DYNAMIC_CALL, this is why PREEMPT_DYNAMIC is
> not set by default there.

Neither does amd64. So it appears something else is causing
CONFIG_PREEMPT_DYNAMIC to be enabled on amd64.

On Sunday, 8 October 2023 19:17:58 CEST Diederik de Haas wrote:
> In the Debian kernel it's not explicitly enabled and it's having a different
> effect on amd64 compared to arm64.

I think it should be explicitly enabled (or disabled) and not be it dependent
on some other, possibly unrelated, Kconfig option being enabled

me@pc:~/dev/debian/salsa/kernel-team/linux$ grep -rn PREEMPT debian/config/
debian/config/config:6446:# CONFIG_PREEMPT_NONE is not set
debian/config/config:6447:CONFIG_PREEMPT_VOLUNTARY=y
debian/config/config:6448:# CONFIG_PREEMPT is not set
debian/config/config:6554:# CONFIG_PREEMPT_TRACER is not set
debian/config/config:6633:CONFIG_DEBUG_PREEMPT=y
debian/config/featureset-rt/config:5:# CONFIG_PREEMPT_VOLUNTARY is not set
debian/config/featureset-rt/config:6:CONFIG_PREEMPT_RT=y
debian/config/featureset-rt/defines:5:part-long-rt: This kernel includes the PREEMPT_RT realtime patch set.
debian/config/featureset-rt/defines:6:part-short-rt: PREEMPT_RT
debian/config/kernelarch-mips/config.loongson-3:110:# CONFIG_PREEMPT_VOLUNTARY is not set
debian/config/kernelarch-mips/config.loongson-3:111:CONFIG_PREEMPT=y

In commit 89b4f0559d598f5b61ad26fe12d787c21b1aea61 (dd 2009-09-25)
CONFIG_PREEMPT_VOLUNTARY was enabled and its Kconfig entry mentions this:
"Select this if you are building a kernel for a desktop system."

The PREEMPT_DYNAMIC (and related options) are NOT en-/disabled explicitly.
The PREEMPT_DYNAMIC Kconfig entry mentions the following:
1. "This option allows to define the preemption model on the kernel command
line parameter and thus override the default preemption model"
2. "The feature is primarily interesting for Linux distributions"
3. "Interesting if you want the same pre-built kernel should be used for
both Server and Desktop workloads."

ad 1) I _think_ running benchmarks is not that useful as its result will/should
vary based on the default preemption model and whether that is overridden
on the kernel command line
ad 3) It seems (to me) that it is a better choice then PREEMPT_VOLUNTARY for
Debian as it's used both as Server and Desktop, on various architectures.

I think https://bugs.debian.org/1016151#15 means that PREEMPT_DYNAMIC
should be enabled, it just hasn't been made explicit in Debian's kernel config.

[Vincent Blut]

Hi,

Le 2023-10-10 13:54, Diederik de Haas a écrit :
> On Tuesday, 10 October 2023 12:10:07 CEST Emanuele Rocca wrote:
> > CONFIG_PREEMPT_DYNAMIC is set to 'y' by default on amd64 due to
> > HAVE_PREEMPT_DYNAMIC_CALL being 'y', see:
> > https://sources.debian.org/src/linux/6.5.6-1/kernel/Kconfig.preempt/?hl=101#
> > L101
> >
> > arm64 does not have PREEMPT_DYNAMIC_CALL, this is why PREEMPT_DYNAMIC is
> > not set by default there.
>
> Neither does amd64. So it appears something else is causing
> CONFIG_PREEMPT_DYNAMIC to be enabled on amd64.

From the commit message introducing PREEMPT_DYNAMIC:¹

CONFIG_PREEMPT_DYNAMIC is automatically selected by CONFIG_PREEMPT if
the architecture provides the necessary support (CONFIG_STATIC_CALL_INLINE,
CONFIG_GENERIC_ENTRY, and provide with __preempt_schedule_function() /
__preempt_schedule_notrace_function()).

Cheers,
Vincent

¹ https://lore.kernel.org/all/20210118141223.1...@kernel.org/T/#u

[Emanuele Rocca]

Hi Diederik,

On 2023-10-10 01:54, Diederik de Haas wrote:
> On Tuesday, 10 October 2023 12:10:07 CEST Emanuele Rocca wrote:
> > CONFIG_PREEMPT_DYNAMIC is set to 'y' by default on amd64 due to
> > HAVE_PREEMPT_DYNAMIC_CALL being 'y', see:
> > https://sources.debian.org/src/linux/6.5.6-1/kernel/Kconfig.preempt/?hl=101#
> > L101
> >
> > arm64 does not have PREEMPT_DYNAMIC_CALL, this is why PREEMPT_DYNAMIC is
> > not set by default there.
>
> Neither does amd64. So it appears something else is causing
> CONFIG_PREEMPT_DYNAMIC to be enabled on amd64.

Here is where HAVE_PREEMPT_DYNAMIC_CALL is set on x86:
https://sources.debian.org/src/linux/6.5.6-1/arch/x86/Kconfig/?hl=271#L271

> I think it should be explicitly enabled (or disabled) and not be it dependent
> on some other, possibly unrelated, Kconfig option being enabled

Right, but HAVE_PREEMPT_DYNAMIC_CALL is not a config option. It's a
setting that defines whether the architecture can use inline static
calls to handle the preemption model passed to the kernel command line
with preempt=

To my understanding, only x86 can do that. From the PREEMPT_DYNAMIC
Kconfig help:

The runtime overhead is negligible with HAVE_STATIC_CALL_INLINE enabled
but if runtime patching is not available for the specific architecture
then the potential overhead should be considered.

> ad 1) I _think_ running benchmarks is not that useful as its result will/should
> vary based on the default preemption model and whether that is overridden
> on the kernel command line

Sorry, I wasn't clear in my previous message about what we should be
benchmarking. The current Debian kernel on arm64 has PREEMPT_VOLUNTARY=y
and PREEMPT_DYNAMIC not set. We want to run benchmarks to see if a
kernel with PREEMPT_DYNAMIC=y booted with preempt=voluntary has any
significant slowdowns compared to the current kernels. We don't want to
benchmark the various preemption models.

[Diederik de Haas]

On Tuesday, 10 October 2023 14:17:50 CEST Vincent Blut wrote:
> > Neither does amd64. So it appears something else is causing
> > CONFIG_PREEMPT_DYNAMIC to be enabled on amd64.
>
> From the commit message introducing PREEMPT_DYNAMIC:¹
>
> CONFIG_PREEMPT_DYNAMIC is automatically selected by CONFIG_PREEMPT if
> the architecture provides the necessary support
> (CONFIG_STATIC_CALL_INLINE, CONFIG_GENERIC_ENTRY, and provide with
> __preempt_schedule_function() / __preempt_schedule_notrace_function()).

And CONFIG_PREEMPT was explicitly disabled on 2007-05-21 in
8ec9ee1435ba3ac97f9f0e7c5b879aac8d636f4a (I couldn't find an older commit).
So I guess the heuristic isn't applied as the precondition already fails.

Thanks for finding that part :-)

[Marc Haber]

Whatever you do, check whether having a serial console still works with
that setting. I had trouble with that in kernel 6.5, had some fruitless
discussions on the LKML and developer interest abruptly died when I
found out that the bug I encountered could be worked around by having
those settings:

[1/4998]mh@fan:~ $ grep PREEMPT /boot/config-6.5.6-zgws1
CONFIG_PREEMPT_BUILD=y

# CONFIG_PREEMPT_NONE is not set

CONFIG_PREEMPT_VOLUNTARY=y

# CONFIG_PREEMPT is not set

CONFIG_PREEMPT_COUNT=y
CONFIG_PREEMPTION=y
CONFIG_PREEMPT_DYNAMIC=y
CONFIG_PREEMPT_RCU=y
CONFIG_HAVE_PREEMPT_DYNAMIC=y
CONFIG_HAVE_PREEMPT_DYNAMIC_CALL=y
CONFIG_PREEMPT_NOTIFIERS=y
CONFIG_DRM_I915_PREEMPT_TIMEOUT=640
CONFIG_DRM_I915_PREEMPT_TIMEOUT_COMPUTE=7500
# CONFIG_DEBUG_PREEMPT is not set

# CONFIG_PREEMPT_TRACER is not set

# CONFIG_PREEMPTIRQ_DELAY_TEST is not set

Greetings
Marc


--
-----------------------------------------------------------------------------
Marc Haber | "I don't trust Computers. They | Mailadresse im Header
Leimen, Germany | lose things." Winona Ryder | Fon: *49 6224 1600402
Nordisch by Nature | How to make an American Quilt | Fax: *49 6224 1600421

[Diederik de Haas]

Hi Emanuele,

On Tuesday, 10 October 2023 15:02:07 CEST Emanuele Rocca wrote:
> On 2023-10-10 01:54, Diederik de Haas wrote:
> > On Tuesday, 10 October 2023 12:10:07 CEST Emanuele Rocca wrote:
> > > CONFIG_PREEMPT_DYNAMIC is set to 'y' by default on amd64
> > > due to HAVE_PREEMPT_DYNAMIC_CALL being 'y'
> > >

> > > arm64 does not have PREEMPT_DYNAMIC_CALL, this is why PREEMPT_DYNAMIC is
> > > not set by default there.

"by default" would be my emphasis

$ grep -A6 "config PREEMPT_DYNAMIC" kernel/Kconfig.preempt
config PREEMPT_DYNAMIC
bool "Preemption behaviour defined on boot"
depends on HAVE_PREEMPT_DYNAMIC && !PREEMPT_RT
select JUMP_LABEL if HAVE_PREEMPT_DYNAMIC_KEY
select PREEMPT_BUILD
default y if HAVE_PREEMPT_DYNAMIC_CALL
help

So HAVE_PREEMPT_DYNAMIC is the requirement afaict.
As you said in your other mail, that can be done by _CALL, but also by _KEY.

$ grep -rnE "(select|depends on) HAVE_PREEMPT_DYNAMIC_(KEY|CALL)"
arch/arm64/Kconfig:224: select HAVE_PREEMPT_DYNAMIC_KEY
arch/riscv/Kconfig:137: select HAVE_PREEMPT_DYNAMIC_KEY if !XIP_KERNEL
arch/x86/Kconfig:273: select HAVE_PREEMPT_DYNAMIC_CALL

On arm64 it's selected if ARM64 is set and on risv when RISCV is set.

> Right, but HAVE_PREEMPT_DYNAMIC_CALL is not a config option.

It looks like any config option starting with HAVE_ is not selectable.

On Tuesday, 10 October 2023 12:10:07 CEST Emanuele Rocca wrote:

> SUSE and Fedora both seem to have CONFIG_HAVE_PREEMPT_DYNAMIC=y on
> arm64, see:
> https://github.com/SUSE/kernel-source/blob/SLE15-SP6/config/arm64/default#L1
> 23 https://src.fedoraproject.org/rpms/kernel/blob/f34/f/kernel.spec#_4458
>

> Setting CONFIG_HAVE_PREEMPT_DYNAMIC=y on arm64 is likely safe for us
> too, but we may want to run some benchmark first to see if there are any
> noticeable slowdowns.

Similarly, you *can't* explicitly/directly enable HAVE_PREEMPT_DYNAMIC.

> > I think it should be explicitly enabled (or disabled) and not be it
> > dependent on some other, possibly unrelated, Kconfig option being enabled

It seems an important difference between your view and mine is that you look at
the end result, while I look at what's configured in the Debian kernel repo.

While some options are enabled in the end result, NONE is configured in the
Debian kernel repo. And that could have major, unwanted, consequences.

I don't know if it's the right call to do so, but PREEMPT_DYNAMIC *is*
directly/explicitly selectable and if that's what we want, then we should make
that setting in the Debian kernel repo.

> From the PREEMPT_DYNAMIC Kconfig help:
>
> The runtime overhead is negligible with HAVE_STATIC_CALL_INLINE enabled
> but if runtime patching is not available for the specific architecture
> then the potential overhead should be considered.
>
> > ad 1) I _think_ running benchmarks is not that useful as its result
> > will/should vary based on the default preemption model and whether that
> > is overridden on the kernel command line
>
> Sorry, I wasn't clear in my previous message about what we should be
> benchmarking. The current Debian kernel on arm64 has PREEMPT_VOLUNTARY=y
> and PREEMPT_DYNAMIC not set. We want to run benchmarks to see if a
> kernel with PREEMPT_DYNAMIC=y booted with preempt=voluntary has any
> significant slowdowns compared to the current kernels. We don't want to
> benchmark the various preemption models.

Ok, thanks for the clarification.

[Diederik de Haas]

On Tuesday, 17 October 2023 19:07:30 CEST Ben Hutchings wrote:
> > > > I think it should be explicitly enabled (or disabled) and not be it
> > > > dependent on some other, possibly unrelated, Kconfig option being
> > > > enabled
> >
> > It seems an important difference between your view and mine is that you
> > look at the end result, while I look at what's configured in the Debian
> > kernel repo.
> >
> > While some options are enabled in the end result, NONE is configured in
> > the Debian kernel repo. And that could have major, unwanted, consequences.
>

> Where are you seeing that?

[salsa.d.o/kernel-team/linux]$ grep -r PREEMPT_DYNAMIC debian/config/
[0 results]

> We enable CONFIG_PREEMPT_VOLUNTARY in debian/config/config.
> That's replaced with CONFIG_PREEMPT_RT for the RT featureset, and with
> CONFIG_PREEMPT for loongson-3 (as a bug workaround).
>
> The actual kernel configurations for 6.4 and 6.5 seem to be consistent
> with this, except for the addition of CONFIG_PREEMPT_DYNAMIC on x86
> non-RT configurations.

Yes, that's what I meant with 'end result':

[amd64]$ grep PREEMPT_DYNAMIC /boot/config-6.5.0-2-amd64
CONFIG_PREEMPT_DYNAMIC=y
CONFIG_HAVE_PREEMPT_DYNAMIC=y
CONFIG_HAVE_PREEMPT_DYNAMIC_CALL=y

[arm64]$ grep PREEMPT_DYNAMIC /boot/config-6.1.0-13-arm64

# CONFIG_PREEMPT_DYNAMIC is not set
CONFIG_HAVE_PREEMPT_DYNAMIC=y
CONFIG_HAVE_PREEMPT_DYNAMIC_KEY=y

(I don't think the kernel version difference matters in this case)

[Emanuele Rocca]

Hi,

On 2023-10-08 06:20, Johannes Schauer Marin Rodrigues wrote:
> I'm wondering whether it'd make sense for the Debian kernel to be built with
> CONFIG_PREEMPT_DYNAMIC=y.

On amd64 we already have PREEMPT_DYNAMIC=y, so I thought of running some
benchmarks on my Ryzen system to evaluate the performance overhead it
introduces.

To my surprise, I found that in Debian you get a 20% (!) performance
improvement on kernel-heavy workloads simply by setting PREEMPT_DYNAMIC
to 'n'. I brought up the topic on LKML [1], and it turns out that what
is really eating performance for us is DEBUG_PREEMPT=y. There is a
performance penalty in PREEMPT_DYNAMIC=y as well, but it's much smaller,
around 3 or 4%.

The Kconfig help entry for DEBUG_PREEMPT also clearly mentions the
performance overhead:

This option has potential to introduce high runtime overhead,
depending on workload as it triggers debugging routines for each
this_cpu operation. It should only be used for debugging purposes.

Regardless of what we are going to decide about PREEMPT_DYNAMIC, we
surely have to disable DEBUG_PREEMPT. I opened [2] for that.

[1] https://lore.kernel.org/lkml/ZTJFA_Ac6nWawIHb@ariel/
[2] https://salsa.debian.org/kernel-team/linux/-/merge_requests/874