Google Groups no longer supports new Usenet posts or subscriptions. Historical content remains viewable.
Dismiss
Bug#804857: linux: New feature: enable CONFIG_NO_HZ_FULL and CONFIG_RCU_NOCB_CPU/CONFIG_RCU_NOCB_CPU_NONE
55 views
Skip to first unread message
Quentin Deldycke
Nov 12, 2015, 8:20:03 AM11/12/15
to
Source: linux
Severity: wishlist
Dear Maintainer,
Could you please enable kernel options CONFIG_NO_HZ_FULL, CONFIG_RCU_NOCB_CPU and CONFIG_RCU_NOCB_CPU_NONE.
As explained in: https://www.kernel.org/doc/Documentation/timers/NO_HZ.txt
This options:
- Don't modify default kernel behavior
- Enable kernel command line nohz_full to enable tickless cpus
- Enable kernel command line rcu_nocbs to offload rcu callbacks
The combination of these parameters + cpu-pinning of virtual machines greatly improve performance.
It also reduces latency in these virtual machines as the host system doesn't try to use it. (if your cgroup is correctly made)
These parameters are disabled in all debian kernels.
These parameters are enabled for various linux distribution by default without problem. (rhel 7 being one of them)
(note that rhel 7 uses CONFIG_RCU_NOCB_CPU_ALL and user can pin offload kthreads to a core)
The usage of CONFIG_RCU_NOCB_CPU_NONE doesn't offload by default. But possibly this behavior is not the best.
Quentin.
-- System Information:
Debian Release: stretch/sid
APT prefers stable-updates
APT policy: (500, 'stable-updates'), (500, 'unstable'), (500, 'testing'), (500, 'stable'), (500, 'oldstable'), (1, 'experimental')
Architecture: amd64 (x86_64)
Foreign Architectures: i386
Kernel: Linux 4.3.0-rc7-amd64 (SMP w/8 CPU cores)
Locale: LANG=en_US.UTF-8, LC_CTYPE=en_US.UTF-8 (charmap=UTF-8)
Shell: /bin/sh linked to /bin/dash
Init: systemd (via /run/systemd/system)
Severity: wishlist
Dear Maintainer,
Could you please enable kernel options CONFIG_NO_HZ_FULL, CONFIG_RCU_NOCB_CPU and CONFIG_RCU_NOCB_CPU_NONE.
As explained in: https://www.kernel.org/doc/Documentation/timers/NO_HZ.txt
This options:
- Don't modify default kernel behavior
- Enable kernel command line nohz_full to enable tickless cpus
- Enable kernel command line rcu_nocbs to offload rcu callbacks
The combination of these parameters + cpu-pinning of virtual machines greatly improve performance.
It also reduces latency in these virtual machines as the host system doesn't try to use it. (if your cgroup is correctly made)
These parameters are disabled in all debian kernels.
These parameters are enabled for various linux distribution by default without problem. (rhel 7 being one of them)
(note that rhel 7 uses CONFIG_RCU_NOCB_CPU_ALL and user can pin offload kthreads to a core)
The usage of CONFIG_RCU_NOCB_CPU_NONE doesn't offload by default. But possibly this behavior is not the best.
Quentin.
-- System Information:
Debian Release: stretch/sid
APT prefers stable-updates
APT policy: (500, 'stable-updates'), (500, 'unstable'), (500, 'testing'), (500, 'stable'), (500, 'oldstable'), (1, 'experimental')
Architecture: amd64 (x86_64)
Foreign Architectures: i386
Kernel: Linux 4.3.0-rc7-amd64 (SMP w/8 CPU cores)
Locale: LANG=en_US.UTF-8, LC_CTYPE=en_US.UTF-8 (charmap=UTF-8)
Shell: /bin/sh linked to /bin/dash
Init: systemd (via /run/systemd/system)
Ben Hutchings
Nov 12, 2015, 12:40:04 PM11/12/15
to
On Thu, 2015-11-12 at 14:13 +0100, Quentin Deldycke wrote:
> Source: linux
> Severity: wishlist
>
> Dear Maintainer,
>
> Could you please enable kernel options CONFIG_NO_HZ_FULL,
> CONFIG_RCU_NOCB_CPU and CONFIG_RCU_NOCB_CPU_NONE.
>
> As explained in: https://www.kernel.org/doc/Documentation/timers/NO_HZ.txt
>
> This options:
> - Don't modify default kernel behavior
> - Enable kernel command line nohz_full to enable tickless cpus
> - Enable kernel command line rcu_nocbs to offload rcu callbacks
>
> The combination of these parameters + cpu-pinning of virtual machines
> greatly improve performance.
> It also reduces latency in these virtual machines as the host system
> doesn't try to use it. (if your cgroup is correctly made)
[...]
> Source: linux
> Severity: wishlist
>
> Dear Maintainer,
>
> Could you please enable kernel options CONFIG_NO_HZ_FULL,
> CONFIG_RCU_NOCB_CPU and CONFIG_RCU_NOCB_CPU_NONE.
>
> As explained in: https://www.kernel.org/doc/Documentation/timers/NO_HZ.txt
>
> This options:
> - Don't modify default kernel behavior
> - Enable kernel command line nohz_full to enable tickless cpus
> - Enable kernel command line rcu_nocbs to offload rcu callbacks
>
> The combination of these parameters + cpu-pinning of virtual machines
> greatly improve performance.
> It also reduces latency in these virtual machines as the host system
> doesn't try to use it. (if your cgroup is correctly made)
But for other configurations it is worse:
config NO_HZ_FULL
bool "Full dynticks system (tickless)"
...
This is implemented at the expense of some overhead in user <-> kernel
transitions: syscalls, exceptions and interrupts. Even when it's
dynamically off.
Say N.
I think we should enable this for the rt featureset when it's next
available, but not by default.
Ben.
--
Ben Hutchings
Logic doesn't apply to the real world. - Marvin Minsky
Hillel Lubman
Feb 20, 2018, 9:10:04 PM2/20/18
to
On Thu, 12 Nov 2015 17:29:35 +0000 Ben Hutchings <b...@decadent.org.uk> wrote:
>
> But for other configurations it is worse:
>
> config NO_HZ_FULL
> bool "Full dynticks system (tickless)"
> ...
> This is implemented at the expense of some overhead in user <-> kernel
> transitions: syscalls, exceptions and interrupts. Even when it's
> dynamically off.
>
> Say N.
>
> I think we should enable this for the rt featureset when it's next
> available, but not by default.
>
> Ben.
What about CONFIG_RCU_NOCB_CPU? There is currently a nasty hardware freeze / reboot bug that
affects AMD Ryzen CPUs on Linux: https://bugzilla.kernel.org/show_bug.cgi?id=196683
Many report, that setting rcu-nocbs kernel parameter works around it at least partially, but it requires
the kernel to be built with CONFIG_RCU_NOCB_CPU=Y. Can it be enabled in Debian, or there are reasons
not to do it?
Thanks!
Hillel.
Witold Baryluk
Nov 14, 2018, 1:10:03 PM11/14/18
to
Package: src:linux
Followup-For: Bug #804857
Hi, I am just curious, and wanted to know if there is some progress on
enableing NO_HZ_FULL in generic debian on most architectures?
I wouldn't also mind separate kernel image with CONFIG_PREEMPT (full
prempetion) and HZ=1000 for desktop / soft real time work (studio audio
work, gaming, etc).
Best regards,
Witold
-- System Information:
Debian Release: buster/sid
Locale: LANG=en_US.UTF-8, LC_CTYPE=en_US.UTF-8 (charmap=UTF-8), LANGUAGE=en_US.UTF-8 (charmap=UTF-8)
Shell: /bin/sh linked to /usr/bin/dash
Init: systemd (via /run/systemd/system)
LSM: AppArmor: enabled
Followup-For: Bug #804857
Hi, I am just curious, and wanted to know if there is some progress on
enableing NO_HZ_FULL in generic debian on most architectures?
I wouldn't also mind separate kernel image with CONFIG_PREEMPT (full
prempetion) and HZ=1000 for desktop / soft real time work (studio audio
work, gaming, etc).
Best regards,
Witold
-- System Information:
Debian Release: buster/sid
Architecture: amd64 (x86_64)
Foreign Architectures: i386
Kernel: Linux 4.18.0-2-amd64 (SMP w/32 CPU cores)
Foreign Architectures: i386
Locale: LANG=en_US.UTF-8, LC_CTYPE=en_US.UTF-8 (charmap=UTF-8), LANGUAGE=en_US.UTF-8 (charmap=UTF-8)
Shell: /bin/sh linked to /usr/bin/dash
Init: systemd (via /run/systemd/system)
LSM: AppArmor: enabled
OECIPOLLETTI
Jul 11, 2019, 4:50:04 AM7/11/19
to
Good Morning,
I want to talk to you privately, kindly get back to me via my personal email: gucha...@gmail.com
Georg Müller
Oct 27, 2021, 4:30:04 AM10/27/21
to
> But for other configurations it is worse:
>
> config NO_HZ_FULL
> bool "Full dynticks system (tickless)"
> ...
> This is implemented at the expense of some overhead in user <-> kernel
> transitions: syscalls, exceptions and interrupts. Even when it's
> dynamically off.
>
> Say N.
>
Upstream commit 176b8906 changed the description regarding NO_HZ_FULL:
>
> config NO_HZ_FULL
> bool "Full dynticks system (tickless)"
> ...
> This is implemented at the expense of some overhead in user <-> kernel
> transitions: syscalls, exceptions and interrupts. Even when it's
> dynamically off.
>
> Say N.
>
https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=176b8906c399a170886ea4bad5b24763c6713d61
> By default, without passing the nohz_full parameter, this behaves just
> like NO_HZ_IDLE.
>
> If you're a distro say Y.
Salvatore Bonaccorso
Oct 30, 2021, 10:10:03 AM10/30/21
to
Control: tags -1 + moreinfo
While this is changed, and distros encouraged to select it, selecting
this would enable both CONFIG_VIRT_CPU_ACCOUNTING_GEN and
CONFIG_RCU_NOCB_CPU.
For CONFIG_VIRT_CPU_ACCOUNTING_GEN
Select this option to enable task and CPU time accounting on full
dynticks systems. This accounting is implemented by watching every
kernel-user boundaries using the context tracking subsystem.
The accounting is thus performed at the expense of some significant
overhead.
For now this is only useful if you are working on the full
dynticks subsystem development.
If unsure, say N.
which indicates some significant overhead.
And for CONFIG_RCU_NOCB_CPU
Use this option to reduce OS jitter for aggressive HPC or
real-time workloads. It can also be used to offload RCU
callback invocation to energy-efficient CPUs in battery-powered
asymmetric multiprocessors. The price of this reduced jitter
is that the overhead of call_rcu() increases and that some
workloads will incur significant increases in context-switch
rates.
This option offloads callback invocation from the set of CPUs
specified at boot time by the rcu_nocbs parameter. For each
such CPU, a kthread ("rcuox/N") will be created to invoke
callbacks, where the "N" is the CPU being offloaded, and where
the "x" is "p" for RCU-preempt (PREEMPTION kernels) and "s" for
RCU-sched (!PREEMPTION kernels). Nothing prevents this kthread
from running on the specified CPUs, but (1) the kthreads may be
preempted between each callback, and (2) affinity or cgroups can
be used to force the kthreads to run on whatever set of CPUs is
desired.
Say Y here if you need reduced OS jitter, despite added overhead.
Say N here if you are unsure.
Adding as well overhead.
Is this still to be considered true?
Regards,
Salvatore
this would enable both CONFIG_VIRT_CPU_ACCOUNTING_GEN and
CONFIG_RCU_NOCB_CPU.
For CONFIG_VIRT_CPU_ACCOUNTING_GEN
Select this option to enable task and CPU time accounting on full
dynticks systems. This accounting is implemented by watching every
kernel-user boundaries using the context tracking subsystem.
The accounting is thus performed at the expense of some significant
overhead.
For now this is only useful if you are working on the full
dynticks subsystem development.
If unsure, say N.
which indicates some significant overhead.
And for CONFIG_RCU_NOCB_CPU
Use this option to reduce OS jitter for aggressive HPC or
real-time workloads. It can also be used to offload RCU
callback invocation to energy-efficient CPUs in battery-powered
asymmetric multiprocessors. The price of this reduced jitter
is that the overhead of call_rcu() increases and that some
workloads will incur significant increases in context-switch
rates.
This option offloads callback invocation from the set of CPUs
specified at boot time by the rcu_nocbs parameter. For each
such CPU, a kthread ("rcuox/N") will be created to invoke
callbacks, where the "N" is the CPU being offloaded, and where
the "x" is "p" for RCU-preempt (PREEMPTION kernels) and "s" for
RCU-sched (!PREEMPTION kernels). Nothing prevents this kthread
from running on the specified CPUs, but (1) the kthreads may be
preempted between each callback, and (2) affinity or cgroups can
be used to force the kthreads to run on whatever set of CPUs is
desired.
Say Y here if you need reduced OS jitter, despite added overhead.
Say N here if you are unsure.
Adding as well overhead.
Is this still to be considered true?
Regards,
Salvatore
Henning Schild
Nov 4, 2021, 5:10:04 PM11/4/21
to
Am Sat, 30 Oct 2021 16:04:35 +0200
schrieb Salvatore Bonaccorso <car...@debian.org>:
I can not answer that from the back of my head. Would have to dig as
well. Might get back in about two weeks if nobody else finds an answer.
But i took the liberty to include Frederic into Cc, the author of the
"distro reassure" patch.
Not sure such a change would be allowed for bullseye (5.10) and if the
answer for 5.10 would be another than for i.e. 5.15
Reading what it is, maybe it can in fact be decoupled from NO_HZ_FULL.
Which would mean an upstream patch and backporting (if preempt would do
that, but could be considered a "performance bug" i guess)
> And for CONFIG_RCU_NOCB_CPU
>
> Use this option to reduce OS jitter for aggressive HPC or
> real-time workloads. It can also be used to offload RCU
> callback invocation to energy-efficient CPUs in
> battery-powered asymmetric multiprocessors. The price of this
> reduced jitter is that the overhead of call_rcu() increases and that
> some workloads will incur significant increases in context-switch
> rates.
>
> This option offloads callback invocation from the set of
> CPUs specified at boot time by the rcu_nocbs parameter. For each
> such CPU, a kthread ("rcuox/N") will be created to invoke
> callbacks, where the "N" is the CPU being offloaded, and
> where the "x" is "p" for RCU-preempt (PREEMPTION kernels) and "s" for
> RCU-sched (!PREEMPTION kernels). Nothing prevents this
> kthread from running on the specified CPUs, but (1) the kthreads may
> be preempted between each callback, and (2) affinity or cgroups can
> be used to force the kthreads to run on whatever set of
> CPUs is desired.
>
> Say Y here if you need reduced OS jitter, despite added
> overhead. Say N here if you are unsure.
>
> Adding as well overhead.
>
> Is this still to be considered true?
probably but only for people that actively choose to use it and only
for the CPUs they choose. "rcu_nocbs" cmdline param, if not set
everything will be as it was.
I already indicated that in the commit message of my MR:
https://salsa.debian.org/kernel-team/linux/-/merge_requests/385
regards,
Henning
>
> Regards,
> Salvatore
>
schrieb Salvatore Bonaccorso <car...@debian.org>:
well. Might get back in about two weeks if nobody else finds an answer.
But i took the liberty to include Frederic into Cc, the author of the
"distro reassure" patch.
Not sure such a change would be allowed for bullseye (5.10) and if the
answer for 5.10 would be another than for i.e. 5.15
Reading what it is, maybe it can in fact be decoupled from NO_HZ_FULL.
Which would mean an upstream patch and backporting (if preempt would do
that, but could be considered a "performance bug" i guess)
> And for CONFIG_RCU_NOCB_CPU
>
> Use this option to reduce OS jitter for aggressive HPC or
> real-time workloads. It can also be used to offload RCU
> callback invocation to energy-efficient CPUs in
> battery-powered asymmetric multiprocessors. The price of this
> reduced jitter is that the overhead of call_rcu() increases and that
> some workloads will incur significant increases in context-switch
> rates.
>
> This option offloads callback invocation from the set of
> CPUs specified at boot time by the rcu_nocbs parameter. For each
> such CPU, a kthread ("rcuox/N") will be created to invoke
> callbacks, where the "N" is the CPU being offloaded, and
> where the "x" is "p" for RCU-preempt (PREEMPTION kernels) and "s" for
> RCU-sched (!PREEMPTION kernels). Nothing prevents this
> kthread from running on the specified CPUs, but (1) the kthreads may
> be preempted between each callback, and (2) affinity or cgroups can
> be used to force the kthreads to run on whatever set of
> CPUs is desired.
>
> Say Y here if you need reduced OS jitter, despite added
> overhead. Say N here if you are unsure.
>
> Adding as well overhead.
>
> Is this still to be considered true?
for the CPUs they choose. "rcu_nocbs" cmdline param, if not set
everything will be as it was.
I already indicated that in the commit message of my MR:
https://salsa.debian.org/kernel-team/linux/-/merge_requests/385
regards,
Henning
>
> Regards,
> Salvatore
>
Salvatore Bonaccorso
Nov 7, 2021, 5:30:04 AM11/7/21
to
Hi Henning,
Thanks, lets see what Frederic can comment on it.
> Not sure such a change would be allowed for bullseye (5.10) and if the
> answer for 5.10 would be another than for i.e. 5.15
>
> Reading what it is, maybe it can in fact be decoupled from NO_HZ_FULL.
> Which would mean an upstream patch and backporting (if preempt would do
> that, but could be considered a "performance bug" i guess)
It would for now defintively be anyway for the uploads following 5.15.
Enabling it now for the bullseye kernel is mostly out of scope I
think, unless not further discussed again. But the first aim here
would be to make it for unstable.
Right, but that is not fully clear to me from the above if it's only
an overhead when as well only the parameter or if adds overhead
already by enabling the option as well.
Regards,
Salvatore
> Not sure such a change would be allowed for bullseye (5.10) and if the
> answer for 5.10 would be another than for i.e. 5.15
>
> Reading what it is, maybe it can in fact be decoupled from NO_HZ_FULL.
> Which would mean an upstream patch and backporting (if preempt would do
> that, but could be considered a "performance bug" i guess)
Enabling it now for the bullseye kernel is mostly out of scope I
think, unless not further discussed again. But the first aim here
would be to make it for unstable.
an overhead when as well only the parameter or if adds overhead
already by enabling the option as well.
Regards,
Salvatore
Frederic Weisbecker
Nov 12, 2021, 6:40:03 PM11/12/21
to
On Thu, Nov 04, 2021 at 10:05:02PM +0100, Henning Schild wrote:
CONFIG_NO_HZ_IDLE=y
# CONFIG_RCU_NOCB_CPU is not set
CONFIG_TICK_CPU_ACCOUNTING=y
Then nohz_full support has been introduced which allows userspace
tasks to run without being annoyed by tick interrupts. But ths feature
is for extreme workloads. So we arranged for this support not to add
additional overhead when it is not used.
This means that
CONFIG_NO_HZ_FULL=y
which also automatically selects:
CONFIG_RCU_NOCB_CPU=y
CONFIG_VIRT_CPU_ACCOUNTING_GEN=y
are not expected to bring more overhead than their traditional
counterparts, unless kernel boot parameters such as "nohz_full="
or "rcu_nocbs=" are passed.
So you can safely enable CONFIG_NO_HZ_FULL=y. I guess the only corner
case is when you optimize your kernel for size and you are sure you
won't have any user of nohz_full for your kernel, but I suspect some
debian users, like me for example, might be interested in that feature.
I should clarify the help text for CONFIG_VIRT_CPU_ACCOUNTING_GEN that
is definitely out of date.
Thanks.
Salvatore Bonaccorso
Nov 13, 2021, 2:50:04 AM11/13/21
to
Hi Frederic,
Thank you. I pushed the change for the next upload:
https://salsa.debian.org/kernel-team/linux/-/commit/f6aad27f05c007d6f30b34ff77bc7ea47844f117
> I should clarify the help text for CONFIG_VIRT_CPU_ACCOUNTING_GEN that
> is definitely out of date.
Ack, thank you.
Regards,
Salvatore
https://salsa.debian.org/kernel-team/linux/-/commit/f6aad27f05c007d6f30b34ff77bc7ea47844f117
> I should clarify the help text for CONFIG_VIRT_CPU_ACCOUNTING_GEN that
> is definitely out of date.
Regards,
Salvatore
Henning Schild
Nov 15, 2021, 5:50:03 AM11/15/21
to
Am Sat, 13 Nov 2021 00:27:12 +0100
schrieb Frederic Weisbecker <fred...@kernel.org>:
Thanks for the clarification. Maybe the patches dealing with the docs
should be backported also to stable kernels where applicable. So other
distros could find the "if you are a distro" i.e. in a recent 5.10 or
maybe even 4.x
regards,
Henning
> Thanks.
schrieb Frederic Weisbecker <fred...@kernel.org>:
should be backported also to stable kernels where applicable. So other
distros could find the "if you are a distro" i.e. in a recent 5.10 or
maybe even 4.x
regards,
Henning
> Thanks.
0 new messages