RT/Linux SCHED_RR/_FIXED to combat latency?
Cedric Blancher
We're experiencing rather very bad latency spikes on busy Linux
systems, for example if one machine is the jumphost (ssh -J) for a few
hundred connections, while at the same time handles CPU intensive
tasks.
Would RT/Linux SCHED_FIXED or SCHED_RR be of help in such a case, e.g.
put all ssh processes into the SCHED_FIXED scheduling class, with a
priority higher than the non-interactive compute processes?
Also, do I interpret it correctly that each forwarded TCP connection
has its own process?!
Ced
--
Cedric Blancher <cedric....@gmail.com>
[https://plus.google.com/u/0/+CedricBlancher/]
Institute Pasteur
_______________________________________________
openssh-unix-dev mailing list
openssh-...@mindrot.org
https://lists.mindrot.org/mailman/listinfo/openssh-unix-dev
Thorsten Glaser
>We're experiencing rather very bad latency spikes on busy Linux
>systems, for example if one machine is the jumphost (ssh -J) for a few
>hundred connections, while at the same time handles CPU intensive
>tasks.
>
>Would RT/Linux SCHED_FIXED or SCHED_RR be of help in such a case, e.g.
Did you already check the old and tried method of nice(2)?
If the other load is CPU-intensive, this is usually sufficient.
Normally you’d nice the CPU-intensive load (so anything else on
the system is not affected), but you can also negative-nice the
sshd processes (and therefore, the children) which however may
not be sufficient and could require to negative-nice some other
processes or kernel tasks as well, so see if your scenario can
just positive-nice the load instead.
gl hf,
//mirabilos
--
Infrastrukturexperte • tarent solutions GmbH
Am Dickobskreuz 10, D-53121 Bonn • http://www.tarent.de/
Telephon +49 228 54881-393 • Fax: +49 228 54881-235
HRB AG Bonn 5168 • USt-ID (VAT): DE122264941
Geschäftsführer: Dr. Stefan Barth, Kai Ebenrett, Boris Esser, Alexander Steeg
****************************************************
/⁀\ The UTF-8 Ribbon
╲ ╱ Campaign against Mit dem tarent-Newsletter nichts mehr verpassen:
╳ HTML eMail! Also, https://www.tarent.de/newsletter
╱ ╲ header encryption!
****************************************************
Cedric Blancher
>
> On Thu, 10 Aug 2023, Cedric Blancher wrote:
>
> >We're experiencing rather very bad latency spikes on busy Linux
> >systems, for example if one machine is the jumphost (ssh -J) for a few
> >hundred connections, while at the same time handles CPU intensive
> >tasks.
> >
> >Would RT/Linux SCHED_FIXED or SCHED_RR be of help in such a case, e.g.
>
> Did you already check the old and tried method of nice(2)?
>
> If the other load is CPU-intensive, this is usually sufficient.
Also, I think nice(2) is to govern scheduler priorities, but does
little to improve latency
Ced
--
Cedric Blancher <cedric....@gmail.com>
[https://plus.google.com/u/0/+CedricBlancher/]
Institute Pasteur
Damien Miller
>
> We're experiencing rather very bad latency spikes on busy Linux
> systems, for example if one machine is the jumphost (ssh -J) for a few
> hundred connections, while at the same time handles CPU intensive
> tasks.
>
> Would RT/Linux SCHED_FIXED or SCHED_RR be of help in such a case, e.g.
> put all ssh processes into the SCHED_FIXED scheduling class, with a
> priority higher than the non-interactive compute processes?
scheduling class isn't likely to help.
> Also, do I interpret it correctly that each forwarded TCP connection
> has its own process?!
ControlMaster/ControlPersist) then connections from the same user
through the same jump host can be shared.
-d
Nico Kadel-Garcia
<cedric....@gmail.com> wrote:
>
> Good morning!
>
> We're experiencing rather very bad latency spikes on busy Linux
> systems, for example if one machine is the jumphost (ssh -J) for a few
> hundred connections, while at the same time handles CPU intensive
> tasks.
>
> Would RT/Linux SCHED_FIXED or SCHED_RR be of help in such a case, e.g.
> put all ssh processes into the SCHED_FIXED scheduling class, with a
> priority higher than the non-interactive compute processes?
Real Time Linux doesn't solve these problems. It attempts to handle
the variety of interrupts more consistently and equitably, but
precisely the sort of "I'm too danged busy with this high priority
process" issues of a highly burdened server are likely to be
*aggraveted* by the incorrect guesses of what processes really matter
on a "real-time" system.
If you know which problems are most important and can raise their
priority, great, but unpredictable delays are usually the sign of a
"too-busy with too many processes" kernel.
Darren Tucker
[...]
> systems, for example if one machine is the jumphost (ssh -J) for a few
> hundred connections, while at the same time handles CPU intensive
> tasks.
Connection establishment is the most computationally expensive part of
the process by some margin, and if you have clients synchronized I
could imagine that causing load spikes.
If that's the case you could try disabling the more expensive key
exchange algorithms ("KexAlgorithms in the config of either the client
or server) or host key algos (HostKeyAlgorithms in the server config).
Try benchmarking the available options, but I'd bet the post-quantum
safe default KexAlgorithm (sntrup761x2...@openssh.com) is the
most expensive one.
--
Darren Tucker (dtucker at dtucker.net)
GPG key 11EAA6FA / A86E 3E07 5B19 5880 E860 37F4 9357 ECEF 11EA A6FA
Good judgement comes with experience. Unfortunately, the experience
usually comes from bad judgement.