Tuning the scheduler? Desktop with a CPU-intensive task becomes rapidly unusable.
jan....@bristol.ac.uk
pretty standard) is providing abysmal interactive performance on an
eight-core machine whenever I try to do anything CPU-intensive (such as
building a port).
Basically, trying to build anything from ports rapidly renders everything
else so "non-interactive" in the eyes of the scheduler that, for instance,
switching between virtual desktops (I have six of them in reasonably
frequent use) takes about a minute of painful waiting on redraws to
complete.
Once I pay attention to any particular window, the scheduler rapidly
(like, in 15 agonising seconds or so) decides that the processes
associated with that particular window are "interactive" and performance
there picks up again. But it only takes 10 seconds (not timed; ballpark
figures) or so of inattention for a window's processes to lapse back into
a low-priority state, with the attendant performance problems.
I don't think my desktop usage is particularly abnormal; I doubt my level
of frustration is, either :-) I think the issue here is that a modern
desktop has quite a lot of associated processes, and you can't keep them
all sufficiently "interactive" in the eyes of sched_ule to ensure they
stay responsive.
Are there particular tunables associated with sched_ule (the manpage says
not) that I might look at to try to address this? Or process flags I can
set on my desktop tasks to keep them nearer the interactive end of the
spectrum?
Claimed is:
o Interactivity heuristics that detect interactive applications
and schedules them preferentially under high load.
but compared to the performance under sched_4bsd, what I'm seeing is an
atrocious user experience.
At the moment I'm fiddling around with cpusets to try to pin my port
builds to a subset of the available processors.
Suggestions are welcome!
Cheers,
jan
PS. I've stuck it out with sched_ule since it's been available, but I
should point out this isn't a sudden change in its behaviour; it's done
this for a while.
--
jan grant, ISYS, University of Bristol. http://www.bris.ac.uk/
Tel +44 (0)117 3317661 http://ioctl.org/jan/
"No generalised law is without exception." A self-demonstrating axiom.
_______________________________________________
freebsd...@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-stable
To unsubscribe, send any mail to "freebsd-stabl...@freebsd.org"
Ivan Voras
> I'm running -STABLE with a kde-derived desktop. This setup (which is
> pretty standard) is providing abysmal interactive performance on an
> eight-core machine whenever I try to do anything CPU-intensive (such as
> building a port).
>
> Basically, trying to build anything from ports rapidly renders everything
> else so "non-interactive" in the eyes of the scheduler that, for instance,
> switching between virtual desktops (I have six of them in reasonably
> frequent use) takes about a minute of painful waiting on redraws to
> complete.
Are you sure this is about the scheduler or maybe bad X11 drivers?
> Once I pay attention to any particular window, the scheduler rapidly
> (like, in 15 agonising seconds or so) decides that the processes
> associated with that particular window are "interactive" and performance
> there picks up again. But it only takes 10 seconds (not timed; ballpark
> figures) or so of inattention for a window's processes to lapse back into
> a low-priority state, with the attendant performance problems.
"windows" in X11 have nothing to do with the scheduler (contrary to MS
Windows where the OS actually "re-nices" processes whose windows have
focus) - here you are just interacting with a process.
> I don't think my desktop usage is particularly abnormal; I doubt my level
> of frustration is, either :-) I think the issue here is that a modern
I'm writing this on a quad-core Core2 machine with 4 GB RAM, amd64 arch,
Radeon 2500 HD, with KDE4 with most of the 3D visual effects turned on.
I have not yet experienced problems like you describe.
On the other hand, I have noticed that a 2xQuad-core machine I have
access too has more X11 interactivity problems than this single
quad-core machine, though again not as serious as yours. I don't know
why this is. From the hardware side it might be the shared FSB or from
the software side it might be the scheduler. If you want to try
something I think it's easier for you to disable one CPU in BIOS or pin
X.org and its descendant processes to CPUs of a single socket than to
diagnose scheduler problems.
> but compared to the performance under sched_4bsd, what I'm seeing is an
> atrocious user experience.
It would be best if you could quantify this in some way. I have no idea how.
jan....@bristol.ac.uk
> On 09/01/10 15:08, jan....@bristol.ac.uk wrote:
> > I'm running -STABLE with a kde-derived desktop. This setup (which is
> > pretty standard) is providing abysmal interactive performance on an
> > eight-core machine whenever I try to do anything CPU-intensive (such as
> > building a port).
> >
> > Basically, trying to build anything from ports rapidly renders everything
> > else so "non-interactive" in the eyes of the scheduler that, for instance,
> > switching between virtual desktops (I have six of them in reasonably
> > frequent use) takes about a minute of painful waiting on redraws to
> > complete.
>
> Are you sure this is about the scheduler or maybe bad X11 drivers?
Not 100%, but mostly convinced; I've just started looking at this. It's my
first stab at what might be going on. X11 performance is usually pretty
snappy. There's no paging pressure at all.
> > Once I pay attention to any particular window, the scheduler rapidly
> > (like, in 15 agonising seconds or so) decides that the processes
> > associated with that particular window are "interactive" and performance
> > there picks up again. But it only takes 10 seconds (not timed; ballpark
> > figures) or so of inattention for a window's processes to lapse back into
> > a low-priority state, with the attendant performance problems.
>
> "windows" in X11 have nothing to do with the scheduler (contrary to MS Windows
> where the OS actually "re-nices" processes whose windows have focus) - here
> you are just interacting with a process.
Yup, and it does seem that by interacting with the process, things'll
start to pick up again - for the processes associated with that window.
> On the other hand, I have noticed that a 2xQuad-core machine I have access too
> has more X11 interactivity problems than this single quad-core machine, though
> again not as serious as yours. I don't know why this is. From the hardware
> side it might be the shared FSB or from the software side it might be the
> scheduler. If you want to try something I think it's easier for you to disable
> one CPU in BIOS or pin X.org and its descendant processes to CPUs of a single
> socket than to diagnose scheduler problems.
>
> > but compared to the performance under sched_4bsd, what I'm seeing is an
> > atrocious user experience.
>
> It would be best if you could quantify this in some way. I have no idea how.
Yeah, I realised that my report was "this doesn't work [very well]!" which
is fairly terrible when it comes to tracking things down; mostly, I was
hoping to prompt none, some or lots of "same here"s to see if the problem
was commonly seen. Also (as you're aware) a desktop environment is a
complex beast, and putting numbers against "look and feel" is tricky -
however in this situation, I can get numbers from a wall-clock, the
behaviour is that pronounced. I'll certainly try getting the whole X tree
onto a single socket, to see if that makes any difference.
I'll certainly have a stab with your suggestion - thanks Ivan.
--
jan grant, ISYS, University of Bristol. http://www.bris.ac.uk/
Tel +44 (0)117 3317661 http://ioctl.org/jan/
Q: What's yellow and equivalent to the axiom of choice? A: Zorn's lemon.
Andriy Gapon
> On Wed, 1 Sep 2010, Ivan Voras wrote:
>
>> On 09/01/10 15:08, jan....@bristol.ac.uk wrote:
>>> I'm running -STABLE with a kde-derived desktop. This setup (which is
>>> pretty standard) is providing abysmal interactive performance on an
>>> eight-core machine whenever I try to do anything CPU-intensive (such as
>>> building a port).
>>>
>>> Basically, trying to build anything from ports rapidly renders everything
>>> else so "non-interactive" in the eyes of the scheduler that, for instance,
>>> switching between virtual desktops (I have six of them in reasonably
>>> frequent use) takes about a minute of painful waiting on redraws to
>>> complete.
>>
>> Are you sure this is about the scheduler or maybe bad X11 drivers?
>
> Not 100%, but mostly convinced; I've just started looking at this. It's my
> first stab at what might be going on. X11 performance is usually pretty
> snappy. There's no paging pressure at all.
>From my experience:
1. system with Athlon II X2 250 CPU and onboard AMD graphics - no issues with
interaction between buildworld and GUI with all KDE4 effects enabled (OpenGL).
2. system with comparable Core2 Duo CPU and onboard Intel graphics (G33) -
enabling OpenGL desktop effects in KDE4 leads to the consequences like what you
describe. With all GUI bells and whistles disabled the system behaves quite
like the AMD system.
--
Andriy Gapon
jan....@bristol.ac.uk
> on 02/09/2010 12:08 jan....@bristol.ac.uk said the following:
> > On Wed, 1 Sep 2010, Ivan Voras wrote:
> >
> >> On 09/01/10 15:08, jan....@bristol.ac.uk wrote:
> >>> I'm running -STABLE with a kde-derived desktop. This setup (which is
> >>> pretty standard) is providing abysmal interactive performance on an
> >>> eight-core machine whenever I try to do anything CPU-intensive (such as
> >>> building a port).
> >>>
> >>> Basically, trying to build anything from ports rapidly renders everything
> >>> else so "non-interactive" in the eyes of the scheduler that, for instance,
> >>> switching between virtual desktops (I have six of them in reasonably
> >>> frequent use) takes about a minute of painful waiting on redraws to
> >>> complete.
> >>
> >> Are you sure this is about the scheduler or maybe bad X11 drivers?
> >
> > Not 100%, but mostly convinced; I've just started looking at this. It's my
> > first stab at what might be going on. X11 performance is usually pretty
> > snappy. There's no paging pressure at all.
>
> From my experience:
> 1. system with Athlon II X2 250 CPU and onboard AMD graphics - no issues with
> interaction between buildworld and GUI with all KDE4 effects enabled (OpenGL).
> 2. system with comparable Core2 Duo CPU and onboard Intel graphics (G33) -
> enabling OpenGL desktop effects in KDE4 leads to the consequences like what you
> describe. With all GUI bells and whistles disabled the system behaves quite
> like the AMD system.
All desktop effects are disabled. The graphics are from an nVidia GeForce
8500 GT (G86) with the X.org driver. (It's not _just_ desktop behaviour
that's affected, though: the box runs a number of small headless
[interactive] server processes which also appear to get rapidly starved of
CPU time.)
The behaviour isn't visible with the 4bsd scheduler; "stuff" generally
remains snappy and responsive.
I'll keep poking around and see if I can get to the bottom of it.
--
jan grant, ISYS, University of Bristol. http://www.bris.ac.uk/
Tel +44 (0)117 3317661 http://ioctl.org/jan/
Rereleasing dolphins into the wild since 1998.
Luca Pizzamiglio
My machine has a similar behavior. For instance, during intensive
workload (portupgrade), everything is quite not-responsive.
I made an alias of portupgrade, nice -n 5 portupgrade, that solves the
problem just in that particular case.
My system is AMD Athlon(tm) 64 Processor 3000+ (1809.28-MHz 686-class
CPU) with openGL effects disabled, KDE as desktop environment.
There is an interesting mib kern.sched.interact. But I don't know the
meaning of it (my value is 30).
Cheers,
Luca
--
Mit herzlichen Grüßen,
Luca Pizzamiglio
Systementwicklung
BALLY WULFF Entertainment GmbH
Maybachufer 48-51
12045 Berlin
Phone: +49(30)62002 149
Jim Bryant
started a /usr/src make clean, and had to abort the qt-creator build to
get the make clean to finish. it was taking forever to even paint the
xterm in the make clean window.
-stable built as of last week, amd64 kernel, core2 duo e8200, 4G ram
(3.9G usable), intel dq45ek motherboard, kde4, compositing turned off.
Michal Varga
> i just noticed this too... had a build going of qt-creator, and then
> started a /usr/src make clean, and had to abort the qt-creator build to
> get the make clean to finish. it was taking forever to even paint the
> xterm in the make clean window.
This has been the state of -stable for at least an year, I have yet to
see a 7-stable machine that doesn't exhibit this behavior. This wasn't
the case with 7 in the very beginning and only started slowly building
up over time, particularly around the time of one specific xorg import -
which one it was? 7.4? I guess. Every bit of performance went down the
drain hole by that time and it's on that same level ever since (it's
rather easy to get used to it while working on a 7-stable desktop, but
would be nice having the old pre-ULE performance levels back, sometime).
On the other hand, at least from some of my observations, the terrible
desktop performance isn't strictly CPU-bound, I/O definitely has some
say in this. You can (you should, mileage may vary) see this by trying
to extract a few-GB archive in the background. While clearly no more
than a single CPU is ever occupied by that process (and there's few
other happily idling), you can spend waiting up to a few minutes just to
get a new application launched (or even just a running one getting
redrawn, in case part of it was swapped out at the moment).
But as I said, for 7-stable, this has been the case for a veeeeeery long
time.
m.
--
Michal Varga,
Stonehenge (Gmail account)
Jeremy Chadwick
> On Fri, 2010-09-03 at 14:03 -0500, Jim Bryant wrote:
> > i just noticed this too... had a build going of qt-creator, and then
> > started a /usr/src make clean, and had to abort the qt-creator build to
> > get the make clean to finish. it was taking forever to even paint the
> > xterm in the make clean window.
>
> On the other hand, at least from some of my observations, the terrible
> desktop performance isn't strictly CPU-bound, I/O definitely has some
> say in this. You can (you should, mileage may vary) see this by trying
> to extract a few-GB archive in the background. While clearly no more
> than a single CPU is ever occupied by that process (and there's few
> other happily idling), you can spend waiting up to a few minutes just to
> get a new application launched (or even just a running one getting
> redrawn, in case part of it was swapped out at the moment).
Could this be caused by lack of disk I/O scheduler on FreeBSD, at least
with regards to launching a new application? Can you try making use of
gsched(8) and see if things improve in this regard?
Just be aware of this problem[1] when using it. (I've been working on a
proper fix -- not a hack -- for the problem for about a week now.
Stress level is very high given the ambiguous nature of many aspects of
GEOM and libgeom lacking in numerous areas. So far I've managed to
figure out how to parse the results from geom_gettree() in attempt to
replace kern.geom.conftxt...)
[1]: http://lists.freebsd.org/pipermail/freebsd-current/2010-April/016883.html
--
| Jeremy Chadwick j...@parodius.com |
| Parodius Networking http://www.parodius.com/ |
| UNIX Systems Administrator Mountain View, CA, USA |
| Making life hard for others since 1977. PGP: 4BD6C0CB |
Bruce Cran
Jeremy Chadwick <fre...@jdc.parodius.com> wrote:
> Just be aware of this problem[1] when using it. (I've been working
> on a proper fix -- not a hack -- for the problem for about a week now.
> Stress level is very high given the ambiguous nature of many aspects
> of GEOM and libgeom lacking in numerous areas. So far I've managed to
> figure out how to parse the results from geom_gettree() in attempt to
> replace kern.geom.conftxt...)
I'm hoping to replace most of the geom code in sysinstall for 9.0 - it
needs to parse the output of geom_gettree, use gpart to create
partitions etc. So far I've got code that can parse the existing
partition layout but not much more. Take a look at
user/ae/usr.sbin/sade in svn to see how to interact with geom - ae@ has
been working on adding support for gpt, zfs etc. to sade.
--
Bruce Cran
David Xu
I always tune
it up to 200 on my desktop machine which is running gnome and other GUI
applications,
for a heavy GUI deskkop, I would tune it up to 224 to get better result.
Regards,
David Xu
jhell
For reference how did you arrive at 224 for a result ?
--
jhell,v
Andriy Gapon
> I think sysctl kern.sched.preempt_thresh is too low, default is only 64. I
> always tune
> it up to 200 on my desktop machine which is running gnome and other GUI
> applications,
> for a heavy GUI deskkop, I would tune it up to 224 to get better result.
And turns out I have this in my sysctl.conf :-)
# should give more responsivness on desktop
# suggested by David Xu <dav...@freebsd.org>
kern.sched.preempt_thresh=220
Apparently I have it for so long in there that I forgot that I do.
Thanks again!
--
Andriy Gapon
Jeremy Chadwick
> > 64. I always tune it up to 200 on my desktop machine which is
> > running gnome and other GUI applications, for a heavy GUI deskkop, I
> > would tune it up to 224 to get better result.
> >
>
Can someone explain exactly what this sysctl does? The description is
only useful if you have familiarity with the scheduler internals:
$ sysctl -d kern.sched.preempt_thresh
kern.sched.preempt_thresh: Min priority for preemption, lower priorities have greater precedence
The source code doesn't really explain it either -- but I will point out
that there's a change in the default value based on an option called
FULL_PREEMPTION:
src/sys/kern/sched_ule.c
192 #ifdef PREEMPTION
193 #ifdef FULL_PREEMPTION
194 static int preempt_thresh = PRI_MAX_IDLE;
195 #else
196 static int preempt_thresh = PRI_MIN_KERN;
197 #endif
198 #else
199 static int preempt_thresh = 0;
200 #endif
src/sys/sys/priority.h
81 #define PRI_MAX (255) /* Lowest priority. */
97 #define PRI_MIN_KERN (64)
...
121 #define PRI_MAX_IDLE (PRI_MAX)
Secondly, this tunable isn't mentioned in any man page, so I'm not sure
how users would even come across it:
$ zgrep -r preempt_thresh /usr/share/man
$
Thirdly, does this sysctl only exist if "options PREEMPTION" is
included in one's kernel config? (I did not dig through kernel source
thoroughly)
--
| Jeremy Chadwick j...@parodius.com |
| Parodius Networking http://www.parodius.com/ |
| UNIX Systems Administrator Mountain View, CA, USA |
| Making life hard for others since 1977. PGP: 4BD6C0CB |
_______________________________________________
Andriy Gapon
> Can someone explain exactly what this sysctl does? The description is
> only useful if you have familiarity with the scheduler internals:
>
> $ sysctl -d kern.sched.preempt_thresh
> kern.sched.preempt_thresh: Min priority for preemption, lower priorities have greater precedence
>
> The source code doesn't really explain it either -- but I will point out
> that there's a change in the default value based on an option called
> FULL_PREEMPTION:
>
> src/sys/kern/sched_ule.c
> 192 #ifdef PREEMPTION
> 193 #ifdef FULL_PREEMPTION
> 194 static int preempt_thresh = PRI_MAX_IDLE;
> 195 #else
> 196 static int preempt_thresh = PRI_MIN_KERN;
> 197 #endif
> 198 #else
> 199 static int preempt_thresh = 0;
> 200 #endif
>
> src/sys/sys/priority.h
> 81 #define PRI_MAX (255) /* Lowest priority. */
> 97 #define PRI_MIN_KERN (64)
> ...
> 121 #define PRI_MAX_IDLE (PRI_MAX)
>
Well, I think you quoted almost all relevant source to get an understanding.
Take a look also at sched_shouldpreempt() in sched_ule.c.
--
Andriy Gapon
Andriy Gapon
> On 09/03/2010 20:14, David Xu wrote:
>> I think sysctl kern.sched.preempt_thresh is too low, default is only
>> 64. I always tune it up to 200 on my desktop machine which is
>> running gnome and other GUI applications, for a heavy GUI deskkop, I
>> would tune it up to 224 to get better result.
>>
>
> For reference how did you arrive at 224 for a result ?
As Jeremy has already discovered, take a look at sys/sys/priority.h, especially
PRI_MIN_IDLE.