aging threads - performance issue

[Vlada]

Our Solaris 7 production system is having performance issue.
Multithreading process slows down during the time until it almost
seems to be hanging.

The number of "famous" Solaris aging threads is growing during the
time. In the pstack, I can see number of aging threads like one below
being more than 300.

----------------- lwp# 40811 --------------------------------
fd719f34 lwp_cond_wait (fd7f3fe0, fd7f3ff0, f9345cf0) fd711a00
_lwp_cond_timedwait (fd7f3fe0, fd7f3ff0, 0, 40437bab, 0, 0) + 90
fd7ca94c _age (fd7eca6c, fd7ed7ec, 1, 0, 0, fe401000) + 90
fd7ca8b4 _qswtch (0, 0, 0, 0, 0, 0) + 10c
------------------

These threads are supposed to be reused or discard after some time but
it seems Solaris threading library does not work as it is supposed to.

All threads are detached. Some threads are bound-threads
(PTHREAD_SCOPE_SYSTEM) but some are not (PTHREAD_SCOPE_PROCESS). All
other threading parameters are default. There is internal control that
limits number of newly created threads and in general, the number of
threads running in the same time is much below 300.

I see very high context switching (cs = 20000). Complete vmstat is
posted below. Could this context switching be result of lwp_cond_wait?

Vmstat:
procs memory page disk faults
cpu
r b w swap free re mf pi po fr de sr m0 m1 m2 m3 in sy cs
us sy id
19 0 0 1232 1176 0 0 0 0 0 0 0 14 0 1 1 4294967196 0 0
-43 -26 -46
1 0 0 5364656 1401240 0 1713 0 60 48 0 0 8 0 0 0 3166 45093 9413
50 43 7
7 0 0 5363088 1403584 0 996 0 0 0 0 0 3 0 0 0 6947 48796
24532 61 36 2
70 0 0 5362384 1404080 4 1465 0 4 4 0 0 10 0 1 0 6070 48996
18092 57 38 5

This is Solaris 7 with the latest threading library patch.

Based on posts that I found, it seems that having all threads to be
bound-threads may help with this.
Is there any other settings related to thread schedule that may help
and force these aging threads to be discard?

Thanks a lot,
Vlad

[Casper H.S. Dik]

vlad...@yahoo.com (Vlada) writes:

>Based on posts that I found, it seems that having all threads to be
>bound-threads may help with this.
>Is there any other settings related to thread schedule that may help
>and force these aging threads to be discard?

The best thing to do, really, is to move to a Solaris release with
the 1:1 thread library. (Solaris 8 /usr/lib/lwp or Solaris 9 and later)

Casper
--
Expressed in this posting are my opinions. They are in no way related
to opinions held by my employer, Sun Microsystems.
Statements on Sun products included here are not gospel and may
be fiction rather than truth.