Possible Threading problem with -CURRENT / MySQL?

[Lasse Laursen]

Hi,

I posted this on the 2freebsq-questions" list some time ago but didn't get
any response. Maybe someone on this list can help:

We are running MySQL 4.0.20 on FreeBSD 5.2-CURRENT. The machine is a dual
Xeon with 4GByte of memory with HTT enabled.

Some info about the system:

FreeBSD dbnode3 5.2-CURRENT FreeBSD 5.2-CURRENT #0: Tue Jun 8 14:41:02 CEST
2004 laursen@dbnode3:/usr/obj/usr/src/sys/DBNODE3 i386

the /etc/libmap.conf contains:

libc_r.so.5 libpthread.so.1
libc_r.so libpthread.so

/etc/make.conf contains:

# -- use.perl generated deltas -- #
# Created: Wed Jun 9 10:18:52 2004
# Setting to use base perl from ports:
PERL_VER=5.8.4
PERL_VERSION=5.8.4
PERL_ARCH=mach
NOPERL=yo
NO_PERL=yo
NO_PERL_WRAPPER=yo

MySQL 4.0.20 was compiled with --without-libwrap (libwrapper seems to be
broken) and WITH_PROC_SCOPE_PTH set to yes.

WE use the SCHED_ULE, and:

# Memory options
options MAXDSIZ="(1536*1024*1024)"
options MAXSSIZ="(1024*1024*1024)"
options DFLDSIZ="(1536*1024*1024)"

The server runs fine until a single thread/query suddenly locks up the
entire MySQL daemon. After that all queries are just queued and a restart of
the daemon is needed to unlock the system. The system itself is stable
enough as far as I can see. 'top' reports a lot of locks (*Giant) so I
assume that it's some weird problem with the threading? We used to use
FreeBSD on non-SMP machines without any problems.

I have tried with linux threads as well but the same problem occurs.

Have anyone experienced any similar problems and found a solution to this
rather weird problem?

Feel free to request any further information about the system/setup and I
will do my best to provide it to the list.

Regards
--
Lasse Laursen · VP, Hosting Technology · NetGroup Processing Aps
St. Kongensgade 40H · DK-1264 Copenhagen K, Denmark
Phone: +45 3370 1526 · Fax: +45 3313 0066

- Don't be fooled by cheap finnish imitations - BSD is the One True Code

[Julian Elischer]

On Mon, 14 Jun 2004, Lasse Laursen wrote:

>
> MySQL 4.0.20 was compiled with --without-libwrap (libwrapper seems to be
> broken) and WITH_PROC_SCOPE_PTH set to yes.
>
> WE use the SCHED_ULE, and:
>
> # Memory options
> options MAXDSIZ="(1536*1024*1024)"
> options MAXSSIZ="(1024*1024*1024)"
> options DFLDSIZ="(1536*1024*1024)"
>
> The server runs fine until a single thread/query suddenly locks up the
> entire MySQL daemon. After that all queries are just queued and a restart of
> the daemon is needed to unlock the system. The system itself is stable
> enough as far as I can see. 'top' reports a lot of locks (*Giant) so I
> assume that it's some weird problem with the threading? We used to use
> FreeBSD on non-SMP machines without any problems.
>
> I have tried with linux threads as well but the same problem occurs.

This does suggest a bug in MySQL, but let's investigate anyhow..

>
> Have anyone experienced any similar problems and found a solution to this
> rather weird problem?
>
> Feel free to request any further information about the system/setup and I
> will do my best to provide it to the list.

the output of "ps -alxH might be useful.

also:
do you have teh kernel debugger compiled into the kernel?
(option DDB)

if so..
note the PID of the process
sysctl debug.enter_debugger=ddb
[you are now in the debugger]
ps
[note the Thread address of the threads that seem to be blocking things]
(I may ask for specific threads after seeing the 'ps' output)
do

show thread {address}

to get a stack backtrace of those threads.

if possible use a serial console to do all this.. That way you can log
it all.

to make a serial console, connect 2 machines com1 by a null-modem cable.

on the other machine do:
script
tip com1

on the SQL machine add
console="comconsole"
into /boot/loader.conf and reboot

all console out put from teh boot-loader-on will now go to teh serial
port and appear in teh 'tip' output. (which is being logged by the
'script' command).

julian

[Lasse Laursen]

Hi,

Please find answers below:

Regards
--
Lasse Laursen · VP, Hosting Technology · NetGroup Processing Aps
St. Kongensgade 40H · DK-1264 Copenhagen K, Denmark
Phone: +45 3370 1526 · Fax: +45 3313 0066

- Don't be fooled by cheap finnish imitations - BSD is the One True Code

----- Original Message -----
From: "Julian Elischer" <jul...@elischer.org>
To: "Lasse Laursen" <lau...@netgroup.dk>
Cc: <freebsd...@freebsd.org>; <freebsd...@freebsd.org>
Sent: Monday, June 14, 2004 8:15 PM
Subject: Re: Possible Threading problem with -CURRENT / MySQL?

> On Mon, 14 Jun 2004, Lasse Laursen wrote:
>
> >
> > MySQL 4.0.20 was compiled with --without-libwrap (libwrapper seems to be
> > broken) and WITH_PROC_SCOPE_PTH set to yes.
> >
> > WE use the SCHED_ULE, and:
> >
> > # Memory options
> > options MAXDSIZ="(1536*1024*1024)"
> > options MAXSSIZ="(1024*1024*1024)"
> > options DFLDSIZ="(1536*1024*1024)"
> >
> > The server runs fine until a single thread/query suddenly locks up the
> > entire MySQL daemon. After that all queries are just queued and a
restart of
> > the daemon is needed to unlock the system. The system itself is stable
> > enough as far as I can see. 'top' reports a lot of locks (*Giant) so I
> > assume that it's some weird problem with the threading? We used to use
> > FreeBSD on non-SMP machines without any problems.
> >
> > I have tried with linux threads as well but the same problem occurs.
>
> This does suggest a bug in MySQL, but let's investigate anyhow..

We have used FreeBSD on non-smp machines for a long time (even the 5.x
branch) without any problems. However on our new servers that are SMP the
problems occur. It's the exact same version of MySQL that runs on the
servers.

> the output of "ps -alxH might be useful.

I will post that the next time the machine locks up. :)

I will look into that as well - thanks for the promte reply.

[Daniel Eischen]

On Mon, 14 Jun 2004, Lasse Laursen wrote:

Try using SCHED_4BSD, then try removing all memory options.

--
Dan Eischen

[Lasse Laursen]

Hi,

Please find answers below:

Regards
--
Lasse Laursen · VP, Hosting Technology · NetGroup Processing Aps
St. Kongensgade 40H · DK-1264 Copenhagen K, Denmark
Phone: +45 3370 1526 · Fax: +45 3313 0066

- Don't be fooled by cheap finnish imitations - BSD is the One True Code
----- Original Message -----
From: "Daniel Eischen" <eis...@vigrid.com>
To: "Lasse Laursen" <lau...@netgroup.dk>
Cc: <freebsd...@freebsd.org>; <freebsd...@freebsd.org>
Sent: Monday, June 14, 2004 8:58 PM
Subject: Re: Possible Threading problem with -CURRENT / MySQL?

> > WE use the SCHED_ULE, and:
> >
> > # Memory options
> > options MAXDSIZ="(1536*1024*1024)"
> > options MAXSSIZ="(1024*1024*1024)"
> > options DFLDSIZ="(1536*1024*1024)"
>
> Try using SCHED_4BSD, then try removing all memory options.

Wouldn't this limit us to 512 MB of memory per process running on the
server? we tried with each scheduler with same results. I will try to
recompile without the memory options and use the old scheduler. Any hints
why this might work?

[Daniel Eischen]

On Mon, 14 Jun 2004, Lasse Laursen wrote:

> > > WE use the SCHED_ULE, and:
> > >
> > > # Memory options
> > > options MAXDSIZ="(1536*1024*1024)"
> > > options MAXSSIZ="(1024*1024*1024)"
> > > options DFLDSIZ="(1536*1024*1024)"
> >
> > Try using SCHED_4BSD, then try removing all memory options.
>
> Wouldn't this limit us to 512 MB of memory per process running on the
> server? we tried with each scheduler with same results. I will try to
> recompile without the memory options and use the old scheduler. Any hints
> why this might work?

No, just trying to see what happens with a more stock
kernel. Also, try removing /etc/my.conf to use default
MySQL settings.

--
Dan

[Lasse Laursen]

Hi,

I will try that.

Would it be better to use 4.10 and MySQL w. LinuxThreads instead? It's
production databases that we run on the machine so stability is a must (yes
I know - 5.2 isn't stable yet...)

The reason why we decided to use 5.2-CURRENT was that the improved SMP
support - which seems to still have some bugs?

Regards
--
Lasse Laursen · VP, Hosting Technology · NetGroup Processing Aps
St. Kongensgade 40H · DK-1264 Copenhagen K, Denmark
Phone: +45 3370 1526 · Fax: +45 3313 0066

- Don't be fooled by cheap finnish imitations - BSD is the One True Code
----- Original Message -----
From: "Daniel Eischen" <eis...@vigrid.com>
To: "Lasse Laursen" <lau...@netgroup.dk>
Cc: <freebsd...@freebsd.org>; <freebsd...@freebsd.org>
Sent: Monday, June 14, 2004 9:10 PM
Subject: Re: Possible Threading problem with -CURRENT / MySQL?

[Daniel Eischen]

On Mon, 14 Jun 2004, mike wrote:

> welcome to our hell. we've been experiencing mysql problems on freebsd 5.x
> as well. it sounds like scheduler/threading is to blame but we were not
> able to give sufficient or proper motivation to the folks who could
> examine this deeper - we even offered $500 cash to whomever stepped up to
> help resolve this.
>
> linux runs almost 2x as fast on the same hardware with no configuring -
> and we get nearly the same results running in single CPU mode vs. dual CPU
> mode on fbsd... something is definately fubar with the mysql+fbsd5.x
> combination.

The problem is not the same. Lasse is getting lockups even
with Linuxthreads. You were able to run with all threading
libraries, but native linux was faster.

--
DE

[Lasse Laursen]

Hi,

Also on a SMP machine? I think that the problem is somehow related to SMP
machines since the problems started when we moved to a SMP box :(

Regards
--
Lasse Laursen · VP, Hosting Technology · NetGroup Processing Aps
St. Kongensgade 40H · DK-1264 Copenhagen K, Denmark
Phone: +45 3370 1526 · Fax: +45 3313 0066

- Don't be fooled by cheap finnish imitations - BSD is the One True Code

----- Original Message -----
From: "Petri Helenius" <pe...@he.iki.fi>
To: "mike" <mi...@mike2k.com>
Cc: "Lasse Laursen" <lau...@netgroup.dk>; <freebsd...@freebsd.org>;
<freebsd...@freebsd.org>
Sent: Monday, June 14, 2004 10:37 PM
Subject: Re: Possible Threading problem with -CURRENT / MySQL?

> mike wrote:
>
> >linux runs almost 2x as fast on the same hardware with no configuring -
> >and we get nearly the same results running in single CPU mode vs. dual
CPU
> >mode on fbsd... something is definately fubar with the mysql+fbsd5.x
> >combination.
> >
> >
> >

> mysql runs fine on freebsd 5.x both with linuxthreads and pthreads (on
> -CURRENT). 5.2.1 release was broken though.
> This is on i386.
>
> Pete
>
>

[Lasse Laursen]

Hi,

No. PAE isn't enabled. I have copied the config file for the kernel to a
webserver:

http://solidcore.dk/DBNODE3

Feel free to comment the settings. The machine is an IBM-x-something, dual
Xeon with HTT enabled. 4 GByte of memory and a ServeRAID controller w. raid
10.

Regards
--
Lasse Laursen · VP, Hosting Technology · NetGroup Processing Aps
St. Kongensgade 40H · DK-1264 Copenhagen K, Denmark
Phone: +45 3370 1526 · Fax: +45 3313 0066

- Don't be fooled by cheap finnish imitations - BSD is the One True Code
----- Original Message -----
From: "Petri Helenius" <pe...@he.iki.fi>
To: "Lasse Laursen" <lau...@netgroup.dk>
Cc: "mike" <mi...@mike2k.com>; <freebsd...@freebsd.org>;
<freebsd...@freebsd.org>
Sent: Monday, June 14, 2004 10:46 PM
Subject: Re: Possible Threading problem with -CURRENT / MySQL?

> >Also on a SMP machine? I think that the problem is somehow related to SMP
> >machines since the problems started when we moved to a SMP box :(
> >
> >
> >

> With both "classic" dual CPU, dual Xeons with HT enabled or disabled.
>
> But I´ve seen motherboards where SMP is not an option although they have
> two CPU sockets :-)
>
> BTW, do you have PAE enabled? We don´t.

[Lasse Laursen]

Hi,

I tried with both schedulers with the same result :( I think that I have
tried every possible combination in order to track down the problem ...

Regards
--
Lasse Laursen · VP, Hosting Technology · NetGroup Processing Aps
St. Kongensgade 40H · DK-1264 Copenhagen K, Denmark
Phone: +45 3370 1526 · Fax: +45 3313 0066

- Don't be fooled by cheap finnish imitations - BSD is the One True Code
----- Original Message -----
From: "Petri Helenius" <pe...@he.iki.fi>
To: "Lasse Laursen" <lau...@netgroup.dk>
Cc: "mike" <mi...@mike2k.com>; <freebsd...@freebsd.org>;
<freebsd...@freebsd.org>
Sent: Monday, June 14, 2004 11:04 PM
Subject: Re: Possible Threading problem with -CURRENT / MySQL?

> Lasse Laursen wrote:
>
> >Hi,
> >
> >No. PAE isn't enabled. I have copied the config file for the kernel to a
> >webserver:
> >
> >http://solidcore.dk/DBNODE3
> >
> >Feel free to comment the settings. The machine is an IBM-x-something,
dual
> >Xeon with HTT enabled. 4 GByte of memory and a ServeRAID controller w.
raid
> >10.
> >
> >
> >

> My suspect (without proof) would be SCHED_ULE.
>
> Not sure also how 4G works out. We have 1-3G on the boxes.
>
> Pete
>
>

[Lasse Laursen]

I find it hard to believe that it's the RAM amount that can cause it - also
the current RAM setup is some exotic IBM configuration with some nice
features (according to their product page) - also the disk controller is too
expensive to replace :(

The sig: We had some really nasty problems with Linux in the early days
(2.0.14) ... :(

Regards
--
Lasse Laursen · VP, Hosting Technology · NetGroup Processing Aps
St. Kongensgade 40H · DK-1264 Copenhagen K, Denmark
Phone: +45 3370 1526 · Fax: +45 3313 0066

- Don't be fooled by cheap finnish imitations - BSD is the One True Code
----- Original Message -----
From: "Petri Helenius" <pe...@he.iki.fi>
To: "Lasse Laursen" <lau...@netgroup.dk>
Cc: "mike" <mi...@mike2k.com>; <freebsd...@freebsd.org>;
<freebsd...@freebsd.org>
Sent: Monday, June 14, 2004 11:11 PM
Subject: Re: Possible Threading problem with -CURRENT / MySQL?

> Lasse Laursen wrote:
>
> >Hi,
> >

> >I tried with both schedulers with the same result :( I think that I have
> >tried every possible combination in order to track down the problem ...
> >
> >
> >

> I would take out one DIMM to reduce the memory and if that would not
> help, swap the hardware.
>
> Obviously the earlier questions about where it does hang are valid.
>
> Pete

>
> >- Don't be fooled by cheap finnish imitations - BSD is the One True Code
> >
> >

> P.S. funny sig :)
>
>

[Robert Watson]

On Mon, 14 Jun 2004, Lasse Laursen wrote:

> Also on a SMP machine? I think that the problem is somehow related to
> SMP machines since the problems started when we moved to a SMP box :(

If you haven't already, it would certainly be worth removing SMP from the
kernel on that box and just running a UP kernel to confirm that it's a
problem using SMP. I don't doubt it's the case, but it's worth checking
anyway to be sure.

Robert N M Watson FreeBSD Core Team, TrustedBSD Projects
rob...@fledge.watson.org Senior Research Scientist, McAfee Research

>
> Regards
> --
> Lasse Laursen · VP, Hosting Technology · NetGroup Processing Aps
> St. Kongensgade 40H · DK-1264 Copenhagen K, Denmark
> Phone: +45 3370 1526 · Fax: +45 3313 0066
>
> - Don't be fooled by cheap finnish imitations - BSD is the One True Code
> ----- Original Message -----
> From: "Petri Helenius" <pe...@he.iki.fi>
> To: "mike" <mi...@mike2k.com>
> Cc: "Lasse Laursen" <lau...@netgroup.dk>; <freebsd...@freebsd.org>;
> <freebsd...@freebsd.org>
> Sent: Monday, June 14, 2004 10:37 PM
> Subject: Re: Possible Threading problem with -CURRENT / MySQL?
>
>
> > mike wrote:
> >
> > >linux runs almost 2x as fast on the same hardware with no configuring -
> > >and we get nearly the same results running in single CPU mode vs. dual
> CPU
> > >mode on fbsd... something is definately fubar with the mysql+fbsd5.x
> > >combination.
> > >
> > >
> > >
> > mysql runs fine on freebsd 5.x both with linuxthreads and pthreads (on
> > -CURRENT). 5.2.1 release was broken though.
> > This is on i386.
> >
> > Pete
> >
> >
>
>

> _______________________________________________
> freebsd...@freebsd.org mailing list
> http://lists.freebsd.org/mailman/listinfo/freebsd-current
> To unsubscribe, send any mail to "freebsd-curre...@freebsd.org"
>

[Martin Blapp]

Hi,

> Feel free to comment the settings. The machine is an IBM-x-something, dual
> Xeon with HTT enabled. 4 GByte of memory and a ServeRAID controller w. raid
> 10.

We have the same config as you have. IBM X-series 345, FreeBSD Current and
have no locking problems with mysql on a very busy SMP server. With HTT
enabled, we encountered the same problems you had, and some severe crashes
additionally from time to time.

Please turn off HTT and try again.. It has been told to us that the chipset
(serverworks) has buggy HTT support.

Martin

[Lasse Laursen]

Hi,

Have you had any lockups in MySQL with your setup? We have tried to disable
HTT and hopefully that will solve the problem.

Regards
--
Lasse Laursen · VP, Hosting Technology · NetGroup Processing Aps
St. Kongensgade 40H · DK-1264 Copenhagen K, Denmark
Phone: +45 3370 1526 · Fax: +45 3313 0066

- Don't be fooled by cheap finnish imitations - BSD is the One True Code
----- Original Message -----
From: "Cedric Tabary" <ced...@carpediem.fr>
To: <freebsd...@freebsd.org>
Cc: "Lasse Laursen" <lau...@netgroup.dk>; <freebsd...@freebsd.org>;
"mike" <mi...@mike2k.com>; "Petri Helenius" <pe...@he.iki.fi>
Sent: Tuesday, June 15, 2004 1:06 PM
Subject: Re: Possible Threading problem with -CURRENT / MySQL?

> On 14/06/2004 17:43, Robert Watson wrote:
> > On Mon, 14 Jun 2004, Lasse Laursen wrote:
> >
> > > Also on a SMP machine? I think that the problem is somehow related to
> > > SMP machines since the problems started when we moved to a SMP box :(
> >
> > If you haven't already, it would certainly be worth removing SMP from
the
> > kernel on that box and just running a UP kernel to confirm that it's a
> > problem using SMP. I don't doubt it's the case, but it's worth checking
> > anyway to be sure.
>

> I have a similar smp config (dell poweredge 2650, dual Xeon 2.8, RAID5)
>
> I have best performance with linux 2.4.21-smp (not tried 2.6),
> but very poor (5x less) with mysqld using libpthreads on CURRENT.
> mysqld process is always in state 'kserel' or sometimes '*Giant'
>
> http://grumly.eu.org/~ced/dmesg.txt
> http://grumly.eu.org/~ced/CED-SMP.txt (no invariants, no witness)
>
> mysqld with linuxthreads seems to work better ...
> and even better with libc_r (but using only 1 cpu)
>
> Everything with HTT disabled in bios. Enabling HTT gives even worst
> performance :/

[Petri Helenius]

[Petri Helenius]

Lasse Laursen wrote:

>Hi,
>
>Also on a SMP machine? I think that the problem is somehow related to SMP
>machines since the problems started when we moved to a SMP box :(
>
>
>

With both "classic" dual CPU, dual Xeons with HT enabled or disabled.

But I“ve seen motherboards where SMP is not an option although they have
two CPU sockets :-)

BTW, do you have PAE enabled? We don“t.

Pete

[Petri Helenius]

Lasse Laursen wrote:

>Hi,
>
>I tried with both schedulers with the same result :( I think that I have
>tried every possible combination in order to track down the problem ...
>
>
>

I would take out one DIMM to reduce the memory and if that would not
help, swap the hardware.

Obviously the earlier questions about where it does hang are valid.

Pete

>- Don't be fooled by cheap finnish imitations - BSD is the One True Code
>
>
P.S. funny sig :)

[Cedric Tabary]

On 14/06/2004 17:43, Robert Watson wrote:

> On Mon, 14 Jun 2004, Lasse Laursen wrote:
>
> > Also on a SMP machine? I think that the problem is somehow related to
> > SMP machines since the problems started when we moved to a SMP box :(
>
> If you haven't already, it would certainly be worth removing SMP from the
> kernel on that box and just running a UP kernel to confirm that it's a
> problem using SMP. I don't doubt it's the case, but it's worth checking
> anyway to be sure.

I have a similar smp config (dell poweredge 2650, dual Xeon 2.8, RAID5)

I have best performance with linux 2.4.21-smp (not tried 2.6),
but very poor (5x less) with mysqld using libpthreads on CURRENT.
mysqld process is always in state 'kserel' or sometimes '*Giant'

http://grumly.eu.org/~ced/dmesg.txt
http://grumly.eu.org/~ced/CED-SMP.txt (no invariants, no witness)

mysqld with linuxthreads seems to work better ...
and even better with libc_r (but using only 1 cpu)

Everything with HTT disabled in bios. Enabling HTT gives even worst
performance :/

--
Cédric Tabary

[mike]

welcome to our hell. we've been experiencing mysql problems on freebsd 5.x
as well. it sounds like scheduler/threading is to blame but we were not
able to give sufficient or proper motivation to the folks who could
examine this deeper - we even offered $500 cash to whomever stepped up to
help resolve this.

linux runs almost 2x as fast on the same hardware with no configuring -

and we get nearly the same results running in single CPU mode vs. dual CPU
mode on fbsd... something is definately fubar with the mysql+fbsd5.x
combination.

> _______________________________________________
> freebsd...@freebsd.org mailing list
> http://lists.freebsd.org/mailman/listinfo/freebsd-threads
> To unsubscribe, send any mail to "freebsd-threa...@freebsd.org"
>

[Petri Helenius]

Lasse Laursen wrote:

>Hi,
>
>No. PAE isn't enabled. I have copied the config file for the kernel to a
>webserver:
>
>http://solidcore.dk/DBNODE3
>
>Feel free to comment the settings. The machine is an IBM-x-something, dual
>Xeon with HTT enabled. 4 GByte of memory and a ServeRAID controller w. raid
>10.
>
>
>

[mike]

there's a difference between running "fine" and actually performing to the
hardware's specifications.

if you tell me 5.2.1 was broken, fine, but i believe JG tried some other
versions and still came out with the same results.

this was on AMD64 but it sounds like all platforms are having the problems
- it's just that there is no linuxthreads port for AMD64 so we could not
test with that (however he did install i386 freebsd and tried linuxthreads
and there were no significant performance gains)

[Lasse Laursen]

Hi,

We just had our first lockup for today - and the last one. Next step is to
downgrade to Linux :( *crud*

Regards
--
Lasse Laursen · VP, Hosting Technology · NetGroup Processing Aps
St. Kongensgade 40H · DK-1264 Copenhagen K, Denmark
Phone: +45 3370 1526 · Fax: +45 3313 0066

- Don't be fooled by cheap finnish imitations - BSD is the One True Code
----- Original Message -----
From: "Cedric Tabary" <ced...@carpediem.fr>
To: "Lasse Laursen" <lau...@netgroup.dk>
Cc: <freebsd...@freebsd.org>; "mike" <mi...@mike2k.com>; "Petri
Helenius" <pe...@he.iki.fi>; <freebsd...@freebsd.org>
Sent: Tuesday, June 15, 2004 3:07 PM
Subject: Re: Possible Threading problem with -CURRENT / MySQL?

> On 15/06/2004 13:12, Lasse Laursen wrote:
> > Hi,
> >
> > Have you had any lockups in MySQL with your setup? We have tried to
disable
> > HTT and hopefully that will solve the problem.
>

> No lockups for now, but it is not heavily loaded.
>
> --
> Cédric Tabary
>

[Alexander Leidinger]

On Tue, 15 Jun 2004 13:12:18 +0200
"Lasse Laursen" <lau...@netgroup.dk> wrote:

> Hi,
>
> Have you had any lockups in MySQL with your setup? We have tried to disable
> HTT and hopefully that will solve the problem.

I'm not sure I've seen all replies, but I think I haven't seen a mention
of the malloc flags so far.

If you haven't changed the default malloc flags please do so and try
again:
---snip---
% ll /etc/malloc.conf
lrwx------ 1 root wheel 2B 18 Aug 2001 /etc/malloc.conf@ -> aj
---snip---

The 'a' isn't important here, but the 'j' affects the performance.

Bye,
Alexander.

--
I'm available to get hired (preferred in .lu).

http://www.Leidinger.net Alexander @ Leidinger.net
GPG fingerprint = C518 BC70 E67F 143F BE91 3365 79E2 9C60 B006 3FE7

[Daniel Eischen]

On Tue, 15 Jun 2004, Lasse Laursen wrote:

> Hi,
>
> We just had our first lockup for today - and the last one. Next step is to
> downgrade to Linux :( *crud*

We still haven't heard whether removing memory options from
the kernel, disabling HTT, and removing /etc/my.conf have
had any effect.

--
DE

[Martin Blapp]

Hi all,

Just to list our working setup:

(I don't know if mysql is very fast, but since I rewrote all mysql queries to
use LEFT/RIGHT JOIN and UNION and redesigned the databases, the mysql load is
not significant anymore)

Has someone a test database and e benchmark tool ?

- We use SCHED_4BSD.
- HTT is disabled in the BIOS
- WITNESS and INVARIANTS are turned off (important) !

uptime
5:07PM up 35 days, 3:52, 9 users, load averages: 2.27, 1.70, 1.54

No hangs and crashes or lockups.

38073 idms 20 -15 55160K 17200K kserel 0 26:45 0.00% 0.00% mysqld

my.cnf config:

[mysqld]
skip-innodb
set-variable = max_connections=1000
set-variable = key_buffer_size=16M
set-variable = table_cache=256
set-variable = record_buffer=1M
set-variable = sort_buffer=16M

CPU: Intel(R) Xeon(TM) CPU 3.06GHz (3059.98-MHz 686-class CPU)
Origin = "GenuineIntel" Id = 0xf27 Stepping = 7

Features=0xbfebfbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CLFLUSH,DTS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE>
real memory = 2147397632 (2047 MB)
avail memory = 2095964160 (1998 MB)
ACPI APIC Table: <IBM SERONYXP>
FreeBSD/SMP: Multiprocessor System Detected: 2 CPUs
cpu0 (BSP): APIC ID: 0
cpu1 (AP): APIC ID: 6

Hope this helps anybody:
Martin

[Claus Guttesen]

> > We just had our first lockup for today - and the
> last one. Next step is to
> > downgrade to Linux :( *crud*
>

Maby not the greatest comfort, but I've had a Dell
2650 with the Perc3-controller working for weeks. I
installed rt3 (troubleticket-system) which needed
Mysql (or postgresql). I installed Mysql 4.0 and
haven't had problems with it. I increased the amount
of RAM than Mysql can utilize, but that's about it.

All installed from /usr/ports using portinstall.

FreeBSD 5.2.1, 2 GB RAM, dual Xeon @ 2.4 Ghz. Other
than enabling -O2 and CPUTYPE=i686 in /etc/make.conf
and make a customized kernel with amr in my kernel,
everything else is pretty much default.

I never touched HTT in BIOS nor using sysctl, that's
working out of the box.

/etc/make.conf:
CPUTYPE=i686
CFLAGS= -O2 -pipe -funroll-loops -fno-strict-aliasing
COPTFLAGS= -O2 -pipe -funroll-loops
...
etc. etc. etc.

/usr/src/sys/i386/conf/DELL2650:
machine i386
cpu I686_CPU
ident DELL
options SCHED_4BSD #4BSD scheduler
options SMP # Symmetric
MultiProcessor Kernel
device apic
device amr # AMI MegaRAID
device bge # Broadcom BCM570xx
Gigabit Ethernet

regards
Claus

Yahoo! Mail (http://dk.mail.yahoo.com) - Gratis: 6 MB lagerplads, spamfilter og virusscan

[Lasse Laursen]

Hi Claus,

How many queries per second does the machine handle? We are at approx 100
queries per second in prime time. The load on the server is low and the
performance is OK (haven't tried anything else but FreeBSD so I can't
compare with Linux). Everything is working fine until a single thread
suddenly locks up the entire system. :(

There are no special compile options for neither FreeBSD or for MySQL (from
ports).

Regards
--
Lasse Laursen · VP, Hosting Technology · NetGroup Processing Aps
St. Kongensgade 40H · DK-1264 Copenhagen K, Denmark
Phone: +45 3370 1526 · Fax: +45 3313 0066

- Don't be fooled by cheap finnish imitations - BSD is the One True Code
----- Original Message -----
From: "Claus Guttesen" <cgut...@yahoo.dk>
To: "Daniel Eischen" <eis...@vigrid.com>; "Lasse Laursen"
<lau...@netgroup.dk>
Cc: <freebsd...@freebsd.org>; "Cedric Tabary" <ced...@carpediem.fr>;
"mike" <mi...@mike2k.com>; <freebsd...@freebsd.org>
Sent: Tuesday, June 15, 2004 7:12 PM
Subject: Re: Possible Threading problem with -CURRENT / MySQL?

> _______________________________________________
> freebsd...@freebsd.org mailing list
> http://lists.freebsd.org/mailman/listinfo/freebsd-current
> To unsubscribe, send any mail to "freebsd-curre...@freebsd.org"
>

[Steve Kargl]

On Tue, Jun 15, 2004 at 07:12:25PM +0200, Claus Guttesen wrote:
>>> We just had our first lockup for today - and the
>>> last one. Next step is to
>>> downgrade to Linux :( *crud*
>>
>

> /etc/make.conf:
> CPUTYPE=i686
> CFLAGS= -O2 -pipe -funroll-loops -fno-strict-aliasing
> COPTFLAGS= -O2 -pipe -funroll-loops

man make.conf

Now read the descriptions for CFLAGS and COPTFLAGS.

--
Steve

[Julian Elischer]

On Mon, 14 Jun 2004, Lasse Laursen wrote:

> The server runs fine until a single thread/query suddenly locks up the
> entire MySQL daemon. After that all queries are just queued and a restart of
> the daemon is needed to unlock the system. The system itself is stable
> enough as far as I can see. 'top' reports a lot of locks (*Giant) so I
> assume that it's some weird problem with the threading? We used to use
> FreeBSD on non-SMP machines without any problems.
>
> I have tried with linux threads as well but the same problem occurs.
>
> Have anyone experienced any similar problems and found a solution to this
> rather weird problem?

When this happens you need to examine the state of each thread.
I have not seen teh result of a 'ps alxH' when this happens..

Also you should try 'ktrace' the process for a while when stopped as it
might be retrying something , which may show up.

julian

[Julian Elischer]

On Mon, 14 Jun 2004, mike wrote:

> welcome to our hell. we've been experiencing mysql problems on freebsd 5.x
> as well. it sounds like scheduler/threading is to blame but we were not
> able to give sufficient or proper motivation to the folks who could
> examine this deeper - we even offered $500 cash to whomever stepped up to
> help resolve this.
>
> linux runs almost 2x as fast on the same hardware with no configuring -
> and we get nearly the same results running in single CPU mode vs. dual CPU
> mode on fbsd... something is definately fubar with the mysql+fbsd5.x
> combination.
>
>

You complained about this some time ago and you have still not responded
with the information I suggest..

---- quote from my previous mail---------------
I'm not saying I can help particularly
but the problem is that the people who can help are generally not
interested in databases.. what needs to happen is for
some analysis to be made as to where the time is being spent..
I vaguely remember you saying htat teh cpu was not piegged when you had
this problem... You didn't indicate whether the disks were at 100%.

to check this you should try running systat -vmstat
while noticing the slowdown and try see what is at 100%.
if it turns out that you are NOT seeing disk or CPU limitted
then the next step would be to take a snapshot of operations using
ktrace -d -p (pid) for 20 second followe by krtace -C to turn it off
again.

then looking at the output may give an idea of what operations are
taking too long, and what is waiting on what..

----------------------------------------------

Julian

[Daryl Chance]

Sorry for the top post.

We've had this same problem (with mysqld locking up).
I was able to get a trace, but since it's a live sql
box, I didn't have time to find your email (If I knew
I had kept a copy in my inbox, I would have done this
correctly, instead I thought I had to look up the
email in the archives) but I was able to get a ktrace
-p pid output of the process. I let it run about 10
seconds and then killall -11 mysqld'd the process (it
allows mysqld to shutdown gracefully, thus not
corrupting our tables).

Here is the link to the .out file
http://sql.tribalwar.com/ktrace.out

here is the link to the .txt file of the kdump output
http://sql.tribalwar.com/kdump.txt

That was all I could remember to do in the short time
I had to get it back up and running :).

It's a stock kernel (w/ some minor things commented
out). It's FBSD 5.2.1 p6
http://sql.tribalwar.com/SQL

Hopefully this can help out a little.

Daryl

-----Original Message-----
From: owner-free...@freebsd.org
[mailto:owner-free...@freebsd.org] On Behalf
Of Julian Elischer
Sent: Tuesday, June 15, 2004 7:21 PM
To: mike
Cc: Lasse Laursen; freebsd...@freebsd.org;
freebsd...@freebsd.org
Subject: Re: Possible Threading problem with -CURRENT
/ MySQL?

----------------------------------------------

Julian

_______________________________________________

freebsd...@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to
"freebsd-curre...@freebsd.org"

__________________________________________________
Do You Yahoo!?
Tired of spam? Yahoo! Mail has the best spam protection around
http://mail.yahoo.com

[Robert Watson]

On Tue, 15 Jun 2004, Julian Elischer wrote:

> On Mon, 14 Jun 2004, mike wrote:
>
> > welcome to our hell. we've been experiencing mysql problems on freebsd 5.x
> > as well. it sounds like scheduler/threading is to blame but we were not
> > able to give sufficient or proper motivation to the folks who could
> > examine this deeper - we even offered $500 cash to whomever stepped up to
> > help resolve this.
> >
> > linux runs almost 2x as fast on the same hardware with no configuring -
> > and we get nearly the same results running in single CPU mode vs. dual CPU
> > mode on fbsd... something is definately fubar with the mysql+fbsd5.x
> > combination.
>
> You complained about this some time ago and you have still not responded
> with the information I suggest..

I sent this to Jeremy privately, since it was just some preliminary
measurements, but figured I'd send it publically since the results were
interesting (if tentative, I need to do a lot more work to make them
useful. There are a number of variables I need to look at including:

- Disabling HTT. A chat with Scott Long this evening suggests that HTT
may be substantially hurting the test cases given increased IPIs, etc.
Unfortunately, it looks like I can't easily twiddle HTT without being
local to the machine, and I'm at home right now (it being 1:30am and
all). Removing HTT may help substantially with the dip in performance
in the SMP configuration.

- I'd like to compare against RELENG_4 and a recent Linux kernel.
Unfortunately, the box is configured for neither right now.

- I need to try twiddling schedulers -- this was with SCHED_ULE, and I'd
like to try SCHED_4BSD.

- This was without adaptive mutexes, which seem to be helpful for others,
so I should give them a try.

I don't have any amd64 hardware, so I don't know what if any role it will
play in the results. The performance drop observed in the report appears
to be on amd64 (I may have misread).

Robert N M Watson FreeBSD Core Team, TrustedBSD Projects
rob...@fledge.watson.org Senior Research Scientist, McAfee Research

----
Date: Wed, 16 Jun 2004 01:15:39 -0400 (EDT)
From: Robert Watson <rwa...@FreeBSD.org>
To: JG <amd6...@jpgsworld.com>

Subject: Re: Possible Threading problem with -CURRENT / MySQL?

On Tue, 15 Jun 2004, JG wrote:

> Fwiw, it has to be something that was committed between May 18th and
> yesterday. ~May 18th was the last time I built -CURRENT during my last
> round of testing and I did not have any of these problems. Then someone
> emailed me recently and said there were some commits that might effect
> the outcome of the mysql benchmarks.

Ok, so these results are on a dual-processor XEON + hyperthreads, so four
logical processors. I used two dates off CVS, 20040515 and 20040615. I
also benchmarked my netperf branch. I don't have RELENG_4 on the box, but
might be able to load RELENG_4 on it later this week. In each case, I
took ten samples, dropped the first value as getting into the cache, and
took the mean of the rest. For this test, I used the select test; I'll
try the other smack query set tomorrow. In each case, I ran with "10
1000" as the arguments to the test. I used the default threading
configuration in -CURRENT, which is libpthread (libkse).

Mean Stdev
20040515-UP 4752.27 14.63
20040515-SMP 2550.35 19.23

20040615-UP 4898.71 22.39
20040615-SMP 2666.93 32.01

Netperf-UP-giant 4902.41 14.3
Netperf-SMP-giant 2566.18 16.83
Netperf-UP-mpsafe 4799.35 22.04
Netperf-SMP-mpsafe 3022.51 18.06

Unfortunately, I can't turn off HTT remotely, and I'm guessing it damages
the SMP numbers a fair amount due to additional IPIs without benefit.
However, the numbers basically suggest that on my hardware, the UP
configuration is marginally faster than it was last month, and that if you
throw in the netperf branch, the SMP case is a moderate amount faster.
This suggests that either I'm just lucky, or that the performance loss
might be specific to the amd64 version of FreeBSD. I'm going to run some
more numbers tomorrow and try to post something more rigorous to the
-threads list.

I don't have RELENG_4 on the box or Linux on the box, but I may get a
chance to later this week.

[Daniel Eischen]

On Tue, 15 Jun 2004, Daryl Chance wrote:

> Sorry for the top post.
>
> We've had this same problem (with mysqld locking up).
> I was able to get a trace, but since it's a live sql
> box, I didn't have time to find your email (If I knew
> I had kept a copy in my inbox, I would have done this
> correctly, instead I thought I had to look up the
> email in the archives) but I was able to get a ktrace
> -p pid output of the process. I let it run about 10
> seconds and then killall -11 mysqld'd the process (it
> allows mysqld to shutdown gracefully, thus not
> corrupting our tables).
>
> Here is the link to the .out file
> http://sql.tribalwar.com/ktrace.out
>
> here is the link to the .txt file of the kdump output
> http://sql.tribalwar.com/kdump.txt
>
> That was all I could remember to do in the short time
> I had to get it back up and running :).
>
> It's a stock kernel (w/ some minor things commented
> out). It's FBSD 5.2.1 p6
> http://sql.tribalwar.com/SQL

I don't think 5.2.1 is going to tell us much. We know there
was a problem in 5.2-release. We really need -current.

--
Dan Eischen

[Daryl Chance]

We're actually upgrading tomorrow night to -CURRENT,
if it continues, I will let you guys know. :).

-----Original Message-----
From: owner-free...@freebsd.org
[mailto:owner-free...@freebsd.org] On Behalf

Of Daniel Eischen
Sent: Wednesday, June 16, 2004 1:45 AM
To: Daryl Chance
Cc: freebsd...@freebsd.org; jul...@elischer.org;
mi...@mike2k.com; freebsd...@freebsd.org
Subject: RE: Possible Threading problem with -CURRENT
/ MySQL?

--
Dan Eischen

_______________________________________________

[Robert Watson]

On Wed, 16 Jun 2004, Robert Watson wrote:

> Netperf-UP-giant 4902.41 14.3
> Netperf-SMP-giant 2566.18 16.83
> Netperf-UP-mpsafe 4799.35 22.04
> Netperf-SMP-mpsafe 3022.51 18.06

FYI, when I add ADAPTIVE_MUTEXES on this box, the mean of 3000 q/s goes to
4000 q/s on the netperf+smp+mpsafenet number.

I'm off to bed now, but it seems like (at least in the HTT configuration),
it makes a big difference. I'll run the remainder of that set with
ADAPTIVE_MUTEXES tomorrow as well.

[Robert Watson]

On Wed, 16 Jun 2004, Robert Watson wrote:

> On Wed, 16 Jun 2004, Robert Watson wrote:
>
> > Netperf-UP-giant 4902.41 14.3
> > Netperf-SMP-giant 2566.18 16.83
> > Netperf-UP-mpsafe 4799.35 22.04
> > Netperf-SMP-mpsafe 3022.51 18.06
>
> FYI, when I add ADAPTIVE_MUTEXES on this box, the mean of 3000 q/s goes
> to 4000 q/s on the netperf+smp+mpsafenet number.
>
> I'm off to bed now, but it seems like (at least in the HTT
> configuration), it makes a big difference. I'll run the remainder of
> that set with ADAPTIVE_MUTEXES tomorrow as well.

And switching to 4BSD from ULE takes the mean to 6426 q/s when combined
with netperf, smp, debug.mpsafenet=1, and adaptive mutexes.

(Keeping in mind that this mysql benchmark is basically an
inter-thread/process kernel IPC benchmark using UNIX domain sockets, since
the workload is fairly minimal in userspace, this makes reasonable sense).

Now I'm really going to bed.

[Julian Elischer]

On Wed, 16 Jun 2004, Robert Watson wrote:

>
> On Wed, 16 Jun 2004, Robert Watson wrote:
>
> > On Wed, 16 Jun 2004, Robert Watson wrote:
> >
> > > Netperf-UP-giant 4902.41 14.3
> > > Netperf-SMP-giant 2566.18 16.83
> > > Netperf-UP-mpsafe 4799.35 22.04
> > > Netperf-SMP-mpsafe 3022.51 18.06
> >
> > FYI, when I add ADAPTIVE_MUTEXES on this box, the mean of 3000 q/s goes
> > to 4000 q/s on the netperf+smp+mpsafenet number.
> >
> > I'm off to bed now, but it seems like (at least in the HTT
> > configuration), it makes a big difference. I'll run the remainder of
> > that set with ADAPTIVE_MUTEXES tomorrow as well.
>
> And switching to 4BSD from ULE takes the mean to 6426 q/s when combined
> with netperf, smp, debug.mpsafenet=1, and adaptive mutexes.
>
> (Keeping in mind that this mysql benchmark is basically an
> inter-thread/process kernel IPC benchmark using UNIX domain sockets, since
> the workload is fairly minimal in userspace, this makes reasonable sense).
>
> Now I'm really going to bed.

One thing that was happenning at one time, was that when a process is
made runnable on
one processor, If another processor is idle, it remains idle until the
next scheduling tick.

I proposed patches about a year ago that fixed this but "as far as I
know" it is still true.

The solution is to generate an IPI when you make a thread runnable and
you see an idle processor..

To see if this is the problem a quick test would be to move HZ to 5000
or something and see what happens.

[Julian Elischer]

I'd like to see what happens with HZ=1000
(or even 5000)

> _______________________________________________
> freebsd...@freebsd.org mailing list
> http://lists.freebsd.org/mailman/listinfo/freebsd-threads
> To unsubscribe, send any mail to "freebsd-threa...@freebsd.org"
>

[Cedric Tabary]

On 15/06/2004 13:12, Lasse Laursen wrote:

> Hi,
>
> Have you had any lockups in MySQL with your setup? We have tried to disable
> HTT and hopefully that will solve the problem.

No lockups for now, but it is not heavily loaded.

--
Cédric Tabary

[Julian Elischer]

Robert, can you try see if HZ=5000 (up from 100) changes
the performance?

On Wed, 16 Jun 2004, Robert Watson wrote:

[Robert Watson]

On Wed, 16 Jun 2004, Julian Elischer wrote:

> Robert, can you try see if HZ=5000 (up from 100) changes the
> performance?

I haven't explored the HZ=BIGGER case very thoroughly, but ran a couple of
test cases with HZ at 1000 instead of 100, and found that generally it
resulted in a slight (1.5%) performance drop in the smattering of cases I
looked at. The cases I looked at were NETPERF-SMP-MPSAFE-ADMTX-HZ=1000,
NETPERF-SMP-MPSAFE-ADMTX-4BSD-!HTT-HZ=1000.

In general, I found that the biggest variables improving performance
relative to out-of-box SMP configuration were:

- Removing Giant from UNIX domain sockets
- Running with SCHED_4BSD instead of SCHED_ULE
- Disabling HTT
- Using ADAPTIVE_MUTEXES

By doing these things, I went from an out-of-box queries/sec for the
simple select "smack" with 11 clients from 2667 q/s to 6955 q/s, or a bit
over doubling the transaction rate. That as compared to 4939 q/s, which
was the best UP result I got (Giant-free UNIX domain sockets, SCHED_4BSD):

20040615-UP 4898.71 q/s (22.39 stddev)
20040615-SMP 2666.93 q/s (32.01 stddev)
NETPERF-UP-MPSAFE-4BSD 4939.89 q/s (74.06 stddev)
NETPERF-SMP-MPSAFE-ADMTX-4BSD-!HTT 6955.18 q/s (156.91 stddev)

Some of my 4BSD performance numbers were also slightly pessimized with
what appears to be a property of the 4BSD scheduling period being longer
than ULE. With ULE, the results settled on one loop of the benchmark.
With 4BSD, it sometimes took 2-3 loops of the benchmark to settle. As a
result, 4BSD stddev's are also generally higher. I haven't tried
regenerating the results dropping those outliers.

Using out-of-box -CURRENT without the netperf patches necessary to run
UNIX domain sockets safely, I got best results using:

20040615-UP-4BSD 4886.84 q/s (46.03 stddev)
20040615-SMP-ADMTX-4BSD-!HTT 5838.76 q/s (45.11 stddev)

In my environment, I couldn't easily test ULE without HTT, since the HTT
disabling sysctl/tunable with ULE results in a hang.

(Obviously, in all of the above, WITNESS and INVARIANTS are disabled; I
didn't disable the userspace malloc debugging flags).

[Jon Noack]

On 06/17/04 01:07, Brad Knowles wrote:

> At 1:31 AM -0400 2004-06-17, Robert Watson wrote:
>> - Removing Giant from UNIX domain sockets
>

> Out of curiosity, is this something that could be relatively safely
> done in general? Any ideas on what the plan is for doing this as the
> default on -CURRENT?

Wasn't this already done? See this commit:
http://lists.freebsd.org/pipermail/cvs-src/2004-June/025082.html

>> - Disabling HTT - Using ADAPTIVE_MUTEXES
>

> These both sound like typical improvements, based on what I've seen
> on this list. Any ideas on when they might become the default?

ADAPTIVE_MUTEXES was already enabled by default on amd64. See this
commit (read thread for discussion):
http://lists.freebsd.org/pipermail/cvs-src/2004-June/025234.html

It appears no comprehensive testing has been done to check whether it
really does improve performance. Many signs do point that way, though.

>> - Running with SCHED_4BSD instead of SCHED_ULE
>

> This is the only one that really concerns me. This shows that we
> clearly need more work on ULE. Is there one particular thing that we
> seem to be tripping up on, or is it a multitude of things?

I think I'll switch to 4BSD until I see more work being done on ULE (I
read cvs-src as a hobby already so I'll know when to switch back). I
noticed a performance drop from 5.2.1-p8 (with 4BSD) to -CURRENT (with
ULE) when I upgraded, but I attributed it to other things. However, the
system still feels slower than before despite having since disabled
INVARIANTS, WITNESS, and userspace malloc debugging flags.

Jon Noack

[Brad Knowles]

At 1:31 AM -0400 2004-06-17, Robert Watson wrote:

> - Removing Giant from UNIX domain sockets

Out of curiosity, is this something that could be relatively

safely done in general? Any ideas on what the plan is for doing this
as the default on -CURRENT?

> - Disabling HTT
> - Using ADAPTIVE_MUTEXES

These both sound like typical improvements, based on what I've

seen on this list. Any ideas on when they might become the default?

> - Running with SCHED_4BSD instead of SCHED_ULE

This is the only one that really concerns me. This shows that we

clearly need more work on ULE. Is there one particular thing that we
seem to be tripping up on, or is it a multitude of things?

--
Brad Knowles, <brad.k...@skynet.be>

"They that can give up essential liberty to obtain a little temporary
safety deserve neither liberty nor safety."
-Benjamin Franklin, Historical Review of Pennsylvania.

SAGE member since 1995. See <http://www.sage.org/> for more info.

[Julian Elischer]

Robert, when it is going as fast as yu can make it, what is the bottleneck?
CPU? disk? net?
or just a procedural bottleneck somewhere.?

BTW what about libthr?

[Petri Helenius]

Brad Knowles wrote:

>
>> - Disabling HTT
>> - Using ADAPTIVE_MUTEXES
>
>
> These both sound like typical improvements, based on what I've
> seen on this list. Any ideas on when they might become the default?
>

HTT defaults to the BIOS setting, which I think is the logical course of
action. I don´t think the OS should make assumptions the way or another
over what the user has set in the BIOS.
(this used not to be the case earlier, and I asked it to default to
disable HTT back then)

Pete

[Robert Watson]

I agree in principle; however, as a variable in this benchmark, it seems
like regardless of scheduler it's a net loss. A question we haven't yet
answered is whether (for this workload, or other interesting workloads) it
should be a net loss, and therefore whether it's simply our support to HTT
is not yet mature enough to offer the theoretical advantage, or whether
HTT is just a bad idea. :-)

[Robert Watson]

On Thu, 17 Jun 2004, Jon Noack wrote:

> On 06/17/04 01:07, Brad Knowles wrote:
> > At 1:31 AM -0400 2004-06-17, Robert Watson wrote:
> >> - Removing Giant from UNIX domain sockets
> >
> > Out of curiosity, is this something that could be relatively safely
> > done in general? Any ideas on what the plan is for doing this as the
> > default on -CURRENT?
>
> Wasn't this already done? See this commit:
> http://lists.freebsd.org/pipermail/cvs-src/2004-June/025082.html

The locks have been introduced to protect UNIX domain sockets, but the
merge of locking for the cross-protocol socket infrastructure isn't
complete yet. This means that if you disable Giant, the UNIX domain bits
work fine, but the socket buffers, socket state, and socket event handling
will probably corrupt themselves rapidly on SMP. Per an earlier e-mail,
I'm working to complete that merge, but took a break to run some
benchmarks. Looks for more information here in about a week and a half.

> >> - Disabling HTT - Using ADAPTIVE_MUTEXES
> >
> > These both sound like typical improvements, based on what I've seen
> > on this list. Any ideas on when they might become the default?
>
> ADAPTIVE_MUTEXES was already enabled by default on amd64. See this
> commit (read thread for discussion):
> http://lists.freebsd.org/pipermail/cvs-src/2004-June/025234.html
>
> It appears no comprehensive testing has been done to check whether it
> really does improve performance. Many signs do point that way, though.

Agreed. I think we need to run a broad range of benchmarks across a
number of hardware configurations in order to make this decision. Results
on a couple of interesting benchmarks, though, are very suggestive, so we
should probably do this sooner rather than later.

> >> - Running with SCHED_4BSD instead of SCHED_ULE
> >
> > This is the only one that really concerns me. This shows that we
> > clearly need more work on ULE. Is there one particular thing that we
> > seem to be tripping up on, or is it a multitude of things?
>
> I think I'll switch to 4BSD until I see more work being done on ULE (I
> read cvs-src as a hobby already so I'll know when to switch back). I
> noticed a performance drop from 5.2.1-p8 (with 4BSD) to -CURRENT (with
> ULE) when I upgraded, but I attributed it to other things. However, the
> system still feels slower than before despite having since disabled
> INVARIANTS, WITNESS, and userspace malloc debugging flags.

ULE seems to do a very good job of scheduling interactive tasks over other
workloads, resulting in a very "snappy" feel on my boxes, despite heavy
CPU load from background builds, etc. The workload I looked at had no
real "interactive" component, although it was a latency-centric RPC test,
so timely hand-off as well as high throughput would be important. I know
that Jeff's measurement work on ULE had a substantial focus on deadlines
-- whether or not ULE was timely in scheduling tasks, etc, and that he
demonstrated that it was much stronger than most other available
schedulers in this area.

One of the next obvious steps in optimizing either ULE or 4BSD is going to
be to spend a lot of time sitting with KTR(4) and looking at context
switch traces for "dumb things", such as bouncing between CPUs, rapid
switches back and forth, undo multiple wakeups, etc.

[Robert Watson]

On Thu, 17 Jun 2004, Brad Knowles wrote:

> At 1:31 AM -0400 2004-06-17, Robert Watson wrote:
>
> > - Removing Giant from UNIX domain sockets
>
> Out of curiosity, is this something that could be relatively
> safely done in general? Any ideas on what the plan is for doing this as
> the default on -CURRENT?

I'm in the process of merging the necessary support to CVS, but took a
break for a day or two to run some benchmarks and wait for some reviews of
specific pieces of it. I'll restart the merge process this evening. You
can find more more information at:

http://www.watson.org/~robert/freebsd/netperf/

> > - Disabling HTT
> > - Using ADAPTIVE_MUTEXES
>
> These both sound like typical improvements, based on what I've
> seen on this list. Any ideas on when they might become the default?

There's an interaction with HTT that prevents the OS from disabling HTT
when running SCHED_ULE. The benefits and costs of HTT vary based on load;
with this particular benchmark, HTT hurts quite a bit.

Regarding ADAPTIVE_MUTEXES -- I think we still need more information. I'm
convinced that on high-contention local IPC workloads, ADAPTIVE_MUTEXES is
a clear win across all other variables -- regardless of scheduler, use of
HTT, fine-grained locking, etc. However, I haven't benchmarked more
userspace-intensive workloads, and that's where you'd expect it might
hurt. ADAPTIVE_MUTEXES should (may?) hurt "less" for high user loads when
contention spots are brief, such as in a more fine-grained locking
scenario rather than with a Giant lock. After I finish the netperf merge,
I'll be doing a set of detailed performance measurements and optimizations
and we'll see if that notion plays out or not. It could also well be that
ADAPTIVE_MUTEXES are simply a win. There are also two proposed
modifications to ADAPTIVE_MUTEXES, one by Bosko Milekic and the other by
Alfred Perlstein, and we'll want to look at them as well.

> > - Running with SCHED_4BSD instead of SCHED_ULE
>
> This is the only one that really concerns me. This shows that
> we clearly need more work on ULE. Is there one particular thing that we
> seem to be tripping up on, or is it a multitude of things?

It's not immediately clear; this benchmark is fairly thread-centric, so it
may just be that ULE is doing a bad job of sharing a quantum among
multiple threads in the same process. Or, we could be looking at a
load-balancing issue or other issue. For this benchmark, which involved
lots of smaller IPCs across processors, ULE hurt quite a bit compared to
4BSD. On UP, the results were identical. Since this benchmark is a very
specific and narrow workload (yet important), I think we don't yet have
enough information to make a decision as to whether to keep refining ULE
for 5.3, or switch back to 4BSD for 5.3. Presumably the largest single
concern is actually whether Jeff or others will have time to work on ULE.y

[Jon Noack]

Well, I switched to 4BSD and noticed this right away. I occasionally
get sound hiccups under heavy i/o with ULE [1], but with 4BSD it's a bit
ridiculous. There's a PR open about it (can't remember which off the
top of my head), but sound will skip and slow down (play at half speed
or slower). I know the sound locking is not all it should be, so this
may not be a representative test. Regardless, my workstations are back
on ULE...

Jon

[1] Often when using tar or bsdtar. To reproduce:
cd /usr/ports/www/firefox; make extract

[Robert Watson]

Hrm. So, I'm not really a scheduler guy, but I have some ideas about how
to investigate what's going on. If you feel like playing with kernel
tracing facilities, there are some really neat things you can do with
ktr(4). It allows you took hook context switch and interrupt delivery
events and dump a trace to userspace.

One possible cause of the symptoms you're seeing is poor interrupt
handling response time -- possibly a failure to schedule the ithread in a
timely manner. It would be quite interesting to compare the latency in
scheduling (and running) the ithread across ULE and 4BSD and see how well
each is managing to do in getting it running quickly. I'm not sure
there's a tutorial on how to get KTR(4) running -- I found the man page
helpful, but a little confusing. What you'll want to do, assuming you're
willing to get into this, is turn on KTR tracing with KTR_INTR and
KTR_PROC. You'll need to set the flags in the kernel compile, and then
set them using a sysctl. You can use ktrdump to dump a record trace --
I'd use a number of flags present to add timestamps, cpu numbers if SMP,
etc. With some post-processing, it should be possible to generate a
distribution of "interrupt time" to "running interrupt handler".

It would be useful to demonstrate to oneself that ithreads are running
quickly, preempting things they need to, etc.

[Bruce Evans]

On Thu, 17 Jun 2004, Robert Watson wrote:

> On Thu, 17 Jun 2004, Jon Noack wrote:
>

> > > ... I know that Jeff's measurement work

> > > on ULE had a substantial focus on deadlines -- whether or not ULE was
> > > timely in scheduling tasks, etc, and that he demonstrated that it was
> > > much stronger than most other available schedulers in this area.
> >
> > Well, I switched to 4BSD and noticed this right away. I occasionally
> > get sound hiccups under heavy i/o with ULE [1], but with 4BSD it's a bit
> > ridiculous. There's a PR open about it (can't remember which off the
> > top of my head), but sound will skip and slow down (play at half speed
> > or slower). I know the sound locking is not all it should be, so this
> > may not be a representative test. Regardless, my workstations are back
> > on ULE...
>
> Hrm. So, I'm not really a scheduler guy, but I have some ideas about how
> to investigate what's going on. If you feel like playing with kernel
> tracing facilities, there are some really neat things you can do with
> ktr(4). It allows you took hook context switch and interrupt delivery
> events and dump a trace to userspace.

Interrupt delivery if well known to be broken (interrupts that can't
switch to their ithread immediately, perhaps because they are scheduled
while in a critical region, just set a flag, and the flag is not checked
until return to user mode which may happen a long time later, so the
current current thread keeps running until it either gives up control
or another interrupt that can be switched to immediately is recieved
(then the flag becomes irrelevant and the highest priority ithread
is switched too; it may even be the one for the first interrupt).

The effects of this can be seen in ping latency. For
"ping -fq -c1000000 besplex" over 100Mb/S ethernet using an old version
of ping that I keep for this benchmark:

Under -current on an unloaded Celeron 366:
--- besplex.bde.org ping statistics ---
100002 packets transmitted, 100000 packets received, 0% packet loss
round-trip min/avg/max/stddev = 0.172/0.393/20.336/0.191 ms
13.60 real 0.74 user 7.66 sys

The latency is sometimes 20ms (2 clock ticks).

Under my version of -current (which is missing the bug) on the same machine:
--- besplex.bde.org ping statistics ---
100001 packets transmitted, 100000 packets received, 0% packet loss
round-trip min/avg/max/stddev = 0.156/0.204/2.449/0.099 ms
10.92 real 0.80 user 5.91 sys

The worst-case latency is still bad, but is much smaller, and the real
and system times are much smaller too (the latter may not be significant --
flood pings sometimes give strange behaviour (capture effect?) where
synchronization gives a larger latency but a smaller real time. But here
the latency is smaller too.

> One possible cause of the symptoms you're seeing is poor interrupt
> handling response time -- possibly a failure to schedule the ithread in a
> timely manner. It would be quite interesting to compare the latency in
> scheduling (and running) the ithread across ULE and 4BSD and see how well
> each is managing to do in getting it running quickly. I'm not sure

The scheduler really shouldn't affect scheduling of ithreads. The 4BSD
scheduler just picks the highest priority ithread (round robin among equal
priorities) and runs it, and doing much more than that would be wrong.
ULE may be reducing the effect of bugs by scheduling more often. Or perhaps
it is better for sound because it handles competition between user threads
and ithreads better. Ithreads are supposed to always have higher priority
than user threads; however, the 4BSD scheduler still has the bugfeature of
not enforcing this, so that when a user thread in kernel mode wakes up with
a high kernel priority, it essentially keeps the kernel priority when it
returns to user mode (until it gives up control or is interrupted). This
bugfeature seems to be more responsible for the 4BSD scheduler's good
interactive behaviour than anything it does intentionally. I believe the
ULE scheduler does similar things intentionally.

> there's a tutorial on how to get KTR(4) running -- I found the man page
> helpful, but a little confusing. What you'll want to do, assuming you're
> willing to get into this, is turn on KTR tracing with KTR_INTR and
> KTR_PROC. You'll need to set the flags in the kernel compile, and then
> set them using a sysctl. You can use ktrdump to dump a record trace --
> I'd use a number of flags present to add timestamps, cpu numbers if SMP,
> etc. With some post-processing, it should be possible to generate a
> distribution of "interrupt time" to "running interrupt handler".
>
> It would be useful to demonstrate to oneself that ithreads are running
> quickly, preempting things they need to, etc.

I think listening to the sound is the actually easiest way to see if
interrupt latency is not too large, at least if 100uS (corresponding
to 10kHz) is not too large :-). Too bad I can't hear it well enough.
When I fixed interrupt problems for 386BSD-0.0, I set up a visual
indicator using about 10 words of VGA memory at the top of the screen.
It was updated on every change to the interrupt masks. Latency bugs
in the 100uS range are far too short to see, but I could easily see
some larger bugs.

Bruce

[David Xu]

When this pthread program runs under ULE, I can not interrupt it
pressing ctrl+c has no effect, I have this problem for a long time.
http://people.freebsd.org/~davidxu/kse/locktest.c

David Xu

Robert Watson wrote:
> ULE seems to do a very good job of scheduling interactive tasks over
> other
> workloads, resulting in a very "snappy" feel on my boxes, despite heavy
> CPU load from background builds, etc. The workload I looked at had no
> real "interactive" component, although it was a latency-centric RPC
> test,
> so timely hand-off as well as high throughput would be important. I
> know
> that Jeff's measurement work on ULE had a substantial focus on deadlines
> -- whether or not ULE was timely in scheduling tasks, etc, and that he
> demonstrated that it was much stronger than most other available
> schedulers in this area.
>
> One of the next obvious steps in optimizing either ULE or 4BSD is going
> to
> be to spend a lot of time sitting with KTR(4) and looking at context
> switch traces for "dumb things", such as bouncing between CPUs, rapid
> switches back and forth, undo multiple wakeups, etc.
>
> Robert N M Watson FreeBSD Core Team, TrustedBSD Projects
> rob...@fledge.watson.org Senior Research Scientist, McAfee Research
>

> _______________________________________________
> freebsd...@freebsd.org mailing list
> http://lists.freebsd.org/mailman/listinfo/freebsd-current

> To unsubscribe, send any mail to

> "freebsd-curre...@freebsd.org"
>

[Claus Guttesen]

> When this pthread program runs under ULE, I can not
> interrupt it
> pressing ctrl+c has no effect, I have this problem
> for a long time.
> http://people.freebsd.org/~davidxu/kse/locktest.c
> David Xu

frodo~/temp%>cc -O2 -Wall -o pthread_lock
pthread_lock.c -lpthread

frodo~/temp%>./pthread_lock
^C
frodo~/temp%>echo $?
130

Pressing ctrl-c worked here. ULE, current from June
15'th.

regards
Claus

Yahoo! Mail (http://dk.mail.yahoo.com) - Gratis: 6 MB lagerplads, spamfilter og virusscan

[Christian Jachmann]

On Tue, Jun 15, 2004 at 05:09:51PM +0200, Martin Blapp wrote:
>
> set-variable = key_buffer_size=16M
> set-variable = table_cache=256
> set-variable = record_buffer=1M
> set-variable = sort_buffer=16M

Why only so uninspired values?
> avail memory = 2095964160 (1998 MB)
there seems to be lot of memory?

--
Christian Jachmann

[Daniel Eischen]

On Fri, 18 Jun 2004, [iso-8859-1] Claus Guttesen wrote:

> > When this pthread program runs under ULE, I can not
> > interrupt it
> > pressing ctrl+c has no effect, I have this problem
> > for a long time.
> > http://people.freebsd.org/~davidxu/kse/locktest.c
> > David Xu
>
> frodo~/temp%>cc -O2 -Wall -o pthread_lock
> pthread_lock.c -lpthread
>
> frodo~/temp%>./pthread_lock
> ^C
> frodo~/temp%>echo $?
> 130
>
> Pressing ctrl-c worked here. ULE, current from June
> 15'th.

SMP or UP?

--
Dan Eischen

[Claus Guttesen]

> > frodo~/temp%>cc -O2 -Wall -o pthread_lock
> > pthread_lock.c -lpthread
> >
> > frodo~/temp%>./pthread_lock
> > ^C
> > frodo~/temp%>echo $?
> > 130
> >
> > Pressing ctrl-c worked here. ULE, current from
> June
> > 15'th.
>
> SMP or UP?

SMP, ACPI APIC Table: <ASUS CUV4XDLS>, dual PIII, 1
GB of RAM.

[jesk]

i got problems too with mysql under freebsd 5.2.1 and CURRENT.

i got these problems on two machines with different hardware, but both are
single cpu machines.
used scheduler is ULE.

problem description:
under high i/o load the mysqld isnt responding any more.
i tested this with dd if=/dev/urandom of=testfile bs=128k on the hdd an was
not
able to become any ouput from simple select-statements within a long period
of time
(above minutes).
i dont know if this problem occurs from mysql and its threads or if the i/o
subsystem has
any problems to deal fast enough with two intensives read and write
processes and its
priorities.

best regards,
christian