[AOLSERVER] futex problems

Wolfgang Winkler

unread,

Apr 24, 2013, 3:12:59 AM4/24/13

to aolserv...@lists.sourceforge.net

Hi!

We have a couple of servers running as openvz guests and on bare metal. Some of them are much slower due to a high latency when calling even very small pages. The strace output (see below) suggests, that the interpreters are initialized on each call.

There have been problems in some kernels with leap seconds, so we followed the advice found on the web and updated the kernels. The problem is still there. On one machine I've compiled the latest AOLserver version from git, still with the same problem.

Systems with these kernel are working fine:

Linux 2.6.32-238.9.1.el5.028stab089.1
Linux 2.6.32-5-openvz-amd64

Systems with these not:
Linux 2.6.32-5-openvz-amd64
Linux 2.6.26-1-amd64

Has anybody encountered this problem as well?

Regards,

Wolfgang

futex(0x12fd118, FUTEX_WAIT_PRIVATE, 2, NULL) = -1 EAGAIN (Resource temporarily unavailable)
futex(0x12fd118, FUTEX_WAKE_PRIVATE, 1) = 0
futex(0x12fd118, FUTEX_WAKE_PRIVATE, 1) = 1
futex(0x12fd0d8, FUTEX_WAKE_PRIVATE, 1) = 1
futex(0x12fd118, FUTEX_WAKE_PRIVATE, 1) = 1
futex(0x12fd118, FUTEX_WAIT_PRIVATE, 2, NULL) = -1 EAGAIN (Resource temporarily unavailable)
futex(0x12fd118, FUTEX_WAKE_PRIVATE, 1) = 1
futex(0x12fd118, FUTEX_WAKE_PRIVATE, 1) = 1
futex(0x12fd118, FUTEX_WAKE_PRIVATE, 1) = 1
futex(0x12fd118, FUTEX_WAKE_PRIVATE, 1) = 1
futex(0x12fd118, FUTEX_WAKE_PRIVATE, 1) = 1
futex(0x12fd1d8, FUTEX_WAIT_PRIVATE, 2, NULL) = -1 EAGAIN (Resource temporarily unavailable)
futex(0x12fd1d8, FUTEX_WAKE_PRIVATE, 1) = 0
futex(0x12fd118, FUTEX_WAKE_PRIVATE, 1) = 1
futex(0x12fd118, FUTEX_WAKE_PRIVATE, 1) = 1
futex(0x12fd0d8, FUTEX_WAKE_PRIVATE, 1) = 1
futex(0x12fd0d8, FUTEX_WAIT_PRIVATE, 2, NULL) = -1 EAGAIN (Resource temporarily unavailable)
futex(0x12fd0d8, FUTEX_WAKE_PRIVATE, 1) = 0
futex(0x12fd0d8, FUTEX_WAKE_PRIVATE, 1) = 1
futex(0x12fd0d8, FUTEX_WAIT_PRIVATE, 2, NULL) = -1 EAGAIN (Resource temporarily unavailable)
futex(0x12fd0d8, FUTEX_WAKE_PRIVATE, 1) = 0
futex(0x12fd098, FUTEX_WAKE_PRIVATE, 1) = 1
futex(0x12fd0d8, FUTEX_WAKE_PRIVATE, 1) = 1
futex(0x12fd0d8, FUTEX_WAIT_PRIVATE, 2, NULL) = -1 EAGAIN (Resource temporarily unavailable)
futex(0x12fd0d8, FUTEX_WAKE_PRIVATE, 1) = 0
futex(0x12fd118, FUTEX_WAKE_PRIVATE, 1) = 1
futex(0x12fd0d8, FUTEX_WAKE_PRIVATE, 1) = 1
futex(0x12fd118, FUTEX_WAKE_PRIVATE, 1) = 1
futex(0x12fd118, FUTEX_WAIT_PRIVATE, 2, NULL) = -1 EAGAIN (Resource temporarily unavailable)
futex(0x12fd118, FUTEX_WAKE_PRIVATE, 1) = 1
futex(0x12fd0d8, FUTEX_WAIT_PRIVATE, 2, NULL) = -1 EAGAIN (Resource temporarily unavailable)
futex(0x12fd0d8, FUTEX_WAKE_PRIVATE, 1) = 0
futex(0x12fd118, FUTEX_WAKE_PRIVATE, 1) = 1
futex(0x12fd118, FUTEX_WAKE_PRIVATE, 1) = 1
futex(0x12fd118, FUTEX_WAIT_PRIVATE, 2, NULL) = -1 EAGAIN (Resource temporarily unavailable)
futex(0x12fd118, FUTEX_WAKE_PRIVATE, 1) = 0
futex(0x12fd0d8, FUTEX_WAIT_PRIVATE, 2, NULL) = -1 EAGAIN (Resource temporarily unavailable)
futex(0x12fd0d8, FUTEX_WAKE_PRIVATE, 1) = 0
futex(0x12fd0d8, FUTEX_WAKE_PRIVATE, 1) = 1
futex(0x12fd0d8, FUTEX_WAKE_PRIVATE, 1) = 1
futex(0x12fd0d8, FUTEX_WAIT_PRIVATE, 2, NULL) = -1 EAGAIN (Resource temporarily unavailable)
futex(0x12fd0d8, FUTEX_WAKE_PRIVATE, 1) = 0
futex(0x12fd118, FUTEX_WAKE_PRIVATE, 1) = 1
futex(0x12fd0d8, FUTEX_WAIT_PRIVATE, 2, NULL) = -1 EAGAIN (Resource temporarily unavailable)
futex(0x12fd0d8, FUTEX_WAKE_PRIVATE, 1) = 0
futex(0x12fd0d8, FUTEX_WAIT_PRIVATE, 2, NULL) = -1 EAGAIN (Resource temporarily unavailable)
futex(0x12fd0d8, FUTEX_WAKE_PRIVATE, 1) = 0
futex(0x12fd0d8, FUTEX_WAIT_PRIVATE, 2, NULL) = -1 EAGAIN (Resource temporarily unavailable)
futex(0x12fd0d8, FUTEX_WAKE_PRIVATE, 1) = 0
futex(0x12fd0d8, FUTEX_WAIT_PRIVATE, 2, NULL) = -1 EAGAIN (Resource temporarily unavailable)
futex(0x12fd0d8, FUTEX_WAKE_PRIVATE, 1) = 0
futex(0x12fd0d8, FUTEX_WAIT_PRIVATE, 2, NULL) = -1 EAGAIN (Resource temporarily unavailable)
futex(0x12fd0d8, FUTEX_WAKE_PRIVATE, 1) = 0
futex(0x12fd118, FUTEX_WAKE_PRIVATE, 1) = 1
futex(0x12fd0d8, FUTEX_WAIT_PRIVATE, 2, NULL) = -1 EAGAIN (Resource temporarily unavailable)
futex(0x12fd0d8, FUTEX_WAKE_PRIVATE, 1) = 0
futex(0x12fd0d8, FUTEX_WAKE_PRIVATE, 1) = 1
futex(0x12fd0d8, FUTEX_WAIT_PRIVATE, 2, NULL) = -1 EAGAIN (Resource temporarily unavailable)
futex(0x12fd0d8, FUTEX_WAKE_PRIVATE, 1) = 0
futex(0x12fd118, FUTEX_WAKE_PRIVATE, 1) = 1
futex(0x1339b24, FUTEX_WAIT_PRIVATE, 1893, NULL) = 0
futex(0x1357610, FUTEX_WAKE_PRIVATE, 1) = 0
futex(0x1339b24, FUTEX_WAIT_PRIVATE, 1895, NULL) = 0
futex(0x1357610, FUTEX_WAKE_PRIVATE, 1) = 0
futex(0x1339b24, FUTEX_WAIT_PRIVATE, 1897, NULL) = 0
futex(0x1357610, FUTEX_WAKE_PRIVATE, 1) = 0
futex(0x1339aa4, FUTEX_CMP_REQUEUE_PRIVATE, 1, 2147483647, 0x1357610, 2640) = 3
futex(0x1357610, FUTEX_WAKE_PRIVATE, 1) = 1
madvise(0x7fb9fe647000, 5615616, MADV_DONTNEED) = 0
futex(0x12fd218, FUTEX_WAIT_PRIVATE, 2, NULL) = -1 EAGAIN (Resource temporarily unavailable)
futex(0x12fd218, FUTEX_WAKE_PRIVATE, 1) = 0
futex(0x12fd1d8, FUTEX_WAIT_PRIVATE, 2, NULL) = -1 EAGAIN (Resource temporarily unavailable)
futex(0x12fd1d8, FUTEX_WAKE_PRIVATE, 1) = 0
futex(0x12fd0d8, FUTEX_WAIT_PRIVATE, 2, NULL) = -1 EAGAIN (Resource temporarily unavailable)
futex(0x12fd0d8, FUTEX_WAKE_PRIVATE, 1) = 0
uname({sys="Linux", node="peregrin-04-vhost-01", ...}) = 0
getuid()ï¿½ï¿½ï¿½ï¿½ï¿½ï¿½ï¿½ï¿½ï¿½ï¿½ï¿½ï¿½ï¿½ï¿½ï¿½ï¿½ï¿½ï¿½ï¿½ï¿½ï¿½ï¿½ï¿½ï¿½ï¿½ï¿½ï¿½ï¿½ï¿½ï¿½ï¿½ = 1001
open("/etc/passwd", O_RDONLY|O_CLOEXEC) = 32
lseek(32, 0, SEEK_CUR)ï¿½ï¿½ï¿½ï¿½ï¿½ï¿½ï¿½ï¿½ï¿½ï¿½ï¿½ï¿½ï¿½ï¿½ï¿½ï¿½ï¿½ = 0
fstat(32, {st_mode=S_IFREG|0644, st_size=1525, ...}) = 0
mmap(NULL, 1525, PROT_READ, MAP_SHARED, 32, 0) = 0x7fba13fff000
lseek(32, 1525, SEEK_SET)ï¿½ï¿½ï¿½ï¿½ï¿½ï¿½ï¿½ï¿½ï¿½ï¿½ï¿½ï¿½ï¿½ï¿½ = 1525
munmap(0x7fba13fff000, 1525)ï¿½ï¿½ï¿½ï¿½ï¿½ï¿½ï¿½ï¿½ï¿½ï¿½ï¿½ = 0
close(32)ï¿½ï¿½ï¿½ï¿½ï¿½ï¿½ï¿½ï¿½ï¿½ï¿½ï¿½ï¿½ï¿½ï¿½ï¿½ï¿½ï¿½ï¿½ï¿½ï¿½ï¿½ï¿½ï¿½ï¿½ï¿½ï¿½ï¿½ï¿½ï¿½ï¿½ = 0
futex(0x12fd0d8, FUTEX_WAIT_PRIVATE, 2, NULL) = 0
futex(0x12fd0d8, FUTEX_WAKE_PRIVATE, 1) = 0
lstat("/usr", {st_mode=S_IFDIR|0755, st_size=4096, ...}) = 0
lstat("/usr/local", {st_mode=S_IFDIR|S_ISGID|0775, st_size=4096, ...}) = 0
lstat("/usr/local/lib", {st_mode=S_IFDIR|S_ISGID|0775, st_size=4096, ...}) = 0
lstat("/usr/local/lib/tcl8.5", {st_mode=S_IFDIR|S_ISGID|0755, st_size=4096, ...}) = 0
access("/usr/local/lib/tcl8.5/init.tcl", F_OK) = 0
stat("/usr/local/lib/tcl8.5/init.tcl", {st_mode=S_IFREG|0644, st_size=25031, ...}) = 0
open("/usr/local/lib/tcl8.5/init.tcl", O_RDONLY) = 32
fcntl(32, F_SETFD, FD_CLOEXEC)ï¿½ï¿½ï¿½ï¿½ï¿½ï¿½ï¿½ï¿½ï¿½ = 0
ioctl(32, SNDCTL_TMR_TIMEBASE or TCGETS, 0x7fba5745f7c0) = -1 ENOTTY (Inappropriate ioctl for device)
read(32, "# init.tcl --\n#\n# Default system"..., 4096) = 4096
read(32, "cmd.exe\n\t\t} else {\n\t\tï¿½ï¿½ï¿½ set env"..., 4096) = 4096
read(32, "} {\n\t\tset ::errorInfo $savedErro"..., 4096) = 4096
read(32, " error \"invalid command name \\\"$"..., 4096) = 4096
read(32, "(making sure that foo:::::bar wi"..., 4096) = 4096
read(32, " {[info exists env(WINDIR)]} {\n\t"..., 4096) = 4096
read(32, "verlapping.\nï¿½ï¿½ï¿½ # \nï¿½ï¿½ï¿½ # On Unix"..., 4096) = 455
read(32, "", 4096)ï¿½ï¿½ï¿½ï¿½ï¿½ï¿½ï¿½ï¿½ï¿½ï¿½ï¿½ï¿½ï¿½ï¿½ï¿½ï¿½ï¿½ï¿½ï¿½ï¿½ï¿½ = 0
close(32)ï¿½ï¿½ï¿½ï¿½ï¿½ï¿½ï¿½ï¿½ï¿½ï¿½ï¿½ï¿½ï¿½ï¿½ï¿½ï¿½ï¿½ï¿½ï¿½ï¿½ï¿½ï¿½ï¿½ï¿½ï¿½ï¿½ï¿½ï¿½ï¿½ï¿½ = 0
futex(0x12fd0d8, FUTEX_WAKE_PRIVATE, 1) = 1
futex(0x12fd0d8, FUTEX_WAKE_PRIVATE, 1) = 1
futex(0x12fd0d8, FUTEX_WAKE_PRIVATE, 1) = 1

Am 2013-04-23 17:12, schrieb Ulf Meyer:

Hallo Wolfgang,

Tommaso erzï¿½hlte mir es gï¿½be bei euch gewisse Serverprobleme.
Irgendetwas von zyklischen Aussetzern im 400ms Bereich ï¿½
Kï¿½nnen wir helfen ?

Melde dich doch einfach bei mir ?

Gruï¿½

Ulf

--

Wolfgang Winkler
Geschï¿½ftsfï¿½hrung
wolfgang...@digital-concepts.com
mobil +43.699.19971172

dc:bï¿½ro
digital concepts Novak Winkler OG
Software & Design
Landstraï¿½e 68, 5. Stock, 4020 Linz
www.digital-concepts.com
tel +43.732.997117.72
tel +43.699.1997117.72

Firmenbuchnummer: 192003h
Firmenbuchgericht: Landesgericht Linz

PS: BESUCHEN SIE UNSERE NEUE SHOP INFO SEITE: www.shop-info.at

Jeff Rogers

unread,

Apr 24, 2013, 1:43:50 PM4/24/13

to Wolfgang Winkler, aolserv...@lists.sourceforge.net

Wolfgang Winkler wrote:
> Hi!
>
> We have a couple of servers running as openvz guests and on bare metal.
> Some of them are much slower due to a high latency when calling even
> very small pages. The strace output (see below) suggests, that the
> interpreters are initialized on each call.

Are the two sets of servers running with the same configuration, in
particular the "maxconnections" setting in the ns/servers/$servername
section?

-J

>
> There have been problems in some kernels with leap seconds, so we
> followed the advice found on the web and updated the kernels. The
> problem is still there. On one machine I've compiled the latest
> AOLserver version from git, still with the same problem.
>
> Systems with these kernel are working fine:
>
> Linux 2.6.32-238.9.1.el5.028stab089.1
> Linux 2.6.32-5-openvz-amd64
>
> Systems with these not:
> Linux 2.6.32-5-openvz-amd64
> Linux 2.6.26-1-amd64

------------------------------------------------------------------------------
Try New Relic Now & We'll Send You this Cool Shirt
New Relic is the only SaaS-based application performance monitoring service
that delivers powerful full stack analytics. Optimize and monitor your
browser, app, & servers with just a few lines of code. Try New Relic
and get this awesome Nerd Life shirt! http://p.sf.net/sfu/newrelic_d2d_apr
_______________________________________________
aolserver-talk mailing list
aolserv...@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/aolserver-talk

Wolfgang Winkler

unread,

Apr 25, 2013, 2:17:29 AM4/25/13

to aolserv...@lists.sourceforge.net

Hi!

The servers are running with the same configuration and I tried various maxconnections settings. ns_pools get default returns

minthreads 60 maxthreads 200 idle 60 current 60 maxconns 0 queued 58366 timeout 1200 spread 20

I also tried with maxconns settings of 10000 and various other values. And I don't see the "exiting: exceeded max connections per thread" messages in the log.

Regards, Wolfgang

Am 2013-04-24 19:43, schrieb Jeff Rogers:

Wolfgang Winkler wrote:

Hi!

We have a couple of servers running as openvz guests and on bare metal.
Some of them are much slower due to a high latency when calling even
very small pages. The strace output (see below) suggests, that the
interpreters are initialized on each call.

Are the two sets of servers running with the same configuration, in particular the "maxconnections" setting in the ns/servers/$servername section?

-J

There have been problems in some kernels with leap seconds, so we
followed the advice found on the web and updated the kernels. The
problem is still there. On one machine I've compiled the latest
AOLserver version from git, still with the same problem.

Systems with these kernel are working fine:

Linux 2.6.32-238.9.1.el5.028stab089.1
Linux 2.6.32-5-openvz-amd64

Systems with these not:
Linux 2.6.32-5-openvz-amd64
Linux 2.6.26-1-amd64

--

Wolfgang Winkler
Geschäftsführung
wolfgang...@digital-concepts.com
mobil +43.699.19971172

dc:büro

digital concepts Novak Winkler OG
Software & Design

Landstraße 68, 5. Stock, 4020 Linz

Wolfgang Winkler

unread,

Apr 25, 2013, 3:00:31 AM4/25/13

to Maurizio Martignano, aolserv...@lists.sourceforge.net

Hi Maurizio!

I don't think virtualizaton itself is the problem, as we have systems running on openvz which behave perfectly and others not virtualized which show the problem. Also I can see in the stack trace, that the threads are trying to get initialized repeatedly.

Wolfgang

Am 2013-04-25 08:36, schrieb Maurizio Martignano:

Dear Wolfgang,

I never used OpenVZ technology but I used VMware quite a lot.

With VMware the adopted resource allocation policies are very important.

If they are dynamic, the machine hasn’t got its resources actually allocated to it till the machine uses them (e.g. to serve a request). And after a while it doesn’t use these resources, they get released.

This dynamic allocation of resources might explain the latency you are observing.

If the resource allocation is static, then the latency due to the dynamic adjustments disappears.

Of course all of this is happening at virtualization level and this is why the Aolserver configuration parameters might be not so relevant.

Hope it helps,

Maurizio

Maurizio Martignano

unread,

Apr 25, 2013, 2:36:51 AM4/25/13

to Wolfgang Winkler, aolserv...@lists.sourceforge.net

Dear Wolfgang,

I never used OpenVZ technology but I used VMware quite a lot.

With VMware the adopted resource allocation policies are very important.

If they are dynamic, the machine hasn’t got its resources actually allocated to it till the machine uses them (e.g. to serve a request). And after a while it doesn’t use these resources, they get released.

This dynamic allocation of resources might explain the latency you are observing.

If the resource allocation is static, then the latency due to the dynamic adjustments disappears.

Of course all of this is happening at virtualization level and this is why the Aolserver configuration parameters might be not so relevant.

Hope it helps,

Maurizio

From: Wolfgang Winkler [mailto:wolfgang...@digital-concepts.com]
Sent: 25 April 2013 08:17
To: aolserv...@lists.sourceforge.net
Subject: Re: [AOLSERVER] futex problems