[AOLSERVER] Diagnosing an AOLserver performance problem

[Janine Sisk]

I host a couple of sites based on an old version of OpenACS with a
proprietary (and fairly primitive) CMS. I didn't write any of it, and
up to now have not had to look at things very closely. But a
combination of increased traffic and more stuff running on the system
has made performance problems that were there all along rear some
really ugly heads, and I need to figure out what's going on.

From watching the output of top I can see that both Postgres and
AOLserver are doing their share of CPU hogging. I've just turned on
log_min_duration_statement in Postgres, to see where it's spending
most of its time. I'm less sure of what to do for AOLserver. I've
increased maxconnections and maxthreads and that seemed to help a
little, but not enough. It has been a long time since I last needed
to tune AOLserver and I'm a bit at a loss. Any suggestions on where
to start? I'd love to know what it's doing when it suddenly jumps up
to using 86% of the cpu, then drops back to almost nothing again.

BTW, this is AOLserver 4.0.10, with this patch applied:

http://sourceforge.net/tracker/index.php?func=detail&group_id=3152&atid=103152&aid=1615787

(it didn't seem to help) I can try upgrading if necessary but nothing
I've read has made me think that 4.5 will give me any better
performance, so for the moment I'm erring on the side of changing as
little as possible.

thanks,

janine

---
Janine Sisk
President/CEO of furfly, LLC
503-693-6407

--
AOLserver - http://www.aolserver.com/

To Remove yourself from this list, simply send an email to <list...@listserv.aol.com> with the
body of "SIGNOFF AOLSERVER" in the email message. You can leave the Subject: field of your email blank.

[Gustaf Neumann]

Janine,

my recommendation is to give aolserver 4.5.1 a try. We had a similar
observation
especially when many connection threads are configured. Depending on
your query-patterns, it might easily happen with earlier versions
that more or less all connection threads will be refreshed
(restarted) at the same time. The more threads are defined the
worse it will become. Aolserver 4.5.1 addresses this problem by
providing a randomization spread.

best regards
-gustaf neumann

Janine Sisk schrieb:

[Janine Sisk]

Ok... I just upgraded all of the sites on this system.

Things look a bit better, but I still occasionally see an nsd pop up
to 50%+ for a few seconds. It doesn't seem to last as long, but that
may just be because it's Sunday (and Mother's Day in the US), so
traffic is always going to be a bit lighter.

Is there anything else I can try?

janine

[Eduardo Santos]

Hi Janine,

I'm not an expert myself, but I've been doing some study in performance problems, as you can see here:

http://www.openacs.org/forums/message-view?message_id=1539800

The first thing to do is to find out exactly the performance problems you have, wich can usually be:

- cpu
- amount of memory
- memory bandwidth
- i/o
- DB

In this analysis, you have to be aware that AOLServer almost always consums all the available memory, depending on the values of maxthreads and minthreads. Making this assumption, try to discover what's your main concern and we can follow from here.

2009/5/10 Janine Sisk <jan...@furfly.net>

[Gustaf Neumann]

Janine,

the values make sense, since the 4.5.1 does not omit the cleanups
but instead of doing these at the same time, it distributes these
(by default by a randomization factor of 20%). The longer the
server runs the spikes should go down, since the cleanup
times will differ more.

Can you post the config file from that site?

[Janine Sisk]

On May 11, 2009, at 12:11 AM, Gustaf Neumann wrote:

> Janine,
>

> the values make sense, since the 4.5.1 does not omit the cleanups
> but instead of doing these at the same time, it distributes these
> (by default by a randomization factor of 20%). The longer the
> server runs the spikes should go down, since the cleanup
> times will differ more.
>
> Can you post the config file from that site?

Sure.

When I first started working on this problem, the settings were

ns_param maxconnections 5
ns_param maxthreads 5
ns_param minthreads 5

I changed minthreads to 0, which dramatically sped up server restart
time, and increased connections/threads first to 50/10 and then
100/20. I didn't see much of an increase in the last change so I left
it there.

I have turned on log_min_duration_statement in postgresql.conf and I
see in the PG log that there are many statements which are taking more
than the 1000ms threshold I set; I'm sure some of them could be
improved. But I'm not seeing PG take up a lot of CPU time, just nsd,
and I'm really not sure what it's doing.

Here's the config file:

ns_log notice "nsd.tcl: starting to read config file..."

######################################################################
#
# Instance-specific settings
# These default settings will only work in limited circumstances
# Two servers with default settings cannot run on the same host
#
######################################################################

#---------------------------------------------------------------------
# change to 80 and 443 for production use
set httpport 80
set httpsport 8443

# If setting port below 1024 with AOLServer 4, read daemontools/run

# The hostname and address should be set to actual values.
#set hostname [ns_info hostname]
set hostname big5.hrichina.org
#set address [ns_info address]
set address 209.162.194.91

# Note: If port is privileged (usually < 1024), OpenACS must be
# started by root, and, in AOLserver 4, the run script have a
# '-b address' flag which matches the address given above

set server "big5"
set servername "HRIC System Prototype"

set serverroot "/export/sites/${server}/live"

#---------------------------------------------------------------------
# which database do you want? postgres or oracle
set database postgres

set db_name $server

if { $database == "oracle" } {
set db_password "mysitepassword"
} else {
set db_host localhost
set db_port "5432"
set db_user hric
}

#---------------------------------------------------------------------
# if debug is false, all debugging will be turned off
set debug false

set homedir /usr/local/aolserver4.5.1
set bindir [file dirname [ns_info nsd]]

#---------------------------------------------------------------------
# which modules should be loaded? Missing modules break the server, so
# don't uncomment modules unless they have been installed.

ns_section ns/server/${server}/modules
ns_param nssock ${bindir}/nssock.so
ns_param nslog ${bindir}/nslog.so
ns_param nssha1 ${bindir}/nssha1.so
ns_param nscache ${bindir}/nscache.so

#nsrewrite is not used by any standard OpenACS code
#ns_param nsrewrite ${bindir}/nsrewrite.so

#---------------------------------------------------------------------
# nsopenssl will fail unless the cert files are present as specified
# later in this file, so it's disabled by default
#ns_param nsopenssl ${bindir}/nsopenssl.so

# Full Text Search
#ns_param nsfts ${bindir}/nsfts.so

# PAM authentication
#ns_param nspam ${bindir}/nspam.so

# LDAP authentication
#ns_param nsldap ${bindir}/nsldap.so

# These modules aren't used in standard OpenACS installs
#ns_param nsperm ${bindir}/nsperm.so
#ns_param nscgi ${bindir}/nscgi.so
#ns_param nsjava ${bindir}/libnsjava.so

if { [ns_info version] >= 4 } {
# Required for AOLserver 4.x
ns_param nsdb ${bindir}/nsdb.so
} else {
# Required for AOLserver 3.x
ns_param libtdom ${bindir}/libtdom.so
}

#---------------------------------------------------------------------
#
# Rollout email support
#
# These procs help manage differing email behavior on
# dev/staging/production.
#
#---------------------------------------------------------------------

ns_section ns/server/${server}/acs/acs-rollout-support

# EmailDeliveryMode can be:
# default: Email messages are sent in the usual manner.
# log: Email messages are written to the server's error log.
# redirect: Email messages are redirected to the addresses specified
# by the EmailRedirectTo parameter. If this list is absent
# or empty, email messages are written to the server's
error log.
# filter: Email messages are sent to in the usual manner if the
# recipient appears in the EmailAllow parameter, otherwise
they
# are logged.

#ns_param EmailDeliveryMode redirect
#ns_param EmailRedirectTo some...@yourdomain.test, othe...@yourdomain.test
#ns_param EmailAllow some...@yourdomain.test,othe...@yourdomain.test

######################################################################
#
# End of instance-specific settings
#
# Nothing below this point need be changed in a default install.
#
######################################################################

#---------------------------------------------------------------------
#
# AOLserver's directories. Autoconfigurable.
#
#---------------------------------------------------------------------

#---------------------------------------------------------------------
# Where are your pages going to live ?
#
set pageroot ${serverroot}/www
set directoryfile index.tcl,index.adp,index.html,index.htm

#---------------------------------------------------------------------
# Global server parameters
#---------------------------------------------------------------------

ns_section ns/parameters
ns_param serverlog /export/sites/${server}/logs/live-$
{server}.err
ns_param home $homedir
ns_param maxkeepalive 0
ns_param logroll on
ns_param maxbackup 5
ns_param debug $debug
#ns_param mailhost localhost

# Unicode by default:
# see http://dqd.com/~mayoff/encoding-doc.html
ns_param HackContentType 1
ns_param DefaultCharset utf-8
ns_param HttpOpenCharset utf-8
ns_param OutputCharset utf-8
ns_param URLCharset utf-8

#---------------------------------------------------------------------
# Thread library (nsthread) parameters
#---------------------------------------------------------------------

ns_section ns/threads
ns_param mutexmeter true ;# measure lock contention
# The per-thread stack size must be a multiple of 8k for AOLServer to
run under MacOS X
ns_param stacksize [expr 128 * 8192]

#
# MIME types.
#
# Note: AOLserver already has an exhaustive list of MIME types, but in
# case something is missing you can add it here.
#

ns_section ns/mimetypes
ns_param Default text/plain
ns_param NoExtension text/plain
ns_param .pcd image/x-photo-cd
ns_param .prc application/x-pilot
ns_param .xls application/vnd.ms-excel
ns_param .doc application/vnd.ms-word

#
# Tcl Configuration
#
ns_section ns/server/${server}/tcl
ns_param library ${serverroot}/tcl
ns_param autoclose on
ns_param debug $debug

#---------------------------------------------------------------------
#
# Server-level configuration
#
# There is only one server in AOLserver, but this is helpful when
multiple
# servers share the same configuration file. This file assumes that
only
# one server is in use so it is set at the top in the "server" Tcl
variable
# Other host-specific values are set up above as Tcl variables, too.
#
#---------------------------------------------------------------------

ns_section ns/servers
ns_param $server $servername

#
# Server parameters
#
ns_section ns/server/${server}
ns_param directoryfile $directoryfile
ns_param pageroot $pageroot
#ns_param maxconnections 5
#ns_param maxconnections 50
ns_param maxconnections 100
ns_param maxdropped 0
#ns_param maxthreads 5
#ns_param maxthreads 10
ns_param maxthreads 20
#ns_param minthreads 5
ns_param minthreads 0
ns_param threadtimeout 120
ns_param globalstats false ;# Enable built-in statistics
ns_param urlstats false ;# Enable URL statistics
ns_param maxurlstats 1000 ;# Max number of URL's to do
stats on
#ns_param directoryadp $pageroot/dirlist.adp ;# Choose one or
the other
#ns_param directoryproc _ns_dirlist ;# ...but not both!
#ns_param directorylisting fancy ;# Can be simple or
fancy

#
# Special HTTP pages
#

ns_param NotFoundResponse "/global/file-not-found.html"
ns_param ServerBusyResponse "/global/busy.html"
ns_param ServerInternalErrorResponse "/global/error.html"

#---------------------------------------------------------------------
#
# ADP (AOLserver Dynamic Page) configuration
#
#---------------------------------------------------------------------

ns_section ns/server/${server}/adp
ns_param map /*.adp ;# Extensions to parse as ADP's
#ns_param map "/*.html" ;# Any extension can be mapped
ns_param enableexpire false ;# Set "Expires: now" on all
ADP's
ns_param enabledebug $debug ;# Allow Tclpro debugging with
"?debug"
ns_param defaultparser fancy

ns_section ns/server/${server}/adp/parsers
ns_param fancy ".adp"

#---------------------------------------------------------------------
#
# Socket driver module (HTTP) -- nssock
#
#---------------------------------------------------------------------

ns_section ns/server/${server}/module/nssock
ns_param timeout 120
ns_param address $address
ns_param hostname $hostname
ns_param port $httpport
ns_param maxinput [expr 5 * 1024 * 1024] ;# Maximum File
Size for uploads in bytes
ns_param recvwait [expr 5 * 60] ;# Maximum request time in
minutes

#---------------------------------------------------------------------
#
# OpenSSL for Aolserver 3.3 and 4
#
#---------------------------------------------------------------------

if { [ns_info version] < 4} {

#---------------------------------------------------------------------
# OpenSSL for Aolserver 3.3

#---------------------------------------------------------------------

ns_section "ns/server/${server}/module/nsopenssl"

ns_param ModuleDir ${serverroot}/etc/certs

# NSD-driven connections:
ns_param ServerPort $httpsport
ns_param ServerHostname $hostname
ns_param ServerAddress $address
ns_param ServerCertFile certfile.pem
ns_param ServerKeyFile keyfile.pem
ns_param ServerProtocols "SSLv2, SSLv3, TLSv1"
ns_param ServerCipherSuite "ALL:!ADH:RC4+RSA:+HIGH:
+MEDIUM:+LOW:+SSLv2:+EXP"
ns_param ServerSessionCache true
ns_param ServerSessionCacheID 1
ns_param ServerSessionCacheSize 512
ns_param ServerSessionCacheTimeout 300
ns_param ServerPeerVerify false
ns_param ServerPeerVerifyDepth 3
ns_param ServerCADir ca
ns_param ServerCAFile ca.pem
ns_param ServerTrace false

# For listening and accepting SSL connections via Tcl/C API:
ns_param SockServerCertFile certfile.pem
ns_param SockServerKeyFile keyfile.pem
ns_param SockServerProtocols "SSLv2, SSLv3, TLSv1"
ns_param SockServerCipherSuite "ALL:!ADH:RC4+RSA:+HIGH:
+MEDIUM:+LOW:+SSLv2:+EXP"
ns_param SockServerSessionCache true
ns_param SockServerSessionCacheID 2
ns_param SockServerSessionCacheSize 512
ns_param SockServerSessionCacheTimeout 300
ns_param SockServerPeerVerify false
ns_param SockServerPeerVerifyDepth 3
ns_param SockServerCADir internal_ca
ns_param SockServerCAFile internal_ca.pem
ns_param SockServerTrace false

# Outgoing SSL connections
ns_param SockClientCertFile certfile.pem
ns_param SockClientKeyFile keyfile.pem
ns_param SockClientProtocols "SSLv2, SSLv3, TLSv1"
ns_param SockClientCipherSuite "ALL:!ADH:RC4+RSA:+HIGH:
+MEDIUM:+LOW:+SSLv2:+EXP"
ns_param SockClientSessionCache false
ns_param SockClientSessionCacheID 3
ns_param SockClientSessionCacheSize 512
ns_param SockClientSessionCacheTimeout 300
ns_param SockClientPeerVerify false
ns_param SockServerPeerVerifyDepth 3
ns_param SockClientCADir ca
ns_param SockClientCAFile ca.pem
ns_param SockClientTrace false

# OpenSSL library support:
#ns_param RandomFile /some/file
ns_param SeedBytes 1024
} else {

#---------------------------------------------------------------------
# OpenSSL for Aolserver 4
#---------------------------------------------------------------------

ns_section "ns/server/${server}/module/nsopenssl"

# We explicitly tell the server which SSL contexts to use as
defaults when an
# SSL context is not specified for a particular client or server
SSL
# connection. Driver connections do not use defaults; they must
be explicitly
# specificied in the driver section. The Tcl API will use the
defaults as there
# is currently no provision to specify which SSL context to use
for a
# particular connection via an ns_openssl Tcl command.

# Note this portion of the configuration is not perfect, and you
# will get errors in the your error.log. However, it does
# work. Fixes welcome.

# ---------------------------------------------------------
# this is used by acs-tcl/tcl/security-procs.tcl to get the
# https port.
# ---------------------------------------------------------
ns_param ServerPort $httpsport

ns_section "ns/server/${server}/module/nsopenssl/defaults"
ns_param server users
ns_param client client

ns_section "ns/server/${server}/module/nsopenssl/sslcontext/users"
ns_param Role server
ns_param ModuleDir ${serverroot}/etc/certs
ns_param CertFile certfile.pem
ns_param KeyFile keyfile.pem
#ns_param CADir ca-client/dir
#ns_param CAFile ca-client/ca-client.crt
ns_param Protocols "SSLv3, TLSv1"
ns_param CipherSuite "ALL:!ADH:RC4+RSA:+HIGH:+MEDIUM:
+LOW:+SSLv2:+EXP"
ns_param PeerVerify false
ns_param PeerVerifyDepth 3
ns_param Trace false

#ns_section "ns/server/${server}/module/nsopenssl/sslcontext/
admins"
#ns_param Role server
#ns_param ModuleDir /path/to/dir
#ns_param CertFile server/server.crt
#ns_param KeyFile server/server.key
#ns_param CADir ca-client/dir
#ns_param CAFile ca-client/ca-client.crt
#ns_param Protocols "All"
#ns_param CipherSuite "ALL:!ADH:RC4+RSA:+HIGH:+MEDIUM:
+LOW:+SSLv2:+EXP"
#ns_param PeerVerify false
#ns_param PeerVerifyDepth 3
#ns_param Trace false

ns_section "ns/server/${server}/module/nsopenssl/sslcontext/client"
ns_param Role client
ns_param ModuleDir ${serverroot}/etc/certs
ns_param CertFile certfile.pem
ns_param KeyFile keyfile.pem
#ns_param CADir ${serverroot}/etc/certs
#ns_param CAFile certfile.pem
ns_param Protocols "SSLv2, SSLv3, TLSv1"
ns_param CipherSuite "ALL:!ADH:RC4+RSA:+HIGH:+MEDIUM:
+LOW:+SSLv2:+EXP"
ns_param PeerVerify false
ns_param PeerVerifyDepth 3
ns_param Trace false

# SSL drivers. Each driver defines a port to listen on and an
explitictly named
# SSL context to associate with it. Note that you can now have
multiple driver
# connections within a single virtual server, which can be tied
to different
# SSL contexts. Isn't that cool?

ns_section "ns/server/${server}/module/nsopenssl/ssldrivers"
ns_param users "Driver for regular user access"
ns_param admins "Driver for administrator access"

ns_section "ns/server/${server}/module/nsopenssl/ssldriver/users"
ns_param sslcontext users
# ns_param port $httpsport_users
ns_param port $httpsport
ns_param hostname $hostname
ns_param address $address

ns_section "ns/server/${server}/module/nsopenssl/ssldriver/admins"
ns_param sslcontext admins
# ns_param port $httpsport_admins
ns_param port $httpsport
ns_param hostname $hostname
ns_param address $address

}

#---------------------------------------------------------------------
#
# Database drivers
# The database driver is specified here.
# Make sure you have the driver compiled and put it in {aolserverdir}/
bin
#
#---------------------------------------------------------------------

ns_section "ns/db/drivers"
if { $database == "oracle" } {
ns_param ora8 ${bindir}/ora8.so
} else {
ns_param postgres ${bindir}/nspostgres.so ;# Load
PostgreSQL driver
}

if { $database == "oracle" } {
ns_section "ns/db/driver/ora8"
ns_param maxStringLogLength -1
ns_param LobBufferSize 32768
}

#
# Database Pools: This is how AOLserver ``talks'' to the RDBMS. You
need
# three for OpenACS: main, log, subquery. Make sure to replace
``yourdb''
# and ``yourpassword'' with the actual values for your db name and the
# password for it, if needed.

# AOLserver can have different pools connecting to different databases
# and even different different database servers.
#
ns_section ns/db/pools
ns_param pool1 "Pool 1"
ns_param pool2 "Pool 2"
ns_param pool3 "Pool 3"

ns_section ns/db/pool/pool1
ns_param maxidle 1000000000
ns_param maxopen 1000000000
ns_param connections 300
ns_param verbose $debug
ns_param extendedtableinfo true
ns_param logsqlerrors $debug
if { $database == "oracle" } {
ns_param driver ora8
ns_param datasource {}
ns_param user $db_name
ns_param password $db_password
} else {
ns_param driver postgres
ns_param datasource ${db_host}:${db_port}:${db_name}
ns_param user $db_user
ns_param password ""
}

ns_section ns/db/pool/pool2
ns_param maxidle 1000000000
ns_param maxopen 1000000000
ns_param connections 300
ns_param verbose $debug
ns_param extendedtableinfo true
ns_param logsqlerrors $debug
if { $database == "oracle" } {
ns_param driver ora8
ns_param datasource {}
ns_param user $db_name
ns_param password $db_password
} else {
ns_param driver postgres
ns_param datasource ${db_host}:${db_port}:${db_name}
ns_param user $db_user
ns_param password ""
}

ns_section ns/db/pool/pool3
ns_param maxidle 1000000000
ns_param maxopen 1000000000
ns_param connections 300
ns_param verbose $debug
ns_param extendedtableinfo true
ns_param logsqlerrors $debug
if { $database == "oracle" } {
ns_param driver ora8
ns_param datasource {}
ns_param user $db_name
ns_param password $db_password
} else {
ns_param driver postgres
ns_param datasource ${db_host}:${db_port}:${db_name}
ns_param user $db_user
ns_param password ""
}

ns_section ns/db/pool/old
ns_param maxidle 1000000000
ns_param maxopen 1000000000
ns_param connections 5
ns_param verbose $debug
ns_param extendedtableinfo true

ns_param logsqlerrors $debug
ns_param driver postgres
ns_param datasource localhost::hric
ns_param user nsadmin
ns_param password ""

ns_section ns/server/${server}/db
ns_param pools "pool1,pool2,pool3,old"
ns_param defaultpool pool1

ns_section ns/server/${server}/acs/database
ns_param database_names {default old}
ns_param pools_default {pool1 pool2 pool3}
ns_param pools_old {old}

ns_section ns/server/${server}/redirects
ns_param 404 "global/file-not-found.html"
ns_param 403 "global/forbidden.html"

#---------------------------------------------------------------------
#
# Access log -- nslog
#
#---------------------------------------------------------------------

ns_section ns/server/${server}/module/nslog
ns_param debug false
ns_param dev false
ns_param enablehostnamelookup false
ns_param file /export/sites/${server}/logs/live-$
{server}.log
ns_param logcombined true
ns_param extendedheaders COOKIE
#ns_param logrefer false
#ns_param loguseragent false
ns_param maxbackup 1000
ns_param rollday *
ns_param rollfmt %Y-%m-%d-%H:%M
ns_param rollhour 0
ns_param rollonsignal true
ns_param rolllog true

#---------------------------------------------------------------------
#
# nsjava - aolserver module that embeds a java virtual machine.
Needed to
# support webmail. See http://nsjava.sourceforge.net for
further
# details. This may need to be updated for OpenACS4 webmail
#
#---------------------------------------------------------------------

ns_section ns/server/${server}/module/nsjava
ns_param enablejava off ;# Set to on to enable nsjava.
ns_param verbosejvm off ;# Same as command line -debug.
ns_param loglevel Notice
ns_param destroyjvm off ;# Destroy jvm on shutdown.
ns_param disablejitcompiler off
ns_param classpath /usr/local/jdk/jdk118_v1/lib/classes.zip:
${bindir}/nsjava.jar:${pageroot}/webmail/java/activation.jar:$
{pageroot}/webmail/java/mail.jar:${pageroot}/webmail/java

#---------------------------------------------------------------------
#
# CGI interface -- nscgi, if you have legacy stuff. Tcl or ADP files
inside
# AOLserver are vastly superior to CGIs. I haven't tested these params
but they
# should be right.
#
#---------------------------------------------------------------------

#ns_section "ns/server/${server}/module/nscgi"
# ns_param map "GET /cgi-bin/ ${serverroot}/cgi-bin"
# ns_param map "POST /cgi-bin/ ${serverroot}/cgi-bin"
# ns_param Interps CGIinterps

#ns_section "ns/interps/CGIinterps"
# ns_param .pl "/usr/bin/perl"

#---------------------------------------------------------------------
#
# PAM authentication
#
#---------------------------------------------------------------------

ns_section ns/server/${server}/module/nspam
ns_param PamDomain "pam_domain"

#---------------------------------------------------------------------
#
# WebDAV Support (optional, requires oacs-dav package to be installed
#
#---------------------------------------------------------------------

ns_section ns/server/${server}/tdav
ns_param propdir ${serverroot}/data/dav/properties
ns_param lockdir ${serverroot}/data/dav/locks
ns_param defaultlocktimeout "300"

ns_section ns/server/${server}/tdav/shares
ns_param share1 "OpenACS"
#ns_param share2 "Share 2 description"

ns_section ns/server/${server}/tdav/share/share1
ns_param uri "/*"
# all WebDAV options
ns_param options "OPTIONS COPY GET PUT MOVE DELETE HEAD MKCOL POST
PROPFIND PROPPATCH LOCK UNLOCK"

#ns_section ns/server/${server}/tdav/share/share2
#ns_param uri "/share2/path/*"
# read-only WebDAV options
#ns_param options "OPTIONS COPY GET HEAD MKCOL POST PROPFIND PROPPATCH"

ns_log notice "nsd.tcl: finished reading config file."

[Alexey Pechnikov]

Hello!

I think you can try these:

1. Disable https (HAProxy+stunnel is better).
2. Disable fancy parser if simple parser is enough.
3. The database pool of 300 connections to Oracle or PostgreSQL is
huge. 25 or may be 50 is more reason.
4. Disable logs (may be replaced to HAProxy logging).
5. Set stacksize to [expr 256*1024] if this enough.
6. Disable debug.

And of course you may enable statistics. There are a lot of helpful
information in AOL stats.

Best regards, Alexey Pechnikov.
http://pechnikov.tel/

[Gustaf Neumann]

Janine Sisk schrieb:

>
> When I first started working on this problem, the settings were
>
> ns_param maxconnections 5
> ns_param maxthreads 5
> ns_param minthreads 5

maxconnections of 5 is for most applications to small. The term
"maxconnections" is misleading since it does not mean "maximum
number of incoming connections", but is a counter per thread, when
it will shut down. With the value above, every thread will exit
and be recreated after 5 requests. If the application is well written,
no garbage should be left after the request, therefore the value can
be set to e.g. 1000.

>
> I changed minthreads to 0, which dramatically sped up server restart
> time, and increased connections/threads first to 50/10 and then
> 100/20. I didn't see much of an increase in the last change so I left
> it there.

when start with minthreads 5 takes long, you seem to have quite a slow
machine.
Do you have some usage figures, how many concurrent users you have to serve?
By concurrent, i mean running requests at the same time.

You can as well increase the threadtimeout to reduce the number of
thread restarts

....
ns_param threadtimeout 120
....

regarding the pools:

I would recommend to comment out the "maxidle" and "maxopen"
values (were needed for old aolserver versions) and - more important -
to reduce the number of connections. When you have maxthreads
at a low value (e.g. 20), you should not need more than e.g. 20
connections for the first two pools.

>
> ns_section ns/db/pool/pool1
> ns_param maxidle 1000000000
> ns_param maxopen 1000000000
> ns_param connections 300
> ns_param verbose $debug

maybe the 600+ connections you have makes things slower than
needed.

best regards
-gustaf neumann

[Janine Sisk]

On May 12, 2009, at 1:23 PM, Gustaf Neumann wrote:

> Janine Sisk schrieb:
>>
>> When I first started working on this problem, the settings were
>>
>> ns_param maxconnections 5
>> ns_param maxthreads 5
>> ns_param minthreads 5
> maxconnections of 5 is for most applications to small. The term
> "maxconnections" is misleading since it does not mean "maximum
> number of incoming connections", but is a counter per thread, when
> it will shut down. With the value above, every thread will exit
> and be recreated after 5 requests. If the application is well written,
> no garbage should be left after the request, therefore the value can
> be set to e.g. 1000.

Interesting - I did not know that. I don't have much faith in this
application to be well written so I haven't made this change yet, but
I will try it next.

>> I changed minthreads to 0, which dramatically sped up server
>> restart time, and increased connections/threads first to 50/10 and
>> then 100/20. I didn't see much of an increase in the last change
>> so I left it there.
> when start with minthreads 5 takes long, you seem to have quite a
> slow machine.
> Do you have some usage figures, how many concurrent users you have
> to serve?
> By concurrent, i mean running requests at the same time.

Unfortunately I have no idea; they process their own stats and so far
have not shared (I have asked but they keep forgetting).

The system is not the best, but it shouldn't be this bad. I've
watched the error log as the site comes up and there is a lot of added
"stuff" going on in the CMS, looks like it's loading in something from
the database. It shouldn't be this slow, but I'm leaving delving into
the code as a last resort because it looks like quicksand.

> You can as well increase the threadtimeout to reduce the number of
> thread restarts
>
> ....
> ns_param threadtimeout 120

I commented out the maxopen/maxidle and dropped the connections to 20
each. I didn't notice the 300 - that was clearly out of whack.

With just those changes I'm still seeing the nsd process consume 50%+
of the CPU from time to time but it looks like it's happening for a
shorter period of time most of the time, so that is at least an
improvement.

Thanks!

janine

> ....
>
> regarding the pools:
>
> I would recommend to comment out the "maxidle" and "maxopen"
> values (were needed for old aolserver versions) and - more important -
> to reduce the number of connections. When you have maxthreads
> at a low value (e.g. 20), you should not need more than e.g. 20
> connections for the first two pools.
>>
>> ns_section ns/db/pool/pool1
>> ns_param maxidle 1000000000
>> ns_param maxopen 1000000000
>> ns_param connections 300
>> ns_param verbose $debug
> maybe the 600+ connections you have makes things slower than
> needed.
>
> best regards
> -gustaf neumann
>
>
> --
> AOLserver - http://www.aolserver.com/
>
> To Remove yourself from this list, simply send an email to <list...@listserv.aol.com
> > with the
> body of "SIGNOFF AOLSERVER" in the email message. You can leave the
> Subject: field of your email blank.

---

Janine Sisk
President/CEO of furfly, LLC
503-693-6407

[Tom Jackson]

On Tue, 2009-05-12 at 22:23 +0200, Gustaf Neumann wrote:
> Janine Sisk schrieb:
> >
> > When I first started working on this problem, the settings were
> >
> > ns_param maxconnections 5
> > ns_param maxthreads 5
> > ns_param minthreads 5
> maxconnections of 5 is for most applications to small. The term
> "maxconnections" is misleading since it does not mean "maximum
> number of incoming connections", but is a counter per thread, when
> it will shut down. With the value above, every thread will exit
> and be recreated after 5 requests. If the application is well written,
> no garbage should be left after the request, therefore the value can
> be set to e.g. 1000.

This sounds like great general advice. If I had to boil down a few
rules, I would say at bare minimum:

maxconnections > 5 * maxthreads.

But 10 to 20 times would be much better. Also, unless you have a memory
leaking application, start with a minimum of 100 for maxconnections.

You should also be using threadpools. If you don't, then every request
gets dumped into a default threadpool that is shared by all virtual
servers. What this means is that it is possible for multiple interps to
exist in each thread, up to one for each virtual server. At a bare
minimum, each virtual server should have its own default threadpool
which matches every request to that vs.

So the problem with one nsd-process-wide threadpool is the memory
footprint of each thread will be at least the sum of each vs interp.

Maybe I should point out that one advantage of threads is to reduce the
memory footprint by specializing the interp code.

If you setup a number of threadpools, you can partition your requests
into static and dynamic (at the very least). A few threads can handle
many static requests. You should also partition your requests by need
run tcl code/filters.

Do you want to require filter processing for css and basic image files?
Personally I would use a different port for pure static content. I have
used publicfile: http://cr.yp.to/publicfile.html but I have heard that
AOLserver is pretty fast at static file handling (I'm trying to catch
up).

tom jackson

[Janine Sisk]

Are you referring to Rob Mayoff's threadpool code from 2004? Does it
still work in 4.5.1?

I'm trying to avoid modifying the code as much as possible; I may get
to that point, but I'm trying to exhaust my tuning options first. I
have a feeling that once I crack open the cover I'm going to be stuck
in buggy quicksand and I'd rather not go there if I don't have to,
despite how financially rewarding it would be. :)

janine

---

Janine Sisk
President/CEO of furfly, LLC
503-693-6407

[Tom Jackson]

On Wed, 2009-05-13 at 09:54 -0700, Janine Sisk wrote:
> Are you referring to Rob Mayoff's threadpool code from 2004? Does it
> still work in 4.5.1?
>

No AOLserver has url based, connection threadpools built in. But by
default they don't get configured very well. Rob's code was more like a
worker thread pool, something like warmed up, non-connection threads.

There isn't any need to modify any code, just configure them. Use nscp
and telnet in and check out the ns_pools command.

[Gustaf Neumann]

Janine Sisk wrote:
>
> Unfortunately I have no idea; they process their own stats and so far
> have not shared (I have asked but they keep forgetting).

...

>
> With just those changes I'm still seeing the nsd process consume 50%+
> of the CPU from time to time but it looks like it's happening for a
> shorter period of time most of the time, so that is at least an
> improvement.

From my experience, you can obtain much more improvement by
looking for bottlenecks in the application than by tuning a few
parameter in the aolserver configuration. So far, i did not see any
case of "slow" behavior like you are describing attributable to aolserver.
Your problem is most likely more an "application performance problem"
than a "aolserver performance problem".
If you have an application running since several years, it is
not unlikely that some tables have grown quite large, such
that missing indices and sequentual scans might have become
a problem over the years. Furthermore, many OpenACS user don't
care much about the performance of scheduled jobs, but these can lead
exactly to the behavior you are observing.

I would recommend to monitor the behavior more closely,
by using e.g. the xotcl request monitor and the aolserver statistics

http://aolserver.cvs.sourceforge.net/viewvc/aolserver/aolserver/examples/config/stats.tcl?view=markup

Since you seem to run quite an old version of OpenACS:
Another option, i would consider is to upgrade OpenACS.
We fixed many performance problems over the last years
in the base framework.

[Janine Sisk]

I think you are right, Gustaf - I've given my client a choice of which
way to go from here, one of which is to start fixing the code.

janine

---

Janine Sisk
President/CEO of furfly, LLC
503-693-6407

[aT]

> maxconnections of 5 is for most applications to small. The term
> "maxconnections" is misleading since it does not mean "maximum
> number of incoming connections", but is a counter per thread, when
> it will shut down. With the value above, every thread will exit
> and be recreated after 5 requests. If the application is well written,
> no garbage should be left after the request, therefore the value can
> be set to e.g. 1000.

This statement changes my understanding of max connection , I use to
think that Max connection defines the maximum number of connections any
aolserver instance can receive and any further connection will receive
"Server Busy " .

Why the thread has to shutdown ?

Does this mean if i have Max thread 20 and Max connection 100 ,
Aolserver will queue 1980 connections as any given time only 20
connections can be served .
When any given thread has served 100 connections it will shutdown and
aolserver will respawn another thread to serve the pending queue ? is my
understand correct>?

Once the thread has served a connection , what state it moves to so that
aolserver can pass more connection requests to it ?

Thanks & Regards

--
Syed Atif Ali
D. +971 4 3911914
F. +971 4 3911915
___________________________________________
The world is coming to an end! Repent and return those library books!