[AOLSERVER] Error reading pool.tcl file

[Thorpe Mayes]

Hi,

This is in AOLserver 4.5.1.

When I start the server I get this error:

Error: tcl: source /usr/local/aolserver/modules/tcl/pools.tcl failed: maxconns cannot be less than 1

The maxconns in the pool file is listed as 0. 

I can change that and that error will go away, but I do not think that is what I should be doing. 

I did not have this issue in 4.5.0.

Thanks,

Thorpe

Here is my config file:

# $Header: /cvsroot/aolserver/aolserver/nsd/Attic/sample-config.tcl,v 1.2.4.1 2002/09/17 23:52:03 jgdavidson Exp $

#

# sample-config.tcl --  The AOLserver Startup Script

#

#      This is a Tcl script that is sourced when AOLserver starts up.

#      A detailed reference is in "doc/config.txt".

#

ns_log notice "nsd.tcl: starting to read config file for server0..."

#

# Set some Tcl variables that are commonly used throughout this file.

#

set httpport               80

set httpsport              8443

# The hostname and address should be set to actual values.

set hostname               [ns_info hostname]

set address                0.0.0.0

set host                   www.somedomain.com

set location               http://www.somedomain.com

set servername             "server0"

set serverdesc             "Server 0"

set homedir                /usr/local/aolserver

set bindir                 /usr/local/aolserver/bin

set pageroot               ${homedir}/servers/${servername}/pages 

set directoryfile          index.tcl,index.adp,index.html,index.htm

set ext .so

set serverlog              ${homedir}/servers/${servername}/server_log/server.log

set mailhost               mail.fotizo.net

set smtphost               mail.fotizo.net

#

# Global server parameters

#

ns_section "ns/parameters"

ns_param   home            $homedir

ns_param   debug           false

ns_param   serverlog       $serverlog

ns_param   user            nsadmin

ns_param   logroll         on

ns_param   logmaxbackup    10

#ns_param   mailhost        $the_mailhost  ;# do not set this parameter - it causes the server to die

ns_param   maxkeepalive    100; # http://aolserver.am.net/docs/tuning.adpx

ns_param   keepalivetimeout  30; # keepalivetimeout

#

# Thread library (nsthread) parameters

#

ns_section "ns/threads"

ns_param   mutexmeter      true      ;# measure lock contention

ns_param   stacksize [expr 1024*1024] ;# Per-thread stack size.

#

# MIME types

#

#  Note: AOLserver already has an exhaustive list of MIME types, but in

#  case something is missing you can add it here.

#

ns_section "ns/mimetypes"

ns_param   default         "*/*"     ;# MIME type for unknown extension.

ns_param   noextension     "*/*"     ;# MIME type for missing extension.

ns_param   ".xls"          "application/vnd.ms-excel"

ns_param   ".qtl"          "application/x-quicktimeplayer"

ns_param   ".gif"          "image/gif"

############################################################

#

# Server-level configuration

#

#  There is only one server in AOLserver, but this is helpful when multiple

#  servers share the same configuration file.  This file assumes that only

#  one server is in use so it is set at the top in the "server" Tcl variable.

#  Other host-specific values are set up above as Tcl variables, too.

#

ns_section "ns/servers"

ns_param   $servername     $serverdesc

#

ns_section "ns/server/${servername}"

ns_param   directoryfile   $directoryfile

ns_param   pageroot        $pageroot

ns_param   globalstats     false      ;# Enable built-in statistics.

ns_param   urlstats        false      ;# Enable URL statistics.

ns_param   maxurlstats     1000      ;# Max number of URL's to do stats on.

#

# Scaling and Tuning Options

#

#  Note: These values aren't necessarily the defaults.

#

ns_param   connsperthread  1         ;# Normally there's one conn per thread

ns_param   flushcontent    false     ;# Flush all data before returning

# ns_param   maxconnections  5       ;# Max connections to put on queue

ns_param   maxconns        0         ; # after the number of specified service connections, a connection thread will be destroyed - per http://aolserver.am.net/docs/tuning.adpx

ns_param   maxdropped      0         ;# Shut down if dropping too many conns

ns_param   maxthreads      10        ;# Tune this to scale your server - http://aolserver.am.net/docs/tuning.adpx - was 5

ns_param   minthreads      0         ;# Tune this to scale your server - http://aolserver.am.net/docs/tuning.adpx - was 5

ns_param   socktimeout     30

# ns_param   threadtimeout   120       ;# Idle threads die at this rate

ns_param   threadtimeout   1000000   ; # http://aolserver.am.net/docs/tuning.adpx

ns_param   maxkeepalive   100

ns_param   MaxWait         100       ; # http://aolserver.am.net/docs/tuning.adpx

ns_param   MaxWaitTime     60        ; # http://aolserver.am.net/docs/tuning.adpx

ns_param   MinBusyThreads   0        ; # http://aolserver.am.net/docs/tuning.adpx

ns_param   MaxBusyThreads   10       ; # http://aolserver.am.net/docs/tuning.adpx

# Directory listings -- use an ADP or a Tcl proc to generate them.

#ns_param   directoryadp    $pageroot/dirlist.adp ;# Choose one or the other.

ns_param   directoryproc   _ns_dirlist           ;#  ...but not both!

ns_param   directorylisting simple               ;# Can be simple or fancy.

# miscellaneous

ns_param   enabletclpages  true     ;# Parse *.tcl files in pageroot.

ns_param   enableaolpress  false

ns_param checkmodifiedsince true

ns_param flushcontent       false

ns_param noticedetail       true

ns_param headercase         preserve

ns_param outputcharset      UTF-8

ns_param sendfdthreshold    2048

ns_param errorminsize       514

ns_param maxline            16384

ns_param maxheaders         65536

ns_param maxpost            62914560

#

# Internal redirects

#

ns_section "ns/server/${servername}/redirects"

ns_param    404 /redirects/notfound.tcl

ns_param    500 /redirects/servererror.tcl

ns_param    403 /redirects/forbidden.tcl

#

# fastpath serves html

#

ns_section "ns/server/${servername}/fastpath"

ns_param     cache            10

ns_param     cachemaxentry    524288

ns_param     cachemaxsize     5242880

ns_param     mmap             false

ns_param     pageroot         $pageroot

ns_param     directorylisting fancy

ns_param     directoryproc    css_dirlist

ns_param     directoryfile    $directoryfile

ns_param     directoryadp     {}

#

# Tcl interpreter

#

ns_section "ns/server/${servername}/tcl"

ns_param    autoclose     true

ns_param    debug         false

ns_param    nsvbuckets    8

ns_param    statlevel     0

ns_param    statmaxbuf    100

ns_param    library       ${homedir}/servers/${servername}/modules/tcl

#

# ADP (AOLserver Dynamic Page) configuration

#

ns_section "ns/server/${servername}/adp"

ns_param   map             /*.adp  ;# Extensions to parse as ADP's.

ns_param   map             /*.html ;# Any extension can be mapped.

ns_param   enableexpire    false     ;# Set "Expires: now" on all ADP's.

ns_param   enabledebug     true     ;# Allow Tclpro debugging with "?debug".

ns_param   defaultparser   fancy

ns_param   cachesize       5242880

# do we need these?

ns_param   cache           on        ; # caching is enabled

# ADP special pages

ns_param   errorpage      ${pageroot}/errorpage.adp ;# ADP error page.

#

# ADP custom parsers -- see adp.c

#

ns_section "ns/server/${servername}/adp/parsers"

ns_param   adp       ".adp"    ;# adp is the default parser.

ns_param   fancy     .adp

#

# Modules to load

#

ns_section "ns/server/${servername}/modules"

ns_param   nssock          ${bindir}/nssock${ext}

ns_param   nslog           ${bindir}/nslog${ext}

ns_param   nsperm          ${bindir}/nsperm${ext} ;# Edit passwd before using.

# ns_param   nsxml           ${bindir}/nsxml${ext}   ; # old nsxml module

ns_param   nsdb            ${bindir}/nsdb${ext}

# ns_param php    ${bindir}/libphp5${ext}

# ns_param   nscache         ${bindir}/nscache${ext}   ; # ns_cache module

# PHP   

# ns_section "ns/server/${servername}/module/php"

# ns_param map *.php

   

#

# Socket driver module (HTTP)  -- nssock

#

ns_section "ns/server/${servername}/module/nssock"

ns_param   port            $httpport

ns_param   hostname        $host ; # was $hostname

ns_param   address         $address

ns_param   location        $location

ns_param   maxsock         100; # per http://aolserver.am.net/docs/tuning.adpx

ns_param   backlog         5; # http://aolserver.am.net/docs/tuning.adpx

ns_param   keepwait        30; # http://aolserver.am.net/docs/tuning.adpx

ns_param   socktimeout   30; # http://aolserver.am.net/docs/tuning.adpx

#

# Access log -- nslog

#

ns_section "ns/server/${servername}/module/nslog"

ns_param   file            ${homedir}/servers/${servername}/access.log

ns_param   enablehostnamelookup false

ns_param   formattedtime   true

ns_param   logcombined     true

ns_param   logrefer        true

ns_param   loguseragent    true

ns_param   rollday         *

ns_param   rollfmt         %Y-%m-%d-%H:%M

ns_param   rolllog         true      ;# Should we roll log?

ns_param   rollonsignal    true      ;# Roll log on SIGHUP.

ns_param   rollhour        0         ;# Time to roll log.

ns_param   maxbackup       10        ;# Max number to keep around when rolling.

#

# Database drivers

#

ns_section "ns/db/drivers"

ns_param    postgres     ${bindir}/nspostgres${ext}

#

# Database pools

#

ns_section "ns/db/pools"

ns_param    tmayes     "Pool for postgres database"

#

# database pool () configuration

#

ns_section "ns/db/pool/tmayes"

ns_param    driver              postgres

ns_param    connections         20

ns_param    datasource          localhost:5432:testdb

ns_param    user                nsadmin

ns_param    password            {}

ns_param    verbose             true

ns_param    logsqlerrors        true

ns_param    extendedtableinfo   true

ns_param    maxidle             1000000000

ns_param    maxopen             1000000000

#

# Database pools accessible by server

#

ns_section "ns/server/${servername}/db"

ns_param    pools     *

ns_param    defaultpool  "tmayes"

# set file max size

# ns_limits set default -maxupload [expr 15 * 1024 * 1024]  ;# Maximum file size set to 15Mb - default is 10Mb

ns_log notice "nsd.tcl: finished reading config file for server0)."

[Bjoern Kiesbye]

Hi Thorpe,

I think Aolserver 4.5.0 ignores this settings from config.tcl, and if
not set explicitly using ns_pools set command, Aolserver will use
deafult values.

I would suggest to in crease the maxconns value to at least 100, it
does not cost extra pre occupied resources, it just indicates how many
client connections (per thread) can wait for a thread to become
available again, before the Aolserver refuses to establish new client
connections.

This includes all requests including images, css/js files etc. not just
requests for tcl/adp pages.
If for example you have a Page showing 20 Images from fotoalbum, once
the html is loaded and the browser starts requesting the page elements
(20 images + 4 css/js files == 24 elements), with your current setting
maxthreads 10 and maxconns 1 ( 10 * 1 == 10 globalmaxconns) , this
will lead to 14 browser requests to be rejected by Aolserver (assuming
the Browser makes all requests at exactly the same time).
The product of maxthread and maxconns must be greater then the sum of
all simultaneous client connections, other wise some client requests
will be rejected by Aolserver. Increasing the maxthreads setting can
help too but each thread requires extra Memory (for a OpenAcs core only
install each thread requires about 40 MB).

Bjoern

-----Ursprüngliche Mitteilung-----
Von: Thorpe Mayes <tma...@ecognizant.net>
An: aolserver-talk <aolserv...@lists.sourceforge.net>
Verschickt: Fr, 27 Jul 2012 12:29 am
Betreff: [AOLSERVER] Error reading pool.tcl file

Hi,

Thanks,

Thorpe

set ext .so

#

-------------------------------------------------------------------------
-----Live Security Virtual ConferenceExclusive live event will cover
all the ways today's security and threat landscape has changed and how
IT managers can respond. Discussions will include endpoint security,
mobile security and the latest in malware threats.
http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/

_______________________________________________aolserver-talk mailing
listaolserver-talk@lists.sourceforge.nethttps://lists.sourceforge.net/lis
ts/listinfo/aolserver-talk

------------------------------------------------------------------------------
Live Security Virtual Conference
Exclusive live event will cover all the ways today's security and
threat landscape has changed and how IT managers can respond. Discussions
will include endpoint security, mobile security and the latest in malware
threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/
_______________________________________________
aolserver-talk mailing list
aolserv...@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/aolserver-talk

[Jeff Rogers]

Bjoern Kiesbye wrote:
> Hi Thorpe, I think Aolserver 4.5.0 ignores this settings from
> config.tcl, and if not set explicitly using ns_pools set command,
> Aolserver will use deafult values. I would suggest to in crease the
> maxconns value to at least 100, it does not cost extra pre occupied
> resources, it just indicates how many client connections (per thread)
> can wait for a thread to become available again, before the Aolserver
> refuses to establish new client connections. This includes all

Not quite - maxconns is the number of requests a thread will be allowed
to serve before it exits and a new one is spawned. Reusing an existing
thread is vastly more efficient than spawning a new one, but sometimes
resources can leak if a thread sticks around too long. If there's not
an available thread to handle a request, it gets queued.

Another reason to set maxconns higher is that if it is set too low, the
server can crash under load due to a race condition. I agree with 100
as a minimum; 1000 is quite reasonable.

-J

> requests including images, css/js files etc. not just requests for

> tcl/adp pages.If for example you have a Page showing 20 Images from

> fotoalbum, once the html is loaded and the browser starts requesting
> the page elements (20 images + 4 css/js files == 24 elements), with
> your current setting maxthreads 10 and maxconns 1 ( 10 * 1 == 10
> globalmaxconns) , this will lead to 14 browser requests to be
> rejected by Aolserver (assuming the Browser makes all requests at

> exactly the same time).The product of maxthread and maxconns must be

> greater then the sum of all simultaneous client connections, other
> wise some client requests will be rejected by Aolserver. Increasing
> the maxthreads setting can help too but each thread requires extra
> Memory (for a OpenAcs core only install each thread requires about 40
> MB).
>
> Bjoern

[Jim Davidson]

Howdy,

The leak is more precisely bloat, i.e., Tcl interps that create more and more objects and various per-thread thingers that grow in size and hang around, e.g., ADP caches and output buffers. These resources are somewhat reclaimed when a thread exists. But, assuming the normal "zippy" memory allocator is in use, the various small blocks of memory used by the allocator in the dying thread are flushed to a "shared" pool for other threads. The memory is not actually "returned" via a munmap, e.g.. Nate Folkman and I spent tons of time trying to figure this out years ago with all sorts of coalescing, unmapping, etc. fun but failed. It's possible other memory allocators have the same or better performance in speed and space today.

So, while it's very smart to have maxconns > 0 (and 100 to 1000 seems good), you will often still see continued bloat of the server size as reported by, e.g., top.


Anyway, what's the race condition? Curious about that one.

-Jim

[Thorpe Mayes]

Hi, 

I really appreciate your help with this. I got things working now. 

The interaction between these various variables re pools.tcl is interesting and not intuitive for me. I ended up leaving the code in pools.tcl alone and set the configuration file to: 

ns_section "ns/server/${servername}"

    ns_param directoryfile $directoryfile

    ns_param pageroot $pageroot

    ns_param maxthreads 10

    ns_param minthreads 0

    ns_param urlcharset "utf-8"

    ns_param outputcharset "utf-8"

    ns_param inputcharset "utf-8"

    ns_param enabletclpages true     ;# Parse *.tcl files in pageroot.

    # ns_param maxconns 4000

    ns_param maxconnections 4000 ; # controls maxconn in pools.tcl

    ns_param threadtimeout 1000000 ; # controls timeout in pools.tcl

Where did I come up those values? Various suggestions. I am certainly open to other values. 

As for the race condition, as Haig pointed out, there was a typo in the servererror.tcl file which caused the looping. My stupidity. 

Thanks,

Thorpe

Thorpe Mayes

tma...@ecognizant.com

405.514.9753

[Jeff Rogers]

Jim Davidson wrote:
>
> Howdy,
>
> The leak is more precisely bloat, i.e., Tcl interps that create more

> and more objects and [...] The memory is not

> actually "returned" via a munmap, e.g.. Nate Folkman and I spent
> tons of time trying to figure this out years ago with all sorts of
> coalescing, unmapping, etc. fun but failed. It's possible other
> memory allocators have the same or better performance in speed and
> space today.

Things may be a bit better today, as (I believe) the core tcl allocator
has a separate pool specifically for Tcl_Objs, which are a extremely
common small allocation.

I think the 'vtamalloc' allocator by Zoran Vasiljevec (?) was
specifically written to munmap/deallocate memory so that it could be
reclaimed by the system and keep the high-water mark down. It requires
you to build tcl with nonstandard defines tho, so you don't get the
standard tcl threaded allocator, which I think is a direct derivative of
zippy.

> Anyway, what's the race condition? Curious about that one.

I'm going off memory, because looking at the code it seems that it
shouldn't happen, but it was and my change fixed what I was seeing.

The problem is that a pthreads thread starts to run immediately upon
creation, and if maxconns was set too low then the conn thread could run
and exit before Ns_ThreadCreate (which is more or less a thin wrapper
around pthread_create) ran to completion, adding itself to the list of
threads to be reaped, and the next thread to be created would reap the
dead threads list but the thread id never got written in
Ns_ThreadCreate. I could only reproduce the error with maxconns less
than about 10, and running a benchmark like 'ab' with a fast request
like a fastpath file. pthread_join would be called with a null tid, and
the server would segfault.

I "fixed" this by passing the Ns_Thread* passed to Ns_ThreadCreate
directly through to pthread_create; on linux at least the tid is written
to the pthread_t in pthread_create before the new thread starts
running, but POSIX offers no such guarantee, so my "fix" might not work
on solaris or elsewhere (I only dug into so many library files).

I'm confusing myself now tho, because it certainly looks like the
threads are only ever reaped by the driver thread, which should
absolutely finish Ns_ThreadCreate before it can call it again. Conn
threads create a new thread to replace themselves when they hit
maxconns, but they don't reap at that time, so it should be ok.

Either way, I had easily reproducible segfaults, and tweaking the thread
code eliminated them, so I'm pretty sure I saw something real :)

-J

[Jeff Rogers]

Jeff Rogers wrote:

> I'm confusing myself now tho, because it certainly looks like the
> threads are only ever reaped by the driver thread, which should
> absolutely finish Ns_ThreadCreate before it can call it again. Conn
> threads create a new thread to replace themselves when they hit
> maxconns, but they don't reap at that time, so it should be ok.

Ok, I think I remembered how it happened.

D is DriverThread, which reaps when it creates conn threads
C is ConnThread, which upon exiting on hitting maxconns adds itself to
the reap list and creates a new conn thread without reaping.

"create XXX" is Ns_CreateThread, and actually starts the thread running
"finish creating" is when the tid is written to the ConnData. The gap
between these is the race.

D: create C1 ....
C1: run ... die ... add C1 to reap list ... create C2 ...
C2: run ... die ... add C2 to reap list ... create C3 ...
C1: finish creating C2 ... exit
D: finish creating C1 ... reap (C1, C2) *splat*

Of course D is queuing conns this whole time and doesn't create a thread
on every call to QueueConn, so there's a few steps left out here, but
I'm mostly sure this captures the scenario.

As I said before, my tweak to Ns_CreateThread ensures that the 'finish
creating' (tid written to ConnData) happens before the thread starts
running on Linux, but other implementations (even possibly a different
version of NPTL) might not get such a guarantee. And as with all race
conditions, scheduling matters, so you might never see this on a 4-core
box, for example.