ANN: AOLserver 4.5.0 released!
Dossy
June 27, 2006
========================================
On behalf of the AOLserver Team, I have the honor of announcing the
latest major release of AOLserver: 4.5.0.
AOLserver 4.5.0 is a major upgrade including several new Tcl commands,
C API's, and resource management improvements.
A summary of changes is provided at the end of this document.
WHAT IS AOLSERVER?
AOLserver is America Online's Open-Source web server. AOLserver is
the backbone of the largest and busiest production environments in
the world. AOLserver is a multithreaded, Tcl-enabled web server
used for large scale, dynamic web sites.
Visit the project's website:
HOW CAN I GET AOLSERVER?
Download the source code from SourceForge:
http://aolserver.com/downloads/aolserver-4.5.0-src.tar.gz
I FOUND A BUG! WHAT DO I DO?
File it in the Bug Tracker at SourceForge:
IS THERE A MAILING LIST? WHERE'S THE USER COMMUNITY?
Yes! There's an announcements-only and a general discussion mailing
list. Instructions on how to subscribe are here:
http://aolserver.com/lists.php
There is also a wiki-web set up for AOLserver:
CHANGES SINCE LAST RELEASE?
AOLserver 4.5 - June 2006
-------------------------
AOLserver 4.5 is a major upgrade including several new Tcl commands,
C API's, and resource management improvements.
Contents:
--------
New and Updated Binaries
New Modules
New and Updated Tcl Commands
New and Updated C Routines
Additional Updates
Changes to the Build Process
Bug Fixes
Known Issues
New and Updated Binaries:
------------------------
libnsd:
The AOLserver library now includes an entry point suitable
for loading into an ordinary, thread-enabled, tclsh, e.g.,:
# tclsh
% load /usr/local/aolserver/lib/libnsd.so
% ns_time
This provides the full set of Tcl commands within the
AOLserver core although several commands will generate
errors for lack of an HTTP request context (e.g., ns_conn).
nstclsh:
A simple Tcl-shell linked against libnsd. It's equivalent
to loading libnsd into an ordinary tclsh as described above.
New Modules:
-----------
nszlib:
The popular nszlib module written by Vlad Seryakov has been
integrated into the core release with modifications to
enable the new Ns_Gzip routine. The module also provides
the "ns_zlib" command for compressing and uncompressing
strings and files.
nsproxy:
The nsproxy modules provides the "ns_proxy" command which
enables sending Tcl scripts to a proxy process connected
via a pipe. The proxy includes core Tcl commands as well
as AOLserver commands from the libnsd library. Evaluating
scripts in a proxy process can be used to isolate and/or
timeout thread-unsafe or otherwise unsafe 3rd party code.
New and Updated Tcl Commands:
----------------------------
The following Tcl commands are new or updated from the last release.
As few of the man pages have yet to be written, see the source code
for usage details and options.
ns_pools:
The "ns_pools" command enables configuration of one or more
pools of connection processing threads. The pools allow
certain requests to be handled by specific threads. This
could, for example, ensure multiple long running requests
don't block other short running requests. Pools are selected
based on method/url pairs similar to the mappings managed
by the "ns_register_proc" command. By default, all requests
are handled by a single, unlimited, "default" pool. There
is also an "error" pool as described below. Coupled with
the new "ns_limits" command, pools can provide for sophisticated
resource management.
ns_limit:
The "ns_limit" command enables setting various resource
limits for specified method/url combinations. These limits
include such items as max concurrent connections, max file
upload size, and timeouts waiting for connection processing.
When limits are exceeded, connections are immediately
dispatched to a dedicated "error" connection processing
pool to generate a quick error response. By default all
requests share the same default limits. Coupled with the
new "ns_pools" command, URL-based limits can provide for
sophisticated resource management.
ns_register_fastpath:
The new "ns_register_fastpath" command enables mapping of
specific directories to the internal static file response
code. See also the new Ns_FastPathOp API.
ns_adp_flush:
The "ns_adp_flush" command allows buffered ADP output to
be immediately flushed to the client. If no output has been
sent, the server will first generate the appropriate HTTP
response headers for a streaming response. An implicit
flush will also be performed when the ADP buffer overflows.
ns_adp_close:
The "ns_adp_close" command flushes content as with ns_adp_flush
and then closes the connection so no additional content can
be sent. This can be useful to keep the client from waiting
while post-processing steps continue.
ns_adp_registerscript
The "ns_adp_registerscript" command is similar to the
ns_adp_regsiteradp command except it registers an ordinary
script instead of text/script ADP code to execute for a
given tag.
ns_adp_ictl:
The new "ns_adp_ictl" command provides various options to
manage the ADP execution environment for current Tcl
interpreter, e.g.:
ns_adp_ctl bufsize ?size?; # Max output before forced flush
ns_adp_ctl autoabort ?bool?; # Abort execution on client drop?
ns_adp_ctl singlescript ?bool?; # Compile pages as a single script?
ns_adp_ctl gzip ?bool?; # Gzip output if enabled in server?
ns_adp_ctl trimspace ?bool?; # Trim leading whitespace in response?
ns_adp_ctl channel ?chan?; # Set channel for ADP output
ns_adp_ctl trace ?bool?; # Trace ADP execution
ns_adp_ctl detailerror ?bool?; # Generate detailed error messages
ns_adp_ctl displayerror ?bool?; # Format error message in page output
ns_adp_ctl stricterror ?bool?; # Abort ADP execution on flush error
ns_adp_include:
The "ns_adp_include" command has been enhanced with the new
"-cache ttl" and "-nocache" options to support execution
results caching. Applications which isolate fully dynamic
page components (e.g., personalized content) from partially
dynamic page components (e.g., slowly changing news headlines)
can benefit from using these options to cache and reuse the
results of slowly changing components. This approach has
the potential to dramatically improve the performance of
such applications.
ns_adp_ident:
The new "ns_adp_ident" command can be used to tag and query
ADP file versions.
ns_cache:
The "ns_cache" command enables creation and management of
memory-based caches utilizing the underlying Ns_Cache C
API's.
ns_driver
The new "ns_driver" command provides a "list" option to
enumerate all driver instances and a "query" option to
return a Tcl list of various statistics for a given driver.
ns_ictl:
The "ns_ictl" command has been enhanced with several new
options including:
trace:
Enables Tcl-script based callbacks at points defined
by the Ns_TclRegisterTrace routine.
threads:
Returns a list of all threads with Tcl interps for
current virtual server.
cancel:
Sends an async-cancel message to a given thread,
interrupting any Tcl execution in any AOLserver-based
interpreter executing on the thread.
package:
Registers a Tcl package to be automatically required
in each interp for the current virtual server.
once:
Registers a script to be executed exactly once for
a virtual server.
ns_internalredirect:
The "ns_internalredirect" command enables re-authorization
and restart of a connection as if it where originally for
a different URL. This command provides access to the
previously private routines used internally for such purposes,
e.g., by the fastpath to handle directory URL's mapped to
an ADP index files such as "index.adp".
ns_for:
ns_foreach:
ns_while:
These commands are replacements for the standard "for",
"foreach", and "while" command which have the same semantics
but also allow for monitoring and control by the "ns_loop_ctl"
command described below.
ns_loop_ctl:
The "ns_loop_ctl" command enables listing, pause, resume,
cancel, and evaluation of a script within "ns_for",
"ns_foreach", and "ns_while" loops executing in other
threads. In addition, the "install" option can be used at
to swap the "ns_for", "ns_foreach", and "ns_while" commands for
the "for", "foreach", and "while" commands, enabling monitoring
of all loops in unmodified code.
ns_register_encoding
The new "ns_register_encoding" command enables mapping of
method/URL combinations to specific charset encodings used
to decode the request. If no mapping exists for a given
method/URL, the server will use that defined for the virtual
server or process wide by the "urlcharset" config setting.
This setting defaults to NULL, or no encoding, thus assuming
UTF-8 characters in the request data.
ns_returnmoved:
The "ns_returnmoved" command is similar to the "ns_returnredirect"
command but returns a 301 "permanently moved" HTTP status
code along with the redirected location.
New and Updated C Routines:
--------------------------
The following routines are new or updated from the last release.
As few of the man pages have yet to be written, see the source code
for parameter and results details.
Ns_BinPath:
New routine to construct file pathnames relative to the
bin/ subdirectory similar to the Ns_LibPath and Ns_HomePath
routines.
Ns_ConnContent:
Return a pointer to request content. If the content is not
in memory and has been spooled to a temp file, a new file
mapping will be created and returned. Note that the content
is likely, but not guaranteed, to be followed by a terminating
null byte. See also the "content" option of the "ns_conn"
command.
Ns_ConnContentFd:
Return a file descriptor open on a temp file containing
request content. If the content is in memory only, it will
first be copied to a newly allocated temp file. You must
not close the returned fd as it is owned by the server core
which will reuse the open file for subsequent requests.
For an example, see nscgi/nscgi.c which has been updated
to use Ns_ConnContentFd instead of spooling to a temp file
directly. See also the "contentchannel" option to the
"ns_conn" command which returns a duplicate of the fd
returned via Ns_ConnContentFd registered as a Tcl channel.
Ns_ConnContentOnDisk:
Returns 1 if the request content is already located in a
temp file or 0 if the content is located in memory. A call
to Ns_ConnContent after Ns_ConnContentOnDisk has returned
1 will result in a new file memory mapping being created.
Ns_ConnGetFile:
Ns_ConnFirstFile:
Ns_ConnNextFile:
Routines to find a specific file or enumerate all file
objects embedded within a multipart form. See also the
"files" option to the "ns_conn" command.
Ns_ConnSendFdEx:
Send contents of an open file descriptor from a specific
file offset. This routine uses pread on Unix to enable
multiple simultaneous calls with the same open fd and is
currently unsupported on Win32.
Ns_ConnReturnOpenFdEx:
Generate a complete response with content from open file
descriptor at given offset. This routine uses Ns_ConnSendFdEx
and as such is also unsupported on Win32.
Ns_ConnReturnServiceUnavailable:
New routine to return an HTTP 503 response.
Ns_CheckStack:
Checks if the stack appears to have overrun. Note that the
ADP evaluation engine now calls Ns_CheckStack on entry to
each call frame and will generate an exception in the case
of an endlessly recursive ADP condition.
Ns_ClsAlloc:
Ns_ClsSet:
Ns_ClsGet:
New interface, similar to thread-local storage, which enables
user data to be saved and retrieved within the connection
structure. This interface is intended to be used with the
Ns_QueueWait interface so that data can be saved in the
driver thread and retrieved later in a connection processing
thread.
Ns_ConnSetStatus:
Ns_ConnGetStatus:
New routines to set and get the HTTP status code to be used
in responses generated via Ns_ConnFlush. See also the
"status" option to the "ns_conn" command.
Ns_ConnSetType:
Enhanced to more intelligently set the input and output
character encodings based on intended output charset. See
also the "ns_adp_mimetype" command and the "encoding" and
"urlencoding" options to the "ns_conn" command.
Ns_ConnGetGzipFlag:
Ns_ConnSetGzipFlag:
Ns_ConnGetKeepAliveFlag:
Ns_ConnSetKeepAliveFlag:
New utility routines to query or set the gzip and connection
keep-alive advisory flags.
Ns_ConnFlush:
New general purpose routine to construct and return dynamically
generated, text-based responses, e.g., as generated from
ADP. The routine incorporates much of the logic previously
embedded in the ADP interface including the ability to
automatically covert from UTF-8 to the output encoding
and/or gzip-compress the output. In addition, the routine
supports streaming responses, generating headers on the
first call followed by HTTP/1.1-style chunked-encoding
blocks on subsequent calls. See also the "ns_adp_flush" and
"ns_adp_close" commands.
Ns_ConnFlushDirect:
Similar to Ns_ConnFlush but without any modification of the
given buffer, thus useful for non-text content.
Ns_DevNull:
Returns a shared file descriptor open on /dev/null. This
fd is owned by the AOLserver process and must not be closed.
Ns_FastPathOp:
New request callback to utilize the internal "fastpath"
caching-file response code. See also the new "ns_register_fastpath"
command.
Ns_AdpRequestEx:
Extended version of Ns_AdpRequest with an additional Ns_Time
argument, specifying the timeout of optional execution
results caching.
Ns_FindCharset
Locates the charset string within a full mimetype string,
e.g., locating "utf-8" in "text/html; charset=utf-8".
Ns_Gzip:
Ns_SetGzipProc:
Routine to gzip-compress a buffer if a compression function
has been registered with a call to Ns_SetGzipProc. The new
"nszlib" module registers such a callback.
Ns_GetAllAddrByHost:
New routine to return all addresses for a given host. See
also the "ns_addrbyhost -all" command.
Ns_ParseRequestEx:
New extended version of Ns_ParseRequest which takes an
optional Tcl_Encoding argument specifying the charset to
decode the URL. The default is based on the context of the
request.
Ns_TaskCreate:
Ns_TaskEnqueue:
Ns_TaskRun:
Ns_TaskCancel:
Ns_TaskWait:
Ns_TaskCallback:
Ns_TaskDone:
Ns_TaskFree:
Ns_CreateTaskQueue:
Ns_DestroyTaskQueue:
New facility for event-based callback queues designed to
replace the legacy Ns_SockCallback interface. As an example,
see the "ns_http" command which has been updated to use the
task interface.
Ns_TclRegisterTrace:
New routine to registere callbacks at specific points in
the lifetime of a Tcl interpreter. This routine is a
replacement for the legacy Ns_TclRegisterAt* routines. See
also the "trace", "runtraces", and "gettraces", options to
the "ns_ictl" command.
Ns_QueueWait:
New interface to enable event-driven callbacks in the driver
thread before dispatching to pools for processing. This
allows drivers to augment data received from the client
(headers, request, content) with additional data fetched
over the network (likely stored in connection-local storage
via Ns_Cls). An example would be to add certain personalization
data received via a Web service. The benefit of this approach
is to accumulate additional data efficiently with driver-thread
based event callbacks instead of through potentially blocking
calls within connection threads.
Ns_SockWaitEx:
New extended version of Ns_SockWait using higher-resolution
millisecond timeouts instead of 1-second resolution.
ns_poll:
New wrapper around system-provided poll or an emulation
based on select when poll isn't available (Win32) or broken
(OS/X).
Ns_LibInit:
NsThreads_LibInit:
New routines to initialize the libnsd and libnsthread
libraries. It is normally not necessary to call these
routines as the dynamic library entry points as well as the
Ns_Main and Ns_TclInit routines call these init routines
at startup. They are available if necessary to initialize
specialized, statically linked applications.
Additional Updates:
------------------
* Enabled a global, pseudo-server, to support all Tcl commands
outside the context of a virtual server, e.g., when the libnsd.so
library is loaded into a Tcl shell.
* Updated Tcl trace execution order to be FIFO and LIFO for init
and cleanup callbacks respectively.
* Moved more of the legacy (and generally confusing) Tcl interp
duplication code from C into the the bootstrap init.tcl script
utilizing the ns_ictl command.
* ADP output buffer is now configurable with a maximum size (see
the "ns_adp_ctl" command above). Upon overflow, the current output
will be flushed to the client immediately in streaming mode to avoid
consuming unbounded memory.
* Restored AOLserver 3.x ADP parsing order, based on a patch from
Alastair Young, where registered tags, <% ... %> scripts, and
<script> ... </script> tags all have the same precedence. The ADP
parser now also properly handles nested sequences, e.g.:
<% ns_adp_puts [ns_adp_eval {<% ... %>} %>
* Enhanced ADP error messaging capabilities which can be managed
on a per-interp basis with ns_adp_ctl or configured by default,
e.g.:
ns_section "ns/server/server1/adp"
ns_param stricterror false ;# Stop execution on error
ns_param detailerror true ;# Include connection detail
ns_param displayerror false ;# Return error message to client
When "detailerror" and/or "displayerror" are set, an error message
similar to the following would be appended to the server log and/or
included in the output sent to the client:
This is an error.
while executing
"error "This is an error.""
at line 4 of adp file "/aolserver/servers/server1/pages/error.adp"
while processing connection #2:
GET /error.adp HTTP/1.1
Accept: */*
Accept-Language: en
Accept-Encoding: gzip, deflate
User-Agent: Mozilla/5.0 (Macintosh; U; Intel Mac OS X; en) ...
Connection: keep-alive
Host: 192.168.2.1:8000
* New ADP execution tracing which can be enabled for a request with
the ns_adp_ctl command or configured by default, e.g.:
ns_section "ns/server/server1/adp"
ns_param trace false ;# Enable text/script debug info.
ns_param tracesize 40 ;# Max chars of each chunk to log.
The output of ADP tracing is somewhat limited and poorly formatted.
* Communication driver threads are now dedicated to each instance
of loaded drivers instead of a single thread multiplexing all I/O.
This both simplified the code and enabled the new Ns_QueueWait API.
* Communication driver threads now truncate instead of freeing I/O
dstring buffers, avoiding repeated realloc calls between connections.
* Re-ordered inits to ensure startup fds and log file are initialized
first.
* Communication drivers and connection data access routines have
been updated to support large request content spooled to temp files
instead of always copying to potentially large in-memory regions.
See the Ns_ConnContentFd API for more details.
* Updated to HTTP response version (1.0 or 1.1) to match client
version.
* Updated "ns_db" Tcl API to be a native Tcl_Obj command which
should improve performance over the previous technique of multiple
comparisons of command option strings.
* Added debug driver configuration option which will enable logging
of unusual I/O or protocol errors.
* Simplified charset/encoding config for process-wide url decoding
and per-virtual server defaults for dynamic Tcl requests.
* Several new default file extension to mimetype mappings have been
added in nsd/mimetype.c, e.g., .mp3 = audio/mpeg.
* Created new default index.adp page which displays a number of
AOLserver configuration settings.
* The OS/X dynamic library loading code has been removed in favor
of the more standard dlopen/dlsym API now provided by OS/X 10.4.
Changes to the Build Process:
----------------------------
* The Unix and Win32 build environments have been tightly integrated.
Basically, the makefiles support both gmake on Unix and nmake on
Win32 by including a platform-specific include/ns.mak created by
the nsconfig.tcl Tcl script. As a side effect, several makefile
variable names had to be changed, e.g., "LIB" on Unix is now "DLL"
on Unix and Win32. In addition, platform-independent Tcl scripts
are used for file removal and copying, a further dependency on a
working Tcl installation while building.
* On Unix, backwards compatible Makefile.module and Makefile.global
are provided which simply include the new ns.mak and map old variables
to new names if present, e.g., "LIB" to "DLL".
* On Win32, the legacy Visual Studio build directories have been
removed entirely has their maintenance proved irritating and
complicated.
* A new util/nsinstall-man.sh script installs man pages and also
optionally converts the pages to HTML using groff on Unix.
* Several new makefile install directives have been added, including
install-tcl, install-skel, install-util, install-bins, and install-docs.
* Added new stubs and makefile targets to support building a library
or module in a consistent way which separates the library code
installed as both a static archive and dynamic lib from the library
and/or module init calls.
* Added mynsd example directory showing how to build a custom static
linked nsd binary. The previous conditional startup code and
makefile support has been removed.
* Compiling for 64-bit is known to work in general on Unix. It is
unclear if AOLserver has ever been successfully compiled on 64-bit
Windows and there is a known issue with the Win32 thread interface
which would appear to make this not possible (the thread exit status
pointer is cast to an smaller-sized integer). In addition, while
AOLserver runs on 64-bit Unix systems, the interfaces have not been
scrubbed and updated to support all 64-bit features, e.g., large
file offsets. A more extensive 64-bit certification effort is planned
for the next release.
* Applied patches from Stephen Deasey for better compile time error
checking, e.g., now checking printf-style arguments in Ns_DStringPrintf.
Bug Fixes:
---------
* Fixed crash bug caused by certain command sequences generated
when using the "ns_chan" Tcl API (SF #1094480).
* Fixed crash bug in channel tables cleanup and closing code.
* Fixed crash bug loading nsdb outside a virtual server.
* Fixed invalid return code in Ns_ConnReadLine().
* Updated simple HTML response routines (e.g., Ns_ConnReturnNotice,
ns_returnnotice) to properly encode output from UTF-8.
* Fixed bug where log level "dev" wasn't properly supported.
* Switched to strtol() from atoi() for improved thread safety.
* Added check for Darwin (OS/X) to disable detection of poll() which
appears broken, using the poll() emulation code based on select()
instead.
* Fixed infinite recursion crash bug in GetAddr().
* Removed minor memory leak when no CGI mappings are present.
* Removed most panics when compiling on 64 bit machines.
* A thread race condition which could lead to a crash when using
"ns_server threads" has been fixed.
Known Issues:
------------
* Documentation remains incomplete. We are still missing useful
overview and getting started materials and, while significant
progress has been made, most of the man pages are still empty stubs.
* Loading libnsd into a tclsh and then creating new threads with
the ns_thread command will result in a crash when those threads
exit. The issues has to do with finalization of the async-cancel
context used to support the new "ns_ictl cancel" feature. This bug
is not present when using the "nstclsh" binary.
* The Win32 port does not support all of the new features of temp-file
based request content. In particular, while large requests will
spool to a temp file, a subsequent call to Ns_ConnContent will fail
as the corresponding file mapping code has not yet been written.
* The build environment in general is annoying. As described, there
was an effort in this release to support Unix and Win32 from the
same makefiles which, while a bit clever, likely irritated more
Unix developers than excited Win32 developers. In addition, the
Unix configure script is weak, overly reliant on the context from
a properly built Tcl installation, and the makefiles rely too much
on GNU make features. Creating a proper Unix configure script which
creates proper Makefiles is planned for the next release along with
a possible shift in the Win32 strategy (see next note).
* AOLserver does not compile with the latest Visual C++ 2005 from
Microsoft. While some of the new safety and security features in
Visual C++ are interesting and the debugger has always been excellent,
the complexity of Visual Studio and incompatibility of the Microsoft
command line tools with Unix equivalents makes supporting Win32
increasingly irritating. As such, we will be exploring a shift to
Msys-based Win32 build in the next release, enabling us to utilize
our Unix-based development approach while still allowing AOLserver
to run against native Win32 libraries. Feedback is appreciated.
For now, AOLserver can be compiled on Win32 as described in the
README file using the previous release of Visual C++.
--
Dossy Shiobara | do...@panoptic.com | http://dossy.org/
Panoptic Computer Network | http://panoptic.com/
"He realized the fastest way to change is to laugh at your own
folly -- then you can let go and quickly move on." (p. 70)
Robert Hicks
:Robert
Michael Schlenker
> Does AOLserver building on Windows? Or is there a binary for it?
See the end of the Announcement, where it talks about the VC/nmake based
build environment for Windows.
Michael
Dossy
Indeed, there are some build issues on Win32, but it is definitely
possible to produce Win32 builds.
-- Dossy