Google Groups no longer supports new Usenet posts or subscriptions. Historical content remains viewable.
Dismiss

ANNOUNCE: NaviServer 4.99.16 available

41 views
Skip to first unread message

gustafn

unread,
Dec 31, 2017, 11:24:32 AM12/31/17
to
I am pleased to announce the availability of NaviServer 4.99.16 from [1] and [2]. NaviServer is a scalable, extensible, multi threaded web server implemented in C and scriptable via Tcl.

Below are the release notes describing changes since 4.99.14.

all the best in the new year!

-gustaf neumann
[1] https://sourceforge.net/projects/naviserver/
[2] https://bitbucket.org/naviserver/naviserver/



=======================================
NaviServer 4.99.16, released 2017-12-29
=======================================

345 files changed, 16488 insertions(+), 10401 deletions(-)

New Features:
-------------

- ns_cache improvements:

* Allow runtime reaction and re-configuration of caches
(e.g. grow/shrink a cache via new cmd ns_cache_configure)

* Added cache transaction semantics

Background: When ns_cache_* commands are used within a database
transaction (e.g. in OpenACS), it might occur, that partial
results of the transaction are cached before the transaction is
committed. When the transaction is rolled back, invalid values
might be kept in the stack leading to erroneous and hard to
debug behavior. Furthermore, information about changes might
leak into other concurrent threads via the cache, even before
the transaction is committed.

The new cache transaction semantics is implemented via the three
commands

- ns_cache_transaction_begin
- ns_cache_transaction_commit
- ns_cache_transaction_rollback

When no ns_cache_transaction* commands are used, the behavior is
exactly as before.

When ns_cache_transaction* commands are in use, the following
functionalities become available

- The ability to rollback of the values since the matching
ns_cache_transaction_begin

- Isolation of behavior: cached values from incomplete cache
transactions are just visible from the current thread, but
not from other threads.

- Nesting: transactions can be nested
(up to a compile time constant, per default: 16)

- Introspection: the statistics about cache commits and
rollbacks are included in the cache statistics.

* Support caches sizes larger than 2GB

* Summary of new ns_cache* commands
. ns_cache_configure to change cache parameters at runtime
. ns_cache_exists: This command is is more than an order of
magnitude faster than [expr {"foo" in [ns_cache_names]]
. ns_cache_transaction_begin
. ns_cache_transaction_commit
. ns_cache_transaction_rollback


- Virtual server improvements

Make mapping of host entries in the virtual server definition more
intelligent to avoid newbie gotchas and hard to find
mis-configurations (entries in e.g. nssock/servers)
a) add automatically an entry with the port, when non is given
b) complain, when driver is not listening on the specified port
c) add automatically an entry without the port, when the driver
listening on the default port.
Old configurations (doing a-z manually) should continue to work

- Improved integration of NaviServer with e.g service files (systemd):
When the server is starting in a forking mode, return a non-zero
return code, when initialization of the server went wrong (e.g.
error in the config file)


- nsdb:
provide interface for obtaining session_ids. When implementing
e.g. prepared statements over the nsdbpg driver, the prepared
statement is just valid for one session. by providing the
session_id on the Tcl layer, prepared statements can be provided
selectively on the Tcl level.


- https:
* Support OpenSSL 1.1.0 (configuration and runtime)
* Support LibreSSL 2.6.3 and 2.6.4
* Improved support for ancient versions of OpenSSL (e.g. in CentOS 5.0)
* Added support for for client side Server Name Indication (SNI)
* Report attempted TLS handshakes on plain connections to error.log


- nscgi: new config parameter "allowstaticresources"

New parameter "allowstaticresources": provide control on whether
static resources can be delivered from a cgi-bin directory.

Background: When CGI scripts are called from a cgi-bin directory
without an explicit extension mapping to a script interpreter
(e.g. Perl, Bash, ...) the script has to be set executable. If
this is not the case, the CGI script is delivered in source code,
which might reveal unwanted information. The reason for this
behavior was, that one can so deliver also static resources
(e.g. images) from a cgi-bin directory. Now this behavior con be
controlled with the configure parameter "allowstaticresources",
which is by default off. If an application depends on this
behavior, please turn it on in the config file (in the nscgi
section).


- nsproxy:
New subcommand "ns_proxy stats" to provide usage and runtime
statistics per nsproxy pool.


- logging:

* Better error.log entries: Include the connection id in the
thread name to ease debugging, when multiple requests of the
same connection thread write to the error.log and it is not
clear, where the first ends and the next starts.

* Write notice message to error.log when entities are too large
to provide information about possible misconfigurations or
attacks in the error log

* Added new config parameter for nslog: "logthreadname"

When the parameter is specified, the thread name is placed as
second field in the access.log. This eases debugging, since one
can now link the lines in error.log caused by an request with
the line in the access.log without the need of guessing via
time-stamps. The thread name contains the the name of the
connection pool and a connection id.

* When "mutexlocktrace" is activated, use thread name instead
of id to ease debugging


- Added parameter "defaultextension" for adp section in config file.
Can be set e.g. to ".adp" to test for FILENAME.adp, when adp is
mapped to FILENAME



Performance Improvements:
-------------------------

* Improved performance of ns_urlencode/ns_urldecode in common
cases by about a factor of 2

* Improved performance by replacing the slow "snprintf(... %d ...)"
function by a the new functions ns_uint32toa() and ns_uint64toa().
This change improved the performance of EnterSet() by nearly a
factor of 2.5. Before, snprintf() took in this function 3x (!) as
long as Tcl_CreateHashEntry(); the new ns_u32toa() is about 6x
faster than Tcl_CreateHashEntry().

* Improved performance of ns_quotehtml:

In real-world applications (with e.g. OpenACS) this function is
one the the most frequently used functions of NaviServer. This
change improves the performance of "ns_quotehtml" by about 40-60% by
avoiding per-character calls to *DStringAppend and some more
improvements.

* Improved performance of server internal "ns_set" operations: remove
the need of 3 malloc()/free() pairs per request

* Use Tcl_CreateHashEntry() in frequently called functions instead
of Tcl_FindHashEntry() since the second one uses the first.

* Avoid strlen() operations on several occasions
* Improved speed of line-counting in .adp files by a factor of 2
* Reduced scope of mutex lock when registering filters


Bug Fixes:
----------

- Allow fully qualified domain names in "Host" header field
(as allowed in RFC 2976)

- Avoid potential hangs of nxproxy in cases the evaltimeout is very
small and the client is writing much data causing blocking write
operations

- urlencode reform:

* The new code conforms to RFC3986 (2005) rather than RFC1738
(1994) and RFC1808 (1995) vs. RFC2396 (1998). The newer RFC3986
has a more precise and differently structured definition (e.g. no
'unwise' characters) of characters encoded in URLs. The coding of
the query part is actually defined in the HTML 4.01
definitions. Legacy sites can use the old encoding, when
compiling NaviServer with RFC1738 defined.

* Produce a warning, when a not properly encoded URL is passed to
the location header field (e.g. ns_returnredirect). Some newer
browsers might reject those redirects. Only characters, which
have to be always coded, are flagged.

* ns_urldecode: Make sure, that only valid percent codes are
converted back.

* ns_urledecode: Use URL encoding charset, when no charset is
specified. This fixes an old (at least 10 year old) bug which
can allow to sneak in binary nulls into query variables, which
provide a potential injection attack vector


- ns_server active|all|queued: Better handling of querying data from
concurrent threads. Handle now semi-parsed requests: these are not
"queued", but not all information is available for querying it. We
do now the best effort to report on fields that are trusted.

- Complain on connection output operations when the connection is
already closed. Previous version of NaviServer swallowed output
operations on already-closed connections more or less silently,
leading to hard to understand messages in the error log. This
happens in particular often during error handling in OpenACS,
where a "ad_script_obort" is missing. The new code raises now an
error message, pointing to the erroneous code. When the old
behavior is necessary for a while for legacy installations, the
global configuration parameter "rejectalreadyclosedconn" can be
set to "false".

- Fix duplicate DriverClose calls in context of UDP drivers

- Undo potential crlf-translations in the body of (e.g. POST)
requests with Content-Type "www-form-urlencoded"

Background: When a hidden form fields contains line-breaks, these
are transformed pn transit into CRLF by current browsers in POST
requests. In this step the value of a hidden form-field might
become larger on every iteration, which might result in a
quadratic growth on multiple edits of such a field. The new
behavior undoes this effect.

- Improved handling of requests in ancient (pre HTTP/1.0 1996) HTTP
requests in combination with writer threads. Previous versions
could run into a too restrictive sanity assert, assuming that
reply headers must be always present (which is not the case in the
rather informal HTTP versions before HTTP/1.0)

- Virtual server fixes:
* Avoid potential double-initialization of driver modules when
multiple servers are used.
* Postpone registration of virtual servers until all servers
are defined. Before it was possible that NaviServer boot was
terminated, when a default server of a driver was not yet defined.

- Improved handling of binary data:
NsTclObjIsByteArray() behaves now more like the (Tcl internal)
TclIsPureByteArray() to figure out, when the bytearray data has
to be treated as binary. Cases of pure and impure bytearrays
are handled now by NaviServer.

- Fixed old bug, were SSL_shutdown could hang (observed on Linux
systems)

- Fixed potential crashes as flagged by fb infer.

- Improved robustness on invalid URLs: Don't Fatal on invalid URLs
passed to NSDriverClientOpen(), but spit out a warning.

- Improved Tcl code safety (guarding "file delete" against names
starting with a "--")

- Improved handling of invalid input from config file for rolling
files

- Make sure "ns_server serverdir" and "ns_server pagedir" return
normalized paths to avoid potential confusions

- Improved windows portability:
* Cope with changes in Universal CRT in Visual Studio 2015 where e.g.
vsnprintf() is no longer identical to _vsnprintf().

* Improved Windows support for recent versions of visual studio
(versions after visual studio 2010) by introducing NS_EINPROGRESS
and NS_EINTR for abstracting from the raw constants (many thanks to
Maurizio for the input)

* Make nsproxy compile-able under windows

- Improved Unix Portability:
Support systems without RLIMIT_AS (e.g. OpenBSD 6.2)

- Improved Tcl portability:

* Starting with Tcl 8.7a1, Tcl has actually two different types
for bytearrays, the old "tclByteArrayType" and a new
"properByteArrayType", where both have the string name
"bytearray". NsTclObjIsByteArray() is now extended to handle
both types.



Documentation improvements:
---------------------------

- Updated several man pages
* admin-maintenance.man
* commandlist.man
* ns_accesslog.man
* ns_adp_exception.man
* ns_adp_include.man
* ns_base64decode.man
* ns_buildsqldate.man
* ns_cache.man
* ns_cancel.man
* ns_chan.man
* ns_connchan.man
* ns_db.man
* ns_driver.man
* ns_gmtime.man
* ns_http.man
* ns_info.man
* ns_job.man
* ns_locationproc.man
* ns_parseheader.man
* ns_perm.man
* ns_proxy.man
* ns_serverrootproc.man
* ns_shutdown.man
* ns_sockcallback.man
* ns_urlencode.man
* ns_write.man
* nssock.man
* nsssl.man
* ns_setpriveleges.man
* returnstatus-cmds.man
* tcl-overview.man

- Additional global documentation changes:
* Added various cross references with "see also"
* Added keywords for "global builtin" and "server builtin" to
distinguish between per-server commands and global commands
* Added for every module the name of the module as "keyword"
* Make markup of options more consistent

- Improved banner of online man pages:
* fix markup
* make URLs protocol agnostic
* use new .io domain of sourceforge

- Fixed spelling errors


Tcl API Changes:
- ns_connchan: new option "-driver" for "ns_connchan open"
- ns_cache_create returns now boolean to flag success
- "ns_http queue|run" new parameter "-hostname" for handling SNI

- Removed useless and undocumented subcommand "ns_set idelete"

* ns_urlencode: New flag "-uppercase" added for supporting encoding
for OAuth (RFC 5849); note that the "path" segment encoding has
to be used to avoid coding space as "+".

* ns_urlencode and ns_urldecode: Additional accepted value "cookies
"for parameter "-part" to request cookie encoding/decoding.

* ns_getcookie: New option "-all"

Background: By using a cookie domain, it is possible that a
cookie with a specified name is given two value, one from
e.g. the parent domain, one from the current; this can lead to
situation not easy to debug, when e.g. domain cookies are
introduced from other applications in the same domain. The "-all"
option provides infrastructure support to detect such situations
by returning potentially multiple values for the cookie.

* ns_connchan:
New (experimental) subcommand
ns_connchan listen ?-driver d? ?-server s? ?-bind? address port script
Start a listening connection for the ns_connchan interface;
Still missing: https variant via specifying "-driver" +
handling of "-server" (and obtaining defaultserver)

* sendmail.tcl
- For the deprecated warning, do not write entire body of email to
the log
- Use local time (with timezone) instead of UTC in Date header


C API Changes:
--------------

- New API function NsConnRequire() to make handling of required
connections and error results more regular.

- New API functions
. Ns_CookieEncode()
. Ns_CookieDecode()
. NSDriverSockNew(): create an initialized driver Sock structure
. Ns_HttpMessageParse(): parse a (potentially incomplete) HTTP message
. ns_uint32toa()
. ns_uint64toa(): substantially faster conversion of integer to
string than snprintf()

- Ns_SockListenCallback: alter interface to (a) return the listening
socket and (b) to be able to bind to the fresh socket. This is
necessary for e.g. FTPD's passive server connections, where the
server offers a freshly bound socket to the client to connect to
(EPSV, extended passive mode).

- driver.c:
* Factored out LookupDriver() to improve reusability
* New function NSDriverSockNew() to create an initialized driver Sock
structure,


Configuration Changes:
----------------------

- Removed need to specify port for virtual servers in */servers
section of the config files

- Extended sample config files:

* nsd-config.tcl
. Added new nslog parameter "logthreadname"
. Added new global parameter "rejectalreadyclosedconn"

* openacs-config.tcl
. Added configuration for strict-transport-security (HSTS)
. Added setting of OpenACS kernel parameter
"NsShutdownWithNonZeroExitCode" in sample config file
. Eased configuration for nsssl by simply setting https port
. Added new nslog parameter "logthreadname"
. Added new global parameter "rejectalreadyclosedconn"

* sample-config.tcl:
. Included global parameter "shutdowntimeout" in sample config file
. Included parameter "allowstaticresources" in nscgi section
. Added new nslog parameter "logthreadname"
. Added new global parameter "rejectalreadyclosedconn"
. Make it work out-of-the box


Code Changes:
-------------

- Reworked TLS cleanup (necessary for Tcl 8.7 and beyond); align it
to the Tcl practices. Add compatibility for Tcl 8.5- w.r.t
notifier-thread init after the fork.

- Switched to the recommended way of creating detached threads via
pthread attribute (see e.g.
https://docs.oracle.com/cd/E19455-01/806-5257/6je9h032l/index.html)

- Extended regression tests
* ns_urlspace.test
* cookies.test
* ns_sls.test
* tclresp.test
* ns_urlencode.test
* url2file.test
* ns_conn_host.test
* ns_pagepath.test
* ns_serverpath.test
* ns_server.test
* ns_proxy.test
* http.test

- Code cleanup
* Added const declarations
* Added missing prototypes
* Don't shadow local variables
* Marked unused arguments as UNUSED
* Removed old-style function definitions
* Reduced implicit conversions (gcc7)
* Prefer type use "bool" over "int" when applicable
* Don't pass implementation-defined NULL after the last typed
argument to a variadic function
* Reduced number of return statements before end of function
* Reduced variable scopes
* Reduced size of huge switch statements

- API modernization
* Prefer usage of Ns_ParseObjv over manual parsing and
error message generation
* Prefer Tcl_SetObjResult over Tcl_SetResult
* Stop using readdir_r() unless it is forced via compile flag
* Aligned casts to Tcl 8.6
* Replaced all occurrences of strcpy() with more safe variants
* Replaced all occurrences of sprintf() by snprintf()
to protect better against buffer overflows
* Prefer void* over legacy caddr_r
* Replaced all calls to deprecated function Tcl_DStringTrunc()
by Tcl_DStringSetLength()

- Implemented deprecated commands as Tcl proc and complain on
its usage
. ns_puts
. ns_returnadminnotice
. env

- Silenced static checker

- Add CFLAGSs for convenient testing/cleanup
(CFLAGS_TIDY, CFLAGS_DEFINITION, CFLAGS_CONVERSION)

- Improved error messages


Modules:
--------

- nsstats:
. Format values in human readable form (unless &raw=1 is used in query)
. Return proxy stats form all proxy pools
. Include expire date of certificate in "process" page
. Added "commit" and "rollback" to cache statistics

- nsdbpg:
. Report more detailed version numbers used during built process
. Honor "logsqlerrors" settings from config file
. Truncate SQL statement in error.log to 10.000 chars if longer
(to avoid overlong log file entries)
. Improved backwards compatibility with ancient versions of PostgreSQL

- nsimap:
. Added optional flag "-novalidatecert" to "ns_imap open"
(name from the flag of the IMAP implementation)
. Use NaviServer built-in config check for SSL libraries and use same libraries
. Updated to meet engineering standards

- revproxy:
. Added url_rewrite_callback
. Added timeout handling
. Added support for handling requests with provided content (POST, PUT, ...)
. Generalized error handling, reverse callback registration as suggested by David.
. Close connection explicitly after every request

- letsencrypt:
. New module for managing server certificates via Let's Encrypt

- websocket:
. Varius small updates (links, spelling errors, ...)

0 new messages