Announcing Tornado 1.2

[Ben Darnell]

We are pleased to announce the release of Tornado 1.2, available from

http://github.com/downloads/facebook/tornado/tornado-1.2.tar.gz

Backwards compatibility notes:

* This release includes the backwards-incompatible security change from

  version 1.1.1.  Users upgrading from 1.1 or earlier should read the

  release notes from that release:

  http://groups.google.com/group/python-tornado/browse_thread/thread/b36191c781580cde

* StackContexts that do something other than catch exceptions may need to 

  be modified to be reentrant.  

  https://github.com/facebook/tornado/commit/7a7e24143e77481d140fb5579bc67e4c45cbcfad

* When XSRF tokens are used, the token must also be present on PUT and 

  DELETE requests (anything but GET and HEAD)

New features:

* A new HTTP client implementation is available in the module 

  tornado.simple_httpclient.  This HTTP client does not depend on pycurl.

  It has not yet been tested extensively in production, but is intended

  to eventually replace the pycurl-based HTTP client in a future release of 

  Tornado.  To transparently replace tornado.httpclient.AsyncHTTPClient with 

  this new implementation, you can set the environment variable 

  USE_SIMPLE_HTTPCLIENT=1 (note that the next release of Tornado will

  likely include a different way to select HTTP client implementations)

* Request logging is now done by the Application rather than the

  RequestHandler.  Logging behavior may be customized by either overriding

  Application.log_request in a subclass or by passing log_function

  as an Application setting

* Application.listen(port): Convenience method as an alternative to

  explicitly creating an HTTPServer

* tornado.escape.linkify(): Wrap urls in <a> tags

* RequestHandler.create_signed_value(): Create signatures like the

  secure_cookie methods without setting cookies.

* tornado.testing.get_unused_port(): Returns a port selected in the same

  way as inAsyncHTTPTestCase

* AsyncHTTPTestCase.fetch(): Convenience method for synchronous fetches

* IOLoop.set_blocking_signal_threshold(): Set a callback to be run when

  the IOLoop is blocked.

* IOStream.connect(): Asynchronously connect a client socket

* AsyncHTTPClient.handle_callback_exception(): May be overridden

  in subclass for custom error handling

* httpclient.HTTPRequest has two new keyword arguments, validate_cert and

  ca_certs. Setting validate_cert=False will disable all certificate checks

  when fetching https urls.  ca_certs may be set to a filename containing

  trusted certificate authorities (defaults will be used if this is 

  unspecified)

* HTTPRequest.get_ssl_certificate(): Returns the client's SSL certificate

  (if client certificates were requested in the server's ssl_options

* StaticFileHandler can be configured to return a default file (e.g. 

  index.html) when a directory is requested

* Template directives of the form "{% from x import y %}" are now supported (in

  addition to the existing support for "{% import x %}"

* FacebookGraphMixin.get_authenticated_user now accepts a new

  parameter 'extra_fields' which may be used to request additional information

  about the user

Bug fixes:

* auth: Fixed KeyError with Facebook offline_access

* auth: Uses request.uri instead of request.path as the default redirect

  so that parameters are preserved.

* escape: xhtml_escape() now returns a unicode string, not utf8-encoded bytes

* ioloop: Callbacks added with add_callback are now run in the order they 

  were added

* ioloop: PeriodicCallback.stop can now be called from inside the callback.

* iostream: Fixed several bugs in SSLIOStream

* iostream: Detect when the other side has closed the connection even with

  the select()-based IOLoop

* iostream: read_bytes(0) now works as expected

* iostream: Fixed bug when writing large amounts of data on windows

* iostream: Fixed infinite loop that could occur with unhandled exceptions

* httpclient: Fix bugs when some requests use proxies and others don't

* httpserver: HTTPRequest.protocol is now set correctly when using the

  built-in SSL support

* httpserver: When using multiple processes, the standard library's

  random number generator is re-seeded in each child process

* httpserver: With xheaders enabled, X-Forwarded-Proto is supported as an

  alternative to X-Scheme

* httpserver: Fixed bugs in multipart/form-data parsing

* locale: format_date() now behaves sanely with dates in the future

* locale: Updates to the language list

* stack_context: Fixed bug with contexts leaking through reused IOStreams

* stack_context: Simplified semantics and improved performance

* web: The order of css_files from UIModules is now preserved

* web: Fixed error with default_host redirect

* web: StaticFileHandler works when os.path.sep != '/' (i.e. on Windows)

* web: Fixed a caching-related bug in StaticFileHandler when a file's

  timestamp has changed but its contents have not.

* web: Fixed bugs with HEAD requests and e.g. Etag headers

* web: Fix bugs when different handlers have different static_paths

* web: @removeslash will no longer cause a redirect loop when applied to the 

  root path

* websocket: Now works over SSL

* websocket: Improved compatibility with proxies

Many thanks to everyone who contributed patches, bug reports, and feedback

that went into this release!

-Ben

[HENG]

Wow! Thank ben....

2011/2/21 Ben Darnell <b...@bendarnell.com>

--
--------------------------------------------------------------------
HengZhou
---------------------------------------------------------------------
--

[Bill Janssen]

Hmmm...

% wget http://github.com/downloads/facebook/tornado/tornado-1.2.tar.gz
--2011-02-20 16:35:30-- http://github.com/downloads/facebook/tornado/tornado-1.2.tar.gz
Resolving github.com... 207.97.227.239
Connecting to github.com|207.97.227.239|:80... connected.
HTTP request sent, awaiting response... 301 Moved Permanently
Location: https://github.com/downloads/facebook/tornado/tornado-1.2.tar.gz
[following]
--2011-02-20 16:35:30-- https://github.com/downloads/facebook/tornado/tornado-1.2.tar.gz
Connecting to github.com|207.97.227.239|:443... connected.
ERROR: cannot verify github.com's certificate, issued by `/C=US/
ST=Arizona/L=Scottsdale/O=GoDaddy.com, Inc./OU=http://
certificates.godaddy.com/repository/CN=Go Daddy Secure Certification
Authority/serialNumber=07969287':
Self-signed certificate encountered.
ERROR: certificate common name `*.github.com' doesn't match requested
host name `github.com'.
To connect to github.com insecurely, use `--no-check-certificate'.
%

Not sure if that's a problem, but I thought I'd mention it.

Bill

On Feb 20, 3:44 pm, HENG <ourwait...@gmail.com> wrote:
> Wow! Thank ben....
>
> 2011/2/21 Ben Darnell <b...@bendarnell.com>
>
>
>
>
>
> > We are pleased to announce the release of Tornado 1.2, available from
> >http://github.com/downloads/facebook/tornado/tornado-1.2.tar.gz
>
> > Backwards compatibility notes:
> > * This release includes the backwards-incompatible security change from
> >   version 1.1.1.  Users upgrading from 1.1 or earlier should read the
> >   release notes from that release:
>

> >http://groups.google.com/group/python-tornado/browse_thread/thread/b3...

> > * StackContexts that do something other than catch exceptions may need to
> >   be modified to be reentrant.
>

> >https://github.com/facebook/tornado/commit/7a7e24143e77481d140fb5579b...

[Ben Darnell]

It's apparently a wget problem.  Curl can download the file with no problem.

http://support.github.com/discussions/site/2230-wget-doesnt-like-the-ssl-certificate

-Ben

[Jeremy Kelley]

Great work Ben et al.

Thanks!

-jeremy

Sent from my iPhone

[Didip Kerabat]

On the latest master branch, Tornado `sudo python setup.py install` raised error:

Exception: `curl-config' not found -- please install the libcurl development files

See complete stacktrace: https://gist.github.com/836523

Tornado is looking for the latest pycurl, which is version 7.19.0. And that's where the error happens.
The same error raised when running `sudo easy_install pycurl`

I usually install pycurl via `apt-get install python-pycurl` so i'm not sure whether pycurl in easy_install repo has always been broken or not.

OS: Ubuntu 10.04 LTS 64bit

Thought this might be informative.

- Didip -

[Didip Kerabat]

Some more information,

installing pycurl via `apt-get install python-pycurl` on Ubuntu 10.04 TLS works as expected (version: 1.19.0-3).
Here's the dpkg info: https://gist.github.com/836555

I guess I'm surprised why easy_install didn't recognize it.

- Didip -

[Ben Darnell]

In your first message when "setup.py install" failed, was the python-pycurl ubuntu package installed or not?  If it was installed but setup.py still failed, then there's a problem.  If it wasn't installed, then that's kind of expected - installing pycurl from source (which is what pip/easy_install do) requires the libcurl development files to be installed separately.  On ubuntu, that's the awkwardly-named libcurl4-gnutls-dev package.  

-Ben

[Ben Darnell]

[re-adding the mailing list]

After a little experimentation with a clean ubuntu 10.10 VM, here's what I've found:

* Without a virtualenv and with the python-pycurl ubuntu package installed, it works no matter how I try to install tornado

* With virtualenv and --no-site-packages (which is also what you get with pip -E), the development packages (libcurl4-gnutls-dev and python-dev) must be installed so a new copy of pycurl can be built inside the virtualenv.  pip seems to get confused by the ubuntu installation of pycurl and won't automatically install a new copy inside the virtualenv, so you need to "pip install pycurl" separately.

* With a virtualenv that can use the system site-packages and the python-pycurl ubuntu package installed, pip and easy_install work as expected and use the system pycurl package.  Running setup.py install by hand tries to install a new copy of pycurl and therefore requires the development packages.  The workaround appears to be to always use pip in a virtualenv instead of running setup.py by hand.  (You can use "pip install ." to install a locally-modified copy instead of downloading the 1.2 release)

-Ben

On Sun, Feb 20, 2011 at 7:27 PM, Ben Darnell <b...@bendarnell.com> wrote:

Hmm, that's unfortunate.  Most of my testing was with pip and virtualenv, although now that I've tried it again I see that it shouldn't have worked in the first place.  pip in a virtualenv appears to see out to the system's version of pycurl when it shouldn't (easy_install does the right thing, i.e. it can't see pycurl from inside a --no-site-packages virtualenv).  Everything works for me when I try to install tornado system-wide, though.  Are you using virtualenv (with or without --no-site-packages)?

-Ben

On Sun, Feb 20, 2011 at 6:42 PM, Didip Kerabat <did...@gmail.com> wrote:

Unfortunately, python-pycurl ubuntu package was already installed.

This is an upgrade attempt to existing Tornado app in production.

[Didip Kerabat]

I'm sorry for the late reply, I had family dinner party to take care.

* Without a virtualenv and with the python-pycurl ubuntu package installed, it works no matter how I try to install tornado

That was the path I chose (no virtualenv and no pip) and easy_install couldn't seemed to figure out the pycurl dependency.

After further debugging, pip solves my problem. It can figure out the pycurl dependency:

easy_install pip

cd /path/to/tornado

git pull

python setup.py build

pip install . 

See result below:

Unpacking /path/to/tornado

    Running setup.py egg_info for package from file:///path/to/tornado

Requirement already satisfied (use --upgrade to upgrade): pycurl in /usr/lib/pymodules/python2.6 (from tornado)

Installing collected packages: tornado

    Running setup.py install for tornado

Successfully installed tornado

Cleaning up...

ls -la /usr/local/lib/python2.6/dist-packages reveals tornado-1.2-py2.6.egg and tornado-1.2.egg-info. Problem solved =)

(For archive purpose, the OS is Ubuntu 10.04 LTS 64bit)

Thanks for spending a lot of time in this, Ben.

- Didip -

[Ben Darnell]

On Mon, Feb 21, 2011 at 5:13 PM, Didip Kerabat <did...@gmail.com> wrote:

I'm sorry for the late reply, I had family dinner party to take care.

* Without a virtualenv and with the python-pycurl ubuntu package installed, it works no matter how I try to install tornado

That was the path I chose (no virtualenv and no pip) and easy_install couldn't seemed to figure out the pycurl dependency.

Interesting.  This appears to be a difference between ubuntu 10.4 and 10.10.  I don't have a 10.4 image handy but my old 9.10 VM has the same problem.  

I had added the install_requires declaration in this release so that pip/easy_install would be able to pick up the dependency automatically, but since the best way to install pycurl for most people is going to be apt-get, maybe it's the wrong way to go (we've gotten this far without automatic dependency installation, and I hope that in 1.3 we'll drop the pycurl requirement as SimpleAsyncHTTPClient becomes the default).

-Ben

[Walter Wefft]

Ben Darnell wrote:
> On Mon, Feb 21, 2011 at 5:13 PM, Didip Kerabat <did...@gmail.com
> <mailto:did...@gmail.com>> wrote:
>
>
>> * Without a virtualenv and with the python-pycurl ubuntu package
>> installed, it works no matter how I try to install tornado
>
> That was the path I chose (no virtualenv and no pip) and
> easy_install couldn't seemed to figure out the pycurl dependency.
>
>
> Interesting. This appears to be a difference between ubuntu 10.4 and
> 10.10. I don't have a 10.4 image handy but my old 9.10 VM has the
> same problem.
>
> I had added the install_requires declaration in this release so that
> pip/easy_install would be able to pick up the dependency
> automatically, but since the best way to install pycurl for most
> people is going to be apt-get, maybe it's the wrong way to go (we've
> gotten this far without automatic dependency installation, and I hope
> that in 1.3 we'll drop the pycurl requirement as SimpleAsyncHTTPClient
> becomes the default).
>
> -Ben
>
>
>

FWIW, pip/virtualenv works for me (Ubuntu 9.10) using symlinks to the
system (apt-get installed) pycurl.

I use this wrapper for virtualenv:

http://pypi.python.org/pypi/orb/

which does the symlinking automatically, giving a typical setup like:

$ easy_install -Z orb
$ orb init working.orb --no-site-packages
$ cd working.orb
$ orb install tornado
$ # alternatively, orb install -e
git+https://github.com/facebook/tornado.git@master#egg=tornado-master
$ orb link pycurl psycopg2 mx PIL etc. (anything installed with apt-get
or manually installed)

for example.

Another datapoint - light testing of both async clients with a Kyoto
Tycoon server shows no problems.

[Artemi Krymski]

don't know about ubuntu but for the record tornado + pycurl is very
painful to install on OS X. Different versions of OS X come with
different pycurl versions, and you have to download an appropriate
version from the pycurl website manually and compile it, and even then
it's more of an art than science as I've done this 5 times and each
time took me several hours, god knows why.

my point: let's try to switch to the simple http client as soon as possible ;)

[Jeremy Kelley]

On Tue, Feb 22, 2011 at 5:43 AM, Artemi Krymski <akry...@gmail.com> wrote:

don't know about ubuntu but for the record tornado + pycurl is very
painful to install on OS X.  Different versions of OS X come with
different pycurl versions, and you have to download an appropriate
version from the pycurl website manually and compile it, and even then
it's more of an art than science as I've done this 5 times and each
time took me several hours, god knows why.

my point: let's try to switch to the simple http client as soon as possible ;)

I'm curious, which version of OS X are you using?  I'm on 10.6.6 and have had no problems with pycurl.

I'm actually curious about the differences in performance between pycurl and pure Python.  

Has anyone done any benchmarking?  I ran some very informal tests on my laptop doing a multipart mime post for files using pycurl vs a pure python implementation and the Python was faster by about 15%, but it also used a ton more ram (encoded the entire file in RAM). My tests were just using the shell command "time" on some simple scripts and uploading the same files about 5x.

-j

--
The Christian ideal has not been tried and found wanting;
it has been found difficult and left untried – G. K. Chesterton

[Arek Bochinski]

Artemi,

I think the pure python version of simple http client can use ssl already, so
if you update to the latest version, you can give it a go. I haven't tested
the new client implementation much outside of doing GET over HTTPS,
but it worked without a problem.

[Artemi Krymski]

I am already using the simple http client for FB oauth - no probs.
Just wishing it was the default out of the box.

@Jeremy: the newer the OS X version, the easier it is generally
speaking :) 10.6 does work, but 10.5.8 is a nightmare, and everything
below that is even worse.

Though it is just a matter of time until libcurl gets updated ahead of
an os x release, and the linux guys make the easy_install pycurl work
against the latest libcurl - that's when problems creep in - anytime
you have to install a different libcurl version on os x...

[Ben Darnell]

Yeah, the plan is for the simple client to be the default in the next release.  

We used to have one-line installation instructions for pycurl on OS X 10.5 (there's a way to force easy_install to use an older version of pycurl that matches 10.5's version of libcurl), but that version of libcurl happens to have bugs in its asynchronous interface that cannot be worked around so we had to drop support for that version.

If you're doing server development on an older mac, I recommend using macports or something like it.  It's kind of annoying that it has to install a whole parallel universe of tools, but once you've got it set up it's easier to keep up to date than the libraries that come with the OS.  I still use the python interpreter from macports even though I'm now on 10.6.

-Ben