Is Pylons a meta-package ?

17 views
Skip to first unread message

Tycon

unread,
Jan 11, 2009, 8:33:27 AM1/11/09
to pylons-discuss
Pylons has some useful things in it, however most of them are actually
independent packages that pylons just depends on. The definition of a
meta-package - in the context of Debian's APT, for instance - is a
package that doesn't have any content of its own, but it's simply a
list of dependencies on other package, so that users can get all
those other packages by installing the meta-package.
The question is what does pylons add of its own to make it more than a
meta-package (which is just a list of dependencies). It has a bunch of
useful decorators - https, beaker_cache, validate - but they are all
so flawed (https throws away query params, beaker cant handle
memcache, validate has flawed design and produces invalid html) that I
had to write my own.
I considered the basic strength to be the HTTP server which comes from
Paste which I guess together with some other sub-modules provides all
the HTTP request processing. But then I found out about CherryPy 3 and
confirmed it gives much better performance and a dispatching
architecture, so what exactly is left in pylons ?
Maybe Routes, but if the only route you need is "controller/action/
id" (which I think CherryPy supports) dispatch is enough, then we
don't even need routes fancy (and most likely wasteful and expensive)
mapping.

Mako, SQLAlchemy, and FormEncode are all great but can be used
independently.

Then I found out the real purpose of pylons in the thread below -
basically to be a framework for easy integration of different
projects. That's nice but it's not exactly what I need. My emphasis is
on light-weight and high-performance, so I'm trying to find out what
other parts of Pylons I need, or if I can just use cherrypy+mako
+formencode + sqlalchemy+memcache

http://groups.google.com/group/pylons-discuss/browse_frm/thread/c6c8c5bb596115c2/81b12ce891a9be04?lnk=gst&q=cherrypy#81b12ce891a9be04

Lawrence Oluyede

unread,
Jan 11, 2009, 8:47:06 AM1/11/09
to pylons-...@googlegroups.com
On Sun, Jan 11, 2009 at 2:33 PM, Tycon <adi...@gmail.com> wrote:
> Pylons has some useful things in it, however most of them are actually
> independent packages that pylons just depends on. The definition of a
> meta-package - in the context of Debian's APT, for instance - is a
> package that doesn't have any content of its own, but it's simply a
> list of dependencies on other package, so that users can get all
> those other packages by installing the meta-package.

That's where META PACKAGE definition does not apply to Pylons. Pylons
has source code, glue code and utility code. Hence is hardly just a
meta package. Yes, you are not constrained to use the default
packages, but that's not being a meta package, that's being a
pluggable architecture.

Debian metapackages are just a list of dependencies to install and a label

> The question is what does pylons add of its own to make it more than a
> meta-package (which is just a list of dependencies). It has a bunch of
> useful decorators - https, beaker_cache, validate - but they are all
> so flawed (https throws away query params, beaker cant handle
> memcache, validate has flawed design and produces invalid html) that I
> had to write my own.

It's also a series of variables and objects you use during the
development phase. I don't use those decorators myself but it's not
true that beaker can't handle memcached
At least not the version I use ;-)

> I considered the basic strength to be the HTTP server which comes from
> Paste which I guess together with some other sub-modules provides all
> the HTTP request processing. But then I found out about CherryPy 3 and
> confirmed it gives much better performance and a dispatching
> architecture, so what exactly is left in pylons ?

Paste and CP3 are not meant to be used in the deployment phase. I
really don't care about who's fast during development mode (either
Paste or CP3 or something else), I do care that mod_wsgi in Apache or
nginx or lighttpd are solid.

> Maybe Routes, but if the only route you need is "controller/action/
> id" (which I think CherryPy supports) dispatch is enough, then we
> don't even need routes fancy (and most likely wasteful and expensive)
> mapping.
> Mako, SQLAlchemy, and FormEncode are all great but can be used
> independently.

Yes, you can use pure WSGI too instead of Pylons

I'm losing you here, what are you trying to prove?

--
Lawrence, http://oluyede.org - http://twitter.com/lawrenceoluyede
"It is difficult to get a man to understand
something when his salary depends on not
understanding it" - Upton Sinclair

Mike Orr

unread,
Jan 11, 2009, 3:45:44 PM1/11/09
to pylons-...@googlegroups.com
On Sun, Jan 11, 2009 at 5:47 AM, Lawrence Oluyede <l.ol...@gmail.com> wrote:
>
> On Sun, Jan 11, 2009 at 2:33 PM, Tycon <adi...@gmail.com> wrote:
>> Pylons has some useful things in it, however most of them are actually
>> independent packages that pylons just depends on. The definition of a
>> meta-package - in the context of Debian's APT, for instance - is a
>> package that doesn't have any content of its own, but it's simply a
>> list of dependencies on other package, so that users can get all
>> those other packages by installing the meta-package.
>
> That's where META PACKAGE definition does not apply to Pylons. Pylons
> has source code, glue code and utility code. Hence is hardly just a
> meta package. Yes, you are not constrained to use the default
> packages, but that's not being a meta package, that's being a
> pluggable architecture.
>
> Debian metapackages are just a list of dependencies to install and a label

Pylons is everything that's in the Pylons package itself rather than
in dependencies. The essential part is the PylonsApp class and the
application template. The decorators are peripheral; they were just
thrown in as convenience tools.

Pylons' philosophy is to use third-party distributions as much as
possible. But certain features were not available in any third-party
distribution, so they had to be written from scratch. A Routes-based
dispatcher, for instance. A fully WSGI request manager.

>> I considered the basic strength to be the HTTP server which comes from
>> Paste which I guess together with some other sub-modules provides all
>> the HTTP request processing. But then I found out about CherryPy 3 and
>> confirmed it gives much better performance and a dispatching
>> architecture, so what exactly is left in pylons ?
>
> Paste and CP3 are not meant to be used in the deployment phase. I
> really don't care about who's fast during development mode (either
> Paste or CP3 or something else), I do care that mod_wsgi in Apache or
> nginx or lighttpd are solid.
>
>> Maybe Routes, but if the only route you need is "controller/action/
>> id" (which I think CherryPy supports) dispatch is enough, then we
>> don't even need routes fancy (and most likely wasteful and expensive)
>> mapping.
>> Mako, SQLAlchemy, and FormEncode are all great but can be used
>> independently.

You can use CherryPy with Pylons. It's just a small change in the
[server] section of the ini file, which you can find in the list
archive and maybe on the wiki. Pylons-on-CherryPy is reported to be
better in high-traffic deployments than Pylons-on-PasteHTTPServer.
(Although mod_wsgi may be faster still?)

If you don't want Routes, plain CherryPy may be a good alternative to
Pylons. You can certainly use Mako, SQLAlchemy, and FormEncode with
it.

The "Pylons Execution Analysis" may help explain what's going on and
which module is responsible for what. (It was written for 0.9.6 so a
few things like templating have changed, but most of it is still
accurate.)

http://docs.pythonweb.org/display/pylonscookbook/Pylons+Execution+Analysis+0.9.6

Basically, although the Pylons application *template* is specific to
Pylons, a created application is the user's code, and when it runs
certain portions are not under "Pylons'" control.

So, you run "paster serve", and it reads the config file and
instantiates the server. This is all Paste, not Pylons. The server
may be multithreaded or multiprocess (or mod_wsgi, in which case most
of this is skipped). The application advertises a Setuptools entry
point for myapp.config.middleware:make_app(). Paste uses this to
instantiate the Pylons app.

make_app() -- the users' code -- instantiates a PylonsApp and adds
some middleware around it. There's some trickery in that the
configuration is put into a Pylons config object, and PylonsApp looks
in the config object to find out where the application is. (This was
done to keep boilerplate code in the users' modules to a minimum.)

When a request comes in, Paste calls the PylonsApp as a WSGI
application. The PylonsApp does routing to determine the controller,
instantiates it, and calls it as a WSGI application. The controller
is user code but its base class is Pylons code. The base controller
chooses an action method, looks at its signature to determine which
arguments to call it with, calls it, and incorporates the return value
into the WSGI response. That's about all Pylons does, besides
managing the context variables (request, c, etc).

CherryPy is simpler but less flexible. I"m taking pure CherryPy here,
not Pylons-on-CherryPy. That simplicity may increase runtime speed,
but maybe not enough to be noticed. The Routes overhead is basically
a few method calls. The context variables each add one call per use
plus a couple per request (because they are proxies).

CherryPy 3 does support Routes as an option, not that you would care
since you're trying to avoid overhead.

The main difference between Pylons and CherryPy is philosophical.
CherryPy grafted WSGI and Routes on as options. Pylons was designed
around them in the core, and doesn't include legacy alternatives.

--
Mike Orr <slugg...@gmail.com>

Tycon

unread,
Jan 11, 2009, 4:19:22 PM1/11/09
to pylons-discuss
Thanks, that's the information I was looking for - an explanation of
what the actual pylons code (that is not part of the thrid party
libraries) is doing.

I guess that means routes can be taken out if I remove the routes
middleware and use my own url_for (which I was just about ready to
do).
So in your assessment a simple hello world on pylons using cherrypy
instead of paster (that is my current config) is going to be as fast
as just as an equivalent cherrypy (or pretty close, but not twice as
slow) ?

P.S. - mod_wsgi is a hack, it's an unstable and unscalable
architecture to embed the app server in the web server. And using
daemon mode makes even less sense as using apache+mod_wsgi as a
process monitor for the app server (it's better to use a dedicated and
specialiazed monitoring) as well as a proprietary communication
protocol when we already have http and fcgi, is silly.


On Jan 11, 12:45 pm, "Mike Orr" <sluggos...@gmail.com> wrote:
> On Sun, Jan 11, 2009 at 5:47 AM, Lawrence Oluyede <l.oluy...@gmail.com> wrote:
> http://docs.pythonweb.org/display/pylonscookbook/Pylons+Execution+Ana...
> Mike Orr <sluggos...@gmail.com>

Ian Bicking

unread,
Jan 11, 2009, 4:41:00 PM1/11/09
to pylons-...@googlegroups.com
On Sun, Jan 11, 2009 at 3:19 PM, Tycon <adi...@gmail.com> wrote:
I guess that means routes can be taken out if I remove the routes
middleware and use my own url_for (which I was just about ready to
do).
So in your assessment a simple hello world on pylons using cherrypy
instead of paster (that is my current config) is going to be as fast
as just as an equivalent cherrypy (or pretty close, but not twice as
slow) ?

There's a category error here.

There are a few Paste-Deploy-compatible web servers.  paste.httpserver and CherryPy's servers are two examples (the two most popular for Pylons users).  There's also the flup servers (which aren't http), Spawning, and... some other ones I can't remember now.  Whether you choose paste.httpserver or CherryPy's server is up to you, it doesn't affect any other choice.  They act almost identically (CherryPy is somewhat faster, paste.httpserver has some debugging tools... maybe there's some HTTP/1.1 features that one supports that the other doesn't, but I haven't kept track).

Paste Deploy is a configuration setup to glue together middleware and other components into a single http-serving-wsgi-application.  It also lets you select a web server.

paster serve takes the Paste Deploy configuration and actually runs it.  It's kind of like an app server, but a very small app server.

There's not really any runtime over head to Paste Deploy and paster serve -- they just glue things together, but by the time you are actually serving HTTP they are finished with their jobs.


--
Ian Bicking  |  http://blog.ianbicking.org

Mike Orr

unread,
Jan 11, 2009, 5:09:15 PM1/11/09
to pylons-...@googlegroups.com
On Sun, Jan 11, 2009 at 1:19 PM, Tycon <adi...@gmail.com> wrote:
> I guess that means routes can be taken out if I remove the routes
> middleware and use my own url_for (which I was just about ready to
> do).

What Pylons needs is 'wsgiorg.routing_args', 'routes.route', and
'routes.url' in the WSGI environment. In other words, something has
to emulate the RoutesMiddleware, but it does not have to use Routes.

'wsgiorg.routing_args' is used by the Pylons dispatcher. This was
intentionally defined as a neutral standard so that Routes could be
replaced with something else.

The 'routes.url' object is put onto pylons.url. You'll probably have
to provide a dummy object if you don't have a real one, and I imagine
'None' would be OK. url() is called only when the user calls it
(although the user may call it indirectly via redirect_to()).

I'm not sure if Pylons does anything with 'routes.route', but it may
stick it in the config at least, in which case a dummy value would be
needed.

> So in your assessment a simple hello world on pylons using cherrypy
> instead of paster (that is my current config) is going to be as fast
> as just as an equivalent cherrypy (or pretty close, but not twice as
> slow) ?

You're still using paster. You're just replacing PasteHTTPServer with
CherryPy's server component. The CherryPy *server* handles high
traffic better than PasteHTTPServer, according to several user
reports. The difference between a Pylons-CherryPy application vs a
pure CherryPy application is a different question. I would guess the
pure CherryPy application would be slightly more efficient.

But a sizeable application is going to have significant different
stats compared to "Hello World". The overhead of the framework will
be much less than the overhead of the application, so even if one
framework is 10% more efficient, the net difference in application
speed may be only 1%.

> P.S. - mod_wsgi is a hack, it's an unstable and unscalable
> architecture to embed the app server in the web server. And using
> daemon mode makes even less sense as using apache+mod_wsgi as a
> process monitor for the app server (it's better to use a dedicated and
> specialiazed monitoring) as well as a proprietary communication
> protocol when we already have http and fcgi, is silly.

I use Supervisor to manage PasteHTTPServer daemons, and I like having
a HTTP separate daemon for each application so that I can:

- Start and stop them separately ("supervisor restart myapp")
- See their individual memory usage using 'top'.
- Send localhost HTTP requests to them for troubleshooting.

Efficiency is not on my radar because I have plenty of server
capacity. And the efficiency I *do* worry about is large data
structures in memory, which has nothing to do with the deployment
method (except for the impact of multithreaded vs multiprocess). And
if I *did* have to worry about Python overhead so strictly, I'd also
evaluate even more radical choices like not using Python, using a
cloud service, etc.

Fcgi has a long reputation of being buggy. Perhaps it's better now,
but that's why people like me don't use it. SCGI is a *language
neutral* replacement that was written specifically to be simpler and
more reliable. mod_proxy is of course standard. WSGI may be Python
specific, but it does a reasonably good job of making Python
frameworks interchangeable.

--
Mike Orr <slugg...@gmail.com>

Tycon

unread,
Jan 11, 2009, 5:40:40 PM1/11/09
to pylons-discuss
Does WSGI support remote processes (e.g. on other machines) ? If not
then it's a dead end for scalability. I don't see any reason to use
proprietary protocol when more standard ones exist. I'm only talking
about the spec for daemon mode, not the rest of WSGI call spec.

On Jan 11, 2:09 pm, "Mike Orr" <sluggos...@gmail.com> wrote:
> Mike Orr <sluggos...@gmail.com>

Mike Orr

unread,
Jan 11, 2009, 8:35:04 PM1/11/09
to pylons-...@googlegroups.com
On Sun, Jan 11, 2009 at 2:40 PM, Tycon <adi...@gmail.com> wrote:
>
> Does WSGI support remote processes (e.g. on other machines) ? If not
> then it's a dead end for scalability. I don't see any reason to use
> proprietary protocol when more standard ones exist. I'm only talking
> about the spec for daemon mode, not the rest of WSGI call spec.

I don't even know what you're asking. WSGI is a function signature
and return value, see PEP 333. It doesn't have a daemon mode or even
a server.

PasteHTTPServer is a simple multithreaded server, and that's why it's
the default. It's not meant to be a high-end server for the most
demanding tasks. You can write your own server if you don't approve
of any of the existing ones.

The WSGI environ dict can be pickled and sent via RPC (assuming you
use StringIO objects rather than real file handles). But you can't
pickle a function or iterator, so you can't wholly do WSGI over RPC.
But that's not its purpose. There are other protocols such as SCGI
and HTTP for talking to other computers. WSGI is intended for
in-process chaining of chunks of Python code that may have been
written by different people.

--
Mike Orr <slugg...@gmail.com>

mk

unread,
Jan 12, 2009, 1:05:15 PM1/12/09
to pylons-...@googlegroups.com
Lawrence Oluyede wrote:

> Paste and CP3 are not meant to be used in the deployment phase. I
> really don't care about who's fast during development mode (either
> Paste or CP3 or something else), I do care that mod_wsgi in Apache or
> nginx or lighttpd are solid.

Would you recommend using ngxinx + mod_wsgi (as described in following
link) as production configuration?

http://wiki.codemongers.com/NginxNgxWSGIModule

Regards,
mk


Tycon

unread,
Jan 12, 2009, 3:22:16 PM1/12/09
to pylons-discuss
No, mod_wsgi is a hack. Embedded mode is bad, no serious website is
running app server embedded in web server. Daemon mode is even more
stupid, an unstable (and non scalable) way of using web server to
manage the app server, and invent a new communication protocol between
them, when standard ones that support distributed architecture already
exist.

Dalius Dobravolskas

unread,
Jan 12, 2009, 3:44:40 PM1/12/09
to pylons-...@googlegroups.com
On Mon, Jan 12, 2009 at 10:22 PM, Tycon <adi...@gmail.com> wrote:
>
> No, mod_wsgi is a hack. Embedded mode is bad, no serious website is
> running app server embedded in web server. Daemon mode is even more
> stupid, an unstable (and non scalable) way of using web server to
> manage the app server, and invent a new communication protocol between
> them, when standard ones that support distributed architecture already
> exist.

Tycon, are you talking about nginx's mod_wsgi or apache's mod_wsgi.
That's two completely different projects IMHO.

As for mod_wsgi on Apache I doubt that it is not scalable. At least it
should be as much scalable as Apache. And it is not hack it is WSGI
implementation for Apache. Graham could answer better here.

BTW, I recommend using Apache with mod_wsgi. I'm running 6 pylons
applications under Apache with mod_wsgi + MySql and I fit into 256 Mb
of RAM and it runs really fast. From other side I don't have high
loads and don't expect that soon.

--
Dalius
http://blog.sandbox.lt

Mike Orr

unread,
Jan 12, 2009, 3:46:35 PM1/12/09
to pylons-...@googlegroups.com
On Mon, Jan 12, 2009 at 12:22 PM, Tycon <adi...@gmail.com> wrote:
>
> No, mod_wsgi is a hack. Embedded mode is bad, no serious website is
> running app server embedded in web server. Daemon mode is even more
> stupid, an unstable (and non scalable) way of using web server to
>
>
> On Jan 12, 10:05 am, mk <mrk...@gmail.com> wrote:
>> Lawrence Oluyede wrote:
>> > Paste and CP3 are not meant to be used in the deployment phase. I
>> > really don't care about who's fast during development mode (either
>> > Paste or CP3 or something else), I do care that mod_wsgi in Apache or
>> > nginx or lighttpd are solid.

There are a lot of opinions being thrown about as if they're the
absolute truth, and I'm afraid it may be confusing people. I can't
speak officially for Pylons (only Ben can), but I can say that Pylons
has been deployed on PasteHTTPServer, CherryPy, mod_wsgi, mod_proxy,
nginx, and others, and all have been stable and are reasonable
choices. It's not true that "embedded mode is bad", "daemon mode is
even more stupid", or "Paste and CP3 are not meant to be used in the
deployment phase" (especially CP3). They may be too underperformant
for certain situations, but that does not make them bad across the
board even if certain people think so.

The argument against PasteHTTPServer and CherryPy3 seems to be
efficiency. The arguments against mod_wsgi and daemon mode seem to be
ideological. (And I'm lost now: if you don't have mod_wsgi and you
don't have an application daemon, what other choice is there?)

--
Mike Orr <slugg...@gmail.com>

Graham Dumpleton

unread,
Jan 12, 2009, 4:51:30 PM1/12/09
to pylons-discuss


On Jan 13, 7:44 am, "Dalius Dobravolskas"
<dalius.dobravols...@gmail.com> wrote:
> On Mon, Jan 12, 2009 at 10:22 PM, Tycon <adie...@gmail.com> wrote:
>
> > No, mod_wsgi is a hack. Embedded mode is bad, no serious website is
> > running app server embedded in web server. Daemon mode is even more
> > stupid, an unstable (and non scalable) way of using web server to
> > manage the app server, and invent a new communication protocol between
> > them, when standard ones that support distributed architecture already
> > exist.
>
> Tycon, are you talking about nginx's mod_wsgi or apache's mod_wsgi.
> That's two completely different projects IMHO.
>
> As for mod_wsgi on Apache I doubt that it is not scalable. At least it
> should be as much scalable as Apache. And it is not hack it is WSGI
> implementation for Apache. Graham could answer better here.

I have learned that there is no point trying to counter these sorts of
accusations. They never want to provide proof of why what they claim
is true and even when you point out where their arguments are wrong,
make no sense, or have been proven otherwise through actual use, they
don't listen and just keep repeating the same FUD. They also never
want to accept the basic concept of choice and that different systems
are going to suit different peoples requirements or mindsets.

The #pylons, #wsgi and #python.web irc channels seem to be a breeding
ground for this sort of stuff. If I didn't know any better I would
think this Tycon is the twin brother of one particular person on those
irc channels. Although, that other person may have toned down their
rhetoric by now, I haven't bothered to check lately what they are
saying.

It is pretty sad really, I would be quite happy adding those irc
channels to my watch list and helping out people with WSGI, mod_wsgi,
mod_python and Apache questions, but I don't because I would have to
spend most of my time just countering the FUD that was being put
around about both mod_python, mod_wsgi and use of Apache.

Graham

Lawrence Oluyede

unread,
Jan 12, 2009, 7:17:08 PM1/12/09
to pylons-...@googlegroups.com

I honestly don't know anything about nginx, altough I know personally
the author of its mod_wsgi. Never used. We use modwsgi for Apache

Lawrence Oluyede

unread,
Jan 12, 2009, 7:20:42 PM1/12/09
to pylons-...@googlegroups.com
On Mon, Jan 12, 2009 at 9:46 PM, Mike Orr <slugg...@gmail.com> wrote:
> There are a lot of opinions being thrown about as if they're the
> absolute truth, and I'm afraid it may be confusing people.

Yeah, like saying that modwsgi + Apache are bad...

> I can't
> speak officially for Pylons (only Ben can), but I can say that Pylons
> has been deployed on PasteHTTPServer, CherryPy, mod_wsgi, mod_proxy,
> nginx, and others, and all have been stable and are reasonable
> choices.

Yes I'm aware of that, I did use PasteHTTPServer in production for a
while. What I was trying to say is that there are better options if we
talk about scaling or performances (what Tycon was talking about all
along)

> It's not true that "embedded mode is bad", "daemon mode is
> even more stupid", or "Paste and CP3 are not meant to be used in the
> deployment phase" (especially CP3). They may be too underperformant
> for certain situations, but that does not make them bad across the
> board even if certain people think so.

I don't think they are bad, that's for sure

> The argument against PasteHTTPServer and CherryPy3 seems to be
> efficiency. The arguments against mod_wsgi and daemon mode seem to be
> ideological. (And I'm lost now: if you don't have mod_wsgi and you
> don't have an application daemon, what other choice is there?)

Asynchronous web servers I guess

Mike Orr

unread,
Jan 12, 2009, 9:03:06 PM1/12/09
to pylons-...@googlegroups.com
On Mon, Jan 12, 2009 at 4:20 PM, Lawrence Oluyede <l.ol...@gmail.com> wrote:
>> The argument against PasteHTTPServer and CherryPy3 seems to be
>> efficiency. The arguments against mod_wsgi and daemon mode seem to be
>> ideological. (And I'm lost now: if you don't have mod_wsgi and you
>> don't have an application daemon, what other choice is there?)
>
> Asynchronous web servers I guess

Does Pylons even work with asynchronous servers? I guess as long as
you don't have a database.

Last I heard, Twisted ran WSGI applications in a thread because they
might block.

--
Mike Orr <slugg...@gmail.com>

Lawrence Oluyede

unread,
Jan 13, 2009, 3:45:30 AM1/13/09
to pylons-...@googlegroups.com
On Tue, Jan 13, 2009 at 3:03 AM, Mike Orr <slugg...@gmail.com> wrote:
>
> On Mon, Jan 12, 2009 at 4:20 PM, Lawrence Oluyede <l.ol...@gmail.com> wrote:
>>> The argument against PasteHTTPServer and CherryPy3 seems to be
>>> efficiency. The arguments against mod_wsgi and daemon mode seem to be
>>> ideological. (And I'm lost now: if you don't have mod_wsgi and you
>>> don't have an application daemon, what other choice is there?)
>>
>> Asynchronous web servers I guess
>
> Does Pylons even work with asynchronous servers?

Don't think so. We should ask Manlio Perillo, nginx's modwsgi author,
you can find him on the web-sig ML

mk

unread,
Jan 13, 2009, 7:40:49 AM1/13/09
to pylons-...@googlegroups.com
Lawrence Oluyede wrote:
> I honestly don't know anything about nginx, altough I know personally
> the author of its mod_wsgi. Never used. We use modwsgi for Apache

That's the response on what you actually use in production, thanks. :-)

Regards,
mk


Tycon

unread,
Jan 13, 2009, 9:52:43 AM1/13/09
to pylons-discuss
I think CherryPy is awesome, much better than "Paster" HTTP (which
cant even do HTTPS).

In general I prefer to separate the web server and app server (over
the embedded approach) as it is more modular, flexible and scalable.
But embedded deployment has a theoretical performance advantage
because there is no overhead of communication between the web and app
server.

The best option for embedded deployment is apache+modwsgi (but apache
+mod_python or nginx+mod_wsgi also exist). Apache+modwsgi (embedded)
is faster than Apache as a reverse proxy, and also outperfoms
standalone CherryPy or PasterHTTP, but it's not faster than nginx as
reverse proxy with CherryPy (performance is about the same for dynamic
content, and nginx serves static content faster). In addition the
memory consumption with apache+modwsgi embedded mode is much higher.
So despite the theoretical advantage of embedded mode (for small
deployment), it does not materialize into actual advantage for apache
+modwsgi.

Last word on modwsgi and its "daemon" mode, which is similar to
reverse proxy and fcgi in that it separates the web server and app
server. As such, it has the same theoretical performance as reverse
proxy and fcgi (which in fact provide the same performance), but it
uses a proprietary communication protocol, and inlike proxy or fcgi,
it requires the app and web server processes to be on the same machine
(so it does not scale across a pool of machines). Also this "modwsgi"
provides a rudimentary way for the web server (e.g. apache) to monitor
the app server, but this is a poor substitute for actual server
monitoring tools. So In summary modwsgi daemon mode is completely
useless, while modwsgi embedded mode is the best choice when using
apache as the webserver, if you only have a single node and enough
memory on it.

Tycon

unread,
Jan 13, 2009, 10:11:37 AM1/13/09
to pylons-discuss
Actually one correction: standalone CherryPy smokes everything else
(including apache+modwsgi embedded mode) for dynamic content (It's
almost TWICE as fast). But CheeryPy (or any python HTTP server) is
slower for serving static files. So if you don't have many static
files, it might be better to use standalone CherryPy over apache
+modwsgi embedded mode.

Mike Orr

unread,
Jan 13, 2009, 6:42:32 PM1/13/09
to pylons-...@googlegroups.com
On Tue, Jan 13, 2009 at 6:52 AM, Tycon <adi...@gmail.com> wrote:
> Last word on modwsgi and its "daemon" mode, which is similar to
> reverse proxy and fcgi in that it separates the web server and app
> server. As such, it has the same theoretical performance as reverse
> proxy and fcgi (which in fact provide the same performance), but it
> uses a proprietary communication protocol, and inlike proxy or fcgi,
> it requires the app and web server processes to be on the same machine

Is *that* what you're talking about when you say "daemon mode" and
"proprietary protocol". I thought you meant daemon mode as in running
PasteHTTPServer or CherryPy as a daemon, and proprietary protocol as
in WSGI or SCGI.

The main point of mod_wsgi's daemon mode is to isolate bugs/memory
leaks between the web application and the server, and to track the
application's individual resource usage in the 'ps' listing. It's not
designed for multi-machine scalability.

As for its "proprietary" protocol, I consider that an internal matter
of mod_wsgi. What matters is whether it works, and I haven't heard
any complaints in that regard.

Ultimately it comes down to the sysadmin's time of setting up mod_wsgi
now and possibly switching to something else later, vs setting up
something multi-machine scalable now (which is more work up front).
And that depends on how likely a traffic onslaught is, how quickly the
load will accelerate, and the sysadmin's future availability.

--
Mike Orr <slugg...@gmail.com>

Graham Dumpleton

unread,
Jan 13, 2009, 8:53:14 PM1/13/09
to pylons-discuss


On Jan 14, 10:42 am, "Mike Orr" <sluggos...@gmail.com> wrote:
You don't need to switch to something else when you want to go to a
multi machine configuration. This is just part of the FUD that they
were pushing on the irc channels in the past to try and discredit
Apache/mod_wsgi. It just seems that Tycon is parroting this same
message. I wouldn't be surprised if he has never even used Apache/
mod_wsgi. Certainly the originator of a lot of this FUD on the irc
channels in the past freely admitted he had installed neither
mod_python or mod_wsgi with Apache.

When you want to start looking at horizontal scaling you do exactly
what you would do were you scaling up Apache for any other scenario.
That is, you stick perbal, pound, nginx or some other proxying/load
balancing solution in front and run an Apache instance of each of the
machines in your cluster.

Since running nginx in front of Apache/mod_wsgi to handle static files
is a common scenario, that same nginx instance could be used for the
job, given that it still likely going to handle the load of both
static files and proxying fine because of being event driven rather
than threaded.

Hmmm, this sounds exactly like what is because suggested if using
paste server or cherrypy wsgi server. Strange that.

When you read this persons other posts I think it is quite clear that
he doesn't really understand the difference between the WSGI
specification and mod_wsgi as an implementation of it. Specifically
that WSGI is a programmatic API and not a wire protocol. It is also
questionable how much he knows how to setup Apache and mod_wsgi.
Probably will never see anything about how they actually configured
Apache and mod_wsgi for these benchmarks that he surely must have run
to come to these conclusions. Like most benchmarks they are probably
flawed due to not setting up his tests properly so equal comparison
was being performed, or not even benchmarking something realistic.

What I find particularly clueless about comparisons between different
hosting mechanisms for dynamic web applications is that they quite
often test a hello world application and not a real application. As
such any figures are pretty meaningless given that any difference
between different hosting mechanism is likely in milliseconds. When
for a large application the overall request time is in the 10-100
milliseconds, the difference just disappears as noise within the real
bottleneck which is the application and or database access. Add on top
of that that you would never want to run your web server at maximum
capacity on an ongoing basis, you would generally have more than
enough headroom even if different hosting solutions perform a bit
different, thus differences don't matter anyway.

Overall one would be much better off focusing your time on improving
the performance of your application and database access than having a
pissing contest about raw request throughput for some unrealistic
pattern of traffic that your site will never experience.

Graham

Jorge Vargas

unread,
Jan 13, 2009, 10:18:56 PM1/13/09
to pylons-...@googlegroups.com

hi Ian, is this explanation somewhere in pythonpaste? or somewhere
else? I see this being asked a lot in several places (irc, other
mailing lists,etc) I'll like to have a link to point to.

Jorge Vargas

unread,
Jan 13, 2009, 10:21:03 PM1/13/09
to pylons-...@googlegroups.com
On Sun, Jan 11, 2009 at 4:40 PM, Tycon <adi...@gmail.com> wrote:
>
> Does WSGI support remote processes (e.g. on other machines) ? If not
> then it's a dead end for scalability. I don't see any reason to use
> proprietary protocol when more standard ones exist. I'm only talking
> about the spec for daemon mode, not the rest of WSGI call spec.
>

wsgi != mod_wsgi.

WSGI is a python protocol for app-app communication and contrary to
proprietary it is today probably the most used "standard" in python's
web development.

Jorge Vargas

unread,
Jan 13, 2009, 10:32:35 PM1/13/09
to pylons-...@googlegroups.com
On Mon, Jan 12, 2009 at 2:22 PM, Tycon <adi...@gmail.com> wrote:
>
> No, mod_wsgi is a hack. Embedded mode is bad, no serious website is
> running app server embedded in web server. Daemon mode is even more
> stupid, an unstable (and non scalable) way of using web server to
> manage the app server, and invent a new communication protocol between
> them, when standard ones that support distributed architecture already
> exist.
>

you are right no self respecting site will use that piece of junk!
although I saw this email some time ago
http://groups.google.com/group/modwsgi/browse_thread/thread/88de3e07ea574ddb/95612a8e94613bf7
so you may reconsider that statement given this line.

"Apache/2.2.9 (Debian) mod_python/3.3.1 Python/2.5.2 mod_wsgi/2.3
Server at pypi.python.org Port 80"

Tycon

unread,
Jan 13, 2009, 11:10:32 PM1/13/09
to pylons-discuss
Actually I have apache+modwsgi running flawlessly, and everything I
said is based on meticulous performance benchmarking and theoretical
profiling of deployment architectures.

All benchmarks were performed on pylons full stack production mode,
with debugging and logging turned off. The test was a simple "hello
world" page with no template rendering, database access or other
external links or references. Apache is prefork MPM v2.2.8 with
modwsgi 1.3 using python 2.5.2.

The benchmarks CLEARLY show that using a stand-alone app server is
MUCH faster then using apache+modwsgi to serve a page (returned from
the aforementioned "hello world" controller action). When using
CherryPy as the HTTP server for pylons, the req/sec is almost twice as
fast as apache+modwsgi. PasteHTTP is 15% slower than CherryPy but
still much faster then Apache+modwsgi.

But of course using a stand-alone app-server as the web-server has a
few drawbacks:

1. Static files are also served by pylons, which is slow.
2. Cannot use multiple processes, which is required to make optimal
use of system resources and allow for scalability across multiple CPUs
and multiple machines.

So in most cases you will want to have a reverse proxy front-end that
acts as a load balancer as well as serve the static files. As you
mention, nginx is a good choice. In this configuration, I would assume
it would be better to have nginx forward the requests directly to a
pool of standalone pylons app-servers, rather than to a pool of apache
+modwsgi processes.

So, while using only standalone CherryPy to serve a pylons app has
some drawbacks, those drawbacks are avoided if using reverse proxy
that serves as a load balancer and to serve static files.

So when is it best to use apache+modwsgi ? If we have a single node
with mutiple cores and a lot of RAM, and we need to serve a lot of
static files, then apache+modwsgi would be better than using a
standalone app server. But even for this scenario, nginx as a reverse
proxy gives about the same performance.

On Jan 13, 5:53 pm, Graham Dumpleton <Graham.Dumple...@gmail.com>
wrote:

Jorge Vargas

unread,
Jan 13, 2009, 11:29:47 PM1/13/09
to pylons-...@googlegroups.com
On Tue, Jan 13, 2009 at 10:10 PM, Tycon <adi...@gmail.com> wrote:
>
> Actually I have apache+modwsgi running flawlessly, and everything I
> said is based on meticulous performance benchmarking and theoretical
> profiling of deployment architectures.
>
> All benchmarks were performed on pylons full stack production mode,
> with debugging and logging turned off. The test was a simple "hello
> world" page with no template rendering, database access or other
> external links or references. Apache is prefork MPM v2.2.8 with
> modwsgi 1.3 using python 2.5.2.
>
> The benchmarks CLEARLY show that using a stand-alone app server is
> MUCH faster then using apache+modwsgi to serve a page (returned from
> the aforementioned "hello world" controller action). When using
> CherryPy as the HTTP server for pylons, the req/sec is almost twice as
> fast as apache+modwsgi. PasteHTTP is 15% slower than CherryPy but
> still much faster then Apache+modwsgi.
>

I'm sorry but that isn't a viable testing environment. template
rendering and db access are the most time consuming tasks pylons will
undertake. On the other hand you never pointed out how many request
per second are you sending to the system.

Graham Dumpleton

unread,
Jan 13, 2009, 11:36:02 PM1/13/09
to pylons-discuss


On Jan 14, 3:10 pm, Tycon <adie...@gmail.com> wrote:
> Actually I have apache+modwsgi running flawlessly, and everything I
> said is based on meticulous performance benchmarking and theoretical
> profiling of deployment architectures.
>
> All benchmarks were performed on pylons full stack production mode,
> with debugging and logging turned off. The test was a simple "hello
> world" page with no template rendering, database access or other
> external links or references. Apache is prefork MPM v2.2.8 with
> modwsgi 1.3 using python 2.5.2.

Old version of mod_wsgi. If using daemon mode of mod_wsgi then 2.X is
somewhat faster.

Using embedded mode, prefork especially can stuff up benchmarking
because the burst in traffic will cause Apache to create lots of
additional child worker processes to handle load. Every one of those
then has to load application and that can kill machine performance
while occurring, interfering with benchmark results. As such, for
applications with high load cost, actually better to use daemon mode
with fixed number of processes and with worker MPM for Apache.
Stripping Apache of unused modules that take a lot of memory,
especially PHP, is also advisable.

> The benchmarks CLEARLY show that using a stand-alone app server is
> MUCH faster then using apache+modwsgi to serve a page (returned from
> the aforementioned "hello world" controller action). When using
> CherryPy as the HTTP server for pylons, the req/sec is almost twice as
> fast as apache+modwsgi. PasteHTTP is 15% slower than CherryPy but
> still much faster then Apache+modwsgi.

Generally contrary results on MacOSX, with both CherryPy and Paste
server running quite badly in comparison.

On Ubuntu CherryPy runs about the same and Paste server slower.

What operating system are you using?

Did your tests take into consideration that Apache/mod_wsgi lazy loads
application object on first request where as the others preload. This
in part plays into why benchmarking results can look bad with Apache/
mod_wsgi, people are actually counting startup costs when they
shouldn't.

Graham

Graham Dumpleton

unread,
Jan 13, 2009, 11:38:50 PM1/13/09
to pylons-discuss


On Jan 14, 2:32 pm, Jorge Vargas <jorge.var...@gmail.com> wrote:
> On Mon, Jan 12, 2009 at 2:22 PM, Tycon <adie...@gmail.com> wrote:
>
> > No, mod_wsgi is a hack. Embedded mode is bad, no serious website is
> > running app server embedded in web server. Daemon mode is even more
> > stupid, an unstable (and non scalable) way of using web server to
> > manage the app server, and invent a new communication protocol between
> > them, when standard ones that support distributed architecture already
> > exist.
>
> you are right no self respecting site will use that piece of junk!
> although I saw this email some time agohttp://groups.google.com/group/modwsgi/browse_thread/thread/88de3e07e...
> so you may reconsider that statement given this line.
>
> "Apache/2.2.9 (Debian) mod_python/3.3.1 Python/2.5.2 mod_wsgi/2.3
> Server at pypi.python.org Port 80"

The pycon site mustn't know what they are doing either. ;-)

Graham

Tycon

unread,
Jan 14, 2009, 12:51:27 AM1/14/09
to pylons-discuss
Im using Ubunto 8.04 LTS and running the test using ab -n 10000 -c 10
to localhost, in order to focus on the request handling code in the
server stack and remove or reduce the weight of all other application
logic such as db access and page rendering. This is not supposed to be
a realistic timing of a real world request, but a highlight of the
differences in efficiency of different request handling stacks
including network layer, HTTP and routing code.

All software package veriosns I'm using are the versions supplied in
the OFFICIAL repositores for the distribution. Pylons and cherrypy
were installed using (non)easy-install so their version is the latest
rather than a version tied to any official repository.

I'm running in a full VM (single core) so I'm pretty sure my results
are correct: CherryPy (started using "paster serve prodduction.ini")
is twice as fast (1300 req/sec for "hello world") as Apache+modwsgi
(650 req/sec). The only difference you may have is using worker MPM
instead of pre-fork MPM, and newer versions of apche and modwsgi. In
addition, apache is also logging the requests which is an additional
overhead that pylons doesn't do in my configuration. But even so I
doubt if Apache could match standalone pylons using CherryPy.

On Jan 13, 8:36 pm, Graham Dumpleton <Graham.Dumple...@gmail.com>

Mike Orr

unread,
Jan 14, 2009, 1:08:44 AM1/14/09
to pylons-...@googlegroups.com

Those are just toy sites for a hobbyist programming language, not big
mean serious e-commerce servers. And they're so inefficient they must
waste a huge number of bogomips!

--
Mike Orr <slugg...@gmail.com>

Colin Flanagan

unread,
Jan 14, 2009, 1:13:20 AM1/14/09
to pylons-...@googlegroups.com
Performance considerations aside, why is mod_wsgi a "hack"? Because it doesn't perform as wel as other solutions? Can you address your concerns about the implementation vs the performance benchmarks?

Jorge Vargas

unread,
Jan 14, 2009, 1:54:38 AM1/14/09
to pylons-...@googlegroups.com
On Tue, Jan 13, 2009 at 11:51 PM, Tycon <adi...@gmail.com> wrote:
>
> Im using Ubunto 8.04 LTS and running the test using ab -n 10000 -c 10
> to localhost, in order to focus on the request handling code in the
> server stack and remove or reduce the weight of all other application
> logic such as db access and page rendering. This is not supposed to be
> a realistic timing of a real world request, but a highlight of the
> differences in efficiency of different request handling stacks
> including network layer, HTTP and routing code.
>
> All software package veriosns I'm using are the versions supplied in
> the OFFICIAL repositores for the distribution. Pylons and cherrypy
> were installed using (non)easy-install so their version is the latest
> rather than a version tied to any official repository.
>
http://code.google.com/p/modwsgi/downloads/list doesn't even lists 1.x anymore
as a note "official debian package" is 2.x
http://packages.debian.org/changelogs/pool/main/m/mod-wsgi/mod-wsgi_2.3-1/changelog

now it seems ubuntu doesn't has a 2.0 package
http://packages.ubuntu.com/hardy/libapache2-mod-wsgi

so your "definitive" results are totally coerced by the fact that your
operating system is running outdated code for modwsgi,

in other words you are running the very last version of a software
with something that is almost 2 years old. And you expect us to
believe it's not bias?

lets run firefox 1.0 vs ie 7 today and see who is better, ummm wait
bad example. But we are talking about competent projects here.

Jorge Vargas

unread,
Jan 14, 2009, 1:58:57 AM1/14/09
to pylons-...@googlegroups.com
On Wed, Jan 14, 2009 at 12:54 AM, Jorge Vargas <jorge....@gmail.com> wrote:
> On Tue, Jan 13, 2009 at 11:51 PM, Tycon <adi...@gmail.com> wrote:
>>
>> Im using Ubunto 8.04 LTS and running the test using ab -n 10000 -c 10
>> to localhost, in order to focus on the request handling code in the
>> server stack and remove or reduce the weight of all other application
>> logic such as db access and page rendering. This is not supposed to be
>> a realistic timing of a real world request, but a highlight of the
>> differences in efficiency of different request handling stacks
>> including network layer, HTTP and routing code.
>>
>> All software package veriosns I'm using are the versions supplied in
>> the OFFICIAL repositores for the distribution. Pylons and cherrypy
>> were installed using (non)easy-install so their version is the latest
>> rather than a version tied to any official repository.
>>
> http://code.google.com/p/modwsgi/downloads/list doesn't even lists 1.x anymore
> as a note "official debian package" is 2.x
> http://packages.debian.org/changelogs/pool/main/m/mod-wsgi/mod-wsgi_2.3-1/changelog
>
> now it seems ubuntu doesn't has a 2.0 package
> http://packages.ubuntu.com/hardy/libapache2-mod-wsgi
>
sorry I was looking at the wrong ubuntu version it does have a 2.3
package as shown here
http://packages.ubuntu.com/intrepid/libapache2-mod-wsgi

but you are still running 1.3 as you posted above, so everything else
still stands

Tycon

unread,
Jan 14, 2009, 5:21:54 AM1/14/09
to pylons-discuss
I installed modwsgi 2.3 and apache worker MPM and performance improved
by 30% but CherryPy is still significantly faster.

On Jan 13, 10:58 pm, Jorge Vargas <jorge.var...@gmail.com> wrote:
> On Wed, Jan 14, 2009 at 12:54 AM, Jorge Vargas <jorge.var...@gmail.com> wrote:
> > On Tue, Jan 13, 2009 at 11:51 PM, Tycon <adie...@gmail.com> wrote:
>
> >> Im using Ubunto 8.04 LTS and running the test using ab -n 10000 -c 10
> >> to localhost, in order to focus on the request handling code in the
> >> server stack and remove or reduce the weight of all other application
> >> logic such as db access and page rendering. This is not supposed to be
> >> a realistic timing of a real world request, but a highlight of the
> >> differences in efficiency of different request handling stacks
> >> including network layer, HTTP and routing code.
>
> >> All software package veriosns I'm using are the versions supplied in
> >> the OFFICIAL repositores for the distribution. Pylons and cherrypy
> >> were installed using (non)easy-install so their version is the latest
> >> rather than a version tied to any official repository.
>
> >http://code.google.com/p/modwsgi/downloads/listdoesn't even lists 1.x anymore
> > as a note "official debian package" is 2.x
> >http://packages.debian.org/changelogs/pool/main/m/mod-wsgi/mod-wsgi_2...
>
> > now it seems ubuntu doesn't has a 2.0 package
> >http://packages.ubuntu.com/hardy/libapache2-mod-wsgi
>
> sorry I was looking at the wrong ubuntu version it does have a 2.3
> package as shown herehttp://packages.ubuntu.com/intrepid/libapache2-mod-wsgi
> >>> > > Probably will never see anything about how they actually...
>
> read more »

Tycon

unread,
Jan 14, 2009, 8:36:58 AM1/14/09
to pylons-discuss
I specifically meant modwsgi's daemon mode is a hack, which is like I
explained a proprietary unreliable and non scalable communication
protocol.
As for embedded mode, it's not a hack but as it turns out its
performance isn't all that great given that it under-performs a pure
python HTTP server like the one in CherryPy.
> > > > enough...
>
> read more »

mk

unread,
Jan 14, 2009, 9:00:13 AM1/14/09
to pylons-...@googlegroups.com
Tycon wrote:
> I installed modwsgi 2.3 and apache worker MPM and performance improved
> by 30% but CherryPy is still significantly faster.

Formally speaking, there's also event-driven Apache hack available:

http://packages.ubuntu.com/intrepid/apache2-mpm-event

Although Uber-Guru says "premature optimization is the root of all
evil". :-)

Regards,
mk

Jorge Vargas

unread,
Jan 14, 2009, 11:46:39 AM1/14/09
to pylons-...@googlegroups.com
You are simply FUD. You are cherrypicking on which posts to reply to,
why didn't you reply to Graham's point regarding lazy loading? it
almost seems like you want to make mod_wsgi bad...

Using apache has always been about optimizations, I'm certain your
apache process is loading a ton of stuff, if I recall correctly
ubuntu's package will enable all modules installed by everyone so if
you have say wordpress on that machine, even if it's a testing site
you ones added. That means your apache is loading mod_php as part of
your "benchmark".

Bottom line is, you keep comparing a sub-optimal mod_wsgi installation
with a perfect CP deploy, and say it's faster. Maybe faster for the
unlearned to deploy but not faster to the CPU to run it.

Oh and before you say it I'm no where near close to an apache expert,
so I'm probably going to come up with very suboptimal apache installs,
that said I'm aware it is my fault rather than the package.

So I'm going to stop reading your posts regarding this, be happy with
your CP + reverse_proxy, I have used that some times. it really
depends on your taste and all 3 solutions are good enough. Your
bottleneck is going to be the DB anyway, it always is. Because of one
simple fact, disk I/O operations are the most expensive thing (in time
of course) in Computers Today.

Wyatt Baldwin

unread,
Jan 14, 2009, 12:56:04 PM1/14/09
to pylons-discuss
On Jan 14, 8:46 am, Jorge Vargas <jorge.var...@gmail.com> wrote:
> You are simply FUD.

Any relation to Elmer FUD?

(Sorry, I couldn't help it.)


> You are cherrypicking on which posts to reply to,
> why didn't you reply to Graham's point regarding lazy loading? it
> almost seems like you want to make mod_wsgi bad...
>
> Using apache has always been about optimizations, I'm certain your
> apache process is loading a ton of stuff, if I recall correctly
> ubuntu's package will enable all modules installed by everyone so if
> you have say wordpress on that machine, even if it's a testing site
> you ones added. That means your apache is loading mod_php as part of
> your "benchmark".
>
> Bottom line is, you keep comparing a sub-optimal mod_wsgi installation
> with a perfect CP deploy, and say it's faster. Maybe faster for the
> unlearned to deploy but not faster to the CPU to run it.
>
> Oh and before you say it I'm no where near close to an apache expert,
> so I'm probably going to come up with very suboptimal apache installs,
> that said I'm aware it is my fault rather than the package.
>
> So I'm going to stop reading your posts regarding this, be happy with
> your CP + reverse_proxy, I have used that some times. it really
> depends on your taste and all 3 solutions are good enough. Your
> bottleneck is going to be the DB anyway, it always is. Because of one
> simple fact, disk I/O operations are the most expensive thing (in time
> of course) in Computers Today.
>
> ...
>
> read more »

Graham Dumpleton

unread,
Jan 14, 2009, 5:35:27 PM1/14/09
to pylons-discuss


On Jan 14, 9:21 pm, Tycon <adie...@gmail.com> wrote:
> I installed modwsgi 2.3 and apache worker MPM and performance improved
> by 30% but CherryPy is still significantly faster.

Which indicates that you definitely have something not right with your
configuration or how you are running your tests.

This is because the improvement in 2.X only applies to daemon mode. In
embedded mode you should see about the same performance, irrespective
of whether you switch form prefork to worker MPM. Even if you did
switch to daemon mode, it can't be faster than embedded mode of 1.3.

The only way that performance could improve is that the preloading
issue I talked about was a factor, but if you haven't done anything
about that, although worker may perform a bit better than prefork,
your benchmarking would still appear to be out.

Such variability simply calls into question your results and
conclusions you are coming to.

Graham

> On Jan 13, 10:58 pm, Jorge Vargas <jorge.var...@gmail.com> wrote:
>
> > On Wed, Jan 14, 2009 at 12:54 AM, Jorge Vargas <jorge.var...@gmail.com> wrote:
> > > On Tue, Jan 13, 2009 at 11:51 PM, Tycon <adie...@gmail.com> wrote:
>
> > >> Im using Ubunto 8.04 LTS and running the test using ab -n 10000 -c 10
> > >> to localhost, in order to focus on the request handling code in the
> > >> server stack and remove or reduce the weight of all other application
> > >> logic such as db access and page rendering. This is not supposed to be
> > >> a realistic timing of a real world request, but a highlight of the
> > >> differences in efficiency of different request handling stacks
> > >> including network layer, HTTP and routing code.
>
> > >> All software package veriosns I'm using are the versions supplied in
> > >> the OFFICIAL repositores for the distribution. Pylons and cherrypy
> > >> were installed using (non)easy-install so their version is the latest
> > >> rather than a version tied to any official repository.
>
> > >http://code.google.com/p/modwsgi/downloads/listdoesn'teven lists 1.x anymore
> ...
>
> read more »

Graham Dumpleton

unread,
Jan 14, 2009, 5:54:14 PM1/14/09
to pylons-discuss


On Jan 15, 12:36 am, Tycon <adie...@gmail.com> wrote:
> I specifically meant modwsgi's daemon mode is a hack, which is like I
> explained a proprietary

The wire protocol used is actually only a slight variation on the wire
protocol used by SCGI. The differences are that there is an initial
handshaking to determine if daemon process feels it needs to restart
before handling the request, and that header length information is as
binary byte values rather than ascii, so that it can be decoded more
efficiently.

As mentioned by someone else the wire protocol is irrelevant as what
is important is the WSGI API calling interface. As such daemon mode
should be seen as a black box, the internals of which you shouldn't
need to care about.

> unreliable

I have not seen you report any bugs to the mod_wsgi list or issue
tracker detailing how it is failing and neither has anyone else for
that matter.

The only known issue with the way daemon mode communication works is a
blocking problem with large posts greater than system UNIX socket
buffer size when content not consumed by application and it then in
turn sends a response greater than UNIX socket buffer size. This
problem also affects mod_scgi, mod_cgi, mod_cgid and theoretically in
some configurations mod_proxy (from reading source code). In case of
mod_proxy it probably is hard to trigger as it is using INET sockets
and not UNIX sockets.

Only platform this issue really affects is those with very small
default UNIX socket buffer size like MacOSX. To cope with this there
are configuration options in mod_wsgi to increase the buffer sizes.
There are also socket timeouts to detect the situation and recover
from it automatically. In practice, no one has been reporting the
problem in real installations.

> and non scalable communication
> protocol.

As I explained previously and which you are ignoring, horizontal
scaling is performed exactly like you would with any web server. You
stick a proxy/load balancer in front and spread everything across
multiple machines in a cluster.

There is actually specific code in mod_wsgi to deal with this
situation for case where each machine only has a single daemon
process, but many processes across a cluster. The feature is a way of
ensuring that wsgi.multiprocess flag is set correctly to True rather
than False as would be case when only one machine on a process. For
other WSGI hosting mechanisms have seen, not sure they have a way of
overriding this and so each machine thinks it has the only process,
which is wrong. If applications rely on wsgi.multiprocess they could
then behave incorrectly. Only answer in that case is to hack a WGSI
middleware into stack that overrides it.

BTW, technically FASTCGI is not faster. This is because with FASTCGI
you have to use flup code. Being Python code that slows things down
much more than the pure C code in the mod_wsgi daemon process doing
the same thing.

> As for embedded mode, it's not a hack but as it turns out its
> performance isn't all that great given that it under-performs a pure
> python HTTP server like the one in CherryPy.

Which is still debatable, plus have already pointed out, the
differences get swallowed up as noise in overall request time as this
isn't where your bottleneck is going to be.

As I pointed out right at the beginning. Even when you try and correct
the misinformation that people put out, they still don't change their
tune. As such, this is all the time I am going to waste on this.

I will though do again my own benchmarking. Maybe CherryPy WSGI server
has improved since last time I looked. Either way I am not going to be
too concerned as I am not out to make Apache/mod_wsgi the fastest
thing in the world, nor am I expecting everyone to use it. I work on
it purely because I find it an interesting thing to work on. So, as I
always say, use whatever you feel like using, but if you are going to
try and discredit other options, please back it up with real evidence
in the way of actual benchmark data and configuration information,
which you still haven't done so.

Graham

mk

unread,
Jan 16, 2009, 7:50:26 AM1/16/09
to pylons-...@googlegroups.com
Mike Orr wrote:

> Those are just toy sites for a hobbyist programming language, not big
> mean serious e-commerce servers. And they're so inefficient they must
> waste a huge number of bogomips!

Somewhat off topic, where I work (software group at big IT corporation)
there is lots and lots of resistance towards dynamic typing. It *is*
felt that dynamic typing, even in the model of strong dynamic typing of
Python, belongs precisely to the realm of "hobbyist languages like PHP".

I've heard loads and loads of argumentation from software architects how
(specifically, Java) static typing supposedly saves programmer from
creating lots of bugs and how dynamic languages like Python are "for kids".

And Websphere gives me a stiff left buttock.

Regards,
mk

Mike Orr

unread,
Jan 16, 2009, 3:45:10 PM1/16/09
to pylons-...@googlegroups.com

And Iraq had WMDs too. There is now a significant amount of
established Python software (mostly in websites and niche products)
that can be compared directly to its Java/C counterparts. You would
expect an inferior, bug-prone language to require more development and
maintenance resources compared to similar Java/C programs, yet the
opposite is the case.

And where are the hordes of bugs that strong typing prevents? Static
typing affects only a certain class of bugs, the confusion of one
variable with another. But bugs related to passing the wrong variable
to a function are not that common. Much more common is a variable
that's unexpectedly None/null, but here Java behaves exactly like
Python does. In web applications, a small number of errors can be
traced to integers vs numeric strings, which strong typing would help,
but this has to be weighed against the significant burden of adding
typedefs everywhere and not being able to use flexible-type
constructs.

But of course, none of this will convince the die-hards. Yet there
are an increasing number of programmers who use both static- and
dynamic-typed languages.

A stronger argument for static typing is its easier convertability to
machine language.

--
Mike Orr <slugg...@gmail.com>

Christopher Barker

unread,
Jan 23, 2009, 12:58:49 PM1/23/09
to pylons-...@googlegroups.com
mk wrote:
> Somewhat off topic, where I work (software group at big IT corporation)
> there is lots and lots of resistance towards dynamic typing. It *is*
> felt that dynamic typing, even in the model of strong dynamic typing of
> Python, belongs precisely to the realm of "hobbyist languages like PHP".
>
> I've heard loads and loads of argumentation from software architects how
> (specifically, Java) static typing supposedly saves programmer from
> creating lots of bugs and how dynamic languages like Python are "for kids".

A while back, I was digging into the Python vs. Java question, in
response to someone that proposed Java for our web dev.

I came to the conclusion, like the above, that the real question is
dynamic vs. static typing. Beyond that, it's really taste (Ruby vs.
Python -- which syntax do you like better?)

Anyway, as I trolled the web, reading the arguments on both sides, I
came to the following conclusion:

Static vs. Dynamic typing is a trade off between safety and
productivity. Being a fan of dynamic typing, I think that the static
typing fans both:

1) Underestimate the productivity gains of dynamic typing.
After all, how long does it take to type a few lines of type
declarations? Indeed, if you write Python like Java, you don't get a lot
of productivity gains. A C++ fan I work with once said: "Python is like
C++ with some nice built-in container classes". If that's how you think
about dynamic typing, no wonder you don't want to give up the safety.


2) Over-estimate the safety you get from static typing.
I think this is because most of them haven't REALLY used a dynamic
language. However, many of them have used a weakly typed language: C
with type casted pointers, Fortran with no compiler type checking (what
a nightmare that was!). These are quite different than a strong but
dynamically typed language.

Also, static typing does let the compiler catch a LOT of bugs -- most
programmers find the compiler catching type bugs frequently. The problem
is that they assume that if the compiler didn't catch those bugs, then
they'd remain in the code. However, with a dynamic language, two things
happen:
a) they aren't bugs -- so you passed a tuple in instead of a list --
so what? it's a sequence anyway.
b) they are bugs, but you catch them the first time you run the code.

It doesn't really matter how many bugs are caught in the first draft of
a piece of code, the bugs that are a problem are the tricky, hard to
find ones, and those are very rarely type bugs. They are logic bugs, or
edge-case bugs, etc. The only way you're going to catch those is good
testing. Period.

If a given path through the code has not been tested, then you don't
know if it's correct -- it helps very, very, little if the compiler has
told you that the types are correct.

Another observation I made when researching this issue is that there are
a few studies that tried to objectively assess programmer productivity.
These all concluded that there is greater product ivy and fewer bugs
with dynamic languages. These studies are few, limited and flawed, but
it's what we have. There are also all the anecdotes, and this is what
I've found interesting:

There are lots of anecdotes from folks proclaiming that they found
themselves far more productive using a dynamic rather than a static
language. However, I couldn't find a SINGLE anecdote that demonstrated
the advantages of static languages, and/or the dangers of dynamic ones.
Not one -- lots of folks talking about how they are afraid of the bugs
that would sneak in, but no stories of actual type error that snuck in
and caused a dangerous, hard to find bug.

If anyone has seen such anecdotes -- please post links -- I'd like to
know, I feel like I haven't given it a fair shake.

-Chris


--
Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR&R (206) 526-6959 voice
7600 Sand Point Way NE (206) 526-6329 fax
Seattle, WA 98115 (206) 526-6317 main reception

Chris....@noaa.gov

Reply all
Reply to author
Forward
0 new messages