That's where META PACKAGE definition does not apply to Pylons. Pylons
has source code, glue code and utility code. Hence is hardly just a
meta package. Yes, you are not constrained to use the default
packages, but that's not being a meta package, that's being a
pluggable architecture.
Debian metapackages are just a list of dependencies to install and a label
> The question is what does pylons add of its own to make it more than a
> meta-package (which is just a list of dependencies). It has a bunch of
> useful decorators - https, beaker_cache, validate - but they are all
> so flawed (https throws away query params, beaker cant handle
> memcache, validate has flawed design and produces invalid html) that I
> had to write my own.
It's also a series of variables and objects you use during the
development phase. I don't use those decorators myself but it's not
true that beaker can't handle memcached
At least not the version I use ;-)
> I considered the basic strength to be the HTTP server which comes from
> Paste which I guess together with some other sub-modules provides all
> the HTTP request processing. But then I found out about CherryPy 3 and
> confirmed it gives much better performance and a dispatching
> architecture, so what exactly is left in pylons ?
Paste and CP3 are not meant to be used in the deployment phase. I
really don't care about who's fast during development mode (either
Paste or CP3 or something else), I do care that mod_wsgi in Apache or
nginx or lighttpd are solid.
> Maybe Routes, but if the only route you need is "controller/action/
> id" (which I think CherryPy supports) dispatch is enough, then we
> don't even need routes fancy (and most likely wasteful and expensive)
> mapping.
> Mako, SQLAlchemy, and FormEncode are all great but can be used
> independently.
Yes, you can use pure WSGI too instead of Pylons
I'm losing you here, what are you trying to prove?
--
Lawrence, http://oluyede.org - http://twitter.com/lawrenceoluyede
"It is difficult to get a man to understand
something when his salary depends on not
understanding it" - Upton Sinclair
Pylons is everything that's in the Pylons package itself rather than
in dependencies. The essential part is the PylonsApp class and the
application template. The decorators are peripheral; they were just
thrown in as convenience tools.
Pylons' philosophy is to use third-party distributions as much as
possible. But certain features were not available in any third-party
distribution, so they had to be written from scratch. A Routes-based
dispatcher, for instance. A fully WSGI request manager.
>> I considered the basic strength to be the HTTP server which comes from
>> Paste which I guess together with some other sub-modules provides all
>> the HTTP request processing. But then I found out about CherryPy 3 and
>> confirmed it gives much better performance and a dispatching
>> architecture, so what exactly is left in pylons ?
>
> Paste and CP3 are not meant to be used in the deployment phase. I
> really don't care about who's fast during development mode (either
> Paste or CP3 or something else), I do care that mod_wsgi in Apache or
> nginx or lighttpd are solid.
>
>> Maybe Routes, but if the only route you need is "controller/action/
>> id" (which I think CherryPy supports) dispatch is enough, then we
>> don't even need routes fancy (and most likely wasteful and expensive)
>> mapping.
>> Mako, SQLAlchemy, and FormEncode are all great but can be used
>> independently.
You can use CherryPy with Pylons. It's just a small change in the
[server] section of the ini file, which you can find in the list
archive and maybe on the wiki. Pylons-on-CherryPy is reported to be
better in high-traffic deployments than Pylons-on-PasteHTTPServer.
(Although mod_wsgi may be faster still?)
If you don't want Routes, plain CherryPy may be a good alternative to
Pylons. You can certainly use Mako, SQLAlchemy, and FormEncode with
it.
The "Pylons Execution Analysis" may help explain what's going on and
which module is responsible for what. (It was written for 0.9.6 so a
few things like templating have changed, but most of it is still
accurate.)
http://docs.pythonweb.org/display/pylonscookbook/Pylons+Execution+Analysis+0.9.6
Basically, although the Pylons application *template* is specific to
Pylons, a created application is the user's code, and when it runs
certain portions are not under "Pylons'" control.
So, you run "paster serve", and it reads the config file and
instantiates the server. This is all Paste, not Pylons. The server
may be multithreaded or multiprocess (or mod_wsgi, in which case most
of this is skipped). The application advertises a Setuptools entry
point for myapp.config.middleware:make_app(). Paste uses this to
instantiate the Pylons app.
make_app() -- the users' code -- instantiates a PylonsApp and adds
some middleware around it. There's some trickery in that the
configuration is put into a Pylons config object, and PylonsApp looks
in the config object to find out where the application is. (This was
done to keep boilerplate code in the users' modules to a minimum.)
When a request comes in, Paste calls the PylonsApp as a WSGI
application. The PylonsApp does routing to determine the controller,
instantiates it, and calls it as a WSGI application. The controller
is user code but its base class is Pylons code. The base controller
chooses an action method, looks at its signature to determine which
arguments to call it with, calls it, and incorporates the return value
into the WSGI response. That's about all Pylons does, besides
managing the context variables (request, c, etc).
CherryPy is simpler but less flexible. I"m taking pure CherryPy here,
not Pylons-on-CherryPy. That simplicity may increase runtime speed,
but maybe not enough to be noticed. The Routes overhead is basically
a few method calls. The context variables each add one call per use
plus a couple per request (because they are proxies).
CherryPy 3 does support Routes as an option, not that you would care
since you're trying to avoid overhead.
The main difference between Pylons and CherryPy is philosophical.
CherryPy grafted WSGI and Routes on as options. Pylons was designed
around them in the core, and doesn't include legacy alternatives.
--
Mike Orr <slugg...@gmail.com>
I guess that means routes can be taken out if I remove the routes
middleware and use my own url_for (which I was just about ready to
do).
So in your assessment a simple hello world on pylons using cherrypy
instead of paster (that is my current config) is going to be as fast
as just as an equivalent cherrypy (or pretty close, but not twice as
slow) ?
What Pylons needs is 'wsgiorg.routing_args', 'routes.route', and
'routes.url' in the WSGI environment. In other words, something has
to emulate the RoutesMiddleware, but it does not have to use Routes.
'wsgiorg.routing_args' is used by the Pylons dispatcher. This was
intentionally defined as a neutral standard so that Routes could be
replaced with something else.
The 'routes.url' object is put onto pylons.url. You'll probably have
to provide a dummy object if you don't have a real one, and I imagine
'None' would be OK. url() is called only when the user calls it
(although the user may call it indirectly via redirect_to()).
I'm not sure if Pylons does anything with 'routes.route', but it may
stick it in the config at least, in which case a dummy value would be
needed.
> So in your assessment a simple hello world on pylons using cherrypy
> instead of paster (that is my current config) is going to be as fast
> as just as an equivalent cherrypy (or pretty close, but not twice as
> slow) ?
You're still using paster. You're just replacing PasteHTTPServer with
CherryPy's server component. The CherryPy *server* handles high
traffic better than PasteHTTPServer, according to several user
reports. The difference between a Pylons-CherryPy application vs a
pure CherryPy application is a different question. I would guess the
pure CherryPy application would be slightly more efficient.
But a sizeable application is going to have significant different
stats compared to "Hello World". The overhead of the framework will
be much less than the overhead of the application, so even if one
framework is 10% more efficient, the net difference in application
speed may be only 1%.
> P.S. - mod_wsgi is a hack, it's an unstable and unscalable
> architecture to embed the app server in the web server. And using
> daemon mode makes even less sense as using apache+mod_wsgi as a
> process monitor for the app server (it's better to use a dedicated and
> specialiazed monitoring) as well as a proprietary communication
> protocol when we already have http and fcgi, is silly.
I use Supervisor to manage PasteHTTPServer daemons, and I like having
a HTTP separate daemon for each application so that I can:
- Start and stop them separately ("supervisor restart myapp")
- See their individual memory usage using 'top'.
- Send localhost HTTP requests to them for troubleshooting.
Efficiency is not on my radar because I have plenty of server
capacity. And the efficiency I *do* worry about is large data
structures in memory, which has nothing to do with the deployment
method (except for the impact of multithreaded vs multiprocess). And
if I *did* have to worry about Python overhead so strictly, I'd also
evaluate even more radical choices like not using Python, using a
cloud service, etc.
Fcgi has a long reputation of being buggy. Perhaps it's better now,
but that's why people like me don't use it. SCGI is a *language
neutral* replacement that was written specifically to be simpler and
more reliable. mod_proxy is of course standard. WSGI may be Python
specific, but it does a reasonably good job of making Python
frameworks interchangeable.
--
Mike Orr <slugg...@gmail.com>
I don't even know what you're asking. WSGI is a function signature
and return value, see PEP 333. It doesn't have a daemon mode or even
a server.
PasteHTTPServer is a simple multithreaded server, and that's why it's
the default. It's not meant to be a high-end server for the most
demanding tasks. You can write your own server if you don't approve
of any of the existing ones.
The WSGI environ dict can be pickled and sent via RPC (assuming you
use StringIO objects rather than real file handles). But you can't
pickle a function or iterator, so you can't wholly do WSGI over RPC.
But that's not its purpose. There are other protocols such as SCGI
and HTTP for talking to other computers. WSGI is intended for
in-process chaining of chunks of Python code that may have been
written by different people.
--
Mike Orr <slugg...@gmail.com>
> Paste and CP3 are not meant to be used in the deployment phase. I
> really don't care about who's fast during development mode (either
> Paste or CP3 or something else), I do care that mod_wsgi in Apache or
> nginx or lighttpd are solid.
Would you recommend using ngxinx + mod_wsgi (as described in following
link) as production configuration?
http://wiki.codemongers.com/NginxNgxWSGIModule
Regards,
mk
Tycon, are you talking about nginx's mod_wsgi or apache's mod_wsgi.
That's two completely different projects IMHO.
As for mod_wsgi on Apache I doubt that it is not scalable. At least it
should be as much scalable as Apache. And it is not hack it is WSGI
implementation for Apache. Graham could answer better here.
BTW, I recommend using Apache with mod_wsgi. I'm running 6 pylons
applications under Apache with mod_wsgi + MySql and I fit into 256 Mb
of RAM and it runs really fast. From other side I don't have high
loads and don't expect that soon.
--
Dalius
http://blog.sandbox.lt
There are a lot of opinions being thrown about as if they're the
absolute truth, and I'm afraid it may be confusing people. I can't
speak officially for Pylons (only Ben can), but I can say that Pylons
has been deployed on PasteHTTPServer, CherryPy, mod_wsgi, mod_proxy,
nginx, and others, and all have been stable and are reasonable
choices. It's not true that "embedded mode is bad", "daemon mode is
even more stupid", or "Paste and CP3 are not meant to be used in the
deployment phase" (especially CP3). They may be too underperformant
for certain situations, but that does not make them bad across the
board even if certain people think so.
The argument against PasteHTTPServer and CherryPy3 seems to be
efficiency. The arguments against mod_wsgi and daemon mode seem to be
ideological. (And I'm lost now: if you don't have mod_wsgi and you
don't have an application daemon, what other choice is there?)
--
Mike Orr <slugg...@gmail.com>
I honestly don't know anything about nginx, altough I know personally
the author of its mod_wsgi. Never used. We use modwsgi for Apache
Yeah, like saying that modwsgi + Apache are bad...
> I can't
> speak officially for Pylons (only Ben can), but I can say that Pylons
> has been deployed on PasteHTTPServer, CherryPy, mod_wsgi, mod_proxy,
> nginx, and others, and all have been stable and are reasonable
> choices.
Yes I'm aware of that, I did use PasteHTTPServer in production for a
while. What I was trying to say is that there are better options if we
talk about scaling or performances (what Tycon was talking about all
along)
> It's not true that "embedded mode is bad", "daemon mode is
> even more stupid", or "Paste and CP3 are not meant to be used in the
> deployment phase" (especially CP3). They may be too underperformant
> for certain situations, but that does not make them bad across the
> board even if certain people think so.
I don't think they are bad, that's for sure
> The argument against PasteHTTPServer and CherryPy3 seems to be
> efficiency. The arguments against mod_wsgi and daemon mode seem to be
> ideological. (And I'm lost now: if you don't have mod_wsgi and you
> don't have an application daemon, what other choice is there?)
Asynchronous web servers I guess
Does Pylons even work with asynchronous servers? I guess as long as
you don't have a database.
Last I heard, Twisted ran WSGI applications in a thread because they
might block.
--
Mike Orr <slugg...@gmail.com>
Don't think so. We should ask Manlio Perillo, nginx's modwsgi author,
you can find him on the web-sig ML
That's the response on what you actually use in production, thanks. :-)
Regards,
mk
Is *that* what you're talking about when you say "daemon mode" and
"proprietary protocol". I thought you meant daemon mode as in running
PasteHTTPServer or CherryPy as a daemon, and proprietary protocol as
in WSGI or SCGI.
The main point of mod_wsgi's daemon mode is to isolate bugs/memory
leaks between the web application and the server, and to track the
application's individual resource usage in the 'ps' listing. It's not
designed for multi-machine scalability.
As for its "proprietary" protocol, I consider that an internal matter
of mod_wsgi. What matters is whether it works, and I haven't heard
any complaints in that regard.
Ultimately it comes down to the sysadmin's time of setting up mod_wsgi
now and possibly switching to something else later, vs setting up
something multi-machine scalable now (which is more work up front).
And that depends on how likely a traffic onslaught is, how quickly the
load will accelerate, and the sysadmin's future availability.
--
Mike Orr <slugg...@gmail.com>
hi Ian, is this explanation somewhere in pythonpaste? or somewhere
else? I see this being asked a lot in several places (irc, other
mailing lists,etc) I'll like to have a link to point to.
wsgi != mod_wsgi.
WSGI is a python protocol for app-app communication and contrary to
proprietary it is today probably the most used "standard" in python's
web development.
you are right no self respecting site will use that piece of junk!
although I saw this email some time ago
http://groups.google.com/group/modwsgi/browse_thread/thread/88de3e07ea574ddb/95612a8e94613bf7
so you may reconsider that statement given this line.
"Apache/2.2.9 (Debian) mod_python/3.3.1 Python/2.5.2 mod_wsgi/2.3
Server at pypi.python.org Port 80"
I'm sorry but that isn't a viable testing environment. template
rendering and db access are the most time consuming tasks pylons will
undertake. On the other hand you never pointed out how many request
per second are you sending to the system.
Those are just toy sites for a hobbyist programming language, not big
mean serious e-commerce servers. And they're so inefficient they must
waste a huge number of bogomips!
--
Mike Orr <slugg...@gmail.com>
Formally speaking, there's also event-driven Apache hack available:
http://packages.ubuntu.com/intrepid/apache2-mpm-event
Although Uber-Guru says "premature optimization is the root of all
evil". :-)
Regards,
mk
> Those are just toy sites for a hobbyist programming language, not big
> mean serious e-commerce servers. And they're so inefficient they must
> waste a huge number of bogomips!
Somewhat off topic, where I work (software group at big IT corporation)
there is lots and lots of resistance towards dynamic typing. It *is*
felt that dynamic typing, even in the model of strong dynamic typing of
Python, belongs precisely to the realm of "hobbyist languages like PHP".
I've heard loads and loads of argumentation from software architects how
(specifically, Java) static typing supposedly saves programmer from
creating lots of bugs and how dynamic languages like Python are "for kids".
And Websphere gives me a stiff left buttock.
Regards,
mk
And Iraq had WMDs too. There is now a significant amount of
established Python software (mostly in websites and niche products)
that can be compared directly to its Java/C counterparts. You would
expect an inferior, bug-prone language to require more development and
maintenance resources compared to similar Java/C programs, yet the
opposite is the case.
And where are the hordes of bugs that strong typing prevents? Static
typing affects only a certain class of bugs, the confusion of one
variable with another. But bugs related to passing the wrong variable
to a function are not that common. Much more common is a variable
that's unexpectedly None/null, but here Java behaves exactly like
Python does. In web applications, a small number of errors can be
traced to integers vs numeric strings, which strong typing would help,
but this has to be weighed against the significant burden of adding
typedefs everywhere and not being able to use flexible-type
constructs.
But of course, none of this will convince the die-hards. Yet there
are an increasing number of programmers who use both static- and
dynamic-typed languages.
A stronger argument for static typing is its easier convertability to
machine language.
--
Mike Orr <slugg...@gmail.com>
A while back, I was digging into the Python vs. Java question, in
response to someone that proposed Java for our web dev.
I came to the conclusion, like the above, that the real question is
dynamic vs. static typing. Beyond that, it's really taste (Ruby vs.
Python -- which syntax do you like better?)
Anyway, as I trolled the web, reading the arguments on both sides, I
came to the following conclusion:
Static vs. Dynamic typing is a trade off between safety and
productivity. Being a fan of dynamic typing, I think that the static
typing fans both:
1) Underestimate the productivity gains of dynamic typing.
After all, how long does it take to type a few lines of type
declarations? Indeed, if you write Python like Java, you don't get a lot
of productivity gains. A C++ fan I work with once said: "Python is like
C++ with some nice built-in container classes". If that's how you think
about dynamic typing, no wonder you don't want to give up the safety.
2) Over-estimate the safety you get from static typing.
I think this is because most of them haven't REALLY used a dynamic
language. However, many of them have used a weakly typed language: C
with type casted pointers, Fortran with no compiler type checking (what
a nightmare that was!). These are quite different than a strong but
dynamically typed language.
Also, static typing does let the compiler catch a LOT of bugs -- most
programmers find the compiler catching type bugs frequently. The problem
is that they assume that if the compiler didn't catch those bugs, then
they'd remain in the code. However, with a dynamic language, two things
happen:
a) they aren't bugs -- so you passed a tuple in instead of a list --
so what? it's a sequence anyway.
b) they are bugs, but you catch them the first time you run the code.
It doesn't really matter how many bugs are caught in the first draft of
a piece of code, the bugs that are a problem are the tricky, hard to
find ones, and those are very rarely type bugs. They are logic bugs, or
edge-case bugs, etc. The only way you're going to catch those is good
testing. Period.
If a given path through the code has not been tested, then you don't
know if it's correct -- it helps very, very, little if the compiler has
told you that the types are correct.
Another observation I made when researching this issue is that there are
a few studies that tried to objectively assess programmer productivity.
These all concluded that there is greater product ivy and fewer bugs
with dynamic languages. These studies are few, limited and flawed, but
it's what we have. There are also all the anecdotes, and this is what
I've found interesting:
There are lots of anecdotes from folks proclaiming that they found
themselves far more productive using a dynamic rather than a static
language. However, I couldn't find a SINGLE anecdote that demonstrated
the advantages of static languages, and/or the dangers of dynamic ones.
Not one -- lots of folks talking about how they are afraid of the bugs
that would sneak in, but no stories of actual type error that snuck in
and caused a dangerous, hard to find bug.
If anyone has seen such anecdotes -- please post links -- I'd like to
know, I feel like I haven't given it a fair shake.
-Chris
--
Christopher Barker, Ph.D.
Oceanographer
Emergency Response Division
NOAA/NOS/OR&R (206) 526-6959 voice
7600 Sand Point Way NE (206) 526-6329 fax
Seattle, WA 98115 (206) 526-6317 main reception