Google Groups no longer supports new Usenet posts or subscriptions. Historical content remains viewable.
Dismiss

Python Web Programming - looking for examples of solid high-traffic sites

13 views
Skip to first unread message

Victor Kryukov

unread,
May 16, 2007, 5:04:17 PM5/16/07
to
Hello list,

our team is going to rewrite our existing web-site, which has a lot of
dynamic content and was quickly prototyped some time ago.

Today, as we get better idea of what we need, we're going to re-write
everything from scratch. Python is an obvious candidate for our team:
everybody knows it, everybody likes it, it has *real* objects, nice
clean syntax etc.

Our main requirement for tools we're going to use is rock-solid
stability. As one of our team-members puts it, "We want to use tools
that are stable, has many developer-years and thousands of user-years
behind them, and that we shouldn't worry about their _versions_." The
main reason for that is that we want to debug our own bugs, but not
the bugs in our tools.

Our problem is - we yet have to find any example of high-traffic,
scalable web-site written entirely in Python. We know that YouTube is
a suspect, but we don't know what specific python web solution was
used there.

TurboGears, Django and Pylons are all nice, and provides rich features
- probably too many for us - but, as far as we understand, they don't
satisfy the stability requirement - Pylons and Django hasn't even
reached 1.0 version yet. And their provide too thick layer - we want
something 'closer to metal', probably similar to web.py -
unfortunately, web.py doesn't satisfy the stability requirement
either, or so it seems.

So the question is: what is a solid way to serve dynamic web pages in
python? Our initial though was something like python + mod_python +
Apache, but we're told that mod_python is 'scary and doesn't work very
well'.

And although http://www.python.org/about/quotes/ lists many big names
and wonderful examples, be want more details. E.g. our understanding
is that Google uses python mostly for internal web-sites, and
performance is far from perfect their. YouTube is an interesting
example - anybody knows more details about that?

Your suggestions and comments are highly welcome!

Best Regards,
Victor.

John Nagle

unread,
May 16, 2007, 10:53:40 PM5/16/07
to
Victor Kryukov wrote:
> Hello list,
>
> our team is going to rewrite our existing web-site, which has a lot of
> dynamic content and was quickly prototyped some time ago.
...

> Our main requirement for tools we're going to use is rock-solid
> stability. As one of our team-members puts it, "We want to use tools
> that are stable, has many developer-years and thousands of user-years
> behind them, and that we shouldn't worry about their _versions_." The
> main reason for that is that we want to debug our own bugs, but not
> the bugs in our tools.

You may not be happy with Python, then.

Having spent the last several months implementing a reasonably complex
web site in Python, I now have an understanding of the problems involved.
Which are non-trivial.

Some key web site components, like the SSL interface and the
MySQL interface, are written in C and maintained by third parties,
often by a single person. Getting the correct version for your
platform and making it work can be difficult. Getting the right
versions of MySQL, OpenSSL, and Python to all play together is
non-trivial. Expect to have to build from source, debug the build
process, look at source repositories, submit bug reports, fix
library bugs yourself, and maintain a local set of library patches.

Few hosting companies will have these modules available for you.
It's not like Perl or PHP, where it works out of the box. WebFaction
claims to support Python well, but few other hosting companies bother.
Most Linux distributions ship with older versions of Python, and
that's what most hosting companies will give you. Python 2.4
is par for the course.

High traffic sites are a problem. There are a number of
"frameworks", all supported by different people and with different
capabilities. See

http://www.polimetrix.com/pycon/slides/

for a discussion. I haven't tried most of them, so I won't
say anything about that issue.

> Our problem is - we yet have to find any example of high-traffic,
> scalable web-site written entirely in Python. We know that YouTube is
> a suspect, but we don't know what specific python web solution was
> used there.

Not that much of YouTube is really in Python. The video
codecs aren't; they'd take forever if they were. And since Google
took over, the old YouTube search engine (which was terrible) was
replaced by Google's search technology.


John Nagle

Message has been deleted
Message has been deleted

Luis M. González

unread,
May 17, 2007, 12:03:27 AM5/17/07
to
> And althoughhttp://www.python.org/about/quotes/lists many big names

> and wonderful examples, be want more details. E.g. our understanding
> is that Google uses python mostly for internal web-sites, and
> performance is far from perfect their. YouTube is an interesting
> example - anybody knows more details about that?
>
> Your suggestions and comments are highly welcome!
>
> Best Regards,
> Victor.


Reddit.com was built with webpy, and it's amongst the top 1000
websites in volume of traffic.
So if you liked it, go with it...
I believe one of the authors of reddit is also webpy's developer.

Luis

Istvan Albert

unread,
May 17, 2007, 10:10:25 AM5/17/07
to
On May 16, 5:04 pm, Victor Kryukov <victor.kryu...@gmail.com> wrote:

> Our main requirement for tools we're going to use is rock-solid
> stability. As one of our team-members puts it, "We want to use tools
> that are stable, has many developer-years and thousands of user-years
> behind them, and that we shouldn't worry about their _versions_." The
> main reason for that is that we want to debug our own bugs, but not
> the bugs in our tools.

I think this is a requirement that is pretty much impossible to
satisfy. Only dead frameworks stay the same. I have yet to see a
framework that did not have incompatible versions.

Django has a very large user base, great documentation and is deployed
for several online new and media sites. It is fast, it's efficient and
is simple to use. Few modern frameworks (in any language) are
comparable, and I have yet to see one that is better,

http://code.djangoproject.com/wiki/DjangoPoweredSites

i.

Jarek Zgoda

unread,
May 17, 2007, 10:21:29 AM5/17/07
to
Victor Kryukov napisał(a):

> Our main requirement for tools we're going to use is rock-solid
> stability. As one of our team-members puts it, "We want to use tools
> that are stable, has many developer-years and thousands of user-years
> behind them, and that we shouldn't worry about their _versions_." The
> main reason for that is that we want to debug our own bugs, but not
> the bugs in our tools.

I don't think you find anything even remotely resembling that idea here.
Moreover, I don't think you find it elsewhere. Maybe even such tools do
not exist in nature?

--
Jarek Zgoda

"We read Knuth so you don't have to."

Steve Holden

unread,
May 17, 2007, 10:38:56 AM5/17/07
to pytho...@python.org
Victor Kryukov wrote:
> Hello list,
>
> our team is going to rewrite our existing web-site, which has a lot of
> dynamic content and was quickly prototyped some time ago.
>
And has stayed around to dog the developers, as so many quick fixes do ...

> Today, as we get better idea of what we need, we're going to re-write
> everything from scratch. Python is an obvious candidate for our team:
> everybody knows it, everybody likes it, it has *real* objects, nice
> clean syntax etc.
>

Yes indeedy, that's our language!

> Our main requirement for tools we're going to use is rock-solid
> stability. As one of our team-members puts it, "We want to use tools
> that are stable, has many developer-years and thousands of user-years
> behind them, and that we shouldn't worry about their _versions_." The
> main reason for that is that we want to debug our own bugs, but not
> the bugs in our tools.
>

Given that even Apache has bugs despite its venerable status I think you
are setting your sights a little high here.

> Our problem is - we yet have to find any example of high-traffic,
> scalable web-site written entirely in Python. We know that YouTube is
> a suspect, but we don't know what specific python web solution was
> used there.
>

Zope? Plone? Django? TurboGears? All are handling large volumes of data
on a daily basis, and I wouldn't use either of them (but that's just a
personal issue).

> TurboGears, Django and Pylons are all nice, and provides rich features
> - probably too many for us - but, as far as we understand, they don't
> satisfy the stability requirement - Pylons and Django hasn't even
> reached 1.0 version yet. And their provide too thick layer - we want
> something 'closer to metal', probably similar to web.py -
> unfortunately, web.py doesn't satisfy the stability requirement
> either, or so it seems.
>
> So the question is: what is a solid way to serve dynamic web pages in
> python? Our initial though was something like python + mod_python +
> Apache, but we're told that mod_python is 'scary and doesn't work very
> well'.
>

Python CGI? mod_python is OK, but like all frameworks you have to be
aware of its limitations.

> And although http://www.python.org/about/quotes/ lists many big names
> and wonderful examples, be want more details. E.g. our understanding
> is that Google uses python mostly for internal web-sites, and
> performance is far from perfect their. YouTube is an interesting
> example - anybody knows more details about that?
>

Google use Python for all sorts of stuff.

> Your suggestions and comments are highly welcome!
>
> Best Regards,
> Victor.
>

I think you are chasing a chimera (and I wrote a *book* called "Python
Web Programming").

regards
Steve
--
Steve Holden +1 571 484 6266 +1 800 494 3119
Holden Web LLC/Ltd http://www.holdenweb.com
Skype: holdenweb http://del.icio.us/steve.holden
------------------ Asciimercial ---------------------
Get on the web: Blog, lens and tag your way to fame!!
holdenweb.blogspot.com squidoo.com/pythonology
tagged items: del.icio.us/steve.holden/python
All these services currently offer free registration!
-------------- Thank You for Reading ----------------

Paul Boddie

unread,
May 17, 2007, 11:35:29 AM5/17/07
to
John Nagle wrote:

> Victor Kryukov wrote:
> >
> > Our main requirement for tools we're going to use is rock-solid
> > stability. As one of our team-members puts it, "We want to use tools
> > that are stable, has many developer-years and thousands of user-years
> > behind them, and that we shouldn't worry about their _versions_." The

When settling on an environment you need to have a clear strategy
about component versions and compatibility. Although Python provides a
fairly reasonable backwards compatibility story, one still needs to be
aware of changes between versions and their implications, but this is
the case for a large proportion of the software produced today.
Despite the baggage maintained by Microsoft so that ancient
applications might still run in recent versions of Windows, to choose
a fairly extreme example in certain respects, there are still many
things to be aware of when maintaining or upgrading your environment.
Saying that something needs "Python plus Windows/Linux plus some
database system" isn't going to be enough if you're emphasizing
stability.

> > main reason for that is that we want to debug our own bugs, but not
> > the bugs in our tools.
>
> You may not be happy with Python, then.
>
> Having spent the last several months implementing a reasonably complex
> web site in Python, I now have an understanding of the problems involved.
> Which are non-trivial.

I'm sure they are, which is why I attempt to use the tools and
services available to reduce the impact of those problems on myself.

> Some key web site components, like the SSL interface and the
> MySQL interface, are written in C and maintained by third parties,
> often by a single person. Getting the correct version for your
> platform and making it work can be difficult. Getting the right
> versions of MySQL, OpenSSL, and Python to all play together is
> non-trivial. Expect to have to build from source, debug the build
> process, look at source repositories, submit bug reports, fix
> library bugs yourself, and maintain a local set of library patches.

I think we went through this before, but to what extent did you rely
on your distribution to handle many of these issues? Or was this on
Windows? If so, you're the master of your own distribution rather than
a group of helpful people who've probably already done most of that
work.

> Few hosting companies will have these modules available for you.
> It's not like Perl or PHP, where it works out of the box. WebFaction
> claims to support Python well, but few other hosting companies bother.
> Most Linux distributions ship with older versions of Python, and
> that's what most hosting companies will give you. Python 2.4
> is par for the course.

There's nothing wrong with Python 2.4. Perhaps you've inadvertently
highlighted an issue with the Python community: that everyone wants
the latest and greatest, and sees the CPython distribution as the
primary means of delivering it. If so, you've just included yourself
in the group causing yourself all those problems described above.

Paul

P.S. The inquirer may wish to visit sites representing the different
frameworks in order to seek testimonials, and to look for sites using
those frameworks in order to assess their popularity. Zope, Plone,
Django, TurboGears, Webware and many others have delivered high-
profile sites at various times, as far as I recall.

Chris Cioffi

unread,
May 17, 2007, 12:07:24 PM5/17/07
to Victor Kryukov, pytho...@python.org
I think the first question I would have is what kind of dynamic
content are you talking about? Is this a web app kind of thing, or
just a content pushing site?

While Django might not be v1.0 yet, it seems very solid and stable,
and perfect for quickly building powerful content based dynamic sites.

Other Python frameworks might be well suited to other types of dynamic
sites. (I like TurboGears for actual web apps...)

Depending on your needs you might also consider a non-python solution
(!) like Drupal. (I'm not a PHP fan, but since other people have done
all that great work....:)

As others have said, there might be no framework currently in
existence that meets all of your requirements. You'll need to work
with your team to decide what kinds of compromises you can all live
with.

Chris
--
"A little government and a little luck are necessary in life, but only
a fool trusts either of them." -- P. J. O'Rourke

John Nagle

unread,
May 17, 2007, 12:35:15 PM5/17/07
to

Sure they do. I have a complex web site, "http://www.downside.com",
that's implemented with Perl, Apache, and MySQL. It automatically reads SEC
filings and parses them to produce financial analyses. It's been
running for seven years, and hasn't been modified in five, except once
when the NASDAQ changed the format of their ticker symbol file.

It's not that useful at this point, because its purpose was to predict
failing dot-coms, but it's still running and doing a sizable amount of
work every day to update itself.

John Nagle

Alex Martelli

unread,
May 18, 2007, 12:22:28 AM5/18/07
to
Victor Kryukov <victor....@gmail.com> wrote:
...

> And although http://www.python.org/about/quotes/ lists many big names
> and wonderful examples, be want more details. E.g. our understanding
> is that Google uses python mostly for internal web-sites, and
> performance is far from perfect their. YouTube is an interesting
> example - anybody knows more details about that?

Hmmm, I do, but exactly because I _do_ know a lot more details I cannot
discuss them unless you're under the proper Non-Disclosure Agreement
with Google, Inc. The best I can do otherwise is to point you to
already existing webpages -- I'm not going to reveal Google-Confidential
information, nor go to the substantial trouble to get such info cleared
by Legal and all other Google stakeholders.

For example, it HAS been published elsewhere that YouTube uses lighttpd,
not Apache: <http://trac.lighttpd.net/trac/wiki/PoweredByLighttpd>.

Fortunately, I managed to convince my YouTube colleagues to publically
present many more details about the evolution of their architecture
which had been discussed in Google-confidential talks and presentations
-- and my wife Anna, who's on the program selection committee of OSCON,
may have helped that talk get accepted (must not have been a hard
job:-). See:
<http://conferences.oreillynet.com/cs/os2007/view/e_sess/13435>

I hope to see you (and anybody else interested in solid technical
details about using Python for websites on YouTube's scale) in Portland,
OR on July 26 -- that will also be your first and best chance to ask
Mike Solomon specific questions that his talk might not directly
address. Once that's done, maybe somebody can convince the YouTube guys
to contribute a "Python Success Story" so that future querants about
this can be easily pointed to a URL!-)

I would also encourage anybody who's so keenly interested in Python to
visit our jobs-listing web app, e.g., if within the US, at
<http://www.google.com/support/jobs/bin/topic.py?loc_id=1100&dep_id=1173
&by_loc=1>. Of course, one should *never* have the implementation
language of a web-app show up in the URL, and I believe we've carefully
avoided that error in other external-facing web-apps, such as (one I can
reveal is indeed in Python, because that was mentioned at
<http://www.sauria.com/~twl/conferences/pycon2005/20050325/Python%20at%2
0Google.html>) code.google.com. Etc, etc.

Performance of web-apps (be they internal or external) depends more on
the architecture than on the implementation language (as long as highly
optimized C or C++, NOT Java or Python or whatever, is used for the few
jobs that are extremely CPU-intensive, such as codecs, of course:-).

So, if some Python-coded internal web-apps at Google perform badly
(which may be the case as you say, though I can't think of any
off-hand), it must be an issue of architecture. For example, a heavily
used internal web-app at Google is Mondrian,
<http://www.niallkennedy.com/blog/archives/2006/11/google-mondrian.html>
, Guido van Rossum's web-app for code review -- it's got a good, solid
architecture, and its performance is so good that many Googlers, me
included, have switched to it for all the reviews we do (and, believe
me, we do MANY -- _nothing_ is submitted to the global code repository
until it's been OK'd in a code review). I can't mention other such apps
because, AFAIK, they haven't been previously talked about in public and
so they're Google Confidential by default until otherwise determined.


Alex

Daniel Nogradi

unread,
May 18, 2007, 6:14:40 AM5/18/07
to pytho...@python.org
> For example, it HAS been published elsewhere that YouTube uses lighttpd,
> not Apache: <http://trac.lighttpd.net/trac/wiki/PoweredByLighttpd>.

How do you explain these, then:

http://www.youtube.com/results.xxx
http://www.youtube.com/results.php
http://www.youtube.com/results.py

Just wondering,
Daniel

Jarek Zgoda

unread,
May 18, 2007, 6:22:40 AM5/18/07
to
Daniel Nogradi napisał(a):

Server signature is usually configurable.

Bruno Desthuilliers

unread,
May 18, 2007, 8:48:42 AM5/18/07
to
John Nagle a écrit :

> Victor Kryukov wrote:
>> Hello list,
>>
>> our team is going to rewrite our existing web-site, which has a lot of
>> dynamic content and was quickly prototyped some time ago.
> ...
>> Our main requirement for tools we're going to use is rock-solid
>> stability. As one of our team-members puts it, "We want to use tools
>> that are stable, has many developer-years and thousands of user-years
>> behind them, and that we shouldn't worry about their _versions_." The
>> main reason for that is that we want to debug our own bugs, but not
>> the bugs in our tools.
>
> You may not be happy with Python, then.

John, I'm really getting tired of your systemic and totally
unconstructive criticism. If *you* are not happy with Python, by all
means use another language.

Bruno Desthuilliers

unread,
May 18, 2007, 8:50:21 AM5/18/07
to
Istvan Albert a écrit :

Then have a look at Pylons.

Alex Martelli

unread,
May 18, 2007, 10:12:06 AM5/18/07
to
Jarek Zgoda <jzg...@o2.usun.pl> wrote:

> Daniel Nogradi napisa?(a):


>
> >> For example, it HAS been published elsewhere that YouTube uses lighttpd,
> >> not Apache: <http://trac.lighttpd.net/trac/wiki/PoweredByLighttpd>.
> >
> > How do you explain these, then:
> >
> > http://www.youtube.com/results.xxx
> > http://www.youtube.com/results.php
> > http://www.youtube.com/results.py
>
> Server signature is usually configurable.

Yeah, but I don't know why it's configured it that way. A good example
of a question that looks perfectly appropriate for YouTube's OSCON
session.


Alex

Daniel Nogradi

unread,
May 18, 2007, 11:00:51 AM5/18/07
to pytho...@python.org
> > >> For example, it HAS been published elsewhere that YouTube uses lighttpd,
> > >> not Apache: <http://trac.lighttpd.net/trac/wiki/PoweredByLighttpd>.
> > >
> > > How do you explain these, then:
> > >
> > > http://www.youtube.com/results.xxx
> > > http://www.youtube.com/results.php
> > > http://www.youtube.com/results.py
> >
> > Server signature is usually configurable.
>
> Yeah, but I don't know why it's configured it that way. A good example
> of a question that looks perfectly appropriate for YouTube's OSCON
> session.

Let us know what they say! :)

John Nagle

unread,
May 18, 2007, 1:44:26 PM5/18/07
to

Denying the existence of the problem won't fix it.

Many of the basic libraries for web related functions do have
problems. Even standard modules like "urllib" and "SSL" are buggy,
and have been for years. Outside the standard modules, it gets
worse, especially for ones with C components. Version incompatibility
for extensions is a serious problem. That's reality.

It's a good language, but the library situation is poor. Python as
a language is better than Perl, but CPAN is better run than Cheese Shop.

As a direct result of this, neither the Linux distro builders like
Red Hat nor major hosting providers provide Python environments that
just work. That's reality.

John Nagle

John Nagle

unread,
May 18, 2007, 2:15:39 PM5/18/07
to

YouTube's home page is PHP. Try "www.youtube.com/index.php".
That works, while the obvious alternatives don't.
If you look at the page HTML, you'll see things like

<a href="/login?next=/index.php"
onclick="_hbLink('LogIn','UtilityLinks');">Log In</a>

So there's definitely PHP inside YouTube.

If you look at the HTML for YouTube pages, there seem to be two
drastically different styles. Some pages begin with "<!-- machid: 169 -->",
and have their CSS stored in external files. Those seem to be generated
by PHP. Other pages start with
"<meta http-equiv="Content-Type" content="text/html; charset=UTF-8">",
with no "machine ID". It looks like the stuff associated with
accounts and logging in is on the second system (Python?) while the
search and view related functions are on the PHP system.

Shortly after Google bought YouTube, they replaced YouTube's search
engine (which was terrible) with one of their own. At that time,
Google search syntax, like "-", started working. That's probably
when the shift to PHP happened.

John Nagle

Daniel Nogradi

unread,
May 18, 2007, 2:13:17 PM5/18/07
to pytho...@python.org
> > >> For example, it HAS been published elsewhere that YouTube uses lighttpd,
> > >> not Apache: <http://trac.lighttpd.net/trac/wiki/PoweredByLighttpd>.
> > >
> > > How do you explain these, then:
> > >
> > > http://www.youtube.com/results.xxx
> > > http://www.youtube.com/results.php
> > > http://www.youtube.com/results.py
> >
> > Server signature is usually configurable.
>
> Yeah, but I don't know why it's configured it that way. A good example
> of a question that looks perfectly appropriate for YouTube's OSCON
> session.

Actually, the fact that http://www.youtube.com/results.php returns
legitimate content might be explainable by the fact that youtube was
started as a PHP app so they might provide this URL for backward
compatibility although today there is no PHP at all. See the abstract
of Mike Solomon's OSCON talk: "YouTube began as a small PHP
application. [...]"
http://conferences.oreillynet.com/cs/os2007/view/e_sess/13435

Michele Simionato

unread,
May 18, 2007, 2:16:17 PM5/18/07
to
On May 16, 11:04 pm, Victor Kryukov <victor.kryu...@gmail.com> wrote:
>
> Our main requirement for tools we're going to use is rock-solid
> stability. As one of our team-members puts it, "We want to use tools
> that are stable, has many developer-years and thousands of user-years
> behind them, and that we shouldn't worry about their _versions_." The
> main reason for that is that we want to debug our own bugs, but not
> the bugs in our tools.
>
> Our problem is - we yet have to find any example of high-traffic,
> scalable web-site written entirely in Python. We know that YouTube is
> a suspect, but we don't know what specific python web solution was
> used there.
>
> TurboGears, Django and Pylons are all nice, and provides rich features
> - probably too many for us - but, as far as we understand, they don't
> satisfy the stability requirement - Pylons and Django hasn't even
> reached 1.0 version yet. And their provide too thick layer - we want
> something 'closer to metal', probably similar to web.py -
> unfortunately, web.py doesn't satisfy the stability requirement
> either, or so it seems.
>
> So the question is: what is a solid way to serve dynamic web pages in
> python? Our initial though was something like python + mod_python +
> Apache, but we're told that mod_python is 'scary and doesn't work very
> well'.

AFAIK mod_python is solid and works well, but YMMV of course.
If you want rock solid stability, you want a framework where there is
little
development going on. In that case, I have a perfect match for your
requirements: Quixote. It has been around for ages, it is the most bug
free framework I have seen and it *very* scalable. For instance
http://www.douban.com
is a Quixote-powered chinese site with more than 2 millions of pages
served per
day. To quote from a message on the Quixote mailing list:

"""
Just to report-in the progress we're making with a real-world Quixote
installation: yesterday douban.com celebrated its first 2 million-
pageview day. Quixote generated 2,058,207 page views. In addition,
there're about 640,000 search-engine requests. These put the combined
requests at around 2.7 millions. All of our content pages are
dynamic, including the help and about-us pages.

We're still wondering if we're the busiest one of all the python/ruby
supported websites in the world.

Quixote runs on one dual-core home-made server (costed us US$1500).
We have three additional servers dedicated to lighttpd and mysql. We
use memcached extensively as well.

Douban.com is the most visible python establishment on the Chinese
web, so there's been quite a few django vs. quixote threads in the
Chinese language python user mailing lists.
"""

Michele Simionato

John Nagle

unread,
May 18, 2007, 2:21:04 PM5/18/07
to
John Nagle wrote:

> YouTube's home page is PHP. Try "www.youtube.com/index.php".
> That works, while the obvious alternatives don't.
> If you look at the page HTML, you'll see things like
>
> <a href="/login?next=/index.php"
> onclick="_hbLink('LogIn','UtilityLinks');">Log In</a>
>
> So there's definitely PHP inside YouTube.

Not sure; that "next" field is just the URL of the page you're on,
inserted into the output HTML. It's "index.php" because the page was
"index.php".

But it's an Apache server, with all the usual Apache messages.

John Nagle

Jorge Godoy

unread,
May 18, 2007, 7:18:51 PM5/18/07
to
John Nagle <na...@animats.com> writes:

> As a direct result of this, neither the Linux distro builders like
> Red Hat nor major hosting providers provide Python environments that
> just work. That's reality.

Try SuSE, OpenSUSE, Ubuntu... They "just work". I've never had any
problem installing any library or module for Python. Even the ones that
require huge libraries or compiling something.

--
Jorge Godoy <jgo...@gmail.com>

Josiah Carlson

unread,
May 18, 2007, 8:48:42 PM5/18/07
to
John Nagle wrote:
> Many of the basic libraries for web related functions do have
> problems. Even standard modules like "urllib" and "SSL" are buggy,
> and have been for years. Outside the standard modules, it gets
> worse, especially for ones with C components. Version incompatibility
> for extensions is a serious problem. That's reality.
>
> It's a good language, but the library situation is poor. Python as
> a language is better than Perl, but CPAN is better run than Cheese Shop.

You know, submitting bug reports, patches, etc., can help make Python
better. And with setuptools' easy_setup, getting modules and packages
installed from the Cheese Shop is pretty painless.

- Josiah

Bruno Desthuilliers

unread,
May 20, 2007, 3:09:05 PM5/20/07
to
John Nagle a écrit :

> Bruno Desthuilliers wrote:
>
>> John Nagle a écrit :
>>
>>> Victor Kryukov wrote:
>>>
>>>> Hello list,
>>>>
>>>> our team is going to rewrite our existing web-site, which has a lot of
>>>> dynamic content and was quickly prototyped some time ago.
>>>
>>>
>>> ...
>>>
>>>> Our main requirement for tools we're going to use is rock-solid
>>>> stability. As one of our team-members puts it, "We want to use tools
>>>> that are stable, has many developer-years and thousands of user-years
>>>> behind them, and that we shouldn't worry about their _versions_." The
>>>> main reason for that is that we want to debug our own bugs, but not
>>>> the bugs in our tools.
>>>
>>>
>>>
>>> You may not be happy with Python, then.
>>
>>
>>
>> John, I'm really getting tired of your systemic and totally
>> unconstructive criticism. If *you* are not happy with Python, by all
>> means use another language.
>
>
> Denying the existence of the problem won't fix it.
>

Neither will keeping on systematically criticizing on this newsgroup
instead of providing bug reports and patches.

> As a direct result of this, neither the Linux distro builders like
> Red Hat nor major hosting providers provide Python environments that
> just work. That's reality.
>

I've been using Python for web applications (Zope, mod_python, fast cgi
etc) on Gentoo and Debian for the 4 or 5 past years, and it works just
fine. So far, I've had much more bugs and compatibility problems with
PHP (4 and 5) than with Python.

Michael Bayer

unread,
May 20, 2007, 7:57:59 PM5/20/07
to
On May 16, 5:04 pm, Victor Kryukov <victor.kryu...@gmail.com> wrote:
>
> Our main requirement for tools we're going to use is rock-solid
> stability. As one of our team-members puts it, "We want to use tools
> that are stable, has many developer-years and thousands of user-years
> behind them, and that we shouldn't worry about their _versions_." The
> main reason for that is that we want to debug our own bugs, but not
> the bugs in our tools.

youre not going to find a web development platform in any language at
all where you will not come across bugs. you will always have to
"worry" about versions. I have a day job where we worry about bugs
and versions in struts, hibernate, mysql, and spring all day long, and
each of those products probably has more users than all python
frameworks combined (feel free to whomever to bring out numbers, id be
glad to be proven wrong).

The web platform for Python which has the longest running time and the
most thousands-of-whatever hours is Zope (and Plone). All the others
which are popular today have only a tiny fraction of the in-production
time that Zope has. so if thats your criterion, then zope is what
you'd probably have to use.

> TurboGears, Django and Pylons are all nice, and provides rich features
> - probably too many for us - but, as far as we understand, they don't
> satisfy the stability requirement - Pylons and Django hasn't even
> reached 1.0 version yet. And their provide too thick layer - we want
> something 'closer to metal', probably similar to web.py -
> unfortunately, web.py doesn't satisfy the stability requirement
> either, or so it seems.

I would seriously reconsider the notion that Pylons is "too thick" of
a layer. Pylons is quite open ended and non-opinionated. the
approaches of Pylons and Django couldnt be more different, so I would
suggest digging a little deeper into the various frameworks before
dismissing them on based on shallow judgments. Also, I understand
reddit is built on web.py, which is pretty high-traffic/proven/etc.

> So the question is: what is a solid way to serve dynamic web pages in
> python? Our initial though was something like python + mod_python +
> Apache, but we're told that mod_python is 'scary and doesn't work very
> well'.

mod_python works fantastically in my experience. that would satisfy
your requirement of stability as well as "close to the metal". but
youre going to have to roll your own pretty much everything...theres
only the most rudimdental controller layer, not much of an idea of url
resolution, and of course youd still have to figure out database/
templating. if you built a whole lot of custom mod_python handlers,
youd be tied to a very specific kind of process model and couldnt
really branch out into something like fcgi/mod_proxy->WSGI etc.

I think you guys have to either be less rigid about your requirements
and be willing to get your hands a little dirtier...web.py does seem
to be the kind of thing you guys would like, but if it has some issues
then youd just have to ....*shudder*....*contribute!* to that project
a little bit. its sort of par for the course in the field of open
source that youre going to have to be willing to contribute, if not
patches, then at least feedback and test cases to the developers for
issues found. if youre not willing to do that, you might have to
stick with J2EE for now.


Rico

unread,
May 20, 2007, 8:03:12 PM5/20/07
to
On May 16, 2:04 pm, Victor Kryukov <victor.kryu...@gmail.com> wrote:
> Hello list,
>
> our team is going to rewrite our existing web-site, which has a lot of
> dynamic content and was quickly prototyped some time ago.
>
> Today, as we get better idea of what we need, we're going to re-write
> everything from scratch. Python is an obvious candidate for our team:
> everybody knows it, everybody likes it, it has *real* objects, nice
> clean syntax etc.
>
> Our main requirement for tools we're going to use is rock-solid
> stability. As one of our team-members puts it, "We want to use tools
> that are stable, has many developer-years and thousands of user-years
> behind them, and that we shouldn't worry about their _versions_." The
> main reason for that is that we want to debug our own bugs, but not
> the bugs in our tools.
>
> Our problem is - we yet have to find any example of high-traffic,
> scalable web-site written entirely in Python. We know that YouTube is
> a suspect, but we don't know what specific python web solution was
> used there.
>
> TurboGears, Django and Pylons are all nice, and provides rich features
> - probably too many for us - but, as far as we understand, they don't
> satisfy the stability requirement - Pylons and Django hasn't even
> reached 1.0 version yet. And their provide too thick layer - we want
> something 'closer to metal', probably similar to web.py -
> unfortunately, web.py doesn't satisfy the stability requirement
> either, or so it seems.
>
> So the question is: what is a solid way to serve dynamic web pages in
> python? Our initial though was something like python + mod_python +
> Apache, but we're told that mod_python is 'scary and doesn't work very
> well'.
>
> And althoughhttp://www.python.org/about/quotes/lists many big names

> and wonderful examples, be want more details. E.g. our understanding
> is that Google uses python mostly for internal web-sites, and
> performance is far from perfect their. YouTube is an interesting
> example - anybody knows more details about that?
>
> Your suggestions and comments are highly welcome!
>
> Best Regards,
> Victor.

Teenwag runs on Python, with a hacked up Framework and recieves about
2million visitors a day and is constantly increasing
http://teenwag.com/profile?friendid=326

Ivan Tikhonov

unread,
May 21, 2007, 1:39:49 PM5/21/07
to
Use php. I am lead programmer/maintainer of big website with a lot of
interactive stuff in user's backoffice and with a lot of interraction
to our non-web apps.

PHP is a crap, but programming for web in python is a pain in the ass.
And php programmers are cheaper. Especialy avoid mod_python.

IMHO.

Matthew Nuzum

unread,
May 21, 2007, 4:42:14 PM5/21/07
to
On May 16, 4:04 pm, Victor Kryukov <victor.kryu...@gmail.com> wrote:
> Hello list,
>
> our team is going to rewrite our existing web-site, which has a lot of
> dynamic content and was quickly prototyped some time ago.
See #3 below

> Our main requirement for tools we're going to use is rock-solid
> stability. As one of our team-members puts it, "We want to use tools
> that are stable, has many developer-years and thousands of user-years
> behind them, and that we shouldn't worry about their _versions_."

> TurboGears, Django and Pylons are all nice, and provides rich features


> - probably too many for us - but, as far as we understand, they don't
> satisfy the stability requirement - Pylons and Django hasn't even
> reached 1.0 version yet.

See #3 below

> And their provide too thick layer - we want
> something 'closer to metal', probably similar to web.py -

See #1 below

> Your suggestions and comments are highly welcome!

Victor et al,

I would propose that you ask some different questions. I propose these
out of personal experience, much of it from making poor decisions and
learning the hard way. Sorry if these sound gruff, I think its hard to
avoid when using bullet points; in my mind, these are all being
phrased as kindly and gently as I can.

#1
Seek flexibility over being "closer to metal." I say this because
whenever I've thought I wanted to be closer to metal, its because I
didn't want to be constrained by a framework's assumptions. I guess
another reason would be the need for raw performance with
computations, but I think you would have said that if that was your
goal. Still, "more flexible" is not always better. "flexible and well
integrated" is slightly better than "flexible to the Nth degree."

#2
Look for projects that deal with updates and security fixes in a way
that is sensitive to users of critical applications. This is better
than programs that change little. For example, does the "vendor"
provide patches containing just critical updates? Do they have good,
clear communication about changes that may break compatibility? How
long are older versions maintained? Asking these questions will help
you find a thriving project that is actively maintained and supported.
(contrast to abandonware, which hasn't changed in ages)

#3
Why haven't you mentioned maintainability or scalability? It sounds
like you're coming from a platform that you have outgrown, either
because your app can't keep up with it's load, or because you can't
enhance it with the features you want. You're not simply refactoring
it, you're starting over from the beginning. How often do you want to
start from scratch? If the answer is, "this is the last time," then
I'd worry *way* more about this and point #2 than anything else you
mentioned.

#4 (optional)
Has your dev team built many python web-apps? I'm guessing no, or
you'd already have picked a platform because of experience. If they
have not, I'd personally also ask for a platform that is easy to
learn, works well with the dev tools (IDE, debugger, version control)
you're familiar with, and has good documentation (dead tree or
online).

The unfortunate news, I'm afraid to say, is that because of the python
culture, you're still going to face tough decisions as there are
several mature products who score high marks in the areas I've listed
above. It seems the python community at large is insanely bent on
doing things the "right" way, so there may just be too many good
options to choose from.

But its better to ask the right questions.

--
Matthew Nuzum
newz2000 on freenode

Bruno Desthuilliers

unread,
May 21, 2007, 5:30:44 PM5/21/07
to
Ivan Tikhonov a écrit :

> Use php. I am lead programmer/maintainer of big website with a lot of
> interactive stuff in user's backoffice and with a lot of interraction
> to our non-web apps.
>
> PHP is a crap, but programming for web in python is a pain in the ass.

Strange enough, MVHO on this is that PHP is crap *and* that programming
"for the web" with it is a king-size PITA, *specially* compared to Python.

> And php programmers are cheaper.

"cheaper", yes. On an hourly basis. TCO is another problem...

> Especialy avoid mod_python.

Unless you have to deeply integrate with (and are willing to be forever
tied to) Apache, I totally agree on this last one.

Bruno Desthuilliers

unread,
May 21, 2007, 6:07:23 PM5/21/07
to
John Nagle a écrit :
(snip)

> YouTube's home page is PHP. Try "www.youtube.com/index.php".
> That works, while the obvious alternatives don't.
> If you look at the page HTML, you'll see things like
>
> <a href="/login?next=/index.php"
> onclick="_hbLink('LogIn','UtilityLinks');">Log In</a>
>
> So there's definitely PHP inside YouTube.

What about learning more on web servers configuration ?
http://zope.alostnet.eu/index.php

"definitively php", hu ?

João Santos

unread,
May 21, 2007, 5:25:45 PM5/21/07
to pytho...@python.org
Please have a look at Plone and Zope.

"During the month of January 2006, we've had approx. 167 million hits"
plone.org


On 2007-05-16 23:04:17 +0200, Victor Kryukov <victor....@gmail.com> said:

> Hello list,
>
> our team is going to rewrite our existing web-site, which has a lot of
> dynamic content and was quickly prototyped some time ago.
>

> Today, as we get better idea of what we need, we're going to re-write
> everything from scratch. Python is an obvious candidate for our team:
> everybody knows it, everybody likes it, it has *real* objects, nice
> clean syntax etc.
>

> Our main requirement for tools we're going to use is rock-solid
> stability. As one of our team-members puts it, "We want to use tools
> that are stable, has many developer-years and thousands of user-years

> behind them, and that we shouldn't worry about their _versions_." The
> main reason for that is that we want to debug our own bugs, but not
> the bugs in our tools.
>
> Our problem is - we yet have to find any example of high-traffic,
> scalable web-site written entirely in Python. We know that YouTube is
> a suspect, but we don't know what specific python web solution was
> used there.
>

> TurboGears, Django and Pylons are all nice, and provides rich features
> - probably too many for us - but, as far as we understand, they don't
> satisfy the stability requirement - Pylons and Django hasn't even

> reached 1.0 version yet. And their provide too thick layer - we want


> something 'closer to metal', probably similar to web.py -

> unfortunately, web.py doesn't satisfy the stability requirement
> either, or so it seems.
>
> So the question is: what is a solid way to serve dynamic web pages in
> python? Our initial though was something like python + mod_python +
> Apache, but we're told that mod_python is 'scary and doesn't work very
> well'.
>

> And although http://www.python.org/about/quotes/ lists many big names


> and wonderful examples, be want more details. E.g. our understanding
> is that Google uses python mostly for internal web-sites, and
> performance is far from perfect their. YouTube is an interesting
> example - anybody knows more details about that?
>

> Your suggestions and comments are highly welcome!
>

> Best Regards,
> Victor.


--
João Santos
ma...@thegoldenaura.com
www.thegoldenaura.com


Michael Ströder

unread,
May 21, 2007, 2:22:19 AM5/21/07
to
John Nagle wrote:
> Sure they do. I have a complex web site, "http://www.downside.com",
> that's implemented with Perl, Apache, and MySQL. It automatically reads
> SEC
> filings and parses them to produce financial analyses. It's been
> running for seven years, and hasn't been modified in five, except [..]

Well, how can you be then sure that you don't have any security hole in
there?

Ciao, Michael.

Victor Kryukov

unread,
May 24, 2007, 9:40:04 PM5/24/07
to
Hello list,

thanks a lot to everybody for their suggestions. We're yet to make our
final decision, and the information you've provided is really helpful.

Best,
Victor.

0 new messages