Account Options

  1. Sign in
Google Groups Home
« Groups Home
Recommendation against Python?
There are currently too many topics in this group that display first. To make this topic appear first, remove this option from another topic.
There was an error processing your request. Please try again.
flag
  Messages 1 - 25 of 39 - Collapse all  -  Translate all to Translated (View all originals)   Newer >
The group you are posting to is a Usenet group. Messages posted to this group will make your email address visible to anyone on the Internet.
Your reply message has not been sent.
Your post was successful
 
From:
To:
Cc:
Followup To:
Add Cc | Add Followup-to | Edit Subject
Subject:
Validation:
For verification purposes please type the characters you see in the picture below or the numbers you hear by clicking the accessibility icon. Listen and type the numbers you hear
 
kgn  
View profile  
 More options Nov 6 2009, 4:22 pm
From: kgn <kaben.nanl...@gmail.com>
Date: Fri, 6 Nov 2009 13:22:36 -0800 (PST)
Local: Fri, Nov 6 2009 4:22 pm
Subject: Recommendation against Python?
Thomas, Jeffrey, or Collin,

I'm told that Google employees are being discouraged from using Python
for new projects, which seems silly given how much Python code and
Python support comes out of Google.

I've hunted for more information but haven't found any. Do you know
any details, or is this just a rumor?

Thanks -- K


 
You must Sign in before you can post messages.
To post a message you must first join this group.
Please update your nickname on the subscription settings page before posting.
You do not have the permission required to post.
Collin Winter  
View profile  
 More options Nov 6 2009, 4:49 pm
From: Collin Winter <collinwin...@google.com>
Date: Fri, 6 Nov 2009 13:49:45 -0800
Local: Fri, Nov 6 2009 4:49 pm
Subject: Re: [unladen-swallow] Recommendation against Python?
Hi,

On Fri, Nov 6, 2009 at 1:22 PM, kgn <kaben.nanl...@gmail.com> wrote:
> Thomas, Jeffrey, or Collin,

> I'm told that Google employees are being discouraged from using Python
> for new projects, which seems silly given how much Python code and
> Python support comes out of Google.

> I've hunted for more information but haven't found any. Do you know
> any details, or is this just a rumor?

Well, simple common sense is going to limit Python's applicability
when operating at Google's scale: it's not as fast as Java or C++,
threading sucks, memory usage is higher, etc. One of the design
constraints we face when designing any new system is, "what happens
when the load goes up by 10x or 100x? What happens if the whole planet
thinks your new service is awesome?" Any technology that makes
satisfying that constraint harder -- and I think Python falls into
this category -- *should* be discouraged if it doesn't have a very
strong case made in its favor on other merits. You have to balance
Python's strengths with its weaknesses: your engineers may be more
productive using Python, but if they have to work around more
platform-level performance/scaling limitations as volume increases, do
you come out ahead? etc.

Thanks,
Collin Winter


 
You must Sign in before you can post messages.
To post a message you must first join this group.
Please update your nickname on the subscription settings page before posting.
You do not have the permission required to post.
Luis Gonzalez  
View profile  
 More options Nov 7 2009, 10:24 am
From: Luis Gonzalez <luis...@gmail.com>
Date: Sat, 7 Nov 2009 07:24:47 -0800 (PST)
Local: Sat, Nov 7 2009 10:24 am
Subject: Re: Recommendation against Python?
...but Unladen Swallow will change that. Right?
At this moment, what are the leassons learned (I know you wrote
something about it, but it was some time ago...), how has your
expectations changed, and when do you think Unladen Swallow will
deliver on its promises? (what does your heart tell you?)??

Luis

On 6 nov, 18:49, Collin Winter <collinwin...@google.com> wrote:


 
You must Sign in before you can post messages.
To post a message you must first join this group.
Please update your nickname on the subscription settings page before posting.
You do not have the permission required to post.
Collin Winter  
View profile  
 More options Nov 9 2009, 7:41 pm
From: Collin Winter <collinwin...@google.com>
Date: Mon, 9 Nov 2009 16:41:04 -0800
Local: Mon, Nov 9 2009 7:41 pm
Subject: Re: [unladen-swallow] Re: Recommendation against Python?
Hi Luis,

On Sat, Nov 7, 2009 at 7:24 AM, Luis Gonzalez <luis...@gmail.com> wrote:
> ...but Unladen Swallow will change that. Right?
> At this moment, what are the leassons learned (I know you wrote
> something about it, but it was some time ago...), how has your
> expectations changed, and when do you think Unladen Swallow will
> deliver on its promises? (what does your heart tell you?)??

Unladen Swallow aims to shift the balancing points in that tradeoff to
make it possible to use Python in more places where it would currently
be unsuitable, but it's not going to be a panacea. Python will still
be slower than C and Java, use more memory and have inferior threading
until someone decides to invest resources into Python comparable to
what, say, Sun has invested in their JVM. I hope a focus on Python
performance by the developers will start a snowballing effect: more
companies are interested, more resources can be devoted, more grad
students will work on Python (and actually commit their work), etc.

Thanks,
Collin Winter


 
You must Sign in before you can post messages.
To post a message you must first join this group.
Please update your nickname on the subscription settings page before posting.
You do not have the permission required to post.
Luis Gonzalez  
View profile  
 More options Nov 9 2009, 8:11 pm
From: Luis Gonzalez <luis...@gmail.com>
Date: Mon, 9 Nov 2009 17:11:50 -0800 (PST)
Local: Mon, Nov 9 2009 8:11 pm
Subject: Re: Recommendation against Python?
Hi Collin,

Thank you very much for your reply!
It is clear that python will still be slower than c or java, but I
wonder how python (in this case, unladen swallow) will perform
compared to the v8 javascript engine, for example.
Do you think it can get close? Or will it take a complete rewrite from
scratch to be competitive with this engine?

Luis


 
You must Sign in before you can post messages.
To post a message you must first join this group.
Please update your nickname on the subscription settings page before posting.
You do not have the permission required to post.
Collin Winter  
View profile  
 More options Nov 9 2009, 8:12 pm
From: Collin Winter <collinwin...@google.com>
Date: Mon, 9 Nov 2009 17:12:28 -0800
Local: Mon, Nov 9 2009 8:12 pm
Subject: Re: [unladen-swallow] Re: Recommendation against Python?
Hey Luis,

On Mon, Nov 9, 2009 at 5:01 PM, Luis M. Gonzalez <luis...@gmail.com> wrote:

> Thank you very much for your reply!
> It is clear that python will still be slower than c or java, but I wonder
> how python (in this case, unladen swallow) will perform compared to the v8
> javascript engine, for example.
> Do you think it can get close? Or will it take a complete rewrite from
> scratch to be competitive with this engine?

I don't think it's possible to make an implementation like CPython as
fast as an engine like V8 or SquirrelFish Extreme that was designed to
be fast above all else. We've come up with some optimizations already
that would simply be too difficult to implement in CPython, and so we
had to discard them. Being a volunteer-run open-source project,
CPython requires somewhat different priorities than V8: CPython places
a heavy emphasis on simplicity, the idea being that a simple, slower
core will be easier for people to maintain in their free time than a
more complicated, faster core.

I have high hopes for one of the other Python implementations to
provide a longer-term performance solution designed without the
shackles of C-level backwards compatibility. We're thinking "what can
we accomplish in the short term to make things better *now*?", while a
project like PyPy is approach performance from the perspective of
"what can we accomplish if we spend a decade really getting it right?"

Thanks,
Collin Winter


 
You must Sign in before you can post messages.
To post a message you must first join this group.
Please update your nickname on the subscription settings page before posting.
You do not have the permission required to post.
Reid Kleckner  
View profile  
 More options Nov 10 2009, 10:22 am
From: Reid Kleckner <r...@mit.edu>
Date: Tue, 10 Nov 2009 10:22:29 -0500
Local: Tues, Nov 10 2009 10:22 am
Subject: Re: [unladen-swallow] Re: Recommendation against Python?

On Mon, Nov 9, 2009 at 8:12 PM, Collin Winter <collinwin...@google.com> wrote:
> I don't think it's possible to make an implementation like CPython as
> fast as an engine like V8 or SquirrelFish Extreme that was designed to
> be fast above all else. We've come up with some optimizations already
> that would simply be too difficult to implement in CPython, and so we
> had to discard them. Being a volunteer-run open-source project,
> CPython requires somewhat different priorities than V8: CPython places
> a heavy emphasis on simplicity, the idea being that a simple, slower
> core will be easier for people to maintain in their free time than a
> more complicated, faster core.

Yeah, I was thinking about this, and it's interesting to note that
there are very few CPython developers who are paid to work on CPython.
 They're all volunteers.  As such, performance really isn't a priority
the way it is for the JVM.  The Ruby folks are getting funding (from
EngineYard and others) to work on their VMs because people have
realized that MRI is too darn slow to serve web applications and they
want to fix it.

Reid


 
You must Sign in before you can post messages.
To post a message you must first join this group.
Please update your nickname on the subscription settings page before posting.
You do not have the permission required to post.
Ad  
View profile  
 More options Nov 10 2009, 3:29 am
From: Ad <ada...@gmail.com>
Date: Tue, 10 Nov 2009 00:29:32 -0800 (PST)
Local: Tues, Nov 10 2009 3:29 am
Subject: Re: Recommendation against Python?
What about Jython?

 
You must Sign in before you can post messages.
To post a message you must first join this group.
Please update your nickname on the subscription settings page before posting.
You do not have the permission required to post.
Alex Gaynor  
View profile  
 More options Nov 10 2009, 11:15 am
From: Alex Gaynor <alex.gay...@gmail.com>
Date: Tue, 10 Nov 2009 11:15:38 -0500
Local: Tues, Nov 10 2009 11:15 am
Subject: Re: [unladen-swallow] Re: Recommendation against Python?

On Tue, Nov 10, 2009 at 3:29 AM, Ad <ada...@gmail.com> wrote:
> What about Jython?

What about Jython?  Collin has indicated the primary issue is resource
(CPU and RAM) at Google's scale, Jython has no significant advantage
in either of these departments.

Alex

--
"I disapprove of what you say, but I will defend to the death your
right to say it." -- Voltaire
"The people's good is the highest law." -- Cicero
"Code can always be simpler than you think, but never as simple as you
want" -- Me


 
You must Sign in before you can post messages.
To post a message you must first join this group.
Please update your nickname on the subscription settings page before posting.
You do not have the permission required to post.
Reid Kleckner  
View profile  
 More options Nov 10 2009, 11:52 am
From: Reid Kleckner <r...@mit.edu>
Date: Tue, 10 Nov 2009 11:52:32 -0500
Local: Tues, Nov 10 2009 11:52 am
Subject: Re: [unladen-swallow] Re: Recommendation against Python?

On Tue, Nov 10, 2009 at 11:15 AM, Alex Gaynor <alex.gay...@gmail.com> wrote:
> On Tue, Nov 10, 2009 at 3:29 AM, Ad <ada...@gmail.com> wrote:
>> What about Jython?

> What about Jython?  Collin has indicated the primary issue is resource
> (CPU and RAM) at Google's scale, Jython has no significant advantage
> in either of these departments.

Another one of unladen-swallow's major goals is to maintain
source-level compatibility with C extension modules, which Google uses
a lot of.  Using Jython would require moving that infrastructure from
SWIG to JNI, which would be a huge pain in the butt.

Reid


 
You must Sign in before you can post messages.
To post a message you must first join this group.
Please update your nickname on the subscription settings page before posting.
You do not have the permission required to post.
Collin Winter  
View profile  
 More options Nov 10 2009, 1:04 pm
From: Collin Winter <collinwin...@google.com>
Date: Tue, 10 Nov 2009 10:04:44 -0800
Local: Tues, Nov 10 2009 1:04 pm
Subject: Re: [unladen-swallow] Re: Recommendation against Python?

On Tue, Nov 10, 2009 at 8:15 AM, Alex Gaynor <alex.gay...@gmail.com> wrote:
> On Tue, Nov 10, 2009 at 3:29 AM, Ad <ada...@gmail.com> wrote:
>> What about Jython?

> What about Jython?  Collin has indicated the primary issue is resource
> (CPU and RAM) at Google's scale, Jython has no significant advantage
> in either of these departments.

Jython has a similarly-small number of paid full-time developers (and
Frank W just left Sun, as you may have seen). To date, IronPython and
Jython have had to dedicate a significant portion of their engineering
resources to achieving compatibility with CPython, with relatively
little time left over for performance optimization. It remains to be
seen how these factors will be balanced as Jython, IronPython, PyPy,
etc move to support Python 3.

Collin Winter


 
You must Sign in before you can post messages.
To post a message you must first join this group.
Please update your nickname on the subscription settings page before posting.
You do not have the permission required to post.
Collin Winter  
View profile  
 More options Nov 10 2009, 1:10 pm
From: Collin Winter <collinwin...@google.com>
Date: Tue, 10 Nov 2009 10:10:03 -0800
Local: Tues, Nov 10 2009 1:10 pm
Subject: Re: [unladen-swallow] Re: Recommendation against Python?

On Tue, Nov 10, 2009 at 8:52 AM, Reid Kleckner <r...@mit.edu> wrote:
> On Tue, Nov 10, 2009 at 11:15 AM, Alex Gaynor <alex.gay...@gmail.com> wrote:
>> On Tue, Nov 10, 2009 at 3:29 AM, Ad <ada...@gmail.com> wrote:
>>> What about Jython?

>> What about Jython?  Collin has indicated the primary issue is resource
>> (CPU and RAM) at Google's scale, Jython has no significant advantage
>> in either of these departments.

> Another one of unladen-swallow's major goals is to maintain
> source-level compatibility with C extension modules, which Google uses
> a lot of.  Using Jython would require moving that infrastructure from
> SWIG to JNI, which would be a huge pain in the butt.

Reid is correct: moving large applications from SWIG-wrapped
infrastructure (e.g., Bigtable) to their Java equivalents would be a
tremendous pain in the ass and would almost certainly introduce
exceedingly-subtle bugs due to differences in handling rare edge/error
cases. This is the primary reason why we did not choose Jython as a
baseline.

Collin Winter


 
You must Sign in before you can post messages.
To post a message you must first join this group.
Please update your nickname on the subscription settings page before posting.
You do not have the permission required to post.
Tom Machinski  
View profile  
 More options Nov 10 2009, 2:53 pm
From: Tom Machinski <tom.machin...@gmail.com>
Date: Tue, 10 Nov 2009 11:53:00 -0800 (PST)
Subject: Re: Recommendation against Python?
What about Cython?

http://cython.org/

 -- Tom


 
You must Sign in before you can post messages.
To post a message you must first join this group.
Please update your nickname on the subscription settings page before posting.
You do not have the permission required to post.
Collin Winter  
View profile  
 More options Nov 10 2009, 5:26 pm
From: Collin Winter <collinwin...@google.com>
Date: Tue, 10 Nov 2009 14:26:38 -0800
Local: Tues, Nov 10 2009 5:26 pm
Subject: Re: [unladen-swallow] Recommendation against Python?

Since this has come up off-list, let me be more specific about this point:

One area where I believe Python (and other dynamic languages) do
particularly well is projects with lots of developers working on lots
of little subsystems, all contributing to the same codebase or the
same released binary. In a language like C or C++, the more developers
you add, the more fragile your binary becomes: it only takes one
segfault to kill a running binary (and hence lose those pending
requests), and the probability of introducing that segfault goes up
with the number of developers/subsystems/integration points/etc.
Dynamic languages, on the other hand, are much easier to sandbox in
this regard. If you want to isolate failures in one particular
component in your Python system, you can just throw a try/except
around it and you're basically good to go. That kind of agility and
flexibility is an important advantage, and it's one of the "other
merits" I was alluding to above.

Python may not be a good fit for millisecond-critical systems, but it
does have advantages; we wouldn't use it if it didn't.

Collin Winter


 
You must Sign in before you can post messages.
To post a message you must first join this group.
Please update your nickname on the subscription settings page before posting.
You do not have the permission required to post.
Tom Machinski  
View profile  
 More options Nov 10 2009, 5:40 pm
From: Tom Machinski <tom.machin...@gmail.com>
Date: Tue, 10 Nov 2009 14:40:19 -0800
Local: Tues, Nov 10 2009 5:40 pm
Subject: Re: [unladen-swallow] Recommendation against Python?

Just to clarify my question about Cython:

It seems to satisfy both requirements that Collin mentioned above, namely:
 1. It is a technology that is usable right now.
 2. It offers very significant efficiency gains.

So even for a company like Google, wouldn't it make sense to write a
prototype with Python, then gradually convert performance critical parts to
Cython, which still offers most of the benefits of pure Python, paired with
C-like efficiency and optimization potential?

 -- Tom


 
You must Sign in before you can post messages.
To post a message you must first join this group.
Please update your nickname on the subscription settings page before posting.
You do not have the permission required to post.
Sebastian  
View profile  
 More options Nov 10 2009, 5:41 pm
From: Sebastian <flashbus...@gmail.com>
Date: Tue, 10 Nov 2009 14:41:30 -0800 (PST)
Local: Tues, Nov 10 2009 5:41 pm
Subject: Re: Recommendation against Python?
I always had the impression that java is much faster than python, but
sucks up a rather big amount of memory, so this isn't true in your
experience?
My impression more or less came from
http://shootout.alioth.debian.org/u64q/benchmark.php?test=all&lang=ja...
as i didn't test that myself.

sebastian


 
You must Sign in before you can post messages.
To post a message you must first join this group.
Please update your nickname on the subscription settings page before posting.
You do not have the permission required to post.
Leon Sit  
View profile  
 More options Nov 10 2009, 5:46 pm
From: Leon Sit <wing1127ai...@gmail.com>
Date: Tue, 10 Nov 2009 17:46:40 -0500
Local: Tues, Nov 10 2009 5:46 pm
Subject: Re: [unladen-swallow] Recommendation against Python?
I believe that Cython has good performance increase in numerical code
and not so much on other application. Moreover, Cython relies on C
compiler which most Windows systems do not have. To get major speedup
in Cython, you would need to add type information which involves
syntax outside of standard Python. I do not see the reason why Cython
is what Google is looking for.

Leon


 
You must Sign in before you can post messages.
To post a message you must first join this group.
Please update your nickname on the subscription settings page before posting.
You do not have the permission required to post.
Tom Machinski  
View profile  
 More options Nov 10 2009, 5:54 pm
From: Tom Machinski <tom.machin...@gmail.com>
Date: Tue, 10 Nov 2009 14:54:27 -0800
Local: Tues, Nov 10 2009 5:54 pm
Subject: Re: [unladen-swallow] Recommendation against Python?

On Tue, Nov 10, 2009 at 2:46 PM, Leon Sit <wing1127ai...@gmail.com> wrote:
> I believe that Cython has good performance increase in numerical code and
> not so much on other application.

Is this belief based on factual data?  I have working examples demonstrating
>100x performance gains for common, non-"numerical" operations such as

looping, iteration, object construction, etc.

> Moreover, Cython relies on C compiler which most Windows systems do not
> have.

How is that relevant for the many server-side users of Python, such as
Google?

> To get major speedup in Cython, you would need to add type information
> which involves syntax outside of standard Python.

Yes. Still far less work than rewriting the entire prototype in C, or
writing it in C from scratch.

 -- Tom


 
You must Sign in before you can post messages.
To post a message you must first join this group.
Please update your nickname on the subscription settings page before posting.
You do not have the permission required to post.
Brian Harring  
View profile  
 More options Nov 10 2009, 8:28 pm
From: Brian Harring <ferri...@gmail.com>
Date: Tue, 10 Nov 2009 17:28:49 -0800
Local: Tues, Nov 10 2009 8:28 pm
Subject: Re: [unladen-swallow] Recommendation against Python?

On Tue, Nov 10, 2009 at 02:54:27PM -0800, Tom Machinski wrote:
>    On Tue, Nov 10, 2009 at 2:46 PM, Leon Sit <[1]wing1127ai...@gmail.com>
>    wrote:

>      I believe that Cython has good performance increase in numerical
>      code and not so much on other application.

>    Is this belief based on factual data?  I have working examples
>    demonstrating >100x performance gains for common, non-"numerical"
>    operations such as looping, iteration, object construction, etc.

Fundamentally cython is the wrong approach for this in my opinion.  
It's syntactic glue to translate pseudo python into c code- much like
pyrex.  This isn't a bad thing mind you.  It's just not applicable to
speeding up the core processing of the language.

To use cython, they'd have to go converting large chunks of stdlib to
cython.  Yes those components would be faster, but when I write

def f(*args):
    some_cython_func(args[0])
    ... do other things

This still invokes the mainline cpython vm, meaning you still get
.5us hit for just generating the function frame itself.  This is
ignoring the FFI cost there also- for passing bits into the extension,
it has to convert it into tuples/keywords (hence *args, **kwargs,
although METH_O/METH_NONE are modifications to this).

All of these things take time, and are not something cython can
address unless you're proposing shifting the actual vm implementation
itself to cython... which isn't really possible.

>      Moreover, Cython relies on C compiler which most Windows systems do
>      not have.

>    How is that relevant for the many server-side users of Python, such as
>    Google?

Rebasing the implementation upon llvm, they don't have to care if it's
windows or unix (or client or server)- they just need to either
compile llvm into the resultant binary, or ensure the library is
there.

This is *far* more simple then trying to ensure cython machinery still
works and avoids excluding platforms.

>      To get major speedup in Cython, you would need to add type
>      information which involves syntax outside of standard Python.

>    Yes. Still far less work than rewriting the entire prototype in C, or
>    writing it in C from scratch.

The gains from cython aren't comparable to llvm gains on a grand
scale.  Speeding up chunks of code is nice (a faster os.path.join for
example), but that's hot spot optimization.  The potential gains from
it aren't comparable to the intended goal of a 5x increase in
raw/normal python code (this is the target of u-s).

Or that's the theory at least.  Either way I'd suggest you dig into
what u-s is targeting, and what cython is targeting- they're two
rather orthogonally different things.  The one nifty thing is that via
u-s trying to maintain transparent cpy extension, you probably will be
able to use cython generated modules w/ u-s.  That's the current
theory at least...

Hope that clarifies things.
~harring

  application_pgp-signature_part
< 1K Download

 
You must Sign in before you can post messages.
To post a message you must first join this group.
Please update your nickname on the subscription settings page before posting.
You do not have the permission required to post.
Tom Machinski  
View profile  
 More options Nov 10 2009, 10:25 pm
From: Tom Machinski <tom.machin...@gmail.com>
Date: Tue, 10 Nov 2009 19:25:43 -0800
Local: Tues, Nov 10 2009 10:25 pm
Subject: Re: [unladen-swallow] Recommendation against Python?

On Tue, Nov 10, 2009 at 5:28 PM, Brian Harring <ferri...@gmail.com> wrote:

> The gains from cython aren't comparable to llvm gains on a grand
> scale.  Speeding up chunks of code is nice (a faster os.path.join for
> example), but that's hot spot optimization.  The potential gains from
> it aren't comparable to the intended goal of a 5x increase in
> raw/normal python code (this is the target of u-s).

So if your CPU-bound program spends a conservative 90% of its
execution time running a small set of tight loops, and Cython can
speed those loops up by 100x or more, it's not comparable to an
expected 5x gain that U-S will hopefully deliver to the entire
application?

And you realize that most CPU-bound programs are like this, spending
most of their execution time - probably more than 90% - in very
specific hot spots, so dismissing "hot spot optimization" is
inappropriate in this context?

> To use cython, they'd have to go converting large chunks of stdlib to
> cython.

They don't have be Cython, just C extension modeuls, which many
performance-intensive parts of the stdlib already are.

> Yes those components would be faster, but when I write

> def f(*args):
>    some_cython_func(args[0])
>    ... do other things

> This still invokes the mainline cpython vm, meaning you still get
> .5us hit for just generating the function frame itself.

Correct. The sensible design here is that if f() is being invoked
enough times to matter - i.e. it is itself inside a tight loop - then
optimizing it entails translating f() itself and its containing loop
to Cython.

Don't get me wrong: I'm excited by U-S and hope it meets its
beneficial goals. Unfortunately, I also agree with Collin when he says
that even if all the project goals are met, Python on U-S would still
not be a match for Java or C++.

Cython exists right now, and offers efficiency gains that bring it
close to compiled code of these two languages. So if Google, or any
other company, is entirely ruling out Python for performance intensive
tasks, it's interesting to ask whether they considered it, and on what
grounds (if any) was it ruled out.

 -- Tom


 
You must Sign in before you can post messages.
To post a message you must first join this group.
Please update your nickname on the subscription settings page before posting.
You do not have the permission required to post.
Craig Citro  
View profile  
 More options Nov 10 2009, 11:47 pm
From: Craig Citro <craigci...@gmail.com>
Date: Tue, 10 Nov 2009 20:47:28 -0800
Local: Tues, Nov 10 2009 11:47 pm
Subject: Re: [unladen-swallow] Recommendation against Python?

> Just to clarify my question about Cython:

> It seems to satisfy both requirements that Collin mentioned above, namely:
>  1. It is a technology that is usable right now.
>  2. It offers very significant efficiency gains.

> So even for a company like Google, wouldn't it make sense to write a
> prototype with Python, then gradually convert performance critical parts to
> Cython, which still offers most of the benefits of pure Python, paired with
> C-like efficiency and optimization potential?

I think what you're saying could be a very good plan for *new* Python
code being developed at Google. It's exactly how we develop code for
the Sage library, in fact -- write it and get it working correctly in
Python, write some tests, then move the mission-critical sections into
Cython for speed. However, I think that the comments Collin makes
above are all in the context of making Google's *existing* Python code
run faster, without having to do anything to the source at all. In
particular, they also want something they can do without having to
worry about the prospect of introducing subtle bugs in corner cases
(like, say, the difference between C and Python semantics for
arithmetic).

That said, this will also be a much more interesting question a little
further down the line. In particular, the goal is for Cython to
support all of the Python language in the near future, and there are
things like Robert Bradshaw's recent start of a type inferencer ...
once Cython can actually compile all the code one might run into in
the wild, it'll be interesting to see how competitive it is.

Even still, though, Cython and Unladen Swallow really are orthogonal
in several ways -- in particular, with LLVM in the mix, Unladen
Swallow has the prospect of introducing lots of interesting *runtime*
optimization and specialization that Cython could never do ...

-cc


 
You must Sign in before you can post messages.
To post a message you must first join this group.
Please update your nickname on the subscription settings page before posting.
You do not have the permission required to post.
Reid Kleckner  
View profile  
 More options Nov 11 2009, 12:47 am
From: Reid Kleckner <r...@mit.edu>
Date: Wed, 11 Nov 2009 00:47:16 -0500
Local: Wed, Nov 11 2009 12:47 am
Subject: Re: [unladen-swallow] Recommendation against Python?

> Even still, though, Cython and Unladen Swallow really are orthogonal
> in several ways -- in particular, with LLVM in the mix, Unladen
> Swallow has the prospect of introducing lots of interesting *runtime*
> optimization and specialization that Cython could never do ...

This is actually a really important point.  The HotSpot JVM can beat
g++ on some benchmarks because it can devirtualize method calls with
type feedback where the C++ compiler doesn't have enough information
to generate the fastest code.  This suggests to me that it shouldn't
be too hard to beat a *naive* static compilation of pure Python code
with Cython.

Cython is great if you have a specific application for which you
really care about performance, and you're willing to profile it, find
the hot loops, and turn objects into native C data types.  If you
aren't willing to dig in and force your pure Python code into Cython,
a JIT compiler like u-s can do a much better job optimizing the pure
Python code.  It has so much more information than a static compiler.
That's the theory, anyway.  :)

Here's another anecdote for why Cython may not be the right answer for
all Python performance problems.  I wrote an optimizing Decaf compiler
for a class in Python, and I could not get Cython to speed it up.  I
only spent one afternoon working on it, but I was surprised that
everything I tried (in particular, turning the IR classes into Cython
classes) resulted in slowdowns, not speedups.  There were none of the
mythical "doubly-nested for loops" that I could just provide type
annotations for to make my code fast.

So, in summary, Cython is great for bridging that performance gap for
projects that are concerned about performance, and u-s is targeted
more towards everyone else who will take what performance they can
get.

Reid


 
You must Sign in before you can post messages.
To post a message you must first join this group.
Please update your nickname on the subscription settings page before posting.
You do not have the permission required to post.
Thomas_H  
View profile  
 More options Nov 11 2009, 9:09 am
From: Thomas_H <the...@googlemail.com>
Date: Wed, 11 Nov 2009 06:09:02 -0800 (PST)
Local: Wed, Nov 11 2009 9:09 am
Subject: Re: Recommendation against Python?

On Nov 10, 4:22 pm, Reid Kleckner <r...@mit.edu> wrote:

> Yeah, I was thinking about this, and it's interesting to note that
> there are very few CPython developers who are paid to work on CPython.
>  They're all volunteers.  As such, performance really isn't a priority
> the way it is for the JVM.  The Ruby folks are getting funding (from
> EngineYard and others) to work on their VMs because people have
> realized that MRI is too darn slow to serve web applications and they
> want to fix it.

This comes as kind of a surprise, knowing that GvR is working for
Google, and obviously you U-S guys are getting funded. Wouldn't it be
only too logical that Google throws a couple of FTE at CPython?!

T.


 
You must Sign in before you can post messages.
To post a message you must first join this group.
Please update your nickname on the subscription settings page before posting.
You do not have the permission required to post.
Luis Gonzalez  
View profile  
 More options Nov 11 2009, 11:42 am
From: Luis Gonzalez <luis...@gmail.com>
Date: Wed, 11 Nov 2009 08:42:34 -0800 (PST)
Local: Wed, Nov 11 2009 11:42 am
Subject: Re: Recommendation against Python?
Collin,

We know that U-S is an effort to speed up cpython *right now*, as an
incremental step to the current implementation (as opposed to Pypy,
for example, which is what can be done in a decade...).

But I wonder if, considering the importance of Python for Google, and
seeing that V8 is a Google project...
Is there any slim chance that, one day, Google will do for Python the
same they are doing for Javascript?

After all, Python is not very different than Javascript. They are both
dynamic languages.
And the knowledge and brains are there.
Do you imagine working with Lars Bak & Co.? Or perhaps, applying the
same strategy and techniques for implementing python?

I know it sounds easier than it is, but wouldn't it be great if you
guys could leverage all this knowledge and experience for Python?
If not Google, which is already doing it for Javascript, who else?

Luis

On 10 nov, 19:26, Collin Winter <collinwin...@google.com> wrote:


 
You must Sign in before you can post messages.
To post a message you must first join this group.
Please update your nickname on the subscription settings page before posting.
You do not have the permission required to post.
Collin Winter  
View profile  
 More options Nov 13 2009, 5:25 pm
From: Collin Winter <collinwin...@google.com>
Date: Fri, 13 Nov 2009 14:25:00 -0800
Subject: Re: [unladen-swallow] Recommendation against Python?
Hey Tom,

On Tue, Nov 10, 2009 at 7:25 PM, Tom Machinski <tom.machin...@gmail.com> wrote:

[snip]

> Don't get me wrong: I'm excited by U-S and hope it meets its
> beneficial goals. Unfortunately, I also agree with Collin when he says
> that even if all the project goals are met, Python on U-S would still
> not be a match for Java or C++.

> Cython exists right now, and offers efficiency gains that bring it
> close to compiled code of these two languages. So if Google, or any
> other company, is entirely ruling out Python for performance intensive
> tasks, it's interesting to ask whether they considered it, and on what
> grounds (if any) was it ruled out.

Right, the obvious hotspots are the easy part: you code them by hand
in C, use a tool like Cython, bring in Psyco, etc. We're seeking to
speed up the entire application from end to end without the need for
special tools, to have that performance be more adaptive to code
changes, and to make the existing Python language faster without
language extensions. We feel that such extensions and exceptions
increase the barrier to adoption, increase possibly-problematic edge
cases, and increase the training needed for engineers to produce
Python applications.

Thanks,
Collin Winter


 
You must Sign in before you can post messages.
To post a message you must first join this group.
Please update your nickname on the subscription settings page before posting.
You do not have the permission required to post.
Messages 1 - 25 of 39   Newer >
« Back to Discussions « Newer topic     Older topic »