[ann] first release of PyPy

2 views
Skip to first unread message

holger krekel

unread,
May 20, 2005, 5:38:07 PM5/20/05
to pytho...@python.org
Welcome to PyPy 0.6
--------------------

*The PyPy Development Team is happy to announce the first
public release of PyPy after two years of spare-time and
half a year of EU funded development. The 0.6 release
is eminently a preview release.*

What it is and where to start
-----------------------------

Getting started: http://codespeak.net/pypy/index.cgi?doc/getting_started.html

PyPy Documentation: http://codespeak.net/pypy/index.cgi?doc

PyPy Homepage: http://codespeak.net/pypy/

PyPy is a MIT-licensed reimplementation of Python written in
Python itself. The long term goals are an implementation that
is flexible and easy to experiment with and retarget to
different platforms (also non-C ones) and such that high
performance can be achieved through high-level implementations
of dynamic optimisation techniques.

The interpreter and object model implementations shipped with 0.6 can
be run on top of CPython and implement the core language features of
Python as of CPython 2.3. PyPy passes around 90% of the Python language
regression tests that do not depend deeply on C-extensions. Some of
that functionality is still made available by PyPy piggy-backing on
the host CPython interpreter. Double interpretation and abstractions
in the code-base make it so that PyPy running on CPython is quite slow
(around 2000x slower than CPython ), this is expected.

This release is intended for people that want to look and get a feel
into what we are doing, playing with interpreter and perusing the
codebase. Possibly to join in the fun and efforts.

Interesting bits and highlights
---------------------------------

The release is also a snap-shot of our ongoing efforts towards
low-level translation and experimenting with unique features.

* By default, PyPy is a Python version that works completely with
new-style-classes semantics. However, support for old-style classes
is still available. Implementations, mostly as user-level code, of
their metaclass and instance object are included and can be re-made
the default with the ``--oldstyle`` option.

* In PyPy, bytecode interpretation and object manipulations
are well separated between a bytecode interpreter and an
*object space* which implements operations on objects.
PyPy comes with experimental object spaces augmenting the
standard one through delegation:

* an experimental object space that does extensive tracing of
bytecode and object operations;

* the 'thunk' object space that implements lazy values and a 'become'
operation that can exchange object identities.

These spaces already give a glimpse in the flexibility potential of
PyPy. See demo/fibonacci.py and demo/sharedref.py for examples
about the 'thunk' object space.

* The 0.6 release also contains a snapshot of our translation-efforts
to lower level languages. For that we have developed an
annotator which is capable of infering type information
across our code base. The annotator right now is already
capable of successfully type annotating basically *all* of
PyPy code-base, and is included with 0.6.

* From type annotated code, low-level code needs to be generated.
Backends for various targets (C, LLVM,...) are included; they are
all somehow incomplete and have been and are quite in flux. What is
shipped with 0.6 is able to deal with more or less small/medium examples.


Ongoing work and near term goals
---------------------------------

Generating low-level code is the main area we are hammering on in the
next months; our plan is to produce a PyPy version in August/September
that does not need to be interpreted by CPython anymore and will
thus run considerably faster than the 0.6 preview release.

PyPy has been a community effort from the start and it would
not have got that far without the coding and feedback support
from numerous people. Please feel free to give feedback and
raise questions.

contact points: http://codespeak.net/pypy/index.cgi?contact

contributor list: http://codespeak.net/pypy/index.cgi?doc/contributor.html

have fun,

Armin Rigo, Samuele Pedroni,

Holger Krekel, Christian Tismer,

Carl Friedrich Bolz


PyPy development and activities happen as an open source project
and with the support of a consortium funded by a two year EU IST
research grant. Here is a list of partners of the EU project:

Heinrich-Heine University (Germany), AB Strakt (Sweden)

merlinux GmbH (Germany), tismerysoft GmbH(Germany)

Logilab Paris (France), DFKI GmbH (Germany)

ChangeMaker (Sweden)

Kay Schluehr

unread,
May 20, 2005, 6:50:07 PM5/20/05
to

holger krekel wrote:
> Welcome to PyPy 0.6
> --------------------
>
> *The PyPy Development Team is happy to announce the first
> public release of PyPy after two years of spare-time and
> half a year of EU funded development. The 0.6 release
> is eminently a preview release.*

Congratulation to You and Your team!

PyPy is really awesome and if it succeeds in speed demands after the
translation phase I believe that the project will shift the power
within the Python community on the long run. There are moments I'm
almost shocked about it and think about the fate of other programming
programming languages like LISP. PyPy can be resolved to "Python in
Python" but also "Python multiplied/powered by itself" which is much
more triumphant. A short review of the 'thunks' objspace example gives
me the impression that the language development process as we know it
comes to an end and makes a kind of transition. This is both very
exciting and dangerous, like every philosophical event.

Regards,
Kay

beli...@aol.com

unread,
May 20, 2005, 11:02:16 PM5/20/05
to
Kay Schluehr wrote:
> holger krekel wrote:
> > Welcome to PyPy 0.6
> > --------------------
> >
> > *The PyPy Development Team is happy to announce the first
> > public release of PyPy after two years of spare-time and
> > half a year of EU funded development. The 0.6 release
> > is eminently a preview release.*
>
> Congratulation to You and Your team!
>
> PyPy is really awesome and if it succeeds in speed demands after the
> translation phase I believe that the project will shift the power
> within the Python community on the long run.

Could you please explain this statement? Who will gain power, and who
will lose it? Are you suggesting that CPython and PyPy developers are
competing?

Paul Rubin

unread,
May 20, 2005, 11:08:25 PM5/20/05
to
beli...@aol.com writes:
> > PyPy is really awesome and if it succeeds in speed demands after the
> > translation phase I believe that the project will shift the power
> > within the Python community on the long run.
>
> Could you please explain this statement? Who will gain power, and who
> will lose it? Are you suggesting that CPython and PyPy developers are
> competing?

I interpreted it to mean that it would bring power into the Python
community from outside, e.g., Python would become more competitive with
other languages that are currently beating it on performance.

Christian Tismer

unread,
May 21, 2005, 12:20:51 AM5/21/05
to beli...@aol.com, pytho...@python.org
beli...@aol.com wrote:

The Python community will gain power, and nobody will loose some.
The big win is that we gain a new flexibility that did not
exist before, even if PyPy should completely miss its speed
promises. Having an extremely flexible implementation in a
very high-level language (which happens to be Python) enables
possibilities which have not been seen, before.

There is of course a chance for some community of C programmers
to loose interest, if PyPy really gets as efficient as we hope
for. But this is a) still a long, uncertain path and b) not
a real danger, but more likely an advantage for the involved people.

> Are you suggesting that CPython and PyPy developers are
> competing?

No idea how we could get onto this track. If there is a competition,
then only if PyPy gets into a position where it is comparable
with CPython. This is not the case, at least not in a well-ordered manner.
It is not really faster, but it is definately much more flexible.
Comparisons are not suitable at all, bcause there are too many
qualities to compare about.

And I see no point for any competition in any future. We all love
Python. It is a language, and languages are communities.
If a particular implementation gets more interest for some reasons,
then because it ibetter s more efficient or more interesting, whatever
reasoning gives it popularity. But we are all with Python!

Surely we are comparing our performance with CPython's. This is
not the real point. Note also, that many of the PyPy team members belong
to CPython core developers, as well. This is not a competition, but
a huge new branch, exploring what is doable and what not.

You might also give our website a try which is quite informative
and gives you an insight into what we are aiming for.

http://codespeak.net/pypy

PyPy is just a completely new approach to interpreted languages,
almost based upon known compiler technology, but applying this in a
consequent manner, that has no comparable prior example.

I wish to repeat the congratulations to the team for the first release!
-----------------------------------------------------------------------

ciao -- chris

--
Christian Tismer :^) <mailto:tis...@stackless.com>
tismerysoft GmbH : Have a break! Take a ride on Python's
Johannes-Niemeyer-Weg 9A : *Starship* http://starship.python.net/
14109 Berlin : PGP key -> http://wwwkeys.pgp.net/
work +49 30 802 86 56 mobile +49 173 24 18 776 fax +49 30 80 90 57 05
PGP 0x57F3BF04 9064 F4E1 D754 C2FF 1619 305B C09C 5A3B 57F3 BF04
whom do you want to sponsor today? http://www.stackless.com/

Paul Rubin

unread,
May 21, 2005, 12:42:10 AM5/21/05
to
Christian Tismer <tis...@stackless.com> writes:
> PyPy is just a completely new approach to interpreted languages,
> almost based upon known compiler technology, but applying this in a
> consequent manner, that has no comparable prior example.

Is there a web page describing what's new? Compile-and-go interactive
languages have been around for decades.

Kay Schluehr

unread,
May 21, 2005, 1:31:00 AM5/21/05
to

Christian Tismer wrote:
> beli...@aol.com wrote:
>
> > Kay Schluehr wrote:
> >
> >>holger krekel wrote:
> >>
> >>>Welcome to PyPy 0.6
> >>>--------------------
> >>>
> >>>*The PyPy Development Team is happy to announce the first
> >>>public release of PyPy after two years of spare-time and
> >>>half a year of EU funded development. The 0.6 release
> >>>is eminently a preview release.*
> >>
> >>Congratulation to You and Your team!
> >>
> >>PyPy is really awesome and if it succeeds in speed demands after
the
> >>translation phase I believe that the project will shift the power
> >>within the Python community on the long run.
> >
> >
> > Could you please explain this statement? Who will gain power, and
who
> > will lose it?
>
> The Python community will gain power, and nobody will loose some.
> The big win is that we gain a new flexibility that did not
> exist before, even if PyPy should completely miss its speed
> promises. Having an extremely flexible implementation in a
> very high-level language (which happens to be Python) enables
> possibilities which have not been seen, before.

But not only flexibility IN the current language but also beyond it.
It's not anymore clear what the language as a set of well-defined
syntactical and semantical rules really is if You can change the
semantics in an arbitrary module representing an object-space. I
currently don't know how modular the parser is but adding syntax-rules
should not be that hard either. Once You get enough speed out of the
PyPy-runtime and the community shifts to it the PEP-process degenerates
in the view of a PyPythonista to discussions about aspects of the
std-objectspace and language design patterns. There will be some
CPython compliance - that's all.

Empowering the community means beheading the BDFL and that's currently
not only a person but a principle. Well, maybe that's o.k. but at least
inevitable and Guido finally finds the time to clip roses, writes his
memoirs, educates children and polishs his medals of honour.

Regards,
Kay

Torsten Bronger

unread,
May 21, 2005, 2:39:26 AM5/21/05
to
Hallöchen!

"Kay Schluehr" <kay.sc...@gmx.net> writes:

> [...]
>
> [...] Once You get enough speed out of the PyPy-runtime and the


> community shifts to it the PEP-process degenerates in the view of
> a PyPythonista to discussions about aspects of the std-objectspace
> and language design patterns. There will be some CPython
> compliance - that's all.

Please could somebody explain to us non-CS people why PyPy could
have speed features CPython can't have?

Tschö,
Torsten.

--
Torsten Bronger, aquisgrana, europa vetus

Paul Rubin

unread,
May 21, 2005, 2:41:10 AM5/21/05
to
Torsten Bronger <bro...@physik.rwth-aachen.de> writes:
> Please could somebody explain to us non-CS people why PyPy could
> have speed features CPython can't have?

Does the one-word answer "compiler" explain enough?

Torsten Bronger

unread,
May 21, 2005, 3:13:19 AM5/21/05
to
Hallöchen!

No, just more questions. ;-)

What's supposed to be compiled? Only PyPy itself or also the
programs it's "interpreting"?

<http://www.python.org/pycon/dc2004/papers/27/>: "In the next step
of the project, we will generate C code or machine code from the
source of Pypy, thereby reducing the speed penalty."

I've been told by so many books and on-line material that Python
cannot be compiled (unless you cheat). So how is this possible?

Ville Vainio

unread,
May 21, 2005, 3:29:30 AM5/21/05
to
>>>>> "Torsten" == Torsten Bronger <bro...@physik.rwth-aachen.de> writes:

Torsten> What's supposed to be compiled? Only PyPy itself or also
Torsten> the programs it's "interpreting"?

PyPy is written in python, if it can be compiled then the programs can
be as well.

Torsten> I've been told by so many books and on-line material that
Torsten> Python cannot be compiled (unless you cheat). So how is
Torsten> this possible?

These guys are exploring a new territory. OTOH, Lisp is a dynamic
language like python and it can be compiled to native code. Pyrex
demonstrates the "trivial" way to compile python to native code, the
real problem is making the resulting code fast. Typically this
requires type inference (i.e. figuring out the type of an object from
the context because there are no type declarations) to avoid dict
lookups in method dispatch.

This is not about PyPy but it might help:

http://www.python.org/pycon/dc2004/papers/1/paper.pdf

--
Ville Vainio http://tinyurl.com/2prnb

Ville Vainio

unread,
May 21, 2005, 3:30:22 AM5/21/05
to
>>>>> "Ville" == Ville Vainio <vi...@spammers.com> writes:

Ville> This is not about PyPy but it might help:

Ville> http://www.python.org/pycon/dc2004/papers/1/paper.pdf

(It's about starkiller, sorry about the opaque url)

Shane Hathaway

unread,
May 21, 2005, 4:19:08 AM5/21/05
to Torsten Bronger, pytho...@python.org
Torsten Bronger wrote:
> Hallöchen!
>
> "Kay Schluehr" <kay.sc...@gmx.net> writes:
>
>
>>[...]
>>
>>[...] Once You get enough speed out of the PyPy-runtime and the
>>community shifts to it the PEP-process degenerates in the view of
>>a PyPythonista to discussions about aspects of the std-objectspace
>>and language design patterns. There will be some CPython
>>compliance - that's all.
>
>
> Please could somebody explain to us non-CS people why PyPy could
> have speed features CPython can't have?

The idea is to shift more of the responsibility to optimize code from
the human to the computer. Since C code is at a low level, the computer
can only infer low level intent and thus perform low level
optimizations. Humans optimize C by making thousands of speed-oriented
decisions directly in the code. Python code is at a much higher level,
which should enable the computer to discover more of the programmer's
intent and perform deep optimizations.

In the end, the computer's automated optimization could turn out better
than a human's manual optimization. Thus, by expressing the Python
interpreter in a high level manner, PyPy is a first step toward deep
optimizations that aren't possible to automate in C.

Even if things don't turn out that way, note that each generation of
programming languages builds on its predecessors, and PyPy could help
bootstrap the next generation. Assemblers first had to be written in
machine code; when it was possible to write assemblers in assembly,
people started writing complex grammars and came up with C. C compilers
first had to be written in assembly; when it was possible to write C
compilers in C, people started inventing high level languages. Now
people are experimenting with high level compilers written in high level
languages. Where will this pattern lead? Who knows. :-)

Shane

Kay Schluehr

unread,
May 21, 2005, 4:19:54 AM5/21/05
to

Torsten Bronger wrote:
> Hallöchen!
>
> Paul Rubin <http://phr...@NOSPAM.invalid> writes:
>
> > Torsten Bronger <bro...@physik.rwth-aachen.de> writes:
> >
> >> Please could somebody explain to us non-CS people why PyPy could
> >> have speed features CPython can't have?
> >
> > Does the one-word answer "compiler" explain enough?
>
> No, just more questions. ;-)
>
> What's supposed to be compiled? Only PyPy itself or also the
> programs it's "interpreting"?

There is no "PyPy itself". The distinction bewteen interpreter-level
code and application-level code is nothing but a set of coding
conventions which are usefull but not necessary to let the type
inferencer ( "annotator" in PyPy slang ) terminate definitely on a set
of machine-translateable types in case of interpreter-level code. At
least the interpreter should not interpret it's own code after an
inititalisation phase. Currently type annotated application level code
will still be compiled into bytecodes but it is not only possible to
JIT and to specialize it by means of Psyco, but it should be possible
to compile parts of it into native code like we do today with
C-extensions.

It's hard for me to recognize a fixedpoint in this process or a clear
boundary between interpreter and application level code. This is IMO a
pure heuristic not a categorial distinction but is also clear there
will ever be a remaining gap due to the dynanism of the language. I
think a lot of research understanding this distinction will follow in
the next years.

As a conclusion: with PyPy Python will still be interpreted, but a
large corpus of Python code may be compiled into native code of the
underlying machine.

Regards,
Kay

Kay Schluehr

unread,
May 21, 2005, 4:33:48 AM5/21/05
to

Shane Hathaway wrote:

> Now people are experimenting with high level compilers written in
high level
> languages. Where will this pattern lead? Who knows. :-)

Drift from old Europe ( greek Pythons ) to old India to "Nagas" and
other snake-beings and goddesses :-)

http://www.khandro.net/mysterious_naga.htm#many-headed

Regards,
Kay

Carl Friedrich Bolz

unread,
May 21, 2005, 6:14:58 AM5/21/05
to Ville Vainio, pytho...@python.org
Hi!

Ville Vainio wrote:
>>>>>>"Torsten" == Torsten Bronger <bro...@physik.rwth-aachen.de> writes:
>
>
> Torsten> What's supposed to be compiled? Only PyPy itself or also
> Torsten> the programs it's "interpreting"?
>
> PyPy is written in python, if it can be compiled then the programs can
> be as well.

That's correct in the sense that if a program adherses to the same
staticness conditions as the PyPy code, it can be compiled. The core
parts of the PyPy interpreter are written in "Restricted Python"
(RPython), which imposes some limits to the features you are allowed to
use. This is done in such a way that the annotator can perform type
inference, e.g. you are not allowed to assign values with different
types to a variable (plus some more restrictions). See

http://codespeak.net/pypy/index.cgi?doc/coding-guide.html#restricted-python

for more details about RPython.

>
> Torsten> I've been told by so many books and on-line material that
> Torsten> Python cannot be compiled (unless you cheat). So how is
> Torsten> this possible?
>
> These guys are exploring a new territory. OTOH, Lisp is a dynamic
> language like python and it can be compiled to native code. Pyrex
> demonstrates the "trivial" way to compile python to native code, the
> real problem is making the resulting code fast. Typically this
> requires type inference (i.e. figuring out the type of an object from
> the context because there are no type declarations) to avoid dict
> lookups in method dispatch.

There is some preliminary documentation about the type infering (which
is called annotation here) and the translation process:

http://codespeak.net/pypy/index.cgi?doc/translation.html

Regards,

Carl Friedrich

beli...@aol.com

unread,
May 21, 2005, 6:56:24 AM5/21/05
to
Shane Hathaway wrote:

<snip>

> > Please could somebody explain to us non-CS people why PyPy could
> > have speed features CPython can't have?
>
> The idea is to shift more of the responsibility to optimize code from
> the human to the computer. Since C code is at a low level, the
computer
> can only infer low level intent and thus perform low level
> optimizations. Humans optimize C by making thousands of
speed-oriented
> decisions directly in the code. Python code is at a much higher
level,
> which should enable the computer to discover more of the programmer's
> intent and perform deep optimizations.
>
> In the end, the computer's automated optimization could turn out
better
> than a human's manual optimization. Thus, by expressing the Python
> interpreter in a high level manner, PyPy is a first step toward deep
> optimizations that aren't possible to automate in C.

I am less optimistic but hope I am wrong :).

C++ is a higher level language than C, but it's not clear to me that
compilers are able to optimize C++ code using higher-level features
such as the Standard Library so that they run as fast as the equivalent
C code. OTOH, some C++ experts have advocated template metaprogramming
as a way of speeding up programs. I wonder how widely this technique is
used.

Fortran 95 is a considerably higher level language than Fortran 77, but
I get the impression from comp.lang.fortran that it is harder to
optimize. Fortran compilers compete in a performance-driven market, and
AFAIK they are written in C.

John Roth

unread,
May 21, 2005, 8:46:37 AM5/21/05
to
"Torsten Bronger" <bro...@physik.rwth-aachen.de> wrote in message
news:87is1dy...@wilson.rwth-aachen.de...

> Hallöchen!
>
> Paul Rubin <http://phr...@NOSPAM.invalid> writes:
>
>> Torsten Bronger <bro...@physik.rwth-aachen.de> writes:
>>
>>> Please could somebody explain to us non-CS people why PyPy could
>>> have speed features CPython can't have?
>>
>> Does the one-word answer "compiler" explain enough?
>
> No, just more questions. ;-)
>
> What's supposed to be compiled? Only PyPy itself or also the
> programs it's "interpreting"?

To be more specific, the (possible) speed increase will come
from JIT (Just In Time) compilation technology. JIT technology
is quite capable of handling dynamic languages. That has to
come after they get a compilable interpreter working, but I
believe it was in the original project vision statement.

A JIT compiler within the interpreter will put PyPy pretty
much on a par with Java as far as speed goes.

> <http://www.python.org/pycon/dc2004/papers/27/>: "In the next step
> of the project, we will generate C code or machine code from the
> source of Pypy, thereby reducing the speed penalty."
>
> I've been told by so many books and on-line material that Python
> cannot be compiled (unless you cheat). So how is this possible?

JIT compilers cheat. Specifically, they compile for the observed
object environment of a statement, and then insert a test to make
sure that the actual environment matches the expected environment.
If it doesn't, it goes back to interpretation for that code segment.

John Roth

Skip Montanaro

unread,
May 21, 2005, 11:25:08 AM5/21/05
to beli...@aol.com, pytho...@python.org

beliavsky> C++ is a higher level language than C,

>From the compiler's viewpoint C++ is not much higher level than C. It has
the same basic types, (structs, unions and C++ classes are really the same
thing data-wise, though C++ classes can be somewhat more complex
layout-wise) and supports pointers to those types as well as void pointers
(pointers to untyped memory). In addition, the operators are essentially
the same.

Python has a somewhat higher-level set of objects, doesn't have pointers,
nor does it allow untyped pointers to random chunks of memory. I would
think that a run-time specializing compiler like Psyco could potentially do
more with that than a C/C++ compiler can do with the data structures and
operations it has to work with.

Skip

Mike Meyer

unread,
May 21, 2005, 12:46:32 PM5/21/05
to
Shane Hathaway <sh...@hathawaymix.org> writes:

> Torsten Bronger wrote:
> Even if things don't turn out that way, note that each generation of
> programming languages builds on its predecessors, and PyPy could help
> bootstrap the next generation. Assemblers first had to be written in
> machine code; when it was possible to write assemblers in assembly,
> people started writing complex grammars and came up with C. C compilers
> first had to be written in assembly; when it was possible to write C
> compilers in C, people started inventing high level languages. Now
> people are experimenting with high level compilers written in high level
> languages. Where will this pattern lead? Who knows. :-)

Your history of programming languages skips so many steps that it's
misleading.

For instance, C didn't arrive ab initio. It was preceeded by B, which
was a derivative of BCPL. From the history at <URL:
http://www.cs.bell-labs.com/who/dmr/chist.html >, it seems that B
slowly evolved into C. B started life as an interpreted language, with
a compiler that generated pseudo-code. The first B compiler was
written in TMG, which was a high-level language designed for creating
compilers - well, sorta. Based in that history, it seems likely that
the first program that compiled a language called C was written in B.

I'm used to seeing the term "high level languages" used for languages
a lot like C, to distinguish them from assembler. See <URL:
http://www.computerhope.com/jargon/h/highll.htm > for one
definition. "Very high level languages" used to be popular, but I
haven't seen it used much. At least one person classifies Python as
such <URL: http://www.everything2.com/index.pl?node_id=735359 >. In
any case, powerful dynamic languages - of which python is an example -
date back to LISP. The first LISP compiler in LISP almost certainly
predates C.

Basically, there's a *lot* of history in programming languages. I'd
hate to see someone think that we went straight from assembler to C,
or that people didn't understand the value of dynamic languages very
early.

<mike
--
Mike Meyer <m...@mired.org> http://www.mired.org/home/mwm/
Independent WWW/Perforce/FreeBSD/Unix consultant, email for more information.

Christian Tismer

unread,
May 21, 2005, 8:42:30 PM5/21/05
to Torsten Bronger, pytho...@python.org
Torsten Bronger wrote:

...

> I've been told by so many books and on-line material that Python
> cannot be compiled (unless you cheat). So how is this possible?

Have a look at Psyco, that will be folded into and improved
by PyPy.

Christian Tismer

unread,
May 21, 2005, 8:47:23 PM5/21/05
to Ville Vainio, pytho...@python.org
Ville Vainio wrote:

>>>>>>"Torsten" == Torsten Bronger <bro...@physik.rwth-aachen.de> writes:
>
>
> Torsten> What's supposed to be compiled? Only PyPy itself or also
> Torsten> the programs it's "interpreting"?
>
> PyPy is written in python, if it can be compiled then the programs can
> be as well.

Well, this is not really true. PyPy is written in RPython,
a sub-language of Python that is implicitly defined by
"simple and static enough to be compilable".

We have not yet started to work on the dynamic nature of
Python, that needs different technology (Psyco).

> Torsten> I've been told by so many books and on-line material that
> Torsten> Python cannot be compiled (unless you cheat). So how is
> Torsten> this possible?
>
> These guys are exploring a new territory. OTOH, Lisp is a dynamic
> language like python and it can be compiled to native code. Pyrex
> demonstrates the "trivial" way to compile python to native code, the
> real problem is making the resulting code fast. Typically this
> requires type inference (i.e. figuring out the type of an object from
> the context because there are no type declarations) to avoid dict
> lookups in method dispatch.

Type inference works fine for our implementation of Python,
but it is in fact very limited for full-blown Python programs.
Yoou cannot do much more than to try to generate effective code
for the current situation that you see. But that's most often
quite fine.

Paul Rubin

unread,
May 21, 2005, 8:57:17 PM5/21/05
to
Christian Tismer <tis...@stackless.com> writes:
> Type inference works fine for our implementation of Python,
> but it is in fact very limited for full-blown Python programs.
> Yoou cannot do much more than to try to generate effective code
> for the current situation that you see. But that's most often
> quite fine.

Type inference (or static type declarations) is one part of compiling
dynamic languages but I think its importance is overblown in these
Python compiler threads. There's lots of compiled Lisp code out there
that's completely dynamic, with every operation dispatching on the
type tags in the Lisp objects. Yes, the code runs slower than when
the compiler knows the type in advance, but it's still much faster
than interpreted code.

I'd expect one of the worst bottlenecks in Python is the multiple
levels of dictionary lookup needed when you say a.x(). The
interpreter has to search through the method dictionaries for class(a)
and all of its superclasses. It has to do this every time you do the
operation, since those dictionaries can change at any time. Being
able to do that, it seems to me, is NOT in the interest of reliable or
maintainable programming--look at the cruft in socket.py, for example.
Being able to statically generate the method call (like a C++ compiler
does) or even just being able to cache a method list in each class
(avoiding searching through all the superclasses on subsequent calls
to any operation) would probably make a big difference in execution
speed in both the compiler and interpreter. It would require a change
to the Python language but I think the change would be a beneficial
one both from the software maintainability and the performance point
of view.

Jp Calderone

unread,
May 21, 2005, 9:14:24 PM5/21/05
to pytho...@python.org
On 21 May 2005 17:57:17 -0700, Paul Rubin <"http://phr.cx"@nospam.invalid> wrote:
>Christian Tismer <tis...@stackless.com> writes:
>> Type inference works fine for our implementation of Python,
>> but it is in fact very limited for full-blown Python programs.
>> Yoou cannot do much more than to try to generate effective code
>> for the current situation that you see. But that's most often
>> quite fine.
>
>Type inference (or static type declarations) is one part of compiling
>dynamic languages but I think its importance is overblown in these
>Python compiler threads. There's lots of compiled Lisp code out there
>that's completely dynamic, with every operation dispatching on the
>type tags in the Lisp objects. Yes, the code runs slower than when
>the compiler knows the type in advance, but it's still much faster
>than interpreted code.
>
>I'd expect one of the worst bottlenecks in Python is the multiple
>levels of dictionary lookup needed when you say a.x().
> [snip]

Have you profiler data in support of this? Suggesting optimizations, especially ones which require semantic changes to existing behavior, without actually knowing that they'll speed things up, or even that they are targetted at bottleneck code, is kind of a waste of time.

Jp

Paul Rubin

unread,
May 21, 2005, 9:57:58 PM5/21/05
to
Jp Calderone <exa...@divmod.com> writes:
> Have you profiler data in support of this? Suggesting
>optimizations, especially ones which require semantic changes to
>existing behavior, without actually knowing that they'll speed things
>up, or even that they are targetted at bottleneck code, is kind of a
>waste of time.

I don't have measurements for Python (no idea whether PyPy supports
profiling, and CPython interpreters aren't very interesting since
the effect matters mostly for compiled code), but Flavors faced
a very similar problem and caching was a big win. I'm sure all
serious CLOS implementations do something similar.

holger krekel

unread,
May 22, 2005, 1:23:49 PM5/22/05
to ionel, pytho...@python.org
On Sun, May 22, 2005 at 19:18 +0200, ionel wrote:
> this is interesting
> anyway i'm to lazy to read so i'll just ask:
> can PyPy at the current state of develepment help me improve my python
> programs? (speed)

no, it can't at this stage. You might check out Psyco,
the specializing compiler for Python: http://psyco.sf.net

holger

Ville Vainio

unread,
May 23, 2005, 2:34:35 AM5/23/05
to
>>>>> "Christian" == Christian Tismer <tis...@stackless.com> writes:

>> PyPy is written in python, if it can be compiled then the programs
>> can
>> be as well.

Christian> Well, this is not really true. PyPy is written in
Christian> RPython, a sub-language of Python that is implicitly
Christian> defined by "simple and static enough to be compilable".

Could it be possible to tag some modules in application code as
RPython-compatible, making it possible to implement the speed critical
parts in RPython?

Alex Stapleton

unread,
May 23, 2005, 3:05:27 AM5/23/05
to Kay Schluehr, pytho...@python.org
The question still remains, can it run it's self? ;)

On 20 May 2005, at 23:50, Kay Schluehr wrote:

>
> holger krekel wrote:
>
>> Welcome to PyPy 0.6
>> --------------------
>>
>> *The PyPy Development Team is happy to announce the first
>> public release of PyPy after two years of spare-time and
>> half a year of EU funded development. The 0.6 release
>> is eminently a preview release.*
>>
>
> Congratulation to You and Your team!
>
> PyPy is really awesome and if it succeeds in speed demands after the
> translation phase I believe that the project will shift the power

> within the Python community on the long run. There are moments I'm
> almost shocked about it and think about the fate of other programming
> programming languages like LISP. PyPy can be resolved to "Python in
> Python" but also "Python multiplied/powered by itself" which is much
> more triumphant. A short review of the 'thunks' objspace example gives
> me the impression that the language development process as we know it
> comes to an end and makes a kind of transition. This is both very
> exciting and dangerous, like every philosophical event.
>
> Regards,
> Kay
>
> --
> http://mail.python.org/mailman/listinfo/python-list
>
>

Michael Sparks

unread,
May 23, 2005, 4:28:43 AM5/23/05
to
holger krekel wrote:

> The PyPy 0.6 release


> --------------------
>
> *The PyPy Development Team is happy to announce the first
> public release of PyPy after two years of spare-time and
> half a year of EU funded development. The 0.6 release
> is eminently a preview release.*

Yay! Congratulations guys!


Michael.
--
Michael...@rd.bbc.co.uk
British Broadcasting Corporation, Research and Development
Kingswood Warren, Surrey KT20 6NP

This message (and any attachments) may contain personal views
which are not the views of the BBC unless specifically stated.


Christian Tismer

unread,
May 23, 2005, 5:32:06 AM5/23/05
to Ville Vainio, pytho...@python.org, pypy-dev
Ville Vainio wrote:

Interesting idea.
Especially since we have automatic translation from RPythonic
application code to interpreter level.
Maybe not for now, but I'm cc-ing pypy-dev.

@rpythonic :-)

Rocco Moretti

unread,
May 23, 2005, 9:50:07 AM5/23/05
to
Alex Stapleton wrote:
> The question still remains, can it run it's self? ;)
>

I think they try, every once in a while, to self host. The only problem
at this stage of the game is the ~2000x speed slowdown. Using that
figure, a five second startup time for PyPy on CPython would take about
3 hours for PyPy on PyPy on CPython (5s*2000). Running a 1 second (on
CPython) Python program would take a month and a half for PyPy on PyPy
on CPython. (1s*2000*2000)

Once they get the speed issue licked, the self hosting problems should
be no trouble. ;)

Carl Friedrich Bolz

unread,
May 23, 2005, 1:26:14 PM5/23/05
to Rocco Moretti, pytho...@python.org
Rocco Moretti wrote:
> Alex Stapleton wrote:
>
>>The question still remains, can it run it's self? ;)
>>

This allready worked in the past, though it doesn't at the moment.

>
>
> I think they try, every once in a while, to self host. The only problem
> at this stage of the game is the ~2000x speed slowdown. Using that
> figure, a five second startup time for PyPy on CPython would take about
> 3 hours for PyPy on PyPy on CPython (5s*2000). Running a 1 second (on
> CPython) Python program would take a month and a half for PyPy on PyPy
> on CPython. (1s*2000*2000)
>
> Once they get the speed issue licked, the self hosting problems should
> be no trouble. ;)

Speed isn't even the biggest problem when running PyPy on itself. PyPy
still 'fakes' some objects, e.g. borrows them from the underlying
Python. This is mostly the case for things that have direct access to
the OS, e.g. files. If you run PyPy on PyPy on CPython you try to fake
the faked objects again, which gives trouble. Since we have to handle
faked objects differently in the future anyway we decided that at the
moment it isn't worth the effort to keep the self-hosting working.

Regards,

Carl Friedrich

Shane Hathaway

unread,
May 23, 2005, 3:26:40 PM5/23/05
to Mike Meyer, pytho...@python.org
Mike Meyer wrote:
> Basically, there's a *lot* of history in programming languages. I'd
> hate to see someone think that we went straight from assembler to C,
> or that people didn't understand the value of dynamic languages very
> early.

Yes, although I wasn't following historical events; I was following the
trends of what programmers in general have used. Theory has always been
far ahead of practice... and generalizations are never correct. ;-)

Shane

Kay Schluehr

unread,
May 23, 2005, 4:39:24 PM5/23/05
to

Carl Friedrich Bolz wrote:
> Rocco Moretti wrote:
> > Alex Stapleton wrote:
> >
> >>The question still remains, can it run it's self? ;)
> >>
>
> This allready worked in the past, though it doesn't at the moment.
>
> >
> >
> > I think they try, every once in a while, to self host. The only
problem
> > at this stage of the game is the ~2000x speed slowdown. Using that
> > figure, a five second startup time for PyPy on CPython would take
about
> > 3 hours for PyPy on PyPy on CPython (5s*2000). Running a 1 second
(on
> > CPython) Python program would take a month and a half for PyPy on
PyPy
> > on CPython. (1s*2000*2000)
> >
> > Once they get the speed issue licked, the self hosting problems
should
> > be no trouble. ;)
>
> Speed isn't even the biggest problem when running PyPy on itself.
PyPy
> still 'fakes' some objects, e.g. borrows them from the underlying
> Python.

Does it mean You create an RPython object that runs on top of CPython,
but is just an RPython facade wrapped around a CPython object? So You
have four kinds of Pythons:

RPy - translateable into LL code
APy - non-translateable but interpretable by translated RPy
RPy* - non-translateable but consistent interface with RPy. Calls
APy*
APy* - not translateable and not interpreteable by translated RPy

"Selfhosting" would imply vanishing RPy* and APy*. But the problem
seems to be that selfhosting must somehow be broken because the system
needs to interact with OS-dependend librarys. As long as You run the
system upon CPython the problem does not occur but once You drop it, a
kind of "extension objectspace" must be created which is translated
into code with nice interfacing properties. Or do You think that
RPython translations will be sufficient and another ext-objectspace is
just useless epi-cycling?

Kay

Mike Meyer

unread,
May 23, 2005, 9:47:05 PM5/23/05
to
Shane Hathaway <sh...@hathawaymix.org> writes:

Well, I'd say that generalization isn't correct. I recall a period before C
became popular when a plethora of different languages were widely used,
depending on the application domain. COBOL, FORTRAN, ALGOL, LISP, Pascal,
Snobol, PL/I, PL/360, APL, various assemblers and others all had their uses.

C (and later C++) has come to dominate a lot of application domains. But
it inherited a lot from those other languages, as did the VHLL's that
have started displacing C in some application domains. Those languages are
still worth studying, because you can see what they did wrong. And what,
in retrospect, they did right that was forgotten by C.

Carl Friedrich Bolz

unread,
May 24, 2005, 11:20:20 AM5/24/05
to Kay Schluehr, pytho...@python.org
Kay Schluehr wrote:
>>Speed isn't even the biggest problem when running PyPy on itself.
>>PyPy still 'fakes' some objects, e.g. borrows them from the underlying
>>Python.
>
>
> Does it mean You create an RPython object that runs on top of CPython,
> but is just an RPython facade wrapped around a CPython object?

yes. It means that there are objects that behave like they should in
PyPy but are implemented by keeping creating a regular CPython instance
of the object and delegating all calls on PyPy-level back to the CPython
level.

> So You have four kinds of Pythons:
>
> RPy - translateable into LL code
> APy - non-translateable but interpretable by translated RPy
> RPy* - non-translateable but consistent interface with RPy. Calls
> APy*
> APy* - not translateable and not interpreteable by translated RPy
>
> "Selfhosting" would imply vanishing RPy* and APy*. But the problem
> seems to be that selfhosting must somehow be broken because the system
> needs to interact with OS-dependend librarys. As long as You run the
> system upon CPython the problem does not occur but once You drop it, a
> kind of "extension objectspace" must be created which is translated
> into code with nice interfacing properties. Or do You think that
> RPython translations will be sufficient and another ext-objectspace is
> just useless epi-cycling?

Not exactly sure what you mean here. It's clear that we have to handle
faked objects differently to get a stand-alone PyPy version. One
possibility would be, that the RPython code calls certain functions
which are implemented in Python (can be regular Python), that are not
translated but replaced by a proper C function. For example we have a
function intmask at the moment which takes a long and removes as many
bits from it as neccessary to make it fit into an int again. This
function is left out when translating, since in C an int can obviously
not overflow to a long.

Regards,

Carl Friedrich

ionel

unread,
May 24, 2005, 2:07:50 PM5/24/05
to pytho...@python.org
so what could this PyPy do in the future ? .. concretely ...
hope this is not a stupid question

--
ionel.

Carl Friedrich Bolz

unread,
May 24, 2005, 3:11:59 PM5/24/05
to ionel, pytho...@python.org
ionel wrote:
> so what could this PyPy do in the future ? .. concretely ...
> hope this is not a stupid question
>
Maybe the description from the homepage says it best:

The PyPy project aims at producing a flexible and fast Python
implementation. The guiding idea is to translate a Python-level
description of the Python language itself to lower level languages.
Rumors have it that the secret goal is being faster-than-C which is
nonsense, isn't it?


Regards,

Carl Friedrich

holger krekel

unread,
May 26, 2005, 8:35:18 AM5/26/05
to Kay Schluehr, pytho...@python.org
Hi Kay,

On Mon, May 23, 2005 at 13:39 -0700, Kay Schluehr wrote:
> Does it mean You create an RPython object that runs on top of CPython,
> but is just an RPython facade wrapped around a CPython object? So You
> have four kinds of Pythons:
>
> RPy - translateable into LL code
> APy - non-translateable but interpretable by translated RPy
> RPy* - non-translateable but consistent interface with RPy. Calls
> APy*
> APy* - not translateable and not interpreteable by translated RPy
>
> "Selfhosting" would imply vanishing RPy* and APy*. But the problem
> seems to be that selfhosting must somehow be broken because the system
> needs to interact with OS-dependend librarys. As long as You run the
> system upon CPython the problem does not occur but once You drop it, a
> kind of "extension objectspace" must be created which is translated
> into code with nice interfacing properties.

You are mostly right but 'extension objectspace' is misleading.
Object Spaces are only responsible for manipulating Python
application objects.

To get rid of 'faked' objects we need implementations for IO
access and operating system interactions. Those can sometimes
even be written in pure python (applevel) as is the case for
a preliminary version of a file object.

> RPython translations will be sufficient and another ext-objectspace is
> just useless epi-cycling?

Conceptually, we need a good concept to perform foreign
function invocation (FFI) much like ctypes or other approaches do.
However, concretely, we might at first just write some very
low-level code (even lower level than RPython) to interact
with os-level APIs and weave that into the translation process.
This is an area we are beginning to explore in more depth currently.

cheers,

holger

Anton Vredegoor

unread,
May 26, 2005, 10:08:49 AM5/26/05
to
Carl Friedrich Bolz wrote:

> Rumors have it that the secret goal is being faster-than-C which is
> nonsense, isn't it?

Maybe not. If one can call functions from a system dll (a la ctypes,
some other poster already mentioned there was some investigation in
this area) one can skip a layer of the hierarchy (remove the c-coded
middleman!) and this would possibly result in faster code.

I'm not involved in PyPy myself but this would seem a logical
possibility. To go a step further, if the compiler somehow would know
about the shortest machine code sequence which would produce the
desired effect then there would be no reason to limit onself to only
those relatively inefficent standard code sequences that are inside
system dll's.

Just design specific optimized dll's on the fly :-)

(Now going into turbo overdive) One could have a central computer
checking which data transformations (at a polymorfic level) a specific
program is accomplishing and 'reengineer or restructure' the code
inductively to check whether some other coder already had 'said the
same thing' in 'better python code'. So one would get a warning when
reinventing the wheel even if one had invented a square one :-) or if
one had distributed functionality in an inefficent way. Next, after
standardizing the input code this way one could have a list of these
'frequently used standard sequences' memoized at the central location
in order to speed up the compilation phase. Of course the central
interpreter would be sensitive to local code history so this would ease
the code recognition process.

This would work like the way human attention works in that we recognize
the word 'wheel' sooner if we first saw a picture of a car. The only
problem with this approach seems to be that it looks like a straight
path to borghood ...

Anton

'resistance is futile, all your codes are belong to us!'

Kay Schluehr

unread,
May 28, 2005, 1:23:03 PM5/28/05
to
Anton Vredegoor wrote:

> I'm not involved in PyPy myself but this would seem a logical
> possibility. To go a step further, if the compiler somehow would know
> about the shortest machine code sequence which would produce the
> desired effect then there would be no reason to limit onself to only
> those relatively inefficent standard code sequences that are inside
> system dll's.

Are You shure that this problem is effectively solvable in any
language? Since You did not precise Your idea I'm not shure whether You
want to solve the halting-problem in PyPy or not ;)

Kay

Anton Vredegoor

unread,
May 29, 2005, 9:04:02 AM5/29/05
to

Since PyPy is covering new territory it seemed important to provide new
ideas so that they have something to look forward to and will not fall
asleep at the entrance of the new area. Maybe I failed with the new
part but at least I tried :-)

Whether they are supposed to solve the halting problem or if that can
reasonably be expected I don't now either. Is it ethical to send people
on an impossible mission in order to harvest the spinoff? Some evil
genius might have created this universe in order to do just that!

However, people posting code to this list are often reminded of other
algorithms (are you sorting this list? why not use quicksort?) so it
seems possible at least for humans to guess the intentions of another
coder sometimes, and provide better code.

Every time something is described at a higher level (these levels
cannot be found within the original system but must be created by a
leap of the imagination or by divine intervention) there seem to be
ways to remove superfluous things and be more effective even at the
lower level.

Anton

'answering all questions destroys the universe?'

Rocco Moretti

unread,
May 30, 2005, 11:42:24 AM5/30/05
to

I'm always amazed at how many people take the intractableness of the
halting problem as a reason to not even try. Sure, you can't tell if an
*arbitrary* program halts or not, but there are many where you can.
(Assuming a perfect Python Interpreter):

c = 5 + 6

halts.

c = 5
while 1:
c = c + 6

doesn't.

A trip through the Zope internals may be a little harder to decide. But
one might argue that code that does not give a clear "yes, it halts" is
poorly written, and should be rewritten to be better behaved.

At any rate, Anton isn't talking about solving the halting problem, but
is hinting more along the lines of introspecting the code and, for
example, substituting a bubble sort for a quick sort when the specific
circumstances determine that it would be quicker. Granted, giving the
"best" code sequence would be tough to impossible to determine, but a
"better" code sequence would still make people happy.

Reply all
Reply to author
Forward
0 new messages