*The PyPy Development Team is happy to announce the first public release of PyPy after two years of spare-time and half a year of EU funded development. The 0.6 release is eminently a preview release.*
What it is and where to start -----------------------------
PyPy is a MIT-licensed reimplementation of Python written in Python itself. The long term goals are an implementation that is flexible and easy to experiment with and retarget to different platforms (also non-C ones) and such that high performance can be achieved through high-level implementations of dynamic optimisation techniques.
The interpreter and object model implementations shipped with 0.6 can be run on top of CPython and implement the core language features of Python as of CPython 2.3. PyPy passes around 90% of the Python language regression tests that do not depend deeply on C-extensions. Some of that functionality is still made available by PyPy piggy-backing on the host CPython interpreter. Double interpretation and abstractions in the code-base make it so that PyPy running on CPython is quite slow (around 2000x slower than CPython ), this is expected.
This release is intended for people that want to look and get a feel into what we are doing, playing with interpreter and perusing the codebase. Possibly to join in the fun and efforts.
Interesting bits and highlights ---------------------------------
The release is also a snap-shot of our ongoing efforts towards low-level translation and experimenting with unique features.
* By default, PyPy is a Python version that works completely with new-style-classes semantics. However, support for old-style classes is still available. Implementations, mostly as user-level code, of their metaclass and instance object are included and can be re-made the default with the ``--oldstyle`` option.
* In PyPy, bytecode interpretation and object manipulations are well separated between a bytecode interpreter and an *object space* which implements operations on objects. PyPy comes with experimental object spaces augmenting the standard one through delegation:
* an experimental object space that does extensive tracing of bytecode and object operations;
* the 'thunk' object space that implements lazy values and a 'become' operation that can exchange object identities.
These spaces already give a glimpse in the flexibility potential of PyPy. See demo/fibonacci.py and demo/sharedref.py for examples about the 'thunk' object space.
* The 0.6 release also contains a snapshot of our translation-efforts to lower level languages. For that we have developed an annotator which is capable of infering type information across our code base. The annotator right now is already capable of successfully type annotating basically *all* of PyPy code-base, and is included with 0.6.
* From type annotated code, low-level code needs to be generated. Backends for various targets (C, LLVM,...) are included; they are all somehow incomplete and have been and are quite in flux. What is shipped with 0.6 is able to deal with more or less small/medium examples.
Ongoing work and near term goals ---------------------------------
Generating low-level code is the main area we are hammering on in the next months; our plan is to produce a PyPy version in August/September that does not need to be interpreted by CPython anymore and will thus run considerably faster than the 0.6 preview release.
PyPy has been a community effort from the start and it would not have got that far without the coding and feedback support from numerous people. Please feel free to give feedback and raise questions.
PyPy development and activities happen as an open source project and with the support of a consortium funded by a two year EU IST research grant. Here is a list of partners of the EU project:
Heinrich-Heine University (Germany), AB Strakt (Sweden)
holger krekel wrote: > Welcome to PyPy 0.6 > --------------------
> *The PyPy Development Team is happy to announce the first > public release of PyPy after two years of spare-time and > half a year of EU funded development. The 0.6 release > is eminently a preview release.*
Congratulation to You and Your team!
PyPy is really awesome and if it succeeds in speed demands after the translation phase I believe that the project will shift the power within the Python community on the long run. There are moments I'm almost shocked about it and think about the fate of other programming programming languages like LISP. PyPy can be resolved to "Python in Python" but also "Python multiplied/powered by itself" which is much more triumphant. A short review of the 'thunks' objspace example gives me the impression that the language development process as we know it comes to an end and makes a kind of transition. This is both very exciting and dangerous, like every philosophical event.
Kay Schluehr wrote: > holger krekel wrote: > > Welcome to PyPy 0.6 > > --------------------
> > *The PyPy Development Team is happy to announce the first > > public release of PyPy after two years of spare-time and > > half a year of EU funded development. The 0.6 release > > is eminently a preview release.*
> Congratulation to You and Your team!
> PyPy is really awesome and if it succeeds in speed demands after the > translation phase I believe that the project will shift the power > within the Python community on the long run.
Could you please explain this statement? Who will gain power, and who will lose it? Are you suggesting that CPython and PyPy developers are competing?
beliav...@aol.com writes: > > PyPy is really awesome and if it succeeds in speed demands after the > > translation phase I believe that the project will shift the power > > within the Python community on the long run.
> Could you please explain this statement? Who will gain power, and who > will lose it? Are you suggesting that CPython and PyPy developers are > competing?
I interpreted it to mean that it would bring power into the Python community from outside, e.g., Python would become more competitive with other languages that are currently beating it on performance.
>>>*The PyPy Development Team is happy to announce the first >>>public release of PyPy after two years of spare-time and >>>half a year of EU funded development. The 0.6 release >>>is eminently a preview release.*
>>Congratulation to You and Your team!
>>PyPy is really awesome and if it succeeds in speed demands after the >>translation phase I believe that the project will shift the power >>within the Python community on the long run.
> Could you please explain this statement? Who will gain power, and who > will lose it?
The Python community will gain power, and nobody will loose some. The big win is that we gain a new flexibility that did not exist before, even if PyPy should completely miss its speed promises. Having an extremely flexible implementation in a very high-level language (which happens to be Python) enables possibilities which have not been seen, before.
There is of course a chance for some community of C programmers to loose interest, if PyPy really gets as efficient as we hope for. But this is a) still a long, uncertain path and b) not a real danger, but more likely an advantage for the involved people.
> Are you suggesting that CPython and PyPy developers are > competing?
No idea how we could get onto this track. If there is a competition, then only if PyPy gets into a position where it is comparable with CPython. This is not the case, at least not in a well-ordered manner. It is not really faster, but it is definately much more flexible. Comparisons are not suitable at all, bcause there are too many qualities to compare about.
And I see no point for any competition in any future. We all love Python. It is a language, and languages are communities. If a particular implementation gets more interest for some reasons, then because it ibetter s more efficient or more interesting, whatever reasoning gives it popularity. But we are all with Python!
Surely we are comparing our performance with CPython's. This is not the real point. Note also, that many of the PyPy team members belong to CPython core developers, as well. This is not a competition, but a huge new branch, exploring what is doable and what not.
You might also give our website a try which is quite informative and gives you an insight into what we are aiming for.
PyPy is just a completely new approach to interpreted languages, almost based upon known compiler technology, but applying this in a consequent manner, that has no comparable prior example.
I wish to repeat the congratulations to the team for the first release! -----------------------------------------------------------------------
ciao -- chris
-- Christian Tismer :^) <mailto:tis...@stackless.com> tismerysoft GmbH : Have a break! Take a ride on Python's Johannes-Niemeyer-Weg 9A : *Starship* http://starship.python.net/ 14109 Berlin : PGP key -> http://wwwkeys.pgp.net/ work +49 30 802 86 56 mobile +49 173 24 18 776 fax +49 30 80 90 57 05 PGP 0x57F3BF04 9064 F4E1 D754 C2FF 1619 305B C09C 5A3B 57F3 BF04 whom do you want to sponsor today? http://www.stackless.com/
Christian Tismer <tis...@stackless.com> writes: > PyPy is just a completely new approach to interpreted languages, > almost based upon known compiler technology, but applying this in a > consequent manner, that has no comparable prior example.
Is there a web page describing what's new? Compile-and-go interactive languages have been around for decades.
Christian Tismer wrote: > beliav...@aol.com wrote:
> > Kay Schluehr wrote:
> >>holger krekel wrote:
> >>>Welcome to PyPy 0.6 > >>>--------------------
> >>>*The PyPy Development Team is happy to announce the first > >>>public release of PyPy after two years of spare-time and > >>>half a year of EU funded development. The 0.6 release > >>>is eminently a preview release.*
> >>Congratulation to You and Your team!
> >>PyPy is really awesome and if it succeeds in speed demands after the > >>translation phase I believe that the project will shift the power > >>within the Python community on the long run.
> > Could you please explain this statement? Who will gain power, and who > > will lose it?
> The Python community will gain power, and nobody will loose some. > The big win is that we gain a new flexibility that did not > exist before, even if PyPy should completely miss its speed > promises. Having an extremely flexible implementation in a > very high-level language (which happens to be Python) enables > possibilities which have not been seen, before.
But not only flexibility IN the current language but also beyond it. It's not anymore clear what the language as a set of well-defined syntactical and semantical rules really is if You can change the semantics in an arbitrary module representing an object-space. I currently don't know how modular the parser is but adding syntax-rules should not be that hard either. Once You get enough speed out of the PyPy-runtime and the community shifts to it the PEP-process degenerates in the view of a PyPythonista to discussions about aspects of the std-objectspace and language design patterns. There will be some CPython compliance - that's all.
Empowering the community means beheading the BDFL and that's currently not only a person but a principle. Well, maybe that's o.k. but at least inevitable and Guido finally finds the time to clip roses, writes his memoirs, educates children and polishs his medals of honour.
> [...] Once You get enough speed out of the PyPy-runtime and the > community shifts to it the PEP-process degenerates in the view of > a PyPythonista to discussions about aspects of the std-objectspace > and language design patterns. There will be some CPython > compliance - that's all.
Please could somebody explain to us non-CS people why PyPy could have speed features CPython can't have?
Torsten Bronger <bron...@physik.rwth-aachen.de> writes: > Please could somebody explain to us non-CS people why PyPy could > have speed features CPython can't have?
Does the one-word answer "compiler" explain enough?
Paul Rubin <http://phr...@NOSPAM.invalid> writes: > Torsten Bronger <bron...@physik.rwth-aachen.de> writes:
>> Please could somebody explain to us non-CS people why PyPy could >> have speed features CPython can't have?
> Does the one-word answer "compiler" explain enough?
No, just more questions. ;-)
What's supposed to be compiled? Only PyPy itself or also the programs it's "interpreting"?
<http://www.python.org/pycon/dc2004/papers/27/>: "In the next step of the project, we will generate C code or machine code from the source of Pypy, thereby reducing the speed penalty."
I've been told by so many books and on-line material that Python cannot be compiled (unless you cheat). So how is this possible?
Torsten> What's supposed to be compiled? Only PyPy itself or also Torsten> the programs it's "interpreting"?
PyPy is written in python, if it can be compiled then the programs can be as well.
Torsten> I've been told by so many books and on-line material that Torsten> Python cannot be compiled (unless you cheat). So how is Torsten> this possible?
These guys are exploring a new territory. OTOH, Lisp is a dynamic language like python and it can be compiled to native code. Pyrex demonstrates the "trivial" way to compile python to native code, the real problem is making the resulting code fast. Typically this requires type inference (i.e. figuring out the type of an object from the context because there are no type declarations) to avoid dict lookups in method dispatch.
>>[...] Once You get enough speed out of the PyPy-runtime and the >>community shifts to it the PEP-process degenerates in the view of >>a PyPythonista to discussions about aspects of the std-objectspace >>and language design patterns. There will be some CPython >>compliance - that's all.
> Please could somebody explain to us non-CS people why PyPy could > have speed features CPython can't have?
The idea is to shift more of the responsibility to optimize code from the human to the computer. Since C code is at a low level, the computer can only infer low level intent and thus perform low level optimizations. Humans optimize C by making thousands of speed-oriented decisions directly in the code. Python code is at a much higher level, which should enable the computer to discover more of the programmer's intent and perform deep optimizations.
In the end, the computer's automated optimization could turn out better than a human's manual optimization. Thus, by expressing the Python interpreter in a high level manner, PyPy is a first step toward deep optimizations that aren't possible to automate in C.
Even if things don't turn out that way, note that each generation of programming languages builds on its predecessors, and PyPy could help bootstrap the next generation. Assemblers first had to be written in machine code; when it was possible to write assemblers in assembly, people started writing complex grammars and came up with C. C compilers first had to be written in assembly; when it was possible to write C compilers in C, people started inventing high level languages. Now people are experimenting with high level compilers written in high level languages. Where will this pattern lead? Who knows. :-)
> >> Please could somebody explain to us non-CS people why PyPy could > >> have speed features CPython can't have?
> > Does the one-word answer "compiler" explain enough?
> No, just more questions. ;-)
> What's supposed to be compiled? Only PyPy itself or also the > programs it's "interpreting"?
There is no "PyPy itself". The distinction bewteen interpreter-level code and application-level code is nothing but a set of coding conventions which are usefull but not necessary to let the type inferencer ( "annotator" in PyPy slang ) terminate definitely on a set of machine-translateable types in case of interpreter-level code. At least the interpreter should not interpret it's own code after an inititalisation phase. Currently type annotated application level code will still be compiled into bytecodes but it is not only possible to JIT and to specialize it by means of Psyco, but it should be possible to compile parts of it into native code like we do today with C-extensions.
It's hard for me to recognize a fixedpoint in this process or a clear boundary between interpreter and application level code. This is IMO a pure heuristic not a categorial distinction but is also clear there will ever be a remaining gap due to the dynanism of the language. I think a lot of research understanding this distinction will follow in the next years.
As a conclusion: with PyPy Python will still be interpreted, but a large corpus of Python code may be compiled into native code of the underlying machine.
Shane Hathaway wrote: > Now people are experimenting with high level compilers written in high level > languages. Where will this pattern lead? Who knows. :-)
Drift from old Europe ( greek Pythons ) to old India to "Nagas" and other snake-beings and goddesses :-)
Ville Vainio wrote: >>>>>>"Torsten" == Torsten Bronger <bron...@physik.rwth-aachen.de> writes:
> Torsten> What's supposed to be compiled? Only PyPy itself or also > Torsten> the programs it's "interpreting"?
> PyPy is written in python, if it can be compiled then the programs can > be as well.
That's correct in the sense that if a program adherses to the same staticness conditions as the PyPy code, it can be compiled. The core parts of the PyPy interpreter are written in "Restricted Python" (RPython), which imposes some limits to the features you are allowed to use. This is done in such a way that the annotator can perform type inference, e.g. you are not allowed to assign values with different types to a variable (plus some more restrictions). See
> Torsten> I've been told by so many books and on-line material that > Torsten> Python cannot be compiled (unless you cheat). So how is > Torsten> this possible?
> These guys are exploring a new territory. OTOH, Lisp is a dynamic > language like python and it can be compiled to native code. Pyrex > demonstrates the "trivial" way to compile python to native code, the > real problem is making the resulting code fast. Typically this > requires type inference (i.e. figuring out the type of an object from > the context because there are no type declarations) to avoid dict > lookups in method dispatch.
There is some preliminary documentation about the type infering (which is called annotation here) and the translation process:
> > Please could somebody explain to us non-CS people why PyPy could > > have speed features CPython can't have?
> The idea is to shift more of the responsibility to optimize code from > the human to the computer. Since C code is at a low level, the computer > can only infer low level intent and thus perform low level > optimizations. Humans optimize C by making thousands of speed-oriented > decisions directly in the code. Python code is at a much higher level, > which should enable the computer to discover more of the programmer's > intent and perform deep optimizations.
> In the end, the computer's automated optimization could turn out better > than a human's manual optimization. Thus, by expressing the Python > interpreter in a high level manner, PyPy is a first step toward deep > optimizations that aren't possible to automate in C.
I am less optimistic but hope I am wrong :).
C++ is a higher level language than C, but it's not clear to me that compilers are able to optimize C++ code using higher-level features such as the Standard Library so that they run as fast as the equivalent C code. OTOH, some C++ experts have advocated template metaprogramming as a way of speeding up programs. I wonder how widely this technique is used.
Fortran 95 is a considerably higher level language than Fortran 77, but I get the impression from comp.lang.fortran that it is harder to optimize. Fortran compilers compete in a performance-driven market, and AFAIK they are written in C.
>>> Please could somebody explain to us non-CS people why PyPy could >>> have speed features CPython can't have?
>> Does the one-word answer "compiler" explain enough?
> No, just more questions. ;-)
> What's supposed to be compiled? Only PyPy itself or also the > programs it's "interpreting"?
To be more specific, the (possible) speed increase will come from JIT (Just In Time) compilation technology. JIT technology is quite capable of handling dynamic languages. That has to come after they get a compilable interpreter working, but I believe it was in the original project vision statement.
A JIT compiler within the interpreter will put PyPy pretty much on a par with Java as far as speed goes.
> <http://www.python.org/pycon/dc2004/papers/27/>: "In the next step > of the project, we will generate C code or machine code from the > source of Pypy, thereby reducing the speed penalty."
> I've been told by so many books and on-line material that Python > cannot be compiled (unless you cheat). So how is this possible?
JIT compilers cheat. Specifically, they compile for the observed object environment of a statement, and then insert a test to make sure that the actual environment matches the expected environment. If it doesn't, it goes back to interpretation for that code segment.
>From the compiler's viewpoint C++ is not much higher level than C. It has
the same basic types, (structs, unions and C++ classes are really the same thing data-wise, though C++ classes can be somewhat more complex layout-wise) and supports pointers to those types as well as void pointers (pointers to untyped memory). In addition, the operators are essentially the same.
Python has a somewhat higher-level set of objects, doesn't have pointers, nor does it allow untyped pointers to random chunks of memory. I would think that a run-time specializing compiler like Psyco could potentially do more with that than a C/C++ compiler can do with the data structures and operations it has to work with.
Shane Hathaway <sh...@hathawaymix.org> writes: > Torsten Bronger wrote: > Even if things don't turn out that way, note that each generation of > programming languages builds on its predecessors, and PyPy could help > bootstrap the next generation. Assemblers first had to be written in > machine code; when it was possible to write assemblers in assembly, > people started writing complex grammars and came up with C. C compilers > first had to be written in assembly; when it was possible to write C > compilers in C, people started inventing high level languages. Now > people are experimenting with high level compilers written in high level > languages. Where will this pattern lead? Who knows. :-)
Your history of programming languages skips so many steps that it's misleading.
For instance, C didn't arrive ab initio. It was preceeded by B, which was a derivative of BCPL. From the history at <URL: http://www.cs.bell-labs.com/who/dmr/chist.html >, it seems that B slowly evolved into C. B started life as an interpreted language, with a compiler that generated pseudo-code. The first B compiler was written in TMG, which was a high-level language designed for creating compilers - well, sorta. Based in that history, it seems likely that the first program that compiled a language called C was written in B.
I'm used to seeing the term "high level languages" used for languages a lot like C, to distinguish them from assembler. See <URL: http://www.computerhope.com/jargon/h/highll.htm > for one definition. "Very high level languages" used to be popular, but I haven't seen it used much. At least one person classifies Python as such <URL: http://www.everything2.com/index.pl?node_id=735359 >. In any case, powerful dynamic languages - of which python is an example - date back to LISP. The first LISP compiler in LISP almost certainly predates C.
Basically, there's a *lot* of history in programming languages. I'd hate to see someone think that we went straight from assembler to C, or that people didn't understand the value of dynamic languages very early.
<mike -- Mike Meyer <m...@mired.org> http://www.mired.org/home/mwm/ Independent WWW/Perforce/FreeBSD/Unix consultant, email for more information.
Ville Vainio wrote: >>>>>>"Torsten" == Torsten Bronger <bron...@physik.rwth-aachen.de> writes:
> Torsten> What's supposed to be compiled? Only PyPy itself or also > Torsten> the programs it's "interpreting"?
> PyPy is written in python, if it can be compiled then the programs can > be as well.
Well, this is not really true. PyPy is written in RPython, a sub-language of Python that is implicitly defined by "simple and static enough to be compilable".
We have not yet started to work on the dynamic nature of Python, that needs different technology (Psyco).
> Torsten> I've been told by so many books and on-line material that > Torsten> Python cannot be compiled (unless you cheat). So how is > Torsten> this possible?
> These guys are exploring a new territory. OTOH, Lisp is a dynamic > language like python and it can be compiled to native code. Pyrex > demonstrates the "trivial" way to compile python to native code, the > real problem is making the resulting code fast. Typically this > requires type inference (i.e. figuring out the type of an object from > the context because there are no type declarations) to avoid dict > lookups in method dispatch.
Type inference works fine for our implementation of Python, but it is in fact very limited for full-blown Python programs. Yoou cannot do much more than to try to generate effective code for the current situation that you see. But that's most often quite fine.
-- Christian Tismer :^) <mailto:tis...@stackless.com> tismerysoft GmbH : Have a break! Take a ride on Python's Johannes-Niemeyer-Weg 9A : *Starship* http://starship.python.net/ 14109 Berlin : PGP key -> http://wwwkeys.pgp.net/ work +49 30 802 86 56 mobile +49 173 24 18 776 fax +49 30 80 90 57 05 PGP 0x57F3BF04 9064 F4E1 D754 C2FF 1619 305B C09C 5A3B 57F3 BF04 whom do you want to sponsor today? http://www.stackless.com/
Christian Tismer <tis...@stackless.com> writes: > Type inference works fine for our implementation of Python, > but it is in fact very limited for full-blown Python programs. > Yoou cannot do much more than to try to generate effective code > for the current situation that you see. But that's most often > quite fine.
Type inference (or static type declarations) is one part of compiling dynamic languages but I think its importance is overblown in these Python compiler threads. There's lots of compiled Lisp code out there that's completely dynamic, with every operation dispatching on the type tags in the Lisp objects. Yes, the code runs slower than when the compiler knows the type in advance, but it's still much faster than interpreted code.
I'd expect one of the worst bottlenecks in Python is the multiple levels of dictionary lookup needed when you say a.x(). The interpreter has to search through the method dictionaries for class(a) and all of its superclasses. It has to do this every time you do the operation, since those dictionaries can change at any time. Being able to do that, it seems to me, is NOT in the interest of reliable or maintainable programming--look at the cruft in socket.py, for example. Being able to statically generate the method call (like a C++ compiler does) or even just being able to cache a method list in each class (avoiding searching through all the superclasses on subsequent calls to any operation) would probably make a big difference in execution speed in both the compiler and interpreter. It would require a change to the Python language but I think the change would be a beneficial one both from the software maintainability and the performance point of view.
>Christian Tismer <tis...@stackless.com> writes: >> Type inference works fine for our implementation of Python, >> but it is in fact very limited for full-blown Python programs. >> Yoou cannot do much more than to try to generate effective code >> for the current situation that you see. But that's most often >> quite fine.
>Type inference (or static type declarations) is one part of compiling >dynamic languages but I think its importance is overblown in these >Python compiler threads. There's lots of compiled Lisp code out there >that's completely dynamic, with every operation dispatching on the >type tags in the Lisp objects. Yes, the code runs slower than when >the compiler knows the type in advance, but it's still much faster >than interpreted code.
>I'd expect one of the worst bottlenecks in Python is the multiple >levels of dictionary lookup needed when you say a.x(). > [snip]
Have you profiler data in support of this? Suggesting optimizations, especially ones which require semantic changes to existing behavior, without actually knowing that they'll speed things up, or even that they are targetted at bottleneck code, is kind of a waste of time.
Jp Calderone <exar...@divmod.com> writes: > Have you profiler data in support of this? Suggesting >optimizations, especially ones which require semantic changes to >existing behavior, without actually knowing that they'll speed things >up, or even that they are targetted at bottleneck code, is kind of a >waste of time.
I don't have measurements for Python (no idea whether PyPy supports profiling, and CPython interpreters aren't very interesting since the effect matters mostly for compiled code), but Flavors faced a very similar problem and caching was a big win. I'm sure all serious CLOS implementations do something similar.
|
