First release of Shed Skin, a Python-to-C++ compiler.

[Mark Dufour]

After nine months of hard work, I am proud to introduce my baby to the
world: an experimental Python-to-C++ compiler. It can convert many
Python programs into optimized C++ code, without any user intervention
such as adding type declarations. It uses rather advanced static type
inference techniques to deduce type information by itself. In
addition, it determines whether deduced types may be parameterized,
and if so, it generates corresponding C++ generics. Based on deduced
type information, it also attempts to convert heap allocation into
stack and static preallocation (falling back to libgc in case this
fails.)

The compiler was motivated by the belief that in many cases it should
be possible to automatically deduce C++ versions of Python programs,
enabling users to enjoy both the productivity of Python and the
efficiency of C++. It works best for Python programs written in a
relatively static C++-style, in essence enabling users to specify C++
programs at a higher level.

At the moment the compiler correctly handles 124 unit tests, six of
which are serious programs of between 100 and 200 lines:

-an othello player
-two satisfiability solvers
-a japanese puzzle solver
-a sudoku solver
-a neural network simulator

Unfortunately I am just a single person, and much work remains to be
done. At the moment, there are several limitations to the type of
Python programs that the compiler accepts. Even so, there is enough of
Python left to be able to remain highly productive in many cases.
However, for most larger programs, there are probably some minor
problems that need to be fixed first, and some external dependencies
to be implemented/bridged in C++.

With this initial release, I hope to attract other people to help me
locate remaining problems, help implement external dependencies, and
in the end hopefully even to contribute to the compiler itself. I
would be very happy to receive small programs that the compiler does
or should be able to handle. If you are a C++ template wizard, and you
would be interested in working on the C++ implementation of builtin
types, I would also love to get in contact with you. Actually, I'd
like to talk to anyone even slightly interested in the compiler, as
this would be highly motivating to me.

The source code is available at the following site. Please check the
README for simple installation/usage instructions. Let me know if you
would like to create ebuild/debian packages.

Sourceforge site: http://shedskin.sourceforge.net
Shed Skin blog: http://shed-skin.blogspot.com

Should you reply to this mail, please also reply to me directly. Thanks!

Credits

Parts of the compiler have been sponsored by Google, via its Summer of
Code program. I am very grateful to them for keeping me motivated
during a difficult period. I am also grateful to the Python Software
Foundation for chosing my project for the Summer of Code. Finally, I
would like to thank my university advisor Koen Langendoen for guiding
this project.

Details

The following describes in a bit more detail various aspects of the
compiler. Before seriously using the compiler, please make sure to
understand especially its limitations.

Main Features

-very precise, efficient static type inference (iterative object
contour splitting, where each iteration performs the cartesian product
algorithm)
-stack and static pre-allocation (libgc is used as a fall-back)
-support for list comprehensions, tuple assignments, anonymous funcs
-generation of arbitrarily complex class and function templates
(even member templates, or generic, nested list comprehensions)
-binary tuples are internally analyzed
-some understanding of inheritance (e.g. list(dict/list) becomes
list<pyiter<A>>)
-hierarchical project support: generation of corresponding C++
hierarchy, including (nested) Makefiles; C++ namespaces
-annotation of source code with deduced types
-builtin classes, functions (enumerate, sum, min, max, range, zip..)
-polymorphic inline caches or virtual vars/calls (not well tested)
-always unbox scalars (compiler bails out with error if scalars are
mixed with pointer types)
-full source code available under the MIT license

Main Limitations/TODO's

-Windows support (I don't have Windows, sorry)
-reflection (getattr, hasattr), dynamic inheritance, eval, ..
-mixing scalars with pointer types (e.g. int and None in a single variable)
-mixing unrelated types in single container instance variable other
than tuple-2
-holding different types of objects in tuples with length >2;
builtin 'zip' can only take 2 arguments.
-exceptions, generators, nested functions, operator overloading
-recursive types (e.g. a = []; a.append(a))
-expect some problems when mixing floats and ints together
-varargs (*x) are not very well supported; keyword args are not supported yet
-arbitrary-size arithmetic
-possible non-termination ('recursive customization', have not
encountered it yet)
-profiling will be required for scaling to very large programs
-combining binary-type tuples with single-type tuples (e.g. (1,1.0)+(2,))
-unboxing of small tuples (should form a nice speedup)
-foreign code has to be modeled and implemented/bridged in C++
-some builtins are not implemented yet, e.g. 'reduce' and 'map'

[Paul Rubin]

Mark Dufour <mark....@gmail.com> writes:
> After nine months of hard work, I am proud to introduce my baby to the
> world: an experimental Python-to-C++ compiler.

Wow, looks really cool. But why that instead of Pypy?

[Carl Friedrich Bolz]

Hi!

adDoc's networker Phil wrote:
>>
>> experimental Python-to-C++ compiler.

>>
>> why that instead of Pypy?
>>
>

> . pypy compiles to llvm (low-level virtual machine) bytecode
> which is obviously not as fast as the native code coming from c++ compilers;

PyPy can currently compile Python code to C code and to LLVM bytecode.
Note that even for LLVM bytecode the argument is void since LLVM
(despite its name, which might lead one to think that it is Java-like)
compiles its bytecode to native assembler.

> but the primary mission of pypy
> is just having a python system that is
> written in something like python rather than c or c++

it's really just plain python (it completely runs on top of CPython
after all) together with some restrictions -- which seem similar to the
restictions that shedskin imposes btw.

> . there is no reason why the pypy project can't have a .NET architecture
> instead of the java-like arrangement I assume it has now

Sorry, I can't really follow you here. In what way does PyPy have a
Java-like arrangement?

> . without such a pypy.NET system,
> shedskin is offering a service that pypy can't yet provide:
> a ( python -> c++ )-conversion allows me to
> smoothly integrate python contributions
> with my already-staggering c++ library
> . I'm not suggesting that pypy should be another
> Mono rewritten in python,
> because the essential mission of the .NET architecture
> is being able to compile
> any language of the user`s choice,
> to some intermediate language designed to be
> far more efficiently compiled to
> any machine language of the user`s choice
> than any human-readable language such as c++
> . perhaps llvm bytecode can serve as such an intermediate language?
> then llvm could be the new c++ (our defacto IL (intermediate language))
> and shedskin (python -> IL=c++) could then be replaced by
> the combination of pypy (python -> IL=llvm)
> and some incentive for all target platforms
> to develope a highly optimized
> ( llvm -> native code)-compiler
> -- assuming also, that there is available
> a highly optimized ( c++ -> llvm bytecode )-compiler .

there is. look at the LLVM page for details: www.llvm.org

Cheers,

Carl Friedrich Bolz

[Mark Dufour]

>> After nine months of hard work, I am proud to introduce my baby to the
>> world: an experimental Python-to-C++ compiler.

>Wow, looks really cool. But why that instead of Pypy?

I agree with anyone that a JIT compiler that supports the full Python
semantics (which I thought to be the goal of PyPy?) is probably the
best long term solution. It will really be a lot of work, however, and
in general probably result in slower code than what my compiler
produces, when it works (because of run-time overheads, lack of global
optimizations.)

Considering that Shed Skin does what it does in only about 7500 lines
(5500 without C++ implementations of builtins), and is extreme in both
the requirements it puts on the compiler and in the speed of the
resulting code, for me personally it is just a very appealing
alternative to investigate (I like extremes :-)) It should work for
many programs of the type I usually write (algorithms, compilers..),
so it's also a case of scratching my own itch.

thanks!
mark.

[Paul Boddie]

Carl Friedrich Bolz wrote:
>
> > . there is no reason why the pypy project can't have a .NET architecture
> > instead of the java-like arrangement I assume it has now
>
> Sorry, I can't really follow you here. In what way does PyPy have a
> Java-like arrangement?

I imagine that this remark was made in reference to the just-in-time
compilation techniques that PyPy may end up using, although I was under
the impression that most CLR implementations also use such techniques
(and it is possible to compile Java to native code as gcj proves).

But on the subject of LLVM: although it seems like a very interesting
and versatile piece of software, it also seems to be fairly difficult
to build; my last attempt made the old-style gcc bootstrapping process
seem like double-clicking on setup.exe. Does this not worry the PyPy
team, or did I overlook some easier approach? (Noting that a Debian
package exists for LLVM 1.4 but not 1.5.)

Paul

[Michael Sparks]

Mark Dufour wrote:

> With this initial release, I hope to attract other people to help me
> locate remaining problems,

Well, you did say you want help with locating problems. One problem with
this is it doesn't build...

If I try and build (following your instructions), I get presented with a
whole slew of build errors - knock on errors from the first few:

In file included from builtin_.cpp:1:
builtin_.hpp:4:29: gc/gc_allocator.h: No such file or directory
builtin_.hpp:5:23: gc/gc_cpp.h: No such file or directory
In file included from builtin_.cpp:1:
builtin_.hpp:89: error: syntax error before `{' token
builtin_.hpp:93: error: virtual outside class declaration

Which C++ libraries are you dependent on? (Stating this would be really
useful, along with specific versions and if possible where you got them :)

For reference, I'm building this on SuSE 9.3, under which I also have
boehm-gc-3.3.5-5 installed. I suspect you're using the same gc library
(having downloaded libgc from sourceforge and finding the includes don't
match the above include names) but a different version. For reference this
version/distribution of boehm-gc has the following file structure:

/usr/include/gc.h
/usr/include/gc_backptr.h
/usr/include/gc_config_macros.h
/usr/include/gc_cpp.h
/usr/include/gc_local_alloc.h
/usr/include/gc_pthread_redirects.h
/usr/lib/libgc.a
/usr/lib/libgc.la
/usr/lib/libgc.so
/usr/lib/libgc.so.1
/usr/lib/libgc.so.1.0.1

It's specifically the gc_cpp.h file that makes me suspect it's the same gc.

Regards,

Michael.

[Carl Friedrich Bolz]

Hi Paul!

Paul Boddie wrote:
> Carl Friedrich Bolz wrote:
>>>. there is no reason why the pypy project can't have a .NET architecture
>>>instead of the java-like arrangement I assume it has now
>>Sorry, I can't really follow you here. In what way does PyPy have a
>>Java-like arrangement?
>
> I imagine that this remark was made in reference to the just-in-time
> compilation techniques that PyPy may end up using, although I was under
> the impression that most CLR implementations also use such techniques
> (and it is possible to compile Java to native code as gcj proves).

Well, PyPy is still quite far from having a JIT build in. Plus the
JIT-techniques will probably differ quite a bit from Java _and_ the CLR :-).

> But on the subject of LLVM: although it seems like a very interesting
> and versatile piece of software, it also seems to be fairly difficult
> to build; my last attempt made the old-style gcc bootstrapping process
> seem like double-clicking on setup.exe. Does this not worry the PyPy
> team, or did I overlook some easier approach? (Noting that a Debian
> package exists for LLVM 1.4 but not 1.5.)

We are not that worried about this since

a) building LLVM is not _that_ bad (you don't need to build the
C-frontend, which is the really messy part) and

b) the LLVM-backend is one of the more experimental backends we have
anyway (in fact, we have discovered some bugs in LLVM with PyPy
already). Since the C backend is quite stable we are not dependent
solely on LLVM so this is not too big a problem. Note that this doesn't
mean that the LLVM backend is not important: it's the only other backend
(apart from the C one) that can succesfully translate the whole PyPy
interpreter.

Cheers,

Carl Friedrich

[Paul Boddie]

Michael Sparks wrote:
> Well, you did say you want help with locating problems. One problem with
> this is it doesn't build...

I found that I needed both the libgc and libgc-dev packages for my
Kubuntu distribution - installing them fixed the include issues that
you observed - and it does appear to be the Boehm-Demers-Weiser GC
library, yes. The only other issue I observed was the importing of the
profile and pstats modules which don't exist on my system, but those
imports seemed to be redundant and could be commented out anyway.

Paul

[A.B., Khalid]

Mark Dufour wrote:
> After nine months of hard work, I am proud to introduce my baby to the
> world: an experimental Python-to-C++ compiler.

Good work.

I have good news and bad news.

First the good news: ShedSkin (SS) more or less works on Windows. After
patching gc6.5 for MinGW, building it, and testing it on WinXP with
some succuess, and after patching my local copy of SS, I can get the
test.py to compile from Python to C++, and it seems that I can get
almost all the unit tests in unit.py to pass.

Here is what I used:

1. shedskin-0.0.1

2. pyMinGW patched and MinGW compiled Python 2.4.1 from CVS:
Python 2.4.1+ (#65, Aug 31 2005, 22:34:14)
[GCC 3.4.4 (mingw special)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>>

3. MinGW 3.4.4:
g++ -v
Reading specs from
e:/UTILIT~1/PROGRA~1/MINGW/BIN/../lib/gcc/mingw32/3.4.4/specs
Configured with: ../gcc/configure --with-gcc --with-gnu-ld
--with-gnu-as --host=mingw32 --target=mingw32 --prefix=/mingw
--enable-threads --disable-nls
--enable-languages=c,c++,f77,ada,objc,java --disable-win32-registry
--disable-shared --enable-sjlj-exceptions --enable-libgcj
--disable-java-awt --without-x --enable-java-gc=boehm
--disable-libgcj-debug --enable-interpreter
--enable-hash-synchronization --enable-libstdcxx-debug
Thread model: win32
gcc version 3.4.4 (mingw special)

4. Also using:
- mingw-runtime 3.8
- w32api-3.3
- binutils-2.16.91-20050827-1
- gc6.5 (Bohem GC) locally patched

Now the bad news. Four tests in Unit.py fail, brief output is as
follows[1].

[SKIP 19532 lines]
*** tests failed: 4
[(60, '__class__ and __name__ attributes'), (85, 'ifa: mixing strings
and lists of strings in the same list'), (122, 'neural network
simulator XXX later: recursive customization, plus some small fixes'),
(124, 'small factorization program by Rohit Krishna Kumar')]

Moreover, and since the GC system you used only works in "recent
versions of Windows", it follows that this solution will not work in
all versions. I tested it on Win98 and both GC tests and SS's unit.py
tests crash; although SS can still seem to compile the tests to C++.

At any rate, if anyone is interested in the patches they can be
downloaded from [2].

Regards,
Khalid

[1] The entire output of unit.py can also be found at [2]
[2] http://jove.prohosting.com/iwave/ipython/Patches.html

[Carl Friedrich Bolz]

Hi Mark!

Mark Dufour wrote:
>>>After nine months of hard work, I am proud to introduce my baby to the
>>>world: an experimental Python-to-C++ compiler.

>>Wow, looks really cool. But why that instead of Pypy?
>
> I agree with anyone that a JIT compiler that supports the full Python
> semantics (which I thought to be the goal of PyPy?) is probably the
> best long term solution. It will really be a lot of work, however, and
> in general probably result in slower code than what my compiler
> produces, when it works (because of run-time overheads, lack of global
> optimizations.)

Although it is one of the goals of the PyPy project is producing a JIT
for Python at some point in the future, it also contains a _static_
Python compiler right now: The whole source code of PyPy's interpreter
is written in RPython, a restricted subset of the Python language. This
subset is choosen in a way to make type inference possible and translate
the code to a more low level language (at the moment C and LLVM are
supported). This part of PyPy, the translator
(http://codespeak.net/pypy/dist/pypy/doc/translation.html) has quite
some similarities to Shed Skin. The kind of code that Shed Skin supports
and RPython look quite similar (the biggest difference I see right now
is that RPython supports exceptions). This was probably what Paul was
referring to.

The translator is needed to translate the whole PyPy interpreter into a
low level language to get speed back, so Shed Skin and the translator
exist for the same reason.

> Considering that Shed Skin does what it does in only about 7500 lines
> (5500 without C++ implementations of builtins), and is extreme in both
> the requirements it puts on the compiler and in the speed of the
> resulting code, for me personally it is just a very appealing
> alternative to investigate (I like extremes :-)) It should work for
> many programs of the type I usually write (algorithms, compilers..),
> so it's also a case of scratching my own itch.

Shed Skin looks indeed quite impressive. If you are interested in
discussing static translation of subsets of Python a bit or are even
interested in getting to know PyPy a bit better feel free to contact us
(http://codespeak.net/pypy/dist/pypy/doc/contact.html).

Cheers,

Carl Friedrich Bolz

[Michael Sparks]

Paul Boddie wrote:

Mark's also let me know this. Part of the problem is the version in SuSE 9.3
of the GC used is ancient - it should be version 6.5 onwards. Also for
people compiling from source you (at minimum) should be using the
configure line along the lines of:

./configure --enable-cplusplus

If you don't, you get build problems because one of the needed libraries
isn't built by default.

I started off with the obvious "hello world" type program :

----------------------------------------
print "GAME OVER"
----------------------------------------

Which compiled cleanly and worked as expected. I then read Mark's short
paper linked from his blog "Efficient Implementation of Modern Imperative
Languages; Application to Python", and got concerned by the comments:

"""We have decided not to investigate two types of features: [...snip...];
and those features that may be turned off without affecting correct
programs, e.g. array bounds checking, and exceptions"""

That set some alarm bells ringing, largely because LBYL being deprecated by
many people in favour of exceptions based code. (And more to the point,
widely used as a result)

As a result, I tried a trivial, but obvious program that should have clear
behaviour:

----------------------------------------
x = []
print "GAME OVER"

x.append(5)
print x[0]
try:
print x[1]
print "This should never be seen..."
except IndexError:
print "It's OK, we caught it..."
----------------------------------------

This compiles, but unfortunately has the following behaviour:

GAME OVER
5
0
This should never be seen...
It's OK, we caught it...

Obviously, neither the 0 nor the message following should have been
displayed. It's a pity that this assumption was made, but given the short
time the project's been going I can understand it, hopefully Mark will
continue towards greater python compliance :)

Michael.

[Paul Boddie]

Carl Friedrich Bolz wrote:
> a) building LLVM is not _that_ bad (you don't need to build the
> C-frontend, which is the really messy part)

That piece of wisdom must have passed me by last time, when I probably
heeded the scary warning from the configure script and made the mistake
of getting the C front end. This time, the build process was virtually
effortless, and I'll now have to investigate LLVM further.

Thanks for the tip!

Paul

[Mark Dufour]

>Obviously, neither the 0 nor the message following should have been
>displayed. It's a pity that this assumption was made, but given the short
>time the project's been going I can understand it, hopefully Mark will
>continue towards greater python compliance :)

The latter is certainly my goal. I just haven't looked into supporting
exceptions yet, because I personally never use them. I feel they
should only occur in very bad situations, or they become goto-like
constructs that intuitively feel very ugly. In the 5500 lines of the
compiler itself, I have not needed to use a single exception. For
example, I prefer to check whether a file exists myself, rather than
executing code that can suddenly jump somewhere else. There's probably
some use for exceptions, but I don't (want to?) see it.

Seeing how often exceptions are used by other people though, it's
probably one of the first things I should look into.. :-)

thanks!
mark.

[Mark Dufour]

>First the good news: ShedSkin (SS) more or less works on Windows. After
>patching gc6.5 for MinGW, building it, and testing it on WinXP with
>some succuess, and after patching my local copy of SS, I can get the
>test.py to compile from Python to C++, and it seems that I can get
>almost all the unit tests in unit.py to pass.

Thank you so much for your efforts! I will try to download your
patches this afternoon on a roommate's Windows computer, and try to
see if I can fix the remaining tests.

>Moreover, and since the GC system you used only works in "recent
>versions of Windows", it follows that this solution will not work in
>all versions. I tested it on Win98 and both GC tests and SS's unit.py
>tests crash; although SS can still seem to compile the tests to C++.

Thanks!! Since adding GC support took about 10 lines of C++ code, I
guess it won't be hard to switch to a different system.. I'll try and
see if I can add support for a version that works with Win98..

BTW if anyone is interested in running Shed Skin under OSX.. I got
this comment on my blog (http://shed-skin.blogspot.com)

(why doesn't everybody just run Gentoo? :P)

>Wow, very cool. Congratulations!
>
>Here's what I had to do to get it working on the Mac (OS X 10.4):
>
>1. Install the garbage collector from
>http://www.hpl.hp.com/personal/Hans_Boehm/gc/
>
>2. Add #include <cmath> above #include <vector> in builtin_.hpp
>
>3. Change makelib to: g++ -dynamiclib -o libss.dylib builtin_.cpp
sets_.cpp >random_.cpp math_.cpp copy_.cpp -lgc -lm
>
>4. Edit ss to use the appropriate path
>
>5. Use python 2.4 instead of the version 2.2 or 2.3 that comes with
OS X (in my >case, I just had to put /usr/local at the start of my
path)
>
>6. Run ./ss test.py
>
>7. Compile the resulting cpp file with: g++ -L. test.cpp -lss -lgc
>
>8. Run ./a.out and watch in awe.

thanks!
mark.

[Fuzzyman]

In general it's considered quite pythonic to catch exceptions :-)

It's a particularly useful way of implementing duck typing for example.

I'm not sure if I've got *any* code that doesn't use exceptions
somewhere....

;-)

All the best,

Fuzzyman
http://www.voidspace.org.uk/python/index.shtml

>
> thanks!
> mark.

[Brian Quinlan]

Mark Dufour wrote:
> The latter is certainly my goal. I just haven't looked into supporting
> exceptions yet, because I personally never use them. I feel they
> should only occur in very bad situations, or they become goto-like
> constructs that intuitively feel very ugly. In the 5500 lines of the
> compiler itself, I have not needed to use a single exception. For
> example, I prefer to check whether a file exists myself, rather than
> executing code that can suddenly jump somewhere else. There's probably
> some use for exceptions, but I don't (want to?) see it.

I don't understand your example here. When you check that a file exists,
you feel safe that openning it will succeed? What if:

o you don't have permission to open the file
o the file is deleted in the time between you checking for it's
existance and opening it (possible race condition)
o the system doesn't have sufficient resources to open the file
e.g. too many open file handles
o the file is already open with exclusive read/write permission

Cheers,
Brian

[Mark Dufour]

You're right, I don't feel safe about that. It's a bad example. I just
prefer error codes, because the code usually becomes cleaner (at least
to me). I would really like it if I could do something like this:

f = file(name)
if not f:
print 'error opening file %s: %s', name, str(f.error)
sys.exit()

Error codes may be more work in some cases, but I really like the
(subjectively) cleaner code. Again, exceptions are probably useful in
some programs, but I wouldn't know.. :)

thanks!
mark.

[Mark Dufour]

>In general it's considered quite pythonic to catch exceptions :-)
>It's a particularly useful way of implementing duck typing for example.
>I'm not sure if I've got *any* code that doesn't use exceptions
>somewhere....

Hehe. Okay. It will probably always be the case that you have to lose
some Python features if you want the code to run really fast. I
suppose PyPy's restricted Python subset doesn't support duck typing
either. Luckily not all code is performance critical, or you could
just try and optimize some performance critical part. But anyway, I'm
starting to understand that Shed Skin should probably support
exceptions wherever possible :-)

The main goal of Shed Skin is to be able to specify C++-like code at a
higher level, not to be able to optimize arbitrary Python programs..
:-) For the kinds of things I write (algorithmic-like code), I really
don't need the full flexibility of Python. It's just great to be able
to leave out type declarations, and to use the beautiful Python
syntax.

>;-)
>All the best,

thanks!
mark.

[beli...@aol.com]

A.B., Khalid wrote:
> Mark Dufour wrote:
> > After nine months of hard work, I am proud to introduce my baby to the
> > world: an experimental Python-to-C++ compiler.
>
> Good work.
>
> I have good news and bad news.
>
> First the good news: ShedSkin (SS) more or less works on Windows. After
> patching gc6.5 for MinGW, building it, and testing it on WinXP with
> some succuess, and after patching my local copy of SS, I can get the
> test.py to compile from Python to C++, and it seems that I can get
> almost all the unit tests in unit.py to pass.

I am reluctant to attempt an arduous installation on Windows, but if
Mr. Dufour or someone else could create a web site that would let you
paste in Python code and see a C++ translation, I think this would
expand the user base. Alternatively, a Windows executable would be
nice.

[Mark Dufour]

>I am reluctant to attempt an arduous installation on Windows, but if
>Mr. Dufour or someone else could create a web site that would let you
>paste in Python code and see a C++ translation, I think this would
>expand the user base. Alternatively, a Windows executable would be
>nice.

The web site is a great idea, thanks. Maybe I will get to that, but I
have so much other things on my TODO list.. The source code of the
compiler can be downloaded, so anyone is free to tinker with it.

The problem of making a Windows executable is that the compiler itself
makes Windows executables, and therein lies the problem. It would be
nice if somebody would come up with a complete package, because I have
no access to Windows nor would I know how to approach that..

thanks!
mark.

[Fuzzyman]

Mark Dufour wrote:
> >In general it's considered quite pythonic to catch exceptions :-)
> >It's a particularly useful way of implementing duck typing for example.
> >I'm not sure if I've got *any* code that doesn't use exceptions
> >somewhere....
>
> Hehe. Okay. It will probably always be the case that you have to lose
> some Python features if you want the code to run really fast. I
> suppose PyPy's restricted Python subset doesn't support duck typing
> either. Luckily not all code is performance critical, or you could
> just try and optimize some performance critical part. But anyway, I'm
> starting to understand that Shed Skin should probably support
> exceptions wherever possible :-)
>

Ok - the point I was trying to make was that exceptions were pretty
integral to Python. I accept that losing the more dynamic features of
Python for 'compilation' is a possibly worthwhile tradeoff.

How easy is it going to be to call your c++ code from Python (and vice
versa) ?

Fuzzyman
http://www.voidspace.org.uk/python/index.shtml

[Mark Dufour]

>> Hehe. Okay. It will probably always be the case that you have to lose
>> some Python features if you want the code to run really fast. I
>> suppose PyPy's restricted Python subset doesn't support duck typing
>> either. Luckily not all code is performance critical, or you could
>> just try and optimize some performance critical part. But anyway, I'm
>> starting to understand that Shed Skin should probably support
>> exceptions wherever possible :-)

>Ok - the point I was trying to make was that exceptions were pretty
>integral to Python. I accept that losing the more dynamic features of
>Python for 'compilation' is a possibly worthwhile tradeoff.

Shed Skin's main use will be in algorithmic-like code, which is also
in most need of optimization. Typically such code is not very dynamic,
or uses many exceptions, if any. It may be the case that most programs
written in Python are not algorithmic-like at this point, which would
be logical, since Python is traditionally not one of the fastest
languages.. In that case applying Shed Skin to existing programs may
not be very useful in most cases.

But in any case, exceptions are not really dynamic (I think..) so,
yeah, they should be supported..

>How easy is it going to be to call your c++ code from Python (and vice
>versa) ?

I haven't really thought deeply about this, but I guess it shouldn't
be too hard to use existing C++/Python bridges there. However, I have
no experience at all in this area.. (hint hint)

thanks!
mark.

[Brian Quinlan]

Mark Dufour wrote:

> You're right, I don't feel safe about that. It's a bad example. I just
> prefer error codes, because the code usually becomes cleaner (at least
> to me). I would really like it if I could do something like this:
>
> f = file(name)
> if not f:
> print 'error opening file %s: %s', name, str(f.error)
> sys.exit()

I think you meant:
print 'error opening file %s: %s' % (name, str(f.error))

You forgot to check for an error when:
o when you wrote f.error [attribute "error" might not exist e.g. f is
None]
o you called str(f.error) [might contain unicode characters that can't
be converted to a string using the default
encoding]
o when you used the % operator [format string might be wrong]

And, of course, pretty much every expression can result in a memory error.

Exception handling is a boon because almost EVERY expression that you
write can result in a error and checking each one is tedious in the
extreme. So people forget and then their programs exhibit very odd
behavior that is very difficult to debug.

If you look at the Python C source, you'll notice that probably 50% of
the code is devoted to error handling (that was a guess).

Cheers,
Brian

[Mark Dufour]

> You forgot to check for an error when:
> o when you wrote f.error [attribute "error" might not exist e.g. f is
> None]
> o you called str(f.error) [might contain unicode characters that can't
> be converted to a string using the default
> encoding]
> o when you used the % operator [format string might be wrong]
> And, of course, pretty much every expression can result in a memory error.

I don't mind the program bailing out in such a case and telling me
where it went wrong, so I can fix the problem. I just don't really see
the need for catching exceptions yourself. Especially not in
well-tested code, potentially to be compiled.

> Exception handling is a boon because almost EVERY expression that you
> write can result in a error and checking each one is tedious in the
> extreme. So people forget and then their programs exhibit very odd
> behavior that is very difficult to debug.

What do you mean odd behaviour? If they don't catch exceptions, the
program will bail out, showing what went wrong, yeah?

> If you look at the Python C source, you'll notice that probably 50% of
> the code is devoted to error handling (that was a guess).

That's a lot of error handling..

thanks!
mark.

[Robert Kern]

Mark Dufour wrote:

[Brian Quinlan wrote:]

>>You forgot to check for an error when:
>>o when you wrote f.error [attribute "error" might not exist e.g. f is
>> None]
>>o you called str(f.error) [might contain unicode characters that can't
>> be converted to a string using the default
>> encoding]
>>o when you used the % operator [format string might be wrong]
>>And, of course, pretty much every expression can result in a memory error.
>
> I don't mind the program bailing out in such a case and telling me
> where it went wrong, so I can fix the problem. I just don't really see
> the need for catching exceptions yourself. Especially not in
> well-tested code, potentially to be compiled.

grep the standard library. Tons of use-cases. It really is fundamental
to the way most of us code Python.

>>Exception handling is a boon because almost EVERY expression that you
>>write can result in a error and checking each one is tedious in the
>>extreme. So people forget and then their programs exhibit very odd
>>behavior that is very difficult to debug.
>
> What do you mean odd behaviour? If they don't catch exceptions, the
> program will bail out, showing what went wrong, yeah?

He was referring to the "error code" method of error handling. People
forget to check the error code (or else wrongly assume that nothing
wrong can happen). Uncaught exceptions *do* make the program bail out at
the point of bogosity (more or less).

>>If you look at the Python C source, you'll notice that probably 50% of
>>the code is devoted to error handling (that was a guess).
>
> That's a lot of error handling..

It's code that needs to be robust but can't avail itself of exceptions.

--
Robert Kern
rk...@ucsd.edu

"In the fields of hell where the grass grows high
Are the graves of dreams allowed to die."
-- Richard Harter

[Steve Holden]

Mark Dufour wrote:
>>You forgot to check for an error when:
>>o when you wrote f.error [attribute "error" might not exist e.g. f is
>> None]
>>o you called str(f.error) [might contain unicode characters that can't
>> be converted to a string using the default
>> encoding]
>>o when you used the % operator [format string might be wrong]
>>And, of course, pretty much every expression can result in a memory error.
>
>
> I don't mind the program bailing out in such a case and telling me
> where it went wrong, so I can fix the problem. I just don't really see
> the need for catching exceptions yourself. Especially not in
> well-tested code, potentially to be compiled.
>
>

>>Exception handling is a boon because almost EVERY expression that you
>>write can result in a error and checking each one is tedious in the
>>extreme. So people forget and then their programs exhibit very odd
>>behavior that is very difficult to debug.
>
>
> What do you mean odd behaviour? If they don't catch exceptions, the
> program will bail out, showing what went wrong, yeah?
>
>

>>If you look at the Python C source, you'll notice that probably 50% of
>>the code is devoted to error handling (that was a guess).
>
>
> That's a lot of error handling..
>

Mark:

You have achieved so much with the first release of Shed Skin that it's
strange to see you apparently trying to argue that exceptions aren't
necessary when in fact they are such a fundamental part of Python's
philosophy.

Their real value is in removing the necessity to perform explicit error
checking after each operation in a program. For example, when I write

try:
func1(...)
func2(...)
except SomeException:
handle the exception

the authors of func1()) and func2() don't have to worry about explicitly
returning control if some condition they can't handle occurs. They
just write

raise SomeException

knowing that this will immediately cause all calling contexts to be
unstacked until the context is reached that is prepared to handle the
exception.

Without such an ability the logic of the functions tends to look
something like

if (errorCondition1):
return SomeErrorFlag1
elif (errorCondition2):
return SomeErrorFlag2
else:
perform normal computation
return Value

and the calling code then has to check for the flags before handling the
return value.

So basically exceptions allow simplification of program structure with
more effective and simpler error handling. If you read through (for
example) the code you'll find in the standard library and you'll get a
better feel for the usefulness of exceptions.

regards
Steve
--
Steve Holden +44 150 684 7255 +1 800 494 3119
Holden Web LLC http://www.holdenweb.com/

[A.B., Khalid]

Mark Dufour wrote:
> Thank you so much for your efforts!

Don't mention it.

> I will try to download your patches this afternoon on a roommate's Windows
> computer, and try to see if I can fix the remaining tests.
>

Okay. An update.

- Test (124, 'small factorization program by Rohit Krishna Kumar') now
passes on Windows. My mistake in my patching of SS header files. The
patch is now updated. Please download again if you have an earlier
copy.

- Test (122, 'neural network simulator XXX later: recursive
customization, plus some small fixes') was not really failing, it was
just not passing. :)

You see the Python version calls random like so:
self.weight = (random()-0.5)/2 # [float]

And testing output for random generated values is not going to make the
test pass. So if we change that line to a random value generated by
Python in the same way like so:
self.weight = (0.88996634365870131-0.5) # [float]

We have then output from Python like so:
[[0.97483328216510368], [0.50575070454495774], [0.50047176765739709],
[0.93429133063585856], [0.50083898389362214], [0.98098364981984132],
[0.5033858371718114], [0.94838636704849744], [0.50002510730868799],
[0.50910910041727786], [0.5128172933740105], [0.50010155471769424]]

and from the compiled SS generated C++ files like so:
[[0.974833], [0.505751], [0.500472], [0.934291], [0.500839],
[0.980984], [0.503386], [0.948386], [0.500025], [0.509109], [0.512817],
[0.500102]]

Which is okay I guess, wouldn't you agree?

- On to Test (85, 'ifa: mixing strings and lists of strings in the same
list'). It is still failing. It is crashing in the compiled C++
generated from SS. But did you know that the test in pure Python is
crashing as well?

"""
#test03.py
#('ifa: mixing strings and lists of strings in the same list', '''
def row_perm_rec():
hoppa_row = 'impossible' # [str]
hoppa_row = [] # [list(str)]

a = hoppa_row # [pyobj]
hoppa_row.extend(a) # []
hoppa_row.append('u') # []

return hoppa_row # [pyobj]

a = [[7]] # [list(list(int))]
s = row_perm_rec() # [pyobj]
puzzleboard = [['']] # [list(list(str))]
puzzleboard[1][1] = s[1] # [str]
#''', '''
#check('s', ['pyobj'])
#check('a', ['list(list(int))'])
#check('puzzleboard', ['list(list(str))'])
#output()
"""

Running that file produces this error in all Python versions I have.:
Traceback (most recent call last):
File "test03.py", line 15, in ?
puzzleboard[1][1] = s[1] # [str]
IndexError: list index out of range

- Finally Test (60, '__class__ and __name__ attributes') needs
patience. :) In Python it outputs the following:
__main__.evert
evert
<type 'instance'>
instance
equal str!

In the compiled C++ generated by SS it outputs the following:
class evert
evert
class evert
evert
equal!
[Big nasty crash]

So there. :)

Regards,
Khalid

[Mark Dufour]

>You have achieved so much with the first release of Shed Skin that it's
>strange to see you apparently trying to argue that exceptions aren't
>necessary when in fact they are such a fundamental part of Python's
>philosophy.

To be honest, I am a relative newcomer to Python, and Shed Skin is the
first large program I have written in it. My background is mostly
writing algorithms in C, and I don't think I've ever (voluntarily)
caught an exception in my life, or that doing so could have made my
code any cleaner. So for me personally, and for the types of programs
I usually write, I'm still not sure if they will ever be very useful.
I guess I have just naturally ignored them.. :-)

Thank you for stressing the importance of exceptions in Python in
general. I'm always happy to learn more about the language and its
philosophy..

thanks!
mark.

[Luis M. Gonzalez]

This is great news. Congratulations!

By the way, I read in your blog that you would be releasing a windows
intaller soon.
Have you, or anyone else, managed to do it?

Cheers,
Luis

[Mark Dufour]

> By the way, I read in your blog that you would be releasing a windows
> intaller soon.
> Have you, or anyone else, managed to do it?

I just finished making a 20 MB (!) package for Windows XP (I'm not
sure which older versions of Windows it will run on.) It includes the
Boehm garbage collector and a C++ compiler (MingW), which hopefully
will make it really easy to create executables. However, I'm not
releasing it until somebody with XP can test it for me :-) If you'd
like to try what I have so far, please download
http://kascade.org/shedskin-0.0.2.zip, unzip it and follow some simple
steps in the README file. I would really like to know about anything
that doesn't work, or is unclear!

BTW, I also fixed all OSX problems, but I'm waiting for a friend to
give it a final test.

What kind of program would you like to compile?

thanks!
mark.

[A.B., Khalid]

Here is the very end of a very long output of unit.py run in Python
2.4.1 on WinXP Pro SP2:

[generating c++ code..]
*** compiling & running..
rm test.o test.exe
g++ -O3 -IG:/Downloads/Temp/ss2/shedskin -c test.cpp
g++ -O3 -IG:/Downloads/Temp/ss2/shedskin test.o
G:/Downloads/Temp/ss2/shedskin/libss.a -lgc -o test
output:
[3, 3, 3, 1097, 70201]

*** success: small factorization program by Rohit Krishna Kumar 124
*** no failures, yay!

:)

Well done. So what was causing that crash in test '__class__ and
__name__ attributes' after all?

I'll also try to test it on Win98.

Regards,
Khalid

[Mark Dufour]

> *** success: small factorization program by Rohit Krishna Kumar 124
> *** no failures, yay!
>
>
> :)
>
> Well done. So what was causing that crash in test '__class__ and
> __name__ attributes' after all?

Well, I did something like this:

class_ c(..);
class_ *cp = &c;

class list {
list() {
this->class = cp;
}
}

constant_list = new list(..);

Now, depending on the order of things, I think this->class became
somewhat undefined. In any case, putting all initializations in the
right order in an initialization function called from main() fixed the
problem.

The problem with test 85 was that it should not actually be passed to
g++, since it is indeed incorrect code :-) However, because of some
bug in unit.py, on sys.platform 'win32' g++ would always be called.

Thanks again for your help. Your Makefile made it very easy for me to
create a Windows package. I'm glad to learn about Mingw too.. very
nice.

> I'll also try to test it on Win98.

I think you said the GC wouldn't work in this case..? Anyway, I've had
my share of Windows for a while.. I would be really glad if somebody
else could look into this.. :)

Have you tried compiling any code of your own yet..?

thanks!
mark.