CPython "ES" concept

[Andy O'Meara]

Hi tinypy!

I recently learned about tinypy from a thread started from
comp.lang.python. Rather than copy and paste everything from there to
this thread, I'll just post a link to it:

http://groups.google.com/group/comp.lang.python/browse_thread/thread/9d995e4a1153a1b2#

For those not looking for the big picture, I've made an abridged
version (below). However, if this topic interests you or you have
questions or commentary, please first check out the linked thread
above.

...

I'm CTO at a small software company that makes music visualization
software (you can check us out at www.soundspectrum.com). About two
years ago we went with decision to use embedded python in a couple of
our new products, given all the great things about python. We were
close to using lua but for various reasons we decided to go with
python. However, over the last two years, there's been one area of
grief that sometimes makes me think twice about our decision to go
with python...

Some background first... Our software is used for entertainment and
centers around real time, high-performance graphics, so python's
performance, embedded flexibility, and stability are the most
important issues for us. Our software targets a large cross section
of hardware and we currently ship products for Win32, OS X, and the
iPhone and since our customers are end users, our products have to be
robust, have a tidy install footprint, and be foolproof. Basically,
we use embedded python and use it to wrap our high performance C++
class set which wraps OpenGL, DirectX and our own software renderer.
In addition to wrapping our C++ frameworks, we use python to perform
various "worker" tasks on worker thread (e.g. image loading and
processing). However, we require *true* thread/interpreter
independence so CPython has been frustrating to say the
least.

Sadly, the only way we could get truly independent interpreters was
to
put python in a dynamic library, have our installer make a
*duplicate*
copy of it during the installation process (e.g. python.dll/.bundle -
>
python2.dll/.bundle) and load each one explicitly in our app, so we
can get truly independent interpreters. In other words, we load a
fresh dynamic lib for each thread-independent interpreter (you can't
reuse the same dynamic library because the OS will just reference the
already-loaded one).

After much back and forth and debate with people on comp.lang.python,
it's now clear that CPython is just too big and lumbering for there to
be any hope of getting changes in that are critical for embedded
python use. We'd like to use drop in an implementation of python
that scales and uses a interpreter "context" concept (where
interpreters are independent by means of a C object/struct and it's up
to the embedding app not to exhibit hazardous behavior). This allows
the app to have complete freedom for handling "embarrassingly easy"
parallelization scenarios. Here's some posts that go into this more:

http://groups.google.com/group/comp.lang.python/browse_thread/thread/9d995e4a1153a1b2/edae2840ab432344?lnk=st&q=video+thread#edae2840ab432344
http://groups.google.com/group/comp.lang.python/browse_thread/thread/9d995e4a1153a1b2/5c9cbb80f37f0e9e?lnk=st&q=video+thread#5c9cbb80f37f0e9e
http://groups.google.com/group/comp.lang.python/browse_thread/thread/9d995e4a1153a1b2/09aaca3d94ee7a04?lnk=st&q=video+thread#09aaca3d94ee7a04

So, our company is pretty serious about supporting the
development a CPython "ES" concept (not to mention that it'd be cool
as
hell for the community and app developers!). Of course, once I
learned about tinypy, this was exactly the CPythonES concept I was
imagining. However, upon a brief review of the codebase, my
impression is that there's some distance to go before tinypy is at a
level that app developers would really consider it for their app.
Again, I've only glanced at things, so I'd love to hear a more
detailed assessment of the codebase in the context raised here. Just
to be clear, when I say "ES", I only mean it in a good and clean API
sense (e.g. Lua), not in a literal sense that the goal is to get
python running on embedded systems. To the contrary, the goal is to
provide prospective python embedders of with a C python implementation
that has an emphasis on performance and true interpreter/"context"
independence. The biggest frustration myself and a number of other
commercial developers who really care about this issue is that a
significant portion of the CPython dev community doesn't seem to "get"
the needs of end-user, commercial software developers because many of
them are in theoretical, scientific, or academic lines of work.
Again, for more details, please check out the above links.

Meanwhile, our company is fairly invested in python at this point, so
our
interest and budget for taking tinypy to the next level is very
real. As I've indicated in my posts to comp.lang.python, we've
already had
to burn lots of time dealing with the pains (and therefore company
resources) that I've raised in my posts, in addition to
pains that I didn't get into (because of some of the folks there who
apparently like to tune out the needs of end-user app developers). My
biggest regret is that I didn't start this initiative two years ago,
when we first had to start burning resources trying to engineer ways
to make interpreters run truly independently).

Anyway, if this sort of thing is something that you
could really see yourself getting into (and getting paid for it), then
I'd love to move this all further along.

There's lot's of stuff to chew on here, and I highly recommend going
through my main posts in the above linked thread for the full
background and details.

Thanks,
Andy O'Meara

[Dean Hall]

I read through the highlights of the thread. Sounds like neat stuff
you're working on. Here are some Pros and Cons that come to mind for
your situation.

These features of tinypy would work for you:

- tinypy has a small, clean API meant to be embedded in larger apps
(much like lua's design)
- you could put an independent tinypy interpreter in separate threads
spawned within C code
- you could wrap your C/C++ libraries with tinypy and call your
functions from within python code

These features of tinypy would be a hinderance to you:

- tinypy has not been used widely and is not as "exercised" as CPython
(bugs exist).
- tinypy has math, random and pygame libraries, but that's it; no
string, re, etc.
- you would have to manually wrap your libraries, none of the
automated tools like SWIG can wrap for tinypy
- tinypy has a small developer community (1 lead developer, 2 summer
interns that are now back to school, a few misc)

!!Dean

[Andy O'Meara]

Hi Dean!

On Nov 4, 3:42 pm, Dean Hall <dwhall...@gmail.com> wrote:
> These features of tinypy would work for you:
>
> - tinypy has a small, clean API meant to be embedded in larger apps  
> (much like lua's design)
> - you could put an independent tinypy interpreter in separate threads  
> spawned within C code
> - you could wrap your C/C++ libraries with tinypy and call your  
> functions from within python code

Yep, yep, and yep!! :^)

>
> These features of tinypy would be a hinderance to you:
>
> - tinypy has not been used widely and is not as "exercised" as CPython  
> (bugs exist).

Yeah, I figure, but moving forward, that stuff will have to start to
evaporate as TP finds its way into commercial use, where we have our
microscopes and tools out to be looking for the problem areas.

> - tinypy has math, random and pygame libraries, but that's it; no  
> string, re, etc.

Understood... However, as that guy Patrick from another commercial dev
house raised in that other thread, generally the core feature set is
plenty. However, it's me feeling we'll see a rapid influx of module
development once the benefits of TP are marketed and made clear to
prospective app developers.

> - you would have to manually wrap your libraries, none of the  
> automated tools like SWIG can wrap for tinypy

Not problem there, and I venture to say that this is the case with
many app developers. The last of our problems is maintaining a macro
or header file that declares a bunch of function maps. Hah, I *wish*
that was the largest amount of work ahead. :^)

> - tinypy has a small developer community (1 lead developer, 2 summer  
> interns that are now back to school, a few misc)

Well, as I mentioned, my company is ready and willing to really jump
in with development. Now that it's clear CPython is riddled with
issues for us, I personally love the idea helping bring TP to the next
level. I look through the folks here at TP and I feel like we're
really on the same wavelength with respect to the "right" way a fully
flexible API needs to look.

Ok, so to talk turkey...

We're in the process of launching a new product in a month, so that
unfortunately ties most of our engineering resources up for the next
two months, but I'd love to start coming up with the big-picture
proposals.

First, how about everyone that'd have a potential interest in this
endeavor chime in. This would include comments such as "not
interested", "interested but can't participate", "interested and
willing to help code", "interested and willing to help design/lead",
and "interested but only for compensation". Personally and
professionally, I'm very interested and willing help code as well as
lead. Getting people to chime in here is pretty important since it
will gave the leadership and idea of where the ceiling is.

Finally, it'd be great if I could get access to the tinypy repos so I
can start a branch. I understand if that's not desired, so just let
me know if that's the case and I'll just work off my own copy (but I
anticipate this will become increasingly inefficient as our changes/
contributions grow).

Just looking at the codebase, here's several over-arching things that
strike me as the first in taking the codebase to the next level:

1) integer formalization: I see widespread use of "int" and "long" but
this of course means trouble for 64 bit portability and overall long-
term sanity. Our experience with API and interpreter design and
development has made us a big fan of using a SInt16, UInt16, SInt32,
UInt32, SInt64, UInt64 approach for all prototypes and declaration.
Proc implementations don't have to be that formal (hey, that's what
compiler conversion warnings are for), so the bottom-line amount of
work/pain here actually isn't bad at all.

2) prototype declarations: some compiler flavors/settings don't like
the implicit proc declarations that return a tp_obj (including Xcode/
gccs default settings). This is of course easily addressed with
prototype declarations.

3) C++ compatibility: like the above this one should be a straight-
forward one.

4) a simple sample project for Xcode and VS that runs some embedded
python code

We're very willing to hit the ground running on all 4 of these fronts
very rapidly if we're given a branch. If we don't get a branch, we'll
still proceed but we'll have to move forward with more of a fork
mindset (which is also fine--we look forward to sharing our work with
the TP community either way). If we go down the fork path, I
anticipate that we'll transition to a name like "CPythonES".

Beyond that, in the spirit of CPython, I anticipate us making a
"bridge" API that resembles CPython's but is implemented by TP. Of
course, as discussed, the difference would be that every proc would
ultimately have access to the respective interpreter/context object
(which would a tp_vm). The reasoning behind the CPython-like API
would be to ease the learning curve for devs like like us that see all
the payoffs of TP vs. CPython. Just to be clear, we're talking about
an API wrapper that only *resembles* CPython (and is not meant to be
an exact replacement).

Looking forward to hearing everyone's 2 cents!

Andy

[Mykhailo Parfeniuk]

Hi all!

Here are some thoughts about possible TinyPy improvements. It can solve some problems that were mentioned. First of all when I discovered TinyPy project I was looking for small implementation of scripting language, which will help me implement feature I called "auto binding". Main idea is simple: each extension module is simply DLL which also contain meta information about symbols it exports(for example for function it can be return type, export name, calling convention, alias, arguments types). For C\C++ side such information can be generated by using GCC-XML or custom C++ parser for example based on Doxygen one. Then runtime using meta information can automatically expose this symbols into language. Trying to implement this I find that llvm library can handle all problems with calling external functions(for my test example I was using DynCall library). Also I liked idea generating parser for python in python itself. Also the main advantage of using llvm is that it can compile or JIT bytecode. The tricky part is to make parser generate llvm bytecode - but I think this is solvable(PyPy has solution). Another useful feature can be using some standart type library instead of self-written types. This can impact code generation strategy.

I think such approach can help solve some problems with bindings and performance issues. But one of drawback is that it can demand significant efforts.

Best regards,

Mykhailo Parfeniuk.

[Phil Hassey]

Hey, So this is where I stand on the tinypy project -- I've pretty much done what I can to get it started :)  But I'm a bit busy with all my game dev projects to really keep it moving forward.  I would love to see it grow into something useful .. that I could actually use!  (I've got a few projects that tinypy would be quite useful for.  Some that I might use it for next year :) I like the principal of "small code" and I'd love to see that continue.  I think it actually improves the code quality of the project.  It has helped me be diligent to review my code very carefully before adding it to the project - ensuring that everything in it is essential.  This has made tinypy be quite powerful, clean, and small :)  So keep that in mind as you dev. On a general note, you appear to be quite involved in python.  How well do you understand lua?  I built tinypy by reading quite a few of the lua design docs.  So if you know lua internals, that will give you a head start into comprehending tinypy. Anyway, I'll be glad to set you folks up with a branch*.  The changes you are suggesting all sound like good improvements to tinypy.  If at all possible, it would be swell if you resolved a few of the issues that are lingering in our issues list as well.  I really want to get another release out soon, but I just haven't had the time to resolve those items. Regarding next release - is there anyone who cares about the cpython module?  If nobody does, we may have to remove it, or copy the trunk to a cpython-module branch or something so someone can care about it in the future.  I'd rather make a new release that works than have a semi-broken module sitting there forever. Cheers! -Phil * I need to know your google account info. --- On Wed, 11/5/08, Andy O'Meara <and...@gmail.com> wrote:

[Andy O'Meara]

Thanks for your thoughts, Phil... Glad you mentioned the Lua
internals thing, because I was starting to get a little nervous that
most of the code doesn't have a single comment. :^)

> * I need to know your google account info.
>

and...@gmail.com

[Andy O'Meara]

Ok, so I got into the code a bit more the other day and have the
following questions... The mailist isn't that small so if there's any
lurkers out there who have some good info to share, it'd be great if
you could post since your info could save myself and others a ton of
time.

- I don't see any debug/diagnostic code anywhere, so a big question
mark is the potential list of unknown memory or gc related issues that
be be lurking. How much diagnostic work has been done in this area,
if any? Past the 4 items listed in my other post, I would anticipate
this area to be what I'd work on next. Specifically, inserting
various debug check/diagnostic code and macros that will let us feel
confident that memory and performance are behaving the way we expect
them to.

- After looking at different parts, the trickiest part to easily get
is the VM code. Phil, do you think it'd be possible to whip up just a
really brief spec for each VM instruction? Similarly, it would be
very helpful to have even the briefest writeup of tp_vm, tp_code, and
the assumptions general principles of the TP vm. Or perhaps there's
some lua docs that are highly similar here?

- Has anyone use TP enough to gauge how it fares for performance
compared to Lua or CPython? Is anyone aware of any known performance
bottlenecks or issues w/ the TP bytecode scheme or the TP vm?

- Phil, regarding your comment on the CPython module, I don't follow
you there... What part of TP are you referring to exactly?

Thanks,
Andy

[Seth Lemons]

Andy,

I've been busy this week, so I didn't have a chance to reply earlier.
Great to see more folks interested in tinypy. I'd love to help with
the changes you've talked about. I'm a grad student, though, so the
time I have available to help will fluctuate often.

What we've done so far for debug/diagnostics is the unit tests
("tests.py" in the main code directory and most modules' directories)
and valgrind to find memory leaks. Not perfect, but it saves adding
tons of extra stuff to the code.

Have you seen the (admittedly incomplete) docs at
http://code.google.com/p/tinypy/wiki/Index?tm=6 ? I've been wanting to
add to and update the docs for a while now, but it's never quite
floated to the top of my todo list.

The CPython module refers to code written mostly by Dennis, our Summer
of Code student, as part of his sandbox project (
http://code.google.com/soc/2008/psf/appinfo.html?csaid=F659D27687537FDE
). Essentially, it's a CPython module which allows you to open a
tinypy VM inside CPython which you can use as a sandboxed environment.
Rene has also poked at the idea of using the sandboxed environment for
threads, since it bypasses the GIL and would actually run in parallel.

Once I finish my work for this week, I'll look back over this thread
and see what ideas look like ones I'd be best suited to help with. If
I don't get a chance to help much, I'll still keep an eye on the
mailing list to answer questions or share my ideas on possible
changes.

Seth

[Phil Hassey]

Hey, I think Seth answered most of the questions.  Here are a couple things that came to my mind as well: - We've used valgrind extensively to ensure that tinypy has no memory leaks or memory errors, etc.  So I think it's pretty solid.  Any further dev on tinypy I want to pass the valgrind test.  (It's even an option for the build process.) - It is faster than python for a few cases (say, rendering a mandelbrot fractal).  But it is slower than python in most other cases.  I don't have any super great benchmarks .. running the tinypy compiler code is slower, for example.  I've used callgrind a bit to track down slowness in tinypy, and most major areas of slowness have been fixed, but there is plenty of room for improvement. - Automated benchmark tests of performance against certain code samples would be nice, we don't have any of those.  - Most of the VM codes are pretty easy to follow.  Read up on lua's VM: http://www.tecgraf.puc-rio.br/~lhf/ftp/doc/jucs05.pdf tinypy isn't exactly the same, but pretty similar. Cheers! -Phil

[Phil Hassey]

As a follow-up to that, any features that are added to tinypy MUST be accompanied by a test case in tests.py.  I encourage test-first dev for tinypy - create a test case that doesn't work, and then create the code that resolves the test-case.  This has really made dev on tinypy quite easy, because if you mess anything up, the tests immediately tell you what you upset.  (Or in some cases, what you changed, and then you are able to clearly communicate to people how your change has required a modification to certain test cases, etc.) -Phil --- On Thu, 11/6/08, Phil Hassey <philh...@yahoo.com> wrote:

[Andy O'Meara]

Just to be sure, can you confirm for me that procs with a "_" prefix
are intended to be internal/private?

Andy

[John Szakmeister]

On Tue, Nov 4, 2008 at 1:19 PM, Andy O'Meara <and...@gmail.com> wrote:
[snip]

> Some background first... Our software is used for entertainment and
> centers around real time, high-performance graphics, so python's
> performance, embedded flexibility, and stability are the most
> important issues for us. Our software targets a large cross section
> of hardware and we currently ship products for Win32, OS X, and the
> iPhone and since our customers are end users, our products have to be
> robust, have a tidy install footprint, and be foolproof. Basically,
> we use embedded python and use it to wrap our high performance C++
> class set which wraps OpenGL, DirectX and our own software renderer.
> In addition to wrapping our C++ frameworks, we use python to perform
> various "worker" tasks on worker thread (e.g. image loading and
> processing). However, we require *true* thread/interpreter
> independence so CPython has been frustrating to say the
> least.

We've been there as well. Thankfully, most of our hard work is
actually I/O bound, so it's been less of a problem.

> Sadly, the only way we could get truly independent interpreters was
> to
> put python in a dynamic library, have our installer make a
> *duplicate*
> copy of it during the installation process (e.g. python.dll/.bundle -
>>
> python2.dll/.bundle) and load each one explicitly in our app, so we
> can get truly independent interpreters. In other words, we load a
> fresh dynamic lib for each thread-independent interpreter (you can't
> reuse the same dynamic library because the OS will just reference the
> already-loaded one).

Ouch. And you can't share easily between interpreters (well, you can,
but not without pickle or something similar).

[snip]

> So, our company is pretty serious about supporting the
> development a CPython "ES" concept (not to mention that it'd be cool
> as
> hell for the community and app developers!). Of course, once I
> learned about tinypy, this was exactly the CPythonES concept I was
> imagining. However, upon a brief review of the codebase, my
> impression is that there's some distance to go before tinypy is at a
> level that app developers would really consider it for their app.
> Again, I've only glanced at things, so I'd love to hear a more
> detailed assessment of the codebase in the context raised here. Just
> to be clear, when I say "ES", I only mean it in a good and clean API
> sense (e.g. Lua), not in a literal sense that the goal is to get
> python running on embedded systems. To the contrary, the goal is to
> provide prospective python embedders of with a C python implementation
> that has an emphasis on performance and true interpreter/"context"
> independence. The biggest frustration myself and a number of other
> commercial developers who really care about this issue is that a
> significant portion of the CPython dev community doesn't seem to "get"
> the needs of end-user, commercial software developers because many of
> them are in theoretical, scientific, or academic lines of work.
> Again, for more details, please check out the above links.

I think Guido's holding out for something less error prone than
threads on the concurrency front. I completely understand that
viewpoint.

[snip]

> Anyway, if this sort of thing is something that you
> could really see yourself getting into (and getting paid for it), then
> I'd love to move this all further along.
>
> There's lot's of stuff to chew on here, and I highly recommend going
> through my main posts in the above linked thread for the full
> background and details.

Myself and my company could use tinypy in this sort of context as
well. If you can help put the issues out there, I'd be happy to pick
some of them off and contribute.

-John

[Andy O'Meara]

Hey John, thanks for chiming in!

Everyone, I'm now on the tinypy dev list, so now I'll be able to start
a new branch -- I'll post here as I have updates...

Now that I've gotten into the codebase a bit, here's the path that I'm
seeing ahead...

1) Make C++ and Xcode compatible

Done -- fairly trivial stuff

2) For readability and sanity, rename/formalize/typedef various
structs and procs

Just about done... I've added some typedefs and debug-build-only
checks that provide protection from various nasty things (I've never
been a fan of casting procs and structs when typedefs and unions can
replace them for free). I'm a huge huge fan of self-documenting code,
meaning good whitespace practices, well chosen field, proc, param, and
var names, and easy-to-follow logic and control--that stuff always
saves myself and others *tons* of time in the log run.

3) Make first draft of "bridge" API that resembles CPython but is
implemented by TP

I'm still mulling over this one... The biggest decision is to go with
a ref counting scheme (like CPython) or not--I'm currently leaning
towards ref counting. The more this API looks like CPython, (a) the
lower the learning curve and (b) the easier it will be for people to
interchange their CPython host code with this new API. Speaking of
that, it'd be nice to commit to a API name, so if you have any other
suggestions other than "CPythonES", then please chime in. Other ideas
are PythonES, PyES, and CPythonTP. I personally like "ES" somewhere
in the name because it says a lot about the nature of the API and who
it's intended for (like OpenGLES, for example).

4) Make a handful of test and sample embedded projects for VS and
Xcode

These will set up the next step...

5) Closely evaluate and analyze performance and compare with CPython.

We need to build up our knowledge on TP's performance characteristics
and a feel for its strengths and weaknesses.

6) *IF* performance is generally lower to CPython, look into root
issues and improve. If performance is still an issue, look at for VM
design improvements.

The over-arching idea here is to find out if the TP VM needs design
enhancements to get it at a level most app developers would be happy
with. Otherwise, TP/CPythonES will be a tough sell to prospective
developers if performance is worse the CPython--so basically we need
to get our arms around if the VM (or its data structures) need a
performance overhaul. This area is my bread and butter, so I feel
pretty good there.

So, overall, there's a lot of fun and promising things ahead.
Unfortunately, time is scarce for me but the interest is definitely
there... Assuming the VM gives us hope that we can equal or greatly
exceed CPython's performance (which I'm confident of, based on what
I've seen of CPython and my own background), I have some options with
respect to support and funding to increase the development pace.
Before that, however, this new "ES" API needs to be really thought out
and designed carefully, so I'll be looking forward to everyone's input
on my drafts! :^)


Andy

[Dean Hall]

I disagree with point (a). I believe having to learn where and why to
incr/decr a refcount adds to the learning and the opportunity for
errors. Automatic memory management increases reliability; I consider
refcounting to be only semi-automatic. I agree with point (b), but I
think if you're porting from CPython, then deleting or defining the
incr/decr macros as empty would also be easy.

!!Dean

[Phil Hassey]

Hey, #2 - tinypy uses incremental garbage collection .. so you just plain don't need to bother with reference counting. #3 - why?  tinypy has an external API already.  Not that I'm against having a Cpython-like API for convenience of python coders .. is that the reason?  (I guess that's a decent reason .. it just wasn't one of my reasons, since I don't know the Cpython-API.) Cheers! -Phil

--- On Mon, 11/10/08, Andy O'Meara <and...@gmail.com> wrote:

From: Andy O'Meara <and...@gmail.com> Subject: [tinypy] Re: CPython "ES" concept To: "tinypy" <tin...@googlegroups.com>

[Andy O'Meara]

On Mon, Nov 10, 2008 at 1:29 PM, Dean Hall <dwha...@gmail.com> wrote:
>
> I disagree with point (a). I believe having to learn where and why to
> incr/decr a refcount adds to the learning and the opportunity for
> errors. Automatic memory management increases reliability; I consider
> refcounting to be only semi-automatic. I agree with point (b), but I
> think if you're porting from CPython, then deleting or defining the
> incr/decr macros as empty would also be easy.

I definitely agree on all counts... I should have underscored that I
haven't gotten into the issue intensely yet, so all my thoughts on the
issue are highly preliminary... In principle, I agree, but there may
be some special or proxy objects or something that make sense...
Again, I have no idea yet and I'm just speaking generally. Ideally,
module authors should be able to create modules with as much power as
CPython, so that will play a large part into what the bridge API needs
to be able to support, etc.

[Andy O'Meara]

On Mon, Nov 10, 2008 at 2:14 PM, Phil Hassey <philh...@yahoo.com> wrote:
> Hey,
>
> #2 - tinypy uses incremental garbage collection .. so you just plain don't
> need to bother with reference counting.

Yeah, but I'm trying to think through all the cases (including ones
where it may make sense to offer alternative C idioms to make all that
machinery that much more easier to use).

>
> #3 - why? tinypy has an external API already. Not that I'm against having
> a Cpython-like API for convenience of python coders .. is that the reason?
> (I guess that's a decent reason .. it just wasn't one of my reasons, since I
> don't know the Cpython-API.)
>

In my mind, the likeness to CPython is pretty key in my mind (even if
the idioms turn out to be empty macros as Dean mentioned) for some big
reasons:
- Easy for prospective app developers to switch from CPython (my
company, for starters!) -- this is a BIG one, imho. Also, this is a
perfect way for me to learn TP inside and out and offer some
implementation flexibility.
- Docs... Since tp's current documentation is kind on the thin
side, everyone wins from just being able to look at CPython's docs and
conventions.
- No detriment to tp or its performance -- my intention is to make
this API as pass-through and macro based as possible. Also, folks can
simply not use it if they choose.
- Less verbose... I'm a big believer that APIs need to permit
succinct code (not to be confused with overly succinct, abbreviated,
or otherwise confusing code), so if it take multiple statements to do
commonplace things (such as make and track an object) then we need to
look at ways to improve this (of course, macros and good documentation
are often the way to go).

Honestly, as I mentioned to Dean, I can't say much more until I start
to see what it takes for existing CPython client code to mesh with TP.
It could be the case macros do it, but it could be the case that a
more substantial wrapper is needed. As I mentioned, I think in the
long run this will add a lot of power and strength to this project
since it allows the TP implementation to have more degrees of freedom.

[Andy O'Meara]

Quick question...

I was wondering what the motivation for using C89 was (as opposed to
C99). Is there any real reason to stay married to C89 in this
project? I definitely appreciate uber-compatible codebases, but
there's a point where the PITA factor gets bad (such as // comments).


Andy

[Krzysztof Kowalczyk]

Windows (msvc) support.

-- kjk

[Rockins Chen]

I think #2 is a good idea, but #3 is not. Why make TP's API compatible with CPython? Don't you find CPython's API too complicated and too obscure? I remembered that one aim of TP is to make a lightweight interface, rather than a more heavy one. Actually, I think it's time to reduce and clarify TP's API, rather than adding more messing stuffs into it.

F.Y.I.

BRs,

Rockins Chen

2008/11/11 Andy O'Meara <and...@gmail.com>

--
BRs,
Rockins Chen
~~~~~~~~~~~~~~~~~~~~~~~~~~~~
UESTC, Chengdu, Sichuan, China
http://rockins.cn/
http://rockins.cublog.cn
ybc...@gmail.com

[Andy O'Meara]

Perhaps you can be a little more specific? I'm aware that MSVC
doesn't have all the C99 math functions, but as far as I can tell none
of those are in use (not to mention the trivial workaround to just add
them explicitly).

[Andy O'Meara]

As I mentioned in my response to Phil, there's a pretty big reasons to
offer a wrapper that are CPython friendly.

The key is to look at it from the point of view of a developer that's
currently using CPython but is interested in trying TP out. That
person or company will have zero interest in editing hundreds of lines
of their code (or more) just to *try* TP--it just won't happen. If
they can drop something in that is nearly transparent (load wise) and
minimal changes are needed, now you're talking a different ball game.
As I also mentioned, then you've also got the learning curve
liability.

My feeling here is that if we do this API right, people are likely to
take a *real* interest in this endeavor (not that I care about
popularity, but it would be cool to see the light go on for a lot of
people who are frustrated with CPython and just want to ship their
software and not bork with their code.

Andy

[Krzysztof Kowalczyk]

MSVC doesn't support C99. I can't give you specifics because the
extent of C99 non-support isn't documented in detail, as far as I
know.

The simplest approach to making sure that the code compiles on msvc is
to force C89 mode with gcc. That greatly decreases probability that
people who don't compile with msvc will break msvc build.

-- kjk

[Dean Hall]

I believe the number of people wanting to develop for TP that already
have CPython C code is going to be smaller than people who simply want
to develop for TP.

Also, I believe there would not be a learning period for a petite TP
API because the TP API reflects exactly the Python operations and
class methods developers already know with only a little extra overhead.

So, my vote would be against a CPython mimic wrapper. But, as this is
your branch, I recognize you are free to do as you choose.

!!Dean

[Andy O'Meara]

> I believe the number of people wanting to develop for TP that already
> have CPython C code is going to be smaller than people who simply want
> to develop for TP.

Well, the issue is that no dev in their right mind will want to change
their existing code to any large extent to even try TP. In my mind, a
goal is to welcome prospective embedders and developers, not to make
it easy for them to walk away from at least trying TP.

For example, that last thing I want to do is convert my existing
codebase to use TP's API. Unfortunately, TP basically has no module
support right now so in the event that I throw my hands up on this
project for whatever reason (most likely because of lack of time), I
can't be married to TP's API. For my case alone, it makes sense.

>
> Also, I believe there would not be a learning period for a petite TP
> API because the TP API reflects exactly the Python operations and
> class methods developers already know with only a little extra overhead.
>

Well, there's always a learning curve unfortunately... What needs
ownership statements, what doesn't? What are the expectations and
specs for controlling gc? Also, unless I'm missing something, there's
little to no docs that discuss conventions, protocols, and client/impl
expectations and responsibilities. For example, I'm new to the scene
and keep finding myself looking at all kinds of .c files just to
understand what certain functions are doing and how the params are
used. Keep in mind that when you're a developer in a codebase it's
pretty tough to have an outside perspective on it. Take me for
example: I'm on the hook for supporting, maintaining, and growing a
codebase that has about 400k paid customers and when I look at TP and
its lack of documentation and shipping projects, I have some concerns.
I don't mean this in any negative way, but is anyone here actually
shipping TP in paid or official software (i.e. software that if it
requires extensive support or becomes unmaintainable, you can't pay
your mortgage or you have to lay off people)?

While we're talking about liabilities, one area of concern is the
maintainability of encode.py and parse.py. My fear is that if/when
there's a bug or it needs to be extended, it's sort of scary to think
about getting into that code and making changes.

As for TP already having gc, unless I'm missing something, at the C
level one still needs to communicate to the gc when an object is no
longer owned. For example, if your app wants to keep some object
around for easy access at the C level, when it is done with it, it
needs to ultimately needs to tell the gc that it's no longer
referenced. Point being, weather it's reference counting or gc calls,
there's no escape from required calls at the C level to manage object
ownership.

So please don't misunderstand any of this as throwing stones or trying
to be negative. I think TP is impressively cool and really on the
right track with respect to how a python impl should look, but we
unfortunately need to recognize and identify some of the important
hallmarks of industrial-caliber packages that app developers look for
and really count on.

Andy

[Dean Hall]

An automatic GC does not need any attention from the C developer other
than to make sure any new objects are reachable from the roots list (a
pointer suffices). During a sweep, the roots list is followed to find
all objects that are in use and the remaining unused chunks are
reclaimed. The sweeper is invoked according to some policy, which is
incremental according to Phil. I haven't studied TP's GC yet, but I
have studied GC theory and Incremental is the way to go for embedded
systems. It avoids the delays of a full mark & sweep when memory is
needed, so it is a good tradeoff to have automatic memory management,
but not take large chunks of CPU time during sweep.

!!Dean

[Andy O'Meara]

On Tue, Nov 11, 2008 at 11:03 AM, Dean Hall <dwha...@gmail.com> wrote:
>
> An automatic GC does not need any attention from the C developer other
> than to make sure any new objects are reachable from the roots list (a
> pointer suffices).

Right, so at the C level, one ultimately needs to message the gc when
an object is about to go away. That's what I meant about regarding my
comment that there's no escape from required calls at the C level to
manage object ownership (and hence CPython's approach to use ref
counting). I appreciate the benefits of gc, but unfortunately as long
as we're talking C/C++, object owners need to manage their ownership
(in one way or another--whether that's tp_track() and tp_delete() or
an add/release ref scheme).

Speaking of the gc, without any docs, it's a bit tricky to understand
TP's conventions of ownership (ie. gc C level communication) under
TP... For example, tp_track() will cause strings already in the
string dict to be deleted from under you (so the caller has to be
careful--and knowledgeable--about what the gc is doing and you'd only
ever know by looking at the implementation). Also, it's not clear how
ownership of objects is moved into or out of the VM (to/from the host
app) -- this is probably just me not finding the right sample code.
All this stuff is part of the learning curve I was referring to.

[Dean Hall]

No, really! The GC finds reachable objects and reclaims non-reachable
objects automatically.

http://en.wikipedia.org/wiki/Garbage_collection_(computer_science)

!!Dean

[Andy O'Meara]

On Wed, Nov 12, 2008 at 1:10 PM, Dean Hall <dwha...@gmail.com> wrote:
>
> No, really! The GC finds reachable objects and reclaims non-reachable
> objects automatically.
>
> http://en.wikipedia.org/wiki/Garbage_collection_(computer_science)
>

Sure, but keep in mind that all those code samples of syntactic
garbage are in a managed language layer. When you're talking about
storing a reference to an object in a C/C++ struct/class, you still
need to somehow message the gc when a reference goes away.

Maybe the disconnect here is that I'm operating under the assumption
that an embedable interpreter has has to allow the C/C++ level to keep
references to objects, one way or another. From my experience, it's
pretty essential to allow the C layer store a reference to certain
objects for easy and fast access otherwise there can be a pretty
serious hit on performance when you're talking about a python method
that keeps making calls back into the C layer and the C callback has
to then re-fetch everything again (rather than use stored references)
before it can commence work.

[illume]

On Nov 6, 12:37 pm, Phil Hassey <philhas...@yahoo.com> wrote:
>
> Regarding next release - is there anyone who cares about the cpython module?  If nobody does, we may have to remove it, or copy the trunk to a cpython-module branch or something so someone can care about it in the future.  I'd rather make a new release that works than have a semi-broken module sitting there forever.
>

Hi,

I care about it. I've been on holiday for a month :) However I
haven't had any of my issues/patches addressed which I posted (I
think)? If I had svn access I could move it forward.

(Haven't caught up on all the tinypy emails yet... and a month of
other emails).


cu,

[illume]

Hi,

one approach I've been thinking of is a CPython API emulation layer.
The idea with this was to make porting CPython modules easier.

However I just got to the basics of it... I imagine it would actually
be impossible... but you never know. Being able to have a module that
compiles under cpython or tinypy was the goal... However I'm not sure
if that's possible really.

My notes are in the pages section of the tinypy group.


cu,


On Nov 11, 6:23 am, "Andy O'Meara" <and...@gmail.com> wrote:

[Phil Hassey]

Rene, I've added you to the project.  As a reminder, at least for starters, post patches to the ML before committing to subversion.  That way we can get some peer review on things.  I'll do my best to be prompt in my part :)  (And I'm always glad if others chip in on that too!) Thanks! -Phil --- On Sun, 11/16/08, illume <ren...@gmail.com> wrote:

From: illume <ren...@gmail.com> Subject: [tinypy] cpython module... Re: CPython "ES" concept To: "tinypy" <tin...@googlegroups.com>