Project Phoenix update
Robin Dunn
http://wiki.wxpython.org/ProjectPhoenix
http://groups.google.com/group/wxPython-dev/browse_thread/thread/451c5420afc8d402#
I just love it when a plan comes together and it actually works even
better than I thought it would at the outset.
Executive Summary
-----------------
Last week I reached my first major milestone, the successful completion
of what I've been calling the end-to-end test, you can also think of it
as a proof-of-concept test. In a nutshell I've got everything in place
to be able to go from the XML output from Doxygen for selected classes
and headers, build a collection of objects that represent that
information[1], tweak[2] the objects for special cases as needed, and
generate code[3] for the wrapper tool (SIP) used to create the Python
binary extension modules.
More Details
------------
If you don't care about all the gory details feel free to skip the rest
of this message. ;-)
Code is currently being generated for about 100 classes and their
supporting enums, functions, etc. This is enough to be able to create a
wxApp and a wxFrame, bind event handlers, have those events delivered to
the handlers, run the main event loop, etc. This was the criteria I had
set for myself to be able to call this test a success.
In addition, I've also gotten the following more advanced things working
along the way, or have been able to take advantage of SIP features that
implement them:
* Build scripts that are much cleaner and more sane that the hodge-podge
that had grown up for the old wxPython over the years. It automatically
(mostly) deals with the dependencies between the ETG scripts, the SIP
source and the generated C++ source. It works well, but there is still
a chance that I may dump distutils and use something a little more powerful.
* Function/method overloading works well, without losing the ability to
use keyword arguments and without requiring us to rename the overloads.
I do have to explicitly ignore some overloads if one or more of them
would end up with argument signatures in Python that are ambiguous
and/or can't be distinguished from each other at runtime.
* Overriding of C++ virtual methods in Python classes is supported,
works well and is easy to do, although I did have some troubles in some
cases. See the Speed Bumps section for more info.
* Along the way a lot more classes have been added than just those
needed for the end-to-end test since they become required if they are
used for parameter or return types in the classes that I do need for the
test. In some cases I just added a forward declaration for those
classes so I could avoid opening those cans of worms until later, but
most of the time I just added the class since it's so easy to do.
* Things like autoconverting between wxString and Python string or
unicode objects, or allowing a sequence of 2 integers to be passed
instead of wxPoint or wxSize objects are implemented, and are all
somewhat simpler than how it was done with SWIG.
* A unit test suite has been started[4]. Mostly I've just done tests
for things that are customized or tweaked in ways that I'm not certain
of at the outset, and I assume that those things that are wrapped
without tweaks are working okay and don't need tests. However that
should be changed later.
Speed Bumps
-----------
* There were some errors in the interface header code that Doxygen
processes and so the XML processed by some etg scripts were flawed to
the point of causing compilation errors when building the extension
modules. These were fairly easy to fix.
* There were also some items missing from the interface files that I
discovered when trying to implement things like the same properties that
are already in the old wxPython classes. So I've added some methods and
some classes where needed, although mostly with very sparse (or missing)
documentation.
* The way that the backend generators (that I've investigated so far)
handle the overriding of C++ virtual methods means that they not only
need to know that the method is virtual in the base class, but they need
to know which of the derived classes override it. This is because if
the method is not implemented in the Python derived class, or if the
base version is being called from the Python class, then the C++ code
will do something like "instance->BaseClass::methodName()" in order to
avoid looping into the derived class again. The problem happens because
the wx interface files do not usually document where the virtuals are
reimplemented, and even if they did it can vary by platform. For
example wxWindow::Show is reimplemented in wxTopLevelWindow on Windows,
and in wxFrame on Mac. Since the backend generator only knew about
Show() in the wxWindow class then there were times that wxWindow::Show
was explicitly called when it should have been calling wxFrame::Show.
The workaround is not difficult, but brings with it a new set of
questions. By adding a declaration for the virtual in the derived
classes then the backend generator would act as if it was overridden in
the derived class and in the example above would call wxFrame::Show and
then C++ would route the call to the actual class in the hierarchy where
the appropriate Show method is implemented. This is good, but it has
some problems for the Tweaker stage of the etg scripts. Namely it will
either have to know a lot more about the implementation than is
represented in the interface files, (meaning much more maintenance work)
and most likely would require generating different versions of the
wrapper code for each platform (which is something I wanted to avoid
this time) or we would have to add declarations for all virtuals to all
derived classes, which would end up with a lot of code bloat.
I finally decided on a solution that is a compromise of sorts. Instead
of allowing every virtual C++ method to be overridden in Python classes
the ETG scripts remove the virtual keyword from all methods and then add
it back (or add a new declaration to the class if needed) for just a few
selected methods where it is very likely that people will want to
reimplement the method in Python. (For example, it is very unlikely
that Show will ever need to be overridden in a Python class, but
DoGetBestSize will be in a large number of cases.) This helps eliminate
some of the bloat and reduces the number of methods that need to have
close attention, and it is also easily automated. Taking this approach
reduced the number of virtuals in wxWindow (that the backend will see)
from about 137 to about 39.
* I've run into only one kludge from the old wxPython that is more
difficult to implement with SIP than it was with SWIG. The wx.PyEvent
and wx.PyCommandEvent classes are special in that when they are
Clone()'d any Python attributes that were set in the original object are
carried over to the clone object. With SWIG I could do this simply by
giving the clone a reference to the original event's proxy object, and
then later on passing the original to the event handler instead of the
cloned object. Since the wrapper classes are more tightly coupled with
the C++ classes in SIP (they are actual Python Types instead of simple
classes that just glue together a collection of functions) then a kludge
like the old one simply would not work. Three or four additional ideas
I had would not work either (although one of them might in the future if
my enhancement request is implemented in SIP.) Finally I figured out
that I could get or create the wrapper object for the existing event and
the newly cloned one, and then copy the Python attributes from the old
to the new[5]. This makes the Clone method be fairly ugly, and I also
have to customize the Clone method wrapper a bit, but it seems to work
well so far, although I still need to verify proper reference counts and
object ownership in one use case. The silver lining of this problem is
that without the old kludge I no longer have to have ugly special case
code in wxPyCallback::EventThunker. Currently all that code is
commented out and there really isn't much code left in the new
EventThunker, you can see it at [6].
[1] See http://trac.wxwidgets.org/browser/wxPython/Phoenix/trunk/etgtools
[2] See: http://trac.wxwidgets.org/browser/wxPython/Phoenix/trunk/etg
Notice that most of the tweaks are just for ignoring the things we don't
want or can't use yet, adding Properties, adding custom implementations
for some methods, etc.
[3] http://trac.wxwidgets.org/browser/wxPython/Phoenix/trunk/sip/gen
[4] http://trac.wxwidgets.org/browser/wxPython/Phoenix/trunk/unittests
[5] http://trac.wxwidgets.org/browser/wxPython/Phoenix/trunk/src/pyevent.sip
[6]
http://trac.wxwidgets.org/browser/wxPython/Phoenix/trunk/src/event_ex.cpp
--
Robin Dunn
Software Craftsman
http://wxPython.org
Robin Dunn
> Background info:
> http://wiki.wxpython.org/ProjectPhoenix
>
> http://groups.google.com/group/wxPython-dev/browse_thread/thread/451c5420afc8d402#
>
>
>
> I just love it when a plan comes together and it actually works even
> better than I thought it would at the outset.
>
>
> Executive Summary
> -----------------
> Last week I reached my first major milestone, the successful completion
> of what I've been calling the end-to-end test, you can also think of it
> as a proof-of-concept test. In a nutshell I've got everything in place
> to be able to go from the XML output from Doxygen for selected classes
> and headers, build a collection of objects that represent that
> information[1], tweak[2] the objects for special cases as needed, and
> generate code[3] for the wrapper tool (SIP) used to create the Python
> binary extension modules.
>
Here's one more link for the curious. This is the simple little program
used for the test:
http://trac.wxwidgets.org/browser/wxPython/Phoenix/trunk/samples/simple/events.py
Matthias
> Background info:
> http://wiki.wxpython.org/ProjectPhoenix
>
> http://groups.google.com/group/wxPython-dev/browse_thread/thread/451c5420afc8d402#
>
>
> I just love it when a plan comes together and it actually works even
> better than I thought it would at the outset.
Really great work, Robin! Everything you tell sounds more than promising.
> * Build scripts that are much cleaner and more sane that the hodge-podge
> that had grown up for the old wxPython over the years. It automatically
> (mostly) deals with the dependencies between the ETG scripts, the SIP
> source and the generated C++ source. It works well, but there is still
> a chance that I may dump distutils and use something a little more
> powerful.
One good tool for this might be http://www.scons.org . It's a python
replacement for make and has a dependency graph which you can customize
with custom dependency tracking, building and so on. distutils is simpler
though.
> * A unit test suite has been started[4]. Mostly I've just done tests
> for things that are customized or tweaked in ways that I'm not certain
> of at the outset, and I assume that those things that are wrapped
> without tweaks are working okay and don't need tests. However that
> should be changed later.
This is also nice. In the far future maybe there could be a unit test
suite for things like agw, too (but that's completely off topic here).
> [virtual method overloading bumps]
I think it's not a very big limitation if you enable virtuals only for a
subset of the methods. It will also help fighting bigger binary sizes I
guess.
One additional question: Do you have any indication yet what the
speed/size of Project Phoenix is compared to current wxPython?
-Matthias
Robin Dunn
> Am 27.12.2010, 21:27 Uhr, schrieb Robin Dunn <ro...@alldunn.com>:
>
>> * Build scripts that are much cleaner and more sane that the
>> hodge-podge that had grown up for the old wxPython over the years. It
>> automatically (mostly) deals with the dependencies between the ETG
>> scripts, the SIP source and the generated C++ source. It works well,
>> but there is still a chance that I may dump distutils and use
>> something a little more powerful.
>
> One good tool for this might be http://www.scons.org . It's a python
> replacement for make and has a dependency graph which you can customize
> with custom dependency tracking, building and so on. distutils is
> simpler though.
I'm looking at waf, which is supposed to be a lot faster than scons.
Unfortunately waf seems very non-intuitive to me, at least so far.
> One additional question: Do you have any indication yet what the
> speed/size of Project Phoenix is compared to current wxPython?
Not yet, although I think there will be at least a measurable
improvement in speed since for most things there will be a whole layer
removed in the call stack. IOW, instead of
User code --> Python wrapper code --> C++ wrapper code --> wx
it will be
User code --> C++ wrapper code --> wx