Prototype for a vispy component architecture was Questioning the current vispy design paradigm

78 views
Skip to first unread message

Dr. Volker Jaenisch

unread,
Jun 23, 2015, 6:47:51 PM6/23/15
to visp...@googlegroups.com
Hi Luke and Coworkers!

As promised yesterday today I am coming up with the first prototype of a flexible architecture for the graph components of vispy (and maybe pyqtgraph).

The code you can fetched from our mercurial repository

hg clone https://hg.inqbus.de/volker/graph_components

The directory structure is twofold. Under /src you find /graph with the examples from the ongoing discussion. Under /graph_ZCA you find the
new componet oriented implementation.                                                                                                                                                                                                                   

.                                                                                                                                                                                                                    
└── src                                                                                                                                                                                                              
    ├── graph                                                                                                                                                                                                        
    │   ├── app.py                                                                                                                                                                                                   
    │   ├── components.py                                                                                                                                                                                            
    │   ├── __init__.py                                                                                                                                                                                              
    │   └── interfaces.py                                                                                                                                                                                            
    └── graph_ZCA                                                                                                                                                                                                    
        ├── app.py                                                                                                                                                                                                   
        ├── base.py                                                                                                                                                                                                  
        ├── components.py                                                                                                                                                                                            
        ├── __init__.py                                                                                                                                                                                              
        └── interfaces.py 
The main file to run is in both case app.py. There are only two external dependencies:
zope.interface
zope.component

If not included already in your python. Please install them using
pip install zope.interface
pip install zope.component
or the way you like to install python packages on your machine.

Sorry for not coming up with fully elaborated packages.

Let have a look at the app.py:

==============================================================
# Here starts the boiler plate
from zope.interface import implements   # <- this is needed here to form the PlotImage component
from zope.component import getGlobalSiteManager, adapts   # <- Only needed for the extension afterwards
gsm = getGlobalSiteManager()                              # <- Only needed for the extension afterwards
# End of Boiler plate

# Import of the sub components.
from graph_ZCA.components import BaseComponent, Image, Axis, ViewBox

# Only needed for the extension afterwards.
from graph_ZCA.interfaces import IAxis, IViewBox, IBaseComponent
from graph_ZCA.components import TickLabelComponent, AxisComponent


# Set up a new interface for the component PlotImage. Here usually properties, functions and other exposed items
# of a PlotImage instance would be documented and registered to the framework. Since the only function render is
# already documented for IBaseComponent nothing has to be done here.
class IPlotImage(IBaseComponent):
    """
    Component to plot a 2d graph of a given image
    """

# Here the component itself is constructed. The component inherits from a simple BaseComponent without much importance.
# The only crucial requirement is that every component knows its parent component as self.parent, which is the first
# parameter to the constructor. The top most component has a parent of NONE.
class PlotImage( BaseComponent):
    """
    Makes a 2D plot for an image
    """
    implements(IPlotImage)   # here the interface specified above is used.

    def __init__(self, parent):
        super(PlotImage, self).__init__(parent)   # tribute to the super class

        self.vb = ViewBox(self)()   # make a ViewBox. Please notice the double Braces. These are neccessary for the framework.
                                    # In fact here in no instance generated but a factory. And with the second brace
                                    # the factory produces the instance.
                                    # The PlotImage "self" is assigned as parent for the ViewBox

        x_axis = Axis(self.vb)()    # Same pattern here
                                    # The axis are assigned to the viewbox
        x_axis.orientation='bottom'
        x_axis.start=0
        x_axis.end=8

        y_axis = Axis(self.vb)()    # and here
        y_axis.orientation='left'
        y_axis.start=-5
        y_axis.end=3


        self.vb.addItem( x_axis )   # Finally the axis are given to the viewbox. This step may be not necessary and
                                    # may be done automatically while assigning the parent to the axis.
        self.vb.addItem( y_axis )

    def render(self):               # In this prototype code every component optains a render method that is used to generate
                                    # some simple output
        return self.vb.render()


# # The following statements can be uncommented to illustrate how to swap classes.
# #
# # This example shows how to swap all Axis components which parents are a ViewBoxes to be of class NewAxisComponent
# class NewAxisComponent(AxisComponent):
#      adapts(IViewBox)
#
# gsm.registerAdapter(NewAxisComponent)
#
# # This example shows how to exchange the TickLabelComponent for all Axis
# class NewTickLabelComponent(TickLabelComponent):
#     adapts(IAxis)
#
#     def render(self):
#         return " ".join([ "q%i" % i for i in range(self.axis.start, self.axis.end)])
#
# gsm.registerAdapter(NewTickLabelComponent)


image_data = [1,2,3,4]

plot =  PlotImage( None)
image = Image(plot.vb)(image_data)
plot.vb.addItem(image)

print( plot.render())
====================================================

At first impression the code looks like before. But a closer look reveals some changes in detail:
1) getting a Instance of a conponent "A" is yet no longer self.a = A() but self.a = A(self)(). This is due to using Factories in the framework backend. The first A(self) invokes the factory. The factory then looks up the best class to be used in this context. The second () invokes the constructor of the class returned by the factory. If you look at the end of the file you wil see:

image = Image(plot.vb)(image_data)

Here an Image is created. The first call Image(plot.vb) creates a factory that looks for an apropriate Image class for an Image that is attached to a instance of plot.vb. This factory can be changend and extended easily to deliver a class of Ximage or of Qimage or whatever you may think of dependend on the knowlegde you build into this factory.
The second call  (image_data) invokes the constructor of the delivered Image class to fill it with data.

If you run the code as is the output will be:

AxisComponent: bottom:0 1 2 3 4 5 6 7
AxisComponent: left:-5 -4 -3 -2 -1 0 1 2
ImageComponent:[1, 2, 3, 4]

Where the first in Row is the class of the Item followed by its representation.

If you comment in

class NewAxisComponent(AxisComponent):
     adapts(IViewBox)
gsm.registerAdapter(NewAxisComponent)

You introduce a new Axis class NewAxisComponent that inherits from AxisComponent (Standard). Also you stated that
this new class should be used for every Axis class which parent class is a ViewBox.

NewAxisComponent: bottom:0 1 2 3 4 5 6 7
NewAxisComponent: left:-5 -4 -3 -2 -1 0 1 2
ImageComponent:[1, 2, 3, 4]

And you can be more specific:

class NewAxisComponent(AxisComponent):
     adapts(IPlotImage, IViewBox)
gsm.registerAdapter(NewAxisComponent)

This will affect only the Axis classes that are in a ViewBox that lives in a PlotImage component.

NewAxisComponent: bottom:0 1 2 3 4 5 6 7
NewAxisComponent: left:-5 -4 -3 -2 -1 0 1 2
ImageComponent:[1, 2, 3, 4]

Ok the outcome does not change in that case.

Now lets dig a bit deeper. The TickLabels are generated some levels below our discussed code. But they are implemented like the axis. (please have a look at components.py)

class NewTickLabelComponent(TickLabelComponent):
    adapts(IAxis)

    def render(self):
         return " ".join([ "q%i" % i for i in range(self.axis.start, self.axis.end)])

gsm.registerAdapter(NewTickLabelComponent)

This will introduce a NewTickLabelComponent. It will prefix every tick of any axis with a "q".
As shown above you can also be specific.

class NewTickLabelComponent(TickLabelComponent):
    adapts(IPlotImage, Interface, IAxis)

will confine this modification to any PlotImage instance, with an arbitrary Component below but an Axis component following in the Parent vector.

This is only a prototype and many questions are still open.

What I liked to demostrate is, that with the use of ZCA and some clever base classes a powerful API can be crafted.
This API delivers the user
* an OOP-Like environment with the same look and feel as normal OOP.
* a non changing API if a developer likes to swap a component class.

To the developer this framework gives a unified way to make subcomponents swappable, later on.

How does it work?

It's no magic.
The whole logic lies in roughly 40 lines of code in base.py in the class BaseFactory.

The logic is based on two assumptions:
1) Each componet has a parent component, or it is a top level Componetn that has a parent reference of None.
2) Each component has a Interface specified that is derived of a sole BaseInterface.

If the code looks for a components instance not the componet class itself is invoked but a factory. This factory is specific for the component: For an Axis Component there is a Axis Factory.

All Factories are based on BaseFactory.

BaseFactory gets the Parent of the component that is meant to create. Also BaseFactory knows the Interface of the instance that has to be created.

class AxisComponent(BaseComponent):
    """
    Represents an axis
    """
    implements(IAxis)
    adapts(Interface)
    def __init__(self, parent, orientation=None, start=None, end=None):
        super(AxisComponent, self).__init__(parent)
        self.orientation = orientation
        self.start = start
        self.end = end
    def get_tick_labels(self):
        return TickLabel(self)().render()
    def render(self):
        return '%s: ' % self.__class__.__name__ + self.orientation + ':' + ' '.join(self.get_tick_labels())
gsm.registerAdapter(AxisComponent)

    
class Axis(BaseFactory):
    target_interface = IAxis

As you can see Axis is the Factory and AxisComponent is the Instance that is created by the Axis factory. The Axis-Factory knows the Interface of the Instance to be created by means of the Class-Variable "target_interface". This boiler plate is tribute to the ZCA.
The lookup of the apropiate class for a factory.
If a factory is invoked it firstly creates a parent_vector  which is a vector containing a reference to all parent instances of the current parent object to new class. What?

An example makes it more clear. Let us think of the creation of a TickLabel-Instance by calling

TickLabel(self)().render()

in a Axis Component. [componets.py line 49].

The hierarchy of the components is:

PlotImage -> ViewBox -> Axis -> TickLabel

So the Factory for TickLabe first looks for a multiadpater that is registered for the tuple of
(PlotImage, ViewBox,Axis)
If no one is found it looks for an adapter for
(ViewBox,Axis)
and finally for
(Axis)
until it finds a match.

If the PlotIten instance will become a part of a larger scene the vector is expanded automatically to take the scene level into account.

scene = Scene()

plot = PlotImage(scene)()

class NewAxisComponent(AxisComponent):
     adapts(IScene, IPlotImage, IViewBox)

gsm.registerAdapter(NewAxisComponent)

In this case only the Axis of this Scene class will get the NewAxisComponent class get swapped in. All other Instances of PlotImages will not be affected.

There may be more applications of the ZCA for the vispy project. But I focussed on swapping classes on runtime without sideeffects and code changes.

Cheers

Volker
 
-- 
=========================================================
   inqbus Scientific Computing    Dr.  Volker Jaenisch
   Richard-Strauss-Straße 1       +49(08861) 690 474 0
   86956 Schongau-West            http://www.inqbus.de
=========================================================	    
	    

Dr. Volker Jaenisch

unread,
Jun 23, 2015, 7:04:42 PM6/23/15
to visp...@googlegroups.com
Hi Luke and Coworkers!

Sorry, I just noticed that our repository can not be convinced to be on
open access.

Since the code is not that heavy 34KB I will attach it at this post.
graph_components.zip

Luke Campagnola

unread,
Aug 9, 2015, 7:30:37 PM8/9/15
to visp...@googlegroups.com
Thanks for putting this together! It's quite a lot to digest and there are some very interesting ideas in here, and I will continue to consider these issues as we develop vispy. 

However, I will need to re-read this a few more times to fully understand the design of the architecture, and therein lies its weakness--it is so unfamiliar that it would increase the already steep learning curve for new developers (not to mention the existing developers). I am left wondering whether there is a compromise solution that provides the primary benefit you need--being able to easily customize the construction of the complex widgets--without introducing such unfamiliar concepts. (If the answer turns out to be "no", then perhaps it really does make sense to move in this direction.)

For example, the use of factories like `Component(obj)(args)` would initially draw many questions, but might be rewritten in a way that is more readable, such as `obj.create(Component, args)`. Likewise the ability to arbitrarily specify how new components will be created based on the parent component hierarchy is very powerful, but comes with the cost of tripling the number of classes and potentially introducing very difficult situations that invite user error. Perhaps if we narrow this down to a less general, more targeted approach, it can be implemented in such a way that it does not require such deep architectural changes. 

..or perhaps what you demonstrate really is the ideal approach, but it would be hard for me to evaluate until I have had a lot more experience with this architecture and became familiar with its uses and limitations.



--
You received this message because you are subscribed to the Google Groups "vispy-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to vispy-dev+...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/vispy-dev/5589E607.4050000%40inqbus.de.
For more options, visit https://groups.google.com/d/optout.

Dr. Volker Jaenisch

unread,
Aug 11, 2015, 10:29:29 AM8/11/15
to visp...@googlegroups.com
Hi Luke!

Had given up any hope, that someone will come up with this topic. So I am very pleased to meet you again!


Am 10.08.2015 um 01:30 schrieb Luke Campagnola:
Thanks for putting this together! It's quite a lot to digest and there are some very interesting ideas in here, and I will continue to consider these issues as we develop vispy.
Fine!

 
However, I will need to re-read this a few more times to fully understand the design of the architecture, and therein lies its weakness--it is so unfamiliar that it would increase the already steep learning curve for new developers (not to mention the existing developers). I am left wondering whether there is a compromise solution that provides the primary benefit you need--being able to easily customize the construction of the complex widgets--without introducing such unfamiliar concepts. (If the answer turns out to be "no", then perhaps it really does make sense to move in this direction.)
You are right. The learning curve of component architecture is a lot steeper than of pure OOP. One may compare it to the difference steapness between procedural programming and OOP. But this is the nature of paradigms - a new paradigm implies that some old things have to be buried and new (usually more complex) things are born.


For example, the use of factories like `Component(obj)(args)` would initially draw many questions, but might be rewritten in a way that is more readable, such as `obj.create(Component, args)`. Likewise the ability to arbitrarily specify how new components will be created based on the parent component hierarchy is very powerful, but comes with the cost of tripling the number of classes
Yes, in some cases you may triple the number of classes. But in more concrete cases e.g. a factory can be build by writing a simple python function.
But let's be realistic. In e.g. Java to use a simple dict or list you will have to write a class. But is this a point against JAVA in general?
In componet architecture classes are no longer glued 1:1 to entities like in classic OOP. Classes become now building blocks to build higher level patterns that represent the entities quite more flexible.
In component architecture entities are represented in terms of interfaces. And in difference to OOP a component which is represented by a class instance may have several interfaces at once.
And in addition an adapter can provide a new interface to a set of interface it matches.
In terms of chemistry spoken: In OOP we build atoms to interact with each other. In component architecture we build molecules and catalysts to interact with each other.
and potentially introducing very difficult situations that invite user error. Perhaps if we narrow this down to a less general, more targeted approach, it can be implemented in such a way that it does not require such deep architectural changes.
This approach will most likely fail. When the ZCA was introduced in the Zope community many developers complained about the steep learning curve and called for some shortcuts for beginners to build a bridge between the "old way" and the ZCA. There were some very cool and usefull approaches like grok http://grok.zope.org/ to hide the complexity of the ZCA from the end users. But in the long run Grok died. The problem with grok was that to deal with the complexity of what you can build with the ZCA you need an as well as complex framework. So grok becomes a superframework to the ZCA using Python metaclasses.
Python Metaclasses are a very powerful tool, but only for some very limited cases. In the end Grok becomes so automagic that debugging grok code was nearly impossible. From this experiance I strongly recommend not to repeat this desaster.

If you deal with lots of code and lots of classes further or later you will need more than OOP. The new paradigm it may be ZCA or something else will introduce new ideas and patterns. If you try to hide the new paradigm from the developers you will cause confusion.The bitter truth is: To do complex things you need complex tools with a steep learning curve.

But in the end when your complex component code is up and running, you can build an API for the end user with e.g. the portal pattern to hide the complexity of the code to the end user.

To repeat myself: One has to distingiush between developers (going the steep way) and end users (Using the API but not fiddling with the core). The end user will not understand the concepts of ZCA but he can go after examples to mimic existing code and customize it to their needs. Lets have a look at the Plone community. There we have two very distinct populations of coders. There is a small group of 40-60 core programmers wade knee deep in ZCA. But there are several thausands advanced web programmers  using and extending the plone framework and also using the ZCA but on a limited scope and mainly by customizing (copy/paste) existing code. I think this kind of community is not to far of the vispy community?

 

..or perhaps what you demonstrate really is the ideal approach, but it would be hard for me to evaluate until I have had a lot more experience with this architecture and became familiar with its uses and limitations.
It took me some time to get familiar with ZCA. To be honest, it took me several months. Components are not OOP+ they are a new paradign. But afterwards I taught the ZCA to my apprentice within a fortnight. And at the end of a month she programmed her first adapters.

I will do my best to help you on your way to understand ZCA. Please feel free to ask me any question.
Reply all
Reply to author
Forward
0 new messages