I love it if shedskin could support __del__ destructors...

[Paul Haeberli]

I've been trying a few things with shedskin.  So far I've

been amazed by what if can do!  Thanks so much for creating

this software!  

I'd like to use shedskin to make my python code run faster,

but I need support for __del__ for various data structures 

implemented in c, like canvases, OpenGL texture maps, frame 

buffer objects, etc.

I'm building a ui view archiecture in python to support writing

iOS/Android/OSX applications.  A work in progress video is here:

    http://www.youtube.com/watch?v=KeEh9xKfg8c

This project is 5000 lines of python calling 20,000 lines of C to 

do graphics, etc.

I've attached a .zip file with an example project that shows

what I'm trying to do.  Having support for being able to

define my own __del__ method would be a wonderful thing.

Please let me know if you have any comments or questions.

Paul Haeberli

paulha...@gmail.com

To compile and run this example, do this:

    shedskin -L lib imageproc.py

    make

    ./imageproc

to remove the files generated do:

    rm Makefile

    rm frame.tga

    rm red.tga

    rm saturate.tga

    rm shift.tga

    rm small.tga

    rm imageproc

    rm imageproc.cpp

    rm imageproc.hpp

The module libgfx support calls to alloc and free

canvases - arrays of 32 bit ints that hold image data.

    canvas *cannew(int sizex, int sizey)

    {

        canvas *c;

        c = (canvas *)malloc(sizeof(canvas));

        c->sizex = sizex;

        c->sizey = sizey;

        c->data = (unsigned int *)malloc(sizex*sizey*sizeof(unsigned int));

        nalloc++;

        return c;

    }

    void canfree(canvas *c)

    {

        if(!c)

            return;

        free(c->data);

        free(c);

        nalloc--;

    }

Image processing happens in C like this:

    void cansaturate(canvas *c, float sat)

    {

        int npix = c->sizex*c->sizey;

        unsigned int *lptr = c->data;

        while(npix--) {

            int r = RVAL(*lptr);

            int g = GVAL(*lptr);

            int b = BVAL(*lptr);

            int lum = ILUM(r,g,b);

            *lptr++ = CPACK(pixlerp(lum,r,sat),pixlerp(lum,g,sat),pixlerp(lum,b,sat),255);

        }

    }

Here's some of my python code that supports canvas in python.  The python 

code keeps an int with a pointer to the c canvas data structure in self.can

I'd really like __del__ to work so the c data sctructures can be freed, etc.

class canvas():

    def __init__(self):

        self.name = ""

    def init(self):

        self.rect = rectsize(self.sizex, self.sizey)

        self.name = ""

        self.readonly = False

        self.diagsize = float(math.sqrt(self.sizex*self.sizex+self.sizey*self.sizey))

        self.units = float(max(self.sizex, self.sizey))

        self.origin = point(0.0,0.0)

    def initsize(self, sizex, sizey):

        self.can = libgfx.gfx_cannew(sizex, sizey)

        self.sizex = sizex

        self.sizey = sizey

        self.nchans = 4

        self.init()

    def initpointer(self, ptr):

        self.can = ptr

        size = libgfx.gfx_cansize(self.can)

        self.sizex = size[0]

        self.sizey = size[1]

        self.nchans = size[2]

        self.init()

    def __str__(self):

        return "canvas: size: " + str(self.rect.sizex) + "," + str(self.rect.sizey) + " nchans: " + str(self.nchans)

    def __del__(self):

        print "canvas: DESTRUCT"

        print self

        if self.can != 0:

            libgfx.gfx_canfree(self.can)   # this is never called by shedskin

            self.can = 0

    def setreadonly(self, ro):

        self.readonly = ro

    def touched(self):

        if self.readonly:

            print "Error: Canvas is readonly"

            print self

    def tofile(self, filename):

        libgfx.gfx_cantofile(self.can, filename)

        return self

    # misc utils

    def putpix(self, x, y, pix):

        libgfx.gfx_canputpix(self.can, x, y, pix)

    def getpix(self, x, y):

        return libgfx.gfx_cangetpix(self.can, x, y)

    def saturate(self, sat):

        self.touched()

        libgfx.gfx_cansaturate(self.can, sat)

        return self

    def shift(self, shiftx, shifty):    # pix dim

        c = canvas()

        c.initpointer(libgfx.gfx_canshift(self.can, int(shiftx), int(shifty)))

        c.origin.x = self.origin.x - shiftx

        c.origin.y = self.origin.y - shifty

        return c

    def frame(self, r, g, b, a, width): # pix dim

        c = canvas()

        c.initpointer(libgfx.gfx_canframe(self.can, r, g, b, a, width))

        c.origin.x = self.origin.x + width

        c.origin.y = self.origin.y + width

        return c

    def subimage(self, r):              # pix dim

        c = canvas()

        c.initpointer(libgfx.gfx_cansubimg(self.can, int(r.orgx), int(r.orgy), int(r.sizex), int(r.sizey)))

        c.origin.x = self.origin.x - int(r.orgx)

        c.origin.y = self.origin.y - int(r.orgy)

        return c

    def zoom(self, zoomx, zoomy):

        c = canvas()

        c.initpointer(libgfx.gfx_canzoom(self.can, zoomx, zoomy))

        c.origin.x = c.origin.x * zoomx

        c.origin.y = c.origin.y * zoomy

        return c

    def clone(self):

        c = canvas()

        c.initpointer(libgfx.gfx_canclone(self.can))

        c.origin = self.origin

        return c

    def new(self):

        c = canvas()

        c.initpointer(libgfx.gfx_cannew(self.rect.sizex,self.rect.sizey))

        c.origin = self.origin

        return c

    def blend(self, c, r, g, b, a):

        self.touched()

        libgfx.gfx_canblend(self.can, c.can, r, g, b, a)

        return self

    def mult(self, c):

        self.touched()

        libgfx.gfx_canmult(self.can, c.can)

        return self

    def scalergba(self, r, g, b, a):

        self.touched()

        libgfx.gfx_canscalergba(self.can, r, g, b, a)

        return self

    def clear(self, r, g, b, a):

        self.touched()

        libgfx.gfx_canclear(self.can, r, g, b, a)

        return self

def canvasfromfile(filename):

    can = canvas()

    ptr = libgfx.gfx_canfromfile(filename)

    can.initpointer(ptr)

    return can

def canvasnew(sizex, sizey):

    can = canvas()

    can.initsize(sizex, sizey)

    return can

def canvasnalloc():

    return libgfx.gfx_cannalloc()

[Mark Dufour]

hi paul,

On Mon, Jan 21, 2013 at 9:56 PM, Paul Haeberli <paulha...@gmail.com> wrote:

I've been trying a few things with shedskin.  So far I've

been amazed by what if can do!  Thanks so much for creating

this software!  

thanks a lot for the feedback! it's always motivating to see an interesting new program work with shedskin. perhaps we could even add yours to shedskin/examples at some point, as an example of how to integrate with manual C code..?

I'd like to use shedskin to make my python code run faster,

but I need support for __del__ for various data structures 

implemented in c, like canvases, OpenGL texture maps, frame 

buffer objects, etc.

I just made some changes to GIT to make __del__ work in the most basic way. thanks for triggering! some caveats at this point:

-since there is no refcounting in shedskin, the boehm GC may decide at a much later point to call __dell__. or under low memory pressure, to never even call it at all (I think cpython doesn't give a guarantee that __del__ is called either). the order of calls may also be completely different, and I haven't even started thinking about reference cycles and inheritance.

-there are some thread unsafe/non-reentrant parts in shedskin, especially in printing and string formatting. now since the boehm GC apparently can call __del__ while in the process of running another call to __del__, printing, string formatting and probably joining strings will all lead to a segfault at this point. unfortunately fixing this quickly will make the respective code much slower when run in a single thread. if anyone is interesting in fixing this so it works well in both cases, please let me know (a starting point might be to grep for "__print_cache").

-to make __del__ work right now, there also has to be an explicit __init__ method. this can be fixed later.
 
-using __del__ may well make your program much slower, since each class now inherits from the boehm GC 'gc_cleanup' class instead of 'gc'. well I guess we could inherit from gc_cleanup only for classes that implement __del__, but I don't know how to do that cleanly right now. I haven't measured in any case, so no idea really about the potential slowdown this may cause in practice.

thanks again, let me know how this works for you,
mark.

--
You received this message because you are subscribed to the Google Groups "shedskin-discuss" group.
To view this discussion on the web visit https://groups.google.com/d/msg/shedskin-discuss/-/UcZNzfe3_V8J.
To post to this group, send email to shedskin...@googlegroups.com.
To unsubscribe from this group, send email to shedskin-discu...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/shedskin-discuss?hl=en.

--
http://www.youtube.com/watch?v=E6LsfnBmdnk

[Paul Haeberli]

Hi Mark -

Thanks for implementing __del__.  I have not been able to make it work

for my example, but maybe I'm doing something wrong.  There are many

points I'd like to reply to in detail once I get your new stuff working.

Here's what I did:

I Downloaded 

    shedskin-mainline-master.tar.gz

from

    http://gitorious.org/shedskin/mainline/trees/master

Unzipped to shedskin-mainline

    cd shedskin-mainline

    python setup.py install

    shedskin test.py

    make

    ./test

    

    this works

When I try to compile the example, I still get the warning about __del__ not being

supported.

    shedskin -L lib imageproc.py

.

.

.

*WARNING* imageproc.py:401: function (class canvas, 'initsize') not called!

*WARNING* imageproc.py:419: '__del__' is not supported

*WARNING* imageproc.py:426: function (class canvas, 'setreadonly') not called!

.

.

.

When I run my example it runs the same as before.

    ./imageproc

.

.

.

N CANVASES STILL ALLOCATED: 25

I'd really like the canveses to be freed by __del__!

Am I doing something wrong?  AM I running your new version?

-Paul

[Mark Dufour]

hi paul,

I kept the warning because of the caveats, but it looks like you are running the correct version, because the warning about '__del__' not being called seems gone now.  do you see the __del__ method in the generated .?pp code?

if you only allocate 25 objects, there's no need for the GC to call __del__ (no memory pressure). what happens when you allocate many more..?

thanks!
mark.
 

Visit this group at http://groups.google.com/group/shedskin-discuss?hl=en.
For more options, visit https://groups.google.com/groups/opt_out.
 
 

--
http://www.youtube.com/watch?v=E6LsfnBmdnk

[Paul Haeberli]

I tried running mark's new build with my program.

Here's the output now:

g++  -O2 -Wno-deprecated  -I. -I/Library/Python/2.7/site-packages/shedskin/lib -Ilib -I/usr/local/include -D__SS_GC_CLEANUP lib/libgfx.cpp imageproc.cpp /Library/Python/2.7/site-packages/shedskin/lib/sys.cpp /Library/Python/2.7/site-packages/shedskin/lib/re.cpp /Library/Python/2.7/site-packages/shedskin/lib/math.cpp /Library/Python/2.7/site-packages/shedskin/lib/builtin.cpp -lgc -lpcre  -L/usr/local/lib -o imageproc

LERP: 0.5

FMOD: 2.0

LIMIT: 0.0

DELTA: 0.1

point: 0.0 60.0

point: 0.0 0.0

point: 10.0 0.0

point: 0.0 20.0

DIFFER

rect: org: 50,50 size: 50,50

N CANVASES ALLOCATED: 0

smallpic: canvas: size: 776,386 nchans: 4

GET PIX: 45

Error: grr... canfree called from multiple threads

Marks's comments comment roved true:

"boehm GC may decide at a much later point to call __del__"

"since the boehm GC apparently can call __del__ while in the 

process of running another call to __del__, printing"

Yes - canfree gets called twice by different threads I guess!

I had modified the code to include this in libgfx.c

int infree = 1;

void canfree(canvas *c)

{

    infree++;

    if(infree == 2) {

        fprintf(stderr, "ERROR: grr... canfree called from multiple threads\n");

        exit(1);

    }

    if(!c) {

        infree--;

        return;

    }

    free(c->data);

    free(c);

    nalloc--;

    infree--;

}

And added these lines to the file imageproc.py

for i in range(100):

    testcanvases()

    print "N CANVASES STILL ALLOCATED: " + str(canvasnalloc())

An updated version of my example code is attached.

[Paul Haeberli]

Is there some way to tell the boehm GC to collect all

garbage?

I guess I could put a lock in to make canfree only called

be one thread right?

-P

[Mark Dufour]

hi paul,

Yes - canfree gets called twice by different threads I guess!

I had modified the code to include this in libgfx.c

the problem with printing from inside __del__ was not caused  by multiple threads but a non-reentrant function being called recursively (__del__ calls print2, print2 calls GC_MALLOC, GC_MALLOC calls gc finalization which calls ~canvas which calls __del__ which calls print2 again.. :P). but the problem is related, in that a function cannot simply mess around with global variables.

in any case I don't think there are multiple threads running at the same time. running 'info threads' in gdb gives only one thread. and iirc boehm should run in a single thread by default and that's why it calls finalization from GC_MALLOC. I'm far from an expert in threading though.
 

int infree = 1;

void canfree(canvas *c)

{

    infree++;

    if(infree == 2) {

this doesn't seem correct, infree should probably be initialized at 0..?

after removing the print statements initializing infree at 0 and adding print statements to gfx_canputpix and gfx_canfree, as follows:

    printf("canput %d\n", ic);
    printf("canfree! %d\n", ic);

it looks like canput always crashes for ic 95, which is interesting, because of the following lines:

N CANVASES STILL ALLOCATED: 95
..
canfree! 95
..
canput 95

I'm afraid I don't have more time to look into this today, perhaps over the weekend. but perhaps this gives you a small clue as to what the problem could be..?

thanks!

mark.
--
http://www.youtube.com/watch?v=E6LsfnBmdnk

[Mark Dufour]

hi paul,

On Thu, Jan 24, 2013 at 6:14 PM, Paul Haeberli <paulha...@gmail.com> wrote:

Is there some way to tell the boehm GC to collect all

garbage?

yes, but it remains tricky business:

http://www.hpl.hp.com/personal/Hans_Boehm/gc/gcinterface.html

for example I have no idea whether gc_gcollect triggers finalization.

I've been reading a bit about finalization, and so far get the idea that it might be best to run finalization in a separate thread, to avoid all sorts of "funny" situations. but again unfortunately not much time at the moment to look into this further.
 
thanks!
mark.

--

You received this message because you are subscribed to the Google Groups "shedskin-discuss" group.
To post to this group, send email to shedskin...@googlegroups.com.
To unsubscribe from this group, send email to shedskin-discu...@googlegroups.com.
Visit this group at http://groups.google.com/group/shedskin-discuss?hl=en.
For more options, visit https://groups.google.com/groups/opt_out.
 
 

--
http://www.youtube.com/watch?v=E6LsfnBmdnk

[Paul Haeberli]

Hi Mark -

Thanks for finding and fixing my counting bug in canfree.

I added support for explicitly calling CG_gcollect(), but this 

does not actually cause garbage collection to happen.

void *gfx_gccollect(void) {

    GC_gcollect();

    return NULL;

}

The Boehm GC acts in complicated ways.  When I run this kind

of python code with CPython, things work very nicely - it uses 

reference counting, and in my experience, whenever the reference

count on a python object goes to zero, and the object does not have

any circular references, it is immediately freed and __del__ is called.

<quote>

    from http://docs.python.org/2/reference/datamodel.html

    CPython implementation detail: CPython currently uses a

    reference-counting scheme with (optional) delayed detection

    of cyclically linked garbage, which collects most objects

    as soon as they become unreachable, but is not guaranteed

    to collect garbage containing circular references. See the

    documentation of the gc module for information on controlling

    the collection of cyclic garbage. Other implementations act

    differently and CPython may change. Do not depend on immediate

    finalization of objects when they become unreachable (ex:

    always close files).

</quote>

To support freeing memory or hardware resources in libraries 

called from python, reference counting instead of garbage collection

might be a better solution.

I hate to suggest more work, but I wonder if you could comment on

how hard it might be to make shedskin use reference counting instead

of garbage collection?

Thanks!

Paul H

[Mark Dufour]

hi paul,

I hate to suggest more work, but I wonder if you could comment on

how hard it might be to make shedskin use reference counting instead

of garbage collection?

that would be a lot of work.. and I think in most cases __del__ should work fine with shedskin, especially when it just releases non-GC'd resources.

if you don't beat me to it, I will have another look at why imageproc is currently failing. unfortunately I'm a bit busy though this weekend..

thanks!
mark.

To unsubscribe from this group and stop receiving emails from it, send an email to shedskin-discu...@googlegroups.com.

To post to this group, send email to shedskin...@googlegroups.com.

Visit this group at http://groups.google.com/group/shedskin-discuss?hl=en.
For more options, visit https://groups.google.com/groups/opt_out.
 
 

--
http://www.youtube.com/watch?v=E6LsfnBmdnk

[Mark Dufour]

hi paul,

this line seems to cause the problem, because gfx_canfree is later called via __del__, leading gfx_canfree to be called a second time.

libgfx.gfx_canfree(sh.can)

commenting out this line or adding 'sh.can=0' right below it seems to avoid the observed crash. or replacing this line with:

sh.__del__()

(that and you still have to comment out the print statements in __del__).

note that 'del sh' doesn't work, because this decreases the refcount, and __del__ is only called when the refcount reaches 0.

does that solve the problem for you as well?

thanks!
mark.

--
http://www.youtube.com/watch?v=E6LsfnBmdnk

[Paul Haeberli]

Mark -

I fixed the bug you found and fixed a few other bugs and it  

looks like it's working!  calling

gccollect()

seems to run the garbage collector and it actually runs and

__del__ gets called in a reasonable way.

Now I'm going to try to get shed skin running under iOS -

compiled in 32 bit mode and see if I can get my little UI

system running with shedskin.

Thanks for you help with this.  Please feel free to add my

code to your example set.  Also let me know if you have

suggestions of how to improve this example in any way before

you release it.

Please see the attached tar file - Now with handy Makefile

-Paul Haeberli