Images sometimes fail to load

[David Woods]

I have a complex application for the academic analysis of video, audio,
and now still image data that I've been working on for several years
now. (http://www.transana.org) I'm stuck on a bug that defies
simplification, and am seeking suggestions on how to proceed.

As part of my application, one can assemble reports which can get quite
long. At present, for example, my testing database produces a report
that would be about 550 pages long if you were foolish enough to print
it. It includes information on the coding of over 1,150 video clips and
dozens of coded still images. At the moment, my report generator will
reliably fail on the display of one image in this large report. Only on
Windows. It works perfectly on OS X.

For testing purposes, each coded image is included in the report at
least twice. Only one copy of the image will fail. The other copy (or
as many as 7 copies) of the same image will be included successfully.
And it's not always the same image that fails.

The line of code that fails is:

# Load the image
self.bgImage = wx.Image(self.obj.image_filename)

At the moment, I get a console message that says:

Application transferred too few scanlines

(I have no idea where this message comes from) and self.bgImage.IsOk()
returns false.

The error is NOT related to the image. If I alter the parameters of the
report, a different image will fail. It's just that occasionally the
wx.Image call won't work, even though it works most of the time.

More details:

I'm using wxPython 2.9.4.0-unicode on Python 2.6.6.0 on Windows 7. The
same code using the same wxPython version on OS X works flawlessly.

The report is built in a wx.RichTextCtrl, and all images are loaded by
passing through a FloatCanvas using a ScaledBitmap2 object, although the
wx.Image line that fails is the one that loads the image to make the
call to the FloatCanvas.ScaledBitmap2 object, so the failure occurs
before the FloatCanvas is invoked.

Finally, in exploring this issue, I've found that explicitly deleting my
FloatCanvas Objects after I've copied the image from them to put into
the report and explicitly invoking garbage collection (using
gc.collect()) seem to make the problem occur less frequently.

At this point, I'm at a loss about what to do next in trying to sort out
this problem. Any suggestions?

David

[Cody]

Hi,

At this point, I'm at a loss about what to do next in trying to sort out this problem.  Any suggestions?

Just a guess but, it sounds like you may be running out of system resources. Especially since you say forcing cleanup of some UI objects causes it to occur less frequently.

On Windows I would monitor the GDI handle count for the application while you are running the report and see where the number climbs to. In Windows XP a single process is limited to 10,000 GDI handles and will begin to show problems like this when that point is reached. 

Every Image, Window, Brush, Color object, ect.. uses one or more GDI handles.

Cody

[Chris Barker - NOAA Federal]

David,

My first that thought was that might be a limited resource issue -- either too many file handles, or too many GDI objects or... though I know far too little of Windows internals to have any guess as to what.

# Load the image
self.bgImage = wx.Image(self.obj.image_filename)

At the moment, I get a console message that says:

Application transferred too few scanlines

(I have no idea where this message comes from) and self.bgImage.IsOk() returns false.

I hope that's familiar to someone -- maybe libjpeg?

I'm using wxPython 2.9.4.0-unicode on Python 2.6.6.0 on Windows 7. The same code using the same wxPython version on OS X works flawlessly.

so could be a limited resource on Windows issue.. 

The report is built in a wx.RichTextCtrl, and all images are loaded by passing through a FloatCanvas using a ScaledBitmap2 object, although the wx.Image line that fails is the one that loads the image to make the call to the FloatCanvas.ScaledBitmap2 object, so the failure occurs before the FloatCanvas is invoked.

Finally, in exploring this issue, I've found that explicitly deleting my FloatCanvas Objects after I've copied the image from them to put into the report and explicitly invoking garbage collection (using gc.collect()) seem to make the problem occur less frequently.

ah -- this fits too -- the FloatCanvas ScaledBitmap objects keep a copy of the wx.Image around -- so you probably don't want to keep those around -- and I woulnd't be surprised if gc.collect() were required -- this is a complex enough system that circular references are possible (likely?)

but you say "less frequently", which implies there are still issues.

 

At this point, I'm at a loss about what to do next in trying to sort out this problem.  Any suggestions?

have you tracked memory use? Maybe you're bumping into limits there, and libpeg doesn't give a good message if it can't allocate what it needs.

Can you be more vigorous about deleting the FloatCanvas objects and wx.Images after you are done with them?

I imagine the more Windows-savy folks on this list will be able to suggest a way to track resourced used by your app at run time -- maybe there will be some hints there.

not much help, I know.

-Chris

--

Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR&R            (206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115       (206) 526-6317   main reception

Chris....@noaa.gov

[Robin Dunn]

David Woods wrote:
>
> The line of code that fails is:
>
> # Load the image
> self.bgImage = wx.Image(self.obj.image_filename)
>
> At the moment, I get a console message that says:
>
> Application transferred too few scanlines
>
> (I have no idea where this message comes from) and self.bgImage.IsOk()
> returns false.

It looks like it is coming from libjpeg. See
wxWidgets/src/jpeg/jerror.h line #116. It's used in the functions
jpeg_finish_compress and jpeg_finish_decompress, with code like this:

if (cinfo->next_scanline < cinfo->image_height)
ERREXIT(cinfo, JERR_TOO_LITTLE_DATA);

My guess is that it's probably a memory or resource issue, and that the
actual problem is happening some time before the lines of code above,
but that it is manifesting as not enough image data being sent to those
functions so that is where it is caught.

--
Robin Dunn
Software Craftsman
http://wxPython.org

[David Woods]

My first that thought was that might be a limited resource issue -- either too many file handles, or too many GDI objects or... though I know far too little of Windows internals to have any guess as to what.

# Load the image
self.bgImage = wx.Image(self.obj.image_filename)

At the moment, I get a console message that says:

Application transferred too few scanlines

(I have no idea where this message comes from) and self.bgImage.IsOk() returns false.

I hope that's familiar to someone -- maybe libjpeg?

I'm using wxPython 2.9.4.0-unicode on Python 2.6.6.0 on Windows 7. The same code using the same wxPython version on OS X works flawlessly.

so could be a limited resource on Windows issue.. 

The report is built in a wx.RichTextCtrl, and all images are loaded by passing through a FloatCanvas using a ScaledBitmap2 object, although the wx.Image line that fails is the one that loads the image to make the call to the FloatCanvas.ScaledBitmap2 object, so the failure occurs before the FloatCanvas is invoked.

Finally, in exploring this issue, I've found that explicitly deleting my FloatCanvas Objects after I've copied the image from them to put into the report and explicitly invoking garbage collection (using gc.collect()) seem to make the problem occur less frequently.

ah -- this fits too -- the FloatCanvas ScaledBitmap objects keep a copy of the wx.Image around -- so you probably don't want to keep those around -- and I woulnd't be surprised if gc.collect() were required -- this is a complex enough system that circular references are possible (likely?)

but you say "less frequently", which implies there are still issues.

 

At this point, I'm at a loss about what to do next in trying to sort out this problem.  Any suggestions?

have you tracked memory use? Maybe you're bumping into limits there, and libpeg doesn't give a good message if it can't allocate what it needs.

Can you be more vigorous about deleting the FloatCanvas objects and wx.Images after you are done with them?

I imagine the more Windows-savy folks on this list will be able to suggest a way to track resourced used by your app at run time -- maybe there will be some hints there.

I track GDI Resources using the following:

# import the win32 methods and constants we need to determine the number of GDI Resources in use.
# This is based on code found in wxWidgets ticket #4451
from win32api import GetCurrentProcess
from win32con import GR_GDIOBJECTS
from win32process import GetGuiResources

def GDIReport():
    """ Report the number of GDI Resources in use by Transana """
    # Get the number of GDI Resources in use from the Operating System
    numGDI = GetGuiResources(GetCurrentProcess(), GR_GDIOBJECTS)
    # Return the number of GDI Resources in use
    return numGDI

What I see is that GDI resources increase by about 40 per image inserted, but that the error arises somewhere between 1667 and 2007 GDI resources in use (different on different runs), which is FAR less than the 10,000 figure generally accepted as the limit.  I had some GDI issues with the wx.RichTextCtrl in a previous wxPython release, but these issues *appear* to be resolved in wxPython 2.9.4.0.

As far as tracking memory, I use:

wx.GetFreeMemory()

which reports a cool 4095 GB of free memory, a number which does not change during report generation.

Using an object counter I found at http://stackoverflow.com/questions/10610570/dramatic-memory-leak-in-wxpython-application , I'm not seeing any unreasonable increase in program objects.

I'm WAY out of my depth here.  If anyone has further suggestions of what to track and how to track it, I'm all ears.

David

[Chris Barker - NOAA Federal]

David,

How about a total kludge:

for tries in range(10):

    img = wx.Image(....)

    if IsOk():

        break

    time.sleep(0.001)

else:

    raise IOError("couldn't load an image")

maybe this is one of those stochastic errors....

-Chris

--
You received this message because you are subscribed to the Google Groups "wxPython-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to wxpython-user...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.
 
 

[David Woods]

Chris,

I tried something quite like that, although without the time.sleep() call.  It didn't work.  But now that you mention it, adding something along the lines of time.sleep() or wx.Yield() might help.  The next image in the report always loads, so I just need something there to let the system clear the cobwebs from its brain.

Thanks for the suggestion.

David

[Chris Barker - NOAA Federal]

On Fri, May 3, 2013 at 2:41 PM, David Woods <tran...@gmail.com> wrote:

I tried something quite like that, although without the time.sleep() call.  It didn't work.  But now that you mention it, adding something along the lines of time.sleep() or wx.Yield() might help.  The next image in the report always loads, so I just need something there to let the system clear the cobwebs from its brain.

maybe a gc.collect() even...

I'd probably log it too, nice to know how often it's getting hit.

 

-Chris

[David Woods]

Hi everyone,

Thanks for all of your help. I've finally resolved the issue.

It turns out that my wx.Image call was the problem. I wasn't Destroying
it properly. Since it was defined in the context of a wx.Frame, I
assumed it would be destroyed when the frame was destroyed, but
apparently that isn't the way it works. It wasn't a child of the frame,
and the image object was persisting after the frame was destroyed. As I
built my report, 25 or 30 images would build up in memory until there
was no room for additional images. It's not clear to me why one image
failing caused later images to be able to work, but that's what was
happening. I guess something in wx.Python/wxWidgets was freeing up the
image memory at that point that my attempts at explicit garbage
collection weren't freeing up.

The solution was to add an explicit Image.Destroy() call in the
EVT_CLOSE handler for the Frame object. No more crashes, even with
significantly more images.

The increasing number of GDI resources being used was a bit of a red
herring. I knew they weren't getting high enough that they should be
causing problems. I'm not sure why explicit Frame.Destroy() calls for
the form and explicit garbage collection isn't freeing those resources,
but I've found another solution for that. Since the form I use to build
my complex image is never displayed during report generation, I can skip
creating its toolbars, which were what was using all the GDI resources.

Is there a common technique for tracking how much memory a program has
to work with in a case like this? My wx.GetFreeMemory() call clearly
was not measuring what I thought, as its value never changed as images
built up in memory. A little googling hasn't revealed a commonly used
technique, although I admit I haven't devoted a lot of time to this.

David

[Robin Dunn]

David Woods wrote:
>>> The line of code that fails is:
>>>
>>> # Load the image
>>> self.bgImage = wx.Image(self.obj.image_filename)
>>>
>>> At the moment, I get a console message that says:
>>>
>>> Application transferred too few scanlines
>>>
>>> (I have no idea where this message comes from) and self.bgImage.IsOk()
>>> returns false.
>> It looks like it is coming from libjpeg. See
>> wxWidgets/src/jpeg/jerror.h line #116. It's used in the functions
>> jpeg_finish_compress and jpeg_finish_decompress, with code like this:
>>
>> if (cinfo->next_scanline< cinfo->image_height)
>> ERREXIT(cinfo, JERR_TOO_LITTLE_DATA);
>>
>> My guess is that it's probably a memory or resource issue, and that the
>> actual problem is happening some time before the lines of code above,
>> but that it is manifesting as not enough image data being sent to those
>> functions so that is where it is caught.
>
> Hi everyone,
>
> Thanks for all of your help. I've finally resolved the issue.
>
> It turns out that my wx.Image call was the problem. I wasn't Destroying
> it properly. Since it was defined in the context of a wx.Frame, I
> assumed it would be destroyed when the frame was destroyed, but

Was it holding a reference to the image, something like "self.img =
wx.Image(...)" or was it just used as a local variable like "img =
wx.Image(...)" ?

> apparently that isn't the way it works. It wasn't a child of the frame,
> and the image object was persisting after the frame was destroyed. As I
> built my report, 25 or 30 images would build up in memory until there
> was no room for additional images. It's not clear to me why one image
> failing caused later images to be able to work, but that's what was
> happening. I guess something in wx.Python/wxWidgets was freeing up the
> image memory at that point that my attempts at explicit garbage
> collection weren't freeing up.
>
> The solution was to add an explicit Image.Destroy() call in the
> EVT_CLOSE handler for the Frame object. No more crashes, even with
> significantly more images.

IIUC the garbage collector is not able to do anything with objects that
are implemented as extension module types (unless they implement the
proper protocols) or with classes that have a __del__ method. The
wrapped wx classes in Classic fall into the latter category, and their
.this attribute (a SwigObject) falls into the first category. So object
cleanup of wxPython objects relies solely on old-fashioned reference
counting only.

In the case of wx.Image there is no other object that the ownership of
the C++ object will be transferred to, like is done with widgets that
have a parent window, etc. so the C++ image object should always be
owned by the Python proxy object and will be destroyed when the proxy's
refcount reaches zero. Calling Destroy will force the C++ object to be
deleted and will tell the proxy that it no longer owns the C++ object,
but the proxy will continue to exist until it's refcount reaches zero
and if any of those references tries to use the image then there will be
an exception or a crash.

So it sounds to me like perhaps there are some extra references to the
image object, or perhaps a reference cycle. Since the GC can't wipe
those out you may want to figure out where they are and break the cycles
yourself. Using Destroy will release the bulk of the memory needed for
each image object, but if the proxies are hanging around then you still
have a leak.

>>> img = wx.Image('/Users/robind/Desktop/Snap015.png')
>>> sys.getrefcount(img)
2
>>> img.Destroy()
>>> sys.getrefcount(img)
2

This shows that Destroy does not reduce the refcount of the proxy object
(so the C++ object is not holding a reference to its proxy like we do
with widget objects and a few others.) The count of 2 is expected in
this case, one for the img variable and one for passing it as a
parameter to getrefcount.

> The increasing number of GDI resources being used was a bit of a red
> herring. I knew they weren't getting high enough that they should be
> causing problems. I'm not sure why explicit Frame.Destroy() calls for
> the form and explicit garbage collection isn't freeing those resources,
> but I've found another solution for that. Since the form I use to build
> my complex image is never displayed during report generation, I can skip
> creating its toolbars, which were what was using all the GDI resources.
>
> Is there a common technique for tracking how much memory a program has
> to work with in a case like this? My wx.GetFreeMemory() call clearly
> was not measuring what I thought, as its value never changed as images
> built up in memory. A little googling hasn't revealed a commonly used
> technique, although I admit I haven't devoted a lot of time to this.
>

With Python programs it is kind of an inexact science anyway, since
anything allocated for Python objects is usually not released back to
the OS. The memory will be reused when allocating new Python objects,
but the Python memory manager never fully lets go of it. IIUC. The
wrapped C++ objects are a different story of course.

[David Woods]

Hi Robin,

>>>> The line of code that fails is:
>>>>
>>>> # Load the image
>>>> self.bgImage = wx.Image(self.obj.image_filename)
>>>>
>>>> At the moment, I get a console message that says:
>>>>
>>>> Application transferred too few scanlines
>>>>
>>>> (I have no idea where this message comes from) and self.bgImage.IsOk()
>>>> returns false.
>>> It looks like it is coming from libjpeg. See
>>> wxWidgets/src/jpeg/jerror.h line #116. It's used in the functions
>>> jpeg_finish_compress and jpeg_finish_decompress, with code like this:
>>>
>>> if (cinfo->next_scanline< cinfo->image_height)
>>> ERREXIT(cinfo, JERR_TOO_LITTLE_DATA);
>>>
>>> My guess is that it's probably a memory or resource issue, and that the
>>> actual problem is happening some time before the lines of code above,
>>> but that it is manifesting as not enough image data being sent to those
>>> functions so that is where it is caught.
>>

>> Thanks for all of your help. I've finally resolved the issue.
>>
>> It turns out that my wx.Image call was the problem. I wasn't Destroying
>> it properly. Since it was defined in the context of a wx.Frame, I
>> assumed it would be destroyed when the frame was destroyed, but
>
> Was it holding a reference to the image, something like "self.img =
> wx.Image(...)" or was it just used as a local variable like "img =
> wx.Image(...)" ?

It was (and is) a self.img reference, as I reuse the image in a couple
of methods in the object.

For context, this part of the application is for doing analytic mark-up
of still images using a FloatCanvas. If the user chooses to clear or
temporarily hide some of their mark-up, I need to redraw, and I reuse
self.img during that process.

For the reports, I open the markup window without showing it, load the
image from the file and the mark-up from the database, extract the
marked-up image to put into the report, and close the markup window
without ever showing it.

In terms of avoiding memory leaks, am I better off retaining the
self.img reference, or am I better off just remembering the file name
and creating a new local img when I need one within the markup window?

Okay.... I mostly understand what you're saying, I think. Maybe. But I
haven't got a clue how to DO this "breaking the cycles yourself" of
which you speak.

Basically, I open the SnapshotWindow (a wx.Frame that houses some
toolbars and a FloatCanvas object named canvas, and which loads an image
from a file) without showing it, draw the markup (using _DrawObjects),
and copy the image to tmpBMP, then I close and destroy the Snapshot Window.

Here's the critical code:

# Open a HIDDEN Snapshot Window
tmpSnapshotWindow = SnapshotWindow.SnapshotWindow(menuWindow, -1,
tmpObj.id, tmpObj,
showWindow=False)
# Reset the Bounding Box to avoid NaN problems
tmpSnapshotWindow.canvas._ResetBoundingBox()
# Get the image's Bounding Box
box = tmpSnapshotWindow.canvas.BoundingBox
# Create an empty Bitmap the size of the image
tmpBMP = wx.EmptyBitmap(tmpSnapshotWindow.canvas.GetSize()[0],
tmpSnapshotWindow.canvas.GetSize()[1])
# Get the MemoryDC for the empty bitmap
tempDC = wx.MemoryDC(tmpBMP)
# Set the bitmap's background colour to WHITE
tempDC.SetBackground(wx.Brush("white"))
tempDC.Clear()
# Create a ClientDC for the hidden Shapshot Window
tempDC2 = wx.ClientDC(tmpSnapshotWindow)
# Create another MemoryDC (although I'm not sure why!)
tempDC3 = wx.MemoryDC()
# Get the image from the hidden Snapshot Window's Canvas and
# put it in the Device Contexts we just created
tmpSnapshotWindow.canvas._DrawObjects(tempDC,
tmpSnapshotWindow.canvas._DrawList,
tempDC2, box, tempDC3)
# Close the hidden Snapshot Window
tmpSnapshotWindow.Close()
# Explicitly Delete the Temporary Snapshot Window
tmpSnapshotWindow.Destroy()

To the extent that I understand this, I think I've made a copy of image
drawn on the SnapshotWindow's canvas and don't think I retain any
reference to that canvas or to the wx.Image object used inside the
SnapshotWindow object to populate the FloatCanvas object. But to be
honest, I'm a bit fuzzy about device contexts and exactly how the image
I need ends up in tmpBMP here. I have no idea why 3 device contexts
seem to come into play, for example.

So if Destroy() isn't going to do what I need here, what do I do? Do I
need to explicitly call del(tmpBMP) or del(tmpSnapshotWindow)? Is there
something more I need to do?

Sorry for my ignorance here, but I'm a psychologist by training and am
self-taught with computers, and there are holes in my understanding of
some of this.

Thanks, as always, for your help.

David

[Robin Dunn]

Ok.

>
> For context, this part of the application is for doing analytic mark-up
> of still images using a FloatCanvas. If the user chooses to clear or
> temporarily hide some of their mark-up, I need to redraw, and I reuse
> self.img during that process.
>
> For the reports, I open the markup window without showing it, load the
> image from the file and the mark-up from the database, extract the
> marked-up image to put into the report, and close the markup window
> without ever showing it.
>
> In terms of avoiding memory leaks, am I better off retaining the
> self.img reference, or am I better off just remembering the file name
> and creating a new local img when I need one within the markup window?

No it's best to keep the image object, loading it from disk and
reconverting that data to a wx.Image each time you need it is overhead
that is easily avoidable. The issue here is mainly making sure that
everything will get released when you don't need it any more.

A cycle in this case means a set of objects that have references to each
other such that you can start at one object and follow some path of
references from object to object and eventually get back to the original
object. For example, object A has a reference to object B, which has a
reference to object C, which has a reference to object A. This is the
type of things that Python's garbage collector was designed to take care
of, but as I mentioned before it's not able to deal with objects that
have a __del__ method, or those that are extension module types that
don't have the necessary protocols implemented. It's not just the A, B,
and C objects that can get stuck in memory, but anything else that they
have references to. For example, if B has a reference to your image
then it will be stuck in memory too until that reference is released.

"Breaking the cycle yourself" means providing some way to disconnect one
of the references that is causing the reference graph to have a cycle in
it. For example if object A has a method like this:

def forgetB(self):
self.B = None

Then if that enables B's reference count to drop to zero then it will be
destroyed and the references it had to the image and to C to be
decremented. If their reference count also drops to zero then they will
be destroyed, and so on.

It's also possible to assist the garbage collector after it has detected
unreachable and uncollectable objects by examining the contents of the
gc.garbage list and dealing with your objects that are there. See the
docs for the gc module for more info.

The Blit()'s being done in the _DrawObjects method are essentially just
moving pixel data from one DC to the other. So yes, at that point you
are done with the image object and canvas and etc. However, just
because you don't have a reference to those object from the code above
(other than the local variables that will go away when this code returns
to its caller) that doesn't mean that there isn't a cycle involving your
SnapShot window class, or perhaps something in floatcanvas.

I think I would start by trying to reorganize and simplify the above.
For example it seems to me that getting a bitmap from the snapshot
window would be the job of the snapshot window and not code that is
external to it. So I would give it a method that could create and
return the bitmap. Secondly, creating a window that will never be shown
just to get it to create a bitmap smells really really bad. So I would
probably create a new class that can create and manage the collection of
FC draw objects, and use that class both from this code and also from
SnapshotWindow

> I have no idea why 3 device contexts seem
> to come into play, for example.

It looks like the 3rd DC is used to maintain the hit testing bitmap
inside of the floatcanvas code, so since you are never showing this
window and never interacting with it then I expect that you can just let
it default to None.

>
> So if Destroy() isn't going to do what I need here, what do I do? Do I
> need to explicitly call del(tmpBMP) or del(tmpSnapshotWindow)?

No, local variables are always disposed of at the end of the scope and
that will decrement the reference count of the objects they were
referring to.

> Is there
> something more I need to do?

A. Look for object references that are saved in some other object, and
see if there is some other path back to the original object.

B. Some time after the code in question has run (so you expect the
objects to all have been destroyed naturally by that time) run
gc.collect() and then look at gc.garbage to see if there are any objects
there that you think should have been disposed of but were not.

[Chris Barker - NOAA Federal]

David and Robin,

This all may be all my fault!

I was thinking that there may well be circular references in FloatCanvas, and on the bus this morning  I think I remembered where:

when you add a DrawObject to a canvas, it gets put in self._DrawList -- as you'd expect.

But, it also puts a reference to the Canvas itself in the DrawObject:

    def AddObject(self, obj):

        # put in a reference to the Canvas, so remove and other stuff can work

        obj._Canvas = self

I think that may be the source of the circular reference: the canvas has a reference to a list, the list has a reference to the DrawObject, the DrawObject has a reference to the Canvas, which has a reference to the list.....

I never liked putting in a reference to the canvas in the DrawObject, but there are a couple things that won't work without it.

Robin:

1) is this indeed a circular reference that would cause this problem?

2)  can you think of a way to clean this up? When the DrawObject is remove from the list, is there a way to make sure the extra references go away?

I'm still confused as to why gc.collect wouldn't work, but I'll trust you on that!

-Chris

--
You received this message because you are subscribed to the Google Groups "wxPython-users" group.

To unsubscribe from this group and stop receiving emails from it, send an email to wxpython-users+unsubscribe@googlegroups.com.

For more options, visit https://groups.google.com/groups/opt_out.

[Robin Dunn]

Chris Barker - NOAA Federal wrote:
> David and Robin,
>
> This all may be all my fault!
>
> I was thinking that there may well be circular references in
> FloatCanvas, and on the bus this morning I think I remembered where:
>
> when you add a DrawObject to a canvas, it gets put in self._DrawList --
> as you'd expect.
>
> But, it also puts a reference to the Canvas itself in the DrawObject:
>
> def AddObject(self, obj):
> # put in a reference to the Canvas, so remove and other stuff
> can work
> obj._Canvas = self
>
> I think that may be the source of the circular reference: the canvas has
> a reference to a list, the list has a reference to the DrawObject, the
> DrawObject has a reference to the Canvas, which has a reference to the
> list.....
>
> I never liked putting in a reference to the canvas in the DrawObject,
> but there are a couple things that won't work without it.
>
> Robin:
>
> 1) is this indeed a circular reference that would cause this problem?

Yes, it is.

> 2) can you think of a way to clean this up? When the DrawObject is
> remove from the list, is there a way to make sure the extra references
> go away?

Something as simple as:

obj._Canvas = None

should work. Removing the objects from the list should do it too
however, so if you're doing that already and the draw objects are still
hanging around then perhaps there are some other references contributing
to the cycle somewhere. (Assuming that you empty the list when the
canvas window is closed.) Hmm... I wonder if the WIT could/should grow
a GC Cycle Visualization Tool, (one that wouldn't actually make the
problem worse by holding it's own references to the objects.)

>
> I'm still confused as to why gc.collect wouldn't work, but I'll trust
> you on that!

This is the reason usually given about it, from
http://docs.python.org/2/library/gc.html:

"""
Objects that have __del__() methods and are part of a reference cycle
cause the entire reference cycle to be uncollectable, including objects
not necessarily in the cycle but reachable only from it. Python doesn�t
collect such cycles automatically because, in general, it isn�t possible
for Python to guess a safe order in which to run the __del__() methods.

[David Woods]

FWIW, after cleaning up my code quite a bit, I'm currently left with
only 4 wxCursors and a PyTimer object in the uncollected garbage. I
can't say that they come from FloatCanvas rather than my code yet, as I
still have some work to do.

And many thanks, Robin, for your suggestions and your patience.

David

[David Woods]

On 05/10/2013 04:59 PM, Chris Barker - NOAA Federal wrote:
> This all may be all my fault!

Hi Chris,

It's definitely not all your fault. With Robin's guidance, I've been
able to resolve most of the issues by adjusting my code.

FloatCanvas is a fabulous tool that saved me many, many hours. I'm
grateful to you for developing it.

David

[Chris Barker - NOAA Federal]

On Mon, May 13, 2013 at 8:47 AM, David Woods <tran...@gmail.com> wrote:

On 05/10/2013 04:59 PM, Chris Barker - NOAA Federal wrote:

This all may be all my fault!

 

It's definitely not all your fault.  With Robin's guidance, I've been able to resolve most of the issues by adjusting my code.

I woulnd't hav used teh term "fault" if I was talking about anyone but myself.

But I think one _should_ be able to create, add, remove, delete an unlimited number of DrawObjects without filling up memory.

But I'm still not sure how to do that. The "right" way would be to remove the reference to the Canvas in the objects, but that's going to take more af a refactor than I can do at the moment.

I'll poke at it a bit more...

-Chris

 

FloatCanvas is a fabulous tool that saved me many, many hours.  I'm grateful to you for developing it.

David

--
You received this message because you are subscribed to the Google Groups "wxPython-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to wxpython-users+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

[Chris Barker]

Robin et al,

I'm kept this old thread highlighted in my email for al ong time, as I really want to fix the problem in FloatCanvas:

I was thinking that there may well be circular references in
FloatCanvas, and on the bus this morning  I think I remembered where:

 

     def AddObject(self, obj):

         # put in a reference to the Canvas, so remove and other stuff
can work
         obj._Canvas = self

so I had been thinking that I'd get rid of that -- but it would require a significant refactoring of the event handling, so have not gotten around to it.

However, I recently ran into this with another library, and solved it by using a weak reference. So in this case, something like:

import weakref

def AddObject(self, obj):

         # put in a reference to the Canvas, so remove and other stuff

can work

         obj._Canvas = weakref.proxy(self)

I'm still confused as to why gc.collect wouldn't work, but I'll trust
you on that!

This is the reason usually given about it, from http://docs.python.org/2/library/gc.html:

"""

Objects that have __del__() methods and are part of a reference cycle cause the entire reference cycle to be uncollectable, including objects not necessarily in the cycle but reachable only from it. Python doesn’t collect such cycles automatically because, in general, it isn’t possible for Python to guess a safe order in which to run the __del__() methods.

"""

weak references are supposed to solve that, but  compiled object need to be "weak referenceable":

http://docs.python.org/2/extending/newtypes.html#weakref-support

Do the wxPython types do that???

-Chris

[Robin Dunn]

Chris Barker wrote:
>
> weak references are supposed to solve that, but compiled object need to
> be "weak referenceable":
>
> http://docs.python.org/2/extending/newtypes.html#weakref-support
>
> Do the wxPython types do that???
>

Classic doesn't use extension Types, just Python classes with the
methods calling functions in the extension module. However for widgets
classes and some others there are references to the objects being held
in C++ code so I think that may mess up weak references.

In Phoenix extension Types are used for the wrapped classes. I would
have to check for sure but I think that they do implement what is needed
for use with weakref.

[Chris Barker]

On Fri, Mar 14, 2014 at 9:52 PM, Robin Dunn <ro...@alldunn.com> wrote:

Chris Barker wrote:

weak references are supposed to solve that, but  compiled object need to
be "weak referenceable":

http://docs.python.org/2/extending/newtypes.html#weakref-support

Do the wxPython types do that???

Classic doesn't use extension Types, just Python classes with the methods calling functions in the extension module.  However for widgets classes and some others there are references to the objects being held in C++ code

hmmm -- so that _should_ be plain old python __del__ methods, which will defat teh Garbage Collector, but _should_ work with weak references.

 

so I think that may mess up weak references.

or not ;-)

I'll have to give it a try. 

 

In Phoenix extension Types are used for the wrapped classes.  I would have to check for sure but I think that they do implement what is needed for use with weakref.

Cool --once I get some test code going with Classic, maybe I"ll see if someone wants to test with Phoenix -- or finally get around to giving it a try myself ....

Thanks,

  -Chris