QGraphicsProxyWidget causes a crash when reloading

[vince touache]

hello, 

I'm wrapping a QWidget inside a QGraphicsItem, which is in turn used in a QGraphicsView/Scene (all using PySide2)

To do so, I'm using a QGraphicsProxyWidget (first time I'm using it), like below : 

proxy = QGraphicsProxyWidget(self)  # I'm inside my QGraphicsItem, so self == QGraphicsItem

proxy.setWidget(myWidget)

It works, but after 4 reloads/instanciating my view, it crashes my application (in that case, Maya). 

import graphic_view;reload(graphic_view)

v = graphic_view.graphic_view()

v.close()

# ---- x1 is ok ------

# ---- x2 is ok ------

# ---- x3 is ok ------

# ---- x4 ==> crashing ------

I noticed that if I use another variable to store my new instance (x instead of v, they y, z, w, ....) it doesn't crash

So I guess it has to do with how the memory is released (or not released where it should ! Maybe some parts are, some other not, which would mess with the garbage collection mechanism ?).

If anyone has a better understanding of proxy widgets and what I am doing wrong, that'd be really appreciated. And more generally speaking, I always feel a bit powerless when I face situations like this : I usually uncomment pieces of code until I isolate the issue, when I get crashes, but that seems really like a cheap debugging workflow. But no way to attach debugger or anything like that, since the crash will simply close the application. How do you guys deal with this kind of problem ?

Thank you !

[Marcus Ottosson]

Hi Vincent!

Because it’s a broad question, I’ll provide a broad answer.

This is most likely the result of reloading without freeing resources claimed by Qt. When you reload, Python does a good job garbage collecting all of its items, but it cannot know the order in which Qt’s objects should be freed. In this case, it’s likely the assigned widget to your proxy need to be cleaned up first; but as Python doesn’t know of this relationship, it cannot take that information into account during garbage collection, which would in that case delete something that Qt then tried to delete again, which in C++ land would be undefined. Undefined means it may crash, or it may not. Which means the first 3 times you try, you’re in luck. And on the 4th you’re not. Undefined doesn’t necessarily mean random though, so it may even consistently not-crash the first 3, and crash consistently on the 4th.

You can avoid situations like this altogether with a little programming hygene. What I typically do is two things.

1. Anything you install needs an equivalent uninstall
2. Instead of freeing with reload(), free with sys.modules.pop

The uninstall in this case could consist of un-assigning the widgets to the proxy before freeing. Qt is able to automatically free objects if they form a hierarchy, like when you assign widgets to a layout. I would expect it to also handle cases where you assign a widget like in your example, but odds are it doesn’t in which case you need to do it yourself.

Generally, I structure my projects into an install() and uninstall() function, where uninstall() does everything install() does in reverse; including making connections to databases, or creating menus in Maya or loading images and fonts off of disk etc. On calling uninstall() I expect a Maya in the exact same condition that it was when I first called install(). This could happen either at a package, module and/or class level.

Freeing with sys.modules.pop("mymodule") is relevant when you work with multiple Python modules that reference each other. You’ve probably already found that if you reload a module that imports another module, that imported module isn’t reloaded. It would hold onto any references it made - including widgets that the parent module made use of - which means the module you reload would re-import that persistent module and thus gain access to whatever data it previously held. You could work around it by reloading that module too, but the issue arises when modules form a hierarchy and worse yet when there’s diamond or cyclic dependency between modules.

In that case you’ve probably also found that the order in which you reload becomes important and then down the rabbit hole you go.

To solve this you can instead divide the work of unloading and loading by removing entries from sys.modules.

# For example
import mymodule

import sys
for module in list(sys.modules):
  if module.startswith("mymodule"):
    sys.modules.pop(module)

import mymodule

The .pop() has the same effect as reload, except it doesn’t automatically re-import it, the .startswith ensures that any sub-modules are taken into account as well. Then when you next import, it will import it in whatever order it was imported the first time around. This also guarantees that the modules you’ve now imported won’t contain any references from before and that it’s an entirely clean slate.

It will also make apparent which of those references you were dependent on and throw lots more errors due to things actually disappearing and not lingering. These are the errors you want to get rid of.

Somewhat general advice, but having followed this method for many years I haven’t had these issues when iterating on modules or packages in Maya (or any embedded environment) and certainly no crashes because of it.

For more specific advice about this specific crash, maybe try and replicate it in mayapy, in e.g. a loop that imports it over and over until it crashes. Such that you can cause a crash fast and consistently. Once you’ve got it, reduce it down to the few lines that cause it and take things from there.

--
You received this message because you are subscribed to the Google Groups "Python Programming for Autodesk Maya" group.
To unsubscribe from this group and stop receiving emails from it, send an email to python_inside_m...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/python_inside_maya/318b7121-df53-4743-9355-a1946d6e3da5%40googlegroups.com.

[vince touache]

Hi Marcus ! Hope you're doing well =]

Thanks for the detailed answer ! In fact, I was already flushing my main module through the sys.modules, but never really implemented it fully in my code, reload always did the trip (and when it did not, it was on a basic level I could solve from the script editor)

I'm now flushing all the modules I want to reload from within the main module, and it works fine, so as suspected, some weirdness in the garbage collection (which in fact, seems rather common when I start dealing with views, scenes, or less straight-forward configurations) !

Now I'm wondering why there is nothing better than reload that has been developped yet ? Some sort of one optimised function to flush'em all ! 

Ages ago, I found this page (http://pyunit.sourceforge.net/notes/reloading.html), which seems to start something, but is there anything out there which works in every situation and is clean / easy to use enough for everybody to implement it in their code ?

Also, I was wondering how the "as" was handled (e.g. import module as md). I never really thought about it, but at first glance I would say this is essentially a variable storing the module itself ; therefore, as long as you flush the module and re-import it, you don't have to worry about the namespace you use via the "as", correct ?

Anyway, thanks ! I feel stupid because even though I knew how to flush properly, I just couldn't think of this by myself, which makes me feel useless -___-' 

To unsubscribe from this group and stop receiving emails from it, send an email to python_inside_maya+unsub...@googlegroups.com.

[Justin Israel]

Hey Vincent,

Marcus did a nice job explaining some of the issues with reload(). I wanted to expand on that.

Really, the reload() function should not be used in a production environment; it should just be a development tool. As Marcus pointed out, reload will not recursively reload all of the imports triggered by that main module. ipython provided a deepreload() to cover this need, but again ipython is meant for development and not production:
https://ipython.org/ipython-doc/stable/api/generated/IPython.lib.deepreload.html 

So one thing that hasn't been mentioned yet is the impact of reloading compiled python extension modules (like Qt). When we try and reload just a bunch of pure python modules we already have the issue that not every module will be recursively reloaded, and that any references that were imported by some other module are going to be the old references. When a compiled module is imported into python, at least on posix systems, a dlopen() loads that compiled module. There is really no way to completely unload that module from the processes memory (as far as I know) and a dlclose() only unloads the module by sort of tossing the reference out so that it can be dlopen()'d again. From the man page of dlclose

DLOPEN(3)
...

DESCRIPTION:
...

dlclose()
    A  successful  return  from  dlclose()  does not guarantee that the symbols associated with handle are removed from the caller's address space. 

This doesn't directly pertain to your situation of reloading your python qt source code, since you don't need the Qt modules to be reloaded, but I just wanted to point out how reload() isn't a reliable production solution. 

That parent-child garbage collection mechanism in Qt can both work for you and against you. It's nice in the sense that entire Qt hierarchies will be cleaned up when a top level parent is deleted. But if any bit of code in the entire Maya process were to hold onto the reference of an object from your code, that chain may not be deleted when you clean up and reload. So at any point later a garbage collection could be triggered on stale python wrappers around C++ pointers and result in a segfault. This can happen if things were parented to, say, the maya main window.

Is it possible to completely remove the use of reload from your production deployments and only use it when iterating on development code, and just accept that it is not reliable?

Justin

To unsubscribe from this group and stop receiving emails from it, send an email to python_inside_m...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/python_inside_maya/318b7121-df53-4743-9355-a1946d6e3da5%40googlegroups.com.

--

You received this message because you are subscribed to the Google Groups "Python Programming for Autodesk Maya" group.

To unsubscribe from this group and stop receiving emails from it, send an email to python_inside_m...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/python_inside_maya/a0892932-833a-4986-9ac2-c1f7b84da110%40googlegroups.com.

[vince touache]

then, you have 2 versions of the same code giving 2 different results, this is what i dont like ^^ but i can livre with deleting manually the modules. At least for now. Take a bit more manuel work but at least I have consistant résults, which is similar to what I would have without a maya persistent env.

I m still not sure why the child dont get removed in the garbage collection , considering the parent child relation exists. Also, can we re launch mayapy separately from maya ? that would automatically flush it family !

thanks

To view this discussion on the web visit https://groups.google.com/d/msgid/python_inside_maya/CAPGFgA1cVz8vcVAFL8x2wEp5TgB%3DTtftAMJFQfSDQuUg%3DipWjw%40mail.gmail.com.