[Maya-Python] Thread with callback

296 views
Skip to first unread message

Marcus Ottosson

unread,
Mar 25, 2015, 2:39:28 PM3/25/15
to python_in...@googlegroups.com

Hi all,

I threw something together on a whim and would like your opinion of it.

Inspired by JavaScript and it’s use of callbacks for IO-bound functions:

expensiveFunction("argument", function(result) {
    console.log("The results are: " + result);
})

I did this.

def callback(result):
    print "The results are: %s" % result

invoke(expensive_function, callback)

Code here:
https://gist.github.com/mottosso/c20df396f4ecc882b53c

Question is, is there already a way of doing this? If not, what could be made better, faster, stronger? I’m calling it “invoke” but I’m sure there’s already an established term for it, do you know of any?

Best,
Marcus

​
--
Marcus Ottosson
konstr...@gmail.com

Justin Israel

unread,
Mar 25, 2015, 3:11:52 PM3/25/15
to python_in...@googlegroups.com

This looks kind of similar to something I did a while back

https://gist.github.com/justinfx/6183367

We even both use the term "invoke"  :-)
Mine uses a posted event to the main loop instead of signals and slots.


--
You received this message because you are subscribed to the Google Groups "Python Programming for Autodesk Maya" group.
To unsubscribe from this group and stop receiving emails from it, send an email to python_inside_m...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/python_inside_maya/CAFRtmOD6jDvWtY39eBEbQVU95HB2T0wF89GHeXSAdHEnUn5yDA%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Marcus Ottosson

unread,
Mar 25, 2015, 3:56:45 PM3/25/15
to python_in...@googlegroups.com

Great minds think alike? :)

You also use “dispatch” which sounds like another option; we’re dispatching a function onto a thread that triggers a callback once finished.

dispatch(expensive_function, callback=on_completed)

I didn’t quite get the interface for your version, does it require a subclass?

Here’s what I’ve been doing previously and was looking to simplify:

# From within a class
self.finished.connect(self.on_finished)

def worker():
    time.sleep(3)

    # Relies on an existing signal in the caller
    self.finished.emit("result")

# And creates junk variables that will never get referenced.
thread = threading.Thread(target=worker)
thread.daemon = True
thread.start()

Which is quite the many lines for such a simple thing; I’m refraining from doing too much of it due to the visual clutter it causes, along with having to make signals serve single-shot purposes, one per unique task more or less.

Are you using anything like this in production? I had never seen or used anything like it before outside of JavaScript, but in there it’s everywhere and is incredibly useful and intuitive and I find a need for it in quite a few places.

​

To view this discussion on the web visit https://groups.google.com/d/msgid/python_inside_maya/CAPGFgA1QfFCxB0wUv-Z8kQefi-koshdUjLQZgwTZbx-juAW1uw%40mail.gmail.com.

For more options, visit https://groups.google.com/d/optout.



--
Marcus Ottosson
konstr...@gmail.com

Justin Israel

unread,
Mar 25, 2015, 3:59:41 PM3/25/15
to python_in...@googlegroups.com
If I remember correctly, the original motivation for my version was that I encountered an issue with callbacks where they were methods of objects that could be deleted before the callback gets a chance to run, and the fact that holding a reference to the methods would prevent garbage collection (which also triggers signal/slot auto cleanup, etc). So I had come across some information about weakref callbacks. I made some modifications to some examples I had found. The result is that you can use bound methods as callbacks, and run your initial logic through a thread pool. Then the callback will be executed in the main thread, but can quietly fail if the owning object was deleted. And it won't hold a reference to the owning object. 


To unsubscribe from this group and stop receiving emails from it, send an email to python_inside_maya+unsub...@googlegroups.com.

Justin Israel

unread,
Mar 25, 2015, 4:07:21 PM3/25/15
to python_in...@googlegroups.com
On Thu, Mar 26, 2015 at 8:56 AM Marcus Ottosson <konstr...@gmail.com> wrote:

Great minds think alike? :)

You also use “dispatch” which sounds like another option; we’re dispatching a function onto a thread that triggers a callback once finished.

dispatch(expensive_function, callback=on_completed)

I didn’t quite get the interface for your version, does it require a subclass?


There are usage example right at the bottom of the gist. It gets used like a normal threadpool:
self._pool = CallbackThreadPool(4)
...
self._pool.apply_async(action, (i, i), callback=callback)
​

You don't have to subclass anything by default, since it will just direct the event to the "Invoker" instance. But you *can* direct it to specific objects which can handle the posted event specifically. 
 
Once you have a thread pool, you just use it. You can either have one thread pool for your app, or thread pool instances in your specific classes. Whatever you want. I found that I needed to weakref functionality in there, if you are dealing with bound methods. And having the thread pool is nice to keep a bounded amount of threads running in the app. 

Here’s what I’ve been doing previously and was looking to simplify:

# From within a class
self.finished.connect(self.on_finished)

def worker():
    time.sleep(3)

    # Relies on an existing signal in the caller
    self.finished.emit("result")

# And creates junk variables that will never get referenced.
thread = threading.Thread(target=worker)
thread.daemon = True
thread.start()

Which is quite the many lines for such a simple thing; I’m refraining from doing too much of it due to the visual clutter it causes, along with having to make signals serve single-shot purposes, one per unique task more or less.

Are you using anything like this in production? I had never seen or used anything like it before outside of JavaScript, but in there it’s everywhere and is incredibly useful and intuitive and I find a need for it in quite a few places.

I'm using this CallbackThreadPool in production, in one of my Qt applications. I use it to process non-gui stuff in the threadpool and then safely callback to GUI logic when it is done.
 
To unsubscribe from this group and stop receiving emails from it, send an email to python_inside_maya+unsub...@googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "Python Programming for Autodesk Maya" group.
To unsubscribe from this group and stop receiving emails from it, send an email to python_inside_maya+unsub...@googlegroups.com.
--
You received this message because you are subscribed to the Google Groups "Python Programming for Autodesk Maya" group.
To unsubscribe from this group and stop receiving emails from it, send an email to python_inside_maya+unsub...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/python_inside_maya/CAFRtmODMVxFJFzdk6%2BEvRiQZte514-uwYZXPLTwT40vjLTaq%2BQ%40mail.gmail.com.

Marcus Ottosson

unread,
Mar 25, 2015, 5:18:07 PM3/25/15
to python_in...@googlegroups.com
Perfect, thanks a bunch Justin!

On 25 March 2015 at 20:07, Justin Israel <justin...@gmail.com> wrote:
On Thu, Mar 26, 2015 at 8:56 AM Marcus Ottosson <konstr...@gmail.com> wrote:

Great minds think alike? :)

You also use “dispatch” which sounds like another option; we’re dispatching a function onto a thread that triggers a callback once finished.

dispatch(expensive_function, callback=on_completed)

I didn’t quite get the interface for your version, does it require a subclass?


There are usage example right at the bottom of the gist. It gets used like a normal threadpool:

```python
To unsubscribe from this group and stop receiving emails from it, send an email to python_inside_m...@googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "Python Programming for Autodesk Maya" group.
To unsubscribe from this group and stop receiving emails from it, send an email to python_inside_m...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/python_inside_maya/CAPGFgA1QfFCxB0wUv-Z8kQefi-koshdUjLQZgwTZbx-juAW1uw%40mail.gmail.com.

For more options, visit https://groups.google.com/d/optout.



--
Marcus Ottosson
konstr...@gmail.com

--
You received this message because you are subscribed to the Google Groups "Python Programming for Autodesk Maya" group.
To unsubscribe from this group and stop receiving emails from it, send an email to python_inside_m...@googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "Python Programming for Autodesk Maya" group.
To unsubscribe from this group and stop receiving emails from it, send an email to python_inside_m...@googlegroups.com.
 To view this discussion on the web visit https://groups.google.com/d/msgid/python_inside_maya/CAPGFgA0X0tnrCPzkhOkwO2YMUM_W2BA7svG-huz-zySq%3DTJy_g%40mail.gmail.com.

For more options, visit https://groups.google.com/d/optout.



--
Marcus Ottosson
konstr...@gmail.com

Robert White

unread,
Mar 26, 2015, 2:26:29 AM3/26/15
to python_in...@googlegroups.com
If you're worried about event handlers keeping callback objects alive when they should have already been wiped out.

Use the weakref module.


I ended up using a variation of this for my MSceneMessage handler system. Been working really well for me.
To unsubscribe from this group and stop receiving emails from it, send an email to python_inside_maya+unsubscribe@googlegroups.com.

Justin Israel

unread,
Mar 26, 2015, 3:23:55 AM3/26/15
to python_in...@googlegroups.com


On Thu, 26 Mar 2015 7:26 PM Robert White <robert....@gmail.com> wrote:

If you're worried about event handlers keeping callback objects alive when they should have already been wiped out.

Use the weakref module.

They have a pretty good example at http://code.activestate.com/recipes/578298-bound-method-weakref/.

I ended up using a variation of this for my MSceneMessage handler system. Been working really well for me.

That's what my version uses. The callbacks are stored as weakrefs


On Wednesday, March 25, 2015 at 2:59:41 PM UTC-5, Justin Israel wrote:

If I remember correctly, the original motivation for my version was that I encountered an issue with callbacks where they were methods of objects that could be deleted before the callback gets a chance to run, and the fact that holding a reference to the methods would prevent garbage collection (which also triggers signal/slot auto cleanup, etc). So I had come across some information about weakref callbacks. I made some modifications to some examples I had found. The result is that you can use bound methods as callbacks, and run your initial logic through a thread pool. Then the callback will be executed in the main thread, but can quietly fail if the owning object was deleted. And it won't hold a reference to the owning object. 


On Thu, Mar 26, 2015 at 8:11 AM Justin Israel <justin...@gmail.com> wrote:


This looks kind of similar to something I did a while back

https://gist.github.com/justinfx/6183367

We even both use the term "invoke"  :-)
Mine uses a posted event to the main loop instead of signals and slots.


On Thu, 26 Mar 2015 7:39 AM Marcus Ottosson <konstr...@gmail.com> wrote:


Hi all,

I threw something together on a whim and would like your opinion of it.

Inspired by JavaScript and it’s use of callbacks for IO-bound functions:

expensiveFunction("argument", function(result) { console.log("The results are: " + result); })

I did this.

def callback(result): print "The results are: %s" % result invoke(expensive_function, callback)

Code here:
https://gist.github.com/mottosso/c20df396f4ecc882b53c

Question is, is there already a way of doing this? If not, what could be made better, faster, stronger? I’m calling it “invoke” but I’m sure there’s already an established term for it, do you know of any?

Best,
Marcus

​


--

Marcus Ottosson
konstr...@gmail.com


--
You received this message because you are subscribed to the Google Groups "Python Programming for Autodesk Maya" group.

To unsubscribe from this group and stop receiving emails from it, send an email to python_inside_m...@googlegroups.com.


To view this discussion on the web visit https://groups.google.com/d/msgid/python_inside_maya/CAFRtmOD6jDvWtY39eBEbQVU95HB2T0wF89GHeXSAdHEnUn5yDA%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "Python Programming for Autodesk Maya" group.

To unsubscribe from this group and stop receiving emails from it, send an email to python_inside_m...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/python_inside_maya/fa3f0627-14a3-43bf-95b9-9ed02bc4d4fa%40googlegroups.com.

Marcus Ottosson

unread,
Mar 26, 2015, 3:53:38 AM3/26/15
to python_in...@googlegroups.com

Yes, I would agree with that, weakref is one way of solving that. Haven’t run into a scenario where the thread outlives the caller yet though, but suspect I might.

The result is that you can use bound methods as callbacks, and run your initial logic through a thread pool. Then the callback will be executed in the main thread, but can quietly fail if the owning object was deleted. And it won’t hold a reference to the owning object.

The version I posted takes bound methods too, it also runs in the main loop, due to QtCore.Qt.BlockingQueuedConnection.

The remaining difference between our results looks to be that you are limiting thread count; where have you found that to come in handy? Why limit the amount of threads to begin with? How many threads would you end up with otherwise?

​

To view this discussion on the web visit https://groups.google.com/d/msgid/python_inside_maya/CAPGFgA2fRpf38gaFUFMFYpZ788VEcPrkcAMeCSf3gRLZx-zi%2BQ%40mail.gmail.com.

For more options, visit https://groups.google.com/d/optout.



--
Marcus Ottosson
konstr...@gmail.com

Marcus Ottosson

unread,
Apr 7, 2015, 10:51:50 AM4/7/15
to python_in...@googlegroups.com
Hey Justin,

I've been using Invoke now since we last spoke of it and things have been going quite well, and I haven't yet seen the need to limit the amount of threads running within the application with a thread pool.

But I have gotten another problem for which a thread pool might be helpful, which is managing the order in which threads finish. 

What's happening now is that threads are spawned independently of each other and finishes whenever they finish, and even though I have no immediate coupling between them, I do get some rather odd unexplainable things happening - particularly related to the creation and destruction of objects that are somehow managed by separate threads.

I'm suspecting it may be due to one thread finishing, calling back to the main thread to perform some function, but as it performs, another thread finishes and makes an update to the same data structure.

Either way, what I was thinking is that a thread pool could be the solution to this, while also making the implementation easier to understand. I'd simply have a single threadpool, with a single worker, and pass all work to it. I would still get asynchronousy, which is the only goal, but with synchronous behaviour.

The point is of course null if multiple workers are introduced, so was wondering what your thoughts on this?
--
Marcus Ottosson
konstr...@gmail.com

Justin Israel

unread,
Apr 7, 2015, 5:17:28 PM4/7/15
to python_in...@googlegroups.com
On Wed, Apr 8, 2015 at 2:51 AM Marcus Ottosson <konstr...@gmail.com> wrote:
Hey Justin,

I've been using Invoke now since we last spoke of it and things have been going quite well, and I haven't yet seen the need to limit the amount of threads running within the application with a thread pool.

Right, it just depends on how your app is structured. An example would be if your app listening to external triggers of some sort and does work. Since you have no control over these triggers, you could have unbounded threads being launched. So in that circumstance a threadpool would give you some form of control.

Also, normally one does work in a background thread if it is something that will take some time and would block the main gui thread. So if it is possible that there are triggers within your own application that can schedule multiple requests, then it would also make sense to bound it in some way. 

Anyways, if you didn't need it in your app, then its totally understandable. 
 

But I have gotten another problem for which a thread pool might be helpful, which is managing the order in which threads finish. 

What's happening now is that threads are spawned independently of each other and finishes whenever they finish, and even though I have no immediate coupling between them, I do get some rather odd unexplainable things happening - particularly related to the creation and destruction of objects that are somehow managed by separate threads.

I'm suspecting it may be due to one thread finishing, calling back to the main thread to perform some function, but as it performs, another thread finishes and makes an update to the same data structure.

Either way, what I was thinking is that a thread pool could be the solution to this, while also making the implementation easier to understand. I'd simply have a single threadpool, with a single worker, and pass all work to it. I would still get asynchronousy, which is the only goal, but with synchronous behaviour.

The point is of course null if multiple workers are introduced, so was wondering what your thoughts on this?

Sounds like a good plan to me when you need to maintain order to the queued commands. If a threadpool wasn't available, you would just end up writing a Queue and a single thread that sits on it, and putting work into the queue. It just happens that a threadpool handles that for you when set to one thread :-)
 
Reply all
Reply to author
Forward
0 new messages