Dispatcher and Win32 message pump blues

[Jeremiah Morrill]

In my WPF MediaKit, I create a new thread/Dispatcher to run all my COM components.  I do this primarily because:

1. WPF runs STA, and sometimes I want to run MTA.
2. There are some blocking calls on the DirectShow that would block the UI thread.
3. I sometimes need a Win32 message pump.  WPF's Dispatcher provides this.
4. It's very convenient. 

Here's an example of the issue I'm having.  

UI Dispatcher does an async BeginInvoke to DShow Dispatcher, telling it to OpenFile().  The UI Dispatcher may, in some cases, execute a BeginInvoke on the DShow Dispatcher telling it to Play().  That Play() method is queued in the DShow Dispatcher (good).  While the DShow Dispatcher is executing my "OpenFile()", it is initializing a lot of COM stuffs.  These calls appear to be syncronous, but under the hood, they are posting Win32 messages.  So I may do:

dshowGraph.RenderFile(filename, string.empty)

But that call will reactivate the WPF Dispatcher (as it handles win32 message pumping) and it will invoke the queued "Play()" command before it even finishes opening up the media.  I played around with Dispatcher.DisableProcessing, but that just ends up with a "Dispatcher processing has been suspended, but messages are still being processed." exception.  I don't want to disable processing anyways (deadlocks in Dshow if u do), I just want it to process Win32 messages at this point time and ignore any delegates in the queue.

I've created my own rudimentary Dispatcher/DispatcherObject that processes CLR delegates first, then processes the Win32 message pump manually.  This seems to fix my issue generally, but I'd still like to use the WPF Dispatcher as its more robust and slightly more efficient...and ironically, trust it a bit more :).

So my question is, is there something totally fubar with my general approach?  I'd rather try to get this done right vs. covering it up with turd.  It's not a critical bug as it doesnt cause any crashes...but it does throw an MDA alert if connected via debugger, which is annoying.  I have a hard time convincing people this is NOT an exception...but because VS makes it look like one..to them, it is.

Thanks for any suggestions you guys may have!

-Jer

[Josh Smith]

Have ya tried googling it yet?  (joking)

[Jeremiah Morrill]

Funny, Google told me to ask you guys!

[David Anson]

Bing and decide.

[Jeremiah Morrill]

Bing didn't have much to say about this issue...but it did find a nice restaurant I would like to take my g/f to.

-Jer

[Eric Burke]

Can you share any of your code, even if privately?

---

From: Jeremiah Morrill <jeremiah...@gmail.com>
To: wpf-di...@googlegroups.com
Sent: Wed, September 23, 2009 7:26:00 PM
Subject: [WPF Disciples] Dispatcher and Win32 message pump blues

[Jeremiah Morrill]

Out of office now, but check out the latest wpfmediakit.codeplex.com.  The workerdispatcher/workerdispatcher classes are my simple dispatcher implementations.   I have to send u my test code when I get back to show the issue.  

[Jaime Rodriguez]

Hey Jer,

 

I Dwayned* it in-house and this is what I got back..

The WPF dispatcher is not just a queue of delegates.  It is deeply integrated with the Win32 message pump design.  For instance, delegates queued at “foreground” priorities are interleaved with other posted win32 messages.  Further, we guarantee that delegates queued at “background” priorities will only be dispatched after all other posted/sent/input Win32 messages are dispatched.  We don’t provide any way to cheat.  Because cheating would break the contract for other consumers of the Dispatcher API that depend on that contract.

 

The DShow APIs you are calling are evidently running their own nested message pumps.  This means that they call GetMessage() and DispatchMessage().  If they dispatch our ProcessQueue message, we will process the next delegate in the queue in priority order.  If we didn’t we would be breaking the contract.  Imagine code that posts a work item at Normal priority.   This item, by definition, must be executed before input is processed.  But the nested message pump will very likely pump input messages.  So if we don’t respond when it is our turn, then we break the contract.

 

Nested message pumps cause reentrancy.  WPF is designed to continue working even if an app causes reenentracy – such as your app is causing by calling into DShow.  This is important, because we need layout, rendering, animations, etc, etc to continue working.  We do offer a mode, as you have noted, where we will throw an exception if there is any reentrancy (this is a defensive mode for code paths that cannot survive reentrancy).  But we do not offer a mode where we simply suspend the dispatcher during reentrancy.

 

Now, of course, you can fix for your scenario in several ways.  For example, you could keep your own list of async operations.  Then you could post one work item to do the next thing in your list.  You could even ensure that you won’t do anything on your list until you have finished the previous item (refuse reentrancy).  Of course, I’m not sure how much value the WPF dispatcher is then providing for you…

 

 

*Dwayne Need is a Dev Manager in WPF. He is not as handsome as dr. wpf, but he knows his stuff equally well and he owns the dispatcher.  

[Jeremiah Morrill]

Wow, thats an awesome and very thorough answer!  Dwayne has always come through with great info!  Thanks guys!

I think the interleaved Win32/Delegate queue is what was causing me pain with the WPF Dispatcher.  Very understandable on why it is the way it is, but it surely doesn't play nice with the odd ball stuff I'm doing with it.  I don't know if anyone has ever peeked at the Dispatcher code...it's quite a bit more complex than the 4 line pump we're used to seeing in Win32 apps.

I'm going the route of finishing/un-nastifing up my own Dispatcher-like pump.  My prototype had nasty reset events w/ a short timeout, mixed with PeekMessage.  Now it's a little more efficient, using GetMessage and no reset events.  I wake up the pump by posting a user message to the thread (WM_DISPATCHER_NOTIFY).  On this, I process the managed queue separate from the win32 message queue and so far so good!  

Here's the heart of what I'm doing.

            while (GetMessage(out msg, IntPtr.Zero, 0, 0) && !m_shutdown)

            {

                switch (msg.message)

                {

                    case WM_DISPATCHER_NOTIFY:

                        DoManagedQueue();

                        break;

                    default:

                        break;

                }

                TranslateMessage(ref msg);

                DispatchMessage(ref msg);

            }

[Daniel Vaughan]

What an excellent set of posts guys, cheers.

Daniel

On Sep 24, 7:46 am, Jaime Rodriguez <jai...@microsoft.com> wrote:
> Hey Jer,
>
> I Dwayned* it in-house and this is what I got back..
> The WPF dispatcher is not just a queue of delegates.  It is deeply integrated with the Win32 message pump design.  For instance, delegates queued at "foreground" priorities are interleaved with other posted win32 messages.  Further, we guarantee that delegates queued at "background" priorities will only be dispatched after all other posted/sent/input Win32 messages are dispatched.  We don't provide any way to cheat.  Because cheating would break the contract for other consumers of the Dispatcher API that depend on that contract.
>
> The DShow APIs you are calling are evidently running their own nested message pumps.  This means that they call GetMessage() and DispatchMessage().  If they dispatch our ProcessQueue message, we will process the next delegate in the queue in priority order.  If we didn't we would be breaking the contract.  Imagine code that posts a work item at Normal priority.   This item, by definition, must be executed before input is processed.  But the nested message pump will very likely pump input messages.  So if we don't respond when it is our turn, then we break the contract.
>

> Nested message pumps cause reentrancy.  WPF is designed to continue working even if an app causes reenentracy - such as your app is causing by calling into DShow.  This is important, because we need layout, rendering, animations, etc, etc to continue working.  We do offer a mode, as you have noted, where we will throw an exception if there is any reentrancy (this is a defensive mode for code paths that cannot survive reentrancy).  But we do not offer a mode where we simply suspend the dispatcher during reentrancy.
>
> Now, of course, you can fix for your scenario in several ways.  For example, you could keep your own list of async operations.  Then you could post one work item to do the next thing in your list.  You could even ensure that you won't do anything on your list until you have finished the previous item (refuse reentrancy).  Of course, I'm not sure how much value the WPF dispatcher is then providing for you...
>
> *Dwayne Need<http://blogs.msdn.com/dwayneneed> is a Dev Manager in WPF. He is not as handsome as dr. wpf, but he knows his stuff equally well and he owns the dispatcher.

>
> From: wpf-di...@googlegroups.com [mailto:wpf-di...@googlegroups.com] On Behalf Of Jeremiah Morrill
> Sent: Wednesday, September 23, 2009 5:40 PM
> To: wpf-di...@googlegroups.com
> Subject: [WPF Disciples] Re: Dispatcher and Win32 message pump blues
>
> Bing didn't have much to say about this issue...but it did find a nice restaurant I would like to take my g/f to.
>
> -Jer

> On Wed, Sep 23, 2009 at 5:35 PM, David Anson <david...@microsoft.com<mailto:david...@microsoft.com>> wrote:
>
> Bing and decide.
>

> From: wpf-di...@googlegroups.com<mailto:wpf-di...@googlegroups.com> [mailto:wpf-di...@googlegroups.com<mailto:wpf-di...@googlegroups.com>] On Behalf Of Jeremiah Morrill
> Sent: Wednesday, September 23, 2009 5:14 PM
> To: wpf-di...@googlegroups.com<mailto:wpf-di...@googlegroups.com>
> Subject: [WPF Disciples] Re: Dispatcher and Win32 message pump blues
>
> Funny, Google told me to ask you guys!
>

> On Wed, Sep 23, 2009 at 5:02 PM, Josh Smith <flappleja...@gmail.com<mailto:flappleja...@gmail.com>> wrote:
>
> Have ya tried googling it yet?  (joking)
>

> On Wed, Sep 23, 2009 at 4:26 PM, Jeremiah Morrill <jeremiah.morr...@gmail.com<mailto:jeremiah.morr...@gmail.com>> wrote:
>
> In my WPF MediaKit, I create a new thread/Dispatcher to run all my COM components.  I do this primarily because:
>

>   1.  WPF runs STA, and sometimes I want to run MTA.
>   2.  There are some blocking calls on the DirectShow that would block the UI thread.
>   3.  I sometimes need a Win32 message pump.  WPF's Dispatcher provides this.
>   4.  It's very convenient.

[Corrado Cavalli]

Great explanation!

 

Thanks Jaime

 

From: wpf-di...@googlegroups.com [mailto:wpf-di...@googlegroups.com] On Behalf Of Jaime Rodriguez
Sent: giovedì 24 settembre 2009 07:47
To: wpf-di...@googlegroups.com
Subject: [WPF Disciples] Re: Dispatcher and Win32 message pump blues

 

Hey Jer,

 

I Dwayned* it in-house and this is what I got back..

The WPF dispatcher is not just a queue of delegates.  It is deeply integrated with the Win32 message pump design.  For instance, delegates queued at “foreground” priorities are interleaved with other posted win32 messages.  Further, we guarantee that delegates queued at “background” priorities will only be dispatched after all other posted/sent/input Win32 messages are dispatched.  We don’t provide any way to cheat.  Because cheating would break the contract for other consumers of the Dispatcher API that depend on that contract.

 

The DShow APIs you are calling are evidently running their own nested message pumps.  This means that they call GetMessage() and DispatchMessage().  If they dispatch our ProcessQueue message, we will process the next delegate in the queue in priority order.  If we didn’t we would be breaking the contract.  Imagine code that posts a work item at Normal priority.   This item, by definition, must be executed before input is processed.  But the nested message pump will very likely pump input messages.  So if we don’t respond when it is our turn, then we break the contract.

 

Nested message pumps cause reentrancy.  WPF is designed to continue working even if an app causes reenentracy – such as your app is causing by calling into DShow.  This is important, because we need layout, rendering, animations, etc, etc to continue working.  We do offer a mode, as you have noted, where we will throw an exception if there is any reentrancy (this is a defensive mode for code paths that cannot survive reentrancy).  But we do not offer a mode where we simply suspend the dispatcher during reentrancy.

 

Now, of course, you can fix for your scenario in several ways.  For example, you could keep your own list of async operations.  Then you could post one work item to do the next thing in your list.  You could even ensure that you won’t do anything on your list until you have finished the previous item (refuse reentrancy).  Of course, I’m not sure how much value the WPF dispatcher is then providing for you…

 

 

*Dwayne Need is a Dev Manager in WPF. He is not as handsome as dr. wpf, but he knows his stuff equally well and he owns the dispatcher.  

[Jeremiah Morrill]

I just committed my message pump here...

So if anyone ever needs a simple, async-only, non-interleaved win32/clr-delegate message pump...you know where to find it.

-Jer

[Peter O'Hanlon]

Good job Jer.

--
Peter O'Hanlon