Silverlight rendered in WPF with no-airspace and interactivity!

[Jeremiah Morrill]

I have been spending some of my free time working on a fun library.  My goal is to render silverlight to WPF, performantly and with interactivity.  I came across a little known native COM api for silverlight.  Not many people have used it, but it allows for hosting silverlight like any activex.  Any activex wasn't good enough because airspace in WPF is weak-sauce.  Then I remembered silverlight does support "Windowless" (IOleInPlaceObjectWindowless) rendering .  Windowless/IOleInPlaceObjectWindowless lets you take over rendering of an hwnd.  After a crap ton of crazy ass Win32/COM stuffs, I finally have it working...the prototype at least.  Interactivity isn't right just yet, but will work just as if it was any control in WPF.

Now we can do DeepZoom in WPF :).

Here is a video

http://silverlight.services.live.com/invoke/36456/Silverlight%20in%20WPF/iframe.html

I still have to do a major overhaul to get this thing ready for release...but I'm excited to see it work...and excited to see it doesnt need 1000% CPU usage.

[Jeremiah Morrill]

BTW, this is the silverlight app I'm showing.

http://www.lab101.be/wp-content/Silverlight/Plasma/index.html

All i need to do is supply the url to the xap.  Also, I should be able to do communication between the WPF and SL apps too.

[Paul Stovell]

Hi Jeremiah,

That is very cool.

Was the the bulk of the work to do with making IOleInPlaceObjectWindowless work with WPF, or with Silverlight specific work? I'm wondering if this technique will work for other components that support this interface?

Paul

--
Paul Stovell

[Mike Brown]

Better question does the IE Browser control support that? You are amazing!

[Jeremiah Morrill]

The bulk of the work was really learning how to use these COM interfaces.  I wasn't sure what I had to implement and what could return "not implemented".  The other hard part was translating all these interfaces to C#.  I chose to do this pure c#/com interop because I found that most .NET developers won't touch it or I spend a lot of time teaching people how to compile C++.

Anyways, at the end of the day, the code is pretty simple.  Implement some of these COM interfaces on a Winforms user control, create your activex.  On your WndProc, send windows messages to your ActiveX via IOleInPlaceObjectWindowless (for interactivity).  You need to handle your WM_PAINT (or OnPaint) and in there you use QI the IViewObject from your activex and tell it to draw to your offscreen bitmap.  Then I throw an event for each on paint and pass that to WPF, where I use a WriteableBitmap (maybe i should use something faster) to copy the offscreen bitmap to the WriteableBitmap backbuffer.

Any activex that supports IViewObject should be able to render to an offscreen bitmap, but IOleInPlaceObjectWindowless is key for interactivity.  I think IE supports at least IViewObject

[Cory Plotts]

Very nice! Wow. I like it when something doesn't chew up 1000% either. :)

[Shawn Wildermuth]

The thing with Windowless mode isn't that it takes more CPU, its that is performs more poorly with lots of animations.

[Jaime Rodriguez]

This is very cool Jeremiah..  can't wait to  use it..

[Jeremiah Morrill]

Shawn,

I'm just speculating from what I understand as of yet...Hw acceleration (BitmapCache) in SL aside, I believe windowless mode performs worse in a browser because the browser must compose both the rendered browser html/document and the rendered silverlight screen together.  I believe this is why with windowless, you can have html content on top of silverlight content.  In the end, SL, windowless or not, still renders with GDI, the only difference here would be the hDC it renders to.  That's why with the hosted solution I've hacked together the overhead of composition with the browser is non-existent...but I won't pretend there's no overhead with WPF composition ;).  WriteableBitmap is a turd cake.  I suspect InteropBitmap will be even better, but and if I can't get the dirty area region, it might be the way to go.

-Jer

[Jeremiah Morrill]

Thanks Jaime,

It's been a lotta fun.  I need to do a massive clean up.  Right now I use a Winforms control and a WindowsFormsHost in order to get my message loop running.  That needs to be changed to run it's own message loop to drop the WindowsFormsHost requirement.  Then I need to make some API to take in pseudo windows messages. So, for instance, I can take WPF mouse-move events and pump them into the Silverlight Active-X.

What's kind of crazy cool, is this COM API lets you (makes you?) create your own mechanisms for what the browser handles normally, like downloading files.  I think you maybe be able to make your own "SL -> JS DOM" bridge.  Of course there'd be no JS and no DOM with no browser, but I think here, a WPF -> SL bridge could be constructed.

I'll send you guys a link when it's more there, hopefully this week.

-Jer

[Laurent Bugnion]

Hey Jer,

Back then at Siemens we were starting to talk about using some SL within WPF and one path we were thinking of exploring was to render Silverlight withing the WPF web browser, and to use the JavaScript bridge to communicate. Unfortunately i never had time to try this and then i left Siemens.

Would you say that what you did here is similar to that, or did I miss something? One advantage i can think of for your solution is that it should perform better, right? How about the Windowless area, is it square or can it have any shape? I was out yesterday but will check your mails on a real computer soon.

Thanks,
Laurent
--
Sent from mobile

-original message-
Subject: Re: [WPF Disciples] Re: Silverlight rendered in WPF with no-airspace and interactivity!

From: Jeremiah Morrill <jeremiah...@gmail.com>
Date: 15.12.2009 07:21

Shawn,

I'm just speculating from what I understand as of yet...Hw acceleration
(BitmapCache) in SL aside, I believe windowless mode performs worse in a
browser because the browser must compose both the rendered browser
html/document and the rendered silverlight screen together. I believe this
is why with windowless, you can have html content on top of silverlight
content. In the end, SL, windowless or not, still renders with GDI, the
only difference here would be the hDC it renders to. That's why with the
hosted solution I've hacked together the overhead of composition with the
browser is non-existent...but I won't pretend there's no overhead with WPF
composition ;). WriteableBitmap is a turd cake. I suspect InteropBitmap
will be even better, but and if I can't get the dirty area region, it might
be the way to go.

-Jer

On Mon, Dec 14, 2009 at 8:04 PM, Shawn Wildermuth <
shawn...@agilitrain.com> wrote:

> The thing with Windowless mode isn't that it takes more CPU, its that is
> performs more poorly with lots of animations.
>
>
>

> *From:* wpf-di...@googlegroups.com [mailto:
> wpf-di...@googlegroups.com] *On Behalf Of *Cory Plotts
> *Sent:* Monday, December 14, 2009 10:55 PM
> *To:* wpf-di...@googlegroups.com
> *Subject:* Re: [WPF Disciples] Re: Silverlight rendered in WPF with

[Jeremiah Morrill]

This solution does not use a webbrowser at all and does not render using any win32 interop in WPF.  It uses WriteableBitmap ATM.  Currently with a web browser hosted in WPF, you will not be able to put any transforms, opacity, or even use it on 3D.  With this, you can (like in the video).  I can't say this solution will perform better than with native GDI, but I will say it's very little overhead for a big reward.

I have only experimented with RGB24 bitmaps thus far.  But if SL supports RGBA (which some IViewObject's do), it won't be restricted to be a rectangular area....but even not, no airspace restrictions is still pretty cool.  This also is nothing really new.  The technology has been around since 96 and many people have used the same features to render flash to open gl or what not...I'm just discovering it now, thanks to stackoverflow and chatting with people like Evan.

Also, it's worthy to mention in the SL native API, there are some methods to render any xaml/visual to an RGBA bitmap.    You can essentially use SL to be just a rendering engine for anything.  Don't know what to use it for...but it's sick that its there!

-Jer

[Peter O'Hanlon]

This is crazy insane stuff Jer. Tres cool.

--
Peter O'Hanlon

[Laurent Bugnion]

Yeah I got that you didn’t use a WebBrowser at all, don’t get me wrong, I think it is pretty insanely cool! I was just trying to sort out the various options we have to do that.

 

Sometimes I think if we didn’t have you and Evan we would have to invent you guys. I know this kind of things is totally over my head J

 

Cheers,

Laurent

[Daniel Vaughan]

Nice one Jer!
DeepZoom in WPF eh...wicked.


On Dec 15, 3:58 am, Jeremiah Morrill <jeremiah.morr...@gmail.com>
wrote:

> I have been spending some of my free time working on a fun library.  My goal
> is to render silverlight to WPF, performantly and with interactivity.  I
> came across a little known native COM api for silverlight.  Not many people
> have used it, but it allows for hosting silverlight like any activex.  Any
> activex wasn't good enough because airspace in WPF is weak-sauce.  Then I
> remembered silverlight does support "Windowless"
> (IOleInPlaceObjectWindowless) rendering .
>  Windowless/IOleInPlaceObjectWindowless lets you take over rendering of an
> hwnd.  After a crap ton of crazy ass Win32/COM stuffs, I finally have it
> working...the prototype at least.  Interactivity isn't right just yet, but
> will work just as if it was any control in WPF.
>
> Now we can do DeepZoom in WPF :).
>

> Here is a videohttp://silverlight.services.live.com/invoke/36456/Silverlight%20in%20...

[Sacha Barber]

Yeah I want DeepZoom in WPF. Let us know man.

 

As for Transforms on WebBrowser in WPF and 3D support, you could try this approach:

 

http://sachabarber.net/?p=609

--
Sacha Barber
sacha....@gmail.com

[Jeremiah Morrill]

Laurent,

If it wasn't for the likes of you and everyone else...well my SL/WPF would still blow up in blend and "separation of concerns" would mean "Separate your eyes from reviewing my code so you won't be so concerned about it!"

WPF Disciples is great because we got a mixed bag.  It's like the movie Breakfast Club.  We have a "nerd, a jock, a prom queen, a delinquent, and a loner."  Now we just need to queue up some cornball 80's music..."Don't you....forget about meee"

[Josh Smith]

We also have a KDAWG.

[Bill Kempf]

Who'd you call a prom queen?

On Tue, Dec 15, 2009 at 2:40 PM, Jeremiah Morrill

--
Quidquid latine dictum sit, altum sonatur.
- Whatever is said in Latin sounds profound.

War is peace. Freedom is slavery. Bugs are features.

[Josh Smith]

Bea?

[Peter O'Hanlon]

Bill's a porn queen? Sorry, I misread that sentence.

--
Peter O'Hanlon