Electrolysis update: out-of-process plugins and more

14 views
Skip to first unread message

Chris Jones

unread,
Jun 15, 2009, 2:05:22 PM6/15/09
to dev-te...@lists.mozilla.org, dev-pl...@lists.mozilla.org, Benjamin Smedberg
Ben Turner and I have been working on running plugins in separate
processes (one per plugin shared object), as a first step for the
Electrolysis project. This work is ongoing in an experimental m-c
clone, http://hg.mozilla.org/users/bturner_mozilla.com/libchromiumipc/,
but will likely move elsewhere soon. We'll update as this takes shape.
Also, we will be using mozilla.dev.tech.dom to discuss Electrolysis
issues, at least for the near future. And as a quick reminder, we are
documenting the high level plans and status of Electrolysis here
(https://wiki.mozilla.org/Content_Processes).

As of late Wednesday the 10th, plugins were at the following state: on
GTK2/GNU/Linux systems, we can load a single instance of a single plugin
in windowed mode out of process. We haven't yet implemented the
NPRuntime (scripting wrapper for plugins), but theoretically
non-scripted plugins should be able to load and draw to the screen.
Unfortunately we haven't observed this yet, because both flash and our
"test" plugin want the NPRuntime :S. But most importantly, our IPC
architecture is starting to come together, and adding NPRuntime support
should not be a massive burden. We are ready to start implementing the
IPDL ("I piddle"), inter-*-communication protocol definition language,
compiler described somewhat here
(https://wiki.mozilla.org/IPC_Protocols), although this page is
outdated, inaccurate, and incomplete. This language should hopefully
make writing IPC code more productive and less error prone.

We have a complete short-term story for plugin graphics on X11.
Windowed plugins are very easy to support, as X windows are identified
with cross-process XID's; we need only to create the window in the Gecko
process and ship the ID across to the plugin. Windowless plugins are
more difficult to support, but we can reuse som work done for Hildon;
essentially, we create a window for the windowless plugins and create a
surface within that window for the plugin to draw to, but use Xdamage
and Xcomposite to composite the Gecko window with the plugin's window.
(Graphics folks, if this is a bad summary of this technique, please
correct me.) Ben Turner will likely be working on plugin graphics for
Windows, but we're not yet sure of our approach to plugin graphics for OS X.

In other Electrolysis news, our graphics team has a solid plan of attack
for multi-process Gecko rendering, both for the short and long terms.
They will probably document this plan publicly, soon. Ben Turner is
also working on creating our "content" process, in which web pages will
be rendered (as opposed to the "chrome" process in which the Firefox UI
will reside). We expect to have an initial prototype of this up and
running "soon."

Cheers,
Chris

Jeff Muizelaar

unread,
Jun 15, 2009, 2:23:17 PM6/15/09
to Chris Jones, dev-pl...@lists.mozilla.org, dev-te...@lists.mozilla.org, Benjamin Smedberg

On 15-Jun-09, at 2:05 PM, Chris Jones wrote:
>
> We have a complete short-term story for plugin graphics on X11.
> Windowed plugins are very easy to support, as X windows are
> identified with cross-process XID's; we need only to create the
> window in the Gecko process and ship the ID across to the plugin.
> Windowless plugins are more difficult to support, but we can reuse
> som work done for Hildon; essentially, we create a window for the
> windowless plugins and create a surface within that window for the
> plugin to draw to, but use Xdamage and Xcomposite to composite the
> Gecko window with the plugin's window. (Graphics folks, if this is a
> bad summary of this technique, please correct me.) Ben Turner will
> likely be working on plugin graphics for Windows, but we're not yet
> sure of our approach to plugin graphics for OS X.

Shouldn't windowless plugins be easier to support then windowed ones?
Is there any reason you can't just create a surface in a region of
memory shared between the plugin process and the Gecko process? It's
not clear to me what problems are solved by using Xdamage and
Xcomposite.

This shared memory approach should also work fine on OS X.

-Jeff

Jeff Muizelaar

unread,
Jun 15, 2009, 3:58:53 PM6/15/09
to Chris Jones, dev-pl...@lists.mozilla.org, dev-te...@lists.mozilla.org, Benjamin Smedberg
Adding back dev-platform cc as it seems like a better list then dev-
tech-dom.

On 15-Jun-09, at 3:17 PM, Chris Jones wrote:

> It's just disgustingly easy to ship XID's from the content to plugin
> process. Also, all of the Xdamage/Xcomposite code is written and
> tested. Again, this is the short-term (i.e. something shippable as
> soon as possible) and not long-term (i.e. something we're entirely
> happy with) plan.

If the Xdamage/Xcomposite code you're talking about is the Fennec/
Hildon code, you won't be able to use that code out of the box because
it's designed for making all windowed plugins act like windowless
plugins and does scary and limited event forwarding. I've also only
ever tested it with flash and a test plugin.

Currently we have code paths for:
- Windowed plugins
- Windowless plugins
and an optional one (#ifdef'd to replace the windowed plugins path) for:
- Windowed plugins that act like windowless plugins
with this approach we'd add a another path for:
- Windowless plugins that act like windowed plugins that act like
windowless plugins
and that scares me a little bit.

My guess is that modifying Xdamage/Xcomposite code to fit your needs
won't be any easier then just using a shared memory surface. Further,
if we're not going to use that approach in the end it seems better to
just do it right the first time. However, if you've already got
windowless plugins working using Xdamage/Xcomposite, I guess we can
always move to a saner approach later on.

-Jeff

Jeff Muizelaar

unread,
Jun 16, 2009, 11:06:01 AM6/16/09
to Chris Jones, dev-te...@lists.mozilla.org, dev-pl...@lists.mozilla.org, dev-te...@lists.mozilla.org, Benjamin Smedberg

On 15-Jun-09, at 2:05 PM, Chris Jones wrote:
> We have a complete short-term story for plugin graphics on X11.
> Windowed plugins are very easy to support, as X windows are
> identified with cross-process XID's; we need only to create the
> window in the Gecko process and ship the ID across to the plugin.
> Windowless plugins are more difficult to support, but we can reuse
> som work done for Hildon; essentially, we create a window for the
> windowless plugins and create a surface within that window for the
> plugin to draw to, but use Xdamage and Xcomposite to composite the
> Gecko window with the plugin's window. (Graphics folks, if this is a
> bad summary of this technique, please correct me.) Ben Turner will
> likely be working on plugin graphics for Windows, but we're not yet
> sure of our approach to plugin graphics for OS X.

For the record, here's my rough understanding of what WebKit does for
out of process plugins on Snow Leopard.

The implementation is in WebKit/mac/Plugins/Hosted. Unfortunately, the
WebKit tree does not seem to include the client code???

Overview and background
- All of the IPC is done using mach ports
- They have a client (presumably the plugin process) and a host
- CoreAnimation (CA) is a new UI API introduced in Leopard and on the
iPhone.
- A CALayer is a high level abstraction of a backing store. The
backing store is usually in video ram and CALayers are usually drawn
using a textured quads
- CARenderer is an object responsible for drawing a hierarchy of layers.

On host initialization
- The host manager (NetscapePluginHostManager.mm) creates a render
server using WKInitializeRenderServer() which returns a renderServerPort
- This renderServerPort is passed into _WKPHCheckInWithPluginHost(),
however I'm not sure what that does or where it's implemented.

On plugin initialization (WebHostedNetscapePluginView.mm:createPlugin)
- instantiatePlugin() is called which returns a renderContextID and a
useSoftwareRenderer flag
- the renderContextID is presumably associated with the RenderServer
created earlier

- if (useSoftwareRenderer) then
WKSoftwareCARendererCreate(renderContextID)
- the softwareRenderer draws in drawRect() using
WKSoftwareCARendererRender()
- I'm not sure what the advantage of using CoreAnimation here is,
because I assume the client process wouldn't have more then one layer...

- otherwise we call WKMakeRenderLayer(renderContextID)
- WKMakeRenderLayer presumably creates one side of a CALayer that
is shared between the host and the client
- This shared layer is added to the existing layer hierarchy in the
browser and presumably drawn as though it was a regular layer

- Printing is done using either the softwareRenderer or shuttling a
bitmap between the plugin and the host.

Overall this seems like a good way to solve the cross-process hardware
acceleration problem. The shared CALayer is simple and flexible
(though it's unclear how they deal with synchronization).
Unfortunately, this method is implemented using private webkit symbols
(WKInitializeRenderServer, WKMakeRenderLayer) so it is not easy for us
to use unless Apple publicly exports the interface that these
functions use. I think it would be worth while to put pressure on
Apple to do so.

-Jeff

Mike Shaver

unread,
Jun 16, 2009, 11:11:10 AM6/16/09
to Jeff Muizelaar, Benjamin Smedberg, dev-te...@lists.mozilla.org, dev-pl...@lists.mozilla.org, dev-te...@lists.mozilla.org, Chris Jones
On Tue, Jun 16, 2009 at 11:06 AM, Jeff Muizelaar<jmuiz...@mozilla.com> wrote:
> Unfortunately, this method is
> implemented using private webkit symbols (WKInitializeRenderServer,
> WKMakeRenderLayer) so it is not easy for us to use unless Apple publicly
> exports the interface that these functions use. I think it would be worth
> while to put pressure on Apple to do so.

Yes, let's get radar on file, and we can ask some of our Apple
contacts for help getting those things exposed.

Mike

Robert O'Callahan

unread,
Jun 16, 2009, 2:55:58 PM6/16/09
to Jeff Muizelaar, Chris Jones, Benjamin Smedberg
On 16/6/09 6:23 AM, Jeff Muizelaar wrote:
> On 15-Jun-09, at 2:05 PM, Chris Jones wrote:
>> We have a complete short-term story for plugin graphics on X11.
>> Windowed plugins are very easy to support, as X windows are identified
>> with cross-process XID's; we need only to create the window in the
>> Gecko process and ship the ID across to the plugin. Windowless plugins
>> are more difficult to support, but we can reuse som work done for
>> Hildon; essentially, we create a window for the windowless plugins and
>> create a surface within that window for the plugin to draw to, but use
>> Xdamage and Xcomposite to composite the Gecko window with the plugin's
>> window. (Graphics folks, if this is a bad summary of this technique,
>> please correct me.) Ben Turner will likely be working on plugin
>> graphics for Windows, but we're not yet sure of our approach to plugin
>> graphics for OS X.
>
> Shouldn't windowless plugins be easier to support then windowed ones? Is
> there any reason you can't just create a surface in a region of memory
> shared between the plugin process and the Gecko process?

Yes, there is a reason. Windowless plugins require an X drawable to be
passed into them.

Rob

Jeff Muizelaar

unread,
Jun 16, 2009, 6:49:38 PM6/16/09
to Robert O'Callahan, Chris Jones, dev-pl...@lists.mozilla.org, Benjamin Smedberg

On 16-Jun-09, at 2:55 PM, Robert O'Callahan <rob...@ocallahan.org>
wrote:

Oh. Well in that case things should be even easier. Just ship a
drawable id to the plugin process and have it draw to that just like
you would a window id.

-Jeff

Robert O'Callahan

unread,
Jun 16, 2009, 7:51:49 PM6/16/09
to

Do you mean "whenever we paint our content window, ship the drawable ID
to the plugin process and have it synchronously draw to that"? That
would work, but it would lose some of the benefits of out-of-process
plugins; a slow or hung plugin would retard drawing of the window. The
overhead of the synchronous IPC would also impact painting.

Or do you mean "have the plugin process paint to an X pixmap
asynchronously, and copy that pixmap whenever we paint our content
window"? That would work, but for transparent plugins you actually need
two pixmaps, one with a white background and one with a black
background, so you can compute the plugin's alpha values.

We discussed this at length on IRC a while back. I guess no-one took
notes, oops.

Rob

Chris Jones

unread,
Jun 16, 2009, 9:00:26 PM6/16/09
to Robert O'Callahan, dev-te...@lists.mozilla.org, dev-pl...@lists.mozilla.org, dev-te...@lists.mozilla.org

I saved the raw IRC conversation, but didn't extract clean notes from
it. Here's the raw conversation:

http://pastebin.mozilla.org/657908

These notes will probably materialize near
http://wiki.mozilla.org/Content_Processes, but they're not my top
priority at the moment.

Cheers,
Chris

Reply all
Reply to author
Forward
0 new messages