Windowless plugin rendering issues
Robert O'Callahan
Pre-Firefox 4, we rendered windowless plugins by asking them to paint during
our normal paint phase. In most cases we could pass in our "current drawing
context" (HDC or X surface) and get the plugin to draw directly into it.
This worked reasonably well, but this approach was abandoned in Firefox 4
for two good reasons. First, asking the plugin to paint during our paint
phase is dangerous because it leads to security and stability issues I don't
want to describe in a public forum. Second, when we're doing accelerated
rendering we just don't have an HDC or X surface that represents "the window
contents". For example the content under the plugin might only be available
in D3D surfaces.
So in Firefox 4 Oleg, Benjamin and others implemented asynchronous plugin
rendering. The idea is that we periodically render the plugin to a pixmap
asynchronously, not during our normal paint phase. Whenever we want to paint
the plugin into our window, we just paint the latest pixmap we got from the
plugin. This is easy to do except when the plugin instance is partially
transparent. In that case we need to get a bitmap with alpha transparency
values. But standard NPAPI plugin drawing calls --- which use HDCs and X
surfaces --- don't fill the bitmap with alpha values. But we did some
investigation and found that, at least on Windows, if we drew Flash to a
32-bits-per-pixel bitmap, it would fill in correct alpha values. This gave
us a performant solution for asynchronous rendering of transparent Flash
instances ... or so we thought.
The Problem:
More testing in the wild shows that in some cases Flash doesn't give us
correct alpha values. Instead it sets alpha values to 1, which leads to
black regions incorrectly showing up in the Flash object. We were able to
figure out and hack around some of the situations that trigger this, but in
other situations we don't know what's going on. Bug 626206 is tracking these
unresolved issues.
There is a side issue that our asynchronous rendering approach forces all
plugin rendering through an RGBA bitmap, which means the plugin can't do
subpixel antialiasing correctly, so there's a reduction in text quality.
This doesn't worry me as much as the black-background bugs though.
Possible Solutions:
1) Ship with bug 626206 unfixed. I don't think this is acceptable.
2) Turn off asynchronous plugin rendering, and graphics acceleration on
Windows. I don't think this is acceptable either.
3) Use "alpha recovery" to extract alpha values from the plugin instance, by
asking it to paint twice per frame, once onto a black background and once
onto a white background, and computing alpha values from the difference. We
already do this in some cases (and also did pre-Firefox-4 in some cases), so
this is low risk, but there's a significant performance penalty, especially
with X. We could try to find a way to trigger alpha recovery only when bug
626206 is happening, but I don't think that will be possible. I've asked
Adobe if they can help with that, but I suspect they won't.
4) Introduce a new approach to asynchronous windowless plugin rendering,
more like what Chrome does: capture the "background" of the plugin area
while we paint, send that background asynchronously over to the plugin
process, ask the plugin to draw over a copy of that background, then draw
the opaque result in place of the plugin. This should work well when the
content behind the plugin is static, which it usually is. It can be made to
perform well if we do a good job of detecting when the background has
changed. In some cases we'll need to do asynchronous readback of data from
GPU memory, but this can probably be done efficiently.
5) Do a mix of solutions 3 and 4: Implement solution 4 in a way that works
for most plugin instances, and fall back to solution 3 when things get
tricky.
Proposal:
I propose doing option 5 for Firefox 4. It's some new code, and I
desperately wish we didn't have to do it, but to me it seems like the best
of a bad set of options. A sketch of the design is in bug 626602.
All feedback appreciated.
Rob
--
"Now the Bereans were of more noble character than the Thessalonians, for
they received the message with great eagerness and examined the Scriptures
every day to see if what Paul said was true." [Acts 17:11]
Jeff Muizelaar
On 18-Jan-11, at 4:56 PM, Robert O'Callahan wrote:
> Proposal:
>
> I propose doing option 5 for Firefox 4. It's some new code, and I
> desperately wish we didn't have to do it, but to me it seems like
> the best
> of a bad set of options. A sketch of the design is in bug 626602.
Option 5 sounds good to me.
-Jeff
Neil
>2) Turn off asynchronous plugin rendering, and graphics acceleration on Windows.
>
What about the case where graphics acceleration was never on in the
first place?
--
Warning: May contain traces of nuts.
Robert O'Callahan
> Robert O'Callahan wrote:
>
> 2) Turn off asynchronous plugin rendering, and graphics acceleration on
>> Windows.
>>
>> What about the case where graphics acceleration was never on in the first
> place?
>
Then we don't need to turn it off.
Neil
>On Wed, Jan 19, 2011 at 10:52 PM, Neil <ne...@parkwaycc.co.uk> wrote:
>
>>Robert O'Callahan wrote:
>>
>>
>>>2) Turn off asynchronous plugin rendering, and graphics acceleration on Windows.
>>>
>>>
>>What about the case where graphics acceleration was never on in the first place?
>>
>Then we don't need to turn it off.
>
>
Would you still need to turn off asynchronous plugin rendering to get
the "benefit" of not having graphics acceleration?
Neil
>asking the plugin to paint during our paint phase is dangerous
>
>
Would content processes make the problem go away?
Robert O'Callahan
> Robert O'Callahan wrote:
>
> asking the plugin to paint during our paint phase is dangerous
>>
>>
> Would content processes make the problem go away?
No.
Robert O'Callahan
> Would you still need to turn off asynchronous plugin rendering to get the
> "benefit" of not having graphics acceleration?
We'd need to turn off asynchronous plugin rendering to make plugins work the
way they used to pre-Firefox 4.
Henri Sivonen
> investigation and found that, at least on Windows, if we drew Flash to
> a
> 32-bits-per-pixel bitmap, it would fill in correct alpha values. This
> gave
> us a performant solution for asynchronous rendering of transparent
> Flash
> instances ... or so we thought.
> 1) Ship with bug 626206 unfixed. I don't think this is acceptable.
> I've asked Adobe if they can help with that, but I suspect they won't.
> I propose doing option 5 for Firefox 4. It's some new code, and I
> desperately wish we didn't have to do it, but to me it seems like the
> best of a bad set of options.
Is there a reason to believe that if Firefox shipped as-is, Adobe wouldn't change Flash Player to draw consistently with an alpha channel reasonably soon (considering that evidently there's already some capability to draw with alpha in Flash Player)? Is there some history that leads to a conclusion that browsers need to accommodate plug-ins rather than the other way round? (I observe that 17% of the remaining hard blockers are in the Plug-ins Bugzilla component.)
(I'm *not* trying to suggest a different conclusion. I'm just genuinely curious.)
--
Henri Sivonen
hsiv...@iki.fi
http://hsivonen.iki.fi/
Mike Beltzner
> Is there a reason to believe that if Firefox shipped as-is, Adobe
> wouldn't change Flash Player to draw consistently with an alpha
> channel reasonably soon (considering that evidently there's already
> some capability to draw with alpha in Flash Player)? Is there some
We have contacts on that team, and we can ask them. However, we also
know that they're at the RC phase of Flash 10.2, and so our cycles are
not well aligned to service our users.
I'm pretty sure that they'd be open to changing, but I'm also quite sure
that their willingness would be completely separate from the issue at hand.
> history that leads to a conclusion that browsers need to accommodate
> plug-ins rather than the other way round? (I observe that 17% of the
> remaining hard blockers are in the Plug-ins Bugzilla component.)
>
> (I'm *not* trying to suggest a different conclusion. I'm just
> genuinely curious.)
I agree that it's concerning that so many of our blockers seem to be
caught up in problems co-ordinating with plugins, both at the event
model and the development team levels.
cheers,
mike
Robert O'Callahan
a choice.
One good thing is that this work mostly moves us in the direction of greater
isolation between plugins and the browser.
Justin Dolske
> 3) Use "alpha recovery" to extract alpha values from the plugin instance, by
> asking it to paint twice per frame, once onto a black background and once
> onto a white background, and computing alpha values from the difference. We
> already do this in some cases (and also did pre-Firefox-4 in some cases), so
> this is low risk, but there's a significant performance penalty, especially
> with X.
I sure like "already do this" and "low risk", especially at this point
in the cycle. How big is the performance penalty? It would only be on
pages using windowless plugins, right?
Could we special-case plugins on a flat background (<div
bgcolor=red><plugin></div>), or is that effectively as much work as #4/5
or otherwise not worth it?
> I propose doing option 5 for Firefox 4. It's some new code, and I
> desperately wish we didn't have to do it, but to me it seems like the best
> of a bad set of options. A sketch of the design is in bug 626602.
How much risk (w.r.t. both code and time) is this? From your description
it _sounds_ like the basic scheme is straight-forward to implement (and
is exactly what Chrome already does), but that detecting background
changes might be tricky?
Justin
Robert O'Callahan
> On 1/18/11 1:56 PM, Robert O'Callahan wrote:
>
> 3) Use "alpha recovery" to extract alpha values from the plugin instance,
>> by
>> asking it to paint twice per frame, once onto a black background and once
>> onto a white background, and computing alpha values from the difference.
>> We
>> already do this in some cases (and also did pre-Firefox-4 in some cases),
>> so
>> this is low risk, but there's a significant performance penalty,
>> especially
>> with X.
>>
>
> I sure like "already do this" and "low risk", especially at this point in
> the cycle. How big is the performance penalty?
"It depends". It's super-bad on X, less bad (but variable) on Windows.
> It would only be on pages using windowless plugins, right?
>
Yes, but that's a lot of important pages.
Could we special-case plugins on a flat background (<div
> bgcolor=red><plugin></div>), or is that effectively as much work as #4/5 or
> otherwise not worth it?
>
That's heading towards #5.
> I propose doing option 5 for Firefox 4. It's some new code, and I
>> desperately wish we didn't have to do it, but to me it seems like the best
>> of a bad set of options. A sketch of the design is in bug 626602.
>>
>
> How much risk (w.r.t. both code and time) is this? From your description it
> _sounds_ like the basic scheme is straight-forward to implement (and is
> exactly what Chrome already does), but that detecting background changes
> might be tricky?
>
I don't know yet.