NPImageExpose
Doug Turner
flash plugin running on the N900 device. It allows them to quickly
draw directly to a shared image buffer. This improves drawing
performance on in Fennec as we can avoid X.
Here is a sketch of the proposal. I invite you to consider and
respond:
We are proposing a NPAPI extension which would allow a plugin to
render directly into 32-bit buffer in shared memory. The new extension
will be designed so that plugin explicitly makes requests of the user
agent whether the extension is supported (with NPN_GetValue and
NPNVSupportsWindowlessLocal) and then enables the new rendering mode
at runtime (with NPN_SetValue and NPPVpluginWindowlessLocalBool). In
this way other plugins unaware of new extension will not be disrupted.
When new rendering mode is enabled the Xdrawable field of a standard
XGraphicsExpose event holds a pointer to the new NPImageExpose
structure which among other things contains pointer to the shared
memory buffer, its dimensions, translation and scale factors.
typedef struct _NPImageExpose
{
char* data; /* image pointer */
int32 stride; /* Stride of data image buffer */
int32 depth; /* Depth of image pointer */
int32 x; /* Expose x */
int32 y; /* Expose y */
uint32 width; /* Expose width */
uint32 height; /* Expose height */
NPSize dataSize; /* Data buffer size */
float translateX; /* translate X matrix value */
float translateY; /* translate Y matrix value */
float scaleX; /* scale X matrix value */
float scaleY; /* scale Y matrix value */
} NPImageExpose;
Doug Turner
Boris Zbarsky
> We are proposing a NPAPI extension which would allow a plugin to
> render directly into 32-bit buffer in shared memory.
Can this shared memory be shared between processes?
-Boris
Oleg Romashin
Yes, this is the main idea of N900 browser.
See: http://docs.google.com/present/view?id=ajk2s2df9zhr_144fb8zrrck
Slide 6.
Oleg Romashin
It depends what engine and plugins supports.... it can be any buffer
format
> int32 x; /* Expose x */
> int32 y; /* Expose y */
> uint32 width; /* Expose width */
> uint32 height; /* Expose height */
> Is this in terms of buffer coordinates or plugin coordinates?
As I remember it is the buffer coordinates.
> NPSize dataSize; /* Data buffer size */
> Why is this needed?
We are giving whole rendering buffer to plugin... not only part which
is related to buffer.
In this case plugin can render stuff once (with or without alpha), and
engine should not do anything more (copy, composite....)
Also there are one idea, about providing plugin own buffer to the
engine.
For example if plugin cannot paint in engine provided image format,
then plugin does rendering in own buffer, then replacing whole
NPImageExpose structure, or some fields values according to buffer
parameters provided by plugin... and then engine checking that plugin
did rendering into own buffer, and should try to composite plugin
buffer into engine buffer.
Martin Husemann
>> int32 depth; /* Depth of image pointer */
>> What is the format of the buffer? In particular are there alpha values?
>>
>
> It depends what engine and plugins supports.... it can be any buffer
> format
>
plugin force a special format, or does the browser force the most
efficiently blittable one and the plugin has to implement all possible
modes?
Martin
Benjamin Smedberg
> Here is a sketch of the proposal. I invite you to consider and
> respond:
Proposals should be specified at https://wiki.mozilla.org/Plugins
> When new rendering mode is enabled the Xdrawable field of a standard
> XGraphicsExpose event holds a pointer to the new NPImageExpose
> structure which among other things contains pointer to the shared
> memory buffer, its dimensions, translation and scale factors.
Does this only apply in the context of windowless plugin instances? Or would
all plugin instances be forced into windowless mode when this drawing mode
was enabled?
How does this interact with transparent (partially-drawn) plugins? Is the
plugin only allowed to draw to the shared-memory buffer at certain times, so
that it can integrate properly with z-order painting?
This proposal seems very similar, at least in theory, to the pepper
platform-independent-plugins proposal that Google has drawn up:
https://wiki.mozilla.org/Plugins:PlatformIndependentNPAPI#Plugins_and_rendering
One of the particular advantages of pepper (and the pepper-like drawing
model that Chrome uses for windowless plugins currently) is that you give
the plugin a drawable which is *not* the direct screen surface. This means
that the plugin host can composite that drawable with other DOM elements
(underlays and overlays) asynchronously, and you never have to synchronously
wait for the plugin to paint.
--BDS
Oleg Romashin
plug-in.
Also even when new drawing model is accepted, plug-in should be ready
to paint in old modes too. (XWindow context)
I'm not sure what is the best way to do it... but we should notify
plugin somehow what XGraphicsExposeEvent.drawable contain,
NPImageExpose or XDrawable pointer..
Currently we are setting XGraphicsExposeEvent.display = 0 to identify
plugin that this expose event is not X-related.
>
> How does this interact with transparent (partially-drawn) plugins? Is the
> plugin only allowed to draw to the shared-memory buffer at certain times, so
> that it can integrate properly with z-order painting?
z-ordering is the same as in normal windowless pipeline.. builder
DisplayList define when plugin should paint it's data.
>
> This proposal seems very similar, at least in theory, to the pepper
> platform-independent-plugins proposal that Google has drawn up:https://wiki.mozilla.org/Plugins:PlatformIndependentNPAPI#Plugins_and...
I'll check it.
Oleg Romashin
> This proposal seems very similar, at least in theory, to the pepper
do we have any bug in mozilla about this new API? patches for mozilla,
plugin-examples? something to try and check the list of possible
problems...
Doug Turner
In talking with a few people, there is a lot of improvement we can
make on this API -- maybe pepper is the right approach.
The API I was spec-ing out above exists and is in use on the Nokia
n900 devices. Assuming that we address issues raised above w/r/t
format of the image buffer, clipping, and what does the plugin
actually have to draw into the image buffer, I would like to propose
this API as a Maemo only extension to the NPAPI.
Thoughts? Objections?
Doug
Benjamin Smedberg
> The API I was spec-ing out above exists and is in use on the Nokia
> n900 devices. Assuming that we address issues raised above w/r/t
> format of the image buffer, clipping, and what does the plugin
> actually have to draw into the image buffer, I would like to propose
> this API as a Maemo only extension to the NPAPI.
>
> Thoughts? Objections?
The biggest issue I want to address is the painting model. Chrome experience
with OOP plugins shows that you really want to have an asynchronous painting
model such as the one provided by pepper.
Is there a reason pepper wouldn't give the performance needed by Flash,
instead of having yet another rendering model?
--BDS
Doug Turner
I am not suggesting that we do not do pepper for Maemo.
Right now, the n900 is just about done. I do not expect there to be
any significant changes in flash before we ship Fennec 1.0. We have
something that has been implemented by the platform (the n900), and we
can either take advantage of it, or not. I think we do want to take
advantage of it for speed, but advise other plugins not to use this
approach.
Doug
Benjamin Smedberg
> Right now, the n900 is just about done. I do not expect there to be
> any significant changes in flash before we ship Fennec 1.0. We have
> something that has been implemented by the platform (the n900), and we
> can either take advantage of it, or not. I think we do want to take
> advantage of it for speed, but advise other plugins not to use this
> approach.
Since we'll have OOP plugins (or OOP tabs, which have the same rendering
problem) in the 1.1 release, and OOP plugins cannot use the approach as it's
currently specced, it seems like we'd be digging ourself into a hole which
only partly-works and isn't very well designed.
--BDS
Doug Turner
This API clearly doesn't satifiy our requirements for OOP and it is
X11 specific. However, it is what it is and support for it is on the
n900 devices. If we want faster flash on the n900, we will want to
support this NPAPI feature -- even if it is just on maemo.
Doug
Doug Turner
> X11 specific. However, it is what it is and support for it is on the
> n900 devices. If we want faster flash on the n900, we will want to
> support this NPAPI feature -- even if it is just on maemo.
I am trying to be pragmatic, but I really dislike this API too. It is
a shame that we couldn't influence the design and build something that
would be more useful. Josh -- do you have any objection with
documenting this as what it is and implementing support for it on
Maemo only?