Nokia created an extension to NPAPI and it has been supported by the flash plugin running on the N900 device. It allows them to quickly draw directly to a shared image buffer. This improves drawing performance on in Fennec as we can avoid X.
Here is a sketch of the proposal. I invite you to consider and respond:
We are proposing a NPAPI extension which would allow a plugin to render directly into 32-bit buffer in shared memory. The new extension will be designed so that plugin explicitly makes requests of the user agent whether the extension is supported (with NPN_GetValue and NPNVSupportsWindowlessLocal) and then enables the new rendering mode at runtime (with NPN_SetValue and NPPVpluginWindowlessLocalBool). In this way other plugins unaware of new extension will not be disrupted.
When new rendering mode is enabled the Xdrawable field of a standard XGraphicsExpose event holds a pointer to the new NPImageExpose structure which among other things contains pointer to the shared memory buffer, its dimensions, translation and scale factors.
typedef struct _NPImageExpose { char* data; /* image pointer */ int32 stride; /* Stride of data image buffer */ int32 depth; /* Depth of image pointer */ int32 x; /* Expose x */ int32 y; /* Expose y */ uint32 width; /* Expose width */ uint32 height; /* Expose height */ NPSize dataSize; /* Data buffer size */ float translateX; /* translate X matrix value */ float translateY; /* translate Y matrix value */ float scaleX; /* scale X matrix value */ float scaleY; /* scale Y matrix value */
> int32 depth; /* Depth of image pointer */ > What is the format of the buffer? In particular are there alpha values?
It depends what engine and plugins supports.... it can be any buffer format
> int32 x; /* Expose x */ > int32 y; /* Expose y */ > uint32 width; /* Expose width */ > uint32 height; /* Expose height */ > Is this in terms of buffer coordinates or plugin coordinates?
As I remember it is the buffer coordinates.
> NPSize dataSize; /* Data buffer size */ > Why is this needed?
We are giving whole rendering buffer to plugin... not only part which is related to buffer. In this case plugin can render stuff once (with or without alpha), and engine should not do anything more (copy, composite....)
Also there are one idea, about providing plugin own buffer to the engine. For example if plugin cannot paint in engine provided image format, then plugin does rendering in own buffer, then replacing whole NPImageExpose structure, or some fields values according to buffer parameters provided by plugin... and then engine checking that plugin did rendering into own buffer, and should try to composite plugin buffer into engine buffer.
>> int32 depth; /* Depth of image pointer */ >> What is the format of the buffer? In particular are there alpha values?
> It depends what engine and plugins supports.... it can be any buffer > format
How does the plugin (or the browser) know which format is used? Can the plugin force a special format, or does the browser force the most efficiently blittable one and the plugin has to implement all possible modes?
> When new rendering mode is enabled the Xdrawable field of a standard > XGraphicsExpose event holds a pointer to the new NPImageExpose > structure which among other things contains pointer to the shared > memory buffer, its dimensions, translation and scale factors.
Does this only apply in the context of windowless plugin instances? Or would all plugin instances be forced into windowless mode when this drawing mode was enabled?
How does this interact with transparent (partially-drawn) plugins? Is the plugin only allowed to draw to the shared-memory buffer at certain times, so that it can integrate properly with z-order painting?
One of the particular advantages of pepper (and the pepper-like drawing model that Chrome uses for windowless plugins currently) is that you give the plugin a drawable which is *not* the direct screen surface. This means that the plugin host can composite that drawable with other DOM elements (underlays and overlays) asynchronously, and you never have to synchronously wait for the plugin to paint.
> Does this only apply in the context of windowless plugin instances? Or would
Only for context... final decision about drawing mode support is up to plug-in. Also even when new drawing model is accepted, plug-in should be ready to paint in old modes too. (XWindow context) I'm not sure what is the best way to do it... but we should notify plugin somehow what XGraphicsExposeEvent.drawable contain, NPImageExpose or XDrawable pointer.. Currently we are setting XGraphicsExposeEvent.display = 0 to identify plugin that this expose event is not X-related.
> How does this interact with transparent (partially-drawn) plugins? Is the > plugin only allowed to draw to the shared-memory buffer at certain times, so > that it can integrate properly with z-order painting?
z-ordering is the same as in normal windowless pipeline.. builder DisplayList define when plugin should paint it's data.
In talking with a few people, there is a lot of improvement we can make on this API -- maybe pepper is the right approach.
The API I was spec-ing out above exists and is in use on the Nokia n900 devices. Assuming that we address issues raised above w/r/t format of the image buffer, clipping, and what does the plugin actually have to draw into the image buffer, I would like to propose this API as a Maemo only extension to the NPAPI.
> The API I was spec-ing out above exists and is in use on the Nokia > n900 devices. Assuming that we address issues raised above w/r/t > format of the image buffer, clipping, and what does the plugin > actually have to draw into the image buffer, I would like to propose > this API as a Maemo only extension to the NPAPI.
> Thoughts? Objections?
The biggest issue I want to address is the painting model. Chrome experience with OOP plugins shows that you really want to have an asynchronous painting model such as the one provided by pepper.
Is there a reason pepper wouldn't give the performance needed by Flash, instead of having yet another rendering model?
> > The API I was spec-ing out above exists and is in use on the Nokia > > n900 devices. Assuming that we address issues raised above w/r/t > > format of the image buffer, clipping, and what does the plugin > > actually have to draw into the image buffer, I would like to propose > > this API as a Maemo only extension to the NPAPI.
> > Thoughts? Objections?
> The biggest issue I want to address is the painting model. Chrome experience > with OOP plugins shows that you really want to have an asynchronous painting > model such as the one provided by pepper.
> Is there a reason pepper wouldn't give the performance needed by Flash, > instead of having yet another rendering model?
> --BDS
I am not suggesting that we do not do pepper for Maemo.
Right now, the n900 is just about done. I do not expect there to be any significant changes in flash before we ship Fennec 1.0. We have something that has been implemented by the platform (the n900), and we can either take advantage of it, or not. I think we do want to take advantage of it for speed, but advise other plugins not to use this approach.
> Right now, the n900 is just about done. I do not expect there to be > any significant changes in flash before we ship Fennec 1.0. We have > something that has been implemented by the platform (the n900), and we > can either take advantage of it, or not. I think we do want to take > advantage of it for speed, but advise other plugins not to use this > approach.
Since we'll have OOP plugins (or OOP tabs, which have the same rendering problem) in the 1.1 release, and OOP plugins cannot use the approach as it's currently specced, it seems like we'd be digging ourself into a hole which only partly-works and isn't very well designed.
On Oct 14, 2:16 pm, Benjamin Smedberg <benja...@smedbergs.us> wrote:
> On 10/14/09 5:13 PM, Doug Turner wrote:
> > Right now, the n900 is just about done. I do not expect there to be > > any significant changes in flash before we ship Fennec 1.0. We have > > something that has been implemented by the platform (the n900), and we > > can either take advantage of it, or not. I think we do want to take > > advantage of it for speed, but advise other plugins not to use this > > approach.
> Since we'll have OOP plugins (or OOP tabs, which have the same rendering > problem) in the 1.1 release, and OOP plugins cannot use the approach as it's > currently specced, it seems like we'd be digging ourself into a hole which > only partly-works and isn't very well designed.
> --BDS
This API clearly doesn't satifiy our requirements for OOP and it is X11 specific. However, it is what it is and support for it is on the n900 devices. If we want faster flash on the n900, we will want to support this NPAPI feature -- even if it is just on maemo.
> This API clearly doesn't satisfy our requirements for OOP and it is > X11 specific. However, it is what it is and support for it is on the > n900 devices. If we want faster flash on the n900, we will want to > support this NPAPI feature -- even if it is just on maemo.
I am trying to be pragmatic, but I really dislike this API too. It is a shame that we couldn't influence the design and build something that would be more useful. Josh -- do you have any objection with documenting this as what it is and implementing support for it on Maemo only?
