Implementing DOM elements in JS

[Blake Kaplan]

Hello everyone,

This post is a bit retroactive in that we've already implemented a bunch of
DOM objects in JavaScript. However, I noticed that we've never had an explicit
conversation about doing so. In particular, there are a bunch of reasons to
implement DOM APIs in JS over C++ as well as a bunch of downsides to doing so.
It seems worth it to decide explicitly that we want to make JS implementations
of DOM APIs something that we support and that we're willing to extend the
platform to make easier.

As I see it (and after a conversation with khuey the other day) here are some
of the advantages of implementing DOM APIs in JS:

* Lower barrier to entry for prospective implementors.
* Many people find JS to be easier to write and maintain.
* While it's possible to introduce security bugs in any programming
language, JS eliminates a large class of memory corruption bugs, reducing
our attack surface (see also: the Rust programming language!).

And some downsides:

* The lower barrier to entry might tempt more people to create
lower-standard implementations of DOM APIs, especially from extensions.
* Given that JS is less structured, the resulting code might be more
difficult to maintain in the long run.
* Because JS is also the language of untrusted code on the web, we're going
to have to be a lot more careful about accidental capability leaks and
sharing of objects between chrome and content.
* More languages means more complexity, especially in terms of debugging.

Of particular note here is that it isn't a given that we want to attract new
developers to implement DOM APIs. As we're providing a platform to the web, we
want to make sure that any APIs we expose are of high quality both in their
interface and their implementation. Using a more popular language with a lower
barrier to entry might invite sub-par DOM API implementations.

Personally, I feel that the ship has already sailed: we already have several
DOM APIs implemented in JS. The quality of these APIs has, to my knowledge,
not been noticeably lower than our C++-implemented APIs and given that we
already have established a strong review process at Mozilla, we already
control the quality of implementations that we accept. To that end, I think
that we should be extending the platform where needed to allow more JS DOM
implementations.

Any other opinions would be welcome.
--
Blake Kaplan

[Boris Zbarsky]

On 5/30/12 8:48 PM, Blake Kaplan wrote:
> It seems worth it to decide explicitly that we want to make JS implementations
> of DOM APIs something that we support and that we're willing to extend the
> platform to make easier.

I think this is fine in general. Just a few notes:

1) We actually need to put some work into implementing WebIDL correctly
for these APIs. Right now what we have is ... very broken in terms of
actually following specs.

2) Implementing prototypes in JS that "inherit" from prototypes
implemented in C++ is possible, but needs some DOM work.

3) Implementing objects in C++ that "inherit" from prototypes
implemented in JS is not much harder than just doing pure-JS stuff,
apart from the details of the prototype hookup process.

4) WebIDL has all sorts of things that depend on a distinction between
"platform objects" and other objects. We'll need a way to flag our
JS-implemented objects as "platform objects" from the point of view of
the C++ bindings and our C++-implemented objects as "platform objects"
from the point of view of the JS WebIDL bindings. I'm not quite sure
what the right way to do this is; right now we basically use the JSClass
on the C++ side to determine whether we have a platform object.

5) JS DOM implementations can't easily do the tricks with examining the
JS caller to decide whether to grant permission for something that C++
can do. We'd need to fix that.

> Personally, I feel that the ship has already sailed: we already have several
> DOM APIs implemented in JS. The quality of these APIs has, to my knowledge,
> not been noticeably lower than our C++-implemented APIs

It's hard to say. We have a lot of weird edge cases in our
JS-implemented APIs. For example, window.console.foopy() is a silent
no-op instead of throwing. Various other things have properties
directly on objects instead of on prototypes, etc. The infrastructure
to actually correctly implement APIs in JS without rocket science being
involved is just not present...

-Boris

[Boris Zbarsky]

On 5/30/12 9:23 PM, Boris Zbarsky wrote:
> I think this is fine in general. Just a few notes:

One more:

6) For performance-sensitive stuff, we'll depend on IonMonkey, since JM
is not likely to actually grow fast-paths for scripted getters and
setters, apparently.

-Boris

[Boris Zbarsky]

On 5/30/12 9:51 PM, Justin Lebar wrote:
> From a DOM perspective, we're adding some new methods and attributes
> to<iframe>. The methods are implemented in JS, currently in flagrant
> violation of all sorts of things, if I'm understanding Boris
> correctly.

Pretty much, if you're talking about things that are expressed in
WebIDL. ;)

For example,
https://bug753595.bugzilla.mozilla.org/attachment.cgi?id=625319 is
putting the new method on the object, not on the prototype. It's
binding it, so grabbing it off one iframe and applying it to another
won't work as it should per spec afaict. Neither will grabbing the
method and applying it to some non-iframe object throw, as it should,
etc, etc.

On the other hand, you're also not exposing this API to web content as
far as I can tell. So there isn't really a problem in that sense, just
like there's no problem with the fact that the properties/fields XBL
adds work nothing like WebIDL stuff.

-Boris

[Boris Zbarsky]

On 5/30/12 9:51 PM, Justin Lebar wrote:
> From a DOM perspective, we're adding some new methods and attributes
> to<iframe>.

And actually, this is another dimension of "implement stuff in JS" that
is interesting: having a mix of JS-implemented and C++-implemented stuff
on the same proto, with either a JS-implemented or C++-implemented
object. This can generally work, I think, if we get the other pieces
working. The hard part will be the prototype setup and Xrays.

-Boris

[Blake Kaplan]

David Humphrey <david.h...@senecacollege.ca> wrote:
> Would you mind mentioning which ones have been done in JS already?

A bunch of the new APIs that we're adding for Boot 2 Gecko are implemented in
JS glancing at a non-exhaustive list, I see:

* Webapps (http://mxr.mozilla.org/mozilla-central/source/dom/base/Webapps.js),
* Wifi (http://mxr.mozilla.org/mozilla-central/source/dom/wifi/DOMWifiManager.js),
* Contacts (http://mxr.mozilla.org/mozilla-central/source/dom/contacts/ContactManager.js),
* Settings (http://mxr.mozilla.org/mozilla-central/source/dom/settings/SettingsManager.js)
--
Blake Kaplan

[Blake Kaplan]

Boris Zbarsky <bzba...@mit.edu> wrote:
> 1) We actually need to put some work into implementing WebIDL correctly
> for these APIs. Right now what we have is ... very broken in terms of
> actually following specs.

I totally forgot to mention this (and this was one of khuey's biggest
objections).

This raises a second question: right now all of the JS implemented DOM APIs
use XPConnect to reflect themselves into content. As we implement HTML5 and
move towards WebIDL, what's the best path forward to make them behave
properly? The path of least resistance would be to use XPConnect on the
backend and write C++ shims that do the Right Thing with the new DOM bindings
(and generally work around XPConnect in the process). The best solution might
be to have some fancy way of hooking new DOM bindings up to JS (possibly with
a library .jsm to get edge cases right).

> 2) Implementing prototypes in JS that "inherit" from prototypes
> implemented in C++ is possible, but needs some DOM work.
>
> 3) Implementing objects in C++ that "inherit" from prototypes
> implemented in JS is not much harder than just doing pure-JS stuff,
> apart from the details of the prototype hookup process.

I don't think we've run into this yet and I'd like to ignore it as long as
possible. For now, things like nodes will have to be implemented in C++ and
IMO allowing new node implementations in JS is very low priority.

> 4) WebIDL has all sorts of things that depend on a distinction between
> "platform objects" and other objects. We'll need a way to flag our
> JS-implemented objects as "platform objects" from the point of view of
> the C++ bindings and our C++-implemented objects as "platform objects"
> from the point of view of the JS WebIDL bindings. I'm not quite sure
> what the right way to do this is; right now we basically use the JSClass
> on the C++ side to determine whether we have a platform object.

Where does this distinction occur? No matter what we do on the backend side,
we're almost certainly going to end up with a "chrome" JS object and a DOM
reflection of that object into content (for things like state and helper
methods. gal might know more from his experience with dom.js). We can make that reflection have any JS class we want and forward
appropriately.

> 5) JS DOM implementations can't easily do the tricks with examining the
> JS caller to decide whether to grant permission for something that C++
> can do. We'd need to fix that.

We've worked around this in the current implementations by installing our
objects manually as the result of events that pass the window to the
implementation and taking a snapshot of who we're being created for. Is that
not sufficient?

> It's hard to say. We have a lot of weird edge cases in our
> JS-implemented APIs. For example, window.console.foopy() is a silent
> no-op instead of throwing. Various other things have properties

That's wild, is there a bug?

> directly on objects instead of on prototypes, etc. The infrastructure
> to actually correctly implement APIs in JS without rocket science being
> involved is just not present...

Yeah. So, this is why this question needs to be asked and answered. There's
work to be done and we have to decide that we're going to do it. Otherwise,
we're going to end up with "real" DOM APIs and these odd DOM-like APIs that
mostly look right but don't behave quite like they should. I think that we
need to avoid that world at all costs (that is to say, I'd rather implement
everything in C++ than end up there).
--
Blake Kaplan

[smaug]

On 05/31/2012 03:48 AM, Blake Kaplan wrote:
> Hello everyone,
>
> This post is a bit retroactive in that we've already implemented a bunch of
> DOM objects in JavaScript. However, I noticed that we've never had an explicit
> conversation about doing so. In particular, there are a bunch of reasons to
> implement DOM APIs in JS over C++ as well as a bunch of downsides to doing so.
> It seems worth it to decide explicitly that we want to make JS implementations
> of DOM APIs something that we support and that we're willing to extend the
> platform to make easier.
>
> As I see it (and after a conversation with khuey the other day) here are some
> of the advantages of implementing DOM APIs in JS:
>
> * Lower barrier to entry for prospective implementors.
> * Many people find JS to be easier to write and maintain.

And some people find JS harder to write and maintain ;)


Anyhow, is this about random DOM objects or about elements like the subject says?

> * While it's possible to introduce security bugs in any programming
> language, JS eliminates a large class of memory corruption bugs, reducing
> our attack surface (see also: the Rust programming language!).
>
> And some downsides:
>
> * The lower barrier to entry might tempt more people to create
> lower-standard implementations of DOM APIs, especially from extensions.
> * Given that JS is less structured, the resulting code might be more
> difficult to maintain in the long run.
> * Because JS is also the language of untrusted code on the web, we're going
> to have to be a lot more careful about accidental capability leaks and
> sharing of objects between chrome and content.
> * More languages means more complexity, especially in terms of debugging.
>
> Of particular note here is that it isn't a given that we want to attract new
> developers to implement DOM APIs. As we're providing a platform to the web, we
> want to make sure that any APIs we expose are of high quality both in their
> interface and their implementation. Using a more popular language with a lower
> barrier to entry might invite sub-par DOM API implementations.
>
> Personally, I feel that the ship has already sailed: we already have several
> DOM APIs implemented in JS. The quality of these APIs has, to my knowledge,
> not been noticeably lower than our C++-implemented APIs

Well, so far non of those APIs have been actually used in the web.
Most of them are b2g-only (I mean, a random web site doesn't use them yet).

[smaug]

On 05/31/2012 03:48 AM, Blake Kaplan wrote:

One downside: Add more stuff to the GC. We'd need to make sure we don't regress
GC handling.

[Peter Van der Beken]

On 31/05/12 02:48, Blake Kaplan wrote:
> It seems worth it to decide explicitly that we want to make JS implementations
> of DOM APIs something that we support and that we're willing to extend the
> platform to make easier.

Thanks for bringing this up. I'd already suggested to jst a while back
that we should discuss this at the Monday DOM bindings meeting. It would
be good if people with an interest in this would actually attend that
meeting when we do discuss it.

I think we should scope this well. Seems to me that there are at least
two categories of objects that are of interest: DOM objects that are
implemented in JS and are self-contained, ie they don't inherit from C++
prototypes, and DOM object that are implemented in JS and do inherit
from C++ prototypes. Other objects should be out of scope IMO. WRT the
second set (inherit from C++ prototypes) there should probably be a
limit to the prototypes they can inherit from, probably a whitelist? I
think we should for example explicitly rule out inheriting from Node for
now (directly or indirectly), because of the complexity of changing our
native code to be able to deal with those.

Like Boris I do worry about the correctness of these implementations,
and we should try to provide infrastructure (helper modules, ...) to
make it easy to implement correctly.

Maybe we should discuss this at the next DOM binding meeting?

Peter

[Boris Zbarsky]

On 5/31/12 3:54 AM, Blake Kaplan wrote:
> This raises a second question: right now all of the JS implemented DOM APIs
> use XPConnect to reflect themselves into content. As we implement HTML5 and
> move towards WebIDL, what's the best path forward to make them behave
> properly? The path of least resistance would be to use XPConnect on the
> backend and write C++ shims that do the Right Thing with the new DOM bindings
> (and generally work around XPConnect in the process).

Yeah, this would actually be pretty painless (for some values of
painless: you still have to write the shim). For cases where
performance doesn't matter too much, this is a perfectly viable approach.

> The best solution might be to have some fancy way of hooking new DOM bindings up to JS (possibly with
> a library .jsm to get edge cases right).

Indeed.

>> 2) Implementing prototypes in JS that "inherit" from prototypes
>> implemented in C++ is possible, but needs some DOM work.
>>
>> 3) Implementing objects in C++ that "inherit" from prototypes
>> implemented in JS is not much harder than just doing pure-JS stuff,
>> apart from the details of the prototype hookup process.
>
> I don't think we've run into this yet and I'd like to ignore it as long as
> possible. For now, things like nodes will have to be implemented in C++ and
> IMO allowing new node implementations in JS is very low priority.

Item #2 above people have already run into, I thought. Think
EventTarget for the C++ proto and "anything that's implemented in JS and
is an EventTarget" for the JS thing.

The basic pattern for implementing such things would be to have a C++
object the JS can instantiate that would then have the JS-provided proto
as its proto. The JS-provided proto would chain up to the existing C++
protos (e.g. EventTarget). At that point the higher-up protos would
just operate on the underlying C++ object, and the JS proto can
implement things however it wants.

Note that this requires us to create some infrastructure for every C++
proto we want to thus subclass (e.g. this object that the JS would
instantiate).

>> 4) WebIDL has all sorts of things that depend on a distinction between
>> "platform objects" and other objects. We'll need a way to flag our
>> JS-implemented objects as "platform objects" from the point of view of
>> the C++ bindings and our C++-implemented objects as "platform objects"
>> from the point of view of the JS WebIDL bindings. I'm not quite sure
>> what the right way to do this is; right now we basically use the JSClass
>> on the C++ side to determine whether we have a platform object.
>
> Where does this distinction occur?

Typically during argument conversion. So for example, when converting
to a non-callback interface type you have to check that the object is
actually a platform object that implements that interface. That's not
too bad, if we implement things inheriting from c++ protos per my
proposal above and expose a way to do platform object detection to JS.
But there are some worse things. For example, when converting to a
sequence type, the only allowed inputs are platform array objects, JS
Arrays, and platform objects that support indexed properties. So we
need a way to detect in both the JS and C++ version of bindings, for an
arbitrary object, whether it's a "platform object that supports indexed
properties". Also, platform objects are not allowed to be converted to
dictionary types or callback interface types, or to callback function
types.

That _might_ be all the cases that care about the distinction.

> No matter what we do on the backend side, we're almost certainly going to end up with a "chrome" JS object and a DOM
> reflection of that object into content (for things like state and helper
> methods. gal might know more from his experience with dom.js). We can make that reflection have any JS class we want and forward
> appropriately.

Good, good.

>> 5) JS DOM implementations can't easily do the tricks with examining the
>> JS caller to decide whether to grant permission for something that C++
>> can do. We'd need to fix that.
>
> We've worked around this in the current implementations by installing our
> objects manually as the result of events that pass the window to the
> implementation and taking a snapshot of who we're being created for. Is that
> not sufficient?

I don't know. If an extension then touches your objects, will they
perceive that as being called "from chrome" or not? Maybe it doesn't
matter too much for them anyway, of course; this is only important for
cases where we want chrome to be able to do things with a DOM object
that the web page that object belongs to can't do.

>> It's hard to say. We have a lot of weird edge cases in our
>> JS-implemented APIs. For example, window.console.foopy() is a silent
>> no-op instead of throwing. Various other things have properties
>
> That's wild, is there a bug?

There is now. https://bugzilla.mozilla.org/show_bug.cgi?id=760144

> Yeah. So, this is why this question needs to be asked and answered. There's
> work to be done and we have to decide that we're going to do it.

Agreed.

I, like Peter, would be interested in just sorting this out at the
bindings meeting on Monday.

-Boris

[Kent James]

On 5/30/2012 6:23 PM, Boris Zbarsky wrote:
> 2) Implementing prototypes in JS that "inherit" from prototypes
> implemented in C++ is possible, but needs some DOM work.

Although it may not be relevant to the DOM discussions, in case there is
any interest I had to implement a method to create JS XPCOM objects that
can inherit from C++ XPCOM objects in order to implement JS new account
types in Thunderbird. The basic structure of Thunderbird relies on base
C++ objects that are then extended (traditionally using C++ inheritance)
to create the actual protocol-specific types (IMAP, POP3, or NNTP). To
do new protocols in javascript, I had to develop the ability to use JS
to extend C++ objects.

I wrote about this a year or so ago at
http://mesquilla.com/2010/12/28/overriding-c-xpcom-objects-using-javascript/
Essentially the concept is that the C++ object has to implement a
special interface to be extendable, and the javascript objects have to
explicitly tell the C++ object through this interface that they exist,
and also specify exactly what methods they want to override.

When this is all done, there is a C++ glue layer that is account-type
independent, and exists in a binary extension "New Account Types". A
specific account type though can then be implemented completely using
JS, as demonstrated by a Twitter account type in the extension "TweeQuilla".

Specific code for the glue layer (which now supports Windows, OSX, and
Linux) is available at https://bitbucket.org/rkentjames/skinkglue while
the demonstration Twitter account type is at
https://bitbucket.org/rkentjames/tweequilla

rkent

[Ehsan Akhgari]

I find the debugging problem very important. Even though we have a
script debugger under development, I'm not sure when it's going to be
useful to debug these kinds of APIs. And when it reaches that stage,
we still wouldn't be able to debug cross-language scenarios, so this
will make it very hard to debug problems unless the JS implemented
APIs are pretty much self-contained, and interact with the native APIs
as little as possible in both directions.

There's also the question of other developer tools not working on
these kinds of APIs. For example, console.log and friends tend to be
extremely slow -- but debugging them is impossible with the tools that
we currently have at hand (at least, until the future wonderful world
where the JS stacks are reported to the Gecko Profiler, and our chrome
JS goes through IonMonkey and such.)

--
Ehsan
<http://ehsanakhgari.org/>

> _______________________________________________
> dev-platform mailing list
> dev-pl...@lists.mozilla.org
> https://lists.mozilla.org/listinfo/dev-platform

[Blake Kaplan]

Peter Van der Beken <pet...@propagandism.org> wrote:
> Maybe we should discuss this at the next DOM binding meeting?

This sounds like a plan.
--
Blake Kaplan