Binary Components - Another Approach

[Bobby Holley]

I work on both js-ctypes and XPConnect, so I'm in involved in the pain here
one way or another. ;-)

A lot of folks have expressed criticism of js-ctypes over the past few
days. Much of it is totally valid. And to be honest, I don't think that
js-ctypes is the right solution for a lot of the people trying to use it.
js-ctypes is designed for extensions that do most of their work in JS, and
just need to make a few isolated native calls. Rewriting a large extension
with tight coupling between JS and C++ is likely to be very painful for the
reasons people have already mentioned (in particular, lifetime and memory
management).

The thing is, tight coupling between JS and C++ isn't the actual problem.
You can write an extension that uses 100k lines of native component code
and it will (most likely) remain compatible across major releases. Binary
incompatibility doesn't intrinsically come from using binary components; it
comes from using Mozilla interfaces (like nsINode and nsIURI) in C++
extension code. This is _really_ important to recognize. JS can use nsIFoo
across major releases, C++ cannot. The extent to which binary extensions
are a problem depends entirely on this, and not at all on whether they us
js-ctypes or XPConnect to talk to their _own_ native code.

But I think this hasn't been at all clear in our communication. The message
has been "if you use js-ctypes, you won't have to worry about rapid
release." This is technically true. But it falsely implies the converse -
that is, "you must use js-ctypes in order to avoid problems with rapid
release." This is false. So I have a sense that there's a lot of
misdirected effort, with people building crazy JS refcounting mechanisms
when all they really need to do is to stop using nsIFoo from C++.

So, extension developers: What if Mozilla promised to support binary
components over a reasonable timeframe, provided that those extensions used
only a limited set of mozilla headers (nsISupports, maybe things like
nsIArray, etc) from C++? Would this make you happy? I'm essentially
proposing a return to frozen interfaces, except that the frozen set would
be much, much smaller - just core XPCOM stuff, ideally countable on one
hand. You can keep using XPCOM to your hearts' content, so long as you
manipulate your nsIFoo objects entirely in JS and never pass them to C++.

We could enforce this rule on technical level (keeping stable headers in a
separate directory, for example) and automatically identify compliance.
This would probably be helpful for both AMO reviewers and extension
developers themselves, since the latter group could be sure they weren't
accidentally pulling in something that could cause incompatibility down the
line.

There are various hurdles, like [noscript] interfaces. But I think that
they're tractable problems if we decide to go down this road. There are
also surely extensions for which this isn't a practical approach. But if we
can alleviate the pain for 90% of extensions, I think it's totally
worthwhile.

So. Would this approach alleviate your pain?

Cheers,
bholley

[Simon Kornblith]

This would be perfect for our binary components.

Simon

[Kyle Huey]

The problem is that nobody has identified the set (or even a set) of
interfaces to freeze. And it's possible that the set that extension
authors would desire would conflict with the set of interfaces that we
absolutely do not want to freeze (e.g. nsINode).

- Kyle

[Bobby Holley]

I more or less proposed one: nsISupports, and possibly various pieces of
glue (like nsIArray) that we don't touch often. No DOM, no nsIURI, no
observers, none of that stuff. Just the basics to use XPCOM.

bholley

[Wes Garland]

On 19 January 2012 15:52, Kyle Huey <m...@kylehuey.com> wrote:

> The problem is that nobody has identified the set (or even a set) of
> interfaces to freeze. And it's possible that the set that extension
> authors would desire would conflict with the set of interfaces that we
> absolutely do not want to freeze (e.g. nsINode).
>

You know, you could just walk the list of all interfaces and freeze the
ones which you think are easily freezable -- then let developers file bugs
when they find they can't do what they want with the frozen interfaces.

I'm assuming nsINode is a DOM Node interface, and it seems reasonable to me
to not freeze that.

What about coming up with some kind of blended JS/C++ architecture that
lets people easily pass C++ data to JS code, and push DOM manipulation into
script? *This* is a place where js-ctypes could really shine instead of
confounding matters.

We do work on the server-side with CommonJS modules that are part C, part
JS, and find it to be a very powerful pairing. The key to making it work
well in our case is ensuring that all data that matters is held in a JS
object somehow, so that the JS GC knows everything it needs to know to make
memory management a breeze.

Wes

--
Wesley W. Garland
Director, Product Development
PageMail, Inc.
+1 613 542 2787 x 102

[Bobby Holley]

On Thu, Jan 19, 2012 at 1:07 PM, Wes Garland <w...@page.ca> wrote:

> You know, you could just walk the list of all interfaces and freeze the
> ones which you think are easily freezable -- then let developers file bugs
> when they find they can't do what they want with the frozen interfaces.
>

This is not what I'm proposing. I think this would eventually just lead us
back to where we are. I'm suggesting that we just freeze the bare minimum
to use XPCOM. No gecko stuff.

>
> What about coming up with some kind of blended JS/C++ architecture that
> lets people easily pass C++ data to JS code, and push DOM manipulation into
> script? *This* is a place where js-ctypes could really shine instead of
> confounding matters.
>
>

I'm pretty sure people can roll their own IDL for whatever they need to do
send into JS-land. It's possible that marshalling the stuff into JS is
impractical performance-wise for large amounts of data, but we could
probably work on that.

bholley

[Jonas Sicking]

On Thu, Jan 19, 2012 at 1:06 PM, Bobby Holley <bobby...@gmail.com> wrote:

> On Thu, Jan 19, 2012 at 12:52 PM, Kyle Huey <m...@kylehuey.com> wrote:
>
>> The problem is that nobody has identified the set (or even a set) of
>> interfaces to freeze.  And it's possible that the set that extension
>> authors would desire would conflict with the set of interfaces that we
>> absolutely do not want to freeze (e.g. nsINode).
>

> I more or less proposed one: nsISupports, and possibly various pieces of
> glue (like nsIArray) that we don't touch often. No DOM, no nsIURI, no
> observers, none of that stuff. Just the basics to use XPCOM.

Additionally, we wouldn't really be "freezing" nsISupports (and
whatever other interfaces we end up choosing), right? We'd just change
it a whole lot less often and with lots of heads-up warnings.

One thing that doesn't seem addressed by this proposal though is
people wanting to do more heavy interaction between their C++ and JS
code. Currently a good way to do this is to write a few XPCOM-IDL
interfaces in your C++ code, and then use these from JS. This is a
more powerful and author friendly mechanism than using js-ctypes. When
doing that you'd most likely also use your own XPCOM-IDL interfaces
from C++ code in order to not have to duplicate interfaces.

Based on the threads the last couple of days it sounds like this would
help people who are writing binary components. It also seems to me
that there's little risk involved in this pattern as long as we can
ensure that those XPCOM-IDL interfaces don't get mixed up with "our"
interfaces.

/ Jonas

[Bobby Holley]

On Thu, Jan 19, 2012 at 1:59 PM, Jonas Sicking <jo...@sicking.cc> wrote:

> Additionally, we wouldn't really be "freezing" nsISupports (and
> whatever other interfaces we end up choosing), right? We'd just change
> it a whole lot less often and with lots of heads-up warnings.
>

Exactly.

>
> One thing that doesn't seem addressed by this proposal though is
> people wanting to do more heavy interaction between their C++ and JS
> code. Currently a good way to do this is to write a few XPCOM-IDL
> interfaces in your C++ code, and then use these from JS. This is a
> more powerful and author friendly mechanism than using js-ctypes. When
> doing that you'd most likely also use your own XPCOM-IDL interfaces
> from C++ code in order to not have to duplicate interfaces.
>

This isn't a problem (unless I'm misunderstanding you). It's fine for
authors to use their own IDL interfaces in C++, because they're the ones
choosing when to change it. We just want them to not use _gecko_ IDL in C++.

>
> Based on the threads the last couple of days it sounds like this would
> help people who are writing binary components. It also seems to me
> that there's little risk involved in this pattern as long as we can
> ensure that those XPCOM-IDL interfaces don't get mixed up with "our"
> interfaces.
>

Hooray! I think we can do this pretty easily by explicitly exporting them
(or symlinking them) into a separate 'public include' directory in the SDK.

bholley

[Robert O'Callahan]

On Fri, Jan 20, 2012 at 9:02 AM, Bobby Holley <bobby...@gmail.com> wrote:

> A lot of folks have expressed criticism of js-ctypes over the past few
> days. Much of it is totally valid. And to be honest, I don't think that
> js-ctypes is the right solution for a lot of the people trying to use it.
> js-ctypes is designed for extensions that do most of their work in JS, and
> just need to make a few isolated native calls. Rewriting a large extension
> with tight coupling between JS and C++ is likely to be very painful for the
> reasons people have already mentioned (in particular, lifetime and memory
> management).
>

Maybe it would be a better investment, and more fun, to enhance js-ctypes
to make it easier to use where it's currently painful? E.g., adding some
support for C++, and providing a hook for wrapper destruction?

Rob
--
"If we claim to be without sin, we deceive ourselves and the truth is not
in us. If we confess our sins, he is faithful and just and will forgive us
our sins and purify us from all unrighteousness. If we claim we have not
sinned, we make him out to be a liar and his word is not in us." [1 John
1:8-10]

[Bobby Holley]

On Thu, Jan 19, 2012 at 2:52 PM, Robert O'Callahan <rob...@ocallahan.org>wrote:

> Maybe it would be a better investment, and more fun, to enhance js-ctypes
> to make it easier to use where it's currently painful?

E.g., adding some support for C++, and providing a hook for wrapper
> destruction?
>

I don't think so. js-ctypes just isn't mean to provided the large-scale
reflection that people use XPCOM for. You have to manually declare each
function (no typelibs), so I don't think C++ support would buy much for
ease-of-conversion. Additionally, I don't think we _can_ support C++ or
destructors, because the JS GC doesn't guarantee finalization ordering. So
all the assumptions about destructor ordering for stack-allocated objects
go out the window.

More importantly though, I think that focusing on js-ctypes is missing the
point. Extensions using components rather than js-ctypes isn't the problem.
Extensions using gecko interfaces in native code is the problem, and this
is an issue that developers are going to have to solve regardless of which
FFI they use. The only difference is that js-ctypes automatically precludes
the problematic behavior, whereas we have yet to discourage bad behavior
for components. This is where I think we should focus our attention.

bholley

[Robert O'Callahan]

On Fri, Jan 20, 2012 at 12:06 PM, Bobby Holley <bobby...@gmail.com>wrote:

> I don't think so. js-ctypes just isn't mean to provided the large-scale
> reflection that people use XPCOM for. You have to manually declare each
> function (no typelibs), so I don't think C++ support would buy much for
> ease-of-conversion.

We could have a tool that builds the js-ctypes boilerplate from C++ debug
information.

Additionally, I don't think we _can_ support C++ or destructors, because
> the JS GC doesn't guarantee finalization ordering. So all the assumptions
> about destructor ordering for stack-allocated objects go out the window.
>

For heap-allocated objects this isn't much of a problem.

For RAII support I guess we'd need a JS language extension.

Wrapping binary code in XPIDL isn't the greatest developer experience
either, and I'm hopeful we could beat it with something based on js-ctypes.

More importantly though, I think that focusing on js-ctypes is missing the
> point. Extensions using components rather than js-ctypes isn't the problem.
> Extensions using gecko interfaces in native code is the problem, and this
> is an issue that developers are going to have to solve regardless of which
> FFI they use. The only difference is that js-ctypes automatically precludes
> the problematic behavior, whereas we have yet to discourage bad behavior
> for components. This is where I think we should focus our attention.
>

"Freeze some interfaces, discourage usage of unfrozen interfaces" was our
strategy for most of the lifetime of Mozilla, and it didn't work very well.
I'm pessimistic about the effectiveness of verbal encouragement.

[Ted Mielczarek]

On Thu, Jan 19, 2012 at 7:49 PM, Robert O'Callahan <rob...@ocallahan.org> wrote:
> On Fri, Jan 20, 2012 at 12:06 PM, Bobby Holley <bobby...@gmail.com>wrote:
>
>> I don't think so. js-ctypes just isn't mean to provided the large-scale
>> reflection that people use XPCOM for. You have to manually declare each
>> function (no typelibs), so I don't think C++ support would buy much for
>> ease-of-conversion.
>
>
> We could have a tool that builds the js-ctypes boilerplate from C++ debug
> information.

We'd basically be building something like SWIG[1]. I've used it to
generate Python wrappers for C++ in the past (from C++ header files),
and it does an okay job. You tend to have to add additional metadata
to make things work properly. Also, if you want to take an idiomatic
C++ interface and turn it into an idiomatic interface in Python (or
whatever other language), you have to do a lot more work.

> Additionally, I don't think we _can_ support C++ or destructors, because
>> the JS GC doesn't guarantee finalization ordering. So all the assumptions
>> about destructor ordering for stack-allocated objects go out the window.
>>
>
> For heap-allocated objects this isn't much of a problem.
>
> For RAII support I guess we'd need a JS language extension.
>
> Wrapping binary code in XPIDL isn't the greatest developer experience
> either, and I'm hopeful we could beat it with something based on js-ctypes.

This is true. XPIDL generates terrible interfaces for both C++ and
JavaScript. It may be possible to do better, since XPIDL is by design
targeting the lowest-common-denominator of language neutrality.

-Ted

1. http://www.swig.org/

[Bobby Holley]

On Thu, Jan 19, 2012 at 4:49 PM, Robert O'Callahan <rob...@ocallahan.org>wrote:

> Wrapping binary code in XPIDL isn't the greatest developer experience
> either, and I'm hopeful we could beat it with something based on js-ctypes.
>

It might leave something to be desired, but developers using it isn't
really a problem for us. My point is that it's unproductive to use our
energy and political capital trying to move developers off of XPCOM when
they're happy with it and it doesn't cause us trouble.

> "Freeze some interfaces, discourage usage of unfrozen interfaces" was our
> strategy for most of the lifetime of Mozilla, and it didn't work very well.
>

I don't think this is a fair comparison, and I wish I hadn't made it. I'm
not talking about the sort of freezing we did in the past. In fact, I'm
really only talking about "freezing" one interface (nsISupports). I used
the plural simply because Mook asked on IRC if extensions could use
nsIArray too (rather than duplicating it), and I didn't see why not, given
that it hasn't been touched in (at least) 5 years.

> I'm pessimistic about the effectiveness of verbal encouragement.
>

Unless we're going to unilaterally unsupport binary components, we're going
to have to engage with the their developers. So rather than spending energy
verbally encouraging them to rewrite their code in js-ctypes, something
that (a) they don't want to do, and (b) doesn't buy us much at all, we
should encourage them to do what matters: stop using Gecko interfaces in
C++, so that the extensions remain compatible across releases.

Also, we could do a bit more than verbal encouragement if we were so
inclined. For example, we could alter the SDK to just stop exporting gecko
headers, and force them to manually include
"headers/private/goaway/seriously/noreallyyoudontwanttoincludethese/nsIFoo.h"
in order to access them.

bholley

[Asa Dotzler]

I'm really excited about this. If we can make the lives of add-on
developers better by committing to a couple of already rarely changed
interfaces and doing developer outreach, I think that'd be great progress.

- A

[Zack Weinberg]

Hmm, would this preclude switching to a more efficient QueryInterface
implementation, as bsmedberg said upthread might be possible if we
didn't have to maintain IIDs?

(I am pretty seriously tempted to find the time to see what breaks and
what the performance delta is if we blow up QueryInterface and replace
it with dynamic_cast<>.)

zw

[Asa Dotzler]

What's the timeframe for that work? Could we buy ourselves and our
extension community some time by not screwing around with it until we
were sure we had a real improvement?

- A

[Bobby Holley]

On Thu, Jan 19, 2012 at 6:24 PM, Zack Weinberg <za...@panix.com> wrote:

> Hmm, would this preclude switching to a more efficient QueryInterface
> implementation, as bsmedberg said upthread might be possible if we didn't
> have to maintain IIDs?
>

I'm not positive, but at first glance I don't see it as a significant
problem.

First there's the C++ case: If extensions aren't touching gecko from native
code, then we don't need any sort of IID mechanism to ensure that they get
the component they expect (because they're not getting _any_ components).
They can just use whatever new nsISupports we come up with, so the only
thing that will require version checking is nsISupports itself (along with
any trivial helpers like nsIArray). This can be accomplished by a simple
XPCOM versioning scheme, where we rev to an incompatible version if we ever
really need to change nsISupports.

Now, for the JS case, extension code would indeed be accessing gecko
interfaces. But thankfully, that JS code runs using whatever version of
XPConnect shipped with the given release, so it would automatically get the
right behavior. It would still throw if it tried to use a method whose
signature was changed in an incompatible way, but this is true for any
extension (and throwing a JS exception is much better than crashing).

This is all sort of academic though, because I think that getting rid of
IIDs is a mammoth task, and unlikely to happen any time soon. According to
grep, there are 795 lines containing the string "iid" in XPConnect alone,
so you're in for a real treat if you try to remove them entirely. Perhaps a
better strategy for experimentation would be to just make IIDs 64 bits?
This would be less invasive, and probably just require changing a dozen or
so hard-coded constants. You'd be unlikely to run into collisions with a
single m-c snapshot, and it'd give us an idea of the performance overhead
of IID comparisons.

bholley

[Zack Weinberg]

On 2012-01-19 6:29 PM, Asa Dotzler wrote:
> On 1/19/2012 6:24 PM, Zack Weinberg wrote:
>> Hmm, would this preclude switching to a more efficient QueryInterface
>> implementation, as bsmedberg said upthread might be possible if we
>> didn't have to maintain IIDs?
>>
>> (I am pretty seriously tempted to find the time to see what breaks and
>> what the performance delta is if we blow up QueryInterface and replace
>> it with dynamic_cast<>.)
>

> What's the timeframe for that work? Could we buy ourselves and our
> extension community some time by not screwing around with it until we
> were sure we had a real improvement?

*I* am unlikely to have time to do it in the near future, and regardless
of who does it, it certainly is the sort of thing that ought to bake on
a branch until we're all sure it's a big win. (It might also be the
sort of thing that's an enormous pain in the ass to maintain off trunk
for any length of time, though.)

Tangenting, something else we badly need for other reasons, and that
might mitigate a lot of the pain extension devs are feeling, is an IDL
compiler that generates JS-to-C++-or-vice-versa glue that *isn't*
XPCOM-based.

zw

[Kyle Huey]

That'll totally break js implemented components (among others), and faking
compiler specific RTTI information is going to be far harder than faking
vtables (which we have existing code for that has a decade of testing).

I don't think I would take any plan to change how QI works (as opposed to
how big its argument is) seriously.

- Kyle

[Asa Dotzler]

On 1/19/2012 9:34 PM, Zack Weinberg wrote:
> Tangenting, something else we badly need for other reasons, and that
> might mitigate a lot of the pain extension devs are feeling, is an IDL
> compiler that generates JS-to-C++-or-vice-versa glue that *isn't*
> XPCOM-based.

Can you explain this further? What would this buy us in terms of
compatibility?

- A

[Brian Smith]

Bobby Holley wrote:
> I more or less proposed one: nsISupports, and possibly various pieces
> of glue (like nsIArray) that we don't touch often. No DOM, no nsIURI,
> no observers, none of that stuff. Just the basics to use XPCOM.

Forgive my naive question: Why couldn't/shouldn't we freeze things like nsIURI? It is a very simple interface that depends on only core XPCOM types. I don't know about other Necko/Gecko developers, but I wouldn't mind maintaining nsIURI the same way that nsIX509Cert was maintained (see nsIX509Cert2, nsIX509Cert3, etc.) or doing other things to avoid changing the interface unnecessarily.

IMO, things like nsIURI are not a problem as far as maintaining backward compatibility. Things like nsICache*, or NSS, which we are more implementation details that likely to make major changes to, and which aren't really designed for addons to use, are the problematic cases. I think the solution for them is basically to hide "internal" interfaces (at least new ones) in an internal namespace that addons can't access (i.e. stop exporting them from libxul or whatever library they are in, whenever possible).

- Brian

[Mike Hommey]

On Fri, Jan 20, 2012 at 07:21:37AM +0100, Kyle Huey wrote:
> On Fri, Jan 20, 2012 at 3:24 AM, Zack Weinberg <za...@panix.com> wrote:
>

> That'll totally break js implemented components (among others), and faking
> compiler specific RTTI information is going to be far harder than faking
> vtables (which we have existing code for that has a decade of testing).

Do compilers even allow dynamic_cast<> without RTTI being enabled? we
surely don't want RTTI enabled.

Mike

[Kyle Huey]

On Fri, Jan 20, 2012 at 8:49 AM, Mike Hommey <m...@glandium.org> wrote:

> Do compilers even allow dynamic_cast<> without RTTI being enabled? we
> surely don't want RTTI enabled.

Other than dynamic_cast<void*>, no.

- Kyle

[Robert O'Callahan]

On Fri, Jan 20, 2012 at 8:32 PM, Brian Smith <bsm...@mozilla.com> wrote:

> IMO, things like nsIURI are not a problem as far as maintaining backward
> compatibility.
>

We changed nsIURI twice last year.

[Mike Hommey]

On Fri, Jan 20, 2012 at 09:38:06PM +1300, Robert O'Callahan wrote:
> On Fri, Jan 20, 2012 at 8:32 PM, Brian Smith <bsm...@mozilla.com> wrote:
>
> > IMO, things like nsIURI are not a problem as far as maintaining backward
> > compatibility.
> >
>
> We changed nsIURI twice last year.

None of which required breaking binary compatibility, but did.
That's the other problem with the evolution of the mozilla codebase.

Mike

[Matthew Gertner]

On Jan 20, 2012, at 12:06 AM, Bobby Holley wrote:
> I don't think so. js-ctypes just isn't mean to provided the large-scale
> reflection that people use XPCOM for. You have to manually declare each
> function (no typelibs), so I don't think C++ support would buy much for
> ease-of-conversion.

As Ted mentioned, swig could provide inspiration and or code for this. We used a branch (see http://swig.svn.sourceforge.net/viewvc/swig/branches/gsoc2008-maciekd/Doc/Manual/C.html and http://swig.svn.sourceforge.net/svnroot/swig/branches/gsoc2008-maciekd/) to generate a C wrapper for a large C++ library. I don't see any reason why a complete JS wrapper couldn't be generated for C++ code, so this could be improved over time. No need to use debug info since you're always going to have headers (right?).

> Additionally, I don't think we _can_ support C++ or
> destructors, because the JS GC doesn't guarantee finalization ordering. So
> all the assumptions about destructor ordering for stack-allocated objects
> go out the window.

How does JS code get its hands on stack-allocated objects? A more realistic use case would seem to me a C/C++ API that returns heap-allocated objects that need to be freed by the caller. If these are wrapped using JS objects and js-ctypes, there should be a way to free them when the associated wrapper is freed.

> More importantly though, I think that focusing on js-ctypes is missing the
> point. Extensions using components rather than js-ctypes isn't the problem.
> Extensions using gecko interfaces in native code is the problem, and this
> is an issue that developers are going to have to solve regardless of which
> FFI they use. The only difference is that js-ctypes automatically precludes
> the problematic behavior, whereas we have yet to discourage bad behavior
> for components. This is where I think we should focus our attention.

The problem for add-on developers right now is that FF releases are very frequent and each one explicitly breaks all binary components using kVersion. I guess some developers are just updating their add-on every time, but I know that for a lot of developers (including us) this is not an appealing option, whatever interfaces you use. There are three kinds of extensions using binary XPCOM components today:

1) Those that use XPCOM only with custom IDL to access a C/C++ library from JS.
2) Those that contain C++ code for performance, obfuscation, integration with legacy code, out of habit, etc. and use interfaces like nsIFile, nsIURI, nsIIOService, nsIPrefBranch, etc. even though that part of the code could technically be written in JS.
3) Those who do deeper hacking of Gecko using nsIDocShell, nsIDocument, nsIAppShell, etc.

I think it's fair to say that those in 2) need to separate their code and do the XPCOM stuff from JS. I also think it's fair to say that those in 3) need to update their extension every 6 weeks if they really can't do what they need from JS (I can't believe this is a very large group anyway. Those in 1) (which includes us for the project I mentioned) are kind of out of luck right now. If you can find a way to selectively ban use of IDL interfaces (rather than relying on kVersion) then this would be a perfectly satisfying solution. I find the idea of turning js-ctypes into a really workable JS <-> C/C++ bridge to be more aesthetically pleasing, but maybe it's too hard to make sense right now.

Matt

[Pavol Mišík]

On 19. 1. 2012 21:02, Bobby Holley wrote:
> I work on both js-ctypes and XPConnect, so I'm in involved in the pain here
> one way or another. ;-)
>

> A lot of folks have expressed criticism of js-ctypes over the past few
> days. Much of it is totally valid. And to be honest, I don't think that
> js-ctypes is the right solution for a lot of the people trying to use it.
> js-ctypes is designed for extensions that do most of their work in JS, and
> just need to make a few isolated native calls. Rewriting a large extension
> with tight coupling between JS and C++ is likely to be very painful for the
> reasons people have already mentioned (in particular, lifetime and memory
> management).
>

> Cheers,
> bholley

Hi.

I think this is the minimalist set of interfaces you need in order to
register XPCOM component and deal with interface changes dynamically.
I’m not sure it is complete.
nsISupports
nsICategoryManager
nsIComponentManager
nsIComponentRegistrar
nsIServiceManager
nsIFactory
nsIFile
nsIModule
nsIObserverService
nsIObserver
nsIInterfaceRequestor
nsIInterfaceInfoManager
nsIInterfaceInfo
nsIMemory
nsISimpleEnumerator
nsIXULAppInfo
nsIClassInfo

It would be also more convenient if also following interfaces were
frozen or at least nsIRunnable.

nsIEventTarget
nsIRunnable
nsIThread
nsIThreadManager
nsIWeakReference
nsISupportsWeakReference


pm-

[Robert O'Callahan]

How would you extend nsIURI without breaking binary compatibility?

[Mike Hommey]

On Fri, Jan 20, 2012 at 10:49:23PM +1300, Robert O'Callahan wrote:
> On Fri, Jan 20, 2012 at 9:45 PM, Mike Hommey <m...@glandium.org> wrote:
>
> > On Fri, Jan 20, 2012 at 09:38:06PM +1300, Robert O'Callahan wrote:
> > > On Fri, Jan 20, 2012 at 8:32 PM, Brian Smith <bsm...@mozilla.com> wrote:
> > >
> > > > IMO, things like nsIURI are not a problem as far as maintaining
> > backward
> > > > compatibility.
> > > >
> > >
> > > We changed nsIURI twice last year.
> >
> > None of which required breaking binary compatibility, but did.
> > That's the other problem with the evolution of the mozilla codebase.
> >
>
> How would you extend nsIURI without breaking binary compatibility?

It pretty much depends on what changes you want to make, but adding
virtual functions doesn't require breaking binary compatibility. Adding
new data members doesn't require breaking binary compatibility as long
as objects are not created with "new" in the component. There are many
binary compatible changes possible.

However, those that happened last year happened to change the order of
the functions in the vtable, and that is definitely not binary
compatible. AFAICS, the changes could have been made without doing that.

Mike

[Philipp Kewisch]

On 1/20/12 2:07 AM, Ted Mielczarek wrote:
> On Thu, Jan 19, 2012 at 7:49 PM, Robert O'Callahan<rob...@ocallahan.org> wrote:
>> On Fri, Jan 20, 2012 at 12:06 PM, Bobby Holley<bobby...@gmail.com>wrote:
>> We could have a tool that builds the js-ctypes boilerplate from C++ debug
>> information.
>
> We'd basically be building something like SWIG[1]. I've used it to
> generate Python wrappers for C++ in the past (from C++ header files),
> and it does an okay job. You tend to have to add additional metadata
> to make things work properly. Also, if you want to take an idiomatic
> C++ interface and turn it into an idiomatic interface in Python (or
> whatever other language), you have to do a lot more work.

There is already a a tool that does this at least for C. It doesn't
catch all cases (like enums), but it did a pretty good job on libical.
Check out the js-ctypes part of this blog post:

http://philikon.wordpress.com/2010/09/17/mozilla-four-months-later-a-status-update/

I've made some slight modifications locally that made it work for what I
needed, I'd be happy to contribute it back if needed.

Philipp

[Benjamin Smedberg]

On 1/20/2012 3:47 AM, Pavol Mišík wrote:
>
>
> I think this is the minimalist set of interfaces you need in order to
> register XPCOM component and deal with interface changes dynamically.
> I’m not sure it is complete.

I think you've significantly misunderstood the proposal. The proposal at
hand is *not* to support "dealing" with interface changes. It is merely
to allow extensions to implement their own interfaces. I would even like
to extend the proposal so that extension cannot use the component
manager or service manager: they would instead just ask for components
directly from their binary library. This gives the 'benefit' of
XPConnect integration without the capability of poking into mozilla
internals in a meaningful way.

So for instance, extension JS code might look like this:

var binarymodule =
Components.utils.importBinaryComponent("myextension.dll");
binarymodule.createInstance("contractid", interface);

In this case, the *only* interfaces we would freeze are nsISupports,
nsIFactory, and mozilla::Module.

--BDS

[Zack Weinberg]

On 2012-01-19 11:32 PM, Brian Smith wrote:
> Bobby Holley wrote:
>> I more or less proposed one: nsISupports, and possibly various
>> pieces of glue (like nsIArray) that we don't touch often. No DOM,
>> no nsIURI, no observers, none of that stuff. Just the basics to use
>> XPCOM.
>
> Forgive my naive question: Why couldn't/shouldn't we freeze things
> like nsIURI? It is a very simple interface that depends on only core
> XPCOM types. I don't know about other Necko/Gecko developers, but I
> wouldn't mind maintaining nsIURI the same way that nsIX509Cert was
> maintained (see nsIX509Cert2, nsIX509Cert3, etc.) or doing other
> things to avoid changing the interface unnecessarily.

Bug 234856 says the nsIX509Cert* approach doesn't always work.

zw

[Zack Weinberg]

It would be an alternative avenue to giving Javascript access to
third-party C++ DLLs (that is, alternative to the proposal elsethread
that freezes nsISupports, nsIFactory, and mozilla::Module so that
third-party C++ can continue to define XPIDL interfaces).

I imagine the code it would generate would be sort of like the code I
wrote by hand for js::Perf last year:
http://mxr.mozilla.org/mozilla-central/source/js/src/perf/jsperf.cpp

zw

[Zack Weinberg]

On 2012-01-19 10:21 PM, Kyle Huey wrote:
> On Fri, Jan 20, 2012 at 3:24 AM, Zack Weinberg<za...@panix.com> wrote:
>>
>> (I am pretty seriously tempted to find the time to see what breaks and
>> what the performance delta is if we blow up QueryInterface and replace it
>> with dynamic_cast<>.)
>
> That'll totally break js implemented components (among others)

It doesn't have to, although I do see that a significant amount of glue
might have to be written in that direction.

> and faking
> compiler specific RTTI information is going to be far harder than faking
> vtables (which we have existing code for that has a decade of testing).

At least initially, we shouldn't need that. We get the easy 90% -- C++
on both sides of the QI, no games with tearoffs or anything -- with

template <class T, class Dest>
inline nsresult
CallQueryInterface(T* aSource, Dest** aDest)
{
Dest* d = dynamic_cast<Dest*>(aSource);
if (d) {
NS_ADDREF(*aDest = d);
return NS_OK;
}
return aSource->QueryInterface(NS_GET_TEMPLATE_IID(Dest),
reinterpret_cast<void **>(aDest));
}

And that should give us enough of the win to tell whether the idea is
worth pursuing further.

zw

[Zack Weinberg]

On 2012-01-19 11:49 PM, Mike Hommey wrote:
>
> Do compilers even allow dynamic_cast<> without RTTI being enabled? we
> surely don't want RTTI enabled.

We do not know that the perf hit of RTTI is greater than the perf hit of
not being able to use dynamic_cast where it makes sense, and besides,
turning it on is a large step on the road to modern C++.

zw

[Jonathan Protzenko]

As far as I know, no compilers implement fast dynamic casting using
integer modulo (see
www2.research.att.com/~bs/fast_dynamic_casting.pdf), unfortunately.

jonathan

> _______________________________________________
> dev-platform mailing list
> dev-pl...@lists.mozilla.org
> https://lists.mozilla.org/listinfo/dev-platform

[Joshua Cranmer]

On 1/19/2012 8:24 PM, Zack Weinberg wrote:
> (I am pretty seriously tempted to find the time to see what breaks and
> what the performance delta is if we blow up QueryInterface and replace
> it with dynamic_cast<>.)

Knowing that LLVM (which is a more modern codebase and uses several C++
features more heavily) uses its own replacement-for-dynamic-cast makes
me believe that dynamic_cast<> would not be a speed improvement over a
hand-rolled replacement that would still give us the ability to allow
non-C++ XPCOM components without having more ABI-specific code.

[Mike Hommey]

Compilers are pretty bad at doing things fast with RTTI, without even
accounting the size overhead of the RTTI data, which, for the supposed
benefit, is just too big.

Mike

[Jason Orendorff]

On 1/19/12 7:07 PM, Ted Mielczarek wrote:
> On Thu, Jan 19, 2012 at 7:49 PM, Robert O'Callahan<rob...@ocallahan.org> wrote:
>> On Fri, Jan 20, 2012 at 12:06 PM, Bobby Holley<bobby...@gmail.com>wrote:
>>

>>> I don't think so. js-ctypes just isn't mean to provided the large-scale
>>> reflection that people use XPCOM for. You have to manually declare each
>>> function (no typelibs), so I don't think C++ support would buy much for
>>> ease-of-conversion.
>>

>> We could have a tool that builds the js-ctypes boilerplate from C++ debug
>> information.
> We'd basically be building something like SWIG[1].

I designed a lot of the API for js-ctypes. I don't think it's a good fit
for wrapping arbitrary C++ code. There are a million reasons for that;
having to declare each function is actually a very small part of the
problem.

We've considered going in the direction roc proposes, but if we're going
to impose a build-time boilerplate-generating step, why not just go
ahead and have that step emit C++ glue code? In the use case we're
talking about, the extension developer is already compiling and shipping
C++ code on all the platforms they care about, right?

And we already have tools that emit C++ (qsgen.py and dombindingsgen.py,
both close relatives of the desired tool).

-j

[Wes Garland]

On 20 January 2012 12:34, Jason Orendorff <joren...@mozilla.com> wrote:

> We've considered going in the direction roc proposes, but if we're going
> to impose a build-time boilerplate-generating step, why not just go ahead
> and have that step emit C++ glue code?

Indeed: In order to overcome the portability hazards of traditional FFI,
this is exactly what we had to do with GPSEE: pick a set of supported
interfaces and emit some C code when we build the platform.

Now, we stopped short of emitting complete JS bindings and instead emitted
code which could be reliably bound to with our FFI, but that distinction is
not really important.

Wes

--
Wesley W. Garland
Director, Product Development
PageMail, Inc.
+1 613 542 2787 x 102

[Zack Weinberg]

On 2012-01-20 9:25 AM, Mike Hommey wrote:
> On Fri, Jan 20, 2012 at 07:46:14AM -0800, Zack Weinberg wrote:
>> On 2012-01-19 11:49 PM, Mike Hommey wrote:
>>>
>>> Do compilers even allow dynamic_cast<> without RTTI being enabled? we
>>> surely don't want RTTI enabled.
>>
>> We do not know that the perf hit of RTTI is greater than the perf
>> hit of not being able to use dynamic_cast where it makes sense, and
>> besides, turning it on is a large step on the road to modern C++.
>
> Compilers are pretty bad at doing things fast with RTTI

The compilers I have available to me on this Mac right now (GCC 4.2 and
some ancient fork of LLVM) aren't especially clever at optimizing
base-to-derived or cross-hierarchy dynamic_cast. However, the code they
generate may still be faster than the code we use now!

All derived-to-base QIs are optimized to static_cast. That is
potentially huge all by itself.

All other (C++-to-C++, no-tearoff, etc) QIs become *at worst* a linear
series of pointer comparisons, versus the linear series of 128-bit block
comparisons that they are now. That is also potentially huge.

> without even
> accounting the size overhead of the RTTI data, which, for the supposed
> benefit, is just too big.

Is it bigger than the data and code that supports QI now?

zw

[Zack Weinberg]

On 2012-01-20 9:01 AM, Joshua Cranmer wrote:
> On 1/19/2012 8:24 PM, Zack Weinberg wrote:
>> (I am pretty seriously tempted to find the time to see what breaks and
>> what the performance delta is if we blow up QueryInterface and replace
>> it with dynamic_cast<>.)
>
> Knowing that LLVM (which is a more modern codebase and uses several C++
> features more heavily) uses its own replacement-for-dynamic-cast

This is news to me; can you point me at the code implementing the
replacement?

... I get very frustrated at the attitude that we're stuck with the
present state of C++ compilers (other than MSVC). If dynamic_cast
doesn't turn out to be a win -- and QI is so inefficient that I have
trouble believing it wouldn't -- the right thing is to *fix
dynamic_cast*, not continue ignoring it. If it's slow on our use cases
it's because nobody over in GCC or LLVM has prioritized making it fast,
and that in turn is because we haven't convinced them it matters.

zw

[Bobby Holley]

Zack, I forked this into another thread. Can you please take this
discussion there?

> ______________________________**_________________
> dev-platform mailing list
> dev-pl...@lists.mozilla.org
> https://lists.mozilla.org/**listinfo/dev-platform<https://lists.mozilla.org/listinfo/dev-platform>
>

[Neil]

Zack Weinberg wrote:

> All derived-to-base QIs are optimized to static_cast. That is
> potentially huge all by itself.

Staticly casting to nsISupports is usually wrong.

--
Warning: May contain traces of nuts.

[Sheppy]

On Jan 19, 3:02 pm, Bobby Holley <bobbyhol...@gmail.com> wrote:

> So. Would this approach alleviate your pain?

I just ask that if this happens, be sure to cc me on the relevant bug,
or email me once the decision is made as to which interfaces will be
put on this list, so I can be sure it's documented. Thanks!

Eric Shepherd
Developer Documentation Lead
Mozilla
http://www.bitstampede.com/

[Robert O'Callahan]

On Fri, Jan 20, 2012 at 11:16 PM, Mike Hommey <m...@glandium.org> wrote:

> On Fri, Jan 20, 2012 at 10:49:23PM +1300, Robert O'Callahan wrote:
> > How would you extend nsIURI without breaking binary compatibility?
>
> It pretty much depends on what changes you want to make, but adding
> virtual functions doesn't require breaking binary compatibility. Adding
> new data members doesn't require breaking binary compatibility as long
> as objects are not created with "new" in the component. There are many
> binary compatible changes possible.
>

Historically the assumption has been that any XPCOM interface could be
implemented by third party components, and since adding a function even at
the end of the vtable would break such implementations, adding a function
anywhere has to be considered an ABI change.

[Robert O'Callahan]

On Sat, Jan 21, 2012 at 8:56 PM, Robert O'Callahan <rob...@ocallahan.org>wrote:

> Historically the assumption has been that any XPCOM interface could be
> implemented by third party components, and since adding a function even at
> the end of the vtable would break such implementations, adding a function
> anywhere has to be considered an ABI change.
>

It may make sense to add an attribute that signals third-party binary
components can't implement an interface, or even extend builtinclass to
mean that. Then we could safely add methods to the end of the vtable of
such classes without breaking ABI. But that would be a new thing.

I'm also not sure that nsIURI could be marked that way, since some
extension protocols mint their own nsIURI subclasses IIRC.

[Robert O'Callahan]

On Sat, Jan 21, 2012 at 7:58 AM, Zack Weinberg <za...@panix.com> wrote:

> ... I get very frustrated at the attitude that we're stuck with the
> present state of C++ compilers (other than MSVC).
>

That's a big "but". If something isn't well-implemented on MSVC then we
can't use it, even if we start up an LLVM team.

[Mike Hommey]

On Sat, Jan 21, 2012 at 08:56:57PM +1300, Robert O'Callahan wrote:
> On Fri, Jan 20, 2012 at 11:16 PM, Mike Hommey <m...@glandium.org> wrote:
>
> > On Fri, Jan 20, 2012 at 10:49:23PM +1300, Robert O'Callahan wrote:
> > > How would you extend nsIURI without breaking binary compatibility?
> >
> > It pretty much depends on what changes you want to make, but adding
> > virtual functions doesn't require breaking binary compatibility. Adding
> > new data members doesn't require breaking binary compatibility as long
> > as objects are not created with "new" in the component. There are many
> > binary compatible changes possible.
> >
>

> Historically the assumption has been that any XPCOM interface could be
> implemented by third party components, and since adding a function even at
> the end of the vtable would break such implementations, adding a function
> anywhere has to be considered an ABI change.

Oh, somehow I missed the "extend" in your message. The problem with such
assumptions is that it blocks the way of those who just need to use
nsIURI methods in C++, for the benefit of those probably much fewer people
who need to derive or implement it. Arguably, it's hard to prevent the
latter.

Mike

[Pavol Mišík]

> So. Would this approach alleviate your pain?
>

> Cheers,
> bholley

Hi.

I think this is the minimalist set of interfaces you need in order to
register XPCOM component and deal with interface changes dynamically.
I’m not sure it is complete.

[Ben Bucksch]

[snip - thanks for the valuable insight and clarification]

On 19.01.2012 21:02, Bobby Holley wrote:
> What if Mozilla promised to support binary
> components over a reasonable timeframe, provided that those extensions used
> only a limited set of mozilla headers (nsISupports, maybe things like
> nsIArray, etc) from C++? Would this make you happy? I'm essentially
> proposing a return to frozen interfaces, except that the frozen set would
> be much, much smaller - just core XPCOM stuff, ideally countable on one
> hand.

No. The old set of frozen interfaces was already much too small.

For example, most non-trivial binary components I saw need to run on
threads. Some even spin their own event loops, so they must run in a
thread and dispatch back to the Mozilla main thread. They all used
GetProxyForObject() and will be hitting the wall real hard in FF12 after
that being removed prematurely.

And this is just one example from the last days where API changes are
painful.

Again, also note that no amount of "new approach"es can help the
existing codebases. Often, the migration is so expensive as to not be an
option, and if a migration is forced, they might well drop Mozilla entirely.
I appreciate your new thinking and ideas, it's great and helpful, but it
must be clear to all participants that any new approach (be it JetPack,
c-types, minimal XPCOM or anything else) is not a solution for current
problems and projects and will only help new projects.

Ben

[Ben Bucksch]

On 23.01.2012 00:00, Ben Bucksch wrote:
> [snip - thanks for the valuable insight and clarification]
>
> On 19.01.2012 21:02, Bobby Holley wrote:
>> only a limited set of mozilla headers (nsISupports, maybe things like
>> nsIArray, etc)
>

> No. The old set of frozen interfaces was already much too small.

To expand a bit:

We were explicitly asked *not* to pass file paths as strings (thanks to
dumb Mac OS), but to use nsIFile always. We obeyed. Similarly, we use
nsIURI, directory service, file streams. To avoid problems with proxies
and similar, we use the necko network library instead of linking an
entirely different network lib, sometimes actively migrated away from a
separate network lib to Mozilla's.

Dropping all these interfaces amounts to a complete rewrite. You can't
ask that of any project and be treated seriously and politely. I hope
that shows the scope of the problem.

[Ben Bucksch]

On 20.01.2012 01:49, Robert O'Callahan wrote:
> "Freeze some interfaces, discourage usage of unfrozen interfaces" was our
> strategy for most of the lifetime of Mozilla, and it didn't work very well.

Yeah. And the reason was mostly that dev has no choice but to use
unfrozen APIs :).

Ben

[Boris Zbarsky]

And when they did have a choice, they used the unfrozen APIs anyway, by
and large. Which is why we have a problem....

-Boris

[Ben Bucksch]

On 20.01.2012 18:34, Jason Orendorff wrote:

I designed a lot of the API for js-ctypes. I don't think it's a good fit for wrapping arbitrary C++ code. There are a million reasons for that; having to declare each function is actually a very small part of the problem.

Can you expand, please?

And: Do you have an idea which direction to go?

For new projects, the goal should be:

1. I take an arbitrary (including arbitrary complex, but sane) C or C++ (both!) library source code, including callbacks, threads, file pointers and whatnot.
   
2. the interface C++ <-> JS (2 way) is then generated automatically using some build tool, and ends up being a comfortable API that directly reflects the C++ API in JS, but with JS expectations about life-cycle, automatic types, errors reported as exceptions, etc.

why not just go ahead and have that step emit C++ glue code?

Sure, why not? As long as it's not too big (download size) and doesn't have to be recompiled for every FF release...

Ben

[Boris Zbarsky]

One more note, because this is important. Even when they were using
used-to-be frozen APIs, keeping them frozen is no longer an option in
some cases. The only way we can freeze nsIDOMNode is if we give up on
evolving the web platform.

Yes, this means that use of all DOM interfaces from binary code is not
really possible as things stand today. The "right" solution to that
would be to have an ABI-stable API (i.e. one not using vtables), probably.

The issue of nsIFile, nsIURI, etc is a bigger problem, because we don't
_have_ to change those interfaces per se. Except that some aspects of
those interfaces are extremely broken by design leading to pervasive
correctness and security bugs due to ease of misuse (nsIURI is a prime
example, actually). So then the question becomes whether we fix those
problems (breaking compat in the process) or just keep accepting those
sources of bugs. I believe the right thing here is to do the former,
with sufficient heads-up and so forth. Then at some point various code
using nsIURI will need to update to the new API.... I don't see how to
make this transition smooth, honestly. :( If you do, I'd love to know.

-Boris

[Mike Hommey]

Nothing prevents you from implementing access to these in js, and call
your js from your C++ if you really need to.

Mike

[Joshua Cranmer]

On 1/23/2012 1:27 AM, Mike Hommey wrote:
> Nothing prevents you from implementing access to these in js, and call
> your js from your C++ if you really need to.

The APIs JS can use for I/O are extremely crippled and binary I/O is
quite painful to code, not to mention that between xpconnect and the
loss of access to non-copying I/O, your performance is also likely to
drop considerably if I/O were a bottleneck.

[Bobby Holley]

You can still do I/O and other intensive operations in your own C++
components. You just can't interact with Gecko from C++.

[Zack Weinberg]

On 2012-01-21 12:01 AM, Robert O'Callahan wrote:
> On Sat, Jan 21, 2012 at 7:58 AM, Zack Weinberg<za...@panix.com> wrote:
>
>> ... I get very frustrated at the attitude that we're stuck with the
>> present state of C++ compilers (other than MSVC).
>>
>
> That's a big "but". If something isn't well-implemented on MSVC then we
> can't use it, even if we start up an LLVM team.

If we get to the point where MSVC is holding us back, maybe we should
dump MSVC.

zw

[Joshua Cranmer]

And therefore stop supporting Windows--gcc certainly can't compile
MSVC's C++ ABI, clang is nowhere near ready enough either, and I am not
aware of any other freely-available compilers for Windows.

[Zack Weinberg]

Well, we are by no means there yet, but why does it *matter* that we are
compatible with MSVC's C++ ABI? Remember that this whole conversation
has been about not letting third party code poke at our C++ guts, so in
that regard I see not using the same C++ ABI as the OS as a bit of a
win, actually.

(I do realize that by the time we get to where MSVC is definitely the
weakest link, we might be better off rewriting in Rust. ;-)

zw

[Joshua Cranmer]

On 1/23/2012 1:05 PM, Zack Weinberg wrote:
> On 2012-01-23 10:43 AM, Joshua Cranmer wrote:
>> On 1/23/2012 12:29 PM, Zack Weinberg wrote:
>>>
>>> If we get to the point where MSVC is holding us back, maybe we should
>>> dump MSVC.
>>
>> And therefore stop supporting Windows--gcc certainly can't compile
>> MSVC's C++ ABI, clang is nowhere near ready enough either, and I am not
>> aware of any other freely-available compilers for Windows.
>
> Well, we are by no means there yet, but why does it *matter* that we
> are compatible with MSVC's C++ ABI? Remember that this whole
> conversation has been about not letting third party code poke at our
> C++ guts, so in that regard I see not using the same C++ ABI as the OS
> as a bit of a win, actually.

Anywhere where we use any C++ APIs: comm-central needs it for MAPI
access. I don't know everywhere where mozilla-central itself uses it,
but parental controls is one. Grep indicates that there are also uses
for network, DnD, audio, file picker, taskbar preview, text thingies,
and DirectX. (This was done by searching for nsRefPtr<I and removing
"obvious" false positives). So yes, we do use this quite a lot.

Oh, and I forgot: we rely on the ABI for XPConnect (there's the
hardcoded assembly files in xpcom/reflect/xptcall).

FWIW, not using the same C++ ABI would actually probably be somewhat
worse, since people would still /try/ to hook into our guts (the
compiler almost certainly wouldn't scream) and fail miserably at runtime
because the callee and the caller are doing two different things.

[Robert O'Callahan]

On Tue, Jan 24, 2012 at 7:29 AM, Zack Weinberg <za...@panix.com> wrote:

> If we get to the point where MSVC is holding us back, maybe we should dump
> MSVC.

In some hypothetical future where we have an open source Windows compiler
that's compatible with the Microsoft ABI and produces code that's just as
good as MSVC PGO, sure.

[Zack Weinberg]

On 2012-01-23 11:19 AM, Joshua Cranmer wrote:
> On 1/23/2012 1:05 PM, Zack Weinberg wrote:
>>
>> Well, we are by no means there yet, but why does it *matter* that we
>> are compatible with MSVC's C++ ABI? Remember that this whole
>> conversation has been about not letting third party code poke at our
>> C++ guts, so in that regard I see not using the same C++ ABI as the OS
>> as a bit of a win, actually.
>
> Anywhere where we use any C++ APIs: comm-central needs it for MAPI
> access. I don't know everywhere where mozilla-central itself uses it,
> but parental controls is one. Grep indicates that there are also uses
> for network, DnD, audio, file picker, taskbar preview, text thingies,
> and DirectX.

Hmph. Okay, I guess we're stuck with that constraint then.

This is getting really far off topic, so this will be the last thing I
say on the subject, but I think it is not in our long run best interest
to devote even as much development effort to Windows (or OSX) as we do
right now, because it is in the OS vendor's interest to make our life as
difficult as possible on both of those platforms, and the resources we
spend on coping with that could be better directed elsewhere.

zw

[Ben Bucksch]

On 20.01.2012 08:32, Brian Smith wrote:
> Forgive my naive question: Why couldn't/shouldn't we freeze things like nsIURI? It is a very simple interface that depends on only core XPCOM types. I don't know about other Necko/Gecko developers, but I wouldn't mind maintaining nsIURI the same way that nsIX509Cert was maintained (see nsIX509Cert2, nsIX509Cert3, etc.) or doing other things to avoid changing the interface unnecessarily.
>
> IMO, things like nsIURI are not a problem as far as maintaining backward compatibility. Things like nsICache*, or NSS, which we are more implementation details that likely to make major changes to, and which aren't really designed for addons to use, are the problematic cases. I think the solution for them is basically to hide "internal" interfaces (at least new ones) in an internal namespace that addons can't access (i.e. stop exporting them from libxul or whatever library they are in, whenever possible).

+1

[Taras Glek]

See https://bugzilla.mozilla.org/show_bug.cgi?id=563742 for better js
file apis.
I have not seen evidence that the browser needs to do non-copying IO for
perf reasons, all of the cases of that could be implemented via
sendfile/splice(ie large/fast downloads).

At the moment ctypes provides no finalization facility so such API would
be leaky. This prevents us from implementing a safe-ish file api.
https://bugzilla.mozilla.org/show_bug.cgi?id=720771 will provide just
enough finalization support in ctypes to do really basic fd/etc cleanup.

Taras

[Jonas Sicking]

On Thu, Jan 19, 2012 at 11:32 PM, Brian Smith <bsm...@mozilla.com> wrote:
> Bobby Holley wrote:
>> I more or less proposed one: nsISupports, and possibly various pieces
>> of glue (like nsIArray) that we don't touch often. No DOM, no nsIURI,
>> no observers, none of that stuff. Just the basics to use XPCOM.

>
> Forgive my naive question: Why couldn't/shouldn't we freeze things like nsIURI? It is a very simple interface that depends on only core XPCOM types. I don't know about other Necko/Gecko developers, but I wouldn't mind maintaining nsIURI the same way that nsIX509Cert was maintained (see nsIX509Cert2, nsIX509Cert3, etc.) or doing other things to avoid changing the interface unnecessarily.
>
> IMO, things like nsIURI are not a problem as far as maintaining backward compatibility. Things like nsICache*, or NSS, which we are more implementation details that likely to make major changes to, and which aren't really designed for addons to use, are the problematic cases. I think the solution for them is basically to hide "internal" interfaces (at least new ones) in an internal namespace that addons can't access (i.e. stop exporting them from libxul or whatever library they are in, whenever possible).

My experience is that there are *very* few interfaces that we won't
want to change every now and then. On nsIURI I'd love to remove all
mutating methods for example. Or at least change them to return a new
object rather than mutating the current one.

This is why I think Bobby's proposal is good. The smaller number of
interfaces that we claim are stable, the more likely it is that we can
honor that claim.

Even nsISupports will change. I have a patch to change the call
signature on windows to not use stdcall, bsmedberg or taras (iirc) has
written patches to change QueryInterface to return the new pointer
rather than using out parameters. So the fewer interfaces that we
claim are stable, the better off we'll be.

/ Jonas

[Blair McBride]

On 24/01/2012 9:48 a.m., Zack Weinberg wrote:
> I think it is not in our long run best interest to devote even as much
> development effort to Windows (or OSX) as we do right now

I'm having trouble grokking this... are you saying we should spend less
development time on Windows, where ~80% of our users are?

- Blair

[Asa Dotzler]

I'm having trouble understanding where you get ~80% from :P

92% of our active daily installs are Windows. Throw in Mac OS X and
you're up to 98%.

If it's desktop versus mobile we're talking about, today about 7-8% of
Web usage comes from phones and tablets. I expect that number will
double in the next two or three years but I don't see it catching up to
desktop/laptop for many, many years.

But even if "mobile" is growing fast, and even if it is the long term
future (I think there's always going to be a split, myself,) failing to
delivering a killer Windows browser over the next few years -- failing
to invest even *more* development efforts into Windows in the near and
medium term, would hugely damage our chances of being successful in
pretty much any of our other efforts.

We have nearly 500 million users. If we let that fall on the floor, our
brand will go to hell and no one will trust us on any future products.
(Would you use a "Netscape Mobile" browser if it came out tomorrow and
was decent? Hell no you wouldn't because that brand is shit because they
turned it into shit.)

The keyboard and mouse based Firefox browser isn't going anywhere any
time soon. We're going to see pretty much all the obvious form factors
succeed and we need to be on all of them to fulfill the Mozilla Mission.
That means continuing to invest, and even increasing our investment in
the keyboard and mouse based Firefox and especially on Windows where
nearly half a billion users are already counting on us.

- A

[db_cooper]

Also, is it not possible that Windows 8 or 9 will become a major mobile (tablets, phones, ARM netbooks etc) OS in the future ...

[Zack Weinberg]

Yes, and I'd say that even if the number of non-Windows users was _zero_
(literally zero, not just rounded down to zero).

It is in our _short term_ best interest to continue investing in
Windows, as Asa points out downthread. It is _not_ in our _long term_
best interest to invest in Windows, or OSX, or even Android, at the
expense of open platforms where could actually cooperate with the OS
vendor rather than be at their mercy; _such as_ B2G, but also the Linux
desktops (which, unlike the proprietary desktops, are actually
innovating right now) or the other open mobile stacks.

Count me as another person doubting that desktop will ever truly go
away, by the way, and also as someone who thinks that we _hurt_ our
attempts to make the Web take over the conventional application space by
refusing to collaborate with existing conventional applications.

zw

[Ehsan Akhgari]

On Tue, Jan 24, 2012 at 5:58 PM, Jonas Sicking <jo...@sicking.cc> wrote:

> Even nsISupports will change. I have a patch to change the call
> signature on windows to not use stdcall, bsmedberg or taras (iirc) has
> written patches to change QueryInterface to return the new pointer
> rather than using out parameters. So the fewer interfaces that we
> claim are stable, the better off we'll be.

FWIW, I was the one who wrote that patch, and we decided that we're not
going to take it.

--
Ehsan
<http://ehsanakhgari.org/>

[Simon Kornblith]

Is there an ultimate conclusion to this discussion? We would like to
know whether we should begin migrating our XPCOM component (which
makes use of no non-trivial XPCOM interfaces besides our own, but has
a somewhat complex OO API) to js-ctypes, or if we should stick it out
with the expectation that we'll eventually be able to target multiple
releases with an XPCOM component again.

Thanks,
Simon

On Jan 19, 10:22 pm, Bobby Holley <bobbyhol...@gmail.com> wrote:
> On Thu, Jan 19, 2012 at 6:24 PM, Zack Weinberg <za...@panix.com> wrote:
> > Hmm, would this preclude switching to a more efficient QueryInterface
> > implementation, as bsmedberg said upthread might be possible if we didn't
> > have to maintain IIDs?
>
> I'm not positive, but at first glance I don't see it as a significant
> problem.
>
> First there's the C++ case: If extensions aren't touching gecko from native
> code, then we don't need any sort of IID mechanism to ensure that they get
> the component they expect (because they're not getting _any_ components).
> They can just use whatever new nsISupports we come up with, so the only
> thing that will require version checking is nsISupports itself (along with
> any trivial helpers like nsIArray). This can be accomplished by a simple
> XPCOM versioning scheme, where we rev to an incompatible version if we ever
> really need to change nsISupports.
>
> Now, for the JS case, extension code would indeed be accessing gecko
> interfaces. But thankfully, that JS code runs using whatever version of
> XPConnect shipped with the given release, so it would automatically get the
> right behavior. It would still throw if it tried to use a method whose
> signature was changed in an incompatible way, but this is true for any
> extension (and throwing a JS exception is much better than crashing).
>
> This is all sort of academic though, because I think that getting rid of
> IIDs is a mammoth task, and unlikely to happen any time soon. According to
> grep, there are 795 lines containing the string "iid" in XPConnect alone,
> so you're in for a real treat if you try to remove them entirely. Perhaps a
> better strategy for experimentation would be to just make IIDs 64 bits?
> This would be less invasive, and probably just require changing a dozen or
> so hard-coded constants. You'd be unlikely to run into collisions with a
> single m-c snapshot, and it'd give us an idea of the performance overhead
> of IID comparisons.
>
> bholley