No more binary components in extensions

[Benjamin Smedberg]

(Followup questions or comments to mozilla.dev.extensions only, please.)

With the landing of bug 1159737, I have removed support for binary XPCOM
components in extensions. This is planned to ride the Firefox 40 train.

This change is necessary because we no longer expose or intend to expose
a binary-stable API to XPCOM. Most addons have already moved away from
binary XPCOM components, but those that haven't are a source of
instability around Firefox releases.

Extension authors that need to use native binaries are encouraged to do
so using the addon SDK "system/child_process" pipe mechanism:
https://developer.mozilla.org/en-US/Add-ons/SDK/Low-Level_APIs/system_child_process

If this is not sufficient, JS-ctypes may be an alternative mechanism to
use shared libraries, but this API is much more fragile and it's easy to
write unsafe code.

I will be updating MDN documentation and removing or archiving old
documentation about binary XPCOM components in the next few weeks.

--BDS

[Joshua Cranmer 🐧]

On 5/4/2015 10:03 AM, Benjamin Smedberg wrote:
> (Followup questions or comments to mozilla.dev.extensions only, please.)
>
> With the landing of bug 1159737, I have removed support for binary
> XPCOM components in extensions. This is planned to ride the Firefox 40
> train.
>
> This change is necessary because we no longer expose or intend to
> expose a binary-stable API to XPCOM. Most addons have already moved
> away from binary XPCOM components, but those that haven't are a source
> of instability around Firefox releases.

Is it not possible to let applications choose whether or not they want
to support binary XPCOM components in extensions? It's really dismaying
that such a major change was made with no discussion or prior
notification that includes developers of major binary addons, most
notably Lightning, which has at least 1M users...

--
Joshua Cranmer
Thunderbird and DXR developer
Source code archæologist

[daniel....@gmail.com]

Le lundi 4 mai 2015 17:17:16 UTC+2, Joshua Cranmer 🐧 a écrit :

> Is it not possible to let applications choose whether or not they want
> to support binary XPCOM components in extensions? It's really dismaying
> that such a major change was made with no discussion or prior
> notification that includes developers of major binary addons, most
> notably Lightning, which has at least 1M users...

Currently contracting for Postbox, I'm also interested in the future of Lightning, a major add-on with lots of users relying on binary components.
And honestly, 4 days from bug creation to feature removal w/o prior information/discussion or time to let users adapt code and strategy is, as said Joshua, pretty surprising.

</Daniel>

[Eric Shepherd]

Joshua Cranmer 🐧 wrote:
>> With the landing of bug 1159737, I have removed support for binary
>> XPCOM components in extensions. This is planned to ride the Firefox
>> 40 train.
>>
>> This change is necessary because we no longer expose or intend to
>> expose a binary-stable API to XPCOM. Most addons have already moved
>> away from binary XPCOM components, but those that haven't are a
>> source of instability around Firefox releases.
>
> Is it not possible to let applications choose whether or not they want
> to support binary XPCOM components in extensions? It's really
> dismaying that such a major change was made with no discussion or
> prior notification that includes developers of major binary addons,
> most notably Lightning, which has at least 1M users...

Wow, that's a really dramatic change that's come very abruptly and with
little warning. The docs team is going to get hammered for this one; we
tend to be the ones people complain to when the stuff our docs cover no
longer works. This is also going to mean a massive overhaul to our
add-ons material.

Plus it will kill off a few genuinely valuable extensions...

--

Eric Shepherd
Senior Technical Writer
Mozilla <https://www.mozilla.org/>
Blog: http://www.bitstampede.com/
Twitter: http://twitter.com/sheppy

[Eric Shepherd]

Benjamin Smedberg wrote:
> I will be updating MDN documentation and removing or archiving old
> documentation about binary XPCOM components in the next few weeks.

Please ping me before outright deleting anything; I'd like to be sure
we're able to continue to support people embedding Gecko or targeting
projects other than Firefox. Thanks!

[Benjamin Smedberg]

On 5/4/2015 11:17 AM, Joshua Cranmer 🐧 wrote:
> On 5/4/2015 10:03 AM, Benjamin Smedberg wrote:
>> (Followup questions or comments to mozilla.dev.extensions only, please.)
>>
>> With the landing of bug 1159737, I have removed support for binary
>> XPCOM components in extensions. This is planned to ride the Firefox
>> 40 train.
>>
>> This change is necessary because we no longer expose or intend to
>> expose a binary-stable API to XPCOM. Most addons have already moved
>> away from binary XPCOM components, but those that haven't are a
>> source of instability around Firefox releases.
>
> Is it not possible to let applications choose whether or not they want
> to support binary XPCOM components in extensions?

Is it possible right now? Probably yes, with a little work to get the
flag from the app config down into XPCOM. I'd review the patch, although
I'm not going to write it.

But be aware that this is part of a longer-term strategy to start
removing XPCOM from the critical path of anything in gecko. As we start
adopting rust/servo components, we're going to progressively make XPCOM
less valuable, to the point where we can eventually remove the XPCOM
component system from Firefox completely. So mainly if you do this you'd
be buying some time.

> It's really dismaying that such a major change was made with no
> discussion or prior notification that includes developers of major
> binary addons, most notably Lightning, which has at least 1M users...
>

I'm sorry that this came as so much of a surprise, but I still think
it's the right call for our codebase, and do not intend to delay it
further. We have been talking on this list about binary XPCOM being on
the way out for more than a year. See prior links:

https://groups.google.com/forum/#!msg/mozilla.dev.extensions/LEpxE1zs9eA/Vr7WijR54A8J
https://groups.google.com/d/msg/mozilla.dev.extensions/UPBgQ20z2FA/tN1w7900DPAJ
https://groups.google.com/d/msg/mozilla.dev.extensions/tP4fsGG6t2Q/CKL_OlvENAkJ

--BDS

[Joshua Cranmer 🐧]

On 5/4/2015 12:12 PM, Benjamin Smedberg wrote:
>
>
> On 5/4/2015 11:17 AM, Joshua Cranmer 🐧 wrote:
>> On 5/4/2015 10:03 AM, Benjamin Smedberg wrote:
>>> (Followup questions or comments to mozilla.dev.extensions only,
>>> please.)
>>>
>>> With the landing of bug 1159737, I have removed support for binary
>>> XPCOM components in extensions. This is planned to ride the Firefox
>>> 40 train.
>>>
>>> This change is necessary because we no longer expose or intend to
>>> expose a binary-stable API to XPCOM. Most addons have already moved
>>> away from binary XPCOM components, but those that haven't are a
>>> source of instability around Firefox releases.
>>
>> Is it not possible to let applications choose whether or not they
>> want to support binary XPCOM components in extensions?
>
> Is it possible right now? Probably yes, with a little work to get the
> flag from the app config down into XPCOM. I'd review the patch,
> although I'm not going to write it.

I was actually thinking of a configure-level flag, which would (if my
understanding of the patch is correct) require only a single ifdef
within the file.

> But be aware that this is part of a longer-term strategy to start
> removing XPCOM from the critical path of anything in gecko. As we
> start adopting rust/servo components, we're going to progressively
> make XPCOM less valuable, to the point where we can eventually remove
> the XPCOM component system from Firefox completely. So mainly if you
> do this you'd be buying some time.

I've long held that XPCOM and binary components are going to die in the
future, but my estimated time frames range in the region of 10 years or
so. The two most important binary extensions in TB would probably go
aware within a year, one being integrated in core and the other moving
its core to JS. Actually, Lightning can already be implemented in JS,
but it constitutes a major performance regression at present (something
about calculating timezones being painful in JS).

>
>> It's really dismaying that such a major change was made with no
>> discussion or prior notification that includes developers of major
>> binary addons, most notably Lightning, which has at least 1M users...
>>
> I'm sorry that this came as so much of a surprise, but I still think
> it's the right call for our codebase, and do not intend to delay it
> further. We have been talking on this list about binary XPCOM being on
> the way out for more than a year.

There is a distinction, to me at least, between aspirationally saying
"we'd like to do this in the future" and giving a heads up "we want to
make this change in a few days, any objections?" Particularly since this
project has a very good history of failing to make progress on many of
its aspirational "we'd like to do this in the future" changes, which
makes it hard to gauge the seriousness of particular requests.

[Philipp Kewisch]

On 5/4/15 9:51 PM, Joshua Cranmer 🐧 wrote:
>>> Is it not possible to let applications choose whether or not they
>>> want to support binary XPCOM components in extensions?
>>
>> Is it possible right now? Probably yes, with a little work to get the
>> flag from the app config down into XPCOM. I'd review the patch,
>> although I'm not going to write it.
>
> I was actually thinking of a configure-level flag, which would (if my
> understanding of the patch is correct) require only a single ifdef
> within the file.

Yes, as the Lightning owner I would appreciate if this could be done on
a configure-level flag.

Firefox uses Intent to deprecate/Intent to unship messages for lots of
things, I think its a bit unfair to not do it for this feature. I know
the HTTP discussion has exploded and that might not make it lucrative to
get into discussion before actually writing the patch, but it is still a
pretty big feature.

I've ported our binary component to javascript, but performance is not
up to par. Code that takes a few hundred milliseconds with the binary
component take about 6 seconds with the javascript port. I've went
through the javascript with a js engine expert, but they couldn't
identify anything but micro optimzations.

I've also attempted to port the component using jsctypes, but there is a
blocking bug 1059370 in the way and the person who is capable of
debugging and fixing the issue unfortunately doesn't have time.

The only quick fix I see is to compile the binary component into
Thunderbird, but then again this would defeat the purpose of an
extension and also require Seamonkey and other Products to do the same.

I would be less uncomfortable with this if we had more time to work on
alternatives and our options weren't so little. I understand that
getting rid of xpcom is a long term goal, but as it takes time to remove
xpcom it will also take time to rewrite a project based on xpcom to be a
project independent of xpcom. Especially in a community project with
just a handful of voluntary contributors. Cutting of support for binary
components at this end makes it so much harder to transition.

Philipp

[com...@gmail.com]

On Monday, May 4, 2015 at 7:27:23 PM UTC-4, Philipp Kewisch wrote:
> I've ported our binary component to javascript, but performance is not
> up to par. Code that takes a few hundred milliseconds with the binary
> component take about 6 seconds with the javascript port. I've went
> through the javascript with a js engine expert, but they couldn't
> identify anything but micro optimzations.

Out of curiosity, have you tried Emscripten?

[Philipp Kewisch]

I don't quite recall the result of testing it, but I know I considered
it back then. I believe there were some issues that didn't allow me to
actually compile the library with emscripten. I could give that another
try, but moving to emscripten is still a major undertaking.

Philipp

[Benjamin Smedberg]

On 5/4/2015 6:53 PM, Philipp Kewisch wrote:
>
> So to be clear, this is just removed/disabled for Firefox? Other
> projects like Thunderbird are not affected?

Followups to dev-extensions please!

That is incorrect. This is currently disabled for all gecko applications.

B2G has asked that binary component support be restored for
distribution/bundles only, and that is being done in bug 1161212.

As I said on the other list, I will review a patch which makes this
configurable for Thunderbird, but I don't plan to write that patch myself.

--BDS

[Philipp Kewisch]

Sorry about that, I missed that when first reading. I'm subscribed to
m.d.extensions now. I'm sure we can put a little time aside to write
that patch, given its fairly easy to disable it it should probably be
fairly easy to make it configurable.

Thanks!
Philipp

[Doug Turner]

One thing I should point out is that binary components in B2G are NOT user installable. Instead, binaries components are used by companies building FirefoxOS devices.

For example, Qualcomm has some special implementation for Geolocation and the radio interface layer (RIL). When a company wants to build a FirefoxOS device on Qualcomm hardware, Qualcomm hands them a bunch of binaries. These binaries obviously aren’t compiled into LIBXUL and thus we need XPCOM to continue looking for and loading binary components.

I hope this helps.

[Norton Browser Extensions]

I wish to add to the call for "buying time" for adoption of this change (and not just via a config option in FF40). My group authors binary FF extensions included with products that have millions of installs. We knew the context that XPCOM would be going away sometime this year, but were surprised by the lead-time of the announcement.

Based on our experience with refactoring work done for other browser extensions, the proposed timeframe for this change to come with FF40 is challenging for us to deliver compatible extensions. We risk disturbing our user experience and disrupt the features they've come to expect with our extensions.

Benjamin, can you please consider a delay in committing the changes - perhaps one major FF release version later?

Regards,

Matt Powers
Symantec Corporation

[donbugd...@gmail.com]

Goodbye Firefox, Hello Chrome.

[Kostas]

So, is there any future in JS-CTypes?
Because I have just written an addon using that technology and I hate saying
goodbye to all the work done.
Your documentation didn't say anything about you ditching this way of
writing addons in the future.
And in your announcement that the new FF will support Chrome extensions (on
which I have no objection by the way) you did not say anything on what the
future of JS-CTypes will be.
Will it be continued to be supported or it will die out?

_______________________________________________
dev-extensions mailing list
dev-ext...@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-extensions

[David Rajchenbach-Teller]

So far, we have no plans to ditch js-ctypes. I have heard of plans for a
simpler, safer & faster js-ctypes 2, based on WebASM and TypedObjects,
but I don't think anybody is working on it right now, as there are still
a number of technological blocker.

Cheers,
David

On 30/09/15 09:44, Kostas wrote:
> So, is there any future in JS-CTypes?
> Because I have just written an addon using that technology and I hate saying
> goodbye to all the work done.
> Your documentation didn't say anything about you ditching this way of
> writing addons in the future.
> And in your announcement that the new FF will support Chrome extensions (on
> which I have no objection by the way) you did not say anything on what the
> future of JS-CTypes will be.
> Will it be continued to be supported or it will die out?


--

David Rajchenbach-Teller, PhD
Performance Team, Mozilla

[Benjamin Smedberg]

On 9/30/2015 3:44 AM, Kostas wrote:
> So, is there any future in JS-CTypes?
> Because I have just written an addon using that technology and I hate saying
> goodbye to all the work done.
> Your documentation didn't say anything about you ditching this way of
> writing addons in the future.
> And in your announcement that the new FF will support Chrome extensions (on
> which I have no objection by the way) you did not say anything on what the
> future of JS-CTypes will be.
> Will it be continued to be supported or it will die out?

We don't have a plan to explicitly keep or kill js-ctypes right now.
ctypes code is a footgun because of pointer/GC hazards, and so I
recommend other solutions if at all possible. In particular, the APIs
which communicate with an external binary using pipes are going to be
far safer and probably easier to code:

system/child_process is the Firefox SDK API for launching and
communicating with a bundled executable.
Chrome's equivalent is the nativeMessaging API, which interacts with an
executable which is already installed on the system (not installed via
the addon). Both of these systems communicate with the native code using
pipes.

--BDS

[Kostas]

Thanks for the info!

I have found some related material:
https://developer.mozilla.org/en-US/Add-ons/SDK/Low-Level_APIs/system_child_
process

I guess (from what you have said) that your "system/child_process" API will
continue to have the advantage (over Chrome addons) that we can package the
native executable in our XPI extension and retrieve its extracted path in
order to start it (as I used to do with DLL files and JS-CTypes).
That is great. :)
Thanks for keeping the best (although not so well documented - I think...)
parts of your API alive.

-----Original Message-----
From: Benjamin Smedberg [mailto:benj...@smedbergs.us]
Sent: Wednesday, September 30, 2015 6:28 PM
To: Kostas; dev-ext...@lists.mozilla.org
Subject: Re: No more binary components in extensions