API Compatibility in Mozilla 2+
Benjamin Smedberg
meeting in June, as well as presentations at the Mozilla summit. Please
direct questions and followup to the mozilla.dev.platform group.
In order to improve performance and maintain the Mozilla codebase, we have
decided that now is the right time to make more aggresive changes which
break older code in ways which are not backwards-compatible. This is already
evident in the changes to XPCOM registration, and will be continued in the
future. As part of this decision, we have switched the Gecko version number
to "2.0" for the next major release (Firefox 4).
The new rules for interface compatibility are written here:
https://developer.mozilla.org/En/Developer_Guide/Interface_Compatibility
There are different "layers" of compatibility:
* Web content APIs: very stable
* Jetpack APIs (when the Jetpack SDK reaches version 1.0). Stable.
Compatibility will preserved if at all possible.
* Scriptable APIs (JS/XUL/XBL): May change between major versions.
Developers should carefully consider the impact of changes, however, and
design APIs to be backward- and forward-compatible when appropriate.
* Binary APIs (XPCOM interfaces): extensions with XPCOM binaries will need
to be recompiled with each major version of the platform (2.0, 2.1, etc).
Interfaces formerly marked with @status FROZEN are no longer considered
frozen and may change in the future.
We have been preparing for this change for a while: the ctypes and jetpack
projects are designed so that developers can transition from unstable XPCOM
interfaces to stable APIs.
Note that the change to "Mozilla 2" does not mean that we are switching to
GCed XPCOM objects, nor are we switching to C++ exceptions. This is not
"opening the floodgates" to make lots of arbitrary changes to APIs just
because they are now unfrozen. All changes should be carefully considered,
weighing the potential benefits against the costs both within the Mozilla
codebase and to third-party code such as extensions.
We are not planning on re-freezing XPCOM interfaces in the future.
--BDS
Ben Bucksch
> * Scriptable APIs (JS/XUL/XBL): May change between major versions.
> Developers should carefully consider the impact of changes, however,
> and design APIs to be backward- and forward-compatible when appropriate.
> * Binary APIs (XPCOM interfaces): extensions with XPCOM binaries will
> need to be recompiled with each major version of the platform (2.0,
> 2.1, etc). Interfaces formerly marked with @status FROZEN are no
> longer considered frozen and may change in the future.
>
So, are you saying that you are dropping the whole concept of frozen
XPCOM APIs?
Allowing frozen interfaces to change exactly once, namely before Gecko
2.0 release, is one thing. Not considering them frozen at all anymore in
the future and continue to break them in every major release is another.
For me, that's a major step backwards for the platform.
> We have been preparing for this change for a while: the ctypes and
> jetpack projects are designed so that developers can transition from
> unstable XPCOM interfaces to stable APIs.
jetpack is not even remotely usable for 95% of the current extensions,
and won't be sufficient for all of them, neither will ctypes. That's
basically forcing a rewrite of all C++ components, and doesn't help
those extensions which need third party code.
We need stable C++ APIs. For embedders, xulrunner applications with
binary components, and similar.
This post basically means that the entire Mozilla platform is dropped
for anybody using C++. I don't think that's acceptable.
Ben
Benjamin Smedberg
> So, are you saying that you are dropping the whole concept of frozen XPCOM
> APIs?
Yes.
> Allowing frozen interfaces to change exactly once, namely before Gecko 2.0
> release, is one thing. Not considering them frozen at all anymore in the
> future and continue to break them in every major release is another.
>
> For me, that's a major step backwards for the platform.
That may be, but we cannot either make all the changes we want right now,
nor do we want flag day in which all sorts of changes are crammed in at
once. That's not good engineering. The future of our platform is in the JS
APIs, not the binary APIs. We are focusing on making both the stable JS APIs
(content and jetpack) more full-featured, as well as making the less-stable
ones (XUL/XBL/chrome) as full-featured as possible.
In order to do this and still be productive as a project, we need to shed
some weight of binary compatibility that has been slowing us done and making
our code less readable and less performant for years.
> jetpack is not even remotely usable for 95% of the current extensions, and
> won't be sufficient for all of them, neither will ctypes. That's basically
> forcing a rewrite of all C++ components, and doesn't help those extensions
> which need third party code.
It is true that jetpack will not serve everyone's needs. But jetpack has the
builtin ability to write a module to do the very specific thing that isn't
builtin. If that module is generally useful, they will try to accept and
maintain it long-term.
ctypes won't solve everyone's problem either, that's true. But our
experience has shown that it can probably solve 90% of the problem of
interfacing with third-party code.
> This post basically means that the entire Mozilla platform is dropped for
> anybody using C++. I don't think that's acceptable.
No! We are not dropping C++/binary support. We will continue to support this
case for a long while: we're just saying that if you want to use these APIs,
you will need to recompile for each major version.
--BDS
Joshua Cranmer
> Allowing frozen interfaces to change exactly once, namely before Gecko
> 2.0 release, is one thing. Not considering them frozen at all anymore
> in the future and continue to break them in every major release is
> another.
>
> For me, that's a major step backwards for the platform.
future versions, nor do they necessarily even guarantee source
compatibility. That doesn't mean that interfaces are changed in breaking
ways in every major release just because it's not frozen. Given the
number and usefulness (at least in the extensions I write) of non-frozen
interfaces, I don't think the change is as devastating in practice as
you assume it to be.
Ben Bucksch
> We will continue to support [C++ binary components] for a long while:
> recompile for each major version.
Before I answer your post, I need clarification for the last sentence:
Given that you suggest to move to JS components, are you guaranteeing
that XPCOM APIs that are currently frozen will continue to be "frozen"
towards JavaScript, i.e. not change in ways that breaks JS code? I.e.
only the C++ ABI changes? I.e. XPCOM interfaces are not removed, only added?
Shawn Wilsher
> Given that you suggest to move to JS components, are you guaranteeing
> that XPCOM APIs that are currently frozen will continue to be "frozen"
> towards JavaScript, i.e. not change in ways that breaks JS code? I.e.
> only the C++ ABI changes? I.e. XPCOM interfaces are not removed, only
> added?
don't think we'd want to mark them as "frozen" either (for JS
consumers). The point here is to not tie our hands to bad decisions of
the past.
Cheers,
Shaw
Ben Bucksch
> In general we don't try to needlessly break APIs for JavaScript, but I
> don't think we'd want to mark them as "frozen" either (for JS consumers).
Given the mindless breakage we've seen in the past before the frozen
APIs, the frozen XPCOM IDLs were a major boon for the platform as a
whole, including for extensions and XULRunner. We do need some rules.
> The point here is to not tie our hands to bad decisions of the past.
With Gecko 2.0, you now have the chance to fix them. That's the point of
a major tookit/library release.
Afterward, you have to comment-mark and prepare the IDL with your
intended change, maybe even create an additional, new IDL that looks
like you want to, and then can make that change for Gecko 3.0 in 5 years.
E.g. you realize that nsIDownloader is inherently broken, you make an
nsIFileTransfer and design it properly, and you @deprecate
nsIDownloader, but support both APIs until the next .0, at which point
you remove nsIDownloader and support only nsIFileTransfer. That gives
apps a few years to transition, and allows you to progress and improve
gradually. In fact, that's precisely how IDL is supposed to be used.
If you only guarantee JS API stability, it's even easier: You can add
new functions to the existing IDL, and @deprecate the old functions (but
still support them until the next .0).
Yes, keeping APIs working is a bit of work. Compared to the work that
all consumers of the API need to do, it's minor, though. I am missing
the understanding of this basic relation.
Ben
Shawn Wilsher
> Given the mindless breakage we've seen in the past before the frozen
> APIs, the frozen XPCOM IDLs were a major boon for the platform as a
> whole, including for extensions and XULRunner. We do need some rules.
behavior common more recently on non-frozen APIs?
> With Gecko 2.0, you now have the chance to fix them. That's the point of
> a major tookit/library release.
>
> Afterward, you have to comment-mark and prepare the IDL with your
> intended change, maybe even create an additional, new IDL that looks
> like you want to, and then can make that change for Gecko 3.0 in 5 years.
And tie our hands for another five years? Please re-read bsmedberg's
post about why we don't want to freeze interfaces.
> If you only guarantee JS API stability, it's even easier: You can add
> new functions to the existing IDL, and @deprecate the old functions (but
> still support them until the next .0).
We tend to do this when we can. For example, Storage will deprecate
something and then keep it around for two or so versions and then drop it.
> Yes, keeping APIs working is a bit of work. Compared to the work that
> all consumers of the API need to do, it's minor, though. I am missing
> the understanding of this basic relation.
It's not always minor, and that's the problem. Again, bsmedberg covered
why we don't want to freeze in another post in this thread.
Cheers,
Shawn
Pavol Mišík
change?
What about to change interface design that every interface will be
inherited from nsISupports? Current issue is that some interfaces are
inherited from other interfaces. If you change parent interface (add or
remove method) you indirectly change child interface for binary
extension. Unfortunately this kind of changes comes without change of
iid of child interfaces. If there were a rule that interfaces are
inherited from nsISupport then this kind of problems would not happen.
It would simplify live for extension developers.
I know that this can affect performance therefore there is a question if
this change would affect performance significantly or not?
pm-
Benjamin Smedberg
> Are you going to preserve calling convention of methods or this also can
> change?
I think it *could* change: would making a change improve performance?
> What about to change interface design that every interface will be inherited
> from nsISupports? Current issue is that some interfaces are inherited from
> other interfaces. If you change parent interface (add or remove method) you
> indirectly change child interface for binary extension. Unfortunately this
> kind of changes comes without change of iid of child interfaces. If there
> were a rule that interfaces are inherited from nsISupport then this kind of
> problems would not happen. It would simplify live for extension developers.
> I know that this can affect performance therefore there is a question if
> this change would affect performance significantly or not?
We are certainly not going to make a rule that all interfaces should inherit
from nsISupports. That would just bloat classes with extra vtables
(sometimes *many* extra vtables). Since you will have to recompile for each
major version anyway, I don't think the child-interface problem will be a
serious issue.
--BDS
Pavol Mišík
> On 8/10/10 3:48 AM, Pavol Mišík wrote:
>
>> Are you going to preserve calling convention of methods or this also can
>> change?
>
> I think it *could* change: would making a change improve performance?
I think __stdcall is best choice for interoperability and performance.
pm-
Benjamin Smedberg
>> I think it *could* change: would making a change improve performance?
> I don't think.
> I think __stdcall is best choice for interoperability and performance.
Do you have any data on this? I know that I'd kinda like to rid our codebase
of the needless NS_IMETHOD/NS_IMETHODIMP macros and just use "virtual", to
make the code more C++-like and readable, but I also don't have any data on
it and it's very low on my list of things to do.
--BDS
Pavol Mišík
> Do you have any data on this? I know that I'd kinda like to rid our
> codebase of the needless NS_IMETHOD/NS_IMETHODIMP macros and just use
> "virtual", to make the code more C++-like and readable, but I also don't
> have any data on it and it's very low on my list of things to do.
>
> --BDS
that Visual C++ uses __thiscall calling convention for virtual member of
class where *this* pointer is stored in ecx register. I'm not sure if
GCC support it as Visual C++ does.
http://en.wikipedia.org/wiki/X86_calling_conventions#thiscall
If you change it then crash dump analysis can get more difficult on
Windows because you couldn't be able to get all arguments of call. With
current stdcall calling convention you have pointer to this passed as
first argument on stack so you know with which instance of object you
right work.
The difference is just for x86. On x86-64(amd64) there is just one
calling convention.
pm-
Benjamin Smedberg
> If you get rid of NS_IMETHOD/NS_IMETHODIMP then the difference would be that
> Visual C++ uses __thiscall calling convention for virtual member of class
> where *this* pointer is stored in ecx register. I'm not sure if GCC support
Yes, that is true, and it may perform better.
> it as Visual C++ does.
> http://en.wikipedia.org/wiki/X86_calling_conventions#thiscall
No, GCC uses cdecl by default, where "this" is the first argument.
> If you change it then crash dump analysis can get more difficult on Windows
> because you couldn't be able to get all arguments of call. With current
I don't think this is a big problem in practice, especially since the
crash-stats tools don't use the arguments at all.
--BDS
Justin Dolske
> The new rules for interface compatibility are written here:
>
> https://developer.mozilla.org/En/Developer_Guide/Interface_Compatibility
Just to be clear: the term "major release" isn't really defined... I
assume the intention is that this would include releases like Firefox
3.5 and 3.6 (Gecko 1.9.1, 1.9.2)?
Justin
Benjamin Smedberg
Yes. I don't think we've decided whether the gecko after gecko 2 will be
numbered gecko 3 or gecko 2.1, but whatever we call it, that's the intent.
--BDS
Pavol Mišík
> No, GCC uses cdecl by default, where "this" is the first argument.
>
have the guaranteed application binary interface (ABI) for components.
Because there are differences in compiler you need to force compiler to
fulfils ABI.
Because of compiler differences I think you will always need those
macros. There is also question if gcc have an option to declare method
with __thiscall calling convention. If not then this can be good reason
to not to change it.
A change of calling convention will also require changes in corner stone
of xpcom – xpcom/reflect/xptcall.
pm-
Georg Maaß
> Yes. I don't think we've decided whether the gecko after gecko 2 will be
> numbered gecko 3 or gecko 2.1, but whatever we call it, that's the intent.
So the decision is simple: migrate the remaining toolkit based
applications to webkit applications and let FF die alone.
Boris Zbarsky
> So the decision is simple: migrate the remaining toolkit based
> applications to webkit applications and let FF die alone.
How does that help? Webkit promises neither source nor binary compat
between webkit versions.
-Boris