Add-on SDK Repacks Proposal

[dcm]

Rocketeers!

One thing we have planned to make part of the Add-on SDK story is to have "automatic repacks" for developers using the SDK. What this means is that if you host your add-on, built with the SDK, on AMO (addons.mozilla.org), we would automatically rebuild it everytime we release a new version of the Add-on SDK to maintain compatibility with a new version of Firefox. This would make sure that your add-on was always up-to-date with the latest version of Firefox.

This is still the plan, but for our first attempt we noticed some things that were wrong with the plan as we currently have it and would like to propose a way of getting past these problems and getting to a new repack process.

What Went Wrong

The first problem we faced is that the repacker is the first service to use AMO's brand-new APIs for updating add-ons, and we hadn't tested the interaction between the two sites sufficiently beforehand, so we had to implement fixes for issues we encountered during the repack testing, which slowed down the testing.

Second, our testing was short and not deep enough for what we were trying to accomplish. The repack team didn't get enough testing in themselves, nor did they make it possible for other project participants to test repacks and suggests issues and fixes before the production run of repacking.

Finally, what we are fundamentally trying to do is repackage a derived format without the source code it came from as reference, like decompiling a program, making changes, and then recompiling it. This is something that is hard to do and as it turned out, didn't work very well. We could fix that by including the source code in the XPI. However, including the source code in the XPI would defeat our goal of making the XPI size as small as possible (this is extremely important for mobile versions of Firefox). 

A Proposal

So what can we do about these issues? First of all, the repack team will answer the testing problems by starting testing earlier and sharing the results with everyone. In addition, the use of the AMO APIs is now solved as the calls we are using are now verified and tested. 

However, the issue about the required sources for a clean rebuild is a bit more difficult and needs a new proposal.

Short-term

In the short-term I propose that we only repack add-ons which are built using the Add-on Builder. This is the quickest and safest solution as we have access to all the sources with Builder-based add-ons. The repack process is much simpler too as we don't have to unpack a XPI first. For those who use a local installation of the Add-on SDK to build an add-on, we will document how to repack your add-on and the process for pushing the new one to AMO. In addition, you can move your add-on to the Add-on Builder and take advantage of the automatic repacks if you'd like, but do keep in mind that the Add-on Builder is still in beta at this point.

Long-term

The long-term solutions will need to be studied and debated a bit more. One option we have is to create a "source package" that can be pushed to AMO for use in repacking. However, this would mean that AMO would have to store these source packages and generate XPIs from them in addition to taking on responsibility for repacking when a new SDK is released. Obviously this would involve adding a great deal of functionality to AMO, as well as tooling up the services side to store these new, larger packages.

Another option is to land parts of the Add-on SDK into Firefox itself, including API implementations and the module loader. If an API already in Firefox needs updating to maintain compatibility with changes to Firefox, we won't need to repack add-ons that use that API, as the updates can be made to that new version of the browser rather than each individual add-on. Doing this has other benefits as well. For example, it may help reduce XPI sizes tremendously. However, this too would take a bit of time to implement and is thus a long-term solution.

At this point we think that the short-term solution of just repacking Add-on Builder-based add-ons would be the best approach. We would also provide documentation describing the simplest approach for repacking add-ons along with best practices and troubleshooting. In addition, we would look into how best to notify authors when their add-ons need to be repacked manually.

Still, we also want to know what you think about this as well as the long-term ideas. Please feel free to leave your comments or suggestions about this proposal.

Thanks!

Dave

[KWierso]

There were people saying that due to the repack, their addons got removed from the review queue and put at the end of the line. Could anyone verify this? (All of my AMO-hosted Jetpack addons use recent Github clones, so they didn't get repacked.)

If repacks are pulling addons out of the queue, is there anything we or AMO (or the repacker) can do to stop that? Waiting for a few weeks for a review, only to be pulled out of the queue to start over with a new multi-week estimate would really suck.

[Erik Vold]

This is probably due to https://bugzilla.mozilla.org/show_bug.cgi?id=656122

Erik

On Tue, Oct 4, 2011 at 14:02, KWierso <kwi...@gmail.com> wrote:

There were people saying that due to the repack, their addons got removed from the review queue and put at the end of the line. Could anyone verify this? (All of my AMO-hosted Jetpack addons use recent Github clones, so they didn't get repacked.)

If repacks are pulling addons out of the queue, is there anything we or AMO (or the repacker) can do to stop that? Waiting for a few weeks for a review, only to be pulled out of the queue to start over with a new multi-week estimate would really suck.

--
Erik Vergobbi Vold

Email: erik...@gmail.com
Website: http://erikvold.com/

[Drugoy]

I think it's the best solution.

dcm:

[Jorge Villalobos]

Note https://bugzilla.mozilla.org/show_bug.cgi?id=689635, that I marked as a dependency.

- Jorge

--
You received this message because you are subscribed to the Google Groups "mozilla-labs-jetpack" group.
To post to this group, send email to mozilla-la...@googlegroups.com.
To unsubscribe from this group, send email to mozilla-labs-jet...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/mozilla-labs-jetpack?hl=en.

[James Burke]

On Oct 4, 1:47 pm, dcm <masons...@gmail.com> wrote:
> Finally, what we are fundamentally trying to do is repackage a derived
> format without the source code it came from as reference, like decompiling a
> program, making changes, and then recompiling it. This is something that is
> hard to do and as it turned out, didn't work very well. We could fix that by
> including the source code in the XPI. However, including the source code in
> the XPI would defeat our goal of making the XPI size as small as possible
> (this is extremely important for mobile versions of Firefox).

I'm curious to know what sort of issues you hit with the repackaging.
Was it missing modules, the parsing of dependencies were harder to do?
My interest is mostly from a module analysis point of view. I would
like to learn from jetpack's real world experience. I can follow up
off-list if this is too much noise for this thread.

James