Jetpack-sdk now being tested on mozilla-central
John O'Duinn
mozilla-central for mac and linux - tracked in bug#570251. Getting it
working windows is being tracked in bug#602391. While doing this, we
also landed some other misc changes:
* bug#517475: Now that unittests are running on debug builds on *test*
machines, we disabled running bloat log steps on debug builds on *build*
machines. This will slightly improve build time for these builders.
* bug#570248: adds run_jetpack.sh to tools repo for running in bug 570251
* bug#603012: make env a required arg in UnittestPackagedBuildFactory
* bug#506404: Fix case where test failures can turn a build red for nanojit
* bug#595187: Fix for zombie buildbot master processes left around after
'make stop'
* bug#599795: 'debug test mochitest-other' fails to unpack dmg on 10.5.8
These changes were tested in staging before landing, required only
reconfigs of our production/try masters. If you see any unexpected
changes, please let us know.
***NOTE: This did not require a downtime. All systems are running
normally, and all jobs were being processed normally before/during/after
this reconfig.
Benjamin Smedberg
> At 12:45 PDT today, we landed fixes to enable jetpack-sdk testing on
> mozilla-central for mac and linux - tracked in bug#570251.
What does this mean in practice? Where are these test results reported? Do
we have any policy for what happens if a mozilla-central change causes the
jetpack tests to fail? I don't *think* that we want that to close the tree
(yet).
--BDS
Mike Beltzner
I don't think we want to close the tree, but I think that the JetPack team needs to be involved and we all need to quickly evaluate what caused JetPack to break. Without these tests, f.e., we missed that GC Compartments breaks JetPack.
I'd be interested to hear other people's thoughts about this.
cheers,
mike
johnjbarton
I think you ought to apply the same logic to the Firebug nightly tests,
for the same reasons.
jjb
>
> cheers,
> mike
Rob Campbell
On 2010-10-19, at 10:18 , Mike Beltzner wrote:
> On 2010-10-19, at 8:48 AM, Benjamin Smedberg wrote:
>
> I don't think we want to close the tree, but I think that the JetPack team needs to be involved and we all need to quickly evaluate what caused JetPack to break. Without these tests, f.e., we missed that GC Compartments breaks JetPack.
>
> I'd be interested to hear other people's thoughts about this.
Being able to see the results would be a fine first step. I was surprised to find that some jetpacks were broken the other day (and some still are) as were the Jetpack team, I'm sure.
Knowing about this in a more timely fashion might have saved some regression-range hunting and confusion.
Should we have backed-out Compartments for this? I'm not sure. The Compartments devs might have benefitted from the additional test coverage before landing if these tests were available as a set on Try server.
I guess we're going to need to figure out where Jetpack breakage fits into our landing processes. I would say that after feature freeze (b7), breaking jetpacks should be a no-no, and would likely require a backout.
~ rob
Lukas Blakk
the tree, and if you want to see what's happening with them you can go
to http://tinderbox.mozilla.org/showbuilds.cgi?tree=Firefox&noignore=1
Cheers,
Lukas
> _______________________________________________
> dev-planning mailing list
> dev-pl...@lists.mozilla.org
> https://lists.mozilla.org/listinfo/dev-planning
Benjamin Smedberg
> I don't think we want to close the tree, but I think that the JetPack
> team needs to be involved and we all need to quickly evaluate what caused
> JetPack to break. Without these tests, f.e., we missed that GC
> Compartments breaks JetPack.
>
> I'd be interested to hear other people's thoughts about this.
In general, I think we want to end up in a place where "working jetpacks" is
part of our regular test suite. But I am very worried about having code
outside of the main repository which affects test results. I think that as
the Jetpack SDK gets to 1.0 and we promise not to break the APIs, we should
import the SDK into mozilla-central. That way, if SDK changes are needed for
a project branch, they can be checked in and tested along with the code
changes. Having multiple repositories could cause a lot of friction during
the development process, and even more when trying to regression-hunt.
This same kind of decision also affects other projects (such as developer
tools) which are considering whether to put their projects into other kinds
of source control. I'm not opposed to it in principle, but I really think at
the end of the process all our in-product code should live in a single
repository.
--BDS
Rob Campbell
+1 to this. This is my biggest fear when considering an alternate source control mechanism for devtools work. I'd even suggest pulling it back in before 1.0 so we can stay in sync while API-completeness is being worked on. If the jetpack devs want to use a separate repo, spin a branch or use user repos for "rapid iterations".
Right now we've got bug 604641 as an excellent example of why the current scenario doesn't work.
~ rob
--
Rob Campbell
- blog: http://antennasoft.net/robcee
- twitter: http://twitter.com/robcee
johnjbarton
...
> Right now we've got bug 604641 as an excellent example of why the current scenario doesn't work.
As I read it, but 604641 is an example of testing breakdown, unrelated
to source code repository. The organizational goal is to ship Firefox
with Jetpack. Implicit in that goal is "Firefox + Jetpack" works, but
the testing system only applies to "Firefox" and "Jetpack". Therefore
sooner or later (b7 in this case) "Firefox + Jetpack" fails.
If you puts Jetpack in the same repository as Firefox, but still don't
test them together, the problem remains. (Of course my interest here is
the case where "Jetpack" is replaced by "Firebug").
Indeed, applying the logic of "everything in one repository" would
require Windows and Linux as well as all addons in mozilla-central.
Doesn't make sense.
jjb
Benjamin Smedberg
> If you puts Jetpack in the same repository as Firefox, but still don't test
> them together, the problem remains. (Of course my interest here is the case
> where "Jetpack" is replaced by "Firebug").
I think you misread or misunderstood my concern. I think we are all agreed
that we should test Firefox+Jetpack. The question is what we do with those
results, and how they affect the development process.
The current results of Firefox+Jetpack results depend on two things: the
-central changeset, and the jetpack-SDK changeset. If the tests start
failing, the developer must look through two different changelogs to find
the possible cause of the regression.
It's also the case that Firefox development will intentionally break the
current Jetpack SDK at times. If fixing the breakage requires checking out a
different repository, that is a fair bit of additional friction in the
development process.
At some point, the Jetpack SDK is going to reach version 1.0 and be declared
API-stable. At that point, I think we should commit to not breaking the SDK
by making the SDK part of mozilla-central. Until then, breaking the Jetpack
SDK probably shouldn't close the tree.
> Indeed, applying the logic of "everything in one repository" would require
> Windows and Linux as well as all addons in mozilla-central. Doesn't make sense.
No, the Windows/Linux version is a constant. And we don't run automated
regression tests against addons for the same reason: they are a moving
target, and are *expected* to break during Firefox development.
We don't commit to not breaking addons during development.
--BDS
johnjbarton
> On 10/25/10 11:04 AM, johnjbarton wrote:
>
>> If you puts Jetpack in the same repository as Firefox, but still don't
>> test
>> them together, the problem remains. (Of course my interest here is the
>> case
>> where "Jetpack" is replaced by "Firebug").
>
> I think you misread or misunderstood my concern. I think we are all
> agreed that we should test Firefox+Jetpack. The question is what we do
> with those results, and how they affect the development process.
Yes, and I am suggesting a different way they should affect the process.
>
> The current results of Firefox+Jetpack results depend on two things: the
> -central changeset, and the jetpack-SDK changeset. If the tests start
> failing, the developer must look through two different changelogs to
> find the possible cause of the regression.
For regression testing there are four things: Firefox-before,
Firefox-after, jetpack-before, jetpack-after. The initial state is
Firefox-before works with jetpack-before. The test state for Firefox is
Firefox-after+jetpack-before: Firefox should never be responsible for
jetpack changesets. The test state for jetpack is Firefox-before +
jetpack-after: jetpack should not be responsible for firefox changesets
for regressions.
(This is the reason Firebug releases span 2 versions of Firefox).
> It's also the case that Firefox development will intentionally break the
> current Jetpack SDK at times. If fixing the breakage requires checking
> out a different repository, that is a fair bit of additional friction in
> the development process.
Intentional breakage should be an entirely different problem. When
Firefox-after + jetpack-before fails because of intentional breakage,
the developer should know why. There should be an established procedure
for this case, with a notification and response plan.
> At some point, the Jetpack SDK is going to reach version 1.0 and be
> declared API-stable. At that point, I think we should commit to not
> breaking the SDK by making the SDK part of mozilla-central. Until then,
> breaking the Jetpack SDK probably shouldn't close the tree.
This approach does not scale, as you know from experience. By
synchronizing all development in the entire project on a single point,
more and more time goes in to resolutions at check-in.
>
>> Indeed, applying the logic of "everything in one repository" would
>> require
>> Windows and Linux as well as all addons in mozilla-central. Doesn't
>> make sense.
>
> No, the Windows/Linux version is a constant. And we don't run automated
> regression tests against addons for the same reason: they are a moving
> target, and are *expected* to break during Firefox development.
Addons are not a vague moving target. Like firefox, they have -before
and -after versions. Changes in Firefox can break addons on purpose or
by mistake. Testing addons-before against firefox-after is an effective
way to ensure that firefox changes are intended.
I'm not suggesting that Firefox checkins be blocked by testing all
addons. That would not be practical. But the complete opposite, no
testing at all, also is not sensible. What we have now is an willy
nilly ad hoc testing solution. A small change in approach could have
significant gains.
>
> We don't commit to not breaking addons during development.
But you do commit to having addons that work when Firefox ships. The
current plan for meeting that commitment could be better. Integrating
testing earlier and establishing a procedure for changes that
intentionally break dependent layers would help find problems earlier
with less pain and the rapid-cycling model would be more effective.
jjb
Mike Beltzner
>> At some point, the Jetpack SDK is going to reach version 1.0 and be
>> declared API-stable. At that point, I think we should commit to not
>> breaking the SDK by making the SDK part of mozilla-central. Until then,
>> breaking the Jetpack SDK probably shouldn't close the tree.
>
> This approach does not scale, as you know from experience. By synchronizing all development in the entire project on a single point, more and more time goes in to resolutions at check-in.
I don't think it needs to scale. JetPack is meant to be a stable SDK, the rest of the Mozilla API set is not. For things we declare as stable, we should have tests which flag when changes have caused instability.
>> We don't commit to not breaking addons during development.
>
> But you do commit to having addons that work when Firefox ships. The current plan for meeting that commitment could be better. Integrating testing earlier and establishing a procedure for changes that intentionally break dependent layers would help find problems earlier with less pain and the rapid-cycling model would be more effective.
We commit to working with certain addons to ensure that there's an upgrade path when Firefox ships, yes. It's not a global commitment nor even a "this many addons must work" commitment. Firebug is one of those add-ons, and as such I think we should have an agreed upon set of APIs that Firefox can assume will remain stable, and indeed a set of tests that help us understand when changes will break Firebug. Same to for add-ons that we "bless" in this fashion, and the JetPack SDK which will hopefully cover an increasing space of add-ons.
cheers,
mike
Shawn Wilsher
> As I read it, but 604641 is an example of testing breakdown, unrelated
> to source code repository. The organizational goal is to ship Firefox
> with Jetpack. Implicit in that goal is "Firefox + Jetpack" works, but
> the testing system only applies to "Firefox" and "Jetpack". Therefore
> sooner or later (b7 in this case) "Firefox + Jetpack" fails.
is to ship Firefox with jetpack. Jetpack is an SDK that an add-on would
ship with. The SDK would be compatible with certain versions of
Firefox, but, ideally, provide a stable API to add-ons so an add-on can
easily drop in a newer SDK and work in a newer version of Firefox.
All this means is that Firefox needs to ship with the needed APIs to let
jetpack work.
Cheers,
Shawn
Benjamin Smedberg
> I think there is some misunderstanding here. I don't believe the intent is
> to ship Firefox with jetpack. Jetpack is an SDK that an add-on would ship
> with. The SDK would be compatible with certain versions of Firefox, but,
> ideally, provide a stable API to add-ons so an add-on can easily drop in a
> newer SDK and work in a newer version of Firefox.
>
> All this means is that Firefox needs to ship with the needed APIs to let
> jetpack work.
No misunderstanding on my part, at least. We need to test the combination of
"Jetpack SDK + Firefox", preferably on-checkin. The fact that the SDK code
doesn't end up in Firefox doesn't really change how we have to test it to
keep our compatibility promises.
--BDS