Mozbase discussion today at 2:30pm EST

[Andrew Halberstadt]

There is a bi-weekly mozbase meeting at 2:30pm EST/ 11:30am PST in
jgriffin's vidyo room. I figured I'd send out a broader announcement as
we will be making decisions about some things that will affect others
outside the project.

The etherpad is: https://etherpad.mozilla.org/great-automation-refactor

Summary:
Currently the way mozbase gets used by our automation is sort of crazy.
Some things use the in-tree version, some things pull mozbase off of
puppetagain (an internal pypi server), it lives in two locations and
gets synced from one to the other. This all makes it really confusing
for consumers / developers.

Some questions we will be asking:
* Should mozbase be moved into mozilla-central?
* Should the non-canonical location (either github or m-c) be deleted?
* Should the test harnesses continue to use the puppetagain packages?
* Should dependency versioning be strict? (== vs >=)
* What should constitute a version bump?

If you have any opinions on any of these questions I invited you to join
us and let them be heard!

Cheers,
Andrew

[Andrew Halberstadt]

In case you missed the meeting, I filled out some meeting notes here:
https://wiki.mozilla.org/Auto-tools/Projects/Mozbase/Automation-Refactor/Meetings/2013-04-09

tl;dr - We are leaning towards moving mozbase in-tree and using strict
versioning, but no consensus was reached on either.

The current action items are to use in-tree mozbase for all in-tree
automation (replacing puppetagain in some cases) and to finish hooking
up the mozbase tests to make check. These are both pre-requisites to
mozbase in m-c and need to be finished either way. I'll start a
discussion post once these items are finished where we can revisit the
argument if we need to, or figure out time frames and logistics if we don't.

Thanks,
Andrew

[William Lachance]

On 04/09/2013 04:29 PM, Andrew Halberstadt wrote:
> In case you missed the meeting, I filled out some meeting notes here:
> https://wiki.mozilla.org/Auto-tools/Projects/Mozbase/Automation-Refactor/Meetings/2013-04-09
>
>
> tl;dr - We are leaning towards moving mozbase in-tree and using strict
> versioning, but no consensus was reached on either.
>
> The current action items are to use in-tree mozbase for all in-tree
> automation (replacing puppetagain in some cases) and to finish hooking
> up the mozbase tests to make check. These are both pre-requisites to
> mozbase in m-c and need to be finished either way. I'll start a
> discussion post once these items are finished where we can revisit the
> argument if we need to, or figure out time frames and logistics if we
> don't.

So one thing which I brought up during the meeting, but didn't explore
the full implications of perhaps, is how talos interacts with m-c, and
how that might change. For those not aware, we use a .zip distribution
of talos to test m-c. This .zip file is created an arbitrary point in
time and is referenced from inside the tree via a file called
talos.json. To create the .zip distribution, we pull down pegged
versions of mozbase from pypi in order to create a talos zip
distribution that is used in production:

http://hg.mozilla.org/build/talos/file/c10f4a861b3d/create_talos_zip.py

We explicitly moved to only mirroring mozbase on every version bump in
order to keep talos always in sync with the m-c version of mozbase (in
particular mozdevice). The idea is that on every merge of a mozbase
component, we would also bump the version number of said component and
update talos to request it. If we moved maintenance over to m-c,
effectively every commit to mozbase would be such a merge. Would we then
versionbump mozdevice, etc. on every commit (and update talos
accordingly) in order to keep things in sync? That seems somewhat
unworkable to me, so something would have to give -- either giving up on
talos being in sync with the tree or somehow making it use the in-tree
copy of mozbase.

This sort of issue is why I'm rather lukewarm to negative to moving
mozbase in-tree. Yes, the current approach has a number of issues, but
it's a devil we know (we've been doing it for over a year) and one we
can address with specific steps (whether that be more frequent
mirroring, in-tree tests, whatever).

Will

[Ed Morley]

On 09 April 2013 22:36:02, William Lachance wrote:
> Would we
> then versionbump mozdevice, etc. on every commit (and update talos
> accordingly) in order to keep things in sync? That seems somewhat
> unworkable to me, so something would have to give -- either giving up
> on talos being in sync with the tree or somehow making it use the
> in-tree copy of mozbase.

Fwiw talos is regularly out of sync with m-c for things like mozcrash,
so not version-bumping every time wouldn't exactly be changing the
status quo (not that it's ideal anyway, but still, we should compare
against reality rather than the ideal current scenario).

Ed

[William Lachance]

I hadn't been following talos that closely recently, though I can't I'm
that surprised by this. If making talos track changes to m-c's test
infrastructure is really that important, I suspect the only sensible way
forward is to integrate it into m-c proper.

Moving back to mozbase, I wonder if we shouldn't consider going back to
more frequent full-repo mozbase->m-c merges (without the extra burden of
releasing new versions to pypi) as another option, since it sounds like
the argument for not doing that is no longer compelling.

Will

[Jeff Hammel]

On 04/10/2013 01:47 AM, Ed Morley wrote:
> On 09 April 2013 22:36:02, William Lachance wrote:
>> Would we
>> then versionbump mozdevice, etc. on every commit (and update talos
>> accordingly) in order to keep things in sync? That seems somewhat
>> unworkable to me, so something would have to give -- either giving up
>> on talos being in sync with the tree or somehow making it use the
>> in-tree copy of mozbase.
>
> Fwiw talos is regularly out of sync with m-c for things like mozcrash,
> so not version-bumping every time wouldn't exactly be changing the
> status quo (not that it's ideal anyway, but still, we should compare
> against reality rather than the ideal current scenario).
>

> Ed
> _______________________________________________
AIUI, talos on windows is still on python 2.4. This (still) not only
holds back talos efforts but in addition makes for fragility as we can't
develop whatever mozbase software we use there past 2.4. Fun. AFAIK
this is the only technical barrier for putting talos in-tree.

[Jeff Hammel]

On 04/10/2013 08:09 AM, William Lachance wrote:

> I hadn't been following talos that closely recently, though I can't
> I'm that surprised by this. If making talos track changes to m-c's
> test infrastructure is really that important, I suspect the only
> sensible way forward is to integrate it into m-c proper.
>
> Moving back to mozbase, I wonder if we shouldn't consider going back
> to more frequent full-repo mozbase->m-c merges (without the extra
> burden of releasing new versions to pypi) as another option, since it
> sounds like the argument for not doing that is no longer compelling.
>
> Will
>

Digging up this old thread that I've been meaning to respond to for
weeks....

The reason we changed from full-repo mozbase->m-c merges to versioned
package merges is so that what is used in m-c is pinned to discrete
versions that other tools can also be pinned to. If mozbase is intended
to be a set of packages that is m-c agnostic, then I would argue that
this continues to be important. If mozbase is not supposed to be m-c
agnostic, then I would argue that yes we should move mozbase to m-c. Is
the github mirror only for staging mozbase development, doing RTD, and
writing mirroring bugs and software? I don't find those compelling use
cases.

FWIW, my true perspective is and has always been that mozbase should not
be in mozilla-central; that mozbase was a collection of utilities. I am
not even sure why they are in the same repository; generally, if I am
making a software package, I will have a discrete repository for it. My
perspective is directed towards growing ecosystems of open-source
software. While some have agreed in theory, much more interest has been
expressed in keeping status quo of everything goes in m-c if buildbot
touches it, including mozbase. This being the case, I don't see any
particular reason for keeping a separate repository for mozbase's
development.

I have also tried to raise the issue (albeit, I am not the best
communicator) in moving towards standardization of development practices
across teams -- in this case, "if i'm developing software like mozbase,
how should I mirror my work to m-c (or should I)?" -- not for
classification or enforcement purposes, but to consolidate effort around
tools to do so and documentation about why such and such is hard or easy
or should be avoided. Again, not much traction.

I am all for getting things done today, which seems to be the guiding
principle of these priorities. But I do want us -- all of us -- to be
moving towards the direction we really want to go in the longer term and
not coding in circles for continuing course-corrections.

Jeff

[Andrew Halberstadt]

On 05/07/2013 01:17 PM, Jeff Hammel wrote:
> Digging up this old thread that I've been meaning to respond to for
> weeks....
>
> The reason we changed from full-repo mozbase->m-c merges to versioned
> package merges is so that what is used in m-c is pinned to discrete
> versions that other tools can also be pinned to. If mozbase is intended
> to be a set of packages that is m-c agnostic, then I would argue that
> this continues to be important. If mozbase is not supposed to be m-c
> agnostic, then I would argue that yes we should move mozbase to m-c. Is
> the github mirror only for staging mozbase development, doing RTD, and
> writing mirroring bugs and software? I don't find those compelling use
> cases.

I think mozbase is meant to be m-c agnostic, but that the primary and
most important (by far) consumers of mozbase all live in m-c.

> FWIW, my true perspective is and has always been that mozbase should not
> be in mozilla-central; that mozbase was a collection of utilities. I am
> not even sure why they are in the same repository; generally, if I am
> making a software package, I will have a discrete repository for it. My
> perspective is directed towards growing ecosystems of open-source
> software. While some have agreed in theory, much more interest has been
> expressed in keeping status quo of everything goes in m-c if buildbot
> touches it, including mozbase. This being the case, I don't see any
> particular reason for keeping a separate repository for mozbase's
> development.

I agree with this sentiment. I also still think that from an open source
project/community point of view, it makes more sense to have mozbase in
github. But from a practical point of view, the people who are going to
be using mozbase are the same people who are intimately familiar with
m-c anyway, meaning mirroring is unnecessary work.

> I have also tried to raise the issue (albeit, I am not the best
> communicator) in moving towards standardization of development practices
> across teams -- in this case, "if i'm developing software like mozbase,
> how should I mirror my work to m-c (or should I)?" -- not for
> classification or enforcement purposes, but to consolidate effort around
> tools to do so and documentation about why such and such is hard or easy
> or should be avoided. Again, not much traction.

The only thing I'm adamantly against is our current situation where we
mirror from one repository to the other. See
https://bugzilla.mozilla.org/show_bug.cgi?id=860110 for an example of
why this is bad.

> I am all for getting things done today, which seems to be the guiding
> principle of these priorities. But I do want us -- all of us -- to be
> moving towards the direction we really want to go in the longer term and
> not coding in circles for continuing course-corrections.
>
> Jeff

Agreed. I think it doesn't make sense to have half our automation living
in m-c and half outside. So either we move all our harnesses (mochitest,
reftest, xpcshell, marionette, etc) out of m-c, or we move mozbase into
it. I don't think the former will ever happen, so the latter seems like
our best course.

Andrew

[Jeff Hammel]

On 05/07/2013 10:48 AM, Andrew Halberstadt wrote:
> <snip/>

>> I have also tried to raise the issue (albeit, I am not the best
>> communicator) in moving towards standardization of development practices
>> across teams -- in this case, "if i'm developing software like mozbase,
>> how should I mirror my work to m-c (or should I)?" -- not for
>> classification or enforcement purposes, but to consolidate effort around
>> tools to do so and documentation about why such and such is hard or easy
>> or should be avoided. Again, not much traction.
>

> The only thing I'm adamantly against is our current situation where we
> mirror from one repository to the other. See
> https://bugzilla.mozilla.org/show_bug.cgi?id=860110 for an example of
> why this is bad.

I don't find this a particularly compelling case of why this is bad.
Why not?

- this situation was anticipated > 1 year ago; it was decided that it
was unworth putting time into preventing (again, from my method-centric
POV, I would have done the work up front even though I'm guess to date
that more time would have been spent this way)

- someone did not follow directions; admittedly, working on the various
pieces of m-c it is hard to figure out what directions are. someone also
did not follow up with communication, putting the burden on us. While
this is symptomatic of the mirroring problem...it is also symptomatic of
a whole lot of things.

- this is technically solvable/preventable if we care; there are several
solutions (scripts/tools to automagically/semiautomagically do this,
mirroring eggs so this isn't possible, policy enforcement since we
actually just soaked a policy violation, etc) available if we found this
sort of mirroring to provide some value; if we didn't then we shouldn't
do it, and not just for this reason

[Andrew Halberstadt]

On 05/07/2013 02:03 PM, Jeff Hammel wrote:
> - someone did not follow directions; admittedly, working on the various
> pieces of m-c it is hard to figure out what directions are. someone also
> did not follow up with communication, putting the burden on us. While
> this is symptomatic of the mirroring problem...it is also symptomatic of
> a whole lot of things.
>
> - this is technically solvable/preventable if we care; there are several
> solutions (scripts/tools to automagically/semiautomagically do this,
> mirroring eggs so this isn't possible, policy enforcement since we
> actually just soaked a policy violation, etc) available if we found this
> sort of mirroring to provide some value; if we didn't then we shouldn't
> do it, and not just for this reason

One thing I've realized from working on B2G is that it is very difficult
to get developers to spend their valuable time looking into test
failures. So yes, we could solve this technically or through policy
enforcement. But anytime we add complexity/extra steps it pushes
developers further away. Our job is to make their lives easier, not more
complicated.

-Andrew