A merged comm-/mozilla-central repository is now available
Joshua Cranmer 🐧
background first:
The build team has been more or less promising a build system blitz of
changes in Q1 2016 to finally move to a saner build system, and the
current c-c/m-c repository directory structure is a difficult roadblock
to those changes. As someone who has worked on the build system on
occasion in the past, I can attest to all the ugliness needed to keep
c-c reworking in this regard. It's pretty clear that our current build
setup is completely untenable to maintain through the end of Q1 2016,
and actually it has been rather untenable for the past several months. I
will point out that it's not feasible for us to not use Mozilla's build
system, so long as we have substantial C++ code that needs to interact
with Gecko internals, and anyone who thinks otherwise is misinformed.
The only feasible path forward is for us to merge comm-central and
mozilla-central into one repository. I (and others) have held out hope
that Mozilla could be convinced to let mozilla-central be that
repository, but it is pretty clear at this point that it is not to be,
and so we must resign ourselves to the fallback option of forking
mozilla-central and periodically merging new revisions into mozilla-central.
Fortunately, I already have a script that can merge the two repositories
together into one. The resulting repository is at
<http://hg.mozilla.org/projects/cypress/>, and it was produced by a
fully-automated script that pulls comm-central, deletes the build system
(well, the remnants thereof, most having been gutted by cc-rework quite
some time ago), rewrites references to the mozilla/ directory to the
top-level, and then pulls and merges mozilla-central with an automated
resolution of the two remaining conflicting files (.hgtags and AUTHORS).
The cypress repository is dedicated to testing release automation for
the necessary changes that need to be made. Once these changes can be
identified and we get working builds and tests on all the platform, I
expect that we will have a day where we perform the merge and push to
comm-central. I'm using bug 787208 to track this change.
One major important note about the change is that, for the first time
since 2008, all the code to build Thunderbird will be in a single
repository. As such, client.py will no longer be necessary to build
Thunderbird (and indeed, can't be used, since mozilla-central maintains
a file named that for importing third-party source repos). It sounds
like the SeaMonkey maintainers are planning on pulling DOM inspector and
chatzilla into the new repository as well, but I'm not directly involved
in that decision.
--
Joshua Cranmer
Thunderbird and DXR developer
Source code archæologist
_______________________________________________
tb-planning mailing list
tb-pl...@mozilla.org
https://mail.mozilla.org/listinfo/tb-planning
Tony Mechelynck
> […] It sounds like the SeaMonkey
> maintainers are planning on pulling DOM inspector and chatzilla into the new
> repository as well, but I'm not directly involved in that decision.
The current client.py (at least when building the Suite) already
grafts the cZ and DOMi repositories as (IIRC) subdirectories of
mozilla/extensions/, on its grafted mozilla-central repo; so IIUC the
change wouldn't be a big one for SeaMonkey. If comm-central becomes
the master repo for both cZ and DOMi, where the change would be big
would be for cZ and DOMi themselves, who would no longer be able to
release their own XPIs without having a clone of the full (and even
bigger then than is it now) comm-central; so my guess would be that
the SeaMonkey Council (of which I am admittedly not a member) might
welcome such a merge, but that the final decision would rest with the
owners of cZ and DOMi themselves. I don't know, and can't tell, which
scale of the pair will be more weighty for them.
This is only a side issue for this particular news thread anyway.
Best regards,
Tony
Robert Kaiser
> The only feasible path forward is for us to merge comm-central and
> mozilla-central into one repository. I (and others) have held out hope
> that Mozilla could be convinced to let mozilla-central be that
> repository, but it is pretty clear at this point that it is not to be,
> and so we must resign ourselves to the fallback option of forking
> mozilla-central and periodically merging new revisions into
> mozilla-central.
\o/
Even though this is the second-best solution, it's great to see.
> The cypress repository is dedicated to testing release automation for
> the necessary changes that need to be made. Once these changes can be
> identified and we get working builds and tests on all the platform, I
> expect that we will have a day where we perform the merge and push to
> comm-central.
As the person who created both the c-c repo and its build system, yes,
please merge this to c-c once it works well. We'll need to make noise
about it for sure esp. for those people who build Firefox off
c-c/mozilla but otherwise c-c is where things should live.
How often or in what cadence do you think we'll merge m-c into c-c in
the future? Can this be automated? How far? Do we need a comm-inbound
where we test the merge and fix up issues that keep Thunderbird (and IB,
SM, whatever) from building?
KaiRo
Ben Bucksch
> The build team has been more or less promising a build system blitz of
> changes in Q1 2016 to finally move to a saner build system, and the
> current c-c/m-c repository directory structure is a difficult
> roadblock to those changes.
Fair enough
> it's not feasible for us to not use Mozilla's build system, so long as
> we have substantial C++ code that needs to interact with Gecko
> internals, and anyone who thinks otherwise is misinformed.
Agreed.
> The only feasible path forward is for us to merge comm-central and
> mozilla-central into one repository.
There's another option that's much simpler: m-c and c-c can keep being
different repositories, but they can live in the same directory on-disk.
You can simply check out both repos in the same on-disk repository.
mail/ and mailnews/ would live on the same level as browser/ .
For the build system, that would be strictly identical to a merged repo.
If hg has an issue with that, because it expects e.g. a root/.hg/
directory, then you can simply ignore/delete the m-c .hg/ dir (because
we make almost no changes there anyway) and keep the c-c .hg/ .
> forking mozilla-central and periodically merging new revisions into
> mozilla-central
I would be concerned about that, due to repository size. m-c is 20 times
larger than c-c.
> Fortunately, I already have a script that can merge the two
> repositories together into one. The resulting repository is at
> <http://hg.mozilla.org/projects/cypress/>, and it was produced by a
> fully-automated script that pulls comm-central, deletes the build
> system (well, the remnants thereof, most having been gutted by
> cc-rework quite some time ago), rewrites references to the mozilla/
> directory to the top-level, and then pulls and merges mozilla-central
> with an automated resolution of the two remaining conflicting files
> (.hgtags and AUTHORS).
I was thinking along the same lines, just that I propose that this
happens as part of the local checkout system (client.py), instead of
pushing m-c changes into the official c-c repo.
Ben
Patrick Cloke
> Joshua Cranmer 🐧 wrote on 17.12.2015 02:59:
>> The only feasible path forward is for us to merge comm-central and
>> mozilla-central into one repository.
> There's another option that's much simpler: m-c and c-c can keep being
> different repositories, but they can live in the same directory
> on-disk. You can simply check out both repos in the same on-disk
> repository. mail/ and mailnews/ would live on the same level as
> browser/ .
control.
> If hg has an issue with that, because it expects e.g. a root/.hg/
> directory, then you can simply ignore/delete the m-c .hg/ dir (because
> we make almost no changes there anyway) and keep the c-c .hg/ .
How would you update the Mozilla code then? This isn't a reasonable
solution IMO.
>> forking mozilla-central and periodically merging new revisions into
>> mozilla-central
>
> I would be concerned about that, due to repository size. m-c is 20
> times larger than c-c.
Why is this a concern? Instead of pulling c-c, then m-c, you pull just
c-c which includes m-c. The size should be roughly equivalent (and can
be checked very easily by pulling Joshua's test repo). I don't believe
this argument without numbers to back it up.
>> Fortunately, I already have a script that can merge the two
>> repositories together into one. The resulting repository is at
>> <http://hg.mozilla.org/projects/cypress/>, and it was produced by a
>> fully-automated script that pulls comm-central, deletes the build
>> system (well, the remnants thereof, most having been gutted by
>> cc-rework quite some time ago), rewrites references to the mozilla/
>> directory to the top-level, and then pulls and merges mozilla-central
>> with an automated resolution of the two remaining conflicting files
>> (.hgtags and AUTHORS).
>
> I was thinking along the same lines, just that I propose that this
> happens as part of the local checkout system (client.py), instead of
> pushing m-c changes into the official c-c repo.
Doing it locally for every person seems quite inefficient and prone to
issues if conflicts ever arise. I.e. it's harder to ensure that everyone
ACTUALLY has the same code or not.
From previously conversations with Joshua and other developers, I agree
with him that the periodic merging of m-c into c-c is the 'best' way
forward.
Great work Joshua,
--Patrick
Nomis101 🐝
Can this be build the same way like Firefox with the "mach" command?
Ben Bucksch
> This is much more complicated for people who are not experts in version
> control.
Nothing changes for you.
>> I would be concerned about that, due to repository size. m-c is 20
>> times larger than c-c.
> Why is this a concern? Instead of pulling c-c, then m-c, you pull just
> c-c which includes m-c.
larger. The repo is the crown perls of the project.
I don't care much about the download size (that's the same either way),
I care about the repo size, because it's the repo that you need to work
with when you diff, backup etc.pp..
Also, you have all the m-c commit in-between the c-c commits in the
commit log.
Joshua Cranmer 🐧
> Patrick Cloke wrote on 17.12.2015 17:53:
>> This is much more complicated for people who are not experts in version
>> control.
>
> That would be done by a script. You currently use "client.py
> checkout". Nothing changes for you.
One of the goals of the merger is to improve new contributor's lives by
making it easier to get and build Thunderbird. The client.py script is
itself a major impediment to that (and also a major headache for build
automation).
>
>>> I would be concerned about that, due to repository size. m-c is 20
>>> times larger than c-c.
>> Why is this a concern? Instead of pulling c-c, then m-c, you pull just
>> c-c which includes m-c.
>
> Right. But m-c is 20 times the size of c-c, so c-c is then 20 times
> larger. The repo is the crown perls of the project.
>
> I don't care much about the download size (that's the same either
> way), I care about the repo size, because it's the repo that you need
> to work with when you diff, backup etc.pp..
You do realize that, no matter what, you have to have m-c on-disk
anyways if you care about building? It may have been thought in 2007 and
2008 that it could not be the case one day, but it's been clear since
2011 that it is very much the case that c-c needs m-c to build and that
is to remain the situation for quite some time.
--
Joshua Cranmer
Thunderbird and DXR developer
Source code archæologist
_______________________________________________
Joshua Cranmer 🐧
> As the person who created both the c-c repo and its build system, yes,
> please merge this to c-c once it works well. We'll need to make noise
> about it for sure esp. for those people who build Firefox off
> c-c/mozilla but otherwise c-c is where things should live.
Actually, I'll rerun the script when making it live, rather than merging
cypress into comm-central (cypress will likely end up with lots of heads).
> How often or in what cadence do you think we'll merge m-c into c-c in
> the future? Can this be automated? How far? Do we need a comm-inbound
> where we test the merge and fix up issues that keep Thunderbird (and
> IB, SM, whatever) from building?
It should be able to be automatic, since I don't foresee any merge
conflicts happening. The other questions are effectively independent
management questions that can be answered at greater leisure. I've
personally generally thought that merging m-c every day or every m-c
push (waiting ~15min to coalesce multiple tree pushes) (the latter
happening about every 6-24h) seemed about right.
--
Joshua Cranmer
Thunderbird and DXR developer
Source code archæologist
_______________________________________________
ace
Subject: Re: A merged comm-/mozilla-central repository is now available
From: Joshua Cranmer 🐧 <pidg...@gmail.com>
To: tb-pl...@mozilla.org
Date: Thu, 17 Dec 2015 15:47:05 -0600
>>>> I would be concerned about that, due to repository size. m-c is 20
>>>> times larger than c-c.
>>> Why is this a concern? Instead of pulling c-c, then m-c, you pull just
>>> c-c which includes m-c.
>>
>> Right. But m-c is 20 times the size of c-c, so c-c is then 20 times
>> larger. The repo is the crown perls of the project.
>>
>> I don't care much about the download size (that's the same either
>> way), I care about the repo size, because it's the repo that you need
>> to work with when you diff, backup etc.pp..
>
> You do realize that, no matter what, you have to have m-c on-disk
> anyways if you care about building? It may have been thought in 2007 and
> 2008 that it could not be the case one day, but it's been clear since
> 2011 that it is very much the case that c-c needs m-c to build and that
> is to remain the situation for quite some time.
Ben probably meant the repo size as in how many files the hg
qpop/qref/qpush commands need to parse when creating patches. And there,
doing any command on m-c is much slower than on c-c (I do it sometimes).
So I agree with Ben here. It is true that the download/on disk size is
the same, but it matters how it is partitioned into repos/logical units.
R Kent James
> It should be able to be automatic, since I don't foresee any merge
> conflicts happening.
I think this is short-sighted. I've been using a mozilla-central branch
for a year in comm-esr38, and in spite of lack of enthusiasm here, I
intend to continue to do that, and start doing it for comm-beta, in the
next cycle. We do need to have subtle differences between m-c and our
as-built version, and we need a concrete way to manage that.
You should have some method to merge m-c into something that is a
canonical copy of m-c, that should not have conflicts, and some other
mechanism to manage a set of patches that are either included or
excluded from our builds.
:rkent
Magnus Melin
> So I agree with Ben here. It is true that the download/on disk size is
> the same, but it matters how it is partitioned into repos/logical units.
live without mozilla-central.
Of course working with hg is slightly slower on a larger repo, but the
benefits sure outweigh the cost here.
-Magnus
ishikawa
> The cypress repository is dedicated to testing release automation for the
> necessary changes that need to be made. Once these changes can be identified
> and we get working builds and tests on all the platform, I expect that we
> will have a day where we perform the merge and push to comm-central. I'm
> using bug 787208 to track this change.
So this means that occasional patch contributors should not use this
repository, but
we should stand back and let the people who deal with release automation to
look into the
minor details to weed out kinks?
Or do you need some volunteers to occasionally fetch and build locally to
see if anything
worth reporting is found, etc?
It is a pity that a batch synchronization like this is necessary.
But I think it is the only option right now from the look of the discussion
that took place on the mailing list.
Thank *you* and other people within mozilla who have made it possible to
build C-C TB on the current infrastructure so far.
Magnus Melin
> On 12/17/2015 1:56 PM, Joshua Cranmer 🐧 wrote:
>> It should be able to be automatic, since I don't foresee any merge
>> conflicts happening.
>
> I think this is short-sighted. I've been using a mozilla-central branch for
> a year in comm-esr38, and in spite of lack of enthusiasm here, I intend to
> continue to do that, and start doing it for comm-beta, in the next cycle. We
> do need to have subtle differences between m-c and our as-built version, and
> we need a concrete way to manage that.
>
> You should have some method to merge m-c into something that is a canonical
> copy of m-c, that should not have conflicts, and some other mechanism to
> manage a set of patches that are either included or excluded from our builds.
I don't see the problem. Wouldn't it work just like it works now, only that
the automatic merge would merge the mozilla-central thunderbird-version-branch
(not the default branch) into comm-central? For the cases the version branch
is used.
-Magnus
Magnus Melin
mozilla-central as such, would it be possible to have comm-central as a long
time branch under mozilla-central? The main opposition to merging seems to be
that people "shouldn't have to consider comm" and on a branch it certainly
would be opt-in.
-Magnus
_______________________________________________
Joshua Cranmer 🐧
> I don't know if it's been discussed, but if we can't merge
> comm-central into mozilla-central as such, would it be possible to
> have comm-central as a long time branch under mozilla-central? The
> main opposition to merging seems to be that people "shouldn't have to
> consider comm" and on a branch it certainly would be opt-in.
It does make pulling slightly harder (you have to remember to pull a
branch option), and I think the objection of "but comm-central should be
independent of mozilla-central" would still apply. I don't see any real
benefit to the ordeal.
--
Joshua Cranmer
Thunderbird and DXR developer
Source code archæologist
_______________________________________________
ishikawa
> Fortunately, I already have a script that can merge the two repositories
> together into one. The resulting repository is at
> <http://hg.mozilla.org/projects/cypress/>, and it was produced by a
> fully-automated script that pulls comm-central, deletes the build system
> (well, the remnants thereof, most having been gutted by cc-rework quite some
> time ago), rewrites references to the mozilla/ directory to the top-level,
> and then pulls and merges mozilla-central with an automated resolution of
> the two remaining conflicting files (.hgtags and AUTHORS). The cypress
> repository is dedicated to testing release automation for the necessary
> changes that need to be made. Once these changes can be identified and we
> get working builds and tests on all the platform, I expect that we will have
> a day where we perform the merge and push to comm-central. I'm using bug
> 787208 to track this change.
Thank you again for the great work.
I re-read the bug 787208 and I realized that, (sorry for catching up very
slowly), that
the main opposition seemed to be that by having the C-C directory
at the same level of M-C, it was feared that the C-C directory has to be
merged completely into the same HG repository.
Was this intended?
I wonder what the structure of the repository (or repositories) under cypress.
Do you intend to put C-C and M-C into the same repository?
Background of my question:
After having to work with C-C and M-C dichotomy for several years (OK, maybe
a few years),
I thought it was not intended but the intention was simply having the
C-C directory to be maintained at the same directory position of other
modules in M-C and
still have the different HG repositories.
Even this level of integration would make things much simpler in terms of
BUILD system integration, I think, by simply looking at the bustages of
build: the contortion of having to reference mozilla module subdirectories
under ./mozilla/modulename instead of simple ./modulename has caused many
failures in the past few years.
I think most of the recent build-related bustages are due these pathname
issues, ugh.
But then the MAIN opposition in bug 787208 seemed to be the
INCLUSION of C-C into M-C hg repository.
I didn't catch this subtle difference until now.
I wonder how many of C-C patch contributors want to have a single gigantic
repository instead of separate repositories.
Having the source code directory in the same directory position as those of
other modules in M-C is a good idea.
Then, we can simply address the mozilla modules by
simple ../modulename and be done with it. Right?
Having separate repositories may not be such a bad idea given the merit of
the above only,
and the separate repositories setup will make things easier in terms of the
FF push
not triggering automatic build of C-C or vice versa.
(I am not familiar how to set up such IGNORE feature at the top-level of M-C.)
But I have been living with the separate repositories of M-C and C-C (and
C-C's M-C lives under
./mozilla subdirectory), merged C-C editing and pushing can be done
once we move under ./C-C.
Only hitch would be when we want to build C-C TB, then we have to figure out
how to create C-C TB on the push.
Maybe we have to tell the build infrastructure ONCE for ALL that
`mail' program ought to be built under ./C-C
and we probably need to use the infamous mozilla-* patch name convention to
force the
applying of patches under ./mozilla (i.e., the separate direcotry), this time
maybe using thunderbird-* patch convention to force applying the patches to
./C-C subdirectory (i.e., the separate C-C
repository).
Oh well. There are still many details to work out, but
the continued impact on build infrastructure, which TB needs to survive,
will be minimized greatly IMHO.
Unfortunately this point does not seem to be understood very well outside
C-C TB developer community. :-(
TIA
Tony Mechelynck
[…]
> I wonder how many of C-C patch contributors want to have a single gigantic
> repository instead of separate repositories.
Hard to tell.
One thing is certain: anyone working only with Firefox couldn't care
less about the code for Mail, LDAP, Suite, ChatZilla and DOM
Inspector. OTOH, people working on SeaMonkey need them all, people
working on Thunderbird need a subtantial part of them, and both kinds
of people need Gecko, Toolkit, NSS, NSPR, etc., let's say most or all
of what lives in mozilla-central or its branch avatars.
In the past (and, I think, in the present), the c-c client.py took
care of that by pulling and updating all six (six! or now five now
that Venkman is dead) repositories (at least when preparing to buils
SeaMonkey), in the proper order, and graft them all into a single
directory tree which could be used either for compiling or (e.g. with
the mq extension to Mercurial) to work on proposed patches. The result
is a single gigantic tree structure, which make treats as a recursive
and (hopefully) consistent unit (with Makefiles and config scripts at
several levels) while to Mercurial each of the handful of repos is
totally independant of all the others.
I have in the past contributed a few patches to SeaMonkey but I
wouldn't regard myself as a "developer": these patches were
elementary, mostly one-liner or few-liner portings from Firefox or
Thunderbird. I had no real difficulties with it, but all my changes
were in comm-central. If I had been working on Toolkit instead, and
more experienced, I might (considering that I'm not on MoCo's payroll
;-) ) taken care of the side-effects of my patches on not only Firefox
but also Thunderbird and SeaMonkey and provided patches, or rather
*sets* of patches, straddling mozilla-central and comm-central. I'm
saying sets of patches because not only patches to Toolkit, Browser,
Mail and Suite need to pass review with different reviewers, pushing
to momzilla-central and pushing to comm-central MUST be done
severally. I was always conscious of that, but I have seen young patch
contributors who tried to mix suite/ and mozilla/ in a single patch,
and of course it didn't pass review. Last I tried (but that was maybe
a year ago) this repository structure didn't pose insurmountable
problems, but OTOH I've seen the SeaMonkey RelEng team struggle to
publish releases and nightlies on a more or less regular schedule, and
Kairo saying in this very thread that (I paraphrase) that client.py
system was falling in pieces, and I wouldn't say that to them it's a
piece of cake. So who am I to stand against a simplification of
procedures, even at the cost of repository "gianthood"? I won't.
Now that we will be (collectively I mean: TB+SM) masters of the
repository (or repositories) used to build our beloved mailer and
suite, we could either put everything in a Single Mercurial repository
or take advantage of something I have seen mentioned time and again on
the Mercurial mailing list (merc...@selenic.com IIRC) but haven't
studied, namely a Mercurial extension called "subrepository". I'm not
saying we necessarily should adopt it, just that we should be aware of
its existence. I think it would have both advantages (such as allowing
to pull smaller repositories, and thus, for instance, run a lesser
risk of network timeout in the case of a cloning) and inconvenients
(such as the necessity, I think, to make sure to have a consistent
repository set before going over to the compile phase, the way we did
with branch tags and client.py).
So in conclusion, I think there are several kinds of c-c "developers"
(in the wide sense of the word) who would prefer one repo rather than
five:
— old hands, who are aware of the complexity of the present system,
and feel bothered by the necessity of always keeping it in mind,
though they do;
— RelEng engineers, whose task would be enormously simplified,
especially on aurora, beta and release branches (where the changesets
to be pulled are not necessarily the head of the default branch for
every repository, especially, in the present situation, ChatZilla and
DOM Inspector);
— people trying their hand at "good first bugs", who would run the
less risk of coming afoul of the repository structure, the simpler it
would be.
Others might prefer separate repositories:
— the developers for cZ and DOMi, who would have to reserve disk space
for a full m-c + c-c + LDAP + cZ + DOMi clone, and not just their own
source. I'm not surprised that they seem to be still uncertain whether
they'll join the party (the feast I mean, not the pressure group).
— people who never "compile their own" but merely contribute patches
and maybe have them built by the Thunderbird try-servers in order to
test them.
— people with a slow Internet connection or a heavily busy CPU, who
would run less risk of connection timeouts when pulling a number of
successive new changesets and updating to the last of them if there
were less data to be pulled in each Mercurial run (considering that if
there is a timeout, Mercurial will throw to the dustbin anything
pulled so far).
People who, like I used to, contribute patches maybe a handful of
times a year, and are loath of building a c-c application anyway
because of the incredible RAM+swap total needed to link the libxul,
could probably do with either repository architecture.
Best regards,
Tony.