One Firefox repository to rule them all

[Gregory Szorc]

I'm pleased to announce the immediate availability of some *experimental* read-only Mercurial repositories containing the combined, useful history of the various Firefox repositories, all in chronological order and stored in a more efficient format that is faster to clone and pull from and results in faster client operations.

The repositories can be found at https://hg.mozilla.org/experimental. The repository you likely want to clone is https://hg.mozilla.org/experimental/firefox-unified. A visualization showing the chronological history of the repo can be seen at https://hg.mozilla.org/experimental/firefox-unified/graph.

The primary goal of these repositories is to provide developers (and eventually automation) with more efficient interaction with the Firefox source repositories. There are several secondary and side-benefits, including improving the scalability of Try and MozReview's repositories.

More documentation about these repos is available at [1]. tl;dr

* The repositories contain all the commits from the Firefox repositories you use everyday (central, inbound, fx-team, aurora, beta, esr, etc).

* The repositories do not contain all the *_RELBRANCH branches (which basically have no value to the average developer).

* Thes unified repositories are ~300MB *smaller* than mozilla-central despite containing ~28,000 more commits. This was achieved through light magic.

* Mercurial bookmarks are used to track the heads of the various Firefox repos.

* The pushlog data is derived from the first known push of a changeset, so it should match what's on e.g. central, inbound, etc.

* Sadly, git-cinnabar won't be able to talk to these repos just yet due to git-cinnabar not supporting some modern Mercurial features. A GitHub issue is on file at [2].

If you use the "firefoxtree" extension to manage a unified repository today, you should consider switching to one of these new unified repositories instead: it should be faster and easier to reason about.

The repositories have the "experimental" label attached so we can reserve the right to make changes without people complaining too loudly about backwards compatibility. (But I wouldn't worry too much about stability - I'm committed to keeping these running and improving them.) The goal is to flush out issues with these repositories then remove the "experimental" label. After that, we can have automation start consuming these repositories. After that, we can perhaps start thinking about consolidating around a single, canonical repository, including pushing. But that's a topic for another day.

I'm very anxious for feedback on these repositories. Please make noise in dev-versi...@lists.mozilla.org, #vcs, the "Developer Services: Mercurial: hg.mozilla.org" bug component, or in bug 1108729.

[1] https://mozilla-version-control-tools.readthedocs.org/en/latest/hgmozilla/unifiedrepo.html
[2] https://github.com/glandium/git-cinnabar/issues/64

[Dave Townsend]

On Thu, Apr 14, 2016 at 5:22 PM, Gregory Szorc <g...@mozilla.com> wrote:

I'm pleased to announce the immediate availability of some *experimental* read-only Mercurial repositories containing the combined, useful history of the various Firefox repositories, all in chronological order and stored in a more efficient format that is faster to clone and pull from and results in faster client operations.

Does this mean that it contains the full history including CVS?

[Andrew Halberstadt]

This is really cool!

Though I much prefer firefoxtree's namespace updating to keep track of
remote heads over using bookmarks. I want a label that will always point
to the last known head on the server, so e.g
`hg update central && hg commit -m "Foo"` should not move 'central'.
Using bookmarks to track the remote heads is also incompatible with my
bookbinder extension which I've come to rely quite heavily on. This
would be a personal blocker for me to make the switch.

Maybe firefoxtree could be adapted to work with this new repo as well.
Or maybe I could look into doing something with remotenames.

Andrew

On 14/04/16 08:22 PM, Gregory Szorc wrote:
> I'm pleased to announce the immediate availability of some *experimental*
> read-only Mercurial repositories containing the combined, useful history of
> the various Firefox repositories, all in chronological order and stored in
> a more efficient format that is faster to clone and pull from and results
> in faster client operations.
>

_______________________________________________
firefox-dev mailing list
firef...@mozilla.org
https://mail.mozilla.org/listinfo/firefox-dev

[Gregory Szorc]

No. Adding a Mercurial repo with CVS history isn't on the priority list. Make a case for it improving developer productivity and it could be.

[Kris Maglione]

On Fri, Apr 15, 2016 at 01:29:10PM -0700, Gregory Szorc wrote:
>On Thu, Apr 14, 2016 at 6:45 PM, Dave Townsend <dtow...@mozilla.com>
>wrote:

>> Does this mean that it contains the full history including CVS?
>
>No. Adding a Mercurial repo with CVS history isn't on the priority list.
>Make a case for it improving developer productivity and it could be.

I often start looking at the blame history in a file I'm working
in, run into a dead end at the initial CVS import, and then have
to switch to gecko-dev to find what I'm looking for.

That's not great when I'm working locally, and happen to have
up-to-date copies of both repos. But when I'm looking at blame
data that I got to from DXR, and then have to switch to Github,
find the file again, search through the history, and then
eventually give up and check out a copy of gecko-dev so I can
check locally, it's a major waste of time and energy.

[Gregory Szorc]

On Fri, Apr 15, 2016 at 7:16 AM, Andrew Halberstadt <ahalbe...@mozilla.com> wrote:

This is really cool!

Though I much prefer firefoxtree's namespace updating to keep track of
remote heads over using bookmarks. I want a label that will always point
to the last known head on the server, so e.g
`hg update central && hg commit -m "Foo"` should not move 'central'.
Using bookmarks to track the remote heads is also incompatible with my
bookbinder extension which I've come to rely quite heavily on. This
would be a personal blocker for me to make the switch.

Maybe firefoxtree could be adapted to work with this new repo as well.
Or maybe I could look into doing something with remotenames.

There is actually a mode in firefoxtree where the server can advertise the labels. We're not using it on the unified repo yet, but we could. Let's hash something out in bug 1264814.

[Gregory Szorc]

On Fri, Apr 15, 2016 at 7:16 AM, Andrew Halberstadt <ahalbe...@mozilla.com> wrote:

This is really cool!

Though I much prefer firefoxtree's namespace updating to keep track of
remote heads over using bookmarks. I want a label that will always point
to the last known head on the server, so e.g
`hg update central && hg commit -m "Foo"` should not move 'central'.
Using bookmarks to track the remote heads is also incompatible with my
bookbinder extension which I've come to rely quite heavily on. This
would be a personal blocker for me to make the switch.

Maybe firefoxtree could be adapted to work with this new repo as well.
Or maybe I could look into doing something with remotenames.

When pulling from these experimental repos with the latest version of firefoxtree, the bookmarks on these repos will be converted to firefoxtree's special namespace and local bookmarks matching what were formally pulled from these repos will be deleted. i.e. it works just like firefoxtree does with the canonical repos which don't have bookmarks.

You will need firefoxtree from version-control-tools as of about ~15 minutes ago to get this new feature.

[Eric Shepherd]

This has potential to be a Huge Deal for the MDN content team, too. Figuring out when things were added or removed or changed can be hard across so many repositories.

I'm pleased to announce the immediate availability of some *experimental* read-only Mercurial repositories containing the combined, useful history of the various Firefox repositories, all in chronological order and stored in a more efficient format that is faster to clone and pull from and results in faster client operations.

The repositories can be found at https://hg.mozilla.org/experimental. The repository you likely want to clone is https://hg.mozilla.org/experimental/firefox-unified. A visualization showing the chronological history of the repo can be seen at https://hg.mozilla.org/experimental/firefox-unified/graph.

The primary goal of these repositories is to provide developers (and eventually automation) with more efficient interaction with the Firefox source repositories. There are several secondary and side-benefits, including improving the scalability of Try and MozReview's repositories.

--

Eric Shepherd
Senior Technical Writer
Mozilla Developer Network
Blog: https://www.bitstampede.com/
Twitter: http://twitter.com/sheppy
Doodle: http://doodle.com/the.sheppy

[Eric Shepherd]

For what it's worth it would be an absolute game changer for the developer docs team. I already commented that having the unified repo is a big deal, but having it go all the way to the beginning would rock our worlds.

Having to wade through by hand and basically do a manual binary search to figure out which release a change took place in can take a lot of our time. If we could get this all in one place -- holy cow, I would be bursting with happy sparkly rainbow kittens!

No. Adding a Mercurial repo with CVS history isn't on the priority list. Make a case for it improving developer productivity and it could be.

[Gregory Szorc]

The unified Firefox repo has now been moved out of experimental status and is available at https://hg.mozilla.org/mozilla-unified. The repository is read-only and contains most of the commits from the main Firefox repos in chronological order.

Pre-generated bundles for the repo are available on our CDN (https://hg.cdn.mozilla.net/) and are 300MB smaller than mozilla-central.

When converting to the unified repo, I recommend using the one-line bootstrapper to clone the repo:

$ wget https://hg.mozilla.org/mozilla-central/raw-file/tip/python/mozboot/bin/bootstrap.py

$ python2.7 bootstrap.py

(The bootstrapper now runs the [recently rewritten] Mercurial config wizard and offers to clone the repo using optimal settings. But it only offers to clone if you aren't in a checkout, so you need to run the bootstrapper standalone - not via `mach bootstrap`.)

More docs for working with the unified repo are available at https://mozilla-version-control-tools.readthedocs.io/en/latest/hgmozilla/unifiedrepo.html.

The experimental URL still works (but is a slightly different repo). If you have already cloned/pulled from https://hg.mozilla.org/experimental/firefox-unified, you can simply update your .hg/hgrc's [paths] to point to https://hg.mozilla.org/mozilla-unified.

(Note: https://hg.mozilla.org/firefox was the initial planned location for the unified repo and some of you may have started using it. There was a last minute change of name to "mozilla-unified." https://hg.mozilla.org/firefox is no longer updating and will eventually be deleted.)

[Mike Hommey]

On Mon, Jun 27, 2016 at 02:23:52PM -0700, Gregory Szorc wrote:
> The unified Firefox repo has now been moved out of experimental status and
> is available at https://hg.mozilla.org/mozilla-unified

> <https://hg.mozilla.org/firefox>.

Which one is it? firefox/ or mozilla-unified/? Both exist and don't seem
to contain the same things.

> (Note: https://hg.mozilla.org/firefox was the initial planned location for
> the unified repo and some of you may have started using it. There was a
> last minute change of name to "mozilla-unified."
> https://hg.mozilla.org/firefox is no longer updating and will eventually be
> deleted.)

Okay, so here's the answer, but why mention both?

Mike

[Mike Hommey]

On Tue, Jun 28, 2016 at 07:36:52AM +0900, Mike Hommey wrote:
> On Mon, Jun 27, 2016 at 02:23:52PM -0700, Gregory Szorc wrote:
> > The unified Firefox repo has now been moved out of experimental status and
> > is available at https://hg.mozilla.org/mozilla-unified
> > <https://hg.mozilla.org/firefox>.
>
> Which one is it? firefox/ or mozilla-unified/? Both exist and don't seem
> to contain the same things.

BTW, firefox/ has bundles on hg.cdn.mozilla.net, but mozilla-unified/ doesn't.

[Gregory Szorc]

On Mon, Jun 27, 2016 at 3:36 PM, Mike Hommey <m...@glandium.org> wrote:

On Mon, Jun 27, 2016 at 02:23:52PM -0700, Gregory Szorc wrote:
> The unified Firefox repo has now been moved out of experimental status and
> is available at https://hg.mozilla.org/mozilla-unified
> <https://hg.mozilla.org/firefox>.

Which one is it? firefox/ or mozilla-unified/? Both exist and don't seem
to contain the same things.

> (Note: https://hg.mozilla.org/firefox was the initial planned location for
> the unified repo and some of you may have started using it. There was a
> last minute change of name to "mozilla-unified."
> https://hg.mozilla.org/firefox is no longer updating and will eventually be
> deleted.)

Okay, so here's the answer, but why mention both?

This was my email client (gmail web) not changing the link when I changed the URL text of the draft message I composed 2 weeks ago :/

And the bundle issue is known. mozilla-unified will use the bundles for the "firefox" repo for another few hours until the bundles are regenerated.