Initial deployment

[mrc...@gmail.com]

The Bugzilla integration is just about complete; we just need to polish the comment-diff mirroring, and we should be good.

I am, however, still confused by the hg-review-repo side of things. It's pretty far outside my domain, much more of a releng speciality than A-Team. Does https://github.com/laggyluke/reviewboard-mercurial-hook#readme contain everything we need from that side? What kind of disk space do we need? When we set up reviewboard.allizom.org and reviewboard-dev.allizom.org around 6 months ago, the idea of having an hg review repository wasn't on the radar, so we probably don't have much disk space available right now. Now that fubar is back from PTO, if we need something large, we need to start setting it up now.

What kind of maintenance do we need going forward? I noticed a thread on hg scaling in this group, but I didn't really see any conclusion. Unless we are treating the initial deployment as a temporary throwaway solution (and I don't think we are), we need to deal with maintenance questions now. As gps remarked, the try server has caused a lot of headaches with its thousands of heads, and I'm pretty sure no one in releng wants to go down that path again unless we have a really solid plan. I know we want a minimum viable product, but I consider some plan for scalability to be part of that minimum; putting this off until later will cause more pain in the long run.

Mark

[Gregory Szorc]

We should install Mercurial 3.0 with generaldelta revlog encoding. We
should plan to upgrade the server quickly when new releases come out so
we can have the server running bundle2 ASAP.

There are scaling problems with thousands of heads in Mercurial.
However, I've identified the likely bottleneck and I'm optimistic the
situation will improve with Mercurial 3.1 or 3.2 in the next 6 months.
When those planned fixes are in place, repos should scale past tens of
thousands of heads and we shouldn't need to periodically reset repos.
Though, I still think bundles might be a better distribution mechanism.
That's a discussion we should have down the road.

[Mark Côté]

Six months sounds like a long time. Do you think it very unlikely that
we'll run into the too-many-heads problem before then? As a contingency
plan, we could reset repos, but will that break existing reviews?

How much disk space do you think is reasonable?

Mark

[Gregory Szorc]

I believe resetting repos may break existing reviews.

I have a ~26,000 head clone of the Try repo that is consuming ~3 GB of
storage. I can't imagine the review repo growing beyond 10 GB in the
next few months.

Also, please put the repos on an SSD if possible. This especially holds
true if using generaldelta encoding. But if generaldelta isn't used, the
repos will grow in size much quicker (we're taking 20-40% more storage
per push).

Yes, I think we may hit the bottleneck before Mercurial 3.1 or 3.2. We
can alleviate things by limiting the number of hooks and rollbacks that
occur.

[Ehsan Akhgari]

Resetting repos is definitely going to be a problem.  The one and only requirement of the hg hosting side that I know of is for the published changesets to live forever as they would essentially be the data backing our code review data.  Taras previously said that this is on his radar, any updates on what we want to do here, Taras?

On the issue of the hg repos, I'm of course fine with running the latest and greatest version of hg, but we should also note that we've been talking about upgrading the hg version on hg.m.o for a number of years now (I can confidently say 2, but I'm probably underestimating) and it hasn't happened yet, so I'd rather us to keep that in mind before putting all of our eggs in that basket.  I'd defer to Taras on that issue too, of course.

Cheers,

--
Ehsan
<http://ehsanakhgari.org/>

--
You received this message because you are subscribed to the Google Groups "mozilla-code-review" group.
To unsubscribe from this group and stop receiving emails from it, send an email to mozilla-code-review+unsub...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Gregory Szorc]

It's my understanding that we haven't upgraded hg.m.o due to a lot of
FUD and lack of test coverage for our hooks and extensions there. IMO
the code review repo is its own deployment and should be looked at
independently. If starting a new HG server today, I would be comfortable
using 3.0. If I were really paranoid, I'd do 2.9.2. The paranoid me
would feel comfortable running 3.0.1 when that is released in the next
week or so.

On 5/27/14, 11:49 AM, Ehsan Akhgari wrote:
> Resetting repos is definitely going to be a problem. The one and only
> requirement of the hg hosting side that I know of is for the published
> changesets to live forever as they would essentially be the data backing
> our code review data. Taras previously said that this is on his radar,
> any updates on what we want to do here, Taras?
>
> On the issue of the hg repos, I'm of course fine with running the latest
> and greatest version of hg, but we should also note that we've been
> talking about upgrading the hg version on hg.m.o for a number of years
> now (I can confidently say 2, but I'm probably underestimating) and it
> hasn't happened yet, so I'd rather us to keep that in mind before
> putting all of our eggs in that basket. I'd defer to Taras on that
> issue too, of course.
>
> Cheers,
>
> --
> Ehsan
> <http://ehsanakhgari.org/>
>
>
> On Tue, May 27, 2014 at 1:48 PM, Gregory Szorc <g...@mozilla.com
> <mailto:g...@mozilla.com>> wrote:
>
> On 5/27/14, 10:39 AM, Mark Côté wrote:
>
> On 2014-05-27, 1:31 PM, Gregory Szorc wrote:
>
> On 5/27/14, 10:22 AM, mrc...@gmail.com

> <mailto:mrc...@gmail.com> wrote:
>
> The Bugzilla integration is just about complete; we just
> need to
> polish the comment-diff mirroring, and we should be good.
>
> I am, however, still confused by the hg-review-repo side
> of things.
> It's pretty far outside my domain, much more of a releng
> speciality
> than A-Team. Does

> https://github.com/laggyluke/__reviewboard-mercurial-hook#__readme

> <https://github.com/laggyluke/reviewboard-mercurial-hook#readme>
> contain everything we need from that side? What kind of
> disk space
> do we need? When we set up reviewboard.allizom.org

> <http://reviewboard.allizom.org> and
> reviewboard-dev.allizom.org
> <http://reviewboard-dev.allizom.org> around 6 months

> send an email to mozilla-code-rev...@googlegroups.com
> <mailto:mozilla-code-review%2Bunsu...@googlegroups.com>.
> For more options, visit https://groups.google.com/d/__optout
> <https://groups.google.com/d/optout>.
>
>

[Taras Glek]

Ehsan Akhgari wrote:

Resetting repos is definitely going to be a problem.  The one and only requirement of the hg hosting side that I know of is for the published changesets to live forever as they would essentially be the data backing our code review data.  Taras previously said that this is on his radar, any updates on what we want to do here, Taras?

On the issue of the hg repos, I'm of course fine with running the latest and greatest version of hg, but we should also note that we've been talking about upgrading the hg version on hg.m.o for a number of years now (I can confidently say 2, but I'm probably underestimating) and it hasn't happened yet, so I'd rather us to keep that in mind before putting all of our eggs in that basket.  I'd defer to Taras on that issue too, of course.

Don't have a good update here. My understanding is that resetting has been done before mainly due to respository corruption. As far as I know that was due to people not spending the time to debug the "wedged hg" problems when resetting was a viable option.

We need to stop doing that and the problem will work itself out. :)

Taras

Cheers,

--
Ehsan
<http://ehsanakhgari.org/>

To unsubscribe from this group and stop receiving emails from it, send an email to mozilla-code-re...@googlegroups.com.

[Hal Wine]

Just coming to this thread, so my apologies if I get the context incorrect.

On 2014-05-27, 13:12 , Taras Glek wrote:

Ehsan Akhgari wrote:

Resetting repos is definitely going to be a problem.  The one and only requirement of the hg hosting side that I know of is for the published changesets to live forever as they would essentially be the data backing our code review data.  Taras previously said that this is on his radar, any updates on what we want to do here, Taras?

Which repos are we talking about resetting? If only "try", then as I understand it, holding a 2 week history is "sufficient", and we have plans to support that on try.

If we're talking about resetting any other repos, especially m-c, m-a, m-b, m-r or any of the b2g variants, we need some more discussion to ensure we don't mess with the downstream b2g partners view of gecko.git.

On the issue of the hg repos, I'm of course fine with running the latest and greatest version of hg, but we should also note that we've been talking about upgrading the hg version on hg.m.o for a number of years now (I can confidently say 2, but I'm probably underestimating) and it hasn't happened yet, so I'd rather us to keep that in mind before putting all of our eggs in that basket.  I'd defer to Taras on that issue too, of course.

Don't have a good update here. My understanding is that resetting has been done before mainly due to respository corruption. As far as I know that was due to people not spending the time to debug the "wedged hg" problems when resetting was a viable option.

The issues blocking hg server software upgrade were largely "no one owned hg" (imo), so resources were not aligned across the groups in a timely fashion. Now Laura owns repos, so "things have changed".

We did get in one server upgrade last year. And, from discussions with a "major hg user" (facebook), it seems like we'll be able to fasttrack upgrades withing a point release of stable. I'm going to go out on a limb here, and say we can do an hg server upgrade in Q3.

As for the "wedged" problems -- we have gotten some decent debugging (thanks to Glandium) and theories (thanks to GPS) within the last month. Much more is known, so we can say with confidence that:
 a) there is no silver bullet server release
 b) staying closer to "tip" will help
 c) preventing commiter's from interrupting a commit will really help (social and technical solutions available)

--Hal

[Ehsan Akhgari]

On Tue, May 27, 2014 at 4:30 PM, Hal Wine <hw...@mozilla.com> wrote:

Just coming to this thread, so my apologies if I get the context incorrect.

On 2014-05-27, 13:12 , Taras Glek wrote:

Ehsan Akhgari wrote:

Resetting repos is definitely going to be a problem.  The one and only requirement of the hg hosting side that I know of is for the published changesets to live forever as they would essentially be the data backing our code review data.  Taras previously said that this is on his radar, any updates on what we want to do here, Taras?

Which repos are we talking about resetting? If only "try", then as I understand it, holding a 2 week history is "sufficient", and we have plans to support that on try.

If we're talking about resetting any other repos, especially m-c, m-a, m-b, m-r or any of the b2g variants, we need some more discussion to ensure we don't mess with the downstream b2g partners view of gecko.git.

To give a bit of a background on what we're trying to achieve here, we're working on a new code review workflow that replaces the current workflow of attaching textual diffs to bugs to pushing your changes to a remote code repository and have the review tool look at that code repository to get the information about the code being reviewed, etc. from there.  The hg repo that I'm talking about here would be conceptually similar to the try repo in that people will push their local trees to that repo so it's separate from m-c/m-a/etc. but we do need to ensure that the changesets that are pushed to that repo will persist forever because otherwise going back to a code review of 6 months ago would not work since you would be asking for a changeset which no longer exists on that repo.  So we won't be able to reset that repo if it gets affected by similar problems to the perf issues that we've been struggling with on try.

On the issue of the hg repos, I'm of course fine with running the latest and greatest version of hg, but we should also note that we've been talking about upgrading the hg version on hg.m.o for a number of years now (I can confidently say 2, but I'm probably underestimating) and it hasn't happened yet, so I'd rather us to keep that in mind before putting all of our eggs in that basket.  I'd defer to Taras on that issue too, of course.

Don't have a good update here. My understanding is that resetting has been done before mainly due to respository corruption. As far as I know that was due to people not spending the time to debug the "wedged hg" problems when resetting was a viable option.

The issues blocking hg server software upgrade were largely "no one owned hg" (imo), so resources were not aligned across the groups in a timely fashion. Now Laura owns repos, so "things have changed".

We did get in one server upgrade last year. And, from discussions with a "major hg user" (facebook), it seems like we'll be able to fasttrack upgrades withing a point release of stable. I'm going to go out on a limb here, and say we can do an hg server upgrade in Q3.

As for the "wedged" problems -- we have gotten some decent debugging (thanks to Glandium) and theories (thanks to GPS) within the last month. Much more is known, so we can say with confidence that:
 a) there is no silver bullet server release
 b) staying closer to "tip" will help
 c) preventing commiter's from interrupting a commit will really help (social and technical solutions available)

Thanks!  So based on the above, do you think we would be able to host a code repository system where we persist the changesets pushed to it forever?

Cheers,
Ehsan

[Hal Wine]

On 2014-05-27, 15:04 , Ehsan Akhgari wrote:

On Tue, May 27, 2014 at 4:30 PM, Hal Wine <hw...@mozilla.com> wrote:

Just coming to this thread, so my apologies if I get the context incorrect.

On 2014-05-27, 13:12 , Taras Glek wrote:

Ehsan Akhgari wrote:

Resetting repos is definitely going to be a problem.  The one and only requirement of the hg hosting side that I know of is for the published changesets to live forever as they would essentially be the data backing our code review data.  Taras previously said that this is on his radar, any updates on what we want to do here, Taras?

Which repos are we talking about resetting? If only "try", then as I understand it, holding a 2 week history is "sufficient", and we have plans to support that on try.

If we're talking about resetting any other repos, especially m-c, m-a, m-b, m-r or any of the b2g variants, we need some more discussion to ensure we don't mess with the downstream b2g partners view of gecko.git.

To give a bit of a background on what we're trying to achieve here, we're working on a new code review workflow that replaces the current workflow of attaching textual diffs to bugs to pushing your changes to a remote code repository and have the review tool look at that code repository to get the information about the code being reviewed, etc. from there.  The hg repo that I'm talking about here would be conceptually similar to the try repo in that people will push their local trees to that repo so it's separate from m-c/m-a/etc. but we do need to ensure that the changesets that are pushed to that repo will persist forever because otherwise going back to a code review of 6 months ago would not work since you would be asking for a changeset which no longer exists on that repo.  So we won't be able to reset that repo if it gets affected by similar problems to the perf issues that we've been struggling with on try.

On the issue of the hg repos, I'm of course fine with running the latest and greatest version of hg, but we should also note that we've been talking about upgrading the hg version on hg.m.o for a number of years now (I can confidently say 2, but I'm probably underestimating) and it hasn't happened yet, so I'd rather us to keep that in mind before putting all of our eggs in that basket.  I'd defer to Taras on that issue too, of course.

Don't have a good update here. My understanding is that resetting has been done before mainly due to respository corruption. As far as I know that was due to people not spending the time to debug the "wedged hg" problems when resetting was a viable option.

The issues blocking hg server software upgrade were largely "no one owned hg" (imo), so resources were not aligned across the groups in a timely fashion. Now Laura owns repos, so "things have changed".

We did get in one server upgrade last year. And, from discussions with a "major hg user" (facebook), it seems like we'll be able to fasttrack upgrades withing a point release of stable. I'm going to go out on a limb here, and say we can do an hg server upgrade in Q3.

As for the "wedged" problems -- we have gotten some decent debugging (thanks to Glandium) and theories (thanks to GPS) within the last month. Much more is known, so we can say with confidence that:
 a) there is no silver bullet server release
 b) staying closer to "tip" will help
 c) preventing commiter's from interrupting a commit will really help (social and technical solutions available)

Thanks!  So based on the above, do you think we would be able to host a code repository system where we persist the changesets pushed to it forever?

Knowing that "keep 10's of thousands of heads forever" is a requirement is a good start to achieving that goal. I'm late to the game on this topic, so before I promise anything, let me get with some folks and get up to speed. I'm 99% confident that, for some definition of "keep", it's technically doable. (E.g. stable urls, but slower response for changesets older than X months, etc.)

--Hal

[Gregory Szorc]

Someone please follow up with me offline about this scaling problem.
There's too much context that I don't want to take the time to type up.
I would be glad to brain dump in a meeting though.

In addition to what I've typed so far, head exchange in Mercurial will
eventually make pushing bandwidth prohibitive. e.g. 10,000 heads
requires a 400+ KB blob be transferred from the server. This could be
fixed in a future Mercurial version, but it's not being planned. FWIW,
Git has the same problem (albeit worse since refs have names).

[Ehsan Akhgari]

Note that we do not require to persist the "heads" but we do need to persist the changesets.  So for example a solution which merges the heads in a dumb way to just reduce the number of heads will work fine.  Or any other variation of that idea...

Cheers,

--
Ehsan
<http://ehsanakhgari.org/>

--
You received this message because you are subscribed to the Google Groups "mozilla-code-review" group.

To unsubscribe from this group and stop receiving emails from it, send an email to mozilla-code-review+unsub...@googlegroups.com.

[Mike Hommey]

On Tue, May 27, 2014 at 08:06:18PM -0400, Ehsan Akhgari wrote:
> Note that we do not require to persist the "heads" but we do need to
> persist the changesets. So for example a solution which merges the heads
> in a dumb way to just reduce the number of heads will work fine. Or any
> other variation of that idea...

Having a dummy branch where everything is merged into with something like
http://mercurial.selenic.com/wiki/TipsAndTricks#mergemineortheir , which
is equivalent to git merge -s ours, would avoid deltas growing out of
proportion without generaldelta.

Mike

[Ehsan Akhgari]

I'm very glad to hear that!  Out of curiosity, is there a technical reason why we have not done this for try?  Is it just that we haven't gotten around to doing it yet?

But I think this answers the question of whether we can make hosting this repo work.  Do you agree, Hal?

Thanks!

--
Ehsan
<http://ehsanakhgari.org/>

--
You received this message because you are subscribed to the Google Groups "mozilla-code-review" group.

To unsubscribe from this group and stop receiving emails from it, send an email to mozilla-code-re...@googlegroups.com.

[Mike Hommey]

On Tue, May 27, 2014 at 09:34:13PM -0400, Ehsan Akhgari wrote:
> I'm very glad to hear that! Out of curiosity, is there a technical reason
> why we have not done this for try? Is it just that we haven't gotten
> around to doing it yet?

Presumably because we hit other problems before the number of heads
becomes one.

Mike

[Gregory Szorc]

I still think using generaldelta is advised. The main reason you don't
want to use generaldelta today is because of adverse performance impact
on large push/pull operations. Those concerns will go away with bundle2
(or at least be moved off the server to clients). Furthermore, the
review repository should never be cloned, so I don't believe it has this
concern to begin with. It differs from Try in this regard (although if
our automation seeded clones with a bundle and then performed an
incremental pull from Try things would be different).

We want to optimize the review repository for a) fast pushes b) fast
(small) reads (as opposed to clones). I've profiled Mercurial and
manifest resolution is always the slowest part. Specifically, zlib
decompression (or compression for large pushes) accounts for most of the
CPU time, even for things like exporting a single changeset. Both
generaldelta and lz4 compression (via Facebook's lz4revlog extension)
help speed up manifest resolution. The former reduces the size of deltas
and the amount of data to be decompressed. The latter is a faster
compression algorithm (at the expense of storage space). Both negatively
impact push/pull times, but I don't think it matters for our use case
(only small pushes).

[Hal Wine]

Folks -- let's keep this thread to a single topic. In this case the topic is supporting the rollout of Review Board into a staging environment.

On 2014-05-27, 18:34 , Ehsan Akhgari wrote:

I'm very glad to hear that!  Out of curiosity, is there a technical reason why we have not done this for try?  Is it just that we haven't gotten around to doing it yet?

Please take to another thread - there are reasons.

But I think this answers the question of whether we can make hosting this repo work.  Do you agree, Hal?

I do not agree. In the past, the dummy branch method proposed below was implemented with the "try repo" and failed to yield improvements (and was then removed). That was sever server versions ago, and by all accounts, things may have changed. But as far as I know, this technique has not been shown to improve performance in a situation where the performance problem had been recreated.

As I said in a previous email, I will get with some folks and get back to this group. I did leave out a time frame. I'll get back to this group no later than Friday, June 6. Between now and then, I'll be meeting with:
 - dev services, who have hard data on current infrastructure capacity and alternatives
 - GPS, who has replicated the "try repo" performance problem, and has had long discussion with the Hg dev team.
 - someone from the Review Board program who can speak to what the plans are for the staging environment.

Ehsan - can you provide me with that last item? Or point me to whomever can?

Thanks,
--Hal

One of my favorites: http://www.c2.com/cgi/wiki?DifferenceBetweenTheoryAndPractice

[Ehsan Akhgari]

On Tue, May 27, 2014 at 10:39 PM, Hal Wine <hw...@mozilla.com> wrote:

Folks -- let's keep this thread to a single topic. In this case the topic is supporting the rollout of Review Board into a staging environment.

On 2014-05-27, 18:34 , Ehsan Akhgari wrote:

I'm very glad to hear that!  Out of curiosity, is there a technical reason why we have not done this for try?  Is it just that we haven't gotten around to doing it yet?

Please take to another thread - there are reasons.

Sure.  That was a tangent.

But I think this answers the question of whether we can make hosting this repo work.  Do you agree, Hal?

I do not agree. In the past, the dummy branch method proposed below was implemented with the "try repo" and failed to yield improvements (and was then removed). That was sever server versions ago, and by all accounts, things may have changed. But as far as I know, this technique has not been shown to improve performance in a situation where the performance problem had been recreated.

As I said in a previous email, I will get with some folks and get back to this group. I did leave out a time frame. I'll get back to this group no later than Friday, June 6. Between now and then, I'll be meeting with:
 - dev services, who have hard data on current infrastructure capacity and alternatives
 - GPS, who has replicated the "try repo" performance problem, and has had long discussion with the Hg dev team.
 - someone from the Review Board program who can speak to what the plans are for the staging environment.

Ehsan - can you provide me with that last item? Or point me to whomever can?

I would say that person is me.  :-)  Unless you mean the Review Board project, in which case I can reach out to them as well.  Please let me know what you'd like to know about the plans for staging environment.

Taras, in the interest of moving this forward further, do you think we can do something sooner than June 6th?

Thanks!

Ehsan

[Ehsan Akhgari]

I don't claim to understand half of what's involved in this proposal, but from my basic understanding what you're saying makes sense.  So if this is feasible and won't overly complicate things, let's do all of this!  :-)

Cheers,
Ehsan

[Mike Hommey]

On Tue, May 27, 2014 at 07:34:33PM -0700, Gregory Szorc wrote:
> On 5/27/2014 6:30 PM, Mike Hommey wrote:
> > On Tue, May 27, 2014 at 08:06:18PM -0400, Ehsan Akhgari wrote:
> >> Note that we do not require to persist the "heads" but we do need to
> >> persist the changesets. So for example a solution which merges the heads
> >> in a dumb way to just reduce the number of heads will work fine. Or any
> >> other variation of that idea...
> >
> > Having a dummy branch where everything is merged into with something like
> > http://mercurial.selenic.com/wiki/TipsAndTricks#mergemineortheir , which
> > is equivalent to git merge -s ours, would avoid deltas growing out of
> > proportion without generaldelta.
>
> I still think using generaldelta is advised. The main reason you don't
> want to use generaldelta today is because of adverse performance impact
> on large push/pull operations. Those concerns will go away with bundle2
> (or at least be moved off the server to clients). Furthermore, the
> review repository should never be cloned, so I don't believe it has this
> concern to begin with. It differs from Try in this regard (although if
> our automation seeded clones with a bundle and then performed an
> incremental pull from Try things would be different).

That's what they already do with hgtool, using the try bundle:
https://ftp-ssl.mozilla.org/pub/mozilla.org/firefox/bundles/try.hg

Mike

[Hal Wine]

[Moving most to bcc]

On 2014-05-27, 19:49 , Ehsan Akhgari wrote:

On Tue, May 27, 2014 at 10:39 PM, Hal Wine <hw...@mozilla.com> wrote:

Folks -- let's keep this thread to a single topic. In this case the topic is supporting the rollout of Review Board into a staging environment.

On 2014-05-27, 18:34 , Ehsan Akhgari wrote:

I'm very glad to hear that!  Out of curiosity, is there a technical reason why we have not done this for try?  Is it just that we haven't gotten around to doing it yet?

Please take to another thread - there are reasons.

Sure.  That was a tangent.

But I think this answers the question of whether we can make hosting this repo work.  Do you agree, Hal?

I do not agree. In the past, the dummy branch method proposed below was implemented with the "try repo" and failed to yield improvements (and was then removed). That was sever server versions ago, and by all accounts, things may have changed. But as far as I know, this technique has not been shown to improve performance in a situation where the performance problem had been recreated.

As I said in a previous email, I will get with some folks and get back to this group. I did leave out a time frame. I'll get back to this group no later than Friday, June 6. Between now and then, I'll be meeting with:
 - dev services, who have hard data on current infrastructure capacity and alternatives
 - GPS, who has replicated the "try repo" performance problem, and has had long discussion with the Hg dev team.
 - someone from the Review Board program who can speak to what the plans are for the staging environment.

Ehsan - can you provide me with that last item? Or point me to whomever can?

I would say that person is me.  :-)  Unless you mean the Review Board project, in which case I can reach out to them as well.  Please let me know what you'd like to know about the plans for staging environment.

Ehsan, my interest here is ensuring we all share the same expectations (please feel free to point me to any docs that cover this, if they exist). It may make more sense to have a chat/vidyo about this subject. Basically what I know at this time is standing up the staging environment is a Q2 goal, and the staging environment needs an hg repository. And our repository creation process routes via RelEng (usually me) per https://wiki.mozilla.org/ReleaseEngineering/RepositoryCreationRequest. This is a "first of it's kind" use case for source control at Mozilla, and hence there are some additional questions.

Here are some open questions I have:
 - is this staging-to-be-used-as-POC, or staging-to-work-out-operational-issues-before-production-deployment?
 - how many devs will use the staging environment?
 - is there a planned duration?
 - what are the goals of the staging? Feature tests, UX tests, performance tests, ???
 - what level of support are you planning on needing? (set & forget; or frequent repository resets; or "on call"?)
 - is there a business requirement to host the Review Board repository in the same URL space as served by hg.m.o?
 - are there any special security or privacy issues with the contents of this repository?
 - do devs need to interact with this repository directly, or is all interaction via the review board tooling and UI? ssh only, or web view required?

Here are some assumptions I'm making -- I could be completely wrong on a number of them -- feel free to correct me. The goal is to get to agreement, not for me to have guessed accurately. :)
 - there is only one repository instance needed in staging or production
 - the repository is completely unstructured (i.e. it can be used for any code review, not just those that are "mozilla-central" based - multiple roots)
 - the staging environment will not have any production data in it. (i.e. no requirements for backups)
 - the staging environment would not be migrated to production - the production instances will be rebuilt with empty databases & repositories as applicable
 - any urls into the staging environment will not be preserved in production. (for sure, the host name will differ.)

If this is a completely separate repository, with quite separate needs, it may make more sense to have it on it's own server, and not be tied to some of the constraints most of hg.m.o operates under. That's an example of what I'm hoping to learn from these questions to find an optimal solution.

Hoping we can get you unblocked quickly,
--Hal

[Taras Glek]

Ehsan Akhgari

Tuesday, May 27, 2014 19:49

On Tue, May 27, 2014 at 10:39 PM, Hal Wine <hw...@mozilla.com> wrote:

Folks -- let's keep this thread to a single topic. In this case the topic is supporting the rollout of Review Board into a staging environment.

On 2014-05-27, 18:34 , Ehsan Akhgari wrote:

I'm very glad to hear that!  Out of curiosity, is there a technical reason why we have not done this for try?  Is it just that we haven't gotten around to doing it yet?

Please take to another thread - there are reasons.

Sure.  That was a tangent.

But I think this answers the question of whether we can make hosting this repo work.  Do you agree, Hal?

I do not agree. In the past, the dummy branch method proposed below was implemented with the "try repo" and failed to yield improvements (and was then removed). That was sever server versions ago, and by all accounts, things may have changed. But as far as I know, this technique has not been shown to improve performance in a situation where the performance problem had been recreated.

As I said in a previous email, I will get with some folks and get back to this group. I did leave out a time frame. I'll get back to this group no later than Friday, June 6. Between now and then, I'll be meeting with:
 - dev services, who have hard data on current infrastructure capacity and alternatives
 - GPS, who has replicated the "try repo" performance problem, and has had long discussion with the Hg dev team.
 - someone from the Review Board program who can speak to what the plans are for the staging environment.

Ehsan - can you provide me with that last item? Or point me to whomever can?

I would say that person is me.  :-)  Unless you mean the Review Board project, in which case I can reach out to them as well.  Please let me know what you'd like to know about the plans for staging environment.

Taras, in the interest of moving this forward further, do you think we can do something sooner than June 6th?

What's the rush. I think we can rely on try as is during the testing period.

Hal Wine

Tuesday, May 27, 2014 19:39

Folks -- let's keep this thread to a single topic. In this case the topic is supporting the rollout of Review Board into a staging environment.

On 2014-05-27, 18:34 , Ehsan Akhgari wrote:

I'm very glad to hear that!  Out of curiosity, is there a technical reason why we have not done this for try?  Is it just that we haven't gotten around to doing it yet?

Please take to another thread - there are reasons.

But I think this answers the question of whether we can make hosting this repo work.  Do you agree, Hal?

I do not agree. In the past, the dummy branch method proposed below was implemented with the "try repo" and failed to yield improvements (and was then removed). That was sever server versions ago, and by all accounts, things may have changed. But as far as I know, this technique has not been shown to improve performance in a situation where the performance problem had been recreated.

As I said in a previous email, I will get with some folks and get back to this group. I did leave out a time frame. I'll get back to this group no later than Friday, June 6. Between now and then, I'll be meeting with:
 - dev services, who have hard data on current infrastructure capacity and alternatives
 - GPS, who has replicated the "try repo" performance problem, and has had long discussion with the Hg dev team.
 - someone from the Review Board program who can speak to what the plans are for the staging environment.

Ehsan - can you provide me with that last item? Or point me to whomever can?

Thanks,
--Hal

One of my favorites: http://www.c2.com/cgi/wiki?DifferenceBetweenTheoryAndPractice

Thanks!

--
Ehsan
<http://ehsanakhgari.org/>

On Tue, May 27, 2014 at 9:30 PM, Mike Hommey <glan...@mozilla.com> wrote:

On Tue, May 27, 2014 at 08:06:18PM -0400, Ehsan Akhgari wrote:
> Note that we do not require to persist the "heads" but we do need to
> persist the changesets.  So for example a solution which merges the heads
> in a dumb way to just reduce the number of heads will work fine.  Or any
> other variation of that idea...

Having a dummy branch where everything is merged into with something like
http://mercurial.selenic.com/wiki/TipsAndTricks#mergemineortheir , which
is equivalent to git merge -s ours, would avoid deltas growing out of
proportion without generaldelta.

Mike

--
You received this message because you are subscribed to the Google Groups "mozilla-code-review" group.
To unsubscribe from this group and stop receiving emails from it, send an email to mozilla-code-re...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Ehsan Akhgari

Tuesday, May 27, 2014 18:34

I'm very glad to hear that!  Out of curiosity, is there a technical reason why we have not done this for try?  Is it just that we haven't gotten around to doing it yet?

But I think this answers the question of whether we can make hosting this repo work.  Do you agree, Hal?

Thanks!

--
Ehsan
<http://ehsanakhgari.org/>

Mike Hommey

Tuesday, May 27, 2014 18:30

Having a dummy branch where everything is merged into with something like
http://mercurial.selenic.com/wiki/TipsAndTricks#mergemineortheir , which
is equivalent to git merge -s ours, would avoid deltas growing out of
proportion without generaldelta.

Mike

Ehsan Akhgari

Tuesday, May 27, 2014 17:06

Note that we do not require to persist the "heads" but we do need to persist the changesets.  So for example a solution which merges the heads in a dumb way to just reduce the number of heads will work fine.  Or any other variation of that idea...

Cheers,

--
Ehsan
<http://ehsanakhgari.org/>

[Ehsan Akhgari]

On Wed, May 28, 2014 at 12:40 PM, Taras Glek <tg...@mozilla.com> wrote:

Ehsan Akhgari

Tuesday, May 27, 2014 19:49

On Tue, May 27, 2014 at 10:39 PM, Hal Wine <hw...@mozilla.com> wrote:

Folks -- let's keep this thread to a single topic. In this case the topic is supporting the rollout of Review Board into a staging environment.

On 2014-05-27, 18:34 , Ehsan Akhgari wrote:

I'm very glad to hear that!  Out of curiosity, is there a technical reason why we have not done this for try?  Is it just that we haven't gotten around to doing it yet?

Please take to another thread - there are reasons.

Sure.  That was a tangent.

But I think this answers the question of whether we can make hosting this repo work.  Do you agree, Hal?

I do not agree. In the past, the dummy branch method proposed below was implemented with the "try repo" and failed to yield improvements (and was then removed). That was sever server versions ago, and by all accounts, things may have changed. But as far as I know, this technique has not been shown to improve performance in a situation where the performance problem had been recreated.

As I said in a previous email, I will get with some folks and get back to this group. I did leave out a time frame. I'll get back to this group no later than Friday, June 6. Between now and then, I'll be meeting with:
 - dev services, who have hard data on current infrastructure capacity and alternatives
 - GPS, who has replicated the "try repo" performance problem, and has had long discussion with the Hg dev team.
 - someone from the Review Board program who can speak to what the plans are for the staging environment.

Ehsan - can you provide me with that last item? Or point me to whomever can?

I would say that person is me.  :-)  Unless you mean the Review Board project, in which case I can reach out to them as well.  Please let me know what you'd like to know about the plans for staging environment.

Taras, in the interest of moving this forward further, do you think we can do something sooner than June 6th?

What's the rush. I think we can rely on try as is during the testing period.

We can't deploy the push hook on try, right?

[Hal Wine]

tl;dr: I stumbled on https://bugzil.la/1016006 and that changes the discussion entirely -- this repo _must_ be on a server other than hg.m.o -- it has to live on the review board servers.

Details and remaining discussion inline below.

On 2014-05-28, 06:53 , Ehsan Akhgari wrote:

On Wed, May 28, 2014 at 1:13 AM, Hal Wine <hw...@mozilla.com> wrote:

[Moving most to bcc]

Not sure why you dropped the public mailing list from the CC list.  I don't think anything in this email is confidential and cannot be discussed in public.  If that's true, please add the list back to the CC list.  Specifically there are people on the list who I'd like to verify what I'm saying below for accuracy.  Thanks!

Not confidential, but I viewed it as "getting Hal up to speed, so noise for everyone else". You did such a good job getting me up to speed, adding the list back in. Thanks!

 

On 2014-05-27, 19:49 , Ehsan Akhgari wrote:

On Tue, May 27, 2014 at 10:39 PM, Hal Wine <hw...@mozilla.com> wrote:

Folks -- let's keep this thread to a single topic. In this case the topic is supporting the rollout of Review Board into a staging environment.

On 2014-05-27, 18:34 , Ehsan Akhgari wrote:

I'm very glad to hear that!  Out of curiosity, is there a technical reason why we have not done this for try?  Is it just that we haven't gotten around to doing it yet?

Please take to another thread - there are reasons.

Sure.  That was a tangent.

But I think this answers the question of whether we can make hosting this repo work.  Do you agree, Hal?

I do not agree. In the past, the dummy branch method proposed below was implemented with the "try repo" and failed to yield improvements (and was then removed). That was sever server versions ago, and by all accounts, things may have changed. But as far as I know, this technique has not been shown to improve performance in a situation where the performance problem had been recreated.

As I said in a previous email, I will get with some folks and get back to this group. I did leave out a time frame. I'll get back to this group no later than Friday, June 6. Between now and then, I'll be meeting with:
 - dev services, who have hard data on current infrastructure capacity and alternatives
 - GPS, who has replicated the "try repo" performance problem, and has had long discussion with the Hg dev team.
 - someone from the Review Board program who can speak to what the plans are for the staging environment.

Ehsan - can you provide me with that last item? Or point me to whomever can?

I would say that person is me.  :-)  Unless you mean the Review Board project, in which case I can reach out to them as well.  Please let me know what you'd like to know about the plans for staging environment.

Ehsan, my interest here is ensuring we all share the same expectations (please feel free to point me to any docs that cover this, if they exist).

And, the docs, such as they exist for this are in https://bugzil.la/1016006 and it appears this repo already needs to be on the review board server, so can not be part of the regular cluster. (Just found this after responding below,  so some may read a bit odd.)

That means the following items need to be addressed:
 - what flows are needed to the review board server
  -- ssh to in same manner as hg.m.o
  -- https access? (advantages to not providing -- seee below)
  -- are the existing dev & staging instances large enough (ram, disk) to support both review board & hg
  -- some method for the review board project folks to access the box.
  -- name the repo -- since it's on the review board servers, I have zero interest in naming. More below.

It may make more sense to have a chat/vidyo about this subject. Basically what I know at this time is standing up the staging environment is a Q2 goal, and the staging environment needs an hg repository. And our repository creation process routes via RelEng (usually me) per https://wiki.mozilla.org/ReleaseEngineering/RepositoryCreationRequest. This is a "first of it's kind" use case for source control at Mozilla, and hence there are some additional questions.

Just to make sure that we're talking about the same thing, this is not going to be a staging environment in the usual sense of the word.  It can probably be more accurately described as an initial deployment of the tool in order to test it in practice with a small target audience. 

Okay, in our world, that is a HUGE difference. Thanks for clarifying

About the timelines, this might be a Q2 goal for some teams/people but from my perspective, I'd really like to move on this as fast as we can, since we're still in the process of building the tool.  The goal for the initial deployment is to get feedback from our test group and based on the feedback, we might need to go back and change various things in the tool.  So it's crucial to try to move things forward as fast as we can, and we've been doing that with reasonable success so far.

Agreed - Q2 goal means "no later than", but also means "we knew you were coming" (which is good).

Looking at https://wiki.mozilla.org/ReleaseEngineering/RepositoryCreationRequest, I'm not sure if I understand what the exact pieces for RelEng and IT look like exactly. 

And releng/it relationship is changing as we speak - for the better.

What we need here is an L1 hg repository where we have the option of installing our hg hooks on.  Additionally we'd like to ensure that the changesets pushed to this repo are not destroyed by a reset/etc.  In the future we'd like to do some of the things mentioned in the thread so far to ensure the repo's performance as it's being used by more people, but that is not required at this stage.  Also, we won't need to mirror this repo to git.

Now, on to answering your questions below.

 

Here are some open questions I have:
 - is this staging-to-be-used-as-POC, or staging-to-work-out-operational-issues-before-production-deployment?

I'd say more of the latter.  Except that I'm not sure what you mean by "operational issues".  We're mostly interested in feedback on the code review tools and workflows at this stage, nothing else.

Hal's definitions:
 - "dev" or "staging" - project team is responsible for operations, trouble shooting, etc.
 - "in production" means devs have documented everything well enough that IT can run things at the needed SLA
I.e. until IT can rebuild and run it on stock IT infrastructure, devs "carry the pager" (I still carry the pager for all the vcs-sync stuff).
Any change from that is a exception that should be negotiated with IT in advance.

 

 - how many devs will use the staging environment?

Part of Dave Camp's team.  According to phonebook, there are ~30 people reporting to him, not sure how many of those are targeted for this experiment.  CCing him.

 

 - is there a planned duration?

No.

 

 - what are the goals of the staging? Feature tests, UX tests, performance tests, ???

Testing our code review tools and workflows.  We're not interested in any repository level performance tests for now.

 

 - what level of support are you planning on needing? (set & forget; or frequent repository resets; or "on call"?)

Set and forget.  We might of course need to do things like deploy newer versions of our hook to the repo, etc.  I don't think we'll ever need a reset.

Good to know -- we'll see if we can make hook deployment self serve (perhaps not immediately).

 

 - is there a business requirement to host the Review Board repository in the same URL space as served by hg.m.o?

If you mean hosting it on hg.m.o, no.  I'd like to have it hosted wherever is faster and easier.

 

 - are there any special security or privacy issues with the contents of this repository?

No.

 

 - do devs need to interact with this repository directly, or is all interaction via the review board tooling and UI? ssh only, or web view required?

Developers will push to this repository through hg push.  ssh access is obviously required for pushing.  Web view is helpful but not required.  But this is something that the ReviewBoard devs who read the public list should confirm -- I'm not sure how ReviewBoard accesses hg (it might be through web view).

Http tickles the poor performance parts of many headed repos. (http headers get large, which makes load balancing harder, etc...)  If we can avoid the need for multiple web heads to serve devs, things will be simpler to operate.

I'd like to encourage folks to try without http access. I.e. embrace the new workflow entirely to see if it works. But, of course, this is the team's decision. It just may bring scaling problems later, or constrain the set of options to scaling.

 

Here are some assumptions I'm making -- I could be completely wrong on a number of them -- feel free to correct me. The goal is to get to agreement, not for me to have guessed accurately. :)
 - there is only one repository instance needed in staging or production

Yes, we won't need a separate repository when we open this up to a broader audience.

 

 - the repository is completely unstructured (i.e. it can be used for any code review, not just those that are "mozilla-central" based - multiple roots)

It will only be used for mozilla-central based code reviews, not arbitrary other projects.  Not sure what you mean by multiple roots.

Then there are no multiple roots. (unrelated trees in the same repository implies multiple roots).

New assumption: project team will write any needed hooks to enforce "mozilla-central" based if needed.

 

 - the staging environment will not have any production data in it. (i.e. no requirements for backups)

We need to ensure the persistence of the data in the repository somehow.  If this can be done by the usual Mozilla hosted backups (not sure what our existing practices for hg backups look like) then great!  If that might add multiple weeks (or months) to the timeline here, I'd be perfectly happy with an externally hosted backup solution (which might just be a bunch of people keeping updated clones of this repo on their machines.)

This will need some more discussion -- as you say there are "crowd sourced" solutions that will work for now, but also various requirements on location of data-to-be-backed-up for when it goes to production.

Also, since this is a dedicated repository for review board, any discussion of "keep forever", vs pruning, needs to be worked out with them. It is very expensive to engineer and maintain workarounds as products evolve when they are linked

 

 - the staging environment would not be migrated to production - the production instances will be rebuilt with empty databases & repositories as applicable

That's not correct.  We would definitely like to preserve the data in the system when moving this to a broader audience.

yep - that follows from the clarification of purpose, which I wanted to tease out. check.

 

 - any urls into the staging environment will not be preserved in production. (for sure, the host name will differ.)

While changing the URLs won't be the end of the world, it's unnecessary pain for the users which I'd like to avoid if possible.

Okay, this may take some planning. However, it sounds like it that can be a quick phase 2 to stabilize URLs. This most likely depends on exactly what the review board folks support in terms of old/new writable repositories.

 

If this is a completely separate repository, with quite separate needs, it may make more sense to have it on it's own server, and not be tied to some of the constraints most of hg.m.o operates under. That's an example of what I'm hoping to learn from these questions to find an optimal solution.

I don't know what those constraints are, so hopefully the above would make it obvious whether this is the case.

Yes - at the moment, I don't see any business requirement to have this repository tied to the main server at all. My mental model is shifting to "the repo is just yet another database instance used by review board". IT runs multiple database servers for this purpose, we should consider running this as a completely separate instance, as long as we can define the flows needed (and I think we can).

And, when I found that bug, that's exactly what is needed. :)

 

Hoping we can get you unblocked quickly,
--Hal

Thanks!
Ehsan

[Hal Wine]

Clarifying -- all further discussion & questions should be in https://bugzil.la/1016006. I've transfered the requirements from this thread to comment 3 -- corrections welcome in the bug.