Making "external" patch contributions easier
Zack Weinberg
problems faced by new (or not-so-new but not-so-integrated into the
team, either) contributors.
Suppose we set up a secondary Hg server with a web interface more or
less totally stolen from Github. Anyone can create an account on this
server, they don't have to ask a human for permission. They can then
create personal clones of any repository on hg.mozilla.org, make changes
and upload them -- same concept as the "fork this project" github
button. (To keep it focused, though, they can't just create their own
/de novo/ project, at least not at first -- maybe after we get some
experience with this system, that could be an interesting addition.)
Once someone's pushed some changes to this server, they can request a
try server run (such runs have lower priority than pushes to try by
people with "real" commit access, and if necessary there could be a
limit to the number of try invocations per day). Try server reports
come back to this interface, so you don't have to learn tbpl as well.
Finally, there is a button "request review of my changes" which is not
entirely unlike Github's pull-request button. You have to explain the
motivation of your changes, and it then automagically figures out an
appropriate reviewer and sticks your patch in Bugzilla.
This addresses the problems of creating patches in the proper format,
getting access to the test farm, and requesting that someone pay
attention to your changes. It does not solve any problems related to
Bugzilla, and it also doesn't solve the rather more social problem of
getting attention *actually paid* to your changes. However, I think it
might remove enough speedbumps to be worth doing.
zw
Dustin J. Mitchell
> Suppose we set up a secondary Hg server with a web interface more or
> less totally stolen from Github. Anyone can create an account on this
> server, they don't have to ask a human for permission. They can then
> create personal clones of any repository on hg.mozilla.org, make changes
> and upload them -- same concept as the "fork this project" github
> button. (To keep it focused, though, they can't just create their own
> /de novo/ project, at least not at first -- maybe after we get some
> experience with this system, that could be an interesting addition.)
> Once someone's pushed some changes to this server, they can request a
> try server run (such runs have lower priority than pushes to try by
> people with "real" commit access, and if necessary there could be a
> limit to the number of try invocations per day). Try server reports
> come back to this interface, so you don't have to learn tbpl as well.
> Finally, there is a button "request review of my changes" which is not
> entirely unlike Github's pull-request button. You have to explain the
> motivation of your changes, and it then automagically figures out an
> appropriate reviewer and sticks your patch in Bugzilla.
I think this is a super-sweet idea. Two concerns, though:
1. Let's shake of NIH syndrome and just use an existing service like
Bitbucket. I know Github offers "hosted" versions of their site for a
fee - maybe Bitbucket does too?
2. For security reasons, we cannot build arbitrary code contributed by
people without Mozilla commit access using the try system. That said,
the addition of a few scripts could let people who *do* have commit
access do a basic is-this-sane review, then cause it to be run in try
under their authority.
Dustin
David Ascher
> 2. For security reasons, we cannot build arbitrary code contributed by
> people without Mozilla commit access using the try system. That said,
> the addition of a few scripts could let people who *do* have commit
> access do a basic is-this-sane review, then cause it to be run in try
> under their authority.
>
allow us to mitigate some of those security concerns?
--da
Steve Fink
> Suppose we set up a secondary Hg server with a web interface more or
> less totally stolen from Github. Anyone can create an account on this
> server, they don't have to ask a human for permission. They can then
> create personal clones of any repository on hg.mozilla.org, make
> changes and upload them -- same concept as the "fork this project"
> github button. (To keep it focused, though, they can't just create
> their own /de novo/ project, at least not at first -- maybe after we
> get some experience with this system, that could be an interesting
> addition.) Once someone's pushed some changes to this server, they can
> request a try server run (such runs have lower priority than pushes to
> try by people with "real" commit access, and if necessary there could
> be a limit to the number of try invocations per day). Try server
> reports come back to this interface, so you don't have to learn tbpl
> as well. Finally, there is a button "request review of my changes"
> which is not entirely unlike Github's pull-request button. You have
> to explain the motivation of your changes, and it then automagically
> figures out an appropriate reviewer and sticks your patch in Bugzilla.
>
The immediately obvious problem is that you're opening up a large pool
of machines to trivially easy abuse. A popular new test might be "How
many spam messages can I send out before someone notices and shuts me down?"
You could try to sandbox the try servers, but it's a very very large
sandbox.
The closest thing I can see that would be practical is to have an "L0"
level of access to try (though I thought the L1-L3 distinctions were
already supposed to go away...) where you can't push to try until you
get a cursory "do no harm" review of your changes. Then make the bar to
becoming an accepted reviewer for that purpose very low. Crowdsource the
basic safety check.
As for hosting repositories, is that even necessary? If you're making a
change, you'll probably want to have the code locally, and there's not
much difference between downloading the full tree and downloading an hg
repo. Why not just use a local repo until you're ready to share it? Then
you'd push the changes to try, but when going through this path no
builds would be triggered; the bits would just be available for review.
Ben Hearsum
As far as I know, we specifically still use a separate pool of machines
for Try builds, specifically *because* Try has wider access than other
things. These machines do not have write access to anywhere we publish
other builds or updates to. We run tests on a common pool, but those
machines have no write access to *anything*.
I don't see a reason we'd want to segregate our pool further for this.
Ben Hearsum
As far as I know, we specifically still use a separate pool of machines
Richard Newman
> Bitbucket. I know Github offers "hosted" versions of their site for a
> fee - maybe Bitbucket does too?
Better yet, use one of the Git mirrors of mc that is already on GitHub. It would go a really long way towards helping new contributors if the process was "click fork, send a pull request", just like every other software project these days. Accepted patches can be bidirectionally absorbed back into mc so our build tools can use them.
Git and GitHub won.
Wes Garland
> 1. Let's shake of NIH syndrome and just use an existing service like
> Bitbucket.
We use BitBucket as part our toolchain at PageMail and are quite pleased
with it. We use it specifically because we wanted a private "GitHub", that
runs Mercurial, and that we didn't have to invest any real time into. I
wholeheartedly recommend it.
Wes
--
Wesley W. Garland
Director, Product Development
PageMail, Inc.
+1 613 542 2787 x 102
Christian Legnitto
My 2 cents:
Something like this is too much change right now and needs to be managed very carefully. Once we get into the faster, more streamlined release cycle we can start attacking the underlying development model. Additionally, we are now encouraging project repos which will help socialize a change like this and allow developers to see benefits and pitfalls.
Christian
> _______________________________________________
> dev-planning mailing list
> dev-pl...@lists.mozilla.org
> https://lists.mozilla.org/listinfo/dev-planning
Axel Hecht
stuff anyone in mozilla should want.
I'd rephrase it, I guess.
I'm not sure that going for an open hosting service is necessarily the
key. We could do bundles, or pulling repos from the web by authorized
folks. That allows us to check for licensing issues outside of our repos.
I'm not sure if "request review" is necessarily great, I think I
conceptually prefer legneato's thinking about associating a flock of
repos with a bug. github's notes on revs is nice, though.
Oh, and now I head off, "don't drink and drive, take drugs and fly" style.
Iff we'd drop our hg web interface for something beyond, it should, IMHO,
- still support being a remote hg repo
- have a network view (not requesting a manual fork, IMHO)
- diff arbitrary revisions from arbitrary repos (to some extent)
-- extra feature of logic-aware diffs
- offer two-way integration with builds
-- display builds for a revision, or push of revision, independent of tree
--- with respins
-- offer to trigger builds, with test selection (now try), or nightlies
- easy "fork" UI, of course.
And, to boldly go where even drugs don't take me
- a webui for editing, so one can, for example, hack on simple patches
without a local repo, or even just fix the review comments inline.
Axel
Mike Beltzner
> I think that focusing on "external contributors" is wrong here. This is stuff anyone in mozilla should want.
+1
+2, even
Shawn Wilsher
> Better yet, use one of the Git mirrors of mc that is already on GitHub. It would go a really long way towards helping new contributors if the process was "click fork, send a pull request", just like every other software project these days. Accepted patches can be bidirectionally absorbed back into mc so our build tools can use them.
different version control systems for the same tree is ideal.
I get that people like git, but it's not the silver bullet.
Cheers,
Shawn
David Humphrey
This isn't altogether true, if you're talking about external/new people
getting involved. People use github much more than bitbucket.
Bitbucket may have the same features, but we both know that's not how
decisions get made in reality. More people are using git/github, and
when you want to make it easier for people to become involved in your
project, using the tools/workflows they already know helps.
Not a reason to switch, nor am I making that argument. But I do think
it's important to not pretend these are the same, especially in the eyes
of the web dev community--a community I think we should be working
harder to get involved in the project.
Dave
Joe Walker
> On 3/25/2011 1:45 PM, Richard Newman wrote:
>> Better yet, use one of the Git mirrors of mc that is already on
>> GitHub. It would go a really long way towards helping new
>> contributors if the process was "click fork, send a pull request",
>> just like every other software project these days. Accepted patches
>> can be bidirectionally absorbed back into mc so our build tools can
>> use them.
> Bitbucket has the same functionality. I don't think having two
> different version control systems for the same tree is ideal.
Along similar lines, I've been mulling for a while the idea of
'patchzilla'. Bugzilla isn't great at handling patches, and especially
their evolution, but perhaps we could have a database of patches with
extra info like:
- dependencies
- range of affected lines (so we can grok conflicts)
- patch state (broken, ready for commit, in progress, etc)
If we had that then we could get a ton of useful data to help everyone
commit code.
More detailed thoughts: http://etherpad.mozilla.com:9000/patchzilla
Joe.
Christian Legnitto
Why even use bugzilla as a poor man's patch queue? Why not have a repo attached to a bug one could push to? Or multiple repos attached to a bug?
Christian
Richard Newman
>
> Bitbucket has the same functionality. I don't think having two different version control systems for the same tree is ideal.
>
Yes, I think Git knocks Mercurial into a cocked hat. That wasn't actually my point. I was suggesting that using GitHub would make contributing easier for large swathes of the developer community. (It would have picked me up as a contributor long ago if m-c had a GitHub fork and a nice README.markdown!)
Point one: GitHub is more popular.
http://trends.google.com/websites?q=github.com,+bitbucket.org,+launchpad.net&sa=N
GitHub absolutely swamps BitBucket in terms of users. If you take a random person who might want to contribute to Mozilla, the chances of them using BitBucket are practically nil when compared to GitHub.
There are 51,577 public projects on BitBucket right now. GitHub hit a million last year.
Point two: BitBucket is not as good. For example, pull requests.
Compare:
https://github.com/blog/712-pull-requests-2-0
to
GitHub has the ability to apply changes through the web UI with a single click, with a GUI to hint whether various commits will apply cleanly. No comparison.
If we were to try to expand Mozilla's contributor base by using non-Mozilla tools, I'd consider GitHub to be vastly more worthwhile than BitBucket, particularly when you consider that the hg nuts (i.e., BitBucket's users) would be fairly happy using hg.m.o. That this involves another version control system seems beside the point; there's bidirectional integration, lots of internal expertise (lots of new Mozilla projects are starting on GitHub by default, and some people even use Git locally for m-c work, despite no formal support), and most of the contributors that would be picked up in this way wouldn't voluntarily use hg regardless. It's the contributor that matters, not the VCS, and GitHub would remove a bunch of stumbling blocks for a whole category.
Richard Newman
This is how some people are using GitHub. There are tools to attach a redirect 'patch' to Bugzilla which automatically redirects to a GitHub pull request when viewed. Code and code review lives in GitHub, and Bugzilla is used for issue tracking. Seems like a good partition.
Joe Walker
> On Mar 28, 2011, at 10:55 AM, Joe Walker wrote:
>> On 28/03/2011 18:18, Shawn Wilsher wrote:
>>> On 3/25/2011 1:45 PM, Richard Newman wrote:
>>> Bitbucket has the same functionality. I don't think having two different version control systems for the same tree is ideal.
>> - dependencies
>> - range of affected lines (so we can grok conflicts)
>> - patch state (broken, ready for commit, in progress, etc)
>> If we had that then we could get a ton of useful data to help everyone commit code.
>>
>> More detailed thoughts: http://etherpad.mozilla.com:9000/patchzilla
>>
That's probably a better method, I was coming at it from the perspective
of smaller tweaks to the way we do things, but if that's possible, it's
certainly better.
Joe.
Jim Blandy
reports and feature requests, as compared to our in-house team. GitHub
is clearly on a roll at the moment, but companies change, people move
on, and so on. And unlike many projects, we do have the resources to
put up the essentials for ourselves.
I know that one question we got answered before we switched to
Mercurial was how available the hg developers or outside contractors
would be to fix our problems. I don't know how that panned out in
practice, but with GitHub, we wouldn't even have the escape valve of
just fixing stuff in-house. Working in Open Source, it's easy to fall
into the assumption that that's always an option; it's not with GitHub
(or BitBucket).
Axel Hecht
>> Why even use bugzilla as a poor man's patch queue? Why not have a repo attached to a bug one could push to? Or multiple repos attached to a bug?
> This is how some people are using GitHub. There are tools to attach a redirect 'patch' to Bugzilla which automatically redirects to a GitHub pull request when viewed. Code and code review lives in GitHub, and Bugzilla is used for issue tracking. Seems like a good partition.
crossing tool boundaries.
There's the side of me that doesn't believe in single stream
non-threaded communication, so I could see a value in taking the back
and forth on a patch outside of the stream of comments that deal with
the bug and the design of its solution. But going to an independent
website is just a hack.
Axel
Richard Newman
>
> There's the side of me that doesn't believe in single stream non-threaded communication, so I could see a value in taking the back and forth on a patch outside of the stream of comments that deal with the bug and the design of its solution. But going to an independent website is just a hack.
I agree that it's a hack... but the alternative is _attaching diffs to bugs_, and posting review responses in _bug comments_. All very 1990s, and a much bigger hack IMO.
I'd love to see GitHub or something equally usable brought in-house (http://fi.github.com/ !), but in the mean time I'd rather take a dependency on an external website than use Bugzilla for code review.
The 'correct' solution is to have the bug tracker and the VCS aware of each other (a la Trac, GitHub's own terrible issue tracker, etc.), so that WIP can be committed to a branch, and issues automatically amended as merging operations take place... but I'm not holding my breath for that.
Axel Hecht
github folks ever did.
As other have mentioned, going for a closed-source solution for
something as essential as our dev chain doesn't sound right.
Axel
Robert O'Callahan
> I know that one question we got answered before we switched to
> Mercurial was how available the hg developers or outside contractors
> would be to fix our problems. I don't know how that panned out in
> practice,
Not very well. We haven't been able to get
http://mercurial.selenic.com/bts/issue1576 fixed for example.
but with GitHub, we wouldn't even have the escape valve of
> just fixing stuff in-house. Working in Open Source, it's easy to fall
> into the assumption that that's always an option; it's not with GitHub
> (or BitBucket).
>
If Github use is optional and there's always a way to extract the important
data from it, what's the harm?
Rob
--
"Now the Bereans were of more noble character than the Thessalonians, for
they received the message with great eagerness and examined the Scriptures
every day to see if what Paul said was true." [Acts 17:11]
Jim Blandy
> If Github use is optional and there's always a way to extract the important
> data from it, what's the harm?
Well, the more valuable it is to use GitHub, the more trouble it will
cause if we have to leave it. You can't have it both ways.
Kevin Dangoor
I'll also note that GitHub has an open source competitor (http://gitorious.org/) should we use GitHub actively and then later change our minds.
It looks like people are generally agreed that improving this workflow would help everyone who contributes to Mozilla projects. From there, it's a matter of figuring out what workflow people actually want and then doing the usual "build vs. buy" sort of analysis. There are always tradeoffs!
Kevin
Robert O'Callahan
> It looks like people are generally agreed that improving this workflow
> would help everyone who contributes to Mozilla projects. From there, it's a
> matter of figuring out what workflow people actually want and then doing the
> usual "build vs. buy" sort of analysis. There are always tradeoffs!
>
I think the main issue here is that Github has network effects that can't be
replicated in-house (or by using one of their more open competitors).
Kevin Dangoor
I think the main issue here is that Github has network effects that can't be replicated in-house (or by using one of their more open competitors).
Kevin
Dietrich Ayala
pointed at QT as an example: https://qt.gitorious.org/.
It still doesn't get us the network-effect benefits that GitHub has.
On Tue, Mar 29, 2011 at 9:44 AM, Kevin Dangoor <kdan...@mozilla.com> wrote:
> On Monday, March 28, 2011 at 7:29 PM, Jim Blandy wrote:
> On Mon, Mar 28, 2011 at 4:05 PM, Robert O'Callahan <rob...@ocallahan.org> wrote:
>> > If Github use is optional and there's always a way to extract the important
>> > data from it, what's the harm?
>>
>> Well, the more valuable it is to use GitHub, the more trouble it will
>> cause if we have to leave it. You can't have it both ways.
> That's true, but it seems like that's also the case for all sorts of products and services that Mozilla uses. In the case of GitHub, the most important data we'd have with them (the source code and its history) would actually also be in the hands of everyone else with a clone of the repository. If we had to leave GitHub, we'd essentially be back in the same position we are in today.
>
> I'll also note that GitHub has an open source competitor (http://gitorious.org/) should we use GitHub actively and then later change our minds.
>
> It looks like people are generally agreed that improving this workflow would help everyone who contributes to Mozilla projects. From there, it's a matter of figuring out what workflow people actually want and then doing the usual "build vs. buy" sort of analysis. There are always tradeoffs!
>
> Kevin
Max Kanat-Alexander
> Why even use bugzilla as a poor man's patch queue? Why not have a repo attached to a bug one could push to? Or multiple repos attached to a bug?
This is more or less what Launchpad does with its "merge requests"
and it works really well. There's already a fair bit of Perl library
support that I've written that would make this possible; it's just a
matter of adapting it to this usage and writing a lot of UI code.
-Max
Max Kanat-Alexander
> The 'correct' solution is to have the bug tracker and the VCS aware of each other (a la Trac, GitHub's own terrible issue tracker, etc.), so that WIP can be committed to a branch, and issues automatically amended as merging operations take place... but I'm not holding my breath for that.
There's already the start of this work, from the "show me what
commits in the tree were associated with this bug" side:
Axel Hecht
Note that there are alternative web uis for mercurial out there, too,
like http://packages.python.org/RhodeCode/index.html.
(Yes, I love mercurial for having an obvious command at every stage to
verify what you did, and I curse git for either not having that, or for
making finding that too hard to find.)
Axel
Ehsan Akhgari
David Lawrence
have direct access to the
VCS servers for two way communication. I don't know if this is currently
possible.
Dave
On 3/29/11 1:28 PM, Ehsan Akhgari wrote:
> On 11-03-29 6:14 AM, Max Kanat-Alexander wrote:
> Is that something which we can get on bugzilla.mozilla.org?
>
> Ehsan
--
David Lawrence
d...@mozilla.com
Dave Miller
David Lawrence wrote on 3/29/11 1:58 PM:
> We would need to get IT involved as the Bugzilla web apps would need to
> have direct access to the
> VCS servers for two way communication. I don't know if this is currently
> possible.
>
> Dave
>
>
> On 3/29/11 1:28 PM, Ehsan Akhgari wrote:
>> On 11-03-29 6:14 AM, Max Kanat-Alexander wrote:
>> Is that something which we can get on bugzilla.mozilla.org?
>>
>> Ehsan
>
--
Dave Miller http://www.justdave.net/
System Administrator, Mozilla Corporation http://www.mozilla.com/
Project Leader, Bugzilla Bug Tracking System http://www.bugzilla.org/
David Lawrence
this up and
running post-4.0 release. We would need to look into any performance impact
by getting it running on staging and also there is the issue of getting
all of the
Moose dependencies installed.
Dave
On 3/29/11 2:10 PM, Dave Miller wrote:
> It can probably be arranged.
>
> David Lawrence wrote on 3/29/11 1:58 PM:
>> We would need to get IT involved as the Bugzilla web apps would need to
>> have direct access to the
>> VCS servers for two way communication. I don't know if this is currently
>> possible.
>>
>> Dave
>>
>>
>> On 3/29/11 1:28 PM, Ehsan Akhgari wrote:
>>> On 11-03-29 6:14 AM, Max Kanat-Alexander wrote:
>>> Is that something which we can get on bugzilla.mozilla.org?
>>>
>>> Ehsan
>>
>
>
--
David Lawrence
d...@mozilla.com
Jim Blandy
history. But what about patch discussions? Those are valuable, too.
Reading bugzilla discussions teaches me a lot.
Jim Blandy
> On Mon, Mar 28, 2011 at 4:05 PM, Robert O'Callahan <rob...@ocallahan.org> wrote:
>> If Github use is optional and there's always a way to extract the important
>> data from it, what's the harm?
Escape costs aren't ameliorated just by being able to get one's data
and metadata. You also need a way to put it up somewhere and continue
reviewing/discussing/committing. Git makes it practical to continue
committing elsewhere, but the rest are still an issue, as far as I
know.
Christian Legnitto
Christian
On Mar 29, 2011, at 11:29 AM, David Lawrence wrote:
> Then I personally do not have any objections to looking into getting this up and
> running post-4.0 release. We would need to look into any performance impact
> by getting it running on staging and also there is the issue of getting all of the
> Moose dependencies installed.
>
> Dave
>
>
> On 3/29/11 2:10 PM, Dave Miller wrote:
>> It can probably be arranged.
>>
>> David Lawrence wrote on 3/29/11 1:58 PM:
>>> We would need to get IT involved as the Bugzilla web apps would need to
>>> have direct access to the
>>> VCS servers for two way communication. I don't know if this is currently
>>> possible.
>>>
>>> Dave
>>>
>>>
>>> On 3/29/11 1:28 PM, Ehsan Akhgari wrote:
>>>> On 11-03-29 6:14 AM, Max Kanat-Alexander wrote:
>>>> Is that something which we can get on bugzilla.mozilla.org?
>>>>
>>>> Ehsan
>>>
>>
>>
>
> --
> David Lawrence
> d...@mozilla.com
>
Andrew Sutherland
> Escape costs aren't ameliorated just by being able to get one's data
> and metadata. You also need a way to put it up somewhere and continue
> reviewing/discussing/committing. Git makes it practical to continue
> committing elsewhere, but the rest are still an issue, as far as I
> know.
The Pull Request API appears to provide all the information required to
extract the discussion for posterity without needing to engage in
scraping or other potentially brittle practices:
http://develop.github.com/p/pulls.html
Presumably the fancy extension or a simple daemon could utilize this to
simulcast all pull request responses? Thanks to the fact that the
existing bar is "some text in a bugzilla comment field", it's not a lot
of work to create an export format that is reasonably human friendly.
For example, my reviewboard "integration"[1] for mozilla bugzilla for
mozilla-central/comm-central provides a simple export format you can
copy and paste into the bugzilla text field for both initial reviews and
comments on those reviews. (Obviously, a polished solution would not
involve those extra human steps, but I find it more palatable than using
the current alternative of directly marking up diffs without seeing
things in context/syntax-highlighted/etc.)
Andrew
Clint Talbert
> We would need to get IT involved as the Bugzilla web apps would need to
> have direct access to the
> VCS servers for two way communication. I don't know if this is currently
> possible.
>
that bug to do something like this.
Clint
Max Kanat-Alexander
> We would need to get IT involved as the Bugzilla web apps would need to
> have direct access to the
> VCS servers for two way communication. I don't know if this is currently
> possible.
Actually, it would be best if the VCS servers were mirrored onto the
Bugzilla servers locally, or onto a mirror close to the Bugzilla
servers, although giving the Bugzilla web-heads direct access to the
VCSes is a simpler solution (and will still work).
-Max
matthew zeier
On Mar 30, 2011, at 5:25 AM, Max Kanat-Alexander wrote:
> On Mar 29, 10:58 am, David Lawrence <d...@mozilla.com> wrote:
>> We would need to get IT involved as the Bugzilla web apps would need to
>> have direct access to the
>> VCS servers for two way communication. I don't know if this is currently
>> possible.
Anything is possible with proper review.
>
> Actually, it would be best if the VCS servers were mirrored onto the
> Bugzilla servers locally
Why is that best? That turns into a storage problem where every web server has to have the same amount of storage that VCS has.
> , or onto a mirror close to the Bugzilla
> servers
At present, VCS is physically a rack or two away from Bugzilla - how does locality affect this?
David Lawrence
On 3/30/11 11:15 AM, matthew zeier wrote:
> On Mar 30, 2011, at 5:25 AM, Max Kanat-Alexander wrote:
>
>> On Mar 29, 10:58 am, David Lawrence<d...@mozilla.com> wrote:
>>> We would need to get IT involved as the Bugzilla web apps would need to
>>> have direct access to the
>>> VCS servers for two way communication. I don't know if this is currently
>>> possible.
> Anything is possible with proper review
Definitely would need a security review for something like this before
we went live. We could
get it setup on one of our internal staging servers and then file the
proper tickets to get the
security team to go through it.
>> Actually, it would be best if the VCS servers were mirrored onto the
>> Bugzilla servers locally
> Why is that best? That turns into a storage problem where every web server has to have the same amount of storage that VCS has.
web heads typically do not have that much storage unlike the db server.
We definitely don't want to
be mirroring the entire VCS server. And if we only mirror parts, people
will always be wanting more
and more.
>> , or onto a mirror close to the Bugzilla
>> servers
> At present, VCS is physically a rack or two away from Bugzilla - how does locality affect this?
With Red Hat we were very tight with which server could talk to any
other server even on the
same networks. Certain holes had to be created for specific purposes and
only when absolutely
necessary.
The way I understand the current implementation of the Bugzilla
extension for VCS access, hooks are added
to the VCS itself that parse the commit message for bug information. It
then sends a XMLRPC request to the
Bugzilla server filling in some basic information in Bugzilla tables.
Bugzilla then uses VCS abstraction code
to talk to the VCS server natively to get more detailed information
about commits for display in the
Bugzilla UI.
So at a minimum we would need to setup the network so that the VCS
server can send the XMLRPC request
to Bugzilla (port 443) and that Bugzilla is allowed to communicate with
the VCS servers using their
specific port ranges.
Sounds feasible?
Dave
--
David Lawrence
d...@mozilla.com
matthew zeier
On Mar 30, 2011, at 9:05 AM, David Lawrence <d...@mozilla.com> wrote:
>
> So at a minimum we would need to setup the network so that the VCS server can send the XMLRPC request
> to Bugzilla (port 443) and that Bugzilla is allowed to communicate with the VCS servers using their
> specific port ranges.
>
> Sounds feasible?
>
Absolutely. Just wanted to make sure we weren't locking ourselves into a design that prevented bugzilla and vcs from moving around. Firewall rules are easy.
We'll work offline on staging this.
Christian Legnitto
Thr only piece we'd need is a simple script to catch pulse messages and stick data into the bugzilla-vcs table for display in the UI (or we could directly POST somewhere from pulse)
Food for thought...
On Mar 30, 2011, at 9:05 AM, David Lawrence <d...@mozilla.com> wrote:
> Thanks Matthew.
>
> On 3/30/11 11:15 AM, matthew zeier wrote:
>> On Mar 30, 2011, at 5:25 AM, Max Kanat-Alexander wrote:
>>
>>> On Mar 29, 10:58 am, David Lawrence<d...@mozilla.com> wrote:
>>>> We would need to get IT involved as the Bugzilla web apps would need to
>>>> have direct access to the
>>>> VCS servers for two way communication. I don't know if this is currently
>>>> possible.
>> Anything is possible with proper review
>
> Definitely would need a security review for something like this before we went live. We could
> get it setup on one of our internal staging servers and then file the proper tickets to get the
> security team to go through it.
>
>>> Actually, it would be best if the VCS servers were mirrored onto the
>>> Bugzilla servers locally
>> Why is that best? That turns into a storage problem where every web server has to have the same amount of storage that VCS has.
>
> web heads typically do not have that much storage unlike the db server. We definitely don't want to
> be mirroring the entire VCS server. And if we only mirror parts, people will always be wanting more
> and more.
>
>>> , or onto a mirror close to the Bugzilla
>>> servers
>> At present, VCS is physically a rack or two away from Bugzilla - how does locality affect this?
>
> With Red Hat we were very tight with which server could talk to any other server even on the
> same networks. Certain holes had to be created for specific purposes and only when absolutely
> necessary.
>
> The way I understand the current implementation of the Bugzilla extension for VCS access, hooks are added
> to the VCS itself that parse the commit message for bug information. It then sends a XMLRPC request to the
> Bugzilla server filling in some basic information in Bugzilla tables. Bugzilla then uses VCS abstraction code
> to talk to the VCS server natively to get more detailed information about commits for display in the
> Bugzilla UI.
>
> So at a minimum we would need to setup the network so that the VCS server can send the XMLRPC request
> to Bugzilla (port 443) and that Bugzilla is allowed to communicate with the VCS servers using their
> specific port ranges.
>
> Sounds feasible?
>
> Dave
>
> --
> David Lawrence
> d...@mozilla.com
>
David Lawrence
(logs, files, etc.) which could start to
fill up the db fairly quickly. And also run an additional background
service to monitor the queue for
relevant information. With the bugzilla-vcs extension the data stays on
the VCS server for the most part.
Dave
On 3/30/11 12:09 PM, Christian Legnitto wrote:
> As a 2nd option we could use pulse as it already has hg commit messages (currently polling but the hook is in the process of being rolled out to all repos). hg.mozilla.org can already connect to pulse as well. The security concerns would be largely irrelevant and data would show up instantly. Loosely tying two existing systems together is a main use case for a message broker.
>
> Thr only piece we'd need is a simple script to catch pulse messages and stick data into the bugzilla-vcs table for display in the UI (or we could directly POST somewhere from pulse)
>
> Food for thought...
>
>
--
David Lawrence
d...@mozilla.com
Christian Legnitto
Max Kanat-Alexander
> > Actually, it would be best if the VCS servers were mirrored onto the
> > Bugzilla servers locally
>
> Why is that best? That turns into a storage problem where every web server has to have the same amount of storage that VCS has.
It's only best from a performance perspective. When a checkin is
done, Bugzilla grabs information about the checkin from the VCS.
Usually there's a checkin hook in the VCS to do this. Bugzilla talking
to the VCS is obviously faster if it's local than remote, but the
performance hit is only during the checkin hook (not during normal
Bugzilla operation, because Bugzilla caches all the info in the DB) so
it's probably not a huge deal.
> At present, VCS is physically a rack or two away from Bugzilla - how does locality affect this?
That should be fine.
-Max
Gervase Markham
> I agree that it's a hack... but the alternative is _attaching diffs
> to bugs_, and posting review responses in _bug comments_. All very
> 1990s, and a much bigger hack IMO.
Splinter is, I believe, part of the BMO 4.0 upgrade coming in April:
http://blog.fishsoup.net/2009/09/23/splinter-patch-review/
(
although a quick check suggests it might not be working at the moment on
bugzilla-stage-tip:
https://bugzilla-stage-tip.mozilla.org/page.cgi?id=splinter.html&bug=626997&attachment=507833
)
Does that improve things?
Gerv
David Lawrence
The reason it seems to not be working is that the attachments on
bugzilla-stage-tip have been removed as
part of the database sanitizing we had to do to get approval to open
testing up to a broader audience.
To test out the Splinter functionality, simply go to any test bug you
wish and attach a patch to it. Then
you can try it out on the new attachment.
> Does that improve things?
>
> Gerv
>
> _______________________________________________
> dev-planning mailing list
> dev-pl...@lists.mozilla.org
> https://lists.mozilla.org/listinfo/dev-planning
--
David Lawrence
d...@mozilla.com
Andrew Sutherland
> Splinter is, I believe, part of the BMO 4.0 upgrade coming in April:
> Does that improve things?
Splinter is a fantastic improvement on the alternative of directly
editing a diff.
The notable 'BUT' is that you are still just marking up a diff; if you
want to check out the surrounding context of the diff, you need to turn
elsewhere. Review board makes it a matter of clicking on something and
magic AJAX filling in the (syntax highlighted) rest of the elided file
contents. Although mozilla patch standards require 8 lines of context,
most mozilla code is sufficiently complex that 8 lines of context is not
really sufficient for a thorough review. While thorough review will
also entail applying the patch locally, I'm not sure there is a good
hookup between a locally applied patch and review mechanisms, etc.
It's worth noting that it does not appear that github's pull requests
can do the expanding/syntax highlighted tricks that review board can do
(at this time) and so have a lot in common with splinter's UI.
Andrew
David Lawrence
> On 03/31/2011 04:03 AM, Gervase Markham wrote:
> editing a diff.
>
> The notable 'BUT' is that you are still just marking up a diff; if you
> want to check out the surrounding context of the diff, you need to
> turn elsewhere. Review board makes it a matter of clicking on
> something and magic AJAX filling in the (syntax highlighted) rest of
> the elided file contents. Although mozilla patch standards require 8
> lines of context, most mozilla code is sufficiently complex that 8
> lines of context is not really sufficient for a thorough review.
> While thorough review will also entail applying the patch locally, I'm
> not sure there is a good hookup between a locally applied patch and
> review mechanisms, etc.
>
It is on the to-do list for future version of Splinter to add the
ability to pull in the rest of the file that the patch
is applied to for better context. This should be come easier in the
future as we also plan to use an extension
that allows Bugzilla to communicate with the various VCS systems in use.
https://bugzilla.mozilla.org/show_bug.cgi?id=603037
dkl
--
David Lawrence
d...@mozilla.com
Andrew Sutherland
> It is on the to-do list for future version of Splinter to add the
> ability to pull in the rest of the file that the patch
> is applied to for better context. This should be come easier in the
> future as we also plan to use an extension
> that allows Bugzilla to communicate with the various VCS systems in use.
This sounds great! Does "to-do list" == "wish list", or someone is
actively working full-time on it now? If it's "wish list", can someone
in the automation or developer tools team either work on it or fund
someone to work on it? (Or whatever aspect of MoCo is tasked with the
responsibility and resources to address this problem domain.)
Andrew
David Lawrence
Well the to-dos and the nice-to-haves will always have some overlap but
the good news is that we now
have two full-time BMO hackers (myself included) to work on this stuff
now. So the prospects of getting
this implemented in the near term are much better.
Nicholas Nethercote
>
> Splinter is, I believe, part of the BMO 4.0 upgrade coming in April:
Cool! I was just daydreaming about a better patch review system like
this the other day :)
Nick