[Wikitech-l] MediaWiki core deployments starting in April, and how that might work
Rob Lanphier
This email is going to briefly describe the old SVN workflow, and then
use that as a baseline to describe what we should do for Git. I
haven't had a chance to coordinate this mail with Chad (or anyone
else), so I'll reserve the right for him to completely contradict me
here. This is meant to provoke a discussion about how we're really
going to use Git, and to establish a plan for taking advantage of the
new workflow to move to much more frequent deployments.
In the old world, we had this:
trunk
├── REL1_17
│ └── 1.17wmf1 (branched from REL1_17)
├── REL1_18
│ └── 1.18wmf1 (branched from REL1_18)
└── REL1_19
└── 1.19wmf1 (branched from REL1_19)
Tarball releases would come out of the respective REL1_xx branches,
and deployments would come out of the 1.xxwmf1 branches. REL1_xx
branches have all extensions, and 1.xxwmf1 branches have only
Wikimedia production code. Each would be a relatively long lived
branch (6-18 months) into which critical fixes and priority features
would be merged from trunk.
Looking ahead to deployments, there's a couple of different ways to go
about this:
One plan would be to have a "wmf" branch that does not trail far
behind the master. The extensions we deploy to the cluster can be
included as submodules for that given branch. The process for
deployment at that point will be "merge from master" or "update
submodule reference" on the wmf branch. Then on fenari, you will git
pull and git submodule update before scapping like you're currently
used to. The downside of this approach is that there's not an obvious
way to have multiple production branches in play (heterogeneous
deploy). Seems solvable (e.g wmf1, wmf2, etc), but that also seems
messy.
Another possible plan would be to have something *somewhat* closer to
what we have today, with new branches off of trunk for each
deployment, and deployments happening as frequently as weekly.
master
├── 1.20wmf01
├── 1.20wmf02
├── 1.20wmf03
...
├── 1.20wmf11
├── 1.20wmf12
├── REL1_20
├── 1.21wmf01
├── 1.21wmf02
├── 1.21wmf03
...
This is how I was envisioning the process working, and just didn't get
a chance to sync up with Chad to find out what the issues of this
approach would be.
Since we don't have an imminent deployment coming from Git, we have a
little time to figure this situation out.
Regardless of the branching strategy, the goal would be to start as
early as April with much more frequent deployments to production. The
deployment plan would look something like this:
* Deploy 1.20wmf01 to test2 real soon now (say, no later than April 16).
* Deploy 1.20wmf01 to mediawiki.org a couple deploy days after that
("deploy day" meaning Monday through Thursday)
* Let simmer for some short-ish amount of time (TBD)
* Roll out 1.20wmf01 to more wikis, eventually making it to all of them
Given the way APC caches and other caching works, I suspect we can't
get away with having more than two simultaneous versions out on the
production cluster, but we could conceivably have a situation where,
for example, a deploy day or two after rolling out 1.20wmf01 out to
the last of the wikis, we then roll out 1.20wmf02 out to test2.
This topic is partially covered here:
https://www.mediawiki.org/wiki/Git/Workflow#Who_can_review.3F_Gerrit_project_owners
...but I imagine we'll probably need to revise that based on this
conversation and perhaps break this out into a separate page.
There's a few of us that plan to meet in a couple of weeks to
formalize something here, but perhaps we can get this all hammered out
on-list prior to that.
Thoughts on this process?
Rob
_______________________________________________
Wikitech-l mailing list
Wikit...@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Brion Vibber
the advantage that you've got a previous branch to revert to quickly if
needed. That's a plus for deployment work.
-- brion
Chad
> I kinda like the first model (it feels more organic), but the second has
> the advantage that you've got a previous branch to revert to quickly if
> needed. That's a plus for deployment work.
>
> -- brion
I kinda like the idea of both. We need a wmf-branch (specific hacks for
us, ability to trail master when need be), but having tags is super useful
too.
Why not just have the branch but make a new tag before any scap?
Best of both worlds ;-)
-Chad
Rob Lanphier
> On Thu, Mar 22, 2012 at 3:57 PM, Brion Vibber <bvi...@wikimedia.org> wrote:
>> I kinda like the first model (it feels more organic), but the second has
>> the advantage that you've got a previous branch to revert to quickly if
>> needed. That's a plus for deployment work.
>>
>> -- brion
>
> I kinda like the idea of both. We need a wmf-branch (specific hacks for
> us, ability to trail master when need be), but having tags is super useful
> too.
Well, we need two wmf-branches when we're mid-deploy, which is going
to be far more frequently in the new regime. When mediawiki.org,
meta, commons, and nlwiki are on 1.20wmf03 for a week, while enwiki
and others are on 1.20wmf02, we'll need to be mindful of what we might
backport to both branches.
Now, in this new model, backporting in general should be far, far less
frequent, and only done for the most urgent bugfixes.
> Why not just have the branch but make a new tag before any scap?
> Best of both worlds ;-)
There are almost certainly going to be times when the wmf branch isn't
linear (e.g. a security fix needs to be deployed to all wikis, even
though we're not ready to push all wikis to the tip of the wmf branch)
Rob
Chad
> On Fri, Mar 23, 2012 at 7:33 AM, Chad <innocen...@gmail.com> wrote:
>> On Thu, Mar 22, 2012 at 3:57 PM, Brion Vibber <bvi...@wikimedia.org> wrote:
>>> I kinda like the first model (it feels more organic), but the second has
>>> the advantage that you've got a previous branch to revert to quickly if
>>> needed. That's a plus for deployment work.
>>>
>>> -- brion
>>
>> I kinda like the idea of both. We need a wmf-branch (specific hacks for
>> us, ability to trail master when need be), but having tags is super useful
>> too.
>
> Well, we need two wmf-branches when we're mid-deploy, which is going
> to be far more frequently in the new regime. When mediawiki.org,
> meta, commons, and nlwiki are on 1.20wmf03 for a week, while enwiki
> and others are on 1.20wmf02, we'll need to be mindful of what we might
> backport to both branches.
>
Then a branch for each one we're deploying. No harm in having wmf-1.20
and wmf-1.19 at the same time.
> Now, in this new model, backporting in general should be far, far less
> frequent, and only done for the most urgent bugfixes.
>
Agreed. We want to keep them as close to master as possible as it is.
>> Why not just have the branch but make a new tag before any scap?
>> Best of both worlds ;-)
>
> There are almost certainly going to be times when the wmf branch isn't
> linear (e.g. a security fix needs to be deployed to all wikis, even
> though we're not ready to push all wikis to the tip of the wmf branch)
>
I disagree here. The wmf branches should always be linear, and you
should never merge to it without accepting that it may get scapped.
-Chad
Rob Lanphier
> I disagree here. The wmf branches should always be linear, and you
> should never merge to it without accepting that it may get scapped.
Actually, you should never merge to master without accepting that it
may get scapped. Why have two layers here?
Rob
Chad
> On Fri, Mar 23, 2012 at 10:03 AM, Chad <innocen...@gmail.com> wrote:
>> I disagree here. The wmf branches should always be linear, and you
>> should never merge to it without accepting that it may get scapped.
>
> Actually, you should never merge to master without accepting that it
> may get scapped. Why have two layers here?
>
Well true, but I still don't see an instance where you'd merge
to master but *not* be willing to have it deployed.
-Chad
Rob Lanphier
> On Fri, Mar 23, 2012 at 1:11 PM, Rob Lanphier <ro...@wikimedia.org> wrote:
>> On Fri, Mar 23, 2012 at 10:03 AM, Chad <innocen...@gmail.com> wrote:
>>> I disagree here. The wmf branches should always be linear, and you
>>> should never merge to it without accepting that it may get scapped.
>>
>> Actually, you should never merge to master without accepting that it
>> may get scapped. Why have two layers here?
>>
>
> Well true, but I still don't see an instance where you'd merge
> to master but *not* be willing to have it deployed.
Chad and I discussed this, and I think we figure out where we were
talking past each other. I'll let him explain in more detail when he
gets a chance, but the short answer is we'll probably be doing
something roughly like I described as the second plan in my original
email, with the addition of liberal use of tagging. We could even
*try* to get away with tagging instead of branching, and then only
branch when necessary.
Rob
Marcin Cieslak
> On Fri, Mar 23, 2012 at 10:13 AM, Chad <innocen...@gmail.com> wrote:
>> On Fri, Mar 23, 2012 at 1:11 PM, Rob Lanphier <ro...@wikimedia.org> wrote:
>>> On Fri, Mar 23, 2012 at 10:03 AM, Chad <innocen...@gmail.com> wrote:
>>>> I disagree here. The wmf branches should always be linear, and you
>>>> should never merge to it without accepting that it may get scapped.
>>>
>>> Actually, you should never merge to master without accepting that it
>>> may get scapped. Why have two layers here?
>>>
>>
>> Well true, but I still don't see an instance where you'd merge
>> to master but *not* be willing to have it deployed.
>
> Chad and I discussed this, and I think we figure out where we were
> talking past each other. I'll let him explain in more detail when he
> gets a chance, but the short answer is we'll probably be doing
> something roughly like I described as the second plan in my original
> email, with the addition of liberal use of tagging. We could even
> *try* to get away with tagging instead of branching, and then only
> branch when necessary.
How "git committers" (i.e. mere mortals) will be able to propose
changes to be deployed in wmf production (a.k.a. "1.19wmf1" tag
in [[Special:Code]]?
If we plan to use gerrit for that (submitting commits for review
in the "wmf" branches by the means of refs/for/wmf-something),
I would ask for one thing - can we full go end-to-end and test
the workflow with actual commands?
I have noticed potential issues with gerrit+git combination vs
pure git workflow:
* It seems I can't have a longer running branch with my
changes on some topic - those commits cannot be easily
submitted to gerrit (since my branch will likely
contain merges from master and we don't want't them
to be resubmitted via the dependency tree)
* It seems necessary to change "Change-Id" everywhere
for "git cherry-pick" to work without -n - it seems
that cherry-pick does not invoke our commit-msg hook
when committing. Neither does "git merge".
Those things limit possibilities in moving commits
between branches (official or local ones), so I think
it would be good to review the process in the test tree
using some actual commands (and with some test users
that don't have privilege to cross the gated trunk
involved).
More on the second case in separate thread.
//Saper
Roan Kattouw
> Another possible plan would be to have something *somewhat* closer to
> what we have today, with new branches off of trunk for each
> deployment, and deployments happening as frequently as weekly.
>
> master
> ├── 1.20wmf01
> ├── 1.20wmf02
> ├── 1.20wmf03
> ...
> ├── 1.20wmf11
> ├── 1.20wmf12
> ├── REL1_20
> ├── 1.21wmf01
> ├── 1.21wmf02
> ├── 1.21wmf03
> ...
>
> This is how I was envisioning the process working, and just didn't get
> a chance to sync up with Chad to find out what the issues of this
> approach would be.
>
better, as described by others on this thread. However, I'd recommend using
a naming convention with a prefix, e.g. wmf/1.20/01 , so we can handle
permissions on these deployment branches in a managable way.
Roan
Chad
> On Mar 22, 2012 3:44 PM, "Rob Lanphier" <ro...@wikimedia.org> wrote:
>> Another possible plan would be to have something *somewhat* closer to
>> what we have today, with new branches off of trunk for each
>> deployment, and deployments happening as frequently as weekly.
>>
>> master
>> ├── 1.20wmf01
>> ├── 1.20wmf02
>> ├── 1.20wmf03
>> ...
>> ├── 1.20wmf11
>> ├── 1.20wmf12
>> ├── REL1_20
>> ├── 1.21wmf01
>> ├── 1.21wmf02
>> ├── 1.21wmf03
>> ...
>>
>> This is how I was envisioning the process working, and just didn't get
>> a chance to sync up with Chad to find out what the issues of this
>> approach would be.
>>
> This seems like the best approach to me, because it meshes with het deploy
> better, as described by others on this thread. However, I'd recommend using
> a naming convention with a prefix, e.g. wmf/1.20/01 , so we can handle
> permissions on these deployment branches in a managable way.
>
*nod* Makes sense. And Rob and I totally agree actually with how we
should be handling deployments. We should branch from master liberally
and often. When it comes time to do an actual scap, you should also
tag your release so we have a clearly repeatable state if we need to
roll-back or figure out what broke.
Ideally, scap would become something like:
`mw-deploy <tag> <wiki1,wiki2>`
-Chad
Arthur Richards
>
> Regardless of the branching strategy, the goal would be to start as
> early as April with much more frequent deployments to production. The
> deployment plan would look something like this:
> * Deploy 1.20wmf01 to test2 real soon now (say, no later than April 16).
> * Deploy 1.20wmf01 to mediawiki.org a couple deploy days after that
> ("deploy day" meaning Monday through Thursday)
> * Let simmer for some short-ish amount of time (TBD)
> * Roll out 1.20wmf01 to more wikis, eventually making it to all of them
Now that April is upon us, is there a more concrete timeline for getting
deployment branches into Git and using them for deployment? As much as I
love making patches between a Git repo and the deployment branch, I'm very
eager for the migration as I expect it will make deployments much easier
and less time consuming.
Also, what is the impact that this migration strategy will have on
extension deployments? The mobile team makes code deployments very
frequently and it would be helpful to be able to plan ahead around the
migration timeline.
--
Arthur Richards
Software Engineer, Mobile
[[User:Awjrichards]]
IRC: awjr
+1-415-839-6885 x6687
Chad
> Now that April is upon us, is there a more concrete timeline for getting
> deployment branches into Git and using them for deployment? As much as I
> love making patches between a Git repo and the deployment branch, I'm very
> eager for the migration as I expect it will make deployments much easier
> and less time consuming.
>
We've got a meeting Wednesday to work on this. There's some
rough consensus already, so it should just be a matter finishing
off the details.
> Also, what is the impact that this migration strategy will have on
> extension deployments? The mobile team makes code deployments very
> frequently and it would be helpful to be able to plan ahead around the
> migration timeline.
>
The plan right now is to use extensions as submodules on the
wmf branch. Updating your extension in deployment is just a
matter of updating the referenced commit and then running
`git submodule update` on fenari.
-Chad
Antoine Musso
> The plan right now is to use extensions as submodules on the
> wmf branch. Updating your extension in deployment is just a
> matter of updating the referenced commit and then running
> `git submodule update` on fenari.
Which, as a side effect, means that extensions would have to either:
- make sure master is production grade
Or:
- maintain a deployment branch
--
Antoine "hashar" Musso
Chad
> Le 03/04/12 03:34, Chad a écrit :
>> The plan right now is to use extensions as submodules on the
>> wmf branch. Updating your extension in deployment is just a
>> matter of updating the referenced commit and then running
>> `git submodule update` on fenari.
>
> Which, as a side effect, means that extensions would have to either:
> - make sure master is production grade
> Or:
> - maintain a deployment branch
>
Well we should all be submitting production grade code.
Especially with pre-commit review ;-)
More seriously though: just as it is in SVN, you are expected
to keep deployed code runnable at all times. This has never
ever changed. If you need to develop some experimental
feature that you don't want to deploy, that's what branches are
for (and was a major reason we switched to begin with).
Regarding deployment branches...I'd really prefer this to not
happen, since master should be the "this is the current stable
version that I want people to use."
-Chad