Updating the organization of the Django Project

[Aymeric Augustin]

Hello,

I’ve been working on updating our organization: https://github.com/django/django/pull/2947

This proposal attempts to address several issues with our current organization. There’s no short version and any simplistic interpretation will be wrong. Here are the main factors at play.

1) In theory, the core team is the group of committers, each of whom has been judged capable of making code design decisions. (Astute readers will have noticed that it isn’t true in practice.) This restrictive approach to staffing makes it hard to cover all of our HR needs. Specifically:
a) It creates a chasm between non-core and core contributors, which has perverse side effects and creates tons of frustration.
b) It drives away would-be contributors whose help wouldn’t involve committing code to the main Django repository.
c) Even if such contributors are found, it’s hard to convince the core team to bring them on board.

2) Since the BDFLs have stepped down, there’s no obvious way to counteract honest mistakes made by core developers. This is making the core team uncomfortable at times. While BDFLs hardly ever had to intervene, their mere existence played a role. We need to recreate that role in a more democratic fashion.

3) We’re good at burning out our most prolific contributors. Since we lack structure, it’s too easy to become responsible for everything, until you can’t handle it anymore and throw the towel. We must classify roles, write down who takes what role, fill the gaps with new volunteers, and remove pressure around stepping down.

4) As we have grown, having no explicit organization within the core team makes it complicated for newcomers to figure who does what and how they fit in the picture. It doesn’t erase the power structure. It merely hides it.

My proposal builds upon years of discussions at DjangoCons. It has gone through many rounds of feedback inside the core team already. It’s an evolution, not a revolution. It takes into account the growth of the project, acknowledges and formalizes some things that we’re already doing, and introduces just enough formal organization to make everyone comfortable.

It doesn’t encompass everything we could do to improve our organization. In particular I expect some follow up work on how we manage roles in order to avoid burnout.

I would like to ask the core team for a formal vote on this pull request, according to our guidelines. [1] Please vote by the end of July in UTC (2014-08-01T00:00:00Z).

Obviously, I’m voting +1.

Thank you,

--
Aymeric.


[1] https://docs.djangoproject.com/en/stable/internals/contributing/bugs-and-features/#how-we-make-decisions

[Tim Graham]

+1.

It might be better to move the more temporal information (team.txt and roles.txt) to a protected wiki page (or something similar) so there's one canonical place to view and update that information (rather than the documentation which has a branch for each release of Django). 

[Carl Meyer]

+1

Carl

[Loic Bistuer]

+1

Thanks for the hard work you’ve put into this.

--
Loic

[Chris Beaven]

Looks good, Aymeric!

+1

[charettes]

+1

Thanks for putting this up together Aymeric

Simon

[Paul McMillan]

+1

Thanks for your hard work, Aymeric.

-Paul

> --
> You received this message because you are subscribed to the Google Groups
> "Django developers" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to django-develop...@googlegroups.com.
> To post to this group, send email to django-d...@googlegroups.com.
> Visit this group at http://groups.google.com/group/django-developers.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/django-developers/6e625452-74f2-46b1-8553-6167effcd7f5%40googlegroups.com.
>
> For more options, visit https://groups.google.com/d/optout.

[Shai Berger]

+1

What they said.

[Claude Paroz]

Count me in the +1 storm :-)

Claude

[Erik Romijn]

+1

Thanks for all the work on this.

Erik

[Jannis Leidel]

This makes me very happy, thanks for plowing through it, Aymeric.

+1

Jannis

[Florian Apolloner]

+1

[Russell Keith-Magee]

Hi Aymeric.

A big +1 from me. Thanks for all your work drafting these modifications.

Russ %-)

[Chris Foresman]

As a non-core community member, I welcome a streamlined way for new potential coders to contribute.

[Jacob Kaplan-Moss]

+1.

Aymeric, I can't thank you enough for taking this on and running with it. 

Jacob

--
You received this message because you are subscribed to the Google Groups "Django developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to django-develop...@googlegroups.com.
To post to this group, send email to django-d...@googlegroups.com.
Visit this group at http://groups.google.com/group/django-developers.

To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/4920b823-debe-411c-8b3f-6fad13c10604%40googlegroups.com.

[Honza Král]

+1

Thanks for pushing for this!

To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/CAK8PqJGXO%3DZzkPkDFd2dfGfJs2k9x9S0An5%2BCCHhhNBOk2XDOg%40mail.gmail.com.

[Baptiste Mispelon]

Another big +1 from me.


Thanks for yet another awesome refactor,

Baptiste

To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/CADoCwr2nVQgavmZE8xGR09xKXhB5tZyZE4OYya6ZrE-KS%2Bk-Nw%40mail.gmail.com.

[Aymeric Augustin]

Hello,

Since the proposal was accepted with sixteen +1, I committed it: https://github.com/django/django/commit/8c2b405ba8343e68bc32ac2c860be0c6cfd7d631. The next step is to elect the first technical board, as soon as 1.7 final is released.

As Tim noticed, team.txt and roles.txt are unrelated to Django versions. I suggest to keep master up to date and leave each version whatever was there when its branch was forked. I’d rather not make the wiki an authoritative source of information. If someone wants to build a better solution, that’s cool.

I would like to thank all those who took the time to comment on the proposal as well as the entire core team for accepting to relinquish a few of its individual prerogatives.

--
Aymeric.

[Christian Schmitt]

Since you've introduced these changes, wouldn't it be suitable to change Django's commit / review system entirely?

Like introduce gerrit, where very commit / ticket needs to be reviewed by X people and then it would be marked as merge ready and a "core" or whoever member could merge that.

There are a lot of projects which uses this kind of workflow.

Also there are like 120 Pull Requests which are over an year old sometimes.Which scary's off a lot of people. We should do something against it to have a "cleaner" queue.

[Collin Anderson]

Also there are like 120 Pull Requests which are over an year old sometimes.Which scary's off a lot of people. We should do something against it to have a "cleaner" queue.

In theory every pull request has a trac ticket and the queue is all handled by queries such as:

https://code.djangoproject.com/query?status=!closed&stage=Ready+for+checkin

There are links to other useful reports here:

https://code.djangoproject.com/wiki/Reports

[Aymeric Augustin]

On 1 août 2014, at 18:31, Christian Schmitt <c.sc...@briefdomain.de> wrote:

Since you've introduced these changes, wouldn't it be suitable to change Django's commit / review system entirely?

“Hey, you climbed this hill just fine, why don’t you climb the Everest while you’re at it?” ;-) Even though the final vote looks like everything went fine, preparing it took three months and drained a lot of energy.

Like introduce gerrit, where very commit / ticket needs to be reviewed by X people and then it would be marked as merge ready and a "core" or whoever member could merge that.

There are a lot of projects which uses this kind of workflow.

I’ve discussed that idea with other core devs several times. Generally speaking, we’re interested in shifting from a commit-oriented culture to a review-oriented culture. We haven’t done it (yet) for four main reasons:

- Someone has to organize this process, set up the tools, and teach everyone. No one has volunteered.

- The drawbacks of adding bureaucracy may exceed the advantages of reviews for a vast majority of patches, which are small fixes or improvements.

- We have only one person reviewing patches regularly in his free time, so requiring as few as two reviews for a patch to be merged is already unrealistic. Contrast this with the number of full-time paid developers on OpenStack.

- Overall e’re not convinced this process would be appropriate for Django and we don’t know what would happen if it turns out to be impractical.

Also there are like 120 Pull Requests which are over an year old sometimes.Which scary's off a lot of people. We should do something against it to have a "cleaner" queue.

Unfortunately, GitHub’s issues and PRs management tools are somewhere between primitive and non-existent. The only option GitHub gives us to move a PR that requires discussion out of the review queue is to close it. Often that triggers a heated argument with its author. That has killed all attempts to clean the queue. Thus we’ve been beaten into the path of least resistance and now we're letting PRs that cannot be merged as-is rot.

Overall, I don’t think that change is impossible, but I think we’re going to see how the dust settles before considering further (r)evolutions.

-- 

Aymeric.

[Schmitt, Christian]

i know that such things drains a lot of energy. I'm working for a very small company and we changed our commit / review system entirely.

currently I think you are right at your four points, but one of your four points is somehow really bad 

> We have only one person reviewing patches regularly in his free time, so requiring as few as two reviews for a patch to be merged is already unrealistic

I know that everybody is working here. I mean some people are working for django quite hard or working for their company and doing a lot of work for django, too. But still reviewing PR's is really important for a community project. 

I know lots of things already changed and this change is great as well.

Still I think there are parts where some upstream work could be done way better by the community than by a few (with few i mean core) and thats the review process, thats why I came up with gerrit. I actually don't think that gerrit actually suites everybody's projects ( and I came to the same four conclusions in my company ;))

But maybe we are finding something that is as good as gerrit but suites the djangos project size a little bit better.

i mean i still think commits needs control, else there would be a lot of circlejerk as the master/replica thing. but still it looks that pr's take way too long for django. and thats the last thing that django could do better.

especially since there are a lots of ideas how to solve such problems in the real world..

i mean maybe we should really look at a review tool and maybe say if two people of the community reviewed it gets pipelined to the "core" which can do a final review. so it would be more community oriented and fewer "bad patches" will get to the commit pipeline.

still it takes a lot of time to teach people and setup everything. but it will put the project further ahead.

(also I'm not a core member but if we had such a review system i would definitely do more of these reviews since thats the thing i do at my company, too)

--

You received this message because you are subscribed to the Google Groups "Django developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to django-develop...@googlegroups.com.
To post to this group, send email to django-d...@googlegroups.com.
Visit this group at http://groups.google.com/group/django-developers.

To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/428911F3-292F-4F97-AB3C-B2AC5AC76798%40polytechnique.org.

[Carl Meyer]

Hi Christian,

On 08/01/2014 01:37 PM, Schmitt, Christian wrote:
[snip]

> i mean maybe we should really look at a review tool and maybe say if two
> people of the community reviewed it gets pipelined to the "core" which
> can do a final review. so it would be more community oriented and fewer
> "bad patches" will get to the commit pipeline.
> still it takes a lot of time to teach people and setup everything. but
> it will put the project further ahead.
> (also I'm not a core member but if we had such a review system i
> would definitely do more of these reviews since thats the thing i do at
> my company, too)

I'm curious why the system you propose (requiring two community reviews
to fast-track a patch) would motivate you to do more reviews than the
current system, in which it only takes a single review (by any community
member) to mark a ticket as "Ready for Check-in", which already gets it
fast-tracked for commit.

Although it does happen sometimes, there isn't a big problem (AFAIK)
with lots of patches getting marked RFC when they aren't ready;
currently there are only 3 tickets in RFC state. The bigger problem is
there aren't many people doing reviews and moving tickets to RFC state.
According to dashboard.djangoproject.com, there are 52 patches awaiting
review.

Maybe it's mostly a tooling issue, in that the current process requires
reviewers to read the contribution docs, understand the ticket state
flow, and then use both GitHub and Trac and manually mark the ticket RFC
themselves after doing a review, rather than having a review tool with a
big +1 button on it that automatically handles the state changes.

Carl

[Schmitt, Christian]

its a matter of trust.

gerrit tracks your reviews so its way easier to build up trust that the things you mark as RFC is really RFC ;)

And yes its a matter of tooling since some tools provide a better way to build trust for such community processes.

Also I'm quite young. I wouldn't want to make a "community review" just by myself since I will definitely do some mistakes over time as everybody will do.

to come back to the tooling. gerrit is just one tool. i don't know more than gerrit that has such a large feature set. but i think there are definitly some out.

In our company we use a lot of atlassian tools and atlassian has a "gerrit" tool, too but I didn't take a look at it yet. so thats another option since atlassian tools are free for community use.

the next thing is, reading contribution docks is easy, using trac is not. trac is a really blocker for a lot of things on my side. i think the ux is somewhat bad, but thats just an own perspective.

--
You received this message because you are subscribed to the Google Groups "Django developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to django-develop...@googlegroups.com.
To post to this group, send email to django-d...@googlegroups.com.
Visit this group at http://groups.google.com/group/django-developers.

To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/53DBEF42.8090000%40oddbird.net.

[Tim Graham]

Christian, have you seen the patch review checklist I put together? 

https://docs.djangoproject.com/en/dev/internals/contributing/writing-code/submitting-patches/#patch-review-checklist

That is basically what I do when I review a patch. If a patch meets those guidelines, please mark it RFC. A core dev will always do a final review, but it helps a lot if we don't have to explain those basics to new contributors. Sure, you will make mistakes, but that is the only way to learn and the risks are low. Over time, the core team will see that the patches your mark as RFC are really RFC as you said, and we're likely to ask you the join the team so you can just commit those patches that you review yourself.

Regarding Trac, would you mind explaining what Trac is blocking you from doing? Perhaps we can try to address some of those issues.

[Schmitt, Christian]

> Christian, have you seen the patch review checklist I put together?

Yep and I will definitely do some more work in the future.

Everything i write now is my personal opinion.

first of all trac has some of the weakest Query Filters. (thats my personal opinion)

the next thing is that changing the ticket and adding comments share the same button, which leads to strange things (sometimes).

also its really hard to review for specific versions. 

voting for a new feature is done on the mailing list which is okai, but i think that would be better to do in an issue tracker. (voting through tooling seems to be way simpler than posting a +1.)

also there is no way of saving filters to my profile. I mean i could bookmark them but my bookmarks are way too much already.

also it would be "a cool" feature if trac would automatically add pr's as an info to the ticket..

I mean most of these things are comfort features and still trac feels less intuitive than other tools since they are a complete platform or link everything together.

However none of these points would stop me or another individual from helping / commiting / reviewing. its just a matter of time.

tools should help us to use less time.

tools should link/work together since its easier to start if you have everything together. also i talked about gerrit, if gerrit would be introduced it woud be even harder to keep these things together since its another platform, too.

what i want to do in short is that sometimes we also should look at the infrastructure and improve on these things. i mean i'm just somebody but i think other people would say something about the currents, too.

with infrastructure i mean the things like documentation, issue tracker, code review, etc.. and i think there are plenty improvements.

like the online documentation.

To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/e41b99a9-e998-48b9-871c-c9f8b2183728%40googlegroups.com.