On usage of mailing lists/forums for development discussions

[Francesco Guardiani]

Hi,

I would love to talk a bit about how we have technical discussions with topics a bit broader than a bug fix. I hope this mail doesn't sound like a rant but more like a productive feedback with a clear action item at the end.

When I joined the knative community ~2 years ago, my biggest struggle was "where do these people discuss? And why do we need to have meetings to discuss? Can't we just use mails?"

I still struggle with these problems after 2 years. A usual technical discussion goes this way:

• Starts from a Github Issue
• Then it's discussed on a WG meeting
• The meeting ends, the discussion then continues on slack
• Somebody chime in and say "we need a design doc for it"
• The doc is developed, but then people start adding "important comments" making harder to follow the discussion on the "limited" comment system of docs and the discussion goes back to slack (sometimes even DMs) and meetings
• After going in circles for a bunch of times between Github/Slack/doc/meetings, we need to reach a consensus
• There is no clear understanding of how we reach consensus, if leads say "yes or no", if it's just lazy consensus (but then we need somebody that writes on the pr /lgtm and /approve), if there should be a vote, ...
  

IMHO there is a very simple solution: when we need to talk about a feature, and we need to talk about it, not execute it, let's use mailing lists, forums, or any other system that allows threading and "organized" discussions. At the same time, we need to commit as a community to not use any other system we have (slack/meetings) to follow up on these discussions. Everything has to remain in the same place to make this working. This has the important benefits of being more inclusive (anybody can jump in the discussion, read the thread and comment) and helps to keep track of the discussions.

If there is a need to create a document to enhance the discussion, that's fine, but then let's continue to discuss the doc in the mails themselves (unless of course typos etc).

If at some point nobody replies to a thread anymore, that's fine, lazy consensus kicks in and that means people are ok with that. This will encourage people to be engaged in the discussions, more than rarely pointing out something wrong in a meeting without actually joining the whole discussion and providing more context on "what is wrong".

Now, I don't actually care if we want to end up using this mailing list, Github discussions, some old but fancy bbforum, or whatever other tool we like. My personal preference is mails, they are the openness for excellence, they don't require accounts to use them, they're easy to index, to search and to organize, and also I believe they have this "mind power" of making people reason more before sending them (because you can't remove the mails!). But again, I want to emphasize that the problem is not which tool, but rather the way we organize it. If we want to use a tool that has a little CSS in it, I'm fine with it 😀. What matters is the way we communicate, and how we keep "discussions" organized and in the same spot to let other people chime in and follow the discussion from the beginning to the end.

As I said, I didn't mean this mail to be a rant, I put myself between one of those people that made all the mistakes above underlined. All I want is to help make this community more inclusive for us and to attract new contributors.

I will personally try from today onward to use mails, or any other tool the community wants to use, to discuss features I'm working on, and I will encourage other people to reply me in the place where the discussion is actually happening. Prepare to be asked in WG meetings to go on the mailing list and reply to me 😂. I encourage every other community member to do the very same and to keep discussions outside other means of communication, except mails (or again, any other tool we would like to use for this purpose).

Let me know your thoughts,

FG

--

Francesco Guardiani

Software Engineer

Red Hat Srl

fgua...@redhat.com   

[Matthias Wessendorf]

On Fri, Feb 12, 2021 at 9:44 AM Francesco Guardiani <fgua...@redhat.com> wrote:

Hi,

I would love to talk a bit about how we have technical discussions with topics a bit broader than a bug fix. I hope this mail doesn't sound like a rant but more like a productive feedback with a clear action item at the end.

When I joined the knative community ~2 years ago, my biggest struggle was "where do these people discuss? And why do we need to have meetings to discuss? Can't we just use mails?"

+1 I do like mails for discussions and voting on topic that need a clear/clean decision.

 

I still struggle with these problems after 2 years. A usual technical discussion goes this way:

• Starts from a Github Issue
• Then it's discussed on a WG meeting
• The meeting ends, the discussion then continues on slack
• Somebody chime in and say "we need a design doc for it"
• The doc is developed, but then people start adding "important comments" making harder to follow the discussion on the "limited" comment system of docs and the discussion goes back to slack (sometimes even DMs) and meetings
• After going in circles for a bunch of times between Github/Slack/doc/meetings, we need to reach a consensus
• There is no clear understanding of how we reach consensus, if leads say "yes or no", if it's just lazy consensus (but then we need somebody that writes on the pr /lgtm and /approve), if there should be a vote, ...
  

ACK

 

IMHO there is a very simple solution: when we need to talk about a feature, and we need to talk about it, not execute it, let's use mailing lists, forums, or any other system that allows threading and "organized" discussions. At the same time, we need to commit as a community to not use any other system we have (slack/meetings) to follow up on these discussions. Everything has to remain in the same place to make this working. This has the important benefits of being more inclusive (anybody can jump in the discussion, read the thread and comment) and helps to keep track of the discussions.

If there is a need to create a document to enhance the discussion, that's fine, but then let's continue to discuss the doc in the mails themselves (unless of course typos etc).

I do like "Feature Track" docs, but I think that somethings it's just simple to provide "feedback", inside the GDoc - However, doing a discussion inside of GDoc is completely wrong. Fully agree on your point there.

 

If at some point nobody replies to a thread anymore, that's fine, lazy consensus kicks in and that means people are ok with that. This will encourage people to be engaged in the discussions, more than rarely pointing out something wrong in a meeting without actually joining the whole discussion and providing more context on "what is wrong".

true! I like the Apache rules for this subject. Works fine for a lot of projects there.

 

Now, I don't actually care if we want to end up using this mailing list, Github discussions, some old but fancy bbforum, or whatever other tool we like. My personal preference is mails, they are the openness for excellence, they don't require accounts to use them, they're easy to index, to search and to organize, and also I believe they have this "mind power" of making people reason more before sending them (because you can't remove the mails!). But again, I want to emphasize that the problem is not which tool, but rather the way we organize it. If we want to use a tool that has a little CSS in it, I'm fine with it 😀. What matters is the way we communicate, and how we keep "discussions" organized and in the same spot to let other people chime in and follow the discussion from the beginning to the end.

As I said, I didn't mean this mail to be a rant, I put myself between one of those people that made all the mistakes above underlined. All I want is to help make this community more inclusive for us and to attract new contributors.

I will personally try from today onward to use mails, or any other tool the community wants to use, to discuss features I'm working on, and I will encourage other people to reply me in the place where the discussion is actually happening. Prepare to be asked in WG meetings to go on the mailing list and reply to me 😂. I encourage every other community member to do the very same and to keep discussions outside other means of communication, except mails (or again, any other tool we would like to use for this purpose).

I think this mail is spot on. Slack to me is sometimes like poison. The `@` mentions cause some level of stress (I am guilty here too), and sometimes I feel the discussion needs to be happen NOW. NO.... it should happen two minutes ago.

Email: Gives you the full power of async communications. You have time to prepare anweser. You can put much better scope/context into an email, rather than doing slack chatters, they feel to me like fire drills.

-M

 

Let me know your thoughts,

FG

--

Francesco Guardiani

Software Engineer

Red Hat Srl

fgua...@redhat.com   

--
You received this message because you are subscribed to the Google Groups "Knative Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to knative-dev...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/knative-dev/CAMtVRjsRf0x3PO5od7zvok1gKOForu-MSJo6dA%3Dc5L3Z4mWjJg%40mail.gmail.com.

--

Matthias Wessendorf

github: https://github.com/matzew 
twitter: http://twitter.com/mwessendorf

[DL duglin]

I can understand where you're coming from. I definitely would prefer a single communication channel, and I'd prefer for it to be GitHub so that the conversations live as close as possible to the code. However, I also realize that GitHub is not good for everything. As you noted, we sometimes need to work on a doc and while editing a file via GitHub PRs is do-able, it's not the best. And sometimes the immediate back-n-forth of slack (or calls) really speeds things up.

So, to me the non-github channels are more like secondary mechanisms that help the overall goal. But, and this might be the key thing that's missing, once those side-chats are done I think we need to move all formal resolutions back into GitHub for official tracking purposes. Then if those side-chats vanish (which we should delete to avoid confusion when people look at old docs) we still have a record of what was agreed to, why and who approved it.

This doesn't help the fact that you may still need to monitor multiple channels, but it will at least help with the tracking and ability to see the history.

just my 2c

-Doug

[Francesco Guardiani]

> And sometimes the immediate back-n-forth of slack (or calls) really speeds things up.

That may be true in the short term, but in the long term calls have the following "side effects": excluding people not joining in the calls, like users and individual contributors, making it harder to follow then what the conversation outcome was. I personally lost track of some topics I personally bootstrapped, because of those side effects. And also note that I wrote this mail because I personally noticed that 99% of technical discussions we have in meetings can actually happen async, I rarely saw situations where we actually needed to have them in a meeting.

> But, and this might be the key thing that's missing, once those side-chats are done I think we need to move all formal resolutions back into GitHub for official tracking purposes

Is there any reason why emails can't be "official"? Is there any reason why lazy consensus cannot work in Knative and we need A, B and C taking a decision? And who are A, B and C? In any case, I don't want to sidetrack the discussion here, decision making deserves a mail thread on its own (feel free to open a new thread to talk about it).

> This doesn't help the fact that you may still need to monitor multiple channels, but it will at least help with the tracking and ability to see the history.

What I'm saying is that we need just 2 channels: Github for action items, bugs and code. The rest IMO should go on a mailing list (or similar tool) and, when necessary, docs to elaborate complex proposals.

[Markus Thömmes]

Heya,

I agree with pretty much all you're saying regarding asynchronous communication and us currently not doing a great job with them always. I'd absolutely be game in trying more ML based communications throughout, though we have to recognize that meanwhile people are used to a certain style of working/chatting etc. Breaking habits and forming new ones is super hard.

One baby-step I could imagine being worthwhile is requiring to share every Feature Track document to the mailing lists. That'd make sure everybody sees what's going on and give people a place to discuss the Feature Track in broader strokes than might be possible inside the Google Doc.

Overall: +1, thanks for bringing this important topic up Francesco!

Cheers

Markus

[Lionel Villard]

While I totally agree with the "there is no clear understanding of how we reach consensus" part, I'm more skeptical about forcing on people just one communication channel, which IMO won't solve the core problem you raised. 

 

I would rather work on improving the Feature Track process, and as Markus already suggested, at the very least advertising FTs on the mailing list(s). But that's not enough. There should be a clear path from proposal to approval. Another idea is every week, during WG meetings, we should bring up all active FTs for a very quick status update and act on it (eg. appoint reviewers and put a deadline). I also think that when appropriate, for big FT, we should be less hesitant to create a task force and a dedicated slack channel for focused discussion (as an example, look at the async and vanity domain FTs). 

 

I also like Doug's suggestion of keeping track of discussions in Github, close the code, as a single point of entry. It does not mean that all discussions have to happen in Github, but they should be referenced from there (eg. link to a slack thread, or slack channel, google doc, video recording, etc...). From the issue there should be a link to the FT which is, at the end of the day, the authoritative document.

 

Hope this helps.

Lionel

 

 

 

To view this discussion on the web visit https://groups.google.com/d/msgid/knative-dev/57e47adb-f995-4848-a213-e5ac78e3135fn%40googlegroups.com.

 

[DL duglin]

> > But, and this might be the key thing that's missing, once those side-chats are done I think we need to move all formal resolutions back into GitHub for official tracking purposes
>
> Is there any reason why emails can't be "official"? Is there any reason why lazy consensus cannot work in Knative and we need A, B and C taking a decision? And who are A, B and C? In any case, I don't want to sidetrack the discussion here, decision making deserves a mail thread on its own (feel free to open a new thread to talk about it).

I think those are two separate things. I don't think emails should be the "final official" statement for things that should be done in GitHub. When someone, a year later, wants to understand the status of things I think it would be a much nicer UX if they could just scroll thru a GitHub issue/PR to see the latest status. Having to jump to a remote medium (email archive) to see the final resolution is less than ideal - especially if there are many emails referenced in that issue and the latest one may not be the "decision" one.

re: Lazy consensus... no comment on that.

> > This doesn't help the fact that you may still need to monitor multiple channels, but it will at least help with the tracking and ability to see the history.
>
> What I'm saying is that we need just 2 channels: Github for action items, bugs and code. The rest IMO should go on a mailing list (or similar tool) and, when necessary, docs to elaborate complex proposals.

In general, sure I think it makes sense to reduce the number of channels, but I think reality is that you'll never get people to not use slack, calls or docs to discuss things. So keep trying :-) but in the meantime let's ensure that someone wanting to find the status of something can do so easily and consistently.

-Doug

[Cory Cross]

• Somebody chime in and say "we need a design doc for it"
• The doc is developed, but then people start adding "important comments" making harder to follow the discussion on the "limited" comment system of docs and the discussion goes back to slack (sometimes even DMs) and meetings

The Tekton project writes their documents in markdown so you can have semi-threaded conversations:

https://github.com/tektoncd/community/pull/137

Cory

[Francesco Guardiani]

> There should be a clear path from proposal to approval.

I agree with that. but as I said this thread is about the actual discussion, not the approval process. Let's open another thread for that.

> Another idea is every week, during WG meetings, we should bring up all active FTs for a very quick status update and act on it (eg. appoint reviewers and put a deadline). I also think that when appropriate, for big FT, we should be less hesitant to create a task force and a dedicated slack channel for focused discussion (as an example, look at the async and vanity domain FTs).

> (eg. link to a slack thread, or slack channel, google doc, video recording, etc...)

IMHO none of these solutions actually solves the problem of including users, individual contributors and even us (which I classify in this thread as full-time contributors). On the contrary it will just encourage closing the discussions even more around the same group of people. We need to expand this community and make it easier for new people to join, not closing it more and more to make us more efficient and more focused on things. Users, individual contributors and even part of us will never be directly involved in everyday WG meetings, subscribed to all slack channels, they won't watch video recordings etc. And sometimes there are reasons that we can't really "control" that will never make them join in meetings, like lack of time, bad time block, not being comfortable with spoken english (my GSoC student, now a prolific individual contributor, had this problem), and I can go on with more and more reasons. I don't want to let these people feel excluded, I want to include them. My thesis is that encouraging sync communication channels will just make these people feel more and more excluded.

I personally put myself between people who "excluded", asking to individual contributors to join meetings when it was not necessary (and where they eventually not joined), and between who "feels excluded", because for example I would love to follow the progress of productivity WG and help, but I can't for the clash with another meeting.

Of course, I'm not saying we should abandon meetings, but I'm saying that they should be our last resort for technical discussions and we should be as much async as possible. Ideally, we should do WG meetings only if there's an actual need for discussing something that for some reason can't be brought up on a mail or a github issue. Meetings are also useful as a "bulletin", but we can actually do that with mails, like the weekly bulletin of FTs you proposed. I believe google groups even have the automatic bulletin feature.

About slack: IM discussions are "ephemeral" and unstructured, I put myself between those people that need 10 messages to explain a concept, where actually 9 of them were not useful to actually explain it. This very same discussion would have taken 1/2 meetings and probably 100+ IM messages (there is already a thread open in #toc-steering-questions long 30~ messages). Mails/forums solve this efficiency problem, they force people to structure their concepts, reason about their comments. Like I always say "when I write a mail (even this one), I read it 10 times before sending it to make sure what I wrote makes sense". IM and meetings don't have this "mind power" at all and don't help having an overall view of the problem.

> When someone, a year later, wants to understand the status of things I think it would be a much nicer UX if they could just scroll thru a GitHub issue/PR to see the latest status

If that github issue is full of slack links (maybe most of them expired), video recordings, countless draft docs, the UX is not nice at all, It will just make people give up.

Let me give you an example, a mess I personally made, look at my issue about the enhanced trigger filtering: https://github.com/knative/eventing/issues/3359. It quotes a bit the steps that happened throughout this discussion, but it's not clear the status, why we didn't choose JS (which is explained in one of the other nested issues/prs), what are we trying to do now (which I talked about during wg meetings, but it's written nowhere in knative channels), the previous work with CEL is just vaguely quoted, the json schema thing people didn't really liked (it was sad during meetings, but i don't recall which WG), and so on.

Another example, a simpler discussion that had the same problem: https://github.com/knative/eventing/issues/4515. This one jumped through 2 different working groups, slack and github issues. At the end of the day the decision was made "between us" and no external user will ever know how, unless they dig into the 2/3 meetings we had where we talked about it. Today I've found out a problem about that issue, and I'll go yet another time around 2/3 meetings to discuss the implementation detail.

I can talk at least for Eventing, but most "big discussions" IMO are in a situation like that (I'm not pointing to anybody and again, I put myself between people that contributed to these problems).

If the discussion is in a single mail thread, I'll just go through the thread, read it and understand the full story of the discussion. If I don't want to read the full thread, that's fine, I'll just go to the end and read the last mails to understand the status. Maybe I want to read just the doc? Cool, I'll search the doc link in the thread and go read it, which is being updated with the feedback from the mail replies. No need to jump through video recordings, long and inefficient slack discussions, then dig in 10 different github issues/PRs, and so on. A discussion thread and a design doc updated with the feedback is all we really need IMO.

And even without considering all the above mentioned problems about slack and meetings, when you claim "let's have a github issue where we track the discussions with links etc", realistically who's going to maintain such issues?

[Ahmed Abdalla]

Hi Francesco,

thanks for speaking up your thoughts, this is definitely an important topic which I personally had a hard time with in Knative. I'll put my thoughts in bullet points

1. If I were to describe an ideal desired state of a community, it'd be not very far from what you've described. However, I keep reminding myself that we need to stay open and realistic that changing people's habits let alone a whole community is very hard (ack Markus point). So, let's change things gradually and see if we'll improve.
   
2. It is indeed difficult now to understand what's in flight, and figure out the context unless you're involved in many communications channels (slack, online meetings, etc).
3. Decision making has been a big gray area in Knative for me (specially in eventing) where the rule I have in mind became (you need to get those few key people on board for a decision to be made). How? On which communication channel? is it clear what to do if some of the key people don't have enough enthusiasm to push the decision forward? no to all that

someone pointing out something wrong in a meeting without actually joining the whole discussion and providing more context on "what is wrong" ~

This happened many times with me in the project, and I concur it. Sometimes a key person might point out that "this is wrong" during a meeting or even a comment on a GitHub issue, without providing more context on "what is wrong" or even "how can I make it right" and I'd be left chasing people all over. While I can imagine if there's a standard decision making process with a standard communication channel (ML or GitHub issue), others would be able to chime in or you'd not be blocked because of the lazy consensus. 

Now that said, I believe the change we can focus on, is enhancing the proposals process (Feature Tracks?) regarding:

1. Source of truth: there should be a clear expectation of what is the single source of truth of the status. Is it a GitHub issue that links to the FT doc? the FT doc itself? (i.e copy key comments from the GitHub issues into the doc?) else? (Markdown with the FT akin to TEP)? ML?
   
2. Decisions communication channel expectations: There needs to be one communication channel that is "the formal" communication channel for decisions, their technical discussions, and their status updates (approve, reject, etc) which meets at least these two criteria: 
   
1. Fully asynchronous, ideally a contributor (not necessarily WG leads) wouldn't even need to join any meeting to smoothly propose a FT, follow up its approval/adoption and implement it.
   
2. Preserves history
   
3. Enhancing decision making. AI for me: as Eventing execution lead I will bring up the need to have a clear decision making process in the WG and propose the lazy consensus idea to the WG leads and rest of the WG members.

So, concretely I propose we enhancing the proposal process by making a choice for each of these:

1. Source of truth (choose one)
   
1. Feature Track docs
   
2. KNEP ( KNative Enhancement Proposals) as MD files
   
2. Formal communication channel for technical discussions and decisions of proposals. (choose one)
   
1. Mailing list
   
2. Github Issues
   
3. Adopting lazy consensus as a decision making process. (You can see Apache's here https://community.apache.org/committers/lazyConsensus.html)

To summarize, the change I'm trying to introduce that might improve the situation you brought up Francesco, is to not say "hey let's stop using slack or meetings" , because that's very disturbing and a big change from a large number of folks that got accustomed to, I'm saying let's introduce one change aside to what we already do: a better structured proposals process (source of truth, communication channel, decision making). 

I believe if we introduced that proposal process and started exercising it, the synchronous coms like meetings, IMs will lose its current importance and there's a big chance at some point they become secondary optional means of comms or totally dropped. 

--

You received this message because you are subscribed to the Google Groups "Knative Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to knative-dev...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/knative-dev/CAF33DkoaMRH8yg27EM0UD57hPnNDKS%3DwtejU83JCCsWzECZEDw%40mail.gmail.com.

--

Ahmed Abdalla

[Francesco Guardiani]

> So, concretely I propose we enhancing the proposal process by making a choice for each of these:

For me the choices are 1 and 1 and +1000 on the apache lazy consensus, I can elaborate on why my choices.

Can you try to elaborate this decision process in another mail thread? :) We don't need to discuss it in the WG meetings, we can just tell people to get on the thread and reply there.

> Source of truth: there should be a clear expectation of what is the single source of truth of the status. Is it a GitHub issue that links to the FT doc? the FT doc itself? (i.e copy key comments from the GitHub issues into the doc?) else? (Markdown with the FT akin to TEP)? ML?

Source of truth is the doc IMO, but the doc should not contain the actual discussion, which should live in a "persistent" and threaded discussion system like mails and forums. Doc comment systems are awful for complex discussions and you can clearly see it from our FT docs comments where important concepts are scattered in tiny comment boxes that sometimes disappear because one removes the sentence where the comment was added. I like markdown committed to a github repo, a lot of communities does that (to bring another example, the Golang community) but having a tool where two or more people can easily contribute in a doc is usually better, so my personal preference is that a system like google docs or hack.md is fine as long as we don't abuse their comment systems.

> I believe if we introduced that proposal process and started exercising it, the synchronous coms like meetings, IMs will lose its current importance and there's a big chance at some point they become secondary optional means of comms or totally dropped.

+1000 on this point, my motive behind this thread was more "I would love to have both formal and informal technical discussions in mails", hence my action item "I'll try to start with my discussions and I'll encourage people to reply my threads, I encourage all of you to do the same". That sounds pretty gradual, isn't it? :) I'm not saying you all must not use chats and meetings, I'm saying that for the discussions started/moderated by me I will prefer from now on mails. I want to underline that we don't have an actual policy of whether communication tool we should use: this mailing list is, after all, an official communication channel. So I encourage you Ahmed to do the same with your decision process proposal!

 

To sum up the decision process you proposed, I like your plan, let's do it, but we need to commit to don't having technical discussions about FT outside the formal channels, otherwise it won't work. It will just make the situation worse, adding another channel where the discussion is scattered. Generally, as you underlined, in my ideal community, both formal and informal technical discussions are on mails. I want to underline that having informal discussions on mails is as important as having formal ones, because usually a FT comes after having a discussion on a problem, on a missing feature, it doesn't come out of nowhere. Sometimes missing the first part of the discussion, before the FT, makes a difference. And also note that a lot of actual important discussions don't need FT, maybe because they're changes not worth a whole doc, so I'm not bought in forcing people in a formal process for every fairly small change. I believe it's really up to us to understand when a doc is needed and when it's not: if we have the discussions in the same place, figuring out where this "border" between doc and non-doc lives becomes less important.

[Julian Friedman1]

I’m pretty sympathetic to the idea of “one channel, which should be async and persistent”, but it seems like inevitably - given the need to code review (etc) - a lot of relevant discussion will be on GitHub isssues / PRs, and given many users will report issues/raise feature requests on GitHub and will not (I think?) join a mailing list to see responses it seems like *either* GitHub issues/PRs/discussions are the one channel, *or* there are multiple channels. Is this a fair statement or do you disagree?

Sent from my iPhone

On 14 Feb 2021, at 09:31, Francesco Guardiani <fgua...@redhat.com> wrote:



To view this discussion on the web visit https://groups.google.com/d/msgid/knative-dev/CAMtVRjtAOzUYGLnZ%3D8vAjanDN5mjE1OktZxhvyKqcY16FqgkMg%40mail.gmail.com.

Unless stated otherwise above:
IBM United Kingdom Limited - Registered in England and Wales with number 741598.
Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU

[Francesco Guardiani]

> a lot of relevant discussion will be on GitHub isssues / PRs,

> will not (I think?) join a mailing list to see responses

For straightforward feature requests and issues, I don't expect any user to join a mailing list (nor a slack chat or a meeting), everything will likely happen and remain on the original Github issue and PR, like it happens today. I'm talking about "more complex" technical discussions, the ones often related with feature tracks, the ones we have scattered through several meetings, issues and slack chats. And as a matter of fact, I expect users and individual contributors to join mailing lists more than joining slack or meetings, it's very usual in other communities to get asked to "develop a ft doc and send it to the mailing list". Today we do the same, but we ask users to go talking about it in a meeting.

When the "broader" discussion starts to get more "concrete", I expect the discussion to move to github PRs, because that's of course where the code lives. But that doesn't mean the discussion becomes disorganized, because in the ML we'll always have links to the actual "action" on GH, and this will likely be almost natural: e.g. after discussing a feature, after we agreed on it, i'll just reply to the thread with "hey, this is the implementation PR, check it out". If in the PR the issue is strictly about the PR, I expect to talk about it in the PR. If the issue is broader, I expect to talk about it in the mail thread. That's the very same of what we do today, except the latter is done on slack or in meetings. What I'm trying to say is that IMO there's no need to "force the discussion organization".

> Is this a fair statement or do you disagree?

I disagree in the sense that, at the end of the day, Github discussions is a different communication channel from issues and PRs. Yes, they're on the same website, but they're not meant to have the same kind of discussions: Github discussions is more or less a forum (with editable and removable comments), so it resembles more the kind of discussions you could have on a mailing list than on a PR. My personal preference is mails, we have a mailing list already in place and I don't see any reason to configure yet another tool, but as I said in the first mail I'm fine if the community wants Github discussions or mybb or whatever other tool.

[Scott Nichols]

I think this email thread is a great example of hard to follow email chains. 🙆

To view this discussion on the web visit https://groups.google.com/d/msgid/knative-dev/CAMtVRjsCp7kqjmBQNypfvi2EM1F6ockq1em4gnYkZ6FYAjCR_Q%40mail.gmail.com.

[Ahmed Abdalla]

On Sun, Feb 14, 2021, 21:11 Scott Nichols <deo...@gmail.com> wrote:

I think this email thread is a great example of hard to follow email chains. 🙆

I'd counter argue that if the same thread content was on slack, it'd have been impossible to follow. Specially given the fact that it started a few days ago

To view this discussion on the web visit https://groups.google.com/d/msgid/knative-dev/CAFvqHVRB1PPObp4ONz5pVnoB_KLpxLrA-1Y9xjW6MO2o4ZOw5g%40mail.gmail.com.

[Scott Nichols]

But they result in the same thing...

[Ahmed Abdalla]

I'm not sure they do. I like the explanation of the difference in this blog post https://emmanuelbernard.com/blog/2021/02/10/communicating-within-remote-teams/#use-emails-and-not-some-instant-message-communication-tool-slack-etc

[Jimmy Lin]

Thanks Francesco for raising the issue. I would vote for using github as the only source of truth because:

1. The open source project I worked on before using email threads, but there is also an automatic job to convert email threads to a google group for better reading experience. IMO that's because it is before github became popular and there were already discussions about moving it to github.

2. If you watched the github project like me, you will get all the email updates anyway.

To view this discussion on the web visit https://groups.google.com/d/msgid/knative-dev/CAM595D%3DBmPBKpTrzppZ1w7qbVAQfjj%2BGgW%2BwFTgBUs%2BxcOiqyA%40mail.gmail.com.

[Francesco Guardiani]

> The open source project I worked on before using email threads, but there is also an automatic job to convert email threads to a google group for better reading experience. IMO that's because it is before github became popular and there were already discussions about moving it to github.

Mailing lists are still very popular among most open source projects :)

> I would vote for using github as the only source of truth because:

The problem I see with Github discussions/issues is that they're strongly coupled to the organization of the repos and not of the "community", IMO that model doesn't fit "larger" discussions. E.g.: where the sources WG should discuss a specific problem? Where the event delivery WG should talk about kafka configs unification (note: there are 2 kafka repos, and there may be even more)? Where productivity WG should discuss a specific release process change that spans across several repos?

So Github is fine as soon as the problem is "smaller", connected to some specific code in a specific repo, but things get hard to follow when you have broader discussions (which might want to have attention from other WGs too) and, for that reason, I generally prefer a system which allows us to organize discussions based on community organization, more than code organization. Today we have a single "dev" mailing list, but if the traffic increases, we might at some point decide to split it in serving and eventing MLs. If the traffic gets too high again, we might decide to have an ML for every WGs and even have specific MLs for task forces. What I'm trying to say is that IMO, on the long term, mailing lists/forums just scale better and scale in relation to the community more than to the code.

[Aizhamal Nurmamat kyzy]

Hi all,

I wanted to chime in with a little bit of my experience from working with Apache open source projects (mainly Beam and Airflow). It is due to topics like these that Apache Software Foundation(ASF) projects follow the principle of "If it didn't happen on the mailing list - it didn't happen". Below is the summary:

When all discussion happens on mailing list and is openly available in a single place:

• A geographically distributed community can work together asynchronously
  
• Users can monitor features, and roadmaps through it
• New contributors can use it to map the state of the project
• Ensures that important decisions are not taken privately
• Communication in open channels builds a shared vision

As a result we strive for:

• Transparency that allows open development
  
• Ensuring that everyone has an equal voice
• Creating an even playing field with equal opportunity to contribute

As for smaller fixes, projects use GitHub issues, and pull requests for discussions. All big code changes are usually accompanied by design docs that are discussed in the mailing list first, and later their final version is added to a public repository of design documents for the project. For example, both Beam and Airflow use cwiki to store theirs[1][2]. This helps stakeholders to track the project roadmap and status for each feature development.

The ASF halso has guidance/customs around decision making. It is -again- always on the mailing list, and via voting. The general process for a vote is the following:

1. The decision making happens by voting with +1 (approve), 0 (neutral), -1(disapprove). When voting -1 - always need to provide context why you disagree and if possible, provide alternative solutions
2. The author that opens the thread should allow at least 3 business days to receive feedback. This is to accommodate folks from different time zones and allow enough time for reviewers to provide thoughtful feedback.
3. If the thread doesn't get any feedback, "lazy consensus" is assumed after 3 days.

This has worked pretty well for our projects. Hopefully some of this will be helpful for the Knative community as you kickstart these discussions. 

I am happy to share more if anyone is interested. I once prepared these slides(https://s.apache.org/apache-way-for-everyone) to talk about the Apache Way. I'd love it if they could help you in any way.

To view this discussion on the web visit https://groups.google.com/d/msgid/knative-dev/CAMtVRjurVsbXynHm_PUpJ1gq8PtgzXM6ceeRRe34rJuP0k4wwA%40mail.gmail.com.

[Omer Bensaadon]

Hey folks, wanted to chime in here with some UX WG perspective. 

I agree with the overall ethos of this discussion and am glad it is being broached.  Email lists are great, but they are not discoverable.

I can easily point a user or contributor to an on-going Github Discussion and say "you should chime in here!" doing the same thing via email is considerably more difficult. Github and Slack also have integrations, which means we could have new discussions post in Slack so everyone can see them in the same place they see everything else Knative related. 

Overall, I believe Github Discussions to be a worthwhile experiment, at the very least. Is there a way we could "test" this process with a Github Discussion about just 1 feature?

Best,

Omer B. 

[Omer Bensaadon]

PS: when I say "discoverable" here I mean discoverable via Google (i.e. folks outside of our community) 

[Brenda Chan]

I agree that we should experiment with Github discussions and I'm happy to enable this on the Github side if any WGs are willing to try it out.

My assumption is that most folks who are interested in Knative-related discussions already have a GitHub account so the barrier for contributing to the discussion will be low compared to having to sign up to a mailing list/slack org. 

---

From: knati...@googlegroups.com <knati...@googlegroups.com> on behalf of Omer Bensaadon <omerbe...@knative.team>
Sent: Wednesday, February 17, 2021 12:01 PM
To: Knative Developers <knati...@googlegroups.com>
Subject: Re: On usage of mailing lists/forums for development discussions

 

To view this discussion on the web visit https://groups.google.com/d/msgid/knative-dev/afb3e2ba-0915-4850-8ae2-6c15c1feca75n%40googlegroups.com.

[Francesco Guardiani]

> I can easily point a user or contributor to an on-going Github Discussion and say "you should chime in here!" doing the same thing via email is considerably more difficult.
> Email lists are great, but they are not discoverable.

With Google Groups, this is not true. Every mail thread has a link, each mail has a link and mails are very well indexed in search engines. This is this thread permalink https://groups.google.com/u/0/g/knative-dev/c/O4gbnJ8pZ5Y and this is your mail link https://groups.google.com/g/knative-dev/c/O4gbnJ8pZ5Y/m/sMU1wTo1FgAJ . Try to search any average-complex question about go and you'll probably land on go-nuts google group.

> contributing to the discussion will be low compared to having to sign up to a mailing list/slack org.

We can have people join this mailing list without being part of the org, so people can join just by going to google groups and sign-in, it should be relatively straightforward I guess. TBH I never understood why in first place this mailing list is "closed".

To view this discussion on the web visit https://groups.google.com/d/msgid/knative-dev/MN2PR05MB68788707DDFED3DF025CB4DFD2869%40MN2PR05MB6878.namprd05.prod.outlook.com.

[Omer Bensaadon]

I couldn't find this thread on Google^^^ 

The fact that it can be accessed doesn't mean it is easily accessed... 

Having everything on GitHub encourages wider participation IMO 

[Ahmed Abdalla]

Hi Omer,

Thanks for chiming in. I understand where you are coming from. These are my thoughts:

1. I believe we need to make sure our priorities are set straight and we have the right tool for the purpose. I think we're here talking about a communication channel intended for the "contributors". I.e users who need to awaringly take some initial steps to become "contributors". Things like join slack, the ML (that's mandatory to view any meeting recording, design doc or WG notes). So with that in mind, I think the barrier to entry becomes irrelevant/insignificant? no?
2. With that in mind, I think if we set a "contributor" persona in mind for the "dev" mailing list, we're looking for a tool that achieves the following in priority (as I mentioned early in the thread):
1. Be fully asynchronous. 
2. Pertains the history
3. Have a high SNR, and I would like to clarify some thoughts here.
1. IMHO Slack/Discussion tools have a much lower SNR compared to a ML (i.e higher noise than MLs)
1. The structure of the information is predetermined for all users. (You setup channels in predetermined order) and you can't "filter" the messages you need from there (unlike email, where it's a free form, you get all messages and you can set up the labels, categories, sub-categories, folders....etc)
2. There's no "collapse read messages in a thread except new ones" nor "if a thread is all read, keep it off my sight" as you can do with email. You get a notification about a thread and you have to scroll through all the messages to see the newest ones at the bottom unless. 
2. In slack/discussions etc, you need to jump into all different "channels/categories" that were "predetermined" for you to get the information you want, while in email, the information comes to you if that makes sense.
   
4. Have a high delivery guarantee. Email has a "delivery guarantee", "you've been informed". Think of it as your post mailbox where there's a guarantee that you'll receive your ballot as long as you've the correct address and name on the mailbox. That's important for communities adopting "lazy consensus", cause literally every single contributor gets his own copy of proposals/decisions.
5. Discoverable/linkable/searchable
1. for the intended users "i.e contributors", IMHO searching my own inbox is way more efficient than searching slack/discussions. 

I think Discussions/Slack/StackOverflow etc are very useful and have their own other benefits, but for the "contributors formal communication channel", the one where f "If it didn't happen on - it didn't happen email/MLs are a clear winner IMHO.

To view this discussion on the web visit https://groups.google.com/d/msgid/knative-dev/f33f2b93-bf0b-4513-881e-dc9de7f5e8c4n%40googlegroups.com.

--

Ahmed Abdalla

[Paul Morie]

I'm +1 on using mailing lists for async communication. We faced similar problems in Kubernetes at one point where decisions would be made in slack or even on twitter (!!), causing confusion and alienation amongst the community, especially developers.

What we did is:

- Move decision making / debates to issues / mailing list / meetings (with lazy consensus) and away from slack/twitter

- Emphasize lazy consensus to fight the FOMO effect

It helps, but it definitely takes some adjustment / work to fully adopt. The temptation will be there to follow muscle memory. As long as people are willing to work to adopt this scheme, and trust each other to get there in time, I think it can be very successful.

Paul

To view this discussion on the web visit https://groups.google.com/d/msgid/knative-dev/CAM595DntN535yDoG3o3M78k%2BpCt9MdDKqtcF9idttM1ivKZkqA%40mail.gmail.com.

[Jimmy Lin]

FYI, this is an example of how LLVM community (a popular compiler framework) improves the readability of email threads by automatically converting them to a forum. I think small scope discussion in github and larger scope for decision in ML makes sense to me.

https://groups.google.com/g/llvm-dev/c/XgEkN9oNdVs

To view this discussion on the web visit https://groups.google.com/d/msgid/knative-dev/CACNw8mqLVLP4GmKjXKgrOYOOEbcJfFjXpPZhrOPk092W%3DGXFAw%40mail.gmail.com.

[Ahmed Abdalla]

Hey folks,

I created a proposal for the TOC https://github.com/knative/community/issues/474 . Feel free to chime in.

--

Ahmed Abdalla