Release transparency

[Daniel Martí]

Hi all,

We've struggled with timely releases in the past few years; I realise
that steps were taken to improve the freeze and bug tracking processes.
Plus it's been a weird year, and there's a lot in flight right now
between big refactors and security bugs.

That said, I wanted to bring up 1.16beta1. We're just over two weeks
behind schedule (https://github.com/golang/go/wiki/Go-Release-Cycle)
and, as an outside contributor, it can be hard to tell what went wrong
or how to help.

Bryan helpfully provided this link on Slack, which does point to a few
release blockers which can't wait until after the first beta:

https://github.com/golang/go/issues?q=is%3Aissue+is%3Aopen+milestone%3AGo1.16+label%3Arelease-blocker+-label%3Aokay-after-beta1

I'd like to make a constructive suggestion: if a scheduled release is
going to be late, the release team should write a brief email to
golang-dev outlining the situation at their earliest convenience. For
instance:

* Where can one find the list of blocking bugs?
* Is help wanted from any outside contributors? (e.g. people with M1 macs)
* Is there a current estimate for the release date?
* Do we plan any changes to the release process for the future?

For example, 1.16beta1 was supposedly scheduled for Tuesday the 1st, so
one would expect an email to golang-dev from the release team that week,
be it because the release is out or because it's late.

This is not to say that the release team owes an explanation, or that Go
users are entitled to timely releases under any circumstances. But I
still think such proactive status updates would be a good way to keep
all contributors in the loop. At least from my point of view as an
outside contributor, late releases have generally been a black box.

Have a good winter break :)

[Alex Rakoczy]

Hi Dan,

Thank you for bringing this up. As you know, we’ve been trying to communicate better for each release, and some releases we do a better job than others. I think that Go has a lot to gain by the release team communicating more proactively.

Beta release timing is entirely determined by GitHub issues. The release team meets every Thursday with a subset of the Go team to discuss release-related issues. For the Beta 1 release, our main checklist that we go through is this GitHub query: 

https://github.com/golang/go/issues?q=is%3Aissue+is%3Aopen+milestone%3AGo1.16+label%3Arelease-blocker+-label%3Aokay-after-beta1

The notable components are:

• is:issue
• is:open 
• milestone:Go1.16 
• label:release-blocker 
• -label:okay-after-beta1

This is finding issues in Go 1.16 that are labeled release-blocker, that are not labeled okay-after-beta1. We go through as many of these as we have time to every week, and ensure that progress is being made.

As part of making the beta release, we remove the okay-after-beta1 label from all issues in the milestone, and they are all considered to be blocking issues for the Go release. Once there are few of these issues left, we have a discussion internally to determine if the upcoming Go release is in a good place to ask people to test an RC, and we cut that RC as soon as possible.

Another thing that we look at is issues that we file internally when testing against Google’s codebase. These issues are triaged by the Go team, and if an issue is found in Go, the issue is fixed in the public Go repository and is imported to our internal codebase. Ideally a public issue is filed for every fix, but we have room for improvement here. The internal issues do not block the beta, but they do block the RC and final release, as we are committed to running the RC ourselves internally before we ask others to trust their production systems with it. We use a public release blocking issue to track this item, see golang.org/issue/43253.

We have plenty of opportunities to do better here to keep the entire Go community on the same page as us, and invite help to get releases unblocked. In no particular order:

• Starting today, we will be sending a bi-weekly email outlining the status of the release during the code freeze.
• We’re discussing how to ensure we are filing public issues for Go issues that we discover internally.
• We’re also constantly evolving our process to keep ourselves informed, and Go stable. We might need to update the wiki to outline some of the processes I've mentioned above.

Finally, we know that our communication currently gets worse as we get further behind, as we’re often doing everything we can to unblock the release, and manual emails fall down in our list of todos. I want to make sure that does not happen again. Hopefully the bi-weekly status update helps to resolve that mistake. Our release team is also smaller than it was several months ago, and we’re doing the best to ensure that every Go release is the best one ever.

Thanks again for all your help and insight. Please let us know what you think.

Cheers,

Alex for the Go team

[Matthew Dempsky]

On Thu, Dec 17, 2020 at 2:03 PM Alex Rakoczy <al...@golang.org> wrote:

Beta release timing is entirely determined by GitHub issues. The release team meets every Thursday with a subset of the Go team to discuss release-related issues.

I think it would help if more of these regular Go team meetings were conducted openly, rather than privately within Google. For example, my impression is the golang-tools group is doing a good job at this: https://github.com/golang/go/wiki/golang-tools

I don't think this is an issue specific to the release team, or something that I think is intentionally done. E.g., I think the compiler/runtime team tends to default to discussing a lot of things internally even when most of it could be discussed openly too. But I think it would help live up to our "Go needs everyone's help, and everyone isn't here" motto if we strived to make more recurring meetings public by default.

[Ulderico Cirello]

I always refrain participating in threads in this mailing list. But this post by mdempsky kind of inspires me to join the conversation - at least to make a couple of points.

First, the fact that there are Google(rs) only meetings is sad to say the least but I believe the word is broken. Unless you're a Googler you don't have a seat on the table in many conversations. For instance, the group think struck hard with the pkg.go.dev. To the point of a) forcing me (and more strongly networkimprov) to act to prevent what would be the de facto closing source of godoc (the idea was to release pkg.go.dev in private and release an open source much later) and then b) causing Russ Cox the embarrassment of having to apologize for this event.

But we all know the root cause right? It's a Google project, ran by a lot of Googlers. And they try really hard saying that's not the case by pointing the number of external contributors - but all critical development is done inside Google.

Some say that exposing these meetings to non-Googlers would also be a nice touch in terms of transparency, but here's the problem: if you cannot always let non-Googlers to know what Googlers are discussing internally, due to the overlap of Google needs and Go programming language goals in some of these discussions. This became quite evident when Russ Cox explained that he could not just open source the code of pkg.go.dev! Alas, it had Google code in it.

Also, take in consideration the real consequences of having all the meetings in the open. You may or may not know, but there's an issue in GH tracker that analyze proposals, every week, Russ Cox posts an updates with their current status. And the top of the posts, he CCs the people who participated in the meeting. With the notable exception of Brad Fitzpatrick (now an ex-Google), they are all Googlers or people close to Googlers.

It may seem that my tone indicates that I am somehow opposed to this setup. In fact I am not. I prefer this way. If anything, I would like to see more non-Googlers being hired by Google to work on Go. 

If there's one thing that years of open source has shown, is that unless a project has a solid corporate backing to it, it dies. Google has deep pockets, and the solution is not to dilute power but to make sure Google takes more money of its treasure vault and increase the head count of the Go team. 

And be open about that. Stop pretending Go is a pure open source project. Be frank and honest that this project is a business tool, with funding, and more importantly with an ecosystem that no one can neglect anymore (if you got a service and it has an API, the one of the officially supported SDKs has to be in Go).

Go need more Googlers. For sure, better communication; but not more frustration-inducing transparency. That's the way to convert a dynamic that's broken and sad to a dynamic that's only sad. Sad trumps broken at any time.

Thanks for the patience of reading my response. 

--
You received this message because you are subscribed to the Google Groups "golang-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to golang-dev+...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/golang-dev/CAF52%2BS7MEQbu0nHnXATBwGRSr67DKd7mWaVnamddJFG%2BT5phdQ%40mail.gmail.com.

[Alex Rakoczy]

I appreciate everyone's feedback, and we'll consider a public version of the meeting. Currently, we spend a portion of the time discussing issues with Go in the Google codebase. We'd have to be mindful of everyone's time if we decide to split it into public and private meetings.

If someone is interested in what we talk about, we make all of our notes on GitHub issues. Our main goal is to make sure that progress is being made on issues labeled release-blocker. The rest of the meeting is reviewing issues labeled CherryPickCandidate, and either making a decision, or following up with package owners on the best path forward. The notes from those discussions appear on the appropriate GitHub issues. Our source of truth of the status of the release remains GitHub issues, and you can follow along on release-blocker and CherryPickCandidate issues to know as much as we do.

Because the vast majority of our notes are already recorded on issues, I'm hoping that the addition of the status email helps resolve the biggest missing part on our release communication. 

Please keep letting us know how we're doing, and we'll strive to keep improving.

Thanks,

Alex

[Daniel Martí]

Hi Alex,

Thanks for the thoughtful response. I hope you all had a good winter
break.

> - Starting today, we will be sending a bi-weekly email outlining the

> status of the release during the code freeze.

This sounds like a great idea :)

> - We’re discussing how to ensure we are filing public issues for Go

> issues that we discover internally.

I think this falls in place naturally if the release team exclusively
uses the public issue tracker to track release blockers and progress. It
should be evident when there's an internal issue that's not part of the
picture.

> - We’re also constantly evolving our process to keep ourselves informed,

> and Go stable. We might need to update the wiki to outline some of the
> processes I've mentioned above.

Definitely agree. Perhaps the wiki page could also clarify what happens
when the scheduled release date, for example February 1st, falls on a
Saturday or on a US public holiday.

Regarding Matthew's point about mimicking the golang-tools transparency;
I'll reply to that sub-thread separately, but I just wanted to point out
that I think your workflow seems to be more similar to the weekly
proposal review committee. They do leave updates on each github thread,
but they also use a review minutes thread [1] to allow users to
subscribe to all their activity and follow links.

I think that's better than just posting comments on the issues (most
users won't be subscribed to all of those issues), and could perhaps be
easier for you to link to issues, and for others to decide whether or
not to subscribe to the feed separately from golang-dev.

[1]: https://github.com/golang/go/issues/33502

[Daniel Martí]

> > Beta release timing is entirely determined by GitHub issues. The release
> > team meets every Thursday with a subset of the Go team to discuss
> > release-related issues.
>
> I think it would help if more of these regular Go team meetings were
> conducted openly, rather than privately within Google. For example, my
> impression is the golang-tools group is doing a good job at this:
> https://github.com/golang/go/wiki/golang-tools
>
> I don't think this is an issue specific to the release team, or something
> that I think is intentionally done. E.g., I think the compiler/runtime team
> tends to default to discussing a lot of things internally even when most of
> it could be discussed openly too. But I think it would help live up to our "Go
> needs everyone's help, and everyone isn't here"

> <https://blog.golang.org/open-source> motto if we strived to make more

> recurring meetings public by default.

I have mixed feelings about this idea. On one hand, I've attended almost
all golang-tools calls and I think they work great, and will hopefully
set a good example for other Go sub-teams to open up more to
non-Googlers.

On the other hand, I think it's necessary to say that the "tooling
team", meaning primarily cmd/go and gopls, is special in that its nature
requires it to have close ties with many external contributors and
projects. For example, a big portion of each monthly meeting is
discussing gopls issues which affect third party editors and
integrations. Another good example is getting early feedback on cmd/go
changes from tool authors, before the changes get fully released.

I am fully on board with other sub-teams becoming more transparent and
diluting the "googler or external contributor" line like golang-tools
has. I just have a slight worry that the compiler or release teams might
not be quite as well suited to that model, so perhaps they should take
more gradual steps towards it rather than doing a big jump.

The regular and public status updates sound like a great first step :)
Perhaps the compiler or runtime folks could do something similar by
publishing minutes from existing regular meetings on a GitHub thread.

[Jeremy Faller]

The compiler and runtime team has been discussing adding meeting minutes since Daniel proposed this, and we've started collecting them. While we're not fixed on the format we've chosen, you can find them at the below link.

https://github.com/golang/go/issues/43930

Note, this is an experiment. We might discontinue this, or radically change our approach here. But, we wanted to address the community's desires.

--
You received this message because you are subscribed to the Google Groups "golang-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to golang-dev+...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/golang-dev/20210103085727.sdxaopkns346yf3b%40carbon.localhost.

--

Warmest Regards,
Jeremy Faller

[Frederic Marand]

Since that "minutes" thread is not open to public comments, what is the best way to talk to the group about issues discussed in these meetings ?

[Than McIntosh]

>what is the best way to talk to the group about issues discussed in these meetings

Opening a thread in golang-dev or golang-nuts is the best option. E.g. 

In this comment, issue ABC was mentioned, and I'd like to say XYZ about ...

Thanks, Than

To view this discussion on the web visit https://groups.google.com/d/msgid/golang-dev/530f42e2-4a8e-41fe-ba3c-52df5844e0dan%40googlegroups.com.