plans to redirect godoc.org to go.dev

6,035 views
Skip to first unread message

Dan Kortschak

unread,
Jan 31, 2020, 4:46:14 PM1/31/20
to golang-dev
In the "Next steps for pkg.go.dev" blog post[1] there is an indication
that godoc.org will be redirected to pkg.go.dev.

I want to say that I think this is badly misguided given the current
status of go.dev's ability to correctly (notwithstanding [2] being
closed) identify licenses (and the restrictive set of licenses that it
will even try to identify) and the consequences for packages that are
not identified as having a "correct" license.

I don't use licenses like WTFPL or CC0 (though some gonum packages are
dual licensed BSD3/CC0-1.0-PDD), but some people in the community do
use these licenses and due to [3], if godoc.org redirects to
pkg.go.dev, these people will no longer have a commonly known public-
facing documentation outlet. It should be noted that there are many
license that are listed as OSI-approved, but which don't fit into the
narrow list of what is acceptable to pkg.go.dev.

This heavy handed approach to what is happening is really disappointing
to me.

Dan

[1]https://blog.golang.org/pkg.go.dev-2020
[2]https://github.com/golang/go/issues/36758
[3]https://pkg.go.dev/license-policy
[4]https://opensource.org/licenses/alphabetical


Nate Finch

unread,
Jan 31, 2020, 9:23:24 PM1/31/20
to golang-dev
I share Dan's concern, and I'd like to hear more about why this change is being made. Godoc.org has been good to the community for a long time, and has not been concerned with what license the repo uses.

In my opinion there should be a way to tell pkg.go.dev that it's ok to display information from a repo without having to change the license on your code generally.

Perhaps if repo owners could add a marker file that specifically gives pkg.go.dev rights to display their package, then it can be an escape hatch for people who don't want to generally relicense, but also want their code on the site.

Dan Kortschak

unread,
Jan 31, 2020, 9:30:06 PM1/31/20
to golang-dev
This was an idea I was considering as well; perhaps a micro-license
that allows pkg.go.dev to render the API and documentation in
perpetuity. This is essentially what the marker file would amount to
legally.

Dan

Elias Naur

unread,
Feb 1, 2020, 4:37:09 AM2/1/20
to golang-dev
FWIW, I suggested a marker on (the now closed) #36840: https://github.com/golang/go/issues/36840#issuecomment-579402752

-- elias

Dan Kortschak

unread,
Feb 1, 2020, 4:52:37 AM2/1/20
to Elias Naur, golang-dev
So it look like they see that as a non-starter.

The impact of this decision will be to put a significant number of
packages into the dark where their documentation is not accessible,
either because the licensing is not correctly recognised or not
Correct™ in the sense that it doesn't conform to the limited set of
licenses that pkg.go.dev wants people to use. This is a pretty tone
dead approach.

Dan
> --
> You received this message because you are subscribed to the Google
> Groups "golang-dev" group.
> To unsubscribe from this group and stop receiving emails from it,
> send an email to golang-dev+...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/golang-dev/349f4b9b-4c17-4acc-8b59-f8cb224219b9%40googlegroups.com
> .


Jan Mercl

unread,
Feb 1, 2020, 4:54:06 AM2/1/20
to Dan Kortschak, golang-dev
On Fri, Jan 31, 2020 at 10:46 PM Dan Kortschak <d...@kortschak.io> wrote:
>
> In the "Next steps for pkg.go.dev" blog post[1] there is an indication
> that godoc.org will be redirected to pkg.go.dev.
>
> I want to say that I think this is badly misguided given the current
> status of go.dev's ability to correctly (notwithstanding [2] being
> closed) identify licenses (and the restrictive set of licenses that it
> will even try to identify) and the consequences for packages that are
> not identified as having a "correct" license.

Thanks Dan for bringing this to my attention. I have now read the blog
post and I'm concerned about the impact of the intent to redirect
godoc.org to pkg.go.dev IMO will have on the contributors community.

Checking whois godoc.org says"Registrant Organization: Google LLC",
while whois go.dev returns "Registrant Name: REDACTED FOR PRIVACY".

Why has Google no problem with package licenses at godoc.org? I don't
know who actually owns go.dev and why is that information not public,
but let me assume it's Google as well. If so, why the same company can
publish documentation for any publicly visible package whatsoever,
regardless of license, at one of its sites and cannot do the same at
another of its sites?

I have nothing against pkg.go.dev and I appreciate the efforts that
went into creating it. But I am not happy with the output. Nearly
every of my (tens) of Go packages that I have published in the last
ten years do not show the README nor the documentation. I'm not
offended, but neither I think treating me and (probably) thousands of
other members of the contributors community this way is fair and/or
justified.

Let me try to be constructive.

1) godoc.org does not understand modules/versions they said. I see no
reason why this cannot be fixed. I am willing to participate in a
group of volunteers, if there are any others, and spent some of my
precious free time, for free, to try to make that happen. I believe
it's feasible, even when started from scratch. Or actually/better
starting from the open code for `go doc` or `godoc` (I'll never learn
the difference) even when I have no idea ATM where it now resides.

And now let me be somehow less constructive.

2) Even though pkg.go.dev has more features that godoc.org, unless it
is 100% backwards feature compatible with godoc.org, the idea of
killing godoc.org by redirecting it to pkg.go.dev is IMO not a good
idea. Throwing away uncountable hours of free work made by the
community in the past 10 years, by hiding the API documentation and
the README at the redirected site, is a hard to accept move for me -
and perhaps I'm not the only one. Yet, the blog clearly communicates
that's the plan quote "later this year" end quote.

Please reconsider, thanks.

PS: Is someone feeling like creating an online voting place or
something like a petition that folks can use? It perhaps could provide
Google an opportunity to show how much it _really_ cares about the Go
community - as expressed _verbally_ at other places and occasions.

gugl.z...@gmail.com

unread,
Feb 1, 2020, 2:29:52 PM2/1/20
to golang-dev
pkg.go.dev has several faults when compared to godoc.org:

1. It's a heavier and slower website. The page for the package
    github.com/gorilla/mux on godoc.org required seven HTTP requests and
    it loads in about 1.3 seconds on my computer and Internet
    connection. The page for the same package on pkg.go.dev requires 19
    requests, and loads in about 2.1 seconds.

2.  pkg.go.dev is a closed-source product, while godoc.org is
    open-source software.  I cannot inspect the source code of
    pkg.go.dev, which is something I and other developers are concerned
    about.  You did the same with proxy.golang.org, and honestly this
    pattern is very disturbing.

Both problems can be solved, but until they aren't, I don't think it's
wise to redirect everything to the new website.

Liam

unread,
Feb 1, 2020, 3:16:10 PM2/1/20
to golang-dev
Dan, thanks for continuing to raise this.

I get the impression that there's an unstated (?) corporate rationale motivating the go.dev license policy and shutdown of godoc.org. I'd guess that stems in part from the Oracle lawsuit re Java APIs.

If so, we've reached a point where the interests of the Go project and those of Alphabet/Google conflict, and it's time to separate Go governance from Google.

Kubernetes also originated in Google, but has been governed outside Google since v1.0. What are the reasons not do the same for Go?

Ulderico Cirello

unread,
Feb 1, 2020, 7:41:16 PM2/1/20
to golang-dev
I want to share my agreement and express it with different terms.

I see two class of problems.

In one, I believe there's a change of policy that is frankly not acceptable and may risk leading to unnecessary forks of effort in the community. pkg.go.dev is closed source and its development has been done in private taking very little input from the community (not because you are not responsive - Andy Bons was prompt in his response to me; but because you are making decisions despite of our expectations).

The other one has to do with functionality: pkg.go.dev does not have feature parity with godoc.org.

With all that said, I have two things more to say.

One - If Google plans to keep control of pkg.go.dev and developing it in private; I am willing to start a movement with people of the community to keep a publicly version of godoc.org available.

Two - regarding the many repositories that are not visible due to license check issues, the solution is quite simple: just adopt https://spdx.org/licenses/ lines in license files. This response by Julie Qiu (https://github.com/golang/go/issues/36840#issuecomment-579816850) seems tone-deaf to me and very discouraging to the necessary debate of a solution that works for everyone in the Go ecosystem. 


Thanks,
Ulderico


--
You received this message because you are subscribed to the Google Groups "golang-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to golang-dev+...@googlegroups.com.

Akhil Indurti

unread,
Feb 1, 2020, 9:37:32 PM2/1/20
to golang-dev
The motivation behind the license policy seems opaque. The comment on #36840 states they want to "encourage package authors to fix their licenses for the broader open source ecosystem," but a "fixed" license suggests that projects whose licenses are approved are recommended for usage over others. However, companies often have varying restrictions on licenses for third-party software they'll adopt. Go for example won't pull in an MIT-licensed dependency unless its maintainer signs a Google CLA, so even then, the list of approved licenses doesn't really help with the goal of "evaluating dependencies". At the end of the day, the programmer, or at worst, a legal team will still have to vet the dependency.

Manlio Perillo

unread,
Feb 2, 2020, 1:17:11 PM2/2/20
to golang-dev
On Saturday, February 1, 2020 at 8:29:52 PM UTC+1, gugl.z...@gmail.com wrote:
pkg.go.dev has several faults when compared to godoc.org:


> [...] 
2.  pkg.go.dev is a closed-source product, while godoc.org is
    open-source software.  I cannot inspect the source code of
    pkg.go.dev, which is something I and other developers are concerned
    about.  You did the same with proxy.golang.org, and honestly this
    pattern is very disturbing.


I would prefer for pkg.go.dev and proxy.golang.org to be open source, but why do you say that the pattern is disturbing?
The two applications runs on Google servers, not on my or your computer.
And even if the source code is released, there is no guarantee that the applications on the Google servers will use the open source code.


Manlio

gugl.z...@gmail.com

unread,
Feb 2, 2020, 4:45:48 PM2/2/20
to golang-dev
> I would prefer for pkg.go.dev and proxy.golang.org to be open source,
> but why do you say that the pattern is disturbing?
> The two applications runs on Google servers, not on my or your
> computer.
> And even if the source code is released, there is no guarantee that
> the applications on the Google servers will use the open source code.

Thoughts about that in no particular order:

1.  Having the source code is *always* preferred to not having it.
    That should pretty much be an axiom by now.
2.  There is no reason why they should not be public.  And none were
    really provided, which is a failure of communication from the
    company's side.
3.  I can host my own version of godoc.org or golang.org.  Why not
    proxy.golang.org or go.dev?  I know about Athens, that's not the
    point.
4.  Now we know that it wasn't just a one-off thing with
    proxy.golang.org, and that the Go people from the company want to go
    on making non-free software for the Go ecosystem.  Yes, those don't
    run on my computer *yet,* but what's stopping them now from making
    a non-free superoptimising compiler or static analysis tool?
5.  We can't see those services' source code, which can mean that the
    authors have something to hide.  Technical flaws?  Security holes?
    License mismatches?  Either way, this atmosphere of not trusting the
    community is harmful.
6.  The argument about “no guarantee that the Goog runs the same code”
    applies to pretty much any FOSS service.  I would like to live in
    a world where we could cryptographically verify that the server
    we're talking to runs the correct software, but AFAIK there is no
    way to do so now.

That's just off the top of my head.  Really, if I wanted my programming
language to be tightly controlled by a shady corporate entity that
dislikes and distrusts FOSS, I would just learn a certain programming
language with a coffee-related name.  I don't want Go, the language
I love, to become that.

Sam Whited

unread,
Feb 2, 2020, 4:58:06 PM2/2/20
to rprichard via golang-dev
Adding to this that the Go team is now taking data about package
downloads and fetching away from package owners and handing it to
Google, this becomes a pattern of bad behavior. I'm sure the Go team has
the best of intentions, but their intentions are meaningless when their
actions are so harmful to the community. I believe them when they say
that Google isn't doing anything with the data they're gathering on Go
packages, but that doesn't mean some new head of the Go team or some new
higher-up at Google won't try to do something with it later.

This pattern of the Go team building closed source servers and
integrating Google run services with little to no input from the
community (and general dismissive comments about the communities
concerns) is absolutely unacceptable and must stop.


—Sam
> --
> You received this message because you are subscribed to the Google
> Groups "golang-dev" group. To unsubscribe from this group and stop
> receiving emails from it, send an email to golang-
> dev+uns...@googlegroups.com. To view this discussion on the web
> visit
> https://groups.google.com/d/msgid/golang-dev/f6336f98-d8ff-417b-af3b-0e38bddce797%40googlegroups.com
> <https://groups.google.com/d/msgid/golang-dev/f6336f98-d8ff-417b-af3b-0e38bddce797%40googlegroups.com?utm_medium=email&utm_source=footer>
> .

--
Sam Whited

Robert Engels

unread,
Feb 2, 2020, 5:42:00 PM2/2/20
to Sam Whited, rprichard via golang-dev
Wow. The victim/outrage culture is reaching new heights - actively insulting the good work and character of those whose output you are then using.

Google services billions of users - none have the source code to these services.

As the Go team pointed out - you always have the ability to fork. Go is open source.

Attempting to dictate how a private company does their business is silly - as long as they comply with applicable laws.

I would ask that you be more respectful in your commentary.

> On Feb 2, 2020, at 3:58 PM, Sam Whited <s...@samwhited.com> wrote:
>
> Adding to this that the Go team is now taking data about package
> To unsubscribe from this group and stop receiving emails from it, send an email to golang-dev+...@googlegroups.com.
> To view this discussion on the web visit https://groups.google.com/d/msgid/golang-dev/02112c90-6b6a-4c0c-b51c-5c3c897df146%40www.fastmail.com.

hes...@google.com

unread,
Feb 2, 2020, 7:20:52 PM2/2/20
to golang-dev
Hi. I usually don't say much on these forums, but I disagree strongly with a lot of what's been said about proxy.golang.org and I wanted to share my perspective. I worked on proxy.golang.org (and sum, and index) and was involved in the design from the beginning. That said, everything here is my opinion, and other members of the Go team may disagree. I'm only speaking for myself and any mistakes are my own.


> 1. Having the source code is *always* preferred to not having it. That 
> should pretty much be an axiom by now. 
I agree with this.

> 2. There is no reason why they should not be public.
There is. The mirror and checksum database are critical services depended on by the majority of Go developers on a daily basis. They absolutely must be correct, reliable, and secure. Open sourcing, while desirable, is not an actual requirement. My opinion was and is that we could provide a better service to Go developers by using proprietary Google technologies. Looking at the source, there is very little that isn't entangled with those technologies. We could have tried to split the pieces up, but that would have created significant complexity in the project, and complexity is the enemy of reliability.

> And none were 
> really provided, which is a failure of communication from the 
> company's side.
While we could have been more open about this (and I'm trying to be now) I don't really agree that the world is owed an in-depth explanation of every decision I make.

> 3. I can host my own version of godoc.org or golang.org. Why not 
proxy.golang.org or go.dev? I know about Athens, that's not the 
> point.
As I've said above, you don't have access to the things that proxy.golang.org is built on. Unfortunate but in my opinion the right tradeoff.

> 4. Now we know that it wasn't just a one-off thing with 
proxy.golang.org, and that the Go people from the company want to 
> go on making non-free software for the Go ecosystem. Yes, those 
> don't run on my computer *yet,* but what's stopping them now from 
> making a non-free superoptimising compiler or static analysis tool? 
I don't think I have anything useful to say here.

> 5. We can't see those services' source code, which can mean that the 
> authors have something to hide.
True. I don't think I have anything to hide, but I doubt you'll find that convincing.

> Technical flaws? Security holes? 
> License mismatches? Either way, this atmosphere of not trusting the 
> community is harmful. 
I don't think any of the decisions made about whether to open source proxy.golang.org had anything whatsoever to do with trusting or not trusting the community.

> 6. The argument about “no guarantee that the Goog runs the same code” 
> applies to pretty much any FOSS service. I would like to live in a 
> world where we could cryptographically verify that the server we're 
> talking to runs the correct software, but AFAIK there is no way to 
> do so now. 
Yes. However, you can verify that the output of proxy.golang.org matches what the origin has. If you find that something suspicious has happened, you can use sum.golang.org to irrefutably prove it. If you don't trust Google, I encourage you to do so and publish the results.

On Sunday, February 2, 2020 at 4:58:06 PM UTC-5 Sam Whited wrote:
Adding to this that the Go team is now taking data about package
downloads and fetching away from package owners and handing it to
Google, this becomes a pattern of bad behavior.
I am not a lawyer and I cannot offer an authoritative interpretation of our privacy policy. However, at the end of https://proxy.golang.org/privacy is the statement:
"We intend to aggregate and anonymize usage metrics to measure popularity for Go modules and share this popularity data with the Go community."
In my opinion, that is the only allowed use of download metrics other than monitoring and debugging. I believe Google services should get the same version of that data that everybody else gets.

I hope this is helpful. I don't think I have much more to say than this, so I probably won't respond any further. I would ask that you (all) remember that these are projects worked on by actual people who are, in my experience, generally just trying to make Go the best it can be.

Liam Breck

unread,
Feb 2, 2020, 7:59:41 PM2/2/20
to hes...@google.com, golang-dev
This thread had gone off-topic. The issue Dan raised is that pkg.go.dev is not ready to replace godoc.org.


thepud...@gmail.com

unread,
Feb 2, 2020, 8:39:26 PM2/2/20
to golang-dev
 > "The issue Dan raised is that pkg.go.dev is not ready to replace godoc.org."

Hello fellow gophers,

FWIW, as far as I have followed, it seems everyone is in agreement that pkg.go.dev is not ready to replace godoc.org, including it seems that the very people working on pkg.go.dev are also saying pkg.go.dev is not ready to replace godoc.org.

The recent blog post (https://blog.golang.org/pkg.go.dev-2020) that was published a couple of days ago said (emphasis added):

  "__later this year__ we are planning to redirect traffic from godoc.org to the corresponding page on pkg.go.dev"

and the next paragraph begins:

  "Your feedback will inform our transition plan"

at which point the blog post then asks for feedback via a variety of channels.

As far as I have followed, there have been a few statements that a rough, early version was launched so that people could comment on something concrete, but that it is not finished yet.

The mid-November blog post also said (https://blog.golang.org/go.dev):

  "Today’s launch is our minimum viable product for go.dev, so we can share what we’ve built to help the community and get feedback."

In any event, it seems the people working on go.dev agree that some work remains, including on the license detection pieces. About a month ago, there was this comment (https://github.com/golang/go/issues/33654#issuecomment-569711045):

   "We plan to respond to all issues in the new year, and to continue working with 
    our legal team to resolve any license issues. 
    Thanks for being patient with us as we work through them!"
 

   "Our current volume of special case issues is low enough that we plan to keep the manual 
    review process for now (rather than building an automatic solution). 
    We can always revisit this strategy as other specific issues come up."

That sounds like they currently have a manual strategy that seems to be working, but also seems to say that they are open to improvements.

I don't have any particular special knowledge here and I am not on the Go team, so apologies in advance if I've said something off-base.

Dan, from a quick look over the different license issues on the issue tracker, it looks like you have put a fair amount of effort into helping work through the licensing problems, so I wanted to say a quick thanks for that from one gopher to another.

Best,
thepudds

Ulderico Cirello

unread,
Feb 2, 2020, 8:57:38 PM2/2/20
to thepud...@gmail.com, golang-dev
Unfortunately, there's more than meet the eyes here. And although it might seem we are veering off the OP, the fundamental question is not so much whether pkg.go.dev is ready to replace godoc.org (obviously it is not), but whether this transition should happen at all.

One way or another you could run godoc or gddo locally: their source code was available and there was some semblance that Google was playing fair by using some aligned version of the software both inside and outside. 

Or to reframe the problem. Right now, if I go and open a CL to change godoc - everyone has the opportunity to chime in and participate. 


pkg.go.dev on the other hand is being developed in the private. We cannot contribute, we cannot review how it's being done, we cannot contribute to its license detection system, we cannot do any of the things we could do to participate. 


More, and at least for me disturbing, Google is using go.dev as a place to publish and foster Google cloud software. Browse go.dev and you're going to see that they prominently added their products ahead of the rest of the ecosystem. 

Thus, not only pkg.go.dev is not ready to replace godoc.org; but go.dev as a domain is a closed service offered by Google to which we all are being extracted away. 


The documentation website is one of the components of go.dev and we, as members of the ecosystem, should be watchful about what's going on elsewhere. 


And finally, Googlers claimed that go.dev is a MVP. Which by the way is problem of their own creation. Godoc.org worked fine and golang.org worked great too. The Go ecosystem does NOT need go.dev. At least not the one Google and the Go team is building in private alienating the ecosystem by constraining the communications to write only channels (go-discovery-feedback) and stonewall (like the response from a Googler in this thread who stated that proxy.golang is too tied to Google infra as a fait accompli). 

Sad days that I would hope to never see in Go. Like I told Andy Bons, the Go team is breaking my heart. 



--
You received this message because you are subscribed to the Google Groups "golang-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to golang-dev+...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/golang-dev/88d6aa3e-3fdb-4fa7-9e4b-12b0d7309789%40googlegroups.com.

minux

unread,
Feb 2, 2020, 9:50:09 PM2/2/20
to golang-dev
On Sun, Feb 2, 2020 at 8:57 PM Ulderico Cirello <ulderi...@gmail.com> wrote:
> Unfortunately, there's more than meet the eyes here. And although it might seem we are veering off the OP, the fundamental question is not so much whether pkg.go.dev is ready to replace godoc.org (obviously it is not), but whether this transition should happen at all.
>
> One way or another you could run godoc or gddo locally: their source code was available and there was some semblance that Google was playing fair by using some aligned version of the software both inside and outside.
>
> Or to reframe the problem. Right now, if I go and open a CL to change godoc - everyone has the opportunity to chime in and participate.
>
>
> pkg.go.dev on the other hand is being developed in the private. We cannot contribute, we cannot review how it's being done, we cannot contribute to its license detection system, we cannot do any of the things we could do to participate.

pkg.go.dev uses https://github.com/google/licensecheck to detect
licenses, and you are certainly welcome contribute to it.
(It puzzles me that a lot of people seem to indicate they're willing
to fork or otherwise enhance related projects, but no one seems to
want to actually contribute and fix the exact issue that this thread
talks about.)

pkg.go.dev is closed source, but the blog post
(https://blog.golang.org/pkg.go.dev-2020) also explicitly mentioned
that the team is planning to develop an open-source equivalent that
you can host on your own servers.

Ulderico Cirello

unread,
Feb 2, 2020, 9:58:26 PM2/2/20
to minux, golang-dev
Thanks for the response Minux.

Equivalent is not the same. 

A library that a project uses is not the project itself. 

As much as I can contribute, I do. And I did in other areas (playground and documentation examples). 

And again, making it a new closed source service that barely replaces an open source one that works is a solution for a problem that didn't exist first off. 

In any case, I might be beating a dead horse by now. I apologize. 

--
You received this message because you are subscribed to the Google Groups "golang-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to golang-dev+...@googlegroups.com.

Akhil Indurti

unread,
Feb 2, 2020, 11:17:44 PM2/2/20
to golang-dev
I thought this discussion was about documentation not being displayed,
but it seems to have trailed off into FUD. The whole point of a time gap
till later this year is to garner feedback.

That being said, any project that is closed-source must make the extra
effort to clarify its intent. Such information should be public and visible
on the documentation, not on an issue tracker. However, given the
relatively recent launch of the site, I'm willing to give them the benefit
of the doubt.

On Friday, January 31, 2020 at 4:46:14 PM UTC-5, Dan Kortschak wrote:

Liam

unread,
Feb 3, 2020, 4:16:58 AM2/3/20
to golang-dev
Besides the (temporary) problem of license recognition and (permanent?) limitations on accepted licenses, there is a bizarre belief that anyone can publish another author's work under arbitrary terms, described in

roger peppe

unread,
Feb 3, 2020, 5:25:30 AM2/3/20
to Liam, golang-dev
About losing package visibility: I agree that it's a problem when packages that have historically been available in godoc.org are no longer available on the new (and much better, IMHO!) pkg.go.dev site.

Two possible workarounds for this:

 - only redirect from godoc.org to pkg.go.dev when the package/module is available on the new site.
 - grandfather all the packages on godoc.org, so that all the old content is still visible. New content would still need a compatible license (ISTM if you're publishing new content, you're probably in a position to change to using an accepted license).


--
You received this message because you are subscribed to the Google Groups "golang-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to golang-dev+...@googlegroups.com.

Ian Davis

unread,
Feb 3, 2020, 6:16:32 AM2/3/20
to golan...@googlegroups.com
On Mon, 3 Feb 2020, at 2:49 AM, minux wrote:
>
> pkg.go.dev uses https://github.com/google/licensecheck to detect
> licenses, and you are certainly welcome contribute to it.
> (It puzzles me that a lot of people seem to indicate they're willing
> to fork or otherwise enhance related projects, but no one seems to
> want to actually contribute and fix the exact issue that this thread
> talks about.)
>

I think that is a little unfair. Dan Kortschak, for example, raised relevant issues[1] but the response is that a redesign of the library is needed. Without sharing any thinking behind that redesign then it's difficult to guess what would be accepted as a fix.

[1] https://github.com/google/licensecheck/issues/6

-- Ian

Russ Cox

unread,
Feb 3, 2020, 5:17:07 PM2/3/20
to golang-dev, Dan Kortschak
Hi all,

Julie's blog post was first and foremost a request for feedback, so
thanks to Dan and everyone else for sharing your thoughts on this
thread. As I hope the post made clear, the discovery site is not done
yet, we know some ways it needs to be made better, and more
importantly we know we are probably blind to others, hence the call
for feedback. Please bear with us while we digest that feedback and
work on fixes. Please also keep in mind that, as I've noted before, we
are all pulling in the same direction. We all want what's best for Go.

I also want to say thanks for keeping us honest about all of this. The
last year has been hectic and we haven't been completely clear about
some of the reasons for recent work. (I take full responsibility for
this and am definitely not faulting anyone else on the Go team!)
It's very helpful when people call us out on that. The high standards
you all demand truly are one of my favorite things about open source.

This mail is a bit long but I hope it addresses most of the discussion
over the weekend. If you feel I'm missing anything, please reply and
I'm happy to continue the discussion.

# Background about go.dev

One thing we've come to understand over the past year or so is that
many in the next wave of Go adopters want a “one-stop” web site full
of resources about Go, including how to get started, who is using it
already, links to learning resources, package docs, and so on. The new
go.dev is meant to be that one-stop site. In the blog announcing it,
we called it “a new hub for Go developers.”

# Why is go.dev different from golang.org?

Personally, I thought that separating the two would allow go.dev to be
more inclusive of community content. Historically, golang.org is the
place where the Go project speaks authoritatively about Go: it has the
language spec, standard library docs, official downloads, and so on.
It has always been important to us not to dilute that with all the
other Go content in the world. The new site seemed to be an
opportunity to create a more inclusive place for other resources,
hence a second site.

# Background about godoc.org

Gary Burd created and ran godoc.org starting in late 2012. It was and
remains both a brilliant idea and clearly a valuable service to the Go
community. After a year or so, Gary decided he didn't want to run and
pay for the servers anymore, so he gave it to Go project to adopt. We
were happy to do that, to make sure this resource remained available
to all Go users. We said it back in 2014 when we adopted godoc.org and
I will say it again: I am incredibly grateful to Gary for what he
created.

# Why is there a new package docs site at all? Why not update
godoc.org in place?

With the introduction of modules and the notion of multiple versions
of a package, we knew we had to update the godoc.org experience. After
a hard look, it seemed worth starting anew, especially since the
godoc.org server design, with its single-VM database, had been
starting to show its age. In addition to the modules work, there are
other things we're addressing, such as accessibility and overall
scalability of the service.

As a side note, there's almost nothing in the Go distribution that has
survived eight years without being redone. The compiler, the
assembler, the linker, the go command itself, most of the standard
library: all of them have been massively overhauled one or more times
since the start of Go. That's how we take what we learn and make
things better.

This kind of rewrite always involves a transition period in which the
old version is still the workhorse that most people use and the new
version has a new name for early adopters to test and find bugs in.

# Why is the new package docs site on pkg.go.dev?

Docs for all the packages in the entire ecosystem are exactly the kind
of community-generated content that go.dev is meant to help find, so
pkg.go.dev seemed like a good name. Especially since go.dev has a much
broader scope than godoc.org, it makes sense to take the opportunity
to fold it in and reduce the number of sites a typical user has to be
aware of (once the transition is complete).

# Why are you talking about redirecting godoc.org to pkg.go.dev
when pkg.go.dev is still so broken?

Bluntly, so that you will help us understand what's broken, so we can
fix it before the redirect happens. We know about some things but
would not be surprised to find there are things we're completely
unaware of. Better to find out earlier rather than later. Again, the
blog post was first and foremost a request for feedback about what
needs to happen before we actually do the redirect.

# Why does pkg.go.dev require a detected license to show docs?
Why doesn't godoc.org?

The teams working on the proxy and on pkg.go.dev have spent a lot of
time talking to Google's lawyers about what we can and can't do with
Go source code downloaded from the internet. The rule we've been given
to follow is that serving a pretty HTML version of the docs is
displaying a modified version of the original, and we can only do that
if there's a recognized known-good license that gives us that
permission.

When we adopted godoc.org from Gary Burd back in 2014, it did not
occur to any of us to put it through that kind of review. If we had,
maybe the community would have gone through this licensing pain
earlier. For now we are focusing on making changes to pkg.go.dev
rather than correcting past mistakes on godoc.org. (At this point,
more scrutiny of what godoc.org does is not likely to have an outcome
that anyone likes.)

# What fraction of popular packages don't display on pkg.go.dev?

Right now it looks like pkg.go.dev sees 1,200 modules imported by at
least 100 other modules. Of those, it looks like 82 are flagged as not
redistributable, so that we can't show their docs. That's under 7%,
and we're working to understand that better. If any of those are
mistakes on our end, we'll fix them.

Another thing that was suggested that I think is a great idea is to
change the “no docs available” page to have a command-line to bring up
the docs in your own local godoc command.

[Aside: Speaking generally (not just about Go), you would be surprised
at the number of software packages that have a declared "license type"
in some kind of metadata (GitHub meta, package.json, SPDX codes, etc)
but no actual license text. This makes the license impossible to comply
with! For example, the MIT license requires that "The above copyright
notice and this permission notice shall be included in all copies or
substantial portions of the Software." But if there's no such notice
to include, only an "// SPDX-License-Identifier: MIT" comment, there's
literally no way to comply. It can be a real mess. If you have never before
encountered the differences between how programmers see the world
and how lawyers do, let me recommend "What Colour are your bits?"]

# Why hasn't pkg.go.dev been open sourced?

There's no conspiracy here. The original plan was to open source it,
but open sourcing puts pressure on the code base to be reusable in
other contexts. Right now the code is only written for one context:
the global pkg.go.dev site.

I helped edit the blog post and was responsible for the "Will
pkg.go.dev be open-sourced so I can run it at work for my private
code?" heading. I apologize for the offense I caused so many of you.
I did not mean to imply that there were no other reasons to run a doc server,
only to highlight as an example what I thought was the most common
reason Go developers would want the code.

In much the same way that we published an open source reference
implementation of the proxy that was not the global proxy site, I
still hope that we will publish an open source reference
implementation of a module-aware package docs site that need not be
exactly the global site. An implementation that doesn't have to hit
the same global scale can be much simpler to run—whether it's at work
with private code or on your laptop with completely open-source code.
I also hope the same server can serve index queries that make tools
like goimports and gopls faster. In contrast pkg.go.dev probably can't
serve such queries, at least not the same way, for both scaling
concerns and privacy concerns.

So the reason in the blog post is the real reason: the current code is
not what you probably should run, whether at work or on your laptop in
offline mode, and I thought we could deliver a much better answer for
that.

But I very much hear all of you who want to see the code that is
running on the site and possibly contribute to it, whether it makes sense
to run in other contexts or not. I am going to look into that.

Again, thanks for calling us out when we're not communicating clearly
with all of you. It really helps. I hope this mail helps too, and if
not, let me know.

Best,
Russ

Dan Kortschak

unread,
Feb 3, 2020, 7:37:46 PM2/3/20
to Russ Cox, golang-dev
Thanks Russ,

The issues that I am concerned about are; 1. the going dark of
unacceptably licensed packages, and 2. the limited definition of what
is acceptable in licensing of packages. The other aspects of this
discussion are interesting, but weren't the primary focus of my
original post.

The explanation for the need for care in licensing makes sense (and
brings to mind the old induction-based joke about lawyers in benthic
environments). However, I continue to have issues about how this will
work mechanically. From posts at issues, it appears that at least for
the short term this will be papered over by human vetting; I don't see
this changing in the longer term given that licensing is complicated
(see more below about issues that we skirt in licensing Gonum source).

I think a reasonable approach would be to have first stage license
detection performed mechanically (with an improved detection algorithm
that makes use of SPDX license descriptions - see licensecheck#6), and
a complete set of licenses, with a fallback to render a note explaining
why it is not rendered as documentation, a link to the original source
host, and a mechanism that allows it to be flagged as actually licensed
(without a need to file an issue) that is acted on outside go.dev
release cycles.

At the moment I don't believe that even the cases that appear to be
correctly marked with licenses are correctly marked (this is
golang.org/issue/36953), and I am not convinced that it is actually
possible to correctly mark licenses without human intervention. If the
license notes are purely there as a brief note to explain why go.dev
has the opinion it does, then it's possibly reasonable, but I doubt
that most people would read them that way. Maybe that could be made
more clear.

*(note from above) Part of the ethos for the Gonum project is proper
ascription of thanks for work. In this we are very careful with
licensing (we have a THIRD_PARTY_LICENSES directory that includes
licenses for code that we have worked from in porting routines to Go -
whether we need to do this is not entirely clear, the FSF would say
yes, but many Gophers appear not to agree with this; for example, the
Go project itself does not include third party license information for
a set of code that comes from other sources and have historically come
through to the projects source from e.g. plan9/Inferno). We do this to
1. thank the original authors and acknowledge their contributions, but
also to 2. allow our users to decide for themselves whether the
licensing of the code needs to include these third parties. go.dev
ignores these. I'm not sure this problem is solvable.

Dan

Dan Kortschak

unread,
Feb 3, 2020, 7:54:07 PM2/3/20
to Russ Cox, golang-dev
As an addendum to this, there is an argument that the pkg.go.dev team
has given me the rights to see the source of pkg.go.dev as a
consequence of displaying this package[1] and the first clause of the
license which grants me "access to the licensee's hardware".

The license text quoted here:
```
Copyright 2019 Dan Kortschak

Redistribution and use in source and binary forms, with or without
modification, are permitted provided that the following conditions are
met:

1. Redistributions of source code must retain the above copyright
notice, this list of conditions and the following disclaimer, and grant
access to the licensee's hardware.

2. Redistributions in binary form must reproduce the above copyright
notice, this list of conditions and the following disclaimer in the
documentation and/or other materials provided with the distribution.

3. Neither the name of the copyright holder nor the names of its
contributors may be used to endorse or promote products derived from
this software without specific prior written permission.

THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS
IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED
TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A
PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT
HOLDER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
(INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
```

This package was written with the intention of illustrating the danger
of lawyering a machinā.

Dan

[1]https://pkg.go.dev/github.com/kortschak/unlicensable?tab=doc
> --
> You received this message because you are subscribed to the Google
> Groups "golang-dev" group.
> To unsubscribe from this group and stop receiving emails from it,
> send an email to golang-dev+...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/golang-dev/ec0e25723a2c86fb045347cc98e3f5b4e2f209af.camel%40kortschak.io
> .


Ian Lance Taylor

unread,
Feb 3, 2020, 8:12:28 PM2/3/20
to Dan Kortschak, Russ Cox, golang-dev
On Mon, Feb 3, 2020 at 4:37 PM Dan Kortschak <d...@kortschak.io> wrote:
>
> *(note from above) Part of the ethos for the Gonum project is proper
> ascription of thanks for work. In this we are very careful with
> licensing (we have a THIRD_PARTY_LICENSES directory that includes
> licenses for code that we have worked from in porting routines to Go -
> whether we need to do this is not entirely clear, the FSF would say
> yes, but many Gophers appear not to agree with this; for example, the
> Go project itself does not include third party license information for
> a set of code that comes from other sources and have historically come
> through to the projects source from e.g. plan9/Inferno). We do this to
> 1. thank the original authors and acknowledge their contributions, but
> also to 2. allow our users to decide for themselves whether the
> licensing of the code needs to include these third parties. go.dev
> ignores these. I'm not sure this problem is solvable.

The Go project does of course include this additional license
information in the files themselves (e.g.,
https://golang.org/src/cmd/compile/internal/gc/gsubr.go). Is your
concern here that the information does not appear in the top level
LICENSE file?

Ian

Dan Kortschak

unread,
Feb 3, 2020, 8:17:14 PM2/3/20
to Ian Lance Taylor, Russ Cox, golang-dev
Thanks, Ian. Yes, that's the issue (or at least that it's not as
obviously licensed as it could be). We've discussed this before and I
didn't press it, but since we're here I thought I'd bring it up again.

It's based on a broader problem of whether people actually include
those licenses in their software. I doubt most Go users are even aware
that the Lucent and Vita Nuova licenses exist, let alone making sure
that they are included in compiled products (I'd say that people don't
include the Go license often in compiled products).

Dan
> --
> You received this message because you are subscribed to the Google
> Groups "golang-dev" group.
> To unsubscribe from this group and stop receiving emails from it,
> send an email to golang-dev+...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/golang-dev/CAOyqgcXwC%3DuXVCQzo85cL6c22oodRZLRhfo%2BFPR1EJwLX6UxFA%40mail.gmail.com
> .


Ian Lance Taylor

unread,
Feb 3, 2020, 8:53:25 PM2/3/20
to Dan Kortschak, Russ Cox, golang-dev
On Mon, Feb 3, 2020 at 5:17 PM Dan Kortschak <d...@kortschak.io> wrote:
>
> Thanks, Ian. Yes, that's the issue (or at least that it's not as
> obviously licensed as it could be). We've discussed this before and I
> didn't press it, but since we're here I thought I'd bring it up again.
>
> It's based on a broader problem of whether people actually include
> those licenses in their software. I doubt most Go users are even aware
> that the Lucent and Vita Nuova licenses exist, let alone making sure
> that they are included in compiled products (I'd say that people don't
> include the Go license often in compiled products).

It's worth noting that the Lucent and Vita Nuova copyrights only exist
on Go tools like the compiler and linker, not on any part of the Go
library. There is no requirement or expectation that Go users include
those licenses with arbitrary programs written in Go. The only
possible question is how they should be handled in distributions of
the Go tools.

Ian

Dan Kortschak

unread,
Feb 3, 2020, 9:08:06 PM2/3/20
to Ian Lance Taylor, Russ Cox, golang-dev
They are indeed mainly in the compiler, but also exist in the runtime
for memmove and memclr (also vlop and vlrt for some architectures).

Dan