plans to redirect godoc.org to go.dev

5786 views
Skip to first unread message

Dan Kortschak

unread,
Jan 31, 2020, 4:46:14 PM1/31/20
to golang-dev
In the "Next steps for pkg.go.dev" blog post[1] there is an indication
that godoc.org will be redirected to pkg.go.dev.

I want to say that I think this is badly misguided given the current
status of go.dev's ability to correctly (notwithstanding [2] being
closed) identify licenses (and the restrictive set of licenses that it
will even try to identify) and the consequences for packages that are
not identified as having a "correct" license.

I don't use licenses like WTFPL or CC0 (though some gonum packages are
dual licensed BSD3/CC0-1.0-PDD), but some people in the community do
use these licenses and due to [3], if godoc.org redirects to
pkg.go.dev, these people will no longer have a commonly known public-
facing documentation outlet. It should be noted that there are many
license that are listed as OSI-approved, but which don't fit into the
narrow list of what is acceptable to pkg.go.dev.

This heavy handed approach to what is happening is really disappointing
to me.

Dan

[1]https://blog.golang.org/pkg.go.dev-2020
[2]https://github.com/golang/go/issues/36758
[3]https://pkg.go.dev/license-policy
[4]https://opensource.org/licenses/alphabetical


Nate Finch

unread,
Jan 31, 2020, 9:23:24 PM1/31/20
to golang-dev
I share Dan's concern, and I'd like to hear more about why this change is being made. Godoc.org has been good to the community for a long time, and has not been concerned with what license the repo uses.

In my opinion there should be a way to tell pkg.go.dev that it's ok to display information from a repo without having to change the license on your code generally.

Perhaps if repo owners could add a marker file that specifically gives pkg.go.dev rights to display their package, then it can be an escape hatch for people who don't want to generally relicense, but also want their code on the site.

Dan Kortschak

unread,
Jan 31, 2020, 9:30:06 PM1/31/20
to golang-dev
This was an idea I was considering as well; perhaps a micro-license
that allows pkg.go.dev to render the API and documentation in
perpetuity. This is essentially what the marker file would amount to
legally.

Dan

Elias Naur

unread,
Feb 1, 2020, 4:37:09 AM2/1/20
to golang-dev
FWIW, I suggested a marker on (the now closed) #36840: https://github.com/golang/go/issues/36840#issuecomment-579402752

-- elias

Dan Kortschak

unread,
Feb 1, 2020, 4:52:37 AM2/1/20
to Elias Naur, golang-dev
So it look like they see that as a non-starter.

The impact of this decision will be to put a significant number of
packages into the dark where their documentation is not accessible,
either because the licensing is not correctly recognised or not
Correct™ in the sense that it doesn't conform to the limited set of
licenses that pkg.go.dev wants people to use. This is a pretty tone
dead approach.

Dan
> --
> You received this message because you are subscribed to the Google
> Groups "golang-dev" group.
> To unsubscribe from this group and stop receiving emails from it,
> send an email to golang-dev+...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/golang-dev/349f4b9b-4c17-4acc-8b59-f8cb224219b9%40googlegroups.com
> .


Jan Mercl

unread,
Feb 1, 2020, 4:54:06 AM2/1/20
to Dan Kortschak, golang-dev
On Fri, Jan 31, 2020 at 10:46 PM Dan Kortschak <d...@kortschak.io> wrote:
>
> In the "Next steps for pkg.go.dev" blog post[1] there is an indication
> that godoc.org will be redirected to pkg.go.dev.
>
> I want to say that I think this is badly misguided given the current
> status of go.dev's ability to correctly (notwithstanding [2] being
> closed) identify licenses (and the restrictive set of licenses that it
> will even try to identify) and the consequences for packages that are
> not identified as having a "correct" license.

Thanks Dan for bringing this to my attention. I have now read the blog
post and I'm concerned about the impact of the intent to redirect
godoc.org to pkg.go.dev IMO will have on the contributors community.

Checking whois godoc.org says"Registrant Organization: Google LLC",
while whois go.dev returns "Registrant Name: REDACTED FOR PRIVACY".

Why has Google no problem with package licenses at godoc.org? I don't
know who actually owns go.dev and why is that information not public,
but let me assume it's Google as well. If so, why the same company can
publish documentation for any publicly visible package whatsoever,
regardless of license, at one of its sites and cannot do the same at
another of its sites?

I have nothing against pkg.go.dev and I appreciate the efforts that
went into creating it. But I am not happy with the output. Nearly
every of my (tens) of Go packages that I have published in the last
ten years do not show the README nor the documentation. I'm not
offended, but neither I think treating me and (probably) thousands of
other members of the contributors community this way is fair and/or
justified.

Let me try to be constructive.

1) godoc.org does not understand modules/versions they said. I see no
reason why this cannot be fixed. I am willing to participate in a
group of volunteers, if there are any others, and spent some of my
precious free time, for free, to try to make that happen. I believe
it's feasible, even when started from scratch. Or actually/better
starting from the open code for `go doc` or `godoc` (I'll never learn
the difference) even when I have no idea ATM where it now resides.

And now let me be somehow less constructive.

2) Even though pkg.go.dev has more features that godoc.org, unless it
is 100% backwards feature compatible with godoc.org, the idea of
killing godoc.org by redirecting it to pkg.go.dev is IMO not a good
idea. Throwing away uncountable hours of free work made by the
community in the past 10 years, by hiding the API documentation and
the README at the redirected site, is a hard to accept move for me -
and perhaps I'm not the only one. Yet, the blog clearly communicates
that's the plan quote "later this year" end quote.

Please reconsider, thanks.

PS: Is someone feeling like creating an online voting place or
something like a petition that folks can use? It perhaps could provide
Google an opportunity to show how much it _really_ cares about the Go
community - as expressed _verbally_ at other places and occasions.

gugl.z...@gmail.com

unread,
Feb 1, 2020, 2:29:52 PM2/1/20
to golang-dev
pkg.go.dev has several faults when compared to godoc.org:

1. It's a heavier and slower website. The page for the package
    github.com/gorilla/mux on godoc.org required seven HTTP requests and
    it loads in about 1.3 seconds on my computer and Internet
    connection. The page for the same package on pkg.go.dev requires 19
    requests, and loads in about 2.1 seconds.

2.  pkg.go.dev is a closed-source product, while godoc.org is
    open-source software.  I cannot inspect the source code of
    pkg.go.dev, which is something I and other developers are concerned
    about.  You did the same with proxy.golang.org, and honestly this
    pattern is very disturbing.

Both problems can be solved, but until they aren't, I don't think it's
wise to redirect everything to the new website.

Liam

unread,
Feb 1, 2020, 3:16:10 PM2/1/20
to golang-dev
Dan, thanks for continuing to raise this.

I get the impression that there's an unstated (?) corporate rationale motivating the go.dev license policy and shutdown of godoc.org. I'd guess that stems in part from the Oracle lawsuit re Java APIs.

If so, we've reached a point where the interests of the Go project and those of Alphabet/Google conflict, and it's time to separate Go governance from Google.

Kubernetes also originated in Google, but has been governed outside Google since v1.0. What are the reasons not do the same for Go?

Ulderico Cirello

unread,
Feb 1, 2020, 7:41:16 PM2/1/20
to golang-dev
I want to share my agreement and express it with different terms.

I see two class of problems.

In one, I believe there's a change of policy that is frankly not acceptable and may risk leading to unnecessary forks of effort in the community. pkg.go.dev is closed source and its development has been done in private taking very little input from the community (not because you are not responsive - Andy Bons was prompt in his response to me; but because you are making decisions despite of our expectations).

The other one has to do with functionality: pkg.go.dev does not have feature parity with godoc.org.

With all that said, I have two things more to say.

One - If Google plans to keep control of pkg.go.dev and developing it in private; I am willing to start a movement with people of the community to keep a publicly version of godoc.org available.

Two - regarding the many repositories that are not visible due to license check issues, the solution is quite simple: just adopt https://spdx.org/licenses/ lines in license files. This response by Julie Qiu (https://github.com/golang/go/issues/36840#issuecomment-579816850) seems tone-deaf to me and very discouraging to the necessary debate of a solution that works for everyone in the Go ecosystem. 


Thanks,
Ulderico


--
You received this message because you are subscribed to the Google Groups "golang-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to golang-dev+...@googlegroups.com.

Akhil Indurti

unread,
Feb 1, 2020, 9:37:32 PM2/1/20
to golang-dev
The motivation behind the license policy seems opaque. The comment on #36840 states they want to "encourage package authors to fix their licenses for the broader open source ecosystem," but a "fixed" license suggests that projects whose licenses are approved are recommended for usage over others. However, companies often have varying restrictions on licenses for third-party software they'll adopt. Go for example won't pull in an MIT-licensed dependency unless its maintainer signs a Google CLA, so even then, the list of approved licenses doesn't really help with the goal of "evaluating dependencies". At the end of the day, the programmer, or at worst, a legal team will still have to vet the dependency.

Manlio Perillo

unread,
Feb 2, 2020, 1:17:11 PM2/2/20
to golang-dev
On Saturday, February 1, 2020 at 8:29:52 PM UTC+1, gugl.z...@gmail.com wrote:
pkg.go.dev has several faults when compared to godoc.org:


> [...] 
2.  pkg.go.dev is a closed-source product, while godoc.org is
    open-source software.  I cannot inspect the source code of
    pkg.go.dev, which is something I and other developers are concerned
    about.  You did the same with proxy.golang.org, and honestly this
    pattern is very disturbing.


I would prefer for pkg.go.dev and proxy.golang.org to be open source, but why do you say that the pattern is disturbing?
The two applications runs on Google servers, not on my or your computer.
And even if the source code is released, there is no guarantee that the applications on the Google servers will use the open source code.


Manlio

gugl.z...@gmail.com

unread,
Feb 2, 2020, 4:45:48 PM2/2/20
to golang-dev
> I would prefer for pkg.go.dev and proxy.golang.org to be open source,
> but why do you say that the pattern is disturbing?
> The two applications runs on Google servers, not on my or your
> computer.
> And even if the source code is released, there is no guarantee that
> the applications on the Google servers will use the open source code.

Thoughts about that in no particular order:

1.  Having the source code is *always* preferred to not having it.
    That should pretty much be an axiom by now.
2.  There is no reason why they should not be public.  And none were
    really provided, which is a failure of communication from the
    company's side.
3.  I can host my own version of godoc.org or golang.org.  Why not
    proxy.golang.org or go.dev?  I know about Athens, that's not the
    point.
4.  Now we know that it wasn't just a one-off thing with
    proxy.golang.org, and that the Go people from the company want to go
    on making non-free software for the Go ecosystem.  Yes, those don't
    run on my computer *yet,* but what's stopping them now from making
    a non-free superoptimising compiler or static analysis tool?
5.  We can't see those services' source code, which can mean that the
    authors have something to hide.  Technical flaws?  Security holes?
    License mismatches?  Either way, this atmosphere of not trusting the
    community is harmful.
6.  The argument about “no guarantee that the Goog runs the same code”
    applies to pretty much any FOSS service.  I would like to live in
    a world where we could cryptographically verify that the server
    we're talking to runs the correct software, but AFAIK there is no
    way to do so now.

That's just off the top of my head.  Really, if I wanted my programming
language to be tightly controlled by a shady corporate entity that
dislikes and distrusts FOSS, I would just learn a certain programming
language with a coffee-related name.  I don't want Go, the language
I love, to become that.

Sam Whited

unread,
Feb 2, 2020, 4:58:06 PM2/2/20
to rprichard via golang-dev
Adding to this that the Go team is now taking data about package
downloads and fetching away from package owners and handing it to
Google, this becomes a pattern of bad behavior. I'm sure the Go team has
the best of intentions, but their intentions are meaningless when their
actions are so harmful to the community. I believe them when they say
that Google isn't doing anything with the data they're gathering on Go
packages, but that doesn't mean some new head of the Go team or some new
higher-up at Google won't try to do something with it later.

This pattern of the Go team building closed source servers and
integrating Google run services with little to no input from the
community (and general dismissive comments about the communities
concerns) is absolutely unacceptable and must stop.


—Sam
> --
> You received this message because you are subscribed to the Google
> Groups "golang-dev" group. To unsubscribe from this group and stop
> receiving emails from it, send an email to golang-
> dev+uns...@googlegroups.com. To view this discussion on the web
> visit
> https://groups.google.com/d/msgid/golang-dev/f6336f98-d8ff-417b-af3b-0e38bddce797%40googlegroups.com
> <https://groups.google.com/d/msgid/golang-dev/f6336f98-d8ff-417b-af3b-0e38bddce797%40googlegroups.com?utm_medium=email&utm_source=footer>
> .

--
Sam Whited

Robert Engels

unread,
Feb 2, 2020, 5:42:00 PM2/2/20
to Sam Whited, rprichard via golang-dev
Wow. The victim/outrage culture is reaching new heights - actively insulting the good work and character of those whose output you are then using.

Google services billions of users - none have the source code to these services.

As the Go team pointed out - you always have the ability to fork. Go is open source.

Attempting to dictate how a private company does their business is silly - as long as they comply with applicable laws.

I would ask that you be more respectful in your commentary.

> On Feb 2, 2020, at 3:58 PM, Sam Whited <s...@samwhited.com> wrote:
>
> Adding to this that the Go team is now taking data about package
> To unsubscribe from this group and stop receiving emails from it, send an email to golang-dev+...@googlegroups.com.
> To view this discussion on the web visit https://groups.google.com/d/msgid/golang-dev/02112c90-6b6a-4c0c-b51c-5c3c897df146%40www.fastmail.com.

hes...@google.com

unread,
Feb 2, 2020, 7:20:52 PM2/2/20
to golang-dev
Hi. I usually don't say much on these forums, but I disagree strongly with a lot of what's been said about proxy.golang.org and I wanted to share my perspective. I worked on proxy.golang.org (and sum, and index) and was involved in the design from the beginning. That said, everything here is my opinion, and other members of the Go team may disagree. I'm only speaking for myself and any mistakes are my own.


> 1. Having the source code is *always* preferred to not having it. That 
> should pretty much be an axiom by now. 
I agree with this.

> 2. There is no reason why they should not be public.
There is. The mirror and checksum database are critical services depended on by the majority of Go developers on a daily basis. They absolutely must be correct, reliable, and secure. Open sourcing, while desirable, is not an actual requirement. My opinion was and is that we could provide a better service to Go developers by using proprietary Google technologies. Looking at the source, there is very little that isn't entangled with those technologies. We could have tried to split the pieces up, but that would have created significant complexity in the project, and complexity is the enemy of reliability.

> And none were 
> really provided, which is a failure of communication from the 
> company's side.
While we could have been more open about this (and I'm trying to be now) I don't really agree that the world is owed an in-depth explanation of every decision I make.

> 3. I can host my own version of godoc.org or golang.org. Why not 
proxy.golang.org or go.dev? I know about Athens, that's not the 
> point.
As I've said above, you don't have access to the things that proxy.golang.org is built on. Unfortunate but in my opinion the right tradeoff.

> 4. Now we know that it wasn't just a one-off thing with 
proxy.golang.org, and that the Go people from the company want to 
> go on making non-free software for the Go ecosystem. Yes, those 
> don't run on my computer *yet,* but what's stopping them now from 
> making a non-free superoptimising compiler or static analysis tool? 
I don't think I have anything useful to say here.

> 5. We can't see those services' source code, which can mean that the 
> authors have something to hide.
True. I don't think I have anything to hide, but I doubt you'll find that convincing.

> Technical flaws? Security holes? 
> License mismatches? Either way, this atmosphere of not trusting the 
> community is harmful. 
I don't think any of the decisions made about whether to open source proxy.golang.org had anything whatsoever to do with trusting or not trusting the community.

> 6. The argument about “no guarantee that the Goog runs the same code” 
> applies to pretty much any FOSS service. I would like to live in a 
> world where we could cryptographically verify that the server we're 
> talking to runs the correct software, but AFAIK there is no way to 
> do so now. 
Yes. However, you can verify that the output of proxy.golang.org matches what the origin has. If you find that something suspicious has happened, you can use sum.golang.org to irrefutably prove it. If you don't trust Google, I encourage you to do so and publish the results.

On Sunday, February 2, 2020 at 4:58:06 PM UTC-5 Sam Whited wrote:
Adding to this that the Go team is now taking data about package
downloads and fetching away from package owners and handing it to
Google, this becomes a pattern of bad behavior.
I am not a lawyer and I cannot offer an authoritative interpretation of our privacy policy. However, at the end of https://proxy.golang.org/privacy is the statement:
"We intend to aggregate and anonymize usage metrics to measure popularity for Go modules and share this popularity data with the Go community."
In my opinion, that is the only allowed use of download metrics other than monitoring and debugging. I believe Google services should get the same version of that data that everybody else gets.

I hope this is helpful. I don't think I have much more to say than this, so I probably won't respond any further. I would ask that you (all) remember that these are projects worked on by actual people who are, in my experience, generally just trying to make Go the best it can be.

Liam Breck

unread,
Feb 2, 2020, 7:59:41 PM2/2/20
to hes...@google.com, golang-dev
This thread had gone off-topic. The issue Dan raised is that pkg.go.dev is not ready to replace godoc.org.


thepud...@gmail.com

unread,
Feb 2, 2020, 8:39:26 PM2/2/20
to golang-dev
 > "The issue Dan raised is that pkg.go.dev is not ready to replace godoc.org."

Hello fellow gophers,

FWIW, as far as I have followed, it seems everyone is in agreement that pkg.go.dev is not ready to replace godoc.org, including it seems that the very people working on pkg.go.dev are also saying pkg.go.dev is not ready to replace godoc.org.

The recent blog post (https://blog.golang.org/pkg.go.dev-2020) that was published a couple of days ago said (emphasis added):

  "__later this year__ we are planning to redirect traffic from godoc.org to the corresponding page on pkg.go.dev"

and the next paragraph begins:

  "Your feedback will inform our transition plan"

at which point the blog post then asks for feedback via a variety of channels.

As far as I have followed, there have been a few statements that a rough, early version was launched so that people could comment on something concrete, but that it is not finished yet.

The mid-November blog post also said (https://blog.golang.org/go.dev):

  "Today’s launch is our minimum viable product for go.dev, so we can share what we’ve built to help the community and get feedback."

In any event, it seems the people working on go.dev agree that some work remains, including on the license detection pieces. About a month ago, there was this comment (https://github.com/golang/go/issues/33654#issuecomment-569711045):

   "We plan to respond to all issues in the new year, and to continue working with 
    our legal team to resolve any license issues. 
    Thanks for being patient with us as we work through them!"
 

   "Our current volume of special case issues is low enough that we plan to keep the manual 
    review process for now (rather than building an automatic solution). 
    We can always revisit this strategy as other specific issues come up."

That sounds like they currently have a manual strategy that seems to be working, but also seems to say that they are open to improvements.

I don't have any particular special knowledge here and I am not on the Go team, so apologies in advance if I've said something off-base.

Dan, from a quick look over the different license issues on the issue tracker, it looks like you have put a fair amount of effort into helping work through the licensing problems, so I wanted to say a quick thanks for that from one gopher to another.

Best,
thepudds

Ulderico Cirello

unread,
Feb 2, 2020, 8:57:38 PM2/2/20
to thepud...@gmail.com, golang-dev
Unfortunately, there's more than meet the eyes here. And although it might seem we are veering off the OP, the fundamental question is not so much whether pkg.go.dev is ready to replace godoc.org (obviously it is not), but whether this transition should happen at all.

One way or another you could run godoc or gddo locally: their source code was available and there was some semblance that Google was playing fair by using some aligned version of the software both inside and outside. 

Or to reframe the problem. Right now, if I go and open a CL to change godoc - everyone has the opportunity to chime in and participate. 


pkg.go.dev on the other hand is being developed in the private. We cannot contribute, we cannot review how it's being done, we cannot contribute to its license detection system, we cannot do any of the things we could do to participate. 


More, and at least for me disturbing, Google is using go.dev as a place to publish and foster Google cloud software. Browse go.dev and you're going to see that they prominently added their products ahead of the rest of the ecosystem. 

Thus, not only pkg.go.dev is not ready to replace godoc.org; but go.dev as a domain is a closed service offered by Google to which we all are being extracted away. 


The documentation website is one of the components of go.dev and we, as members of the ecosystem, should be watchful about what's going on elsewhere. 


And finally, Googlers claimed that go.dev is a MVP. Which by the way is problem of their own creation. Godoc.org worked fine and golang.org worked great too. The Go ecosystem does NOT need go.dev. At least not the one Google and the Go team is building in private alienating the ecosystem by constraining the communications to write only channels (go-discovery-feedback) and stonewall (like the response from a Googler in this thread who stated that proxy.golang is too tied to Google infra as a fait accompli). 

Sad days that I would hope to never see in Go. Like I told Andy Bons, the Go team is breaking my heart. 



--
You received this message because you are subscribed to the Google Groups "golang-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to golang-dev+...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/golang-dev/88d6aa3e-3fdb-4fa7-9e4b-12b0d7309789%40googlegroups.com.

minux

unread,
Feb 2, 2020, 9:50:09 PM2/2/20
to golang-dev
On Sun, Feb 2, 2020 at 8:57 PM Ulderico Cirello <ulderi...@gmail.com> wrote:
> Unfortunately, there's more than meet the eyes here. And although it might seem we are veering off the OP, the fundamental question is not so much whether pkg.go.dev is ready to replace godoc.org (obviously it is not), but whether this transition should happen at all.
>
> One way or another you could run godoc or gddo locally: their source code was available and there was some semblance that Google was playing fair by using some aligned version of the software both inside and outside.
>
> Or to reframe the problem. Right now, if I go and open a CL to change godoc - everyone has the opportunity to chime in and participate.
>
>
> pkg.go.dev on the other hand is being developed in the private. We cannot contribute, we cannot review how it's being done, we cannot contribute to its license detection system, we cannot do any of the things we could do to participate.

pkg.go.dev uses https://github.com/google/licensecheck to detect
licenses, and you are certainly welcome contribute to it.
(It puzzles me that a lot of people seem to indicate they're willing
to fork or otherwise enhance related projects, but no one seems to
want to actually contribute and fix the exact issue that this thread
talks about.)

pkg.go.dev is closed source, but the blog post
(https://blog.golang.org/pkg.go.dev-2020) also explicitly mentioned
that the team is planning to develop an open-source equivalent that
you can host on your own servers.

Ulderico Cirello

unread,
Feb 2, 2020, 9:58:26 PM2/2/20
to minux, golang-dev
Thanks for the response Minux.

Equivalent is not the same. 

A library that a project uses is not the project itself. 

As much as I can contribute, I do. And I did in other areas (playground and documentation examples). 

And again, making it a new closed source service that barely replaces an open source one that works is a solution for a problem that didn't exist first off. 

In any case, I might be beating a dead horse by now. I apologize. 

--
You received this message because you are subscribed to the Google Groups "golang-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to golang-dev+...@googlegroups.com.

Akhil Indurti

unread,
Feb 2, 2020, 11:17:44 PM2/2/20
to golang-dev
I thought this discussion was about documentation not being displayed,
but it seems to have trailed off into FUD. The whole point of a time gap
till later this year is to garner feedback.

That being said, any project that is closed-source must make the extra
effort to clarify its intent. Such information should be public and visible
on the documentation, not on an issue tracker. However, given the
relatively recent launch of the site, I'm willing to give them the benefit
of the doubt.

On Friday, January 31, 2020 at 4:46:14 PM UTC-5, Dan Kortschak wrote:

Liam

unread,
Feb 3, 2020, 4:16:58 AM2/3/20
to golang-dev
Besides the (temporary) problem of license recognition and (permanent?) limitations on accepted licenses, there is a bizarre belief that anyone can publish another author's work under arbitrary terms, described in

roger peppe

unread,
Feb 3, 2020, 5:25:30 AM2/3/20
to Liam, golang-dev
About losing package visibility: I agree that it's a problem when packages that have historically been available in godoc.org are no longer available on the new (and much better, IMHO!) pkg.go.dev site.

Two possible workarounds for this:

 - only redirect from godoc.org to pkg.go.dev when the package/module is available on the new site.
 - grandfather all the packages on godoc.org, so that all the old content is still visible. New content would still need a compatible license (ISTM if you're publishing new content, you're probably in a position to change to using an accepted license).


--
You received this message because you are subscribed to the Google Groups "golang-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to golang-dev+...@googlegroups.com.

Ian Davis

unread,
Feb 3, 2020, 6:16:32 AM2/3/20
to golan...@googlegroups.com
On Mon, 3 Feb 2020, at 2:49 AM, minux wrote:
>
> pkg.go.dev uses https://github.com/google/licensecheck to detect
> licenses, and you are certainly welcome contribute to it.
> (It puzzles me that a lot of people seem to indicate they're willing
> to fork or otherwise enhance related projects, but no one seems to
> want to actually contribute and fix the exact issue that this thread
> talks about.)
>

I think that is a little unfair. Dan Kortschak, for example, raised relevant issues[1] but the response is that a redesign of the library is needed. Without sharing any thinking behind that redesign then it's difficult to guess what would be accepted as a fix.

[1] https://github.com/google/licensecheck/issues/6

-- Ian

Russ Cox

unread,
Feb 3, 2020, 5:17:07 PM2/3/20
to golang-dev, Dan Kortschak
Hi all,

Julie's blog post was first and foremost a request for feedback, so
thanks to Dan and everyone else for sharing your thoughts on this
thread. As I hope the post made clear, the discovery site is not done
yet, we know some ways it needs to be made better, and more
importantly we know we are probably blind to others, hence the call
for feedback. Please bear with us while we digest that feedback and
work on fixes. Please also keep in mind that, as I've noted before, we
are all pulling in the same direction. We all want what's best for Go.

I also want to say thanks for keeping us honest about all of this. The
last year has been hectic and we haven't been completely clear about
some of the reasons for recent work. (I take full responsibility for
this and am definitely not faulting anyone else on the Go team!)
It's very helpful when people call us out on that. The high standards
you all demand truly are one of my favorite things about open source.

This mail is a bit long but I hope it addresses most of the discussion
over the weekend. If you feel I'm missing anything, please reply and
I'm happy to continue the discussion.

# Background about go.dev

One thing we've come to understand over the past year or so is that
many in the next wave of Go adopters want a “one-stop” web site full
of resources about Go, including how to get started, who is using it
already, links to learning resources, package docs, and so on. The new
go.dev is meant to be that one-stop site. In the blog announcing it,
we called it “a new hub for Go developers.”

# Why is go.dev different from golang.org?

Personally, I thought that separating the two would allow go.dev to be
more inclusive of community content. Historically, golang.org is the
place where the Go project speaks authoritatively about Go: it has the
language spec, standard library docs, official downloads, and so on.
It has always been important to us not to dilute that with all the
other Go content in the world. The new site seemed to be an
opportunity to create a more inclusive place for other resources,
hence a second site.

# Background about godoc.org

Gary Burd created and ran godoc.org starting in late 2012. It was and
remains both a brilliant idea and clearly a valuable service to the Go
community. After a year or so, Gary decided he didn't want to run and
pay for the servers anymore, so he gave it to Go project to adopt. We
were happy to do that, to make sure this resource remained available
to all Go users. We said it back in 2014 when we adopted godoc.org and
I will say it again: I am incredibly grateful to Gary for what he
created.

# Why is there a new package docs site at all? Why not update
godoc.org in place?

With the introduction of modules and the notion of multiple versions
of a package, we knew we had to update the godoc.org experience. After
a hard look, it seemed worth starting anew, especially since the
godoc.org server design, with its single-VM database, had been
starting to show its age. In addition to the modules work, there are
other things we're addressing, such as accessibility and overall
scalability of the service.

As a side note, there's almost nothing in the Go distribution that has
survived eight years without being redone. The compiler, the
assembler, the linker, the go command itself, most of the standard
library: all of them have been massively overhauled one or more times
since the start of Go. That's how we take what we learn and make
things better.

This kind of rewrite always involves a transition period in which the
old version is still the workhorse that most people use and the new
version has a new name for early adopters to test and find bugs in.

# Why is the new package docs site on pkg.go.dev?

Docs for all the packages in the entire ecosystem are exactly the kind
of community-generated content that go.dev is meant to help find, so
pkg.go.dev seemed like a good name. Especially since go.dev has a much
broader scope than godoc.org, it makes sense to take the opportunity
to fold it in and reduce the number of sites a typical user has to be
aware of (once the transition is complete).

# Why are you talking about redirecting godoc.org to pkg.go.dev
when pkg.go.dev is still so broken?

Bluntly, so that you will help us understand what's broken, so we can
fix it before the redirect happens. We know about some things but
would not be surprised to find there are things we're completely
unaware of. Better to find out earlier rather than later. Again, the
blog post was first and foremost a request for feedback about what
needs to happen before we actually do the redirect.

# Why does pkg.go.dev require a detected license to show docs?
Why doesn't godoc.org?

The teams working on the proxy and on pkg.go.dev have spent a lot of
time talking to Google's lawyers about what we can and can't do with
Go source code downloaded from the internet. The rule we've been given
to follow is that serving a pretty HTML version of the docs is
displaying a modified version of the original, and we can only do that
if there's a recognized known-good license that gives us that
permission.

When we adopted godoc.org from Gary Burd back in 2014, it did not
occur to any of us to put it through that kind of review. If we had,
maybe the community would have gone through this licensing pain
earlier. For now we are focusing on making changes to pkg.go.dev
rather than correcting past mistakes on godoc.org. (At this point,
more scrutiny of what godoc.org does is not likely to have an outcome
that anyone likes.)

# What fraction of popular packages don't display on pkg.go.dev?

Right now it looks like pkg.go.dev sees 1,200 modules imported by at
least 100 other modules. Of those, it looks like 82 are flagged as not
redistributable, so that we can't show their docs. That's under 7%,
and we're working to understand that better. If any of those are
mistakes on our end, we'll fix them.

Another thing that was suggested that I think is a great idea is to
change the “no docs available” page to have a command-line to bring up
the docs in your own local godoc command.

[Aside: Speaking generally (not just about Go), you would be surprised
at the number of software packages that have a declared "license type"
in some kind of metadata (GitHub meta, package.json, SPDX codes, etc)
but no actual license text. This makes the license impossible to comply
with! For example, the MIT license requires that "The above copyright
notice and this permission notice shall be included in all copies or
substantial portions of the Software." But if there's no such notice
to include, only an "// SPDX-License-Identifier: MIT" comment, there's
literally no way to comply. It can be a real mess. If you have never before
encountered the differences between how programmers see the world
and how lawyers do, let me recommend "What Colour are your bits?"]

# Why hasn't pkg.go.dev been open sourced?

There's no conspiracy here. The original plan was to open source it,
but open sourcing puts pressure on the code base to be reusable in
other contexts. Right now the code is only written for one context:
the global pkg.go.dev site.

I helped edit the blog post and was responsible for the "Will
pkg.go.dev be open-sourced so I can run it at work for my private
code?" heading. I apologize for the offense I caused so many of you.
I did not mean to imply that there were no other reasons to run a doc server,
only to highlight as an example what I thought was the most common
reason Go developers would want the code.

In much the same way that we published an open source reference
implementation of the proxy that was not the global proxy site, I
still hope that we will publish an open source reference
implementation of a module-aware package docs site that need not be
exactly the global site. An implementation that doesn't have to hit
the same global scale can be much simpler to run—whether it's at work
with private code or on your laptop with completely open-source code.
I also hope the same server can serve index queries that make tools
like goimports and gopls faster. In contrast pkg.go.dev probably can't
serve such queries, at least not the same way, for both scaling
concerns and privacy concerns.

So the reason in the blog post is the real reason: the current code is
not what you probably should run, whether at work or on your laptop in
offline mode, and I thought we could deliver a much better answer for
that.

But I very much hear all of you who want to see the code that is
running on the site and possibly contribute to it, whether it makes sense
to run in other contexts or not. I am going to look into that.

Again, thanks for calling us out when we're not communicating clearly
with all of you. It really helps. I hope this mail helps too, and if
not, let me know.

Best,
Russ

Dan Kortschak

unread,
Feb 3, 2020, 7:37:46 PM2/3/20
to Russ Cox, golang-dev
Thanks Russ,

The issues that I am concerned about are; 1. the going dark of
unacceptably licensed packages, and 2. the limited definition of what
is acceptable in licensing of packages. The other aspects of this
discussion are interesting, but weren't the primary focus of my
original post.

The explanation for the need for care in licensing makes sense (and
brings to mind the old induction-based joke about lawyers in benthic
environments). However, I continue to have issues about how this will
work mechanically. From posts at issues, it appears that at least for
the short term this will be papered over by human vetting; I don't see
this changing in the longer term given that licensing is complicated
(see more below about issues that we skirt in licensing Gonum source).

I think a reasonable approach would be to have first stage license
detection performed mechanically (with an improved detection algorithm
that makes use of SPDX license descriptions - see licensecheck#6), and
a complete set of licenses, with a fallback to render a note explaining
why it is not rendered as documentation, a link to the original source
host, and a mechanism that allows it to be flagged as actually licensed
(without a need to file an issue) that is acted on outside go.dev
release cycles.

At the moment I don't believe that even the cases that appear to be
correctly marked with licenses are correctly marked (this is
golang.org/issue/36953), and I am not convinced that it is actually
possible to correctly mark licenses without human intervention. If the
license notes are purely there as a brief note to explain why go.dev
has the opinion it does, then it's possibly reasonable, but I doubt
that most people would read them that way. Maybe that could be made
more clear.

*(note from above) Part of the ethos for the Gonum project is proper
ascription of thanks for work. In this we are very careful with
licensing (we have a THIRD_PARTY_LICENSES directory that includes
licenses for code that we have worked from in porting routines to Go -
whether we need to do this is not entirely clear, the FSF would say
yes, but many Gophers appear not to agree with this; for example, the
Go project itself does not include third party license information for
a set of code that comes from other sources and have historically come
through to the projects source from e.g. plan9/Inferno). We do this to
1. thank the original authors and acknowledge their contributions, but
also to 2. allow our users to decide for themselves whether the
licensing of the code needs to include these third parties. go.dev
ignores these. I'm not sure this problem is solvable.

Dan

Dan Kortschak

unread,
Feb 3, 2020, 7:54:07 PM2/3/20
to Russ Cox, golang-dev
As an addendum to this, there is an argument that the pkg.go.dev team
has given me the rights to see the source of pkg.go.dev as a
consequence of displaying this package[1] and the first clause of the
license which grants me "access to the licensee's hardware".

The license text quoted here:
```
Copyright 2019 Dan Kortschak

Redistribution and use in source and binary forms, with or without
modification, are permitted provided that the following conditions are
met:

1. Redistributions of source code must retain the above copyright
notice, this list of conditions and the following disclaimer, and grant
access to the licensee's hardware.

2. Redistributions in binary form must reproduce the above copyright
notice, this list of conditions and the following disclaimer in the
documentation and/or other materials provided with the distribution.

3. Neither the name of the copyright holder nor the names of its
contributors may be used to endorse or promote products derived from
this software without specific prior written permission.

THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS
IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED
TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A
PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT
HOLDER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
(INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
```

This package was written with the intention of illustrating the danger
of lawyering a machinā.

Dan

[1]https://pkg.go.dev/github.com/kortschak/unlicensable?tab=doc
> --
> You received this message because you are subscribed to the Google
> Groups "golang-dev" group.
> To unsubscribe from this group and stop receiving emails from it,
> send an email to golang-dev+...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/golang-dev/ec0e25723a2c86fb045347cc98e3f5b4e2f209af.camel%40kortschak.io
> .


Ian Lance Taylor

unread,
Feb 3, 2020, 8:12:28 PM2/3/20
to Dan Kortschak, Russ Cox, golang-dev
On Mon, Feb 3, 2020 at 4:37 PM Dan Kortschak <d...@kortschak.io> wrote:
>
> *(note from above) Part of the ethos for the Gonum project is proper
> ascription of thanks for work. In this we are very careful with
> licensing (we have a THIRD_PARTY_LICENSES directory that includes
> licenses for code that we have worked from in porting routines to Go -
> whether we need to do this is not entirely clear, the FSF would say
> yes, but many Gophers appear not to agree with this; for example, the
> Go project itself does not include third party license information for
> a set of code that comes from other sources and have historically come
> through to the projects source from e.g. plan9/Inferno). We do this to
> 1. thank the original authors and acknowledge their contributions, but
> also to 2. allow our users to decide for themselves whether the
> licensing of the code needs to include these third parties. go.dev
> ignores these. I'm not sure this problem is solvable.

The Go project does of course include this additional license
information in the files themselves (e.g.,
https://golang.org/src/cmd/compile/internal/gc/gsubr.go). Is your
concern here that the information does not appear in the top level
LICENSE file?

Ian

Dan Kortschak

unread,
Feb 3, 2020, 8:17:14 PM2/3/20
to Ian Lance Taylor, Russ Cox, golang-dev
Thanks, Ian. Yes, that's the issue (or at least that it's not as
obviously licensed as it could be). We've discussed this before and I
didn't press it, but since we're here I thought I'd bring it up again.

It's based on a broader problem of whether people actually include
those licenses in their software. I doubt most Go users are even aware
that the Lucent and Vita Nuova licenses exist, let alone making sure
that they are included in compiled products (I'd say that people don't
include the Go license often in compiled products).

Dan
> --
> You received this message because you are subscribed to the Google
> Groups "golang-dev" group.
> To unsubscribe from this group and stop receiving emails from it,
> send an email to golang-dev+...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/golang-dev/CAOyqgcXwC%3DuXVCQzo85cL6c22oodRZLRhfo%2BFPR1EJwLX6UxFA%40mail.gmail.com
> .


Ian Lance Taylor

unread,
Feb 3, 2020, 8:53:25 PM2/3/20
to Dan Kortschak, Russ Cox, golang-dev
On Mon, Feb 3, 2020 at 5:17 PM Dan Kortschak <d...@kortschak.io> wrote:
>
> Thanks, Ian. Yes, that's the issue (or at least that it's not as
> obviously licensed as it could be). We've discussed this before and I
> didn't press it, but since we're here I thought I'd bring it up again.
>
> It's based on a broader problem of whether people actually include
> those licenses in their software. I doubt most Go users are even aware
> that the Lucent and Vita Nuova licenses exist, let alone making sure
> that they are included in compiled products (I'd say that people don't
> include the Go license often in compiled products).

It's worth noting that the Lucent and Vita Nuova copyrights only exist
on Go tools like the compiler and linker, not on any part of the Go
library. There is no requirement or expectation that Go users include
those licenses with arbitrary programs written in Go. The only
possible question is how they should be handled in distributions of
the Go tools.

Ian

Dan Kortschak

unread,
Feb 3, 2020, 9:08:06 PM2/3/20
to Ian Lance Taylor, Russ Cox, golang-dev
They are indeed mainly in the compiler, but also exist in the runtime
for memmove and memclr (also vlop and vlrt for some architectures).

Dan

Ian Davis

unread,
Feb 4, 2020, 7:35:40 AM2/4/20
to golan...@googlegroups.com
On Mon, 3 Feb 2020, at 2:49 AM, minux wrote:
> On Sun, Feb 2, 2020 at 8:57 PM Ulderico Cirello <ulderi...@gmail.com> wrote:
> > pkg.go.dev on the other hand is being developed in the private. We cannot contribute, we cannot review how it's being done, we cannot contribute to its license detection system, we cannot do any of the things we could do to participate.
>
> pkg.go.dev uses https://github.com/google/licensecheck to detect
> licenses, and you are certainly welcome contribute to it.
> (It puzzles me that a lot of people seem to indicate they're willing
> to fork or otherwise enhance related projects, but no one seems to
> want to actually contribute and fix the exact issue that this thread
> talks about.)

Some licensing displays problems are not fixable by changing the licensecheck package. For example, see https://pkg.go.dev/github.com/iand/salience?tab=licenses which dedicated to the public domain using the Unlicense, which is explicitly supported by licensecheck. As recommended by https://unlicense.org/ the license text is in a file called UNLICENSE which pkg.go.dev does not consider when calling licensecheck.

Where should I file this issue? On the Go issue tracker?

-- Ian

Kevin Chadwick

unread,
Feb 4, 2020, 8:20:42 AM2/4/20
to golan...@googlegroups.com
On 2020-02-03 22:16, Russ Cox wrote:
> Historically, golang.org <http://golang.org> is the
> place where the Go project speaks authoritatively about Go: it has the
> language spec, standard library docs,

I wonder how much pain it would be for the spec and std lib docs to be bundled
in whatever fashion with the GO packages download again (doc server removal),
otherwise I don't really see how go.dev would become the one-stop shop?

I know that when I started, having the go doc server in the application menu
helped me find the docs quickly, and proved useful offline.

thepud...@gmail.com

unread,
Feb 4, 2020, 8:31:32 AM2/4/20
to golang-dev
Hello Ian,

From what I understand, you have at least two options for questions or issues with a specific license: you can start a conversation by sending an email to go-discove...@google.com, or you can open a "pkg.dev:" issue on this Go issue tracker on github.

For a license question such as you posed, personally I would suggest opening an issue on github, which of course enables others to see the conversation, comment, subscribe, and so on.

From a portion of the blog from last week (https://blog.golang.org/pkg.go.dev-2020):

  "As always, our license policy is at pkg.go.dev/license-policy. If you are having issues, 
   feel free to file an issue on the Go issue tracker, or email go-discove...@google.com 
   so that we can work with you directly!"

thepudds

Russ Cox

unread,
Feb 4, 2020, 8:36:20 AM2/4/20
to Dan Kortschak, golang-dev
On Mon, Feb 3, 2020 at 7:37 PM Dan Kortschak <d...@kortschak.io> wrote:
The issues that I am concerned about are; 1. the going dark of
unacceptably licensed packages, and 2. the limited definition of what
is acceptable in licensing of packages. The other aspects of this
discussion are interesting, but weren't the primary focus of my
original post.

They were the focus of others on the thread. I was not replying only to you.
 
I think a reasonable approach would be ...

I am starting to have to question whether you are engaging in good faith.
I've explained the legal reasons that we can't display docs for packages
without a license that permits such a use. I've also explained what our 
lawyers are comfortable having us do to check for such a license.
And it appears that your gonum packages are confirmed as having an
acceptable license and are displayed on pkg.go.dev. I don't understand
what is left that is affecting you that we can fix. It doesn't matter what you
think is reasonable. What matters is the relevant copyright law and what
efforts our lawyers have asked us to make to comply with it.

It does not help to try to engage in hypotheticals like your unlicensable
package, meant to show that there is a fuzzy boundary being enforced.
We all agree the boundary is fuzzy, but that doesn't make it less important.
If this were a restaurant enforcing a "no shirt, no shoes, no service" policy
mandated by the local health department, it really wouldn't help to try 
walking in with less and less of a shirt to demonstrate that the boundary
is fuzzy. Wearing a shirt made of saran wrap or fishing net does not prove
that the policy is bad or unenforceable. It only distracts the restaurant
employees - who didn't make and can't change the policy anyway - from 
doing their actual jobs.

If there is something concretely wrong with the license detection for an
actual package of yours, please do let us know. If not, let's give up on the
hypothetical problems please.

Thanks.

Best,
Russ

Sam Whited

unread,
Feb 4, 2020, 9:23:36 AM2/4/20
to rprichard via golang-dev
RE: the license issues:

Would giving pkg.go.dev to a foundation that's not controlled by a risk
averse behemoth like Google not fix the problem? Then you could display
the package because it's just common sense and the lawyers are just
being overly cautious, and if someone does sue you you don't have any
assets they can go after anyways except what's been donated by Google
and others to run the servers.

The underlying problem in my mind is that all of this is controlled by
Google which has now decided (probably in good faith on the Go team's
part, but decided never the less) to start pulling Google's normal bait
and switch like they do with all of their larger products. Starting a
foundation that is not a wholly owned subsidiary of Google seems like a
good way to relieve whatever pressures there are and make sure the legal
risk isn't too much to just display an API since the foundation doesn't
have many assets and isn't currently being sued by Oracle.

—Sam

Ian Lance Taylor

unread,
Feb 4, 2020, 9:32:46 AM2/4/20
to Dan Kortschak, Russ Cox, golang-dev
On Mon, Feb 3, 2020 at 6:08 PM Dan Kortschak <d...@kortschak.io> wrote:
>
> They are indeed mainly in the compiler, but also exist in the runtime
> for memmove and memclr (also vlop and vlrt for some architectures).

Oh yeah, I keep forgetting about those.

Ian Lance Taylor

unread,
Feb 4, 2020, 9:42:42 AM2/4/20
to Sam Whited, rprichard via golang-dev
It really does not seem fair to describe this as a bait and switch.

Let's fix the real problems and then see where we stand.

Ian

Sam Whited

unread,
Feb 4, 2020, 9:48:35 AM2/4/20
to Ian Lance Taylor, rprichard via golang-dev
On Tue, Feb 4, 2020, at 09:42, Ian Lance Taylor wrote:
> It really does not seem fair to describe this as a bait and switch.
>
> Let's fix the real problems and then see where we stand.

Naturally, I disagree. While I am trying to be careful to attribute good
motives to the Go team, taking a more or less distributed package
manager and then rebuilding important pieces of it so that package
maintainers lose data and Google gets all the data (and the best we can
do is trust that they'll release some data for us after the fact at some
point and play fair with it even though they had it first and have it on
lock down), and so that everything has to go through Google servers
seems like a bait and switch.

If the package management system of Go, to continue using this
example since I feel most strongly about it, had required that all my
packages be served through Google when I first started learning Go I
would have found some other language and package ecosystem to use.
Now I'm more or less locked into this system after having built a
career and a lot of libraries around it, and I have no recourse
except to accept the promises of my Google overlords that they won't
do anything evil going forward, just like all their other products
which then broke those promises.

I understand not everyone will share my deep dislike of Google, but the
tighter they integrate Go services the less happy I'll be. A language is
fundamental computing infrastructure, and should not be controlled by
any one company or you wind up with different, but related problems,
like the ones that have been mentioned here and a lot of unhappy users.

—Sam

Ian Lance Taylor

unread,
Feb 4, 2020, 10:19:18 AM2/4/20
to Sam Whited, rprichard via golang-dev
Perhaps I misunderstand, but I feel like you aren't talking about
godoc.org or pkg.go.dev.

I think we've worked pretty hard to make sure that the Go
infrastructure is open source and available for other people to run
their own versions.

Ian

Paul Jolly

unread,
Feb 4, 2020, 10:24:11 AM2/4/20
to Sam Whited, Ian Lance Taylor, rprichard via golang-dev
"...so that everything has to go through Google servers"

"...required that all my packages be served through Google"

"... more or less locked into this system"

"... I have no recourse"

Please can you provide specific evidence for the quotes above?

Because my understanding on the points from which the quotes are
extract does not match yours.

Thanks

Sam Whited

unread,
Feb 4, 2020, 10:39:31 AM2/4/20
to Paul Jolly, Ian Lance Taylor, rprichard via golang-dev
On Tue, Feb 4, 2020, at 10:23, Paul Jolly wrote:
> "...so that everything has to go through Google servers"
>
> "...required that all my packages be served through Google"
>
> "... more or less locked into this system"
>
> "... I have no recourse"

I am of course talking about pkg.go.dev, which is not open source. I
know they're planning on building a different open source version that
doesn't have the same scaling characteristics as the Google version,
which uses internal Google products to scale, but this also gives Google
their own special version while everyone else is relegated to a version
that can't be scaled to the same degree.

I am also talking about the proxy and checksum databases, which are
similar. These for me are the biggest problems: Google should not
control how my package gets downloaded. I understand that I can turn
them off, or in theory use my own servers eventually when the open
source versions of those eventually become available, but now I have to
jump through hoops and all my users will likely be using the Google
version or not use my library if I ensure it's not available through the
Google version. With pkg.go.dev, something similar happens: if I run my
own docs server, the Google one remains the default that most people
will go to to find packages. If my package has a license that Google
doesn't like out of an abundance of caution, saying that I can run my
own documentation isn't really an alternative: users will search on
pkg.go.dev, not find my library, and it will never be adopted. Google
has given themselves a monopoly in this space.

—Sam

--
Sam Whited

Paul Jolly

unread,
Feb 4, 2020, 11:06:19 AM2/4/20
to Sam Whited, Ian Lance Taylor, rprichard via golang-dev
> I am of course talking about pkg.go.dev, which is not open source.

<snip>

Forgive me, but I didn't see anything in your answer that answered my question.

Instead:

> "...so that everything has to go through Google servers"

You appear to acknowledge that not everything has to go through Google servers.

> "...required that all my packages be served through Google"

You appear to acknowledge that not all packages must be served through Google.

> "... more or less locked into this system"

You outline a means by which you are not locked into this system.

> "... I have no recourse"

You outline at least one alternative approach, i.e. your recourse.

Rather your answer appears to deflect onto other points of hyperbole
that reflect your "deep dislike of Google".

> Google has given themselves a monopoly in this space.

Again, you will need to explain to me how a monopoly has been and will
remain established here. Because I can list Athens and the GoCenter
(https://search.gocenter.io/stats) as two alternatives (in various
forms) as of today.

I might be alone here, but I fail to see how your argument is
constructive. In such exchanges online I think it's beholden on those
more experienced in an area, such as yourself, to set an example to
others. Being patient, thoughtful, respectful, charitable and
constructive are important values in that respect.

Sam Whited

unread,
Feb 4, 2020, 11:10:35 AM2/4/20
to Paul Jolly, Ian Lance Taylor, rprichard via golang-dev
On Tue, Feb 4, 2020, at 11:06, Paul Jolly wrote:
> Forgive me, but I didn't see anything in your answer that answered my
> question.

I apologize, I'm not trying to be dense but on re-reading it over and
over I don't see how it doesn't answer the question.

Let's just focus on package proxying so that we have one example
(although I think pkg.go.dev works the exact same way since it will be
the place people discover new packages): If we build and deploy our own
package hosting, no one will use our packages because in their default
install of Go if they try to fetch them it won't work since the Go
install has Google's proxy set as the default. Yes, we can
*technically* run a package server, but if the end result is that no
one can use our packages without having to jump through hoops
(following our instructions to reconfigure Go to use our package
proxy), they won't, therefore I am forced to allow my package to be
served by Google's proxy.

I hope that's more clear.

—Sam


--
Sam Whited

Paul Jolly

unread,
Feb 4, 2020, 11:21:20 AM2/4/20
to Sam Whited, Ian Lance Taylor, rprichard via golang-dev
> I apologize, I'm not trying to be dense but on re-reading it over and
> over I don't see how it doesn't answer the question.

It doesn't answer my question because you didn't provide specific
evidence to back up that point that "EVERYTHING has to go through
Google servers" (emphasis my own).

My point is that your emails are littered with hyperbolic absolute
statements which, thus far, do not have any evidence behind them.

> Let's just focus on package proxying so that we have one example
> (although I think pkg.go.dev works the exact same way since it will be
> the place people discover new packages): If we build and deploy our own
> package hosting, no one will use our packages because in their default
> install of Go if they try to fetch them it won't work since the Go
> install has Google's proxy set as the default. Yes, we can
> *technically* run a package server, but if the end result is that no
> one can use our packages without having to jump through hoops
> (following our instructions to reconfigure Go to use our package
> proxy), they won't, therefore I am forced to allow my package to be
> served by Google's proxy.

Again, more hyperbole (and hypotheticals):

* "no one will use our packages"
* "they won't"
* "I am forced"

Honestly, I'm personally having a hard time reading past the hyperbole.

Would a good place to start be explaining what problem you are trying
to solve for here (in this thread or another) and specific issues you
have encountered with people using some of the alternatives you
outline?

Sam Whited

unread,
Feb 4, 2020, 11:44:33 AM2/4/20
to Paul Jolly, Ian Lance Taylor, rprichard via golang-dev
On Tue, Feb 4, 2020, at 11:21, Paul Jolly wrote:
> Again, more hyperbole (and hypotheticals):

Hypotheticals and hyperbole are fine. It should be obvious that I am
saying "Go went from an equal and fair platform to an unacceptable level
of being tied to Google services".

You know my views, and I think my examples of why this is bad for the
ecosystem and only to the benefit of Google are good (if not well
expressed, sorry for that, my rage at the direction the Go team has
taken the project is almost certainly affecting my ability to make a
good argument here and I recognize that). Further discussion is
unproductive. I am done, but may try to write a more thoughtful blog
post on the topic if I can convince work to give me the time.

—Sam

--
Sam Whited

Russ Cox

unread,
Feb 4, 2020, 12:13:26 PM2/4/20
to Sam Whited, golang-dev
On Tue, Feb 4, 2020 at 9:23 AM Sam Whited <s...@samwhited.com> wrote:
Would giving pkg.go.dev to a foundation that's not controlled by a risk
averse behemoth like Google not fix the problem? Then you could display
the package because it's just common sense and the lawyers are just
being overly cautious, and if someone does sue you you don't have any
assets they can go after anyways except what's been donated by Google
and others to run the servers.

I am not a lawyer, and this is not legal advice, but speaking in the most 
general possible terms, "set up a foundation with no money, so it can do illegal
things, in the hopes that no one looks at who is behind the curtain" is not a
strategy that typically leads to a good outcome for the parties involved.

Put a different way, a foundation is not a magic shield from the law.
Instead, foundations have lawyers too, lawyers who are likely to give similar
advice about the importance of following the law.

Best,
Russ

Sam Whited

unread,
Feb 4, 2020, 12:15:24 PM2/4/20
to Russ Cox, rprichard via golang-dev
On Tue, Feb 4, 2020, at 12:13, Russ Cox wrote:
> I am not a lawyer, and this is not legal advice, but speaking in the
> most general possible terms, "set up a foundation with no money, so it
> can do illegal things, in the hopes that no one looks at who is behind
> the curtain" is not a strategy that typically leads to a good outcome
> for the parties involved.

I am also not a lawyer, but this is not illegal and is probably
perfectly fine: Google is just overly risk averse in case someone takes
issue with it and tries to interpret the law in a different way and sue.

—Sam

Robert Engels

unread,
Feb 4, 2020, 12:40:16 PM2/4/20
to Sam Whited, Russ Cox, rprichard via golang-dev
What do you care if no one uses your packages? The Go ecosystem is not designed towards commercialization, so what’s your agenda?

It certainly seems like you have a axe to grind with Google, and you are mistakenly attacking Go as a proxy.

No one is forcing you to use Go. There are plenty of alternatives, and for in house usage none of these changes have any bearing. So what’s your point other than “I think Google is bad and I’m taking this opportunity to say so”?

> On Feb 4, 2020, at 11:15 AM, Sam Whited <s...@samwhited.com> wrote:
> --
> You received this message because you are subscribed to the Google Groups "golang-dev" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to golang-dev+...@googlegroups.com.
> To view this discussion on the web visit https://groups.google.com/d/msgid/golang-dev/71664392-d267-476b-9e4c-21b94629f1d9%40www.fastmail.com.

Ralph Corderoy

unread,
Feb 4, 2020, 1:56:46 PM2/4/20
to golan...@googlegroups.com
Hi,

Russ wrote:
> The teams working on the proxy and on pkg.go.dev have spent a lot of
> time talking to Google's lawyers about what we can and can't do with
> Go source code downloaded from the internet. The rule we've been
> given to follow is that serving a pretty HTML version of the docs is
> displaying a modified version of the original, and we can only do that
> if there's a recognized known-good license that gives us that
> permission.

That suggests to me that turning documentation written as a man page,
asciidoc, etc., into HTML and serving it would also fall foul. Many
distros, man-page sites, and GitHub do this without checking the licence
conditions.

I expect a different set of lawyers may reach a different conclusion.
Like lawyers for a Go Foundation, akin to the {Python,Linux,Apache,...}
Foundations. Yes, funding required, but corporate funding could come
from the same player that is directly funding it now, Google, without
Google being so wary of protecting itself in legal opinion. And it
would allow others to chip in.

More overhead than Google just running the show, but Go may reach a time
where it's worth it.

--
Cheers, Ralph.

Chris Broadfoot

unread,
Feb 4, 2020, 2:01:00 PM2/4/20
to Ralph Corderoy, golang-dev
On Tue, Feb 4, 2020 at 10:56 AM Ralph Corderoy <ra...@inputplus.co.uk> wrote:
Hi,

Russ wrote:
> The teams working on the proxy and on pkg.go.dev have spent a lot of
> time talking to Google's lawyers about what we can and can't do with
> Go source code downloaded from the internet.  The rule we've been
> given to follow is that serving a pretty HTML version of the docs is
> displaying a modified version of the original, and we can only do that
> if there's a recognized known-good license that gives us that
> permission.

That suggests to me that turning documentation written as a man page,
asciidoc, etc., into HTML and serving it would also fall foul.  Many
distros, man-page sites, and GitHub do this without checking the licence
conditions.

By using GitHub, you give them a license to do this:

"You grant us and our legal successors the right to store, parse, and display Your Content, and make incidental copies as necessary to render the Website and provide the Service. This includes the right to do things like copy it to our database and make backups; show it to you and other users; parse it into a search index or otherwise analyze it on our servers; share it with other users; and perform it, in case Your Content is something like music or video."
 
I expect a different set of lawyers may reach a different conclusion.
Like lawyers for a Go Foundation, akin to the {Python,Linux,Apache,...}
Foundations.  Yes, funding required, but corporate funding could come
from the same player that is directly funding it now, Google, without
Google being so wary of protecting itself in legal opinion.  And it
would allow others to chip in.

More overhead than Google just running the show, but Go may reach a time
where it's worth it.

--
Cheers, Ralph.

--
You received this message because you are subscribed to the Google Groups "golang-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to golang-dev+...@googlegroups.com.

Liam Breck

unread,
Feb 4, 2020, 2:38:22 PM2/4/20
to Russ Cox, Sam Whited, golang-dev
Many search services do what godoc.org does, print part of a published document. It's not generally considered illegal.

The normal means to protest such use is a DMCA takedown request. Google publishes data on those it receives at transparencyreport.google.com. Have you ever received a takedown request re godoc.org?

Have you heard of any search service which refuses to print full results for open source projects that don't use one of a certain set of licenses?

There is no need for the anxious policy re open source docs which you plan to adopt. If your lawyers are convinced there is, and you can't seek a second opinion, a Go foundation is a reasonable solution.

At the very least, hand godoc.org off to a third party.

Ian Lance Taylor

unread,
Feb 4, 2020, 4:15:21 PM2/4/20