Try our new module mirror!

[Katie Hockman]

Hey Gophers!

In the blog post Go Modules in 2019, we announced our intent to provide a module mirror for accelerating Go module downloads, an index for discovering new modules, and a checksum database for authenticating module content.

We are excited to share that we are ready for developers to start alpha testing our module mirror, index, and checksum database!

Our privacy policy explains how we collect and use your information. The privacy policy for all of these services is proxy.golang.org/privacy.

The module mirror at proxy.golang.org serves the go command’s proxy protocol. To make the go command use it (when in module mode), set GOPROXY=https://proxy.golang.org.

The module index at index.golang.org serves a feed of module versions in the order they are discovered. For example, see https://index.golang.org/index?since=2019-03-04T18:00:15.161182-07:00

The module checksum database at sum.golang.org serves the URLs described in the Secure the Public Go Module Ecosystem proposal. If you use the Go 1.13 development version of the go command, setting GOPROXY=https://proxy.golang.org will automatically use the checksum database as well. To use the checksum database to protect other code downloads, set GOSUMDB=sum.golang.org explicitly. See the development version go command docs for details. If you are using Go 1.12 or earlier, you can manually check a go.sum file against the checksum database with gosumcheck:

go get golang.org/x/exp/sumdb/gosumcheck

gosumcheck /path/to/go.sum

We hope you’ll try out these new services! Note that these are in the early stages, so even though we don’t have any planned outages, you should recognize it is an alpha release and instability and bugs are expected. We’ll be actively working through these and improving our services, so please file issues if you spot them, with the title prefix “proxy.golang.org:” (or index.golang.org, or sum.golang.org). We look forward to hearing about how it’s working for you!

Cheers,

Katie Hockman

[t...@heckman.io]

Hi Katie,

Thank you for this announcement! Are these sites blocked from China like other parts of golang.org? If so, how will we make it so this subset of our community can rely on the same infrastructure as the rest of us? Are there other countries that are subject to the same blocks and can you provide that list?

I'm asking so that I can provide an answer to anyone who may come in to the Gophers Slack Workspace or GolangBridge Forum looking for help trying to use these services, while being in a country that is blocked.

Cheers!

-Tim

[marwan....@nytimes.com]

Congratulations on building all of these services! 

When will proxy.golang.org be opened sourced (if at all)?

Thanks!

[Jakub Cajka]

----- Original Message -----
> From: "Katie Hockman" <ka...@golang.org>
> To: golan...@googlegroups.com
> Sent: Monday, April 29, 2019 10:35:38 PM
> Subject: [golang-dev] Try our new module mirror!
>
> Hey Gophers!
>

> In the blog post Go Modules in 2019 <https://blog.golang.org/modules2019>,

> we announced our intent to provide a module mirror for accelerating Go
> module downloads, an index for discovering new modules, and a checksum
> database for authenticating module content.
>
> We are excited to share that we are ready for developers to start alpha
> testing our module mirror, index, and checksum database!
>
> Our privacy policy explains how we collect and use your information. The
> privacy policy for all of these services is proxy.golang.org/privacy.

Hello,

if I'm not mistaken the page that you are linking for privacy information of the proxy seems not related to it at all. It looks like some sort of generic terms that Google is using for all of its services. It is not describing what is collected and what for, how and where it is retained and stored, etc. in context of the proxy/this new google service.

Could you expand on these topics in actual go proxy context, please? What are you collecting/are you planning on collecting and what for?

>
> The module mirror at proxy.golang.org serves the go command’s proxy
> protocol. To make the go command use it (when in module mode), set GOPROXY=
> https://proxy.golang.org.
>
> The module index at index.golang.org serves a feed of module versions in
> the order they are discovered. For example, see
> https://index.golang.org/index?since=2019-03-04T18:00:15.161182-07:00
>
> The module checksum database at sum.golang.org serves the URLs described in
> the Secure the Public Go Module Ecosystem

> <https://go.googlesource.com/proposal/+/master/design/25530-sumdb.md>

With brief look on the proposal, just from the technical perspective(kind of including modules too). Little has changed from my perspective since the initial proposal. I'm still worried that I will have to disabled/de-configured or at worst case scenario even patch it (out) to make our build system in Fedora work with Go sources that we are curating/shipping/using(i.e. occasional need to carry downstream/backport patches). Leaving out for now the privacy/MITM concerns which look nearly the same(i.e. IMHO not resolved) as previously.

> proposal. If you use the Go 1.13 development version of the go command,
> setting GOPROXY=https://proxy.golang.org will automatically use the
> checksum database as well. To use the checksum database to protect other
> code downloads, set GOSUMDB=sum.golang.org explicitly. See the development
> version go command docs

> <https://tip.golang.org/cmd/go/#hdr-Module_authentication_failures> for

> details. If you are using Go 1.12 or earlier, you can manually check a
> go.sum file against the checksum database with gosumcheck

> <https://godoc.org/golang.org/x/exp/sumdb/gosumcheck>:

>
> go get golang.org/x/exp/sumdb/gosumcheck
>
> gosumcheck /path/to/go.sum
>
> We hope you’ll try out these new services! Note that these are in the early
> stages, so even though we don’t have any planned outages, you should
> recognize it is an alpha release and instability and bugs are expected.
> We’ll be actively working through these and improving our services, so

> please file issues <https://github.com/golang/go/issues/new> if you spot

> them, with the title prefix “proxy.golang.org:” (or index.golang.org, or
> sum.golang.org). We look forward to hearing about how it’s working for you!
>
> Cheers,
>
> Katie Hockman

Is there anywhere a place where I or anyone else could pull the sources and contribute to all of these new Go features/services that you are deploying/running, or so I/anyone could potentially even run my own instances of proxy.golang.org, sum.golang.org and index.golang.org and help with devel?

Thanks for any info. Sorry if I have missed some details.

JC


>
> --
> You received this message because you are subscribed to the Google Groups
> "golang-dev" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to golang-dev+...@googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.
>

[Russ Cox]

On Mon, Apr 29, 2019 at 5:44 PM <t...@heckman.io> wrote:

Thank you for this announcement! Are these sites blocked from China like other parts of golang.org?

Almost certainly yes, but that block is imposed by China, not by Google.

One thing we had to do to launch the Go home page in China at golang.google.cn was remove the ability for the playground to retrieve user-generated content (the "shared" pages). It's hard to see how a module mirror fly if playground sharing does not, but we haven't pursued that directly. For now we are concerned primarily with getting the system working, making it robust, and so on. Making the home page available was a major effort, and as I understand it that effort did not even include the Go Blog.

If so, how will we make it so this subset of our community can rely on the same infrastructure as the rest of us?

Many parts of the Go community are partly disconnected from the internet, including those in various corporate or government environments. The proxy and checksum database protocols are designed specifically to be cacheable and mirrorable, so that it is straightforward to set up an appropriate server within those restricted environments. We will be publishing more about how to set up internal proxies. (And of course there is already the Athens project at docs.gomod.io.)

  

Are there other countries that are subject to the same blocks and can you provide that list?

I'm unaware of any other countries that block Google. My current understanding is that Google does not serve golang.org and related sites (nor much else) to the US-Embargoed Countries, but I have never had occasion to test that directly. Again the hurdles to serving that traffic are largely outside the Go team's control, as much as I would love to make Go available to absolutely everyone.

I'm asking so that I can provide an answer to anyone who may come in to the Gophers Slack Workspace or GolangBridge Forum looking for help trying to use these services, while being in a country that is blocked.

Do you get that kind of question often? I would expect anyone who can connect to Gophers Slack or the GolangBridge Forum would be able to connect to golang.org too. 

Thanks.

Russ

[Russ Cox]

On Mon, Apr 29, 2019 at 11:18 PM <marwan....@nytimes.com> wrote:

When will proxy.golang.org be opened sourced (if at all)?

That server is not likely to be open-sourced directly. As you know from your work on Athens, of course, a basic module proxy is not a fundamentally complicated program. The vast majority of our code is dealing with various Google-internal serving infrastructure, not proxy details. It shells out to "go mod download" to get modules, same as Athens.

As part of working on proxy.golang.org, we've also been working on a more detailed description of expected proxy behavior (beyond 'go help goproxy'), and I expect we will release that document along with a very basic proxy of a couple hundred lines that others can adapt or use as a reference.

Best,

Russ

[Russ Cox]

On Tue, Apr 30, 2019 at 5:07 AM Jakub Cajka <jca...@redhat.com> wrote:

> Our privacy policy explains how we collect and use your information. The
> privacy policy for all of these services is proxy.golang.org/privacy.

if I'm not mistaken the page that you are linking for privacy information of the proxy seems not related to it at all. It looks like some sort of generic terms that Google is using for all of its services. It is not describing what is collected and what for, how and where it is retained and stored, etc. in context of the proxy/this new google service.

Could you expand on these topics in actual go proxy context, please? What are you collecting/are you planning on collecting and what for?

That's right - the privacy policy for the servers is currently Google's standard privacy policy, so that URL is a simple redirect. We very much want to provide more specific information in the future. When that happens, proxy.golang.org/privacy will be updated to either redirect to the more specific policy or serve it directly. That's all I can say right now.

> The module checksum database at sum.golang.org serves the URLs described in
> the Secure the Public Go Module Ecosystem
> <https://go.googlesource.com/proposal/+/master/design/25530-sumdb.md>

  With brief look on the proposal, just from the technical perspective(kind of including modules too). Little has changed from my perspective since the initial proposal. I'm still worried that I will have to disabled/de-configured or at worst case scenario even patch it (out) to make our build system in Fedora work with Go sources that we are curating/shipping/using(i.e. occasional need to carry downstream/backport patches). Leaving out for now the privacy/MITM concerns which look nearly the same(i.e. IMHO not resolved) as previously.

Yes, we still have open issues to address those various concerns. Note that right now the proxy and checksum database are still disabled by default. Even once they are enabled, all you need to do to bypass them is:

    export GOPROXY=direct; export GONOSUMDB='*'

Now that the server is available for public alpha testing I intend to circle back to trying to work out the best path forward for repackagers like Fedora and for those concerned about privacy. It's easier to have those conversations if you can show working code.

Is there anywhere a place where I or anyone else could pull the sources and contribute to all of these new Go features/services that you are deploying/running, or so I/anyone could potentially even run my own instances of proxy.golang.org, sum.golang.org and index.golang.org and help with devel?

There isn't right now, because they are tied a bit to Google infrastructure.

For proxy.golang.org, as I mentioned in my reply to Marwan, we intend to publish a short reference proxy that people can adapt as needed, and of course there is also Athens already.

For sum.golang.org, there is a reference implementation in golang.org/cl/161665 and related CLs; those will land at golang.org/x/exp/sumdb/... and eventually move to a more permanent location.

There's no reference for index.golang.org but it's little more than a seekable append-only file.

Best,

Russ

[wayneash...@gmail.com]

Just tested and working very well, apart from private modules - do you think there will eventually be support for those?

[Russ Cox]

On Wed, May 1, 2019 at 8:37 AM <wayneash...@gmail.com> wrote:

Just tested and working very well, apart from private modules - do you think there will eventually be support for those?

Private modules are not part of our plan for proxy.golang.org - it's an unauthenticated, public mirror only. We want to do more to make it easy to play nicely with private modules, though. Go 1.13 will support listing multiple proxies in GOPROXY (for people who want to put a private proxy with only private modules in front of the public proxy) and also configuring module paths that should always bypass any proxy with GONOPROXY (for people who want to fetch private modules directly).

Best,

Russ

[Aaron Schlesinger]

You can also set up a private Athens proxy to host your private code. https://docs.gomods.io has setup instructions if you’re interested.

On May 1, 2019, at 06:36, amn...@gmail.com wrote:

>> Just tested and working very well, apart from private modules - do you think there will eventually be support for those?
>

> Go 1.13 will allow GOPATH="https://proxy.golang.org/,direct"
> so that go get will fall back to direct vcs access for private modules which are not accessible to a public proxy.

[xia...@gmail.com]

for chain mainland gophers, you can have a try this open source proxy.

http://goproxy.io/

[wayneash...@gmail.com]

Thanks for that response Russ, it makes complete sense. Glad to hear that support for multiple proxies is coming - and even happier to hear about GONOPROXY which is definitely they route I'll be taking initially.

[jerome....@gmail.com]

Hello,

I read here https://twitter.com/FiloSottile/status/1123325055830769666 that this proxy allows to prevent the problem with dependency that was removed (and no longer available) when the dependency's license is compatible.

Does it exist a way to check whether the dependency is actually save on the proxy?

Thx in adv

Jérôme

[Tim Heckman]

Hi Russ,

Thank you for taking the time to reply. I'll respond inline as well.

On Tuesday, April 30, 2019 at 6:45:05 AM UTC-7, rsc wrote:

On Mon, Apr 29, 2019 at 5:44 PM <t...@heckman.io> wrote:

Thank you for this announcement! Are these sites blocked from China like other parts of golang.org?

Almost certainly yes, but that block is imposed by China, not by Google.

One thing we had to do to launch the Go home page in China at golang.google.cn was remove the ability for the playground to retrieve user-generated content (the "shared" pages). It's hard to see how a module mirror fly if playground sharing does not, but we haven't pursued that directly. For now we are concerned primarily with getting the system working, making it robust, and so on. Making the home page available was a major effort, and as I understand it that effort did not even include the Go Blog.

If so, how will we make it so this subset of our community can rely on the same infrastructure as the rest of us?

Many parts of the Go community are partly disconnected from the internet, including those in various corporate or government environments. The proxy and checksum database protocols are designed specifically to be cacheable and mirrorable, so that it is straightforward to set up an appropriate server within those restricted environments. We will be publishing more about how to set up internal proxies. (And of course there is already the Athens project at docs.gomod.io.)

 

While I think that's great we're thinking about corporate and government environments, there are plenty of individuals who would not fall in the bucket of working at a company who would be impacted by this. This is a learning I've come to by interacting directly with users on Slack and the forum. I think we need to consider all users of Go, and not just the corporate users, because as I've learned directly interacting with the users of Go (through Slack and the forum) that there are quite a few who do it outside of work. Are individual developers in China going to have the resources to run their own proxy somewhere that allows them to pull packages, including the golang.org ones? I don't know, but my gut tells me not always.

I feel strongly that this is a topic we should consider researching, and to understand what we can do to better alleviate this issue and strengthen our community.

 

Are there other countries that are subject to the same blocks and can you provide that list?

I'm unaware of any other countries that block Google. My current understanding is that Google does not serve golang.org and related sites (nor much else) to the US-Embargoed Countries, but I have never had occasion to test that directly. Again the hurdles to serving that traffic are largely outside the Go team's control, as much as I would love to make Go available to absolutely everyone.

I'm sorry to ask such a pointed question, but what have we researched or considered to solve the problem of making Go as-available to everyone as we can? Does it make sense to discuss the idea of moving the core of the Go infrastructure out to a non-profit that's unaffiliated with Google, centered in a country that doesn't follow the (arguably) frivolous trade embargoes set forth by the US Government? Who is the best person to talk to about starting this sort of effort?

If we weren't restricted by US trade embargoes, and weren't related to Google, would we be able to build a community that's inclusive to people from all countries based on their qualities as a person and not the place where they live or their ethnicity? I'm a citizen of the US, but I don't feel discrimination based on your ethnicity/country of origin is acceptable, and it feels like that's inherently what an embargo accomplishes.

Are we okay with that? Can we do more to tear down those artificial walls?

 

I'm asking so that I can provide an answer to anyone who may come in to the Gophers Slack Workspace or GolangBridge Forum looking for help trying to use these services, while being in a country that is blocked.

Do you get that kind of question often? I would expect anyone who can connect to Gophers Slack or the GolangBridge Forum would be able to connect to golang.org too. 

It's hard to say it's often, but it's a non-zero amount of time over the past year. I've answered the golang.org from China question a few times on the Slack Workspace, so it seems like we're not blocked in China like golang.org is. I don't speak Mandarin or Cantonese, so it's likely the questions are being asked in channels using those languages so I don't see them.

 

Thanks.

Russ

Thank you again for the response.

Cheers!

-Tim 

[Andrew Bonventre]

We are actively looking into solutions for users with connectivity issues such as those in China.

We will not be changing Go's governance structure or moving any ownership to an entity other than where it resides today.

Thanks

--

[amn...@gmail.com]

The multiple proxy feature seems to be get defeated by the sum.golang.org verification.
See https://github.com/golang/go/issues/31818

[Russ Cox]

On Fri, May 3, 2019 at 1:05 PM <amn...@gmail.com> wrote:

The multiple proxy feature seems to be get defeated by the sum.golang.org verification.
See https://github.com/golang/go/issues/31818

Replied there, thanks.

Russ

[perry.j...@gmail.com]

Thank you!

With this, I'll be able to get our security team (and, transitively, their PCI auditors) off my back. The last thing that I could really benefit from is something like npm's audit functionality.

-P

[marwan....@nytimes.com]

The new release of The Athens Project now proxies https://sum.golang.org by default :) 

Furthermore, Athens can also block import paths from being proxied by returning a Forbidden http status to the client ensuring that the user use GONOSUMDB on private modules.

For guides on how to do so, check out the docs here: https://docs.gomods.io/configuration/sumdb

Feel free to take a look and all feedback is welcome. 

Thanks!

On Monday, April 29, 2019 at 4:35:54 PM UTC-4, Katie Hockman wrote:

[Russ Cox]

On Thu, May 9, 2019 at 11:19 PM <marwan....@nytimes.com> wrote:

The new release of The Athens Project now proxies https://sum.golang.org by default :) 

Furthermore, Athens can also block import paths from being proxied by returning a Forbidden http status to the client ensuring that the user use GONOSUMDB on private modules.

Very nice!

Thanks.

Russ

[Aaron Schlesinger]

I'm really happy this is out in a release now :)

[Paul Jolly]

Congratulations, Katie and team!

Quick question about the delay in updating the list of versions for each module.

At the time of writing, I've just released v0.0.10
github.com/myitcv/gobin. go get github.com/myitcv/gobin in a clean
environment returns v0.0.9. Somewhat to be expected because I'm
assuming there's a periodic refresh of modules?

If however I do go get github.com/myitcv/go...@v0.0.10, and then retry
go get github.com/myitcv/gobin I still get v0.0.9 for the latter.

Is there perhaps some optimisation to be had here, by updating the
list for github.com/myitcv/gobin when it sees a new version "pass
through"?

Thanks

[Brad Fitzpatrick]

I see it in https://proxy.golang.org/github.com/myitcv/gobin/@v/list now. I don't know if that list is supposed to be sorted semver-wise there, though.

Actually https://golang.org/cmd/go/#hdr-Module_proxy_protocol only says:

> GET $GOPROXY/<module>/@v/list returns a list of all known versions of the given module, one per line.

So cmd/go should do the sorting, which it does:

$ GOPROXY=https://proxy.golang.org/ go list -m --versions github.com/myitcv/gobin

go: finding github.com/myitcv/gobin v0.0.10

github.com/myitcv/gobin v0.0.1 v0.0.2 v0.0.3 v0.0.4-alpha v0.0.4-alpha.1 v0.0.4 v0.0.5 v0.0.6 v0.0.7 v0.0.8 v0.0.9 v0.0.10

To view this discussion on the web visit https://groups.google.com/d/msgid/golang-dev/CACoUkn4dxUv-p%2BKY3%2BBcJi1S4NTL0bVkji%2Bc6UJCe6HTsGeHqA%40mail.gmail.com.

[Hyang-Ah Hana Kim]

There is delay - the module proxy is refreshing the version periodically (currently ~30min) 

and also http caching is involved on the serving path. We are currently working on reducing the version discovery latency. 

Correct. /@v/list doesn't return versions in sorted order. We let the go command on the client side does the sorting.

To view this discussion on the web visit https://groups.google.com/d/msgid/golang-dev/CAFzRk03WFLcwyjqYPUf8_4dJFP_H-VJWaWKNXooWC0FBFm8kHg%40mail.gmail.com.

For more options, visit https://groups.google.com/d/optout.

--

__

[Paul Jolly]

Hi team,

As a follow up, I'm randomly seeing some calls to a module's (not
specific to a given module) /@latest endpoint be very slow to return.

I haven't yet narrowed down any particular behaviour pattern from my
side that leads to this, nor am I able to actually confirm it's
limited to the /@latest endpoint

For example I just did:

go get -x -v -m myitcv.io@latest

and the following step took ~10-15secs:

Fetching https://proxy.golang.org/myitcv.io/@latest

If I repeat the same command again it returns extremely quickly.

Any thoughts?

Thanks,


Paul

[Hyang-Ah Hana Kim]

Hi  Paul,

If data for the request is not cached in the system, the proxy goes and fetches the data. 

/@latest query is particularly interesting since we can't cache that for long time without sacrificing the data freshness.

I didn't look into the particular query closely yet and there must be other optimization opportunity on our side.

But it's likely that your first /@latest query probably triggered the refreshing logic; on the other hand, the next queries could be answered quickly because the previous result is cached.

- Hana

To view this discussion on the web visit https://groups.google.com/d/msgid/golang-dev/CACoUkn6uNGKtqJux%2B1RdS-uPrO0GMVSPHAyZDht4fVzJrYdY8Q%40mail.gmail.com.

For more options, visit https://groups.google.com/d/optout.

--

__

[Paul Jolly]

Hi Hana,

Thanks for the quick response, that makes total sense.

At least with this approach the proxy's data freshness is a function
of how frequently that module is accessed. Which I guess means for
less frequently accessed modules a higher percentage of /@latest
requests will potentially hit the slow path?

For more frequently accessed modules, do you remove the slow path from
the end user and instead do a periodic background check, always
returning the cached version to the user?

Thanks,


Paul

[roger peppe]

I wonder if it might be worth allowing the GOPROXY URL to specify update latency tolerance. eg  GOPROXY=https://proxy.golang.org?maxdelay=1h

Then you could use maxdelay=0 to force an immediate sync when you know a new branch has been pushed, or 24h if you really don't care much.

To view this discussion on the web visit https://groups.google.com/d/msgid/golang-dev/CAM7qfQLi9sGtbHr9zsnVAOWzvSa-FDTK1xtzdoFFPrvBLptU5w%40mail.gmail.com.

[Katie Hockman]

As an update, related to https://golang.org/issue/32805 and https://golang.org/issue/32879#issuecomment-507861612:
Go 1.13 includes some changes to the go command which tighten up the validation of module versions. Now that we have upgraded proxy.golang.org to use the 1.13beta1 version of the go command, we will be invalidating old module versions that are currently being served from proxy.golang.org but are no longer valid in 1.13. We don't expect any major issues from users since the request count is low for each of the module versions that will be invalidated, but please file issues against the issue tracker, prefixed "proxy.golang.org", if you encounter any problems. Thanks to everyone who has filed issues so far and given us feedback!

[Akram Ahmad]

This is a very positive development. Keep up the good work!

[Sulaiman, Marwan]

Hi there, 

Could this issue be possibly related to the recent change? https://github.com/golang/go/issues/33513

Thanks!

--
You received this message because you are subscribed to a topic in the Google Groups "golang-dev" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/golang-dev/go6_lZqew6g/unsubscribe.
To unsubscribe from this group and all its topics, send an email to golang-dev+...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/golang-dev/882fc92f-cd5a-4d40-961d-7dbe2e1c310d%40googlegroups.com.

[Katie Hockman]

Yep, https://github.com/golang/go/issues/33513 is related! I commented there for some additional context. Thanks.

You received this message because you are subscribed to the Google Groups "golang-dev" group.

To unsubscribe from this group and stop receiving emails from it, send an email to golang-dev+...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/golang-dev/CAJtS%2BrfMgsGU8LTmKD4VUCdfCSmiLZ-BnHwBegwVGyhrqgFGyA%40mail.gmail.com.

[Dave Cheney]

proxy.golang.org has started to return 410's for artefacts which were
working last week :(

https://github.com/golang/go/issues/33558

> To view this discussion on the web visit https://groups.google.com/d/msgid/golang-dev/CALvTBvd3yeC891ZGUPgPdGiS_ViGUMHHc3MWgSoc7ZETtgvHng%40mail.gmail.com.