Recomended GOPROXY (or not) for Open Source projects?

[josvazg]

When working on internal company projects, it makes sense to use a company wide GO Proxy assuring that all go dependency code is available and immutable. But when you move to an Open Source project, you cannot longer use such private proxy.

I wonder what is the best practice recommendation for Open Source projects.

For instance, reading about https://proxy.golang.org/ is says:

> Why did a previously available module become unavailable in the mirror?

> proxy.golang.org does not save all modules forever. 

Which means, your project may not compile anymore if someone pulls one of your dependencies and proxy.golang.org decides to drop it.

When you read that, you may decide to just track the vendor/ folder in your repo and forget about proxies for OSS projects.

What is the recommendation from the Go community about this?

- Are there public go proxies can be used for OSS projects ensuring you will never lose any dependency?

- Is https://goproxy.io/ giving such guarantee maybe?

- Should we just vendor and forget about Go-Proxies for Open Source?

Thanks,

Jose

[David Finkel]

On Fri, Jun 25, 2021 at 1:23 PM josvazg <jos...@gmail.com> wrote:

When working on internal company projects, it makes sense to use a company wide GO Proxy assuring that all go dependency code is available and immutable. But when you move to an Open Source project, you cannot longer use such private proxy.

I wonder what is the best practice recommendation for Open Source projects.

I can't comment about whether anything specifically is a "best practice", but here are my thoughts:

 

For instance, reading about https://proxy.golang.org/ is says:

> Why did a previously available module become unavailable in the mirror?

> proxy.golang.org does not save all modules forever. 

Which means, your project may not compile anymore if someone pulls one of your dependencies and proxy.golang.org decides to drop it.

The next line in that FAQ section mentions specific reasons why proxy.golang.org wouldn't persist a module version in its cache. (e.g. licences it doesn't recognize as letting it keep/distribute copies of the code)

This seems like a non-issue if you are careful about your dependencies (and specifically, their licenses (which seems to be easy to tell now that pkg.go.dev will flag unrecognized licenses)).

When you read that, you may decide to just track the vendor/ folder in your repo and forget about proxies for OSS projects.

What is the recommendation from the Go community about this?

- Are there public go proxies can be used for OSS projects ensuring you will never lose any dependency?

- Is https://goproxy.io/ giving such guarantee maybe?

- Should we just vendor and forget about Go-Proxies for Open Source?

Now that Go Modules is relatively mature, please don't use vendoring in open source projects unless you have a really good reason.

Vendoring makes it impossible for the rest of us to tell whether there are modifications to those dependencies and blows up your repo's size at the same time. (at least without some really complicated diffing)

Thanks,

Jose

--
You received this message because you are subscribed to the Google Groups "golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/golang-nuts/47c445ca-884d-49b5-8357-aeae8802e937n%40googlegroups.com.

[josvazg]

The problem with dependencies is that you can control what you depend on directly, but not what those dependencies depend on.

I understand what you are basically saying is https://proxy.golang.org should be safe as long as all your dependencies, including transitive ones, have proper licenses. Still, if you want to ensure your open source project will always compile, vendoring still seems like the simplest way forward.

> Vendoring makes it impossible for the rest of us to tell whether there are modifications to those dependencies and blows up your repo's size at the same time. (at least without some really complicated diffing)

Indeed, all true. All specially important to consider if you are writing a library rather than a final program such as a CLI.

That is why it would be great if there were a public proxy infrastructure that would blindly retain ANY code seen forever, the same way a private proxy would. Because if that does not exist, the incentives for people to do the right thing are against the odds, IMHO.

On the other hand, such an infrastructure may lead to go open source projects not grooming their dependencies properly. Using unlicensed code, even indirectly, should be considered a liability even before such code might actually disappear. That aside of the fact that such public infra is given away for free and potentially growing storage needs without control. Tough call I guess.

Thanks,

Jose

[josvazg]

Maybe the answer is something like:

- DO NOT vendor in OSS go library repos. Use go proxies, or at least, let library consumers use go proxies with your library and do not force your deps on them.

- Have the CLI/compilable repo do the vendoring.

  - Even better, keep all your repos unvendored and a separate repo "for insurance" doing the vendoring for your project or projects. In case something goes south.

An "insurance repo" can be fully automated, running only on releases of your consumable repos. That avoids you looking at huge diffs of dependency changes for no good reason.

This is not as simple as just vendor on your same repo, but on the flip side makes your exported repos more trustworthy and easier to consume.

Are there better ways to have the cake and eat it too?

WDYT?

Thanks,

Jose

[Brian Candler]

If we're talking about an OSS project here, then having one of the dependencies vanish is "just" a case of replacing that dependency in the project.  That is, it's part of normal project maintenance, in the same way that if a severe security issue were discovered in a dependency, you'd have to update that dependency.  The same would be required if its build scripts didn't work with a newer version of Go (the Go compatibility promise applies to your code but not to the tooling and CLI interfaces)

If you're flinging an OSS project into the wild with no intention of maintaining it, then you're assuming that anyone who cares about it will fork it and maintain it themselves.

Speaking personally: if I were using someone else's OSS project, which was (a) obviously unmaintained and (b) depended on some third-party application which had been withdrawn from proxies, then I would be highly suspicious of it.  I'd have to weigh up the balance of maintaining that code myself, versus using something else or writing my own.

If the withdrawn dependency were still available in a frozen proxy or vendor directory, that wouldn't sway much in its favour.  It means I'd then have to choose whether to fork and maintain that dependency myself as well - or cut out the dependency altogether.  And as others have said, if the dependency were withdrawn due to licensing issues, then I'd obviously be at risk if I continued to use it.

Put another way: if the code is OSS, but it depends on something that's not OSS and freely available, then can the code claim to be called OSS in the first place?

[josvazg]

I do agree with all that. All very good points indeed.

What about prevention?

It can happen that a project looks very active and trustworthy when you chose to depend on it. But later on, it decides to take on new dependencies that are less reliable, or starts being less and less maintained. If that happens I am not sure how you would notice in advance, unless you actively check all your deps manually... which in my experience will not happen in most cases in real life. So in the end it is likely that you might run into trouble on the worst possible moment. For instance, your project is in maintenance mode, it does not get a lot of new features but needs to be keep up to date, etc. A dep disappears right after you need to send a security patch ASAP, and suddenly your SLA is in trouble.

So still you might need or want some insurance stronger than a public go proxy to recover the latest copy you were using and start your own fork.

BTW, any tooling you recommend to check your dependencies health automatically? Meaning checking licenses, activity stats, and anything that might make you detect you might want to revisit or fork a dependency before it actually becomes a liability?

Or maybe you should use something like https://dependabot.com/ to keep an eye on your deps in an automated way.

Thanks,

Jose

[Axel Wagner]

On Mon, Jun 28, 2021, 09:33 josvazg <jos...@gmail.com> wrote:

I do agree with all that. All very good points indeed.

What about prevention?

It can happen that a project looks very active and trustworthy when you chose to depend on it. But later on, it decides to take on new dependencies that are less reliable, or starts being less and less maintained. If that happens I am not sure how you would notice in advance, unless you actively check all your deps manually... which in my experience will not happen in most cases in real life. So in the end it is likely that you might run into trouble on the worst possible moment. For instance, your project is in maintenance mode, it does not get a lot of new features but needs to be keep up to date, etc. A dep disappears right after you need to send a security patch ASAP, and suddenly your SLA is in trouble.

So still you might need or want some insurance stronger than a public go proxy to recover the latest copy you were using and start your own fork.

I'm a bit confused about your use case TBH. On the one hand, you seem to posit an unstaffed Open Source project in maintenance mode, which can't afford to run a proxy. On the other hand you talk about having an SLA (like, not even just SLOs). That doesn't really jibe. 

Note that it's completely possible to run your own proxy, which keeps all of your dependencies indefinitely. *If* any of them vanish from public proxies, your users can then point their GOPROXY variable at your own and still build. Like,there is nothing magical about the public Go module proxy.

Of course running your own proxy is non-trivial effort. So, for most open source projects, my recommendation would be to not worry about it, use the public proxy and to just not make any legally binding promises that your stuff will work in the indefinite future. On fact, pretty much all free licenses have a clause to that effect. That's how I would expect open source to work.

The public proxy is a free service, that works well for the ecosystem as a whole, even with the restrictions on applicable licenses and the lack of an official SLA. If you want stricter promises than the ones you yet for free, you have to run your own or pay someone to do it for me - that seems like a no-brainer to me. And if you want to make promises about your service level, you obviously need to make the necessary investment to keep them - or not make them in the first place.

To view this discussion on the web visit https://groups.google.com/d/msgid/golang-nuts/17edab03-a467-4b70-b2bb-62446e3ed03dn%40googlegroups.com.