Struggling with go mod, micro-repos and private forks

[Shaun Crampton]

Hi All,

My team own a large product that sprawls across a lot of repos.  We also manage a private, commercial fork and we occasionally do new development in private and then merge across to open source.  While a monorepo might have been easier to manage, we're stuck with what we've got.  We've recently moved to go mod and it just seems like we're constantly fighting the tool.  I'm hoping you can either suggest some good workflows or maybe improve the tool!

I think the biggest problem we have is when working with private repos.   What I want to express to the tool is 

My module requires commit abcd1234 (or version v1.2.3) of dependency x/y/z

Look for any instances of dependency x/y/z in git repo g...@github.com:ourfork instead of the default.

However, what I can express to the tool is

My module requires version ??? of dependency x/y/z

Replace  x/y/z @ version ??? with <other module> @ abcd1234

 

This throws up a couple of problems:

• What should version ??? be?  It's only there to be replaced, which seems like a bit of a smell.  
• If I set it to the commit ID it gets resolved and I have to change three places in the file each time I move the pin.
• If I set it to a particular version, that seems misleading.  
• I guess I can set it to v0.0.0, but again that seems misleading.
• There's nowhere to specify the details of the repo (e.g. connection/auth type), all that has to magically work according to my git settings and the defaults aren't great for private repos (which we access over ssh).

We also just ran into the new GOPRIVATE env var and had to update all our builds to use that.  Couldn't that just fall back to the private behaviour if it gets a cache miss, it seems over-engineered to require an explicit exception!

Another issue we've had is making a build mode where we can build against local copies of the dependencies for quick development.  We can add replace directives to point to local directories, which is part of the puzzle, but it's hard to do that "just for one build".  Not sure what we're looking for here; maybe an optional go.mod override file that can be passed in for just one build?  

-Shaun

[Bryan Mills]

Thanks for the feedback.

The good news is that we're aware of (and planning to address) most of these pain points; the bad news is that we haven't been able to get to most of them yet.

Detailed responses inline.

On Tuesday, October 22, 2019 at 6:42:05 AM UTC-4, Shaun Crampton wrote:

Hi All,

My team own a large product that sprawls across a lot of repos.  We also manage a private, commercial fork and we occasionally do new development in private and then merge across to open source.  While a monorepo might have been easier to manage, we're stuck with what we've got.  We've recently moved to go mod and it just seems like we're constantly fighting the tool.  I'm hoping you can either suggest some good workflows or maybe improve the tool!

I think the biggest problem we have is when working with private repos.   What I want to express to the tool is 

My module requires commit abcd1234 (or version v1.2.3) of dependency x/y/z

Look for any instances of dependency x/y/z in git repo g...@github.com:ourfork instead of the default.

 

I believe this is https://golang.org/issue/28176.

It's an intriguing idea, but problematic if the dependencies are specified as pseudo-versions, because the commit hashes in a fork in general will not match the hashes in the upstream module. We're still exploring alternatives for this use-case.

However, what I can express to the tool is

My module requires version ??? of dependency x/y/z

Replace  x/y/z @ version ??? with <other module> @ abcd1234

 

This throws up a couple of problems:

• What should version ??? be?  It's only there to be replaced, which seems like a bit of a smell.  
• If I set it to the commit ID it gets resolved and I have to change three places in the file each time I move the pin.
• If I set it to a particular version, that seems misleading.  
• I guess I can set it to v0.0.0, but again that seems misleading.

Currently it should be `v0.0.0-00010101000000-000000000000`.

We're also rethinking this behavior for `replace` in general; see https://golang.org/issue/33370.

• There's nowhere to specify the details of the repo (e.g. connection/auth type), all that has to magically work according to my git settings and the defaults aren't great for private repos (which we access over ssh).

See https://golang.org/cmd/go/#hdr-Remote_import_paths.

In general we expect that either you have an HTTPS server to locate the repo for a given import path (credentials can be stored in a `.netrc` file; see also https://golang.org/issue/26232), or you include an explicit VCS extension somewhere in the import path (and have corresponding credentials configured per-user in your VCS).

We also just ran into the new GOPRIVATE env var and had to update all our builds to use that.  Couldn't that just fall back to the private behaviour if it gets a cache miss, it seems over-engineered to require an explicit exception!

The checksum database and proxy fallback are designed to provide integrity by default.

Without making the opt-out explicit, an origin could serve one set of sources (and checksum) to you, and another version to someone else, and simply refuse to serve anything at all in response to queries from the checksum database (so that the entry would always be missing), and no one would be the wiser.

Another issue we've had is making a build mode where we can build against local copies of the dependencies for quick development.  We can add replace directives to point to local directories, which is part of the puzzle, but it's hard to do that "just for one build".  Not sure what we're looking for here; maybe an optional go.mod override file that can be passed in for just one build?

You mean like https://golang.org/issue/34506? 🙂 

 

[Shaun Crampton]

The good news is that we're aware of (and planning to address) most of these pain points; the bad news is that we haven't been able to get to most of them yet.

Great to see, thanks!

 

I think the biggest problem we have is when working with private repos.   What I want to express to the tool is 

My module requires commit abcd1234 (or version v1.2.3) of dependency x/y/z

Look for any instances of dependency x/y/z in git repo g...@github.com:ourfork instead of the default.

 

I believe this is https://golang.org/issue/28176.

It's an intriguing idea, but problematic if the dependencies are specified as pseudo-versions, because the commit hashes in a fork in general will not match the hashes in the upstream module. We're still exploring alternatives for this use-case.

So, the problem that pseudo-versions are trying to solve is ordering in the face of unordered commit hashes?  For my use case, the way I want this to work is that, once a dependency is pinned to my private fork, it's as if the public repo didn't exist.  In 99% of cases, all the pins that I'm working with are my organisation's repos anyway so once we pin to a private fork that's what we want to use everywhere, transitively, even in downstream repos.  In extreme cases, even the version numbering scheme in the private fork may be different so there's not necessarily any relation between the open source v2.3.0 and the private v2.3.0 tags.

Our minority use case is forking some public repo to make a fix.  In that case, a perfect tool would have a way to say "we need a version >= open source version that must have (fix) commit abcd in its commit history"  Then, once our fix is upstreamed, the open source version will become a candidate.  I suspect that's too much to ask though!

 

However, what I can express to the tool is

My module requires version ??? of dependency x/y/z

Replace  x/y/z @ version ??? with <other module> @ abcd1234

 

Currently it should be `v0.0.0-00010101000000-000000000000`.

We're also rethinking this behavior for `replace` in general; see https://golang.org/issue/33370.

Good to know that there's a good value to put there.  Would be nice if it could be made shorter, though, I'll have to look that up.  

• There's nowhere to specify the details of the repo (e.g. connection/auth type), all that has to magically work according to my git settings and the defaults aren't great for private repos (which we access over ssh).

See https://golang.org/cmd/go/#hdr-Remote_import_paths.

In general we expect that either you have an HTTPS server to locate the repo for a given import path (credentials can be stored in a `.netrc` file; see also https://golang.org/issue/26232), or you include an explicit VCS extension somewhere in the import path (and have corresponding credentials configured per-user in your VCS).

Maybe I'm just out-of-date, go now tries git+ssh as well as https://, was that new in v1.13?  Since we only use github, that should be good enough for me.

We also just ran into the new GOPRIVATE env var and had to update all our builds to use that.  Couldn't that just fall back to the private behaviour if it gets a cache miss, it seems over-engineered to require an explicit exception!

The checksum database and proxy fallback are designed to provide integrity by default.

Without making the opt-out explicit, an origin could serve one set of sources (and checksum) to you, and another version to someone else, and simply refuse to serve anything at all in response to queries from the checksum database (so that the entry would always be missing), and no one would be the wiser.

Fair enough, I suppose.  From a selfish point of view, making the go tooling auto-trust github.com private repos would be nice :-)  I can't easily do that with a single entry in GOPRIVATE since whether a repo is private or not can only be guessed by whether you need to use credentials or ssh to access it.

 

Another issue we've had is making a build mode where we can build against local copies of the dependencies for quick development.  We can add replace directives to point to local directories, which is part of the puzzle, but it's hard to do that "just for one build".  Not sure what we're looking for here; maybe an optional go.mod override file that can be passed in for just one build?

You mean like https://golang.org/issue/34506? 🙂 

 

Sounds like just what we need, thanks! 

[Shaun Crampton]

I circed some of the suggestions round my team.  Sounds like others had already tried some of the suggestions with mixed results:

• Go v1.13 still has trouble authenticating to github without an "insteadof" in the config.  We use 2FA on github, which seems to make HTTPS fail in a way that doesn't fall back to git+ssh mode?
• Another team member said they tried using v0.0.0-00010101000000-000000000000 as version but it got replaced.

[Bryan C. Mills]

On Thu, Oct 24, 2019 at 5:36 AM Shaun Crampton <sh...@tigera.io> wrote:

I think the biggest problem we have is when working with private repos.   What I want to express to the tool is 

My module requires commit abcd1234 (or version v1.2.3) of dependency x/y/z

Look for any instances of dependency x/y/z in git repo g...@github.com:ourfork instead of the default.

 

I believe this is https://golang.org/issue/28176.

It's an intriguing idea, but problematic if the dependencies are specified as pseudo-versions, because the commit hashes in a fork in general will not match the hashes in the upstream module. We're still exploring alternatives for this use-case.

So, the problem that pseudo-versions are trying to solve is ordering in the face of unordered commit hashes?

Yes.

 

For my use case, the way I want this to work is that, once a dependency is pinned to my private fork, it's as if the public repo didn't exist.  In 99% of cases, all the pins that I'm working with are my organisation's repos anyway so once we pin to a private fork that's what we want to use everywhere, transitively, even in downstream repos.  In extreme cases, even the version numbering scheme in the private fork may be different so there's not necessarily any relation between the open source v2.3.0 and the private v2.3.0 tags.

In that case, you want a `replace` directive without a version on the left side. It will prune out more of your transitive dependency graph than is strictly necessary, but that's not usually a major problem.

(I'm hoping we can offer something better in the future, but there isn't enough time for it to make 1.14.)

Our minority use case is forking some public repo to make a fix.  In that case, a perfect tool would have a way to say "we need a version >= open source version that must have (fix) commit abcd in its commit history"  Then, once our fix is upstreamed, the open source version will become a candidate.  I suspect that's too much to ask though!

If you expect the upstream fix to be accepted before the next patch release, generally the procedure there is:

require example.com/some/dependency <current version>

replace example.com/some/dependency <current version> => my.fork <some version>

Then when you upgrade, you'll get a version higher than <current version> and your `replace` directive will no longer apply.

If the patch doesn't come through, then you update <current version> in the `replace` directive and try again next time.

However, what I can express to the tool is

My module requires version ??? of dependency x/y/z

Replace  x/y/z @ version ??? with <other module> @ abcd1234

 

Currently it should be `v0.0.0-00010101000000-000000000000`.

We're also rethinking this behavior for `replace` in general; see https://golang.org/issue/33370.

Good to know that there's a good value to put there.  Would be nice if it could be made shorter, though, I'll have to look that up.  

That's the version that the `go` command automatically inserts when it sees a `replace` directive without an explicit version.

Ideally we shouldn't need to insert a version at all, though.

• There's nowhere to specify the details of the repo (e.g. connection/auth type), all that has to magically work according to my git settings and the defaults aren't great for private repos (which we access over ssh).

See https://golang.org/cmd/go/#hdr-Remote_import_paths.

In general we expect that either you have an HTTPS server to locate the repo for a given import path (credentials can be stored in a `.netrc` file; see also https://golang.org/issue/26232), or you include an explicit VCS extension somewhere in the import path (and have corresponding credentials configured per-user in your VCS).

Maybe I'm just out-of-date, go now tries git+ssh as well as https://, was that new in v1.13?  Since we only use github, that should be good enough for me.

I'm not sure. I do know that 1.12 had a bug in its `.netrc` handling, which is fixed in 1.13 and might let you proceed past some step that failed previously.

[Bryan C. Mills]

On Thu, Oct 24, 2019 at 1:07 PM Shaun Crampton <sh...@tigera.io> wrote:

I circed some of the suggestions round my team.  Sounds like others had already tried some of the suggestions with mixed results:

• Go v1.13 still has trouble authenticating to github without an "insteadof" in the config.  We use 2FA on github, which seems to make HTTPS fail in a way that doesn't fall back to git+ssh mode?

We have https://golang.org/issue/26232 open for 2FA workflows in general.

In the meantime, you may need to configure a Personal Access Token to get HTTPS to work.

• Another team member said they tried using v0.0.0-00010101000000-000000000000 as version but it got replaced.

That probably means that you have a higher version requirement from some other dependency. (The `go` command automatically updates the `go.mod` file to maintain consistency.)

--
You received this message because you are subscribed to a topic in the Google Groups "golang-nuts" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/golang-nuts/Kg-X2qZQohg/unsubscribe.
To unsubscribe from this group and all its topics, send an email to golang-nuts...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/golang-nuts/930aff92-8600-4df1-8236-06b07916a57b%40googlegroups.com.

[Shaun Crampton]

We have https://golang.org/issue/26232 open for 2FA workflows in general.

In the meantime, you may need to configure a Personal Access Token to get HTTPS to work.

We use 2FA for accessing the github UI but we tend to use SSH for push and pull.  Would be easier if it tried ssh rather than bailing out.

 

• Another team member said they tried using v0.0.0-00010101000000-000000000000 as version but it got replaced.

That probably means that you have a higher version requirement from some other dependency. (The `go` command automatically updates the `go.mod` file to maintain consistency.)

But then we go round in circles, right?  The start of this discussion was that I wanted a "nil" version to put in my require statement so that I could replace it with exactly the commit that I need using a "replace".  Now, if the tool helpfully updates the require then that defeats the point of having a "nil" version in the first place.

[Bryan Mills]

On Friday, October 25, 2019 at 10:19:03 AM UTC-4, Shaun Crampton wrote:

We have https://golang.org/issue/26232 open for 2FA workflows in general.

In the meantime, you may need to configure a Personal Access Token to get HTTPS to work.

We use 2FA for accessing the github UI but we tend to use SSH for push and pull.  Would be easier if it tried ssh rather than bailing out.

 Ah, that one is https://golang.org/issue/26134.

• Another team member said they tried using v0.0.0-00010101000000-000000000000 as version but it got replaced.

That probably means that you have a higher version requirement from some other dependency. (The `go` command automatically updates the `go.mod` file to maintain consistency.)

But then we go round in circles, right?  The start of this discussion was that I wanted a "nil" version to put in my require statement so that I could replace it with exactly the commit that I need using a "replace".  Now, if the tool helpfully updates the require then that defeats the point of having a "nil" version in the first place.

If you don't care what version you're replacing, use a `replace` directive without a version and don't worry about the version that the `go` command adds in the `require` directive.