Go Create

[Kevin Chadwick]

Inspired by:

"https://twitter.com/rakyll/status/1348723364894961666"

"Yet another couple of hours spent on go.mod and go.sum. If I was giving #golang
a first try today, I'd just give up and look for another option. What are we
doing as a community to address this?"

Couldn't go have a folder based Go create command akin to Flutter create. That
creates a project in the current folder perhaps with a unique 128 bit ID or just
local folder based. IOW Hybrid between Go Path and Go mod (cd based, no GOPATH
env). Maybe there could be a Go import for adding another folder as a lib to the
.mod file?

Go Mod with require keyword still seems unintuitive to me, even with a unique
domain. I still get network calls that I don't want, too. Sometimes I am in one
folder down and get odd failure messages WRT domains, before realising too.

[Artur Vianna]

I don't think this tweet is very true, it took me many months of learning Go to start learning about go.mod. I think that's not the first thing one learns about a language, you can just 'vim main.go' and begin experimenting.

How do you propose this tool will work?

--
You received this message because you are subscribed to the Google Groups "golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/golang-nuts/452f4e85-7ddb-9a55-f888-0a93d37f502c%40gmail.com.

[Axel Wagner]

On Tue, Jan 12, 2021 at 10:26 AM Kevin Chadwick <m8il...@gmail.com> wrote:

Inspired by:

"https://twitter.com/rakyll/status/1348723364894961666"

"Yet another couple of hours spent on go.mod and go.sum. If I was giving #golang
a first try today, I'd just give up and look for another option. What are we
doing as a community to address this?"

Couldn't go have a folder based Go create command akin to Flutter create. That
creates a project in the current folder perhaps with a unique 128 bit ID or just
local folder based. IOW Hybrid between Go Path and Go mod (cd based, no GOPATH
env). Maybe there could be a Go import for adding another folder as a lib to the
.mod file?

I don't understand what you are suggesting here. ISTM creating a new Go project folder is

git init

go mod init <module-path>

I guess you *could* safe the `git init` part, but is that really worth the added complexity?

FTR, I don't think what you are trying to solve here is a problem Jaana is having. But I also don't really know where her problems lie - I assume it has something to do with what the ecosystem around Kubernetes does, which I don't really use, so that I don't encounter them.

 

I still get network calls that I don't want, too.

AIUI adding `-mod readonly` should address most of that and I think I remember that this has become the default in more situations in go 1.16. So, your problems might just go away soon.

Sometimes I am in one
folder down and get odd failure messages WRT domains, before realising too.

[Kevin Chadwick]

On 1/12/21 9:43 AM, Axel Wagner wrote:
> git init
> go mod init <module-path>
>
> I guess you *could* safe the `git init` part, but is that really worth the added
> complexity?

Sorry I meant replace keyword, not require in the original mail. Though
requiring a domain name is equally unintuitive.

I guess the point is that a domain is good for one scenario and env variables
shouldn't need editing for GoPATH. If env editing isn't needed for go path then
they should be put on an equal footing, maybe GoPATH put back in primary
position. I've never really used GoPATH much but I'm not sure a vendor folder
should be required.

I usually init git from within vscode afterwards. I wonder if doing it pre would
save me any trouble.

AFAICT go mod init works great, if you use remote git.

If you work locally then you have to manually edit the .mod files. For me with a
replace directive to bring in general libraries. That is the likely scenario of
a new user. I have noticed that GoPATH is more prominent again in the docs.

Generelly Gos docs and methods are far simpler and better than Dart/Flutters but
in this regard. Flutter create followed by opening the folder in an IDE with
import files as required, is a lot simpler. I believe they may have some complex
cyclic dependency checking though.

> FTR, I don't think what you are trying to solve here is a problem Jaana is having.

True, but I think maybe more pertinent to the issue of onboarding. After all if
you are using Kubernetes, then you aren't exactly opting for simplicity anyway.

[Kevin Chadwick]

On 1/12/21 9:39 AM, Artur Vianna wrote:
> I don't think this tweet is very true, it took me many months of learning Go to
> start learning about go.mod. I think that's not the first thing one learns about
> a language, you can just 'vim main.go' and begin experimenting.
>

Fair enough. I had forgotten how easy that is tbh. I did that too, though my
experience was importing packages quite quickly.

I guess it's the

Call code in an external package
"https://golang.org/doc/tutorial/getting-started"

That is a little hairy.

> How do you propose this tool will work?

Not sure exactly. I guess it would ask for remote Git or local Folder/Git option
and maybe folders to import. Then create a .mod and file with main() and those
imports. Maybe I should just be using filesystem links instead of replace
though. I might have tried that and ran into issues though. I should probably
just set up a local git server and /etc/hosts

It just seems that manually editing a .mod with replace entries as well as
imports in the project files to a non existent url location is a bolt on and not
a thoughtful solution.

[Axel Wagner]

Hi,

I'm having trouble understanding and separating your core problems and what you are suggesting to solve them.

The solution to describe, sounds to me a lot like what `-mod vendor` achieves. I don't use that very often, so I don't really understand its shortcomings - so it might be helpful if you could describe the differences and what shortcomings of vendor mode you are trying to address.

You also talk about onboarding - but I don't understand how the rest of your message relates to that. The replace-directive is an advanced feature that newcomers to the language don't need to be concerned with, as far as I can tell. It's really only useful to overwrite the dependencies of your dependencies with patched versions - which is pretty fickle and IMO not something newcomers should attempt.

And they also don't need to a domain name in their module path - `go mod init foo` is all you need to start coding. Having a discoverable moudle path is only necessary once you actually want to publish it and make it discoverable to other people. This *is* indeed a bit of a learning curve, but note that other approaches also require you to set up code-hosting, for example, to make your code discoverable to other people. I think it's a fine onboarding-workflow to push code to github and learn that your code is now discoverable as `github.com/Username/project`. I don't think that's particularly easier or harder to communicate, than having to choose a globally unique name on crates.io (or wherever) and reference it by that.

On Tue, Jan 12, 2021 at 11:41 AM Kevin Chadwick <m8il...@gmail.com> wrote:

I guess the point is that a domain is good for one scenario and env variables
shouldn't need editing for GoPATH. If env editing isn't needed for go path then

they should be put on an equal footing, maybe GoPATH put back in primary
position. I've never really used GoPATH much but I'm not sure a vendor folder
should be required.

I really don't understand what you are saying here. But FTR, I just tried, in a clean user (i.e. no environment variables set):

mkdir foo

go mod init foo

And I could start writing code immediately.

If you work locally then you have to manually edit the .mod files. For me with a
replace directive to bring in general libraries. That is the likely scenario of
a new user.

I strongly disagree that a new user commonly or likely needs `replace`. Much less for common libraries. New users, most likely, will either a) work in an open source environment, in which they can just `go get` whatever library they need, or b) work in a closed source environment, in which they should use whatever workflow appropriate for their org.

There is one relevant scenario, I think, which is an org wanting to newly adopt Go in a closed-source setting. I agree that there is friction here and a certain learning curve. The ideal solution would require setting up an internal module hosting solution - though, FTR, that internal module hosting solution can be as simple as an NFS share. Either way, `replace` shouldn't be a part of the solution *either way*.

There is also the scenario where you have your code slit over many repositories and need to make changes to one, to change another. But IMO, this is a hairy problem in *any* language. I'd be interested to see, how other languages solve this better.

 

I have noticed that GoPATH is more prominent again in the docs.

Generelly Gos docs and methods are far simpler and better than Dart/Flutters but
in this regard. Flutter create followed by opening the folder in an IDE with
import files as required, is a lot simpler. I believe they may have some complex
cyclic dependency checking though.

> FTR, I don't think what you are trying to solve here is a problem Jaana is having.

True, but I think maybe more pertinent to the issue of onboarding. After all if
you are using Kubernetes, then you aren't exactly opting for simplicity anyway.

--
You received this message because you are subscribed to the Google Groups "golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/golang-nuts/930e33cd-be9d-c2d7-bc8c-ffa84e66e234%40gmail.com.

[Axel Wagner]

On Tue, Jan 12, 2021 at 1:20 PM Kevin Chadwick <m8il...@gmail.com> wrote:

I guess it's the

Call code in an external package
"https://golang.org/doc/tutorial/getting-started"

That is a little hairy.

What is hairy about this? To me, that seems like pretty simple steps:

1. Find the library you need and add an import

2. Run "go mod init foo"

3. Write code that uses the library

 

Again, would be interesting how other languages can make this simpler, because I genuinely can't imagine.

> How do you propose this tool will work?

Not sure exactly. I guess it would ask for remote Git or local Folder/Git option
and maybe folders to import. Then create a .mod and file with main() and those
imports. Maybe I should just be using filesystem links instead of replace
though. I might have tried that and ran into issues though. I should probably
just set up a local git server and /etc/hosts

It just seems that manually editing a .mod with replace entries as well as
imports in the project files to a non existent url location is a bolt on and not
a thoughtful solution.

--
You received this message because you are subscribed to the Google Groups "golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/golang-nuts/a3c5862e-460a-796f-40cc-4d605c8c4e36%40gmail.com.

[Kevin Chadwick]

On 1/12/21 12:33 PM, Axel Wagner wrote:
> You also talk about onboarding - but I don't understand how the rest of your
> message relates to that. The replace-directive is an advanced feature that
> newcomers to the language don't need to be concerned with, as far as I can tell.
> It's really only useful to overwrite the dependencies of your dependencies with
> patched versions - which is pretty fickle and IMO not something newcomers should
> attempt.

Maybe I need to look into -mod vendor as I didn't know that existed. Otherwise
disconnect your internet and then import a folder of local .go files that you
use in multiple projects?

[Kevin Chadwick]

On 1/12/21 12:37 PM, Axel Wagner wrote:
> Again, would be interesting how other languages can make this simpler, because I
> genuinely can't imagine.

The issue is that I believe go avoids files in order to speed up compilation.
Which I am thankful for. It's just I don't believe the replace directive was
meant for this and yet upon googling. It is the only way that I have found. mod
vendor doesn't seem to do what I want.

They are fairly similar otherwise.

Flutter create
open lib/main.dart
add import to file locations in any dart files where you wish to access that
particular code (like in most languages)
add package to pubspec.yaml
IDE asks to run flutter pub get
add import referencing package where needed

vs

go mod init
edit .mod with weird replace line that links a domain to a file location
add import matching replace directive domain/url
go get package
add import for the package

As I said. I should probably just set up a local git server and sync up that
way. symlinks were only an issue in some programs and only when using Windows too.

[Axel Wagner]

On Tue, Jan 12, 2021 at 1:49 PM Kevin Chadwick <m8il...@gmail.com> wrote:

Maybe I need to look into -mod vendor as I didn't know that existed. Otherwise
disconnect your internet and then import a folder of local .go files that you
use in multiple projects?

Ah, I see. I agree that this could be simpler. I just filed https://github.com/golang/go/issues/43646 to that effect. I also describe a workaround there: Set `GOPROXY=file://$GOPATH/pkg/mod/cache/download`.

Note, that this is essentially one of the workflows suggested here (which I would consider the canonical documentation for the "eng org wants to onboard Go" case):

https://golang.org/ref/mod#private-modules

In particular, the "private proxy serving all modules" section says:

There are several existing implementations of GOPROXY servers that may be used this way. A minimal implementation would serve files from a module cache directory and would use go mod download (with suitable configuration) to retrieve missing modules.

i.e. a minimal version of this enterprise proxy would be an NFS share that gets mounted on the clients, set via GOPROXY and downloaded into via `go mod download`.

--
You received this message because you are subscribed to the Google Groups "golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/golang-nuts/448d0354-65b5-4d0c-1566-88603fd9052e%40gmail.com.

[Axel Wagner]

On Tue, Jan 12, 2021 at 2:49 PM Kevin Chadwick <m8il...@gmail.com> wrote:

On 1/12/21 12:37 PM, Axel Wagner wrote:
> Again, would be interesting how other languages can make this simpler, because I
> genuinely can't imagine.

The issue is that I believe go avoids files in order to speed up compilation.

I don't believe it is.

 

It's just I don't believe the replace directive was meant for this and yet upon googling.

I tend to agree. I still do not understand why you feel the need to use it. It has two fairly limited uses: 1. overwrite the dependency of one of your dependencies, e.g. to use a fork which contains critical patches you need. And 2. work with a local fork of a dependency, for experimentation (e.g. to find a bug or try out an unpublished API). Neither of these are things that should be encountered by newcomers to the language - or very often at all.

If you are using replace as part of your regular workflow, I do agree that you are probably misusing it. It's just hard to understand why, from what you said so far (barring the "I don't have an internet connection and want to use something from my module cache" case, which I find surprising, if that's part of your regular workflow).

It is the only way that I have found. mod
vendor doesn't seem to do what I want.

They are fairly similar otherwise.

Flutter create
open lib/main.dart
add import to file locations in any dart files where you wish to access that
particular code (like in most languages)
add package to pubspec.yaml
IDE asks to run flutter pub get
add import referencing package where needed

vs

go mod init
edit .mod with weird replace line that links a domain to a file location
add import matching replace directive domain/url
go get package
add import for the package

As I said. I should probably just set up a local git server and sync up that
way. symlinks were only an issue in some programs and only when using Windows too.

--
You received this message because you are subscribed to the Google Groups "golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/golang-nuts/8417466a-deb0-98f6-3635-3240278dbcc7%40gmail.com.

[Jim Ancona]

I have a third use case for replace:

3. Simultaneously developing a library and client for that library. For example, if I have a library to access a network service or piece of hardware and want to extend its API, I find that working on the client along with the library helps me iterate to a good API. I use replace to get that to work. But I think that newcomers to the language are unlikely to run into that case either.

Jim

To view this discussion on the web visit https://groups.google.com/d/msgid/golang-nuts/CAEkBMfF6axmeHxYMecoHfR%2BhBVdto_froMGVswRYaoKYWodHnQ%40mail.gmail.com.

[Rob Pike]

Jim, I used replace this way just yesterday. It works fine, but is not easily discoverable as a process and it took me a minute or two to figure it out. Also, one must be sure to remove the replace directive before releasing the code, which is an easy detail to forget.

The same trick (and required care) applies when patching a library.

Modules work well but you do need to read and understand the documentation. The posts in blog.golang.org are good. You can start here: https://blog.golang.org/v2-go-modules

-rob

To view this discussion on the web visit https://groups.google.com/d/msgid/golang-nuts/CAKYY9A%2BKrPwHSK00VjQcsyZoKxumaHWoR-pUi1Jww7hfRtijBA%40mail.gmail.com.

[Kevin Chadwick]

>If you are using replace as part of your regular workflow, I do agree
>that
>you are probably misusing it. It's just hard to understand why, from
>what
>you said so far (barring the "I don't have an internet connection and
>want
>to use something from my module cache" case, which I find surprising,

"https://github.com/golang/go/issues/37554"

A folder import in whichever file has main() would be nice, but nvm.

I guess it's best to throw up a http server. Would be nice if the docs made it clear that http is the simplest solution though. As filesystem use is expected to have first class support by most engineers.

If only I knew a good language for writing a web server 😉? (Joke)

[Wojciech S. Czarnecki]

Dnia 2021-01-13, o godz. 07:22:52
Rob Pike <r...@golang.org> napisał(a):

>Also, one must be sure to remove the replace directive before
> releasing the code, which is an easy detail to forget.

I'd humbly summon https://github.com/golang/go/issues/26640 here.
Opened at the modules advent, still it sees no love from the Team.

--
Wojciech S. Czarnecki
<< ^oo^ >> OHIR-RIPE

[Randall O'Reilly]

There's also this: https://github.com/golang/go/issues/37755 -- just retain GOPATH mode

Although, now that I've been using modules more, and my various interdependent packages are more stable, I am more sympathetic to something like #26640 instead of retaining GOPATH. Anyway, having SOME kind of good solution to this distributed workflow scenario would be great!

- Randy

> --
> You received this message because you are subscribed to the Google Groups "golang-nuts" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts...@googlegroups.com.

> To view this discussion on the web visit https://groups.google.com/d/msgid/golang-nuts/20210113145309.61f42301%40xmint.

[Kevin Chadwick]

> Note, that this is essentially one of the workflows suggested here (which I
> would consider the canonical documentation for the "eng org wants to onboard
> Go" case):
> https://golang.org/ref/mod#private-modules

I have realised a goproxy may be needed rather than a http server as I expect a
domain with https, will be required for security despite being LOCALLY secure.

(reminds me of web filesystem apis and PWAs)

Can you run a GO proxy on http without TLS, such as tunnelled through SSH?

> though, FTR, that internal module hosting solution can be as simple as an NFS
share.

How would something like this work. Do I mount an sftp folder into my projects
'main' despite the library being project 'library'. Would I rename the 'library'
package as 'main' also and use it like an 'extension' to packages with main()?

Thanks, Kc

[Kevin Chadwick]

>
>> Note, that this is essentially one of the workflows suggested here (which I
>> would consider the canonical documentation for the "eng org wants to onboard
>> Go" case):
>> https://golang.org/ref/mod#private-modules
>
> I have realised a goproxy may be needed rather than a http server as I expect a
> domain with https, will be required for security despite being LOCALLY secure.
>

>> Jim, I used replace this way just yesterday. It works fine, but is not easily
>> discoverable as a process and it took me a minute or two to figure it out.

It works well mostly but seems buggy at times. Saying functions aren't defined
with illogical fixes like connecting the internet and "updating transitive
dependencies. I'm starting to reluctantly come to the conclusion that I will
simply have to spend the time setting up a GoProxy, to be in a more tested place.

Having to occasionally delete empty lines and put them back seemingly randomly
after a build tag is enough to deal with already.

Maybe I should dos2unix all my files or something?

[David Riley]

On Jan 12, 2021, at 5:40 AM, Kevin Chadwick <m8il...@gmail.com> wrote:
>
> On 1/12/21 9:43 AM, Axel Wagner wrote:
>> git init
>> go mod init <module-path>
>>
>> I guess you *could* safe the `git init` part, but is that really worth the added
>> complexity?
>

> <snip>

> I usually init git from within vscode afterwards. I wonder if doing it pre would
> save me any trouble.
>
> AFAICT go mod init works great, if you use remote git.
>
> If you work locally then you have to manually edit the .mod files. For me with a
> replace directive to bring in general libraries. That is the likely scenario of
> a new user. I have noticed that GoPATH is more prominent again in the docs.

Truthfully, "git init" isn't required at all if you're not going to share. You can name your module whatever you want in go.mod and as long as you're consistent with it elsewhere in your local project, it works fine.

I understand the need for the domain as part of the module namespacing, but it does make a certain assumption that all of these things are going to be on resolvable domains, which isn't true for everyone. This is particularly a hassle when building with additional local resources so that you can test new library changes without having to do a full push to a remote git server, update go.sum, etc. And I've still got a bit of a bee in my bonnet about the lack of workable relative imports, but I'm not interested in relitigating that right now.

What I wonder might be useful is if go.[mod,sum] were able to pick up additional local overrides (e.g. a go.mod.local file to do the replace directives to point to local variants, with go.sum.local taking the corresponding changes so they don't wind up in go.sum). We've run into this problem quite a bit at work where people point at their local workspaces and then accidentally commit the modified go.mod (often when they've made an actual change to go.mod that needs to be committed and they forget to temporarily remove it). That way, go.mod.local would be easy enough to add to .gitignore, .dockerignore, etc. so that it doesn't escape into the wild.


- Dave

[Kevin Chadwick]

On 2/3/21 2:46 PM, David Riley wrote:
> I understand the need for the domain as part of the module namespacing, but it does make a certain assumption that all of these things are going to be on resolvable domains, which isn't true for everyone.

I have a resolvable domain and run a mail server with a lets encrypt certificate
but there are other issues.

Why would I trust DNS
Why would I setup letsencrypt
What if we have a minimal internet use developer machine policy

Replace isn't used by the masses and works ok but it isn't as well tested as
demonstrated by various gopls and other bugs. How do we know the issue we are
having with gopls is caused by being in a minority with replace or build tags.
Actually a giant package main is likely better tested.

It was said that replace probably should not be used in a daily workflow, yet
there are numerous issues stating to do so. It's possible that the main mistake
was in giving the stdlib special treatment! If Go needed to do something, likely
others will too.

I am currently looking into this proxy as Athens that uses Docker is likely not
cross platform because Docker is not cross platform. The docs like this thread
seem to assume people will use github!

"https://github.com/goproxy/goproxy"

[Kevin Chadwick]

On 2/3/21 5:14 PM, Kevin Chadwick wrote:
> Why would I trust DNS
> Why would I setup letsencrypt
> What if we have a minimal internet use developer machine policy

https://roberto.selbach.ca/go-proxies



Oh, this will be annoying.

https://golang.org/ref/mod#module-proxy

A module proxy must always serve the same content for successful responses for
$base/$module/$version.mod and $base/$module/$version.zip queries. This content
is cryptographically authenticated using go.sum files and, by default, the
checksum database.



Aha, this may be the key

The go command may download module source code and metadata directly from a
version control repository. Downloading a module from a proxy is usually faster,
but connecting directly to a repository is necessary if a proxy is not available
or if a module's repository is not accessible to a proxy (frequently true for
private repositories). Git, Subversion, Mercurial, Bazaar, and Fossil are
supported. A version control tool must be installed in a directory in PATH in
order for the go command to use it.

To download specific modules from source repositories instead of a proxy, set
the GOPRIVATE or GONOPROXY environment variables. To configure the go command to
download all modules directly from source repositories, set GOPROXY to direct.
See Environment variables for more information.

[Kevin Chadwick]

I am surprised this did not come up in the thread. I'm not sure if I never knew this was possible or I have forgotten during my time using replace. Turns out, that you can import a folder within the same module folder as mod identifier/package and it is automatically imported by gopls when you reference that package, which never worked with replace.

This allows existing version control tools to be used across languages too. Just pull the folder into the root module and away you go.

Being able to reference external folders such as with replace is nice but causes issues with gopls and mod updates and possibly erroneous build messages.

I believe there are some documentation bugs. The following does not mention this or link to "how to write go code".

https://golang.org/doc/tutorial/getting-started

The following alludes to this functionality, but could make this possibility, a lot clearer.

https://golang.org/doc/code.html