Weird error when trying to fetch a module.

[Bruno Albuquerque]

It is likely that I am doing something stupid but as I am out of ideas, here goes nothing.

I pushed an initial Go flatgeobuf implementation here:

github.com/flatgeobuf/flatgeobuf/src/go

This directory in the repository has a go.mod file with the following contents:

module github.com/flatgeobuf/flatgeobuf/src/go
go 1.20
require github.com/google/flatbuffers v22.11.23+incompatible

Just in case it might be relevant, the actual repository is github.com/flatgeobuf/flatgeobuf, src/go is a subdirectory there.

If I try to use this module, it fails. For example:

# go get -u github.com/flatgeobuf/flatgeobuf/src/go
go: downloading github.com/flatgeobuf/flatgeobuf v0.0.0-20230914202020-25c11d75fe28
go: module github.com/flatgeobuf/flatgeobuf@upgrade found (v0.0.0-20230914202020-25c11d75fe28), but does not contain package github.com/flatgeobuf/flatgeobuf/src/go

What am I missing?

-Bruno

[Lenny Kneller]

Hi Bruno,

This command works `github.com/flatgeobuf/flatgeobuf/src/go@master`.

I wonder if `@master` is needed because the repo has no packages published?

There's more info about pseudo-versions here

-Lenny-

[Jim Idle]

It’s an absolute nightmare to publish modules from subdirectories. I gave up for ANTLR and created a new repo with the source code in the root directory. 

But the art is that you need to create a tag named with the subdirectory. I was able to make that work until we needed a v{n} tag. Which as far as I can tell, just doesn’t work. 

I found that go get will duplicate the entire repo anyway, will create a pseudo tag from the commit, will download after every commit anyway, even if the go source wasn’t updated, and neither was the tag… the list goes on.  I looked at the source code for this part of go - it seems the authors wrestle with this on every release. There are open issues around it too. I can see how hairy the issues are and I am surprised this does not come up more often - when I first asked here about it, it was crickets all the way to the bottom. Also, the import will need to have your subdirectories in the path, which looks a bit wonky even if it works. 

So basically. Don’t do this as you are doing it, even though, on paper, go does not impose any particular structure on your repo beyond a few basic things such as excluding testdata, internal, and so on. 

Keep your go source at the root of the repo and tag that. If you have a choice, keep non-go artifacts in another repo, otherwise place that code in sub-directories, not the go code. 

If you have not created a tag, the go will create a pseudo version for you, which is its default. That’s ok, but it’s difficult to know what source level you are at from the git commit hash. 

I recommend using git flow methodology to make this stuff easier, though there is nothing forcing you to do so of course. 

--
You received this message because you are subscribed to the Google Groups "golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/golang-nuts/98b7c5b7-5d5b-4a84-8237-b099dd623a53n%40googlegroups.com.

[Brian Candler]

> Keep your go source at the root of the repo and tag that

Or would it be OK just to put go.mod at the root of the repo, containing

module github.com/flatgeobuf/flatgeobuf

- and leave the rest of the source where it is?

I believe that will still allow someone to import or get

github.com/flatgeobuf/flatgeobuf/src/go

(not that that's a pretty import path). But I don't know if it will behave any differently to what the OP sees today.

[Jim Idle]

The go.mod at the root was one thing that might work, but it will cause the entire repo to be pulled in so the tag will still have to be sec/go/ etc. 

--

You received this message because you are subscribed to the Google Groups "golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/golang-nuts/9638fac0-caa9-491e-bfb2-1d5e51df678en%40googlegroups.com.

[Jan Mercl]

On Fri, Sep 15, 2023 at 10:31 AM 'Jim Idle' via golang-nuts <golan...@googlegroups.com> wrote:

> The go.mod at the root was one thing that might work, but it will cause the entire repo to be pulled in so the tag will still have to be sec/go/ etc.

tl;dr: Putting a go.mod in the repository root has no additional costs for the repo users.

I was curious if the entire repo would be pulled, that would surprise me. Turns out that's not the case.

----
0:jnml@e5-1650:/tmp/mod$ ls -la
total 500
drwxr-xr-x   2 jnml jnml   4096 Sep 15 10:47 .
drwxrwxrwt 164 root root 487424 Sep 15 10:49 ..
-rw-r--r--   1 jnml jnml    937 Sep 15 10:45 go.mod
-rw-r--r--   1 jnml jnml   5461 Sep 15 10:45 go.sum
-rw-r--r--   1 jnml jnml     83 Sep 15 10:45 main.go
0:jnml@e5-1650:/tmp/mod$ cat go.mod
module example.com/mod

go 1.21.1

require modernc.org/ccgo/v4 v4.0.0

require (
        github.com/dustin/go-humanize v1.0.1 // indirect
        github.com/google/uuid v1.3.0 // indirect
        github.com/kballard/go-shellquote v0.0.0-20180428030007-95032a82bc51 // indirect
        github.com/mattn/go-isatty v0.0.16 // indirect
        github.com/remyoudompheng/bigfft v0.0.0-20230129092748-24d4a6f8daec // indirect
        golang.org/x/mod v0.11.0 // indirect
        golang.org/x/sys v0.9.0 // indirect
        golang.org/x/tools v0.10.0 // indirect
        lukechampine.com/uint128 v1.2.0 // indirect
        modernc.org/cc/v3 v3.41.0 // indirect
        modernc.org/cc/v4 v4.14.2 // indirect
        modernc.org/ccgo/v3 v3.16.15 // indirect
        modernc.org/gc/v2 v2.3.0 // indirect
        modernc.org/libc v1.24.1 // indirect
        modernc.org/mathutil v1.6.0 // indirect
        modernc.org/memory v1.7.0 // indirect
        modernc.org/opt v0.1.3 // indirect
        modernc.org/strutil v1.2.0 // indirect
        modernc.org/token v1.1.0 // indirect
)
0:jnml@e5-1650:/tmp/mod$ cat main.go
package main

import "modernc.org/ccgo/v4"

func main() {
        println(&ccgo.Task{})
}
0:jnml@e5-1650:/tmp/mod$ go clean -cache -modcache -testcache ; sudo rm -rf ~/pkg ~/.cache/go-build/
0:jnml@e5-1650:/tmp/mod$ go build -v -x |& grep 'ccgo.*\.zip'
# get https://proxy.golang.org/modernc.org/ccgo/v4/@v/v4.0.0.zip
# get https://proxy.golang.org/modernc.org/ccgo/v4/@v/v4.0.0.zip: 200 OK (0.111s)
# get https://proxy.golang.org/modernc.org/ccgo/v3/@v/v3.16.15.zip
# get https://proxy.golang.org/modernc.org/ccgo/v3/@v/v3.16.15.zip: 200 OK (0.032s)
0:jnml@e5-1650:/tmp/mod$
----

Note: ccgo/v4 depends on ccgo/v3 but that's not relevant to this experiment.

-j

[Bruno Albuquerque]

Hey Lenny.

Generally speaking @master is not required but it appears it is in this case due to it being in a subdirectory. I understand the issues involved (changes outside the module would affect its version anyway) but I am fine with that as long as the module can be used. It looks to me this is actually broken.

-Bruno

--
You received this message because you are subscribed to the Google Groups "golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/golang-nuts/98b7c5b7-5d5b-4a84-8237-b099dd623a53n%40googlegroups.com.

[Bruno Albuquerque]

Unfortunately that would be difficult for me to do as it is not my project. I just implemented the Go version of it so I had to follow the standard they set (implementations are inside src/[language]. But thanks for the detailed summary of the issues. I will try the tag approach and see how it goes.

-Bruno

To view this discussion on the web visit https://groups.google.com/d/msgid/golang-nuts/CAGPPfg-YT%2BQ4B1R3Wyj%2BQqcfHyeiTJErnZwVKWhMy1%3DzhCQJgg%40mail.gmail.com.

[Bruno Albuquerque]

That might be doable. I will discuss this with the maintainers. Thanks for the suggestion.

It is still a shame that workarounds like this need to exist.

-Bruno

--

You received this message because you are subscribed to the Google Groups "golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/golang-nuts/9638fac0-caa9-491e-bfb2-1d5e51df678en%40googlegroups.com.

[Bruno Albuquerque]

Ok, this is promising. Thanks for testing it.

-Bruno

--
You received this message because you are subscribed to the Google Groups "golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/golang-nuts/CAA40n-UbYxMCJ8T6QMApV1xme9C9JNR3EjRjh_CT0xoyoU1Wew%40mail.gmail.com.

[Bruno Albuquerque]

So, I am still curious about something. Flatbuffers, which is a project I depend on, does have its go code in a subdirectory and I can go get it:

➜  ~ go get -x github.com/google/flatbuffers/go
# get https://proxy.golang.org/github.com/@v/list
# get https://proxy.golang.org/github.com/google/flatbuffers/@v/list
# get https://proxy.golang.org/github.com/google/flatbuffers/go/@v/list
# get https://proxy.golang.org/github.com/google/@v/list
# get https://proxy.golang.org/github.com/@v/list: 404 Not Found (0.156s)
# get https://proxy.golang.org/github.com/google/@v/list: 404 Not Found (0.230s)
# get https://proxy.golang.org/github.com/google/flatbuffers/go/@v/list: 200 OK (0.236s)
# get https://proxy.golang.org/github.com/google/flatbuffers/@v/list: 200 OK (0.236s)
# get https://proxy.golang.org/github.com/google/flatbuffers/go/@latest
# get https://proxy.golang.org/github.com/google/flatbuffers/go/@latest: 404 Not Found (0.023s)
go: added github.com/google/flatbuffers v23.5.26+incompatible

As comparison, here is what I get for flatgeobuf:

➜  ~ go get -x github.com/flatgeobuf/flatgeobuf/go
# get https://proxy.golang.org/github.com/@v/list
# get https://proxy.golang.org/github.com/flatgeobuf/@v/list
# get https://proxy.golang.org/github.com/flatgeobuf/flatgeobuf/@v/list
# get https://proxy.golang.org/github.com/flatgeobuf/flatgeobuf/go/@v/list
# get https://proxy.golang.org/github.com/@v/list: 404 Not Found (0.124s)
# get https://proxy.golang.org/github.com/flatgeobuf/@v/list: 404 Not Found (0.126s)
# get https://proxy.golang.org/github.com/flatgeobuf/flatgeobuf/@v/list: 200 OK (0.130s)
# get https://proxy.golang.org/github.com/flatgeobuf/flatgeobuf/@latest
# get https://proxy.golang.org/github.com/flatgeobuf/flatgeobuf/@latest: 200 OK (0.023s)
# get https://proxy.golang.org/github.com/flatgeobuf/flatgeobuf/go/@v/list: 404 Not Found (4.152s)
go: module github.com/flatgeobuf/flatgeobuf@upgrade found (v0.0.0-20230914202020-25c11d75fe28), but does not contain package github.com/flatgeobuf/flatgeobuf/go

Is the difference simply that flatbuffers has an associated tag and flatgeobuf currently does not?

While we are at it, I can not find the go.mod for flatbuffers! It is not in the root of the repository and also not inside the go subdir where the code resides. Is there some special casing for this project in place?

-Bruno

[Brian Candler]

> I can not find the go.mod for flatbuffers! It is not in the root of the repository and also not inside the go subdir where the code resides

Presumably relies on this:

https://go.dev/ref/mod#non-module-compat

See also

https://stackoverflow.com/questions/67030123/can-a-go-module-have-no-go-mod-file

[Sean Liao]

Modules in subdirectories should just work, tags are not necessary.

I see there is a recent commit to correct the module name, that was necessary.

As for your most recent invocation, the module path was wrong (missing a /src/ component)

$ go get github.com/flatgeobuf/flatgeobuf/src/g...@v0.0.0-20230914202020-25c11d75fe28
go: downloading github.com/flatgeobuf/flatgeobuf v0.0.0-20230914202020-25c11d75fe28
go: downloading github.com/flatgeobuf/flatgeobuf/src/go v0.0.0-20230914202020-25c11d75fe28
go: added github.com/flatgeobuf/flatgeobuf/src/go v0.0.0-20230914202020-25c11d75fe28
go: added github.com/google/flatbuffers v22.11.23+incompatible

- sean

--
You received this message because you are subscribed to the Google Groups "golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/golang-nuts/CAEd86TwVCmcVBfKasTKYv804oJeJJ_dEoQ1SVPaJ2_nEhO0RUw%40mail.gmail.com.

[Bruno Albuquerque]

You are right but the error was just at this run. Before I was actually using the right path and it was failing. Now it *IS* working apparently. Maybe it had something to do with the module cache?

Thanks for the heads up.

To view this discussion on the web visit https://groups.google.com/d/msgid/golang-nuts/CAGabyPp%2BaNKujM6Hb-J7WF9EYQH2aMvaGrYjhhjxrqdgXjc_Yw%40mail.gmail.com.