go mod download fails on docker for 1.20 (but not for 1.18)

[Joao Miguel Ferreira]

Hello all,

My docker build is ok with FROM golang:1.18-alpine, FROM golang:1.18-buster and FROM golang:1.18-bullseye images. The application works. All is fine on the 3 cases.

But the docker build fails with FROM golang:1.20-alpine, FROM golang:1.20-buster and FROM golang:1.20-bullseye images.

This is the error message I get:

go: github.com/davecgh/go-...@v1.1.1: Get "https://proxy.golang.org/github.com/davecgh/go-spew/@v/v1.1.1.mod": dial tcp [2a00:1450:4003:801::2011]:443: connect: cannot assign requested address

It happens during go mod download. It happens when building on Debian11, Debian12 and Ubuntu22 hosts.

I have tried many things including going inside the container and downloading the module with wget. It works fine. Maybe something changed from 1.18 to 1.20 related to ipv6.

I would like to be able to do the build with one golang 1.20 image. But I can not understand what is wrong. Can you help clarify that error and how to overcome it?

[Jim Idle]

My guess is that IPV6 is disabled in your containers. If you use 

docker run -it xxx yyy -c "ip -6 addr show dev eth0; ip -6 route show"

Then you can find out if that is the case.

--
You received this message because you are subscribed to the Google Groups "golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/golang-nuts/6629812f-534e-4d01-a46c-ef5bdeac4c12n%40googlegroups.com.

[TheDiveO]

? base system distribution?

? do you have IPv6 connectivity at all?

? which docker version?

[Joao Miguel Ferreira]

Hello Jim and Harald,

please find below the answers to your questions. if anything else could help, please let me know

about the container configuration for ipv6, it seems that there is no such configuration for containers running from the base images (1.18, 1.20, bullseye seems to not have the ip command installed)

$ docker run -it golang:1.18-buster sh -c "ip -6 addr show dev eth0; ip -6 route show"
$ docker run -it golang:1.18-bullseye sh -c "ip -6 addr show dev eth0; ip -6 route show"
sh: 1: ip: not found
sh: 1: ip: not found
$ docker run -it golang:1.18-alpine sh -c "ip -6 addr show dev eth0; ip -6 route show"
$ docker run -it golang:1.20-buster sh -c "ip -6 addr show dev eth0; ip -6 route show"
$ docker run -it golang:1.20-bullseye sh -c "ip -6 addr show dev eth0; ip -6 route show"
sh: 1: ip: not found
sh: 1: ip: not found
$ docker run -it golang:1.20-alpine sh -c "ip -6 addr show dev eth0; ip -6 route show"

I add here a sample of the same command for ipv4 just to clarify:

$ docker run -it golang:1.20-buster sh -c "ip -4 addr show dev eth0; ip -4 route show"
52: eth0@if53: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default  link-netnsid 0
    inet 172.17.0.2/16 brd 172.17.255.255 scope global eth0
       valid_lft forever preferred_lft forever
default via 172.17.0.1 dev eth0
172.17.0.0/16 dev eth0 proto kernel scope link src 172.17.0.2
 

On Mon, May 22, 2023 at 7:55 AM TheDiveO <harald....@gmx.net> wrote:

? base system distribution?

? do you have IPv6 connectivity at all?

? which docker version?

b) about my host and docker version and connectivity

I am using Debian11 with apt provided docker; docker version is 20.10.5. I am also using 2 VMs with Debian 12 and Ubuntu 22 and I have the same problem there.

I have an ipv6 address on the host network interface and I can ping6 to localhost and to google, also. Please find more details below:

jmf@deb11tp:~$ ping6 -c 3 localhost
PING localhost(localhost (::1)) 56 data bytes
64 bytes from localhost (::1): icmp_seq=1 ttl=64 time=0.043 ms
64 bytes from localhost (::1): icmp_seq=2 ttl=64 time=0.035 ms
64 bytes from localhost (::1): icmp_seq=3 ttl=64 time=0.054 ms

jmf@deb11tp:~$ ping6 -c 3 www.google.com
PING www.google.com(mad41s13-in-x04.1e100.net (2a00:1450:4003:80e::2004)) 56 data bytes
64 bytes from mad41s13-in-x04.1e100.net (2a00:1450:4003:80e::2004): icmp_seq=1 ttl=58 time=16.6 ms
64 bytes from mad41s13-in-x04.1e100.net (2a00:1450:4003:80e::2004): icmp_seq=2 ttl=58 time=16.9 ms
64 bytes from mad41s13-in-x04.1e100.net (2a00:1450:4003:80e::2004): icmp_seq=3 ttl=58 time=16.4 ms

jmf@deb11tp:~$ ip addr show dev wlp5s0

3: wlp5s0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether 18:56:80:2f:33:48 brd ff:ff:ff:ff:ff:ff
    inet 192.168.1.65/24 brd 192.168.1.255 scope global dynamic noprefixroute wlp5s0
       valid_lft 1832sec preferred_lft 1832sec
    inet6 2001:8a0:ffa9:9800:a681:184e:ee1b:1800/64 scope global dynamic noprefixroute
       valid_lft 89765sec preferred_lft 89765sec
    inet6 fe80::d3e8:44e:9bfa:c6f8/64 scope link noprefixroute

       valid_lft forever preferred_lft forever

I try to be clear: this happens during "docker build", on the "go mod download" and in some cases also on the "go mod tidy". it does not happen on any of the 1.18 base images, whatever host I use. It happens on all 1.20 base images. 

Thank you for your attention

João

 

--
You received this message because you are subscribed to the Google Groups "golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/golang-nuts/31dcfd80-fc3a-444b-879f-8b0ca75a3a54n%40googlegroups.com.

[TheDiveO]

The thing that finally caught my eye is the "connect: cannot assign  requested address" part in the error message. You can search for it in combination with golang. I'm not exactly clear but this could be related to running out of socket or port resources, maybe in combination with changes in how many parallel HTTP requests are fired off by go mod tidy and go mod download. The overall communication/socket "load" of your system might also be a factor to look at.   

[Eduardo Luís]

Did anyone found a fix for this? I still have this problem after 6 months.
It allows me to up a single container but it seems to have a 5 minute cooldown before I can up another one without an error, which takes a long time on projects with multiple services.

Error example on Ubuntu22 with golang:1.20-alpine image
```
go: github.com/alexliesenfeld/hea...@v0.8.0: Get "https://proxy.golang.org/github.com/alexliesenfeld/health/@v/v0.8.0.mod": dial tcp [2a00:1450:4003:80e::2011]:443: connect: cannot assign requested address
```

[Dmitry Anderson (4nd3rs0n)]

I'm having the same problem. For now I just vendor modules to a project modules and run without installing anything from the container, but also would like to find a fix.

On Monday, May 22, 2023 at 2:29:08 AM UTC+2 Joao Miguel Ferreira wrote:

[Sebastiaan van der Meulen]

Same issue here without solution. Hoping anyone finds one

Op zondag 14 januari 2024 om 22:53:54 UTC+1 schreef Dmitry Anderson (4nd3rs0n):

[TheDiveO]

Are you still using Debian 11 and the outdated Debian docker.io package with Docker 18? What happens when you use a recent Docker, either 24.x or hot-off-the-press 25.0.1? And then build using a Go-Alpine base image? Do you still use Debian's broken Docker seccomp profile...?

I'm on an IPv6 uplink and in all these years never seen this issue. I'm not using Debian due to its totally outdated cloud/container packages, but instead a *ubuntu host with docker-ce packages, using alpine base images for building ... and never such an issue. So my bet would be on Debian and/or your ISP. From my bad experience with many nightmarish late hour diagnosis sessions with oversees my bet is on Debian, especially the LTS long-term-broken variants.

[sprynger]

In my case it might be an ISP problem, since this only happens to me when I work from home. At work, the exact same project builds all at once without any issues.

[TheDiveO]

carrier-grade NAT?

[Bryan C. Mills]

In https://go.dev/issue/52545 we saw failures in the cmd/go tests on Google Cloud VMs due to a combination of a small number of available NAT ports and port exhaustion from TIME_WAIT connections, but the failure mode there was different (timed-out git commands rather than "cannot assign requested address").

If the problem is the total number of open connections, rather than connections in the TIME_WAIT state, you could try setting `GODEBUG=cmdgonetlimit=4` or similar in your environment, although I don't think that will prevent the accumulation of idle connections. It would probably be helpful to figure out exactly how many connections you have open (and what state they're in) when the error occurs.

(When I was investigating #52545 I wrote a little `ss` wrapper to monitor connection states; see https://github.com/bcmills/cmd/blob/main/watchnet/main.go.)