Why does Go ignore HTTP_PROXY and HTTPS_PROXY if the proxy address is localhost

[Orson Cart]

From the documentation for ProxyFromEnvironment:

"As a special case, if req.URL.Host is "localhost" (with or without a port number), then a nil URL and nil error will be returned."

Can anyone explain the reasoning behind this? It rather interferes with debugging traffic using a local proxy like Fiddler.

I've seen suggestions to define an alternative hostname in /etc/hosts but as far as I can tell this doesn't work either.

[wagner riffel]

On Tue Apr 13, 2021 at 2:14 PM -03, Orson Cart wrote:
> Can anyone explain the reasoning behind this? It rather interferes with
> debugging traffic using a local proxy like Fiddler.
>

My guess here it's for preventing the remote from tricking the proxy to
make request to internal services that couldn't be reached otherwise.

> I've seen suggestions to define an alternative hostname in /etc/hosts
> but as far as I can tell this doesn't work either.
>

The code you linked doesn't seen to try that hard to block loopback
requests, I do think that an /etc/hosts entry to 127.0.0.1 should
bypass the proxy, if not I don't see any other way around other than
implementing the RoundTripper yourself, which shouldn't be hard for
such simple use.

--wagner

[Orson Cart]

Please accept my ay apologies. I'd misunderstood the documentation.

I was looking for a reason why my go application's requests aren't being sent to the proxy that I've specified using HTTP_PROXY and HTTPS_PROXY. When I saw the comment in the documentation I thought I'd found an explanation even if I didn't understand the reason for it being that way.

So now I'm left with my original problem: I have HTTP_PROXY set to http://localhost:8888 and HTTPS_PROXY set to https://localhost:8888. I also have a proxy (Fiddler) running on localhost:8888 but requests from my application appear to go direct, bypassing the proxy.

I'm perplexed :(

[wagner riffel]

On Tue Apr 13, 2021 at 7:07 PM UTC, Orson Cart wrote:
> I'm perplexed :(
>

Did you tried the /etc/hosts? I think that would do it, if not,
something like this may do the trick:

if debug {
t := http.DefaultTransport.(*http.Transport)
t.Proxy = func(r *http.Request) (*url.URL, error) {
return url.Parse("http://localhost:8888")
}
}

note: i did not tested, just shooting.

--wagner

[Orson Cart]

Thanks for the reply. 

Yes I did add an alias to the /etc/hosts file and it makes no difference. The requests still bypass the proxy.

As you point out it's possible to achieve the goal in code but I'd really like to do it without changing the code if at all possible.

I'm also just really curious now as to why the go libraries aren't honouring the environment variables.

Thanks for the input.

[Orson Cart]

So, to clarify: 

1/ I have a proxy (Fiddler) running locally on port 8888

2/ my /etc/hosts file has an entry as follows: 127.0.0.1 fiddler

3/ My HTTP_PROXY and HTTPS_PROXY environment variables are set to http://fiddler:8888 and  https://fiddler:8888 respectively

if I then issue a request via curl then that request is picked up by the proxy.

However if I make the same request from my go application, the request appears to bypass the proxy and is sent directly. I'm perplexed.

Any advice would be appreciated.

[Adrian Ho]

On 14/4/21 6:39 am, Orson Cart wrote:

So, to clarify: 

1/ I have a proxy (Fiddler) running locally on port 8888

2/ my /etc/hosts file has an entry as follows: 127.0.0.1 fiddler

3/ My HTTP_PROXY and HTTPS_PROXY environment variables are set to http://fiddler:8888 and  https://fiddler:8888 respectively

if I then issue a request via curl then that request is picked up by the proxy.

However if I make the same request from my go application, the request appears to bypass the proxy and is sent directly. I'm perplexed.

The "Name Resolution" section of https://golang.org/pkg/net/ says:

On Unix systems, the resolver has two options for resolving names. It can use a pure Go resolver that sends DNS requests directly to the servers listed in /etc/resolv.conf, or it can use a cgo-based resolver that calls C library routines such as getaddrinfo and getnameinfo.

By default the pure Go resolver is used, because a blocked DNS request consumes only a goroutine, while a blocked C call consumes an operating system thread.

That might have some bearing on your problem. To see if it's the case, as the page suggests:

export GODEBUG=netdns=cgo   # force cgo resolver

Best Regards,
Adrian

[Adrian Ho]

On 14/4/21 6:39 am, Orson Cart wrote:

So, to clarify: 

1/ I have a proxy (Fiddler) running locally on port 8888

2/ my /etc/hosts file has an entry as follows: 127.0.0.1 fiddler

3/ My HTTP_PROXY and HTTPS_PROXY environment variables are set to http://fiddler:8888 and  https://fiddler:8888 respectively

if I then issue a request via curl then that request is picked up by the proxy.

However if I make the same request from my go application, the request appears to bypass the proxy and is sent directly. I'm perplexed.

[Orson Cart]

On Wednesday, 14 April 2021 at 18:06:53 UTC+1 Adrian Ho wrote:

The "Name Resolution" section in https://golang.org/pkg/net/ says:

On Unix systems, the resolver has two options for resolving names. It can use a pure Go resolver that sends DNS requests directly to the servers listed in /etc/resolv.conf, or it can use a cgo-based resolver that calls C library routines such as getaddrinfo and getnameinfo.

By default the pure Go resolver is used, because a blocked DNS request consumes only a goroutine, while a blocked C call consumes an operating system thread.

That might have some bearing on your problem. To see if it's the case, as the page suggests:

export GODEBUG=netdns=cgo   # force cgo resolver

Best Regards,
Adrian

Thanks Adrian. For the record, I'm working on Windows.

For the record I should also admit that I've been missing the blindingly obvious. At least it's blindingly obvious to me now.

I never occurred to me before that the client instance that I'm using has a non-default Transport. The way that the transport is initialised means that it has a nil Proxy - hence whatever I set in the environment didn't matter as it was being ignored.

This thread hasn't been my finest hour! 

Thanks to all for the help.

[Jeff Peck]

Just a complete shot in the dark. Have you verified that the environment variables you set are indeed getting picked up inside of your application? Try checking the output of os.GetEnv("HTTP_PROXY"), os.GetEnv("HTTPS_PROXY"), and os.GetEnv("NO_PROXY"). Make sure that if NO_PROXY is set that it isn't overriding your local proxy.

You could also try setting the lowercase versions of these env vars:

https://github.com/golang/net/blob/master/http/httpproxy/proxy.go

func FromEnvironment() *Config {
    return &Config{
        HTTPProxy:  getEnvAny("HTTP_PROXY", "http_proxy"),
        HTTPSProxy: getEnvAny("HTTPS_PROXY", "https_proxy"),
        NoProxy:    getEnvAny("NO_PROXY", "no_proxy"),
        CGI:        os.Getenv("REQUEST_METHOD") != "",
    }
}

--
You received this message because you are subscribed to the Google Groups "golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/golang-nuts/ac6c1bae-2153-41b3-9037-110fc0eb12cfn%40googlegroups.com.

[Michael Baker]

On 14/04/2021 20:25, Jeff Peck wrote:
>
> Just a complete shot in the dark. Have you verified that the
> environment variables you set are indeed getting picked up inside of
> your application? Try checking the output of os.GetEnv("HTTP_PROXY"),
> os.GetEnv("HTTPS_PROXY"), and os.GetEnv("NO_PROXY"). Make sure that if
> NO_PROXY is set that it isn't overriding your local proxy.
>
>
> You could also try setting the lowercase versions of these env vars:
>
> https://github.com/golang/net/blob/master/http/httpproxy/proxy.go
>
> func FromEnvironment() *Config {
>     return &Config{
>         HTTPProxy:  getEnvAny("HTTP_PROXY", "http_proxy"),
>         HTTPSProxy: getEnvAny("HTTPS_PROXY", "https_proxy"),
>         NoProxy:    getEnvAny("NO_PROXY", "no_proxy"),
>         CGI:        os.Getenv("REQUEST_METHOD") != "",
>     }
> }
>

Thanks for the ideas Jeff. I already found out what the issue was though
- see my other reply.