Why does net/http ProxyFromEnvironment refuse to proxy localhost requests?

[Kekoa Vincent]

When using net/http ProxyFromEnvironment to specify proxy using HTTP_PROXY environment variable, there is a special case to prevent use of a proxy when connecting to a localhost or 127.0.0.1 service. 

Using a debugging proxy with local services is a common practice, and it was surprising to find this special case, as I can't find any logical reason why it is included. I did find it was documented (discussed in History below), but the document only describes the code and not any reason why.

This behavior is surprising, which is the worst part, because it required debugging and work to figure out why my proxy wasn't working. The proxy works great for everything until someone tries to point to localhost. I would expect to be able to use a proxy to localhost. Why not? Surely, there's a reason. Here's what I dug up.

History

Here's my best attempt at reconstructing the history behind this change.

1. Issue which prompted the change: https://github.com/golang/go/issues/1589 

It appears that this was introduced to fix tests.

@rsc:

[...] we should not use $http_proxy for 127.0.0.1.

This statement doesn't imply that this should be a universal law, but seems to apply only to this issue.

2. Commit which introduced the localhost and loopback special case: https://github.com/golang/go/commit/8dad7fec1d3c12a7ff50c2ad5178f4b77d4e48ef 

As a result of issue 1589, instead of fixing this for just the tests, proxy to localhost requests was disabled for ALL future usage of ProxyFromEnvironment.

My best guess is that it was simply a hasty decision to fix some broken tests, but had further consequences than was probably intended. I can't find any other reason for including this special case in such a widely used package as net/http.

An alternative solution may have been to use NO_PROXY="localhost 127.0.0.1" in the calling code that required such behavior.

3. Issue which raises the same question I'm asking, but results in only documentation being changed. https://github.com/golang/go/issues/7256

@bradfitz:

I don't remember why we do that. (in func useProxy)
If we have to keep it, it at least needs to be documented.

Commit with documentation change: https://codereview.appspot.com/97590043

Documentation now reads:

As a special case, if req.URL.Host is "localhost" (with or without a port number), then a nil URL and nil error will be returned.

4. Current code looks to be at https://github.com/golang/go/blob/master/src/vendor/golang_org/x/net/http/httpproxy/proxy.go#L183-L191 (net/http uses x/net/http/httpproxy as the implementation)

The special case of not proxying for localhost and loopback IPs is retained in the relocated code.

Pre-Proposal Feedback

Before I write a proposal to change- I thought I'd reach out to see if anyone knew a deeper reason why this is the case.

This is what I would propose:

1. Remove the localhost and loopback restriction for using a proxy in the net/http (and now x/net/http/httpproxy) package.

2. Fix any broken tests by using NO_PROXY environment variable (or some other equivalent mechanism).

Reasons:

1. Eliminate surprising library behavior.

2. The restriction is artificial, and prompts workarounds which shouldn't be necessary.

3. Could be serious if a user expects the proxy to be used when actually the proxy is being ignored.

Implications:

I'm not quite sure the impact of this change. More research would be needed. The obvious impact would be:

1. Any usage of ProxyFromEnvironment which relies on the special case of not proxying to localhost requests will now actually use the proxy.

As far as knowing how many people rely on this behavior, I can't say at this point. The consequence of proxying isn't well known either because that is application-specific. My gut says this would be odd to require this behavior.

Thanks,

Kekoa