[pkg-discuss] granular configuration of http_proxy
Tim Foster
I'm looking to allow users to selectively use web proxies with pkg(1),
letting them set different web proxies for different URIs, and perhaps
none for others. Here's the approach I was going to take, comments
would be most welcome.
On the server-side, we'll add a new "proxied-urls" list property to each
existing "publishers" entry returned by the publisher/0/ response.
The list contains a series of [uri, proxy] lists so that clients using
-p can discover a suggested proxy to use automatically[1] Older
clients ignore the "proxied-urls" list, so I don't think this needs a
bump of the publisher/ response version.
That property name is intentionally the same as the existing
"proxied-urls" list served by the syspub/0/ response, and I intend to
modify that to use special [uri, "syspub"] values, rather than just the
[uri, uri, ... ] values used today, indicating that the system publisher
proxy should be used.
On the client-side, when we discover that a URI is associated with a
proxy, we pass that proxy information to the transport (support for
passing a proxy already exists with the work that was done for the
system publisher)
To support granular proxies from command line, we'll add a --proxy
option to "pkg set-publisher", which, when used with any "-g" or "-m"
URIs, associates that proxy with those URIs and overrides any proxy
configuration supplied from the server.
When used with the "-p" option, it associates that proxy with all URIs
that were discovered at the address supplied.
Finally, the "pkg/system-repository" and "pkg/mirror" SMF services will
be modified to automatically proxy requests through the correct proxy
URI, as configured in the image from which they derive their
configuration. Right now, the services have a single
"config/http[s]_proxy" property set on the SMF instance that applies to
all URIs - I'd propose to have that override the --proxy settings in the
image from which each SMF service gains its configuration.
Let me know if any of the above sounds wrong, or if you've any
questions: I'm still exploring the code here, so there's time to change
tack if I've got anything askew.
cheers,
tim
[1] this ties into the pkg mirror work I'm investigating at the moment,
to allow a user to do
# pkg publisher
PUBLISHER TYPE STATUS URI
# pkg set-publisher -p http://mymirror:9091
pkg set-publisher:
Added publisher(s): extra, internal-only, on-extra, solaris,
solarisstudio, userland
where the client automatically gets configured with all publishers
served by from that mirror, with the --proxy property set to that mirror
for each URI.
_______________________________________________
pkg-discuss mailing list
pkg-d...@opensolaris.org
http://mail.opensolaris.org/mailman/listinfo/pkg-discuss
joha...@opensolaris.org
> On the server-side, we'll add a new "proxied-urls" list property to
> each existing "publishers" entry returned by the publisher/0/
> response.
>
> The list contains a series of [uri, proxy] lists so that clients
> using -p can discover a suggested proxy to use automatically[1]
> Older clients ignore the "proxied-urls" list, so I don't think this
> needs a bump of the publisher/ response version.
>
> That property name is intentionally the same as the existing
> "proxied-urls" list served by the syspub/0/ response, and I intend
> to modify that to use special [uri, "syspub"] values, rather than
> just the [uri, uri, ... ] values used today, indicating that the
> system publisher proxy should be used.
I don't have a problem with this for the system publisher, but in any
other situation I don't agree with letting the server specify the proxy
for the client. First, how would the client contact the server if it
didn't already know the proxy configuration? Second, the publisher may
be in a faraway place, and unable to make a good generalization about
configuration for the clients. Third, this would seem to setup a very
simple man-in-the-middle attack. IMO, this should be configured on the
client side for non system publishers.
> On the client-side, when we discover that a URI is associated with a
> proxy, we pass that proxy information to the transport (support for
> passing a proxy already exists with the work that was done for the
> system publisher)
>
> To support granular proxies from command line, we'll add a --proxy
> option to "pkg set-publisher", which, when used with any "-g" or
> "-m" URIs, associates that proxy with those URIs and overrides any
> proxy configuration supplied from the server.
A few questions about this. Should users environment variables override
this setting? Currently libcurl honors the environment variables unless
the CURLOPT_PROXY is set. In that case, the proxy option takes
precedence.
The current code will not proxy requests to localhost unless
CURLOPT_PROXY is set. Do you expect customers to have a longer list of
hosts not to proxy? Have you thought about how you want to manage this?
Do you plan on supporting authentication to the proxy? Users can
specify a username and password in the proxy string, but if we want that
to be more secure, you'll have to figure out how to store the
credentials safely and then use CURLOPT_PROXYUSERPWD or
CURLOPT_PROXYUSER and CURLOPT_PROXYPASSWORD.
What types of proxies do you plan to support? Curl lets you choose
multiple types. There's an option to do this, but in 7.21.7, you can
specify the type in the URL instead. Have you thought about how you
want to make this visibile to the user? Should we upgrade libcurl >=
7.21.7?
> Finally, the "pkg/system-repository" and "pkg/mirror" SMF services
> will be modified to automatically proxy requests through the correct
> proxy URI, as configured in the image from which they derive their
> configuration. Right now, the services have a single
> "config/http[s]_proxy" property set on the SMF instance that applies
> to all URIs - I'd propose to have that override the --proxy settings
> in the image from which each SMF service gains its configuration.
The idea here is that pkg will be setting properties in the SMF
service's instance?
-j
Tim Foster
> On Tue, Mar 13, 2012 at 12:59:41PM +1300, Tim Foster wrote:
>> The list contains a series of [uri, proxy] lists so that clients
>> using -p can discover a suggested proxy to use automatically[1]
> I don't have a problem with this for the system publisher, but in any
> other situation I don't agree with letting the server specify the proxy
> for the client.
Ok,
> First, how would the client contact the server if it
> didn't already know the proxy configuration? Second, the publisher may
> be in a faraway place, and unable to make a good generalization about
> configuration for the clients. Third, this would seem to setup a very
> simple man-in-the-middle attack. IMO, this should be configured on the
> client side for non system publishers.
Those are very excellent arguments against including this support.
The main reason I wanted this, was to allow the pkg/mirror service
configure clients automatically (where we explicitly want clients who've
configured their publishers via the mirror service to use the mirror
itself as their http proxy: we assume that clients can reach the mirror
and that the mirror is the preferred way of contacting those repositories)
But given those arguments (particularly the 3rd) I'm happy to drop the
automatic proxy support. I guess users of the mirror would just as
easily be able to do:
# pkg set-publisher -p http://mirror --proxy http://mirror
Side-tracking a bit, with this work, it would also be nice if pkg(1) was
able to take duplicate URIs so long as they have different proxy values,
something like:
# pkg set-publisher -g http://ipkg.us.oracle.com/solaris11/dev/ solaris
# pkg set-publisher -g http://ipkg.us.oracle.com/solaris11/dev/ \
--proxy=http://mirror solaris
PUBLISHER TYPE STATUS URI
solaris origin online proxy://http://ipkg.us.oracle.com/solaris11/dev/
solaris origin online http://ipkg.us.oracle.com/solaris11/dev/
letting transport work out which was faster, at risk of blurring the
reason why users wanted proxying in the first place (because the above
example assumes the client can reach the repository directly, as well as
through the proxy).
Personally, I'm in it for the performance gain when updating my 3
Solaris boxes here in NZ from repositories in .sfbay. It'll be much
nicer when they all go through the same proxy :-)
>> To support granular proxies from command line, we'll add a --proxy
>> option to "pkg set-publisher", which, when used with any "-g" or
>> "-m" URIs, associates that proxy with those URIs and overrides any
>> proxy configuration supplied from the server.
>
> A few questions about this. Should users environment variables override
> this setting? Currently libcurl honors the environment variables unless
> the CURLOPT_PROXY is set. In that case, the proxy option takes
> precedence.
>
> The current code will not proxy requests to localhost unless
> CURLOPT_PROXY is set. Do you expect customers to have a longer list of
> hosts not to proxy? Have you thought about how you want to manage this?
I think user environment variables ought to override any values set for
any publishers in the image.
> Do you plan on supporting authentication to the proxy? Users can
> specify a username and password in the proxy string, but if we want that
> to be more secure, you'll have to figure out how to store the
> credentials safely and then use CURLOPT_PROXYUSERPWD or
> CURLOPT_PROXYUSER and CURLOPT_PROXYPASSWORD.
> What types of proxies do you plan to support? Curl lets you choose
> multiple types. There's an option to do this, but in 7.21.7, you can
> specify the type in the URL instead. Have you thought about how you
> want to make this visibile to the user? Should we upgrade libcurl>=
> 7.21.7?
Urgh, I hadn't thought of those, thanks.
I think for both of these, Apache as used by the system repository is
going to be the limiting factor.
Apache can only support http and https proxies, and the ProxyRemote
directives used there don't allow authentication at all at the moment[1]
We could potentially support authenticating proxies only if there were
no zones on the system, or come up with restrictions on which
repositories could be supported by the system publisher (ie. those which
didn't use authenticating proxies), but that's really unpleasant, I
think - I'd rather only support basic proxies for now.
>> Finally, the "pkg/system-repository" and "pkg/mirror" SMF services
>> will be modified to automatically proxy requests through the correct
>> proxy URI, as configured in the image from which they derive their
>> configuration. Right now, the services have a single
>> "config/http[s]_proxy" property set on the SMF instance that applies
>> to all URIs - I'd propose to have that override the --proxy settings
>> in the image from which each SMF service gains its configuration.
>
> The idea here is that pkg will be setting properties in the SMF
> service's instance?
Not quite. At present, the system-repository SMF service takes most of
its configuration from the image whose repositories it provides access to.
However, since any proxies normally used to access those repositories
aren't configured in the image, admins must manually set those in the
SMF instance instead.
With this change, the system-repository would respect any --proxy
settings found in the image, applying those to its Apache configuration,
unless the user has explicitly set the "config/http[s]_proxy" SMF
property. Does that make sense?
cheers,
tim
[1] https://issues.apache.org/bugzilla/show_bug.cgi?id=40808
https://issues.apache.org/bugzilla/show_bug.cgi?id=37355
- there was some movement on the bug late last year, with a patch
available, but it hasn't made it in to either 2.2 or 2.4 yet.