[pkg-discuss] when to bump ImageConfig version? (and granular proxy progress report)
Tim Foster
I'm making good progress in getting us granular configuration of web
proxies, allowing some origins/mirrors use proxies set in the image, yet
still allowing others to contact repositories directly.
The current unit tests all pass with these bits now, and my manual tests
are also looking good, but I do still need to write new unit tests.
With these changes, older pkg bits reading new config files will fail to
proxy URLs, but will otherwise still function. However, I'm not sure
whether the changes here should really constitute an ImageConfig version
bump, and if so, what the consequences of that are? Perhaps a client
API bump as well?
The intent is to add two new properties, "origin_info" and
"mirror_info", each of which take a list of lists (tuples, maybe?). In
each element, we store the URI, and the proxy used to access that URI.
So in the pkg5.image config file, this looks something like:
origins = ['http://foo', 'http://bar', 'http://baz']
origin_info = [ ['http://foo', 'http://webcache:8080'],
['http://bar', 'http://www-proxy:80'],
['http://bar', 'http://www-proxy2:80'],
['http://baz', 'None'] ]
which translates to the the 'pkg publisher' output below:
timf@kura[36] pkg publisher
PUBLISHER TYPE STATUS URI
foo origin online proxy://http://foo/
baz origin online http://baz/
bar origin online proxy://http://bar/
bar origin online proxy://http://bar/
timf@kura[37] pkg publisher bar
Publisher: bar
Alias:
Origin URI: proxy://http://bar/
Proxy: http://www-proxy2:80
SSL Key: None
SSL Cert: None
Origin URI: proxy://http://bar/
Proxy: http://www-proxy:80
SSL Key: None
SSL Cert: None
Client UUID: 064cf0d2-7195-11e1-b4ab-d4bed990482b
Catalog Updated: March 19, 2012 07:28:07 AM
Enabled: Yes
We reuse the "proxy://" prefixes introduced with the system repository
work, as that seemed like the path of least surprise, though a -o option
to 'pkg publisher' would be nice to be able to display what the proxies
actually are, without the long form 'pkg publisher <pub>' output)
We allow duplicate origin/mirror RepositoryURI objects iff they use
different proxies. The thinking is, that transport could do optimization
if it has several paths to the repository. I've a bit of work yet to do
to make this work properly in client/transport/stats.py which currently
keys simply by URI alone)
Ultimately, we could the origin_info and mirror_info properties that are
stored per-origin/mirror (say to include ssl key/certs), but that's not
this webrev.
A draft of what I've got so far, which isn't ready for review, but might
be useful for context is at:
I'd appreciate any comments people might have on the approach here (even
if they're "Wait, stop - are you mad!?") In the meantime, I'll get
started on the tests.
cheers,
tim
_______________________________________________
pkg-discuss mailing list
pkg-d...@opensolaris.org
http://mail.opensolaris.org/mailman/listinfo/pkg-discuss
Danek Duvall
> With these changes, older pkg bits reading new config files will fail to
> proxy URLs, but will otherwise still function. However, I'm not sure
> whether the changes here should really constitute an ImageConfig version
> bump, and if so, what the consequences of that are? Perhaps a client API
> bump as well?
Older clients would fail on those origins, anyway, right? I don't think
there's a compatibility issue sufficient to bump either version.
> The intent is to add two new properties, "origin_info" and "mirror_info",
> each of which take a list of lists (tuples, maybe?). In each element, we
> store the URI, and the proxy used to access that URI.
>
> So in the pkg5.image config file, this looks something like:
>
> origins = ['http://foo', 'http://bar', 'http://baz']
> origin_info = [ ['http://foo', 'http://webcache:8080'],
> ['http://bar', 'http://www-proxy:80'],
> ['http://bar', 'http://www-proxy2:80'],
> ['http://baz', 'None'] ]
Why a generic name like "info" for something so specific? Either
"origin_proxies" (or just "proxies"; why have separate ones for origins and
for mirrors?) or make "origin_info" be a dictionary that maps keys (such as
"proxies") to lists of tuples pairing URLs with their proxies, which can
then later be extended to have other keys (like key/cert like you point out
later).
> which translates to the the 'pkg publisher' output below:
>
> timf@kura[36] pkg publisher
> PUBLISHER TYPE STATUS URI
> foo origin online proxy://http://foo/
> baz origin online http://baz/
> bar origin online proxy://http://bar/
> bar origin online proxy://http://bar/
I'm not thrilled with the proxy:// bit. I kinda see it for the ones which
go through the zone proxy, but I don't see why it's necessary here. I find
it confusing visually, and I don't believe there's any standard that makes
it acceptable for general use.
Danek
Tim Foster
Apologies for the late reply - your mail got lost in my inbox.
On 03/20/12 08:47 AM, Danek Duvall wrote:
> Looks exciting. Just a couple of comments ...
>
>> With these changes, older pkg bits reading new config files will fail to
>> proxy URLs, but will otherwise still function. However, I'm not sure
>> whether the changes here should really constitute an ImageConfig version
>> bump, and if so, what the consequences of that are? Perhaps a client API
>> bump as well?
>
> Older clients would fail on those origins, anyway, right? I don't think
> there's a compatibility issue sufficient to bump either version.
Ok - I've tested snv_151a with one of these config files, and yes, it
just doesn't see the additional proxied origins, it just sees a single
instance of an origin at a given URI and otherwise passes through the
contents of the origin_info property.
It becomes a little more complex if someone removes an origin with 151a
pkg bits, causing the origin_info property to become stale when viewed
on gate-tip bits. I'll try to deal with that gracefully.
>> The intent is to add two new properties, "origin_info" and "mirror_info",
>> each of which take a list of lists (tuples, maybe?). In each element, we
>> store the URI, and the proxy used to access that URI.
>>
>> So in the pkg5.image config file, this looks something like:
>>
>> origins = ['http://foo', 'http://bar', 'http://baz']
>> origin_info = [ ['http://foo', 'http://webcache:8080'],
>> ['http://bar', 'http://www-proxy:80'],
>> ['http://bar', 'http://www-proxy2:80'],
>> ['http://baz', 'None'] ]
>
> Why a generic name like "info" for something so specific? Either
> "origin_proxies" (or just "proxies"; why have separate ones for origins and
> for mirrors?) or make "origin_info" be a dictionary that maps keys (such as
> "proxies") to lists of tuples pairing URLs with their proxies, which can
> then later be extended to have other keys (like key/cert like you point out
> later).
Right was called "_info" because I was foreseeing other stuff eventually
being stored here. I was using a list of lists to try to keep the
config file format as simple as possible.
Talking to Shawn this week, he was eager for the config file format to
remain easy for humans to read/edit. I don't have a problem converting
to a dictionary instead, since at least there'd be key/value pairs,
rather than a mysterious list of ordered values - Shawn?
>> which translates to the the 'pkg publisher' output below:
>>
>> timf@kura[36] pkg publisher
>> PUBLISHER TYPE STATUS URI
>> foo origin online proxy://http://foo/
>> baz origin online http://baz/
>> bar origin online proxy://http://bar/
>> bar origin online proxy://http://bar/
>
> I'm not thrilled with the proxy:// bit. I kinda see it for the ones which
> go through the zone proxy, but I don't see why it's necessary here. I find
> it confusing visually, and I don't believe there's any standard that makes
> it acceptable for general use.
Sure - this was really to try to unify the "pkg publisher" output
between the global and non-global zones, and to disambiguate proxied
URIs from ones which we communicate directly with.
I'm happy to drop the "proxy://" prefix, but if I do that, I'd really
want to do it for both system publisher URIs as well as GZ publishers.
In that case, we get:
$ pkg publisher
PUBLISHER TYPE STATUS URI
foo origin online http://foo/
baz origin online http://baz/
bar origin online http://bar/
bar origin online http://bar/
Eventually we could add a -o flag to pkg publisher to allow two new
fields, "method" and "proxy" (we can bikeshed about those names later,
"method" feels wrong), then we could do something like:
$ pkg publisher -o publisher,method,uri,proxy
PUBLISHER METHOD URI PROXY
foo proxy http://foo/ http://webcache:8080
baz direct http://baz/ http://www-proxy:80
bar proxy http://bar/ http://www-proxy2:80
bar proxy http://bar/ -
Within a zone, we'd have:
root@test:~# pkg publisher -o publisher,type,method,proxy
PUBLISHER TYPE METHOD URI PROXY
solaris (syspub) origin proxy http://foo http://localhost:1008
How does that sound? I'd like to work on 'publisher -o' after this
putback, probably as part of 7133972 (and friends - we have a few -o
bugs/rfes) as though. With this putback proxy information would only be
shown in "pkg publisher <publisher>" output.
cheers,
tim