Two small proposals for http library (client-specific)

124 views
Skip to first unread message

Michael Hoisie

unread,
Jan 11, 2010, 2:47:25 PM1/11/10
to golang-nuts
I've been doing quite a bit of work with Go's http library (couchdb
client, web downloads, and facebook api), and I believe the following
two changes would make Go's http client library much more useful:

1. Add an http.Client type, and move the existing Get and Post
functions as methods of this type. The client type will store the tcp
connection which can be re-used if multiple requests are made on the
same hostname. This is useful for keep-alive connections, and making
repeated calls to REST apis without having to recreate a connection
each time. Later on , the client can store other state, like cookie
data or session transcripts.

2. http.Client should have a method with the following format:

func (client *Client) Request ( method string, url string, headers
[string]string, body string )

Rationale: This lets you issue general http requests ( PUT, DELETE,
etc.. ) which is very important for things like REST APIS.

Any thoughts?
Mike

Petar Maymounkov

unread,
Jan 11, 2010, 3:49:32 PM1/11/10
to golang-nuts
Something along these lines makes, sense but a few questions:

(1) You say the Client object should keep a cookie database.
That doesn't seem to belong there. Why? Say you are
implementing a proxy and you want to keep the cookies
of one side of the flow. Is it the client or the server that keeps
them? You don't want them stored twice. On the other hand
if you are implementing just a client or just a server you want
them stored in either case. The sum total of this is that
cookie database doesn't seem to belong in Client or Server.

(2) The Request spec should not be a header map [string]string.
Why? The point of the Request and Response objects is that
if you assign the header through their special fields, they do some
sanitizing for inconsistencies for you. On the flip side, if your
proposed methods are intended for pure convenience, you
could pass a header map to your call (say Get or whatever you
wanna call it) and have the implementation parse it with
the Request object to clean it up.

P

Michael Hoisie

unread,
Jan 11, 2010, 4:20:13 PM1/11/10
to golang-nuts
1. I'm not saying it should have a cookie database -- just that
different types of state could be stored between http requests. Right
now the only thing I care about is saving the tcp connection (to avoid
initiating a connection each request), but later on different
properties can be added to it. Whether or not a cookie database should
be there is debatable. But for instance, you could store proxy
information in the client object.

2. From my experience, if you're writing code that interacts with web
apis, it's easier to pass a map[string]string than repeatedly calling
addHeader on a request object. Also, the client.Request method can
sanitize the headers that are passed along the map.

linuxhacker

unread,
Jan 11, 2010, 9:21:01 PM1/11/10
to golang-nuts
I have been playing around with a spec for something like this. What
you are asking for will probably not be in the OS libs. My design is
so far more or less a copy of urllib in python XD.

If you want something like this you are going to need to make it
yourself.

On Jan 11, 2:47 pm, Michael Hoisie <hoi...@gmail.com> wrote:

Mike Ramirez

unread,
Jan 11, 2010, 9:51:26 PM1/11/10
to golan...@googlegroups.com
On Monday 11 January 2010 18:21:01 linuxhacker wrote:
> If you want something like this you are going to need to make it
> yourself.
>

For now I would think/hope, but once a package is created, why can't it be
distributed and shared with others? After awhile if it's found useful and
important enough it should be added to the core.

Regardless if it's part of the core or not, take sqlalchemy for python, great
package, worthy of being in the core, but it's not is it and how many people
still use it for everyday things (I even like to use it over QtSQL with PyQT).

While the http client is already part of the core, I don't think it's unfair
to say that an extended version of it can be distributed outside of the core.

Which brings me to what does Go have in mind for 3rd party packages,
deployment and management utilities? I'm a python person and really in love
with pypi and easy_install/pip, for managing python related packages.

Also if I want my packages to be global, I haven't seen it (still looking for
it in the docs, but haven't done a throrough search for it), do I install them
to $GOROOT/lib/$GOOS_$GOARCH after the are compiled to .a or .o? Is there
plans for a site-packages like directory with in the lib/$GOOS_$GOARCH
directory for such things, so they don't get mixed up with the core packages?


Mike


--
I think $[ is more like a coelacanth than a mastadon.
-- Larry Wall in <1997051019...@wall.org>

signature.asc

Michael Hoisie

unread,
Jan 11, 2010, 9:58:05 PM1/11/10
to golang-nuts
Well if you look at issue 155 ( http client rewrite) , there's just a
few things people actually want:

1. support for generic http methods beyond GET and POST
2. persistent connections
3. (stretch) support for https
4. (stretch) support for cookies

See: http://code.google.com/p/go/issues/detail?id=155

Adding a client type with a generic request method addresses all of
these items. I disagree that it can't be an os lib -- a lot of Go's
requirements are driven internally at Google, and undoubtedly they'll
have to support a decent http library. It would be easier if it's
sooner than later.

I already wrote an http library that handles this ( http://github.com/hoisie/httplib.go
), but I realized the difference from Go's http package is so small,
it would be easier just to modify that.

- Mike

Russ Cox

unread,
Jan 11, 2010, 11:37:33 PM1/11/10
to Michael Hoisie, golang-nuts
I want to see better client support in the standard
package; there's no need to assume the standard
packages cannot improve.

> 1. support for generic http methods beyond GET and POST
> 2. persistent connections
> 3. (stretch) support for https
> 4. (stretch) support for cookies

In particular, I want to see all of these. Petar has
proposed a redesign that includes an explicit Client
type and is keen to implement it. I think that will
give us a place to plug in things like a cookie jar,
persistent connections, and maybe even fine-grained
control over following redirects.

I can imagine that the Client would take an optional
value of

type Cookies interface {
Cookies(scheme, host string) []string
SetCookie(host string, cookie string)
}

to manage the cookies for that connection.

Https is a simple matter of implementing the "get CA
roots" function on Linux, FreeBSD, and OS X.
I think the code for the first two is done, so it's just a
matter of learning how to parse the OS X root CA file,
which must be documented or reverse engineered somewhere.

Russ

Reply all
Reply to author
Forward
0 new messages