net/http: question about RoundTripper

[Anuj Agrawal]

I am trying to understand in what cases would it make sense to
implement my own RoundTripper.

If I search about it, I come across examples of RoundTripper that try
to do things like caching, retries, authentication, etc. I also read
somewhere that there are many RoundTripper implementations that just
set the User-Agent header on a request.

I know that the documentation says "RoundTrip should not attempt to
handle higher-level protocol details such as redirects,
authentication, or cookies." And, I also understand that RoundTripper
would be a bad place for things like caching.

However, I am not able to figure out why is it such a bad idea to use
RoundTripper as a middleware that allows me to do some of the higher
level things like authentication. After all, authentication is just
about interpreting and/or manipulating some headers. In some cases, it
could be just as good as setting the User-Agent where all that happens
is setting the Authorization header with a token. In some other cases,
it could mean interpreting the challenge thrown by the server and then
making the same call again with a response to the challenge.

Can someone please help me understand this better?

[Kevin Conway]

I'll make an attempt to answer this question but I could be wrong given some deeper historical context between the early Go developers.

There are two chunks of the http.RoundTripper comments that folks typically ask about: "should not attempt to handle higher-level protocol details" and "should not modify the request". Unfortunately, the rationale for why these statements are made is neither presented in the comments nor in the patches that introduced them.

It appears "should not modify the request" most likely refers to a concurrency issue that must be mitigated by copying the request before mutating it in the RoundTripper. Here's the change that claims to have resolved an issue surrounding this: https://codereview.appspot.com/5284041. The original comments explicitly allowed for mutating the request but this was changed to "should not" after this patch due to the bug that it resolved.

It's a little harder to find direct evident of the author's intent for "should not attempt to handle higher-level protocol details". This part of the comment has been in the code for nearly a decade and it becomes fairly difficult to track the origin past a set of major renames and large movements of files from place to place within the early Go source code. Reading https://github.com/golang/go/commit/e0a2c5d4b540934e06867710fe7137661a2a39ec makes it seem like these notes were meant for the author or for other Go core devs who were building the original HTTP stack rather than those of us who would use it later. For example, it appears to signal that standard library developers should isolate higher level features within the Client type rather than in the ClientTransport (now RoundTripper) type. I haven't found anything, yet, that suggests the comments are meant for anyone other than developers of the http package in the Go standard library.

From a more practical perspective, you don't really have another choice when it comes to HTTP client middleware that are generally useful in Go applications than the http.RoundTripper interface. If everyone applied the "accept interfaces, return structs" guidelines then you would have more options. For example, if everything that needed an HTTP client accepted a "type Doer { Do(r *http.Request) (*http.Response, Error) }" style interface then you could target your middleware as wrappers for the http.Client. Unfortunately, most projects that allow for injection of a custom HTTP client do so by accepting an instance of *http.Client. Accepting that specific, concrete type makes wrapping anything other than the http.RoundTripper a practical impossibility.

Personally, I've been using http.RoundTripper middleware for several years without issue. It's a solid pattern that can provide an enormous amount of value by allowing re-usable layers of behavior that can be injected into virtually any library or framework that uses an HTTP client. I don't worry about the comments in the standard library for the reasons I listed.

--
You received this message because you are subscribed to the Google Groups "golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/golang-nuts/CABnk5p0k77he6c7Pw0APQJF%2B_FDJFOFdJp_BJ8eS66jbw6%3DG1w%40mail.gmail.com.

--

Kevin Conway

[Anuj Agrawal]

Thanks Kevin for these insights. It does seem like the documentation
notes were meant for Go core devs. It would have helped, if the
authors threw in more insight.

I have also been using RoundTripper as client middleware, but so far
largely for authentication. I wanted to expand the scope of the client
middleware in my implementations to do more but looking at the
RoundTripper documentation, I wanted to have views on its use and see
if I can find anti-patterns that I should be aware of.

In fact, since I could not find a lot of useful information around it,
I even felt like writing a blog post highlighting good and bad
patterns using RoundTripper based on the notes I collect.