What's the best way to copy an http.Request?

[Tomás Senart]

Hello there,

In http://github.com/tsenart/vegeta, I have the need to re-use http.Requests. Before, I was using these structs concurrently which led to unexpected results.
Hence, I learned that http.Requests are not thread-safe. Perhaps the documentation should be improved on this point.

My first approach to solve the problem was to create enough http.Requests to satisfy my needs, by copying.

However, copying the request Body doesn't seem straightforward as it doesn't get reset after the first full read.

How would you suggest I solve this without any extra state besides the http.Request itself?

Regards,
Tomás Senart

[Donovan Hide]

Depends what you want to re-use the request for. If you just want to store it for matching with the response, might be worth looking at: 

Have a read of http://golang.org/pkg/net/http/httputil/#DumpRequest

--
You received this message because you are subscribed to the Google Groups "golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Caleb Spare]

In the past I've done this by using a custom reader for Body which
reads from a shared (fixed) buffer. I copy the request and then swap
out the body for a new one.

https://github.com/cespare/steller/blob/master/steller.go#L28-L53
https://github.com/cespare/steller/blob/master/steller.go#L154-L155

I think I decided to do this after reading through httputil.ReverseProxy code

http://golang.org/src/pkg/net/http/httputil/reverseproxy.go

(although it doesn't do quite the same sort of thing).

-Caleb

[Tomás Senart]

On Wednesday, April 9, 2014 5:31:29 PM UTC+1, Donovan wrote:

Depends what you want to re-use the request for. If you just want to store it for matching with the response, might be worth looking at: 

Have a read of http://golang.org/pkg/net/http/httputil/#DumpRequest

I want to use it and re-use it to pass it to client.Do. I want to issue the same request many times.
 

[Donovan Hide]

Is the use case that you want to be able to dynamically alter the body of some future Requests while some Requests are in flight?

https://github.com/tsenart/vegeta/blob/request-body/lib/targets_test.go#L99-L105

If so, you could do something similar to Caleb's suggestion, and wrap a RWMutex around the byte slice embedded in a custom reader...

[Tomás Senart]

On Wednesday, April 9, 2014 5:37:14 PM UTC+1, Caleb Spare wrote:

In the past I've done this by using a custom reader for Body which
reads from a shared (fixed) buffer. I copy the request and then swap
out the body for a new one.

https://github.com/cespare/steller/blob/master/steller.go#L28-L53
https://github.com/cespare/steller/blob/master/steller.go#L154-L155

Assuming that io.Reader isn't used concurrently, then I think it can be improved by setting off = 0 before returning an io.EOF.
http://play.golang.org/p/tiBBiDVAhl

[Tomás Senart]

On Wednesday, April 9, 2014 5:46:13 PM UTC+1, Donovan wrote:

Is the use case that you want to be able to dynamically alter the body of some future Requests while some Requests are in flight?

No. I don't want to alter the body at all. I just want to either be able to use the same http.Request concurrently (which I can't) or to be able to copy an http.Request in its integrity.
 

[Donovan Hide]

> No. I don't want to alter the body at all. I just want to either be able to use the same http.Request concurrently (which I can't) or to be able to copy an http.Request in its integrity.

Well a Request has state, such as the Content-Length header based on the size of the given body, so this is probably never going to work. Why are you trying to reuse requests when they are cheap to make? If you want huge bodies, just re-use the reader and build new requests on demand. Your race is probably happening here:

https://github.com/tsenart/vegeta/blob/request-body/lib/targets.go#L83

Perhaps the docs should point out that the Request struct fields aren't settable:

http://golang.org/src/pkg/net/http/request.go#L118

If you changed your API to be func NewTargets(lines []string, body io.ReadCloser, header http.Header) (Targets, error) 

https://github.com/tsenart/vegeta/blob/request-body/lib/targets.go#L37

I'm sure your problems would disappear :-)

[Donovan Hide]

Probably:

func NewTargets(lines []string, body []byte, header http.Header) (Targets, error)

would be better and create a new bytes.Reader in the function body.

[Tomás Senart]

On Wed, Apr 9, 2014 at 6:15 PM, Donovan Hide <donov...@gmail.com> wrote:

> No. I don't want to alter the body at all. I just want to either be able to use the same http.Request concurrently (which I can't) or to be able to copy an http.Request in its integrity.

Well a Request has state, such as the Content-Length header based on the size of the given body, so this is probably never going to work. Why are you trying to reuse requests when they are cheap to make? If you want huge bodies, just re-use the reader and build new requests on demand. Your race is probably happening here:

https://github.com/tsenart/vegeta/blob/request-body/lib/targets.go#L83

It's definitely happening there, although this is still in a branch. Note the BUG comment above ;)

Reusing http.Requests is out of question as we already saw. That leaves us with copying.
https://github.com/tsenart/vegeta/blob/master/lib/attack.go#L42

Since two go routines could potentially receive the same request out of that slice, I want to copy the request (aka Target) before putting it into the channel. The only hurdle I am having currently is with copying the Body as all the rest is straightforward.

Perhaps the docs should point out that the Request struct fields aren't settable:

http://golang.org/src/pkg/net/http/request.go#L118

If you changed your API to be func NewTargets(lines []string, body io.ReadCloser, header http.Header) (Targets, error) 

https://github.com/tsenart/vegeta/blob/request-body/lib/targets.go#L37

This is a better API indeed (using body as []byte would be better though), but it doesn't solve the problem as I need to copy the requests in the above mentioned location.
Also, Vegeta is a library besides a CLI utility (which wraps the library). I am not sure how many people are using it as such though.

[Tomás Senart]

I ended up with this: https://github.com/tsenart/vegeta/pull/44/files#diff-60240d1051814a7662b4c474131a5442R99
Have a look at the PR if you like.

Thanks for your help :)