How can I access a http.Response.Body before the entire body is read

[Chinmay V Padhye]

When I use the http library calls such as http.Get, etc the entire body is read before the call is returned.

I am looking for a way to access the resp.Body before the full Body is read. Can I do this without implementing a custom RoundTripper?

In other words the example below is a blocking call.

resp, err := http.Get("http://example.com/")

What I want is to get access to the resp.Body as soon as something is received. Is a custom RoundTripper the only way?

[Jesse McNelis]

On Sun, Aug 23, 2015 at 3:49 PM, Chinmay V Padhye <echi...@gmail.com> wrote:
> When I use the http library calls such as http.Get, etc the entire body is
> read before the call is returned.

It doesn't. The http.Get() blocks until the response headers come back.
The body of the response is an io.Reader that will read from the
network connection in to a buffer.

> What I want is to get access to the resp.Body as soon as something is
> received. Is a custom RoundTripper the only way?

Just read from the resp.Body.


Perhaps if you explained the problem you're trying to solve I could be
of more help.

[Chinmay]

Let me explain a bit more in detail.

The web page I am downloading is quite large. I want to process the html page and do some processing. Lets say I want to count the number of http://www.wikipedia.org/* links in the page. The current process is serial. That means the entire body (html page) is downloaded by the http client. The client returns from http.Get or similar function with the http.Response.Body that has the entire html page. The html page can then be parsed/searched for these links.

However, what I am looking at is getting the Body as a stream. I can then run some string search algorithm that can process streams like the Rabin Karp algorithm. Hence by the time the page is fully downloaded the count is also there. I dont need any extra processing.

Let me know if I should further clarify.

Rest of my answers inline starting with my name

On 24 August 2015 at 11:31, Jesse McNelis <jes...@jessta.id.au> wrote:

On Sun, Aug 23, 2015 at 3:49 PM, Chinmay V Padhye <echi...@gmail.com> wrote:
> When I use the http library calls such as http.Get, etc the entire body is
> read before the call is returned.

It doesn't. The http.Get() blocks until the response headers come back.
The body of the response is an io.Reader that will read from the
network connection in to a buffer.

[Chinmay] This is exactly what I wanted and thought this is the way my client would behave. However I wrote a simple http server and client to observe the behavior.

The server is at - http://play.golang.org/p/ijCaj-kyDc

The client is at - http://play.golang.org/p/FVhuyRi9x9

The server expects a file with filename sendfile.txt. The contents of this is simply -

<p>Links:</p><ul><li><a
 href="foo">Foo</a>
<li><a href="/bar/baz">BarBaz</a></ul>

 The above is taken from the html parser example.

[saj...@gmail.com]

> The client returns from http.Get or similar function with the http.Response.Body that has the entire html page

No it doesnt. The client returns from http.Get as soon as the headers are in, doesnt wait for body. And you can stream http.Response.Body without the need for whole body to come in.

[Bruno Albuquerque]

Body is an io.Reader, so it does exactly what you want.

--
You received this message because you are subscribed to the Google Groups "golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Mat Evans]

Chinmay,

It might do to read a bit more about io.Readers and the other associated interfaces - specifically in regard to streaming data.  There are plenty of examples in the standard library - the encoders have some good ones.

Also plenty of blog posts around the net - for example https://www.datadoghq.com/blog/crossing-streams-love-letter-gos-io-reader/

This is the point of the io.Reader exposed in http.Response.Body - so you don't have to read the whole body to a new variable before using it, like you suggested in a streaming manner.

[Chinmay]

Thanks for the tip folks. Will poke around.

I think there maybe something strange with the way I am writing my test server.

Cheers ....

Chinmay

[Giulio Iotti]

On Monday, August 24, 2015 at 4:04:52 PM UTC+3, Chinmay V Padhye wrote:

Thanks for the tip folks. Will poke around.

I think there maybe something strange with the way I am writing my test server.

Cheers ....

Chinmay

At a glance, maybe you need to flush your writer before time.After? 

if f, ok := w.(http.Flusher); ok {

f.Flush()

}

-- 

Giulio Iotti

https://twitter.com/dullboy

[Chinmay V Padhye]

Thanks Giulio. That is what I was looking for.

Regards,

Chinmay