HTTP client - streaming POST body?

[Jim Smart]

Hi all,

I'm looking for an HTTP client that allows my code to write the body content down the pipe while doing a POST.

The use case here is that I'm wishing to send very large UPDATE/INSERT queries/commands to an HTTP endpoint, and the body content of those queries/commands is actually generated from a database. So it would be better, memory- and performance- wise, if I could write that straight down the pipe during the request, as opposed to manifesting the whole of the query/command into a buffer before doing the POST.

Is this possible at all with the stdlib http.Client? Perhaps with an io.Pipe as the request body?

I did look at the code for http.Client a little, but there's a lot of it, and it's not the simplest of code to follow. I do have a base understanding of the HTTP 1.1 protocol, but there's a lot more to http.Client than just that, of course.

I've also tried looking for existing examples showing similar functionality, but did not seem to find anything. Which is partly what makes me wonder if perhaps the stdlib http.Client cannot operate like this.

If I can't do this with the stdlib http.Client, and I have to roll-my-own client of sorts, are there any parts of the existing http package that I should be making use of?

Any tips / pointers / info greatly appreciated.

Thanks,

/Jim

[Bruno Albuquerque]

The body is just an io.Reader. You just need to write one that materializes and streams the data you want instead of using a buffer with all of it. The entire design is to allow things like what you want to do. :)

-Bruno

--
You received this message because you are subscribed to the Google Groups "golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/golang-nuts/545d0597-66a0-4412-8ca7-3d447f88ccafn%40googlegroups.com.

[Amnon]

As Bruno said.

Try something like this:

func uploadBig(url string ) error {
    file, err := os.Open("bigfile.txt")
    if err != nil {
        return err
    }

    defer file.Close()
    resp, err := http.Post(url, "text/plain", file)
    if err != nil {
        return err
    }
    defer resp.Body.Close()
    if resp.StatusCode >= 400 {
        return fmt.Errorf("Bad Status %s", resp.Status)
    }
    _, err = io.Copy(os.Stdout, resp.Body)
    return err
    }

[Yaw Boakye]

I could be misleading you here, but my immediate thought is that whether you

can stream the body/payload depends on what form you're receiving the query

result in? if it's an io.Reader then it looks like you could use it directly. otherwise

i'm not sure what you could do to the HTTP client to change that (since the

query result would have already materialized).

To view this discussion on the web visit https://groups.google.com/d/msgid/golang-nuts/d9e4e00f-2251-453c-88f2-5e2bb8e61c8bn%40googlegroups.com.

--

Curried programmer

Homepage: http://yawboakye.com

I'm tweeting when I'm not coding when I'm not holding my niece.

[Wojciech Kaczmarek]

Hello,

I think this code example rather streams back the response the server sent from its POST handler
and it is not exactly what the OP requested ;) (not to mention that the typical result of a POST is
an immediate redirect, possibly without body at all).

I believe what Jim needs is constructing http.Request with a body in a form of an io.Reader, then executing it.
Here's the minimal example showing that it can be streamed:

```go

func uploadStdin(url string) error {
req, err := http.NewRequest("POST", url, os.Stdin)

if err != nil {
return err
}

resp, err := http.DefaultClient.Do(req)

if err != nil {
return err
}
defer resp.Body.Close()
if resp.StatusCode >= 400 {

return fmt.Errorf("bad status: %s", resp.Status)

}
_, err = io.Copy(os.Stdout, resp.Body)
return err
}

```

I just tested it and indeed, it gets uploaded after I give some input via stdin and hit ctrl+D.

Best,

Wojtek

[Wojciech Kaczmarek]

PS. My choice of http.Request with client.Do was misleading, http.Post can be used as well:

func uploadStdin(url string) error {
    resp, err := http.Post(url, "text/plain", os.Stdin)

    if err != nil {
        return err
    }

    defer resp.Body.Close()
    if resp.StatusCode >= 400 {
        return fmt.Errorf("bad status: %s", resp.Status)
    }
    _, err = io.Copy(os.Stdout, resp.Body)
    return err
}

[Christian Stewart]

No... The previous example streams the file as the post request. The only difference in your example is that you're using stdin.

It's not that complicated, files also implement io.reader. 

To view this discussion on the web visit https://groups.google.com/d/msgid/golang-nuts/f909420c-6662-4ae3-866a-77b2faba9c76n%40googlegroups.com.

[Wojciech Kaczmarek]

You are right, I overlooked the "bigfile" name clearly suggesting it's about the big content being uploaded. My bad.

Anyway, what's important for the OP is that net/http is well-designed and will let you do what you need in this case.

cheers,

-W.

[Jim Smart]

The issue here, isn’t that I am uploading a big file — that’s easy.

As I said in my initial post:

> The use case here is that I'm wishing to send very large UPDATE/INSERT queries/commands to an HTTP endpoint, and the body content of those queries/commands is actually generated from a database.

The content I wish to push to the server, is /generated/ content. So really what I want is a something I can directly write into.

I am trying to avoid generating my upload content into a buffer first. Because the data can be very large.

— It’s easy to say “write a reader” but writing it as a reader involves doing complete inversion of control on my code, and isn’t really feasible. I can’t easily make the code I have which has complex logic to build the upload data using Writes, into something that is then driven by Reads.

Which is why I asked if it was possible to somehow Write straight down the connection.


— Thanks for the suggestions all the same.

/J

[Christian Stewart]

you can achieve this using an io.Pipe. The io.Pipe function returns a connected pair of *PipeReader and *PipeWriter, where writes to the *PipeWriter are directly read from the *PipeReader. The Write call on the *PipeWriter will block until the data is read from the *PipeReader.

pr, pw := io.Pipe()

Pass the pr as the body and write to the pw.

--
You received this message because you are subscribed to the Google Groups "golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/golang-nuts/C5B2823D-2458-4F91-A09D-6B12F74CD8B3%40jimsmart.org.

[burak serdar]

On Wed, Mar 22, 2023 at 8:32 PM 'Christian Stewart' via golang-nuts <golan...@googlegroups.com> wrote:

you can achieve this using an io.Pipe. The io.Pipe function returns a connected pair of *PipeReader and *PipeWriter, where writes to the *PipeWriter are directly read from the *PipeReader. The Write call on the *PipeWriter will block until the data is read from the *PipeReader.

pr, pw := io.Pipe()

Pass the pr as the body and write to the pw.

And when you are done generating the content, close pw. 

On Wed, Mar 22, 2023, 7:06 PM 'Jim Smart' via golang-nuts <golan...@googlegroups.com> wrote:

The issue here, isn’t that I am uploading a big file — that’s easy.

As I said in my initial post:

> The use case here is that I'm wishing to send very large UPDATE/INSERT queries/commands to an HTTP endpoint, and the body content of those queries/commands is actually generated from a database.

The content I wish to push to the server, is /generated/ content. So really what I want is a something I can directly write into.

I am trying to avoid generating my upload content into a buffer first. Because the data can be very large.

— It’s easy to say “write a reader” but writing it as a reader involves doing complete inversion of control on my code, and isn’t really feasible. I can’t easily make the code I have which has complex logic to build the upload data using Writes, into something that is then driven by Reads.

Which is why I asked if it was possible to somehow Write straight down the connection.


— Thanks for the suggestions all the same.

/J


--
You received this message because you are subscribed to the Google Groups "golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/golang-nuts/C5B2823D-2458-4F91-A09D-6B12F74CD8B3%40jimsmart.org.

--
You received this message because you are subscribed to the Google Groups "golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/golang-nuts/CA%2Bh8R2pTUXsWPpywxhRJi_Bs3ZP0hmVP%3DkqZMe6FwJWJS-3BDg%40mail.gmail.com.

[Jim Smart]

Ok, cool, thanks — that was what I asked in my OP :)

Is this possible at all with the stdlib http.Client? Perhaps with an io.Pipe as the request body?

/J