Handling large HTTP responses with Web Client

[jamesne...@gmail.com]

Hi,

Reading through the Vert.x Web Client docs, it mentions the following for handling large  responses:

"When large response are expected, use the BodyCodec.pipe"

Then there's the code snippet:

client.get(8080, "myserver.mycompany.com", "/some-uri")
   .as(BodyCodec.pipe(writeStream))
   .send()

Any pointers on how the writeStream part should be implemented? I've not been able to find any examples.

Cheers,

--

James

[andriusk]

AsyncFile implements WriteStream, so you can write it into a file.

AsyncFile file = vertx.fileSystem()
.openBlocking("file", new OpenOptions());

client.get(8080, "myserver.mycompany.com", "/some-uri")

.as(BodyCodec.pipe(file))

.send() 

There is a non-blocking way to open files too.

Thanks

Andrius

[Thomas SEGISMONT]

Hi,

Usually you dont' implement write streams yourself but you use one of the API types: File, HttpServerResponse... etc.

Can you tell a bit more about your use case?

Thanks,

Thomas

--
You received this message because you are subscribed to the Google Groups "vert.x" group.
To unsubscribe from this group and stop receiving emails from it, send an email to vertx+un...@googlegroups.com.
To view this discussion on the web, visit https://groups.google.com/d/msgid/vertx/7382144c-ec32-49af-a9ce-575269963434n%40googlegroups.com.

[jamesne...@gmail.com]

Ok, then I'm probably confused about how things are supposed to work.

The JavaDoc of HttpResponse states:

"Keep in mind that using this HttpResponse impose to fully buffer the response body and should be used for payload that can fit in memory."

So if the payload cannot fit in memory, what's the alternative? I'm thinking of something similar to OkHttp which allows you to stream the response body via byteStream etc.

[Thomas SEGISMONT]

Can you tell a bit more about your use case please?

To view this discussion on the web, visit https://groups.google.com/d/msgid/vertx/5c7c7aa7-c583-4ded-84e0-a2f117e26220n%40googlegroups.com.

[jamesne...@gmail.com]

> Can you tell a bit more about your use case please?

I want to consume a response from an HTTP endpoint which may either be small, or potentially very large - 100s of MBs or GBs.

[Thomas SEGISMONT]

Le mar. 1 mars 2022 à 13:08, jamesne...@gmail.com <jamesne...@gmail.com> a écrit :

> Can you tell a bit more about your use case please?

I want to consume a response from an HTTP endpoint which may either be small, or potentially very large - 100s of MBs or GBs.

What do you plan to do with it? Relay over http? Dump to file?

 

To view this discussion on the web, visit https://groups.google.com/d/msgid/vertx/ba7952d4-d0b9-48a4-8444-ac59a49ce3fan%40googlegroups.com.

[Ben]

Hello @tsegi..., 
I would like to know more about that too. 

In my case I would like to do a small processing in the json response (a mapping) and dump the result in a csv file.

Is there a way to do that in a streaming way ?

I would love if someone could point me to examples...

thanks

[Thomas SEGISMONT]

Hi Ben,

Yes you can do that in a streaming way with the HttpClient.

You could create a JsonParser that reads the HttpClientResponse: https://vertx.io/docs/vertx-core/java/#_json_parser

See the section about objectValueMode which explains how to get events for small objects in a big JSON array.

Then for each Json object, create a CSV line as a String, and write to an AsyncFile, which is a WriteStream: https://vertx.io/docs/vertx-core/java/#streams

See the example using writeQueueFull on the WriteStream and pause on the Readstream to handle backpressure during the transfer.

HTH

To view this discussion on the web, visit https://groups.google.com/d/msgid/vertx/affcada6-cd3a-40cd-bc38-90ef8ded40b9n%40googlegroups.com.