webclient response body length limit

[wenb...@gmail.com]

Hi,

I am using webclient (4.0.3) to crawl some urls and got OutOfMemory error. With Memory Analyzer, I got informed that the response body length is 1,392,508,944. I am thinking to use buffer to control the size of response body.

My question is, is there a way to control body length to a limit to avoid OOM? Thanks.

Regards,

Kevin

[Thomas SEGISMONT]

Hi,

The Web Client  is designed to buffer response data before invoking callbacks. If you know you are going to get large responses, you can use the pipe BodyCodec:

https://vertx.io/docs/apidocs/io/vertx/ext/web/codec/BodyCodec.html#pipe-io.vertx.core.streams.WriteStream-

It pipes the content to any WriteStream (could be a file on disk denoted by a Vert.x AsyncFile) before invoking callbacks.

If you need to examine status code and/or headers before deciding what to do, consider using the Vert.x Core HttpClient.

Regards

--
You received this message because you are subscribed to the Google Groups "vert.x" group.
To unsubscribe from this group and stop receiving emails from it, send an email to vertx+un...@googlegroups.com.
To view this discussion on the web, visit https://groups.google.com/d/msgid/vertx/350d08d5-5dd3-4cac-aa73-4d5f86fbbcc1n%40googlegroups.com.

[kevin]

Thanks a lot. I think I need to use the Vert.x Core HttpClient.

Regards,

Kevin