vertx-web and handling large HTTP payloads (not file uploads)

[Claus Ibsen]

Hi

We have use-cases with Apache Camel that runs on Quarkus, which uses the vertx-http web server from Quarkus.

Because Camel users are enterprises and they can do things such as sending GBs of data over HTTP (not file uploads) or any other fancy stuff. 

Just send data over HTTP as chunks.

This works in Camel with Servlet components and has historically worked for this with older Camel releases running on Karaf / Tomcat / JBoss EAP / Spring Boot (servlet).

However for Camel on Quarkus we have a problem, as we use directly the Quarkus HTTP server (vertx-web).

This JIRA has more details

https://issues.apache.org/jira/browse/CAMEL-19905

What I am asking is, if there is anything in vertx-web that can make this possible. I have looked at the vetx Buffer, and the BodyHandler implementation from vertx-web, and to my limited knoweldge it seems it uses Buffer.append which adds data into memory. And this would lead to OOME when you send GBs of data.

I assume I am not the first person asking, so there is likely something easy I miss.

/Claus Ibsen

[Paulo Lopes]

If you’re sending multipart data it should be stored as a file upload. In this case the data is backed by a file, the limit is the storage capacity. If you’re sending as raw data its hold in memory hence the issue. Users should be aware of this because by default we don’t allow more than 10mb raw uploads. In that case an explicit flag must be enabled in code to allow it.

An improvement can be made to stream to disk when the flag is enabled and a user explicitly states the intent.

On 22 Sep 2023, at 11:12, Claus Ibsen <claus...@gmail.com> wrote:

Hi

--
You received this message because you are subscribed to the Google Groups "vert.x" group.
To unsubscribe from this group and stop receiving emails from it, send an email to vertx+un...@googlegroups.com.
To view this discussion on the web, visit https://groups.google.com/d/msgid/vertx/e88a8520-1188-4df9-a582-5bf6598c174an%40googlegroups.com.