[Support] Process multipart request with files attached

[Abel Salgado Romero]

Hi,

I am implementing a gateway based on the microservices example (http://www.sczyh30.com/vertx-blueprint-microservice/api-gateway.html) and so far I had no issues until I have to re-route a multi-part request.

We have a POST method to upload a document and some additional metadata (form params named "file" and "metadata"), and I am having minor issue and a blocking one.

First thing -the minnor- is retrieving the parameters. With the snippet below I am able to get the "metadata" but I don't see how to add them to the outbound request. How do you create a multipart request in a client?

I found this example, http://www.programcreek.com/java-api-examples/index.php?api=io.vertx.core.http.HttpClientRequest, but it seems to me a bit overkill to have to create the body manually.

```

context.request().formAttributes().forEach( attr -> {

            //toReq.write(attr.getKey(), attr.getValue());

            System.out.println("found: "+attr.getKey());

        });

```

Then, -the blocking- issue, when I try to retrieve the file with the following snippet (as seen in http://vert-x3-preview.michel-kraemer.com/docs/vertx-core/java/) I get the error "java.lang.IllegalStateException: Request has already been read"

```

context.request().uploadHandler(upload -> {

            System.out.println(upload.name());

        });

```

I tried this in the dispatchRequests method as well as in the server configuration, with and without lambdas and the error is always the same. Here is my last attempt. The 

```

        vertx.createHttpServer()

            //.requestHandler(router::accept)

            .requestHandler(request -> {

                boolean expectMultipart = request.isExpectMultipart();

                request.setExpectMultipart(true);

                request.endHandler(vh -> {

                    request.uploadHandler(event -> {

                        System.out.println(event.name());

                    });

                    });

                });

            })

```

Could it be an issue? Or the CircuitBreaker interfering?

Is it possible to create a gateway that routes "normal" messages along with multiparts?

PS: the project is great :) 

cheers,

[Abel Salgado Romero]

Finally, I saw that multipart is supported in a different way and that the current example does not allow to forward it like other POST request (https://github.com/vert-x3/vertx-web/issues/351).

We are considering exposing our upload service in a completely different way, but just to confirm this I'd like to get more details in how big files are managed.

Apparently, the files are not processed and body is empty, however, if I add a bodyHandler as seen below, the whole body appears to be loaded into memory (size is initialized and the memory of the java process increased accordingly)

Is is so? Is it possible to prevent Vert.x form accepting a multipart message? or limit the size of a message?

```

        vertx.createHttpServer()

            .requestHandler(req -> {

                req.setExpectMultipart(true);

                req.bodyHandler(buffer -> {

                    try {

                        int size = buffer.length() / (1024 * 1024);

                        Files.write(Paths.get("c:/bulk.txt"), buffer.getBytes());

                    } catch (IOException e) {

                        e.printStackTrace();

                    }

                });

```

[Paulo Lopes]

To avoid multipart parsing (the default behavior) just do:

req.setExpectMultipart(false)

To limit the length, as you noticed if you have 2 choices:

req.handler(Handler<Buffer>)

where data is processed in chunks so you need to keep track of multiple invocation (but safer).

or:

bodyHandler(Handler<Buffer>)

where data is first kept in memory and called only once.

As you noticed if you use bodyHandler you might have trouble with large uploads and in fact it can be a DDoS. If you're doing web development I'd recommend to use Vert.x web which does the chunk approach and has more utilities like routing, templating, common handler, etc...

[Abel Salgado Romero]

Thanks a lot.

Did some testing and it works exactly as you mention as long as is defined in the server (more info below). Shame that handle method is not documented in the class.

I can return the favor with a PR with that, just point me to the correct branch, repo I assume is https://github.com/eclipse/vert.x/.

Also, I can add a note to https://github.com/sczyh30/vertx-blueprint-microservice regarding the support of multipart.

Regarding the last comment, I am already using vetx-web, but when I use the same code inside a route handler I always get the exception (my initial post):

```

java.lang.IllegalStateException: Request has already been read

at io.vertx.core.http.impl.HttpServerRequestImpl.checkEnded(HttpServerRequestImpl.java:426) ~[vertx-core-3.3.3.jar:?]

at io.vertx.core.http.impl.HttpServerRequestImpl.setExpectMultipart(HttpServerRequestImpl.java:312) ~[vertx-core-3.3.3.jar:?]

at io.vertx.ext.web.impl.HttpServerRequestWrapper.setExpectMultipart(HttpServerRequestWrapper.java:192) ~[vertx-web-3.3.3.jar:?]

at my.gateway.MainGateway.dispatchRequests(MainGateway.java:140) ~[classes/:?]

at io.vertx.ext.web.impl.RouteImpl.handleContext(RouteImpl.java:215) ~[vertx-web-3.3.3.jar:?]

```

I have found that the only way to access the body and attachments is with a handle directly defined with the server requestHandler method,

Any call that ends up calling HttpServerRequestImpl.checkEnded() (handler, endHandler, uploadHandler, setExpectMultipart) from the request of a RoutingContext handler cause the 'Request has already been read' error.

Honestly, it doesn't make much sense to me, but I assumed it was an intended thing.

[Paulo Lopes]

On Wednesday, November 9, 2016 at 10:56:06 AM UTC+1, Abel Salgado Romero wrote:

Thanks a lot.

Did some testing and it works exactly as you mention as long as is defined in the server (more info below). Shame that handle method is not documented in the class.

I can return the favor with a PR with that, just point me to the correct branch, repo I assume is https://github.com/eclipse/vert.x/.

Yes that's the one.
 

Also, I can add a note to https://github.com/sczyh30/vertx-blueprint-microservice regarding the support of multipart.

Please do.
 

Regarding the last comment, I am already using vetx-web, but when I use the same code inside a route handler I always get the exception (my initial post):

```

java.lang.IllegalStateException: Request has already been read

at io.vertx.core.http.impl.HttpServerRequestImpl.checkEnded(HttpServerRequestImpl.java:426) ~[vertx-core-3.3.3.jar:?]

at io.vertx.core.http.impl.HttpServerRequestImpl.setExpectMultipart(HttpServerRequestImpl.java:312) ~[vertx-core-3.3.3.jar:?]

at io.vertx.ext.web.impl.HttpServerRequestWrapper.setExpectMultipart(HttpServerRequestWrapper.java:192) ~[vertx-web-3.3.3.jar:?]

at my.gateway.MainGateway.dispatchRequests(MainGateway.java:140) ~[classes/:?]

at io.vertx.ext.web.impl.RouteImpl.handleContext(RouteImpl.java:215) ~[vertx-web-3.3.3.jar:?]

```

I have found that the only way to access the body and attachments is with a handle directly defined with the server requestHandler method,

Any call that ends up calling HttpServerRequestImpl.checkEnded() (handler, endHandler, uploadHandler, setExpectMultipart) from the request of a RoutingContext handler cause the 'Request has already been read' error.

Honestly, it doesn't make much sense to me, but I assumed it was an intended thing.

From the documentation:

The usage of this handler requires that it is installed as soon as possible in the router since it needs to install handlers to consume the HTTP request body and this must be done before executing any async call.

But this is not clear most of the times. The reason for this is that multipart parsing is done at netty level and needs to be enabled when a request arrives if a async call happens then the parsing of the request continues and it is assumed that we're not interested in the body.

[masas dani]

Hi, 

I experienced the same problem with microservice, i'm implementing a gateway based on the microservices example (http://www.sczyh30.com/vertx-blueprint-microservice/api-gateway.html) also. and had the the problem on uploading multipart file. 

it seems that nothing change from the sample repos. tried other Proxy implementation from vertx sample (https://github.com/vert-x3/vertx-examples/blob/master/core-examples/src/main/java/io/vertx/example/core/http/proxy/Proxy.java) but not working also. 

tried using pump like (https://github.com/vert-x3/vertx-examples/blob/master/core-examples/src/main/java/io/vertx/example/core/http/upload/Client.java) but not working also. tried other method by setting header as x-www-form-urlencoded, but it strangely handled by my microservice as string body. 

how do you solve the problem ?

can you please link me to resources that i can try ?

I'm glad if you can help.

been stuck in several days.

Thanks,

Masas