large file uploads using vertx-web

[Kenneth Westelinck]

Hi,

We're on Vertx 3.2.0 and we have an HttpVerticle accepting file uploads. These uploads can get quite large (2GB). The verticle is accepting HTTP requests through the vertx-web framework. What we see is:

- upload is quite slow

- application takes a lot of memory while uploading, roughly size of upload times 1.5

I have created another verticle, doing the same thing, but with standard vertx.createHttpServer() and a lot of callback handlers. In this POC code, I don't see the same behavior. Upload speed is normal and memory footprint is low.

I've been debugging and profiling and found this method in the default body handler (io.vertx.ext.web.handler.impl.BodyHandlerIml) of vertx-web:

@Override
public void handle(Buffer buff) {
  if (failed) {
    return;
  }
  if (bodyLimit != -1 && (body.length() + buff.length()) > bodyLimit) {
    failed = true;
    context.fail(413);
  } else {
    body.appendBuffer(buff);
  }
}

While debugging, this method gets called. And when the upload is finished, this gets called:

void doEnd() {
  if (failed || ended) {
    return;
  }
  ended = true;
  HttpServerRequest req = context.request();
  if (mergeFormAttributes && req.isExpectMultipart()) {
    req.params().addAll(req.formAttributes());
  }
  context.setBody(body);
  context.next();
}

Putting the entire upload on the context, as far as I can see.

I don't think this should happen. When sending a multi part request, we should not try to handle the buffer, but rely on the file that was streamed here:

public BHandler(RoutingContext context) {
  this.context = context;
  Set<FileUpload> fileUploads = context.fileUploads();
  makeUploadDir(context.vertx().fileSystem());

  context.request().setExpectMultipart(true);
  context.request().exceptionHandler(context::fail);
  context.request().uploadHandler(upload -> {
    // We actually upload to a file with a generated filename
    uploadCount.incrementAndGet();
    String uploadedFileName = new File(uploadsDir, UUID.randomUUID().toString()).getPath();
    upload.streamToFileSystem(uploadedFileName);
    FileUploadImpl fileUpload = new FileUploadImpl(uploadedFileName, upload);
    fileUploads.add(fileUpload);
    upload.exceptionHandler(context::fail);
    upload.endHandler(v -> uploadEnded());
  });
}

Or am I not making sense here?

regards,

Kenneth

[Tim Fox]

I believe this is a known issue - there should be an issue for it in vertx-web. If not, please add one :)

--
You received this message because you are subscribed to the Google Groups "vert.x" group.
To unsubscribe from this group and stop receiving emails from it, send an email to vertx+un...@googlegroups.com.
Visit this group at https://groups.google.com/group/vertx.
To view this discussion on the web, visit https://groups.google.com/d/msgid/vertx/4095f585-51ef-4cf0-8f52-e038d8ca033f%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Arnold Schrijver]

The issue was probably #351 and was fixed in https://github.com/vert-x3/vertx-web/commit/d8f90f86fed10c51923b4ac929d140c415995c4e

[Kenneth Westelinck]

Thank you all for your very quick replies. I guess I will be giving the 3.3.0-SNAPSHOT a try :)