How to avoid saving uploaded file data to the upload directory for the multipart form request?

[Kidong Lee]

Hi,

I have implemented HTTP server for the multipart form using BodyHandler.create() with Router.

But as I understand BodyHandlerImpl class, first, the uploaded file should be save to the upload directory

after that, from the routing context like ctx.fileUploads(), the uploaded file can be read from the upload directory.

I want to avoid such saving uploaded file data into the directory,

Without saving uploaded file to the directory, I just want to read the uploaded file data directly which should be sent to kafka afterwards.

I have thought that for my case, performance issue could occur with this approach like saving uploaded file data into directory first.

Thus I have implemented a custom BodyHandler which is similar to BodyHandlerImpl except saving uploaded file data to directory.

It works good for me for now,

but I am not sure, it is a correct way or there is another alternative way to solve my problem.

Thanks,

- Kidong Lee.

P.S: I have added some codes from my custome body handler.

The following listing is BHandler class in my custom MultipartBodyHandlerImpl class, it is very similar to the original BHandler in BodyHandlerImpl:

....

public BHandler(RoutingContext context, long contentLength) {
    this.context = context;
    Set<FileUpload> fileUploads = context.fileUploads();

    final String contentType = context.request().getHeader(HttpHeaders.CONTENT_TYPE);
    if (contentType == null) {
        isMultipart = false;
        isUrlEncoded = false;
    } else {
        final String lowerCaseContentType = contentType.toLowerCase();
        isMultipart = lowerCaseContentType.startsWith(HttpHeaderValues.MULTIPART_FORM_DATA.toString());
        isUrlEncoded = lowerCaseContentType.startsWith(HttpHeaderValues.APPLICATION_X_WWW_FORM_URLENCODED.toString());
    }

    initBodyBuffer(contentLength);

    if (isMultipart || isUrlEncoded) {
        context.request().setExpectMultipart(true);

        context.request().uploadHandler(upload -> {
            if (bodyLimit != -1 && upload.isSizeAvailable()) {
                // we can try to abort even before the upload starts
                long size = uploadSize + upload.size();
                if (size > bodyLimit) {
                    failed = true;
                    context.fail(413);
                    return;
                }
            }
            if (handleFileUploads) {
                // we actually upload to a file with a generated filename
                uploadCount.incrementAndGet();

                // NOTE: upload buffer stream will not be saved to the upload directory.
                // buffer data will be set to MultipartFileUploadImpl instance which will be forwarded through Routing Context. 
                Pump p = Pump.pump(upload, new UploadWriteStream(fileUploads, upload));
                p.start();

                upload.resume();
                upload.exceptionHandler(t -> {
                    deleteFileUploads();
                    context.fail(t);
                });
                upload.endHandler(v -> uploadEnded());
            }
        });
    }

    context.request().exceptionHandler(t -> {
        deleteFileUploads();
        context.fail(t);
    });
}

private class UploadWriteStream implements WriteStream<Buffer>
{
    private Set<FileUpload> fileUploads;
    private HttpServerFileUpload upload;

    public UploadWriteStream(Set<FileUpload> fileUploads, HttpServerFileUpload upload)
    {
        this.fileUploads = fileUploads;
        this.upload = upload;
    }

    public WriteStream<Buffer> exceptionHandler(Handler<Throwable> handler) {
        return this;
    }

    public WriteStream<Buffer> write(Buffer buffer) {
        byte[] data = buffer.getBytes();

        // Here, MultipartFileUploadImpl containing uploaded file data bytes will be added to Set<FileUpload> of routing context 
        MultipartFileUploadImpl fileUpload = new MultipartFileUploadImpl(null, upload, data);
        fileUploads.add(fileUpload);

        return this;
    }

    public void end() {
    }

    public WriteStream<Buffer> setWriteQueueMaxSize(int maxSize) {
        return this;
    }

    public boolean writeQueueFull() {
        return false;
    }

    public WriteStream<Buffer> drainHandler(Handler<Void> handler) {
        return this;
    }
}

..........


public class MultipartFileUploadImpl implements FileUpload {
    private final String uploadedFileName;
    private final HttpServerFileUpload upload;
    private final byte[] data;

    public MultipartFileUploadImpl(String uploadedFileName, HttpServerFileUpload upload, byte[] data) {
        this.uploadedFileName = uploadedFileName;
        this.upload = upload;
        this.data = data;
    }

    public byte[] getData() {
        return data;
    }

    public String name() {
        return this.upload.name();
    }

    public String uploadedFileName() {
        return this.uploadedFileName;
    }

    public String fileName() {
        return this.upload.filename();
    }

    public long size() {
        return this.upload.size();
    }

    public String contentType() {
        return this.upload.contentType();
    }

    public String contentTransferEncoding() {
        return this.upload.contentTransferEncoding();
    }

    public String charSet() {
        return this.upload.charset();
    }
}
.........

[Paulo Lopes]

The reason body handler does not keep upload files in memory is because it could lead to DDoS attacks. If you want to get better performance without having to adapt any code (when deploying on linux) you could look into ramfs or tmpfs to mount the upload dir. Both filesystems have pros and cons, this old article can give you quick overview:

https://www.thegeekstuff.com/2008/11/overview-of-ramfs-and-tmpfs-on-linux/

In this case even though the uploads are seen as a filesystem, it all happens in memory so you're not really limited by the disc spinning latencies...

[Kidong Lee]

Thank for your explanation about multipart handling in vert.x body handler, Paulo.

I will try to do so as you suggested.

Cheers,

-Kidong

2019년 1월 22일 (화) 오후 9:04, Paulo Lopes <pml...@gmail.com>님이 작성:

--
You received this message because you are subscribed to the Google Groups "vert.x" group.
To unsubscribe from this group and stop receiving emails from it, send an email to vertx+un...@googlegroups.com.
Visit this group at https://groups.google.com/group/vertx.
To view this discussion on the web, visit https://groups.google.com/d/msgid/vertx/2acc03fe-1e7a-4a32-82c6-b659b4896f66%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Hasani]

Hi Kidong,

If you want to direct process, do as mentioned in the vertx core doc at https://vertx.io/docs/vertx-core/java/#_handling_requests, scroll down to Handling HTML Forms / Handling form file upload. Please note that you can not mixed this with vertx.web bodyHandler

Cheers

[Kidong Lee]

Thank your instruction, hasani.

I have already considered your suggestion.

But I have also wanted to use routing context, thus why I have written my own body handler.

For more details, I have replied to your answer in my thread: 

https://groups.google.com/d/msgid/vertx/e6c85e19-407d-4f1f-8d8d-417810819216%40googlegroups.com?utm_medium=email&utm_source=footer

Cheers,

- Kidong 

2019년 1월 23일 (수) 오후 12:34, Hasani <arief....@gmail.com>님이 작성:

Hi Kidong,

If you want to direct process, do as mentioned in the vertx core doc at https://vertx.io/docs/vertx-core/java/#_handling_requests, scroll down to Handling HTML Forms / Handling form file upload. Please note that you can not mixed this with vertx.web bodyHandler

Cheers

--

You received this message because you are subscribed to the Google Groups "vert.x" group.
To unsubscribe from this group and stop receiving emails from it, send an email to vertx+un...@googlegroups.com.
Visit this group at https://groups.google.com/group/vertx.

To view this discussion on the web, visit https://groups.google.com/d/msgid/vertx/5cfe983c-8c02-44f7-91cc-493113d42e12%40googlegroups.com.