HttpClientRequest post image request with multipart?

[Bhav]

Hi people,

I'm looking to see how we can post images to an external server using vertx. The server side application is using spring controllers and expects a multipart request with the images sent to a particular url.

I have looked at the doc which suggest Vertx is capable of handling mutlipart requests, but posting using the HttpClientRequest does not seem trivial?

I've played a bit with the actual request, i.e. tried setting the request header Content-Type: multipart/form-data; boundary=<some boundary>

and in the request body setting the multipart parts with the

--<some boundary>

Content-Disposition: form-data; name="a_file"; filename="somefile"

Content-Type: "image/jpeg"

Content-Transfer-Encoding: "binary"

<binary data here>

--<some boundary>--

and all I get is an error back from the server with an error code 500 when it is trying to read the next byte from this request,

I'd like to know if I am on the right path or if I'm missing a trick here. I am actually new to Vert.x so apologies if I've missed something.

Best,

Bhav 

[Alexander Lehmann]

I assume that should work, you only have to make sure that the server understands chunked encoding if you use it or provide a correct content length if it doesn't.

While vert.x doesn't have a method to construct the form-data request, it should go through if you construct it.

[Bhav]

Thank you Alexander. I have implemented the call to post. I see that debugging on the server side, the following exception is thrown twice. Once on read, once on close. (You can see the stack trace details)

nested exception is org.apache.commons.fileupload.FileUploadBase$IOFileUploadException: Processing of multipart/form-data request failed. Stream ended unexpectedly"

      at org.apache.commons.fileupload.MultipartStream$ItemInputStream.makeAvailable(MultipartStream.java:983)

      at org.apache.commons.fileupload.MultipartStream$ItemInputStream.read(MultipartStream.java:887)

      at java.io.InputStream.read(InputStream.java:101)

      at org.apache.commons.fileupload.util.Streams.copy(Streams.java:94)

      at org.apache.commons.fileupload.util.Streams.copy(Streams.java:64)

      at org.apache.commons.fileupload.FileUploadBase.parseRequest(FileUploadBase.java:362)

      at org.apache.commons.fileupload.servlet.ServletFileUpload.parseRequest(ServletFileUpload.java:126)

      at org.springframework.web.multipart.commons.CommonsMultipartResolver.parseRequest(CommonsMultipartResolver.java:156)

      at org.springframework.web.multipart.commons.CommonsMultipartResolver.resolveMultipart(CommonsMultipartResolver.java:139)

      at org.springframework.web.servlet.DispatcherServlet.checkMultipart(DispatcherServlet.java:1047)

      at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:892)

      at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:856)

      at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:953)

      at org.springframework.web.servlet.FrameworkServlet.doPost(FrameworkServlet.java:855)

      at javax.servlet.http.HttpServlet.service(HttpServlet.java:637)

      at org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:829)

      at javax.servlet.http.HttpServlet.service(HttpServlet.java:717)

      at org.apache.commons.fileupload.MultipartStream$ItemInputStream.makeAvailable(MultipartStream.java:983)

      at org.apache.commons.fileupload.MultipartStream$ItemInputStream.close(MultipartStream.java:924)

      at org.apache.commons.fileupload.MultipartStream$ItemInputStream.close(MultipartStream.java:904)

      at org.apache.commons.fileupload.util.Streams.copy(Streams.java:119)

      at org.apache.commons.fileupload.util.Streams.copy(Streams.java:64)

      at org.apache.commons.fileupload.FileUploadBase.parseRequest(FileUploadBase.java:362)

      at org.apache.commons.fileupload.servlet.ServletFileUpload.parseRequest(ServletFileUpload.java:126)

      at org.springframework.web.multipart.commons.CommonsMultipartResolver.parseRequest(CommonsMultipartResolver.java:156)

      at org.springframework.web.multipart.commons.CommonsMultipartResolver.resolveMultipart(CommonsMultipartResolver.java:139)

      at org.springframework.web.servlet.DispatcherServlet.checkMultipart(DispatcherServlet.java:1047)

      at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:892)

      at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:856)

      at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:953)

      at org.springframework.web.servlet.FrameworkServlet.doPost(FrameworkServlet.java:855)

      at javax.servlet.http.HttpServlet.service(HttpServlet.java:637)

      at org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:829)

      at javax.servlet.http.HttpServlet.service(HttpServlet.java:717)

There is also a comment in that bit of code just before the exception is thrown which says:

// The last pad amount is left in the buffer.

// Boundary can't be in there so signal an error

// condition.

And below is my vertx request code. I think this is right, do you see any obvious issues?

package com.test.upload;

import org.vertx.java.core.Handler;

import org.vertx.java.core.http.HttpClient;

import org.vertx.java.core.http.HttpClientRequest;

import org.vertx.java.core.http.HttpClientResponse;

import org.vertx.java.platform.Verticle;

import java.io.IOException;

import java.nio.file.Files;

import java.nio.file.Paths;

public class UploadVerticle extends Verticle {

    

    public void start() {

        super.start();

        HttpClient client = vertx.createHttpClient().setPort(8080).setHost("localhost");

        final HttpClientRequest req = client.post("/api/images/post", new Handler<HttpClientResponse>() {

            public void handle(HttpClientResponse response) {

                System.out.println("File upload attempted " + response.statusCode());

            }

        }).setChunked(false).setTimeout(60000);

        String filename = "/Users/me/Documents/blob.jpg";

        String fileSize = null;

        try {

            fileSize = String.valueOf(Files.size(Paths.get(filename)));

        } catch(IOException e) {

            e.printStackTrace();

        }

        // For a non-chunked upload you need to specify size of upload up-front

        try {

            String imageString = new String(Files.readAllBytes(Paths.get("/Users/me/Documents/blob.jpg")));

            System.out.println(fileSize);

            req.putHeader("Content-Type", "multipart/form-data; boundary=MyBoundary");

            req.putHeader("accept", "application/json");

            req.putHeader("Content-Length", fileSize);

            req.write("--MyBoundary\r\n" +

                    "Content-Disposition: form-data; name=\"photo\"; filename=\"blob.jpg\"\r\n" +

                    "Content-Type: image/jpeg\r\n" +

                    "Content-Transfer-Encoding: binary\r\n\r\n" +

                    imageString + "\r\n" +

                    "--MyBoundary--").end();

        } catch (Exception e) {

            e.printStackTrace();

        }

    }

[Alexander Lehmann]

I think you have to account for the boundary and the additional header lines in the content length.

I'm not sure if the string constructed uses the correct binary encoding, maybe this will not work if it uses utf-8 by default (cannot check now, please check if the size of the string is correct).

[Bhav]

Cheers, I guess that was kind of obvious! Yes, you do need to account for the whole request body in the content length. I managed to hit the correct controller method on the server side, but it seems like the multipart payload is missing. I tried changing the request code to write a Buffer instead of strings to simplify things but still the multipart is missing on the server side method. I'll play around a bit, but it does look like an encoding issue for the binary jpeg image.

Thank you for your time and help!

On Tuesday, July 15, 2014 9:34:39 PM UTC+1, Bhav wrote:

[Alexander Lehmann]

Mixing strings with binary data is not working well in java in my experience, if you can use a Buffer for the data, you are probably better off (to test if it works, you can maybe try a text file so that you will not get encoding problems)

To verify that you are really creating a correct POST request, you could post your request to a netcat listener, this way you can catch the complete request and check if it is using the correct length etc (or you could use something like Wireshark to capture the request).

I'm pretty sure the issue is from an error in the content-length or additional or too few newlines or something.

[Alexander Lehmann]

Ok, two things I noticed fiddling around with your code (not sure how to test the post without an app server right now)

I think you need a newline at the end of the final boundary line, at least regular browsers add that and I'm not completely sure if whether you need \r\n, maybe it works with both.

(if you get it running, it would be great if you could put the solution up on github, I think somebody asked about the same problem a while back)

On Wednesday, July 16, 2014 10:26:00 PM UTC+2, Bhav wrote:

[Bhav]

Success! Thank you. It was actually working all along. As part of the form data header in the body I was sending the wrong parameter name 'photo', when in fact the server was expecting 'image'. I managed to persist the image on the server side and compared it and they were identical, though I did notice the keywords metadata was missing. All the other metadata was present. I still need to pamper up the code a bit and test with different image types, sizes, but below is a working skeletal version of the code. Cheers!

package com.test.upload;

import org.vertx.java.core.Handler;

import org.vertx.java.core.http.HttpClient;

import org.vertx.java.core.http.HttpClientRequest;

import org.vertx.java.core.http.HttpClientResponse;

import org.vertx.java.platform.Verticle;

import java.io.IOException;

import java.nio.file.Files;

import java.nio.file.Paths;

public class UploadVerticle extends Verticle {

    

    public void start() {

        super.start();

        HttpClient client = vertx.createHttpClient().setPort(8080).setHost("localhost");

        final HttpClientRequest req = client.post("/api/images/post", new Handler<HttpClientResponse>() {

            public void handle(HttpClientResponse response) {

                System.out.println("File upload attempted " + response.statusCode());

            }

        }).setChunked(false).setTimeout(60000);

        Buffer bodyBuffer = getBody("/Users/me/Documents/blob.jpg");

        req.putHeader("Content-Type", "multipart/form-data; boundary=MyBoundary");

        req.putHeader("accept", "application/json");

        req.end(bodyBuffer);

    }

    private Buffer getBody(String filename) {

        Buffer buffer = new Buffer();

        buffer.appendString("--MyBoundary\r\n");

        buffer.appendString("Content-Disposition: form-data; name=\"image\"; filename=\"blob.jpg\"\r\n");

        buffer.appendString("Content-Type: application/octet-stream\r\n");

        buffer.appendString("Content-Transfer-Encoding: binary\r\n");

        buffer.appendString("\r\n");

        try {

            buffer.appendBytes(Files.readAllBytes(Paths.get(filename)));

            buffer.appendString("\r\n");

        } catch (IOException e) {

            e.printStackTrace();

        }

        buffer.appendString("--MyBoundary--\r\n");

        return buffer;

    }

}

On Tuesday, July 15, 2014 9:34:39 PM UTC+1, Bhav wrote: