Marshalling FormData to multi-part mime with one field per part

[Eric Swenson]

AWS/S3 HTTP Post with a policy requires an HTTP post with multipart-mime -- one part for the file and one part, each, for various form parameters. When I use akka.http and attempt to marshal FormData to an Entity and use that to create a multi-part mime message, all the form data go into one mime-part using the query string format (foo=bar&baz=quux). AWS rejects such a request as it doesn't want all form parameters in a single www-url-encoded-form-data part, but rather separate mime parts for each parameter. How to I cause a Form to be martialled in this way using akka.http?

[Scott Maher]

I can't test this as I am not at home but there are multiple form marshallers, one for url encoding and one for multipart called, I think, Multipart.FormData. sorry if you already know this.

Hi Eric! :P

On Oct 11, 2015 9:18 PM, "Eric Swenson" <er...@swenson.org> wrote:

AWS/S3 HTTP Post with a policy requires an HTTP post with multipart-mime -- one part for the file and one part, each, for various form parameters. When I use akka.http and attempt to marshal FormData to an Entity and use that to create a multi-part mime message, all the form data go into one mime-part using the query string format (foo=bar&baz=quux). AWS rejects such a request as it doesn't want all form parameters in a single www-url-encoded-form-data part, but rather separate mime parts for each parameter. How to I cause a Form to be martialled in this way using akka.http?

--
>>>>>>>>>> Read the docs: http://akka.io/docs/
>>>>>>>>>> Check the FAQ: http://doc.akka.io/docs/akka/current/additional/faq.html
>>>>>>>>>> Search the archives: https://groups.google.com/group/akka-user
---
You received this message because you are subscribed to the Google Groups "Akka User List" group.
To unsubscribe from this group and stop receiving emails from it, send an email to akka-user+...@googlegroups.com.
To post to this group, send email to akka...@googlegroups.com.
Visit this group at http://groups.google.com/group/akka-user.
For more options, visit https://groups.google.com/d/optout.

[Eric Swenson]

Hey Scott!

When you get a chance, if you can point me to the right one, I'd appreciate it. I've been looking at the sources, tests, and examples, and I can't find how to do it.  What I'm after is a way to take a Form, and emit a multi-part MIME message with one part for each field in the form and one part for the single file I need to upload. Using code like this:

     val formBodyPartFuture = for {

      httpEntity <- Marshal(form).to[HttpEntity]

      strictEntity <- httpEntity.toStrict(timeout)

    } yield Multipart.FormData.BodyPart("foo", strictEntity)

gives me one body part with Content-Type x-www-url-encoded, and the set of all fields are encoded as in a query string.If you can point me to the "other" way of marshalling a form to a set of mime body parts, I'd appreciate it. I had pretty much given up (and googling and using stack overflow have turned up nothing) and was going to generate the parts manually.  i'm sure there must be a correct way to do this.  Thanks. --  Eric

[Eric Swenson]

I have not been able to find any way to marshall a set of HTTP Post form parameters AND a file using akka-http, where the form data parameters are NOT www-url-encoded, but rather each placed in a separate MIME part, with no Content-Type header. This appears to be the format required in order to do an HTTP POST to AWS S3 with a policy.  Furthermore, it appears AWS/S3 is not happy with Chunked HTTP POST requests, and therefore I've had to use toStrict on all my entities.  I've resorted to some hacky code, which works, but I find it surprising that there is no built-in way to generate a multi-part MIME message where post parameters are not combined into one www-url-encoded query string.  

Here is my (admittedly ugly) code:

    val m = Map(

      "key" -> "upload/quux.txt",

      "acl" -> acl,

      "policy" -> policy,

      "X-amz-algorithm" -> "AWS4-HMAC-SHA256",

      "X-amz-credential" -> s"$awsAccessKeyId/$shortDate/$region/$service/aws4_request",

      "X-amz-date" -> date,

      "X-amz-expires" -> expires.toString,

      "X-amz-signature" -> signature

    )

    

    implicit val timeout: FiniteDuration = 10.seconds

    

    val uri = s"http://$bucket.$service-$region.amazonaws.com"

    

    // create request entities for each of the POST parameters. Note that generating Strict entities is an asynchronous

    // operation (returns a future). Strict entities are NOT chunked. It appears AWS is not happy with chunked MIME

    // body parts.

    val strictFormDataEntityFuture = m.map { case (k, v) => (k, HttpEntity(ContentTypes.NoContentType, v).toStrict(timeout)) }

    

    val responseFuture = for {

      strictEntityMap <- Future.sequence(strictFormDataEntityFuture.map(entry => entry._2.map(i => (entry._1, i)))).map(_.toMap) recoverWith { case t: Throwable => throw t }

      formDataBodyPartList = strictEntityMap.map {e => Multipart.FormData.BodyPart.Strict(e._1, e._2)}.toList

      fileEntity <- HttpEntity(ContentTypes.`application/octet-stream`, 3, Source.single(ByteString("foo"))).toStrict(timeout) recoverWith { case t: Throwable => throw t }

      multipartForm = Multipart.FormData(Source(formDataBodyPartList ++List(Multipart.FormData.BodyPart("file", fileEntity, Map("filename" -> "foo.bin")))))

      requestEntity <- Marshal(multipartForm).to[RequestEntity] recoverWith { case t: Throwable => throw t }

      strictRequestEntity <- requestEntity.toStrict(timeout) recoverWith { case t: Throwable => throw t }

      response <- Http().singleRequest(HttpRequest(method = HttpMethods.POST, uri = uri, entity = strictRequestEntity)) recoverWith { case t: Throwable => throw t }

      responseEntity <- Unmarshal(response.entity).to[String] recoverWith { case t: Throwable => throw t }

    } yield (responseEntity, response)

If anyone can suggest a better approach, I'd appreciate it.  Thanks. 

[Mathias]

Eric,

you can create a multipart/formdata request like this, for example:

    val formData = {
      def utf8TextEntity(content: String) = {
        val bytes = ByteString(content)
        HttpEntity.Default(ContentTypes.`text/plain(UTF-8)`, bytes.length, Source.single(bytes))
      }
     
      import Multipart._
      FormData(Source(
        FormData.BodyPart("foo", utf8TextEntity("FOO")) ::
          FormData.BodyPart("bar", utf8TextEntity("BAR")) :: Nil))
    }

    val request = HttpRequest(POST, entity = formData.toEntity())

This will render a chunked request that looks something like this:

    POST / HTTP/1.1
    Host: example.com
    User-Agent: akka-http/test
    Transfer-Encoding: chunked
    Content-Type: multipart/form-data; boundary=o5bjIygPrPrCCYUp8FyfvsQe

    71
    --o5bjIygPrPrCCYUp8FyfvsQe
    Content-Type: text/plain; charset=UTF-8
    Content-Disposition: form-data; name=foo


    3
    FOO
    73

    --o5bjIygPrPrCCYUp8FyfvsQe
    Content-Type: text/plain; charset=UTF-8
    Content-Disposition: form-data; name=bar


    3
    BAR
    1e

    --o5bjIygPrPrCCYUp8FyfvsQe--
    0

If you want to force the rendering of an unchunked request you can call `toStrict` on the `formData` first.
This would create a request looking something like this:

    POST / HTTP/1.1
    Host: example.com
    User-Agent: akka-http/test
    Content-Type: multipart/form-data; boundary=1QipXPrd9L25eJwcQYgiNbfA
    Content-Length: 264

    --1QipXPrd9L25eJwcQYgiNbfA
    Content-Type: text/plain; charset=UTF-8
    Content-Disposition: form-data; name=foo

    FOO
    --1QipXPrd9L25eJwcQYgiNbfA
    Content-Type: text/plain; charset=UTF-8
    Content-Disposition: form-data; name=bar

    BAR
    --1QipXPrd9L25eJwcQYgiNbfA--

There is no www-url-encoding happening anywhere.

Cheers,
Mathias

[Eric Swenson]

Hi Mathias,

I think you are suggesting pretty much exactly what I did, but I did it in such a way that I didn’t have to make each body part (corresponding to a form parameter) manually (I used mapping).  

However, AWS/S3 will not accept the Content-Type: text/plain in the body parts. It thinks the request is trying to “upload” multiple files.  I had to use ContentTypes.NoContentType. 

See below for my approach (where m is a map of the parameter/value pairs):

    val strictFormDataEntityFuture = m.map { case (k, v) => (k, HttpEntity(ContentTypes.NoContentType, v).toStrict(timeout)) }

That produces a Map[String, Future[HttpEntity.Strict]], which is then converted to a Future[Map[String,HttpEntity.Strict]] with:

for {

      strictEntityMap <- Future.sequence(strictFormDataEntityFuture.map(entry => entry._2.map(i => (entry._1, i)))).map(_.toMap)

…

AWS/S3 doesn’t want the form data parts, nor the whole HTTP POST to be chunked. So those entities have to be made “strict”.  So I had to make each MIME part be strict, as well as the final Multipart.FormData entity.

The point of my original post was that it seems the kaka-http library has made a policy decision that the only way to convert a FormData to an entity (in a multi-part MIME body) is to create a single part, using www-url-encoding, and include all fields in query string syntax. This doesn’t work for AWS and therefore it seems a limiting policy decision. It should be equally possible/easy to convert a FormData to a collection of parts. The requirement that there should be a Content-Type header in these parts, also should not be dictated by the framework.

— Eric

You received this message because you are subscribed to a topic in the Google Groups "Akka User List" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/akka-user/ChTzhtmTvyU/unsubscribe.
To unsubscribe from this group and all its topics, send an email to akka-user+...@googlegroups.com.

[Johannes Rudolph]

On Wednesday, October 14, 2015 at 7:17:14 PM UTC+2, Eric Swenson wrote:

AWS/S3 doesn’t want the form data parts, nor the whole HTTP POST to be chunked. So those entities have to be made “strict”.  So I had to make each MIME part be strict, as well as the final Multipart.FormData entity.

Have you seen/tried http://docs.aws.amazon.com/AmazonS3/latest/API/sigv4-streaming.html? It seems to suggest that chunked transfer-encoding is supported, though in a bit complicated format where on top of chunked transfer-encoding AWS seems to introduce their own chunked content-encoding layer where each chunk is signed separately.

 

The point of my original post was that it seems the kaka-http library has made a policy decision

WDYM? akka-http supports both `FormData` = www-url-encoding and `multipart/formdata` encodings, so, I wonder which "policy decision" you mean?

 

that the only way to convert a FormData to an entity (in a multi-part MIME body) is to create a single part, using www-url-encoding, and include all fields in query string syntax. This doesn’t work for AWS and therefore it seems a limiting policy decision.

Yes, because `scaladsl.model.FormData` models www-url-encoding and not multipart/formdata. You seem to suggest that there should be a more general encoding that would allow marshalling to either of both variants. In any case, there's no policy decision but, as explained above, two different models for two different kinds of modelling things.

It should be equally possible/easy to convert a FormData to a collection of parts.

Yes, that could be useful, or otherwise a more general representation of form data that unifies both kinds.

 

The requirement that there should be a Content-Type header in these parts, also should not be dictated by the framework.

 I see that you have to fight a bit to get akka-http to render exactly what you want, but the reason seems to be mostly that AWS has very specific rules and interpretations of how to use HTTP. akka-http is a general purpose HTTP library and cannot foresee all possible deviations or additional constraints HTTP APIs try to enforce. So, in the end, akka-http should make it possible to connect to these APIs (as long as they support HTTP to a large enough degree) but it may mean that the extra complexity the API enforces lies in your code. You could see that as a feature.

That said, I'm pretty sure that there's some sugar missing in the APIs that would help you build those requests more easily. If you can distill the generic helpers that are missing from your particular use case we could discuss how to improve things.

Here are some things I can see:

 * you can directly build a `Multipart.FormData.Strict` which can be directly converted into a Strict entity, I guess one problem that we could solve is that there's only a non-strict FormData.BodyPart.fromFile` which builds a streamed part to prevent loading the complete file into memory. There's no `FormData.BodyPart.fromFile` that would actually load the file and create a strict version of it. We could add that (even if it wouldn't be recommended to load files into memory...)

 * having to run marshalling and unmarshalling manually could be replaced by some sugar

 * dealing with those chains of `T => Future[U]` functions is cumbersome, for this and the previous point, we had the simple `pipelining` DSL in spray which seems to have been lost in parts in the transition to akka-http

Btw. I don't think your code is too bad, if you break it down into methods for every step and put it into a utility object, you can reuse it and don't need to deal with it any more.

Johannes

[Eric Swenson]

On Thursday, October 15, 2015 at 12:24:48 AM UTC-7, Johannes Rudolph wrote:

Have you seen/tried http://docs.aws.amazon.com/AmazonS3/latest/API/sigv4-streaming.html? It seems to suggest that chunked transfer-encoding is supported, though in a bit complicated format where on top of chunked transfer-encoding AWS seems to introduce their own chunked content-encoding layer where each chunk is signed separately.

Yes, but I don't know if this works for S3 HTTP POSTs, and I don't need each chunk separately signed.  

 

The point of my original post was that it seems the kaka-http library has made a policy decision

WDYM? akka-http supports both `FormData` = www-url-encoding and `multipart/formdata` encodings, so, I wonder which "policy decision" you mean?

Perhaps I just couldn't find out how to do it (except very manually as you saw in my code sample). You can easily create a single HTTP entity via Marshal(formData).to[HttpEntity], where that entity will encode all form parameters/values as a query string. How do you, using existing marshallers, create a List[HttpEntity] given a FormData object? The bias is that there is no direct way to do this. I can clearly map over all the fields in the FormData myself and marshall each name/value pair to an HttpEntity. But by default, doing so uses ContentType.`text/plain`. You need your own custom logic to force the content type to NoContentType, which is what we need in this case. 

 

that the only way to convert a FormData to an entity (in a multi-part MIME body) is to create a single part, using www-url-encoding, and include all fields in query string syntax. This doesn’t work for AWS and therefore it seems a limiting policy decision.

Yes, because `scaladsl.model.FormData` models www-url-encoding and not multipart/formdata. You seem to suggest that there should be a more general encoding that would allow marshalling to either of both variants. In any case, there's no policy decision but, as explained above, two different models for two different kinds of modelling things.

That is the bias I'm talking about. FormData is biased toward www-url-encoding rather than multipart/formdata. Both should be provided. 

It should be equally possible/easy to convert a FormData to a collection of parts.

Yes, that could be useful, or otherwise a more general representation of form data that unifies both kinds.

 

The requirement that there should be a Content-Type header in these parts, also should not be dictated by the framework.

 I see that you have to fight a bit to get akka-http to render exactly what you want, but the reason seems to be mostly that AWS has very specific rules and interpretations of how to use HTTP.

Actually, AWS is expecting an entity that is exactly what all browsers would generate when you include one or more files in an HTTP form.  All the form fields including the files go in separate multi-part mime parts. The files have a Content-Type header and the non-files don't. Now I don't have any issue with the non-strict bias in akka-http. That is the more general, and more async-friendly approach. So it doesn't bother me that you have to do something special to get the various entities to be strict. Here, an expression that maps over the body parts converting each from non-strict to strict and returning a Future[Multipart.FormData] (or whatever it would be) would be fine.

akka-http is a general purpose HTTP library and cannot foresee all possible deviations or additional constraints HTTP APIs try to enforce. So, in the end, akka-http should make it possible to connect to these APIs (as long as they support HTTP to a large enough degree) but it may mean that the extra complexity the API enforces lies in your code. You could see that as a feature.

I don't disagree with anything you've said there. But there is a bias toward www-url-encode form data. And it would be helpful if akka-http made it as easy to create Multipart.FormData entities. 

That said, I'm pretty sure that there's some sugar missing in the APIs that would help you build those requests more easily. If you can distill the generic helpers that are missing from your particular use case we could discuss how to improve things.

Here are some things I can see:

 * you can directly build a `Multipart.FormData.Strict` which can be directly converted into a Strict entity, I guess one problem that we could solve is that there's only a non-strict FormData.BodyPart.fromFile` which builds a streamed part to prevent loading the complete file into memory. There's no `FormData.BodyPart.fromFile` that would actually load the file and create a strict version of it. We could add that (even if it wouldn't be recommended to load files into memory...)

 * having to run marshalling and unmarshalling manually could be replaced by some sugar

Yes, that would be good.

 

 * dealing with those chains of `T => Future[U]` functions is cumbersome, for this and the previous point, we had the simple `pipelining` DSL in spray which seems to have been lost in parts in the transition to akka-http

Yes. It took me a while to figure out how to change the collections of futures to a future collection of non-futures. But I learned a lot in the process!   

Btw. I don't think your code is too bad, if you break it down into methods for every step and put it into a utility object, you can reuse it and don't need to deal with it any more.

Thanks. I'll think about how to do break up that big/nasty for, as well as the original creation of the Name -> Entity map.  

-- Eric