Akka-Http handle Multipart formData

[lucian enache]

Hello everyone, for akka-http when handling a POST request, if I have my entity as a Multipart.FormData , 

and I POST some JSON and a binary file, how can I split the two when I parse the formData ?

This is what I come up with so far

 path("upload") { /* uploads a file to the database */
          post {
            entity(as[Multipart.FormData]) { (formData) =>


              val uploadedUrlsFuture = formData.parts.map(_.entity.dataBytes).mapAsync(parallelism = 1)(part =>
                part
                  .map(_.toArray)
                  .runFold(ByteString())((totalBytes, bytes) => totalBytes ++ bytes)
                  .map(_.toArray[Byte]).map(ResourceManager.saveFile(_)) /* TODO update this to a service */
              ).grouped(100).runWith(Sink.head)
              complete(OK)
            }
          }
        }

So if I have an Content-Length of 2 in a scenario where I post an JSON Object and a binary file,

is there a way to split that up and handle the two separately?

On top is this a good way to handle file upload to a backend or there are better ways (the file size should be at most 100mb)?

[Konrad Malawski]

Hi Lucian,

As discussed on gitter, I believe this is doable using the formFields directive for example:
formFields('json, 'file.as[StrictForm.FileData]) { case (json: String, file: StrictForm.FileData) => // not tested though...

I agree that this should be a) very simple and b) well documented, which it isn't currently, thus I opened a ticket to improve this, please let us know of your progress or comments in there: https://github.com/akka/akka/issues/18471

As for limiting the maximal size of an upload, it’s currently already possible as you can use this cookbook pattern:
http://doc.akka.io/docs/akka-stream-and-http-experimental/1.0/scala/stream-cookbook.html#Limit_the_number_of_bytes_passing_through_a_stream_of_ByteStrings

We’re contemplating if it can be made available as a simple to use directive, here’s the ticket (feel free to voice your opinion there or jump in and help us out :-)):
https://github.com/akka/akka/issues/18493

​

--
>>>>>>>>>> Read the docs: http://akka.io/docs/
>>>>>>>>>> Check the FAQ: http://doc.akka.io/docs/akka/current/additional/faq.html
>>>>>>>>>> Search the archives: https://groups.google.com/group/akka-user
---
You received this message because you are subscribed to the Google Groups "Akka User List" group.
To unsubscribe from this group and stop receiving emails from it, send an email to akka-user+...@googlegroups.com.
To post to this group, send email to akka...@googlegroups.com.
Visit this group at http://groups.google.com/group/akka-user.
For more options, visit https://groups.google.com/d/optout.

--

Cheers,

Konrad 'ktoso' Malawski

Akka @ Typesafe

[Johannes Rudolph]

Hi,

the approach that Konrad showed will work, however, the problem with the suggested approach is that it will load the complete file into memory which limits the number of concurrent uploading requests severely. Any multipart entity is a natural stream of parts so your original approach is a good start.

On Friday, September 11, 2015 at 4:12:02 PM UTC+2, lucian enache wrote:

Hello everyone, for akka-http when handling a POST request, if I have my entity as a Multipart.FormData , 

and I POST some JSON and a binary file, how can I split the two when I parse the formData ?

What do you mean by "split"? If you post them as multipart you will have several parts which are split automatically and will be delivered as elements on the `parts` stream. Inside of your `mapAsync` you can inspect the metadata (like the filename of a part, or the field name for which it was posted) before doing anything with the data (but then don't do `.map(_.entity.dataBytes)` first), so there's no really the need to "split" processing but just decide which processing you want to do based on the metadata of each part. If you need to correlate both parts with some context things will be a bit harder, I would then suggest to return some result for each part inside of mapAsync (e.g. the filename of the part together with the future representing the result of processing that part) and then collect all these result and do whatever is necessary before completing.

Here's a sketch:

case class ProcessingResult(partFileName: String, result: Result)

def processBinaryPart(part): Future[ProcessingResult] = ...

def processJsonPart(part): Future[ProcessingResult] = ...

entity(as[Multipart.FormData]) { (formData) =>

  val processedParts = formData.parts.mapAsync(1) { part =>

    part.filename match {

      case Some("xyz.bin") => processBinaryPart(part)

      case Some("abc.json") => processJsonPart(json)

    }

  } // result of mapAsync is Source[ProcessingResult, ...]

  // now collect all results into a map for further processing

  .runFold(Map.empty[String, ProcessingResult])((map, res) => map += res.partFileName -> res)

  // result is a Future[Map[String, ProcessingResult]]

  

  // wait for processing to have finished and then create response

  onSuccess(processedParts) { resultMap =>

    complete { ...

    }

  }

}

Cheers,

Johannes