mulit-part form data

[Tim Pigden]

Hi

A source is uploading files to the lagom service that need to be parsed as spreadsheet data. 

If I upload files using curl or something like that there's no problem, I can parse the bytes and all that sort of stuff using POI.

But the data is arriving as multipart form data

Previous advise from James was

a) get the gateway to handle it

b) use "some lower level play api"

a) is not practical at this point.

b) unfortunately leads to the fact that I don't know Play and it looks like I need to understand a lot to do something with these apis.

I have a byte string and ti's got header and tail info wrapped around it so this code

private class WorkbookDeserializer(charset: String) extends NegotiatedDeserializer[Workbook, ByteString] {
  override def deserialize(wire: ByteString): Workbook = {
    val asArray = wire.toArray
    val ins = new ByteArrayInputStream(asArray)
    try {
      // write out to file
      FileUtils.writeByteArrayToFile(new File(s"data/wakefield/uploads/${UUID.randomUUID()}.xlsx"), asArray)

writes the spreadsheet data with the multipart header stuff.

Is there some simple instructions for applying a parser to extract the necessary info. Or should I just go looking for another library. After all I guess all I really need to do is strip off a few bytes, but there may be gotchas

[James Roper]

Here's an (untested) example of using PlayServiceCall to parse multipart/form-data files.  This example doesn't buffer the file in memory (like your example does), but rather writes it straight to disk, so there's no risk of running out of memory (though there is a risk of running out of disk, but Play's built in multipart/form-data parser has a configurable limit to how much data it will write to disk per request).  Using fully qualified classes to make it clear.

def myServiceCall: ServiceCall[NotUsed, SomeResult] = PlayServiceCall { wrapCall =>

  // Create a Play action, and give it the multipartFormData body parser to parse the body

  play.api.mvc.Action(play.api.mvc.BodyParsers.parse.multipartFormData({ fileInfo =>

    // This is a file part handler, it will be invoked once for each file part that

    // the request contains, passing in the fileInfo which gives you access to things

    // like the name of the field they entered it into, the name of the file on their

    // machine, and the files content type. You're expected to return an accumulator

    // (wrapper for an Akka streams Sink) that will be used to consume the file, for

    // example, that will write it to disk. The result of the accumulator will then

    // be accessible below once all files have been uploaded. In this case, since

    // we're generating a random file name, we'll map the accumulator to return that

    // file name, so we can then do further processing on the file.

    val file = new File(s"data/wakefield/uploads/${UUID.randomUUID()}.xlsx")

    val sink = akka.stream.scaladsl.FileIO.toPath(file.toPath)

    play.api.libs.streams.Accumulator(sink).map {

      case akka.stream.IOResult(bytesWritten, status) =>

        // Check if status.success is true if you want, or whatever. In this case,

        // we'll just return the file that was written.

        file

    }

  }) { request =>

    // request is a play.api.mvc.Request[play.api.mvc.MultipartFormData[File]]

    // You could just directly handle the request here, and generate a Play Result,

    // but if you want Lagom to handle serializing the result of the ServiceCall,

    // then you can use wrapCall to convert a ServiceCall that returns SomeResult

    // into an action, and then invoke that action.

    val wrappedAction = wrapCall(ServiceCall { _ =>

      

      // Here's all the uploaded files, you can do with them what you want. If

      // you returned something other than File from the file part handler, then

      // the type parameter to FilePart would be the type of that thing that you

      // returned.

      val files: Seq[FilePart[File]] = request.body.files

      // Create the result, can be whatever you want, will be serialized by Lagom.

      val someResult = ....

      Future.successful(someResult)

    })

    wrappedAction.apply(request).run()

  }

}

--
You received this message because you are subscribed to the Google Groups "Lagom Framework Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to lagom-framework+unsubscribe@googlegroups.com.
To post to this group, send email to lagom-framework@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/lagom-framework/b3b32b8f-790e-4b7d-bc23-9d3705ae99d0%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

James Roper
Software Engineer

Lightbend – Build reactive apps!
Twitter: @jroper

[Tim Pigden]

So a supplementary question as I try to implement this.

My original service is 

def postItemXls: ServiceCall[List[Item], CheckedDone]

I have implemented a serializer that works with byte arrays representing the spreadsheet.

This works just fine. I can test with curl.

I actually have quite a few things where I want to send data around but I may instead upload a file. For example a json file or an xml file. In the case of the json this is the regular json serializer.

So in each case I have appropriate serializers in place and working.

What I'd rather not do is implement a separate end-point for multipartForm and for regular data. So I want a composite wrapper that will either test for multipart or drop into "standard" mode if the file is not wrapped.

So what I'm confused about is your wrapCall stuff. The output of your process is expecting me to separately parse the file within the body as one of many files. But what I really want to do is pass the bytes of the temporary file to my serializer and ideally do all the testing for multi-part. Any suggestions as to how to manage that?

[Tim Pigden]

I think we need play.api.mvc.Action.async here. Can you confirm please? Otherwise you get an extra Future

To unsubscribe from this group and stop receiving emails from it, send an email to lagom-framewo...@googlegroups.com.
To post to this group, send email to lagom-f...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/lagom-framework/b3b32b8f-790e-4b7d-bc23-9d3705ae99d0%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[James Roper]

The Lagom server takes all your ServiceCalls and turns them into EssentialAction - EssentialAction is what Play uses to handle requests.

The PlayServiceCall allows you to return your own custom EssentialAction for when you want to do things that Lagom doesn't support.  But what if you want to also take advantage of Lagom's logic?  The wrapCall callback is basically an injection of Lagom's logic, it allows you to give it a ServiceCall, and it will wrap it in an EssentialAction, the same way that happens when you don't use PlayServiceCall but just implement a regular SerivceCall, which you can then invoke from your custom EssentialAction, and so have the best of both worlds.

So, if you only want to provide custom logic for handling multipart/form-data, but want Lagom to handle JSON request for you, then you can do this:

def myServiceCall: ServiceCall[SomeRequest, SomeResult] = PlayServiceCall { wrapCall =>

  // Need to use EssentialAction so that we can check the content type before we choose

  // a body parser

  EssentialAction { requestHeader =>

    val action = if (requestHeader.contentType == "application/json") {

      // Just delegate straight to Lagom's logic

      wrapCall(ServiceCall { request =>
        // Do something

        Future.successful(someResult)
      })

    } else {

      // Otherwise, use the multipart/form-data logic

      play.api.mvc.Action.async(play.api.mvc.BodyParsers.parse.multipartFormData({ fileInfo =>

        val file = new File(s"data/wakefield/uploads/${UUID.randomUUID()}.xlsx")

        val sink = akka.stream.scaladsl.FileIO.toPath(file.toPath)

        play.api.libs.streams.Accumulator(sink).map {

          case akka.stream.IOResult(bytesWritten, status) =>

            file

        }

      }) { request =>

        // Do something

        Future.successful(Results.Ok)

      }

    }

    action(requestHeader)

  }

}

Note that this time I didn't use wrapCall in the multipart/form-data logic.  Why?  Because in this case, the action that wrapCall is going to return is expecting to be able to parse a body of type SomeRequest as JSON.  If I really wanted to, I could either build a JSON byte string to feed into it, and then I could use wrapCall, or I could also define a custom message serializer that, when the message was empty, used some default instance.

At this stage though I'd be questioning about whether all this was worth it.  We have two requirements here, one to handle JSON, and another to upload files.  Handling them with the same service call doesn't feel right to me, and as you can see, makes things quite complex.

--
You received this message because you are subscribed to the Google Groups "Lagom Framework Users" group.

To unsubscribe from this group and stop receiving emails from it, send an email to lagom-framework+unsubscribe@googlegroups.com.
To post to this group, send email to lagom-framework@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/lagom-framework/f9101b10-2d8e-46a2-89e1-8a9fde483cb6%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.