Marshaller to handle content negotiation derived from existing marshalling support

[Adam Mackler]

I apologize if this is answered somewhere, but I cannot figure this out, though I see much discussion around the issue.

Using spray-routing, I'm wanting to marshal my custom case class appropriately for the value of the request's Accept header.  I have no problem creating a Marshaller that will produce one type, say XML or JSON, but how do I create one Marshaller to handle either depending on the request?  In particular, I'm confused by the fact that I need to use existing Marshallers.  If I am extending DefaultJsonProtocol to handle marshalling JSON, there's that.  Then I see there's a NodeSeqMarshaller, so I ought to be able use that somehow with some inline XML to create my desired XML-format response entity (but still that's marshalling a NodeSeq and not my custom case class).  And then suppose I also want to handle text/plain with just a string.  I can create a Marshaller to handle all desired types, match on the ContentType (or MediaType, not sure which), and then what?  How do I connect that to the json support or the NodeSeqMarshaller?

I have a pretty good idea conceptually of what has to happen but I cannot figure out how to put all the pieces together to make it work.  Is there an example somewhere of a Marshaller for a custom case-class that will handle requests with Accept headers of either text/plain, application/xml or application/json, incorporating the spray json support, the NodeSeq marshaller, and some of my own marshalling code that doesn't rely on any existing marshallers?

I know this is not that complicated, but the solution eludes me.  An example would be invaluable, and from my perspective, the documentation would be wonderfully improved if it included such an example or reference to one.

Thanks very much,

--

Adam Mackler

[Adam Mackler]

Here's what I have so far:

case class Foo(name: String, quantity: Int)

object Main extends App with SimpleRoutingApp {

  implicit val system = ActorSystem("my-system")

  implicit val m = FooObj.FooMarshaller

  startServer(interface = "localhost", port = 8080) {

    complete( Foo("Alice", 33) )

  }

}

object FooObj extends DefaultJsonProtocol {

  val allTypes = Seq (

    `text/xml`,

    `application/xml`,

    `application/json`,

    `text/plain`

  ).map(ContentType(_))

  val fooFormat: RootJsonFormat[Foo] = jsonFormat2(Foo)

  val FooMarshaller = new Marshaller[Foo] {

    def apply(foo: Foo, ctx: MarshallingContext) {

      ctx.tryAccept(allTypes) match {

case Some(contentType @ ContentType(mediaType,charset)) =>

 ctx.marshalTo(HttpEntity(

   contentType,

   (mediaType: @unchecked) match {

     case `application/xml` | `text/xml` =>

<foo quantity={foo.quantity.toString}>{foo.name}</foo>.toString

     case `application/json` =>

PrettyPrinter(fooFormat.write(foo))

     case `text/plain` =>

s"${foo.name} is on record for ${foo.quantity}\n"

   }

 ))

case None => ctx.rejectMarshalling(allTypes)

      }

    }

  }

}

Is this the proper way to be doing content negotiation?

Also, if anyone feels like answering some basic non-spray questions, I have two: first, how can I make this Marshaller be implicitly available in the Main object without declaring the variable that references it?  I tried declaring the Marshaller as implicit val FooMarshaller and then saying import FooObj._ in the Main class, which seems as if it should work to me, but didn't.

Second, is there a way I can change the name of FooObj to Foo instead, so that it becomes the companion to the case class Foo?  Changing its name as such causes jsonFormat2() to fail for reasons I have not been able to figure out; apparently the parameter of jsonFormat2() is a function returning a Foo rather than a Foo-type value, so maybe there's a way to rewrite the invocation of jsonFormat2 so its argument is explicitly a function literal?

The reason I'm asking is this passage from the Marshalling documentation:

you best define the Marshaller for T in the companion object of T. This way your marshaller is always in-scope, without anyimport tax.

So ought I to be putting this FooMarshaller in the companion object to the case class Foo?  And if so, how do I then get a JsonWriter[Foo]?

[Mathias Doenitz]

Adam,

your questions are very valid, so let me go through them one by one:

> I have no problem creating a Marshaller that will produce one type, say XML or JSON, but how do I create one Marshaller to handle either depending on the request?

spray relies completely on type-classes for associating marshaling logic with a type. Type-classes are resolved statically at compile-time, not at runtime, which means that your program cannot pick a different marshaler for your type depending on what the client sent along in the `Accept` header of the request.
There must be exactly one marshaler for your type, which is able to produce *all* supported resource representations (JSON, XML, …).

However, this does not mean that this one marshaller cannot delegate (parts of) the job of actually producing the representations to other (sub-)marshallers.
Here is an example (untested):

case class Foo(bar: String)

object Foo {
import DefaultJsonProtocol._

val jsonMarshaller = SprayJsonSupport.sprayJsonMarshaller(jsonFormat1(Foo.apply))
val xmlMarshaller = Marshaller.delegate[Foo, NodeSeq](`text/xml`) { foo => <foo>${foo.bar}</foo> }
val textMarshaller = Marshaller.delegate[Foo, String](`text/plain`) { _.toString }

val supportedContentTypes = List[ContentType](`application/json`, `text/xml`, `text/plain`)

implicit val marshaller =
Marshaller[Foo] { (foo, ctx) =>
ctx.tryAccept(supportedContentTypes) match {
case Some(`application/json`) => jsonMarshaller(foo, ctx)
case Some(`text/xml`) => xmlMarshaller(foo, ctx)
case Some(`text/plain`) => textMarshaller(foo, ctx)
case _ => ctx.rejectMarshalling(supportedContentTypes)
}
}
}

We realize that this is not yet as easy and straightforward as it could (and should) be.
There already is a ticket for a larger refactoring of the (Un)Marshalling infrastructure, which will also make "composed" (un)marshallers much easier:
https://github.com/spray/spray/issues/293

> first, how can I make this Marshaller be implicitly available in the
> Main object without declaring the variable that references it? I tried
> declaring the Marshaller as implicit val FooMarshaller and then saying import
> FooObj._ in the Main class, which seems as if it should work to me, but
> didn't.

As shown above the best way is usually to put the `Marshaller[Foo]` directly into the companion object of Foo.
Then it will automatically in scope while still be "overridable" by explicitly importing another `Marshaller[Foo]` if needed.

> Second, is there a way I can change the name of FooObj to Foo instead, so
> that it becomes the companion to the case class Foo? Changing its name as
> such causes jsonFormat2() to fail for reasons I have not been able to
> figure out; apparently the parameter of jsonFormat2() is a function
> returning a Foo rather than a Foo-type value, so maybe there's a way to
> rewrite the invocation of jsonFormat2 so its argument is explicitly a
> function literal?

Yes, you were missing the `There is one additional quirk:` paragraph in the spray-json docs:
https://github.com/spray/spray-json.

HTH and cheers,
Mathias

---
mat...@spray.io
http://spray.io

> The reason I'm asking is this passage from the Marshalling documentation<http://spray.io/documentation/1.2-M8/spray-httpx/marshalling/>
> :

>
> you best define the Marshaller for T in the companion object of T. This way

>> your marshaller is always in-scope, without anyimport tax<http://eed3si9n.com/revisiting-implicits-without-import-tax>
>> .

>
>
> So ought I to be putting this FooMarshaller in the companion object to the
> case class Foo? And if so, how do I then get a JsonWriter[Foo]?
>
> Thanks very much,
> --
> Adam Mackler
>

> --
> You received this message because you are subscribed to the Google Groups "spray-user" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to spray-user+...@googlegroups.com.
> For more options, visit https://groups.google.com/groups/opt_out.

[Adam Mackler]

On Tuesday, September 17, 2013 5:50:33 AM UTC-4, Mathias wrote:

Yes, you were missing the `There is one additional quirk:` paragraph in the spray-json docs:
https://github.com/spray/spray-json.

HTH and cheers,
Mathias

Awesome!!  Thanks so much; that example feels great.  I knew it had to be something like that.  Incidentally, as someone who is still trying to get my head around type classes, I found section 7.3 of Joshua Suereth's "Scala in Depth" to be very helpful.  One thing in particular got my attention: apparently the Marshaller trait could be annotated like this:

@annotation.implicitNotFound(msg = "No Marshaller for ${T} found in scope")

and it will fix the wording of the possibly confusing default error message.

That example you gave would go great in the Marshaller and/or routing documentation. I'll also mention that I had never even seen the README for spray-json, which contains much more information than the documentation site's page on spray-json support.  A link to the former in the latter as well as in the scaladocs would be valuable.

Thanks again!

--

Adam Mackler

[Adam Mackler]

Question:

On the spray-json README, under heading "Usage" there's a example line of code with a comment that reads 

import DefaultJsonProtocol._ // !!! IMPORTANT, else `convertTo` and `toJson` won't work

Is that comment correct about toJson?  I see toJson is a member of PimpedAny, and the implicit conversion to PimpedAny is in the spray.json package object, not DefaultJsonProtocol, so wouldn't the necessary import for making toJson work be spray.json._?

In any event import spray.json._ got toJson working for me; import DefaultJsonProtocol._ did not.

--

Adam Mackler

[Age Mooij]

Hi Adam,

It's not that those functions are declared in DefaultJsonProtocol but they won't be very useful without importing it because all the default implementations of the JsonFormat type classes are exposed through it, as well as the jsonFormat1, etc. methods used for converting your own case classes.

I think a better wording would have been "won't work correctly".

Age

[Johannes Rudolph]

On Wed, Sep 18, 2013 at 9:35 AM, Age Mooij <age....@gmail.com> wrote:
> I think a better wording would have been "won't work correctly".

Thanks for the suggestion Age, I fixed that.

--
Johannes

-----------------------------------------------
Johannes Rudolph
http://virtual-void.net

[hbf]

I have run into similar question as the OP and I wanted to share my code, in the hope somebody else may find it helpful:

trait TestService extends spray.routing.HttpService {

  // The case classes to marshall/unmarshall

  case class Point(x: Int, y: Int)

  object JsonProtocols extends spray.json.DefaultJsonProtocol {

    implicit val point = jsonFormat2(Point)

    // ... more of these, if you have more case classes to marshall/unmarshall

  }

  val supportedContentTypes = List[ContentType](`application/json`, `text/plain`)

  implicit def marshaller[T](implicit fmt: RootJsonFormat[T]) =

    Marshaller[T] { (foo, ctx) =>

      supportedContentTypes.mapFind(ctx.tryAccept).map {

        _.mediaType match {

          case MediaTypes.`application/json` => spray.httpx.SprayJsonSupport.sprayJsonMarshaller.apply(foo, ctx)

          case MediaTypes.`text/plain` => Marshaller.delegate[T, String](`text/plain`) { _.toString } .apply(foo, ctx)

          case _ => ctx.rejectMarshalling(supportedContentTypes)

        }

      }

    }

  import JsonProtocols._

  val route =

      pathPrefix("test") {

        get {

          complete(Point(1,2))

        }

      }

}

Cheers,

Kaspar

[Mathias Doenitz]

Thanks, Kaspar!

Cheers,

Mathias

---
mat...@spray.io
http://spray.io

>> mat...@spray.io <javascript:>
>> http://spray.io
>>
>> On 17.09.2013, at 07:10, Adam Mackler <adamm...@gmail.com <javascript:>>

>> an email to spray-user+...@googlegroups.com <javascript:>.

[Adam Mackler]

On Monday, September 30, 2013 5:36:53 AM UTC-4, Mathias wrote:

Thanks, Kaspar!

 

Yes, thank you.  That's helpful to see, and gosh I feel selfish now.   Here's the current state of what I've come up with: I have a base class, then a class for each type of resource that needs to be (un)marshalled.  There's a JSON marshaller, and an XML marshaller, and then an HTML marshaller that runs the output of the XML marshaller through an XSLT translation.   Each subclass defines (1) the method toXml, invoked by the superclass XML marshaller, and which uses XML literals to define the XML marshalling; (2) a value member whose value is the name of the XSLT sheet to use; and (3) a member value, prepareHtmlTransformer, that is a function can be overridden to set parameters on the XSLT translation for things that are in the HTML but not the XML representation.

First the base class, Representation:

abstract class Representation[T] extends MyJsonProtocol with SprayJsonSupport {

  val log = org.slf4j.LoggerFactory.getLogger(this.getClass)

  val acceptedContentTypes = List[ContentType] (

    `application/xhtml+xml`,

    `application/xml`,

    `text/xml`,

    `application/json`

  )

  val xmlMarshaller = Marshaller.delegate[T, String](

    `text/xml`,`application/xml`

  ) { (obj: T) => toXml(obj).toString }

  protected def toXml(obj: T): scala.xml.Elem = <error/>

  protected val xsltFilename: String = ""

  protected val prepareHtmlTransformer: Function[XsltTransformer,Unit] = {

    (x:XsltTransformer) =>

  }

  /** Is XHTML not HTML */

  val htmlMarshaller: Marshaller[T] = Marshaller.delegate[T, String](

    `application/xhtml+xml`

  ){ (obj,contentType) =>

      val xmlReader = new PipedReader

      val xmlWriter = new PipedWriter(xmlReader)

      new Thread { override def run {

try {

 XML.write(xmlWriter, toXml(obj), "UTF-8", false, null)

} catch {

 case e: Exception => log.error(e.getMessage)

} finally {

 xmlWriter.close()

}

      }}.start

      val xsltTransformer = xsltTransformerDef(xsltFilename)

      prepareHtmlTransformer(xsltTransformer)

      xsltTransformer.setSource(new javax.xml.transform.stream.StreamSource(xmlReader))

      val stringWriter = new java.io.StringWriter()

      xsltTransformer.setDestination(new Serializer(stringWriter))

      xsltTransformer.transform()

      stringWriter.toString

  }

  // Translation sheet will be reloaded on every request.

  // Uncomment next line and delete following line for production use. 

  //private lazy val xsltTransformer: net.sf.saxon.s9api.XsltTransformer = {

  def xsltTransformerDef(filename: String) = {

    val processor = new net.sf.saxon.s9api.Processor(false)

    val compiler = processor.newXsltCompiler

    compiler.compile {

      new javax.xml.transform.sax.SAXSource (

        scala.xml.Source fromFile filename

      )

    } load()

  }

  val jsonMarshaller: Marshaller[T]

  implicit val marshaller = Marshaller[T] {

    (obj: T, ctx: MarshallingContext) =>

      (ctx.tryAccept(acceptedContentTypes): @unchecked) match {

case Some(ContentType(`application/xhtml+xml`,_)) =>

 htmlMarshaller(obj, ctx)

case Some(ContentType(`application/json`,_)) => jsonMarshaller(obj, ctx)

case Some(ContentType(`application/xml`,_)) | Some(ContentType(`text/xml`,_)) =>

 xmlMarshaller(obj, ctx)

case _ => ctx.rejectMarshalling(acceptedContentTypes)

      }

  }

}

Then, for example, people requesting the root path of my site get a representation of the Index class.

import spray.json.BasicFormats

import collection.JavaConversions._

case class IndexEntry( name: String, foo: Int, etc... )

class Index { entries: Set[IndexEntry] = getEntries }

object Index extends Representation[Index] {

  import spray.json._

  def apply = new Index()

  override def toXml(index: Index): scala.xml.Elem =

    <index>{ index.entries.map(e =>

      <entry name={e.name}

             foo={e.foo}

      /> )

    }</index>

  override val xsltFilename: String = "src/main/resources/index.xsl"

  override val prepareHtmlTransformer = { (xsltTransformer: XsltTransformer) =>

    xsltTransformer.setParameter (new QName("myParam"), paramValue)

  }

  val jsonMarshaller = sprayJsonMarshaller(

    new spray.json.RootJsonWriter[Index] {

      implicit val entryFormat = jsonFormat7(IndexEntry)

      def write(index: Index) = index.entries.toList.toJson

    }

  )

}

I still have some implementing to do, but the general outline of a type that unmarshals so far is looking something like this:

case class Foo (

  name   : String,

  date   : DateTime

)

object Foo extends Representation[Foo] {

  import spray.json._

  import spray.httpx.unmarshalling._

  implicit val fooJsonFormat = jsonFormat5(Foo.apply)

  val jsonMarshaller = sprayJsonMarshaller(fooJsonFormat)

  implicit val unmarshaller = Unmarshaller[Foo] (

    `application/json`,`application/xml`,`application/x-www-form-urlencoded`

  ) {

    case entity @ NonEmpty(ContentType(mediaType,charset), bytes) => mediaType match {

      case `application/x-www-form-urlencoded` =>

val formFields = entity.as[FormData] match {

 case Right(formData) => formData.fields

 case Left(ContentExpected) => throw new Exception("Missing content")

 case Left(MalformedContent(message,cause)) => throw new Exception(message)

 case Left(UnsupportedContentType(errorMessage)) => throw new Exception(errorMessage)

}

        val name = formFields("name")

        val date = formFields("date")

        Foo(name, date)

      case `application/json` => throw new Exception("json not implemented")

      case `application/xml` => throw new Exception("xml not implemented")

      case o => throw new Exception(s"Unrecognized media type $o")

    }

  }

}

--

Adam Mackler

[Adam Mackler]

Couple followup points:

1. Kaspar: a line in your marshaller is:

On Sunday, September 29, 2013 8:08:13 PM UTC-4, hbf wrote:

      supportedContentTypes.mapFind(ctx.tryAccept).map { 

 

I had to change it to:

ctx.tryAccept(supportedContentTypes).map {

I don't know if I'm missing something.

2. I noticed the code that I posted refers to a missing trait, MyJsonProtocol.  I'm a big fan of JodaTime.  Here's the missing trait.

import spray.json._

trait DateTimeJsonFormat extends JsonFormat[DateTime] {

  private val dateTimeFmt = org.joda.time.format.ISODateTimeFormat.dateTime

  def write(x: DateTime) = JsString(dateTimeFmt.print(x))

  def read(value: JsValue) = value match {

    case JsString(x) => dateTimeFmt.parseDateTime(x)

    case x => deserializationError("Expected DateTime as JsString, but got " + x)

  }

}

trait LocalDateJsonFormat extends JsonFormat[LocalDate] {

  def write(x: LocalDate) = JsString(x.toString)

  def read(value: JsValue) = value match {

    case JsString(x) => org.joda.time.LocalDate.Parse(x)

    case x => deserializationError("Expected LocalDate as JsString, but got " + x)

  }

}

trait MyJsonProtocol extends DefaultJsonProtocol {

  implicit object dtf extends DateTimeJsonFormat

  implicit object ldf extends LocalDateJsonFormat

}

--

Adam Mackler

[hbf]

Adam, thank you for sharing your code and the comments!

On Tuesday, 1 October 2013 04:22:39 UTC-7, Adam Mackler wrote:

Couple followup points:

1. Kaspar: a line in your marshaller is:

On Sunday, September 29, 2013 8:08:13 PM UTC-4, hbf wrote:

      supportedContentTypes.mapFind(ctx.tryAccept).map { 

I am using Spray 1.2-M8 (see MarshallingContext.scala). mapFind is from import spray.util._ 

Cheers!

Kaspar

[Adam Mackler]

On Tuesday, October 1, 2013 11:38:39 AM UTC-4, hbf wrote:

      supportedContentTypes.mapFind(ctx.tryAccept).map { 

I am using Spray 1.2-M8 (see MarshallingContext.scala). mapFind is from import spray.util._ 

I see mapFind, but tryAccept takes one argument of type Seq[ContentType] . But  mapFind takes an argument of type A => Option[B], where A is the parameterized type of the Seq, and mapFind then wants to apply tryAccept to each element of the Seq one at a time.  So if I pass tryAccept as the argument to mapFind, then tryAccept would see each ContentType one-at-a-time, not all-at-once as the Seq[ContentType] it wants, and I get an error

type mismatch;

[error]  found   : Seq[spray.http.ContentType] => Option[spray.http.ContentType]

[error]  required: spray.http.ContentType => Option[?]

In other words, the argument to mapFind is function that takes a ContentType, but tryAccept takes a Seq[ContentType] not a ContentType.  So I don't see how you can get that to compile.

--

Adam Mackler

[Lily]

Hi, Mathias:

在 2013年9月17日星期二 UTC+8下午5:50:33，Mathias写道：

Adam,

your questions are very valid, so let me go through them one by one:

> I have no problem creating a Marshaller that will produce one type, say XML or JSON, but how do I create one Marshaller to handle either depending on the request?

spray relies completely on type-classes for associating marshaling logic with a type. Type-classes are resolved statically at compile-time, not at runtime, which means that your program cannot pick a different marshaler for your type depending on what the client sent along in the `Accept` header of the request.

----- "pick a different marshaler for your type depending on what the client sent along in the `Accept` header of the request" Is this support now? 

 

There must be exactly one marshaler for your type, which is able to produce *all* supported resource representations (JSON, XML, …).

However, this does not mean that this one marshaller cannot delegate (parts of) the job of actually producing the representations to other (sub-)marshallers.
Here is an example (untested):

-----Does the example  mean this one marshaller can realize that the program cannot pick a different marshaler for your type depending on what the client sent along in the `Accept` header of the reques. Thank you very much!