Building Record interface to REST api

[pelagic]

Hey Everyone,

I'm building a Record interface for use with RESTful api backends and would like some advice. So far I've been using JSONRecord from the couchdb-record project, which has turned out great for serializing json into Records, and Databinder Dispatch for HTTP stuff. 

Right now I'm trying to figure out the best way to define the different endpoints (uri) associated with a record. For right now I would like to handle the following cases:

/foo

/foo/:id

/foo/bar

/foo/:id/bar

Currently I've chosen to build the uri via three methods.

trait RestRecord[MyType <: RestRecord[MyType]] extends JSONRecord[MyType] {

   

  /**  Defines the RESTful endpoint for this resource: /foo   */

  val uri: List[String]

  

  /**  Defines the RESTful suffix for endpoint: /foo/:id/bar    */

  val uriSuffix: List[String] = Nil

  /**  Defines and uri identifier for this resource: /foo/:id or /foo/:id/bar   */

  def id: Box[String] = Empty

}

Using HTTP verbs and the endpoints I can find, create, update and delete the record. The uri is also partially defined my the HTTP verb:

GET         uri ::: maybe the id ::: uriSuffix

PUT         uri ::: maybe the id ::: uriSuffix

POST       uri ::: uriSuffix    // no id because it hasn't been created yet

DELETE  uri ::: maybe the id ::: uriSuffix

Now you can define Records like this

/** /foo/:id  */

class Foo extends RestRecord[Foo] {

  override val uri = "foo" :: Nil

  override def id = Full(id.is)

  object id extends StringField(this)

}

/** /foo/:id/bar  */

class Bar extends RestRecord[Bar] {

  override val uri = "foo" :: Nil

  override val uriSuffix = "bar" :: Nil

  override def id = Full(id.is)

  object id extends OptionalStringField(this)

}

Does this seems like a reasonable approach to define the uri? I would love to hear any feedback.

Thanks for you help and time,

Greg

[Tobias Pfeiffer]

Hi,

Am Thu, 4 Oct 2012 15:10:54 -0700 (PDT) schrieb pelagic:
> I'm building a Record interface for use with RESTful api backends and

> would like some advice. [...] I would love to hear any feedback.

I recently read "REST in Practice" from Webber/Parastatidis/Robinson
<http://shop.oreilly.com/product/9780596805838.do> and one thing I
learned from that book is that it's not necessarily a good idea to
abstract away the "remote" character of a remote RESTful service.

Quoting them:
"Unfortunately, WSDL’s conceptual model and the naïve tooling based
around it attempt to hide distribution as though programmers need to be
protected from it. [Footnote: Nothing could be further from the truth.
We need to know about distribution boundaries so that we can write code
that deals gracefully with failures outside our control.] In hiding the
remote aspects of a distributed system, we hide necessary complexity
to the extent that we can’t build services that are tolerant of their
inherent latencies, failure characteristics, and ownership
boundaries." (p. 385)

"The classic paper by Waldo et al. emphasizes that distribution cannot
be safely hidden from developers. [Footnote:
http://research.sun.com/techrep/1994/abstract-29.html] If you try to
hide distribution in a distributed system, the abstractions will leak
at inconvenient times and cause significant problems. It’s best to
acknowledge the distribution in distributed systems—it is necessary
complexity— and to plan accordingly." (p. 409)

In the recent past, I've built two Lift applications that get (part of)
their data from RESTful backend services. In one, a 500 error or a
delayed response from the REST service will cause problems all over the
place, probably because I tried too hard to hide the REST data source
behind a normal model class. In the other one, I absolutely wanted the
application to continue running when the backend service has problems,
so I basically Box'ed everything and I'm now able to run that whole
application even when the backend service is not running. Probably there
are still better solutions, though ;-)

While I think it would be great to treat a RESTful CRUD service just
like any other Record, I'd keep in mind what should happen if, say, the
server needs 5 seconds to send you certain resource data.

Just my 2 cent
Tobias

[pelagic]

Hey Tobias,

Thanks for you input, I really appreciate you sharing your experience on working with RESTful services.

Could you ellaborate more on what kind of problems you were having

>In one, a 500 error or a delayed response from the REST service will cause problems all over the
>place, probably because I tried too hard to hide the REST data source
>behind a normal model class

I'm currently working on a app that is backed entirely by an internal restful api and I'd really like to incorporate all the awesomeness of Record. So for me it might be worth it go down this road, maybe not, i'm trying to figure that out.

Greg

[Tobias Pfeiffer]

Hi Greg,

Am Freitag, 5. Oktober 2012, 18:02 schrieb pelagic:
> Thanks for you input, I really appreciate you sharing your experience
> on working with RESTful services.
>
> Could you ellaborate more on what kind of problems you were having

Well, what you might be tempted to do if you want to map a model class
to a REST resource, is the following:

class Article extends LongKeyedMapper[Article] with IdPK {
lazy val data: NodeSeq = (fetch XML data from RESTful API using id)

def title = data \ "title" text
def body = data \ "body" text
def numberOfCcomments = asLong(data \ "comments" text) getOrElse 0L
// ...
}

And then in your source code, you just treat that class as a normal
mapper class, as in:

Article.find(By(Article.id, 7L)).get.title

and let's assume the ".get" is not a problem because you know there is
an entry with id 7, then this will work fine – if your backend is up and
running. If it takes a long time to respond, then the statement blocks,
and if it returns a response code that your code is not prepared to deal
with (for example, Http(myRequest as_str) in dispatch 0.8.x), then you
will get an exception. If the distributedness of your data store is
abstracted away and you treat Article as any other Mapper/Record class,
then you easily step into that trap.


In a second project where I was more aware of that issue, I defined my
REST-mapped model classes more like this:

class Product extends LongKeyedMapper[Product] with IdPK {
lazy val data: Box[NodeSeq] = tryo {
(try to fetch XML data using id)
}

def title = data.map(_ \ "name" text).getOrElse("missing title")
def price = asDouble(data.map(_ \ "list_price" text).getOrElse("0.0")
).getOrElse(0.0)
// ...
}

Here, I still have the problem of blocking, but I get around this by
LazyLoading snippets that show product data as much as possible. On the
other hand, if the service is down or returns an unexpected response
code, my page will still work fine, just display "missing title"
everywhere. Just need to make sure that nobody buys three items "missing
title" with a total price of 0.0 EUR ;-)


Now I'm also wondering what the best way to model this situation
actually is. Right after I sent my last email, I discovered that in
dispatch 0.9.x, network delays and failures are dealt with by wrapping
responses to HTTP requests into Promise objects. So maybe (!) one of the
following is the "right" way to do it?

class ProductOption extends LongKeyedMapper[ProductOption] with IdPK {
lazy val data: Promise[Option[xml.Elem]] =
Http(myRequest OK as.xml.Elem).option

def title: Promise[Option[String]] = data.map(_.map(_ \ "name" text))
def price: Promise[Option[Double]] = data.map(_.map(_ \ "price"
text).flatMap(asDouble))
}

OR:

class ProductEither extends LongKeyedMapper[ProductEither] with IdPK {
lazy val data: Promise[Either[String, xml.Elem]] = {
val res = Http(myRequest OK as.xml.Elem).either
for (exc <- res.left) yield "ERROR: " + exc.getMessage
}

def title: Promise[Either[String, String]] = {
def extractTitle(xml: scala.xml.Elem): Either[String, String] = {
val nodes = xml \ "title" map (_.text)
nodes.headOption.toRight("invalid data")
}
for {
xml <- data.right
t <- Promise(extractTitle(xml)).right
} yield t
}

def price: Promise[Either[String, Double]] = {
def extractPrice(xml: scala.xml.Elem): Either[String, Double] = {
val nodes = xml \ "price" map (_.text) flatMap (asDouble)
nodes.headOption.toRight("invalid data")
}
for {
xml <- data.right
t <- Promise(extractPrice(xml)).right
} yield t
}

// ...
}

Both of these methods neither block nor do they throw exceptions if
something goes wrong in the backend. In particular the latter method
always keeps the error message, in case you want to log or display it.
However, you haven't dealt with updating or saving your data yet... ;-)

Tobias

[Tobias Pfeiffer]

Hi again,

I just realized that actually using Lift's Box gives you the best of the
Option and Either world, as in:

class ProductBox extends LongKeyedMapper[ProductBox] with IdPK {
lazy val data: Promise[Box[xml.Elem]] = {

val res = Http(myRequest OK as.xml.Elem).either

res.map(r => r match {
case Left(t) => Failure(t.getMessage, Full(t), Empty)
case Right(x) => Full(x)
})
}

def title: Promise[Box[String]] = data.map(_.map(_ \ "name" text))
def price: Promise[Box[Double]] = data.map(_.map(_ \ "price"
text).flatMap(asDouble))
// ...
}

Kind regards
Tobias

[pelagic]

Hey Tobias,

Thanks a ton for the response. Using a Promise seems like a slick way of handling all of the connections to the backend. My current approach is similar to the one where you wrap everything in a Box, expect I'm not using the non-blocking io.

Thanks again,
Greg