[Play 2 - Scala] JSON Validation

[Yann Simon]

Hi,

I'm using Json.Reads API to validate a JSON

(http://www.playframework.com/documentation/2.1.x/ScalaJsonCombinators)

The API is very good.

Some questions:

- how do the validation of one field can depend on another field

For example:

the field 'fieldA' or the field 'fieldB' must exist

- what is the difference between "andKeep" and "keepAnd"?

I did not see any practical difference.

- is it possible to transform the value of one field before validating it.

For example, I wand to trim() a value before checking for the length.

Do I have to use JSON transformers before using Json.Reads?

(http://www.playframework.com/documentation/2.1.x/ScalaJsonTransformers)

Thanks!

Yann

[Yann Simon]

up

2013/9/4 Yann Simon <yann.s...@gmail.com>

[Pascal Voitot Dev]

missed your mail (holidays ;))

On Wed, Sep 4, 2013 at 2:58 PM, Yann Simon <yann.s...@gmail.com> wrote:

Hi,

I'm using Json.Reads API to validate a JSON

(http://www.playframework.com/documentation/2.1.x/ScalaJsonCombinators)

The API is very good.

Some questions:

- how do the validation of one field can depend on another field

For example:

the field 'fieldA' or the field 'fieldB' must exist

You can't validate a field depending on another one because validation is always contextual and relative. It's not aware of the rest of the Json tree.
But you can go around it:

For ex:

__.json.pick keepAnd

((__ \ "fieldA").read[Int] and (__ \ "fieldB").read[Int]).tupled.filter(ValidationError("fieldA should be > fieldB)){ case (a,b) => a > b }

 

- what is the difference between "andKeep" and "keepAnd"?

I did not see any practical difference.

"Reads1 andKeep Reads2" validates Reads1 and Reads2 and if both succeeds, it returns only result of Reads1
"Reads1 keepAnd Reads2" validates Reads1 and Reads2 and if both succeeds, it returns only result of Reads2

(same as ~> and <~ in parser combinators for the analogy)

 

- is it possible to transform the value of one field before validating it.

For example, I wand to trim() a value before checking for the length.

you must read it first to a type and then validate it.

(__ \ "field").read[String].map(_.trim).filter(ValidationError("toto"))( a => // your validation }.map( JsString(_) )

If you want to get Json branch with validation, you can do the following.

(__ \ "field").read[String].map(_.trim).filter(ValidationError("toto"))( a => // your validation } andKeep (__ \ "field").json.pickBranch

You read the branch twice but the code is clear.

Do I have to use JSON transformers before using Json.Reads?

(http://www.playframework.com/documentation/2.1.x/ScalaJsonTransformers)

Json Transformers is just a way to Validate Json to Json ie Reads[JsValue].
It's just helpers actually.

regards
Pascal

 

Thanks!

Yann

--
You received this message because you are subscribed to the Google Groups "play-framework" group.
To unsubscribe from this group and stop receiving emails from it, send an email to play-framewor...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

[Yann Simon]

Thanks for the answer.

Still one question:

2013/9/11 Pascal Voitot Dev <pascal.v...@gmail.com>

my initial idea was that:

  def trim(implicit reads:Reads[String]) = reads.map(_.trim)

  def notEmptyString(implicit reads:Reads[String]) = Reads.filterNot[String](ValidationError("mandatory"))( _.isEmpty)(reads)

  (__ \ "field").read[String](trim andKeep notEmptyString)

but it cannot work.

The reads in parenthesis can only be validator?

[Pascal Voitot Dev]

it should work!

don't forget to import:

import play.api.libs.json._
import play.api.libs.functional.syntax._

this is what imports all functional tools to allow "and" and "andKeep/keepAnd"

[Yann Simon]

it compiles, it is not the problem.

The problem is that the string is not trimmed in the second validation.

[Pascal Voitot Dev]

On Wed, Sep 11, 2013 at 3:01 PM, Yann Simon <yann.s...@gmail.com> wrote:

it compiles, it is not the problem.

The problem is that the string is not trimmed in the second validation.

Which is quite normal. And/andKeep/keepAnd compose Reads but doesn't chain them!

map is done for this (or andThen but it's limited to "Reads[JsValue] andThen Reads[A])"

  (__ \ "field").read[String].map(_.trim).filterNot[String](

ValidationError("mandatory"))( _.isEmpty)

[Yann Simon]

2013/9/11 Pascal Voitot Dev <pascal.v...@gmail.com>

On Wed, Sep 11, 2013 at 3:01 PM, Yann Simon <yann.s...@gmail.com> wrote:

it compiles, it is not the problem.

The problem is that the string is not trimmed in the second validation.

Which is quite normal. And/andKeep/keepAnd compose Reads but doesn't chain them!

map is done for this (or andThen but it's limited to "Reads[JsValue] andThen Reads[A])"

  (__ \ "field").read[String].map(_.trim).filterNot[String](

ValidationError("mandatory"))( _.isEmpty)

OK, I understand.

And how can you write the example like that:

(__ \ "field").read[String].map(_.trim)(notEmptyString andKeep otherValidation1 andKeep otherValidation2)

?

Can we use andThen for this?

(__ \ "field").read[String].map(_.trim) andThen (notEmptyString andKeep otherValidation1 andKeep otherValidation2)

(does not compile)

In English:

"first, trim the value, then check the trimmed value against some validations"

[Pascal Voitot Dev]

On Wed, Sep 11, 2013 at 3:28 PM, Yann Simon <yann.s...@gmail.com> wrote:

2013/9/11 Pascal Voitot Dev <pascal.v...@gmail.com>

On Wed, Sep 11, 2013 at 3:01 PM, Yann Simon <yann.s...@gmail.com> wrote:

it compiles, it is not the problem.

The problem is that the string is not trimmed in the second validation.

Which is quite normal. And/andKeep/keepAnd compose Reads but doesn't chain them!

map is done for this (or andThen but it's limited to "Reads[JsValue] andThen Reads[A])"

  (__ \ "field").read[String].map(_.trim).filterNot[String](

ValidationError("mandatory"))( _.isEmpty)

OK, I understand.

And how can you write the example like that:

(__ \ "field").read[String].map(_.trim)(notEmptyString andKeep otherValidation1 andKeep otherValidation2)

?

Can we use andThen for this?

(__ \ "field").read[String].map(_.trim) andThen (notEmptyString andKeep otherValidation1 andKeep otherValidation2)

(does not compile)

In English:

"first, trim the value, then check the trimmed value against some validations"

and this?
(__ \ "field").read[JsString].map{ case JsString(s) => JsString(s.trim) } andThen (notEmptyString andKeep otherValidation1 andKeep otherValidation2)

(__ \ "field").read[JsString] == (__ \ "field").json.pick[JsString]

But you could also just read a String and use filter with function composition

notEmptyString = { s: String => !s.isEmpty }

and use function composition too...

[René Grüner Vangsgaard]

Hi Pascal, thank you for showing how to trim the values - just what I was looking for.

With no intend to hijack the thread, do you know if something similar can be done with HTML forms?

-René

[Pascal Voitot Dev]

On Wed, Sep 11, 2013 at 10:00 PM, René Grüner Vangsgaard <rene.va...@gmail.com> wrote:

Hi Pascal, thank you for showing how to trim the values - just what I was looking for.

With no intend to hijack the thread, do you know if something similar can be done with HTML forms?

As evoked in Play roadmap, we (mainly Julien Tournay and a bit me as moral supporter) are currently working on generalizing Json validation API to other types of data such as forms and if it's good enough, it will be released in Play 2.3... Be patient a bit ;)
(teasing: we will present some parts of this new API at scala.io end of October in Paris)

 

regards

Pascal

[Yann Simon]

direct function composition would not work, as the type of the second function would be Boolean => ... and not String => ...

What I did:

def every[A](predicates: (A => Boolean)*) = (a: A) => predicates.foldLeft(true){ case(valid, predicate) => valid && predicate(a) }

val notEmpty = { s: String => !s.isEmpty}

def maxLength(max: Int) = { s: String => s.length <= max }

val fieldValid = every(notEmpty, maxLength(20))

(__ \ "field").read[JsString].map(_.trim).filter(fieldValid)

With this solution, it is not possible to return a different validation error message for each validation.

But in some case, it is not needed.

Thanks for the help!

[Yann Simon]

2013/9/11 Pascal Voitot Dev <pascal.v...@gmail.com>

missed your mail (holidays ;))

On Wed, Sep 4, 2013 at 2:58 PM, Yann Simon <yann.s...@gmail.com> wrote:

Hi,

I'm using Json.Reads API to validate a JSON

(http://www.playframework.com/documentation/2.1.x/ScalaJsonCombinators)

The API is very good.

Some questions:

- how do the validation of one field can depend on another field

For example:

the field 'fieldA' or the field 'fieldB' must exist

You can't validate a field depending on another one because validation is always contextual and relative. It's not aware of the rest of the Json tree.
But you can go around it:

For ex:

__.json.pick keepAnd

((__ \ "fieldA").read[Int] and (__ \ "fieldB").read[Int]).tupled.filter(ValidationError("fieldA should be > fieldB)){ case (a,b) => a > b }

thanks for the info.

My 2 cents: I personally find it quite complex, compared to ad-hoc constraints in forms (http://www.playframework.com/documentation/2.1.x/ScalaForms)

It would be quite good to have this simplicity in json.

[Pascal Voitot Dev]

On Thu, Sep 12, 2013 at 10:45 AM, Yann Simon <yann.s...@gmail.com> wrote:

2013/9/11 Pascal Voitot Dev <pascal.v...@gmail.com>

missed your mail (holidays ;))

On Wed, Sep 4, 2013 at 2:58 PM, Yann Simon <yann.s...@gmail.com> wrote:

Hi,

I'm using Json.Reads API to validate a JSON

(http://www.playframework.com/documentation/2.1.x/ScalaJsonCombinators)

The API is very good.

Some questions:

- how do the validation of one field can depend on another field

For example:

the field 'fieldA' or the field 'fieldB' must exist

You can't validate a field depending on another one because validation is always contextual and relative. It's not aware of the rest of the Json tree.
But you can go around it:

For ex:

__.json.pick keepAnd

((__ \ "fieldA").read[Int] and (__ \ "fieldB").read[Int]).tupled.filter(ValidationError("fieldA should be > fieldB)){ case (a,b) => a > b }

thanks for the info.

My 2 cents: I personally find it quite complex, compared to ad-hoc constraints in forms (http://www.playframework.com/documentation/2.1.x/ScalaForms)

It would be quite good to have this simplicity in json.

 

my code seems complex because I wrote it in one-liner. Moreover, I keep the Json in input and output which makes it somewhat strange but generally you don't do that.

// my interdependent field validator

val validateDependentFields = (

  (__ \ "fieldA").read[Int] and
  (__ \ "fieldB").read[Int]
).tupled.filter(ValidationError("fieldA should be > fieldB)){ case (a,b) => a > b }

case class MyType(...)
object MyType {
  implicit val reader = Json.reads[MyType] keepAnd validateDependentFields

}

Don't hesitate to compose validators, to store them and reuse them...

Actually Form API seems simple (for simple cases) but it's quite restricting and not easy to extend/compose... Json API is much more consistent than Form actually.

Pascal

[Julien L.]

Hi Pascal,

Thanks for these exemple, very instructive. I hurted the same problem few days ago, and I still don't understand one point, maybe you could enlighten me

__.reads[String]

res4: play.api.libs.json.Reads[String]

scala> __.read[String](Reads.email)

res5: play.api.libs.json.Reads[String]

__.read[String].map(s => s)

res7: play.api.libs.json.Reads[String] // Looks like the same reader as res4 !

scala> __.read[String].map(s => s)(Reads.email)

<console>:14: error: play.api.libs.json.Reads[String] does not take parameters

              __.read[String].map(s => s)(Reads.email)

Why the map(s => s) make the reader not working as expected?

[Pascal Voitot Dev]

On Thu, Sep 12, 2013 at 11:02 AM, Julien L. <yotsu...@gmail.com> wrote:

Hi Pascal,

Thanks for these exemple, very instructive. I hurted the same problem few days ago, and I still don't understand one point, maybe you could enlighten me

__.reads[String]

res4: play.api.libs.json.Reads[String]

scala> __.read[String](Reads.email)

res5: play.api.libs.json.Reads[String]

__.read[String].map(s => s)

res7: play.api.libs.json.Reads[String] // Looks like the same reader as res4 !

scala> __.read[String].map(s => s)(Reads.email)

<console>:14: error: play.api.libs.json.Reads[String] does not take parameters

              __.read[String].map(s => s)(Reads.email)

Why the map(s => s) make the reader not working as expected?

let's look at signatures:

class JsPath {
  def read[T](implicit r: Reads[T]): Reads[T] = ...
}

class Reads[T] {
  def map[B](f: A => B): Reads[B] = ...
}

So the implicit Reads[T] is on JsPath.read

"map" is the map function of Reads[T]

__.read[String].map(s => s)(Reads.email)

here Reads.email is a parameter of map and map has no currified second param or implicit

__.read[String](Reads.email).map(s => s) YES

[Pascal Voitot Dev]

you're right :D

 

What I did:

def every[A](predicates: (A => Boolean)*) = (a: A) => predicates.foldLeft(true){ case(valid, predicate) => valid && predicate(a) }

val notEmpty = { s: String => !s.isEmpty}

def maxLength(max: Int) = { s: String => s.length <= max }

val fieldValid = every(notEmpty, maxLength(20))

(__ \ "field").read[JsString].map(_.trim).filter(fieldValid)

With this solution, it is not possible to return a different validation error message for each validation.

you're right too... you could call filter on each predicate...

but actually I would write like that but cutting validation in several steps and compose Reads functionally speaking:

val notEmptyString = Reads.of[String].collect(ValidationError("shouldn't be empty")){ case s if !s.isEmpty => s }

def maxLengthString(max: Int) = Reads.of[String].collect(ValidationError(s"shouldn't be more than $max")){ case s if s.length <= max => s }

val readAndTrimJsString = (__ \ "field").read[JsString].map{case JsString(s) => JsString(s.trim) }

val fullReader = readAndTrimJsString andThen (notEmptyString keepAnd maxLengthString(5))

 

Pascal

[Yann Simon]

2013/9/12 Pascal Voitot Dev <pascal.v...@gmail.com>

Yes, it was the second idea I wanted to explore.

To avoid duplicating .read[JsString].map{case JsString(s) => JsString(s.trim) }, I wrote an implicit conversion:

  class TrimJsPath(val jsPath: JsPath) {

    def readTrim(implicit r: Reads[String]) = {

      jsPath.read[JsString].map{ case JsString(s) => JsString(s.trim) }

    }

  }

  implicit def ToTrimJsPath(jsPath: JsPath) = new TrimJsPath(jsPath)

that I can use like that:

(__ \ "field1").readTrim andThen (notEmptyString keepAnd maxLengthString(5))

(__ \ "field2").readTrim andThen (notEmptyString keepAnd maxLengthString(8))

Would you see an easier way to achieve that?

And another question:

I am trying the same experience with optional fields:

(__ \ "field3").readNullable[JsString].map{ case Some(JsString(s)) => Some(JsString(s.trim)) } andThen (notEmptyString keepAnd maxLengthString(5))

That cannot compile:

"Cannot prove that Some[play.api.libs.json.JsString] <:< play.api.libs.json.JsValue."

Do you have an idea?

Many many thanks for the help!

[Pascal Voitot Dev]

no it's not bad!

 

And another question:

I am trying the same experience with optional fields:

(__ \ "field3").readNullable[JsString].map{ case Some(JsString(s)) => Some(JsString(s.trim)) } andThen (notEmptyString keepAnd maxLengthString(5))

That cannot compile:

"Cannot prove that Some[play.api.libs.json.JsString] <:< play.api.libs.json.JsValue."

This is normal: Reads[A] andThen Reads[B] expets the A <:< play.api.libs.json.JsValue (this is due to the nature of Reads[A] that is simply a function doing JsValue => JsResult[A])

 

Do you have an idea?

Can you try this? (haven't tried to compile it):

(__ \ "field3").readNullable[String](__.readTrim andThen (notEmptyString keepAnd maxLengthString(5)))
 

[Yann Simon]

It's working, thanks! 

[virtualeyes]

scala.io lineup looks quite interesting, even Guillaume Laforge from Groovy is getting in on the action

Spray and Slick talks look especially appealing.

Hmmmm, just got down here to the Landes region, to work or surf, that is the question ;-)

[Yann Simon]

2013/9/12 Yann Simon <yann.s...@gmail.com>

After some experimentation, some feedback:

I find that the validation with the JSON API is not the best for our use-case, because:

1. the API is quite complex for us (we have an imperative background, that can explain this)

2. we wanted to re-use the validation outside JSON deserialization.

At the end, I wrote some little validation helpers:

  sealed trait ValidationResult[+A]

  case class ValidationSuccess[+A](result: A) extends ValidationResult[A]

  case class ValidationError(errorKey: String) extends ValidationResult[Nothing]

  /**

   * define a simple validation.

   * A => ValidationResult[A]

   */

  trait Validation[A] extends (A => ValidationResult[A]) {

    self =>

    /**

     * combine 2 validations in one.

     * The 2nd validation is checked only the the first one is successful

     * @param nextStep the next validation

     */

    def and(nextStep: A => ValidationResult[A]): Validation[A] = new Validation[A]{

      def apply(a: A): ValidationResult[A] = {

        self.apply(a) match {

          case e @ ValidationError(_) => e

          case ValidationSuccess(value) => nextStep(value)

        }

      }

    }

  }

  object Validation {

    /**

     * creates a validation from a simple function

     */

    def apply[A](validation: A => ValidationResult[A]): Validation[A] = new Validation[A] {

      def apply(a: A) = validation(a)

    }

  }

It can be used like this:

  val trim = Validation { s: String => ValidationSuccess(s.trim) }

  val removeAllSpaces = Validation { s: String =>  ValidationSuccess(s.replaceAll("\\s", "")) }

  val notEmpty = { s: String => if (s.isEmpty) ValidationError("error.empty") else ValidationSuccess(s) }

  def length(length: Int) = { s: String => if (s.length == length) ValidationSuccess(s) else ValidationError("error.length") }

  def maxLength(max: Int) = { s: String => if (s.length > max) ValidationError("error.maxLength") else ValidationSuccess(s) }

  val field1Validation = trim and notEmpty and maxLength(20)

  val field2Validation = removeAllSpaces and notEmpty and maxLength(5)

for validation outside JSON:

field1Validation("value1") match ...

And I wrote an implicit conversion from Validation[A] to Reads[A]:

    import scala.language.implicitConversions

    implicit def validationToReads[A](predicate: Validation[A])(implicit reads:Reads[A]) =

      Reads[A]( js => reads.reads(js).flatMap { o =>

        predicate(o) match {

          case ValidationError(error) => JsError(error)

          case ValidationSuccess(value) => JsSuccess(value)

        }

      })

I can then re-use the validation in JSON like this:

val field1 = (__ \ "field1").read[String](field1Validation)

val field2 = (__ \ "field2").readNullable[String](field2Validation)

The main difference with the Reads[] is that the validations are not accumulated, only the first error is reported.

This behavior is suitable for our application, but may be not for others.

I am looking forward to see your next API design!

Cheers,

Yann

[Pascal Voitot Dev]

Don't see exactly what you mean with imperative background...

 

2. we wanted to re-use the validation outside JSON deserialization.

At the end, I wrote some little validation helpers:

  sealed trait ValidationResult[+A]

  case class ValidationSuccess[+A](result: A) extends ValidationResult[A]

  case class ValidationError(errorKey: String) extends ValidationResult[Nothing]

  /**

   * define a simple validation.

   * A => ValidationResult[A]

   */

  trait Validation[A] extends (A => ValidationResult[A]) {

    self =>

    /**

     * combine 2 validations in one.

     * The 2nd validation is checked only the the first one is successful

     * @param nextStep the next validation

     */

    def and(nextStep: A => ValidationResult[A]): Validation[A] = new Validation[A]{

      def apply(a: A): ValidationResult[A] = {

        self.apply(a) match {

          case e @ ValidationError(_) => e

          case ValidationSuccess(value) => nextStep(value)

        }

      }

    }

  }

 

It's not the same semantic as our "and" operator which is an functional builder aggregating different types :
Reads[A1] and Reads[A2] => Reads[A1~A2]

your "and" is more a "compose" on the same type:
Validation[A] compose Validation[A] => Validation[A]

But if this fits your needs, no problem for me ;)

Which is one of the main features introduced in our Json API also :D

Actually, trim isn't a validation, it's a pure transformation and you want it to be mutualized for several validation.

The main limitation of current Json validation is that you can only do:

Reads[A <: JsValue] andThen Reads[B]

but you want to do:

Validation[A <: JsValue, String] andThen Validation[String, String] andThen Validation[String, String]

But, you could implement your Validation as an Applicative Functor as we did for our JsResult and then, you would be able to take benefits of our ApplicativeBuilder facilities and you would have aggregation of your errors and mixed with your implicit conversion to our Reads, you would have something quite satisfying ;)

 

I am looking forward to see your next API design!

it's just the Json concepts generalized.

it shouldn't change the JSON API and be fully compatible!

But the new API will allow much more as it won't suppose input data is only JsValue...