JSON data validation similar to PlayFramework (Scala)?

[Jari Pennanen]

Hello!

I've been dabbling to F# lately, and wondering could I reimplement PlayFramework JSON validation/reading/writing library with F#? Has anyone tried, or have similar library already?

To me, the exciting part is the syntax. Suppose following Scala code:

  val addressJson = {

    r(__ \ 'address \ 'name)(of[JsString]) and

    ?(__ \ 'address \ 'address_1)(of[JsString]) and

    ?(__ \ 'address \ 'address_2)(of[JsString]) and

    ?(__ \ 'address \ 'code)(of[JsString]) and

    ?(__ \ 'address \ 'city)(of[JsString]) and

    ?(__ \ 'address \ 'country)(of[JsString])

  }.reduce

  val clientJson = {

    r(__ \ 'name)(of[JsString]) and

    ?(__)(addressJson) and

    ?(__ \ 'email)(of[JsString] keepAnd email)

  }.reduce

  val invoiceRowJson = (

    r(__ \ 'title)(of[JsString]) and

    ?(__ \ 'quantity)(of[JsNumber] keepAnd min(0)) and

    ?(__ \ 'quantityUnit)(of[JsString]) and

    ?(__ \ 'price)(of[JsNumber]) and

    ?(__ \ 'tax)(of[JsNumber] keepAnd min(0)) and

    ?(__ \ 'taxPrice)(of[JsNumber] keepAnd min(0)) and

    ?(__ \ 'totalTaxPrice)(of[JsNumber]) and

    ?(__ \ 'totalPrice)(of[JsNumber])).reduce

  val invoiceJson = (

    r(__ \ 'title)(of[JsString]) and

    ?(__ \ 'client)(of[JsString] keepAnd foreignkeyInCollection("client")) and

    ?(__ \ 'address)(addressJson) and

    ?(__ \ 'invoiceRows)(list(invoiceRowJson).map(JsArray(_)))).reduce

  // Note "r" is required, and "?" is optional, foreignKeyInCollection checks

  // that the client ID value is actually found in database

Above code is readable and self-documenting without knowing much about small helpers I put on the library. (It may contain bugs though I just edited it somewhat during the post)

However F# has computation expressions and type providers. I'd expect to have even cleaner syntax and parsed output value that can be accessed neatly with type providers?

I couldn't find any similar libraries, it doesn't mean there isn't any, but I'll try to implement my own.

Thanks.

- Jari

[Tomas Petricek (Info)]

Hi,

The closest thing that you’ll find in the F# world is probably the JSON type provider: http://fsharp.github.io/FSharp.Data/library/JsonProvider.html

 

The type provider gives you a nice way for reading JSON – there is some work in progress on writing JSON data too, although this is not quite complete yet. Validation would be an interesting thing to tackle! Currently, the JSON provider works by inferring the type form a sample document – which works surprisingly well, but is somewhat fragile compared to an explicitly specified schema.

 

I think that the most interesting thing to do would be to add support for some form of JSON Schema. I’m not an expert, but this seems to be a standard thing: http://json-schema.org. This way you could pass standard JSON schema to the type provider and it would generate validator/parser for you:  

 

type Sample = JsonProvider< """{

        "title": "Example Schema",

        "type": "object",

        "properties": {

               "firstName": { "type": "string" },

               "lastName": { "type": "string" },

               "age": { "description": "Age in years", "type": "integer", "minimum": 0 }

        },

        "required": ["firstName", "lastName"]

}""">

 

This includes the schema inline, but it could also be included in a separate file – which would be probably more realistic use case. Doing things this way does not involve designing any custom syntax for the specifications (unlike in Scala), but I think it’s probably more useful – if you want some validation, then JSON schema seems to be the standard thing.

 

If you find this interesting, then start an issue/discussion on F# Data: https://github.com/fsharp/FSharp.Data

 

Tomas

--
--
To post, send email to fsharp-o...@googlegroups.com
To unsubscribe, send email to
fsharp-opensou...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/fsharp-opensource
---
You received this message because you are subscribed to the Google Groups "F# Discussions" group.
To unsubscribe from this group and stop receiving emails from it, send an email to fsharp-opensou...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Mauricio Scheffer]

You could also validate the result of the JSON type provider with FSharpx (google "fsharpx validation" for many examples). With FSharp.Data you could also work with the JSON values as in your example, though I don't see why one would want to validate and then keep the "untyped" JSON instead of a typed object tree.

Another option is Fleece ( https://github.com/mausch/Fleece ), which lets you define exactly how to map JSON to your types. It's easy to add extra validation in there, as functions over Choice. 

Here's some code based on your sample domain: https://gist.github.com/mausch/10616448 which is similar to what's shown in http://semberal.github.io/json-validation-and-error-handling-in-play-2-1.html .

--

Mauricio

[Jari Pennanen]

On Monday, April 14, 2014 3:31:33 AM UTC+3, Tomas Petricek wrote:

The closest thing that you’ll find in the F# world is probably the JSON type provider: http://fsharp.github.io/FSharp.Data/library/JsonProvider.html

I looked that example and I was dumbstruck! That's how I ended up looking to recreate the PlayFramework's API with F# in the first place. 

If I understand Type Providers (and Computation Expressions) correctly, this means I could have syntax like:

let invoiceRowModel = validator {

"title" is typeString and required

"quantity" is typeNumber and min(0)

}

let invoiceModel = validator {

"title" is typeString and required

"client" is foreignKeyInCollection("client")

"address" is addressModel

"invoiceRows" is typeList<invoiceRowModel>

}

let a = invoiceModel.parseJson("... json ...")

print a.title

If I can somehow manage to figure out the F# language to write such API. That would be soo clean way to do the model definition / validation syntax.

if you want some validation, then JSON schema seems to be the standard thing.

Problem with JSON schema is that it's not extensible. How would you write a "foreingKeyInCollection" for instance with it?

It would be far better to have functional validation API like in PlayFramework, and then serialize (as much as possible) it to JSON schema if needs to be used elsewhere.

On Monday, April 14, 2014 8:06:11 AM UTC+3, Mauricio Scheffer wrote:

You could also validate the result of the JSON type provider with FSharpx (google "fsharpx validation" for many examples). With FSharp.Data you could also work with the JSON values as in your example, though I don't see why one would want to validate and then keep the "untyped" JSON instead of a typed object tree.

Another option is Fleece ( https://github.com/mausch/Fleece ), which lets you define exactly how to map JSON to your types. It's easy to add extra validation in there, as functions over Choice. 

Here's some code based on your sample domain: https://gist.github.com/mausch/10616448 which is similar to what's shown in http://semberal.github.io/json-validation-and-error-handling-in-play-2-1.html .

That's interesting, thanks. I will look on how the Fleece is done.

I like the Coast-to-Coast design in PlayFramework JSON, it means one should keep the models and classes out and keep the JSON as primary structure.

I think in F# the underlying datatype would be record. So in this coast-to-coast design, there is no models just simple record validator, which parses the json using the single validator and outputs Future[Record] or some such.

[Mauricio Scheffer]

Computation expressions admit custom operations/keywords but I don't think you could have that syntax exactly (i.e. starting each line with a string).

If you google for 'F# computation expression "customoperation"' you'll see many examples that should give you an idea of what's possible to do with computation expressions.

Still, I don't think that using English as a model for a validation DSL is a good idea (or for most programming-related things, for that matter). Either the DSL will be very limited, or will grow to a point where you'll have the English part of the DSL interacting with other parts of the language (since embedded DSLs inherit language features), and then not only will it stop looking like English but it will actually hinder readability. More about this: http://pragdave.me/blog/2008/03/10/the-language-in-domainspecific-language-doesnt-mean-english-or-french-or-japanese-or-/

Scala can sort of get away with it because its syntax is quite flexible, certainly much more flexible than F#.

I also don't think that the "coast-to-coast design" is generally a good idea. Since it doesn't capture the structure validated from the JSON in the type system, it's essentially throwing away this valuable information. 

As an example, imagine that the code in https://github.com/mandubian/play2-json-demo/blob/master/json-coast-to-coast/app/controllers/Application.scala#L118 grows to a point where you want to refactor and extract the map function to a separate, named function. This new function will have a parameter of type JsObject, which doesn't say anything about the kind of data it should have (namely, a Person in this case).

--

Mauricio

--

[Tomas Petricek (Info)]

You can definitely come up with nice syntax for specifying the schema in F# - computation expressions are not quite as flexible as you are imagining in the sample below and I’d probably prefer using F# data types (records with some annotations), but it is definitely an option.

 

However, I think that F# follows quite different philosophy here. With type providers, you can *import* data and schema from the outside. This completely removes the need for specifying the schema *again* inside your programming language. For example, you do not need to write classes to describe your database – you can just import the classes using a type provider!

 

So I think that a similar approach would be much more interesting for JSON. The big benefit is that people can start using that immediately. When you build your own DSL for describing the validation, people will have to decide to use it, learn it, etc. This is quite a blocking issue for adoption. However, if you import the schema from something that people already use (be it JSON schema or something else – I’m not an expert here), then everyone can start benefiting from that almost immediately. So that’s why I think looking into some way of importing schema from other specifications or sources would be much more interesting and better aligned with the F# philosophy.

 

That said, I’m sure that following another direction would be fun & interesting too!

 

Tomas

 

From: fsharp-o...@googlegroups.com [mailto:fsharp-o...@googlegroups.com] On Behalf Of Jari Pennanen

Sent: Monday, April 14, 2014 6:57 PM
To: fsharp-o...@googlegroups.com; fsharp-o...@googlegroups.com

--