Schema: Keep coerced values when errors occur

[Frederik Claus]

Is it possible to keep coerced values when an error occurs? I have further programmatic validation that would immensely benefit using coerce values such as time validation. What I currently do is check if the values are time objects and then execute some validation. If the validation creates any additional errors, I merge them with the error map from schema. Now that works if the input data is according to the schema defined and that would usually be sufficient, because it would first display an error about the wrong type and once the type is correct more semantic validation. However, I need to be able to fully validate a subset of the data while the rest may not be according to the schema. For example, I might have an a form that has several steps. On the first step the user inputs a start and end date that requires programmatic validation. Now, the validation cannot happen until the last step, because nil values in other fields that have not been filled yet are also considered as errors. How can I work around this issue? What I really need is a specification of a type and a seperate @Required annotation (thinking about Java Bean validation) where the coercing succeeds even though the @Required annotation fails, but that will not stop other fields from coercing and validation properly.

[Jason Wolfe]

Happy to try to help, but I'm not sure I understand the situation in detail.  Can you give a concrete example of a schema and datum and the output you get and would hope to see instead? 

Thanks, Jason

[Frederik Claus]

Sure. If I execute the following code:

((coerce/coercer  

   {:a {:b s/Str

        :c s/Int}}

   coerce/json-coercion-matcher)

  {:a {:b nil

       :c "1"}})

I receive

#schema.utils.ErrorContainer{:error {:a {:b (not (instance? java.lang.String nil)), :c (not (integer? "1"))}}}

What I would like to have is the error container above plus the coerced input (:c "1" -> :c 1)

{:a {:b nil

     :c 1}}

This coerced value can then be used for further programmatic validation.

This is all happening in the context of a REST API. It accepts JSON which first gets de-serialized using cheshire to "simple" clojure data-types. Route handlers usually validate the data which may involve coercion.

[Jason Wolfe]

Thanks, that's very helpful. 

Is your desire specifically that you don't want errors on nil / missing fields during coercion, or do you more generally want to separate the "successful" vs. "invalid" parts of coercion for all kinds of invalid values?  

If the former, I would probably advise just making all fields optional in your route schema and then doing a separate pass to make sure all the required values are present (possibly with a second version of the schema -- where the former could potentially be produced by a programmatic transformation of the latter).  If you want something more complicated I'll have to think about it more. 

Cheers, 

Jason

[Frederik Claus]

Thanks for your suggestions.

Ideally I want to "successful" vs. "invalid" parts of coercion, but that's probably like having a cake and eating it. Programmatic transformation of the schema also came to my mind when I searched for a solution on my own. I guess what I am really asking (still not entirely sure ;) ) is how to properly integrate Schema into a REST API. In Java Land I am used to the following: You get a string, you use jackson to de-serialize/coerce it to an object, you use Bean Validation to validate it. Both steps use the same model and since the processes are separate, validation (for example: @Required) does not interfere with the de-serialization/coercion process. Being two steps, this would be identical to programmatic transformation of the model using only optional fields and then the second "validation" pass with the original schema. My problem with the programmatic schema transformation is that it doesn't feel right to me. Let me explain:  I have a compojure middleware that uses Cheshire (= Jackson) to create Clojure datastructure, then I would use Schema with the "optional" schema and then Schema with the "original" schema. Since schema does not support "class constraints" (borrowing Bean Validation terminology here), I execute the validation fns that are attached to the schema (this is home-brewed). This is a lot of customization for a pretty standard requirement in web development.

[Tommi Reiman]

just picking one partial sentence ;)

"...how to properly integrate Schema into a REST API"

just wrote a spec-guide for the new reitit routing library: https://metosin.github.io/reitit/ring/coercion.html

If you want to use compojure, you could check out compojure-api, https://github.com/metosin/compojure-api.

Neither does the two-pass validation, but report also the original value on error. Two sweeps would mean twice as slow.

[Jason Wolfe]

Gotcha.  Unfortunately I'm not at all familiar with the existing Java Land tools, and am also sadly rather out of touch with the Clojure land tools as well.  

Generally Schema doesn't try to solve all the world's problems, but rather to be a general-purpose tool that can be adapted to many possible applications.  There might be existing third-party tools that build upon Schema to provide a more direct solution to your problem.  Or this might be solved better by the clojure.spec ecosystem, which I expect will be the target for most other Clojure libraries in the future.  (Metosin's stuff Tommi linked to is probably a good place to start for either of those options).  

That said, I'm also happy to help talking through building your own thing with Schema if you are still interested in that. (It seems like what you might really want is to coerce each field value into an Either of sorts with an error or coerced value.  But I'm not entirely sure.  If that's what you want, I think it should be pretty easy to do, but it might require making your own coercion-like thing by adapting `schema.coerce/coercer`. )

Best, Jason