spec and à la carte conformance of map vals (and maybe a bug)

[Josh Tilles]

Hi all,

On several occasions, Alex Miller has emphasized that we should be wary of “baking in” conformers when defining specs. I agree with him, but it’s not clear to me how to “opt in” to using conformers on map vals. As far as I can tell, I’m either limited to using unqualified keys —which enables swapping specs from alternative namespaces for a given map-entry— or adapting any conformer to operate on the whole map (e.g., with update) rather than the val of a map-entry directly.

So, some questions:

1. Am I missing something? Is there actually a way to specify conformance à la carte for the vals of a map with qualified keys? If not…
2. Is there a plan to add this functionality? (I would be happy to create a JIRA issue if it would be better to take this conversation there.) If not…
3. Is this seen as an acceptable tradeoff of an API that the core Clojure devs are otherwise happy with? Or is it in fact a deliberate limitation, in line with the “map specs should be of keysets only” design decision?

Thank you,

Josh

P.S. To confirm my understanding that there is not a way to specify conformance inline, I attempted to create and use a spec defined like (s/keys :req [(s/and ::foo (s/conformer ,,,))]). To my surprise, clojure.spec used it without complaint, although the behavior of s/conform was unaffected. Should I create a JIRA issue for this?

P.P.S. A previous draft of this email had detailed illustrative examples, cut out of fear of being tiresomely long-winded. If anything I described isn’t clear, let me know and I’ll go ahead and include the examples in the conversation.

[Alex Miller]

On Monday, December 19, 2016 at 2:42:49 PM UTC-6, Josh Tilles wrote:

Hi all,

On several occasions, Alex Miller has emphasized that we should be wary of “baking in” conformers when defining specs. I agree with him, but it’s not clear to me how to “opt in” to using conformers on map vals. As far as I can tell, I’m either limited to using unqualified keys —which enables swapping specs from alternative namespaces for a given map-entry— or adapting any conformer to operate on the whole map (e.g., with update) rather than the val of a map-entry directly.

So, some questions:

1. Am I missing something? Is there actually a way to specify conformance à la carte for the vals of a map with qualified keys? If not…

You're not missing anything.

1. Is there a plan to add this functionality? (I would be happy to create a JIRA issue if it would be better to take this conversation there.) If not…

In the way that you seem to be envisioning it, probably not.

1. Is this seen as an acceptable tradeoff of an API that the core Clojure devs are otherwise happy with? Or is it in fact a deliberate limitation, in line with the “map specs should be of keysets only” design decision?

It is deliberate that s/keys works only on attributes and that you can't inline value specs. If you really need to do a wholesale coercion of all your attributes, it seems like you can do this explicitly as a pre-step before validating the map with spec and that may be the better way to go. I'm not sure what spec is buying you over just explicitly transforming the map using normal Clojure functions? You could still do individual transformations by conforming non-registered specs if desired.

 

Thank you,

Josh

P.S. To confirm my understanding that there is not a way to specify conformance inline, I attempted to create and use a spec defined like (s/keys :req [(s/and ::foo (s/conformer ,,,))]). To my surprise, clojure.spec used it without complaint, although the behavior of s/conform was unaffected. Should I create a JIRA issue for this?

Right now anything that's not a keyword gets filtered out and ignored. I'd be in favor of erroring on other stuff, but not sure if that's at odds with how Rich thinks about it. So, could do an enhancement jira if you wish.

[Josh Tilles]

Alex,

Thank you for your reply! (I apologize for the delay in my own.)

I’ll first address your final point (regarding what spec is buying me vs. just using normal functions): I’m still trying to figure that out myself! 😁 I.e., the boundaries of spec’s applicability/appropriateness are not yet apparent to me.

Second, your suggestion of an explicit coercion step before using spec to validate the map indicates to me that we were envisioning different things, although perhaps I’ve misunderstood the role of spec’s conformers. I was thinking about a situation like processing maps of named timestamps that originated as JSON sent over HTTP. The maps’ JSON origins force the timestamp values to be strings, but it would be a lot more convenient for downstream processing if the timestamps were in a format more structured than a string, like a map (akin to how s/cat specs conform, perhaps with :year, :month and :day components) or even an org.joda.time.DateTime instance. Hence, in this hypothetical code, I’d like to ensure the strings nested in the maps look properly timestamp-y before preparing the data for downstream processing by converting the strings to the more-structured format. So the coercions I had in mind would make more sense as a post-step to validation, but does this seem like an inappropriate application of conformers to you? I’d thought that s/conform was meant for this kind of validate-then-convert behavior, but I see how coercion before validation would be valuable in a different way, enabling more-granular specification, with more-focused predicate implementations and more-precise error reporting.

Ultimately, I’m trying to discern which of the following scenarios is the case:

• I’m misusing spec by trying to “opt in” to using conformers on map values.
• I’m using spec appropriately and the ability to specify conformance à la carte is a feature worthy of consideration.
• I’m using spec appropriately and the inability to specify conformance à la carte is unfortunate yet tolerable & unlikely to change.

From what you wrote before, it seems likely that either the first or the third is true, but I wanted to make sure I wasn’t being misunderstood first.

Thank you,

Josh

On Dec 19, 2016, at 5:26 PM, Alex Miller <al...@puredanger.com> wrote:

On Monday, December 19, 2016 at 2:42:49 PM UTC-6, Josh Tilles wrote:

1. Is this seen as an acceptable tradeoff of an API that the core Clojure devs are otherwise happy with? Or is it in fact a deliberate limitation, in line with the “map specs should be of keysets only”design decision?

It is deliberate that s/keys works only on attributes and that you can't inline value specs.

Right, and that’s a decision that I’m not contesting, btw.

You could still do individual transformations by conforming non-registered specs if desired.

Do you mean that I could s/conform maps against an “anonymous” spec like (s/and ::something (s/conformer #(try (update % ::created timestamp->map) (catch SomeException _ ::s/invalid))))? Or are you referring to something else? 

 

Thank you,

Josh

P.S. To confirm my understanding that there is not a way to specify conformance inline, I attempted to create and use a spec defined like (s/keys :req [(s/and ::foo (s/conformer ,,,))]). To my surprise, clojure.spec used it without complaint, although the behavior of s/conform was unaffected. Should I create a JIRA issue for this?

Right now anything that's not a keyword gets filtered out and ignored. I'd be in favor of erroring on other stuff, but not sure if that's at odds with how Rich thinks about it. So, could do an enhancement jira if you wish.

OK, I’ll look into doing that this weekend.

 

P.P.S. A previous draft of this email had detailed illustrative examples, cut out of fear of being tiresomely long-winded. If anything I described isn’t clear, let me know and I’ll go ahead and include the examples in the conversation.

-- 
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clo...@googlegroups.com
Note that posts from new members are moderated - please be patient with your first post.
To unsubscribe from this group, send email to
clojure+u...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
--- 
You received this message because you are subscribed to the Google Groups "Clojure" group.
To unsubscribe from this group and stop receiving emails from it, send an email to clojure+u...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Sean Corfield]

At World Singles, we use clojure.spec for that scenario: conforming / validating string input data and producing non-string conformed values.

 

For each string-to-non-string type, we write a conformer that accepts the string and either produces the valid, parsed non-string valid or produces ::s/invalid. If we need further constraints on the non-string value, we use s/and to combine those.

 

We also have generators for all of these: we use gen/fmap of a non-string-to-string formatter over a (custom) generator for the non-string values.

 

Here’s an example, for string input representing dates:

 

(defn coerce->date

  "Given a string or date, produce a date, or throw an exception.

  Low level utility used by spec predicates to accept either a

  date or a string that can be converted to a date."

  [s]

  (if (instance? java.util.Date s)

    s

    (-> (tf/formatter t/utc "yyyy/MM/dd" "MM/dd/yyyy"

                      "EEE MMM dd HH:mm:ss zzz yyyy")

        (tf/parse s)

        (tc/to-date))))

 

(defn ->date

  "Spec predicate: conform to Date else invalid."

  [s]

  (try (coerce->date s)

       (catch Exception _ ::s/invalid)))

 

(defmacro api-spec

  "Given a coercion function and a predicate / spec, produce a

  spec that accepts strings that can be coerced to a value that

  satisfies the predicate / spec, and will also generate strings

  that conform to the given spec."

  [coerce str-or-spec & [spec]]

  (let [[to-str spec] (if spec [str-or-spec spec] [str str-or-spec])]

    `(s/with-gen (s/and (s/conformer ~coerce) ~spec)

       (fn [] (g/fmap ~to-str (s/gen ~spec))))))

 

(s/def ::dateofbirth (api-spec ->date #(dt/format-date % "MM/dd/yyyy") inst?))

 

Sean Corfield -- (970) FOR-SEAN -- (904) 302-SEAN
An Architect's View -- http://corfield.org/

"If you're not annoying somebody, you're not really alive."
-- Margaret Atwood

 

On 1/13/17, 9:56 PM, "Josh Tilles" <clo...@googlegroups.com on behalf of jo...@signafire.com> wrote:

 

Alex,

 

Thank you for your reply! (I apologize for the delay in my own.)

 

I’ll first address your final point (regarding what spec is buying me vs. just using normal functions): I’m still trying to figure that out myself! 😁 I.e., the boundaries of spec’s applicability/appropriateness are not yet apparent to me.

 

Second, your suggestion of an explicit coercion step before using spec to validate the map indicates to me that we were envisioning different things, although perhaps I’ve misunderstood the role of spec’s conformers. I was thinking about a situation like processing maps of named timestamps that originated as JSON sent over HTTP. The maps’ JSON origins force the timestamp values to be strings, but it would be a lot more convenient for downstream processing if the timestamps were in a format more structured than a string, like a map (akin to how s/cat specs conform, perhaps with :year, :month and :day components) or even an org.joda.time.DateTime instance. Hence, in this hypothetical code, I’d like to ensure the strings nested in the maps look properly timestamp-y before preparing the data for downstream processing by converting the strings to the more-structured format. So the coercions I had in mind would make more sense as a post-step to validation, but does this seem like an inappropriate application of conformers to you? I’d thought that s/conform was meant for this kind of validate-then-convert behavior, but I see how coercion before validation would be valuable in a different way, enabling more-granular specification, with more-focused predicate implementations and more-precise error reporting.

 

Ultimately, I’m trying to discern which of the following scenarios is the case:

·         I’m misusing spec by trying to “opt in” to using conformers on map values.

·         I’m using spec appropriately and the ability to specify conformance à la carte is a feature worthy of consideration.

·         I’m using spec appropriately and the inability to specify conformance à la carte is unfortunate yet tolerable & unlikely to change.

 

From what you wrote before, it seems likely that either the first or the third is true, but I wanted to make sure I wasn’t being misunderstood first.

 

Thank you,

Josh

 

On Dec 19, 2016, at 5:26 PM, Alex Miller <al...@puredanger.com> wrote:

 

On Monday, December 19, 2016 at 2:42:49 PM UTC-6, Josh Tilles wrote:

1.     Is this seen as an acceptable tradeoff of an API that the core Clojure devs are otherwise happy with? Or is it in fact a deliberate limitation, in line with the “map specs should be of keysets only”design decision?

[Josh Tilles]

Sean,

Thank you for sharing your input and code! Although I am interested by what you shared about your team’s usage of spec, I don’t believe it addresses my original problem of wanting to omit conformers when registering specs, yet later opt in to use a conformer when consuming the specs.

---

Josh Tilles

79 Madison Ave, 4th Floor
New York, New York 10016

Tel: (646) 685-8379
signafire.com

---

clojure+unsubscribe@googlegroups.com

For more options, visit this group at
http://groups.google.com/group/clojure?hl=en

--- 
You received this message because you are subscribed to the Google Groups "Clojure" group.

To unsubscribe from this group and stop receiving emails from it, send an email to clojure+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

 

--
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clo...@googlegroups.com
Note that posts from new members are moderated - please be patient with your first post.
To unsubscribe from this group, send email to

clojure+unsubscribe@googlegroups.com

For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
---
You received this message because you are subscribed to the Google Groups "Clojure" group.

To unsubscribe from this group and stop receiving emails from it, send an email to clojure+unsubscribe@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clo...@googlegroups.com
Note that posts from new members are moderated - please be patient with your first post.
To unsubscribe from this group, send email to

clojure+unsubscribe@googlegroups.com

For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
---
You received this message because you are subscribed to the Google Groups "Clojure" group.

To unsubscribe from this group and stop receiving emails from it, send an email to clojure+unsubscribe@googlegroups.com.