clojure.spec merge+or bug?

[Jenny Finkel]

I think I found a bug in spec when combining merge and or. Basically, when you conform a map where one of the keys has an or, and the spec comes from a clojure.spec/merge, one of the underlying keys will conform it, while the others don't, and then when the results get merged together you can end up with the unconformed version in the result:

user> (require '[clojure.spec :as s])

user> (s/def ::a (s/or :even even? :odd odd?))

user> (s/def ::b (s/or :even even? :odd odd?))

user> (s/def ::m1 (s/keys :req-un [::a]))

user> (s/def ::m2 (s/keys :req-un [::b]))

user> (s/def ::mm (s/merge ::m1 ::m2))

user> (s/valid? ::mm {:a 1 :b 2})

true

user> (s/conform ::mm {:a 1 :b 2})

{:a 1, :b [:even 2]}

user> (s/unform ::mm {:a [:odd 1] :b [:even 2]})

{:a 1, :b [:even 2]}

user> (s/unform ::mm (s/conform ::mm {:a 1 :b 2}))

UnsupportedOperationException nth not supported on this type: Long  clojure.lang.RT.nthFrom (RT.java:962)

I guess that valid? checks if it satisfies all the merged specs, and conform conforms on each and merges, and then unform can end up with a result that it can't unform (and similarly, if you give unform a properly conformed thing it can't unform properly due to how it merges results as well). I think the fix would be an update to clojure.spec/merge to make it smarter about which keys to keep from each map on conforming and unforming, though looking at the current code I don't think it's an easy fix.

-Jenny

[Jenny Finkel]

PS - I just realize I wasn't using the latest version of spec, as is evident from my require, but I just tried again with the latest and it doesn't change the result. And I should have mentioned that I'm happy to take a stab at a fix, assuming I'm correct that this is a bug.

[Alex Miller]

This is actually the expected result. s/merge doesn't flow like s/and - only the conformed version of the last map spec in the merge is used. There are thus some unexpected results in combination with the -un options as they are the only link towards conforming.

One thing you could try instead is to use s/and which does flow and might give you some of the behavior you're looking for (but won't gen as well).

[Jenny Finkel]

thanks for the fast reply! do you think it will always be this way? it does seem to violate the expected conform/unform relationship.

On Wed, May 31, 2017 at 9:51 PM, Alex Miller <al...@puredanger.com> wrote:

This is actually the expected result. s/merge doesn't flow like s/and - only the conformed version of the last map spec in the merge is used. There are thus some unexpected results in combination with the -un options as they are the only link towards conforming.

One thing you could try instead is to use s/and which does flow and might give you some of the behavior you're looking for (but won't gen as well).

--
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clo...@googlegroups.com
Note that posts from new members are moderated - please be patient with your first post.
To unsubscribe from this group, send email to
clojure+unsubscribe@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
---
You received this message because you are subscribed to a topic in the Google Groups "Clojure" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/clojure/r8WO24rHsi0/unsubscribe.
To unsubscribe from this group and all its topics, send an email to clojure+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Jenny Finkel]

It also seems to come up with coll-of + or:

user> (s/def ::a (s/or :even even? :odd odd?))

user> (s/def ::c (s/coll-of ::a))

user> (s/conform ::c [1 2 3 4])

[[:odd 1] [:even 2] [:odd 3] [:even 4]]

user> (s/unform ::c (s/conform ::c [1 2 3 4]))

[[:odd 1] [:even 2] [:odd 3] [:even 4]]

It looks like coll-of calls every-impl which just has identity as it's unform fn.

-Jenny

[Alex Miller]

You can file a bug on the s/merge unform - anything that doesn't roundtrip should be a bug.

On the coll-of one, I thought that was just fixed in the latest spec.alpha release (see https://dev.clojure.org/jira/browse/CLJ-2076) - are you using latest there?

[shlomi...@gmail.com]

Achhh, just spent the last few hours fighting this unexpected behavior with s/merge, until I finally came to realize that this is what it was..

I see this thread is quite old, did anyone open a bug for it as mentioned above? 

@Alex, you said this was the expected behavior, but then asked to open a bug because it does not round-trip.. I am slightly confused, does the problem lie in this "expected" behavior, or does it lie in s/unform? How would such a bug be closed?

Thanks,

Shlomi

[Alex Miller]

The non-flowing behavior on conform is expected behavior.

Failure to roundtrip conform then unform is a bug (so I'd so the bug here is in unform).

On a quick search, I don't believe this was filed, but I could have missed it.

[shlomi...@gmail.com]

Thank you for explaining!

Just in case, I opened a bug here https://dev.clojure.org/jira/browse/CLJ-2388 

[Jenny Finkel]

Sorry, I meant to file a bug and even try to write a fix, but I was very pregnant at the time and then I gave birth so it sort of fell by the wayside. I'm interested in understanding why non-flowing behavior on conform is expected/desired behavior. I've found that when clojure design decisions go contrary to my intuitions, I usually learn a lot from understanding the design motivation. Would you mind explaining?

Cheers,

 Jenny

PS - I did try to google for an answer, and I found the thread where the docstring for merge got updated to reflect this, but I couldn't find an explanation.

--

[Alex Miller]

With s/and, conformed values flow through the predicates. This allows you to take advantage of structured values from early preds in later preds, so you're not having to re-understand the structure. (There are cases where having a non-flowing s/and would be useful and we've talked about adding that since way back, still TBD). For single values, there's not much difference, but it matters when you've got more structure, either from regex (really s/& then) or s/or, etc.

With s/merge the idea is to combine 2 or more map specs and "merge" their specs, both during conform and gen. The mental picture I have is that the same data flows into all the merged preds in parallel (whereas s/and snakes them through in serial order). I don't have a good example at hand that illustrates where is critical, but certainly in cases where you combining s/keys with a collection view of map tuples with s/coll (sometimes called "hybrid maps"), this is essential. 

[Alex Whitt]

I got bitten by this today as well... makes it hard to add a predicate that destructures the map and compares its values. 

clo...@googlegroups.com

For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
---
You received this message because you are subscribed to a topic in the Google Groups "Clojure" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/clojure/r8WO24rHsi0/unsubscribe.

To unsubscribe from this group and all its topics, send an email to clo...@googlegroups.com.