Spec for the destructuring language

[Sam Estep]

I can’t seem to find it right now, but I remember reading somewhere that there are plans to add specs for the existing functions and macros in the Clojure core library, before the release of Clojure 1.9. While writing a macro recently, I found that I didn’t have a way to validate or conform binding forms, so I wrote a spec for such forms as specified by the destructuring guide:

(require '[clojure.spec :as s])

(s/def ::form
  (s/or :symbol simple-symbol?
        :sequential ::sequential
        :associative ::associative))

(s/def ::sequential
  (s/and vector?
         (s/conformer identity (comp vec (partial mapcat #(if (seq? %) % [%]))))
         (s/cat :elements (s/* (s/and (complement #{'&}) ::form))
                :tail (s/? (s/cat :delimiter #{'&} :form ::form))
                :all (s/? (s/cat :delimiter #{:as} :form ::form)))))

(s/def ::associative
  (s/and map?
         (s/conformer #(let [keywords #{:as :or :keys :strs :syms}]
                         (merge (some->> (apply dissoc % keywords)
                                         not-empty
                                         (assoc {} :mappings))
                                (select-keys % keywords)))
                      #(merge (:mappings %) (dissoc % :mappings)))
         (s/keys :opt-un [::mappings ::as ::or ::keys ::strs ::syms])))

(s/def ::mappings
  (s/and (s/conformer identity
                      #(into {} (for [[k v] %] [(s/unform ::form k) v])))
         (s/map-of ::form any? :conform-keys true)))

(s/def ::as simple-symbol?)
(s/def ::or ::mappings)

(s/def ::keys (s/coll-of ident? :kind vector?))
(s/def ::strs (s/coll-of ident? :kind vector?))
(s/def ::syms (s/coll-of ident? :kind vector?))

The above spec seems to push the limits of what you can do easily with the current functionality in clojure.spec (as of 1.9.0-alpha10). Specifically:

• In ::sequential, using unform with the cat spec doesn’t work correctly (is there an existing issue in JIRA for this, or should I create one?), so an extra conformer is needed.
• In ::associative, since there’s no easy way to spec a hybrid map, a particularly complex conformer is required to even facilitate validation.
• In ::mappings, :conform-keys doesn’t do anything for unform, which necessitates another conformer.

Also, I would have really liked to get a working ::form generator out of this spec, but after my attempts to tweak the generator were thwarted by stack overflows, I gave up. If you know a way to fix the generator, I’d love to see it!

Aside from that, validation, conform, and unform all work, as far as I’m aware. I’ll leave it to you to test the validation. For an example of conform/unform, here’s what it looks like for the argument vector from the second print-contact-info function in the Where to destructure section:

(let [form
      '[{:keys [f-name l-name phone company title]
         {:keys [street city state zip]} :address
         [fav-hobby second-hobby] :hobbies}]
      conformed (s/conform ::form form)
      unformed (s/unform ::form conformed)]
  (pprint [form conformed unformed]))
;; [[{:keys [f-name l-name phone company title],
;;    {:keys [street city state zip]} :address,
;;    [fav-hobby second-hobby] :hobbies}]
;;  [:sequential
;;   {:elements
;;    [[:associative
;;      {:mappings
;;       {[:associative {:keys [street city state zip]}] :address,
;;        [:sequential
;;         {:elements [[:symbol fav-hobby] [:symbol second-hobby]]}]
;;        :hobbies},
;;       :keys [f-name l-name phone company title]}]]}]
;;  [{{:keys [street city state zip]} :address,
;;    [fav-hobby second-hobby] :hobbies,
;;    :keys [f-name l-name phone company title]}]]

I tried to be as true to the actual destructuring language as possible. As you can see from my implementation, there are a few quirks about what’s required and what’s not:

• In a ::sequential form, the :as form can be any destructuring form, but in an ::associative form, the :as form must be a symbol. I’d prefer to be more strict and only accept symbols in :as forms for both ::sequential and ::associative destructuring, but I kept the existing behavior to fit with the general Clojure strategy of backwards compatibility.
• Clojure seems to support using keywords instead of symbols as ::mappings keys. I would have liked to retain this behavior in the spec for backwards compatibility, but since no other part of the destructuring language allows the use of keywords in place of symbols, allowing that sort of usage would have required large changes to the structure of the above spec, so I left it out.

Is a set of specs for the core Clojure functionality planned for 1.9? If so, I hope that the above spec can help set the ball rolling for adding a destructuring spec to Clojure. If not, please let me know; in that case, I’ll probably publish the above spec in a library on Clojars or something.

​

[Alex Miller]

Hi Sam,

I have a complete spec for destructuring which will be included in a future alpha. It has been tested extensively on a large number of community projects as well. I see a number of places where I do not think this spec is completely accurate, it is also missing the new namespace key destructuring introduced in 1.9.0-alpha8 (which admittedly was challenging to cover well with the existing tools and has driven some not yet released enhancements that Rich has been working on). 

The specific problem of combining vector? and s/cat is one that has come up a lot and Rich have I discussed it several times without a definitive conclusion, although the option with the fewest tradeoffs is probably introducing s/vcat to cover that specific case (regex in a vector).

On Thu, Aug 11, 2016 at 8:00 AM, Sam Estep <samuel...@gmail.com> wrote:

I can’t seem to find it right now, but I remember reading somewhere that there are plans to add specs for the existing functions and macros in the Clojure core library, before the release of Clojure 1.9. While writing a macro recently, I found that I didn’t have a way to validate or conform binding forms, so I wrote a spec for such forms as specified by the destructuring guide:

(require '[clojure.spec :as s])

(s/def ::form
  (s/or :symbol simple-symbol?
        :sequential ::sequential
        :associative ::associative))

(s/def ::sequential
  (s/and vector?
         (s/conformer identity (comp vec (partial mapcat #(if (seq? %) % [%]))))
         (s/cat :elements (s/* (s/and (complement #{'&}) ::form))
                :tail (s/? (s/cat :delimiter #{'&} :form ::form))
                :all (s/? (s/cat :delimiter #{:as} :form ::form)))))

The name following :as can only be a simple-symbol?, not a general ::form. 

(s/def ::associative
  (s/and map?
         (s/conformer #(let [keywords #{:as :or :keys :strs :syms}]
                         (merge (some->> (apply dissoc % keywords)
                                         not-empty
                                         (assoc {} :mappings))
                                (select-keys % keywords)))
                      #(merge (:mappings %) (dissoc % :mappings)))
         (s/keys :opt-un [::mappings ::as ::or ::keys ::strs ::syms])))

There are better solutions to this hybrid map and the use of a conformer here is definitely not ideal. It is better to treat this as an s/merge of the special keys and an s/every that validates the tuples.

I wrote a lengthy blog post about handling hybrid maps but Rich asked me to defer releasing it as it will likely be superseded by other enhancements.

 

(s/def ::mappings
  (s/and (s/conformer identity
                      #(into {} (for [[k v] %] [(s/unform ::form k) v])))
         (s/map-of ::form any? :conform-keys true)))

the keys of the mappings must be simple-symbol?, not any?.  

(s/def ::as simple-symbol?)
(s/def ::or ::mappings)

(s/def ::keys (s/coll-of ident? :kind vector?))
(s/def ::strs (s/coll-of ident? :kind vector?))
(s/def ::syms (s/coll-of ident? :kind vector?))

These can be more finely scoped as well. strs must be simple-symbol?. syms must be symbol?.

 

The above spec seems to push the limits of what you can do easily with the current functionality in clojure.spec (as of 1.9.0-alpha10). Specifically:

• In ::sequential, using unform with the cat spec doesn’t work correctly (is there an existing issue in JIRA for this, or should I create one?), so an extra conformer is needed.

If you can boil this down to a concise example, sure. 

• In ::associative, since there’s no easy way to spec a hybrid map, a particularly complex conformer is required to even facilitate validation.

I described an approach above, which I've used successfully for this. 

• In ::mappings, :conform-keys doesn’t do anything for unform, which necessitates another conformer.

That's interesting. I'd have to think about whether there's something more that needs to exist. (But generally I'd say you should be avoiding most of the conformers in the first place.)

Also, I would have really liked to get a working ::form generator out of this spec, but after my attempts to tweak the generator were thwarted by stack overflows, I gave up. If you know a way to fix the generator, I’d love to see it!

The version I have generates and uses no conformers, although I can't say generation creates anything very fun (without spending more effort constraining the any? generators).

I'm using the more restrictive form currently and have not found a failing case in the wild yet. I do not know of any good reason to allow destructuring that sequential :as (since you're already destructuring the same data).

• Clojure seems to support using keywords instead of symbols as ::mappings keys. I would have liked to retain this behavior in the spec for backwards compatibility, but since no other part of the destructuring language allows the use of keywords in place of symbols, allowing that sort of usage would have required large changes to the structure of the above spec, so I left it out.

keywords are allowed in specific places (namely in :keys to support auto-resolved namespaced keywords). Prior to 1.9.0-alpha8 there were some bugs around the mapping keys although that behavior was not intentional and was never documented or specified anywhere. That code has been tightened up. The :or mapping keys *must* be simple-symbol? and are always the local name being bound in the let (which cannot be namespaced or a keyword). There are a few uses of this out in the wild and that code will cause errors when it did not before, however with the new let spec those will produce compilation errors, so should fail early.

 

Is a set of specs for the core Clojure functionality planned for 1.9? If so, I hope that the above spec can help set the ball rolling for adding a destructuring spec to Clojure. If not, please let me know; in that case, I’ll probably publish the above spec in a library on Clojars or something.

There will be an official spec for most of the core macros and that will be the one loaded and enforced by the compiler during macroexpansion. I expect there will also be something for core functions although that will require instrumentation to enable - still discussion about what exactly will be provided there. I have working and tested specs currently for let, if-let, when-let, fn, defn, defn-, and ns which cover some of the hardest "dsl" specs in core - I expect all of those will land in an upcoming alpha.

 

​

--
You received this message because you are subscribed to the Google Groups "Clojure Dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to clojure-dev+unsubscribe@googlegroups.com.
To post to this group, send email to cloju...@googlegroups.com.
Visit this group at https://groups.google.com/group/clojure-dev.
For more options, visit https://groups.google.com/d/optout.

[Sam Estep]

Thank you for the detailed response, Alex!

I have a complete spec for destructuring which will be included in a future alpha. It has been tested extensively on a large number of community projects as well.

Awesome, I’m really glad to hear that. It is a bit unfortunate that we both spent time developing specs for the same thing independently, but writing this was good clojure.spec practice for me anyway.

I see a number of places where I do not think this spec is completely accurate, it is also missing the new namespace key destructuring introduced in 1.9.0-alpha8 (which admittedly was challenging to cover well with the existing tools and has driven some not yet released enhancements that Rich has been working on).

I wouldn’t be surprised if my spec has some errors, but the particular notes that you have made are incorrect. I’ve included further explanation below of the correctness of my spec in those particular areas.

I could be mistaken about what specifically you mean by “the new namespace key destructuring introduced in 1.9.0-alpha8”, but if you mean the part about auto-resolved keywords in the Associative Destructuring section of the destructuring guide, my spec does validate those:

(let [{:keys [::foo]} {::foo :bar}] foo)
;;=> :bar

(s/conform ::form {:keys [::foo]})
;;=> [:associative {:keys [:my.ns/foo]}]

The specific problem of combining vector? and s/cat is one that has come up a lot and Rich have I discussed it several times without a definitive conclusion, although the option with the fewest tradeoffs is probably introducing s/vcat to cover that specific case (regex in a vector).

OK, thanks! A vcat macro sounds great.

The name following :as can only be a simple-symbol?, not a general ::form.

In Clojure 1.9.0-alpha10, :as can be a general ::form:

(let [[:as [foo]] [:bar]] foo)
;;=> :bar

(let [[:as {foo 0}] [:bar]] foo)
;;=> :bar

As I said in my original post, I would prefer for :as to only support simple symbols, and you may have already changed it for the next alpha, but regardless, my spec is correct for Clojure 1.9.0-alpha10.

There are better solutions to this hybrid map and the use of a conformer here is definitely not ideal. It is better to treat this as an s/merge of the special keys and an s/every that validates the tuples.

I wrote a lengthy blog post about handling hybrid maps but Rich asked me to defer releasing it as it will likely be superseded by other enhancements.

I’m not entirely sure I understand your solution; however, it’s good to know that there are (and will be) better solutions to hybrid maps than explicit conformers. If you have the time to post your solution as an answer to my question on Stack Overflow, I would really appreciate it!

the keys of the mappings must be simple-symbol?, not any?.

The second argument to map-of is the value spec, not the key spec:

(:arglists (meta #'s/map-of))
;;=> ([kpred vpred & opts])

Regardless, at least in Clojure 1.9.0-alpha10, neither the keys nor the values are restricted to simple-symbol?:

(let [{[foo bar] :baz} {:baz [:foo :bar]}] [foo bar])
;;=> [:foo :bar]

These can be more finely scoped as well. strs must be simple-symbol?. syms must be symbol?.

In Clojure 1.9.0-alpha10, any ident? can be used in :strs or :syms:

(let [{:strs [foo foo/bar :baz :baz/qux]}
      {"foo" :foo "foo/bar" :bar ":baz" :baz ":baz/qux" :qux}]
  [foo bar baz qux])
;;=> [:foo :bar :baz :qux]

(let [{:syms [foo foo/bar :baz :baz/qux]}
      '{foo :foo foo/bar :bar baz :baz baz/qux :qux}]
  [foo bar baz qux])
;;=> [:foo :bar :baz :qux]

If you can boil this down to a concise example, sure.

OK, will do.

I described an approach above, which I’ve used successfully for this.

See my above comment.

That’s interesting. I’d have to think about whether there’s something more that needs to exist. (But generally I’d say you should be avoiding most of the conformers in the first place.)

I certainly agree; however, using a conformer here is the only way I can see to make the destructuring ::form keys in ::mappings consistent with other ::forms when conformed.

The version I have generates and uses no conformers, although I can’t say generation creates anything very fun (without spending more effort constraining the any? generators).

Cool, I look forward to seeing it!

I’m using the more restrictive form currently and have not found a failing case in the wild yet. I do not know of any good reason to allow destructuring that sequential :as (since you’re already destructuring the same data).

OK, good to know. I wouldn’t expect that anyone would use something besides a symbol for :as, because as you said, it doesn’t seem to provide any additional power. I only put it in my spec to be consistent with the current (1.9.0-alpha10) Clojure behavior.

keywords are allowed in specific places (namely in :keys to support auto-resolved namespaced keywords). prior to 1.9.0-alpha8 there were some bugs around the mapping keys although that behavior was not intentional and was never documented or specified anywhere. that code has been tightened up. the :or mapping keys must be simple-symbol? and are always the local name being bound in the let (which cannot be namespaced or a keyword). there are a few uses of this out in the wild and that code will cause errors when it did not before, however with the new let spec those will produce compilation errors, so should fail early.

I’m guessing that you mean you’ve changed this since the latest alpha release; however, the behaviors with the keyword keys in mappings and general ::form keys in :or are still extant in 1.9.0-alpha10:

(let [{:foo :bar} {:bar :baz}] foo)
;;=> :baz

(let [{[foo bar] :foobar :or {[foo bar] [:baz :qux]}} {}] [foo bar])
;;=> [:baz :qux]

There will be an official spec for most of the core macros and that will be the one loaded and enforced by the compiler during macroexpansion. I expect there will also be something for core functions although that will require instrumentation to enable - still discussion about what exactly will be provided there. I have working and tested specs currently for let, if-let, when-let, fn, defn, defn-, and ns which cover some of the hardest “dsl” specs in core - I expect all of those will land in an upcoming alpha.

That’s quite awesome that you’ve already done all those specs! I’m looking forward to seeing them in a future release.

​

To unsubscribe from this group and stop receiving emails from it, send an email to clojure-dev...@googlegroups.com.

​

[Alex Miller]

On Thu, Aug 11, 2016 at 12:00 PM, Sam Estep <samuel...@gmail.com> wrote:

I could be mistaken about what specifically you mean by “the new namespace key destructuring introduced in 1.9.0-alpha8”, but if you mean the part about auto-resolved keywords in the Associative Destructuring section of the destructuring guide, my spec does validate those:

No, I mean the new syntax added via CLJ-1919 such as:

(let [{:foo/keys [a b]} {:foo/a 1 :foo/b 2}] [a b]) ;;=> [1 2] 

The name following :as can only be a simple-symbol?, not a general ::form.

In Clojure 1.9.0-alpha10, :as can be a general ::form:

I know that's accepted as valid now, but I don't believe that behavior has ever been specified or documented as an intentional feature and I'm proposing to lock that down with the spec. Having a spec allows us to more precisely say what is actually allowed.

There are better solutions to this hybrid map and the use of a conformer here is definitely not ideal. It is better to treat this as an s/merge of the special keys and an s/every that validates the tuples.

I wrote a lengthy blog post about handling hybrid maps but Rich asked me to defer releasing it as it will likely be superseded by other enhancements.

I’m not entirely sure I understand your solution; however, it’s good to know that there are (and will be) better solutions to hybrid maps than explicit conformers. If you have the time to post your solution as an answer to my question on Stack Overflow, I would really appreciate it!

The basic idea is along the lines of (simplified):

(s/def ::a keyword?)

(s/def ::b string?)

(s/def ::m

  (s/merge (s/keys :opt-un [::a ::b])

                 (s/every (s/or :int (s/tuple int? int?)

                                       :option (s/tuple keyword? any?))

                               :into {})))

(s/valid? ::m {1 2, 3 4, :a :foo, :b "abc"}) ;; true

Something like that should gen just fine.

 

the keys of the mappings must be simple-symbol?, not any?.

Sorry, I did say a confusing thing - apologies. Yes, you can nest map destructuring - what I said re mappings was incorrect. 

My point was that the :or map is not ::mappings - the keys (where you re-use ::mappings) are expected to be simple symbols, not further destructurings: (s/map-of simple-symbol? any?) 

 

These can be more finely scoped as well. strs must be simple-symbol?. syms must be symbol?.

In Clojure 1.9.0-alpha10, any ident? can be used in :strs or :syms:

Again, this is a case of the current code allowing things broader than what is actually intended. Supported forms are:

  :keys - can be qualified or simple keywords or symbols

  :syms - can be qualified or simple symbols

 :strs - can be simple symbols

Rich and I wrote some updated docs for this but I don't think I ever got those actually published. I should do so!

keywords are allowed in specific places (namely in :keys to support auto-resolved namespaced keywords). prior to 1.9.0-alpha8 there were some bugs around the mapping keys although that behavior was not intentional and was never documented or specified anywhere. that code has been tightened up. the :or mapping keys must be simple-symbol? and are always the local name being bound in the let (which cannot be namespaced or a keyword). there are a few uses of this out in the wild and that code will cause errors when it did not before, however with the new let spec those will produce compilation errors, so should fail early.

I’m guessing that you mean you’ve changed this since the latest alpha release; however, the behaviors with the keyword keys in mappings and general ::form keys in :or are still extant in 1.9.0-alpha10:

Again, some things exist now that are accidental, not intentional. The changes I was referring to above were around using namespaced keywords as keys in :or.

[Sam Estep]

It seems that I misunderstood your meaning. Thank you for clarifying, and for bringing my attention to CLJ-1919; I was not aware of the new syntax. I appreciate your explanation of a hybrid map spec using merge and every; your solution is certainly more elegant than my solution with conformer. I’m glad that the new destructuring spec will specify the destructuring language more precisely; that will make everything easier in the future.