[ANN] Clojure 1.9.0-alpha1

[Alex Miller]

Clojure 1.9.0-alpha1 is now available.

Try it via

- Download: https://repo1.maven.org/maven2/org/clojure/clojure/1.9.0-alpha1

- Leiningen: [org.clojure/clojure "1.8.0-alpha2"]

1.9.0-alpha1 includes the first release of clojure.spec.

A usage guide for spec is now available: http://clojure.org/guides/spec.

Enjoy!

Alex

[Alex Miller]

You know how you always see the typo right after you hit send?

- Leiningen: [org.clojure/clojure "1.9.0-alpha1"]

[Gary Trakhman]

The inevitable question: is there a release schedule for 1.9 yet? I would love to start writing greenfield spec code ASAP, knowing that would help me communicate tradeoffs.

--
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clo...@googlegroups.com
Note that posts from new members are moderated - please be patient with your first post.
To unsubscribe from this group, send email to
clojure+u...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
---
You received this message because you are subscribed to the Google Groups "Clojure" group.
To unsubscribe from this group and stop receiving emails from it, send an email to clojure+u...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Alex Miller]

No. I would expect on the order of months.

[Nando Breiter]

Alex,

The spec usage guide is excellent! Thank you!

Perhaps a small correction? Below the example:

(s/valid? ::person {::first-name "Elon" ::last-name "Musk" ::email "el...@example.com"})

;; Fails required key check

You might want to include what that spec call returns under it, which I assume would be true. I got a little confused trying to figure out why you were saying it was failing ... until I figured out the comment applied to the next example. 

Aria Media Sagl

+41 (0)76 303 4477 cell
skype: ariamedia

--

[Alex Miller]

Yes, I'll add that in the next deploy. Thanks.

[Leif]

Spec usage guide questions:

There is a small typo in the 'Collections' section (pretend this is a question):

(s/conform (s/coll-of number? #{} #{5 10 2}))
;; should be
(s/conform (s/coll-of number? #{}) #{5 10 2})

Is there a technical reason why the paths reported by 's/explain' for regex ops are incomplete?  E.g., in the "A game of cards" section, I would expect:

(s/explain ::game ...)
At: [::players 0 ::hand 0 1] val: :banana fails spec: ::card predicate: suit?
;; but the real output is this, indices from regex matches missing
At: [::players ::hand 1] val: :banana fails spec: ::card predicate: suit?

It's fine here, but it could get harder to find the failure point in a larger data structure.

Is there a way to write a map spec that contains only the required keyset?  This is against the spirit of the 'composition' arguments in the Rationale, but sometimes I really want to pin down the exact data in a map, especially for data about to escape into the wild (e.g. return values of a web API).

The behavior of 's/and' really surprised me (again, pretend this is a question).  The conformed values propagating through is semantically weird when you have 'core/and' in mind.  Also, if you have data keywords and spec keywords, it could get complicated to keep track of what's what in the later specs.  More examples in the "Composing predicates" section, right when 's/and' in introduced, might take away the surprise and replace it with luxurious comfort.

Is there a recommended way to introspect specs for our own purposes (coercion, code generation)?  An interpreter on the output of 'describe' might work (although it's a little complicated for fn specs), but I wanted to know if you all had any thoughts or plans for the future here.

Thanks,
Leif

[Alex Miller]

On Tuesday, May 24, 2016 at 4:54:40 PM UTC-5, Leif wrote:

Spec usage guide questions:

There is a small typo in the 'Collections' section (pretend this is a question):

(s/conform (s/coll-of number? #{} #{5 10 2}))
;; should be
(s/conform (s/coll-of number? #{}) #{5 10 2})

Fixing...

 

Is there a technical reason why the paths reported by 's/explain' for regex ops are incomplete?  E.g., in the "A game of cards" section, I would expect:

(s/explain ::game ...)
At: [::players 0 ::hand 0 1] val: :banana fails spec: ::card predicate: suit?
;; but the real output is this, indices from regex matches missing
At: [::players ::hand 1] val: :banana fails spec: ::card predicate: suit?

It's fine here, but it could get harder to find the failure point in a larger data structure.

The paths refer to tags in the schemas, not keys in the map. However, this has been asked about several times today and Rich has added support for a :in clause that will track the key paths to master and that will be in alpha2. 

Is there a way to write a map spec that contains only the required keyset?  This is against the spirit of the 'composition' arguments in the Rationale, but sometimes I really want to pin down the exact data in a map, especially for data about to escape into the wild (e.g. return values of a web API).

Nothing built in, but it's easy enough to write that predicate yourself if you need it.

 

The behavior of 's/and' really surprised me (again, pretend this is a question).  The conformed values propagating through is semantically weird when you have 'core/and' in mind.  Also, if you have data keywords and spec keywords, it could get complicated to keep track of what's what in the later specs.  More examples in the "Composing predicates" section, right when 's/and' in introduced, might take away the surprise and replace it with luxurious comfort.

I'll consider it.

 

Is there a recommended way to introspect specs for our own purposes (coercion, code generation)?  An interpreter on the output of 'describe' might work (although it's a little complicated for fn specs), but I wanted to know if you all had any thoughts or plans for the future here.

Could you give more details on what question you would like to ask?

[Sean Corfield]

We have 1.9.0 Alpha 1 in QA at World Singles and plan to start using clojure.spec this week. We’ve gone to production on Alpha builds ever since 1.3 and almost never had any problems (in five years – time has flown past!).

 

What sort of tradeoffs have you in mind regarding whether to use clojure.spec or not? I ask because I’m thinking about adding an optional namespace to clojure.java.jdbc that would provide fn-specs for that library – my assumption is that folks not yet using a 1.9.0 build would simply not require that spec namespace…

 

Sean Corfield -- (904) 302-SEAN
An Architect's View -- http://corfield.org/

"If you're not annoying somebody, you're not really alive."
-- Margaret Atwood

[Leif]

Hi, Alex, thanks for the responsiveness.

The paths refer to tags in the schemas, not keys in the map. However, this has been asked about several times today and Rich has added support for a :in clause that will track the key paths to master and that will be in alpha2. 

 Where is this discussion thread?  I wouldn't want to ask duplicate questions.

 

Is there a recommended way to introspect specs for our own purposes (coercion, code generation)?  An interpreter on the output of 'describe' might work (although it's a little complicated for fn specs), but I wanted to know if you all had any thoughts or plans for the future here.

 

Could you give more details on what question you would like to ask?

 
Better people to ask would be those that have a lot of experience writing translators for specs, like the Schema devs.  But I'll give my muddled thoughts here:

Right now, the internals of different instances of Spec are private.  So, to write a translator from a Spec to the approximate json or avro schema it specifies, or translate from a Spec to a function that coerces a String to a data structure that conforms, I would have to:

1. Parse the output of 'describe' back into a description of the Spec's internals (if I can get at all of them)
2. Interpret / translate the parsed tree

In fact, the current Spec protocol, in my mind, is actually 4 protocols, one for 4 different translators you all have written:

(defprotocol Spec
  ;; Spec->Conformer
  (conform* [spec x])
  :: Spec->Explainer
  (explain* [spec path via x])
  ;; Spec->Gen
  (gen* [spec overrides path rmap])
  (with-gen* [spec gfn])
  ;; Spec->Describer
  (describe* [spec]))

But this implementation hiding dooms clojure/core to write all the translators.  This just seems weird given how most other Clojure features are open and user-extensible.

--Leif

[Timothy Baldridge]

The output of `spec/form` is helps a bit with this, as it namespaces vars, keywords and symbols correctly. I've had it on my list to write a translator for specs sometime in the near future, but I haven't done it yet. True the output of `form` isn't as uniform as I would like (sexprs instead of maps) but I don't think it would be too much work. 

Timothy

--

You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clo...@googlegroups.com
Note that posts from new members are moderated - please be patient with your first post.
To unsubscribe from this group, send email to
clojure+u...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
---
You received this message because you are subscribed to the Google Groups "Clojure" group.
To unsubscribe from this group and stop receiving emails from it, send an email to clojure+u...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

“One of the main causes of the fall of the Roman Empire was that–lacking zero–they had no way to indicate successful termination of their C programs.”
(Robert Firth)

[Colin Fleming]

I'll also need a spec translator (from s/form or s/describe, I guess - I'm not clear on the difference), and I'm very interesting in seeing any details if someone gets to it before I do.

Cheers,

Colin

[Ambrose Bonnaire-Sergeant]

Thanks for the guide Alex.

Is it possible to get more information in the "no method" error returned by

multi-spec?

(s/explain :event/event {:event/type :event/restart})

;; val: {:event/type :event/restart} fails predicate: my.domain/event-type, no method

It would be nice to mention exactly which method is missing here (ie. :event/restart).

Thanks,

Ambrose

[Mark Engelberg]

One thing that has always limited the value of pre and post conditions is that they don't give useful error messages.

I would like to see function in the spec namespace that returns true if input conforms to a spec, otherwise returns false and prints to standard out an explanation of the failure.  Sort of a valid-or-explain predicate (but a shorter name would be better).

In some sense, fdef is superior to using pre and post assertions, so maybe at some point they will feel obsolete, but possibly not -- things defined with fdef have to be explicitly instrumented, but pre and post assertions automatically follow the behavior set for all assertions.

So as long as pre and post assertions have value, I think many people are going to cobble together their own implementation of valid-or-explain.  Better to have one standard implementation that everyone can rely on.

[Mark Engelberg]

On Tue, May 24, 2016 at 8:55 PM, Mark Engelberg <mark.en...@gmail.com> wrote:

I would like to see function in the spec namespace that returns true if input conforms to a spec, otherwise returns false and prints to standard out an explanation of the failure.  Sort of a valid-or-explain predicate (but a shorter name would be better).

My point is that this would be a useful thing to use inside a pre or post condition.

[se...@corfield.org]

Bear in mind that you can catch the exception thrown by a pre-/post-condition so printing to stdout is not the right behavior.

 

Are you thinking about something like this?

 

(when-not (s/valid? spec arg)

 (throw (AssertionError. (with-out-str (s/explain spec arg))

                          (ex-info “SpecFailure” (s/explain-data spec arg)))))

 

Sean Corfield -- (904) 302-SEAN
An Architect's View -- http://corfield.org

 

--

[Mark Engelberg]

On Tue, May 24, 2016 at 11:01 PM, <se...@corfield.org> wrote:

Bear in mind that you can catch the exception thrown by a pre-/post-condition so printing to stdout is not the right behavior.

Ah, I didn't realize that it was okay to throw an error inside a pre-/post-condition test, but that makes sense.

 

 

Are you thinking about something like this?

 

(when-not (s/valid? spec arg)

 (throw (AssertionError. (with-out-str (s/explain spec arg))

                          (ex-info “SpecFailure” (s/explain-data spec arg)))))

 

So yes, this looks like exactly the kind of thing I would like to have a convenient name for, for easy use inside a pre or post condition.

Maybe `assert-valid` would be a good name.

[Tatu Tarvainen]

Are there plans for what will be the release content of 1.9?

I didn't see anything in the Clojure jira.

Particularly I'm interested if namespaced maps syntax will be included as clojure.spec encourages

namespacing keyword keys.

[Rick Moynihan]

Congratulations on the alpha release!

I'm also curious what the plans are for 1.9 beyond clojure.spec (which
looks awesome btw).

R.

[Alex Miller]

On Tuesday, May 24, 2016 at 5:45:47 PM UTC-5, Sean Corfield wrote:

We have 1.9.0 Alpha 1 in QA at World Singles and plan to start using clojure.spec this week. We’ve gone to production on Alpha builds ever since 1.3 and almost never had any problems (in five years – time has flown past!).

 

What sort of tradeoffs have you in mind regarding whether to use clojure.spec or not? I ask because I’m thinking about adding an optional namespace to clojure.java.jdbc that would provide fn-specs for that library – my assumption is that folks not yet using a 1.9.0 build would simply not require that spec namespace…

Well if you want to validate your data and functions then I think using clojure.spec would be useful. :) If you want to check them at runtime then there is obviously a performance cost in doing so.

[Alex Miller]

On Tuesday, May 24, 2016 at 7:26:57 PM UTC-5, Leif wrote:

Hi, Alex, thanks for the responsiveness.

The paths refer to tags in the schemas, not keys in the map. However, this has been asked about several times today and Rich has added support for a :in clause that will track the key paths to master and that will be in alpha2. 

 Where is this discussion thread?  I wouldn't want to ask duplicate questions.

It was on the clojurians.net Slack room #clojure_spec

 

Is there a recommended way to introspect specs for our own purposes (coercion, code generation)?  An interpreter on the output of 'describe' might work (although it's a little complicated for fn specs), but I wanted to know if you all had any thoughts or plans for the future here.

 

Could you give more details on what question you would like to ask?

 
Better people to ask would be those that have a lot of experience writing translators for specs, like the Schema devs.  But I'll give my muddled thoughts here:

Right now, the internals of different instances of Spec are private.  So, to write a translator from a Spec to the approximate json or avro schema it specifies, or translate from a Spec to a function that coerces a String to a data structure that conforms, I would have to:

1. Parse the output of 'describe' back into a description of the Spec's internals (if I can get at all of them)

I'm not sure why you need the internals - the vocabulary of spec is relatively small (+ the open-ended world of predicates). It doesn't seem possible to translate open-ended predicates to fixed types in json or avro, but you could look for known predicates.

2. Interpret / translate the parsed tree

In fact, the current Spec protocol, in my mind, is actually 4 protocols, one for 4 different translators you all have written:

I'm not sure you need to know or care about this to translate specs into something else.

 

(defprotocol Spec
  ;; Spec->Conformer
  (conform* [spec x])
  :: Spec->Explainer
  (explain* [spec path via x])
  ;; Spec->Gen
  (gen* [spec overrides path rmap])
  (with-gen* [spec gfn])
  ;; Spec->Describer
  (describe* [spec]))

But this implementation hiding dooms clojure/core to write all the translators.  This just seems weird given how most other Clojure features are open and user-extensible.

The core vocabulary for spec is relatively small - the "open" part comes through predicates (which can be anything) or conditional mechanisms like multi-spec.

 

--Leif

[Alex Miller]

Probably - can you file a jira for that?

[Gary Trakhman]

It would be helpful to make clojure.spec available as a lib for 1.8.  I have opted to backport clojure.spec (a 30-second copy/paste job) into our application running on 1.8 for an easy transition, and maybe others would be more comfortable bringing in a lib until 1.9 is further along.  That would also allow related libs/fn-specs to show up and be useful sooner.

--

[Alex Miller]

So something like

(defn valid-or-explain [spec data] 

  (let [v (s/valid? spec data)] 

    (when-not v (s/explain spec data))

    v))

I'll mention it to Rich, not sure though.

[Alex Miller]

We will be adding CLJ-1910 and CLJ-1919 for namespaced map literals and namespaced map destructuring in an alpha relatively soon.

Nothing else to announce re 1.9 right now.

[Alex Miller]

We do not plan to create or support a backport lib for Clojure 1.8. Mostly just a matter of choosing not to spend our time on maintaining multiple code bases.

[Alex Miller]

Rich pointed out that this is telling you the predicate :my.domain/event-type and the value it received: {:event/type :event/restart} which seems to be telling you everything you need to know wrt to the missing method. 

The dispatch method is opaque (even though it is simple here), so doesn't make sense to generically add anything additional to this message.

On Tuesday, May 24, 2016 at 10:53:43 PM UTC-5, Ambrose Bonnaire-Sergeant wrote:

[Ambrose Bonnaire-Sergeant]

In practice this is a lot of information though, and often it's a coincidence that

the dispatch entry comes first since maps are unordered.

See the following error I found in practice using namespaced keywords pervasively

missing a :spec-play.destruct/type|or method. It's rather large and

I'm glad the relevant entry came first.

CompilerException clojure.lang.ExceptionInfo: Call to #'spec-play.destruct/parse-type did not conform to spec:

At: [:ret] val: {:spec-play.destruct/type|type :spec-play.destruct/type|or, :spec-play.destruct/or|types [{:spec-play.destruct/type|type :spec-play.destruct/type|N}]} fails predicate: spec-play.destruct/type-type,  no method

:clojure.spec/args  ((clojure.core/or N))

Here are two ideas that might solve this:

1. Use the result of applying the dispatch function to the input in the error message.

2. Allow custom error messages in the default method (ie. return something more

interesting than nil).

Thanks,

Ambrose

--

[Sean Corfield]

I meant more that a failed pre-/post-condition throws AssertionError rather than printing to the console and that a failure using clojure.spec should still do the same thing.

 

The valid-or-explain function suggested would “work” in terms of being a predicate but the AssertionError wouldn’t reflect the actual details of the failure and you’re assuming stdout is easily accessible.

 

I think it would be better if the spec-based predicate failed by throwing an exception with the explanation in it (but I haven’t tested how it would actually interact with the pre-/post-condition machinery).

 

Sean Corfield -- (904) 302-SEAN

An Architect's View -- http://corfield.org/

"If you're not annoying somebody, you're not really alive."
-- Margaret Atwood

[Mark Engelberg]

On Wed, May 25, 2016 at 6:38 AM, Alex Miller <al...@puredanger.com> wrote:

So something like

(defn valid-or-explain [spec data] 

  (let [v (s/valid? spec data)] 

    (when-not v (s/explain spec data))

    v))

Right, that's what I was originally thinking. The form Sean Corfield suggested might make more sense:
(defn assert-valid [spec data]
    (when-not (s/valid? spec data)
         (throw (AssertionError. (with-out-str (s/explain spec data))
                                  (ex-info “SpecFailure” (s/explain-data spec data))))))

 

My objective in suggesting it is that we figure out what is the most useful thing inside a pre-/post-condition, and implement it once and standardize on it rather than a bunch of people coming up with their own ad-hoc solution in their own libraries for printing a meaningful error message when something doesn't conform.

[Mike Rodriguez]

Is there a recommended way to introspect specs for our own purposes (coercion, code generation)?  An interpreter on the output of 'describe' might work (although it's a little complicated for fn specs), but I wanted to know if you all had any thoughts or plans for the future here.

 

Could you give more details on what question you would like to ask?

 
Better people to ask would be those that have a lot of experience writing translators for specs, like the Schema devs.  But I'll give my muddled thoughts here:

Right now, the internals of different instances of Spec are private.  So, to write a translator from a Spec to the approximate json or avro schema it specifies, or translate from a Spec to a function that coerces a String to a data structure that conforms, I would have to:

1. Parse the output of 'describe' back into a description of the Spec's internals (if I can get at all of them)

I'm not sure why you need the internals - the vocabulary of spec is relatively small (+ the open-ended world of predicates). It doesn't seem possible to translate open-ended predicates to fixed types in json or avro, but you could look for known predicates.

I will add here that I have found quite a bit of leverage from being able to use Prismatic schema's data-representation for generating Avro schema from records.  Yes, we only support a subset of all possible Schema, since Schema also allows for arbitrary predicates as schema.  We've also had other occasions where we could write general functions that gave default values to fields that had the same Schema instead of relying on what that field is named etc.  

I always really liked that Prismatic Schema had a "data representation" and that seems to be the Clojure-way anyways.  I haven't dug into this too much yet, but I'm hoping that the Spec's do have some way to programmatically inspect them and utilize their structures for other purposes. 

[kovas boguta]

On Wed, May 25, 2016 at 3:01 PM, Mike Rodriguez <mjr...@gmail.com> wrote:

I always really liked that Prismatic Schema had a "data representation" and that seems to be the Clojure-way anyways.  I haven't dug into this too much yet, but I'm hoping that the Spec's do have some way to programmatically inspect them and utilize their structures for other purposes. 

Nothing precludes a system of data-described predicates. 

Just implement a datatype that implements IFn, in addition to some collection interface. Basically something like Records, except that (f x) will invoke the predicate described by f's data rather than looking up a value in the collection that f also represents. 

How this would work out in practice I have no idea but it seems worth trying. 

[Leif]

This is what the *spec-impl functions in clojure.spec do, essentially.  But the only internal field a user can get at is 'form'.  That's why I think the easiest way to programmatically inspect them, currently, is to call the 's/form' function (thanks, Timothy) and parse it back into a description of the internal fields.  This will probably work for a lot of use cases.  Seems a little roundabout, though.

But, if my reading of the code is correct, the generators are hidden, so someone wanting to write code that, e.g., 'completes' partial data conforming to a spec would have to--well, I don't know what, exactly, but some kind of trickery.

[Leif]

I guess I'm confused why the Clojure philosophy of "data > fns > macros" is being ignored in this particular case, when the other schema libraries show that there is an army of end-users that want to do unexpected things with specifications, and to do that, it would be easiest if they had data IRs, c.f. the Schema model:

macros to look nice ->
    data repr ->
        -compile to-> validator or
        -translate to-> data repr with subgenerators attached ->
            -compile to-> generator or
            -compile to-> ??? something a clever tester dreamed up ??? or
        -compile to-> fn returning nice error message or
        -compile to-> ??? something a clever user dreamed up ???

I'll leave it at that, I've put my 6 cents in.

[Rich Hickey]

I’d advise everyone concerned/confused about the relationship between spec and data representations to please spend some more time trying to understand spec and, especially, dial back the rhetoric. I know what the Clojure philosophy is, and it’s not some triviality.

specs are fundamentally code (predicates and logic)

Clojure code is a great data format for code, designed to be written and read by humans. There is a reason we don’t write code in AST maps:

(if x y z)

vs

{:tag :if-expr :test x :then y :else z}

The ‘form’ function will return the spec as sexpr data. This is currently missing gens but that will come (if gens don’t end up somewhere else entirely). You can eval this data and get a running spec. Or not. If you don’t want to, that doesn’t make the fact that you can bad. Data doesn’t run, so any data-driven ‘language’ is going to need an interpreter/compiler.

Claiming sexprs aren't ‘data’ suitable for programmatic manipulation denies a 50+ year Lisp history.

Expecting runtime specs to ‘be’ data is no more desirable than leaving code as data at runtime - it would always need interpretation. Runtime specs are (sets of) functions, and are as binary as functions. What matters is that they are available as data, which they are.

If you prefer an AST-like thing you are only a spec and conform call away from it. This is the key point people are missing who haven’t used spec enough. If you have an sexpr or other data-format syntax you don’t need to write a parser, you need to write a spec. conform will destructure for you.

We intend to provide specs for spec which can be used for these and other purposes.

Rich

[Glen Mailer]

Is the plan that any lib which adds specs will force the rest of the running system up to 1.9, or is it possible to keep the specs in a different namespace and not blow up?

Would it be reasonable to release something for 1.8 which contains all the spec vars but no-ops them?

[Mike Rodriguez]

Thanks for this explanation.  I think that cleared up some of this for me more.  I'm certainly excited about this new addition.  I should have started off with that.

[Sean Corfield]

On 5/26/16, 1:43 AM, "Glen Mailer" <clo...@googlegroups.com on behalf of glen...@gmail.com> wrote:
>Is the plan that any lib which adds specs will force the rest of the running system up to 1.9, or is it possible to keep the specs in a different namespace and not blow up?

I would expect to be able to keep the specs in a separate namespace. I plan to try this with clojure.java.jdbc as an exercise “soon”.