Thoughts on clojure.spec
1,762 views
Skip to first unread message
Mark Engelberg
Jul 10, 2016, 6:04:39 AM7/10/16
to clojure
I've played around now with implementing specs for a couple of my projects.
What I'm finding is that writing specs to check the inputs of a function is easy-ish, but building useful random generators is very hard -- in my projects, this seems too hard to do with any reasonable amount of effort.
This isn't due to anything inherent in clojure.spec, it's just that for non-trivial functions, coming up with relevant random input is a very hard problem. For example, let's say I have a function that takes two integers. Odds are that not any two randomly chosen integers will work. Some combinations of integers are non-sensical and could trigger an error, other combinations may cause the function to run way too long. As a concrete example, I just tried to spec out a SAT solver (which tries to solve NP-complete problems). The input should be a vector of vectors of ints, but many combinations of inputs will just run forever. How to generate "good" SAT problems? I have no idea.
So for the most part, I've ignored the generation aspect of specs because it just feels like too much work. But without the generators, clojure.spec's utility is significantly diminished.
1. No way to test function output specs. For documentation purposes, I want to have an output spec on my function. However, as far as I know, after instrumentation-triggered checking of output specs was removed a couple of alphas ago, the only way remaining to check against output specs is to use randomly generated tests. So if I can't make good generators, I have no way to confirm that my output spec works the way I think it does. My documentation could be totally out of touch with reality, and that displeases me.
2. Limited ability for testing that functions take and receive what you expect. Whereas a static type system can prove certain properties about whether functions are called with valid inputs, with spec, you only get those guarantees if you pump a function with enough valid random data to trigger the function calling all its callees with interesting combinations of data. But if I use the naive generators, the function will never even complete with most of the randomly generated input, let alone call other functions in a useful way. And in many cases I don't see how to generate something of better quality.
So looking back at my experiments, my preliminary impression is that by adding specs to my public APIs, I've gained some useful documentation, and I've given users the ability to instrument functions in order to get high-quality assertion-checking of the inputs. In some cases, the error messages for bad input when instrumented are also more useful than I would have otherwise gotten, but in some cases they aren't. Overall, I've struggled to write generators, and without them, the value proposition isn't as great.
One other issue I've had, unrelated to generators, is that I'm struggling to express higher-order type constraints, for example, this function takes a vector v1 of anything and a vector v2 of anything, but the type of the things in vector v1 better match the type of the things in vector v2.
What are other people finding? Do you find it easy/hard to write generators? (If you think it's easy, I'd love to know your tricks). Do you find it easy/hard to read specs as a form of documentation about the contract of a function? Do you find it frustrating that there's no way to turn on instrumentation of function outputs for manual testing? Do you feel your generators are providing sufficient code coverage when exercising callee functions?
What I'm finding is that writing specs to check the inputs of a function is easy-ish, but building useful random generators is very hard -- in my projects, this seems too hard to do with any reasonable amount of effort.
This isn't due to anything inherent in clojure.spec, it's just that for non-trivial functions, coming up with relevant random input is a very hard problem. For example, let's say I have a function that takes two integers. Odds are that not any two randomly chosen integers will work. Some combinations of integers are non-sensical and could trigger an error, other combinations may cause the function to run way too long. As a concrete example, I just tried to spec out a SAT solver (which tries to solve NP-complete problems). The input should be a vector of vectors of ints, but many combinations of inputs will just run forever. How to generate "good" SAT problems? I have no idea.
So for the most part, I've ignored the generation aspect of specs because it just feels like too much work. But without the generators, clojure.spec's utility is significantly diminished.
1. No way to test function output specs. For documentation purposes, I want to have an output spec on my function. However, as far as I know, after instrumentation-triggered checking of output specs was removed a couple of alphas ago, the only way remaining to check against output specs is to use randomly generated tests. So if I can't make good generators, I have no way to confirm that my output spec works the way I think it does. My documentation could be totally out of touch with reality, and that displeases me.
2. Limited ability for testing that functions take and receive what you expect. Whereas a static type system can prove certain properties about whether functions are called with valid inputs, with spec, you only get those guarantees if you pump a function with enough valid random data to trigger the function calling all its callees with interesting combinations of data. But if I use the naive generators, the function will never even complete with most of the randomly generated input, let alone call other functions in a useful way. And in many cases I don't see how to generate something of better quality.
So looking back at my experiments, my preliminary impression is that by adding specs to my public APIs, I've gained some useful documentation, and I've given users the ability to instrument functions in order to get high-quality assertion-checking of the inputs. In some cases, the error messages for bad input when instrumented are also more useful than I would have otherwise gotten, but in some cases they aren't. Overall, I've struggled to write generators, and without them, the value proposition isn't as great.
One other issue I've had, unrelated to generators, is that I'm struggling to express higher-order type constraints, for example, this function takes a vector v1 of anything and a vector v2 of anything, but the type of the things in vector v1 better match the type of the things in vector v2.
What are other people finding? Do you find it easy/hard to write generators? (If you think it's easy, I'd love to know your tricks). Do you find it easy/hard to read specs as a form of documentation about the contract of a function? Do you find it frustrating that there's no way to turn on instrumentation of function outputs for manual testing? Do you feel your generators are providing sufficient code coverage when exercising callee functions?
Michael Gardner
Jul 10, 2016, 6:37:18 AM7/10/16
to clo...@googlegroups.com
It might be possible to leverage something like American Fuzzy Lop[1] for better random input generation. I've never used AFL myself, but I know SQLite (one of the best-tested libraries I know of) has had good success with it[2], and it does work on Java.
[1] http://lcamtuf.coredump.cx/afl/
[2] https://www.sqlite.org/testing.html#aflfuzz
> --
> You received this message because you are subscribed to the Google
> Groups "Clojure" group.
> To post to this group, send email to clo...@googlegroups.com
> Note that posts from new members are moderated - please be patient with your first post.
> To unsubscribe from this group, send email to
> clojure+u...@googlegroups.com
> For more options, visit this group at
> http://groups.google.com/group/clojure?hl=en
> ---
> You received this message because you are subscribed to the Google Groups "Clojure" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to clojure+u...@googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.
[1] http://lcamtuf.coredump.cx/afl/
[2] https://www.sqlite.org/testing.html#aflfuzz
> You received this message because you are subscribed to the Google
> Groups "Clojure" group.
> To post to this group, send email to clo...@googlegroups.com
> Note that posts from new members are moderated - please be patient with your first post.
> To unsubscribe from this group, send email to
> clojure+u...@googlegroups.com
> For more options, visit this group at
> http://groups.google.com/group/clojure?hl=en
> ---
> You received this message because you are subscribed to the Google Groups "Clojure" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to clojure+u...@googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.
kovas boguta
Jul 10, 2016, 1:39:05 PM7/10/16
to clo...@googlegroups.com
On Sun, Jul 10, 2016 at 6:04 AM, Mark Engelberg <mark.en...@gmail.com> wrote:
This isn't due to anything inherent in clojure.spec, it's just that for non-trivial functions, coming up with relevant random input is a very hard problem.
This is a very interesting observation.
I've written generators (we'd call it enumeration) for a fair number of computational systems when I was at Wolfram, so I have some observations and suggestions.
This problem comes up anytime the *behavior* of the function is nontrivial. For most human-coded functions, behavior is trivial, its only the implementation that is nontrivial. Clojure.spec optimizes for this case.
All NP-complete problems, by definition, exhibit a kind of universality, meaning they can be programmed. In general there is no way to know the behavior of the inputs without an irreducible amount of computation. In specific cases, there are ornate mathematical theories, which can also help with generation, but which have no unifying framework universal across all systems.
So I don't think there is any general solution clojure.spec can offer.
On generating formulas:
I would just enumerate them systematically with increasing number of variables (starting at 1), and see how far I can get within an allotted runtime. Being able to time-out surely must be important also.
With most nontrivial computational systems you don't have go far to hit nontrivial behavior. So only being able to enumerate to a modest size isn't a deal breaker.
If you want to generate big, satisfiable formulas, i think you'll have to make them satisfiable by construction. You can do with this with a simple term-rewriting system. Anytime you have a satisfiable expr, you can "or-in" failing branches. It would actually probably be pretty illuminating so see the runtime performance under various schemas for doing this.
If the goal is to test with a diverse set of formulas, your best bet is to generate them using using a secondary computational system (for the elementary cellular automata) and then map its behavior into formulas.
You might find the following links interesting, both for potential methods and for building intuition of what might one expect when enumerating through mathematical systems:
Rich Hickey
Jul 11, 2016, 10:01:05 AM7/11/16
to clo...@googlegroups.com
Kovas is right on the money here. Inherently when you make something ‘programmable’ generating correct programs for it is hard.
I would say though, I frequently see people struggling to spec more complex logic are going directly to independent input generation. Thus, how will inputs be reasonably coordinated etc. The better approach is to first generate a consistent model, then generate inputs from that model. gen/bind is your friend. Typically the approach is:
(gen/bind model-generator (fn [model] data-generator))
Each model then need not be exhaustively large because many models will be generated, and the model lets you share things between what would otherwise be independent generators.
On the specific points:
> 1. No way to test function output specs. For documentation purposes, I want to have an output spec on my function. However, as far as I know, after instrumentation-triggered checking of output specs was removed a couple of alphas ago, the only way remaining to check against output specs is to use randomly generated tests. So if I can't make good generators, I have no way to confirm that my output spec works the way I think it does. My documentation could be totally out of touch with reality, and that displeases me.
Running return-value instrument-style checking on whatever few hand-written tests you might have isn’t going to give you better coverage than a simple (even hardwired) generator that captures similar ranges. And you can directly exercise your :ret spec - it’s just another spec. You can also spec/assert on your return - the tests will disappear in production, although this is similarly as weak as instrument-triggered return checking, so I’m not going to help you do it, but you can do it yourself :)
> 2. Limited ability for testing that functions take and receive what you expect. Whereas a static type system can prove certain properties about whether functions are called with valid inputs, with spec, you only get those guarantees if you pump a function with enough valid random data to trigger the function calling all its callees with interesting combinations of data. But if I use the naive generators, the function will never even complete with most of the randomly generated input, let alone call other functions in a useful way. And in many cases I don't see how to generate something of better quality.
You can certainly use core.typed if you want to check the kinds of things it can check in the way it can check them. But saying specs aren’t types says little about either, and using the word ‘valid’ in both contexts without qualification equates things that are not the same. It’s not as if most type systems can check the predicates spec can (Idris et al aside). There’s a big difference between type-correct inputs/outputs and stakeholder-correct programs. spec is oriented towards the latter, but, the bar, as you have seen, is generation. It is, however, a local challenge - ‘this particular fn is hard to gen for’. The bar for static typing is flowing types everywhere, a bar I consider to be more global, harder to meet, and less expressive.
> One other issue I've had, unrelated to generators, is that I'm struggling to express higher-order type constraints, for example, this function takes a vector v1 of anything and a vector v2 of anything, but the type of the things in vector v1 better match the type of the things in vector v2.
Parameterized types definitely have advantages for the subset of things they can check in this regard. But again, it is a logical fallacy to equate spec == type then switch to static type checking ‘advantages' as if they had the same expressive power - in general they don’t, since specs are predicative and of runtime values. Most type systems can’t tell you that the things in v1 satisfy the same predicates as do the things in v2 for any predicate other than ‘is statically a T’ (Idris et al aside). That said, I have been thinking about the ‘parameterized spec’ problem, nothing concrete to show yet.
Rich
I would say though, I frequently see people struggling to spec more complex logic are going directly to independent input generation. Thus, how will inputs be reasonably coordinated etc. The better approach is to first generate a consistent model, then generate inputs from that model. gen/bind is your friend. Typically the approach is:
(gen/bind model-generator (fn [model] data-generator))
Each model then need not be exhaustively large because many models will be generated, and the model lets you share things between what would otherwise be independent generators.
On the specific points:
> 1. No way to test function output specs. For documentation purposes, I want to have an output spec on my function. However, as far as I know, after instrumentation-triggered checking of output specs was removed a couple of alphas ago, the only way remaining to check against output specs is to use randomly generated tests. So if I can't make good generators, I have no way to confirm that my output spec works the way I think it does. My documentation could be totally out of touch with reality, and that displeases me.
> 2. Limited ability for testing that functions take and receive what you expect. Whereas a static type system can prove certain properties about whether functions are called with valid inputs, with spec, you only get those guarantees if you pump a function with enough valid random data to trigger the function calling all its callees with interesting combinations of data. But if I use the naive generators, the function will never even complete with most of the randomly generated input, let alone call other functions in a useful way. And in many cases I don't see how to generate something of better quality.
> One other issue I've had, unrelated to generators, is that I'm struggling to express higher-order type constraints, for example, this function takes a vector v1 of anything and a vector v2 of anything, but the type of the things in vector v1 better match the type of the things in vector v2.
Rich
Oliver George
Jul 11, 2016, 7:16:58 PM7/11/16
to Clojure
Do you find it frustrating that there's no way to turn on instrumentation of function outputs for manual testing?
Yes.
In particular when I'm refactoring code and want to know when I'm returning something surprising.
I haven't yet had much success with spec generators for CLJS / UI code. Too many non-clojure types to deal with - my gen-fu is limited.
I suspect the argument for reading an instrument flag to turn on return value testing is just that:
- to help those who aren't able to leverage generators in their problem domain.
- informal exploration testing
I see that spec/assert would give me the same results. That's a fantastic addition but is also a code intrusion.
Final though: I suspect you can have it for the cost of a small helper function. The instrument-all code doesn't seem complex but there may be a few private vars between you and happiness.
Alan Thompson
Jul 11, 2016, 8:03:29 PM7/11/16
to clo...@googlegroups.com
Do you find it frustrating that there's no way to turn on instrumentation of function outputs for manual testing?
Yes.
--
Maarten Truyens
Jul 12, 2016, 3:36:39 AM7/12/16
to Clojure
I would also truly appreciate instrumentation of function outputs for manual outputs. I understand the rationale for not having it as the default, but could it perhaps be specified as an option s/instrument? (Considering that it was present in the first alphas, I would assume that such option should not be far-fetched.)
Beau Fabry
Jul 18, 2016, 8:36:07 PM7/18/16
to Clojure
> Do you find it frustrating that there's no way to turn on instrumentation of function outputs for manual testing?
Yes. I've mentioned this elsewhere but I think being able to turn on output checking in lower environments (dev, test, master, staging) is getting extra values from specs basically for free. Being able to do it seems pragmatic. I'm hoping it won't be too difficult to write an `overinstrument-all` that gives me that when I want it.
Oliver George
Jul 18, 2016, 10:47:12 PM7/18/16
to clojure
Here's the commit removing that aspect of instrument-all. Not a big change.
As an aside, I also love the idea of the Clojure community fostering a culture of gen testing each chunk of well defined functionality. If it's truly achievable the Clojure community could become known as an unstoppable force of robust code.
It would be something of a challenge for many of us... especially those wanting this particular feature!
--
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clo...@googlegroups.com
Note that posts from new members are moderated - please be patient with your first post.
To unsubscribe from this group, send email to
clojure+u...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
---
You received this message because you are subscribed to a topic in the Google Groups "Clojure" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/clojure/jcVnjk1MOWY/unsubscribe.
To unsubscribe from this group and all its topics, send an email to clojure+u...@googlegroups.com.
Oliver George
Director, Condense
0428 740 978
Sean Corfield
Jul 18, 2016, 10:53:49 PM7/18/16
to Clojure Mailing List
Rich has given a pretty good explanation of why this was removed elsewhere. And in this thread, a week ago, he explained again why gen-testing :ret and :fn specs was the better approach.
Sean Corfield -- (970) FOR-SEAN -- (904) 302-SEAN
An Architect's View -- http://corfield.org/
"If you're not annoying somebody, you're not really alive."
-- Margaret Atwood
Beau Fabry
Jul 18, 2016, 11:50:07 PM7/18/16
to Clojure
I think that was an explanation of why it's not particularly valuable in unit tests, but not really an explanation of why it wouldn't be useful in lower environments or canary boxes in distributed deployments. This thread has also touched on how not everything is gen-testable because of complexity, and I'd add that side-effects are another reason for that. We also have "you can just use assert on the return value" which is true, but seeing as I already have a database of expected return values that I've defined then it seems natural to be able to use that database to gain some extra testing value rather than define it again.
I'm not trying to argue for inclusion, if clojure core doesn't want to implement the feature then those who see value in it can trivially implement it themselves, but I haven't read anything that's made me think it wouldn't be useful.
se...@corfield.org
Jul 19, 2016, 1:51:09 AM7/19/16
to Clojure
Well, both Alex and Rich have said the change is deliberate and there are no plans to change that decision – and Rich talked about ways you can add return value testing manually based on specs (if you want, but he won’t help you) – so it seems like a “closed” topic to me? (and Alex has shut down a couple of other threads that have continued on past a clear line of decision)
I was sad to see :ret checking go away but I accept Rich’s line of thinking on this and I’ll adjust my workflow accordingly. I find Rich’s point that instrumentation is now about ensuring functions are _called_ correctly rather than trying to establish that they _behave_ correctly oddly compelling, now that I’ve had some time to think about it and play with it 😊
Sean Corfield -- (904) 302-SEAN
An Architect's View -- http://corfield.org
--
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clo...@googlegroups.com
Note that posts from new members are moderated - please be patient with your first post.
To unsubscribe from this group, send email to
clojure+u...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
---
You received this message because you are subscribed to the Google Groups "Clojure" group.
To unsubscribe from this group and stop receiving emails from it, send an email to clojure+u...@googlegroups.com.
Beau Fabry
Jul 19, 2016, 2:30:27 AM7/19/16
to Clojure
Right, and I don't think the "this is closed we shouldn't discuss it anymore" line is great when people are advocating for a piece of functionality. I understand Alex doesn't want endless threads bikeshedding basically arbitrary naming choices, but that's not the same as people making simple points of "I think X would be a good addition because of Y" with no back and forth.
Maybe enough people saying "yes that sounds like a good idea because Y" in this thread will convince someone else that they should create a lib that mirrors the old functionality, this is the general Clojure group and not clojure-dev after all.
Sorry about the meta.
Gary Fredericks
Jul 20, 2016, 3:27:04 PM7/20/16
to Clojure
Brandon Bloom
Dec 6, 2016, 6:42:44 PM12/6/16
to Clojure
I was just very surprised to discover my :ret specs were not being checked via instrument. I've read the rationale above, but I'm not yet convinced. While of course I can (and will) use spec/assert from a post condition, I lose the nice selective instrumentation features.
I'd also like to make a counter-point to this reasoning:
Running return-value instrument-style checking on whatever few hand-written tests you might have isn’t going to give you better coverage than a simple (even hardwired) generator that captures similar ranges.
In my case, I was already testing the root-level return value, but this function is deeply recursive. A small non-conforming value deep in the evaluation stack was yielding non-conformance at a path deep in the root return value. I was confused by why the :ret check wasn't finding it earlier. By relying on the top-level check rather than inline checks per-call (as per instrument or :pre/:post) I lost access to the call stack. Adding the post condition pin-pointed the problem instantly.
Ben Brinckerhoff
Dec 10, 2016, 4:20:43 PM12/10/16
to Clojure
I would certainly welcome and use an `overinstrument` function if added to schpec (or another library).
One reason spec is great because it provides "a la carte" validation: a can add an appropriate amount of validation to the parts of my code that would most benefit from it. In my experience using spec so far, optionally running `fn` and `ret` specs during instrumentation would give me more options to pick the correct amount of validation.
For instance, since side-effecting functions are harder to unit test (and difficult to test generatively), I would like to spec the functions and then run integration tests. Running `ret`/`fn` specs would help me detect errors closer to their source and also confirm the specs are accurate, which improves the value specs as documentation. In my current project, I have been avoiding writing `ret` specs for side-effecting functions because I worry they'll get out-of-sync and mislead other programmers.
Also, having `ret`/`fn` specs checked during instrumentation would add value during development when I'm not yet ready to build generative tests. For example, during early development of a function, I might just spec a parameter with `map?` even though it actually requires certain keys. That spec far too vague for generative testing, but immediately adds value when I turn on instrumentation, since it catches bugs where I've passed, say, `nil` as the parameter. When the feature is more stable, I can invest the time in creating a generator and running generative tests. With the current implementation of `instrument`, I find that I've been tempted to over-spec early in development in order to get working generative tests that will confirm my `ret` and `fn` specs are correct.
I understand Rich and Alex have thought all this through and have provided a detailed rationale for the current behavior. But in my own workflow, being able to turn on `ret`/`fn` specs during instrumentation would be a big help, and I'd very much like to see this feature added to schpec. I'm happy to look at this when my schedule allows (not this week or next due to travel). Perhaps the best approach is to add an `over-instrument` function that has options for turning on/off `args`, `fn`, and `ret` specs (enabling all by default) and see how people use it in practice?
Thanks to everyone who have worked so hard on all aspects of spec!
Ben
Dean Thompson
Apr 9, 2018, 11:02:06 AM4/9/18
to Clojure
For others, like me, who are finding this old thread interesting reading, there's a newer library that nicely addresses a concern expressed here, "Do you find it frustrating that there's no way to turn on instrumentation of function outputs for manual testing?"
I am really enjoying and getting value from orchestra.
Dean
I am really enjoying and getting value from orchestra.
Dean
Reply all
Reply to author
Forward
0 new messages