wondering why spec.test/instrument doesn't check :ret

[Tom Connors]

spec.test/instrument doesn't check the return value of spec'd functions and according to the spec guide this is because we should be checking function implementations at testing time, not development time. It seems to me that adding :ret and :fn checks to instrumented functions would be worthwhile even though tests are a better place to do those checks - since instrumentation is meant to be turned off in production the performance hit doesn't matter, and since we're already verifying that the arguments conform to spec, why not check the return value as well? A benefit would be that if we haven't yet written (or won't write) generators and tests, we still get a confidence increase about our instrumented functions' return values. 

I'm sure this was considered, so I'm just wondering if "do that in your tests" is the entire reason for this design choice.

[Josh Tilles]

I’m not sure it answers your question directly, but a quote from Rich in another thread seems relevant:

On Monday, July 11, 2016 at 10:01:05 AM UTC-4, Rich Hickey wrote:

On Sunday, July 10, 2016 at 6:04:39 AM UTC-4, puzzler wrote:

1. No way to test function output specs.  For documentation purposes, I want to have an output spec on my function.  However, as far as I know, after instrumentation-triggered checking of output specs was removed a couple of alphas ago, the only way remaining to check against output specs is to use randomly generated tests.  So if I can't make good generators, I have no way to confirm that my output spec works the way I think it does.  My documentation could be totally out of touch with reality, and that displeases me.

Running return-value instrument-style checking on whatever few hand-written tests you might have isn’t going to give you better coverage than a simple (even hardwired) generator that captures similar ranges. And you can directly exercise your :ret spec - it’s just another spec. You can also spec/assert on your return - the tests will disappear in production, although this is similarly as weak as instrument-triggered return checking, so I’m not going to help you do it, but you can do it yourself :) 

Incidentally, I think your question is common enough that it might be worth writing something up for http://clojure.org/guides/faq.

[Sean Corfield]

> I'm sure this was considered

 

Indeed, and when clojure.spec was first introduced, instrument did check :ret and :fn specs. That feature was deliberately removed and the reason for it is (paraphrased):

 

* clojure.spec/instrument helps check that your functions are _called_ correctly

* clojure.spec.test/check helps check that your functions are _implemented_ correctly

 

Those are two different concerns and they should not be complected into one function. The argument checks performed by instrument are straightforward: are the arguments valid? The function invariant is checked by performing generative testing.

 

See this thread for some of the discussion around this (and Rich has also discussed in on Slack quite a bit):

 

https://groups.google.com/d/topic/clojure/RLQBFJ0vGG4/discussion

 

Sean Corfield -- (970) FOR-SEAN -- (904) 302-SEAN
An Architect's View -- http://corfield.org/

"If you're not annoying somebody, you're not really alive."
-- Margaret Atwood

[Tom Connors]

Thank you both for the responses and links, they've been helpful.