[spec] instrument should check :ret and :fn specs

Description

Problem: a function instrumented via clojure.spec.test.alpha/instrument checks the :args part of its spec, but not the :ret and :fn parts. It's confusing and forces people to use a 3rd-party library[1].

Solution: add an option to turn on :ret and :fn checking to clojure.spec.test.alpha/instrument.

[1]: https://github.com/jeaye/orchestra

Environment

None

Activity

Former user 
July 17, 2018 at 6:57 PM

Well, fair enough!

Re example-based tests: what I had in mind is 1) calling a function (presumably in repl) with a specific input to see if the output still conforms to :ret and :fn specs; 2) testing a function A which calls spec'ed functions B1..Bn to see which of those fail their :ret/:fn specs (essentially it gives you free unit tests for each integration test). I agree that it's theoretically possible to do that in vanilla clojure.spec with a custom generator, but that certainly doesn't worth the effort.

Anyway, I guess you've already heard this argument, so I'll stop there.

Alex Miller 
July 17, 2018 at 6:14 PM

As I said, we're aware of the request and why people want it. From a core team perspective, we don't need an issue to track it.

Use `check` if you want to use property based testing to verify the correctness of a function with respect to its spec. Generally if you want to test specific possible inputs, most people use example-based tests with clojure.test instead - that's not in the guide as it doesn't really have anything to do with spec. It is possible to override the args generator with a set spec of explicit inputs, but I'm not sure if the effort of that is worth the work most of the time.

Former user 
July 17, 2018 at 4:44 PM

Sorry, I've only searched through open issues before creating this ticket.

Is it at all possible to keep this issue open so people could vote for it and subscribe to it, since (as Alex stated) it's not set in stone yet?


(The thing is, I was confused by this behavior and asked on irc and elsewhere, and was told by several people that it's weird and looks like a bug. I've filed it as a "feature" since the doc clearly states that's the intended behavior. Also, it's not quite clear how to use check to verify correctness of a function for a specific subset of possible inputs (think checking the fn in repl as you develop it) - maybe the spec guide could elaborate a bit more on what the supposed workflow should look like? Maybe that'd clear up some confusion; though, I understand that's out of the scope.)

Ghadi Shayban 
July 17, 2018 at 12:04 AM

https://dev.clojure.org/display/community/Creating+Tickets
https://groups.google.com/forum/#!topic/clojure/RLQBFJ0vGG4/discussion
There's also some commentary from Rich in Slack, but I don't have a link handy.
("we should do X" tickets are almost universally declined.)
dup of https://dev.clojure.org/jira/browse/CLJ-1961

Alex Miller 
July 16, 2018 at 11:59 PM

The basic idea here is that instrument is designed to verify that a function is invoked correctly. For determining whether the internals of a function are correct, you can use `check`.

This has been discussed at great length, both on the core team and in the broader community and we’re familiar with all the arguments. While I’m not ruling out the possibility that we will do something in this area in the future, we are not currently planning on changing this, so I’m going to decline this for now. I’ll repeat, saying no to this now does not prevent us from saying yes later.

Declined

Details

Assignee

Reporter

Labels

Priority

Affects versions

Created July 16, 2018 at 11:38 PM
Updated July 17, 2018 at 6:57 PM
Resolved July 17, 2018 at 6:57 PM