WIP: To triage issues that people have with clojure.test with the goal of finding areas where it could potentially be improved.
Constraints: Retain backwards compatibility with existing callers.
Some slack conversations on the topic:
- turning into this thread:
Just for easy lookup, here are all the open issues: JIRA open issues, labeled clojure.test
thrown?test doesn't attempt to unwrap exceptions. It should probably test the main exception type and, in some situations, test the cause type as well (for
ExceptionInfo– maybe others?). See https://dev.clojure.org/jira/browse/CLJ-2458
Assertion error reporting
Problems making assertions
arecan be awkward at times, e.g. sometimes you want to put an
(is ,,,, msg)inside it for more control, e.g. of the message — which leads to the reporting of the error twice because are also wraps an
isover your form.
- How do you share a block of assertions across different tests, parameterize them, and retain good errors?
- Expressiveness: provides very little "language" for assertions:
areprovide basic equality/truthiness checks, and there are simple assertions around exceptions, but nothing to help with assertions about more complex data structures without very verbose code (see, for example, Expectations'
Usability of fixtures
- Complects test namespace with fixtures/scope. Choices are wrap each test in an ns separately in a fixture, or wrap all tests in this ns, but no way to opt out, or opt-in.
- No support for per-run fixture (allowing use cases like loading (and then dropping) fixture data for all tests in a suite.
test-varslets you run a single test, it provides no success/failure feedback and requires
Vars rather than just symbols – https://dev.clojure.org/jira/browse/CLJ-1908 would address this.
Problems people have in structuring tests, test suites etc. How to map ns's and functions to test ns's and functions? 1-1 mapping, 1-many mapping etc?
- Composability: if tooling wants to override the reporting behavior (or the assertion behavior), it's often impossible to combine such tooling with other tooling. An example is that Leiningen monkey-patches some of the clojure.test machinery which can then break other tooling that tries to override behavior. I recently created expectations/clojure-test to provide an Expectations-like DSL over clojure.test and found that it didn't work well with Paul Stadig's Humane Test Output because both tools extend the assert-expr multimethod (I ended up making e/c-t conditionally aware of HTO and changing its behavior accordingly).
Using clojure.spec with clojure.test
Is this even a good idea? If not do clojure.spec property based tests also need a test runner? Perhaps other projects such as leiningen need to plug a gap here?
clojure.test development speed is tied to clojure.core's releases
Alex Miller mentioned there was a possibility of splitting clojure.test out as an external dependency from clojure.core so it can evolve at its own pace.
Other test libraries
This (incomplete) list is provided purely for information, as possible sources of inspiration for changes that could be made or pain points that people have (that have already turned into a library):