add test.generative to Clojure's tests

Description

The 0.1.5 release of test generative includes data generators and specs based on data generation, plus a runner that can run both c.t and c.t.g. tests under the same umbrella.

Environment

None

Attachments

1
  • 22 Aug 2012, 04:23 AM

Activity

Show:

Stuart Sierra September 7, 2012 at 5:34 PM

Patches have been applied as of September 1, 2012.

Fogus August 22, 2012 at 6:59 PM

Thanks for the clarification Stu. You're right that the exception behavior is consistent, the confusion was my own.

Stuart Halloway August 22, 2012 at 5:51 PM

The stats reports are aggregate, i.e. they rollup both the generative and clojure.test stats. It would be straightforward to separate these in the report. I will look at doing that in the next drop of c.t.g.

The output already includes stack traces for errors, but not for test failures. I thought this was consistent with clojure.test, but maybe I missed something.

Fogus August 22, 2012 at 4:32 PM

I agree with you position on the cyclic load tests. The patch looks sane to me and ran without hitch. A couple points of note:

  • This requires one to re-run antsetup – not a problem, but it might trip a few people

  • The final output of the test run is different than before. In the case of a successful run we now see only the generative success stats. Is the regular clojure.test stats included? It's not clear to me, but I don't think they are. In the fail case we see something like the following. I actually like the map output better, but one advantage to the stack trace output was that it sometimes helped identify code bugs. In any case, it would be nice to see an aggregate score at the end. This is a test.generative feature request I know.

[java] {:clojure.test/vars (test-equality), [java] :thread/name "main", [java] :pid 8682, [java] :thread 1, [java] :type :assert/fail, [java] :level :warn, [java] :test/actual (not (not true)), [java] :test/expected (not (= nil nil)), [java] :line 28, [java] :tstamp 1345649256308, [java] :file "data_structures.clj"}

In all, I like this! Now we need to start thinking up more specs.

Stuart Halloway August 22, 2012 at 4:26 AM

Note that this patch also removes the cyclic load tests. Those tests relied on leaving broken code on the test classpath, which is very hostile to tooling (in this case the test runner's ability to walk the code in the test directory.)

I believe such tests are an antipattern, and should be implemented differently or e.g. placed in an ancillary test suite.

Completed

Details

Assignee

Reporter

Approval

Ok

Patch

Code and Test

Priority

Fix versions

Created August 22, 2012 at 4:23 AM
Updated March 1, 2013 at 3:49 PM
Resolved March 1, 2013 at 3:49 PM