<< Back to previous view

[TCHECK-84] Add gen/uniform-double Created: 12/Nov/15  Updated: 10/Apr/16

Status: Open
Project: test.check
Component/s: None
Affects Version/s: None
Fix Version/s: None

Type: Enhancement Priority: Major
Reporter: Gary Fredericks Assignee: Gary Fredericks
Resolution: Unresolved Votes: 0
Labels: None


 Description   

Would be somewhat of a utility generator – generates a double from 0.0 (inclusive) to 1.0 (exclusive), and doesn't shrink.

gen/choose could even be rewritten with gen-fmap if we had this.






[TCHECK-85] prop/for-all should let later bindings refer to earlier bindings Created: 18/Nov/15  Updated: 10/Apr/16

Status: Open
Project: test.check
Component/s: None
Affects Version/s: None
Fix Version/s: None

Type: Enhancement Priority: Major
Reporter: Dan Burton Assignee: Gary Fredericks
Resolution: Unresolved Votes: 0
Labels: None


 Description   

For example:

(prop/for-all [x (gen/nat)
y (gen/choose 0 (dec x))] ;; y is a nat less than x
...)



 Comments   
Comment by Gary Fredericks [ 20/Nov/15 10:13 PM ]

I've thought about this before, and I'm pretty hesitant since it would probably mean using bind whenever there are multiple bindings, which means that old-style uses with independent generators would all of a sudden start shrinking poorly (due to the difficulty of shrinking bind in general).

I can see the value in consistency with gen/let though.

Another option would be an alternate for-all with the bind behavior, though I'm not sure would it would be called.





[TCHECK-38] Generators for functions? Created: 15/Aug/14  Updated: 10/Apr/16

Status: Open
Project: test.check
Component/s: None
Affects Version/s: None
Fix Version/s: None

Type: Enhancement Priority: Major
Reporter: Michael Sperber Assignee: Gary Fredericks
Resolution: Unresolved Votes: 0
Labels: None


 Description   

test.check has generators for first-order types, but not for functions.

In the original QuickCheck, an additional type class ("Arbitrary") handles
functions.

If I implemented this and submitted a patch, would it be likely to be included?

Background: I wrote my own Clojure QuickCheck clone here:

https://bitbucket.org/sperber/deinprogramm.quickcheck

This has arbitraries and function generators, but not all of test.check's goodies.

I'm trying to decide whether to continue maintaining my QuickCheck clone or
instead to contribute ot test.check, so feedback would be much appreciated.



 Comments   
Comment by Reid Draper [ 21/Aug/14 2:16 PM ]

I'd love a patch for this! Would you like to write up a quick implementation proposal first, so we can iron out any architectural details before looking at the code?

Comment by Michael Sperber [ 25/Aug/14 2:06 AM ]

Would you be willing to look at my QuickCheck clone as a starting point? Its general implementation approach is very similar to yours, so I think you should feel right at home.

The caveat is that it introduces a type separate from generators - the "arbitrary", corresponding to the `Arbitray' type class in the Haskell code. Doing this to `test.check' would change the API, so maybe there, the `Generator' type should be extended by an optional `transformer' field.

Comment by Reid Draper [ 26/Aug/14 2:26 PM ]

Sure, I've already read through it a bit, but need to read in more detail. How does your implementation handle printing (showing) functions? Do functions shrink? Have you seen Showing and Shrinking Functions by Claessen ?

Comment by Michael Sperber [ 27/Aug/14 2:21 AM ]

Yes. Haven't done that yet, but it would be on my list. Given that functions are objects in Clojure, I think printing should be a little easier than in Haskell.





[TCHECK-44] for-all should support nesting Created: 22/Sep/14  Updated: 10/Apr/16

Status: Open
Project: test.check
Component/s: None
Affects Version/s: None
Fix Version/s: None

Type: Enhancement Priority: Major
Reporter: Michael Sperber Assignee: Gary Fredericks
Resolution: Unresolved Votes: 0
Labels: None

Attachments: File for-alls-nest.diff    
Patch: Code and Test

 Description   

Haskell QuickCheck allows for-all expressions to nest. This is useful when there are dependencies between generated values. test.check should allow this, too.

Currently, nested for-alls always succeed, which is somewhat pernicious.

I've added a patch that implements this.



 Comments   
Comment by Reid Draper [ 24/Sep/14 11:32 AM ]

Thanks Michael. I appreciate the patch, but there's a few design details that could be discussed before we get to code-level detail. As a separate, but related issue, I've been wanting to implement something like your CheckResult type, but as a record with several more fields. These fields would hold things like a plain-text description of any errors found, statistics (ala QuickCheck's collect, classify, etc.). I'd also like to write a protocol that allows basic types like booleans to be turned into this record. This would be analogous to Haskell QuickCheck's Testable Type Class. While this is technically a separate issue, I think it would behoove us to solve it in conjunction with nestable for-alls, particularly since nested for-alls can be simulated by just using bind at the generator level. Does this make sense?

Comment by Michael Sperber [ 25/Sep/14 2:19 AM ]

Absolutely.

I personally would start this patch, and work from there, unless you want to do things fundamentally differently rather than add more stuff.

Either way, how can I help make it happen?

Comment by Reid Draper [ 25/Sep/14 11:10 PM ]

Great question. Let me think on that and get back to you ASAP. I'd also love to make this happen soon.

Comment by Reid Draper [ 21/Oct/14 10:23 PM ]

Sorry for the delay, here's my sketch I've been working with:

diff --git a/src/main/clojure/clojure/test/check/properties.clj b/src/main/clojure/clojure/test/check/properties.clj
index 99b5222..139ae9a 100644
--- a/src/main/clojure/clojure/test/check/properties.clj
+++ b/src/main/clojure/clojure/test/check/properties.clj
@@ -8,13 +8,47 @@
 ;   You must not remove this notice, or any other, from this software.
 
 (ns clojure.test.check.properties
+  (:import clojure.test.check.generators.Generator)
   (:require [clojure.test.check.generators :as gen]))
 
+(defrecord Result [result pass? message stamps])
+
+(defprotocol ToResult
+  (to-result [a]))
+
+(extend java.lang.Object
+  ToResult
+  {:to-result (fn [b]
+               ;; not checking for caught exceptions here
+               (->Result b (not (false? b)) nil nil))})
+
+(extend nil
+  ToResult
+  {:to-result (fn [b]
+               (->Result b false nil nil))})
+
+(extend java.lang.Boolean
+  ToResult
+  {:to-result (fn [b]
+               (->Result b b nil nil))})
+
+(extend Generator
+  ToResult
+  {:to-result identity})
+
+(extend Result
+  ToResult
+  {:to-result identity})
+
+(defn message
+  [m property]
+  (assoc property :message m))
+
 (defn- apply-gen
   [function]
   (fn [args]
-    (let [result (try (apply function args) (catch Throwable t t))]
-      {:result result
+    (let [result (to-result (try (apply function args) (catch Throwable t t)))]
+      {:result (:result result)
        :function function
        :args args})))
 
@@ -29,9 +63,18 @@
   (for-all* [gen/int gen/int] (fn [a b] (>= (+ a b) a)))
   "
   [args function]
-  (gen/fmap
-    (apply-gen function)
-    (apply gen/tuple args)))
+  (gen/bind
+    (apply gen/tuple args)
+    (fn [a]
+      (let [result ((apply-gen function) a)]
+        (cond (gen/generator? result) (gen/fmap (fn [r] (println "foo") (update-in r :args #(conj % a))) result)
+              ;; NOTE: quick note to myself before I leave this code for the night,
+              ;; this :else is getting hit because we're wrapping the result
+              ;; with a {:result ...} map. Should probably do that conditionally.
+              ;; We also need two result types I think, a result to return from
+              ;; a property itself, and a reuslt that tacks the 'args' on top of this.
+              :else (do (println "bar") (gen/return result)))))
+    ))
 
 (defn binding-vars
   [bindings]
Comment by Michael Sperber [ 22/Oct/14 2:00 AM ]

Looks OK. However, it's difficult to see why that would get you more quickly where you said you want to go than my patch ...

Comment by Reid Draper [ 28/Oct/14 10:55 PM ]

Looks OK. However, it's difficult to see why that would get you more quickly where you said you want to go than my patch ...

Fair enough. Part of this it that it was easier for me to write up a sketch. The main things I'm trying to cover when supporting nested generators are making sure:

  1. We also support the upcoming ability to collect and return statistics about a test, ala collect from Haskell QuickCheck
  2. We have a sane way of returning failing tests to a user. Right now, in the :fail and :smallest keys of the returned map, we tell the user the failing arguments. They're always wrapped in at least one vector, since you may use more than one generator using prop/for-all. What do we do with nested properties? How do we distinguish between multiple generators at the 'same level', vs nested properties? Or do we not need to distinguish? Can whatever we decide to do be backward compatible?

Point being, I want to make sure we're not committing ourselves to nested-properties until we have some of those answers, and for me personally, it's easier to try and play with these things together, and see how they will fit together.

Comment by Lars Andersen [ 04/Oct/15 4:42 AM ]

I don't really want to have to nest for-all, I'd much rather prefer it work like let where we can refer to previous values. That said, no matter which solution, this is my biggest gripe with test.check at the moment, so any solution would be preferable to another year of hammock time.

Here's one solution, taken from the wild. I'm sure there are many others, people want this badly, but this one was taken from Nathan Marz's specter library:

(defmacro for-all+ [bindings & body]
  (let [parts (partition 2 bindings)
        vars (vec (map first parts))
        genned (reduce
                (fn [curr [v code]]
                  `(gen/bind ~code (fn [~v] ~curr)))
                `(gen/return ~vars)
                (reverse parts))]
    `(prop/for-all [~vars ~genned]
                   ~@body )))
Comment by Gary Fredericks [ 04/Oct/15 9:47 AM ]

We've been thinking about a similar problem in TCHECK-81, and in the meantime there's an even-fancier variant of for-all here.





[TCHECK-60] string from regular expression generator Created: 23/Jan/15  Updated: 10/Apr/16

Status: Open
Project: test.check
Component/s: None
Affects Version/s: None
Fix Version/s: None

Type: Enhancement Priority: Major
Reporter: Steve Miner Assignee: Gary Fredericks
Resolution: Unresolved Votes: 0
Labels: None


 Description   

A regular expression is a succinct and flexible way to describe a set of strings. It would be useful to have generator that takes a regular expression and generates matching strings.

There was some hallway discussion about this feature request at the Clojure Conj in November 2014.



 Comments   
Comment by Steve Miner [ 23/Jan/15 9:12 AM ]

https://github.com/gfredericks/test.chuck recently added string-from-regex. That's a good work-around for most users.

Comment by Steve Miner [ 23/Jan/15 9:16 AM ]

https://github.com/miner/herbert recently added a new implementation of a string from regex generator (in v0.6.7). Herbert supports a more limited regex syntax than the generator from test.chuck, but it has no dependencies on other libraries so it can more easily be adopted in a contrib library.





[TCHECK-57] Performance regression in map generator between 0.5.8 and 0.5.9 Created: 03/Jan/15  Updated: 10/Apr/16

Status: Open
Project: test.check
Component/s: None
Affects Version/s: None
Fix Version/s: None

Type: Defect Priority: Major
Reporter: Brian Kim Assignee: Gary Fredericks
Resolution: Unresolved Votes: 0
Labels: None
Environment:

Clojure 1.5.1
OSX 10.9.5



 Description   

Tried updating test.check version to latest, but it seems like the map generator has gotten slower from version 0.5.9 up. I tried to make this easy to reproduce in the git repo below, but please let me know if you'd like more information/can't reproduce. Thanks!
https://github.com/brainkim/testcheckperf



 Comments   
Comment by Reid Draper [ 04/Jan/15 2:01 PM ]

Thanks Brian. I'm still digging in to this, but it's appearing like the performance regression is in keyword generation, not map generation. There was also a major change to keywords between 0.5.8 and 0.5.9 in 4528b5ed391d6127b79f4db6bc5e086613da0d81.





[TCHECK-31] Doc improvements: how to separate "size" from "range" Created: 11/Jun/14  Updated: 10/Apr/16

Status: Open
Project: test.check
Component/s: None
Affects Version/s: None
Fix Version/s: None

Type: Enhancement Priority: Major
Reporter: Michael Nygard Assignee: Gary Fredericks
Resolution: Unresolved Votes: 1
Labels: release-0.10.0


 Description   

Size is discussed mainly in terms of recursive generators. However, it is crucial to understanding other aspects such as vector length and integer ranges.

Some examples like these would be helpful:

  • How to get a small number of samples with a large range.
  • How to get a large number of samples with a small range.


 Comments   
Comment by Reid Draper [ 12/Jun/14 9:13 AM ]

Agree. I'll try and get a patch up in the next couple of days. Do you just have something in mind going in the doc/ dir?

Comment by Brian Craft [ 12/Nov/14 12:05 AM ]

I have exactly this problem. Are there any examples from projects using test.check?

Comment by Reid Draper [ 12/Nov/14 9:12 AM ]

Brian, can you be more specific about your problem? Some helpful functions regarding sizing and domain-generation are documented here. Take a look specifically at gen/choose, gen/resize, and gen/sized.

Comment by Brian Craft [ 12/Nov/14 10:29 AM ]

I don't need to test, for example, strings that are 500 chars long, but do need to test lists of 500 strings. If I have a generator that builds lists of strings & size it larger, both the lists and strings get larger. Then I start hitting irrelevant resource limits, like db field widths, or jvm memory. It seems like I need a way to specify how different data elements grow. Just setting the range, like (gen/vector ... 0 5000), seems wrong, because then it always picks up to 5000, while I suspect for shrinking it needs to respond to "size". Something like this?

(gen/sized (fn [size] (gen/resize (* size size) (gen/vector (gen/resize size gen/int)))))

forcing the size of the vector to increase more quickly than the size of the elements in the vector.

I gave this a try, but hit the vector stack overflow problem. I will try the fmap/flatten workaround, but am wondering if this is the right approach. Also wondering about the range of "size". From the code it looks like it goes to 100 due to sample-seq. So functions of size passed to gen/sized should expect range from 0-99, and return the largest meaningful data when given size parameter 99?

Comment by Reid Draper [ 15/Nov/14 12:03 PM ]

I don't need to test, for example, strings that are 500 chars long, but do need to test lists of 500 strings. If I have a generator that builds lists of strings & size it larger, both the lists and strings get larger.

Indeed, by default, all generators will 'grow' together. And as you've correctly noticed, you can use gen/sized and gen/resize to further control this.

Just setting the range, like (gen/vector ... 0 5000), seems wrong, because then it always picks up to 5000, while I suspect for shrinking it needs to respond to "size". Something like this?

Shrinking is completely separate from the size parameter. You can safely use (gen/vector ... 0 n) without negatively affecting shrinking.

(gen/sized (fn [size] (gen/resize (* size size) (gen/vector (gen/resize size gen/int)))))

forcing the size of the vector to increase more quickly than the size of the elements in the vector.

I gave this a try, but hit the vector stack overflow problem. I will try the fmap/flatten workaround, but am wondering if this is the right approach.

Your example looks fine. And you can avoid the stack overflow problem by simply not allowing size to grow above a certain limit, say 5000: (gen/resize (min 5000 (* size size)) ...).

(gen/sized (fn [size] (gen/resize (* size size) (gen/vector (gen/resize size gen/int)))))

forcing the size of the vector to increase more quickly than the size of the elements in the vector.

Also wondering about the range of "size". From the code it looks like it goes to 100 due to sample-seq. So functions of size passed to gen/sized should expect range from 0-99, and return the largest meaningful data when given size parameter 99?

When testing, the default max size is 200. However, you can override this simply by passing in a :max-size n argument to tc/quick-check. For example: (tc/quick-check 100 my-prop :max-size 75).

Hope this helps, and don't hesitate to ask anything else.

Comment by Brian Craft [ 18/Nov/14 11:41 AM ]

At the moment the hardest bit for me is understanding the shrinking. Composing generators, I inadvertently build things that won't shrink. E.g. to generate a 2d matrix of numbers with row & column names, I make use of bind so I can size vectors of names to match the matrix size, but then the resulting matrix/vectors will not shrink.

Here's a small example. Generate a vector of short strings, and test that they are all unique.

(tc/quick-check 40 (prop/for-all [f (gen/bind gen/int (fn [i] (gen/vector (gen/resize 5 (gen/such-that not-empty gen/string-ascii)) i)))] (do (println "f" f "count" (count f)) (= (count f) (count (set f))))))

Run this a few times & it will hit a duplicate string, failing the test. It is then unable to shrink the vector.

Here's a case generating two vectors of the same size by generating one vector, then using bind to generate a second of the same size. Testing uniqueness on the second vector, it is able to shrink, however it does so by regenerating the 2nd vector for every test.

(tc/quick-check 40 (prop/for-all [f (gen/bind (gen/vector (gen/resize 5 (gen/such-that not-empty gen/string-ascii))) (fn [v] (gen/hash-map :a (gen/return v) :b (gen/vector gen/int (count v)))))] (do (println "f" f "count" (count (:a f))) (= (count (:b f)) (count (set (:b f)))))))

I expect this will shrink poorly in many cases, because regenerating the second vector destroys the (possibly very rare) failure condition. I suspect this is why my 2d matrices shrink poorly. One really wants to be able to shrink by dropping the same positional element from both vectors. In the case of a fixed dimension (two), you can rewrite it to generate a vector of tuples. For a 2d matrix, though, I'm not sure what to do. Ideally it would shrink by dropping either a column or a row, and the corresponding column or row label. Perhaps I need to write a recursive generator like vector?

Are there any docs/papers describing how the shrinking works?

Comment by Gary Fredericks [ 20/Nov/14 9:45 AM ]

I've included a generator called cap-size in the test.chuck library.

Comment by Reid Draper [ 30/Nov/14 6:11 PM ]

I think both max-size and min-size functions would be a nice addition to test.check proper.

Comment by Jay A. Patel [ 22/Jan/15 7:29 PM ]

Hi Reid –

I am not sure if this is the right place for my comment, but it's closely related to this issue –

As you show, one can use `resize` to reconfigure the default generators. However, here is one issue with default generators for numbers. The default numeric generators (`int`, `nat`, `pos-int`, etc.) all generate numbers < 100 because default `size` is 100. Given JVM interns Integer/Long objects <= 127:

user=> (identical? 127 127)
true
user=> (identical? 128 128)
false

This could be an issue where test.check default number generators miss out on certain class of errors. I opine that a default value of `size` as 100 makes good sense for collections, but not for numbers.

If you agree, a few alternatives could be:

  • change default `size` to > 128, say 256
  • change `int` generator to choose between >1 multiple of size
  • have two different defaults: one for `length` (or `range`) and one for `size`

I don't know though. I think all alternatives have a drawback. I maybe missing something.

Comment by Gary Fredericks [ 22/Jan/15 8:03 PM ]

We've talked a bit about your third option, perhaps having two different classes of numeric generators. Clearly not ever generating big numbers is a pretty bad default for a lot of applications.

Comment by Gary Fredericks [ 26/Feb/16 9:29 AM ]

Jay,

A couple more comments:

  1. It's actually always been the case that the default size is 200 rather than 100, so the edge case you're describing should already be tested for tests that have more than 128 runs; I assume the reason you thought the default size was 100 is because that's what gen/sample uses for no reason in particular, so it might be worth changing that to 200 just to avoid this sort of confusion in the future
  2. As of 0.9.0 there are new integer generators (gen/large-integer and gen/large-integer*) aimed at use cases where small integers don't make sense, so that should help as well




[TCHECK-34] clojure.test reporting is uninformative Created: 21/Jun/14  Updated: 10/Apr/16

Status: Open
Project: test.check
Component/s: None
Affects Version/s: None
Fix Version/s: None

Type: Defect Priority: Major
Reporter: Philip Potter Assignee: Gary Fredericks
Resolution: Unresolved Votes: 0
Labels: release-0.10.0


 Description   

The clojure.test reporting results in output like:

FAIL in (always-fail) (clojure_test.clj:18)
expected: result
  actual: false

Note that the file and line number are always clojure_test.clj:18 and the expected value is always the literal string "result".

I'm not sure what the right output would be for expected/actual, but the incorrect file and line reporting means that clojure-test-mode consistently highlights the wrong line.



 Comments   
Comment by Philip Potter [ 21/Jun/14 9:10 AM ]

I think this is because the assertion is checked by calling clojure.test/is. clojure.test/is is being called for its return value, but it has the side effect of reporting failures. It's a macro, and it takes the literal symbol "result" as its expected expression, and takes the current file and line number to attach to the failure. It's really designed to be called by user code, not library code.

Comment by John Walker [ 11/Aug/14 3:55 PM ]

Well, one solution would involve taking advantage of the report functionality.

https://github.com/clojure/clojure/blob/master/src/clj/clojure/test.clj#L204
https://github.com/clojure/clojure/blob/master/src/clj/clojure/test.clj#L371





[TCHECK-96] quick-check should accept an arg for maximum shrinking time Created: 29/Feb/16  Updated: 10/Apr/16

Status: Open
Project: test.check
Component/s: None
Affects Version/s: None
Fix Version/s: None

Type: Enhancement Priority: Major
Reporter: Gary Fredericks Assignee: Gary Fredericks
Resolution: Unresolved Votes: 0
Labels: release-0.10.0





[TCHECK-95] quick-check should report how long it ran Created: 29/Feb/16  Updated: 22/Apr/16

Status: Open
Project: test.check
Component/s: None
Affects Version/s: None
Fix Version/s: None

Type: Enhancement Priority: Major
Reporter: Gary Fredericks Assignee: Gary Fredericks
Resolution: Unresolved Votes: 0
Labels: release-0.10.0

Attachments: Text File add_time_to_result_map.patch    

 Description   

Just an extra key in the return value, showing milliseconds that it was running.

Could maybe also add a key to the shrunk map saying how long it spent shrinking.



 Comments   
Comment by Unnikrishnan Geethagovindan [ 20/Apr/16 2:55 AM ]

Adding

  • :time-elapsed "100 ms" for success
  • :failed-after "2 ms" for failure
    • :time-spent-shrinking "2 ms" for time spent shrinking

or would a common key :elapsed "10 ms" would be better for all the 3 cases ?

Comment by Nicolás Berger [ 22/Apr/16 8:34 AM ]

I think this feature could be implemented as a reporter-fn. It doesn't even need to be in the c.t.check or c.t.c.clojure-test namespace.





[TCHECK-99] Create a generator that will limit the size of a generated sequence Created: 30/Mar/16  Updated: 06/May/16

Status: Open
Project: test.check
Component/s: None
Affects Version/s: None
Fix Version/s: None

Type: Enhancement Priority: Major
Reporter: Matthew O. Smith Assignee: Gary Fredericks
Resolution: Unresolved Votes: 0
Labels: None

Attachments: Text File TCHECK-99-2.patch     Text File TCHECK-99-3.patch     Text File TCHECK-99-4.patch     Text File TCHECK-99-5.patch     Text File TCHECK-99.patch    

 Description   

There are a lot of generators that return a sequence (list, vector, tuple, map, string, etc). Sometimes it us useful to set size limits, either min or max, on the generated sequence. It would be nice to have a generator that would accept a sequence generator and ensure that the sequences are of a certain length. Three examples would be:
(of-length min max gen)
(of-max-length max gen) => (of-length 0 max gen)
(of-min-length min gen) => (of-length min nil gen)

of-length check the length of the generated sequence. If it is too small, use (take min (cycle s)) to extend the length of the sequence.
If it is too long, use (take max s) to return the max length
It will need to be careful to return the same type it received.
If it does not receive a sequence, treat it as though it was one element sequence.
If min is not 0, use such-that not nil to ensure a proper seq is generated.



 Comments   
Comment by Gary Fredericks [ 30/Mar/16 5:49 PM ]

Most of the collection generators already accept options for controlling their size; is there a use case you have in mind where those controls are not sufficient?

Comment by Matthew O. Smith [ 30/Mar/16 7:22 PM ]

I completely missed those that take the :min and :max options

In particular I was looking at the string based generators as I need to populate database tables and so the strings have to match the column definition like not null and varchar (15). As I was thinking about the problem it seemed like writing a composable generator to enforce length restraints might make more sense than trying to retro fit all the generators that produce sequence-like structures.

A short check of generators.cljc shows list, hash-map and even the array based ones could use min and max options.

Comment by Matthew O. Smith [ 31/Mar/16 9:54 AM ]

If I try to update string from a def to a defn in an attempt to allow multiple arguments for sizes like

(defn string 
  ([]  (fmap clojure.string/join (vector char)))
  ([size]  (fmap clojure.string/join (vector char size)))
  ([min max]  (fmap clojure.string/join (vector char min max))))

Then (sample string) no longer works but (sample (string)) (sample (string 5)) (sample (string 3 7)) all work

Is there a trick to getting a just "string" to work? Or is there a better way to modify string to accept multiple arugments?

Comment by Gary Fredericks [ 31/Mar/16 10:32 AM ]

I don't think there's any clean approach to solving the problem you noted, which is one reason that I don't see an obvious solution here.

I faced this issue when creating the large-integer and double generators, and I decided to create two generators: large-integer is a generator with default behavior, and large-integer* is a function that takes options and returns a generator.

In hindsight I don't know if this was the best choice, since it's confusing. I asked David MacIver about how he handles this in hypothesis (a similar python library) and he said he doesn't have any raw generators, just functions that return generators (sometimes with 0 arguments). I like the uniformity of that approach, but it would obviously be a big breaking change for test.check (though there are some hacky tricks that could preserve backwards compatibility).

That issue is a bit bigger than this ticket. With respect to the code you showed, I think I'd prefer an API closer to what gen/set has, with an options map rather than positional args.

Now that you've got me thinking about this, I'm starting to get fixated on the idea of a big breaking API change that uses hacks to preserve backwards compatibility for a few versions, for the sake of cleaning up a lot of inconsistencies. But again, that's bigger than this ticket.

If we can't come up with a clean short-term approach, my standards for accepting things are much more relaxed at test.chuck ☺.

Comment by Matthew O. Smith [ 31/Mar/16 11:22 AM ]

Which do you prefer? adding something like of-length to limit existing seq generators or create (def string ...) and (defn string* ...)? I am just finishing up the Unicode implementation for TCHECK-97 as well so I'll follow which ever pattern we decide.

Comment by Gary Fredericks [ 31/Mar/16 1:28 PM ]

Don't we have at least three generators for strings? Are you thinking of having alternatives to each one?

Comment by Matthew O. Smith [ 31/Mar/16 3:21 PM ]

My very thought for making this request. Do we really want to double the number of string generators? Not to mention other things like arrays etc.

Another thought would be to add a single new generator, to-string, that can take a different character generator and use it to make strings based on an element generator:

(defn to-string
   ([element-gen] (to-string element-gen {})))
   ([element-gen [{:keys [num-elements min-elements max-elements max-tries ] :or {num-elements nil min-elements 0 max-elements nil max-retries 10}}]
                   (fmap clojure.string/join (vector char-gen ==apply-options==))))

I left out the code to handle the options properly but hopefully you get the idea.

It would be called like (to-string char {:min-elements 5 :max-elements 10}) or (to-string char-ascii {:num-elements 5})

This way it would be possible to add new character generators that could easily be converted to strings.

Comment by Matthew O. Smith [ 31/Mar/16 4:52 PM ]

Looking at it, coll-distinct-by almost does exactly what I need, except the allow-dups? flag does not do what I was expecting.

(sample (coll-distinct-by [] identity true false (elements [\a \b \c \d \e ]) {:num-elements 4})) always return distinct elements even though allow-dups? is true

Comment by Gary Fredericks [ 31/Mar/16 6:05 PM ]

coll-distinct-by is pretty specialized and probably not relevant here.

I think you could translate `[num-elements min-elements max-elements]` to the args taken by `gen/vector` pretty easily. `max-retries` shouldn't apply.

Something like to-string might be good, especially if it fits with whatever you're thinking about for TCHECK-97 (e.g., TCHECK-97 provides args to use with to-string).

Comment by Matthew O. Smith [ 31/Mar/16 7:50 PM ]

A patch generated with git --no-pager diff master feature/TCHECK-99 on https://github.com/m0smith/test.check.git

Comment by Matthew O. Smith [ 31/Mar/16 7:54 PM ]

Ignore that patch. I am way out of date with master. New patch on its way

Comment by Matthew O. Smith [ 31/Mar/16 8:01 PM ]

TCHECK-99-2.patch was generated after syncing test.check with my fork. Should be the latest.

Comment by Matthew O. Smith [ 01/Apr/16 10:34 AM ]

I also had a thought on how to allow a function to be treated like a generator. Add a :generator-factory meta data to a function like

(defn to-string
  "Generates strings of element-gen, default to char"
   {:generator-factory to-string}
   ([] (to-string char {})))
   ([element-gen] (to-string element-gen {})))
   ([element-gen [{:keys [num-elements min-elements max-elements]}]
                   (fmap clojure.string/join (vector char-gen ==apply-options==))))

The generator? could then be updated to accept anything with a :generator-factory meta data
call-gen could be updated to look for the new meta data and if there, call the associated function and use the result as the Generator.

Comment by Gary Fredericks [ 01/Apr/16 10:50 AM ]

yeah, that's basically the idea I had for preserving backward compatibility – the metadata generator would have some sort of deprecation warning attached somehow.

Comment by Matthew O. Smith [ 01/Apr/16 11:00 AM ]

Also, on the to-string, clojure.string/join supports a seperator which we be easily added as an additional option like:

(to-string int {:sep ","})

If you are ok with it, I will update the patch.

Comment by Matthew O. Smith [ 02/Apr/16 3:41 PM ]

TCHECK-99-3.patch is updated to support a :sep option. Also, had it based upon a new function 'apply-to' that will be useful for TCHECK-97 as well. See https://github.com/m0smith/test.check/tree/feature/TCHECK-97 for the proposed Unicode implementation

Comment by Matthew O. Smith [ 03/Apr/16 12:57 PM ]

This one has support for suffix and prefix. Using this one makes thing like HTML much easier

Comment by Matthew O. Smith [ 03/Apr/16 4:25 PM ]

Here is an example of how to-string would be used to simulate the lines in a tab-separated file

(def gen-char-upper (gen/fmap char (gen/choose \A \Z)))
(def gen-char-lower (gen/fmap char (gen/choose \a \z)))
(def gen-char-digit (gen/fmap char (gen/choose \0 \9)))
(def gen-char-postal-format (gen/fmap char (gen/one-of [(gen/return \#) 
                                                        (gen/return \@)
                                                        gen-char-upper])))

(def gen-iso-2 (gen/to-string gen-char-upper {:num-elements 2}))
(def gen-iso-3 (gen/to-string gen-char-upper {:num-elements 3}))
(def gen-iso-numeric (gen/to-string gen-char-digit {:num-elements 3}))
(def gen-fips (gen/to-string gen-char-upper {:num-elements 2}))
(def gen-country gen/string)
(def gen-capital gen/string)
(def gen-area  (gen/large-integer* {:min 100 :max 999999999}))
(def gen-population (gen/large-integer* {:min 100 :max 999999999}))
(def gen-continent (gen/elements ["AS" "NA" "SA" "EU" "OC" "AF"]))
(def gen-tld (gen/to-string gen-char-lower {:num-elements 2 :prefix "."}))
(def gen-currency-code (gen/to-string gen-char-upper {:num-elements 3}))
(def gen-currency-name gen/string)
(def gen-phone  (gen/large-integer* {:min 1 :max 999}))
(def gen-simple-language (gen/to-string gen-char-lower {:num-elements 2}))
(def gen-language (gen/one-of [gen-simple-language 
                               (gen/to-string [gen-simple-language (gen/return "-") gen-iso-2])]))
(def gen-languages (gen/to-string gen-language {:sep ","}))
(def gen-geonameid (gen/large-integer* {:min 1 :max 999999999}))
(def gen-neighbors (gen/to-string gen-iso-2 {:sep ","}))
(def gen-equivilent-fips (gen/to-string gen-char-upper {:num-elements 2}))

(defn to-postal-regex [postal-format]
  (str "^" (clojure.string/join (map #(cond (= \# %) "\\d" (= \@ %) "[A-Z]" :default %) postal-format)) "$"))

(def gen-country-line 
  (gen/let [postal-format (gen/to-string gen-char-postal-format {:min-elements 2 :max-element 9})]
    (gen/to-string [gen-iso-2 gen-iso-3 gen-iso-numeric gen-fips
                    gen-country gen-capital gen-area gen-population gen-continent
                    gen-tld gen-currency-code gen-currency-name gen-phone
                    (gen/return postal-format) (gen/return (to-postal-regex postal-format))
                    gen-languages gen-geonameid gen-neighbors gen-equivilent-fips
                    ] {:sep \tab})))
Comment by Gary Fredericks [ 10/Apr/16 5:08 PM ]

If I had been trying to write a generator like gen-country-line I probably would have combined gen/tuple and string/join, e.g.:

(gen/let [fields (gen/tuple gen-iso-2 gen-iso-3 ... gen-equivilent-fips)]
  (string/join \tab fields))

it looks like you wrote `gen/to-string` to accept either a generator or a vector of generators? I don't like that kind of ambiguity generally, and I think the example used of gen/let just above demonstrates that you can live without the vector feature without much pain.

Is the :sep feature still as compelling without the vector feature? I'm a little unsure about :sep, at least inasmuch as I'm hesitant to add features in general without a strong reason to.

Comment by Matthew O. Smith [ 11/Apr/16 4:42 PM ]

I see what you are saying and that makes a lot of sense. It would be good to include that in the docs as an example.

The :sep is used between the fields of the result regardless if one generator or multiple are provided. It makes it nice to create a comma separated list of integers (to-string int {:sep ","})

Comment by Matthew O. Smith [ 06/May/16 3:09 PM ]

I removed all the extra options to to-string. I also added an example of creating an XML generator.





[TCHECK-112] bind doesn't shrink very well Created: 22/Jun/16  Updated: 26/Jun/16

Status: Open
Project: test.check
Component/s: None
Affects Version/s: None
Fix Version/s: None

Type: Defect Priority: Major
Reporter: Gary Fredericks Assignee: Gary Fredericks
Resolution: Unresolved Votes: 0
Labels: None


 Description   

This is a long-standing Hard Problem.

The Problem

The classic example is shrinking a 2d matrix with a poisoned element in it:

(def gen-matrix
  (gen/let [width gen/nat]
    (gen/vector (gen/vector gen/large-integer width))))

(quick-check 10000
             (prop/for-all [matrix gen-matrix]
               (->> matrix
                    (apply concat)
                    (not-any? #{42}))))
;; =>
{:result false,
 :seed 1466646880334,
 :failing-size 16,
 :num-tests 17,
 :fail [[[290 42 10 3 1 3 196]
         [-1793 3484 -5795 -206 -1 -8 464]
         [2 3 -1951 -761 -28829 5518 1]
         [-32 4477 1 -4086 0 1640 -22185]
         [-485 3156 -625 4082 -2 -845 513]
         [-3 -1 26 323 232 5 -1]
         [32 51 -1 240 -1814 0 -190]
         [2417 -4239 326 -4096 -8 1898 75]
         [-509 1 0 466 199 -1 10]
         [-23 5838 -441 30741 -6724 -1169 -171]
         [-4 3974 -1432 -4 698 -56 1210]
         [-2148 -6526 -1 453 19 -5343 461]]],
 :shrunk {:total-nodes-visited 31,
          :depth 8,
          :result false,
          ;; would be better if shrunk to [[[42]]]
          ;;
          ;; note that it's plausible the smallest value here could have the same width (7)
          ;; as the original failure -- the only reason it was able to shrink the width at
          ;; all was that it got lucky by generating an entirely new matrix with the smaller
          ;; width that happened to have 42 in it
          :smallest [[[0 42 0]]]}}

Ideas






[TCHECK-106] Variable-sized collection generators have exponential sizing issues when composed Created: 06/Jun/16  Updated: 06/Jun/16

Status: Open
Project: test.check
Component/s: None
Affects Version/s: None
Fix Version/s: None

Type: Defect Priority: Major
Reporter: Gary Fredericks Assignee: Gary Fredericks
Resolution: Unresolved Votes: 0
Labels: None


 Description   

E.g., this sort of code can OOM:

(-> gen/boolean 
    gen/vector
    gen/vector
    gen/vector
    gen/vector
    gen/vector
    gen/vector
    gen/generate)





[TCHECK-102] Provide a mechanism for getting output from a test run after interrupting it. Created: 10/Apr/16  Updated: 06/Aug/16

Status: Open
Project: test.check
Component/s: None
Affects Version/s: None
Fix Version/s: None

Type: Enhancement Priority: Major
Reporter: Gary Fredericks Assignee: Gary Fredericks
Resolution: Unresolved Votes: 0
Labels: None


 Description   

Related: TCHECK-4, TCHECK-8.



 Comments   
Comment by Gary Fredericks [ 10/Apr/16 5:59 PM ]

For TCHECK-4 I imagine this could only be done by some seriously sad monkeypatching.





[TCHECK-101] Add a prop/for-all alternative to c.t.c.clojure-test that uses clojure.test/is &c Created: 10/Apr/16  Updated: 06/Aug/16

Status: Open
Project: test.check
Component/s: None
Affects Version/s: None
Fix Version/s: None

Type: Enhancement Priority: Major
Reporter: Gary Fredericks Assignee: Gary Fredericks
Resolution: Unresolved Votes: 0
Labels: None


 Description   

This would be similar to https://github.com/gfredericks/test.chuck/blob/9f424c474b02f6e89b423049b5c5daebf71cec9c/src/com/gfredericks/test/chuck/clojure_test.cljc#L112.

Have to think carefully about when to print clojure.test's standard failure stuff with respect to shrinking.

on the other hand

am I just wanting to do this because of the "test output is bad" problem? Is there a better solution that attacks that directly?






[TCHECK-97] Support Unicode chararacters for char based generators Created: 06/Mar/16  Updated: 06/Aug/16

Status: Open
Project: test.check
Component/s: None
Affects Version/s: None
Fix Version/s: None

Type: Enhancement Priority: Major
Reporter: Matthew O. Smith Assignee: Gary Fredericks
Resolution: Unresolved Votes: 0
Labels: None


 Description   

Currently the default char generator is only in the range from 0 to 255. Java chars can range from \0000 to \FFFF. If this is something of interest, I will add a patch as I need to do this anyway.



 Comments   
Comment by Gary Fredericks [ 06/Mar/16 1:53 PM ]

There's definitely a need for this, but I don't think the details of a solution are obvious – in particular what distribution such a generator ought to have.

My hazy understanding of unicode is that a great many (the majority I think) of the code points are not assigned to any particular character, and so if you picked code points at random you would mostly get unprintable stuff.

I spent a while on this issue when I implemented string-from-regex in test.chuck (using a uniform distribution), and you can see the results by doing (gen/sample (com.gfredericks.test.chuck.generators/string-from-regex #".*")):

("" "󲦯" "" "򝡁񌨠񆂧" "" "􇆞򸣒𡖎򫶯" "򅬢򘏁󏉊񪦛" "z𫺑" "񷴡𽟔󙤓" "")

I could see an argument that this is the only reasonable implementation of gen/char, but I don't think that's obviously true since from a user's perspective it would mostly be annoying to see and seem rather silly.

I'm also open to arguments of the form "yes there's no good comprehensive solution to gen/char but implementation X is at least strictly better than the current implementation".

I'd be interested to have somebody research what other quickchecks (in particular haskell & python) do for this as well.

Comment by Matthew O. Smith [ 07/Mar/16 7:38 AM ]

You make some great points. I will also review the Java Character class as it seems to have some Unicode information encoded that could be put to good use.

Comment by Matthew O. Smith [ 02/Apr/16 3:44 PM ]

;;
;; Unicode support for test.check
;;
;; Unicode support is divided into 2 sections: char based and code-point/int based
;;
;; Ranges and choices
;; Ranges are a vector of range defs
;; A range def is either
;; A single character
;; A pair (vector) of the start and end of a range
;;
;; choices is a generator that choose from a vector of ranges. For example,
;; (choices [1 2 [100 200])
;; would return 1 and 2 and the numbers from 100 to 200. The members of the range pair 100 and 200 in this
;; example, can be anything accepted by choose.
;;
;;
;; The char based Unicode support mirrors the normal char and string generators
;;

Standard Generator Unicode Generator Generates
char uchar valid Unicode characters (char) from \u0000 to \uFFFF.
char-asciii uchar-alpha letter Unicode characters.
  uchar-numeric digit Unicode characters
char-alphanumeric uchar-alphanumeric letter and digit Unicode characters
string ustring Unicode strings consisting of only chars
string-alphanumeric ustring-alphanumeric Unicode alphanumeric strings.
  ustring-choices Unicode strings in the given ranges.
namespace unamespace Unicode strings suitable for use as a Clojure namespace
keyword ukeyword Unicode strings suitable for use as a Clojure keyword
keyword-ns ukeyword-ns Unicode strings suitable for use as a Clojure keyword with optional namespace
symbol usymbol Unicode strings suitable for use as a Clojure symbol
symbol-ns usymbol-ns Unicode strings suitable for use as a Clojure symbol with optional namespace

;; Code-point or int based characters

Standard Generator Unicode Generator Unicode Desc
string ustring-from-code-point Generates Unicode strings consisting of any valid code point.
char code-point Generates a valid Unicode code point
Comment by Gary Fredericks [ 10/Apr/16 5:53 PM ]

Are you thinking that these generators will generally have uniform distributions, and that the problem of mostly-unprintable-values is not a big enough problem to do anything special about?

Should the second group of generators include analogs for keyword, symbol, etc. as well?

I think anything that involves dozens of new generators I'll be inclined to put in a separate namespace.

Comment by Matthew O. Smith [ 11/Apr/16 12:00 PM ]

I listed all the new generators I was wanting to build. Basically, I want to map the normal string based generators to have similar behavior to current ones. For example, keywords and symbols have a ukeyword and usymbol for unicode keywords and symbols.

Adding the apply-to from TCHECK-99 will make it easier for people to create a Unicode string generators.

I expect the Unicode versions of the functions to have a very similar distribution to the current versions. The exception is the ones based on "choices" which distributes even across each range, regardless of the size of the range.





[TCHECK-87] New stats feature: Adds ability to assign labels to test cases to report the test case distribution when running a test Created: 03/Dec/15  Updated: 06/Aug/16

Status: Open
Project: test.check
Component/s: None
Affects Version/s: None
Fix Version/s: None

Type: Enhancement Priority: Major
Reporter: Nicolás Berger Assignee: Nicolás Berger
Resolution: Unresolved Votes: 0
Labels: None

Attachments: Text File add-stats-collect-and-classify.patch    
Patch: Code and Test

 Description   

Inspired in the analog feature that exists in Haskell's QuickCheck. Adds a classify fn that is intended to be used as a wrapper of prop/for-all, returning a property (a generator) appropriate for test.check/quick-check that will augment the result-map returned by the underlying property, adding the collected labels under the :labels key. Also triggers a new event :stats in the default-reporter-fn whose default implementation calls test.check.stats/print, printing the classification of trials with the following format:

12.7% :lt-30
14.5% :gte-30
29.1% :lt-30, :lt-20
43.6% :lt-30, :lt-10, :lt-20

(note that multiple labels might be assigned to some test cases)

I think it answers the question "How could we collect stats about the sorts of things generated?" from the test.check design page



 Comments   
Comment by Nicolás Berger [ 28/Feb/16 7:52 PM ]

The patches here no longer apply after the introduction of reporter-fn.

While I'm working on new patches, I'd like to know other people's (and especially gfredericks) opinion on having this new :labels concept in test.check. Each trial's result map can optionally include some :labels that are included in the :complete report so the stats can be computed.

An alternative would be to implement the stats as an external library/reporter, but to make that possible we need the trials loop to keep arbitrary state from each trial's result-map where we could accumulate the labels. Or maybe we could make the (reporter-fn :type :trial ...) to include the entire result-map so the reporter-fn has the chance to "save" this state, but this way doesn't look very elegant...

Comment by Gary Fredericks [ 29/Feb/16 9:40 PM ]

Things to think about:

  • The stats are only a function of the args, so is it necessary to wrap the entire property?
    • haskell does this though
  • On the other hand, I like the idea of stats being a particular instance of a generic feature of the check namespace, but the current implementation doesn't quite achieve that since we have lots of label stuff in the check namespace
  • Buuuut it's hard to move it out because if check just exposed a generic reduce feature, then users going directly to quick-check would have to supply two things – a wrapped property, and the proper reduce function
Comment by Nicolás Berger [ 12/Mar/16 1:07 PM ]

Uploaded a new patch that takes advantage of the new reporter-fn mechanism.

In this new patch some stuff was simplified:

  • Removed the new report-stats var that was going to be used as a way to enable/disable printing of stats for each test. The default-reporter-fn will now print the stats only when there are labels present in the result-map. To not print the stats it's just a matter of using a different :reporter-fn that doesn't print the stats.
  • In the stats report only lines with labels are printed now. In the previous version there was a line with the percentage of trials with "No labels". I removed it now to make it less verbose.

Also docstrings were added, made it sure the stats work in clojurescript and other minor improvements.

Comment by Nicolás Berger [ 19/Mar/16 2:26 PM ]

I'm thinking it would be nice to give more ways for assigning labels to trials. For example:

1. (classify prop): It just uses the vector of args as the label.
2. (classify prop label-fn): Obtains the label by applying a label-fn to the args. Example: (classify prop count) - assigns the count of the arg (a vector, string, etc) as the label
3. (classify prop label-fn pred): Applies label-fn only when pred yields a truthy value. Example: (classify prop count #(> (count %) 1)) - assigns the count of the arg as the label, but only when count is greater than 1.
4. (classify prop pred): Uses the vector of args as the label, but only when pred yields a truthy value. Example: (classify prop #(<= (count %) 1)) - assigns the count of the arg as the label, but only when count is lower or equal to 1.
5. (classify prop pred label): This is the current signature. Assigns label only when pred yields truthy

So I'm thinking about changing the signature to receive a map of (:pred :label-fn :label) possible keys. The three keys are optional. :label and :label-fn can't be both present at the same time.

From what I could understand, Haskell QuickCheck (https://hackage.haskell.org/package/QuickCheck-2.8.2/docs/Test-QuickCheck-Property.html) has different functions to provide similar alternatives:
(classify prop) would be similar to collect in Haskell QC
(classify prop label-fn) would be similar to label in Haskell QC
(classify prop label-fn pred) would be similar to classify in Haskell QC

What do you think @gfredericks?

Comment by Gary Fredericks [ 05/Apr/16 10:02 PM ]

After discussing it, my hunch is that 2 and 5 are the most natural ones, maybe calling 2 collect to mirror the haskell version.

We talked about maybe treating nil as a flag for not labelling in collect, but I don't particularly like that idea I don't think.

Comment by Gary Fredericks [ 05/Apr/16 10:09 PM ]

I should also not forget that this might overlap with changes to address the "Test Failure Feedback" issue on the confluence page: http://dev.clojure.org/display/design/test.check

Comment by Gary Fredericks [ 05/Apr/16 10:09 PM ]

I should also not forget that this might overlap with changes to address the "Test Failure Feedback" issue on the confluence page: http://dev.clojure.org/display/design/test.check

Comment by Nicolás Berger [ 10/Apr/16 10:49 PM ]

We talked about maybe treating nil as a flag for not labelling in collect, but I don't particularly like that idea I don't think.

Perhaps we can use a namespaced keyword to signal that a label should be ignored? Something like clojure.test.check.stats/ignore. This way we can easily implement classify in terms of collect by creating a function that returns stats/ignore when pred doesn't match:

(defn collect
  [prop label-fn]
  (gen/fmap
    (fn [{:keys [args] :as result-map}]
      (let [label (apply label-fn args)]
        (if (= ::ignore label)
          result-map
          (update result-map :labels conj label))))
    prop))

(defn classify
  [prop pred label]
  (collect prop (fn [& args]
                  (if (apply pred args)
                    label
                    ::ignore))))

Another option could be to add an extra arity in collect, to receive a flag about whether nil should be treated as a label or if it should be ignored. I like :stats/ignore more

Comment by Nicolás Berger [ 19/Apr/16 9:19 AM ]

Replaced with new patch that adds both stats/classify & stats/collect as discussed. I think this new patch implements what was discussed, covering cases 2 & 5 with function names analog to the haskell impl, leaving out any special treatment for nil (it is a valid label) and not adding :stats/ignore as I suggested in a previous comment.





[TCHECK-70] Redesign gen/choose Created: 06/Jun/15  Updated: 06/Aug/16

Status: Open
Project: test.check
Component/s: None
Affects Version/s: None
Fix Version/s: None

Type: Enhancement Priority: Major
Reporter: Gary Fredericks Assignee: Gary Fredericks
Resolution: Unresolved Votes: 0
Labels: release-0.10.0


 Description   

Currently gen/choose is the generator that everything else is based off of, and it's also a user-facing generator. Its current implementation takes a [0,1) double and scales it to the desired range. This approach degrades w.r.t. distribution as you get higher, and presumably has pretty terrible properties once your range is larger than 2^53. It could also be optimized better for when the range is a power of 2 (no need to go through Double). Seeing as how gen/choose is user facing, we should attempt to redesign it so that it doesn't degrade in this fashion, and can handle bigints.

Requirements

gen/choose should have a uniform distribution regardless of bounds

gen/choose should accept arbitrary integer bounds and handle them correctly

Things to investigate

  • Note that the SplittableRandom class generates integer ranges without going through floating point. We should check if this is faster, since it's definitely a more uniform distribution.
  • Is it worth optimizing for the power-of-two case, the same with SplittableRandom does in the link above?
  • When it's possible for the generator to return a long instead of a bigint is somewhat subtle. E.g., if you call (gen/choose Long/MIN_VALUE Long/MAX_VALUE), then when analyzing the size of the range you'll have to use a bigint (because 2^64 is not representable as a long). gen/choose should generate longs when both args are in long range, probably?
  • Is it actually possible to satisfy the requirements above without a performance hit? You'd think it was just a generator-creation-time change, but bind can blur the line between generator creation and generation time.


 Comments   
Comment by Gary Fredericks [ 12/Nov/15 7:32 AM ]

On second thought, I might be more in favor of leaving choose the way it is, discouraging its use by users, and adding a uniform-integer generator that is specifically designed to handle arbitrary ranges. A big question is what it should do in JavaScript, and I have a couple ideas about that.





[TCHECK-111] The latest recursive-gen algorithm seems to exhibit a peculiar lack of variety in depths Created: 22/Jun/16  Updated: 29/Aug/16

Status: Open
Project: test.check
Component/s: None
Affects Version/s: None
Fix Version/s: None

Type: Defect Priority: Major
Reporter: Gary Fredericks Assignee: Gary Fredericks
Resolution: Unresolved Votes: 0
Labels: release-0.10.0


 Description   

In particular the output of (generate (recursive-gen vector nat) 1000) is rather less varied than I would hope.



 Comments   
Comment by Felix Andrews [ 29/Aug/16 10:18 PM ]

Yes, recursive-gen seems not to generate scalars, which means any-printable doesn't generate scalars.

(require '[clojure.test.check.generators :as g])
(->> (g/sample g/any-printable 1000) (remove coll?))
;; ()
Comment by Gary Fredericks [ 29/Aug/16 10:25 PM ]

recursive-gen should generate scalars on master

Comment by Felix Andrews [ 29/Aug/16 10:31 PM ]

D'oh, sorry for the noise.





[TCHECK-21] Rerunning a particular failure is difficult. Created: 11/Apr/14  Updated: 06/Aug/16

Status: Open
Project: test.check
Component/s: None
Affects Version/s: None
Fix Version/s: None

Type: Enhancement Priority: Major
Reporter: Gary Fredericks Assignee: Gary Fredericks
Resolution: Unresolved Votes: 0
Labels: None


 Description   

Test.check has the concept of a seed that is used when a property is tested. The seed can be supplied or generated by the current time. Either way, the single seed is used to generate all the trials for the test. If a test fails, the only way to try to re-test the property on the same inputs is to run all of the tests again using the same seed. If the tests were running for hours before a failure was found, this is prohibitive.

I would like instead to be able to check the property for exactly the input that failed. One step in that direction is to have an explicit different seed for each trial (perhaps generated from an initial metaseed), and on failure to report a [seed size] pair that can be used to run a test a single time.

The way this interacts with shrinking is interesting though, since it's just as likely that I'll want to re-run on the shrunk value from a failure. Since the shrunk value was never explicitly generated from a known [seed size], just that information is insufficient. We could perhaps report [seed size shrink-path] where the third element is a list of indexes to walk through the shrink tree with. Probably the worst part about that is it might be rather long.

In any case I think something like this would be more useful than the current setup. I'll probably be hacking up something similar for a fork branch I'm maintaining, but if we can settle on a specific design I'd be happy to cook up a patch.



 Comments   
Comment by Reid Draper [ 12/Apr/14 10:28 AM ]

When a test fails, we do return the arguments that originally failed, and the arguments of the shrunk test. Is this insufficient?

For example:

{:result false,
       :failing-size 45,
       :num-tests 46,
       :fail [[10 1 28 40 11 -33 42 -42 39 -13 13 -44 -36 11 27 -42 4 21 -39]],
       :shrunk {:total-nodes-visited 38,
                :depth 18,
                :result false,
                :smallest [[42]]}}

see :fail and [:shrunk :smallest].

Comment by Gary Fredericks [ 12/Apr/14 11:00 AM ]

It's certainly adequate to reproduce in theory, but I don't know of any built in mechanism that makes this easy. Here's what I end up doing:

Given this:

(defspec foo
  (prop/for-all [x gen]
    (f x)))

and after getting a failure with a 50-line shrunk value (e.g. :bar), I manually pretty-print it and paste it back into my file like so:

(def shrank
  ':bar)

(defspec foo
  (prop/for-all [x #_gen (gen/return shrank)]
    (f x)))

And now I can run (foo 1) to re-run with the failing value.

This awkwardness is partly due to three facts:

  1. My generators create large unwieldy values
  2. My property is a large body of code rather than a single function
  3. There's no easy way to test a property on a specific value

I could mitigate some of the pain by fixing #2 – having each one of my tests split into a defspec / prop/for-all and a plain function. But #1 is not easy to fix, and having a small tuple is rather nicer for me.

I've already written a POC for this and it's working rather well. I used the term :key to refer to the tuple, so the term doesn't conflict with :seed. I end up calling the test like (foo 0 :key [329489249329323 19 []]), but I'll probably think up something else that doesn't require passing a dummy first arg.

Comment by Gary Fredericks [ 03/Mar/16 10:09 AM ]

I think this kind of issue becomes more important with TCHECK-96, because if a shrink aborts prematurely it would be nice to be able to run it again without the time limit (without having to run all the prior passing tests like you would if you re-ran quick-check with the same seed).

I just had an intermediate idea for this that doesn't have the downside of the arbitrarily-long "key" that I've used in my fork: we promote the RNG to be a true serializable value, and then in quick-check we keep track of the [rng size] pair used for each trial, and on a failure we report that pair. Then we have a new function similar to quick-check that takes a property and a pair and runs a single trial (with shrinking if it fails).





[TCHECK-119] Checking of interesting edge cases is not very creative Created: 03/Sep/16  Updated: 03/Sep/16

Status: Open
Project: test.check
Component/s: None
Affects Version/s: None
Fix Version/s: None

Type: Enhancement Priority: Major
Reporter: Gary Fredericks Assignee: Gary Fredericks
Resolution: Unresolved Votes: 0
Labels: None


 Description   

There may not be a single solution to all of these, and some of them may be hard.

Some examples:

  • When specifying bounded integer ranges, we don't intentionally check near all of min,0,max
    • At first glance this sounds easy, but it involves making not-obvious decisions about the meaning of "small" examples and where to shrink to. E.g., should (gen/generate (gen/choose 5 Integer/MAX_VALUE) 3) be highly likely to generate Integer/MAX_VALUE? (Intuitively that may not match what the user means when passing 3 as the size); and if a test fails for 1000 but not for any smaller numbers, should it shrink to 1000 or to Integer/MAX_VALUE since the latter is "simpler" in a certain sense?
  • The size of a collection correlates with the size of its elements
    • i.e., (gen/list gen/nat) is highly unlikely to generate a list of 20 0's.





[TCHECK-120] size of args is conflated with depth in shrink tree Created: 03/Sep/16  Updated: 03/Sep/16

Status: Open
Project: test.check
Component/s: None
Affects Version/s: None
Fix Version/s: None

Type: Defect Priority: Major
Reporter: Gary Fredericks Assignee: Gary Fredericks
Resolution: Unresolved Votes: 0
Labels: None


 Description   

This came up when working on TCHECK-114.

I had written a generator of the form (gen/frequency [[1 gen-1] [2 gen-2]]) and was trying to figure out why failures that originated in gen-2 didn't always shrink to gen-1.

When I examined the shrink tree, I realized that the shrinking algorithm did try shrinking to gen-1, but even after finding a failure there it would continue looking for alternatives and eventually shrunk to something from gen-2, because it was deeper in the shrink tree and therefore believed to be smaller.

So the shrinking algorithm has no way to compare the "size" of inputs except for their depth in the search tree, which is sometimes misleading. I doubt there's a general solution to this besides maybe adding an option to quick-check for a custom comparator function.






[TCHECK-115] run tests in parallel on the jvm Created: 15/Aug/16  Updated: 03/Sep/16

Status: Open
Project: test.check
Component/s: None
Affects Version/s: None
Fix Version/s: None

Type: Enhancement Priority: Major
Reporter: Gary Fredericks Assignee: Gary Fredericks
Resolution: Unresolved Votes: 0
Labels: None


 Description   

Goals

  • deterministic – calling quick-check with parallel opts gives the same answer as without
    • we have to give up some determinism though, because I don't think we could reasonably
      say that the reporter-fn will be called the same way every time, or even that
      the :num-tests value returned should always be the same
  • does the best possible thing on the jvm w.r.t. things hanging
  • is it necessary to have some sort of dynamic var with a thread id or something similar,
    so users can use it to coordinate usage of global resources? alternatively they could
    use a ThreadLocal by hand, or a pool

Issues

  • Is this really worth it? The downside is it makes the API and impl of quick-check more complex, and arguably most use cases could be served by calling quick-check multiple times from separate threads; would a generic test-suite parallelizer do the job for most people?





[TCHECK-72] Poisson generator Created: 07/Jun/15  Updated: 07/Jun/15

Status: Open
Project: test.check
Component/s: None
Affects Version/s: None
Fix Version/s: None

Type: Enhancement Priority: Minor
Reporter: Jason Felice Assignee: Gary Fredericks
Resolution: Unresolved Votes: 0
Labels: None

Attachments: File poisson.diff    
Patch: Code

 Description   

Adds `gen/poisson`, which samples numbers from a Poisson distribution.



 Comments   
Comment by Jason Felice [ 07/Jun/15 1:40 PM ]
user=> (gen/sample (gen/poisson 1/500) 500)
(0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0)
user=> (gen/sample (gen/poisson 1/5) 500)
(1 0 0 0 0 0 0 0 0 0 0 0 0 1 0 1 0 0 0 0 0 0 0 0 0 0 0 0 1 0 0 0 1 0 0 1 0 0 0 0 0 0 0 0 1 1 1 0 0 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 1 0 0 0 1 1 0 1 0 0 0 0 0 0 0 0 0 1 0 0 0 0 1 0 0 0 0 0 0 0 1 0 0 0 0 0 0 1 0 0 1 1 0 0 0 0 0 1 0 0 0 1 0 0 0 0 0 0 0 1 0 0 1 0 0 0 0 0 0 0 0 0 0 0 1 0 0 1 0 0 0 0 0 0 1 0 1 0 0 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 0 0 0 1 0 0 0 0 0 0 0 0 1 0 1 0 1 0 0 0 0 0 0 0 1 1 0 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 1 1 0 0 0 0 0 1 0 0 0 1 0 0 0 0 0 1 1 0 1 0 0 0 1 0 0 0 0 0 0 0 0 1 0 0 1 1 0 0 0 0 0 0 0 0 1 1 1 0 1 0 0 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 1 0 1 0 0 1 1 0 1 1 0 0 0 0 0 1 0 0 0 0 0 0 0 0 1 0 0 1 1 0 0 1 1 0 0 0 1 0 0 0 0 0 0 0 0 0 1 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 0 0 1 1 0 0 0 0 0 0 0 0 0 0 1 1 1 0 0 1 1 0 1 0 0 0 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 0 1 1 0 0 0 0 0 0 1 0 0 0 0 0 0)
user=> (gen/sample (gen/poisson 1) 500)
(2 2 1 1 1 0 2 2 0 0 0 1 0 1 2 1 2 2 2 2 1 1 2 2 0 1 2 0 0 1 1 2 1 1 0 1 2 0 1 1 0 1 1 1 0 2 2 0 2 1 1 1 0 1 1 0 0 1 2 0 1 1 1 2 0 1 1 0 0 0 0 2 2 0 1 1 2 2 0 0 1 0 2 0 0 1 2 1 1 1 1 1 2 2 0 1 0 1 1 1 0 1 0 2 1 0 1 0 1 0 0 0 2 1 1 0 0 2 0 1 2 1 0 2 1 0 1 0 0 1 1 2 1 2 0 1 1 2 0 1 1 2 1 1 0 0 2 1 2 1 0 0 2 1 1 2 0 1 2 1 1 0 2 2 1 1 2 1 1 2 0 1 0 1 0 0 0 1 2 0 2 1 1 2 1 0 1 0 0 1 1 0 1 0 1 1 2 0 0 2 1 0 2 0 1 0 2 1 1 0 2 1 0 1 1 0 1 0 2 0 0 0 0 2 1 0 0 1 1 2 1 0 2 2 2 2 0 1 2 1 0 1 1 1 2 1 0 0 1 0 0 1 2 1 0 0 0 2 2 0 1 1 1 0 2 0 0 0 1 1 1 0 0 0 2 1 2 1 0 0 2 1 1 1 2 1 1 2 2 0 1 2 1 2 2 0 0 1 2 2 0 1 1 2 0 1 1 2 1 1 1 1 0 2 2 2 1 0 1 1 0 1 2 0 1 1 1 1 2 1 0 1 2 0 0 2 1 0 2 0 0 1 0 1 1 1 2 1 1 0 0 2 1 1 1 2 2 0 0 2 1 1 1 0 0 2 2 0 1 0 1 2 1 1 0 0 0 2 0 1 1 1 2 2 1 1 0 2 1 0 1 2 1 0 0 0 1 0 0 2 1 1 0 1 2 1 1 0 2 1 0 0 0 1 2 1 1 2 1 1 0 2 0 0 2 2 1 0 2 0 2 2 2 1 1 2 0 0 0 1 0 1 2 0 2 1 2 0 1 0 1 2 1 0 1 0 1 0 0 0 1 1 2 2 0 0 2 1 1 1 2 1 1 0 2 1 2 0 0 1 0 0 1 0 1 0 0 0 1 1 1 0 1 0 1 0 0 0 0 2)




[TCHECK-71] Exponential generator Created: 07/Jun/15  Updated: 07/Jun/15

Status: Open
Project: test.check
Component/s: None
Affects Version/s: None
Fix Version/s: None

Type: Enhancement Priority: Minor
Reporter: Jason Felice Assignee: Gary Fredericks
Resolution: Unresolved Votes: 0
Labels: None

Attachments: File exponential.diff     File exponential.diff    
Patch: Code

 Description   

A generator for generating exponential values. I'm using this to generate queue latency values, to simulate out-of-order processing of events in some test.check specifications.



 Comments   
Comment by Jason Felice [ 07/Jun/15 1:31 PM ]

Corrected exponential.diff (previous one missing parenthesis)

Comment by Jason Felice [ 07/Jun/15 1:33 PM ]
user=> (require '[clojure.test.check.generators :as gen])
nil
user=> (gen/sample (gen/exponential 1/500))
(2377.5877207475132 320.68681344358293 144.18262694905863 1422.342802626106 300.8014248144241 79.8053977813952 36.18465309993865 93.65252446233575 806.4125368253829 37.88916712893676)
user=> (gen/sample (gen/exponential 1/500))
(61.431732383617884 93.51860239112023 557.6183626329674 57.553819177701584 60.4171805107351 7.152938152340775 31.84874217796871 215.71248467032976 470.69904975577214 222.89410284392252)
user=> (gen/sample (gen/exponential 1/500))
(231.68194071961852 74.5153426725424 135.57104449489148 48.2988483633354 566.3041916582175 925.9673009296939 169.65855270732143 914.1156710720605 1445.3554035600935 674.1395344157389)
user=> (gen/sample (gen/exponential 1/500))
(1467.6817487657213 1084.3796561112276 444.15866332358667 491.2566970512585 737.2979791960513 92.05590733338545 341.58867531092477 254.66969942333122 1150.8085459403007 111.21710097838688)




[TCHECK-53] property creating function similar to map as alternative to for-all* Created: 17/Nov/14  Updated: 10/Apr/16

Status: Open
Project: test.check
Component/s: None
Affects Version/s: None
Fix Version/s: None

Type: Enhancement Priority: Minor
Reporter: Steve Miner Assignee: Gary Fredericks
Resolution: Unresolved Votes: 0
Labels: None


 Description   

As an alternative to the `for-all*` syntax, I suggest using a function signature that looks more like `map` with the predicate as the first arg and the generators as the rest of the args.

For example, these forms would all be equivalent:

(property (comp integer? *) gen/int gen/int gen/int) ;; proposed style

(for-all* [gen/int gen/int gen/int] (comp integer? *)) ;; current style



 Comments   
Comment by Steve Miner [ 17/Nov/14 8:20 AM ]

It's easy enough to write for myself. I'll make a patch if you want it.

(defn property 
    ([pred gen] (prop/for-all* [gen] pred))
    ([pred gen1 gen2] (prop/for-all* [gen1 gen2] pred))
    ([pred gen1 gen2 & more] 
        (prop/for-all* (list* gen1 gen2 more) pred)))




[TCHECK-7] Add nested property support Created: 04/Mar/14  Updated: 10/Apr/16

Status: Open
Project: test.check
Component/s: None
Affects Version/s: None
Fix Version/s: None

Type: Enhancement Priority: Minor
Reporter: Jean Niklas L'orange Assignee: Gary Fredericks
Resolution: Unresolved Votes: 2
Labels: None


 Description   

Including functionality to add support for nested properties would be beneficial. An example of how this may work follows:

(defn exists?
  [e coll]
  (some #{e} coll))

(defn remove-elem
  [e coll]
  (remove #{e} coll))

;; this test should fail, and illustrates nested properties
(quick-check (for-all [v (gen/vector gen/int)]
               (for-all [e (elements v)]
                 #(exists? e (remove-elem e v)))))

The requirement for this is support for reifying properties, which is not currently implemented.

Original issue - GitHub #10



 Comments   
Comment by Maciej Jaśkowski [ 14/May/14 4:14 PM ]

What's the status of this issue?
It would help us a lot to have this functionality up and running in the following case: I want to verify my path finding algorithm finds optimal solution in the given graph G, so I generate the graph and a big number of random paths and check that all the random paths are "less optimal" then the one found by the algorithm.

Having nested for-all that would be simple. Is there a way around?

Comment by Reid Draper [ 14/May/14 4:24 PM ]

Sorry for the lack of update here. Nested properties are still in a bit of hammock-time on my part. The good news is they are merely convenient, and you can achieve the exact same effect with clojure.test.check.generators/bind. You can see an example of bind being used here.

Comment by Maciej Jaśkowski [ 15/May/14 12:08 AM ]

Apparently I need some more hints because all I can generate (using gen/bing, btw) are pairs '(graph list-of-random-paths)

That is useful as long as the algorithm works as expected. However, once it fails it's pretty hard to figure out which of random-paths failed it.

Comment by Reid Draper [ 15/May/14 1:02 PM ]

Can you show me how you would write the test with nested properties, and then perhaps I can help you rewrite it to use bind instead?

Comment by Maciej Jaśkowski [ 15/May/14 3:58 PM ]

Great! Thank you!

Please, find the most important part below.

(defn gen-matrix-and-paths [size]
    (gen/bind
      (gen-matrix gen/s-pos-int size)
      (fn [matrix]
        (let [ nodes (into [] (range (count matrix)))
               gen-perms (gen-permutations nodes)
               perms (distinct (gen/sample gen-perms 500)) ]
          (gen/tuple 
            (gen/return matrix) 
            (gen/return perms))))))

gen-matrix generates a square matrix of incidence e.g. [[0 1] [4 0]]
gen-permutations generates a permutation of provided vector

Comment by Reid Draper [ 15/May/14 4:41 PM ]

I'm confused. What issue are you running into? The code you've pasted uses bind, and not a nested property.

Comment by Maciej Jaśkowski [ 17/May/14 3:26 AM ]

Ok but now I can only write something like this:

(prop/for-all [data (gen-matrix-and-paths 6)]
   (let [ [matrix random-paths] data ]
     (not-worse-then-others? tsp matrix random-paths))))

which, if my 'tsp' algorithm is not working correctly would point me to a matrix and a vector (length 500) of paths one of which is counterexample. It would be better if instead I got the matrix and a single counterexample.

Comment by Reid Draper [ 17/May/14 12:44 PM ]

I'm still not sure what this would look like using a nested-property. Can you please write the code the way you'd like to see it as a nested property? This will involve at least two for-alls.

Comment by Maciej Jaśkowski [ 18/May/14 4:56 PM ]

Sure!

I would like to write something like that:

(prop/for-all [matrix (gen-matrix 6)]
  (prop/for-all [random-path (gen-random-path matrix)]
    (<= (cost (tsp matrix)) (cost random-path))))
Comment by Maciej Jaśkowski [ 23/May/14 6:22 AM ]

ping!

Comment by Reid Draper [ 26/May/14 11:47 AM ]

Ok, let's translate that to using gen/bind. Just like with nested properties, bind allows you to create generators that depend on the value of another generator. For example, your gen-random-path depends on the previously generated matrix variable. So let's write a generator that returns both a matrix and a random-path.

(def matrix-and-path
  "Return a two-tuple of [matrix random-path]"
  (gen/bind (gen-matrix 6)
            (fn [matrix]
               (gen/tuple (gen/return matrix) (gen-random-path matrix)))))

(prop/for-all [[matrix random-path] matrix-and-path]
  (<= (cost (tsp matrix)) (cost random-path))




[TCHECK-41] Add helpers for deprecating generators Created: 01/Sep/14  Updated: 10/Apr/16

Status: Open
Project: test.check
Component/s: None
Affects Version/s: None
Fix Version/s: None

Type: Enhancement Priority: Minor
Reporter: John Walker Assignee: Gary Fredericks
Resolution: Unresolved Votes: 0
Labels: None

Attachments: File deprecation-helpers.diff    
Patch: Code

 Description   

It should be clearly communicated to the user when generators are deprecated. This patch supplies helpers for easily deprecating generators.

def-throwing-generator receives the metadata that its generator will throw if its called. As an example, suppose deprecated-generator was deprecated in 0.5.9.

(def-throwing-generator deprecated-generator
  "deprecated-generator has been deprecated. Use current-generator instead."
  {:deprecated "0.5.9"})

Then if the user looked up its documentation, they would see from cider:

clojure.test.check.generators/deprecated-generator
Deprecated in 0.5.9
  deprecated-generator has been deprecated. Use current-generator instead.

or if they used the generator and ran cider-test, they would see:

Test Summary
foobar.core-test

Ran 1 tests, in 1 test functions
1 errors


Results

Error in a-test
FIXME, I fail.
expected: (= "a" (sample deprecated-generator 5))
  actual: clojure.lang.ExceptionInfo: deprecated-generator has been deprecated. Use current-generator instead. {:deprecated "0.5.9"}





[TCHECK-19] Permit a data-structure containing generators to be used as a generator Created: 11/Apr/14  Updated: 10/Apr/16

Status: Open
Project: test.check
Component/s: None
Affects Version/s: None
Fix Version/s: None

Type: Enhancement Priority: Minor
Reporter: Darrick Wiebe Assignee: Gary Fredericks
Resolution: Unresolved Votes: 0
Labels: None

Attachments: File add_record.diff    

 Description   

I'm working through some of the examples in John Hughes' "QuickCheck Testing for Fun and Profit" paper. I'd like the structures I generate in those tests to look like the this:

[:apply `reg [(gen-name) (gen/elements pids)]]

Rather than the way they must be expressed now:

(gen/tuple (gen/return :apply) (gen/return `reg) (gen/tuple (gen-name) (gen/elements pids)))

I think a simple recursive function could be used to generate these, with the caveat that currently generators create a map {:gen #<fn...>}, which I'd propose to change to a record so that we could distinguish between a generator and a map data-structure.

This seems to be implemented to good effect in Quvig QuickCheck already:

In general, Quviq QuickCheck permits any data-structure containing embedded generators to be used as a generator for data-structures of that shape—something which is very convenient for users, but quite impossible in Haskell, where embedding a generator in a data-structure would normally result in a type error. This technique is used throughout the generators written at Ericsson. – Hughes



 Comments   
Comment by Darrick Wiebe [ 11/Apr/14 5:06 PM ]

Playing around in the REPL today, I came up with this code that seems to work well converting arbitrary literals into generators:

(defprotocol LiteralGenerator
  (-literal->generator [literal]))

(defn literal->generator [literal]
  (cond
    (satisfies? LiteralGenerator literal) (-literal->generator literal)
    (vector? literal) (apply gen/tuple (mapv literal->generator literal))
    (and (map? literal) (= [:gen] (keys literal)) (fn? (:gen literal))) literal
    (map? literal) (gen/fmap (partial into {}) (literal->generator (mapv vec literal)))
    (set? literal) (gen/fmap set (literal->generator (vec literal)))
    (list? literal) (gen/fmap (partial apply list) (literal->generator (vec literal)))
    :else (gen/return literal)))

Generating from a record is probably something that would be generally useful, so I added this as well:

(defn record->generator
  ([record]
   ; Is there a better way to do this?
   (let [ctor (eval (read-string (str "map->" (last (clojure.string/split (str (class record)) #"\.")))))]
     (record->generator ctor record)))
  ([ctor record]
   (gen/fmap ctor (literal->generator (into {} record)))))

Which enables me to extend a record like this:

(defrecord Foo [a b]
  LiteralGenerator
  (-literal->generator [this] (record->generator map->AbcDef this)))

I haven't looked at the possibility of baking this code into test.check at all yet. I'd like feedback on what I've got so far before pursuing this any further.

Comment by Reid Draper [ 11/Apr/14 5:31 PM ]

So, I have kind of mixed feelings about this. I agree it's convenient, but I also worry that being loose like that can allow more accidents to occur, and doesn't force users to make the distinction between values and generators. For example, what if you do something like this: [int int]. Did you mean to type [gen/int gen/int], or do you really intend to have written the equivalent of [(gen/return int) (gen/return int)]? If every function that expects a generator can also take a value, we can no longer start adding error-checking to make sure you're correctly passing generators. On that same token, I do see the benefit of being able to use data-structure literals. What if we wrote a function that you had to use to create a generator with this syntax, something like: {{(gen/literal [:age gen/int])}}. That way you could opt-in to this syntax, but only within the scope of gen/literal?

Comment by Darrick Wiebe [ 11/Apr/14 5:40 PM ]

I agree that is a concern, and since you're not guaranteed to see generator output, it might be better to be explicit about using gen/literal. It's still a nice clean solution. Fortunately, that's exactly what I've written! We'd just need to rename literal->generator to literal and install it into the clojure.test.check.generators namespace.

Comment by Reid Draper [ 11/Apr/14 6:56 PM ]

I think the first step is changing generators to use Records. Interested in making a patch for that?

Comment by Darrick Wiebe [ 11/Apr/14 8:31 PM ]

A very simple patch to use a Generator record rather than a simple {:gen ƒ) map.

Comment by Reid Draper [ 12/Apr/14 6:15 PM ]

I've applied the record-patch in ef132b5f85a07879f01417c9104aa6dea771fdb4. Thanks. I've also added a generator? helper function.

Comment by Philip Potter [ 22/Jun/14 4:39 PM ]

I spotted something which seemed relevant: ztellman/collection-check defines a tuple* fn which is like gen/tuple but automatically wraps literals with gen/return.

Notably, despite not solving the general case, this would have solved the example in the issue description.

Comment by Gary Fredericks [ 20/Aug/15 10:28 PM ]

There is now a library aimed at this: https://github.com/holguinj/jen





[TCHECK-2] pos-int is confusingly named Created: 04/Mar/14  Updated: 10/Apr/16

Status: Open
Project: test.check
Component/s: None
Affects Version/s: None
Fix Version/s: None

Type: Enhancement Priority: Minor
Reporter: Jean Niklas L'orange Assignee: Gary Fredericks
Resolution: Unresolved Votes: 2
Labels: None


 Description   

The generators neg-int and pos-int generate 0, which is confusing as (= (pos? 0} (neg? 0) false). The suggestion is to change s-pos-int and s-neg-int to pos-int and neg-int respectively, and replace the current neg-int and pos-int. Suggestions for new names include non-x-int and not-x-int.

Original issue






[TCHECK-9] Properly document how :max-size works Created: 04/Mar/14  Updated: 10/Apr/16

Status: Open
Project: test.check
Component/s: None
Affects Version/s: None
Fix Version/s: None

Type: Enhancement Priority: Minor
Reporter: Jean Niklas L'orange Assignee: Gary Fredericks
Resolution: Unresolved Votes: 0
Labels: release-0.10.0


 Description   

I have had several issues where I thought increasing quick-check's num-tests argument would increase the time taken linearly, but surprisingly it grew quadratically. This is related to the auto-increasing of the size to call-gen, which I believe it isn't documented well enough. It also seems like other people have had trouble with this functionality, see the reply to announcement of clojure.test.check.

Preferably, it should be mentioned in the docstring and the tutorial that the size of the structures generated grows linearly with the amount of tests you have, up to 200. As a result, doubling the amount of tests does not necessarily double the amount of time taken to run the tests.

In some of my tests, this behaviour increases the running time exponentially, and I actually used quite some time figuring out why. The documentation on recursive generators decreases the size of the binary tree in half, but it is not explained why it is split in half. While obvious in hindsight, explicitly explaining that recursive structures should have an amount of nodes proportional to the input size argument and how it is done, would be beneficial.



 Comments   
Comment by Reid Draper [ 04/Mar/14 8:50 PM ]

+1. This is not well-documented now. Further, I'm still not 100% sure that halving inside of a recursive generator is the right thing to do. For some structures, log2 may be more appropriate. In the mailing list post you mention, gen/any is clearly not bounded enough at roughly size=90, consider it's using more than 1GB of heap.

Comment by Reid Draper [ 14/Apr/14 8:58 PM ]

I've made things a little better in 57df9b9b8f2fe61a798fbf683046e8d1aa128228.





[TCHECK-26] defspec does not respect (is (thrown? ...)) Created: 20/Apr/14  Updated: 10/Apr/16

Status: Open
Project: test.check
Component/s: None
Affects Version/s: None
Fix Version/s: None

Type: Defect Priority: Minor
Reporter: Sung Pae Assignee: Gary Fredericks
Resolution: Unresolved Votes: 0
Labels: release-0.10.0


 Description   

The following test does not pass:

(quick-check 1
  (for-all [x g/any]
    (is (thrown? Throwable (assert false)))))

This can be worked around by wrapping the (is ...) form with a call to boolean:

(quick-check 1
  (for-all [x g/any]
    (boolean (is (thrown? Throwable (assert false))))))

The (is (thrown? ...)) form does not return true|false, but will return the Exception itself when it passes.

Thank you!



 Comments   
Comment by Reid Draper [ 13/May/14 6:14 PM ]

This is correct. And I'm not yet sure what the best way forward is, but my hunch is that we'll need to support non-boolean return values from tests. Maybe something that implements a ->pass? function.





[TCHECK-14] re-organize README and doc/ Created: 29/Mar/14  Updated: 10/Apr/16

Status: Open
Project: test.check
Component/s: None
Affects Version/s: None
Fix Version/s: None

Type: Enhancement Priority: Minor
Reporter: Reid Draper Assignee: Gary Fredericks
Resolution: Unresolved Votes: 0
Labels: release-0.10.0


 Description   

I'd like to re-organize the README, and focus on making it easier for users to find the information they need. The README is currently pretty long, and could be information overload. My current thinking is that the new README should be short, and focus on providing links to people based on their experience with test.check. I think something like the following three headings would be useful:

1. First time user. Show me what this is all about.

  • Focus on 'seeing something cool' within five minutes. Maybe include some example tests in examples/. This way new users can simply clone the repository and run a test in under two minutes.

2. Still getting started, show me some tutorials and examples.

  • This will probably build on the existing doc/intro.md file, which currently describes how to write generators.

3. Existing user, show me API documentation, release notes, advanced techniques.

  • API documentation
  • Release-notes, make it clear these are up-to-date, and have clear instructions when there are breaking changes
  • Advanced techniques/tutorials





[TCHECK-8] Bound tests and shrinks in time Created: 04/Mar/14  Updated: 10/Apr/16

Status: Open
Project: test.check
Component/s: None
Affects Version/s: None
Fix Version/s: None

Type: Enhancement Priority: Minor
Reporter: Jean Niklas L'orange Assignee: Gary Fredericks
Resolution: Unresolved Votes: 1
Labels: release-0.10.0


 Description   

It should be possible to bound tests and shrinks in time.

The original issue also mentions shrink interruption through Ctrl-c, but this is already covered in #TCHECK-4.

Original issue - #9 GitHub



 Comments   
Comment by Gary Fredericks [ 10/Apr/16 5:58 PM ]

Idea: a new function in c.t.check that takes a collection of arglists for c.t.check/quick-check, as well as a max-time, and runs the test in an interleaving fashion.

Not sure how c.t.c.clojure-test integration would look.





[TCHECK-89] the clojure-test namespace has a global atom Created: 23/Feb/16  Updated: 14/Apr/16

Status: Open
Project: test.check
Component/s: None
Affects Version/s: None
Fix Version/s: None

Type: Defect Priority: Minor
Reporter: Gary Fredericks Assignee: Unassigned
Resolution: Unresolved Votes: 0
Labels: release-0.10.0

Attachments: Text File remove_global_atom.patch    

 Description   

There's a global atom in the clojure-test namespace, for tracking something or other across calls to the report functions I think.

This might be problematic for an implementation of parallel tests. I'm not sure, I think it would depend on the details of that implementation.



 Comments   
Comment by Nicolás Berger [ 05/Apr/16 10:24 PM ]

The atom is there as part of the trial-report-periodic implementation, it's not used for anything else. I think we could reimplement trial-report-function under the new reporter-fn mechanism, with a function that encloses the atom, thus avoiding having a global atom.

Comment by Unnikrishnan Geethagovindan [ 12/Apr/16 2:16 AM ]

The atom being set when `:being-test-var` is called and used in the `trial-report-periodic` ? Wouldn't the atom have to be global for it to work ?

This might be a silly question (considering I'm fairly new to clojure), How can this be implemented without the atom being global ? If someone can point me in the right direction, I can try to fix this.

Comment by Nicolás Berger [ 12/Apr/16 7:30 AM ]

The atom being set when `:being-test-var` is called and used in the `trial-report-periodic` ? Wouldn't the atom have to be global for it to work ?

You are totally right: to reset the atom in :begin-test-var the atom needs to be global. But if we find how to reset the atom in a way that it doesn't need to be global, we could make it non-global . I'm thinking of something like:

(defn make-periodic-reporter-fn
  ([] (make-periodic-reporter-fn *trial-report-period*))
  ([report-period] (make-periodic-reporter-fn report-period default-reporter-fn))
  ([report-period reporter]
   (let [last-trial-report (atom 0)]
     (fn [m]
       (case (:type m)
         :started
         (reset! last-trial-report (get-current-time-millis))

         :trial
         (let [t (get-current-time-millis)]
           (when (> (- t report-period) @last-trial-report)
             (with-test-out*
               (fn []
                 (println "Passing trial"
                          (-> m ::trial first) "/" (-> m ::trial second)
                          "for" (get-property-name m))))
             (reset! last-trial-report t)))
         ;; else
         (reporter m))))))

We would also need to add the following at the beginning of c.t.c/quick-check:

(reporter-fn {:type :started
              :property property
              :num-tests num-tests})

This would allow us to use it as (defspec some-spec {:reporter-fn (make-periodic-reporter-fn)} some-prop)

To make it usable via the *report-trials* mechanism (so it is backwards-compatible), we could do something like:

(def trial-report-periodic
  ;; I'm using (constantly nil) as the underlying reporter-fn because we only need to support the :type ::trial
  ;; message. The other messages are already handled by default-reporter-fn, which is going to call us on :type ::trial
  (make-periodic-reporter-fn *trial-report-period* (constantly nil)))
Comment by Unnikrishnan Geethagovindan [ 12/Apr/16 1:01 PM ]

Yea, this works, with a minor change in println

(:so-far m) "/" (:num-tests m)

Should I submit a patch with these following changes incorporated along with corresponding change in tests ?

Comment by Nicolás Berger [ 12/Apr/16 1:12 PM ]

Thanks for the fix in the println . Adding a patch sounds good to me, but of course we'll have to hear what @gfredericks thinks about it.

Comment by Unnikrishnan Geethagovindan [ 14/Apr/16 1:29 AM ]

Just realised that we will need to add :type ::started to the default-reporter-fn so that it is backward-compatible !

Comment by Unnikrishnan Geethagovindan [ 14/Apr/16 1:30 AM ]

Attaching patch





[TCHECK-110] Enable shrinking on recursive generators Created: 22/Jun/16  Updated: 22/Jun/16

Status: Open
Project: test.check
Component/s: None
Affects Version/s: None
Fix Version/s: None

Type: Enhancement Priority: Minor
Reporter: Timothy Baldridge Assignee: Gary Fredericks
Resolution: Unresolved Votes: 0
Labels: None


 Description   

I wrote a AST generator for a search engine using test.check. It works quite well, but for some reason recursive generators don't shrink "vertically", instead they only shrink vertically. For exmaple, let's say this AST failed:

{:op :and
 :args [{:op :and
         :args [{:op :term ...}
                {:op :term ...}
                {:op :term ...}]}]}

After Shrinking

{:op :and
 :args [{:op :and
         :args [{:op :term ...}]}]}

The problem is in `{:op :term}` but for some reason test.check doesn't try to remove the recursive `:and` nodes. Not a big deal, but would be a nice feature to have.



 Comments   
Comment by Gary Fredericks [ 22/Jun/16 7:12 PM ]

Does this use gen/recursive-gen?

Comment by Timothy Baldridge [ 22/Jun/16 7:17 PM ]

Yes, I'm using recursive-gen.

Comment by Gary Fredericks [ 22/Jun/16 7:25 PM ]

So this looks like the same issue?

(quick-check 10000
             (prop/for-all [tree (gen/recursive-gen gen/vector gen/large-integer)]
               (->> tree
                    (tree-seq coll? seq)
                    (not-any? #{4242}))))
;; =>
{:result false,
 :seed 1466641044276,
 :failing-size 151,
 :num-tests 3152,
 :fail [[[-250312923371676 -2634403398808308]
         [-134580]
         1190117809715
         [1736827773692]
         [91379147228 281572852]
         []
         []
         [264322377680727]
         [-2 -2005122340306]
         []
         []
         [2133414023]
         []
         [-7203148411369 2093087]
         [-1 -1]
         [-350804570003194 -24726]
         [-2145238760990835556]
         [-4410884650149229158 27914810]
         []
         [21126727]
         [816412492 102]
         [1]
         [-119 -2132126120873]
         [50]
         [1590594626470485464 -555554916273244]
         [4242 322325]]],
 :shrunk {:total-nodes-visited 57,
          :depth 11,
          :result false,
          ;; should have shrunk to `4242`
          :smallest [[[4242]]]}}
Comment by Gary Fredericks [ 22/Jun/16 7:36 PM ]

I'm not 100% sure there's a clean solution to this, but note to self: shrinking to sub-trees should be the first thing we try

Comment by Gary Fredericks [ 22/Jun/16 8:58 PM ]

If TCHECK-112 ends up being figured out it might apply here too.





[TCHECK-58] Alternate clojure.test integration Created: 05/Jan/15  Updated: 06/Aug/16

Status: Open
Project: test.check
Component/s: None
Affects Version/s: None
Fix Version/s: None

Type: Enhancement Priority: Minor
Reporter: Colin Williams Assignee: Gary Fredericks
Resolution: Unresolved Votes: 0
Labels: None


 Description   

I've got a macro that integrates with clojure.test at a different level than defspec, which I've found useful in migrating tests from non-generative to generative:

(deftest something-important
  (checking "something really important" [x gen/int]
    (is (= (important-calculation x) (inc x)))))

I've found it natural to reach the above from something like this:

(deftest something-important
  (testing "something really important"
    (is (= (important-calculation 1) 2))))

There are still a lot of rough edges, but I wanted to get peoples reactions before I started polishing it for a patch.



 Comments   
Comment by Reid Draper [ 20/Jan/15 10:36 PM ]

Thanks for the comment Colin. At the moment, I'd like to keep the integration simple (meaning there is one way to do things), and keep 'alternative syntax' macros and functions out of test.check itself. Gary's test.chuck might be a good home for this.





[TCHECK-4] Handle sigint Created: 04/Mar/14  Updated: 06/Aug/16

Status: Open
Project: test.check
Component/s: None
Affects Version/s: None
Fix Version/s: None

Type: Enhancement Priority: Minor
Reporter: Jean Niklas L'orange Assignee: Gary Fredericks
Resolution: Unresolved Votes: 0
Labels: None


 Description   

During shrinking, this should simply return the best shrink so-far. During testing, perhaps it should just return the number of tests that've been run. In both cases, some indication should be given in the return value that the user pressed ctrl-c.

Original issue - #41 on GitHub






[TCHECK-48] Add sublist generator Created: 28/Oct/14  Updated: 06/Aug/16

Status: Open
Project: test.check
Component/s: None
Affects Version/s: None
Fix Version/s: None

Type: Enhancement Priority: Minor
Reporter: Reid Draper Assignee: Gary Fredericks
Resolution: Unresolved Votes: 0
Labels: release-0.10.0


 Comments   
Comment by Gary Fredericks [ 21/May/15 9:40 PM ]

Related (but probably different): https://github.com/gfredericks/test.chuck/blob/20f17550acecdfed4ca3259ebdad34e088fd453a/src/com/gfredericks/test/chuck/generators.clj#L141-150





[TCHECK-15] gen/for macro for alternate combinator syntax Created: 02/Apr/14  Updated: 10/Apr/16

Status: Open
Project: test.check
Component/s: None
Affects Version/s: None
Fix Version/s: None

Type: Enhancement Priority: Minor
Reporter: Gary Fredericks Assignee: Gary Fredericks
Resolution: Unresolved Votes: 3
Labels: None

Attachments: Text File TCHECK-15-p1.patch     Text File TCHECK-15-p2.patch     Text File TCHECK-15-p3.patch     Text File TCHECK-15-p4.patch     Text File TCHECK-15.patch    

 Description   

I think the syntax of clojure.core/for would be a good fit for test.check's combinators. For example:

(defn gen-even-subset
  "Returns a generator that generates an even-cardinality
   subset of the given elements"
  [elements]
  (gen/for [bools (apply gen/tuple (repeat (count elements) gen/boolean))
            :let [true-count (->> bools (filter identity) (count))]
            :when (even? true-count)]
    (->> (map list bools elements)
         (filter first)
         (map second)
         (set))))

This combines the capabilities of fmap, bind, and such-that into a familiar syntax.

One downside here is the temptation to use multiple clauses for independent generators, resulting in a use of gen/bind when gen/tuple would be simpler and presumably shrink easier. An approach to this is an additional supported clause, perhaps called :parallel, that uses the syntax of :let to provide the functionality of gen/tuple:

(gen/for [:parallel [n1 gen/nat
                     n2 gen/nat]
          :let [sum (+ n1 n2)]]
  {:nums [n1 n2] :sum sum})

Compared to gen/tuple, this has the advantage of placing generators syntactically next to names, rather than segregating the generators from the names.

The :parallel feature has not been added to the current patches.



 Comments   
Comment by Gary Fredericks [ 05/Apr/14 3:23 PM ]

I think there might be some design ambiguity around the meaning of :when. In particular, in the following contrived example:

(for [n nat
      v (vec (return n))
      :let [sum (reduce + v)]
      :when (pos? sum)]
  v)

In my default design this can hang, for the same reason that this code can hang:

(bind nat 
      (fn [n]
        (such-that
          (fn [v] (pos? (reduce + v)))
          (vector (return n)))))

But it could just as well have been written:

(such-that 
  (fn [v] (pos? (reduce + v)))
  (bind nat (fn [n] (vector (return n)))))

So the issue is whether a :when filter is applied to just the previous generator or to all of the previous generators. I have some hazy notion that the latter would be less efficient in some cases, but I'm not sure what. So I think our options are:

  1. Decide to always do it one way or the other
  2. Provide a third keyword (:when-all?) with different behavior
  3. Don't write this macro at all because it's too difficult to understand

My gut is to do option 1 and just apply :when to the previous generator.

Comment by Gary Fredericks [ 08/Apr/14 9:24 PM ]

Attached my initial draft. The implementation took a lot more thought than I expected, and is a bit subtle, so I included some inline comments explaining the structure of the macro.

Comment by Gary Fredericks [ 13/Apr/14 8:33 PM ]

Attached TCHECK-15-p1.patch, updated to apply to the current master.

Comment by Gary Fredericks [ 16/Apr/14 9:51 PM ]

Attached TCHECK-15-p2.patch which adds a note to the docstring about independent clauses, shrinking, and tuple.

Comment by Gary Fredericks [ 16/Apr/14 9:58 PM ]

Attached TCHECK-15-p3.patch which fixes one bug and one redundancy in namespace aliases.

Comment by Gary Fredericks [ 13/May/14 10:37 AM ]

Attached TCHECK-15-p4.patch which fixes a bug with destructuring (and adds a regression test).

Comment by Gary Fredericks [ 13/May/14 10:38 AM ]

Also might be helpful to note that I've put this in my test.check utility library for now: https://github.com/fredericksgary/test.chuck#for.

Comment by Michael Blume [ 08/Jan/15 12:21 AM ]

I wonder if it'd be possible to avoid :parallel by analyzing the code and checking whether the bindings can be run in parallel?

Comment by Gary Fredericks [ 08/Jan/15 9:16 AM ]

I think it's possible in theory, but we'd need access to a non-buggy code walker.

Additionally you might argue that it makes the meaning of the code a lot more subtle.





[TCHECK-116] With clojure.test, debug prn should be optional, so you can run tests with minimal output Created: 18/Aug/16  Updated: 25/Aug/16

Status: Open
Project: test.check
Component/s: None
Affects Version/s: None
Fix Version/s: None

Type: Enhancement Priority: Minor
Reporter: Brent Spell Assignee: Gary Fredericks
Resolution: Unresolved Votes: 3
Labels: release-0.10.0
Environment:

osx el capitan, leiningen 2.6.1, clojure 1.8, test.check 0.9.0



 Description   

There is an unconditional prn in assert-check (https://github.com/clojure/test.check/blob/master/src/main/clojure/clojure/test/check/clojure_test.cljc#L19) that outputs the test configuration (seed, etc.) every time a defspec is used. For projects with many property tests/defspecs, this results in a large test output when tests are successful.

For tests that pass, it seems like this output is unnecessary, since failing tests always report the seed, etc. Currently, the only obvious way to suppress this message is to monkey-patch assert-check and disable prn. It would be nice to have an option to disable these messages through configuration for successful runs.



 Comments   
Comment by Gary Fredericks [ 18/Aug/16 6:54 PM ]

Agreed – is there any cleaner mechanism for this than a dedicated global var?

Perhaps a general callback function would be better than a flag for printing. E.g.,

(def ^:dynamic *output-fn* prn)

On second thought, the master branch has a :reporter-fn option to quick-check, and the clojure-spec namespace has a default reporter fn that could be overrideable. If the printing were folded into that, then it could be disabled that way, though that's rather more roundabout for the user :/.

Comment by Brent Spell [ 25/Aug/16 8:11 AM ]

Both of those solutions sound like a reasonable burden on the user to me, and they would be more robust than having to monkey-patch the assert-check function.





[TCHECK-121] for-all's signature suggests multiple expressions are allowed, but this doesn't work as you'd expect Created: 04/Sep/16  Updated: 10/Sep/16

Status: Open
Project: test.check
Component/s: None
Affects Version/s: None
Fix Version/s: None

Type: Enhancement Priority: Minor
Reporter: lvh Assignee: Gary Fredericks
Resolution: Unresolved Votes: 0
Labels: release-0.10.0


 Description   

The signature of for-all, which includes & body, might suggest you could write something like:

(defspec empty-schema-validation
  (prop/for-all
   [v (s/gen ::js/json-type)]
   (nil? (explain-with-schema {} v))
   (nil? (explain-with-spec {} v))))

That works, but IIUC not as you might expect; in that only the latter expr is checked. I'm not sure if the signature should be amended, or if for-all should just create a fn for each of those exprs. The latter is more work, but seems more useful.



 Comments   
Comment by Gary Fredericks [ 10/Sep/16 5:54 PM ]

I appreciate the confusion, and I think maybe keeping it to one expression would have been a better idea, but at this point it can't be changed in a backwards-compatible way.

I'm planning to overhaul the docstrings before the next release though, so I will make sure to clarify this.





Generated at Sun Sep 25 03:59:03 CDT 2016 using JIRA 4.4#649-r158309.