<< Back to previous view

[TCHECK-21] Rerunning a particular failure is difficult. Created: 11/Apr/14  Updated: 12/Apr/14

Status: Open
Project: test.check
Component/s: None
Affects Version/s: None
Fix Version/s: None

Type: Enhancement Priority: Major
Reporter: Gary Fredericks Assignee: Reid Draper
Resolution: Unresolved Votes: 0
Labels: None


 Description   

Test.check has the concept of a seed that is used when a property is tested. The seed can be supplied or generated by the current time. Either way, the single seed is used to generate all the trials for the test. If a test fails, the only way to try to re-test the property on the same inputs is to run all of the tests again using the same seed. If the tests were running for hours before a failure was found, this is prohibitive.

I would like instead to be able to check the property for exactly the input that failed. One step in that direction is to have an explicit different seed for each trial (perhaps generated from an initial metaseed), and on failure to report a [seed size] pair that can be used to run a test a single time.

The way this interacts with shrinking is interesting though, since it's just as likely that I'll want to re-run on the shrunk value from a failure. Since the shrunk value was never explicitly generated from a known [seed size], just that information is insufficient. We could perhaps report [seed size shrink-path] where the third element is a list of indexes to walk through the shrink tree with. Probably the worst part about that is it might be rather long.

In any case I think something like this would be more useful than the current setup. I'll probably be hacking up something similar for a fork branch I'm maintaining, but if we can settle on a specific design I'd be happy to cook up a patch.



 Comments   
Comment by Reid Draper [ 12/Apr/14 10:28 AM ]

When a test fails, we do return the arguments that originally failed, and the arguments of the shrunk test. Is this insufficient?

For example:

{:result false,
       :failing-size 45,
       :num-tests 46,
       :fail [[10 1 28 40 11 -33 42 -42 39 -13 13 -44 -36 11 27 -42 4 21 -39]],
       :shrunk {:total-nodes-visited 38,
                :depth 18,
                :result false,
                :smallest [[42]]}}

see :fail and [:shrunk :smallest].

Comment by Gary Fredericks [ 12/Apr/14 11:00 AM ]

It's certainly adequate to reproduce in theory, but I don't know of any built in mechanism that makes this easy. Here's what I end up doing:

Given this:

(defspec foo
  (prop/for-all [x gen]
    (f x)))

and after getting a failure with a 50-line shrunk value (e.g. :bar), I manually pretty-print it and paste it back into my file like so:

(def shrank
  ':bar)

(defspec foo
  (prop/for-all [x #_gen (gen/return shrank)]
    (f x)))

And now I can run (foo 1) to re-run with the failing value.

This awkwardness is partly due to three facts:

  1. My generators create large unwieldy values
  2. My property is a large body of code rather than a single function
  3. There's no easy way to test a property on a specific value

I could mitigate some of the pain by fixing #2 – having each one of my tests split into a defspec / prop/for-all and a plain function. But #1 is not easy to fix, and having a small tuple is rather nicer for me.

I've already written a POC for this and it's working rather well. I used the term :key to refer to the tuple, so the term doesn't conflict with :seed. I end up calling the test like (foo 0 :key [329489249329323 19 []]), but I'll probably think up something else that doesn't require passing a dummy first arg.





[TCHECK-32] Default sizing on gen/any needs re-evaluation Created: 13/Jun/14  Updated: 04/Aug/14

Status: Open
Project: test.check
Component/s: None
Affects Version/s: None
Fix Version/s: None

Type: Defect Priority: Major
Reporter: Luke VanderHart Assignee: Reid Draper
Resolution: Unresolved Votes: 0
Labels: None


 Description   

The following innocuous-looking test blows the heap and then crashes a 4GB JVM with an out-of-memory error:

(tc/defspec merge-is-idempotent
  100
  (prop/for-all [m (gen/map gen/any gen/any)]
    (= m (merge m m))))

I understand how this happens, and how to fix it by adjusting the size parameters.

However, it would be great if using the defaults did not have the potential for such a nasty failure mode (particularly as the unwary user will have trouble determining that the fault is not in their code).



 Comments   
Comment by Philip Potter [ 21/Jun/14 8:29 AM ]

I was going to raise a separate ticket for this, but it seems like it might be related: gen/any (and any-printable) seem to take exponential time in the size of the input. See for example this session:

user> (time (dorun (gen/sample (gen/resize 50 gen/any-printable))))
"Elapsed time: 2204.284643 msecs"
nil
user> (time (dorun (gen/sample (gen/resize 55 gen/any-printable))))
"Elapsed time: 2620.717337 msecs"
nil
user> (time (dorun (gen/sample (gen/resize 60 gen/any-printable))))
"Elapsed time: 5923.636336 msecs"
nil
user> (time (dorun (gen/sample (gen/resize 65 gen/any-printable))))
"Elapsed time: 9035.762191 msecs"
nil
user> (time (dorun (gen/sample (gen/resize 70 gen/any-printable))))
"Elapsed time: 15393.687184 msecs"
nil
user> (time (dorun (gen/sample (gen/resize 75 gen/any-printable))))
"Elapsed time: 9510.571668 msecs"
nil
user> (time (dorun (gen/sample (gen/resize 80 gen/any-printable))))
"Elapsed time: 39591.543565 msecs"
nil

Apart from the anomaly at 75, adding 10 to the size seems to increase runtime by almost 3x.

Comment by Reid Draper [ 23/Jun/14 5:07 PM ]

I believe I should have a fix for this today, as well as an easier way to write custom, recursive generators.

Comment by Reid Draper [ 23/Jun/14 5:21 PM ]

This isn't fixing the OOM yet, but is making a big difference in making the size have a linear relationship with the number of elements generated, in a recursive generator. Here's branch: https://github.com/clojure/test.check/compare/feature;recursive-generator-helpers.

Comment by Reid Draper [ 23/Jun/14 7:16 PM ]

Fixed in https://github.com/clojure/test.check/commit/19ca756c95141af3fb9caa5e053b9d01120e5d7e. I'll try and get a snapshot build up tonight. Will be 0.5.9-SNAPSHOT. Check out the commit-message for a full explanation of how this now works.

Comment by Philip Potter [ 28/Jun/14 3:26 PM ]

awesome work! The new recursive-gen fn is great, too: I can immediately see a use case for generating arbitrary JSON-compatible data (ie vectors, maps, strings, numbers, bools, but no rationals, characters, symbols...).

Comment by Philip Potter [ 29/Jun/14 6:48 AM ]

Just re-tested on my machine; performance is vastly improved though still superlinear:

clojure.test.mode> (time (dorun (gen/sample (gen/resize 100 gen/any-printable))))
"Elapsed time: 101.907628 msecs"
nil
clojure.test.mode> (time (dorun (gen/sample (gen/resize 200 gen/any-printable))))
"Elapsed time: 302.341697 msecs"
nil
clojure.test.mode> (time (dorun (gen/sample (gen/resize 400 gen/any-printable))))
"Elapsed time: 1154.098163 msecs"
nil
clojure.test.mode> (time (dorun (gen/sample (gen/resize 800 gen/any-printable))))
"Elapsed time: 2954.889396 msecs"
nil
clojure.test.mode> (time (dorun (gen/sample (gen/resize 1600 gen/any-printable))))
"Elapsed time: 22335.200578 msecs"
nil

although since the default max-size is 200, this is very much no big deal.

Comment by Reid Draper [ 03/Jul/14 10:52 AM ]

I'm not too surprised that performance is still superlinear. What should be linear is the number of leaf nodes in the generated tree. We may eventually be able to do something a little more complex and have the total number of nodes be linear (including internal nodes). For now, however, it should be considered a bug if the relationship between leaf nodes doesn't grow linearly with the size parameter.

Comment by Reid Draper [ 04/Aug/14 8:35 PM ]

I'm still seeing that the original example is OOMing. I'm not sure the best way to solve that. For a single layer of nested generators, the above patch seems to be making a big difference. But when you make a map of gen/any, map doesn't know that it's arguments are themselves recursive generators. This means you might create 100 keys and values, each themselves, large recursive generators. There's a balance that has to be had between making the default recursive generators be 'large enough' by themselves, but not too large when used like this. Hmm.. I think I'll go ahead and release 0.5.9, and keep on thinking of ways to make this even better.





[TCHECK-34] clojure.test reporting is uninformative Created: 21/Jun/14  Updated: 11/Aug/14

Status: Open
Project: test.check
Component/s: None
Affects Version/s: None
Fix Version/s: None

Type: Defect Priority: Major
Reporter: Philip Potter Assignee: Reid Draper
Resolution: Unresolved Votes: 0
Labels: None


 Description   

The clojure.test reporting results in output like:

FAIL in (always-fail) (clojure_test.clj:18)
expected: result
  actual: false

Note that the file and line number are always clojure_test.clj:18 and the expected value is always the literal string "result".

I'm not sure what the right output would be for expected/actual, but the incorrect file and line reporting means that clojure-test-mode consistently highlights the wrong line.



 Comments   
Comment by Philip Potter [ 21/Jun/14 9:10 AM ]

I think this is because the assertion is checked by calling clojure.test/is. clojure.test/is is being called for its return value, but it has the side effect of reporting failures. It's a macro, and it takes the literal symbol "result" as its expected expression, and takes the current file and line number to attach to the failure. It's really designed to be called by user code, not library code.

Comment by John Walker [ 11/Aug/14 3:55 PM ]

Well, one solution would involve taking advantage of the report functionality.

https://github.com/clojure/clojure/blob/master/src/clj/clojure/test.clj#L204
https://github.com/clojure/clojure/blob/master/src/clj/clojure/test.clj#L371





[TCHECK-38] Generators for functions? Created: 15/Aug/14  Updated: 27/Aug/14

Status: Open
Project: test.check
Component/s: None
Affects Version/s: None
Fix Version/s: None

Type: Enhancement Priority: Major
Reporter: Michael Sperber Assignee: Reid Draper
Resolution: Unresolved Votes: 0
Labels: None


 Description   

test.check has generators for first-order types, but not for functions.

In the original QuickCheck, an additional type class ("Arbitrary") handles
functions.

If I implemented this and submitted a patch, would it be likely to be included?

Background: I wrote my own Clojure QuickCheck clone here:

https://bitbucket.org/sperber/deinprogramm.quickcheck

This has arbitraries and function generators, but not all of test.check's goodies.

I'm trying to decide whether to continue maintaining my QuickCheck clone or
instead to contribute ot test.check, so feedback would be much appreciated.



 Comments   
Comment by Reid Draper [ 21/Aug/14 2:16 PM ]

I'd love a patch for this! Would you like to write up a quick implementation proposal first, so we can iron out any architectural details before looking at the code?

Comment by Michael Sperber [ 25/Aug/14 2:06 AM ]

Would you be willing to look at my QuickCheck clone as a starting point? Its general implementation approach is very similar to yours, so I think you should feel right at home.

The caveat is that it introduces a type separate from generators - the "arbitrary", corresponding to the `Arbitray' type class in the Haskell code. Doing this to `test.check' would change the API, so maybe there, the `Generator' type should be extended by an optional `transformer' field.

Comment by Reid Draper [ 26/Aug/14 2:26 PM ]

Sure, I've already read through it a bit, but need to read in more detail. How does your implementation handle printing (showing) functions? Do functions shrink? Have you seen Showing and Shrinking Functions by Claessen ?

Comment by Michael Sperber [ 27/Aug/14 2:21 AM ]

Yes. Haven't done that yet, but it would be on my list. Given that functions are objects in Clojure, I think printing should be a little easier than in Haskell.





[TCHECK-44] for-all should support nesting Created: 22/Sep/14  Updated: 28/Oct/14

Status: Open
Project: test.check
Component/s: None
Affects Version/s: None
Fix Version/s: None

Type: Enhancement Priority: Major
Reporter: Michael Sperber Assignee: Reid Draper
Resolution: Unresolved Votes: 0
Labels: None

Attachments: File for-alls-nest.diff    
Patch: Code and Test

 Description   

Haskell QuickCheck allows for-all expressions to nest. This is useful when there are dependencies between generated values. test.check should allow this, too.

Currently, nested for-alls always succeed, which is somewhat pernicious.

I've added a patch that implements this.



 Comments   
Comment by Reid Draper [ 24/Sep/14 11:32 AM ]

Thanks Michael. I appreciate the patch, but there's a few design details that could be discussed before we get to code-level detail. As a separate, but related issue, I've been wanting to implement something like your CheckResult type, but as a record with several more fields. These fields would hold things like a plain-text description of any errors found, statistics (ala QuickCheck's collect, classify, etc.). I'd also like to write a protocol that allows basic types like booleans to be turned into this record. This would be analogous to Haskell QuickCheck's Testable Type Class. While this is technically a separate issue, I think it would behoove us to solve it in conjunction with nestable for-alls, particularly since nested for-alls can be simulated by just using bind at the generator level. Does this make sense?

Comment by Michael Sperber [ 25/Sep/14 2:19 AM ]

Absolutely.

I personally would start this patch, and work from there, unless you want to do things fundamentally differently rather than add more stuff.

Either way, how can I help make it happen?

Comment by Reid Draper [ 25/Sep/14 11:10 PM ]

Great question. Let me think on that and get back to you ASAP. I'd also love to make this happen soon.

Comment by Reid Draper [ 21/Oct/14 10:23 PM ]

Sorry for the delay, here's my sketch I've been working with:

diff --git a/src/main/clojure/clojure/test/check/properties.clj b/src/main/clojure/clojure/test/check/properties.clj
index 99b5222..139ae9a 100644
--- a/src/main/clojure/clojure/test/check/properties.clj
+++ b/src/main/clojure/clojure/test/check/properties.clj
@@ -8,13 +8,47 @@
 ;   You must not remove this notice, or any other, from this software.
 
 (ns clojure.test.check.properties
+  (:import clojure.test.check.generators.Generator)
   (:require [clojure.test.check.generators :as gen]))
 
+(defrecord Result [result pass? message stamps])
+
+(defprotocol ToResult
+  (to-result [a]))
+
+(extend java.lang.Object
+  ToResult
+  {:to-result (fn [b]
+               ;; not checking for caught exceptions here
+               (->Result b (not (false? b)) nil nil))})
+
+(extend nil
+  ToResult
+  {:to-result (fn [b]
+               (->Result b false nil nil))})
+
+(extend java.lang.Boolean
+  ToResult
+  {:to-result (fn [b]
+               (->Result b b nil nil))})
+
+(extend Generator
+  ToResult
+  {:to-result identity})
+
+(extend Result
+  ToResult
+  {:to-result identity})
+
+(defn message
+  [m property]
+  (assoc property :message m))
+
 (defn- apply-gen
   [function]
   (fn [args]
-    (let [result (try (apply function args) (catch Throwable t t))]
-      {:result result
+    (let [result (to-result (try (apply function args) (catch Throwable t t)))]
+      {:result (:result result)
        :function function
        :args args})))
 
@@ -29,9 +63,18 @@
   (for-all* [gen/int gen/int] (fn [a b] (>= (+ a b) a)))
   "
   [args function]
-  (gen/fmap
-    (apply-gen function)
-    (apply gen/tuple args)))
+  (gen/bind
+    (apply gen/tuple args)
+    (fn [a]
+      (let [result ((apply-gen function) a)]
+        (cond (gen/generator? result) (gen/fmap (fn [r] (println "foo") (update-in r :args #(conj % a))) result)
+              ;; NOTE: quick note to myself before I leave this code for the night,
+              ;; this :else is getting hit because we're wrapping the result
+              ;; with a {:result ...} map. Should probably do that conditionally.
+              ;; We also need two result types I think, a result to return from
+              ;; a property itself, and a reuslt that tacks the 'args' on top of this.
+              :else (do (println "bar") (gen/return result)))))
+    ))
 
 (defn binding-vars
   [bindings]
Comment by Michael Sperber [ 22/Oct/14 2:00 AM ]

Looks OK. However, it's difficult to see why that would get you more quickly where you said you want to go than my patch ...

Comment by Reid Draper [ 28/Oct/14 10:55 PM ]

Looks OK. However, it's difficult to see why that would get you more quickly where you said you want to go than my patch ...

Fair enough. Part of this it that it was easier for me to write up a sketch. The main things I'm trying to cover when supporting nested generators are making sure:

  1. We also support the upcoming ability to collect and return statistics about a test, ala collect from Haskell QuickCheck
  2. We have a sane way of returning failing tests to a user. Right now, in the :fail and :smallest keys of the returned map, we tell the user the failing arguments. They're always wrapped in at least one vector, since you may use more than one generator using prop/for-all. What do we do with nested properties? How do we distinguish between multiple generators at the 'same level', vs nested properties? Or do we not need to distinguish? Can whatever we decide to do be backward compatible?

Point being, I want to make sure we're not committing ourselves to nested-properties until we have some of those answers, and for me personally, it's easier to try and play with these things together, and see how they will fit together.





[TCHECK-31] Doc improvements: how to separate "size" from "range" Created: 11/Jun/14  Updated: 22/Jan/15

Status: Open
Project: test.check
Component/s: None
Affects Version/s: None
Fix Version/s: None

Type: Enhancement Priority: Major
Reporter: Michael Nygard Assignee: Reid Draper
Resolution: Unresolved Votes: 1
Labels: None


 Description   

Size is discussed mainly in terms of recursive generators. However, it is crucial to understanding other aspects such as vector length and integer ranges.

Some examples like these would be helpful:

  • How to get a small number of samples with a large range.
  • How to get a large number of samples with a small range.


 Comments   
Comment by Reid Draper [ 12/Jun/14 9:13 AM ]

Agree. I'll try and get a patch up in the next couple of days. Do you just have something in mind going in the doc/ dir?

Comment by Brian Craft [ 12/Nov/14 12:05 AM ]

I have exactly this problem. Are there any examples from projects using test.check?

Comment by Reid Draper [ 12/Nov/14 9:12 AM ]

Brian, can you be more specific about your problem? Some helpful functions regarding sizing and domain-generation are documented here. Take a look specifically at gen/choose, gen/resize, and gen/sized.

Comment by Brian Craft [ 12/Nov/14 10:29 AM ]

I don't need to test, for example, strings that are 500 chars long, but do need to test lists of 500 strings. If I have a generator that builds lists of strings & size it larger, both the lists and strings get larger. Then I start hitting irrelevant resource limits, like db field widths, or jvm memory. It seems like I need a way to specify how different data elements grow. Just setting the range, like (gen/vector ... 0 5000), seems wrong, because then it always picks up to 5000, while I suspect for shrinking it needs to respond to "size". Something like this?

(gen/sized (fn [size] (gen/resize (* size size) (gen/vector (gen/resize size gen/int)))))

forcing the size of the vector to increase more quickly than the size of the elements in the vector.

I gave this a try, but hit the vector stack overflow problem. I will try the fmap/flatten workaround, but am wondering if this is the right approach. Also wondering about the range of "size". From the code it looks like it goes to 100 due to sample-seq. So functions of size passed to gen/sized should expect range from 0-99, and return the largest meaningful data when given size parameter 99?

Comment by Reid Draper [ 15/Nov/14 12:03 PM ]

I don't need to test, for example, strings that are 500 chars long, but do need to test lists of 500 strings. If I have a generator that builds lists of strings & size it larger, both the lists and strings get larger.

Indeed, by default, all generators will 'grow' together. And as you've correctly noticed, you can use gen/sized and gen/resize to further control this.

Just setting the range, like (gen/vector ... 0 5000), seems wrong, because then it always picks up to 5000, while I suspect for shrinking it needs to respond to "size". Something like this?

Shrinking is completely separate from the size parameter. You can safely use (gen/vector ... 0 n) without negatively affecting shrinking.

(gen/sized (fn [size] (gen/resize (* size size) (gen/vector (gen/resize size gen/int)))))

forcing the size of the vector to increase more quickly than the size of the elements in the vector.

I gave this a try, but hit the vector stack overflow problem. I will try the fmap/flatten workaround, but am wondering if this is the right approach.

Your example looks fine. And you can avoid the stack overflow problem by simply not allowing size to grow above a certain limit, say 5000: (gen/resize (min 5000 (* size size)) ...).

(gen/sized (fn [size] (gen/resize (* size size) (gen/vector (gen/resize size gen/int)))))

forcing the size of the vector to increase more quickly than the size of the elements in the vector.

Also wondering about the range of "size". From the code it looks like it goes to 100 due to sample-seq. So functions of size passed to gen/sized should expect range from 0-99, and return the largest meaningful data when given size parameter 99?

When testing, the default max size is 200. However, you can override this simply by passing in a :max-size n argument to tc/quick-check. For example: (tc/quick-check 100 my-prop :max-size 75).

Hope this helps, and don't hesitate to ask anything else.

Comment by Brian Craft [ 18/Nov/14 11:41 AM ]

At the moment the hardest bit for me is understanding the shrinking. Composing generators, I inadvertently build things that won't shrink. E.g. to generate a 2d matrix of numbers with row & column names, I make use of bind so I can size vectors of names to match the matrix size, but then the resulting matrix/vectors will not shrink.

Here's a small example. Generate a vector of short strings, and test that they are all unique.

(tc/quick-check 40 (prop/for-all [f (gen/bind gen/int (fn [i] (gen/vector (gen/resize 5 (gen/such-that not-empty gen/string-ascii)) i)))] (do (println "f" f "count" (count f)) (= (count f) (count (set f))))))

Run this a few times & it will hit a duplicate string, failing the test. It is then unable to shrink the vector.

Here's a case generating two vectors of the same size by generating one vector, then using bind to generate a second of the same size. Testing uniqueness on the second vector, it is able to shrink, however it does so by regenerating the 2nd vector for every test.

(tc/quick-check 40 (prop/for-all [f (gen/bind (gen/vector (gen/resize 5 (gen/such-that not-empty gen/string-ascii))) (fn [v] (gen/hash-map :a (gen/return v) :b (gen/vector gen/int (count v)))))] (do (println "f" f "count" (count (:a f))) (= (count (:b f)) (count (set (:b f)))))))

I expect this will shrink poorly in many cases, because regenerating the second vector destroys the (possibly very rare) failure condition. I suspect this is why my 2d matrices shrink poorly. One really wants to be able to shrink by dropping the same positional element from both vectors. In the case of a fixed dimension (two), you can rewrite it to generate a vector of tuples. For a 2d matrix, though, I'm not sure what to do. Ideally it would shrink by dropping either a column or a row, and the corresponding column or row label. Perhaps I need to write a recursive generator like vector?

Are there any docs/papers describing how the shrinking works?

Comment by Gary Fredericks [ 20/Nov/14 9:45 AM ]

I've included a generator called cap-size in the test.chuck library.

Comment by Reid Draper [ 30/Nov/14 6:11 PM ]

I think both max-size and min-size functions would be a nice addition to test.check proper.

Comment by Jay A. Patel [ 22/Jan/15 7:29 PM ]

Hi Reid –

I am not sure if this is the right place for my comment, but it's closely related to this issue –

As you show, one can use `resize` to reconfigure the default generators. However, here is one issue with default generators for numbers. The default numeric generators (`int`, `nat`, `pos-int`, etc.) all generate numbers < 100 because default `size` is 100. Given JVM interns Integer/Long objects <= 127:

user=> (identical? 127 127)
true
user=> (identical? 128 128)
false

This could be an issue where test.check default number generators miss out on certain class of errors. I opine that a default value of `size` as 100 makes good sense for collections, but not for numbers.

If you agree, a few alternatives could be:

  • change default `size` to > 128, say 256
  • change `int` generator to choose between >1 multiple of size
  • have two different defaults: one for `length` (or `range`) and one for `size`

I don't know though. I think all alternatives have a drawback. I maybe missing something.

Comment by Gary Fredericks [ 22/Jan/15 8:03 PM ]

We've talked a bit about your third option, perhaps having two different classes of numeric generators. Clearly not ever generating big numbers is a pretty bad default for a lot of applications.





[TCHECK-60] string from regular expression generator Created: 23/Jan/15  Updated: 23/Jan/15

Status: Open
Project: test.check
Component/s: None
Affects Version/s: None
Fix Version/s: None

Type: Enhancement Priority: Major
Reporter: Steve Miner Assignee: Reid Draper
Resolution: Unresolved Votes: 0
Labels: None


 Description   

A regular expression is a succinct and flexible way to describe a set of strings. It would be useful to have generator that takes a regular expression and generates matching strings.

There was some hallway discussion about this feature request at the Clojure Conj in November 2014.



 Comments   
Comment by Steve Miner [ 23/Jan/15 9:12 AM ]

https://github.com/gfredericks/test.chuck recently added string-from-regex. That's a good work-around for most users.

Comment by Steve Miner [ 23/Jan/15 9:16 AM ]

https://github.com/miner/herbert recently added a new implementation of a string from regex generator (in v0.6.7). Herbert supports a more limited regex syntax than the generator from test.chuck, but it has no dependencies on other libraries so it can more easily be adopted in a contrib library.





[TCHECK-59] s-pos-int and s-pos-int generators have an off-by-1 issue Created: 22/Jan/15  Updated: 22/Jan/15

Status: Open
Project: test.check
Component/s: None
Affects Version/s: None
Fix Version/s: None

Type: Defect Priority: Major
Reporter: Jay A. Patel Assignee: Reid Draper
Resolution: Unresolved Votes: 0
Labels: None
Environment:

Clojure 1.6.0
test.check 0.6.2



 Description   

=== Problem ===

gen/s-pos-int and gen/s-neg-int do not strictly respect size parameter. Generated values can be off-by-1 from the size value.

=== Samples ===

test=> (gen/sample (gen/resize 5 gen/s-pos-int))
(6 1 3 2 2 1 3 6 6 3)

test=> (gen/sample (gen/resize 5 gen/s-neg-int))
(-1 -6 -2 -6 -5 -3 -2 -1 -4 -3)

=== Root Cause ===

(def s-pos-int
"Generate strictly positive integers bounded by the generator's `size`
parameter."
(fmap inc nat))

(def s-neg-int
"Generate strictly negative integers bounded by the generator's `size`
parameter."
(fmap dec neg-int))

=== Suggestions ===

Instead of using fmap, use such-that to remove the 0 value?



 Comments   
Comment by Jay A. Patel [ 22/Jan/15 12:24 AM ]

I didn't change the priority value. Not a major issue for me.





[TCHECK-57] Performance regression in map generator between 0.5.8 and 0.5.9 Created: 03/Jan/15  Updated: 04/Jan/15

Status: Open
Project: test.check
Component/s: None
Affects Version/s: None
Fix Version/s: None

Type: Defect Priority: Major
Reporter: Brian Kim Assignee: Reid Draper
Resolution: Unresolved Votes: 0
Labels: None
Environment:

Clojure 1.5.1
OSX 10.9.5



 Description   

Tried updating test.check version to latest, but it seems like the map generator has gotten slower from version 0.5.9 up. I tried to make this easy to reproduce in the git repo below, but please let me know if you'd like more information/can't reproduce. Thanks!
https://github.com/brainkim/testcheckperf



 Comments   
Comment by Reid Draper [ 04/Jan/15 2:01 PM ]

Thanks Brian. I'm still digging in to this, but it's appearing like the performance regression is in keyword generation, not map generation. There was also a major change to keywords between 0.5.8 and 0.5.9 in 4528b5ed391d6127b79f4db6bc5e086613da0d81.





[TCHECK-51] Add set generator Created: 12/Nov/14  Updated: 21/May/15

Status: Open
Project: test.check
Component/s: None
Affects Version/s: None
Fix Version/s: None

Type: Enhancement Priority: Major
Reporter: Daniel Compton Assignee: Gary Fredericks
Resolution: Unresolved Votes: 0
Labels: None


 Description   

test.check has generators for maps, vectors, and lists. Can we also add sets as a generator with the same arguments as vector?



 Comments   
Comment by Reid Draper [ 15/Nov/14 1:53 PM ]

There are a few subtleties to adding a set generator with the same arguments as vector, particularly, asking for a particular size set. For example, what should the set generator do with the following arguments: gen/set gen/boolean 5? The domain of the boolean generator is only two elements, yet we've asked for a set with 5 elements. A vector can have duplicates, so this is no problem.

In the meantime, a set can be created:

(defn set-gen
  [vector-gen]
  (gen/fmap set vector-gen))
Comment by Gary Fredericks [ 21/May/15 9:47 PM ]

I have some code for this on a branch, and I think it will probably be included in the next release.

The trick was just to bite the bullet and write this is as a low-level primitive instead of trying to use combinators. You also have to include an abort mechanism similar to such-that – if the generator passed in can't generate unique elements fast enough, you give up.





[TCHECK-61] defspec throwing errors in cljs Created: 18/Mar/15  Updated: 01/Jun/15

Status: Open
Project: test.check
Component/s: None
Affects Version/s: None
Fix Version/s: None

Type: Defect Priority: Major
Reporter: Viktor Eriksson Assignee: David Nolen
Resolution: Unresolved Votes: 1
Labels: cljs
Environment:

OSX



 Description   

Hi,

I have the following issue:

I tried to integrate test.check into my current cljs.test setup. The generators are working like a charm but I have issues with the defspec macro. I get the an error when trying the example from: https://github.com/clojure/test.check#cljstest-integration

(defspec first-element-is-min-after-sorting ;; the name of the test 
         100 ;; the number of iterations for test.check to test 
         (prop/for-all [v (gen/not-empty (gen/vector gen/int))] 
                       (= (apply min v) 
                          (first (sort v)))))

The error is the following:

ERROR in (first-element-is-min-after-sorting) (TypeError:NaN) 
Uncaught exception, not in assertion. 
expected: nil 
  actual: 
#<TypeError: 'undefined' is not an object (evaluating 'f.cljs$lang$maxFixedArity')>

I saw that there is a function called for-all* as well and tried it but it didn't work either, but I got a different error:

ERROR in (first-element-is-min-after-sorting) (Error:NaN) 
Uncaught exception, not in assertion. 
expected: nil 
  actual: 
#<Error: Assert failed: Args to tuple must be generators 
(every? generator? generators)>

Any ideas what the issue might be? Anyone that successfully ran test.check in cljs with defspec?

The tests are run with phantomjs using this example: https://gitlab.com/keeds/cljsinit and my ns requires the following:

[cljs.test.check.generators :as gen] 
[cljs.test.check.cljs-test :as ct :refer-macros [defspec]] 
[cljs.test.check.properties :as prop :include-macros true] 
[cljs.test :as test :refer-macros [deftest testing is use-fixtures]] 

versions: 
[org.clojure/test.check "0.7.0"] 
[org.clojure/clojurescript "0.0-2843"]


 Comments   
Comment by Alex Engelberg [ 30/May/15 4:38 PM ]

defspec's macroexpansion includes (apply cljs.test.check/quick-check ...). However, you are not requiring cljs.test.check in your namespace. And that specific error arises whenever you try to call apply on an undefined function. Therefore, the workaround is to simply add (:require cljs.test.check) at the top of your namespace.

Could we prevent this error from happening in general by simply adding (:require cljs.test.check) to the top of cljs/test/check/cljs_test.cljs?

Comment by Gary Fredericks [ 30/May/15 6:01 PM ]

My guess is that this is basically the same thing as TCHECK-33 (except in clojurescript), so hopefully both can be fixed together.

Comment by Gary Fredericks [ 30/May/15 6:02 PM ]

in the CLJ side simply fixing it with a require as you suggest isn't possible because it would create a circular dependency. So there's some larger refactoring that needs to happen.

Comment by Gary Fredericks [ 30/May/15 6:02 PM ]

adding more details on TCHECK-33

Comment by David Nolen [ 01/Jun/15 8:49 AM ]

Just chiming in to agree that this indeed looks like the ClojureScript version of TCHECK-33.





[TCHECK-70] Redesign gen/choose Created: 06/Jun/15  Updated: 06/Jun/15

Status: Open
Project: test.check
Component/s: None
Affects Version/s: None
Fix Version/s: None

Type: Enhancement Priority: Major
Reporter: Gary Fredericks Assignee: Gary Fredericks
Resolution: Unresolved Votes: 0
Labels: None


 Description   

Currently gen/choose is the generator that everything else is based off of, and it's also a user-facing generator. Its current implementation takes a [0,1) double and scales it to the desired range. This approach degrades w.r.t. distribution as you get higher, and presumably has pretty terrible properties once your range is larger than 2^53. It could also be optimized better for when the range is a power of 2 (no need to go through Double). Seeing as how gen/choose is user facing, we should attempt to redesign it so that it doesn't degrade in this fashion, and can handle bigints.

Requirements

gen/choose should have a uniform distribution regardless of bounds

gen/choose should accept arbitrary integer bounds and handle them correctly

Things to investigate

  • Note that the SplittableRandom class generates integer ranges without going through floating point. We should check if this is faster, since it's definitely a more uniform distribution.
  • Is it worth optimizing for the power-of-two case, the same with SplittableRandom does in the link above?
  • When it's possible for the generator to return a long instead of a bigint is somewhat subtle. E.g., if you call (gen/choose Long/MIN_VALUE Long/MAX_VALUE), then when analyzing the size of the range you'll have to use a bigint (because 2^64 is not representable as a long). gen/choose should generate longs when both args are in long range, probably?
  • Is it actually possible to satisfy the requirements above without a performance hit? You'd think it was just a generator-creation-time change, but bind can blur the line between generator creation and generation time.





[TCHECK-73] gen/int can generate doubles Created: 17/Jun/15  Updated: 17/Jun/15

Status: Open
Project: test.check
Component/s: None
Affects Version/s: None
Fix Version/s: None

Type: Defect Priority: Major
Reporter: Gary Fredericks Assignee: Reid Draper
Resolution: Unresolved Votes: 0
Labels: None


 Description   

gen/int will generate doubles if the size parameter is a double, which is possible when using gen/resize in a sloppy manner.

(gen/generate gen/int 20) => -9
(gen/generate gen/int 20.0) => 5.0





[TCHECK-69] Create a UUID Generator Created: 29/May/15  Updated: 29/May/15

Status: Open
Project: test.check
Component/s: None
Affects Version/s: None
Fix Version/s: None

Type: Defect Priority: Major
Reporter: Gary Fredericks Assignee: Reid Draper
Resolution: Unresolved Votes: 0
Labels: None


 Description   

UUIDs have become first-class objects in clojure with the built-in reader tag #uuid. I can't think of any thorny issues surrounding generating uuids, so it seems like something test.check should have.

Probably it should always generate type-4 UUIDs, i.e. it should set the version field to 4.

I think the only question is whether it should shrink by default. I'm inclined to say no, as most code handling UUIDs is probably treating it opaquely, and so not prone to bugs caused by specific values, and therefore shrinking the UUID would be a waste of time.

Shrinking also gets messy if users are relying on different generated values being unique (which is mildly questionable but useful as well; on the other hand we should have better support for unique values in the near future).

On the other hand calling gen/no-shrink yourself is pretty easy, and having the generator not shrink would be an exception among existing test.check generators.






[TCHECK-74] gen-int generates very small ints Created: 02/Jul/15  Updated: 07/Jul/15

Status: Open
Project: test.check
Component/s: None
Affects Version/s: None
Fix Version/s: None

Type: Defect Priority: Major
Reporter: Jonathan Leonard Assignee: Reid Draper
Resolution: Unresolved Votes: 0
Labels: None


 Description   

Sorry for creating an issue for this but I couldn't find any way to communicate with the developers of this project otherwise. Can someone explain why taking 100,000 ints never yields an int larger than 100? Is it due to the 'size' parameter? Does the runtime just adjust 'size' on the fly to reach 2.7 billion or what not?

If manually sampling a generator, how can one adjust the size?



 Comments   
Comment by Gary Fredericks [ 02/Jul/15 3:22 PM ]

This is a pretty major gotcha currently, and one that definitely would have been fixed by now if there was an obvious best way to fix it.

See the "Numerics" section of this page if you're interested in my ideas for fixing it.

Fortunately there's nothing blocking you from generating what you want once you understand how the pieces work together.

By default the size parameter ranges from 0-200 in normal testing (even smaller if you run your tests less than 200 times at once), which means that gen/int and similar generators stay quite small.

There are at least a couple options for opting-in to larger numbers currently:

  • use the new gen/scale function, e.g. (gen/scale #(* % 10000) gen/int)
  • use the gen/choose function, with the main downside being it ignores the size parameter and generates uniformly within the given range (but it does shrink in the normal way)

As for setting the size when sampling, I recommend the new gen/generate function, which takes an optional size argument.

Comment by Gary Fredericks [ 02/Jul/15 3:23 PM ]

I was looking for another ticket on this topic since I figured one must exist, but I didn't find it. So I'll probably leave this ticket open as a representative of the broader numerics issue.

Comment by Jonathan Leonard [ 02/Jul/15 4:24 PM ]

Thanks for the reply/explanation.

However, doesn't the `gen/scale` solution still have the drawback that it will be biased towards larger numbers-- i.e., it will be impossible to return integers between 1 and 9999 in your example code.

Comment by Gary Fredericks [ 02/Jul/15 8:23 PM ]

The distribution should be a uniform range of size roughly (* 10000 size), so small numbers are definitely possible:

user> (def g (gen/scale #(* % 10000) gen/int))
#'user/g
user> (gen/sample g)
(0 -3423 -14429 788 -16910 49010 4363 56297 -1776 58167)
Comment by Jonathan Leonard [ 07/Jul/15 12:34 PM ]

But is 10,000 a large enough factor to make Long/MAX_VALUE possible? When I tried scaling by (/ MAX_VALUE 100), I didn't see any small numbers.

Comment by Gary Fredericks [ 07/Jul/15 3:01 PM ]

They'll be there, they're just less likely. This is part of what makes in not obvious what the best built-in generators should be, different people have different expectations.

It sounds like what you want is probably something that generates longs with a uniform distribution over the log of the number? So small numbers are just as likely as large numbers?

Comment by Jonathan Leonard [ 07/Jul/15 3:19 PM ]

Yes, I think uniform distribution over the width of the number (whether 8-bit, 16-bit, 32-bit, 64-bit or what have you) makes the most sensible default. Can you show an example of how that would be achieved for one of the above widths?

Thanks!

Comment by Gary Fredericks [ 07/Jul/15 4:22 PM ]

The thing that makes it slightly trickier is how it interacts with size. It's not too hard to write a generator that ignores size and gives you the uniform-width distribution.

Maybe another option is something like:

(gen/bind (gen/sized (fn [size] (let [max-bits (min 60 size)] (gen/choose 0 max-bits)))) 
          #(gen/resize (bit-shift-left 1 %) gen/int))

I used 60 instead of 63 or 64 because I got errors with 63 that I expect are related to how gen/int works with a very high size. I'm not sure if that should be considered a bug or not – depends on if generators should be able to work with arbitrarily large sizes.





[TCHECK-3] Add ability to customize test output when a test fails Created: 04/Mar/14  Updated: 04/Mar/14

Status: Open
Project: test.check
Component/s: None
Affects Version/s: None
Fix Version/s: None

Type: Enhancement Priority: Minor
Reporter: Jean Niklas L'orange Assignee: Reid Draper
Resolution: Unresolved Votes: 0
Labels: None


 Description   

From the original description:

I would like the ability to control / enhance / replace the output generated when a test fails. Printing the raw values works well as long as those values have a useful output for the test at hand, but this doesn't always hold true.

To give a bit of background, I was playing with generating sequences of partially-applied functions and then applying them to an initial state. Here's a small excerpt from the full gist:

:smallest [(#<core$partial$fn__4190 clojure.core$partial$fn__4190@2a799171>)]

What would be more useful to me is if the output had something like this, which is more meaningful:

:smallest [(- 45 %)]

In this example, (- 45 %) is all my custom output string. 45 is a value from an upstream generator (and would thus change), and % is just a string modeled after the anonymous function syntax.

I could even see my particular application rolled up as a generator:

(def gen-inc
  (gen/bind gen/pos-int #(gen/partial + %)))

Which could handle some of the heavy lifting for the user.

Original issue






[TCHECK-4] Handle sigint Created: 04/Mar/14  Updated: 04/Mar/14

Status: Open
Project: test.check
Component/s: None
Affects Version/s: None
Fix Version/s: None

Type: Enhancement Priority: Minor
Reporter: Jean Niklas L'orange Assignee: Reid Draper
Resolution: Unresolved Votes: 0
Labels: None


 Description   

During shrinking, this should simply return the best shrink so-far. During testing, perhaps it should just return the number of tests that've been run. In both cases, some indication should be given in the return value that the user pressed ctrl-c.

Original issue - #41 on GitHub






[TCHECK-8] Bound tests and shrinks in time Created: 04/Mar/14  Updated: 04/Mar/14

Status: Open
Project: test.check
Component/s: None
Affects Version/s: None
Fix Version/s: None

Type: Enhancement Priority: Minor
Reporter: Jean Niklas L'orange Assignee: Reid Draper
Resolution: Unresolved Votes: 1
Labels: None


 Description   

It should be possible to bound tests and shrinks in time.

The original issue also mentions shrink interruption through Ctrl-c, but this is already covered in #TCHECK-4.

Original issue - #9 GitHub






[TCHECK-9] Properly document how :max-size works Created: 04/Mar/14  Updated: 14/Apr/14

Status: Open
Project: test.check
Component/s: None
Affects Version/s: None
Fix Version/s: None

Type: Enhancement Priority: Minor
Reporter: Jean Niklas L'orange Assignee: Reid Draper
Resolution: Unresolved Votes: 0
Labels: documentation


 Description   

I have had several issues where I thought increasing quick-check's num-tests argument would increase the time taken linearly, but surprisingly it grew quadratically. This is related to the auto-increasing of the size to call-gen, which I believe it isn't documented well enough. It also seems like other people have had trouble with this functionality, see the reply to announcement of clojure.test.check.

Preferably, it should be mentioned in the docstring and the tutorial that the size of the structures generated grows linearly with the amount of tests you have, up to 200. As a result, doubling the amount of tests does not necessarily double the amount of time taken to run the tests.

In some of my tests, this behaviour increases the running time exponentially, and I actually used quite some time figuring out why. The documentation on recursive generators decreases the size of the binary tree in half, but it is not explained why it is split in half. While obvious in hindsight, explicitly explaining that recursive structures should have an amount of nodes proportional to the input size argument and how it is done, would be beneficial.



 Comments   
Comment by Reid Draper [ 04/Mar/14 8:50 PM ]

+1. This is not well-documented now. Further, I'm still not 100% sure that halving inside of a recursive generator is the right thing to do. For some structures, log2 may be more appropriate. In the mailing list post you mention, gen/any is clearly not bounded enough at roughly size=90, consider it's using more than 1GB of heap.

Comment by Reid Draper [ 14/Apr/14 8:58 PM ]

I've made things a little better in 57df9b9b8f2fe61a798fbf683046e8d1aa128228.





[TCHECK-19] Permit a data-structure containing generators to be used as a generator Created: 11/Apr/14  Updated: 22/Jun/14

Status: Open
Project: test.check
Component/s: None
Affects Version/s: None
Fix Version/s: None

Type: Enhancement Priority: Minor
Reporter: Darrick Wiebe Assignee: Reid Draper
Resolution: Unresolved Votes: 0
Labels: None

Attachments: File add_record.diff    

 Description   

I'm working through some of the examples in John Hughes' "QuickCheck Testing for Fun and Profit" paper. I'd like the structures I generate in those tests to look like the this:

[:apply `reg [(gen-name) (gen/elements pids)]]

Rather than the way they must be expressed now:

(gen/tuple (gen/return :apply) (gen/return `reg) (gen/tuple (gen-name) (gen/elements pids)))

I think a simple recursive function could be used to generate these, with the caveat that currently generators create a map {:gen #<fn...>}, which I'd propose to change to a record so that we could distinguish between a generator and a map data-structure.

This seems to be implemented to good effect in Quvig QuickCheck already:

In general, Quviq QuickCheck permits any data-structure containing embedded generators to be used as a generator for data-structures of that shape—something which is very convenient for users, but quite impossible in Haskell, where embedding a generator in a data-structure would normally result in a type error. This technique is used throughout the generators written at Ericsson. – Hughes



 Comments   
Comment by Darrick Wiebe [ 11/Apr/14 5:06 PM ]

Playing around in the REPL today, I came up with this code that seems to work well converting arbitrary literals into generators:

(defprotocol LiteralGenerator
  (-literal->generator [literal]))

(defn literal->generator [literal]
  (cond
    (satisfies? LiteralGenerator literal) (-literal->generator literal)
    (vector? literal) (apply gen/tuple (mapv literal->generator literal))
    (and (map? literal) (= [:gen] (keys literal)) (fn? (:gen literal))) literal
    (map? literal) (gen/fmap (partial into {}) (literal->generator (mapv vec literal)))
    (set? literal) (gen/fmap set (literal->generator (vec literal)))
    (list? literal) (gen/fmap (partial apply list) (literal->generator (vec literal)))
    :else (gen/return literal)))

Generating from a record is probably something that would be generally useful, so I added this as well:

(defn record->generator
  ([record]
   ; Is there a better way to do this?
   (let [ctor (eval (read-string (str "map->" (last (clojure.string/split (str (class record)) #"\.")))))]
     (record->generator ctor record)))
  ([ctor record]
   (gen/fmap ctor (literal->generator (into {} record)))))

Which enables me to extend a record like this:

(defrecord Foo [a b]
  LiteralGenerator
  (-literal->generator [this] (record->generator map->AbcDef this)))

I haven't looked at the possibility of baking this code into test.check at all yet. I'd like feedback on what I've got so far before pursuing this any further.

Comment by Reid Draper [ 11/Apr/14 5:31 PM ]

So, I have kind of mixed feelings about this. I agree it's convenient, but I also worry that being loose like that can allow more accidents to occur, and doesn't force users to make the distinction between values and generators. For example, what if you do something like this: [int int]. Did you mean to type [gen/int gen/int], or do you really intend to have written the equivalent of [(gen/return int) (gen/return int)]? If every function that expects a generator can also take a value, we can no longer start adding error-checking to make sure you're correctly passing generators. On that same token, I do see the benefit of being able to use data-structure literals. What if we wrote a function that you had to use to create a generator with this syntax, something like: {{(gen/literal [:age gen/int])}}. That way you could opt-in to this syntax, but only within the scope of gen/literal?

Comment by Darrick Wiebe [ 11/Apr/14 5:40 PM ]

I agree that is a concern, and since you're not guaranteed to see generator output, it might be better to be explicit about using gen/literal. It's still a nice clean solution. Fortunately, that's exactly what I've written! We'd just need to rename literal->generator to literal and install it into the clojure.test.check.generators namespace.

Comment by Reid Draper [ 11/Apr/14 6:56 PM ]

I think the first step is changing generators to use Records. Interested in making a patch for that?

Comment by Darrick Wiebe [ 11/Apr/14 8:31 PM ]

A very simple patch to use a Generator record rather than a simple {:gen ƒ) map.

Comment by Reid Draper [ 12/Apr/14 6:15 PM ]

I've applied the record-patch in ef132b5f85a07879f01417c9104aa6dea771fdb4. Thanks. I've also added a generator? helper function.

Comment by Philip Potter [ 22/Jun/14 4:39 PM ]

I spotted something which seemed relevant: ztellman/collection-check defines a tuple* fn which is like gen/tuple but automatically wraps literals with gen/return.

Notably, despite not solving the general case, this would have solved the example in the issue description.





[TCHECK-41] Add helpers for deprecating generators Created: 01/Sep/14  Updated: 01/Sep/14

Status: Open
Project: test.check
Component/s: None
Affects Version/s: None
Fix Version/s: None

Type: Enhancement Priority: Minor
Reporter: John Walker Assignee: Reid Draper
Resolution: Unresolved Votes: 0
Labels: None

Attachments: File deprecation-helpers.diff    
Patch: Code

 Description   

It should be clearly communicated to the user when generators are deprecated. This patch supplies helpers for easily deprecating generators.

def-throwing-generator receives the metadata that its generator will throw if its called. As an example, suppose deprecated-generator was deprecated in 0.5.9.

(def-throwing-generator deprecated-generator
  "deprecated-generator has been deprecated. Use current-generator instead."
  {:deprecated "0.5.9"})

Then if the user looked up its documentation, they would see from cider:

clojure.test.check.generators/deprecated-generator
Deprecated in 0.5.9
  deprecated-generator has been deprecated. Use current-generator instead.

or if they used the generator and ran cider-test, they would see:

Test Summary
foobar.core-test

Ran 1 tests, in 1 test functions
1 errors


Results

Error in a-test
FIXME, I fail.
expected: (= "a" (sample deprecated-generator 5))
  actual: clojure.lang.ExceptionInfo: deprecated-generator has been deprecated. Use current-generator instead. {:deprecated "0.5.9"}





[TCHECK-14] re-organize README and doc/ Created: 29/Mar/14  Updated: 29/Mar/14

Status: Open
Project: test.check
Component/s: None
Affects Version/s: None
Fix Version/s: None

Type: Enhancement Priority: Minor
Reporter: Reid Draper Assignee: Reid Draper
Resolution: Unresolved Votes: 0
Labels: documentation


 Description   

I'd like to re-organize the README, and focus on making it easier for users to find the information they need. The README is currently pretty long, and could be information overload. My current thinking is that the new README should be short, and focus on providing links to people based on their experience with test.check. I think something like the following three headings would be useful:

1. First time user. Show me what this is all about.

  • Focus on 'seeing something cool' within five minutes. Maybe include some example tests in examples/. This way new users can simply clone the repository and run a test in under two minutes.

2. Still getting started, show me some tutorials and examples.

  • This will probably build on the existing doc/intro.md file, which currently describes how to write generators.

3. Existing user, show me API documentation, release notes, advanced techniques.

  • API documentation
  • Release-notes, make it clear these are up-to-date, and have clear instructions when there are breaking changes
  • Advanced techniques/tutorials





[TCHECK-7] Add nested property support Created: 04/Mar/14  Updated: 26/May/14

Status: Open
Project: test.check
Component/s: None
Affects Version/s: None
Fix Version/s: None

Type: Enhancement Priority: Minor
Reporter: Jean Niklas L'orange Assignee: Reid Draper
Resolution: Unresolved Votes: 2
Labels: None


 Description   

Including functionality to add support for nested properties would be beneficial. An example of how this may work follows:

(defn exists?
  [e coll]
  (some #{e} coll))

(defn remove-elem
  [e coll]
  (remove #{e} coll))

;; this test should fail, and illustrates nested properties
(quick-check (for-all [v (gen/vector gen/int)]
               (for-all [e (elements v)]
                 #(exists? e (remove-elem e v)))))

The requirement for this is support for reifying properties, which is not currently implemented.

Original issue - GitHub #10



 Comments   
Comment by Maciej Jaśkowski [ 14/May/14 4:14 PM ]

What's the status of this issue?
It would help us a lot to have this functionality up and running in the following case: I want to verify my path finding algorithm finds optimal solution in the given graph G, so I generate the graph and a big number of random paths and check that all the random paths are "less optimal" then the one found by the algorithm.

Having nested for-all that would be simple. Is there a way around?

Comment by Reid Draper [ 14/May/14 4:24 PM ]

Sorry for the lack of update here. Nested properties are still in a bit of hammock-time on my part. The good news is they are merely convenient, and you can achieve the exact same effect with clojure.test.check.generators/bind. You can see an example of bind being used here.

Comment by Maciej Jaśkowski [ 15/May/14 12:08 AM ]

Apparently I need some more hints because all I can generate (using gen/bing, btw) are pairs '(graph list-of-random-paths)

That is useful as long as the algorithm works as expected. However, once it fails it's pretty hard to figure out which of random-paths failed it.

Comment by Reid Draper [ 15/May/14 1:02 PM ]

Can you show me how you would write the test with nested properties, and then perhaps I can help you rewrite it to use bind instead?

Comment by Maciej Jaśkowski [ 15/May/14 3:58 PM ]

Great! Thank you!

Please, find the most important part below.

(defn gen-matrix-and-paths [size]
    (gen/bind
      (gen-matrix gen/s-pos-int size)
      (fn [matrix]
        (let [ nodes (into [] (range (count matrix)))
               gen-perms (gen-permutations nodes)
               perms (distinct (gen/sample gen-perms 500)) ]
          (gen/tuple 
            (gen/return matrix) 
            (gen/return perms))))))

gen-matrix generates a square matrix of incidence e.g. [[0 1] [4 0]]
gen-permutations generates a permutation of provided vector

Comment by Reid Draper [ 15/May/14 4:41 PM ]

I'm confused. What issue are you running into? The code you've pasted uses bind, and not a nested property.

Comment by Maciej Jaśkowski [ 17/May/14 3:26 AM ]

Ok but now I can only write something like this:

(prop/for-all [data (gen-matrix-and-paths 6)]
   (let [ [matrix random-paths] data ]
     (not-worse-then-others? tsp matrix random-paths))))

which, if my 'tsp' algorithm is not working correctly would point me to a matrix and a vector (length 500) of paths one of which is counterexample. It would be better if instead I got the matrix and a single counterexample.

Comment by Reid Draper [ 17/May/14 12:44 PM ]

I'm still not sure what this would look like using a nested-property. Can you please write the code the way you'd like to see it as a nested property? This will involve at least two for-alls.

Comment by Maciej Jaśkowski [ 18/May/14 4:56 PM ]

Sure!

I would like to write something like that:

(prop/for-all [matrix (gen-matrix 6)]
  (prop/for-all [random-path (gen-random-path matrix)]
    (<= (cost (tsp matrix)) (cost random-path))))
Comment by Maciej Jaśkowski [ 23/May/14 6:22 AM ]

ping!

Comment by Reid Draper [ 26/May/14 11:47 AM ]

Ok, let's translate that to using gen/bind. Just like with nested properties, bind allows you to create generators that depend on the value of another generator. For example, your gen-random-path depends on the previously generated matrix variable. So let's write a generator that returns both a matrix and a random-path.

(def matrix-and-path
  "Return a two-tuple of [matrix random-path]"
  (gen/bind (gen-matrix 6)
            (fn [matrix]
               (gen/tuple (gen/return matrix) (gen-random-path matrix)))))

(prop/for-all [[matrix random-path] matrix-and-path]
  (<= (cost (tsp matrix)) (cost random-path))




[TCHECK-26] defspec does not respect (is (thrown? ...)) Created: 20/Apr/14  Updated: 13/May/14

Status: Open
Project: test.check
Component/s: None
Affects Version/s: None
Fix Version/s: None

Type: Defect Priority: Minor
Reporter: Sung Pae Assignee: Reid Draper
Resolution: Unresolved Votes: 0
Labels: None


 Description   

The following test does not pass:

(quick-check 1
  (for-all [x g/any]
    (is (thrown? Throwable (assert false)))))

This can be worked around by wrapping the (is ...) form with a call to boolean:

(quick-check 1
  (for-all [x g/any]
    (boolean (is (thrown? Throwable (assert false))))))

The (is (thrown? ...)) form does not return true|false, but will return the Exception itself when it passes.

Thank you!



 Comments   
Comment by Reid Draper [ 13/May/14 6:14 PM ]

This is correct. And I'm not yet sure what the best way forward is, but my hunch is that we'll need to support non-boolean return values from tests. Maybe something that implements a ->pass? function.





[TCHECK-53] property creating function similar to map as alternative to for-all* Created: 17/Nov/14  Updated: 17/Nov/14

Status: Open
Project: test.check
Component/s: None
Affects Version/s: None
Fix Version/s: None

Type: Enhancement Priority: Minor
Reporter: Steve Miner Assignee: Reid Draper
Resolution: Unresolved Votes: 0
Labels: None


 Description   

As an alternative to the `for-all*` syntax, I suggest using a function signature that looks more like `map` with the predicate as the first arg and the generators as the rest of the args.

For example, these forms would all be equivalent:

(property (comp integer? *) gen/int gen/int gen/int) ;; proposed style

(for-all* [gen/int gen/int gen/int] (comp integer? *)) ;; current style



 Comments   
Comment by Steve Miner [ 17/Nov/14 8:20 AM ]

It's easy enough to write for myself. I'll make a patch if you want it.

(defn property 
    ([pred gen] (prop/for-all* [gen] pred))
    ([pred gen1 gen2] (prop/for-all* [gen1 gen2] pred))
    ([pred gen1 gen2 & more] 
        (prop/for-all* (list* gen1 gen2 more) pred)))




[TCHECK-2] pos-int is confusingly named Created: 04/Mar/14  Updated: 04/Mar/14

Status: Open
Project: test.check
Component/s: None
Affects Version/s: None
Fix Version/s: None

Type: Enhancement Priority: Minor
Reporter: Jean Niklas L'orange Assignee: Reid Draper
Resolution: Unresolved Votes: 2
Labels: None


 Description   

The generators neg-int and pos-int generate 0, which is confusing as (= (pos? 0} (neg? 0) false). The suggestion is to change s-pos-int and s-neg-int to pos-int and neg-int respectively, and replace the current neg-int and pos-int. Suggestions for new names include non-x-int and not-x-int.

Original issue






[TCHECK-5] Include a float generator Created: 04/Mar/14  Updated: 10/Oct/14

Status: Open
Project: test.check
Component/s: None
Affects Version/s: None
Fix Version/s: None

Type: Enhancement Priority: Minor
Reporter: Jean Niklas L'orange Assignee: Reid Draper
Resolution: Unresolved Votes: 3
Labels: None


 Description   

There seems to be a demand for a general float generator.

One of the bigger issues with a float generator is shrinking, which is not straightforward. Reid Draper intially considered including a small implementation, using fmap to coerce a ratio into a float:

(def float (gen/fmap float gen/ratio))
(gen/sample float 20)
;; => (0.0 0.0 -1.0 0.0 -1.0 0.75 0.5 0.8 -2.0 -0.5714286 2.6666667 -2.0
;;     -0.46153846 -0.125 -12.0 -1.75 -2.4 0.41666666 0.64285713 -1.125)

The Haskell implementation is performing float shrinking through fractions.

Chas Emerick implemented a bigdec generator using ideas from data.generators, but mentioned that manipulating the long and int bits would be more performant. However, such shrinking would require good knowledge on how IEEE 754 works, in order to avoid bugs whenever shrinking is performed.

A slower, but more understandable and portable double generator was also suggested, again by Chas Emerick. The implementation of said generator could be found at https://gist.github.com/cemerick/7599452

Original issue - GitHub #36



 Comments   
Comment by Gary Fredericks [ 10/Oct/14 1:41 PM ]

I've written one and added it to test.chuck here.

It scales exponentially with the size so it very quickly fills the whole double range (I think once size > 53 or so).

It will generate +/-Infinity but not NaN.

It shrinks in an okayish way.

I'm not sure what the design goals are here, so don't know if it's worth submitting yet.





[TCHECK-43] Test.check lacks a generator for all floats and floats in a range Created: 17/Sep/14  Updated: 14/Dec/14

Status: Open
Project: test.check
Component/s: None
Affects Version/s: None
Fix Version/s: None

Type: Enhancement Priority: Minor
Reporter: Adam Clements Assignee: Reid Draper
Resolution: Unresolved Votes: 1
Labels: enhancement

Attachments: File float-generators.diff    
Patch: Code and Test

 Description   

Test.check lacks generators for floats and float ranges. I have implemented a float generator which shrinks to zero based on gen/ratio and also a float-in-range which is bounded by size.



 Comments   
Comment by Reid McKenzie [ 14/Dec/14 12:22 PM ]

Duplicate of http://dev.clojure.org/jira/browse/TCHECK-5





[TCHECK-15] gen/for macro for alternate combinator syntax Created: 02/Apr/14  Updated: 08/Jan/15

Status: Open
Project: test.check
Component/s: None
Affects Version/s: None
Fix Version/s: None

Type: Enhancement Priority: Minor
Reporter: Gary Fredericks Assignee: Reid Draper
Resolution: Unresolved Votes: 3
Labels: None

Attachments: Text File TCHECK-15-p1.patch     Text File TCHECK-15-p2.patch     Text File TCHECK-15-p3.patch     Text File TCHECK-15-p4.patch     Text File TCHECK-15.patch    

 Description   

I think the syntax of clojure.core/for would be a good fit for test.check's combinators. For example:

(defn gen-even-subset
  "Returns a generator that generates an even-cardinality
   subset of the given elements"
  [elements]
  (gen/for [bools (apply gen/tuple (repeat (count elements) gen/boolean))
            :let [true-count (->> bools (filter identity) (count))]
            :when (even? true-count)]
    (->> (map list bools elements)
         (filter first)
         (map second)
         (set))))

This combines the capabilities of fmap, bind, and such-that into a familiar syntax.

One downside here is the temptation to use multiple clauses for independent generators, resulting in a use of gen/bind when gen/tuple would be simpler and presumably shrink easier. An approach to this is an additional supported clause, perhaps called :parallel, that uses the syntax of :let to provide the functionality of gen/tuple:

(gen/for [:parallel [n1 gen/nat
                     n2 gen/nat]
          :let [sum (+ n1 n2)]]
  {:nums [n1 n2] :sum sum})

Compared to gen/tuple, this has the advantage of placing generators syntactically next to names, rather than segregating the generators from the names.

The :parallel feature has not been added to the current patches.



 Comments   
Comment by Gary Fredericks [ 05/Apr/14 3:23 PM ]

I think there might be some design ambiguity around the meaning of :when. In particular, in the following contrived example:

(for [n nat
      v (vec (return n))
      :let [sum (reduce + v)]
      :when (pos? sum)]
  v)

In my default design this can hang, for the same reason that this code can hang:

(bind nat 
      (fn [n]
        (such-that
          (fn [v] (pos? (reduce + v)))
          (vector (return n)))))

But it could just as well have been written:

(such-that 
  (fn [v] (pos? (reduce + v)))
  (bind nat (fn [n] (vector (return n)))))

So the issue is whether a :when filter is applied to just the previous generator or to all of the previous generators. I have some hazy notion that the latter would be less efficient in some cases, but I'm not sure what. So I think our options are:

  1. Decide to always do it one way or the other
  2. Provide a third keyword (:when-all?) with different behavior
  3. Don't write this macro at all because it's too difficult to understand

My gut is to do option 1 and just apply :when to the previous generator.

Comment by Gary Fredericks [ 08/Apr/14 9:24 PM ]

Attached my initial draft. The implementation took a lot more thought than I expected, and is a bit subtle, so I included some inline comments explaining the structure of the macro.

Comment by Gary Fredericks [ 13/Apr/14 8:33 PM ]

Attached TCHECK-15-p1.patch, updated to apply to the current master.

Comment by Gary Fredericks [ 16/Apr/14 9:51 PM ]

Attached TCHECK-15-p2.patch which adds a note to the docstring about independent clauses, shrinking, and tuple.

Comment by Gary Fredericks [ 16/Apr/14 9:58 PM ]

Attached TCHECK-15-p3.patch which fixes one bug and one redundancy in namespace aliases.

Comment by Gary Fredericks [ 13/May/14 10:37 AM ]

Attached TCHECK-15-p4.patch which fixes a bug with destructuring (and adds a regression test).

Comment by Gary Fredericks [ 13/May/14 10:38 AM ]

Also might be helpful to note that I've put this in my test.check utility library for now: https://github.com/fredericksgary/test.chuck#for.

Comment by Michael Blume [ 08/Jan/15 12:21 AM ]

I wonder if it'd be possible to avoid :parallel by analyzing the code and checking whether the bindings can be run in parallel?

Comment by Gary Fredericks [ 08/Jan/15 9:16 AM ]

I think it's possible in theory, but we'd need access to a non-buggy code walker.

Additionally you might argue that it makes the meaning of the code a lot more subtle.





[TCHECK-58] Alternate clojure.test integration Created: 05/Jan/15  Updated: 20/Jan/15

Status: Open
Project: test.check
Component/s: None
Affects Version/s: None
Fix Version/s: None

Type: Enhancement Priority: Minor
Reporter: Colin Williams Assignee: Reid Draper
Resolution: Unresolved Votes: 0
Labels: None


 Description   

I've got a macro that integrates with clojure.test at a different level than defspec, which I've found useful in migrating tests from non-generative to generative:

(deftest something-important
  (checking "something really important" [x gen/int]
    (is (= (important-calculation x) (inc x)))))

I've found it natural to reach the above from something like this:

(deftest something-important
  (testing "something really important"
    (is (= (important-calculation 1) 2))))

There are still a lot of rough edges, but I wanted to get peoples reactions before I started polishing it for a patch.



 Comments   
Comment by Reid Draper [ 20/Jan/15 10:36 PM ]

Thanks for the comment Colin. At the moment, I'd like to keep the integration simple (meaning there is one way to do things), and keep 'alternative syntax' macros and functions out of test.check itself. Gary's test.chuck might be a good home for this.





[TCHECK-65] add a float generator Created: 06/Apr/15  Updated: 15/Apr/15

Status: Open
Project: test.check
Component/s: None
Affects Version/s: None
Fix Version/s: None

Type: Enhancement Priority: Minor
Reporter: Tom Crayford Assignee: Reid Draper
Resolution: Unresolved Votes: 0
Labels: None

Attachments: File add_float_generator.diff    
Patch: Code

 Description   

Adds a `gen/float` generator. Internally it uses `gen/ratio` and `gen/int`



 Comments   
Comment by Gary Fredericks [ 15/Apr/15 9:55 AM ]

Looks like you added a stray (:import java.util.Random)?

Comment by Gary Fredericks [ 15/Apr/15 9:58 AM ]

I expect the reason we haven't added something like this before is that it wasn't clear what domain we should be targeting; in particular whether all possible floats are likely to be generated. This fits into the broader category of numeric generators that have been in need of a redesign.

For another approach this generator in test.chuck attempts to generate all possible Doubles.





[TCHECK-63] add sizing arguments to map generator Created: 06/Apr/15  Updated: 28/Apr/15

Status: Open
Project: test.check
Component/s: None
Affects Version/s: None
Fix Version/s: None

Type: Enhancement Priority: Minor
Reporter: Tom Crayford Assignee: Reid Draper
Resolution: Unresolved Votes: 0
Labels: None

Attachments: File better_map_generation.diff    
Patch: Code

 Description   

Adds sizing arguments to `gen/map`, just like `gen/vector`.



 Comments   
Comment by Gary Fredericks [ 15/Apr/15 10:04 AM ]

This doesn't seem like a bad idea, but glancing at the code I suspect that it might inadvertently return smaller maps when there are duplicate keys; does that sound right?

Might be tricky to get this to perform well especially under shrinking.

Comment by Tom Crayford [ 15/Apr/15 6:59 PM ]

Yeah, that's right. I don't think that's a huge issue though - there's no guarantee that it will hit the max limit.

By "perform well" - are you thinking about the results, or how slow shrinking will be? I assume the results will be ok.

Comment by Gary Fredericks [ 15/Apr/15 8:21 PM ]

Well by "smaller" I meant "smaller than the minimum" in particular for the arity where you set a minimum size.

Comment by Gary Fredericks [ 28/Apr/15 12:16 PM ]

I just hacked up some generators for distinct collections that should make it easier to implement this in a solid way.

Not sure how soon it'll make it into master/release, just wanted to make a note here.





[TCHECK-48] Add sublist generator Created: 28/Oct/14  Updated: 21/May/15

Status: Open
Project: test.check
Component/s: None
Affects Version/s: None
Fix Version/s: None

Type: Enhancement Priority: Minor
Reporter: Reid Draper Assignee: Reid Draper
Resolution: Unresolved Votes: 0
Labels: None


 Comments   
Comment by Gary Fredericks [ 21/May/15 9:40 PM ]

Related (but probably different): https://github.com/gfredericks/test.chuck/blob/20f17550acecdfed4ca3259ebdad34e088fd453a/src/com/gfredericks/test/chuck/generators.clj#L141-150





[TCHECK-33] test.check is entangled with clojure.test Created: 21/Jun/14  Updated: 30/May/15

Status: Open
Project: test.check
Component/s: None
Affects Version/s: None
Fix Version/s: None

Type: Defect Priority: Minor
Reporter: Philip Potter Assignee: Gary Fredericks
Resolution: Unresolved Votes: 0
Labels: None


 Description   

test.check is tightly coupled to clojure.test; this means that it's difficult to use it with other test frameworks.

There is a dependency cycle between clojure.test.check and clojure.test.check.clojure-test.

quick-check uses clojure.test/report to print dots to the screen, meaning that quick-check necessarily depends on clojure.test



 Comments   
Comment by Philip Potter [ 21/Jun/14 8:54 AM ]

I wonder if the solution is to allow you to provide your own reporting functions to quick-check to report successes, failures, and trials? That way different frameworks can inject different behaviour as required.

Comment by Gary Fredericks [ 23/Mar/15 2:07 PM ]

This problem, according to anecdote, has the more severe effect of requiring a user of clojure.test.check.clojure-test to require clojure.test.check even if it's not otherwise being used (the defspec macro attempts to fix this but I believe that trick doesn't actually work, though the problem doesn't usually surface).

Comment by Gary Fredericks [ 30/May/15 6:03 PM ]

I was working on this a while ago but somehow got side-tracked.

I think my main idea was to add another option arg to quick-check which would be a reporting callback function. Then defspec would simply supply a function that does whatever it currently does, maybe integrating with clojure.test somehow.





[TCHECK-72] Poisson generator Created: 07/Jun/15  Updated: 07/Jun/15

Status: Open
Project: test.check
Component/s: None
Affects Version/s: None
Fix Version/s: None

Type: Enhancement Priority: Minor
Reporter: Jason Felice Assignee: Gary Fredericks
Resolution: Unresolved Votes: 0
Labels: None

Attachments: File poisson.diff    
Patch: Code

 Description   

Adds `gen/poisson`, which samples numbers from a Poisson distribution.



 Comments   
Comment by Jason Felice [ 07/Jun/15 1:40 PM ]
user=> (gen/sample (gen/poisson 1/500) 500)
(0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0)
user=> (gen/sample (gen/poisson 1/5) 500)
(1 0 0 0 0 0 0 0 0 0 0 0 0 1 0 1 0 0 0 0 0 0 0 0 0 0 0 0 1 0 0 0 1 0 0 1 0 0 0 0 0 0 0 0 1 1 1 0 0 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 1 0 0 0 1 1 0 1 0 0 0 0 0 0 0 0 0 1 0 0 0 0 1 0 0 0 0 0 0 0 1 0 0 0 0 0 0 1 0 0 1 1 0 0 0 0 0 1 0 0 0 1 0 0 0 0 0 0 0 1 0 0 1 0 0 0 0 0 0 0 0 0 0 0 1 0 0 1 0 0 0 0 0 0 1 0 1 0 0 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 0 0 0 1 0 0 0 0 0 0 0 0 1 0 1 0 1 0 0 0 0 0 0 0 1 1 0 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 1 1 0 0 0 0 0 1 0 0 0 1 0 0 0 0 0 1 1 0 1 0 0 0 1 0 0 0 0 0 0 0 0 1 0 0 1 1 0 0 0 0 0 0 0 0 1 1 1 0 1 0 0 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 1 0 1 0 0 1 1 0 1 1 0 0 0 0 0 1 0 0 0 0 0 0 0 0 1 0 0 1 1 0 0 1 1 0 0 0 1 0 0 0 0 0 0 0 0 0 1 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 0 0 1 1 0 0 0 0 0 0 0 0 0 0 1 1 1 0 0 1 1 0 1 0 0 0 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 0 1 1 0 0 0 0 0 0 1 0 0 0 0 0 0)
user=> (gen/sample (gen/poisson 1) 500)
(2 2 1 1 1 0 2 2 0 0 0 1 0 1 2 1 2 2 2 2 1 1 2 2 0 1 2 0 0 1 1 2 1 1 0 1 2 0 1 1 0 1 1 1 0 2 2 0 2 1 1 1 0 1 1 0 0 1 2 0 1 1 1 2 0 1 1 0 0 0 0 2 2 0 1 1 2 2 0 0 1 0 2 0 0 1 2 1 1 1 1 1 2 2 0 1 0 1 1 1 0 1 0 2 1 0 1 0 1 0 0 0 2 1 1 0 0 2 0 1 2 1 0 2 1 0 1 0 0 1 1 2 1 2 0 1 1 2 0 1 1 2 1 1 0 0 2 1 2 1 0 0 2 1 1 2 0 1 2 1 1 0 2 2 1 1 2 1 1 2 0 1 0 1 0 0 0 1 2 0 2 1 1 2 1 0 1 0 0 1 1 0 1 0 1 1 2 0 0 2 1 0 2 0 1 0 2 1 1 0 2 1 0 1 1 0 1 0 2 0 0 0 0 2 1 0 0 1 1 2 1 0 2 2 2 2 0 1 2 1 0 1 1 1 2 1 0 0 1 0 0 1 2 1 0 0 0 2 2 0 1 1 1 0 2 0 0 0 1 1 1 0 0 0 2 1 2 1 0 0 2 1 1 1 2 1 1 2 2 0 1 2 1 2 2 0 0 1 2 2 0 1 1 2 0 1 1 2 1 1 1 1 0 2 2 2 1 0 1 1 0 1 2 0 1 1 1 1 2 1 0 1 2 0 0 2 1 0 2 0 0 1 0 1 1 1 2 1 1 0 0 2 1 1 1 2 2 0 0 2 1 1 1 0 0 2 2 0 1 0 1 2 1 1 0 0 0 2 0 1 1 1 2 2 1 1 0 2 1 0 1 2 1 0 0 0 1 0 0 2 1 1 0 1 2 1 1 0 2 1 0 0 0 1 2 1 1 2 1 1 0 2 0 0 2 2 1 0 2 0 2 2 2 1 1 2 0 0 0 1 0 1 2 0 2 1 2 0 1 0 1 2 1 0 1 0 1 0 0 0 1 1 2 2 0 0 2 1 1 1 2 1 1 0 2 1 2 0 0 1 0 0 1 0 1 0 0 0 1 1 1 0 1 0 1 0 0 0 0 2)




[TCHECK-71] Exponential generator Created: 07/Jun/15  Updated: 07/Jun/15

Status: Open
Project: test.check
Component/s: None
Affects Version/s: None
Fix Version/s: None

Type: Enhancement Priority: Minor
Reporter: Jason Felice Assignee: Gary Fredericks
Resolution: Unresolved Votes: 0
Labels: None

Attachments: File exponential.diff     File exponential.diff    
Patch: Code

 Description   

A generator for generating exponential values. I'm using this to generate queue latency values, to simulate out-of-order processing of events in some test.check specifications.



 Comments   
Comment by Jason Felice [ 07/Jun/15 1:31 PM ]

Corrected exponential.diff (previous one missing parenthesis)

Comment by Jason Felice [ 07/Jun/15 1:33 PM ]
user=> (require '[clojure.test.check.generators :as gen])
nil
user=> (gen/sample (gen/exponential 1/500))
(2377.5877207475132 320.68681344358293 144.18262694905863 1422.342802626106 300.8014248144241 79.8053977813952 36.18465309993865 93.65252446233575 806.4125368253829 37.88916712893676)
user=> (gen/sample (gen/exponential 1/500))
(61.431732383617884 93.51860239112023 557.6183626329674 57.553819177701584 60.4171805107351 7.152938152340775 31.84874217796871 215.71248467032976 470.69904975577214 222.89410284392252)
user=> (gen/sample (gen/exponential 1/500))
(231.68194071961852 74.5153426725424 135.57104449489148 48.2988483633354 566.3041916582175 925.9673009296939 169.65855270732143 914.1156710720605 1445.3554035600935 674.1395344157389)
user=> (gen/sample (gen/exponential 1/500))
(1467.6817487657213 1084.3796561112276 444.15866332358667 491.2566970512585 737.2979791960513 92.05590733338545 341.58867531092477 254.66969942333122 1150.8085459403007 111.21710097838688)




Generated at Tue Jul 07 21:30:20 CDT 2015 using JIRA 4.4#649-r158309.