<< Back to previous view

[TCHECK-54] shuffle assumes vector arg Created: 18/Nov/14  Updated: 18/Nov/14

Status: Open
Project: test.check
Component/s: None
Affects Version/s: None
Fix Version/s: None

Type: Defect Priority: Minor
Reporter: Steve Miner Assignee: Reid Draper
Resolution: Unresolved Votes: 0
Labels: None


 Description   

(gen/sample (gen/shuffle (range 3)))

ClassCastException clojure.lang.LazySeq cannot be cast to clojure.lang.IFn clojure.test.check.generators/swap

(gen/sample (gen/shuffle (vec (range 3)))) works



 Comments   
Comment by Steve Miner [ 18/Nov/14 9:20 AM ]

The easy fix is to change gen/swap...

(defn- swap
  [coll [i1 i2]]
  (let [v (vec coll)]
    (assoc v i2 (v i1) i1 (v i2))))

I can provide a patch if desired.

Comment by Steve Miner [ 18/Nov/14 9:25 AM ]

Or change the documentation to say that only a vector is allowed as opposed to a collection.





[TCHECK-53] property creating function similar to map as alternative to for-all* Created: 17/Nov/14  Updated: 17/Nov/14

Status: Open
Project: test.check
Component/s: None
Affects Version/s: None
Fix Version/s: None

Type: Enhancement Priority: Minor
Reporter: Steve Miner Assignee: Reid Draper
Resolution: Unresolved Votes: 0
Labels: None


 Description   

As an alternative to the `for-all*` syntax, I suggest using a function signature that looks more like `map` with the predicate as the first arg and the generators as the rest of the args.

For example, these forms would all be equivalent:

(property (comp integer? *) gen/int gen/int gen/int) ;; proposed style

(for-all* [gen/int gen/int gen/int] (comp integer? *)) ;; current style



 Comments   
Comment by Steve Miner [ 17/Nov/14 8:20 AM ]

It's easy enough to write for myself. I'll make a patch if you want it.

(defn property 
    ([pred gen] (prop/for-all* [gen] pred))
    ([pred gen1 gen2] (prop/for-all* [gen1 gen2] pred))
    ([pred gen1 gen2 & more] 
        (prop/for-all* (list* gen1 gen2 more) pred)))




[TCHECK-52] defspec in 0.6.0 doesn't allow symbolic options Created: 15/Nov/14  Updated: 17/Nov/14  Resolved: 17/Nov/14

Status: Resolved
Project: test.check
Component/s: None
Affects Version/s: None
Fix Version/s: None

Type: Defect Priority: Major
Reporter: Steve Miner Assignee: Reid Draper
Resolution: Completed Votes: 0
Labels: None

Attachments: Text File TCHECK-52-fix-defspec-options.patch    
Patch: Code and Test

 Description   

The new 0.6.0 release has a `defspec` that no longer allows symbolic or calculated options. Basically, the macro was improperly defined to require literals as options. In previous releases, the number of trials for a `defspec` could be a regular Clojure expression that was evaluated at runtime. My project broke on the new test.check because I was using my own var to define the number of trials for all my defspecs. Easy enough to work around once I understood the issue, but it's still needlessly backward incompatible.



 Comments   
Comment by Steve Miner [ 15/Nov/14 6:54 PM ]

fix for defspec

Comment by Steve Miner [ 15/Nov/14 6:55 PM ]

patch and test attached

Comment by Reid Draper [ 15/Nov/14 11:09 PM ]

Thanks Steve, this is definitely a bug. I'll review the patch in the morning, and get a 0.6.1 release out ASAP.

Comment by Steve Miner [ 16/Nov/14 5:55 AM ]

clean up patch

Comment by Reid Draper [ 16/Nov/14 12:28 PM ]

It will probably take some time to make it up to maven central, but 0.6.1 has been released with this patch. Thanks!

Comment by Steve Miner [ 16/Nov/14 4:01 PM ]

version 0.6.1 fixed my problem. Thanks.

Comment by Reid Draper [ 17/Nov/14 10:21 PM ]

Fixed in 30673274b035d78163efa83878515ced67eee474.





[TCHECK-51] Add set generator Created: 12/Nov/14  Updated: 15/Nov/14

Status: Open
Project: test.check
Component/s: None
Affects Version/s: None
Fix Version/s: None

Type: Enhancement Priority: Major
Reporter: Daniel Compton Assignee: Reid Draper
Resolution: Unresolved Votes: 0
Labels: None


 Description   

test.check has generators for maps, vectors, and lists. Can we also add sets as a generator with the same arguments as vector?



 Comments   
Comment by Reid Draper [ 15/Nov/14 1:53 PM ]

There are a few subtleties to adding a set generator with the same arguments as vector, particularly, asking for a particular size set. For example, what should the set generator do with the following arguments: gen/set gen/boolean 5? The domain of the boolean generator is only two elements, yet we've asked for a set with 5 elements. A vector can have duplicates, so this is no problem.

In the meantime, a set can be created:

(defn set-gen
  [vector-gen]
  (gen/fmap set vector-gen))




[TCHECK-50] Sampling one in the obvious way could return a more random value Created: 06/Nov/14  Updated: 06/Nov/14

Status: Open
Project: test.check
Component/s: None
Affects Version/s: None
Fix Version/s: None

Type: Enhancement Priority: Major
Reporter: Jessica Kerr Assignee: Reid Draper
Resolution: Unresolved Votes: 0
Labels: generator


 Description   

To get values from a generator I need to call gen/sample. This returns a sequence of values of increasing size, starting from 0.

From the outside, the obvious way to get one value from a generator is (first (gen/sample g)). This always produces a value of generator's size 0. If it's a vector or string, it's empty. If it's an integer, it's 0.

This is not documented or obvious.

There are many uses for generators outside of testing. At work we use them to populate test databases. And for example tests that are in the process of becoming property tests but not there yet. Let's make that harder to screw up.

suggestions: 1) specify this in docs around sample
2) either reverse sample OR
3) create a sample-one function that provides one value with a size like the default max size in checked properties (currently 200).



 Comments   
Comment by Jessica Kerr [ 06/Nov/14 5:09 PM ]

didn't mean to make it Major. Three attempts to resubmit later...

Comment by Andy Fingerhut [ 06/Nov/14 11:59 PM ]

Jessica, you only get one chance to pick the ticket attributes when you create one, unless you have been granted permission to edit them. This permission is given to people who have signed a Clojure Contributor Agreement: http://clojure.org/contributing

No worries, though. Those responding to tickets can easily modify them if they wish.





[TCHECK-49] StackOveflowError when generating large vectors Created: 03/Nov/14  Updated: 17/Nov/14  Resolved: 17/Nov/14

Status: Resolved
Project: test.check
Component/s: None
Affects Version/s: None
Fix Version/s: None

Type: Defect Priority: Major
Reporter: James Aley Assignee: Reid Draper
Resolution: Completed Votes: 0
Labels: None
Environment:

java version "1.7.0_71"
OpenJDK Runtime Environment (IcedTea 2.5.3) (Arch Linux build 7.u71_2.5.3-1-x86_64)
OpenJDK 64-Bit Server VM (build 24.65-b04, mixed mode)

Clojure 1.6.0
test.check 0.5.9


Attachments: Text File TCHECK-49-p1.patch    

 Description   

If you attempt to generate a vector with tens of thousands of elements, you'll get a StackOverflowError:

(gen/sample (gen/vector gen/int 1 20000))
;; StackOverflowError   clojure.test.check.generators/gen-bind/fn--10435 (generators.clj:77)

Because it's fun to be meta, here's a test.check test to reproduce it

(defspec does-not-overflow 1000
  (prop/for-all [max (gen/fmap (partial apply *)
                               (gen/tuple gen/s-pos-int gen/s-pos-int))]
                (< 0 (count (gen/sample (gen/vector gen/int 1 max))))))

I'm generating the the max parameter with that product, because it seems gen/s-pos-int doesn't actually generate large enough values to produce the error by itself.

Further, I'm actually just looking for a way to generate two things:

  • Byte arrays under 250kb
  • Bute arrays over 250kb

My app has different expected behaviour for each case. I was going about this by generating vectors and fmapping byte-array over it, but hit this overflow issue. I couldn't see a nice way to just generate data with the given size constraints directly. Perhaps I missed something? This is actually an issue I have quite often with test.check – integers in a particular range, strings of some minimum or maximum length, etc. Am I doing it wrong?



 Comments   
Comment by James Aley [ 03/Nov/14 12:06 PM ]

Oh - ignore my question about integers - I just found gen/choose

Comment by Reid Draper [ 08/Nov/14 2:04 PM ]

Thanks for filing this issue. First, the StackOverflow is indeed a problem. I believe I've figured out the root cause, but figuring out an acceptable fix may take some time. However, I believe I have some workarounds for you.

It sounds like your function/system has a magic number at 250Kb. I've run into some tests like this before, and what I've been able to do is to actually make this magic number parameterizable, at least for tests. During testing, could you just set the magic number to say, 250 bytes? This will work around the stack overflow issue, and make your tests both faster and less resource-intensive.

If you do end up needing to generate large values, you can get around this by simply combining several smaller generators into a large one. The stack overflow only happens when one collection generator is large. So instead of:

(def my-gen (gen/vector gen/int 10000))

try:

(def my-gen (gen/fmap flatten (gen/vector (gen/vector gen/int 100) 100)))

You're just flattening together 100 one-hundred element vectors, effectively doing the same thing.

Does this help?

Comment by Gary Fredericks [ 16/Nov/14 1:26 PM ]

I'm thinking the cleanest thing to do is to abandon the generic impl of sequence and replace it with smarter seq-of-gens->gen-of-seqs code (since that's all sequence is used for anyhow). Any objections to this?

Comment by Reid Draper [ 16/Nov/14 1:29 PM ]

I have no objection to that.

Comment by Gary Fredericks [ 16/Nov/14 1:33 PM ]

Patch is forthcoming.

Comment by James Aley [ 16/Nov/14 1:38 PM ]

Reid - sorry, the activity on this thread just reminded me that I still owe you a reply!

Yes, you're absolutely right about the magic number. I did end up refactoring the code to make it a parameter, and so the work-arounds weren't necessary.

Thanks for the great work on test.check, by the way. I'm really enjoying breaking my code with it!

Comment by Gary Fredericks [ 16/Nov/14 1:58 PM ]

Attached TCHECK-49-p1.patch.

Comment by Reid Draper [ 17/Nov/14 10:23 PM ]

Resolved in 40ccf07085bf987dd977306a3d9b69d8ce4a18ca.





[TCHECK-48] Add sublist generator Created: 28/Oct/14  Updated: 28/Oct/14

Status: Open
Project: test.check
Component/s: None
Affects Version/s: None
Fix Version/s: None

Type: Enhancement Priority: Minor
Reporter: Reid Draper Assignee: Reid Draper
Resolution: Unresolved Votes: 0
Labels: None





[TCHECK-47] tuple generator accepts bad arguments, then throws NullPointerException when sampled Created: 13/Oct/14  Updated: 28/Oct/14  Resolved: 28/Oct/14

Status: Resolved
Project: test.check
Component/s: None
Affects Version/s: None
Fix Version/s: None

Type: Enhancement Priority: Trivial
Reporter: Alistair Roche Assignee: John Walker
Resolution: Completed Votes: 0
Labels: None
Environment:

[org.clojure/test.check "0.5.9"]
org.clojure/clojure "1.6.0"]


Attachments: Text File TCHECK-47-p1.patch    

 Description   

test-reqs.generate=> (gen/sample (gen/tuple "asdf"))

test-reqs.generate=> (print-stack-trace *e 3)
java.lang.NullPointerException:
clojure.test.check.generators/gen-bind/fn generators.clj: 77
clojure.test.check.generators/gen-bind/fn generators.clj: 79
clojure.test.check.generators/gen-bind/fn generators.clj: 77
test-reqs.generate=>



 Comments   
Comment by John Walker [ 18/Oct/14 8:33 PM ]

Bad things can be passed to all generators, and they'll do the same thing when they're sampled.

Comment by John Walker [ 18/Oct/14 9:29 PM ]

We could address this by adding :pre and :post conditions where generators are expected.

This is similar to http://dev.clojure.org/jira/browse/TCHECK-46

(defn tuple
  "Create a generator that returns a vector, whose elements are chosen
  from the generators in the same position. The individual elements shrink
  according to their generator, but the value will never shrink in count.

  Examples:

      (def t (tuple gen/int gen/boolean))
      (sample t)
      ;; => ([1 true] [2 true] [2 false] [1 false] [0 true] [-2 false] [-6 false]
      ;; =>  [3 true] [-4 false] [9 true]))
  "
  [& generators]
  {:pre [(every? generator? generators)]}
  (gen-bind (sequence gen-bind gen-pure generators)
            (fn [roses]
              (gen-pure (rose/zip core/vector roses)))))

Usage:

(ns experimental-clojure.tcheck.jira49
  (:require [clojure.test.check :as tc]
            [clojure.test.check.generators :as gen]
            [clojure.test.check.properties :as prop]))

(gen/tuple 5)
CompilerException java.lang.AssertionError: Assert failed: (every? generator? generators), compiling:(/home/john/development/experimental-clojure/src/experimental_clojure/tcheck/jira47.clj:4:55) 
Comment by Reid Draper [ 21/Oct/14 9:55 PM ]

I'd definitely be interested in a patch for this. If not, I'd be happy to add it myself, just may be a little while.

Comment by Gary Fredericks [ 21/Oct/14 10:17 PM ]

John Walker did you assign this to yourself because you're working on a patch?

I'll throw one together if not.

Comment by John Walker [ 21/Oct/14 10:25 PM ]

Yes. I don't mind if you take it, though.

Comment by Gary Fredericks [ 25/Oct/14 9:43 AM ]

Uploaded TCHECK-47-p1.patch.

Comment by John Walker [ 28/Oct/14 10:05 PM ]

Looks good to me.

Comment by Reid Draper [ 28/Oct/14 10:43 PM ]

Thanks! Applied in 8ae6a077b6b0d22d89b37c6108fbd323b53bd2d6.





[TCHECK-46] Better error message when accidentally using something else as a generator Created: 07/Oct/14  Updated: 28/Oct/14  Resolved: 28/Oct/14

Status: Resolved
Project: test.check
Component/s: None
Affects Version/s: None
Fix Version/s: None

Type: Enhancement Priority: Major
Reporter: Gary Fredericks Assignee: Reid Draper
Resolution: Completed Votes: 0
Labels: None


 Description   

Currently if you accidentally use something else as a generator the error is a NullPointerException:

(gen/sample 42) ;; => throws NullPointerException

This could either be fixed with explicit type checking in the base functions that take generators, or alternatively be converting those same functions into a protocol and letting the normal protocol dispatch exceptions do the work.



 Comments   
Comment by Reid Draper [ 21/Oct/14 9:57 PM ]

See TCHECK-47http://dev.clojure.org/jira/browse/TCHECK-47 too. A precondition seems like a reasonable fix.

Comment by Reid Draper [ 28/Oct/14 10:48 PM ]

Fixed in 78b7cf8b3fbf6f96c3d5c706134fb67c40dcbb7e.





[TCHECK-45] Add ns to such-that example in README.md Created: 22/Sep/14  Updated: 24/Sep/14  Resolved: 24/Sep/14

Status: Resolved
Project: test.check
Component/s: None
Affects Version/s: None
Fix Version/s: None

Type: Defect Priority: Trivial
Reporter: Ryan Fowler Assignee: Reid Draper
Resolution: Completed Votes: 0
Labels: None

Attachments: File add-ns-for-such-that.diff    
Patch: Code

 Description   

The clojure.test integration example is missing a "gen/" in front of the such-that function call.

The attached patch just makes it slightly easier to copy/paste the example into a project.



 Comments   
Comment by Reid Draper [ 24/Sep/14 11:23 AM ]

Thanks. I ended up just pushing a commit to use gen/not-empty, which is a convenience function. Fixed in e54646f667af10034cdc637adeb96df3948b2b2c.





[TCHECK-44] for-all should support nesting Created: 22/Sep/14  Updated: 28/Oct/14

Status: Open
Project: test.check
Component/s: None
Affects Version/s: None
Fix Version/s: None

Type: Enhancement Priority: Major
Reporter: Michael Sperber Assignee: Reid Draper
Resolution: Unresolved Votes: 0
Labels: None

Attachments: File for-alls-nest.diff    
Patch: Code and Test

 Description   

Haskell QuickCheck allows for-all expressions to nest. This is useful when there are dependencies between generated values. test.check should allow this, too.

Currently, nested for-alls always succeed, which is somewhat pernicious.

I've added a patch that implements this.



 Comments   
Comment by Reid Draper [ 24/Sep/14 11:32 AM ]

Thanks Michael. I appreciate the patch, but there's a few design details that could be discussed before we get to code-level detail. As a separate, but related issue, I've been wanting to implement something like your CheckResult type, but as a record with several more fields. These fields would hold things like a plain-text description of any errors found, statistics (ala QuickCheck's collect, classify, etc.). I'd also like to write a protocol that allows basic types like booleans to be turned into this record. This would be analogous to Haskell QuickCheck's Testable Type Class. While this is technically a separate issue, I think it would behoove us to solve it in conjunction with nestable for-alls, particularly since nested for-alls can be simulated by just using bind at the generator level. Does this make sense?

Comment by Michael Sperber [ 25/Sep/14 2:19 AM ]

Absolutely.

I personally would start this patch, and work from there, unless you want to do things fundamentally differently rather than add more stuff.

Either way, how can I help make it happen?

Comment by Reid Draper [ 25/Sep/14 11:10 PM ]

Great question. Let me think on that and get back to you ASAP. I'd also love to make this happen soon.

Comment by Reid Draper [ 21/Oct/14 10:23 PM ]

Sorry for the delay, here's my sketch I've been working with:

diff --git a/src/main/clojure/clojure/test/check/properties.clj b/src/main/clojure/clojure/test/check/properties.clj
index 99b5222..139ae9a 100644
--- a/src/main/clojure/clojure/test/check/properties.clj
+++ b/src/main/clojure/clojure/test/check/properties.clj
@@ -8,13 +8,47 @@
 ;   You must not remove this notice, or any other, from this software.
 
 (ns clojure.test.check.properties
+  (:import clojure.test.check.generators.Generator)
   (:require [clojure.test.check.generators :as gen]))
 
+(defrecord Result [result pass? message stamps])
+
+(defprotocol ToResult
+  (to-result [a]))
+
+(extend java.lang.Object
+  ToResult
+  {:to-result (fn [b]
+               ;; not checking for caught exceptions here
+               (->Result b (not (false? b)) nil nil))})
+
+(extend nil
+  ToResult
+  {:to-result (fn [b]
+               (->Result b false nil nil))})
+
+(extend java.lang.Boolean
+  ToResult
+  {:to-result (fn [b]
+               (->Result b b nil nil))})
+
+(extend Generator
+  ToResult
+  {:to-result identity})
+
+(extend Result
+  ToResult
+  {:to-result identity})
+
+(defn message
+  [m property]
+  (assoc property :message m))
+
 (defn- apply-gen
   [function]
   (fn [args]
-    (let [result (try (apply function args) (catch Throwable t t))]
-      {:result result
+    (let [result (to-result (try (apply function args) (catch Throwable t t)))]
+      {:result (:result result)
        :function function
        :args args})))
 
@@ -29,9 +63,18 @@
   (for-all* [gen/int gen/int] (fn [a b] (>= (+ a b) a)))
   "
   [args function]
-  (gen/fmap
-    (apply-gen function)
-    (apply gen/tuple args)))
+  (gen/bind
+    (apply gen/tuple args)
+    (fn [a]
+      (let [result ((apply-gen function) a)]
+        (cond (gen/generator? result) (gen/fmap (fn [r] (println "foo") (update-in r :args #(conj % a))) result)
+              ;; NOTE: quick note to myself before I leave this code for the night,
+              ;; this :else is getting hit because we're wrapping the result
+              ;; with a {:result ...} map. Should probably do that conditionally.
+              ;; We also need two result types I think, a result to return from
+              ;; a property itself, and a reuslt that tacks the 'args' on top of this.
+              :else (do (println "bar") (gen/return result)))))
+    ))
 
 (defn binding-vars
   [bindings]
Comment by Michael Sperber [ 22/Oct/14 2:00 AM ]

Looks OK. However, it's difficult to see why that would get you more quickly where you said you want to go than my patch ...

Comment by Reid Draper [ 28/Oct/14 10:55 PM ]

Looks OK. However, it's difficult to see why that would get you more quickly where you said you want to go than my patch ...

Fair enough. Part of this it that it was easier for me to write up a sketch. The main things I'm trying to cover when supporting nested generators are making sure:

  1. We also support the upcoming ability to collect and return statistics about a test, ala collect from Haskell QuickCheck
  2. We have a sane way of returning failing tests to a user. Right now, in the :fail and :smallest keys of the returned map, we tell the user the failing arguments. They're always wrapped in at least one vector, since you may use more than one generator using prop/for-all. What do we do with nested properties? How do we distinguish between multiple generators at the 'same level', vs nested properties? Or do we not need to distinguish? Can whatever we decide to do be backward compatible?

Point being, I want to make sure we're not committing ourselves to nested-properties until we have some of those answers, and for me personally, it's easier to try and play with these things together, and see how they will fit together.





[TCHECK-43] Test.check lacks a generator for all floats and floats in a range Created: 17/Sep/14  Updated: 17/Sep/14

Status: Open
Project: test.check
Component/s: None
Affects Version/s: None
Fix Version/s: None

Type: Enhancement Priority: Minor
Reporter: Adam Clements Assignee: Reid Draper
Resolution: Unresolved Votes: 0
Labels: enhancement

Attachments: File float-generators.diff    
Patch: Code and Test

 Description   

Test.check lacks generators for floats and float ranges. I have implemented a float generator which shrinks to zero based on gen/ratio and also a float-in-range which is bounded by size.






[TCHECK-42] Interruptibility Created: 03/Sep/14  Updated: 21/Oct/14

Status: Open
Project: test.check
Component/s: None
Affects Version/s: None
Fix Version/s: None

Type: Enhancement Priority: Minor
Reporter: Gary Fredericks Assignee: Reid Draper
Resolution: Unresolved Votes: 1
Labels: None

Attachments: Text File TCHECK-42.patch    

 Description   

Both the main test loop and the shrink loop should have checks for thread interrupts

Patch coming imminently.



 Comments   
Comment by Gary Fredericks [ 03/Sep/14 3:55 PM ]

Attached TCHECK-42.patch.

Comment by Reid Draper [ 21/Oct/14 9:54 PM ]

Thanks, sorry for the delay. Maybe this is a lein repl thing, but when I ctrl-c a test, while using this patch, I get a java.lang.ThreadDeath exception as the :result. Maybe the repl spawns your code in another thread, unlike running lein test?





[TCHECK-41] Add helpers for deprecating generators Created: 01/Sep/14  Updated: 01/Sep/14

Status: Open
Project: test.check
Component/s: None
Affects Version/s: None
Fix Version/s: None

Type: Enhancement Priority: Minor
Reporter: John Walker Assignee: Reid Draper
Resolution: Unresolved Votes: 0
Labels: None

Attachments: File deprecation-helpers.diff    
Patch: Code

 Description   

It should be clearly communicated to the user when generators are deprecated. This patch supplies helpers for easily deprecating generators.

def-throwing-generator receives the metadata that its generator will throw if its called. As an example, suppose deprecated-generator was deprecated in 0.5.9.

(def-throwing-generator deprecated-generator
  "deprecated-generator has been deprecated. Use current-generator instead."
  {:deprecated "0.5.9"})

Then if the user looked up its documentation, they would see from cider:

clojure.test.check.generators/deprecated-generator
Deprecated in 0.5.9
  deprecated-generator has been deprecated. Use current-generator instead.

or if they used the generator and ran cider-test, they would see:

Test Summary
foobar.core-test

Ran 1 tests, in 1 test functions
1 errors


Results

Error in a-test
FIXME, I fail.
expected: (= "a" (sample deprecated-generator 5))
  actual: clojure.lang.ExceptionInfo: deprecated-generator has been deprecated. Use current-generator instead. {:deprecated "0.5.9"}





[TCHECK-40] Typo: alpha-numeric should be alphanumeric Created: 25/Aug/14  Updated: 17/Nov/14  Resolved: 17/Nov/14

Status: Resolved
Project: test.check
Component/s: None
Affects Version/s: None
Fix Version/s: None

Type: Defect Priority: Trivial
Reporter: John Walker Assignee: Reid Draper
Resolution: Completed Votes: 0
Labels: None

Attachments: File alphanumeric2.diff    
Patch: Code and Test

 Description   

This replaces all instances of alpha-numeric with alphanumeric. This includes renaming the following functions:

char-alpha-numeric -> char-alphanumeric
string-alpha-numeric -> string-alphanumeric



 Comments   
Comment by Reid Draper [ 26/Aug/14 2:22 PM ]

test.check hasn't reached 1.0, so we can break backward compatibility. However, I want to still communicate clearly to users. Maybe we can leave a definition of the hyphenated versions that raises a 'deprecated/removed' exception?

Comment by John Walker [ 27/Aug/14 10:44 AM ]

This seems reasonable. Are generators of this form alright?

(defn make-deprecated-generator
  ([msg map]
     (make-gen
      (fn [_ _]
        (throw (ex-info msg map)))))
  ([msg map cause]
     (make-gen
      (fn [_ _]
        (throw (ex-info msg map cause))))))

(def
  ^{:deprecated "TODO"}
  char-alpha-numeric
  "DEPRECATED - use char-alphanumeric instead."
  (make-deprecated-generator "DEPRECATED - use char-alphanumeric instead." {:deprecated "TODO"}))

If so, I can supply a macro to create these things. It would attach the metadata like docstrings and the deprecation version to the var, and usage might look like

(def-deprecated-generator char-alpha-numeric 
  "DEPRECATED - use char-alphanumeric instead."
  {:deprecated "TODO"})
Comment by Reid Draper [ 01/Sep/14 10:31 PM ]

I think I may be changing my mind here. I'm thinking we should have one release where the two names are both present. Make some noise about it in the release notes, and attach the :deprecated tag on the metadata. Would you mind making a patch to just add the new names, and add a ^{:deprecated ...} tag on the existing names? Sorry for the back and forth.

Comment by John Walker [ 02/Sep/14 2:11 PM ]

Thats fair. How should we handle it after that release?

Comment by John Walker [ 02/Sep/14 4:26 PM ]

This patch replaces alpha-numeric with alphanumeric, and attaches :deprecated "0.6.0" to char-alpha-numeric and string-alpha-numeric in favor of char-alphanumeric and string-alphanumeric.

Comment by Reid Draper [ 02/Sep/14 7:59 PM ]

Patch applied in 6def75ebc80cc5c1ab66ae956c76a7b402198852.





[TCHECK-39] such-that stops after 10 tries Created: 23/Aug/14  Updated: 26/Aug/14  Resolved: 26/Aug/14

Status: Resolved
Project: test.check
Component/s: None
Affects Version/s: None
Fix Version/s: None

Type: Defect Priority: Major
Reporter: Paul Lange Assignee: Reid Draper
Resolution: Declined Votes: 0
Labels: None
Environment:

clojure 1.5.1, test.check 0.5.9



 Description   

The documentation for such-that states that "it will keep trying forever" till it finds values that satisfy f.

I use it with a prime? predicate to generate primes: (such-that prime? gen/pos-int)

This sometimes results in this error when running a tc/quick-check which contradicts the docs:
ExceptionInfo Couldn't satisfy such-that predicate after 10 tries. clojure.core/ex-info (core.clj:4327)



 Comments   
Comment by John Walker [ 25/Aug/14 5:27 PM ]

Where do you see this in the documentation?

https://github.com/clojure/test.check/blob/d6edf70cfa6cb20316f9377edf9630772ccef969/src/main/clojure/clojure/test/check/generators.clj#L268

"By default, `such-that` will try 10 times to generate a value that
satisfies the predicate. If no value passes this predicate after this number
of iterations, a runtime exception will be throw." [sic]

Comment by Paul Lange [ 26/Aug/14 2:52 AM ]

Thanks, saw it here (which seem out of date): https://clojure.github.io/test.check/clojure.test.check.generators.html#var-such-that

Comment by Reid Draper [ 26/Aug/14 2:19 PM ]

I've just updated that page. Apologies on it being out of date. Would be nice to have a little version selector thing on it, so you can know you're viewing docs for the version of test.check that you use.





[TCHECK-38] Generators for functions? Created: 15/Aug/14  Updated: 27/Aug/14

Status: Open
Project: test.check
Component/s: None
Affects Version/s: None
Fix Version/s: None

Type: Enhancement Priority: Major
Reporter: Michael Sperber Assignee: Reid Draper
Resolution: Unresolved Votes: 0
Labels: None


 Description   

test.check has generators for first-order types, but not for functions.

In the original QuickCheck, an additional type class ("Arbitrary") handles
functions.

If I implemented this and submitted a patch, would it be likely to be included?

Background: I wrote my own Clojure QuickCheck clone here:

https://bitbucket.org/sperber/deinprogramm.quickcheck

This has arbitraries and function generators, but not all of test.check's goodies.

I'm trying to decide whether to continue maintaining my QuickCheck clone or
instead to contribute ot test.check, so feedback would be much appreciated.



 Comments   
Comment by Reid Draper [ 21/Aug/14 2:16 PM ]

I'd love a patch for this! Would you like to write up a quick implementation proposal first, so we can iron out any architectural details before looking at the code?

Comment by Michael Sperber [ 25/Aug/14 2:06 AM ]

Would you be willing to look at my QuickCheck clone as a starting point? Its general implementation approach is very similar to yours, so I think you should feel right at home.

The caveat is that it introduces a type separate from generators - the "arbitrary", corresponding to the `Arbitray' type class in the Haskell code. Doing this to `test.check' would change the API, so maybe there, the `Generator' type should be extended by an optional `transformer' field.

Comment by Reid Draper [ 26/Aug/14 2:26 PM ]

Sure, I've already read through it a bit, but need to read in more detail. How does your implementation handle printing (showing) functions? Do functions shrink? Have you seen Showing and Shrinking Functions by Claessen ?

Comment by Michael Sperber [ 27/Aug/14 2:21 AM ]

Yes. Haven't done that yet, but it would be on my list. Given that functions are objects in Clojure, I think printing should be a little easier than in Haskell.





[TCHECK-37] Make random-number generation in test.check thread-safe Created: 15/Aug/14  Updated: 05/Sep/14

Status: Open
Project: test.check
Component/s: None
Affects Version/s: None
Fix Version/s: None

Type: Enhancement Priority: Major
Reporter: Michael Sperber Assignee: Reid Draper
Resolution: Unresolved Votes: 0
Labels: enhancement

Attachments: File your-patch-file.diff    
Patch: Code and Test

 Description   

test.check uses the Java random-number generator, which is imperative.

This makes the testing non-deterministic in many settings.

It would be better if test.check used a purely functional rng, just
like the original Haskell version of QuickCheck.

I've attached a patch that does this. All but one of the original test
pass; I've modified the only failing test to not rely on an imperative
rng.

Note that it still has quick-check/make-rng supply a random seed.
IMHO, it should supply a fixed seed.



 Comments   
Comment by Reid Draper [ 15/Aug/14 5:45 PM ]

Thanks for the patch! I haven't had a chance to look it over in detail, but I do have a few introductory questions. You're correct that test.check currently uses a mutable RNG. While it may not be ideal, I haven't run into any user-facing cases where the use is non-deterministic. Do you have some particular example in mind where test.check is not deterministic now, given the same RNG seed? One place this could be an issue in the future is if we changed the way we walk the shrink tree, which would cause different values to be generated. I'd be keen on fixing this, but it's not an issue at the moment.

tl;dr: Under what circumstances, given the same seed, is test.check non-deterministic?

Comment by Michael Sperber [ 20/Aug/14 6:50 AM ]

Thanks for the prompt answer, and sorry for taking so long to reply.

The answer to your question is: None I'm currently encountering.

My motivation for the patch was a bit different, though:

1. I don't see an easy way to make things deterministic with defspec, running tests from, say, Leiningen.

2. While I can get reproducibility specifying the seed explicitly, this complicates the testing process, which should be as smooth as possible.

3. With the purely-functional rng, the generator monad is splittable: This enables me to write deterministic tests for concurrent programs.

#1 and #2 could also be addressed with the imperative RNG, not #3, though.

Comment by Reid Draper [ 21/Aug/14 2:15 PM ]

I don't see an easy way to make things deterministic with defspec, running tests from, say, Leiningen.

Indeed, this is simply an omission in the defspec macro. Calling the quick-check function directly allows you to pass in a seed. You can see the tests for this.

While I can get reproducibility specifying the seed explicitly, this complicates the testing process, which should be as smooth as possible.

This is how Haskell QuickCheck works too. If you don't specify a seed, tests are random. Would you prefer to get the same result, even without explicitly supplying a seed?

With the purely-functional rng, the generator monad is splittable: This enables me to write deterministic tests for concurrent programs.

You can write tests for concurrent programs now. The generated values are completely constructed before your code that may use threads is called. What am I missing?

Comment by Michael Sperber [ 25/Aug/14 2:02 AM ]

Ah, I didn't realize the shipped QuickCheck does this - sorry. But yes, I would greatly prefer to get the same result, unless explicitly instructed otherwise: This would solve the `defspec' problem.

Also, some test suites will pass or fail depending on the seed - particularly those involving floating-point may need a lot of test cases before they fail, and non-reproducibility might convince you that they pass - for a while.

You can write tests for concurrent programs now.

I was being less than clear: I meant concurrent tests for concurrent programs, where the concurrency is initiated by the test itself. In that case, you may want to call `quickcheck' from different threads, and would thus like to give each its own random-number generator.

Comment by Reid Draper [ 04/Sep/14 9:00 PM ]

Just noticed that newer Haskell QuickCheck uses this random number generator now .

Comment by Reid Draper [ 05/Sep/14 1:45 PM ]

Some more discussion of the problems with the proposed implementation:





[TCHECK-36] doc typo: 'excepts' should be 'accepts' Created: 05/Aug/14  Updated: 17/Nov/14  Resolved: 17/Nov/14

Status: Resolved
Project: test.check
Component/s: None
Affects Version/s: None
Fix Version/s: None

Type: Defect Priority: Trivial
Reporter: Steve Miner Assignee: Reid Draper
Resolution: Completed Votes: 0
Labels: None

Attachments: Text File typo-excepts-should-be-accepts.patch    
Patch: Code

 Description   

typo in doc/intro.md



 Comments   
Comment by Reid Draper [ 07/Aug/14 11:51 AM ]

Thanks! Pushed in d2f46ad2fba3e7363cc0f22f40272088f2f01eb5.





[TCHECK-35] Improve symbol and keyword generators Created: 11/Jul/14  Updated: 23/Jul/14  Resolved: 23/Jul/14

Status: Resolved
Project: test.check
Component/s: None
Affects Version/s: None
Fix Version/s: None

Type: Enhancement Priority: Major
Reporter: Eric Normand Assignee: Reid Draper
Resolution: Completed Votes: 0
Labels: enhancement, generator

Attachments: File namespaced-symbols-2014-07-11-14-38.diff     File namespaced-symbols-2014-07-23.diff    
Patch: Code

 Description   

Create a generator for symbols.

Also modify the keyword generator to allow :'s (as per the spec) (http://clojure.org/reader)

Create generators for namespaced symbols and keywords, as these were not tested before.

The new generators are added to gen/any.

A patch (namespaced-symbols-2014-07-11-14-38.diff) is attached.

Also a suggestion (but not in the patch): the edn-roundtrips test should be run separately on each type in gen/any to adequately search the space for errors.



 Comments   
Comment by Reid Draper [ 22/Jul/14 8:20 PM ]

Aside from +or--digit?, are all of the rest of the definitions in this patch intended to be public? It's a little unclear to me which of these are public generators and which are just helpers.

Comment by Eric Normand [ 23/Jul/14 2:06 PM ]

This patch is the same as the previous one with the non-interface functions set as private.

Patch: namespaced-symbols-2014-07-23.diff

Comment by Reid Draper [ 23/Jul/14 7:08 PM ]

Thanks! Merged in bc1650d4136d3453a6c16a363c1d5db36ad9a6b1.





[TCHECK-34] clojure.test reporting is uninformative Created: 21/Jun/14  Updated: 11/Aug/14

Status: Open
Project: test.check
Component/s: None
Affects Version/s: None
Fix Version/s: None

Type: Defect Priority: Major
Reporter: Philip Potter Assignee: Reid Draper
Resolution: Unresolved Votes: 0
Labels: None


 Description   

The clojure.test reporting results in output like:

FAIL in (always-fail) (clojure_test.clj:18)
expected: result
  actual: false

Note that the file and line number are always clojure_test.clj:18 and the expected value is always the literal string "result".

I'm not sure what the right output would be for expected/actual, but the incorrect file and line reporting means that clojure-test-mode consistently highlights the wrong line.



 Comments   
Comment by Philip Potter [ 21/Jun/14 9:10 AM ]

I think this is because the assertion is checked by calling clojure.test/is. clojure.test/is is being called for its return value, but it has the side effect of reporting failures. It's a macro, and it takes the literal symbol "result" as its expected expression, and takes the current file and line number to attach to the failure. It's really designed to be called by user code, not library code.

Comment by John Walker [ 11/Aug/14 3:55 PM ]

Well, one solution would involve taking advantage of the report functionality.

https://github.com/clojure/clojure/blob/master/src/clj/clojure/test.clj#L204
https://github.com/clojure/clojure/blob/master/src/clj/clojure/test.clj#L371





[TCHECK-33] test.check is entangled with clojure.test Created: 21/Jun/14  Updated: 21/Jun/14

Status: Open
Project: test.check
Component/s: None
Affects Version/s: None
Fix Version/s: None

Type: Defect Priority: Minor
Reporter: Philip Potter Assignee: Reid Draper
Resolution: Unresolved Votes: 0
Labels: None


 Description   

test.check is tightly coupled to clojure.test; this means that it's difficult to use it with other test frameworks.

There is a dependency cycle between clojure.test.check and clojure.test.check.clojure-test.

quick-check uses clojure.test/report to print dots to the screen, meaning that quick-check necessarily depends on clojure.test



 Comments   
Comment by Philip Potter [ 21/Jun/14 8:54 AM ]

I wonder if the solution is to allow you to provide your own reporting functions to quick-check to report successes, failures, and trials? That way different frameworks can inject different behaviour as required.





[TCHECK-32] Default sizing on gen/any needs re-evaluation Created: 13/Jun/14  Updated: 04/Aug/14

Status: Open
Project: test.check
Component/s: None
Affects Version/s: None
Fix Version/s: None

Type: Defect Priority: Major
Reporter: Luke VanderHart Assignee: Reid Draper
Resolution: Unresolved Votes: 0
Labels: None


 Description   

The following innocuous-looking test blows the heap and then crashes a 4GB JVM with an out-of-memory error:

(tc/defspec merge-is-idempotent
  100
  (prop/for-all [m (gen/map gen/any gen/any)]
    (= m (merge m m))))

I understand how this happens, and how to fix it by adjusting the size parameters.

However, it would be great if using the defaults did not have the potential for such a nasty failure mode (particularly as the unwary user will have trouble determining that the fault is not in their code).



 Comments   
Comment by Philip Potter [ 21/Jun/14 8:29 AM ]

I was going to raise a separate ticket for this, but it seems like it might be related: gen/any (and any-printable) seem to take exponential time in the size of the input. See for example this session:

user> (time (dorun (gen/sample (gen/resize 50 gen/any-printable))))
"Elapsed time: 2204.284643 msecs"
nil
user> (time (dorun (gen/sample (gen/resize 55 gen/any-printable))))
"Elapsed time: 2620.717337 msecs"
nil
user> (time (dorun (gen/sample (gen/resize 60 gen/any-printable))))
"Elapsed time: 5923.636336 msecs"
nil
user> (time (dorun (gen/sample (gen/resize 65 gen/any-printable))))
"Elapsed time: 9035.762191 msecs"
nil
user> (time (dorun (gen/sample (gen/resize 70 gen/any-printable))))
"Elapsed time: 15393.687184 msecs"
nil
user> (time (dorun (gen/sample (gen/resize 75 gen/any-printable))))
"Elapsed time: 9510.571668 msecs"
nil
user> (time (dorun (gen/sample (gen/resize 80 gen/any-printable))))
"Elapsed time: 39591.543565 msecs"
nil

Apart from the anomaly at 75, adding 10 to the size seems to increase runtime by almost 3x.

Comment by Reid Draper [ 23/Jun/14 5:07 PM ]

I believe I should have a fix for this today, as well as an easier way to write custom, recursive generators.

Comment by Reid Draper [ 23/Jun/14 5:21 PM ]

This isn't fixing the OOM yet, but is making a big difference in making the size have a linear relationship with the number of elements generated, in a recursive generator. Here's branch: https://github.com/clojure/test.check/compare/feature;recursive-generator-helpers.

Comment by Reid Draper [ 23/Jun/14 7:16 PM ]

Fixed in https://github.com/clojure/test.check/commit/19ca756c95141af3fb9caa5e053b9d01120e5d7e. I'll try and get a snapshot build up tonight. Will be 0.5.9-SNAPSHOT. Check out the commit-message for a full explanation of how this now works.

Comment by Philip Potter [ 28/Jun/14 3:26 PM ]

awesome work! The new recursive-gen fn is great, too: I can immediately see a use case for generating arbitrary JSON-compatible data (ie vectors, maps, strings, numbers, bools, but no rationals, characters, symbols...).

Comment by Philip Potter [ 29/Jun/14 6:48 AM ]

Just re-tested on my machine; performance is vastly improved though still superlinear:

clojure.test.mode> (time (dorun (gen/sample (gen/resize 100 gen/any-printable))))
"Elapsed time: 101.907628 msecs"
nil
clojure.test.mode> (time (dorun (gen/sample (gen/resize 200 gen/any-printable))))
"Elapsed time: 302.341697 msecs"
nil
clojure.test.mode> (time (dorun (gen/sample (gen/resize 400 gen/any-printable))))
"Elapsed time: 1154.098163 msecs"
nil
clojure.test.mode> (time (dorun (gen/sample (gen/resize 800 gen/any-printable))))
"Elapsed time: 2954.889396 msecs"
nil
clojure.test.mode> (time (dorun (gen/sample (gen/resize 1600 gen/any-printable))))
"Elapsed time: 22335.200578 msecs"
nil

although since the default max-size is 200, this is very much no big deal.

Comment by Reid Draper [ 03/Jul/14 10:52 AM ]

I'm not too surprised that performance is still superlinear. What should be linear is the number of leaf nodes in the generated tree. We may eventually be able to do something a little more complex and have the total number of nodes be linear (including internal nodes). For now, however, it should be considered a bug if the relationship between leaf nodes doesn't grow linearly with the size parameter.

Comment by Reid Draper [ 04/Aug/14 8:35 PM ]

I'm still seeing that the original example is OOMing. I'm not sure the best way to solve that. For a single layer of nested generators, the above patch seems to be making a big difference. But when you make a map of gen/any, map doesn't know that it's arguments are themselves recursive generators. This means you might create 100 keys and values, each themselves, large recursive generators. There's a balance that has to be had between making the default recursive generators be 'large enough' by themselves, but not too large when used like this. Hmm.. I think I'll go ahead and release 0.5.9, and keep on thinking of ways to make this even better.





[TCHECK-31] Doc improvements: how to separate "size" from "range" Created: 11/Jun/14  Updated: 20/Nov/14

Status: Open
Project: test.check
Component/s: None
Affects Version/s: None
Fix Version/s: None

Type: Enhancement Priority: Major
Reporter: Michael Nygard Assignee: Reid Draper
Resolution: Unresolved Votes: 1
Labels: None


 Description   

Size is discussed mainly in terms of recursive generators. However, it is crucial to understanding other aspects such as vector length and integer ranges.

Some examples like these would be helpful:

  • How to get a small number of samples with a large range.
  • How to get a large number of samples with a small range.


 Comments   
Comment by Reid Draper [ 12/Jun/14 9:13 AM ]

Agree. I'll try and get a patch up in the next couple of days. Do you just have something in mind going in the doc/ dir?

Comment by Brian Craft [ 12/Nov/14 12:05 AM ]

I have exactly this problem. Are there any examples from projects using test.check?

Comment by Reid Draper [ 12/Nov/14 9:12 AM ]

Brian, can you be more specific about your problem? Some helpful functions regarding sizing and domain-generation are documented here. Take a look specifically at gen/choose, gen/resize, and gen/sized.

Comment by Brian Craft [ 12/Nov/14 10:29 AM ]

I don't need to test, for example, strings that are 500 chars long, but do need to test lists of 500 strings. If I have a generator that builds lists of strings & size it larger, both the lists and strings get larger. Then I start hitting irrelevant resource limits, like db field widths, or jvm memory. It seems like I need a way to specify how different data elements grow. Just setting the range, like (gen/vector ... 0 5000), seems wrong, because then it always picks up to 5000, while I suspect for shrinking it needs to respond to "size". Something like this?

(gen/sized (fn [size] (gen/resize (* size size) (gen/vector (gen/resize size gen/int)))))

forcing the size of the vector to increase more quickly than the size of the elements in the vector.

I gave this a try, but hit the vector stack overflow problem. I will try the fmap/flatten workaround, but am wondering if this is the right approach. Also wondering about the range of "size". From the code it looks like it goes to 100 due to sample-seq. So functions of size passed to gen/sized should expect range from 0-99, and return the largest meaningful data when given size parameter 99?

Comment by Reid Draper [ 15/Nov/14 12:03 PM ]

I don't need to test, for example, strings that are 500 chars long, but do need to test lists of 500 strings. If I have a generator that builds lists of strings & size it larger, both the lists and strings get larger.

Indeed, by default, all generators will 'grow' together. And as you've correctly noticed, you can use gen/sized and gen/resize to further control this.

Just setting the range, like (gen/vector ... 0 5000), seems wrong, because then it always picks up to 5000, while I suspect for shrinking it needs to respond to "size". Something like this?

Shrinking is completely separate from the size parameter. You can safely use (gen/vector ... 0 n) without negatively affecting shrinking.

(gen/sized (fn [size] (gen/resize (* size size) (gen/vector (gen/resize size gen/int)))))

forcing the size of the vector to increase more quickly than the size of the elements in the vector.

I gave this a try, but hit the vector stack overflow problem. I will try the fmap/flatten workaround, but am wondering if this is the right approach.

Your example looks fine. And you can avoid the stack overflow problem by simply not allowing size to grow above a certain limit, say 5000: (gen/resize (min 5000 (* size size)) ...).

(gen/sized (fn [size] (gen/resize (* size size) (gen/vector (gen/resize size gen/int)))))

forcing the size of the vector to increase more quickly than the size of the elements in the vector.

Also wondering about the range of "size". From the code it looks like it goes to 100 due to sample-seq. So functions of size passed to gen/sized should expect range from 0-99, and return the largest meaningful data when given size parameter 99?

When testing, the default max size is 200. However, you can override this simply by passing in a :max-size n argument to tc/quick-check. For example: (tc/quick-check 100 my-prop :max-size 75).

Hope this helps, and don't hesitate to ask anything else.

Comment by Brian Craft [ 18/Nov/14 11:41 AM ]

At the moment the hardest bit for me is understanding the shrinking. Composing generators, I inadvertently build things that won't shrink. E.g. to generate a 2d matrix of numbers with row & column names, I make use of bind so I can size vectors of names to match the matrix size, but then the resulting matrix/vectors will not shrink.

Here's a small example. Generate a vector of short strings, and test that they are all unique.

(tc/quick-check 40 (prop/for-all [f (gen/bind gen/int (fn [i] (gen/vector (gen/resize 5 (gen/such-that not-empty gen/string-ascii)) i)))] (do (println "f" f "count" (count f)) (= (count f) (count (set f))))))

Run this a few times & it will hit a duplicate string, failing the test. It is then unable to shrink the vector.

Here's a case generating two vectors of the same size by generating one vector, then using bind to generate a second of the same size. Testing uniqueness on the second vector, it is able to shrink, however it does so by regenerating the 2nd vector for every test.

(tc/quick-check 40 (prop/for-all [f (gen/bind (gen/vector (gen/resize 5 (gen/such-that not-empty gen/string-ascii))) (fn [v] (gen/hash-map :a (gen/return v) :b (gen/vector gen/int (count v)))))] (do (println "f" f "count" (count (:a f))) (= (count (:b f)) (count (set (:b f)))))))

I expect this will shrink poorly in many cases, because regenerating the second vector destroys the (possibly very rare) failure condition. I suspect this is why my 2d matrices shrink poorly. One really wants to be able to shrink by dropping the same positional element from both vectors. In the case of a fixed dimension (two), you can rewrite it to generate a vector of tuples. For a 2d matrix, though, I'm not sure what to do. Ideally it would shrink by dropping either a column or a row, and the corresponding column or row label. Perhaps I need to write a recursive generator like vector?

Are there any docs/papers describing how the shrinking works?

Comment by Gary Fredericks [ 20/Nov/14 9:45 AM ]

I've included a generator called cap-size in the test.chuck library.





[TCHECK-30] Make Namespacing Consistent in generators.clj Docstrings Created: 31/May/14  Updated: 17/Nov/14  Resolved: 17/Nov/14

Status: Resolved
Project: test.check
Component/s: None
Affects Version/s: None
Fix Version/s: None

Type: Enhancement Priority: Trivial
Reporter: Michael S. Daines Assignee: Reid Draper
Resolution: Completed Votes: 0
Labels: docstring

Attachments: Text File prefix.patch    
Patch: Code

 Description   

Three of the docs with examples in generators.clj aren't namespaced with "gen/", whereas everything else is. Add these in to make the documentation consistent and easy for people learning the library to cut and paste to test.






[TCHECK-29] Docstring typo Created: 31/May/14  Updated: 17/Nov/14  Resolved: 17/Nov/14

Status: Resolved
Project: test.check
Component/s: None
Affects Version/s: None
Fix Version/s: None

Type: Defect Priority: Trivial
Reporter: Gary Fredericks Assignee: Reid Draper
Resolution: Completed Votes: 0
Labels: None

Attachments: Text File docstring-typo.patch    

 Description   

Patch attached.



 Comments   
Comment by Reid Draper [ 17/Nov/14 10:28 PM ]

Fixed in 6c553aaa891e0b3c3bc1c6fba3791d75664c530e.





[TCHECK-28] gen/elements should work with arbitrary collections Created: 23/May/14  Updated: 23/May/14  Resolved: 23/May/14

Status: Resolved
Project: test.check
Component/s: None
Affects Version/s: None
Fix Version/s: None

Type: Defect Priority: Major
Reporter: Gary Fredericks Assignee: Reid Draper
Resolution: Completed Votes: 0
Labels: None

Attachments: Text File TCHECK-28.patch    

 Description   

gen/elements currently fails for any collection on which nth fails.

We can fix this by calling vec on the input, which has the added benefit of ensuring that the subsequent indexed lookups (one per generated value) are constant time.

A patch to that effect is imminent.



 Comments   
Comment by Gary Fredericks [ 23/May/14 8:52 AM ]

Added TCHECK-28.patch.

Comment by Reid Draper [ 23/May/14 4:52 PM ]

Applied in 1df5a635bcf195c2f98717da3929f6c0d832ee0e.





[TCHECK-27] stateful PRNG + bind => nondeterministic shrink tree Created: 10/May/14  Updated: 11/May/14

Status: Open
Project: test.check
Component/s: None
Affects Version/s: None
Fix Version/s: None

Type: Defect Priority: Major
Reporter: Gary Fredericks Assignee: Reid Draper
Resolution: Unresolved Votes: 0
Labels: None

Attachments: Text File TCHECK-27-p1.patch    

 Description   

bind inherently needs to consume randomness while shrinking, since the inner generator has to be run multiple times. Since the shrink tree is lazy and there's only one Random object providing the randomness, the resulting shrank values are sensitive to what order the tree is walked in.

I don't believe this affects normal usage, but there are various modifications I've made that I think are useful (e.g., resuming slow shrinks) that are thwarted by this nondeterminism.

Ideally we could stop using a stateful PRNG, but I think this would be a rather extreme change. A much smaller fix is to change bind to create a new Random object each time the inner function is called, based on a seed from the outer Random. I will have a patch that does this shortly, and it seems to fix the issues I've had.

Code for reproducing the issue:

(ns user
  (:require [clojure.test.check.generators :as gen]
            [clojure.test.check.rose-tree :as rose]))

(def the-gen
  (gen/bind gen/nat
            (fn [n]
              (gen/fmap (fn [m] [n m]) gen/nat))))

(defn reproduced?
  [seed]
  (let [make-rose-tree #(gen/call-gen the-gen (java.util.Random. seed) 100)
        rose-1 (make-rose-tree)
        rose-2 (doto (make-rose-tree)
                 ((fn [rt1]
                    ;; force the tree to be realized in a different way
                    (dorun (for [rt2 (rose/children rt1)
                                 rt3 (rose/children rt2)]
                             42)))))
        last-child #(-> % rose/children last rose/root)]
    ;; when these two are not equal, we've reproduced the problem
    (not= (last-child rose-1)
          (last-child rose-2))))

(frequencies (map reproduced? (range 10000)))
;; => {false 9952, true 48}


 Comments   
Comment by Gary Fredericks [ 10/May/14 2:16 PM ]

Attached TCHECK-27-p1, which makes the change I described in the description to the bind-helper function.

Comment by Gary Fredericks [ 11/May/14 8:27 AM ]

The only function that uses bind-helper is bind; however, bind is also implemented with gen-bind, and several other functions use gen-bind. So we'll have to check if gen-bind is sensitive to this as well (e.g., perhaps frequencies has this issue?). If so, I'm not sure what that means about the best way to fix it. Maybe there's an alternative fix that could happen in gen-bind, or maybe the other functions could be rewritten with bind?





[TCHECK-26] defspec does not respect (is (thrown? ...)) Created: 20/Apr/14  Updated: 13/May/14

Status: Open
Project: test.check
Component/s: None
Affects Version/s: None
Fix Version/s: None

Type: Defect Priority: Minor
Reporter: Sung Pae Assignee: Reid Draper
Resolution: Unresolved Votes: 0
Labels: None


 Description   

The following test does not pass:

(quick-check 1
  (for-all [x g/any]
    (is (thrown? Throwable (assert false)))))

This can be worked around by wrapping the (is ...) form with a call to boolean:

(quick-check 1
  (for-all [x g/any]
    (boolean (is (thrown? Throwable (assert false))))))

The (is (thrown? ...)) form does not return true|false, but will return the Exception itself when it passes.

Thank you!



 Comments   
Comment by Reid Draper [ 13/May/14 6:14 PM ]

This is correct. And I'm not yet sure what the best way forward is, but my hunch is that we'll need to support non-boolean return values from tests. Maybe something that implements a ->pass? function.





[TCHECK-25] shuffle generator Created: 17/Apr/14  Updated: 26/May/14

Status: Open
Project: test.check
Component/s: None
Affects Version/s: None
Fix Version/s: None

Type: Enhancement Priority: Major
Reporter: Gunnar Völkel Assignee: Reid Draper
Resolution: Unresolved Votes: 0
Labels: None


 Description   

Adding a generator that shuffles the output of a given collection generator would be really useful.
Short example:

(gen/sample (gen/shuffle (gen/vector get/nat))) ;=> ([5 2 4] [9 1 7 8] ...)

This could be used to add a generator for permutations of integers 1,2,...,n which I also missed several times already.



 Comments   
Comment by Reid Draper [ 13/May/14 6:17 PM ]

Yeah this would be cool. Should it shrink toward the original collection-ordering?

Comment by Gunnar Völkel [ 26/May/14 10:55 AM ]

Yes, that was the idea when we talked about that topic on #clojure. Otherwise, you can just use gen/int to create a seed and use java.util.Random together with java.util.Collections/shuffle.





[TCHECK-24] Add a retry-limited version of `such-that` Created: 17/Apr/14  Updated: 13/May/14  Resolved: 13/May/14

Status: Resolved
Project: test.check
Component/s: None
Affects Version/s: None
Fix Version/s: None

Type: Enhancement Priority: Minor
Reporter: Jean Niklas L'orange Assignee: Reid Draper
Resolution: Completed Votes: 0
Labels: None


 Description   

The docstring of such-that states that

Eventually we will add another generator combinator that only tries N times before giving up.

So I found it reasonable to add an issue just so it's not forgotten. Seems reasonable to have it throw a RuntimeException with an explanation that such-that attempted to do more than N tries.

Additionally, I think it would be valuable to have a runtime-error in the general such-that implementation, e.g. on 10000 retries, possibly with an explanation on how to override the default retry-limit. I've heard of many people (myself included) not being really sure where their infinite loop is within their test.check code, and I guess such-that is the culprit in very many cases. This might help debugging those issues, although it would break backwards compatibility.



 Comments   
Comment by Reid Draper [ 13/May/14 6:12 PM ]

Fixed in 719d64d60419638f22e3ceca5e37fe075df652ea. Should be in the next release.





[TCHECK-23] Clojure Keywords must begin with a non-digit. Created: 15/Apr/14  Updated: 16/Apr/14  Resolved: 16/Apr/14

Status: Resolved
Project: test.check
Component/s: None
Affects Version/s: None
Fix Version/s: None

Type: Defect Priority: Minor
Reporter: Eric Normand Assignee: Reid Draper
Resolution: Completed Votes: 0
Labels: None

Attachments: File better-keywords.diff    

 Description   

Per http://clojure.org/reader, Clojure Keywords must begin with a non-numeric character. Currently, test.check generates keywords from strings of alpha-numeric characters, and sometimes the first character is a digit. Although the current reader on the JVM allows it, the ClojureScript reader fails with keywords like :1, :2aa, etc.

Here is the current code:

(def keyword
"Generate keywords."
(->> string-alpha-numeric
(such-that #(not= "" %))
(fmap core/keyword)))

Proposed fix:

(def keyword
"Generate keywords."
(->> (tuple char-alpha string-alpha-numeric)
(fmap #(apply str %))))
(fmap core/keyword)))

char-alpha will need to be defined.

This would also alleviate the need for the such-that to filter out empty strings.



 Comments   
Comment by Reid Draper [ 15/Apr/14 8:03 PM ]

+1. Do you have a CA signed, Eric? And would you like to provide a patch? If not, I'd be happy to fix this.

Comment by Eric Normand [ 16/Apr/14 6:35 AM ]

My CA is filed with Cognitect.

I'll get a patch in today.

Comment by Eric Normand [ 16/Apr/14 9:21 AM ]

better-keywords.diff contains a patch to conform to the reader specs. All tests pass.

Comment by Reid Draper [ 16/Apr/14 10:22 PM ]

Merged in 4528b5ed391d6127b79f4db6bc5e086613da0d81

Comment by Reid Draper [ 16/Apr/14 10:22 PM ]

Thanks Eric!





[TCHECK-22] Move rose-tree code into a separate namespace. Created: 13/Apr/14  Updated: 13/Apr/14  Resolved: 13/Apr/14

Status: Resolved
Project: test.check
Component/s: None
Affects Version/s: None
Fix Version/s: None

Type: Enhancement Priority: Minor
Reporter: Gary Fredericks Assignee: Reid Draper
Resolution: Completed Votes: 0
Labels: None

Attachments: Text File rose-namespace.patch    

 Description   

The rose tree code is logically separate from the rest of the generator code, and is already pseudo-namespaced by the inclusion of the word "rose" in the function names.



 Comments   
Comment by Gary Fredericks [ 13/Apr/14 2:35 PM ]

Attached a patch that moves the rose-tree functions into their own namespace, and removes "rose" from the function names.

The latter part is a bit trickier, and a couple times involved naming collisions for t he names root and children which are now functions in the rose-tree namespace but are also often locals in that namespaces as well. I twice changed the locals to the-root or the-namespace to resolve the collision.

Comment by Reid Draper [ 13/Apr/14 4:58 PM ]

Merged in 9cc5cac08a9dc5ec0d327e12c502dea1f7741958. Thanks!





[TCHECK-21] Rerunning a particular failure is difficult. Created: 11/Apr/14  Updated: 12/Apr/14

Status: Open
Project: test.check
Component/s: None
Affects Version/s: None
Fix Version/s: None

Type: Enhancement Priority: Major
Reporter: Gary Fredericks Assignee: Reid Draper
Resolution: Unresolved Votes: 0
Labels: None


 Description   

Test.check has the concept of a seed that is used when a property is tested. The seed can be supplied or generated by the current time. Either way, the single seed is used to generate all the trials for the test. If a test fails, the only way to try to re-test the property on the same inputs is to run all of the tests again using the same seed. If the tests were running for hours before a failure was found, this is prohibitive.

I would like instead to be able to check the property for exactly the input that failed. One step in that direction is to have an explicit different seed for each trial (perhaps generated from an initial metaseed), and on failure to report a [seed size] pair that can be used to run a test a single time.

The way this interacts with shrinking is interesting though, since it's just as likely that I'll want to re-run on the shrunk value from a failure. Since the shrunk value was never explicitly generated from a known [seed size], just that information is insufficient. We could perhaps report [seed size shrink-path] where the third element is a list of indexes to walk through the shrink tree with. Probably the worst part about that is it might be rather long.

In any case I think something like this would be more useful than the current setup. I'll probably be hacking up something similar for a fork branch I'm maintaining, but if we can settle on a specific design I'd be happy to cook up a patch.



 Comments   
Comment by Reid Draper [ 12/Apr/14 10:28 AM ]

When a test fails, we do return the arguments that originally failed, and the arguments of the shrunk test. Is this insufficient?

For example:

{:result false,
       :failing-size 45,
       :num-tests 46,
       :fail [[10 1 28 40 11 -33 42 -42 39 -13 13 -44 -36 11 27 -42 4 21 -39]],
       :shrunk {:total-nodes-visited 38,
                :depth 18,
                :result false,
                :smallest [[42]]}}

see :fail and [:shrunk :smallest].

Comment by Gary Fredericks [ 12/Apr/14 11:00 AM ]

It's certainly adequate to reproduce in theory, but I don't know of any built in mechanism that makes this easy. Here's what I end up doing:

Given this:

(defspec foo
  (prop/for-all [x gen]
    (f x)))

and after getting a failure with a 50-line shrunk value (e.g. :bar), I manually pretty-print it and paste it back into my file like so:

(def shrank
  ':bar)

(defspec foo
  (prop/for-all [x #_gen (gen/return shrank)]
    (f x)))

And now I can run (foo 1) to re-run with the failing value.

This awkwardness is partly due to three facts:

  1. My generators create large unwieldy values
  2. My property is a large body of code rather than a single function
  3. There's no easy way to test a property on a specific value

I could mitigate some of the pain by fixing #2 – having each one of my tests split into a defspec / prop/for-all and a plain function. But #1 is not easy to fix, and having a small tuple is rather nicer for me.

I've already written a POC for this and it's working rather well. I used the term :key to refer to the tuple, so the term doesn't conflict with :seed. I end up calling the test like (foo 0 :key [329489249329323 19 []]), but I'll probably think up something else that doesn't require passing a dummy first arg.





[TCHECK-20] Code cleanup Created: 11/Apr/14  Updated: 12/Apr/14  Resolved: 12/Apr/14

Status: Resolved
Project: test.check
Component/s: None
Affects Version/s: None
Fix Version/s: None

Type: Task Priority: Trivial
Reporter: Gary Fredericks Assignee: Reid Draper
Resolution: Completed Votes: 0
Labels: None

Attachments: Text File cleanup.patch     Text File core-slash.patch     Text File flatten.patch    

 Description   

I've been doing some hacking on test.check, so just trying to leave things better than I found them.



 Comments   
Comment by Gary Fredericks [ 11/Apr/14 8:53 PM ]

Attached another patch that changes clojure.core/ to core/ in the generators namespace, in case you like that sort of thing.

Comment by Gary Fredericks [ 12/Apr/14 6:53 AM ]

Attached a third patch for removing a use of flatten.

Comment by Reid Draper [ 12/Apr/14 6:16 PM ]

Thanks Gary. I've merged all three patches in 63bc79fdded00211f4b9a51561920965d4e2c67d.





[TCHECK-19] Permit a data-structure containing generators to be used as a generator Created: 11/Apr/14  Updated: 22/Jun/14

Status: Open
Project: test.check
Component/s: None
Affects Version/s: None
Fix Version/s: None

Type: Enhancement Priority: Minor
Reporter: Darrick Wiebe Assignee: Reid Draper
Resolution: Unresolved Votes: 0
Labels: None

Attachments: File add_record.diff    

 Description   

I'm working through some of the examples in John Hughes' "QuickCheck Testing for Fun and Profit" paper. I'd like the structures I generate in those tests to look like the this:

[:apply `reg [(gen-name) (gen/elements pids)]]

Rather than the way they must be expressed now:

(gen/tuple (gen/return :apply) (gen/return `reg) (gen/tuple (gen-name) (gen/elements pids)))

I think a simple recursive function could be used to generate these, with the caveat that currently generators create a map {:gen #<fn...>}, which I'd propose to change to a record so that we could distinguish between a generator and a map data-structure.

This seems to be implemented to good effect in Quvig QuickCheck already:

In general, Quviq QuickCheck permits any data-structure containing embedded generators to be used as a generator for data-structures of that shape—something which is very convenient for users, but quite impossible in Haskell, where embedding a generator in a data-structure would normally result in a type error. This technique is used throughout the generators written at Ericsson. – Hughes



 Comments   
Comment by Darrick Wiebe [ 11/Apr/14 5:06 PM ]

Playing around in the REPL today, I came up with this code that seems to work well converting arbitrary literals into generators:

(defprotocol LiteralGenerator
  (-literal->generator [literal]))

(defn literal->generator [literal]
  (cond
    (satisfies? LiteralGenerator literal) (-literal->generator literal)
    (vector? literal) (apply gen/tuple (mapv literal->generator literal))
    (and (map? literal) (= [:gen] (keys literal)) (fn? (:gen literal))) literal
    (map? literal) (gen/fmap (partial into {}) (literal->generator (mapv vec literal)))
    (set? literal) (gen/fmap set (literal->generator (vec literal)))
    (list? literal) (gen/fmap (partial apply list) (literal->generator (vec literal)))
    :else (gen/return literal)))

Generating from a record is probably something that would be generally useful, so I added this as well:

(defn record->generator
  ([record]
   ; Is there a better way to do this?
   (let [ctor (eval (read-string (str "map->" (last (clojure.string/split (str (class record)) #"\.")))))]
     (record->generator ctor record)))
  ([ctor record]
   (gen/fmap ctor (literal->generator (into {} record)))))

Which enables me to extend a record like this:

(defrecord Foo [a b]
  LiteralGenerator
  (-literal->generator [this] (record->generator map->AbcDef this)))

I haven't looked at the possibility of baking this code into test.check at all yet. I'd like feedback on what I've got so far before pursuing this any further.

Comment by Reid Draper [ 11/Apr/14 5:31 PM ]

So, I have kind of mixed feelings about this. I agree it's convenient, but I also worry that being loose like that can allow more accidents to occur, and doesn't force users to make the distinction between values and generators. For example, what if you do something like this: [int int]. Did you mean to type [gen/int gen/int], or do you really intend to have written the equivalent of [(gen/return int) (gen/return int)]? If every function that expects a generator can also take a value, we can no longer start adding error-checking to make sure you're correctly passing generators. On that same token, I do see the benefit of being able to use data-structure literals. What if we wrote a function that you had to use to create a generator with this syntax, something like: {{(gen/literal [:age gen/int])}}. That way you could opt-in to this syntax, but only within the scope of gen/literal?

Comment by Darrick Wiebe [ 11/Apr/14 5:40 PM ]

I agree that is a concern, and since you're not guaranteed to see generator output, it might be better to be explicit about using gen/literal. It's still a nice clean solution. Fortunately, that's exactly what I've written! We'd just need to rename literal->generator to literal and install it into the clojure.test.check.generators namespace.

Comment by Reid Draper [ 11/Apr/14 6:56 PM ]

I think the first step is changing generators to use Records. Interested in making a patch for that?

Comment by Darrick Wiebe [ 11/Apr/14 8:31 PM ]

A very simple patch to use a Generator record rather than a simple {:gen ƒ) map.

Comment by Reid Draper [ 12/Apr/14 6:15 PM ]

I've applied the record-patch in ef132b5f85a07879f01417c9104aa6dea771fdb4. Thanks. I've also added a generator? helper function.

Comment by Philip Potter [ 22/Jun/14 4:39 PM ]

I spotted something which seemed relevant: ztellman/collection-check defines a tuple* fn which is like gen/tuple but automatically wraps literals with gen/return.

Notably, despite not solving the general case, this would have solved the example in the issue description.





[TCHECK-18] Output of failed defspec should result in copy and paste-able data. Created: 08/Apr/14  Updated: 17/Nov/14  Resolved: 17/Nov/14

Status: Resolved
Project: test.check
Component/s: None
Affects Version/s: None
Fix Version/s: None

Type: Enhancement Priority: Minor
Reporter: Jason Gilman Assignee: Reid Draper
Resolution: Completed Votes: 0
Labels: None


 Description   

During a failure of a defspec the generated example data that caused the failure is not output in a way that makes it easy to copy and paste for testing. For example if the failed data was {:a "foo" :b "bar"} it will be displayed in the REPL as {:a foo, :b bar}. This is happening because the result of testing is printed by clojure.test.check.clojure-test/assert-check with println:

(defn- assert-check
[{:keys [result] :as m}]
(println m)
(if (instance? Throwable result)
(throw result)
(ct/is result)))

If it was changed to use pr-str then the output would be copy and paste-able:

(defn- assert-check
[{:keys [result] :as m}]
(println (pr-str m))
(if (instance? Throwable result)
(throw result)
(ct/is result)))

The following example spec output is better when I manipulate a local copy of test.check.

(defspec test-output 1
(for-all [n (gen/return {:a "foo"})]
(= n {:b "foo"})))

Output without change:
{:test-var test-output, :result false, :failing-size 0, :num-tests 1, :fail [{:a foo}], :shrunk {:total-nodes-visited 0, :depth 0, :result false, :smallest [{:a foo}]}}

Output with change:
{:test-var "test-output", :result false, :failing-size 0, :num-tests 1, :fail [{:a "foo"}], :shrunk {:total-nodes-visited 0, :depth 0, :result false, :smallest [{:a "foo"}]}}



 Comments   
Comment by Reid Draper [ 08/Apr/14 9:20 PM ]

Agree. This should get fixed.

Comment by Reid Draper [ 17/Nov/14 10:29 PM ]

Resolved in e857a084725889d0be7bee2a4e53ad70cd9b5f12.





[TCHECK-17] shrinking/test running treat ctrl-c (InterruptedException) as Created: 04/Apr/14  Updated: 06/Apr/14

Status: Open
Project: test.check
Component/s: None
Affects Version/s: None
Fix Version/s: None

Type: Defect Priority: Major
Reporter: Tom Crayford Assignee: Reid Draper
Resolution: Unresolved Votes: 0
Labels: None


 Description   

If you're testing a long running or blocking test (for example, I've been
working on finding race conditions with state machines ala John Hughes' keynote
at clojure west), it's very frustrating to not be able to ctrl-c tests when
they're blocking/stalled/slow. This is because the shrinking/running processes
are naive, and only catch Exception, without special casing
InterruptedException.

I'd very much like to see this fixed - it's a blocker on me actually using
test.check on my project, and very definitely a blocker on the race
condition/state machine work I've been doing. I've had a couple of runs at it,
but got pretty confused at aborting the shrink correctly (likely that's me
being dumb though).

To reproduce:

(tc/quick-check (prop/for-all [a (gen/any)] (Thread/sleep 10000)))

I'd expect to see:

the test run finishes

What happens instead:

test.check treats the InterruptedException as a failure and continues shrinking
as though that run had failed.



 Comments   
Comment by Reid Draper [ 06/Apr/14 11:44 AM ]

Agree. I have some ideas about what the best longterm solution is here, but am still a bit fuzzy. That being said, I think there are some simple incremental solutions that can help people now. I'll take a stab at something this week and gather some feedback.





[TCHECK-16] docstrings/comment cleanup Created: 03/Apr/14  Updated: 06/Apr/14  Resolved: 06/Apr/14

Status: Resolved
Project: test.check
Component/s: None
Affects Version/s: None
Fix Version/s: None

Type: Task Priority: Trivial
Reporter: Gary Fredericks Assignee: Reid Draper
Resolution: Completed Votes: 0
Labels: None

Attachments: Text File vocabularic-pedantry.patch    

 Description   

Deleted a few apostrophes.



 Comments   
Comment by Reid Draper [ 06/Apr/14 10:53 AM ]

Applied in 2b2e5fac123e798a88a01e536bd12219395b09e2.





[TCHECK-15] gen/for macro for alternate combinator syntax Created: 02/Apr/14  Updated: 23/May/14

Status: Open
Project: test.check
Component/s: None
Affects Version/s: None
Fix Version/s: None

Type: Enhancement Priority: Minor
Reporter: Gary Fredericks Assignee: Reid Draper
Resolution: Unresolved Votes: 1
Labels: None

Attachments: Text File TCHECK-15-p1.patch     Text File TCHECK-15-p2.patch     Text File TCHECK-15-p3.patch     Text File TCHECK-15-p4.patch     Text File TCHECK-15.patch    

 Description   

I think the syntax of clojure.core/for would be a good fit for test.check's combinators. For example:

(defn gen-even-subset
  "Returns a generator that generates an even-cardinality
   subset of the given elements"
  [elements]
  (gen/for [bools (apply gen/tuple (repeat (count elements) gen/boolean))
            :let [true-count (->> bools (filter identity) (count))]
            :when (even? true-count)]
    (->> (map list bools elements)
         (filter first)
         (map second)
         (set))))

This combines the capabilities of fmap, bind, and such-that into a familiar syntax.

One downside here is the temptation to use multiple clauses for independent generators, resulting in a use of gen/bind when gen/tuple would be simpler and presumably shrink easier. An approach to this is an additional supported clause, perhaps called :parallel, that uses the syntax of :let to provide the functionality of gen/tuple:

(gen/for [:parallel [n1 gen/nat
                     n2 gen/nat]
          :let [sum (+ n1 n2)]]
  {:nums [n1 n2] :sum sum})

Compared to gen/tuple, this has the advantage of placing generators syntactically next to names, rather than segregating the generators from the names.

The :parallel feature has not been added to the current patches.



 Comments   
Comment by Gary Fredericks [ 05/Apr/14 3:23 PM ]

I think there might be some design ambiguity around the meaning of :when. In particular, in the following contrived example:

(for [n nat
      v (vec (return n))
      :let [sum (reduce + v)]
      :when (pos? sum)]
  v)

In my default design this can hang, for the same reason that this code can hang:

(bind nat 
      (fn [n]
        (such-that
          (fn [v] (pos? (reduce + v)))
          (vector (return n)))))

But it could just as well have been written:

(such-that 
  (fn [v] (pos? (reduce + v)))
  (bind nat (fn [n] (vector (return n)))))

So the issue is whether a :when filter is applied to just the previous generator or to all of the previous generators. I have some hazy notion that the latter would be less efficient in some cases, but I'm not sure what. So I think our options are:

  1. Decide to always do it one way or the other
  2. Provide a third keyword (:when-all?) with different behavior
  3. Don't write this macro at all because it's too difficult to understand

My gut is to do option 1 and just apply :when to the previous generator.

Comment by Gary Fredericks [ 08/Apr/14 9:24 PM ]

Attached my initial draft. The implementation took a lot more thought than I expected, and is a bit subtle, so I included some inline comments explaining the structure of the macro.

Comment by Gary Fredericks [ 13/Apr/14 8:33 PM ]

Attached TCHECK-15-p1.patch, updated to apply to the current master.

Comment by Gary Fredericks [ 16/Apr/14 9:51 PM ]

Attached TCHECK-15-p2.patch which adds a note to the docstring about independent clauses, shrinking, and tuple.

Comment by Gary Fredericks [ 16/Apr/14 9:58 PM ]

Attached TCHECK-15-p3.patch which fixes one bug and one redundancy in namespace aliases.

Comment by Gary Fredericks [ 13/May/14 10:37 AM ]

Attached TCHECK-15-p4.patch which fixes a bug with destructuring (and adds a regression test).

Comment by Gary Fredericks [ 13/May/14 10:38 AM ]

Also might be helpful to note that I've put this in my test.check utility library for now: https://github.com/fredericksgary/test.chuck#for.





[TCHECK-14] re-organize README and doc/ Created: 29/Mar/14  Updated: 29/Mar/14

Status: Open
Project: test.check
Component/s: None
Affects Version/s: None
Fix Version/s: None

Type: Enhancement Priority: Minor
Reporter: Reid Draper Assignee: Reid Draper
Resolution: Unresolved Votes: 0
Labels: documentation


 Description   

I'd like to re-organize the README, and focus on making it easier for users to find the information they need. The README is currently pretty long, and could be information overload. My current thinking is that the new README should be short, and focus on providing links to people based on their experience with test.check. I think something like the following three headings would be useful:

1. First time user. Show me what this is all about.

  • Focus on 'seeing something cool' within five minutes. Maybe include some example tests in examples/. This way new users can simply clone the repository and run a test in under two minutes.

2. Still getting started, show me some tutorials and examples.

  • This will probably build on the existing doc/intro.md file, which currently describes how to write generators.

3. Existing user, show me API documentation, release notes, advanced techniques.

  • API documentation
  • Release-notes, make it clear these are up-to-date, and have clear instructions when there are breaking changes
  • Advanced techniques/tutorials





[TCHECK-13] Add generated API documentation for test.check Created: 27/Mar/14  Updated: 29/Mar/14  Resolved: 29/Mar/14

Status: Resolved
Project: test.check
Component/s: None
Affects Version/s: None
Fix Version/s: None

Type: Enhancement Priority: Major
Reporter: Jason Gilman Assignee: Reid Draper
Resolution: Completed Votes: 0
Labels: None


 Description   

The simple-check README had a link to API documentation (http://reiddraper.github.io/simple-check) that was very useful. This appears to be missing for test.check. Please add generated documentation and a link to it on the test.check README.



 Comments   
Comment by Reid Draper [ 29/Mar/14 5:47 PM ]

Pushed: http://clojure.github.io/test.check/

Comment by Nicola Mometto [ 29/Mar/14 5:54 PM ]

Reid, clojure contrib repositories have the webdoc managed by autodoc so that they are shown in http://clojure.github.io/

I suggest you ping Tom Faulhaber so that he can set up the autodoc page for test.check, he will also add a hook so that the autodoc page will be updated automatically at every change





[TCHECK-12] Random-number seed isn't returned on test failure Created: 24/Mar/14  Updated: 25/Mar/14  Resolved: 25/Mar/14

Status: Resolved
Project: test.check
Component/s: None
Affects Version/s: None
Fix Version/s: None

Type: Defect Priority: Major
Reporter: Reid Draper Assignee: Reid Draper
Resolution: Completed Votes: 0
Labels: None


 Description   

The random seed used to run the test is returned on test success, but not failure. We should return it on failure too.



 Comments   
Comment by Reid Draper [ 25/Mar/14 11:33 AM ]

Fixed in https://github.com/clojure/test.check/commit/9ba775a9fd69591ef065051be6c2b727387daff4.





[TCHECK-11] clojure.test.check.generators/rand-range includes upper bound Created: 14/Mar/14  Updated: 15/Apr/14  Resolved: 15/Apr/14

Status: Resolved
Project: test.check
Component/s: None
Affects Version/s: None
Fix Version/s: None

Type: Defect Priority: Minor
Reporter: Steve Kim Assignee: Reid Draper
Resolution: Completed Votes: 1
Labels: None

Attachments: File private-rand-range.diff    

 Description   

While playing with the rand-range function in the clojure.test.check.generators namespace, I was surprised that

(range-range lower upper)

can generate integers equal to the upper bound. This behavior is inconsistent with the zero-based numbering convention that ranges of integers are lower-inclusive, upper-exclusive.

I know that this function appears in a section that is commented

;; Internal Helpers

but it isn't a private function, and it is useful for users who need to implement custom generators. Can it be changed? It looks like the choose function in the same namespace is the only other function that would need to be changed too, along with unit tests.

If others agree that this is a bug that ought to be fixed, I volunteer to submit a patch, after I provide my signed Contributor Agreement.



 Comments   
Comment by Steve Kim [ 14/Mar/14 2:04 PM ]

Sorry for the typo. I meant

(rand-range rnd lower upper)
Comment by Reid Draper [ 16/Mar/14 8:06 PM ]

Both Erlang and Haskell QuickCheck do this too. I'll admit I mostly just followed in their steps, and that's it's a bit unexpected, compared to Clojure functions like 'range'. However, without being inclusive on the top-end, you wouldn't be able to create generators that fill an entire 'type range'. For example: (gen/sample (gen/choose Integer/MIN_VALUE Integer/MAX_VALUE)). Thoughts?

Comment by Chas Emerick [ 17/Mar/14 4:06 AM ]

I agree that it's surprising at first, but the utility is undeniable (compared to e.g. range as mentioned, the behaviour of which you need to compensate for in various ways in order to include upper bounds).

Comment by Jean Niklas L'orange [ 17/Mar/14 6:11 AM ]

On the visibility of this function: As far as I can see, rand-range is just used once in generators, and not used anywhere else. Whereas it is a useful function in general, I don't think it is that useful for people who need to implement custom generators (You can bind/fmap over choose instead).

I assume it wouldn't be a problem to make this private.

Comment by Steve Kim [ 17/Mar/14 10:12 AM ]

I did not know the QuickCheck convention, and I had not considered the need to fill an entire type range like

(gen/sample (gen/choose Integer/MIN_VALUE Integer/MAX_VALUE))
. Thanks for the explanation!

The docstring for choose states that it is inclusive, whereas rand-range is not documented. I think that new users would be less confused if either (1) rand-range had a docstring, or (2) rand-range were made private, because it only makes sense as a helper to implement the choose function. This is a trivial nitpick; I won't complain if this issue is resolved as Won't fix.

Comment by Reid Draper [ 17/Mar/14 10:20 AM ]

Sounding like we're all in favor of making range-range private. I'm happy to take a patch if someone wants to contribute. Otherwise I'll fix it today or tomorrow.

Comment by Jean Niklas L'orange [ 15/Apr/14 5:49 AM ]

I have attached the straightforward patch to this issue.

There is not much to elaborate on, only that the test suite now refers to the var containing the function, not the function itself (as it now is private).

Comment by Reid Draper [ 15/Apr/14 7:58 PM ]

Fixed in 364710da692e66fca7310bade4388cf9439e3ef1, thanks!

Comment by Reid Draper [ 15/Apr/14 7:58 PM ]

364710da692e66fca7310bade4388cf9439e3ef1





[TCHECK-10] defspec makes it impossible to specify :max-size (and seed) Created: 04/Mar/14  Updated: 06/Sep/14  Resolved: 06/Sep/14

Status: Resolved
Project: test.check
Component/s: None
Affects Version/s: None
Fix Version/s: None

Type: Enhancement Priority: Major
Reporter: Jean Niklas L'orange Assignee: Reid Draper
Resolution: Completed Votes: 0
Labels: None

Attachments: File defspec.diff    

 Description   

While defspec is very valuable for integration with clojure.test, it is almost impossible to specify :max-size for the quick-check call. The only way I've found around it right now is by using the following macro:

(defmacro ^:private change-max-size
  [name max-size]
  `(alter-meta! (var ~name) assoc
                :test (fn [] (#'ct/assert-check
                             (assoc (~name ct/*default-test-count*
                                           :max-size ~max-size)
                               :test-var (str '~name))))))

This macro alters the meta of a named test such that the quick-check-call uses a specified expression for max-size, but obviously this is a very hacky solution. Preferably, I would be able to specify the :max-size value directly through defspec. I imagine that something like this could work:

(defspec my-test
  {:max-size 40, :num-tests 400}
  (my-test-here))

Essentially, if the optional argument (currently named default-times) is a map, inspect the map for values given, or use defaults if not.



 Comments   
Comment by Reid Draper [ 04/Mar/14 8:52 PM ]

I like the idea of making the first (optional) argument a map. This allows us to add in more options in the future, without breaking existing code.

Comment by John Walker [ 03/Sep/14 10:06 AM ]

Do we want to maintain backwards compatibility with

(defspec my-test number-of-times (my-test-here))

?

Comment by Reid Draper [ 03/Sep/14 6:02 PM ]

I spoke with Gary Fredericks about this earlier today, we're both thinking it seems reasonable to keep the current Int as a value, but also accept a map. The map will contain the same optional keys as the quick-check function, with an additional num-tests key. That seem reasonable?

Comment by John Walker [ 03/Sep/14 8:34 PM ]

Yes, I think that is the right approach. I'll get a first draft in for us tonight.

Comment by Reid Draper [ 03/Sep/14 8:54 PM ]

Great, thanks! As an aside, I've noticed that quick-check takes var-args, and not a last-argument-map. I'd probably like to change this as well, but am not sure how to keep backward compatibility there...

Comment by John Walker [ 03/Sep/14 9:14 PM ]

This patch adds a map as an alternative for defspec's options. Users can specify max-size, num-tests and seed using it. It also changes the way in which the function arguments produced by defspec are qualified. Example: Before it was max-size_auto_1448, now it is max-size.

Comment by John Walker [ 03/Sep/14 9:18 PM ]

Great, thanks!

You're welcome! Thanks for test.check.

As an aside, I've noticed that quick-check takes var-args, and not a last-argument-map. I'd probably like to change this as well, but am not sure how to keep backward compatibility there...

Probably the best way to maintain backward compatibility here would be to do some parsing on the varargs ourselves. It would require a check to see if the input is a map, etc. It's not a lot of trouble, but it's definitely a compromise.

Comment by Reid Draper [ 06/Sep/14 4:51 PM ]

Merged, with some white-space cleanup and added tests in c3d6134a5eef3a3c45369cc3f3f9416d23d30f61. Thanks!





[TCHECK-9] Properly document how :max-size works Created: 04/Mar/14  Updated: 14/Apr/14

Status: Open
Project: test.check
Component/s: None
Affects Version/s: None
Fix Version/s: None

Type: Enhancement Priority: Minor
Reporter: Jean Niklas L'orange Assignee: Reid Draper
Resolution: Unresolved Votes: 0
Labels: documentation


 Description   

I have had several issues where I thought increasing quick-check's num-tests argument would increase the time taken linearly, but surprisingly it grew quadratically. This is related to the auto-increasing of the size to call-gen, which I believe it isn't documented well enough. It also seems like other people have had trouble with this functionality, see the reply to announcement of clojure.test.check.

Preferably, it should be mentioned in the docstring and the tutorial that the size of the structures generated grows linearly with the amount of tests you have, up to 200. As a result, doubling the amount of tests does not necessarily double the amount of time taken to run the tests.

In some of my tests, this behaviour increases the running time exponentially, and I actually used quite some time figuring out why. The documentation on recursive generators decreases the size of the binary tree in half, but it is not explained why it is split in half. While obvious in hindsight, explicitly explaining that recursive structures should have an amount of nodes proportional to the input size argument and how it is done, would be beneficial.



 Comments   
Comment by Reid Draper [ 04/Mar/14 8:50 PM ]

+1. This is not well-documented now. Further, I'm still not 100% sure that halving inside of a recursive generator is the right thing to do. For some structures, log2 may be more appropriate. In the mailing list post you mention, gen/any is clearly not bounded enough at roughly size=90, consider it's using more than 1GB of heap.

Comment by Reid Draper [ 14/Apr/14 8:58 PM ]

I've made things a little better in 57df9b9b8f2fe61a798fbf683046e8d1aa128228.





[TCHECK-8] Bound tests and shrinks in time Created: 04/Mar/14  Updated: 04/Mar/14

Status: Open
Project: test.check
Component/s: None
Affects Version/s: None
Fix Version/s: None

Type: Enhancement Priority: Minor
Reporter: Jean Niklas L'orange Assignee: Reid Draper
Resolution: Unresolved Votes: 1
Labels: None


 Description   

It should be possible to bound tests and shrinks in time.

The original issue also mentions shrink interruption through Ctrl-c, but this is already covered in #TCHECK-4.

Original issue - #9 GitHub






[TCHECK-7] Add nested property support Created: 04/Mar/14  Updated: 26/May/14

Status: Open
Project: test.check
Component/s: None
Affects Version/s: None
Fix Version/s: None

Type: Enhancement Priority: Minor
Reporter: Jean Niklas L'orange Assignee: Reid Draper
Resolution: Unresolved Votes: 2
Labels: None


 Description   

Including functionality to add support for nested properties would be beneficial. An example of how this may work follows:

(defn exists?
  [e coll]
  (some #{e} coll))

(defn remove-elem
  [e coll]
  (remove #{e} coll))

;; this test should fail, and illustrates nested properties
(quick-check (for-all [v (gen/vector gen/int)]
               (for-all [e (elements v)]
                 #(exists? e (remove-elem e v)))))

The requirement for this is support for reifying properties, which is not currently implemented.

Original issue - GitHub #10



 Comments   
Comment by Maciej Jaśkowski [ 14/May/14 4:14 PM ]

What's the status of this issue?
It would help us a lot to have this functionality up and running in the following case: I want to verify my path finding algorithm finds optimal solution in the given graph G, so I generate the graph and a big number of random paths and check that all the random paths are "less optimal" then the one found by the algorithm.

Having nested for-all that would be simple. Is there a way around?

Comment by Reid Draper [ 14/May/14 4:24 PM ]

Sorry for the lack of update here. Nested properties are still in a bit of hammock-time on my part. The good news is they are merely convenient, and you can achieve the exact same effect with clojure.test.check.generators/bind. You can see an example of bind being used here.

Comment by Maciej Jaśkowski [ 15/May/14 12:08 AM ]

Apparently I need some more hints because all I can generate (using gen/bing, btw) are pairs '(graph list-of-random-paths)

That is useful as long as the algorithm works as expected. However, once it fails it's pretty hard to figure out which of random-paths failed it.

Comment by Reid Draper [ 15/May/14 1:02 PM ]

Can you show me how you would write the test with nested properties, and then perhaps I can help you rewrite it to use bind instead?

Comment by Maciej Jaśkowski [ 15/May/14 3:58 PM ]

Great! Thank you!

Please, find the most important part below.

(defn gen-matrix-and-paths [size]
    (gen/bind
      (gen-matrix gen/s-pos-int size)
      (fn [matrix]
        (let [ nodes (into [] (range (count matrix)))
               gen-perms (gen-permutations nodes)
               perms (distinct (gen/sample gen-perms 500)) ]
          (gen/tuple 
            (gen/return matrix) 
            (gen/return perms))))))

gen-matrix generates a square matrix of incidence e.g. [[0 1] [4 0]]
gen-permutations generates a permutation of provided vector

Comment by Reid Draper [ 15/May/14 4:41 PM ]

I'm confused. What issue are you running into? The code you've pasted uses bind, and not a nested property.

Comment by Maciej Jaśkowski [ 17/May/14 3:26 AM ]

Ok but now I can only write something like this:

(prop/for-all [data (gen-matrix-and-paths 6)]
   (let [ [matrix random-paths] data ]
     (not-worse-then-others? tsp matrix random-paths))))

which, if my 'tsp' algorithm is not working correctly would point me to a matrix and a vector (length 500) of paths one of which is counterexample. It would be better if instead I got the matrix and a single counterexample.

Comment by Reid Draper [ 17/May/14 12:44 PM ]

I'm still not sure what this would look like using a nested-property. Can you please write the code the way you'd like to see it as a nested property? This will involve at least two for-alls.

Comment by Maciej Jaśkowski [ 18/May/14 4:56 PM ]

Sure!

I would like to write something like that:

(prop/for-all [matrix (gen-matrix 6)]
  (prop/for-all [random-path (gen-random-path matrix)]
    (<= (cost (tsp matrix)) (cost random-path))))
Comment by Maciej Jaśkowski [ 23/May/14 6:22 AM ]

ping!

Comment by Reid Draper [ 26/May/14 11:47 AM ]

Ok, let's translate that to using gen/bind. Just like with nested properties, bind allows you to create generators that depend on the value of another generator. For example, your gen-random-path depends on the previously generated matrix variable. So let's write a generator that returns both a matrix and a random-path.

(def matrix-and-path
  "Return a two-tuple of [matrix random-path]"
  (gen/bind (gen-matrix 6)
            (fn [matrix]
               (gen/tuple (gen/return matrix) (gen-random-path matrix)))))

(prop/for-all [[matrix random-path] matrix-and-path]
  (<= (cost (tsp matrix)) (cost random-path))




[TCHECK-6] Rose-filter may assume too much Created: 04/Mar/14  Updated: 07/Aug/14

Status: Open
Project: test.check
Component/s: None
Affects Version/s: None
Fix Version/s: None

Type: Enhancement Priority: Minor
Reporter: Jean Niklas L'orange Assignee: Reid Draper
Resolution: Unresolved Votes: 0
Labels: None


 Description   

Gary Fredericks commented that the rose-filter will discard a node and all of its children if the node doesn't pass the predicate. On the other hand, some of its children may pass the predicate.

Reid Draper said that he has attempted to solve this previously, but it was not evident how one should exclude a node from the tree. One solution considered was to replace the node with all of its children, then repeat the procedure recursively.

Gary Fredericks came with another solution, where you promote the first child to the root and concatenate the remaining children to the first child's children.

Original issue - GitHub #33



 Comments   
Comment by Scott Feeney [ 06/Aug/14 1:21 AM ]

Am I correct in thinking the motivation here is to make such-that applied to generators still find the minimal failing example in all cases?

If so, a complete fix seems impractical. Say we have a tree that lets us shrink from C to B to A, but B fails the predicate while C and A pass. Then it's fairly straightforward to keep A in:

     C
    / \
   B  ...
  / \
 A  ...

becomes

     C
    / \
   A  ...

But what if multiple levels are taken out? Say the shrink is like Y→X1→X2→X3→...→Xn→W where Y and W pass the predicate but all the Xs don't? If we accommodate that, then a single call to (first (children Y)) could, in the worst case, require eager exploration of the entire tree under Y, which could easily be exponential.

Comment by Reid Draper [ 07/Aug/14 3:21 PM ]

Indeed. I believe your analysis is correct. And so far, not having this optimization has not resulted in sub-optimal shrinks, as far as I know. Perhaps we should close this issue, and re-open it if we're able to find a specific test whose shrinking is poor because?

Comment by Scott Feeney [ 07/Aug/14 4:28 PM ]

That seems like a reasonable course of action. But I have a question, and feel free to stop me if this issue isn't the correct place to discuss this, but what do people actually use such-that for? Would it not be preferable to use fmap?

For example: generating even integers. (gen/such-that even? gen/int) has a potential problem finding the optimal shrink, but (gen/fmap #(* 2 %) gen/int) does not. Also, once every thousand runs, the former will error out without finding an appropriate value; the latter is reliable. It does produce larger values, but if it's important, you could cancel that out with:

(defn half-size [f]
  (gen/sized
    (fn [size]
      (gen/resize (/ size 2) f))))

(gen/fmap #(* 2 %) (half-size gen/int))

Isn't it better to find a mapping from all values to the subset of values you want? In what case would that not be possible (yet such-that is sufficiently reliable)?





[TCHECK-5] Include a float generator Created: 04/Mar/14  Updated: 10/Oct/14

Status: Open
Project: test.check
Component/s: None
Affects Version/s: None
Fix Version/s: None

Type: Enhancement Priority: Minor
Reporter: Jean Niklas L'orange Assignee: Reid Draper
Resolution: Unresolved Votes: 1
Labels: None


 Description   

There seems to be a demand for a general float generator.

One of the bigger issues with a float generator is shrinking, which is not straightforward. Reid Draper intially considered including a small implementation, using fmap to coerce a ratio into a float:

(def float (gen/fmap float gen/ratio))
(gen/sample float 20)
;; => (0.0 0.0 -1.0 0.0 -1.0 0.75 0.5 0.8 -2.0 -0.5714286 2.6666667 -2.0
;;     -0.46153846 -0.125 -12.0 -1.75 -2.4 0.41666666 0.64285713 -1.125)

The Haskell implementation is performing float shrinking through fractions.

Chas Emerick implemented a bigdec generator using ideas from data.generators, but mentioned that manipulating the long and int bits would be more performant. However, such shrinking would require good knowledge on how IEEE 754 works, in order to avoid bugs whenever shrinking is performed.

A slower, but more understandable and portable double generator was also suggested, again by Chas Emerick. The implementation of said generator could be found at https://gist.github.com/cemerick/7599452

Original issue - GitHub #36



 Comments   
Comment by Gary Fredericks [ 10/Oct/14 1:41 PM ]

I've written one and added it to test.chuck here.

It scales exponentially with the size so it very quickly fills the whole double range (I think once size > 53 or so).

It will generate +/-Infinity but not NaN.

It shrinks in an okayish way.

I'm not sure what the design goals are here, so don't know if it's worth submitting yet.





[TCHECK-4] Handle sigint Created: 04/Mar/14  Updated: 04/Mar/14

Status: Open
Project: test.check
Component/s: None
Affects Version/s: None
Fix Version/s: None

Type: Enhancement Priority: Minor
Reporter: Jean Niklas L'orange Assignee: Reid Draper
Resolution: Unresolved Votes: 0
Labels: None


 Description   

During shrinking, this should simply return the best shrink so-far. During testing, perhaps it should just return the number of tests that've been run. In both cases, some indication should be given in the return value that the user pressed ctrl-c.

Original issue - #41 on GitHub






[TCHECK-3] Add ability to customize test output when a test fails Created: 04/Mar/14  Updated: 04/Mar/14

Status: Open
Project: test.check
Component/s: None
Affects Version/s: None
Fix Version/s: None

Type: Enhancement Priority: Minor
Reporter: Jean Niklas L'orange Assignee: Reid Draper
Resolution: Unresolved Votes: 0
Labels: None


 Description   

From the original description:

I would like the ability to control / enhance / replace the output generated when a test fails. Printing the raw values works well as long as those values have a useful output for the test at hand, but this doesn't always hold true.

To give a bit of background, I was playing with generating sequences of partially-applied functions and then applying them to an initial state. Here's a small excerpt from the full gist:

:smallest [(#<core$partial$fn__4190 clojure.core$partial$fn__4190@2a799171>)]

What would be more useful to me is if the output had something like this, which is more meaningful:

:smallest [(- 45 %)]

In this example, (- 45 %) is all my custom output string. 45 is a value from an upstream generator (and would thus change), and % is just a string modeled after the anonymous function syntax.

I could even see my particular application rolled up as a generator:

(def gen-inc
  (gen/bind gen/pos-int #(gen/partial + %)))

Which could handle some of the heavy lifting for the user.

Original issue






[TCHECK-2] pos-int is confusingly named Created: 04/Mar/14  Updated: 04/Mar/14

Status: Open
Project: test.check
Component/s: None
Affects Version/s: None
Fix Version/s: None

Type: Enhancement Priority: Minor
Reporter: Jean Niklas L'orange Assignee: Reid Draper
Resolution: Unresolved Votes: 2
Labels: None


 Description   

The generators neg-int and pos-int generate 0, which is confusing as (= (pos? 0} (neg? 0) false). The suggestion is to change s-pos-int and s-neg-int to pos-int and neg-int respectively, and replace the current neg-int and pos-int. Suggestions for new names include non-x-int and not-x-int.

Original issue






[TCHECK-1] Namespace reorganization Created: 13/Feb/14  Updated: 17/Nov/14  Resolved: 17/Nov/14

Status: Resolved
Project: test.check
Component/s: None
Affects Version/s: None
Fix Version/s: None

Type: Enhancement Priority: Major
Reporter: Reid Draper Assignee: Reid Draper
Resolution: Declined Votes: 0
Labels: None


 Description   

Currently test.check (and simple-check) use three separate namespaces: core, generators and properties. Bronsa has suggested renaming test.check.core to simply test.check. This has me wondering if the entire public API should be exported from just the 'test.check' namespace. This would mean users only need to import one namespace. Thoughts?

[1] https://github.com/clojure/test.check/commit/a2f270ae8e6fa99f337eaff6fbb2051a66dac392#commitcomment-5303415



 Comments   
Comment by Peter Taoussanis [ 28/Feb/14 11:01 AM ]

+1 I'd definitely welcome merging the namespaces. Even merged, the lot would be < 1000 loc - and the core namespace isn't much use w/o the other two.

Comment by Reid Draper [ 28/Feb/14 2:42 PM ]

I'd still like to keep them as separate files, if we do this. Any easy way to do that? Just declare the same namespace in each file?

Comment by Nicola Mometto [ 28/Feb/14 5:22 PM ]

Put the ns decl in the main file and then simply clojure.core/load the different files.

Comment by Bruce Adams [ 01/Mar/14 9:01 AM ]

Merging core and properties is fine, but I like having the generators in a separate namespace.

The complex part of my uses of test.check have been building generators. Having a clear name space for generators helps make my code easier to read. Here are some trivially small examples from my code:

(ns whatever.generators
  (:require (clojure.test.check [generators :as gen])))

(def gen-non-blank-string
  (gen/such-that #(not-empty (.trim %)) gen/string))

(defn gen-nil-or
  [generator]
  (gen/one-of [(gen/return nil) generator]))

(defn gen-set
  [generator]
  (gen/fmap set (gen/vector generator)))
Comment by Chas Emerick [ 03/Apr/14 7:52 AM ]

+1 to making the properties bits available in clojure.test.check. I've never been a fan of using load to separate namespaces into different files; var immigration is another option that could work.

I agree with Bruce that keeping the generators namespace separate is desirable.

Comment by Reid Draper [ 08/Apr/14 9:49 PM ]

Currently I'm thinking this is pretty low priority, given the other issues on the repo. Any disagreement?

Comment by Bruce Adams [ 09/Apr/14 7:19 AM ]

I agree, this is lower priority than its current "Major".





Generated at Sun Nov 23 16:41:01 CST 2014 using JIRA 4.4#649-r158309.