test.check

Make random-number generation in test.check thread-safe

Details

  • Type: Enhancement Enhancement
  • Status: Open Open
  • Priority: Major Major
  • Resolution: Unresolved
  • Affects Version/s: None
  • Fix Version/s: None
  • Component/s: None
  • Labels:
  • Patch:
    Code and Test

Description

test.check uses the Java random-number generator, which is imperative.

This makes the testing non-deterministic in many settings.

It would be better if test.check used a purely functional rng, just
like the original Haskell version of QuickCheck.

I've attached a patch that does this. All but one of the original test
pass; I've modified the only failing test to not rely on an imperative
rng.

Note that it still has quick-check/make-rng supply a random seed.
IMHO, it should supply a fixed seed.

Activity

Hide
Reid Draper added a comment -

Thanks for the patch! I haven't had a chance to look it over in detail, but I do have a few introductory questions. You're correct that test.check currently uses a mutable RNG. While it may not be ideal, I haven't run into any user-facing cases where the use is non-deterministic. Do you have some particular example in mind where test.check is not deterministic now, given the same RNG seed? One place this could be an issue in the future is if we changed the way we walk the shrink tree, which would cause different values to be generated. I'd be keen on fixing this, but it's not an issue at the moment.

tl;dr: Under what circumstances, given the same seed, is test.check non-deterministic?

Show
Reid Draper added a comment - Thanks for the patch! I haven't had a chance to look it over in detail, but I do have a few introductory questions. You're correct that test.check currently uses a mutable RNG. While it may not be ideal, I haven't run into any user-facing cases where the use is non-deterministic. Do you have some particular example in mind where test.check is not deterministic now, given the same RNG seed? One place this could be an issue in the future is if we changed the way we walk the shrink tree, which would cause different values to be generated. I'd be keen on fixing this, but it's not an issue at the moment. tl;dr: Under what circumstances, given the same seed, is test.check non-deterministic?
Hide
Michael Sperber added a comment -

Thanks for the prompt answer, and sorry for taking so long to reply.

The answer to your question is: None I'm currently encountering.

My motivation for the patch was a bit different, though:

1. I don't see an easy way to make things deterministic with defspec, running tests from, say, Leiningen.

2. While I can get reproducibility specifying the seed explicitly, this complicates the testing process, which should be as smooth as possible.

3. With the purely-functional rng, the generator monad is splittable: This enables me to write deterministic tests for concurrent programs.

#1 and #2 could also be addressed with the imperative RNG, not #3, though.

Show
Michael Sperber added a comment - Thanks for the prompt answer, and sorry for taking so long to reply. The answer to your question is: None I'm currently encountering. My motivation for the patch was a bit different, though: 1. I don't see an easy way to make things deterministic with defspec, running tests from, say, Leiningen. 2. While I can get reproducibility specifying the seed explicitly, this complicates the testing process, which should be as smooth as possible. 3. With the purely-functional rng, the generator monad is splittable: This enables me to write deterministic tests for concurrent programs. #1 and #2 could also be addressed with the imperative RNG, not #3, though.
Hide
Reid Draper added a comment -

I don't see an easy way to make things deterministic with defspec, running tests from, say, Leiningen.

Indeed, this is simply an omission in the defspec macro. Calling the quick-check function directly allows you to pass in a seed. You can see the tests for this.

While I can get reproducibility specifying the seed explicitly, this complicates the testing process, which should be as smooth as possible.

This is how Haskell QuickCheck works too. If you don't specify a seed, tests are random. Would you prefer to get the same result, even without explicitly supplying a seed?

With the purely-functional rng, the generator monad is splittable: This enables me to write deterministic tests for concurrent programs.

You can write tests for concurrent programs now. The generated values are completely constructed before your code that may use threads is called. What am I missing?

Show
Reid Draper added a comment -
I don't see an easy way to make things deterministic with defspec, running tests from, say, Leiningen.
Indeed, this is simply an omission in the defspec macro. Calling the quick-check function directly allows you to pass in a seed. You can see the tests for this.
While I can get reproducibility specifying the seed explicitly, this complicates the testing process, which should be as smooth as possible.
This is how Haskell QuickCheck works too. If you don't specify a seed, tests are random. Would you prefer to get the same result, even without explicitly supplying a seed?
With the purely-functional rng, the generator monad is splittable: This enables me to write deterministic tests for concurrent programs.
You can write tests for concurrent programs now. The generated values are completely constructed before your code that may use threads is called. What am I missing?
Hide
Michael Sperber added a comment -

Ah, I didn't realize the shipped QuickCheck does this - sorry. But yes, I would greatly prefer to get the same result, unless explicitly instructed otherwise: This would solve the `defspec' problem.

Also, some test suites will pass or fail depending on the seed - particularly those involving floating-point may need a lot of test cases before they fail, and non-reproducibility might convince you that they pass - for a while.

You can write tests for concurrent programs now.

I was being less than clear: I meant concurrent tests for concurrent programs, where the concurrency is initiated by the test itself. In that case, you may want to call `quickcheck' from different threads, and would thus like to give each its own random-number generator.

Show
Michael Sperber added a comment - Ah, I didn't realize the shipped QuickCheck does this - sorry. But yes, I would greatly prefer to get the same result, unless explicitly instructed otherwise: This would solve the `defspec' problem. Also, some test suites will pass or fail depending on the seed - particularly those involving floating-point may need a lot of test cases before they fail, and non-reproducibility might convince you that they pass - for a while.
You can write tests for concurrent programs now.
I was being less than clear: I meant concurrent tests for concurrent programs, where the concurrency is initiated by the test itself. In that case, you may want to call `quickcheck' from different threads, and would thus like to give each its own random-number generator.
Hide
Reid Draper added a comment -

Just noticed that newer Haskell QuickCheck uses this random number generator now .

Show
Reid Draper added a comment - Just noticed that newer Haskell QuickCheck uses this random number generator now .

People

Vote (0)
Watch (1)

Dates

  • Created:
    Updated: