Skip to:

  1. Skip to Jira Navigation
  2. Skip to Side Navigation
  3. Skip to Main Content

Rerunning a particular failure is difficult.

Description

Test.check has the concept of a seed that is used when a property is tested. The seed can be supplied or generated by the current time. Either way, the single seed is used to generate all the trials for the test. If a test fails, the only way to try to re-test the property on the same inputs is to run all of the tests again using the same seed. If the tests were running for hours before a failure was found, this is prohibitive.

I would like instead to be able to check the property for exactly the input that failed. One step in that direction is to have an explicit different seed for each trial (perhaps generated from an initial metaseed), and on failure to report a [seed size] pair that can be used to run a test a single time.

The way this interacts with shrinking is interesting though, since it's just as likely that I'll want to re-run on the shrunk value from a failure. Since the shrunk value was never explicitly generated from a known [seed size], just that information is insufficient. We could perhaps report [seed size shrink-path] where the third element is a list of indexes to walk through the shrink tree with. Probably the worst part about that is it might be rather long.

In any case I think something like this would be more useful than the current setup. I'll probably be hacking up something similar for a fork branch I'm maintaining, but if we can settle on a specific design I'd be happy to cook up a patch.

Environment

None

Activity

Show:

gfredericks March 3, 2016 at 4:09 PM

I think this kind of issue becomes more important with TCHECK-96, because if a shrink aborts prematurely it would be nice to be able to run it again without the time limit (without having to run all the prior passing tests like you would if you re-ran quick-check with the same seed).

I just had an intermediate idea for this that doesn't have the downside of the arbitrarily-long "key" that I've used in my fork: we promote the RNG to be a true serializable value, and then in quick-check we keep track of the [rng size] pair used for each trial, and on a failure we report that pair. Then we have a new function similar to quick-check that takes a property and a pair and runs a single trial (with shrinking if it fails).

gfredericks April 12, 2014 at 5:00 PM

It's certainly adequate to reproduce in theory, but I don't know of any built in mechanism that makes this easy. Here's what I end up doing:

Given this:

and after getting a failure with a 50-line shrunk value (e.g. :bar), I manually pretty-print it and paste it back into my file like so:

And now I can run (foo 1) to re-run with the failing value.

This awkwardness is partly due to three facts:

  1. My generators create large unwieldy values

  2. My property is a large body of code rather than a single function

  3. There's no easy way to test a property on a specific value

I could mitigate some of the pain by fixing #2 – having each one of my tests split into a defspec / prop/for-all and a plain function. But #1 is not easy to fix, and having a small tuple is rather nicer for me.

I've already written a POC for this and it's working rather well. I used the term :key to refer to the tuple, so the term doesn't conflict with :seed. I end up calling the test like (foo 0 :key [329489249329323 19 []]), but I'll probably think up something else that doesn't require passing a dummy first arg.

Reid Draper April 12, 2014 at 4:28 PM

When a test fails, we do return the arguments that originally failed, and the arguments of the shrunk test. Is this insufficient?

For example:

see :fail and [:shrunk :smallest].

Details

Assignee

Reporter

Priority

Created April 12, 2014 at 2:20 AM
Updated August 6, 2016 at 11:17 PM