test.check

New stats feature: Adds ability to assign labels to test cases to report the test case distribution when running a test

Details

  • Type: Enhancement Enhancement
  • Status: Open Open
  • Priority: Major Major
  • Resolution: Unresolved
  • Affects Version/s: None
  • Fix Version/s: None
  • Component/s: None
  • Labels:
    None
  • Patch:
    Code and Test

Description

Inspired in the analog feature that exists in Haskell's QuickCheck. Adds a classify fn that is intended to be used as a wrapper of prop/for-all, returning a property (a generator) appropriate for test.check/quick-check that will augment the result-map returned by the underlying property, adding the collected labels under the :labels key. Also triggers a new event :stats in the default-reporter-fn whose default implementation calls test.check.stats/print, printing the classification of trials with the following format:

12.7% :lt-30
14.5% :gte-30
29.1% :lt-30, :lt-20
43.6% :lt-30, :lt-10, :lt-20

(note that multiple labels might be assigned to some test cases)

I think it answers the question "How could we collect stats about the sorts of things generated?" from the test.check design page

Activity

Hide
Nicolás Berger added a comment -

The patches here no longer apply after the introduction of reporter-fn.

While I'm working on new patches, I'd like to know other people's (and especially gfredericks) opinion on having this new :labels concept in test.check. Each trial's result map can optionally include some :labels that are included in the :complete report so the stats can be computed.

An alternative would be to implement the stats as an external library/reporter, but to make that possible we need the trials loop to keep arbitrary state from each trial's result-map where we could accumulate the labels. Or maybe we could make the (reporter-fn :type :trial ...) to include the entire result-map so the reporter-fn has the chance to "save" this state, but this way doesn't look very elegant...

Show
Nicolás Berger added a comment - The patches here no longer apply after the introduction of reporter-fn. While I'm working on new patches, I'd like to know other people's (and especially gfredericks) opinion on having this new :labels concept in test.check. Each trial's result map can optionally include some :labels that are included in the :complete report so the stats can be computed. An alternative would be to implement the stats as an external library/reporter, but to make that possible we need the trials loop to keep arbitrary state from each trial's result-map where we could accumulate the labels. Or maybe we could make the (reporter-fn :type :trial ...) to include the entire result-map so the reporter-fn has the chance to "save" this state, but this way doesn't look very elegant...
Hide
Gary Fredericks added a comment - - edited

Things to think about:

  • The stats are only a function of the args, so is it necessary to wrap the entire property?
    • haskell does this though
  • On the other hand, I like the idea of stats being a particular instance of a generic feature of the check namespace, but the current implementation doesn't quite achieve that since we have lots of label stuff in the check namespace
  • Buuuut it's hard to move it out because if check just exposed a generic reduce feature, then users going directly to quick-check would have to supply two things – a wrapped property, and the proper reduce function
Show
Gary Fredericks added a comment - - edited Things to think about:
  • The stats are only a function of the args, so is it necessary to wrap the entire property?
    • haskell does this though
  • On the other hand, I like the idea of stats being a particular instance of a generic feature of the check namespace, but the current implementation doesn't quite achieve that since we have lots of label stuff in the check namespace
  • Buuuut it's hard to move it out because if check just exposed a generic reduce feature, then users going directly to quick-check would have to supply two things – a wrapped property, and the proper reduce function
Hide
Nicolás Berger added a comment - - edited

Uploaded a new patch that takes advantage of the new reporter-fn mechanism.

In this new patch some stuff was simplified:

  • Removed the new report-stats var that was going to be used as a way to enable/disable printing of stats for each test. The default-reporter-fn will now print the stats only when there are labels present in the result-map. To not print the stats it's just a matter of using a different :reporter-fn that doesn't print the stats.
  • In the stats report only lines with labels are printed now. In the previous version there was a line with the percentage of trials with "No labels". I removed it now to make it less verbose.

Also docstrings were added, made it sure the stats work in clojurescript and other minor improvements.

Show
Nicolás Berger added a comment - - edited Uploaded a new patch that takes advantage of the new reporter-fn mechanism. In this new patch some stuff was simplified:
  • Removed the new report-stats var that was going to be used as a way to enable/disable printing of stats for each test. The default-reporter-fn will now print the stats only when there are labels present in the result-map. To not print the stats it's just a matter of using a different :reporter-fn that doesn't print the stats.
  • In the stats report only lines with labels are printed now. In the previous version there was a line with the percentage of trials with "No labels". I removed it now to make it less verbose.
Also docstrings were added, made it sure the stats work in clojurescript and other minor improvements.
Hide
Nicolás Berger added a comment - - edited

I'm thinking it would be nice to give more ways for assigning labels to trials. For example:

1. (classify prop): It just uses the vector of args as the label.
2. (classify prop label-fn): Obtains the label by applying a label-fn to the args. Example: (classify prop count) - assigns the count of the arg (a vector, string, etc) as the label
3. (classify prop label-fn pred): Applies label-fn only when pred yields a truthy value. Example: (classify prop count #(> (count %) 1)) - assigns the count of the arg as the label, but only when count is greater than 1.
4. (classify prop pred): Uses the vector of args as the label, but only when pred yields a truthy value. Example: (classify prop #(<= (count %) 1)) - assigns the count of the arg as the label, but only when count is lower or equal to 1.
5. (classify prop pred label): This is the current signature. Assigns label only when pred yields truthy

So I'm thinking about changing the signature to receive a map of (:pred :label-fn :label) possible keys. The three keys are optional. :label and :label-fn can't be both present at the same time.

From what I could understand, Haskell QuickCheck (https://hackage.haskell.org/package/QuickCheck-2.8.2/docs/Test-QuickCheck-Property.html) has different functions to provide similar alternatives:
(classify prop) would be similar to collect in Haskell QC
(classify prop label-fn) would be similar to label in Haskell QC
(classify prop label-fn pred) would be similar to classify in Haskell QC

What do you think @gfredericks?

Show
Nicolás Berger added a comment - - edited I'm thinking it would be nice to give more ways for assigning labels to trials. For example: 1. (classify prop): It just uses the vector of args as the label. 2. (classify prop label-fn): Obtains the label by applying a label-fn to the args. Example: (classify prop count) - assigns the count of the arg (a vector, string, etc) as the label 3. (classify prop label-fn pred): Applies label-fn only when pred yields a truthy value. Example: (classify prop count #(> (count %) 1)) - assigns the count of the arg as the label, but only when count is greater than 1. 4. (classify prop pred): Uses the vector of args as the label, but only when pred yields a truthy value. Example: (classify prop #(<= (count %) 1)) - assigns the count of the arg as the label, but only when count is lower or equal to 1. 5. (classify prop pred label): This is the current signature. Assigns label only when pred yields truthy So I'm thinking about changing the signature to receive a map of (:pred :label-fn :label) possible keys. The three keys are optional. :label and :label-fn can't be both present at the same time. From what I could understand, Haskell QuickCheck (https://hackage.haskell.org/package/QuickCheck-2.8.2/docs/Test-QuickCheck-Property.html) has different functions to provide similar alternatives: (classify prop) would be similar to collect in Haskell QC (classify prop label-fn) would be similar to label in Haskell QC (classify prop label-fn pred) would be similar to classify in Haskell QC What do you think @gfredericks?
Hide
Gary Fredericks added a comment -

After discussing it, my hunch is that 2 and 5 are the most natural ones, maybe calling 2 collect to mirror the haskell version.

We talked about maybe treating nil as a flag for not labelling in collect, but I don't particularly like that idea I don't think.

Show
Gary Fredericks added a comment - After discussing it, my hunch is that 2 and 5 are the most natural ones, maybe calling 2 collect to mirror the haskell version. We talked about maybe treating nil as a flag for not labelling in collect, but I don't particularly like that idea I don't think.
Hide
Gary Fredericks added a comment -

I should also not forget that this might overlap with changes to address the "Test Failure Feedback" issue on the confluence page: http://dev.clojure.org/display/design/test.check

Show
Gary Fredericks added a comment - I should also not forget that this might overlap with changes to address the "Test Failure Feedback" issue on the confluence page: http://dev.clojure.org/display/design/test.check
Hide
Gary Fredericks added a comment -

I should also not forget that this might overlap with changes to address the "Test Failure Feedback" issue on the confluence page: http://dev.clojure.org/display/design/test.check

Show
Gary Fredericks added a comment - I should also not forget that this might overlap with changes to address the "Test Failure Feedback" issue on the confluence page: http://dev.clojure.org/display/design/test.check
Hide
Nicolás Berger added a comment -

We talked about maybe treating nil as a flag for not labelling in collect, but I don't particularly like that idea I don't think.

Perhaps we can use a namespaced keyword to signal that a label should be ignored? Something like clojure.test.check.stats/ignore. This way we can easily implement classify in terms of collect by creating a function that returns stats/ignore when pred doesn't match:

(defn collect
  [prop label-fn]
  (gen/fmap
    (fn [{:keys [args] :as result-map}]
      (let [label (apply label-fn args)]
        (if (= ::ignore label)
          result-map
          (update result-map :labels conj label))))
    prop))

(defn classify
  [prop pred label]
  (collect prop (fn [& args]
                  (if (apply pred args)
                    label
                    ::ignore))))

Another option could be to add an extra arity in collect, to receive a flag about whether nil should be treated as a label or if it should be ignored. I like :stats/ignore more

Show
Nicolás Berger added a comment -
We talked about maybe treating nil as a flag for not labelling in collect, but I don't particularly like that idea I don't think.
Perhaps we can use a namespaced keyword to signal that a label should be ignored? Something like clojure.test.check.stats/ignore. This way we can easily implement classify in terms of collect by creating a function that returns stats/ignore when pred doesn't match:
(defn collect
  [prop label-fn]
  (gen/fmap
    (fn [{:keys [args] :as result-map}]
      (let [label (apply label-fn args)]
        (if (= ::ignore label)
          result-map
          (update result-map :labels conj label))))
    prop))

(defn classify
  [prop pred label]
  (collect prop (fn [& args]
                  (if (apply pred args)
                    label
                    ::ignore))))
Another option could be to add an extra arity in collect, to receive a flag about whether nil should be treated as a label or if it should be ignored. I like :stats/ignore more
Hide
Nicolás Berger added a comment -

Replaced with new patch that adds both stats/classify & stats/collect as discussed. I think this new patch implements what was discussed, covering cases 2 & 5 with function names analog to the haskell impl, leaving out any special treatment for nil (it is a valid label) and not adding :stats/ignore as I suggested in a previous comment.

Show
Nicolás Berger added a comment - Replaced with new patch that adds both stats/classify & stats/collect as discussed. I think this new patch implements what was discussed, covering cases 2 & 5 with function names analog to the haskell impl, leaving out any special treatment for nil (it is a valid label) and not adding :stats/ignore as I suggested in a previous comment.
Hide
Nicolás Berger added a comment -

Added a new patch TCHECK87-add-stats-feature-2.patch that is rebased on current master. I don't think it's the final version (I'd like to find a way to not pollute the main quickcheck loop with the labels stuff for example) but it's hopefully getting closer.

Show
Nicolás Berger added a comment - Added a new patch TCHECK87-add-stats-feature-2.patch that is rebased on current master. I don't think it's the final version (I'd like to find a way to not pollute the main quickcheck loop with the labels stuff for example) but it's hopefully getting closer.
Hide
Nicolás Berger added a comment -

Added new patch TCHECK87-add-stats-feature-3.patch rebased on current master with some squashed commits. Also fixes the print-stats test in ClojureScript.

Show
Nicolás Berger added a comment - Added new patch TCHECK87-add-stats-feature-3.patch rebased on current master with some squashed commits. Also fixes the print-stats test in ClojureScript.

People

Vote (0)
Watch (1)

Dates

  • Created:
    Updated: