test.check

Metadata for tuning on generated values

Details

  • Type: Enhancement Enhancement
  • Status: Open Open
  • Priority: Minor Minor
  • Resolution: Unresolved
  • Affects Version/s: None
  • Fix Version/s: None
  • Component/s: None

Description

It can be really useful for optimization of generative testing(with specs) if values outputted by generators included attempts and time as metadata.
Example:

(-> (s/exercise string? 1)
first
meta)
;; => {:time-ms 2, :attemts 1}

It can help in finding slowest generators which can benefit the most from customization.

Activity

Hide
Alex Miller added a comment -

I move this to test.check as that’s where the generators live.

Note that not every value can carry metadata (number, strings, etc) but it’s an interesting idea.

Show
Alex Miller added a comment - I move this to test.check as that’s where the generators live. Note that not every value can carry metadata (number, strings, etc) but it’s an interesting idea.
Hide
JAre added a comment - - edited

> Note that not every value can carry metadata (number, strings, etc) but it’s an interesting idea.

It's not that big of a deal since the metadata will be directly useful mostly with recursive/nested specs for spec tooling devs. And a person that wants to use it for this can work around the limitation by wrapping primitives.

For the end-user API something similar to spec/explain(s/explain-data, s/explain-str..) would be great.
I'll call it s/complexity, the function takes a spec and returns (in the case of s/complexity-data) structure of the same shape as the original spec filled with generation complexity data (average time, attempts, mb the spec form...) It should handle too complex specs in a reasonable way i.e. Instead of throwing (as with generators) it preferably reports which part of the spec is too complex and what is individual complexity of its elements.

Show
JAre added a comment - - edited > Note that not every value can carry metadata (number, strings, etc) but it’s an interesting idea. It's not that big of a deal since the metadata will be directly useful mostly with recursive/nested specs for spec tooling devs. And a person that wants to use it for this can work around the limitation by wrapping primitives. For the end-user API something similar to spec/explain(s/explain-data, s/explain-str..) would be great. I'll call it s/complexity, the function takes a spec and returns (in the case of s/complexity-data) structure of the same shape as the original spec filled with generation complexity data (average time, attempts, mb the spec form...) It should handle too complex specs in a reasonable way i.e. Instead of throwing (as with generators) it preferably reports which part of the spec is too complex and what is individual complexity of its elements.
Hide
JAre added a comment -

Also `s/complexity` probably should have per node(spec element) timeout (mb configurable) since it's mostly debugging/profiling tool.

Show
JAre added a comment - Also `s/complexity` probably should have per node(spec element) timeout (mb configurable) since it's mostly debugging/profiling tool.
Hide
Gary Fredericks added a comment -

What does "attempts" mean here? Does it only apply to gen/such-that generators?

Show
Gary Fredericks added a comment - What does "attempts" mean here? Does it only apply to gen/such-that generators?
Hide
JAre added a comment -

Yeah I in the context of specs. How many times generator had to retry before it managed to satisfy spec.

Show
JAre added a comment - Yeah I in the context of specs. How many times generator had to retry before it managed to satisfy spec.
Hide
Gary Fredericks added a comment -

I just noticed the :time-ms aspect of this, which would presumably apply to a lot more than just such-that.

I agree that being able to easily debug generator performance would be useful, but I don't think it's obvious how this approach applies to some of the combinators, like gen/fmap and gen/bind. I'd have an easier time considering a more concrete proposal, that includes details for those kinds of cases.

Show
Gary Fredericks added a comment - I just noticed the :time-ms aspect of this, which would presumably apply to a lot more than just such-that. I agree that being able to easily debug generator performance would be useful, but I don't think it's obvious how this approach applies to some of the combinators, like gen/fmap and gen/bind. I'd have an easier time considering a more concrete proposal, that includes details for those kinds of cases.
Hide
JAre added a comment -

but I don't think it's obvious how this approach applies to some of the combinators, like gen/fmap

The metadata shouldn't be available from withing the function that gen/fmap applies. Instead the time spent inside the function and other performance stats should be assoc-ed with the gen/fmap inputs stats and then returned attached to the outputs. So the data will be carried by the context between nested gen/ calls.

The metadata shouldn't be available from withing the function that gen/fmap calls.

Alternatively you can allow to peak into the inputs stats but not alter it. Not sure about that. It may be too confusing.

Show
JAre added a comment -
but I don't think it's obvious how this approach applies to some of the combinators, like gen/fmap
The metadata shouldn't be available from withing the function that gen/fmap applies. Instead the time spent inside the function and other performance stats should be assoc-ed with the gen/fmap inputs stats and then returned attached to the outputs. So the data will be carried by the context between nested gen/ calls.
The metadata shouldn't be available from withing the function that gen/fmap calls.
Alternatively you can allow to peak into the inputs stats but not alter it. Not sure about that. It may be too confusing.
Hide
JAre added a comment - - edited

Actually if you allow peeking it might be used not only for debugging but also to alter the generation for performance.
For amortisation of the cumulative generator complexity, to meet some deadline by switching routes.

  • - - - -

@gfredericks For some reason I can't add a new comments to any issue so I update the old one.

For:

(def g1 ...)

(def g2 (gen/fmap f g1))

complexity function should output

(complexit g1) => {:name <generator-name> :time-ms 10}

(complexit g2) => {:name :fmap :time-ms 5 :args [{:name <generator-name> :time-ms 10}]}

here :time-ms 5 is time spent inside f function.

Show
JAre added a comment - - edited Actually if you allow peeking it might be used not only for debugging but also to alter the generation for performance. For amortisation of the cumulative generator complexity, to meet some deadline by switching routes.
  • - - - -
@gfredericks For some reason I can't add a new comments to any issue so I update the old one. For:
(def g1 ...)

(def g2 (gen/fmap f g1))
complexity function should output
(complexit g1) => {:name <generator-name> :time-ms 10}

(complexit g2) => {:name :fmap :time-ms 5 :args [{:name <generator-name> :time-ms 10}]}
here :time-ms 5 is time spent inside f function.
Hide
Gary Fredericks added a comment -

For example, let's say I have:

(def g1 ...)

(def g2 (gen/fmap f g1))

When I generate a value from g2, what kind of information will I get? Will it be about g1, or about f, or about both somehow?

Show
Gary Fredericks added a comment - For example, let's say I have:
(def g1 ...)

(def g2 (gen/fmap f g1))
When I generate a value from g2, what kind of information will I get? Will it be about g1, or about f, or about both somehow?
Hide
Gary Fredericks added a comment -

JAre I believe your comment permissions are fixed.

Okay, I can see that it might be a well-defined concept. It would be a non-trivial amount of work, though, and I'd want to be sure that there wouldn't be a major performance difference. If there is, we'd probably want a way to have it off by default, and make sure that there's not a performance hit even when off.

Show
Gary Fredericks added a comment - JAre I believe your comment permissions are fixed. Okay, I can see that it might be a well-defined concept. It would be a non-trivial amount of work, though, and I'd want to be sure that there wouldn't be a major performance difference. If there is, we'd probably want a way to have it off by default, and make sure that there's not a performance hit even when off.
Hide
JAre added a comment -

Yeah works now. Thx.

Show
JAre added a comment - Yeah works now. Thx.

People

  • Assignee:
    Unassigned
    Reporter:
    JAre
Vote (0)
Watch (0)

Dates

  • Created:
    Updated: