core.logic

Make benchmark suite as easy to run as `lein test`

Details

  • Type: Enhancement Enhancement
  • Status: Open Open
  • Priority: Major Major
  • Resolution: Unresolved
  • Affects Version/s: None
  • Fix Version/s: None
  • Component/s: None
  • Labels:
    None

Description

`lein benchmark` (or some other non-lein-based incantation) prints a report listing the name of each benchmark and its timing.

Example:

$ lein benchmark
membero 2839
zebra 152738
$ lein benchmark comparison-report {:baseline "benchmark-5.9.2013-1"
                                    :runs 5
                                    :diffs-only true
                                    :threshold 25} ; only consider different if the delta is > 25 ms.                                     
membero +68ms
zebra -122ms
$ lein benchmark {:pretty true}
Thu May  9 11:21:41 PDT 2013

Linux mars 2.6.32-5-amd64 #1 SMP Fri Feb 15 15:39:52 UTC 2013 x86_64 GNU/Linux

java version "1.7.0_21"
Java(TM) SE Runtime Environment (build 1.7.0_21-b11)
Java HotSpot(TM) 64-Bit Server VM (build 23.21-b01, mixed mode)

membero     2839 ms
zebra     152738 ms

I haven't looked for any Clojure benchmarking libs, but ideally this would be a trivial script that automates the repetitive manual task of running benchmark tests. Unlike the test suite, we aren't looking for binary success or failure. Every run will generate unique results, so the script should accommodate a fuzzier comparison.

A "pretty" output that includes system info would be great for bug reports.

Activity

People

Vote (0)
Watch (0)

Dates

  • Created:
    Updated: