<< Back to previous view

[LOGIC-136] Make benchmark suite as easy to run as `lein test` Created: 09/May/13  Updated: 09/May/13

Status: Open
Project: core.logic
Component/s: None
Affects Version/s: None
Fix Version/s: None

Type: Enhancement Priority: Major
Reporter: Austin Haas Assignee: David Nolen
Resolution: Unresolved Votes: 0
Labels: None


 Description   

`lein benchmark` (or some other non-lein-based incantation) prints a report listing the name of each benchmark and its timing.

Example:

$ lein benchmark
membero 2839
zebra 152738
$ lein benchmark comparison-report {:baseline "benchmark-5.9.2013-1"
                                    :runs 5
                                    :diffs-only true
                                    :threshold 25} ; only consider different if the delta is > 25 ms.                                     
membero +68ms
zebra -122ms
$ lein benchmark {:pretty true}
Thu May  9 11:21:41 PDT 2013

Linux mars 2.6.32-5-amd64 #1 SMP Fri Feb 15 15:39:52 UTC 2013 x86_64 GNU/Linux

java version "1.7.0_21"
Java(TM) SE Runtime Environment (build 1.7.0_21-b11)
Java HotSpot(TM) 64-Bit Server VM (build 23.21-b01, mixed mode)

membero     2839 ms
zebra     152738 ms

I haven't looked for any Clojure benchmarking libs, but ideally this would be a trivial script that automates the repetitive manual task of running benchmark tests. Unlike the test suite, we aren't looking for binary success or failure. Every run will generate unique results, so the script should accommodate a fuzzier comparison.

A "pretty" output that includes system info would be great for bug reports.



 Comments   
Comment by David Nolen [ 09/May/13 1:44 PM ]

Sounds like an excellent enhancement to me. Patch welcome for this.

Generated at Fri Oct 24 19:03:58 CDT 2014 using JIRA 4.4#649-r158309.