http://dev.clojure.org/jira This file is an XML representation of an issue en-us 4.4 649 25-07-2011 http://dev.clojure.org/jira/browse/CLJS-178 ClojureScript <p><tt>0001-PersistentHashMap-ported-from-Clojure.patch</tt> contains a port of Clojure's <tt>PersistentHashMap</tt>.</p> <p><tt>0002-...-.patch</tt> rewires <tt>cljs.core/hash-map</tt> and <tt>cljs.compiler/emit</tt>'s <tt>:map</tt> method to use <tt>cljs.core.PersistentHashMap</tt>. With that in place, the regular cljs test suite runs fine.</p> <p>As a quick sanity check,</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>(loop [m cljs.core.PersistentHashMap/EMPTY i 0] (if (&lt; i 10000) (recur (assoc m i :foo) (inc i)) (loop [i 0] (if (&lt; i 10000) (do (assert (= :foo (get m i))) (recur (inc i))))))) </pre> </div></div> <p>runs fine under Rhino. Timing this loop produces the expected results: it takes 1.1s on my machine as written above and 10.5s with cljs.core.HashMap/EMPTY bound to m. With 100000 iterations, PersistentHashMap takes 11.8s, whereas HashMap takes forever.</p> <p>TODO:</p> <ul> <li>iron out any remaining bugs, add in missing protocols (if any) etc.,</li> <li>test HashCollisionNode and related code,</li> <li>use other map types for small maps and switch at some threshold.</li> </ul> <p>See <a href="https://groups.google.com/d/topic/clojure-dev/DMcV4QwApuY/discussion">https://groups.google.com/d/topic/clojure-dev/DMcV4QwApuY/discussion</a>.</p> CLJS-178

PersistentHashMap implementation

Enhancement Major Resolved Completed Unassigned Michał Marczyk Wed, 11 Apr 2012 18:09:48 -0500 Thu, 19 Apr 2012 22:41:18 -0500 Thu, 19 Apr 2012 22:41:18 -0500 0 2 <p>Note that I'm developing this code on a branch at</p> <p><a href="https://github.com/michalmarczyk/clojurescript/tree/persistent-hash-map">https://github.com/michalmarczyk/clojurescript/tree/persistent-hash-map</a></p> <p>The branch may be ahead of the patch here (as it is at the time of this writing); I'll update the patch periodically.</p> <p>The rewiring gets tested on a separate branch:</p> <p><a href="https://github.com/michalmarczyk/clojurescript/tree/persistent-hash-map-tests">https://github.com/michalmarczyk/clojurescript/tree/persistent-hash-map-tests</a></p> <p>Some notes after a quick look. Don't use js*. We have (make-array n) now.</p> <p>Don't create a protocol for the node interface. Protocols are nice but introduce overhead. Just add those methods via Object.</p> <p>Thanks! Updated both branches with these changes, plus a new bit-shift-right-zero-fill function in core (with accompanying compiler macro).</p> <p>Please attach new patches here with the changes - thanks.</p> <p>I also notice that you're using js* in a couple of other places to inline array constructors. But array is already a compiler macro - again no need for js*. </p> <p>These patches bring master in line with my persistent-hash-map branch; the "rewiring" are coming in a moment.</p> <p>Ah, actually the original rewiring patch still works.</p> <p>@David Nolen: Thanks for the review! No js* left in PHM now.</p> <p>Incidentally, I've run a couple of tests of hash collision handling &#8211; seems to work fine so far. Also, the way sets are implemented in cljs, they too are persistent with the PHM impl wired for use in cljs.core/hash-map.</p> <p>One more thing: PHM needs a function to count bits set in a given (presumably integral) number, which I've introduced under the name of cljs.core/bit-count. It is currently not marked private, but it resides in the PHM section of core.cljs. I think it might be generally useful, so I'm inclined to add a docstring and add it to the bit ops section; otherwise it should probably be marked private.</p> <p>Cool. Can you please construct a single patch with all the changes, thanks.</p> <p>@David Nolen: Sure. The new patch includes all changes (in particular it wires up the compiler and hash-map to use PHM) and places bit-count in the bit-ops section (with a docstring).</p> <p>Please remove any code behind #_. Looking more closely at your code, why are you using atoms? Is the added-leaf field only set once per node? If so why not a ^:mutable field for those deftypes. </p> <p><tt>added-leaf?</tt> is set at most once per an `assoc` to a PHM to signal that the new PHM needs to have an incremented count in relation to the original. Clojure's impl uses mutable Boxes for this purpose. Looking at this again, I guess a singleton array would make more sense than an atom; I'll make the change.</p> <p>Sounds good.</p> <p>All in one.</p> <p>Looks good! I ran some quick perf tests - access times on small maps vs. ObjMap is actually a significant performance hit. I'll review the patch more thoroughly. I do note that update performance on V8 is pretty stellar.</p> <p>Fantastic work! I think we're getting very close.</p> <p>I've got an updated version on a branch here:</p> <p><a href="https://github.com/michalmarczyk/clojurescript/tree/phm-wip-2">https://github.com/michalmarczyk/clojurescript/tree/phm-wip-2</a></p> <p>Unfortunately I've got to dash right now, so I'll post a new monolithic patch sometime later. I'm closing in on a transient hash map implementation too &#8211; it should be ready sometime <b>very</b> soon now.</p> <p>Many thanks for the review and all the helpful tips, David! The latest branch incorporates some of the changes you suggested, with others due to land next time I'm at the keyboard.</p> <p>The latest changes are available in two branches on my fork, <tt>phm</tt> and <tt>phm-transient</tt>.</p> <p><tt>phm</tt> is just the old monolithic commit. <tt>phm-transient</tt>, on the other hand, introduces TransientHashMap, TransientObjMap, various performance-related fixes (as per David's suggestions; one thing I haven't got around to yet is the alternative bit-count impl &#8211; will do that soon) and I believe even a bug fix or two. It's available for viewing at</p> <p><a href="https://github.com/michalmarczyk/clojurescript/tree/phm-transient">https://github.com/michalmarczyk/clojurescript/tree/phm-transient</a></p> <p>I'll post the patch here in a moment. I'll also rip it apart into single-concern commits if appropriate &#8211; I'm afraid I got into the flow on this during a longish train ride, so it's another monolithic commit for now.</p> <p>Meant to be applied on top of the previous monolithic patch.</p> <p>Yes please move the transient work into a separate patch, thanks!</p> <p>Ok, no problem. Will do that in the next batch sometime soon, since again I need to dash right now. Should any bugs surface in the meantime, I'll fix them as soon as possible too.</p> <p>PHM without the transient stuff available at</p> <p><a href="https://github.com/michalmarczyk/clojurescript/tree/phm2">https://github.com/michalmarczyk/clojurescript/tree/phm2</a></p> <p>Will attach patch in a moment.</p> <p>Here's the patch. Applies cleanly onto the current master.</p> <p>Before moving forward with this it would be useful to get a jsperf benchmark comparing performance to ObjMap and HashMap for big/small, access/update, string/non-string keys. Mind putting this together? Once we have that we can pick a strategy that won't slow down existing programs.</p> <p>Here's a first cut at some tests... but I must have something or other mixed up, because the results are crazy. Any hints will be greatly appreciated.</p> <p>This is the code used by the test cases:</p> <p><a href="https://github.com/michalmarczyk/cljs-persistent-hash-map-test">https://github.com/michalmarczyk/cljs-persistent-hash-map-test</a></p> <p>And here is the current batch of tests:</p> <p><a href="http://jsperf.com/cljs-persistent-hash-map-tiny-assoc">http://jsperf.com/cljs-persistent-hash-map-tiny-assoc</a><br/> <a href="http://jsperf.com/cljs-persistent-hash-map-small-assoc">http://jsperf.com/cljs-persistent-hash-map-small-assoc</a><br/> <a href="http://jsperf.com/cljs-persistent-hash-map-large-assoc">http://jsperf.com/cljs-persistent-hash-map-large-assoc</a><br/> <a href="http://jsperf.com/cljs-persistent-hash-map-access">http://jsperf.com/cljs-persistent-hash-map-access</a></p> <p>A further pair of test cases for associng non-string keys:</p> <p><a href="http://jsperf.com/cljs-persistent-hash-map-assoc-non-string">http://jsperf.com/cljs-persistent-hash-map-assoc-non-string</a><br/> <a href="http://jsperf.com/cljs-persistent-hash-map-large-assoc-non-string">http://jsperf.com/cljs-persistent-hash-map-large-assoc-non-string</a></p> <p>Also, to make the earlier comment clearer &#8211; the results are crazy, because in Chrome PHM just totally trounces everything, even at small map sizes.</p> <p>I have a new branch based off of the latest PHM code and including transient support for PHM (but not ObjMap &#8211; leaving that on the previous phm-transient branch for now).</p> <p>Here's the branch:</p> <p><a href="https://github.com/michalmarczyk/clojurescript/tree/phm2-transient">https://github.com/michalmarczyk/clojurescript/tree/phm2-transient</a></p> <p>And here's a jsPerf test (with transient/assoc!/persistent! being twice as fast to build a PHM of 10000 entries than assoc in Chrome):</p> <p><a href="http://jsperf.com/cljs-persistent-hash-map-transient-support">http://jsperf.com/cljs-persistent-hash-map-transient-support</a></p> <p>Here's the patch (applies cleanly on top of the previous PHM patch).</p> <p>I've updated the test with slightly modified preparation code (I used a version with cljs.core/hash-map not using transient &#8211; no difference in this case, but still, now it's using the actual code from the patch).</p> <p>Please a create a new ticket and put the transient patch there, thanks!</p> <p>Makes sense, will do right now.</p> <p>Done:</p> <p><a href="http://dev.clojure.org/jira/browse/CLJS-181">http://dev.clojure.org/jira/browse/CLJS-181</a></p> <p>There is an issue which hasn't occurred to me previously &#8211; a lot of code (including TwitterBuzz) will simply break with PHM wired in due to the way in which people use <tt>-strobj</tt>. Of course nobody ever promised <tt>-strobj</tt> to work... But a replacement would be nice.</p> <p>For the sort of uses of <tt>-strobj</tt> I see at first glance, making <tt>js-obj</tt> accept varargs like <tt>hash-map</tt> does (and <tt>array</tt>!) and construct a JS object with appropriate entries would be an adequate solution; the rest could be expected to use a conversion function.</p> <p>If breaking <tt>-strobj</tt> proves to be a major problem, though, I do have code ready (on an older branch previously linked to from here, phm-transient) which causes ObjMaps to be constructed for small maps with string keys (then a PHM is built using a transient when a non-string key is encountered or the OM grows too large).</p> <p>Added a test case for 6 key `assoc` @ <a href="http://jsperf.com/cljs-persistent-hash-map-miniscule-assoc">http://jsperf.com/cljs-persistent-hash-map-miniscule-assoc</a>. PHM still wins in Chrome and FF1 and loses by a reasonable margin to ObjMap on Iceweasel.</p> <p>Also, for completeness, here's a 20 key, "mutable is fine" `assoc` / `aset` test with JS object included: <a href="http://jsperf.com/cljs-persistent-hash-map-vs-js-object-small">http://jsperf.com/cljs-persistent-hash-map-vs-js-object-small</a>, where unsurprisingly JS objects dominate. (Note that transients are <b>a lot</b> faster than PHM for building large maps &#8211; see the test linked to in a previous comment, <a href="http://jsperf.com/cljs-persistent-hash-map-transient-support">http://jsperf.com/cljs-persistent-hash-map-transient-support</a>.)</p> <p>4 key `assoc`: <a href="http://jsperf.com/cljs-persistent-hash-map-minuscule-assoc">http://jsperf.com/cljs-persistent-hash-map-minuscule-assoc</a>.<br/> 8 key `assoc`: <a href="http://jsperf.com/cljs-persistent-hash-map-teeny-assoc">http://jsperf.com/cljs-persistent-hash-map-teeny-assoc</a>.</p> <p>Same results, basically.</p> <p>This great. For completeness can we compare against defrecord w/ 4 keys? Thanks!</p> <p>Sure, here's a test:</p> <p><a href="http://jsperf.com/cljs-persistent-hash-map-vs-records-minuscule">http://jsperf.com/cljs-persistent-hash-map-vs-records-minuscule</a></p> <p>Initial test run suggests that a record is significantly faster for {assoc} onto fields and significantly slower if using {_<em>extmap} (NB. here the {</em>_extmap} is itself a PHM).</p> <p>fixed, <a href="https://github.com/clojure/clojurescript/commit/ef0d52541b02bcf767f2f54f497c8808c6afa91d">https://github.com/clojure/clojurescript/commit/ef0d52541b02bcf767f2f54f497c8808c6afa91d</a></p> Global Rank Patch Code