http://dev.clojure.org/jira This file is an XML representation of an issue en-us 4.4 649 25-07-2011 http://dev.clojure.org/jira/browse/CLJ-829 Clojure <p>Clojure 1.2.1 and 1.3.0-beta1 both exhibit the following behavior:</p> <p>user&gt; (let [m (into {} (for [x (range 100000)] <span class="error">&#91;(rand) (rand)&#93;</span>))]<br/> (println (count (distinct (map hash (keys m)))))<br/> ((juxt count identity) (persistent!<br/> (reduce dissoc! (transient m) (keys m)))))<br/> 99999<br/> <span class="error">&#91;2 {0.42548900739367024 0.8725039567983159}&#93;</span></p> <p>We create a large transient map with random keys and values, and check to see how many unique hashcodes we get. Then, we iterate over all the keys, dissoc'ing each out of the transient map. The resulting map has one element in it (wrong - it should be empty, since we dissoc'ed all the keys), and reports its count as being two (wrong - not sure whether it should be zero or one given the other breakage). As far as I can tell, each duplicated hash value is represented once in the output map, and the map's count is the number of keys that hashed to something duplicated.</p> <p>The problem seems to be restricted to transients, as if we remove the transient/persistent! pair and use dissoc instead of dissoc!, the map is always empty.</p> <p>Inspired by discussion at <a href="http://groups.google.com/group/clojure/browse_thread/thread/313ac122667bb4b5/c3e7faa8635403f1">http://groups.google.com/group/clojure/browse_thread/thread/313ac122667bb4b5/c3e7faa8635403f1</a></p> CLJ-829

Transient hashmaps mishandle hash collisions

Defect Major Closed Completed Christophe Grand Alan Malloy Wed, 24 Aug 2011 03:30:21 -0500 Fri, 1 Mar 2013 12:47:05 -0600 Fri, 7 Oct 2011 09:12:36 -0500 Release 1.3 Release 1.4 0 3 <p>By the way, since this involves randomness, on occasion it doesn't fail. With the input as given it seems to fail around 80% of the time, but if you want to be sure to reproduce you can add another 0 to the input size.</p> <p>Thanks for the update. I was able to reproduce with the extra zero. Moving this ticket to Release.Next so that it will ship with 1.3.</p> <p>Sorry for the delay. Here is a fix and a reduced test case.</p> <p>I have checked behavior with an independent test</p> <p><a href="https://github.com/clojure/test.generative/commit/b1350235eb219b76f5b7c4cc21c8255f567892b3">https://github.com/clojure/test.generative/commit/b1350235eb219b76f5b7c4cc21c8255f567892b3</a></p> <p>which looks good, but don't have context to evaluate the code (particularly the array allocation) in the time I have available today.</p> <p>The array alloc looks suspicious:</p> <p>+ Object[] newArray = new Object<span class="error">&#91;2*(array.length+1)&#93;</span>; // make room for next assoc<br/> + System.arraycopy(array, 0, newArray, 0, 2*count);</p> <p>should it not be array.length + 2, (or 2*(count + 1), whichever?</p> <p>Thanks Rich for spotting this copy&amp;paste error.</p> <p>I attached an updated patch.</p> <p>The problem reported by Alan was double:</p> <ul> <li>the misplaced removedLeaf.val = removedLeaf was causing the global count to be incorrect</li> <li>the missing array copy in ensureEditable was causing the seq returned by (keys m) to be mutated (shortened) and this is why all values were not dissoced.</li> </ul> <p>Mutable code is hard, one should invent a cool language with sane state management <img class="emoticon" src="http://dev.clojure.org/jira/images/icons/emoticons/smile.gif" height="20" width="20" align="absmiddle" alt="" border="0"/></p> <p>New tests continue to pass.</p> <p>It seems to me that the allocation in the second path is too big by two in some cases (e.g. it gets called in the dissoc! path when the array needs to be copied, but not get bigger). But this might be considered innocuous. The same method is called in the assoc! path where the +2 is needed, so avoiding two objects worth of allocation would require a more substantial patch.</p> <p>The larger array allocation is similar to the one performed in BitmapIndexedNode.ensureEditable. My line of thought behind this heuristic is that the copying dominates the allocation and that I prefer one slightly larger array than having to allocates two arrays in case of growth.</p> <p>Anyway it's on collision nodes so it's a rare occurence: I won't bother arguing further if switching to a more conservative allocation helps the patch landing in 1.3. </p> Approval Ok Global Rank Patch Code and Test