<< Back to previous view

[CLJ-883] Clojure head holding bug Created: 27/Nov/11  Updated: 01/Mar/13  Resolved: 27/Nov/12

Status: Closed
Project: Clojure
Component/s: None
Affects Version/s: Release 1.3
Fix Version/s: None

Type: Defect Priority: Major
Reporter: Gerrard McNulty Assignee: Unassigned
Resolution: Declined Votes: 0
Labels: None
Environment:

java 6 jdk on linux, jre 6 on windows 7



 Description   

I've a head holding problem that I believe is a bug in clojure 1.3. I
wrote the following function to split a a lazy seq of strings across
files of x size:

(defn split-file
 ([path strs size]
    (trampoline split-file path (seq strs) size 0))
 ([path strs size part]
    (with-open [f (clojure.java.io/writer (str path "." part))]
      (loop [written 0, ss strs]
        (when ss
          (if (>= written size)
            #(split-file path ss size (inc part))
            (let [s (first ss)]
              (.write f s)
              (recur (+ written (.length s)) (next ss)))))))))

If I call the 3 arg version of the function:
(split-file "foo" (repeat 100000000 "blah blah blah") 100000000)

I see memory usage increases as I'm writing each file with the usual
gc slow down, then memory usage goes back down again as I get to a new
split file.

Memory usage is fine if I call the 4 arg version (which only writes
one part of the split file):
(split-file "foo" (repeat 100000000 "blah blah blah") 100000000 0)

I can also avoid the head holding problem by removing trampoline and
recursively calling split-file directly, but then those recursive
calls use up stack and don't close files until all calls complete



 Comments   
Comment by a_strange_guy [ 16/Jan/12 4:15 AM ]

There is actually no problem with head retention in your code.
It's zust the garbage collector that doesn't collect the seq right away. If there were problems, then you would get a OutOfMemoryError pretty fast.

compare
(split-file "foo" (repeat 100000000 "blah blah blah") 100000000 0)
with
(doall (repeat 100000000 "blah blah blah"))
to see the difference

Comment by Timothy Baldridge [ 27/Nov/12 2:29 PM ]

Closing due to lack of reproducible tests. On my system I actually saw memory usage not move at all when I ran the demo code, and at one point the GC kicked in and the memory usage went down a bit.

Generated at Sun Apr 20 01:06:27 CDT 2014 using JIRA 4.4#649-r158309.