[CLJ-883] Clojure head holding bug

Fri, 1 Mar 2013 12:46:59 -0600

I've a head holding problem that I believe is a bug in clojure 1.3. I
wrote the following function to split a a lazy seq of strings across
files of x size:

(defn split-file
 ([path strs size]
    (trampoline split-file path (seq strs) size 0))
 ([path strs size part]
    (with-open [f (clojure.java.io/writer (str path "." part))]
      (loop [written 0, ss strs]
        (when ss
          (if (>= written size)
            #(split-file path ss size (inc part))
            (let [s (first ss)]
              (.write f s)
              (recur (+ written (.length s)) (next ss)))))))))

If I call the 3 arg version of the function:
(split-file "foo" (repeat 100000000 "blah blah blah") 100000000)

I see memory usage increases as I'm writing each file with the usual
gc slow down, then memory usage goes back down again as I get to a new
split file.

Memory usage is fine if I call the 4 arg version (which only writes
one part of the split file):
(split-file "foo" (repeat 100000000 "blah blah blah") 100000000 0)

I can also avoid the head holding problem by removing trampoline and
recursively calling split-file directly, but then those recursive
calls use up stack and don't close files until all calls complete

Clojure JIRA

[CLJ-883] Clojure head holding bug