Clojure

Clojure head holding bug

Details

  • Type: Defect Defect
  • Status: Closed Closed
  • Priority: Major Major
  • Resolution: Declined
  • Affects Version/s: Release 1.3
  • Fix Version/s: None
  • Component/s: None
  • Labels:
    None
  • Environment:
    java 6 jdk on linux, jre 6 on windows 7

Description

I've a head holding problem that I believe is a bug in clojure 1.3. I
wrote the following function to split a a lazy seq of strings across
files of x size:

(defn split-file
 ([path strs size]
    (trampoline split-file path (seq strs) size 0))
 ([path strs size part]
    (with-open [f (clojure.java.io/writer (str path "." part))]
      (loop [written 0, ss strs]
        (when ss
          (if (>= written size)
            #(split-file path ss size (inc part))
            (let [s (first ss)]
              (.write f s)
              (recur (+ written (.length s)) (next ss)))))))))

If I call the 3 arg version of the function:
(split-file "foo" (repeat 100000000 "blah blah blah") 100000000)

I see memory usage increases as I'm writing each file with the usual
gc slow down, then memory usage goes back down again as I get to a new
split file.

Memory usage is fine if I call the 4 arg version (which only writes
one part of the split file):
(split-file "foo" (repeat 100000000 "blah blah blah") 100000000 0)

I can also avoid the head holding problem by removing trampoline and
recursively calling split-file directly, but then those recursive
calls use up stack and don't close files until all calls complete

Activity

People

Vote (0)
Watch (1)

Dates

  • Created:
    Updated:
    Resolved: