Clojure

Clojure head holding bug

Details

  • Type: Defect Defect
  • Status: Closed Closed
  • Priority: Major Major
  • Resolution: Declined
  • Affects Version/s: Release 1.3
  • Fix Version/s: None
  • Component/s: None
  • Labels:
    None
  • Environment:
    java 6 jdk on linux, jre 6 on windows 7

Description

I've a head holding problem that I believe is a bug in clojure 1.3. I
wrote the following function to split a a lazy seq of strings across
files of x size:

(defn split-file
 ([path strs size]
    (trampoline split-file path (seq strs) size 0))
 ([path strs size part]
    (with-open [f (clojure.java.io/writer (str path "." part))]
      (loop [written 0, ss strs]
        (when ss
          (if (>= written size)
            #(split-file path ss size (inc part))
            (let [s (first ss)]
              (.write f s)
              (recur (+ written (.length s)) (next ss)))))))))

If I call the 3 arg version of the function:
(split-file "foo" (repeat 100000000 "blah blah blah") 100000000)

I see memory usage increases as I'm writing each file with the usual
gc slow down, then memory usage goes back down again as I get to a new
split file.

Memory usage is fine if I call the 4 arg version (which only writes
one part of the split file):
(split-file "foo" (repeat 100000000 "blah blah blah") 100000000 0)

I can also avoid the head holding problem by removing trampoline and
recursively calling split-file directly, but then those recursive
calls use up stack and don't close files until all calls complete

Activity

Hide
Timothy Baldridge added a comment -

Closing due to lack of reproducible tests. On my system I actually saw memory usage not move at all when I ran the demo code, and at one point the GC kicked in and the memory usage went down a bit.

Show
Timothy Baldridge added a comment - Closing due to lack of reproducible tests. On my system I actually saw memory usage not move at all when I ran the demo code, and at one point the GC kicked in and the memory usage went down a bit.
Hide
a_strange_guy added a comment -

There is actually no problem with head retention in your code.
It's zust the garbage collector that doesn't collect the seq right away. If there were problems, then you would get a OutOfMemoryError pretty fast.

compare
(split-file "foo" (repeat 100000000 "blah blah blah") 100000000 0)
with
(doall (repeat 100000000 "blah blah blah"))
to see the difference

Show
a_strange_guy added a comment - There is actually no problem with head retention in your code. It's zust the garbage collector that doesn't collect the seq right away. If there were problems, then you would get a OutOfMemoryError pretty fast. compare (split-file "foo" (repeat 100000000 "blah blah blah") 100000000 0) with (doall (repeat 100000000 "blah blah blah")) to see the difference

People

Vote (0)
Watch (1)

Dates

  • Created:
    Updated:
    Resolved: