Clojure

Indirect function calls through Var instances fail to clear locals

Details

  • Type: Defect Defect
  • Status: Open Open
  • Priority: Minor Minor
  • Resolution: Unresolved
  • Affects Version/s: Release 1.4
  • Fix Version/s: None
  • Component/s: None
  • Labels:
  • Environment:
    Probably all, but observed on Ubuntu 12.04, OpenJDK 6
  • Patch:
    Code
  • Approval:
    Vetted

Description

If you make a function call indirectly by invoking a Var object (which derefs itself and invokes the result), the invocation parameters remain in the thread's local stack for the duration of the function call, even though they are needed only long enough to be passed into the deref'd function. As a result, passing a lazy seq into a function invoked in its Var form may run out of memory if the seq is forced inside that function. For example:

(defn total [xs] (reduce + 0 xs))
(total (range 1000000000)) ; this works, though takes a while
(#'total (range 1000000000)) ; this dies with out of memory error

I can provide a patch if it would be useful. The fix should be trivial, something along the lines of wrapping each argN in clojure/lang/Var.java inside a Util.ret1(argN, argN = null) as is done in RestFn.java.

Activity

Hide
Spencer Tipping added a comment -

Sounds great, and the patch looks good too. Let me know if I need to do anything else.

Show
Spencer Tipping added a comment - Sounds great, and the patch looks good too. Let me know if I need to do anything else.
Hide
Timothy Baldridge added a comment -

And as a note, both examples in the original report now have extremely similar memory usages.

Show
Timothy Baldridge added a comment - And as a note, both examples in the original report now have extremely similar memory usages.
Hide
Timothy Baldridge added a comment - - edited

adding a patch. Since most of Clojure ends up running this code in one way or another, I'd assert that tests are included as part of the normal Clojure test process.

Patch simply calls Util.ret1(argx, argx=null) on all invoke calls.

Show
Timothy Baldridge added a comment - - edited adding a patch. Since most of Clojure ends up running this code in one way or another, I'd assert that tests are included as part of the normal Clojure test process. Patch simply calls Util.ret1(argx, argx=null) on all invoke calls.
Hide
Timothy Baldridge added a comment -

Couldn't reproduce the exception, but the 2nd example did chew through about 4x the amount of memory. Vetting.

Show
Timothy Baldridge added a comment - Couldn't reproduce the exception, but the 2nd example did chew through about 4x the amount of memory. Vetting.
Hide
Timothy Baldridge added a comment -

fixed typeo in example

Show
Timothy Baldridge added a comment - fixed typeo in example
Hide
Spencer Tipping added a comment -

Sorry, I typo'd the example. (defn total ...) should be (defn sum ...).

Show
Spencer Tipping added a comment - Sorry, I typo'd the example. (defn total ...) should be (defn sum ...).

People

Vote (0)
Watch (2)

Dates

  • Created:
    Updated: