Clojure

Indirect function calls through Var instances fail to clear locals

Details

  • Type: Defect Defect
  • Status: Closed Closed
  • Priority: Minor Minor
  • Resolution: Completed
  • Affects Version/s: Release 1.4
  • Fix Version/s: Release 1.6
  • Component/s: None
  • Labels:
  • Environment:
    Probably all, but observed on Ubuntu 12.04, OpenJDK 6
  • Patch:
    Code
  • Approval:
    Ok

Description

If you make a function call indirectly by invoking a Var object (which derefs itself and invokes the result), the invocation parameters remain in the thread's local stack for the duration of the function call, even though they are needed only long enough to be passed into the deref'd function. As a result, passing a lazy seq into a function invoked in its Var form may run out of memory if the seq is forced inside that function. For example:

(defn total [xs] (reduce + 0 xs))
(total (range 1000000000))   ; this works, though takes a while
(#'total (range 1000000000)) ; this dies with out of memory error

Solution: Similar to RestFn, wrap each argN in Var inside a Util.ret1(argN, argN = null).

Patch: var-clear-locals-patch-v2.diff

Screened by: Alex Miller

  1. var-clear-locals.diff
    29/Nov/12 2:57 PM
    19 kB
    Timothy Baldridge
  2. var-clear-locals-patch-v2.diff
    22/Oct/13 9:13 AM
    18 kB
    Alex Miller
  3. var-clear-locals-patch-v2.txt
    05/Sep/13 6:18 PM
    18 kB
    Andy Fingerhut

Activity

Hide
Spencer Tipping added a comment -

Sorry, I typo'd the example. (defn total ...) should be (defn sum ...).

Show
Spencer Tipping added a comment - Sorry, I typo'd the example. (defn total ...) should be (defn sum ...).
Timothy Baldridge made changes -
Field Original Value New Value
Assignee Timothy Baldridge [ halgari ]
Hide
Timothy Baldridge added a comment -

fixed typeo in example

Show
Timothy Baldridge added a comment - fixed typeo in example
Timothy Baldridge made changes -
Description If you make a function call indirectly by invoking a Var object (which derefs itself and invokes the result), the invocation parameters remain in the thread's local stack for the duration of the function call, even though they are needed only long enough to be passed into the deref'd function. As a result, passing a lazy seq into a function invoked in its Var form may run out of memory if the seq is forced inside that function. For example:

(defn total [xs] (reduce + 0 xs))
(sum (range 1000000000)) ; this works, though takes a while
(#'sum (range 1000000000)) ; this dies with out of memory error

I can provide a patch if it would be useful. The fix should be trivial, something along the lines of wrapping each argN in clojure/lang/Var.java inside a Util.ret1(argN, argN = null) as is done in RestFn.java.
If you make a function call indirectly by invoking a Var object (which derefs itself and invokes the result), the invocation parameters remain in the thread's local stack for the duration of the function call, even though they are needed only long enough to be passed into the deref'd function. As a result, passing a lazy seq into a function invoked in its Var form may run out of memory if the seq is forced inside that function. For example:

(defn total [xs] (reduce + 0 xs))
(total (range 1000000000)) ; this works, though takes a while
(#'total (range 1000000000)) ; this dies with out of memory error

I can provide a patch if it would be useful. The fix should be trivial, something along the lines of wrapping each argN in clojure/lang/Var.java inside a Util.ret1(argN, argN = null) as is done in RestFn.java.
Hide
Timothy Baldridge added a comment -

Couldn't reproduce the exception, but the 2nd example did chew through about 4x the amount of memory. Vetting.

Show
Timothy Baldridge added a comment - Couldn't reproduce the exception, but the 2nd example did chew through about 4x the amount of memory. Vetting.
Timothy Baldridge made changes -
Approval Vetted [ 10003 ]
Assignee Timothy Baldridge [ halgari ]
Timothy Baldridge made changes -
Assignee Timothy Baldridge [ halgari ]
Hide
Timothy Baldridge added a comment - - edited

adding a patch. Since most of Clojure ends up running this code in one way or another, I'd assert that tests are included as part of the normal Clojure test process.

Patch simply calls Util.ret1(argx, argx=null) on all invoke calls.

Show
Timothy Baldridge added a comment - - edited adding a patch. Since most of Clojure ends up running this code in one way or another, I'd assert that tests are included as part of the normal Clojure test process. Patch simply calls Util.ret1(argx, argx=null) on all invoke calls.
Timothy Baldridge made changes -
Attachment var-clear-locals.diff [ 11729 ]
Timothy Baldridge made changes -
Assignee Timothy Baldridge [ halgari ]
Timothy Baldridge made changes -
Patch Code [ 10001 ]
Hide
Timothy Baldridge added a comment -

And as a note, both examples in the original report now have extremely similar memory usages.

Show
Timothy Baldridge added a comment - And as a note, both examples in the original report now have extremely similar memory usages.
Hide
Spencer Tipping added a comment -

Sounds great, and the patch looks good too. Let me know if I need to do anything else.

Show
Spencer Tipping added a comment - Sounds great, and the patch looks good too. Let me know if I need to do anything else.
Hide
Alex Miller added a comment -

Switching back to Triaged as afaict Rich has not Vetted this one.

Show
Alex Miller added a comment - Switching back to Triaged as afaict Rich has not Vetted this one.
Alex Miller made changes -
Approval Vetted [ 10003 ] Triaged [ 10120 ]
Rich Hickey made changes -
Approval Triaged [ 10120 ] Vetted [ 10003 ]
Fix Version/s Release 1.6 [ 10157 ]
Hide
Andy Fingerhut added a comment -

Patch var-clear-locals-patch-v2.txt is identical to var-clear-locals.diff (and preserves credit to its author), except it eliminates trailing whitespace in some added lines that cause git to give warnings when applying the patch.

Show
Andy Fingerhut added a comment - Patch var-clear-locals-patch-v2.txt is identical to var-clear-locals.diff (and preserves credit to its author), except it eliminates trailing whitespace in some added lines that cause git to give warnings when applying the patch.
Andy Fingerhut made changes -
Attachment var-clear-locals-patch-v2.txt [ 12238 ]
Alex Miller made changes -
Approval Vetted [ 10003 ] Screened [ 10004 ]
Description If you make a function call indirectly by invoking a Var object (which derefs itself and invokes the result), the invocation parameters remain in the thread's local stack for the duration of the function call, even though they are needed only long enough to be passed into the deref'd function. As a result, passing a lazy seq into a function invoked in its Var form may run out of memory if the seq is forced inside that function. For example:

(defn total [xs] (reduce + 0 xs))
(total (range 1000000000)) ; this works, though takes a while
(#'total (range 1000000000)) ; this dies with out of memory error

I can provide a patch if it would be useful. The fix should be trivial, something along the lines of wrapping each argN in clojure/lang/Var.java inside a Util.ret1(argN, argN = null) as is done in RestFn.java.
If you make a function call indirectly by invoking a Var object (which derefs itself and invokes the result), the invocation parameters remain in the thread's local stack for the duration of the function call, even though they are needed only long enough to be passed into the deref'd function. As a result, passing a lazy seq into a function invoked in its Var form may run out of memory if the seq is forced inside that function. For example:

{code}
(defn total [xs] (reduce + 0 xs))
(total (range 1000000000)) ; this works, though takes a while
(#'total (range 1000000000)) ; this dies with out of memory error
{code}

*Solution:* Similar to RestFn, wrap each argN in Var inside a Util.ret1(argN, argN = null).

*Patch:* var-clear-locals-patch-v2.txt

*Screened by:* Alex Miller
Alex Miller made changes -
Description If you make a function call indirectly by invoking a Var object (which derefs itself and invokes the result), the invocation parameters remain in the thread's local stack for the duration of the function call, even though they are needed only long enough to be passed into the deref'd function. As a result, passing a lazy seq into a function invoked in its Var form may run out of memory if the seq is forced inside that function. For example:

{code}
(defn total [xs] (reduce + 0 xs))
(total (range 1000000000)) ; this works, though takes a while
(#'total (range 1000000000)) ; this dies with out of memory error
{code}

*Solution:* Similar to RestFn, wrap each argN in Var inside a Util.ret1(argN, argN = null).

*Patch:* var-clear-locals-patch-v2.txt

*Screened by:* Alex Miller
If you make a function call indirectly by invoking a Var object (which derefs itself and invokes the result), the invocation parameters remain in the thread's local stack for the duration of the function call, even though they are needed only long enough to be passed into the deref'd function. As a result, passing a lazy seq into a function invoked in its Var form may run out of memory if the seq is forced inside that function. For example:

{code}
(defn total [xs] (reduce + 0 xs))
(total (range 1000000000)) ; this works, though takes a while
(#'total (range 1000000000)) ; this dies with out of memory error
{code}

*Solution:* Similar to RestFn, wrap each argN in Var inside a Util.ret1(argN, argN = null).

*Patch:* var-clear-locals-patch-v2.diff

*Screened by:* Alex Miller
Attachment var-clear-locals-patch-v2.diff [ 12366 ]
Rich Hickey made changes -
Approval Screened [ 10004 ] Ok [ 10007 ]
Stuart Halloway made changes -
Resolution Completed [ 1 ]
Status Open [ 1 ] Closed [ 6 ]

People

Vote (0)
Watch (2)

Dates

  • Created:
    Updated:
    Resolved: