Go blocks leak memory
Description
Environment
clojure 1.7.0
core.async 0.1.346.0-17112a-alpha
Java HotSpot(TM) Client VM 1.8.0_31-b13
Attachments
- 17 Feb 2017, 06:35 PM
- 15 Feb 2017, 10:58 PM
- 14 Feb 2017, 10:54 AM
- 13 Feb 2017, 11:13 PM
- 17 Dec 2015, 04:17 PM
- 16 Dec 2015, 09:17 AM
- 15 Dec 2015, 03:34 PM
Activity
Alex Miller February 17, 2017 at 3:10 PM
Applied v5 version.
Nicola Mometto February 16, 2017 at 9:12 AM
yes, to showcase the two different transformations we do in case of primitive or non primitive type propagation. I'll expand on that in the ticket description so it's clearer what i meant
Alex Miller February 16, 2017 at 2:12 AM
Did you mean to have ^String and clojure.core/int in that example?
Nicola Mometto February 15, 2017 at 10:58 PM
Updated the patch to handle with primitive type hints.
In case of primitive type hints, rather than emitting
(let [^String a (a')] ..)we emit
(let [a (clojure.core/int (a'))] ..)as hinting is not valid and proper unboxing is required
Alex Miller February 15, 2017 at 9:36 PM
I am reopening this and reverting the change for the moment. This example fails when using Clojure 1.9.0-alpha12+ (anything after the commit for CLJ-1224).
(require '[clojure.core.async :refer (go)])
(defprotocol P
(x [p]))
(defrecord R [z]
P
(x [_]
(go
(loop []
(recur)))))
CompilerException java.lang.UnsupportedOperationException: Can't type hint a primitive localI believe this is due to the code in the patch related to applying meta to the local binding, specifically this comes into interaction with the code now generated in hasheq after CLJ-1224.
The following example, after running for a few minutes, generates an OutOfMemoryError.
(let [c (chan)] (go (while (<! c))) (let [vs (range)] (go (doseq [v vs] (>! c v)))))By contrast, the following example will run indefinitely without causing an OutOfMemoryError.
(let [c (chan)] (go (while (<! c))) (go (let [vs (range)] (doseq [v vs] (>! c v)))))The only significant difference I see between the two examples is that the
(range)is created outside the go block in the first example but is created inside the go block in the second example. It appears that the go block in the first example is referencingvsin such a way as to prevent if from being garbage-collected.This behavior is also be the cause of ASYNC-32.
Approach:
The attached patch applies a transformation from
(let [a large-seq] (loop [..] .. a ..))where the loop exemplifies the state machine loop created by `go`, into
(let [a large-seq a' (^:once fn* [] a)] (loop [..] .. (let [a (a')] ..) ..))which allows the closed over locals to cross into the state machine loop without getting held. While this might look unsafe, because of how the `go` loop state machine works, we're guaranteed that `(a')` will only be invoked on the first iteration of the loop.
The patch looks way more complex than it actually is, because it needs to deal with: renaming locals so that shadowing of special symbols works properly, preserving locals type hints and handle nested go blocks, where the value of `&env` will be provided by tools.analyzer rather than from Compiler.java
There are two different type propagations that we need to do, one in case of non primitive objects and one in the case of primitive values:
if we're propagating type info for a non primitive object, the transformation is as follow:
(let [a ^Klass my-obj] (loop [..] .. a ..)) -> (let [a ^Klass my-obj a' (^:once fn* [] a)] (loop [..] .. (let [^Klass a (a')] ..) ..))if we're propagating type info for a primitive local, a type hint won't do – we actually need to unbox the value out of the thunk, so the transformation will be:
(let [a (int x)] (loop [..] .. a ..)) -> (let [a (int x) a' (^:once fn* [] a)] (loop [..] .. (let [a (clojure.core/int (a'))] ..) ..))Patch: 0001-ASYNC-138-allow-for-clearing-of-closed-over-locals-v6.patch