core.async

Pub/sub leaks memory

Details

  • Type: Defect Defect
  • Status: Open Open
  • Priority: Major Major
  • Resolution: Unresolved
  • Affects Version/s: None
  • Fix Version/s: None
  • Component/s: None
  • Labels:
  • Environment:
  • Approval:
    Triaged

Description

The following code will cause OOME:

(require '[clojure.core.async :refer [chan close! <! pub sub unsub go timeout]])

(def p* (chan))

(def p (pub p* :topic))

(go
  (loop []
    (let [s (chan)
          t (rand-int Integer/MAX_VALUE)]
      (sub p t s)
      (<! (timeout 10))
      (unsub p t s)
      (close! s)
      (recur))))

(It grows slowly: to see the OOME in a reasonable amount of time, either give JVM very small
memory like 64m, or remove the timeout step.)

I tried to profile the code, and the reason seems to be that even though I
unsubed the channel from the port, something is still retained which causes
the heap to be used up.

Activity

Hide
Ziyang Hu added a comment -

Here is the problem:

https://github.com/clojure/core.async/blob/96de9a47ac511d9bb4309645a3bc594a2fc0c33a/src/main/clojure/clojure/core/async.clj#L826-L828

When unsub* is called, it just untaps the channel from the mult specified by the topic. The mult still remains in the atom called mults even if the mult has no taps left.

I can't think of a clean fix for this problem, since currently the channels which are tapping a mult aren't exposed, i.e., we currently have no way of knowing if a mult has any taps on it.

Show
Ziyang Hu added a comment - Here is the problem: https://github.com/clojure/core.async/blob/96de9a47ac511d9bb4309645a3bc594a2fc0c33a/src/main/clojure/clojure/core/async.clj#L826-L828 When unsub* is called, it just untaps the channel from the mult specified by the topic. The mult still remains in the atom called mults even if the mult has no taps left. I can't think of a clean fix for this problem, since currently the channels which are tapping a mult aren't exposed, i.e., we currently have no way of knowing if a mult has any taps on it.
Hide
Ghadi Shayban added a comment -

I also cannot think of a clean fix, as mults do not expose their registrants. (The notion of "current" is a concurrent system is subtle)
Besides saying (and perhaps amending the docstring) that pubs are indeed resources, their footprint grows by the # of seen topics, and that resources should almost always be bounded.

Show
Ghadi Shayban added a comment - I also cannot think of a clean fix, as mults do not expose their registrants. (The notion of "current" is a concurrent system is subtle) Besides saying (and perhaps amending the docstring) that pubs are indeed resources, their footprint grows by the # of seen topics, and that resources should almost always be bounded.
Hide
Daniel Compton added a comment -

This bit us too in https://github.com/apa512/clj-rethinkdb/issues/97. My suggestion for a docstring addition is:

pubs are resources, and their footprint grows by the number of seen topics. unpub does not reclaim this resource. You should not use pub to subscribe to an unbounded number of topics.

Are there any changes/suggestions? Would you like me to create a patch for this?

Show
Daniel Compton added a comment - This bit us too in https://github.com/apa512/clj-rethinkdb/issues/97. My suggestion for a docstring addition is:
pubs are resources, and their footprint grows by the number of seen topics. unpub does not reclaim this resource. You should not use pub to subscribe to an unbounded number of topics.
Are there any changes/suggestions? Would you like me to create a patch for this?
Hide
Daniel Compton added a comment -

Also, is a pub able to be garbage collected when it's no longer used? When I was doing some limited testing, it didn't seem like it was, but I may be wrong.

Show
Daniel Compton added a comment - Also, is a pub able to be garbage collected when it's no longer used? When I was doing some limited testing, it didn't seem like it was, but I may be wrong.
Hide
Daniel Compton added a comment -

A partial workaround (I think?) is to use

(unsub-all pub topic)
if you know you don't need a topic anymore.

Show
Daniel Compton added a comment - A partial workaround (I think?) is to use
(unsub-all pub topic)
if you know you don't need a topic anymore.

People

Vote (5)
Watch (5)

Dates

  • Created:
    Updated: