<< Back to previous view

[CLJ-1032] seque leaks threads from the send-off pool Created: 25/Jul/12  Updated: 01/Mar/13  Resolved: 19/Aug/12

Status: Closed
Project: Clojure
Component/s: None
Affects Version/s: None
Fix Version/s: Release 1.5

Type: Defect Priority: Major
Reporter: Alan Malloy Assignee: Unassigned
Resolution: Completed Votes: 5
Labels: None

Attachments: Text File seque.patch    
Patch: Code and Test
Approval: Ok

 Description   

Because a new (fill) action is created every time an item is (drain)ed, the LBQ gets loaded up with piles of EOS markers. If the output seq is consumed faster than the input sequence can produce items, then the original agent action does all of the filling in a single action, while the agent's queue continues to back up with (fill) actions. Eventually the entire sequence is consumed, and each queued action does a blocking .put of an EOS marker; once the queue has filled up, one of these actions blocks forever, since nobody will ever .take from the queue again.

The attached patch does two things:

  • Make sure to put exactly one EOS marker in the stream, by setting the agent's state to nil if and only if a .offer of EOS has been accepted.
  • Replace all occurrences of .put with .offer (and appropriate checking of the return value). This is necessary because if .put ever blocks, it's possible for the system to remain in that state forever (say, because the client stops consuming the queue), thus leading to the same leaked-thread scenario.

The diff is a little bit inconvenient to read because of indentation changes; https://gist.github.com/b7ecd4395a0d3d473de6 is an ignore-whitespace view of the patch for convenience.



 Comments   
Comment by Andy Fingerhut [ 26/Jul/12 3:50 AM ]

Sorry for me just triggering on the function "seque" without a deep understanding, but is this by any chance the same issue as described in CLJ-823? If so, since this one has a patch and that one doesn't, might be nice to mark CLJ-823 as a duplicate of this one.

Comment by Alan Malloy [ 26/Jul/12 4:48 AM ]

No, these are separate issues. CLJ-823 is just a special case of the general problem that seque is not usable from within an agent action. I have an implementation of seque using futures instead of agents so that this isn't a problem, but that has other problems of its own, specifically if you don't fully consume the seque you wind up leaking a future object.

Comment by Alan Malloy [ 19/Aug/12 8:11 PM ]

Marking as resolved since the patch has been applied in master.

Generated at Thu Apr 17 11:07:05 CDT 2014 using JIRA 4.4#649-r158309.