seque leaks threads from the send-off pool

Description

Because a new (fill) action is created every time an item is (drain)ed, the LBQ gets loaded up with piles of EOS markers. If the output seq is consumed faster than the input sequence can produce items, then the original agent action does all of the filling in a single action, while the agent's queue continues to back up with (fill) actions. Eventually the entire sequence is consumed, and each queued action does a blocking .put of an EOS marker; once the queue has filled up, one of these actions blocks forever, since nobody will ever .take from the queue again.

The attached patch does two things:

  • Make sure to put exactly one EOS marker in the stream, by setting the agent's state to nil if and only if a .offer of EOS has been accepted.

  • Replace all occurrences of .put with .offer (and appropriate checking of the return value). This is necessary because if .put ever blocks, it's possible for the system to remain in that state forever (say, because the client stops consuming the queue), thus leading to the same leaked-thread scenario.

The diff is a little bit inconvenient to read because of indentation changes; https://gist.github.com/b7ecd4395a0d3d473de6 is an ignore-whitespace view of the patch for convenience.

Environment

None

Attachments

1

Activity

Show:

Alan Malloy August 20, 2012 at 2:11 AM

Marking as resolved since the patch has been applied in master.

Alan Malloy July 26, 2012 at 10:48 AM

No, these are separate issues. is just a special case of the general problem that seque is not usable from within an agent action. I have an implementation of seque using futures instead of agents so that this isn't a problem, but that has other problems of its own, specifically if you don't fully consume the seque you wind up leaking a future object.

Andy Fingerhut July 26, 2012 at 9:50 AM

Sorry for me just triggering on the function "seque" without a deep understanding, but is this by any chance the same issue as described in CLJ-823? If so, since this one has a patch and that one doesn't, might be nice to mark as a duplicate of this one.

Completed

Details

Assignee

Reporter

Approval

Patch

Priority

Fix versions

Created July 26, 2012 at 12:03 AM
Updated March 1, 2013 at 3:49 PM
Resolved March 1, 2013 at 3:49 PM