Clojure

Piping seque into seque can deadlock

Details

  • Type: Defect Defect
  • Status: Open Open
  • Priority: Minor Minor
  • Resolution: Unresolved
  • Affects Version/s: Release 1.3
  • Fix Version/s: Release 1.7
  • Component/s: None
  • Labels:
    None
  • Environment:
    Windows 7; JVM 1.6; Clojure 1.3 beta 1
  • Patch:
    Code and Test
  • Approval:
    Screened

Description

I'm not sure if this is a supported scenario, but the following deadlocks in Clojure 1.3:

(let [xs (seque (range 150000))
      ys (seque (filter odd? xs))]
  (apply + ys))

Cause: As I understand it, the problem is that ys' fill takes place on an agent thread, so when it calls xs' drain, the (send-off agt fill) does not immediately trigger xs' fill, but is instead put on the nested list to be performed when ys' agent returns. Unfortunately, ys' fill will eventually block trying to take from xs, and so it never returns and the pending send-offs are never sent.

Approach: Use (release-pending-sends) in seque's drain function to avoid the deadlock when a seque is fed into another seque.

Patch: clj-823-v1.patch

Screened by: Alex Miller

Activity

Hide
Peter Monks added a comment -

Reproduced on 1.4.0 and 1.5.0-RC1 as well, albeit with this example:

(seque 3 (seque 3 (range 10)))

Show
Peter Monks added a comment - Reproduced on 1.4.0 and 1.5.0-RC1 as well, albeit with this example: (seque 3 (seque 3 (range 10)))
Hide
Stuart Halloway added a comment -

release-pending-sends?

Show
Stuart Halloway added a comment - release-pending-sends?
Stuart Halloway made changes -
Field Original Value New Value
Approval Triaged [ 10120 ]
Rich Hickey made changes -
Approval Triaged [ 10120 ] Vetted [ 10003 ]
Rich Hickey made changes -
Fix Version/s Release 1.7 [ 10250 ]
Hide
Andy Fingerhut added a comment -

clj-823-v1.patch uses (release-pending-sends) in seque's drain function in an attempt to avoid the deadlock when a seque is fed into another seque, as suggested by Stuart Halloway. It adds Peter Monks's small quick test case demonstrating the deadlock, which fails (i.e. hangs until killed) without the change and passes with it.

Show
Andy Fingerhut added a comment - clj-823-v1.patch uses (release-pending-sends) in seque's drain function in an attempt to avoid the deadlock when a seque is fed into another seque, as suggested by Stuart Halloway. It adds Peter Monks's small quick test case demonstrating the deadlock, which fails (i.e. hangs until killed) without the change and passes with it.
Andy Fingerhut made changes -
Attachment clj-823-v1.patch [ 13245 ]
Andy Fingerhut made changes -
Patch Code and Test [ 10002 ]
Alex Miller made changes -
Description I'm not sure if this is a supported scenario, but the following deadlocks in Clojure 1.3:

{{(let [xs (seque (range 150000))}}
{{ys (seque (filter odd? xs))]}}
{{(apply + ys))}}

As I understand it, the problem is that ys' fill takes place on an agent thread, so when it calls xs' drain, the {{(send-off agt fill)}} does not immediately trigger xs' fill, but is instead put on the nested list to be performed when ys' agent returns. Unfortunately, ys' fill will eventually block trying to take from xs, and so it never returns and the pending send-offs are never sent. Wrapping the send-off in drain to:

{{(future (send-off agt fill))}}

is a simple (probably not optimal) way to fix the deadlock.
I'm not sure if this is a supported scenario, but the following deadlocks in Clojure 1.3:

{code}
(let [xs (seque (range 150000))
      ys (seque (filter odd? xs))]
  (apply + ys))
{code}

As I understand it, the problem is that ys' fill takes place on an agent thread, so when it calls xs' drain, the {{(send-off agt fill)}} does not immediately trigger xs' fill, but is instead put on the nested list to be performed when ys' agent returns. Unfortunately, ys' fill will eventually block trying to take from xs, and so it never returns and the pending send-offs are never sent. Wrapping the send-off in drain to:

{code}
(future (send-off agt fill))
{code}

is a simple (probably not optimal) way to fix the deadlock.
Alex Miller made changes -
Approval Vetted [ 10003 ] Screened [ 10004 ]
Description I'm not sure if this is a supported scenario, but the following deadlocks in Clojure 1.3:

{code}
(let [xs (seque (range 150000))
      ys (seque (filter odd? xs))]
  (apply + ys))
{code}

As I understand it, the problem is that ys' fill takes place on an agent thread, so when it calls xs' drain, the {{(send-off agt fill)}} does not immediately trigger xs' fill, but is instead put on the nested list to be performed when ys' agent returns. Unfortunately, ys' fill will eventually block trying to take from xs, and so it never returns and the pending send-offs are never sent. Wrapping the send-off in drain to:

{code}
(future (send-off agt fill))
{code}

is a simple (probably not optimal) way to fix the deadlock.
I'm not sure if this is a supported scenario, but the following deadlocks in Clojure 1.3:

{code}
(let [xs (seque (range 150000))
      ys (seque (filter odd? xs))]
  (apply + ys))
{code}

*Cause:* As I understand it, the problem is that ys' fill takes place on an agent thread, so when it calls xs' drain, the {{(send-off agt fill)}} does not immediately trigger xs' fill, but is instead put on the nested list to be performed when ys' agent returns. Unfortunately, ys' fill will eventually block trying to take from xs, and so it never returns and the pending send-offs are never sent.

*Approach:* Use (release-pending-sends) in seque's drain function to avoid the deadlock when a seque is fed into another seque.

*Patch:* clj-823-v1.patch

*Screened by:* Alex Miller

People

Vote (1)
Watch (5)

Dates

  • Created:
    Updated: