Clojure

pmap level of parallelism inconsistent for chunked vs non-chunked input

Details

  • Type: Defect Defect
  • Status: Open Open
  • Priority: Minor Minor
  • Resolution: Unresolved
  • Affects Version/s: Release 1.1, Release 1.2, Release 1.3, Release 1.4
  • Fix Version/s: None
  • Component/s: None
  • Labels:
    None
  • Patch:
    Code and Test

Description

The code of pmap creates futures for the set of map operations it needs to perform with (map #(future %) coll), then acts on that sequence in a fashion obviously intended to keep only #CPUs+2 unfinished futures realized at any given time. It works exactly this way for non-chunked input seqs, but when pmap is passed a chunked seq, the seq of futures also becomes chunked. This causes an arbitrary number of futures to be realized as the #CPUs+2 window and chunk-size window interact.

The doc-string for pmap doesn't promise any particular level of parallelism, but I think the inconsistent parallelism for chunked vs non-chunked input comprises a bug.

Activity

Stuart Halloway made changes -
Field Original Value New Value
Fix Version/s Approved Backlog [ 10034 ]
Stuart Halloway made changes -
Fix Version/s Approved Backlog [ 10034 ]
Fix Version/s Reviewed Backlog [ 10053 ]
Jim Blomo made changes -
Attachment pmap-chunking-862.diff [ 11265 ]
Jim Blomo made changes -
Patch Code and Test [ 10002 ]
Jim Blomo made changes -
Assignee Jim Blomo [ jim.blomo ]
Alex Miller made changes -
Fix Version/s Reviewed Backlog [ 10053 ]
Fix Version/s Backlog [ 10035 ]
Alex Miller made changes -
Fix Version/s Backlog [ 10035 ]

People

Vote (0)
Watch (2)

Dates

  • Created:
    Updated: