pipeline does not stop consuming `from` when `to` is closed
Description
The docs for pipeline say "Will stop consuming the from channel if the to channel closes." however the following code will print 0..49 (and would continue consuming from forever).
Cause: Currently, when the output go loop puts to the to channel and gets a closed result indicator (logical false), it only stops the output go block, but lets the ingest and compute go loops continue, so the pipeline keeps consuming the from channel and computing values it can't use in the output. In other words, it does not do what it says it will do.
Alternatives:
Change the docstring to not promise to stop consuming from(this seems wrong)
When to is closed, signal the ingest and compute go blocks to stop, either via state (atom), or a control channel
Push the signal back through the compute and ingest blocks somehow, maybe by closing the jobs and results channels and watching for that?
Proposed:
Create an atom indicating ingest processing should stop.
In the output loop, if the to channel is found to be closed, reset the stop atom to true.
In the ingest loop, check the stop atom - if set, stop ingesting and close the jobs chan (as we already do when the from channel is closed)
Note that some work may be in flight (1 may be in-process in the output go block, N may be in the results channel, and N may be parked in flight in process go blocks)
Patch: async-217-2.patch (same change in clj and cljs)
The example above usually prints 0…2 with the patch.
Screened by: Ghadi
Environment
None
Attachments
3
Activity
Alex Miller
November 22, 2022 at 4:22 AM
-3 patch is alternate approach to use a control channel to shutdown the ingest go loop and job threads. have not yet replicated to cljs
Alex Miller
February 3, 2022 at 2:58 PM
-2 patch is same, just merges context in the patch to make it easier to glance at
Alex Miller
August 16, 2018 at 8:48 PM
Tianxiang: I'm inclined to agree with you, just being thorough in listing options.
import
May 17, 2018 at 6:22 PM
Comment made by: xiongtx
One option would be to change the doc string and not promise this behavior.
The expectation should be that we don't endlessly process inputs when we're not consuming outputs, which would certainly be the case when the to channel is closed.
Critical
The docs for
pipelinesay "Will stop consuming thefromchannel if thetochannel closes." however the following code will print 0..49 (and would continue consumingfromforever).Cause: Currently, when the output go loop puts to the
tochannel and gets a closed result indicator (logical false), it only stops the output go block, but lets the ingest and compute go loops continue, so the pipeline keeps consuming thefromchannel and computing values it can't use in the output. In other words, it does not do what it says it will do.Alternatives:
Change the docstring to not promise to stop consuming
from(this seems wrong)When
tois closed, signal the ingest and compute go blocks to stop, either via state (atom), or a control channelPush the signal back through the compute and ingest blocks somehow, maybe by closing the jobs and results channels and watching for that?
Proposed:
Create an atom indicating ingest processing should stop.
In the output loop, if the
tochannel is found to be closed, reset the stop atom to true.In the ingest loop, check the stop atom - if set, stop ingesting and close the jobs chan (as we already do when the
fromchannel is closed)Note that some work may be in flight (1 may be in-process in the output go block, N may be in the results channel, and N may be parked in flight in process go blocks)
Patch: async-217-2.patch (same change in clj and cljs)
The example above usually prints 0…2 with the patch.
Screened by: Ghadi