<< Back to previous view

[ASYNC-214] pending puts are never signalled when channels are closed Created: 12/Feb/18  Updated: 16/Feb/18  Resolved: 16/Feb/18

Status: Closed
Project: core.async
Component/s: None
Affects Version/s: None
Fix Version/s: None

Type: Defect Priority: Major
Reporter: Pierre-Yves Ritschard Assignee: Unassigned
Resolution: Declined Votes: 0
Labels: bug

Reproduced with core.async 0.3.365 and 0.4.474

Attachments: Text File 0001-ASYNC-214-signal-close-for-putters-later-down-the-li.patch    


Putting on a closed channel should ensure that the producer's put callback be notified with false when a channel is closed.
Currently that is the case when the channel is closed prior to a value being produced on the channel.

When using fixed buffers, there is a race condition if the following sequence of operations occur:

  • A buffer gets filled up
  • A subsequent put blocks, waiting for the buffer to recover capacity
  • The channel is closed

In this case, a put! operation will never see its callback being called, nor would a greater-than-bang! or greater-than-bang-bang! operation ever return (can't figure out the correct way to mark those in JIRA's markup).
From the documenation, the correct behavior should be to deliver false to the put callback.

Here is a small reproduction example:

(require '[clojure.core.async :refer [chan put! close!]])

(let [in (chan 1)
      p  (promise)]
   (put! in :first)
   (put! in :second #(deliver p %))
   (close! in)
   (prn (deref p 200 :blocked)))

;; Output without patch: :blocked
;; Output with patch:    false

Unless I am misunderstanding the semantics or intended behavior of put, I have a fix which walks the list of pending puts when a
close operation occurs on a channel and delivers false to the list of putter callbacks, as well as a corresponding test case. You'll find both as attached files.

If this is deemed valid I can see if the CLJS part also suffers from the same behavior (which at a quick glance seems to be the case).

As it stands, CLJS seems unaffected by the issue, as evidenced by the test I attached to this issue.

Comment by Pierre-Yves Ritschard [ 16/Feb/18 9:19 AM ]

So as it turns out, one way to deal with this is to drain the closed channel, like so:

(a/go (while (a/<! in)))

This ensures no pending puts are blocked. Thanks go out to Tim Baldridge and Alex Miller for helping reason around the issue.

[ASYNC-210] Puts committed on take! when buffer is full and buffer contains an expanding transducer Created: 06/Jan/18  Updated: 07/Jan/18

Status: Open
Project: core.async
Component/s: None
Affects Version/s: None
Fix Version/s: None

Type: Defect Priority: Major
Reporter: Brian James Rubinton Assignee: Unassigned
Resolution: Unresolved Votes: 0
Labels: bug

macOS 10.12.6
clojure 1.7.0 with core.async 0.3.466-SNAPSHOT (rev 0498ba69fda5d930e3e7568cc90ca933c19c42ca)


macOS 10.12.6
clojure 1.9.0 with core.async 0.3.465

Attachments: Text File code_and_test.patch    
Patch: Code


This issue came up here: https://groups.google.com/d/topic/clojure/w_p4f3gNo3s/discussion
Another (possibly) related previous discussion: https://groups.google.com/d/topic/clojure-dev/9Ai-ZuCezOY/discussion

The scenario:
1. a channel has a fixed buffer of 1 and an expanding transducer `(mapcat identity)`
2. a single put! is performed with a collection (C1)
3. a blocking take! is initiated in another thread
4. a second put! is performed with another collection

I expected any put!'s to fail until all elements of collection C1 are taken from the channel.

Non-blocking put!'s (via offer!) do fail until then.

Blocking put!'s do block – but only until a single take! is executed on the channel. That's surprising. Where does the put!'s value go? Everything does continue to "work" as expected, but the buffer acts like it has grown to size 2 (2 input collections, 1 partially output via the transducer)

I believe this is resolved by checking whether the buffer is full during a take! after the call to `(impl/remove! buffer)` but before processing any puts. See the patch for exactly what I mean. I'm not sure whether this change breaks anything as I'm not too familiar with this codebase; though tests do pass.

Comment by Brian James Rubinton [ 07/Jan/18 11:17 AM ]

Replaced the patch with one including a test demonstrating the behavior with the change and without.

[ASYNC-205] Pausing in mix is note effective Created: 22/Sep/17  Updated: 22/Sep/17

Status: Open
Project: core.async
Component/s: None
Affects Version/s: None
Fix Version/s: None

Type: Defect Priority: Major
Reporter: Gabor Hermann Assignee: Unassigned
Resolution: Unresolved Votes: 0
Labels: bug

org.clojure/clojure "1.8.0", org.clojure/core.async "0.3.443"


The following happens when using mix:

(def c1 (to-chan (range 100 105)))
(def c2 (to-chan (range 10 15)))
(def mch (chan))
(def m (mix mch))
(admix m c1)
(admix m c2)
(toggle m {c1 {:pause true}})
=> true
(<!! mch)
=> 100
(<!! mch)
=> 101
(<!! mch)
=> 10

[ASYNC-110] 0.1.346, using >! crashes with no method 'setTimeout', when targeting node Created: 23/Dec/14  Updated: 21/Sep/16  Resolved: 12/Feb/15

Status: Closed
Project: core.async
Component/s: None
Affects Version/s: None
Fix Version/s: None

Type: Defect Priority: Major
Reporter: William Assignee: Unassigned
Resolution: Completed Votes: 3
Labels: bug, nodejs

[[org.clojure/clojure "1.6.0"]
[org.clojure/core.async "0.1.346.0-17112a-alpha"]
[org.clojure/clojurescript "0.0-2511"]]

Attachments: Text File async-110-exportSymbol.patch     Text File br.log    


When targeting node, running a simple >! crashes immediately:

(let [a (chan)]
  (go (>! a :hi))
  (go (println (<! a))))

Crashes with:

goog.global.setTimeout(cb, 0);
TypeError: Object #<Object> has no method 'setTimeout'

This works fine with 0.1.338.0-5c5012-alpha.

Comment by Joel Wilsson [ 01/Feb/15 3:34 PM ]

The problem is really in the ClojureScript compiler. It does not export those NodeJS global functions. I have a fix that works for me and seems sound, so I submitted a pull request: https://github.com/clojure/clojurescript/pull/45

Comment by David Nolen [ 01/Feb/15 3:47 PM ]

I'm not convinced yet that there are any issues with the ClojureScript compiler nor that your PR represents the solution we should pursue.

Before we even begin assessing any solution first someone needs to identify precisely why the goog.async.nextTick detection doesn't pick up the Node.js globals. The first thing that Google Closure library does is map the global environment to goog.global. I'm curious as to why this isn't working in this case.

Comment by Joel Wilsson [ 01/Feb/15 3:47 PM ]

Patch from the pull request attached.

Comment by Joel Wilsson [ 03/Feb/15 3:49 PM ]

Yes, the first thing the Google Closure library is set up a reference to the global environment with goog.global = this;

However, this is not the global environment in Node.js. From http://nodejs.org/api/vm.html#vm_sandboxes:

the this expression within the global scope of the context evaluates to the empty object ({}) instead of to your sandbox.

If you start a Node.js REPL and enter console.log(this) you will see that this is in fact the global environment. However, if you put that statement in a file and run it with nodejs <filename>, the output will be "{}".

Some possible solutions:

  1. Use goog.exportSymbol to get these functions into goog.global, as in the patch I attached. A drawback of this approach is that it is using goog to fix goog, so it must be done after goog has been loaded.
  2. Explicitly set properties on this before doing anything else. This is the simplest solution. Preamble:
    this.setTimeout = setTimeout;
    this.clearTimeout = clearTimeout;
    this.setInterval = setInterval;
    this.clearInterval = clearInterval;
  3. Use vm.Script.runInThisContext, as bootstrap_node.js does to load base.js with this referencing the global environment found in the variable global.
    A bit tricky in this case, since the file we're executing needs to read itself, but it can be done with this preamble:
    if (!this.__nodeFilename) {
        var vm = require('vm');
        var fs = require('fs');
        global.__nodeFilename = __filename;
        vm.runInThisContext.call(global, fs.readFileSync(__nodeFilename), __nodeFilename);

    For this to work the hashbang must be removed, since #!/usr/bin/env node is not valid JavaScript. Since node is a symlink to ax25-node on Ubuntu, and the Node.js binary is called "nodejs", this might be a good idea either way.
    While this solution is the closest to bootstrap_node.js, it is complex and may be a bit too clever.

  4. Convince the Closure library maintainers to check if base.js is being loaded in Node.js and do goog.global = global; if that is the case.

With 3 and 4, goog.global will contain a lot of Node.js stuff. That may or may not be a good thing. Considering the effort in bootstrap node to hide module and exports, I'm guessing it's a bad thing.

From what I can tell, programs using the Closure library, and which have been compiled by the Closure compiler into a single file, have never worked properly if they have been using anything that needs setTimeout and friends.

Comment by David Nolen [ 03/Feb/15 3:53 PM ]

Joel, many thanks for all the extra information. Will sort through this and think about it.

Comment by David Nolen [ 10/Feb/15 12:00 PM ]

Issue is confirmed on this end. Will look into.

Comment by David Nolen [ 10/Feb/15 12:41 PM ]

So I looked into this a bit, this does not actually appear to be a real issue, at least not yet. The Node.js environment must be bootstrapped for ClojureScript to work. However this process could be simpler and we will address the usability bits in ClojureScript. When it's resolved there and we can confirm via automated testing in core.async we will close this ticket.

Comment by Stuart Mitchell [ 11/Feb/15 8:16 PM ]

Not quite sure what you mean by 'not a real problem' as I have a clojurescript project that targets atom-shell that worked with 0.1.338.0-5c5012-alpha but now doesn't.

Do you mean that its actually a clojurescript issue, if so is there a bug report/ work around?

Comment by Joel Wilsson [ 12/Feb/15 7:58 AM ]

Stuart, could you please try the four lines from option 2 in my previous comment as a preamble and let me know if that solves the problem for you? Ie. put them in a file preamble.js on your project's classpath (I use resources/) and add :preamble "preamble.js" to your :compiler settings. Assuming that you are using lein-cljsbuild, that is.

This is a real issue for me as well, because it stops me from using core.async on AWS Lambda.

Comment by David Nolen [ 12/Feb/15 8:10 AM ]

Sorry Stuart I'm not saying it's not real a issue but rather that I'm trying to determine the scope and precise nature of the issue. I noticed that the problem does not manifest when I use :optimizations :none. Are you and Joel using :simple, :advanced? Thanks.

Comment by Joel Wilsson [ 12/Feb/15 8:23 AM ]

I use :simple to get a single JavaScript file, which I can then use with AWS Lambda with this in main.js:

var r = require("./clojurescript-output.js");
exports.handler = r.hello_lambda.core.handler;

where clojurescript-output.js is generated from a file like

(ns hello_lambda.core)
(defn ^:export handler [event context]
  (println "Hello AWS Lambda")
  (.done context))

:advanced doesn't work for me: it just creates a JavaScript file with a single (function () {...})(), which makes it useless for creating a module (the symbols are not exported properly), but that's another issue.

Comment by David Nolen [ 12/Feb/15 9:11 AM ]

this was in fact an upstream ClojureScript issue, fixed as of this commit https://github.com/clojure/clojurescript/commit/e8ae2c8a9c3f2429555e136cc17c1669be5e993a

Comment by Dusan Maliarik [ 12/Jul/16 3:41 PM ]

Not entirely fixed. If you use cljs.core.async/pub in the namespace,

eg. http://stackoverflow.com/questions/24698536/how-to-memoize-a-function-that-uses-core-async-and-non-blocking-channel-read/38069830#38069830
(then using it in some namespace)

(ns foo
  (:require ....))

(def foo
    (fn [...] ...)))

then goog.global.setTimeout is called before the cljs.nodejscli is evaled. Therefore causing

    goog.global.setTimeout(a, 0);

TypeError: goog.global.setTimeout is not a function
    at Function.setImmediate_ (/home/skrat/Workspace/xxx/yyy.drie/bin/convert:37684:17)
    at Object.goog.async.nextTick (/home/skrat/Workspace/xxx/yyy.drie/bin/convert:37630:270)
    at Object.cljs.core.async.impl.dispatch.queue_dispatcher (/home/skrat/Workspace/xxx/yyy.drie/bin/convert:37718:21)
    at Object.cljs.core.async.impl.dispatch.run (/home/skrat/Workspace/xxx/yyy.drie/bin/convert:37722:40)
    at Function.cljs.core.async.pub.cljs$core$IFn$_invoke$arity$3 (/home/skrat/Workspace/xxx/yyy.drie/bin/convert:40775:33)
    at Function.cljs.core.async.pub.cljs$core$IFn$_invoke$arity$2 (/home/skrat/Workspace/xxx/yyy.drie/bin/convert:40703:30)
    at Object.yyy.drie.io.async.memoize_async (/home/skrat/Workspace/xxx/yyy.drie/bin/convert:42231:89)
    at Object.<anonymous> (/home/skrat/Workspace/xxx/yyy.drie/bin/convert:59163:41)
    at Module._compile (module.js:541:32)
    at Object.Module._extensions..js (module.js:550:10)

I tried to require cljs.nodejscli in the ns that contains my main fn, but it results in

SEVERE: cljs.nodejscli:1: ERROR - namespace "cljs.nodejscli" cannot be provided twice

Jul 12, 2016 2:34:34 PM com.google.javascript.jscomp.LoggerErrorManager printSummary
WARNING: 1 error(s), 0 warning(s)
ERROR: JSC_DUPLICATE_NAMESPACE_ERROR. namespace "cljs.nodejscli" cannot be provided twice at cljs.nodejscli line 1 : 0
Comment by Dusan Maliarik [ 12/Jul/16 3:44 PM ]

I believe the solution is to include cljs.nodejscli as soon as possible, instead of appending it at the end. This also means it has to be split into two pieces, one dealing with goog.global (included asap), and another one dealing with main-cli-fn (included at the end as is the case now).

Comment by Nick Alexander [ 21/Sep/16 5:34 PM ]

I am seeing this, with `optimizations :advanced` and `:simple`, with the following project https://github.com/ncalexan/datomish-user-agent-service/commit/af99ad3a16eece881a48fedd0b0f62517bb5cb25. It's not going to be 100% easy to duplicate, since an underlying library isn't published on Maven Central yet, but I can reproduce this consistently.

Any help?

Comment by Nick Alexander [ 21/Sep/16 6:10 PM ]

For the next poor sod who runs into this, it is triggered by having a `go` at top-level.

Hat tip to user "Roc King" in the post http://clojure.2344290.n4.nabble.com/Will-advanced-optimizations-be-officially-supported-on-nodejs-tp7762p7855.html, who writes [sic]:

After got more familiar with clojurescript, I realized that ASYNC-110 fully described(and gave several working solutions!)[3] this issue.
But if "goog.global" was used before "goog.global = global"(the solution used in clojurescript[4]), the same problem will occur.
This is the exactly situation I have encountered several days ago: I wrote '(go (<! (timeout 1212)) ...)' in top level.

This was not clear to me until I read Roc's note. After I remove my top-level `go`, everything is happy.

[ASYNC-59] Channel returned by cljs.core.async/map> is missing protocol method Channel.closed? Created: 08/Mar/14  Updated: 10/Sep/14  Resolved: 02/Sep/14

Status: Closed
Project: core.async
Component/s: None
Affects Version/s: None
Fix Version/s: None

Type: Defect Priority: Minor
Reporter: Kevin Neaton Assignee: Unassigned
Resolution: Declined Votes: 0
Labels: bug, patch

[org.clojure/clojure "1.5.1"]
[org.clojure/clojurescript "0.0-2173"]
[org.clojure/core.async ""]

Attachments: Text File 0001-Fixed-map-by-including-impl.-for-closed.patch    
Patch: Code



(let [ch (->> (chan) (map> inc) (filter> even?))]
  (doseq [i (range 10)] (put! ch i)))

When filter> checks to see if the channel returned by map> is closed?, this code fails because the channel returned by map> does not implement the Channel.closed? protocol method.

Comment by Alex Miller [ 02/Sep/14 9:54 AM ]

map> and filter> have been deprecated and will be removed in a future release. They have been replaced with applying transducers to a channel which is now available.

Comment by Kevin Neaton [ 10/Sep/14 9:08 AM ]

Great, thanks for the update.

[ASYNC-48] Recur fails within catch Created: 21/Dec/13  Updated: 09/May/14  Resolved: 09/May/14

Status: Closed
Project: core.async
Component/s: None
Affects Version/s: None
Fix Version/s: None

Type: Defect Priority: Major
Reporter: Gerrit Jansen van Vuuren Assignee: Timothy Baldridge
Resolution: Declined Votes: 0
Labels: bug


This is related to

Having a loop in a go block and the recur inside a catch statement causes an IllegalArgumentException.

The code is:

(loop [i 0]
(+ 1 1))
(catch Exception e (do
(prn e)
(recur (inc i)))))))

;; IllegalArgumentException No implementation of method: :emit-instruction of protocol: #'clojure.core.async.impl.ioc-
;; macros/IEmittableInstruction found for class: clojure.core.async.impl.ioc_macros.Jmp clojure.core/-cache-protocol-fn

Comment by Leon Grapenthin [ 25/Dec/13 9:14 AM ]

This doesn't work outside of a go block either.

Comment by Timothy Baldridge [ 09/May/14 11:30 AM ]

Closing due to example not being valid code.

[ASYNC-29] 50 parallel blocking takes in go-blocks as loop-binding or recur argument break go Created: 28/Oct/13  Updated: 29/Oct/13  Resolved: 29/Oct/13

Status: Resolved
Project: core.async
Component/s: None
Affects Version/s: None
Fix Version/s: None

Type: Defect Priority: Major
Reporter: Leon Grapenthin Assignee: Rich Hickey
Resolution: Completed Votes: 0
Labels: bug, go, go-macro



This problem raises the question whether what the implications are of using <!! and >!! inside of go and whether it is legal.

What if I provide API functions that are using <!! >!! and may be invoked from within go?
Should I then also provide !! and ! versions?

Please evalute this code at the REPL to reproduce the problem: https://www.refheap.com/20225

Comment by Leon Grapenthin [ 29/Oct/13 9:46 AM ]

I'd like to rephrase my question: Do I have to provide {!} and {!!} macros that do the same thing and use {<!} {>!} and {<!!} {>!!} respectively? Would it make sense for core.async to provide a defasync macro that creates those two versions from the same body where you could for example use {<!!!} and {>!!!}, {alts!!!} and so on so that they would be replaced by a postwalk before the macro is defined defined twice with {!} and {!!} appended to the name? Or are there other plans of abstraction? [Quoting symbols because of jira markup]

Comment by Rich Hickey [ 29/Oct/13 9:55 AM ]

You shouldn't be using <!! and >!! in library code. We may at some point be able to detect at runtime their use in go blocks and throw errors, but currently do not.

[ASYNC-20] Case macro in go block ignores default expression Created: 20/Aug/13  Updated: 27/Sep/13  Resolved: 27/Sep/13

Status: Closed
Project: core.async
Component/s: None
Affects Version/s: None
Fix Version/s: None

Type: Defect Priority: Major
Reporter: Leon Grapenthin Assignee: Timothy Baldridge
Resolution: Completed Votes: 0
Labels: bug, go

Clojure 1.5.1 in the JVM, core.async-0.1.0-20130819.220150-69


To reproduce the error use this code:
(go (case true
false false

An IllegalArgumentException will be thrown (as if there was no default expression).
Exception in thread "async-dispatch-46" java.lang.IllegalArgumentException: No matching clause: true

Comment by Timothy Baldridge [ 27/Sep/13 8:34 AM ]

fixed in: https://github.com/clojure/core.async/commit/f9b24552cf4f6b8b86f3377c7b7a42a618a9fd76

[ASYNC-18] CompilerException attempting to perform a transaction inside go block Created: 15/Aug/13  Updated: 16/Aug/13  Resolved: 16/Aug/13

Status: Closed
Project: core.async
Component/s: None
Affects Version/s: None
Fix Version/s: None

Type: Defect Priority: Major
Reporter: Adam Wittenberg Assignee: Timothy Baldridge
Resolution: Declined Votes: 0
Labels: bug


When setting up a transaction inside of a go block, receiving the following error:

CompilerException java.lang.RuntimeException: No such var: clojure.core/runInTransaction, compiling:(NO_SOURCE_PATH:2:8)

(require '[clojure.core.async :as async :refer :all])

(def bar (ref []))
(def c (chan))

(defn foo []
(<!! (go (dosync
(doseq [i (range 10)]
(alter bar conj (<! c)))))))

Comment by Timothy Baldridge [ 16/Aug/13 9:08 AM ]

There are two problems with the example given.

a) dosync wraps the body of the body of the transaction in a fn. Go block translation stops at fn boundaries. The same applies to doseq.

b) The bigger problem is that this example code is incorrect. A transaction body may executed multiple times in that case some messages received from the channel may be dropped.

Closing this issue as the examples are flawed. However I will recommend that you look into doing the takes outside of the transaction, and then using a function to execute the transaction outside the body of a go.

Generated at Tue Sep 25 15:51:35 CDT 2018 using JIRA 4.4#649-r158309.