<< Back to previous view

[ASYNC-124] put! into channel that has mapcat transducer fails to dispatch more than one waiting takes Created: 30/May/15  Updated: 28/Aug/15

Status: Open
Project: core.async
Component/s: None
Affects Version/s: None
Fix Version/s: None

Type: Defect Priority: Critical
Reporter: Jozef Wagner Assignee: Alex Miller
Resolution: Unresolved Votes: 1
Labels: None
Environment:

Clojure 1.7.0-RC1, latest core.async head (May 2015)


Attachments: Text File async-124-2.patch     Text File async-124-3.patch     Text File async-124-4.patch     Text File async-124-5.patch     Text File async-124-6.patch     Text File async-124-7.patch     Text File async-124-8.patch     Text File async-124.patch    
Patch: Code and Test
Approval: Screened

 Description   

This issue is similar to one that has caused the revert of ASYNC-103. When channel has multiple pending takes, one put will dispatch only one take, even if transducer attached to the channel produces multiple values.

(require '[clojure.core.async :refer (chan put! take!)])
(let [xf (mapcat #(range % (+ % 5)))
        c (chan 10 xf)]
    (dotimes [x 5]
      (take! c #(println (str x %))))
    (put! c 10)
    (put! c 15))

The output of the above code is

0 10
1 11

but it should be

0 10
1 11
2 12
3 13
4 14

Note that if we change the order and execute puts before takes, the result will be as expected.

Cause: When executing a write on a channel, no more than one pending take will be satisfied. Other pending takes will wait until the next put. When a put could release at most one take, this was sufficient but with expanding transducers, a put can potentially satisfy many pending takers.

Approach: Once a put succeeds into the buffer (through the transducer), instead of finding a single taker, do the prior work in a loop, pairing up callbacks with values removed from the buffer. Once the loop is complete, release the mutex and dispatch all paired takes.

Patch: async-124-8.patch

Screened by:



 Comments   
Comment by Jozef Wagner [ 30/May/15 10:30 AM ]

patch with test

Comment by Jozef Wagner [ 30/May/15 10:32 AM ]

Note that with attached patch the actual dispatch/run on takes happens before (.unlock mutex) and before (when done? (abort this)) are called.

Also note that added test only works with 1.7.0 version of clojure

Comment by Jozef Wagner [ 30/May/15 12:46 PM ]

Added new patch async-124-2.patch that dispatches after unlocking and aborting. Test was also changed as it had race condition.

Comment by Alex Miller [ 06/Jul/15 9:23 AM ]

We can't add a test that is 1.7-only. See https://github.com/clojure/core.async/blob/master/src/test/clojure/clojure/core/pipeline_test.clj#L7 for an example of how to patch in a transducer impl that will work in pre-1.7.

Comment by Jozef Wagner [ 06/Jul/15 10:41 AM ]

Added async-124-3.patch that works with Clojure <1.7

Comment by Alex Miller [ 10/Aug/15 3:53 PM ]

Jozef, I don't see any reason why the independent callbacks in your test will necessarily conj in order as they contend for the atom. When I run the test I usually get a failure. If you agree, could wrap expected and actual in a set. Or if that seems wrong, then let's decide why.

Comment by Jozef Wagner [ 11/Aug/15 10:49 AM ]

Attached async-124-4.patch that uses unordered comparison in test

Comment by Alex Miller [ 14/Aug/15 1:01 PM ]

Added async-124-5.patch which should be semantically identical as -4 but retains the structure of a single dispatch point.

Comment by Alex Miller [ 14/Aug/15 3:03 PM ]

New -6 patch tries to retain more of the original algorithm and leverages the original loop instead of adding a new one.

Comment by Alex Miller [ 18/Aug/15 8:25 AM ]

New -7 patch addresses some bugs in the prior version and cleans up the loop code, also adds better tests.

Comment by Alex Miller [ 28/Aug/15 11:53 AM ]

test tweaks added in -8, per Stuart Sierra

Comment by Stuart Sierra [ 28/Aug/15 12:01 PM ]

Screened. Clojure version good. Still needs corresponding fix to ClojureScript version.





[ASYNC-4] Record literals become ordinary maps Created: 15/Jul/13  Updated: 23/Jun/14

Status: Open
Project: core.async
Component/s: None
Affects Version/s: None
Fix Version/s: None

Type: Defect Priority: Major
Reporter: Alex Miller Assignee: Unassigned
Resolution: Unresolved Votes: 0
Labels: None
Environment:

CLJS


Approval: Triaged

 Description   
(require '[clojure.core.async :refer (<!! go)])
(defrecord Foo [x])
(def f (Foo. 4))
(<!! (go f))
;; => #user.Foo{:x 4}
;; OK

(<!! (go #clojure.core.async.Foo{:x 4}))
;; CLJ: => #user.Foo{:x 4}   ;; expected
;; CLJS => {:x 4}            ;; wrong

Approach: Query the analyzer to know if we have a record or not.

(Copied from GitHub issue #13 - https://github.com/clojure/core.async/issues/13)



 Comments   
Comment by Ghadi Shayban [ 03/Aug/13 2:08 PM ]

0c6e663493 contains a fix on the Clojure side, would appreciate help porting to cljs.

Comment by David Nolen [ 08/Aug/13 8:35 AM ]

I think on the ClojureScript we'll have to query the analyzer to know if we have a record or not.





[ASYNC-27] Compilation errors inside go block always reported as first line of block Created: 08/Oct/13  Updated: 09/May/14

Status: Open
Project: core.async
Component/s: None
Affects Version/s: None
Fix Version/s: None

Type: Defect Priority: Major
Reporter: Howard Lewis Ship Assignee: Unassigned
Resolution: Unresolved Votes: 3
Labels: errormsgs


 Description   

I've noticed that when there are any errors inside a go block, the line number of the error is always the line containing the go symbol.

I suspect that some meta data on the forms that are converted into a state machine is being lost in the process.

This is quite annoying and quite leaky (in the abstraction sense). It makes it that much harder to track down the source of errors.



 Comments   
Comment by Timothy Baldridge [ 22/Nov/13 2:49 PM ]

Working on this, may be a few weeks out yet.

Comment by Hugo Duncan [ 11/Feb/14 5:18 PM ]

This effects both compilation errors and line numbers in stack trace frames.

Comment by Timothy Baldridge [ 09/May/14 11:26 AM ]

Completed in CLJ, once we get a tools.analyzer.cljs I'll add this to CLJS as well.





[ASYNC-32] onto-chan retains head of input sequence causing OutOfMemoryError Created: 06/Nov/13  Updated: 11/Aug/15

Status: Open
Project: core.async
Component/s: None
Affects Version/s: None
Fix Version/s: None

Type: Defect Priority: Major
Reporter: Brian Lubeski Assignee: Unassigned
Resolution: Unresolved Votes: 1
Labels: memory
Environment:

org.clojure/core.async 0.1.242.0-44b1e3-alpha


Attachments: File patch.clj    
Patch: Code
Approval: Triaged

 Description   
(require '[clojure.core.async :as a])
(let [c (a/chan)]
  (a/onto-chan c (iterate inc 0))
  (while true
    (a/<!! c)))

onto-chan is holding on to the head of the input sequence as it is unfolded, resulting in an (eventual) OutOfMemoryError.

I've attached a diff showing changes I made to onto-chan that fixed the problem for me.



 Comments   
Comment by Colin Taylor [ 07/Oct/14 7:13 PM ]

Just to note, to-chan uses onto-chan so is similarly affected.
We ran into this, Brian's patch worked fine.

Comment by Alex Miller [ 10/Aug/15 3:22 PM ]

This seems like a viable change, but the patch needs a better test (even if not in the patch) and to be properly formatted for git apply (per http://dev.clojure.org/display/community/Developing+Patches).

Comment by Brian Lubeski [ 11/Aug/15 12:00 AM ]

This issue might be caused by ASYNC-138.





[ASYNC-46] Add optional default channel to 'pub' Created: 09/Dec/13  Updated: 05/May/14

Status: Open
Project: core.async
Component/s: None
Affects Version/s: None
Fix Version/s: None

Type: Enhancement Priority: Major
Reporter: Steffen Dienst Assignee: Unassigned
Resolution: Unresolved Votes: 0
Labels: None

Attachments: Text File 0001-Add-default-channel-for-pub.patch    
Patch: Code and Test

 Description   

If there is no subscriber for a topic the message gets silently dropped. I suggest to add an optional default channel that receives all those messages. The default channel can then be used in scenarios, where the complete set of topics might be unknown beforehand.
This functionality resembles the 'dead letter' pattern in messaging systems.






[ASYNC-49] cljs IOC macro issue with double-dot form Created: 12/Jan/14  Updated: 05/Jun/15

Status: Open
Project: core.async
Component/s: None
Affects Version/s: None
Fix Version/s: None

Type: Defect Priority: Major
Reporter: Paul Butcher Assignee: Unassigned
Resolution: Unresolved Votes: 1
Labels: None
Environment:

ClojureScript 0.0-2138
core.async 0.1.267.0-0d7780-alpha



 Description   

This ClojureScript compiles and runs as expected:

(let [circle (dom/getElement "circle")]
  (go-loop [x 20]
    (<! (timeout 10))
    (set! (.-value (.-baseVal (.-cx circle))) x)
    (recur (inc x))))

But if I change the set! line to:

(set! (.. circle -cx -baseVal -value) x)

I get the following error:

Wrong number of args (3) passed to: core$-DOT

For further discussion see https://groups.google.com/d/topic/clojurescript/ONMaEho4K0c/discussion



 Comments   
Comment by Andrew S [ 05/Jun/15 8:49 AM ]

I too noticed this, but only when using the .. operator in a go loop. The linked thread further suggests this is a core.async problem with this operator.





[ASYNC-51] Core.async memory leak in Safari and Mobile Safari Created: 21/Jan/14  Updated: 15/Oct/14

Status: Open
Project: core.async
Component/s: None
Affects Version/s: None
Fix Version/s: None

Type: Defect Priority: Major
Reporter: Bruce Hauman Assignee: Unassigned
Resolution: Unresolved Votes: 0
Labels: None
Environment:

Mobile Safari on iOS 7.0; Safari 6.1 on OSX Lion



 Description   

Chaining together channels apparently leaks memory in Safari.

A statement such as

(def test [input-chan]
  (map< identity (map< identity (map< identity input-chan))))

will leak memory when the channels are used. The longer the chain the more memory.

I have created an example repository and an example page.

The repo and the example code is here:
https://github.com/bhauman/checkmemleak/blob/master/src/checking_safari_leak/core.cljs

The demonstration page is here:
This link will leak
http://rigsomelight.com/checkmemleak/?leak=true
This link will not
http://rigsomelight.com/checkmemleak

The leak is pretty darn severe.



 Comments   
Comment by Bruce Hauman [ 21/Jan/14 4:55 PM ]

I have just confirmed this is also a problem in Safari Version 7.0.1 (9537.73.11) on Mavericks.

Comment by Bruce Hauman [ 23/Jan/14 7:13 PM ]

I updated the example page.

http://rigsomelight.com/checkmemleak/index.html

This link wont leak:
http://rigsomelight.com/checkmemleak/index.html#comp-partial

This link will leak:
http://rigsomelight.com/checkmemleak/index.html#map<

There is navigation to try the different implementations and optimization modes.

To be clear map<-chain, custom-map, map<, and map> all exhibit the leak in :none and simple optimization modes.

Comment by David Nolen [ 14/Oct/14 6:00 AM ]

To make sure this ticket has all the necessary information - if I remember correctly even advanced optimization is affected? Am I correct Bruce?

Comment by Bruce Hauman [ 15/Oct/14 12:53 PM ]

Yes, it affected advanced optimization. I haven't checked if this is still an issue.





[ASYNC-58] mult channel deadlocks when untapping a consuming channel whilst messages are being queued/blocked Created: 20/Feb/14  Updated: 23/Jun/14

Status: Open
Project: core.async
Component/s: None
Affects Version/s: None
Fix Version/s: None

Type: Defect Priority: Major
Reporter: Mathieu Gauthron Assignee: Unassigned
Resolution: Unresolved Votes: 1
Labels: deadlock, mult, untap
Environment:

Mac 10.7.5; java version "1.7.0_40"; [org.clojure/clojure "1.5.1"]; [org.clojure/core.async "0.1.267.0-0d7780-alpha"]; Tested with cider and emacs 24.3


Approval: Triaged

 Description   

I have two (or more) listeners tapped onto a mult channel. I want to use them all then have one (or more) of them to leave at will without blocking the other consumer(s) or the publisher. Initially they work fine until one of them wants to stop listening. I thought the listener which drops out needs to (be a good citizen and) untap its channel from mult (otherwise a deadlock is systematic). However if messages are put into the mult before the leaving listener has had a chance to untap its channel, it creates a deadlock on the main thread (which is putting more messages simultaneously). I do not find a way to guarantee that I can untap the channel in time to avoid this race condition.

Once I have reproduced the deadlock, the repl is frozen until I interrupt with ctrl-c.
I have also tried to close the tapped channel before untapping it but the result was the same.

In the following snippet, the last (println "I'm done. You will never see this") is never reached. The publisher and the remaining consumer (consumer 1) are deadlocked even though consumer 2 was trying to leave in good terms.

(require '[clojure.core.async :refer (chan go <! >!! mult tap untap)])
(let [to-mult (chan 1)
      m (mult to-mult)]

  ;;consumer 1
  (let [c (chan 1)]
    (tap m c)
    (go (loop []
          (when-let [v (<! c)]
            (println "1 Got! " v)
            (recur))
          (println "1 Exiting!"))))

  ;;consumer 2
  (let [c (chan 1)]
    (tap m c)
    (go (loop []
          (when-let [v (<! c)]
            (when (= v 42)  ;; exit when value is not 42
              (println "2 Got! " v)
              (recur)))
          (println "2 about to leave!")
          (Thread/sleep 5000) ;; wait a bit to exacerbate the race condition
          (untap m c) ;; before unsubscribing this reader
          (println "2 Exiting."))))

   (println "about to put a few messages that work")
   (doseq [a (range 10)]
     (>!! to-mult 42))
   (println "about to put a message that will force the exit of 2")
   (>!! to-mult 43)
   (println "about to put a few more messages before reader 2 is unsubscribed to show the deadlock")
   (doseq [a (range 10)]
     (println "putting msg" a)
     (>!! to-mult 42))
   (println "I'm done. You will never see this"))
about to put a few messages that work
2 Got!  42
1 Got!  42
2 Got!  42
1 Got!  42
1 Got!  42
2 Got!  42
1 Got!  42
1 Got!  42
2 Got!  42
2 Got!  42
2 Got!  42
2 Got!  1 Got!  42
422 Got!  42

1 Got!  42
1 Got!  42
2 Got!  42
1 Got!  42
about to put a message that will force the exit of 2
1 Got!  42
2 Got!  about to put a few more messages before reader 2 is unsubscribed to show the deadlock
42
putting msg 1 Got!  0
2 about to leave!
43
1 Got!  42
putting msg 1
putting msg 2
putting msg 3
1 Got!  42
2 Exiting.


 Comments   
Comment by Ghadi Shayban [ 22/Apr/14 10:18 AM ]

Mathieu, this is probably expected. It's important to note that to guarantee correct ordering/flow when using a mult, you should enforce it on the source/producer side of the mult, and not asynchronously on the tap side.

Mult will deref a stable set taps just before distributing a value to them, and does not adjust dynamically during value distribution except when a tap has been closed [1]. If you would like to stably untap without closing the tap you can/should let the 'producer' do it in an ordered fashion in between values on the input channel.

Knowing that a put occurred to a closed channel is new on release 0.1.278.

In general, walking away on the consuming side of a channel is tricky. Depending on the semantics of your processes, if the producer side of a channel isn't aware that a close! can happen from the consumer side, you might have to launch a draining operation.

(defn drain [c] (go (when (some? (<! c)) (recur))))

Golang disallows closing a read-only channel FWIW [2]

Better documentation is probably warranted.

[1] https://github.com/clojure/core.async/blob/master/src/main/clojure/clojure/core/async.clj#L680-L682
[2] http://golang.org/ref/spec#Close





[ASYNC-64] Race condition when closing mults Created: 29/Apr/14  Updated: 16/Oct/14

Status: Open
Project: core.async
Component/s: None
Affects Version/s: None
Fix Version/s: None

Type: Defect Priority: Major
Reporter: James Reeves Assignee: Unassigned
Resolution: Unresolved Votes: 0
Labels: mult

Approval: Triaged

 Description   

When a mult is tapped at around the same time as the source channel is closed, the tapped channel may not be closed.

(require '[clojure.core.async :refer (chan mult tap close!)])
(let [s (chan)
      m (mult s)
      c (chan)]
  (tap m c)
  (close! s)
  (impl/closed? c))

The above code will sometimes return true, and sometimes return false.

Cause: This is caused by the following code in the mult function:

(if (nil? val)
  (doseq [[c close?] @cs]
    (when close? (close! c)))

Any channels tapped after cs is dereferenced will not be closed.

Approach: A possible solution to this could be to always close channels tapped to a closed source. i.e.

(let [s (chan)
      m (mult s)
      c (chan)]
  (close! s)
  (tap m c))  ;; will always close c

This could be achieved by adding a flag to the cs atom to denote whether the mult is open or closed. If it's closed, any tapped channel is closed automatically.



 Comments   
Comment by James Reeves [ 30/Apr/14 6:05 AM ]

For reference, below is the custom fix for mult I'm using:

(defn mult [ch]
  (let [state (atom [true {}])
        m (reify
            Mux
            (muxch* [_] ch)
            Mult
            (tap* [_ ch close?]
              (let [add-ch    (fn [[o? cs]] [o? (if o? (assoc cs ch close?) cs)])
                    [open? _] (swap! state add-ch)]
                (when-not open? (close! ch))
                nil))
            (untap* [_ ch]
              (swap! state (fn [[open? cs]] [open? (dissoc cs ch)]))
              nil)
            (untap-all* [_]
              (swap! state (fn [[open? _]] [open? {}]))))
        dchan (chan 1)
        dctr (atom nil)
        done (fn [_] (when (zero? (swap! dctr dec))
                       (put! dchan true)))]
    (go-loop []
      (let [val (<! ch)]
        (if (nil? val)
          (let [[_ cs] (swap! state (fn [[_ cs]] [false cs]))]
            (doseq [[c close?] cs]
              (when close? (close! c))))
          (let [chs (keys (second @state))]
            (reset! dctr (count chs))
            (doseq [c chs]
              (when-not (put! c val done)
                (swap! dctr dec)
                (untap* m c)))
            (when (seq chs)
              (<! dchan))
            (recur)))))
    m))
Comment by David Nolen [ 14/Oct/14 6:10 AM ]

Is this also fixed in master? Thanks.

Comment by Ghadi Shayban [ 15/Oct/14 11:09 PM ]

I understand the scenario, but honestly I'm not sure this is a bug in mult or the usage. A channel shouldn't be expected to always yield a take. The consumer of the "late tap" can guard against it with alts or some other mechanism, and also you can enforce a no-late-taps through a policy on the "production" side of things.

Rich Hickey can you weigh in?

Comment by James Reeves [ 16/Oct/14 3:51 AM ]

The "tap" function currently has an explicit "close?" flag, and if a tapped channel isn't guaranteed to close when the source channel closes, that argument probably shouldn't exist. Also, if auto-closing taps is taken out, should we remove the "close?" argument on "sub" as well?

Comment by Ghadi Shayban [ 16/Oct/14 11:34 AM ]

It's more than respecting the flag. Related to the close behavior, channels can tap and untap without receiving anything while the mult process happily distributes a value to another set of channels (like the ABA problem). Could also make it an error to tap after the close is distributed to the last deref'ed set of channels. That is different than the familiar permanent nil receive, but mults already differ from simple channels.





[ASYNC-70] documentation of thread macro should include behavior of nil (closes the channel) Created: 15/May/14  Updated: 14/Nov/14

Status: Reopened
Project: core.async
Component/s: None
Affects Version/s: None
Fix Version/s: None

Type: Defect Priority: Major
Reporter: Howard Lewis Ship Assignee: Unassigned
Resolution: Unresolved Votes: 0
Labels: reader
Environment:

0.1.278.0


Attachments: Text File async-70-patch.1.txt    
Patch: Code

 Description   

I needed the ability to invoke some code in a thread, and have the channel close if nil was returned.

Digging though the code, I discovered it already does this, but it is no documented in the docstring.

I'll supply a patch shortly.



 Comments   
Comment by Alex Miller [ 23/Jun/14 11:15 AM ]

The thread docstring says: "Returns a channel which will receive the result of the body when completed." The special case of a null return value is actually handled by ignoring it (because you are not allowed to explicitly put nils on a channel). The channel is closed on completion regardless. I'm not sure I get what needs to be added here and no patch, so closing. Reopen if there is a concrete suggestion to evaluate.

Comment by Howard Lewis Ship [ 14/Nov/14 10:41 AM ]

It's a tiny little patch. DRY is great for code, less so for docs!





[ASYNC-71] exception behavior for thread macro is hard coded Created: 15/May/14  Updated: 06/Aug/14

Status: Open
Project: core.async
Component/s: None
Affects Version/s: None
Fix Version/s: None

Type: Enhancement Priority: Major
Reporter: Howard Lewis Ship Assignee: Unassigned
Resolution: Unresolved Votes: 1
Labels: exceptions
Environment:

0.1.278.0-76b25b-alpha



 Description   

Currently, the thread macro's behavior when an exception occurs is to print the exception (with println), and swallow it.

Although I make a habit of wrapping a try around code inside a thread form, I still find this a bit limited; it would be nice if there was a function that could be dynamically bound, that handled the case of an exception inside a thread.



 Comments   
Comment by Alex Miller [ 06/Aug/14 10:29 AM ]

Patches on ASYNC-76 have been applied and exceptions will now flow up to the top of the thread, where they can be caught by the standard Thread uncaught exception handler mechanism or ultimately by the default uncaught exception handler, which can be set for the application. There may still be further changes to support exception handling in thread/go.





[ASYNC-73] try-catch-finally broken inside go-blocks Created: 21/Jun/14  Updated: 21/Jun/14

Status: Open
Project: core.async
Component/s: None
Affects Version/s: None
Fix Version/s: None

Type: Defect Priority: Major
Reporter: Moritz Ulrich Assignee: Unassigned
Resolution: Unresolved Votes: 0
Labels: None
Environment:

clojure 1.6.0, clojurescript 0.0-2234, core.async 0.1.303.0-886421-alpha1



 Description   

The following will cause the failed assertion (ioc_helpers.cljs:155)
when evaluated as a whole, and will correctly catch the Error when
just the `try' gets evaluated. The `finally' block runs only if the
inner block is evaluated:

(go
  (try
    (throw (js/Error. "asdf"))
    (catch ExceptionInfo e
      (println "ExceptionInfo"))
    (catch js/Error e
      (println "js/Error"))
    (finally
      (println "finally"))))

Another notable observation is that changing the order of the `catch'
blocks will change the behavior: If the (catch js/Error ...) is the
first catch block, it will work just as expected.






[ASYNC-74] Provide a chan? function to determine if a given var is a channel Created: 23/Jun/14  Updated: 30/Jun/14

Status: Open
Project: core.async
Component/s: None
Affects Version/s: None
Fix Version/s: None

Type: Enhancement Priority: Major
Reporter: Reno Reckling Assignee: Unassigned
Resolution: Unresolved Votes: 2
Labels: None

Attachments: File drupp-async-74.diff    

 Description   

When using channels, for example for streaming http bodies, it would be really helpful if there would be a chan? function to check whether a given var is a channel or not.

Aleph already does it like that in it's http-server/client implementation using lamina channels and it feels quite consistent.

It would of course also be more consistent with the rest of clojure to provide a type checking function for basic objects.



 Comments   
Comment by David Rupp [ 30/Jun/14 9:58 AM ]

Implement chan? predicate.

Comment by Timothy Baldridge [ 30/Jun/14 10:50 AM ]

I'm in favor of this, but last time I asked Rich about it his quote was "do you want a predicate for every single interface?".

Due to the implementation of core.async you'd probably need two additional predicates. One for read-port? and write-port?. You can use (satisfies? clojure.core.async.impl.protocols/ReadPort ...) but that's an internal implementation, so I'd rather have a new predicate than to tell people to touch the innards of core.async. But this call is up to Rich.

Comment by Reno Reckling [ 30/Jun/14 11:07 AM ]

I understand that reasoning. But then we would have to step up on documentation and provide a way to easily determine which interfaces are implemented by the return values of for example (chan). For me, going into the innards of core.async and trying to determine which interfaces I want, especially on very basic things like a channel, is not a user friendly approach at all.





[ASYNC-78] deadlock in multi catch Created: 05/Jul/14  Updated: 05/Jul/14

Status: Open
Project: core.async
Component/s: None
Affects Version/s: None
Fix Version/s: None

Type: Defect Priority: Major
Reporter: Lars Bohl Assignee: Unassigned
Resolution: Unresolved Votes: 0
Labels: None
Environment:

ubuntu 14.04
Leiningen 2.4.2 on Java 1.7.0_60 Java HotSpot(TM) 64-Bit Server VM
[org.clojure/clojure "1.6.0"]
[org.clojure/core.async "0.1.303.0-886421-alpha"]



 Description   

The following apparently never returns:

(defn thread-death []
  (<!! (go
        (<!
         (let [ch (chan 1)]
           (>! ch 0)
           (>! ch (try (<! ch)
                       (assert false)
                       (catch Exception _ 5)
                       (catch AssertionError _ 4)))
           ch)))))

After removing the (catch Exception _ 5) catch block it returns 4 as expected.






[ASYNC-79] (ClojureScript) go macro not correctly transforming (case) within a macro Created: 23/Jul/14  Updated: 23/Jul/14

Status: Open
Project: core.async
Component/s: None
Affects Version/s: None
Fix Version/s: None

Type: Defect Priority: Major
Reporter: Tom Locke Assignee: Unassigned
Resolution: Unresolved Votes: 2
Labels: cljs, macro
Environment:

[org.clojure/clojure "1.6.0"]
[org.clojure/clojurescript "0.0-2234"]
[org.clojure/core.async "0.1.303.0-886421-alpha"]


Attachments: File core_async_bug.tgz    

 Description   

(case-let) is a macro to handle messages of the form [:message-tag, arg1, arg2, ...] with the arguments bound to local variables. It fails to work correctly when used within a go block. Note that a simple macro with a case, e.g. (defmacro my-case [expr & cases] `(case ~expr ~@cases)) does work.

(Sample project attached)

(case-let) definition:

(ns core-async-bug.macros)

(defmacro case-let [expr & cases]
  (let [msg (gensym)]
    `(let [~msg ~expr]
       (case (first ~msg)
         ~@(apply concat (for [[key [args & body]] (partition 2 cases)]
                    [key `(let [~args (rest ~msg)] ~@body)]))))))

ClojureScript test code:

(ns core-async-bug.core
  (:require-macros [cljs.core.async.macros :refer [go]]
                   [core-async-bug.macros :refer [case-let]])
  (:require [cljs.core.async :refer[<! put! chan]]))

(enable-console-print!)

; go block with manual case + lets - works
(let [c (chan)]
  (go
    (let [msg (<! c)]
      (case (first msg)
        :a (let [[x] (rest msg)] (println "First :a" x))
        :b (let [[y] (rest msg)] (println "First :b" y)))))
  (put! c [:b 123]))

; case-let outside of go - works
(case-let [:b 123]
  :a ([x] (println "Second :a" x))
  :b ([y] (println "Second :b" y)))

; case-let inside go - broken
(let [c (chan)]
  (go
    (case-let (<! c)
      :a ([x] (println "Third :a" x))
      :b ([y] (println "Third :b" y))))
  (put! c [:b 123]))

Browser console output:

Second :b 123
First :b 123
Third :a 123          <-- Should not be there!
Third :b 123


 Comments   
Comment by Tom Locke [ 23/Jul/14 3:46 AM ]

More discussion here: https://groups.google.com/forum/#!topic/clojurescript/w21nNWkKR-c

Comment by Tom Locke [ 23/Jul/14 1:58 PM ]

I've discovered an easy workaround for this problem. During macro-expansion core names like case become fully qualified, i.e. cljs.core/case, and it seems that the go macro then fails to recognise the case as such. Replacing case with ~'case in the definition of let-case fixes the problem.

I would hope this leads to an easy fix for someone who knows the core.async codebase.

The working macro is:

(defmacro case-let [expr & cases]
  (let [msg (gensym)]
    `(let [~msg ~expr]
       (~'case (first ~msg)
         ~@(apply concat (for [[key [args & body]] (partition 2 cases)]
                    [key `(let [~args (rest ~msg)] ~@body)]))))))




[ASYNC-90] Pub/sub leaks memory Created: 13/Sep/14  Updated: 15/Oct/14

Status: Open
Project: core.async
Component/s: None
Affects Version/s: None
Fix Version/s: None

Type: Defect Priority: Major
Reporter: Ziyang Hu Assignee: Unassigned
Resolution: Unresolved Votes: 0
Labels: None
Environment:

[org.clojure/clojure "1.6.0"]
[org.clojure/core.async "0.1.303.0-886421-alpha"]

java version "1.7.0_45"
Java(TM) SE Runtime Environment (build 1.7.0_45-b18)
Java HotSpot(TM) 64-Bit Server VM (build 24.45-b08, mixed mode) (reproducible on OpenJDK 7 as well)

OS X 10.9.4 (reproducible on Ubuntu Linux 14.04 as well)



 Description   

The following code will cause OOME:

(require '[clojure.core.async :refer [chan close! <! pub sub unsub go timeout]])

(def p* (chan))

(def p (pub p* :topic))

(go
  (loop []
    (let [s (chan)
          t (rand-int Integer/MAX_VALUE)]
      (sub p t s)
      (<! (timeout 10))
      (unsub p t s)
      (close! s)
      (recur))))

(It grows slowly: to see the OOME in a reasonable amount of time, either give JVM very small
memory like 64m, or remove the timeout step.)

I tried to profile the code, and the reason seems to be that even though I
unsubed the channel from the port, something is still retained which causes
the heap to be used up.



 Comments   
Comment by Ziyang Hu [ 13/Sep/14 8:50 AM ]

Here is the problem:

https://github.com/clojure/core.async/blob/96de9a47ac511d9bb4309645a3bc594a2fc0c33a/src/main/clojure/clojure/core/async.clj#L826-L828

When unsub* is called, it just untaps the channel from the mult specified by the topic. The mult still remains in the atom called mults even if the mult has no taps left.

I can't think of a clean fix for this problem, since currently the channels which are tapping a mult aren't exposed, i.e., we currently have no way of knowing if a mult has any taps on it.

Comment by Ghadi Shayban [ 15/Oct/14 11:27 PM ]

I also cannot think of a clean fix, as mults do not expose their registrants. (The notion of "current" is a concurrent system is subtle)
Besides saying (and perhaps amending the docstring) that pubs are indeed resources, their footprint grows by the # of seen topics, and that resources should almost always be bounded.





[ASYNC-92] go macro removes binding forms that are intialized with logical false value Created: 03/Oct/14  Updated: 30/Oct/14

Status: Open
Project: core.async
Component/s: None
Affects Version/s: None
Fix Version/s: None

Type: Defect Priority: Major
Reporter: Oleh Palianytsia Assignee: Unassigned
Resolution: Unresolved Votes: 1
Labels: None
Environment:

org.clojure/core.async "0.1.346.0-17112a-alpha"


Attachments: File fix-async-92.diff    
Approval: Triaged

 Description   
(require '[clojure.core.async :as a])

(a/go (let [a nil] (a/alts! (if a <whatever> <whatever>)))) // Unable to resolve a
(a/go (let [a nil] (a/<! (if a <whatever> <whatever>))) // Unable to resolve a

Seems that 'go' macro removes falsely initialized symbols that are used as channels, because
in both cases there's exception, that says " Unable to resolve symbol: a in this context".



 Comments   
Comment by Willy Blandin [ 17/Oct/14 12:19 PM ]

Confirmed.
Bug was introduced between 0.1.278.0-76b25b-alpha and 0.1.295.0-9ea6ef-alpha.

Comment by Willy Blandin [ 17/Oct/14 12:27 PM ]

Worked around with:

(defmacro workaround-async-92
  "Hack to workaround core.async bug
   cf. http://dev.clojure.org/jira/browse/ASYNC-92"
  []
  ;; has to be a list
  `(do nil))

(let [a (workaround-async-92)]
  ...)
Comment by Leon Grapenthin [ 23/Oct/14 11:55 AM ]

modifies two methods of the RawCode inst so that they check:collected-locals in locals via contains? before ignoring them

Comment by Ghadi Shayban [ 23/Oct/14 5:19 PM ]

Hi Leon, thanks for the patch. Can you fill out a Contributor Agreement? http://clojure.org/contributing

Comment by Leon Grapenthin [ 24/Oct/14 7:17 AM ]

I did, yesterday. Got an automatic confirmation email saying Rich Hickey signed it. Anything else I should do with it?





[ASYNC-94] Allow user supplied executor/thread-pool (and potentially other options) to be used in go, thread macros and chan Created: 09/Oct/14  Updated: 17/Jul/15

Status: Open
Project: core.async
Component/s: None
Affects Version/s: None
Fix Version/s: None

Type: Enhancement Priority: Major
Reporter: Max Penet Assignee: Unassigned
Resolution: Unresolved Votes: 9
Labels: None

Attachments: File ASYNC-94e.diff    
Patch: Code
Approval: Triaged

 Description   

Right now thread pools used by core.async are defined as globals, they are shared by all go or thread macro on their respective pools:

see https://github.com/clojure/core.async/blob/ac0f1bfb40237a18dc0f03c0db5df41657cd23a6/src/main/clojure/clojure/core/async.clj#L391-L411
and https://github.com/clojure/core.async/blob/ac0f1bfb40237a18dc0f03c0db5df41657cd23a6/src/main/clojure/clojure/core/async/impl/dispatch.clj#L16

It would be very useful to have more control over this, hopefully not via a simple setter, like the one we have in core for agents (set-agent-send-executor!), since it only allows one pool to be used at a time.
One possible solution would be to do that via an option map parameter to thread-call, and/or some other mechanism for go blocks (go-call with the same kind of signature). Making this parameter a map would also allow to further extend internals in the future, if necessary, without requiring heavy changes to the API.



 Comments   
Comment by Max Penet [ 15/Mar/15 9:35 AM ]

I have a patch here https://github.com/mpenet/core.async/commit/1c2b0a9af9a5e891af0f1631b8debd337a73999d that adds this features.

It adds 2 macros: go* and thread*. These are identical to go/thread, but takes first an option map, then the body. The option map can receive an :executor key with an clojure.core.async.impl.protocols/Executor implementation.

I took the liberty to move the `thread` executor in the same namespace as the ioc/go one, and have it support the same interface instead of using a raw j.u.c Executor instance. Both instances declared in this namespace are now lazy (in a delay block), preventing useless thread pool initialisations/leaks.

Lastly I also modified thread-call to support that same option map.

I'll attach the .patch file to the ticket shortly.

Comment by Max Penet [ 15/Mar/15 12:22 PM ]

well actually this patch is incomplete, I am just discovering that ioc-macros rewriting leads to calls on the global thread pool (via put!, take! callbacks in particular).

We could allow to pass an executor at channel creation to force ops on this channel to run there relatively easily (partial example here https://github.com/mpenet/core.async/commit/1449b3842052033cdf917ae92259ad9789722fdb).

I believe this is how manifold handles it as well, but there might be more unknowns for me at this point.

Comment by Max Penet [ 15/Mar/15 2:51 PM ]

patch with `chan` modifications included. chan now can take an additional parameter that would be an clojure.core.async.impl.protocols/Executor compatible executor, this executor will then be used instead of the global default pool when supplied.

Comment by Max Penet [ 17/Mar/15 9:52 AM ]

provide patch as single commit (more readable). You can also view the changes here: https://github.com/mpenet/core.async/commit/e7dea04553935863d1abb1880d84bbdc273854ec

Comment by Ghadi Shayban [ 17/Mar/15 1:32 PM ]

Hi Max. I think that having some better control over this is necessary.

There are basically two needs:
1) callbacks need to wake up parked go-blocks.
2) running heavier `thread`s (which may block and thus cannot share the same pool)

Right now channel callbacks don't know what they are waking up. I think that's a good design. If they are waking up a thread, it's only indirectly doing so by delivering a promise [1]
I think there should be no mention of a specific executor in channels.clj Having multiple different places to run go-blocks is probably an antipattern as go-blocks should not be blocking or doing I/O.

I very much welcome some control over the `thread` block's executors though.

[1] https://github.com/clojure/core.async/blob/master/src/main/clojure/clojure/core/async.clj#L137

Comment by Max Penet [ 18/Mar/15 3:18 AM ]

Hi Ghadi,

Thanks for the feedback, but I don't think you read my patch correctly.

I do not change the case you mention (promise), the only thing that's changed is that for the few cases where clojure.core.async.impl.dispatch/run is called, meaning the function is run in a globally defined threadpool used by everything in core.async with the exception of thread, it can *optionally* be done via a function argument supplied threadpool. The default is unchanged, if you don't pass an executor you get the current master core.async behavior without any change at all.

Few people mentioned that need, some publicly, some in private, myself included, and I know of a team who expressed the same concerns and just switched one of their systems from core.async to manifold because the latter offer more knobs of that sort.

In the end it's about control, it changes nothing to core.async and it's current execution model if you don't care about it, but it's important to people who needs that kind of fine grained tuning.

It came up again a couple of days ago on twitter: https://twitter.com/puredanger/status/576378306062262272 and you could certainly find reference of it in the irc logs as well.

I would have released this as a lib if that was possible, but the code implicated is sometimes very deeply rooted and make it nearly impossible to do without forking the whole library, hence that patch.

Comment by Max Penet [ 18/Mar/15 4:02 AM ]

same as previous patch, added missing arity to promise-chan

Comment by Stuart Halloway [ 11/Jul/15 7:38 AM ]

Consider my upvote to be for the problem, not necessarily for the approach taken here.

It seems to me that go and thread are separate cases, deserving of separate consideration. It isn't clear to me why chan needs consideration at all, and I don't want to have to read a patch (or learn about impl namespaces) to discover why. Can somebody explain the case for chan without reference to implementation?

Is there a dev list discussion on this?

Comment by Max Penet [ 17/Jul/15 12:42 AM ]

Sure, as long as this is improved/fixed.

If you would read the patch, or the source for channels you would see
that channels run their callbacks on a threadpool.

When this patch was submitted I spent time discussing it with Timothy
Baldridge on IRC and made some changes at his suggestion. Since you
probably share offices with him I suggest you discuss the current
approach with him as he is more familiar with the codebase than both
of us.





[ASYNC-97] CLJS: alts! sporadically failing to unblock despite channel activity (Safari 7) Created: 15/Oct/14  Updated: 16/Oct/14

Status: Open
Project: core.async
Component/s: None
Affects Version/s: None
Fix Version/s: None

Type: Defect Priority: Major
Reporter: Oliver Charles Assignee: Unassigned
Resolution: Unresolved Votes: 0
Labels: None

Attachments: Text File gistfile1.txt     File main.cljs     File socket.cljs    

 Description   

Hi all, bit of a tricky bug to report here... We're seeing some problems with using core.async in ClojureScript on Safari 7. Our application is built around a large event loop that blocks on a message from one of many channels that correspond to user activity or API calls. The problem seems to lie within this event loop - we are using alts! to pull a message out of any available channel, but sometimes logging shows that we reach alts! and never unblock. However, with a little more logging, I can see that there are subsequent writes to one of the channels in the list of channels passed to alts!, so I'm not really sure what's going on.

That's the high level overview, now on to some code.

Our main event loop is as follows:

(log "Entering main event loop.")
  (go
    (while true
      (log "alts! channel hashes: " (map hash (:channels @app)))
      (let [[message channel] (alts! (seq (:channels @app)))]
        (log "alts! unblocked, calling our process-message")
        (swap! app process-message message channel)
        (log "process-message completed, looping"))))

process-message here is our a function internal to our application, but I don't think it's details are necessarily important. In the scenario where Safari gets stuck, the log looks like:

[Log] process-message completed, looping (main.js, line 62)
[Log] alts! channel hashes:  (16 12 19 33) (main.js, line 82)
[Log] Socket connected. (socket.js, line 309)
[Log] put! to channel with hash  19 (socket.js, line 86)
[Log] The message is [:metronome [:staff [{:description nil, :deletable true, :email nil, :isAdmin true, :isTrainer false, :telephone nil, :name "Fynder Admin", :picture nil, :userId 1} {:description nil, :deletable fa... (socket.js, line 87)
[Log] put! callback gave us true (socket.js, line 89)
[Debug] Metronome: staff data decoded. put! complete.: 12.282ms (socket.js, line 93)
[Log] put! to channel with hash  19 (socket.js, line 86)
[Log] The message is [:metronome [:class-types [{:deletable false, :picture nil, :name "CycleCore", :id 2, :description "CycleCore is a 55-minute dual workout concept that combines 30 minutes of intense cardiovascular ... (socket.js, line 87)
[Log] put! callback gave us true (socket.js, line 89)
[Debug] Metronome: class-types data decoded. put! complete.: 1.288ms (socket.js, line 93)
[Log] put! to channel with hash  19 (socket.js, line 86)
[Log] The message is [:metronome [:locations [{:studios [{:deletable false, :name "Kensington", :id 1, :locationId 1, :description "Studio (11a) sits just off Stratford Road in Stratford Studios. To find us, just pass ... (socket.js, line 87)
[Debug] Metronome: locations data decoded. put! complete.: 0.884ms (socket.js, line 93)

Note that we see a log entry for "alts! channel hashes", but we never seen "alts! unblocked". However, note the list of hashes passed to alts!. Channel 19 is mentioned, but subsequently we put! to channel 19... yet we still don't get unblocked. Something that also strikes me as suspicious, is that while we're blocked at alts!, two calls to put! have succeeded immediately, for a channel that is bounded to contain only one element at a time. Maybe I'm misunderstanding something, but I wouldn't expect the immediate-put callback to be invoked more than once. Interestingly that last put! doesn't invoke the callback.

Unfortunately, reproduction of this bug is reasonably difficult. I can somewhat reliably reproduce it by quitting Safari, re-opening it, and navigating to the dev server. About 1 in 15 attempts get stuck in this fashion. I wondered if it was something to do with Safari's MessageChannel implementation - you can see in the log entries where nexttick.js calls its callback, which seems to be how dispatch is working in my browser.

I'd be very happy to help provide any more information that's useful, but this problem is now outside my ability to debug it. While the code is proprietary, I'd be happy to temporarily add people to the Github project in order to try and get this fixed. We have development APIs servers that you can point at, so it should be just a case of running lein cljs.

I've attached our code for our Socket.io wrapper and our main event loop. Sadly I do not yet have a minimal test-case - I wouldn't really know where to begin.



 Comments   
Comment by Oliver Charles [ 15/Oct/14 7:37 AM ]

I went deep into the guts of the Google Closure library and changed getSetImmediateEmulator_ to:

goog.async.nextTick.getSetImmediateEmulator_ = function() {
  // Fall back to setTimeout with 0. In browsers this creates a delay of 5ms
  // or more.
  return function(cb) {
    goog.global.setTimeout(cb, 0);
  };
};

and I haven't been able to get it stuck. So maybe MessageChannel has problems in Safari...

Comment by Ghadi Shayban [ 15/Oct/14 10:40 PM ]

Hi Oliver, seems like a race, and we'll figure this out.

Would you mind compare running upon 0.1.319.0-6b1aca-alpha vs 0.1.346.0-17112a-alpha ?

alts! should be passed an indexed collection/vector btw. Passing a seq wouldn't cause this bug, just something to note.

Comment by Oliver Charles [ 16/Oct/14 6:53 AM ]

Hi Ghadi,

0.1.319.0-6b1aca-alpha is what the initial report was against - I should have mentioned that. So 0.1.319.0-6b1aca-alpha does get stuck.

0.1.346.0-17112a-alpha however does not get stuck, which is odd - as I'm sure I tried upgrading to this! I've tried on two Macs that are normally problematic, and they didn't get stuck once. I'm pushing this out to more of our testers and will see what happens.

Comment by Oliver Charles [ 16/Oct/14 7:16 AM ]

Aha, I knew it wouldn't be that easy! Upon releasing this to production, it immediately froze again. The dev server runs with very different optimisations though, so I'm going to build a production release and serve that locally - will see what happens there.

Comment by Oliver Charles [ 16/Oct/14 7:28 AM ]

Yep, definitely a problem with optimisations. Here is my Shadow Build configuration

(ns fynder.shadowbuild
  (:require [clojure.java.io :as io]
            [shadow.cljs.build :as cljs]))

(defn define-modules [state]
  (-> state
      (cljs/step-configure-module :cljs '[cljs.core clojure.walk clojure.string cljs.reader cljs.core.async] #{})
      (cljs/step-configure-module :test-support '[inflections.core no.en.core enfocus.bind fynder.winchan] #{:cljs})
      (cljs/step-configure-module :devel '[fynder.devel] #{:cljs})
      (cljs/step-configure-module :admin '[fynder-admin.main] #{:cljs})
      (cljs/step-configure-module :trainer '[fynder-trainer.main] #{:cljs})
      (cljs/step-configure-module :mobile '[fynder-mobile.main] #{:cljs})
      (cljs/step-configure-module :sweatybetty '[fynder-sweatybetty.main] #{:cljs})
      (cljs/step-configure-module :loader '[fynder-loader.loader] #{:cljs})))

(defn dev
  "build the project, wait for file changes, repeat"
  [& args]
  (let [state (-> (cljs/init-state)
                  (cljs/enable-source-maps)
                  (assoc :optimizations :advanced
                         :pretty-print false
                         :work-dir (io/file "target/cljs-work")
                         :public-dir (io/file "resources/dev")
                         :public-path "")
                  (cljs/step-find-resources-in-jars)
                  (cljs/step-find-resources "src/cljs/")
                  (cljs/step-finalize-config)
                  (cljs/step-compile-core)
                  (define-modules))]
    ;; compile, flush, reload, repeat
    (loop [state state]
      (let [new-state (try
                        (-> state
                            (cljs/step-compile-modules)
                            (cljs/flush-unoptimized)
                            (cljs/wait-and-reload!))
                        (catch Throwable t
                          (prn [:failed-to-compile t])
                          (.printStackTrace t)
                          (cljs/wait-and-reload! state)))]
        (recur new-state)))))

(defn prod
  "build the project, wait for file changes, repeat"
  [& args]
  (-> (cljs/init-state)
      (cljs/enable-emit-constants)
      (cljs/enable-source-maps)
      (assoc :optimizations :advanced
             :pretty-print false
             :work-dir (io/file "target/cljs-work")
             :public-dir (io/file "resources/prod")
             :externs ["react/externs/react.js"
                       "externs/base32.js"
                       "externs/jquery.js"
                       "externs/react_addons.js"
                       "externs/fastclick.js"
                       "externs/socket.io.js"
                       "externs/moment.js"
                       "externs/papa.js"
                       "externs/markdown.js"
                       "externs/xss.js"
                       "externs/facebook.js"
                       "externs/checkout.js"])
      (cljs/step-find-resources-in-jars)
      (cljs/step-find-resources "src/cljs/")
      (cljs/step-finalize-config)
      (cljs/step-compile-core)
      (define-modules)
      (cljs/step-compile-modules)
      (cljs/closure-optimize)
      (cljs/flush-to-disk)
      (cljs/flush-modules-to-disk)))

If I run in the dev profile, then I can't get it stuck. If I switch over to the production profile (and serve the result of lein publish with python's SimpleHTTPServer), then Safari does get stuck.





[ASYNC-99] go block with <! inside protocol method invocation fails to compile (dispatch :protocol-invoke on '-item-to-ssa') Created: 23/Oct/14  Updated: 04/Nov/14

Status: Open
Project: core.async
Component/s: None
Affects Version/s: None
Fix Version/s: None

Type: Defect Priority: Major
Reporter: Valentin Waeselynck Assignee: Unassigned
Resolution: Unresolved Votes: 3
Labels: go-macro, protocols
Environment:

[org.clojure/core.async "0.1.346.0-17112a-alpha"]
happened on both [org.clojure/clojure "1.7.0-alpha1"] and [org.clojure/clojure "1.6.0"]

java version "1.8.0_05"
Java(TM) SE Runtime Environment (build 1.8.0_05-b13)
Java HotSpot(TM) 64-Bit Server VM (build 25.5-b02, mixed mode)



 Description   

I was programming a function involving a core.async go block, when I stumbled on a strange compilation error :

CompilerException java.lang.IllegalArgumentException: No method in multimethod '-item-to-ssa' for dispatch value: :protocol-invoke, compiling:(NO_SOURCE_PATH:2:3)

I experimented a little to try and strip down the problem, and found it was very generic. Say I have any protocol MyProtocol :

(defprotocol MyProtocol
  (do-something [this param] "some method"))

The following code will not compile, failing with the exception I showed you above :

(defn uncompilable! [me ch] 
  (go 
    (do-something me (<! ch)) ;; apparently, it hurts to use <! in a protocol method invocation 
    ))

However, the following 2 will compile without any problem :

(defn compilable! [me ch] 
  (let [call-it #(do-something me %)] ; wrapping the protocol call in a function
    (go 
     (call-it (<! ch))
     )))

(defn compilable-2! [me ch] 
  (go 
    (let [my-value (<! ch)] ; taking out the <! call
      (do-something me my-value))
    ))

It seems to me the '<! inside protocol method invocation form' is a situation which the go macro fails to handle.






[ASYNC-100] core.async with multiple catch blocks causing weird loop behaviour Created: 27/Oct/14  Updated: 18/Jun/15

Status: Open
Project: core.async
Component/s: None
Affects Version/s: None
Fix Version/s: None

Type: Defect Priority: Major
Reporter: Tom Coupland Assignee: Unassigned
Resolution: Unresolved Votes: 3
Labels: None

Attachments: Text File ASYNC-100_2.patch    
Patch: Code and Test
Approval: Triaged

 Description   

I've been seeing this weird looping behavior with some go loops over the last few days. An exception is being thrown from a function within the loop and rather than logging and looping back around to a waiting take, the loop seems to just hop back to the line before the function call.

I've managed to boil it down to the following:

(def s (chan))
(def single
  (go-loop []
    (try
      (prn "Awaiting single")
      (<! s)
      (prn "Single")
      (throw (new Throwable))
      (catch Throwable t
        (prn t)))
    (recur)))

(def d (chan))
(def double
  (go-loop []
    (try
      (prn "Awaiting double")
      (<! d)
      (prn "Double")
      (throw (new Throwable))
      (catch Exception re
        (prn re))
      (catch Throwable t
        (prn t)))
    (recur)))

Now if you (>!! s :a), you'll see the throwable printed out and the loop go back to waiting on the s channel. However, (>!! d :a) and you'll get to enjoy an infinite stream of 'Double'. In actual fact you can remove the -loop from double and get the same result.

Not sure what's going on here at all. In the macro expanded version of double '(prn t)' doesn't appear at all (it does in single's expansion), so it looks like it's not surviving the move into the state machine and instead is routing back to (prn "Double") or the take isn't really completing somehow, leaving the :a on the chan.



 Comments   
Comment by Paavo Parkkinen [ 03/Dec/14 5:30 PM ]

From my experiments, it seems what's causing the issue isn't the two catch clauses, but the fact that you are catching an Exception first, and then the Throwable. When I switched the two catch clauses (Throwable first) the issue went away.

You don't even need the two catch clauses to trigger it, a single catch clause of an Exception will do.

;; Works
(require '[clojure.core.async :as async])
(def c (async/chan))
(def st
  (async/go-loop []
    (try
      (prn "Awaiting")
      (async/<! c)
      (prn "Received")
      (throw (new Throwable))
      (catch Throwable t
             (prn t)))
    (recur)))
(async/>!! c :a)
;; Doesn't work
(require '[clojure.core.async :as async])
(def c (async/chan))
(def se
  (async/go-loop []
    (try
      (prn "Awaiting")
      (async/<! c)
      (prn "Received")
      (throw (new Throwable))
      (catch Exception t
             (prn t)))
    (recur)))
(async/>!! c :a)
Comment by Paavo Parkkinen [ 04/Dec/14 12:18 AM ]

Attaching a patch that fixes the original issue, but not the one in my comment above with an uncaught exception. All test cases pass.

Instead of just picking the first catch block, and creating a state machine block and exception frame for that one, I create blocks and frames for all of them.

I tried creating a test case too, but was unable to create one that would reproduce the error. I'll spend some more time trying to write a test case, but I wanted to submit the code patch without the test coverage first.

Comment by Paavo Parkkinen [ 04/Dec/14 7:45 PM ]

Attached patch with test case included.

Comment by Stuart Sierra [ 18/Jun/15 2:52 PM ]

Possible workaround until this is fixed: Just have one catch clause, catching all Throwable. Then examine the type of the throwable object inside the catch block to decide what to do.





[ASYNC-102] implement IDeref, IBlockingDeref for channels Created: 05/Nov/14  Updated: 12/Aug/15

Status: Open
Project: core.async
Component/s: None
Affects Version/s: None
Fix Version/s: None

Type: Enhancement Priority: Major
Reporter: Stuart Halloway Assignee: Unassigned
Resolution: Unresolved Votes: 0
Labels: None

Attachments: Text File async-102-2.patch     Text File async-102.patch    
Patch: Code and Test
Approval: Vetted
Waiting On: Alex Miller

 Description   
  • each deref will take a value from the channel
  • deref from a closed channel returns nil
  • deref with timeout must either timeout or take value from the channel (not both!). Might just want to call some variant of alt!! through the var
  • use internals efficiently
  • do NOT implement IPending/realized, neither interpretation is that great

Patch: async-102-2.patch

The implementation of deref and timed deref is conceptually straightforward, but is difficult to implement due to circular namespace dependencies. Suggestions welcome.



 Comments   
Comment by Stuart Halloway [ 09/Nov/14 8:24 AM ]

The async-102.patch does not match the description "a channel becomes realized when it is closed" – instead, a channel becomes realized when a value is currently available. Both of these interpretations seem problematic to me, and I think the definition of realized? needs clarification to support this new case. The other uses of realized? in Clojure all guarantee that subsequent derefs will be filled immediately, but that will not be true for channels if another consumer takes the value.

Once that is solved, I need better doc strings for the core.async implementation protocols before I can screen this. What guarantees are made about use of threads? Where does the work enqueue, and what happens if the queue is full?

Comment by Alex Miller [ 09/Nov/14 10:32 PM ]

My implementation of realized? is based on Rich's response to your question (from internal chat): "open+empty = false, else true" and I believe that is what's implemented and reflected in the tests. This also makes more sense to me than only being realized on close.

Your comment vs other cases of realized? seems accurate, so I agree that's a question.

Can you be more specific on which implementation protocol? I'm guessing you specifically mean Channel and Buffer. Any thread could be calling into the Channel. The M2MC protects its internal state with a mutex and will forward calls down to the Buffer. All calls into the buffer are protected by the channel mutex. M2MC enqueues pending puts and takes in the puts and takes lists. Those are bounded by the fixed limit clojure.core.async.impl.protocols/MAX-QUEUE-SIZE (1024), at which point an exception is thrown.

Comment by Alex Miller [ 05/Jan/15 12:33 PM ]

The conceptual implementations of IDeref and IBlockingDeref on ManyToManyChannel using existing async constructs is relatively straightforward:

(deftype ManyToManyChannel
  #_existing_code...
  IDeref
  (deref [this] (<!! this))

  IBlockingDeref
  (deref [this ms timeoutValue]
    (alt!!
      this ([val _] val)
      (timeout ms) timeoutValue)))

However, M2MC is defined in clojure.core.async.impl.channels. <!!, alt!!, and timeout are all defined in clojure.core.async, which depends on clojure.core.async.impl.channels, so there is a cyclic dependency problem here. The <!! and timeouts are pretty easy to deal with looking up the var behind delay like this:

(def ^:private <!!' (delay (find-var 'clojure.core.async/<!!)))

;; then:
  IDeref
  (deref [this] (@<!!' this))

However, I'm a little stumped on how to do the equivalent with alt!!, which is a macro around do-alt and alts!!.

Comment by Ghadi Shayban [ 05/Jan/15 1:14 PM ]

Don't rely on resolving the circular dependency. Instead make a shared alt-flag and enqueue the two handlers [1]. fret will be delivering a promise, and then deref the promise.

[1] https://github.com/clojure/core.async/blob/master/src/main/clojure/clojure/core/async.clj#L230-L231

Comment by Alex Miller [ 06/Jan/15 11:42 AM ]

New patch goes back to prior implementation of blocking deref (basically guts of <!!) and adds new alts-based version of timed deref.

Comment by Fogus [ 09/Jan/15 1:11 PM ]

> However, I'm a little stumped on how to do the equivalent with
> alt!Unable to render embedded object: File (, which is a macro around do-alt and alts) not found.!.

I wonder if the following will solve the alts!! problem:

(def ^:private ^:macro alts!!' (deref (find-var 'clojure.core.async/alts!!)))

And then using it in the deref impl behind IBlockingDeref. As it stands it's difficult to grok the code as written given that it's using some core.async implementation details (indeed, the same code). Whereas your implementation of IBlockingDeref.deref is crystal clear, the implementation prompted by the cyclic dependency is... less so. Not to belabor the point (which I seem to be), but alts!! is a clear implementation for this, but using the guts muddies the water. If I had my druthers, I'd prefer clarity.

That said, being a human macro-expander I was able to reason through the implementation eventually.

Comment by Alex Miller [ 28/Jan/15 2:23 PM ]

That doesn't seem to work to me. If you have some formulation that would, that would be great though!





[ASYNC-103] promise-chan Created: 05/Nov/14  Updated: 28/Aug/15

Status: Reopened
Project: core.async
Component/s: None
Affects Version/s: None
Fix Version/s: None

Type: Enhancement Priority: Major
Reporter: Stuart Halloway Assignee: Alex Miller
Resolution: Unresolved Votes: 3
Labels: None

Attachments: Text File async-103-2.patch     Text File async-103-3.patch     Text File async-103-4.patch     Text File async-103-5.patch     Text File async-103-6.patch     Text File async-103.patch    
Patch: Code and Test
Approval: Screened
Waiting On: Alex Miller

 Description   

A promise-chan is a new kind of core.async channel that gives promise semantics to a channel.

  • Takes block prior to put or close!
  • After a put, all pending takes are notified, and any subsequent takes return the put value immediately
  • After a close (and no put), all pending takes are notified with nil, and any subsequent takes return nil immediately

In practice, a promise-chan is a normal channel that uses a promise-buf:

  • buf can only take on one value (or nil if closed) ever
  • buf is never emptied: N consumers can all get the same, one and only value by reading from the channel

In order to make close! sensible with this new kind of buf

  • all buffers need to be closeable
  • closing a channel closes its buffer
  • all basic buffer types do nothing when closed
  • a promise-buf make a one-time transition to having no value when closed (if not previously delivered)
  • closing a buffer is implementation detail not exposed to end users

Approach:

  • Buffer protocol now has a close-buf! function (close! would have overlapped the Channel function close!). close-buf! is invoked on a buffer when the channel is closed, allowing it to update itself if necessary.
  • Existing buffers implement close-buf! and do nothing (buffer still available for taking)
  • New promise-buffer implementation. Makes a one-time transition when value is supplied or buffer is closed. value is held in an unsynchronized-mutable deftype field - updates via add!* or close-buf! always happen under the containing channel mutex.
  • New api functions: promise-chan creates a channel with a promise-buffer and promise-buffer.

Patch: async-103-6.patch



 Comments   
Comment by Ghadi Shayban [ 07/Nov/14 4:06 PM ]

My initial gut reaction was that this is more related to semantics of the channel, not the buffer, and I'm wary of significant changes to esp. the impl protocols. But after seeing an impl, it looks straightforward and the changes aren't too significant. (Another possibility is to make another simpler implementation of a channel, with just slots for the value, lock and pending takers before the value is delivered. No slots for xfn or putters or buffer handling would be needed.)

Note an atom is not needed in the PromiseBuffer, just a set! on a mutable field to be inline with the other buffer flavors. If the patch continues using an atom, maybe guard val to not swap unnecessarily.

Comment by Alex Miller [ 07/Nov/14 11:09 PM ]

Good call on the atom - backed off to just using existing clojure.lang.Box. If that's too weird, will just deftype it.

Comment by Alex Miller [ 07/Nov/14 11:18 PM ]

Dur, just need the val field itself.

Comment by Fogus [ 09/Jan/15 4:27 PM ]

I'd really love to see the reason for the current impl over a more pointed promise channel (perhaps as described by Ghadi). This is a clear implementation, but with the addition of close-buf! some complexity is added for implementations in the future. Specifically, I'd like to at least see a dcostring in the protocol describing the meaning and desired semantics around close-buf! and what it should return and when. Like I said, this is a quality patch, but I'm just concerned that there are unstated assumptions around close-buf! that escape me.

Comment by Alex Miller [ 28/Jan/15 12:43 PM ]

Fogus, I looked into implementing the promise-chan a bit today but it requires replicating a significant portion of the existing chan implementation. I believe that's the primary reason to do it this way, just leveraging the majority chan code.

New -5 patch has two changes - I commented the Buffer protocol fns, and I removed promise-buffer from the api, as I think users should only use promise-chan.

Comment by Fogus [ 30/Jan/15 10:06 AM ]

The patch looks good. My only (minor) reservation is that the Buffer docstrings are written under the assumption that people will use instances only in the context of a guarded channel. I understand why of course, so I think I might be too pedantic here. Because of this I see no reason not to +1 this patch.

Comment by Alex Miller [ 23/Feb/15 8:32 AM ]

Applied patch.

Comment by Alex Miller [ 03/Apr/15 3:57 PM ]

Reopening (and likely reverting commit).

This example demonstrates that only one of the waiting blocks will be notified, instead of all, as we'd like:

(let [c (promise-chan)] 
  ;; takers
  (dotimes [_ 3] 
    (go (println (<! c)))) 

  ;; put, should get to all takers
  (go (>! c 1)))
Comment by Jozef Wagner [ 30/May/15 10:35 AM ]

ASYNC-124 fixes the issue mentioned by Alex

Comment by Leon Grapenthin [ 11/Jun/15 11:44 AM ]

Quote: "buf is never emptied: N consumers can all get the same, one and only value by reading from the channel"

This is not true with the new closing semantics introduced here, correct? If I deref a Clojure promise, I get the same value everytime. But what is the purpose in closing promise-chans after the promise has been delivered?

When is a good time to close a promise-chan? If I close after the put, which is what many core.async processes do, takers must have taken before the close which means it will often be a race condition unless explicit care is taken.

Is there a benefit or am I simply missing something here?

Comment by Herwig Hochleitner [ 08/Jul/15 2:23 PM ]

Here is my cljs implementation of a promise channel: https://github.com/webnf/webnf/blob/master/cljs/src/webnf/promise.cljs
I'm the sole author and you are free to use any of it.

Comment by Alex Miller [ 12/Aug/15 3:00 PM ]

I believe that this patch can be re-evaluated after ASYNC-124 is applied as that patch addresses the reason this was retracted.

Comment by Leon Grapenthin [ 23/Aug/15 6:46 AM ]

It seems that there are two different approaches to how a promise-chan is supposed to or can be used.

1. All consumers are supposed to be takers before the promise is delivered. Taking after delivery is not intended, hence producers should close the channel after delivery.
2. " can take at any time and either have to wait for the promise (take blocks) or they get its value right away.

(2) models the behavior of real promises. (1) rather models that of a mutable reference.

It seems that the current patch explicitly makes room for (1) by giving producers the possibility to close.

Now if you want to use a promise-chan like (1) as a producer you can't create and return a promise-chan. Instead you need to be passed a promise-chan which has (a) taker(s) already. On the consumer side, it seems like an annoying quirk to enqueue takes before being able to pass the promise-chan to the producer (since he could close the promise immediately). I can't think of a scenario where one would want to do such thing. So my new question is:

Given the lengths taken here to enable (1), are there practical usecases for that? Why is closing a promise-chan not just going to be a no op?

Comment by Alex Miller [ 24/Aug/15 12:56 PM ]

Taking after delivery is intended. It is hard sometimes to know whether a take will happen before or after delivery in asynchronous systems so both must work. close! will deliver nil to all pending or future takers - in this case (as with all channels) nil indicates a drained channel, meaning that no delivery will ever happen.

However, one good question here is - what happens when you close a promise-chan that has already been delivered?

Currently:

  • (def pc (promise-chan))
  • take from promise-chan (blocks)
  • (>!! pc 5) - delivers 5 to blocked taker(s)
  • take from promise-chan (immediately returns 5) ;; for any number of takes
  • (close! pc 5)
  • take from promise-chan (immediately returns nil) ;; ???

I could easily be persuaded that the last answer is wrong and that it should always deliver 5 regardless of whether the pc gets closed (that is closing has no effect on delivered promise-chans).

Comment by Alex Miller [ 24/Aug/15 1:03 PM ]

I just confirmed with Rich that once delivered, a promise-chan should always deliver that value to takers, regardless of whether it is subsequently closed.

Comment by Stuart Sierra [ 28/Aug/15 4:02 PM ]

Screened. Clojure version good, assuming ASYNC-124 is applied first. ClojureScript version may or may not need additional work.

I agree that the addition of close-buf! and the no-op implementations feel like working around some missing abstraction. (What does it even mean to "close" a buffer?) But the Buffer protocol is not part of the public API, it's buried down in clojure.core.async.impl.protocols. The Channel and Buffer implementations are already tightly coupled, so adding another point of collaboration does not seem likely to cause problems. Adding close-buf! is certainly preferable to duplicating the implementation of ManyToManyChannel.





[ASYNC-109] Clarify timeout doc to mention that close! should not be called on a timeout channel Created: 11/Dec/14  Updated: 30/Dec/14

Status: Open
Project: core.async
Component/s: None
Affects Version/s: None
Fix Version/s: None

Type: Defect Priority: Major
Reporter: Ryan Neufeld Assignee: Unassigned
Resolution: Unresolved Votes: 2
Labels: documentation

Attachments: Text File clarify-timeout-doc.patch    
Patch: Code

 Description   

After running into a race-condition involving closed timeout channels, it seems like it would be appropriate to mention that `close!` should never be called on a timeout channel in its docstring. The attached patch tweaks the doc string to that effect. Please advise if you'd like the wording changed a bit.



 Comments   
Comment by Howard Lewis Ship [ 12/Dec/14 11:03 AM ]

Alternately/additionally, it would be nice if close! on a timeout channel would throw an exception.

Comment by Erik Assum [ 30/Dec/14 8:25 AM ]

or alternately, make it a no-op?





[ASYNC-116] Convert core.async ClojureScript tests to cljs.test Created: 12/Feb/15  Updated: 12/Feb/15

Status: Open
Project: core.async
Component/s: None
Affects Version/s: None
Fix Version/s: None

Type: Task Priority: Major
Reporter: David Nolen Assignee: David Nolen
Resolution: Unresolved Votes: 1
Labels: None


 Description   

This is a good dog-fooding opportunity for cljs.test's async testing functionality.






[ASYNC-117] let-bindings unavailable in #js literals in go-blocks Created: 13/Feb/15  Updated: 13/Feb/15

Status: Open
Project: core.async
Component/s: None
Affects Version/s: None
Fix Version/s: None

Type: Defect Priority: Major
Reporter: Rasmus Erik Voel Jensen Assignee: Unassigned
Resolution: Unresolved Votes: 1
Labels: None
Environment:

[org.clojure/clojure "1.7.0-alpha5"]
[org.clojure/clojurescript "0.0-2760"]
[org.clojure/core.async "0.1.346.0-17112a-alpha"]



 Description   

As far as I can see, let bindings are unavailable in #js-literals within go-blocks, ie:

(go (let [a 1] (js/console.log #js[a])))

prints `[ undefined ]` instead of `[1]`

It happens both with `#js[..]` and `#js{..}`

To make it easier to reproduce the bug, I've created a minimal repository github.com/rasmuserik/cljs-bug with project.clj etc.






[ASYNC-119] ClojureScript: combine cljs.core.async.macros and cljs.core.async Created: 15/Mar/15  Updated: 15/Mar/15

Status: Open
Project: core.async
Component/s: None
Affects Version/s: None
Fix Version/s: None

Type: Enhancement Priority: Major
Reporter: Joshua Choi Assignee: Unassigned
Resolution: Unresolved Votes: 1
Labels: enhancement


 Description   

The macros in cljs.core.async.macros can now be moved into the cljs.core.async namespace, now that combined .cljs/.clj namespaces are possible in ClojureScript. This, along with ClojureScript 0.0-2755's recent improvements, would enable users to use `(require [cljs.core.async :refer [go go-loop ...])` without having to distinguish between functions and macros, and without having to use `require-macros` at all.






[ASYNC-122] Parking in finally block replaces result Created: 06/May/15  Updated: 06/May/15

Status: Open
Project: core.async
Component/s: None
Affects Version/s: None
Fix Version/s: None

Type: Defect Priority: Major
Reporter: Vesa Karvonen Assignee: Unassigned
Resolution: Unresolved Votes: 0
Labels: None
Environment:

[org.clojure/clojurescript "0.0-3211"]
[org.clojure/clojure "1.6.0"]
[org.clojure/core.async "0.1.346.0-17112a-alpha"]



 Description   

Because

(try 1 (finally 2))

evaluates to 1, I would expect

(go (println (try 1 (finally (<! (go 2))))))

to print 1, but it prints 2 in both CLJ and CLJS. This can be worked around by replacing try-finally with try-catch.






[ASYNC-123] Channel operations fail in for comprehension. Created: 25/May/15  Updated: 14/Jun/15

Status: Open
Project: core.async
Component/s: None
Affects Version/s: None
Fix Version/s: None

Type: Defect Priority: Major
Reporter: Christian Weilbach Assignee: Unassigned
Resolution: Unresolved Votes: 0
Labels: bug
Environment:

[org.clojure/clojure "1.6.0"]
[org.clojure/core.async "0.1.346.0-17112a-alpha"]



 Description   

``` clojure
(<!! (go (for [a (<! (go [1 2 3]))]
a)))
;=> (1 2 3)
```

works fine. But:

```
(go (for [a [1 2 3]
:let [b (<! (go a))]]
b))
```

1. Caused by java.lang.IllegalArgumentException
No method in multimethod '-item-to-ssa' for dispatch value: :fn

MultiFn.java: 160 clojure.lang.MultiFn/getFn
MultiFn.java: 227 clojure.lang.MultiFn/invoke
ioc_macros.clj: 492 clojure.core.async.impl.ioc-macros/item-to-ssa
ioc_macros.clj: 547 clojure.core.async.impl.ioc-macros/let-binding-to-ssa/fn
ioc_macros.clj: 122 clojure.core.async.impl.ioc-macros/all/fn/fn
ArrayChunk.java: 58 clojure.lang.ArrayChunk/reduce
protocols.clj: 98 clojure.core.protocols/fn
protocols.clj: 19 clojure.core.protocols/fn/G
protocols.clj: 31 clojure.core.protocols/seq-reduce
protocols.clj: 54 clojure.core.protocols/fn
protocols.clj: 13 clojure.core.protocols/fn/G
core.clj: 6289 clojure.core/reduce
ioc_macros.clj: 120 clojure.core.async.impl.ioc-macros/all/fn
ioc_macros.clj: 554 clojure.core.async.impl.ioc-macros/eval22522/fn/fn
ioc_macros.clj: 600 clojure.core.async.impl.ioc-macros/eval22585/fn/fn
ioc_macros.clj: 806 clojure.core.async.impl.ioc-macros/parse-to-state-machine/fn
ioc_macros.clj: 80 clojure.core.async.impl.ioc-macros/get-plan
ioc_macros.clj: 802 clojure.core.async.impl.ioc-macros/parse-to-state-machine
ioc_macros.clj: 1066 clojure.core.async.impl.ioc-macros/state-machine
async.clj: 384 clojure.core.async/go
RestFn.java: 442 clojure.lang.RestFn/invoke
Var.java: 388 clojure.lang.Var/invoke
AFn.java: 160 clojure.lang.AFn/applyToHelper
Var.java: 700 clojure.lang.Var/applyTo
Compiler.java: 6552 clojure.lang.Compiler/macroexpand1
Compiler.java: 6630 clojure.lang.Compiler/analyzeSeq
Compiler.java: 6445 clojure.lang.Compiler/analyze
Compiler.java: 6406 clojure.lang.Compiler/analyze
Compiler.java: 3719 clojure.lang.Compiler$InvokeExpr/parse
Compiler.java: 6646 clojure.lang.Compiler/analyzeSeq
Compiler.java: 6445 clojure.lang.Compiler/analyze
Compiler.java: 6406 clojure.lang.Compiler/analyze
Compiler.java: 5782 clojure.lang.Compiler$BodyExpr$Parser/parse
Compiler.java: 5217 clojure.lang.Compiler$FnMethod/parse
Compiler.java: 3846 clojure.lang.Compiler$FnExpr/parse
Compiler.java: 6642 clojure.lang.Compiler/analyzeSeq
Compiler.java: 6445 clojure.lang.Compiler/analyze
Compiler.java: 6700 clojure.lang.Compiler/eval
Compiler.java: 7130 clojure.lang.Compiler/load
REPL: 1 geschichte.p2p.hooks/eval32348
Compiler.java: 6703 clojure.lang.Compiler/eval
Compiler.java: 6666 clojure.lang.Compiler/eval
core.clj: 2927 clojure.core/eval
main.clj: 239 clojure.main/repl/read-eval-print/fn
main.clj: 239 clojure.main/repl/read-eval-print
main.clj: 257 clojure.main/repl/fn
main.clj: 257 clojure.main/repl
RestFn.java: 1523 clojure.lang.RestFn/invoke
interruptible_eval.clj: 67 clojure.tools.nrepl.middleware.interruptible-eval/evaluate/fn
AFn.java: 152 clojure.lang.AFn/applyToHelper
AFn.java: 144 clojure.lang.AFn/applyTo
core.clj: 624 clojure.core/apply
core.clj: 1862 clojure.core/with-bindings*
RestFn.java: 425 clojure.lang.RestFn/invoke
interruptible_eval.clj: 51 clojure.tools.nrepl.middleware.interruptible-eval/evaluate
interruptible_eval.clj: 183 clojure.tools.nrepl.middleware.interruptible-eval/interruptible-eval/fn/fn
interruptible_eval.clj: 152 clojure.tools.nrepl.middleware.interruptible-eval/run-next/fn
AFn.java: 22 clojure.lang.AFn/run
ThreadPoolExecutor.java: 1145 java.util.concurrent.ThreadPoolExecutor/runWorker
ThreadPoolExecutor.java: 615 java.util.concurrent.ThreadPoolExecutor$Worker/run
Thread.java: 745 java.lang.Thread/run

Possibly this is due to the function boundaries of go-blocks, but I think it would be nice if this would work similar to the binding for `a`. I have basically one big for-comprehension to map a highly nested seq and rewriting it will make the code much more verbose. Note that I know that I cannot execute channel ops inside the body of the for-comprehension due to the function boundaries in lazy seqs. This is no problem, but I cannot fix the for-comprehension bindings easily.
I played around with extending the -item-to-ssa multimethod, but some pointers of how to fix it properly would be nice. Then I could try to implement a patch.



 Comments   
Comment by Christian Weilbach [ 14/Jun/15 9:11 AM ]

This is actually a problem for many more cases, only a take operation in the first binding works, because the for-comprehension macro in core.clj wraps every following binding in a lazy-seq. The best fix to me seemed to be a custom version of the for-comprehension macro.

I have simplified the macro to remove lazyness and rewritten it with go-loops. This works fairly well, but it is eager and channel operations are sideeffects, so channels representing constants on which iteration happens repeatedly should be put in an immutable collection before (e.g. let binding around the comprehension).

https://github.com/ghubber/geschichte/blob/c57bacdcf8e3335d17dc370941384c559ae80373/src/cljx/geschichte/go_for.cljx#L25





[ASYNC-125] Closing a tap with a pending item blocks the mult input channel Created: 07/Jun/15  Updated: 07/Jun/15

Status: Open
Project: core.async
Component/s: None
Affects Version/s: None
Fix Version/s: None

Type: Defect Priority: Major
Reporter: Klaus Wuestefeld Assignee: Unassigned
Resolution: Unresolved Votes: 0
Labels: None
Environment:

org.clojure:clojure:1.7.0-alpha5
org.clojure:core.async:0.1.346.0-17112a-alpha


Approval: Triaged

 Description   

Closing a tap without a pending item is OK but closing a tap with a pending item blocks the mult input channel:

(require '[clojure.core.async :refer :all])
(def c (chan))
(def m (mult c))
(def t (chan))
(tap m t)
(>!! c :a)
(close! t)
(>!! c :b)  ; BLOCKS


 Comments   
Comment by Klaus Wuestefeld [ 07/Jun/15 6:22 PM ]

A more general case:

Doing this:

(go (println (>! c 42)))

and then closing c will cause the >! to block, instead of returning false.

If c is closed before that, >! will return false.

Is this race condition the intended behavior?





[ASYNC-126] Add public channel "closed?" predicate Created: 08/Jun/15  Updated: 19/Jul/15

Status: In Progress
Project: core.async
Component/s: None
Affects Version/s: None
Fix Version/s: None

Type: Enhancement Priority: Major
Reporter: Stuart Sierra Assignee: Stuart Sierra
Resolution: Unresolved Votes: 1
Labels: None

Attachments: Text File 0001-ASYNC-126-Add-public-channel-closed-predicate.patch     Text File 0002-ASYNC-126-Add-public-channel-closed-predicate.patch    
Patch: Code
Approval: Triaged

 Description   

Current patch file: 0002-ASYNC-126-Add-public-channel-closed-predicate.patch

Add a public function closed? in clojure.core.async to determine if a channel is closed without putting or taking any values.

This is a trivial wrapper around clojure.core.async.impl.protocols/closed?

There could still be a race between closed? and close! on another thread. This is just a check to avoid doing unnecessary work if a channel is already closed.

Example use case: a producer process which puts values on a channel can check if a channel is closed? before doing the work to produce the next value.



 Comments   
Comment by Alejandro [ 18/Jul/15 11:41 AM ]

The patch only adds 'closed?' to the ClojureScript version of core.async, is there a reason for not adding it to the Clojure version?

Comment by Stuart Sierra [ 19/Jul/15 12:07 PM ]

Previous patch was missing the Clojure(JVM) implementation. Fixed in new patch 0002-ASYNC-126-Add-public-channel-closed-predicate.patch.





[ASYNC-127] mult distribution behavior doesn't work as intended Created: 08/Jun/15  Updated: 08/Jun/15

Status: Open
Project: core.async
Component/s: None
Affects Version/s: None
Fix Version/s: None

Type: Defect Priority: Major
Reporter: Leon Grapenthin Assignee: Unassigned
Resolution: Unresolved Votes: 1
Labels: mult

Attachments: Text File async-127.patch    
Approval: Triaged

 Description   

Quote from docstring: "[...] each tap must accept before the next item is distributed."

(def ch (chan))

(def m (mult ch))

(def t-1 (chan))
(def t-2 (chan))
(def t-3 (chan))

(def t-1-takes (atom []))

(defn log [l] (partial swap! l conj))

(tap m t-1)
(tap m t-2)
(tap m t-3)

(close! t-3)

(take! t-1 (log t-1-takes))

(take! t-1 (log t-1-takes)) ;; this take shouldn't be happening before
                            ;; a take on t-2

(put! ch true)

(put! ch true)

@t-1-takes

;-> [true true] ;; but it does.

The reason is that the internal atom dctr is decreased twice when a tapped channel is already closed.



 Comments   
Comment by Leon Grapenthin [ 08/Jun/15 1:53 PM ]

Fixing this for clj/cljs





[ASYNC-128] or evaluation not stopped when exp nests take Created: 09/Jun/15  Updated: 09/Jun/15

Status: Open
Project: core.async
Component/s: None
Affects Version/s: None
Fix Version/s: None

Type: Defect Priority: Major
Reporter: Leon Grapenthin Assignee: Unassigned
Resolution: Unresolved Votes: 0
Labels: None
Environment:

CLJS



 Description   
(go (or true
        (= (<! (js/console.log "this should not happen"))
           :doesnt-matter)))





[ASYNC-129] Channels with transducer using reduced don't work as intended Created: 09/Jun/15  Updated: 11/Jun/15

Status: Open
Project: core.async
Component/s: None
Affects Version/s: None
Fix Version/s: None

Type: Defect Priority: Major
Reporter: Leon Grapenthin Assignee: Ghadi Shayban
Resolution: Unresolved Votes: 0
Labels: None
Environment:

CLJ 1.7.0-RC1, (CLJS not tested)



 Description   
(def ch (chan 1 (take 2)))

(put! ch 1)
(put! ch 2)
(put! ch 3)

(take! ch identity)

ClassCastException clojure.lang.PersistentVector cannot be cast to java.util.concurrent.locks.Lock clojure.core.async.impl.channels.ManyToManyChannel (channels.clj:55)

This does not happen if a take is made before the first put so that the buffer size fits. The problem doesn't seem to be related only to the buffer size though.

(def ch (chan 1 (take-while true?)))

(put! ch true)
(put! ch false)
(put! ch false)

(take! ch identity)

ClassCastException clojure.lang.PersistentVector cannot be cast to java.util.concurrent.locks.Lock clojure.core.async.impl.channels.ManyToManyChannel (channels.clj:55)






[ASYNC-132] Can't close over locals in #js in go Created: 09/Jul/15  Updated: 09/Jul/15

Status: Open
Project: core.async
Component/s: None
Affects Version/s: None
Fix Version/s: None

Type: Defect Priority: Major
Reporter: Leon Grapenthin Assignee: Unassigned
Resolution: Unresolved Votes: 0
Labels: None
Environment:

CLJS 3308



 Description   
(go
  (let [bar 42]
    #js{:foo bar}))

Use of undeclared Var my.core/bar






[ASYNC-137] Make (<! (timeout 0)) be closer to 0 Created: 03/Aug/15  Updated: 25/Aug/15

Status: Open
Project: core.async
Component/s: None
Affects Version/s: None
Fix Version/s: None

Type: Enhancement Priority: Major
Reporter: Mike Thompson Assignee: Unassigned
Resolution: Unresolved Votes: 4
Labels: None


 Description   

Currently `(<! (timeout 0))` will take about 4ms on a modern browser because it ends up as `(js/setTimeout f 0)`. On a modern browser, setTimeout won't come back in under 4ms, even if given a 0. And 4ms is too long if you are just wanting to hand back control to the browser for the minimum about of time.

See:
https://github.com/clojure/core.async/blob/d073896192fa55fab992eb4c9ea57b86ec5cf076/src/main/clojure/cljs/core/async/impl/timers.cljs#L161-L164

followed by:
https://github.com/clojure/core.async/blob/d073896192fa55fab992eb4c9ea57b86ec5cf076/src/main/clojure/cljs/core/async/impl/dispatch.cljs#L35-L36

I was imagining this tiny rewrite:

(defn queue-delay [f delay]
(if (= 0 delay)
(goog.async.nextTick f) ;; <---- new path for 0
(js/setTimeout f delay)) ;; <---- existing path



 Comments   
Comment by Patrick O'Brien [ 03/Aug/15 10:31 PM ]

I think the use of goog.Timer.callOnce should work for any amount of time and would still solve the problem posed by the use of js/setTimeout.

Comment by Mike Thompson [ 04/Aug/15 2:38 AM ]

From what I can see, goog.Timer.callOnce use setTimeout. But that's specifically what we're trying to avoid in the case of a 0ms duration. So I don't see callOnce as a solution.

Comment by Patrick O'Brien [ 04/Aug/15 7:27 AM ]

I think you are right, Mike. Hard as it is to believe, since obviously Google knows about the problem. But now that I'm rereading the goog.Timer code I can't see that they've fixed it here. Looks like goog.async.nextTick is the only solution.

Comment by Patrick O'Brien [ 04/Aug/15 7:33 AM ]

I just wanted to mention that this issue is for the ClojureScript version of core.async, and that it appears to be fairly idiomatic to use (<! (timeout 0)) as a way to temporarily pass control back to the js event processing. So I think it is very important to make this efficient.

Comment by Ghadi Shayban [ 06/Aug/15 1:56 PM ]

In reading this "bug", it seems that the actual problem is having more varied goblock dispatch in CLJS. It could/should be improved, but making (<! (timeout 0)) – non-sensical code – behave differently is not addressing the problem in a meaningful way.

We should eliminate the need for this hack at the root, rather than make undefined behavior more specific. ASYNC-131 and ASYNC-43 are related and probably better starting points.

Comment by Mike Thompson [ 06/Aug/15 10:32 PM ]

@Ghadi

  1. I can't see how issue 131 is closely enough related to act as a "better starting point". It has a different thrust, represents different needs, and will likely have a different solution (as evidenced by the current suggestions).
  1. It almost looks to me as if Issue 43 could be closed with the switch to Google Closure dispatch noted by David Nolen. BTW, you'll note that my suggestion is to indeed use goog.async.nextTick (so I'm adhering to the use Google Closure approach).

Also, I would disagree with your assertion that (<! (timeout 0)) is "non-sensical code". In a single threaded environment, like the browser, there is sometimes a need to explicitly "hand back control" to the browser from within go loops (if that go loop is, for a time, hogging the CPU). At the moment (<! (timeout 0)) is the only way I know to do that. It could reworked to be (<! (yield-cpu)) in order to make the intent clearer, and that might be nice, but what I care about is having a way of yielding which doesn't always cost 4ms. I believe this need is genuine, and very "sensical"

Comment by Mike Thompson [ 06/Aug/15 11:06 PM ]

@Ghadi
Just to be clear ... you refer to this as a "bug" in double quotes. Almost as if it isn't a bug.

But for me this is a sufficiently real "bug" that I'll have to either work with a fork OR move away from using core.async entirely.

I need (<! (timeout 0)) to mean 0, or as close to it as possible. I need it to work as advertised. The current implementation does not deliver on 0 (or as close to it as possible), and instead delivers the same as (<! (timeout 4)) and 4ms is FOREVER compared to 0ms (or close to it). At 4ms each time around, a goloop can only iterate 250 times a second.

Anyone who has a goloop listening to the output of a bursty websocket will have this problem. They need to process like crazy (no 4ms delays) BUT also hand back control so the browser can do what it needs to, every now and again. All of this get exacerbated when a browser page loses "focus", goes into the background, and the js gets throttled, animation frames slow down, etc.

Comment by Patrick O'Brien [ 15/Aug/15 9:59 PM ]

In case it helps anyone who might be reading this, I have begun using the following as a temporary solution:

(defn yield []
  (let [ch (chan)]
    (goog.async.nextTick #(close! ch))
    ch))

(go
  ; Some code...
  (<! (yield))  ;; Instead of (<! (timeout 0))
  ...)
Comment by Mike Thompson [ 25/Aug/15 5:04 AM ]


That's nice! Certainly provides a placeholder solution.





[ASYNC-138] Go blocks leak memory Created: 10/Aug/15  Updated: 11/Aug/15

Status: Open
Project: core.async
Component/s: None
Affects Version/s: None
Fix Version/s: None

Type: Defect Priority: Major
Reporter: Brian Lubeski Assignee: Unassigned
Resolution: Unresolved Votes: 0
Labels: memory
Environment:

clojure 1.7.0
core.async 0.1.346.0-17112a-alpha
Java HotSpot(TM) Client VM 1.8.0_31-b13


Approval: Triaged

 Description   

The following example, after running for a few minutes, generates an OutOfMemoryError.

(let [c (chan)]
  (go (while (<! c)))
  (let [vs (range)]
    (go
      (doseq [v vs]
        (>! c v)))))

By contrast, the following example will run indefinitely without causing an OutOfMemoryError.

(let [c (chan)]
  (go (while (<! c)))
  (go
    (let [vs (range)] 
      (doseq [v vs]
        (>! c v)))))

The only significant difference I see between the two examples is that the (range) is created outside the go block in the first example but is created inside the go block in the second example. It appears that the go block in the first example is referencing vs in such a way as to prevent if from being garbage-collected.

This behavior might also be the cause of ASYNC-32.






[ASYNC-141] Convert project to use Reader Conditionals Created: 18/Aug/15  Updated: 18/Aug/15

Status: Open
Project: core.async
Component/s: None
Affects Version/s: None
Fix Version/s: None

Type: Enhancement Priority: Major
Reporter: Daniel Compton Assignee: Unassigned
Resolution: Unresolved Votes: 0
Labels: None


 Description   

There is a fair amount of duplicated code between the Clojure and ClojureScript versions of core.async. Converting the library to use Reader Conditionals would get rid of a lot of code volume and reduce the chances of changes in one library not being correctly implemented in the other. It may also make maintenance and source reading easier by making the platform specific code stick out more, where the 'essence' of core.async stays in pure Clojure. This could also be an opportunity to bring cljs.core.async.macros into the main async namespace using reader conditionals.

Despite these benefits there are a few downsides to this:

  • Currently the two ports are able to exist fairly separate from each other and the domain experts in each can work on their own part without affecting the other. If this was a concern, more tests may be able to be added to ensure there aren't breakages
  • Combining the two ports may make it harder to read either version in the same source file.

As this is a fairly large change, I wanted to get feedback on this proposal before going any further.



 Comments   
Comment by Alex Miller [ 18/Aug/15 8:34 AM ]

For the time being, we are still supporting Clojure 1.6 with core.async so this is not an option. At some point, that will change, not sure when.

Another aspect here is that reader conditionals are designed for code that is very similar across platforms but the code for clj and cljs is substantially different in many cases, so it is not an obvious candidate. Rather, I would expect to see alternate versions of the same namespace in many cases.

Changing namespaces would also of course break existing users, so we would need to plan ahead for this change.

While I think there is an endpoint with more consolidated code base (and a better build system, which I have been working on), I do not think it's wise for anyone outside the core team to work on it.





[ASYNC-142] Rename ClojureScript namespace from cljs.core.async to clojure.core.async Created: 18/Aug/15  Updated: 18/Aug/15

Status: Open
Project: core.async
Component/s: None
Affects Version/s: None
Fix Version/s: None

Type: Enhancement Priority: Major
Reporter: Daniel Compton Assignee: Unassigned
Resolution: Unresolved Votes: 0
Labels: None


 Description   

I'm not sure what the original reasoning for using separate namespaces for the Clojure and ClojureScript versions of core.async were, but now with Reader Conditionals people are able to write cross platform code that targets both at the same time. For that reason, it would be nice to only need to require clojure.core.async so that the require didn't need to be split into two platform specific requires for the same library.

The biggest downside to this is that it would break backwards compatibility. However core.async is still in alpha, and updating code to reference the new namespace should be simple and able to be done quickly. It may be possible to leave a shim namespace in place for some time to allow people to migrate their code.

Thoughts?



 Comments   
Comment by Alex Miller [ 18/Aug/15 8:36 AM ]

I think we should do this, but it will likely be in tandem with a bigger set of changes (build, moving away from Clojure 1.6, etc).





[ASYNC-143] Assert that fixed buffers must have size > 0 Created: 18/Aug/15  Updated: 18/Aug/15

Status: Open
Project: core.async
Component/s: None
Affects Version/s: None
Fix Version/s: None

Type: Enhancement Priority: Major
Reporter: Alex Miller Assignee: Unassigned
Resolution: Unresolved Votes: 0
Labels: None

Attachments: Text File async-143.patch    
Patch: Code
Approval: Triaged

 Description   

(chan 0) is invalid (and is not the same thing as an unbuffered (chan)).

Approach: Add an assertion to throw on (chan 0). I found based on some failing tests that one place this can come up is in to-chan, which creates a buffer sized to the collection (0). The patch also changes to-chan to return an empty, closed chan if the collection is of size 0.

Patch: async-143.patch






[ASYNC-39] Processes spawned by mix never terminate Created: 21/Nov/13  Updated: 05/May/14

Status: Open
Project: core.async
Component/s: None
Affects Version/s: None
Fix Version/s: None

Type: Defect Priority: Minor
Reporter: Leon Grapenthin Assignee: Unassigned
Resolution: Unresolved Votes: 0
Labels: chan, mix
Environment:

"0.1.256.0-1bf8cf-alpha"



 Description   

Once a mix has been created, the go-loop inside mix will always recur. Obviously, input-channels can be unmixed and the output-channel could be closed, but the process would still never terminate.

Probably mixes should support something like (stop) to to make the mix-associated process garbage-collectable. Operations on a stopped mix should probably throw.



 Comments   
Comment by Ghadi Shayban [ 22/Apr/14 10:44 AM ]

On 0.1.278 the mix process terminates when its output channel closes [1].

[1] https://github.com/clojure/core.async/blob/master/src/main/clojure/clojure/core/async.clj#L778

Comment by Leon Grapenthin [ 24/Apr/14 3:16 AM ]

Yes, it has been fixed using the new return behavior of put!. (I can't find an option to close this issue)





[ASYNC-40] do not transform forms that have :no-transform metadata attached Created: 29/Nov/13  Updated: 05/May/14

Status: Open
Project: core.async
Component/s: None
Affects Version/s: None
Fix Version/s: None

Type: Enhancement Priority: Minor
Reporter: David Nolen Assignee: Unassigned
Resolution: Unresolved Votes: 2
Labels: None


 Description   

Interaction between core.match clauses and the core.async transform is undesirable. core.async should respect some hook so that some forms are left alone. For example:

(match [x y]
  [1 2] ...
  [3 4] ...)

All code generated for [1 2] and [3 4] would have this metadata attached to it.



 Comments   
Comment by Rich Hickey [ 30/Nov/13 9:25 AM ]

Can you be more specific? This seems like a bad idea, and I wonder why it's desired.

Comment by David Nolen [ 30/Nov/13 2:12 PM ]

To show how bad the interaction is between core.async and a library like core.match that also generates a lot of code consider the following:

(defn foo [e]
  (match [e]
    [{:type :mouse :client-x x :client-y y}] [x y]
    [{:type :mouse :page-x x :page-y y}] [x y]
    [{:type :touch :page-x x :page-y y}] [x y]
    [{:type :key :char-code c}] c))

Without core.async, core.match produces a reasonable amount of code for this typical use of core.match to efficiently match maps - ~230 lines of pretty printed JavaScript.

But if the user wraps this typical expression in a go block:

(defn foo [in]
  (go (while true
        (let [[e c] (<! in)]
          (match [e]
            [{:type :mouse :client-x x :client-y y}] [x y]
            [{:type :mouse :page-x x :page-y y}] [x y]
            [{:type :touch :page-x x :page-y y}] [x y]
            [{:type :key :char-code c}] c)))))

It generates nearly 4200 lines of pretty-printed JavaScript. I fail to see the value of core.async transforming the optimized conditionals generated by core.match, it just generates 18X more code and the extra generated code is obviously useless - a user is matching a value, they cannot put arbitrary computations in the patterns.

https://gist.github.com/swannodette/7723758





[ASYNC-55] Notification of items dropped from sliding/dropping buffers Created: 12/Feb/14  Updated: 05/May/14

Status: Open
Project: core.async
Component/s: None
Affects Version/s: None
Fix Version/s: None

Type: Enhancement Priority: Minor
Reporter: Chris Perkins Assignee: Unassigned
Resolution: Unresolved Votes: 0
Labels: None

Attachments: File drop-notification.diff    
Patch: Code and Test

 Description   

I would like to know when items are dropped from sliding or dropping buffers, so that I can do things like logging and keeping metrics.

The attached patch has one possible mechanism for doing this, by adding an optional second argument to sliding-buffer and dropping-buffer - a one-argument function that will be called with dropped items.



 Comments   
Comment by Ghadi Shayban [ 22/Apr/14 10:41 AM ]

Hi Chris, I think this use-case can be handled the combination of a go process + collection directly without modification to the existing buffer impls. I'd be concerned about making buffers pay for a non-essential field.

I like using the library primitives to make sure they are minimally complete.

(If you really really want to make your own buffer, no one will stop you =) It's still an impl protocol, but it's relatively stable. Just beware that buffer operations run within the context of a channel lock, so make sure to dispatch the overflow function to the async threadpool using dispatch/run).





[ASYNC-61] Exceptions thrown inside go/thread blocks propagate up and out of ThreadPoolExcecutor Created: 24/Mar/14  Updated: 06/Aug/14

Status: Open
Project: core.async
Component/s: None
Affects Version/s: None
Fix Version/s: None

Type: Enhancement Priority: Minor
Reporter: Howard Lewis Ship Assignee: Unassigned
Resolution: Unresolved Votes: 2
Labels: exceptions
Environment:

core.async 0.1.267.0-0d7780-alpha on JDK 1.7



 Description   

If a go or thread blocks throws an exception, there is nothing in Clojure to catch and handle (or report) the exception. Instead, it propagates up to the ThreadExcecutor which invokes its NO-OP afterExecute() method and is re-thrown, ultimately being displayed on the System.err:

Exception in thread "async-dispatch-32" java.lang.IllegalStateException: Fall down, go boom!
at flashiz.resources.orders$index.invoke(orders.clj:26)
at clojure.lang.Var.invoke(Var.java:411)
at clojure.lang.AFn.applyToHelper(AFn.java:159)
at clojure.lang.Var.applyTo(Var.java:532)
at clojure.core$apply.invoke(core.clj:617)
at io.aviso.rook$rook_dispatcher.invoke(rook.clj:225)
at flashiz.async$wrap_sync_handler$fn__9005.invoke(async.clj:34)
at flashiz.resources$authorize_async_rook_middleware$fn_9356$fn9402$state_machine3245auto__9403$fn_9405.invoke(resources.clj:21)
at flashiz.resources$authorize_async_rook_middleware$fn_9356$fn9402$state_machine3245auto___9403.invoke(resources.clj:21)
at clojure.core.async.impl.ioc_macros$run_state_machine.invoke(ioc_macros.clj:945)
at clojure.core.async.impl.ioc_macros$run_state_machine_wrapped.invoke(ioc_macros.clj:949)
at clojure.core.async.impl.ioc_macros$take_BANG_$fn__3261.invoke(ioc_macros.clj:958)
at clojure.core.async.impl.channels.ManyToManyChannel$fn__2256.invoke(channels.clj:80)
at clojure.lang.AFn.run(AFn.java:24)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:724)

It would be nice if execution of the go/thread block was wrapped in an exception handler that delegated to a default function to report the exception. My goal is to be able to alter that function to report it more nicely and/or write it to a persistent log file.



 Comments   
Comment by Alex Miller [ 06/Aug/14 10:30 AM ]

Patches on ASYNC-76 have been applied and exceptions will now flow up to the top of the thread, where they can be caught by the standard Thread uncaught exception handler mechanism or ultimately by the default uncaught exception handler, which can be set for the application. There may still be further changes to support exception handling in thread/go.





[ASYNC-63] Variable called 'new' in vector in go block in CLJS causes "Object has no method 'call'" error Created: 07/Apr/14  Updated: 05/May/14

Status: Open
Project: core.async
Component/s: None
Affects Version/s: None
Fix Version/s: None

Type: Defect Priority: Minor
Reporter: James Henderson Assignee: Unassigned
Resolution: Unresolved Votes: 0
Labels: bug, cljs
Environment:

ClojureScript 0.0-2202, core.async 0.1.267.0-0d7780-alpha, openjdk 7u51


Attachments: File compiled-cljs.js     File test_cases.clj    

 Description   

I seem to be having trouble with the combination of:

  • a variable named 'new'
  • wrapped in a vector
  • after a <! call in a go block.

Removing any of these conditions seems to work fine.

I've attached a few minimal examples in test_cases.clj and the JavaScript that the first error compiles down to in compiled-cljs.js - the errors occur on line 4 (I think inst_39771 is 21 in this case)

An obvious workaround is not to name the variable 'new'

Please let me know if you need anything more from me, and please go easy on me if this is in the wrong place/badly labelled etc - it's my first Clojure JIRA bug report

Thanks,

James



 Comments   
Comment by James Henderson [ 07/Apr/14 4:07 PM ]

Sorry, just spotted there's a newer release of core.async - I've reproduced this with 0.1.278.0-76b25b-alpha as well.

James





[ASYNC-65] Change chan returned from pipe to internal go block out channel Created: 09/May/14  Updated: 09/May/14

Status: Open
Project: core.async
Component/s: None
Affects Version/s: None
Fix Version/s: None

Type: Enhancement Priority: Minor
Reporter: Ghadi Shayban Assignee: Unassigned
Resolution: Unresolved Votes: 0
Labels: None

Approval: Vetted

 Description   

Can (pipe src dest close?) return the chan of its internal go block instead of 'dest'? This would provide a useful 'done' channel.






[ASYNC-66] Add drain! to consume and discard a channel Created: 09/May/14  Updated: 09/May/14

Status: Open
Project: core.async
Component/s: None
Affects Version/s: None
Fix Version/s: None

Type: Enhancement Priority: Minor
Reporter: Ghadi Shayban Assignee: Unassigned
Resolution: Unresolved Votes: 0
Labels: None

Approval: Vetted

 Description   

Do we want this for consumer walkaway scenarios or something baked into the channel impl?

(defn drain!
 "Fully consumes and discards a channel. Returns a channel that
  closes when ch is fully drained"
 [ch]
 (go-loop []
   (if (some? (<! ch))
     (recur))))





[ASYNC-67] Can we get a generic sink operation? Created: 09/May/14  Updated: 07/Oct/14

Status: Open
Project: core.async
Component/s: None
Affects Version/s: None
Fix Version/s: None

Type: Enhancement Priority: Minor
Reporter: Ghadi Shayban Assignee: Unassigned
Resolution: Unresolved Votes: 2
Labels: None

Approval: Triaged

 Description   

Can we get a generic sink operation?

(defn sink
 "Fully consumes and discards a channel, applying f to each val. Returns a channel that
  closes when ch is fully drained"
 [f ch]
 (go-loop []
   (when-some [v (<! ch))]
     (f v)
     (recur))))





[ASYNC-69] How to better communicate "mix" lifecycle wrt coordination Created: 09/May/14  Updated: 09/May/14

Status: Open
Project: core.async
Component/s: None
Affects Version/s: None
Fix Version/s: None

Type: Enhancement Priority: Minor
Reporter: Timothy Baldridge Assignee: Unassigned
Resolution: Unresolved Votes: 0
Labels: None

Approval: Vetted

 Description   

Confusion regarding 'mix' lifecycle. Users have a false expectation that a mix's output should magically close when its source channels close, not realizing its a dynamic coordination process. Actually ensuring correctness when using mult/mix/pub isn't so straightforward, people need to control consistency from the producer end, and not ad hoc from the consumer side (e.g. late taps of a mult).






[ASYNC-77] StackOverflowError in go macro with cemerick.cljs.test (CLJS) Created: 04/Jul/14  Updated: 04/Jul/14

Status: Open
Project: core.async
Component/s: None
Affects Version/s: None
Fix Version/s: None

Type: Defect Priority: Minor
Reporter: Daniel Skarda Assignee: Unassigned
Resolution: Unresolved Votes: 0
Labels: None
Environment:

[org.clojure/clojure "1.6.0"]
[org.clojure/core.async "0.1.303.0-886421-alpha"]
[org.clojure/clojurescript "0.0-2261"]
[com.cemerick/clojurescript.test "0.3.1"]



 Description   

I got an StackOverflowError during compilation of CLJS tests using clojurescript.test from Chas Emerick.

I narrowed the problem to following code:

(ns async.overflow.cljs
  (:require-macros [cljs.core.async.macros :refer [go]]
                   [cemerick.cljs.test :refer [deftest is]])
  (:require [cemerick.cljs.test]))

(deftest foobar
  (go
    (is (= 1 1))
    (is (= 1 1))
    (is (= 1 1))
    (is (= 1 1))
    (is (= 1 1))
    (is (= 1 1))
    (is (= 1 1))
    (is (= 1 1))))

If you do not get the exception, try to increase number of repetitions of 'is' call.

I was not able to replicate the issue in CLJ. cemerick.cljs.test/is macro is probably far more complex than original CLJ version in clojure.test/is.






[ASYNC-91] 'and' does not short circuit within go block in clojurescript Created: 14/Sep/14  Updated: 25/Sep/14

Status: Open
Project: core.async
Component/s: None
Affects Version/s: None
Fix Version/s: None

Type: Defect Priority: Minor
Reporter: Jan Krueger Assignee: Unassigned
Resolution: Unresolved Votes: 1
Labels: None
Environment:

core.async 0.1.338.0-5c5012-alpha



 Description   

I have the following piece of ClojureScript code within a go block:

(cond
(and (vector? x) (= (first x) :some-key)) ...)

This generates the following piece of JavaScript code for the cond case:

if (40 === f) { return d = b[9], f = cljs.core.vector_QMARK_.call(null, d), d = cljs.core.first.call(null, d), d = cljs.core._EQ_.call(null, d, new cljs.core.Keyword(null, "some-key", "some-key", -978969032)), cljs.core.truth_(f && d) ? b[1] = 42 : b[1] = 43, new cljs.core.Keyword(null, "recur", "recur", -437573268); }

This looks to me like both and arguments would actually get evaluated. As a result my code crashes whenever it hits this cond case and 'x' is not seqable.



 Comments   
Comment by Francis Avila [ 25/Sep/14 2:52 AM ]

CLJS-864 is likely the same issue. (It also has a small test project to reproduce.)





[ASYNC-96] FixedBuffer's full checking in cljs Created: 13/Oct/14  Updated: 13/Oct/14

Status: Open
Project: core.async
Component/s: None
Affects Version/s: None
Fix Version/s: None

Type: Defect Priority: Minor
Reporter: Nguyễn Tuấn Anh Assignee: Unassigned
Resolution: Unresolved Votes: 0
Labels: None

Attachments: Text File cljs-fixed-buffer-overflow.patch    
Patch: Code and Test

 Description   

In the cljs implementation of fixed buffer, "full?" uses "=" instead of ">=". This makes it incorrect after an overflow.






[ASYNC-98] Less hostile message for #'go stopping at (fn [] ) boundaries Created: 15/Oct/14  Updated: 15/Oct/14

Status: Open
Project: core.async
Component/s: None
Affects Version/s: None
Fix Version/s: None

Type: Enhancement Priority: Minor
Reporter: Ghadi Shayban Assignee: Unassigned
Resolution: Unresolved Votes: 0
Labels: errormsgs


 Description   

ASYNC-93 has an example of a not nice message when #'go attempts to shred through a (fn []) form. Maybe we can improve this, but then again creating a (fn []) in a block should be permitted, though the inner contents of it will not be transformed.






[ASYNC-107] Improved docstring for alt! Created: 10/Nov/14  Updated: 10/Nov/14

Status: Open
Project: core.async
Component/s: None
Affects Version/s: None
Fix Version/s: None

Type: Enhancement Priority: Minor
Reporter: Howard Lewis Ship Assignee: Unassigned
Resolution: Unresolved Votes: 2
Labels: docstring

Attachments: Text File core-async-alt-doc-1.patch    
Patch: Code

 Description   

I had a little remembering exactly how alt! is used, and put together some better documentation that more clearly identifies the differences between taking from and putting to channels.






[ASYNC-108] cljs to-chan hangs on infinite lazy sequences Created: 20/Nov/14  Updated: 20/Nov/14

Status: Open
Project: core.async
Component/s: None
Affects Version/s: None
Fix Version/s: None

Type: Defect Priority: Minor
Reporter: Felix Andrews Assignee: Unassigned
Resolution: Unresolved Votes: 0
Labels: cljs
Environment:

[org.clojure/core.async "0.1.346.0-17112a-alpha"]


Patch: Code

 Description   

e.g. this hangs the browser:

(go-loop [ch (to-chan (range))]
(println (<! ch))
(<! (timeout 1000))
(recur ch))

The problem is in
https://github.com/clojure/core.async/blob/53bf7866f195e6ba247ff7122b99784e66e9f1bb/src/main/clojure/cljs/core/async.cljs#L362
where the order of arguments to bounded-count is wrong.






[ASYNC-111] maps are represented different inside and outside of go blocks Created: 27/Dec/14  Updated: 27/Dec/14

Status: Open
Project: core.async
Component/s: None
Affects Version/s: None
Fix Version/s: None

Type: Defect Priority: Minor
Reporter: Sven Richter Assignee: Unassigned
Resolution: Unresolved Votes: 0
Labels: None
Environment:

clojure 1.6.0, clojurescript "0.0-2371", W7, Oracle JDK 8



 Description   

When initializing a state atom with a set "conjuring" a map behaves differently depending of the scope where it happens:

(def state (atom {:transformations []}))
(reset! state {:transformations #{:foo "bar"}})

; throws: Uncaught Error: compare on non-nil objects of different types
(...(go ;some async stuff
(swap! state conj {:name "baz"}))...)

; works
(...(go ;some async stuff)
(swap! state conj {:name "baz"}))

I am not sure if this is a defect at all as maps are not comparable as Tom in the thread stated. However, the fact that it works in one case and does not in the other is weird.
You can find the complete thread here: https://groups.google.com/forum/#!topic/clojurescript/lmvltpa3k8s






[ASYNC-113] alt!! should not evaluate the :default expr if a non-default clause is selected Created: 13/Jan/15  Updated: 27/Jul/15

Status: Open
Project: core.async
Component/s: None
Affects Version/s: None
Fix Version/s: None

Type: Defect Priority: Minor
Reporter: Caleb Spare Assignee: Unassigned
Resolution: Unresolved Votes: 0
Labels: None
Environment:

Clojure 1.5.0, core.async 0.1.346.0-17112a-alpha


Approval: Triaged

 Description   

alt!! will always evaluate the expr given in the :default clause. For instance, consider this code:

(require '[clojure.core.async :as async])
(let [c1 (async/chan 1)
      c2 (async/chan 1)]
  (async/alt!!
    [[c1 "a"]] (println "first")
    [[c2 "a"]] (println "second")
    :default (println "default")))

It will print either "first" or "second", but also (always) "default". Reading the documentation, technically it doesn't say whether the non-selected clauses are evaluated (only that the value of the selected expr is returned) but it's certainly not the behavior I would expect.

I haven't checked whether alt! does the same thing as well.



 Comments   
Comment by Daniel Compton [ 27/Jul/15 7:37 PM ]

It looks to me like :default is expected to be a value, not a function, and println is being run as part of evaluating the alt!! expression, not as the return value.





[ASYNC-114] Allow a second arity on channel ex-handler which takes val as well as throwable Created: 27/Jan/15  Updated: 19/Feb/15

Status: Open
Project: core.async
Component/s: None
Affects Version/s: None
Fix Version/s: None

Type: Enhancement Priority: Minor
Reporter: Derek Troy-West Assignee: Unassigned
Resolution: Unresolved Votes: 2
Labels: None

Attachments: Text File 0001-Pass-the-val-along-with-the-exception-to-the-channel.patch    

 Description   

When a transducer is applied to a channel and an exception occurs during transformation the ex-handler will be called with
the Throwable as an argument.

Can we also send the val which caused the transformation exception to the ex-handler? When dealing with transducer errors it might be useful to have the full picture.

If that's agreeable, and the solution is a second-arity on ex-handler that takes val and throwable, and the call applied is:

clojure.core.async.impl.channels becomes:

(defn chan
...
(try
(add! buf val)
(catch Throwable t
(handle buf exh t val)))))))))

I would be happy to supply a patch.

Apologies if I have missed something obvious.
Derek



 Comments   
Comment by Nivedita Priyadarshini [ 19/Feb/15 5:56 AM ]

Found the same need when using core.async/pipeline in my code. Wrote a patch of sorts (demonstrative) which allows you to have val in the exh for clj. Happy to write a full patch if this looks okay.





[ASYNC-118] (ClojureScript) A let-binding named 'arguments' not bound in go block on nodejs Created: 14/Mar/15  Updated: 14/Mar/15

Status: Open
Project: core.async
Component/s: None
Affects Version/s: None
Fix Version/s: None

Type: Defect Priority: Minor
Reporter: Sean Doig Assignee: Unassigned
Resolution: Unresolved Votes: 0
Labels: binding, cljs, nodejs
Environment:

OS X 10.10, node 0.10.35, Clojure 1.6.0, ClojureScript 0.0-3030, core.async 0.1.346.0-17112a-alpha



 Description   

Discovered this while using tools.cli 0.3.1 where it's handy to destructure one of their returned maps which has a key :arguments, but you can reproduce it like so:

(let [arguments [:a :b]]
    (println "Args Sync:" arguments)
    (go (println "Args Async:" arguments)))

which prints

Args Sync: [:a :b]
Args Async: #js {:0 #js [#<function (state_12638){
switch(arguments.length){
case 0:
return dls$rip$core$go_BANG__$_state_machine__6252__auto____0.call(this);
case 1:
return dls$rip$core$go_BANG__$_state_machine__6252__auto____1.call(this,state_12638);
}
throw(new Error('Invalid arity: ' + arguments.length));
}> 1 nil nil nil nil #<[object Object]>]}

I haven't tested this in the browser yet, so it might not be limited to node.



 Comments   
Comment by Sean Doig [ 14/Mar/15 12:34 PM ]

Just tried, can't reproduce in Chrome.





[ASYNC-121] compilation warning when calling 'satisfies?' inside a go block Created: 21/Apr/15  Updated: 21/Apr/15

Status: Open
Project: core.async
Component/s: None
Affects Version/s: None
Fix Version/s: None

Type: Defect Priority: Minor
Reporter: Yehonathan Sharvit Assignee: Unassigned
Resolution: Unresolved Votes: 0
Labels: compiler
Environment:

[org.clojure/core.async "0.1.346.0-17112a-alpha"]
[org.clojure/clojurescript "0.0-3196"]



 Description   

This piece of code is causing the following compilation warning:

(defprotocol a)
(defrecord b [])
(go (satisfies? a (b.)))

WARNING: Use of undeclared Var cljs-explore.me/bit_4884auto_ at line 12 src/cljs_explore/me.cljs






[ASYNC-130] Improve put! docstring to mention no more than 1024 pending puts are allowed Created: 21/Jun/15  Updated: 21/Jun/15

Status: Open
Project: core.async
Component/s: None
Affects Version/s: None
Fix Version/s: None

Type: Enhancement Priority: Minor
Reporter: Daniel Compton Assignee: Unassigned
Resolution: Unresolved Votes: 0
Labels: docstring


 Description   

The put! docstring says:

> "Asynchronously puts a val into port, calling fn0 (if supplied) when
complete. nil values are not allowed. Will throw if closed. If
on-caller? (default true) is true, and the put is immediately
accepted, will call fn0 on calling thread. Returns nil."

It would be good if it also mentioned that it can throw if more than 1024 pending puts are already on the channel. It could be something like:

> "Asynchronously puts a val into port, calling fn0 (if supplied) when
complete. nil values are not allowed. Will throw if closed or if there are more than 1024 pending puts. If
on-caller? (default true) is true, and the put is immediately
accepted, will call fn0 on calling thread. Returns nil."






[ASYNC-131] go! or "go-now" for CLJS Created: 29/Jun/15  Updated: 06/Jul/15

Status: Open
Project: core.async
Component/s: None
Affects Version/s: None
Fix Version/s: None

Type: Enhancement Priority: Minor
Reporter: Thomas Heller Assignee: Unassigned
Resolution: Unresolved Votes: 0
Labels: None


 Description   

I'd like to see a variant of the "go" macro that executes immediately until the first parkable operation instead of dispatching (effectively calling "setTimeout" twice). This is primarily for CLJS since CLJ can get away with blocking operations but it still might be useful there as well.

My Use-Case:

I currently use core.async to orchestrate CSS animations together and while it works perfectly fine with "go" it does require careful attention to detail on the order of operations since "go" will yield control back to the browser which may decide to reflow/repaint the screen although the animation wasn't properly started yet.

Example:

(defn show-toast [message]
  (let [toast (create-toast-dom message)]
    (dom/append js/document.body toast)
    ;; X - PROBLEM: REPAINT HAPPENS NOW
    (go (<! (anim/slide-in-from-bottom animation-time toast))
        (<! (timeout 3000))
        (<! (anim/fade-out animation-time toast))
        (dom/remove toast)
        )))

This example tries to create a "Toast" [1] DOM node which is animated in from the bottom and removed after 3 seconds. A real implementation would require handling clicks and so forth but the example is easy enough to describe the problem. The (anim/...) functions set the appropriate CSS and return a (async/timeout time) channel which lets us wait for the "completion".

The problem at X is that the go is dispatched and control is given back to the browser. Therefore the "setup" part of the animation is delayed and the toast dom node is rendered by the browser as is. Usually this means that the browser will do a repaint. This will have a "flicker" effect as the node appears and then disappears again and animates (depending on the default CSS).

The simple solution is to move the dom/append call into the go block, which is easy enough in this example but hard for more complex. Another solution is to start the animation outside the go:

(defn show-toast [message]
  (let [toast (create-toast-dom message)]
    (dom/append js/document.body toast)
    (let [slide-in (anim/slide-in-from-bottom animation-time toast)]
      ;; REPAINT, but slide-in animation already started so no "flicker"
      (go (<! slide-in)
          (<! (timeout 3000))
          (<! (anim/fade-out animation-time toast))
          (dom/remove toast)
          ))))

Again, simple enough here, hard for more complex things.

What I would prefer to see is a "go!" variant which instead of dispatching the go block just starts executing it.

(defn show-toast [message]
  (let [toast (create-toast-dom message)]
    (dom/append js/document.body toast)
    (go! (<! (anim/slide-in-from-bottom animation-time toast)) ;; REPAINT ON <!
         (<! (timeout 3000))
         (<! (anim/fade-out animation-time toast))
         (dom/remove toast)
         )))

The subtle difference is that the browser does not get a chance to do anything before we are ready and would wait anyway. After fine-tuning a few of those animation effects myself it always requires moving stuff out of or into go blocks as it stands now.

I'm aware that this is a rather specific use-case and core.async might not even be the best way to do this at all but so far I had great success with it and the code is very easy to reason about.

I have not looked too much into the internals of core.async but go! should be a pretty simple addition. If this change is accepted I would start looking into things and create a patch, just wanted to make sure there were no objections before doing so.

Hope my intent is clear. CSS animations can be a bit counter-intuitive which might not make it obvious where the problem actually lies.

[1] http://www.google.com/design/spec/components/snackbars-toasts.html#snackbars-toasts-specs



 Comments   
Comment by Thomas Heller [ 06/Jul/15 6:06 AM ]

I looked into the issue to find out what needed to be done on the core.async side to facilitate this change, turns out nothing.

(defmacro go!
  "just like go, just executes immediately until the first put/take"
  [& body]
  `(let [c# (cljs.core.async/chan 1)
         f# ~(ioc/state-machine body 1 &env ioc/async-custom-terminators)
         state# (-> (f#)
                    (ioc/aset-all! cljs.core.async.impl.ioc-helpers/USER-START-IDX c#))]
     (cljs.core.async.impl.ioc-helpers/run-state-machine state#)
     c#))

Works perfectly fine so far and can live perfectly well inside a library, I'm beginning to think that this might be a better "default" for CLJS since there are no "threads" that could start running the "go".

With the normal "go" the flow of execution is:

code before go
setTimeout/nextTick
code inside go until first take/put
setTimeout/nextTick

With go! it is:

code before go!
code inside go! until first take/put
setTimeout/nextTick

Given the non-blocking nature of Javascript the extra setTimeout does not hurt it doesn't provide much benefit either.

Since no changes to core.async are required I can simply keep this macro in my library. My use-case is rather specific so feel free to close the issue if there is no interest to make this "common".

Comment by Leon Grapenthin [ 06/Jul/15 5:46 PM ]

As a default this would violate the async execution guarantee of go which would be a breaking change. Some observations from my user perspective: For readability I'd prefer both examples you proposed for usage of go over go-now because go conveys async execution of its entire body whereas in go-now I'd have to scan the body for a parkable operation to determine the part which executes synchronously. I find it difficult to convince myself that there could really be problems so complex that go-now would be preferable over refactoring go. (I don't think you can safely rely on impl namespaces in library code )





[ASYNC-134] async/alts!! with an empty vector throws ArrayIndexOutOfBoundsException Created: 27/Jul/15  Updated: 28/Jul/15

Status: Open
Project: core.async
Component/s: None
Affects Version/s: None
Fix Version/s: None

Type: Defect Priority: Minor
Reporter: Daniel Compton Assignee: Unassigned
Resolution: Unresolved Votes: 0
Labels: None


 Description   

I'm not sure if this is intended, but this code throws an exception.

(async/alts!! [])
;; => ArrayIndexOutOfBoundsException 1  clojure.core.async/random-array (async.clj:182)

Is this desired behaviour, or is there another way we could represent that this collection is empty? I'm thinking of the pattern where we take from any channel, and recur with that channel removed from the collection until the collection is empty. Perhaps this usage should be discouraged. If it should be discouraged, should it be documented in the docstring?



 Comments   
Comment by Alex Miller [ 28/Jul/15 9:08 AM ]

What would you expect this to do?

Comment by Daniel Compton [ 28/Jul/15 5:20 PM ]

You could return [nil nil]? That's not great either though. Documentation is probably a better fix.





[ASYNC-144] pipeline-async docstring correction Created: 27/Aug/15  Updated: 27/Aug/15

Status: Open
Project: core.async
Component/s: None
Affects Version/s: None
Fix Version/s: None

Type: Defect Priority: Minor
Reporter: Derek Troy-West Assignee: Unassigned
Resolution: Unresolved Votes: 0
Labels: None


 Description   

Currently says:

"af must close! the channel before returning."

more correctly:

"af is responsible for closing the channel.", or similar.

af is asynchronous so should return immediately, the thread or go-block it spawns should close the channel later.






[ASYNC-54] MAX-QUEUE-SIZE has a wrong type-hint Created: 11/Feb/14  Updated: 23/Jun/14

Status: Open
Project: core.async
Component/s: None
Affects Version/s: None
Fix Version/s: None

Type: Defect Priority: Trivial
Reporter: Nicola Mometto Assignee: Unassigned
Resolution: Unresolved Votes: 0
Labels: None

Attachments: Text File 0001-Fix-MAX-QUEUE-SIZE-type-hint-def-evaluates-the-metad.patch    

 Description   

Since `def` evaluated the metadata on the symbol by design, primitive type hints needs to be quoted to avoid resolving to the homonimous clojure.core function



 Comments   
Comment by Alex Miller [ 23/Jun/14 3:22 PM ]

Actually, I think no hint is needed here at all and it should just be removed. Clojure will just inline a primitive long for the MAX-QUEUE-SIZE where it is used.





Generated at Sun Aug 30 03:09:37 CDT 2015 using JIRA 4.4#649-r158309.