Error formatting macro: pagetree: java.lang.NullPointerException
Skip to end of metadata
Go to start of metadata
You are viewing an old version of this page. View the current version. Compare with Current  |   View Page History

problem statement

Many real-world problems can be reduced to handling queues of messages. The messages are often ordered; it's not enough to define a single handler for all messages, since one message can dictate how future messages are handled. These messages also often represent future side effects; once they are modified, filtered, and accumulated, they will generally be sent over the network, printed to the screen, or written to disk.

It's this last fact that points to a crucial difference between these message queues and Clojure sequences: adding a message to the queue is not always transactionally safe. However, this doesn't mean that messages can never be transactionally enqueued. If enqueuing the message doesn't directly result in a side effect, transactions are perfectly fine.

In other words, transactions can safely encompass the modification, filtering, and accumulation of messages, as long as there's a clear separation between that process and using the end result to create side effects. A good solution should maximize the extent to which this is possible, and minimize the potential for mistakes.

Something which satisfied all of the above would be a good candidate for inclusion in contrib.

some possible approaches

pure functions (a la Ring)


  • Leverages existing operators for composing and applying functions
  • Easy to understand and reason about, hides away side effects entirely


  • Assumes simple call and response model for communication
  • Works best when messages are unordered; one message cannot effect the handling of another without the introduction of state
  • Works best when response is immediate

blocking lazy-seqs


  • Leverages existing operators for sequences
  • Widely understood abstraction


  • Immutability doesn't make it safe: neither LinkedBlockingQueues or queues of promises are transactionally safe to enqueue into (if deliver is used in a transaction that retries, it will throw an exception the second time around)
  • Must manually allocate one thread per consumer
  • No concept of timeout, consumer threads can starve forever
  • Somewhat leaky abstraction; (next seq) takes a predictable amount of time pretty much everywhere else, and holding onto the head of the seq is a non-obvious trap to avoid

pub-sub events


  • node.js does it, so can we
  • Simplest asynchronous mechanism there is


  • If no one's listening, the data disappears
  • node.js imposes order on events by using a single thread, to do the same Clojure must either:
    • Make events per-thread (a la Erlang actors), which doesn't play nicely with Clojure's existing concurrency primitives
    • Impose an ordering using state, which is arguably something that should be abstracted away

the proposed approach

Channels are described in detail here, and a method of using them to compose asynchronous workflows is described here.

Some notable qualities about channels, to contrast against the other approaches:

  • When no callbacks are registered, messages queue up
  • Callbacks can either consume all messages (receive-all) or just some messages (receive, receive-while)

The combination of these two features means that we can choose between a push or pull model, depending on the situation. It also saves us from the transaction/side-effect issue, since can divert messages into a receiver-less channel while in a transaction, and then use those messages to achieve side effects outside the transaction.