problem statement

Many real-world problems can be reduced to handling queues of messages. The messages are often ordered; it's not enough to define a single handler for all messages, since one message can dictate how future messages are handled. These messages also often represent future side effects; once they are modified, filtered, and accumulated, they will generally be sent over the network, printed to the screen, or written to disk.

It's this last fact that points to a crucial difference between these message queues and Clojure sequences: adding a message to the queue is not always transactionally safe. However, this doesn't mean that messages can never be transactionally enqueued. If enqueuing the message doesn't directly result in a side effect, transactions are perfectly fine.

In other words, transactions can safely encompass the modification, filtering, and accumulation of messages, as long as there's a clear separation between that process and using the end result to create side effects. A good solution should maximize the extent to which this is possible, and minimize the potential for mistakes.

Something which satisfied all of the above would be a good candidate for inclusion in contrib.

some possible approaches

pure functions (a la Ring)

Pros

• Leverages existing operators for composing and applying functions
• Easy to understand and reason about, hides away side effects entirely

Cons

• Assumes simple call and response model for communication
• Works best when messages are unordered; one message cannot effect the handling of another without the introduction of state
• Works best when response is immediate

blocking lazy-seqs

Pros

• Leverages existing operators for sequences
• Widely understood abstraction

Cons

• Immutability doesn't make it safe: neither LinkedBlockingQueues or queues of promises are transactionally safe to enqueue into (if deliver is used in a transaction that retries, it will throw an exception the second time around)
• Must manually allocate one thread per consumer
• No concept of timeout, consumer threads can starve forever
• Somewhat leaky abstraction; (next seq) takes a predictable amount of time pretty much everywhere else, and holding onto the head of the seq is a non-obvious trap to avoid

pub-sub events

Pros

• node.js does it, so can we
• Simplest asynchronous mechanism there is

Cons

• If no one's listening, the data disappears
• node.js imposes order on events by using a single thread, to do the same Clojure must either:
  • Make events per-thread (a la Erlang actors), which doesn't play nicely with Clojure's existing concurrency primitives
  • Impose an ordering using state, which is arguably something that should be abstracted away

the proposed approach

Channels are described in detail here, and a method of using them to compose asynchronous workflows is described here.

Some notable qualities about channels, to contrast against the other approaches:

• When no callbacks are registered, messages queue up
• Callbacks can either consume all messages (receive-all) or just some messages (receive, receive-while)

The combination of these two features means that we can choose between a push or pull model, depending on the situation. It also saves us from the transaction/side-effect issue, since can divert messages into a receiver-less channel while in a transaction, and then use those messages to achieve side effects outside the transaction.