problem statement
Many real-world problems can be reduced to handling queues of messages. The messages are often ordered; it's not enough to define a single handler for all messages, since one message can dictate how future messages are handled. These messages also often represent future side effects; once they are modified, filtered, and accumulated, they will generally be sent over the network, printed to the screen, or written to disk.
It's this last fact that points to a crucial difference between these message queues and Clojure sequences: adding a message to the queue is not always transactionally safe. However, this doesn't mean that messages can never be transactionally enqueued. If enqueuing the message doesn't directly result in a side effect, transactions are perfectly fine.
In other words, transactions can safely encompass the modification, filtering, and accumulation of messages, as long as there's a clear separation between that process and using the end result to create side effects. A good solution should maximize the extent to which this is possible, and minimize the potential for mistakes.
Something which satisfied all of the above would be a good candidate for inclusion in contrib.
some possible approaches
pure functions (a la Ring)
Pros
- Leverages existing operators for composing and applying functions
- Easy to understand and reason about, hides away side effects entirely
Cons
- Assumes simple call and response model for communication
- Works best when messages are unordered; one message cannot effect the handling of another without the introduction of state
- Works best when response is immediate
blocking lazy-seqs
Pros
- Leverages existing operators for sequences
- Widely understood abstraction
Cons
- Immutability doesn't make it safe: neither LinkedBlockingQueues or queues of promises are transactionally safe to enqueue into (if deliver is used in a transaction that retries, it will throw an exception the second time around)
- Must manually allocate one thread per consumer
- No concept of timeout, consumer threads can starve forever
- Somewhat leaky abstraction; (next seq) takes a predictable amount of time pretty much everywhere else, and holding onto the head of the seq is a non-obvious trap to avoid
pub-sub events
Pros
- node.js does it, so can we
- Simplest asynchronous mechanism there is
Cons
- If no one's listening, the data disappears
- node.js imposes order on events by using a single thread, to do the same Clojure must either:
- Make events per-thread (a la Erlang actors), which doesn't play nicely with Clojure's existing concurrency primitives
- Impose an ordering using state, which is arguably something that should be abstracted away
the proposed approach
Channels are described in detail here, and a method of using them to compose asynchronous workflows is described here.
Some notable qualities about channels, to contrast against the other approaches:
- When no callbacks are registered, messages queue up
- Callbacks can either consume all messages (receive-all) or just some messages (receive, receive-while)
The combination of these two features means that we can choose between a push or pull model, depending on the situation. It also saves us from the transaction/side-effect issue, since can divert messages into a receiver-less channel while in a transaction, and then use those messages to achieve side effects outside the transaction.