Copy Page

You are not logged in. Any changes you make will be marked as anonymous. You may want to Log In if you already have an account. You can also Sign Up for a new account.

<h2>problem statement</h2>

<p>Many real-world problems can be reduced to handling queues of messages.  The messages are often ordered; it's not enough to define a single handler for all messages, since one message can dictate how future messages are handled.  These messages also often represent future side effects; once they are modified, filtered, and accumulated, they will generally be sent over the network, printed to the screen, or written to disk.</p>

<p>It's this last fact that points to a crucial difference between these message queues and Clojure sequences: adding a message to the queue is not always transactionally safe.  However, this doesn't mean that messages can <strong>never</strong> be transactionally enqueued.  If enqueuing the message doesn't directly result in a side effect, transactions are perfectly fine.</p>

<p>In other words, transactions can safely encompass the modification, filtering, and accumulation of messages, as long as there's a clear separation between that process and using the end result to create side effects.  A good solution should maximize the extent to which this is possible, and minimize the potential for mistakes.</p>

<p>Something which satisfied all of the above would be a good candidate for inclusion in contrib.</p>

<h2>some possible approaches</h2>

<h3>pure functions (a la Ring)</h3>

<p><strong>Pros</strong></p>
<ul>
	<li>Leverages existing operators for composing and applying functions</li>
	<li>Easy to understand and reason about, hides away side effects entirely</li>
</ul>

<p><strong>Cons</strong></p>
<ul>
	<li>Assumes simple call and response model for communication</li>
	<li>Works best when messages are unordered; one message cannot effect the handling of another without the introduction of state</li>
	<li>Works best when response is immediate</li>
</ul>

<h3>blocking lazy-seqs</h3>

<p><strong>Pros</strong></p>
<ul>
	<li>Leverages existing operators for sequences</li>
	<li>Widely understood abstraction</li>
</ul>

<p><strong>Cons</strong></p>
<ul>
	<li>Immutability doesn't make it safe: neither LinkedBlockingQueues or queues of promises are transactionally safe to enqueue into (if deliver is used in a transaction that retries, it will throw an exception the second time around)</li>
	<li>Must manually allocate one thread per consumer</li>
	<li>No concept of timeout, consumer threads can starve forever</li>
	<li>Somewhat leaky abstraction; (next seq) takes a predictable amount of time pretty much everywhere else, and holding onto the head of the seq is a non-obvious trap to avoid</li>
</ul>

<h3>pub-sub events</h3>

<p><strong>Pros</strong></p>
<ul>
	<li>node.js does it, so can we</li>
	<li>Simplest asynchronous mechanism there is</li>
</ul>

<p><strong>Cons</strong></p>
<ul>
	<li>If no one's listening, the data disappears</li>
	<li>node.js imposes order on events by using a single thread, to do the same Clojure must either:
	<ul>
		<li>Make events per-thread (a la Erlang actors), which doesn't play nicely with Clojure's existing concurrency primitives</li>
		<li>Impose an ordering using state, which is arguably something that should be abstracted away</li>
	</ul>
	</li>
</ul>

<h2>the proposed approach</h2>

<p>Channels are described in detail <a href="http://github.com/ztellman/lamina/wiki/Channels">here</a>, and a method of using them to compose asynchronous workflows is described <a href="http://github.com/ztellman/lamina/wiki/Pipelines">here</a>.</p>

<p>Some notable qualities about channels, to contrast against the other approaches:</p>

<ul>
	<li>When no callbacks are registered, messages queue up</li>
	<li>Callbacks can either consume all messages (receive-all) or just some messages (receive, receive-while)</li>
</ul>

<p>The combination of these two features means that we can choose between a push or pull model, depending on the situation.  It also saves us from the transaction/side-effect issue, since can divert messages into a receiver-less channel while in a transaction, and then use those messages to achieve side effects outside the transaction.  </p>