Copy Page

You are not logged in. Any changes you make will be marked as anonymous. You may want to Log In if you already have an account. You can also Sign Up for a new account.

<p>Design page for Promises / Futures with callbacks.</p><h2>Motivation</h2><p>Clojure's <code>future</code> and <code>promise</code> provide implementations of <code>deref</code> that block the calling thread. There is currently no facility to invoke callback functions on values when they become available. Asynchronous styles of programming are not possible without this facility, especially in single-threaded runtimes such as ClojureScript.</p><h2>Proposed APIs</h2><h3>Low-Level Primitives</h3><p><code>promise</code> and <code>future</code> will return objects implementating a new protocol, <code>INotify</code>, supporting a function <code>attend</code> with two arities:</p><pre class="example">    attend [promise fn] [promise fn executor]
</pre><p>This function attaches a callback to the promise/future which will be invoked on the executor when the promise is delivered with a value. (<em>attend</em>, transitive verb, &quot;occur with or as a result of&quot;)</p><p>Promises also implement the new protocol <code>IDeliver</code>, supporting two functions:</p><pre class="example">    deliver [promise value]
    fail [promise exception]
</pre><p>The <code>deliver</code> function provides a value to the promise, invokes any pending callbacks, and releases any pending derefs. The <code>fail</code> function delivers an exception to the promise (see &quot;Exceptions&quot; below).</p><p>The behavior of <code>deref</code> on futures/promises does not change, except that a failed promise may throw an exception (see below).</p><h3>Higher-Level Functions &amp; Macros</h3><p><span>Two macros facilitate composing promises:</span></p><pre>    then [promise binding-form &amp; body]</pre><pre>    recover [promise symbol &amp; body]</pre><p>The <code>then</code> macro waits for the promise to be delivered successfully and executes <code>body</code> in a callback with the value of the promise bound to <code>binding-form</code>. It returns a new promise which will be delivered with the value of body. If the promise fails or if <code>body</code> throws an exception, <code>then</code> propagates the exception to the returned promise.</p><p>The <code>recover</code> macro catches exceptions thrown from a failed promise, executes <code>body</code> with the exception bound to symbol, and returns a new promise which will be delivered with the value of body. If the promise does not fail, <code>recover</code> propagates its value to the returned promise. This behavior is intended to be similar to <code>try/catch</code>.</p><p>These macros can be composed with the threading macro:</p><table class="wysiwyg-macro" data-macro-name="code" style="background-image: url(/plugins/servlet/confluence/placeholder/macro-heading?definition=e2NvZGV9&amp;locale=en_GB&amp;version=2); background-repeat: no-repeat;" data-macro-body-type="PLAIN_TEXT"><tr><td class="wysiwyg-macro-body"><pre> (-> a-promise
    (then x ...)
    (then y ...)
    (then z ...)
    (recover ex ...)) 
</pre></td></tr></table><p>The <code>then</code> and <code>recover</code> macros are implemented in terms of two functions, <code>then-call</code> and <code>recover-call</code>, which take functions as arguments.</p><p><span style="color: rgb(0,0,0);font-size: 1.4em;font-weight: bold;line-height: normal;">Execution Context</span></p><p><span style="color: rgb(0,0,0);">If not supplied, </span><code>executor</code><span style="color: rgb(0,0,0);"> defaults to a dynamic Var, </span><code>*callback-executor*</code><span style="color: rgb(0,0,0);">, resolved </span><strong>when the callback is created</strong><span style="color: rgb(0,0,0);">, that is, at the time </span><code>attend</code><span style="color: rgb(0,0,0);"> was called. This Var is initially set to the Agent send-off thread pool executor, and can be globally reset or thread-bound by applications.</span></p><h3><span style="color: rgb(0,0,0);">Running Callbacks Locally</span><span style="font-size: 10.0pt;font-weight: normal;line-height: 13.0pt;"> </span></h3><p>If callbacks are known to be fast, it may be more efficient to run them directly on the thread that invoked <code>deliver</code>. This is enabled by calling <code>attend</code> with a <code>nil</code> executor (or binding <code>*callback-executor*</code> to <code>nil</code>).<span style="color: rgb(0,0,0);font-size: 10.0pt;line-height: 13.0pt;"> </span></p><p>Guava provides a similar capability with <a href="http://docs.guava-libraries.googlecode.com/git-history/release/javadoc/com/google/common/util/concurrent/MoreExecutors.html#sameThreadExecutor()">sameThreadExecutor</a>. It is somewhat related to the JDK's <a href="http://docs.oracle.com/javase/6/docs/api/java/util/concurrent/ThreadPoolExecutor.CallerRunsPolicy.html">CallerRunsPolicy</a> for rejected execution.</p><h2>Further Analysis</h2><h3>Delivery</h3><p>The APIs in this proposal can be delivered either as a library or as an extension to the Clojure language.</p><p>Extending the Clojure language has the benefits of enhancing the existing <code>promise</code> and <code>future</code> in a standard way. Having a standard mechanism for callbacks at the language level will facilitate sharing callbacks across different libraries and frameworks.</p><h3>Exceptions</h3><p>A callback function (really, any function) can do one of three things:</p><ul><li>Return a value</li><li>Throw an exception</li><li>Never return at all</li></ul><p>Failing to return is almost certainly a bug anywhere execpt on the main application thread. I assume that returning a value indicates a successful execution, while throwing an exception indicates failure. However, it is also possible to <strong>return</strong> an exception object as a value, which does not necessarily indicate failure. (For example, a logging framework might handle exceptions as values.)</p><p>Given that throwing an exception is different from returning a value, promises need to handle it on a different path. This is the <code>fail</code> function, which acts like <code>deliver</code> but takes an exception as its argument and puts the promise in a &quot;failed&quot; state.</p><p>When it comes time to invoke callback functions, there are several possible ways to handle failure:</p><ol><li>Provide separate callback attachment points for success and failure</li><li>Pass the exception to the callback as a normal value</li><li>Wrap the value or exception in a union type</li></ol><p>Method 1 leads to a proliferation of error-handling functions. Method 2 makes it impossible to tell if the exception object was thrown or delivered as a normal value. Method 3 is more natural in a statically-typed language (Scala does this). In both 2 and 3 the callback function must always check whether its argument is a normal value or an exception.</p><p>I have chosen a variant of 3 by using the promise itself as the argument to the callback function. The callback is responsible for calling <code>deref</code> on the promise. If the promise is in a failed state, dereferencing it will throw the exception. Callback functions can use standard try/catch blocks to handle the exception or allow it to propagate. Macros such as <code>on</code> handle propagation automatically and allow users to program in terms of values.</p><p>Delivering a promise A as a value to another promise B causes B to <code>attend</code> on A. Whenever A is delivered/failed, B is delivered/failed with the same value. This makes it convenient to implement multiple try/retry operations with promises, as in this example:</p><table class="wysiwyg-macro" data-macro-name="code" style="background-image: url(/plugins/servlet/confluence/placeholder/macro-heading?definition=e2NvZGV9&amp;locale=en_GB&amp;version=2); background-repeat: no-repeat;" data-macro-body-type="PLAIN_TEXT"><tr><td class="wysiwyg-macro-body"><pre>(defn do-operation [tries]
  (if (zero? tries)
    (throw (ex-info "Operation failed" {}))
    (-> (future ...)       ; long operation which may fail
        (then value ...)   ; handle success
        (recover ex        ; handle failure with retry
          (do-operation (dec tries))))))
 
(do-operation 3)
</pre></td></tr></table><h3>Java Interop</h3><p>Promises implement java.util.concurrent.Future. Extending them to similar APIs such as Guava's ListenableFuture is trivial.</p><h3>Cancellation</h3><p>There is no API to remove or cancel a callback created with <code>attend</code>. However, it is possible to deliver a value to the promise returned by the <code>then</code> macro <strong>before</strong> the underlying promise is delivered. In that case, the callback function will not be executed. <span style="font-size: 10.0pt;line-height: 13.0pt;">There is an implicit race condition with cancellation, so this cannot guarantee that the callback function will never be executed at all.</span></p><p>Futures can still be cancelled with <code>future-cancel</code>, which attempts to stop the Future's thread.</p><p>Calling <code>future-cancel</code> on a promise is equivalent to calling <code>fail</code> with an instance of java.util.concurrent.CancellationException.</p><h3>Progress Reporting</h3><p>Promises do not provide an API to report partial progress during a long-running operation. Progress reporting can be handled through other means such as watches.</p><h3>ClojureScript</h3><p>Promises can be implemented in ClojureScript, subject to the following limitations:</p><ul><li>There is no notion of executors apart from &quot;execute immediately&quot; or &quot;execute asynchronously&quot; <span>(e.g., with </span><code>setTimeout</code><span>)</span></li><li><code style="font-size: 10.0pt;line-height: 13.0pt;">deref</code><span style="font-size: 10.0pt;line-height: 13.0pt;"> on an unrealized promise cannot block; instead it throws an exception</span></li></ul><h2>Discussions</h2><ul><li><a href="https://groups.google.com/d/topic/clojure-dev/7BKQi9nWwAw/discussion">Promise/Futures with callbacks</a> (Clojure-dev group)</li><li><a href="https://groups.google.com/d/topic/clojure/5lEABZyD1L0/discussion">lamina and channels-driven concurrency with clojure</a> (Clojure group)</li><li><a href="http://dev.clojure.org/display/design/reactive+programming">reactive programing</a> (dev.clojure.org)</li><li><a href="https://groups.google.com/d/topic/datomic/u1UTCws2hKU/discussion">can a better Future be exposed by API?</a> (Datomic group)</li></ul><h2>Implementations</h2><ul><li><a href="https://github.com/stuartsierra/cljque/blob/master/src/cljque/promises.clj">cljque promises</a> (Stuart Sierra)</li></ul><h2>References / Related Work</h2><ul><li>Java<ul><li><a href="http://code.google.com/p/guava-libraries/wiki/ListenableFutureExplained">Listenable Futures</a> in Google Guava</li><li><a href="http://asynchttpclient.github.com/async-http-client/apidocs/reference/com/ning/http/client/ListenableFuture.html">ListenableFuture</a> in <a href="http://asynchttpclient.github.com/async-http-client/">Async HTTP Client</a></li></ul></li><li>Scala<ul><li><a href="http://docs.scala-lang.org/sips/pending/futures-promises.html">SIP-14: Futures and Promises</a></li></ul></li><li>JavaScript<ul><li><a href="http://wiki.commonjs.org/wiki/Promises/A">CommonJS Promises/A</a></li><li><a href="http://promises-aplus.github.io/promises-spec/">Promises/A+</a> (extension of Promises/A)</li><li><a href="https://gist.github.com/3889970">You're Missing the Point of Promises</a></li><li><a href="https://github.com/kriskowal/q">Q</a></li><li><a href="http://docs.angularjs.org/api/ng.$q">Angular JS promise/deferred</a></li></ul></li><li><span style="font-size: 10.0pt;line-height: 13.0pt;">C# / .NET</span><ul><li><a href="http://msdn.microsoft.com/en-us/library/dd997364.aspx" style="font-size: 10.0pt;line-height: 13.0pt;">Cancellation in Managed Threads</a></li></ul></li><li>Python<ul><li><a href="http://packages.python.org/defer/defer.html">Defer</a></li><li><a href="https://twistedmatrix.com/trac/ticket/990">Deferred cancellation</a></li></ul></li></ul><h3>Comparison to Guava ListenableFuture</h3><p>Guava's ListenableFuture takes the same basic approach of passing a Runnable and an Executor for the callback. It should be trivial to extend the INotify and IDeliver protocols to Guava's <a href="http://docs.guava-libraries.googlecode.com/git-history/release/javadoc/com/google/common/util/concurrent/ListenableFuture.html">ListenableFuture</a> and <a href="http://docs.guava-libraries.googlecode.com/git-history/release/javadoc/com/google/common/util/concurrent/SettableFuture.html">SettableFuture</a>.</p><p>The <a href="https://code.google.com/p/guava-libraries/source/browse/src/com/google/common/util/concurrent/AbstractFuture.java?name=v9.0">implementation</a> in Guava is more sophisticated than the one in <a href="https://github.com/stuartsierra/cljque/blob/master/src/cljque/promises.clj">cljque</a>, using an <a href="http://docs.oracle.com/javase/7/docs/api/java/util/concurrent/locks/AbstractQueuedSynchronizer.html">AbstractQueuedSynchronizer</a> instead of monitors. </p>