Copy Page

You are not logged in. Any changes you make will be marked as anonymous. You may want to Log In if you already have an account. You can also Sign Up for a new account.

<h3>Rationale</h3>

<p>It can be difficult to manage resource lifetimes, especially when laziness is involved. Simply local scoping can be inadequate. Resource scopes can decouple resource lifetimes from creating scope.</p>

<ul>
	<li>Generalize from resource scopes to general scopes (succeed/fail/exit)</li>
	<li>with-open et al make scopes</li>
	<li>Some design work and code has been done
	<ul>
		<li>find it and write it up</li>
		<li><a href="http://github.com/clojure/clojure/blob/streams/src/clj/clojure/core.clj#L2005)">scope work in streams branch</a></li>
	</ul>
	</li>
</ul>

<h3>Use Cases</h3>

<ul>
	<li>REPL usage: open stream and it closes before I lazily consume it. Doh!</li>
	<li>Start an activity, and then have nested resource lifetime automatically tie to it
	<ul>
		<li>Task A calls Iib B which allocates resources</li>
	</ul>
	</li>
	<li>Prevent resource lifetimes from nesting
	<ul>
		<li>lib C knows it doesn't want its lifetime governed by Task A</li>
	</ul>
	</li>
	<li>All works across threads</li>
</ul>

<h3>Problems</h3>

<ul>
	<li>Other than memory allocation, the proper freeing of resources (e.g. closing open files) is on the user
	<ul>
		<li>thus, easy to get wrong</li>
	</ul>
	</li>
	<li>'He who acquires releases' is one strategy
	<ul>
		<li>can be automated, e.g. with-xxx
		<ul>
			<li>but with-xxx puts an envelope around the availability of the resource, tied to the acquirer</li>
			<li>we trip over this when we create and return lazy seqs that reference resources</li>
			<li>anywhere else?, could we limit the scope of the solution to lazy seqs?
			<ul>
				<li>even then, needs care</li>
				<li>e.g. closing on seq exhaustion not triggered by partial consumption</li>
			</ul>
			</li>
		</ul>
		</li>
	</ul>
	</li>
	<li>piggybacking on GC gets around this
	<ul>
		<li>i.e. finalizers</li>
		<li>but is bad, because we can't know when/if they will run
		<ul>
			<li>non-option?</li>
		</ul>
		</li>
	</ul>
	</li>
	<li>reference counting works in some langs
	<ul>
		<li>but not on the JVM where aliasing is common and undetectable</li>
		<li>is it possible to wrap resources with a counter/tracker?
		<ul>
			<li>open question</li>
		</ul>
		</li>
	</ul>
	</li>
	<li>can tag things as closable and call close everywhere defensively
	<ul>
		<li>but everywhere is a big and dangerous place</li>
	</ul>
	</li>
	<li>Fundamentally - how do you know when something is (or contains) a 'resource'
	<ul>
		<li>and how do you know when no one cares about it anymore?
		<ul>
			<li>and what does it mean to care?</li>
			<li>could just fold down to  - care means would be unhappy if closed</li>
		</ul>
		</li>
		<li>and what needs to be done about it when that is the case?
		<ul>
			<li>who knows what that is?</li>
		</ul>
		</li>
	</ul>
	</li>
	<li>Is the problem as simple as closing?
	<ul>
		<li>are there ever two paths?
		<ul>
			<li>e.g. happy path saves and unhappy path discards</li>
		</ul>
		</li>
	</ul>
	</li>
	<li>are there any similarities to exception handling?
	<ul>
		<li>one part of code has problem, another deals with it</li>
	</ul>
	</li>
</ul>

<h3>Issues</h3>

<ul>
	<li>Consider threads as well as nesting</li>
	<li>How do scopes interact with io! and stm</li>
	<li>Can we hydrate the fork/join tree where we need it?</li>
	<li>How to prevent mess?
	<ul>
		<li>(when-scope :fails ...) et al are very declarative</li>
		<li>but are setting up non-local effects</li>
		<li>potentially leading to wtf when triggered</li>
		<li>is this the right external API?
		<ul>
			<li>any way to ensure correctness</li>
			<li>will we be beset with requests for order control</li>
		</ul>
		</li>
	</ul>
	</li>
	<li>Is the nearest enclosing scope always the right one?
	<ul>
		<li>what to do if not
		<ul>
			<li>named scopes</li>
			<li>scopes as args</li>
		</ul>
		</li>
	</ul>
	</li>
</ul>

<h3>Proposal</h3>

<ul>
	<li>dynamic binding <code>*scope*</code> holds collection of things needing cleanup</li>
	<li>with-open binds a new <code>*scope*</code>, cleans up <code>*scope*</code> at end</li>
	<li>lib APIs can call a new fn <code>(scope thing)</code> to add a resource to the current bound scope
	<ul>
		<li>throws exception if no scope available</li>
	</ul>
	</li>
	<li>REPL cleans up scope before every loop</li>
	<li>*scope* is an atom, so N &quot;child&quot; threads can add to it
	<ul>
		<li>binding conveyance makes this work</li>
		<li>still up to you to make sure &quot;parent&quot; thread outlives children</li>
	</ul>
	</li>
	<li><a href="http://dev.clojure.org/jira/browse/CLJ-2">this patch</a> demonstrates the idea</li>
</ul>

<p>IMO this nails all the use cases and problems above, with a few<br />
caveats that I think can be subsumed into a single example as follows:</p>

<p>Caller A creates a scope. Lib B combines several resources to do some<br />
work for A, some of which will be lazy/incomplete when returning to<br />
A. The challenge is that B wants to clean up some, but not all, of the<br />
scoped things it used. Some possibilities:</p>

<ul>
	<li>You are on your own in this case. Don't use scopes.</li>
	<li>Scopes are more queryable/manipulable to help lib B author, e.g. named scopes.
	<ul>
		<li>complicated!</li>
	</ul>
	</li>
	<li>Scope push: Some way to opt back out of as scope (push me to the enclosing scope).
	<ul>
		<li>complicated and difficult to reason about</li>
	</ul>
	</li>
	<li>Make resources aware of their own cleanup logic, so that B can say &quot;I am done with C but still need D and E&quot;.
	<ul>
		<li>non-starter, pollutes every API in the universe with bookkeeping to propagate cleanup logic.</li>
	</ul>
	</li>
	<li>Leverage close-on-consume
	<ul>
		<li>This is an antipatten today, but a potentially useful optimization with scopes in play</li>
	</ul>
	</li>
	<li>Scope grab
	<ul>
		<li>Give Lib B a way to grab the scope associate with a single resource and clean it up:
<table class="wysiwyg-macro" data-macro-name="code" style="background-image: url(/plugins/servlet/confluence/placeholder/macro-heading?definition=e2NvZGV9&amp;locale=en_GB&amp;version=2); background-repeat: no-repeat;" data-macro-body-type="PLAIN_TEXT"><tr><td class="wysiwyg-macro-body"><pre>
;; scope-grab binds a scope and returns it without cleanup
(let [[c c-scope] (scope-grab (c-making-thing))
       d (d-making-thing)]
  ;; do some work that finishes with c
  (release-scope c-scope)
  ;; continue work
)
</pre></td></tr></table></li>
		<li>Easier to compose than named scopes
		<ul>
			<li>C doesn't have to do anything special to be used in this way</li>
			<li>B doesn't have to know anything about C's use of scopes</li>
		</ul>
		</li>
		<li>Easier to componse that scope push
		<ul>
			<li>simple inversion: cleanup what you are done with, instead of enumerating the things still in play</li>
		</ul>
		</li>
	</ul>
	</li>
</ul>

<p>I believe that 95% of the current problems people experience are solved with the simple mechanism proposed here. My preference is to start with &quot;you're on your own&quot; and evolve to &quot;scope grab&quot; if necessary. The latter is an easy, non-breaking extension.</p>

<p>TBD to flesh out this approach</p>

<ul>
	<li>investigate interaction with fork/join
	<ul>
		<li>think this reduces to &quot;make binding conveyance work with fork/join&quot;</li>
	</ul>
	</li>
	<li>new protocol or IFn for bits of cleanup logic placed in a scope
	<ul>
		<li>handling Closeable and IFn in an if statement for now</li>
		<li>code in core happens before protocols are available</li>
	</ul>
	</li>
	<li>verify that change to with-open is non-breaking for existing code
	<ul>
		<li>but note no benefit either, until you start calling (scope ...)</li>
	</ul>
	</li>
</ul>

<p>Scenarios</p>

<ol>
	<li>Creating a resource whose lifetime is bounded to yours: use with-open, as you do today</li>
	<li>Creating a resource that will be passed back to a caller unconsumed: call (scope res)
	<ol>
		<li>caller must have made a scope</li>
		<li>mandatory scope is not a limitation here, it is a sensible requirement</li>
	</ol>
	</li>
	<li>Returning &gt;1 resources from a function with different lifecycles and cleanup rules: not supported
	<ol>
		<li>not going to worry about this without realistic example</li>
	</ol>
	</li>
	<li>Create resources from child threads
	<ol>
		<li>works if thread creation conveys bindings</li>
		<li>you are responsible for waiting on child threads</li>
	</ol>
	</li>
	<li>Consuming one resource, pass another through to caller: put with-open around the one you are consuming
<table class="wysiwyg-macro" data-macro-name="code" style="background-image: url(/plugins/servlet/confluence/placeholder/macro-heading?definition=e2NvZGV9&amp;locale=en_GB&amp;version=2); background-repeat: no-repeat;" data-macro-body-type="PLAIN_TEXT"><tr><td class="wysiwyg-macro-body"><pre>
;; a cleaned up here, b cleaned up by caller
(let [a (with-open [a ...] ...
      b (scope ...)]
  b)
</pre></td></tr></table></li>
	<li>Decide on the fly whether to clean up or let parent do it
<table class="wysiwyg-macro" data-macro-name="code" style="background-image: url(/plugins/servlet/confluence/placeholder/macro-heading?definition=e2NvZGV9&amp;locale=en_GB&amp;version=2); background-repeat: no-repeat;" data-macro-body-type="PLAIN_TEXT"><tr><td class="wysiwyg-macro-body"><pre>
;; capture-scope? grab-scope? capture-cleanup?
(let [[a s] (capture-scope (make-a))]
  (if (done? ...)
    (s) ;; cleanup myself
    (scope s) ;; punt
))
</pre></td></tr></table></li>
</ol>