<< Back to previous view

[TNS-20] tracker's unload order sometimes incorrect Created: 23/Aug/14  Updated: 30/Jan/15  Resolved: 30/Jan/15

Status: Resolved
Project: tools.namespace
Component/s: None
Affects Version/s: None
Fix Version/s: None

Type: Defect Priority: Major
Reporter: Andy Fingerhut Assignee: Stuart Sierra
Resolution: Completed Votes: 0
Labels: None

Attachments: Text File tns-20-v1.patch     Text File tns-20-v2.patch    


The attached patch contains a new test namespace clojure.tools.namespace.load-unload-order-test that demonstrates the incorrect unload order, if you leave out the proposed fix in file track.clj

Comment by Andy Fingerhut [ 23/Aug/14 3:02 PM ]

Patch tns-20-v1.patch contains a test that demonstrates the bug, and contains a proposed fix. Please scrutinize the proposed fix carefully, as I haven't yet convinced myself 100% that it is the right fix.

Also adds one more dependency check in dependency_test.clj that I discovered while reviewing those tests, to model my dependency checking tests on.

Comment by Andy Fingerhut [ 24/Aug/14 4:52 PM ]

I've got some code that checks that the load and unload orders in a tools.namespace tracker are consistent with its dependencies. It is similar to the checks done now in dependencies_test.clj, except it assumes that the dependencies in the tracker are correct and verifies the load and unload orders against those.

If you would be interested in a patch that added these checks to tools.namespace, perhaps along with a debug/consistency-check flag that when true causes these checks to be run every time a tracker is updated, let me know.

Comment by Stuart Sierra [ 07/Sep/14 11:11 AM ]

Thanks for this, Andy. I will need to study it further to convince myself this is the correct behavior.

To help, can you describe what the observed problem would be from a user's point of view?

We cannot assume that the dependencies in the tracker are always correct. I have demonstrable cases where they are wrong. Usually that takes the form of nonexistent namespaces left in the dependency graph from files that were deleted or had an invalid ns declaration.

Comment by Andy Fingerhut [ 07/Sep/14 8:10 PM ]

I have not used tools.namespace for the reload workflow that it was originally developed for, so I can't say with certainty what the effect on a user would be, but I can make some guesses that you can probably confirm more quickly than I can.

The test case in the patch is one where a few Clojure source files are added to a tracker using function c.t.n.dir/scan-all. No other changes are made to the tracker. At that point, the dependencies are completely correct, and the load order calculated from those dependencies honors them, but the unload order does not.

If that tracker were then used in a call to c.t.n.reload/track-reload, track-reload would call remove-lib on a library B before calling remove-lib on a library A, even though A requires or uses B.

I guess your question confuses me a bit. Do you believe tools.namespace should only create trackers that have load and unload orders that honor the dependencies? Or is that a wrong assumption I was making from reading the implementation?

Comment by Andy Fingerhut [ 08/Sep/14 5:52 PM ]

Attaching slightly cleaned up patch tns-20-v2.patch. Identical to tns-20-v1.patch except it avoids copying a function into the new test namespace by adding a dependency on the test namespace where it is defined.

Also updates the name of a deftest I had copied but not renamed in v1 of the patch.

Comment by Stuart Sierra [ 19/Sep/14 4:19 PM ]

I think I'm starting to get a handle on this. There's a lot going on here, and it's been more than two years since I wrote most of this code, so bear with me.

At track.clj line 78 we compute the dependency order for unloading namespaces based on the dependency graph (the local deps) as it existed before the most recent set of changes. I believe that this is correct, or at least the correct intention. If we change a file such that its dependencies are different, the order in which we unload namespaces should reflect the namespaces in memory, as they were before we changed the file.

When using c.t.n.dir/scan to detect and reload changed files this works correctly, at least most of the time.

When adding files to a new tracker for the first time, the old dependency graph is empty, so the unload order is undefined. This is arguably incorrect but effectively meaningless, since those namespaces have not yet been loaded.

In release 0.2.6, there was a bad commit which mistakenly changed c.t.n.dir/scan and scan-all to remove the files which have changed before adding them again. As a result, the dependencies of changed files were removed from the tracker's dependency graph before the unload order could be calculated, so unload order was always undefined. I have reverted that change — thanks for drawing my attention to it!

Now the unload order should be "correct" after scan or scan-all, except for the first time files are added to the tracker, when unload order is undefined.

The tracker doesn't currently have a way to distinguish between changed files which need unload+load and new files which only need load. Even if there were a way to distinguish between new and changed files, we can't be certain that a namespace has not been loaded by other means (e.g. require at the REPL) so it's safest to unload everything before reloading.

In general, unload order shouldn't matter at all. Clojure doesn't care when namespaces are removed: the Vars are still referenced from wherever they are used.

Comment by Andy Fingerhut [ 20/Sep/14 1:08 PM ]

OK, makes sense. Is it already documented anywhere that the unload order is independent of the dependencies after scan-all (and perhaps other calls)? I can create a separate ticket for that if you think it is worth adding such documentation.

It sounds like for the application I had in mind, the reverse of the load order is always defined, and is a correct unload order. You are welcome to close this ticket.

Comment by Stuart Sierra [ 26/Sep/14 9:44 AM ]

After release 0.2.7, which fixed the regression in 0.2.6, :unload order should be correct in all cases except the first time files are added to a new tracker, even with scan-all. I would appreciate it if you can confirm this in your application.

It may be possible to fix the new-tracker case too. For example, commit c0b6b93d, currently on the branch TNS-20-fix-initial-unload-order. This works for the tests in your patch.

I'm not sure if the same change is needed at track.clj line 104 as well.

Comment by Andy Fingerhut [ 02/Oct/14 5:12 PM ]

My application right now is very simple compared to the component workflow – simply use dir/scan-all to get the namespaces and their dependencies, and then print them in a particular order consistent with a correct unload order. I'm sorry, but I don't have the interest right now in testing whether the unload order is correct after doing additional operations on the tracker, since it isn't needed in my application.

I do have a sanity check in my application that confirms the load order is consistent with the dependencies in my application, and will file a bug if I ever see that trigger anything, but I don't expect there is a bug there.

I have switched to using the reverse of the load order in my application for what I believe should always be a correct unload order.

I may look into the branch change you made, but it may fall off my radar unless my needs change.

Comment by Stuart Sierra [ 30/Jan/15 9:57 AM ]

On Git master as of https://github.com/clojure/tools.namespace/commit/4465f8ee13f1a95907725dae4f0000ab7f9fd24e

Will be in 0.2.9 release

[TNS-21] tools.namespace ignores ns form dependencies inside vectors Created: 23/Aug/14  Updated: 30/Jan/15  Resolved: 30/Jan/15

Status: Resolved
Project: tools.namespace
Component/s: None
Affects Version/s: None
Fix Version/s: None

Type: Defect Priority: Minor
Reporter: Andy Fingerhut Assignee: Stuart Sierra
Resolution: Completed Votes: 0
Labels: None

Attachments: Text File 0002-TNS-21-Allow-ns-clauses-to-be-vectors-instead-of-lis.patch     Text File tns-21-v1.patch    
Patch: Code and Test


Cross reference: TLOG-13

For example, tools.logging has this ns form (metadata omitted):

(ns clojure.tools.logging
   [clojure.string :only [trim-newline]]
   [clojure.pprint :only [code-dispatch pprint with-pprint-dispatch]]]
  [:require [clojure.tools.logging.impl :as impl]])

tools.namespace ignores these dependencies, only processing those that are in list subforms of the ns form.

Not sure if the best way to handle this is to update tools.logging or tools.namespace. tools.logging seems pretty unusual in using vectors like this, but the Clojure compiler seems to accept it just fine.

Comment by Stuart Sierra [ 24/Aug/14 1:29 PM ]

The ns macro docstring is quite clear that references inside it should be lists. I've seen a lot of weird ns forms, and this is the first time I've seen vectors used this way.

I would say I'm surprised that vectors work here, except nothing weird about the ns macro surprises me any more.

Comment by Andy Fingerhut [ 22/Dec/14 8:46 PM ]

Temporarily reopening the ticket just to assign a patch for possible consideration. Will return it to the state I found it in when done.

Comment by Andy Fingerhut [ 22/Dec/14 8:47 PM ]

Proposed patch tns-21-v1.patch dated Dec 22 2014 would enable tools.namespace to recognize dependencies in ns forms where the references are vectors, in addition to lists.

Comment by Stuart Sierra [ 30/Jan/15 9:43 AM ]

On Git master as of https://github.com/clojure/tools.namespace/commit/d6a4a65c11f571280d02382dfb20c9efd743d70a

Will be in 0.2.9 release

[TNS-32] Accept symbols for 'require' and 'use' inside 'ns' Created: 19/Dec/14  Updated: 30/Jan/15  Resolved: 30/Jan/15

Status: Resolved
Project: tools.namespace
Component/s: None
Affects Version/s: None
Fix Version/s: None

Type: Enhancement Priority: Minor
Reporter: Stuart Sierra Assignee: Stuart Sierra
Resolution: Completed Votes: 1
Labels: None

Attachments: Text File 0001-TNS-32-handle-non-keyword-clause-heads-in-ns-form.patch     Text File 0002-TNS-32-handle-non-keyword-clause-heads-in-ns-form.patch    
Patch: Code and Test


clojure.core.ns allows much more syntactic variation than its docstring describes.

One example is (ns foo (require bar)) where require should be the keyword :require.

As of 0.2.9, tools.namespace silently ignores these forms.

Comment by Andy Fingerhut [ 22/Dec/14 1:34 PM ]

Code change and test look perfect to me.

Comment by Petr Gladkikh [ 23/Dec/14 11:39 AM ]

Line 48 in 0001-TNS-32-handle-non-keyword-clause-heads-in-ns-form.patch:
"(def t-clauses-without-keywords" -> "(deftest t-clauses-without-keywords" ?

Comment by Andy Fingerhut [ 23/Dec/14 12:35 PM ]

Good catch, Petr. I missed that.

Comment by Stuart Sierra [ 30/Jan/15 9:42 AM ]

On Git master as of https://github.com/clojure/tools.namespace/commit/bb472283a71156bf4bda1265389b866e7326bf0d

Will be in 0.2.9 release

Generated at Sat Jan 31 18:32:13 CST 2015 using JIRA 4.4#649-r158309.