<< Back to previous view

[TNS-45] File in invalid path will mark namespace for reload Created: 13/Sep/16  Updated: 21/Oct/16

Status: Open
Project: tools.namespace
Component/s: None
Affects Version/s: None
Fix Version/s: None

Type: Enhancement Priority: Major
Reporter: Juho Teperi Assignee: Stuart Sierra
Resolution: Unresolved Votes: 0
Labels: None

Attachments: Text File TNS-45-2.patch    

 Description   

Having a cljc file at path public/js/out/foo/bar.cljc with ns form (ns foo.bar) will result in namespace foo.bar being reloaded.

This is problematic because ClojureScript compiler will copy all the input files to :output-dir for source-map use. Lately as more libraries have started using cljc, and this has started causing problems in cases where library is used on Clojure env. Cljs compilation will result in reload of the library code, which can redefine protocols etc. and break the Clojure env.

I think it would make sense for tools.namespace to ignore changes where file path and namespace don't match.
Another question and perhaps way to fix this, is to understand why dependency resolution doesn't work in this case: namespaces which depend on the protocols at a cljc file on output-dir aren't reloaded.



 Comments   
Comment by Juho Teperi [ 13/Sep/16 3:50 PM ]

This is nearly the same to http://dev.clojure.org/jira/browse/TNS-24, but this case the inconsistently named ns will be reloade because there is a copy in the correct path.

Comment by Juho Teperi [ 18/Sep/16 10:00 AM ]

Proposed patch with a test.

find-sources-in-dir is the easiest place to do the check, because we need the directory path.

  • Possibly breaking for direct users of find-sources-in-dir?
  • Now both find-sources-in-dir and file/files-and-deps need to read the ns form.
Comment by Juho Teperi [ 26/Sep/16 6:43 AM ]

Fixed a typo on the patch.

Comment by Dominic Monroe [ 03/Oct/16 10:51 AM ]

I've also experienced this bug. It's particularly bad when combined with bidi due to it's use of protocols and records. This patch resolved the issue for me perfectly.

Comment by Stuart Sierra [ 21/Oct/16 10:57 AM ]

The root cause of this problem is the common practice of putting compiled "web assets" such as compiled ClojureScript code on the Java classpath at resources/public.

Two easy solutions are available right now:

1. Configure ClojureScript to put compiled files in a directory not on the classpath
2. Use set-refresh-dirs to omit those directories from the tools.namespace search.

Anything which changes the behavior of tools.namespace, especially in the lower-level APIs, has the potential to break other use cases, so must be considered carefully. For example, a code linter might use find-sources-in-dir to search for all Clojure source files, including those with invalid ns declarations.

Comment by Juho Teperi [ 21/Oct/16 11:05 AM ]

> 1. Configure ClojureScript to put compiled files in a directory not on the classpath

Using classpath to serve files is the most common way, and is unlikely to change. Many tools, like Boot, encourage using classpath to serve files.

> 2. Use set-refresh-dirs to omit those directories from the tools.namespace search.

This is not possible on Boot. In Boot classpath is managed by Boot and one directory will contain files from several sources, e.g. Boot-cljs and source-paths.





[TNS-42] scan-dirs / scan-all can return incorrect dependencies when both clj and cljc files define same namespace Created: 30/Dec/15  Updated: 06/Jan/16

Status: Open
Project: tools.namespace
Component/s: None
Affects Version/s: None
Fix Version/s: None

Type: Defect Priority: Major
Reporter: Andy Fingerhut Assignee: Stuart Sierra
Resolution: Unresolved Votes: 0
Labels: None
Environment:

tools.namespace 0.2.11 and 0.3.0-alpha2



 Description   

Originally filed as an issue for function scan-all, which at the time I was unaware was deprecated in 0.3.0 alphas. I have tested against the newer scan-dirs as well, and it has the same problem.

When there are both .clj and .cljc files defining the same namespace, tools.namespace scan-dirs and scan-all can use the dependencies from the .cljc file, ignoring those in the .clj file, whereas for Clojure/Java the opposite should be done.

See sample project and its README here for steps to reproduce: https://github.com/jafingerhut/tnstest

Note that Clojure/Java's behavior is to prefer .clj file over a .cljc file defining the same namespace, even if the .clj file is in a directory that is later in the classpath than a .cljc file. According to Alex Miller email on clojure-dev Google group, this behavior is by design. Link: https://groups.google.com/d/msg/clojure-dev/d5Hb1E7zfHE/sPybIAxgCwAJ



 Comments   
Comment by Stuart Sierra [ 05/Jan/16 11:03 AM ]

I can confirm this is a bug.

When I added multiple file extensions, I didn't consider the priority order.

Fixing this may be tricky.

It starts c.t.n.find. c.t.n.find/find-sources-in-dir filters by file extension (controlled by the platform argument in 0.3) but it only considers one directory at a time. Since the files could be in two different directories, it cannot necessarily discover two files with the same name but different extensions.

The next layer is c.t.n.file, which currently doesn't look at file extensions at all, but maybe it should.

In c.t.n.dir we have enough information to prioritize and filter files for the same namespace by extension. But to do it correctly, it has to handle updates too. For example, what happens to a project that has a .cljc file, then adds a .clj file. c.t.n.dir should treat that as removing the .cljc file from the dependency graph.

I think this will require storing the association between a namespace name and a file name, something I'd been hoping to avoid.

Comment by Stuart Sierra [ 05/Jan/16 11:21 AM ]

It might even make sense to make platform a property of the tracker, to make sure the same rules are always used when adding more files.

Comment by Andy Fingerhut [ 06/Jan/16 10:54 AM ]

Agreed that fixing this is not a small change to tools.namespace, hence the reason I don't have a proposed patch already. Came across this while thinking about how Eastwood could/should handle .cljc files. I'll add any proposed patches here if I come up with something.





[TNS-37] c.t.n.move: does not support to move .cljc files Created: 05/Sep/15  Updated: 20/Dec/15

Status: Open
Project: tools.namespace
Component/s: None
Affects Version/s: None
Fix Version/s: None

Type: Enhancement Priority: Major
Reporter: Benedek Fazekas Assignee: Stuart Sierra
Resolution: Unresolved Votes: 0
Labels: None

Attachments: Text File 0001-c.t.n.move-add-support-for-.cljc-file-extension.patch     Text File 0001-make-c.t.n.move-platform-aware-guess-platform.patch     Text File 0001-make-c.t.n.move-platform-aware.patch    
Patch: Code

 Description   

it seems that c.t.n.move did not get any reader conditionals love. I use this namespace to power mranderson: a source inlining tool used by cider-nrepl and refactor-nrepl in order to avoid clashes with the code/projects their users work on. lately we started adding dependencies to refactor-nrepl (well, tools.namespace itself) which may use .cljc files. source inlining started failing for these source files.



 Comments   
Comment by Stuart Sierra [ 06/Sep/15 12:57 PM ]

Good idea - glad to see how people are actually using tools.namespace in other dev cools.

Could this be extended further to take 'platform' as a parameter, as in c.t.n.find or c.t.n.dir?

(I deliberately excluded c.t.n.move to limit the scope of changes in 0.3.0, but your patch demonstrates that it may not not be such a difficult change to make.)

Comment by Benedek Fazekas [ 06/Sep/15 3:52 PM ]

first patch does what you asked for I hope. Kept original signiture of move-ns for backward compatibility: clj platform used by default.

second patch is pushing this a bit further: if platform is not provided it tries to guess it based on the file's extension of the old-sym. you might see this as over complication: feel free to pick the first patch then.

Comment by Stuart Sierra [ 06/Nov/15 3:02 PM ]

Thanks for working on these patches. I did some more testing, and uncovered some more complications.

Guessing the file extension leads to unpredictable results, because you don't know which file gets moved. It's not uncommon to have both .clj and .cljs version of the same namespace, for example with :require-macros in ClojureScript.

Unfortunately, even specifying a "platform" argument doesn't completely solve the problem, because you can have .cljc file at the same time as .clj and .cljs. This may not be common, but it's explicitly allowed for in the design of reader conditionals. For example, a library may have a platform-neutral .cljc file and override it with a platform-specific file. In this case, even if you specify a "platform" argument, you still can't control which file gets moved.

So now I think the only way for an operation like "move namespace" to make sense is to move all the files defining that namespace, preserving their file extensions. This seems like what one would want from a user perspective as well.

So move-ns should take new/old symbols, find all the matching files with any extension, move/rename them all while preserving extensions, then apply the textual transformation to all .clj, .cljs, and .cljc files.

If we're searching for files, we might as well eliminate the need to specify the source path, so this also relates to TNS-39.

Comment by Benedek Fazekas [ 20/Dec/15 3:44 PM ]

I have not abandoned this ticket or the patches but it seems it is related to a bigger problem in terms of supporting multiplatform projects therefore related to TNS-38 too.

I am also contemplating the wider context: `move-ns` is not only used in mranderson for source inlining but there is a refactor-nrepl feature (rename file or dir) which basically reimplements this functionality using other parts of TNS.

The mranderson story for me is about exploring different ways of dependency handling really, there was a good lightning talk by Glen Mailer summarising the problem on clojureX this year. My ideas are along creating local, deeply nested dependencies and perhaps enhance the ns macro to be able to load them – not even sure that is feasible. This would eliminate the need for source inlining and more importantly would be much more hygienic approach I think.

Hope this makes sense. Sorry for the brain dump which is obviously exceeds the scope of this ticket.





[TNS-6] Attempt to reload deleted file Created: 14/Dec/12  Updated: 12/Aug/15

Status: Open
Project: tools.namespace
Component/s: None
Affects Version/s: None
Fix Version/s: None

Type: Defect Priority: Major
Reporter: Stuart Sierra Assignee: Stuart Sierra
Resolution: Unresolved Votes: 1
Labels: None


 Description   

I can't identify the exact circumstances, but I have seen events where a source code file has been deleted but clojure.tools.namespace.repl/refresh still tries to reload it. Because the file doesn't exist, there's an exception when you try to load it, so you're stuck.



 Comments   
Comment by Gary Fredericks [ 12/Aug/15 10:21 AM ]

This happens to me pretty frequently, especially when switching branches (which is ironically the best use case for calling refresh).

Comment by Stuart Sierra [ 12/Aug/15 10:24 AM ]

I still don't know exactly how this occurs.

The workaround for now is to call c.t.n.repl/clear, added in 0.2.5

Comment by Gary Fredericks [ 12/Aug/15 1:26 PM ]

yep, noticed that independently and just confirmed that it works.





[TNS-46] Add project.clj for easier development Created: 13/Sep/16  Updated: 13/Sep/16

Status: Open
Project: tools.namespace
Component/s: None
Affects Version/s: None
Fix Version/s: None

Type: Enhancement Priority: Minor
Reporter: Juho Teperi Assignee: Stuart Sierra
Resolution: Unresolved Votes: 0
Labels: None

Attachments: Text File TNS-46.patch    
Patch: Code

 Description   

Tools.namespace doesn't currently have have a Leiningen configuration which would make it easy to start a repl and start developing tools.namespace.

Attached patch adds simple project.clj file with deps and source-paths.






[TNS-43] ns-decl? treats (ns) as valid Created: 28/Mar/16  Updated: 01/Apr/16

Status: Open
Project: tools.namespace
Component/s: None
Affects Version/s: None
Fix Version/s: None

Type: Defect Priority: Minor
Reporter: James Laver Assignee: Stuart Sierra
Resolution: Unresolved Votes: 0
Labels: None

Attachments: Text File ns-decl.patch    
Patch: Code

 Description   

If you try to declare an empty namespace with `(ns)`, you get an ArityException, but ns-decl? does not test for this.

The attached patch simply tests whether it is at least two items long.

I have previously signed the Clojure CA.



 Comments   
Comment by James Laver [ 28/Mar/16 6:45 AM ]

I'm wondering if it's worth updating my patch to also check that the namespace name is a symbol.

Comment by Stuart Sierra [ 01/Apr/16 9:33 AM ]

I feel like I need some more context here. What is the motivation for making a change in this case? Have you encountered real code in which (ns) causes a problem because tools.namespace accepts it as a valid namespace declaration?





[TNS-39] Single classpath argument for c.t.n.move Created: 06/Nov/15  Updated: 06/Nov/15

Status: Open
Project: tools.namespace
Component/s: None
Affects Version/s: None
Fix Version/s: None

Type: Enhancement Priority: Minor
Reporter: Stuart Sierra Assignee: Stuart Sierra
Resolution: Unresolved Votes: 0
Labels: None


 Description   

clojure.tools.namespace.move/move-ns requires that the caller provide both 1) the directory containing the file to be moved; and 2) the collection of directories containing all source files to be updated with the new name. This is redundant.

Instead, we can search for the file in all source directories, move it within the same root directory, then update all files in all directories.

In addition, with java.classpath in TNS-36, the set of directories to search can default to directories on the classpath. This will be much more convenient for REPL use.

This change can be made without breaking older arities of move-ns.






[TNS-24] Broken tracker after mis-named namespace Created: 07/Sep/14  Updated: 11/Sep/14

Status: Open
Project: tools.namespace
Component/s: None
Affects Version/s: None
Fix Version/s: None

Type: Defect Priority: Minor
Reporter: Stuart Sierra Assignee: Stuart Sierra
Resolution: Unresolved Votes: 0
Labels: None


 Description   

A tools.namespace dependency tracker can get into an inconsistent state after attempting to load a file with a namespace declaration that does not match the file's name.

Steps to reproduce:

In a new Clojure project, create a file foo.clj on the classpath containing the namespace declaration (ns wrong-ns-declaration-for-foo)

Then at the REPL:

user=> (use 'clojure.tools.namespace.repl)
nil
user=> (refresh)
:reloading (wrong-ns-declaration-for-foo)
:error-while-loading wrong-ns-declaration-for-foo
#<FileNotFoundException java.io.FileNotFoundException: Could not locate wrong_ns_declaration_for_foo__init.class or wrong_ns_declaration_for_foo.clj on classpath: >

Edit the file foo.clj so that its namespace declaration is correct: (ns foo)

But at the REPL, refresh still doesn't work:

user=> (refresh)
:reloading (foo wrong-ns-declaration-for-foo)
:error-while-loading wrong-ns-declaration-for-foo
#<FileNotFoundException java.io.FileNotFoundException: Could not locate wrong_ns_declaration_for_foo__init.class or wrong_ns_declaration_for_foo.clj on classpath: >

Since tools.namespace 0.2.5, the workaround is to call clear:

user=> (clear)
{}
user=> (refresh)
:reloading (foo)
:ok


 Comments   
Comment by Andy Fingerhut [ 11/Sep/14 6:38 PM ]

Such files introduce problems for all kinds of tools, not just tools.namespace. I don't know all of the consequences, but two are listed here: https://github.com/jonase/eastwood#check-consistency-of-namespace-and-file-names

I have added warnings in Eastwood whenever it finds files like this, and doesn't do any linting checks on any files until such things are corrected.





[TNS-7] Stack overflow on refresh after circular dependency detected Created: 17/Apr/13  Updated: 17/Apr/13

Status: Open
Project: tools.namespace
Component/s: None
Affects Version/s: None
Fix Version/s: None

Type: Defect Priority: Minor
Reporter: Stuart Sierra Assignee: Stuart Sierra
Resolution: Unresolved Votes: 1
Labels: None
Environment:

tools.namespace 0.2.3



 Description   
user> (refresh)
Exception Circular dependency between com.example.foo and com.example.bar  clojure.tools.namespace.dependency.MapDependencyGraph (dependency.clj:74)
user> (refresh)
StackOverflowError   clojure.lang.PersistentHashMap$BitmapIndexedNode.index (PersistentHashMap.java:576)
user> (clojure.repl/pst)
StackOverflowError 
	clojure.lang.PersistentHashMap$BitmapIndexedNode.index (PersistentHashMap.java:576)
	clojure.lang.PersistentHashMap$BitmapIndexedNode.find (PersistentHashMap.java:676)
	clojure.lang.PersistentHashMap$ArrayNode.find (PersistentHashMap.java:396)
	clojure.lang.PersistentHashMap.valAt (PersistentHashMap.java:152)
	clojure.lang.PersistentHashMap.valAt (PersistentHashMap.java:156)
	clojure.lang.RT.get (RT.java:645)
	clojure.tools.namespace.dependency/transitive (dependency.clj:52)
	clojure.tools.namespace.dependency/transitive/fn--7043 (dependency.clj:51)
	clojure.core.protocols/fn--6022 (protocols.clj:79)
	clojure.core.protocols/fn--5979/G--5974--5992 (protocols.clj:13)
	clojure.core/reduce (core.clj:6177)
	clojure.tools.namespace.dependency/transitive (dependency.clj:52)
nil





Generated at Tue Dec 06 11:58:44 CST 2016 using JIRA 4.4#649-r158309.