<< Back to previous view

[TNS-48] Ability to track files that are not namespaces Created: 03/Nov/17  Updated: 16/Jan/18

Status: Open
Project: tools.namespace
Component/s: None
Affects Version/s: None
Fix Version/s: None

Type: Enhancement Priority: Major
Reporter: Ray Huang Assignee: Stuart Sierra
Resolution: Unresolved Votes: 0
Labels: None


Support tracking non-clojure files when performing a refresh. The idea here is to make the reloaded workflow work well for libraries like HugSQL (which may depend on non-clojure files in the project).

See https://github.com/layerware/hugsql/issues/72 for some additional context

Comment by Stuart Sierra [ 16/Jan/18 4:54 PM ]

I have been aware of the value of this feature for a long time now, but I expect it would require significant, breaking changes to the internal data structures used by tools.namespace.

[TNS-45] File in invalid path will mark namespace for reload Created: 13/Sep/16  Updated: 01/Jan/18

Status: Open
Project: tools.namespace
Component/s: None
Affects Version/s: None
Fix Version/s: None

Type: Enhancement Priority: Major
Reporter: Juho Teperi Assignee: Stuart Sierra
Resolution: Unresolved Votes: 5
Labels: None

Attachments: Text File TNS-42-3.patch     Text File TNS-45-2.patch     Text File TNS-45-4.patch     Text File TNS-45-5.patch    
Patch: Code and Test


Having a cljc file at path public/js/out/foo/bar.cljc with ns form (ns foo.bar) will result in namespace foo.bar being reloaded.

This is problematic because ClojureScript compiler will copy all the input files to :output-dir for source-map use. Lately as more libraries have started using cljc, and this has started causing problems in cases where library is used on Clojure env. Cljs compilation will result in reload of the library code, which can redefine protocols etc. and break the Clojure env.

I think it would make sense for tools.namespace to ignore changes where file path and namespace don't match.
Another question and perhaps way to fix this, is to understand why dependency resolution doesn't work in this case: namespaces which depend on the protocols at a cljc file on output-dir aren't reloaded.

Comment by Juho Teperi [ 13/Sep/16 3:50 PM ]

This is nearly the same to http://dev.clojure.org/jira/browse/TNS-24, but this case the inconsistently named ns will be reloade because there is a copy in the correct path.

Comment by Juho Teperi [ 18/Sep/16 10:00 AM ]

Proposed patch with a test.

find-sources-in-dir is the easiest place to do the check, because we need the directory path.

  • Possibly breaking for direct users of find-sources-in-dir?
  • Now both find-sources-in-dir and file/files-and-deps need to read the ns form.
Comment by Juho Teperi [ 26/Sep/16 6:43 AM ]

Fixed a typo on the patch.

Comment by Dominic Monroe [ 03/Oct/16 10:51 AM ]

I've also experienced this bug. It's particularly bad when combined with bidi due to it's use of protocols and records. This patch resolved the issue for me perfectly.

Comment by Stuart Sierra [ 21/Oct/16 10:57 AM ]

The root cause of this problem is the common practice of putting compiled "web assets" such as compiled ClojureScript code on the Java classpath at resources/public.

Two easy solutions are available right now:

1. Configure ClojureScript to put compiled files in a directory not on the classpath
2. Use set-refresh-dirs to omit those directories from the tools.namespace search.

Anything which changes the behavior of tools.namespace, especially in the lower-level APIs, has the potential to break other use cases, so must be considered carefully. For example, a code linter might use find-sources-in-dir to search for all Clojure source files, including those with invalid ns declarations.

Comment by Juho Teperi [ 21/Oct/16 11:05 AM ]

> 1. Configure ClojureScript to put compiled files in a directory not on the classpath

Using classpath to serve files is the most common way, and is unlikely to change. Many tools, like Boot, encourage using classpath to serve files.

> 2. Use set-refresh-dirs to omit those directories from the tools.namespace search.

This is not possible on Boot. In Boot classpath is managed by Boot and one directory will contain files from several sources, e.g. Boot-cljs and source-paths.

Comment by Dominic Monroe [ 13/Dec/16 12:01 PM ]

Would you accept this patch if `do-refresh` took an option in it's map of `sane-paths-only?` which it passes through it's calls down to `find/find-sources-in-dir`? Where `find-sources-in-dir` took an optional third argument specifying whether `sane-paths-only?`?

It would be nice to refactor the final argument to `find-sources-in-dir` into an options map, but breaking that API is perhaps out of scope.

tl;dr would you accept this patch if `sane-paths-only?` only ran for the repl/refresh, and was passed down via arguments?

Comment by Malcolm Sparks [ 09/Jan/17 9:13 AM ]

I agree that you shouldn't compile artefacts to the classpath.

However, this is extremely widespread practise in the Clojure world, because of uberjars.

Personally, I don't like uberjars, but I seem to be in the minority! As long as people use uberjars in production, they will want to server from the classpath in development.

Hence I think that both the easy solutions Stuart proposes are unsatisfactory.

This is a really annoying bug for anyone who uses libraries that use cljc (e.g. bidi). It would be great if another solution can be proposed.

Comment by Dominic Monroe [ 15/Feb/17 4:22 PM ]

> 2. Use set-refresh-dirs to omit those directories from the tools.namespace search.

Another issue I've noticed with this strategy is that it doesn't handle leiningen profiles with differing `:source-paths` in each. You have to alter the call to `set-refresh-dirs` each time you switch profile (and be confused when you get it wrong).

Comment by Stuart Sierra [ 25/Apr/17 4:40 PM ]

tools.namespace was never intended to be used in deployed applications, so I think it is safe to ignore the case of uberjars for now.

If files in unexpected paths are skipped silently, it would be difficult to understand why a mis-named file is not being loaded. As a possible compromise, c.t.n.find could print a warning to *err* when it encounters a file whose ns declaration does not match its path, and omit it from the search results.

That leaves the problem of spurious error messages. As noted in comments above, Leiningen profiles and Boot make it difficult to statically define the set of directories which *should* be scanned for source files. If we could specify directories to *exclude* from the search, would that be sufficient to work around the problem of ClojureScript copying .cljc files to a resources directory?

Comment by Juho Teperi [ 25/Apr/17 4:51 PM ]

I think uberjars were mentioned in reference to why people serve files from classpath, not because people use c.t.n with deployed applications.

Would the excluded directories be defined as file paths or classpath prefixes?

In Boot the user doesn't control the directories which are included in classpath. Instead Boot creates a few temp directories which it controls and which are included in classpath. Files from several sources (project resource-paths, Cljs artifacts etc.) are all copied into same temp directory, so it is not possible to exclude Cljs artifacts based on that.

Defining classpath prefixes to ignore would probably work with Boot.

Comment by Stuart Sierra [ 30/Apr/17 10:07 PM ]

Excluding by classpath prefix might be possible, but tricky. At the moment, t.n.s.dir searches arbitrary filesystem directories, without knowing that those directories are on the classpath. Still, if exclusions were expressed as classpath-relative directory paths, they could be resolved to real filesystem paths.

Yet another possibility, which should have occurred to me earlier, is using file hashes, rather than timestamps as is done now, to determine when a file is "changed." This would prevent reloads of .cljc files just because the ClojureScript compiler has copied them. On the other hand, it would be a more significant change, and it would prevent the use of touch (or re-saving in an editor) to force a file to be reloaded.

Comment by Stuart Sierra [ 11/Nov/17 9:49 AM ]

New patch TNS-42-3.patch shifts the responsibility up a layer from c.t.n.find to c.t.n.dir. This avoids any breaking to c.t.n.find/find-sources-in-dir while achieving the desired behavior for c.t.n.repl/refresh.

This may be an adequate compromise, although I am still slightly concerned about mis-named files being silently ignored leading to confusion.

Comment by Dominic Monroe [ 13/Nov/17 7:12 AM ]

Discarded files/namespaces could be added to the tracker, to mitigate confusion. Refresh could then warn about those files with `prn` as it does now.

Something would need to remove files from the discarded list if they ever became valid.

I'm not sure this should block TNS-45, but could be something for someone to pick up as follow-up.

Comment by Stuart Sierra [ 31/Dec/17 6:34 PM ]

New patch file TNS-45-4.patch prints a warning and adds the ignored directories to the tracker, following suggestions in comment by Dominic Monroe.

This applies only to c.t.n.dir/scan-dirs, so other uses of the lower-level APIs in tools.namespace should not be affected.

On the first occurrence of a file not matching scan-dirs will print a warning will print a warning to *err* with the the declared ns name and the path to the file that did not match. After that, the entire directory will be ignored, and a notification will be printed on each invocation of scan-dirs. I hope this strikes a good balance between informative messages and not too many spurious warnings.

Note: previous patch file TNS-42-3.patch was mis-named but was always meant to apply to TNS-45.

Comment by Stuart Sierra [ 31/Dec/17 6:36 PM ]

Using c.t.n.repl, an ignored directory can be added back to the scan by calling c.t.n.repl/clear to initialize a new tracker.

Comment by Juho Teperi [ 01/Jan/18 6:40 AM ]


I applied and installed this locally and tested with one Boot projects where I know Cljs output contains some [[.cljc}} namespaces, and this indeed ignores those.

Two notes:

Every time I call refresh, I get the warning about directory being ignored. This does add a bit clutter, but is probably acceptable, as system restart will probably log a few lines anyway in most cases.

One thing that might be a problematic, is that if single file has bad ns declaration at some point, e.g. user accidentally has typo on the name, the whole directory is ignored untill clear is called. This will probably cause confusion. Would it make sense automatically remove ignored directories if the problematic files are fixed? Looks like that would be possible in find-files.

Comment by Juho Teperi [ 01/Jan/18 6:42 AM ]

-5 patch modifies find-files to check ignored directories if all the ns declarations in the directory have been fixed. I had to move the side-effects from predicate functions to find-files so this is not the cleanest change, but should work. I should be able to clean this a bit if this seems like good approach.

[TNS-42] scan-dirs / scan-all can return incorrect dependencies when both clj and cljc files define same namespace Created: 30/Dec/15  Updated: 06/Jan/16

Status: Open
Project: tools.namespace
Component/s: None
Affects Version/s: None
Fix Version/s: None

Type: Defect Priority: Major
Reporter: Andy Fingerhut Assignee: Stuart Sierra
Resolution: Unresolved Votes: 0
Labels: None

tools.namespace 0.2.11 and 0.3.0-alpha2


Originally filed as an issue for function scan-all, which at the time I was unaware was deprecated in 0.3.0 alphas. I have tested against the newer scan-dirs as well, and it has the same problem.

When there are both .clj and .cljc files defining the same namespace, tools.namespace scan-dirs and scan-all can use the dependencies from the .cljc file, ignoring those in the .clj file, whereas for Clojure/Java the opposite should be done.

See sample project and its README here for steps to reproduce: https://github.com/jafingerhut/tnstest

Note that Clojure/Java's behavior is to prefer .clj file over a .cljc file defining the same namespace, even if the .clj file is in a directory that is later in the classpath than a .cljc file. According to Alex Miller email on clojure-dev Google group, this behavior is by design. Link: https://groups.google.com/d/msg/clojure-dev/d5Hb1E7zfHE/sPybIAxgCwAJ

Comment by Stuart Sierra [ 05/Jan/16 11:03 AM ]

I can confirm this is a bug.

When I added multiple file extensions, I didn't consider the priority order.

Fixing this may be tricky.

It starts c.t.n.find. c.t.n.find/find-sources-in-dir filters by file extension (controlled by the platform argument in 0.3) but it only considers one directory at a time. Since the files could be in two different directories, it cannot necessarily discover two files with the same name but different extensions.

The next layer is c.t.n.file, which currently doesn't look at file extensions at all, but maybe it should.

In c.t.n.dir we have enough information to prioritize and filter files for the same namespace by extension. But to do it correctly, it has to handle updates too. For example, what happens to a project that has a .cljc file, then adds a .clj file. c.t.n.dir should treat that as removing the .cljc file from the dependency graph.

I think this will require storing the association between a namespace name and a file name, something I'd been hoping to avoid.

Comment by Stuart Sierra [ 05/Jan/16 11:21 AM ]

It might even make sense to make platform a property of the tracker, to make sure the same rules are always used when adding more files.

Comment by Andy Fingerhut [ 06/Jan/16 10:54 AM ]

Agreed that fixing this is not a small change to tools.namespace, hence the reason I don't have a proposed patch already. Came across this while thinking about how Eastwood could/should handle .cljc files. I'll add any proposed patches here if I come up with something.

[TNS-37] c.t.n.move: does not support to move .cljc files Created: 05/Sep/15  Updated: 20/Dec/15

Status: Open
Project: tools.namespace
Component/s: None
Affects Version/s: None
Fix Version/s: None

Type: Enhancement Priority: Major
Reporter: Benedek Fazekas Assignee: Stuart Sierra
Resolution: Unresolved Votes: 0
Labels: None

Attachments: Text File 0001-c.t.n.move-add-support-for-.cljc-file-extension.patch     Text File 0001-make-c.t.n.move-platform-aware-guess-platform.patch     Text File 0001-make-c.t.n.move-platform-aware.patch    
Patch: Code


it seems that c.t.n.move did not get any reader conditionals love. I use this namespace to power mranderson: a source inlining tool used by cider-nrepl and refactor-nrepl in order to avoid clashes with the code/projects their users work on. lately we started adding dependencies to refactor-nrepl (well, tools.namespace itself) which may use .cljc files. source inlining started failing for these source files.

Comment by Stuart Sierra [ 06/Sep/15 12:57 PM ]

Good idea - glad to see how people are actually using tools.namespace in other dev cools.

Could this be extended further to take 'platform' as a parameter, as in c.t.n.find or c.t.n.dir?

(I deliberately excluded c.t.n.move to limit the scope of changes in 0.3.0, but your patch demonstrates that it may not not be such a difficult change to make.)

Comment by Benedek Fazekas [ 06/Sep/15 3:52 PM ]

first patch does what you asked for I hope. Kept original signiture of move-ns for backward compatibility: clj platform used by default.

second patch is pushing this a bit further: if platform is not provided it tries to guess it based on the file's extension of the old-sym. you might see this as over complication: feel free to pick the first patch then.

Comment by Stuart Sierra [ 06/Nov/15 3:02 PM ]

Thanks for working on these patches. I did some more testing, and uncovered some more complications.

Guessing the file extension leads to unpredictable results, because you don't know which file gets moved. It's not uncommon to have both .clj and .cljs version of the same namespace, for example with :require-macros in ClojureScript.

Unfortunately, even specifying a "platform" argument doesn't completely solve the problem, because you can have .cljc file at the same time as .clj and .cljs. This may not be common, but it's explicitly allowed for in the design of reader conditionals. For example, a library may have a platform-neutral .cljc file and override it with a platform-specific file. In this case, even if you specify a "platform" argument, you still can't control which file gets moved.

So now I think the only way for an operation like "move namespace" to make sense is to move all the files defining that namespace, preserving their file extensions. This seems like what one would want from a user perspective as well.

So move-ns should take new/old symbols, find all the matching files with any extension, move/rename them all while preserving extensions, then apply the textual transformation to all .clj, .cljs, and .cljc files.

If we're searching for files, we might as well eliminate the need to specify the source path, so this also relates to TNS-39.

Comment by Benedek Fazekas [ 20/Dec/15 3:44 PM ]

I have not abandoned this ticket or the patches but it seems it is related to a bigger problem in terms of supporting multiplatform projects therefore related to TNS-38 too.

I am also contemplating the wider context: `move-ns` is not only used in mranderson for source inlining but there is a refactor-nrepl feature (rename file or dir) which basically reimplements this functionality using other parts of TNS.

The mranderson story for me is about exploring different ways of dependency handling really, there was a good lightning talk by Glen Mailer summarising the problem on clojureX this year. My ideas are along creating local, deeply nested dependencies and perhaps enhance the ns macro to be able to load them – not even sure that is feasible. This would eliminate the need for source inlining and more importantly would be much more hygienic approach I think.

Hope this makes sense. Sorry for the brain dump which is obviously exceeds the scope of this ticket.

[TNS-6] Attempt to reload deleted file Created: 14/Dec/12  Updated: 12/Aug/15

Status: Open
Project: tools.namespace
Component/s: None
Affects Version/s: None
Fix Version/s: None

Type: Defect Priority: Major
Reporter: Stuart Sierra Assignee: Stuart Sierra
Resolution: Unresolved Votes: 2
Labels: None


I can't identify the exact circumstances, but I have seen events where a source code file has been deleted but clojure.tools.namespace.repl/refresh still tries to reload it. Because the file doesn't exist, there's an exception when you try to load it, so you're stuck.

Comment by Gary Fredericks [ 12/Aug/15 10:21 AM ]

This happens to me pretty frequently, especially when switching branches (which is ironically the best use case for calling refresh).

Comment by Stuart Sierra [ 12/Aug/15 10:24 AM ]

I still don't know exactly how this occurs.

The workaround for now is to call c.t.n.repl/clear, added in 0.2.5

Comment by Gary Fredericks [ 12/Aug/15 1:26 PM ]

yep, noticed that independently and just confirmed that it works.

Generated at Tue Jan 23 22:19:14 CST 2018 using JIRA 4.4#649-r158309.