Skip to:

  1. Skip to Jira Navigation
  2. Skip to Side Navigation
  3. Skip to Main Content

Duplicate source files passed to Closure Compiler causes ModuleLoader#resolvePaths to throw "Duplicate module path after resolving"

Description

I have a small (not minimal) repo here: https://github.com/au-phiware/boot-parsets

The repo includes a es6 module and a cljs file that both depend on a node module, which produces the following error.

Writing main.cljs.edn... Compiling ClojureScript... • main.js java.lang.Thread.run Thread.java: 748 java.util.concurrent.ThreadPoolExecutor$Worker.run ThreadPoolExecutor.java: 617 java.util.concurrent.ThreadPoolExecutor.runWorker ThreadPoolExecutor.java: 1142 java.util.concurrent.FutureTask.run FutureTask.java: 266 ... clojure.core/binding-conveyor-fn/fn core.clj: 2022 adzerk.boot-cljs/compile-1/fn boot_cljs.clj: 160 adzerk.boot-cljs/compile boot_cljs.clj: 72 boot.pod/call-in* pod.clj: 413 ... org.projectodd.shimdandy.impl.ClojureRuntimeShimImpl.invoke ClojureRuntimeShimImpl.java: 102 org.projectodd.shimdandy.impl.ClojureRuntimeShimImpl.invoke ClojureRuntimeShimImpl.java: 109 ... boot.pod/call-in* pod.clj: 410 boot.pod/eval-fn-call pod.clj: 359 clojure.core/apply core.clj: 657 ... adzerk.boot-cljs.impl/compile-cljs impl.clj: 151 cljs.build.api/build api.clj: 205 cljs.closure/build closure.clj: 2595 cljs.closure/handle-js-modules closure.clj: 2496 cljs.closure/process-js-modules closure.clj: 2389 cljs.closure/convert-js-modules closure.clj: 1680 com.google.javascript.jscomp.Compiler.parse Compiler.java: 995 com.google.javascript.jscomp.Compiler.parseInputs Compiler.java: 1731 com.google.javascript.jscomp.deps.ModuleLoader.<init> ModuleLoader.java: 92 com.google.javascript.jscomp.deps.ModuleLoader.resolvePaths ModuleLoader.java: 276 java.lang.IllegalArgumentException: Duplicate module path after resolving: /home/corin/Projects/Demos/boot-parsets/node_modules/d3/d3.js clojure.lang.ExceptionInfo: Duplicate module path after resolving: /home/corin/Projects/Demos/boot-parsets/node_modules/d3/d3.js from: :boot-cljs clojure.lang.ExceptionInfo: Duplicate module path after resolving: /home/corin/Projects/Demos/boot-parsets/node_modules/d3/d3.js line: 33

Run `boot cljs` to reproduce the issue.

The patch attach removes duplicates from the set of input source files before they are preprocessed. With this patch the repo compiles correctly.

Environment

None

Attachments

2
  • 20 Nov 2017, 02:49 PM
  • 19 Nov 2017, 09:28 AM

Activity

Show:

Mike Fikes May 12, 2019 at 2:46 AM

CLJS-2402.patch added to Patch Tender Info

Mike Fikes November 20, 2017 at 6:08 PM

Thanks Corin. The entire test suite passes for me with your latest patch.

Corin Lawson November 20, 2017 at 3:02 PM

Hi Mike,

I hope this is to your's (and BDFL's) satisfaction now; I ran lein test for both proposed solutions and I do not receive any failures. I do receive errors, however, that do not occur in assertions. I assume that the cause is something peculiar (or lack thereof) in my setup. Let me know if you require anything else from me.

Cheers,
Corin.

Corin Lawson November 20, 2017 at 2:49 PM

Attached proposed Solution B

Corin Lawson November 20, 2017 at 2:47 PM

Steps to reproduce the problem.

Consider the following three source files:

src/distinct_inputs/core.cljs

(ns distinct-inputs.core (:require [d3] [circle :refer [circle]])) (-> d3 (.select "body") (.append "svg") (.attr "width" 200) (.attr "height" 200) (.call circle "steelblue"))

es6/circle.js

import * as d3 from 'd3'; export function circle(sel, color) { return sel .append("circle") .attr("cx", "100px") .attr("cy", "100px") .attr("r", "100px") .attr("fill", color); }

build.clj

(require 'cljs.build.api) (cljs.build.api/build "src" {:main 'distinct-inputs.core :output-to "out/main.js" :install-deps true :foreign-libs [{:file "es6" :module-type :es6}] :npm-deps {:d3 "3.5.16"}})

Execute cljs:

java -cp cljs.jar:src clojure.main build.clj

Expected outcome

cljs should produce nothing to the standard output, exit cleanly and write the following files (approximately).

out/main.js

var CLOSURE_UNCOMPILED_DEFINES = {}; var CLOSURE_NO_DEPS = true; if(typeof goog == "undefined") document.write('<script src="out/goog/base.js"></script>'); document.write('<script src="out/goog/deps.js"></script>'); document.write('<script src="out/cljs_deps.js"></script>'); document.write('<script>if (typeof goog == "undefined") console.warn("ClojureScript could not load :main, did you forget to specify :asset-path?");</script>'); document.write('<script>goog.require("process.env");</script>'); document.write('<script>goog.require("distinct_inputs.core");</script>');

out/distinct_inputs/core.js

goog.provide('distinct_inputs.core'); goog.require('cljs.core'); goog.require('module$distinct_inputs$node_modules$d3$d3'); goog.require('module$distinct_inputs$es6$circle'); module$distinct_inputs$node_modules$d3$d3.select("body").append("svg").attr("width",(200)).attr("height",(200)).call(module$distinct_inputs$es6$circle.circle,"steelblue"); //# sourceMappingURL=core.js.map

out/es6/circle.js

goog.provide("module$distinct_inputs$es6$circle");goog.require("module$distinct_inputs$node_modules$d3$d3");function circle$$module$distinct_inputs$es6$circle(sel,color){return sel.append("circle").attr("cx","100px").attr("cy","100px").attr("r","100px").attr("fill",color)}module$distinct_inputs$es6$circle.circle=circle$$module$distinct_inputs$es6$circle

Actual outcome.

cljs exits with exit code 1 and produces the following standard out.

stdout

Exception in thread "main" java.lang.IllegalArgumentException: Duplicate module path after resolving: /distinct_inputs/node_modules/d3/d3.js, compiling:(/distinct_inputs/build.clj:3:1) at clojure.lang.Compiler.load(Compiler.java:7391) at clojure.lang.Compiler.loadFile(Compiler.java:7317) at clojure.main$load_script.invokeStatic(main.clj:275) at clojure.main$script_opt.invokeStatic(main.clj:335) at clojure.main$script_opt.invoke(main.clj:330) at clojure.main$main.invokeStatic(main.clj:421) at clojure.main$main.doInvoke(main.clj:384) at clojure.lang.RestFn.invoke(RestFn.java:408) at clojure.lang.Var.invoke(Var.java:379) at clojure.lang.AFn.applyToHelper(AFn.java:154) at clojure.lang.Var.applyTo(Var.java:700) at clojure.main.main(main.java:37) Caused by: java.lang.IllegalArgumentException: Duplicate module path after resolving: /distinct_inputs/node_modules/d3/d3.js at com.google.javascript.jscomp.deps.ModuleLoader.resolvePaths(ModuleLoader.java:276) at com.google.javascript.jscomp.deps.ModuleLoader.<init>(ModuleLoader.java:92) at com.google.javascript.jscomp.Compiler.parseInputs(Compiler.java:1731) at com.google.javascript.jscomp.Compiler.parse(Compiler.java:995) at cljs.closure$convert_js_modules.invokeStatic(closure.clj:1680) at cljs.closure$process_js_modules.invokeStatic(closure.clj:2371) at cljs.closure$handle_js_modules.invokeStatic(closure.clj:2495) at cljs.closure$build.invokeStatic(closure.clj:2592) at cljs.build.api$build.invokeStatic(api.clj:204) at cljs.build.api$build.invoke(api.clj:189) at cljs.build.api$build.invokeStatic(api.clj:192) at cljs.build.api$build.invoke(api.clj:189) at user$eval24.invokeStatic(build.clj:3) at user$eval24.invoke(build.clj:3) at clojure.lang.Compiler.eval(Compiler.java:6927) at clojure.lang.Compiler.load(Compiler.java:7379) ... 11 more

None of the aforementioned expected files are produced.

Cause of the exception.

The exception emitted by ModuleLoader#resolvePaths is a result of the same input file (i.e. node_modules/d3/d3.js) having been specified more than once to Compiler#initModules. There happens to be this note in Compiler#getAllInputsFromModules:

src/com/google/javascript/jscomp/Compiler.java

// NOTE(nicksantos): If an input is in more than one module, // it will show up twice in the inputs list, and then we // will get an error down the line.

cljs.closure/process-js-modules is provided a :foreign-libs vector which contains a repeated entry for node_modules/d3/d3.js (and also it's package.json). That vector is a result of multiple invocations of cljs.closure/node-inputs; once for out/cljs$node_modules.js (which is presumably a result of the dependency in distinct_inputs/core) and again for es6/circle.js.

In short, the dependency on D3 is pulled in by both ClojureScript source files and JavaScript module source files.

Proposed solution.

In this scenario the :foreign-libs vector contains repeated entries dispite the use of distinct within cljs.closure/node-inputs. A possible solution would to remove the use of distinct within cljs.closure/node-inputs and require the caller of cljs.closure/node-inputs to use distinct.

Solution A

From 063e35080c14d35189ab7827f25f071e958ab5b4 Mon Sep 17 00:00:00 2001 From: Corin Lawson <corin@responsight.com> Date: Tue, 21 Nov 2017 01:31:53 +1100 Subject: [PATCH] CLJS-2402: Ensure :foreign-libs vector contains distinct entries. --- src/main/clojure/cljs/closure.clj | 19 ++++++++++--------- 1 file changed, 10 insertions(+), 9 deletions(-) diff --git a/src/main/clojure/cljs/closure.clj b/src/main/clojure/cljs/closure.clj index a686f878..74a0cc86 100644 --- a/src/main/clojure/cljs/closure.clj +++ b/src/main/clojure/cljs/closure.clj @@ -2219,7 +2219,7 @@ (when env/*compiler* (:options @env/*compiler*)))) ([entries opts] - (into [] (distinct (mapcat #(node-module-deps % opts) entries))))) + (into [] (mapcat #(node-module-deps % opts) entries)))) (defn index-node-modules ([modules] @@ -2480,14 +2480,15 @@ output-dir (util/output-directory opts) opts (update opts :foreign-libs (fn [libs] - (into (if (= target :nodejs) - [] - (index-node-modules node-required)) - (into expanded-libs - (node-inputs (filter (fn [{:keys [module-type]}] - (and (some? module-type) - (not= module-type :amd))) - expanded-libs)))))) + (distinct + (into (if (= target :nodejs) + [] + (index-node-modules node-required)) + (into expanded-libs + (node-inputs (filter (fn [{:keys [module-type]}] + (and (some? module-type) + (not= module-type :amd))) + expanded-libs))))))) opts (if (some (fn [ijs] (let [dest (io/file output-dir (rel-output-path (assoc ijs :foreign true) opts))] -- 2.13.0

A more general solution is cljs.closure/process-js-modules must ensure the set of input files (i.e. js-modules) is distinct. This patch would be simpler (i.e. doesn't mess with code I don't understand) and closer to the call to Google Closure Compiler.

Solution B

From 6bf11a24b93642e118e6d29c5af8a137fa01ea94 Mon Sep 17 00:00:00 2001 From: Corin Lawson <corin@responsight.com> Date: Sun, 19 Nov 2017 20:25:31 +1100 Subject: [PATCH] CLJS-2402: Ensure input source files are distinct. --- src/main/clojure/cljs/closure.clj | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/src/main/clojure/cljs/closure.clj b/src/main/clojure/cljs/closure.clj index a686f878..24421bde 100644 --- a/src/main/clojure/cljs/closure.clj +++ b/src/main/clojure/cljs/closure.clj @@ -2364,7 +2364,8 @@ (let [;; Modules from both :foreign-libs (compiler options) and :ups-foreign-libs (deps.cljs) ;; are processed together, so that files from both sources can depend on each other. ;; e.g. commonjs module in :foreign-libs can depend on commonjs module from :ups-foreign-libs. - js-modules (filter :module-type (concat (:foreign-libs opts) (:ups-foreign-libs opts)))] + js-modules (filter :module-type (concat (:foreign-libs opts) (:ups-foreign-libs opts))) + js-modules (distinct js-modules)] (if (seq js-modules) (util/measure (:compiler-stats opts) "Process JS modules" -- 2.13.0

FWIW: I prefer Solution B.

Details

Assignee

Reporter

Labels

Patch

Code

Affects versions

Priority

Created November 19, 2017 at 9:28 AM
Updated May 12, 2019 at 2:46 AM

Flag notifications