<< Back to previous view

[MATCH-111] [CLJS]: Usage of core.match within core.async/go block causes compilation to fail when match cases exceed a certain count Created: 08/Aug/15  Updated: 08/Aug/15

Status: Open
Project: core.match
Component/s: None
Affects Version/s: None
Fix Version/s: None

Type: Defect Priority: Major
Reporter: Dennis Heihoff Assignee: David Nolen
Resolution: Unresolved Votes: 0
Labels: bug, compiler
Environment:

Mac OS X Mavericks, JDK 1.8



 Description   

This works in Clojure but fails in ClojureScript.
Reproduced in this repo: https://github.com/den1k/core-match-test

Failure case:

(ns core-match-test.core
(:require-macros [cljs.core.async.macros :refer [go]]
[cljs.core.match :refer [match]]))

;; The first two cases below will compile while the third will not
;; To reproduce run `lein cljsbuild auto
;; after successful compilation comment in last test case and save
;; note console output

;; This will compile
(defn compile-test [vec-to-match]
(match [vec-to-match]
[[:test :1]] (prn vec-to-match)
[[:test :2]] (prn vec-to-match)
[[:test :3]] (prn vec-to-match)
[[:test :4]] (prn vec-to-match)
[[:test :5]] (prn vec-to-match)
[[:test :6]] (prn vec-to-match)
[[:test :7]] (prn vec-to-match)
[[:test :8 arg]] (prn vec-to-match arg)
[[:test :9]] (prn vec-to-match)))

;; This will also compile
(defn compile-test2 [vec-to-match]
(go ;; <= added `go`
(match [vec-to-match]
[[:test :1]] (prn vec-to-match)
[[:test :2]] (prn vec-to-match)
[[:test :3]] (prn vec-to-match)
[[:test :4]] (prn vec-to-match)
[[:test :5]] (prn vec-to-match)
;[[:test :6]] (prn vec-to-match)
[[:test :7]] (prn vec-to-match)
[[:test :8 arg]] (prn vec-to-match arg)
[[:test :9]] (prn vec-to-match))))

;; This will not compile
#_(defn compile-test3 [vec-to-match]
(go
(match [vec-to-match]
[[:test :1]] (prn vec-to-match)
[[:test :2]] (prn vec-to-match)
[[:test :3]] (prn vec-to-match)
[[:test :4]] (prn vec-to-match)
[[:test :5]] (prn vec-to-match)
[[:test :6]] (prn vec-to-match) ;; <= commented in line
[[:test :7]] (prn vec-to-match)
[[:test :8 arg]] (prn vec-to-match arg)
[[:test :9]] (prn vec-to-match)
)))



 Comments   
Comment by Dennis Heihoff [ 08/Aug/15 10:28 AM ]

I could not reproduce this on a different machine, running JDK 1.7. I assume this is a caching issue in the compiler? I will investigate further, this can be closed.

Comment by Dennis Heihoff [ 08/Aug/15 10:38 AM ]

Exception:

Compiling "resources/public/js/compiled/core_match_test.js" failed.
java.lang.StackOverflowError:
AFn.java:148 clojure.lang.AFn.applyToHelper
RestFn.java:132 clojure.lang.RestFn.applyTo
core.clj:630 clojure.core/apply
core.clj:3605 clojure.core/print
RestFn.java:137 clojure.lang.RestFn.applyTo
core.clj:630 clojure.core/apply
core.clj:4581 clojure.core/print-str
RestFn.java:408 clojure.lang.RestFn.invoke
compiler.cljc:175 cljs.compiler$emits.doInvoke
RestFn.java:137 clojure.lang.RestFn.applyTo
core.clj:630 clojure.core/apply
compiler.cljc:173 cljs.compiler$emits.doInvoke
RestFn.java:408 clojure.lang.RestFn.invoke
compiler.cljc:1020 cljs.compiler$eval3142$fn__3144.invoke
MultiFn.java:229 clojure.lang.MultiFn.invoke
compiler.cljc:166 cljs.compiler$emit.invoke
compiler.cljc:172 cljs.compiler$emits.doInvoke
RestFn.java:137 clojure.lang.RestFn.applyTo
core.clj:630 clojure.core/apply
compiler.cljc:183 cljs.compiler$emitln.doInvoke
RestFn.java:408 clojure.lang.RestFn.invoke
compiler.cljc:746 cljs.compiler$eval2946$fn__2948.invoke
MultiFn.java:229 clojure.lang.MultiFn.invoke
compiler.cljc:166 cljs.compiler$emit.invoke
compiler.cljc:172 cljs.compiler$emits.doInvoke
RestFn.java:408 clojure.lang.RestFn.invoke
compiler.cljc:786 cljs.compiler$emit_let$fn__2971.invoke
compiler.cljc:771 cljs.compiler$emit_let.invoke
compiler.cljc:793 cljs.compiler$eval2982$fn__2983.invoke
MultiFn.java:229 clojure.lang.MultiFn.invoke
compiler.cljc:166 cljs.compiler$emit.invoke
compiler.cljc:172 cljs.compiler$emits.doInvoke
RestFn.java:137 clojure.lang.RestFn.applyTo
core.clj:630 clojure.core/apply





[CLJS-1216] varargs not passed properly Created: 21/Apr/15  Updated: 23/Apr/15  Resolved: 23/Apr/15

Status: Closed
Project: ClojureScript
Component/s: None
Affects Version/s: 0.0-3196
Fix Version/s: 0.0-3211

Type: Defect Priority: Critical
Reporter: Karsten Schmidt Assignee: David Nolen
Resolution: Completed Votes: 2
Labels: compiler


 Description   

The following is broken as of 3196 (3165 is still working correctly)

(defn foo
  ([a] (foo a 10))
  ([a b & [c]] [a b c]))

(foo 1)
;; => [1 (10) nil] => should be [1 10 nil]

(foo 1 2)
;; => [1 (2) nil] => should be [1 2 nil]

(foo 1 2 3)
;; => [1 (2 3) nil] => should be [1 2 3]


 Comments   
Comment by Stephen Nelson [ 21/Apr/15 7:18 PM ]

This only occurs with defns that have multiple bodies, it does not occur with fns or with defns with a single body, or when the defn is called via apply rather than directly. Seems like it might be an off-by-one error in the calculation of required arguments in the defn dispatcher function. Test case:

test/dispatch_test.cljs
(ns test.destructuring-test
  (:require [cljs.test :refer-macros [deftest is]]))

(defn destructure
  ([kvs]
   kvs)
  ([k v & args]
   [k v args]))

(deftest test-destructuring
  (is (= [1 2 [3 4]]                                        ;; fails
         (destructure 1 2 3 4)))
  (is (= [1 2 [3 4]]                                        ;; passes
         (apply destructure [1 2 3 4])))
  )

In the compiled javascript the default case of the dispatcher only pulls one argument off before calling the variadic body, whereas the applyTo body pulls two off. Seems like the dispatcher is at fault.

test/dispatch_test.js
...
test.destructuring_test.destructure = (function test$destructuring_test$destructure() {
    var G__48557 = arguments.length;
    switch (G__48557) {
        case 1:
            return test.destructuring_test.destructure.cljs$core$IFn$_invoke$arity$1((arguments[(0)]));

            break;
        default:
            var argseq__17948__auto__ = (new cljs.core.IndexedSeq(Array.prototype.slice.call(arguments, 1), (0)));
            return test.destructuring_test.destructure.cljs$core$IFn$_invoke$arity$variadic((arguments[(0)]), argseq__17948__auto__);

    }
});

test.destructuring_test.destructure.cljs$core$IFn$_invoke$arity$1 = (function (kvs) {
    return kvs;
});

test.destructuring_test.destructure.cljs$core$IFn$_invoke$arity$variadic = (function (k, v, args) {
    return new cljs.core.PersistentVector(null, 3, 5, cljs.core.PersistentVector.EMPTY_NODE, [k, v, args], null);
});

test.destructuring_test.destructure.cljs$lang$applyTo = (function (seq48553) {
    var G__48554 = cljs.core.first.call(null, seq48553);
    var seq48553__$1 = cljs.core.next.call(null, seq48553);
    var G__48555 = cljs.core.first.call(null, seq48553__$1);
    var seq48553__$2 = cljs.core.next.call(null, seq48553__$1);
    return test.destructuring_test.destructure.cljs$core$IFn$_invoke$arity$variadic(G__48554, G__48555, seq48553__$2);
});
...
Comment by Karsten Schmidt [ 22/Apr/15 5:20 AM ]

You're right, Stephen! I've narrowed down the first occurrence, it's starting at r3178 - most likely with the changes done to cljs.core in this commit: https://github.com/clojure/clojurescript/commit/576fb6e054dd50ec458a3c9e4172a5a0002c7aea

will dig more & attempt a patch later today...

Comment by David Nolen [ 23/Apr/15 12:34 AM ]

I have a fix for this, just a simple error around computing the max fixed arity.

Comment by David Nolen [ 23/Apr/15 12:39 AM ]

fixed https://github.com/clojure/clojurescript/commit/c2296b11cc89a81baef1cd6a11567150d48605d9

Comment by David Nolen [ 23/Apr/15 12:51 AM ]

Cut 0.0-3211 with this fix.





[CLJS-1178] Compiler does not know Math ns is not not-native Created: 29/Mar/15  Updated: 29/Apr/15  Resolved: 29/Apr/15

Status: Closed
Project: ClojureScript
Component/s: None
Affects Version/s: None
Fix Version/s: None

Type: Defect Priority: Minor
Reporter: Francis Avila Assignee: Unassigned
Resolution: Completed Votes: 0
Labels: compiler

Attachments: Text File cljs-1178.patch    
Patch: Code

 Description   

With static-fns true, compiler will emit invocations of functions in the Math "namespace" with a conditional arity check or .call() because it does not know that the Math namespace is not not-native. Example with the fix function (advanced compilation, pseudo-names true):

function $cljs$core$fix$$($q$$28$$) {
  return 0 <= $q$$28$$ ? Math.floor.$cljs$core$IFn$_invoke$arity$1$ ? Math.floor.$cljs$core$IFn$_invoke$arity$1$($q$$28$$) : Math.floor.call(null, $q$$28$$) : Math.ceil.$cljs$core$IFn$_invoke$arity$1$ ? Math.ceil.$cljs$core$IFn$_invoke$arity$1$($q$$28$$) : Math.ceil.call(null, $q$$28$$);
}

With attached patch, the following is emitted instead:

function $cljs$core$fix$$($q$$28$$) {
  return 0 <= $q$$28$$ ? Math.floor($q$$28$$) : Math.ceil($q$$28$$);
}


 Comments   
Comment by David Nolen [ 29/Apr/15 7:04 AM ]

fixed https://github.com/clojure/clojurescript/commit/4c50d43592074a72a84ca35eceaf5e43334da49e





[CLJS-1059] Simple interface wanted to convert cljs forms to js Created: 22/Feb/15  Updated: 07/May/15

Status: Open
Project: ClojureScript
Component/s: None
Affects Version/s: 0.0-3211
Fix Version/s: None

Type: Enhancement Priority: Minor
Reporter: Stuart Mitchell Assignee: Unassigned
Resolution: Unresolved Votes: 0
Labels: analyzer, compiler

Patch: Code

 Description   

In our project (a clojurescript debugger) we want to convert cljs forms or a sequence of forms into javascript so that they can be executed in the javascript console.

We would like something similar to closure/compile-form-seq (https://github.com/clojure/clojurescript/blob/master/src/clj/cljs/closure.clj#L308)

However, we need to supply, the namespace requires and locals in an env like this

{:ns {:name "test.core" :requires {(quote gstring) (quote goog.string)}} :locals {}}

This code seems to do what we want.

(defn compile-form-seq
    \"Compile a sequence of forms to a JavaScript source string.\"
    [forms env]
    (env/ensure
    (compiler/with-core-cljs nil
      (fn []
        (with-out-str
            (doseq [form forms]
              (compiler/emit (analyzer/analyze env form))))))))

I am not sure why I need env/ensure.

Would you be able to patch compile-form-seq to provide the needed interface, or suggest what we should be doing.

Thanks
Stu



 Comments   
Comment by Mike Thompson [ 22/Feb/15 10:09 PM ]

Just to be clear:
1. when our debugger is at a breakpoint,
2. the user can type in an expression at the repl
3. in response, our debugger has to compile the user-typed-in expression to javascript (and then execute it, showing a result)
4. taking into account any local bindings. <---- this is the key bit.

To satisfy point 4, our tool extracts all the 'locals' from the current call-frame, and then supplies all these local bindings in env/locals, so the compiler doesn't stick a namespace on the front of them.

For example, if there was a local binding for 'x' in the callstack, and the user's repl-entered-expression involves 'x', then we want the compiler to leave the symbol 'x' alone and to not put some namespace on the front of it. In the final javascript, it must still be 'x', not 'some.namespace.x'

Our method to achieve this is to put 'x' into env/locals when compiling – and it all works. Except, with the recent changes this has become more of a challenge. Hence this ticket asking for a way to pass in env.

Comment by Thomas Heller [ 23/Feb/15 3:19 AM ]

You could wrap the user expression in an fn, that would allow you to skip messing with the locals. The REPL basically does the same trick for *1,*2,...

(fn [x]
  ~user-expression-here)
Comment by David Nolen [ 29/Apr/15 7:21 AM ]

Seems like something useful to add to a cljs.compiler.api namespace.





[CLJS-1036] cljs.closure/build does not find upstream dependencies when called from worker thread Created: 13/Feb/15  Updated: 20/Feb/15  Resolved: 20/Feb/15

Status: Closed
Project: ClojureScript
Component/s: None
Affects Version/s: None
Fix Version/s: None

Type: Defect Priority: Major
Reporter: Michael Griffiths Assignee: Unassigned
Resolution: Completed Votes: 1
Labels: cljs, closure, compiler

Attachments: Text File CLJS-1036--001.patch    

 Description   

Example stacktrace: https://www.refheap.com/97198 (context is using figwheel in a clojure REPL from CIDER)

Because cljs.closure/build calls the 0-arity form of get-upstream-deps, it is implicitly using the current thread's classloader to find the deps.cljs resources. It is then assoc'ing the result into opts, so the caller has no way of gathering the dependencies themselves and passing them in.



 Comments   
Comment by Michael Griffiths [ 13/Feb/15 1:40 PM ]

Patch CLJS-1036--001.patch, attached, uses merge instead of assoc to allow passing of the upstream dependency data via opts.

Comment by Michael Griffiths [ 13/Feb/15 1:45 PM ]

We could also allow passing of a :resources-classloader opt so callers don't have to gather the dependencies themselves, if you'd prefer a patch that does that.

Comment by Bruce Hauman [ 15/Feb/15 2:17 PM ]

This is a problem with calling cljs.closure/build inside nREPL in general when upstream deps are involved.

Comment by David Nolen [ 15/Feb/15 2:34 PM ]

I would like Chas Emerick to chime in on this thread before considering it.

Comment by Chas Emerick [ 16/Feb/15 9:02 AM ]

Could someone point me to a simple (sample?) project that uses upstream dependencies? I haven't used them yet, and don't have such a thing handy.

The proposed patch is confusing to me; if the problem is in resolving dependencies due to classloader state/configuration, how does it help to give opts the opportunity to clobber what get-upstream-deps returns? That is, doesn't this just move the problem downstream to who/whatever is calling build?

Comment by Michael Griffiths [ 16/Feb/15 10:10 AM ]

Chas: yes, the proposed patch simply moves the responsibility to the caller. I was under the assumption that changing how get-upstream-deps/build resolves the classloader by default would be too-breaking of a change, and had not even considered the fact that this might be a broader issue with nREPL itself.

Example usage of get-upstream-deps in the wild (that even mentions classloader issues): https://github.com/immutant/immutant-release/blob/ea538feb548bde86ebce20ec679da7a19b639259/integration-tests/apps/ring/basic-ring/src/basic_ring/core.clj#L51

Note that Weasel (at least) is currently gathering the upstream deps itself too: https://github.com/tomjakubowski/weasel/commit/5cd7009f584100f0d89fc7f00079458bb1f2c016

I'll set up an example project tonight if you like, but I think all you need is to add [cljsjs/react "0.12.2-5"] as a dependency to an existing cljs project and then require [cljsjs.react] in one of its namespaces.

Comment by Bruce Hauman [ 16/Feb/15 1:04 PM ]

Chas, David here is a very quick example:

If you do a

lein new om-mies hello-world
cd hello-world
lein repl
user> (require '[cljs.closure :as cc])
user> (cc/build "src" {})

Assuming lein repl started an nREPL repl. This will produce the error mentioned above. The problem is that om is utilizing the new deps.js facility to autmatically import cljsjs.react. get-upstream-deps depends on being run on the main thread.

https://github.com/clojure/clojurescript/blob/master/src/clj/cljs/closure.clj#L1116

A solution that fixes this in get-upstream-deps* would be preferable.

Is this simple solution to use the system classloader when not on the main thread:

(if (not= (.getName (Thread/currentThread)) "main")
  (java.lang.ClassLoader/getSystemClassLoader)
  (. (Thread/currentThread) (getContextClassLoader)))

To naive? My experience with Java threads and classloaders is minimal.

Or should we always use the (java.lang.ClassLoader/getSystemClassLoader) in this case?

Comment by Chas Emerick [ 17/Feb/15 3:35 PM ]

Using the system classloader isn't necessary. get-upstream-deps is currently using .findResources, not .getResources; the difference is that the former looks only for matching resources in the target classloader, while the latter first delegates to the parent classloader, and only looks locally if nothing is found there. So, just using .getResources instead will make all current usage work as expected, I think. e.g.

;; `lein repl` nREPL here
user=> (enumeration-seq (.getResources (. (Thread/currentThread) (getContextClassLoader))  "deps.cljs"))
(#<URL jar:file:/home/chas/.m2/repository/cljsjs/react/0.12.2-5/react-0.12.2-5.jar!/deps.cljs>)
user=> (enumeration-seq (.findResources (. (Thread/currentThread) (getContextClassLoader))  "deps.cljs"))
nil

There are some classloader edge cases that might motivate making get-upstream-deps perform a more comprehensive search, explicitly pinging all classloaders for matching resources. This is sometimes necessary for these sorts of scanning scenarios if e.g. a deps.cljs file is available via a parent classloader, which will therefore "shadow" other deps.cljs files in child classloaders. In that case, you need something like cemerick.pomegranate/resources.

Hope that helps.

Comment by David Nolen [ 17/Feb/15 3:46 PM ]

Chas's suggested fix has been committed to master. It works on the simple tests I've run but I'd like to hear more confirmation. Thanks everyone.

https://github.com/clojure/clojurescript/commit/79208f5bf15825a2ba2d0ce95aae6d2b71966494

Comment by David Nolen [ 20/Feb/15 4:36 PM ]

Closing this one. Chas's solution is the correct one.





[CLJS-905] Dependency tree fail Created: 11/Dec/14  Updated: 14/Dec/14  Resolved: 12/Dec/14

Status: Closed
Project: ClojureScript
Component/s: None
Affects Version/s: None
Fix Version/s: None

Type: Defect Priority: Major
Reporter: Dusan Maliarik Assignee: Unassigned
Resolution: Declined Votes: 0
Labels: bug, compiler


 Description   

In the following repo

https://github.com/skrat/deporder-bug-cljs

Looking at the compiled source, I would expect the `bug.b` module to be initialized before `bug.a`. It is not, and it leads to a runtime error about `bug.b/registry` being undefined. Now, when I switch things around (`around` branch in the repo), rename `bug.a` to `bug.b` and vice versa, I will get a compiler warning about `bug.a/registry` being undeclared, but at runtime everything works as expected.

This example was extracted from a large project that uses similar macro that expands to `swap!`, where with `none` optimizations everything works, but with `simple` and `advanced` it throws during runtime.



 Comments   
Comment by Thomas Heller [ 11/Dec/14 6:38 AM ]

Technically not a "bug" but a weakness due to the nature ClojureScript handles macros.

'bug.macros requires 'bug.b cause it emits code calling it. 'bug.a requires 'bug.macro and therefore technically requires 'bug.b but since it cannot figure this out you technically have to tell it by also doing a (:require [bug.b]) in 'bug.a.

I had this problem a while back [1] and implemented a workarround by letting the macro declare what namespaces it requires, which in your case would look like

(def macro
  ^{:js-requires '[bug.b]}
  zwap [v]
  `(swap! bug.b/registry conj ~v))

But I was turned down so the feature never made it.

The recommended way is to have macros in namespaces that have a matching CLJS namespace and using :include-macros/:refer-macros when requiring them. Has its own issues cause a) everyone needs to know that bug.b uses macros and b) everyone needs to know what actually is a macro.

Anyways, in your case move bug/macros.clj to bug/b.clj and rewrite bug/a.cljs to

(ns bug.a
  (:require [bug.b :refer-macros (zwap)]))

[1] https://groups.google.com/forum/?fromgroups#!searchin/clojurescript/macro$20require/clojurescript/sLRGCIa8E1c/N9sFcTP_i9wJ

Comment by Thomas Heller [ 11/Dec/14 6:49 AM ]

Shameless plug: shadow-build [1] supports the macro meta feature. Still sort of hack-ish since it only works when macros are referred directly but my use of macros is very limited, basically only whats in [2].

[1] https://github.com/thheller/shadow-build
[2] https://github.com/thheller/shadow/blob/master/src/clj/shadow/macros.clj

Comment by Dusan Maliarik [ 11/Dec/14 8:33 AM ]

I wonder how this works in Clojure land, but I would still say it's a bug in ClojureScript compiler, because, afaik, it first expands macros, which results in a symbol from another namespace ('bug.b), and at that point, 'bug.b should be added as a dependency, resulting in goog.require('bug.b') in the compiled 'bug.a, isn't that right?

Comment by Francis Avila [ 11/Dec/14 9:18 AM ]

In java Clojure you still need to ensure that the symbols you expand in your macros are resolvable by the time the expanded form is executed. For example, if you create a macro that includes a clojure.data/diff call, you need to require clojure.data somewhere in your program, ideally in the namespace that defines the macro. Symbols which refer to other namespaces do not automatically require those namespaces. Most likely you will get a {{ClassNotFoundException [namespace-part]}} when you try to execute the form.

However Clojurescript introduces the extra problem that the macro expansion and execution environments are different. (In Clojure they are the same.) So there are additional constraints not present in Clojure:

1. A separate, completely different namespace dependency tree exists at compile time vs runtime.
2. Because of the Google Closure compiler, the entire runtime dependency tree must be known statically at compile time. (Otherwise some code you need at runtime might not be compiled into the final js.)
3. Because of javascript's dynamism, some symbols may not be known even to the Google Closure compiler. For example, if you have a macro which expands to calls to jQuery code, neither clojure, clojurescript, nor google closure, nor even your browser have a mechanism to "require" jQuery. The code must simply trust that you have done whatever is necessary to ensure jQuery/whatever is resolvable when it finally executes.

Additionally, automatically requiring namespaces from macro expansion is impossible (not a bug in the ClojureScript compiler):
1. The namespaces might not exist (i.e. the symbol is being used as a symbol, rather than as a reference or var).
2. The namespace/symbol might not be resolveble right now. This is clearer in the clojurescript case: is that symbol a reference to a clojure namespace that I should require right now, or is it a reference to a clojurescript namespace which I need to expand code to require, but I can't actually expect it to exist until runtime?
3. The macro might create symbols dynamically, in which case there is no way to know what namespaces the macro requires without executing the macro.

Comment by Dusan Maliarik [ 11/Dec/14 9:36 AM ]

Yes that's clear to me. Let's factor the non-ClojureScript deps like jQuery out, that's of course impossible to get right for everyone. What I propose is:

  1. run the macros
  2. walk through the expanded code
  3. when a symbol from another namespace is found, if this namespace isn't already required in that package, do either
    • add the require for that namespace
    • print out a warning

Currently it doesn't do anything, it just breaks during the runtime.

Comment by Thomas Heller [ 11/Dec/14 9:46 AM ]

I get a Warning?

WARNING: Use of undeclared Var bug.b/registry at line 8 src/bug/a.cljs

Comes down to a tooling problem again (hint: use shadow-build) that the warning is only printed once with lein-cljsbuild.

Comment by Dusan Maliarik [ 12/Dec/14 4:58 AM ]

Still, why does the different naming produce different results?

Comment by David Nolen [ 12/Dec/14 6:22 AM ]

This is not a bug.

Comment by Dusan Maliarik [ 12/Dec/14 7:14 AM ]

The behavior we talked about earlier might not be a bug indeed, but the fact that merely changing namespace names breaks things, sure is a bug. David, I advise to checkout the repo, try for yourself.

Comment by David Nolen [ 12/Dec/14 10:23 AM ]

I looked at the repo there is no compiler bug that I can discern. You just have two different invalid ClojureScript programs. All the salient points have been covered by other commenters.

Comment by Dusan Maliarik [ 12/Dec/14 11:31 AM ]

I won't insist anymore, nuff was said. Still the point about naming choice affecting the compiler output, wasn't addressed (compare 'master' and 'around' branches). "invalid ClojureScript programs" made me laugh though thanks

Comment by Thomas Heller [ 14/Dec/14 5:32 AM ]

Naming has very little to do with it. If you declare no dependency when using the macro (either bug.b or bug.a depending on branch) the analyzer cannot establish its "correct" position in the dependency graph. Therefore it is basically "luck" whether it will end up in the correct position or not. Declare the correct position and it will always be in the correct position.

Pretty sure the around branch will behave exaclty like master when you switch the order of the :require, but again: DO NOT RELY on such undefined behavior. Declare the dependencies!

(ns bug.core
  (:require [bug.b :refer [woot]]
            [bug.a :refer [registry]]
            ))




[CLJS-893] Nodejs target shouldn't insert shebang by default Created: 02/Dec/14  Updated: 02/Dec/14  Resolved: 02/Dec/14

Status: Closed
Project: ClojureScript
Component/s: None
Affects Version/s: None
Fix Version/s: None

Type: Defect Priority: Major
Reporter: Dusan Maliarik Assignee: Unassigned
Resolution: Duplicate Votes: 0
Labels: bug, compiler


 Description   

If people want, they will prepend the shebang themselves, or run the file using node, where-ever it is on PATH. This is not good default behavior, and breaks things when the intention is to compile a node.js module.



 Comments   
Comment by David Nolen [ 02/Dec/14 12:52 PM ]

Whether it was a good or bad default is water under the bridge at this point. Do you have any suggested solutions? :node-module comes to mind for a new target that doesn't append the shebang. Another option would be something to explicitly suppress it.

Comment by Dusan Maliarik [ 02/Dec/14 1:20 PM ]

Sorry, my fault
https://github.com/clojure/clojurescript/commit/00a9b7bde0c8823175560b453a00e7be09ddd250

I just have to :hashbang false. Unfortunately I was unable to find a complete list of supported options to the compiler. Is it documented somewhere?

Comment by David Nolen [ 02/Dec/14 1:29 PM ]

It is not but the wiki is publicly editable - nothing is stopping anyone from creating a page that documents all the knobs and their relationships.





[CLJS-885] read-string result raising a warning Created: 11/Nov/14  Updated: 05/Dec/14  Resolved: 05/Dec/14

Status: Closed
Project: ClojureScript
Component/s: None
Affects Version/s: None
Fix Version/s: None

Type: Defect Priority: Major
Reporter: Stefano Pugnetti Assignee: David Nolen
Resolution: Completed Votes: 0
Labels: Compiler, bug
Environment:

Ubuntu 12.04 (64 bit)
Oracle JDK 1.7.0_72
leiningen 2.5.0
ClojureScript 0.0-2371
lein-cljsbuild 1.0.4-SNAPSHOT



 Description   

The result of read-string is not correctly recognized as a number in the example discussed here:

https://groups.google.com/forum/#!topic/clojurescript/kwJNH2MBbao

Wrapping it in a call to the identity function makes the symptom disappear.



 Comments   
Comment by David Nolen [ 05/Dec/14 1:37 PM ]

fixed https://github.com/clojure/clojurescript/commit/ecc5765d599cadb0d22e2d66635df9eb68c60774





[CLJS-875] Compiler error on :" Created: 22/Oct/14  Updated: 05/Nov/14  Resolved: 05/Nov/14

Status: Closed
Project: ClojureScript
Component/s: None
Affects Version/s: None
Fix Version/s: None

Type: Defect Priority: Major
Reporter: Stuart Mitchell Assignee: David Nolen
Resolution: Completed Votes: 0
Labels: compiler, errormsgs
Environment:

Ubuntu 14.04
org.clojure/clojurescript "0.0-2277"



 Description   

(str "this will break" :"the parser")
This simple form (note the misplaced ':' causes the compiler to barf with the following error
Caused by: clojure.lang.ExceptionInfo: EOF while reading string

no mention of the ':' and in larger blocks of code the unmatched brackets several lines away is flagged

in clojure the error is
RuntimeException Invalid token: : clojure.lang.Util.runtimeException (Util.java:221)
RuntimeException Unmatched delimiter: ) clojure.lang.Util.runtimeException (Util.java:221)

Note it mentions the errant ':'



 Comments   
Comment by Stuart Mitchell [ 22/Oct/14 11:04 PM ]

I would like the error reporting from the compiler improved in this case, so that it mentions that the problem is caused by the ':'.

Comment by Nicola Mometto [ 23/Oct/14 7:53 AM ]

This appears to be an issue caused by tools.reader, I'm working on a fix for it

Comment by Nicola Mometto [ 23/Oct/14 9:17 AM ]

I just pushed a 0.8.10 release of tools.reader that fixes this issue

Comment by David Nolen [ 05/Nov/14 6:33 AM ]

fixed https://github.com/clojure/clojurescript/commit/ecb13f06e766182e1daa7d227def3bad98bd8c49





[CLJS-869] When preamble is not found in source directory, compiler does not report it Created: 03/Oct/14  Updated: 02/Dec/14  Resolved: 08/Oct/14

Status: Closed
Project: ClojureScript
Component/s: None
Affects Version/s: None
Fix Version/s: None

Type: Defect Priority: Minor
Reporter: Timothy Pratley Assignee: Timothy Pratley
Resolution: Completed Votes: 0
Labels: cljsbuild, compiler, patch

Attachments: Text File CLJS-869.patch    

 Description   

file target is not validated
https://github.com/clojure/clojurescript/blob/master/src/clj/cljs/closure.clj#L525-L531
externs might need checking also.

Steps:
in project.clj add :preamble ["nosuchfile"] to the :compiler map

Expected:
"Could not find preamble file X in source path"

Actual:
Stacktrace gives no indication of the problem (without reading the compiler code)::

Compiling "resources/public/js/device_manager.js" failed.
java.lang.IllegalArgumentException: No implementation of method: :make-reader of protocol: #'clojure.java.io/IOFactory found for class: nil
at clojure.core$_cache_protocol_fn.invoke(core_deftype.clj:544)
at clojure.java.io$fn_8628$G8610_8635.invoke(io.clj:69)
at clojure.java.io$reader.doInvoke(io.clj:102)
at clojure.lang.RestFn.invoke(RestFn.java:410)
at clojure.lang.AFn.applyToHelper(AFn.java:154)
at clojure.lang.RestFn.applyTo(RestFn.java:132)
at clojure.core$apply.invoke(core.clj:626)
at clojure.core$slurp.doInvoke(core.clj:6390)
at clojure.lang.RestFn.invoke(RestFn.java:410)
at cljs.closure$preamble_from_paths$fn__3018.invoke(closure.clj:524)
at clojure.core$map$fn__4245.invoke(core.clj:2557)
at clojure.lang.LazySeq.sval(LazySeq.java:40)
at clojure.lang.LazySeq.seq(LazySeq.java:49)
at clojure.lang.RT.seq(RT.java:484)
at clojure.core$seq.invoke(core.clj:133)
at clojure.core$apply.invoke(core.clj:624)
at cljs.closure$preamble_from_paths.invoke(closure.clj:524)
at cljs.closure$make_preamble.invoke(closure.clj:529)
at cljs.closure$optimize.doInvoke(closure.clj:592)
at clojure.lang.RestFn.applyTo(RestFn.java:139)
at clojure.core$apply.invoke(core.clj:626)
at cljs.closure$build.invoke(closure.clj:980)
at cljs.closure$build.invoke(closure.clj:923)
at cljsbuild.compiler$compile_cljs$fn__3200.invoke(compiler.clj:58)
at cljsbuild.compiler$compile_cljs.invoke(compiler.clj:57)
at cljsbuild.compiler$run_compiler.invoke(compiler.clj:159)
at user$eval3326$iter_33443348$fn3349$fn_3361.invoke(form-init6680721970243583147.clj:1)
at user$eval3326$iter_33443348$fn_3349.invoke(form-init6680721970243583147.clj:1)
at clojure.lang.LazySeq.sval(LazySeq.java:40)
at clojure.lang.LazySeq.seq(LazySeq.java:49)
at clojure.lang.RT.seq(RT.java:484)
at clojure.core$seq.invoke(core.clj:133)
at clojure.core$dorun.invoke(core.clj:2855)
at clojure.core$doall.invoke(core.clj:2871)
at user$eval3326.invoke(form-init6680721970243583147.clj:1)
at clojure.lang.Compiler.eval(Compiler.java:6703)
at clojure.lang.Compiler.eval(Compiler.java:6693)
at clojure.lang.Compiler.load(Compiler.java:7130)
at clojure.lang.Compiler.loadFile(Compiler.java:7086)
at clojure.main$load_script.invoke(main.clj:274)
at clojure.main$init_opt.invoke(main.clj:279)
at clojure.main$initialize.invoke(main.clj:307)
at clojure.main$null_opt.invoke(main.clj:342)
at clojure.main$main.doInvoke(main.clj:420)
at clojure.lang.RestFn.invoke(RestFn.java:421)
at clojure.lang.Var.invoke(Var.java:383)
at clojure.lang.AFn.applyToHelper(AFn.java:156)
at clojure.lang.Var.applyTo(Var.java:700)
at clojure.main.main(main.java:37)



 Comments   
Comment by Timothy Pratley [ 05/Oct/14 1:02 PM ]

Reports a warning

Preamble resource file not found: <file1> <file2> <file3>

Comment by David Nolen [ 08/Oct/14 4:59 PM ]

fixed https://github.com/clojure/clojurescript/commit/9fd6bf5bd55421c3d5becacc5230ed661d6fb3c3





[CLJS-812] Recurring from a case statement emits invalid JavaScript Created: 09/Jun/14  Updated: 13/Jun/14  Resolved: 13/Jun/14

Status: Closed
Project: ClojureScript
Component/s: None
Affects Version/s: None
Fix Version/s: None

Type: Defect Priority: Critical
Reporter: Luke VanderHart Assignee: Unassigned
Resolution: Completed Votes: 0
Labels: bug, compiler


 Description   

In 0.2227, compiling the following form produces syntactically invalid JavaScript:

(defn reproduce
  [value]
  (case value
    :a (recur :b)
    :b 0))

Yields:

bug_repro_test.reproduce = (function reproduce(value) {
    while (true) {
        var G__5832 = (((value instanceof cljs.core.Keyword)) ? value.fqn : null);
        var caseval__5833;
        switch (G__5832) {
            case "b":
                caseval__5833 = 0
                break;
            case "a":
                caseval__5833 = {
                    var G__5834 = cljs.core.constant$keyword$67;
                    value = G__5834;
                    continue;
                }

                break;
            default:
                caseval__5833 = (function () {
                    throw (new Error(("No matching clause: " + cljs.core.str.cljs$core$IFn$_invoke$arity$1(value))))
                })()
        }
        return caseval__5833;
        break;
    }
});

When evaluated in any JavaScript environment (including the Google Closure compiler) environment, this yields a syntax error in this code:

caseval__5833 = {
                    var G__5834 = cljs.core.constant$keyword$67;
                    value = G__5834;
                    continue;
                }


 Comments   
Comment by David Nolen [ 10/Jun/14 7:56 AM ]

Good catch. I suspect that throw may be similarly problematic.

Comment by David Nolen [ 13/Jun/14 3:59 PM ]

fixed in master https://github.com/clojure/clojurescript/commit/cc11c7996ba8522d6767fb45df2f76e20e4c1773





[CLJS-748] Dump analyzer state during a compilation. Created: 13/Jan/14  Updated: 03/Jan/15  Resolved: 03/Jan/15

Status: Closed
Project: ClojureScript
Component/s: None
Affects Version/s: None
Fix Version/s: None

Type: Enhancement Priority: Minor
Reporter: Gary Trakhman Assignee: Unassigned
Resolution: Completed Votes: 0
Labels: Compiler

Attachments: Text File CLJS_748.patch    
Patch: Code

 Description   

CLJS doesn't have the luxury of a fully-reified runtime environment, therefore effective tooling needs a view of the internal compiler state. This issue addresses this need by enabling a dump of the compiler state based on additional cljs compiler options.

The compiler state is a map contained within an atom, either passed in as the 'compiler-env' arg to cljs.closure/build or contained globally within cljs.env/compiler.

The prototype is implemented as such:

:dump-analysis-to
either a string or :print, when :print, output will be written to out, when a string, the argument will be passed to clojure.java.io/writer.

:dump-analysis-full
checked for truthiness, default is false
When this is switched on, the full structure of the compiler state will be printed, which is impractically large for most use cases. In normal operation, :env keys and extended :method-params will be removed from the output, making the analysis represent simply the globally reachable environment. Additionally, only the contents under :cljs.analyzer/namespaces and :cljs.analyzer/constants will be printed.



 Comments   
Comment by Gary Trakhman [ 13/Jan/14 6:46 PM ]

Added implementing patch

Comment by David Nolen [ 16/Jan/14 5:21 PM ]

I question the utility of :dump-analysis-full for now. Lets remove it.

Comment by Gary Trakhman [ 16/Jan/14 5:32 PM ]

I was thinking it might be good along with :print followed by a pipe, but... I don't have an explicit use-case. I'll create a new patch.

Comment by David Nolen [ 03/Jan/15 6:13 PM ]

This ticket is subsumed by :cache-analysis support





[CLJS-740] undeclared-ns warnings are broken Created: 02/Jan/14  Updated: 17/Jan/14  Resolved: 17/Jan/14

Status: Closed
Project: ClojureScript
Component/s: None
Affects Version/s: None
Fix Version/s: None

Type: Defect Priority: Minor
Reporter: Erik Ouchterlony Assignee: Unassigned
Resolution: Completed Votes: 0
Labels: Compiler, bug, patch,

Attachments: File fix-undeclared-ns-warnings.diff    
Patch: Code

 Description   

In the recent versions of cljs, the undeclared-ns warnings have stopped working. This patch seems to be the culprit.



 Comments   
Comment by David Nolen [ 02/Jan/14 12:58 PM ]

Great thanks!

Comment by David Nolen [ 07/Jan/14 10:32 AM ]

CLJS-615

Comment by David Nolen [ 17/Jan/14 9:26 AM ]

fixed, https://github.com/clojure/clojurescript/commit/b8cf302b1500e88e0602e72fa6aec6f7328a1a00





[CLJS-697] top-level symbol reference doesn't get an automatically inserted ns-name Created: 23/Nov/13  Updated: 23/Nov/13  Resolved: 23/Nov/13

Status: Closed
Project: ClojureScript
Component/s: None
Affects Version/s: None
Fix Version/s: None

Type: Defect Priority: Major
Reporter: Limbo Peng Assignee: Unassigned
Resolution: Completed Votes: 0
Labels: Compiler, bug, namespace
Environment:

org.clojure/clojurescript "0.0-2030"



 Description   

I'm trying to use a Node.js module (with nested namespaces) in ClojureScript - the code goes like this:

(ns myapp)
(def evernote (js/require "evernote"))
(def token "TOKEN")
(defn do-sth []
  (let [client (evernote.Evernote.Client. (js-obj "token" token))]
    (.log js/console client)))
(do-sth)

which gets compiled (with :simple optimization) to:

var myapp = {}
myapp.evernote = require("evernote")
myapp.token = "TOKEN"
myapp.do_sth = function() {
  var a = new evernote.Evernote.Client({token:myapp.token})
  return console.log(a)
}
myapp.do_sth()

which will obviously fail with error "Uncaught ReferenceError: evernote is not defined".



 Comments   
Comment by David Nolen [ 23/Nov/13 11:55 PM ]

fixed, https://github.com/clojure/clojurescript/commit/d4bf88269e1d96468a19fd481f32628d4eafec9d





[CLJS-685] Cannot call method 'fromArray' of undefined -- Clojurescript 0.0-2030 Created: 17/Nov/13  Updated: 26/Nov/13  Resolved: 22/Nov/13

Status: Closed
Project: ClojureScript
Component/s: None
Affects Version/s: None
Fix Version/s: None

Type: Defect Priority: Minor
Reporter: John Chijioke Assignee: Unassigned
Resolution: Completed Votes: 0
Labels: Compiler, bug, errormsgs
Environment:

Linux 3.2.0-52-generic x86_64 GNU/Linux, java 1.7, clojure 1.5.1



 Description   

Clojurescript 0.0-2030

This line from compile cljs.core is causing problems:

cljs.core.PersistentQueue.EMPTY = (new cljs.core.PersistentQueue(null, 0, null, cljs.core.with_meta(cljs.core.PersistentVector.EMPTY, cljs.core.PersistentArrayMap.fromArray([new cljs.core.Keyword(null, "end-line", "end-line", 2693041432), 3820, new cljs.core.Keyword(null, "end-column", "end-column", 3799845882), 69], true)), 0));

error message: Uncaught TypeError: Cannot call method 'fromArray' of undefined.

That's the first mention of fromArray in that file. I don't know if it's an ordering problem.



 Comments   
Comment by John Chijioke [ 17/Nov/13 11:10 PM ]

I solved it by replacing [] with cljs.core.PersistentVector.EMPTY. I think this must be a reader problem.

Comment by David Nolen [ 17/Nov/13 11:32 PM ]

This ticket needs more details, how can this error be reproduced?

Comment by Peter Taoussanis [ 22/Nov/13 3:03 AM ]

Hi, I'm seeing the same problem with tools.reader 0.8.0.

Any Clojurescript file (even an empty file) will produce the error.

Clojure: 1.6.0-alpha2
Clojurescript: 0.0-2030
Cljsbuild: 1.0.0
tools.reader: 0.8.0

Tried `lein cljsbuild clean`.

Problem is resolved by dropping back to tools.reader 0.7.10.

Update: have created an issue on the tools.reader GitHub page: https://github.com/clojure/tools.reader/issues/7

Update 2: this isn't something specific to Cljs 0.0-2030 btw, tools.reader 0.8.0 seems to produce the same error against at least Cljs 0.0-2060, 0.0-2027, 0.0-2024.

Comment by Nicola Mometto [ 22/Nov/13 6:49 AM ]

tools.reader 0.8.0 introduces end-column/end-line metadata, this needs to be elided as per line/column to avoid this bootstrapping issue.

Comment by David Nolen [ 22/Nov/13 8:02 AM ]

fixed, http://github.com/clojure/clojurescript/commit/36d401797f85c99794eef8a71239641930c36871

Comment by Peter Taoussanis [ 22/Nov/13 10:30 AM ]

Thanks a lot David, Nicola - much appreciated! Cheers

Comment by John Chijioke [ 26/Nov/13 6:32 AM ]

Thanks David. Cheers!





[CLJS-575] cljsc.bat emit FileNotFoundException when compile samples in windows Created: 25/Aug/13  Updated: 19/Jun/14

Status: Open
Project: ClojureScript
Component/s: None
Affects Version/s: None
Fix Version/s: None

Type: Defect Priority: Major
Reporter: Park Sang Kyu Assignee: Unassigned
Resolution: Unresolved Votes: 0
Labels: Compiler, bug, patch
Environment:

in windows 7


Attachments: File cljsc.bat.diff     File cljsc-path.bat    
Patch: Code

 Description   

cljsc.bat emit FileNotFoundException when it compile samples of the ClojureScript project in windows like below.

------------------------------------------------
Exception in thread "main" java.io.FileNotFoundException: Could not locate cljs/closure__init.class
or cljs/closure.clj on classpath:
------------------------------------------------

It is caused by lack of a backslash in the end of path of the system environment variable, %CLOJURESCRIPT_HOME% set by a user.
In the case CLASSPATH is set to "C:\\clojure\clojurescriptsrc\clj;C:\\clojure\clojurescriptsrc\cljs" and this make it impossible for javac to find cljs/clojure.clj file.

So it can be solved by adding a backslash to the path of %CLOJURESCRIPT_HOME%.

I attached the patched file, "cljsc-path.bat"



 Comments   
Comment by David Nolen [ 04/Sep/13 11:04 PM ]

Can we please get a proper git diff thanks (and please send in your CA)! Also would be nice to get Windows users to check this out.

Comment by Park Sang Kyu [ 15/Sep/13 3:16 AM ]

git diff

Comment by David Nolen [ 05/Oct/13 11:55 AM ]

Thank you! Have you sent in your CA? http://clojure.org/contributing

Comment by Park Sang Kyu [ 19/Jun/14 10:24 AM ]

Yes i have sent my CA.

Comment by David Nolen [ 19/Jun/14 10:27 AM ]

Excellent, the patch is not correctly formatted. Can we get a new patch that conforms to http://github.com/clojure/clojurescript/wiki/Patches





[CLJS-573] checkboxes 'indeterminate' attribute gets munged in advanced mode Created: 19/Aug/13  Updated: 21/Aug/13  Resolved: 21/Aug/13

Status: Closed
Project: ClojureScript
Component/s: None
Affects Version/s: None
Fix Version/s: None

Type: Defect Priority: Major
Reporter: Xavi Caballé Assignee: Unassigned
Resolution: Declined Votes: 0
Labels: Compiler, bug
Environment:

ClojureScript 0.0-1853, Clojure 1.5.1, Java 1.6.0_51, OS X 10.7.5



 Description   

HTML5 defines an indeterminate attribute for checkboxes
http://www.w3.org/TR/html5/forms.html#checkbox-state-(type=checkbox)
Trying to set this attribute with ClojureScript
(set! (.-indeterminate checkbox-element) true))
doesn't work when compiling in advanced mode.

Looking at the generated JavaScript code, I see that the indeterminate property gets munged. I don't know which externs files the ClojureScript compiler uses by default, but the indeterminate property is not referenced in Closure Compiler's externs for HTML5
https://code.google.com/p/closure-compiler/source/browse/externs/html5.js

In my project I've worked around this by adding this to my own externs file

/** @type {boolean} */
HTMLInputElement.prototype.indeterminate;

but I guess this should be "built-in" into the ClojureScript compiler (or Google's Closure Compiler)?



 Comments   
Comment by Jozef Wagner [ 19/Aug/13 6:26 AM ]

This issue should be addressed in the Closure Compiler bugtracker

Until they fix it, you can use your own externs file (tutorial), which is a perfectly valid approach.

Comment by Xavi Caballé [ 21/Aug/13 5:09 AM ]

ok, thanks.
I just submitted a patch to fix this in the Closure Compiler
https://code.google.com/p/closure-compiler/issues/detail?id=1068





[CLJS-476] Reading a value from a module does not work if the module is def'ed Created: 22/Feb/13  Updated: 19/Nov/13  Resolved: 19/Nov/13

Status: Closed
Project: ClojureScript
Component/s: None
Affects Version/s: None
Fix Version/s: None

Type: Defect Priority: Minor
Reporter: Paul Gearon Assignee: Unassigned
Resolution: Declined Votes: 0
Labels: Compiler, bug, scope
Environment:

Clojure 1.5.0-RC16
Clojurescript 0.0-1586
java version "1.7.0_04"
Java(TM) SE Runtime Environment (build 1.7.0_04-b21)
Java HotSpot(TM) 64-Bit Server VM (build 23.0-b21, mixed mode)
OSX Mountain Lion 10.8.2



 Description   

Referring to a value in a module can have a scoping issue when using the "static accessor" operator of module/VALUE_NAME. The static accessor works if the module is loaded into a local value but not if def'ed. This example uses the mmap module for Node.js, and successfully reads the PROT_READ value:

(ns stat.core)
(defn -main []
  (let [m (js/require "mmap")]
    (println "value: " m/PROT_READ)))
(set! *main-cli-fn* -main)

This correctly prints "value: 1"

However, if the value for m is def'ed instead, then the compiler assumes that the reference to m is local to the function and therefore not defined:

(ns stat.core)
(def m (js/require "mmap"))
(defn -main []
  (println "value: " m/PROT_READ))
(set! *main-cli-fn* -main)

/Users/pag/src/test/clj/stat/target/stat.js:12836
  return cljs.core.println.call(null, "value: ", m.PROT_READ)
                                                 ^
ReferenceError: m is not defined
    at Function.stat.core._main (/Users/pag/src/test/clj/stat/target/stat.js:12836:50)
    at cljs.core.apply.b (/Users/pag/src/test/clj/stat/target/stat.js:5621:14)
    at cljs.core.apply.a (/Users/pag/src/test/clj/stat/target/stat.js:5656:18)
    at Object.<anonymous> (/Users/pag/src/test/clj/stat/target/stat.js:12844:17)
    at Module._compile (module.js:449:26)
    at Object.Module._extensions..js (module.js:467:10)
    at Module.load (module.js:356:32)
    at Function.Module._load (module.js:312:12)
    at Module.runMain (module.js:492:10)
    at process.startup.processNextTick.process._tickCallback (node.js:244:9)

In this case, the value of m.PROT_READ should have been stat.core.m.PROT_READ.

On the other hand, using . syntax fixes the scoping issue:

(ns stat.core)
(def m (js/require "mmap"))
(defn -main []
  (println "value: " (.-PROT_READ m)))
(set! *main-cli-fn* -main)


 Comments   
Comment by David Nolen [ 19/Nov/13 9:26 PM ]

the slash syntax is reserved for real CLJS namespaces for everything else the dot syntax must be used.





[CLJS-475] Node.js target fails with optimizations set to :none or :whitespace Created: 21/Feb/13  Updated: 02/Dec/14  Resolved: 02/Dec/14

Status: Closed
Project: ClojureScript
Component/s: None
Affects Version/s: None
Fix Version/s: None

Type: Defect Priority: Minor
Reporter: Paul Gearon Assignee: Unassigned
Resolution: Completed Votes: 0
Labels: Compiler, bug
Environment:

Clojure 1.5.0-RC16
Clojurescript 0.0-1586
java version "1.7.0_04"
Java(TM) SE Runtime Environment (build 1.7.0_04-b21)
Java HotSpot(TM) 64-Bit Server VM (build 23.0-b21, mixed mode)
OSX Mountain Lion 10.8.2


Attachments: GZip Archive out-none.tar.gz     GZip Archive out-whitespace.tar.gz     File pr.js-none     File pr.js-whitespace    

 Description   

Compiling a hello world program for Node.js works fine if using optimizations of :advanced or :simple, but if using :none or :whitespace then an error will be reported for either "goog undefined" or "goog.string" undefined respectively.

The program is shown here:

(ns pr.core)
(defn -main []
  (println "Hello World!"))
(set! *main-cli-fn* -main)

This program is in src/cljs/pr/core.cljs. The repl line used to compile is:

(cljs.closure/build "src/cljs" {:output-to "src/js/pr.js" :target :nodejs :pretty-print true :optimizations :none})

When compiled with optimizations of :none, the output is:

$ node src/js/pr.js 

/Users/pag/src/test/clj/pr/src/js/pr.js:1
(function (exports, require, module, __filename, __dirname) { goog.addDependen
                                                              ^
ReferenceError: goog is not defined
    at Object.<anonymous> (/Users/pag/src/test/clj/pr/src/js/pr.js:1:63)
    at Module._compile (module.js:449:26)
    at Object.Module._extensions..js (module.js:467:10)
    at Module.load (module.js:356:32)
    at Function.Module._load (module.js:312:12)
    at Module.runMain (module.js:492:10)
    at process.startup.processNextTick.process._tickCallback (node.js:244:9)

When running with optimizations of :whitespace the output is:

$ node src/js/pr.js 

/Users/pag/src/test/clj/pr/src/js/pr.js:493
goog.string.Unicode = {NBSP:"\u00a0"};
                    ^
TypeError: Cannot set property 'Unicode' of undefined
    at Object.<anonymous> (/Users/pag/src/test/clj/pr/src/js/pr.js:493:21)
    at Module._compile (module.js:449:26)
    at Object.Module._extensions..js (module.js:467:10)
    at Module.load (module.js:356:32)
    at Function.Module._load (module.js:312:12)
    at Module.runMain (module.js:492:10)

When running with optimizations of either :simple or :advanced, the output is:

$ node src/js/pr.js 
Hello World!

I have included the two javascript output files that match the above errors.



 Comments   
Comment by Paul Gearon [ 21/Feb/13 4:40 PM ]

Remaining generated files

Comment by David Nolen [ 25/Feb/13 3:46 PM ]

This is a known bug. We need goog.require/provide to actually mean something to Node.js. I'm not sure how this can be made to work. I've been hoping for a patch for this since ClojureScript was first announced, but I haven't seen anything yet.

Comment by David Nolen [ 02/Dec/14 6:20 AM ]

long ago fixed in master.





[CLJS-100] binding breaks in macros Created: 11/Nov/11  Updated: 27/Jul/13  Resolved: 10/Jan/12

Status: Closed
Project: ClojureScript
Component/s: None
Affects Version/s: None
Fix Version/s: None

Type: Defect Priority: Major
Reporter: Heinz N. Gies Assignee: Unassigned
Resolution: Duplicate Votes: 0
Labels: Compiler
Environment:

unimportant



 Description   

When creating a macro that results in a binding the name of the global bound variables resolves into namespace bound names - this breaks the assert in clojurescripts let code.

example (simplified):
(ns clgl.macros)
(defmacro with-gl [[gl prog] & body]
`(binding [*gl* ~gl *prog* ~brog] ~@body))

(ns clgl.core
(:require-macros [clgl.macros :as m]))
...
(with-gl [gl prog]
(do-something))



 Comments   
Comment by David Nolen [ 20/Nov/11 1:18 PM ]

I'm not sure I follow. What does gl and prog get namespaced to?

Comment by Jozef Wagner [ 09/Jan/12 2:46 PM ]

I think he has a problem with

*gl*
resolving into
clgl.macros/*gl*
when expanding macro, and this produces error in clojurescript (no such ns exists in cljs).

Solution is to use

~'*gl*
instead of
*gl*
inside macro.

So IMO this issue is not a CLJS defect and should be closed.

Comment by David Nolen [ 10/Jan/12 11:20 PM ]

http://dev.clojure.org/jira/browse/CLJS-114





[CLJ-1793] Clear 'this' before calls in tail position Created: 05/Aug/15  Updated: 24/Aug/15

Status: Open
Project: Clojure
Component/s: None
Affects Version/s: Release 1.8
Fix Version/s: None

Type: Defect Priority: Critical
Reporter: Alex Miller Assignee: Alex Miller
Resolution: Unresolved Votes: 0
Labels: compiler
Environment:

1.8.0-alpha2 - 1.8.0-alpha4


Attachments: Text File 0001-Clear-this-before-calls-in-tail-position.patch     Text File clj-1793-2.patch    
Patch: Code and Test
Approval: Vetted

 Description   

(This ticket started life as CLJ-1250, was committed in 1.8.0-alpha2, pulled out after alpha4, and this is the new version that fixes the logic about whether in a tail call as well as addresses direct linking added in 1.8.0-alpha3.)

Problem: Original example was with reducers holding onto the head of a lazy seq:

(time (reduce + 0 (map identity (range 1e8))))    ;; works
(time (reduce + 0 (r/map identity (range 1e8))))  ;; oome from holding head of range

Example of regression from CLJ-1250 that now works correctly (doesn't clear this in nested loop) with CLJ-1793:

(let [done (atom false)
        f (future-call
            (fn inner []
              (while (not @done)
                (loop [found []]
                  (println (conj found 1))))))]
    (doseq [elem [:a :b :c :done]]
      (println "queue write " elem))
    (reset! done true)
    @f)

Approach: When invoking a method in a tail call, clear this prior to invoking.

The criteria for when a tail call is a safe point to clear 'this':

1) Must be in return position
2) Not in a try block (might need 'this' during catch/finally)
3) When not direct linked

Return position (#1) isn't simply (context == C.RETURN) because loop bodies are always parsed in C.RETURN context

A new dynvar METHOD_RETURN_CONTEXT tracks whether an InvokeExpr in tail position can directly leave the body of the compiled java method. It is set to RT.T in the outermost parsing of a method body and invalidated (set to null) when a loop body is being parsed where the context for the loop expression is not RETURN parsed. Added clear in StaticInvokeExpr as that is now a thing with direct linking again.

Removes calls to emitClearLocals(), which were a no-op.

Patch: clj-1793-2.patch

Screened by: Alex Miller



 Comments   
Comment by Alex Miller [ 05/Aug/15 12:16 PM ]

The this ref is cleared prior to the println, but the next time through the while loop it needs the this ref to look up the closed over done field (via getfield).

Adding an additional check to the inTailCall() method to not include tail call in a loop addresses this case:

static boolean inTailCall(C context) {
-    return (context == C.RETURN) && (IN_TRY_BLOCK.deref() == null);
+    return (context == C.RETURN) && (IN_TRY_BLOCK.deref() == null) && (LOOP_LOCALS.deref() == null);
}

But want to check some more things before concluding that's all that's needed.

Comment by Alex Miller [ 05/Aug/15 1:36 PM ]

This change undoes the desired behavior in the original CLJ-1250 (new tests don't pass). For now, we are reverting the CLJ-1250 patch in master.

Comment by Ghadi Shayban [ 05/Aug/15 3:12 PM ]

Loop exit edges are erroneously being identified as places to clear 'this'. Only exits in the function itself or the outermost loop are safe places to clear.

Comment by Ghadi Shayban [ 05/Aug/15 8:43 PM ]

Patch addresses this bug and the regression in CLJ-1250.

See the commit message for an extensive-ish comment.

Comment by Alex Miller [ 18/Aug/15 12:33 PM ]

New patch is same as old, just adds jira id to beginning of commit message.

Comment by Rich Hickey [ 24/Aug/15 10:00 AM ]

Not doing this for 1.8, more thought needs to go into whether this is the right solution to the problem. And, what is the problem? This title of this patch is just something to do.

Comment by Alex Miller [ 24/Aug/15 10:21 AM ]

changing to vetted so this is at a valid place in the jira workflow

Comment by Ghadi Shayban [ 24/Aug/15 10:45 AM ]

Rich the original context is in CLJ-1250 which was a defect/problem. It was merged and revert because of a problem in the impl. This ticket is the continuation of the previous one, but unfortunately the title lost the context and became approach-oriented and not problem-oriented. Blame Alex. (I kid, it's an artifact of the mutable approach to issue management.)





[CLJ-1786] Unused defaults are evaluted in `:or` destructoring Created: 26/Jul/15  Updated: 26/Jul/15  Resolved: 26/Jul/15

Status: Closed
Project: Clojure
Component/s: None
Affects Version/s: Release 1.7
Fix Version/s: None

Type: Defect Priority: Major
Reporter: Leon Grapenthin Assignee: Unassigned
Resolution: Duplicate Votes: 0
Labels: compiler


 Description   
(defn example [{:keys [foo] :or {foo (println "Evaluated!")}}])

(example {:foo 42})
;; prints "Evaluated!"


 Comments   
Comment by Alex Miller [ 26/Jul/15 8:38 AM ]

Dupe of CLJ-1676





[CLJ-1776] Test that collections are valid statements Created: 08/Jul/15  Updated: 08/Jul/15

Status: Open
Project: Clojure
Component/s: None
Affects Version/s: None
Fix Version/s: None

Type: Enhancement Priority: Minor
Reporter: Michael Blume Assignee: Michael Blume
Resolution: Unresolved Votes: 0
Labels: compiler, test

Attachments: Text File clj-1776-v1.patch    
Patch: Code and Test

 Description   

It's possible to break the compiler such that vectors are emitted incorrectly when they're in statement position (I accidentally did this). This doesn't break any part of the Clojure test suite, but does break valid Clojure code (for me it hit taoensso's encore). Add tests to the test suite so defects of this kind are caught.






[CLJ-1759] macroexpand throws runtime exception on symbol bound to a class Created: 17/Jun/15  Updated: 18/Jun/15

Status: Open
Project: Clojure
Component/s: None
Affects Version/s: Release 1.6
Fix Version/s: None

Type: Defect Priority: Minor
Reporter: W. David Jarvis Assignee: Unassigned
Resolution: Unresolved Votes: 0
Labels: compiler
Environment:

OSX 10.10.3, Leiningen 2.5.1, Java 1.8.0_45 64-bit.


Attachments: Text File clj-1759.patch    
Patch: Code
Approval: Triaged

 Description   

The use of macroexpand on short class name symbols triggers a RuntimeException.

user=> (import 'java.net.URI)
java.net.URI
user=> (macroexpand '(java.net.URI "http://google.com")) ;; fine
(java.net.URI "http://google.com")
user=> (macroexpand '(URI "http://google.com")) ;; huh?
java.lang.RuntimeException: Expecting var, but URI is mapped to class java.net.URI
user=> (pst *e)
RuntimeException Expecting var, but URI is mapped to class java.net.URI
	clojure.lang.Util.runtimeException (Util.java:221)
	clojure.lang.Compiler.lookupVar (Compiler.java:7092)
	clojure.lang.Compiler.isMacro (Compiler.java:6571)
	clojure.lang.Compiler.macroexpand1 (Compiler.java:6626)
	clojure.core/macroexpand-1 (core.clj:3870)
	clojure.core/macroexpand (core.clj:3879)

Neither of these should throw an error during macroexpansion (basically should be same after expansion. Both should throw the same error when evaluated (ClassCast trying to invoke a Class as an IFn).

Approach: Throw the runtime error in lookupVar only if internNew is true. In that case we unexpectedly found something other than a var and should still report. Otherwise, just let lookupVar flow through to return a null (no var found.



 Comments   
Comment by Alex Miller [ 18/Jun/15 6:19 AM ]

The compiler is trying to determine if the thing in function position is a var that is a macro that requires expansion in Compiler.isMacro().

In the case of (java.net.URI "http://google.com"), lookupVar determines that java.net.URI is an unmapped symbol and does nothing, meaning no expansion is necessary (this of course will fail at evaluation time with "ClassCastException java.lang.Class cannot be cast to clojure.lang.IFn").

In the case of (URI "http://google.com"), lookupVar finds a symbol mapped to something that's not a var and throws the RuntimeException that is seen.

I would expect that neither of these should throw an error during macroexpansion (basically the same thing they start as) and that both should throw the same error when evaluated. Attaching a patch that will only throw the error if internNew - in that case you unexpectedly found something other than a var and you should still report (otherwise, just return null - lookupVar didn't find a var).

The internNew case comes up with:

(import java.net.URI) 
(def URI "abc") ;; java.lang.RuntimeException: Expecting var, but URI is mapped to class java.net.URI

with the patch:

user=> (macroexpand '(java.net.URI "http://google.com")) 
(java.net.URI "http://google.com") 
user=> (macroexpand '(URI "http://google.com")) 
(URI "http://google.com") 
user=> (java.net.URI "http://google.com") 
ClassCastException java.lang.Class cannot be cast to clojure.lang.IFn user/eval9 (NO_SOURCE_FILE:6) 
user=> (URI "http://google.com") 
ClassCastException java.lang.Class cannot be cast to clojure.lang.IFn user/eval11 (NO_SOURCE_FILE:7)




[CLJ-1745] Some compiler exceptions wrapped in new CompilerException Created: 04/Jun/15  Updated: 17/Jun/15  Resolved: 17/Jun/15

Status: Closed
Project: Clojure
Component/s: None
Affects Version/s: Release 1.7
Fix Version/s: Release 1.7

Type: Defect Priority: Minor
Reporter: Andy Fingerhut Assignee: Unassigned
Resolution: Completed Votes: 0
Labels: compiler, error-reporting, regression

Attachments: Text File clj-1745.patch    
Patch: Code and Test
Approval: Ok

 Description   

Clojure error reporting changed in CLJ-1169 to wrap exceptions thrown during macro evaluation in CompilerException to give more input:

Clojure 1.6

(defmacro demo [] (throw (ex-info "fail" {})))
(demo)
ExceptionInfo fail  clojure.core/ex-info (core.clj:4403)
(class *e)
=> clojure.lang.ExceptionInfo

Clojure 1.7.0-alpha2 to 1.7.0-RC1

(defmacro demo [] (throw (ex-info "fail" {})))
(demo)
ExceptionInfo fail  clojure.core/ex-info (core.clj:4403)
;; NOTE: lein repl will instead print: CompilerException clojure.lang.ExceptionInfo: fail {}, compiling:(form-init8304622754337237403.clj:1:1)
(class *e)
=> clojure.lang.Compiler$CompilerException

This change has caused some breakage for users that throw exceptions in macros and expect to see the same exception at the top of the exception chain, not wrapped in a CompilerException. This change is somewhat masked in the Clojure REPL because clojure.main/root-cause unwraps CompilerException wrappers and prints the root cause when an exception occurs.

More background can be found in some messages on:
https://groups.google.com/d/msg/clojure/ccZuKTYKDPc/xpaz44UDqYwJ

Approach: The attached patch rolls back most of the change for CLJ-1169, specifically the part that wraps exceptions in CompilerException and the tests that were affected by this change (good examples of the kind of breakage others are seeing). I left the parts of CLJ-1169 that added quotes in the error message and those equivalent tests.

Patch: clj-1745.patch



 Comments   
Comment by Stuart Halloway [ 04/Jun/15 7:15 PM ]

All the stuff about lein in this ticket is just noise, and I am removing it. (Please don't use the phrase "small reproducing case" for anything that includes lein.) Clojure's behavior changed: improved error reporting. Code that explicitly relied on less-good error reporting broke.

Comment by Alex Miller [ 05/Jun/15 7:50 AM ]

Pulling this into 1.7 just for tracking discussion.

Comment by Alex Miller [ 05/Jun/15 4:02 PM ]

There is one confusing factor in replicating this re lein vs Clojure. The Clojure REPL will unpeel CompilerExceptions (see clojure.main/root-cause) so the repl actually prints the same in 1.7 as in 1.6 (but the exception chain and class are wrapped in one more CompilerException than before). Leiningen's repl will actually show the full structure.

Comment by Alex Miller [ 05/Jun/15 4:36 PM ]

I went back and looked at CLJ-1169 and while I think the intentions are good there, I do think that wrapping exceptions that happen to be thrown out of macro bodies like this does create unexpected (certainly different) behavior. I've attached a patch that rolls back most of the CLJ-1169 change.





[CLJ-1744] Clear unused locals Created: 03/Jun/15  Updated: 04/Jun/15

Status: Open
Project: Clojure
Component/s: None
Affects Version/s: None
Fix Version/s: None

Type: Enhancement Priority: Critical
Reporter: Nicola Mometto Assignee: Unassigned
Resolution: Unresolved Votes: 2
Labels: compiler, locals-clearing

Attachments: Text File 0001-CLJ-1744-clear-unused-locals.patch     Text File 0001-CLJ-1744-clear-unused-locals-v2.patch    
Patch: Code

 Description   

Clojure currently doesn't clear unused locals. This is problematic as some form of destructuring can generate unused/unusable locals that the compiler cannot clear and thus can cause head retention:

;; this works
user=> (loop [xs (repeatedly 2 #(byte-array (quot (.maxMemory (Runtime/getRuntime)) 10)))] (when (seq xs) (recur (rest xs))))
nil
;; this doesn't
user=>  (loop [[x & xs] (repeatedly 200 #(byte-array (quot (.maxMemory (Runtime/getRuntime)) 10)))] (when (seq xs) (recur xs)))
OutOfMemoryError Java heap space  clojure.lang.Numbers.byte_array (Numbers.java:1252)

Here's a macroexpansion that explicits this issue:

user=> (macroexpand-all '(loop [[a & b] c] [a b]))
(let* [G__21 c 
       vec__22 G__21
       a (clojure.core/nth vec__22 0 nil)
       b (clojure.core/nthnext vec__22 1)]
 (loop* [G__21 G__21]
   (let* [vec__23 G__21
          a (clojure.core/nth vec__23 0 nil)
          b (clojure.core/nthnext vec__23 1)]
     [a b])

The first two bindings of a and b will hold onto the head of c since they are never used and not accessible from the loop body they cannot be cleared.

The attached patch allows the compiler to clear unused locals making both examples work by trivially popping the binding value off the stack rather than assigning it to the local variable if analysis reveals it's never used.

Patch: 0001-CLJ-1744-clear-unused-locals-v2.patch



 Comments   
Comment by Michael Blume [ 03/Jun/15 12:57 PM ]

Nice =)





[CLJ-1743] Avoid compile-time static initialization of classes when using inheritance Created: 02/Jun/15  Updated: 27/Jul/15

Status: Open
Project: Clojure
Component/s: None
Affects Version/s: Release 1.6, Release 1.7
Fix Version/s: None

Type: Enhancement Priority: Major
Reporter: Abe Fettig Assignee: Unassigned
Resolution: Unresolved Votes: 4
Labels: aot, compiler, interop

Attachments: Text File 0001-Avoid-compile-time-class-initialization-when-using-g.patch     Text File clj-1743-2.patch    
Patch: Code
Approval: Triaged

 Description   

I'm working on a project using Clojure and RoboVM. We use AOT compilation to compile Clojure to JVM classes, and then use RoboVM to compile the JVM classes to native code. In our Clojure code, we call Java APIs provided by RoboVM, which wrap the native iOS APIs.

But we've found an issue with inheritance and class-level static initialization code. Many iOS APIs require inheriting from a base object and then overriding certain methods. Currently, Clojure runs a superclass's static initialization code at compile time, whether using ":gen-class" or "proxy" to create the subclass. However, RoboVM's base "ObjCObject" class [1], which most iOS-specific classes inherit from, requires the iOS runtime to initialize, and throws an error at compile time since the code isn't running on a device.

CLJ-1315 addressed a similar issue by modifying "import" to load classes without running static initialization code. I've written my own patch which extends this behavior to work in ":gen-class" and "proxy" as well. The unit tests pass, and we're using this code successfully in our iOS app.

Patch: clj-1743-2.patch

Here's some sample code that can be used to demonstrate the current behavior (Full demo project at https://github.com/figly/clojure-static-initialization):

Demo.java
package clojure_static_initialization;

public class Demo {
  static {
    System.out.println("Running static initializers!");
  }
  public Demo () {
  }
}
gen_class_demo.clj
(ns clojure-static-initialization.gen-class-demo
  (:gen-class :extends clojure_static_initialization.Demo))
proxy_demo.clj
(ns clojure-static-initialization.proxy-demo)

(defn make-proxy []
  (proxy [clojure_static_initialization.Demo] []))

[1] https://github.com/robovm/robovm/blob/master/objc/src/main/java/org/robovm/objc/ObjCObject.java



 Comments   
Comment by Alex Miller [ 18/Jun/15 3:01 PM ]

No changes from previous, just updated to apply to master as of 1.7.0-RC2.

Comment by Alex Miller [ 18/Jun/15 3:03 PM ]

If you had a sketch to test this with proxy and gen-class, that would be helpful.

Comment by Abe Fettig [ 22/Jun/15 8:31 AM ]

Sure, what form would you like for the sketch code? A small standalone project? Unit tests?

Comment by Alex Miller [ 22/Jun/15 8:40 AM ]

Just a few lines of Java (a class with static initializer that printed) and Clojure code (for gen-class and proxy extending it) here in the test description that could be used to demonstrate the problem. Should not have any dependency on iOS or other external dependencies.

Comment by Abe Fettig [ 01/Jul/15 8:49 PM ]

Sample code added, let me know if I can add anything else!

Comment by Abe Fettig [ 27/Jul/15 2:21 PM ]

Just out of curiosity, what are the odds this could make it into 1.8?

Comment by Alex Miller [ 27/Jul/15 6:06 PM ]

unknown.





[CLJ-1741] deftype class literals and instances loaded from different classloaders when recompiling namespace Created: 30/May/15  Updated: 18/Jun/15

Status: Open
Project: Clojure
Component/s: None
Affects Version/s: Release 1.7
Fix Version/s: Release 1.8

Type: Defect Priority: Major
Reporter: Stephen Nelson Assignee: Unassigned
Resolution: Unresolved Votes: 0
Labels: aot, classloader, compiler

Attachments: Text File 0001-CLJ-1714-Don-t-load-AOT-class-when-compiling-already.patch    
Patch: Code
Approval: Incomplete

 Description   

Scenario: Given two files:

src/dispatch/core.clj:

(ns dispatch.core (:require [dispatch.dispatch]))

src/dispatch/dispatch.clj:

(ns dispatch.dispatch)
(deftype T [])
(def t (->T))
(println "T = (class t):" (= T (class t)))

Compile first core, then dispatch:

java -cp src:target/classes:clojure.jar -Dclojure.compile.path=target/classes clojure.main
user=> (compile 'dispatch.core)
T = (class t): true
dispatch.core
user=> (compile 'dispatch.dispatch)
T = (class t): false     ;; expected true
dispatch.dispatch

This scenario more commonly occurs in a leiningen project with :aot :all. Files are compiled in alphabetical order with :all. In this case, dispatch.core will be compiled first, then dispatch.dispatch.

Cause:

(compile 'dispatch.core)

  • transitively compiles dispatch.dispatch
  • writes .class files to compile-path (which is on the classpath)
  • assertion passes

(compile 'dispatch.dispatch)

  • due to prior compile, load dispatch.dispatch__init is loaded via the appclassloader
  • ->T constructor will use new bytecode to instantiate a T instance - this uses appclassloader, loaded from compiled T on disk
  • however, T class literals are resolved with RT.classForName, which checks the dynamic classloader cache, so uses old runtime version of T, instead of on-disk version

In 1.6, RT.classForName() did not check dynamic classloader cache, so loaded T from disk as with instances. This was changed in CLJ-979 to support other redefinition and AOT mixing usages.

Approaches:

1) Compile in reverse dependency order to avoid compiling twice.

Either swap the order of compilation in the first example or specify the order in project.clj:

:aot [dispatch.dispatch dispatch.core]

This is a short-term workaround.

2) Move the deftype into a separate namespace from where it is used so it is not redefined on the second compile. This is another short-term workaround.

3) Do not put compile-path on the classpath (this violates current expectations, but avoids loading dispatch__init)

(set! *compile-path* "foo")
(compile 'dispatch.core)
(compile 'dispatch.dispatch)

This is not easy to set up via Leiningen currently.

4) Compile each file with an independent Clojure runtime - avoids using cached classes in DCL for class literals.

Probably too annoying to actually do right now in Leiningen or otherwise.

5) Make compilation non-transitive. This is in the ballpark of CLJ-322, which is another can of worms. Also possibly where we should be headed though.

Screening: I do not believe the proposed patch is a good idea - it papers over the symptom without addressing the root cause. I think we need to re-evaluate how compilation works with regard to compile-path (#3) and transitivity (CLJ-322) (#5), but I think we should do this after 1.7. - Alex

See also: CLJ-1650



 Comments   
Comment by Alex Miller [ 30/May/15 8:50 PM ]

Pulling into 1.7 for consideration.

Comment by Stephen Nelson [ 30/May/15 8:55 PM ]

I've added a debug flag to my example that causes type instance hashcodes and their class-loaders to be printed.

Compiling dispatch.core
deftype => 652433136 (clojure.lang.DynamicClassLoader@23c30a20)
defmethod => 652433136 (clojure.lang.DynamicClassLoader@23c30a20)
instance => 652433136 (clojure.lang.DynamicClassLoader@23c30a20)
dispatch:  :pass
Compiling dispatch.dispatch
deftype => 652433136 (clojure.lang.DynamicClassLoader@23c30a20)
defmethod => 652433136 (clojure.lang.DynamicClassLoader@23c30a20)
instance => 760357227 (sun.misc.Launcher$AppClassLoader@42a57993)
dispatch:  :fail
Comment by Nicola Mometto [ 01/Jun/15 7:23 AM ]

The compiler has weird loading rules when using `compile` and both a clj file and a class file are present in the classpath.

This bug happens because RT.load will load the AOT class file rebinding the ->Ctor to use the AOT deftype instance.

A fix for this would be making load "loaded libs" aware to avoid unnecessary/harmful reloadings.

Comment by Nicola Mometto [ 01/Jun/15 10:55 AM ]

The attached patch fixes this bug by keeping track of what has already been loaded and loading the AOT class only if necessary

Comment by Alex Miller [ 16/Jun/15 2:24 PM ]

Original description (since replaced):

Type-dispatching multimethods are defined using the wrong type instance

When using a multimethod that dispatches on types, such as print-dup/print-method, the type reference passed to addMethod in the presence of aot is incorrect on the second load of the namespace. This means that if the namespace has already been loaded as a dependency of another file, the second load when the namespace is loaded for aot compilation will produce a multimethod that fails to dispatch correctly.

I've created an example repository:
https://github.com/sfnelson/clj-mm-dispatch

To reproduce independently, create a namespace that contains a deftype and a multimethod dispatching on the type, and a second namespace that requires the first and sorts alphabetically before the first. Aot-compile both namespaces. When the type-defining namespace is loaded via require it produces a class file for the deftype. When it is loaded the second time for aot-compilation, the type corresponding to the existing class file is given to the defmethod, instead of the new class constructed by loading the namespace. This causes the multimethod it fail to dispatch correctly.

To me this issue seems similar to CLJ-979: the type passed to the multimethod is retrieved using the wrong classloader. This suggests that it might have wider implications than AOT and multimethod dispatch.

Comment by Nicola Mometto [ 18/Jun/15 11:09 AM ]

I just realized this ticket is a duplicate of CLJ-1650





[CLJ-1737] Omit java exception class from CompilerException message Created: 23/May/15  Updated: 23/May/15

Status: Open
Project: Clojure
Component/s: None
Affects Version/s: Release 1.5, Release 1.6, Release 1.7
Fix Version/s: None

Type: Enhancement Priority: Minor
Reporter: John Hume Assignee: Unassigned
Resolution: Unresolved Votes: 0
Labels: compiler, errormsgs, patch, usability

Attachments: Text File clearer-CompilerException-messase.patch     File compiler_exception_examples.clj    
Patch: Code

 Description   

A CompilerException is always created with a cause exception. Currently the message is built using cause.toString(), which for all examples I've examined is the cause class, followed by a colon, followed by the cause message. In all those examples, the message of the cause is informative, and the class name provides no additional help.

I propose to switch to using cause.getMessage() rather than cause.toString(). This would make it easier for tools to present compiler errors that don't leak implementation details that may confuse a new user. The cause class would still be shown in the stack trace.

Here are the examples I looked at, with the output from before the attached patch:

Example source '(ns foo)

(def'
Exception message:
 java.lang.RuntimeException: EOF while reading, starting at line 3, compiling:(/Users/jhume/Projects/clojure/clojure/temp.clj:3:1)

Example source ':foo}'
Exception message:
 java.lang.RuntimeException: Unmatched delimiter: }, compiling:(/Users/jhume/Projects/clojure/clojure/temp.clj:1:6)

Example source 'foo'
Exception message:
 java.lang.RuntimeException: Unable to resolve symbol: foo in this context, compiling:(/Users/jhume/Projects/clojure/clojure/temp.clj:14:1)

Example source 'clojure.core/firstt'
Exception message:
 java.lang.RuntimeException: No such var: clojure.core/firstt, compiling:(/Users/jhume/Projects/clojure/clojure/temp.clj:15:1)

Example source '(nil 1)'
Exception message:
 java.lang.IllegalArgumentException: Can't call nil, compiling:(/Users/jhume/Projects/clojure/clojure/temp.clj:1:1)

Example source '("hi" 1)'
Exception message:
 java.lang.ClassCastException: java.lang.String cannot be cast to clojure.lang.IFn, compiling:(/Users/jhume/Projects/clojure/clojure/temp.clj:1:1)

Example source '{:foo}'
Exception message:
 java.lang.RuntimeException: Map literal must contain an even number of forms, compiling:(/Users/jhume/Projects/clojure/clojure/temp.clj:1:7)

Example source '1st'
Exception message:
 java.lang.NumberFormatException: Invalid number: 1st, compiling:(/Users/jhume/Projects/clojure/clojure/temp.clj:1:1)

And the output with the attached patch applied:

Example source '(ns foo)

(def'
Exception message:
 EOF while reading, starting at line 3, compiling:(/Users/jhume/Projects/clojure/clojure/temp.clj:3:1)

Example source ':foo}'
Exception message:
 Unmatched delimiter: }, compiling:(/Users/jhume/Projects/clojure/clojure/temp.clj:1:6)

Example source 'foo'
Exception message:
 Unable to resolve symbol: foo in this context, compiling:(/Users/jhume/Projects/clojure/clojure/temp.clj:14:1)

Example source 'clojure.core/firstt'
Exception message:
 No such var: clojure.core/firstt, compiling:(/Users/jhume/Projects/clojure/clojure/temp.clj:15:1)

Example source '(nil 1)'
Exception message:
 Can't call nil, compiling:(/Users/jhume/Projects/clojure/clojure/temp.clj:1:1)

Example source '("hi" 1)'
Exception message:
 java.lang.String cannot be cast to clojure.lang.IFn, compiling:(/Users/jhume/Projects/clojure/clojure/temp.clj:1:1)

Example source '{:foo}'
Exception message:
 Map literal must contain an even number of forms, compiling:(/Users/jhume/Projects/clojure/clojure/temp.clj:1:7)

Example source '1st'
Exception message:
 Invalid number: 1st, compiling:(/Users/jhume/Projects/clojure/clojure/temp.clj:1:1)





[CLJ-1717] Compiler casts System properties to String without prior type check Created: 29/Apr/15  Updated: 05/Jun/15  Resolved: 18/May/15

Status: Closed
Project: Clojure
Component/s: None
Affects Version/s: Release 1.7
Fix Version/s: None

Type: Defect Priority: Critical
Reporter: Laurent Petit Assignee: Unassigned
Resolution: Declined Votes: 0
Labels: compiler, regression
Environment:

occurs for any JVM / Operating System. Encountered when using Clojure inside the Equinox OSGi framework (reference OSGi implementation used e.g. by Eclipse IDE)


Attachments: Text File clj-1717-1.patch    
Patch: Code
Approval: Triaged

 Description   

The Clojure Compiler loops over all System Properties through their EntrySet.
This interface allows for non-String properties to be found. It is leveraged by the Equinox OSGi framework (reference OSGi implementation used e.g. by the Eclipse IDE).

This means that since this new code has been introduced in the Clojure compiler, the official Clojure Jar cannot be used inside Eclipse.

The problem is that Counterclockwise, the Eclipse-based Clojure IDE, at least, is affected, since it is developed with Clojure code.

The attached patch solves the issue by skipping System Properties key/value pairs whose values aren't Strings.



 Comments   
Comment by Alex Miller [ 29/Apr/15 2:40 PM ]

Probably a regression related to CLJ-1274.

Comment by Laurent Petit [ 29/Apr/15 2:45 PM ]

Yes, CLJ-1274 moved the code from the Compile.java class to the Compiler.java class. The code already had the cast problem, but it was probably not an issue at runtime for CCW when only present in Compile.java

Comment by Alex Miller [ 29/Apr/15 3:00 PM ]

It looks like maybe this is considered a bug in Eclipse land (which it probably should be)?
https://bugs.eclipse.org/bugs/show_bug.cgi?id=445122

Comment by Laurent Petit [ 29/Apr/15 3:21 PM ]

I agree that it is an abuse on the side of Eclipse-and of a System properties hole that allows to put non-String values and then browse them through methods not using Generics.

From the last comments, it appears that it has only be released very recently, and for only the last stable version of Eclipse. So I expect CCW users with disparate Eclipse versions installed to have the problem if I do not apply the patch to my custom version of Clojure for some more months.

I can live with that if you reject the issue.

Comment by Laurent Petit [ 01/May/15 7:28 AM ]

What is the status for this issue? I understand that being considered an abuse from the OSGi implementation of the System properties contract, you might not want to add code for dealing with this in Compiler.
On the other hand, it's a small 3-lines check with an explanation of why it's done that way, so...

Anyway, feel free to get rid of it and close it so it doesn't get in the way if you think it's not worth the trouble.

Comment by Alex Miller [ 01/May/15 8:44 AM ]

We haven't had a chance to discuss it further yet. There is reluctance to change it if it's not necessary.

Comment by Nicola Mometto [ 05/Jun/15 9:56 AM ]

Ticket was declined and marked as resolved but not closed. I'm closing it.





[CLJ-1714] Some static initialisers still run at compile time if used in type hints Created: 22/Apr/15  Updated: 12/Aug/15

Status: Open
Project: Clojure
Component/s: None
Affects Version/s: None
Fix Version/s: None

Type: Defect Priority: Minor
Reporter: Adam Clements Assignee: Unassigned
Resolution: Unresolved Votes: 1
Labels: compiler, typehints

Attachments: Text File CLJ-1714.patch     Text File CLJ-1714-v2.patch    
Patch: Code
Approval: Triaged

 Description   

AOT compiling on an x86 machine to be run on an ARM machine when a Java dependency has a native component and the class with the native dependency is used in a type hint.

In this situation, the only native library available on the classpath is the ARM dependency, and obviously won't load on the compiling x86 machine. Java libraries tend to load the native dependencies in the static initialiser of the class, which will fail in this situation as the architecture is x86 and the dependencies are ARM, for which reason CLJ-1315 made the change to not run static initialisers at compile time.

This covers a case which didn't come up as part of CLJ-1315, that the same problem occurs if rather than constructing the class, you simply use it as a type hint (which IMO is doubly surprising as something to have a side-effect).

This patch fixes that - happy to try and create a test, but would appreciate some advice on the shape such a test would take - presumably loading a java native library would be undesirable. I could simply check for static initialisers being run, but first would need some agreement that this is universally undesirable at compile time.

I have been using this patch in production for over a year with no adverse effects (as has anybody using the clojure-android build of clojure).



 Comments   
Comment by Alex Miller [ 22/Apr/15 10:53 AM ]

I think this might have been logged already but I'm not sure.

Comment by Michael Blume [ 22/Apr/15 12:30 PM ]

Patch won't apply to master for me

Comment by Adam Clements [ 22/Apr/15 2:39 PM ]

Really sorry, don't know what happened there. I checked out a fresh copy of the repo and re-applied the changes, deleted the old patch as it was garbage. Try the new one, timestamped 2:37pm

Comment by Stuart Halloway [ 30/Jul/15 1:52 PM ]

Please add an example of the problem, and if possible a failing test.

Comment by Alex Miller [ 30/Jul/15 5:14 PM ]

Reset to "Open" as moving from Triaged->Incomplete is not valid in our current workflow.

Comment by Adam Clements [ 31/Jul/15 10:56 AM ]

Example problem:
AOT compiling on an x86 machine to be run on an ARM machine when a Java dependency has a native component and the class with the native dependency is used in a type hint.

In this situation, the only native library available on the classpath is the ARM dependency, and obviously won't load on the compiling x86 machine. Java libraries tend to load the native dependencies in the static initialiser of the class, which will fail in this situation as the architecture is x86 and the dependencies are ARM, for which reason CLJ-1315 made the change to not run static initialisers at compile time.

This covers a case which didn't come up as part of CLJ-1315, that the same problem occurs if rather than constructing the class, you simply use it as a type hint (which IMO is doubly surprising as something to have a side-effect).

This patch fixes that - happy to try and create a test, but would appreciate some advice on the shape such a test would take - presumably loading a java native library would be undesirable. I could simply check for static initialisers being run, but first would need some agreement that this is universally undesirable at compile time.

I have been using this patch in production for over a year with no adverse effects (as has anybody using the clojure-android build of clojure).

Comment by Stuart Halloway [ 31/Jul/15 11:34 AM ]

Hi Adam,

Thanks for the quick response. I think checking for static initializers being run is OK for a test.

Comment by Adam Clements [ 12/Aug/15 9:12 AM ]

Added failing tests which now pass





[CLJ-1708] Volatile mutable in deftype is not settable when using try..finally and returning this Created: 17/Apr/15  Updated: 31/Jul/15

Status: Open
Project: Clojure
Component/s: None
Affects Version/s: Release 1.6, Release 1.7
Fix Version/s: None

Type: Defect Priority: Minor
Reporter: Patrick Gombert Assignee: Unassigned
Resolution: Unresolved Votes: 0
Labels: compiler, deftype
Environment:

clojure 1.6.0, clojure 1.7.0-beta1


Approval: Triaged

 Description   

Reproducible Code: https://gist.github.com/patrickgombert/1bcb8a051aeb3e82d855

When using a volatile-mutable field in deftype, compilation fails if the field is set! in a method call that uses both try..finally and returns itself from the method call. Leaving out either the try..finally or returning itself from the method causes compilation to succeed.

Expected behavior: set! should set the volatile-mutable variable and compilation should succeed.



 Comments   
Comment by Kevin Downey [ 17/Apr/15 7:15 PM ]

this must be the same issue as CLJ-1422 and CLJ-701, it has nothing to do with returning `this`, but with the try being in a tail position or not. if the try is not in a tail position the compiler hoists it out in to a thunk. effectively the code is

(deftype SomeType [^:volatile-mutable foo]
  SomeProtocol
  (someFn [_] ((fn [] (try (set! foo 1))))))

which the compiler also rejects, because it doesn't let you mutate fields from functions that are not the immediate protocol functions





[CLJ-1659] compile leaks files Created: 16/Feb/15  Updated: 10/Aug/15  Resolved: 17/Jul/15

Status: Closed
Project: Clojure
Component/s: None
Affects Version/s: Release 1.6
Fix Version/s: Release 1.8

Type: Defect Priority: Major
Reporter: Ralf Schmitt Assignee: Unassigned
Resolution: Completed Votes: 3
Labels: compiler, ft

Attachments: Text File clj-1659.patch     Text File clj-1659-v2.patch     Text File clj-1659-v3.patch    
Patch: Code
Approval: Ok

 Description   

clojure's compile function leaks file descriptors, i.e. it relies on garbage collection to close the files. I'm trying to use boot [1] on windows and ran into the problem, that files could not be deleted intermittently [2]. The problem is that clojure's compile function, or rather clojure.lang.RT.lastModified() relies on garbage collection to close files. lastModified() looks like:

static public long lastModified(URL url, String libfile) throws IOException{
	if(url.getProtocol().equals("jar")) {
		return ((JarURLConnection) url.openConnection()).getJarFile().getEntry(libfile).getTime();
	}
	else {
		return url.openConnection().getLastModified();
	}
}

Here's the stacktrace from file leak detector [3]:

#205 C:\Users\ralf\.boot\tmp\Users\ralf\home\steinmetz\2mg\-x24pa9\steinmetz\fx\config.clj by thread:clojure-agent-send-off-pool-0 on Sat Feb 14 19:58:46 UTC 2015
    at java.io.FileInputStream.(FileInputStream.java:139)
    at java.io.FileInputStream.(FileInputStream.java:93)
    at sun.net.www.protocol.file.FileURLConnection.connect(FileURLConnection.java:90)
    at sun.net.www.protocol.file.FileURLConnection.initializeHeaders(FileURLConnection.java:110)
    at sun.net.www.protocol.file.FileURLConnection.getLastModified(FileURLConnection.java:178)
    at clojure.lang.RT.lastModified(RT.java:390)
    at clojure.lang.RT.load(RT.java:421)
    at clojure.lang.RT.load(RT.java:411)
    ...

Cause: getLastModified() opens the URLConnection's InputStream but does not close it.

Approach: On Stackoverflow [4] there's a discussion on how to close the URLConnection correctly.

On non-Windows operating systems this shouldn't be much of a problem. But on windows this hurts very much, since you can't delete files that are opened by some process.

Patch: clj-1659-v3.patch

Screened by: Alex Miller

[1] http://boot-clj.com/
[2] https://github.com/boot-clj/boot/issues/117
[3] http://file-leak-detector.kohsuke.org/
[4] http://stackoverflow.com/questions/9150200/closing-urlconnection-and-inputstream-correctly



 Comments   
Comment by Pietro Menna [ 06/May/15 11:10 AM ]

First attempt Patch

Comment by Pietro Menna [ 06/May/15 11:13 AM ]

Hi Alex,

This is my first patch to Clojure and to any OSS. So maybe I will need a little guidance. I follow the steps on how to generate the patch and just uploaded the patch to this thread.

The link from Stack Overflow was good, but unfortunately it is not possible to cast to HttpURLConnection in order to have the .disconnect() method.

Please, let me know if I should attempt anything else.

Kind regards,

Pietro

Comment by Andy Fingerhut [ 06/May/15 11:20 AM ]

Seems that creating a test for this to be run on every build might be difficult.

Have you verified that on Windows it has the desired effect?

Comment by Ralf Schmitt [ 06/May/15 4:49 PM ]

I don't understand how the patch solves that issue. It just sets connection to null. Or am I missing something?

You can test for the file leak with the following program. This works on windows and Linux for me.

leak-test.clj
;; test file leak
;; on UNIX-like systems set hard limit on open files via
;; ulimit -H -n 200 and then run
;; java -jar clojure-1.6.0.jar leak-test.clj

(let [file (java.io.File. "test-leak.txt")
      url (.. file toURI toURL)]
  (doseq [x (range 2000)]
    (print x " ")
    (flush)
    (spit file "")
    (clojure.lang.RT/lastModified url nil)
    (assert (.delete file) "delete failed")))
Comment by Ralf Schmitt [ 06/May/15 5:21 PM ]

dammit, the formatting is wrong. but this patch seems to fix the problem for me (tested on linux).

Comment by Ralf Schmitt [ 06/May/15 6:16 PM ]

indentation fixed

Comment by Andy Fingerhut [ 07/May/15 8:45 AM ]

Ralf, thanks for the patch. I can't say if or when this ticket will be considered for a change to Clojure, but I do know that patches are only considered if they were written by someone who has signed a CA. Were you considering doing so? You can do it on-line here if you wish: http://clojure.org/contributing

Also, patches should be in a slightly different format that include the author's name, email, date of change, etc. Instructions for creating a patch in that format are here: http://dev.clojure.org/display/community/Developing+Patches

Comment by Ralf Schmitt [ 07/May/15 8:50 AM ]

Thanks for the explanation. I've signed the CA and I will update the patch.

Comment by Ralf Schmitt [ 07/May/15 9:34 AM ]

patch vs current master, created with git format-patch

Comment by Andy Fingerhut [ 26/May/15 8:41 AM ]

Ralph, it would be good if all attached files on a ticket have different names, for clarity when referring to them. Could you remove your clj-1659.patch and upload it with a different name, e.g. clj-1659-v2.patch ?

Comment by Ralf Schmitt [ 26/May/15 8:49 AM ]

upload my last patch with non-conflicting filename

Comment by Ralf Schmitt [ 26/May/15 8:52 AM ]

yes, sorry about that. I don't know how to remove my previous patches. (ok, found it now)

Comment by Sven Richter [ 08/Jun/15 2:57 AM ]

This one is keeping me from using boot on windows and seeing how far the 1.7 release process is I would like to express my strong wish that this one makes it into the 1.7 release.

Thanks everyone for their hard work

Comment by Alex Miller [ 18/Jun/15 4:09 PM ]

The clj-1659-v3.patch is same as v2, just removed unneeded braces to match rest of compiler.

Comment by Erik Dannenberg [ 06/Aug/15 12:09 PM ]

Issue still persists with Clojure 1.8.0-alpha4 and latest boot. Running Windows 8.1.

Example app that should reproduce the error: https://github.com/danielsz/sente-boot

clone, run boot dev, change a source file => file exists errors

Any chances to get a fix for this into Clojure 1.7? It's pretty much a show stopper for us as it prevents the reloaded workflow for our windows devs.

Comment by Ralf Schmitt [ 06/Aug/15 4:21 PM ]

This must be another issue. Please try to find the leak with http://file-leak-detector.kohsuke.org/

(I once dug into a similar issue in boot's bug tracker and the problem was somewhere in the cljs compiler).

Comment by Erik Dannenberg [ 10/Aug/15 11:46 AM ]

I attached the file leak detector and ran boot with clojure 1.7 and 1.8-alpha4, the stacktraces for both are a bit different but from what i can tell the leak cause is still the same and looking very similiar to the stacktrace posted in the issue description. I do however see the cljs compiler further down in the stacktrace.

Should i open an issue on CLJS tracker?

Comment by Ralf Schmitt [ 10/Aug/15 11:51 AM ]

This is hard to tell without a stacktrace...

Comment by Erik Dannenberg [ 10/Aug/15 11:59 AM ]
1 descriptors are open
#1 C:\Users\ed\.boot\cache\tmp\Users\ed\projekte\sente-system\5ho\-c4hx1l\adzerk\boot_reload.cljs by thread:clojure-agent-send-off-pool-0 on Mon Aug 10 17:54:39 CEST 2015
	at java.io.FileInputStream.<init>(FileInputStream.java:139)
	at java.io.FileInputStream.<init>(FileInputStream.java:93)
	at sun.net.www.protocol.file.FileURLConnection.connect(FileURLConnection.java:90)
	at sun.net.www.protocol.file.FileURLConnection.initializeHeaders(FileURLConnection.java:110)
	at sun.net.www.protocol.file.FileURLConnection.getLastModified(FileURLConnection.java:178)
	at cljs.util$last_modified.invokeStatic(util.cljc:136)
	at cljs.util$last_modified.invoke(util.cljc)
	at cljs.analyzer$requires_analysis_QMARK_.invokeStatic(analyzer.cljc:2179)
	at cljs.analyzer$requires_analysis_QMARK_.invoke(analyzer.cljc)
	at cljs.analyzer$requires_analysis_QMARK_.invokeStatic(analyzer.cljc:2164)
	at cljs.analyzer$requires_analysis_QMARK_.invoke(analyzer.cljc)
	at cljs.analyzer$analyze_file.invokeStatic(analyzer.cljc:2220)
	at cljs.analyzer$analyze_file.invoke(analyzer.cljc)
	at cljs.analyzer$analyze_deps.invokeStatic(analyzer.cljc:1291)
	at cljs.analyzer$analyze_deps.invoke(analyzer.cljc)
	at cljs.analyzer$eval1868$fn__1870.invoke(analyzer.cljc:1545)
	at clojure.lang.MultiFn.invoke(MultiFn.java:251)
	at cljs.analyzer$analyze_seq.invokeStatic(analyzer.cljc:1900)
	at cljs.analyzer$analyze_seq.invoke(analyzer.cljc)
	at cljs.analyzer$analyze$fn__2118.invoke(analyzer.cljc:1992)
	at cljs.analyzer$analyze.invokeStatic(analyzer.cljc:1985)
	at cljs.analyzer$analyze.invoke(analyzer.cljc)
	at cljs.compiler$compile_file_STAR_$fn__3216.invoke(compiler.cljc:1027)
	at cljs.compiler$with_core_cljs.invokeStatic(compiler.cljc:968)
	at cljs.compiler$with_core_cljs.invoke(compiler.cljc)
	at cljs.compiler$compile_file_STAR_.invokeStatic(compiler.cljc:988)
	at cljs.compiler$compile_file_STAR_.invoke(compiler.cljc)
	at cljs.compiler$compile_file$fn__3248.invoke(compiler.cljc:1129)
	at cljs.compiler$compile_file.invokeStatic(compiler.cljc:1109)
	at cljs.compiler$compile_file.invoke(compiler.cljc)
	at cljs.compiler$compile_root.invokeStatic(compiler.cljc:1181)
	at cljs.compiler$compile_root.invoke(compiler.cljc)
	at cljs.closure$compile_dir.invokeStatic(closure.clj:385)
	at cljs.closure$compile_dir.invoke(closure.clj)
	at cljs.closure$eval3614$fn__3615.invoke(closure.clj:425)
	at cljs.closure$eval3567$fn__3568$G__3558__3575.invoke(closure.clj:331)
	at cljs.closure$eval3627$fn__3628.invoke(closure.clj:439)
	at cljs.closure$eval3567$fn__3568$G__3558__3575.invoke(closure.clj:331)
	at adzerk.boot_cljs.impl.CljsSourcePaths$fn__4011.invoke(impl.clj:14)
Comment by Alex Miller [ 10/Aug/15 12:30 PM ]

Looks like similar problem in CLJS to me - I would open a ticket there.

Comment by Erik Dannenberg [ 10/Aug/15 1:35 PM ]

Done. CLJS-1416





[CLJ-1648] Use equals() instead of == when resolving Symbol Created: 22/Jan/15  Updated: 12/May/15  Resolved: 12/May/15

Status: Closed
Project: Clojure
Component/s: None
Affects Version/s: Release 1.7
Fix Version/s: Release 1.7

Type: Defect Priority: Critical
Reporter: Steven Yi Assignee: Unassigned
Resolution: Completed Votes: 0
Labels: Compiler

Attachments: File resolve-symbol-equals.diff    
Patch: Code
Approval: Ok

 Description   

In Compiler.java, resolveSymbol() uses == to compare a Symbol's ns and the found namespace's name. This can result in a false comparison result, though the name's may be equal. In the following example:

ond.core=> (require '[clojure.string])
nil
ond.core=> `(clojure.string/join "," [1 2])
false : true ;; reported from System.out.println code I put into Compiler.java for == vs .equals()
nil

The result is that a new Symbol is allocated, when the previous one should be returned.

Prior to Clojure 1.7, Symbol name and ns were interned so == would actually have worked, but that is no longer the case.

Patch: resolve-symbol-equals.diff

Screened by: Alex Miller

[1] - https://groups.google.com/forum/#!topic/clojure-dev/58fYUSIEfxg






[CLJ-1645] 'javap -v' on protocol class reveals no source file Created: 16/Jan/15  Updated: 17/Jul/15  Resolved: 17/Jul/15

Status: Closed
Project: Clojure
Component/s: None
Affects Version/s: Release 1.6
Fix Version/s: Release 1.8

Type: Defect Priority: Minor
Reporter: Fabio Tudone Assignee: Unassigned
Resolution: Completed Votes: 0
Labels: compiler, ft, protocols, source
Environment:

Mac OS X Yosemite.

java version "1.8.0_25"
Java(TM) SE Runtime Environment (build 1.8.0_25-b17)
Java HotSpot(TM) 64-Bit Server VM (build 25.25-b02, mixed mode)


Attachments: Text File CLJ-1645-protocol-class-has-no-source-file-information.patch     Text File CLJ-1645-protocol-class-has-no-source-file-information-w-repl.patch    
Patch: Code and Test
Approval: Ok

 Description   

Through "javap -v" I can find source filename information in Clojure-generated datatype class files but not in protocol ones.

Approach: In gen-interface, if the *source-path* indicates this is not a REPL-generated interface, invoke the proper ASM method to set the source file. Per JVM expectations, this is the actual file name, not the file path.

Patch: CLJ-1645-protocol-class-has-no-source-file-information-w-repl.patch

Screened by: Alex Miller



 Comments   
Comment by Fabio Tudone [ 22/Jun/15 11:45 AM ]

Any chances this will get into Clojure 1.7.0?

Comment by Alex Miller [ 22/Jun/15 12:28 PM ]

No, sorry.

Comment by Alex Miller [ 22/Jun/15 1:10 PM ]

It looks like the patch does not handle the repl case where the file will be "NO_SOURCE_FILE". Can you add a (when (not= "NO_SOURCE_FILE" source-path) ...) check around calling the visitSource invocation?

Comment by Fabio Tudone [ 22/Jun/15 3:16 PM ]

As requested

Comment by Alex Miller [ 22/Jun/15 3:57 PM ]

My last comment formatted my "earmuff stars" around source-path and you included that in the patch, which doesn't compile. Should be *source-path*, not source-path in the condition.

Comment by Fabio Tudone [ 22/Jun/15 4:02 PM ]

Right, re-attaching.

Comment by Alex Miller [ 22/Jun/15 4:18 PM ]

Fabio, this looks good and I would like to move it forward, but I just realized we don't have a signed Contributor Agreement from you. If you could do so, that would be great - the information and electronic form can be found at http://clojure.org/contributing .

Comment by Fabio Tudone [ 22/Jun/15 4:24 PM ]

Sure, with pleasure. Please also consider that the original patch is not mine though, but Yanxiang Lou's.

Comment by Fabio Tudone [ 22/Jun/15 4:27 PM ]

Signed.





[CLJ-1632] Mark Clojure-generated classes in order for instrumenters to identify them easily Created: 05/Jan/15  Updated: 05/Jan/15  Resolved: 05/Jan/15

Status: Closed
Project: Clojure
Component/s: None
Affects Version/s: Release 1.6
Fix Version/s: None

Type: Enhancement Priority: Minor
Reporter: Fabio Tudone Assignee: Unassigned
Resolution: Declined Votes: 0
Labels: compiler


 Description   

Instrumenting logic specific to Clojure-generated classes should be able to identify them easily.



 Comments   
Comment by Fabio Tudone [ 05/Jan/15 2:23 AM ]

One way could be annotations, another could be interface-marking. Would that be feasible? Any drawbacks?

Comment by Alex Miller [ 05/Jan/15 8:17 AM ]

deftypes have a marker interface clojure.lang.IType.
defrecords have a marker interface clojure.lang.IRecord.
proxy classes have marker interface clojure.lang.IProxy.

I think generic markers for protocols or gen-interface would be undesirable as they may be used to create APIs for external use.

Comment by Fabio Tudone [ 05/Jan/15 8:29 AM ]

Not sure I understand your point about the non-marked (as of now and AFAIK) Clojure features such as protocols and gen-interface; could you elaborate? Does it apply to marking with annotations as well?

Comment by Alex Miller [ 05/Jan/15 10:50 AM ]

My point was that many people wish to generate interfaces that do not extend from interfaces in Clojure and adding those marker interfaces would be seen as a downside for them. Annotations are slightly better but have the same problem (dependencies on parts of Clojure core). You are of course free to add those interfaces or annotations yourself in your own code if that's useful to you!

Comment by Fabio Tudone [ 05/Jan/15 11:15 AM ]

It's clear now, thanks!

Actually my use case is about general tooling that will inspect and instrument all (and only) Clojure-generated code in any application making use of it, so I don't control the code I'm going to examine. This is done in order to add specific runtime features in a general fashion.

I can't find a way to do that reliably on everything that has been generated by Clojure; I could use some imperfect heuristics but I'd rather use a reliable way if one exists. Can you see of any other way of doing this I might have overlooked? Or is some other enhancement possible that would allow me to do this and would not compromise external integrations?

Comment by Alex Miller [ 05/Jan/15 11:31 AM ]

I don't see a general-purpose way to do this now. I do not believe supporting this is something we would spend time on.

Comment by Julien Eluard [ 05/Jan/15 2:03 PM ]

Class#isSynthetic() might be relevant here. If I am not mistaken asm generated classes will be flagged as synthetic.





[CLJ-1620] Constants are leaked in case of a reentrant eval Created: 18/Dec/14  Updated: 19/May/15

Status: Open
Project: Clojure
Component/s: None
Affects Version/s: Release 1.7
Fix Version/s: Release 1.8

Type: Defect Priority: Critical
Reporter: Christophe Grand Assignee: Unassigned
Resolution: Unresolved Votes: 3
Labels: aot, compiler

Attachments: Text File 0001-CLJ-1620-avoid-constants-leak-in-static-initalizer.patch     Text File 0001-CLJ-1620-avoid-constants-leak-in-static-initalizer-v2.patch     Text File 0001-CLJ-1620-avoid-constants-leak-in-static-initalizer-v3.patch     Text File 0001-CLJ-1620-avoid-constants-leak-in-static-initalizer-v4.patch     Text File clj-1620-v5.patch     Text File eval-bindings.patch    
Patch: Code
Approval: Incomplete

 Description   

Compiling a function that references a non loaded (or uninitialized) class triggers its init static. When the init static loads clojure code, some constants (source code I think) are leaked into the constants pool of the function under compilation.

It prevented CCW from working in some environments (Rational) because the static init of the resulting function was over 64K.

Steps to reproduce:

Load the leak.main ns and run the code in comments: the first function has 15 extra fields despite being identical to the second one.

(ns leak.main)

(defn first-to-load []
  leak.Klass/foo)

(defn second-to-load []
  leak.Klass/foo)

(comment
=> (map (comp count #(.getFields %) class) [first-to-load second-to-load])
(16 1)
)
package leak;
 
import clojure.lang.IFn;
import clojure.lang.RT;
import clojure.lang.Symbol;
 
public class Klass {
  static {
    RT.var("clojure.core", "require").invoke(Symbol.intern("leak.leaky"));
  }
  public static IFn foo = RT.var("leak.leaky", "foo");
}
(ns leak.leaky)

(defn foo
  "Some doc"
  []
  "hello")

(def unrelated 42)

https://gist.github.com/cgrand/5dcb6fe5b269aecc6a5b#file-main-clj-L10

Patch: clj-1620-v5.patch



 Comments   
Comment by Christophe Grand [ 18/Dec/14 3:56 PM ]

Patch from Nicola Mometto

Comment by Nicola Mometto [ 18/Dec/14 4:01 PM ]

Attached the same patch with a more informative better commit message

Comment by Laurent Petit [ 18/Dec/14 4:03 PM ]

I'd like to thank Christophe and Alex for their invaluable help in understanding what was happening, formulating the right hypothesis and then finding a fix.

I would also mention that even if non IBM rational environments where not affected by the bug to the point were CCW would not work, they were still affected. For instance the class for a one-liner function wrapping an interop call weighs 700bytes once the patch is applied, when it weighed 90kbytes with current 1.6 or 1.7.

Comment by Laurent Petit [ 18/Dec/14 5:07 PM ]

In CCW for the initial problematic function, the -v2 patch produces exactly the same bytecode as if the referenced class does not load any namespace in its static initializers.
That is, the patch is valid. I will test it live in the IBM Rational environment ASAP.

Comment by Laurent Petit [ 19/Dec/14 12:10 AM ]

I confirm the patch fixes the issue detected initially in the IBM Rational environment

Comment by Michael Blume [ 06/Jan/15 4:03 PM ]

I have absolutely no idea why, but if I apply this patch, and the patch for CLJ-1544 to master, and then try to build a war from this test project https://github.com/pdenhaan/extend-test I get a scary-looking traceback:

$ lein do clean, war!
Exception in thread "main" java.lang.NoSuchFieldError: __thunk__0__, compiling:(route.clj:1:1)
	at clojure.lang.Compiler$InvokeExpr.eval(Compiler.java:3606)
	at clojure.lang.Compiler.compile1(Compiler.java:7299)
	at clojure.lang.Compiler.compile1(Compiler.java:7289)
	at clojure.lang.Compiler.compile(Compiler.java:7365)
	at clojure.lang.RT.compile(RT.java:398)
	at clojure.lang.RT.load(RT.java:438)
	at clojure.lang.RT.load(RT.java:411)
	at clojure.core$load$fn__5415.invoke(core.clj:5823)
	at clojure.core$load.doInvoke(core.clj:5822)
	at clojure.lang.RestFn.invoke(RestFn.java:408)
	at clojure.core$load_one.invoke(core.clj:5613)
	at clojure.core$load_lib$fn__5362.invoke(core.clj:5668)
	at clojure.core$load_lib.doInvoke(core.clj:5667)
	at clojure.lang.RestFn.applyTo(RestFn.java:142)
	at clojure.core$apply.invoke(core.clj:628)
	at clojure.core$load_libs.doInvoke(core.clj:5706)
	at clojure.lang.RestFn.applyTo(RestFn.java:137)
	at clojure.core$apply.invoke(core.clj:628)
	at clojure.core$require.doInvoke(core.clj:5789)
	at clojure.lang.RestFn.invoke(RestFn.java:436)
	at extend_test.core.handler$loading__5301__auto____66.invoke(handler.clj:1)
	at clojure.lang.AFn.applyToHelper(AFn.java:152)
	at clojure.lang.AFn.applyTo(AFn.java:144)
	at clojure.lang.Compiler$InvokeExpr.eval(Compiler.java:3601)
	at clojure.lang.Compiler.compile1(Compiler.java:7299)
	at clojure.lang.Compiler.compile1(Compiler.java:7289)
	at clojure.lang.Compiler.compile(Compiler.java:7365)
	at clojure.lang.RT.compile(RT.java:398)
	at clojure.lang.RT.load(RT.java:438)
	at clojure.lang.RT.load(RT.java:411)
	at clojure.core$load$fn__5415.invoke(core.clj:5823)
	at clojure.core$load.doInvoke(core.clj:5822)
	at clojure.lang.RestFn.invoke(RestFn.java:408)
	at clojure.core$load_one.invoke(core.clj:5613)
	at clojure.core$load_lib$fn__5362.invoke(core.clj:5668)
	at clojure.core$load_lib.doInvoke(core.clj:5667)
	at clojure.lang.RestFn.applyTo(RestFn.java:142)
	at clojure.core$apply.invoke(core.clj:628)
	at clojure.core$load_libs.doInvoke(core.clj:5706)
	at clojure.lang.RestFn.applyTo(RestFn.java:137)
	at clojure.core$apply.invoke(core.clj:628)
	at clojure.core$require.doInvoke(core.clj:5789)
	at clojure.lang.RestFn.invoke(RestFn.java:421)
	at extend_test.core.servlet$loading__5301__auto____7.invoke(servlet.clj:1)
	at clojure.lang.AFn.applyToHelper(AFn.java:152)
	at clojure.lang.AFn.applyTo(AFn.java:144)
	at clojure.lang.Compiler$InvokeExpr.eval(Compiler.java:3601)
	at clojure.lang.Compiler.compile1(Compiler.java:7299)
	at clojure.lang.Compiler.compile1(Compiler.java:7289)
	at clojure.lang.Compiler.compile1(Compiler.java:7289)
	at clojure.lang.Compiler.compile(Compiler.java:7365)
	at clojure.lang.RT.compile(RT.java:398)
	at clojure.lang.RT.load(RT.java:438)
	at clojure.lang.RT.load(RT.java:411)
	at clojure.core$load$fn__5415.invoke(core.clj:5823)
	at clojure.core$load.doInvoke(core.clj:5822)
	at clojure.lang.RestFn.invoke(RestFn.java:408)
	at clojure.core$load_one.invoke(core.clj:5613)
	at clojure.core$compile$fn__5420.invoke(core.clj:5834)
	at clojure.core$compile.invoke(core.clj:5833)
	at user$eval5.invoke(form-init180441230737245034.clj:1)
	at clojure.lang.Compiler.eval(Compiler.java:6776)
	at clojure.lang.Compiler.eval(Compiler.java:6765)
	at clojure.lang.Compiler.eval(Compiler.java:6766)
	at clojure.lang.Compiler.load(Compiler.java:7203)
	at clojure.lang.Compiler.loadFile(Compiler.java:7159)
	at clojure.main$load_script.invoke(main.clj:274)
	at clojure.main$init_opt.invoke(main.clj:279)
	at clojure.main$initialize.invoke(main.clj:307)
	at clojure.main$null_opt.invoke(main.clj:342)
	at clojure.main$main.doInvoke(main.clj:420)
	at clojure.lang.RestFn.invoke(RestFn.java:421)
	at clojure.lang.Var.invoke(Var.java:383)
	at clojure.lang.AFn.applyToHelper(AFn.java:156)
	at clojure.lang.Var.applyTo(Var.java:700)
	at clojure.main.main(main.java:37)
Caused by: java.lang.NoSuchFieldError: __thunk__0__
	at instaparse.core__init.load(Unknown Source)
	at instaparse.core__init.<clinit>(Unknown Source)
	at java.lang.Class.forName0(Native Method)
	at java.lang.Class.forName(Class.java:344)
	at clojure.lang.RT.loadClassForName(RT.java:2141)
	at clojure.lang.RT.load(RT.java:430)
	at clojure.lang.RT.load(RT.java:411)
	at clojure.core$load$fn__5415.invoke(core.clj:5823)
	at clojure.core$load.doInvoke(core.clj:5822)
	at clojure.lang.RestFn.invoke(RestFn.java:408)
	at clojure.core$load_one.invoke(core.clj:5613)
	at clojure.core$load_lib$fn__5362.invoke(core.clj:5668)
	at clojure.core$load_lib.doInvoke(core.clj:5667)
	at clojure.lang.RestFn.applyTo(RestFn.java:142)
	at clojure.core$apply.invoke(core.clj:628)
	at clojure.core$load_libs.doInvoke(core.clj:5706)
	at clojure.lang.RestFn.applyTo(RestFn.java:137)
	at clojure.core$apply.invoke(core.clj:628)
	at clojure.core$require.doInvoke(core.clj:5789)
	at clojure.lang.RestFn.invoke(RestFn.java:436)
	at clout.core$loading__5301__auto____273.invoke(core.clj:1)
	at clout.core__init.load(Unknown Source)
	at clout.core__init.<clinit>(Unknown Source)
	at java.lang.Class.forName0(Native Method)
	at java.lang.Class.forName(Class.java:344)
	at clojure.lang.RT.loadClassForName(RT.java:2141)
	at clojure.lang.RT.load(RT.java:430)
	at clojure.lang.RT.load(RT.java:411)
	at clojure.core$load$fn__5415.invoke(core.clj:5823)
	at clojure.core$load.doInvoke(core.clj:5822)
	at clojure.lang.RestFn.invoke(RestFn.java:408)
	at clojure.core$load_one.invoke(core.clj:5613)
	at clojure.core$load_lib$fn__5362.invoke(core.clj:5668)
	at clojure.core$load_lib.doInvoke(core.clj:5667)
	at clojure.lang.RestFn.applyTo(RestFn.java:142)
	at clojure.core$apply.invoke(core.clj:628)
	at clojure.core$load_libs.doInvoke(core.clj:5706)
	at clojure.lang.RestFn.applyTo(RestFn.java:137)
	at clojure.core$apply.invoke(core.clj:628)
	at clojure.core$require.doInvoke(core.clj:5789)
	at clojure.lang.RestFn.invoke(RestFn.java:482)
	at compojure.core$loading__5301__auto____68.invoke(core.clj:1)
	at compojure.core__init.load(Unknown Source)
	at compojure.core__init.<clinit>(Unknown Source)
	at java.lang.Class.forName0(Native Method)
	at java.lang.Class.forName(Class.java:344)
	at clojure.lang.RT.loadClassForName(RT.java:2141)
	at clojure.lang.RT.load(RT.java:430)
	at clojure.lang.RT.load(RT.java:411)
	at clojure.core$load$fn__5415.invoke(core.clj:5823)
	at clojure.core$load.doInvoke(core.clj:5822)
	at clojure.lang.RestFn.invoke(RestFn.java:408)
	at clojure.core$load_one.invoke(core.clj:5613)
	at clojure.core$load_lib$fn__5362.invoke(core.clj:5668)
	at clojure.core$load_lib.doInvoke(core.clj:5667)
	at clojure.lang.RestFn.applyTo(RestFn.java:142)
	at clojure.core$apply.invoke(core.clj:628)
	at clojure.core$load_libs.doInvoke(core.clj:5706)
	at clojure.lang.RestFn.applyTo(RestFn.java:137)
	at clojure.core$apply.invoke(core.clj:628)
	at clojure.core$require.doInvoke(core.clj:5789)
	at clojure.lang.RestFn.invoke(RestFn.java:457)
	at compojure.route$loading__5301__auto____1508.invoke(route.clj:1)
	at clojure.lang.AFn.applyToHelper(AFn.java:152)
	at clojure.lang.AFn.applyTo(AFn.java:144)
	at clojure.lang.Compiler$InvokeExpr.eval(Compiler.java:3601)
	... 75 more
Subprocess failed
Comment by Michael Blume [ 06/Jan/15 4:06 PM ]

https://github.com/MichaelBlume/clojure/tree/no-field
https://github.com/MichaelBlume/extend-test/tree/no-field

mvn clean install in the one, lein ring uberwar in the other.

Comment by Nicola Mometto [ 06/Jan/15 6:09 PM ]

Michael, thanks for the report, I've tried investigating this a bit but the big amount of moving parts involved make it really hard to figure out why the combination of the two patches causes this issue.

A helpful minimal case would require no lein and no external dependencies, I'd appreciate some help in debugging this issue if anybody has time.

Comment by Michael Blume [ 06/Jan/15 10:56 PM ]

Ok, looks like the minimal case is

(ns foo (:require [instaparse.core]))

(ns bar (:require [foo]))

and then attempt to AOT-compile both foo and bar.

I don't yet know what's special about instaparse.core.

Comment by Michael Blume [ 06/Jan/15 11:30 PM ]

Well, not a minimal case, of course, but one without lein, at least.

Comment by Michael Blume [ 06/Jan/15 11:51 PM ]

ok, problem is instaparse's defclone macro, I've extracted it to a test repo

https://github.com/MichaelBlume/thunk-fail

lein do clean, compile will get you a failure, but the repo has no dependencies so I'm sure there's a way to do that without lein.

Comment by Ghadi Shayban [ 06/Jan/15 11:56 PM ]

Sorry for the barrage of questions, but these classloader bugs are subtle (and close to being solved I hope). Your report is immensely valuable, and yet it will help to be even more specific. There are a cluster of these bugs – and keeping them laser-focused is key.

The minimal case to which you refer is the NoSuchFieldError?
How are is this being invoked this without lein?
What are you calling to AOT? (compile 'bar) ?
What is the classpath? When you invoke originally, is ./target/classes empty?
Does the problem go away with CLJ-979-7 applied?

Comment by Michael Blume [ 07/Jan/15 12:16 AM ]

I have tried and failed to replicate without leiningen. When I just run

java -Dclojure.compile.path=target -cp src:../clojure/target/clojure-1.7.0-aot-SNAPSHOT.jar clojure.lang.Compile thunk-fail.first thunk-fail.second

everything works fine.

Comment by Ghadi Shayban [ 07/Jan/15 12:30 AM ]

The NoSuchFieldError is related to the keyword lookup sites.

Replacing defclone's body with
`(do (:foo {})) is enough to trigger it, with the same ns structure.

Comment by Nicola Mometto [ 07/Jan/15 4:47 AM ]

I have updated the patch for CLJ-1544, now the combination of the new patch + the patch from this ticket should not cause any exception.

That said, a bug in this patch still exists since while the patch for CLJ-1544 had a bug, it was causing a perfectly valid (albeit hardly reproducible) compilation scenario so we should keep debugging this patch with the help of the bugged patch for CLJ-1544.

I guess the first thing to do is figure out what lein compile is doing differently than clojure.Compile

Comment by Nicola Mometto [ 07/Jan/15 4:49 AM ]

Also Ghadi is right, infact replacing the whole body of thunk-fail.core with (:foo {}) is enough.

It would seem like the issue is with AOT (re)compiling top-level keyword lookup sites, my guess is that for some reason this patch is preventing correct generation of the __init static initializer.

Comment by Nicola Mometto [ 07/Jan/15 5:35 AM ]

I still have absolutely no idea what lein compile is doing but I figured out the issue.
The updated patch binds (in eval) the appropriate vars only when already bounded.

Comment by Alex Miller [ 07/Jan/15 9:00 AM ]

Would it be worth using transients on the bindings map now?

Comment by Nicola Mometto [ 07/Jan/15 9:11 AM ]

Makes sense, updated the patch to use a transient map

Comment by Michael Blume [ 07/Jan/15 12:25 PM ]

Is there a test we can add that'll fail in the presence of the v2 patch? preferably independent of the CLJ-1544 patch? I can try to write one myself, but I don't have a lot of familiarity with the Clojure compiler internals.

Comment by Nicola Mometto [ 07/Jan/15 12:32 PM ]

I'll have to think about a way to reproduce that bug, it's not a simple scenario to reproduce.
It involves compiling a namespace from an evaluated context.

Comment by Laurent Petit [ 15/Apr/15 11:14 AM ]

Hello, is there any chance left that this issue will make it to 1.7 ?

Comment by Alex Miller [ 15/Apr/15 11:18 AM ]

Wasn't planning on it - what's the impact for you?

Comment by Laurent Petit [ 29/Apr/15 2:14 PM ]

The impact is that I need to use a patched version of Clojure for CCW.
While it's currently not that hard to follow clojure's main branch and regularly rebase on it or reapply the patch, it's still a waste of time.

Comment by Alex Miller [ 29/Apr/15 2:31 PM ]

I will check with Rich whether it can be screened for 1.7 before we get to RC.

Comment by Alex Miller [ 29/Apr/15 3:49 PM ]

same as v4 patch, but just has more diff context

Comment by Laurent Petit [ 01/May/15 7:25 AM ]

the file mentioned in the patch field is not the right one IMHO

Comment by Alex Miller [ 01/May/15 8:42 AM ]

which one is?

Comment by Laurent Petit [ 01/May/15 8:58 AM ]

I think you previous comment relates to clj-1620-v5.patch, but at the end of the description there's the following line:

Patch: 0001-CLJ-1620-avoid-constants-leak-in-static-initalizer-v4.patch

Comment by Alex Miller [ 01/May/15 9:30 AM ]

Those patches are equivalent with respect to the change they introduce; they just differ in how much diff context they have.

Comment by Alex Miller [ 18/May/15 2:25 PM ]

Rich has ok'ed screening this one for 1.7 but I do not feel that I can mark it screened without understanding it much better than I do. The description, code, and cause information here is not sufficient for me to understand what the problem actually is or why the fix is the right one. The fix seems to address the symptom but I worry that it is just a symptom and that a better understanding of the actual cause would lead to a different or better fix.

The evolution of the patches was driven by bugs in CLJ-1544 (a patch which has been pulled out for being suspect for other reasons). Starting fresh, were those modifications necessary and correct?

Why does this set of vars need to push clean impls into the bindings? Why not some of the other vars (like those pushed in load())? The set chosen here seems to match that from the ReifyParser - why? Why should they only be pushed if they are bound (that is, why is "not bound" not the same as "bound but empty")? Are we affecting performance?

Popping all the way out, is the thing being done by CCW even a thing that should be doable? The description says "Compiling a function that references a non loaded (or uninitialized) class triggers its init static" - should this load even happen? Can we get an example that actually demonstrates what CCW was doing originally?

Comment by Laurent Petit [ 19/May/15 7:12 AM ]

Alex, the question of "should what CCW is doing be doable" can be answered if you answer it on the given example, I think.

The question "should the initialization of the class occur when it could just be loaded" is a good one. Several reports have been made on the Clojure list about this problem, and I guess there is at least one CLJ issue about changing some more classForName into classForNameNonLoading here and there in Clojure.
For instance, it prevents referencing java classes which have code in their static initializers as soon as the code does some supposition about the runtime it is initialized in. This is a problem with Eclipse / SWT, this a problem with Cursive as I remember Colin mentioning a similar issue. And will probably is a problem that can appear each time one tries to AOT compile clojure code interoperating with java classes who happen to have, somewhere within static initializers triggered by the compilation (and this is transitive), assumptions that they are initialized in the proper target runtime environment.

What I don't know is if preventing the initialization to occur in the first place would be sufficient to get rid of the class of problems this bug and the proposed patch tried to solve. I do not claim to totally what is happening either (Christophe and Nicolas were of great help to analyze the issue and create the patch), but as I understand it, it's a kind of "Inception-the-movie-like" bug. Compiling a fn which triggers compiling another fn (here through the loading of clojure namespaces via a java initializer).

If preventing the initialization of class static methods when they are referenced (through interop calls - constructor, field, method, static field, static method-) is the last remaining bit that could cause such "compilation during compilation" scenario, then yes, protecting the compilation process like Nicolas tried to do may not be necessary, and just fixing the undesired loading may be enough.





[CLJ-1614] Clojure does not start: ClassCastException Created: 12/Dec/14  Updated: 05/Jun/15  Resolved: 05/Jun/15

Status: Closed
Project: Clojure
Component/s: None
Affects Version/s: Release 1.7
Fix Version/s: None

Type: Defect Priority: Minor
Reporter: Vladimir Tsichevski Assignee: Unassigned
Resolution: Declined Votes: 0
Labels: compiler
Environment:

Eclipse RCP



 Description   

The clojure.lang.Compiler class static code throws the ClassCastException when reading compiler options from System properties (Compiler.java, line 260 in the git master release). When running Clojure from Eclipse RCP application the System properties may have non-string values.

Checking if the value is String and ignoring non-strings fixes this problem.



 Comments   
Comment by Nicola Mometto [ 05/Jun/15 10:02 AM ]

I believe this is a duplicate of CLJ-1717 which was declined.





[CLJ-1604] AOT'ed code that defs a var with clojure.core symbol name causes IllegalStateException Created: 25/Nov/14  Updated: 21/Feb/15  Resolved: 20/Feb/15

Status: Closed
Project: Clojure
Component/s: None
Affects Version/s: Release 1.7
Fix Version/s: Release 1.7

Type: Defect Priority: Critical
Reporter: Nicola Mometto Assignee: Unassigned
Resolution: Completed Votes: 3
Labels: aot, compiler

Attachments: Text File 0001-fix-AOT-bug-preventing-overriding-of-clojure.core-fu.patch     Text File 0001-fix-AOT-bug-preventing-overriding-of-clojure.core-fu-v2.patch     File 1604-context.diff    
Patch: Code
Approval: Ok

 Description   

AOT'ed code that defs a var that is also a symbol in clojure.core results in an exception at runtime. This problem can be avoided with (:refer-clojure :exclude ...) but this requires a library author to update and release a new version. AOT'ed applications must then wait for all transitive dependencies to update before they can update to a new Clojure version. For some users, this problem prevents them from trying or adopting new releases.

For example, the contrib library data.int-map defines an update function. clojure.core will also have a new update function as of 1.7.0. If this library is AOT'ed, then users of the clojure.data.int-map/update function will see the exception below. This situation can commonly occur when an application uses lein uberjar to compile all of the project+libs. In this case, applications or libraries that use data.int-map (either directly or indirectly) are affected.

java.lang.IllegalStateException: Attempting to call unbound fn: #'clojure.data.int-map/update
 at clojure.lang.Var$Unbound.throwArity (Var.java:43)
    clojure.lang.AFn.invoke (AFn.java:40)
    compiler_update_not_referenced_bug.core$foo.invoke (core.clj:5)

Reproduce with this sample project: https://github.com/yeller/compiler_update_not_referenced_bug

Cause: When AOT compiling a namespace, the def forms are hoisted into the ns__init class (in the example here, clojure.data.int_map__init). The static initializer in this class creates each var in the ns via a call to RT.var(ns, name). For data.int-map the static initializer will properly create the var for clojure.data.int-map/update. But when the ns is loaded (via the clojure.data.int_map.load() method), (refer-clojure) will be called, which will remap clojure.data.int-map/update to point to clojure.core/update.

This problem does not affect non-AOT loading (which doesn't use the ns__init class) and does not affect collisions from any other namespace. Only collisions from clojure.core create this possibility.

Proposed: The proposed patch explicitly refers the Var during ns__init.load() (after Clojure symbols are referred) rather than implicitly during ns__init static {}.

This change in behavior only happens during AOT in the specific case where a core symbol is being shadowed. In that case, clojure.core has already been loaded and v (the looked up var) will have ns=clojure.core. The currentNS will be (for example) data.int-map. If that's the case, and the sym has no ns, then the new logic will be emitted.

In the case of clojure.core itself, NO new bytecode is emitted. From tests on several projects, only shadowed vars during AOT get this additional bytecode.

Patch: 0001-fix-AOT-bug-preventing-overriding-of-clojure.core-fu-v2.patch

Screened by: Alex Miller



 Comments   
Comment by Andy Fingerhut [ 25/Nov/14 11:28 PM ]

When I try latest Clojure master plus patch CLJ-1604-only-core.patch with the small test project created by Tom Crayford to demonstrate this issue: https://github.com/yeller/compiler_update_not_referenced_bug

In that project, I get the same exception thrown when attempting 'lein do clean, uberjar, test' using this patch, as without it. It is because int-map/update in namespace compiler-update-not-referenced-bug.core is an unbound var.

Comment by Nicola Mometto [ 26/Nov/14 4:25 AM ]

Andy, you're right. For some reason I attached the wrong patch to the ticket, this is the correct one

Comment by Nicola Mometto [ 26/Nov/14 5:21 AM ]

I wasn't able to write a test for this, so here's a repl session using the clojure jar demonstrating this issue:

[˷/test]> ls
classes  clojure.jar  test.clj
[˷/test]> cat test.clj
(in-ns 'test)
(clojure.core/refer 'clojure.core)
(def foo "bar")
(def update "foo")
[˷/test]> java -cp classes:clojure.jar:. clojure.main
Clojure 1.7.0-master-SNAPSHOT
user=> (binding [*compile-files* true] (load "test"))
WARNING: update already refers to: #'clojure.core/update in namespace: test, being replaced by: #'test/update
nil
user=> test/foo
"bar"
user=> test/update
"foo"
user=>
[˷/test]> java -cp classes:clojure.jar:. clojure.main
Clojure 1.7.0-master-SNAPSHOT
user=> (load "test")
nil
user=> test/foo
"bar"
user=> test/update
CompilerException java.lang.RuntimeException: No such var: test/update, compiling: (NO_SOURCE_PATH:0:0)
user=>
Comment by Andy Fingerhut [ 26/Nov/14 10:39 AM ]

Thanks. I have not tried to assess the details of the change, other than to say that patch 0001-fix-AOT-bug-preventing-overriding-of-clojure.core-fu.patch dated 26 Nov 2014, when applied to latest Clojure master as of today, enables both 'lein do clean, test' and 'lein do clean, uberjar, test' to work as expected with Tom Crayford's test project, linked above, whereas 'lein do clean, uberjar, test' fails without this patch, due to a var being unbound that should have a value.

Comment by Andy Fingerhut [ 27/Nov/14 10:53 AM ]

Copying a comment here from CLJ-1591, since it is more appropriate here. It is responding to Tom Crayford's posting of his example project to demonstrate the issue: https://github.com/yeller/compiler_update_not_referenced_bug

Tom, looked at your project. Thanks for that. It appears not to have anything like (def inc inc) in it. It throws exception during test step of 'lein do clean, uberjar, test' consistently for me, too, but compiles with only warnings and passes tests with 'lein do clean, test'. I have more test results showing in which Clojure versions these results change. To summarize, the changes to Clojure that appear to make the biggest difference in the results are below (these should be added to the new ticket you create – you are welcome to do so):

Clojure 1.6.0, 1.7.0-alpha1, and later changes up through the commit with description "CLJ-1378: Allows FnExpr to override its reported class with a type hint": No errors or warnings for either lein command above.

Next commit with description "Add clojure.core/update, like update-in but takes a single key" that adds clojure.core/update: 'lein do clean, test' is fine, but 'lein do clean, uberjar' throws exception during compilation, probably due to CLJ-1241.

Next commit with description "fix CLJ-1241": 'lein do clean, test' and 'lein do clean, uberjar' give warnings about clojure.core/update, but no errors or exceptions. 'lein do clean, uberjar, test' throws exception during test step that is same as the one I see with Clojure 1.7.0-alpha4. Debug prints of values of clojure.core/update and int-map/update (in data.int-map and in Tom's namespace compiler-update-not-referenced-bug.core) show things look fine when printed inside data.int-map, and in Tom's namespace when not doing the uberjar, but when doing the uberjar, test, int-map/update is unbound in Tom's namespace.

In case it makes a difference, my testing was done with Mac OS X 10.9.5, Leiningen 2.5.0 on Java 1.7.0_45 Java HotSpot(TM) 64-Bit Server VM

Comment by Nicola Mometto [ 02/Dec/14 9:04 AM ]

The updated patch only emits the interning bytecode when necessary, avoiding the emission when a clojure.core var with the same name exists but is not mapped to the current namespace

Comment by Alex Miller [ 09/Jan/15 9:27 AM ]

Attached 1604-context.diff for purely informational purposes - same diff just more context in it for easier reading.

Comment by Tom Crayford [ 10/Jan/15 4:52 PM ]

Thought I'd add a minor note in here to say I tried testing this patch out on my app (which is where I discovered this AOT bug), and the bug doesn't turn up with this patch applied to clojure (tested by applying 0001-fix-AOT-bug-preventing-overriding-of-clojure.core-fu-v2.patch to 1.7-alpha5)

Comment by Adam Krieg [ 21/Feb/15 12:28 PM ]

I ran into this issue with Korma 0.4.0. I'm still running into it, but there is a twist.

My project depends on an artifact that was built with Clojure 1.7.0-alpha1. If I remove this dependency, everything is fine. However, with this dependency, I run into this issue, even if I declare a dependency on 1.7.0-master-SNAPSHOT in my project and exclude any dependency on clojure-1.7.0-alpha1.

I'm not sure if this is a Maven issue or a Clojure issue. Running Maven with debug on seems to show that it's using the correct version of Clojure.

I have created a dummy project that reproduces this issue if you are interested.

https://github.com/deaddowney/UpdateProblem

Check it out, run "mvn install", and you will get
java.lang.RuntimeException: No such var: korma.core/update
.





[CLJ-1598] Make if forms compile directly to the appropriate branch expression if the test is a literal Created: 24/Nov/14  Updated: 26/May/15

Status: Open
Project: Clojure
Component/s: None
Affects Version/s: None
Fix Version/s: None

Type: Enhancement Priority: Minor
Reporter: Nicola Mometto Assignee: Unassigned
Resolution: Unresolved Votes: 2
Labels: compiler, performance, primitives

Attachments: Text File 0001-if-test-expr-of-an-if-statement-is-a-literal-don-t-e.patch    
Patch: Code
Approval: Triaged

 Description   

This allows expressions like `(cond (some-expr) 1 :else 2)` to be eligible for unboxed use, which is not currently possible since the cond macro always ends up with a nil else branch that the compiler currently takes into account.

With the attached patch, the trailing (if :else 2 nil) in the macroexpansion will be treated as 2 by the Compiler, thus allowing the unboxed usage of the cond expression.






[CLJ-1596] Using keywords in place of symbols for defrecord fields causes a compiler exception with incorrect line number Created: 20/Nov/14  Updated: 25/Mar/15  Resolved: 25/Mar/15

Status: Closed
Project: Clojure
Component/s: None
Affects Version/s: Release 1.6
Fix Version/s: None

Type: Defect Priority: Minor
Reporter: Kyle Kingsbury Assignee: Unassigned
Resolution: Completed Votes: 0
Labels: compiler, defrecord

Approval: Triaged

 Description   

Possibly related to http://dev.clojure.org/jira/browse/CLJ-1261: a defrecord like

(defn foo [x])

(defrecord Bar [:b])

Throws an exception, like you'd expect:

java.lang.ClassCastException: clojure.lang.Keyword cannot be cast to clojure.lang.IObj, compiling:(tesser/quantiles_test.clj:45:15)

However, this exception's line and character indicates the error is in the previous form: the defn, not the defrecord. This can be really tricky to figure out when the expressions are more complicated.



 Comments   
Comment by Alex Miller [ 20/Nov/14 4:17 PM ]

Related: CLJ-1261

Comment by Alex Miller [ 20/Nov/14 4:18 PM ]

Possibly fixed by CLJ-1561, not sure.

Comment by Nicola Mometto [ 25/Mar/15 5:55 PM ]

No longer reproducible, fixed by CLJ-1561





[CLJ-1593] Use PAM for small maps when assigned to a var rather than always using PHMs Created: 15/Nov/14  Updated: 09/Mar/15  Resolved: 09/Mar/15

Status: Closed
Project: Clojure
Component/s: None
Affects Version/s: None
Fix Version/s: None

Type: Enhancement Priority: Minor
Reporter: Nicola Mometto Assignee: Unassigned
Resolution: Declined Votes: 0
Labels: collections, compiler, maps

Attachments: Text File 0001-Use-PAM-rather-than-always-using-PHMs-for-small-maps.patch    
Patch: Code

 Description   

I'm reproposing the fix I implemented for http://dev.clojure.org/jira/browse/CLJ-944 a while ago as an enhancement rather than as a defect.

Currently when a map is used as the value of a `def` expression, unless it's an empty map, it will always be a PersistentHashMap even if it's a small map.

user=> (def a {:foo :bar})
#'user/a
user=> (class a)
clojure.lang.PersistentHashMap

The current patch makes makes small maps be compiled to PAMs, consistently with how it's handled in lexical contexts, only using PHMs when the number of elements is above the threshold

user=> (def a {:foo :bar})
#'user/a
user=> (class a)
clojure.lang.PersistentArrayMap
user=> (class (let [a {:foo :bar}] a))
clojure.lang.PersistentArrayMap
user=> (def a {0 0 1 1 2 2 3 3 4 4 5 5 6 6 7 7 8 8 9 9})
#'user/a
user=> (class a)
clojure.lang.PersistentHashMap


 Comments   
Comment by Alex Miller [ 15/Nov/14 12:17 PM ]

This might be subsumed under the small collections CLJ-1517, not sure.

Comment by Nicola Mometto [ 08/Dec/14 9:19 AM ]

This is now out of scope for CLJ-1517 now that's focused only on vectors.

Comment by Alex Miller [ 08/Dec/14 9:47 AM ]

We're just splitting the ticket apart, maps will be a separate ticket/patch.

Comment by Nicola Mometto [ 09/Mar/15 1:40 PM ]

This change has been included by Rich in commit https://github.com/clojure/clojure/commit/692645c73c86d12c93a97c858dc6e8b0f4280a0b#diff-f17f860d14163523f1e1308ece478ddb





[CLJ-1586] Compiler doesn't preserve metadata for LazySeq literals Created: 12/Nov/14  Updated: 24/Aug/15

Status: Open
Project: Clojure
Component/s: None
Affects Version/s: None
Fix Version/s: Release 1.8

Type: Defect Priority: Major
Reporter: Nicola Mometto Assignee: Unassigned
Resolution: Unresolved Votes: 0
Labels: compiler, metadata, typehints

Attachments: Text File 0001-CLJ-1586-Compiler-doesn-t-preserve-metadata-for-lazy.patch     Text File 0001-Compiler-doesn-t-preserve-metadata-for-lazyseq-liter.patch     Text File clj-1586-2.patch    
Patch: Code and Test
Approval: Ok

 Description   

The analyzer in Compiler.java forces evaluation of lazyseq literals, but loses the compile time original metadata of that form, meaning that a type hint will be lost.

Example demonstrating this issue:

user=> (set! *warn-on-reflection* true)
true
user=> (list '.substring (with-meta (concat '(identity) '("foo")) {:tag 'String}) 0)
(.substring (identity "foo") 0)
user=> (eval (list '.substring (with-meta (concat '(identity) '("foo")) {:tag 'String}) 0))
Reflection warning, NO_SOURCE_PATH:6:1 - call to method substring on java.lang.Object can't be resolved (no such method).
"foo"

Forcing the concat call to an ASeq rather than a LazySeq fixes this issue:

user=> (eval (list '.hashCode (with-meta (seq (concat '(identity) '("foo"))) {:tag 'String})))
101574

This ticket blocks http://dev.clojure.org/jira/browse/CLJ-1444 since clojure.core/sequence might return a lazyseq.

This bug affected both tools.analyzer and tools.reader and forced me to commit a fix in tools.reader to work around this issue, see: http://dev.clojure.org/jira/browse/TANAL-99

The proposed patch trivially preserves the form metadata after realizing the lazyseq

Approach: Keep a copy of the original form, and apply its metadata to the realized lazyseq
Patch: clj-1586-2.patch
Screened by: Alex Miller



 Comments   
Comment by Stuart Halloway [ 31/Jul/15 9:19 AM ]

Please add a test case.

Comment by Nicola Mometto [ 31/Jul/15 9:32 AM ]

Updated patch with testcase

Comment by Alex Miller [ 11/Aug/15 10:55 AM ]

clj-1586-2.patch is same as prior, just updated to apply to current master





[CLJ-1573] Support (Java) transient fields in deftype, e.g. for hashcodes Created: 26/Oct/14  Updated: 29/Dec/14

Status: Open
Project: Clojure
Component/s: None
Affects Version/s: None
Fix Version/s: None

Type: Enhancement Priority: Minor
Reporter: Ghadi Shayban Assignee: Unassigned
Resolution: Unresolved Votes: 4
Labels: compiler, deftype

Attachments: Text File 0001-transient-field-deftype.patch    
Patch: Code and Test

 Description   

Enhance deftypes to allow fields to be marked ACC_TRANSIENT.

strawman syntax:
(deftype AType [^:transient hash])

Came across this need while experimenting with a reified range written in a deftype, not in Java.

Patch doesn't include docstring change, but has a test.



 Comments   
Comment by Adrian Medina [ 29/Dec/14 11:54 AM ]

Perhaps ^:transient-mutable would be a more appropriate modifier name to be consistent with the ^:unsynchronized-mutable and ^:volatile-mutable field modifiers. In any event, this feature would eliminate the need to drop down to Java for types that require transient fields.

Comment by Andy Fingerhut [ 29/Dec/14 12:07 PM ]

Roberto, there is a "Vote" word you can click on to actually vote for tickets, and ticket wranglers actually look at those votes at times to examine popular ones sooner. +1 comments don't do that.





[CLJ-1529] Significantly improve compile time by reducing calls to Class.forName Created: 21/Sep/14  Updated: 14/Nov/14  Resolved: 14/Nov/14

Status: Closed
Project: Clojure
Component/s: None
Affects Version/s: Release 1.7
Fix Version/s: Release 1.7

Type: Enhancement Priority: Critical
Reporter: Zach Tellman Assignee: Unassigned
Resolution: Completed Votes: 28
Labels: compiler, performance

Attachments: File class-for-name.diff     File clj-1529-no-cache-2.diff     File clj-1529-no-cache.diff     PNG File clj-1529.png     File clj-1529-with-cache.diff     Text File maybe-class-cache-2.patch     Text File maybe-class-cache.patch    
Patch: Code
Approval: Ok

 Description   

Compilation speed has been a real problem for a number of my projects, especially Aleph [1], which in 1.6 takes 18 seconds to load. Recently I realized that Class.forName is being called repeatedly on symbols which are lexically bound. Hits on Class.forName are cached, but misses are allowed to go through each time, which translates into tens of thousands of calls after calling `(use 'aleph.http)`.

Proposed: Avoid calling Class.forName() on non-namespaced symbols that do not contain "." or start with "[", don't map to a Class in the ns, and are names in the current local env. Also, adjust the ordering of checks in tagToClass to check for hints before checking for class.

[Note that the latest variant of the patch moves the check from the original patch farther down in the logic to avoid changing the semantics. This still yields virtually all of the performance gains. See comments for details.]

Patch: clj-1529-no-cache-2.diff

Screened by: Stu Halloway. Note that for this change the patch ended up being so small it is easier follow the code than the prose description.

[1] https://github.com/ztellman/aleph



 Comments   
Comment by Ghadi Shayban [ 21/Sep/14 4:30 PM ]

One of our larger projects (not macro-laden) just went from 36 seconds to 23 seconds to start with this patch.

Comment by Ramsey Nasser [ 03/Oct/14 12:34 PM ]

I ported this patch to Clojure-CLR for the Unity integration project and we have seen significant speedups as well. I too agree that this is the behavior I expect as a user.

Comment by Alex Miller [ 06/Oct/14 12:19 PM ]

I ran this on a variety of open-source projects. I didn't find that it produced any unexpected behavior or test errors. Most projects were about 10% faster to run equivalent of "lein test" with a few as high as 35% faster.

Comment by Alex Miller [ 07/Oct/14 12:52 PM ]

We're interested in comparing this and the class caching in fastload branch to get something in for 1.7. Next step is to extract a patch of the stuff in fastload so we can compare them better.

Comment by Alex Miller [ 07/Oct/14 4:06 PM ]

Add maybe class cache patch from fastload branch

Comment by Alex Miller [ 08/Oct/14 8:57 AM ]

Times below to run "time lein test" on a variety of projects with columns:

  • master = current 1.7.0 master
  • maybe-cache = maybe-class-cache.patch extracted from Rich's fastload branch
  • class-for-name = class-for-name.diff from Zach
  • % maybe-cache = % improvement for maybe-cache over master
  • % class-for-name = % improvement for class-for-name patch over master (sorted desc)

project,master,maybe-cache,class-for-name,% maybe-cache,% class-for-name
aleph,25.605,16.572,14.460,35.278,43.527
riemann,40.550,27.656,24.734,31.798,39.004
lamina,37.247,30.072,29.045,19.263,22.021
trapperkeeper,11.979,11.158,10.3,6.854,14.016
plumbing,73.777,68.388,66.922,7.304,9.292
cheshire,5.583,5.089,5.086,8.848,8.902
tools.analyzer,5.411,5.289,5.023,2.255,7.171
core.async,19.161,18.090,17.942,5.589,6.362
tools.reader,4.686,4.435,4.401,5.356,6.082
clara-rules,43.964,42.140,41.542,4.149,5.509
core.typed,158.885,154.954,151.445,2.474,4.683
instaparse,9.286,8.922,8.859,3.920,4.598
schema,45.3,43.914,43.498,3.060,3.978
mandoline,76.295,74.831,74.425,1.919,2.451

The summary is that both patches improve times on all projects. In most cases, the improvement from either is <10% but the first few projects have greater improvements. The class-for-name patch has a bigger improvement in all projects than the maybe-cache patch (but maybe-cache has no change in semantics).

Comment by Nicola Mometto [ 08/Oct/14 9:03 AM ]

Are the two patches mutually exclusive?

Comment by Alex Miller [ 08/Oct/14 9:35 AM ]

They are non-over-lapping. I have not considered whether they could both be applied or whether that makes any sense.

Comment by Alex Miller [ 08/Oct/14 9:53 AM ]

The two patches both essentially cut off the same hot code path, just at different points (class-for-name is earlier), so applying them both effectively should give you about the performance of class-for-name.

Comment by Alex Miller [ 08/Oct/14 2:14 PM ]

Added a picture of the data for easier consumption.

Comment by Deepak Giridharagopal [ 10/Oct/14 4:35 PM ]

One of our bigger projects saw a reduction of startup time of 16% with class-for-name, 14% with maybe-cache, and a whopping 23% with both patches applied. This was actually starting up the program, as opposed to running "lein test", FWIW.

Maybe it's worth re-running the benchmarks with a "both-patches" variant?

Comment by Alex Miller [ 10/Oct/14 5:28 PM ]

Hey Deepak, I did actually run some of them with both patches and saw times similar to class-for-name.

Were your times consistent across restarts? The times in the data above are the best of 3 trials for every data point (although they were relatively consistent).

Comment by Deepak Giridharagopal [ 10/Oct/14 6:08 PM ]

Hi Alex, the tests I ran did 20-iteration loops, and I took the mean (though it was pretty consistent between restarts). I can redo stuff and upload the raw data for you if that will help.

Comment by Deepak Giridharagopal [ 10/Oct/14 6:43 PM ]

So repeating the experiment several times does in fact behave as you suspected...apologies for my previous LOLDATA.

Comment by Alex Miller [ 24/Oct/14 3:01 PM ]

maybe-class-cache-2.patch removes some debugging stuff

Comment by Alex Miller [ 27/Oct/14 4:41 PM ]

I've done more testing and made mods to both patches and moved them closer together.

On the maybe-class-cache patch (new version = clj-1529-with-cache.diff):
1) I found that adding a final else branch that returned null was an improvement - this avoids caching things that will never hit in the future (Cons, PersistentList, Symbols with namespaces, etc). That's both a perf improvement (avoids hashing those things) and a memory improvement.
2) The tagToClass improvement from Zach's patch is orthogonal and also valid here so I added it.
3) I added Zach's check, but moved the placement lower so that it doesn't alter semantics. It helps avoid caching locals that aren't classes.

On the class-for-name patch (new version = clj-1529-no-cache.diff):
1) Same change as #3 above - moved check lower to avoid semantics change.

With these changes, both patches have tagToClass and local checks, neither patch changes semantics, and the only difference is whether to keep or remove the cache.

aleph timings (for "lein test"):

  • 1.7 master = 25.415 s
  • 1.7 + clj-1529-with-cache.diff = 14.329 s
  • 1.7 + clj-1529-no-cache.diff = 14.808 s

lamina timings (for "lein test"):

  • 1.7 master = 37.340 s
  • 1.7 + clj-1529-with-cache.diff = 28.680 s
  • 1.7 + clj-1529-no-cache.diff = 28.759 s

The cache helps slightly in both cases, but it does not seem worth adding the new dynamic var and unbounded memory use inherent in the cache.

Comment by Alex Miller [ 05/Nov/14 11:40 AM ]

Talked to Rich, focusing on no-cache patch. Added new version that fixes tabbing and restores Zach's name to the patch, which seems appropriate.





[CLJ-1516] Throw an exception if def name contains a dot Created: 29/Aug/14  Updated: 29/Aug/14

Status: Open
Project: Clojure
Component/s: None
Affects Version/s: Release 1.6
Fix Version/s: None

Type: Enhancement Priority: Minor
Reporter: Nicola Mometto Assignee: Unassigned
Resolution: Unresolved Votes: 0
Labels: compiler

Attachments: Text File 0001-throw-an-exception-on-def-names-containing-dots.patch    
Patch: Code
Approval: Triaged

 Description   

In this comment: http://dev.clojure.org/jira/browse/CLJ-1100?focusedCommentId=35510&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-35510 Rich said that Vars whose name contains a dot are not supported, but the current implementation allows their definition.
This patch makes `(def foo.bar)` throw a compile-time exception



 Comments   
Comment by Alex Miller [ 29/Aug/14 10:41 AM ]

I'm curious whether this breaks existing code in the wild.

Comment by Nicola Mometto [ 29/Aug/14 10:45 AM ]

I find this hard to believe given the current behaviour:

user=> (def a.b 1)
#'user/a.b
user=> a.b
CompilerException java.lang.ClassNotFoundException: a.b, compiling:(NO_SOURCE_PATH:0:0)

one would need to go out of his way and refer to the var namespace qualified everywhere to make it work

Comment by Nicola Mometto [ 29/Aug/14 11:03 AM ]

After a brief conversation on #clojure, I updated the patch to only throw on non-macro defs so that macros like clojure.core/.. and clojure.core.incubator/.?. will work fine





[CLJ-1492] PersistentQueue objects are improperly eval'd and compiled Created: 06/Aug/14  Updated: 07/Aug/14

Status: Open
Project: Clojure
Component/s: None
Affects Version/s: Release 1.6, Release 1.7
Fix Version/s: None

Type: Defect Priority: Minor
Reporter: Jon Distad Assignee: Unassigned
Resolution: Unresolved Votes: 0
Labels: compiler
Environment:

OS X 10.9.4
java version "1.7.0_60"
Java(TM) SE Runtime Environment (build 1.7.0_60-b19)
Java HotSpot(TM) 64-Bit Server VM (build 24.60-b09, mixed mode)


Attachments: Text File 0001-Exclude-PersistentQueue-from-IPersistentList-eval-co.patch    
Patch: Code and Test
Approval: Triaged

 Description   

PersistentQueue objects do not follow the correct evaluation path in the Compiler.

The simplest case:

user=> (def q (conj clojure.lang.PersistentQueue/EMPTY 1 2 3))
#'user/q
user=> q
#<PersistentQueue clojure.lang.PersistentQueue@7861>
user=> (eval q)
CompilerException java.lang.ClassCastException: clojure.lang.PersistentQueue cannot be cast to java.util.List, compiling:(NO_SOURCE_PATH:4:1)

And you get the same exception when embedding a PersistentQueue:

user=> (eval `(fn [] ~q))
CompilerException java.lang.ClassCastException: clojure.lang.PersistentQueue cannot be cast to java.util.List, compiling:(NO_SOURCE_PATH:2:1)

Instead of the expected:

CompilerException java.lang.RuntimeException: Can't embed unreadable object in code: #<PersistentQueue clojure.lang.PersistentQueue@7861>, compiling:(NO_SOURCE_PATH:3:1)

Since PersistentQueue implements IPersistentCollection and IPersistentList, and is not called out explicitly in the compiler, it is falling into the same compile path as a list. The exception comes from the call to emitValue inside the emitConstants portion of the FnExpr emit path. PersistentQueue does not implement java.util.List and thus the cast in emitListAsObjectArray (Compiler.java:4479) throws. Implementing List would NOT, however, resolve this issue, but would mask it by causing all eval'd PersistedQueues to be compiled as PersistentLists.

The first case is resolved by adding `&& !(form instanceof PersistentQueue)` to the IPersistentCollection branch of Compiler.eval() (Compiler.java:6695-8), allowing the PersistentQueue to fall through to the ConstantExpr case in analyze (Compiler.java:6459). The embedding case is resolved by adding `&& !(value instanceof PersistentQueue)` to the IPersistentList branch in ObjExpr's emitValue (Compiler.java:4639).

This bug also precludes definition of data-readers for PersistentQueue as the read object throws an exception when it is passed to the Compiler.

The attached patch includes the two changes mentioned above, and tests for each case that illustrates the bug.

Clojure-dev thread: https://groups.google.com/forum/#!topic/clojure-dev/LDUQfqjFg9w






[CLJ-1491] External type hint inconsistency between regular functions and primitive functions Created: 05/Aug/14  Updated: 25/Sep/14  Resolved: 25/Sep/14

Status: Closed
Project: Clojure
Component/s: None
Affects Version/s: Release 1.6
Fix Version/s: None

Type: Defect Priority: Minor
Reporter: Gunnar Völkel Assignee: Unassigned
Resolution: Duplicate Votes: 0
Labels: compiler, typehints

Attachments: Text File 0001-preserve-fn-meta-on-invokePrim.patch    
Patch: Code
Approval: Triaged

 Description   

Consider the following example.

(set! *warn-on-reflection* true)

(defn f [n] (java.util.ArrayList. (int n)))

(let [al ^java.util.ArrayList (f 10)]
  (.add al 23))

As expected this does not warn about reflection. The following example shows the same scenario for a primitive function.

(set! *warn-on-reflection* true)

(defn g [^long n] (java.util.ArrayList. n))

(let [al ^java.util.ArrayList (g 10)]
  (.add al 23))
; Reflection warning, NO_SOURCE_PATH:2:3 - call to method add on java.lang.Object can't be resolved (no such method).

So the behavior of external type hints is inconsistent for regular functions and primitive functions.
Most likely, the external type hint information is somehow ignored for primitive functions since the case where they return no primitive value is not treated separately.



 Comments   
Comment by Nicola Mometto [ 05/Aug/14 4:32 AM ]

The following patch preserves the original metadata of the invoke form on the transformed .invokePrim expression

Comment by Alex Miller [ 05/Aug/14 7:40 AM ]

Not challenging the premise at all but workaround:

(let [^java.util.ArrayList al (g 10)]
  (.add al 23))
Comment by Gunnar Völkel [ 05/Aug/14 8:09 AM ]

Well, the example above was already changed such that you can also place the type hint on the binding to check whether that works.
The actual problem arose when using the return value of the function exactly once without an additional binding.

Comment by Jozef Wagner [ 05/Aug/14 10:48 AM ]

Responding to Alex's comment, is there a consensus on which variant is (more) idiomatic? IMHO latter variant seems to be more reliable (as this issue shows, and for primitive hits too), and is consistent with 'place hint on a symbol' idiom which is applied when type hinting vars or fn args.

(let [symbol ^typehint expr] body)
(let [^typehint symbol expr] body)
Comment by Alex Miller [ 05/Aug/14 4:59 PM ]

They have different meanings. Generally the latter covers some cases that the former does not so it's probably the better one. I believe one of the cases is that if expr is a macro, the typehint is lost in the former.

Comment by Nicola Mometto [ 25/Sep/14 9:59 AM ]

The patch for http://dev.clojure.org/jira/browse/CLJ-1533?jwupdated=61127&focusedCommentId=35814 fixes this issue and more and should be preferred over this

Comment by Alex Miller [ 25/Sep/14 10:31 AM ]

Dupe of CLJ-1533





[CLJ-1475] :post condition causes compiler error with recur Created: 25/Jul/14  Updated: 29/Jul/14

Status: Open
Project: Clojure
Component/s: None
Affects Version/s: Release 1.6
Fix Version/s: None

Type: Defect Priority: Minor
Reporter: Steve Miner Assignee: Unassigned
Resolution: Unresolved Votes: 0
Labels: compiler

Attachments: File clj-1475.diff    
Patch: Code and Test
Approval: Triaged

 Description   

Michael O'Keefe <michael.p.okeefe@gmail.com> posted on the mailing list an example of code that causes a compiler error only if a :post condition is added. Here's my slightly modified version:

(defn g
  [xs acc]
  {:pre [(or (nil? xs) (sequential? xs))]
   :post [(number? %)]}
  (if (seq xs)
     (recur (next xs) (+ (first xs) acc))
     acc))

CompilerException java.lang.UnsupportedOperationException: Can only recur from tail position

The work-around is to wrap the body in a loop that simply rebinds the original args.



 Comments   
Comment by Steve Miner [ 25/Jul/14 9:53 AM ]

A macro expansion shows that body is placed in a let form to capture the result for later testing with the post condition, but the recur no longer has a proper target. The work-around of using a loop form is easy once you understand what's happening but it's a surprising limitation.

Comment by Steve Miner [ 25/Jul/14 9:55 AM ]

Use a local fn* around the body and call it with the original args so that the recur has a proper target. Update: not good enough for handling destructuring. Patch withdrawn.

Comment by Michael Patrick O'Keefe [ 25/Jul/14 10:37 AM ]

Link to the original topic discussion: https://groups.google.com/d/topic/clojure/Wb1Nub6wVUw/discussion

Comment by Steve Miner [ 25/Jul/14 1:42 PM ]

Patch withdrawn because it breaks on destructured args.

Comment by Steve Miner [ 25/Jul/14 5:27 PM ]

While working on a patch, I came up against a related issue: Should the :pre conditions apply to every recur "call". Originally, I thought the :pre conditions should be checked just once on the initial function call and never during a recur. People on the mailing list pointed out that the recur is semantically like calling the function again so the :pre checks are part of the contract. But no one seemed to want the :post check on every recursion, so the :post would happen only at the end.

That means automatically wrapping a loop (or nested fn* call) around the body is not going to work for the :pre conditions. A fix would have to bring the :pre conditions inside the loop.

Comment by Steve Miner [ 26/Jul/14 8:54 AM ]

I'm giving up on this bug. My approach was adding too much complexity to handle an edge case. I recommend the "loop" work-around to anyone who runs into this problem.

(defn g2
  [xs acc]
  {:pre [(or (nil? xs) (sequential? xs))]
   :post [(number? %)]}
  (loop [xs xs acc acc]
    (if (seq xs)
       (recur (next xs) (+ (first xs) acc))
       acc)))
Comment by Ambrose Bonnaire-Sergeant [ 26/Jul/14 10:29 AM ]

Add patch that handles rest arguments and destructuring.

Comment by Michael Patrick O'Keefe [ 26/Jul/14 10:57 AM ]

With regard to Steve's question on interpreting :pre, to me I would expect g to act like the case g3 below which uses explicit recursion (which does work and does appear to check the :pre conditions each time and :post condition once):

(defn g3
  [xs acc]
  {:pre [(or (sequential? xs) (nil? xs)) (number? acc)]
   :post [(number? %)]}
  (if (seq xs)
    (g3 (next xs) (+ (first xs) acc))
    acc))
Comment by Ambrose Bonnaire-Sergeant [ 26/Jul/14 11:42 AM ]

Patch clj-1475.diff handles destructuring, preconditions and rest arguments

Comment by Steve Miner [ 26/Jul/14 4:04 PM ]

The clj-1475.diff patch looks good to me.

Comment by Alex Miller [ 27/Jul/14 7:18 AM ]

Please don't use "patch" as a label - that is the purpose of the Patch field. There is a list of good and bad labels at http://dev.clojure.org/display/community/Creating+Tickets

Comment by Steve Miner [ 27/Jul/14 11:32 AM ]

More knowledgeable commenters might take a look at CLJ-701 just in case that's applicable to the proposed patch.

Comment by Kevin Downey [ 29/Jul/14 1:35 AM ]

re clj-701

it is tricky to express loop expression semantics in jvm byte code, so the compiler sort of punts, hoisting expression loops in to anonymous functions that are immediately invoked, closing over whatever is in scope that is required by the loop, this has some problems like those seen in CLJ-701, losing type data which the clojure compiler doesn't track across functions, the additional allocation of function objects (the jit may deal with that pretty well, I am not sure) etc.

where the world of clj-701 and this ticket collide is the patch on this ticket lifts the function body out as a loop expression, which without the patch in clj-701 will have the issues I listed above, but we already have those issues anywhere something that is difficult to express in bytecode as an expression (try and loop) is used as an expression, maybe it doesn't matter, or maybe clj-701 will get fixed in some way to alleviate those issues.

general musings

it seems like one feature people like from asserts is the ability to disable them in production (I have never actually seen someone do that with clojure), assert and :pre/:post have some ability to do that (it may only work at macroexpansion time, I don't recall) since the hoisting of the loop could impact performance it might be nice to have some mechanism to disable it (maybe using the same flag assert does?).





[CLJ-1463] Providing own ClassLoader for eval is broken Created: 10/Jul/14  Updated: 21/Jul/15

Status: Open
Project: Clojure
Component/s: None
Affects Version/s: Release 1.2, Release 1.3, Release 1.5, Release 1.6
Fix Version/s: None

Type: Enhancement Priority: Minor
Reporter: Volkert Oakley Jurgens Assignee: Unassigned
Resolution: Unresolved Votes: 0
Labels: compiler
Environment:

Clojure 1.6.0



 Description   

clojure.lang.Compiler has a method with the signature

public static Object eval(Object form, boolean freshLoader)

but the freshLoader argument is ignored since https://github.com/clojure/clojure/commit/2c2ed386ed0f6f875342721bdaace908e298c7f3

Is there a good reason this still needs to be "hotfixed" like this?

We would like to provide our own ClassLoader for eval to manage the lifecycle of the generated classes.



 Comments   
Comment by Stuart Halloway [ 21/Jul/15 8:04 AM ]

This is not part of the public API of Clojure. We would need to understand more about the use case.





[CLJ-1456] The compiler ignores too few or too many arguments to throw Created: 30/Jun/14  Updated: 04/May/15

Status: Open
Project: Clojure
Component/s: None
Affects Version/s: Release 1.6
Fix Version/s: None

Type: Enhancement Priority: Major
Reporter: Alf Kristian Støyle Assignee: Unassigned
Resolution: Unresolved Votes: 0
Labels: compiler, ft

Attachments: Text File clj-1456-4.patch     Text File v3_0001-CLJ-1456-counting-forms-to-catch-malformed-throw-for.patch    
Patch: Code and Test
Approval: Triaged

 Description   

The compiler does not fail on "malformed" throw forms:

user=> (defn foo [] (throw))
#'user/foo

user=> (foo)
NullPointerException   user/foo (NO_SOURCE_FILE:1)

user=> (defn bar [] (throw Exception baz))
#'user/bar

user=> (bar)
ClassCastException java.lang.Class cannot be cast to java.lang.Throwable  user/bar (NO_SOURCE_FILE:1)

; This one works, but ignored-symbol, should probably not be ignored
user=> (defn quux [] (throw (Exception. "Works!") ignored-symbol))
#'user/quux

user=> (quux)
Exception Works!  user/quux (NO_SOURCE_FILE:1)

The compiler can easily avoid these by counting forms.

Patch: clj-1456-4.patch

Screened by: Alex Miller



 Comments   
Comment by Alf Kristian Støyle [ 30/Jun/14 11:56 AM ]

Not sure how to create a test for the attached patch. Will happily do so if anyone has a suggestion.

Comment by Alex Miller [ 30/Jun/14 12:23 PM ]

Re testing, I think the examples you give are good - you should add tests to test/clojure/test_clojure/compilation.clj that eval the form and expect compilation errors. I'm sure you can find similar examples.

Comment by Alf Kristian Støyle [ 30/Jun/14 2:01 PM ]

Newest patch also contains a few tests.

Comment by Andy Fingerhut [ 29/Aug/14 4:54 PM ]

All patches dated Jun 30 2014 and earlier no longer applied cleanly to latest master after some commits were made to Clojure on Aug 29, 2014. They did apply cleanly before that day.

I have not checked how easy or difficult it might be to update this patch. See section "Updating Stale Patches" on this wiki page for some tips on updating patches: http://dev.clojure.org/display/community/Developing+Patches

Alf, it can help avoid confusion if different patches have different file names. JIRA lets you create multiple attachments with the same name, but I wouldn't recommend it.

Comment by Alf Kristian Støyle [ 30/Aug/14 2:18 AM ]

It was easy to fix the patch. Uploaded the new patch v3_0001-CLJ-1456-counting-forms-to-catch-malformed-throw-for.patch, which applies cleanly to the current master.

Comment by Andy Fingerhut [ 08/Jan/15 6:07 PM ]

Alf, while JIRA can handle multiple attachments for the same ticket with the same name, it can get confusing for people trying to determine which one with the same name is meant. Could you remove or rename one of your identically-named attachments? Instructions for deleting patches are in the "Removing patches" section on this wiki page: http://dev.clojure.org/display/community/Developing+Patches

Comment by Alf Kristian Støyle [ 10/Jan/15 9:12 AM ]

Removed both obsolete attachments. So shouldn't be confusing any more

Comment by Alex Miller [ 04/May/15 9:31 AM ]

-4 patch is same, just refreshed to apply to master





[CLJ-1440] Unable to exclude clojure.lang.Compiler using :refer-clojure Created: 06/Jun/14  Updated: 07/Jun/14  Resolved: 07/Jun/14

Status: Closed
Project: Clojure
Component/s: None
Affects Version/s: Release 1.6
Fix Version/s: None

Type: Defect Priority: Major
Reporter: Howard Lewis Ship Assignee: Unassigned
Resolution: Declined Votes: 0
Labels: compiler, interop


 Description   
(ns io.aviso.twixt.js-minification
  "Provides support for JavaScript minification using the Google Closure compiler."
  (:refer-clojure :exclude [Compiler])
  (:import (com.google.javascript.jscomp CompilerOptions ClosureCodingConvention DiagnosticGroups CheckLevel
                                         SourceFile Result Compiler))
  (:require [clojure.java.io :as io]
            [io.aviso.twixt.utils :as utils]
            [io.aviso.tracker :as t]
            [clojure.string :as str]))

Results in:

clojure.lang.Compiler$CompilerException: java.lang.IllegalStateException: Compiler already refers to: class clojure.lang.Compiler in namespace: io.aviso.twixt.js-minification, compiling:(/Users/hlship/workspaces/annadale/twixt/src/io/aviso/twixt/js_minification.clj:1:1)
        java.lang.IllegalStateException: Compiler already refers to: class clojure.lang.Compiler in namespace: io.aviso.twixt.js-minification
                                     clojure.lang.Namespace.referenceClass                    Namespace.java:  140
                                        clojure.lang.Namespace.importClass                    Namespace.java:  158
                                        clojure.lang.Namespace.importClass                    Namespace.java:  164
                   io.aviso.twixt.js-minification/eval4104/loading--auto--               js_minification.clj:    1
                                   io.aviso.twixt.js-minification/eval4104               js_minification.clj:    1
                                                clojure.lang.Compiler.eval                     Compiler.java: 6703
                                                clojure.lang.Compiler.eval                     Compiler.java: 6692
                                                clojure.lang.Compiler.load                     Compiler.java: 7130
                                   io.aviso.twixt.js-minification/eval4100  form-init4106199735960171933.clj:    1
                                                clojure.lang.Compiler.eval                     Compiler.java: 6703
                                                clojure.lang.Compiler.eval                     Compiler.java: 6666
                                                         clojure.core/eval                          core.clj: 2927
                                      clojure.main/repl/read-eval-print/fn                          main.clj:  239
                                         clojure.main/repl/read-eval-print                          main.clj:  239
                                                      clojure.main/repl/fn                          main.clj:  257
                                                         clojure.main/repl                          main.clj:  257
                                                clojure.lang.RestFn.invoke                       RestFn.java: 1096
             clojure.tools.nrepl.middleware.interruptible-eval/evaluate/fn            interruptible_eval.clj:   56
                                            clojure.lang.AFn.applyToHelper                          AFn.java:  152
                                                  clojure.lang.AFn.applyTo                          AFn.java:  144
                                                        clojure.core/apply                          core.clj:  624
                                               clojure.core/with-bindings*                          core.clj: 1862
                                                clojure.lang.RestFn.invoke                       RestFn.java:  425
                clojure.tools.nrepl.middleware.interruptible-eval/evaluate            interruptible_eval.clj:   41
clojure.tools.nrepl.middleware.interruptible-eval/interruptible-eval/fn/fn            interruptible_eval.clj:  171
                                                      clojure.core/comp/fn                          core.clj: 2402
             clojure.tools.nrepl.middleware.interruptible-eval/run-next/fn            interruptible_eval.clj:  138
                                                      clojure.lang.AFn.run                          AFn.java:   22
                         java.util.concurrent.ThreadPoolExecutor.runWorker           ThreadPoolExecutor.java: 1145
                        java.util.concurrent.ThreadPoolExecutor$Worker.run           ThreadPoolExecutor.java:  615
                                                      java.lang.Thread.run                       Thread.java:  724


 Comments   
Comment by Nicola Mometto [ 06/Jun/14 4:52 PM ]

refer and thus refer-clojure only works for Vars.
a workaround is:

(ns ..)
(ns-unmap *ns* 'Compiler)
(import 'com.google.javascript.jscomp.Compiler)
Comment by Alex Miller [ 07/Jun/14 11:34 AM ]

Ditto what Nicola said. Or just fully-qualify.





[CLJ-1422] Recur around try boxes primitives Created: 14/May/14  Updated: 28/Jul/14

Status: Open
Project: Clojure
Component/s: None
Affects Version/s: Release 1.5, Release 1.6
Fix Version/s: None

Type: Enhancement Priority: Minor
Reporter: Kyle Kingsbury Assignee: Unassigned
Resolution: Unresolved Votes: 0
Labels: compiler, performance, typehints


 Description   

Primitive function and recur variables can't pass through a (try) cleanly; they're boxed to Object instead. This causes reflection warnings for fns or loops that use primitive types.

user=> (set! *warn-on-reflection* true)
true
 
user=> (fn [] (loop [t 0] (recur t)))
#<user$eval676$fn__677 user$eval676$fn__677@3d80023a>
 
user=> (fn [] (loop [t 0] (recur (try t))))
NO_SOURCE_FILE:1 recur arg for primitive local: t is not matching primitive, had: Object, needed: long
Auto-boxing loop arg: t
#<user$eval680$fn__681 user$eval680$fn__681@5419323a>

user=> (fn [^long x] (recur (try x)))
NO_SOURCE_FILE:1 recur arg for primitive local: x is not matching primitive, had: Object, needed: long

CompilerException java.lang.IllegalArgumentException:  recur arg for primitive local: x is not matching primitive, had: Object, needed: long, compiling:(NO_SOURCE_PATH:1:1)


 Comments   
Comment by David James [ 15/Jun/14 10:27 PM ]

Without commenting on the most desirable behavior, the following code does not cause reflection warnings:

user=> (set! *warn-on-reflection* true)
true
user=> (fn [] (loop [t 0] (recur (long (try t)))))
#<user$eval673$fn__674 user$eval673$fn__674@4e56c411>
Comment by Nicola Mometto [ 16/Jun/14 6:33 AM ]

Similar ticket http://dev.clojure.org/jira/browse/CLJ-701

Comment by Kevin Downey [ 21/Jul/14 6:59 PM ]

try/catch in the compiler only implements Expr, not MaybePrimitiveExpr, looking at extending TryExpr with MaybePrimitiveExpr it seems simple enough, but it turns out recur analyzes it's arguments in the statement context, which causes (try ...) to essentially wrap itself in a function like ((fn [] (try ...))), at which point it is an invokeexpr which is much harder to add maybeprimitiveexpr too and it reduces to the same case as CLJ-701

Comment by Kevin Downey [ 22/Jul/14 9:27 PM ]

http://dev.clojure.org/jira/browse/CLJ-701 has a patch that I think solves this

Comment by Alex Miller [ 28/Jul/14 1:56 PM ]

Should I dupe this to CLJ-701?

Comment by Kevin Downey [ 28/Jul/14 5:22 PM ]

if you want the fixes for try out of the return context to be part of CLJ-701 then yes it is a dupe, if you are unsure or would prefer 701 to stay more focused (my patch may not be acceptable, or may be too large and doing too much) then no it wouldn't be a dupe. I sort of took it on myself to solve both in the patch on CLJ-701 because I came to CLJ-701 via Nicola's comment here, and the same compiler machinery can be used for both.

I think the status is pending on the status of CLJ-701.





[CLJ-1411] Special symbols can be shadowed inconsistently Created: 28/Apr/14  Updated: 29/Apr/14

Status: Open
Project: Clojure
Component/s: None
Affects Version/s: Release 1.6
Fix Version/s: None

Type: Defect Priority: Minor
Reporter: Volkert Oakley Jurgens Assignee: Unassigned
Resolution: Unresolved Votes: 0
Labels: compiler


 Description   

The compiler does not complain about let binding (or def-ing) special symbols, but the binding only works if not used at the beginning of a list:

These work:

(let [try :a]
  try)
=> :a
(let [try (constantly :a)]
  (apply try :b))
=> :a

This doesn't work:

(let [try (constantly :a)]
  (try :b))
=> :b

This is true for all special symbols, not just publicly exposed ones like try and new, but also internal ones like fn*.

I would expect consistent behaviour: either the compiler does not permit shadowing special symbols at all, or shadowing them works in all cases.



 Comments   
Comment by Nicola Mometto [ 28/Apr/14 10:01 AM ]

I don't think that shadowing special symbols is a good idea, but probably having all the special symbols namespace qualified (clojure.core/import* is the only one ns-qualified atm) along with checking for the symbol in the locals env first and fallbacking to the special symbols map after that, would probably help in those scenarios

Comment by Volkert Oakley Jurgens [ 29/Apr/14 12:48 AM ]

I think that shadowing special symbols is a bad idea. If that was possible, we'd have to change most macros in clojure.core to make them safe (i.e. explicitly add a namespace to each special symbol usage). And how would we handle special symbols that are not just implementation specific, like try and new? Every 3rd party macro that uses those might become unsafe.

My personal preference would be to prohibit the shadowing of special symbols.

Comment by Nicola Mometto [ 29/Apr/14 5:37 AM ]

That won't be the case since what I'm proposing includes making syntax-quote aware of the namespaced special symbols.
`def would expand to 'clojure.core/def for example.

Comment by Volkert Oakley Jurgens [ 29/Apr/14 5:58 AM ]

That's true, but macros don't have to use the syntax quote. See for example the definition of when.





[CLJ-1407] Recur mismatch might cause multiple evaluation Created: 17/Apr/14  Updated: 17/Apr/14

Status: Open
Project: Clojure
Component/s: None
Affects Version/s: None
Fix Version/s: None

Type: Defect Priority: Minor
Reporter: Nicola Mometto Assignee: Unassigned
Resolution: Unresolved Votes: 0
Labels: compiler, macro


 Description   

Since mismatching recurs cause the loop body to be re-analyzed, macroexpansion in the loop body might happen more than once, causing any side effects that happen during macroexpansion to be evaluated potentially multiple times

Clojure 1.7.0-master-SNAPSHOT
user=> (defmacro x [] (println "foo"))
#'user/x
user=> (fn [] (loop [y 1] (x) (recur (Integer. 1))))
foo
foo
#<user$eval6$fn__7 user$eval6$fn__7@71687585>


 Comments   
Comment by Andy Fingerhut [ 17/Apr/14 6:59 PM ]

This is not a question about whether the behavior in the description is a bug or not, but rather a curiosity about how often people write macros that have side effects at macroexpansion time. I think the following in Clojure itself do, but there may be others:

  • gen-class, and also ns because it uses gen-class
  • gen-interface, and also definterface because it uses gen-interface
  • clojure.core/compile-if (private) calls eval on its expr arg, but as used now doesn't cause macroexpansion-time side effects
  • doc seems to have one case that prints at macroexpansion time
  • I am not sure whether defprotocol or deftype have macroexpansion time side effects, or whether they are limited to run time
Comment by Nicola Mometto [ 17/Apr/14 9:20 PM ]

Andy, I don't think there are that many macros that side-effect at macroexpansion time and I haven't discovered this bug in real code but while thinking about how loop locals invalidation was implemented in Compiler.java.

Because there are a really a small number of side-effecting macros, this is unlikely to cause problems in real code, so I changed the priority to minor.





[CLJ-1401] CompilerException / IllegalStateException when overriding vars Created: 10/Apr/14  Updated: 31/Jul/15

Status: Reopened
Project: Clojure
Component/s: None
Affects Version/s: Release 1.6
Fix Version/s: None

Type: Enhancement Priority: Minor
Reporter: Mike Anderson Assignee: Unassigned
Resolution: Unresolved Votes: 0
Labels: compiler, errormsgs

Approval: Triaged

 Description   
=> (ns foo)
nil
=> (def a 1)
#'foo/a
=> (ns bar (:require [foo :refer :all]))
nil
=> (def a 2)
CompilerException java.lang.IllegalStateException: a already refers to: #'foo/a in namespace: bar, compiling:(NO_SOURCE_PATH:4:1)
	clojure.lang.Compiler.analyzeSeq (Compiler.java:6745)
	clojure.lang.Compiler.analyze (Compiler.java:6529)
	clojure.lang.Compiler.analyze (Compiler.java:6490)
	clojure.lang.Compiler.eval (Compiler.java:6801)
	clojure.lang.Compiler.eval (Compiler.java:6760)
	clojure.core/eval (core.clj:3079)
	clojure.main/repl/read-eval-print--7095/fn--7098 (main.clj:240)
	clojure.main/repl/read-eval-print--7095 (main.clj:240)
	clojure.main/repl/fn--7104 (main.clj:258)
	clojure.main/repl (main.clj:258)
	clojure.main/repl-opt (main.clj:324)
	clojure.main/main (main.clj:422)
Caused by:
IllegalStateException a already refers to: #'foo/a in namespace: bar
	clojure.lang.Namespace.warnOrFailOnReplace (Namespace.java:88)
	clojure.lang.Namespace.intern (Namespace.java:72)
	clojure.lang.Compiler$DefExpr$Parser.parse (Compiler.java:534)
	clojure.lang.Compiler.analyzeSeq (Compiler.java:6738)
	clojure.lang.Compiler.analyze (Compiler.java:6529)

I would expect (at worst) a similar warning to the initial namespace loading, rather than an exception here.



 Comments   
Comment by Alex Miller [ 11/Apr/14 8:26 AM ]

Could you put together a better reproducible test case for this that does not depend on core.matrix? Also, please include the (pst *e) when it occurs.

Comment by Andy Fingerhut [ 11/Apr/14 10:19 AM ]

I have tried the smallest possible Leiningen project I could think of that would cause the warnings about redefinitions, to see if I could get the exception to occur. 'lein new try1' to create the skeleton project, then edit src/try1/core.clj to contain only the following function definitions:

(defn merge
  "This definition of merge replaces clojure.core/merge"
  [x y]
  (- x y))

(defn *
  [x y]
  (* x y))

Then start a REPL with 'lein repl', and I see this behavior:

user=> (require '[try1.core :as c])
WARNING: merge already refers to: #'clojure.core/merge in namespace: try1.core, being replaced by: #'try1.core/merge
WARNING: * already refers to: #'clojure.core/* in namespace: try1.core, being replaced by: #'try1.core/*
nil
user=> (require '[try1.core :as c] )
nil
user=> (require '[try1.core :as c] :reload)
WARNING: merge already refers to: #'clojure.core/merge in namespace: try1.core, being replaced by: #'try1.core/merge
WARNING: * already refers to: #'clojure.core/* in namespace: try1.core, being replaced by: #'try1.core/*
nil

Ths all looks like behavior as I would expect, and I did not see the exception that Mike reports.

It seems that either Ctrl+Alt+L in Counterclockwise does something different than (require ... :reload), or there is something different about Mike's namespace in addition to redefining names in clojure.core that is causing the problem.

Comment by Alex Miller [ 11/Apr/14 11:17 AM ]

Marking this as NR for now - would be happy to see it reopened with an easily reproducible test case.

Comment by Mike Anderson [ 12/Apr/14 12:41 AM ]

To reproduce:

(ns op)
(defn * [a b] (clojure.core/* a b)) ;; gives warning
(ns use-op (:require [op :refer :all])) ;; gives warning
(ns use-op (:require [op :refer :all])) ;; gives error!

I believe Counterclockwise is simply loading the namespace again with CTRL-Alt+L, which is causing the ns form to be re-executed.

The docstring implies that ns can be used multiple times ("Sets ns to the namespace named by name (unevaluated), creating it if needed") so I would certainly expect multiple invocations of ns to be a no-op

Comment by Alex Miller [ 03/Nov/14 10:24 AM ]

Duped in CLJ-1578.

Comment by Mike Anderson [ 09/Jun/15 3:54 AM ]

This is still affecting me, and causing breakage with the latest versions of core.matrix. I don't know if this is a regression or not, but it certainly happens in 1.7.0-RC1

Any chance we can get a fix for 1.7? It is really annoying to have code fail because of this and force of refactoring of user code (my use case is adding a new var to clojure.core.matrix namespace, compiler error in user code that previously defined a var with the same name).

Comment by Mike Anderson [ 09/Jun/15 3:59 AM ]

Closing because I think this is better handled in the related issue

Comment by Mike Anderson [ 09/Jun/15 5:05 AM ]

Reopening because CLJ-1578 apparently does not resolve this specific issue, it only covers vars in clojure.core.

I'd still like to see this fixed for all namespaces, not just clojure.core.

Comment by Mike Anderson [ 09/Jun/15 5:08 AM ]

Reproduction:

=> (ns foo)
nil
=> (def a 1)
#'foo/a
=> (ns bar (:require [foo :refer :all]))
nil
=> (def a 2)
CompilerException java.lang.IllegalStateException: a already refers to: #'foo/a in namespace: bar, compiling:(NO_SOURCE_PATH:1:1)
=> clojure-version
{:major 1, :minor 7, :incremental 0, :qualifier "RC1"}

Stack trace:

CompilerException java.lang.RuntimeException: Unable to resolve symbol: pst in this context, compiling:(NO_SOURCE_PATH:1:1)
clojure.lang.Compiler.analyze (Compiler.java:6543)
clojure.lang.Compiler.analyze (Compiler.java:6485)
clojure.lang.Compiler$InvokeExpr.parse (Compiler.java:3737)
clojure.lang.Compiler.analyzeSeq (Compiler.java:6735)
clojure.lang.Compiler.analyze (Compiler.java:6524)
clojure.lang.Compiler.analyze (Compiler.java:6485)
clojure.lang.Compiler$BodyExpr$Parser.parse (Compiler.java:5861)
clojure.lang.Compiler$FnMethod.parse (Compiler.java:5296)
clojure.lang.Compiler$FnExpr.parse (Compiler.java:3925)
clojure.lang.Compiler.analyzeSeq (Compiler.java:6731)
clojure.lang.Compiler.analyze (Compiler.java:6524)
clojure.lang.Compiler.eval (Compiler.java:6789)
Caused by:
RuntimeException Unable to resolve symbol: pst in this context
clojure.lang.Util.runtimeException (Util.java:221)
clojure.lang.Compiler.resolveIn (Compiler.java:7029)
clojure.lang.Compiler.resolve (Compiler.java:6973)
clojure.lang.Compiler.analyzeSymbol (Compiler.java:6934)
clojure.lang.Compiler.analyze (Compiler.java:6506)
clojure.lang.Compiler.analyze (Compiler.java:6485)

Comment by Nicola Mometto [ 09/Jun/15 5:16 AM ]

As I already commented in CLJ-1578, I don't think this is a bug and I think this ticket should be declined.

Overriding non clojure.core vars has always (since 1.2 at least) caused an exception to be thrown.

Comment by Nicola Mometto [ 09/Jun/15 5:23 AM ]

Mike, maybe it would make sense to bring this issue up in the clojure-dev ml to get some opinions?

Comment by Mike Anderson [ 09/Jun/15 5:42 AM ]

Re-classify it as a feature request, if you prefer.

I still regard it as a defect because I expect :refer :all to work sanely.

Either way, this issue keeps breaking user code in my area (data science / exploratory statistics / data management). The ability to use / refer all is very useful for setting up a convenient namespace for exploratory work, so I don't accept that forcing users to explicitly require every single var used (as Nicola suggests in CLJ-1578) is a practical workaround.

I've also had it cause problems when working at the REPL and reloading namespaces.

If the Clojure core team really wants to keep this annoying behaviour, can we at least have some way to turn it off at the library level? Perhaps some namespace metadata that I can add to the clojure.core.matrix namespace to stop this from triggering?

Comment by Nicola Mometto [ 09/Jun/15 6:23 AM ]

Mike, this is just my personal opinion, I'm not a part of the core team and I don't speak for them, this is why I suggested you wrote on the clojure-dev ml.

Also to clarify, this issue you're reporting cannot manifest itself while reloading namespaces as the exception is thrown as soon as the redefinition happens.

Comment by Alex Miller [ 09/Jun/15 8:47 AM ]

You could use :exclude for this

(ns bar (:require [foo :refer :all :exclude (a)]))
Comment by Mike Anderson [ 09/Jun/15 9:30 AM ]

Hi Alex, that works as a fix when the problem occurs, but doesn't solve the problem of future user code breakage, unless the user accurately anticipates what symbols might get added to "bar" in the future. Which again seems like an unreasonable burden on the user.

What I'm arguing for, I guess, is a default presumption that if the user defines a var in their own namespace, they are happy to replace a similarly named var in any namespaces that they have previously use'd / refer-all'd.

If the user is genuinely concerned about overriding things by accident, I'd be happy with a warn-on-replace which does something analogous to warn-on-reflection. I proposed something similar in CLJ-1257 a while back, even wrote a patch that solves the whole problem in this way. Can we get that patch or something similar in 1.7?

Comment by Nicola Mometto [ 09/Jun/15 10:22 AM ]

CLJ-1746 is relevant and I like that proposal much better than CLJ-1257

Comment by Mike Anderson [ 09/Jun/15 10:43 AM ]

Hi Nicola, CLJ-1746 looks like a reasonable suggestion but still doesn't address the problem here, for the same reason that :exclude doesn't (see my comment to Alex)

The fundamental issue is that the current behaviour causes user code to break when libraries are upgraded to add new vars and requires changes to user code to fix / work around it. I consider that broken behaviour, or at least very bad design.

This is especially since we are encouraged to choose good names: I quote from the library coding standards(http://dev.clojure.org/display/community/Library+Coding+Standards)

"Use good names, and don't be afraid to collide with names in other namespaces. That's what the flexible namespace support is there for."

It isn't very flexible when user code keep breaking.....

Comment by Nicola Mometto [ 09/Jun/15 10:53 AM ]

From the same page, just a two lines below:
"Be explicit and minimalist about dependencies on other packages. (Prefer :require :refer in 1.4+ or :use :only in 1.0-1.3)."

If users carelessly import a whole package, I'd say that's their fault. Since clojure >1.3 the popular consensus has been that :use/:refer :all are not a good idea and people have been moving away from that, preferring :require :refer or :require :as instead.

The few that still use :use/:refer :all do it mostly for backward compatibility or in tests (I myself do it in tools.reader for the former reason).

I really don't think this is such a bad design choice as you seem to think, and I think that most people in the community would agree that the current behaviour and drive towards using :require :refer/require :as in lieu of :use/:refer :all is way more beneficial than harmful

Comment by Mike Anderson [ 09/Jun/15 11:42 AM ]

Hi Nicola, I have no issue with people using explicit :refer / :require, and I agree it is normal practice. It's good to be explicit and I generally do that myself (except in test / demo code).

However we aren't talking about that case : this issue is most relevant in those situations where users (for their own legitimate reasons) have decided to import a whole namespace. This is often convenient (e.g. REPL usage), sometimes it is valuable for specific purposes (e.g. testing).

I personally care about this mostly from the perspective of a library author: I want users to be able to use the library in whatever way is most convenient (which may include importing all vars), and I don't want user code to break randomly when I make a new release. Note that this also means I don't have control over user code so any solution that involves explicit requires, excludes, or some other manual workarounds is not a practical solution.

You can say "that's their fault" but this is currently supposed to be supported in Clojure so I think it ought to work sanely.

Comment by Alex Miller [ 09/Jun/15 12:00 PM ]

I think the library / evolution argument is a good one and I like that it reduces breakage due to evolution of 3rd party libraries. (I feel very strongly about this in clojure.core itself which is auto-referred, less strongly otherwise.)

On the flip side, if we remove this error, we should be explicit about what we are giving up to fully consider it. Presumably we are at least giving up a helpful error that occurs in the case of an accidental override. Are there other impacts?

I do not expect that we're going to change anything in 1.7.

Comment by Stuart Halloway [ 31/Jul/15 3:10 PM ]

Recategorizing as a feature request, as the current behavior was the original intent.

Could a namespace-reflecting macro provide for this use case without requiring a change to core?





[CLJ-1400] Error "Can't refer to qualified var that doesn't exist" should name the bad symbol Created: 09/Apr/14  Updated: 07/Oct/14

Status: Open
Project: Clojure
Component/s: None
Affects Version/s: Release 1.5
Fix Version/s: Release 1.8

Type: Enhancement Priority: Minor
Reporter: Howard Lewis Ship Assignee: Unassigned
Resolution: Unresolved Votes: 0
Labels: Compiler, errormsgs
Environment:

OS X


Attachments: File clj-1400-2.diff     File clj-1400-3.diff     File clj-1400-4.diff    
Patch: Code and Test
Approval: Incomplete

 Description   

Def of var with a ns that doesn't exist will yield this error:

user> (def foo/bar 1)
CompilerException java.lang.RuntimeException: Can't refer to qualified var that doesn't exist, compiling:(NO_SOURCE_PATH:1:1)

Cause: Compiler.lookupVar() returns null if the ns in a qualified var does not exist yet.

Proposed: The error message would be improved by naming the symbol and throwing a CompilerException with file/line/col info. It's not obvious, but this may be the only case where this error occurs. If so, the error message could be more specific that the ns is the part that doesn't exist.

Patch: clj-1400-4.diff

Screened by: Alex Miller



 Comments   
Comment by Scott Bale [ 25/Jun/14 9:58 AM ]

This looks to me like relatively low hanging fruit unless I'm missing something; assigning to myself.

Comment by Scott Bale [ 26/Jun/14 11:23 PM ]

Patch clj-1400-1.diff to Compiler.java.

With this patch the example would now look like:

user> (def foo/bar 1)
CompilerException java.lang.RuntimeException: Qualified symbol foo/bar refers to nonexistent namespace: foo, compiling:(NO_SOURCE_PATH:1:1)

I'm not sure the if(namesStaticMember(sym)) [see below], and the 2nd branch, is even necessary. Just by inspection I suspect it is not.

[footnote]

public static boolean namesStaticMember(Symbol sym){
	return sym.ns != null && namespaceFor(sym) == null;
}
Comment by Scott Bale [ 26/Jun/14 11:24 PM ]

patch: code and test

Comment by Scott Bale [ 26/Jun/14 11:27 PM ]

I tested on an actual source file, and the exception message included the file/line/col info as desired:

user=> CompilerException java.lang.RuntimeException: Qualified symbol goo/bar refers to nonexistent namespace: goo, compiling:(/home/scott/dev/foo.clj:3:1)
Comment by Andy Fingerhut [ 29/Aug/14 4:46 PM ]

Patch clj-1400-1.diff dated Jun 26 2014 no longer applied cleanly to latest master after some commits were made to Clojure on Aug 29, 2014. It did apply cleanly before that day.

I have not checked how easy or difficult it might be to update this patch. See section "Updating Stale Patches" on this wiki page for some tips on updating patches: http://dev.clojure.org/display/community/Developing+Patches

Comment by Scott Bale [ 31/Aug/14 3:53 PM ]

Attached is an updated patch: "clj-1400-2.diff". I removed the stale patch.

Comment by Alex Miller [ 09/Sep/14 9:29 AM ]

Few comments to address:

  • Compiler diff was using spaces, not tabs, which makes it harder to diff. I attached a -3.diff that fixes this.
  • the call to namesStaticMember seems weird. The name of that method is confusing for this use. Beyond that, I think it's doing more than you need. That method is going to attempt resolve the qualified name in terms of the current ns, but I think you don't even want to do that. Rather you just want to know if the sym has a ns (sym.ns != null) - isn't that enough?
  • In what case will the other error "Var doesn't exist" occur? In other words, in what case will lookupVar not succeed in creating a new var here? If there is no such case, then remove this case. If there is such a case, then add a test.
Comment by Scott Bale [ 11/Sep/14 11:19 PM ]

Agree with all three of your bullets. Attached is an updated patch, clj-1400-4.diff.

  • I used tabs in Compiler.java
  • After close inspection of call to lookupVar(...), I believe null is returned only in the case of exactly this ticket (the symbol having a non-null namespace which has not been loaded yet). So I've taken out the conditional and the 2nd branch.
  • (Test is unchanged)
Comment by Scott Bale [ 11/Sep/14 11:22 PM ]

(properly named patch)

Comment by Alex Miller [ 11/Sep/14 11:37 PM ]

You could throw a CompilerException with the location of the problem instead (as the ticket description suggests).

Comment by Scott Bale [ 19/Sep/14 2:37 PM ]

Sorry, I should've mentioned because this wasn't obvious to me either (and in fact I forgot until just now): the RuntimeException is already caught and wrapped in a CompilerException.

I'm not sure which try-catch block within Compiler.java this is happening in, there are multiple. But you can see in the output that the exception is a CompilerException and the file|line|col info is there:

In the Repl...

user> (def foo/bar 1)
CompilerException java.lang.RuntimeException: Qualified symbol foo/bar refers to nonexistent namespace: foo, compiling:(NO_SOURCE_PATH:1:1)

...or in a source file

user=> CompilerException java.lang.RuntimeException: Qualified symbol goo/bar refers to nonexistent namespace: goo, compiling:(/home/scott/dev/foo.clj:3:1)

Also, at the point at which the RuntimeException of this patch is being thrown, the source line and col params to CompilerException are not available, or at least not afaict.

Comment by Alex Miller [ 07/Oct/14 12:34 PM ]

I'll follow up on this patch later - Rich thought it was making too many assumptions. I think we validated many of those but need to double-check those.





[CLJ-1399] missing field munging when recreating deftypes serialized into byte code Created: 02/Apr/14  Updated: 17/Jul/15  Resolved: 17/Jul/15

Status: Closed
Project: Clojure
Component/s: None
Affects Version/s: Release 1.7
Fix Version/s: Release 1.8

Type: Defect Priority: Critical
Reporter: Kevin Downey Assignee: Unassigned
Resolution: Completed Votes: 2
Labels: compiler, deftype, ft

Attachments: File clj-1399.diff     File clj-1399-with-test.diff    
Patch: Code and Test
Approval: Ok

 Description   

deftypes with fields whose names get munged fail when constructed in data reader functions.

user=> (deftype Foo [hello-world])
user.Foo
user=> (alter-var-root #'default-data-readers assoc 'foo (fn [x] (->Foo x)))
{inst #'clojure.instant/read-instant-date, uuid #'clojure.uuid/default-uuid-reader, foo #object[user$eval12$fn__13 0x23c89df9 "user$eval12$fn__13@23c89df9"]}
user=> #foo "1"
CompilerException java.lang.IllegalArgumentException: No matching field found: hello-world for class user.Foo, compiling:(NO_SOURCE_PATH:0:0)

Cause: To embed deftypes in the bytecode the compiler emits the value of each field, then emits a call to the deftypes underlying class's constructor. To get a list of fields the compiler calls .getBasis. The getBasis fields are the "clojure" level field names of the deftype, which the actual "jvm" level field names have been munged (replacing - with _, etc), so the compiler tries to generate code to set values on non-existent fields.

Approach: Munge the field name before emitting it in bytecode.
Patch: clj-1399-with-test.diff
Screened by: Alex Miller



 Comments   
Comment by Kevin Downey [ 02/Apr/14 4:26 PM ]

reproducing case

$ rlwrap java -server -Xmx1G -Xms1G -jar /Users/hiredman/src/clojure/target/clojure-1.6.0-master-SNAPSHOT.jar
Clojure 1.6.0-master-SNAPSHOT
user=> (deftype Foo [hello-world])
user.Foo
user=> (alter-var-root #'default-data-readers assoc 'foo (fn [x] (Foo. x)))
{foo #<user$eval6$fn__7 user$eval6$fn__7@2f953efd>, inst #'clojure.instant/read-instant-date, uuid #'clojure.uuid/default-uuid-reader}
user=> #foo "1"
CompilerException java.lang.IllegalArgumentException: No matching field found: hello-world for class user.Foo, compiling:(NO_SOURCE_PATH:0:0)
user=>
Comment by Kevin Downey [ 02/Apr/14 4:39 PM ]

this patch fixes the issue on the latest master for me

Comment by Chas Emerick [ 02/Apr/14 4:57 PM ]

FWIW, this was precipitated by real experience (I think I created the refheap paste). The workaround is easy (don't use dashes in field names of deftypes you want to return from data reader functions), but I wouldn't expect anyone to guess that that wasn't already oversensitized to munging edge cases.

Comment by Alex Miller [ 29/Apr/15 11:36 AM ]

Could the patch have a test?

Comment by Kevin Downey [ 29/Apr/15 1:20 PM ]

clj-1399-with-test.diff adds a test





[CLJ-1392] AOT vs "var already exists warning" results in NPE Created: 26/Mar/14  Updated: 26/Mar/14  Resolved: 26/Mar/14

Status: Closed
Project: Clojure
Component/s: None
Affects Version/s: Release 1.6
Fix Version/s: None

Type: Defect Priority: Major
Reporter: Aaron Cohen Assignee: Unassigned
Resolution: Duplicate Votes: 0
Labels: aot, compiler


 Description   

I just saw this thread on the cascalog list: https://groups.google.com/forum/#!topic/cascalog-user/Pe5QIpmU0vA

Apparently the "WARNING: some? already refers to: #'clojure.core/some? in namespace: jackknife.seq, being replaced by: #'jackknife.seq/some?" results in a NullPointerException when trying to AOT a namespace that results in the message being output.

Reproducer:

1) lein new aotFail
2) Edit project.clj
;add as appropriate
:aot :all
:dependencies [[org.clojure/clojure "1.6.0"]
[cascalog "2.0.0"]]
2) Add "(:use cascalog.api)" to the ns block of src/aotFail/core.clj
3) lein compile

Output:

Compiling aotFail.core
WARNING: some? already refers to: #'clojure.core/some? in namespace: jackknife.seq, being replaced by: #'jackknife.seq/some?
Exception in thread "main" java.lang.NullPointerException, compiling:(api.clj:1:1)
at clojure.lang.Compiler$InvokeExpr.eval(Compiler.java:3558)
at clojure.lang.Compiler.compile1(Compiler.java:7226)
at clojure.lang.Compiler.compile1(Compiler.java:7216)
at clojure.lang.Compiler.compile(Compiler.java:7292)
at clojure.lang.RT.compile(RT.java:398)
at clojure.lang.RT.load(RT.java:438)
at clojure.lang.RT.load(RT.java:411)
at clojure.core$load$fn__5066.invoke(core.clj:5641)
at clojure.core$load.doInvoke(core.clj:5640)
at clojure.lang.RestFn.invoke(RestFn.java:408)
at clojure.core$load_one.invoke(core.clj:5446)
at clojure.core$load_lib$fn__5015.invoke(core.clj:5486)
at clojure.core$load_lib.doInvoke(core.clj:5485)
at clojure.lang.RestFn.applyTo(RestFn.java:142)
at clojure.core$apply.invoke(core.clj:626)
at clojure.core$load_libs.doInvoke(core.clj:5528)
at clojure.lang.RestFn.applyTo(RestFn.java:137)
at clojure.core$apply.invoke(core.clj:628)
at clojure.core$use.doInvoke(core.clj:5618)
at clojure.lang.RestFn.invoke(RestFn.java:408)
at cascalog2.core$loading_4958auto_.invoke(core.clj:1)
at clojure.lang.AFn.applyToHelper(AFn.java:152)
at clojure.lang.AFn.applyTo(AFn.java:144)
at clojure.lang.Compiler$InvokeExpr.eval(Compiler.java:3553)
at clojure.lang.Compiler.compile1(Compiler.java:7226)
at clojure.lang.Compiler.compile1(Compiler.java:7216)
at clojure.lang.Compiler.compile(Compiler.java:7292)
at clojure.lang.RT.compile(RT.java:398)
at clojure.lang.RT.load(RT.java:438)
at clojure.lang.RT.load(RT.java:411)
at clojure.core$load$fn__5066.invoke(core.clj:5641)
at clojure.core$load.doInvoke(core.clj:5640)
at clojure.lang.RestFn.invoke(RestFn.java:408)
at clojure.core$load_one.invoke(core.clj:5446)
at clojure.core$compile$fn__5071.invoke(core.clj:5652)
at clojure.core$compile.invoke(core.clj:5651)
at user$eval19.invoke(form-init2092370125048380878.clj:1)
at clojure.lang.Compiler.eval(Compiler.java:6703)
at clojure.lang.Compiler.eval(Compiler.java:6693)
at clojure.lang.Compiler.load(Compiler.java:7130)
at clojure.lang.Compiler.loadFile(Compiler.java:7086)
at clojure.main$load_script.invoke(main.clj:274)
at clojure.main$init_opt.invoke(main.clj:279)
at clojure.main$initialize.invoke(main.clj:307)
at clojure.main$null_opt.invoke(main.clj:342)
at clojure.main$main.doInvoke(main.clj:420)
at clojure.lang.RestFn.invoke(RestFn.java:421)
at clojure.lang.Var.invoke(Var.java:383)
at clojure.lang.AFn.applyToHelper(AFn.java:156)
at clojure.lang.Var.applyTo(Var.java:700)
at clojure.main.main(main.java:37)
Caused by: java.lang.NullPointerException
at clojure.lang.Compiler$ObjExpr.emitVar(Compiler.java:4944)
at clojure.lang.Compiler$DefExpr.emit(Compiler.java:437)
at clojure.lang.Compiler.compile1(Compiler.java:7225)
at clojure.lang.Compiler.compile(Compiler.java:7292)
at clojure.lang.RT.compile(RT.java:398)
at clojure.lang.RT.load(RT.java:438)
at clojure.lang.RT.load(RT.java:411)
at clojure.core$load$fn__5066.invoke(core.clj:5641)
at clojure.core$load.doInvoke(core.clj:5640)
at clojure.lang.RestFn.invoke(RestFn.java:408)
at clojure.core$load_one.invoke(core.clj:5446)
at clojure.core$load_lib$fn__5015.invoke(core.clj:5486)
at clojure.core$load_lib.doInvoke(core.clj:5485)
at clojure.lang.RestFn.applyTo(RestFn.java:142)
at clojure.core$apply.invoke(core.clj:626)
at clojure.core$load_libs.doInvoke(core.clj:5524)
at clojure.lang.RestFn.applyTo(RestFn.java:137)
at clojure.core$apply.invoke(core.clj:628)
at clojure.core$use.doInvoke(core.clj:5618)
at clojure.lang.RestFn.invoke(RestFn.java:408)
at cascalog.api$loading_4958auto_.invoke(api.clj:1)
at clojure.lang.AFn.applyToHelper(AFn.java:152)
at clojure.lang.AFn.applyTo(AFn.java:144)
at clojure.lang.Compiler$InvokeExpr.eval(Compiler.java:3553)



 Comments   
Comment by Nicola Mometto [ 26/Mar/14 12:42 PM ]

Duplicate of http://dev.clojure.org/jira/browse/CLJ-1241





[CLJ-1351] Clojure emits an unused "swapThunk" method for functions with keyword callsites Created: 14/Feb/14  Updated: 29/Apr/15

Status: Open
Project: Clojure
Component/s: None
Affects Version/s: None
Fix Version/s: None

Type: Enhancement Priority: Major
Reporter: Nicola Mometto Assignee: Unassigned
Resolution: Unresolved Votes: 0
Labels: compiler, ft, performance

Attachments: Text File 0001-remove-unused-swapThunk-method-generation.patch    
Patch: Code
Approval: Triaged

 Description   

This method is no longer used, I did a quick git blame and it look like it was used for an earlier implementation of keyword callsites and forgot to be removed in this commit https://github.com/clojure/clojure/commit/c7af275d4ee33cdc1794c8df8fa1e6d39039ac84

Removing this should reduce a bit the size of compile functions.

Screened by: Alex Miller






[CLJ-1342] Byte comparison boxes both bytes and converts to longs to compare (which is slow) Created: 06/Feb/14  Updated: 06/Feb/14

Status: Open
Project: Clojure
Component/s: None
Affects Version/s: None
Fix Version/s: None

Type: Enhancement Priority: Minor
Reporter: Alex Miller Assignee: Unassigned
Resolution: Unresolved Votes: 1
Labels: compiler

Attachments: File bytebox.clj    

 Description   

This came up in a much more complicated example but consider a case like this:

(defn simple []
  (let [b (byte-array [(byte 0)])
        m (byte 0)]
    (= m (aget b 0))))

In the compiled bytecode, both m and (aget b 0) are known to be bytes, but both are boxed using Byte.valueOf(), then cast using RT.uncheckedLongCast() and finally compared as longs:

26: iload_2
  27: invokestatic  #69  // Method java/lang/Byte.valueOf:(B)Ljava/lang/Byte;
  30: checkcast     #81  // class java/lang/Number
  33: invokestatic  #85  // Method clojure/lang/RT.uncheckedLongCast:(Ljava/lang/Object;)J

In a tight loop manipulating and matching against byte arrays, this boxing is significant for performance.

Attached is a test that demonstrates the performance difference between the byte[] and long[] performance to get an idea of the difference.



 Comments   
Comment by Nicola Mometto [ 06/Feb/14 9:10 PM ]

The description states that Util.equiv() has a byte/byte comparison variant but it doesn't look like it actually exists.

Comment by Nicola Mometto [ 06/Feb/14 9:17 PM ]

By the way, tools.emitter.jvm uses i2l to cast the byte to a long instead of boxing && unboxing to a long

Comment by Alex Miller [ 06/Feb/14 9:39 PM ]

Thanks Nicola - I must have confused it with the boolean/boolean version.





[CLJ-1330] Class name clash between top-level functions and defn'ed ones Created: 22/Jan/14  Updated: 14/Nov/14  Resolved: 14/Nov/14

Status: Closed
Project: Clojure
Component/s: None
Affects Version/s: None
Fix Version/s: Release 1.7

Type: Defect Priority: Critical
Reporter: Nicola Mometto Assignee: Unassigned
Resolution: Completed Votes: 8
Labels: aot, compiler

Attachments: Text File 0001-CLJ-1093-v3-no-locals-improv.patch     Text File 0001-CLJ-1093-v3.patch     Text File 0001-CLJ-1330-remove-local-binding-name-enhancement.patch     File demo1.clj    
Patch: Code
Approval: Ok

 Description   

Named anonymous fn's are not guaranteed to have unique class names when AOT-compiled.

For example:

(defn g [])
(def xx (fn g []))

When AOT-compiled both functions will emit user$g.class, the latter overwriting the former.

Impact: this affects apps like Cursive, which has been using a patched version of Clojure to get around this issue for quite a while.

Demonstration script: demo1.clj

Patch: 0001-CLJ-1093-v3.patch (already applied, see below)

Approach: Generate unique class names for named fn's the same way as for unnamed anonymous fn's.
The patch contains an additional enhancement to include the name of the local binding in the class name.

Comparison between pre and post patch naming scheme (N denotes unique number):

code before after note
(defn a []) user$a user$a same
(fn []) user$evalN$fn__N user$evalN$fn__N same
(fn a []) user$evalN$a__N user$evaN$a__N same
(let [a (fn [])] a) user$evalN$a__N user$evalN$a__N same
(let [a (fn x [])] a) user$eval1N$x__N user$evalN$a_x_N IMPROVED - contains local binding name
(def a (fn [])) user$a user$a same
(def a (fn x [])) user$x user$a_x_N FIXED conflict with (defn x [])
(def ^{:foo (fn [])} a) user$fn__N user$fn__N same
(def ^{:foo (fn a [])} a) user$a user$a__N FIXED conflict with (defn a [])
(def a (fn [] (fn []))) user$a$fn__N user$a$fn__N same
(def a (fn [] (fn x []))) user$a$x__N user$a$x__N same

See also: This patch also fixes the issue reported in CLJ-1227.

Screened by: Alex Miller - I am not sure whether the local binding name enhancement is worth doing. It improves debugging of which anonymous class you're talking about but has the downsides of increasing class name (and file name) length.

REOPENED Patch: 0001-CLJ-1093-v3-no-locals-improv.patch
REOPENED UPDATE: The local improvement version of this patch increases class names (and thus .class names) and we've had several reports of exceeding file system limits (~143 chars) - see comments for details. Thus, we should rollback the prior patch (0001-CLJ-1093-v3.patch) and apply the version without this enhancement. The replacement patch (0001-CLJ-1093-v3-no-locals-improv.patch) does not have the same effect on class name length.



 Comments   
Comment by Ambrose Bonnaire-Sergeant [ 22/Jan/14 11:12 AM ]

This seems like the reason why jvm.tools.analyzer cannot analyze clojure.core. On analyzing a definline, there is an "attempted duplicate class definition" error.

This doesn't really matter, but I thought it may or may not be useful information to someone.

Comment by Nicola Mometto [ 22/Jan/14 11:35 AM ]

Attached a fix.

This also fixes AOT compiling of code like:

(def x (fn foo []))
(fn foo [])
Comment by Nicola Mometto [ 22/Jan/14 11:39 AM ]

Cleaned up patch

Comment by Alex Miller [ 22/Jan/14 12:43 PM ]

It looks like the patch changes indentation of some of the code - can you fix that?

Comment by Nicola Mometto [ 22/Jan/14 3:57 PM ]

Updated patch without whitespace changes

Comment by Alex Miller [ 22/Jan/14 4:15 PM ]

Thanks, that's helpful.

Comment by Alex Miller [ 24/Jan/14 10:03 AM ]

There is consensus that this is a problem, however this is an area of the code with broad impacts as it deals with how classes are named. To that end, there is some work that needs to be done in understanding the impacts before we can consider it.

Some questions we would like to answer:

1) According to Rich, naming of (fn x []) function classes used to work in the manner of this patch - with generated names. Some code archaeology needs to be done on why that was changed and whether the change to the current behavior was addressing problems that we are likely to run into.

2) Are there issues with recursive functions? Are there impacts either in AOT or non-AOT use cases? Need some tests.

3) Are there issues with dynamic redefinition of functions? With the static naming scheme, redefinition causes a new class of the same name which can be picked up by reload of classes compiled to the old definition. With the dynamic naming scheme, redefinition will create a differently named class so old classes can never pick up a redefinition. Is this a problem? What are the impacts with and without AOT? Need some tests.

Comment by Nicola Mometto [ 24/Jan/14 11:39 AM ]

Looks like the current behaviour has been such since https://github.com/clojure/clojure/commit/4651e60808bb459355a3a5d0d649c4697c672e28

My guess is that Rich simply forgot to consider the (def y (fn x [] ..)) case.

Regarding 2 and 3, the dynamic naming scheme is no different than what happens for anonymous functions so I don't see how this could cause any issue.

Recursion on the fn arg is simply a call to .invoke on "this", it's classname unaware.

I can add some tests to test that

(def y (fn x [] 1))
and
(fn x [] 2)
compile to different classnames but other than that I don't see what should be tested.

Comment by Stuart Halloway [ 27/Jun/14 2:17 PM ]

incomplete pending the answers to Alex Miller's questions in the comments

Comment by Nicola Mometto [ 27/Jun/14 3:20 PM ]

I believe I already answered his questions, I'll try to be a bit more explicit:
I tracked the relevant commit from Rich which added the dynamic naming behaviour https://github.com/clojure/clojure/commit/4651e60808bb459355a3a5d0d649c4697c672e28#diff-f17f860d14163523f1e1308ece478ddbL3081 which clearly shows that this bug was present since then so.

Regarding redefinitions or recursive functions, both of those operations never take in account the generated fn name so they are unaffected.

Comment by Alex Miller [ 12/Sep/14 4:32 PM ]

Summarizing some cases here from before/after the patch:

1) top-level fn (always has name)
	1.6 - namespace$name
	patch - namespace$name
2) non-top-level fn with name
	1.6 - namespace$name (collides with prior case)
	patch - namespace$topname__x__name  	<-- CHANGED
3) anonymous fn (no name)
	1.6 - namespace$name$fn__x
	patch - namespace$name$fn__x
4) top-level anonymous fn (no name, not at all useful :)
	1.6 - namespace$fn__x
	patch - namespace$fn__x

The key problem is that the first 2 cases produce the identical class name on 1.6. The patch alters the non-top-level named fn so there is no conflict.

Prior to the referenced old commit, I believe cases 1 and 2 would both produce namespace$name__x (where x is unique) so they would not collide. The change was made to prevent the top-level name from changing ("don't append numbers on top-level fn class names"). While the similar change was made on non-top-level fn names, I do not think it needed to be.

I've thought through (and tried) a bunch of the implications of this with the help of Nicola's comments above and I do not see an issue with any of the things I've considered. From a binary compatibility point of view with existing AOT code, old code compiled together should be self-consistent and continue to work. New compiled code will also be consistent. I can't think of a way that new code would expect to know the old name of a non-top-level function such that there could be an issue.

One question - why change the code such that the new class name is namespace$name$topname__x__name instead of namespace$name$topname_name__x (or something else?). And relatedly, while the diff is small, could we refactor a couple more lines to make the intent and cases clearer?

I am 90% ok with this patch but want a little thought into that question before I mark screened.

Comment by Nicola Mometto [ 12/Sep/14 4:47 PM ]

Alex, the attached patch munges into ns$topname__name__x, not into ns$topname__x__name.

Comment by Nicola Mometto [ 12/Sep/14 5:22 PM ]

The attached patch 0001-Fix-CLJ-1330refactored.patch contains the same fix from 0001-FixCLJ-1330-make-top-level-named-functions-classnam.patch but also refactors the code that deals with fn name munging

Comment by Alex Miller [ 12/Sep/14 6:22 PM ]

Hmmm.. I will double-check. That's not why I recall seeing when I did AOT.

Comment by Nicola Mometto [ 12/Sep/14 7:26 PM ]

New patch 0001-CLJ-1093-v2.patch improves the fn naming scheme a lot.
I've threw together a number of test cases that show the improvement + bug fixes:

user=> (fn [])
;; pre:
#<user$eval1$fn__2 user$eval1$fn__2@4e13aa4e>
;; post: (no change)
#<user$eval1$fn__3 user$eval1$fn__3@3c92218c>
user=> (fn a [])
;; pre:
#<user$eval5$a__6 user$eval5$a__6@6946a317>
;; post: (no change)
#<user$eval6$a__8 user$eval6$a__8@6f85c59c>
user=> (let [a (fn [])] a)
;; pre:
#<user$eval9$a__10 user$eval9$a__10@15fdf894>
;; post: (no change)
#<user$eval11$a__13 user$eval11$a__13@4d051922>
user=> (let [a (fn x [])] a)
;; pre: (only contains the name of the fn)
#<user$eval17$x__18 user$eval17$x__18@7f0cd67f>
;; post: (contains the name of the local aswell as the name of the fn
#<user$eval21$a__x__23 user$eval21$a__x__23@528ef256>
user=> (def a (fn [])) a
#'user/a
;; pre:
#<user$a user$a@33e1ccbc>
;; post: (no change)
#<user$a user$a@6bef63f9>
user=> (def a (fn x [])) a
#'user/a
;; pre: (BUG!)
#<user$x user$x@59a04a1b> 
;; post: (bug fixed)
#<user$a__x__28 user$a__x__28@5f0bebef>
user=> (def ^{:foo (fn [])} a) (-> (meta #'a) :foo)
#'user/a
;; pre:
#<user$fn__23 user$fn__23@d9c21c6>
;; post: (no change)
#<user$fn__30 user$fn__30@4cf0f2eb>
user=> (def ^{:foo (fn a [])} a) (-> (meta #'a) :foo)
#'user/a
;; pre: (BUG!)
#<user$a user$a@420dd874>
;; post: (bug fixed)
#<user$a__35 user$a__35@37ff95a9>
user=> (def a (fn [] (fn []))) (a)
#'user/a
;; pre:
#<user$a$fn__30 user$a$fn__30@6f57be76>
;; post: (no change)
#<user$a$fn__41 user$a$fn__41@fd34eac>
user=> (def a (fn [] (fn x []))) (a)
#'user/a
;; pre:
#<user$a$x__35 user$a$x__35@79930089>
;; post: (no change)
#<user$a$x__48 user$a$x__48@6fc334de>
user=> (let [x (fn [])] (def a (fn [] x))) a (a)
#'user/a
;; pre:
#<user$eval40$a__43 user$eval40$a__43@6db1694e>
#<user$eval40$x__41 user$eval40$x__41@20bd16bb>
;; post (no change)
#<user$eval54$a__58 user$eval54$a__58@7c721de>
#<user$eval54$x__56 user$eval54$x__56@43f7b41b>
user=> (let [x (fn a [])] (def a (fn [] x))) (a)
#'user/a
;; pre: (the local binding name doesn't appear in the class name)
#<user$eval48$a__49 user$eval48$a__49@75d6d1d4>
;; post: (the local binding name is included in the class name)
#<user$eval64$x__a__66 user$eval64$x__a__66@460d4>

As you can see, this last patch not only fixes the two bugs, but also improves fn naming in let contexts by preserving the name of the local binding in the class name, this I believe will be a great improvement in the understandability of stacktraces.

Comment by Alex Miller [ 25/Sep/14 7:00 AM ]

The patch should be changed to not create suffix if it's not going to be used. Please update the patch to inline that into each branch name = nm.name + "__" + RT.nextID();.

I am unsure whether the "enhancement" part of this patch goes too far. I think it does provide some improvements in debugging but those seem small to me. I am somewhat concerned about greatly increasing the name of the class for nested locals thus making it harder to read stack traces. There is a large limit to class name size of 16 bits (what you can put in the constant table) but class names also map to file names and there have historically been issues on some older Windows architectures with file size limits - we are increasing the risk of running into problems with this. Small risks. I am ok with passing this on to Rich though and he can decide whether to kick that part back or not.

Comment by Nicola Mometto [ 25/Sep/14 7:08 AM ]

0001-CLJ-1093-v3.patch is identical to 0001-CLJ-1093-v2.patch except it doesn't call RT.nextID() when not necessary, as per Alex's request

Alex, if this is ok please change the "Patch:" field in the description, I won't do that myself since this ticket is now screened

Comment by Nicola Mometto [ 06/Oct/14 11:54 AM ]

Addressing the screening comment by Alex Miller, I've attached an alternative patch "0001-CLJ-1093v3-no-locals-improv.patch" which is identical to "0001CLJ-1093-v3.patch" except it doesn't include the local binding name enhancement, so that it can be picked in case Rich decides that that improvement is out of scope for this ticket.

Comment by Alex Miller [ 28/Oct/14 12:05 PM ]

I've reopened this issue based on early reports of breakage due to long file names.

Two reports:
https://groups.google.com/d/msg/clojure-dev/hnkJb9_il_M/4e5smM6mVlIJ
https://groups.google.com/d/msg/clojure/hnkJb9_il_M/QOaTdCo5wmkJ

Comment by Alex Miller [ 28/Oct/14 12:21 PM ]

Here's an example of a class name that is too long on Ubuntu 12.04 LTS 64bit / Java8 - reported max file size is 143 chars:

https://github.com/ska2342/nested-fn-breaks-clojure-17/blob/master/src/nested_fn_breaks_clojure_17/core.clj

With 1.6.0: (95 chars)
core$this_function_breaks_with_clojure_1_7$my_anonymous_function_18$iter1923$fn24$fn_25

With 1.7.0-alpha3: (144 chars)
core$this_function_breaks_with_clojure_1_7$my_anonymous_function_my_anonymous_function19$iter4951auto__iter2024$fn25$fn_26.class

With the alternate patch here, the name would be: (95 chars)
core$this_function_breaks_with_clojure_1_7$my_anonymous_function_19$iter2024$fn25$fn_26

Comment by Nicola Mometto [ 28/Oct/14 12:26 PM ]

patch "0001-CLJ-1330-remove-local-binding-name-enhancement.patch" has the same effect of reverting f149260c14a75367dc9eba91cbe9b78110113566 and applying "0001-CLJ-1093-v3-no-locals-improv.patch" in case this is preferable

Comment by Stefan Kamphausen [ 29/Oct/14 7:44 AM ]

The tiny and unusual max file size of 143 is standard in the Ubuntu 12.04 crypto container for the home directory. You can get it for any directory with 'getconf NAME_MAX /path/to/dir'.

My initial problem (other than the file to reproduce on github) was triggered by the fns in a for-expression. Don't know if that makes any difference for you.





[CLJ-1315] Don't initialize classes when importing them Created: 28/Dec/13  Updated: 22/May/15  Resolved: 07/Oct/14

Status: Closed
Project: Clojure
Component/s: None
Affects Version/s: Release 1.1, Release 1.2, Release 1.3, Release 1.4, Release 1.5
Fix Version/s: Release 1.7

Type: Enhancement Priority: Critical
Reporter: Aaron Cohen Assignee: Unassigned
Resolution: Completed Votes: 9
Labels: aot, compiler, interop

Attachments: Text File 0001-Don-t-initialize-classes-during-import.patch    
Patch: Code
Approval: Ok

 Description   

Problem: When classes are imported in Clojure, the class is loaded using Class.forName(), which causes its static initialisers to be executed. This differs from Java where compilation does not cause classes to be loaded.

Motivation: In many cases when those classes are normally loaded by Java code during execution of a framework of some kind (IntelliJ in my case, and RoboVM is another culprit mentioned in that thread) the initialiser expects some infrastructure to be in place and will fail when it's not. This means that it's impossible to AOT compile namespaces importing these classes, which is a fairly serious limitation.

Approach: Modify ImportExpr to call RT.classForNameNonLoading() instead of Class.forName(), which will load the class but not initialise it. This change causes the Clojure test suite to fail, since clojure.test-clojure.genclass imports a gen-class'ed class which no longer loads its namespace on initialisation. I'm not sure if this is considered an incorrect use of such a class (IIRC with records it's required to import the class and require its namespace), but given that it's in the Clojure test case it's reasonable to assume that this fix would be a breaking change for more code out there. This test failure is also corrected in the attached patch.

Patch: 0001-Don-t-initialize-classes-during-import.patch

Screened by: Alex Miller - I have tested many open source Clojure projects with this change (particularly seeking out large, complicated, or known users of genclass/deftype/etc) and have found no projects adversely impacted. I know that Cursive has been running with this modification for a long time with no known issues. I am ok with unconditionally enabling this change (re the comment below). The impact is described in more detail in the suggested changelog diff in the comments below.

Alternative: This patch enables the change unconditionally, but depending on the extent of breakage it causes, it might need to be enabled with a configuration flag. I propose we make it unconditional in an early 1.7 beta and monitor the fall-out.

Background: This issue has been discussed in the following threads
https://groups.google.com/d/topic/clojure/tWSEsOk_pM4/discussion
https://groups.google.com/forum/#!topic/clojure-dev/qSSI9Z-Thc0



 Comments   
Comment by Alex Miller [ 29/Dec/13 12:23 PM ]

From original post:

This issue was originally reported by Zach Oakes and Colin Fleming and this patch was also tested by Colin.

I'm duplicating here my suggested release notes for this issue, which includes my current thoughts on potential breakage (it's also in the commit message of the patch):

    "import" no longer causes the imported class to be initialized. This
    change better matches Java's import behavior and allows the importing of
    classes that do significant work at initialization time which may fail.
    This semantics change is not expected to effect most code, but certain
    code may have depended on behavior that is no longer true.

    1) importing a Class defined via gen-class no longer causes its defining
    namespace to be loaded, loading is now deferred until first reference. If
    immediate loading of the namespace is needed, "require" it directly.
    2) Some code may have depended on import to initialize the class before it
    was used. It may now be necessary to manually call (Class/forName
    "org.example.Class") when initialization is needed. In most cases, this
    should not be necessary because the Class will be initialized
    automatically before first use.
Comment by Greg Chapman [ 13/May/14 6:25 PM ]

I'm not sure if this should also be fixed, but it would be nice if you could emit the code for a proxy of one of these non-initialized classes without forcing initialization. For example, the following raises an exception (I'm using Java 8):

Clojure 1.6.0
user=> (def cname "javafx.scene.control.ListCell")
#'user/cname
user=> (let [cls (Class/forName cname false (clojure.lang.RT/baseLoader))] (.importClass *ns* cls))
javafx.scene.control.ListCell
user=> (defn fails [] (proxy [ListCell] [] (updateItem [item empty] (proxy-super item empty))))
CompilerException java.lang.ExceptionInInitializerError, compiling:(NO_SOURCE_PATH:3:16)

The exception was ultimately caused by "IllegalStateException Toolkit not initialized", which javafx throws if you attempt to initialize a Control class outside of Application.launch.

Comment by Michael Blume [ 22/May/15 2:58 PM ]

Not sure if this should properly be considered a bug in Cloverage, but since this patch landed, I've been unable to get coverage in gen-class methods.

https://github.com/lshift/cloverage/issues/74





[CLJ-1309] Bindings after :as in list destructuring should throw error Created: 19/Dec/13  Updated: 05/Feb/14

Status: Open
Project: Clojure
Component/s: None
Affects Version/s: Release 1.4, Release 1.5
Fix Version/s: None

Type: Enhancement Priority: Minor
Reporter: ben wolfson Assignee: Unassigned
Resolution: Unresolved Votes: 0
Labels: Compiler, errormsgs


 Description   

If you try to define a vector binding with anything at all after an :as parameter, you do not get a compiler error, and the binding is silently swallowed:

user> ((fn [[:as y z]] y) [1 2])
[1 2]

If you try to actually use the binding, there will be a compiler error (the compiler will complain that there's no binding for the symbol), but the actual error has already happened, and should be reported earlier.






[CLJ-1301] case expression fails to match a BigDecimal Created: 23/Nov/13  Updated: 26/Jan/14  Resolved: 26/Jan/14

Status: Closed
Project: Clojure
Component/s: None
Affects Version/s: Release 1.5, Release 1.6
Fix Version/s: Release 1.6

Type: Defect Priority: Blocker
Reporter: Andy Fingerhut Assignee: Unassigned
Resolution: Completed Votes: 2
Labels: Compiler

Attachments: Text File case-alt.patch     File clj-1301-1.diff    
Patch: Code and Test
Approval: Incomplete

 Description   

In 1.5.1 (anywhere before the CLJ-1118 patch), this is the behavior on BigDecimal case matching:

user=> (defn t [v] (case v 1 "Long" 1.0M "BigDecimal" "none"))
#'user/t
user=> (map t [1 1.0M 1.00M])
("Long" "BigDecimal" "none")

In 1.6 the behavior (post CLJ-1118 patch) has changed:

user=> (defn t [v] (case v 1 "Long" 1.0M "BigDecimal" "none"))
#'user/t
user=> (map t [1 1.0M 1.00M])
("Long" "none" "none")

In 1.6 after CLJ-1118, I expect to see: ("Long" "BigDecimal" "BigDecimal") as they now have the same hash and hasheq.

Cause: The case constants are hashed in the clojure.core/case macro using clojure.core/hash which calls clojure.lang.util/hasheq(). In Compiler.emitExprForHashes(), a call to clojure.lang.Util/hash(). In Clojure 1.5 these hash values are the same (hash of 1.0M == hasheq of 1.0M == 311). In Clojure 1.6, they are different (hash of 1.0M = 311, hasheq of 1.0M = 31).

In any cases where Java's hashCode and Clojure's hasheq return different values, the case statement can fail to do the correct thing.

Approach: Change Compiler.java to use clojure.lang.Util hasheq() to match the case macro use of clojure.core/hash (which calls clojure.lang.Util.hasheq()).

Patch: clj-1301-1.diff

Screened by:



 Comments   
Comment by Andy Fingerhut [ 23/Nov/13 5:00 PM ]

Patch clj-1301-1.diff modifies Compiler.java so that case* statements use hasheq on the test expression value, rather than Java's hashCode. It also adds a test case that currently fails with latest Clojure master, but passes with the patch.

Comment by Andy Fingerhut [ 23/Nov/13 5:01 PM ]

This bug is also the root cause for the recent failures of tests for the test.generative library.

Comment by Alex Miller [ 10/Dec/13 3:22 PM ]

Putting in 1.6 release per Rich.

Comment by Alex Miller [ 13/Dec/13 3:36 PM ]

Andy, I talked to Rich and the conclusion was that we should make the opposite change here such that the case macro should route to the Java hashcode version clojure.lang.util.hash() and the Compiler should be left as is. Can you update the patch?

Comment by Alex Miller [ 13/Dec/13 3:38 PM ]

And in case you were wondering, the reason is that the Java hashcode is generally faster (case is all about speed) and there are easy opportunities for you to properly cast your expression and/or case constants (where as the situations with collections where boxing is difficult to fix generically, that is not true).

Comment by Andy Fingerhut [ 13/Dec/13 5:14 PM ]

Alex, unless I am missing something, changing case to use Java's hashCode() would also require changing its current equality comparison from Clojure = (aka equiv()) to something consistent with hashCode(), which I think must be Java's equals().

Such a change would mean that all of the things that are = but not equals() will not match each other in a case statement, e.g. a case value of (Integer. 5) will not match a (Long. 5) value to compare against in a case branch.

Is that really what is desired here? I almost hesitate to create such a patch, for fear it might be committed

Comment by Alex Miller [ 17/Dec/13 12:06 PM ]

Based on discussion comments, move back to Incomplete until we resolve.

Comment by Alex Miller [ 16/Jan/14 9:37 AM ]

Added better example demonstrating the problem (the specific problem exposed by CLJ-1118).

Comment by Alex Miller [ 16/Jan/14 11:50 AM ]

Simplified examples.

Comment by Alex Miller [ 16/Jan/14 12:29 PM ]

Re Andy's comments above, I walked down that path a bit and built such a patch, however we currently have tests in clojure.test-clojure.control:

(testing "test number equivalence"
    (is (= :1 (case 1N 1 :1 :else))))

which clearly seems to expect Clojure equiv() behavior over Java equals() behavior in case constant matching. So either that is a bad test or this is not a viable approach (it also suggests we could break existing code with this change).

Comment by Andy Fingerhut [ 16/Jan/14 12:55 PM ]

One could consider having the default behavior of case to use hasheq and clojure.core/= everywhere, but add a 'fast' option to use hashCode and Java equals.

Comment by Alex Miller [ 24/Jan/14 9:46 AM ]

Alternative patch in the direction of using hashcode/equals instead of hasheq/equiv. Note that this test causes some test failures. This is not yet a candidate patch - further work needs to be done in evaluating this path.





[CLJ-1297] try to catch using - instead of _ in filenames so the compiler can give a better error message for people who don't know that you need to use _ in file names Created: 19/Nov/13  Updated: 07/Oct/14  Resolved: 07/Oct/14

Status: Closed
Project: Clojure
Component/s: None
Affects Version/s: None
Fix Version/s: Release 1.7

Type: Enhancement Priority: Major
Reporter: Kevin Downey Assignee: Unassigned
Resolution: Completed Votes: 11
Labels: compiler, errormsgs

Attachments: File better-error-messages-for-require.diff     Text File clj-1297-v3.patch     Text File clj-1297-v5.patch    
Patch: Code
Approval: Ok

 Description   

Problem: Clojure requires the files that back a namespace that has dashes in it to have the dashes replaced with underscores on the filesystem (ie a.b_c.clj for namespace a.b-c). If you require a file that has been mistakenly saved as b-c.clj instead, you will get an error message:

Exception in thread "main" java.io.FileNotFoundException: Could not locate a/b_c__init.class or a/b_c.clj on classpath:
...

Proposed:
Fix the bad ending colon in this sentence and add a second sentence only when the file name has an _ in it: "Please check that namespaces with dashes use underscores in the Clojure file name."

Patch: clj-1297-v5.patch

Screened by: Alex Miller



 Comments   
Comment by Joshua Ballanco [ 20/Nov/13 12:15 AM ]

A perhaps even better solution would be to simply allow the use of dashes in *.clj[s] filenames. I can't imagine the extra disk access per-namespace would be a huge performance burden, and (since dashes aren't allowed currently) I don't think there would be any issues with backwards compatibility.

Comment by Gary Fredericks [ 20/Nov/13 8:40 AM ]

It's worth mentioning the combinatorial explosion for namespaces with multiple dashes – if I (require 'foo-bar.baz-bang), should clojure search for all four possible filenames? Does the jvm have a way to search for files by regex or similar to avoid nasty degenerate cases (like (require 'foo-------------))?

Comment by Joshua Ballanco [ 20/Nov/13 11:08 AM ]

According to the docs, the FileSystem class's "getPathMatcher" method accepts path globs, so you'd merely have to replace each instance of "-" or "_" with "{-,_}". Actual runtime characteristics would likely depend on the underlying filesystem's implementation.

Comment by Alex Miller [ 20/Nov/13 12:02 PM ]

I don't think the FileSystem stuff applies when looking up classes on the classpath. Note that Java class names cannot contain "-".

Comment by Phil Hagelberg [ 21/Nov/13 12:05 PM ]

According to the spec, Java class names can't contain dashes (though IIRC OpenJDK and Oracle's JDK accept them anyway) but the requirement that Clojure source files have names which align with their AOT'd class file eqivalents is something we've imposed upon ourselves. Introducing the disconnect between .clj files and .class files makes way more sense than disconnecting namespaces and .clj files, but arguably it's too late to fix that mistake.

In any case a check for dashed files (resulting only in a more informative compiler error, not a more permissive compiler) which only triggers when a .clj file cannot be found imposes zero overhead in the case where things are already working.

Comment by scott tudd [ 09/Dec/13 2:19 PM ]

As Clojure seems to be idiomatic to have sometimes-dashed-namespace-and-function-names as opposed to the ubiquitous camelCaseFunctionNames in java ... I agree to have the compiler automagically handle 'knowing' to look in dir_struct AND dir-struct for requisite files.

or at the least print out a nice message explaining the quirk when files "can't" be found ... WHEN there are dashes and underscores involved... anything to aid in helping things "just work" as one would think they're supposed to.

Comment by Obadz [ 12/Dec/13 5:28 AM ]

I would have saved a few hours as well.

Comment by Alexander Redington [ 14/Feb/14 2:29 PM ]

This patch changes clojure.core/load such that:

  • When loading the resource-root of lib throws a FileNotFoundException, the lib is analyzed...
  • ... if the lib was a name that would be munged, it examines the combinatorial explosion of munge candidates and .clj or .class files in the classpath ...
  • ... if any of these candidates exist, it informs the user of the file's existance, and that a change to that filename would lead to that resource being loaded.
  • ... if none of these candidates exist, it throws the original exception.

It also modifies clojure.lang.RT to expose the behavior around finding clj or class files from a resource root.

Comment by Andy Fingerhut [ 20/Mar/14 1:16 PM ]

I do not know whether it handles all of the cases proposed in this discussion, but I encourage folks to check out the filename/namespace consistency checking in the latest Eastwood release (version 0.1.1) to see if it catches the cases they would hope to catch. It does a static check based on the files in a Leiningen project, nothing at run time. https://github.com/jonase/eastwood

Of course changes to Clojure itself to give warnings about such things can still be very useful, since not everyone will be using a 3rd party tool to check for such things.

Comment by Alex Miller [ 27/Jun/14 2:24 PM ]

Re the screener's note at the top, my preference would be for the simpler approach.

Comment by Rich Hickey [ 29/Aug/14 9:48 AM ]

I see no reason to fish around in the file system at all. Why can't the message simply remind people that underscores are required and to check that they aren't using dashes?

Comment by Gary Fredericks [ 30/Sep/14 4:43 PM ]

The tradeoff is that fishing around in the file system means the error message is only shown if the user likely made the relevant mistake, whereas simply showing the error message would (if I understand correctly) mean this reminder gets shown for every require error, which I would estimate happens a whole lot more often than the mistake in question.

Comment by Andy Fingerhut [ 30/Sep/14 5:05 PM ]

Gary, there is an exception thrown in any case if the load fails. One approach that I am hacking up now is to add to the existing exception's message that maybe they need to replace - chars in file name with _, but only if the name attempted to be loaded has at least one '-' character in it. So no new output, except in an exception already being thrown, and then only if there is at least a possibility that this is the problem.

Comment by Andy Fingerhut [ 30/Sep/14 5:10 PM ]

clj-1297-v2.patch is similar to the previously proposed patch, but does not touch the file system in any way that wasn't already being done before this patch.

It adds an extra hint to the message of the exception already being thrown if the resource is not found, but only if there is a '-' character in the name that failed to load.

Comment by Andy Fingerhut [ 30/Sep/14 6:45 PM ]

clj-1297-v3.patch is nearly identical to clj-1297-v2.patch, described in the previous comment, except it eliminates an unnecessary let expression.

Comment by Alex Miller [ 01/Oct/14 3:05 PM ]

With latest:

user=> (require 'a-b)
FileNotFoundException Could not locate a_b__init.class or a_b.clj on classpath:   Perhaps a file you are attempting to load needs - chars in name replaced with _  clojure.core/load-one/fn--5135 (core.clj:5606)

That looks goofy due to the base message in RT.load(): throw new FileNotFoundException(String.format("Could not locate %s or %s on classpath: ", classfile, cljfile));

I would like to:
1) Fix that RT.load() message to not end with a colon: "Could not locate %s or %s on classpath."
2) Instead of changing load-one, just add the additional sentence in RT.load(). Second optional sentence could be: "Please check that namespaces with dashes use underscores in the Clojure file name."

Final message would then look like:

"Could not locate a_b__init.class or a_b.clj on classpath. Please check that namespaces with dashes use underscores in the Clojure file name."

Comment by Andy Fingerhut [ 02/Oct/14 12:06 AM ]

Patch clj-1297-v4.patch modifies only RT.load, including an extra message only if the file name in the argument contains a '_' character in it, with the message suggested by Alex Miller in his last comment before this.

Comment by Alex Miller [ 02/Oct/14 8:55 AM ]

The bad end colon is not fixed and the optional message is not included in the format string.

Comment by Andy Fingerhut [ 02/Oct/14 11:04 AM ]

clj-1297-v5.patch should be the one. I can only attempt to blame the previous bone-headed failure on lack of sleep, but even then ...





[CLJ-1279] Fix confusing macroexpand1 ArityException handling Created: 16/Oct/13  Updated: 05/Feb/14

Status: Open
Project: Clojure
Component/s: None
Affects Version/s: Release 1.6
Fix Version/s: None

Type: Defect Priority: Major
Reporter: Alex Coventry Assignee: Unassigned
Resolution: Unresolved Votes: 3
Labels: Compiler, errormsgs, macro

Attachments: Text File 0001-Edit-macro-ArityException-in-AFn.patch     Text File 0001-Fix-macroexpand1-s-handling-of-ArityException.patch    
Patch: Code and Test

 Description   

macros can give very confusing error messages when they execute a form which generates an ArityException. clojure.lang.Compiler.macroexpand1 assumes that any ArityException comes from the call to the macro itself, which need not be the case. For instance:

user> (do (defmacro f [] (assoc)) (f))
ArityException Wrong number of args (-2) passed to: core$assoc clojure.lang.Compiler.macroexpand1 (Compiler.java:6488)
user> (use 'clojure.repl) (pst)
nil
ArityException Wrong number of args (-2) passed to: core$assoc
clojure.lang.Compiler.macroexpand1 (Compiler.java:6488)
clojure.lang.Compiler.macroexpand (Compiler.java:6544)
clojure.lang.Compiler.eval (Compiler.java:6618)
clojure.lang.Compiler.eval (Compiler.java:6624)
clojure.lang.Compiler.eval (Compiler.java:6597)
clojure.core/eval (core.clj:2864)
clojure.main/repl/read-eval-print-6596/fn-6599 (main.clj:260)
clojure.main/repl/read-eval-print--6596 (main.clj:260)
clojure.main/repl/fn--6605 (main.clj:278)
clojure.main/repl (main.clj:278)
clojure.tools.nrepl.middleware.interruptible-eval/evaluate/fn--1251 (interruptible_eval.clj:56)
clojure.core/apply (core.clj:617)
nil

Easy enough to see the source of the problem in this case, but because both the number of arguments actually passed is off by two, and the stacktrace element for the call to assoc has been dropped, this shortcut by macroexpand1 can get super confusing.

The attached patch corrects this behavior. E.g.

user=> (do (defmacro f [] (assoc)) (f))
ArityException Wrong number of args (0) passed to: core$assoc clojure.lang.AFn.throwArity (AFn.java:437)
user=> (use 'clojure.repl) (pst)
nil
ArityException Wrong number of args (0) passed to: core$assoc
user/f (NO_SOURCE_FILE:1)
clojure.lang.Var.invoke (Var.java:419)
clojure.lang.Var.applyTo (Var.java:532)
clojure.lang.Compiler.macroexpand1 (Compiler.java:6507)
clojure.lang.Compiler.macroexpand (Compiler.java:6580)
clojure.lang.Compiler.eval (Compiler.java:6654)
clojure.lang.Compiler.eval (Compiler.java:6660)
clojure.lang.Compiler.eval (Compiler.java:6633)
clojure.core/eval (core.clj:2864)
clojure.main/repl/read-eval-print-6594/fn-6597 (main.clj:260)
clojure.main/repl/read-eval-print--6594 (main.clj:260)
clojure.main/repl/fn--6603 (main.clj:278)
nil



 Comments   
Comment by Alex Coventry [ 17/Oct/13 11:01 AM ]

Patch with test

Comment by Alex Coventry [ 23/Oct/13 11:42 PM ]

Amended patch to deal more gracefully with unexpected stack trace structure.

Comment by Alex Miller [ 24/Oct/13 12:09 AM ]

Also see CLJ-397 and CLJ-383.

Comment by Alex Coventry [ 24/Oct/13 2:46 PM ]

Thanks, Alex. It would be easy enough to move most of the logic into ArityException, which would be a compromise between Stu's[1] options 1 and 2. Is that worth doing?

Amending clojure.lang.AFn.throwArity to check whether "this" is a macro and adjust the arg count there accordingly might be the simplest way. I can see why Rich prefers all the logic to go into ArityException, but since ArityExceptions are used for things other than macros, I don't see a way to make an honest error message there without groveling the stack trace.

[1] http://dev.clojure.org/jira/browse/CLJ-397?focusedCommentId=24090&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-24090

Comment by Alex Miller [ 24/Oct/13 2:57 PM ]

I would have to take more time than I have to make an informed opinion but I can say that from a general point of view inspecting StackTraceElements does not seem like the right solution to (almost) any problem.

Comment by Alex Coventry [ 24/Oct/13 10:26 PM ]

This patch causes Var.setMacro to set instance attribute AFn.macrop to true, so that AFn.throwArity can reduce the number of arguments reported.

I'm not used to negotiating java class hierarchies, so it's possible there's a cleaner way. Since Var.fn() returns an IFn, I added macrop handling methods IFn.setMacro and IFn.isMacro. These then needed to be implemented in Ref and Keyword, as well as AFn (where I wanted them) because they implement the IFn interface but don't inherit from AFn.

The real drawback I see with this approach is the duplicated state, though: ^{:macro true} vs AFn.macrop==true.

Comment by Andy Fingerhut [ 25/Oct/13 6:33 PM ]

I have not investigated the reason yet, but neither patch applies cleanly after the latest commits to Clojure master on Oct 25 2013. Given that what kinds of solution methods would be acceptable for this issue, it sounds like more thinking and code changes are probably needed anyway before worrying too much about that.





[CLJ-1274] Unable to set compiler options via system properties except for AOT compilation Created: 02/Oct/13  Updated: 29/Aug/14  Resolved: 29/Aug/14

Status: Closed
Project: Clojure
Component/s: None
Affects Version/s: Release 1.5
Fix Version/s: Release 1.7

Type: Defect Priority: Major
Reporter: Howard Lewis Ship Assignee: Unassigned
Resolution: Completed Votes: 3
Labels: compiler

Attachments: Text File CLJ-1274.patch    
Patch: Code
Approval: Ok

 Description   

The code that converts JVM system properties into keys under the *compiler-options* var is present only inside the clojure.lang.Compile class. This is a problem when using a debugger inside an IDE and not AOT compiling; specifying -Dclojure.compiler.disable-locals-clearing=true has no effect here when it would be most useful!

Patch: CLJ-1274.patch
Screened by: Stu



 Comments   
Comment by Howard Lewis Ship [ 02/Oct/13 4:52 PM ]

Obviously, that's supposed to be *compiler-options*.

Comment by Howard Lewis Ship [ 02/Dec/13 4:03 PM ]

Changes initialization of *compiler-options* to occur statically inside Compiler; now available to all forms of Clojure, not just AOT compilation; however, the initial *compiler-options* value is now defined as a root binding, rather than a per-thread binding, which has slightly different semantics.

Comment by Stuart Halloway [ 27/Jun/14 1:45 PM ]

Patch is straightforward, marking screened.

I am left wondering if other options that are set only in Compile.java ought also to be moved.





[CLJ-1263] Allow static compilation of function invocations Created: 14/Sep/13  Updated: 07/Nov/13

Status: Open
Project: Clojure
Component/s: None
Affects Version/s: None
Fix Version/s: None

Type: Enhancement Priority: Major
Reporter: Mike Anderson Assignee: Unassigned
Resolution: Unresolved Votes: 3
Labels: compiler


 Description   

This proposal is to allow metadata on functions to prevent a fully dynamic var deref to be used whenever the function is called.

When the function is invoked, JVM "invokevirtual" instruction will be used, which is faster than the current implementation (var deref + IFn cast + invokinterface) and has less restrictions (no need to predefine interfaces to match the function parameters). The JVM is generally able to compile such invokevirtual instructions into extremely efficient code - effectively as fast as pure Java.

This is intended to pave the way to better support for statically compiled, high performance code. In particular, it allow:

  • Supporting arbitrary JVM primitives (float, int, byte, char etc.) as well as just double/long.
  • Supporting typed return values e.g. "String". This could eliminate many casts and type checks.
  • Supporting typed reference arguments (e.g. String).

Suggested usage:

(defn ^:static foo ^int [^String a ^String b]
(+ (count a) (Integer/parseInt b)))

Existing code / semantics should not be affected



 Comments   
Comment by Alex Fowler [ 18/Sep/13 5:08 AM ]

Very nice! That is what would really improve experience with certain tasks. I think it will also make possible to work with primitive arrays without the conversions?

Comment by Mike Anderson [ 19/Sep/13 5:44 PM ]

Hi Alex - which aspect of "work with primitive arrays" are you referring to? This feature would certainly help with passing primitive arguments to/from functions that use primitive arrays. It would also potentially help avoid some casts of primitive array arguments themselves. I don't think it helps in any other way - perhaps a separate issue would be appropriate if there is another thing you are trying to do?

Comment by Kevin Downey [ 29/Oct/13 11:50 AM ]

this issue is confusing, because there was/is a :static feature in clojure(which seems to be disabled in the compiler at the moment) and this proposal doesn't mention the existing work at all.

I also think this proposal is begging the question, there is no discussion of other possible solutions to the performance problem (whatever that is) that this is trying to solve.

the (var.deref()(IFn)).invoke(...) is pretty fundamental to the feel of clojure, in fact the existing :static keyword seems to be disabled in the compiler exactly because it complicates those semantics. so we should have a very clear proposal (not a wishlist) if we want to change that with some very clear wins.

maybe an optimizing clojure compiler would be a better approach.

Comment by Mike Anderson [ 30/Oct/13 11:01 PM ]

Hi Kevin,

This is partly in response to this discussion on Clojure-Dev, where we discussed there are quite a lot of performance issues around the way that Clojure passes arguments currently:
https://groups.google.com/d/topic/clojure/H5P25eYKBj4/discussion

Also I believe it reinstates the original intention of "^:static": I can't find where this is/was officially documented, but Arthur's answer in this SO question suggests that this was the case:
http://stackoverflow.com/questions/7552632/what-does-static-do-in-clojure

I think the proposal is relatively clear: it's probably the minimal change required to get static/direct (i.e. not via an indirect var reference / IFn) function invocations without affecting any of the semantics of current code.

This is sufficiently important for me that it's preventing me from shifting some performance-critical code to Clojure (even with primitive type hints). e.g. here's a simple case with a small primitive function:

(defn ^long foo [^long x]
(inc x))

(c/quick-bench (dotimes [i 100] (foo i))) ;; c = criterium
=> Execution time mean : 194.334293 ns

(c/quick-bench (dotimes [i 100] (inc i)))
=> Execution time mean : 71.539048 ns

i.e. the indirect function invocation is costing us nearly 170% overhead. In Java the equivalent functions perform identically: the overhead is zero because with static function invocation the JVM JIT is able to eliminate all the function call overhead.

In the long term, I agree that a proper optimising compiler would be the best way forward (perhaps Clojure 2.0/CinC can give us this?) but in the meantime I think this is a pragmatic way to improve performance with minimal impact on existing code. Even with an optimising compiler, I think we' would need some way to specifiy the "optimised" semantics rather than the indirect var deref behaviour, and "^:static" seems like a reasonable way to do so (unless anyone has a better idea?)

Comment by Kevin Downey [ 04/Nov/13 3:58 PM ]

have you looked at the definition of int and how it uses :inline/definline to avoid the call overhead?

Comment by Mike Anderson [ 05/Nov/13 4:27 AM ]

Good point Kevin - :inline and definline seem like a good approach in many cases (although it's marked as "experimental" - does that mean we can't rely on it to work in future releases?).

This proposal is still somewhat different: the inline solutions and its variants are effectively doing macro expansion to generate code without a function call on the Clojure side. The approach in this proposal would still emit a function call in bytecode, but do so in a way that the JVM can subsequently inline and optimise much more efficiently. Both have their uses, I think?

Commented edited Nov 7 2013 by Andy Fingerhut: Regarding definline marked as experimental, it has been so marked since Clojure 1.0's release, and the plan is to keep it marked that way in the pending Clojure 1.6 release. See discussion thread on CLJ-1281. No plans to remove it that I am aware of.

Comment by Kevin Downey [ 06/Nov/13 2:06 PM ]

my point is your benchmark above is not a comparison of clojure's current deref + cast + invoke vs. invokevirtual, inc is being inlined in to a static method call there

Comment by Kevin Downey [ 06/Nov/13 2:32 PM ]

I've been noodling around this, and it is entirely possible to generate and invoke code in clojure right now without paying the extra deref() cost:

 (defn ^long fib [^long n]
   (case n
     0 0
     1 1
     (+ (fib (dec n))
        (fib  (dec (dec n))))))

can be written as

(declare TheR1798)

(definterface I1797
  (^long fib_Invoke_1 [^long n]))

(deftype R1798 []
  I1797
  (^long fib_Invoke_1
    [this1799 ^long n]
    (case n
      0 0
      1 1
      (+ (.fib_Invoke_1 this1799 (dec n))
         (.fib_Invoke_1 this1799 (dec (dec n)))))))

(def TheR1798 (new R1798))

(defn ^long fib [^long n]
  (.fib_Invoke_1 TheR1798  n)))

now the recursive calls are invokeinterfaces, and the resulting function seems to have mean execution time about 5 times smaller using criterium to bench mark

it is entirely possible to write a macro that translates one in to other, and the weird names in the above are because I have a little proof of concept that does that.

the body of the bytecode for the regular fib function first shown looks something like:

  public final java.lang.Object invokePrim(long);
    Code:
       0: lload_1       
       1: lstore_3      
       2: lload_3       
       3: l2i           
       4: tableswitch   { // 0 to 1
                     0: 28
                     1: 40
               default: 52
          }
      28: lconst_0      
      29: lload_3       
      30: lcmp          
      31: ifne          52
      34: getstatic     #33                 // Field const__1:Ljava/lang/Object;
      37: goto          94
      40: lconst_1      
      41: lload_3       
      42: lcmp          
      43: ifne          52
      46: getstatic     #37                 // Field const__3:Ljava/lang/Object;
      49: goto          94
      52: getstatic     #57                 // Field const__5:Lclojure/lang/Var;
      55: invokevirtual #70                 // Method clojure/lang/Var.getRawRoot:()Ljava/lang/Object;
      58: checkcast     #6                  // class clojure/lang/IFn$LO
      61: lload_1       
      62: invokestatic  #75                 // Method clojure/lang/Numbers.dec:(J)J
      65: invokeinterface #77,  3           // InterfaceMethod clojure/lang/IFn$LO.invokePrim:(J)Ljava/lang/Object;
      70: getstatic     #57                 // Field const__5:Lclojure/lang/Var;
      73: invokevirtual #70                 // Method clojure/lang/Var.getRawRoot:()Ljava/lang/Object;
      76: checkcast     #6                  // class clojure/lang/IFn$LO
      79: lload_1       
      80: invokestatic  #75                 // Method clojure/lang/Numbers.dec:(J)J
      83: invokestatic  #75                 // Method clojure/lang/Numbers.dec:(J)J
      86: invokeinterface #77,  3           // InterfaceMethod clojure/lang/IFn$LO.invokePrim:(J)Ljava/lang/Object;
      91: invokestatic  #81                 // Method clojure/lang/Numbers.add:(Ljava/lang/Object;Ljava/lang/Object;)Ljava/lang/Number;
      94: areturn       
    LineNumberTable:
      line 243: 0
      line 244: 2
      line 247: 52
      line 247: 52
      line 247: 61
      line 248: 70
      line 248: 79
      line 248: 79
    LocalVariableTable:
      Start  Length  Slot  Name   Signature
             2      92     3 G__301   J
             0      94     0  this   Ljava/lang/Object;
             0      94     1     n   J

  public java.lang.Object invoke(java.lang.Object);
    Code:
       0: aload_0       
       1: aload_1       
       2: checkcast     #89                 // class java/lang/Number
       5: invokestatic  #93                 // Method clojure/lang/RT.longCast:(Ljava/lang/Object;)J
       8: invokeinterface #77,  3           // InterfaceMethod clojure/lang/IFn$LO.invokePrim:(J)Ljava/lang/Object;
      13: areturn       

the body of the "optimized" version looks like:

  public long fib_Invoke_1(long);
    Code:
       0: lload_1       
       1: lstore_3      
       2: lload_3       
       3: l2i           
       4: tableswitch   { // 0 to 1
                     0: 28
                     1: 38
               default: 48
          }
      28: lconst_0      
      29: lload_3       
      30: lcmp          
      31: ifne          48
      34: lconst_0      
      35: goto          80
      38: lconst_1      
      39: lload_3       
      40: lcmp          
      41: ifne          48
      44: lconst_1      
      45: goto          80
      48: aload_0       
      49: checkcast     #6                  // class com/thelastcitadel/kernel/I2364
      52: lload_1       
      53: invokestatic  #53                 // Method clojure/lang/Numbers.dec:(J)J
      56: invokeinterface #55,  3           // InterfaceMethod com/thelastcitadel/kernel/I2364.fib_Invoke_1:(J)J
      61: aload_0       
      62: checkcast     #6                  // class com/thelastcitadel/kernel/I2364
      65: lload_1       
      66: invokestatic  #53                 // Method clojure/lang/Numbers.dec:(J)J
      69: invokestatic  #53                 // Method clojure/lang/Numbers.dec:(J)J
      72: invokeinterface #55,  3           // InterfaceMethod com/thelastcitadel/kernel/I2364.fib_Invoke_1:(J)J
      77: invokestatic  #59                 // Method clojure/lang/Numbers.add:(JJ)J
      80: lreturn       
    LineNumberTable:
      line 251: 0
      line 251: 2
      line 251: 48
      line 251: 48
      line 251: 52
      line 251: 61
      line 251: 65
      line 251: 65
    LocalVariableTable:
      Start  Length  Slot  Name   Signature
             2      78     3 G__2363   J
             0      80     0  this   Lcom/thelastcitadel/kernel/R2365;
             0      80     1     n   J

so the calls are not invokevirtual (due to the way clojure compiles stuff, you cannot type anything inside a record as being that record's type), but the interface is unique and only has one instance, so I think the jvm's class hierarchy analysis makes short work of that.

if I have time I may try and complete my macro and release it as a library, but given tools.analyzer.jvm someone should be able to do better than my little proof of concept very quickly.

Comment by Andy Fingerhut [ 07/Nov/13 12:48 PM ]

I don't know if my editing of Mike Anderson's Nov 5 2013 comment is notified to people watching this ticket, so adding a new comment so those interested in definline's experimental status can know to go back and re-read it.





[CLJ-1256] Support type-hinted overrides of function parameters Created: 06/Sep/13  Updated: 09/Sep/13

Status: Open
Project: Clojure
Component/s: None
Affects Version/s: None
Fix Version/s: None

Type: Enhancement Priority: Major
Reporter: Mike Anderson Assignee: Unassigned
Resolution: Unresolved Votes: 2
Labels: compiler, interop, typehints


 Description   

Problem: in many cases, the Clojure compiler has enough information about the type of a function argument to statically emit maximally efficient code on the JVM (i.e. without instance? checks, type casts or other forms of dynamic polymorphic dispatch). We are currently unable to do so in Clojure, which pushes developers with strong performance requirements to use some unidiomatic or convoluted workarounds.

Proposal is simply to allow functions to take type-hinted overloads of function arguments, e.g.

(defn foo
([^double x] (Math/floor x))
([^float x] (Math/floor (double x)))
([^String s] (count s)))

An "Object" version of the code with the correct arity will always be emitted, which will maintain compatibility with the IFn interface and ensure that the function can still be used in dynamic / interactive contexts. If the "Object" version is not explicitly provided, then it will be generated to use instance? checks that subsequently delegate to the appropriate typed version of the function (or throw an InvalidArgumentException if no match is found).

Matching rules would be the same as Java.

This will be backwards compatible with all existing uses of defn. In particular, it should extend / enhance / supercede the existing handling of primitive functions.

In the future, this technique might be used alongside core.typed to ensure that the most efficient function version is chosen based on type inference.






[CLJ-1250] Reducer (and folder) instances hold onto the head of seqs Created: 03/Sep/13  Updated: 07/Aug/15  Resolved: 07/Aug/15

Status: Closed
Project: Clojure
Component/s: None
Affects Version/s: Release 1.5
Fix Version/s: Release 1.8

Type: Defect Priority: Major
Reporter: Christophe Grand Assignee: Unassigned
Resolution: Completed Votes: 10
Labels: compiler, memory, reducers

Attachments: Text File after-change.txt     Text File before-change.txt     Text File CLJ-1250-08-29.patch     Text File CLJ-1250-08-29-ws.patch     Text File CLJ-1250-20131211.patch     Text File clj-1250-2.patch     Text File CLJ-1250-AllInvokeSites-20140204.patch     Text File CLJ-1250-AllInvokeSites-20140320.patch     Text File clj1250.patch     Text File do-not-clear-this-when-direct-linked.patch    
Patch: Code and Test
Approval: Vetted

 Description   

Problem Statement
A shared function #'clojure.core.reducers/reducer holds on to the head of a reducible collection, causing it to blow up when the collection is a lazy sequence.

Reproduction steps:
Compare the following calls:

(time (reduce + 0 (map identity (range 1e8))))
(time (reduce + 0 (r/map identity (range 1e8))))

The second call should fail on a normal or small heap.

(If reducers are faster than seqs, increase the range.)

Cause: #'reducer closes over a collection when in order to reify CollReduce, and the closed-over is never cleared. When code attempts to reduce over this anonymous transformed collection, it will realize the tail while the head is stored in the closed-over.

Patch: clj-1250-2.patch

Approach:

Clear the reference to 'this' on the stack just before a tail call occurs

Removes calls to emitClearLocals(), which is a no-op.

When the context is RETURN (indicating a tail call), and the operation
is an InvokeExpr, StaticMethodExpr, or InstanceMethodExpr, clear the
reference to 'this' which is in slot 0 of the locals.

Edge-case: Inside the body of a try block, we cannot clear 'this' at the tail
position as we might need to keep refs around for use in the catch or finally
clauses. Introduces another truthy dynamic binding var to track position being
inside a try block.

Adds two helpers to emitClearThis and inTailCall.

Advantages: Fixes this case with no user code changes. Enables GC to do reclaim closed-overs references earlier.
Disadvantages: A compiler change.

Screened by: Alex Miller

Alternate Approaches:

1) Reimplement the #'reducer (and #'folder) transformation fns similar to the manner that Christophe proposes here:

(defrecord Reducer [coll xf])

(extend-protocol 
  clojure.core.protocols/CollReduce
  Reducer
      (coll-reduce [r f1]
                   (clojure.core.protocols/coll-reduce r f1 (f1)))
      (coll-reduce [r f1 init]
                   (clojure.core.protocols/coll-reduce (:coll r) ((:xf r) f1) init)))

(def rreducer ->Reducer) 

(defn rmap [f coll]
  (rreducer coll (fn [g] 
                   (fn [acc x]
                     (g acc (f x))))))

Advantages: Relatively non-invasive change.
Disadvantages: Not evident code. Additional protocol dispatch, though only incurred once

2) Alternate approach

from Christophe Grand:
Another way would be to enhance the local clearing mechanism to also clear "this" but it's complex since it may be needed to keep a reference to "this" for caches long after obvious references to "this" are needed.

Advantages: Fine-grained
Disadvantages: Complex, invasive, and the compiler is hard to hack on.

Mitigations
Avoid reducing on lazy-seqs and instead operate on vectors / maps, or custom reifiers of CollReduce or CollFold. This could be easier with some implementations of common collection functions being available (like iterate and partition).

See https://groups.google.com/d/msg/clojure-dev/t6NhGnYNH1A/2lXghJS5HywJ for previous discussion.



 Comments   
Comment by Gary Fredericks [ 03/Sep/13 8:53 AM ]

Fixed indentation in description.

Comment by Ghadi Shayban [ 11/Dec/13 11:08 PM ]

Adding a patch that clears "this" before tail calls. Verified that Christophe's repro case is fixed.

Will upload a diff of the bytecode soon.

Any reason this juicy bug was taken off 1.6?

Comment by Ghadi Shayban [ 11/Dec/13 11:17 PM ]

Here's the bytecode for the clojure.core.reducers/reducer reify before and after the change... Of course a straight diff isn't useful because all the line numbers changed. Kudos to Gary Trakhman for the no.disassemble lein plugin.

Comment by Christophe Grand [ 12/Dec/13 6:58 AM ]

Ghadi, I'm a bit surprised by this part of the patch: was the local clearing always a no-op here?

-		if(context == C.RETURN)
+		if(shouldClear)
 			{
-			ObjMethod method = (ObjMethod) METHOD.deref();
-			method.emitClearLocals(gen);
+                            gen.visitInsn(Opcodes.ACONST_NULL);
+                            gen.visitVarInsn(Opcodes.ASTORE, 0);
 			}

The problem with this approach (clear this on tail call) is that it adds yet another special case. To me the complexity stem from having to keep this around even if the user code doesn't refer to it.

Comment by Ghadi Shayban [ 12/Dec/13 7:19 AM ]

Thank you - I failed to mention this in the commit message: it appears that emitClearLocals() belonging to both ObjMethod and FnMethod (its child) are empty no-ops. I believe the actual local clearing is on line 4855.

I agree re: another special case in the compiler.

Comment by Alex Miller [ 12/Dec/13 8:56 AM ]

Ghadi re 1.6 - this ticket was never in the 1.6 list, it has not yet been vetted by Rich but is ready to do so when we open up again after 1.6.

Comment by Ghadi Shayban [ 12/Dec/13 8:59 AM ]

Sorry I confused the critical list with the Rel1.6 list.

Comment by Ghadi Shayban [ 14/Dec/13 11:16 AM ]

New patch 20131214 that handles all tail invoke sites (InvokeExpr + StaticMethodExpr + InstanceMethodExpr). 'StaticInvokeExpr' seems like an old remnant that had no active code path, so that was left as-is.

The approach taken is still the same as the original small patch that addressed only InvokeExpr, except that it is now using a couple small helpers. The commit message has more details.

Also a 'try' block with no catch or finally clause now becomes a BodyExpr. Arguably a user error, historically accepted, and still accepted, but now they are a regular BodyExpr, instead of being wrapped by a the no-op try/catch mechanism. This second commit can be optionally discarded.

With this patch on my machine (4/8 core/thread Ivy Bridge) running on bare clojure.main:
Christophe's test cases both run i 3060ms on a artificially constrained 100M max heap, indicating a dominant GC overhead. (But they now both work!)

When max heap is at a comfortable 2G the reducers version outpaces the lazyseq at 2100ms vs 2600ms!

Comment by Ghadi Shayban [ 13/Jan/14 10:48 AM ]

Updating stale patch after latest changes to master. Latest is CLJ-1250-AllInvokeSites-20140113

Comment by Ghadi Shayban [ 04/Feb/14 3:50 PM ]

Updating patch after murmur changes

Comment by Tassilo Horn [ 13/Feb/14 4:52 AM ]

Ghadi, I suffer from the problem of this issue. Therefore, I've applied your patch CLJ-1250-AllInvokeSites-20140204.patch to the current git master. However, then I get lots of "java.lang.NoSuchFieldError: array" errors when the clojure tests are run:

     [java] clojure.test-clojure.clojure-set
     [java] 
     [java] java.lang.NoSuchFieldError: array
     [java] 	at clojure.core.protocols$fn__6026.invoke(protocols.clj:123)
     [java] 	at clojure.core.protocols$fn__5994$G__5989__6003.invoke(protocols.clj:19)
     [java] 	at clojure.core.protocols$fn__6023.invoke(protocols.clj:147)
     [java] 	at clojure.core.protocols$fn__5994$G__5989__6003.invoke(protocols.clj:19)
     [java] 	at clojure.core.protocols$seq_reduce.invoke(protocols.clj:31)
     [java] 	at clojure.core.protocols$fn__6017.invoke(protocols.clj:48)
     [java] 	at clojure.core.protocols$fn__5968$G__5963__5981.invoke(protocols.clj:13)
     [java] 	at clojure.core$reduce.invoke(core.clj:6213)
     [java] 	at clojure.set$difference.doInvoke(set.clj:61)
     [java] 	at clojure.lang.RestFn.invoke(RestFn.java:442)
     [java] 	at clojure.test_clojure.clojure_set$fn__1050$fn__1083.invoke(clojure_set.clj:109)
     [java] 	at clojure.test_clojure.clojure_set$fn__1050.invoke(clojure_set.clj:109)
     [java] 	at clojure.test$test_var$fn__7123.invoke(test.clj:704)
     [java] 	at clojure.test$test_var.invoke(test.clj:704)
     [java] 	at clojure.test$test_vars$fn__7145$fn__7150.invoke(test.clj:721)
     [java] 	at clojure.test$default_fixture.invoke(test.clj:674)
     [java] 	at clojure.test$test_vars$fn__7145.invoke(test.clj:721)
     [java] 	at clojure.test$default_fixture.invoke(test.clj:674)
     [java] 	at clojure.test$test_vars.invoke(test.clj:718)
     [java] 	at clojure.test$test_all_vars.invoke(test.clj:727)
     [java] 	at clojure.test$test_ns.invoke(test.clj:746)
     [java] 	at clojure.core$map$fn__2665.invoke(core.clj:2515)
     [java] 	at clojure.lang.LazySeq.sval(LazySeq.java:40)
     [java] 	at clojure.lang.LazySeq.seq(LazySeq.java:49)
     [java] 	at clojure.lang.Cons.next(Cons.java:39)
     [java] 	at clojure.lang.RT.boundedLength(RT.java:1655)
     [java] 	at clojure.lang.RestFn.applyTo(RestFn.java:130)
     [java] 	at clojure.core$apply.invoke(core.clj:619)
     [java] 	at clojure.test$run_tests.doInvoke(test.clj:761)
     [java] 	at clojure.lang.RestFn.applyTo(RestFn.java:137)
     [java] 	at clojure.core$apply.invoke(core.clj:617)
     [java] 	at clojure.test.generative.runner$run_all_tests$fn__527.invoke(runner.clj:255)
     [java] 	at clojure.test.generative.runner$run_all_tests$run_with_counts__519$fn__523.invoke(runner.clj:251)
     [java] 	at clojure.test.generative.runner$run_all_tests$run_with_counts__519.invoke(runner.clj:251)
     [java] 	at clojure.test.generative.runner$run_all_tests.invoke(runner.clj:253)
     [java] 	at clojure.test.generative.runner$test_dirs.doInvoke(runner.clj:304)
     [java] 	at clojure.lang.RestFn.applyTo(RestFn.java:137)
     [java] 	at clojure.core$apply.invoke(core.clj:617)
     [java] 	at clojure.test.generative.runner$_main.doInvoke(runner.clj:312)
     [java] 	at clojure.lang.RestFn.invoke(RestFn.java:408)
     [java] 	at user$eval564.invoke(run_tests.clj:3)
     [java] 	at clojure.lang.Compiler.eval(Compiler.java:6657)
     [java] 	at clojure.lang.Compiler.load(Compiler.java:7084)
     [java] 	at clojure.lang.Compiler.loadFile(Compiler.java:7040)
     [java] 	at clojure.main$load_script.invoke(main.clj:274)
     [java] 	at clojure.main$script_opt.invoke(main.clj:336)
     [java] 	at clojure.main$main.doInvoke(main.clj:420)
     [java] 	at clojure.lang.RestFn.invoke(RestFn.java:408)
     [java] 	at clojure.lang.Var.invoke(Var.java:379)
     [java] 	at clojure.lang.AFn.applyToHelper(AFn.java:154)
     [java] 	at clojure.lang.Var.applyTo(Var.java:700)
     [java] 	at clojure.main.main(main.java:37)
Comment by Ghadi Shayban [ 13/Feb/14 8:23 AM ]

Can you give some details about your JVM/environment that can help reproduce? I'm not encountering this error.

Comment by Tassilo Horn [ 13/Feb/14 9:41 AM ]

Sure. It's a 64bit ThinkPad running GNU/Linux.

% java -version
java version "1.7.0_51"
OpenJDK Runtime Environment (IcedTea 2.4.5) (ArchLinux build 7.u51_2.4.5-1-x86_64)
OpenJDK 64-Bit Server VM (build 24.51-b03, mixed mode)
Comment by Ghadi Shayban [ 13/Feb/14 10:19 AM ]

Strange, that is exactly my mail env, OpenJDK7 on Arch, 64-bit. I have also tested on JDK 6/7/8 on OSX mavericks. Are you certain that the git tree is clean besides the patch? (Arch users unite!)

Comment by Tassilo Horn [ 14/Feb/14 1:13 AM ]

Yes, the tree is clean. But now I see that I get the same error also after resetting to origin/master, so it's not caused by your patch at all. Oh, now the error vanished after doing a `mvn clean`! So problem solved.

Comment by Nicola Mometto [ 19/Feb/14 12:32 PM ]

Ghandi, FnExpr.parse should bind IN_TRY_BLOCK to false before analyzing the fn body, consider the case

(try (do something (fn a [] (heap-consuming-op a))) (catch Exception e ..))

Here in the a function the this local will never be cleared even though it's perfectly safe to.
Admittedly this is an edge case but we should cover this possibility too.

Comment by Ghadi Shayban [ 19/Feb/14 2:06 PM ]

You may have auto-corrected my name to Ghandi instead of Ghadi. I wish I were that wise =)

I will update the patch for FnExpr (that seems reasonable), but maybe after 1.6 winds down and the next batch of tickets get scrutiny. It would be nice to get input on a preferred approach from Rich or core after it gets vetted – or quite possibly not vetted.

Comment by Nicola Mometto [ 19/Feb/14 6:11 PM ]

hah, sorry for the typo on the name

Seems reasonable to me, in the meantime I just pushed to tools.analyzer/tools.emitter complete support for "this" clearing, I'll test this a bit in the next few days to make sure it doesn't cause unexpected problems.

Comment by Andy Fingerhut [ 24/Feb/14 12:13 PM ]

Patch CLJ-1250-AllInvokeSites-20140204.patch no longer applies cleanly to latest master as of Feb 23, 2014. It did on Feb 14, 2014. Most likely some of its context lines are changed by the commit to Clojure master on Feb 23, 2014 – I haven't checked in detail.

Comment by Ghadi Shayban [ 20/Mar/14 4:39 PM ]

Added a patch that 1) applies cleanly, 2) binds the IN_TRY_EXPR to false initially when analyzing FnExpr and 3) uses RT.booleanCast

Comment by Alex Miller [ 22/Aug/14 9:31 AM ]

Can you squash the patch and add tests to cover all this stuff?

Comment by Ghadi Shayban [ 22/Aug/14 9:47 AM ]

Sure. Have any ideas for how to test proper behavior of reference clearing? Know of some prior art in the test suite?

Comment by Alex Miller [ 22/Aug/14 10:25 AM ]

Something like the test in the summary would be a place to start. I don't know of any test that actually inspects bytecode or anything but that's probably not wise anyways. Need to make that kind of a test but get coverage on the different kinds of scenarios you're covering - try/catch, etc.

Comment by Ghadi Shayban [ 22/Aug/14 12:13 PM ]

Attached new squashed patch with a couple of tests.

Removed (innocuous but out-of-scope) second commit that analyzed try blocks missing a catch or finally clause as BodyExprs

Comment by Ghadi Shayban [ 29/Aug/14 11:43 AM ]

Rebased to latest master. Current patch CLJ-1250-08-29

Comment by Jozef Wagner [ 29/Aug/14 2:40 PM ]

CLJ-1250-08-29.patch is fishy, 87k size and it includes many unrelated commits

Comment by Alex Miller [ 29/Aug/14 2:44 PM ]

Agreed, Ghadi that last rebase looks wrong.

Comment by Ghadi Shayban [ 29/Aug/14 3:06 PM ]

Oops. Used format-patch against the wrong base. Updated.

Apologies that ticket is longer than War & Peace

Comment by Alex Miller [ 08/Sep/14 7:02 PM ]

I have not had enough time to examine all the bytecode diffs that I want to on this yet but preliminary feedback:

Compiler.java

  • need to use tabs instead of spaces to blend into the existing code better
  • why do StaticFieldExpr and InstanceFieldExpr not need this same logic?

compilation.clj

  • has some whitespace diffs that you could get rid of
  • there seem to be more cases in the code than are covered in the tests here?
Comment by Ghadi Shayban [ 08/Sep/14 11:19 PM ]

The germ of the issue is to clear the reference to 'this' (arg 0) when transferring control to another activation frame. StaticFieldExpr and InstanceFieldExpr do not transfer control to another frame. (StaticMethod and InstanceMethod do transfer control, and are covered by the patch)

Comment by Alex Miller [ 25/Sep/14 9:03 AM ]

Makes sense - can you address the tabs and whitespace issues?

Comment by Ghadi Shayban [ 26/Sep/14 12:51 PM ]

Latest patch CLJ-1250-08-29-ws.patch with whitespace issues fixed.

Comment by Michael Blume [ 17/Jun/15 4:43 PM ]

Patch doesn't apply, will attempt to fix and upload

Comment by Michael Blume [ 17/Jun/15 4:58 PM ]

Nope, sorry, admitting defeat on this one. Ghadi, can you update?

Comment by Ghadi Shayban [ 22/Jun/15 9:40 PM ]

I've updated the patch for first 1.8 merge window.

Comment by Alex Miller [ 23/Jun/15 7:37 AM ]

Added clj-1250-2.patch which is same as clj1250.patch but squashes commits and fixes whitespace tab/space issues.

Comment by Nicola Mometto [ 17/Jul/15 8:24 PM ]

Not sure what it means but the testcase is failing with an OOM exception in the IBM JDK 1.6 instance http://build.clojure.org/job/clojure-test-matrix/284/jdk=IBM%20JDK%201.6/console

Comment by Alex Miller [ 17/Jul/15 10:29 PM ]

CLJ-1780 created to track the ibm failure. maybe just difference in gc speed or something?

Comment by Alex Miller [ 04/Aug/15 2:32 PM ]

Here's a greatly simplified version of some real code that now fails due to this change:

(let [done (atom 2)]
    (while (pos? @done)
      (swap! done dec)
      (loop [found []]
        (println (conj found 1)))))
  • Before: [1][1]
  • After: [1] NullPointerException
Comment by Ghadi Shayban [ 04/Aug/15 5:18 PM ]

Reproduced on 1.8.0-alpha4. Could not reproduce with the previously committed patch applied directly to 1.7.

Cause: Tail clearing is insensitive to direct linking

In a direct linking static method (through invokeStatic) there's no "this" to clear.

Patch addresses the root cause

Comment by Alex Miller [ 05/Aug/15 9:02 AM ]

Ok, backing out a little, here's another example that still fails with the patch:

(let [done (atom false)
        f (future-call
            (fn inner []
              (while (not @done)
                (loop [found []]
                  (println (conj found 1))))))]
    (doseq [elem [:a :b :c :done]]
      (println "queue write " elem))
    (reset! done true)
    @f)

1.7:
queue write :a
queue write :b
queue write :c
queue write :done
[1]
nil

1.8:
queue write :a
[1]
queue write :b
queue write :c
queue write :done
NullPointerException

Comment by Alex Miller [ 05/Aug/15 10:58 AM ]

Moved latest case to CLJ-1793

Comment by Ghadi Shayban [ 05/Aug/15 8:44 PM ]

Both this regression and the loop regression are fixed on CLJ-1793.

Comment by Alex Miller [ 07/Aug/15 8:56 AM ]

The patch for this ticket has been reverted in master after 1.8.0-alpha4. Further work on re-implementation is moved to CLJ-1793.





[CLJ-1241] NPE when AOTing overrided clojure.core functions Created: 30/Jul/13  Updated: 29/Aug/14  Resolved: 29/Aug/14

Status: Closed
Project: Clojure
Component/s: None
Affects Version/s: Release 1.5, Release 1.6
Fix Version/s: Release 1.7

Type: Defect Priority: Major
Reporter: Phil Hagelberg Assignee: Unassigned
Resolution: Completed Votes: 2
Labels: aot, compiler

Attachments: Text File 0001-fix-CLJ-1241.patch    
Patch: Code
Approval: Ok

 Description   

When performing AOT compilation on a namespace that overrides a clojure.core function without excluding the original clojure.core function from the ns, you get a NullPointerException.

To reproduce aot compile a namespace like "(ns x) (defn get [])"

For example:

$ lein new aot-get
$ cd aot-get
$ sed -i s/foo/get/
$ lein compile :all
WARNING: get already refers to: #'clojure.core/get in namespace: aot-get.core, being replaced by: #'aot-get.core/get
Exception in thread "main" java.lang.NullPointerException
	at clojure.lang.Compiler$ObjExpr.emitVar(Compiler.java:4858)
	at clojure.lang.Compiler$DefExpr.emit(Compiler.java:428)
	at clojure.lang.Compiler.compile1(Compiler.java:7152)
	at clojure.lang.Compiler.compile(Compiler.java:7219)
	at clojure.lang.RT.compile(RT.java:398)
	at clojure.lang.RT.load(RT.java:438)
	at clojure.lang.RT.load(RT.java:411)

Cause: DefExpr.parse does not call registerVar for vars overridding clojure.core ones, thus when AOT compiling the var is not registered in the constant table.

Proposed: The attached patch makes DefExpr.parse call registerVar for vars overridding clojure.core ones.

Patch: 0001-fix-CLJ-1241.patch

Screened by: Alex Miller



 Comments   
Comment by Nicola Mometto [ 30/Jul/13 7:29 PM ]

DefExpr.parse was not calling registerVar for vars overridding clojure.core ones.

Comment by Alex Miller [ 31/Jul/13 12:25 AM ]

Verified on Clojure 1.5.1.

Comment by Javier Neira Sanchez [ 27/Aug/13 8:34 AM ]

Reproduced with `key` function without `(:refer-clojure :exclude [key])`

Comment by Rich Hickey [ 05/Sep/13 8:32 AM ]

This doesn't meet triage guidelines - i.e. there is this problem, therefore we will fix it by _____ so it then does _____

Comment by Aaron Cohen [ 26/Mar/14 12:52 PM ]

This is still present in the 1.6 release. I think it's mis-classified as low priority.

Comment by Aaron Cohen [ 26/Mar/14 12:52 PM ]

See for instance the cascalog mailing list: https://groups.google.com/forum/#!topic/cascalog-user/Pe5QIpmU0vA

Comment by Andy Fingerhut [ 26/Mar/14 1:07 PM ]

It may help if someone could clarify Rich's comment.

Does it mean that the ticket should include a plan of the form "therefore we will fix it by _____ so it then does _____", but this ticket doesn't have that?

Or perhaps it means that the ticket should not include a plan of that form, but this ticket does? If so, I don't see it, except perhaps the very last sentence of the description. If that is a problem for vetting a ticket, perhaps we could just delete that sentence and proceed from there?

Something else?

Comment by Nicola Mometto [ 26/Mar/14 1:13 PM ]

Andy, I added the two last lines in the description after reading Rich's comment to explain why this bug happens and how the patch I attached works around this.

I don't know if this is what he was asking for though.

Comment by Alex Miller [ 27/Mar/14 11:00 AM ]

I think Rich meant that a ticket should have a plan of that form but does not. My own take on "triaged" is that it should state actual and expected results demonstrating a problem - I don't think it needs to actually describe the solution (as that can happen later in development). It is entirely possible that Rich and I differ in our interpretation of that. I will see if I can rework the description a bit to match what I've been doing elsewhere.

Comment by Andy Fingerhut [ 31/Mar/14 9:34 AM ]

Alex, I have looked through the existing wiki pages on the ticket tracking process, and do not recall seeing anything about this desired aspect of a triaged ticket. Is it already documented somewhere and I missed it? Not that it has to be documented, necessarily, but Rich saying "triage guidelines" makes it sound like a filter he applies that ticket creators and screeners maybe should know about.

Comment by Alex Miller [ 31/Mar/14 11:57 AM ]

To me, Triage (and Vetting) is all about having good problem statements. For a defect, it is most important to demonstrate the problem (what happens now) and what you expect to happen instead. I do not usually expect there to necessarily be "by ____" in the ticket - to me that is part of working through the solution (although it is typical to have this in an enhancement). This ticket, as it stands now, seems to have both a good problem statement and a good cause/solution statement so seems to exceed Triaging standards afaik.

Two places where I have tried to write about these things in the past are http://dev.clojure.org/display/community/Creating+Tickets and in the Triage process on the workflow page http://dev.clojure.org/display/community/JIRA+workflow.





[CLJ-1233] Allow ** as a valid symbol name without triggering "not declared dynamic" warnings Created: 23/Jul/13  Updated: 25/Oct/13  Resolved: 25/Oct/13

Status: Closed
Project: Clojure
Component/s: None
Affects Version/s: Release 1.5
Fix Version/s: Release 1.6

Type: Defect Priority: Trivial
Reporter: Mike Anderson Assignee: Unassigned
Resolution: Completed Votes: 0
Labels: Compiler
Environment:

All


Attachments: File clj-1233-minimal.diff     File clj-1233-with-test.diff     File clj-1233-with-test-v2.diff     Text File clj-1233-with-test-v2.txt    
Patch: Code and Test
Approval: Ok

 Description   

Currently declaring a symbol ** triggers a compiler warning:

user=> (defn ** [] nil)
Warning: ** not declared dynamic and thus is not dynamically rebindable, but its name suggests otherwise. Please either indicate ^:dynamic ** or change the name. (NO_SOURCE_PATH:1)
#'user/**

** is a useful symbol in many domains, e.g. as an exponent function or as a matrix multiplication operator.

Cause: This warning checks for a def of a symbol that starts and ends with *.

Solution: Change check for name length >2 to skip this particular case.

Patch: clj-1233-with-test-v2.diff

Screened by: Alex Miller



 Comments   
Comment by Mike Anderson [ 23/Jul/13 6:09 AM ]

Link to discussion on Clojure-Dev
https://groups.google.com/forum/#!topic/clojure-dev/OuTMsZQkxN4

Comment by Mike Anderson [ 23/Jul/13 12:29 PM ]

Minimal patch for the ** case only

Comment by Alex Miller [ 24/Jul/13 12:17 PM ]

Minimal patch would be slightly less minimal with a test.

Comment by Mike Anderson [ 25/Jul/13 5:16 AM ]

Hmmm... is there a standard/reliable method for testing the presence / non-presence of emitted warnings?

Comment by Alex Miller [ 25/Jul/13 3:21 PM ]

There are some test helpers in Clojure's test/clojure/test_helper.clj for capturing error messages.

Comment by Mike Anderson [ 29/Jul/13 12:32 PM ]

New patch with a test that includes using "with-err-print-writer" to detect the avoidance of the warning.

Comment by Andy Fingerhut [ 05/Sep/13 6:24 PM ]

Patch clj-1233-with-test-v2.txt is identical to Mike Anderson's clj-1233-with-test.diff (preserving his authorship), except it avoids git warnings when applying by eliminating trailing whitespace in added lines.





[CLJ-1232] Functions with non-qualified return type hints force import of hinted classes when called from other namespace Created: 18/Jul/13  Updated: 24/Aug/15

Status: Open
Project: Clojure
Component/s: None
Affects Version/s: Release 1.5, Release 1.6, Release 1.7
Fix Version/s: Release 1.8

Type: Defect Priority: Major
Reporter: Tassilo Horn Assignee: Unassigned
Resolution: Unresolved Votes: 8
Labels: compiler, typehints

Attachments: Text File 0001-auto-qualify-arglists-class-names.patch     Text File 0001-auto-qualify-arglists-class-names-v2.patch     Text File 0001-auto-qualify-arglists-class-names-v3.patch     Text File 0001-auto-qualify-arglists-class-names-v4.patch     Text File 0001-CLJ-1232-auto-qualify-arglists-class-names.patch     Text File 0001-throw-on-non-qualified-class-names-that-are-not-auto.patch    
Patch: Code and Test
Approval: Ok

 Description   

You can add a type hint to function arglists to indicate the return type of a function like so.

user> (import '(java.util List))
java.util.List
user> (defn linkedlist ^List [] (java.util.LinkedList.))
#'user/linkedlist
user> (.size (linkedlist))
0

The problem is that now when I call `linkedlist` exactly as above from another namespace, I'll get an exception because java.util.List is not imported in there.

user> (in-ns 'user2)
#<Namespace user2>
user2> (refer 'user)
nil
user2> (.size (linkedlist))
CompilerException java.lang.IllegalArgumentException: Unable to resolve classname: List, compiling:(NO_SOURCE_PATH:1:1)
user2> (import '(java.util List)) ;; Too bad, need to import List here, too.
java.util.List
user2> (.size (linkedlist))
0

There are two workarounds: You can import the hinted type also in the calling namespace, or you always use fully qualified class names for return type hints. Clearly, the latter is preferable.

Approach: Resolve the tags in the defn macro.

Patch: 0001-CLJ-1232-auto-qualify-arglists-class-names.patch

Screened by: Fogus

Alternate approach: Make the compiler resolve the return tags when necessary (tag is not a string, primitive tag (^long) or array tag (^longs)) and update the Var's :arglist appropriately. Patch: 0001-auto-qualify-arglists-class-names-v4.patch. Note that this patch had problems with Hystrix, which was using non-conformant arglists - Hystrix has since been patched.



 Comments   
Comment by Andy Fingerhut [ 16/Apr/14 3:47 PM ]

To make sure I understand, Nicola, in this ticket you are asking that the Clojure compiler change behavior so that the sample code works correctly with no exceptions, the same way as it would work correctly without exceptions if one of the workarounds were used?

Comment by Tassilo Horn [ 17/Apr/14 12:18 AM ]

Hi Andy. Tassilo here, not Nicola. But yes, the example should work as-is. When I'm allowed to use type hints with simple imported class names for arguments, then doing so for return values should work, too.

Comment by Rich Hickey [ 10/Jun/14 10:41 AM ]

Type hints on function params are only consumed by the function definition, i.e. in the same module as the import/alias. Type hints on returns are just metadata, they don't get 'compiled' and if the metadata is not useful to consumers in other namespaces, it's not a useful hint. So, if it's not a type in the auto-imported set (java.lang), it should be fully qualified.

Comment by Alex Miller [ 10/Jun/14 11:55 AM ]

Based on Rich's comment, this ticket should probably morph into an enhancement request on documentation, probably on http://clojure.org/java_interop#Java Interop-Type Hints.

Comment by Andy Fingerhut [ 10/Jun/14 3:13 PM ]

I would suggest something like the following for a documentation change, after this part of the text on the page Alex links in the previous comment:

For function return values, the type hint can be placed before the arguments vector:

(defn hinted
(^String [])
(^Integer [a])
(^java.util.List [a & args]))

-> #user/hinted

If the return value type hint is for a class that is outside of java.lang, which is the part auto-imported by Clojure, then it must be a fully qualified class name, e.g. java.util.List, not List.

Comment by Nicola Mometto [ 10/Jun/14 4:02 PM ]

I don't understand why we should enforce this complexity to the user.
Why can't we just make the Compiler (or even defn itself) update all the arglists tags with properly resolved ones? (that's what I'm doing in tools.analyzer.jvm)

Comment by Alexander Kiel [ 19/Jul/14 10:02 AM ]

I'm with Nicola here. I also think that defn should resolve the type hint according the imports of the namespace defn is used in.

Comment by Max Penet [ 22/Jul/14 7:06 AM ]

Same here, I was bit by this in the past. The current behavior is clearly counterintuitive.

Comment by Nicola Mometto [ 28/Aug/14 12:58 PM ]

Attached two patches implementing two different solutions:

  • 0001-auto-qualify-arglists-class-names.patch makes the compiler automatically qualify all the tags in the :arglists
  • 0001-throw-on-non-qualified-class-names-that-are-not-auto.patch makes the compiler throw an exception for all public defs whose return tag is a symbol representing a non-qualified class that is not in the auto-import list (approach proposed in IRC by Alex Miller)
Comment by Tassilo Horn [ 29/Aug/14 1:49 AM ]

For what it's worth, I'd prefer the first patch because the second doesn't help in situations where the caller lives in a namespace where the called function's return type hinted class is `ns-unmap`-ed. And there a good reasons for doing that. For example, Process is a java.lang class and Process is a pretty generic name. So in some namespace, I want to define my own Process deftype or defrecord. Without unmapping 'Process first to get rid of the java.lang.Process auto-import, I'd get an exception:

user> (deftype Process [])
IllegalStateException Process already refers to: class java.lang.Process in namespace: user  clojure.lang.Namespace.referenceClass (Namespace.java:140)

Now when I call some function from some library that has a `^Process` return type hint (meaning java.lang.Process there), I get the same exception as in my original report.

I can even get into troubles when only using standard Clojure functions because those have `^String` and `^Class` type hints. IMO, Class is also a pretty generic name I should be able to name my custom deftype/defrecord. And I might also want to have a custom String type/record in my astrophysics system.

Comment by Andy Fingerhut [ 30/Sep/14 4:39 PM ]

Not sure whether the root cause of this behavior is the same as the example in the description or not, but seems a little weird that even for fully qualified Java class names hinting the arg vector, it makes a difference whether it is done with defn or def:

Clojure 1.6.0
user=> (set! *warn-on-reflection* true)
true
user=> (defn f1 ^java.util.LinkedList [coll] (java.util.LinkedList. coll))
#'user/f1
user=> (def f2 (fn ^java.util.LinkedList [coll] (java.util.LinkedList. coll)))
#'user/f2
user=> (.size (f1 [2 3 4]))
3
user=> (.size (f2 [2 3 4]))
Reflection warning, NO_SOURCE_PATH:5:1 - reference to field size can't be resolved.
3
Comment by Alex Miller [ 30/Sep/14 6:21 PM ]

Andy, can you file that as a separate ticket?

Comment by Andy Fingerhut [ 30/Sep/14 9:08 PM ]

Created ticket CLJ-1543 for the issue raised in my comment earlier on 30 Sep 2014.

Comment by Andy Fingerhut [ 01/Oct/14 12:38 PM ]

Tassilo (or anyone), is there a reason to prefer putting the tag on the argument vector in your example? It seems that putting it on the Var name instead avoids this issue:

user=> (clojure-version)
"1.6.0"
user=> (set! *warn-on-reflection* true)
true
user=> (import '(java.util List))
java.util.List
user=> (defn ^List linkedlist [] (java.util.LinkedList.))
#'user/linkedlist
user=> (.size (linkedlist))
0
user=> (ns user2)
nil
user2=> (refer 'user)
nil
user2=> (.size (linkedlist))
0

I suppose that only allows a single type tag, rather than an independent one for each arity.

Comment by Tassilo Horn [ 02/Oct/14 3:16 AM ]

I wasn't aware of the fact that you can put it on the var's name. That's not documented at http://clojure.org/java_interop#Java Interop-Type Hints. But IMHO the documented version with putting the tag on the argument vector is more general since it supports different return type hints for the different arity version. In any case, if both forms are permitted then they should be equivalent in the case the function has only one arity.

Comment by Rich Hickey [ 16/Mar/15 12:02 PM ]

Please work on the simplest patch that resolves the names

Comment by Alex Miller [ 16/Mar/15 4:34 PM ]

Nicola, in this:

if (tag != null &&
                        !(tag instanceof String) &&
                        primClass((Symbol)tag) == null &&
                        !tagClass((Symbol) tag).getName().startsWith("["))
                        {
                            argvec = (IPersistentVector)((IObj)argvec).withMeta(RT.map(RT.TAG_KEY, Symbol.intern(tagClass((Symbol)tag).getName())));
                        }

doesn't tagClass already handle most of these cases properly already? Can this be simplified? Is there an optimization case in avoiding lookup for a dotted name?

Comment by Nicola Mometto [ 16/Mar/15 5:10 PM ]

Patch 0001-auto-qualify-arglists-class-names-v2.patch avoids doing unnecessary lookups (dotted names, special tags (primitive tags, array tags)) and adds a testcase

Comment by Michael Blume [ 20/May/15 1:13 PM ]

I'm seeing an odd failure with this patch and hystrix defcommands, will post a small reproduction shortly

Comment by Michael Blume [ 20/May/15 1:20 PM ]

https://github.com/MichaelBlume/hystrix-demo

passes lein check with 1.7 beta3, fails with v3 patch applied

Comment by Nicola Mometto [ 20/May/15 1:40 PM ]

During analysis the compiler understands only arglists in the form of (quote ([..]*)) (see https://github.com/clojure/clojure/blob/master/src/jvm/clojure/lang/Compiler.java#L557-L558), hystrix emits arglists in the form of (list (quote [..])*).

Not sure what to do about this.

Comment by Michael Blume [ 20/May/15 1:51 PM ]

Possibly just ask Hystrix not to do that?

Comment by Alex Miller [ 20/May/15 2:30 PM ]

test.generative uses non-standard arglists too. I haven't looked at the patch, but if it's sensitive to that, it's probably not good enough.

Comment by Nicola Mometto [ 20/May/15 6:51 PM ]

test.generative uses non-standard :tag, not :arglists

Comment by Alex Miller [ 20/May/15 10:22 PM ]

ah, yes. sorry.

Comment by Stuart Halloway [ 10/Jul/15 12:15 PM ]

Opened https://github.com/Netflix/Hystrix/issues/831 to see what is up with hystrix.

Comment by Stuart Halloway [ 31/Jul/15 9:16 AM ]

Can somebody with context update this patch to apply to latest master?

Comment by Nicola Mometto [ 31/Jul/15 9:37 AM ]

patch updated

Comment by Matt Jacobs [ 06/Aug/15 12:59 PM ]

I've released hystrix-clj 1.4.14, with a fix for https://github.com/Netflix/Hystrix/issues/831: https://github.com/Netflix/Hystrix/releases/tag/v1.4.14. If anything else is needed on the Hystrix side, please open another issue.

Comment by Fogus [ 07/Aug/15 11:03 AM ]

Applied the 0001-CLJ-1232-auto-qualify-arglists-class-names.patch to master (1.8) and ran through the example scenario to check the veracity of the change. Additionally, I ran a modified snippet using def and verified that it too worked. Finally, after checking the code it seems reasonable in implementation and scope.

Comment by Rich Hickey [ 08/Aug/15 9:56 AM ]

I'd rather not change the compiler and it seems hystrix was broken. Also please make it clear what the single strategy and patch is at the top of the ticket.

Comment by Alex Miller [ 11/Aug/15 5:01 PM ]

Made preferred approach clear in the description and moved back to vetted. I believe the other ticket was screened by Fogus so this needs to be re-screened with the preferred approach ticket.

Comment by Alex Miller [ 11/Aug/15 5:08 PM ]

Nicola and Fogus, would appreciate an eye as to whether I just made the proper alterations in the description.

Comment by Nicola Mometto [ 12/Aug/15 6:17 AM ]

Alex Miller Reading Fogus' last comment, it seems to me that he actually screened the current patch (the one who resolves tags in defn rather than in the compiler) "Applied the 0001-CLJ-1232-auto-qualify-arglists-class-names.patch to master [..]"

Comment by Fogus [ 14/Aug/15 9:18 AM ]

It would be better if the description said the word "preferred," but based on my reading resolving the tags in defn is the winner (0001-CLJ-1232-auto-qualify-arglists-class-names.patch). Thankfully that's the one that I screened as I agree that non-trivial changes to the compiler are to be avoided. That said, of course that patch does modify the compiler but since it's a change that pulls out some code into a useful method I let it through. To my mind that was a trivial (and justifiable change). Of course, if we want to avoid any compiler changes then this patch would have to be reworked.

Comment by Alex Miller [ 18/Aug/15 5:03 PM ]

Moving back to screened as fogus screened the preferred (by Rich) version.





[CLJ-1226] set! of a deftype field using field-access syntax causes ClassCastException Created: 26/Jun/13  Updated: 11/Aug/15

Status: Open
Project: Clojure
Component/s: None
Affects Version/s: Release 1.7
Fix Version/s: Release 1.8

Type: Defect Priority: Major
Reporter: Nicola Mometto Assignee: Unassigned
Resolution: Unresolved Votes: 0
Labels: compiler, deftype, interop

Attachments: Text File 0001-CLJ-1226-fix-set-of-instance-field-expression-that-r.patch     Text File 0001-CLJ-1226-fix-set-of-instance-field-expression-that-r-v2.patch    
Patch: Code and Test
Approval: Screened

 Description   

set! can be used to set a public field on an instance with (set! (.field inst) val). This does not work inside a protocol function defined on a deftype with a mutable field for an instance of that type itself.

user=> (defprotocol p (f [_]))
p
user=> (deftype t [^:unsynchronized-mutable x] p (f [this] (set! (.x this) 1)))
user.t
user=>  (f (t. 1))   ;; expect: 1
ClassCastException user.t cannot be cast to compile__stub.user.t  user.t (NO_SOURCE_FILE:1

Cause: The type assigned in the bytecode at this point is the compile_stub type, not the expected class type.

Approach: Use getType(targetClass) instead of Type.getType(targetClass)

Patch: 0001-CLJ-1226-fix-set-of-instance-field-expression-that-r-v2.patch
Screened by: Fogus



 Comments   
Comment by Nicola Mometto [ 27/Jun/13 5:30 AM ]

This patch offers a better workaround for CLJ-1075, making it possible to write
(deftype foo [^:unsynchronized-mutable x] MutableX (set-x [this v] (try (set! (.x this) v)) v))

Comment by Nicola Mometto [ 25/Mar/15 4:39 PM ]

Updated patch to apply to current master

Comment by Fogus [ 07/Aug/15 3:10 PM ]

Straight-forward fix and test.

Comment by Rich Hickey [ 08/Aug/15 10:05 AM ]

Screeners - please make sure the patch has an Approach section that explains how and why the patch will fix the problem. Symptom is X and now after patch behavior is Y is not good enough.





[CLJ-1216] Evaling ((fn [do] do) 1) returns nil while ((fn [do] do do) 1) returns 1 Created: 09/Jun/13  Updated: 03/Sep/13

Status: Open
Project: Clojure
Component/s: None
Affects Version/s: None
Fix Version/s: None

Type: Defect Priority: Minor
Reporter: Nicola Mometto Assignee: Unassigned
Resolution: Unresolved Votes: 0
Labels: compiler

Attachments: Text File 0001-Create-a-DoExpr.Parser-class-that-delegates-to-BodyE.patch    
Patch: Code and Test

 Description   

user=> ((fn [do] do) 1)
nil

user=> ((fn [do] (do do)) 1)
1

user=> ((fn [] do))
nil

user=> ((fn [] do do))
CompilerException java.lang.RuntimeException: Unable to resolve symbol: do in this context, compiling:(NO_SOURCE_PATH:0:0)



 Comments   
Comment by Nicola Mometto [ 09/Jun/13 4:31 PM ]

This patch creates a DoExpr class and makes DoExpr.Parser the DO special form parser.

DoExpr.Parser simply removes the 'do' symbol and delegates to BodyExpr, that was previously done by BodyExpr incorrectly.





[CLJ-1214] Compiler runs out of memory on a small snippet of code Created: 31/May/13  Updated: 18/Apr/14  Resolved: 18/Apr/14

Status: Closed
Project: Clojure
Component/s: None
Affects Version/s: Release 1.5
Fix Version/s: None

Type: Defect Priority: Major
Reporter: Praki Prakash Assignee: Unassigned
Resolution: Declined Votes: 0
Labels: compiler
Environment:

Linux 3.2.0-39-generic


Attachments: File fubar.clj    

 Description   

Clojure compiler runs out of memory when loading the attached file. Transcript below.

$ java -cp ~/.m2/repository/org/clojure/clojure/1.5.1/clojure-1.5.1.jar:. clojure.main
Clojure 1.5.1
user=> (load "fubar")
OutOfMemoryError GC overhead limit exceeded  [trace missing]
user=> 

The file contents are:

  (ns fu.bar)

  (defn foo[l] (concat (drop-last l) (repeat (last l))))

  (def ^:const bar (foo [#(print "") #(println ";")]))

  bar

If I remove the metadata on bar, it works. Removal of namespace seems to fix it as well. Pretty strange.

Although I realize this code is not quite kosher, it would be nice to have the compiler deal with it.



 Comments   
Comment by Andy Fingerhut [ 01/Jun/13 7:40 PM ]

If you really do have 'bar' on a line by itself at the end of file fubar.clj, it seems you are asking it to evaluate the value of bar, which by the definitions is an infinite list, and will of course exhaust all available memory if you attempt to evaluate it.

It seems to me the more odd thing is not that it runs out of memory as shown, but that it does not run out of memory when you remove the metadata on bar.

What is the purpose of having 'bar' on a line by itself at the end of the file?

If I try this but remove 'bar' as the last line of the file, loading the file causes no errors regardless of whether there is metadata on bar's definition or not. It is strange that doing (load "fubar") followed by (first fu.bar/bar) seems to go into an infinite loop if the :const is there on bar, but quickly returns the correct answer if the metadata is removed.

Comment by Praki Prakash [ 01/Jun/13 8:40 PM ]

This code snippet is a minimal test case to show that the compiler runs out of memory. What I meant by "it works" was that the compiler doesn't run out of memory and successfully loads the file (or in my real code base, the namespace is compiled).

In my code, I use bar (or whatever the real thing is) as source of sequence of functions. The sole reference to bar is needed to trigger this issue. I believe that bar is not being fully evaluated here and thus no infinite loop. If I try to print it, yes, it will ultimately fail.

Comment by Praki Prakash [ 01/Jun/13 9:04 PM ]

Having thought about this a bit more, it seems to me that when bar is annotated with const, the compiler is trying to evaluate the associated expression which exhausts the heap? I have never looked at the compiler source and thus not sure if this is what is happening. If it is, then it seems like one should be really careful when adding metadata.

That still leaves the other question about the namespace requirement to cause memory exhaustion. I quite distinctly recall having to add the namespace when trying to come up with minimal test case to reproduce the bug.

If you think this is really user error, I would accept it

Comment by Andy Fingerhut [ 02/Jun/13 4:56 AM ]

It is not just any old metadata that causes this issue, only :const metadata. I tried testing with :const replaced with :dynamic and :private, and there was no problem.

This might shed some light on the issue: