Vetted - Backlog (Clojure JIRA)

[CLJ-47] GC Issue 43: Dead code in generated bytecode

Fri, 8 Oct 2010 10:21:00 -0500

Reported by Levente.Santha, Jan 11, 2009
The bug was described in detail in this thread: http://groups.google.com/
group/clojure/browse_thread/thread/81ba15d7e9130441

For clojure.core$last__2954.invoke the correct bytecode would be (notice 
the removed "goto    65" after "41:  goto    0"):

public java.lang.Object invoke(java.lang.Object)   throws 
java.lang.Exception;
  Code:
   0:   getstatic       #22; //Field const__0:Lclojure/lang/Var;
   3:   invokevirtual   #37; //Method clojure/lang/Var.get:()Ljava/lang/
Object;
   6:   checkcast       #39; //class clojure/lang/IFn
   9:   aload_1
   10:  invokeinterface #41,  2; //InterfaceMethod clojure/lang/IFn.invoke:
(Ljava/lang/Object;)Ljava/lang/Object;
   15:  dup
   16:  ifnull  44
   19:  getstatic       #47; //Field java/lang/Boolean.FALSE:Ljava/lang/
Boolean;
   22:  if_acmpeq       45
   25:  getstatic       #22; //Field const__0:Lclojure/lang/Var;
   28:  invokevirtual   #37; //Method clojure/lang/Var.get:()Ljava/lang/
Object;
   31:  checkcast       #39; //class clojure/lang/IFn
   34:  aload_1
   35:  invokeinterface #41,  2; //InterfaceMethod clojure/lang/IFn.invoke:
(Ljava/lang/Object;)Ljava/lang/Object;
   40:  astore_1
   41:  goto    0
   44:  pop
   45:  getstatic       #26; //Field const__1:Lclojure/lang/Var;
   48:  invokevirtual   #37; //Method clojure/lang/Var.get:()Ljava/lang/
Object;
   51:  checkcast       #39; //class clojure/lang/IFn
   54:  aload_1
   55:  aconst_null
   56:  astore_1
   57:  invokeinterface #41,  2; //InterfaceMethod clojure/lang/IFn.invoke:
(Ljava/lang/Object;)Ljava/lang/Object;
   62:  areturn

Our JIT reported incorrect stack size along the basic block introduced by 
the unneeded goto.
The bug was present in SVN rev 1205.

[CLJ-274] cannot close over mutable fields (in deftype)

Fri, 1 Oct 2010 09:35:00 -0500

Simplest case:

user=>
(deftype Bench [#^{:unsynchronized-mutable true} val]
Runnable
(run [_]
(fn [] (set! val 5))))

java.lang.IllegalArgumentException: Cannot assign to non-mutable: val (NO_SOURCE_FILE:5)

Functions should be able to mutate mutable fields in their surrounding deftype (just like inner classes do in Java).

Filed as bug, because the loop special form expands into a fn form sometimes:

user=>
(deftype Bench [#^{:unsynchronized-mutable true} val]
Runnable
(run [_]
(let [x (loop [] (set! val 5))])))
java.lang.IllegalArgumentException: Cannot assign to non-mutable: val (NO_SOURCE_FILE:9)

[CLJ-124] GC Issue 120: Determine mechanism for controlling automatic shutdown of Agents, with a default policy and mechanism for changing that policy as needed

Fri, 12 Apr 2013 09:52:24 -0500

Reported by cemer...@snowtide.com, Jun 01, 2009

There has been intermittent chatter over the past months from a couple of
people on the group (e.g.
http://groups.google.com/group/clojure/browse_thread/thread/409054e3542adc1f)
and in #clojure about some clojure scripts hanging, either for a constant
time (usually reported as a minute or so with no CPU util) or seemingly
forever (or until someone kills the process).

I just hit a similar situation in our compilation process, which invokes
clojure.lang.Compile from ant.  The build process for this particular
project had taken 15 second or so, but after adding a couple of pmap calls,
that build time jumped to ~1:15, with roughly zero CPU utilization over the
course of that last minute.

Adding a call to Agent.shutdown() in the finally block in
clojure.lang.Compile/main resolved the problem; a patch including this
change is attached.  I wouldn't suspect anyone would have any issues with
such a change.

-----
In general, it doesn't seem like everyone should keep tripping over this
problem in different directions.  It's a very difficult thing to debug if
you're not attuned to how clojure's concurrency primitives work under the
hood, and I would bet that newer users would be particularly affected.

After discussion in #clojure, rhickey suggested adding a
*auto-shutdown-agents* var, which:

- if true when exiting one of the main entry points (clojure.main, or the
legacy script/repl entry points), Agent.shutdown() would be called,
allowing for the clean exit of the application

- would be bound by default to true

- could be easily set to false for anyone with an advanced use-case that
requires agents to remain active after the main thread of the application
exits.

This would obviously not help anyone initializing clojure from a different
entry point, but this may represent the best compromise between
least-surprise and maximal functionality for advanced users.

------

In addition to the above, it perhaps might be worthwhile to change the
keepalive values used to create the Threadpools used by c.l.Actor's
Executors.  Currently, Actor uses a default thread pool executor, which
results in a 60s keepalive.  Lowering this to something much smaller (1s?
5s?) would additionally minimize the impact of Agent's threadpools on Java
applications that embed clojure directly (and would therefore not benefit
from *auto-shutdown-agents* as currently conceived, leading to puzzling
'hanging' behaviour).  I'm not in a position to determine what impact this
would have on performance due to thread churn, but it would at least
minimize what would be perceived as undesirable behaviour by users that are
less familiar with the implementation details of Agent and code that
depends on it.

Comment 1  by cemer...@snowtide.com, Jun 01, 2009

Just FYI, I'd be happy to provide patches for either of the suggestions mentioned
above...

[CLJ-420] Some compiler exceptions erroneously using REPL line numbers.

Fri, 12 Apr 2013 09:42:02 -0500

Certain kinds of errors in loaded source files are coming back tagged with the correct source file, but what seems to be the REPL line number.

http://groups.google.com/group/clojure/browse_frm/thread/beb36e7228eabd69/a7ef16dcc45834bc?hl=en#a7ef16dcc45834bc

jawolfe@[~/Projects/testproj]: cat > src/test.clj

bla
jawolfe@[~/Projects/testproj]: cat > src/test2.clj

(bla)
jawolfe@[~/Projects/testproj]: lein repl
user=> (require 'test)
java.lang.Exception: Unable to resolve symbol: bla in this context
(test.clj:1)
user=> (require 'test)a
java.lang.Exception: Unable to resolve symbol: bla in this context
(test.clj:2)
user=> (require 'test)
java.lang.Exception: Unable to resolve symbol: bla in this context
(test.clj:3)
user=> (require 'test2)
java.lang.Exception: Unable to resolve symbol: bla in this context
(test2.clj:2)
user=> (require 'test2)
java.lang.Exception: Unable to resolve symbol: bla in this context
(test2.clj:2)
user=> (require 'test2)
java.lang.Exception: Unable to resolve symbol: bla in this context
(test2.clj:2)
user=> (require 'test)
java.lang.Exception: Unable to resolve symbol: bla in this context
(test.clj:7)
user=>

[CLJ-701] Compiler loses 'loop's return type in some cases

Fri, 12 Apr 2013 09:42:00 -0500

(set! *warn-on-reflection* true)
(fn [] (loop [b 0] (recur (loop [a 1] a))))

Generates the following warnings:

recur arg for primitive local: b is not matching primitive, had: Object, needed: long
Auto-boxing loop arg: b

This is interesting for several reasons. For one, if the arg to recur is a let form, there is no warning:

(fn [] (loop [b 0] (recur (let [a 1] a))))

Also, the compiler appears to understand the return type of loop forms just fine:

(use '[clojure.contrib.repl-utils :only [expression-info]])
(expression-info '(loop [a 1] a))
;=> {:class long, :primitive? true}

The problem can of course be worked around using an explicit cast on the loop form:

(fn [] (loop [b 0] (recur (long (loop [a 1] a)))))

Reported by leafw in IRC: http://clojure-log.n01se.net/date/2011-01-03.html#10:31

[CLJ-1058] StackOverflowError on exception in reducef for PersistentHashMap fold

Fri, 12 Apr 2013 09:42:02 -0500

If reducef throws an exception, the PHM fold code can descend into an infinite loop, causing a stack overflow which masks the problem. This situation is commented "aargh" in PersistentHashMap.java line 444 (as of 412a51d).

To reproduce:

user> (require '[clojure.core.reducers :as r])
nil
user> (r/fold (fn ([]) ([ret k v] (+ 3 "foo") ret)) (into {} (map (juxt identity identity) (range 10000))))
;; boom

This results in a stack like: https://raw.github.com/gist/3bab917287a7fd635a84/f38bfe3e270556e467f3fc02062af7ea10781390/gistfile1.txt

[CLJ-1100] Reader literals cannot contain periods

Thu, 14 Feb 2013 15:04:53 -0600

The LispReader tries to read a record instead of a literal if the tag contains periods:

https://github.com/clojure/clojure/blob/master/src/jvm/clojure/lang/LispReader.java#L1171

Which effectively means that reader tags cannot contain periods.
The EDN spec is unclear on this:

edn supports extensibility through a simple mechanism. # followed immediately by a symbol starting with an alphabetic character indicates that that symbol is a tag.

(issue opened: https://github.com/edn-format/edn/issues/39)

If periods are allowed, then the LispReader should first check to see if the tag is in *data-readers* and only then if not try to initialize a Java class.

I'm happy to write the patch if this behavior is what is desired.

[CLJ-348] reify allows use of qualified name as method parameter

Fri, 1 Mar 2013 12:33:02 -0600

This should complain about using a fully-qualified name as a parameter:

(defmacro lookup []
`(reify clojure.lang.ILookup
(valAt [_ key])))

Instead it simply ignores that parameter in the method body in favour of clojure.core/key.

[CLJ-731] Create macro to variadically unroll a combinator function definition

Fri, 28 Jan 2011 09:40:05 -0600

Clojure contains a set of combinators that are implemented in a similar, but slightly different way. That is, they are implemented as a complete set of variadic overloads on both the call-side and also on the functions that they return. Visually, they all tend to look something like:

(defn foo
  ([f]
     (fn
       ([] do stuff...)
       ([x] do stuff...)
       ([x y] do stuff...)
       ([x y z] do stuff...)
       ([x y z & args] do variadic stuff...)))
  ([f1 f2]
     (fn
       ([] do stuff...)
       ([x] do stuff...)
       ([x y] do stuff...)
       ([x y z] do stuff...)
       ([x y z & args] do variadic stuff...)))
  ([f1 f2 f3]
     (fn
       ([] do stuff...)
       ([x] do stuff...)
       ([x y] do stuff...)
       ([x y z] do stuff...)
       ([x y z & args] do variadic stuff...)))
  ([f1 f2 f3 & fs]
     (fn
       ([] do stuff...)
       ([x] do stuff...)
       ([x y] do stuff...)
       ([x y z] do stuff...)
       ([x y z & args] do variadic stuff...))))

To build this type of function for each combinator is tedious and error-prone.

There must be a way to implement a macro that takes a "specification" of a combinator including:

1. name
2. docstring
3. do stuff template
4. do variadic stuff template

And builds something like the function foo above. This macro should be able to implement the current batch of combinators (assuming that http://dev.clojure.org/jira/browse/CLJ-730 is completed first for the sake of verification).

[CLJ-787] transient blows up when passed a vector created by subvec

Fri, 12 Apr 2013 09:42:00 -0500

Subvectors created with subvec from a PersistentVector cannot be made transient:

user=> (transient (subvec [1 2 3 4] 2))
ClassCastException clojure.lang.APersistentVector$SubVector cannot be cast to clojure.lang.IEditableCollection  clojure.core/transient (core.clj:2864)

[CLJ-823] Piping seque into seque can deadlock

Fri, 12 Apr 2013 09:23:49 -0500

I'm not sure if this is a supported scenario, but the following deadlocks in Clojure 1.3:

(let [xs (seque (range 150000))
ys (seque (filter odd? xs))]
(apply + ys))

As I understand it, the problem is that ys' fill takes place on an agent thread, so when it calls xs' drain, the (send-off agt fill) does not immediately trigger xs' fill, but is instead put on the nested list to be performed when ys' agent returns. Unfortunately, ys' fill will eventually block trying to take from xs, and so it never returns and the pending send-offs are never sent. Wrapping the send-off in drain to:

(future (send-off agt fill))

is a simple (probably not optimal) way to fix the deadlock.

[CLJ-1033] pr-str and read-string don't handle @ symbols inside keywords properly

Sun, 13 Jan 2013 22:58:37 -0600

user=> (read-string (pr-str {(keyword "key@other") :stuff}))
RuntimeException Map literal must contain an even number of forms  clojure.lang.Util.runtimeException (Util.java:170)

pr-str emits "{:key@other :stuff}", which read-string fails to interpret correctly. Either pr-str needs to escape the @ symbol, or read-string needs to handle the symbol inside a keyword.

Background: I'm passing a map with email addresses as keys through Storm bolts, which require a thrift-serializable form. Using the pr-str/read-string combo fails on these keys, so I've fallen back to JSON serialization.

[CLJ-1039] Using 'def with metadata {:type :anything} throws ClassCastException

Fri, 10 Aug 2012 13:40:31 -0500

In lein 1.7.1 (and CCW) the form (def ^{:type :anything} mydef 1) throws "ClassCastException clojure.lang.Var cannot be cast to clojure.lang.IObj clojure.core/with-meta (core.clj:211)".
This seems to be due to setting the :type metadata. With other metadata keys it works well.

If it is intended to forbid setting the :type metadata, then there should be an appropriate error message instead of the ClassCastException

In lein2 repl which is using "REPL-y 0.1.0-beta8" the exception does not occur.

Full stacktrace:
java.lang.ClassCastException: clojure.lang.Var cannot be cast to clojure.lang.IObj
at clojure.core$with_meta.invoke (core.clj:211)
clojure.core$vary_meta.doInvoke (core.clj:617)
clojure.lang.RestFn.invoke (RestFn.java:425)
clojure.core/fn (core_print.clj:76)
clojure.lang.MultiFn.invoke (MultiFn.java:167)
clojure.core$pr_on.invoke (core.clj:3266)
clojure.core$pr.invoke (core.clj:3278)
clojure.lang.AFn.applyToHelper (AFn.java:161)
clojure.lang.RestFn.applyTo (RestFn.java:132)
clojure.core$apply.invoke (core.clj:601)
clojure.core$prn.doInvoke (core.clj:3311)
clojure.lang.RestFn.invoke (RestFn.java:408)
clojure.main$repl$read_eval_print__6405.invoke (main.clj:246)
clojure.main$repl$fn__6410.invoke (main.clj:266)
clojure.main$repl.doInvoke (main.clj:266)
clojure.lang.RestFn.invoke (RestFn.java:512)
user$eval27$acc_3261auto__30$fn_32.invoke (NO_SOURCE_FILE:1)
clojure.lang.AFn.run (AFn.java:24)
java.lang.Thread.run (Thread.java:662)

[CLJ-1056] defprotocol: invalid method overload syntax getting accepted

Thu, 29 Nov 2012 16:03:11 -0600

The compiler accepts both of these erroneous forms which while silly, are not imposible to come up with.

(defprotocol Foo (f ([this]) ([this arg])))
(defprotocol Bar (m [this]) (m [this arg]))

[CLJ-1105] defrecord classes implement IPersistentCollection but not .empty, clojure.walk assumes collections support empty

Fri, 12 Apr 2013 09:42:00 -0500

Using clojure.walk functions fails surprisingly for data containing records defined with defrecord:

user=> (defrecord Foo [x])
user.Foo
user=> (def f (Foo. :x))
#'user/f
user=> (use 'clojure.walk)
nil
user=> (postwalk identity {:foo f})
UnsupportedOperationException Can't create empty: user.Foo user.Foo (NO_SOURCE_FILE:1)

This seems to be because clojure.walk/walk guards a call to (empty form) with a (coll? form) check. The check succeeds because records implement IPersistentCollection, but (empty form) throws an exception. This looks to me like a bug in clojure.walk (it should check records separately and either treat them as atomic or implement a way of walking through them) but perhaps it is a sign of some unclarity in the contract of collections.

[CLJ-1184] Evaling #{do ...} or [do ...] is treated as the do special form

Fri, 12 Apr 2013 09:42:02 -0500

Evaluating a persistent collection for which the function 'first' returns the symbol 'do' leads to that collection being treated as the do special form, even though it may be a vector or even a set. IMHO, the expected result would be to report that 'do' cannot be resolved. For the cause, see the if condition on line 6604 of Compiler.java in clojure-1.5.1.

E.g.:
[do 1 2]
;=> 2

#{"hello" "goodbye" do}
;=> "hello"
; Wat?