<< Back to previous view

[CLJ-1531] inc always warns when *unchecked-math* is set Created: 23/Sep/14  Updated: 23/Sep/14  Resolved: 23/Sep/14

Status: Closed
Project: Clojure
Component/s: None
Affects Version/s: Release 1.7
Fix Version/s: None

Type: Defect Priority: Major
Reporter: Pierre-Yves Ritschard Assignee: Pierre-Yves Ritschard
Resolution: Not Reproducible Votes: 0
Labels: bug, errormsgs


 Description   

While testing 1.7-alpha2 I stumbled this (affects clojure.data.codec amongst others). inc inlines a call to clojure.lang.Numbers's inc method which according to the rules of CLJ-1325 will warn.

I can't find a way around it for now, except maybe having a primitive-inc and primitive-dec java method which would be inlined in that case.

Happy to work on a patch but would prefer discussing it first.



 Comments   
Comment by Nicola Mometto [ 23/Sep/14 12:42 PM ]

I cannot reproduce this:

Clojure 1.7.0-master-SNAPSHOT
user=> (set! *unchecked-math* true)
true
user=> (inc 1)
2

Looking at Numbers.java I see both unchecked_inc and inc have long/double taking methods.

Comment by Pierre-Yves Ritschard [ 23/Sep/14 1:49 PM ]

you're right, i must have been confused.

Comment by Pierre-Yves Ritschard [ 23/Sep/14 1:50 PM ]

not a bug





[CLJ-1525] bean function returns mutable maps Created: 16/Sep/14  Updated: 22/Sep/14  Resolved: 22/Sep/14

Status: Closed
Project: Clojure
Component/s: None
Affects Version/s: Release 1.6
Fix Version/s: None

Type: Defect Priority: Major
Reporter: Simone Mosciatti Assignee: Unassigned
Resolution: Declined Votes: 0
Labels: None
Environment:

Linux



 Description   

Please take a look at this snippet.

user> (import 'java.util.Date)
java.util.Date
user> (def now (Date.))
#'user/now
user> now
#inst "2014-09-17T03:14:13.821-00:00"
user> (def bean-map (bean now))
#'user/bean-map
user> bean-map
{:day 3, :date 17, :time 1410923653821, :month 8, :seconds 13, :year 114, :class java.util.Date, :timezoneOffset -120, :hours 5, :minutes 14}
user> (.setMonth now 1)
nil
user> bean-map
{:day 1, :date 17, :time 1392610453821, :month 1, :seconds 13, :year 114, :class java.util.Date, :timezoneOffset -60, :hours 5, :minutes 14}

The same snippet here. https://gist.github.com/siscia/032bff669bbc6fb0fe57



 Comments   
Comment by Jozef Wagner [ 17/Sep/14 1:32 AM ]

It works as expected. bean fn returns a clojuresque abstraction on top of live bean. map-like abstraction returned from bean is intended to be 'mutable' in sense that it always return the latest value. Otherwise it is read only.

Comment by Simone Mosciatti [ 17/Sep/14 1:42 AM ]

Hi,

sorry, the documentation didn't mention the "mutable" part so I was expecting an immutable map as always.

Sorry, about that.

Greets

Comment by Alex Miller [ 22/Sep/14 9:38 AM ]

this is the expected behavior





[CLJ-1524] SeqIterator constructor change broke binary compatibility in 1.7.0-alpha2 Created: 09/Sep/14  Updated: 09/Sep/14  Resolved: 09/Sep/14

Status: Closed
Project: Clojure
Component/s: None
Affects Version/s: Release 1.7
Fix Version/s: Release 1.7

Type: Defect Priority: Critical
Reporter: Alex Miller Assignee: Unassigned
Resolution: Completed Votes: 0
Labels: None

Attachments: File clj-1524.diff    
Patch: Code
Approval: Ok

 Description   

Running code AOT-compiled against Clojure 1.6.0 (or older) with 1.7.0-alpha2 runtime will encounter this error with SeqIterator:

CompilerException java.lang.NoSuchMethodError: clojure.lang.SeqIterator.<init>(Lclojure/lang/ISeq;)V, compiling:(form-init5913779045640355531.clj:1:11)

Cause: This is due to a type change in the constructor of SeqIterator from ISeq to Object (commit: https://github.com/clojure/clojure/commit/43cc1854508d655e58e377f84836ba128971f90c ).

Proposed: Add the ISeq constructor back so that calls into that constructor retain backwards binary compatibility.

Patch: clj-1524.diff

Screened by:

More: From Datomic mailing list - https://groups.google.com/forum/#!topic/datomic/KZqhY6hUHz0



 Comments   
Comment by Alex Miller [ 09/Sep/14 11:06 AM ]

Patch not applied, but similar change applied directly here:

https://github.com/clojure/clojure/commit/ba41f25b6f3f32729c55f7f7ceb179be597acf94





[CLJ-1518] Patch for removing transient thread owner check broke rrb-vector Created: 03/Sep/14  Updated: 04/Sep/14  Resolved: 04/Sep/14

Status: Closed
Project: Clojure
Component/s: None
Affects Version/s: Release 1.7
Fix Version/s: Release 1.7

Type: Enhancement Priority: Minor
Reporter: Alex Miller Assignee: Alex Miller
Resolution: Completed Votes: 0
Labels: None

Attachments: File clj-1518.diff    
Patch: Code
Approval: Ok

 Description   

The patch for CLJ-1498 changed the field types inside the persistent data structures, which inadvertently broke core.rrb-vector, which relies on reusing some of those internals. It is not necessary to change the type to satisfy the patch, so we would like to rollback that aspect of the change to minimize breakage.



 Comments   
Comment by Alex Miller [ 03/Sep/14 2:05 PM ]

In the patch I rolled back the changes in the Persistent*.java from CLJ-1498 and re-applied. The only "real" changes after the rollback are in ensureEditable(). Tests were left of course.





[CLJ-1512] Create volatile box for managing state Created: 25/Aug/14  Updated: 03/Sep/14  Resolved: 03/Sep/14

Status: Resolved
Project: Clojure
Component/s: None
Affects Version/s: Release 1.7
Fix Version/s: Release 1.7

Type: Enhancement Priority: Major
Reporter: Rich Hickey Assignee: Rich Hickey
Resolution: Completed Votes: 0
Labels: transducers

Attachments: File volatile2.diff     File volatile3.diff    
Patch: Code and Test
Approval: Ok

 Description   

Motivation:

Clojure needs a faster variant of Atom for managing state inside transducers. That is, Atoms do the job, but they provide a little too much capability for the purposes of transducers. Specifically the compare and swap semantics of Atoms add too much overhead. Therefore, it was determined that a simple volatile ref type would work to ensure basic propagation of its value to other threads and reads of the latest write from any other thread. While updates are subject to race conditions, access is controlled by JVM guarantees.

Solution overview: Create a concrete type in Java, akin to clojure.lang.Box, but volatile inside supports IDeref, but not watches etc.

API:

(volatile! x) ;;ctor
(vreset! vol newval) ;;like reset
(vswap! vol f args) ;;same shape as swap!, but MACRO over vreset!

Patch: volatile3.diff

Screened by: fogus



 Comments   
Comment by Alex Miller [ 25/Aug/14 9:11 AM ]

Dumb benchmark before/after...

java -cp target/classes -Xmx512m -server clojure.main
(def t (take 1000000))
(def v (doall (range 1000000)))
(defn bench [t v]
  (time (into [] t v)))
(dotimes [_ 30] (bench t v))

before - 29-32 ms after warmup
after - 22-23 ms after warmup

Comment by Alex Miller [ 25/Aug/14 9:12 AM ]

From Stu H elsewhere:

Three questions:
1) Should we keep volatile? in the public API?
2) Should we work in terms of IVolatile interface (guessing no)
3) Do we need a CLJS version of these APIs?

Comment by Alex Miller [ 25/Aug/14 9:13 AM ]

1. We have many tickets requesting predicates over types that are "internal" and generally I find these to be helpful. They also can help in making core more portable to cljs (maybe those fns would fall back to atoms in cljs?).
2. We have tickets requesting the equivalent of this for IAtom (CLJ-803) etc. I don't think an interface adds any value to us here though. There seems to be some requests for this kind of passthrough interface from tooling as a decoupling point. Not putting my finger on those discussions but I know I've heard this, maybe on the mailing list.
3. I think yes if that allows us to be more efficient than whatever is being done now. Not obvious to me.

Comment by Nicola Mometto [ 25/Aug/14 9:40 AM ]

Why is vswap! a macro?

Comment by Ambrose Bonnaire-Sergeant [ 26/Aug/14 8:04 AM ]

An IAtom conversation: https://groups.google.com/forum/#!searchin/clojure-dev/iatom/clojure-dev/y5QoMqd44Lc/y4YmW09blk0J

Comment by Max Penet [ 26/Aug/14 10:28 AM ]

the vswap! macro is probably for performance reasons (the main motivation of this code to begin with), to avoid using apply or unrolling tons of arities

Comment by Nicola Mometto [ 26/Aug/14 1:07 PM ]

If that is the only reason, why can't it be a regular fn + :inline metadata?

Comment by Jozef Wagner [ 27/Aug/14 3:50 AM ]

why the bang in the name of volatile! function? If the reason is to warn users that this is an 'expert only' stuff, I suggest to use a verbose name instead, e.g. volatile-reference. (This will also be consistent with approach chosen in the names of volatile-mutable and unsynchronized-mutable hints.)

Comment by Rich Hickey [ 27/Aug/14 6:37 AM ]

Can you please lift the with-meta stuff out of the syntax-quote?
Actually, if volatile! ctor returned a type-hinted value that extra hinting might not even be needed. Let's do both for now.

Also the type hint on the volatile? arg makes no sense - it's a predicate asking if something is a volatile.

Comment by Alex Miller [ 28/Aug/14 9:05 AM ]

Made changes as requested.

Comment by Fogus [ 29/Aug/14 11:01 AM ]

I downloaded the patch and applied to latest master. I ran the isolated tests and the full test suite and also ensured that the patch didn't add any reflection warnings. I then modified the ticket description to add a little more context and motivation (for future readers). The code is straight-forward and clean.

Comment by Alex Miller [ 29/Aug/14 4:31 PM ]

Updated to volatile3.diff to address offline comment from Rich.





[CLJ-1511] stack overflow when comparing sequence results Created: 24/Aug/14  Updated: 27/Aug/14  Resolved: 27/Aug/14

Status: Resolved
Project: Clojure
Component/s: None
Affects Version/s: Release 1.7
Fix Version/s: Release 1.7

Type: Defect Priority: Critical
Reporter: Chhi'mèd Künzang Assignee: Unassigned
Resolution: Completed Votes: 0
Labels: transducers
Environment:

OS X 10.9.4


Attachments: Text File 0001-provide-working-implementations-for-LazyTransform-eq.patch    
Patch: Code and Test
Approval: Ok

 Description   

Comparing sequences created with sequence causes a stack overflow when used as first argument to =.

Consider this transducer:

user=> (def map-inc (map inc))
#'user/map-inc

When creating a sequence and comparing with expected results, it works fine as the second argument to the comparison:

user=> (= (range 1 11) (sequence map-inc (range 10)))
true

But a stack overflow occurs when the order of arguments is reversed:

user=> (= (sequence map-inc (range 10)) (range 1 11))

StackOverflowError   clojure.lang.LazyTransformer.equals (LazyTransformer.java:202)
user=> (clojure.stacktrace/print-stack-trace *e 10)
java.lang.StackOverflowError: null
 at clojure.lang.LazyTransformer.equiv (LazyTransformer.java:185)
    clojure.lang.LazyTransformer.equals (LazyTransformer.java:202)
    clojure.lang.LazyTransformer.equiv (LazyTransformer.java:185)
    clojure.lang.LazyTransformer.equals (LazyTransformer.java:202)
    clojure.lang.LazyTransformer.equiv (LazyTransformer.java:185)
    clojure.lang.LazyTransformer.equals (LazyTransformer.java:202)
    clojure.lang.LazyTransformer.equiv (LazyTransformer.java:185)
    clojure.lang.LazyTransformer.equals (LazyTransformer.java:202)
    clojure.lang.LazyTransformer.equiv (LazyTransformer.java:185)
    clojure.lang.LazyTransformer.equals (LazyTransformer.java:202)
nil

The error persists, even if the sequence is forced with doall:

user=> (= (doall (sequence map-inc (range 10))) (doall (range 1 11)))

StackOverflowError   clojure.lang.LazyTransformer.equiv (LazyTransformer.java:185)

It does work as expected, however, if the sequence is converted to a vector:

user=> (= (vec (sequence map-inc (range 10))) (range 1 11))
true


 Comments   
Comment by Nicola Mometto [ 25/Aug/14 4:31 AM ]

Patch provides equiv/equals implementations for LazyTransform based on ASeq equiv/equals





[CLJ-1510] line-seq and read-line don't return nil on Ctrl-D in lein repl Created: 21/Aug/14  Updated: 25/Aug/14  Resolved: 22/Aug/14

Status: Closed
Project: Clojure
Component/s: None
Affects Version/s: Release 1.6
Fix Version/s: None

Type: Defect Priority: Major
Reporter: Geoff Little Assignee: Unassigned
Resolution: Not Reproducible Votes: 0
Labels: reader
Environment:

OSX, lein repl



 Description   

Executing in lein repl either

(doseq [line (line-seq (java.io.BufferedReader. *in*)) :while line]
(println line))

Or

(doseq [line (read-line) :while line]
(println line))

One would expect these to return on a user's enter of Ctrl-D, however they never return.



 Comments   
Comment by Andy Fingerhut [ 21/Aug/14 11:51 PM ]

If instead of 'lein repl' you run:

java -cp clojure.jar clojure.main

at a shell prompt, you get a REPL where the first doseq expression you give does return when you enter Ctrl-D.

The second doseq expression returns a string from (read-line), and then the doseq iterates over that as a sequence of characters, and it returns without needing Ctrl-D after reading a single line in both 'lein repl' and the plain Clojure REPL with the command given above.

I suggest that the behavior with your first doseq expression is an issue to be taken up with the Leiningen developers. Several ways of contacting them are given at http://leiningen.org/#community I would recommend IRC or the email list first, before creating a Github issue.

Comment by Alex Miller [ 22/Aug/14 8:29 AM ]

Agreed with Andy - these examples work fine in base repl but other environments may be affecting your input upstream of clojure.

Comment by Andy Fingerhut [ 25/Aug/14 11:42 AM ]

Geoff, it looks like there have been recent changes committed to the tools.nrepl library related to this issue. tools.nrepl is used within Leiningen to implements its REPL.

https://github.com/clojure/tools.nrepl/commit/eb526fd8498ced1b4bd1555f8ff680f3ad65f1b4





[CLJ-1505] Sorry I have to enter this test bug Created: 15/Aug/14  Updated: 15/Aug/14  Resolved: 15/Aug/14

Status: Closed
Project: Clojure
Component/s: None
Affects Version/s: None
Fix Version/s: None

Type: Defect Priority: Major
Reporter: Karen Assignee: Unassigned
Resolution: Declined Votes: 0
Labels: None


 Description   

Interacting with this site caused FireFox to freak out when populating a field in our app so I need to enter a bug here to figure out why. See http://dev.clojure.org/jira/browse/CLJ-1378 We get the same error message in one of our summary field. It is a FireFox bug so I'll update this when I am done. Apologies.






[CLJ-1500] Cache namespace env Created: 08/Aug/14  Updated: 08/Aug/14  Resolved: 08/Aug/14

Status: Closed
Project: Clojure
Component/s: None
Affects Version/s: None
Fix Version/s: None

Type: Enhancement Priority: Minor
Reporter: Nicola Mometto Assignee: Unassigned
Resolution: Declined Votes: 0
Labels: None


 Description   

See: https://github.com/clojure/tools.analyzer.js/blob/master/src/main/clojure/clojure/tools/analyzer/js.clj#L524-L548






[CLJ-1498] Remove birth-thread check from transients Created: 08/Aug/14  Updated: 29/Aug/14  Resolved: 29/Aug/14

Status: Closed
Project: Clojure
Component/s: None
Affects Version/s: Release 1.6
Fix Version/s: Release 1.7

Type: Enhancement Priority: Major
Reporter: Rich Hickey Assignee: Alex Miller
Resolution: Completed Votes: 0
Labels: collections, transient

Attachments: File clj-1498-2.diff     File clj-1498-3.diff     File clj-1498.diff    
Patch: Code
Approval: Ok

 Description   

Transients protect themselves from use by any thread other than the one that creates them. This is good for safety, however it eliminates certain valid usages of transients. For example, usage in a go-block might occur in subsequent invocations across multiple OS threads (but only one logical thread of control).

Current simple test:

user> (def v (transient []))
#'user/v
user> (persistent! @(future (conj! v 1)))
IllegalAccessError Transient used by non-owner thread  clojure.lang.PersistentVector$TransientVector.ensureEditable (PersistentVector.java:464)

Proposal: Remove the owner check from transient collections. (Leave the edit after persistent check as is.) The test above should succeed.

After:

user=> (def v (transient []))
#'user/v
user=> (persistent! @(future (conj! v 1)))
[1]

The clj-1498-3.diff version of the patch also replaces the AtomicReference<Thread> with AtomicBoolean as we can now track just ownership, not who owns it.

Doc update: Various pieces of documentation will need to be updated with this change, namely http://clojure.org/transients

Patch: clj-1498-3.diff

Alternative: Another idea would be to make this check optional with some kind of option on the transient call (transient coll :check-owner true). Not sure whether what the default would be for that.



 Comments   
Comment by Jozef Wagner [ 09/Aug/14 7:08 AM ]

I suggest to add a functionality to pass ownership of a transient to the different thread, or to release the ownership by passing nil.

user=> (def v (pass! (transient []) nil))
#'user/v
user=> (persistent! @(future (conj! v 1)))
[1]

pass! has to be called by current owner thread, or by any thread if the transient is currently released.

Comment by Alex Miller [ 13/Aug/14 1:42 PM ]

New patch that replaces AtomicReference<Thread> with AtomicBoolean.

Comment by Stuart Halloway [ 19/Aug/14 11:05 AM ]

Alex, can you please expand the example test you provided to a generative test that covers the following combinations:

  1. different collection sizes (above and below the ArrayMap size boundary)
  2. different shapes (vector vs. map)
  3. successful use across threads (positive use case this ticket enables)

data_structures.clj has helpers for generating transient interactions that you can build on.

Comment by Alex Miller [ 20/Aug/14 8:59 AM ]

Enhanced existing generative tests to test random actions against sets, vectors, and both PHM and PAM. Added additional actions to do transient modification actions in other threads as well as originating thread.





[CLJ-1497] sequence with transducers realizes n+2 elements Created: 08/Aug/14  Updated: 08/Aug/14  Resolved: 08/Aug/14

Status: Closed
Project: Clojure
Component/s: None
Affects Version/s: Release 1.7
Fix Version/s: Release 1.7

Type: Defect Priority: Major
Reporter: Nicola Mometto Assignee: Unassigned
Resolution: Completed Votes: 0
Labels: None

Attachments: File clj-1497.diff     File clj-1497v2.diff    
Approval: Ok

 Description   

The first element is realized at creation time:

user=> (def a (sequence (take 2) (map #(do (println (str "~" %)) %) (iterate inc 1))))
~1
#'user/a

Fully realizing the sequence realizes the other n-1 elements, and 2 more:

user=> a
(~2
~3
1 ~4
2)

Compare with version using seq operations:

user=> (def a (sequence (take 2 (map #(do (println (str "~" %)) %) (iterate inc 1)))))
#'user/a
user=> a
(~1
~2
1 2)

Transduce also doesn't seem to exhibit this issue:

user=> (def a (transduce (take 2) conj [] (map #(do (println (str "~" %)) %) (iterate inc 1))))
~1
~2
#'user/a
user=> a
[1 2]


 Comments   
Comment by Alex Miller [ 08/Aug/14 10:02 AM ]

Patch attached that improves the issue - will now only realize n+1 elements.

Comment by Nicola Mometto [ 08/Aug/14 10:16 AM ]

Nice, I added a commit on top of yours to delay the realization of the first element of the lazyseq to the first .next call instead of on SeqIteration creation

Comment by Alex Miller [ 08/Aug/14 11:12 AM ]

Fixed by Rich directly, not by patch.





[CLJ-1494] remove flatmap in favor of mapcat Created: 07/Aug/14  Updated: 03/Sep/14  Resolved: 03/Sep/14

Status: Closed
Project: Clojure
Component/s: None
Affects Version/s: Release 1.7
Fix Version/s: Release 1.7

Type: Enhancement Priority: Major
Reporter: Nicola Mometto Assignee: Unassigned
Resolution: Completed Votes: 1
Labels: None

Attachments: Text File 0001-remove-flatmap-use-1-arity-mapcat-instead.patch    
Patch: Code

 Description   

While all the transducers functions are implemented as an arity in the matching clojure core sequence, for mapcat a new function has been added: flatmap.
The reason for this is, as Rich said in a HN comment, "because mapcat's signature was not amenable to the additional arity".
This patch changes the mapcat signature to take at least one collection so that it's possible to add the 1-arity for the transducer function, eliminating the need for a different function, flatmap.

There has been no loss by removing the 1-arity version of mapcat as a sequence function since trying to use (mapcat f) as currently defined (not as redefined with this patch) would fail before transducers, and after transducers:
Before transducers (mapcat f) would result in a call to (map f) which would fail with an ArityException
After transducers that (map f) call would return a function, which then would be used as an argument to (apply concat the-f), resulting in a IllegalArgumentException since apply expects a sequence but it's been given a fn.



 Comments   
Comment by Alex Miller [ 03/Sep/14 11:02 AM ]

Done as a result of https://github.com/clojure/clojure/commit/7d84a9f6f35a503cddf98487b6544d18937c669e





[CLJ-1487] Variadic unrolling for partial Created: 01/Aug/14  Updated: 01/Aug/14  Resolved: 01/Aug/14

Status: Closed
Project: Clojure
Component/s: None
Affects Version/s: None
Fix Version/s: None

Type: Enhancement Priority: Minor
Reporter: Alan Malloy Assignee: Unassigned
Resolution: Duplicate Votes: 0
Labels: None

Attachments: Text File unroll-partial.patch    
Patch: Code and Test

 Description   

Many of the functions in clojure.core are variadic-unrolled for performance. The implementation of juxt, for example, is 29 lines where it could be just 3 if we didn't care about performance. For the function "partial", this unrolling is, if you will excuse the pun, only partially done. This patch extends the unrolling for partial to mirror that for juxt and comp, and includes a test that partial works for any reasonable number of arguments (the existing test just spot-checks a few arities).

I've done some performance benchmarking, and we get about a 20% speedup on calling partial functions, with no performance penalty for creating them.

You may note that the n-ary case, ([arg1 arg2 arg3 & more] ...), is not unrolled in my patch. Doing so would have worsened performance: because we're having to call apply anyway, there's no real benefit to be had from unrolling, and it costs a little to rebuild the arglist before passing it to apply.



 Comments   
Comment by Ghadi Shayban [ 01/Aug/14 6:45 PM ]

Dupe of CLJ-1430

Comment by Alex Miller [ 01/Aug/14 11:10 PM ]

Dupe as noted in the comments





[CLJ-1476] map-invert should use (empty m) instead of {} Created: 26/Jul/14  Updated: 27/Jul/14  Resolved: 27/Jul/14

Status: Closed
Project: Clojure
Component/s: None
Affects Version/s: Release 1.6
Fix Version/s: None

Type: Enhancement Priority: Minor
Reporter: Gregory Schlomoff Assignee: Unassigned
Resolution: Declined Votes: 0
Labels: None


 Description   

clojure.set/map-invert should reduce with (empty m) instead of {} so that it returns a map of the same type as its argument.

This is a trivial change and I'm willing to submit a patch if nobody opposes.



 Comments   
Comment by Alex Miller [ 26/Jul/14 8:43 AM ]

I don't think that always makes sense. Say you had a map of string to integers with a custom comparator created by sorted-map-by. If you use empty, you'd still have a map with a custom comparator which you would pour integer keys into and would likely throw a ClassCastException.

What is the use case that led you to this ticket?

Comment by Gregory Schlomoff [ 26/Jul/14 9:14 AM ]

Hello Alex, thanks for commenting.

My use case is that I have a custom type that implements IPersistentMap. If I use map-invert over it, I get a regular map back, which is problematic because regular maps don't allow multiple values for the same key, unlike my multimap implementation, so I loose information.

(map-invert (my-multimap :a 1, :b 1))
=> {1 :b} ; lost the (1 :a) entry because regular maps don't allow duplicate keys

Maybe a solution would be to make a version of map-invert that takes a map to insert the inverted entries into?

I'm not adamant over this, if you think there is no elegant solution for this issue we can close it.

Comment by Alex Miller [ 27/Jul/14 7:28 AM ]

I don't think this enhancement makes sense as written - there are cases where it would be a breaking change for existing code.

I do think your specified problem makes sense though. One enhancement might be to have a variant of map-invert (different arity or map-invert-into that took an additional map target param).





[CLJ-1474] `reduced` docstring should be more explicit Created: 25/Jul/14  Updated: 27/Jul/14  Resolved: 25/Jul/14

Status: Closed
Project: Clojure
Component/s: None
Affects Version/s: Release 1.6
Fix Version/s: None

Type: Enhancement Priority: Major
Reporter: Jean Niklas L'orange Assignee: Unassigned
Resolution: Declined Votes: 0
Labels: docstring


 Description   

The documentation for reduced is as follows:

Wraps x in a way such that a reduce will terminate with the value x

From what I gather, this does not specify whether the init value of a reduce could be a reduced value or not. As shown, the fact that the init value is a reduced value is ignored:

(reduce list (reduced 1) [2])
=> (#<Reduced@518a6aa: 1> 2)

The documentation should explicitly mention that a reduce call will not check if the initial value is reduced.



 Comments   
Comment by Alex Miller [ 25/Jul/14 9:09 AM ]

reduced creates a value that has special meaning as the output of invocation of the reducing function. Your example is about an input to that function. I don't see that this makes sense or needs documenting.

You can of course invent a situation where a (reduced 1) input is also the output but again, that seems like a pretty weird use case.

(reduce (fn [a v] a) (reduced 1) [2])
;; 1
Comment by Jean Niklas L'orange [ 25/Jul/14 12:10 PM ]

Right, that's my point. Nowhere in the documentation does it state that this does not apply to the initial value given to reduce. While you and I know this, I don't see how one can conclude this based on the current documentation.

Put differently, someone might wrongly assume that reduce is implemented as an optimised version of this:

(defn reduce [f init coll]
  (cond (reduced? init) (unreduced init)
        (empty? coll)    init
        :else           (recur f (f init (first coll))
                                 (rest coll))))

However, that's not the case, which I think is worth pointing out.

Comment by Alex Miller [ 25/Jul/14 5:01 PM ]

But it might apply to the initial value (as in my example where a reduced value is respected - note that doesn't return (reduced 1), just 1). Your suggested documentation change is talking about input values, but in my mind that leads to incorrect conclusions.

The only change that would make sense to me is clarifying where a "reduced" value is checked (on the result of applying the function passed to reduce). I think that's already implicit in the existing doc string myself. Since we have multiple implementations of "reduce", we have to tread carefully not to refer to explicitly to a particular one.

This use of a reduced initial value does not even make sense; why we would we confuse the docstring to warn about it?

Comment by Jean Niklas L'orange [ 27/Jul/14 7:20 AM ]

Ah, I get your point now, and I see how this would just create more confusion.

Thanks for the explanation.





[CLJ-1468] Add deep-merge and deep-merge-with Created: 18/Jul/14  Updated: 18/Jul/14  Resolved: 18/Jul/14

Status: Closed
Project: Clojure
Component/s: None
Affects Version/s: None
Fix Version/s: None

Type: Enhancement Priority: Minor
Reporter: Stuart Sierra Assignee: Unassigned
Resolution: Declined Votes: 1
Labels: None

Attachments: Text File CLJ-1468-deep-merge-01.patch    
Patch: Code and Test
Approval: Triaged

 Description   

When dealing with nested map structures, one often wants to merge two maps recursively.

The deep-merge-with function was originally written by Chris Houser for clojure.contrib.map-utils but was not maintained after clojure-contrib was split into separate modules.

deep-merge and deep-merge-with are widely copied, usually with the same implementation, in utility libraries. For example:



 Comments   
Comment by Rich Hickey [ 18/Jul/14 11:42 AM ]

Vague semantics and docs strings





[CLJ-1465] SubVector leaks memory by design (?) Created: 14/Jul/14  Updated: 14/Jul/14  Resolved: 14/Jul/14

Status: Closed
Project: Clojure
Component/s: None
Affects Version/s: Release 1.6
Fix Version/s: None

Type: Defect Priority: Minor
Reporter: Szymon Witamborski Assignee: Unassigned
Resolution: Declined Votes: 0
Labels: memory


 Description   

While reading through the code of SubVector class, I've noticed that the only thing that it does is creating a "subview" of an existing vector without doing anything to elements that are not accessible any more. Please correct me if I'm wrong but I think this leads to memory leaks in this scenario:

(let [a [1 2 3 4 5]
      b (subvec a 2 3)]
  b)
[3]

In this example we no longer have any reference to a once the (let...) expression returned. Elements of a that are no longer accessible will not be garbage collected until b is garbage collected because b still holds a reference to a (the v field in SubVector class). Ideally, these elements should be garbage collectible as soon as a is gone.



 Comments   
Comment by Alex Miller [ 14/Jul/14 2:40 PM ]

The subvec docstring says:

"Returns a persistent vector of the items in vector from start (inclusive) to end (exclusive). If end is not supplied, defaults to (count vector). This operation is O(1) and very fast, as the resulting vector shares structure with the original and no trimming is done."

The implementation is intentional to make this a constant-time operation. If you are willing to make the tradeoff re shared structure and object retention, this constant operation has better performance. In other words: working as intended.





[CLJ-1464] Incorrectly named parameter to fold function in reducers.clj Created: 12/Jul/14  Updated: 10/Sep/14  Resolved: 10/Sep/14

Status: Closed
Project: Clojure
Component/s: None
Affects Version/s: None
Fix Version/s: None

Type: Defect Priority: Trivial
Reporter: Jason Jackson Assignee: Unassigned
Resolution: Declined Votes: 0
Labels: None

Attachments: Text File 0001-Rename-a-function-parameter-to-reflect-the-fold-func.patch    
Patch: Code
Approval: Triaged

 Description   

https://github.com/clojure/clojure/blob/master/src/clj/clojure/core/reducers.clj#L95
The 2-arity fold accepts reducef as parameter and then uses it as a combinef.
Instead it should accept combinef as parameter and then use it as a reducef, as every combine fn (monoid) is a reduce fn, but not every reduce fn is a combine fn (it's not associative).



 Comments   
Comment by Jason Jackson [ 12/Jul/14 2:58 PM ]

this is my first patch for clojure please double check everything. CA is done.

Comment by Andy Fingerhut [ 12/Jul/14 7:29 PM ]

Everything gets double checked whether it is your first patch or your 50th

At least as far as the format of the patch being correct, that it applies cleanly to the latest version of Clojure on the master branch, compiles and passes all tests cleanly, all of that is good.

Whether there is interest in taking your proposed change is to be decided by others. It may be some time before it is examined further.

Comment by Jozef Wagner [ 13/Jul/14 4:11 AM ]

This is not a defect. Quoting Rich, "If no combining fn is supplied, the reducing fn is used." (source)

There are three user supplied operations in fold: getting identity element (combinef :: -> T), reducing function (reducef :: T * E -> T) and combining function (combinef :: T * T -> T). For reduce, combining function is not needed but the rest two operations are needed. Thus reducing function (reducef) supplies identity element for reducers and only in folders the identity element is produced by combining function. In case where reducing fn is used for both reducing and combining, it must of course be associative and must handle objects of types T and E as a second argument.

Comment by Jason Jackson [ 14/Jul/14 12:14 AM ]

@Jozef I appreciate the feedback I still think my patch is correct, although I admit everyone's time is better spent not debating this small refactoring so feel free to close it.

One perspective to view my patch from is if we had protocols p/Monoid and p/Reducer. It's possible to reify any object that implements p/Monoid into p/Reducer, but not the other way around since not every p/Reducer is associative.

Comment by Jason Jackson [ 14/Jul/14 12:30 AM ]

From that perspective you could also say that in the 2-arity case the parameter "reducef" requires objects that implement both p/Monoid and p/Reducer, but in the 3-arity case the parameter "reducef" only requires p/Reducer

Comment by Jozef Wagner [ 14/Jul/14 1:38 AM ]

Note that reducef and combinef take different type of second argument, so not every combining function can be used as a reducing one. Your proposal is thus no better than the status quo. Consider following example:

(fold clojure.set/union conj [1 1 2 2 3 3 4 4])




[CLJ-1450] Add a variant of range that's inclusive with respect to its upper boundary Created: 19/Jun/14  Updated: 19/Jun/14  Resolved: 19/Jun/14

Status: Closed
Project: Clojure
Component/s: None
Affects Version/s: Release 1.6
Fix Version/s: None

Type: Enhancement Priority: Minor
Reporter: Bozhidar Batsov Assignee: Unassigned
Resolution: Declined Votes: 0
Labels: None


 Description   

Such a variant of range might behave like this:

```
(irange 1 5)
;;=> (1 2 3 4 5)

(irange 1 2 5)
;;=> (1 3 5)
```

I'm suggesting this because often in practice we have to deal with inclusive ranges and having to add 1 to the upper boundary in such scenarios seems to reduce the clarity of the code. I guess something like this applies to some extend to lower boundary as well:

```
(erange 1 5)
;;=> (2 3 4)
```



 Comments   
Comment by Alex Miller [ 19/Jun/14 2:55 PM ]

We're not interested in adding these to core, thanks.





[CLJ-1448] Suggest alength in error message on attempt to access array length via .length Created: 19/Jun/14  Updated: 19/Jun/14  Resolved: 19/Jun/14

Status: Closed
Project: Clojure
Component/s: None
Affects Version/s: None
Fix Version/s: None

Type: Enhancement Priority: Trivial
Reporter: Colin Taylor Assignee: Unassigned
Resolution: Declined Votes: 0
Labels: errormsgs

Attachments: Text File dot-length-recommend-alength.patch    
Patch: Code

 Description   

Problem:

Newcomers are easily confused by the inability to access <array>.length via (.length <array>).

Approach:

Append to invalid field access message, a suggestion to use alength for this specific case (class.isArray and field = "length")

user=> (.length (int-array 2))
IllegalArgumentException No matching field found: length for class [I, use alength function for array length


 Comments   
Comment by Alex Miller [ 19/Jun/14 2:56 PM ]

We don't have any plans to add anything this specific to the error check, thanks.





[CLJ-1440] Unable to exclude clojure.lang.Compiler using :refer-clojure Created: 06/Jun/14  Updated: 07/Jun/14  Resolved: 07/Jun/14

Status: Closed
Project: Clojure
Component/s: None
Affects Version/s: Release 1.6
Fix Version/s: None

Type: Defect Priority: Major
Reporter: Howard Lewis Ship Assignee: Unassigned
Resolution: Declined Votes: 0
Labels: compiler, interop


 Description   
(ns io.aviso.twixt.js-minification
  "Provides support for JavaScript minification using the Google Closure compiler."
  (:refer-clojure :exclude [Compiler])
  (:import (com.google.javascript.jscomp CompilerOptions ClosureCodingConvention DiagnosticGroups CheckLevel
                                         SourceFile Result Compiler))
  (:require [clojure.java.io :as io]
            [io.aviso.twixt.utils :as utils]
            [io.aviso.tracker :as t]
            [clojure.string :as str]))

Results in:

clojure.lang.Compiler$CompilerException: java.lang.IllegalStateException: Compiler already refers to: class clojure.lang.Compiler in namespace: io.aviso.twixt.js-minification, compiling:(/Users/hlship/workspaces/annadale/twixt/src/io/aviso/twixt/js_minification.clj:1:1)
        java.lang.IllegalStateException: Compiler already refers to: class clojure.lang.Compiler in namespace: io.aviso.twixt.js-minification
                                     clojure.lang.Namespace.referenceClass                    Namespace.java:  140
                                        clojure.lang.Namespace.importClass                    Namespace.java:  158
                                        clojure.lang.Namespace.importClass                    Namespace.java:  164
                   io.aviso.twixt.js-minification/eval4104/loading--auto--               js_minification.clj:    1
                                   io.aviso.twixt.js-minification/eval4104               js_minification.clj:    1
                                                clojure.lang.Compiler.eval                     Compiler.java: 6703
                                                clojure.lang.Compiler.eval                     Compiler.java: 6692
                                                clojure.lang.Compiler.load                     Compiler.java: 7130
                                   io.aviso.twixt.js-minification/eval4100  form-init4106199735960171933.clj:    1
                                                clojure.lang.Compiler.eval                     Compiler.java: 6703
                                                clojure.lang.Compiler.eval                     Compiler.java: 6666
                                                         clojure.core/eval                          core.clj: 2927
                                      clojure.main/repl/read-eval-print/fn                          main.clj:  239
                                         clojure.main/repl/read-eval-print                          main.clj:  239
                                                      clojure.main/repl/fn                          main.clj:  257
                                                         clojure.main/repl                          main.clj:  257
                                                clojure.lang.RestFn.invoke                       RestFn.java: 1096
             clojure.tools.nrepl.middleware.interruptible-eval/evaluate/fn            interruptible_eval.clj:   56
                                            clojure.lang.AFn.applyToHelper                          AFn.java:  152
                                                  clojure.lang.AFn.applyTo                          AFn.java:  144
                                                        clojure.core/apply                          core.clj:  624
                                               clojure.core/with-bindings*                          core.clj: 1862
                                                clojure.lang.RestFn.invoke                       RestFn.java:  425
                clojure.tools.nrepl.middleware.interruptible-eval/evaluate            interruptible_eval.clj:   41
clojure.tools.nrepl.middleware.interruptible-eval/interruptible-eval/fn/fn            interruptible_eval.clj:  171
                                                      clojure.core/comp/fn                          core.clj: 2402
             clojure.tools.nrepl.middleware.interruptible-eval/run-next/fn            interruptible_eval.clj:  138
                                                      clojure.lang.AFn.run                          AFn.java:   22
                         java.util.concurrent.ThreadPoolExecutor.runWorker           ThreadPoolExecutor.java: 1145
                        java.util.concurrent.ThreadPoolExecutor$Worker.run           ThreadPoolExecutor.java:  615
                                                      java.lang.Thread.run                       Thread.java:  724


 Comments   
Comment by Nicola Mometto [ 06/Jun/14 4:52 PM ]

refer and thus refer-clojure only works for Vars.
a workaround is:

(ns ..)
(ns-unmap *ns* 'Compiler)
(import 'com.google.javascript.jscomp.Compiler)
Comment by Alex Miller [ 07/Jun/14 11:34 AM ]

Ditto what Nicola said. Or just fully-qualify.





[CLJ-1439] Reduce keyword cache lookup cost Created: 05/Jun/14  Updated: 01/Aug/14  Resolved: 01/Aug/14

Status: Resolved
Project: Clojure
Component/s: None
Affects Version/s: Release 1.6
Fix Version/s: Release 1.7

Type: Enhancement Priority: Minor
Reporter: Kyle Kingsbury Assignee: Unassigned
Resolution: Completed Votes: 1
Labels: keywords, performance

Attachments: Text File 0001-Improve-Keyword.intern-performance.patch    
Patch: Code
Approval: Ok

 Description   

Background: Symbol is composed of name and namespace strings. Symbol construction interns both of these strings - this reduces memory usage and allows for string == checks inside Symbol. Keywords wrap a Symbol and have an additional cache to reuse Keyword instances.

Problem: Certain applications make heavy use of keywords (in particular the case of parsing or transforming JSON, XML, or other data into Clojure maps with keyword keys). Constructing the same keyword from a string over and over again will cause the string to be interned, a symbol constructed, and the lookup to occur in the keyword cache. In the case where the keyword already exists, this is more work than is necessary, making this path slower than it can be.

Reproduce: The following test simulates rounds of creating many keywords - the unique? flag indicates whether to use new or the same set of keywords each rep. unique?=false should be more similar to parsing a similar JSON record format over and over.

(set! *unchecked-math* true)

(defn kw-new [n unique?]
  (let [base (if unique? (str (rand)) "abcdef")]
    (loop [i 0
           kws (transient [])]
      (if (< i n)
        (recur (inc i) (conj! kws (keyword (str base i))))
        (persistent! kws)))))

(defn bench-kw [reps n unique?]
  (dotimes [_ reps]
    (let [begin (System/nanoTime)]
        (kw-new n unique?)
        (let [end (System/nanoTime)
              elapsed (/ (- end begin) 1000000.0)]
          (println elapsed "ms")))))

(bench-kw 50 10000 false)  ;; expected similar to JSON use case
(bench-kw 50 10000 true)   ;; for comparison

On 1.6, we see about 5.5 ms for repeated and 134 ms for unique after warmup.
With the patch, we see about 2.2 ms for repeated and 120 ms for unique after warmup.

Cause: Keyword construction based on a string involves:

  • Interning string(s) in new kw
  • Constructing Symbol with interned strings
  • Clearing Keywords from the Keyword cache if GC has reclaimed them
  • Constructing a new Keyword
  • Wrapping the Keyword in a WeakReference
  • CHM putIfAbsent on the cache
  • If new, return. If exists, get the old one and return.
  • In the event the Keyword is reclaimed by GC between the last 2 steps, retry.

This process involves a fair amount of speculative interning and object creation if the keyword already exist.

Proposal: Streamline the keyword construction process by reworking the cache implementation and the Keyword.intern() process. The patch changes the cache to key by string name instead of symbol, deferring interning and symbol creation on lookup to when we know the keyword construction is needed. The various Keyword.intern() methods are also reworked to take advantage if called with an existing Symbol to avoid re-creating it.

Patch: 0001-Improve-Keyword.intern-performance.patch

Related: CLJ-1415



 Comments   
Comment by Alex Miller [ 01/Aug/14 11:48 AM ]

Alternate changes were committed today to improve both symbol and keyword creation times.

https://github.com/clojure/clojure/commit/c9e70649d2652baf13b498c4c3ebb070118c4573





[CLJ-1437] Keyword cache lookup could be faster Created: 05/Jun/14  Updated: 05/Jun/14  Resolved: 05/Jun/14

Status: Closed
Project: Clojure
Component/s: None
Affects Version/s: Release 1.6
Fix Version/s: None

Type: Enhancement Priority: Major
Reporter: Alex Miller Assignee: Unassigned
Resolution: Duplicate Votes: 0
Labels: keywords, performance


 Description   

Breaking this piece out from CLJ-1415. Keyword cache lookup includes symbol creation and interning which could be avoided, making cache lookup faster.






[CLJ-1430] Improve performance of partial Created: 23/May/14  Updated: 05/Sep/14  Resolved: 29/Aug/14

Status: Closed
Project: Clojure
Component/s: None
Affects Version/s: None
Fix Version/s: Release 1.7

Type: Enhancement Priority: Minor
Reporter: Timothy Baldridge Assignee: Unassigned
Resolution: Completed Votes: 0
Labels: performance

Attachments: File partial-perf.diff    
Patch: Code
Approval: Ok

 Description   

This patch improves performance of partial by only using apply when needed. The code structure follows that of juxt.

Performance benchmark:

(ns partial-test.core
  (:require [criterium.core :refer [bench]])
  (:gen-class))

(defn -main []
  (let [f (partial + 1 1)]
    (println "Starting")
    (bench (f 1 1))
    (println "Done")))

Results for 1.6.0:

Evaluation count : 228751140 in 60 samples of 3812519 calls.
             Execution time mean : 266.700063 ns
    Execution time std-deviation : 2.966851 ns
   Execution time lower quantile : 262.641023 ns ( 2.5%)
   Execution time upper quantile : 274.207916 ns (97.5%)
                   Overhead used : 1.610513 ns

Found 3 outliers in 60 samples (5.0000 %)
	low-severe	 3 (5.0000 %)
 Variance from outliers : 1.6389 % Variance is slightly inflated by outliers

Results for 1.7.0 with this patch:

 Evaluation count : 348208140 in 60 samples of 5803469 calls.
              Execution time mean : 171.210533 ns
     Execution time std-deviation : 2.011660 ns
    Execution time lower quantile : 168.819526 ns ( 2.5%)
    Execution time upper quantile : 176.015584 ns (97.5%)
                    Overhead used : 2.644128 ns

 Found 3 outliers in 60 samples (5.0000 %)
 	low-severe	 3 (5.0000 %)
  Variance from outliers : 1.6389 % Variance is slightly inflated by outliers

Benchmarks performed via lein uberjar + running via the commandline.

Patch: partial-perf.diff

Screened by: Alex Miller



 Comments   
Comment by Alex Miller [ 23/May/14 10:46 AM ]

Screened, looks as expected.

Comment by Andy Fingerhut [ 02/Jun/14 10:50 AM ]

Timothy, just a nit that I would not have noticed except for my program that checks for name and email address of patch authors, to see if they are on my contributor's list, but do you really have both of the email addresses tbaldridge@gmail.com and tbaldidge@gmail.com (note the spelling difference)? The latter is the one on this patch.

Comment by Timothy Baldridge [ 02/Jun/14 11:04 AM ]

fixed email

Comment by Timothy Baldridge [ 02/Jun/14 11:05 AM ]

nice catch! it was a typeo in my .gitconfig defaults. I've fixed the patch.

Comment by Alex Miller [ 02/Jun/14 11:19 AM ]

Tim (and anyone really) - please let someone know if you need to change a screened patch! Looks fine here, but screener should be notified so they can re-screen.

Comment by Alex Baranosky [ 05/Sep/14 9:11 PM ]

Very nice patch. I've gotten into the habit of not using partial anymore for performance sensitive code. Perhaps this change means I need to rethink that.





[CLJ-1429] Cache unknown multimethod value default dispatch Created: 22/May/14  Updated: 29/Aug/14  Resolved: 29/Aug/14

Status: Closed
Project: Clojure
Component/s: None
Affects Version/s: Release 1.6
Fix Version/s: Release 1.7

Type: Enhancement Priority: Major
Reporter: Alex Miller Assignee: Unassigned
Resolution: Completed Votes: 2
Labels: performance

Attachments: Text File clj-1429.patch    
Patch: Code
Approval: Ok

 Description   

Multimethods maintain a cache from dispatch value (result of the dispatch function) to dispatch method. If the dispatch value does not find a match in the available methods, it falls through to a lookup using the default dispatch value and returns that method. This default dispatch case is NOT recorded in the cache. This means that every case that falls through to the default case incurs a scan of the methodTable (and the class inheritance checks that involves).

Perf test:

(defmulti mm class)
(defmethod mm String [s] s)
(defmethod mm Long [l] l)
(defmethod mm :default [v] v)

(defn perf [reps size]
  (let [data (take size (cycle ["abc" 5 :k]))]
    (dotimes [_ reps]
      (time (doall (map mm data))))))

And results:

;; Without patch:
user=> (perf 5 100000)
"Elapsed time: 1301.262 msecs"
"Elapsed time: 928.888 msecs"
"Elapsed time: 942.905 msecs"
"Elapsed time: 858.513 msecs"
"Elapsed time: 832.314 msecs"

;; With patch:
user=> (perf 5 100000)
"Elapsed time: 134.169 msecs"
"Elapsed time: 28.859 msecs"
"Elapsed time: 45.452 msecs"
"Elapsed time: 13.189 msecs"
"Elapsed time: 13.42 msecs"

Attached patch caches the mapping of unknown value -> default dispatch method and significantly improves the performance for this case.

Patch: clj-1429.patch
Screened by: Stu






[CLJ-1427] restart-agent is ignored inside an fn passed to set-agent-handler. It shall maybe documented ? Created: 19/May/14  Updated: 20/May/14  Resolved: 20/May/14

Status: Closed
Project: Clojure
Component/s: None
Affects Version/s: Release 1.6
Fix Version/s: None

Type: Defect Priority: Minor
Reporter: Rafik NACCACHE Assignee: Unassigned
Resolution: Duplicate Votes: 0
Labels: None
Environment:

Linux, jdk 1.7, emacs / cider



 Description   

If I pass a function containing start-agent to set-error-handler of an agent, if an exception occurs the restart-agent is ignored.
for example:

(def a (agent 0))

(set-error-handler! a (fn [the-agent the-exception] (restart-agent the-agent)) )

If I now issue : (send! a #(/ 1 0)), I still have a failed agent. It did not restart.

I know I can set the error-mode to the agent to :continue to someone have my agent up after a crash, but I wished I could fix the conditions that caused the exception and restart the agent programmatically in the set-error-handler.

Maybe it is a known feature, but it is not documented.



 Comments   
Comment by Rafik NACCACHE [ 20/May/14 11:15 AM ]

Sorry, I probably hit return while typing, this one is duplicate of clj-1428





[CLJ-1426] "/" in keyword leads to unexpected behavior when calling `name` Created: 18/May/14  Updated: 21/May/14  Resolved: 21/May/14

Status: Closed
Project: Clojure
Component/s: None
Affects Version/s: None
Fix Version/s: None

Type: Defect Priority: Minor
Reporter: Darrell Hamilton Assignee: Unassigned
Resolution: Declined Votes: 0
Labels: None


 Description   

Because `clojure.lang.Keyword` delegates it's `getName` functionality to `clojure.lang.Symbol`, there is some unexpected behavior when calling `name` on a keyword that contains a "/" in it. For example:

(name (keyword "foo/bar"))
=> "bar"

This is due to `Symbol` stripping all namespace qualifiers and considering only the content trailing the "/" as part of the name.



 Comments   
Comment by Darrell Hamilton [ 20/May/14 9:29 PM ]

Totally misunderstood the behavior of namespaced keywords. this can be closed.

/derp





[CLJ-1421] It's not a digit but regexp classify this one as a digit. Created: 14/May/14  Updated: 14/May/14  Resolved: 14/May/14

Status: Closed
Project: Clojure
Component/s: None
Affects Version/s: None
Fix Version/s: None

Type: Defect Priority: Major
Reporter: Ale Assignee: Unassigned
Resolution: Not Reproducible Votes: 0
Labels: None

Attachments: PNG File scr.png    

 Comments   
Comment by Ale [ 14/May/14 12:53 AM ]

[org.clojure/clojure "1.5.1"]

Comment by Ale [ 14/May/14 4:20 AM ]

I have no possibility to close this issue. It was wrong "\" before "1".





[CLJ-1415] Keyword cache cleanup incurs linear scan of cache Created: 06/May/14  Updated: 01/Aug/14  Resolved: 01/Aug/14

Status: Resolved
Project: Clojure
Component/s: None
Affects Version/s: Release 1.6
Fix Version/s: Release 1.7

Type: Enhancement Priority: Minor
Reporter: Kyle Kingsbury Assignee: Alex Miller
Resolution: Not Reproducible Votes: 6
Labels: keywords, performance

Attachments: File faster-keywords.diff     File keyword-cache.diff     Text File kw-clean-future.patch     File unified-kw-patch.diff    
Patch: Code
Approval: Vetted

 Description   

If the GC reclaims a keyword, any subsequent attempt to create a keyword requires an O(n) scan over the entire keyword table via Util.clearCache. This is a significant performance cost in keyword-heavy operations; e.g. JSON parsing.

Patch: keyword-cache.diff - patch to defer cleaning till portion of the table is dead and avoid multiple threads cleaning simultaneously.

Patch: kw-clean-future.patch - patch to spin cache cleaning into a future. Found that 1) this introduces some ordering constraints and circularity between Agent and Keyword (fixable) and 2) using the future pool for this means shutdown-agents would always need to be called (in the patch I avoided this by changing agent's soloExecutor to use daemon threads.

Patch: unified-kw-patch.diff - Alternative to keyword-cache and clean-future.patch. Combines all keyword-perf changes, including the EDN kw parser improvement, improved table lookup performance, and has threads cooperate to empty the table refqueue with a minimum of contention.



 Comments   
Comment by Alex Miller [ 06/May/14 5:53 PM ]

Any perf-related ticket will need some clear before/after timings (with good methodology and how to repro) and also a consideration of cases where the change may introduce any perf degradation in normal usage.

Comment by Kyle Kingsbury [ 07/May/14 9:54 PM ]

I've experimented with a patch reducing the cache clearing cost and removing the need for String.intern. Preliminary results are good, but I want to try a few alternative approaches for cache keys. For instance, could we use pure strings like "foo" and "clojure.core/foo" as the cache keys, removing a level of memory indirection? If we're being really sneaky, we could share those same strings with the Symbol _str field to halve our memory use, assuming it's OK to reach in and mutate it.

https://gist.github.com/aphyr/f72e72992dade4578232
http://imgur.com/a/YSgUa#2

Comment by Alex Miller [ 08/May/14 12:29 PM ]

Great start on this - having the perf data is hugely important. One thing I don't see you've covered yet is what the corresponding memory increase you're incurring with CacheKey to get the benefit - we need to quantify both sides of the tradeoff here (latency/throughput vs memory) to fully judge.

Questions/comments on your patch...

1) https://gist.github.com/aphyr/f72e72992dade4578232#file-gistfile1-diff-L101 - do we need the (o instanceof CacheKey) check? If the usage of this is constrained then we might be able to skip it (and it will blow up on the next line if something is wrong).

2) https://gist.github.com/aphyr/f72e72992dade4578232#file-gistfile1-diff-L110 - shouldn't we precompute and save the hash code!? The only thing we're making this for is fast hash comparisons. That hash computation is string length dependent - it ain't cheap.

3) https://gist.github.com/aphyr/f72e72992dade4578232#file-gistfile1-diff-L126 - have you tested with other values here? Should have some justification for this.

4) https://gist.github.com/aphyr/f72e72992dade4578232#file-gistfile1-diff-L126 - have you tested with other values here? Should have some justification that this is a reasonable number.

5) https://gist.github.com/aphyr/f72e72992dade4578232#file-gistfile1-diff-L169 - there is a race here (actually more than one if you include getting the tableSize):

Th1: orphansCount = orphans.get()
Th2: orphansCount = orphans.get()
Th2: orphansNew = orphans.getAndSet(0)
Th2: orphansNew > orphansCount -> start cleaning
<huge gc, 10 zillion orphans are created>
Th1: orphansNew = orphans.getAndSet(0)
Th1: orphansNew > orphansCount -> start cleaning

but I guess this is "safe"; we just have multiple threads cleaning in that case.

6) In general it seems pretty sloppy (I don't mean that pejoratively) how the orphans, rq, and cleaning thread are working together and may be out of sync. I get it and I even believe it will work (usually) but I worry about timing issues that I am too dumb to think of.

Is there a simpler strategy? What if every Nth call to intern() cleaned M entries (or up to M% of table)?

7) https://gist.github.com/aphyr/f72e72992dade4578232#file-gistfile1-diff-L177 - if you made the iterator explicit in this loop, it would be safe to call iterator.remove() instead of table.remove() and I believe that would be faster on CHM.

8) https://gist.github.com/aphyr/f72e72992dade4578232#file-gistfile1-diff-L198 - could have two versions of this with/without the symbol. Not sure if that would be faster but they might both inline better into their callers then?

9) https://gist.github.com/aphyr/f72e72992dade4578232#file-gistfile1-diff-L242 - what's the use case for finding an external CacheKey? Fast re-lookup for specialized use? Do we want to commit to this in the API?

Comment by Kyle Kingsbury [ 08/May/14 2:41 PM ]

1) https://gist.github.com/aphyr/f72e72992dade4578232#file-gistfile1-diff-L101 - do we need the (o instanceof CacheKey) check? If the usage of this is constrained then we might be able to skip it (and it will blow up on the next line if something is wrong).

I'm usually wary of violating equality/hashCode contracts, and this doesn't even appear as a blip on the radar in profiling. I think we could drop it

2) https://gist.github.com/aphyr/f72e72992dade4578232#file-gistfile1-diff-L110 - shouldn't we precompute and save the hash code!? The only thing we're making this for is fast hash comparisons. That hash computation is string length dependent - it ain't cheap.

It's less memoizable than you might think; each CacheKey is only indexed a few times, and only at query time; it also doesn't help us for equality checks, since those only occur after hashing. I can add a memoizing field for it at the cost of another 32 bits/kw; we'll see how it impacts performance.

3) https://gist.github.com/aphyr/f72e72992dade4578232#file-gistfile1-diff-L126 - have you tested with other values here? Should have some justification for this.

I experimented with several values on the Clojure test suite, benchmarks, and some real-world hadoop code. Diminishing returns, as you'd expect. 0.1 and 0.5 are essentially identical in runtime tradeoff. We could drop to 0.01 if desired; it'll only make a difference in large (10-100K) extant keyword benchmarks, as far as I can tell.

4) https://gist.github.com/aphyr/f72e72992dade4578232#file-gistfile1-diff-L126 - have you tested with other values here? Should have some justification that this is a reasonable number.

Same question as #3?

5) https://gist.github.com/aphyr/f72e72992dade4578232#file-gistfile1-diff-L169 - there is a race here (actually more than one if you include getting the tableSize):

Th1: orphansCount = orphans.get()
Th2: orphansCount = orphans.get()
Th2: orphansNew = orphans.getAndSet(0)
Th2: orphansNew > orphansCount -> start cleaning
<huge gc, 10 zillion orphans are created>
Th1: orphansNew = orphans.getAndSet(0)
Th1: orphansNew > orphansCount -> start cleaning

but I guess this is "safe"; we just have multiple threads cleaning in that case.

Yep. This check is only there as an optimization--and note that if a huge GC occurs, it's likely we want to schedule a followup traversal of the table anyway, because the thread that's already cleaned some of the table has probably missed some subsequently GC'ed elements. The number of concurrently cleaning threads is bounded by the rate of GC churn, and in the most pathological case (sadly, I haven't been able to produce this race experimentally), this degenerates to the old Clojure behavior of every thread doing a full scan.

6) In general it seems pretty sloppy (I don't mean that pejoratively) how the orphans, rq, and cleaning thread are working together and may be out of sync. I get it and I even believe it will work (usually) but I worry about timing issues that I am too dumb to think of.

Is there a simpler strategy? What if every Nth call to intern() cleaned M entries (or up to M% of table)?

Every nth call is just fine, but it degrades more poorly for large tables. In general, I try to lean towards scale-invariant solutions, which is why I aimed to reclaim roughly a tenth of the entries in the map every time. Maybe more, maybe less, depending on CAS retries, delayed threads resetting the counter to zero, etc.

Doing M entries or M% is more tricky; gotta figure out which threads collect what fraction when, how you efficiently access that subsection of the hash, make sure elements don't fall through the cracks, etc.

7) https://gist.github.com/aphyr/f72e72992dade4578232#file-gistfile1-diff-L177 - if you made the iterator explicit in this loop, it would be safe to call iterator.remove() instead of table.remove() and I believe that would be faster on CHM.

I agree. I figured Rich had a good reason for doing it this way, but if you concur I'll change it.

8) https://gist.github.com/aphyr/f72e72992dade4578232#file-gistfile1-diff-L198 - could have two versions of this with/without the symbol. Not sure if that would be faster but they might both inline better into their callers then?

I agree. We can do that dispatch statically and cut down on branch misprediction, too.

9) https://gist.github.com/aphyr/f72e72992dade4578232#file-gistfile1-diff-L242 - what's the use case for finding an external CacheKey? Fast re-lookup for specialized use? Do we want to commit to this in the API?

Keep forgetting Java's obsession with encapsulation. I'll privatize.

Comment by Alex Miller [ 08/May/14 10:14 PM ]

On several of these - 2, 7, 8 - I think those are worth a test. If faster, we should consider.

On 9, I thought maybe you were opening it up so it would be possible to save off a CacheKey and reuse it or something else. If it's not needed externally, then might be good to private-ize CacheKey itself so we can change it later.

Comment by Kyle Kingsbury [ 09/May/14 6:04 PM ]

http://imgur.com/a/1bv3P#0
https://gist.github.com/aphyr/f72e72992dade4578232

These charts show the performance impact of several changes. In order, they are:

1.7                 baseline
kw                  initial patch
kw-static-paths     Separate codepaths for interning symbols vs strings. Iterator
                    .remove for cache cleaning. Fix a bug for null comparisons
                    in CacheKey namespaces. Internal functions now protected, not
                    public. Not much performance impact.
kw-memo-hash        Memoize hashcodes for CacheKeys. Performance is a wash.
kw-string-cachekeys Observing that String.indexOf('/') consumed a significant 
                    fraction of interning time, use a combined "ns/name" string for
                    Cachekeys instead of separate strings. Significant performance 
                    boost in all tests; 40% reduction in median latencies in 1000-
                    kw allocation test, for instance.
kw-string-keys      Use raw strings for CacheKeys. Improves performance by removing
                    a level of memory indirection, even without cached hashcodes.
kw-interned-keys    Intern those strings to reduce memory consumption, sharing
                    them with the underlying symbol's strings. Slightly slower.

Performance is even better now. Creating 1000 keywords median latency changed from 900 to 200 micros; .999s lower, throughput from 4000 to 20,000/second. JSON parsing median latency fell from 170 micros to 100 micros; throughput doubled from 17500 docs/sec to 36,000 docs/sec.

We're still suffering from poor dispersal in ConcurrentHashmap's use of the string hashCode on JDK7/8, but maybe that's a subject for a different patch.

Memory impact is now minimal. We intern every key string in the table, and those strings are interned by the symbols anyway, so they're essentially the same object. For namespaced symbols, we pay a slightly higher cost--forcing the interning of the "ns/name" string instead of deferring it to Symbol.toString() time. For non-namespaced symbols, these strings are interned as a part of the symbol creation process so there's no memory overhead.

At the repl, I tested by allocating and retaining a million keywords:

(def x (mapv keyword (map (partial str "test-kw-") (range 1e6))))

Retained size (bytes)             1.7   string-kw
----------------------------------------------------
Total retained heap        221.    MB  221.    MB
clojure.lang.Symbols       104.820 MB   32.900 MB
clojure.lang.Keywords       24.021 MB   56.049 MB
java.lang.Strings           89.537 MB   81.786 MB
clojure.lang.Keyword class  72.447 MB   72.451 MB

Total memory use is unchanged, but note that clojure.lang.Symbol retains less, since its strings are now shared by the Keyword table. Keywords, by contrast, retains more. Strings and the keyword table are essentially unchanged.

Comment by Kyle Kingsbury [ 09/May/14 6:08 PM ]

I can't figure out how to edit the ticket description, but I updated the same gist with the cumulative changes. Comments welcome!

Comment by Alex Miller [ 09/May/14 9:51 PM ]

Excellent, thanks for the data. I added a group to your auth so I think you should be able to edit descriptions now. If not, let me know. I'll re-review the patch next week. It would be good either at this point or after that to turn this into an actual patch file instead of a gist.

Comment by Kyle Kingsbury [ 12/May/14 4:24 PM ]

I've attached a cumulative patch. It's comprised of 8 commits; one for each stage we've discussed. I can rebase into a single commit if you'd like.

Comment by Alex Miller [ 13/May/14 7:31 AM ]

I would like a single cumulative rebased patch. I hope to have some time to look at it today.

Comment by Alex Miller [ 13/May/14 12:39 PM ]

On another look, I think it would be useful to separate this ticket into two parts - the first is about optimizing keyword creation and lookup to avoid unnecessary work (avoiding symbol creation and interning, using Strings as keys in the cache). The second part is really about optimizing cache clearing. Do you think these can be separated into two tickets?

Regarding the cache clearing part, have you tested a simpler strategy of just counting calls to clearCache() and running the clean scan every N calls? If that was almost as good, I'd be in favor of that over what is in the patch.

The kw-static paths version did not seem to be an improvement - perhaps you should try pulling them back together to simplify the code? Only worth it if there is a real improvement from it.

On the various find methods - if you could retain their ordering and minimize noise in the diffs that would really help make it easier to screen.

Finally, we need to do some tests to verify that we have not altered the performance of using keywords and symbols as keys in a map for lookup.

Comment by Kyle Kingsbury [ 05/Jun/14 2:36 PM ]

> On another look, I think it would be useful to separate this ticket into two parts - the first is about optimizing keyword creation and lookup to avoid unnecessary work (avoiding symbol creation and interning, using Strings as keys in the cache). The second part is really about optimizing cache clearing. Do you think these can be separated into two tickets?

Created dev.clojure.org/jira/browse/CLJ-1439 for reduced intern cost

> Regarding the cache clearing part, have you tested a simpler strategy of just counting calls to clearCache() and running the clean scan every N calls? If that was almost as good, I'd be in favor of that over what is in the patch.

I'm not confident that this work will be merged, so I'm kinda reticent to go off and spend another N hours doing benchmarks.

> The kw-static paths version did not seem to be an improvement - perhaps you should try pulling them back together to simplify the code? Only worth it if there is a real improvement from it.

It was obsoleted by a later commit; only included it in the benchmark because you asked about the perf impact.

> On the various find methods - if you could retain their ordering and minimize noise in the diffs that would really help make it easier to screen.

Done.

> Finally, we need to do some tests to verify that we have not altered the performance of using keywords and symbols as keys in a map for lookup.

This doesn't touch the lookup path; costs are equivalent.

Comment by Alex Miller [ 09/Jun/14 1:53 PM ]

reduced patch with only the keyword cache clearing changes

Comment by Alex Miller [ 09/Jun/14 8:53 PM ]

Patch that spins cache cleaning into a future

Comment by Kyle Kingsbury [ 21/Jul/14 2:20 PM ]

Just as a followup: got bit by this issue again in a data analysis project today: JSON parsing thrashes the reference queue really hard. Merging this patch would definitely be appreciated. Yourkit screenshot here: http://aphyr.com/media/clojure-keyword-refqueue.png

Comment by Kyle Kingsbury [ 21/Jul/14 4:58 PM ]

Oh yeah, once these two are merged, here's a patch that speeds up my EDN parsing-heavy hadoop jobs by 2-3x. Avoids constructing the symbol every time.

--- a/src/jvm/clojure/lang/EdnReader.java
+++ b/src/jvm/clojure/lang/EdnReader.java
@@ -299,10 +299,9 @@ private static Object matchSymbol(String s){
                        return null;
                        }
                boolean isKeyword = s.charAt(0) == ':';
-               Symbol sym = Symbol.intern(s.substring(isKeyword ? 1 : 0));
                if(isKeyword)
-                       return Keyword.intern(sym);
-               return sym;
+                       return Keyword.intern(s.substring(1));
+               return Symbol.intern(s);
                }
        return null;
 }
Comment by Kyle Kingsbury [ 21/Jul/14 6:33 PM ]
public static void clearCache() {
  if(rq.poll() != null) {
    Agent.soloExecutor.submit(new Runnable() {
      public void run() {
        Util.clearCache(rq,table);
      }
    });
  }
}

This spawns literally hundreds of new threads – 30-40 at a time in my workload – which fight over the referencequeue. Also it causes a fair bit of contention on the executor itself during keyword-heavy allocation-all threads have to synchronize on the executor's queue-but that seems secondary to the cost of the cache-clearing threads themselves.

How about adding a single-thread executor to Agent for these sorts of housekeeping jobs?

Comment by Alex Miller [ 21/Jul/14 8:14 PM ]

I actually built another patch that did exactly that but never got around to attaching it; a single-threaded executor reserved solely for cache clearing. I tried actually making it completely independent but I found it was pretty easy for it to fall behind - it needs to be connected into the construction process to avoid blowing the queue up too big.

I have not been able to build an isolated test that actually showed any significant performance difference with just your cache-clearing change (what's currently attached as keyword-cache.diff) and not the other changes. I had many problems getting your test code to run but when I did get something to run I was not able to see any significant difference with just the keyword-cache.diff.

Comment by Kyle Kingsbury [ 22/Jul/14 6:43 PM ]

Managed to eliminate the refqueue contention by having only one thread involved in GCing at a time. Also doesn't require messing with background threads, and is less susceptible to the queue-overflow problem. Since the various extant patches don't apply cleanly on top of each other, I've re-written them in unified-kw-patch.diff, attached. Roughly doubles throughput compared to your patch, at least on a 24-core xeon running openjdk7.

http://aphyr.com/media/clojure-reduced-kw-refqueue-contention.png

Can you please reconsider merging? I've put over a hundred hours into writing, testing, and refining this patchset; it's been stable in production for months and has made a dramatic difference in several of our data-heavy Clojure programs.

Comment by Alex Miller [ 23/Jul/14 10:58 AM ]

Hey Kyle, I appreciate the time you've put into this. However, having a big giant patch tuned on a single use case is not an effective way to evolve the language. We need to separate and describe problems, then explore the solution space for each one, as independently as possible, while considering the impacts on all other use cases.

This particular ticket is concerned solely with the linear cleanup of the reference queue. Can you split out just a patch that deals with this issue? It would be helpful to have a test that demonstrates the performance problem and how this patch address the problem. My testing so far with the prior patch did not demonstrate any improvement.

It would also be helpful to have a squashed version of the complement of the changes related to interning on CLJ-1439 for consideration of that as a separate problem. (And maybe there is further splitting that could be done; I have not looked closely at the interning changes.)

Comment by Alex Miller [ 23/Jul/14 11:00 AM ]

The EdnReader changes, for example, should be a separate ticket.

Comment by Kyle Kingsbury [ 23/Jul/14 12:30 PM ]

Could you at least merge dev.clojure.org/jira/browse/CLJ-1439 first? I split it into a separate ticket over a month ago and these changes depend on it.

Comment by Alex Miller [ 23/Jul/14 1:27 PM ]

I would be happy to consider CLJ-1439 first. Can you update the patch there to be current and focused on the intern/cache?

Comment by Kyle Kingsbury [ 23/Jul/14 2:35 PM ]

The patch is current, and it is focused on the intern/cache.

Comment by Andy Fingerhut [ 01/Aug/14 9:31 PM ]

Kyle, CLJ-1439 was completed today via a commit to Clojure master. That also had the side effect of making all of the currently attached patches no longer apply cleanly. I haven't checked how easy or difficult it might be to update them to apply cleanly.

Comment by Alex Miller [ 01/Aug/14 11:35 PM ]

I am not able to reproduce a performance issue due to a linear scan of the reference queue cache. Obviously the scan occurs but in most cases the scan is comparatively fast and does not seem to be a bottleneck in the tests I've run.

If there is a test that can reproduce this issue (on current master) and demonstrates an improvement with this patch, please reopen the ticket for investigation.





[CLJ-1405] clojure runtime does not work with onejar-maven-plugin Created: 16/Apr/14  Updated: 06/Jul/14  Resolved: 06/Jul/14

Status: Closed
Project: Clojure
Component/s: None
Affects Version/s: Release 1.5, Release 1.6, Release 1.7
Fix Version/s: None

Type: Defect Priority: Minor
Reporter: Sean Shubin Assignee: Unassigned
Resolution: Duplicate Votes: 1
Labels: None
Environment:

I was able to repeat this problem on windows, linux, and osx


Attachments: Text File onejar-maven-plugin-fix.patch    
Patch: Code

 Description   

I have created a sample project, steps to repeat the problem, and a proposed patch here:
https://github.com/SeanShubin/clojure-one-jar

I have also attached my proposed patch to this ticket, and pasted the relevant portion of README.md below:

clojure-one-jar
===============

Sample project to demonstrate problem with combining clojure runtime with onejar-maven-plugin

Steps to repeat the problem
===========================

  • mvn package
  • java -jar target/greeter.jar

Behavior before patch is applied
================================
Exception in thread "main" java.lang.reflect.InvocationTargetException
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:483)
at com.simontuffs.onejar.Boot.run(Boot.java:340)
at com.simontuffs.onejar.Boot.main(Boot.java:166)
Caused by: java.lang.ExceptionInInitializerError
at com.seanshubin.clojure_one_jar.Greeter.main(Greeter.java:12)
... 6 more
Caused by: java.lang.NullPointerException
at clojure.lang.RT.lastModified(RT.java:387)
at clojure.lang.RT.load(RT.java:421)
at clojure.lang.RT.load(RT.java:411)
at clojure.lang.RT.doInit(RT.java:447)
at clojure.lang.RT.<clinit>(RT.java:329)
... 7 more

Behavior after patch is applied
===============================
No need to call RT.init() anymore
Hello, world!

Patch
=====
I applied this patch to my local copy of clojure 1.7.0-master-SNAPSHOT
https://github.com/SeanShubin/clojure-one-jar/blob/master/onejar-maven-plugin-fix.patch



 Comments   
Comment by Alex Miller [ 16/Apr/14 9:45 PM ]

I think this is a dupe of CLJ-971 - can you verify?

Comment by Sean Shubin [ 16/Apr/14 9:49 PM ]

Yes, I looked over CLJ-971 and can confirm this matches my observations regarding why this is happening.

Comment by Sean Shubin [ 16/Apr/14 9:54 PM ]

My solution is a bit different in that it still tries to get the date of the file rather than defaulting to the date of the entire jar. I don't have a solid enough understanding of the calling code to know if this is important.

Comment by Alex Miller [ 16/Apr/14 10:17 PM ]

FYI, to consider your patch, I would need you to sign a Contributor Agreement - see http://clojure.org/contributing for details. I do think that is an area that something is needed, will need to assess the options a bit more.

Comment by Sean Shubin [ 16/Apr/14 10:56 PM ]

Sure thing, I just signed and sent a Contributor Agreement, per the instructions at http://clojure.org/contributing

Comment by Gus Heck [ 05/Jul/14 9:42 PM ]

This effects the one-jar-gradle plugin too. Any progress on reviewing Sean's patch?

Comment by Sean Shubin [ 06/Jul/14 12:13 AM ]

Gus, I suggest moving this conversation over to CLJ-971.

The main problem is that I had not time to get it under test coverage, see Stuart Halloway's comment regarding that on CLJ-971. I have been pretty busy with my other projects, and being that I am not familiar with the clojure code base, I am going to have to block out some time to figure out how to get the appropriate tests in place, perhaps I will have some time for that next weekend.

Comment by Alex Miller [ 06/Jul/14 6:11 PM ]

Same issue as CLJ-971, which is farther along in the process.





[CLJ-1404] clojure.core/vals returns nil on an empty map instead of an empty sequence Created: 14/Apr/14  Updated: 14/Apr/14  Resolved: 14/Apr/14

Status: Closed
Project: Clojure
Component/s: None
Affects Version/s: None
Fix Version/s: None

Type: Defect Priority: Trivial
Reporter: Satshabad Khalsa Assignee: Unassigned
Resolution: Declined Votes: 0
Labels: None


 Description   

Is this a bug? maybe I just don't understand. The documentation says: Returns a sequence of the map's values. Is nil a sequence?

This caused an unexpected nil to propagate through a bunch of list processing stuff.



 Comments   
Comment by Alex Miller [ 14/Apr/14 10:54 PM ]

An empty sequence is represented by nil, so this is consistent. For example: (seq (range 0)) => nil





[CLJ-1397] exception testing broken over map Created: 01/Apr/14  Updated: 01/Apr/14  Resolved: 01/Apr/14

Status: Closed
Project: Clojure
Component/s: None
Affects Version/s: Release 1.5
Fix Version/s: None

Type: Defect Priority: Major
Reporter: MG Assignee: Unassigned
Resolution: Declined Votes: 0
Labels: test
Environment:

Linux ... 3.2.0-56-generic-pae #86-Ubuntu SMP ... i686 i686 i386 GNU/Linux



 Description   

Expected: Tests pass
Actual: Two tests fail
To reproduce, run the following test file:

(ns pe.test-test
(:require [clojure.test :refer :all]))
(defn throwexc [m] (throw (Exception. m)))
(defn throwass [m] (assert false m))
(defn nestexc [] (throwexc "exc"))
(defn nestass [] (throwass "ass"))
(defn nestmapexc [] (map throwexc '("a" "b" "c")))
(defn nestmapass [] (map throwass '("a" "b" "c")))
(deftest exceptions-and-assertions-test
(testing "throwing"
(is (thrown? Exception (throwexc "exc")))
(is (thrown? AssertionError (throwass "ass"))))
(testing "nesting"
(is (thrown? Exception (nestexc)))
(is (thrown? AssertionError (nestass))))
(testing "nesting over map"
(is (thrown? Exception (nestmapexc)))
(is (thrown? AssertionError (nestmapass)))))



 Comments   
Comment by MG [ 01/Apr/14 7:25 AM ]

Clarification: The two assertions in "nesting over map" fails, the other four assertions succeed.

Comment by Nicola Mometto [ 01/Apr/14 7:30 AM ]

map is lazy, the exception is never thrown because the sequence is never realized.
either wrap the map in a dorun or use a doseq instead of a map.
map should not be used for side-effecting





[CLJ-1396] Bad link Created: 01/Apr/14  Updated: 01/Apr/14  Resolved: 01/Apr/14

Status: Closed
Project: Clojure
Component/s: None
Affects Version/s: None
Fix Version/s: None

Type: Defect Priority: Trivial
Reporter: MG Assignee: Unassigned
Resolution: Declined Votes: 0
Labels: None
Environment:

http://richhickey.github.io/clojure/



 Description   

http://richhickey.github.io/clojure/
The issue tracker link points to Assembla.



 Comments   
Comment by Nicola Mometto [ 01/Apr/14 7:31 AM ]

That site is deprecated and so is the repo that's hosting it.
This is the updated one http://clojure.github.io/clojure/

Comment by Alex Miller [ 01/Apr/14 7:45 AM ]

Nicola, I agree with your assessment here but only screeners should be closing tickets, thanks.

Comment by Nicola Mometto [ 01/Apr/14 7:50 AM ]

Alex, sorry, I didn't know this, I will refrain from doing so in the future.





[CLJ-1395] get should deref delay, refs atoms etc Created: 01/Apr/14  Updated: 01/Apr/14  Resolved: 01/Apr/14

Status: Closed
Project: Clojure
Component/s: None
Affects Version/s: None
Fix Version/s: None

Type: Enhancement Priority: Minor
Reporter: Gerrit Jansen van Vuuren Assignee: Unassigned
Resolution: Declined Votes: 0
Labels: None


 Description   

(get (delay {:a 1}) :a)
;; nil

(get {:a 1} :a)
;; 1

The above situation can happen easily and there is no way to refactor or check that all code doing gets is in fact doing deref before doing get. Given that everything is dynamic typing, changing a single value from a map to delay or ref can make large parts of the code fail silently on nil for gets.

get would be more consistent (with what is expected of it) if it was to check for refs, atoms and delays and do a deref.



 Comments   
Comment by Alex Miller [ 01/Apr/14 7:43 AM ]

I think this goes beyond what get should do. In regards to silent failure, that is covered by CLJ-1107 which would cause this to throw an exception instead.





[CLJ-1393] clojure.string/trim doesn't trim null character Created: 28/Mar/14  Updated: 28/Mar/14  Resolved: 28/Mar/14

Status: Closed
Project: Clojure
Component/s: None
Affects Version/s: Release 1.6
Fix Version/s: None

Type: Defect Priority: Minor
Reporter: Ryan Fowler Assignee: Unassigned
Resolution: Declined Votes: 0
Labels: string


 Description   

CLJ-935 changed clojure.string/trim to not trim all the characters less than or equal to \u0020 as Java does.

I noticed this because base64 uses null characters to pad the end of encoding blocks.

Clojure 1.6.0's trim leaves the null character in:
user=> (.length (clojure.string/trim "\u0000"))
1

java.lang.String's trim takes it out:
user=> (.length (.trim "\u0000"))
0

Here are the first 21 unicode characters and what Character/isWhitespace says about them.

(dotimes [n 0x20] (printf "
u%04x - %b\n" n (Character/isWhitespace n)))
\u0000 - false
\u0001 - false
\u0002 - false
\u0003 - false
\u0004 - false
\u0005 - false
\u0006 - false
\u0007 - false
\u0008 - false
\u0009 - true
\u000a - true
\u000b - true
\u000c - true
\u000d - true
\u000e - false
\u000f - false
\u0010 - false
\u0011 - false
\u0012 - false
\u0013 - false
\u0014 - false
\u0015 - false
\u0016 - false
\u0017 - false
\u0018 - false
\u0019 - false
\u001a - false
\u001b - false
\u001c - true
\u001d - true
\u001e - true
\u001f - true



 Comments   
Comment by Alex Miller [ 28/Mar/14 12:27 PM ]

The choice was made in CLJ-935 to consistently define whitespace as Character.isWhitespace() across trim, triml, and trimr. There are many possible ways to define "space" (at least two as we see here). If your trimming needs differ from the standard library, then you'll probably need to define your own functions to trim your data. You can still use Java interop to call String.trim() directly if that happens to match your needs.

Comment by Ryan Fowler [ 28/Mar/14 1:03 PM ]

Indeed, it's an easy workaround to use Java interop once you figure out what your problem is.

It's just unintuitive that the character generally used for string termination isn't trimmed by clojure.string/trim.





[CLJ-1392] AOT vs "var already exists warning" results in NPE Created: 26/Mar/14  Updated: 26/Mar/14  Resolved: 26/Mar/14

Status: Closed
Project: Clojure
Component/s: None
Affects Version/s: Release 1.6
Fix Version/s: None

Type: Defect Priority: Major
Reporter: Aaron Cohen Assignee: Unassigned
Resolution: Duplicate Votes: 0
Labels: aot, compiler


 Description   

I just saw this thread on the cascalog list: https://groups.google.com/forum/#!topic/cascalog-user/Pe5QIpmU0vA

Apparently the "WARNING: some? already refers to: #'clojure.core/some? in namespace: jackknife.seq, being replaced by: #'jackknife.seq/some?" results in a NullPointerException when trying to AOT a namespace that results in the message being output.

Reproducer:

1) lein new aotFail
2) Edit project.clj
;add as appropriate
:aot :all
:dependencies [[org.clojure/clojure "1.6.0"]
[cascalog "2.0.0"]]
2) Add "(:use cascalog.api)" to the ns block of src/aotFail/core.clj
3) lein compile

Output:

Compiling aotFail.core
WARNING: some? already refers to: #'clojure.core/some? in namespace: jackknife.seq, being replaced by: #'jackknife.seq/some?
Exception in thread "main" java.lang.NullPointerException, compiling:(api.clj:1:1)
at clojure.lang.Compiler$InvokeExpr.eval(Compiler.java:3558)
at clojure.lang.Compiler.compile1(Compiler.java:7226)
at clojure.lang.Compiler.compile1(Compiler.java:7216)
at clojure.lang.Compiler.compile(Compiler.java:7292)
at clojure.lang.RT.compile(RT.java:398)
at clojure.lang.RT.load(RT.java:438)
at clojure.lang.RT.load(RT.java:411)
at clojure.core$load$fn__5066.invoke(core.clj:5641)
at clojure.core$load.doInvoke(core.clj:5640)
at clojure.lang.RestFn.invoke(RestFn.java:408)
at clojure.core$load_one.invoke(core.clj:5446)
at clojure.core$load_lib$fn__5015.invoke(core.clj:5486)
at clojure.core$load_lib.doInvoke(core.clj:5485)
at clojure.lang.RestFn.applyTo(RestFn.java:142)
at clojure.core$apply.invoke(core.clj:626)
at clojure.core$load_libs.doInvoke(core.clj:5528)
at clojure.lang.RestFn.applyTo(RestFn.java:137)
at clojure.core$apply.invoke(core.clj:628)
at clojure.core$use.doInvoke(core.clj:5618)
at clojure.lang.RestFn.invoke(RestFn.java:408)
at cascalog2.core$loading_4958auto_.invoke(core.clj:1)
at clojure.lang.AFn.applyToHelper(AFn.java:152)
at clojure.lang.AFn.applyTo(AFn.java:144)
at clojure.lang.Compiler$InvokeExpr.eval(Compiler.java:3553)
at clojure.lang.Compiler.compile1(Compiler.java:7226)
at clojure.lang.Compiler.compile1(Compiler.java:7216)
at clojure.lang.Compiler.compile(Compiler.java:7292)
at clojure.lang.RT.compile(RT.java:398)
at clojure.lang.RT.load(RT.java:438)
at clojure.lang.RT.load(RT.java:411)
at clojure.core$load$fn__5066.invoke(core.clj:5641)
at clojure.core$load.doInvoke(core.clj:5640)
at clojure.lang.RestFn.invoke(RestFn.java:408)
at clojure.core$load_one.invoke(core.clj:5446)
at clojure.core$compile$fn__5071.invoke(core.clj:5652)
at clojure.core$compile.invoke(core.clj:5651)
at user$eval19.invoke(form-init2092370125048380878.clj:1)
at clojure.lang.Compiler.eval(Compiler.java:6703)
at clojure.lang.Compiler.eval(Compiler.java:6693)
at clojure.lang.Compiler.load(Compiler.java:7130)
at clojure.lang.Compiler.loadFile(Compiler.java:7086)
at clojure.main$load_script.invoke(main.clj:274)
at clojure.main$init_opt.invoke(main.clj:279)
at clojure.main$initialize.invoke(main.clj:307)
at clojure.main$null_opt.invoke(main.clj:342)
at clojure.main$main.doInvoke(main.clj:420)
at clojure.lang.RestFn.invoke(RestFn.java:421)
at clojure.lang.Var.invoke(Var.java:383)
at clojure.lang.AFn.applyToHelper(AFn.java:156)
at clojure.lang.Var.applyTo(Var.java:700)
at clojure.main.main(main.java:37)
Caused by: java.lang.NullPointerException
at clojure.lang.Compiler$ObjExpr.emitVar(Compiler.java:4944)
at clojure.lang.Compiler$DefExpr.emit(Compiler.java:437)
at clojure.lang.Compiler.compile1(Compiler.java:7225)
at clojure.lang.Compiler.compile(Compiler.java:7292)
at clojure.lang.RT.compile(RT.java:398)
at clojure.lang.RT.load(RT.java:438)
at clojure.lang.RT.load(RT.java:411)
at clojure.core$load$fn__5066.invoke(core.clj:5641)
at clojure.core$load.doInvoke(core.clj:5640)
at clojure.lang.RestFn.invoke(RestFn.java:408)
at clojure.core$load_one.invoke(core.clj:5446)
at clojure.core$load_lib$fn__5015.invoke(core.clj:5486)
at clojure.core$load_lib.doInvoke(core.clj:5485)
at clojure.lang.RestFn.applyTo(RestFn.java:142)
at clojure.core$apply.invoke(core.clj:626)
at clojure.core$load_libs.doInvoke(core.clj:5524)
at clojure.lang.RestFn.applyTo(RestFn.java:137)
at clojure.core$apply.invoke(core.clj:628)
at clojure.core$use.doInvoke(core.clj:5618)
at clojure.lang.RestFn.invoke(RestFn.java:408)
at cascalog.api$loading_4958auto_.invoke(api.clj:1)
at clojure.lang.AFn.applyToHelper(AFn.java:152)
at clojure.lang.AFn.applyTo(AFn.java:144)
at clojure.lang.Compiler$InvokeExpr.eval(Compiler.java:3553)



 Comments   
Comment by Nicola Mometto [ 26/Mar/14 12:42 PM ]

Duplicate of http://dev.clojure.org/jira/browse/CLJ-1241





[CLJ-1388] equality bug on records created with nested calls to map->record Created: 18/Mar/14  Updated: 29/Aug/14  Resolved: 29/Aug/14

Status: Closed
Project: Clojure
Component/s: None
Affects Version/s: Release 1.6
Fix Version/s: Release 1.7

Type: Defect Priority: Major
Reporter: Nicola Mometto Assignee: Unassigned
Resolution: Completed Votes: 0
Labels: defrecord

Attachments: Text File 0001-FIX-CLJ-1388.patch     Text File CLJ-1388.patch     Text File CLJ-1388-record-equality-and-map-record-factory.patch     Text File CLJ-1388v2.patch    
Patch: Code and Test
Approval: Ok

 Description   

Depending on the type of the map passed to a record map constructor, records will not correctly compare for equality:

user> (defrecord a []) 
user.a
user> (def r1 (map->a {:a 1}))
nil
user> (def r2 (map->a r1))
nil
user> (= r1 r2)  ;; expected => true
false
user> (.__extmap r1)
{:a 1}
user> (.__extmap r2)  ;; expected => {:a 1}
#user.a{:a 1}

Cause: The type of the map passed into the map constructor leaks into the __extmap, affecting equality comparison of the record. This bug was described in this post: https://groups.google.com/forum/#!topic/clojure/iN-SPBaTFUw

Approach: Clean the extmap before putting it into the record constructor.

Patch: CLJ-1388-record-equality-and-map-record-factory.patch

Screened by: Alex Miller



 Comments   
Comment by Steve Miner [ 28/Apr/14 3:06 PM ]

The proposed patch 0001-FIX-CLJ-1388.patch makes every map->Record method pay to copy the argument map every time. However, according to my tests, the problem only occurs with records without any fields. So it should be sufficient to generate the (into {} m#) case only when `fields` is empty. [Update: this is wrong, explained below.]

Comment by Steve Miner [ 28/Apr/14 3:10 PM ]

It would be better to fix the problem in the Java Record/create method, but I couldn't figure out how that worked. On the other hand, this bug seems like a fairly rare edge case so I think my patch is acceptable.

Comment by Alex Miller [ 28/Apr/14 3:23 PM ]

Moving out of Screened due to new patch

Comment by Nicola Mometto [ 28/Apr/14 3:35 PM ]

Steve, the problem doesn't occur with records without any fields, your testing was reporting that only because you are only using one record type.

Here's an example that returns true with my patch, but still returns false with yours.

user=> (defrecord a [a])
user.a
user=> (defrecord b [b])
user.b
user=> (def x1 (map->a {:a 1 :b 2}))
#'user/x1
user=> (def x2 (map->a (map->b {:a 1 :b 2})))
#'user/x2
user=> x1
#user.a{:a 1, :b 2}
user=> x2
#user.a{:a 1, :b 2}
user=> (= x1 x2)
false
user=> (.__extmap x2)
#user.b{:b 2}
Comment by Nicola Mometto [ 28/Apr/14 3:37 PM ]

It should also be noted that the overhead of copying the record map is probably insignificant.

Comment by Nicola Mometto [ 28/Apr/14 3:42 PM ]

I also thought at first to fix the problem either on the /create method or on the 3-arity ctor but given that:

  • a fix there would involve messing with the bytecode emitted and thus would be harder to implement than this simple 1-line patch
  • neither the /create method nor the 3-arity ctor is documented and thus should be considered implementation details

I think patching the map->record function is the best way to go.

Comment by Steve Miner [ 28/Apr/14 3:56 PM ]

Nicola, thanks for the correction. I missed the case with multiple records. I withdrew my patch. I'd still like to find a more finely tuned patch, but first I'll have to improve my tests as you demonstrated.

Comment by Steve Miner [ 29/Apr/14 10:17 AM ]

Attached CLJ-1388-record-equality-and-map-record-factory.patch that checks arg to map->Record for MapEquivalence, uses (into {} m#) when necessary. This makes equiv test work correctly with records as the argument (and other map-like values). Added tests with variety of args to map->Record.

Comment by Steve Miner [ 29/Apr/14 10:46 AM ]

A few comments about the new patch... I think the basic issue is a bad interaction between = for records and the generated Record/create method. Everything works when the interal __extmap is a regular map (MapEquivalence), but it fails if __extmap is another record. I think that's because of equiv calling = on the __extmap's.

The user expects to create a new record using the value of another record because it's just like a map. However, = on records respects the record type so it's not = to a map.

The general work-around is to use (into {} x) on the argument to the map->Record. To meet the user's expectation, that `into` call can be incorporated into the map->Record. But I didn't like the defensive copy as most of the time it's unnecessary – the argument is typically a regular map. The `into` work-around is only necessary if the arg is not a MapEquivalence.

There might be a better way to fix the Record/create method but I couldn't figure it out.

Comment by Nicola Mometto [ 29/Apr/14 1:52 PM ]

Steve's last comment made me realize that the root of the problem is on the record .equiv method, where the extmaps are compared via `=`

This new patch (CLJ-1388.patch) addresses this issue by comparing the extmaps with Utils/equiv rather than `=`, which compares maps in a type-indipendent way.

There's still a case where we need recreate the given map, that is when the given map is not an IPersistentMap but simply a java.util.Map.

Steve, my new patch incorporates my fix and your tests, I modified your patch to include only the tests (that were really comprehensive) since I figured it's fair to keep your authorship on those, let me know if that's a problem with you.

Comment by Steve Miner [ 29/Apr/14 2:10 PM ]

Whatever works for you regarding the tests is fine by me.

Comment by Alex Miller [ 30/Apr/14 12:07 AM ]

It seems weird to me that a record should ever contain another record as its extmap. We should be considering the performance aspect but I'm concerned that not locking down extmap more just invites other weirder problems later.

In CLJ-1388.patch, you mention Utils/equiv in your comment but the patch calls Utils/equals - which did you mean?

Also, that patch currently checks if m# is an IPersistentMap - I can't imagine what case we would want to allow where a valid m# is NOT an IPersistentMap?

Comment by Nicola Mometto [ 30/Apr/14 4:15 AM ]

Alex, the Utils/equiv in my comment is wrong (it's easy to confuse between equiv/equals, sorry), Utils/equals in the patch is the right method to use since it compares in a type agnostic way.

Since __extmap is an implementation detail and is only used internally by defrecord for its methods, I don't think it's going to be a problem whether it's a record or a regular clojure map. (Clojure only requires it to be an IPersistentMap)

Regarding the check for m# being an IPersistentMap, Steve in his tests had a case where the map->record ctor was invoked with a java.util.Map, I went to look into the docs for defrecord and it only mentions that the argument to map->record has to be a "map", it doesn't specify that it has to be a clojure map/IPersistentMap, so it seemed right to allow for java maps too and wrap them in IPersistentMaps internally.

Comment by Steve Miner [ 30/Apr/14 8:27 AM ]

My test with java.util.Map was an extension of the idea that anything map-like could be used to initialize a record. That might be a bridge too far, but my patch was testing for MapEquivalence to handle records so it made sense to allow j.u.Map, etc. With Nicola's latest patch, it's probably unnecessary to support non-IPersistentMaps so map->Record doesn't actually need to change.

Comment by Nicola Mometto [ 30/Apr/14 3:57 PM ]

CLJ-1388v2.patch is like CLJ-1388.patch except it doesn't copy non IPersistentMaps in a clojure map.

To summarize, here's the status of the different patches for this ticket:

  • 0001-FIX-CLJ-1388.patch copies the argument of map->record in a clojure map via `(into {} m#)`, be it already a clojure map, a record, or a java.util.Map
  • CLJ-1388-record-equality-and-map-record-factory.patch adopts the same approach except it only copies the arg of map->record into a clojure map if the arg doesn't satisfy clojure.lang.MapEquivalence
  • CLJ-1388.patch fixes the issue by changing the function that compares __extmaps from `=` (type aware) to `clojure.lang.Utils/equals` (type agnostic), this patch also copies the arg of map->record into a clojure map if the arg doesn't satisfy clojure.lang.IPersistentMap
  • CLJ-1388v2.patch is the same as CLJ-1388.patch except it doesn't copy the arg of map->record into a clojure map if the arg doesn't satisfy clojure.lang.IPersitentMap, thus map->record will not work with bare java.util.Maps (which is the behaviour it has already)
Comment by Alex Miller [ 05/May/14 1:59 PM ]

Are these patches all still in play? Having 4 active patches does not help move a ticket forward.

Can someone re-summarize at this point what questions exist?

Comment by Nicola Mometto [ 06/May/14 5:26 AM ]

0001-FIX-CLJ-1388.patch should be superseded by the other 3 patches since they solve the same problem in a more performant way.

To pick between the other patches, we need to chose which approach to go with.
Patches CLJ-1388.patch and CLJ-1388v2.patch fix the issue in the equiv method of the defrecord, patch CLJ-1388-record-equality-and-map-record-factory.patch fixes the issue in the map->record ctor by converting maps that don't implement MapEquivalence to a clojure map.

I'd go with either CLJ-1388.patch or CLJ-1388v2.patch since they both avoid copying alltoghether in the cases where map->record currently works, while CLJ-1388-record-equality-and-map-record-factory.patch needs to copy the arg into a map if the arg is a custom IPersistentMap or a record.

To pick between CLJ-1388.patch or CLJ-1388v2.patch we need to decide whether or not the current behaviour of map->record to require strictly an IPersistentMap is the way to go: if we decide that it's ok to pass non IPersitentMap maps like java.util.Map to map->record then pick CLJ-1388.patch otherwise CLJ-1388v2.patch

Comment by Alex Miller [ 06/May/14 10:22 AM ]

From brief conversation with Rich, we should not allow arbitrary map types in __extmap so would prefer to force a clean map and rely on standard equality checking. I think CLJ-1388-record-equality-and-map-record-factory.patch is the preferred path based on that, which still seems like it should avoid copying in nearly all common cases.

Comment by Alex Miller [ 23/May/14 11:19 AM ]

Screened specifically CLJ-1388-record-equality-and-map-record-factory.patch - use map as is if it supports MapEquivalence (and can thus be compared under a map) and otherwise dump into a clojure map.





[CLJ-1387] reduce-kv on hash map ignores reduced objects in large maps Created: 18/Mar/14  Updated: 22/Mar/14  Resolved: 22/Mar/14

Status: Closed
Project: Clojure
Component/s: None
Affects Version/s: Release 1.6
Fix Version/s: Release 1.6

Type: Defect Priority: Major
Reporter: Jozef Wagner Assignee: Jozef Wagner
Resolution: Completed Votes: 0
Labels: None
Environment:

JRE 7


Attachments: File clj-1387.diff     File clj-1387-v2.diff     File clj-1387-v3.diff    
Patch: Code and Test
Approval: Ok

 Description   

Larger hash maps have nested INodes. As kvreduce implementations in INodes dereference reduced objects, parent INodes continue to reduce.

user=> (defn test-reduce-kv [m] (reduce-kv (fn [_ k v] (when (== 1 k) (reduced :foo))) nil m))
#'user/test-reduce-kv
user=> (test-reduce-kv (zipmap (range 3) (range 3)))
:foo
user=> (test-reduce-kv (zipmap (range 300) (range 300)))
nil

Dereferencing reduced objects should happen only PersistentHashMap/kvreduce - intermediate nodes should pass the Reduced object along.

Patch: clj-1387-v3.diff
Screened-by:



 Comments   
Comment by Alex Miller [ 18/Mar/14 5:11 PM ]

I updated the patch to use a generative test that will try many combinations of map size and the reduced index to bail out on. This test failed before applying the source patch and passes with it.

Comment by Rich Hickey [ 21/Mar/14 7:33 AM ]

if(root != null){ - return root.kvreduce(f,init); + init = root.kvreduce(f,init); + if(RT.isReduced(init)) + return ((IDeref)init).deref(); }

Turns code that always had a return into code that sometimes does.

Comment by Alex Miller [ 21/Mar/14 9:07 AM ]

Added new version of patch that retains the return flow and doesn't fall through.





[CLJ-1384] clojure.core/set should use transients Created: 15/Mar/14  Updated: 29/Aug/14  Resolved: 29/Aug/14

Status: Closed
Project: Clojure
Component/s: None
Affects Version/s: Release 1.5, Release 1.6
Fix Version/s: Release 1.7

Type: Enhancement Priority: Major
Reporter: Gary Fredericks Assignee: Unassigned
Resolution: Completed Votes: 4
Labels: performance

Attachments: Text File CLJ-1384-p1.patch     Text File CLJ-1384-p2.patch     File set-bench.tar    
Patch: Code
Approval: Ok

 Description   

CLJ-1384-p2 uses transients for both create and createWithCheck. This is consistent with the current implementation for map.

clojure.core/vec calls (more or less) PersistentVector.create(...), which uses a transient vector to build up the result.

clojure.core/set on the other hand, calls PersistentHashSet.create(...), which repeatedly calls .cons on a PersistentHashSet, with all the associated speed/GC issues.

Operation count now w/transients
set 5 1.771924 µs 1.295637 µs
into 5 1.407925 µs 1.402995 µs
set 1000000 2.499264 s 1.196653 s
into 1000000 0.977555 s 1.006951 s

Patch: CLJ-1384-p2.patch
Screened by: Stu



 Comments   
Comment by Gary Fredericks [ 15/Mar/14 10:13 PM ]

PersistentHashSet has six methods for creating sets – one for each combination of {with check, without check} and {array (varargs), List, ISeq}. Each of them does not use transients but could.

I believe clojure.core/set only depends on the (without check, ISeq) version.

Should all of these be changed? Three of them? One of them?

Comment by Andy Fingerhut [ 15/Mar/14 10:21 PM ]

I believe that the 'with check' versions are only intended to be used when reading set literals in Clojure source code, and give an error if there are duplicate elements. If you find examples where those set creation functions are called in other situations, I would be interested to learn about them to find out where my misunderstanding lies, or whether that is a problem with the current code.

If the belief above is correct, I would suggest not changing the 'with check' versions, since their speed isn't as critical.

Comment by Gary Fredericks [ 15/Mar/14 10:23 PM ]

Thanks Andy, I'll submit a patch that changes the three non-checked methods.

Comment by Gary Fredericks [ 15/Mar/14 10:46 PM ]

Attached CLJ-1384-p1.patch, which updates the three non-check create methods.

I also added benchmarks. It's about 2-3 times faster for large collections.

Comment by Ambrose Bonnaire-Sergeant [ 11/Apr/14 11:15 AM ]

Added benchmark suite (set-bench.tar).

FWIW results are similar to gfrederick's on my machine:

Clojure 1.6

Small collections (5 elements)

set averages 1.220601 µs
into averages 1.597991 µs

Large collections (1,000,000 elements)

set averages 2.429066 sec
into averages 1.006249 sec

After transients

Small collections (5 elements)

set averages 999.248325 ns
into averages 1.162889 µs

Large collections (1,000,000 elements)

set averages 1.003792 sec
into averages 889.993185 ms

Add full output to the tar.

Comment by Ghadi Shayban [ 11/Apr/14 11:35 AM ]

CLJ-1192 is related to this, but and Andy seems to be indicating the use of reduce as the means to better performance there.

Comment by Gary Fredericks [ 11/Apr/14 11:41 AM ]

Oh that's a good point about reduce. The difference should only apply to chunked seqs, right? It's worth noting that the benchmarks above used range which creates chunked seqs, so that might be why into looks faster on the large collections?

So this change only makes set act like vec; I think whether either/both of them should use reduce is a different question.





[CLJ-1382] Vector sort order should be specified sufficiently to embrace sort-by-juxt Created: 13/Mar/14  Updated: 15/Mar/14  Resolved: 15/Mar/14

Status: Closed
Project: Clojure
Component/s: None
Affects Version/s: Release 1.5
Fix Version/s: None

Type: Enhancement Priority: Minor
Reporter: Phill Wolf Assignee: Unassigned
Resolution: Completed Votes: 0
Labels: data-structures, documentation, idiom


 Description   

Vectors of equal length sort in a way that seems natural – by
comparing their 0th elements, then their 1st, etc., until something
is different:

user> (def vv '(["c" 9] ["a" 100] ["a" 33] ["b" 8]))
#'user/vv
user> (sort vv)
(["a" 33] ["a" 100] ["b" 8] ["c" 9])

This property enables a blisteringly wonderful idiom for sorting
records by multiple keys:

user> (def mm (->> vv (map (fn [[p q]] {:p p :q q}))))
#'user/mm
user> (sort-by (juxt :p :q) mm)
({:p "a", :q 33} {:p "a", :q 100} {:p "b", :q 8} {:p "c", :q 9})

The sort-by-juxt idiom was described on briancarper.net[1], where it
was refined for Clojure 1.1 by Malcolm Sparks. Andy Fingerhut has
also written about it, thoroughly.[2]

Such lore gives it the odor of respectability, but the sort-by-juxt
idiom takes liberties beyond the documented behavior ("contract") of
vectors. APersistentVector indulges the idiom, but the clojure.org
Data Structures page does not say how vectors should compare.

The vector specification should be bolstered with enough traits of
vectors' sorting behavior to make the sort-by-juxt idiom safe to use
wherever Clojure might be implemented.

[1] http://briancarper.net/blog/488/sort-a-clojure-map-by-two-or-more-keys

[2] https://github.com/jafingerhut/thalia/blob/master/doc/other-topics/comparators.md



 Comments   
Comment by Alex Miller [ 13/Mar/14 9:52 PM ]

I don't understand what this ticket is asking for.

Comment by Andy Fingerhut [ 13/Mar/14 10:32 PM ]

It sounds like he is asking that the doc of clojure.core/compare say that vectors of equal length are compared in lexicographic order.

Comment by Phill Wolf [ 15/Mar/14 1:07 PM ]

"(sort-by (juxt" relies on a feature of vectors that the Clojure documentation does not guarantee. But using juxt in this way is part of the cultural fabric and helps make concise programs that "read like a definition" of the problem, to quote Halloway in "Programming Clojure". Therefore, let's document the sort order of equal-length vectors. I looked for this information on the Data Structures page, which has a section on vectors.

Comment by Alex Miller [ 15/Mar/14 11:11 PM ]

I added a line to the Vectors section on the Data Structures (http://clojure.org/data_structures) page: "Vectors are compared first by length, then each element is compared in order."





[CLJ-1378] Hints don't work with #() form of function Created: 11/Mar/14  Updated: 29/Aug/14  Resolved: 29/Aug/14

Status: Closed
Project: Clojure
Component/s: None
Affects Version/s: None
Fix Version/s: Release 1.7

Type: Enhancement Priority: Major
Reporter: Roy Varghese Assignee: Unassigned
Resolution: Completed Votes: 1
Labels: interop, typehints

Attachments: File clj-1378.diff     File clj-1378-v2.diff    
Patch: Code and Test
Approval: Ok

 Description   

Example showing how a local fn can be hinted but an anonymous function cannot:

;; OK
user> (let [ex (java.util.concurrent.Executors/newFixedThreadPool 1)
            f (fn [])]
        (.submit ex ^Runnable f))
nil
;; ERROR - this should work the same as the previous
user> (let [ex (java.util.concurrent.Executors/newFixedThreadPool 1)]
        (.submit ex #()))
CompilerException java.lang.IllegalArgumentException: More than one matching method found: submit, compiling:(/private/var/folders/7r/_1fj0f517rgcxwx79mn79mfc0000gn/T/form-init7901279404687292754.clj:3:9)

Cause: Functions have metadata, but Compiler does not look in them for type hints. Var expressions and local bindings use :tag metadata to override return of getJavaClass(). Compiler parses #() into a FnExpr, which always return AFunction as its class.

Proposed: Change FnExpr.getJavaClass() to return tag as type if it is available.

Patch: clj-1378-v2.diff

Screened by: Alex Miller



 Comments   
Comment by Jozef Wagner [ 12/Mar/14 4:03 AM ]

Functions do have metadata, but Compiler does not look in them for type hints.

user=> (with-meta #() {:foo :bar})
#<clojure.lang.AFunction$1@779325ee>

When compiler is determining which native method to use, it matches method signature with classes of given args. There is a getJavaClass() method in Compiler.java which returns a class for given expression. Vars expressions and local bindings use :tag metadata to override this class, but most other expressions don't. Compiler parses #() into a FnExpr, which always return AFunction as its class.

Most of time this approach is OK, as AFunction implements Runnable and Callable so there is no need for type hint. However, in this particular case, there are overrides for both Runnable and Callable, and as AFunction can be either of them, the expression is ambiguous.

Comment by Jozef Wagner [ 12/Mar/14 4:17 AM ]

Patch added, following expression will now run without error

(.submit (java.util.concurrent.Executors/newCachedThreadPool) ^Runnable #())
Comment by Alex Miller [ 12/Mar/14 9:34 AM ]

Could you add a test to the patch?

Comment by Jozef Wagner [ 12/Mar/14 2:53 PM ]

Attached patch clj-1378-v2.diff which contains both fix and test.





[CLJ-1377] java.lang.IllegalArgumentException: More than one matching method found on calling ExecutorService.submit from let instead of using Var. Created: 11/Mar/14  Updated: 12/Mar/14  Resolved: 11/Mar/14

Status: Closed
Project: Clojure
Component/s: None
Affects Version/s: Release 1.5
Fix Version/s: None

Type: Defect Priority: Major
Reporter: Roy Varghese Assignee: Unassigned
Resolution: Declined Votes: 0
Labels: None


 Description   

Example from Joy Of Clojure doesn't work when Executor service pool is used in (let )

https://groups.google.com/forum/#!msg/clojure/asEXM2uHyxw/aK4rrKOKs1YJ



 Comments   
Comment by Alex Miller [ 11/Mar/14 3:47 PM ]

In the let case, the pool will be tagged with the proper type so the ambiguity is detected.

In the def case, the pool will be seen as an object and the compiler is just deferring to reflection at runtime to figure it out. If you turn on warn-on-reflection, you'll see a reflection warning in this case. Reflection is just picking the first one that matches in that case. If you type hinted the def case, you'd see the same error.

I don't think there is a bug here.

Comment by Roy Varghese [ 11/Mar/14 5:08 PM ]

Actually, I did turn on warn-on-reflection, but didn't see any messages.

I think this creates an element of surprise and uncertainty.

(def A (let [x (java.util.concurrent.Executors/newFixedThreadPool 5)] x))

How would (.submit A ...) behave?

Comment by Alex Miller [ 11/Mar/14 11:15 PM ]

When I ran the prior example, I saw a reflection warning.

In your new example, I would expect x to be typed as a ThreadExecutorPool, A to be typed as an Object, and .submit to get called reflectively (successfully) and produce a reflection warning. It's effectively no different than the def example in the post.

What are you suggesting is the bug and what should happen? Reflector could throw an error when it finds multiple matches but that would certainly break code that is currently working like the example you gave, so I doubt we would consider such a change at this point. There really is an ambiguity here, so I think a type hint is required to make it work unambiguously in either case.

Comment by Roy Varghese [ 12/Mar/14 10:49 AM ]

The bug is that type hints are required in one case, and not required in the other case because of implementation details, unless its documented in the language that (def) and (let) hold different types of Vars.

I the example above, I would expect A and x to refer to the same object, and thus contain the type information. Or not. Without reading the implementation, either case seems possible.

Agree, its a breaking change, but that's a different consideration.

Comment by Roy Varghese [ 12/Mar/14 10:50 AM ]

BTW..trying to fix with hints is what led to CLJ-1378.

Comment by Alex Miller [ 12/Mar/14 11:39 AM ]

x is locally bound to an object (there is no Var here) - the type information is on the local binding, inferred from the type of the expression. In the compiled form, the let expression does know the return type of the evaluated let (if there were multiple branches, there might be many possible return types). The def creates a Var that holds the result of evaluating the let. At compilation time, the type of the result of the let is unknown so is assumed to be Object.

So, x is a local binding and A is a Var. Even if they both refer to the same object, they do so in different contexts using different information. The type hinting and binding behavior in let is described on the special forms page http://clojure.org/special_forms. Remember too that Clojure is a dynamic language - while the Var A might hold a ThreadPoolExecutor instance now, it might hold something else later.

CLJ-1378 is useful - that's phrased more as a problem we can solve and there's a reasonable patch there.

Comment by Alex Miller [ 12/Mar/14 11:55 AM ]

"there is no Var here" == "there is no Var at this point"
"the let expression does know the return type" should have been "does NOT know"





[CLJ-1374] Make PersistentQueue implement List Created: 09/Mar/14  Updated: 31/Mar/14  Resolved: 31/Mar/14

Status: Resolved
Project: Clojure
Component/s: None
Affects Version/s: None
Fix Version/s: None

Type: Enhancement Priority: Minor
Reporter: Andy Fingerhut Assignee: Unassigned
Resolution: Duplicate Votes: 0
Labels: None

Attachments: File clj-1374-1.diff    
Patch: Code and Test

 Description   

Most ordered Clojure collections like lists, vectors, and lazy seqs implement java.util.List, and thus .equals can be true between values of those types and other collections implementing java.util.List, like java.util.Vector and java.util.ArrayList.

Clojure PersistentQueue seems to be the odd man out here, in that it implements Collection but not List, and thus while it can be .equals to Clojure lists, vectors, and lazy seqs, it cannot be .equals to other collections implementing java.util.List.

user=> (instance? java.util.List '())
true
user=> (instance? java.util.List (lazy-seq))
true
user=> (instance? java.util.List [])
true
user=> (instance? java.util.List (vector-of :long))
true
user=> (instance? java.util.List clojure.lang.PersistentQueue/EMPTY)
false

user=> (= '() (java.util.ArrayList.))
true
user=> (= (lazy-seq) (java.util.ArrayList.))
true
user=> (= [] (java.util.ArrayList.))
true
user=> (= (vector-of :long) (java.util.ArrayList.))
true
user=> (= clojure.lang.PersistentQueue/EMPTY (java.util.ArrayList.))
false


 Comments   
Comment by Andy Fingerhut [ 09/Mar/14 6:29 PM ]

Patch clj-1374-1.diff is written assuming that CLJ-1372 patch clj-1372-2.diff or very similar has been committed, because of the tests modified, not because of the change of PersistentQueue to implement the java.util.List interface. I can update this patch as desired if that change does not go in.

Comment by Andy Fingerhut [ 09/Mar/14 6:33 PM ]

Ugh. The subject is definitely a duplicate of CLJ-1059, which I should have checked for before creating this ticket. I will compare patches to see how the approaches compare. Mine is probably a poor substitute for that one, but the tests I add may still be useful to keep in a patch for CLJ-1059.

Comment by Andy Fingerhut [ 31/Mar/14 5:29 PM ]

Problem description is a duplicate of CLJ-1059. Even the patch (independently developed) is nearly the same as the patch with a name beginning with "001" attached to CLJ-1059.





[CLJ-1370] ((println)) should throw CompilerException instead of NPE? Created: 06/Mar/14  Updated: 06/Mar/14  Resolved: 06/Mar/14

Status: Closed
Project: Clojure
Component/s: None
Affects Version/s: Release 1.5
Fix Version/s: None

Type: Defect Priority: Minor
Reporter: The Alchemist Assignee: Unassigned
Resolution: Declined Votes: 0
Labels: bug


 Description   

How to Reproduce

=> ((println))

NullPointerException   hello.core/eval4117 (NO_SOURCE_FILE:1)

What's Wrong?

I might be completely wrong, in which case don't hesitate to close this defect, but I wish this would throw a CompilerException with an IllegalArgumentException, just like ((nil)):

=> ((nil))
CompilerException java.lang.IllegalArgumentException: Can't call nil, compiling:(NO_SOURCE_PATH:1:2)


 Comments   
Comment by Alex Miller [ 06/Mar/14 12:37 PM ]

There is no compilation error here. The error occurs during evaluation.

user> (defn x [] ((println)))  ;; compiles just fine
#'user/x
user> (x)   ;; fails in evaluation
NullPointerException   user/x (NO_SOURCE_FILE:1)

The error is thrown when trying to evaluate (nil) where NullPointerException is a perfectly valid error.





[CLJ-1369] CLJ-738 is marked Closed is not implemented Created: 04/Mar/14  Updated: 04/Mar/14  Resolved: 04/Mar/14

Status: Closed
Project: Clojure
Component/s: None
Affects Version/s: Release 1.5, Release 1.6
Fix Version/s: None

Type: Defect Priority: Minor
Reporter: David Welte Assignee: Unassigned
Resolution: Completed Votes: 0
Labels: None
Environment:

OS X, Java 6



 Description   

CLJ-738 is marked Closed but the attached patch is has not been applied and both Clojure 1.5.1 and 1.6.0-beta2 exhibit the bad behavior listed in CLJ-738. The issue that CLJ-738 is that (<= (Double. Double/NaN) 1) evaluates to true while (<= Double/NaN 1) evaluates to false. See CLJ-738 for many details.



 Comments   
Comment by Alex Miller [ 04/Mar/14 3:20 PM ]

Thanks for letting us know about this - I concur that 738 was incorrectly closed without being applied and I have resurrected that ticket. I am closing this one. In the future, feel free to just comment on a ticket directly, or better (for a closed ticket), comment on one of the mailing lists.





[CLJ-1365] New collection hash functions are too slow Created: 20/Feb/14  Updated: 11/Mar/14  Resolved: 11/Mar/14

Status: Closed
Project: Clojure
Component/s: None
Affects Version/s: Release 1.6
Fix Version/s: Release 1.6

Type: Enhancement Priority: Major
Reporter: Alex Miller Assignee: Unassigned
Resolution: Completed Votes: 0
Labels: collections

Attachments: Text File clj-1365-v1.patch     Text File clj-1365-v2.patch     Zip Archive testclj1365.zip    
Patch: Code
Approval: Ok

 Description   

As reported ( https://groups.google.com/d/msg/clojure-dev/t6LAmVe-RLM/ekLTKxYfU5UJ ) by Mark Engelberg, the new collection hashing functions are slower than invoking the Murmur3 functions directly. See the attached zip for performance tests.

Approach: Made mix-collection-hash, hash-ordered-coll, and hash-unordered-coll use primitive type hints to avoid the bulk of the time.

Patch: clj-1365-v2.patch

Screened by:



 Comments   
Comment by Alex Miller [ 20/Feb/14 11:26 AM ]

Added to 1.6 release.

Comment by Alex Miller [ 20/Feb/14 12:40 PM ]

Made hash functions inline for performance.

Comment by Rich Hickey [ 20/Feb/14 7:55 PM ]

Reported where?

This looks like bad benchmarking.

(dotimes [_ 10] (let [x 1 y 1] (time (dotimes [n 1000000000] (clojure.lang.Murmur3/mixCollHash x y)))))

and

(dotimes [_ 10] (let [x 1 y 1] (time (dotimes [n 1000000000] #_(clojure.lang.Murmur3/mixCollHash x y)))))

take the same time on my machine.

I'd need to see tests where the return was definitely used, it seems this is just more easily ignored by hotspot when not used.

We probably only need to hint count and the return for decent results.

Comment by Alex Miller [ 20/Feb/14 8:55 PM ]

It was reported by Mark Engelberg in his Instaparse rework - he observed these calls taking noticeably longer and overall times 10-20% down. I will ask him to chime in here.

Comment by Rich Hickey [ 04/Mar/14 8:44 AM ]

Could someone please test hinting hint count and the return? I'd hate for the answer to anyone's perf issues be inlining.

Comment by Alex Miller [ 04/Mar/14 9:06 AM ]

I will provide some more data for consideration of the options.

Comment by Alex Miller [ 04/Mar/14 11:07 AM ]

Test project for different variants

Comment by Alex Miller [ 04/Mar/14 11:11 AM ]

Attached a test project with different variants for testing and better benchmarking. To run:

unzip testclj1365.zip
cd clj1365
lein uberjar
java -server -cp target/clj1365-0.1.0-SNAPSHOT-standalone.jar clj1365.core

Results:

mix-collection-hash original
"Elapsed time: 57.777 msecs"
"Elapsed time: 18.034 msecs"
"Elapsed time: 20.591 msecs"
"Elapsed time: 25.179 msecs"
"Elapsed time: 21.781 msecs"
mix-collection-hash hints
"Elapsed time: 14.983 msecs"
"Elapsed time: 8.871 msecs"
"Elapsed time: 8.793 msecs"
"Elapsed time: 8.92 msecs"
"Elapsed time: 8.873 msecs"
mix-collection-hash inline
"Elapsed time: 10.04 msecs"
"Elapsed time: 7.117 msecs"
"Elapsed time: 7.306 msecs"
"Elapsed time: 7.324 msecs"
"Elapsed time: 7.175 msecs"
Murmur3/mixCollHash
"Elapsed time: 9.522 msecs"
"Elapsed time: 7.288 msecs"
"Elapsed time: 7.397 msecs"
"Elapsed time: 7.364 msecs"
"Elapsed time: 7.345 msecs"

From these results, I infer that the unhinted version is slower (21 ms) than a static call (7 ms). Inlining gives you same perf as static. Hinting inputs and return gives almost the same perf (9 ms).





[CLJ-1363] Field access via .- in reflective case does not work Created: 18/Feb/14  Updated: 28/Feb/14  Resolved: 28/Feb/14

Status: Closed
Project: Clojure
Component/s: None
Affects Version/s: Release 1.5
Fix Version/s: Release 1.6

Type: Defect Priority: Major
Reporter: Alex Miller Assignee: Unassigned
Resolution: Completed Votes: 0
Labels: interop

Attachments: Text File clj-1363-v1.patch     Text File clj-1363-v2.patch     Text File clj-1363-v3.patch    
Patch: Code and Test
Approval: Ok

 Description   

The (.-foo instance) syntax will properly resolve to either a field or a no-arg method if the type of instance is known. However, in the reflective case, it will only resolve to a method. This behavior should match the non-reflective case. The method case always be forced by using (. foo (method)).

user> (definterface I (a []))
user.I
user> (deftype T [a] I (a [_] "method"))
user.T
user> (def t (->T "field"))
#'user/t
user=> (. ^T t a)  ;; as expected (prefer method)
"method"
user=> (. ^T t -a) ;; as expected (prefer field)
"field"
user> (. t a)      ;; as expected (prefer method)
"method"
user> (. t -a)     ;; WRONG - should return "field"
"method"

Approach: This case falls into Reflector.invokeNoArgInstanceMember() (this is the only place this method is used). InstanceFieldExpr now takes another flag (requireField) which will be set to true if "-field" and false if "field". InstanceFieldExpr will invoke (or emit) a call to Reflector.invokeNoArgInstanceMember() which now takes the same flag. If the flag is set to true, it first looks only for a field, otherwise it looks for a method and falls back to field which throws an error if necessary. I added a new invokeNoArgInstanceMember() with an arity to match the old arity - existing bytecode compiled on older Clojure versions will be trying to call this arity.

Patch: clj-1363-v3.patch

Screened by:



 Comments   
Comment by Rich Hickey [ 20/Feb/14 7:24 PM ]

You can't change the semantics of invokeNoArgInstanceMember - they are correct when not using '-'. We need to feed the info that '-' was used through InstanceFieldExpr and make field-first conditional on that.

Comment by Alex Miller [ 21/Feb/14 5:42 AM ]

Updated with new patch to thread this case through InstanceFieldExpr.

Comment by Andy Fingerhut [ 28/Feb/14 6:02 AM ]

A patch for this ticket has been committed as part of Clojure 1.6.0-beta2: https://github.com/clojure/clojure/commit/5fda6cb262d1807566ecadd3af9aaafb58ee5544

It appears this ticket could be closed now.





[CLJ-1362] Reduce broken on some primitive vectors Created: 18/Feb/14  Updated: 29/Aug/14  Resolved: 29/Aug/14

Status: Closed
Project: Clojure
Component/s: None
Affects Version/s: Release 1.3, Release 1.4, Release 1.5, Release 1.6
Fix Version/s: Release 1.7

Type: Defect Priority: Major
Reporter: Nathan Davis Assignee: Unassigned
Resolution: Completed Votes: 0
Labels: collections

Attachments: Text File clj-1362-v1.patch    
Patch: Code and Test
Approval: Ok

 Description   

In some cases, reduce over a sequence from a primitive vector created with vector-of will return incorrect answers:

user=> (into [] (drop 32 (into [] (range 33))))
[32]
user=> (into [] (drop 32 (into (vector-of :int) (range 33))))
[0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32]

Second call should return [32] just like the first one.

Cause: VecSeq (seq on primitive Vec obtained with vector-of) maintains two flags: i is the total number of elements prior to the current node in this seq. offset is the offset in the current anode. When using internal-reduce on a VecSeq, the starting index for the reduce was using offset and ignoring i.

Solution: Use (+ i offset) as the starting index.

Patch: clj-1362-v1.patch

Screened by:



 Comments   
Comment by Alex Miller [ 18/Feb/14 10:18 PM ]

We did some debugging on this at the St. Louis Clojure Meetup tonight and suspect the problem is happening when drop walks through the chunked seq over the vector. Specifically, in the VecSeq's implementation of IChunkedSeq.chunkedNext() at https://github.com/clojure/clojure/blob/master/src/clj/clojure/gvec.clj#L116 particularly the offset 0 at the end.

Comment by Alex Miller [ 19/Feb/14 2:41 PM ]

Upon further review, the VecSeq seems to be created properly during chunking. The real issue is in internal-reduce where the starting index is improperly computed.

Comment by Stuart Sierra [ 25/Apr/14 1:05 PM ]

Screened.





[CLJ-1359] Fix changelog typos for 1.6 Created: 18/Feb/14  Updated: 25/Feb/14  Resolved: 25/Feb/14

Status: Closed
Project: Clojure
Component/s: None
Affects Version/s: Release 1.6
Fix Version/s: Release 1.6

Type: Enhancement Priority: Minor
Reporter: Alex Miller Assignee: Alex Miller
Resolution: Completed Votes: 0
Labels: None

Attachments: Text File clj-1359.patch    
Patch: Code
Approval: Ok

 Description   

Some reported problems in the 1.6 changelog:

1) two different issues are both called CLJ-935
2) two issues that are probably different are both called CLJ-1328
3) "Make range consistently return () with a step of 0." This is slightly incorrect. Range now consistently returns an infinite sequence of start with a 0 step.

Patch: clj-1359.patch - updated for these issues, may want to hold this and update for any post-beta1 changes too.






[CLJ-1356] clojure.org/agents calls out deprecated funcs Created: 17/Feb/14  Updated: 17/Feb/14  Resolved: 17/Feb/14

Status: Closed
Project: Clojure
Component/s: None
Affects Version/s: None
Fix Version/s: None

Type: Enhancement Priority: Trivial
Reporter: Ryan Macy Assignee: Unassigned
Resolution: Completed Votes: 0
Labels: agents, documentation, website


 Description   

""If any exceptions are thrown by an action function, no nested dispatches will occur, and the exception will be cached in the Agent itself. When an Agent has errors cached, any subsequent interactions will immediately throw an exception, until the agent's errors are cleared. Agent errors can be examined with agent-errors and cleared with clear-agent-errors.""

While it is true and those functions will do what it describes, they are listed as deprecated in the docs. Should we update this paragraph to reflect usage of `agent-error` and `restart-agent` instead?



 Comments   
Comment by Ryan Macy [ 17/Feb/14 11:38 AM ]

I hope I put this in the right place!

Comment by Alex Miller [ 17/Feb/14 12:32 PM ]

Yep, thanks!

Comment by Alex Miller [ 17/Feb/14 12:40 PM ]

Fixed.





[CLJ-1355] Restore symbol and keyword hashCode to avoid breaking compiled case expressions Created: 17/Feb/14  Updated: 27/Feb/14  Resolved: 27/Feb/14

Status: Closed
Project: Clojure
Component/s: None
Affects Version/s: Release 1.6
Fix Version/s: Release 1.6

Type: Defect Priority: Major
Reporter: Alex Miller Assignee: Unassigned
Resolution: Completed Votes: 0
Labels: None
Environment:

clojure 1.6.0-beta1


Attachments: File clj-1355-cached.diff     Text File clj-1355-v2.patch    
Patch: Code
Approval: Ok

 Description   

case expressions compiled in Clojure 1.5 are broken if run with Clojure 1.6 where hashCode behavior has diverged from hasheq. In particular, Symbol and Keyword fall into this category.

Approach: Cache both hashCode (with 1.5 calculation) and hasheq (new 1.6 calculation) in Symbol and just hasheq in Keyword. In 1.5, these were the same and case expressions compiled with 1.5 will store the old hash calculation. In 1.6, the hashCode of an expression will be used for comparison.

I tested this by AOT compiling a project in clojure 1.5.1 with this function:

(defn check [v]
  (case v
    :k "keyword match"
    'k "symbol match"
    "k" "string match"
    "no match"))

I verified that (check :k) and (check 'v) incorrectly returned "no match" on Clojure 1.6.0-beta1. I then verified that they returned "keyword match" and "symbol match" respectively on Clojure 1.6.0-master with this patch applied.

Patch: clj-1355-v2.patch



 Comments   
Comment by Alex Miller [ 17/Feb/14 9:38 AM ]

Add patch that caches a new hash field for both Symbol and Keyword that retains Clojure 1.5 computations.

Comment by Alex Miller [ 17/Feb/14 10:42 AM ]

There is a concern here that we are adding a new int field to every Symbol and Keyword (and keyword holds a symbol, so it's really 2 for each keyword).

Comment by Rich Hickey [ 20/Feb/14 7:27 PM ]

I don't think we need to cache in keyword, it's just an add

Comment by Alex Miller [ 20/Feb/14 9:24 PM ]

Updated patch to only cache hashCode in symbol and compute in Keyword.





[CLJ-1354] Make the class APersistentVector.SubVector public Created: 17/Feb/14  Updated: 25/Feb/14  Resolved: 25/Feb/14

Status: Closed
Project: Clojure
Component/s: None
Affects Version/s: None
Fix Version/s: Release 1.6

Type: Enhancement Priority: Major
Reporter: Michał Marczyk Assignee: Unassigned
Resolution: Completed Votes: 0
Labels: None

Attachments: Text File 0001-CLJ-1354-make-APersistentVector.SubVector-public.patch    
Patch: Code
Approval: Ok

 Description   

The patch marks APersistentVector.SubVector public so that it can be used as a type hint for reflection-free access to subvec internals. I missed this in CLJ-1150.

core.rrb-vector needs access to the internals of the built-in vector types in order to support their efficient concatenation and (true, RRB-style) slicing.

Patch: 0001-CLJ-1354-make-APersistentVector.SubVector-public.patch

Screened by: Alex Miller



 Comments   
Comment by Michał Marczyk [ 17/Feb/14 7:30 AM ]

This is the exact spot where I'm trying to get at SubVector internals in core.rrb-vector:

https://github.com/clojure/core.rrb-vector/blob/core.rrb-vector-0.0.10/src/main/clojure/clojure/core/rrb_vector/rrbt.clj#L976

With 1.6.0-alpha3, {{(fv/catvec (subvec [0 1 2 3] 1 2) [:foo])}} results in IllegalAccessError tried to access class clojure.lang.APersistentVector$SubVector from class clojure.core.rrb_vector.rrbt$eval2476$fn__2477 clojure.core.rrb-vector.rrbt/eval2476/fn--2477 (rrbt.clj:978). With this patch applied, it works as expected, returning [1 :foo].





[CLJ-1353] Prevent test app from appearing in Mac OS X dock Created: 16/Feb/14  Updated: 27/Feb/14  Resolved: 27/Feb/14

Status: Closed
Project: Clojure
Component/s: None
Affects Version/s: Release 1.6
Fix Version/s: Release 1.6

Type: Enhancement Priority: Trivial
Reporter: Steve Miner Assignee: Unassigned
Resolution: Completed Votes: 1
Labels: None
Environment:

Mac OS X


Attachments: Text File CLJ-1353-no-mac-dock.patch     Text File CLJ-1353-v2.patch     Text File clj-1353-v3.patch     Text File clj-1353-v4.patch    
Patch: Code
Approval: Ok

 Description   

During a local ant build of Clojure (tested with master after release of 1.6.0-beta1), the script/run_test.clj is executed. As a side-effect on the Mac, the Java coffee cup app icon is placed in the Dock, and the test app becomes the active application on the desktop. This is slightly annoying.

Even with this property set, activation of awt causes focus to switch temporarily then switch back (at least on Mac).

Solution: Set the following properties during the build:

java.awt.headless=true

Patch: clj-1353-v4.patch



 Comments   
Comment by Steve Miner [ 16/Feb/14 1:39 PM ]

CLJ-1353-no-mac-dock.patch adds a line to script/run_tests.clj to set the apple.awt.UIElement property. This prevents the test app from appearing in the Dock on Mac OS X.

Comment by Steve Miner [ 16/Feb/14 2:18 PM ]

CLJ-1349 might rearrange the affected source, which would force an update to this patch. Still just a one-liner so maybe it could be added to the patch for CLJ-1349.

Comment by Alex Miller [ 16/Feb/14 5:20 PM ]

I also find this highly annoying.

Comment by Andy Fingerhut [ 17/Feb/14 11:21 AM ]

Patch CLJ-1353-v2.patch is identical to Steve Miner's CLJ-1353-no-mac-dock.patch, except it adds another line to build.xml to set the property there, too. At least on my Mac systems, an icon appears in the dock during compilation, not only during testing, and this added line prevents that. Keeps Steve as the patch author.

Comment by Andy Fingerhut [ 17/Feb/14 11:22 AM ]

I tested CLJ-1353-v2.patch on a Linux system, too, and at least the messages that appear on the console during the execution of "ant" are identical with and without this patch, so no extra warnings appear due to these extra properties being set that are likely ignored by the JVM there.

Comment by Steve Miner [ 17/Feb/14 1:45 PM ]

Adding the sysproperty setting to build.xml sounds like a good idea. Thanks.

Comment by Alex Miller [ 18/Feb/14 1:42 PM ]

I found that even with this property, I still see focus change, then come back, during the build due to the activation of awt. Adding the java.awt.headless=true property made that stop. I updated the patch in both locations and now on Mac focus is never stolen during the build.

FYI: If you see the Java "Allow incoming network connections?" dialog on Mac during the tests in response to creating the Sockets in test/clojure/test_clojure/java/io.clj (test-socket-iofactory), this procedure makes that stop:

http://techblog.willshouse.com/2012/10/17/how-to-allow-java-in-the-firewall-on-os-x-mountain-lion/

Beware tracking down the correct version of Java (for example the 1.6 version) instead of the easier to find 1.7 version - the permissions are separate for each version.

Comment by Andy Fingerhut [ 24/Feb/14 2:35 PM ]

In my testing, the addition of the java.awt.headless=true properties in both build.xml and src/script/run_tests.clj was sufficient to avoid the additional icon appearing, and also avoiding any change of focus. Setting apple.awt.UIElement=true appears to be unnecessary (but harmless).

Comment by Steve Miner [ 24/Feb/14 3:28 PM ]

Yes, it seems that java.awt.headless=true is a better, more general solution for the build process. I think apple.awt.UIElement would be appropriate if you actually needed AWT for user interaction but didn't want the dock icon.

Comment by Alex Miller [ 25/Feb/14 11:33 AM ]

Added v4 patch that only sets java.awt.headless=true and drops the apple property.





[CLJ-1352] clojure.test/test-vars runs :each fixtures for vars without :test metadata Created: 14/Feb/14  Updated: 25/Feb/14  Resolved: 25/Feb/14

Status: Closed
Project: Clojure
Component/s: None
Affects Version/s: Release 1.6
Fix Version/s: Release 1.6

Type: Defect Priority: Major
Reporter: Toby Crawley Assignee: Unassigned
Resolution: Completed Votes: 0
Labels: None

Attachments: File tcrawley-fixtures-with-non-test-vars-2014-02-14.diff    
Patch: Code and Test
Approval: Ok

 Description   

The patch for CLJ-866 introduced a bug with :each fixtures and non-test vars: the fixtures are invoked for every var, not just ones with :test metadata.

Patch: tcrawley-fixtures-with-non-test-vars-2014-02-14.diff

Screened by: Alex Miller



 Comments   
Comment by Andy Fingerhut [ 24/Feb/14 2:37 PM ]

The patch for this ticket has been committed: https://github.com/clojure/clojure/commit/919a7100ddf327d73bc2d50d9ee1411d4a0e8921

but the ticket has not yet been closed.

Comment by Alex Miller [ 24/Feb/14 3:09 PM ]

yeah, I noticed that too. I was going to mention it to Stu the next time we talked.





[CLJ-1350] (/ 1 3) returns Ratio 31/3 Created: 14/Feb/14  Updated: 14/Feb/14  Resolved: 14/Feb/14

Status: Closed
Project: Clojure
Component/s: None
Affects Version/s: Release 1.5
Fix Version/s: None

Type: Defect Priority: Minor
Reporter: Justin Hanekom Assignee: Unassigned
Resolution: Not Reproducible Votes: 0
Labels: None
Environment:

openSuSE 13.1



 Description   

(/ 1 3) incorrectly returns the Ratio 31/3. Other numbers, such as (/ 1 4), work as expected. This could be worked around by using Java interop, but I don't think / it is functioning correctly in this case.



 Comments   
Comment by Justin Hanekom [ 14/Feb/14 1:25 AM ]

$ lein version
Leiningen 2.3.4 on Java 1.7.0_51 OpenJDK 64-Bit Server VM

Comment by Nicola Mometto [ 14/Feb/14 5:35 AM ]

I cannot reproduce this on clojure 1.5.1 or 1.6.0-master-SNAPSHOT

Comment by Alex Miller [ 14/Feb/14 8:14 AM ]

I also could not reproduce on 1.5 or 1.6. Please provide more information on your Clojure environment ({\*clojure-version\*} and also verify that you're not seeing printing obscuring your repl output or something.

user=> *clojure-version*
{:major 1, :minor 5, :incremental 1, :qualifier nil}
user=> (def x (/ 1 3))
#'user/x
user=> (numerator x)
1
user=> (denominator x)
3
user=> x
1/3
Comment by Justin Hanekom [ 14/Feb/14 12:01 PM ]

Today I'm unable to reproduce this behavior, although yesterday I could!? I'm so embarrassed :-*>

Thanks for closing.





[CLJ-1348] Add functions for external collection hashing Created: 10/Feb/14  Updated: 14/Feb/14  Resolved: 14/Feb/14

Status: Closed
Project: Clojure
Component/s: None
Affects Version/s: Release 1.6
Fix Version/s: Release 1.6

Type: Enhancement Priority: Major
Reporter: Alex Miller Assignee: Alex Miller
Resolution: Completed Votes: 0
Labels: None

Attachments: Text File clj-1348-1.patch     Text File clj-1348-2.patch     Text File clj-1348-3.patch    
Patch: Code
Approval: Ok

 Description   

External collections wishing to implement hasheq appropriately must follow the advice at http://clojure.org/data_structures#hash. To simplify the implementation (and avoid unwanted dependencies on the internal Murmur3 class), add two new functions hash-ordered-coll and hash-unordered-coll that provide a proper collection hasheq over entire collections.

Patch: clj-1348-3.patch (fixes [k v])



 Comments   
Comment by Alex Miller [ 10/Feb/14 9:27 AM ]

Added patch. Will need to be refreshed once other patches go in.

Comment by Alex Miller [ 10/Feb/14 4:02 PM ]

oops

Comment by Rich Hickey [ 12/Feb/14 10:53 AM ]

[k,v] => [k v]

Comment by Alex Miller [ 12/Feb/14 11:46 AM ]

New patch fixing [k v].





[CLJ-1345] Add 1.6 beta changelog updates Created: 07/Feb/14  Updated: 14/Feb/14  Resolved: 14/Feb/14

Status: Closed
Project: Clojure
Component/s: None
Affects Version/s: Release 1.6
Fix Version/s: Release 1.6

Type: Enhancement Priority: Major
Reporter: Alex Miller Assignee: Alex Miller
Resolution: Completed Votes: 0
Labels: None

Attachments: Text File clj-1345-2.patch     Text File clj-1345.patch    
Patch: Code
Approval: Ok

 Description   

Update changelog for 1.6 beta.

Patch: clj-1345-2.patch



 Comments   
Comment by Alex Miller [ 10/Feb/14 4:02 PM ]

oops

Comment by Alex Miller [ 12/Feb/14 12:47 PM ]

Updated patch to fix if-some and when-some definitions.





[CLJ-1344] defrecord still uses old hashing algorithm Created: 07/Feb/14  Updated: 14/Feb/14  Resolved: 14/Feb/14

Status: Closed
Project: Clojure
Component/s: None
Affects Version/s: Release 1.6
Fix Version/s: Release 1.6

Type: Defect Priority: Minor
Reporter: Alex Miller Assignee: Unassigned
Resolution: Completed Votes: 0
Labels: defrecord

Attachments: Text File clj-1344-1.patch    
Patch: Code
Approval: Ok

 Description   

defrecord implements hasheq by calling clojure.lang.APersistentMap/mapHasheq. mapHasheq uses the old map hash calculation instead of the new one. At least one external collection (data.avl) also calls this function. It should be updated to match the new hasheq calculations.

I considered changing defrecord to call Murmur3 directly, but this would create a case where the generated class does not work with older Clojure runtimes so I left it at calling mapHasheq instead.

Patch: clj-1344-1.patch



 Comments   
Comment by Alex Miller [ 07/Feb/14 1:33 PM ]

Attached patch to make mapHasheq use new hash map calculation.

Comment by Alex Miller [ 10/Feb/14 4:01 PM ]

oops





[CLJ-1343] Add some?, when-some, if-some for (not (nil? x)) conditions Created: 07/Feb/14  Updated: 15/Feb/14  Resolved: 14/Feb/14

Status: Closed
Project: Clojure
Component/s: None
Affects Version/s: Release 1.5
Fix Version/s: Release 1.6

Type: Enhancement Priority: Major
Reporter: Alex Miller Assignee: Unassigned
Resolution: Completed Votes: 0
Labels: None

Attachments: Text File clj-1343-1.patch     Text File clj-1343-2.patch     Text File clj-1343-3.patch     Text File clj-1343-4.patch    
Patch: Code and Test
Approval: Ok

 Description   

Sometimes it is useful to have a form for non-nil conditions (as opposed to the existing logical true conditions).
Three additions to support this case:

  • some? - same as (not (nil? x))
  • if-some - like if-let, but checks (some? test) instead of test
  • when-some - like when-let, but checks (some? test) instead of test

Patch: clj-1343-4.patch



 Comments   
Comment by Alex Miller [ 10/Feb/14 4:02 PM ]

oops

Comment by Tassilo Horn [ 11/Feb/14 2:32 AM ]

At least to me, the name `some?` doesn't convey the same information as "not nil", so I'd rather prefer a more explicit name like `non-nil?`.

Also, I'm not convinced of the benefit of something like `(when-some x ...)` compared to `(when-not (nil? x) ...)`. A little shorter and one pair of parens less, but IMHO not as clear.

Comment by Jozef Wagner [ 11/Feb/14 2:59 AM ]

In my opinion, some? should be defined as (not (empty? coll)), and used as in "are there 'some' items in this collection?". This will also play nicely with some, which also takes collection as an argument.

Comment by Tassilo Horn [ 12/Feb/14 1:02 AM ]

Jozef, for that purpose, you'd use `seq`. Actually, the definition of `empty?` is `(not (seq coll))`, so your suggestion would boil down to `some?` being `(not (not (seq coll)))`.

Comment by Rich Hickey [ 12/Feb/14 10:56 AM ]

if-some and when-some are supposed to be like if-let and when-let respectively. Changelog will need updating as well

Comment by Alex Miller [ 12/Feb/14 12:38 PM ]

Updated patch to make if-some and when-some similar to if-let and when-let.

Comment by Alex Miller [ 14/Feb/14 10:01 AM ]

New patch that does not use "some?" in if-some and when-some.

Comment by Alex Miller [ 14/Feb/14 10:39 AM ]

New patch that adjusts when-some impl.

Comment by Kyle Kingsbury [ 15/Feb/14 1:04 PM ]

I'd like to echo Jozef Wagner's and Steve Losh's confusion here.

```
user=> (some odd? [1 2 3])
true
user=> (some? odd? [1 2 3])

ArityException Wrong number of args (2) passed to: user$some-QMARK- clojure.lang.AFn.throwArity (AFn.java:437)
```

I might expect (some?) to behave like (some), except returning a boolean instead of a logically true value, but this is clearly not the case. In no other case in the stdlib can I think of two functions which differ only by punctuation yet have completely different semantics.

```
user=> (some? [])
true
```

Given (some)'s association with sequences, I might interpret (some?) to mean "are there some elements here?"; but that's definitely wrong. Given we have (not=), (not-any?), (not-empty), and (not-every?), can we please name this function (not-nil?)? It's only three characters, but makes the interpretation unambiguously clear.

```
user=> (def x nil)
#'user/x
user=> (def y nil)
#'user/y
user=> (some? [x y])
true
user=> (when-some [x y] :i-expect-true)
nil
```

The fact that (when-some) and (if-some) behave like let bindings is, erm, quite surprising to me. The other binding conditionals have -let in their name; perhaps it would be appropriate to use -let here as well?

For that matter, is this use case all that common? I think I reach specifically for a nil? test fewer than 1 in 20 conditionals--in those cases, why not just write

```
(when-let [x (not-nil? y)]
...)
```

instead of

```
(when-some [x y]
...)
```

I'm just not convinced that this pattern is common enough to warrant the confusion of (when-some) having nothing to do with (when (some ...)), haha. What do y'all think? Have I missed some symmetry between (some?) and (some) that helps this all make sense?

Comment by Alex Miller [ 15/Feb/14 4:36 PM ]

Summarizing comments here, mailing list, Twitter, etc:

  • some uses a truthy comparison. some->, some->> use a not nil comparison. This difference existed in 1.5 some?/if-some/when-some follow the latter. This split is unfortunate, but existed before this addition.
  • not-nil?, non-nil?, nnil?, exists?, and all other alternatives I've seen mentioned were considered as options before the existing names were chosen by Rich. Many people have expressed negative feedback about the name choices and I will channel that to Rich for consideration, but ultimately the choice is his.
  • if-some and when-some are likely more useful than some?. In particular, it is commonly needed when reading from core.async channels where nil is a special value (but false is not).
(go
  (if-some [v (<! c)]
    ...))




[CLJ-1339] Empty primitive vectors throw NPE on .equals with non-vector sequential types Created: 04/Feb/14  Updated: 14/Feb/14  Resolved: 14/Feb/14

Status: Closed
Project: Clojure
Component/s: None
Affects Version/s: Release 1.5
Fix Version/s: Release 1.6

Type: Defect Priority: Minor
Reporter: Alex Miller Assignee: Unassigned
Resolution: Completed Votes: 0
Labels: None

Attachments: Text File CLJ-1339.patch    
Patch: Code and Test
Approval: Ok

 Description   

Primitive vectors have several equality cases. In the case where the compared value is not a vector or random access collection but is a sequential or list, an empty primitive vector will throw an NPE:

user> (.equals (vector-of :long) [])   ;; ok
true
user> (.equals (vector-of :long) '())  ;; broken
NullPointerException   clojure.core.Vec (gvec.clj:135)

Cause: In this case of the primitive vector equals() method, seq is called on itself, then .equals() is invoked on the result. seq will return null for an empty primitive vector, causing an NPE.

Solution: Check for this condition and compare with (nil? (seq o)) on the other object.

Patch: CLJ-1339.patch






[CLJ-1338] New Murmur3 class is not public Created: 04/Feb/14  Updated: 07/Feb/14  Resolved: 07/Feb/14

Status: Closed
Project: Clojure
Component/s: None
Affects Version/s: Release 1.6
Fix Version/s: Release 1.6

Type: Defect Priority: Major
Reporter: Alex Miller Assignee: Unassigned
Resolution: Completed Votes: 0
Labels: None

Attachments: Text File clj-1338.patch    
Patch: Code
Approval: Ok

 Description   

The new Murmur3 class added for hashing is not public, which is problematic for code that needs to call it in several other tickets. To separate out this overlapping change, I have provided it here by itself.






[CLJ-1336] Allow external collections to use standard collection hashing Created: 31/Jan/14  Updated: 14/Feb/14  Resolved: 14/Feb/14

Status: Closed
Project: Clojure
Component/s: None
Affects Version/s: Release 1.6
Fix Version/s: Release 1.6

Type: Enhancement Priority: Major
Reporter: Alex Miller Assignee: Alex Miller
Resolution: Completed Votes: 0
Labels: None

Attachments: Text File clj-1336-1.patch     Text File clj-1336-2.patch     Text File clj-1336-3.patch     Text File clj-1336-4.patch    
Patch: Code and Test
Approval: Ok

 Description   

With the change in new hashing algorithms in 1.6, we need to provide a public hook for collections implemented outside of core to participate in the same hash mixing behavior as core collections.

Patch: clj-1336-4.patch
Depends on: CLJ-1338, CLJ-1339, CLJ-1335, CLJ-1331



 Comments   
Comment by Alex Miller [ 04/Feb/14 10:42 AM ]

Updated patch for a couple issues. However, in testing the use of this I discovered that the hash-basis must be an int and the basis accumulation must be based on int-accumulation with int-overflow, so it is not possible to do this in pure Clojure so this function is not currently useful.

I think the best solution would be to provide functions that encapsulate the ordered and unordered algorithms (Murmur3/hashOrdered and Murmur3/hashUnordered basically) such that external collections can implement hasheq correctly and with good performance.

Comment by Alex Miller [ 04/Feb/14 2:45 PM ]

Add new patch that makes Murmur3 class public so it will work for users of mix-collection-hash. Also adds generative tests for comparing the external collection hashing algorithm with hashes produced by internal ordered and unordered collections. These tests currently fail due to CLJ-1335 (empty list and empty lazy seq return wrong hash code).





[CLJ-1335] PersistentList$EmptyList and empty LazySeq still returns old value for hasheq Created: 30/Jan/14  Updated: 14/Feb/14  Resolved: 14/Feb/14

Status: Closed
Project: Clojure
Component/s: None
Affects Version/s: Release 1.6
Fix Version/s: Release 1.6

Type: Defect Priority: Blocker
Reporter: Andy Fingerhut Assignee: Unassigned
Resolution: Completed Votes: 0
Labels: None

Attachments: File clj-1335-v1.diff     Text File clj-1335-v2.patch    
Patch: Code and Test
Approval: Ok

 Description   

After late Jan 2014 changes to hash functions, PersistentList$EmptyList and (lazy-seq) were left behind:

user=> (= '() (lazy-seq) [])
true
user=> (map hash ['() (lazy-seq) []])
(1 1 -2017569654)
user=> (map class ['() (lazy-seq) []])
(clojure.lang.PersistentList$EmptyList clojure.lang.LazySeq clojure.lang.PersistentVector)

PersistentQueue/EMPTY was updated, so should not need any change.

Solution: Update LazySeq.hasheq() and make EmptyList implement IHashEq. EmptyList now creates a static constant for the hash value of an empty ordered collection and returns the constant for hasheq. An alternative would be to have Murmur3 have this constant instead.

Patch: clj-1335-v2.patch
Depends on: CLJ-1338, CLJ-1339, CLJ-1331 (must be applied first)

Patch:



 Comments   
Comment by Andy Fingerhut [ 30/Jan/14 6:33 PM ]

Patch clj-1335-v1.diff adds tests that assume the patch clj-1331-v1.diff on ticket CLJ-1331 have already been committed. If it is desired to combine these into one patch, or commit this one without that one, I can eliminate that dependency.

Makes PersistentList$EmptyList implement IHashEq interface with a straightforward implementation of hasheq(), comments out empty LazySeq special case check that caused it to return old hash value, and fixes a NullPointerException for primitive vectors discovered by the new tests added.





[CLJ-1334] Improve performance of the bean function with caching Created: 30/Jan/14  Updated: 31/Jan/14  Resolved: 31/Jan/14

Status: Closed
Project: Clojure
Component/s: None
Affects Version/s: Release 1.5
Fix Version/s: None

Type: Enhancement Priority: Minor
Reporter: Ron Pressler Assignee: Unassigned
Resolution: Declined Votes: 0
Labels: performance


 Description   

The bean function is a very useful Java interop feature that provides a read-only view of a Java Bean as a Clojure map.
As it stands, the function performs introspection on the bean's class whenever the function is called. We can, however, cache the mapping from keywords to getters using JDK 7's handy ClassValue. The proposed function will look like this:

(def ^:private ^java.lang.ClassValue bean-class-value
(proxy [java.lang.ClassValue]
[]
(computeValue [c]
(reduce1 (fn [m ^java.beans.PropertyDescriptor pd]
(let [name (. pd (getName))
method (. pd (getReadMethod))
type (.getPropertyType pd)]
(if (and method (zero? (alength (. method (getParameterTypes)))))
(assoc m (keyword name) (fn [x] (clojure.lang.Reflector/prepRet type (. method (invoke x nil)))))
m)))
{}
(seq (.. java.beans.Introspector
(getBeanInfo c)
(getPropertyDescriptors)))))))

(defn bean
"Takes a Java object and returns a read-only implementation of the
map abstraction based upon its JavaBean properties."
{:added "1.0"}
[^Object x]
(let [c (. x (getClass))
pmap (.get bean-class-value c)
v (fn [k] ((pmap k) x))
snapshot (fn []
(reduce1 (fn [m e]
(assoc m (key e) ((val e) x)))
{} (seq pmap)))]
(proxy [clojure.lang.APersistentMap]
[]
(containsKey [k] (contains? pmap k))
(entryAt [k] (when (contains? pmap k) (new clojure.lang.MapEntry k (v k))))
(valAt ([k] (when (contains? pmap k) (v k)))
([k default] (if (contains? pmap k) (v k) default)))
(cons [m] (conj (snapshot) m))
(count [] (count pmap))
(assoc [k v] (assoc (snapshot) k v))
(without [k] (dissoc (snapshot) k))
(seq [] ((fn thisfn [plseq]
(lazy-seq
(when-let [pseq (seq plseq)]
(cons (new clojure.lang.MapEntry (first pseq) (v (first pseq)))
(thisfn (rest pseq)))))) (keys pmap))))))



 Comments   
Comment by Jozef Wagner [ 30/Jan/14 9:25 AM ]

associated discussion

Comment by Alex Miller [ 31/Jan/14 8:45 AM ]

Curious if anyone is using this inside production code? I use it at the REPL when exploring Java stuff sometimes but have never used it inside my actual code.

Comment by Stuart Halloway [ 31/Jan/14 12:28 PM ]

This is a cool idea, but doesn't need to be in core. How about https://github.com/clojure/java.data ?

Comment by Ron Pressler [ 31/Jan/14 12:48 PM ]

This might be good for java.data, and I'm certainly not saying it should be in the core except for the fact that it already is. This is a proposal for a performance improvement of a core function.





[CLJ-1331] Primitive vectors should use new hash Created: 29/Jan/14  Updated: 14/Feb/14  Resolved: 14/Feb/14

Status: Closed
Project: Clojure
Component/s: None
Affects Version/s: Release 1.6
Fix Version/s: Release 1.6

Type: Enhancement Priority: Major
Reporter: Andy Fingerhut Assignee: Unassigned
Resolution: Completed Votes: 0
Labels: None

Attachments: File clj-1331-v1.diff     Text File clj-1331-v2.patch    
Patch: Code and Test
Approval: Ok

 Description   

Primitive vectors created via vector-of still use Java hashCode for hasheq.

Solution: Make primitive vectors implement IHashEq and call Murmur3.hashOrdered().

Patch: clj-1331-v2.patch
Depends on: CLJ-1338 (must be applied first)



 Comments   
Comment by Andy Fingerhut [ 29/Jan/14 6:03 PM ]

Patch clj-1331-v1.diff is one way to change primitive vectors to use Murmur3 hash.





[CLJ-1328] Make some Clojure tests independent of hash function used Created: 20/Jan/14  Updated: 07/Feb/14  Resolved: 07/Feb/14

Status: Closed
Project: Clojure
Component/s: None
Affects Version/s: Release 1.6
Fix Version/s: Release 1.6

Type: Defect Priority: Minor
Reporter: Andy Fingerhut Assignee: Unassigned
Resolution: Completed Votes: 0
Labels: None

Attachments: File clj-1328-v3.diff     File clj-1328-v4.diff    
Patch: Code and Test
Approval: Ok

 Description   

The most interesting failures with the new hash function are probably the 3 deftest's in multimethods.clj that define the same multimethod name 'simple', and thus whether they pass or fail depends upon the order that they are executed. They are currently executed in an order that allows them to pass. Found this while testing murmurHash3 changes to Clojure, which caused the deftest's to execute in a different order and fail.

Simplest way to eliminate this dependency on order is to make the multimethod names unique in each test, so none of them depends upon state left behind by the others.



 Comments   
Comment by Andy Fingerhut [ 20/Jan/14 1:18 PM ]

Patch clj-1328-v1.diff makes all defmulti names unique in multimethods.clj, so that no deftest result depends upon state left behind by another.

Comment by Andy Fingerhut [ 29/Jan/14 8:11 PM ]

Updates some, but not all, tests that were recently modified or disabled due to change in hash function.

Comment by Andy Fingerhut [ 29/Jan/14 10:52 PM ]

Updates one more test than the previous patch.

Comment by Andy Fingerhut [ 31/Jan/14 3:43 PM ]

clj-1328-v4.diff is identical to clj-1328-v3.diff, except it adds a comment explaining why the case hash collision tests don't need to change much, and it puts in a couple of missing (is ...) around some equality tests.





[CLJ-1325] Report warnings if *unchecked-math* and boxing happens Created: 16/Jan/14  Updated: 29/Aug/14  Resolved: 29/Aug/14

Status: Closed
Project: Clojure
Component/s: None
Affects Version/s: Release 1.6
Fix Version/s: Release 1.7

Type: Enhancement Priority: Major
Reporter: Alex Miller Assignee: Alex Miller
Resolution: Completed Votes: 1
Labels: errormsgs, math

Attachments: File boxed.diff     Text File boxedmath.txt     Text File clj-1325.patch     Text File clj-1325-v2.patch     Text File clj-1325-v3.patch    
Patch: Code and Test
Approval: Ok

 Description   

Currently, it is difficult to tell that the compiler is using boxed math unless you look at the generated bytecode. The proposed enhancement here is to emit new warnings if *unchecked-math* is on and boxed math is occurring.

Approach: In the compiler, when compiling a StaticMethodExpr, if *unchecked-math* is true and the class is clojure.lang.Numbers and one of the parameters of static method is of type java.lang.Object or java.lang.Number, then emit a warning at compile-time.

In addition, there is a new WarnBoxedMath Java annotation - a small number of methods on Numbers with Object parameters use this annotation to indicate that warning should not take place. The same annotation can be (but is not currently) used to mark methods on Numbers without Object/Number params that should warn. See boxedmath.txt for a list of methods and categories.

Patch: clj-1325-v3.patch

Screened by:



 Comments   
Comment by Alex Miller [ 14/Apr/14 10:56 PM ]

Moving to 1.7.

Comment by Alex Miller [ 15/Apr/14 10:17 AM ]

List of methods in Numbers and whether they should be considered "boxed math" or not, with some questions.

Comment by Alex Miller [ 14/May/14 2:34 PM ]

Ready for screening.

Comment by Alex Miller [ 16/May/14 11:19 AM ]

clj-1325-v2.patch is identical to last except for a cleaned up the commit message.

Comment by Alex Miller [ 16/May/14 11:51 AM ]

Added v3 patch that just reworks block/indentation style to match surrounding code better.

Comment by Stuart Sierra [ 16/May/14 1:15 PM ]

Screened. Comments:

1) There is no way to get both overflow checks and boxed-math warnings at the same time. Maybe this doesn't matter.

2) The error messages aren't ideal, because they refer to clojure.lang.Numbers, but we can assume that anyone savvy enough to be using *unboxed-math* will also be savvy enough to know what clojure.lang.Numbers is.

3) This doesn't protect me from autoboxing in arbitrary Java method calls, but normal reflection warnings should catch most real-world cases, since few Java APIs overload on primitive and Object.





[CLJ-1320] min-key assumes numbers, not comparables. Created: 09/Jan/14  Updated: 09/Jan/14  Resolved: 09/Jan/14

Status: Closed
Project: Clojure
Component/s: None
Affects Version/s: Release 1.2, Release 1.3, Release 1.4, Release 1.5, Release 1.6
Fix Version/s: None

Type: Enhancement Priority: Major
Reporter: Pierre-Yves Ritschard Assignee: Unassigned
Resolution: Declined Votes: 0
Labels: None

Attachments: File min-key.diff    
Patch: Code

 Description   

The min-key function assumes the key-fn will yield a number and thus uses the '<' operator to compare results.
There are cases where one might want to use min-key with comparables instead.

While (first (sort-by key-fn seq)) could also be used, it feels more natural for min-key to use comparables.



 Comments   
Comment by Pierre-Yves Ritschard [ 09/Jan/14 3:18 PM ]

As discussed on the .L, since compare is slower it makes more sense to keep min-key as-is.





[CLJ-1318] Support destructuring maps with namespaced keywords Created: 06/Jan/14  Updated: 23/Feb/14  Resolved: 31/Jan/14

Status: Closed
Project: Clojure
Component/s: None
Affects Version/s: Release 1.5
Fix Version/s: Release 1.6

Type: Enhancement Priority: Major
Reporter: Alex Miller Assignee: Unassigned
Resolution: Completed Votes: 0
Labels: None

Attachments: File clj-1318-1.diff     File clj-1318-2.diff     File clj-1318-3.diff     File clj-1318-4.diff     File clj-1318-5.diff     File clj-1318-6.diff    
Patch: Code and Test
Approval: Ok

 Description   

Current :keys destructuring expects symbols and creates local bindings based on those symbols. This works fine with maps that use non-namespaced keyword keys. This enhancement is to add support for destructuring maps with namespaced keyword keys.

;; typical key destructuring for keyword keys without namespaces
(let [{:keys [a b]} {:a 1 :b 2}] (+ a b))

;; WANT some way to destructure map with namespaced keys
(let [{:keys [????]} {:x/a 1 :y/b 2}] (+ a b))

Approach: Allow keywords (with or without namespaces) in :keys destructuring. Destructure to bindings with the name of the keyword (namespace is ignored).

;; this now works
(let [{:keys [x/a y/b]} {:x/a 1 :y/b 2}] (+ a b))

;; add support for putting keywords into :keys as well to support ::keywords
(let [{:keys [:x/a :y/b]} {:x/a 1 :y/b 2}] (+ a b))
(let [{:keys [::a]} {:user/a 1}] a)

;; syms will also now support namespaced symbols
(let [{:syms [x/a y/b]} {'x/a 1 'y/b 2}] (+ a b))

Patch: clj-1318-6.diff

Screened by: Stuart Sierra. See comments, below.

Doc TODO: Will need to update http://clojure.org/special_forms#binding-forms with new binding form.



 Comments   
Comment by Nicola Mometto [ 06/Jan/14 11:58 AM ]

Why {:keys [:a/b]} and not {:keys [a/b}}?
Also, this should probably be extended to :syms for consistency

Comment by Alex Miller [ 06/Jan/14 12:28 PM ]

Good questions both. For the first question, we want to make locally namespaced keywords (::foo) work and there is no way to say that as a symbol.

I am waiting to hear back from Rich whether support for namespaced :syms is desirable. I think the change to support it is identical to the change to support namespaced keywords as symbols. I'm going to proactively update the patch to support that too.

Comment by Alex Miller [ 06/Jan/14 12:50 PM ]

Added new patch - now supports namespaced symbols or keywords in :keys and namespaced symbols in :syms.

Comment by Rich Hickey [ 06/Jan/14 1:00 PM ]

Should (also) support symbols for names, e.g. {:keys [a/b]}, only limitation is you can't get ns alias resolution. :syms support makes sense, but may seem weird to provide keywords for local names (where it doesn't as much for keywords), but would allow reaching aliases. My preference is no keyword names support for :syms, i.e. {:syms [a/b]} ok, {:syms [:a/b]} not.

Comment by Nicola Mometto [ 06/Jan/14 1:10 PM ]

To me {:syms [:a/b]} doesn't feel any more weird than writing {:keys [:a/b]}.
If this is going to be added, I think it should be consistent for :keys and :syms.
I understand that :syms is rarely used and this should not be an issue realistically, but I would expect everything that works for :keys to work for :syms too and adding only half a feature to :syms might cause unnecessary confusion.

Comment by Nicola Mometto [ 07/Jan/14 2:16 PM ]

With this patch this will now work:

user=> (let [:a/b 1] b)
1

I don't think this is desiderable.

Comment by Alex Miller [ 07/Jan/14 3:52 PM ]

Right, that is a consequence of allowing keywords in the :keys. At a glance this seems hard to address without significant changes unless we catch it prior to processing. Will consider.

Comment by Alex Miller [ 07/Jan/14 4:40 PM ]

Added new patch variant that catches keywords as let binding keys and throws an Exception.

Comment by Alex Miller [ 10/Jan/14 2:24 PM ]

Added one test in -4 showing example of auto-resolved keywords in :keys.

Comment by Stuart Sierra [ 10/Jan/14 3:00 PM ]

Screened. A few comments:

1. The examples in the tests use {:keys (a b)} with lists instead of {:keys [a b]} with vectors. Both forms are accepted both before and after the patch, but the docs at Clojure - special_forms only show vectors.

2. I would like this to work, but it would add some complexity:

(ns com.example.myproject.foo)

  (def data
    {::a 1 ::b 2})

  ;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
  (ns com.example.myproject.bar
    (:require [com.example.myproject.foo :as foo]))

  ;; I would like this to work:
  (let [{:keys [foo/a foo/b]} foo/data]
    [a b])
  ;;=> [nil nil]

  ;; This is good enough, however:
  (let [{:keys [::foo/a ::foo/b]} foo/data]
    [a b])
  ;;=> [1 2]

3. This doesn't produce an error, which is logically consistent but perhaps not desirable:

(let [{:a ::foo/a} foo/data]
    [a])
Comment by Rich Hickey [ 24/Jan/14 10:11 AM ]

please change the tests to use vectors

Comment by Alex Miller [ 24/Jan/14 10:28 AM ]

Added new -5 diff that uses vectors instead of lists in :keys tests.

Comment by Alex Miller [ 24/Jan/14 11:07 AM ]

And also fixing :syms [] in -6 diff.

Comment by Alex Miller [ 24/Jan/14 11:08 AM ]

Changed examples in description to use [].

Comment by Fogus [ 07/Feb/14 2:23 PM ]

A potential point of confusion here is illustrated by the following:

(let [m {:x/a 1, :y/b 2, :x/b 3000}
        {:keys [x/a y/b x/b]} m]
  (+ a b))

//=> 3

To get the answer 3001 one needs to remove the conflicting binding :y/b. Maybe this is not a big deal, but expect questions for the next 100 years.

Comment by David Nolen [ 23/Feb/14 5:01 PM ]

Ported to ClojureScript with CLJS-745





[CLJ-1312] clojure.string/split on empty string includes empty string in results Created: 21/Dec/13  Updated: 07/Sep/14  Resolved: 21/Dec/13

Status: Closed
Project: Clojure
Component/s: None
Affects Version/s: Release 1.5
Fix Version/s: None

Type: Defect Priority: Minor
Reporter: Russell Dunphy Assignee: Unassigned
Resolution: Declined Votes: 0
Labels: string


 Description   

Splitting a string using clojure.string/split with an empty regex includes the empty string in the results - is this expected behaviour?

Example:

Unable to find source-code formatter for language: clojure. Available languages are: javascript, sql, xhtml, actionscript, none, html, xml, java
user=> (clojure.string/split "abc" #"")
["" "a" "b" "c"]


 Comments   
Comment by Alex Miller [ 21/Dec/13 8:05 AM ]

Yes, I think so. This is a case where Clojure defers to the host (Java) for behavior. I think the way to interpret this is that the empty pattern matches all strings. Split checks left to right whether there is a next chunk of string that matches the pattern. The empty pattern matches at the beginning to a string of length 0. Something like that.

Comment by Mark Engelberg [ 07/Sep/14 12:27 PM ]

This bug is a real problem, because it works differently on Windows than on Linux. On Windows, clojure.string/split behaves exactly as you'd expect:

user=> (clojure.string/split "abc" #"")
["a" "b" "c"]

Only on Linux do you get the strange behavior where the empty string shows up at the beginning of the list.

I recently had a student that got burned by this in some webserver code that relied on splitting using the empty regex. It performed flawlessly on her local Windows machine, but mysteriously broke when she uploaded the uberwar to the cloud. The bug was very difficult to track down.

If this were a bug on both Windows and Linux, at least you could plan around it. But right now, it's an obstacle to Clojure's capability of running consistently across platforms.

Comment by Mark Engelberg [ 07/Sep/14 12:40 PM ]

Upon further research, I've found that this is not a Windows/Linux issue, rather it's a difference between Java 7 and Java 8. On Java 8, splitting with the empty string no longer produces a sequence that begins with an empty string.

As you said before, this is just a gotcha relating to Java, not a Clojure issue.





[CLJ-1310] some-> behaves differently than -> when used with macros Created: 19/Dec/13  Updated: 19/Dec/13  Resolved: 19/Dec/13

Status: Closed
Project: Clojure
Component/s: None
Affects Version/s: Release 1.5
Fix Version/s: None

Type: Defect Priority: Minor
Reporter: Chris Perkins Assignee: Unassigned
Resolution: Declined Votes: 0
Labels: None


 Description   

I stumbled across a behavior of some-> when used with macros, and I'm wondering whether it's expected.

I started with something like this:

(some-> value
foo
bar
quux)

Then I realized that quux might throw an exception, which I want to ignore, so had the possibly misguided idea to do this:

(defmacro catch->nil [& body]
`(try ~@body (catch Exception _# nil)))

(some-> value
foo
bar
quux
catch->nil)

My mental model of some-> is that it should end up wrapping the whole expression in a try-catch, as > does, but that does not happen. some> expands into a `let` instead of a deeply-nested form, so the exception is not caught.

Certainly easy to work around, now that I know about it, but I thought perhaps this was not intended.



 Comments   
Comment by Chris Perkins [ 19/Dec/13 7:34 PM ]

After a little reflection, I realize that my mental model of some-> as "thread-first plus magic fairy-dust" is fatally flawed. I withdraw my objections. Please disregard.

Comment by Alex Miller [ 19/Dec/13 7:50 PM ]

withdrawn by submitter





[CLJ-1307] Type hint remains unqualified, resulting in errors Created: 15/Dec/13  Updated: 15/Dec/13  Resolved: 15/Dec/13

Status: Closed
Project: Clojure
Component/s: None
Affects Version/s: None
Fix Version/s: None

Type: Defect Priority: Major
Reporter: Nicola Mometto Assignee: Unassigned
Resolution: Duplicate Votes: 0
Labels: None


 Description   

This snippet (even though unsensical) demonstrates the issue:

user=> (ns foo (:import clojure.lang.RT))
nil
foo=> (defn x ^RT [])
#'foo/x
foo=> (ns bar (:use foo))
nil
bar=> (.hashCode (x))
CompilerException java.lang.IllegalArgumentException: Unable to resolve classname: RT, compiling:(NO_SOURCE_PATH:4:1)

This is because (:tag (meta #'x)) returns the Symbol 'RT, it should either be the Symbol 'clojure.lang.RT or the Class clojure.lang.RT



 Comments   
Comment by Nicola Mometto [ 15/Dec/13 8:43 PM ]

Duplicate: http://dev.clojure.org/jira/browse/CLJ-1232





[CLJ-1306] Cannot reduce over short[] arrays Created: 14/Dec/13  Updated: 14/Dec/13  Resolved: 14/Dec/13

Status: Closed
Project: Clojure
Component/s: None
Affects Version/s: Release 1.5
Fix Version/s: None

Type: Defect Priority: Major
Reporter: Mike Anderson Assignee: Unassigned
Resolution: Duplicate Votes: 0
Labels: bug

Attachments: File clj-1306.diff    

 Description   

Reducing over a short array currently causes an error:

(reduce + (seq (short-array 10)))
=> ClassCastException [S cannot be cast to [Ljava.lang.Object; clojure.core.protocols/fn--6037 (protocols.clj:126)

This appears to occur because ArraySeq is assumed by protocols.clj to contain an Object[] array in the ".array" field, when in fact it is a short[] array.

Proposed solution to to create ArraySeq_short (analogous to the other primitive types ArraySeq_long etc.) to handle short arrays.



 Comments   
Comment by Mike Anderson [ 14/Dec/13 7:37 AM ]

Fix for CLJ-1306

Comment by Alex Miller [ 14/Dec/13 8:17 AM ]

This was also discovered in CLJ-1200 and the patch for that issue includes ArraySeq_short as you propose. It is expected that CLJ-1200 will be included in 1.6.

Comment by Mike Anderson [ 14/Dec/13 9:29 AM ]

OK, thanks Alex!

Just to be sure: Can you confirm that a fix will definitely go into 1.6? This is a defect, and as such should have a higher priority than CLJ-1200 (which appears to be presented as an performance enhancement patch).

My patch also includes a regression test which I think could helpfully be included in the test suite.

Comment by Alex Miller [ 14/Dec/13 11:24 AM ]

Yes, I fully expect CLJ-1200 to be included and talked to Rich about it as recently as yesterday. I could split things out of that patch and pull both of these in separately. That would be objectively better but definitely more work to do all the ticket, patch, and screening work so I'd rather not. If you want to attach just the regression test to 1200, I think we could include it that way as 1200 hasn't been screened yet. Stuart Sierra is planning to screen it in the next few days.





[CLJ-1304] Fixed minor typos in documentation and code comments Created: 09/Dec/13  Updated: 04/Feb/14  Resolved: 04/Feb/14

Status: Closed
Project: Clojure
Component/s: None
Affects Version/s: Release 1.4, Release 1.5
Fix Version/s: Release 1.6

Type: Defect Priority: Trivial
Reporter: Vipul A M Assignee: Unassigned
Resolution: Completed Votes: 0
Labels: docstring

Attachments: File clj-1304-v2.diff     File doc-comment-typos.diff    
Patch: Code
Approval: Screened

 Description   

Fixed minor typos in documentation and code comments across multiple files.



 Comments   
Comment by Andy Fingerhut [ 11/Jan/14 2:53 PM ]

Patch clj-1304-v2.diff dated Jan 11, 2014 is identical to Vipul's patch doc-comment-typos.diff dated Dec 9, 2013, except it applies cleanly to latest master. The only changes are that it removes the part of the patch for files in the ASM library, which was updated in a recent commit to Clojure master.

Comment by Alex Miller [ 04/Feb/14 9:21 PM ]

reopen so that I can set the fix version which didn't get set.

Comment by Alex Miller [ 04/Feb/14 9:22 PM ]

re-close now that fix version is set





[CLJ-1303] Remove (apparently) vestigial forward-defs of unquote and unquote-splicing Created: 05/Dec/13  Updated: 01/Feb/14  Resolved: 31/Jan/14

Status: Closed
Project: Clojure
Component/s: None
Affects Version/s: Release 1.5
Fix Version/s: None

Type: Enhancement Priority: Trivial
Reporter: David Rupp Assignee: Unassigned
Resolution: Declined Votes: 0
Labels: enhancement

Attachments: File generalize-unquote.diff    
Patch: Code

 Description   

clojure/core.clj contains forward defs of unquote and unquote-splicing that seem no longer to be necessary. The pull request at https://github.com/clojure/clojure/pull/45 removes this dead code (also attaching a git diff file). Existing tests pass; no new test code necessary.



 Comments   
Comment by Alex Miller [ 05/Dec/13 9:23 PM ]

FYI for future reference, Clojure doesn't accept pull requests. Thanks for the report though!

Comment by David Rupp [ 05/Dec/13 9:31 PM ]

I noticed. That's why I created the JIRA.

Comment by Andy Fingerhut [ 07/Dec/13 10:25 AM ]

David, I do not have any comment on whether this patch will be accepted or not based on the changes it makes, but patches do need to be in a particular format, including the author's name. See instructions for how to create a patch in this format here: http://dev.clojure.org/display/community/Developing+Patches

Comment by David Rupp [ 07/Dec/13 11:26 AM ]

Submitting properly-formatted patch.

Comment by David Rupp [ 09/Dec/13 7:49 AM ]

Replaced references to clojure.core/unquote and .../unquote-splicing,
which are unbound.

The UNQUOTE and UNQUOTE-SPLICING Symbols in LispReader don't really
refer to anything in clojure.core any longer. They're created and
elided by the reader when it encounters their respective (reader)
macro chars.

Comment by Stuart Halloway [ 31/Jan/14 3:26 PM ]

The patch attached here is poorly-suited for screening. It does more than what it says, (e.g. deleting the def of META) without explaining why.

It also removes things that are commented out. Pretty clear that the BDFL likes having those things stick around.

Comment by David Rupp [ 01/Feb/14 6:57 PM ]

META is not used anywhere. I will explain better next time.

Also, DEREF_BANG has been commented out since 2007 (commit 139ddd146f2a272b7ddda397f54b501ff499c643). Figured it was pretty safe to get rid of at this point. My bad.





[CLJ-1302] keys and vals consistency not mentioned in docstring Created: 04/Dec/13  Updated: 14/Feb/14  Resolved: 14/Feb/14

Status: Closed
Project: Clojure
Component/s: None
Affects Version/s: Release 1.5
Fix Version/s: Release 1.6

Type: Enhancement Priority: Minor
Reporter: Phill Wolf Assignee: Unassigned
Resolution: Completed Votes: 2
Labels: docstring

Attachments: Text File clj-1302-1.patch     Text File clj-1302-2.patch    
Patch: Code
Approval: Ok

 Description   

(keys m) and (vals m) appear to return stuff in a consistent order, so (= m (zipmap (keys m) (vals m))). This consistency is a useful property. The API docs should state whether it is part of the functions' contract.

Patch: clj-1302-2.patch



 Comments   
Comment by Gary Fredericks [ 11/Dec/13 7:18 PM ]

One thing to keep in mind is that the functions can be used on arbitrary instances of java.util.Map, which, aside from being mutable, could hypothetically (though not realistically) generate their entry sets nondeterministically.

I don't know what any of this means about what the docstring should say. It could claim the consistency for clojure's collections at least.

Comment by Andy Fingerhut [ 11/Dec/13 7:44 PM ]

The ticket creator might already realize this, but note that (= m (zipmap (keys m) (vals m))) is guaranteed for Clojure maps, where m is the same identical map, at least by the current implementation. I am not addressing the question whether it is part of the contract, but I think it would be good to make this explicit if it is part of the contract.

The following is not guaranteed for Clojure maps: (= m1 m2) implies that (= (keys m1) (keys m2)).

The set of keys/vals will be equal, but the order of keys/vals could be different for two otherwise equal maps m1, m2.

Comment by Steve Miner [ 27/Dec/13 11:10 AM ]

I think you can depend on a slightly stronger contract: The order of the results from `keys` and `vals` follows the order of the results from `seq`. As with any pure function, `seq` returns consistent results across multiple calls with the same (identical?) map. The order may be arbitrary for a non-sorted map, but it should be consistent.

Some time ago, I looked for this guarantee in the documentation, but I couldn't find it explicitly stated. However, after looking at the implementation, I think it's safe to depend on this invariant.

Comment by Stuart Halloway [ 31/Jan/14 12:48 PM ]

The absence of this property in the docs is correct. You should not rely on this.

Comment by Nicola Mometto [ 31/Jan/14 7:43 PM ]

I have to say this surprises me, I was relying on this undocumented behaviour expecting it to be implicit.

I did a quick search in github and the number of (zipmap (keys m) (do-something (vals m))) is significant, even some experienced clojure developers seem to have given this property for granted (https://groups.google.com/forum/?fromgroups#!topic/clojure/s1sFVF7dAVs).

Could we at least explicitely document the absence of this property in the docs in order to avoid further confusion?

Comment by Peter Taoussanis [ 01/Feb/14 3:21 AM ]

Big surprise here too. Could someone (Stu?) possibly motivate a little why this couldn't/shouldn't be a contractual property? It seems like it has utility. Perhaps more importantly, it seems to be an intuitively reasonable assumption. That's subjective, sure, but I feel like I've seen this pattern come up quite widely.

Anecdotally, am quite sure I've made use of the assumption before (i.e. that `(keys m)` and `(vals m)` will return sequences as per pair order).

Would need to review code to see how frequently I've made the error.

To clarify: not disagreeing, just want to understand the thought that's gone in.

> Could we at least explicitely document the absence of this property in the docs in order to avoid further confusion?

That'd be a big help I think. I'd generally take an

Comment by Peter Taoussanis [ 01/Feb/14 3:58 AM ]

End of comment got mangled somehow.

Was just going to point out that I'm a big fan of how cautious + deliberate Clojure's design tends to be. Being hesitant to pick up needless or half-baked contractual obligations, etc. is a huge strength IMO.

Comment by Rich Hickey [ 01/Feb/14 9:36 AM ]

keys order == vals order == seq order

Comment by Alex Miller [ 05/Feb/14 11:25 AM ]

Tweaked doc.





[CLJ-1301] case expression fails to match a BigDecimal Created: 23/Nov/13  Updated: 26/Jan/14  Resolved: 26/Jan/14

Status: Closed
Project: Clojure
Component/s: None
Affects Version/s: Release 1.5, Release 1.6
Fix Version/s: Release 1.6

Type: Defect Priority: Blocker
Reporter: Andy Fingerhut Assignee: Unassigned
Resolution: Completed Votes: 2
Labels: Compiler

Attachments: Text File case-alt.patch     File clj-1301-1.diff    
Patch: Code and Test
Approval: Incomplete

 Description   

In 1.5.1 (anywhere before the CLJ-1118 patch), this is the behavior on BigDecimal case matching:

user=> (defn t [v] (case v 1 "Long" 1.0M "BigDecimal" "none"))
#'user/t
user=> (map t [1 1.0M 1.00M])
("Long" "BigDecimal" "none")

In 1.6 the behavior (post CLJ-1118 patch) has changed:

user=> (defn t [v] (case v 1 "Long" 1.0M "BigDecimal" "none"))
#'user/t
user=> (map t [1 1.0M 1.00M])
("Long" "none" "none")

In 1.6 after CLJ-1118, I expect to see: ("Long" "BigDecimal" "BigDecimal") as they now have the same hash and hasheq.

Cause: The case constants are hashed in the clojure.core/case macro using clojure.core/hash which calls clojure.lang.util/hasheq(). In Compiler.emitExprForHashes(), a call to clojure.lang.Util/hash(). In Clojure 1.5 these hash values are the same (hash of 1.0M == hasheq of 1.0M == 311). In Clojure 1.6, they are different (hash of 1.0M = 311, hasheq of 1.0M = 31).

In any cases where Java's hashCode and Clojure's hasheq return different values, the case statement can fail to do the correct thing.

Approach: Change Compiler.java to use clojure.lang.Util hasheq() to match the case macro use of clojure.core/hash (which calls clojure.lang.Util.hasheq()).

Patch: clj-1301-1.diff

Screened by:



 Comments   
Comment by Andy Fingerhut [ 23/Nov/13 5:00 PM ]

Patch clj-1301-1.diff modifies Compiler.java so that case* statements use hasheq on the test expression value, rather than Java's hashCode. It also adds a test case that currently fails with latest Clojure master, but passes with the patch.

Comment by Andy Fingerhut [ 23/Nov/13 5:01 PM ]

This bug is also the root cause for the recent failures of tests for the test.generative library.

Comment by Alex Miller [ 10/Dec/13 3:22 PM ]

Putting in 1.6 release per Rich.

Comment by Alex Miller [ 13/Dec/13 3:36 PM ]

Andy, I talked to Rich and the conclusion was that we should make the opposite change here such that the case macro should route to the Java hashcode version clojure.lang.util.hash() and the Compiler should be left as is. Can you update the patch?

Comment by Alex Miller [ 13/Dec/13 3:38 PM ]

And in case you were wondering, the reason is that the Java hashcode is generally faster (case is all about speed) and there are easy opportunities for you to properly cast your expression and/or case constants (where as the situations with collections where boxing is difficult to fix generically, that is not true).

Comment by Andy Fingerhut [ 13/Dec/13 5:14 PM ]

Alex, unless I am missing something, changing case to use Java's hashCode() would also require changing its current equality comparison from Clojure = (aka equiv()) to something consistent with hashCode(), which I think must be Java's equals().

Such a change would mean that all of the things that are = but not equals() will not match each other in a case statement, e.g. a case value of (Integer. 5) will not match a (Long. 5) value to compare against in a case branch.

Is that really what is desired here? I almost hesitate to create such a patch, for fear it might be committed

Comment by Alex Miller [ 17/Dec/13 12:06 PM ]

Based on discussion comments, move back to Incomplete until we resolve.

Comment by Alex Miller [ 16/Jan/14 9:37 AM ]

Added better example demonstrating the problem (the specific problem exposed by CLJ-1118).

Comment by Alex Miller [ 16/Jan/14 11:50 AM ]

Simplified examples.

Comment by Alex Miller [ 16/Jan/14 12:29 PM ]

Re Andy's comments above, I walked down that path a bit and built such a patch, however we currently have tests in clojure.test-clojure.control:

(testing "test number equivalence"
    (is (= :1 (case 1N 1 :1 :else))))

which clearly seems to expect Clojure equiv() behavior over Java equals() behavior in case constant matching. So either that is a bad test or this is not a viable approach (it also suggests we could break existing code with this change).

Comment by Andy Fingerhut [ 16/Jan/14 12:55 PM ]

One could consider having the default behavior of case to use hasheq and clojure.core/= everywhere, but add a 'fast' option to use hashCode and Java equals.

Comment by Alex Miller [ 24/Jan/14 9:46 AM ]

Alternative patch in the direction of using hashcode/equals instead of hasheq/equiv. Note that this test causes some test failures. This is not yet a candidate patch - further work needs to be done in evaluating this path.





[CLJ-1299] Binding conveyance seems to be broken as of 1.6.0-alpha2 Created: 21/Nov/13  Updated: 02/Dec/13  Resolved: 02/Dec/13

Status: Closed
Project: Clojure
Component/s: None
Affects Version/s: Release 1.6
Fix Version/s: Release 1.6

Type: Defect Priority: Blocker
Reporter: Peter Taoussanis Assignee: Unassigned
Resolution: Duplicate Votes: 0
Labels: None
Environment:

Clojure 1.6.0-alpha2
java version "1.7.0_04"
Java(TM) SE Runtime Environment (build 1.7.0_04-b21)
Java HotSpot(TM) 64-Bit Server VM (build 23.0-b21, mixed mode)


Approval: Vetted

 Description   

With Clojure 1.5:

(def ^:dynamic *num* 1)
(binding [*num* 2] (future (dotimes [_ 10] (println *num*))))

Behaves as expected: "2" prints 10 times. With Clojure 1.6.0-alpha2 the same form will print "1"s off the main thread.

This seems to be an interaction between loop/recur and the binding conveyance: the num binding does convey without the loop:

(def ^:dynamic *num* 1)
(binding [*num* 2] (future (println *num*))) ; Prints "2" even with 1.6.0-alpha2


 Comments   
Comment by Alex Miller [ 21/Nov/13 9:02 AM ]

I can confirm that the patch for this ticket is the one that introduced this behavior in 1.6:
http://dev.clojure.org/jira/browse/CLJ-1125

Comment by Alex Miller [ 02/Dec/13 9:57 PM ]

Marking as a dup now handled by the new patch in CLJ-1125 (and test included there).





[CLJ-1294] (vec (range)) causes REPL to hang (infinite loop?) Created: 11/Nov/13  Updated: 11/Nov/13  Resolved: 11/Nov/13

Status: Closed
Project: Clojure
Component/s: None
Affects Version/s: Release 1.5
Fix Version/s: None

Type: Defect Priority: Minor
Reporter: Rene Semmelrath Assignee: Unassigned
Resolution: Declined Votes: 0
Labels: None
Environment:

LightTable and CoutnerClockWise
Windows7



 Description   

I just typed

(vec (range))

in LightTable and it stop working while CPU went to 100%
the same behavior is in Eclipse/Couterclockwise REPL



 Comments   
Comment by Andy Fingerhut [ 11/Nov/13 5:39 PM ]

Rene, this is expected to cause an infinite loop. (range) returns a lazy infinite sequence of integers. vec tries to cause the entire infinite sequence to be realized so it can put all elements into a vector. This is not a bug. You asked for an infinite loop, and you got one

Comment by Alex Miller [ 11/Nov/13 6:10 PM ]

(range) will yield an infinite sequence. vec will walk the sequence, conj'ing each item onto the vector ... which will do bad things.

While Clojure works well with infinite sequences (an abstraction), the collections like vector support only a finite number of elements.





[CLJ-1288] aset-* and aget: on multi-dimensional arrays (e.g. double[][]) these fn reflect (and, thus, perf. poorly) even with type hints. Created: 01/Nov/13  Updated: 01/Nov/13  Resolved: 01/Nov/13

Status: Closed
Project: Clojure
Component/s: None
Affects Version/s: Release 1.5
Fix Version/s: None

Type: Defect Priority: Minor
Reporter: Michael O. Church Assignee: Unassigned
Resolution: Duplicate Votes: 0
Labels: arrays, performance
Environment:

Clojure 1.5.1.

Dependencies: criterium



 Description   

Here's a transcript of the behavior. I don't know for sure that reflection is being done, but the performance penalty (about 1300x) suggests it.

user=> (use 'criterium.core)
nil
user=> (def b (make-array Double/TYPE 1000 1000))
#'user/b
user=> (quick-bench (aget ^"[[D" b 304 175))
WARNING: Final GC required 3.5198021166354323 % of runtime
WARNING: Final GC required 29.172288684474303 % of runtime
Evaluation count : 63558 in 6 samples of 10593 calls.
Execution time mean : 9.457308 µs
Execution time std-deviation : 126.220954 ns
Execution time lower quantile : 9.344450 µs ( 2.5%)
Execution time upper quantile : 9.629202 µs (97.5%)
Overhead used : 2.477107 ns
nil

A (n ugly) workaround is to use multiple agets.

user=> (quick-bench (aget ^"[D" (aget ^"[[D" b 304) 175))
WARNING: Final GC required 40.59820310542545 % of runtime
Evaluation count : 62135436 in 6 samples of 10355906 calls.
Execution time mean : 6.999273 ns
Execution time std-deviation : 0.112703 ns
Execution time lower quantile : 6.817782 ns ( 2.5%)
Execution time upper quantile : 7.113845 ns (97.5%)
Overhead used : 2.477107 ns
nil



 Comments   
Comment by Alex Miller [ 01/Nov/13 1:09 PM ]

Dupe of CLJ-1289.





[CLJ-1287] Change select-keys to return same type of map as it is given Created: 01/Nov/13  Updated: 14/Feb/14  Resolved: 14/Feb/14

Status: Closed
Project: Clojure
Component/s: None
Affects Version/s: Release 1.5
Fix Version/s: None

Type: Enhancement Priority: Minor
Reporter: Andy Fingerhut Assignee: Unassigned
Resolution: Declined Votes: 0
Labels: None

Attachments: File clj-1287-1.diff     File clj-1287-3.diff    
Patch: Code and Test

 Description   

select-keys always returns a map built up from {}. This will sometimes not be the same type of map as the one given as the first arg, e.g. if the first arg is a sorted-map, the returned map will not be. There are situations where it would be nice if it preserved the kind of map it was given, e.g. the developer wishes to preserve the sorting order of the keys.

Discussion thread on Clojure group: https://groups.google.com/forum/#!topic/clojure/l_V1N1nRF-c



 Comments   
Comment by Andy Fingerhut [ 01/Nov/13 9:36 AM ]

clj-1287-1.diff changes the starting set {} to (empty map) in select-keys, and adds tests for the new behavior. The only reason select-keys is moved later in core.clj is to move it after the definition of function 'empty'. Before this patch 'select-keys' is defined before 'declare', even, so we cannot simply put a '(declare empty)' before select-keys.

Comment by Alex Miller [ 01/Nov/13 12:58 PM ]

What happens with records? Does this introduce a new failure case there (due to lack of empty support in records)?

Comment by Andy Fingerhut [ 01/Nov/13 5:43 PM ]

I had not thought about records when making the first patch. It does cause select-keys to fail if given a record, whereas without the patch select-keys works fine on records. clj-1287-2.diff is similar to the earlier patch, but if its argument is a record, it still returns a correct value based upon building up an unsorted map starting from {}. If its argument is not a record, it assumes empty will work on it and uses (empty map) to start with.

Comment by Andy Fingerhut [ 14/Feb/14 12:11 PM ]

Patch clj-1287-3.diff is identical to the earlier clj-1287-2.diff described in an earlier comment, except it updates some diff context lines so that it applies cleanly to the latest Clojure master as of today.

Comment by Alex Miller [ 14/Feb/14 1:14 PM ]

We do not wish to make this change.





[CLJ-1285] Persistent assoc/conj on a transient-created collision node Created: 28/Oct/13  Updated: 11/Nov/13  Resolved: 11/Nov/13

Status: Closed
Project: Clojure
Component/s: None
Affects Version/s: Release 1.5
Fix Version/s: Release 1.6

Type: Defect Priority: Critical
Reporter: Christophe Grand Assignee: Unassigned
Resolution: Completed Votes: 4
Labels: transient

Attachments: File persistent-assoc-after-collision.diff     File transient-generative-test.diff    
Patch: Code and Test
Approval: Ok

 Description   

Bug reported by Zach Tellman https://groups.google.com/d/msg/clojure-dev/HvppNjEH5Qc/1wZ-6qE7nWgJ

Since transients were introduced the invariant array.length == count*2 doesn't hold for HashCollisionNode.
However persistent .without still relies on it.

Hence persistent dissoc on a collision node created by transients fails.

(let [a (reify Object (hashCode [_] 42))
      b (reify Object (hashCode [_] 42))]
      (= (-> #{a b} transient (disj! a) persistent! (conj a))
       (-> #{a b} transient (disj! a) persistent! (conj a))))

returns false.

Patch: persistent-assoc-after-collision.diff

Generative test patch: transient-generative-test.diff

The generative test reliably reproduces the error. It is simpler than the original test that found the bug but tests a series conj/disj/transient/persistent actions on a set. I've included it separately in case we decide not to apply.

Screened by: Alex Miller



 Comments   
Comment by Michał Marczyk [ 29/Oct/13 9:58 PM ]

I can confirm that the patch works for me. As per our #clojure conversation, I've done the ClojureScript port; see CLJ-648.

Comment by Reid Draper [ 29/Oct/13 11:28 PM ]

I've run Zach's original test, as well as my own simple-check test. Both are passing.

Comment by Alex Miller [ 30/Oct/13 9:33 AM ]

I don't suppose we could get a generative test (prob need to use test.generative which is already included) to test this stuff similar to the original test that found the bug?

Very much hoping to get this into 1.6.

Comment by Andy Fingerhut [ 31/Oct/13 10:52 AM ]

Alex, I suspect clojure-dev would reach a much wider audience for your request than a comment on this ticket, which only has 3 watchers, and I don't think many people besides you and I watch the stream of all ticket state changes as they go by.

Comment by Michał Marczyk [ 01/Nov/13 5:19 AM ]

Just wanted to note that this patch, apart from preventing the hash-based collections from failing Zach's test suite, also makes avl.clj collections pass (now that I've released fixes for the two bugs uncovered by the test suite in avl.clj 0.0.9). This provides some cross-validation, I think.

(The built-in sorted collections pass either way, because they don't support transient ops.)

Also, David Nolen has merged the ClojureScript port of the patch.

Comment by Alex Miller [ 01/Nov/13 7:35 AM ]

I'm going to take a crack at repro with test.generative this morning - wish me luck!

Comment by Alex Miller [ 03/Nov/13 10:40 PM ]

Added a simplified version of a test-generative test and marked screened.

Comment by Alex Miller [ 11/Nov/13 11:17 AM ]

Patch was applied to master for 1.6.





[CLJ-1281] Clojure 1.6 - reconsider what is "alpha" in core Created: 23/Oct/13  Updated: 22/Nov/13  Resolved: 22/Nov/13

Status: Closed
Project: Clojure
Component/s: None
Affects Version/s: Release 1.5
Fix Version/s: Release 1.6

Type: Enhancement Priority: Major
Reporter: Alex Miller Assignee: Alex Miller
Resolution: Completed Votes: 0
Labels: docstring

Attachments: Text File alpha.patch    
Patch: Code
Approval: Ok

 Description   

In Clojure 1.5.1, the following things are marked as "alpha - subject to change". We should consider this list and whether some of them are no longer alpha and update them appropriately.

  • Watches (1.0): add-watch, remove-watch
  • Transients (1.1): transient, persistent!, conj!, assoc!, dissoc!, pop!, disj!
  • Exceptions (1.4): ex-info, ex-data
  • Promises (1.1): promise, deliver
  • Compiler warnings (1.4): :disable-locals-clearing
  • Records (1.3) defrecord
  • Types (1.3): deftype
  • Pretty print (1.3): print-table
  • clojure.reflect (1.3) (all)
  • Reducers (1.5) (all)

Patch: alpha.patch

  • Removes alpha marking for everything except reducers, disable-locals-clearing, and clojure.reflect. If Stu wants to remove for clojure.reflect, he should do so.

Screened by: Alex Miller



 Comments   
Comment by Alex Miller [ 04/Nov/13 8:28 AM ]

Pulling into 1.6 as Rich has given me some feedback on what to change here.

Comment by Alex Miller [ 07/Nov/13 10:29 AM ]

Added patch that removes alpha designation from everything but reducers, disable-locals-clearing, and clojure.reflect (still TBD).

Comment by Andy Fingerhut [ 07/Nov/13 10:58 AM ]

definline is marked experimental in its doc string, and has been marked so since Clojure 1.0. Is it ready to be 'promoted', too?

Comment by Alex Miller [ 07/Nov/13 11:03 AM ]

Excellent question, will find out.

Comment by Alex Miller [ 07/Nov/13 12:40 PM ]

Rich says definline is still experimental, so no change.





[CLJ-1276] Can't make a dispatch map containing forward-declared fns Created: 09/Oct/13  Updated: 18/Apr/14  Resolved: 18/Apr/14

Status: Closed
Project: Clojure
Component/s: None
Affects Version/s: Release 1.6
Fix Version/s: None

Type: Defect Priority: Major
Reporter: Alex Coventry Assignee: Unassigned
Resolution: Declined Votes: 0
Labels: None

Attachments: File unbound-eg.tgz    

 Description   

If from (ns tst2) you try to call tst1/c, which calls tst1/f via dispatch map which was defined when tst1/f was forward declared, you get an "unbound fn" error. E.g.

user=> (dorun (map eval 
                   '[(ns tst1) 
                     (declare f) 
                     (def d {:k f}) 
                     (defn c [] ((d :k)))
                     (defn f [] :success)
                     (ns tst2 (:require [tst1]))
                     (tst1/c)]))

IllegalStateException Attempting to call unbound fn: #'tst1/f  clojure.lang.Var$Unbound.throwArity (Var.java:43)
tst2=> (clojure.repl/pst *e)
IllegalStateException Attempting to call unbound fn: #'tst1/f
	clojure.lang.Var$Unbound.throwArity (Var.java:43)
	tst1/c (NO_SOURCE_FILE:5)
	tst2/eval25 (NO_SOURCE_FILE:8)
	clojure.lang.Compiler.eval (Compiler.java:6642)
	clojure.lang.Compiler.eval (Compiler.java:6605)
	clojure.core/eval (core.clj:2883)
	clojure.core/map/fn--4222 (core.clj:2513)
	clojure.lang.LazySeq.sval (LazySeq.java:40)
	clojure.lang.LazySeq.seq (LazySeq.java:49)
	clojure.lang.RT.seq (RT.java:484)
	clojure.core/seq (core.clj:133)
	clojure.core/dorun (core.clj:2811)


 Comments   
Comment by Alex Coventry [ 09/Oct/13 10:43 PM ]

TEttinger pointed out on IRC that the forms in the example run without error if you wrap them in a (do) block. Here is an example using files. Relevant code is in src/unbound_eg/tst[12].clj. Example output shown below.

http://clojure-log.n01se.net/date/2013-10-09.html#23:52

lap% lein repl
nREPL server started on port 50125 on host 127.0.0.1
REPL-y 0.2.1
Clojure 1.5.1
Docs: (doc function-name-here)
(find-doc "part-of-name-here")
Source: (source function-name-here)
Javadoc: (javadoc java-object-or-class-here)
Exit: Control+D or (exit) or (quit)

user=> (require '[unbound-eg.tst2 :as t2])

IllegalStateException Attempting to call unbound fn: #'unbound-eg.tst1/f clojure.lang.Var$Unbound.throwArity (Var.java:43)
user=> (pst)
IllegalStateException Attempting to call unbound fn: #'unbound-eg.tst1/f
clojure.lang.Var$Unbound.throwArity (Var.java:43)
unbound-eg.tst1/c (tst1.clj:4)
unbound-eg.tst2/eval2233 (tst2.clj:3)
clojure.lang.Compiler.eval (Compiler.java:6619)
clojure.lang.Compiler.load (Compiler.java:7064)
clojure.lang.RT.loadResourceScript (RT.java:370)
clojure.lang.RT.loadResourceScript (RT.java:361)
clojure.lang.RT.load (RT.java:440)
clojure.lang.RT.load (RT.java:411)
clojure.core/load/fn--5018 (core.clj:5530)
clojure.core/load (core.clj:5529)
clojure.core/load-one (core.clj:5336)
nil

Comment by Kevin Downey [ 18/Apr/14 12:23 AM ]

this is just a fact of clojure's compilation model and how vars work.

a var is a little mutable cell

(declare foo) declares that a mutable cell exists with the name foo, it doesn't contain a value

foo then gets the value of the mutable cell (which has none)

(defn foo [] 1) then sets the value of the cell named foo to the function created from (fn [] 1)

Comment by Alex Miller [ 18/Apr/14 7:29 AM ]

I agree with Kevin - this is expected behavior.





[CLJ-1274] Unable to set compiler options via system properties except for AOT compilation Created: 02/Oct/13  Updated: 29/Aug/14  Resolved: 29/Aug/14

Status: Closed
Project: Clojure
Component/s: None
Affects Version/s: Release 1.5
Fix Version/s: Release 1.7

Type: Defect Priority: Major
Reporter: Howard Lewis Ship Assignee: Unassigned
Resolution: Completed Votes: 3
Labels: compiler

Attachments: Text File CLJ-1274.patch    
Patch: Code
Approval: Ok

 Description   

The code that converts JVM system properties into keys under the *compiler-options* var is present only inside the clojure.lang.Compile class. This is a problem when using a debugger inside an IDE and not AOT compiling; specifying -Dclojure.compiler.disable-locals-clearing=true has no effect here when it would be most useful!

Patch: CLJ-1274.patch
Screened by: Stu



 Comments   
Comment by Howard Lewis Ship [ 02/Oct/13 4:52 PM ]

Obviously, that's supposed to be *compiler-options*.

Comment by Howard Lewis Ship [ 02/Dec/13 4:03 PM ]

Changes initialization of *compiler-options* to occur statically inside Compiler; now available to all forms of Clojure, not just AOT compilation; however, the initial *compiler-options* value is now defined as a root binding, rather than a per-thread binding, which has slightly different semantics.

Comment by Stuart Halloway [ 27/Jun/14 1:45 PM ]

Patch is straightforward, marking screened.

I am left wondering if other options that are set only in Compile.java ought also to be moved.





[CLJ-1273] def discards ^:macro when used outside the top-level Created: 02/Oct/13  Updated: 17/Jan/14  Resolved: 17/Jan/14

Status: Closed
Project: Clojure
Component/s: None
Affects Version/s: Release 1.5
Fix Version/s: None

Type: Defect Priority: Major
Reporter: André Gustavo Rigon Assignee: Unassigned
Resolution: Duplicate Votes: 0
Labels: :macro, conditional, def, macro, metadata, top-level


 Description   

If I evaluate

(def ^:macro my-defn1 #'defn)

a macro named 'my-defn1' is defined, which I can use just like 'defn'.

However, if I evaluate instead

(if true
  (def ^:macro my-defn2 #'defn))

the var for 'my-defn2' doesn't have the :macro metadata set and I can't use it as a macro, even though the 'def' form is equal to the previous case.

Here is a complete example:

(def ^:macro my-defn1 #'defn)

(if true
  (def ^:macro my-defn2 #'defn))

(println (meta #'my-defn1))    ; => contains :macro

(println (meta #'my-defn2))    ; => doesn't contain :macro!

(my-defn1 hello1 []
          (println "hello 1"))

(hello1)                       ; => prints "hello 1"

(my-defn2 hello2 []            ; => CompilerException: Unable to resolve 
  (println "hello 2"))         ;    symbol: hello2 in this context


 Comments   
Comment by Gary Fredericks [ 09/Dec/13 1:31 PM ]

I hopped around the code for a while and all I could find is that the bindRoot method in Var.java intentionally clears :macro from the metadata.

Comment by Nicola Mometto [ 17/Jan/14 7:03 PM ]

Duplicate of http://dev.clojure.org/jira/browse/CLJ-1021





[CLJ-1271] Reduce protocol callsite overhead Created: 30/Sep/13  Updated: 08/Oct/13  Resolved: 08/Oct/13

Status: Resolved
Project: Clojure
Component/s: None
Affects Version/s: Release 1.6
Fix Version/s: Release 1.6

Type: Enhancement Priority: Critical
Reporter: Alex Miller Assignee: Rich Hickey
Resolution: Completed Votes: 0
Labels: None

Approval: Ok

 Description   

Placeholder for Rich to work on the unused instance member fields emitted by emitProto() in the compiler.






[CLJ-1270] Print control for Java collection types Created: 30/Sep/13  Updated: 18/Oct/13  Resolved: 18/Oct/13

Status: Closed
Project: Clojure
Component/s: None
Affects Version/s: None
Fix Version/s: Release 1.5

Type: Enhancement Priority: Critical
Reporter: Alex Miller Assignee: Stuart Halloway
Resolution: Duplicate Votes: 0
Labels: None

Attachments: Text File clj-1270-1.txt    
Approval: Vetted

 Description   

This feature is described at:
http://dev.clojure.org/display/design/Print+Control+for+Java+Collection+Types



 Comments   
Comment by Andy Fingerhut [ 14/Oct/13 2:08 AM ]

clj-1270-1.txt is just a quick swipe at changing the printing behavior for Java objects with classes implementing the java.util.Queue interface, so pr/pr-str shows them similarly to how they show Clojure lists and seqs. java.util.Set and java.util.List already pr/pr-str similarly to Clojure lists, vectors, or sets.

No change is made in this patch to the behavior of print/print-str, or to how other objects are printed whose classes implement java.util.Collection, but not also one of java.util.Set, .List, or .Queue.

A few tests are added to verify the desired behavior when print-length is set to a value other than nil.

Comment by Andy Fingerhut [ 14/Oct/13 11:24 AM ]

Just a comment: Clojure 1.5.1's behavior regarding print-readably and Java collections does seem a bit odd, perhaps backwards:

user=> (import '[java.util HashSet])
java.util.HashSet
user=> (def x (HashSet. (range 10)))
#'user/x
user=> (binding [*print-readably* true] (pr-str x))
"#{0 1 2 3 4 5 6 7 8 9}"
user=> (binding [*print-readably* false] (pr-str x))
"#<HashSet [0, 1, 2, 3, 4, 5, 6, 7, 8, 9]>"
Comment by Alex Miller [ 18/Oct/13 7:59 AM ]

Rich and Stu said this should be in scope for 1.6. Rich said that Stu would likely do the work.

Comment by Stuart Halloway [ 18/Oct/13 8:41 AM ]

This was actually implemented in 1.5. Andy's suggestion about Queue (see patch and https://groups.google.com/forum/#!topic/clojure-dev/q7h4QJCHEvw) could be considered as a separate feature request.

Comment by Stuart Halloway [ 18/Oct/13 8:42 AM ]

This work was done in commit 92b4fc76e59e68d3bdae4ebd5b8deec915cb7ab5

Comment by Alex Miller [ 18/Oct/13 9:21 AM ]

Ah, sweet - thanks Stu!





[CLJ-1269] RFC: Anonymous functions interaction with -> and ->> threading macros Created: 28/Sep/13  Updated: 09/Dec/13  Resolved: 09/Dec/13

Status: Closed
Project: Clojure
Component/s: None
Affects Version/s: Release 1.6
Fix Version/s: None

Type: Enhancement Priority: Minor
Reporter: Michael O. Church Assignee: Unassigned
Resolution: Declined Votes: 0
Labels: None


 Description   

This is more of a starting point for discussion than a feature request. It'd be easy to write and submit the patch, but I want to ask if it's a good idea, since it would alter the semantics of two core macros, '> and '>>; and if it is a good idea, what considerations need to be balanced.

For threading macros, I'd like to special-case forms that begin with 'fn and 'fn*. It's often useful (but maybe a bad idea; that's why I'd like to start the discussion) to use the threading macros in conjunction with anonymous functions in addition to forms, like so (contrived example):

(defn sigmoid [x] (-> x 
                      - 
                      Math/exp 
                      (+ 1) 
                      #(/ 1 %)))

This won't compile; #(/ 1 %) expands to the form

(fn* [p1__1389#] (/ 1 p1__1389#))

modulo gensym, of course. The threading macro, not treating fn* and fn specially, alters that to:

(fn* (clojure.core/-> (clojure.core/-> (clojure.core/-> x -) Math/exp) (+ 1)) [p1__1417#] (/ 1 p1__1417#))

which is a fn* with a non-symbol (illegal label) before its binding vector, raising an error.

Is this worth "fixing", or are the benefits to small to justify the added complexity of a special case in the ->, ->> threading macros?



 Comments   
Comment by Michael O. Church [ 29/Sep/13 8:52 AM ]

Thanks Alex!

Comment by Gary Fredericks [ 09/Dec/13 7:53 AM ]

Just one of the special cases you'd want to consider is when fn does not mean clojure.core/fn, which is certainly realistic.

Comment by Michael O. Church [ 09/Dec/13 10:03 AM ]

Yes, I realize that.

Looking at the ticket again, I retract my support of this change.

Here's why: (1) that "fix" would alter the semantics of the threading macros, which would break existing code, and (2) there's a really easy way to get anonymous functions into threading macros: just surround the

#(...)
expression with an extra set of parentheses.

I retract my request for this feature. It's easy enough to do this: instead of,

(-> x f #(g h %))

it's this:

(-> x f (#(g h %)))

Saving 2 characters, like so, is not worth a breaking change IMO.

Comment by Alex Miller [ 09/Dec/13 10:15 AM ]

Retracted by submitter





[CLJ-1268] Require Java 1.6 as minimum for Clojure Created: 28/Sep/13  Updated: 25/Oct/13  Resolved: 25/Oct/13

Status: Closed
Project: Clojure
Component/s: None
Affects Version/s: Release 1.6
Fix Version/s: Release 1.6

Type: Enhancement Priority: Blocker
Reporter: Alex Miller Assignee: Unassigned
Resolution: Completed Votes: 0
Labels: build

Attachments: Text File clj-1268.patch    
Patch: Code
Approval: Ok

 Description   

In Clojure 1.6, we plan to move to JDK 1.6 as the minimum JDK.

Patch: clj-1268.patch

This patch changes the build configurations for both Maven and Ant to assume JDK 1.6 as the "source" and "target" runtimes.

Configuration changes will be necessary on Hudson. We already build Clojure and contrib libraries on JDK 1.6 by default, but we will need to remove matrix test builds for JDK 1.5. See for example clojure-test-matrix and data.csv-test-matrix – coordinate with Stuart Sierra for this change.



 Comments   
Comment by Stuart Sierra [ 04/Oct/13 8:45 AM ]

Screened.

I have verified that both the Ant and Maven builds still work (running on JDK 1.7) and that the output .class files contain the bytecode header for JDK 1.6.





[CLJ-1267] Reader Bug for making vector of numbers Created: 26/Sep/13  Updated: 26/Sep/13  Resolved: 26/Sep/13

Status: Closed
Project: Clojure
Component/s: None
Affects Version/s: Release 1.5
Fix Version/s: None

Type: Defect Priority: Major
Reporter: Amin Razavi Assignee: Unassigned
Resolution: Declined Votes: 0
Labels: bug
Environment:

Windows 8 , Core i7



 Description   

When Trying To Make a Vector of Numbers Like 01 04 ... it's OK!
=> (vector 04)
; [4]
But When Trying To Make a Vector of "09" It Says :
NumberFormatException Invalid Number: 09



 Comments   
Comment by Alex Miller [ 26/Sep/13 10:56 AM ]

Numbers with a leading 0 are read as octal (valid digits = 0-7). Example:

> 0100
64

So, this is the expected behavior.

Comment by Amin Razavi [ 26/Sep/13 7:49 PM ]

shame on me , sorry.





[CLJ-1265] various clojure.core functions with ^:static metadata are missing doc strings Created: 23/Sep/13  Updated: 05/Feb/14  Resolved: 05/Feb/14

Status: Closed
Project: Clojure
Component/s: None
Affects Version/s: Release 1.6
Fix Version/s: None

Type: Defect Priority: Trivial
Reporter: Jozef Wagner Assignee: Unassigned
Resolution: Declined Votes: 0
Labels: None


 Description   

Several functions defined with legacy ^:static metadata are not shown in API documentation. Examples are chunk-cons or await1. This defect probably originates in autodoc.



 Comments   
Comment by Tom Faulhaber [ 23/Sep/13 9:25 AM ]

These functions have no doc strings. Autodoc doesn't include Vars without documentation by design.

I'll leave it to others to decide if the functions should have doc strings or not. If they are meant for general use, I would say that the answer is "yes."

Comment by Alex Miller [ 05/Feb/14 12:33 PM ]

The autodoc behavior is intentional. The omission of docstrings is intentional (these are internal functions).





[CLJ-1264] Minor change to Clojure source code to prevent warnings when compiled with JDK 8 Created: 17/Sep/13  Updated: 22/Nov/13  Resolved: 22/Nov/13

Status: Closed
Project: Clojure
Component/s: None
Affects Version/s: Release 1.5
Fix Version/s: Release 1.6

Type: Enhancement Priority: Critical
Reporter: Andy Fingerhut Assignee: Unassigned
Resolution: Completed Votes: 0
Labels: None
Environment:

JDK 8


Attachments: File clj-1264-1.diff     Text File clj-1264-1.txt    
Patch: Code
Approval: Ok

 Description   

When compiling the Clojure source code using the early access version of JDK 8 (I saw this with 1.8.0-ea-b103 in particular), there are 6 warnings produced because the character _ is used as an identifier in Java source code.

    [javac] /home/jafinger/clj/latest-clj/clojure/src/jvm/clojure/lang/PersistentHashMap.java:1089: warning: '_' used as an identifier
    [javac] 	Box _ = new Box(null);
    [javac] 	    ^
    [javac]   (use of '_' as an identifier might not be supported in releases after Java SE 8)

The warning implies that this is likely to continue to be a warning for the lifetime of JDK 8, and could become an error with later JDKs.

Eliminating these warnings is as simple as changing the identifier name used on those 6 lines of one Java source file.

Patch: clj-1264-1.diff

Screened by: Alex Miller



 Comments   
Comment by Andy Fingerhut [ 17/Sep/13 11:23 PM ]

Patch clj-1264-1.txt changes the identifier _ used on 6 lines of file PersistentHashMap.java to the name addedLeaf, which is used for a similar purpose elsewhere in the file.





[CLJ-1262] Set equality broken with BigInteger Created: 13/Sep/13  Updated: 13/Sep/13  Resolved: 13/Sep/13

Status: Closed
Project: Clojure
Component/s: None
Affects Version/s: Release 1.5
Fix Version/s: None

Type: Defect Priority: Major
Reporter: Gary Fredericks Assignee: Unassigned
Resolution: Declined Votes: 0
Labels: None


 Description   

The CLJ-1106 bug is still present when using BigInteger:

(let [n1 -5, n2 (BigInteger. "-5")]
  [(= n1 n2) (= #{n1} #{n2}) (= [n1] [n2])])
;; => [true false true]

Andy Fingerhut pointed out that this may be out of scope in the same way as CLJ-1036.



 Comments   
Comment by Alex Miller [ 13/Sep/13 1:14 PM ]

Yes, Rich considers hash/= consistency out of scope for BigInteger.





[CLJ-1261] Invalid defrecord results in exception attributed to namespace that imports namespace with defrecord Created: 12/Sep/13  Updated: 29/Aug/14  Resolved: 29/Aug/14

Status: Closed
Project: Clojure
Component/s: None
Affects Version/s: Release 1.5
Fix Version/s: Release 1.7

Type: Defect Priority: Minor
Reporter: Howard Lewis Ship Assignee: Unassigned
Resolution: Completed Votes: 0
Labels: defrecord, errormsgs

Attachments: File clj-1261-2.diff     File clj-1261-3.diff     File clj-1261-4.diff     File clj-1261-5.diff    
Patch: Code and Test
Approval: Ok

 Description   

I was introducing a namespace that included a defrecord.

My defrecord was wrong; it used a keyword to define a field, not a symbol. Minimal test case:

% cat src/useclj16/init.clj
(ns useclj16.init)

(defrecord Application [:shutdown-fn])
% cat src/useclj16/app.clj 
(ns useclj16.app
  (:require [useclj16.init :as init]))

However, the exception was perplexing:

% java -cp clojure-1.6.0-master-SNAPSHOT.jar:src clojure.main
user=> (require 'useclj16.app)
ClassCastException clojure.lang.Keyword cannot be cast to clojure.lang.IObj  clojure.core/with-meta (core.clj:214)

user=> (pst *e 100)
ClassCastException clojure.lang.Keyword cannot be cast to clojure.lang.IObj
        clojure.core/with-meta (core.clj:214)
        clojure.core/defrecord/fn--147 (core_deftype.clj:362)
        clojure.core/map/fn--4210 (core.clj:2494)
        clojure.lang.LazySeq.sval (LazySeq.java:42)
        clojure.lang.LazySeq.seq (LazySeq.java:60)
        clojure.lang.RT.seq (RT.java:484)
        clojure.lang.LazilyPersistentVector.create (LazilyPersistentVector.java:31)
        clojure.core/vec (core.clj:354)
        clojure.core/defrecord (core_deftype.clj:362)
        clojure.lang.Var.invoke (Var.java:427)
        clojure.lang.Var.applyTo (Var.java:532)
        clojure.lang.Compiler.macroexpand1 (Compiler.java:6483)
        clojure.lang.Compiler.macroexpand (Compiler.java:6544)
        clojure.lang.Compiler.eval (Compiler.java:6618)
        clojure.lang.Compiler.load (Compiler.java:7079)
        clojure.lang.RT.loadResourceScript (RT.java:370)
        clojure.lang.RT.loadResourceScript (RT.java:361)
        clojure.lang.RT.load (RT.java:440)
        clojure.lang.RT.load (RT.java:411)
        clojure.core/load/fn--5024 (core.clj:5546)
        clojure.core/load (core.clj:5545)
        clojure.core/load-one (core.clj:5352)
        clojure.core/load-lib/fn--4973 (core.clj:5391)
        clojure.core/load-lib (core.clj:5390)
        clojure.core/apply (core.clj:619)
        clojure.core/load-libs (core.clj:5429)
        clojure.core/apply (core.clj:619)
        clojure.core/require (core.clj:5512)
        useclj16.app/eval322/loading--4916--auto----323 (app.clj:1)
        useclj16.app/eval322 (app.clj:1)
        clojure.lang.Compiler.eval (Compiler.java:6634)
        clojure.lang.Compiler.eval (Compiler.java:6623)
        clojure.lang.Compiler.load (Compiler.java:7079)
        clojure.lang.RT.loadResourceScript (RT.java:370)
        clojure.lang.RT.loadResourceScript (RT.java:361)
        clojure.lang.RT.load (RT.java:440)
        clojure.lang.RT.load (RT.java:411)
        clojure.core/load/fn--5024 (core.clj:5546)
        clojure.core/load (core.clj:5545)
        clojure.core/load-one (core.clj:5352)
        clojure.core/load-lib/fn--4973 (core.clj:5391)
        clojure.core/load-lib (core.clj:5390)
        clojure.core/apply (core.clj:619)
        clojure.core/load-libs (core.clj:5429)
        clojure.core/apply (core.clj:619)
        clojure.core/require (core.clj:5512)
        user/eval318 (NO_SOURCE_FILE:1)
        clojure.lang.Compiler.eval (Compiler.java:6634)
        clojure.lang.Compiler.eval (Compiler.java:6597)
        clojure.core/eval (core.clj:2864)
        clojure.main/repl/read-eval-print--6594/fn--6597 (main.clj:260)
        clojure.main/repl/read-eval-print--6594 (main.clj:260)
        clojure.main/repl/fn--6603 (main.clj:278)
        clojure.main/repl (main.clj:278)
        clojure.main/repl-opt (main.clj:344)
        clojure.main/main (main.clj:442)
        clojure.lang.Var.invoke (Var.java:411)
        clojure.lang.Var.applyTo (Var.java:532)
        clojure.main.main (main.java:37)
nil

The error was attributed to app.clj (useclj16.app), a namespace which requires useclj16.init, the namespace containing the defrecord.

No indication that this concerned a defrecord, or even what namespace contained the error, was present in the exception.

Patch: clj-1261-5.diff

Approach: Check explicitly that the fields are all symbols, for both defrecord and deftype, and throw a CompilerException with file, line, and column number if not. Example of exception after patch is applied, in the case give above:

user=> (require 'useclj16.app)
CompilerException java.lang.AssertionError: defrecord and deftype fields must be symbols, useclj16.init.Application had: :shutdown-fn, compiling:(useclj16/init.clj:3:1)

Screened by: Alex Miller



 Comments   
Comment by Alex Miller [ 12/Sep/13 8:58 PM ]

Can you include an example of the defrecord definition just so we're clear what it looks like?

Comment by Alex Miller [ 12/Sep/13 8:59 PM ]

Also, "feedback" is not a useful label. Please use "errormsgs" for stuff like this. See the list of many commonly used labels here: http://dev.clojure.org/display/community/Creating+Tickets

Comment by Howard Lewis Ship [ 13/Sep/13 10:42 AM ]

"Feedback" is my own personal crusade http://tapestryjava.blogspot.com/2013/05/once-more-feedback-please.html

In my case, my invalid code was:

(defrecord Application [:shutdown-fn])

And the mistake was that :shutdown-fn should be a symbol, not a keyword.

Here it is, more completely:

(ns novate.services.initialization
  "Infrastructure for system-as-transient state.")

(defrecord Application [:shutdown-fn])

and

(ns novate.services.activator
  "Responsible for bootstrapping the application by loading certain namespaces and invoking certain functions, guided by data in JAR manifests."
  (:gen-class)
  (:require [clojure.edn :as edn]
            [clojure.java.io :as io]
            [novate.util.logging :as l]
            [novate.services
             [initialization :as init]
             [ordering :as o]])
  (:import [java.io PushbackReader]))

The error was attributed to this file.

Comment by Andy Fingerhut [ 13/Sep/13 11:48 AM ]

Patch clj-1261-v1.txt throws an exception if any fields given to defrecord or deftype are not symbols. They are CompilerExceptions, so include an accurate file, line, and column number.

Comment by Andy Fingerhut [ 13/Sep/13 11:57 AM ]

Updated description to give minimal test case.

Comment by Andy Fingerhut [ 23/Nov/13 1:06 AM ]

Patch clj-1261-2.diff is identical to clj-1261-v1.txt except that it applies cleanly to latest master. The only change was to some lines of context due to recent commits to Clojure.

Comment by Alex Miller [ 18/Apr/14 1:16 PM ]

I think the patch is ok but I have two suggestions in the error message - first, include the record/type ns+name (I think the classname in the patched fn is what you want). Second, I think the wording could be adjusted a bit and the parens should go away - those look like but don't actually have meaning in the original context (since you are filtering out the symbols). Maybe something like:

"defrecord and deftype fields must be symbols, useclj16.init.Application had: :shutdown-fn, :foo-bar"

Comment by Andy Fingerhut [ 30/Apr/14 9:32 AM ]

Patch clj-1261-3.diff attempts to incorporate Alex's suggested error message changes.

There are other errors caught by function validate-fields that could have more details like the namespace and record/type name added to them, but I don't want to go out of scope for the ticket. I can create another patch that does that if there is interest.

Comment by Alex Miller [ 30/Apr/14 2:56 PM ]

Can you update the "after" example in the Approach section of the description to match new?

Comment by Andy Fingerhut [ 01/May/14 4:18 PM ]

Updated example output at end of description to be what is seen after patch clj-1261-3.diff is applied.

Comment by Alex Miller [ 05/May/14 1:56 PM ]

Description looks good, patch looks good. Test?

Comment by Andy Fingerhut [ 05/May/14 2:03 PM ]

I'd be happy to write one, if I had a "similar" one to pattern them on.

By similar, I mean: are there any existing tests that require a namespace that isn't already loaded & compiled when the tests begin running, and catch exceptions thrown during the require?

Comment by Alex Miller [ 05/May/14 3:56 PM ]

I think you should be able to test the right error message here by just invoking the defrecord form.

Otherwise, maybe https://github.com/clojure/clojure/blob/master/test/clojure/test_clojure/ns_libs.clj#L87 ?

Comment by Andy Fingerhut [ 10/May/14 6:43 PM ]

Patch clj-1261-4.diff is identical to clj-1261-3.diff except that it adds a couple of unit tests verifying that an exception of the desired type and with an appropriate message is thrown when keywords are used as defrecord or deftype fields.

Comment by Alex Miller [ 13/May/14 10:41 PM ]

same as -4 but changed final defrecord to deftype in test (seemed like a typo)

Comment by Andy Fingerhut [ 13/May/14 11:40 PM ]

Thanks for the catch on that typo in the tests. You changed it to what I had intended.

Comment by Alex Miller [ 13/May/14 11:54 PM ]

seemed pretty clear





[CLJ-1260] ConcurrentModificationException thrown during action dispatching after commit in LockingTransaction.run() Created: 12/Sep/13  Updated: 21/Jan/14  Resolved: 21/Jan/14

Status: Closed
Project: Clojure
Component/s: None
Affects Version/s: Release 1.4, Release 1.5, Release 1.6
Fix Version/s: Release 1.6

Type: Defect Priority: Major
Reporter: Brandon Ibach Assignee: Unassigned
Resolution: Declined Votes: 0
Labels: STM
Environment:

Discovered on Ubuntu 12.04 with Oracle JDK 1.7.0_25 running Clojure 1.4. Test case produced on Windows 8 with Oracle JDK 1.7.0_09 running Clojure 1.4 and Clojure 1.5.1. Analysis seems to indicate that OS and Java version are not critical. Clojure 1.6 pre-release code has not been tested, but since clojure.lang.LockingTransaction has not changed since Clojure 1.4, it seems likely the defect is still present.


Attachments: File clj-1260.diff     File clj-1260-fixws.diff     Java Source File STMAgentInitBug.java    
Patch: Code
Approval: Vetted

 Description   

Summary

Using the Clojure 1.4 library strictly from Java code, a simple transaction dispatches an action to an Agent. When called from a simple driver, such as a unit test, where there is no interaction with the Clojure library/runtime (specifically, clojure.lang.RT), a ConcurrentModificationException is thrown from inside LockingTransaction.run() while it is iterating through the actions list, dispatching each action to its Agent after committing the transaction.

While the circumstances under which this occurs are probably fairly rare and a simple workaround exists (see final paragraph), thus the "Minor" priority, it seems like it would not be very complicated to fix LockingTransaction to handle the actions list more safely.

Analysis

Based on some debugging, here's what seems to be happening:

  1. Transaction A is run, dispatching action Z, which gets added, via LockingTransaction.enqueue(), to the actions list, which is a java.util.ArrayList<Agent.Action>.
  2. Transaction A completes and is successfully committed.
  3. LockingTransaction.run() does post-commit cleanup, freeing locks and putting a stop() to transaction A, which nulls the transaction's Info object reference.
  4. Notifications are sent and we start iterating the list of actions to be dispatched.
  5. The run() method calls Agent.dispatchAction(). Because the thread's transaction object is no longer considered to be "running" (due to the Info object being null) and no action is being processed on the thread (so its nested vector is null), the action is enqueue()-ed with the Agent.
  6. As part of the enqueue() process, the action is cons()-ed onto the Agent's ActionQueue. Here's where the unique circumstances come into play.
    1. At this point, we haven't really interacted with the Clojure runtime, specifically the clojure.lang.RT class, so its initiation machinery kicks in.
    2. Down in the depths, it executes transaction B to add a library to its list of loaded libraries.
    3. The still-existing-but-not-running thread-local transaction object fires up, runs and commits, with its existing, intact actions list, still containing action Z, enqueued during transaction A, which has not yet finished its post-commit process.
    4. The post-commit process for transaction B runs, including a nested attempt to dispatch action Z, again, which succeeds.
    5. The actions list is cleared before exiting the run() method.
  7. Upon returning way back up the stack to our not-quite-finished-post-processing transaction A, we continue iterating the now-cleared actions list, which promptly throws the ConcurrentModificationException.

A quick perusal of the LockingTransaction code shows that the only interaction with the actions list is adding an item to it in the LockingTransaction.enqueue() method, iterating it in the post-processing section of run() and clearing it in the finally clause of that section, so it's easy to see how a transaction started by any of the action-dispatching machinery would cause problems. Any such activity in the actions themselves would not be an issue, since they'd occur on other threads, but the dispatch stuff all runs on the same thread. The few moving parts that occur in this code seem fairly safe, as long as the runtime, clojure.lang.RT, is already initialized, but if that occurs during this phase, all bets appear to be off.

Test Case

The attached Java class can be compiled and run with just the Clojure 1.4 JAR on the class path. With the change described near the end of the file (comment one line and uncomment another), the Clojure 1.5.1 JAR can be used, instead, producing the same result.

A single Agent named count is created, holding an Integer value of 1. A transaction is run which dispatches an action (referred to as Z in the above description) that will increment the value of count to 2. Following this, another action is dispatched to count to enable monitoring the completion of the incrementing action. Lastly, the final value of count is printed before the application exits.

Running the class with no command-line arguments produces the above-mentioned exception and prints an incorrect final result, due to action Z being run a second time as described in step 6.4. Running with any command-line argument triggers a simple workaround that just references a static value from the clojure.lang.RT class, which invokes the class initialization before anything else happens, such that the exception is not thrown and the correct result is produced.

Patch: clj-1260-fixws.diff

Screened by:



 Comments   
Comment by Alex Miller [ 12/Sep/13 3:29 PM ]

Rich asked me to move this one to Vetted/Release 1.6 when it came in.

Comment by Alex Miller [ 12/Sep/13 3:31 PM ]

FYI: submitter does not have a CA

Comment by Christophe Grand [ 18/Oct/13 4:09 AM ]

Would a patch as simple as triggering the load of RT from the static init of LockingTransaction be ok? (assuming the analysis is correct)

Comment by Guillermo Winkler [ 18/Oct/13 1:07 PM ]

The ConcurrentModificationException is raised because the list of `actions` protocol is being violated.

Last clojure version happens in this loop in LockingTransaction.run(Callable) line: 361

for(Agent.Action action : actions)

{ Agent.dispatchAction(action); }

}

When RT hasn't been initialized before, first time it's used is in PersistentQueue's cons RT.list(o)

public PersistentQueue cons(Object o){ if(f == null) //empty return new PersistentQueue(meta(), cnt + 1, RT.list(o), null); else return new PersistentQueue(meta(), cnt + 1, f, (r != null ? r : PersistentVector.EMPTY).cons(o)); }

With the following call stack.

RT.doInit() line: 447
RT.<clinit>() line: 329
PersistentQueue.cons(Object) line: 127
PersistentQueue.cons(Object) line: 24
Agent.enqueue(Agent$Action) line: 264
Agent.dispatchAction(Agent$Action) line: 255
LockingTransaction.run(Callable) line: 369
LockingTransaction.runInTransaction(Callable) line: 231
STMAgentInitBug.main(String[]) line: 26

Problem is RT initialization triggers a new runInTransaction for new class loading (protocols in this callstack)

Thread [main] (Suspended (breakpoint at line 361 in LockingTransaction))
LockingTransaction.run(Callable) line: 361
LockingTransaction.runInTransaction(Callable) line: 231
protocols__init.load() line: 9
protocols__init.<clinit>() line: not available
Class<T>.forName0(String, boolean, ClassLoader) line: not available [native method]
Class<T>.forName(String, boolean, ClassLoader) line: 247
RT.loadClassForName(String) line: 2099
RT.load(String, boolean) line: 430
RT.load(String) line: 411
core.clj line: 5546
core.clj line: 5545
core$load(RestFn).invoke(Object) line: 408
core__init.load() line: 6174
core__init.<clinit>() line: not available
Class<T>.forName0(String, boolean, ClassLoader) line: not available [native method]
Class<T>.forName(String, boolean, ClassLoader) line: 247
RT.loadClassForName(String) line: 2099
RT.load(String, boolean) line: 430
RT.load(String) line: 411
RT.doInit() line: 447
RT.<clinit>() line: 329
PersistentQueue.cons(Object) line: 127
PersistentQueue.cons(Object) line: 24
Agent.enqueue(Agent$Action) line: 264
Agent.dispatchAction(Agent$Action) line: 255
LockingTransaction.run(Callable) line: 369
LockingTransaction.runInTransaction(Callable) line: 231
STMAgentInitBug.main(String[]) line: 26

So LockingTransaction:run depends indeed on RT to be loaded, since the method re-enters the function if loading is triggered from the inside.

Maybe actions:

final ArrayList<Agent.Action> actions = new ArrayList<Agent.Action>();

Should be better protected to prevent enqueuing during the dispatch loop?

Comment by Brandon Ibach [ 18/Oct/13 2:02 PM ]

The additional detail provided in the last comment is correct and matches what I observed. However, protecting the "actions" list against enqueuing isn't really the issue, since this problem would occur even if the "nested" transaction didn't involve any actions. The problem is that committing a transaction clears the "actions" list.

I think the suggestion to trigger initialization of RT in the static initializer of LockingTransaction would address this issue, though in a somewhat roundabout way. A more direct approach might be to make a temporary copy of the contents of the "actions" list prior to iterating it. Obviously, the latter would make extra work on every transaction commit, so it may not be desirable. I don't know what impact the former approach would have, other than the long-term effect of it being non-obvious to a later observer why that code is needed.

I haven't thought this through completely or done any tests, so my reasoning may be wrong, but one more argument for the approach of making a copy of the "actions" list is that a Ref notification/watch could run a transaction, which, being on the same thread, would also clear the "actions" list. This case wouldn't cause the exception, since the iteration of the list would not have started, yet, but it would cause actions enqueued by the "outer" transaction to not be dispatched.

Comment by Guillermo Winkler [ 18/Oct/13 2:26 PM ]

You're right, problem is not on enqueue are inner transactions clearing the list.

Anyway the protection can be a smarter way of iteration.

Wouldn't a possible solution be treating the action list as a Queue? Pop'in each action would also clear only that action.

Comment by Guillermo Winkler [ 18/Oct/13 2:57 PM ]

Attached patch seems to solve the problem in a cleaner way.

Comment by Andy Fingerhut [ 19/Oct/13 5:32 PM ]

Patch clj-1260-fixws.diff is identical to Guillermo's clj-1260.diff, except it eliminates warnings when applying the patch that were due to trailing white space.

Comment by Stuart Sierra [ 17/Jan/14 1:27 PM ]

Here's my understanding of what happens:

1. User's Java code calls LockingTransaction.runInTransaction.
2. In the transaction's Callable, user calls Agent.dispatch.
3. LockingTransaction adds the agent action to its actions list.
4. The transaction commits.
5. LockingTransaction begins iterating over actions and dispatching them.
6. Agent.dispatchAction checks to see if there is a running transaction, which there is not.
7. Agent.dispatchAction line 255 calls Agent.enqueue.
8. Agent.enqueue line 264 calls PersistentQueue.cons.
9. PersistentQueue.cons line 127 calls RT.list, triggering initization of RT.
10. RT's static initializer calls doInit.
11. doInit loads core.clj.
12. core.clj loads other files which contain ns declarations.
13. ns expands to code which commutes loaded-libs in a transaction.
14. The new transaction gets the same ThreadLocal LockingTransaction.
15. The new transaction executes and clears actions.
16. Back up the stack in the original transaction, LockingTransaction tries to continue iterating over actions and throws ConcurrentModificationException.

In short, this situation can only occur when the mere act of
enqueueing agent actions, not executing them, can create another
transaction on the same thread. This can happen when clojure.lang.RT
has not been initialized.

This would never occur in a Clojure program because RT is already
initialized when the Clojure program starts, and none of the Java code
involved in enqueueing agent actions will create new transactions.

The workaround for Java is trivial: just make sure clojure.lang.RT is
initialized before using any other Clojure classes.

However, the patch clj-1260-fixws.diff is a small and safe change. Its
author, Guillermo Winkler, has a CA.

The approach is to treat the actions list in LockingTransaction
as a queue, popping actions off one at a time, and never to
explicitly clear the queue.

The more important question is if/when RT should be initialized
automatically.

Comment by Guillermo Winkler [ 17/Jan/14 1:46 PM ]

Solving this problem by initializing RT feels to me like complecting two things not necessarily related, if PersistentQueue stops using RT, relationship disappears and you may still have the bug.

Anyway RT could be initialized safely so anyone that may depend on it finds it already there, avoiding potential clashes on initialization.

Comment by Alex Miller [ 17/Jan/14 1:49 PM ]

And following up on that, Clojure 1.6 has a new public Java API in clojure.java.api.Clojure. Loading that class or using any method on the class will serve to load clojure.lang.RT as a side effect.

Comment by Guillermo Winkler [ 21/Jan/14 9:11 AM ]

Alex, do you want a patch for the safe RT initialization here? Or do you prefer to handle it using a different ticket?

Comment by Alex Miller [ 21/Jan/14 9:32 AM ]

When you are using Clojure from Java, it is expected that you should properly initialize the Clojure runtime by (at least) loading the RT class (for Clojure <1.6) or the Clojure class (for Clojure >= 1.6).





[CLJ-1258] (apply and [false true]) gives CompilerException Created: 08/Sep/13  Updated: 09/Sep/13  Resolved: 09/Sep/13

Status: Closed
Project: Clojure
Component/s: None
Affects Version/s: Release 1.4, Release 1.5
Fix Version/s: None

Type: Defect Priority: Major
Reporter: Cynthia Qiu Assignee: Unassigned
Resolution: Declined Votes: 0
Labels: bug
Environment:

Ubuntu 13.04. I run 1.4.0 by "apt-get install", and 1.5.1 by "java -cp clojure-1.5.1.jar clojure.main".



 Description   

user=> (apply and [false true])
CompilerException java.lang.RuntimeException: Can't take value of a macro: #'clojure.core/and, compiling:(NO_SOURCE_PATH:1)
user=> (apply and (list false true))
CompilerException java.lang.RuntimeException: Can't take value of a macro: #'clojure.core/and, compiling:(NO_SOURCE_PATH:2)



 Comments   
Comment by Cynthia Qiu [ 08/Sep/13 10:18 PM ]

I got it. "and" is not a function but a special form. Sorry for that.

Comment by Andy Fingerhut [ 08/Sep/13 11:57 PM ]

Minor comment on yours: Clojure and other Lisps often distinguish between "special forms" and macros. I may be missing some of the distinction, but I am pretty sure the main part is that special forms are usually built into the implementation of Lisp, whereas macros are typically defined by the user via defmacro.

In any case, neither special forms nor macros may be used with apply, only functions.

Comment by Alex Miller [ 09/Sep/13 1:26 AM ]

As per Andy's comment, this is expected behavior when trying to apply to macros.





[CLJ-1252] Clojure reader (incorrectly) accepts keywords starting with a number Created: 04/Sep/13  Updated: 31/Oct/13  Resolved: 31/Oct/13

Status: Closed
Project: Clojure
Component/s: None
Affects Version/s: Release 1.5
Fix Version/s: Release 1.6

Type: Defect Priority: Minor
Reporter: Alex Miller Assignee: Unassigned
Resolution: Declined Votes: 0
Labels: reader

Attachments: Text File numkeyword.patch    
Patch: Code and Test
Approval: Screened

 Description   

The reader page at http://clojure.org/reader states that symbols (and keywords) cannot start with a number and the regex used in LispReader (and EdnReader) also has this intention. But:

user> :5
:5
user> (class *1)
clojure.lang.Keyword

Cause: The pattern for keywords is: "[:]?([\D&&[^/]].*/)?(/|[\D&&[^/]][^/]*)". The intention of the [\D&&[^/]] is to accept all non-digits except /. However, if the : matches and 5 does not, the regex will backtrack and unmatch the :, instead matching it in the non-digit charset. The whole match is sent on in the code and the : is stripped later.

Solution: Prevent back-tracking for the first : match by using the possessive quantifier ?+ instead of ?. Once the first : is matched, it will not backtrack to it. (This is also faster.) The patch makes this change in LispReader and EdnReader, updates a couple tests that were using number keywords, and adds a new negative test to check that :5 isn't read.

Patch: numkeyword.patch

Screened by: Alex Miller



 Comments   
Comment by Andy Fingerhut [ 05/Sep/13 5:25 PM ]

Given how far this one is along now, perhaps CLJ-1003 should be marked as a duplicate of this one?

Comment by Alex Miller [ 31/Oct/13 9:39 PM ]

Because this broke existing code (several lib projects like java.jdbc), we have rolled this back. Will follow up with an alternate ticket to address this in some other way.





[CLJ-1251] The update function: like update-in, for first level Created: 03/Sep/13  Updated: 29/Aug/14  Resolved: 29/Aug/14

Status: Closed
Project: Clojure
Component/s: None
Affects Version/s: Release 1.6
Fix Version/s: Release 1.7

Type: Enhancement Priority: Minor
Reporter: Michael O. Church Assignee: Ambrose Bonnaire-Sergeant
Resolution: Completed Votes: 3
Labels: None

Attachments: Text File CLJ-1251.patch     Text File update.patch    
Patch: Code and Test
Approval: Ok

 Description   

update-in is useful for updating nested structures. Very often we just want to update one level, so an update function optimised for this use case is useful.

It operates identically to update-in with a key path of length one so these are the same:

(update-in m [k] f args...)
(update m k f args...)

Patch: CLJ-1251.patch

Screened by: Alex Miller



 Comments   
Comment by Alex Miller [ 06/Sep/13 9:56 AM ]

I like this - kind of halfway between assoc and update-in.

Comment by Michael O. Church [ 07/Sep/13 12:41 PM ]

It's very useful. I assumed that its non-inclusion was for a reason (hence was hesitant to submit the patch) but it comes in handy a lot. One project I'd like to do with some free time is a library for turn-based strategy games, which use update frequently to express game-state changes.

The downside of this change is that 'update is probably a defined function in a good number of modules written by other people. IMO the strongest reason not to include it is that it's such a common name; but the benefits (in my view) outweigh the downsides.

Comment by Andy Fingerhut [ 14/Feb/14 11:50 AM ]

Patch update.patch dated Sep 3 2013 no longer applies cleanly to latest Clojure master as of Feb 14 2014. It did on Feb 7 2014. I haven't checked in detail, but this is probably simply due to some tests recently added to a test file that require updating some diff context lines.

Comment by Ambrose Bonnaire-Sergeant [ 06/May/14 2:36 PM ]

The vararg validation should be done in the same way as `assoc`.

Comment by Alan Malloy [ 06/May/14 2:41 PM ]

The most obvious reason, to me, that clojure.core/update doesn't exist already is that it's not clear what it should do when given more than 3 arguments. Consider, for example, (update m a b c d). What does this do? There are at least three reasonable interpretations: (update-in m [a] b c d), passing c and d as extra args to the function b; (-> m (update-in [a] b) (update-in [c] d)), treating the args as alternating key/function pairs; (reduce (fn [m k] (update-in m [k] a)) m [b c d]), treating a as a function to apply to each of b, c, and d.

Any of these are plausible meanings for the vague name "update", and there's no obvious behavior to choose, whereas there's only one reasonable way for assoc and assoc-in to behave. If one of them were chosen, it would be a little bit nontrivial to read code using it, at least until it became so well-known that everyone thinks it's obvious. I don't have anything against this function that Michael Church has written, or including it in core, but I don't like naming it update, as if it were the only possible dual to update-in.

Comment by Kyle Kingsbury [ 06/May/14 4:09 PM ]

I'd like to second Alan Malloy's concern; I've defined (update m k f arg1 arg2) in most of my Clojure work to be "change the value for this key to be (f current-value arg1 arg2 ...)"; this is consistent with swap!, update-in, etc., and is in my experience the most common need for update. It also composes well with swap! and other higher-order friends. I suggest we use that variant instead, and rely on assoc or -> threading when updating multiple fields.

Comment by Michael O. Church [ 07/May/14 10:32 AM ]

I agree with Kyle and Alan. There are several interpretations of how update should behave and while it's not clear which one is "correct", Kyle's is most consistent with the rest of the language and therefore probably more right than the one I started with.

The issue I see with including an "update" function is that it will break code for others who've defined it for themselves. Kyle's interpretation is more consistent with the rest of Clojure and will probably involve the least breakage. I'd be happy using his version, and renaming mine to something else.

Comment by Rich Hickey [ 13/May/14 6:09 AM ]

I am in favor, and it should work like everything else: (update m k f args...)

Comment by Ambrose Bonnaire-Sergeant [ 13/May/14 7:18 AM ]

I'm working on a new patch.

Comment by Ambrose Bonnaire-Sergeant [ 13/May/14 7:39 AM ]

update-like-update-in.patch is the new patch as Rich requests.

Comment by Alex Miller [ 13/May/14 8:56 AM ]

Ambrose, I think the example in the description no longer follows the (update m k f args...) form right? Can you update?

Comment by Ambrose Bonnaire-Sergeant [ 13/May/14 9:46 AM ]

Alex, I'm not sure what you're referencing?

Comment by Ambrose Bonnaire-Sergeant [ 13/May/14 9:47 AM ]

If you mean the docstring, I did try and update it for update by copying update-in and change and plural keys to singular.

Comment by Alex Miller [ 13/May/14 10:18 AM ]

I mean the description for this ticket needs to be updated to reflect what we are currently considering.

Comment by Alex Miller [ 13/May/14 12:57 PM ]

In the patch, the docstring has "If the key does not exist, a hash-map will be created." which is not applicable in update right? I think it would be more accurate to say that the fn will be invoked on nil.

This line occurs twice in the tests:

{:a [1 2]}   (update {:a [1]} :a conj 2)

There is no test for what happens when the key is absent. For example:

(update {:a 1} :b str)
=> {:b "", :a 1}
Comment by Ambrose Bonnaire-Sergeant [ 13/May/14 1:30 PM ]

I removed the mention of creating hash-maps, and replaced it with the explicit behaviour of passing `nil` for missing keys.

FWIW I proposed a similar wording in the patch for http://dev.clojure.org/jira/browse/CLJ-373

Added a test for missing key. Removed the duplicate test.

Comment by Gary Fredericks [ 16/May/14 8:45 PM ]

Is it worth unrolling several arities for the sake of premature optimization? e.g., https://github.com/Prismatic/plumbing/blob/master/src/plumbing/core.clj#L33-41

Comment by Alex Miller [ 22/May/14 8:14 AM ]

I think that's probably worth doing - who can update the patch with multiple arities?

Comment by Alex Miller [ 23/May/14 11:25 AM ]

Ambrose, can you (or anyone else really) update the patch to unroll small arities?

Comment by Ambrose Bonnaire-Sergeant [ 23/May/14 11:40 AM ]

Yes will do now.

Comment by Ambrose Bonnaire-Sergeant [ 23/May/14 12:16 PM ]

Add multiple arities + tests (CLJ-1251.patch)





[CLJ-1249] Warning when a record field with the same name as a function exists for it Created: 26/Aug/13  Updated: 31/Jan/14  Resolved: 31/Jan/14

Status: Closed
Project: Clojure
Component/s: None
Affects Version/s: Release 1.4, Release 1.5
Fix Version/s: None

Type: Enhancement Priority: Minor
Reporter: Benjamin Peter Assignee: Unassigned
Resolution: Declined Votes: 0
Labels: errormsgs
Environment:

Linux, clojure jar or leiningen repl



 Description   

Hi,

I had the following problem which took me much longer than it should have. I accidentally had a record's field with the same name as one of the functions from a protocol. When I tried to call the function from within the record I got totally weird behavior and didn't find it, until I removed every piece of code when I found the name conflict.

I wish clojure would warn me in such a case. (Disregarding any naming conventions that could have saved me.)

Following a small example to reproduce the problem:

(defprotocol HasPets
  (dogs [this])
  (cats [this])
  (octopus [this])
  (cute-ones [this]))

; Here the field "dog" is added with the same name as the protocol
(defrecord Petshop [dogs] 
  HasPets
  (dogs [this]
    [:pluto :bethoven])
  (cats [this]
    [:tom])
  (octopus [this]
    [:henry])
  (cute-ones [this]
    ; Here it was intended to call the function "dogs", instead the
    ; field is used.
    (concat (dogs this) (cats this))))

(def dogs-in-stock nil)

(let [petshop (->Petshop dogs-in-stock)]
  (println "Dogs for sale:" (dogs petshop))
  (println "All cute animals: " (cute-ones petshop)))

Results in:

clojure core.clj
Dogs for sale: [:pluto :bethoven]
Exception in thread "main" java.lang.NullPointerException
        at user.Petshop.cute_ones(core.clj:20)
        at user$eval84.invoke(core.clj:26)
        at clojure.lang.Compiler.eval(Compiler.java:6514)
        at clojure.lang.Compiler.load(Compiler.java:6955)
        at clojure.lang.Compiler.loadFile(Compiler.java:6915)
        at clojure.main$load_script.invoke(main.clj:283)
        at clojure.main$script_opt.invoke(main.clj:343)
        at clojure.main$main.doInvoke(main.clj:427)
        at clojure.lang.RestFn.invoke(RestFn.java:408)
        at clojure.lang.Var.invoke(Var.java:415)
        at clojure.lang.AFn.applyToHelper(AFn.java:161)
        at clojure.lang.Var.applyTo(Var.java:532)
        at clojure.main.main(main.java:37)

Expected warning:

Warning: the protocol function "dog" from "HasPets" conflicts with the equally named record field of "Petshop"

Without dog as a record's field:

clojure core.clj
Dogs for sale: [:pluto :bethoven]
All cute animals:  (:pluto :bethoven :tom)

Thanks for your help.

Ben.



 Comments   
Comment by Alex Miller [ 26/Aug/13 1:20 PM ]

Thanks for the report.

It seems like there is a scoping issue here where dogs is bound to the field and there is also a context for it being referred to as a function. It's possible that this should really be treated as a compiler bug and not a warning message problem - that requires some more detective work. Since there is only one function dogs (the field name is a function only in keyword form - :dogs), it seems unambiguous what should be done here.

In the meanwhile, presumably a workaround would be to use extend-protocol, etc to attach the protocol to the record outside the record definition.

Comment by Gary Fredericks [ 09/Dec/13 8:24 AM ]

Alex I can't see a lack of ambiguity here. I read your comment as saying that because dogs is in the call position we can deduce that it's meant to refer to the function rather than the field. But we can't assume that a field is not an IFn or intended to be used as one, so I can't make sense of that.

Another workaround should be using a fully qualified reference to the protocol function.

My expectation as a user of defrecord and deftype has always been that ambiguous references always refer to the fields, as if there were a let around each function body. So I've done this kind of thing intentionally I think. I don't know what that means about whether or not it should be a warning.

Should warnings for this kind of thing be accompanied by a way to suppress individual instances of the warning? E.g., a ^:no-warn metadata somewhere?

Comment by Stuart Halloway [ 31/Jan/14 2:41 PM ]

It is correct and legal code to have a record or type field shadow a var name, and as Gary mentions, the var is still reachable via a qualified name.

This might be a good warning for a linter like https://github.com/jonase/eastwood, if it is not present already.

Comment by Andy Fingerhut [ 31/Jan/14 2:55 PM ]

The Eastwood linter currently has no warning for this situation, but I have created an issue on its Github page to record the enhancement idea: https://github.com/jonase/eastwood/issues/55

Comment by Benjamin Peter [ 31/Jan/14 3:26 PM ]

Okay, thanks guys.





[CLJ-1248] Show type information in reflection warning messages when available Created: 24/Aug/13  Updated: 14/Feb/14  Resolved: 14/Feb/14

Status: Closed
Project: Clojure
Component/s: None
Affects Version/s: None
Fix Version/s: Release 1.6

Type: Enhancement