<< Back to previous view

[CLJ-1757] Inconsistent equals semantics for BigDecimal between = and .equals Created: 16/Jun/15  Updated: 22/Jun/15  Resolved: 22/Jun/15

Status: Closed
Project: Clojure
Component/s: None
Affects Version/s: Release 1.6, Release 1.7
Fix Version/s: None

Type: Defect Priority: Major
Reporter: Greg Mitchell Assignee: Unassigned
Resolution: Declined Votes: 1
Labels: math
Environment:

RHEL5, VirtualBox



 Description   

Numbers.equiv for BigDecimal uses compareTo to test equality instead of equals. This means that = and .equals have different results when the scale of the two BigDecimals are different. For example:
=> (= 1.0M 1.00M)
true
=> (.equals 1.0M 1.00M)
false

I see that another JIRA (http://dev.clojure.org/jira/browse/CLJ-1118) changed this behavior and asserts it in unit tests, but this seems like very incorrect behavior.

The motivation for this issue is a unit test using clojure.data/diff to verify that a test message is the same as a message generated with my platform's code. Our downstream customers care about the scale of the decimal, so Clojure's = operator saying two decimals with a different scale are equal caused a difficult-to-detect bug.

For reference, the line causing the issue is here: https://github.com/clojure/clojure/blob/master/src/jvm/clojure/lang/Numbers.java#L964
And a test asserting this behavior is here: https://github.com/clojure/clojure/blob/master/test/clojure/test_clojure/numbers.clj#L75
Javadoc for BigDecimal: https://docs.oracle.com/javase/7/docs/api/java/math/BigDecimal.html#equals(java.lang.Object)



 Comments   
Comment by Greg Mitchell [ 22/Jun/15 12:42 PM ]

I haven't seen any action on this Jira. Can I provide anything else to facilitate this? I'd be happy to hear other peoples' opinions on if this is actually a bug. My team has tried to work around this issue in a couple of different ways, but re-implementing a diff which doesn't use = is a headache.

Comment by Andy Fingerhut [ 22/Jun/15 1:25 PM ]

Just to set your expectations, unless the Clojure core team considers a bug or enhancement critical, it is not unusual for a ticket to go for months with no changes or comments.

clojure.core/= and Java .equals are different for numeric arguments in many cases, by design. For example, clojure.core/= is true for numerically equal Byte, Short, Integer, Long, and BigInteger arguments, whereas Java .equals returns false if the types are different. I know this is not the issue you are raising in this ticket – it is just an example of one of many ways in which these things are different from each other.

Have you considered creating a modified version of clojure.data/diff that compares BigDecimal values in the way you prefer?

Comment by Alex Miller [ 22/Jun/15 1:29 PM ]

Greg, I haven't managed to get definitive feedback from Rich on this so I have not been able to update it either way. Certainly CLJ-1118 went through his review prior to being committed.

Comment by Greg Mitchell [ 22/Jun/15 2:42 PM ]

Andy - thanks, I didn't realize that. I haven't submitted a Jira to Clojure before. You have a good point, = does more type-coercion work for numerics. I believe that this is qualitatively different because the scale has important informational content in financial and scientific computing. == exists for cases where the type is less relevant, and the linked Jira seems like a reasonable solution for == specifically. As mentioned in my comment, we have tried a couple of ways to re-implement clojure.data/diff, but none of the easy or obvious solutions work. It relies quite deeply on the value-semantics for collections as implemented by = that short-circuits logic to handle BigDecimals specially.

Alex - Understood, thank you for the update.

Comment by Andy Fingerhut [ 22/Jun/15 3:23 PM ]

Greg, agreed that there are cases where clojure.core/= and clojure.core/== differ today, e.g. (== 1 1.0) is true but (= 1 1.0) is false.

If you are arguing that Clojure should change so that (= 1.0M 1.00M) is false, but (== 1.0M 1.00M) is still true, that seems reasonable, and perhaps CLJ-1118 went too far by making them not only == but also = (my bad, if so).

Comment by Alex Miller [ 22/Jun/15 4:19 PM ]

Rich confirmed that the current behavior is desired and we do not plan to make the suggested change.





[CLJ-1756] clojure.java.shell/sh fails with jvm 1.8 Created: 16/Jun/15  Updated: 16/Jun/15  Resolved: 16/Jun/15

Status: Closed
Project: Clojure
Component/s: None
Affects Version/s: Release 1.6, Release 1.7, Release 1.8
Fix Version/s: None

Type: Defect Priority: Major
Reporter: john casu Assignee: Unassigned
Resolution: Declined Votes: 0
Labels: None
Environment:

nREPL server started on port 60186 on host 127.0.0.1 - nrepl://127.0.0.1:60186
REPL-y 0.3.5, nREPL 0.2.6
Clojure 1.6.0
OpenJDK 64-Bit Server VM 1.8.0_45-b13
Docs: (doc function-name-here)
(find-doc "part-of-name-here")
Source: (source function-name-here)
Javadoc: (javadoc java-object-or-class-here)
Exit: Control+D or (exit) or (quit)
Results: Stored in vars *1, *2, *3, an exception in *e



 Description   

user=> (clojure.java.shell/sh "ls /sys/block")

IOException error=2, No such file or directory java.lang.UNIXProcess.forkAndExec (UNIXProcess.java:-2)



 Comments   
Comment by Alex Miller [ 16/Jun/15 4:20 PM ]

I think you want:

(clojure.java.shell/sh "ls" "/sys/block")
Comment by john casu [ 16/Jun/15 4:32 PM ]

I'm such a dumbass.
thanks.
-john

Comment by Alex Miller [ 16/Jun/15 4:37 PM ]

Naw, it's confusing (and inherited from Java).





[CLJ-1753] VerifyError Expecting to find long on stack Created: 09/Jun/15  Updated: 09/Jun/15  Resolved: 09/Jun/15

Status: Closed
Project: Clojure
Component/s: None
Affects Version/s: Release 1.7
Fix Version/s: None

Type: Defect Priority: Major
Reporter: Gerrit Jansen van Vuuren Assignee: Unassigned
Resolution: Duplicate Votes: 0
Labels: bug


 Description   

The following functions cause a VerifyError when defining function m in the repl and via leiningen, it doesnt matter if AOT is enabled or not.
The cause is the function m's let statement (let [ts (select-ts msg)] ...)
If I take take out the select-ts's return type hint of ^long then everything works

(defn valid-ts? [^long ts] )
(defn ^long select-ts [msg] )

(defn m [msg] (let [ts (select-ts msg)] (when (valid-ts? ts) "hi")))

The full exception is:

java.lang.IllegalArgumentException: Unable to resolve classname: clojure.core$long@6df773ef
at clojure.lang.Compiler$HostExpr.tagToClass(Compiler.java:1069)
at clojure.lang.Compiler$InvokeExpr.getJavaClass(Compiler.java:3659)
at clojure.lang.Compiler$LocalBinding.hasJavaClass(Compiler.java:5657)
at clojure.lang.Compiler$LocalBindingExpr.hasJavaClass(Compiler.java:5751)
at clojure.lang.Compiler.maybePrimitiveType(Compiler.java:1283)
at clojure.lang.Compiler$MethodExpr.emitTypedArgs(Compiler.java:1336)
at clojure.lang.Compiler$InstanceMethodExpr.emit(Compiler.java:1523)
at clojure.lang.Compiler$IfExpr.doEmit(Compiler.java:2638)
at clojure.lang.Compiler$IfExpr.emit(Compiler.java:2613)
at clojure.lang.Compiler$BodyExpr.emit(Compiler.java:5826)
at clojure.lang.Compiler$LetExpr.doEmit(Compiler.java:6180)
at clojure.lang.Compiler$LetExpr.emit(Compiler.java:6133)
at clojure.lang.Compiler$BodyExpr.emit(Compiler.java:5826)
at clojure.lang.Compiler$FnMethod.doEmit(Compiler.java:5374)
at clojure.lang.Compiler$FnMethod.emit(Compiler.java:5232)
at clojure.lang.Compiler$FnExpr.emitMethods(Compiler.java:3771)
at clojure.lang.Compiler$ObjExpr.compile(Compiler.java:4410)
at clojure.lang.Compiler$FnExpr.parse(Compiler.java:3904)
at clojure.lang.Compiler.analyzeSeq(Compiler.java:6642)
at clojure.lang.Compiler.analyze(Compiler.java:6445)
at clojure.lang.Compiler.analyzeSeq(Compiler.java:6632)
at clojure.lang.Compiler.analyze(Compiler.java:6445)
at clojure.lang.Compiler.access$100(Compiler.java:38)
at clojure.lang.Compiler$DefExpr$Parser.parse(Compiler.java:538)
at clojure.lang.Compiler.analyzeSeq(Compiler.java:6644)
at clojure.lang.Compiler.analyze(Compiler.java:6445)
at clojure.lang.Compiler.analyze(Compiler.java:6406)
at clojure.lang.Compiler.eval(Compiler.java:6707)
at clojure.lang.Compiler.eval(Compiler.java:6666)
at clojure.core$eval.invoke(core.clj:2927)
at clojure.main$repl$read_eval_print_6625$fn_6628.invoke(main.clj:239)
at clojure.main$repl$read_eval_print__6625.invoke(main.clj:239)
at clojure.main$repl$fn__6634.invoke(main.clj:257)
at clojure.main$repl.doInvoke(main.clj:257)
at clojure.lang.RestFn.invoke(RestFn.java:1096)
at clojure.tools.nrepl.middleware.interruptible_eval$evaluate$fn__5879.invoke(interruptible_eval.clj:56)
at clojure.lang.AFn.applyToHelper(AFn.java:152)
at clojure.lang.AFn.applyTo(AFn.java:144)
at clojure.core$apply.invoke(core.clj:624)
at clojure.core$with_bindings_STAR_.doInvoke(core.clj:1862)
at clojure.lang.RestFn.invoke(RestFn.java:425)
at clojure.tools.nrepl.middleware.interruptible_eval$evaluate.invoke(interruptible_eval.clj:41)
at clojure.tools.nrepl.middleware.interruptible_eval$interruptible_eval$fn_5920$fn_5923.invoke(interruptible_eval.clj:171)
at clojure.core$comp$fn__4192.invoke(core.clj:2402)
at clojure.tools.nrepl.middleware.interruptible_eval$run_next$fn__5913.invoke(interruptible_eval.clj:138)
at clojure.lang.AFn.run(AFn.java:22)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)

VerifyError (class: user$m, method: invoke signature: (Ljava/lang/Object;)Ljava/lang/Object Expecting to find long on stack java.lang.Class.getDeclaredConstructors0 (Class.java:-2)



 Comments   
Comment by Alex Miller [ 09/Jun/15 4:21 PM ]

Maybe

(defn select-ts ^long [msg])

instead?

Comment by Alex Miller [ 09/Jun/15 4:23 PM ]

Where you have the ^long hint, it's resolved to the function clojure.core/long and the function is used as the typehint instead. We've got some other tickets to improve the feedback in this case.

Try:

(meta #'select-ts)

to see the meta better.

Comment by Gerrit Jansen van Vuuren [ 09/Jun/15 4:26 PM ]

that works.

what is the difference between (defn ^long select-ts [msg] ) and (defn select-ts ^long [msg]) ?

Comment by Alex Miller [ 09/Jun/15 4:28 PM ]

See CLJ-1674 for duplicate case w/boolean.

Comment by Gerrit Jansen van Vuuren [ 09/Jun/15 4:36 PM ]

thanks this explains it.





[CLJ-1751] realized? does not work on LongRange Created: 09/Jun/15  Updated: 09/Jun/15  Resolved: 09/Jun/15

Status: Closed
Project: Clojure
Component/s: None
Affects Version/s: Release 1.7
Fix Version/s: None

Type: Defect Priority: Major
Reporter: Logan Linn Assignee: Unassigned
Resolution: Duplicate Votes: 0
Labels: None


 Description   

In pre-1.7 versions, all calls to range produced a LazySeq, but in 1.7.0-beta1 and up, if you call range with a end/start/step, you now get back a clojure.lang.LongRange which does not implement clojure.lang.IPending. Calling realized? on it throws exception.

Clojure 1.6.0
user=> (realized? (range 10))
false
Clojure 1.7.0-RC1

user=> (realized? (range 10))

ClassCastException clojure.lang.LongRange cannot be cast to clojure.lang.IPending  clojure.core/realized? (core.clj:7224)

clojure.lang.LongRange should implement clojure.lang.IPending



 Comments   
Comment by Alex Miller [ 09/Jun/15 1:12 PM ]

This is intentional and there have been a number of discussions and tickets about it already. I have chosen to interpret the meaning of whether a lazy seq is realized as whether the "first" value has been computed and can be returned without computation.

[Another possible interpretation is that a lazy seq is realized when there is a next. This is confusing because both the current value and the next node are forced at the same time as a result of most operations in LazySeq.]

For range, if a LongRange instance exists, its first value has been computed and is available. Thus it is not "pending" and always realized.

In general, sequences may be either IPending, or not. realized? only operates on IPending instances. Thus existing code that generically deals with sequences must have code like this (or it would be throwing ClassCastExceptions):

(defn seq-realized? [s]
  (if (instance? clojure.lang.IPending s)
    (realized? s)
    true))

Although this particular concrete range type has changed in terms of what it supports, generic code like that above will continue to work as before. cycle is the other case in 1.7 that falls into this case. Note that (range) is backed by iterate, which is IPending, so the answer there is actually different.

A separate, likely useful question is: should realized? return true for an instance that is not IPending? If so, then realized? could be used without the guard above. I don't believe that's been filed, but I'd support that change.

Also see: CLJ-1726

Comment by Logan Linn [ 09/Jun/15 1:57 PM ]

Thanks for the fast response, Alex. My apologies for not finding previous discussions prior to creating ticket.

Your points make sense and I realized (no pun intended) after I opened this that I shouldn't have specifically requested LongRange to implement IPending, but that we address behavior of realized? on non-infinite range. I'll file a separate ticket.





[CLJ-1749] evaluate quote with wrong number of arity will not throw any exception Created: 07/Jun/15  Updated: 08/Jun/15  Resolved: 07/Jun/15

Status: Closed
Project: Clojure
Component/s: None
Affects Version/s: Release 1.6, Release 1.7
Fix Version/s: None

Type: Defect Priority: Minor
Reporter: Di Xu Assignee: Unassigned
Resolution: Duplicate Votes: 0
Labels: bug


 Description   

http://clojure.org/special_forms#Special Forms--(quote form)
doc says it should only accept one arity, and it also doesn't make sense if passes multiple arities, but in v1.6 and v1.7

user=> (quote 1 2 3)
1

I think it should throw exception, right?



 Comments   
Comment by Di Xu [ 07/Jun/15 9:39 PM ]

ps. can you guys fix hashtag in http://clojure.org/special_forms it's really hard to share

Comment by Alex Miller [ 07/Jun/15 9:40 PM ]

Dupe of CLJ-1282.

Comment by Alex Miller [ 08/Jun/15 1:10 AM ]

Re the link anchors on the special forms page, the table of contents at the top is actually generated from the headers, so I can't really change it unless I altered the headers more than I'd like. There are actually separate anchors embedded in the page like http://clojure.org/special_forms#quote which will work as well though.

Comment by Di Xu [ 08/Jun/15 1:18 AM ]

ok, thanks for that info. because I usually click link in the toc, and share that url, it's not that convenient.





[CLJ-1745] Some compiler exceptions wrapped in new CompilerException Created: 04/Jun/15  Updated: 17/Jun/15  Resolved: 17/Jun/15

Status: Closed
Project: Clojure
Component/s: None
Affects Version/s: Release 1.7
Fix Version/s: Release 1.7

Type: Defect Priority: Minor
Reporter: Andy Fingerhut Assignee: Unassigned
Resolution: Completed Votes: 0
Labels: compiler, error-reporting, regression

Attachments: Text File clj-1745.patch    
Patch: Code and Test
Approval: Ok

 Description   

Clojure error reporting changed in CLJ-1169 to wrap exceptions thrown during macro evaluation in CompilerException to give more input:

Clojure 1.6

(defmacro demo [] (throw (ex-info "fail" {})))
(demo)
ExceptionInfo fail  clojure.core/ex-info (core.clj:4403)
(class *e)
=> clojure.lang.ExceptionInfo

Clojure 1.7.0-alpha2 to 1.7.0-RC1

(defmacro demo [] (throw (ex-info "fail" {})))
(demo)
ExceptionInfo fail  clojure.core/ex-info (core.clj:4403)
;; NOTE: lein repl will instead print: CompilerException clojure.lang.ExceptionInfo: fail {}, compiling:(form-init8304622754337237403.clj:1:1)
(class *e)
=> clojure.lang.Compiler$CompilerException

This change has caused some breakage for users that throw exceptions in macros and expect to see the same exception at the top of the exception chain, not wrapped in a CompilerException. This change is somewhat masked in the Clojure REPL because clojure.main/root-cause unwraps CompilerException wrappers and prints the root cause when an exception occurs.

More background can be found in some messages on:
https://groups.google.com/d/msg/clojure/ccZuKTYKDPc/xpaz44UDqYwJ

Approach: The attached patch rolls back most of the change for CLJ-1169, specifically the part that wraps exceptions in CompilerException and the tests that were affected by this change (good examples of the kind of breakage others are seeing). I left the parts of CLJ-1169 that added quotes in the error message and those equivalent tests.

Patch: clj-1745.patch



 Comments   
Comment by Stuart Halloway [ 04/Jun/15 7:15 PM ]

All the stuff about lein in this ticket is just noise, and I am removing it. (Please don't use the phrase "small reproducing case" for anything that includes lein.) Clojure's behavior changed: improved error reporting. Code that explicitly relied on less-good error reporting broke.

Comment by Alex Miller [ 05/Jun/15 7:50 AM ]

Pulling this into 1.7 just for tracking discussion.

Comment by Alex Miller [ 05/Jun/15 4:02 PM ]

There is one confusing factor in replicating this re lein vs Clojure. The Clojure REPL will unpeel CompilerExceptions (see clojure.main/root-cause) so the repl actually prints the same in 1.7 as in 1.6 (but the exception chain and class are wrapped in one more CompilerException than before). Leiningen's repl will actually show the full structure.

Comment by Alex Miller [ 05/Jun/15 4:36 PM ]

I went back and looked at CLJ-1169 and while I think the intentions are good there, I do think that wrapping exceptions that happen to be thrown out of macro bodies like this does create unexpected (certainly different) behavior. I've attached a patch that rolls back most of the CLJ-1169 change.





[CLJ-1742] Eduction doesn't implement IReduce Created: 01/Jun/15  Updated: 01/Jun/15  Resolved: 01/Jun/15

Status: Closed
Project: Clojure
Component/s: None
Affects Version/s: Release 1.7
Fix Version/s: None

Type: Defect Priority: Major
Reporter: Leon Grapenthin Assignee: Unassigned
Resolution: Declined Votes: 0
Labels: None

Attachments: Text File eduction.patch    
Patch: Code

 Description   

Eduction doesn't implement IReduce, hence

(reduce + (eduction [])) throws.



 Comments   
Comment by Ghadi Shayban [ 01/Jun/15 11:51 AM ]

I think this is by design. It has been brought up before that IReduce semantics aren't clear.

Comment by Alex Miller [ 01/Jun/15 1:05 PM ]

This is intentional. We are trying to lessen usage of reduce without init.





[CLJ-1740] Reader conditional #?@ only reads first form in some cases Created: 29/May/15  Updated: 29/May/15  Resolved: 29/May/15

Status: Closed
Project: Clojure
Component/s: None
Affects Version/s: Release 1.7
Fix Version/s: None

Type: Defect Priority: Major
Reporter: Karsten Schmidt Assignee: Unassigned
Resolution: Duplicate Votes: 0
Labels: bug, readerconditionals
Environment:

clojure1.7.0-beta3, java 1.8.0_25, osx 10.8.5



 Description   

The unsplicing reader conditional #?@ only reads the first form if it is read top-level:

(read-string {:read-cond :allow} "#?@(:clj [:a :b])")
;; :a

Whereas this produces the correct/expected result:

(read-string {:read-cond :allow} "(do #?@(:clj [:a :b]))")
;; (do :a :b)


 Comments   
Comment by Alex Miller [ 29/May/15 8:59 AM ]

This is a dupe of CLJ-1706 which was fixed in 1.7.0-RC1.

Comment by Karsten Schmidt [ 29/May/15 9:09 AM ]

Sorry, Alex! I did search for "reader conditional" but nothing relevant showed up... glad it's fixed! Thx!





[CLJ-1739] Broken set equality for sets of equal sets Created: 28/May/15  Updated: 28/May/15  Resolved: 28/May/15

Status: Closed
Project: Clojure
Component/s: None
Affects Version/s: Release 1.6, Release 1.7
Fix Version/s: None

Type: Defect Priority: Major
Reporter: Tassilo Horn Assignee: Unassigned
Resolution: Duplicate Votes: 0
Labels: collections, interop


 Description   

With both clojure 1.6.0 and 1.7.0-RC1 I get the following inconsistent behavior.

Different kinds of sets are equal which is expected:

(= #{1 2 3} (flatland.ordered.set/ordered-set 1 2 3)) ;=> true
(= #{1 2 3} (java.util.HashSet. [1 2 3]))             ;=> true

However, sets containing equal sets are not equal:

(= #{#{1 2 3}} #{(flatland.ordered.set/ordered-set 1 2 3)}) ;=> false
(= #{#{1 2 3}} #{(java.util.HashSet. [1 2 3])})             ;=> false

This is similar to http://dev.clojure.org/jira/browse/CLJ-1649 and probably caused by http://dev.clojure.org/jira/browse/CLJ-1372.



 Comments   
Comment by Alex Miller [ 28/May/15 6:14 AM ]

This is a duplicate version of the problem in CLJ-1372. = will check that the persistent collection contains each element from the other collection, but because the hash codes are different for the Java version of the set, the element is not found and equality fails.

user=> (contains? #{#{1 2 3}} #{1 2 3})
true
user=> (contains? #{#{1 2 3}} (java.util.HashSet. [1 2 3]))
false
Comment by Andy Fingerhut [ 28/May/15 10:13 AM ]

There could be an issue where flatland.ordered.set/ordered-set is still using a pre-Clojure-1.6.0 hash function, and should be updated.

Comment by Tassilo Horn [ 28/May/15 2:26 PM ]

Andy, that's its hashCode implementation: https://github.com/flatland/ordered/blob/develop/src/flatland/ordered/set.clj#L52

Comment by Andy Fingerhut [ 28/May/15 2:40 PM ]

Yeah, that was correct with Clojure 1.5.1 and earlier. With Clojure 1.6.0, it should look more like what data.avl's was updated to around the time Clojure 1.6.0 was released, here: https://github.com/clojure/data.avl/blob/master/src/main/clojure/clojure/data/avl.clj#L53-L57

I will file an issue for ordered-set.

Comment by Andy Fingerhut [ 28/May/15 2:43 PM ]

https://github.com/amalloy/ordered/issues/16 was already filed recently. I added a comment with the same link to data.avl example there.

Comment by Andy Fingerhut [ 28/May/15 4:47 PM ]

I've created a PR for the issue with amalloy/ordered sets and maps: https://github.com/amalloy/ordered/pull/18





[CLJ-1738] Document that seqs are incompatible with Java iterators that return the same mutable object every time Created: 27/May/15  Updated: 19/Jun/15  Resolved: 17/Jun/15

Status: Closed
Project: Clojure
Component/s: None
Affects Version/s: Release 1.7
Fix Version/s: Release 1.7

Type: Enhancement Priority: Major
Reporter: Alex Miller Assignee: Stuart Halloway
Resolution: Completed Votes: 0
Labels: regression
Environment:

1.7.0-RC1


Attachments: Text File clj-1738-2.patch     Text File clj-1738-3.patch     Text File clj-1738-4.patch     Text File clj-1738-doc.patch     Text File clj-1738.patch    
Patch: Code
Approval: Ok

 Description   

Some Java libraries return iterators that return the same mutable object on every call:

  • Hadoop ReduceContextImpl$ValueIterator
  • Mahout DenseVector$AllIterator/NonDefaultIterator
  • LensKit FastIterators

While careful usage of seq or iterator-seq over these iterators worked in the past, that is no longer true as of the changes in CLJ-1669 - iterator-seq now produces a chunked sequence. Because next() is called 32 times on the iterator before the first value can be retrieved from the seq, and the same mutable object is returned every time, code on iterators like this now receives different (incorrect) results.

Approach: Sequences cache values and are thus incompatible with holding mutable and mutating Java objects. We will add some clarification about this to seq and iterator-seq docstrings. For those iterators above, it is recommended to either process those iterators in a loop/recur or to wrap them in a lazy-seq that transforms each re-returned mutable object into a proper value prior to caching.

Patch: clj-1738-doc.patch



 Comments   
Comment by Alex Miller [ 27/May/15 7:00 AM ]

I spot-checked some of the perf timings from CLJ-1669 and didn't see anything unexpected.

Comment by Marshall T. Vandegrift [ 27/May/15 7:38 AM ]

In order to maintain compatibility it is also necessary to change `clojure.lang.RT/seqFrom` back to creating non-chunked `IteratorSeq`s. I've verified that these changes are sufficient to restore compatibility for my cases.

Comment by Marshall T. Vandegrift [ 27/May/15 10:05 AM ]

Added updated version of proposed patch which covers RT Iterable->seq coercion and includes a test case.

Comment by Alex Miller [ 01/Jun/15 6:39 AM ]

The seqFrom change is good but I'd prefer not to add the Java class in the test. Can you replace that with a deftype implementing Iterable to reify an Iterator?

Comment by Marshall T. Vandegrift [ 01/Jun/15 10:32 AM ]

Added updated version of patch with pure-Clojure implementation of mutation-based iterator test.

Comment by Alex Miller [ 05/Jun/15 9:12 AM ]

I re-ran the full perf tests from CLJ-1669 and did not see any real changes except for in the last sort over eduction ones. We should still be seeing chunked iterator sequences over eductions which was the primary intent of the original change. We've just fallen back to non-chunked as we had before in the general case.

Comment by Alex Miller [ 05/Jun/15 2:39 PM ]

I think this looks good but since I had a hand in the early development of the patch I'm going to suggest that Stu screen it.

Comment by Alex Miller [ 09/Jun/15 1:18 PM ]

Marshall, can you update the patch so eduction's docstring says "reducible/iterable/seqable" and "reduce/iterator/seq"?

Comment by Marshall T. Vandegrift [ 09/Jun/15 3:08 PM ]

No problem. Updated and attached, but I've also changed the patch author to myself and fleshed out the commit message – if I'm going to do the drudge work I might as well take the credit too!

Comment by Alex Miller [ 09/Jun/15 3:16 PM ]

No problem at all - thanks!

Comment by Alex Miller [ 17/Jun/15 9:33 AM ]

Direction of this ticket changed at Rich's request.

Prior description capture here:

Clojure code that uses iterator-seq to wrap Java iterators that return the same mutable object on every call are broken by the chunked iterator-seq changes from CLJ-1669.

Some examples where this occurs:

  • Hadoop ReduceContextImpl$ValueIterator
  • Mahout DenseVector$AllIterator/NonDefaultIterator
  • LensKit FastIterators

Cause: In 1.6, the iterator-seq wrapper could be used with these to consume a sequence over these iterators element-by-element. In 1.7 RC1, iterator-seq produces a chunked sequence. Because next() is called 32 times on the iterator before the first value can be retrieved from the seq, and the same mutable object is returned every time, code doing this now receives different (incorrect) results.

Approach: Switch iterator-seq back to non-chunked and change eduction to use the chunking iterator-seq strategy as that was the original target. Retain the use of the chunked iterator seq in sequence over the TransformerIterator.

Patch: clj-1738-4.patch

Comment by Marshall T. Vandegrift [ 17/Jun/15 9:57 AM ]

Sorry, what just happened here? Is this no longer being fixed?

Comment by Alex Miller [ 17/Jun/15 10:06 AM ]

Hey Marshall, I thought you might have some questions. As noted above, Rich decided that this should not be a valid usage of seqs over these kinds of iterators (even though your usage happened to suffice in the past). So, you should alter your code to use these iterators in a different way.

Comment by Marshall T. Vandegrift [ 17/Jun/15 10:08 AM ]

Will there be a "breaking changes" section in the release notes for 1.7?

Comment by Alex Miller [ 17/Jun/15 11:20 AM ]

I will add a compatibility section. In this case, it should be considered "already broken" (but you're just now aware of it) I think.

Comment by Mike Rodriguez [ 17/Jun/15 10:09 PM ]

The main question I have is what is the proposed alternative way to interact with these object reusing iterators? I struggle to see what Clojure functions are safe to use on them because anything that internally calls seq must be avoided.

Comment by Alex Miller [ 18/Jun/15 5:23 AM ]

I would say that you shouldn't expect any sequence functions (all of which coerce to seq) to give you useful results. Instead, either consume the iterator in a loop/recur or create a lazy-seq that transforms each re-returned mutable object into a proper value prior to caching.

Comment by Marshall T. Vandegrift [ 18/Jun/15 7:11 AM ]

I expect at this point it isn't possible to change in Clojure/core's mind, but, Alex, your last comment crystallized my specific objection to this change.

You suggest "create a lazy-seq that transforms each re-returned mutable object into a proper value prior to caching." When the seq element-realization semantics match the element-at-a-time `Iterator` element-realization semantics there's an exact function for this: `map`. Specifically that this change breaks existing, working instances of the pattern `(map get-value iterable)` to me clearly demonstrates that the change is not compatible with the semantics of the `Iterator` interface. The fact that the newly-brokenness of the pattern is so non-obvious just emphasizes the point.

An unknown amount of deployed code in the Clojure ecosystem, and a non-trivial amount in my own code bases, are currently using this pattern to handle mutating-element Iterators. From my own tally of used Java-ecosystem libraries which include this pattern, I believe the mutating-Iterator case tmore common than Clojure/core apparently expect. For myself and all the other developers using Clojure to orchestrate large and obtuse Java frameworks, I plea for compatibility.

Comment by Alex Miller [ 18/Jun/15 8:32 AM ]

"When the seq element-realization semantics match the element-at-a-time `Iterator` element-realization semantics" makes an incorrect assumption. As a general guideline, code that relies on how many or when elements of a lazy seq are realized is fragile - Clojure does not make guarantees about that.

Additionally, there is another assumption that the iterator seq will only be traversed once, as you will get different and incorrect results after the first time.

Use of a seq built on this kind of iterator violated these assumptions, even if it happened to work.

Comment by Marshall T. Vandegrift [ 18/Jun/15 8:47 AM ]

I respectfully disagree.

"As a general guideline, code that relies on how many or when elements of a lazy seq are realized is fragile - Clojure does not make guarantees about that." This is true about seqs in general, but the Iterator interface does guarantee that a single element at a time is realized. I strong believe that a correct interop abstraction for generating seqs from Iterators must maintain this guarantee. I'm not making a claim about seqs in general, just for Iterator->seq coercion in order to maintain the semantics of the underlying Iterator and thus provide a useful & correct interop facility.

"Additionally, there is another assumption that the iterator seq will only be traversed once, as you will get different and incorrect results after the first time." Either seqs cache or they don't, yes? I don't believe it is coherent to argue both that mutation is incompatible due to caching and that `map`ing is incompatible because there might not be caching.

Comment by Alex Miller [ 18/Jun/15 8:56 AM ]

The issue is with iterators that return elements that aren't values. Seqs cache values. If you're not using values as elements, then you are outside the bounds of what is supported.

Comment by Marshall T. Vandegrift [ 18/Jun/15 9:40 AM ]

I agree that's their primary intent, but then why functions like `dorun`? For most of Clojure's history seqs have been the primary abstraction for composable iteration over linear collections. With Clojure 1.7 in particular introducing a variety of finer-grained abstractions, I agree this more sharply defines the primary/optimal use for seqs. But this shouldn't come at the cost of invalidating existing code which uses only public interfaces or introducing mismatches with fundamental host platform abstractions like Iterator.

Comment by Mike Rodriguez [ 18/Jun/15 10:28 AM ]

"This is true about seqs in general, but the Iterator interface does guarantee that a single element at a time is realized."

-
This part makes it look like seqs and the Iterator interface are not compatible with one another and Clojure is just pretending they can be.

-
Having a chunking behavior paired with Java Iterators is going to be unreliable because the caller hasn't had a chance to see intermediate elements as they were consumed from the Iterator.

I'm still having difficulty in trying to understand how to interop with an API. My particular case is the (very popular) Hadoop ReduceContextImpl$ValueIterator. I tend to agree with Clojure's strong stance on values and against mutable state like this iterator uses. However, Hadoop apparently has done this from a practical standpoint where the cost of a very large number of object allocations outweighed the cost of adding the mutable state complexity. In this case, Hadoop still did uphold the contract for an Iterator and it made sense for consumers to deal with it against that contract.
When this enters Clojure, we may wrongfully interact with this Iterator as a chunking seq, when it really is not going to be match.

I've been using Clojure for a few years now in my full-time work and this scares me only because I struggle to know what functions I call that may inadvertently "chunk" the Iterator I'm interop'ing with from Java. If I had more clarity on that issue, I may be more comfortable with this. I still don't think the Iterator iterface should really be treated as "chunkable" with by seq though.

Comment by Alex Miller [ 18/Jun/15 11:08 AM ]

There are two ways to interop with an iterator like this - consume it in a loop/recur, or wrap it in a lazy seq. The latter is more similar to whatever you were already doing, so you'd want something like:

(defn iter-seq [iter f]
  (if (.hasNext iter)
    (lazy-seq
      (cons (f (.next iter))
            (iter-seq iter f)))))

which applies a function f to convert from the mutable thing returned from iter to a value. Apply this before doing anything else, then use the result as a normal seq.

Example using the mutating-iterable in the clj-1738-4.patch and AtomicLong.get() as f:

(let [mi (mutating-iterable 10)
      iter (.iterator mi)
      s (iter-seq iter #(.get %))]
    (println s)    ;; (0 1 2 3 4 5 6 7 8 9)
    (println s))   ;; (0 1 2 3 4 5 6 7 8 9)

Again, the real problem here is having a seq that contains mutating objects instead of values. Chunking just exposes that as the problem. If you care about whether chunking happens, then something is wrong.

Comment by Mike Rodriguez [ 18/Jun/15 12:11 PM ]

Thanks Alex. I appreciate the feedback. I certainly think this is valuable and a technique that I will keep in mind to avoid these issues.
My current real-world issue is in my usage of the Hadoop Iterable that has the mutating Iterator behind it. I have currently be using a `reduce` over this Iterable.
In this case, I believe I am safe since `reduce` operates at a different abstraction than the seq abstraction. I believe that is still a correct way to deal with this, but let me know if I'm mistaken.

For completeness, I'd like to make one more point on this topic in regards to "If you care about whether chunking happens, then something is wrong" with respect to Iterators.

I do not think mutability is the only concern of the element-at-a-time contract of an Iterator. The Iterator interface can be used as a stream of elements. This stream allows the consumer to view an individual element at a time, and decide what to do from there - copy it, pull some (derived) value from it, etc.
e.g. The consumer may decide to just look at an individual field of that element and then not need the element at all anymore. A common case is to iterate over an Iterator of items to calculate some summary value. Perhaps the elements are very memory intensive and we do intend to try and fit multiple (to some n count of elements) into memory all at the same time.

My key point is that the Iterator interface leaves the decision of whether or not to hold references to the elements on successive next() calls to the consumer.
Clojure's seq on Iterators makes a decision for the consumer that they can handle having a chunk (e.g. 32 elements) consumed from the Iterator at once. This doesn't have to strictly be a problem of mutable state. This can be about other resource management issues - such as memory in my above example.

I could also see it being that the Iterator is providing elements to the consumer that hold open some resources while the element is being "looked at". When hasNext() is called the Iterator impl could decide to close old connections to resources used in past iterations.
The consumer does not get to have a chance to look at some of these elements at the time they are available anymore, due to the assumption that Clojure makes of being able to read from the Iterator in chunks prior to the consumer seeing the items.

Again, I think I agree that caching and chunking of seqs is at odds with the contract of Iterator. It is because of this, that I find it sneaky how Clojure may behave with them in these circumstances.
I suppose that a seq for Iterator is really only for a special class of Iterators, where there is no concern for holding a chunk in memory at a time or the resource usage to realize a chunk at a time when only a single element may be needed at that point. This is the reality of how laziness interacts with chunking already though for the most part - things will may be lazy, but not necessarily one element at a time lazy.

I certainly can see this change of behavior sneaking up and breaking libraries out there that are interoping with Java Iterators at this point though.

Comment by Marshall T. Vandegrift [ 18/Jun/15 12:14 PM ]

I do understand the available solutions. My dual concern is that they should not be necessary, and that it will not be immediately clear in existing code where the solutions need to be applied.

It seems that I'm not going to be able to convince you, and have no ability to even attempt to convince Rich etc. I'll probably go the route of using a internally-patched version where I want to upgrade existing applications to Clojure 1.7.

Even though we could not come to agreement, I appreciate the time you've taken discussing this issue and seeking resolution for it. Thank you!

Comment by Alex Miller [ 18/Jun/15 2:11 PM ]

While I think you can see the interface of iterators and seqs in a similar way, I perceive them very differently.

I see Java iterators as a stateful interface for external iteration of a source - that is, they provide a processing model. Being stateful, iterators are (in most cases) not safe to share across threads. Once you create one, you have to control access to it.

I see seqs as being a logical list data structure that may have a variety of strategies for production. Caching and immutability are very much bound up in that sense that seqs can be treated as data, passed around safely, etc even if they are built initially on demand.

The sequence you are producing from one of these iterators will give you different results if looked at more than once. This feels deeply wrong to me from a Clojure perspective - as a seq user this violates every conception I have. Yes, you could (previously) use that seq as a form of iteration, but imo that's an abuse of knowing too much about the implementation. If you care about allocation costs, then using a seq that creates and caches seq nodes is a waste of memory. If you care about resource management, laziness is also a bad fit for that need as it's difficult to know when a resource has been completed.

Instead of trying to wedge everything into sequences, consider your new options in 1.7! You could use an eduction to delay processing but eagerly process a stack of transformations without allocation on an iterable source when it's time to do so. Or transduce/into/etc to do it eagerly. Or even sequence to compute it incrementally, which is actually a better answer than the lazy-seq one I gave above. reduce does walk the iterator one-by-one (there's no other way to do it!), and will apply the reducing function to each element before obtaining the next, so using either sequence (if you want caching) or eduction (if you just want delay) or into or reduce/transduce all in combination with a map transducer that produces a value, is another good solution in 1.7:

user=> (def to-val #(.get %)) ;; mutable object to value 
#'user/to-val 
user=> (into [] (map to-val) (mutating-iterable 10)) 
[0 1 2 3 4 5 6 7 8 9] 
user=> (eduction (map to-val) (mutating-iterable 10)) 
(0 1 2 3 4 5 6 7 8 9) 
user=> (sequence (map to-val) (mutating-iterable 10)) 
(0 1 2 3 4 5 6 7 8 9)

All of those are just taking a single "value-convert" but could instead take an arbitrary composition of transducers instead (which will incur none of the intermediate seq node allocation vs using lazy seqs). So thanks for mentioning reduce Mike - those neurons hadn't connected in my head yet.

Comment by Mike Rodriguez [ 19/Jun/15 11:29 AM ]

After reading through your last response I can say I feel more comfortable about this change and the appropriate way to deal with these types of Iterators now.

I appreciate you going into all that detail to explain this. It looks like 1.7 has a lot to offer in allowing for these more "fine-grained" ways to interact with collections like this. +1 to that!





[CLJ-1736] Tweaks to changelog for 1.7 RC2 Created: 22/May/15  Updated: 17/Jun/15  Resolved: 17/Jun/15

Status: Closed
Project: Clojure
Component/s: None
Affects Version/s: Release 1.7
Fix Version/s: Release 1.7

Type: Enhancement Priority: Minor
Reporter: Alex Miller Assignee: Unassigned
Resolution: Completed Votes: 0
Labels: None

Attachments: Text File clj-1736.patch    
Patch: Code
Approval: Ok

 Description   

Just some minor tweaks to the changelog.



 Comments   
Comment by Nicola Mometto [ 22/May/15 11:37 AM ]

https://github.com/clojure/clojure/commit/69afe91ae07a4c75c34615a4af14327f98d78510#commitcomment-10670998





[CLJ-1735] Throwable->map is missing docstring and since Created: 22/May/15  Updated: 17/Jun/15  Resolved: 17/Jun/15

Status: Closed
Project: Clojure
Component/s: None
Affects Version/s: Release 1.7
Fix Version/s: Release 1.7

Type: Defect Priority: Major
Reporter: Alex Miller Assignee: Unassigned
Resolution: Completed Votes: 0
Labels: docstring

Attachments: Text File clj-1735.patch    
Patch: Code
Approval: Ok

 Description   

Throwable->map is missing docstring and since






[CLJ-1731] Transient maps can't be assoc!'d to contain more than 8 elements. Created: 17/May/15  Updated: 17/May/15  Resolved: 17/May/15

Status: Closed
Project: Clojure
Component/s: None
Affects Version/s: Release 1.6
Fix Version/s: None

Type: Task Priority: Major
Reporter: Peter Herniman Assignee: Unassigned
Resolution: Declined Votes: 0
Labels: assoc!, transient
Environment:

Linux, Fedora 20
Clojure 1.6.0
OpenJDK 64-Bit Server VM 1.7.0_79-mockbuild_2015_04_15_06_33-b00



 Description   

(let [x (transient {})] (dotimes [i 30] (assoc! x i 0)) (persistent! x))
Will result in

{0 0, 1 0, 2 0, 3 0, 4 0, 5 0, 6 0, 7 0}

instead of the expected 30 element map.

I'm not sure if this is fixed in the most recent version (development) but it doesn't work in 1.6.0.



 Comments   
Comment by Alex Miller [ 17/May/15 6:54 AM ]

This is not correct usage of transient maps. You must use the return value of assoc! for further updates, not bash the same instance. In other words, the update model is the same as with normal maps. There is an outstanding ticket to tweak the docstring slightly to make this clearer.

See a similar example with transient vectors at http://clojure.org/transients.

Comment by Peter Herniman [ 17/May/15 6:23 PM ]

Ah ok, looking at the example on clojure.org clears things up a lot.
I'm not too sure if the docstring does need updating, the tutorial on transients does state that you need to capture the return value explicitly.
Thanks!





[CLJ-1728] source fn fails for fns with conditional code Created: 10/May/15  Updated: 05/Jun/15  Resolved: 12/May/15

Status: Closed
Project: Clojure
Component/s: None
Affects Version/s: Release 1.7
Fix Version/s: Release 1.7

Type: Defect Priority: Major
Reporter: Mike Fikes Assignee: Unassigned
Resolution: Completed Votes: 0
Labels: reader, repl
Environment:

1.7.0-beta2


Attachments: Text File clj-1728.patch    
Patch: Code
Approval: Ok

 Description   

Note: Similar to issue CLJS-1261.

If you use the source Clojure REPL function on a function defined in a CLJC file, where the function itself contains some conditional code, then the operation will fail with "Conditional read not allowed".

To reproduce:
Do a lein new testme, rename the core.clj file to core.cljc, and then add the following

(defn f 
  "Eff"
  [] 
  1)

(defn g 
  "Gee"
  []
  #?(:clj "clj" :cljs "cljs"))

Additionally, revise the project.clj to specify 1.7.0-beta2.

Require the testme.core namespace, referring :all.

Verify that you can call, get the doc for, and source for f.

But, on the other hand, while you can call and get the doc for g, you can't do (source testme.core/g).

user=> (source testme.core/g)

RuntimeException Conditional read not allowed  clojure.lang.Util.runtimeException (Util.java:221)
user=> (pst)
RuntimeException Conditional read not allowed
	clojure.lang.Util.runtimeException (Util.java:221)
	clojure.lang.LispReader$ConditionalReader.checkConditionalAllowed (LispReader.java:1406)
	clojure.lang.LispReader$ConditionalReader.invoke (LispReader.java:1410)
	clojure.lang.LispReader$DispatchReader.invoke (LispReader.java:682)
	clojure.lang.LispReader.read (LispReader.java:255)
	clojure.lang.LispReader.readDelimitedList (LispReader.java:1189)
	clojure.lang.LispReader$ListReader.invoke (LispReader.java:1038)
	clojure.lang.LispReader.read (LispReader.java:255)
	clojure.lang.LispReader.read (LispReader.java:195)
	clojure.lang.LispReader.read (LispReader.java:190)
	clojure.core/read (core.clj:3638)
	clojure.core/read (core.clj:3636)
nil

Approach: Set {:read-cond :allow} if source file extension is .cljc. Test above works now.

Patch: clj-1728.patch



 Comments   
Comment by Mike Fikes [ 11/May/15 8:05 AM ]

I tested with Alex's cli-1728.patch, and it works for me.

Comment by Mike Fikes [ 12/May/15 7:02 PM ]

Confirmed fixed using master.

Comment by Nicola Mometto [ 05/Jun/15 9:53 AM ]

Patch committed and the ticket marked as resolved but not closed. I'm closing it.





[CLJ-1727] range confused by large bounds Created: 07/May/15  Updated: 05/Jun/15  Resolved: 12/May/15

Status: Closed
Project: Clojure
Component/s: None
Affects Version/s: Release 1.7
Fix Version/s: Release 1.7

Type: Defect Priority: Major
Reporter: Steve Miner Assignee: Unassigned
Resolution: Completed Votes: 0
Labels: regression

Attachments: Text File clj-1709-wip-1.patch     Text File clj-1709-wip-2.patch     Text File clj-1727-2.patch     Text File clj-1727-3.patch     Text File clj-1727-4.patch     Text File clj-1727.patch    
Patch: Code and Test
Approval: Ok

 Description   

There are a number of issues related to counting and overflow in the current LongRange implementation.

expression 1.6.0 1.7.0-beta2 +patch comment
(range (- Long/MAX_VALUE 2) Long/MAX_VALUE) (9223372036854775805 9223372036854775806) OOME (9223372036854775805 9223372036854775806) top of long range
(count (range (- Long/MAX_VALUE 2) Long/MAX_VALUE)) 2 2 2 top of long range
(range (+ Long/MIN_VALUE 2) Long/MIN_VALUE -1) (-9223372036854775806 -9223372036854775807) OOME (-9223372036854775806 -9223372036854775807) bottom of long range
(count (range (+ Long/MIN_VALUE 2) Long/MIN_VALUE -1)) 2 2 2 bottom of long range
(range Long/MIN_VALUE Long/MAX_VALUE Long/MAX_VALUE) ArithmeticEx OOME (-9223372036854775806 -1 9223372036854775806) large positive step
(count (range Long/MIN_VALUE Long/MAX_VALUE Long/MAX_VALUE)) ArithmeticEx 0 3 large positive step
(range Long/MAX_VALUE Long/MIN_VALUE Long/MIN_VALUE) ArithmeticEx OOME (9223372036854775807 -1) large negative step
(count (range Long/MAX_VALUE Long/MIN_VALUE Long/MIN_VALUE)) ArithmeticEx 0 2 large negative step
(count (range 0 Long/MAX_VALUE)) overflows to nonsense -2147483648 ArithmeticEx number of values in range > Integer.MAX_VALUE

Cause: There were several bugs, both old and new, in the counting related code for range, particularly around overflow and (introduced in CLJ-1709) coercion error.

Approach: The patched code:

  • Uses only exact values (no double conversion in there)
  • Fixes the algorithm for integer ceiling to be correct
  • Explicitly does overflow-checking calculations when necessary (chunking, counting, and iterator)
  • In the case of overflow, falls back to slower stepping algorithm if necessary (this is only in pathological cases like above)
  • Added many new tests thanks to Andy Fingerhut and Steve Miner.

One particular question is what to do in the case where the count of a range is > Integer.MAX_VALUE. The choices are:
1. Return Integer.MAX_VALUE (Java collection size() solution)
2. Throw ArithmeticOverflowException (since your answer is going to be wrong)
3. Overflow and let bad stuff happen (Clojure 1.6 does this)

The current patch takes approach #2, per Rich.

Performance check:

expr 1.6 beta1 beta2 beta2+patch
(count (range (* 1024 1024))) 63 ms 0 ms 0 ms 0 ms
(reduce + (map inc (range (* 1024 1024)))) 55 ms 35 ms 34 ms 32 ms
(reduce + (map inc (map inc (range (* 1024 1024))))) 74 ms 59 ms 56 ms 54 ms
(count (keep odd? (range (* 1024 1024)))) 77 ms 52 ms 48 ms 49 ms
(transduce (comp (map inc) (map inc)) + (range (* 1024 1024))) n/a 30 ms 26 ms 26 ms
(reduce + 0 (range (* 2048 1024))) 72 ms 29 ms 29 ms 21 ms
(reduce + 0 (rest (range (* 2048 1024)))) 73 ms 29 ms 30 ms 21 ms
(doall (range 0 31)) 1.38 us 0.97 us 0.73 us 0.75 us
(doall (range 0 32)) 1.38 us 0.99 us 0.76 us 0.77 us
(doall (range 0 4096)) 171 us 126 us 125 us 98 us
(into [] (map inc (range 31))) 1.87 us 1.34 us 1.27 us 1.33 us
(into [] (map inc) (range 31)) n/a 0.76 ms 0.76 ms 0.76 ms
(into [] (range 128)) 5.26 us 2.18 us 2.15 us 2.22 us

Patch: clj-1727-4.patch



 Comments   
Comment by Steve Miner [ 07/May/15 10:49 AM ]

It looks like something is overflowing when doing the count calculation. Here's another example:

(count (range (- Long/MAX_VALUE 7)))
-2147483648

Comment by Alex Miller [ 07/May/15 10:54 AM ]

Looks like lack of overflow checking in the chunk logic maybe.

Comment by Alex Miller [ 07/May/15 11:11 AM ]

The example in the description is overflowing in computing nextStart in forceChunk(). That should be fixable, will consider some options.

The example in the comment overflows calculating the count, which is > max int. I'm not sure there actually is a good answer in that particular case. The Java Collection interface expects count() to return Integer.MAX_VALUE in this case (which is bad, but equally bad as every other incorrect answer when the value is not representable in the type).

Comment by Steve Miner [ 07/May/15 11:18 AM ]

LongRange absCount looks suspicious. That double math might be wrong. If the (end - start) is large, the conversion to double loses integer precision. For example, in Clojure:

(- Long/MAX_VALUE (long (double (- Long/MAX_VALUE 1000))))
1023

You might have expected 1000, but the double conversion was not exact. Of course, it works from some values, like exactly Long/MAX_VALUE.

I think it might be safer to restrict the LongRange to the safe bounds, basically inside (bit-shift-right Long/MAX_VALUE 10). Or just use (long Integer/MAX_VALUE) as a reasonable max.

Comment by Andy Fingerhut [ 07/May/15 12:02 PM ]

Some other fun test cases, if the goal is to make LongRange work for entire range of longs:

user=> (take 5 (range (- Long/MAX_VALUE 2) Long/MAX_VALUE))
(9223372036854775805 9223372036854775806 9223372036854775807 -9223372036854775808 -9223372036854775807)
user=> (take 5 (range (- Long/MAX_VALUE 2) Long/MAX_VALUE 7))
(9223372036854775805 -9223372036854775804 -9223372036854775797 -9223372036854775790 -9223372036854775783)
user=> (take 5 (range (- Long/MAX_VALUE 2) Long/MAX_VALUE Long/MAX_VALUE))
(9223372036854775805 -4 9223372036854775803 -6 9223372036854775801)
Comment by Andy Fingerhut [ 07/May/15 1:24 PM ]

Attachment clj-1709-wip-1.patch is intentionally in the wrong format, since it is only intended as a hint of what might be done.

I think using float or double arithmetic for absCount is a very bad idea, given all the subtle roundoff things that could happen there. Exact arithmetic is best.

It has known limitations mentioned in a few comments in the code. There may be unknown limitations, too.

Generative tests that specifically tried to hit the corner cases, e.g. start values often near Long/MIN_VALUE, end values near Long/MAX_VALUE, step values near 0 and the extreme long values, etc. are much more likely to hit any remaining bugs here, but are also very slow to test given the size of the ranges.

Comment by Andy Fingerhut [ 07/May/15 1:36 PM ]

I am liking Steve Miner's suggestion, or something like it, the more I think on it. That is, check a few more conditions on the values of start, end, and step in clojure.core/range, and if they are not met, avoid LongRange and use Range instead. This would put the common cases in LongRange, and only unusual corner cases in the slower Range. At the same time, it could simplify the code for LongRange and help keep it fast.

Comment by Alex Miller [ 07/May/15 4:17 PM ]

The competing constraint here is of course performance. Adding more checks also makes the fast common path slower. By no means ruling it out, but need to consider it.

Comment by Andy Fingerhut [ 07/May/15 5:23 PM ]

Attachment clj-1709-wip-2.patch is a fleshed out version of the approach suggested by Steve Miner – avoid LongRange/create if the long args to range are such that LongRange would misbehave. The example-based tests could be expanded a bit.

Comment by Alex Miller [ 08/May/15 11:03 AM ]

I have a solution for this that does not need additional up-front checks and has (I think) minimal impact on performance. Polishing it...

Comment by Alex Miller [ 08/May/15 11:03 AM ]

Oh, and big thanks Andy for the tests - I will totally steal those, they were very helpful.

Comment by Andy Fingerhut [ 08/May/15 12:12 PM ]

Here are a couple more that I would recommend stealing (implying that the clojure.core/range return value should be equal to the clojure.lang.Range/create version):

(range -1 Long/MAX_VALUE Long/MAX_VALUE)  ; large step values make (step * CHUNK_SIZE) overflow
(range 1 Long/MIN_VALUE Long/MIN_VALUE)
Comment by Andy Fingerhut [ 08/May/15 7:52 PM ]

Alex, clj-1727.patch looks solid to me. Or at least I couldn't find anything wrong with it in 30-40 minutes.

One question: In count(), why fall back to super.count() if the actual value fits in a long but is greater than Integer.MAX_VALUE ? Because we want to be bug-compatible with count() in that case, sometimes returning overflowed values? (see example below). It seems faster and better to return Integer.MAX_VALUE in that case.

user=> *clojure-version*
{:major 1, :minor 6, :incremental 0, :qualifier nil}
user=> (count (range (+ Integer/MAX_VALUE 2)))
-2147483647
Comment by Alex Miller [ 08/May/15 8:46 PM ]

Yeah I went back and forth on that. I actually I was throwing an exception before this version. There is no good answer.

Comment by Steve Miner [ 09/May/15 2:19 PM ]

I did some quick tests with the patch and it looks good. I have to agree with Andy that it would make sense to return Integer.MAX_VALUE for the overflow cases. The super.count() is likely to be so slow that it might as well be an infinite loop. If I'm reading the code correctly, stepOffset is always (step > 0 ? -1 : l). I would expect that to be a very fast computation (especially with a final step) so it's probably not worth caching in a field.

Comment by Alex Miller [ 09/May/15 3:10 PM ]

New -2 patch avoids the new field (~same perf) and changes behavior on count over max int.

Comment by Steve Miner [ 10/May/15 1:06 PM ]

Testing with patch-2. It looks like there are a couple of edge cases that can give negative results where I would have expected Integer/MAX_VALUE (for any overflow of int count).

user=> Integer/MAX_VALUE
2147483647
user=> (count (range Long/MIN_VALUE -2))
2147483647 ;OK
user=> (count (range Long/MIN_VALUE -1))
-2147483648

user=> (count (range 1 Long/MAX_VALUE))
2147483647 ;OK
user=> (count (range 0 Long/MAX_VALUE))
-2147483648

I think that count() should return Integer.MAX_VALUE when there's an ArithmeticException, instead of trying super.count() there.

Comment by Andy Fingerhut [ 10/May/15 2:53 PM ]

Steve, I think that Integer.MAX_VALUE is often an incorrect value for count(), when rangeCount throws an exception. For example, here are some results with patch-2, tweaked to make rangeCount public:

user=> (def x (range 0 1 2))
#'user/x
user=> (. x rangeCount Long/MIN_VALUE Long/MAX_VALUE Long/MAX_VALUE)
ArithmeticException integer overflow  clojure.lang.Numbers.throwIntOverflow (Numbers.java:1501)
user=> (count (range Long/MIN_VALUE Long/MAX_VALUE Long/MAX_VALUE))
3      ; correct value

Also, it seems that the reason it is returning these incorrect values is because super.count() is ASeq.count(), which starts at int 1, and then when it discovers that the remaining thing is a LongRange, which implemented Counted, it calls LongRange#count() and in turn LongRange#rangeCount() on a sequence one shorter. My debug prints in LongRange#count() confused me mightily until I figured out that this is what it is doing.

That also means that expressions like the one below overflow the stack, because of mutually recursive calls between LongRange#count() and ASeq#count().

user=> (count (range Long/MIN_VALUE 10000))
StackOverflowError   java.lang.Exception.<init> (Exception.java:66)

One way to correct the stack overflow issue would be to effectively copy Aseq#count() code into the place where LongRange#count() calls super.count().

If that loop was then modified to check for (i < 0), to detect overflow, and returning Integer.MAX_VALUE if that ever happened, that would also eliminate overflow for LongRange's (but not all ASeq's).

Comment by Steve Miner [ 10/May/15 3:28 PM ]

Good point about the case of a large step potentially causing the exception, in which case the range may still have a reasonable count.

Comment by Alex Miller [ 11/May/15 2:43 AM ]

New -3 patch changes the fallback to use the iterator. The iterator was also not safe from overflow, so I fixed that too.

For counting ranges > Integer/MAX_VALUE, now:

user=> (count (range Long/MIN_VALUE 0))
2147483647
Comment by Andy Fingerhut [ 12/May/15 10:27 AM ]

Re: clj-1727-4.patch
I, for one, will never be filing a bug that (count (range Long/MIN_VALUE Long/MAX_VALUE)) overflows a long

Comment by Steve Miner [ 12/May/15 3:44 PM ]

And I will resist suggesting a count' (ala inc' and dec'). Thanks for fixing these edge cases. The original report came from real life experience when I was trying to fix my own double math problems with TCHECK-67.

Comment by Nicola Mometto [ 05/Jun/15 9:53 AM ]

Patch committed and the ticket marked as resolved but not closed. I'm closing it.





[CLJ-1726] New iterate and cycle impls have delayed computations but don't implement IPending Created: 06/May/15  Updated: 05/Jun/15  Resolved: 12/May/15

Status: Closed
Project: Clojure
Component/s: None
Affects Version/s: Release 1.7
Fix Version/s: Release 1.7

Type: Defect Priority: Major
Reporter: Alex Miller Assignee: Unassigned
Resolution: Completed Votes: 0
Labels: regression

Attachments: Text File clj-1726-2.patch     Text File clj-1726.patch    
Patch: Code
Approval: Ok

 Description   

When moving from LazySeqs to the new types we lost this but I don't think we should. Tools like Cursive use this for deciding when and how much to realize from a lazy sequence.

Approach:

  • iterate - The head of an iterate will always have the seed value and return 1 realized value. Subsequent elements will start unrealized and then become realized when the iterate function has been invoked to produce the value.
  • cycle - Returns unrealized if _current has been forced (initially null for all nodes after the first node).

(Note that range and repeat effectively always have their first element realized so I have chosen not to implement IPending - there is no delayed computation pending.)

;; setup
(def i (iterate inc 0))
(def c (cycle [1 2 3]))

user=>  (mapv realized? [i (next i) c (next c)])
[true false true false]
user=> (fnext i)
1
user=> (fnext c)
2
user=> (mapv realized? [i (next i) c (next c)])
[true true true true]

Patch: clj-1726-2.patch



 Comments   
Comment by Fogus [ 08/May/15 9:44 AM ]

There are three things that I like about this patch:

1) The implementations of the isRealize method provide meaning to the somewhat opaque encoding inherent in _current and (less opaque) UNREALIZED_SEED.

2) The use of realized? is generally useful outside of IDE contexts.

3) It's small and easy to grasp.

Comment by Nicola Mometto [ 05/Jun/15 9:53 AM ]

Patch committed and the ticket marked as resolved but not closed. I'm closing it.





[CLJ-1725] Add missing transducer creating functions to clojure.org/transducers Created: 04/May/15  Updated: 04/May/15  Resolved: 04/May/15

Status: Closed
Project: Clojure
Component/s: None
Affects Version/s: Release 1.7
Fix Version/s: None

Type: Enhancement Priority: Minor
Reporter: A. R Assignee: Unassigned
Resolution: Completed Votes: 0
Labels: None


 Description   

The following should be added to the list:

  • distinct
  • interpose
  • map-indexed


 Comments   
Comment by Alex Miller [ 04/May/15 3:15 PM ]

Thanks, I'll take care of that.

Comment by A. R [ 04/May/15 4:05 PM ]

Nevermind this comment. Had an old version.

Comment by Alex Miller [ 04/May/15 4:29 PM ]

Added.





[CLJ-1723] NPE with eduction + cat on a collection containing nil Created: 04/May/15  Updated: 05/Jun/15  Resolved: 12/May/15

Status: Closed
Project: Clojure
Component/s: None
Affects Version/s: Release 1.7
Fix Version/s: Release 1.7

Type: Defect Priority: Critical
Reporter: Moritz Heidkamp Assignee: Alex Miller
Resolution: Completed Votes: 0
Labels: transducers

Attachments: Text File clj-1723.patch    
Patch: Code and Test
Approval: Ok

 Description   

Using the cat transducer with eduction leads to an NPE when the collection contains at least one collection with more than one item of which at least one is nil. The shortest reproduction case I could come up with is this:

(eduction cat [[nil nil]])

Cause: An expanding transducer (cat, mapcat) puts the expansion on an internal buffer, which is a ConcurrentLinkedQueue. Java Queue impls do not support adding or removing null b/c null is used as a special value in some of the Queue apis.

Approach: Switch from ConcurrentLinkedQueue to LinkedList. LinkedList supports both Queue and other semantics as well and does support nulls (with caveats that that is a bad thing to do if you're using the Queue apis and expecting those special semantics). However, the TransformerIterator usage does not rely on any of that. LinkedList is also obviously not concurrency friendly, but the buffer is only used by a single thread at a time and the volatile field guarantees visibility, so this is fine.

I re-ran some of the perf tests from CLJ-1669 and found the expanding transducer test there (into [] (eduction (map inc) (mapcat range) s50)) went from 27 us to 24 us, so there is a bit of a perf improvement as well.

Patch: clj-1723.patch



 Comments   
Comment by Alex Miller [ 04/May/15 12:03 PM ]

Gah, Java Queues don't allow null. I have some prior work on other impls for this so I'm working on a fix.

Comment by Fogus [ 08/May/15 9:49 AM ]

This is a very straight-forward solution that works and is easy to justify and grasp.

Comment by Nicola Mometto [ 05/Jun/15 9:53 AM ]

Patch committed and the ticket marked as resolved but not closed. I'm closing it.





[CLJ-1718] (if test then else?) is inconsistent Created: 30/Apr/15  Updated: 05/May/15  Resolved: 30/Apr/15

Status: Closed
Project: Clojure
Component/s: None
Affects Version/s: Release 1.6
Fix Version/s: None

Type: Defect Priority: Major
Reporter: Jigen Daisuke Assignee: Unassigned
Resolution: Declined Votes: 0
Labels: None
Environment:

MacOS X 10.10.2



 Description   

Issue CLJ-1640 was "fixed" by adding a comment to the documentation.
I think that's not enough, because CLJ-1640 actually points to a major inconsistency.
Just have a look at the following code:

;; I know, the definition of y is bad code, and I would
;; never write such rubish, BUT some people do (e.g. the
;; guys at MS, responsible for the sqljdbc4.jar JDBC
;; driver).
(def y (Boolean. false))

(if (= false y)
(println "that's ok"))

(if y
(println "that's inconsistent, because y is " y
" and we proved it with the above if statement"))



 Comments   
Comment by Alex Miller [ 30/Apr/15 3:25 PM ]

CLJ-1640 was closed by the submitter, no change was made for that.

The decision was made long ago by Clojure to canonicalize on Boolean/FALSE as the false value. We do not plan to make any changes in this regard.

In cases where you are dealing with Java libraries that construct new Boolean instances rather than use the canonical ones, you are responsible for normalizing these values. Please file issues with those libraries as appropriate.

Comment by Andy Fingerhut [ 30/Apr/15 3:41 PM ]

The longish example/article on ClojureDocs.org might contain additional useful info. Even Java recommends against ever using (Boolean. false): http://clojuredocs.org/clojure.core/if

Comment by Jigen Daisuke [ 30/Apr/15 5:08 PM ]

But, if Boolean/FALSE is the false value, shouldn't

(if (= (Boolean. false) false) true false)

better return false?

Comment by Andy Fingerhut [ 30/Apr/15 6:12 PM ]

These all return 2:

(if nil 1 2)
(if false 1 2)
(if Boolean/FALSE 1 2)

clojure.core/=, like Java .equals, returns true when comparing Boolean/FALSE and (Boolean. false). Blame Java.

Comment by Jigen Daisuke [ 05/May/15 4:51 PM ]

@Andy Fingerhut

I know all that, but this doesn't make it any more consistent

See, if they won't fix how "if" treats custom made false-values, the next best
thing to do were to manifest this behaviour of "if" into the "=" function (and
yes, that would mean that "=" isn't a mere ".equals" call anymore, because
Boolean false-values would have to be treated in a special way). But then at
least we would have a consistent system - though not exactly in the way I'd
prefer.

Therefore, I can't blame Java. If they had implemented "if" right in the first place,
everything were fine. So this one is on Clojure.

Comment by Andy Fingerhut [ 05/May/15 5:50 PM ]

I'd like to withdraw the last 2 sentences of my previous comment. I think Alex's comment is the shortest accurate answer: It was a choice in Clojure's implementation to do this. As it is right now, there are some compiled 'if' expressions that compare the test expression against both nil (Java null) and Java Boolean.FALSE, and allowing (Boolean. false) to also be treated as false would either require comparing against a third value, or calling a method like Boolean.valueOf() before doing the comparison. Perhaps a micro-optimization, but seems to me like a reasonable one, given the recommendations in Java not to use the Boolean constructors.





[CLJ-1717] Compiler casts System properties to String without prior type check Created: 29/Apr/15  Updated: 05/Jun/15  Resolved: 18/May/15

Status: Closed
Project: Clojure
Component/s: None
Affects Version/s: Release 1.7
Fix Version/s: None

Type: Defect Priority: Critical
Reporter: Laurent Petit Assignee: Unassigned
Resolution: Declined Votes: 0
Labels: compiler, regression
Environment:

occurs for any JVM / Operating System. Encountered when using Clojure inside the Equinox OSGi framework (reference OSGi implementation used e.g. by Eclipse IDE)


Attachments: Text File clj-1717-1.patch    
Patch: Code
Approval: Triaged

 Description   

The Clojure Compiler loops over all System Properties through their EntrySet.
This interface allows for non-String properties to be found. It is leveraged by the Equinox OSGi framework (reference OSGi implementation used e.g. by the Eclipse IDE).

This means that since this new code has been introduced in the Clojure compiler, the official Clojure Jar cannot be used inside Eclipse.

The problem is that Counterclockwise, the Eclipse-based Clojure IDE, at least, is affected, since it is developed with Clojure code.

The attached patch solves the issue by skipping System Properties key/value pairs whose values aren't Strings.



 Comments   
Comment by Alex Miller [ 29/Apr/15 2:40 PM ]

Probably a regression related to CLJ-1274.

Comment by Laurent Petit [ 29/Apr/15 2:45 PM ]

Yes, CLJ-1274 moved the code from the Compile.java class to the Compiler.java class. The code already had the cast problem, but it was probably not an issue at runtime for CCW when only present in Compile.java

Comment by Alex Miller [ 29/Apr/15 3:00 PM ]

It looks like maybe this is considered a bug in Eclipse land (which it probably should be)?
https://bugs.eclipse.org/bugs/show_bug.cgi?id=445122

Comment by Laurent Petit [ 29/Apr/15 3:21 PM ]

I agree that it is an abuse on the side of Eclipse-and of a System properties hole that allows to put non-String values and then browse them through methods not using Generics.

From the last comments, it appears that it has only be released very recently, and for only the last stable version of Eclipse. So I expect CCW users with disparate Eclipse versions installed to have the problem if I do not apply the patch to my custom version of Clojure for some more months.

I can live with that if you reject the issue.

Comment by Laurent Petit [ 01/May/15 7:28 AM ]

What is the status for this issue? I understand that being considered an abuse from the OSGi implementation of the System properties contract, you might not want to add code for dealing with this in Compiler.
On the other hand, it's a small 3-lines check with an explanation of why it's done that way, so...

Anyway, feel free to get rid of it and close it so it doesn't get in the way if you think it's not worth the trouble.

Comment by Alex Miller [ 01/May/15 8:44 AM ]

We haven't had a chance to discuss it further yet. There is reluctance to change it if it's not necessary.

Comment by Nicola Mometto [ 05/Jun/15 9:56 AM ]

Ticket was declined and marked as resolved but not closed. I'm closing it.





[CLJ-1716] IExceptionInfo should print with its ex-data Created: 26/Apr/15  Updated: 12/May/15  Resolved: 12/May/15

Status: Closed
Project: Clojure
Component/s: None
Affects Version/s: Release 1.7
Fix Version/s: Release 1.7

Type: Enhancement Priority: Major
Reporter: Gary Fredericks Assignee: Unassigned
Resolution: Completed Votes: 0
Labels: reader

Attachments: Text File CLJ-1716.patch    
Patch: Code and Test
Approval: Ok

 Description   

The new (for 1.7) data-reader printing format for exceptions does not include the ex-data when relevant:

user> *clojure-version*
{:major 1, :minor 7, :incremental 0, :qualifier "beta2"}
user> (pr (ex-info "msg" {:my-data 42}))
#error {
 :cause "msg"
 :via
 [{:type clojure.lang.ExceptionInfo
   :message "msg"
   :at [clojure.core$ex_info invoke "core.clj" 4591]}]
 :trace
 [[clojure.core$ex_info invoke "core.clj" 4591]
  [user$eval9314 invoke "form-init6701752258113826186.clj" 1]
  ;; ...
]}

Approach: If ExceptionInfo is caught, also print :data key with ex-data. Include :data key for each ExceptionInfo in via.

After:

#error {
 :cause "msg"
 :data {:my-data 42}
 :via
 [{:type clojure.lang.ExceptionInfo
   :message "msg"
   :data {:my-data 42}
   :at [clojure.core$ex_info invoke "core.clj" 4591]}]
 :trace
 [[clojure.core$ex_info invoke "core.clj" 4591]
  [user$eval1 invoke "NO_SOURCE_FILE" 1]
  ;; elided
  ]}

Example with nested ExceptionInfo:

(try 
  (throw (ex-info "cause" {:a :b})) 
  (catch Exception e 
    (throw (ex-info "wrapped" {:c :d} e))))

;; yields:

#error {
 :cause "cause"
 :data {:a :b}
 :via
 [{:type clojure.lang.ExceptionInfo
   :message "wrapped"
   :data {:c :d}
   :at [clojure.core$ex_info invoke "core.clj" 4591]}
  {:type clojure.lang.ExceptionInfo
   :message "cause"
   :data {:a :b}
   :at [clojure.core$ex_info invoke "core.clj" 4591]}]
 :trace
 [[clojure.core$ex_info invoke "core.clj" 4591]
  [user$eval5 invoke "NO_SOURCE_FILE" 4]
  ;; elided
  ]}

Patch: CLJ-1716.patch



 Comments   
Comment by Gary Fredericks [ 26/Apr/15 9:30 PM ]

Added two patch files, the first with tests for the existing code, the second adding the :data key and extra tests for that.

Comment by Alex Miller [ 28/Apr/15 9:55 AM ]

Screening comments:

  • Combine the two patches into a single patch
  • Include :data for all ExceptionInfo in :via (when appropriate) (and leave it where it is)
  • when you git format-patch, throw -W on there to get more context (I think it would help in this case)
  • everything else looks good
Comment by Gary Fredericks [ 29/Apr/15 9:44 AM ]

Cool – I'm assuming "single patch" implies "single commit" since I couldn't find a way to dump two commits into one patch file.

Comment by Alex Miller [ 29/Apr/15 10:08 AM ]

yeah, single commit is what I meant. you can just commit multiple times and format-patch to get a single patch with multiple commits, but would prefer single.

Comment by Gary Fredericks [ 30/Apr/15 9:14 AM ]

Attached CLJ-1716.patch, which has just one commit, and now attaches data both to everything appropriate in :via and at the top level (so the data for the root cause is duplicated).

Added one more test case while I was at it.





[CLJ-1713] Range is not serializable Created: 21/Apr/15  Updated: 24/Apr/15  Resolved: 24/Apr/15

Status: Closed
Project: Clojure
Component/s: None
Affects Version/s: Release 1.7
Fix Version/s: Release 1.7

Type: Defect Priority: Major
Reporter: Alex Miller Assignee: Unassigned
Resolution: Completed Votes: 0
Labels: regression

Attachments: Text File clj-1713.patch    
Patch: Code and Test
Approval: Ok

 Description   

Range is not serializable once it starts getting chunked.

(import [java.io ByteArrayOutputStream ObjectOutputStream])

(defn- serialize
  "Serializes a single object, returning a byte array."
  [v]
  (with-open [bout (ByteArrayOutputStream.)
              oos (ObjectOutputStream. bout)]
    (.writeObject oos v)
    (.flush oos)
    (.toByteArray bout)))

(serialize (range 10))  ;; works fine
;;=> #object["[B" 0x71c14b1d "[B@71c14b1d"]

(def r (range 50))
(nth r 35)  ;; 35
(serialize r)
;;=> NotSerializableException clojure.lang.LongRange$LongChunk  java.io.ObjectOutputStream.writeObject0 (ObjectOutputStream.java:1181)

Cause: LongChunk is not serializable.

Approach: Make it serializable.

Patch: clj-1713.patch



 Comments   
Comment by Fogus [ 24/Apr/15 9:03 AM ]

This patch couldn't be more trivial. Screened and ready to apply.





[CLJ-1712] clojure.walk returns a different kv pair Created: 21/Apr/15  Updated: 21/Apr/15  Resolved: 21/Apr/15

Status: Closed
Project: Clojure
Component/s: None
Affects Version/s: Release 1.7
Fix Version/s: None

Type: Defect Priority: Minor
Reporter: James Conroy-Finn Assignee: Unassigned
Resolution: Declined Votes: 0
Labels: None


 Description   

In Clojure 1.7, a walk that would result in duplicate keys returns different values. Here is a simple example of the behaviour I'm referring to, that passes with Clojure 1.6, and fails with 1.7:

(ns walk.merge-test
  (:require [clojure.walk :refer [keywordize-keys]]
            [clojure.test :refer :all]))

(deftest test-walk-merge
  (is (= (keywordize-keys {:a 1 "a" 2}) {:a 1})))

I don't know if people are relying on this behaviour in production, and consider it a rather pathological example. Regardless, it would certainly be nice if the behaviour were deterministic.

I've pushed a quick demo up to GitHub to make it easier to verify the behaviour should anyone be interested.

https://github.com/jcf/walk-merge



 Comments   
Comment by Alex Miller [ 21/Apr/15 8:34 AM ]

Because maps are unordered, this code should not be expected to have deterministic results.

Changes in which map type is used are the likely cause of this difference, but the prior behavior was not something you should rely on.

Comment by James Conroy-Finn [ 21/Apr/15 9:38 AM ]

Thanks for the update Alex. I really enjoyed your Clojure/West talk, BTW.





[CLJ-1711] Hash map and StructMap cannot be conjoined using "into" Created: 21/Apr/15  Updated: 24/Apr/15  Resolved: 24/Apr/15

Status: Closed
Project: Clojure
Component/s: None
Affects Version/s: Release 1.7
Fix Version/s: Release 1.7

Type: Defect Priority: Major
Reporter: Immo Heikkinen Assignee: Unassigned
Resolution: Completed Votes: 0
Labels: regression
Environment:

Clojure 1.7.0-beta1


Attachments: Text File clj-1711-fix-structmap-iterator.patch    
Patch: Code and Test
Approval: Ok

 Description   
user=> (defstruct foo :bar)
#'user/foo
user=> (into {} (struct-map foo :bar 1))

IllegalArgumentException Don't know how to create ISeq from: clojure.lang.Keyword  clojure.lang.RT.seqFrom (RT.java:528)

In Clojure 1.6 this returns {:bar 1} as expected.



 Comments   
Comment by Alex Miller [ 21/Apr/15 8:43 AM ]

Thanks for the report - looks like an issue with the iter/reduce path for structmaps which changed in 1.7. Another example yielding same error:

(reduce (fn [acc i] (inc acc)) 0 struct-map)
Comment by Immo Heikkinen [ 22/Apr/15 12:42 AM ]

This turned out to be pretty simple to fix so I gave it a go.

Comment by Alex Miller [ 22/Apr/15 10:51 AM ]

Thanks! I'll take a look at it tomorrow!

Comment by Alex Miller [ 23/Apr/15 11:16 AM ]

Looks great, thanks.





[CLJ-1710] count of a range may be incorrect Created: 18/Apr/15  Updated: 19/Apr/15  Resolved: 19/Apr/15

Status: Closed
Project: Clojure
Component/s: None
Affects Version/s: None
Fix Version/s: None

Type: Defect Priority: Major
Reporter: Nelson Morris Assignee: Devin Walters
Resolution: Duplicate Votes: 0
Labels: None
Environment:

clojure 1.7.0-beta1


Attachments: Text File CLJ-1709-1710.patch    
Patch: Code and Test

 Description   

Reproduction:

(def range-count-is-ceil-of-length-divided-by-step                                                                                                                        
  (prop/for-all [start gen/int                                                                                                                                            
                 end gen/int                                                                                                                                              
                 step (gen/such-that #(not= % 0)                                                                                                                          
                                     gen/int)]                                                                                                                            
                (= (count (range start end step))                                                                                                                         
                   (max 0 (long (Math/ceil (double (/ (- end start) step))))))))                                                                                          
                                                                                                                                                                          
(tc/quick-check 100 range-count-is-ceil-of-length-divided-by-step)                                                                                                        
;;=> {:result false, :seed 1429397156819, :failing-size 6, :num-tests 7, :fail [-1 -3 -4], :shrunk {:total-nodes-visited 9, :depth 4, :result false, :smallest [0 -1 -2]}\
}                                                                                                                                                                         
                                                                                                                                                                          
                                                                                                                                                                          
(range 0 -1 -2)                                                                                                                                                           
;;=> (0)                                                                                                                                                                  
(count (range 0 -1 -2))                                                                                                                                                   
;;=> 0                                                                                                                                                                    
                                                                                                                                                                          
(tc/quick-check 100 range-count-is-ceil-of-length-divided-by-step)                                                                                                        
;;=> {:result false, :seed 1429397201424, :failing-size 5, :num-tests 6, :fail [3 4 3], :shrunk {:total-nodes-visited 9, :depth 4, :result false, :smallest [0 1 2]}}     
                                                                                                                                                                          
(range 0 1 2)                                                                                                                                                             
;;=> (0)                                                                                                                                                                  
(count (range 0 1 2))                                                                                                                                                     
;;=> 0


 Comments   
Comment by Devin Walters [ 18/Apr/15 7:11 PM ]

See attached patch and http://dev.clojure.org/jira/browse/CLJ-1709.

Comment by Alex Miller [ 19/Apr/15 9:09 AM ]

Covered by CLJ-1709





[CLJ-1709] Incorrect range contents and count with step != 1 Created: 18/Apr/15  Updated: 24/Apr/15  Resolved: 24/Apr/15

Status: Closed
Project: Clojure
Component/s: None
Affects Version/s: Release 1.7
Fix Version/s: Release 1.7

Type: Defect Priority: Major
Reporter: Nelson Morris Assignee: Devin Walters
Resolution: Completed Votes: 0
Labels: regression
Environment:

clojure 1.7.0-beta1


Attachments: Text File CLJ-1709-1710.patch     Text File clj-1709-2.patch    
Patch: Code and Test
Approval: Ok

 Description   

From https://groups.google.com/d/msg/clojure/dweCm6Bd-vc/atritH--xUEJ.

(range 0 11 2)
;;=> (0 2 4 6 8)
;;Expected: (0 2 4 6 8 10)
(count (range 0 11 2))
;;=> 5
;;Expected: 6

Cause: absCount method in LongRange is not computing count correctly.

Approach: Fix computation in LongRange.absCount() and add a generative test that compares the long and non-long versions of range produce the same ranges.

Screened by: Alex Miller, also verified that generative test failed without the fix.

Patch: clj-1709-2.patch



 Comments   
Comment by Devin Walters [ 18/Apr/15 7:10 PM ]

See attached patch and http://dev.clojure.org/jira/browse/CLJ-1710.

Comment by Alex Miller [ 19/Apr/15 9:07 AM ]

This seems to be related to `clojure.lang.LongRange`. Assuming that, here is a test.check property for being equal to clojure.lang.Range:

(def longrange equals-range               
  (prop/for-all [start gen/int                                                                                                                                            
                 end gen/int                                                                                                                                              
                 step (gen/such-that #(> % 0)                                                                                                                             
                                     gen/nat)]                                                                                                                            
                (= (clojure.lang.Range/create start end step)                                                                                                             
                   (clojure.lang.LongRange/create start end step))))                                                                                                      
                                                                                                                                                                          
(tc/quick-check 100 longrange-equals-range)                                                                                                                               
{:result false, :seed 1429392529363, :failing-size 15, :num-tests 16, :fail [0 15 7], :shrunk {:total-nodes-visited 22, :depth 5, :result false, :smallest [0 4 3]}}
Comment by Alex Miller [ 19/Apr/15 9:08 AM ]

Can we add the test.check property to the patch please? Clojure uses test.check already so this dependency is already taken care of.

Comment by Devin Walters [ 19/Apr/15 4:35 PM ]

@Alex, sure. It looks like transducers is not taking advantage of clojure.test.check's clojure.test integration. Specifically, the use of defspec. Is there a good reason why this is so?

This is what it'd look like:

(defspec longrange-equals-range 100
  (prop/for-all [start gen/int
                 end gen/int
                 step (gen/such-that #(> % 0)
                                     gen/nat)]
                (= (clojure.lang.Range/create start end step)
                   (clojure.lang.LongRange/create start end step))))

When building:

[java] Testing clojure.test-clojure.sequences
[java] {:result true, :num-tests 100, :seed 1429478867534, :test-var longrange-equals-range}

Is this alright? Please advise.

Comment by Devin Walters [ 19/Apr/15 4:44 PM ]

@Alex, I went ahead and did it the way I mentioned above.

Added an updated patch to include generative test to verify LongRange is the same as Range.

Comment by Michael Blume [ 20/Apr/15 12:05 AM ]

@Devin, this may be off-topic, but I'm pretty sure I'm responsible for the lack of defspec in the transducers tests you're talking about, and the reason is simply that I couldn't find a way with defspec to get the clarity of the reporting to the level I wanted.

Comment by Michael Blume [ 20/Apr/15 12:06 AM ]

See CLJ-1621

Comment by Michael Blume [ 20/Apr/15 12:11 AM ]

(I've been meaning for ages to come up with a proposal for test.check itself that would handle that sort of reporting for the user, but never came up with anything I liked enough)

Comment by Devin Walters [ 22/Apr/15 7:47 PM ]

Hey Michael, thanks for the background. I discussed this briefly with Alex at Clojure/West. By the sound of it, using defspec here should be fine. He mentioned he'd take a peek at it.

Comment by Steve Miner [ 23/Apr/15 12:36 PM ]

Regarding the test code, you could use gen/s-pos-int instead of (gen/such-that #(> % 0) gen/nat).

Comment by Alex Miller [ 24/Apr/15 8:20 AM ]

Tweaked one line in the test per Steve's suggestion.

Comment by Nelson Morris [ 24/Apr/15 8:40 AM ]

Since this is still open to suggestions, if I were to write the spec over again I'd probably use (such-that #(not= 0 %) gen/int) for the step. A negative step will produce non-empty lists when end < start, and those might be worth checking. The case to avoid is one that makes an infinite range like (repeat 0 1 0), due to the equality check in the property. A more complicated generator could probably avoid that without a such-that filter, but not sure it would be worth the effort in this case.

Comment by Alex Miller [ 24/Apr/15 9:02 AM ]

Well Rich has now ok'ed it so would prefer no more changes to this patch.





[CLJ-1706] top level conditional splicing ignores all but first element Created: 15/Apr/15  Updated: 21/May/15  Resolved: 21/May/15

Status: Closed
Project: Clojure
Component/s: None
Affects Version/s: Release 1.7
Fix Version/s: Release 1.7

Type: Defect Priority: Major
Reporter: Nicola Mometto Assignee: Unassigned
Resolution: Completed Votes: 2
Labels: reader

Attachments: Text File 0001-CLJ-1706-Make-top-level-reader-conditional-splicing-.patch     Text File clj-1706-2.patch     Text File clj-1706-3.patch     Text File clj-1706-make-error.patch    
Patch: Code and Test
Approval: Ok

 Description   
user=> (def a (java.io.PushbackReader. (java.io.StringReader. "#?@(:clj [1 2])")))
#'user/a
user=> (read {:read-cond :allow} a)
1
user=> (read {:read-cond :allow} a)
RuntimeException EOF while reading  clojure.lang.Util.runtimeException (Util.java:221)

Currently the reader is stateless (read is a static call) but utilizes a stateful reader (and has a few hooks into compiler/runtime state for autoresolving keywords, etc). If the call into the reader at the top level calls a splicing reader conditional, then only the first one will be returned. The remaining forms are stranded in the pendingForms list and will be lost for subsequent reads.

Approach: Make top level reader conditional splicing an error:

user=> (read-string {:read-cond :allow} "#?@(:clj [1 2])")
IllegalStateException Reader conditional splicing not allowed at the top level.  clojure.lang.LispReader.read (LispReader.java:200)

Patch: clj-1706-2.patch

Alternatives:

1. Make top-level reader conditional splicing an error and throw an exception. (SELECTED)

2. Allow the caller to pass in a stateful collection to catch or return the pendingForms. This changes the effective calling API for the reader. You would only need to do this in the cases where reader conditionals were allowed/preserved.

3. Add a static (or threadlocal?) pendingForms attribute to the reader to capture the pendingForms across calls. A static field would have concurrency issues - anyone using the reader across threads would get cross-talk in this field. The pendingForms could be threadlocal which would probably achieve separation in the majority of cases, but also creates a number of lifecycle questions about those forms. When do they get cleared or reset? What happens if reading the same reader happens across threads? Another option would be an identity map keyed by reader instance - would need to be careful about lifecycle management and clean up, as it's basically a cache.

4. Add more state into the reader itself to capture the pendingForms. The reader interfaces and hierarchy would be affected. This would allow the reader to stop passing the pendingForms around inside but modifies the interface in other ways. Again, this would only be needed for the specific case where reader conditionals are allowed so other uses could continue to work as is?

5. If read is going to exit with pendingForms on the stack, they could be printed and pushed back onto the reader. This adds new read/print roundtrip requirements on things at the top level of reader conditionals that didn't exist before.

6. Wrap spliced forms at the top level in a `do`. This seems to violate the intention of splicing reader conditional to read as spliced since it is not the same as if those forms were placed separately in the input stream.



 Comments   
Comment by Alex Miller [ 15/Apr/15 2:05 PM ]

pulling into 1.7 so we can discuss

Comment by Alex Miller [ 24/Apr/15 11:18 AM ]

Compiler.load() makes calls into LispReader.read() (static call). If the reader reads a top-level splicing conditional read form, that will read the entire form, then return the first spliced element. when LispReader.read() returns, the list carrying the other pending forms is lost.

I see two options:

1) Allow the compiler to call the LispReader with a mutable pendingForms list, basically maintaining that state across the static calls from outside the reader. makes the calling model more complicated and exposes the internal details of the pendingform stuff, but is probably the smaller change.

2) Enhance the LineNumberingPushbackReader in some way to remember the pending forms. This would probably allow us to remove the pending form stuff carried around throughout the LispReader and retain the existing (sensible) api. Much bigger change but probably better direction.

Comment by Nicola Mometto [ 24/Apr/15 11:24 AM ]

What about simply disallowing cond-splicing when top level?
Both proposed options are breaking changes since read currently only requires a java.io.PushbackReader

Comment by Alex Miller [ 24/Apr/15 11:42 AM ]

We want to allow top-level cond-splicing.

Comment by Nicola Mometto [ 24/Apr/15 11:52 AM ]

Would automatically wrapping a top-level cond-splicing in a (do ..) form be out of the question?

I'm personally opposed to supporting this feature as it would change the contract of c.c/read, complicate the implementation of LineNumberingPushbackReader or LispReader and complicate significantly the implementaion of tools.reader's reader types, for no significant benefit.
Is it really that important to be able to write

#~@(:clj [1 2])

rather than

(do #~@(:clj [1 2]))

?

Comment by Rich Hickey [ 18/May/15 10:10 AM ]

Please "Make top-level reader conditional splicing an error and throw an exception" for now.

Comment by Nicola Mometto [ 19/May/15 8:50 AM ]

Might be too late since Rich already gave the OK but the proposed patch doesn't prevent single-element top level conditional splicing forms.
e.g

;; this fails
#~@(:clj [1 2])
;; this works
#~@(:clj [1])

Is this intended?

Comment by Alex Miller [ 19/May/15 11:21 AM ]

New -2 patch catches reader conditional splice of 0 or 1 element.

Comment by Nicola Mometto [ 19/May/15 11:59 AM ]

Attached alternative patch that is less intrusive than clj-1706-2.patch

Comment by Alex Miller [ 19/May/15 2:54 PM ]

clj-1706-3.patch is identical to 0001-CLJ-1706Make-top-level-reader-conditional-splicing.patch but with one whitespace change reverted. Marking latest as screened.

Comment by Alex Miller [ 20/May/15 8:38 AM ]

Rich didn't like the dynvar in -3, so switching back to -2.





[CLJ-1703] Pretty print #error data Created: 14/Apr/15  Updated: 24/Apr/15  Resolved: 24/Apr/15

Status: Closed
Project: Clojure
Component/s: None
Affects Version/s: Release 1.7
Fix Version/s: Release 1.7

Type: Enhancement Priority: Major
Reporter: Alex Miller Assignee: Alex Miller
Resolution: Completed Votes: 0
Labels: print

Attachments: Text File clj-1703-2.patch     Text File clj-1703.patch    
Patch: Code
Approval: Ok

 Description   

Some of the work we were doing re socket repls got pushed out but it would still be nice to expose the pretty printed #error formatting as the current version is very hard to read.

1.7.0-beta1:

user=> *99

CompilerException java.lang.RuntimeException: Unable to resolve symbol: *99 in this context, compiling:(/private/var/folders/7r/_1fj0f517rgcxwx79mn79mfc0000gn/T/form-init8696775124159270468.clj:1:1263)
user=> (prn *e)
#error{:cause "Unable to resolve symbol: *99 in this context", :via [{:type clojure.lang.Compiler$CompilerException, :message "java.lang.RuntimeException: Unable to resolve symbol: *99 in this context, compiling:(/private/var/folders/7r/_1fj0f517rgcxwx79mn79mfc0000gn/T/form-init8696775124159270468.clj:1:1263)", :at [clojure.lang.Compiler analyze "Compiler.java" 6543]} {:type java.lang.RuntimeException, :message "Unable to resolve symbol: *99 in this context", :at [clojure.lang.Util runtimeException "Util.java" 221]}], :trace [[clojure.lang.Util runtimeException "Util.java" 221] [clojure.lang.Compiler resolveIn "Compiler.java" 7029] [clojure.lang.Compiler resolve "Compiler.java" 6973] [clojure.lang.Compiler analyzeSymbol "Compiler.java" 6934] [clojure.lang.Compiler analyze "Compiler.java" 6506] [clojure.lang.Compiler analyze "Compiler.java" 6485] [clojure.lang.Compiler eval "Compiler.java" 6796] [clojure.lang.Compiler eval "Compiler.java" 6755] [clojure.core$eval invoke "core.clj" 3082] [clojure.main$repl$read_eval_print__7057$fn__7060 invoke "main.clj" 240] [clojure.main$repl$read_eval_print__7057 invoke "main.clj" 240] [clojure.main$repl$fn__7066 invoke "main.clj" 258] [clojure.main$repl doInvoke "main.clj" 258] [clojure.lang.RestFn invoke "RestFn.java" 1523] [clojure.tools.nrepl.middleware.interruptible_eval$evaluate$fn__599 invoke "interruptible_eval.clj" 53] [clojure.lang.AFn applyToHelper "AFn.java" 152] [clojure.lang.AFn applyTo "AFn.java" 144] [clojure.core$apply invoke "core.clj" 628] [clojure.core$with_bindings_STAR_ doInvoke "core.clj" 1866] [clojure.lang.RestFn invoke "RestFn.java" 425] [clojure.tools.nrepl.middleware.interruptible_eval$evaluate invoke "interruptible_eval.clj" 51] [clojure.tools.nrepl.middleware.interruptible_eval$interruptible_eval$fn__641$fn__644 invoke "interruptible_eval.clj" 183] [clojure.tools.nrepl.middleware.interruptible_eval$run_next$fn__634 invoke "interruptible_eval.clj" 152] [clojure.lang.AFn run "AFn.java" 22] [java.util.concurrent.ThreadPoolExecutor runWorker "ThreadPoolExecutor.java" 1142] [java.util.concurrent.ThreadPoolExecutor$Worker run "ThreadPoolExecutor.java" 617] [java.lang.Thread run "Thread.java" 744]]}

Approach: Do some minimal formatting of the #error data, should be pretty close to (pprint (Throwable->map *e)).

w/patch:

user=> (prn *e)
#error {
 :cause "Unable to resolve symbol: *99 in this context"
 :via
 [{:type clojure.lang.Compiler$CompilerException
   :message "java.lang.RuntimeException: Unable to resolve symbol: *99 in this context, compiling:(NO_SOURCE_PATH:0:0)"
   :at [clojure.lang.Compiler analyze "Compiler.java" 6543]}
  {:type java.lang.RuntimeException
   :message "Unable to resolve symbol: *99 in this context"
   :at [clojure.lang.Util runtimeException "Util.java" 221]}]
 :trace
 [[clojure.lang.Util runtimeException "Util.java" 221]
  [clojure.lang.Compiler resolveIn "Compiler.java" 7029]
  [clojure.lang.Compiler resolve "Compiler.java" 6973]
  [clojure.lang.Compiler analyzeSymbol "Compiler.java" 6934]
  [clojure.lang.Compiler analyze "Compiler.java" 6506]
  [clojure.lang.Compiler analyze "Compiler.java" 6485]
  [clojure.lang.Compiler eval "Compiler.java" 6796]
  [clojure.lang.Compiler eval "Compiler.java" 6755]
  [clojure.core$eval invoke "core.clj" 3079]
  [clojure.main$repl$read_eval_print__7093$fn__7096 invoke "main.clj" 240]
  [clojure.main$repl$read_eval_print__7093 invoke "main.clj" 240]
  [clojure.main$repl$fn__7102 invoke "main.clj" 258]
  [clojure.main$repl doInvoke "main.clj" 258]
  [clojure.lang.RestFn invoke "RestFn.java" 421]
  [clojure.main$repl_opt invoke "main.clj" 324]
  [clojure.main$main doInvoke "main.clj" 422]
  [clojure.lang.RestFn invoke "RestFn.java" 397]
  [clojure.lang.Var invoke "Var.java" 375]
  [clojure.lang.AFn applyToHelper "AFn.java" 152]
  [clojure.lang.Var applyTo "Var.java" 700]
  [clojure.main main "main.java" 37]]}

Patch: clj-1703-2.patch



 Comments   
Comment by Rich Hickey [ 17/Apr/15 9:48 AM ]

Can we please put the kv pairs of via each on their own line?





[CLJ-1700] REPL evaluation of conditional reader forms fails Created: 12/Apr/15  Updated: 24/Apr/15  Resolved: 24/Apr/15

Status: Closed
Project: Clojure
Component/s: None
Affects Version/s: Release 1.7
Fix Version/s: Release 1.7

Type: Defect Priority: Minor
Reporter: Luke VanderHart Assignee: Unassigned
Resolution: Completed Votes: 3
Labels: reader
Environment:

1.7-beta1


Attachments: Text File clj-1700.patch    
Patch: Code
Approval: Ok

 Description   

When using reader conditionals, evaluating a reader conditional in a vanilla command-line REPL (not nRepl or anything like that) results in a "Conditional read not allowed" error message.

Loading the whole file with load-file works as expected.

This breaks the very normal workflow of eval-ing forms from a *.cljc file in a Clojure repl using (e.g.) inferior lisp.

Approach: clojure.main/repl (also used by swank I think) enables reader conditionals at the REPL.

Patch: clj-1700.patch



 Comments   
Comment by Colin Jones [ 21/Apr/15 12:53 AM ]

Looks/works great for me - I quite literally wrote the exact same patch before talking w/ Luke today about this.





[CLJ-1699] data_readers hard coded to .clj extension, should be extended to .cljc Created: 10/Apr/15  Updated: 24/Apr/15  Resolved: 24/Apr/15

Status: Closed
Project: Clojure
Component/s: None
Affects Version/s: Release 1.7
Fix Version/s: Release 1.7

Type: Enhancement Priority: Major
Reporter: David Nolen Assignee: Unassigned
Resolution: Completed Votes: 0
Labels: reader

Attachments: Text File clj-1699.patch    
Patch: Code
Approval: Ok

 Description   

Currently using data_readers in ClojureScript is difficult because the extensions are not available to both compile time and runtime as they are in Clojure. This is fairly straightforward to remedy now given the presence of conditional reading - simply supply data_readers.cljc.

Approach: Find and read both data_readers.clj and data_readers.cljc. For cljc, allow reader conditionals.

Alternative: Another option would be to just allow reader conditionals on the existing data_readers.clj file. That's a simpler patch but possibly confusing given that conditionals are only available in .cljc files right now.

Patch: clj-1699.patch - tested with a variety of manual tests



 Comments   
Comment by David Nolen [ 10/Apr/15 4:23 PM ]

This could be solved trivially by concatenated data_readers.cljc resources to the return value of clojure.core/data-reader-urls.





[CLJ-1698] conditional reading bugs Created: 10/Apr/15  Updated: 24/Apr/15  Resolved: 24/Apr/15

Status: Closed
Project: Clojure
Component/s: None
Affects Version/s: Release 1.7
Fix Version/s: Release 1.7

Type: Defect Priority: Major
Reporter: David Nolen Assignee: Unassigned
Resolution: Completed Votes: 3
Labels: reader

Attachments: Text File 0001-CLJ-1698-fix-conditional-reading-bugs.patch    
Patch: Code and Test
Approval: Ok

 Description   

Bugs in conditional reading:

(ns bug)

#?(:cljs {'a 1 'b 2})

#?(:cljs (let [{{b :b} :a {d :d} :c} {}]))

Requiring / loading this file at the REPL results in the following exception:

CompilerException java.lang.IllegalArgumentException: Duplicate key: null, compiling:
	clojure.lang.Compiler.load (Compiler.java:7244)
	clojure.lang.RT.loadResourceScript (RT.java:371)
	clojure.lang.RT.loadResourceScript (RT.java:362)
	clojure.lang.RT.load (RT.java:446)
	clojure.lang.RT.load (RT.java:412)
	clojure.core/load/fn--5427 (core.clj:5862)
	clojure.core/load (core.clj:5861)
	clojure.core/load-one (core.clj:5667)
	clojure.core/load-lib/fn--5376 (core.clj:5707)
	clojure.core/load-lib (core.clj:5706)
	clojure.core/apply (core.clj:630)
	clojure.core/load-libs (core.clj:5745)
Caused by:
IllegalArgumentException Duplicate key: null
	clojure.lang.PersistentArrayMap.createWithCheck (PersistentArrayMap.java:70)
	clojure.lang.RT.map (RT.java:1537)
	clojure.lang.LispReader$MapReader.invoke (LispReader.java:1152)
	clojure.lang.LispReader.read (LispReader.java:255)
	clojure.lang.LispReader.access$800 (LispReader.java:40)
	clojure.lang.LispReader$ConditionalReader.readCondDelimited (LispReader.java:1376)
	clojure.lang.LispReader$ConditionalReader.invoke (LispReader.java:1448)
	clojure.lang.LispReader$DispatchReader.invoke (LispReader.java:684)
	clojure.lang.LispReader.read (LispReader.java:255)
	clojure.lang.LispReader.readDelimitedList (LispReader.java:1191)
	clojure.lang.LispReader$ListReader.invoke (LispReader.java:1040)
	clojure.lang.LispReader.read (LispReader.java:255)
(ns bug)

(def m #?(:cljs ^{:a :b} {}
          :clj  ^{:a :b} {}))
CompilerException java.lang.IllegalArgumentException: Metadata must be Symbol,Keyword,String or Map, compiling:(bug.cljc:3:25)
	clojure.lang.Compiler.load (Compiler.java:7244)
	clojure.lang.RT.loadResourceScript (RT.java:371)
	clojure.lang.RT.loadResourceScript (RT.java:362)
	clojure.lang.RT.load (RT.java:446)
	clojure.lang.RT.load (RT.java:412)
	clojure.core/load/fn--5427 (core.clj:5862)
	clojure.core/load (core.clj:5861)
	clojure.core/load-one (core.clj:5667)
	clojure.core/load-lib/fn--5376 (core.clj:5707)
	clojure.core/load-lib (core.clj:5706)
	clojure.core/apply (core.clj:630)
	clojure.core/load-libs (core.clj:5745)
Caused by:
IllegalArgumentException Metadata must be Symbol,Keyword,String or Map
	clojure.lang.LispReader$MetaReader.invoke (LispReader.java:790)
	clojure.lang.LispReader.read (LispReader.java:255)
	clojure.lang.LispReader.access$800 (LispReader.java:40)
	clojure.lang.LispReader$ConditionalReader.readCondDelimited (LispReader.java:1376)
	clojure.lang.LispReader$ConditionalReader.invoke (LispReader.java:1448)
	clojure.lang.LispReader$DispatchReader.invoke (LispReader.java:684)
	clojure.lang.LispReader.read (LispReader.java:255)
	clojure.lang.LispReader.readDelimitedList (LispReader.java:1191)
	clojure.lang.LispReader$ListReader.invoke (LispReader.java:1040)
	clojure.lang.LispReader.read (LispReader.java:255)
	clojure.lang.LispReader.read (LispReader.java:195)
	clojure.lang.Compiler.load (Compiler.java:7232)
(ns bug)

#?(:cljs {:a #_:b :c})
CompilerException java.lang.RuntimeException: Map literal must contain an even number of forms

Cause: Not properly handling suppress-read flag.

Patch: 0001-CLJ-1698-fix-conditional-reading-bugs.patch



 Comments   
Comment by Alex Miller [ 11/Apr/15 5:49 AM ]

For Clojure REPL repro:

(def opts {:features #{:clj} :read-cond :allow})
(read-string opts "#?(:cljs {'a 1 'b 2})")
(read-string opts "#?(:cljs (let [{{b :b} :a {d :d} :c} {}]))")
(read-string opts "(def m #?(:cljs ^{:a :b} {} :clj  ^{:a :b} {}))")
(read-string opts "(def m #?(:cljs ^{:a :b} {} :clj ^{:a :b} {}))")
(read-string opts "#?(:cljs {:a #_:b :c}")




[CLJ-1697] Primitive lexical bindings trigger compile time exceptions under certain conditions Created: 07/Apr/15  Updated: 07/Apr/15  Resolved: 07/Apr/15

Status: Closed
Project: Clojure
Component/s: None
Affects Version/s: Release 1.7
Fix Version/s: None

Type: Defect Priority: Major
Reporter: Adrian Medina Assignee: Unassigned
Resolution: Completed Votes: 0
Labels: bug
Environment:

Java 8, Clojure 1.7.0-alpha6



 Description   

A compile time exception is thrown when a value returned from a function with a primitive return signature is lexically bound and used in subsequent bindings within the same let expression.

Example code which exhibits the behavior is available as a gist here:
https://gist.github.com/aamedina/82fee074fb2fb398d4e1

Relevant stack traces are referenced in the comments.



 Comments   
Comment by Adrian Medina [ 07/Apr/15 6:23 PM ]

This does not seem to be a bug. The primitive type hint needs to be precede every argument vector, unlike other return type hinting used to elide reflection.





[CLJ-1696] Generating BigInt sequences with range Created: 04/Apr/15  Updated: 06/Apr/15  Resolved: 06/Apr/15

Status: Closed
Project: Clojure
Component/s: None
Affects Version/s: Release 1.6
Fix Version/s: None

Type: Enhancement Priority: Minor
Reporter: Paul Roush Assignee: Unassigned
Resolution: Declined Votes: 0
Labels: None


 Description   

I'm a relative newcomer to the Clojure language, so this input may cover old ground or be seen as naive. The primary substance is a philosophical question, so the response may be that I'm not thinking about this the "right way".

But, for what it's worth, here is the request.

I found myself in a situation where I wanted 'range' to return a sequence of BigInt's. I naively tried something like (range 5N) and had no luck. Later I found that (range 0N 5) would generate BigInt's rather than Long's.

So my first observation / request is that there are cases where it could be "useful" to be able to generate a sequence of BigInt's with a 1-arity call to range.

The bigger point to me, though, is more a philosophical question of whether the behavior of 'range' is as "consistent" with other core API's as it might be. Obviously, this is very subjective.

My naive thinking was something like this...

(* 2 3N 4) => 24N
(* 2 3 4N) => 24N

etc.

...so passing a BigInt value in as any of the 3 possible args to range would promote the result to a sequence of BigInt.

As I now understand, passing in a BigInt as the 'start' value generates a BigInt sequence, but a BigInt 'end' or 'step' value will generate Long's.

To be precise, this "enhancement request" is to modify 'range' so that passing a BigInt as any one of the 3 possible arguments will yield a sequence of BigInt values.

It might also be viewed as a request to make the documentation more explicit in regard to the current behavior.



 Comments   
Comment by Alex Miller [ 06/Apr/15 8:50 AM ]

This is relatively subtle, but the key is that the values of the range are computed based on start (defaults to 0) and step (default to 1) where 0 and 1 are longs. The end value is used for bounds checking but not for computation. From those pieces of info (which are in the docstring, but the long type of the defaults is implied by the lack of N), plus the default numerics behavior of range, I think it's reasonable to predict how this function will work.

I think that modifying the behavior for the computation based on the type of the end value actually breaks the mental model for me and is harder to understand. This, combined with the infrequency of the use case, leads me to close this enhancement.





[CLJ-1695] Variadic vector-of overload has poor performance Created: 03/Apr/15  Updated: 10/Apr/15  Resolved: 10/Apr/15

Status: Closed
Project: Clojure
Component/s: None
Affects Version/s: Release 1.6
Fix Version/s: Release 1.7

Type: Enhancement Priority: Major
Reporter: Leon Grapenthin Assignee: Unassigned
Resolution: Completed Votes: 0
Labels: performance

Attachments: Text File clj-1695-2.patch     Text File clj-1695.patch    
Patch: Code
Approval: Ok

 Description   
(time (do (apply vector-of :double (repeat 100000 0))
                nil))

Times after a few runs ~335 ms.

(time (do (into (vector-of :double) (repeat 100000 0))
                nil))

Times after a few runs ~5 ms.

Cause: The variadic case for vector-of is missing a type hint and uses reflection - this will be seen in any call to vector-of with more than 4 elements.

Approach: Switch interop call to instead use conj - there is no reason to be using interop here over standard conj on a vector. The after time for the first case above is 6-9 ms depending on GC (the fast reduce path in repeat reduces gc variability I think).

Patch: clj-1695-2.patch



 Comments   
Comment by Leon Grapenthin [ 03/Apr/15 4:57 PM ]

I thought seqs were passed pretty much "as is" with apply. E.g.:

(time (do (apply (fn [& args]
(into (vector-of :double) args))
(repeat 100000 0))
nil))

performs as fast as (into (vector-of :double) (repeat 100000 0)

Comment by Alex Miller [ 06/Apr/15 9:51 AM ]

I'm going to see if we can include this in 1.7, no guarantees. For now a workaround is any use of vector-of that creates, then fills the vector separately as you are doing with into.

The into case actually will get an additional benefit in 1.7 for sources that can reduce themselves faster (repeat happens to be one of those).

Comment by Michael Blume [ 06/Apr/15 11:02 AM ]

If we're adding a type-hint anyway, wouldn't it be more effective to type-hint the return value of vector-of?

Comment by Alex Miller [ 06/Apr/15 11:10 AM ]

I went back-and-forth on that but it didn't seem like that any other code could benefit from the return type hint?

Re-thinking, maybe the hint should actually be more generic and hint to clojure.lang.IPersistentCollection instead.

Comment by Michael Blume [ 06/Apr/15 11:15 AM ]

Actually I'm wrong, type-hinting the return of vector-of (as either Vec or IPersistentCollection) doesn't remove the reflection.

Comment by Alex Miller [ 06/Apr/15 11:37 AM ]

Attached new patch with more generic typehint, placed more specifically where it's needed.

Comment by Alex Miller [ 10/Apr/15 9:52 AM ]

Switched interop call to just





[CLJ-1694] cycle is too eager Created: 03/Apr/15  Updated: 10/Apr/15  Resolved: 10/Apr/15

Status: Closed
Project: Clojure
Component/s: None
Affects Version/s: Release 1.7
Fix Version/s: Release 1.7

Type: Defect Priority: Major
Reporter: Alex Miller Assignee: Unassigned
Resolution: Completed Votes: 0
Labels: regression

Attachments: Text File clj-1694.patch    
Patch: Code and Test
Approval: Ok

 Description   

Same as CLJ-1692, cycle realizes one element ahead in next(), which is different than before CLJ-1603 was applied:

(take 2 (cycle (map #(/ 42 %) '(2 1 0))))

Approach: Delay realization until first is called.

Patch: clj-1694.patch



 Comments   
Comment by Alex Miller [ 03/Apr/15 2:56 PM ]

Actually, the behavior here is no different vs before.

(take 2 (cycle (map #(/ 42 %) (range 31 -1 -1))))
;; exception

(take 2 (cycle (map #(/ 42 %) (range 32 -1 -1))))
;; works

Just a side effect of chunking and afaict not any different than before.

Comment by Nicola Mometto [ 03/Apr/15 3:21 PM ]

The regression does exist, just a bad example.
Here's a valid one:

Clojure 1.7.0-master-SNAPSHOT
user=> (take 2 (cycle (map #(/ 42 %) '(2 1 0))))
ArithmeticException Divide by zero  clojure.lang.Numbers.divide (Numbers.java:158)
Clojure 1.6.0
user=> (take 2 (cycle (map #(/ 42 %) '(2 1 0))))
(21 42)




[CLJ-1692] Iterate is too eager Created: 01/Apr/15  Updated: 10/Apr/15  Resolved: 10/Apr/15

Status: Closed
Project: Clojure
Component/s: None
Affects Version/s: Release 1.7
Fix Version/s: Release 1.7

Type: Defect Priority: Major
Reporter: Ambrose Bonnaire-Sergeant Assignee: Unassigned
Resolution: Completed Votes: 0
Labels: regression

Attachments: Text File clj-1692-2.patch     Text File clj-1692-3.patch     Text File clj1692.patch    
Patch: Code and Test
Approval: Ok

 Description   

1.7's iterate calls its function one extra time than in 1.6

;; clojure 1.6
user=> (take 2 (iterate zero? 0))
(0 true)

;; clojure 1.7-alpha6
user=> (take 2 (iterate zero? 0))
ClassCastException java.lang.Boolean cannot be cast to java.lang.Number  clojure.lang.Numbers.isZero (Numbers.java:92)

This is because Iterate.java calls its function in its "next" method rather than in its "first" method.

Approach: Iterate now holds a prevSeed that will allow computation of a lazily computed seed (the first of the iterate sequence). The very first node will be initialized with the initial seed. All other uses of seed need to ensure first is called to force realization before using seed.

Patch: clj-1692-3.patch

  • note there were two copies of test-iterate so one is removed in the test, allowing the more expansive one to be called.


 Comments   
Comment by Alex Miller [ 02/Apr/15 9:31 AM ]

What about something like clj-1692-2.patch instead? Seems simpler to me. Also, why did the first patch remove the iterate tests?

Comment by Nicola Mometto [ 02/Apr/15 9:55 AM ]

Alex, unless I'm reading it wrong, your patch addresses the symptom of this bug rather than its cause (iterate is now eager than it was before), I personally prefer Ambrose's approach.

WRT removing the iterate tests, it looks like test-iterate is duplicated at line 781 and at line 970 of sequences.clj, Ambrose's patch just removes the duplicate test.

Comment by Ambrose Bonnaire-Sergeant [ 02/Apr/15 10:57 AM ]

To be clear, removing the redundant deftest actually reveals several tests that were previously hidden.

Comment by Alex Miller [ 02/Apr/15 12:01 PM ]

Ok, fair points. I renamed some fields in the first patch to (in my opinion) better reflect their role. There were also several bugs related to using seed without guaranteeing it's computation, for example both of these threw exceptions:

(first (next (next (iterate inc 0))))
(into [] (take 3) (next (iterate inc 0)))

I added those tests as well. I re-ran the perf test from CLJ-1603 and there is definitely some degradation on (doall (take 1000 (iterate inc 0))) from 75 us to 85 us, but that's still a significant win over the prior version.

Comment by Fogus [ 03/Apr/15 1:36 PM ]

I think patch-3 is sound, and I suspect that only Ambrose would have caught it. Thanks!

Comment by Ambrose Bonnaire-Sergeant [ 03/Apr/15 1:48 PM ]

This issue was not caught by Typed Clojure, rather the peculiarities of its implementation https://github.com/clojure/core.typed/commit/1027f792accdc3e5da3475745da0106f0ad5e1cc#diff-eec0f851200aca22af17269fcab94719L1759

I was using iterate to dig down a structure, and was being very careful to only `take` exactly enough.





[CLJ-1691] Occasional failure of seq-and-transducer test Created: 01/Apr/15  Updated: 10/Apr/15  Resolved: 10/Apr/15

Status: Closed
Project: Clojure
Component/s: None
Affects Version/s: Release 1.7
Fix Version/s: Release 1.7

Type: Defect Priority: Major
Reporter: Andy Fingerhut Assignee: Alex Miller
Resolution: Completed Votes: 0
Labels: None

Attachments: Text File build-failure2.txt     Text File clj-1691.patch    
Patch: Code
Approval: Ok

 Description   

I started a 'mvn clean ; mvn test' build loop on a machine yesterday. I saw 2 failures of the test seq-and-transducer in transducers.clj out of 520 runs (and no other failures). The first failure message appears below. The second is in the attachment build-fail2.txt. I haven't yet analyzed them carefully to see whether the generated test is useful.

[java] Testing clojure.test-clojure.transducers
     [java] 
     [java] FAIL in (seq-and-transducer) (transducers.clj:143)
     [java] {:coll [0 0],
     [java]  :actions
     [java]  (->>
     [java]   coll
     [java]   (partition-all 1)
     [java]   (interpose 1)
     [java]   dedupe
     [java]   (remove empty?)
     [java]   (take-while empty?)),
     [java]  :s
     [java]  #error{:cause "Don't know how to create ISeq from: java.lang.Long", :via [{:type java.lang.IllegalArgumentException, :message "Don't know how to create ISeq from: java.lang.Long", :at [clojure.lang.RT seqFrom "RT.java" 528]}], :trace [[clojure.lang.RT seqFrom "RT.java" 528] [clojure.lang.RT seq "RT.java" 509] [clojure.core$seq__4106 invoke "core.clj" 135] [clojure.core$empty_QMARK_ invoke "core.clj" 5946] [clojure.core$complement$fn__4358 invoke "core.clj" 1374] [clojure.core$filter$fn__4558 invoke "core.clj" 2683] [clojure.lang.LazySeq sval "LazySeq.java" 40] [clojure.lang.LazySeq seq "LazySeq.java" 49] [clojure.lang.RT seq "RT.java" 507] [clojure.core$seq__4106 invoke "core.clj" 135] [clojure.core$take_while$fn__4582 invoke "core.clj" 2771] [clojure.lang.LazySeq sval "LazySeq.java" 40] [clojure.lang.LazySeq seq "LazySeq.java" 49] [clojure.lang.RT seq "RT.java" 507] [clojure.core$seq__4106 invoke "core.clj" 135] [clojure.core$dorun invoke "core.clj" 3010] [clojure.core$doall invoke "core.clj" 3026] [clojure.test_clojure.transducers$apply_as_seq invoke "transducers.clj" 82] [clojure.test_clojure.transducers$build_results$fn__26512 invoke "transducers.clj" 115] [clojure.test_clojure.transducers$build_results invoke "transducers.clj" 115] [clojure.test_clojure.transducers$fn__26524 invoke "transducers.clj" 130] [clojure.test.check.rose_tree$fmap invoke "rose_tree.clj" 46] [clojure.core$partial$fn__4505 invoke "core.clj" 2491] [clojure.core$map$fn__4531 invoke "core.clj" 2622] [clojure.lang.LazySeq sval "LazySeq.java" 40] [clojure.lang.LazySeq seq "LazySeq.java" 49] [clojure.lang.RT seq "RT.java" 507] [clojure.core$seq__4106 invoke "core.clj" 135] [clojure.test.check.rose_tree$permutations$iter__26065__26069$fn__26070$iter__26089__26093$fn__26094 invoke "rose_tree.clj" 73] [clojure.lang.LazySeq sval "LazySeq.java" 40] [clojure.lang.LazySeq seq "LazySeq.java" 49] [clojure.lang.RT seq "RT.java" 507] [clojure.core$seq__4106 invoke "core.clj" 135] [clojure.core$map$fn__4531 invoke "core.clj" 2614] [clojure.lang.LazySeq sval "LazySeq.java" 40] [clojure.lang.LazySeq seq "LazySeq.java" 49] [clojure.lang.RT seq "RT.java" 507] [clojure.core$seq__4106 invoke "core.clj" 135] [clojure.core$map$fn__4531 invoke "core.clj" 2614] [clojure.lang.LazySeq sval "LazySeq.java" 40] [clojure.lang.LazySeq seq "LazySeq.java" 49] [clojure.lang.Cons next "Cons.java" 39] [clojure.lang.RT next "RT.java" 674] [clojure.core$next__4090 invoke "core.clj" 64] [clojure.core$nthnext invoke "core.clj" 3039] [clojure.test.check$shrink_loop invoke "check.clj" 94] [clojure.test.check$failure invoke "check.clj" 121] [clojure.test.check$quick_check doInvoke "check.clj" 65] [clojure.lang.RestFn invoke "RestFn.java" 425] [clojure.test_clojure.transducers$fn__26531 invoke "transducers.clj" 139] [clojure.test$test_var$fn__7628 invoke "test.clj" 704] [clojure.test$test_var invoke "test.clj" 704] [clojure.test$test_vars$fn__7650$fn__7655 invoke "test.clj" 722] [clojure.test$default_fixture invoke "test.clj" 674] [clojure.test$test_vars$fn__7650 invoke "test.clj" 722] [clojure.test$default_fixture invoke "test.clj" 674] [clojure.test$test_vars invoke "test.clj" 718] [clojure.test$test_all_vars invoke "test.clj" 728] [clojure.test$test_ns invoke "test.clj" 747] [clojure.core$map$fn__4531 invoke "core.clj" 2622] [clojure.lang.LazySeq sval "LazySeq.java" 40] [clojure.lang.LazySeq seq "LazySeq.java" 49] [clojure.lang.Cons next "Cons.java" 39] [clojure.lang.RT next "RT.java" 674] [clojure.core$next__4090 invoke "core.clj" 64] [clojure.core$reduce1 invoke "core.clj" 907] [clojure.core$reduce1 invoke "core.clj" 898] [clojure.core$merge_with doInvoke "core.clj" 2937] [clojure.lang.RestFn applyTo "RestFn.java" 139] [clojure.core$apply invoke "core.clj" 630] [clojure.test$run_tests doInvoke "test.clj" 762] [clojure.lang.RestFn applyTo "RestFn.java" 137] [clojure.core$apply invoke "core.clj" 628] [user$eval28346 invoke "run_test.clj" 7] [clojure.lang.Compiler eval "Compiler.java" 6792] [clojure.lang.Compiler load "Compiler.java" 7237] [clojure.lang.Compiler loadFile "Compiler.java" 7175] [clojure.main$load_script invoke "main.clj" 275] [clojure.main$script_opt invoke "main.clj" 337] [clojure.main$main doInvoke "main.clj" 421] [clojure.lang.RestFn invoke "RestFn.java" 408] [clojure.lang.Var invoke "Var.java" 379] [clojure.lang.AFn applyToHelper "AFn.java" 154] [clojure.lang.Var applyTo "Var.java" 700] [clojure.main main "main.java" 37]]},
     [java]  :xs (),
     [java]  :xi [],
     [java]  :xe [],
     [java]  :xt []}
     [java] 
     [java] expected: (:result res)
     [java]   actual: false


 Comments   
Comment by Ghadi Shayban [ 01/Apr/15 9:34 AM ]

Both failures seem like a test script problem: an 'empty?' check after an interpose of a scalar long.

Mildly surprised that these didn't show up earlier.

Comment by Nicola Mometto [ 01/Apr/15 9:48 AM ]

Looks like this is caused by the chunkedness of keep:

;; lists are not chunked
user=> (->> '([1] 1) (keep empty?) first)
()
;; vector seqs are chunked
user=> (->> '[[1] 1] (keep empty?) first)
IllegalArgumentException Don't know how to create ISeq from: java.lang.Long  clojure.lang.RT.seqFrom (RT.java:528)

In the non-chunked example (mimicking how transducers work), the call to (empty? 1) never gets executed since only the first value is required, on the chunked example otoh (empty? 1) gets invoked causing the exception.





[CLJ-1685] :eof option in clojure.core/read not handled properly Created: 29/Mar/15  Updated: 10/Apr/15  Resolved: 10/Apr/15

Status: Closed
Project: Clojure
Component/s: None
Affects Version/s: Release 1.7
Fix Version/s: Release 1.7

Type: Defect Priority: Major
Reporter: Adrian Medina Assignee: Alex Miller
Resolution: Completed Votes: 0
Labels: reader

Attachments: Text File 0001-CLJ-1685-correctly-handle-eof-option-in-read-read-st.patch     Text File 0001-CLJ-1685-correctly-handle-eof-option-in-read-read-st-v2.patch     Text File clj-1685-v3.patch    
Patch: Code and Test
Approval: Ok

 Description   

Example form which exhibits the behavior:

(read {:read-cond :allow :eof (Object.)} input)

When EOF is reached in the stream, instead of returning the :eof value specified the boolean value true is always returned instead. If you omit :eof from the option map given to clojure.core/read, false is consistently returned and no EOF error is thrown.
Patch: 0001-CLJ-1685-correctly-handle-eof-option-in-read-read-st-v2.patch

Note: Currently

(read {} stream)
behaves like
(read {:eof nil} stream)
rather than
(read stream)
, the proposed patch makes it believe like
(read {:eof :eofthrow} input)
, the proposed patch changes this so that the default behaviour is always to throw on eof unless a :eof option is explicitly included in the read opts.

Patch: clj-1685-v3.patch

Screened by: Alex Miller



 Comments   
Comment by Nicola Mometto [ 29/Mar/15 2:38 PM ]

Attached patch fixes the issue for both read and read-string

Comment by Andy Fingerhut [ 29/Mar/15 2:47 PM ]

Never try to race Nicola to a patch when he is on the task Thanks, Nicola.

Comment by Nicola Mometto [ 30/Mar/15 8:20 AM ]

Alex, currently calls to read/read-string with an empty options map behave as if {:eof nil} was passed, thus

user=> (read-string "")
RuntimeException EOF while reading  clojure.lang.Util.runtimeException (Util.java:221)
user=> (read-string {} "")
nil

i.e, :eof defaults to nil.
Is this intended? if not, the attached patch 0001-CLJ-1685-correctly-handle-eof-option-in-read-read-st-v2.patch
fixes this issue and changes the behaviour of read/read-string to default to :eofthrow rather than to nil

Comment by Alex Miller [ 06/Apr/15 11:30 AM ]

The -v3 patch is identical to -v2, but adds one clarifying docstring addition.





[CLJ-1684] Transducer eduction test is wrong Created: 27/Mar/15  Updated: 31/Mar/15  Resolved: 31/Mar/15

Status: Closed
Project: Clojure
Component/s: None
Affects Version/s: Release 1.7
Fix Version/s: Release 1.7

Type: Defect Priority: Major
Reporter: Alex Miller Assignee: Unassigned
Resolution: Completed Votes: 0
Labels: regression, transducers

Attachments: Text File clj-1684.patch    
Patch: Code and Test
Approval: Ok

 Description   

Error in build:

[java] ERROR in (seq-and-transducer) (TransformerIterator.java:86)
         [java] Uncaught exception, not in assertion.
         [java] expected: nil
         [java]   actual: java.lang.NullPointerException: null
         [java]  at clojure.lang.TransformerIterator.step (TransformerIterator.java:86)
         [java]     clojure.lang.TransformerIterator.hasNext (TransformerIterator.java:97)
         [java]     clojure.lang.RT.chunkIteratorSeq (RT.java:489)
         [java]     clojure.lang.RT.seqFrom (RT.java:518)
         [java]     clojure.lang.RT.seq (RT.java:509)
         [java]     clojure.core/seq (core.clj:135)
         [java]     clojure.core$print_sequential.invoke (core_print.clj:47)
         [java]     clojure.core/fn (core.clj:7362)
         [java]     clojure.lang.MultiFn.invoke (MultiFn.java:233)
         [java]     clojure.core$pr_on.invoke (core.clj:3549)
         [java]     clojure.core$pr.invoke (core.clj:3561)
         [java]     clojure.pprint$pprint_simple_default.invoke (dispatch.clj:144)

There is an error in the generative transducer eduction test that was added in CLJ-1669. This code in transducers.clj:

(apply eduction (into actions [coll]))

is not invoking eduction with actions and a collection. Rather it is putting the actions inside the collection and effectively using no transformation at all as in {{(eduction [1 2])}}, which always passes. The error seen is due to a bad function being passed - if actions happens to be nil (which I think is happening while shrinking another failure), something like (eduction (constantly nil) []) is being called, which we would not expect to work in the first place.

After fixing the bad eduction handling, I was only able to trigger failures with a high number of iterations and a very large number of transformations. The errors reported under these conditions are difficult to understand, I believe because they are hitting StackOverflow errors and the stack traces are being removed by the JVM.

I did some more investigation into whether we are actually generating useful generative tests and found that due to the large stack of transformations, virtually all tests were just producing exceptions, rather than more interesting behavior. I capped the number of transformations to 5 and saw much more useful and interesting tests being generated. I've also doubled the number of transducer tests being run in the patch and ran it locally with a much higher number with no failures.

Patch: clj-1684.patch






[CLJ-1683] test-reduce doesn't catch off-by-one error in IReduce Created: 25/Mar/15  Updated: 27/Mar/15  Resolved: 27/Mar/15

Status: Closed
Project: Clojure
Component/s: None
Affects Version/s: Release 1.7
Fix Version/s: Release 1.7

Type: Enhancement Priority: Minor
Reporter: Michael Blume Assignee: Unassigned
Resolution: Completed Votes: 1
Labels: None

Attachments: Text File clj-1683-v1.patch    
Patch: Code and Test
Approval: Ok

 Description   

To implement the no-init arity of reduce, you have to use the first element of the sequence as an init value, and then you have to skip the first element when you iterate through the sequence. The tests in test-reduce, which focus on using reduce to sum up a bunch of numbers in a range, don't actually pin down this behavior, because the range used starts with zero, and an extra zero doesn't affect the sum.

Screened by: Alex Miller - this is a good tweak to get better tests for the hard this easy to mess up reduce behavior.






[CLJ-1681] reflection warning throws NPE for literal nil arg Created: 24/Mar/15  Updated: 27/Mar/15  Resolved: 27/Mar/15

Status: Closed
Project: Clojure
Component/s: None
Affects Version/s: Release 1.6, Release 1.7
Fix Version/s: Release 1.7

Type: Defect Priority: Major
Reporter: Michael Blume Assignee: Michael Blume
Resolution: Completed Votes: 0
Labels: regression, typehints

Attachments: Text File clj-1681-v1.patch     Text File clj-1681-v2.patch     Text File clj-1681-v3.patch    
Patch: Code and Test
Approval: Ok

 Description   

Seen on another project, but reproducible with this:

user=> (set! *warn-on-reflection* true)
true
user=> (defn f [a] (.divide 1M a nil))
CompilerException java.lang.NullPointerException, compiling:(NO_SOURCE_PATH:2:13)
user=> (pst *e)
CompilerException java.lang.NullPointerException, compiling:(NO_SOURCE_PATH:21:13)
	clojure.lang.Compiler.analyzeSeq (Compiler.java:6740)
	clojure.lang.Compiler.analyze (Compiler.java:6524)
	clojure.lang.Compiler.analyzeSeq (Compiler.java:6721)
	clojure.lang.Compiler.analyze (Compiler.java:6524)
	clojure.lang.Compiler.analyze (Compiler.java:6485)
	clojure.lang.Compiler$BodyExpr$Parser.parse (Compiler.java:5861)
	clojure.lang.Compiler$FnMethod.parse (Compiler.java:5296)
	clojure.lang.Compiler$FnExpr.parse (Compiler.java:3925)
	clojure.lang.Compiler.analyzeSeq (Compiler.java:6731)
	clojure.lang.Compiler.analyze (Compiler.java:6524)
	clojure.lang.Compiler.analyzeSeq (Compiler.java:6721)
	clojure.lang.Compiler.analyze (Compiler.java:6524)
Caused by:
NullPointerException
	clojure.lang.Compiler.getTypeStringForArgs (Compiler.java:2440)
	clojure.lang.Compiler$InstanceMethodExpr.<init> (Compiler.java:1490)
	clojure.lang.Compiler$HostExpr$Parser.parse (Compiler.java:1000)
	clojure.lang.Compiler.analyzeSeq (Compiler.java:6733)
	clojure.lang.Compiler.analyze (Compiler.java:6524)

Cause: Regression in Clojure 1.6+ from patch for CLJ-1248. In printing the reflection warning message, if the argument's java class is null, an NPE results.

Approach: Check for null before getting class name.

Patch: clj-1681-v3.patch

Screened by: Alex Miller



 Comments   
Comment by Alex Miller [ 24/Mar/15 3:54 PM ]

Patch welcome.

Comment by Alex Miller [ 24/Mar/15 4:31 PM ]

Need test in the patch and having a simple way to repro in the ticket would be great.

Comment by Nicola Mometto [ 25/Mar/15 3:46 PM ]

here's a repro:

Clojure 1.7.0-master-SNAPSHOT
user=> (set! *warn-on-reflection* true)
true
user=> (defn f [a] (.divide 1M a nil))
CompilerException java.lang.NullPointerException, compiling:(NO_SOURCE_PATH:2:13)
Comment by Michael Blume [ 25/Mar/15 3:49 PM ]

Thanks! I was having some trouble with that.

Comment by Nicola Mometto [ 25/Mar/15 3:54 PM ]

Michael, me and Alex were discussing this ticket in IRC and Alex was proposing replacing the if expression with an implementation like:

(arg.hasJavaClass() && arg.getJavaClass() != null) ? arg.getJavaClass().getName() : "unknown"
Comment by Michael Blume [ 25/Mar/15 3:58 PM ]

Hm, that makes sense, I was thinking print "nil" instead of "unknown" but I guess the fact that the user is passing nil doesn't actually tell us the expected class of the argument (apart from that it's not a primitive)





[CLJ-1677] Add setLineNumber to LineNumberingPushbackReader Created: 17/Mar/15  Updated: 20/Mar/15  Resolved: 20/Mar/15

Status: Closed
Project: Clojure
Component/s: None
Affects Version/s: Release 1.6
Fix Version/s: Release 1.7

Type: Enhancement Priority: Major
Reporter: Alex Miller Assignee: Unassigned
Resolution: Completed Votes: 0
Labels: reader

Attachments: Text File clj-1677.patch    
Patch: Code and Test
Approval: Ok

 Description   

Add setLineNumber() to set line number on underlying LineNumberingReader.






[CLJ-1670] Odd error when evaling a quoted object within a macro with a field name containing a '-' Created: 09/Mar/15  Updated: 25/Mar/15  Resolved: 25/Mar/15

Status: Closed
Project: Clojure
Component/s: None
Affects Version/s: Release 1.6
Fix Version/s: None

Type: Defect Priority: Minor
Reporter: Tim Engler Assignee: Unassigned
Resolution: Duplicate Votes: 0
Labels: dash, macro, quote
Environment:

linux gentoo



 Description   

;;Somewhat self explanatory:

(deftype x [ this-is-a-bad-field ])

(defmacro xxx [] `(quote ~(->x 1)))
(xxx)
;;creates error: CompilerException java.lang.IllegalArgumentException: No matching field found: this-is-a-bad-field for class user.x, compiling:(NO_SOURCE_PATH:1:1)

;;But this works:
(deftype y [ ThisIsAGoodField ])

(defmacro yyy [] `(quote ~(->y 1)))
(yyy)

;;returns: #<y user.y@79fd87c8>



 Comments   
Comment by Alex Miller [ 09/Mar/15 12:41 PM ]

possibly a dupe of CLJ-1399

Comment by Tim Engler [ 09/Mar/15 8:21 PM ]

I applied the patch in one of the attachments of CLJ-1399, and this fixed it. So I agree, a dupe.

Comment by Nicola Mometto [ 25/Mar/15 5:22 PM ]

should be closed as dupe of CLJ-1399





[CLJ-1669] Move LazyTransformer to an iterator strategy, extend eduction capabilities Created: 04/Mar/15  Updated: 19/May/15  Resolved: 19/May/15

Status: Closed
Project: Clojure
Component/s: None
Affects Version/s: Release 1.7
Fix Version/s: Release 1.7

Type: Enhancement Priority: Major
Reporter: Alex Miller Assignee: Alex Miller
Resolution: Completed Votes: 0
Labels: transducers

Attachments: Text File clj-1669-2.patch     Text File clj-1669-3.patch     Text File clj-1669-4.patch     Text File clj-1669-5.patch     Text File clj-1669-6.patch     Text File clj-1669.patch    
Patch: Code and Test
Approval: Ok

 Description   
  • LazyTransformer does a lot of work to be a seq. Instead, switch to creating a transforming iterator.
  • Change sequence to wrap iterator-seq around the transforming iterator.
  • Change the iterator-seq implementation to be chunked. IteratorSeq will no longer be used but is left in case of regressions for now.
  • Change Eduction to provide iteration directly via the transforming iterator.
  • Extend eduction to support multiple xforms.

Performance:

Compared:

  • alpha5 == before any change
  • alpha6 == after clj-1669-6.patch was applied
  • beta3 == latest, includes range enhancement, expanding mapcat enhancement, etc

;; using java 1.8
(use 'criterium.core)
(def s (range 1000))
(def v (vec s))
(def s50 (range 50))
(def v50 (vec s50))

expr alpha5 s alpha5 v alpha6 s alpha6 v beta3 s beta3 v
non-chunking transform            
(into [] (->> s (interpose 5) (partition-all 2))) 432 us 437 us 413 us 411 us 353 us 414 us
(into [] (->> s (eduction (interpose 5) (partition-all 2)))) * 117 us 118 us 117 us 113 us 116 us 113 us
1 chunking transform            
(into [] (map inc s)) 43 us 45 us 35 us 31 us 32 us 36 us
(into [] (map inc) s) 19 us 19 us 18 us 18 us 18 us 16 us
(into [] (sequence (map inc) s)) 100 us 54 us 97 us 65 us 66 us 64 us
(into [] (eduction (map inc) s)) 24 us 19 us 24 us 20 us 20 us 19 us
(doall (map inc (eduction (map inc) s))) 219 us 203 us 98 us 78 us 93 us 89 us
2 chunking transforms        
(into [] (map inc (map inc s))) 53 us 56 us 53 us 54 us 61 us 58 us
(into [] (comp (map inc) (map inc)) s) 13 us 26 us 30 us 26 us 31 us 31 us
(into [] (sequence (comp (map inc) (map inc)) s)) 111 us 64 us 98 us 73 us 83 us 80 us
(into [] (eduction (map inc) (map inc) s)) * 58 us 31 us 58 us 31 us 30 us 31 us
(doall (map inc (eduction (map inc) (map inc) s))) * 240 us 212 us 114 us 93 us 105 us 102 us
expand transform            
(into [] (mapcat range (map inc s50))) 74 us 73 us 67 us 68 us 37 us 39 us
(into [] (sequence (comp (map inc) (mapcat range)) s50)) 111 us 102 us 166 us 159 us 99 us 98 us
(into [] (eduction (map inc) (mapcat range) s50)) * 65 us 64 us 57 us 56 us 27 us 27 us
materialized eduction            
(sort (eduction (map inc) s)) ERR ERR 99 us 77 us 77 us 77 us
(->> s (filter odd?) (map str) (sort-by last)) 1.10 ms 1.25 ms 1.15 ms 1.19 ms 1.14 ms 1.13 ms
(->> s (eduction (filter odd?) (map str)) (sort-by last)) ERR ERR 1.18 ms 1.15 ms 1.13 ms 1.13 ms
  • used comp to combine xforms as eduction only took one in the before case

Patch: clj-1669-6.patch

Screened by:



 Comments   
Comment by Michael Blume [ 05/Mar/15 3:52 PM ]

Nice, I like the direction on this.

CLJ-1515 currently breaks this patch (LongRange cannot be converted to Iterable), but I imagine that'll get better when it absorbs the changes from CLJ-1603

Comment by Alex Miller [ 06/Mar/15 8:11 AM ]

Yeah. colls should be mapped through RT.iter() to catch more cases.

Comment by Alex Miller [ 06/Mar/15 9:52 AM ]

To do:

  • remove Seqable from Eduction
  • support Iterable in RT.toArray()
  • more eduction pipeline tests that require realization at end
Comment by Alex Miller [ 06/Mar/15 1:00 PM ]

Perf numbers show pretty worse results from sequence, will dig in further.

Comment by Alex Miller [ 13/Mar/15 7:41 AM ]

For the s timings, we've actually introduced more steps into the stack:

OLD reduce with s:

LazyTransformer
   seq (range) - every transformation is another layer here

NEW reduce with s:

IteratorSeq 
  TransformingIterator (handles N transformations in 1 step)
    SeqIterator
      seq (range)
Comment by Alex Miller [ 20/Mar/15 10:08 AM ]

Look at perf for:

  • ->> eduction transformation
  • transformation comparison that doesn't support chunking
  • more into vector iteration case
Comment by Alex Miller [ 21/Mar/15 8:45 AM ]

The -5 patch is same -3 except all uses of IteratorSeq have been replaced with a ChunkedCons that is effectively a chunked version of the old IteratorSeq. While no one calls it, I left IteratorSeq in the code base in case of regression.

Generally, the chunked iterator seq reduces the cost in a number of the worst cases and also is a clear benefit in making seqs over a result of eduction or sequence faster to traverse (as they are now chunked).

I think the one potential issue is that seqs over iterators are now chunked when they were not before which could change programs that expect their stateful iterator to be traversed one at a time. This change could be isolated to just to sequence and seq-iterator and mitigated by not changing RT.seqFrom() and seq-iterator to use the new chunking behavior only in sequence and/or with a new chunked-iterator-seq to make it more explicit. The sequence over xf is new so no possible regression there, everything else would just be opt-in.

Comment by Rich Hickey [ 27/Mar/15 9:49 AM ]

push as is but leave unresolved, for perf tweaks

Comment by Alex Miller [ 27/Mar/15 10:15 AM ]

clj-1669-6 is identical to clj-1669-5 but removes two commented out debugging lines that were inadvertently included.

Comment by Alex Miller [ 18/May/15 9:47 AM ]

updated for beta3 numbers





[CLJ-1667] Socket test can fail if hard-coded port is unavailable Created: 26/Feb/15  Updated: 27/Mar/15  Resolved: 27/Mar/15

Status: Closed
Project: Clojure
Component/s: None
Affects Version/s: Release 1.7
Fix Version/s: Release 1.7

Type: Defect Priority: Minor
Reporter: Alex Miller Assignee: Stuart Halloway
Resolution: Completed Votes: 0
Labels: io, test

Attachments: Text File socket-test.patch    
Patch: Code
Approval: Ok

 Description   

I was unable to run the Clojure tests due to this problem. There is a test that hardcodes a port and something else on my machine happened to be using that port.

The patch avoids binding a hard-coded port in the test.

Patch: socket-test.patch

Screened by:



 Comments   
Comment by Andy Fingerhut [ 26/Feb/15 11:31 AM ]

I used to try running the prescreen tests in parallel for two different JDKs on the same machine, and I probably stopped doing that because of this. My use case is a very unusual one, and not a good reason to change this by itself, but my use case certainly made this conflict happen regularly.

Comment by Alex Miller [ 26/Feb/15 11:57 AM ]

No good reason not to fix it! Silly test.





[CLJ-1666] change 'fun' to 'f' in doc strings Created: 22/Feb/15  Updated: 22/Feb/15  Resolved: 22/Feb/15

Status: Closed
Project: Clojure
Component/s: None
Affects Version/s: None
Fix Version/s: None

Type: Enhancement Priority: Trivial
Reporter: Patrick Ryan Assignee: Unassigned
Resolution: Declined Votes: 0
Labels: docstring, enhancement

Attachments: File fix-fun-to-f.diff    
Patch: Code

 Description   

fix 'function' term references in core/alter, core/commute, and test/clojure/test_clojure/pprint/test_cl_format.clj for consistency.

Rest of codebase uses 'f' to refer to functions.



 Comments   
Comment by Patrick Ryan [ 22/Feb/15 12:17 PM ]

My first patch, any help/critique welcomed.

Comment by Andy Fingerhut [ 22/Feb/15 1:10 PM ]

It looks like the format of the patch is the one expected, so that is good. Not a big deal, but most people use their actual name and email address to identify themselves, rather than an alias and email address.

I don't know whether this patch is of interest to the Clojure developers or not, but I do know that they will never apply patches written by those who have not signed a Clojure contributor agreement – see http://clojure.org/contributing

I did not see your name on the list there. Were you considering signing the CA?

Comment by Patrick Ryan [ 22/Feb/15 1:19 PM ]

I just signed it about an hour or two ago. Patrick Ryan (phiat99@gmail.com) (phiat on github) Thanks for feedback

Comment by Alex Miller [ 22/Feb/15 1:23 PM ]

Hi Patrick, thanks for navigating the process and submitting the patch! Unfortunately, I don't think this particular change is worth doing so I am going to decline it. Sorry about that and I hope I have not discouraged you on your road to using or contributing to Clojure!

Comment by Patrick Ryan [ 22/Feb/15 1:33 PM ]

No problem Alex! I know its a tiny trivial change, just a small OCD thing when I was looking through docs! Thanks for feedback. I will look for 'bigger fish' to fry





[CLJ-1663] DynamicClassLoader delegates to parent classloader before checking in its URL list Created: 18/Feb/15  Updated: 20/Feb/15  Resolved: 20/Feb/15

Status: Closed
Project: Clojure
Component/s: None
Affects Version/s: Release 1.7
Fix Version/s: Release 1.7

Type: Defect Priority: Critical
Reporter: Nicola Mometto Assignee: Unassigned
Resolution: Completed Votes: 1
Labels: classloader, regression

Attachments: Text File 0001-CLJ-1663-delegate-loadClass-to-super-classloader-bef.patch    
Patch: Code
Approval: Ok

 Description   

See Cursive #748. Cursive calls into Leiningen in-process, and before doing that it creates a new DynamicClassLoader which uses an IntelliJ PluginClassLoader as its parent. This is throwing a CNFE, although the URL containing the class is present in the DynamicClassLoader URL list.

Cause: The patch for CLJ-979 added an implementation of loadClass that delegates to "getParent().loadClass()", which in this case delegates to PluginClassLoader.loadClass(). This was incorrect as the current implementation should have been to delegate to the loadClass method of the superclass, which will take care of delegating to the loadClass method of the parent class loader if necessary.

Approach: The proposed patch replaces the call to "getParent().loadClass()" with a call "super.loadClass()" fixing this issue.

Screened by: Alex Miller



 Comments   
Comment by Colin Fleming [ 18/Feb/15 6:25 AM ]

Unfortunately getClassLoadingLock(name) is only available from Java 1.7+ and I am targeting Java 1.6. However reverting that part of the patch to synchronize on "this" as previously does indeed fix the original problem.

Comment by Nicola Mometto [ 18/Feb/15 7:22 AM ]

I'll revert the getClassLoadingLock change then, it was actually out of scope for this ticket.





[CLJ-1660] Unify Stepper and MultiStepper in LazyTransformer Created: 16/Feb/15  Updated: 04/Mar/15  Resolved: 04/Mar/15

Status: Closed
Project: Clojure
Component/s: None
Affects Version/s: None
Fix Version/s: None

Type: Enhancement Priority: Minor
Reporter: Michael Blume Assignee: Unassigned
Resolution: Declined Votes: 0
Labels: None

Attachments: Text File CLJ-1660-v1.patch    
Patch: Code

 Description   

This seemed worthwhile to me mainly because some of the stepper logic is actually pretty fiddly, so it seems better not to have it duplicated.



 Comments   
Comment by Alex Miller [ 04/Mar/15 1:06 PM ]

Based on CLJ-1669 direction which eliminates this code.





[CLJ-1658] Redefine doseq in terms of reduce Created: 13/Feb/15  Updated: 13/Feb/15  Resolved: 13/Feb/15

Status: Closed
Project: Clojure
Component/s: None
Affects Version/s: Release 1.7
Fix Version/s: None

Type: Enhancement Priority: Minor
Reporter: Timothy Baldridge Assignee: Unassigned
Resolution: Duplicate Votes: 0
Labels: None

Attachments: File doseq-with-reduce.diff    
Patch: Code

 Description   

The current version of doseq calls seq on each input collection in order to obtain a unified interface. If however, the input collection is not a seq, this will cause a fair amount of unnecessary allocation. By modifying doseq to use reduce internally we can not only reduce the amount of code doseq produces but the resulting code is also much faster due to reduction being handed off to the collection itself. The net effect is about a 3x performance boost for vectors, with no impact on code for seqs. 

Approach: we re-define doseq in core.clj after reduce has been defined.

Benchmarks:

Before:

user=> (def v (vec (range (* 1024 1024))))
#'user/v
user=> (dotimes [x 100] (time (doseq [x v] x)))

average: 6ms

user=>  (def s (doall (range (* 1024 1024))))
#'user/s
user=> (dotimes [x 100] (time (doseq [x (seq v)] x)))
average: 2ms

After:

user=> (def v (vec (range (* 1024 1024))))
#'user/v
user=> (dotimes [x 100] (time (doseq [x v] x)))

average: 1.5ms

user=>  (def s (doall (range (* 1024 1024))))
#'user/s
user=> (dotimes [x 100] (time (doseq [x s] x)))
average: 2ms

Notes: All existing doseq tests pass, so no new tests were added.



 Comments   
Comment by Alex Miller [ 13/Feb/15 10:25 AM ]

Summarize benchmark tests and results in description?

Comment by Ghadi Shayban [ 13/Feb/15 10:32 AM ]

dupe of CLJ-1322

Comment by Timothy Baldridge [ 13/Feb/15 10:37 AM ]

well, shoot.





[CLJ-1652] clojure.test counter reports are not thread safe Created: 30/Jan/15  Updated: 30/Jan/15  Resolved: 30/Jan/15

Status: Closed
Project: Clojure
Component/s: None
Affects Version/s: Release 1.6
Fix Version/s: None

Type: Defect Priority: Major
Reporter: David Sargeant Assignee: Unassigned
Resolution: Duplicate Votes: 0
Labels: None

Attachments: File test-thread-safety.diff    
Patch: Code

 Description   

The update to the

*report-counters*
ref in the inc-report-counter function is not atomic and does not produce the expected number of assertions when tests are run on multiple threads.

Failure example:

(use 'clojure.test)

(deftest test-counter-parallel
  (doall (pmap (fn [x] (is true)) (range 1000))))
  
(run-tests)

Ran 1 tests containing 183 assertions.
0 failures, 0 errors.
{:type :summary, :fail 0, :error 0, :pass 183, :test 1}

Same example, but single-threaded:

(use 'clojure.test)

(deftest test-counter
  (doall (map (fn [x] (is true)) (range 1000))))
  
(run-tests)

Ran 1 tests containing 1000 assertions.
0 failures, 0 errors.
{:type :summary, :fail 0, :error 0, :pass 1000, :test 1}

The attached patch fixes the issue.



 Comments   
Comment by Alex Miller [ 30/Jan/15 12:40 PM ]

Dupe of CLJ-1528 I think.





[CLJ-1650] compile forces namespace reloading from AOT classfile Created: 29/Jan/15  Updated: 18/Jun/15  Resolved: 18/Jun/15

Status: Closed
Project: Clojure
Component/s: None
Affects Version/s: Release 1.7
Fix Version/s: Release 1.8

Type: Defect Priority: Major
Reporter: Nicola Mometto Assignee: Unassigned
Resolution: Duplicate Votes: 1
Labels: regression

Attachments: Text File 0001-CLJ-1650-compile-forces-namespace-reloading-from-AOT.patch    
Patch: Code
Approval: Vetted

 Description   

The patch for CLJ-979 exposed an issue with how clojure.core/compile is implemented, which causes the bug reported here: https://groups.google.com/d/msg/clojure-dev/jj87-4yVlWI/YKG4QazhPuAJ

The cause of this regression is that clojure.core/compile doesn't take into account clojure.core/loaded-libs, causing

(binding [*compile-files* true] (require 'some.ns))
(compile 'some.ns)

to reload 'some-ns from the AOT class with the call to compile.

Since the AOT compiled namespace is not loaded by DynamicClassLoader but using the underlying java.net.URLClassLoader, code that relies on deftypes will have references to the AOT versions hardcoded, breaking the class loading policy introduced with CLJ-979 of prefering the in-memory versions to the AOT ones.

The fix for this, as implemented in the attached patch, is to make compile loaded-libs-aware, so that it won't force any namespace re-loading when unnecessary.



 Comments   
Comment by Alex Miller [ 18/Jun/15 11:24 AM ]

Dupe of CLJ-1741





[CLJ-1648] Use equals() instead of == when resolving Symbol Created: 22/Jan/15  Updated: 12/May/15  Resolved: 12/May/15

Status: Closed
Project: Clojure
Component/s: None
Affects Version/s: Release 1.7
Fix Version/s: Release 1.7

Type: Defect Priority: Critical
Reporter: Steven Yi Assignee: Unassigned
Resolution: Completed Votes: 0
Labels: Compiler

Attachments: File resolve-symbol-equals.diff    
Patch: Code
Approval: Ok

 Description   

In Compiler.java, resolveSymbol() uses == to compare a Symbol's ns and the found namespace's name. This can result in a false comparison result, though the name's may be equal. In the following example:

ond.core=> (require '[clojure.string])
nil
ond.core=> `(clojure.string/join "," [1 2])
false : true ;; reported from System.out.println code I put into Compiler.java for == vs .equals()
nil

The result is that a new Symbol is allocated, when the previous one should be returned.

Prior to Clojure 1.7, Symbol name and ns were interned so == would actually have worked, but that is no longer the case.

Patch: resolve-symbol-equals.diff

Screened by: Alex Miller

[1] - https://groups.google.com/forum/#!topic/clojure-dev/58fYUSIEfxg






[CLJ-1646] Small filter performance enhancement Created: 19/Jan/15  Updated: 11/Apr/15  Resolved: 11/Apr/15

Status: Closed
Project: Clojure
Component/s: None
Affects Version/s: Release 1.6
Fix Version/s: None

Type: Enhancement Priority: Critical
Reporter: Alex Miller Assignee: Unassigned
Resolution: Duplicate Votes: 1
Labels: performance

Attachments: Text File clj-1646.patch    
Patch: Code
Approval: Triaged

 Description   

I found when working on other tickets for 1.7 that filter repeats each call to .nth on the chunk.
Reusing the result is a small perf boost for all filter uses.

Timing with criterium quick-bench:

What stock patch applied comment
(into [] (filter odd? (range 1024))) 54.3 µs 50.2 µs 7.5% reduction

Approach: storing the nth value as to avoid double lookup.

Patch: clj-1646.patch
Screened by:



 Comments   
Comment by Michael Blume [ 25/Mar/15 5:09 PM ]

This seems have been rolled into the patch for CLJ-1515 – should it be closed?

Comment by Alex Miller [ 25/Mar/15 5:13 PM ]

I'm not sure yet which patch will be applied for 1515. I will close this if it's included.

Comment by Alex Miller [ 11/Apr/15 6:45 AM ]

included in CLJ-1515 patch





[CLJ-1642] Add mention of new :warn-on-boxed option to doc string of unchecked-math Var Created: 15/Jan/15  Updated: 20/Feb/15  Resolved: 20/Feb/15

Status: Closed
Project: Clojure
Component/s: None
Affects Version/s: Release 1.7
Fix Version/s: Release 1.7

Type: Enhancement Priority: Minor
Reporter: Andy Fingerhut Assignee: Unassigned
Resolution: Completed Votes: 0
Labels: docstring

Attachments: Text File clj-1642-v1.patch    
Patch: Code
Approval: Ok

 Description   

A small doc string enhancement about the new compiler behavior in Clojure 1.7 when *unchecked-math* is bound to :warn-on-boxed

https://github.com/clojure/clojure/blob/master/changes.md#13-warn-on-boxed-math

Patch: clj-1642-v1.patch
Screened by: Alex Miller



 Comments   
Comment by Andy Fingerhut [ 15/Jan/15 11:51 AM ]

Patch clj-1642-v1.patch dated Jan 15 2014 is one way to document the new :warn-on-boxed behavior.





[CLJ-1641] AOT compilation fails for a kind of circular dependency after CLJ-1544 patch Created: 14/Jan/15  Updated: 16/Jan/15  Resolved: 16/Jan/15

Status: Closed
Project: Clojure
Component/s: None
Affects Version/s: Release 1.7
Fix Version/s: Release 1.7

Type: Defect Priority: Major
Reporter: Andy Fingerhut Assignee: Unassigned
Resolution: Declined Votes: 0
Labels: regression

Attachments: Text File 0001-CLJ-1641-disallow-circular-dependencies-even-if-the-.patch    

 Description   

I am assuming the summary and description will be updated from this initial version. This may not be a bug, but simply the way things are after CLJ-1544 is fixed, and AOT compilation of some kinds of cyclic namespace dependencies not throwing an exception in earlier versions of Clojure was an accident.

Behavior before CLJ-1544 patch applied:

% cat project.clj 
(defproject manifold-cycle "0.1.0-SNAPSHOT"
  :dependencies [[org.clojure/clojure "1.7.0-alpha4"]
                 [manifold "0.1.0-alpha4"]]
  :profiles {:uberjar {:aot :all}})

% cat src/manifold_cycle/core.clj 
(ns manifold-cycle.core
  (:require [manifold.stream :as s]))

% lein version
Leiningen 2.5.0 on Java 1.7.0_45 Java HotSpot(TM) 64-Bit Server VM
% lein clean
% lein uberjar
Compiling manifold-cycle.core
Created /Users/jafinger/clj/glosscycle/target/manifold-cycle-0.1.0-SNAPSHOT.jar
Created /Users/jafinger/clj/glosscycle/target/manifold-cycle-0.1.0-SNAPSHOT-standalone.jar

Behavior after CLJ-1544 patch applied, e.g. by changing Clojure version to 1.7.0-alpha5:

% lein clean
% lein uberjar
Compiling manifold-cycle.core
java.lang.Exception: Cyclic load dependency: [ /manifold/stream ]->/manifold/stream/graph->[ /manifold/stream ]->/manifold_cycle/core, compiling:(manifold/stream/graph.clj:1:1)
	at clojure.core$throw_if.doInvoke(core.clj:5612)
	at clojure.lang.RestFn.invoke(RestFn.java:442)
	at clojure.core$check_cyclic_dependency.invoke(core.clj:5763)
	at clojure.core$load.doInvoke(core.clj:5860)
	at clojure.lang.RestFn.invoke(RestFn.java:408)
	at clojure.core$load_one.invoke(core.clj:5653)
	at clojure.core$load_lib$fn__5383.invoke(core.clj:5708)
	at clojure.core$load_lib.doInvoke(core.clj:5707)
	at clojure.lang.RestFn.applyTo(RestFn.java:142)
	at clojure.core$apply.invoke(core.clj:628)
	at clojure.core$load_libs.doInvoke(core.clj:5746)
	at clojure.lang.RestFn.applyTo(RestFn.java:137)
	at clojure.core$apply.invoke(core.clj:628)
	at clojure.core$require.doInvoke(core.clj:5829)
	at clojure.lang.RestFn.invoke(RestFn.java:457)
	at manifold.stream.graph$loading__5322__auto____856.invoke(graph.clj:1)
	at clojure.lang.AFn.applyToHelper(AFn.java:152)
	at clojure.lang.AFn.applyTo(AFn.java:144)
	at clojure.lang.Compiler$InvokeExpr.eval(Compiler.java:3600)
	at clojure.lang.Compiler.compile1(Compiler.java:7290)
	at clojure.lang.Compiler.compile1(Compiler.java:7280)
	at clojure.lang.Compiler.compile(Compiler.java:7356)
	at clojure.lang.RT.compile(RT.java:398)
	at clojure.lang.RT.load(RT.java:438)
	at clojure.lang.RT.load(RT.java:411)
	at clojure.core$load$fn__5436.invoke(core.clj:5863)
	at clojure.core$load.doInvoke(core.clj:5862)
	at clojure.lang.RestFn.invoke(RestFn.java:408)
	at clojure.core$load_one.invoke(core.clj:5653)
	at clojure.core$load_lib$fn__5383.invoke(core.clj:5708)
	at clojure.core$load_lib.doInvoke(core.clj:5707)
	at clojure.lang.RestFn.applyTo(RestFn.java:142)
	at clojure.core$apply.invoke(core.clj:628)
	at clojure.core$load_libs.doInvoke(core.clj:5746)
	at clojure.lang.RestFn.applyTo(RestFn.java:137)
	at clojure.core$apply.invoke(core.clj:628)
	at clojure.core$require.doInvoke(core.clj:5829)
	at clojure.lang.RestFn.applyTo(RestFn.java:137)
	at clojure.lang.Compiler$InvokeExpr.eval(Compiler.java:3600)
	at clojure.lang.Compiler.compile1(Compiler.java:7290)
	at clojure.lang.Compiler.compile(Compiler.java:7356)
	at clojure.lang.RT.compile(RT.java:398)
	at clojure.lang.RT.load(RT.java:438)
	at clojure.lang.RT.load(RT.java:411)
	at clojure.core$load$fn__5436.invoke(core.clj:5863)
	at clojure.core$load.doInvoke(core.clj:5862)
	at clojure.lang.RestFn.invoke(RestFn.java:408)
	at clojure.core$load_one.invoke(core.clj:5653)
	at clojure.core$load_lib$fn__5383.invoke(core.clj:5708)
	at clojure.core$load_lib.doInvoke(core.clj:5707)
	at clojure.lang.RestFn.applyTo(RestFn.java:142)
	at clojure.core$apply.invoke(core.clj:628)
	at clojure.core$load_libs.doInvoke(core.clj:5746)
	at clojure.lang.RestFn.applyTo(RestFn.java:137)
	at clojure.core$apply.invoke(core.clj:628)
	at clojure.core$require.doInvoke(core.clj:5829)
	at clojure.lang.RestFn.invoke(RestFn.java:408)
	at manifold_cycle.core$loading__5322__auto____21.invoke(core.clj:1)
	at clojure.lang.AFn.applyToHelper(AFn.java:152)
	at clojure.lang.AFn.applyTo(AFn.java:144)
	at clojure.lang.Compiler$InvokeExpr.eval(Compiler.java:3600)
	at clojure.lang.Compiler.compile1(Compiler.java:7290)
	at clojure.lang.Compiler.compile1(Compiler.java:7280)
	at clojure.lang.Compiler.compile(Compiler.java:7356)
	at clojure.lang.RT.compile(RT.java:398)
	at clojure.lang.RT.load(RT.java:438)
	at clojure.lang.RT.load(RT.java:411)
	at clojure.core$load$fn__5436.invoke(core.clj:5863)
	at clojure.core$load.doInvoke(core.clj:5862)
	at clojure.lang.RestFn.invoke(RestFn.java:408)
	at clojure.core$load_one.invoke(core.clj:5653)
	at clojure.core$compile$fn__5441.invoke(core.clj:5874)
	at clojure.core$compile.invoke(core.clj:5873)
	at user$eval9$fn__16.invoke(form-init6042653661846269464.clj:1)
	at user$eval9.invoke(form-init6042653661846269464.clj:1)
	at clojure.lang.Compiler.eval(Compiler.java:6767)
	at clojure.lang.Compiler.eval(Compiler.java:6757)
	at clojure.lang.Compiler.load(Compiler.java:7194)
	at clojure.lang.Compiler.loadFile(Compiler.java:7150)
	at clojure.main$load_script.invoke(main.clj:274)
	at clojure.main$init_opt.invoke(main.clj:279)
	at clojure.main$initialize.invoke(main.clj:307)
	at clojure.main$null_opt.invoke(main.clj:342)
	at clojure.main$main.doInvoke(main.clj:420)
	at clojure.lang.RestFn.invoke(RestFn.java:421)
	at clojure.lang.Var.invoke(Var.java:383)
	at clojure.lang.AFn.applyToHelper(AFn.java:156)
	at clojure.lang.Var.applyTo(Var.java:700)
	at clojure.main.main(main.java:37)
Caused by: java.lang.Exception: Cyclic load dependency: [ /manifold/stream ]->/manifold/stream/graph->[ /manifold/stream ]->/manifold_cycle/core
	... 87 more
Exception in thread "main" java.lang.Exception: Cyclic load dependency: [ /manifold/stream ]->/manifold/stream/graph->[ /manifold/stream ]->/manifold_cycle/core, compiling:(manifold/stream/graph.clj:1:1)
	at clojure.core$throw_if.doInvoke(core.clj:5612)
	at clojure.lang.RestFn.invoke(RestFn.java:442)
	at clojure.core$check_cyclic_dependency.invoke(core.clj:5763)
	at clojure.core$load.doInvoke(core.clj:5860)
	at clojure.lang.RestFn.invoke(RestFn.java:408)
	at clojure.core$load_one.invoke(core.clj:5653)
	at clojure.core$load_lib$fn__5383.invoke(core.clj:5708)
	at clojure.core$load_lib.doInvoke(core.clj:5707)
	at clojure.lang.RestFn.applyTo(RestFn.java:142)
	at clojure.core$apply.invoke(core.clj:628)
	at clojure.core$load_libs.doInvoke(core.clj:5746)
	at clojure.lang.RestFn.applyTo(RestFn.java:137)
	at clojure.core$apply.invoke(core.clj:628)
	at clojure.core$require.doInvoke(core.clj:5829)
	at clojure.lang.RestFn.invoke(RestFn.java:457)
	at manifold.stream.graph$loading__5322__auto____856.invoke(graph.clj:1)
	at clojure.lang.AFn.applyToHelper(AFn.java:152)
	at clojure.lang.AFn.applyTo(AFn.java:144)
	at clojure.lang.Compiler$InvokeExpr.eval(Compiler.java:3600)
	at clojure.lang.Compiler.compile1(Compiler.java:7290)
	at clojure.lang.Compiler.compile1(Compiler.java:7280)
	at clojure.lang.Compiler.compile(Compiler.java:7356)
	at clojure.lang.RT.compile(RT.java:398)
	at clojure.lang.RT.load(RT.java:438)
	at clojure.lang.RT.load(RT.java:411)
	at clojure.core$load$fn__5436.invoke(core.clj:5863)
	at clojure.core$load.doInvoke(core.clj:5862)
	at clojure.lang.RestFn.invoke(RestFn.java:408)
	at clojure.core$load_one.invoke(core.clj:5653)
	at clojure.core$load_lib$fn__5383.invoke(core.clj:5708)
	at clojure.core$load_lib.doInvoke(core.clj:5707)
	at clojure.lang.RestFn.applyTo(RestFn.java:142)
	at clojure.core$apply.invoke(core.clj:628)
	at clojure.core$load_libs.doInvoke(core.clj:5746)
	at clojure.lang.RestFn.applyTo(RestFn.java:137)
	at clojure.core$apply.invoke(core.clj:628)
	at clojure.core$require.doInvoke(core.clj:5829)
	at clojure.lang.RestFn.applyTo(RestFn.java:137)
	at clojure.lang.Compiler$InvokeExpr.eval(Compiler.java:3600)
	at clojure.lang.Compiler.compile1(Compiler.java:7290)
	at clojure.lang.Compiler.compile(Compiler.java:7356)
	at clojure.lang.RT.compile(RT.java:398)
	at clojure.lang.RT.load(RT.java:438)
	at clojure.lang.RT.load(RT.java:411)
	at clojure.core$load$fn__5436.invoke(core.clj:5863)
	at clojure.core$load.doInvoke(core.clj:5862)
	at clojure.lang.RestFn.invoke(RestFn.java:408)
	at clojure.core$load_one.invoke(core.clj:5653)
	at clojure.core$load_lib$fn__5383.invoke(core.clj:5708)
	at clojure.core$load_lib.doInvoke(core.clj:5707)
	at clojure.lang.RestFn.applyTo(RestFn.java:142)
	at clojure.core$apply.invoke(core.clj:628)
	at clojure.core$load_libs.doInvoke(core.clj:5746)
	at clojure.lang.RestFn.applyTo(RestFn.java:137)
	at clojure.core$apply.invoke(core.clj:628)
	at clojure.core$require.doInvoke(core.clj:5829)
	at clojure.lang.RestFn.invoke(RestFn.java:408)
	at manifold_cycle.core$loading__5322__auto____21.invoke(core.clj:1)
	at clojure.lang.AFn.applyToHelper(AFn.java:152)
	at clojure.lang.AFn.applyTo(AFn.java:144)
	at clojure.lang.Compiler$InvokeExpr.eval(Compiler.java:3600)
	at clojure.lang.Compiler.compile1(Compiler.java:7290)
	at clojure.lang.Compiler.compile1(Compiler.java:7280)
	at clojure.lang.Compiler.compile(Compiler.java:7356)
	at clojure.lang.RT.compile(RT.java:398)
	at clojure.lang.RT.load(RT.java:438)
	at clojure.lang.RT.load(RT.java:411)
	at clojure.core$load$fn__5436.invoke(core.clj:5863)
	at clojure.core$load.doInvoke(core.clj:5862)
	at clojure.lang.RestFn.invoke(RestFn.java:408)
	at clojure.core$load_one.invoke(core.clj:5653)
	at clojure.core$compile$fn__5441.invoke(core.clj:5874)
	at clojure.core$compile.invoke(core.clj:5873)
	at user$eval9$fn__16.invoke(form-init6042653661846269464.clj:1)
	at user$eval9.invoke(form-init6042653661846269464.clj:1)
	at clojure.lang.Compiler.eval(Compiler.java:6767)
	at clojure.lang.Compiler.eval(Compiler.java:6757)
	at clojure.lang.Compiler.load(Compiler.java:7194)
	at clojure.lang.Compiler.loadFile(Compiler.java:7150)
	at clojure.main$load_script.invoke(main.clj:274)
	at clojure.main$init_opt.invoke(main.clj:279)
	at clojure.main$initialize.invoke(main.clj:307)
	at clojure.main$null_opt.invoke(main.clj:342)
	at clojure.main$main.doInvoke(main.clj:420)
	at clojure.lang.RestFn.invoke(RestFn.java:421)
	at clojure.lang.Var.invoke(Var.java:383)
	at clojure.lang.AFn.applyToHelper(AFn.java:156)
	at clojure.lang.Var.applyTo(Var.java:700)
	at clojure.main.main(main.java:37)
Caused by: java.lang.Exception: Cyclic load dependency: [ /manifold/stream ]->/manifold/stream/graph->[ /manifold/stream ]->/manifold_cycle/core
	... 87 more
Compilation failed: Subprocess failed

It is true that Manifold has a kind of cyclic dependency in its implementation. Namespace manifold.stream's ns form has no require of namespace manifold.stream.graph, but there is an explicit require of the namespace after its ns form here: https://github.com/ztellman/manifold/blob/master/src/manifold/stream.clj#L410

Namespace manifold.stream.graph requires manifold.stream in its ns form: https://github.com/ztellman/manifold/blob/master/src/manifold/stream/graph.clj#L5

Reported by Janne Lemmetti in the Clojure Google group thread announcing Clojure 1.7.0-alpha5's release on Jan 11 2015. He discovered the problem while trying to AOT compile a project that uses the gloss library, which depends upon byte-streams, which depends upon manifold.



 Comments   
Comment by Zach Tellman [ 14/Jan/15 3:46 PM ]

Speaking as the author of the circular dependency, I think what I'm doing here should be allowed. I want to surface vars A and C in a namespace, but C relies on B in another namespace, and B relies on A in my original namespace. Therefore, I was able to do this:

(ns b-ns
(:require [a-ns]))

(def b ...)

(ns a-ns)

(def A ...)

(require 'b-ns)

(def C ...)

The assumption here is that the parts of 'a-ns' that 'b-ns' relies on have been defined once 'b-ns' is required and refers back to 'a-ns'. I was happy to find that this worked, as it meant I didn't have to use my previous approach in these situations, which was to factor out the first half of 'a-ns' into a separate namespace, and then import those vars into 'a-ns' (no one has ever liked import-vars).

If the pre-alpha5 behavior I was relying on was too accidental for everyone's taste, I can get rid of it, but I assert that having some official way to "break" reference loops would be very useful.

Comment by Andy Fingerhut [ 14/Jan/15 5:00 PM ]

Zach, I don't know if it makes any difference to you, but I believe that the CLJ-1544 patch did not break what you are doing in Manifold all of the time, only when AOT compilation is used. That may be significant enough of a use case that your statements still stand.

Comment by Alex Miller [ 14/Jan/15 5:44 PM ]

Pulling this into the 1.7 list just so I'm seeing it, not necessarily implying anything re end result as I haven't looked at it yet.

Comment by Andy Fingerhut [ 15/Jan/15 11:42 AM ]

Added to comments as more background, not necessarily suggesting whether this behavior change is a bug or not: Brief email thread from Oct 2014 in Clojure group between Colin Fleming and Steven (Gilardi?) on why Clojure did not warn about any cyclic dependencies in the Manifold library before: https://groups.google.com/forum/#!topic/clojure/wrVFuCjf0_Y/discussion

Comment by Zach Tellman [ 15/Jan/15 1:41 PM ]

I took another look at my code, and found that the design has changed enough that I no longer "need" circular dependencies, and I could just factor out the shared code to a third namespace. This doesn't need to hold up the 1.7.0 release, I guess, but speaking as someone who uses AOT compilation everywhere, divergent behavior between AOT and standard compilation is worrisome.

Comment by Alex Miller [ 15/Jan/15 2:27 PM ]

One way I can read this is as a sign of the tension between "form as unit of compilation" and "file as code container" where these can be separated in normal source loading but are tangled in AOT.

Comment by Nicola Mometto [ 15/Jan/15 3:52 PM ]

Zach, I agree that having different behaviour between AOT and JIT is wrong.

But I also don't agree that having clojure error out on circular dependencies should be considered a bug, I would argue that the way manifold used to implement the circular dependency between manifold.stream and manifold.stream.graph was a just a hack around lack of validation in require.

My proposal to fix this disparity between AOT and JIT is by making require/use check for circular dependencies before checking for already-loaded namespaces.

This way, both under JIT and AOT code like

(ns foo.a (:require foo.b))
(ns foo.b)
(require 'foo.a)

will fail with a circular depdenency error.

This is what the patch I just attached (0001-CLJ-1641-disallow-circular-dependencies-even-if-the-.patch) does.\

Comment by Alex Miller [ 16/Jan/15 12:59 PM ]

CLJ-1544 is being rolled back in alpha6. I'm not exactly sure what to call the status on this one but we do not plan to take further action on it right now. Any future change for CLJ-1544 will consider this case.





[CLJ-1640] Negating Boolean false is false Created: 13/Jan/15  Updated: 13/Jan/15  Resolved: 13/Jan/15

Status: Closed
Project: Clojure
Component/s: None
Affects Version/s: Release 1.4, Release 1.5, Release 1.6
Fix Version/s: None

Type: Defect Priority: Major
Reporter: Kuldeep Assignee: Unassigned
Resolution: Completed Votes: 0
Labels: bug
Environment:

Ubuntu 14.04



 Description   

% java -version
java version "1.7.0_71"
Java(TM) SE Runtime Environment (build 1.7.0_71-b14)
Java HotSpot(TM) 64-Bit Server VM (build 24.71-b01, mixed mode)

user=> (not (Boolean. "false"))
false
user=> (not (Boolean. false))
false
user=> (not (Boolean. true))
false
user=> (not (Boolean. "true"))
false
user=> (not (Boolean/valueOf "false"))
true



 Comments   
Comment by Kuldeep [ 13/Jan/15 3:55 AM ]

http://clojure.org/special_forms#Special Forms--(if test then else?)





[CLJ-1639] Loading both AOT and non-AOT versions of a namespace causes errors Created: 12/Jan/15  Updated: 29/Jan/15  Resolved: 16/Jan/15

Status: Closed
Project: Clojure
Component/s: None
Affects Version/s: Release 1.7
Fix Version/s: Release 1.7

Type: Defect Priority: Critical
Reporter: Sean Corfield Assignee: Unassigned
Resolution: Declined Votes: 0
Labels: regression
Environment:

Clojure 1.7.0-alpha5



 Description   

Change in behavior here due to CLJ-979 added in 1.7.0-alpha5.

Symptom is that code fails to compile when a namespace is loading, claiming a protocol has no implementation for a method (that it does have).

Stack trace from Sean Corfield's code:

Caused by: java.lang.IllegalArgumentException: No implementation of method: :has? of protocol: #'clojure.core.cache/CacheProtocol found for class: clojure.core.memoize.PluggableMemoization 
        at clojure.core$_cache_protocol_fn.invoke(core_deftype.clj:555) 
        at clojure.core.cache$eval1710$fn__1773$G__1699__1780.invoke(cache.clj:20) 
        at clojure.core.cache$through.invoke(cache.clj:53) 
        at clojure.core.memoize$through_STAR_.invoke(memoize.clj:52) 
        at clojure.lang.Atom.swap(Atom.java:65) 
        at clojure.core$swap_BANG_.invoke(core.clj:2236) 
        at clojure.core.memoize$build_memoizer$fn__12665.doInvoke(memoize.clj:134) 
        at clojure.lang.RestFn.applyTo(RestFn.java:137) 
        at clojure.lang.AFunction$1.doInvoke(AFunction.java:29) 
        at clojure.lang.RestFn.invoke(RestFn.java:408) 
        at clojure.tools.analyzer.jvm$desugar_host_expr.invoke(jvm.clj:117) 
        at clojure.tools.analyzer.jvm$macroexpand_1.invoke(jvm.clj:174) 
        at clojure.tools.analyzer$fn__12389.invoke(analyzer.clj:281) 
        at clojure.lang.MultiFn.invoke(MultiFn.java:238) 
        at clojure.tools.analyzer$fn__12346.invoke(analyzer.clj:68) 
        at clojure.lang.MultiFn.invoke(MultiFn.java:233) 
        at clojure.tools.analyzer$analyze_let.invoke(analyzer.clj:505) 
        at clojure.tools.analyzer$fn__12469.invoke(analyzer.clj:530) 
        at clojure.lang.MultiFn.invoke(MultiFn.java:233) 
        at clojure.tools.analyzer.jvm$fn__13956.invoke(jvm.clj:66) 
        at clojure.lang.MultiFn.invoke(MultiFn.java:233) 
        at clojure.tools.analyzer$fn__12389.invoke(analyzer.clj:283) 
        at clojure.lang.MultiFn.invoke(MultiFn.java:238) 
        at clojure.tools.analyzer$fn__12346.invoke(analyzer.clj:68) 
        at clojure.lang.MultiFn.invoke(MultiFn.java:233) 
        at clojure.tools.analyzer$fn__12389.invoke(analyzer.clj:284) 
        at clojure.lang.MultiFn.invoke(MultiFn.java:238) 
        at clojure.tools.analyzer$fn__12346.invoke(analyzer.clj:68) 
        at clojure.lang.MultiFn.invoke(MultiFn.java:233) 
        at clojure.tools.analyzer$fn__12392.invoke(analyzer.clj:295) 
        at clojure.lang.MultiFn.invoke(MultiFn.java:233) 
        at clojure.tools.analyzer.jvm$fn__13956.invoke(jvm.clj:66) 
        at clojure.lang.MultiFn.invoke(MultiFn.java:233) 
        at clojure.tools.analyzer$analyze_body.invoke(analyzer.clj:366) 
        at clojure.tools.analyzer$analyze_let.invoke(analyzer.clj:520) 
        at clojure.tools.analyzer$fn__12471.invoke(analyzer.clj:540) 
        at clojure.lang.MultiFn.invoke(MultiFn.java:233) 
        at clojure.tools.analyzer.jvm$fn__13956.invoke(jvm.clj:66) 
        at clojure.lang.MultiFn.invoke(MultiFn.java:233) 
        at clojure.tools.analyzer$fn__12389.invoke(analyzer.clj:283) 
        at clojure.lang.MultiFn.invoke(MultiFn.java:238) 
        at clojure.tools.analyzer$fn__12346.invoke(analyzer.clj:68) 
        at clojure.lang.MultiFn.invoke(MultiFn.java:233) 
        at clojure.tools.analyzer$fn__12389.invoke(analyzer.clj:284) 
        at clojure.lang.MultiFn.invoke(MultiFn.java:238) 
        at clojure.tools.analyzer$fn__12346.invoke(analyzer.clj:68) 
        at clojure.lang.MultiFn.invoke(MultiFn.java:233) 
        at clojure.tools.analyzer$fn__12392.invoke(analyzer.clj:295) 
        at clojure.lang.MultiFn.invoke(MultiFn.java:233) 
        at clojure.tools.analyzer.jvm$fn__13956.invoke(jvm.clj:66) 
        at clojure.lang.MultiFn.invoke(MultiFn.java:233) 
        at clojure.tools.analyzer$fn__12389.invoke(analyzer.clj:283) 
        at clojure.lang.MultiFn.invoke(MultiFn.java:238) 
        at clojure.tools.analyzer$fn__12346.invoke(analyzer.clj:68) 
        at clojure.lang.MultiFn.invoke(MultiFn.java:233) 
        at clojure.tools.analyzer$analyze.invoke(analyzer.clj:123) 
        at clojure.tools.analyzer.jvm$analyze$fn__14085.invoke(jvm.clj:476) 
        at clojure.lang.AFn.applyToHelper(AFn.java:152) 
        at clojure.lang.AFn.applyTo(AFn.java:144) 
        at clojure.core$apply.invoke(core.clj:626) 
        at clojure.core$with_bindings_STAR_.doInvoke(core.clj:1864) 
        at clojure.lang.RestFn.invoke(RestFn.java:425) 
        at clojure.tools.analyzer.jvm$analyze.invoke(jvm.clj:474) 
        at clojure.tools.analyzer.jvm$analyze.invoke(jvm.clj:467) 
        at clojure.core.async.impl.ioc_macros$state_machine.invoke(ioc_macros.clj:1062) 
        at clojure.core.async$go.doInvoke(async.clj:384) 
        at clojure.lang.RestFn.invoke(RestFn.java:442) 
        at clojure.lang.Var.invoke(Var.java:388) 
        at clojure.lang.AFn.applyToHelper(AFn.java:160) 
        at clojure.lang.Var.applyTo(Var.java:700) 
        at clojure.lang.Compiler.macroexpand1(Compiler.java:6606)

Stacktrace from Andy Fingerhut's code, in Eastwood:

% lein do clean, eastwood '{:namespaces [clojure.reflect]}'
== Eastwood 0.2.1 Clojure 1.7.0-alpha4c979only JVM 1.7.0_45
== Linting clojure.reflect ==
Exception thrown during phase :analyze+eval of linting namespace clojure.reflect
IllegalArgumentException No implementation of method: :do-reflect of protocol: #'clojure.reflect/Reflector found for class: clojure.reflect.JavaReflector
	clojure.core/-cache-protocol-fn (core_deftype.clj:555)
	clojure.reflect/eval6486/fn--6487/G--6470--6490 (form-init6080826551765301071.clj:1)
	clojure.core/partial/fn--4490 (core.clj:2489)
	clojure.reflect/type-reflect (reflect.clj:100)
	clojure.core/apply (core.clj:632)
	eastwood.copieddeps.dep2.clojure.tools.analyzer.jvm.utils/type-reflect (utils.clj:24)
	eastwood.copieddeps.dep2.clojure.tools.analyzer.jvm.utils/members*--1293 (utils.clj:271)
	clojure.core/apply (core.clj:626)
	eastwood.copieddeps.dep3.clojure.core.memoize/through*/fn--1072 (memoize.clj:66)
	eastwood.copieddeps.dep4.clojure.core.cache/through/fn--872 (cache.clj:55)
	eastwood.copieddeps.dep3.clojure.core.memoize/through*/fn--1068/fn--1069 (memoize.clj:65)
	eastwood.copieddeps.dep3.clojure.core.memoize/d-lay/reify--1063 (memoize.clj:54)
	clojure.core/deref (core.clj:2202)
	eastwood.copieddeps.dep3.clojure.core.memoize/build-memoizer/fn--1123 (memoize.clj:152)
	clojure.lang.AFunction$1.doInvoke (AFunction.java:29)
	eastwood.copieddeps.dep2.clojure.tools.analyzer.jvm.utils/members (utils.clj:280)
	eastwood.copieddeps.dep2.clojure.tools.analyzer.jvm.utils/instance-members (utils.clj:289)
	eastwood.copieddeps.dep2.clojure.tools.analyzer.jvm.utils/instance-methods (utils.clj:299)
	eastwood.copieddeps.dep2.clojure.tools.analyzer.passes.jvm.validate/validate-call (validate.clj:91)
	eastwood.copieddeps.dep2.clojure.tools.analyzer.passes.jvm.validate/eval2056/fn--2058 (validate.clj:148)
	clojure.lang.MultiFn.invoke (MultiFn.java:229)
	eastwood.copieddeps.dep2.clojure.tools.analyzer.passes.jvm.validate/validate (validate.clj:264)
	clojure.lang.Var.invoke (Var.java:379)
	eastwood.copieddeps.dep1.clojure.tools.analyzer.passes/compile-passes/fn--574/fn--579 (passes.clj:171)
	eastwood.copieddeps.dep1.clojure.tools.analyzer.passes/compile-passes/fn--574/fn--581 (passes.clj:173)
	eastwood.copieddeps.dep1.clojure.tools.analyzer.passes/compile-passes/fn--574/fn--581 (passes.clj:173)
	eastwood.copieddeps.dep1.clojure.tools.analyzer.passes/compile-passes/fn--574/fn--581 (passes.clj:173)
	clojure.core/partial/fn--4492 (core.clj:2496)
	eastwood.copieddeps.dep1.clojure.tools.analyzer.ast/walk/walk--191 (ast.clj:102)
	eastwood.copieddeps.dep1.clojure.tools.analyzer.ast/walk/walk--191/walk--192 (ast.clj:96)
	eastwood.copieddeps.dep1.clojure.tools.analyzer.ast/-update-children/fn--182 (ast.clj:51)
	clojure.lang.ArrayChunk.reduce (ArrayChunk.java:63)
	clojure.core.protocols/fn--6433 (protocols.clj:103)
	clojure.core.protocols/fn--6395/G--6390--6404 (protocols.clj:19)
	clojure.core.protocols/seq-reduce (protocols.clj:31)
	clojure.core.protocols/fn--6414 (protocols.clj:65)
	clojure.core.protocols/fn--6369/G--6364--6382 (protocols.clj:13)
	clojure.core/reduce (core.clj:6461)
	eastwood.copieddeps.dep1.clojure.tools.analyzer.ast/-update-children (ast.clj:56)
	eastwood.copieddeps.dep1.clojure.tools.analyzer.ast/update-children-reduced (ast.clj:64)
	eastwood.copieddeps.dep1.clojure.tools.analyzer.ast/walk/walk--191 (ast.clj:99)
	eastwood.copieddeps.dep1.clojure.tools.analyzer.ast/walk/walk--191/walk--192 (ast.clj:96)
	eastwood.copieddeps.dep1.clojure.tools.analyzer.ast/-update-children/fn--182 (ast.clj:51)
	clojure.lang.ArrayChunk.reduce (ArrayChunk.java:63)
	clojure.core.protocols/fn--6433 (protocols.clj:103)
	clojure.core.protocols/fn--6395/G--6390--6404 (protocols.clj:19)
	clojure.core.protocols/seq-reduce (protocols.clj:31)
	clojure.core.protocols/fn--6414 (protocols.clj:65)
	clojure.core.protocols/fn--6369/G--6364--6382 (protocols.clj:13)
	clojure.core/reduce (core.clj:6461)
	eastwood.copieddeps.dep1.clojure.tools.analyzer.ast/-update-children (ast.clj:56)
	eastwood.copieddeps.dep1.clojure.tools.analyzer.ast/update-children-reduced (ast.clj:64)
	eastwood.copieddeps.dep1.clojure.tools.analyzer.ast/walk/walk--191 (ast.clj:99)
	eastwood.copieddeps.dep1.clojure.tools.analyzer.ast/walk/walk--191/walk--192 (ast.clj:96)
	eastwood.copieddeps.dep1.clojure.tools.analyzer.utils/mapv' (utils.clj:208)
	eastwood.copieddeps.dep1.clojure.tools.analyzer.ast/-update-children/fn--182 (ast.clj:51)
	clojure.lang.ArrayChunk.reduce (ArrayChunk.java:58)
	clojure.core.protocols/fn--6433 (protocols.clj:103)
	clojure.core.protocols/fn--6395/G--6390--6404 (protocols.clj:19)
	clojure.core.protocols/seq-reduce (protocols.clj:31)
	clojure.core.protocols/fn--6414 (protocols.clj:65)
	clojure.core.protocols/fn--6369/G--6364--6382 (protocols.clj:13)
	clojure.core/reduce (core.clj:6461)
	eastwood.copieddeps.dep1.clojure.tools.analyzer.ast/-update-children (ast.clj:56)
	eastwood.copieddeps.dep1.clojure.tools.analyzer.ast/update-children-reduced (ast.clj:64)
	eastwood.copieddeps.dep1.clojure.tools.analyzer.ast/walk/walk--191 (ast.clj:99)
	eastwood.copieddeps.dep1.clojure.tools.analyzer.ast/walk/walk--191/walk--192 (ast.clj:96)
	eastwood.copieddeps.dep1.clojure.tools.analyzer.ast/-update-children/fn--182 (ast.clj:51)
	clojure.lang.ArrayChunk.reduce (ArrayChunk.java:58)
	clojure.core.protocols/fn--6433 (protocols.clj:103)
	clojure.core.protocols/fn--6395/G--6390--6404 (protocols.clj:19)
	clojure.core.protocols/seq-reduce (protocols.clj:31)
	clojure.core.protocols/fn--6414 (protocols.clj:65)
	clojure.core.protocols/fn--6369/G--6364--6382 (protocols.clj:13)
	clojure.core/reduce (core.clj:6461)
	eastwood.copieddeps.dep1.clojure.tools.analyzer.ast/-update-children (ast.clj:56)
	eastwood.copieddeps.dep1.clojure.tools.analyzer.ast/update-children-reduced (ast.clj:64)
	eastwood.copieddeps.dep1.clojure.tools.analyzer.ast/walk/walk--191 (ast.clj:99)
	eastwood.copieddeps.dep1.clojure.tools.analyzer.ast/walk/walk--191/walk--192 (ast.clj:96)
	eastwood.copieddeps.dep1.clojure.tools.analyzer.utils/mapv' (utils.clj:208)
	eastwood.copieddeps.dep1.clojure.tools.analyzer.ast/-update-children/fn--182 (ast.clj:51)
	clojure.lang.ArrayChunk.reduce (ArrayChunk.java:58)
	clojure.core.protocols/fn--6433 (protocols.clj:103)
	clojure.core.protocols/fn--6395/G--6390--6404 (protocols.clj:19)
	clojure.core.protocols/seq-reduce (protocols.clj:31)
	clojure.core.protocols/fn--6414 (protocols.clj:65)
	clojure.core.protocols/fn--6369/G--6364--6382 (protocols.clj:13)
	clojure.core/reduce (core.clj:6461)
	eastwood.copieddeps.dep1.clojure.tools.analyzer.ast/-update-children (ast.clj:56)
	eastwood.copieddeps.dep1.clojure.tools.analyzer.ast/update-children-reduced (ast.clj:64)
	eastwood.copieddeps.dep1.clojure.tools.analyzer.ast/walk/walk--191 (ast.clj:99)
	eastwood.copieddeps.dep1.clojure.tools.analyzer.ast/walk/walk--191/walk--192 (ast.clj:96)
	eastwood.copieddeps.dep1.clojure.tools.analyzer.ast/-update-children/fn--182 (ast.clj:51)
	clojure.lang.ArrayChunk.reduce (ArrayChunk.java:63)
	clojure.core.protocols/fn--6433 (protocols.clj:103)
	clojure.core.protocols/fn--6395/G--6390--6404 (protocols.clj:19)
	clojure.core.protocols/seq-reduce (protocols.clj:31)
	clojure.core.protocols/fn--6414 (protocols.clj:65)
	clojure.core.protocols/fn--6369/G--6364--6382 (protocols.clj:13)
	clojure.core/reduce (core.clj:6461)
	eastwood.copieddeps.dep1.clojure.tools.analyzer.ast/-update-children (ast.clj:56)
	eastwood.copieddeps.dep1.clojure.tools.analyzer.ast/update-children-reduced (ast.clj:64)
	eastwood.copieddeps.dep1.clojure.tools.analyzer.ast/walk/walk--191 (ast.clj:99)
	eastwood.copieddeps.dep1.clojure.tools.analyzer.ast/walk/walk--191/walk--192 (ast.clj:96)
	eastwood.copieddeps.dep1.clojure.tools.analyzer.ast/-update-children/fn--182 (ast.clj:51)
	clojure.lang.ArrayChunk.reduce (ArrayChunk.java:63)
	clojure.core.protocols/fn--6433 (protocols.clj:103)
	clojure.core.protocols/fn--6395/G--6390--6404 (protocols.clj:19)
	clojure.core.protocols/seq-reduce (protocols.clj:31)
	clojure.core.protocols/fn--6414 (protocols.clj:65)
	clojure.core.protocols/fn--6369/G--6364--6382 (protocols.clj:13)
	clojure.core/reduce (core.clj:6461)
	eastwood.copieddeps.dep1.clojure.tools.analyzer.ast/-update-children (ast.clj:56)
	eastwood.copieddeps.dep1.clojure.tools.analyzer.ast/update-children-reduced (ast.clj:64)
	eastwood.copieddeps.dep1.clojure.tools.analyzer.ast/walk/walk--191 (ast.clj:99)
	eastwood.copieddeps.dep1.clojure.tools.analyzer.ast/walk/walk--191/walk--192 (ast.clj:96)
	eastwood.copieddeps.dep1.clojure.tools.analyzer.utils/mapv' (utils.clj:208)
	eastwood.copieddeps.dep1.clojure.tools.analyzer.ast/-update-children/fn--182 (ast.clj:51)
	clojure.lang.ArrayChunk.reduce (ArrayChunk.java:58)
	clojure.core.protocols/fn--6433 (protocols.clj:103)
	clojure.core.protocols/fn--6395/G--6390--6404 (protocols.clj:19)
	clojure.core.protocols/seq-reduce (protocols.clj:31)
	clojure.core.protocols/fn--6414 (protocols.clj:65)
	clojure.core.protocols/fn--6369/G--6364--6382 (protocols.clj:13)
	clojure.core/reduce (core.clj:6461)
	eastwood.copieddeps.dep1.clojure.tools.analyzer.ast/-update-children (ast.clj:56)
	eastwood.copieddeps.dep1.clojure.tools.analyzer.ast/update-children-reduced (ast.clj:64)
	eastwood.copieddeps.dep1.clojure.tools.analyzer.ast/walk/walk--191 (ast.clj:99)
	eastwood.copieddeps.dep1.clojure.tools.analyzer.ast/walk/walk--191/walk--192 (ast.clj:96)
	eastwood.copieddeps.dep1.clojure.tools.analyzer.utils/mapv' (utils.clj:208)
	eastwood.copieddeps.dep1.clojure.tools.analyzer.ast/-update-children/fn--182 (ast.clj:51)
	clojure.lang.ArrayChunk.reduce (ArrayChunk.java:63)
	clojure.core.protocols/fn--6433 (protocols.clj:103)
	clojure.core.protocols/fn--6395/G--6390--6404 (protocols.clj:19)
	clojure.core.protocols/seq-reduce (protocols.clj:31)
	clojure.core.protocols/fn--6414 (protocols.clj:65)
	clojure.core.protocols/fn--6369/G--6364--6382 (protocols.clj:13)
	clojure.core/reduce (core.clj:6461)
	eastwood.copieddeps.dep1.clojure.tools.analyzer.ast/-update-children (ast.clj:56)
	eastwood.copieddeps.dep1.clojure.tools.analyzer.ast/update-children-reduced (ast.clj:64)
	eastwood.copieddeps.dep1.clojure.tools.analyzer.ast/walk/walk--191 (ast.clj:99)
	eastwood.copieddeps.dep1.clojure.tools.analyzer.ast/walk/walk--191/walk--192 (ast.clj:96)
	eastwood.copieddeps.dep1.clojure.tools.analyzer.utils/mapv' (utils.clj:208)
	eastwood.copieddeps.dep1.clojure.tools.analyzer.ast/-update-children/fn--182 (ast.clj:51)
	clojure.lang.ArrayChunk.reduce (ArrayChunk.java:63)
	clojure.core.protocols/fn--6433 (protocols.clj:103)
	clojure.core.protocols/fn--6395/G--6390--6404 (protocols.clj:19)
	clojure.core.protocols/seq-reduce (protocols.clj:31)
	clojure.core.protocols/fn--6414 (protocols.clj:65)
	clojure.core.protocols/fn--6369/G--6364--6382 (protocols.clj:13)
	clojure.core/reduce (core.clj:6461)
	eastwood.copieddeps.dep1.clojure.tools.analyzer.ast/-update-children (ast.clj:56)
	eastwood.copieddeps.dep1.clojure.tools.analyzer.ast/update-children-reduced (ast.clj:64)
	eastwood.copieddeps.dep1.clojure.tools.analyzer.ast/walk/walk--191 (ast.clj:99)
	eastwood.copieddeps.dep1.clojure.tools.analyzer.ast/walk/walk--191/walk--192 (ast.clj:96)
	eastwood.copieddeps.dep1.clojure.tools.analyzer.utils/mapv' (utils.clj:208)
	eastwood.copieddeps.dep1.clojure.tools.analyzer.ast/-update-children/fn--182 (ast.clj:51)
	clojure.lang.ArrayChunk.reduce (ArrayChunk.java:63)
	clojure.core.protocols/fn--6433 (protocols.clj:103)
	clojure.core.protocols/fn--6395/G--6390--6404 (protocols.clj:19)
	clojure.core.protocols/seq-reduce (protocols.clj:31)
	clojure.core.protocols/fn--6414 (protocols.clj:65)
	clojure.core.protocols/fn--6369/G--6364--6382 (protocols.clj:13)
	clojure.core/reduce (core.clj:6461)
	eastwood.copieddeps.dep1.clojure.tools.analyzer.ast/-update-children (ast.clj:56)
	eastwood.copieddeps.dep1.clojure.tools.analyzer.ast/update-children-reduced (ast.clj:64)
	eastwood.copieddeps.dep1.clojure.tools.analyzer.ast/walk/walk--191 (ast.clj:99)
	eastwood.copieddeps.dep1.clojure.tools.analyzer.ast/walk (ast.clj:95)
	eastwood.copieddeps.dep1.clojure.tools.analyzer.ast/postwalk (ast.clj:115)
	eastwood.copieddeps.dep1.clojure.tools.analyzer.ast/postwalk (ast.clj:113)
	eastwood.copieddeps.dep1.clojure.tools.analyzer.passes/compile-passes/analyze--586 (passes.clj:175)
	clojure.core/comp/fn--4458 (core.clj:2434)
	clojure.core/comp/fn--4458 (core.clj:2434)
	eastwood.analyze-ns/run-passes (analyze_ns.clj:157)
	eastwood.copieddeps.dep2.clojure.tools.analyzer.jvm/analyze/fn--3553 (jvm.clj:474)
	clojure.core/apply (core.clj:626)
	clojure.core/with-bindings* (core.clj:1864)
	eastwood.copieddeps.dep2.clojure.tools.analyzer.jvm/analyze (jvm.clj:470)
	eastwood.copieddeps.dep2.clojure.tools.analyzer.jvm/analyze+eval (jvm.clj:518)
	eastwood.copieddeps.dep2.clojure.tools.analyzer.jvm/analyze+eval/fn--3574 (jvm.clj:505)
	clojure.core/mapv/fn--6657 (core.clj:6558)
	clojure.lang.ArrayChunk.reduce (ArrayChunk.java:63)
	clojure.core.protocols/fn--6433 (protocols.clj:103)
	clojure.core.protocols/fn--6395/G--6390--6404 (protocols.clj:19)
	clojure.core.protocols/seq-reduce (protocols.clj:31)
	clojure.core.protocols/fn--6418 (protocols.clj:53)
	clojure.core.protocols/fn--6369/G--6364--6382 (protocols.clj:13)
	clojure.core/reduce (core.clj:6461)
	clojure.core/mapv (core.clj:6558)
	eastwood.copieddeps.dep2.clojure.tools.analyzer.jvm/analyze+eval (jvm.clj:503)
	eastwood.analyze-ns/analyze-file/fn--4173/fn--4175 (analyze_ns.clj:279)
	eastwood.analyze-ns/analyze-file/fn--4173 (analyze_ns.clj:276)
	eastwood.analyze-ns/analyze-file (analyze_ns.clj:274)
	eastwood.analyze-ns/analyze-ns (analyze_ns.clj:327)
	eastwood.lint/lint-ns (lint.clj:569)
	eastwood.lint/eastwood-core/fn--6336 (lint.clj:1041)
	eastwood.lint/eastwood-core (lint.clj:1040)
	eastwood.lint/eastwood (lint.clj:1154)
	eastwood.lint/eastwood-from-cmdline (lint.clj:1167)
	clojure.lang.Var.invoke (Var.java:379)
	eastwood.versioncheck/run-eastwood (versioncheck.clj:15)
	user/eval21 (form-init6080826551765301071.clj:1)
	clojure.lang.Compiler.eval (Compiler.java:6767)
	clojure.lang.Compiler.eval (Compiler.java:6757)
	clojure.lang.Compiler.load (Compiler.java:7194)
	clojure.lang.Compiler.loadFile (Compiler.java:7150)
	clojure.main/load-script (main.clj:274)
	clojure.main/init-opt (main.clj:279)
	clojure.main/initialize (main.clj:307)
	clojure.main/null-opt (main.clj:342)
	clojure.main/main (main.clj:420)
	clojure.lang.Var.invoke (Var.java:383)
	clojure.lang.Var.applyTo (Var.java:700)
	clojure.main.main (main.java:37)

The following form was being processed during the exception:

(defprotocol
 TypeReference
 "A TypeReference can be unambiguously converted to a type name on\n   the host platform.\n\n   All typerefs are normalized into symbols. If you need to\n   normalize a typeref yourself, call typesym."
 (typename
  [o]
  "Returns Java name as returned by ASM getClassName, e.g. byte[], java.lang.String[]"))

Shown again with metadata for debugging (some metadata elided for brevity):

^{:line 48}
(^{:line 48} defprotocol
 ^{:line 48} TypeReference
 "A TypeReference can be unambiguously converted to a type name on\n   the host platform.\n\n   All typerefs are normalized into symbols. If you need to\n   normalize a typeref yourself, call typesym."
 ^{:line 54}
 (^{:line 54} typename
  ^{:line 54} [^{:line 54} o]
  "Returns Java name as returned by ASM getClassName, e.g. byte[], java.lang.String[]"))

An exception was thrown while analyzing namespace clojure.reflect 
Lint results may be incomplete.  If there are compilation errors in
your code, try fixing those.  If not, check above for info on the
exception.

Exception thrown while analyzing last namespace.

== Warnings: 0 (not including reflection warnings)  Exceptions thrown: 1
Subprocess failed


 Comments   
Comment by Nicola Mometto [ 12/Jan/15 2:04 PM ]

I can't tell about the exception Sean is getting, but the one Andy is getting with eastwood, I believe should not be considered a bug.

What's happening in eastwood is that the entire clojure.reflect namespace is being reloaded, deftypes and defprotocols included, while a reference to an instance of a previous deftype is being used by the eastwood codebase (from the tools.analyzer.jvm dep)

This used to work only because of the bug that CLJ-979 fixed where since clojure.reflect is AOT compiled, re-evaluating the defprotocol didn't change the underlying interface.

This code demonstrates this:

;; PRE CLJ-979, re-evaluating an AOT compiled defprotocol didn't change the underlying interface used
user=> *clojure-version*
{:interim true, :major 1, :minor 7, :incremental 0, :qualifier "alpha4"}
user=> (use 'clojure.reflect)
nil
user=> (in-ns 'clojure.reflect)
#<Namespace clojure.reflect>
clojure.reflect=> (hash (:on-interface Reflector))
2141314409
clojure.reflect=> (defprotocol Reflector (do-reflect [reflector typeref]))
Reflector
clojure.reflect=> (hash (:on-interface Reflector))
2141314409
clojure.reflect=>


;; AFTER CLJ-979, re-evaluating an AOT compiled defprotocol _does_ change the underlying interface used
user=> *clojure-version*
{:interim true, :major 1, :minor 7, :incremental 0, :qualifier "master"}
user=> (use 'clojure.reflect)
nil
user=> (in-ns 'clojure.reflect)
#<Namespace clojure.reflect>
clojure.reflect=> (hash (:on-interface Reflector))
390902174
clojure.reflect=> (defprotocol Reflector (do-reflect [reflector typeref]))
Reflector
clojure.reflect=> (hash (:on-interface Reflector))
1673776464


;; note that while the new behaviour causes the eastwood bug (and maybe the one Sean is seeing too),
;; the new behaviour is consistent with how redefining a protocol not AOT compiled behaved:
user=> *clojure-version*
{:interim true, :major 1, :minor 7, :incremental 0, :qualifier "alpha4"}
user=> (defprotocol foo)
foo
user=> (hash (:on-interface foo))
1058055630
user=> (defprotocol foo)
foo
user=> (hash (:on-interface foo))
1333687570

Again, I don't know if the exception that Sean is getting is related to this issue without looking at the code, bug I suspect a similar scenario given that Sean told me in IRC that there is indeed some AOT compilation involved with a wrapper around clojure.cache.

I personally wouldn't consider this a bug, I want to emphatize that the exception demonstrated by Andy would have occurred in 1.7.0-alpha4 too hadn't clojure.reflect been AOT compiled and the fact that it worked instead is just an accident of the bug fixed with CLJ-979.

Comment by Sean Corfield [ 12/Jan/15 2:19 PM ]

The only AOT compilation in our entire code base is one small library that implements a DBAppender for log4j. That's a separate project that we AOT compile and then depend on in our main (non-AOT) projects. That AOT project, in order to avoid pulling in a lot of transitive dependencies that get AOT-compiled too (due to the over-enthusiasm of Clojure's AOT process), uses runtime require/resolve to load the symbols it needs from the main project.

Given what you're saying, it sounds like that runtime require/resolve is broken by fixing CLJ-979 (and was therefore only working "by accident" before)?

Comment by Nicola Mometto [ 12/Jan/15 2:34 PM ]

Here a better test case showcasing how 1.7.0-alpha5 behaves consistently between AOT and JIT scenarios while 1.7.0-alpha4 has a different behaviour when AOT compilation is involved. 1.7.0-alpha4 JIT is consistent with 1.7.0-alpha5 JIT and AOT

[~]> cat test.clj
(ns test)
(defprotocol p (f [_]))
(deftype t [] p (f [_] 1))
[~]> mkdir classes
;; 1.7.0-alpha4 AOT
[~]> java -cp .m2/repository/org/clojure/clojure/1.7.0-alpha4/clojure-1.7.0-alpha4.jar:.:classes clojure.main
Clojure 1.7.0-alpha4
user=> (binding [*compile-files* true] (load "test"))
nil
user=> (in-ns 'test)
#<Namespace test>
test=> (def a (t.))
#'test/a
test=> (defprotocol p (f [_]))
p
test=> (deftype t [] p (f [_] 1))
test.t
test=> (f a)
1
;; notice how this is the only call that succeds since the re-defined protocol is still backed by the 
;; AOT compiled class rather than the one generated JIT with the defprotocol call at the repl
test=>
[~]> rm -rf classes/*
;; 1.7.0-alpha4 no AOT
[~]> java -cp .m2/repository/org/clojure/clojure/1.7.0-alpha4/clojure-1.7.0-alpha4.jar:.:classes clojure.main
Clojure 1.7.0-alpha4
user=> (load "test")
nil
user=> (in-ns 'test)
#<Namespace test>
test=> (def a (t.))
#'test/a
test=> (defprotocol p (f [_]))
p
test=> (deftype t [] p (f [_] 1))
test.t
test=> (f a)
IllegalArgumentException No implementation of method: :f of protocol: #'test/p found for class: test.t  clojure.core/-cache-protocol-fn (core_deftype.clj:555)
test=>
;; 1.7.0-alpha5 AOT
[~]> java -cp .m2/repository/org/clojure/clojure/1.7.0-alpha5/clojure-1.7.0-alpha5.jar:.:classes clojure.main
Clojure 1.7.0-alpha5
user=> (binding [*compile-files* true] (load "test"))
nil
user=> (in-ns 'test)
#<Namespace test>
test=> (def a (t.))
#'test/a
test=> (defprotocol p (f [_]))
p
test=> (deftype t [] p (f [_] 1))
test.t
test=> (f a)
IllegalArgumentException No implementation of method: :f of protocol: #'test/p found for class: test.t  clojure.core/-cache-protocol-fn (core_deftype.clj:555)
test=>
[~]> rm -rf classes/*
;; 1.7.0-alpha5 no AOT
[~]> java -cp .m2/repository/org/clojure/clojure/1.7.0-alpha5/clojure-1.7.0-alpha5.jar:.:classes clojure.main
Clojure 1.7.0-alpha5
user=> (load "test")
nil
user=> (in-ns 'test)
#<Namespace test>
test=> (def a (t.))
#'test/a
test=> (defprotocol p (f [_]))
p
test=> (deftype t [] p (f [_] 1))
test.t
test=> (f a)
IllegalArgumentException No implementation of method: :f of protocol: #'test/p found for class: test.t  clojure.core/-cache-protocol-fn (core_deftype.clj:555)
test=>
Comment by Sean Corfield [ 12/Jan/15 2:37 PM ]

I removed the runtime require in the AOT'd project and we still get this exception so I'm less inclined to believe our usage is buggy here. The only other runtime require we do is on our New Relic wrapper - but we get the exception in a pure Clojure project that does not use that at all (and now has no runtime require calls at all, as far as I can tell).

Comment by Nicola Mometto [ 12/Jan/15 2:40 PM ]

Sean, I cannot be sure about your case as I don't know the code/compilation scenario behind the exception.
All I said is however true for the exception reported by Andy and that the two exceptions apperas to be the same and to be caused by the same issue.

I can't also tell you whether this will be considered a regression or if the it will be chosen that your code happened to work by accident before.

I personally don't believe this should be considered a regression since the new behaviour makes JIT and AOT consistent while previously they weren't, but I don't have the authority to decide on this matter

Comment by Sean Corfield [ 12/Jan/15 3:39 PM ]

To rule out some other interactions in our code, I moved our one and only AOT-compiled file/namespace into the main project and removed the dependency on that separate library, and I also made sure there are no require or other dynamic loading occurring at runtime. The AOT-compiled namespace has no dependencies except on clojure.core and I verified that no .class files are generated for anything except that single namespace.

I still get that exception, always on `PluggableMemoization`. I'm going to start going through the libraries we depend on and see if anything else might be bringing in an AOT-version of either core.cache or core.memoize.

Comment by Sean Corfield [ 12/Jan/15 3:47 PM ]

Searching through the libraries we use, core.typed seems to contain AOT'd versions of core.cache and core.memoize so I'm going to build a version of our system without core.typed to see if that's the culprit here.

Comment by Sean Corfield [ 12/Jan/15 4:32 PM ]

Removing core.typed completely from our system seems to resolve the problem. I'm still dealing with the fallout from other, earlier changes made for debugging but at this point I'm fairly confident that without core.typed, our applications will run on Alpha 5. Will update again when I'm done testing.

Comment by Sean Corfield [ 12/Jan/15 5:31 PM ]

Confirmed: removing core.typed allows all of our tests to pass and all of our applications to work correctly on Alpha 5. I'll raised an issue against core.typed at this point. If Clojure/core feel this ticket is not a bug, it can be closed.

Comment by Nicola Mometto [ 13/Jan/15 10:14 AM ]

Sean, for the scenario to be what I described is happening in eastwood, loading an AOT core.[cache/memoize] is not enough.
What needs to go on is also having core.[cache/memoize] realoaded JIT after the AOT compiled version has been loaded, this can happen if for example the version that core.typed ships with is older than the one one of your deps is pulling in.

And again, I personally think that this scenario should not be considered a bug but I don't speak for Clojure/core so it's possible that Alex, Rich or some other screener will have a different opinion than mine.

Comment by Sean Corfield [ 13/Jan/15 12:44 PM ]

Sorry if it wasn't clear from my stream of comments: our apps rely on core.cache and so it is brought in via JIT in some namespaces - as well as the AOT version loaded via core.typed in other namespaces. So, yes, the scenario you describe is definitely happening in our code - it just took me a while to find where the AOT version was coming from.

I consider this a bug against core.typed - it should not ship AOT versions of other libraries that folks might also be using via JIT - and created an issue in JIRA for that.

This fix to CLJ-979 will mean that no library would ever be able to bundle AOT versions of other libraries (that contained classes generated via deftype etc) since any applications that used said library - and also used any of those other libraries - would trip over this. That may well be a good thing: bundling libraries can already cause version conflicts and mixing AOT in just makes that harder to debug and harder to deal with.

Comment by Alex Miller [ 15/Jan/15 2:40 PM ]

I agree that shipping versions of other libraries inside your own jar (as core.typed appears to be doing) is bad.

I am a little concerned that from a user's POV maybe we have just replaced one "AOT is weird and doesn't work like I expect" with another, regardless of how consistent that behavior is.

Still something I'm just mulling through.

Comment by Nicola Mometto [ 15/Jan/15 2:53 PM ]

Alex, I share your worry however I'd like to point out that if the cause of the error Sean is getting is the same as the cause for the exception for eastwood, that is, the reloading of namespaces containg deftypes/defrecord after one instance of a said type/interface has been used, then we should expect things to break as there is now no difference between how that behaves under JIT and under AOT.

In other words, I don't believe it is reasonable to expect code like this to work:

(deftype X [a])
(def a (X. 1))
(deftype X [a])
(.a ^X a)

The fact that in some scenarios (strictly under AOT!) code similar to that used to work rather than throw should not be a reason to consider the new, consistent behaviour, wrong IMHO.

Note that if it turns out that Sean's exception (or any other other reported issue caused by CLJ-979) is caused by a different scenario than the one I described, then I agree that this is a regression that should be addressed

Comment by Alex Miller [ 16/Jan/15 12:56 PM ]

No plans for further action on this ticket and will pursue a fix for the bad core.typed jar under that ticket.

Comment by Sean Corfield [ 16/Jan/15 1:11 PM ]

I agree no further action is needed but have a question / suggestion:

Is there a reasonable way for Clojure to detect the situation and provide a better error message?

Or is it perhaps worth expanding the error message to include a suggestion to check whether a namespace has been reloaded or a type has been redefined?

Comment by Alex Miller [ 16/Jan/15 3:15 PM ]

It's a good question. I'm not sure that it's easy to detect?

Comment by Nicola Mometto [ 29/Jan/15 1:29 PM ]

It's possible that CLJ-1650 is actually what's causing Sean's exception.





[CLJ-1638] Regression - PersistentVector.create(List) was removed in 1.7.0-alpha5 Created: 12/Jan/15  Updated: 20/Mar/15  Resolved: 20/Mar/15

Status: Closed
Project: Clojure
Component/s: None
Affects Version/s: Release 1.7
Fix Version/s: Release 1.7

Type: Defect Priority: Critical
Reporter: Alex Miller Assignee: Unassigned
Resolution: Completed Votes: 0
Labels: collections, regression
Environment:

1.7.0-alpha5


Attachments: Text File clj-1638-2.patch     Text File clj-1638.patch    
Patch: Code
Approval: Ok

 Description   

For CLJ-1546, PersistentVector.create(List) was replaced with PersistentVector.create(ArrayList). At least one library (flatland) was calling this method directly and was broken by the change.

Approach: Change create(ArrayList) to more general prior method create(List).

Patch: clj-1638-2.patch

Screened by:



 Comments   
Comment by Rich Hickey [ 20/Feb/15 7:42 AM ]

Is there a good reason to have both PersistentVector.create(List)and PersistentVector.create(ArrayList)?

Comment by Fogus [ 27/Feb/15 9:08 AM ]

This couldn't possibly be more straight-forward.





[CLJ-1637] vec fails on MapEntry Created: 11/Jan/15  Updated: 20/Feb/15  Resolved: 20/Feb/15

Status: Closed
Project: Clojure
Component/s: None
Affects Version/s: Release 1.7
Fix Version/s: Release 1.7

Type: Defect Priority: Critical
Reporter: Alex Miller Assignee: Alex Miller
Resolution: Completed Votes: 0
Labels: regression
Environment:

1.7.0-alpha5


Attachments: Text File clj-1637-jdevuyst.patch     Text File clj-1637.patch     Text File clj-1637-with-test.patch    
Patch: Code
Approval: Ok

 Description   

After CLJ-1546:

(vec (first {1 2}))

Cause: (if (vector? coll) (with-meta coll nil) ...) checks that something is IPersistentVector, then sends it to something that takes IObj, so anything that is one but not both throws an error. In Clojure itself, this is the set of classes extending from AMapEntry.

Alternatives:

1. Make AMapEntry implement IObj - this fixes everything in Clojure and keeps the vec code as is but still leaves open this gap for any external implementation of IPersistentVector.
2. Check for this case explicitly in vec. (if (vector? coll) (if (instance? clojure.lang.IObj coll) (with-meta coll nil) (clojure.lang.LazilyPersistentVector/create coll)) ...). Perf testing shows no significant difference in performance with the change.
3. Pull the special check for vector? in vec.
4. Check for this case explicitly in vec and return the same instance if it's not an IObj. See clj-1637-jdevuyst.patch.

Approach: patch takes approach #2

Patch: clj-1637-with-test.patch

Screened by: Stu (also added test)



 Comments   
Comment by Nicola Mometto [ 11/Jan/15 8:19 AM ]

The correct fix is to probably make MapEntry an IObj

Comment by Nicola Mometto [ 11/Jan/15 8:21 AM ]

Actually, making AMapEntry an IObj rather than MapEntry would fix the issue for sorted-map kv-pairs too.

user=> (vec (first (sorted-map 1 1)))
ClassCastException clojure.lang.PersistentTreeMap$BlackVal cannot be cast to clojure.lang.IObj  clojure.core/with-meta--4121 (core.clj:216)
Comment by Alex Miller [ 11/Jan/15 8:35 AM ]

There are potentially a couple ways to fix this. I'll look at it next week.

Comment by Jonas De Vuyst [ 14/Jan/15 6:14 AM ]

Slightly modified patch. In the case where coll is a vector but not an IObj, simply return coll.

Comment by Jonas De Vuyst [ 14/Jan/15 7:17 AM ]

If desired I could also update the patch to fix an analogous—albeit somewhat theoretic—bug in `set`.

It does make me wonder if perhaps a `without-meta` function should be added to `clojure.core`.

I think making AMapEntry an IObj might make sense even after applying the above patches. In `AMapEntry`, perhaps `withMeta(m)` could be implemented as `asVector().withMeta(m)`. This, however, would require changing `asVector()` to return some `IObj ∩ IPersistentVector` type (e.g. `PersistentVector`). This would be straightforward to do, but requires deciding if this change in signature may be propagated to the static methods of `LazilyPersistentVector`.

Comment by Alex Miller [ 14/Jan/15 9:28 AM ]

This is a point of some debate, but the intention with the change in the implementation is to retain current behavior, which always gives you a new vector instance. It's not clear to me that there is any point in attaching meta to map entries (which also does not solve the problem for external IPersistentVector, non-IObj instances outside Clojure).

In any case, I'm going to update the description a bit to add this as an alternative.





[CLJ-1636] SeqIterator can return incorrect results Created: 10/Jan/15  Updated: 18/Jan/15  Resolved: 16/Jan/15

Status: Closed
Project: Clojure
Component/s: None
Affects Version/s: Release 1.7
Fix Version/s: Release 1.7

Type: Defect Priority: Blocker
Reporter: Alex Miller Assignee: Unassigned
Resolution: Completed Votes: 0
Labels: regression
Environment:

clojure-1.7.0-alpha5


Attachments: Text File 0001-CLJ-1636-don-t-use-this-as-a-sentinel-in-SeqIterator.patch     Text File 0001-fix-for-CLJ-1636.patch    
Patch: Code
Approval: Ok

 Description   

As of 1.7.0-alpha5, we are seeing SeqIterator return iterated results that do reflect the values of the underlying seq, in particular acting as if the seq contains a nil value when it does not. This problem is intermittent but has at times caused clojure master to fail in compilation (which is why this is marked as a blocker).

Two recent changes during 1.7 have created and exposed this problem:

1) This commit https://github.com/clojure/clojure/commit/43cc1854508d655e58e377f84836ba128971f90c changed the SeqIterator implementation to be lazier and to use "this" as a sentinel object in SeqIterator. (1.7.0-alpha2)
2) CLJ-1546 changed the implementation of vec such that PersistentHashMap and PersistentHashSet are now converted using iterator() rather than seq(). PHS/PHM use SeqIterator for their Iterator implementation. (1.7.0-alpha5)

Because of #2, we are now stressing #1 much more than before. In particular, things like defining defrecords rely heavily on vec (and set) of PHS and PHM.

Example stack trace: https://gist.github.com/puredanger/f56e3253f0668a515ec5 (seen compiling Clojure itself)

Cause: Setting seq==this; in the constructor of SeqIterator is allowing unsafe publication of the partially constructed "this" object, which can cause subtle problems in the hasNext() implementation. In particular, it seems that after inlining, on the first call, the seq==this condition when comparing the cached partially constructed instance in seq and the fully constructed version in this will return false, even though these have the same object identity. This causes the wrong path to be executed in hasNext().

Approach: Do not use this as a sentinel value.

Patch: 0001-CLJ-1636-don-t-use-this-as-a-sentinel-in-SeqIterator.patch

Screened by: Alex Miller



 Comments   
Comment by Colin Jones [ 11/Jan/15 12:40 AM ]

I was able to reproduce this (intermittently) earlier, but I've seen periods of many successful runs in a row (both with that patch reverted and with it in place), so it's been hard for me to trust what I'm seeing locally when it passes. I didn't see any evidence of AOT compilation happening (e.g. no classfiles under `target/`), so I'd have expected the new function `already-compiled?` in CLJ-1544 never to actually run.

It looks like the Cause section of the stacktrace is implicating an error in trying to `(resolve nil)`, where `nil` is an entry in an interfaces collection that should actually be empty. That's based on these two lines (along with the lines higher up in the cause):

...snip...
at clojure.core$set.invoke(core.clj:3944)
at clojure.core$emit_defrecord.invoke(core_deftype.clj:154)
...snip...

(https://github.com/clojure/clojure/blob/3e7cb1a5c840612ad41cf6e0be92480f798bc05d/src/clj/clojure/core_deftype.clj#L154)

The defrecord here in question looks like

(defrecord Foo [x y])

So the `opts+specs` var-arg argument to `defrecord` should be `nil` since there are no entries, which should mean the `interfaces` piece of the `parse-opts+specs` call should return an empty vector.

But that stacktrace confuses me, because it suggests that the `interfaces` vector, instead of being empty, contains a `nil` element. How can this be? Or what misstep have I made in tracing through this?

Comment by Alex Miller [ 11/Jan/15 12:54 AM ]

If the error is intermittent, then my pegging of CLJ-1544 may be wrong. For me, it was repeatable as of clojure commit e5a104e894ed82f244d69513918d570cee5df67d (when CLJ-1544 was applied) and I have not reproduced it prior.

Comment by Nicola Mometto [ 11/Jan/15 4:50 AM ]

Alex, just to be sure – you were able to reproduce this bug with clojure at e5a104e894ed82f244d69513918d570cee5df67d (CLJ-1544) ? I'd like to have confirmation so 9f277c80258b3d2951128ce26a07c30ad0b47af0 (CLJ-979) can be excluded as the culprit

Comment by Alex Miller [ 11/Jan/15 8:09 AM ]

Correct.

Comment by Nicola Mometto [ 11/Jan/15 8:15 AM ]

Well this is weird then.
The only way I can think of that would produce that exception is if this returned nil: https://github.com/clojure/clojure/blob/3e7cb1a5c840612ad41cf6e0be92480f798bc05d/src/clj/clojure/core_deftype.clj#L57

This means a scenario like this:

user=> (defrecord x [] +)
NullPointerException   clojure.lang.Compiler.maybeResolveIn (Compiler.java:7015)
user=> (.printStackTrace *e)
java.lang.NullPointerException, compiling:(NO_SOURCE_FILE:3:1)
	at clojure.lang.Compiler.macroexpand1(Compiler.java:6620)
	at clojure.lang.Compiler.macroexpand(Compiler.java:6678)
	at clojure.lang.Compiler.eval(Compiler.java:6752)
	at clojure.lang.Compiler.eval(Compiler.java:6731)
	at clojure.core$eval.invoke(core.clj:3076)
	at clojure.main$repl$read_eval_print__7044$fn__7047.invoke(main.clj:239)
	at clojure.main$repl$read_eval_print__7044.invoke(main.clj:239)
	at clojure.main$repl$fn__7053.invoke(main.clj:257)
	at clojure.main$repl.doInvoke(main.clj:257)
	at clojure.lang.RestFn.invoke(RestFn.java:421)
	at clojure.main$repl_opt.invoke(main.clj:323)
	at clojure.main$main.doInvoke(main.clj:421)
	at clojure.lang.RestFn.invoke(RestFn.java:397)
	at clojure.lang.Var.invoke(Var.java:375)
	at clojure.lang.AFn.applyToHelper(AFn.java:152)
	at clojure.lang.Var.applyTo(Var.java:700)
	at clojure.main.main(main.java:37)
Caused by: java.lang.NullPointerException
	at clojure.lang.Compiler.maybeResolveIn(Compiler.java:7015)
	at clojure.core$ns_resolve.invoke(core.clj:4200)
	at clojure.core$ns_resolve.invoke(core.clj:4197)
	at clojure.core$resolve.invoke(core.clj:4206)
	at clojure.core$map$fn__4523.invoke(core.clj:2612)
	at clojure.lang.LazySeq.sval(LazySeq.java:40)
	at clojure.lang.LazySeq.seq(LazySeq.java:49)
	at clojure.lang.RT.seq(RT.java:485)
	at clojure.core$seq__4103.invoke(core.clj:135)
	at clojure.core$reduce1.invoke(core.clj:899)
	at clojure.core$set.invoke(core.clj:3944)
	at clojure.core$emit_defrecord.invoke(core_deftype.clj:154)
	at clojure.core$defrecord.doInvoke(core_deftype.clj:374)
	at clojure.lang.RestFn.invoke(RestFn.java:497)
	at clojure.lang.Var.invoke(Var.java:401)
	at clojure.lang.AFn.applyToHelper(AFn.java:171)
	at clojure.lang.Var.applyTo(Var.java:700)
	at clojure.lang.Compiler.macroexpand1(Compiler.java:6607)
	... 16 more
nil

where a var is used as a protocol but no interface is present.

Comment by Nicola Mometto [ 11/Jan/15 8:18 AM ]

Alex, since I cannot reproduce, can you try getting the exception with a patched version of clojure that replaces https://github.com/clojure/clojure/blob/3e7cb1a5c840612ad41cf6e0be92480f798bc05d/src/clj/clojure/core_deftype.clj#L57
with something like

(or (:on (deref (resolve %))) 
    (println % @(resolve %)))

so we can get an idea of what's going on?

Comment by Nicola Mometto [ 11/Jan/15 11:38 AM ]

I was just able to reproduce this issue using clojure at commit 4afd4a7c14c48b5baf3c03196053066483cb4223

This means that CLJ-1544 is not responsable for this bug.

I can also confirm that this bug is intermittent, which makes figuring out what's going on really hard.

Comment by Nicola Mometto [ 11/Jan/15 12:10 PM ]

I still have absolutely no idea how this can happen but adding a bunch of printlns it turned out that for some reason in this binding of the deftype macro:

[interfaces methods opts] (parse-opts+specs opts+specs)

when opts+specs is nil, interfaces is sometimes [nil] as opposed to [].

This makes me think that there's some concurrency bug in the recent changes around the handling of vec, but this is just a guess.

Comment by Nicola Mometto [ 11/Jan/15 12:13 PM ]

I've restricted it down a bit and it looks like this part of opts+spec can bind interfaces to [nil] when impls is {}

interfaces (→ (map #(if (var? (resolve %))
                      (:on (deref (resolve %)))
                      %)
                   (keys impls))
             set
             (disj 'Object 'java.lang.Object)
             vec)
Comment by Nicola Mometto [ 11/Jan/15 12:25 PM ]

Here's an example output from my debugging tests, with the following patch applied:

diff --git a/src/clj/clojure/core_deftype.clj b/src/clj/clojure/core_deftype.clj
index 97e14cc..8f521eb 100644
--- a/src/clj/clojure/core_deftype.clj
+++ b/src/clj/clojure/core_deftype.clj
@@ -60,6 +60,8 @@ (defn- parse-opts+specs [opts+specs]
                        set
                        (disj 'Object 'java.lang.Object)
                        vec)
+        _ (when (nil? opts+specs)
+            (println impls interfaces))
         methods (map (fn [[name params & body]]
                        (cons name (maybe-destructured params body)))
                      (apply concat (vals impls)))]
{} [nil]
Exception in thread "main" java.lang.NullPointerException, compiling:(schema/utils.clj:68:1)
	at clojure.lang.Compiler.analyzeSeq(Compiler.java:6716)
	at clojure.lang.Compiler.analyze(Compiler.java:6500)
	at clojure.lang.Compiler.analyze(Compiler.java:6461)
	at clojure.lang.Compiler$BodyExpr$Parser.parse(Compiler.java:5837)
	at clojure.lang.Compiler$LetExpr$Parser.parse(Compiler.java:6155)
	at clojure.lang.Compiler.analyzeSeq(Compiler.java:6709)
	at clojure.lang.Compiler.analyze(Compiler.java:6500)
	at clojure.lang.Compiler.analyze(Compiler.java:6461)
	at clojure.lang.Compiler$BodyExpr$Parser.parse(Compiler.java:5837)
	at clojure.lang.Compiler$FnMethod.parse(Compiler.java:5272)
	at clojure.lang.Compiler$FnExpr.parse(Compiler.java:3901)
	at clojure.lang.Compiler.analyzeSeq(Compiler.java:6707)
	at clojure.lang.Compiler.analyze(Compiler.java:6500)
	at clojure.lang.Compiler.eval(Compiler.java:6765)
	at clojure.lang.Compiler.load(Compiler.java:7195)
	at clojure.lang.RT.loadResourceScript(RT.java:370)
	at clojure.lang.RT.loadResourceScript(RT.java:361)
	at clojure.lang.RT.load(RT.java:440)
	at clojure.lang.RT.load(RT.java:411)
	at clojure.core$load$fn__5424.invoke(core.clj:5848)
	at clojure.core$load.doInvoke(core.clj:5847)
	at clojure.lang.RestFn.invoke(RestFn.java:408)
	at clojure.core$load_one.invoke(core.clj:5653)
	at clojure.core$load_lib$fn__5373.invoke(core.clj:5693)
	at clojure.core$load_lib.doInvoke(core.clj:5692)
	at clojure.lang.RestFn.applyTo(RestFn.java:142)
	at clojure.core$apply.invoke(core.clj:628)
	at clojure.core$load_libs.doInvoke(core.clj:5731)
	at clojure.lang.RestFn.applyTo(RestFn.java:137)
	at clojure.core$apply.invoke(core.clj:628)
	at clojure.core$require.doInvoke(core.clj:5814)
	at clojure.lang.RestFn.invoke(RestFn.java:457)
	at plumbing.core$eval13998$loading__5316__auto____13999.invoke(core.clj:1)
	at plumbing.core$eval13998.invoke(core.clj:1)
	at clojure.lang.Compiler.eval(Compiler.java:6768)
	at clojure.lang.Compiler.eval(Compiler.java:6757)
	at clojure.lang.Compiler.load(Compiler.java:7195)
	at clojure.lang.RT.loadResourceScript(RT.java:370)
	at clojure.lang.RT.loadResourceScript(RT.java:361)
	at clojure.lang.RT.load(RT.java:440)
	at clojure.lang.RT.load(RT.java:411)
	at clojure.core$load$fn__5424.invoke(core.clj:5848)
	at clojure.core$load.doInvoke(core.clj:5847)
	at clojure.lang.RestFn.invoke(RestFn.java:408)
	at clojure.core$load_one.invoke(core.clj:5653)
	at clojure.core$load_lib$fn__5373.invoke(core.clj:5693)
	at clojure.core$load_lib.doInvoke(core.clj:5692)
	at clojure.lang.RestFn.applyTo(RestFn.java:142)
	at clojure.core$apply.invoke(core.clj:628)
	at clojure.core$load_libs.doInvoke(core.clj:5731)
	at clojure.lang.RestFn.applyTo(RestFn.java:137)
	at clojure.core$apply.invoke(core.clj:628)
	at clojure.core$require.doInvoke(core.clj:5814)
	at clojure.lang.RestFn.invoke(RestFn.java:408)
	at user.fus_threading$eval13994.invoke(fus_threading.clj:6)
	at clojure.lang.Compiler.eval(Compiler.java:6768)
	at clojure.lang.Compiler.load(Compiler.java:7195)
	at clojure.lang.RT.loadResourceScript(RT.java:370)
	at clojure.lang.RT.loadResourceScript(RT.java:361)
	at clojure.lang.RT.load(RT.java:440)
	at clojure.lang.RT.load(RT.java:411)
	at clojure.core$load$fn__5424.invoke(core.clj:5848)
	at clojure.core$load.doInvoke(core.clj:5847)
	at clojure.lang.RestFn.invoke(RestFn.java:408)
	at clojure.core$load_one.invoke(core.clj:5653)
	at clojure.core$load_lib$fn__5373.invoke(core.clj:5693)
	at clojure.core$load_lib.doInvoke(core.clj:5692)
	at clojure.lang.RestFn.applyTo(RestFn.java:142)
	at clojure.core$apply.invoke(core.clj:628)
	at clojure.core$load_libs.doInvoke(core.clj:5731)
	at clojure.lang.RestFn.applyTo(RestFn.java:137)
	at clojure.core$apply.invoke(core.clj:628)
	at clojure.core$require.doInvoke(core.clj:5814)
	at clojure.lang.RestFn.invoke(RestFn.java:421)
	at midje.repl$load_facts$fn__6148.invoke(repl.clj:206)
	at midje.repl$load_facts.doInvoke(repl.clj:192)
	at clojure.lang.RestFn.invoke(RestFn.java:397)
	at user$eval6211.invoke(form-init7109545842773565024.clj:1)
	at clojure.lang.Compiler.eval(Compiler.java:6768)
	at clojure.lang.Compiler.eval(Compiler.java:6758)
	at clojure.lang.Compiler.load(Compiler.java:7195)
	at clojure.lang.Compiler.loadFile(Compiler.java:7151)
	at clojure.main$load_script.invoke(main.clj:274)
	at clojure.main$init_opt.invoke(main.clj:279)
	at clojure.main$initialize.invoke(main.clj:307)
	at clojure.main$null_opt.invoke(main.clj:342)
	at clojure.main$main.doInvoke(main.clj:420)
	at clojure.lang.RestFn.invoke(RestFn.java:421)
	at clojure.lang.Var.invoke(Var.java:383)
	at clojure.lang.AFn.applyToHelper(AFn.java:156)
	at clojure.lang.Var.applyTo(Var.java:700)
	at clojure.main.main(main.java:37)
Caused by: java.lang.NullPointerException
	at clojure.lang.Compiler.resolveIn(Compiler.java:6971)
	at clojure.lang.Compiler.resolve(Compiler.java:6949)
	at clojure.lang.Compiler$NewInstanceExpr.build(Compiler.java:7565)
	at clojure.lang.Compiler$NewInstanceExpr$DeftypeParser.parse(Compiler.java:7490)
	at clojure.lang.Compiler.analyzeSeq(Compiler.java:6709)
	... 91 more
Comment by Nicola Mometto [ 11/Jan/15 12:32 PM ]

Further debugging is convincing me further that some of the recent changes around `vec` are causing this bug.

With the following patch:

diff --git a/src/clj/clojure/core_deftype.clj b/src/clj/clojure/core_deftype.clj
index 97e14cc..9478b04 100644
--- a/src/clj/clojure/core_deftype.clj
+++ b/src/clj/clojure/core_deftype.clj
@@ -53,13 +53,16 @@ (defn- parse-impls [specs]
 (defn- parse-opts+specs [opts+specs]
   (let [[opts specs] (parse-opts opts+specs)
         impls (parse-impls specs)
-        interfaces (-> (map #(if (var? (resolve %)) 
-                               (:on (deref (resolve %)))
-                               %)
-                            (keys impls))
-                       set
-                       (disj 'Object 'java.lang.Object)
-                       vec)
+        ks (keys impls)
+        interfaces' (map #(if (var? (resolve %))
+                            (:on (deref (resolve %)))
+                            %)
+                         ks)
+        interfaces'' (set interfaces')
+        interfaces''' (disj interfaces'' 'Object 'java.lang.Object)
+        interfaces (vec interfaces''')
+        _ (when (nil? opts+specs)
+            (println impls ks interfaces' interfaces'' interfaces''' interfaces))
         methods (map (fn [[name params & body]]
                        (cons name (maybe-destructured params body)))
                      (apply concat (vals impls)))]

I get this println when the NPE occurs:

{} nil () #{} #{} [nil]

Meaning that for some reson, `vec` of `#{}` returns `[nil]` rather than `[]`

Comment by Nicola Mometto [ 11/Jan/15 12:42 PM ]

I confirmed that the bug is in the vec function.
With the following patch, when the NPE occurs, the debug println is triggered:

diff --git a/src/jvm/clojure/lang/PersistentVector.java b/src/jvm/clojure/lang/PersistentVector.java
index 9804a0b..a460b6f 100644
--- a/src/jvm/clojure/lang/PersistentVector.java
+++ b/src/jvm/clojure/lang/PersistentVector.java
@@ -96,19 +96,22 @@ static public PersistentVector create(ArrayList list){
 static public PersistentVector create(Iterable items){
     // optimize common case
     if(items instanceof ArrayList)
         return create((ArrayList)items);

     Iterator iter = items.iterator();
     TransientVector ret = EMPTY.asTransient();
     while(iter.hasNext())
         ret = ret.conj(iter.next());
-    return ret.persistent();
+    PersistentVector r = ret.persistent();
+    if (RT.seq(r) != null && RT.seq(items) == null)
+        System.out.println("bug");
+    return r;
 }

 static public PersistentVector create(Object... items){
        TransientVector ret = EMPTY.asTransient();
        for(Object item : items)
                ret = ret.conj(item);
        return ret.persistent();
 }
Comment by Alex Miller [ 11/Jan/15 1:47 PM ]

And items is a PersistentSet? I've actually been looking at some weirdness on set iterators in the context of CLJ-1499 in consistency between seq and iterators.

Comment by Nicola Mometto [ 11/Jan/15 1:52 PM ]

I believe I have identified the bug, but I cannot make any sense out of it.

The bug apperas to be in SeqIterator.hasNext(), when the NPE occurs, after applying the following patch:

diff --git a/src/jvm/clojure/lang/SeqIterator.java b/src/jvm/clojure/lang/SeqIterator.java
index e6ad481..031fbc8 100644
--- a/src/jvm/clojure/lang/SeqIterator.java
+++ b/src/jvm/clojure/lang/SeqIterator.java
@@ -35,14 +35,18 @@ public SeqIterator(ISeq o){
 public boolean hasNext(){
        if(seq == this){
                seq = START;
                next = RT.seq(next);
                }
        else if(seq == next)
                next = RT.next(seq);
+    else if (RT.seq(next) == null)
+        System.out.println("this shouldn't happen: " + (this == seq));
+    if (RT.seq(next) == null && next != null)
+        System.out.println("bug: " + next);
        return next != null;
 }

 public Object next() throws NoSuchElementException {
        if(!hasNext())
                throw new NoSuchElementException();
        seq = next;

I get the following output:

this shouldn't happen: true
bug: #{}

I have absolutely no idea how it is possible that the last branch gets executed since it is true that seq == this thus the first branch should have been executed.

Comment by Nicola Mometto [ 11/Jan/15 1:58 PM ]

I don't know why, but with the attached patch the bug seems to go away.
This is probably just by accident though as I have no idea what changes between the code pre patch and the code post patch.

Comment by Alex Miller [ 11/Jan/15 3:21 PM ]

Without looking at the patch Id say that non deterministic bug plus impossible state smells like a concurrency / race condition problem.

Comment by Michael Blume [ 11/Jan/15 4:18 PM ]

This isn't the bug, per se, the thing I'm describing should not break anything, but why is the PersistentVector(Iterable) constructor being called on a PersistentHashSet? It looks like we could very easily do

--- i/src/jvm/clojure/lang/LazilyPersistentVector.java
+++ w/src/jvm/clojure/lang/LazilyPersistentVector.java
@@ -26,7 +26,7 @@ static public IPersistentVector createOwning(Object... items){
 static public IPersistentVector create(Object obj){
    if(obj instanceof IReduceInit)
        return PersistentVector.create((IReduceInit) obj);
-   else if(obj instanceof ISeq)
+   else if(obj instanceof Seqable)
        return PersistentVector.create(RT.seq(obj));
    else if(obj instanceof Iterable)
        return PersistentVector.create((Iterable)obj);

and treat the set directly as a seq. Is there some way that would be slower?

Comment by Michael Blume [ 11/Jan/15 4:45 PM ]

It may be that I've just tried it an insufficient number of times, but simply adding 'synchronized' to SeqIterator.hasNext appears to solve the problem. Again, this doesn't really tell us what the problem is.

ETA: Nope, fails sometimes even with synchronized.

Comment by Alex Miller [ 12/Jan/15 8:57 AM ]

The use of ISeq and not Seqable in LazilyPersistentVector is quite intentional. The idea here is that if something is already a seq (effectively a linked list), the best we're going to do is walk that chain. However, many things are Seqable that may have more efficient Iterable implementations which can (statefully) walk a data structure without creating all the intermediate objects required by seq. In particular, via CLJ-1499, map and set will soon be gaining direct Iterable implementations that walk the persistent tree without instantiating a seq object for every element. However, at the moment set and map will create a SeqIterator wrapped around the seq.

CLJ-1546 changed this path - it was walking the seq but is now walking it via SeqIterator. My working theory is that that switch has uncovered a latent race condition in SeqIterator that was never noticed before as the path wasn't exercised.

Note that because CLJ-1499 removes the reliance on SeqIterator, it would have avoided the bug in a different way! However, I have been seeing a number of weird things while doing dev on CLJ-1499 specifically around iterating over sets - the OO around iterator() and seq() in the APersistentSet/PersistentHashSet/PersistentTreeSet has some weird interactions.

I'm going to look a little closer at the suggested patch.

Comment by Nicola Mometto [ 12/Jan/15 9:01 AM ]

Alex, before you waste your time on my patch I want to clarify that I don't think that patch fixes the issue in any way, it's just a random change I made that happens to make the symptoms disappear on my system, I just attached it for debugging purposed.

Maybe there's a reason why the patch solves the bug or maybe it's just masking it, I still can't figure out why this apparent race condition happens.

Comment by Alex Miller [ 12/Jan/15 9:05 AM ]

Gotcha, thx.

Comment by Michael Blume [ 12/Jan/15 11:54 AM ]

Aha, makes sense, thanks =)

Comment by Andy Fingerhut [ 12/Jan/15 12:32 PM ]

I haven't dug into this, and don't have a solution, but SeqIterator's fields are not final, so there is no guarantee that the values assigned to a new instance's fields in its constructors will be visible to other threads, yes? And I believe that if those writes to the fields do eventually become visible, they need not become visible in the order that the assignments occur in the source code, but can become visible in any order.

Comment by Nicola Mometto [ 12/Jan/15 5:13 PM ]

By the way, I've been able to reproduce this bug using jdk 1.8 so it's not just with 1.7

Comment by Michael Blume [ 12/Jan/15 9:10 PM ]

SeqIterator seems to use 'this' as a sentinel value. If I replace it with an explicit 'new Object' sentinel, the problem appears to go away (~40 compilations without an NPE).

Making seq and next volatile doesn't help.

Interestingly, when I synchronize the entire SeqIterator class (both hasNext and next synchronized on this), the problem doesn't go away, so if this is a race condition, it's kind of a weird one.

I can insert a call to seq before the call to set here https://github.com/clojure/clojure/blob/clojure-1.7.0-alpha5/src/clj/clojure/core_deftype.clj#L60 and the problem doesn't go away. I can then print the result of seq before it's passed to set, and, of course, it's a nil.

So somehow, we're basically evaluating (-> nil set (disj 'Object 'java.lang.Object) vec) and getting [nil] instead of []

But when I actually evaluate that expression in a REPL (from 20 threads at once, 1M times in a row) it evaluates to empty vector every time.

So I'm confused.

Debug patch here if anyone wants to check my work: https://gist.githubusercontent.com/MichaelBlume/735c8f601210cfa1ecaf/raw/814f21e5e4abb2ca9d1d5330d0b4cc2b3a4424e6/gistfile1.txt

Comment by Alex Miller [ 12/Jan/15 9:53 PM ]

I'm with you. I'm starting to suspect that this is involved:

https://github.com/clojure/clojure/commit/43cc1854508d655e58e377f84836ba128971f90c

Comment by Andy Fingerhut [ 12/Jan/15 11:57 PM ]

Out of curiosity, I tried adding a field to the SeqIterator class that remembers the thread that constructed each instance, and then checks in the call to hasNext if the calling thread is the same as the creator thread, printing a message if they are ever different. I never saw that while building Clojure, nor running the command on the Midje project. That seems to rule out the possibility of the SeqIterator getting passed from one thread to another.

If that is always true, then I'm with Nicola: I would love an explanation of what is going on here to cause the debug print's he mentions in a comment above to print what they do when a failure occurs. It looks really, really wrong, as in wrong single-threaded program behavior.

Comment by Nicola Mometto [ 13/Jan/15 5:09 AM ]

Michael's and Andy's findings agree with what I observed. There's no multithreading involved yet somehow there's what appears to be a data race condition in SeqIterator.

Also, as Michael observed in his own testings, changing the sentinel from this to an Object instance (START) makes the issue go away, this is exactly what the patch I attached does.

I am also unable to reproduce this issue by repeatedly invoking vec on an empty set.

Comment by Nicola Mometto [ 13/Jan/15 5:15 AM ]

This just happened:

public boolean hasNext(){
    Object a = seq;
    Object b = this;
	if(seq == this){
		seq = START;
		next = RT.seq(next);
		}
	else {
        if (RT.seq(next) == null) {
            System.out.println (a);
            System.out.println (b);
        }
        if(seq == next)
            next = RT.next(seq);
    }
	return next != null;
}
clojure.lang.SeqIterator@3ddc6873
clojure.lang.SeqIterator@3ddc6873
Exception in thread "main" java.util.NoSuchElementException, compiling:(utils.clj:68:1)
Comment by Nicola Mometto [ 13/Jan/15 5:45 AM ]

I'm taking a shot in the dark but I tried reproducing this bug using -Xint (using the jvm interpreter) and I can't seem to be able to reproduce it after many runs.
As absurd as this sounds, I'm starting to think that some hotspot optimization is responsible for this nonsensical behaviour – this would explain the nondeterministic behaviour in the absence of multithreading.

Comment by Alex Miller [ 13/Jan/15 8:50 AM ]

I did some experiments myself yesterday and found many of the same things listed here - single-threaded usage with seemingly impossible results.

I am currently thinking that setting seq = this; in the constructor is unsafe publication of this. "this" is not valid until after the constructor completes, yet we have saved away a reference to it in a field.

Thus seq==this may possibly return false in hasNext() because it is comparing a partially constructed object with the fully constructed object (same object identity!). It may be that this doesn't happen until after hot spot/inlining. By turning on the inline diagnostics, I do see these SeqIterator methods as a hot spot that gets inlined at some point soon before I see the error manifest.

This is my best explanation of the results we're seeing and seems sufficient justification to change the implementation of SeqIterator to me.

Comment by Nicola Mometto [ 13/Jan/15 8:59 AM ]

From the Java Language Spec, section 15.8.3:
"The keyword this may be used only in the body of an instance method or default method, or in the body of a constructor of a class, or in an instance initializer of a class, or in the initializer of an instance variable of a class"

So I don't believe that the current usage of this is incorrect for this reason, however I agree that it's likely caused by some interaction of doing identity check with this and hot spot/inlining issue so I'm more confident that the patch I posted is a right fix for this bug.

Comment by Andy Fingerhut [ 13/Jan/15 10:30 AM ]

The only other scenario of multiple calls to next / hasNext that I could think of with a single thread is if some method call inside of hasNext causes next / hasNext to be called, nested, on the same SeqIterator instance. I did some instrumentation to check for nested calls, unlikely as that would seem from the source code, and saw a failure with no nested call to hasNext.

Comment by Alex Miller [ 13/Jan/15 4:28 PM ]

Midje stack trace (removed from original ticket, left here for safe-keeping). To reproduce:

// set your JAVA_HOME and PATH to JDK 1.7
git clone git@github.com:marick/Midje.git
cd Midje
git co e98cf87
lein with-profile 1.7 midje

Error:

Exception in thread "main" java.lang.NullPointerException, compiling:(fim_collection_diffs.clj:7:1)
	at clojure.lang.Compiler.macroexpand1(Compiler.java:6619)
	at clojure.lang.Compiler.macroexpand(Compiler.java:6677)
	at clojure.lang.Compiler.eval(Compiler.java:6751)
	at clojure.lang.Compiler.load(Compiler.java:7194)
	at clojure.lang.RT.loadResourceScript(RT.java:370)
	at clojure.lang.RT.loadResourceScript(RT.java:361)
	at clojure.lang.RT.load(RT.java:440)
	at clojure.lang.RT.load(RT.java:411)
	at clojure.core$load$fn__5436.invoke(core.clj:5863)
	at clojure.core$load.doInvoke(core.clj:5862)
	at clojure.lang.RestFn.invoke(RestFn.java:408)
	at clojure.core$load_one.invoke(core.clj:5653)
	at clojure.core$load_lib$fn__5383.invoke(core.clj:5708)
	at clojure.core$load_lib.doInvoke(core.clj:5707)
	at clojure.lang.RestFn.applyTo(RestFn.java:142)
	at clojure.core$apply.invoke(core.clj:628)
	at clojure.core$load_libs.doInvoke(core.clj:5746)
	at clojure.lang.RestFn.applyTo(RestFn.java:137)
	at clojure.core$apply.invoke(core.clj:628)
	at clojure.core$require.doInvoke(core.clj:5829)
	at clojure.lang.RestFn.invoke(RestFn.java:421)
	at midje.repl$load_facts$fn__6148.invoke(repl.clj:206)
	at midje.repl$load_facts.doInvoke(repl.clj:192)
	at clojure.lang.RestFn.invoke(RestFn.java:397)
	at user$eval6211.invoke(form-init3965655274254111851.clj:1)
	at clojure.lang.Compiler.eval(Compiler.java:6767)
	at clojure.lang.Compiler.eval(Compiler.java:6757)
	at clojure.lang.Compiler.load(Compiler.java:7194)
	at clojure.lang.Compiler.loadFile(Compiler.java:7150)
	at clojure.main$load_script.invoke(main.clj:274)
	at clojure.main$init_opt.invoke(main.clj:279)
	at clojure.main$initialize.invoke(main.clj:307)
	at clojure.main$null_opt.invoke(main.clj:342)
	at clojure.main$main.doInvoke(main.clj:420)
	at clojure.lang.RestFn.invoke(RestFn.java:421)
	at clojure.lang.Var.invoke(Var.java:383)
	at clojure.lang.AFn.applyToHelper(AFn.java:156)
	at clojure.lang.Var.applyTo(Var.java:700)
	at clojure.main.main(main.java:37)
Caused by: java.lang.NullPointerException
	at clojure.lang.Compiler.maybeResolveIn(Compiler.java:7014)
	at clojure.core$ns_resolve.invoke(core.clj:4200)
	at clojure.core$ns_resolve.invoke(core.clj:4197)
	at clojure.core$resolve.invoke(core.clj:4206)
	at clojure.core$map$fn__4529.invoke(core.clj:2612)
	at clojure.lang.LazySeq.sval(LazySeq.java:40)
	at clojure.lang.LazySeq.seq(LazySeq.java:49)
	at clojure.lang.RT.seq(RT.java:485)
	at clojure.core$seq__4109.invoke(core.clj:135)
	at clojure.core$reduce1.invoke(core.clj:899)
	at clojure.core$set.invoke(core.clj:3944)
	at clojure.core$emit_defrecord.invoke(core_deftype.clj:154)
	at clojure.core$defrecord.doInvoke(core_deftype.clj:374)
	at clojure.lang.RestFn.invoke(RestFn.java:470)
	at clojure.lang.Var.invoke(Var.java:394)
	at clojure.lang.AFn.applyToHelper(AFn.java:165)
	at clojure.lang.Var.applyTo(Var.java:700)
	at clojure.lang.Compiler.macroexpand1(Compiler.java:6606)
Comment by Ghadi Shayban [ 13/Jan/15 4:32 PM ]

Not sure I fully understand the sad path that causes this bug, but START can safely be marked static final in the patch.

Comment by Andy Fingerhut [ 13/Jan/15 4:54 PM ]

Does this seem like it may be a bug in JIT compilation to anyone? I ask because as far as we can tell, this bug occurs completely within a single thread, and as far as I've read, the Java memory model should guarantee that operations in a single thread appear to occur in the order they happen in the source code.

Independent question: It seems that the assumption is that a SeqIterator is only ever accessed from 1 thread. Is it considered an internal implementation detail, and thus on documentation is needed about this assumption?

Comment by Alex Miller [ 13/Jan/15 5:22 PM ]

Re JIT - I wouldn't rule that out, but if it is, that doesn't help us make Clojure work again for everyone with existing JVMs. JCiP section 3.2.1 says "If the this reference escapes during construction, the object is considered not properly constructed." which sounds like what we're doing.

Re threading - I think that the use of iterators inside Clojure itself has (until recently) been unusual. Virtually everything is written to leverage the seq model and iterators were largely provided for Java compliance. With the creation of LazyTransformer and extension of reduce to iterators, this orientation has changed somewhat. However, I'd say that iterators are dirty stateful things and they should be consumed in localized contexts by no more than one thread at a time. If they are used in a thread-confined way and safely published between threads, SeqIterator seems ok.

Comment by Nicola Mometto [ 13/Jan/15 5:32 PM ]

0001-CLJ-1636-don-t-use-this-as-a-sentinel-in-SeqIterator.patch is the same as 0001-fix-for-CLJ-1636.patch except it makes START final as suggested by Ghadi

Comment by Ghadi Shayban [ 15/Jan/15 12:27 PM ]

Alex has screened this – which probably implies that the patch fixes the issue. Just to confirm does the patch clear up the issue for everyone else?

Comment by Michael Blume [ 15/Jan/15 12:48 PM ]

Does for me, yes =)

Comment by Andy Fingerhut [ 15/Jan/15 1:17 PM ]

If you want to collect test results, seems like it would be good for people to respond with the OS version and JVM version they tested on, and whether it was the Midje test in the description of this ticket that they tried, or some other test.

Comment by Andy Fingerhut [ 16/Jan/15 4:59 PM ]

Out of curiosity, does anyone have a smaller test case that causes this incorrect return value from the SeqIterator, without running the 'lein with-profile 1.7 midje' command on Midje? e.g. running a page or less worth of Clojure code?

Comment by Alex Miller [ 16/Jan/15 6:01 PM ]

While we've applied the patch, I would still love to understand wtf is happening here and would love to see that too. For me, I can quite reliably reproduce it building Clojure itself. To support the theory of it happening during an inlining transition, it's unlikely to reproduce outside the context of other code however. I see it get embedded inside a big wad of calling code when I watching inlining.

Comment by Nicola Mometto [ 18/Jan/15 11:36 AM ]

I've spent some time reading through both the jvm and the java specs and I can't find a reasonable explaination for what was happening, I can only think this is a bug in some hotspot inlining optimization.





[CLJ-1635] Make distinct/dedupe/interpose transducer tests play nicely with new reporting Created: 09/Jan/15  Updated: 20/Feb/15  Resolved: 20/Feb/15

Status: Closed
Project: Clojure
Component/s: None
Affects Version/s: None
Fix Version/s: Release 1.7

Type: Enhancement Priority: Minor
Reporter: Michael Blume Assignee: Unassigned
Resolution: Completed Votes: 0
Labels: test, transducers

Attachments: Text File clj-1635.patch    
Patch: Code and Test
Approval: Ok

 Description   

Fix interaction between CLJ-1601 (which introduced new transducers) and CLJ-1621 (which improved transducer tests) to improve test reporting for these new transducer arities as well.

Note from Alex M: I goofed these while rebasing CLJ-1601 after CLJ-1621 went in.

Patch: clj-1635.patch

Screened by: Alex Miller



 Comments   
Comment by Alex Miller [ 09/Jan/15 2:31 PM ]

My fault in the integration process! We'll try to get it fixed in 1.7. Thanks...





[CLJ-1634] Potential bug in trampoline Created: 07/Jan/15  Updated: 07/Jan/15  Resolved: 07/Jan/15

Status: Closed
Project: Clojure
Component/s: None
Affects Version/s: Release 1.6
Fix Version/s: None

Type: Defect Priority: Major
Reporter: Hongseok Yang Assignee: Unassigned
Resolution: Duplicate Votes: 0
Labels: bug, trampoline
Environment:

Java HotSpot(TM) 64-Bit Server VM 1.8.0_25-b17



 Description   

Hi,

While trying to use trampoline to optimise tail recursion in my Clojure project, I came across some strange behaviour of the trampoline function.

=> (defn f [g] (fn [k & args] #(k (apply g args))))
...
=> (trampoline (f list) println 1 2 3)
(#<core$println clojure.core$println@54e517f6> 1 2 3)
nil
=> (((f list) println 1 2 3))
(1 2 3)
nil

I think that (trampoline (f list) ...) and ((f list) ...) should give the same result.

In fact, I asked this question in Stackoverflow before. You can find some further discussion about this potential bug.

https://stackoverflow.com/questions/27819418/strange-behaviour-of-clojure-trampoline

Best wishes,
Hongseok



 Comments   
Comment by Nicola Mometto [ 07/Jan/15 11:48 AM ]

I opened CLJ-1633 just a few moments before you with a patch fixing this issue.





[CLJ-1633] PersistentList/creator doesn't handle ArraySeqs correctly Created: 07/Jan/15  Updated: 10/Jan/15  Resolved: 10/Jan/15

Status: Closed
Project: Clojure
Component/s: None
Affects Version/s: None
Fix Version/s: Release 1.7

Type: Defect Priority: Critical
Reporter: Nicola Mometto Assignee: Unassigned
Resolution: Completed Votes: 1
Labels: collections

Attachments: Text File 0001-CLJ-1633-fix-PersistentList-creator-handling-of-Arra.patch     Text File CLJ-1633-v2.patch     Text File CLJ-1633-v3.patch     Text File generative-seq-tests.patch     Text File generative-seq-tests-v2.patch    
Patch: Code and Test
Approval: Ok

 Description   

This should return '(2 3) but returns '(1 2 3) instead:

user=> ((fn [& args] (apply (fn [a & b] (println a b) (apply list b)) args)) 1 2 3)
1 (2 3)
(1 2 3)

Note that using vector rather than list returns the correct values:

user=> ((fn [& args] (apply (fn [a & b] (println a b) (apply vector b)) args)) 1 2 3)
1 (2 3)
[2 3]

The bug was reported in this stackoverflow question: https://stackoverflow.com/questions/27819418/strange-behaviour-of-clojure-trampoline and the bug identified in this comment: https://stackoverflow.com/questions/27819418/strange-behaviour-of-clojure-trampoline#comments-27821793

A simpler example of this bug:

user=> (apply list (next (clojure.lang.ArraySeq/create (object-array [1 2 3]))))
(1 2 3)

Patch: CLJ-1633-v3.patch



 Comments   
Comment by Michael Blume [ 07/Jan/15 12:28 PM ]

Very nice catch.

This makes me wonder if there's more we can do with generative testing to catch this class of bugs, maybe along the lines of zach tellman's collection-check

Comment by Alex Miller [ 07/Jan/15 2:27 PM ]

There's definitely more we can do. collection-check is great and I've started to integrate some of those ideas into Clojure's tests (see for example the new transducer tests that generate random chains of sequence operations and compare seq and transducer executions). If you have ideas about specific test areas, would be happy to see a jira/patch.

Comment by Michael Blume [ 07/Jan/15 4:59 PM ]

Updated test to get expected list inside the (= ...) form

Comment by Michael Blume [ 07/Jan/15 5:03 PM ]

Oops, rerolling again to apply cleanly to master

Comment by Nicola Mometto [ 07/Jan/15 5:04 PM ]

Thanks for the fix!

Comment by Michael Blume [ 07/Jan/15 5:40 PM ]

No problem =)

Comment by Michael Blume [ 07/Jan/15 5:42 PM ]

Ok, this is kind of crude, and mostly a proof of concept, but this does catch the bug.

Output:

[java] FAIL in (seq-gentest) (sequences.clj:105)
     [java] {:acts1 (-> [] next (->> (cons :foo)) (->> (cons :foo)) next),
     [java]  :acts2
     [java]  (->
     [java]   []
     [java]   next
     [java]   (->> (cons :foo))
     [java]   (->> (cons :foo))
     [java]   into-array-seq
     [java]   next
     [java]   (->> (apply list))),
     [java]  :result1 (:foo),
     [java]  :result2 (:foo :foo),
     [java]  :pass false}
     [java]
     [java] expected: (:result res)
     [java]   actual: false
Comment by Alex Miller [ 09/Jan/15 9:12 AM ]

Rich said to move this forward.

Comment by Nicola Mometto [ 09/Jan/15 5:51 PM ]

This ticket has been closed but no patch has been committed

Comment by Alex Miller [ 09/Jan/15 6:22 PM ]

Stu, doesn't look like this patch was applied but it was closed?





[CLJ-1632] Mark Clojure-generated classes in order for instrumenters to identify them easily Created: 05/Jan/15  Updated: 05/Jan/15  Resolved: 05/Jan/15

Status: Closed
Project: Clojure
Component/s: None
Affects Version/s: Release 1.6
Fix Version/s: None

Type: Enhancement Priority: Minor
Reporter: Fabio Tudone Assignee: Unassigned
Resolution: Declined Votes: 0
Labels: compiler


 Description   

Instrumenting logic specific to Clojure-generated classes should be able to identify them easily.



 Comments   
Comment by Fabio Tudone [ 05/Jan/15 2:23 AM ]

One way could be annotations, another could be interface-marking. Would that be feasible? Any drawbacks?

Comment by Alex Miller [ 05/Jan/15 8:17 AM ]

deftypes have a marker interface clojure.lang.IType.
defrecords have a marker interface clojure.lang.IRecord.
proxy classes have marker interface clojure.lang.IProxy.

I think generic markers for protocols or gen-interface would be undesirable as they may be used to create APIs for external use.

Comment by Fabio Tudone [ 05/Jan/15 8:29 AM ]

Not sure I understand your point about the non-marked (as of now and AFAIK) Clojure features such as protocols and gen-interface; could you elaborate? Does it apply to marking with annotations as well?

Comment by Alex Miller [ 05/Jan/15 10:50 AM ]

My point was that many people wish to generate interfaces that do not extend from interfaces in Clojure and adding those marker interfaces would be seen as a downside for them. Annotations are slightly better but have the same problem (dependencies on parts of Clojure core). You are of course free to add those interfaces or annotations yourself in your own code if that's useful to you!

Comment by Fabio Tudone [ 05/Jan/15 11:15 AM ]

It's clear now, thanks!

Actually my use case is about general tooling that will inspect and instrument all (and only) Clojure-generated code in any application making use of it, so I don't control the code I'm going to examine. This is done in order to add specific runtime features in a general fashion.

I can't find a way to do that reliably on everything that has been generated by Clojure; I could use some imperfect heuristics but I'd rather use a reliable way if one exists. Can you see of any other way of doing this I might have overlooked? Or is some other enhancement possible that would allow me to do this and would not compromise external integrations?

Comment by Alex Miller [ 05/Jan/15 11:31 AM ]

I don't see a general-purpose way to do this now. I do not believe supporting this is something we would spend time on.

Comment by Julien Eluard [ 05/Jan/15 2:03 PM ]

Class#isSynthetic() might be relevant here. If I am not mistaken asm generated classes will be flagged as synthetic.





[CLJ-1631] Issue with type hints on return values Created: 01/Jan/15  Updated: 01/Jan/15  Resolved: 01/Jan/15

Status: Closed
Project: Clojure
Component/s: None
Affects Version/s: Release 1.7
Fix Version/s: None

Type: Defect Priority: Major
Reporter: Mike Anderson Assignee: Unassigned
Resolution: Duplicate Votes: 0
Labels: None


 Description   

When a return value is type hinted in one namespace, errors can occur when the function is used from another namespace. Simple code to reproduce:

(ns foo (:import [java.awt.image BufferedImage]))

(defn f ^BufferedImage []
(BufferedImage. (int 10) (int 10) BufferedImage/TYPE_INT_ARGB))

(ns bar (:require foo))

(.getType (foo/f))
=> CompilerException java.lang.IllegalArgumentException: Unable to resolve classname: BufferedImage, compiling:(NO_SOURCE_PATH:1:1)

I would expect any usage of foo/f to correctly use the fully qualified class name (java.awt.image.BufferedImage). Note that if I add an extra type hint at the call site, it seems to work OK:

(.getType ^java.awt.image.BufferedImage (foo/f))
=> 2



 Comments   
Comment by Nicola Mometto [ 01/Jan/15 4:32 AM ]

Duplicate of http://dev.clojure.org/jira/browse/CLJ-1232





[CLJ-1627] Incorrect error message: "First argument to def must be a Symbol" Created: 27/Dec/14  Updated: 29/Dec/14  Resolved: 28/Dec/14

Status: Closed
Project: Clojure
Component/s: None
Affects Version/s: None
Fix Version/s: None

Type: Defect Priority: Trivial
Reporter: Richard Davies Assignee: Unassigned
Resolution: Declined Votes: 0
Labels: None
Environment:

All



 Description   

(def (symbol "x")) throws the exception: First argument to def must be a Symbol

Given the first argument to this def is a symbol, the error message is incorrect.

This has resulted in:

See http://stackoverflow.com/questions/2486752/in-clojure-how-to-define-a-variable-named-by-a-string

Which so far has >3000 views suggesting that this is something that regularly stumps people the first time they encounter it.

I suggest changing the error message to:

"First argument to def must be an interned Symbol"



 Comments   
Comment by Nicola Mometto [ 28/Dec/14 4:00 AM ]

The first argument to def is not a symbol, it's the list (symbol "x").
def is neither a macro nor a regular function, it's a special form whose first argument is not evaluated. This is documented.

Comment by Alex Miller [ 28/Dec/14 10:39 AM ]

The current error message is literally correct. I don't see how the requested change would help anyone confused by the prior message.

Comment by Richard Davies [ 29/Dec/14 8:46 PM ]

@Ah. It's not explicit in the docs (at least in http://clojure.org/special_forms) or the error message that the first arg isn't evaluated. The docs talk about the "init?" paramater and symbol metadata being evaluated but not that the first argument itself is not actually evaluated. This is probably obvious to those familiar with the workings of the Clojure compiler but as hacking around with vars isn't something I do every day it wasn't to me.

Compare def's error message with trying to do the same thing with var:

(var (symbol "a"))

clojure.lang.Compiler$CompilerException: java.lang.ClassCastException: clojure.lang.PersistentList cannot be cast to clojure.lang.Symbol

At least that gives me a hint that the expression wasn't evaluated rather than just a wtf moment...





[CLJ-1621] Improve reporting in transducers generative test. Created: 21/Dec/14  Updated: 09/Jan/15  Resolved: 09/Jan/15

Status: Closed
Project: Clojure
Component/s: None
Affects Version/s: Release 1.7
Fix Version/s: Release 1.7

Type: Enhancement Priority: Minor
Reporter: Michael Blume Assignee: Unassigned
Resolution: Completed Votes: 0
Labels: transducers

Attachments: Text File transducer-reporting-v1.patch    
Patch: Code and Test
Approval: Ok

 Description   

If the transducers generative test breaks, you get output like this:

[java] {:test-var seq-and-transducer, :result #<ExceptionInfo clojure.lang.ExceptionInfo: Applied actions to coll as seq, sequence transducer, and into transducer and got different results. {:coll [-16 10 -8 8 -5], :actions map clojure.core$dec@782a4056,take 5,partition-by clojure.core$even_QMARK_@2200d281,partition-all 9,map clojure.core$inc@643b798d,drop 9,remove clojure.core$empty_QMARK_@4600f352,remove clojure.core$odd_QMARK_@32dd05af, :s #<ClassCastException java.lang.ClassCastException: clojure.lang.LazySeq cannot be cast to java.lang.Number>, :xs (), :xi [], :xt []}>, :seed 1419199634890, :failing-size 21, :num-tests 22, :fail [[-16 10 -8 8 -5] [{:desc map clojure.core$dec@782a4056, :xf #<core$map$fn__3669 clojure.core$map$fn__3669@28449652>, :seq #<core$partial$fn__3652 clojure.core$partial$fn__3652@506b8505>} {:desc take 5, :xf #<core$take$fn__3712 clojure.core$take$fn__3712@38934406>, :seq #<core$partial$fn__3652 clojure.core$partial$fn__3652@27ce0b6d>} {:desc partition-by clojure.core$even_QMARK_@2200d281, :xf #<core$partition_by$fn__5568 clojure.core$partition_by$fn__5568@5287c159>, :seq #<core$partial$fn__3652 clojure.core$partial$fn__3652@70961c7b>} {:desc partition-all 9, :xf #<core$partition_all$fn__5590 clojure.core$partition_all$fn__5590@3f869b0>, :seq #<core$partial$fn__3652 clojure.core$partial$fn__3652@6f99ed9f>} {:desc map clojure.core$inc@643b798d, :xf #<core$map$fn__3669 clojure.core$map$fn__3669@2f2c41d3>, :seq #<core$partial$fn__3652 clojure.core$partial$fn__3652@2fdbef8d>} {:desc drop 9, :xf #<core$drop$fn__3728 clojure.core$drop$fn__3728@4f7b4b50>, :seq #<core$partial$fn__3652 clojure.core$partial$fn__3652@214b9b5>} {:desc remove clojure.core$empty_QMARK_@4600f352, :xf #<core$filter$fn__3696 clojure.core$filter$fn__3696@6846d654>, :seq #<core$partial$fn__3652 clojure.core$partial$fn__3652@7df231c7>} {:desc remove clojure.core$odd_QMARK_@32dd05af, :xf #<core$filter$fn__3696 clojure.core$filter$fn__3696@5a8ce6dd>, :seq #<core$partial$fn__3652 clojure.core$partial$fn__3652@34ee9000>}]], :shrunk {:total-nodes-visited 32, :depth 12, :result #<ExceptionInfo clojure.lang.ExceptionInfo: Applied actions to coll as seq, sequence transducer, and into transducer and got different results. {:coll [0], :actions map clojure.core$inc@643b798d, :s (1), :xs (0), :xi [0], :xt [0]}>, :smallest [[0] [{:desc map clojure.core$inc@643b798d, :xf #<core$map$fn__3669 clojure.core$map$fn__3669@2f2c41d3>, :seq #<core$partial$fn__3652 clojure.core$partial$fn__3652@2fdbef8d>}]]}}
     [java]
     [java] ERROR in (seq-and-transducer) (core.clj:4566)
     [java] Uncaught exception, not in assertion.
     [java] expected: nil
     [java]   actual: clojure.lang.ExceptionInfo: Applied actions to coll as seq, sequence transducer, and into transducer and got different results.
     [java]  at clojure.core$ex_info.invoke (core.clj:4566)
     [java]     clojure.test_clojure.transducers$seq_and_transducer_same_result.invoke (transducers.clj:103)
     [java]     clojure.lang.AFn.applyToHelper (AFn.java:156)
...etc etc

This has a few problems:

  • when clojure functions are given as arguments, they're full object printouts, with classnames and memory addresses, this is kind of hard to read
  • the combination of the first problem found with the shrunk version means there's a lot of content to read
  • lack of pretty printing makes that content very hard to read
  • the traceback isn't actually that helpful – we know what failed already.

Approach: The attached patch encodes more descriptive info in the actions and does a better job of reporting the difference in an understandable manner:

[java] FAIL in (seq-and-transducer) (transducers.clj:135)
     [java] {:coll [0],
     [java]  :actions (->> coll (map inc)),
     [java]  :s (1),
     [java]  :xs (0),
     [java]  :xi [0],
     [java]  :xt [0]}

Patch: transducer-reporting-v1.patch

Screened by: Alex Miller






[CLJ-1619] PersistentVector implements IReduce but the no init arity throws Created: 17/Dec/14  Updated: 10/Jan/15  Resolved: 10/Jan/15

Status: Closed
Project: Clojure
Component/s: None
Affects Version/s: Release 1.7
Fix Version/s: Release 1.7

Type: Defect Priority: Major
Reporter: Nicola Mometto Assignee: Unassigned
Resolution: Completed Votes: 0
Labels: None

Attachments: Text File 0001-CLJ-1619-Implement-no-init-arity-of-reduce-for-Persi.patch     Text File 0001-Implement-no-init-arity-of-reduce-for-PersistentVect.patch    
Patch: Code and Test
Approval: Ok

 Description   

The reduce arity of IReduce in PersistentVector is implemented as: "throw new UnsupportedOperationException()".

After the CLJ-1572 patch is applied the following code will throw:

(reduce + [1 2])

Approach taken: Implement reduce(f) in PersistentVector.

Alternative: An alternate would be to change PersistentVector from IReduce to IReduceInit and remove the reduce without init function. In this case, reducing a vector would fall back to seqs.

Patch: 0001-CLJ-1619-Implement-no-init-arity-of-reduce-for-Persi.patch

Screened by: Stu



 Comments   
Comment by Alex Miller [ 18/Dec/14 10:59 AM ]

Is that return null there right? In the case of no elements, you should invoke f with no args right?

Comment by Nicola Mometto [ 18/Dec/14 11:04 AM ]

you're right, I didn't know that detail about the behaviour of reduce. Updated the patch to invoke (f) rather than returning nil when the coll is empty

Comment by Alex Miller [ 09/Jan/15 8:14 AM ]

Needs tests

Comment by Nicola Mometto [ 09/Jan/15 8:33 AM ]

Updated patch adding testcases for PersistentVector.reduce in the already existing reduce test

Comment by Stuart Halloway [ 09/Jan/15 11:50 AM ]

I don't understand the problem here. coll-reduce appears to cut off ever hitting this path (the tests call underlying interfaces directly).

  • Need a public API example showing the failure
  • Need tests covering main branches (i.e. the empty case)
Comment by Stuart Halloway [ 09/Jan/15 11:52 AM ]

nevermind, screening 1572





[CLJ-1618] Widen set to take Iterable/IReduceInit Created: 17/Dec/14  Updated: 09/Jan/15  Resolved: 09/Jan/15

Status: Closed
Project: Clojure
Component/s: None
Affects Version/s: Release 1.7
Fix Version/s: Release 1.7

Type: Enhancement Priority: Major
Reporter: Alex Miller Assignee: Unassigned
Resolution: Completed Votes: 0
Labels: None

Attachments: Text File clj-1618.patch    
Patch: Code
Approval: Ok

 Description   

Similar to CLJ-1546 (same thing on vec), set should work on IReducibleInit or Iterable. Currently eduction will work via Iterable but through SeqIterator. set on an IReduceInit will throw an error.

user=> (set (eduction (map inc) (range 100)))  ;; works, but slower path
user=> (set (reify clojure.lang.IReduceInit  
       (reduce [_ f start]
         (reduce f start (range 10)))))
IllegalArgumentException Don't know how to create ISeq from: user$eval1198$reify__1199  clojure.lang.RT.seqFrom (RT.java:506)

Approach: Check for and use IReduceInit path if available, otherwise fallback to seq. Additionally, the patch adds a modification to return a set without it's meta (same approach as CLJ-1546) if a set is passed, which is fast constant time with no change in effective behavior.

Performance: (using Criterium quick-bench)

Timings done with either (count (set coll)) or (count (into #{} coll)):

coll 1.6.0 into 1.6.0 set 1.7.0-alpha4 set 1.7.0-alpha4+patch set
(set (range 100)) 15.4 µs 17.0 µs 11.4 µs 0.0 µs
(vec (range 1000000)) 360.7 ms 702.5 ms 391.1 ms 358.6 ms
(doall (range 1000000)) 363.6 ms 736.9 ms 387.5 ms 371.0 ms
(doall (range 5)) 404.9 ns 612.3 ns 481.9 ns 445.9 ns
(eduction (map identity) (vec (range 100))) n/a n/a 11.3 µs 8.7 µs

See also: CLJ-1546, CLJ-1384

Patch: clj-1618.patch

Screened by:






[CLJ-1616] Frequencies incompatible with eduction Created: 14/Dec/14  Updated: 14/Dec/14  Resolved: 14/Dec/14

Status: Closed
Project: Clojure
Component/s: None
Affects Version/s: None
Fix Version/s: None

Type: Defect Priority: Major
Reporter: Ghadi Shayban Assignee: Unassigned
Resolution: Not Reproducible Votes: 0
Labels: None


 Description   

Reproduction:
This needs the CLJ-1606 patch to apply, so that eduction works.

(frequencies (eduction (take 5) (range 50)))
;; ArityException Wrong number of args (1) passed to: core/frequencies/fn--6730

Cause: The reduce function that 'frequencies' calls is lacking the completing arity.

Simplest fix is to add the completing arity. Could be useful to allow frequencies to take a transducer stack.

mapv/filterv are similarly affected but seem less useful than using into with transducers.



 Comments   
Comment by Alex Miller [ 14/Dec/14 8:14 PM ]

Doesn't this work with CLJ-1572 + CLJ-1606 patches?

Comment by Ghadi Shayban [ 14/Dec/14 9:11 PM ]

No, not when there is something like 'take' in the picture. Transducers imply a reducing function with two different arities [1]. When 'frequencies' reduces over the collection (the eduction), a transducer inside the eduction might terminate early and cause the arity-1 rfn to be called, which will eventually bottom out here and throw the missing arity. [2]

CLJ-1572 helps dispatch properly
CLJ-1606 helps eduction actually work

[1] https://github.com/clojure/clojure/blob/master/src/clj/clojure/core.clj#L6520-L6521
[2] https://github.com/clojure/clojure/blob/master/src/clj/clojure/core.clj#L6859

Comment by Alex Miller [ 14/Dec/14 9:49 PM ]

The example given works for me when I have CLJ-1572 + CLJ-1606 - what am I missing?

Comment by Ghadi Shayban [ 14/Dec/14 10:42 PM ]

Sigh you're not missing anything. I have an active repl that I can reproduce this on...

But with a bare build with CLJ-1572 and CLJ-1606 applied it does not happen. Give me a little bit to track this down. Intuitively it seems correct that something trying to complete frequencies's rfn:

(fn [counts x]
             (assoc! counts x (inc (get counts x 0))))

would fail.

Comment by Ghadi Shayban [ 14/Dec/14 10:49 PM ]

I'll reopen if I can figure out what happened





[CLJ-1614] Clojure does not start: ClassCastException Created: 12/Dec/14  Updated: 05/Jun/15  Resolved: 05/Jun/15

Status: Closed
Project: Clojure
Component/s: None
Affects Version/s: Release 1.7
Fix Version/s: None

Type: Defect Priority: Minor
Reporter: Vladimir Tsichevski Assignee: Unassigned
Resolution: Declined Votes: 0
Labels: compiler
Environment:

Eclipse RCP



 Description   

The clojure.lang.Compiler class static code throws the ClassCastException when reading compiler options from System properties (Compiler.java, line 260 in the git master release). When running Clojure from Eclipse RCP application the System properties may have non-string values.

Checking if the value is String and ignoring non-strings fixes this problem.



 Comments   
Comment by Nicola Mometto [ 05/Jun/15 10:02 AM ]

I believe this is a duplicate of CLJ-1717 which was declined.





[CLJ-1608] add split-at to clojure.string Created: 03/Dec/14  Updated: 03/Dec/14  Resolved: 03/Dec/14

Status: Closed
Project: Clojure
Component/s: None
Affects Version/s: Release 1.6
Fix Version/s: None

Type: Enhancement Priority: Minor
Reporter: Dmitr Sotnikov Assignee: Unassigned
Resolution: Declined Votes: 0
Labels: string

Attachments: Text File string.clj.patch    
Patch: Code

 Description   

Add clojure.string/split-at similar to clojure.core/split-at that accepts a string and a number indicating the position where the string should be split. The function returns a vector containing two strings, first containing the characters from 0-n-1, and second n-length.



 Comments   
Comment by Alex Miller [ 03/Dec/14 9:48 PM ]

I do not think this is an operation that is fundamental (it can be easily composed from existing functions like count and subs) or represents a portability opportunity by being a function available on jvm and js with host performance benefits. It is a non-goal for clojure.string to contain every potentially useful string function.

Comment by Dmitr Sotnikov [ 03/Dec/14 11:04 PM ]

Makes sense, thanks for the clarification.





[CLJ-1606] Transducing an eduction finishes twice Created: 27/Nov/14  Updated: 09/Jan/15  Resolved: 09/Jan/15

Status: Closed
Project: Clojure
Component/s: None
Affects Version/s: Release 1.7
Fix Version/s: Release 1.7

Type: Defect Priority: Major
Reporter: Herwig Hochleitner Assignee: Unassigned
Resolution: Completed Votes: 0
Labels: transducers
Environment:

1.7.0-alpha4


Attachments: Text File CLJ-1606-2.patch     Text File CLJ-1606-3.patch     Text File CLJ-1606-4.patch     Text File CLJ-1606-5.patch     Text File CLJ-1606.patch    
Patch: Code and Test
Approval: Ok

 Description   
> (transduce (map identity)
             (fn
               ([s] (println "Finishing") s)
               ([s i] s))
             nil
             (eduction (map identity) []))
Finishing
Finishing
nil

Cause: transduce passes (xf f) into .reduce of Eduction, which calls transduce, causing completing xf to be called more than once.

Proposed: Eduction reduce should use (completing f) instead of f to isolate completion of inner xf from outer xf.

Patch: CLJ-1606-5.patch

Screened by: Alex Miller



 Comments   
Comment by Alex Miller [ 27/Nov/14 11:01 PM ]

identity is not a valid xf - changed to (map identity)

Comment by Ghadi Shayban [ 27/Nov/14 11:34 PM ]

identity is a valid though nonsensical transducer. fix & test added.

Comment by Ghadi Shayban [ 28/Nov/14 12:06 AM ]

Simple reproduction similar to into:

(transduce (map dec)
           (completing conj! persistent!)
           (transient [])
           (eduction (map inc) (range 6)))

;; ClassCastException clojure.lang.PersistentVector cannot be cast to clojure.lang.ITransientCollection

into doesn't use completing, and conj! has an arity that hides the problem.

Comment by Alex Miller [ 28/Nov/14 8:54 AM ]

I removed trailing whitespace in the patch so it applies cleanly.

Comment by Ghadi Shayban [ 14/Dec/14 11:16 PM ]

This patch is a little more subtle than I thought. Completion of the eduction's rfn needs to be handled separately from the "outer" transduce's xform. Patch coming.

Comment by Ghadi Shayban [ 14/Dec/14 11:32 PM ]

New patch with tests that completes the inner xform without completing the passed in rfn

Comment by Ghadi Shayban [ 15/Dec/14 1:19 AM ]

both -3 and -2 are equivalent. -3 is probably better stylistically.

Comment by Alex Miller [ 15/Dec/14 8:37 AM ]

Added CLJ-1606-4.patch - identical to -3, just fixed whitespace error.

Comment by Andy Fingerhut [ 08/Jan/15 6:09 PM ]

There are two identically named attachments here (containing -2). It looks like it isn't the one under consideration, but it might be nice to remove or rename to avoid the name conflict.

Comment by Ghadi Shayban [ 08/Jan/15 6:24 PM ]

Andy, not sure how to do that, but in any case I just added -5 clarifying language in the comment

Comment by Alex Miller [ 08/Jan/15 6:41 PM ]

Ghadi, that was super confusing. Did you just add a new -5 patch? The -4 patch has already been screened, and you have not removed the duplicate -2 patch so I don't get what the -5 is. Can we just delete the -5 and older -2 patches?

Comment by Andy Fingerhut [ 08/Jan/15 6:44 PM ]

Sorry for adding to the confusion. Ghadi, instructions for deleting patches are in the "Removing patches" section on this wiki page: http://dev.clojure.org/display/community/Developing+Patches

Comment by Ghadi Shayban [ 08/Jan/15 6:50 PM ]

Sorry. Fine by me, though permissions prevent me from deleting one of the patches.

As I read through the screened patch I just tried to clarify the wording. This:

;; NB (completing f) isolates completion of inner xfns from outer xfns
became:
;; NB (completing f) isolates completion of inner rf from outer rf

Feel free to nix that -5 patch if that's worthless

Comment by Alex Miller [ 08/Jan/15 7:12 PM ]

Gotcha. I will take care of the further changes later tonight.

In the future, please don't modify screened patches without letting me know.





[CLJ-1605] Unexpected additional digits are appeared after RuntimeException in repl. Created: 26/Nov/14  Updated: 27/Nov/14  Resolved: 26/Nov/14

Status: Closed
Project: Clojure
Component/s: None
Affects Version/s: Release 1.6
Fix Version/s: None

Type: Defect Priority: Trivial
Reporter: Constantine Potapov Assignee: Unassigned
Resolution: Not Reproducible Votes: 0
Labels: bug
Environment:

$uname -a
Linux um 3.13.0-24-generic #47-Ubuntu SMP Fri May 2 23:30:00 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux

$ java -version
java version "1.7.0_25"
Java(TM) SE Runtime Environment (build 1.7.0_25-b15)
Java HotSpot(TM) 64-Bit Server VM (build 23.25-b01, mixed mode)



 Description   

1) run repl
lein repl

  • evaluate the following value
    ( + (/ 2 3) (/ 3 4 ) )
    17/12
    • result is correct

2) evaluate it with an error, the space after "/" was deleted (/2 3)
user=> ( + (/2 3) (/ 3 4 ))

RuntimeException Invalid token: /2 clojure.lang.Util.runtimeException (Util.java:221)
RuntimeException Unmatched delimiter: ) clojure.lang.Util.runtimeException (Util.java:221)
RuntimeException Unmatched delimiter: ) clojure.lang.Util.runtimeException (Util.java:221)

  • try to evaluate it again
    user=> ( + (/ 2 3) (/ 3 4 ) )
    33/417/12
    • result is incorrect
      actual value is: "33/417/12"
      the value was expected: "17/12"
  • if it runs second time, result is correct again
    user=> ( + (/ 2 3) (/ 3 4 ) )
    17/12


 Comments   
Comment by Nicola Mometto [ 26/Nov/14 6:32 AM ]

The problem is that `/2` is not a valid clojure expression, an error is thrown and 3 is returned, then (/ 3 4) is evaluated and 3/4 is returned.
It looks there's an issue with lein repl that causes "3", "3/4" to be printed after the next expression is evaluated, that is "( + (/ 2 3) (/ 3 4 ) )" which returns 17/12.

This is why you get the apparently wrong result "33/417/12", when in fact it is just printing 3 results in the same line: 3 3/4 17/12

In short, this is not a bug in clojure, it's a bug in how lein repl prints the result. you should open a ticket in the lein issue tracker https://github.com/technomancy/leiningen

Comment by Nicola Mometto [ 26/Nov/14 6:33 AM ]

For instance, here's how a repl run using java -jar clojure.jar prints it:

user=> ( + (/2 3) (/ 3 4 )) 
RuntimeException Invalid token: /2  clojure.lang.Util.runtimeException (Util.java:221)
3
RuntimeException Unmatched delimiter: )  clojure.lang.Util.runtimeException (Util.java:221)
3/4
RuntimeException Unmatched delimiter: )  clojure.lang.Util.runtimeException (Util.java:221)
user=> ( + (/ 2 3) (/ 3 4 ) )
17/12
Comment by Alex Miller [ 26/Nov/14 11:39 AM ]

see Nicola's comment - behavior change is an issue with lein repl

Comment by Constantine Potapov [ 27/Nov/14 2:28 AM ]

I have opened the bug for the lein repl here https://github.com/technomancy/leiningen/issues/1774





[CLJ-1604] AOT'ed code that defs a var with clojure.core symbol name causes IllegalStateException Created: 25/Nov/14  Updated: 21/Feb/15  Resolved: 20/Feb/15

Status: Closed
Project: Clojure
Component/s: None
Affects Version/s: Release 1.7
Fix Version/s: Release 1.7

Type: Defect Priority: Critical
Reporter: Nicola Mometto Assignee: Unassigned
Resolution: Completed Votes: 3
Labels: aot, compiler

Attachments: Text File 0001-fix-AOT-bug-preventing-overriding-of-clojure.core-fu.patch     Text File 0001-fix-AOT-bug-preventing-overriding-of-clojure.core-fu-v2.patch     File 1604-context.diff    
Patch: Code
Approval: Ok

 Description   

AOT'ed code that defs a var that is also a symbol in clojure.core results in an exception at runtime. This problem can be avoided with (:refer-clojure :exclude ...) but this requires a library author to update and release a new version. AOT'ed applications must then wait for all transitive dependencies to update before they can update to a new Clojure version. For some users, this problem prevents them from trying or adopting new releases.

For example, the contrib library data.int-map defines an update function. clojure.core will also have a new update function as of 1.7.0. If this library is AOT'ed, then users of the clojure.data.int-map/update function will see the exception below. This situation can commonly occur when an application uses lein uberjar to compile all of the project+libs. In this case, applications or libraries that use data.int-map (either directly or indirectly) are affected.

java.lang.IllegalStateException: Attempting to call unbound fn: #'clojure.data.int-map/update
 at clojure.lang.Var$Unbound.throwArity (Var.java:43)
    clojure.lang.AFn.invoke (AFn.java:40)
    compiler_update_not_referenced_bug.core$foo.invoke (core.clj:5)

Reproduce with this sample project: https://github.com/yeller/compiler_update_not_referenced_bug

Cause: When AOT compiling a namespace, the def forms are hoisted into the ns__init class (in the example here, clojure.data.int_map__init). The static initializer in this class creates each var in the ns via a call to RT.var(ns, name). For data.int-map the static initializer will properly create the var for clojure.data.int-map/update. But when the ns is loaded (via the clojure.data.int_map.load() method), (refer-clojure) will be called, which will remap clojure.data.int-map/update to point to clojure.core/update.

This problem does not affect non-AOT loading (which doesn't use the ns__init class) and does not affect collisions from any other namespace. Only collisions from clojure.core create this possibility.

Proposed: The proposed patch explicitly refers the Var during ns__init.load() (after Clojure symbols are referred) rather than implicitly during ns__init static {}.

This change in behavior only happens during AOT in the specific case where a core symbol is being shadowed. In that case, clojure.core has already been loaded and v (the looked up var) will have ns=clojure.core. The currentNS will be (for example) data.int-map. If that's the case, and the sym has no ns, then the new logic will be emitted.

In the case of clojure.core itself, NO new bytecode is emitted. From tests on several projects, only shadowed vars during AOT get this additional bytecode.

Patch: 0001-fix-AOT-bug-preventing-overriding-of-clojure.core-fu-v2.patch

Screened by: Alex Miller



 Comments   
Comment by Andy Fingerhut [ 25/Nov/14 11:28 PM ]

When I try latest Clojure master plus patch CLJ-1604-only-core.patch with the small test project created by Tom Crayford to demonstrate this issue: https://github.com/yeller/compiler_update_not_referenced_bug

In that project, I get the same exception thrown when attempting 'lein do clean, uberjar, test' using this patch, as without it. It is because int-map/update in namespace compiler-update-not-referenced-bug.core is an unbound var.

Comment by Nicola Mometto [ 26/Nov/14 4:25 AM ]

Andy, you're right. For some reason I attached the wrong patch to the ticket, this is the correct one

Comment by Nicola Mometto [ 26/Nov/14 5:21 AM ]

I wasn't able to write a test for this, so here's a repl session using the clojure jar demonstrating this issue:

[˷/test]> ls
classes  clojure.jar  test.clj
[˷/test]> cat test.clj
(in-ns 'test)
(clojure.core/refer 'clojure.core)
(def foo "bar")
(def update "foo")
[˷/test]> java -cp classes:clojure.jar:. clojure.main
Clojure 1.7.0-master-SNAPSHOT
user=> (binding [*compile-files* true] (load "test"))
WARNING: update already refers to: #'clojure.core/update in namespace: test, being replaced by: #'test/update
nil
user=> test/foo
"bar"
user=> test/update
"foo"
user=>
[˷/test]> java -cp classes:clojure.jar:. clojure.main
Clojure 1.7.0-master-SNAPSHOT
user=> (load "test")
nil
user=> test/foo
"bar"
user=> test/update
CompilerException java.lang.RuntimeException: No such var: test/update, compiling: (NO_SOURCE_PATH:0:0)
user=>
Comment by Andy Fingerhut [ 26/Nov/14 10:39 AM ]

Thanks. I have not tried to assess the details of the change, other than to say that patch 0001-fix-AOT-bug-preventing-overriding-of-clojure.core-fu.patch dated 26 Nov 2014, when applied to latest Clojure master as of today, enables both 'lein do clean, test' and 'lein do clean, uberjar, test' to work as expected with Tom Crayford's test project, linked above, whereas 'lein do clean, uberjar, test' fails without this patch, due to a var being unbound that should have a value.

Comment by Andy Fingerhut [ 27/Nov/14 10:53 AM ]

Copying a comment here from CLJ-1591, since it is more appropriate here. It is responding to Tom Crayford's posting of his example project to demonstrate the issue: https://github.com/yeller/compiler_update_not_referenced_bug

Tom, looked at your project. Thanks for that. It appears not to have anything like (def inc inc) in it. It throws exception during test step of 'lein do clean, uberjar, test' consistently for me, too, but compiles with only warnings and passes tests with 'lein do clean, test'. I have more test results showing in which Clojure versions these results change. To summarize, the changes to Clojure that appear to make the biggest difference in the results are below (these should be added to the new ticket you create – you are welcome to do so):

Clojure 1.6.0, 1.7.0-alpha1, and later changes up through the commit with description "CLJ-1378: Allows FnExpr to override its reported class with a type hint": No errors or warnings for either lein command above.

Next commit with description "Add clojure.core/update, like update-in but takes a single key" that adds clojure.core/update: 'lein do clean, test' is fine, but 'lein do clean, uberjar' throws exception during compilation, probably due to CLJ-1241.

Next commit with description "fix CLJ-1241": 'lein do clean, test' and 'lein do clean, uberjar' give warnings about clojure.core/update, but no errors or exceptions. 'lein do clean, uberjar, test' throws exception during test step that is same as the one I see with Clojure 1.7.0-alpha4. Debug prints of values of clojure.core/update and int-map/update (in data.int-map and in Tom's namespace compiler-update-not-referenced-bug.core) show things look fine when printed inside data.int-map, and in Tom's namespace when not doing the uberjar, but when doing the uberjar, test, int-map/update is unbound in Tom's namespace.

In case it makes a difference, my testing was done with Mac OS X 10.9.5, Leiningen 2.5.0 on Java 1.7.0_45 Java HotSpot(TM) 64-Bit Server VM

Comment by Nicola Mometto [ 02/Dec/14 9:04 AM ]

The updated patch only emits the interning bytecode when necessary, avoiding the emission when a clojure.core var with the same name exists but is not mapped to the current namespace

Comment by Alex Miller [ 09/Jan/15 9:27 AM ]

Attached 1604-context.diff for purely informational purposes - same diff just more context in it for easier reading.

Comment by Tom Crayford [ 10/Jan/15 4:52 PM ]

Thought I'd add a minor note in here to say I tried testing this patch out on my app (which is where I discovered this AOT bug), and the bug doesn't turn up with this patch applied to clojure (tested by applying 0001-fix-AOT-bug-preventing-overriding-of-clojure.core-fu-v2.patch to 1.7-alpha5)

Comment by Adam Krieg [ 21/Feb/15 12:28 PM ]

I ran into this issue with Korma 0.4.0. I'm still running into it, but there is a twist.

My project depends on an artifact that was built with Clojure 1.7.0-alpha1. If I remove this dependency, everything is fine. However, with this dependency, I run into this issue, even if I declare a dependency on 1.7.0-master-SNAPSHOT in my project and exclude any dependency on clojure-1.7.0-alpha1.

I'm not sure if this is a Maven issue or a Clojure issue. Running Maven with debug on seems to show that it's using the correct version of Clojure.

I have created a dummy project that reproduces this issue if you are interested.

https://github.com/deaddowney/UpdateProblem

Check it out, run "mvn install", and you will get
java.lang.RuntimeException: No such var: korma.core/update
.





[CLJ-1603] cycle, iterate, repeat return vals should IReduceInit Created: 25/Nov/14  Updated: 27/Mar/15  Resolved: 27/Mar/15

Status: Closed
Project: Clojure
Component/s: None
Affects Version/s: None
Fix Version/s: Release 1.7

Type: Enhancement Priority: Major
Reporter: Stuart Halloway Assignee: Stuart Halloway
Resolution: Completed Votes: 0
Labels: None

Attachments: Text File clj-1603-10.patch     Text File clj-1603-11.patch     Text File clj-1603-12-2.patch     Text File clj-1603-12.patch     Text File clj-1603-13.patch     Text File clj-1603-14.patch     Text File clj-1603-15.patch     Text File clj-1603-2.patch     Text File clj-1603-3-2.patch     Text File clj-1603-3-3.patch     Text File clj-1603-3-4.patch     Text File clj-1603-3.patch     Text File clj-1603-4.patch     Text File clj-1603-5.patch     Text File clj-1603-6.patch     Text File clj-1603-7.patch     Text File clj-1603-8.patch     Text File clj-1603-9.patch     Text File clj-1603.patch    
Patch: Code and Test
Approval: Ok

 Description   

Screened

clj-1603-15.patch (Java approach)

Alternatives:

There were a number of possible approaches for these enhancements:
1) Straight Java impl. The benefit of this approach is improving the performance of both the seq and reduce paths at the expense of writing a bunch of Java. See clj-1603-15.patch for the best of these impls.

2) Clojure deftype. This required moving cycle and iterate and providing a repeat1 implementation until deftype is defined (similar to the approach with reduce1). The deftype version returns a Seqable, IReduce object and has effectively the same former implementation for seq with a new fast implementation for IReduce. This makes reduce paths fast, but leaves seq paths about the same, with the benefit of no new Java code. The downside was that the new seqs did not implement the full range of interfaces from prior impls, which could potentially break code. See clj-1603-12-2.patch for a patch that covers this.

3) Add Iterable or IReduceInit directly to LazySeq. Conceptually, this does not make sense for general lazy seqs. Seqs materialize and cache each value once and doing this along with the ability to iterate/reduce introduces issues with caching (might as well use seqs for that) and synchronization. However, doing this for just these specific lazy seqs is possible - see latest patch.

Approach: Latest patch makes LazySeq implement IReduce and internal macro reducible-lazy-seq lets callers supply a function to implement the reduce path.

A few things to note:

  • Added some example-based tests for iterate, cycle, and repeat where I thought they were needed. Did not add generative tests - not clear to me what these would be that would actually be valuable. All of these functions are pretty simple and the examples cover the special cases.

Performance:

Some example timing, all in µs:

Expression 1.6.0 1.7.0-alpha5 master + clj-1603-15 (Java) master + clj-1603-12-2 (deftype) master + clj-1603-14 (split)
(doall (take 1000 (repeat 1))) 87 93 63 89 92
(into [] (take 1000) (repeat 1)) n/a 67 25 27 33
(doall (repeat 1000 1)) 87 94 16 94 89
(into [] (repeat 1000 1)) 99 110 13 12 12
(reduce + 0 (repeat 1000 1)) 99 126 20 22 25
(into [] (take 1000) (repeat 1)) n/a 67 28 33 27
(doall (take 1000 (cycle [1 2 3]))) 101 106 85 108 103
(into [] (take 1000) (cycle [1 2 3])) n/a 73 38 45 44
(doall (take 1000 (iterate inc 0))) 93 98 75 123 116
(into [] (take 1000) (iterate inc 0)) n/a 85 38 40 39

Notes on timings above:

  • All reduce timings (with into) comparable across the impls and significantly better than the current behavior over seqs.
  • The Java impl is faster across the board with doall. doall repeatedly calls seq() and next() to walk the sequence. The Java class versions of Repeat, Cycle, and Iterate extend ASeq and seq() just return this. next() constructs and returns a new instance of the class, which is immutable. In the lazy seq versions, LazySeq is mutable and requires synchronization and handling the caching safely. So, simple immutable instance ftw here.
  • The Java finite repeat has an extra benefit from using a primitive long for the counter.
  • One performance difference that's not visible in the timings is that the Java implementations have the benefit of being both seqs and reducibles as they are traversed, so you can always get a fast reduce. The deftype and split impls are only reducible at the initial instance, walking off that initial head reverts to lazy seqs that are not quickly reducible.

Patch: The two patches in leading contention are:

  • clj-1603-15.patch - for the Java version
  • clj-1603-14.patch - for the split impl

Alex opinion: I have swung back and forth on this but my current recommendation is for the Java implementation (clj-1603-15.patch). It's faster for both seqs and reduce, both in the timings above and importantly in maintaining reducibility as they are traversed. There is more Java, but I've made my peace with that - the code maximally leverages existing ASeq infrastructure and the implementation is easy to understand.



 Comments   
Comment by Ghadi Shayban [ 25/Nov/14 11:01 AM ]

Stu, do you intend these to be in Java or Clojure? It could be trickier to implement in Clojure directly, as loading would have to be deferred until core_deftype loads. It's certainly tractable without breaking any backwards compatibility, and I've explored this while experimenting with Range as a deftype https://github.com/ghadishayban/clojure/commit/906cd6ed4d7c624dc4e553373dabfd57550eeff2

A macro to help with Seq&List participation could be certainly useful, as efficiently being both a Seq/List and IReduceInit isn't a party.

May be useful to list requirements for protocol/iface participation.

It seems like 'repeatedly' is another missing link in the IReduceInit story.

Rich mentioned the future integration of reduce-kv at the conj, it would also be useful to know how that could fit in.

---- Other concerns and ops that may belong better on the mailing list ----

In experimenting with more reducible sources, I put out a tiny repo (github.com/ghadishayban/reducers) a couple weeks ago that includes some sources and operations. The sources were CollReduce and not ISeq.

Relatedly, caching the hashcode as a Java `transient` field is not supported when implementing a collection using deftype (patch w/ test in CLJ-1573).

Sources:
Iterate was one of them https://github.com/ghadishayban/reducers/blob/master/src/ghadi/reducers.clj#L43-L51
Repeatedly https://github.com/ghadishayban/reducers/blob/master/src/ghadi/reducers.clj#L43-L51

Reduce/transduce-based Operations that accept transducers:
some, any, yield-first https://github.com/ghadishayban/reducers/blob/master/src/ghadi/reducers.clj#L52-L80
(any could use a better name, equiv to (first (filter...)))
some and any have a symmetry like filter/remove.

Novelty maybe for 1.8:
A transducible context for Iterables similar to LazyTransformer:
https://github.com/ghadishayban/reducers/blob/master/src/ghadi/reducers.clj#L157-L161

The unless-reduced macro was very useful in implementing the collections:
https://github.com/ghadishayban/reducers/blob/master/src/ghadi/reducers.clj#L7-L15
It is different than the ensure-reduced and unreduced functions in core.

Comment by Alex Miller [ 25/Nov/14 12:01 PM ]

When we discussed this in the past, it was in the vein of reusing some of the range work (in Java) to implement cycle and iterate (per CLJ-1515).

Comment by Ghadi Shayban [ 25/Nov/14 9:20 PM ]

Never mind about 'repeatedly'. Being both ISeq and IReduceInit for repeatedly doesn't make sense for something that relies on side-effects. Current users of repeatedly can reduce over it many times and only realize the elements once.

Comment by Alex Miller [ 05/Dec/14 11:17 PM ]

attached wip Java impl and posted some example timings

Comment by Ghadi Shayban [ 11/Dec/14 4:35 PM ]

NB iterate in this patch does not cache the realized ISeq, but recalcs it at every call to realize the tail. This is not a change in the promised behavior (docstring says "f must be side-effect free") but an implementation change, as worth noting in the changelog.

Comment by Stuart Halloway [ 02/Jan/15 1:32 PM ]

It looks like all the reduce with no inital value paths are still seq-y, and slower, as shown by e.g.

(dotimes [_ 10]
  (time
   (reduce
    +
    (repeat 10000000 1))))

(dotimes [_ 20]
  (time
   (reduce
    +
    0
    (repeat 10000000 1))))
Comment by Alex Miller [ 02/Jan/15 2:01 PM ]

On that example in master before / after patch I see:

before:

  • no init = 844 ms
  • with init = 920 ms

after:

  • no init = 124 ms
  • with init = 90 ms

Is that similar to what you see or not?

Comment by Alex Miller [ 02/Jan/15 4:21 PM ]

The clj-1603-3.patch has been updated to use effectively the same algorithm for both versions of reduce. With the -3 patch, I got ~96 ms on both examples in the prior comment. I re-ran the tests in the description and updated those as well (about the same as expected).

Comment by Stuart Halloway [ 16/Jan/15 1:18 PM ]

The tests do not seem to hit the unseeded reduce branches – do we even want these branches? The original ticket was for IReduceInit.

Comment by Michael Blume [ 18/Jan/15 1:48 PM ]

Probably worth noting – Git will happily apply the latest patch for CLJ-1603 on top of the latest patch for CLJ-1515, but the result does not compile because 1515 uses iterate and 1603 moves the definition of iterate lower in clojure.core. Not sure if this is worth fixing now or just noting for when they're actually applied.

Comment by Michael Blume [ 18/Jan/15 1:52 PM ]

Actually, here, this just moves the declare statement further up the file.

Comment by Michael Blume [ 18/Jan/15 2:19 PM ]

OK, no, the two patches are still incompatible even with the declaration order fixed:

[java] ERROR in (test-range) (LongRange.java:95)
     [java] expected: (= (take 3 (range 3 9 0)) (quote (3 3 3)))
     [java]   actual: java.lang.ClassCastException: clojure.core.InfiniteRepeat cannot be cast to clojure.lang.ISeq
     [java]  at clojure.lang.LongRange.create (LongRange.java:95)
Comment by Alex Miller [ 18/Jan/15 2:31 PM ]

The 1515 patch is actually being reworked right now - we will patch things up at application time if needed.

Comment by Alex Miller [ 19/Jan/15 10:12 AM ]

Removing screened marking so can be re-screened. Added new -7 patch that handles print-method, print-dup, and unmapping the deftype constructors so they're not visible. Thanks to Ghadi in CLJ-1515 for the idea on those.

Comment by Ghadi Shayban [ 20/Jan/15 8:08 AM ]

Review of -7 patch:

Seqable/seq implementations that return a separate ISeq like these do should forward a call to seq on the result, like eduction does. [1] (This is not necessary in these particular impls, as the LazySeqs returned are themselves ISeqs. Also because Cycle's deftype is only constructed for non-empty cycles, the fact that there is a guaranteed seq is implicit. Probably a best practice to add an innocuous seq call if users look to these impls as a recipe.)

The performance regression in (doall (repeat 1000 1)) should go away completely with the dorun tweak in CLJ-1515. This is because dorun is effectively calling seq twice (it calls seq, throws away the result, then calls next.)

minor nits
1) repeat1 seems to be identical to repeat-seq and has both arities necessary for the deftypes
2) inside FiniteRepeat s/(> i 0)/(pos? i) also inside the repeat constructor
3) some things are defn- , some are ^:private
4) Cycle/reduce the recur binding can be (recur rr (or (next s) coll)) rather than nil? check

[1] https://github.com/clojure/clojure/blob/master/src/clj/clojure/core.clj#L7324

Comment by Alex Miller [ 22/Jan/15 10:34 PM ]

Ghadi - good comments! Fixed 1,2,4. #3 - ^:private is because defn- is not yet defined. New -8 patch.

Comment by Alex Miller [ 23/Jan/15 10:03 AM ]

Bah.

user=> (= (repeat 5 :a) (repeat 5 :a))
false
Comment by Alex Miller [ 23/Jan/15 3:04 PM ]

Updated to -9 patch that handles hash and equality for finite repeat case.

Comment by Ghadi Shayban [ 26/Jan/15 2:24 PM ]

metadata in the wrong place on #'repeat1

Comment by Alex Miller [ 26/Jan/15 3:27 PM ]

Thanks, fixed in -10.

Comment by Stuart Halloway [ 20/Feb/15 10:19 AM ]

The collections returned by these APIs promise several things the new deftypes do not deliver. For example, 1.6's repeat currently has the following ancestors that are lost in the propopsed deftype:

  • clojure.lang.ISeq
  • java.io.Serializable
  • clojure.lang.IPending
  • java.util.Collection
  • java.util.List
  • java.lang.Iterable
  • clojure.lang.Obj
  • clojure.lang.IPersistentCollection
  • clojure.lang.IMeta
  • clojure.lang.IObj

Losing metadata and serializability are certainly regressions, other stuff maybe as well. I suspect similar problems in all the other API returns.

We could improve testing by taking advantage of the property-checking fns already in test-clojure/data-structures, e.g. is-same-collection

Comment by Alex Miller [ 20/Feb/15 10:42 AM ]

I think we should consider carefully what is contractually required by the return of repeat. My opinion is that repeat must return something seqable, not literally a seq (or lazy seq). With that perspective, ISeq, IPending (there re lazy seq laziness), Collection, List, Iterable, and IPersistentCollection are non-essential. Obj is a concrete class and we shouldn't commit to that.

  • IObj is a gap, this should work but doesn't: (with-meta (repeat 1 :a) {:a 1})
  • IMeta doesn't need to be added, will never have meta right now and meta handles this correctly
  • Serializable - doesn't make sense for the infinite seqs but should probably fix for finite range
Comment by Alex Miller [ 20/Feb/15 11:03 AM ]

Added -11 patch that adds IObj for all the new deftypes and Serializable for FiniteRepeat. Still need to add more tests.

Comment by Stuart Halloway [ 20/Feb/15 12:27 PM ]

I think all the java. interfaces are mandatory for interop. Leaving out any one of the clojure. interfaces creates observable change in behavior composing with other core fns (admittedly the IPending case would be bizarre, who uses that?), so those seem mandatory too.

Agreed Obj should be irrelevant, and if it isn't the bug is elsewhere.

Comment by Alex Miller [ 27/Feb/15 9:41 AM ]

pending further discussion w/ stu

Comment by Alex Miller [ 13/Mar/15 7:02 AM ]

for a clean run and more explanation of the perf numbers

Comment by Alex Miller [ 17/Mar/15 4:23 PM ]

refreshed numbers in table

Comment by Alex Miller [ 18/Mar/15 9:24 AM ]

Refreshed perf numbers and older patch variants, added some more explanation of the perf numbers for comparison.

Comment by Stuart Halloway [ 22/Mar/15 7:50 AM ]

Screened clj-1603-3-3.patch

Comment by Nicola Mometto [ 23/Mar/15 2:00 PM ]

Shouldn't iterate cache its next slot? the current implementation recalculates it every time:

user=> (def coll (iterate #(do (println %) (inc %)) 0))
#'user/coll
user=> (nth coll 3)
0
1
2
3
user=> (nth coll 3)
0
1
2
3

I realize the docstring for iterate explicitely warns that f must be side-effect free but this is just for demonstration purposes, imagine that f is a computationally-heavy function and you'll understand why I don't think the new proposed behaviour is desiderable.

Comment by Nicola Mometto [ 23/Mar/15 2:26 PM ]

I attached clj-1603-3-4.patch which is the same as clj-1603-3-3.patch except it caches the next slot of iterate.

Here is the diff between -3-3 and -3-4 for ease of review:

diff --git a/src/jvm/clojure/lang/Iterate.java b/src/jvm/clojure/lang/Iterate.java
index f23ddca..aeef998 100644
--- a/src/jvm/clojure/lang/Iterate.java
+++ b/src/jvm/clojure/lang/Iterate.java
@@ -16,6 +16,7 @@ public class Iterate extends ASeq implements IReduce {
 
 private final IFn f;      // never null
 private final Object seed;
+private volatile ISeq _next;
 
 private Iterate(IFn f, Object seed){
     this.f = f;
@@ -37,7 +38,14 @@ public Object first(){
 }
 
 public ISeq next(){
-    return new Iterate(f, f.invoke(seed));
+    ISeq ret = _next;
+    if (ret == null)
+        synchronized(this) {
+            ret = _next;
+            if (ret == null)
+                _next = ret = new Iterate(f, f.invoke(seed));
+        }
+    return ret;
 }
 
 public Iterate withMeta(IPersistentMap meta){
Comment by Alex Miller [ 23/Mar/15 3:32 PM ]

It's a good question. That impl kills all of the performance improvement for iterate on the seq use case (prob the current common case). I'll take a look at it though.

Comment by Nicola Mometto [ 23/Mar/15 3:57 PM ]

My opinion is that the performance improvement for iterate that the benchmark shows won't reflect a real performance improvement in "real" usage cases of iterate where `f` actually does some computation

Comment by Nicola Mometto [ 23/Mar/15 4:04 PM ]

This is also true for its reduce impl btw.
I'm sorry if I'm pointing this out late in the life of this ticket but it never occurred to me that, contrary to cycle, repeat or range (CLJ-1515), this change for iterate will possibly make accessing large sequences slower since we lose the caching aspect of lazy-seqs.

Comment by Alex Miller [ 23/Mar/15 4:58 PM ]

Added new -15 patch, which is the same as -3-3 but caches _next for all three types. This has no impact on the reduce performance and was 2-8 us slower in the timings for the seqs above. That seems like a small hit for the reduction in perf and memory on all cases where the seq is traversed multiple times. That seems like a reasonably common thing to do, particularly for finite repeat.

Comment by Alex Miller [ 23/Mar/15 5:02 PM ]

Stu - the only change in -15 vs -3-3 is in the next() methods of each new type. Each has a new "private volatile ISeq _next;"

Cycle:

public ISeq next(){
    if(_next == null) {
        ISeq next = current.next();
        if (next != null)
            _next = new Cycle(all, next);
        else
            _next = new Cycle(all, all);
    }
    return _next;
}

Iterate:

public ISeq next(){
    if(_next == null) {
        _next = new Iterate(f, f.invoke(seed));
    }
    return _next;
}

Repeat:

public ISeq next() {
    if(_next == null) {
        if(count > 1)
            _next = new Repeat(count - 1, val);
        else if(count == INFINITE)
            _next = this;
    }
    return _next;
}
Comment by Michael Blume [ 23/Mar/15 6:47 PM ]

For Cycle, would it be worth holding the head and linking back to it when we loop, so that the underlying sequence only has to be traversed once? Something like this

https://github.com/MichaelBlume/clojure/commit/cache-cycle-head

On my machine this saves about 10 microseconds on

(doall (take 1000 (cycle [1 2 3])))

Comment by Michael Blume [ 23/Mar/15 7:07 PM ]

Also, for the Cycle reduce implementations, it might make sense to assume the underlying sequence is going to be traversed many times, copy it into an array, and loop over it with an index.

Comment by Michael Blume [ 23/Mar/15 7:17 PM ]

like so: https://github.com/MichaelBlume/clojure/commit/array-loop

Comment by Alex Miller [ 23/Mar/15 8:28 PM ]

The problem with holding a pointer back to the head is that the intervening cycle can be arbitrarily large. You are then literally holding the head on an arbitrary size seq. A silly example that fails with the patch but works without it (depending on your jvm size):

(dorun (take 10000000 (cycle (range 10000000))))

I played with the cycle reduce too but it has the same effective potential issue of creating and holding an arbitrarily sized array. Actually this one is worse in that it realizes an arbitrarily large array before knowing how many inputs it will need - it could be a take 1. Also, things that formerly worked could blow up with an oome. What if someone relies on passing an infinite sequence to cycle in a fallthrough case? For example, this works with the current version but would blow up with the array.

(into [] (take 1000) (cycle (iterate inc 0)))

Neither of these seem worth the risk to me.

Comment by Michael Blume [ 23/Mar/15 10:53 PM ]

Yeah, fair enough.

Comment by Alex Miller [ 25/Mar/15 4:14 PM ]

updated version of the -15 patch that gives proper credit to Ghadi for part.





[CLJ-1602] vals and keys return values should implement IReduceInit or Iterable Created: 25/Nov/14  Updated: 27/Mar/15  Resolved: 27/Mar/15

Status: Closed
Project: Clojure
Component/s: None
Affects Version/s: None
Fix Version/s: Release 1.7

Type: Enhancement Priority: Major
Reporter: Stuart Halloway Assignee: Unassigned
Resolution: Completed Votes: 0
Labels: None

Attachments: File clj-1602-2.diff     File clj-1602-3.diff     File clj-1602-4.diff     File clj-1602-5.diff     File clj-1602-6.diff     File clj-1602.diff    
Patch: Code and Test
Approval: Ok

 Description   

Background: clojure.core/keys calls RT.keys(Object) calls APersistentMap.KeySeq.create(ISeq). RT.keys() creates a sequence (of Map.Entry objects) and KeySeq just wraps it, calling .keyValue(). There is an equivalent vals -> RT.vals() -> ValSeq path. Both of these seq impls extend ASeq and provide Iterable implementations via SeqIterator (iterator wrapped over the seq).

Approach: The important thing here is to avoid creating the sequence and instead directly iterate/reduce over the map. Noting that CLJ-1499 provides support for making PHM directly Iterable and that KeySeq/ValSeq already implement Iterable, I chose to focus on making the instances returned from keys and vals support Iterable directly on the underlying map instead of via the seq.

RT.keys()/vals() created the seq and passed it to KeySeq/ValSeq which made it too late to directly cover the original map iterator. There are a few places that rely on passing a seq of Map.Entry to keys/vals (not just a map instance), so I check for IPersistentMap and in that case pass it directly to a new KeySeq factory method that remembers both the Iterable and the ISeq. There is also support in here for using a direct key/value iterator via IMapIterable as introduced in CLJ-1499.

Questions/notes:

  • Could potentially check for Map or Iterable instead of IPersistentMap in RT.keys()/vals(). Not sure how common it is to pass normal Java maps to keys/vals.
  • The direct Iterable support vanishes once you move off the head of the keys or vals seq. So (rest (keys map)) does not have Iterable support. This is not really possible unless you hold an Iterator and advance it along with the seq, but that seemed to introduce all sorts of possibilities for badness. Since maps are unordered, it seems weird to rely on any ordering or processing only parts of any map, so I suspect doing this would be quite rare.
  • This patch depends on CLJ-1499 for IMapIterable.

Performance: I tested perf using criterium to benchmark as follows:

(use 'criterium.core)
(def m (zipmap (range 1000) (range 1000)))
(bench (reduce + (keys m)))
(bench (reduce + (vals m)))
(bench expr) 1.6.0 1.7.0-alpha5 alpha5 + clj-1499 + patch
(reduce + (keys m)) 69 µs 73 µs 44 µs
(reduce + (vals m)) 75 µs 77 µs 50 µs

Tests:

  • I added some basic tests for subseq and rsubseq as those both rely on the somewhat special behavior of keys accepting a seqable of Map.Entry objects (not just a map itself). There were no other tests for subseq or rsubseq already present.
  • Some coverage from CLJ-1499 tests d760db
  • Show that metadata works on keys/vals
  • Show that iterator works correctly after you step off the head of the seq, invalidating the iterable
  • Show that the fallthrough branch of iterator works (from PersistentTreeMap)

Patch: clj-1602-6.diff

Screened by:

  • the changes in CollReduce are unlikely to stand but are currently essential


 Comments   
Comment by Alex Miller [ 25/Nov/14 11:53 AM ]

Could leverage CLJ-1499 for the bulk of this, may pull that back from 1.8 into 1.7. Waiting on further work till that's answered.

Comment by Alex Miller [ 03/Dec/14 11:24 PM ]

I also have a patch that extends the CLJ-1499 iterators to support providing both key and val iterators that do not require creating and unpacking a Map.Entry. Unfortunately I only saw times that were ~48 µs on the perf benchmark in the description, so it's not a huge benefit (short-lived object allocation is cheap).

Comment by Alex Miller [ 09/Jan/15 9:15 AM ]

waiting on 1499 mods

Comment by Alex Miller [ 14/Jan/15 5:42 PM ]

Updated patch based on latest CLJ-1499-v10 patch.

Comment by Michael Blume [ 15/Jan/15 1:48 AM ]

I find it concerning that the v9 patch passed generative tests, shouldn't we have seen that?

Comment by Alex Miller [ 15/Jan/15 7:21 AM ]

Actually I did see it in the generative tests for this ticket so I moved those tests into the CLJ-1499 patch.

Comment by Michael Blume [ 15/Jan/15 10:10 AM ]

Ah, cool =)

Comment by Stuart Halloway [ 22/Mar/15 8:16 AM ]

clj-1602-5.diff merely updates -4 to apply on current master

Comment by Stuart Halloway [ 22/Mar/15 8:37 AM ]

requested some tests in the description

Comment by Alex Miller [ 23/Mar/15 1:51 PM ]

Updated to -6 patch that adds the requested tests.





[CLJ-1601] transducer arities for map-indexed, distinct, and interpose Created: 25/Nov/14  Updated: 09/Jan/15  Resolved: 09/Jan/15

Status: Closed
Project: Clojure
Component/s: None
Affects Version/s: None
Fix Version/s: Release 1.7

Type: Enhancement Priority: Major
Reporter: Stuart Halloway Assignee: Alex Miller
Resolution: Completed Votes: 0
Labels: transducers

Attachments: