<< Back to previous view

[CLJ-1203] Fallback to hash-based comparison for non-Comparables in Util.compare() Created: 17/Apr/13  Updated: 29/Apr/13  Resolved: 29/Apr/13

Status: Closed
Project: Clojure
Component/s: None
Affects Version/s: None
Fix Version/s: None

Type: Enhancement Priority: Minor
Reporter: Tassilo Horn Assignee: Unassigned
Resolution: Declined Votes: 0
Labels: enhancement, patch

Attachments: Text File 0001-Add-hasheq-based-fallback-to-Util.compare.patch    
Patch: Code

 Description   

I oftentimes use sorted collections, and my comparator functions usually look like that:

(fn [a b]
  (let [x (compare-according-to-my-use-case a b)]
    (if (zero? x)
       (compare a b)
       x)))

That is, I define the sorting order depending on the requirements of my use-case, and if that doesn't suffice to make a distinction, I simply fall back to the standard `compare` function in order to get a stable ordering anyhow. I think that's a widely used idiom, e.g., also used in "Clojure Programming" (for example page 109).

The problem with this approach is that several data structures don't implement Comparable, and thus aren't applicable to `compare` (you'll get a ClassCastException). Examples are maps, records, deftypes, and sequences.

The patch I'm going to attach extends `Util.compare()` with a fallback clause that performs a `hasheq()`-based comparison. This results in a meaningless but at least stable sorting order which suffices for use-cases like the one shown above.



 Comments   
Comment by Andy Fingerhut [ 18/Apr/13 9:01 PM ]

Patch 0001-Add-hasheq-based-fallback-to-Util.compare.patch dated Apr 17 2013 applies cleanly to latest master, but causes several unit tests in data_structures.clj to fail. These are the kinds of things you would expect to fail with the change made in the patch, because the failing tests expect an exception to be thrown when comparing objects that don't implement Comparable, and with this patch's changes they no longer do. If this patch's change is desired, those tests should probably be removed.

Comment by Tassilo Horn [ 19/Apr/13 2:34 AM ]

Thanks Andy, I can't believe I've forgotten to re-run the tests.

Anyway, I'm attaching a new version of the patch that deletes the few sorted-set and sorted-set-by invocations that check that an exception is thrown when creating sorted sets containing non-Comparables.

Comment by Tassilo Horn [ 19/Apr/13 2:35 AM ]

New version of the patch.

Comment by Andy Fingerhut [ 26/Apr/13 2:47 PM ]

Tassilo, you say that one of your use cases is sorted collections. Note that any compare function that returns 0 for two values will cause sorted sets and maps to treat the two compared values as equal, and at most one of them will get into the ordered set/map, the second treated as a duplicate, even though the values are not equal. See https://github.com/jafingerhut/thalia/blob/master/doc/other-topics/comparators.md#comparators-for-sorted-sets-and-maps-are-easy-to-get-wrong for an example (not based on your modified compare, but on a comparator that returns 0 for non-equal items).

With your proposed change to compare, this occurrence would become very dependent upon the hash function used. Probably quite rare, but it would crop up from time to time, and could potentially be very difficult to detect and track down the source.

Comment by Tassilo Horn [ 29/Apr/13 2:29 AM ]

Hi Andy, you are right. It's possible to add an explicit equals()-check in cases the hashcodes are the same, but I think there's nothing you can do in the hashcodes-equal-but-objects-are-not case. That is, there's no way to ensure the rule sgn(compare(x, y)) == -sgn(compare(y, x)), the transitivity rule, and the compare(x, y)==0 ==> sgn(compare(x, z))==sgn(compare(y, z)) for all z rule.

I'm closing that ticket.





[CLJ-1066] No dependency on jsr166y Created: 11/Sep/12  Updated: 20/Oct/12  Resolved: 20/Oct/12

Status: Closed
Project: Clojure
Component/s: None
Affects Version/s: Release 1.5
Fix Version/s: Release 1.5

Type: Defect Priority: Major
Reporter: Wolodja Wentland Assignee: Stuart Halloway
Resolution: Completed Votes: 4
Labels: patch
Environment:

$ java -version
java version "1.7.0_03"
OpenJDK Runtime Environment (IcedTea7 2.1.2) (7u3-2.1.2-2)
OpenJDK 64-Bit Server VM (build 22.0-b10, mixed mode)
$ lein version
Leiningen 2.0.0-preview10 on Java 1.7.0_03 OpenJDK 64-Bit Server VM


Attachments: Text File 0001-Don-t-AOT-compile-clojure.core.reducers.patch    
Patch: Code
Approval: Ok

 Description   

The Clojure 1.5.0-alpha4 jars that have been deployed to maven.org seem to have been compiled against JDK6 which causes
an exception if one tries to use reducers/fold.

— snip —
Setup:

user=> (require '[clojure.core.reducers :as r])
nil
user=> (def v (vec (range 10000)))
#'user/v

;; JDK7 + clojure 1.5.0-alpha4 from maven.org
;; → :dependencies [[org.clojure/clojure "1.5.0-alpha4"]] in project.clj

user=> (r/fold + (r/map inc v))
ClassNotFoundException jsr166y.ForkJoinTask java.net.URLClassLoader$1.run (URLClassLoader.java:366)

;; JDK7 + clojure 1.5.0-alpha4 from maven.org
;; → :dependencies [[org.clojure/clojure "1.5.0-alpha4"]
;; [org.codehaus.jsr166-mirror/jsr166y "1.7.0"]]

user=> (r/fold + (r/map inc v))
50005000

;; JDK7 + clojure 1.5.0-alpha4 locally compiled (mvn install) against OpenJDK7
;; → :dependencies [[org.clojure/clojure "1.5.0-alpha4"]] in project.clj

user=> (r/fold + (r/map inc v))
5000050000
— snip —

It would be wonderful if this issue could be fixed before the release of 1.5.0.

Have a nice day

Wolodja



 Comments   
Comment by Tassilo Horn [ 12/Sep/12 9:44 AM ]

That's really a nasty problem. I wrote the code that makes it possible to compile clojure with either a JDK7 and native ForkJoin or an older JDK + jsr166y. Unfortunately, if you build clojure with a JDK7, reducers' fold requires a JDK7 at runtime, and if you build clojure with a JDK <7 + jsr166y, fold also requires jsr166y at runtime.

The essential problem is that clojure is AOT-compiled to class-files and those are put into the release jars. Ordinary clojure libraries are distributed as jar files containing clj files that are compiled when loaded. There, the implemented approach works just fine. For example, again your example with 1.5.0-alpha4:

;; 1.5.0-master4 compiled with JDK6 + jsr166y running on a JDK7
user=> (require '[clojure.core.reducers :as r])
nil
user=> (def v (vec (range 10000)))
#'user/v
user=> (r/fold + (r/map inc v))
ClassNotFoundException jsr166y.ForkJoinTask  java.net.URLClassLoader$1.run (URLClassLoader.java:366)
;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
;; Now load the reducers.clj source code again, so that it
;; picks the right ForkJoin-impl for the current platform
(load-file "/home/horn/Repos/clj/clojure/src/clj/clojure/core/reducers.clj")
nil
user=> (r/fold + (r/map inc v))
50005000

So IMO, the best thing we can do is to omit AOT-compilation for reducers but to include only its clj file so that it'll be compiled automatically on the platform it eventually runs.

Comment by Tassilo Horn [ 12/Sep/12 11:55 AM ]

This patch removes the clojure.core.reducers namespace from the namespaces to be AOT compiled. So now this namespace will be JIT-compiled when being required, and at that point either the JDK7 ForkJoin classes or the jsr166y classes need to be there.

Comment by Stuart Halloway [ 21/Sep/12 1:31 PM ]

Rich: what is the right approach here?





[CLJ-1061] when-first double evaluation Created: 04/Sep/12  Updated: 20/Oct/12  Resolved: 20/Oct/12

Status: Closed
Project: Clojure
Component/s: None
Affects Version/s: Release 1.4
Fix Version/s: Release 1.5

Type: Defect Priority: Minor
Reporter: Steve Miner Assignee: Stuart Sierra
Resolution: Completed Votes: 1
Labels: patch
Environment:

Mac OS X 10.8.1, Java 1.7.0_06


Attachments: Text File 0001-avoid-double-evaluation-in-when-first.patch    
Patch: Code
Approval: Ok
Waiting On: Rich Hickey

 Description   

The when-first macro will evaluate the xs expression twice. Admittedly, it does exactly what the doc string says, but that seems undesirable to me. Even without side effects, there's a potential performance issue if xs is some expensive operation.

Patch attached. The main diff is:

- `(when (seq ~xs)
- (let [~x (first ~xs)]
- ~@body))))
+ `(when-let xs# (seq ~xs)
+ (let ~x (first xs#)
+ ~@body))))



 Comments   
Comment by Stuart Sierra [ 21/Sep/12 7:39 AM ]

Screened.





[CLJ-1050] Remove a lock in the common case of deref of delay Created: 26/Aug/12  Updated: 18/Sep/12  Resolved: 18/Sep/12

Status: Closed
Project: Clojure
Component/s: None
Affects Version/s: Release 1.4
Fix Version/s: None

Type: Enhancement Priority: Trivial
Reporter: Nicolas Oury Assignee: Unassigned
Resolution: Declined Votes: 0
Labels: enhancement, patch, performance

Attachments: Text File 0001-No-lock-in-the-common-path-for-delays.patch    

 Description   

Currently, when deref is called in Delay.java, a lock on the Delay is always acquired.
This is wasteful as most of the time you just want to read the val.

The attached patch changes this behaviour to the following:

  • val is initialised to a known secret value. (During its constructor so it is visible from any thread).
  • When deref is called, val is checked to the known secret value.
  • If it is not the secret value, then it has to be the value computed by the function and we return it.
  • If it is the secret value, then we lock this object and revert to the current behaviour.

This is faster than what is done currently and can be very much faster if there is contention on the delay.



 Comments   
Comment by Nicolas Oury [ 27/Aug/12 2:37 AM ]

Please close and reject. The patch is not working if val has non final fields.





[CLJ-1042] [PATCH] Allow negative substring indices for (subs) Created: 14/Aug/12  Updated: 19/Sep/12  Resolved: 17/Sep/12

Status: Closed
Project: Clojure
Component/s: None
Affects Version/s: Release 1.5
Fix Version/s: None

Type: Enhancement Priority: Minor
Reporter: Ian Eure Assignee: Unassigned
Resolution: Declined Votes: 0
Labels: enhancement, patch

Attachments: Text File clj-1042-negative-indices-patch3.txt     Text File negative-subs.patch    
Patch: Code and Test

 Description   

This adds Python-style negative string indices for (subs), e.g.:

(subs "foo bar" -3) ;; -> "bar"



 Comments   
Comment by Andy Fingerhut [ 16/Aug/12 7:17 PM ]

Ian, thanks for the patch. It is Rich Hickey's policy only to consider applying patches to Clojure from those who have signed a Clojure contributor agreement: http://clojure.org/contributing

Were you interested in doing so, or perhaps it is already in progress?

Comment by Ian Eure [ 20/Aug/12 11:44 AM ]

I wasn't aware that this was necessary. I'm mailing the form in.

Comment by Andy Fingerhut [ 27/Aug/12 7:56 PM ]

Patch clj-1042-negative-subs-patch2.txt dated Aug 27 2012 is identical to Ian Eure's negative-subs.patch, except it is in the desired git format.

Ian, for future reference on creating patches in the desired format, see the instructions under the heading "Development" on this page: http://dev.clojure.org/display/design/JIRA+workflow

Comment by Ian Eure [ 28/Aug/12 11:47 AM ]

Thanks, will do.

Comment by Steve Miner [ 04/Sep/12 3:53 PM ]

If Clojure decides to support Python-style negative indices, you should also consider adding support to subvec.

Comment by Ian Eure [ 06/Sep/12 12:17 PM ]

Patch extended to support negative indices on (subvec) as well.

Comment by Adrian Bendel [ 07/Sep/12 8:01 AM ]

The arg to rindex should probably be tagged with ^clojure.lang.Counted instead of ^String now.

Comment by Steve Miner [ 07/Sep/12 1:31 PM ]

Regarding the previous comment, String is a Java class so it isn't a clojure.lang.Counted. Is the type hint necessary? Maybe it should be on the call rather than the defn.

Ignoring the type hinting, I'll suggest a slightly simpler way to implement the rindex logic:

(defn rindex [coll i]
(if (neg? i) (+ (count coll) i) i))

In any case, I'm not sure rindex should be public even if you want the subs and subvec enhancements. Someone needs to make the case for adding a new function to core.

The Pythonic negative index is a debatable feature since it's pretty easy to implement for yourself if you want it.

Comment by Adrian Bendel [ 07/Sep/12 11:05 PM ]

Sorry, the type hint on rindex args isn't necessary at all. Just looked up in the source, calling count should never be reflective, since (count ..) emits calls to clojure.lang.RT/count.

Your solution looks good.

Comment by Stuart Halloway [ 17/Sep/12 7:07 AM ]

Negative indices were considered and rejected a long time ago. (I am merely conveying information--I have no strong opinion on this one.)

Comment by Andy Fingerhut [ 17/Sep/12 12:07 PM ]

Note: If some people really like negative index behavior as in Perl or Python, it is straightforward to create a library of functions in a different namespace, perhaps with different names, that can do it. Perhaps a "pythonisms" library?

Comment by Ian Eure [ 18/Sep/12 12:31 PM ]

Would this be accepted as part of clojure.string instead? I considered putting it there, but thought it would be confusing to have multiple substring functions in different namespaces.

This is very helpful in practice, and I'd really like to see at least the (subs) stuff in Clojure.

Comment by Andy Fingerhut [ 18/Sep/12 2:52 PM ]

Disclaimer: I'm no Clojure/core member, so can't speak authoritatively on whether something would or would not be accepted into clojure.string.

However, given that clojure.string is distributed with clojure.core, my guess is it would not be accepted. You'd be able to get things like this out for you and others as a separate library distributed on Clojars. That would also make it easy to include other Python-like things that you don't find in Clojure already.

Comment by Ian Eure [ 18/Sep/12 4:02 PM ]

This isn't about "python-like things," this is about a useful feature. Lots of languages support this: Perl, PHP, Ruby, Python, JavaScript, to name a few. Are you really suggesting that I should create a whole package for a version of a function in clojure.core with slightly different semantics? That's insane.

Anyway, I'm done wasting my time trying to navigate this hopelessly broken process. Maybe it would have been accepted if I included yet another way to interoperate with Java types.

Comment by Michael Klishin [ 18/Sep/12 5:09 PM ]

Stuart, do you remember why specifically negative indexes were rejected? Developing a separate library for a minor improvement to an existing function sounds unreasonable.

Comment by Carlos Cunha [ 18/Sep/12 5:10 PM ]

some explanation about this topic was given by Rich Hickey himself here: http://news.ycombinator.com/item?id=2053908

citation:
"...Yes, there is a backlog from when it was just me, and it will take a while to whittle down. We have finite resources and have to prioritize. I can assure you we have more important things to concentrate on than your negative index substring enhancement, and are doing so. You'll just have to be patient. Or, if you insist, I'll just reject it now because a) IMO it's goofy, b) you can make your own function that works that way c) we don't get a free ride from j.l.String, d) it begs the question of negative indices elsewhere..."

i've been following this thread hoping this feature would be included. but whatever the reason was for the rejection, i'm sure it was thoughtful. great thanks for this wonderful language, and thanks Ian Eure for his effort.

Comment by Steve Miner [ 18/Sep/12 5:25 PM ]

That HN link eventually leads back to CLJ-688 which was rejected.

Comment by Stuart Halloway [ 19/Sep/12 12:03 PM ]

Michael: A proposal for negative indexes would need to be systematic in considering implications for all functions that have index arguments.

Ian: Clojure is driven by design, not incremental piling of features.

All: clojure.string is incomplete in more important and fundamental ways than negative indexes. This sucks now, and will suck worse as more code is written in different dialects. I find myself wishing string was a contrib, so we could iterate faster.

Comment by Andy Fingerhut [ 19/Sep/12 1:34 PM ]

Stuart: Any specific proposals for how you'd like to see clojure.string improve? If it can be made a contrib, that would be cool, but understood if that would be considered too breaking of a change. Even if it isn't made a contrib, having tickets for improvement ideas you are willing to see means patches might get written, and they'll get in some time.





[CLJ-1024] Varargs protococol impls can be defined but not called Created: 10/Jul/12  Updated: 14/Feb/13  Resolved: 13/Feb/13

Status: Closed
Project: Clojure
Component/s: None
Affects Version/s: Release 1.4
Fix Version/s: Release 1.5

Type: Enhancement Priority: Minor
Reporter: Víctor M. Valenzuela Assignee: Stuart Halloway
Resolution: Declined Votes: 1
Labels: patch

Attachments: Text File 0001-CLJ-1024-Check-for-invalid-varags-destrucuring-uses.patch    
Patch: Code

 Description   

The compiler accepts this:

(deftype foo []
clojure.lang.IFn
(invoke [this & xs]))

However calling ((foo.) :bar) will throw an AbstractMethodError. Wouldn't some checking be desirable?



 Comments   
Comment by Tassilo Horn [ 11/Jul/12 1:20 AM ]

First of all, clojure.lang.IFn is no protocol but an interface. And it does not declare a

  public Object invoke(Object... obs)

method. It has an `invoke` method with 20 Object parameters followed by an Object... parameter, but to give an implementation for that, you have to specify every parameter separately, and the last Object... arg is just a normal argument that must be an Object[]. That's because Java-varags Type... parameters are just Java syntax sugar, but in the byte-code its simply a Type-array.

What your example does is provide an `invoke` implementation for the 2-args version, where the first parameter happens to be named `&`, which has no special meaning here. Take that example:

(deftype MyFoo []
  clojure.lang.IFn
  (invoke [this & xs]
    [& xs]))

((MyFoo.) 1 2)
=> [1 2]

But you are right in that `deftype`, `defrecord`, `defprotocol`, and `definferface` probably should error if user's seem to try to use varargs or destructuring.

Comment by Víctor M. Valenzuela [ 11/Jul/12 1:55 AM ]

Cheers for a most clarifying response.

The fact that the meaning of & gets 'subverted' makes the issue only (slightly) worse, in my opinion.

Just for the record, destructuring seems to work, at least for interface impls.

Comment by Tassilo Horn [ 11/Jul/12 2:42 AM ]

> The fact that the meaning of & gets 'subverted' makes the issue only (slightly) worse, in my opinion.

I agree. I'll attach a patch which checks for those invalid usages soon.

> Just for the record, destructuring seems to work, at least for interface impls.

Could you please provide a complete example demonstrating your statement?

I'm rather sure that varargs and destructuring don't work for any of defprotocol, definterface, deftype, defrecord, and reify. But you can use varargs and destructuring when providing dynamic implementations via `extend` (or `extend-protocol`, `extend-type`), because those impls are real functions in contrast to methods.

Comment by Tassilo Horn [ 11/Jul/12 2:43 AM ]

I attached a patch. Here's the commit message:

Check for invalid varags/destrucuring uses.

Protocol, interface method declarations and implementations don't allow for
varags and destructuring support. Currently, for example

(defprotocol FooBar
(foo [this & more]))

compiles just fine, and & is interpreted as a usual argument that happens to be
named & without special meaning. But clearly, the user wanted to specify a
varags parameter here. The same applies to definterface.

Similarly, providing method implementations via defrecord, deftype, and reify
don't allow for destructuring and varags (but dynamic extenions via extend do).

So this patch makes defprotocol, definterface, defrecord, deftype, and reify
throw an IllegalArgumentException if any argument vector contains a
destructuring form or varargs argument.

Comment by Víctor M. Valenzuela [ 12/Jul/12 3:13 AM ]

Glad you've considered my request.

As for destructuring, I was speaking after this example, which may or may not work like it looks like - I don't know.

(deftype foo []
clojure.lang.IFn
(invoke [this [a b] c] (println a b c)))

((foo.) [1 2] 3)

Comment by Tassilo Horn [ 12/Jul/12 8:22 AM ]

Indeed, descructuring seems to work for method implementations. I'll adapt my patch...

Comment by Tassilo Horn [ 12/Jul/12 8:42 AM ]

Revamped patch. Here's the commit message:

Protocol, interface method declarations don't allow for varags and
destructuring support. Currently, for example

  (defprotocol FooBar
    (foo [this & more]))

compiles just fine, and & is interpreted as a usual argument that happens to be
named & without special meaning. But clearly, the user wanted to specify a
varags parameter here. The same applies to definterface.

Similarly, providing method implementations via defrecord, deftype, and reify
don't allow for varags (but dynamic extenions via extend do).

So this patch makes defprotocol and definterface throw an
IllegalArgumentException if a user tries to use varargs and destructuring in
method signatures.

Similarly, defrecord, deftype, and reify throw an IllegalArgumentException if
any method implementation arglist contains a varargs argument.

Comment by Andy Fingerhut [ 12/Jul/12 3:43 PM ]

Tassilo, with your patch 0001-CLJ-1024-Check-for-invalid-varags-destrucuring-uses.patch dated July 12, 2012, I get the following error message while testing, apparently because some metadata is missing on the new functions your patch adds to core:

[java] Testing clojure.test-clojure.metadata
[java]
[java] FAIL in (public-vars-with-docstrings-have-added) (metadata.clj:45)
[java] expected: (= [] (remove (comp :added meta) public-vars-with-docstrin
gs-not-generated))
[java] actual: (not (= [] (#'clojure.core/throw-on-varargs-and-destr #'cl
ojure.core/throw-on-varargs)))

Comment by Tassilo Horn [ 13/Jul/12 2:10 AM ]

Hi Andy, this updated patch declares the two new checking fns private which makes the tests pass again.

Stupid mistake by me: Of course, I've tested the last version, too, but afterwards I decided it wouldn't be bad to add some docstrings, and of course, adding docstrings cannot break anything.

Comment by Aaron Bedra [ 14/Aug/12 7:58 PM ]

Patch applies cleanly against 4004d267e124f12b65b0d7fb6522f32a75e3c4fb. Submitter confirmed as a CA signer.

Comment by Aaron Bedra [ 14/Aug/12 8:02 PM ]

This looks ok to me, but it seems like a fair amount of duplication to accomplish the task. It seems like we should just be able to ask if it is ok to proceed, instead of having to pick which function needs to be called in what case.

Comment by Tassilo Horn [ 15/Aug/12 1:23 AM ]

Aaron, can you please elaborate? I don't get what you mean with duplication and asking if it is ok to proceed.

Comment by Tassilo Horn [ 31/Aug/12 1:40 AM ]

Rebased to apply cleanly on master again.

Comment by Víctor M. Valenzuela [ 31/Aug/12 7:11 AM ]

Pardon the silly contribution, but the added methods' usage of double-negations (when-not ...) seems unnecessary.

Comment by Tassilo Horn [ 31/Aug/12 8:03 AM ]

Hi Victor, this revamped patch removes the double-negation and is more concise and clearer as a result. So your comment wasn't as silly as you thought.

Comment by Tassilo Horn [ 14/Feb/13 1:36 AM ]

Hey Stu, do you mind to explain why you've declined the patch?

Comment by Marek Srank [ 14/Feb/13 8:52 AM ]

@Tassilo: https://groups.google.com/forum/#!topic/clojure-dev/qjkW-cv8nog

Comment by Tassilo Horn [ 14/Feb/13 10:03 AM ]

@Marek, Stu: Thanks, I've left a reply there: https://groups.google.com/d/msg/clojure-dev/qjkW-cv8nog/rMNFqbjNj-EJ





[CLJ-1018] range's behavior is inconsistent Created: 29/Jun/12  Updated: 24/May/13  Resolved: 24/May/13

Status: Closed
Project: Clojure
Component/s: None
Affects Version/s: Release 1.1, Release 1.2, Release 1.3, Release 1.4, Release 1.5
Fix Version/s: Release 1.5, Release 1.6

Type: Defect Priority: Minor
Reporter: Devin Walters Assignee: Unassigned
Resolution: Completed Votes: 0
Labels: patch, range

Attachments: File inconsistent_range_fix.diff    
Patch: Code and Test
Approval: Ok

 Description   

Problem statement: The current behavior of range is inconsistent. (range 0 9 0) has always produced (). (range 0 9 -1) has always produced (). (range 9 0 1) has always produced (). However, (range 9 0 0) produces (9 9 9 9 ...), and (range 0 0 0) produces '(0 0 0 0 ...)

Proposal: Make the behavior of range consistent when using a step of 0 to make it produce an empty list.

Please see attached code and patch.



 Comments   
Comment by Mike Anderson [ 01/Jul/12 4:08 AM ]

Agree it is good to fix the inconsistency, but I think an infinite sequence of zeros is the correct result, as implied by the current docstring definition.

It's also mathematically cleanest: range should produce an arithmetic progression until the end value is equalled or exceeded.

Empty lists only seem to make sense as a return value when start >= end.

Comment by Devin Walters [ 01/Jul/12 12:36 PM ]

Hi Mike,

Could you explain how you think the docstring definition implies this behavior? Maybe I'm missing something, but I was surprised. For instance, in the case of (range 3 9 0), if start were truly inclusive as the docstring says, the result would be (3), not ().

You mentioned that you think the infinite sequence of 0's is consistent and in keeping with the definition of range. I'm not sure I agree. (0,0] is an empty set of length zero, and [0,0) is an empty set of length zero.

You state that empty list only makes sense for start >= end, except this is not the current behavior. Could you help me understand what you believe the appropriate behavior would be in each of the following three cases? (range 0 10 0), (range 10 0 0), and (range 0 0 0)?

A few options to consider:
1) Fix the docstring to reflect the actual behavior of range.
2) Handle the case of (range 9 3 0) => (9 9 9 ...) to make it consistent with the behavior of (range 3 9 0) => ()
3) Stop allowing a step of zero altogether.

Editing to Add (Note: I don't think the way someone else did something is always the right way, just wanted to do some digging on how other people have handled this in the past):
http://docs.python.org/library/functions.html#range (0 step returns ValueError)
http://www2.tcl.tk/10797 (range returns empty list for a zero step)
http://www.scala-lang.org/api/2.7.7/scala/Range.html (zero step is not allowed)

Comment by Dimitrios Piliouras [ 05/Jul/12 4:13 PM ]

It does make sense NOT to allow a step of zero (at least to me)... I wasn't going to say anything about this but if other popular languages do not allow 0, then I guess it makes more sense than I originally gave myself credit for... However, if we want to be mathematically correct then the answer would be to return an infinte seq with the start value like below. After all, what is a step of 0? does it make any difference how many steps you take if with every step you cover 0 distance? obviously not...you start from x and you stay at x forever...we do have repeat for this sort of thing though...

(take 5 (range 3 9 0)) => (3 3 3 3 3)

+1 for not allowing 0 step

Comment by Mike Anderson [ 08/Jul/12 8:49 AM ]

@Devin quite simple: a lazy sequence of nums starting from x with step 0.0 until it reaches an end value of y (where y > x) is an infinite sequence of x.

Consider the case where start is 0 and end is infinity (the default), you would expect sequences to go as follows:

step +2 : 0 2 4 6 8....
step +1 : 0 1 2 3 4....
step +0 : 0 0 0 0 0....

It would be inconsistent to make 0 a special case, all of these are valid arithmetic progressions. And all of these are consistent with the current docstring.

If you make 0 a special case, then people will need to write special case code to handle it. Consider the code to create a multiplication table for example:

(for [x (range 10)]
(take 10 (range 0 Long/MAX_VALUE x)))

This works fine if you produce an infinite sequence of zeros for step 0, but fails if you create an empty list as a special case for step 0.

As a related issue, I'd personally also favour negative step sizes also producing an infinite sequence. If we don't want to allow this though, then at least the docstring should be changed to say "a lazy seq of non-decreasing nums" and a negative step should throw an error.

Comment by Devin Walters [ 09/Jul/12 7:09 PM ]

Carrying over a message from the clojure-dev list by Stuart Sierra:

  • I called the ticket a defect. Does that seem reasonable?

yes

  • Does anyone actually use the 0 step behavior in their programs?

not if they have any sense

  • Has anyone been bitten by this in the past?

not me

  • Is this behavior intentional for some historical reason I don't know about or understand?

I doubt it.

  • Has this been brought up before? I couldn't find any reference to it via Google.

Not that I know of.

  • Are there performance implications to my patch?

I doubt it. Lazy seqs are not ideal for high-performance code anyway.

  • Am I addressing a symptom rather than the problem?

I think the problem is that the result of `range` with a step of 0 was never specified. Don't assume that the tests are specifications. Many of the tests in Clojure were written by over-eager volunteers who defined the tests based on whatever the behavior happened to be. The only specification from the language designer is the docstring. By my reading, that means that `range` with a step of 0 should return an infinite seq of the start value, unless the start and end values are equal.

-S

Comment by Devin Walters [ 09/Jul/12 7:10 PM ]

Carrying over a message by Michal Marczyk:

With a negative step, the comparison is reversed, so that e.g.

(range 3 0 -1)

evaluates to

(3 2 1)

I think this is the useful behaviour; the docstring should perhaps be
adjusted to match it.

Agreed on zero step.

Cheers,
Michał

Comment by Devin Walters [ 20/Jul/12 5:10 PM ]

Adding a new patch after hearing comments. This patch makes (range 9 3 0) => (9 9 9 9 ...), (range 3 9 0) => (3 3 3 3 ...), and () will be returned when (= start end). Also updated the docstring.

Comment by Timothy Baldridge [ 27/Nov/12 3:01 PM ]

Marking as vetted

Comment by Timothy Baldridge [ 27/Nov/12 3:04 PM ]

Patch applies cleanly. We've discussed this issue to death (for as simple as it is). I think it's time to mark it as screened.

Comment by Timothy Baldridge [ 27/Nov/12 3:06 PM ]

For some reason I'm not allowed to edit the attachments list. When you apply the patch, please apply inconsistent_range_fix.diff as that is the most recent version of the fix.

Comment by Rich Hickey [ 09/Dec/12 6:44 AM ]

As someone who has to read these tickets, I'd really appreciate it if you could keep the description up to date and accurate, with examples of the current and intended behavior and the strategy to be used in fixing. I can't follow every thread to see what the latest thinking is, especially on a patch like this where the original mission was changed.

Thanks

Comment by Devin Walters [ 10/Dec/12 3:53 PM ]

@Tim: I've removed the other attachments.

@Rich: Understood. I will update the description this evening.





[CLJ-1011] clojure.data/diff should cope with null and false values in maps Created: 12/Jun/12  Updated: 18/Aug/12  Resolved: 18/Aug/12

Status: Closed
Project: Clojure
Component/s: None
Affects Version/s: Release 1.3, Release 1.4
Fix Version/s: Release 1.5

Type: Defect Priority: Minor
Reporter: Philip Aston Assignee: Unassigned
Resolution: Completed Votes: 0
Labels: enhancement, patch

Attachments: Text File 0001-clojure.data-diff-cope-with-falsey-values-in-maps.patch    
Patch: Code and Test
Approval: Ok

 Description   

Current behaviour of clojure.data/diff:

=> (diff {:a false} {:a true})
(nil {:a true} nil)
=> (diff {:a false} {:a nil})
(nil nil nil)

Proposed behaviour:

=> (diff {:a false} {:a true})
({:a false} {:a true} nil)
=> (diff {:a false} {:a nil})
({:a false} {:a nil} nil)


 Comments   
Comment by Philip Aston [ 12/Jun/12 5:04 AM ]
 
From e03a8060214d122ea2ebadf9e8a368f7f593d9f4 Mon Sep 17 00:00:00 2001
From: Philip Aston <philipa@mail.com>
Date: Sun, 10 Jun 2012 13:11:36 +0100
Subject: [PATCH] clojure.data/diff: cope with falsey values in maps

---
 src/clj/clojure/data.clj           |   18 +++++++++++++++++-
 test/clojure/test_clojure/data.clj |    3 ++-
 2 files changed, 19 insertions(+), 2 deletions(-)

diff --git a/src/clj/clojure/data.clj b/src/clj/clojure/data.clj
index 6e8dbcf..345b234 100644
--- a/src/clj/clojure/data.clj
+++ b/src/clj/clojure/data.clj
@@ -30,6 +30,22 @@
      (vec (repeat (apply max (keys m))  nil))
      m)))
 
+(defn- diff-associative-key
+  "Diff associative things a and b, comparing only the key k."
+  [a b k]
+  (let [va (get a k)
+        vb (get b k)
+        [a* b* ab] (diff va vb)
+        in-a (contains? a k)
+        in-b (contains? b k)
+        same (and in-a in-b
+                  (or (not (nil? ab))
+                      (and (nil? va) (nil? vb))))]
+    [(when (and in-a (or (not (nil? a*)) (not same))) {k a*})
+     (when (and in-b (or (not (nil? b*)) (not same))) {k b*})
+     (when same {k ab})
+     ]))
+
 (defn- diff-associative
   "Diff associative things a and b, comparing only keys in ks."
   [a b ks]
@@ -38,7 +54,7 @@
      (doall (map merge diff1 diff2)))
    [nil nil nil]
    (map
-    (fn [k] (map #(when % {k %}) (diff (get a k) (get b k))))
+    (partial diff-associative-key a b)
     ks)))
 
 (defn- diff-sequential
diff --git a/test/clojure/test_clojure/data.clj b/test/clojure/test_clojure/data.clj
index 9bab766..5a241e0 100644
--- a/test/clojure/test_clojure/data.clj
+++ b/test/clojure/test_clojure/data.clj
@@ -27,5 +27,6 @@
        [#{1} #{3} #{2}] (HashSet. [1 2]) (HashSet. [2 3])
        [nil nil [1 2]] [1 2] (into-array [1 2])
        [nil nil [1 2]] (into-array [1 2]) [1 2]
-       [{:a {:c [1]}} {:a {:c [0]}} {:a {:c [nil 2] :b 1}}] {:a {:b 1 :c [1 2]}} {:a {:b 1 :c [0 2]}}))
+       [{:a {:c [1]}} {:a {:c [0]}} {:a {:c [nil 2] :b 1}}] {:a {:b 1 :c [1 2]}} {:a {:b 1 :c [0 2]}}
+       [{:a nil} {:a false} {:b nil :c false}] {:a nil :b nil :c false} {:a false :b nil :c false}))
 
-- 
1.7.9.5
Comment by Andy Fingerhut [ 12/Jun/12 12:43 PM ]

Philip, thanks for writing that patch. Could you please attach it as a file to this ticket, instead of copying and pasting it into a comment? Instructions are under the heading "Adding patches" on this page:

http://dev.clojure.org/display/design/JIRA+workflow

Rich Hickey's policy is only to include code in Clojure that is written by those who have signed a contributor agreement. You can find out more at http://clojure.org/contributing

Comment by Philip Aston [ 12/Jun/12 2:51 PM ]

Thanks Andy. Patch attached:

0001-clojure.data-diff-cope-with-falsey-values-in-maps.patch

I'll send in a CA.

Comment by Philip Aston [ 16/Jun/12 5:27 AM ]

CA snail-mailed yesterday, I guess it will take a week to arrive.

Comment by Philip Aston [ 23/Jun/12 5:00 AM ]

I now have a CA in place.

Comment by Stuart Halloway [ 20/Jul/12 4:51 PM ]

Yeah, this should be fixed.

Comment by Aaron Bedra [ 14/Aug/12 9:42 PM ]

Patch applies cleanly against 4004d267e124f12b65b0d7fb6522f32a75e3c4fb. Submitter is a confirmed CA signer.





[CLJ-990] Implement clojure.core.reducers/mapcat and some initial reducers tests Created: 10/May/12  Updated: 01/Mar/13  Resolved: 10/May/12

Status: Closed
Project: Clojure
Component/s: None
Affects Version/s: Release 1.5
Fix Version/s: Release 1.5

Type: Enhancement Priority: Minor
Reporter: Tassilo Horn Assignee: Unassigned
Resolution: Completed Votes: 0
Labels: patch, test

Attachments: Text File 0001-Implement-clojure.core.reducers-mapcat.patch     Text File 0002-Add-initial-reducers-tests.patch    
Patch: Code and Test

 Description   

The first patch implements a reducers equivalent of clojure.core/mapcat, and the second patch adds a new reducers.clj to the clojure tests that checks that map, mapcat, filter, and reduce are equivalent in clojure.core and clojure.core.reducers.



 Comments   
Comment by Rich Hickey [ 10/May/12 7:44 PM ]

applied - thanks!





[CLJ-987] pprint doesn't flush the underlying stream Created: 07/May/12  Updated: 01/Mar/13  Resolved: 14/Nov/12

Status: Closed
Project: Clojure
Component/s: None
Affects Version/s: Release 1.5
Fix Version/s: Release 1.5

Type: Defect Priority: Major
Reporter: David Powell Assignee: David Powell
Resolution: Completed Votes: 1
Labels: patch

Attachments: Text File 0002-pprint-now-flushes-the-underlying-stream-similarly-t.patch    
Patch: Code and Test
Approval: Ok

 Description   

Unlike prn, pprint doesn't flush the underlying stream.

pretty_writer currently overrides .flush with behaviour that pushes its own data, but does not flush the underlying stream.

pprint should behave similarly to prn with respect to flushing the underlying stream.



 Comments   
Comment by David Powell [ 07/May/12 9:43 AM ]

Test included.
Patch should ensure that flushing happens after pprint, without introducing any additional unnecessary flushes.

Comment by Stuart Halloway [ 08/Jun/12 3:11 PM ]

I am confused by the tests – they all seem to call prn, though some claim to be calling pprint.

Comment by David Powell [ 09/Jun/12 6:16 AM ]

Ah, sorry. I'd tried to remove duplicate tests before committing them, but removed the wrong ones.
Replacement patch attached:

0002-pprint-now-flushes-the-underlying-stream-similarly-t.patch

Comment by Andy Fingerhut [ 21/Jun/12 5:27 PM ]

Tempting fate once again by changing Approval from Incomplete to None after the reason it was marked as incomplete seems to have been addressed.

Comment by Stuart Sierra [ 09/Nov/12 4:07 PM ]

Screened.

Comment by Stuart Sierra [ 14/Nov/12 9:23 AM ]

Pushed in commit 1588ff3f70e864d9817bc565bd2c30b08189bc6e





[CLJ-982] Implement an interface? predicate to balance class? Created: 04/May/12  Updated: 09/Jun/12  Resolved: 08/Jun/12

Status: Closed
Project: Clojure
Component/s: None
Affects Version/s: Release 1.5
Fix Version/s: None

Type: Enhancement Priority: Trivial
Reporter: David Rupp Assignee: Unassigned
Resolution: Declined Votes: 0
Labels: enhancement, patch, test
Environment:

Any


Attachments: File drupp-add-interface-predicate.diff    
Patch: Code and Test

 Description   

clojure.core implements a class? predicate to detect if a thing is an instance of java.lang.Class. The attached patch implements interface? to further distinguish classes that are also interfaces. This gives us Clojure api for e.g., enumerating all interfaces a thing's base class implements; as in, (filter #(interface? %) (supers (class my-thing))).



 Comments   
Comment by Stuart Halloway [ 08/Jun/12 12:28 PM ]

I would prefer to see this, and other interop/reflective items, in a separate contrib lib.

Comment by David Rupp [ 09/Jun/12 9:08 AM ]

Thanks for the feedback, Stu. Is there an existing contrib lib for stuff like this? Or should I create one?





[CLJ-960] Capture :column metadata (needed for ClojureScript source maps) Created: 27/Mar/12  Updated: 20/Oct/12  Resolved: 20/Oct/12

Status: Closed
Project: Clojure
Component/s: None
Affects Version/s: None
Fix Version/s: Release 1.5

Type: Enhancement Priority: Major
Reporter: Brandon Bloom Assignee: Unassigned
Resolution: Completed Votes: 2
Labels: patch

Attachments: File CLJ-960-tests.diff     Text File columns-1.patch     Text File columns-1-v2.patch    
Patch: Code and Test
Approval: Ok

 Description   

I've begun working on implementing SourceMaps for ClojureScript. For an overview of SourceMaps, see: http://www.html5rocks.com/en/tutorials/developertools/sourcemaps/ For discussion of the feature in ClojureScript, see: https://groups.google.com/d/topic/clojure-dev/zgmXO2iM1JQ/discussion

In order to produce accurate source maps, I need column information in addition to line information from the Clojure reader.

I've made the necessary enhancement to LispReader, etc. but have some cleanup and testing left to do. I'd also like a sanity check from the core team before attaching a formal patch. You can find my work in progress here: https://github.com/brandonbloom/clojure/compare/columns



 Comments   
Comment by David Nolen [ 13/Apr/12 8:29 AM ]

You need to attach a patch so this can be assessed. Thanks!

Comment by Brandon Bloom [ 31/May/12 6:09 PM ]

Sorry David! I added a patch a while ago but forgot to add the patch label(s?) as well as leave a comment.

The patch may be out of date now, but I haven't checked. Hopefully the prescreening process will pick it up automatically run the tests.

Comment by Andy Fingerhut [ 31/May/12 6:32 PM ]

Yes, Brandon, your patch has been on the list of prescreened patches since April 15. It has always applied and built cleanly during that time. That patch label is nice to have for certain JIRA report filters, but isn't necessary for the prescreening process to pick it up.

Comment by John Szakmeister [ 27/Jul/12 5:32 AM ]

v2 now applies against the current master (191b05f1). The original patch seemed to be broken from a whitespace perspective, which was making it difficult to apply--even in a 3-way merge. The only real conflict was in Compiler.java where a "final int line" was added to CompilerException. All the tests passed.

Comment by Brandon Bloom [ 27/Jul/12 7:33 PM ]

It looks like the line field was added to CompilerException in commit 89245c68, but that commit doesn't use it for anything. Maybe a later commit uses it? Also, if we want to keep that field, can we also add a column field for patch v3?

Comment by David Nolen [ 27/Jul/12 7:36 PM ]

This patch is an enhancement. In order for this one to make any movement I believe it will need a design page outlining the problem, what this solves / alternatives.

Comment by Brandon Bloom [ 25/Aug/12 9:38 PM ]

http://dev.clojure.org/display/design/Compiler+tracking+of+columns

I added a design page, but it seems like overkill. This is a straightforward enhancement...

Comment by Rich Hickey [ 04/Oct/12 8:51 AM ]

I've applied the v2 patch (thanks!), but before we close the ticket can we please get a patch comprising some tests?

Comment by Chas Emerick [ 19/Oct/12 11:45 AM ]

Attached CLJ-960-tests.diff to verify :line and :column metadata as now provided by LineNumberingPushbackReader.

Comment by Stuart Halloway [ 19/Oct/12 3:02 PM ]

The tests are a little fragile (exact map comparison) and so will break if the reader ever does more things. In library and app projects I write tests like this using core.match, but that isn't available as a build dependency here.

Marking screened anyway.





Generated at Fri Jul 25 18:06:27 CDT 2014 using JIRA 4.4#649-r158309.