Patch 0001-Add-hasheq-based-fallback-to-Util.compare.patch dated Apr 17 2013 applies cleanly to latest master, but causes several unit tests in data_structures.clj to fail. These are the kinds of things you would expect to fail with the change made in the patch, because the failing tests expect an exception to be thrown when comparing objects that don't implement Comparable, and with this patch's changes they no longer do. If this patch's change is desired, those tests should probably be removed.

Thanks Andy, I can't believe I've forgotten to re-run the tests.

Anyway, I'm attaching a new version of the patch that deletes the few sorted-set and sorted-set-by invocations that check that an exception is thrown when creating sorted sets containing non-Comparables.

New version of the patch.

Tassilo, you say that one of your use cases is sorted collections. Note that any compare function that returns 0 for two values will cause sorted sets and maps to treat the two compared values as equal, and at most one of them will get into the ordered set/map, the second treated as a duplicate, even though the values are not equal. See https://github.com/jafingerhut/thalia/blob/master/doc/other-topics/comparators.md#comparators-for-sorted-sets-and-maps-are-easy-to-get-wrong for an example (not based on your modified compare, but on a comparator that returns 0 for non-equal items).

With your proposed change to compare, this occurrence would become very dependent upon the hash function used. Probably quite rare, but it would crop up from time to time, and could potentially be very difficult to detect and track down the source.

Hi Andy, you are right. It's possible to add an explicit equals()-check in cases the hashcodes are the same, but I think there's nothing you can do in the hashcodes-equal-but-objects-are-not case. That is, there's no way to ensure the rule sgn(compare(x, y)) == -sgn(compare(y, x)), the transitivity rule, and the compare(x, y)==0 ==> sgn(compare(x, z))==sgn(compare(y, z)) for all z rule.

I'm closing that ticket.

I think that this patch would be much more helpful to users if it reported the problem form (both name and params).

(And I wonder if we should be using ex-info for all errors going forward.)

New version of the patch that mentions both method name and argument vector, and uses ex-info as Stu suggested.

Presumuptuously changing Approval from Incomplete back to None, since the reason for marking it Incomplete seems to have been addressed with a new patch.

Notes from the dev mailing list:

The "sources" JAR is generated by another Maven plugin, configured here:
https://github.com/clojure/clojure/blob/clojure-1.5.0-RC15/pom.xml#L169-L181

The simplest solution might be to just exclude the file from the sources jar. It looks like maven-source-plugin has an excludes option which would do the trick:

http://maven.apache.org/plugins/maven-source-plugin/jar-mojo.html#excludes

This patch implements the suggested feature.

The new system property is called clojure.core.loading-verbosely in analogy to the existing clojure.compile.warn-on-reflection.

This is the thread where Max Penet suggested to have 0-arity versions of the two fns:

https://groups.google.com/forum/?fromgroups=#!topic/clojure/IRlN-4LH_U0

Please bring this up on clojure-dev. We'll be able to vet this ticket after that.

Should I respond to my original clojure-dev post about this (linked in the issue description above), or start a new one?

In a quickly hacked up performance test on Mac OS X 10.6.8 + Oracle/Apple JDK 1.6.0_35 which I can attach, I saw about a 9% to 10% slowdown for Colin's patch in printing large vectors.

I have a modified patch that only calls Thread/interrupted after every 20 items are printed, and with that version the slowdown versus the original code was about 3% to 4%.

Colin, would there be any down side to checking Thread/interrupted less often for your purposes?

To run performance test, edit attachment compile.sh to point at your clojure.jar, put file perftest-print.clj in the same directory, and run ./compile.sh It should work on Mac or Linux.

clj-1073-allow-thread-interrupt-in-print-sequential-patch.txt dated Sep 22 2012 is similar to Colin's patch 0001-Allow-thread-interruption-in-print-sequential.patch dated Sep 21 2012, except it only checks interrupted status every 20 (or maybe 21?) times through the loop in print-sequential. It is the one that is 3-4% slower than the current latest master Clojure code in my performance test mentioned above, versus Colin's patch which is about 9-10% slower on that test.

Is this primarily intended for dev-time use? I wouldn't want to lose performance for this if there is any way to implement it as a dev-time feature.

Andy: The only caveat I can think of with checking Thread/interrupted less often is in the case of deeply nested collections.

Stu: Dev-time use was the original reason for opening this, yes. But I can imagine it being needed for anytime a thread can be interrupted, whether that's by ctrl-c or other means.

My original thinking, performance-wise, was that once we're already doing IO, we're probably not too concerned with CPU-bound checks like this, so I didn't bother with it. I guess with an SSD that's not as likely to be true.

Locally (w/ my SSD), I'm seeing that Andy's benchmark of printing a million numbers is about a second slower than the baseline with my original patch (12.08s -> 13.10s), and about the same with Andy's patch (12.08s -> 11.75s). Decreasing to a thousand numbers doesn't really show any difference (every version completes in ~1.3s). Bumping to 2 million adds 2 seconds above the baseline with my patch, and Andy's is again just a bit faster than the baseline (somehow). Bumping to 5 million runs me out of heap space

clj-1073-add-print-interruptibly-patch-v1.txt dated Nov 8 2012 is the same idea as Colin's patch 0001-Allow-thread-interruption-in-print-sequential.patch dated Sep 21 2012, except it only checks (Thread/interrupted) if a new var print-interruptibly is true. Its default value is false.

Performance results of the perftest-print.clj program, as driven by the test.sh script, for Clojure 1.5-beta1 and with two different patches. All run times are elapsed, in seconds, and sorted in increasing order for easier comparison.

Executive summary: Performance results when print-interruptibly is left at default false value are pretty much same as today. With print-interruptibly bound to true, performance results are slower, as expected, and about the same as with Colin's patch.

Original 1.5-beta1 unchanged:
13.75 13.80 13.83 13.87 13.93

With this new print-interruptibly patch, with print-interruptibly
at default false value:
13.86 13.91 14.01 14.08 14.14

With this new print-interruptibly patch, with print-interruptibly
bound to true while printing (so a slightly modified version of
perftest-print.clj that does (binding [*print-interruptibly* true]
...) around the heart of the code):
15.29 15.44 15.45 15.62 15.63

With patch 0001-Allow-thread-interruption-in-print-sequential.patch
applied:
15.38 15.46 15.48 15.49 15.50

Andy's latest patch looks good & makes it easy for the REPL and other interruptible scenarios to opt into the "safe" behavior. I would personally have preferred to have the "safe" behavior on by default, but can understand the performance concerns, and this gets me what I really wanted.

Vetting as it sounds like the performance issues are taken care of.

clj-1073-add-print-interruptibly-patch-v2.txt dated Feb 12 2013 is identical to clj-1073-add-print-interruptibly-patch-v1.txt dated Nov 8 2012 (soon to be removed) except that it applies cleanly to latest master.

$ java -version
java version "1.7.0_03"
OpenJDK Runtime Environment (IcedTea7 2.1.2) (7u3-2.1.2-2)
OpenJDK 64-Bit Server VM (build 22.0-b10, mixed mode)
$ lein version
Leiningen 2.0.0-preview10 on Java 1.7.0_03 OpenJDK 64-Bit Server VM

That's really a nasty problem. I wrote the code that makes it possible to compile clojure with either a JDK7 and native ForkJoin or an older JDK + jsr166y. Unfortunately, if you build clojure with a JDK7, reducers' fold requires a JDK7 at runtime, and if you build clojure with a JDK <7 + jsr166y, fold also requires jsr166y at runtime.

The essential problem is that clojure is AOT-compiled to class-files and those are put into the release jars. Ordinary clojure libraries are distributed as jar files containing clj files that are compiled when loaded. There, the implemented approach works just fine. For example, again your example with 1.5.0-alpha4:

;; 1.5.0-master4 compiled with JDK6 + jsr166y running on a JDK7
user=> (require '[clojure.core.reducers :as r])
nil
user=> (def v (vec (range 10000)))
#'user/v
user=> (r/fold + (r/map inc v))
ClassNotFoundException jsr166y.ForkJoinTask  java.net.URLClassLoader$1.run (URLClassLoader.java:366)
;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
;; Now load the reducers.clj source code again, so that it
;; picks the right ForkJoin-impl for the current platform
(load-file "/home/horn/Repos/clj/clojure/src/clj/clojure/core/reducers.clj")
nil
user=> (r/fold + (r/map inc v))
50005000

So IMO, the best thing we can do is to omit AOT-compilation for reducers but to include only its clj file so that it'll be compiled automatically on the platform it eventually runs.

This patch removes the clojure.core.reducers namespace from the namespaces to be AOT compiled. So now this namespace will be JIT-compiled when being required, and at that point either the JDK7 ForkJoin classes or the jsr166y classes need to be there.

Rich: what is the right approach here?

Mac OS X 10.8.1, Java 1.7.0_06

Screened.

Please close and reject. The patch is not working if val has non final fields.

Ian, thanks for the patch. It is Rich Hickey's policy only to consider applying patches to Clojure from those who have signed a Clojure contributor agreement: http://clojure.org/contributing

Were you interested in doing so, or perhaps it is already in progress?

I wasn't aware that this was necessary. I'm mailing the form in.

Patch clj-1042-negative-subs-patch2.txt dated Aug 27 2012 is identical to Ian Eure's negative-subs.patch, except it is in the desired git format.

Ian, for future reference on creating patches in the desired format, see the instructions under the heading "Development" on this page: http://dev.clojure.org/display/design/JIRA+workflow

Thanks, will do.

If Clojure decides to support Python-style negative indices, you should also consider adding support to subvec.

Patch extended to support negative indices on (subvec) as well.

The arg to rindex should probably be tagged with ^clojure.lang.Counted instead of ^String now.

Regarding the previous comment, String is a Java class so it isn't a clojure.lang.Counted. Is the type hint necessary? Maybe it should be on the call rather than the defn.

Ignoring the type hinting, I'll suggest a slightly simpler way to implement the rindex logic:

(defn rindex [coll i]
(if (neg? i) (+ (count coll) i) i))

In any case, I'm not sure rindex should be public even if you want the subs and subvec enhancements. Someone needs to make the case for adding a new function to core.

The Pythonic negative index is a debatable feature since it's pretty easy to implement for yourself if you want it.

Sorry, the type hint on rindex args isn't necessary at all. Just looked up in the source, calling count should never be reflective, since (count ..) emits calls to clojure.lang.RT/count.

Your solution looks good.

Negative indices were considered and rejected a long time ago. (I am merely conveying information--I have no strong opinion on this one.)

Note: If some people really like negative index behavior as in Perl or Python, it is straightforward to create a library of functions in a different namespace, perhaps with different names, that can do it. Perhaps a "pythonisms" library?

Would this be accepted as part of clojure.string instead? I considered putting it there, but thought it would be confusing to have multiple substring functions in different namespaces.

This is very helpful in practice, and I'd really like to see at least the (subs) stuff in Clojure.

Disclaimer: I'm no Clojure/core member, so can't speak authoritatively on whether something would or would not be accepted into clojure.string.

However, given that clojure.string is distributed with clojure.core, my guess is it would not be accepted. You'd be able to get things like this out for you and others as a separate library distributed on Clojars. That would also make it easy to include other Python-like things that you don't find in Clojure already.

This isn't about "python-like things," this is about a useful feature. Lots of languages support this: Perl, PHP, Ruby, Python, JavaScript, to name a few. Are you really suggesting that I should create a whole package for a version of a function in clojure.core with slightly different semantics? That's insane.

Anyway, I'm done wasting my time trying to navigate this hopelessly broken process. Maybe it would have been accepted if I included yet another way to interoperate with Java types.

Stuart, do you remember why specifically negative indexes were rejected? Developing a separate library for a minor improvement to an existing function sounds unreasonable.

some explanation about this topic was given by Rich Hickey himself here: http://news.ycombinator.com/item?id=2053908

citation:
"...Yes, there is a backlog from when it was just me, and it will take a while to whittle down. We have finite resources and have to prioritize. I can assure you we have more important things to concentrate on than your negative index substring enhancement, and are doing so. You'll just have to be patient. Or, if you insist, I'll just reject it now because a) IMO it's goofy, b) you can make your own function that works that way c) we don't get a free ride from j.l.String, d) it begs the question of negative indices elsewhere..."

i've been following this thread hoping this feature would be included. but whatever the reason was for the rejection, i'm sure it was thoughtful. great thanks for this wonderful language, and thanks Ian Eure for his effort.

That HN link eventually leads back to CLJ-688 which was rejected.

Michael: A proposal for negative indexes would need to be systematic in considering implications for all functions that have index arguments.

Ian: Clojure is driven by design, not incremental piling of features.

All: clojure.string is incomplete in more important and fundamental ways than negative indexes. This sucks now, and will suck worse as more code is written in different dialects. I find myself wishing string was a contrib, so we could iterate faster.

Stuart: Any specific proposals for how you'd like to see clojure.string improve? If it can be made a contrib, that would be cool, but understood if that would be considered too breaking of a change. Even if it isn't made a contrib, having tickets for improvement ideas you are willing to see means patches might get written, and they'll get in some time.

First of all, clojure.lang.IFn is no protocol but an interface. And it does not declare a

  public Object invoke(Object... obs)

method. It has an `invoke` method with 20 Object parameters followed by an Object... parameter, but to give an implementation for that, you have to specify every parameter separately, and the last Object... arg is just a normal argument that must be an Object[]. That's because Java-varags Type... parameters are just Java syntax sugar, but in the byte-code its simply a Type-array.

What your example does is provide an `invoke` implementation for the 2-args version, where the first parameter happens to be named `&`, which has no special meaning here. Take that example:

(deftype MyFoo []
  clojure.lang.IFn
  (invoke [this & xs]
    [& xs]))

((MyFoo.) 1 2)
=> [1 2]

But you are right in that `deftype`, `defrecord`, `defprotocol`, and `definferface` probably should error if user's seem to try to use varargs or destructuring.

Cheers for a most clarifying response.

The fact that the meaning of & gets 'subverted' makes the issue only (slightly) worse, in my opinion.

Just for the record, destructuring seems to work, at least for interface impls.

> The fact that the meaning of & gets 'subverted' makes the issue only (slightly) worse, in my opinion.

I agree. I'll attach a patch which checks for those invalid usages soon.

> Just for the record, destructuring seems to work, at least for interface impls.

Could you please provide a complete example demonstrating your statement?

I'm rather sure that varargs and destructuring don't work for any of defprotocol, definterface, deftype, defrecord, and reify. But you can use varargs and destructuring when providing dynamic implementations via `extend` (or `extend-protocol`, `extend-type`), because those impls are real functions in contrast to methods.

I attached a patch. Here's the commit message:

Check for invalid varags/destrucuring uses.

Protocol, interface method declarations and implementations don't allow for
varags and destructuring support. Currently, for example

(defprotocol FooBar
(foo [this & more]))

compiles just fine, and & is interpreted as a usual argument that happens to be
named & without special meaning. But clearly, the user wanted to specify a
varags parameter here. The same applies to definterface.

Similarly, providing method implementations via defrecord, deftype, and reify
don't allow for destructuring and varags (but dynamic extenions via extend do).

So this patch makes defprotocol, definterface, defrecord, deftype, and reify
throw an IllegalArgumentException if any argument vector contains a
destructuring form or varargs argument.

Glad you've considered my request.

As for destructuring, I was speaking after this example, which may or may not work like it looks like - I don't know.

(deftype foo []
clojure.lang.IFn
(invoke [this [a b] c] (println a b c)))

((foo.) [1 2] 3)

Indeed, descructuring seems to work for method implementations. I'll adapt my patch...

Revamped patch. Here's the commit message:

Protocol, interface method declarations don't allow for varags and
destructuring support. Currently, for example

  (defprotocol FooBar
    (foo [this & more]))

compiles just fine, and & is interpreted as a usual argument that happens to be
named & without special meaning. But clearly, the user wanted to specify a
varags parameter here. The same applies to definterface.

Similarly, providing method implementations via defrecord, deftype, and reify
don't allow for varags (but dynamic extenions via extend do).

So this patch makes defprotocol and definterface throw an
IllegalArgumentException if a user tries to use varargs and destructuring in
method signatures.

Similarly, defrecord, deftype, and reify throw an IllegalArgumentException if
any method implementation arglist contains a varargs argument.

Tassilo, with your patch 0001-CLJ-1024-Check-for-invalid-varags-destrucuring-uses.patch dated July 12, 2012, I get the following error message while testing, apparently because some metadata is missing on the new functions your patch adds to core:

[java] Testing clojure.test-clojure.metadata
[java]
[java] FAIL in (public-vars-with-docstrings-have-added) (metadata.clj:45)
[java] expected: (= [] (remove (comp :added meta) public-vars-with-docstrin
gs-not-generated))
[java] actual: (not (= [] (#'clojure.core/throw-on-varargs-and-destr #'cl
ojure.core/throw-on-varargs)))

Hi Andy, this updated patch declares the two new checking fns private which makes the tests pass again.

Stupid mistake by me: Of course, I've tested the last version, too, but afterwards I decided it wouldn't be bad to add some docstrings, and of course, adding docstrings cannot break anything.

Patch applies cleanly against 4004d267e124f12b65b0d7fb6522f32a75e3c4fb. Submitter confirmed as a CA signer.

This looks ok to me, but it seems like a fair amount of duplication to accomplish the task. It seems like we should just be able to ask if it is ok to proceed, instead of having to pick which function needs to be called in what case.

Aaron, can you please elaborate? I don't get what you mean with duplication and asking if it is ok to proceed.

Rebased to apply cleanly on master again.

Pardon the silly contribution, but the added methods' usage of double-negations (when-not ...) seems unnecessary.

Hi Victor, this revamped patch removes the double-negation and is more concise and clearer as a result. So your comment wasn't as silly as you thought.

Hey Stu, do you mind to explain why you've declined the patch?

@Tassilo: https://groups.google.com/forum/#!topic/clojure-dev/qjkW-cv8nog

@Marek, Stu: Thanks, I've left a reply there: https://groups.google.com/d/msg/clojure-dev/qjkW-cv8nog/rMNFqbjNj-EJ

Agree it is good to fix the inconsistency, but I think an infinite sequence of zeros is the correct result, as implied by the current docstring definition.

It's also mathematically cleanest: range should produce an arithmetic progression until the end value is equalled or exceeded.

Empty lists only seem to make sense as a return value when start >= end.

Hi Mike,

Could you explain how you think the docstring definition implies this behavior? Maybe I'm missing something, but I was surprised. For instance, in the case of (range 3 9 0), if start were truly inclusive as the docstring says, the result would be (3), not ().

You mentioned that you think the infinite sequence of 0's is consistent and in keeping with the definition of range. I'm not sure I agree. (0,0] is an empty set of length zero, and [0,0) is an empty set of length zero.

You state that empty list only makes sense for start >= end, except this is not the current behavior. Could you help me understand what you believe the appropriate behavior would be in each of the following three cases? (range 0 10 0), (range 10 0 0), and (range 0 0 0)?

A few options to consider:
1) Fix the docstring to reflect the actual behavior of range.
2) Handle the case of (range 9 3 0) => (9 9 9 ...) to make it consistent with the behavior of (range 3 9 0) => ()
3) Stop allowing a step of zero altogether.

Editing to Add (Note: I don't think the way someone else did something is always the right way, just wanted to do some digging on how other people have handled this in the past):
http://docs.python.org/library/functions.html#range (0 step returns ValueError)
http://www2.tcl.tk/10797 (range returns empty list for a zero step)
http://www.scala-lang.org/api/2.7.7/scala/Range.html (zero step is not allowed)

It does make sense NOT to allow a step of zero (at least to me)... I wasn't going to say anything about this but if other popular languages do not allow 0, then I guess it makes more sense than I originally gave myself credit for... However, if we want to be mathematically correct then the answer would be to return an infinte seq with the start value like below. After all, what is a step of 0? does it make any difference how many steps you take if with every step you cover 0 distance? obviously not...you start from x and you stay at x forever...we do have repeat for this sort of thing though...

(take 5 (range 3 9 0)) => (3 3 3 3 3)

+1 for not allowing 0 step

@Devin quite simple: a lazy sequence of nums starting from x with step 0.0 until it reaches an end value of y (where y > x) is an infinite sequence of x.

Consider the case where start is 0 and end is infinity (the default), you would expect sequences to go as follows:

step +2 : 0 2 4 6 8....
step +1 : 0 1 2 3 4....
step +0 : 0 0 0 0 0....

It would be inconsistent to make 0 a special case, all of these are valid arithmetic progressions. And all of these are consistent with the current docstring.

If you make 0 a special case, then people will need to write special case code to handle it. Consider the code to create a multiplication table for example:

(for [x (range 10)]
(take 10 (range 0 Long/MAX_VALUE x)))

This works fine if you produce an infinite sequence of zeros for step 0, but fails if you create an empty list as a special case for step 0.

As a related issue, I'd personally also favour negative step sizes also producing an infinite sequence. If we don't want to allow this though, then at least the docstring should be changed to say "a lazy seq of non-decreasing nums" and a negative step should throw an error.

Carrying over a message from the clojure-dev list by Stuart Sierra:

• I called the ticket a defect. Does that seem reasonable?

yes

• Does anyone actually use the 0 step behavior in their programs?

not if they have any sense

• Has anyone been bitten by this in the past?

not me

• Is this behavior intentional for some historical reason I don't know about or understand?

I doubt it.

• Has this been brought up before? I couldn't find any reference to it via Google.

Not that I know of.

• Are there performance implications to my patch?

I doubt it. Lazy seqs are not ideal for high-performance code anyway.

• Am I addressing a symptom rather than the problem?

I think the problem is that the result of `range` with a step of 0 was never specified. Don't assume that the tests are specifications. Many of the tests in Clojure were written by over-eager volunteers who defined the tests based on whatever the behavior happened to be. The only specification from the language designer is the docstring. By my reading, that means that `range` with a step of 0 should return an infinite seq of the start value, unless the start and end values are equal.

-S

Carrying over a message by Michal Marczyk:

With a negative step, the comparison is reversed, so that e.g.

(range 3 0 -1)

evaluates to

(3 2 1)

I think this is the useful behaviour; the docstring should perhaps be
adjusted to match it.

Agreed on zero step.

Cheers,
Michał

Adding a new patch after hearing comments. This patch makes (range 9 3 0) => (9 9 9 9 ...), (range 3 9 0) => (3 3 3 3 ...), and () will be returned when (= start end). Also updated the docstring.

Marking as vetted

Patch applies cleanly. We've discussed this issue to death (for as simple as it is). I think it's time to mark it as screened.

For some reason I'm not allowed to edit the attachments list. When you apply the patch, please apply inconsistent_range_fix.diff as that is the most recent version of the fix.

As someone who has to read these tickets, I'd really appreciate it if you could keep the description up to date and accurate, with examples of the current and intended behavior and the strategy to be used in fixing. I can't follow every thread to see what the latest thinking is, especially on a patch like this where the original mission was changed.

Thanks

@Tim: I've removed the other attachments.

@Rich: Understood. I will update the description this evening.

Philip, thanks for writing that patch. Could you please attach it as a file to this ticket, instead of copying and pasting it into a comment? Instructions are under the heading "Adding patches" on this page:

http://dev.clojure.org/display/design/JIRA+workflow

Rich Hickey's policy is only to include code in Clojure that is written by those who have signed a contributor agreement. You can find out more at http://clojure.org/contributing

Thanks Andy. Patch attached:

0001-clojure.data-diff-cope-with-falsey-values-in-maps.patch

I'll send in a CA.

CA snail-mailed yesterday, I guess it will take a week to arrive.

I now have a CA in place.

Yeah, this should be fixed.

Patch applies cleanly against 4004d267e124f12b65b0d7fb6522f32a75e3c4fb. Submitter is a confirmed CA signer.

Here's an implementation.

There's something strange with my patch. The creation of a composition of a huge seq of functions is much faster with `comp*` than with `comp`, which is expected, because the seq doesn't need to be reversed.

The strange thing however is that the compositions created with `comp` evaluate about 10-20% faster than those created with `comp*` although the arbitrary-arity version of `comp` is defined in terms of `comp*`: `(apply comp* (reverse (list* f1 f2 f3 fs)))`.

For some benchmarking details, see: https://groups.google.com/d/msg/clojure/MizwTxHwLE4/hGLrMfetlP8J

Here's some benchmark:

user> (use 'criterium.core)
nil
user> (let [coll (doall (take 1000000 (repeat inc)))
	   f1 (apply comp* coll) 
	   f2 (apply comp coll)] 
	   (bench (f1 0) :verbose) 
	   (println "---------------------------------------")
	   (bench (f2 0) :verbose))
amd64 Linux 3.4.2-gentoo 2 cpu(s)
OpenJDK 64-Bit Server VM 22.0-b10
Runtime arguments: -agentlib:jdwp=transport=dt_socket,server=y,suspend=n -XX:+TieredCompilation -Xmx1G -Dclojure.compile.path=/home/horn/Repos/clj/testi/target/classes -Dtesti.version=0.1.0-SNAPSHOT -Dclojure.debug=false
Evaluation count             : 600
             Execution time mean : 112.324465 ms  95.0% CI: (112.247218 ms, 112.380682 ms)
    Execution time std-deviation : 6.513809 ms  95.0% CI: (6.477450 ms, 6.553029 ms)
         Execution time lower ci : 105.609401 ms  95.0% CI: (105.609401 ms, 105.622918 ms)
         Execution time upper ci : 122.353763 ms  95.0% CI: (122.353763 ms, 122.405315 ms)
---------------------------------------
amd64 Linux 3.4.2-gentoo 2 cpu(s)
OpenJDK 64-Bit Server VM 22.0-b10
Runtime arguments: -agentlib:jdwp=transport=dt_socket,server=y,suspend=n -XX:+TieredCompilation -Xmx1G -Dclojure.compile.path=/home/horn/Repos/clj/testi/target/classes -Dtesti.version=0.1.0-SNAPSHOT -Dclojure.debug=false
Evaluation count             : 1440
             Execution time mean : 43.519663 ms  95.0% CI: (43.516732 ms, 43.524062 ms)
    Execution time std-deviation : 492.299089 us  95.0% CI: (490.829889 us, 494.198137 us)
         Execution time lower ci : 42.781398 ms  95.0% CI: (42.781398 ms, 42.781398 ms)
         Execution time upper ci : 44.157311 ms  95.0% CI: (44.157311 ms, 44.158513 ms)
nil

Ok, I've tracked down the performance difference. This comes from the fact that `reverse` creates a clojure.lang.PersistentList which can be iterated much faster than a (fully realized) LazySeq. If you provide your fncoll in `(apply comp* fncoll)` as vector or list, the application of the created composition is as fast as for `comp`.

So for me, the patch is good and makes sense.

I'd like to see something much faster than this.

For reference here is a benchmark of a non-reducers (seq based) process that uses partition-by

user=> (def x (vec (range 1e6)))
#'user/x
user=> (bench (reduce + (map count (partition-by #(or (zero? (mod % 3)) (zero? (mod % 5))) x))))
Evaluation count             : 60
             Execution time mean : 1.072157 sec  95.0% CI: (1.070606 sec, 1.073381 sec)
    Execution time std-deviation : 165.818282 ms  95.0% CI: (163.873585 ms, 168.271261 ms)
         Execution time lower ci : 972.562000 ms  95.0% CI: (972.562000 ms, 973.301850 ms)
         Execution time upper ci : 1.419148 sec  95.0% CI: (1.419148 sec, 1.419148 sec)

Found 7 outliers in 60 samples (11.6667 %)
	low-severe	 2 (3.3333 %)
	low-mild	 5 (8.3333 %)
 Variance from outliers : 85.8489 % Variance is severely inflated by outliers
nil
user=>

Same again using r/partition-by from reducer-partition-by.diff

user=> (bench (r/reduce + (r/map count (r/partition-by #(or (zero? (mod % 3)) (zero? (mod % 5))) x))))
Evaluation count             : 60
             Execution time mean : 1.418350 sec  95.0% CI: (1.417738 sec, 1.418948 sec)
    Execution time std-deviation : 66.736477 ms  95.0% CI: (66.186568 ms, 67.610777 ms)
         Execution time lower ci : 1.370419 sec  95.0% CI: (1.370419 sec, 1.370419 sec)
         Execution time upper ci : 1.544151 sec  95.0% CI: (1.544151 sec, 1.544156 sec)

Found 10 outliers in 60 samples (16.6667 %)
	low-severe	 2 (3.3333 %)
	low-mild	 8 (13.3333 %)
 Variance from outliers : 33.5591 % Variance is moderately inflated by outliers
nil
user=> 

(using https://github.com/hugoduncan/criterium for benchmarking)

same again for r/partition-by from reducers-partition-by2.diff

user=> (bench (r/reduce + (r/map count (r/partition-by #(or (zero? (mod % 3)) (zero? (mod % 5))) x))))
Evaluation count             : 180
             Execution time mean : 307.596806 ms  95.0% CI: (307.271339 ms, 307.961550 ms)
    Execution time std-deviation : 34.060809 ms  95.0% CI: (33.613169 ms, 34.416837 ms)
         Execution time lower ci : 285.339333 ms  95.0% CI: (285.339333 ms, 285.339333 ms)
         Execution time upper ci : 385.087950 ms  95.0% CI: (385.087950 ms, 385.087950 ms)

Found 10 outliers in 60 samples (16.6667 %)
	low-severe	 4 (6.6667 %)
	low-mild	 6 (10.0000 %)
 Variance from outliers : 73.8053 % Variance is severely inflated by outliers
nil
user=> 

same again driven using r/fold (for grins) instead of r/reduce

user=> (bench (r/fold + (r/map count (r/partition-by #(or (zero? (mod % 3)) (zero? (mod % 5))) x))))
Evaluation count             : 360
             Execution time mean : 215.214486 ms  95.0% CI: (214.915417 ms, 215.664236 ms)
    Execution time std-deviation : 36.588464 ms  95.0% CI: (36.305548 ms, 36.847846 ms)
         Execution time lower ci : 185.575000 ms  95.0% CI: (185.575000 ms, 185.575000 ms)
         Execution time upper ci : 287.605175 ms  95.0% CI: (286.547833 ms, 287.605175 ms)

Found 6 outliers in 60 samples (10.0000 %)
	low-severe	 3 (5.0000 %)
	low-mild	 3 (5.0000 %)
 Variance from outliers : 87.6303 % Variance is severely inflated by outliers
nil
user=> 

reducers-partition-by2.diff is faster, but I am not wild about introducing a type and a protocol. also not sure about the CollFold impl.

Let's leave fold out for this first patch, please.

reducer-partition-by3.diff is a cleaned up version of reducer-partition-by2.diff without fold

Per Andy Fingerhut's email reducer-partition-by3.diff was failing to apply. This patch should apply cleanly to current master.

Presumptuously changing Approval from Incomplete to None, since the reason for its being marked Incomplete seems to have been addressed with the latest patch.

should this be assigned to someone?

applied - thanks!

Test included.
Patch should ensure that flushing happens after pprint, without introducing any additional unnecessary flushes.

I am confused by the tests – they all seem to call prn, though some claim to be calling pprint.

Ah, sorry. I'd tried to remove duplicate tests before committing them, but removed the wrong ones.
Replacement patch attached:

0002-pprint-now-flushes-the-underlying-stream-similarly-t.patch

Tempting fate once again by changing Approval from Incomplete to None after the reason it was marked as incomplete seems to have been addressed.

Screened.

Pushed in commit 1588ff3f70e864d9817bc565bd2c30b08189bc6e

Any

I would prefer to see this, and other interop/reflective items, in a separate contrib lib.

Thanks for the feedback, Stu. Is there an existing contrib lib for stuff like this? Or should I create one?

You need to attach a patch so this can be assessed. Thanks!

Sorry David! I added a patch a while ago but forgot to add the patch label(s?) as well as leave a comment.

The patch may be out of date now, but I haven't checked. Hopefully the prescreening process will pick it up automatically run the tests.

Yes, Brandon, your patch has been on the list of prescreened patches since April 15. It has always applied and built cleanly during that time. That patch label is nice to have for certain JIRA report filters, but isn't necessary for the prescreening process to pick it up.

v2 now applies against the current master (191b05f1). The original patch seemed to be broken from a whitespace perspective, which was making it difficult to apply--even in a 3-way merge. The only real conflict was in Compiler.java where a "final int line" was added to CompilerException. All the tests passed.

It looks like the line field was added to CompilerException in commit 89245c68, but that commit doesn't use it for anything. Maybe a later commit uses it? Also, if we want to keep that field, can we also add a column field for patch v3?

This patch is an enhancement. In order for this one to make any movement I believe it will need a design page outlining the problem, what this solves / alternatives.

http://dev.clojure.org/display/design/Compiler+tracking+of+columns

I added a design page, but it seems like overkill. This is a straightforward enhancement...

I've applied the v2 patch (thanks!), but before we close the ticket can we please get a patch comprising some tests?

Attached CLJ-960-tests.diff to verify :line and :column metadata as now provided by LineNumberingPushbackReader.

The tests are a little fragile (exact map comparison) and so will break if the reader ever does more things. In library and app projects I write tests like this using core.match, but that isn't available as a build dependency here.

Marking screened anyway.