Please add a patch formatted by "git format-patch" so that attribution is included.

I added the formatted patch a few days ago. Does 'no news is good news' apply here?

And, silly question, will it make it into 1.3? I can't figure out how to tell Jira to show me that.

I fail to see the need for an IAtom, if you want something atom like for couchdb the interfaces are already there. Maybe I ICompareAndSwap. Atoms and couchdb are different so making them appear the same is a bad idea.

http://en.wikipedia.org/wiki/Fallacies_of_Distributed_Computing

http://clojure.org/state one of the distinctions between agents and actors raised in the section titled "Message Passing and Actors" is local vs. distributed and the same distinction can be made between couchdb (regardless of compare and swap) and atoms

This ticket has already been moved into approved backlog. It will be revisited again after the 1.3 release where we will take a closer look at things. For now, this will remain as is.

In Clojure, it is generally the case that redefining something replaces the original, as opposed to augmenting it by merging the old and the new. This makes it easy to reason locally about how code works. One could argue that the following snippet has the same kind of issue you describe:

(defn foo [a] 1)
(defn foo [a b] 1)
(foo 1) ; is it a bug that the first arity is gone?

I also wonder whether the current behavior might be a convenience for some macros. (Clearly it wasn't for yours!) I am changing the type and title of the ticket to better reflect the nature of the request and see what the BDFL says.

Everything in this ticket needs to be said with more precision. I don't know what exactly the problem is nor what the proposed solution is.

One thing to note is that it is necessary to accept definitions under base interfaces, so the class areas are not strict, nor expected to be complete.

Stuart: The two foo forms you give are entirely separate, and to unify the two behaviors you would have to group them together. It's not at all unreasonable to suppose the user wants to define foo once, fiddle with it, and then redefine it - clojure.core does similar stuff with let, reduce, etc.

As written, deftype/reify have a somewhat similar "look" - because there is nothing physically grouping the declaration of Map with its functions, it's not clear what should happen when a heading like Map is given twice, and it's not specified in the docs.

I think the difference is that in the reify case, the two are in the same top-level form, so the compiler can detect that you're trying to do something "weird", so a silent redefinition (reasonable for your defn example) is surprising. There are a number of solutions that would reduce this surprise:

1) Permit or require reify to group things, as in (reify (Comparable (compare [this other] 1))). Then the explicit grouping of Comparable with its methods serves two purposes: it implies that other definitions for Comparable should be included in that grouping; and it makes it easier to do that, because you can just iterate over forms until you find Comparable, and then insert another definition.

2) Throw an exception if an interface is specified twice. This is not ideal because it can be a lot of work for the user to group things together themselves, while it's easy for deftype to do given the grouping it's already doing. However, it would avoid the confusion and surprise, by saying "that's not allowed" rather than leaving the user guessing what's gone wrong.

3) Interpret my original example code as an attempt to open the Map interface, add implementations, and then later add some more implementations.

I would have liked reify to implement (1) to begin with, but at this point I don't think the syntax is backwards-compatible, so it doesn't seem like a good idea. I suppose either (2) or (3) is fine, and they both seem like an improvement over the current confusing behavior. Of course, I prefer (3), but I can understand a desire to make reify reject syntax that is not immediately obvious in intent rather than interpreting it as what I think is the most useful intent.

I was not sure whether the reported arity should match the actual Java method arity or Clojure protocol function arity (including the target). I did the former.

I think it should be the latter. The message is emitted when the protocol methods are being invoked through the corresponding function, so it should be consistent with the errors emitted by regular functions.

+1 for some tests, too. There certainly are tests for reflection warnings and such.

FWIW, I'm happy to take this on if Alex is otherwise occupied.

I am not sure I like this, but I would like a rethink of names and namespaces. Doing a lot of cross language work, it would be great to have protocols for "I have a name" and for "I have a namespace".

With such protocols in place, it would also be possible to separately consider implementing symbol et al in terms of them.

Named being a protocol or an interface seems orthogonal to being able to create a symbol qualified with a namespace when you have a namespace in hand.

I don't think the patch goes far enough, not only should (symbol ns "foo") be supported, but also (symbol ns 'foo), given that (symbol 'foo) works and (symbol "foo") works, (symbol 'bar 'foo) should also work, but doesn't.

if Named is a protocol, and if you extend it to String, and if you make the symbol function create symbols from one or two Named things you still end up having to do (symbol (ns-name ns) 'foo) or (symbol (ns-name ns) "foo")

Stuart, I'm not opposed to the idea of separate protocols for Named and Namespaced. Where should I go about creating a proposal to create those protocols and get them into clojure? I'm interested in doing the leg-work, or being a part of it. But as an outsider, I don't know what to do next – creating a ticket in Jira exhausted my knowledge of the process.

The same enhancement that joe suggests for symbol, would also apply to keyword.

See: http://groups.google.com/group/clojure/browse_thread/thread/222e4abc16df8b20

Probably same/similar solution applies to both issues.

-FrankS.

All

diff.patch of Nov 27, 2011 does not apply cleanly to latest master version of Clojure code (using "patch -p1 < diff.patch", at least). It is preferred by Clojure team that patches are in git format-patch format. Instructions for producing such a patch are given at http://clojure.org/patches

Rahul, are you planning to sign a Clojure Contributor Agreement? Without that, this code cannot be included in Clojure, unless a contributor reimplements it on their own.

In private communication with the patch author today, he expressed an interest in submitting a signed CA so this patch can be considered for inclusion in Clojure.

Converted from http://www.assembla.com/spaces/clojure/tickets/308

Added a CloseableResource protocol and extended it on java.io.Closeable (implemented by all Readers, Writers, Streams, Channels, Sockets). Use it in with-open.

All tests pass.

Seems to be related to Scopes (http://dev.clojure.org/jira/browse/CLJ-2).

Updated patch.

Patch 0001-Added-ClosableResource-protocol-for-with-open.patch dated 08/Mar/12 applies, builds, and tests cleanly on latest master as of Apr 2 2012. Tassilo has signed a CA.

Updated patch to apply cleanly against master again.

Converted from http://www.assembla.com/spaces/clojure/tickets/5

Please see the attached patch which produces a (hopefully more clear) error message as shown below (given the broken code shown in the original bug report):

Clojure 1.4.0-master-SNAPSHOT
user=> (let [x 42 y 43] (+ x y))
85
user=> (let [[x y] {}] x)
UnsupportedOperationException left side of binding must be a symbol (found a PersistentVector instead).  clojure.lang.Compiler.checkLet (Compiler.java:6545)
user=>

In addition, this patch checks the argument of (let) as shown below:

user=> (let 42)
UnsupportedOperationException argument to (let)  must be a vector (found a Long instead).  clojure.lang.Compiler.checkLet (Compiler.java:6553)

Patch produced by doing git diff against commit ba930d95fc (master branch).

Sorry, this patch is wrong: it assumes that the left side of the binding is wrong - the [x y] in :

(let [[x y] {}] x)

because [x y] is a vector, when in fact, the left side is fine (per http://clojure.org/special_forms#let : "Clojure supports abstract structural binding, often called destructuring, in let binding lists".)

So it's the right side (the {}) that needs to be checked and flagged as erroneous, not the [x y].

Add patch better-error-for-let-vector-map-binding

This produces the following:

(let [[x y] {}] x)
Exception map binding to vector is not supported

There are other cases that are not handled by this though — like binding vector to a set

user=> (let [[x y] #{}] x)
UnsupportedOperationException nth not supported on this type: PersistentHashSet

Wondering if it might be better to try convert the map to a seq to support? Although this might be another issue.

Thoughts?

This seems too specific. Is this issue indicative of a larger problem that should be addressed? Even if this is the only case where bindings produce poor error messages, all the cases described above should be addressed in the patch.

Unfortunately, realized that this still does not cover the nested destructuring cases. Coming to the conclusion, that my approach above is not going to work for this.

File: clj-5-destructure-error.diff

Added support for nested destructuring errors

let [[[x1 y1][x2 y2]] [[1 2] {}]]
;=> UnsupportedOperationException let cannot destructure class clojure.lang.PersistentArrayMap.

Converted from http://www.assembla.com/spaces/clojure/tickets/248

awizzard said: Another nice addition could be java.util.NavigableMap, though this is available only since Java 6.

SortedSet-SortedMap-interfaces-248.diff 2012-05-10 is a first pass at the SortedMap interface. If there are no objections, I'll build on the SortedMap implementation to add SortedSet support.

SortedMap-interfaces-248.diff 2012-05-10 supersedes the last diff. It implements and tests the SortedMap interface.

Additionally it includes tests for `sorted-map`, which didn't seem to be tested yet.

SortedMap-SortedSet-interfaces-248.diff 2012-05-18 supersedes the last diff. It implements and tests both SortedMap and SortedSet.

(Concurrent)NavigableMap is not implemented because I think 1.5 is still targeted. In any case, it probably should have its own ticket.

clj-248-SortedMap-SortedSet-interfaces-patch2.txt dated May 24, 2012 is simply an updated version of SortedMap-SortedSet-interfaces-248.diff dated May 17, 2012. The latter patch no longer applied cleanly.

The new patch combines what was internally multiple git commits into one, simply because it was easier for me to create the patch that way. I left out the addition of *.swp and *.swo files to the .gitignore file. Builds and tests cleanly on Oracle/Apple JDK 1.6 on Mac OS X 10.6.8.

I don't know why Priority changed to Blocker when I changed Patch to "Code and Test". I didn't try to do that – suspect that something with the web UI is doing that automatically somehow.

Added CLJ-1008 for NavigableMap interface enhancement.

So I went ahead and did the work of making this change in clojure.core/defmacro instead of clojure.lang.Compiler/macroexpand1. It was even worse than I expected: I didn't realize we don't yet have syntax-quote or apply at this stage in bootstrapping, so writing a non-trivial macroexpansion requires a huge amount of (list `foo (list `bar 'local-name)) and so forth.

I'm sure the version I wrote is not optimal, but it seemed simpler to piggyback on defn, and then use alter-var-root to shim the metadata management in, than it would have been to expand to the correct thing in the first place.

Anyway, attached patch #3 could be applied instead of #2 to resolve the issue in clojure.core instead of clojure.lang. The tests added in patch #1 pass either way.

I realized I can do this with a named private function instead of an anonymous function, reducing the amount of mess defmacro itself has to generate. Patch 4 is, I think, strictly better than Patch 3, if a Clojure implementation is preferred to one in Java.

I prefer patch 0002 in Java over either 0003 or 0004. Patch 0002 keeps the knowledge of how to invoke macro fns (specifically the extra &form and &env args) in one place, macroexpand1 rather than duplicating that knowledge in core.clj as well. Note patch 0001 is just tests.

The proposed default macroexpansion behavior is more useful than what we currently have, but there are two details I'd like to think about a bit more:

1) In exchange for a more useful default, macro writers lose the ability to consume their &form metadata and have control over the resulting form metadata without the &form metadata overridding it. That is, macros are no longer in complete control of their output form.

2) Rule (1) above has hardcoded exceptions for :line and :file, where &form metadata is unable to override the results returned by the macro.

This patch incorporates all previous patches to this issue.

On the clj-dev mailing list, Andy Fingerhut suggested a new metadata key for allowing the macro author to specify "I've looked at their &form metadata, and this form is exactly what I want to expand to, please don't change the metadata any further." I've implemented this, and I think it addresses Chouser's concern about needing a way to "break out" of the improved-default behavior.

One open question is, is :explicit-meta the right key to use? I spent some time tracking down a bug caused by my forgetting the keyword and using :explicit-metadata in my test; perhaps something more difficult to get confused by is available.

Here's an implementation.

There's something strange with my patch. The creation of a composition of a huge seq of functions is much faster with `comp*` than with `comp`, which is expected, because the seq doesn't need to be reversed.

The strange thing however is that the compositions created with `comp` evaluate about 10-20% faster than those created with `comp*` although the arbitrary-arity version of `comp` is defined in terms of `comp*`: `(apply comp* (reverse (list* f1 f2 f3 fs)))`.

For some benchmarking details, see: https://groups.google.com/d/msg/clojure/MizwTxHwLE4/hGLrMfetlP8J

Here's some benchmark:

user> (use 'criterium.core)
nil
user> (let [coll (doall (take 1000000 (repeat inc)))
	   f1 (apply comp* coll) 
	   f2 (apply comp coll)] 
	   (bench (f1 0) :verbose) 
	   (println "---------------------------------------")
	   (bench (f2 0) :verbose))
amd64 Linux 3.4.2-gentoo 2 cpu(s)
OpenJDK 64-Bit Server VM 22.0-b10
Runtime arguments: -agentlib:jdwp=transport=dt_socket,server=y,suspend=n -XX:+TieredCompilation -Xmx1G -Dclojure.compile.path=/home/horn/Repos/clj/testi/target/classes -Dtesti.version=0.1.0-SNAPSHOT -Dclojure.debug=false
Evaluation count             : 600
             Execution time mean : 112.324465 ms  95.0% CI: (112.247218 ms, 112.380682 ms)
    Execution time std-deviation : 6.513809 ms  95.0% CI: (6.477450 ms, 6.553029 ms)
         Execution time lower ci : 105.609401 ms  95.0% CI: (105.609401 ms, 105.622918 ms)
         Execution time upper ci : 122.353763 ms  95.0% CI: (122.353763 ms, 122.405315 ms)
---------------------------------------
amd64 Linux 3.4.2-gentoo 2 cpu(s)
OpenJDK 64-Bit Server VM 22.0-b10
Runtime arguments: -agentlib:jdwp=transport=dt_socket,server=y,suspend=n -XX:+TieredCompilation -Xmx1G -Dclojure.compile.path=/home/horn/Repos/clj/testi/target/classes -Dtesti.version=0.1.0-SNAPSHOT -Dclojure.debug=false
Evaluation count             : 1440
             Execution time mean : 43.519663 ms  95.0% CI: (43.516732 ms, 43.524062 ms)
    Execution time std-deviation : 492.299089 us  95.0% CI: (490.829889 us, 494.198137 us)
         Execution time lower ci : 42.781398 ms  95.0% CI: (42.781398 ms, 42.781398 ms)
         Execution time upper ci : 44.157311 ms  95.0% CI: (44.157311 ms, 44.158513 ms)
nil

Ok, I've tracked down the performance difference. This comes from the fact that `reverse` creates a clojure.lang.PersistentList which can be iterated much faster than a (fully realized) LazySeq. If you provide your fncoll in `(apply comp* fncoll)` as vector or list, the application of the created composition is as fast as for `comp`.

So for me, the patch is good and makes sense.

all

One apparent problem with this patch is that you throw an exception for :refer. You should add that, and make sure there aren't any others missing. Also, #{x y z} is better than (set [x y z]), and you should probably use pr-str rather than str, although I can't think of a case where it matters for the objects in question.

A more minor detail of patch formatting – please attach your patch in git format. See the instructions under the section heading "Development" on this web page: http://dev.clojure.org/display/design/JIRA+workflow

git format-patch version of the diff, with the edits suggested by other maintainers.

Alan: please note that :refer was not mentioned in the docstring for ns, or used in any of the unit tests for clojure.

Are you sure that it is an expected argument, or just an arrangement that happens to work under the current ns macro? The docstring for 'refer itself says to use :use in ns macros instead of calling refer.

I added "refer" to the set of accepted references all the same.

Implemented range. A separate commit is attached, making iterate and range also Seqable, since I'm not sure if that's desired. Apply it or not, as you prefer.

Range should be foldable

Yep, so it should. Time for me to dig into the folding implementations!

Should I fold (har har) all of these commits into one? I don't know what is preferred on JIRA, and I also don't know whether range/iterate should be seqable or if I should just drop the second commit.

Yes, please merge these together, it's hard to see otherwise (I can barely read diffs as is . range and iterate shouldn't be novel in reducers, but just enhanced return values of core fns. The enhancement (e.g. protocol extensions) can come by requiring reducers since it can't be leveraged without it. Also, I'm not sure how I feel about an allocating protocol for 'splittable' - I've avoided it thus far.

So you want clojure.core/range to return some object (a Range), which implements Counted and Seqable (but isn't just a lazy-seq), and then inside of clojure.core.reducers I extend CollReduce and CollFold to that type? Okay, I can do that.

I don't quite follow what you mean by an allocating protocol. I see your point that my fold-by-halves which takes a function in is analogous to a protocol with a single function, but it doesn't allocate anything more than foldvec already does - I just pulled that logic out so that the fork/join fiddly work doesn't need to be repeated in everything foldable. Do you have an alternative recommendation, or is it just something that makes you uneasy and you're still thinking about?

While vector-fold allocs subvecs, the halving-fn must return a new vector, for all implementations. It's ok, I don't think it's likely to dominate (since fj needs new closures anyway). Please proceed, but keep range and iterate in core. They are sources, not transformers, and only transformers (which must be different from their seq-based counterparts) must reside in reducers. Thanks!

One big patch file is preferred, although that file may contain multiple commits if that makes the intent clearer.

When adding a patch, update the description of the ticket to indicate which file is the most recent. Leave old patch files around for historical reference.

It's looking harder than I expected to move iterate and range into core.clj. My plan was to just have them implement Seqable, which is easy enough, but currently they are actually instances of ISeq, because they inherit from LazySeq. A bunch of code all over the place (eg, to print them in the repl) depends on them being ISeq, so I can't just ignore it. To implement all of these methods (around thirty) would take a large amount of code, which can't easily be shared between Iteration, Range, and any future reducible sources that are added to core.clj.

I could write a macro like (defseq Range [start end step] Counted (count [this] ...) ...) which takes normal deftype args and also adds in implementations for ISeq, Collection, and so forth in terms of (.seq this), which will be a LazySeq. However, this seems like a somewhat awkward approach that I would be a little embarrassed to clutter up core.clj with. If anyone has a better alternative I will be pleased to hear it. In the mean time, I will go ahead with this macro implementation, in case it turns out to be the best choice.

– This patch subsumes all previous patches to this issue and to CLJ-992 –

In order to create an object which is both a lazy sequence and a
reducible source, I needed to add a macro named defseq to core_deftype.
It is basically a reimplementation of clojure.lang.LazySeq as a clojure
macro, so that I can "mix in" lazy-sequence functions into a new class
with whatever methods are needed for reducing and folding.

If we wanted, we could use this macro to implement lazy-seq in clojure instead of in java, but that's unrelated so I didn't do that in this patch.

As noted in a previous comment, defseq may not be the right approach, but this works until something better is suggested.

I accidentally included an implementation of drop-while in this patch, which I was playing around with to make sure I understood how this all works. I guess I'll leave it in for the moment, since it works and is useful, but I can remove it, or move it to a new JIRA ticket, if it's not wanted at this time.

Ok, I think this patch is officially off the rails. There must be a better way. Let's start with: touching core/deftype and reimplementing lazy-seq as a macro are off the table. The return value of range doesn't have to be a LazySeq, it has to be a lazy seq, .e.g. implement ISeq (7 methods, not 30) which it can do by farming out to its existing impl. It can also implement some new interface for use by the reducer logic. There is also still clojure.lang.Range still there, which is another approach. Please take an extremely conservative approach in these things.

Okay, thanks for the feedback - I'm glad I went into that last patch knowing it was probably wrong . I thought I would need to implement the java collection interfaces that LazySeq does, eg java.util.List, in order to avoid breaking interop functions like (defn range-list [n] (ArrayList. (range n))). If it's sufficient to implement ISeq (and thus IPersistentCollection), then that's pretty manageable.

It's still an unpleasant chunk of boilerplate for each new source, though; would you welcome a macro like defseq if I didn't put it in core_deftype? If so, it seems like it might as well implement the interop interfaces; if not, I can skip them and implement the 7 (isn't it more like 9?) methods in ISeq, IPersistentCollection, and Seqable for each new source type.

Thanks for pointing out clojure.lang.Range to me - I didn't realize we had it there. Of course with implementation inheritance it would be easy to make Range, Iteration, etc inherit from LazySeq and just extend protocols from them. But that means moving functionality out of clojure and into java, which I didn't think we'd want to do.

I'll put together a patch that just implements ISeq by hand for both of these new types, and attach it probably later today.

So I've written a patch that implements ISeq, but not the java Collections interfaces, and it mostly works but there are definitely assumptions in some parts of clojure.core and clojure.lang that assume seqs are Collections. The most obvious to me (ie, it shows up when running mvn test) is RT/toArray - it tests for Collection, but never for ISeq, implying that it's not willing to handle an ISeq that is not also a collection. Functions which rely on toArray (eg to-array and vec) now fail.

This patch subsumes all previous patches on this issue, but is not suitable for application because it leaves some failing tests behind - it is intended only for intermediate feedback.

It would be a great help if, time permitting, you could please write up the issues, challenges and options you've discovered somewhere on the dev wiki (even a simple table would be fantastic). I realize this has been a challenging task, and at this point perhaps we should opt for the more modest reducers/range and reducers/iterate and leave the two worlds separate. I'd like at some point to unify range, as there are many extant ranges it would be nice to be able to fold, as we can extant vectors.

Should r/range return something Seqable and Counted?

If so, I'll do the same for r/repeat.

I've sketched out a description of the issues and options. I'm not very familiar with the dev wiki and couldn't figure out where was the right place to put this. "release.next" seems to still be about 1.4 issues, and I don't know if it's "appropriate" to create a whole new category for this. It's available as a gist until a better home can be found for it.

Here's a single patch summing up the state Rich suggested "rolling back" to: separate r/range and r/iterate functions. I haven't heard any feedback since doing the writeup Rich asked for, so am not making any further progress at the moment; if something other than this patch is desired just let me know.

I prefer not to see the use of extend like this for new types. Perhaps this code is too DRY? Also, it does a lot in one patch which makes it hard to parse and accept. This adds Range, switches impl of vector folds etc. Can it be broken up into separate tickets that do each step that builds on the previous, e.g. one ticket could be: capture vector fold impl for reuse by similar things.

Okay, I should be able to split it up over the weekend. I'll also see about converting fold-by-halves into a function that is used by Range/Vector, rather than a function that gets extended onto them.

As requested, I have split up the large patch on this issue into four smaller tickets. The other three are: CLJ-1045, CLJ-1046, and CLJ-992.

CLJ-1045 contains the implementation of fold-by-halves, and as such this patch cannot be applied until CLJ-1045 is accepted. This ticket does not depend on the other two, but there will be minor merge conflicts if this is merged before them.

Clojure-dev discussion started here: http://groups.google.com/group/clojure-dev/browse_thread/thread/1f106a21ec1ce3de

Patch clj-957-allow-typehinting-of-method-signatures-in-deftype-patch2.txt dated Aug 19 2012 is identical to Chris Gray's patch 0001-Allow-for-typehinting-of-method-signatures-in-deftyp.patch dated Mar 22, 2012, except it has some updated context lines so that it applies cleanly to latest master.

the same is also true for TransientVectors

{{(contains? (transient [1 2 3]) 0)}}

false

As expected, TransientSets have the same issue; plus an additional, probably related one.

(:x (transient #{:x}))

nil

(get (transient #{:x}) :x)

nil

This is caused by expectations in clojure.lang.RT regarding the type of collections for some methods, e.g. contains() and getFrom(). Checking for contains looks to see if the instance passed in is Associative (a subinterface of PersistentCollection), or IPersistentSet.

This patch refactors several of the Clojure interfaces so that logic abstract from the issue of immutability is pulled out to a general interface (e.g. ISet, IAssociative), but preserves the contract specified (e.g. Associatives only return Associatives when calling assoc()).

With more general interfaces in place the contains() and getFrom() methods were then altered to conditionally use the general interfaces which are agnostic of persistence vs. transience. Includes tests in transients.clj to verify the changes fix this problem.

Rich: Patch doesn't currently apply, but I would like to get your take on approach here. In particular:

1. this represents working back from the defect to rethinking abstractions (good!). Does it go far enough?
2. what are good names for the interfaces introduced here?

Rebased the patch off the latest pull of master as of 3/25/2011, it should apply cleanly now.

Latest patch does not apply as of f5bcf647

clj-700-patch2.txt does patch cleanly to latest Clojure head as of a few mins ago. No changes to patch except in context around changed lines.

Sigh. Git patches applied via 'git am' are fragile beasts indeed. Look at them the wrong way and they fail to apply.

clj-700-patch3.txt applies cleanly to latest master as of Mar 7, 2012, but not if you use this command:

git am -s < clj-700-patch3.txt

I am pretty sure this is because of DOS CR/LF line endings in the file src/jvm/clojure/lang/Associative.java. The patch does apply cleanly if you use this command:

git am --keep-cr -s < clj-700-patch3.txt

This ticket was changed to Incomplete and waiting on Rich when Stuart Halloway asked for feedback on the approach on 28/Jan/2011. Stuart Sierra changed it to not waiting on Rich on 17/Feb/2012 when he noted the patch didn't apply cleanly. Latest patch clj-700-patch3.txt does apply cleanly, but doesn't change the approach used since the time Stuart Halloway's concern was raised. Should it be marked as waiting on Rich again? Something else?

Patch 4 incorporates patch 3, and brings it up to date on hashing (i.e. uses hasheq).

Removed clj-700-patch3.txt in favor of Stuart Halloway's improved clj-700-patch4.txt dated June 8, 2012.

clj-700-patch5.txt dated June 18, 2012 is the same as Stuart Halloway's clj-700-patch4.txt, except for context lines that have changed in Clojure master since Stuart's patch was created. clj-700-patch4.txt no longer applies cleanly.

Adding clj-700-patch6.txt, which is identical to Stuart Halloway's clj-700-patch4.txt, except that it applies cleanly to latest master as of Aug 19, 2012. Note that as described above, you must use the --keep-cr option to 'git am' when applying this patch for it to succeed. Removing clj-700-patch5.txt, since it no longer applies cleanly.

Patch fails as of commit 1c8eb16a14ce5daefef1df68d2f6b1f143003140

Which patch did you try, and what command did you use? I tried applying clj-700-patch6.txt to the same commit, using the following command, and it applied, albeit with the warning messages shown:

% git am --keep-cr -s < clj-700-patch6.txt
Applying: Refactor of some of the clojure .java code to fix CLJ-700.
/Users/jafinger/clj/latest-clj/clojure/.git/rebase-apply/patch:29: trailing whitespace.
public interface Associative extends IPersistentCollection, IAssociative{
warning: 1 line adds whitespace errors.
Applying: more CLJ-700: refresh to use hasheq

Note the --keep-cr option, which is necessary for this patch to succeed. It is recommended in the "Screening Tickets" section of the JIRA workflow wiki page here: http://dev.clojure.org/display/design/JIRA+workflow

Presumptuously changing Approval from Incomplete back to None, since the latest patch does apply cleanly if the --keep-cr option is used. It was in Screened state recently, but I'm not so presumptuous as to change it to Screened

Converted from http://www.assembla.com/spaces/clojure/tickets/373

I've attached my code from the g.group thread in the form of a patch, in case it's sufficient. (Thanks to abedra for the gentle kick in the pants.)

Looks to me like the patch is misusing if-let, e.g.

(when-let [[k & mk] []] "Doh!") 
=> Doh!

Please correct, and add tests for nil and [] keys (at least).

I will write some tests and correct this.

I'm sorry to report that my good intentions of wanting to help clear some of the project backlog has created more work by way of further questions. I'd also like to clarify the desired new behaviours for the test cases.

Heinz proposed that an empty key sequence will not create a new nil key in the returned map.
He also suggested that the following behaviour changes be made (compare old and new* behaviours):

 
(update-in {1 2} [] (constantly {2 3}))
{nil {2 3}, 1 2}

(update-in* {1 2} [] (constantly {2 3}))
{2 3}

(update-in {1 2} [] assoc  1 3)
{nil {1 3}, 1 2}

(update-in* {1 2} [] assoc  1 3)
{1 3}

Chouser also added that nil keys should be honoured, as before:

 
(update-in* {nil 2} [nil] (constantly 3))
{nil 3}

I've added a variety of tests to cover the existing behaviour and would like to confirm that the above is all that's required for new behaviour.

The patch from November 2010 didn't work, but I tweaked it with a when-let as Stuart suggested and placed a check for an empty sequence of keys before the when-let block; because essentially, the primary behaviour change boils down to simply handling an empty sequence of keys, in addition to the existing behaviours.

I'm not entirely convinced that these changes are a good thing, but at least there's now something concrete for discussion. Please have a look at what is there. The good news is that at least there are some tests covering update-in now.

clj-373-alter-behavior-of-update-in-with-empty-key-patch2.txt dated May 24, 2012 supersedes patch 0001-CLJ-373Alter-behaviour-of-update-in-with-empty-key.patch dated May 9, 2012. It makes no changes to the contents of the patch except in the lines of context that have changed since the earlier patch was created. It applies cleanly to the latest master whereas the earlier patch no longer does. Builds and passes tests with Oracle/Apple JDK 1.6 on Mac OS X 10.6.8.

This patch creates a new error mode highlighted by the following:

(update-in {:a 1} [] inc)

; ClassCastException clojure.lang.PersistentArrayMap cannot be cast to java.lang.Number

The reason is that the code that executes in the event that the keys are empty (or nil) blindly assumes that the function given can be applied to the map itself. This is no problem in the case of assoc and the like, but clearly not for inc and many other useful functions.

The old behavior was to throw an NPE and if any code relies on that fact is now broken. Maybe this is not a problem, but I'm kicking it back to get a little more discussion and to request that whatever the ultimate fix happens to be, its behavioral changes and error modes are noted in the docstring for update-in.

I vote for changing `update-in` to support empty key vectors.

Because I think "(apply f m args)" in case of an empty vector is the natural continuation of the usual behavior of update-in:

• update-in with 2 keys the second level map is updated
• update-in with 1 key the first level map (child of given map) is updated
• update-in with no key the given map (zero level) is updated.

Otherwise, you will always have to wrap update-in in something like the following when the keys vector is computed:

(if (seq keys) (apply update-in m keys f args) (apply f m args))

To Fogus' last post: (update-in {:a {:b 1}} [:a] inc) fails similar and is not handled with a special exception.

The problem is the same as the one for http://dev.clojure.org/jira/browse/CLJ-918

The patch linked there was mine but i had no CA signed at that time, and no account on jira.

Here is the same patch in the correct format.

Bug #918 should be closed as this is a duplicate

Ubuntu 12.04, Java 7, Clojure 1.4

clj-1020-inspect-table-skip-nil-rows-patch1.txt of July 12, 2012 is identical to inspector.patch of July 2, 2012, except it is in the desired git format. Proper attribution is given to author Dimitrios Piliouras in the patch.

Jason, have you tried to build this using JDK 1.6.0? I've tried on Mac OS X 10.6.8 + Oracle/Apple JDK 1.6.0 and Ubuntu 11.10 + IBM JDK 1.6.0, and on both it compiles, but during the tests fails with a ClassNotFoundException for class jsr166y.ForkJoinTask.

It builds and tests cleanly on Ubuntu 11.10 + Oracle JDK 1.7.0 for me.

That's an issue that applies to all of core.reducers. Alan Malloy experienced it as well. I tried fixing it, but eventually just upgraded to JDK 1.7. I don't understand why it's happening.

This issue is isolated to mvn test afaik.

When I include clojure inside a leiningen project, and add jsr166y.jar to lib directory, core.reducers works fine with java 1.6.

Jason, you say it applies to all of core.reducers in your May 17, 2012 comment. I don't understand. Without your patch applied, I can run "./antsetup.sh ; ant" in a freshly-pulled Clojure git repo on either of the JDK 1.6.0 versions mentioned in my earlier comment, and do not get any errors during the tests. Are you saying perhaps that core.reducers currently has no tests that exercise the problem now, but your patch adds such tests that fail, even with no other changes to the code?

Yah that's right. Now that you mention it, my patch is the first unit test to call r/fold (the existing tests do non-parallel reductions).

With Stuart Halloway's commit to Clojure master on June 8, 2012 titled "let reducers tests work under ant", patch 0001-repeat-for-clojure.core.reducers.patch dated May 11, 2012 now runs correctly even the new unit tests requiring class jsr166y.ForkJoinTask with Oracle/Apple JDK 1.6 and Linux IBM JDK 1.6.

I'm on the contributors list. Is this patch still needed?
sorry for long long delay.

This patch should wait until http://dev.clojure.org/jira/browse/CLJ-993 is committed. I think there's a some shared code.

Patch clj-1063-add-dissoc-in-patch-v2.txt dated Sep 13 2012 supersedes 001-dissoc-in.diff dated Sep 7 2012. It fixes a typo (missing final " in doc string), and adds a test case for the new function.

Thanks for the fix Andy

This proposed dissoc-in should be compared with the one in clojure.core.incubator which I just happened across. I see they look different, but haven't examined to see if there are any behavior differences.

https://github.com/clojure/core.incubator/blob/master/src/main/clojure/clojure/core/incubator.clj

dissoc-in in clojure.core.incubator recursively removes empty maps

user=> (clojure.core.incubator/dissoc-in {:a {:b {:c 1}}} [:a :b :c])
{}

while the one in this patch doesn't (as I would expect)

user=> (dissoc-in {:a {:b {:c 1}}} [:a :b :c])
{:a {:b {}}}

Please do this work in the incubator.

Hmm.. it's not quite as simple as I thought. My first patch simply had reducer/folder implement IKVReduce with `(clojure.core.protocols/kv-reduce coll (xf f1) init)`, but for map and mapcat, this results in strange behavior (reduce-kv reducefs being called with only 2 args).

Patch 0001 includes modifications to map and mapcat so that the reduce-kv reducef will always be called with 3 args. The previous implementation of mapcat also had the converse problem: during non-kv reduce, if the mapcat fn returned a map, the reducef would be called with 3 args since maps reduce with reduce-kv.

The patch gives behavior like:

user> (->> {1 {2 3 4 5} 6 {7 8 9 10}}
        (r/mapcat #(r/map + %2))
        (reduce-kv #(conj %1 [%2 %3]) []))
[[2 5] [4 9] [7 15] [9 19]]
user> (->> [{2 3 4 5} {7 8 9 10}]
        (r/mapcat identity)
        (r/reduce conj []))
[[2 3] [4 5] [7 8] [9 10]]

Would like to discuss these changes (and auto-kv for maps) on clojure-dev, but my membership is still pending. My CA has been received. Anyone know who to contact to get my membership approved?

I have attached a patch that changes clojure.lang.Var and clojure.core/thread-bound? to only return true if a Var is set!-able.

Removing sync makes clojure build much faster. I wonder why it was added in the first place? I guess only Rich knows? I assume that it is not necessary.

If we are removing sync though, I wouldn't bother with the atomic rename stuff. Doing that sort of thing can cause problems on some platforms, eg with search indexers and virus checkers momentarily locking files after they are created.

The patch seems to be assuming that sync is there for some reason, but my initial assumption would be that sync isn't necessary - perhaps it was working around some issue that no longer exists?

Although its unlikely: there is a possible race condition "loading a paritally written classfile"?:

https://github.com/clojure/clojure/blob/master/src/jvm/clojure/lang/RT.java#L393

The new improve-writeclassfile-perf version of the patch combines the two previous patches into a single patch file and brings them up-to-date with master. I can split the two changes back out into separate patch files if desired, but I figured out current tooling is more geared towards a single patch being applied.

FWIW, both fixes look sane. The first one is a nice cleanup. The second one is a little more interesting in that uses a rename operation to put the final file into place. It removes the sync call, which does make things faster. In general, if we're concerned about on-disk consistency, we should really have a combination of the two: write the full contents to a tmp file, sync it, and atomically rename to the destination file name.

Neither the current master, nor the current patch will guarantee on-disk consistency across a machine wide crash. The current master could crash before the sync() occurs, leaving the file in an inconsistent state. With the patch, the OS may not get the file from file cache to disk before an OS level crash occurs, and leave the file in an inconsistent state as well. The benefit of the patch version is that the whole file does atomically come into view at once. It does have a nasty side effect of leaving around a temp file if the compiler crashes just before the rename though.

Perhaps a little more work to catch an exception and clean up is in order? In general, I like the patched version better.

File.renameTo returns false on (most?) errors, but the patch doesn't check for failure. Docs say "The return value should always be checked to make sure that the rename operation was successful." Failure might be especially likely on Windows, where files are opened by others without FILE_SHARE_DELETE.

Should I have made this implement Seqable as well? It wasn't clear to me, because as far as I could see this was the only function in clojure.core.reducers that's generating a brand-new sequence rather than transforming an existing one.

Previous version neglected to include the seed value of the iteration in the reduce.

Currying iterate seems useless, albeit not harmful.

While implementing repeat, I couldn't use currying. Because 1-arity is already reserved for infinite repeat ([n x] and [x], not [n x] and [n] if currying)

How about we just support currying for functions where last param is reducible?

This new patch replaces the previous patch. As requested, I am splitting up the large issue CLJ-993 into smaller tickets.

Does not depend on any of my other reducer patches, but there will probably be some minor merge conflicts unless it is merged after CLJ-1045 and CLJ-1046, and before CLJ-993.

clj-1087-diff-perf-enhance-patch-v1.txt dated Oct 15 2012 implements Tom's suggested performance enhancement, although not exactly in the way he suggested. It does calculate the union of the two key sequences.

Next person to take a deep look at pmap should probably also think through fork/join.

fork/join is a Java 7 feature. Would a proposed patch need to be able to fall back to Java 5 features?

Clojure/core folks can say more authoritatively, but I believe with the recent reduce enhancements that rely on jsr166 code, Clojure 1.5 will most likely require Java 6 or later, and Java 5 will no longer be supported.

Spinning up more threads than CPU cores is not a good idea when the work is CPU bound. Currently (1.4) pmap uses an unbounded thread pool, so chunked sequences will create more threads than intended. The least invasive change is to use a fixed sized thread pool (ForkJoinPool being one example). pmap is differentiated from core.reducers by the fact that it is lazy. This implies a one-at-a-time ThreadPool.submit model, instead of the recursive fork/join model. Tradeoffs include:

Enforce look-ahead even on chunked sequences:

• + no threadPool changes
• - working against chunking, which is presumably being used for a reason

Move to a fixed size thread pool:

• + reduce contention for cpu-bound functions on chunked sequences
• - increase total realization time for io-bound functions

Use ForkJoinPool for fixed thread pool (instead of newFixedThreadPool):

• + automatic and dynamic parallelism
• - more complex setup (picking Java 6 vs 7 implementation, sharing pool with core.reducers)

I think using a traditional fixed size thread pool is the right option. Most of the time all of pmap's results will be realized, so I don't think it's worth saving work by being strict about the look-ahead size. This is also the decision map has made. Since we're not using ForkJoin's main strength (recursive work queuing), I don't think it is worth setting it up in clojure.core.

I'll use Agent/pooledExecutor as the fixed size thread.

Let me know if I forgot or misunderstood anything.

2012-05-28 pmap-chunking-862.diff uses a fixed size thread pool for pmap functions.

Add one-line patch to clojure.repl/source so that it will find the protocol definition for a given protocol function.

This is the thread where Max Penet suggested to have 0-arity versions of the two fns:

https://groups.google.com/forum/?fromgroups=#!topic/clojure/IRlN-4LH_U0

Can you add a test for the improved functionality?

You bet. I tried to before submitting this but found no existing tests for map-indexed to expand upon. Given that, I decided to just start the conversation first. If you think this is a good addition, I'll find a place to stick the tests and add a new patch file.

Add two unit tests for map-indexed. One tests old behavior (single collection) and the second tests mapping across 3 collections.

There were no existing tests for map-indexed that I could see to expand upon (using git grep map-indexed src/clojure)

clj-842-update-clojure.pprint-metadata-v2.txt dated Nov 14 2012 is identical to Baishampayan Ghose's 0001-Migrate-the-metadata-in-clojure.pprint.-to-the-new-s.patch dated Sep 26, 2011, except it applies cleanly to latest master.

Any

Adding initial patch. A symbol's position is now stored in its metadata, just like a seq when it is read with the LispReader.

The Compiler.LINE and Compiler.COLUMN values are now updated as each symbol is analyzed. This provides more exact line and column numbers for Compiler errors which will resolve ticket CLJ-420.

The Compiler.LINE variable has been interned as clojure.core/line-number

Adding extra metadata to symbols resulted in some tests failing due to the extra values. The failing tests have been updated to ignore the new metadata keys.

Questions posed on the clojure-dev list around how this impacts bit-shift-left: http://groups.google.com/group/clojure-dev/browse_thread/thread/2191cbf0048d8ca6

Patch on CLJ-445 fixes this as well.

This patch fails a test around bit-shifting a BigInt: `(bit-shift-left 1N 10000)`. The reason is that the patch changes the dispatch of (BigInt, Long) from (Object, Object) to (long, int).

Clearly this can't be applied (unless another change makes it possible), but I'm putting it up as a start of the conversation.

My comment from the mailing list:

If the test breaks it likely means Numbers.shiftLeft(long,int) was
selected over Numbers.shiftLeft(Object,Object). Given that 1N is an
Object (one that can exceed the size of a long), the method selection
is incorrect, thus the patch is broken.

The suggestion of "simply" modifying paramArgTypeMatch is not sufficient since the mechanism for preferring one method over another lives in Compiler, and isn't smart enough to make these sorts of decisions.

Considering moving this out of Release.next - soliciting comments from Chas.

I'm afraid I don't have any particular insight into the issues involved at this point. I ran into the problem originally noted a while back, and opened the ticket at Rich's suggestion. I'm sorry if the text of the ticket led anyone down unfruitful paths…

The issues relating to bitshift are moot since the decision was made that bit-shifts are only for 32/64 bit values.

Still a valid issue, but de-prioritized as per Rich.

Modified version of original patch

Alex, would you mind attaching it with a unique file name? I know that JIRA lets us create multiple attachments with the same file name, and I know we can tell them apart by date and the account of the person who uploaded the attachment, but giving them the same name only seems to invite confusion.

Renamed updated patch to unique name

Converted from http://www.assembla.com/spaces/clojure/tickets/107
Attachments:
bit-count.diff - https://www.assembla.com/spaces/clojure/documents/dqone2w4er3RbzeJe5afGb/download/dqone2w4er3RbzeJe5afGb

oranenj said: [file:dqone2w4er3RbzeJe5afGb]

richhickey said: Updating tickets (#8, #19, #30, #31, #126, #17, #42, #47, #50, #61, #64, #69, #71, #77, #79, #84, #87, #89, #96, #99, #103, #107, #112, #113, #114, #115, #118, #119, #121, #122, #124)

clj-107-add-bit-count-v1.txt is probably a correct updated version of the old patch linked above. Added a couple of unit tests.

Converted from http://www.assembla.com/spaces/clojure/tickets/99

richhickey said: Updating tickets (#8, #19, #30, #31, #126, #17, #42, #47, #50, #61, #64, #69, #71, #77, #79, #84, #87, #89, #96, #99, #103, #107, #112, #113, #114, #115, #118, #119, #121, #122, #124)

clj-99-min-key-max-key-performance-v1.txt dated Nov 15 2012 changes min-key and max-key to evaluate the function k on each of its other arguments at most once.

This patch implements the suggested feature.

The new system property is called clojure.core.loading-verbosely in analogy to the existing clojure.compile.warn-on-reflection.

Fix inaccurate documentation string

Apache Commons Beanutils has their own implementation of this, at http://www.docjar.com/html/api/org/apache/commons/beanutils/MethodUtils.java.html#771 – notably, it tries to reflect a method with the given signature and catches the exception on failure, rather than iterating through the whole list. This may be a better approach – I'm unfamiliar with how the cost of exception handling compares with that of reflecting on the full method list of a class.

Prior version of patch were missing new test suite files. Corrected.

Thanks for the patches, Charles. Could you please create a patch in the desired format and attach that, and then remove the obsolete patches? Instructions for creating a patch are under the heading "Development" at this page: http://dev.clojure.org/display/design/JIRA+workflow

Instructions for removing patches are under the heading "Removing patches" on that same page.

Added a patch created per documented process.

I found in my code that it's possible to get a NPE if there is no read-method, for instance on the http://docs.cascading.org/cascading/2.0/javadoc/cascading/flow/hadoop/HadoopFlow.html object which has a setCascade method but no getter. I fixed this in our code by inlining the is-zero-args check into the public-method definition and and-ing the whole thing with 'method' like the original 'bean' code, like so:

public-method (and method (zero? (alength (. method (getParameterTypes))))
(or (and (java.lang.reflect.Modifier/isPublic (. c (getModifiers)))
method)
(public-version-of-method method)))

Charles, I think we should follow Apache BeanUtils on this. Exceptions not thrown are cheap. Ordinarily, exception for control flow are bad, but this is forced by bad design of reflection API.

Probably all, but observed on Ubuntu 12.04, OpenJDK 6

Sorry, I typo'd the example. (defn total ...) should be (defn sum ...).

fixed typeo in example

Couldn't reproduce the exception, but the 2nd example did chew through about 4x the amount of memory. Vetting.

adding a patch. Since most of Clojure ends up running this code in one way or another, I'd assert that tests are included as part of the normal Clojure test process.

Patch simply calls Util.ret1(argx, argx=null) on all invoke calls.

And as a note, both examples in the original report now have extremely similar memory usages.

Sounds great, and the patch looks good too. Let me know if I need to do anything else.

clj-1102-improve-empty-stack-trace-handling-v1.txt dated Nov 4 2012 improves the handling of .getStackTrace returning a length 0 array in two places. I checked all other places .getStackTrace was called and they seem to already handle this case gracefully.

Vetting.

Converted from http://www.assembla.com/spaces/clojure/tickets/200

hlship said: Trickier than I thought because cond is really wired into other fundamentals, like let.

cgrand said: Howard, what do you think of http://gist.github.com/432712 ?

Patch cond-let-clauses.diff on 23/Nov/12 adds inline :let clauses to cond, implementing CLJ-200. The code is based off of code by cgrand, with some tweaks so the implementation only relies on constructs defined earlier in core.clj, since when cond is defined, things aren't yet fully bootstrapped. Also added a test to control.clj.

Some comments: the docstring is missing, I believe you don't have to modify the original cond (except the docstring maybe), just redefine it later on once most of the language is defined – a bit like what is done for let for example.

There is still the unlikely eventuality that some code uses :let as :else. What about shipping a cond which complains on keywords (in test position) other than :else?

cond-let-clauses-with-docstring.diff contains the same patches as cond-let-clauses, but includes the original docstring for cond along with an additional sentence about the :let bindings.

Cgrand, I did see your example of redefining cond after most of the language is defined, but since I was able to figure out how to do it in the proper place, that makes the :let bindings available for users of cond downstream and avoids any unforeseen complications that might come from rebinding.

As for your other point, I think it is highly improbable that someone would have used :let in the :else position. However I can imagine someone intentionally using something like :true or :default. I think the idea of warning for other keywords is actually more likely to cause complications than the unlikely problem it is meant to solve.

I did resubmit the patch with the docstring restored. Thanks for pointing out that problem. I'm excited about this patch – I use :let bindings within the cond in my own code all the time. Thanks again for the blog post that started me on that path.

True, it's :unlikely for :let to happen.
However once :let is officially blessed, it may be better to provision for future other "special" keywords and thus to warn on "unsupported" keywords. Plus it will help out-of-order typists (like myself) to catch earlier a :elt instead of a :let
This is only my point of view. Thanks for trying to get :let in cond supported.

Mark, could you remove the obsolete earlier patch now that you have added the one with the doc string? Instructions for removing patches are under the heading "Removing Patches" on this page: http://dev.clojure.org/display/design/JIRA+workflow

Done.

I haven't figured out what is going wrong yet. I can apply the patch cond-let-clauses-with-docstring.diff to the latest Clojure master just fine. I can do "ant jar" and it will build a jar. When I do "ant", it fails with the new test for cond with :let, throwing a StackOverflowException. I can enter that same form into the REPL and it evaluates just as the test says it should. I can comment out that new test and all of the rest pass. But the new test doesn't pass when inside of the control.clj file. Anyone know why?

It's because of the brutal replacement performed by test/are: the placeholders for this are form are x and y but in Mark's test there are used as local names and are tries to substitute them recursively...
If one changes the local names to a and b for example it works.

cond-let-clauses-fixed-test.diff on 02/Dec/12 contains the same patch, but with the x,y locals in the test case changed to a,b so that it works properly in the are clause which uses x and y.

On Windows, I can't get Clojure's test suite task to work, either via ant or maven, which has made it difficult for me to verify the part of the patch that applies to the test suite works as expected; I had tested it as best I could in the REPL, using a version of Clojure built with the patch applied, but using this process, I missed the subtle interaction between are and the locals in the test case. Sorry about that. If someone can double-check that the test suite task now works with the newest patch, that would be great, and then I'll go ahead and remove the obsoleted patch. Thanks.

clj-200-cond-let-clauses-fixed-test-v2-patch.txt dated Dec 2 2012 is identical to Mark Engelberg's cond-let-clauses-fixed-test.diff of the same date, except it applies cleanly to the latest Clojure master.

I've verified that it compiles and passes all tests with latest Clojure master as of this date.

Mark, I've made sure to keep your name in the patch, since you wrote it. You should be able to remove your two attachments now, so the screener won't be confused which patch should be examined.

Mark, besides general issues with Windows not being used much (or maybe not at all?) by Clojure developers, there is the issue right now filed as CLJ-1076 that not all tests pass when run on Windows due to CR-LF line ending differences that cause several Clojure tests to fail, regardless of whether you use ant or maven to run them.

Converted from http://www.assembla.com/spaces/clojure/tickets/457

Stu and Rich talked about making this an error, but it would break some existing code to do so.

Is there a specific question on this?

Stu, you and I went over this but I can't remember exactly what the question was here.

Tentative patch attached.
Have you an example of existing code which is broken by such a patch (as mention by Aaron Bedra)?

The patch intends to do what? We have only a problem description and code. Please enumerate the plan rather than make us decipher the patch.

As a first principle, I don't want Clojure to promise that such recursively defined values are possible.

The proposal here is to catch recursive seq realization (ie when computing the body of a lazy-seq attempts to access the same seq) and throw an exception.

Currently when such a case happens, the recursive access to the seq returns nil. This results in incorrect code seemingly working but producing incorrect results or even incorrect code producing correct results out of luck (see https://groups.google.com/d/topic/clojure/yD941fIxhyE/discussion for such an example).

So this patch moves around the modification to the LazySeq state (f, sv and s fields) before all potentially recursive method call (.sval in the while of .seq and .invoke in .sval) so that, upon reentrance, the state of the LazySeq is coherent and able to convey the fact the seq is already being computed.

Currently a recursive call may find f and sv cleared and concludes the computation is done and the result is in s despite s being unaffected yet.

Currently:

Note that "Being realized" states overlap with Unrealized or Realized.
(NB: "anything" includes null)

With the patch:

That last comment, Christophe, goes a long way to explaining the idea to me, at least. Any chance comments with similar content could be added as part of the patch?

New patch with a comment explaining the expected states.
Note: I tidied the states table up.

// Before calling user code (f.invoke() in sval and, indirectly,
// ((LazySeq)ls).sval() in seq -- and even RT.seq() in seq), ensure that 
// the LazySeq state is in one of these states:
//
// State            f          sv
// ================================
// Unrealized       not null   null
// Realized         null       null
// Being realized   null       this

CLJ-1119 is also fixed by this patch.

Seems to be a valid enhancement. I can't see any issues we'd have with it. Vetted.

Added patch & test. This patch retains the old performance characteristics of into in the case that there is only one collection argument. For example: (into [] [1 2 3]) .

Since the multi-arity version will be slightly slower, I opted to provide it as a second body instead of unifying both into a single body. If someone has a problem with this, I can rewrite the patch. At least this way, into won't get slower.

This is a good example of an idea for an enhancement I haven't approved, and thus is not yet vetted.

clj-1103-make-conj-assoc-dissoc-handle-args-similarly-v1.txt dated Nov 4 2012 makes conj conj! assoc assoc! dissoc dissoc! handle args similarly to each other.

I too ran into this and started an additional discussion here: https://groups.google.com/d/topic/clojure-dev/wL5hllfhw4M/discussion

In particular, I don't buy the argument that (into coll xs) is sufficient, since into implies conj and there isn't an terse and idiomatic way to write (into map (parition 2 keyvals))

So +1 from me

Whoops, according to http://dev.clojure.org/display/design/JIRA+workflow I should have emailed clojure-dev before filing this ticket. Here is the discussion:

https://groups.google.com/d/topic/clojure-dev/ME3-Ke-RbNk/discussion

Attached 001-make-PersistentQueue-implement-List.diff, 15/Sep/12

Note that this patch has a minor conflict with the one I added to CLJ-1070, because both add an extra interface to PersistentQueue - List in this case, IHashEq in CLJ-1070.

Philip, patch looks pretty good – thanks for doing this. A couple notes:

This is only my opinion, but I prefer imports be listed without wildcards, even if it means an extra couple lines at the top of a .java file.

I noticed the "List stuff" code is a copy of what's in ASeq and EmptyList. I suppose this is copied because EmptyList and PersistentQueue extend Obj and therefore can't extend ASeq. Is this the only reason? It seems a shame to duplicate these method definitions, but I don't know of a better solution, do you?

It would also be nice if the test check a couple of the List methods you've implemented.

oh, also "git am" refused to apply the patch, but I'm not sure why. "patch -p 1" worked perfectly.

did you use the --keep-cr option to git am?

I struggled to know whether I should be adding CRs or not to line endings, because the files I was editing weren't consistent in their usage. If you open them in emacs, half the lines have ^M at the end.

Will submit another patch, with the import changed. I'll have a think about the list implementation and see what ideas I can come up with.

Attached 002-make-PersistentQueue-implement-Asequential.diff

This patch is an alternative to 001-make-PersistentQueue-implement-List.diff

So I took on board what you said about ASeq, but it didn't feel right making PersistentQueue directly implement ISeq, somehow.

So I split ASeq into two parts – ASequential, which implements j.u.{Collection,List} and manages List-equality and hashcodes; and ASeq, which... doesn't seem to be doing much anymore, to be honest.

As a bonus, this patch fixes CLJ-1070 too, so I went and added the tests from that ticket in to demonstrate this fact. It also tidies up PersistentQueue by removing all equals/hashcCode stuff and all Collection stuff.

(It turns out that because ASeq was already implementing Obj, the fact that PersistentQueue was implementing Obj was no barrier to using it.)

Would appreciate comments on this approach, and how it differs from the previous patch here and the patch on CLJ-1070.

Looking at EmptyList's implementation of List, it is a duplicate of the others, but it shouldn't be. I think its implementation of indexOf is the biggest culprit - it should just be 'return -1;' but it has a great big for loop! But this is beyond the scope of this ticket, so I won't patch that here.

Philip, now that the patch for CLJ-1070 has been applied, these patches no longer apply cleanly. Would you be willing to update them? If so, please remove the obsolete patches.

Andy, thanks so much for your efforts to make people aware of these things. I will indeed submit new patches, hopefully later this week.

Replaced existing patches with new ones which apply cleanly to master.

There are two patches:

001-clj-1059-make-persistentqueue-implement-list.diff

This fixes equality by making PersistentQueue implement List directly. I also took the opportunity to remove the wildcard import and to add tests for the List methods, as compared with the previous version of the patch.

002-clj-1059-asequential.diff

This fixes equality by creating a new abstract class ASequential, and making PersistentQueue extend this.

My preferred solution is still the ASequential patch, but I'm leaving both here for comparison.

Vetting.

Philip, this time I think it was patches that were committed for CLJ-1000 that make your patch 002-clj-1059-asequential.diff not apply cleanly. I often fix up stale patches where the change is straightforward and mechanical, but in this case you are moving some methods that CLJ-1000's patch changed the implementation of, so it would be best if someone figured out a way to update this patch in a way that doesn't clobber the CLJ-1000 changes.

Thanks Andy. Submitted a new patch, 002-clj-1059-asequential-rebased-to-cached-hasheq.diff, which supersedes 002-clj-1059-asequential.diff.

The patch 001-clj-1059-make-persistentqueue-implement-list.diff still applies cleanly, and is still an alternative to 002-clj-1059-asequential-rebased-to-cached-hasheq.diff.

Tests pass.

Edward, your patch replaces the expression (or m1 {}) with m1. It was changed from m1 to (or m1 {}) in a commit on Oct 16, 2008 with descriptive text "improved nil handling in merge, merge-with", so I am pretty sure it would be best to leave it as (or m1 {}). I believe the intent is to allow all but one of the map arguments to merge-with be nil, and everything will still work.

The rest of the patch for avoiding one merge call seems sound to me.

Your change would be even better at preserving any metadata on the first non-nil map in the list, if instead of calling with the first map, it called it with the first non-nil item of the list, and then the rest of the list after that.

I figured out that `reduce1` did pass a head of the list for me. (https://github.com/clojure/clojure/blob/master/src/clj/clojure/core.clj#L887)
But case with first nil argument is still valid. Correct me, please, if i'm wrong.

I'm not sure about `(or m1 {})`. I don't see any problems which can happen. Probably behaviour of functions which are used internally was changed since 2008.

(contains? nil :a) ;=> false
(assoc nil :a 1) ;=> {:a 1}
(get nil :a) ;=> nil

I could write some tests for that.

Seems valid. Vetting.

Corrected patch

Patch 0001-report-load-exception-with-file-and-line.diff fails build. Patch 0002-report-load-exception-with-file-and-line.diff applies, builds, and tests cleanly as of March 9, 2012. Hugo has signed a CA.

clj-939-report-load-exceptions-with-file-and-line-patch-v2.txt dated Oct 5 2012 is intended to be an update to Hugo Duncan's patch 0002-report-load-exceptions-with-file-and-line.diff dated Feb 25 2012. Because of Brandon Bloom's recently commited patch adding column numbers in addition to line numbers, this is not simply updating some lines of context, but I think it is correct. It would be good if Hugo could take a look at it and confirm.

Screened.

The error messages are better than what we had before. The line/column numbers are not particularly informative, probably because ns is a macro.

This patch doesn't change the reporting on any other (e.g. nested) exceptions? It looks like it might.

1.2, 1.3 beta3

Modified doctoring to clarify the usage of :hierarchy.

New patch: '0001-CLJ-835-Refine-doc-string-for-defmulti-hierarchy-opt.patch' 10/May/12.

I've attached a new doc string patch in response to Andy Fingerhut's request posted here: https://groups.google.com/forum/?fromgroups#!topic/clojure-dev/i7H82fJYL-U

Andy's request stated "Attached patch seems to have spelling mistakes, and perhaps could be worded more clearly."

I hope my new patch is an improvement upon what was there.

This patch supersedes '0001-Clarify-docstring-for-defmulti.patch' from 14/Sep/11.

This is a million times better than what was there before, but (who knew!?) it could be better. A couple points of contention:

• The order under the :hierarchy section seems off. I would start with an overview of what a hierarchy is including a definition of the global hierarchy map (it's mentioned, but glossed over). I'd then move on to an example impl. Finally, I'd then add the discussion starting "Multimethods expect..." immediately followed by an example.

• In the isa? example it would be great to show the return value.

• You mention that the hierarchy should be supplied as a reference, which is correct, however you talk about #'. The link between e.g. var and #' could be a bit more explicit. Maybe the sentence about #' could be parenthetical?

• It's ok to finish with a very small example of its use in defmethod, in fact a simple example specifying methods for ::shape and ::circle would clarify the intent nicely.

Please leave examples out of docs strings. Use precise language.

v2 of the patch is simply converts the patch to git format. I made sure that Mr. Duffy got proper attribution. I also took a stab at a simple log message. Any problems with the latter are mine, and I'm happy to fix it up if necessary.

This doesn't update the entire doc string. The expansion is updated, but the signature section isn't. Here's what shows up.

user=> (doc extend-type)
-------------------------
clojure.core/extend-type
([t & specs])
Macro
  A macro that expands into an extend call. Useful when you are
  supplying the definitions explicitly inline, extend-type
  automatically creates the maps required by extend.  Propagates the
  class as a type hint on the first argument of all fns.

  (extend-type MyType 
    Countable
      (cnt [c] ...)
    Foo
      (bar [x y] ...)
      (baz ([x] ...) ([x y & zs] ...)))

  expands into:

  (extend MyType
   Countable
     {:cnt (fn [c] ...)}
   Foo
     {:baz (fn ([x] ...) ([x y ...] ...))
      :bar (fn [x y] ...)})

This is very much related to http://dev.clojure.org/jira/browse/CLJ-1024.

NOTE: While varargs are not supported in protocol declarations, dynamic extension of a protocol via extend (extend-type, extend-protocol) does allow for varargs and also destructuring, cause the method impls are actually normal clojure functions.

So if a Foo protocol declares a (foo [this a b c]) method, you can extend/extend-type/extend-protocol it dynamically using (foo [this & more] (do-magick)) where `a b c` are conflated into the `more` parameter.

However, that doesn't work for method implementations defined by deftype, defrecord, and reify.

So with respect to the facts, the example in the docstring is actually correct, so I'm not sure if it should be changed. However, what's supported in which cases should be documented better as it is right now.

The attached patch implements ex-message and ex-cause to work on arbitrary Throwables.