<< Back to previous view

[CLJ-2253] slurp doesn't properly handle URLs that contain authentication information Created: 19/Oct/17  Updated: 22/Oct/17

Status: Open
Project: Clojure
Component/s: None
Affects Version/s: Release 1.8
Fix Version/s: None

Type: Enhancement Priority: Minor
Reporter: Peter Monks Assignee: Unassigned
Resolution: Unresolved Votes: 0
Labels: io
Environment:

n/a


Attachments: Text File 0001-CLJ-2253-Add-support-for-slurping-urls-with-authenti.patch    
Patch: Code
Approval: Triaged

 Description   

Because of limitations in the underlying Java classes [1], clojure.core/slurp doesn't support URLs that contain authentication information i.e. https://user:password@hostname.domain/path/to/resource. It would be great if slurp did support this (e.g. by using Java's UrlConnection class, as suggested in [1]), so that application code doesn't have to include special cases for different types of input string that will be sent to slurp.

[1] https://stackoverflow.com/questions/496651/connecting-to-remote-url-which-requires-authentication-using-java

Approach: Check to see if url contains userInfo, if so, open a connection, set authorisation on it and return its input stream. Otherwise call `openStream` on the url as before
Limitations: This approach uses java.xml.bind.DatatypeConverter, which is known to not work with java9



 Comments   
Comment by Alex Miller [ 19/Oct/17 2:38 PM ]

Re the note, this seems like a non-starter then?

Comment by Peter Monks [ 19/Oct/17 2:55 PM ]

What's the minimum version of the JVM that Clojure supports? Java 1.8 added java.util.Base64, and that may be a better choice for the encoding step (assuming the minimum supported JVM version is 1.8).

Alternatively, it seems like some kind of version-dependent "conditional compilation" would meet all cases - use java.xml.bind.DatatypeConverter for JVM <= 1.7, and use java.util.Base64 for JVM >= 1.8. Apologies for my unfamiliarity with the core Clojure codebase, but is there JVM-version-specific logic elsewhere that I could look at to see how this might be done?

Comment by Erik Assum [ 19/Oct/17 3:26 PM ]

I'd really appreciate some pointers to how to solve this, since Base64 encoding is needed. Could an option be to copy
https://github.com/clojure/data.codec/blob/master/src/main/clojure/clojure/data/codec/base64.clj
to a ns in clojure.core?

There is also http://www.docjar.com/html/api/java/util/prefs/Base64.java.html, which of course is package scoped.
I guess we could use reflection to get to this, but feels hacky.

Or should we just wait until java 8 is the lowest supported java platform?

Comment by Peter Monks [ 19/Oct/17 7:48 PM ]

Posting this may be a career limiting move, but this seems to work (tested on JDK 9 as-is, and on JDK 1.8 as-is as well as with the first clause in the or commented out to ensure both versions of the method got tested):

(let [jvm-version (System/getProperty "java.version")]
  (if (or (clojure.string/starts-with? jvm-version "1.8")
          (clojure.string/starts-with? jvm-version "9"))
    (eval
      (read-string
      "(defn- base64-encode
         [s]
         (.encodeToString (java.util.Base64/getEncoder) (.getBytes s)))"))
    (eval
      (read-string
      "(defn- base64-encode
         [s]
         (javax.xml.bind.DatatypeConverter/printBase64Binary (.getBytes s)))"))))

You'd then unconditionally call base64-encode from the existing patch, Erik.

I'm simultaneously horrified and rather chuffed at this ghastly hack...

Comment by Alex Miller [ 19/Oct/17 10:44 PM ]

Doing nothing is always an option.

Comment by Peter Monks [ 19/Oct/17 10:54 PM ]

Where's the fun in that?

Comment by Alex Miller [ 20/Oct/17 8:55 AM ]

As the one maintaining it down the line, the fun is in avoiding future pain.

Comment by Peter Monks [ 20/Oct/17 10:15 AM ]

How about this approach (also tested as before):

(defmacro base64-encode
  [^String s]
  (let [jvm-version (System/getProperty "java.version")]
    (if (or (clojure.string/starts-with? jvm-version "1.6")
            (clojure.string/starts-with? jvm-version "1.7"))
      `(javax.xml.bind.DatatypeConverter/printBase64Binary (.getBytes ~s))
      `(.encodeToString (java.util.Base64/getEncoder) (.getBytes ~s)))))

?

I also inverted the condition, to make the code more future proof (assuming the JVM retains java.util.Base64 going forward, of course).

Note: there's a separate (unanswered) question regarding whether this macro should be ^:private or not.

Comment by Phill Wolf [ 21/Oct/17 9:02 PM ]

What matters is the Java version that runs the program. The macro is pleasantly legible, but it tests java.version at compile-time.

Comment by Peter Monks [ 22/Oct/17 12:04 AM ]

Is this ns AOT compiled? If not, am I misunderstanding when Clojure compilation occurs?





[CLJ-2252] Add javadoc URL for java 9 to clojure.java.javadoc Created: 17/Oct/17  Updated: 19/Oct/17

Status: Open
Project: Clojure
Component/s: None
Affects Version/s: Release 1.9
Fix Version/s: None

Type: Enhancement Priority: Minor
Reporter: Juraj Martinka Assignee: Unassigned
Resolution: Unresolved Votes: 0
Labels: None

Attachments: Text File clj-2252-2.patch     Text File javadoc-java9.patch    
Patch: Code
Approval: Prescreened

 Description   

Javadoc url for java 9 is missing in clojure.java.javadoc.
As a result, javadoc function is not able to find documentation for java 9 classes:

(javadoc java.lang.StackWalker)
;;=> opens web browser pointing to nonexistent URL: http://docs.oracle.com/javase/8/docs/api/java/lang/StackWalker.html

Patch: clj-2252-2.patch
Prescreened by: Alex Miller



 Comments   
Comment by Alex Miller [ 18/Oct/17 3:04 PM ]

Should probably change the default url too if jdk is unknown. Generally we use the latest stable version of Java for that.

Comment by Juraj Martinka [ 19/Oct/17 2:42 AM ]

Makes sense. I uploaded a new patch.

Comment by Alex Miller [ 19/Oct/17 8:53 AM ]

I added a modified patch that just changes the commit message to include the ticket (attribution retained). Marked prescreened. I don't think we'll get this in for Clojure 1.9, but should be no problem next time.





[CLJ-2249] Document behavior of clojure.core/get on strings and Java arrays Created: 05/Oct/17  Updated: 09/Oct/17

Status: Open
Project: Clojure
Component/s: None
Affects Version/s: None
Fix Version/s: None

Type: Enhancement Priority: Minor
Reporter: Arne Brasseur Assignee: Unassigned
Resolution: Unresolved Votes: 0
Labels: docstring

Attachments: Text File 0001-Expand-docstring-of-clojure.core-get.patch     Text File CLJ-2249-patch-2.patch    
Patch: Code
Approval: Prescreened

 Description   

get's implementation checks in order for

  • ILookup
  • nil
  • Map
  • IPersistentSet
  • String or Java array

The docstring for get currently reads

"Returns the value mapped to key, not-found or nil if key not present."

That this works on maps and associative data can reasonably be inferred if one knows Clojure's data model. That it works on sets, Strings, and arrays is less obvious, and would be helpful to mention.

Patch: CLJ-2249-patch-2.patch
Prescreened by: Alex Miller



 Comments   
Comment by Arne Brasseur [ 05/Oct/17 12:49 PM ]

I've tried to stick to the same terse style, happy to take suggestions for phrasing. I also changed the first sentence slightly, because I found it difficult to parse, but that is an unrelated change that I can undo if that's preferred.

Comment by Alex Miller [ 05/Oct/17 1:14 PM ]

I'd leave the first sentence as is.

All of these collection ops are kind of tricky in being able to succinctly state the intent, while also covering special cases (which are often related to Java types). Here I think the intent is to cover lookup in "associative data structures" which covers Clojure maps, records, vectors, Java maps, and other less obvious things like weird ILookup impls.

The non-obvious inclusions for me are: Clojure sets (I haven't reviewed but undoubtedly this is implicitly used in a bunch of special cases), and the Java special cases which are Strings and arrays. For an example of wording, I would point to `count` and `nth`, which are similarly weird.

So maybe a sentence like: "get also works on sets to return contained values, and on strings and arrays for value by index." ?

We'll need to answer these same questions in the spec too btw. I expect the act of spec'ing core fns to drive more of these tricky questions.

Comment by David Liepmann [ 05/Oct/17 2:18 PM ]

>I'd leave the first sentence as is.

Would a rephrase be welcome in its own issue? Right now the referent of "the value mapped to" is ambiguous. I agree it's difficult to parse.

Comment by Alex Miller [ 05/Oct/17 2:50 PM ]

David Liepmann no thanks

Comment by Arne Brasseur [ 09/Oct/17 3:27 AM ]

Appended new patch.





[CLJ-2244] double division by zero inconsistency Created: 27/Sep/17  Updated: 04/Oct/17

Status: Open
Project: Clojure
Component/s: None
Affects Version/s: Release 1.9
Fix Version/s: None

Type: Defect Priority: Minor
Reporter: Greg Chapman Assignee: Unassigned
Resolution: Unresolved Votes: 0
Labels: math

Approval: Triaged

 Description   

Division by zero with doubles will throw an exception if both arguments are boxed, but not otherwise:

Clojure 1.9.0-beta1
user=> (/ 5.0 0.0)
##Inf
user=> (/ (identity 5.0) 0.0)
##Inf
user=> (/ 5.0 (identity 0.0))
##Inf
user=> (/ (identity 5.0) (identity 0.0))
ArithmeticException Divide by zero  clojure.lang.Numbers.divide (Numbers.java:163)


 Comments   
Comment by Erik Assum [ 28/Sep/17 1:10 PM ]

So, I guess some division by zero is somewhat undefined.
If one removes

if(yops.isZero((Number)y))
	    throw new ArithmeticException("Divide by zero");

then

(/ (identity 5.0) (identity 0.0))

returns infinity as the unboxed versions.

Although doing so, also causes a change to

(/ 5 0)

which then becomes `1/0`, whereas it previously threw Divide by zero.

From Numbers.java we see that unboxed math does not protect against division by zero:

static public double divide(double x, double y){
	return x / y;
}

whereas boxed versions do:

static public Number divide(Object x, Object y){
	if (isNaN(x)){
		return (Number)x;
	} else if(isNaN(y)){
		return (Number)y;
	}
	Ops yops = ops(y);
	if(yops.isZero((Number)y))
		throw new ArithmeticException("Divide by zero");
	return ops(x).combine(yops).divide((Number)x, (Number)y);
}

This inconsistency is also reproducible with clojure 1.8:

Clojure 1.8.0
Java HotSpot(TM) 64-Bit Server VM 1.8.0_144-b01
    Docs: (doc function-name-here)
          (find-doc "part-of-name-here")
  Source: (source function-name-here)
 Javadoc: (javadoc java-object-or-class-here)
    Exit: Control+D or (exit) or (quit)
 Results: Stored in vars *1, *2, *3, an exception in *e

user=> (/ 5.0 0.0)
Infinity
user=> (/ (identity 5.0) (identity 0.0))

ArithmeticException Divide by zero  clojure.lang.Numbers.divide (Numbers.java:158)
user=>
Comment by Erik Assum [ 04/Oct/17 12:43 PM ]

Any thoughts on an approach?
Both throwing on division by zero for natives and allowing it for boxed seems to be a serious change, and the tests expects an exception for boxed division by zero.





[CLJ-2243] clojure.lang.RT should provide a loadObject static method Created: 26/Sep/17  Updated: 27/Sep/17

Status: Open
Project: Clojure
Component/s: None
Affects Version/s: Release 1.8
Fix Version/s: None

Type: Enhancement Priority: Minor
Reporter: Shlomi Assignee: Unassigned
Resolution: Unresolved Votes: 0
Labels: None


 Description   

Similar to CLJ-843, one cannot invoke (System/load "/tmp/a.so") since it successfully loads the so file to the wrong classloader (thanks SUN!)

While trying to make a simple way to load shared-objects embedded in uberjars (using https://github.com/adamheinrich/native-utils/blob/master/src/main/java/cz/adamh/utils/NativeUtils.java), I found that it is not possible to do due to the classloader issue.

Any thoughts?



 Comments   
Comment by Alex Miller [ 27/Sep/17 7:05 AM ]

That linked jira had a patch and was committed, so it seems this already exists as (clojure.lang.RT/loadLibrary "/tmp/a.so") ?

Comment by Shlomi [ 27/Sep/17 11:36 AM ]

Alex,
Sorry for not being clear enough:

(clojure.lang.RT/loadLibrary "/tmp/a.so") - does not work, loadLibrary is expecting properly named library files existing within a path pointed to by either LD_LIBRARY_PATH or java.library.path. For example, if you have the file liba.so existing somewhere on LD_LIBRARY_PATH than (clojure.lang.RT/loadLibrary "a") would work.

For use cases where you dont want to rely on environment settings or preexistence of the .so files, you could use System/load instead of System/loadLibrary. For instance if you want to download the proper .so files for some platform from within the running clojure program, or have the .so files exists inside your jar. In these cases you would want to download or extract them from the jar respectively, place them in some temp folder (which is not in LD_LIBRARY_PATH) and load them using their full path, to be used in the running program (i.e. no special scripts or preparation tasks).

If I was to write a patch to src/jvm/clojure/lang/RT.java, it would be:

// Load a library in the System ClassLoader instead of Clojure's own, from anywhere
public static void loadObject(String libFullPath){ System.load(libFullPath); }





[CLJ-2242] clojure.test: `use-fixtures` should (or should be able to) run for for a `testing` context Created: 25/Sep/17  Updated: 25/Sep/17

Status: Open
Project: Clojure
Component/s: None
Affects Version/s: None
Fix Version/s: None

Type: Enhancement Priority: Minor
Reporter: Víctor M. Valenzuela Assignee: Unassigned
Resolution: Unresolved Votes: 0
Labels: clojure.test


 Description   

As implemented currently, the clojure.test/use-fixtures hooks will run on a per-deftest basis, but not on a per-testing (clojure.test/testing) basis.

So, for a given deftest with multiple testing clauses, use-fixtures will be only run once.

Coming from an RSpec (Ruby) background this is surprising - the default behavior is the opposite, and in my opinion more intuitive.

Personally, this resulted in me writing tests that gave false positives - i.e. it rendered a portion of my test suite useless until I noticed this (by mere luck).

Anecdotically, I commented the issue with another Clojure developer and he also suffered the issue once. I mean, I'm not alone in this one, and surely other developers have stumbled with this despite the lack of JIRA report until now.

In any case, the current working alternative (one deftest per test which depends on use-fixtures) can be overly verbose (in comparison with my original attempt), and also the need for doing that in the first place can be easy to forget.

I'd ask to:

  • Implement an option for use-fixtures called e.g. :run-nested?
  • Make this option mandatory to specify, i.e. programmers pass either :run-nested? true or :run-nested? false, unless a global default is set beforehand. People should be aware of this nuance and be forced to think about it at least once (else they risk false positives/negatives)

I hope this sounds like a reasonable, needed, non-breaking proposal.

Cheers - Victor



 Comments   
Comment by Alex Miller [ 25/Sep/17 6:56 PM ]

Hi Victor, it would be helpful if you provided a full code example in the description.

Fixtures to me seem pretty well defined in the docs (https://github.com/clojure/clojure/blob/master/src/clj/clojure/test.clj#L158-L193) and are strongly tied to the invocation of tests as functions in vars. Invoking them on a per-`testing` basis seems like significantly different to me.

Another option would be to create a new kind of fixture with a different scope that would get reinvoked in `testing` blocks in particular like a `:group` assertion. I guess that would also apply to nested `testing` blocks?

I don't actually have a good sense of what kind of test would need this, hence the request for an example.





[CLJ-2240] Load user.cljc in classpath root. Created: 22/Sep/17  Updated: 22/Sep/17

Status: Open
Project: Clojure
Component/s: None
Affects Version/s: None
Fix Version/s: None

Type: Enhancement Priority: Minor
Reporter: Sameer Rahmani Assignee: Unassigned
Resolution: Unresolved Votes: 0
Labels: None

Attachments: Text File CLJ-2240.patch    
Patch: Code
Approval: Triaged

 Description   

Right now Clojure only loads user.clj but it might be a good idea to load user.cljc if it was in classpath as well.






[CLJ-2238] NPE when print-duping Void/TYPE Created: 18/Sep/17  Updated: 18/Sep/17

Status: Open
Project: Clojure
Component/s: None
Affects Version/s: None
Fix Version/s: None

Type: Defect Priority: Minor
Reporter: Nicola Mometto Assignee: Unassigned
Resolution: Unresolved Votes: 0
Labels: print

Attachments: Text File 0001-CLJ-2238-handle-Void-TYPE-in-print-dup.patch    
Patch: Code
Approval: Triaged

 Description   
user=> (binding [*print-dup* true] (print Void/TYPE))

NullPointerException   java.io.PrintWriter.write (PrintWriter.java:473)
#=(identity user=>

Patch: 0001-CLJ-2238-handle-Void-TYPE-in-print-dup.patch






[CLJ-2237] Provide a type predicate for ex-info? Created: 18/Sep/17  Updated: 19/Sep/17

Status: Open
Project: Clojure
Component/s: None
Affects Version/s: Release 1.9
Fix Version/s: None

Type: Enhancement Priority: Minor
Reporter: Christian Romney Assignee: Unassigned
Resolution: Unresolved Votes: 1
Labels: None

Attachments: Text File Add-ex-info-type-predicate.patch    
Patch: Code and Test

 Description   

A type predicate for ExceptionInfo would be as useful when writing specs as the other nice additions landing in Clojure 1.9 (e.g. nat-int?, pos-int?, etc).



 Comments   
Comment by Alex Miller [ 18/Sep/17 9:39 PM ]

It would be helpful to list an example of some code where this would come in handy (I think examples do exist in catch handling) and to give some indication of how common it is.

Comment by Christian Romney [ 19/Sep/17 7:44 AM ]

One place where it can be useful is specing data that's pulled from a channel. http://swannodette.github.io/2013/08/31/asynchronous-error-handling. David's example immediately throws the exception via the <? macro, but the fact that exception info might be put on the channel at all means it can be seen by functions we might have in a transducer. These functions can leverage spec, where this predicate would come in very handy. I'd paste code, but the example immediately at hand is proprietary. Is this a sufficiently good description, or does the case need more support?

Also, while David's example is CLJS I've used this pattern in Clojure core.async code (e.g. asynchronous Pedestal interceptors).

Comment by Christian Romney [ 19/Sep/17 12:17 PM ]

Also, from an aesthetic[1] point of view adding a type predicate (ex-info?) for a pretty important core Clojure abstraction that already has a constructor (ex-info) and an accessor (ex-data) just seems to round out / complete the API. Not sure how that compelling you or Rich might find that argument, but I figured it couldn't hurt the case.

[1] My sense of aesthetic here is colored by some of the ideas in Mitch Wand & Dan Friedman's Essentials of Programming Languages.





[CLJ-2229] explain-data intermittently produces incorrect report when specs rely on dynamic vars Created: 30/Aug/17  Updated: 30/Aug/17

Status: Open
Project: Clojure
Component/s: None
Affects Version/s: Release 1.9
Fix Version/s: None

Type: Defect Priority: Minor
Reporter: David Chelimsky Assignee: Unassigned
Resolution: Unresolved Votes: 0
Labels: error-reporting, spec


 Description   

We've got some specs whose behavior is controlled by the value of a dynamic var. When we run

(binding [*behave-differently* true] (s/explain-data ::spec-with-conditional-behavior nested-structure-including-maps))

the result shows that some of the nested specs are not operating as expect based on the value of *behave-differently*. I believe that this is due to some implementations of explain* producing lazy seqs that aren't realized until after the binding form has closed. Anecdotally, forcing realization within the binding form results in the expected behavior.



 Comments   
Comment by David Chelimsky [ 30/Aug/17 7:55 AM ]

FYI - the description didn't format as I expected and I don't have edit permissions.

Comment by Alex Miller [ 30/Aug/17 8:09 AM ]

I gave you edit groups David.

Comment by David Chelimsky [ 30/Aug/17 8:24 AM ]

Thanks Alex. I updated the Description.

Comment by Alex Miller [ 30/Aug/17 8:52 AM ]

A working example would help.





[CLJ-2228] Unroll constantly to improve performance of multi-arity calls Created: 28/Aug/17  Updated: 27/Sep/17

Status: Open
Project: Clojure
Component/s: None
Affects Version/s: None
Fix Version/s: None

Type: Enhancement Priority: Minor
Reporter: Nathan Marz Assignee: Unassigned
Resolution: Unresolved Votes: 0
Labels: performance

Attachments: Text File 0001-CLJ-2228-Improve-performance-of-constantly.patch    
Patch: Code and Test
Approval: Prescreened

 Description   

This was found to be a hot spot when testing some Specter use cases.

Review of common uses of constantly shows that the arity 1 case is the most common use of `constantly` in the wild, so only unrolled to two arguments.

Perf test:

(use 'criterium.core)
(def f (constantly 1))
(bench (dotimes [_ 1000] (f)))
(bench (dotimes [_ 1000] (f 1)))
(bench (dotimes [_ 1000] (f 1 2)))

;; Results:
;; Arity Before         After
;; 0     611.455589 ns  607.800747 ns    (no change was expected)
;; 1     3.098828 µs    611.116510 ns    (~5x improvement)
;; 2     3.508726 µs    620.415032 ns    (~5x improvement)

Patch: 0001-CLJ-2228-Improve-performance-of-constantly.patch

Prescreened by: Alex Miller



 Comments   
Comment by Alex Miller [ 29/Aug/17 7:30 AM ]

I want to call out to our workflow diagram at https://dev.clojure.org/display/community/JIRA+workflow for process here. I've triaged this, but it's unlikely to be reviewed for the next release at this point unless it's been prescreened, and that requires a patch. So it would be helpful if you could:

  • mention your use case where this made a difference (Specter)
  • attach a patch
  • include the benchmarks in the ticket description

One question I have is how many arities to unroll. 10 seems like a lot? My guess is that up to 2 or 3 args would cover 90+% of cases but could poke around in https://crossclj.info/fun/clojure.core/constantly.html or do some other research to get an idea.

Comment by Alex Miller [ 29/Aug/17 7:38 AM ]

What is the max arity for constantly you encounter in Specter?

Comment by Alex Miller [ 29/Aug/17 8:04 AM ]

I did some poking around and I suspect arity 1 is the 98% use case and arity 0 making up most of the rest. I failed to actually find an example of any arity >1 although I'm sure there are a few of them out there. I think unrolling to arity 2 would be more than sufficient.

Comment by Nathan Marz [ 27/Sep/17 3:37 PM ]

Specter uses constantly for arity 1, so this patch is sufficient for Specter.





[CLJ-2225] clojure.core/assert docstring is incorrect Created: 22/Aug/17  Updated: 13/Sep/17

Status: Open
Project: Clojure
Component/s: None
Affects Version/s: None
Fix Version/s: None

Type: Enhancement Priority: Minor
Reporter: Gordon Syme Assignee: Unassigned
Resolution: Unresolved Votes: 0
Labels: docstring

Attachments: Text File clj-2225-20170913.patch    
Patch: Code
Approval: Triaged

 Description   

The assert macro has two arities, documented as (assert x) and (assert x message)

The docstring is:

Evaluates expr and throws an exception if it does not evaluate to logical true.

This is quite misleading since assert actually throws an Error. It is tempting to use assert assuming that a failure will lead to a stacktrace in logs, like any other Exception, but the reality is that the JVM will terminate.

The behaviour is correct, assert should cause a program to exit if the asserted condition is true but I regularly come across incorrect uses.

I'll work up a patch if people agree this is an issue.



 Comments   
Comment by Alex Miller [ 22/Aug/17 5:54 AM ]

Error is a subclass of Throwable, just like Exception and has no special behavior (although it does have the special intended meaning that most programs should not catch it). Whether or not your program does catch it, or exit, or log, is totally up to your program and not a property of being an Error.

The only change that I think might make sense here is to be more specific about the exception type.

Comment by Gordon Syme [ 13/Sep/17 5:46 AM ]

clj-2225-20170913.patch updates the assert macro docstring. I've wrapped the docstring at 70 characters, it seems to be a fairly common width used throughout the rest of the file.





[CLJ-2224] Support printing and reading of Java 8 java.time.Instant Created: 19/Aug/17  Updated: 20/Aug/17

Status: Open
Project: Clojure
Component/s: None
Affects Version/s: Release 1.8
Fix Version/s: None

Type: Enhancement Priority: Minor
Reporter: David Bürgin Assignee: Unassigned
Resolution: Unresolved Votes: 3
Labels: instant
Environment:

Java 8


Approval: Triaged

 Description   

In Clojure 1.9 alpha, limited support for Java 8 java.time.Instant was added, namely by (conditionally) extending the Inst protocol to that type.

It would be useful to enhance support for java.time.Instant further by

  • installing a print-method and print-dup for java.time.Instant
  • providing a read-instant function for reading java.time.Instant

This functionality is already provided in Clojure 1.8 today for the types java.util.Date, java.util.Calendar, and java.sql.Timestamp; extending it to java.time.Instant would be very helpful in environments using Java 8.






[CLJ-2215] Extend-protocol for array of Object does not work on array of subtypes of Object. Created: 31/Jul/17  Updated: 31/Jul/17

Status: Open
Project: Clojure
Component/s: None
Affects Version/s: Release 1.8
Fix Version/s: None

Type: Defect Priority: Minor
Reporter: Didier A. Assignee: Unassigned
Resolution: Unresolved Votes: 0
Labels: protocols

Approval: Triaged

 Description   

It appears that when you extend a protocol for "[Ljava.lang.Object;", the dispatch can not resolve for child types of "[Ljava.lang.Object;".

(defprotocol Table
  (t [this]))

(extend-protocol Table
  (Class/forName "[Ljava.lang.Object;")
  (t [this] this))

(t (make-array java.lang.String 0))
=> IllegalArgumentException No implementation of method: :t of protocol: #'test-t/Table found for class: [Ljava.lang.String;  clojure.core/-cache-protocol-fn (core_deftype.clj:568)

(t (make-array java.lang.Object 0))
=> ["[Ljava.lang.Object;" 1512480936 "[Ljava.lang.Object;@5a26a0a8"]

Yet Java is covariant on Object[]:

(instance? (Class/forName "[Ljava.lang.Object;") (make-array java.lang.String 0))
=> true
$ cat > Foo.java
public class Foo {
  public Object[] fooey;
  public Foo() {
    fooey = new String[10];
  }
}
$ javac Foo.java
$


 Comments   
Comment by Alex Miller [ 31/Jul/17 2:18 PM ]

Related: CLJ-1381





[CLJ-2214] Add binding conveyance to reducers. Created: 30/Jul/17  Updated: 30/Jul/17

Status: Open
Project: Clojure
Component/s: None
Affects Version/s: Release 1.8
Fix Version/s: None

Type: Enhancement Priority: Minor
Reporter: Didier A. Assignee: Unassigned
Resolution: Unresolved Votes: 1
Labels: reducers

Approval: Triaged

 Description   

Clojure reducers is the only parallel construct that does not support binding conveyance.

(def ^:dynamic bar "In main thread.")

(binding [bar "In reducers Thread."]
         (r/fold (fn [& args]
                   bar) (vec (range 100000))))
=> "In main thread."

You have to manually use bound-fn:

(binding [bar "In reducers Thread."]
         (r/fold (bound-fn [& args]
                           bar) (vec (range 100000))))
=> "In reducers Thread."

I propose it is enhanced to behave the same as pmap, future, agents and core.async.






[CLJ-2212] docstring for `defmethod` is imprecise Created: 28/Jul/17  Updated: 20/Aug/17

Status: Open
Project: Clojure
Component/s: None
Affects Version/s: Release 1.8
Fix Version/s: None

Type: Enhancement Priority: Minor
Reporter: Marc O'Morain Assignee: Unassigned
Resolution: Unresolved Votes: 0
Labels: docstring

Attachments: Text File clj-2212.patch    
Patch: Code
Approval: Triaged

 Description   

arglist:

(defmethod multifn dispatch-val & fn-tail)

docstring:

Creates and installs a new method of multimethod associated with dispatch-value.

There are a few issues with this docstring that make it hard to understand:

1. The & fn-tail argument is not documented. What is the type / shape of & fn-tail. I don't see any mention in the reference guide either - https://clojure.org/reference/multimethods
2. The docstring and arglists conflict about the name multimethod vs. multifn.
3. The docstring and arglists conflict about the name dispatch-value vs. dispatch-val.

Aside: on clojuredocs.org there is a mention of an optional name being allows on the method https://clojuredocs.org/clojure.core/defmethod#example-542692c7c026201cdc3269cd but I don't see that documented anywhere official. I'm not sure if it supported.



 Comments   
Comment by Alex Miller [ 28/Jul/17 9:00 AM ]

fn-tail here means "additional args as could be passed to fn", which does include the function name, so that is supported.

Comment by Marc O'Morain [ 19/Aug/17 3:01 PM ]

Adding patch

Comment by Alex Miller [ 20/Aug/17 5:03 PM ]

Rather than changing the docstring from multimethod to multifn, I would change the code from multifn to multimethod. Otherwise looks good.





[CLJ-2211] docstring of defn is not precise about the `attr-map?` and `body` arguments Created: 28/Jul/17  Updated: 20/Aug/17

Status: Open
Project: Clojure
Component/s: None
Affects Version/s: Release 1.8
Fix Version/s: None

Type: Enhancement Priority: Minor
Reporter: Marc O'Morain Assignee: Unassigned
Resolution: Unresolved Votes: 0
Labels: docstring

Attachments: Text File clj-2211.patch    
Approval: Triaged

 Description   

The docstring for defn is:

Same as (def name (fn [params* ] exprs*)) or (def name (fn ([params* ] exprs*)+)) with any doc-string or attrs added to the var metadata. prepost-map defines a map with optional keys :pre and :post that contain collections of pre or post conditions.

The arglist is:

(defn name doc-string? attr-map? [params*] prepost-map? body)

There are two issues that made this docstring hard to understand for me:

1. The docstring does not mention attr-map? - it took me a a bit of jumping around the docs to make the leap from attr-map? to "with any [...] attrs added to the var metadata".
2. The docstring makes reference to exprs*, but the arglist refers to body.



 Comments   
Comment by Alex Miller [ 28/Jul/17 8:24 AM ]

Patch welcome, would appreciate smallest change possible.

Comment by Marc O'Morain [ 19/Aug/17 3:24 PM ]

I've attached a patch to address two issues:

  • change 'exprs*' to 'body' to match the arglist.
  • change 'body+)' to 'body)+' when referring to the multi-artiy form.

I didn't address "any attrs" referring to attr-map, I wasn't sure the most "Clojury" way to phrase it.

Comment by Alex Miller [ 20/Aug/17 4:59 PM ]

Seems like the body change is missing a trailing right paren?

You have: (def name (fn ([params* ] body)+)
Should be: (def name (fn ([params* ] body)+))





[CLJ-2202] coll-of :min-count and :gen-max used together cause collections that are too large to be generated Created: 10/Jul/17  Updated: 10/Jul/17

Status: Open
Project: Clojure
Component/s: None
Affects Version/s: Release 1.9
Fix Version/s: Release 1.9

Type: Defect Priority: Minor
Reporter: Sebastian Oberhoff Assignee: Unassigned
Resolution: Unresolved Votes: 0
Labels: spec
Environment:

1.9.0-alpha16


Attachments: Text File clj-2202.patch    
Patch: Code
Approval: Vetted

 Description   

This should specify a spec whose generator always returns collections of size 4 or 5 but instead it generates collections of size 4 to 8:

(require '[clojure.spec.alpha :as s] '[clojure.spec.gen.alpha :as gen])
(map count (gen/sample (s/gen (s/coll-of char? :min-count 4 :gen-max 5))))
;; (4 7 8 8 4 7 4 5 5 7)

Cause: The max logic in s/every-impl is: (c/or max-count (max gen-max (c/* 2 (c/or min-count 0)))). If there is a max-count it's used, otherwise the larger of gen-max or 2*min-count is used. In this case, 2*min-count is 8. Seems like we should never generate more than gen-max though, so propose changing this logic to: (c/or max-count gen-max (c/* 2 (c/or min-count 0))).

Patch: clj-2202.patch






[CLJ-2199] Error attempting to unform unconformed keys (no :conform-keys opt) Created: 05/Jul/17  Updated: 12/Aug/17

Status: Open
Project: Clojure
Component/s: None
Affects Version/s: Release 1.9
Fix Version/s: Release 1.9

Type: Defect Priority: Minor
Reporter: Daniel Stockton Assignee: Unassigned
Resolution: Unresolved Votes: 0
Labels: spec

Attachments: Text File clj-2199.patch     Text File clj-2199.patch    
Approval: Vetted

 Description   

Minimal failing case:

(s/def ::key-spec (s/or :kw keyword? :str string?))
(s/def ::map-spec (s/map-of ::key-spec identity))
(println (s/unform ::map-spec (s/conform ::map-spec {:a :b})))
java.lang.UnsupportedOperationException: nth not supported on this type: Keyword

If keys are not conformed, we should also not attempt to unform them.



 Comments   
Comment by Daniel Stockton [ 08/Aug/17 8:29 AM ]

Add test and fix behaviour

Comment by Daniel Stockton [ 08/Aug/17 8:39 AM ]

Actually, although this passes all tests, it's not alright because it bypasses validation.

Comment by Daniel Stockton [ 12/Aug/17 4:23 AM ]

I think it passes for correctness this time but open to advice if this is not a good approach.





[CLJ-2190] clojure.spec.alpha/exercise-fn should emit a bettor error message when no implementation is found for a symbol Created: 27/Jun/17  Updated: 27/Jun/17

Status: Open
Project: Clojure
Component/s: None
Affects Version/s: Release 1.9
Fix Version/s: Release 1.9

Type: Enhancement Priority: Minor
Reporter: Abhirag Assignee: Unassigned
Resolution: Unresolved Votes: 0
Labels: errormsgs, spec
Environment:

Clojure 1.9.0-alpha17
test.check 0.10.0-alpha2


Attachments: Text File clj-2190.patch    
Patch: Code
Approval: Vetted

 Description   

Here we get a NullPointerException because although we do have a spec for my-reverse, we don't have an implementation for it, a more descriptive error message would help.

(require '[clojure.spec.alpha :as s])

(s/fdef foo :args (s/cat :x int?) :ret int?)
=> user/foo

(s/exercise-fn `foo)
NullPointerException   clojure.core/apply (core.clj:657)

Proposed: Check for a nil function and throw.

Patch: clj-2190.patch






[CLJ-2185] Standardize the running context of all Java to Clojure entry points. Created: 22/Jun/17  Updated: 22/Jun/17

Status: Open
Project: Clojure
Component/s: None
Affects Version/s: Release 1.7, Release 1.8, Release 1.9
Fix Version/s: None

Type: Enhancement Priority: Minor
Reporter: Didier A. Assignee: Unassigned
Resolution: Unresolved Votes: 0
Labels: bootstrap, interop

Approval: Triaged

 Description   

Clojure always must be handed the program counter from Java, the JVM can't load right into Clojure. So there's a few ways you can do it:

1) With a gen-class
2) From an already bootstrapped REPL
3) With clojure.main
4) With the Clojure runtime (RT.java)
5) With the Clojure Java API (https://clojure.github.io/clojure/javadoc/clojure/java/api/Clojure.html)
6) With popular tools such as Leiningen and Boot

Any time you want to run some Clojure code from Java, you need to do it with one of those mechanisms, and you cannot run Clojure code without going to Java first.

The issue is that, not all mechanisms hand over execution with an equal context.

#1, #4 and #5 all hand over execution from the context that the Clojure runtime executes in. That means that you get a minimal context running inside the "clojure.core" namespace. No binding is setup, and ns points to "clojure.core".

#3 and #6 seem all to hand over execution with a similar context, but one that's different from #1, #4 and #5. That is, they set common bindings such as ns, and set the current namespace to a freshly created namespace generally called user which has clojure.core referred in it. Meaning that code executed from #3 and #6 will be able to set common bindings and won't interfere with the clojure.core namespace.

#2's context will depend on how the REPL itself was bootstrapped from Java. In most cases, clojure.main, lein, boot, it will be similar to #3 and #6.

I believe that it would be healthy for Clojure to standardize the execution context when bootstrapping Clojure from Java, so that all above mechanism behave the same. This will make it so that all code will run equally from all bootstrapping mechanism. This is not currently the case, since missing bindings and the clojure.core current namespace can cause certain rare scenarios to fail under #1, #4 and #5, but work under #3 and #6.

To not break backwards compatibility, it would be better to enhance #1, #4 and #5 so that they also set common bindings and change the current namespace to a fresh user namespace with clojure.core referred in it.

Initial discussion on the mailing list: https://groups.google.com/forum/#!topic/clojure/6CXUNuPIUyQ






[CLJ-2181] try accepts multiple catch blocks for the same class Created: 07/Jun/17  Updated: 07/Jun/17

Status: Open
Project: Clojure
Component/s: None
Affects Version/s: Release 1.8, Release 1.9
Fix Version/s: None

Type: Defect Priority: Minor
Reporter: Justin Glenn Smith Assignee: Unassigned
Resolution: Unresolved Votes: 1
Labels: compiler
Environment:

any


Approval: Triaged

 Description   

try silently accepts multiple catch blocks for the same class, but only the first one gets called

user=> (try (/ 1 0) (catch Exception _ (println "a")) (catch Exception _ (println "b")))
a
nil





[CLJ-2172] Error message for non integer index into vector could be improved. Created: 31/May/17  Updated: 31/May/17

Status: Open
Project: Clojure
Component/s: None
Affects Version/s: Release 1.9
Fix Version/s: None

Type: Enhancement Priority: Minor
Reporter: Russ Olsen Assignee: Unassigned
Resolution: Unresolved Votes: 0
Labels: errormsgs

Approval: Triaged

 Description   

The exception generated in APersistentVector/invoke and also in APersistentVector/assoc when the index is not an integer would be better if it included the class of the object actually passed in and would be better still if it included the actual value.



 Comments   
Comment by Alex Miller [ 31/May/17 7:25 PM ]

Needs assessment of current paths through core that reach this (and whether the change in question would be better or worse when that happens).





[CLJ-2166] instrument exception doesn't contain function name in ex-data Created: 24/May/17  Updated: 24/Jun/17

Status: Open
Project: Clojure
Component/s: None
Affects Version/s: Release 1.9
Fix Version/s: Release 1.9

Type: Enhancement Priority: Minor
Reporter: James Reeves Assignee: Unassigned
Resolution: Unresolved Votes: 4
Labels: spec

Attachments: Text File clj2166.patch    
Patch: Code
Approval: Screened

 Description   

When an instrumented function fails, it throws an IExceptionInfo. The ex-data of this exception contains the arguments that failed, but not the function that was called.

(require '[clojure.spec.alpha :as s] '[clojure.spec.test.alpha :as stest])
(defn foo [x] (+ x 1))
(s/fdef foo :args (s/cat :x number?))
(stest/instrument `foo)
(foo "x")  ;; ExceptionInfo Call to #'user/foo did not conform to spec:
(ex-data *e)

{:clojure.spec.alpha/problems
 [{:path [:args :x], :pred number?, :val "x", :via [], :in [0]}],
 :clojure.spec.alpha/args ("x"),
 :clojure.spec.alpha/failure :instrument,
 :clojure.spec.test.alpha/caller
 {:file "form-init1493151512736136730.clj",
  :line 101,
  :var-scope user/eval23284}}

*Proposed: Add an extra key clojure.spec.alpha/fn that has the symbol of the var under instrumentation.

After:

user> (ex-data *e)
=>
{:clojure.spec.alpha/problems [{:path [:args :x],
                                :pred clojure.core/number?,
                                :val "x",
                                :via [],
                                :in [0]}],
 :clojure.spec.alpha/spec #object[clojure.spec.alpha$regex_spec_impl$reify__1210
                                  0x4db4a004
                                  "clojure.spec.alpha$regex_spec_impl$reify__1210@4db4a004"],
 :clojure.spec.alpha/value ("x"),
 :clojure.spec.alpha/fn user/foo,
 :clojure.spec.alpha/args ("x"),
 :clojure.spec.alpha/failure :instrument,
 :clojure.spec.test.alpha/caller {:file "form-init2105027941758372775.clj",
                                  :line 1,
                                  :var-scope user/eval1050}}

Patch: clj2166.patch

Screened by: Alex Miller



 Comments   
Comment by Josh Jones [ 23/Jun/17 3:58 PM ]

I've done a small patch to add this, but is there a consensus on whether the value for the newly-added key should be the var itself (referenced by a :clojure.spec.alpha/var key), or the symbol (referenced by a :clojure.spec.alpha/fn key)?

Comment by Alex Miller [ 23/Jun/17 4:29 PM ]

Symbol

Comment by Josh Jones [ 23/Jun/17 6:22 PM ]

I've attached a one-liner patch, and welcome any comments or issues.

Note that although I signed the CA today (Rich also signed) and can thus officially submit, I am not yet on "Signed Contributor Agreement" list at https://clojure.org/community/contributors . I can't edit posts on Jira yet, which I why I did not edit to mark that a patch is present.

I've read the contrib guide but this is my first clojure patch, so please bear with me if I have made any errors in the workflow or otherwise here. Thanks!

Using the example in the original ticket, the new output of ex-data is:

user> (ex-data *e)
=>
{:clojure.spec.alpha/problems [{:path [:args :x],
                                :pred clojure.core/number?,
                                :val "x",
                                :via [],
                                :in [0]}],
 :clojure.spec.alpha/spec #object[clojure.spec.alpha$regex_spec_impl$reify__1210
                                  0x4db4a004
                                  "clojure.spec.alpha$regex_spec_impl$reify__1210@4db4a004"],
 :clojure.spec.alpha/value ("x"),
 :clojure.spec.alpha/fn user/foo,
 :clojure.spec.alpha/args ("x"),
 :clojure.spec.alpha/failure :instrument,
 :clojure.spec.test.alpha/caller {:file "form-init2105027941758372775.clj",
                                  :line 1,
                                  :var-scope user/eval1050}}
Comment by Alex Miller [ 24/Jun/17 7:29 AM ]

Thanks, looks good. I've also added the groups for you to have edit rights.





[CLJ-2164] case fails when a single single clause with an empty test seq is used Created: 22/May/17  Updated: 22/May/17

Status: Open
Project: Clojure
Component/s: None
Affects Version/s: Release 1.8
Fix Version/s: None

Type: Defect Priority: Minor
Reporter: Chris Blom Assignee: Unassigned
Resolution: Unresolved Votes: 1
Labels: None

Attachments: Text File 0001.patch    
Patch: Code and Test
Approval: Triaged

 Description   

It is not possible to use case with a single empty seq of options, or with a single seq of options and a default clause.

I would expect

(case 1 () :a :none)

to return :none, instead it fails with an uninformative exception: "Unhandled clojure.lang.ArityException: Wrong number of args (-2) passed to: core/max"

I would expect (case 1 () :a) to fail with "java.lang.IllegalArgumentException: No matching clause", but instead it also fails with
"Unhandled clojure.lang.ArityException: Wrong number of args (-2) passed to: core/max"

This seems inconsistent, as passing an empty list of options is fine when there are other alternatives:

(case 1 () :a 2 :b :none)

returns :none, as expected

The attached patch removes the test-clause pairs with empty test lists before further conversion to case*, and adds tests.



 Comments   
Comment by Chris Blom [ 22/May/17 8:54 AM ]

oops, typo in the title (duplicated "single")

Comment by Alex Miller [ 22/May/17 8:55 AM ]

An empty match list in case seems like it should be an error - maybe this should fail to compile rather than ignoring?

Comment by Chris Blom [ 22/May/17 9:07 AM ]

An empty list of options is currently supported when multiple clauses are given, so failing to compile on empty lists would be a breaking change.

This works in 1.8:

(case 1
() :never-happens
1 :ok
:default)
=> :ok

But this does not:

(case 1
() :never-happens
:default)
=> throws clojure.lang.ArityException: Wrong number of args (-2) passed to: core/max

Only failing when no other clauses are given seems very inconsistent to me.

Comment by Nicola Mometto [ 22/May/17 9:39 AM ]

an empty list doesn't make any sense in case as the correct way to match a literal empty list is `(case () (()) :empty)`. I don't see any value in making it not throw and my vote is to have the `case` macro complain at compile time every time a `()` ise used

Comment by Alex Miller [ 22/May/17 9:57 AM ]

() would never match anything now, so failing on () would not break any existing case that matches.

The one case I can imagine might exist though is a macro that creates a case and could potentially programmatically create an empty case list? Something like this:

(defmacro make-case [xs] `(defn ~'foo [e#] (case e# ~xs "matched" "nope")))
Comment by Chris Blom [ 22/May/17 10:00 AM ]

I'm not using this to match empty lists, I ran into this corner case when generating case statements from a DSL.
While it is a pathological case, I disagree that is does not make any sense, an empty list here simply represents no alternatives,
so the clause wil never match and its result-expr will never run.

My point is that now (in clojure 1.8) this is allowed:

(case a
() :never-happens
1 :a
2 :b
:default)

it is equivalent to (as an empty list never matches)

(case a
1 :a
2 :b
:default)

But

(case a
() :never-happens
:default)

gives an uninformative error.

I argue that it should be equivalent to

(case a
:default)

as rejecting empty lists in case statements in general would be a breaking change,
and only rejecting empty lists when no other clauses are present is inconsistent.

Comment by Chris Blom [ 22/May/17 10:11 AM ]

() would never match anything now, so failing on () would not break any existing case that matches.

As () in case is allowed in clojure =<1.8 (just not when its the only clause), letting the compiler reject it would potentially break existing code.

The one case I can imagine might exist though is a macro that creates a case and could potentially programmatically create an empty case list?

That is exactly how i ran into this problem





[CLJ-2162] condp should accept macros as first argument. Also would allow proper inlining Created: 07/May/17  Updated: 15/May/17

Status: Open
Project: Clojure
Component/s: None
Affects Version/s: Release 1.9
Fix Version/s: None

Type: Feature Priority: Minor
Reporter: A. R Assignee: Unassigned
Resolution: Unresolved Votes: 0
Labels: None


 Description   

Currently condp takes the value of the first argument pred. This has two disadvantages:

  1. It won't allow macros to be used as the first argument
  2. It will be less efficient since the function calls can't be inlined anymore (common idiom: condp = x ...)

If pred is a symbol the let binding can be avoided. This would properly inline = which is common.



 Comments   
Comment by Alex Miller [ 07/May/17 3:30 PM ]

Concrete examples would help.

Comment by A. R [ 08/May/17 12:36 AM ]

In my example I have a macro that matches routes. Since they're static this can be optimized during compile time:

(condp route-match? x [:comments :view] 1 [:posts :edit] 2 [_ :delete] 3)

Would also provide a faster assoc for `records` which generates a `condp` with `identical`.





[CLJ-2160] LispReader allows no-ops macros to sneak in certain other forms (namespaced maps, tagged literals and anonymous arguments) Created: 25/Apr/17  Updated: 25/Apr/17

Status: Open
Project: Clojure
Component/s: None
Affects Version/s: Release 1.9
Fix Version/s: None

Type: Defect Priority: Minor
Reporter: Christophe Grand Assignee: Unassigned
Resolution: Unresolved Votes: 0
Labels: reader

Attachments: Text File clj2160-2.patch     Text File clj2160.patch    
Patch: Code
Approval: Triaged

 Description   

No-op macros are line comments (starting with #! or ;), #_, reader conditional (splicing or not) with no matching feature.
Furthermore once a no-op macro has been read regular whitespace are allowed anew.

Examples:
Namespaced map #foo{:bar :baz}

#:#_()#! bang bang
#?(:whatever 42); now a blank line

#?@(:default ())foo
{:bar :baz}

Tagged literal #inst "2017-04-24T09:11:29.878-00:00"

##_()#! bang bang
#?(:whatever 42); now a blank line

inst "2017-04-24T09:11:29.878-00:00"

Anonymous argument: #(do %1)

#(do %#_()#! bang bang
#?(:whatever 42); now a blank line

#?@(:default ())1)

In addition anonymous arguments implementation is leaky (any %n is accepted as long as n is (-2.0 -1.0] (mapping to %&) and [1.0 Infinity) and any representation can be used (bigdec or bigint or float or integers in any basis).

#(list %#_(first arg)1.00000001 %#_(secong arg)2r10 %#_(rest arg)-1.5)


 Comments   
Comment by Christophe Grand [ 25/Apr/17 6:31 AM ]

The patch extracts the body from the read loop to expose a readSome method that returns either a form or the reader (if no valued form has been read starting at the current position).
This patch also adds a regex pattern to validate anonymous args.

Comment by Christophe Grand [ 25/Apr/17 7:35 AM ]

clj2160-2 is clj2160 + two redundant checks that were not removed





[CLJ-2154] Spec macros keys and keys* silently ignores non-keywords given in the vectors for named arguments :req and :req-un Created: 17/Apr/17  Updated: 17/Apr/17

Status: Open
Project: Clojure
Component/s: None
Affects Version/s: Release 1.9
Fix Version/s: None

Type: Defect Priority: Minor
Reporter: Rovanion Luckey Assignee: Unassigned
Resolution: Unresolved Votes: 0
Labels: spec
Environment:

Ubuntu 16.04 32-bit, Clojure 1.9.0-alpha15


Approval: Triaged

 Description   

If we try to pass a non-keyword to `clojure.spec/keys` or `clojure.spec/keys*` on the named argument `:opt` or `:opt-un` we get an assertion error:

(spec/valid? (spec/keys :opt ['a 5]) {})
;1. Caused by java.lang.AssertionError
;   Assert failed: all keys must be namespace-qualified keywords
;   (every? (fn* [p1__13667#] (c/and (keyword? p1__13667#) (namespace
;   p1__13667#))) (concat req-keys req-un-specs opt opt-un))

(spec/valid? (spec/keys* :opt-un ['a 5]) {})
;1. Caused by java.lang.AssertionError
;   Assert failed: all keys must be namespace-qualified keywords                    
;   (every? (fn* [p1__13667#] (c/and (keyword? p1__13667#) (namespace               
;   p1__13667#))) (concat req-keys req-un-specs opt opt-un))

But if we do the same for the named arguments `:req` and `:req-un` the arguments are silently ignored and the call to `keys` returns a spec matching any map without any requirements:

(spec/valid? (spec/keys :req ['a 5]) {})
=> true
(spec/valid? (spec/keys :req-un ['a 5]) {})
=> true
(spec/valid? (spec/keys* :req ['a 5]) {})
=> true
(spec/valid? (spec/keys* :req-un ['a 5]) {})
=> true

An assertion should probably be thrown for the `:req(-un)?` args too.






[CLJ-2135] clojure.spec/Spec implementations that don't implement IObj get silently dropped in s/def Created: 23/Mar/17  Updated: 23/Mar/17

Status: Open
Project: Clojure
Component/s: None
Affects Version/s: None
Fix Version/s: None

Type: Enhancement Priority: Minor
Reporter: Thomas Heller Assignee: Unassigned
Resolution: Unresolved Votes: 0
Labels: spec

Attachments: Text File clj-2135.patch    
Patch: Code
Approval: Triaged

 Description   
(deftype MySpec []
  s/Spec
  (conform* [_ x]
    ::s/invalid))

(s/def ::x (MySpec.))

(s/explain ::x :foo)

This will fail with a "Unable to resolve spec: :user/x" exception, but the def succeeded. Switching the deftype to defrecord fixes the problem.

Cause: The with-name function has cond options for ident?, regex?, and IObj. If none of these succeed, there is no fallthrough case and the s/def will silently return nil.

Proposed: Throw an error in the fallthrough case.

Patch: clj-2135.patch






[CLJ-2134] Update the docstring of `with-redefs` to reflect how little the macro should be used Created: 23/Mar/17  Updated: 23/Mar/17

Status: Open
Project: Clojure
Component/s: None
Affects Version/s: None
Fix Version/s: None

Type: Enhancement Priority: Minor
Reporter: Timothy Baldridge Assignee: Unassigned
Resolution: Unresolved Votes: 0
Labels: docstring

Approval: Triaged

 Description   

Currently the docstring for `with-redefs` recommends itself for use in testing. However there are a number of reasons why using this macro for testing is suboptimal:

  • `with-redefs` "bindings" are not transferred to new threads since it's a global mutation
  • users can get runtime errors if they redef a primitive type-hinted function to a function taking only objects
  • If parts of the body of `with-redefs` is delayed (via a delay, go block, etc.) that code may not see the new root
  • The mutation is global so it "leaks" outside the current scope into other code that may currently be running in another thread
  • Clojure tends to shun global mutation, and yet this macro isn't marked with a `!` nor properly warns users about the dangers mentioned here

Due to these reasons I often encounter new users using `with-redefs` without understanding the ramifications of doing so. All this behavior makes sense if a user understands how Vars work, but that's a lot of knowledge to take on for a new user.

Suggestion:
Remove the suggestion that `with-redefs` be used in testing
Add a few notes of warning about global mutation, and concurrency issues with the macro.






[CLJ-2133] Clarify documentation for the satisfies? function. Created: 23/Mar/17  Updated: 23/Mar/17

Status: Open
Project: Clojure
Component/s: None
Affects Version/s: Release 1.8, Release 1.9
Fix Version/s: None

Type: Enhancement Priority: Minor
Reporter: David Chelimsky Assignee: Unassigned
Resolution: Unresolved Votes: 1
Labels: docstring
Environment:

N/A



 Description   

The docs for satisfies? says "Returns true if x satisfies the protocol", but does not define the meaning of "satisfies". The function returns true when type and protocol are referenced in the same call to either extend-type or extend-protocol even when none of the protocol functions are implemented. I think the doc should be specific about this to avoid confusion.






[CLJ-2124] Catch multiple exceptions in a single catch block Created: 12/Mar/17  Updated: 15/May/17

Status: Open
Project: Clojure
Component/s: None
Affects Version/s: None
Fix Version/s: None

Type: Feature Priority: Minor
Reporter: Daniel Compton Assignee: Unassigned
Resolution: Unresolved Votes: 6
Labels: exceptions, try-catch

Approval: Triaged

 Description   

Java 7 and up support multi-catch exceptions (http://www.oracle.com/technetwork/articles/java/java7exceptions-486908.html). It would be handy if Clojure also supported them to prevent catching something like Exception and then writing manual logic to check the Exception type, or duplicating the logic over multiple catch blocks.

A possible syntax for this could be:

(try (fn-that-throws)
     (catch (UnknownHostException NoRouteToHostException) e
       (go-offline)))

Prior art for this is a try* macro: https://gist.github.com/Gonzih/5814945.

One nuance to handle is

Edit: Note that in Java 7, you cannot both catch ExceptionA& ExceptionB in the same time, if ExceptionB is inherited(directly or indirectly) from ExceptionA. Compiler will complain: The exception ExceptionB is already caught by the alternative ExceptionA. - http://stackoverflow.com/a/3495968/826486

I tried searching to see if this had been asked already, but got mountains of results. I didn't see anything in the first few pages though.






[CLJ-2109] Protocol methods not instrumented Created: 13/Feb/17  Updated: 13/Feb/17

Status: Open
Project: Clojure
Component/s: None
Affects Version/s: Release 1.9
Fix Version/s: None

Type: Defect Priority: Minor
Reporter: Stuart Sierra Assignee: Unassigned
Resolution: Unresolved Votes: 2
Labels: spec
Environment:

Clojure 1.9.0-alpha14


Approval: Triaged

 Description   

Spec instrument does not work on protocol methods. Invalid arguments will be accepted silently with no error. Protocol vars are included in the return value of (instrument).

Steps to reproduce

(require
 '[clojure.spec :as s]
 '[clojure.spec.test :as test])

(defprotocol P
  (method [this arg]))

(defrecord R []
  P
  (method [this arg]
    (str "R.method called with " (pr-str arg))))

(s/fdef method
  :args (s/cat :this any?
               :arg number?))

(defn wrapped [this arg]
  (method this arg))

(s/fdef wrapped
  :args (s/cat :this any?
               :arg number?))

(test/instrument)

(println (method (->R) "not a number"))

(println (wrapped (->R) "not a number"))

This code produces the output:

R.method called with "not a number"
clojure.lang.ExceptionInfo: Call to #'user/wrapped did not conform to spec:
In: [1] val: "not a number" fails at: [:args :arg] predicate: number?
...

Possible resolutions

1. Add support to instrument for protocol methods
2. Document that instrument does not work on protocol methods, do not return protocol method Vars from (instrument), throw exception if protocol method Vars are included in the symbols passed to (instrument syms)

See also

CLJ-1941 describes a different case where instrument does not work. This issue was identified in a comment.

Workarounds

This issue can be avoided by wrapping protocol methods in normal functions and spec'ing the functions. This is already common practice.






[CLJ-2095] Doc s/gen overrides do not take effect inside custom generators Created: 03/Jan/17  Updated: 03/Jan/17

Status: Open
Project: Clojure
Component/s: None
Affects Version/s: Release 1.9
Fix Version/s: None

Type: Enhancement Priority: Minor
Reporter: Russell Mull Assignee: Unassigned
Resolution: Unresolved Votes: 0
Labels: docstring, generator, spec
Environment:

clojure 1.9.0-alpha14



 Description   

Custom generators may build (via fmap/bind) on spec generators. Generator overrides at the top level will not take effect inside custom generators:

(require '[clojure.spec :as s])
(require '[clojure.test.check.generators :as gen])

;; A map that holds a single integer value
(s/def ::val integer?)
(s/def ::body (s/keys :req [::val]))

;; This spec matches stringified versions of 'body'.
;; (read-string is for demonstration purposes only, of course)
(s/def ::stringy-body
  (s/with-gen
    (s/and string? #(s/valid? ::body (read-string %)))
    #(gen/fmap pr-str (s/gen ::body))))

(s/valid? ::stringy-body "{:user/val 37}") ;; => true

;; This makes various stringified maps, as expected
(take 3 (gen/sample (s/gen ::stringy-body)))
;; => ("#:user{:val -1}" "#:user{:val 0}" "#:user{:val -1}")

;; *** But the overrides don't get passed through ***
(take 3 (gen/sample (s/gen ::stringy-body {::val #(s/gen #{42})})))
;; ("#:user{:val -1}" "#:user{:val 0}" "#:user{:val 0}")

Should consider documenting this in s/gen, s/with-gen, etc.



 Comments   
Comment by Alex Miller [ 03/Jan/17 5:39 PM ]

When you use with-gen, you're basically overriding the built-in gen mechanism (which supports overrides) and providing your own (opaque to spec) generator. You should not expect overrides to take effect inside a custom generator.

Comment by Russell Mull [ 03/Jan/17 5:41 PM ]

That makes sense, but in lieu of that I expected (and went looking for) some way to get at the overrides map from the function passed to s/with-gen, and found none.

Comment by Russell Mull [ 03/Jan/17 5:48 PM ]

... I didn't fully parse your comment the first time around. I can see from the implementation that a custom generator (gfn internally) is never passed any of the contextual information that the builtin specs have at hand. As it sounds like this is intentional, it would be useful to note this limitation in the docstring for s/gen or perhaps s/with-gen.

Comment by Alex Miller [ 03/Jan/17 5:58 PM ]

It's not a crazy idea, but it doesn't seem like there's any way this could be done in the current impl without some pretty significant changes.





[CLJ-2082] Improve documentation of clojure.walk/walk Created: 11/Dec/16  Updated: 24/May/17

Status: Open
Project: Clojure
Component/s: None
Affects Version/s: Release 1.8
Fix Version/s: None

Type: Enhancement Priority: Minor
Reporter: David Cook Assignee: Unassigned
Resolution: Unresolved Votes: 0
Labels: docs, docstring, documentation


 Description   

The documentation for the clojure.walk module isn't clear on which methods recurse through data structures, and which only operate on the outermost layer. The documentation for clojure.walk/walk and clojure.walk/postwalk both use forms of the word "traverse," and there's nothing calling out that clojure.walk/walk, unlike the rest of the functions in the namespace, doesn't recurse through the provided form.



 Comments   
Comment by Martin Clausen [ 24/May/17 1:33 AM ]

The potential recursive behaviour of clojure.walk/walk depends on the inner function passed to it and is not inherent to clojure.walk/walk itself. If you look at the source code for clojure.walk/prewalk and clojure.walk/postwalk, they are both implemented using walk, but passed a recursive inner function.





[CLJ-2081] for macro spec should know :let can't go in the first position Created: 09/Dec/16  Updated: 09/Dec/16

Status: Open
Project: Clojure
Component/s: None
Affects Version/s: Release 1.9
Fix Version/s: None

Type: Enhancement Priority: Minor
Reporter: lvh Assignee: Unassigned
Resolution: Unresolved Votes: 0
Labels: spec

Approval: Triaged

 Description   

The for macro does not support :let in the first position. This was reported in CLJ-207 and a patch was produced but rejected. Since we have spec for macros now, it might be an opportunity to provide a better error message. (I think first-place let should be fine, but that's neither here nor there.)






[CLJ-2072] Primitive type aliases do not always work due to meta data evaluation Created: 01/Dec/16  Updated: 01/Dec/16

Status: Open
Project: Clojure
Component/s: None
Affects Version/s: Release 1.8
Fix Version/s: None

Type: Enhancement Priority: Minor
Reporter: Volkert Oakley Jurgens Assignee: Unassigned
Resolution: Unresolved Votes: 0
Labels: typehints


 Description   

Primitive alias do not work when the meta data is evaluated, for example in the case of def.

In this example, char is interpreted to be the function char rather than the type alias. This is because clojure.lang.Compiler$DefExpr$Parser.parse evaluates the meta data on the symbol.

(def ^char x \space)
(String/valueOf x)
=> CompilerException java.lang.IllegalArgumentException: Unable to resolve classname: clojure.core$char@7b82f7d1

Instead, this has to be written as

(def ^Character/TYPE x \space)

However, when using primitive type hints in-line, they work fine:

(def x \space)
(String/valueOf ^char x)

Primitive type aliases should be handled consistently.

Googling shows that this seems to be a well-known problem, but I have not found a Jira issue for it.



 Comments   
Comment by Alex Miller [ 01/Dec/16 7:48 PM ]

Var meta is evaluated. Meta on function signatures and other locations is not. Those two things are that way for historical reasons but would at this point be breaking changes if we changed either of them, so they are not going to change.

One thing that could potentially be done is to detect this particular problem when it happens and create a warning or error. In particular, this would present as a var whose meta :tag is a function.

So, if you want to re-write this ticket as a request for error message, that is something worth considering.





[CLJ-2067] (s/def ::a ::b) throws unable to resolve error if ::b is not defined Created: 22/Nov/16  Updated: 22/Nov/16

Status: Open
Project: Clojure
Component/s: None
Affects Version/s: Release 1.9
Fix Version/s: Release 1.9

Type: Defect Priority: Minor
Reporter: Oliver George Assignee: Unassigned
Resolution: Unresolved Votes: 0
Labels: spec

Approval: Vetted

 Description   

It should be possible to do `(s/def ::a ::b)` before declaring ::b.

Currently this throws an "Unable to resolve" error.

Alex indicated that everything should have delays but that they are missing in some cases. This seems like one of those cases.

Examples where things work fine are `(s/def ::a (s/and ::b))` and `(s/def ::a (s/keys :req [::b]))`.






[CLJ-2065] reduce-kv fails on subvec Created: 20/Nov/16  Updated: 19/Sep/17

Status: Open
Project: Clojure
Component/s: None
Affects Version/s: Release 1.9
Fix Version/s: None

Type: Defect Priority: Minor
Reporter: Steve Miner Assignee: Unassigned
Resolution: Unresolved Votes: 0
Labels: collections

Attachments: Text File CLJ-2065-reduce-kv-for-SubVector.patch     Text File CLJ-2065-Support-IKVReduce-on-SubVector-2.patch    
Patch: Code and Test
Approval: Triaged

 Description   

reduce-kv works as expected on vectors with the element index passed as the "key" argument. However, it fails with a subvec because clojure.lang.APersistentVector$SubVector does not implement IKVReduce.

(reduce-kv + 0 [1 2 3])
9

(reduce-kv + 0 (subvec [1 2 3] 1))
IllegalArgumentException No implementation of method: :kv-reduce of protocol: #'clojure.core.protocols/IKVReduce found for class: clojure.lang.APersistentVector$SubVector clojure.core/-cache-protocol-fn (core_deftype.clj:583)

One work around is to copy the subvec into a vector:

(reduce-kv + 0 (into [] (subvec [1 2 3] 1)))
6

Note however, the `vec` would not work here. Since Clojure 1.7, vec will return a subvec rather than copying.

(reduce-kv + 0 (vec (subvec [1 2 3] 1)))
IllegalArgumentException No implementation of method: :kv-reduce of protocol: #'clojure.core.protocols/IKVReduce found for class: clojure.lang.APersistentVector$SubVector clojure.core/-cache-protocol-fn (core_deftype.clj:583)

The Clojure user expects that a subvector supports all of the normal vector operations and this exception is confusing. The cause of the problem is that subvec returns a clojure.lang.APersistentVector$SubVector which does not implement clojure.lang.IKVReduce. PersistentVector inherits from APersistentVector and implements IKVReduce but SubVector doesn't get any IKVReduce support. This was probably just an oversight.

There are two patches attached. The first fixed the problem by extending the IKVReduce protocol in core.clj. The second, as suggested by Alex Miller, just adds the Java implementation of the IKVReduce interface directly to APersistentVector$SubVector. The same test is included in both patches.



 Comments   
Comment by Steve Miner [ 20/Nov/16 12:53 PM ]

Here is my current work-around:

(extend-type clojure.lang.APersistentVector$SubVector
  clojure.core.protocols/IKVReduce
  (kv-reduce [subv f init]
    (transduce (map-indexed vector)
               (fn ([ret] ret) ([ret [k v]] (f ret k v)))
               init
               subv)))

In my tests it was usually faster to copy the subvec into a regular vector but I like the look of the transduce fix. It would probably be faster to add a native Java implementation in APersistentVector.java. I'm willing to do the work if the Clojure/core team wants a patch.

Comment by Steve Miner [ 09/Mar/17 1:38 PM ]

Revised work-around using more interop for better performance. Comparable to the speed of normal vector reduce-kv.

(when-not (satisfies?   clojure.core.protocols/IKVReduce (subvec [1] 0))
  (extend-type clojure.lang.APersistentVector$SubVector
    clojure.core.protocols/IKVReduce
    (kv-reduce
      [subv f init]
      (let [cnt (.count subv)]
        (loop [k 0 ret init]
          (if (< k cnt)
            (let [val (.nth subv k)
                  ret (f ret k val)]
              (if (reduced? ret)
                @ret
                (recur (inc k) ret)))
            ret))))))
Comment by Steve Miner [ 18/Sep/17 10:35 AM ]

support IKVReduce for SubVector

Comment by Steve Miner [ 18/Sep/17 10:36 AM ]

Patch also adds test for reduce-kv on regular vector and subvector.

Comment by Alex Miller [ 18/Sep/17 10:48 AM ]

Why not implement IKVReduce directly in APersistentVector$SubVector?

Comment by Steve Miner [ 18/Sep/17 11:33 AM ]

Java implementation of IKVReduce on SubVector

Comment by Steve Miner [ 18/Sep/17 11:37 AM ]

Yes, that makes sense. I hesitated because I didn't want to rework the hierarchy to make SubVector a subclass of PersistentVector but that wasn't necessary to fix the bug. CLJ-2065-Support-IKVReduce-on-SubVector.patch is just the Java side, plus the same test.

Comment by Steve Miner [ 19/Sep/17 7:58 AM ]

Updated patch to inline nth() call, avoiding unnecessary bounds checking.





[CLJ-2050] Remove redundant key comparisons in HashCollisionNode Created: 03/Nov/16  Updated: 03/Nov/16

Status: Open
Project: Clojure
Component/s: None
Affects Version/s: None
Fix Version/s: None

Type: Enhancement Priority: Minor
Reporter: Kwang Yul Seo Assignee: Unassigned
Resolution: Unresolved Votes: 0
Labels: collections, performance

Attachments: Text File 0001-Remove-redundant-key-comparisons-in-HashCollisionNod.patch    
Patch: Code
Approval: Prescreened

 Description   

Comparing key to array[idx] is redundant as findIndex already performed key comparison to find the index.

Prescreened by: Alex Miller






[CLJ-2049] Improve clojure.zip documentation Created: 25/Oct/16  Updated: 27/Oct/16

Status: Open
Project: Clojure
Component/s: None
Affects Version/s: Release 1.8, Release 1.9
Fix Version/s: None

Type: Enhancement Priority: Minor
Reporter: Brighid M Assignee: Unassigned
Resolution: Unresolved Votes: 0
Labels: docstring, zip
Environment:

All


Attachments: File improve-zip-docs.diff    
Patch: Code
Approval: Triaged

 Description   

The clojure.zip module has extremely terse docstrings that are helpful as reference material but completely unhelpful to someone approaching the module cold. Expanded docstrings would make this piece of code much more visible to users who might find it helpful.



 Comments   
Comment by Brighid M [ 25/Oct/16 9:03 PM ]

A patch that improves clojure.zip's docstrings.

Comment by Alex Miller [ 27/Oct/16 8:42 AM ]

I would prefer if we could minimize the size of the changes by removing diffs that just add periods, change line breaks, add whitespace, or make other non-essential changes. That will help focus on the things that matter like changes in the ns docstring, zipper, etc. Additionally this ticket talks about doc strings but also makes changes in exception messages - I'd prefer code changes to be in a separate ticket. Also, please don't remove the comment sections in the files.

If you would like to convert the test comment section into actual tests, that would be a great separate ticket (I feel like I might have even written a patch to do that at one point but I don't see a ticket for it!).

Keeping this patch entirely focused on essential docstring changes is the best way to ensure its timely inclusion.

Comment by Brighid M [ 27/Oct/16 4:32 PM ]

Alex — to clarify, I'm hearing "this ticket should include two patches: one with the major prose additions and one with small proofreading-ish changes" and "this ticket's patches should not include the changes to exception messages nor the comment-move"?

Supplemental question: is there a style guide hanging around for the code & documentation style of the Clojure core? I poked around for such a guide on clojure.org and in JIRA: I didn't find one, but maybe I overlooked it. I was looking for a guideline about the comment section. It would be good to have a hint about "please don't touch these," because AFAICT they don't communicate that by themselves.

Comment by Alex Miller [ 27/Oct/16 5:43 PM ]

Actually, I'd prefer to have just one patch with the major prose additions. I don't think the minor changes are worth doing and will be a distraction from the more important prose.

Unfortunately, there is no style guide for Clojure core, and in fact it's been written by many people over many years so there generally isn't a consistent style throughout the code. Generally Rich prefers that debug code or comments that trace to him are left intact (git blame can help there). More generally, the simpler a patch is to review, the easier it is for it to stay good and be easily reviewed.

Thanks!

Comment by Brighid M [ 27/Oct/16 6:34 PM ]

> Actually, I'd prefer to have just one patch with the major prose additions.
Okay, I'll produce that patch and attach it to this ticket.

> I don't think the minor changes are worth doing and will be a distraction from the more important prose. Unfortunately, there is no style guide for Clojure core, and in fact it's been written by many people over many years so there generally isn't a consistent style throughout the code.
I think you just made an argument for why it is worth doing minor changes: consistency generally makes things more accessible. However, now is clearly not the time to litigate that, so I'm gonna come back with just the ns/docstring version of this patch.





[CLJ-2041] clojure.spec/keys requires input collections conform to clojure.core/map? Created: 11/Oct/16  Updated: 08/Mar/17

Status: Open
Project: Clojure
Component/s: None
Affects Version/s: Release 1.9
Fix Version/s: Release 1.9

Type: Defect Priority: Minor
Reporter: Timothy Baldridge Assignee: Unassigned
Resolution: Unresolved Votes: 9
Labels: spec

Approval: Vetted

 Description   

I would like to use specs to validate Datomic entities. However, `s/keys` is too restrictive in that it requires input collections to conform to `clojure.core/map?` instead of some more primitive interface (for example clojure.lang.ILookup or clojure.lang.Associative).



 Comments   
Comment by Alex Miller [ 04/Nov/16 8:21 AM ]

s/keys uses IPersistentMap's Iterable support for iterating through all entries for validation. ILookup and Associative do not support iteration. So, that's why it is the way it is. But, understand the desire.

Comment by Alex Miller [ 04/Nov/16 8:34 AM ]

Datomic entities are seqable so maybe that's a potential path (would be slower for actual PHMs though).

Comment by Odin Standal [ 04/Nov/16 8:35 AM ]

Thanks for following up. So any ideas or guidance on how to use clojure.spec with Datomic entities?

Comment by Alex Miller [ 04/Nov/16 8:37 AM ]

For now, you could use into to pour an entity into a PHM before validating. I hesitate to suggest it, but that could even be in the spec with a leading conformer.

Comment by Alex Miller [ 14/Nov/16 12:16 PM ]

Moving this into 1.9 for the moment just so we don't lose it. Not sure whether we can or will actually do anything with this though.

Comment by Alex Miller [ 08/Mar/17 10:00 AM ]

This is related to CLJ-2080 as it's the same basic issue of what map-of and every-kv actually expect as valid inputs, which is: something that seqs to map entries. We can't really write a predicate for it without an interface "ISeqsToMapEntries" (intentionally bad name) to indicate this. java.util.Map, IPersistentMap, etc imply this, but an object can seq to map entries without meeting all the constraints of those much broader interfaces. ILookup does not and should not imply this.





[CLJ-2038] Clojure.spec/exercise-fn should accept custom generator map Created: 08/Oct/16  Updated: 07/Nov/16

Status: Open
Project: Clojure
Component/s: None
Affects Version/s: Release 1.9
Fix Version/s: None

Type: Enhancement Priority: Minor
Reporter: Andrea Richiardi Assignee: Unassigned
Resolution: Unresolved Votes: 1
Labels: spec

Attachments: File CLJ-2038-exercise-fn-should-accept-custom-generator.diff    
Approval: Triaged

 Description   

I tried to generate some data with exercise-fn but could not do it because I need to carry my own custom generators.
At the moment though, there is no way to add them, like an optional parameter.



 Comments   
Comment by Alex Miller [ 11/Oct/16 12:09 PM ]

Patch would be welcome for this!

Comment by Laszlo Török [ 07/Nov/16 9:23 AM ]

First attempt to form a patch.

I am unsure whether it is ok to use the overloaded 3rd argument approach or take a keyword arg approach for the optional arguments.

i.e. ([sym-or-fn n & {:keys [fspec gen]}])

On the other hand, it only make sense to pass one or the other and given sym-or-fn, fspec-or-gen seems appropriate.

Currently the test suite for c.spec is lacking, I'm happy to add a few example-based tests for exercise-fn, once the approach is approved.
Also





[CLJ-2025] When a generator fails to gen, state which spec/pred failed Created: 21/Sep/16  Updated: 29/Jan/17

Status: Open
Project: Clojure
Component/s: None
Affects Version/s: Release 1.9
Fix Version/s: None

Type: Enhancement Priority: Minor
Reporter: David Collie Assignee: Unassigned
Resolution: Unresolved Votes: 1
Labels: spec

Attachments: Text File better-such-that-info.patch    
Approval: Triaged

 Description   

Given a generator using such-that that fails to find a value, the error does not give enough information to determine which spec or predicate was at fault:

(require '[clojure.spec :as s])
(s/exercise (s/and string? #{"hi"}))
ExceptionInfo Couldn't satisfy such-that predicate after 100 tries.  clojure.core/ex-info (core.clj:4725)

Another special case of this is when providing a custom generator that produces a valid that doesn't satisfy the spec (Clojure adds this filter internally):

(require '[clojure.spec :as s])
(s/exercise (s/with-gen int? #(s/gen #{:a})))

Proposal: Indicate in the error which spec failed to generate and possibly the path in the overall spec if feasible.

(Note: original description moved to comment)



 Comments   
Comment by Alex Miller [ 22/Sep/16 3:54 PM ]

[Original description from ticket:]

I created a generator that did not conform to the spec (doh!). The generator contained the such-that predicate. When I tried creating a sample from the generator I got this error:

ExceptionInfo Couldn't satisfy such-that predicate after 100 tries.  clojure.core/ex-info (core.clj:4725)

I assumed that it referred to my custom generator but that was a red herring because in fact spec must be using such-that to ensure that the generated value conforms to the spec, and it was this such-that that generated the failure, not the one in my custom generator.

Code (with the problem corrected but showing the such-that in my generator:

(defn mod11-checkdigit
  "Calculate the checkdigit see http://freagra.com/imthealth/mitNNC.html"
  [n]
  (let [x (->> (map #(Integer/parseInt (str %)) (take 9 n))
               (map * (range 10 1 -1))
               (reduce +))
        y (mod x 11)
        c (- 11 y)]
    (cond (== 10 c) nil
          (== 11 c) 0
          :else c)))

(def nhs-number-gen
  "Generates a valid NHS number"
  (gen/fmap #(str (+ (* 10 %) (mod11-checkdigit (str %))))
            (gen/such-that #(mod11-checkdigit (str %))
                           (gen/choose 100000000 999999999))))

(defn nhs-number?
  "Returns true if passed a valid nhs number else returns false"
  [n]
  (and (string? n) (= 10 (count n)) (= (str (mod11-checkdigit n)) (str (last n)))))

(s/def ::nhs-number (s/with-gen nhs-number?
                                (fn [] nhs-number-gen)))

It would be nicer if the error thrown due to the generated value being non-conformant with the spec stated this.

Comment by Alex Miller [ 22/Sep/16 4:07 PM ]

I'm not sure that this is possible right now based on what we give to and get back from test.check.

Comment by Griffin Smith [ 29/Jan/17 6:37 PM ]

It looks like test.check is being updated to support error customization: https://github.com/clojure/test.check/commit/5aea0e275257680b672309b1e940be6dae92c17d . I've got a patch which updates clojure.spec to use it, though obviously it doesn't work since the linked commit hasn't made it into a released version of test.check yet.

Comment by Griffin Smith [ 29/Jan/17 6:40 PM ]

See also http://dev.clojure.org/jira/browse/TCHECK-107 I suppose

Comment by Griffin Smith [ 29/Jan/17 6:51 PM ]

Attached the patch for future reference (`better-such-that-info.patch`).





[CLJ-2022] Add fixed arities for mapcat Created: 13/Sep/16  Updated: 13/Sep/16

Status: Open
Project: Clojure
Component/s: None
Affects Version/s: None
Fix Version/s: None

Type: Enhancement Priority: Minor
Reporter: Robert C Faber Assignee: Unassigned
Resolution: Unresolved Votes: 0
Labels: performance

Attachments: Text File CLJ-2022.patch    
Patch: Code

 Description   

Add fixed arities for mapcat, in the pattern of map.

Same change as CLJS-1776.



 Comments   
Comment by Alex Miller [ 13/Sep/16 10:29 AM ]

Presumably this is to improve performance. Please include a benchmark showing the difference.

Comment by Ghadi Shayban [ 13/Sep/16 12:47 PM ]

Please consider interactions with apply and laziness CLJ-1218





[CLJ-2005] Type hint fails with direct linking disabled Created: 19/Aug/16  Updated: 20/Aug/16

Status: Open
Project: Clojure
Component/s: None
Affects Version/s: Release 1.9
Fix Version/s: None

Type: Defect Priority: Minor
Reporter: Viktor Magyari Assignee: Unassigned
Resolution: Unresolved Votes: 0
Labels: compiler, directlinking, typehints

Attachments: Text File 0001-CLJ-2005-assoc-arglist-ret-tag-as-tag-in-constructed.patch    
Patch: Code
Approval: Triaged

 Description   

Minimal example, using 1.9.0-alpha11:

user=> (set! *warn-on-reflection* true)
true
user=> (defn foo ^String [^long x] "")
#'user/foo
user=> (.length (foo 10))
Reflection warning, (...) - reference to field length on java.lang.Object can't be resolved.
0

The warning is present only if direct linking is disabled.

Explanation:
this is another manifestation of CLJ-1533 – because of the lexical transformation the compiler is doing when routing the invoke through invokePrim, the arglists type hints are lost. This doesn't happen when DL is on because invokeStatic isn't compiled via a lexical transformation but through StaticInvokeExpr which properly tracks the original var's type hints

Patch: 0001-CLJ-2005-assoc-arglist-ret-tag-as-tag-in-constructed.patch



 Comments   
Comment by Nicola Mometto [ 19/Aug/16 5:24 PM ]

With DL on:

public static java.lang.Object invokeStatic();
    descriptor: ()Ljava/lang/Object;
    flags: ACC_PUBLIC, ACC_STATIC
    Code:
      stack=2, locals=0, args_size=0
         0: ldc2_w        #12                 // long 10l
         3: invokestatic  #18                 // Method test$foo.invokeStatic:(J)Ljava/lang/Object;
         6: checkcast     #20                 // class java/lang/String
         9: invokevirtual #24                 // Method java/lang/String.length:()I
        12: invokestatic  #30                 // Method java/lang/Integer.valueOf:(I)Ljava/lang/Integer;
        15: areturn
      LineNumberTable:
        line 5: 0
        line 5: 9

with DL off:

public static java.lang.Object invokeStatic();
    descriptor: ()Ljava/lang/Object;
    flags: ACC_PUBLIC, ACC_STATIC
    Code:
      stack=3, locals=0, args_size=0
         0: getstatic     #15                 // Field const__0:Lclojure/lang/Var;
         3: invokevirtual #20                 // Method clojure/lang/Var.getRawRoot:()Ljava/lang/Object;
         6: checkcast     #22                 // class clojure/lang/IFn$LO
         9: ldc2_w        #23                 // long 10l
        12: invokeinterface #28,  3           // InterfaceMethod clojure/lang/IFn$LO.invokePrim:(J)Ljava/lang/Object;
        17: ldc           #30                 // String length
        19: iconst_0
        20: invokestatic  #36                 // Method clojure/lang/Reflector.invokeNoArgInstanceMember:(Ljava/lang/Object;Ljava/lang/String;Z)Ljava/lang/Object;
        23: areturn
      LineNumberTable:
        line 5: 0
        line 5: 12
        line 5: 17
Comment by Nicola Mometto [ 19/Aug/16 5:43 PM ]

bytecode with DL off and current patch:

public static java.lang.Object invokeStatic();
    descriptor: ()Ljava/lang/Object;
    flags: ACC_PUBLIC, ACC_STATIC
    Code:
      stack=3, locals=0, args_size=0
         0: getstatic     #15                 // Field const__0:Lclojure/lang/Var;
         3: invokevirtual #20                 // Method clojure/lang/Var.getRawRoot:()Ljava/lang/Object;
         6: checkcast     #22                 // class clojure/lang/IFn$LO
         9: ldc2_w        #23                 // long 10l
        12: invokeinterface #28,  3           // InterfaceMethod clojure/lang/IFn$LO.invokePrim:(J)Ljava/lang/Object;
        17: checkcast     #30                 // class java/lang/String
        20: invokevirtual #34                 // Method java/lang/String.length:()I
        23: invokestatic  #40                 // Method java/lang/Integer.valueOf:(I)Ljava/lang/Integer;
        26: areturn
      LineNumberTable:
        line 5: 0
        line 5: 12
        line 5: 20




[CLJ-1995] Improved docstring for explain-data Created: 30/Jul/16  Updated: 30/Jul/16

Status: Open
Project: Clojure
Component/s: None
Affects Version/s: Release 1.9
Fix Version/s: None

Type: Enhancement Priority: Minor
Reporter: Marshall Abrams Assignee: Unassigned
Resolution: Unresolved Votes: 0
Labels: docstring, spec

Approval: Triaged

 Description   

In 1.9.0-alpha10, the docstring for explain-data doesn't mention or describe the meaning of some standard keys/values of its return value, and the use of "path" in the docstring could be clarified to avoid conflation with file paths or namespace paths. Here is the current docstring:

Given a spec and a value x which ought to conform, returns nil if x conforms, else a map with at least the key ::problems whose value is a collection of problem-maps, where problem-map has at least :path :pred and :val keys describing the predicate and the value that failed at that path.

Here is a possible replacement:

Given a spec and a value x which ought to conform, returns nil if x conforms, else a map with at least the key ::problems whose value is a collection of problem-maps, where problem-map has at least :path :pred and :val keys describing the predicate and the value that failed at that path (through possibly embedded specs). The map may also contain a :via key for specs that failed, an :in key for data key(s) of the value that failed, and a :reason key for a string describing the reason for failure.

This differs from the existing docstring in two ways:

1. It inserts "(through possibly embedded specs)" at the end of the existing dostring to clarify and disambiguate the meaning of "path" here.

2. It adds an additional sentence describing the :via, :in, and :reason keys.






[CLJ-1990] Add an async macro that behaves the same as ClojureScript's Created: 24/Jul/16  Updated: 15/May/17

Status: Open
Project: Clojure
Component/s: None
Affects Version/s: None
Fix Version/s: None

Type: Feature Priority: Minor
Reporter: Daniel Compton Assignee: Unassigned
Resolution: Unresolved Votes: 2
Labels: clojure.test, portability

Attachments: Text File clj-1990v1.patch    
Approval: Triaged

 Description   

We want to run the same tests between Clojure and ClojureScript. However some of our CLJS tests are asynchronous and require the use of the async macro. Clojure doesn't have the async macro, which makes test portability difficult. If we were to add the async macro, there are at least two possible routes to go down: imitating the async behaviour of cljs.test, or copying the implementation. If we were to imitate the behaviour, we could use the following async macro. It creates a promise, runs the body, blocks on the promise, and done delivers the promise, allowing the test to continue.

(defmacro async
  [done & body]
  `(let [p# (promise)
         ~done #(deliver p# nil)]
     ~@body
     (deref p#)))

This has the advantage that it integrates with the current way tests are run in Clojure, and doesn't require any of the surrounding tooling to be aware of the async testing.

Imitating the implementation would be much more invasive. It would probably mean changing the behaviour of run-tests to return nil like ClojureScript does, and a host of other changes. This means it is likely a non-starter.

I lean heavily towards the first option.

To answer the question "Why do we need this at all when it is a no-op?", this is useful so we can write the same tests in both Clojure and ClojureScript. It would be much more tricky to write without it.



 Comments   
Comment by Daniel Compton [ 26/Jul/16 12:12 AM ]

v1, patch and tests





[CLJ-1989] `let` ported from `test.check/let` to `clojure.spec.gen` Created: 24/Jul/16  Updated: 06/Jul/17

Status: Open
Project: Clojure
Component/s: None
Affects Version/s: Release 1.9
Fix Version/s: None

Type: Feature Priority: Minor
Reporter: Matthew Wampler-Doty Assignee: Unassigned
Resolution: Unresolved Votes: 1
Labels: spec

Attachments: Text File gen-let.patch    

 Description   

When using `clojure.spec` for elaborate specifications and `clojure.spec.gen` for generative testing, developers often find themselves writing code which heavily relies on `clojure.spec.gen/fmap`. This is sometimes unnatural and difficult to read.

To make writing custom generators easier, this patch ports `test.check/let` to `clojure.spec.gen`. Now developers can write generators more simply.



 Comments   
Comment by Matthew Wampler-Doty [ 24/Jul/16 5:55 PM ]

For example, if a user wanted to make a generator of vectors with length between 5 and 11 or 20 to 40 elements, consisting of keywords which were either `:a` or `:b`, they would have to write something like:

(gen/fmap (fn [[n gens]] (take n gens)) 
          (gen/tuple (spec/gen (spec/or :short (int-in 5 11) 
                                        :long (int-in 20 40)))
                     (gen/vector (gen/elements #{:a :b}) 40)))

With this patch they could write this as:

(gen/let [length (spec/or :short (int-in 5 11)
                          :long  (int-in 20 40))]
  (repeat length #{:a :b}))
Comment by Rick Moynihan [ 06/Jul/17 6:57 AM ]

Ran into the lack of this useful macro recently. And spoke to @gfredricks about it on slack. He mentioned that the supplied patch looks to be a complete reimplementation of gen/let with different behaviour.

He suggested an alternative might be to have the gen/let macro make a require call, which when succeeds just expands to use the test.check let implementation. If the require fails the macro can expand into code that raises an appropriate error. Presumably indicating that test.check needs to be included as a dependency.

e.g.

 (defmacro let [& args] (try (require ...) `(...) (catch Exception e `(throw ...)))) 




[CLJ-1984] clojure.spec/double-in should allow strict greater-than, less-than tests Created: 20/Jul/16  Updated: 21/Jul/16

Status: Open
Project: Clojure
Component/s: None
Affects Version/s: Release 1.9
Fix Version/s: None

Type: Enhancement Priority: Minor
Reporter: Marshall Abrams Assignee: Unassigned
Resolution: Unresolved Votes: 0
Labels: spec


 Description   

clojure.spec/double-in defines a spec that tests whether a double is greater than or equal to a minimum value and less than or equal to a maximum value. This seems like an arbitrary choice from the point of view of mathematics and practical concerns. Sometimes you need to test whether a double is greater than a minimum or less than a maximum. Example: The application will divide by the tested double later.

Of course we can add tests to double-in, e.g. like

(s/and (s/double-in :min 0.0 :max 1.0) #(not= 0.0 %))}}

but

#(and (> % 0.0) (<= % 1))

might be clearer if double-in's NaN and Infinity tests aren't needed.

Why not have a common interface to all four interval tests? Rather than four different spec functions, which is one option, I suppose, I suggest adding two keywords to double-in. When true, these would change the >= or <= tests to > or < tests:

:min-greater

(or? :min+, :min-greater-than, :greater-than-min, :strict-min, :min-open, or possibly :infinmum, :inf, but that could be misleading)

:max-less

(or :max- :max-less-than, :less-than-max, :strict-max, :max-open, or possibly :supremum, :sup etc.)

For example,

(s/valid? (s/double-in :min 0.0 :max 0.1 :min-greater true) 0.0)

would return false, but

(s/valid? (s/double-in :min 0.0 :max 0.1 :min-greater false) 0.0)

would return true.

Default values for these keywords should probably be false, for compatibility with the current definition of double-in.






[CLJ-1982] Better explain reporting on a failed zero or one match with an embedded spec. Created: 18/Jul/16  Updated: 26/Aug/16

Status: Open
Project: Clojure
Component/s: None
Affects Version/s: Release 1.9
Fix Version/s: None

Type: Enhancement Priority: Minor
Reporter: Nick Jones Assignee: Unassigned
Resolution: Unresolved Votes: 0
Labels: spec
Environment:

OSX, Java 8, Clojure 1.9.0-alpha10


Approval: Triaged

 Description   

Problem:

When attempting to validate a vector containing an optional map, the spec will validate correctly if the vector contains a valid map. If however the optional map does not satisfy the spec misleading error messages are produced. It would be nice if on a partial match of an optional map that some indication of this would be given to the user.

Example REPL session to illustrate problem:

The optional nested map (:optional-nested-map) below fails validation because :nested-element-b is a string instead of an int however the explain report says the spec fails at the parent predicate: :user/vector-schema at: [:element-value] predicate: string?.

It would be more helpful for the user in this case if spec reported that the optional nested map at :optional-nested-map had failed due to ::nested-element-b failing the int? predicate.

user=> (require '[clojure.spec :as s])
nil
user=> (s/def ::nested-element-a string?)
:user/nested-element-a
user=> (s/def ::nested-element-b int?)
:user/nested-element-b
user=> (s/def ::nested-element-schema
          (s/keys :opt-un [::nested-element-a ::nested-element-b]))
:user/nested-element-schema
user=> (s/def ::vector-schema
         (s/cat :tag-kw               #{:tag}
                :optional-nested-map  (s/? (s/spec ::nested-element-schema))
                :element-value        string?))
:user/vector-schema
user=> (s/valid? ::vector-schema [:tag {:nested-element-a "bla" :nested-element-b 10} "Element"])
true
user=> (s/valid? ::vector-schema [:tag {:nested-element-a "bla" :nested-element-b ""} "Element"])
false
user=> (s/explain ::vector-schema [:tag {:nested-element-a "bla" :nested-element-b ""} "Element"])
In: [1] val: {:nested-element-a "bla", :nested-element-b ""} fails spec: :user/vector-schema at: [:element-value] predicate: string?
nil
user=>


 Comments   
Comment by Alex Miller [ 18/Jul/16 7:43 AM ]

Can you update this description with a self-contained example that demonstrates the problem? It's too hard to repro and understand this larger example.

Comment by Nick Jones [ 19/Jul/16 3:30 AM ]

Hi,

Sorry I don't seem to have access to edit the description of the ticket after creation. Here is a simplified sample that I hope will help illustrate the case better.

When the optional nested map below fails validation because :nested-element-b is a string instead of an int the explain report says the spec fails at the parent predicate: :user/vector-schema at: [:element-value] predicate: string?.

As it is an optional map I could see how this would be the case. When no match is found it moves onto the next predicate in the parent.

That said I think it could be helpful (especially in a large optional nested data structure) that if a partial match is achieved that that could be reported to the user as a potential spot for the spec failing.

user=> (require '[clojure.spec :as s])
nil
user=> (s/def ::nested-element-a string?)
:user/nested-element-a
user=> (s/def ::nested-element-b int?)
:user/nested-element-b
user=> (s/def ::nested-element-schema
          (s/keys :opt-un [::nested-element-a ::nested-element-b]))
:user/nested-element-schema
user=> (s/def ::vector-schema
         (s/cat :tag-kw               #{:tag}
                :optional-nested-map  (s/? (s/spec ::nested-element-schema))
                :element-value        string?))
:user/vector-schema
user=> (s/valid? ::vector-schema [:tag {:nested-element-a "bla" :nested-element-b 10} "Element"])
true
user=> (s/valid? ::vector-schema [:tag {:nested-element-a "bla" :nested-element-b ""} "Element"])
false
user=> (s/explain ::vector-schema [:tag {:nested-element-a "bla" :nested-element-b ""} "Element"])
In: [1] val: {:nested-element-a "bla", :nested-element-b ""} fails spec: :user/vector-schema at: [:element-value] predicate: string?
nil
user=>
Comment by Nick Jones [ 19/Jul/16 3:45 AM ]

Added simplified version of project archive matching comment at 2016-07-19.

Comment by Alex Miller [ 19/Jul/16 8:27 AM ]

Nick, I've given you edit rights here. Generally, we don't like to have external projects for repro - if you can boil it down to a few line example in the description, that would be ideal.

Comment by Nick Jones [ 19/Jul/16 8:15 PM ]

Thanks Alex. I've updated the description and removed the project attachments. I've also added a REPL session to the description to reproduce the problem in a standalone Clojure 1.9.0-alpha10 REPL.





[CLJ-1973] generate-proxy produces unpredictable method order in generated classes Created: 01/Jul/16  Updated: 04/Jul/16

Status: Open
Project: Clojure
Component/s: None
Affects Version/s: Release 1.8, Release 1.9
Fix Version/s: None

Type: Enhancement Priority: Minor
Reporter: James Carnegie Assignee: Unassigned
Resolution: Unresolved Votes: 0
Labels: aot, compiler
Environment:

OSX, OpenJDK 8


Attachments: Text File CLJ-1973-v1.patch     Text File CLJ-1973-v2.patch     Text File CLJ-1973-v3.patch     Text File CLJ-1973-v4.patch     Text File CLJ-1973-v5.patch    
Patch: Code
Approval: Prescreened

 Description   

Using core/proxy to generate proxies for Java classes produces unpredictable method ordering in the generated class files.
This is a problem for repeatable builds (when doing AOT).

Specifically, I'm running Clojure inside Docker, and I'd like my application image layer to be as small as those produced by Java developers (using Meta-inf classpaths and a lib directory). Anyway, to get this working properly so that all dependencies (including those compiled as part of AOT) are on a separate layer, I need the output of compiling my applications' dependencies' proxies to be the same each time I run the build. This reduces build time, image push time, image pull time and container scheduling time.

Example code that exhibits the problem (you'll need to run it a few times to see the issue).

https://github.com/kipz/predictable-proxies

Cause: I've tracked it down to the use of an unsorted map here:

https://github.com/clojure/clojure/blob/master/src/clj/clojure/core_proxy.clj#L186

Approach: Use a sorted map, sorted by hash of the key (which is a vector of method name, seq of param types, and return type).

Patch: CLJ-1973-v5.patch



 Comments   
Comment by James Carnegie [ 01/Jul/16 4:19 PM ]

Patch that uses a sorted-map

Comment by Alex Miller [ 04/Jul/16 9:44 AM ]

I think you can follow the advice at http://clojure.org/guides/comparators to write a simpler comparator for this patch.

Comment by James Carnegie [ 04/Jul/16 11:24 AM ]

Simpler comparator as requested.

Comment by Alex Miller [ 04/Jul/16 12:28 PM ]

I think you lost the sorted set.

Comment by James Carnegie [ 04/Jul/16 1:06 PM ]

Copy paste between branches error. Tested this time.

Comment by James Carnegie [ 04/Jul/16 1:14 PM ]

Now with more consistent formatting of 'let'

Comment by Alex Miller [ 04/Jul/16 2:27 PM ]

While this is probably fine, it might be better to use the hash (Clojure) function rather than .hashCode (Java) function. The map itself is hashed based on the hash so that seems more appropriate.

Comment by James Carnegie [ 04/Jul/16 3:15 PM ]

As requested, using Clojure 'hash' instead.

Thanks Alex - learned about boolean comparators too!

Comment by Alex Miller [ 04/Jul/16 3:52 PM ]

Note that this ordering may still change across Clojure or JVM versions as there is no guarantee of hashing across those. Pre-screening for now.





[CLJ-1965] clojure.spec/def should support an optional doc-string Created: 19/Jun/16  Updated: 18/May/17

Status: Open
Project: Clojure
Component/s: None
Affects Version/s: Release 1.9
Fix Version/s: Release 1.9

Type: Feature Priority: Minor
Reporter: Alexander Kiel Assignee: Unassigned
Resolution: Unresolved Votes: 46
Labels: spec

Approval: Vetted

 Description   

Like clojure.core/def clojure.spec/def should support an optional doc string because one usually likes to describe specs in more detail as one could through keyword naming.



 Comments   
Comment by Moritz Heidkamp [ 03/Nov/16 4:23 PM ]

Building on this idea, I suggest to add first-class metadata support to registered specs and implement doc strings in terms of that (i.e. the same way as with vars).

Comment by Josh Brandoff [ 01/Apr/17 5:02 PM ]

Hi! Was just discussing the potential for a feature like this with a colleague. What's the current status? Was thinking of potentially working on it but wanted to get feedback and guidance from the community first.

Comment by Alex Miller [ 03/Apr/17 9:32 AM ]

We don't have a recommended approach to this yet so not looking for a patch at this time.





[CLJ-1960] Bug in clojure.core/mod with large Double argument Created: 14/Jun/16  Updated: 15/Jun/16

Status: Open
Project: Clojure
Component/s: None
Affects Version/s: Release 1.8
Fix Version/s: None

Type: Defect Priority: Minor
Reporter: William Tozier Assignee: Unassigned
Resolution: Unresolved Votes: 0
Labels: math, numerics
Environment:

Java 8 update 91 on Mac OS X 10.11.5


Approval: Triaged

 Description   

The `clojure.core/mod` function works just as expected for small positive floating-point dividend and small positive integer divisor. But today I was working on some edge case tests and came across the following inexplicable behavior:

REPL_session
user=> (def big  Double/MAX_VALUE)
#'user/big
user=> (mod big 10)
0.0
user=> (mod big 100)
0.0
user=> (mod big 1000)
1.9958403095347198E292
user=> (mod big 999)
-Infinity
user=> (mod big 998)
0.0
user=> (mod big 997)
1.9958403095347198E292
user=> (mod big 996)
0.0
user=> (mod big 995)
0.0
user=> (mod big 994)
0.0
user=> (mod big 1001)
1.9958403095347198E292
user=> (mod big 1002)
0.0
user=> (mod big 1003)
0.0
user=> (mod big 1004)
-Infinity
user=> (mod big 1005)
0.0

No idea whether this is inherited from a Java bug. I can see nothing special about the values chosen, and I suspect if one scanned it'd be easy to find other glitches.



 Comments   
Comment by Alex Miller [ 14/Jun/16 7:12 PM ]

mod is based on rem - from a glance, mod does not seem to account properly for any case of overflow, and I suspect that's at the root of a lot of these problems.

Comment by Gary Fredericks [ 14/Jun/16 7:15 PM ]

Test.check suggests (mod 6.7772677936779424E16 23) => -8.0 is somewhat close to minimal.

Comment by William Tozier [ 15/Jun/16 12:40 PM ]

Actually, just checked, and rem gives the same results. Thus (rem Double/MAX_VALUE 1001) is 1.9958403095347198E292, and (rem 6.7772677936779424E16 23) => -8.0.





[CLJ-1955] .hashCode throws ClassCastException when called on some functions Created: 09/Jun/16  Updated: 01/Dec/16

Status: Open
Project: Clojure
Component/s: None
Affects Version/s: None
Fix Version/s: None

Type: Defect Priority: Minor
Reporter: Georgi Danov Assignee: Unassigned
Resolution: Unresolved Votes: 2
Labels: None

Attachments: Text File clj-1955.patch    
Patch: Code
Approval: Triaged

 Description   
user> some?
#function[clojure.core/some?]
user> (.hashCode map)
72400056
user> (.hashCode str)
ClassCastException clojure.core$str cannot be cast to java.lang.String  /eval39172 (form-init3428514420830954023.clj:5793)
user> (.hashCode (fn []))
1715179801
user> (.hashCode some?)
ClassCastException clojure.core$some_QMARK_ cannot be cast to java.lang.Boolean  /eval39178 (form-init3428514420830954023.clj:5797)
user> (.hashCode #'some?)
1955712430
user> (.hashCode @#'some?)
1726569843


 Comments   
Comment by Nicola Mometto [ 10/Jun/16 3:27 AM ]

This happens because `some?` and `str` have type hints on the Var to signal the type returned by their invocations, but the Compiler thinks those type hints apply to the Var object itself aswell.

An easy fix would be to move those type hints from the Var (old-style) to the argvec (new-style)

Comment by Ghadi Shayban [ 14/Jun/16 3:36 PM ]

agreed with nicola's suggestion - change type hints. This is a dup of CLJ-140 where :tag causes confusion when a var is being invoked vs used in expr context

Comment by Jozef Wagner [ 01/Dec/16 1:29 PM ]

Patch attached





[CLJ-1952] include var->sym in clojure.core Created: 08/Jun/16  Updated: 15/May/17

Status: Open
Project: Clojure
Component/s: None
Affects Version/s: Release 1.9
Fix Version/s: None

Type: Feature Priority: Minor
Reporter: Nicola Mometto Assignee: Unassigned
Resolution: Unresolved Votes: 1
Labels: None

Attachments: Text File clj-1952.patch     Text File clj-1952-v2.patch    
Approval: Prescreened

 Description   

A lot of libraries define their own variant of `var->sym`, clojure.spec recently did so aswell as a private var called `->sym`.

This ticket proposes to move it from `clojure.spec` to `clojure.core` as a public var named `var->sym` and to refactor the two variants to use it.

Patch: clj-1952-v2.patch
Prescreened by: Alex Miller



 Comments   
Comment by Jozef Wagner [ 01/Dec/16 1:38 PM ]

Added patch

Comment by Alex Miller [ 01/Dec/16 7:58 PM ]

Patch has whitespace warnings that should be removed.

Comment by Jozef Wagner [ 02/Dec/16 2:10 AM ]

Attached updated patch clj-1952-v2.patch that has no trailing whitespaces.





[CLJ-1951] bigint? predicate and generator Created: 08/Jun/16  Updated: 15/May/17

Status: Open
Project: Clojure
Component/s: None
Affects Version/s: Release 1.9
Fix Version/s: Release 1.9

Type: Feature Priority: Minor
Reporter: Alex Miller Assignee: Unassigned
Resolution: Unresolved Votes: 2
Labels: generator

Approval: Vetted

 Description   

Add bigint? and spec.gen support.

This part is easy:

(defn bigint?
  "Returns true if n is a BigInt"
  {:added "1.9"}
  [n] (instance? clojure.lang.BigInt n))

The generator is the tricky bit. test.check doesn't have a generator for bigints, just large-integer for things in long range. I think we'd want numbers beyond long range in a bigint generator (as that's a likely place where bugs might lie). Making a really high-quality bigint generator (with good growth and shrinking characteristics) is something that needs more thought.

http://clojure.github.io/test.check/clojure.test.check.generators.html#var-large-integer



 Comments   
Comment by Gary Fredericks [ 17/Dec/16 6:17 PM ]

In case I don't get around to making a patch, I think a generator along these lines would be a decent start:

(def gen-bigint 
  (gen/sized 
   (fn [size] 
     (let [large-integer (gen/resize size gen/large-integer)] 
       ;; scaling gives us relatively small vectors, but using  
       ;; the resized large-integer above means the numbers in 
       ;; the small vectors will still be big 
       (gen/scale #(+ 2 (/ % 20))
                  (gen/fmap (fn [xs] (+ (bigint (first xs)) (reduce * (map bigint (rest xs))))) 
                            (gen/not-empty (gen/vector large-integer))))))))
Comment by Gary Fredericks [ 17/Dec/16 6:21 PM ]

If anything seems inadequate about the sizing in the above generator, I should point out that sizing in test.check is rather subtle but the requirements are also not well-defined. I'd be happy to discuss in detail.

Comment by Gary Fredericks [ 15/May/17 4:39 PM ]

As an update, I've put together a marvelous bigint generator that this margin is too small to contain. It might need tweaking but is probably fine for these purposes. It's not on test.check master yet but I expect it will be soon.





[CLJ-1950] cl-format is too slow for production use Created: 05/Jun/16  Updated: 05/Jun/16

Status: Open
Project: Clojure
Component/s: None
Affects Version/s: Release 1.7, Release 1.9
Fix Version/s: None

Type: Enhancement Priority: Minor
Reporter: Alain Picard Assignee: Unassigned
Resolution: Unresolved Votes: 1
Labels: performance, print
Environment:

Mac OS X - 3GHz i7 16Gb ram


Approval: Triaged

 Description   

Run this example code:

;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
(in-ns 'clojure.pprint)

(println "Basic output using raw str.")
(time
(doseq [i (range 1000)]
(apply str (interpose "," [1 2 3]))))

(println "Test 1 - raw cl-format")
(time
(doseq [i (range 1000)]
(clojure.pprint/cl-format nil "~{D^,~}" [1 2 3])))
;; ==> "Elapsed time: 231.345 msecs"

(println "Test 2 - call on the compiled format")
(def myx
(compile-format "~{D^,~}"))

(time
(doseq [i (range 1000)]
(clojure.pprint/cl-format nil myx [1 2 3])))

(println "Test 3 - using a formatter")
(def myy
(formatter "~{D^,~}"))

(time
(doseq [i (range 1000)]
(myy nil myx [1 2 3])))

(time
(dotimes (i 100000)
(format nil "~{D^,~}" '(1 2 3))))

;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;

It will print something like this:

Basic output using raw str.
"Elapsed time: 2.402 msecs"
Test 1 - raw cl-format
"Elapsed time: 194.405 msecs"
Test 2 - call on the compiled format
"Elapsed time: 87.271 msecs"
Test 3 - using a formatter
"Elapsed time: 199.318 msecs"

So raw `str' is ~ 100X faster.

For reference, on the same hardware, using
SBCL Common Lisp, this test runs in < 1 ms.

There are (at least) 2 problems here:

1. cl-format function begins with a line like:

let [compiled-format (if (string? format-in) (compile-format format-in) format-in)

But there is no api to pass in a compiled-format into it; (as compile-format
is a private function, so can't be used at large) so this is kind of useless.

2. Even using a precompiled formatter is way too slow.

Suggested fix: none, except perhaps warning unwary users that this
function is simply not suitable for tight loops, and should only be
used to pretty print user input strings, etc.

Thank you






[CLJ-1941] Instrumentation of fns with primitive type hints fails Created: 01/Jun/16  Updated: 14/Nov/16

Status: Open
Project: Clojure
Component/s: None
Affects Version/s: Release 1.9
Fix Version/s: Release 1.9

Type: Defect Priority: Minor
Reporter: Kenny Williams Assignee: Unassigned
Resolution: Unresolved Votes: 6
Labels: spec
Environment:

Ubuntu 15.10
Using boot 2.6.0 on openjdk version "1.8.0_91"


Approval: Vetted

 Description   
(require '[clojure.spec :as s] '[clojure.spec.test :as st])
(defn foo [^double val] val)
(s/fdef foo :args (s/cat :val double?))
(st/instrument `foo)
(foo 5.2)

user=> (foo 5.2)
ClassCastException clojure.spec.test$spec_checking_fn$fn__13069 cannot be cast to clojure.lang.IFn$DO
       	user/eval6 (NO_SOURCE_FILE:5)
       	user/eval6 (NO_SOURCE_FILE:5)
       	clojure.lang.Compiler.eval (Compiler.java:6951)
       	clojure.lang.Compiler.eval (Compiler.java:6914)
       	clojure.core/eval (core.clj:3187)
       	clojure.core/eval (core.clj:3183)
       	clojure.main/repl/read-eval-print--9704/fn--9707 (main.clj:241)
       	clojure.main/repl/read-eval-print--9704 (main.clj:241)
       	clojure.main/repl/fn--9713 (main.clj:259)
       	clojure.main/repl (main.clj:259)
       	clojure.main/repl-opt (main.clj:323)
       	clojure.main/main (main.clj:422)

Cause: spec replaces var values with instrumented functions that will not work with primitive function interfaces

Approach: Take primitive interfaces into account and make them work, or document/fail that instrumentation will not work with these.



 Comments   
Comment by Kevin Downey [ 02/Jun/16 1:41 AM ]

spec replaces var values with instrumented functions, which works for the default linking case, var deref cast to ifn, invoke, but in the other cases (primitive functions, direct linking, others?) this won't work

Comment by Kenny Williams [ 02/Jun/16 3:39 PM ]

Hmm. Well this should be at least be documented. So, spec cannot be used on functions with a type hinted arg?

Comment by Sean Corfield [ 02/Jun/16 4:16 PM ]

Spec cannot be used on functions with primitive typed hinted arguments or returns – non-primitive type hints seem to be fine.

But documentation isn't enough here: instrumenting a namespace and then discovering it broke a function (that happened to have a primitive type hint) isn't acceptable. If the instrumentation isn't going to work, the function should be skipped (and a warning produced, hopefully).

Comment by Kevin Downey [ 02/Jun/16 8:10 PM ]

yeah, I was giving the root cause of the issue, not excusing the issue.

Understanding the root cause predicts other places where there will be issues: where ever some non-default function linking strategy is used.

One such place is direct linked functions, but I suspect for direct linked functions, Clojure/Core will just say you should only instrument code for testing, and you should only turn on direct liking for production.

Another case, which I am sort of surprised we haven't heard more about yet is protocol functions.

Comment by Sean Corfield [ 02/Jun/16 8:35 PM ]

Your comment about direct linking made me wonder about the validity of spec'ing and instrumenting clojure.core functions. The examples show clojure.core/symbol, but Clojure's core library is shipped as direct linked, as of 1.8.0 isn't it?

Comment by Kevin Downey [ 03/Jun/16 3:14 AM ]

what alters the calling convention isn't the function being compiled with direct linking on, but a caller of that function being compiled with direct linking on.

This code will throw a non-conforming error for the bogus symbol spec with direct linking off, and return the symbol foo with direct linking on

(require '[clojure.spec :as s])

(s/fdef symbol
  :args string?
  :ret symbol?)

(defn foo
  []
  (symbol 'foo))

(s/instrument-all)

(foo)
Comment by Kevin Downey [ 03/Jun/16 3:26 AM ]

This code returns true because m is a protocol function, if you replace it with a regular function it throws a non-conforming error

(require '[clojure.spec :as s])

(defprotocol P
  (m [_]))

(deftype T []
  P
  (m [_] true))

(s/fdef m
  :args (s/cat :p (constantly false))
  :ret string?)

(defn foo
  []
  (m (T.)))

(s/instrument-all)

(foo)
Comment by Alex Miller [ 13/Jun/16 3:53 PM ]

@Sean instrumenting core functions will work for calls from your code into core (which are presumably not direct-linked), but will not affect calls from one core function to another as they are direct-linked and do not go through the var. One thing we've considered for a long while is building a dev version of core that would not be direct-linked and could potentially turn on instrumentation or other helpful dev-time tooling.

Comment by Sean Corfield [ 14/Jun/16 5:48 PM ]

Thanks for that answer @alexmiller – We have dev set to non-direct-linking but QA / production set to direct linking, so I'm only concerned about possible issues in dev with (s/instrumental-all) and wanting to be sure "code won't break". If instrumentation won't affect existing (direct-linked) calls within core, that's good enough for me. I am concerned about primitive hinting and protocols (and whatever crawls out of the woodwork here) since you don't want to be forced to read the source of every library you include, just to see whether (s/instrument-all) is safe or whether it will bite you in some weird way while you're developing.





[CLJ-1928] Provide meaning error message when eval of function fails. Created: 15/May/16  Updated: 15/May/16

Status: Open
Project: Clojure
Component/s: None
Affects Version/s: Release 1.8
Fix Version/s: None

Type: Defect Priority: Minor
Reporter: Richard Davies Assignee: Unassigned
Resolution: Unresolved Votes: 0
Labels: errormsgs
Environment:

All



 Description   

When attempting to eval a function, in some cases this fails with "No matching ctor found". This error does not clearly indicate the root cause. Suggest something like "cannot eval function" or something similar. See http://dev.clojure.org/jira/browse/CLJ-1206 for history relating to this ticket.






[CLJ-1925] Add uuid and random-uuid functions Created: 10/May/16  Updated: 15/May/17

Status: Open
Project: Clojure
Component/s: None
Affects Version/s: None
Fix Version/s: None

Type: Feature Priority: Minor
Reporter: Daniel Compton Assignee: Unassigned
Resolution: Unresolved Votes: 1
Labels: function


 Description   

ClojureScript has uuid and random-uuid functions. These are handy to have in ClojureScript, and I think would be useful also in Clojure to improve code portability. Is there interest in a patch for this?



 Comments   
Comment by Alex Miller [ 10/May/16 8:00 AM ]

I think the main reason to do this would be portability. It would make most sense to generate java.util.UUIDs - is that harmonious with what is being done in ClojureScript? That is, could the same code for creating and using uuids work on both platforms? If not, then there might not be a good reason to do so.

Comment by Daniel Compton [ 10/May/16 3:45 PM ]

> It would make most sense to generate java.util.UUIDs - is that harmonious with what is being done in ClojureScript?

ClojureScript defines it's own UUID type, as one doesn't exist in JavaScript. https://github.com/clojure/clojurescript/blob/dd589037f242b4eaace113ffa28ab7b3791caf47/src/main/cljs/cljs/core.cljs#L10088-L10128. I'm not quite sure what you mean by harmonious.

> That is, could the same code for creating and using uuids work on both platforms?

The CLJS UUID doesn't support all of the methods of the Java UUID, but the important things are there (equivalence, constructing from a string, printing to a string) and they would be enough to significantly improve portability when working with UUID's.

Comment by Nicola Mometto [ 10/May/16 4:27 PM ]

both clojure and clojurescript have uuid tagged literals, that should be good enough for interop

Comment by Alex Miller [ 11/May/16 2:48 PM ]

I'm aware of that, just wondering if there are any functions you might invoke on a uuid that would need some portable equivalent, like the stuff in http://docs.oracle.com/javase/8/docs/api/java/util/UUID.html.

Comment by Daniel Compton [ 11/May/16 3:27 PM ]

Most of the extra methods here are useful for distinguishing between multiple types of UUID's, or getting information out of time based UUIDs.

clockSequence() - time based
compareTo(UUID val) - not sure if equivalent required?
boolean	equals(Object obj) - no action required
static UUID	fromString(String name) - constructor
long	getLeastSignificantBits() - not sure how important these two are
long	getMostSignificantBits()
int	hashCode() - no action required
static UUID	nameUUIDFromBytes(byte[] name) - is this useful/important?
long	node() - only useful for time UUID
static UUID	randomUUID() - would implement this
long	timestamp() - time based UUID
String	toString() - no action required
int	variant() - for distinguishing between different types of UUID's
int	version() - for distinguishing between different versions of UUID's

I could potentially see an argument for time based UUID's being included in a patch here too, but I'm not sure if they are used enough to be worth it, and they'd need to go into CLJS, e.t.c.

There is use of some of these methods in Clojure code:
https://github.com/search?l=clojure&q=.getLeastSignificantBits&type=Code&utf8=
https://github.com/search?utf8=✓&q=%22.nameUUIDFromBytes%22+language%3Aclojure&type=Code&ref=searchresults

But less than the literal constructor by a factor of ~100:
https://github.com/search?utf8=✓&q=java+util+UUID+language%3Aclojure&type=Code&ref=searchresults (this is a flawed search query, but the best I could do).

Comment by Alex Miller [ 11/May/16 3:56 PM ]

I guess my greater point is: rather than consider just the functions uuid/random-uuid, let's consider the problem to be: how can we add portable uuid support in Clojure/ClojureScript? That's a lot more work, but a lot more valuable in my opinion.

So would also want to consider (some of these exist already, but may not have been tested for portability):

  • construction
  • printing - print, pr, pretty print
  • reading
  • hash code
  • conversion to/from bits
  • conversion to/from string
  • extraction of components

And then I think it's worth considering how much of this should be in core vs in a data.uuid or something.

I think it's probably better to work it off a design page than here (this ticket is but one unit of the greater problem). Perhaps http://dev.clojure.org/pages/viewpage.action?pageId=950382 could suggest some pointers.





[CLJ-1916] AOT compilation sometimes results in extra classes for already compiled namespaces Created: 19/Apr/16  Updated: 19/Apr/16

Status: Open
Project: Clojure
Component/s: None
Affects Version/s: Release 1.8
Fix Version/s: None

Type: Defect Priority: Minor
Reporter: Mike Kaplinskiy Assignee: Unassigned
Resolution: Unresolved Votes: 0
Labels: aot


 Description   

Case-in-point: clojure/tools.logging.

Repro:

  • AOT compile all the namespaces in clojure/tools.logging (clojure.tools.logging & clojure.tools.logging.impl)
  • With the result on the classpath, AOT compile clojure/java.data (clojure.java.data)
  • Observe `clojure/tools/logging$eval32$fn__33.class` in the output of the second compile (make sure to have different output directories for the two compiles).

This is normally harmless, but becomes an issue if you try to cache AOT compilation output. When you try to cache previous AOT runs this way, you sometimes end up with two otherwise unrelated namespaces generating the same filename. If these had the same contents that would be fine, but there's no guarantee that they have the same contents (since 32 & 33 there are just (gensym)s). Depending on which one "wins" in a classpath this could end badly.

I'm not an expert here, but it would be nice if these "extras" were either generated as part of tools.logging or were somehow aliased into the namespace they were compiled from (e.g. clojure/java/data/$clojure$tools$logging$eval32$fn__33.class or clojure/tools/logging/$clojure$java$data$eval32$fn_33.class).



 Comments   
Comment by Kevin Downey [ 19/Apr/16 6:42 PM ]

tools.logging uses eval to generate some code only when certain classes are present on the classpath, eval generates class files, when you have aot compiling turned on, those class files will be output to the filesystem.

the reason the name is the way it is, is because the eval happens when the tools.logging namespace is loading, so the value of the ns var is the tools.logging namespace, which is what the compiler is generating the name from.





[CLJ-1915] Tests for clojure.core/atom Created: 18/Apr/16  Updated: 18/Apr/16

Status: Open
Project: Clojure
Component/s: None
Affects Version/s: Release 1.9
Fix Version/s: None

Type: Enhancement Priority: Minor
Reporter: Linus Ericsson Assignee: Unassigned
Resolution: Unresolved Votes: 0
Labels: atom

Attachments: File 0001-atom-unit-tests.clj    
Patch: Code and Test
Approval: Triaged

 Description   

As per discussion with Alex Miller Mars 3rd 2016 on clojure-dev, Alex suggested we should add tests to clojure.core/atom functionality, of which there is none today.

I proposed tests for

  • the various ways to instatiate atoms (with and without validator and metadata)
  • that validators throws correctly
  • adding and removing watchers and that they trig as one would expected.
  • various ways of changing values (no aim for finding high-load concurrency issues or patological cases or similar).
  • that the arities of the interface in IAtom .swap works as expected - ie no reflection warnings (help/pointers for these type of cases needed!)
  • generative, tests trying to find the glitches while using atoms (the things excluded above).

Alex suggested generative testing, but no performance tests.

The patch "0001-atom-unit-tests.clj" (attached) contains unit tests for

  • creating "bare" atom
  • creating atom with validator
  • that validate-fn triggers and that the atom is unchanged
  • that deref (@) reader macro creates correct '(clojure.core/deref a)
  • that CAS works for ordinary values (no validator-triggering etc).

There are plenty of combinations not covered with these tests, but this is a start.

To cover all cases (like cas-ing with invalid values and other strange things) generative testing is indeed a must.






[CLJ-1911] min-key and max-key should return NaN if any of the argument is NaN Created: 08/Apr/16  Updated: 12/May/16

Status: Open
Project: Clojure
Component/s: None
Affects Version/s: Release 1.8
Fix Version/s: None

Type: Defect Priority: Minor
Reporter: Renzo Borgatti Assignee: Unassigned
Resolution: Unresolved Votes: 3
Labels: None
Environment:

Likely All. Including older version of Clojure.


Attachments: Text File CLJ-1911-contagious-NaN-and-tests.patch     Text File CLJ-1911-contagious-NaN.patch     Text File CLJ-1911-NaN-fix-over-CLJ-99.patch    
Patch: Code
Approval: Triaged

 Description   

It appears that min-key and max-key behave incorrectly (following Java that follows IEEE floating point convention):

(apply max-key last [[:a 10000] [:b (/ 0. 0)] [:c 0]])
[:c 0]

Not sure how this should then propagate forward, but definitely not silently. Options:

1. [:b NaN] (the first item to generate the NaN)
2. NaN (this is changing the expected type)
3. ArithmeticException Operation with at least one NaN operand.

If this was to be patched the same as it was for min/max (http://dev.clojure.org/jira/browse/CLJ-868) it will probably result in option 1.



 Comments   
Comment by Nicholas Antonov [ 14/Apr/16 9:36 PM ]

This implements the first solution of a contagious NaN in the same style as CLJ 868

Comment by Alex Miller [ 15/Apr/16 12:03 AM ]

Patch should have tests...

Comment by Nicholas Antonov [ 15/Apr/16 1:07 AM ]

This latest patch adds tests for min-key and max-key with and without NaN results, as there were none before.

Comment by Alex Miller [ 29/Apr/16 10:06 AM ]

This overlaps with CLJ-99, which has already been prescreened. I would like to base whatever changes this patch requires over the top of that ticket. To build this, apply the CLJ-99 patch, then branch, make you changes, and then create a patch relative to the clj-99 branch. Sorry that's a pain - usually patches don't collide at this level of conflict.

Comment by Nicholas Antonov [ 12/May/16 6:14 AM ]

The latest patch fixes min and max key in the same way, but based over CLJ-99, only evaluating the function once for each item.





[CLJ-1908] Add clojure.test api to run single test with fixtures and report Created: 01/Apr/16  Updated: 15/May/17

Status: Open
Project: Clojure
Component/s: None
Affects Version/s: Release 1.8
Fix Version/s: None

Type: Feature Priority: Minor
Reporter: Howard Lewis Ship Assignee: Unassigned
Resolution: Unresolved Votes: 2
Labels: test

Attachments: Text File CLJ-1908-3.patch    
Approval: Prescreened

 Description   

When developing code, it is sometimes effective to focus on a single failing test, rather than running all tests in a namespace. This can be the case when running the tests takes some amount of time, or when running the tests produces a large volume of failures. The best option for running a single test with fixtures currently is `test-vars` ala:

(use 'clojure.test) 
(def counter (atom 0)) 
(defn setup [f] (swap! counter inc) (f)) ;; a :once fixture with state 
(use-fixtures :once setup) 
(deftest ex (println "counter =" @counter))

(test-vars [#'ex])  ;=> counter = 1 
(test-vars [#'ex])  ;=> counter = 2

However, this has the following issues:

  • No test reporting feedback such as you get with run-tests (on success, there is no output)
  • Need to specify var (not symbols) wrapped in a vector

Proposed: A new macro `run-test` that specifies a single symbol and does the same test reporting you get with `run-tests`. Usage:

(use 'clojure.test) 
(def counter (atom 0)) 
(defn setup [f] (swap! counter inc) (f)) ;; a :once fixture with state 
(use-fixtures :once setup) 
(deftest ex (println "counter =" @counter)) 

(run-test ex)

;=> Testing user
;=> counter = 1

;=> Ran 1 tests containing 0 assertions.
;=> 0 failures, 0 errors.
;=> {:test 1, :pass 0, :fail 0, :error 0, :type :summary}

(run-test ex)

;=> Testing user
;=> counter = 2

;=> Ran 1 tests containing 0 assertions.
;=> 0 failures, 0 errors.
;=> {:test 1, :pass 0, :fail 0, :error 0, :type :summary}

Patch: CLJ-1908-3.patch



 Comments   
Comment by Howard Lewis Ship [ 01/Apr/16 4:12 PM ]

Having trouble with the patch, in that, things that work at the REPL fail when executed via `mvn test`. Tracking down why is taking some time.

Comment by Howard Lewis Ship [ 01/Apr/16 4:40 PM ]

Initial patch; code works but mvn test fails and I haven't figured out why.

Comment by Howard Lewis Ship [ 01/Apr/16 5:44 PM ]

Thanks to Hiredman, was provided with insight that back ticks needed due to how Maven/Ant runs the tests. All tests now pass.

Comment by Alex Miller [ 01/Apr/16 6:43 PM ]

As far as I can tell, this is basically the same intent as CLJ-866 which was completed in Clojure 1.6. You can do this now with test-vars:

user=> (use 'clojure.test) 
nil 
user=> (def counter (atom 0)) 
#'user/counter 
user=> (defn setup [f] (swap! counter inc) (f)) ;; a :once fixture with state 
#'user/setup user=> (use-fixtures :once setup) {:clojure.test/once-fixtures (#object[user$setup 0x7106e68e "user$setup@7106e68e"])} user=> (deftest ex (println "counter =" @counter)) #'user/ex user=> (test-vars [#'ex]) 
counter = 1 
nil 
user=> (test-vars [#'ex]) 
counter = 2 
nil
Comment by Howard Lewis Ship [ 03/Apr/16 12:27 PM ]

I think there is some advantage to being able to run the tests using is symbol, not its var. Further, the change I've suggested also returns the same kind of data that `run-tests` does.

Comment by Alex Miller [ 04/Apr/16 9:23 AM ]

Some changes needed on this patch before I will prescreen it:

  • Patch should be squashed to a single commit
  • Commit message in patch should start with "CLJ-1908"
  • Change run-test* to run-test-var
  • The docstring for run-test-var should be: "Run the test in Var v with fixtures and report." Kill the "called from" sentence".
  • The first sentence of the docstring for run-test should be: "Runs a single test in the current namespace." Remove "This is meant to be invoked interactively, from a REPL.". Last sentence is ok.
  • In run-test, replace (ns-resolve ns test-symbol) with the simpler (resolve test-symbol).
Comment by Howard Lewis Ship [ 04/Apr/16 10:52 AM ]

Thanks for the input; I'll have an updated patch shortly.

Comment by Howard Lewis Ship [ 08/Apr/16 2:51 PM ]

Updated patch, squashed and reflecting all of Alex's comments.





[CLJ-1907] Document non-caching behaviour of `iterate` when used as generator Created: 31/Mar/16  Updated: 31/Mar/16

Status: Open
Project: Clojure
Component/s: None
Affects Version/s: None
Fix Version/s: None

Type: Enhancement Priority: Minor
Reporter: Nicola Mometto Assignee: Unassigned
Resolution: Unresolved Votes: 0
Labels: docstring

Approval: Triaged

 Description   

The non-caching behaviour of `iterate` when used as a generator is not documented and counter-intuitive. It should be documented, just like it's documented for e.g. `eduction`.

Even though the docstring for `iterate` requires `f` to be side-effect free, `f` might take a long time to compute, in which case users should be wary that the computation might happen more than once.






[CLJ-1906] Clojure should make representing iterated api calls easier Created: 30/Mar/16  Updated: 15/Sep/17

Status: Open
Project: Clojure
Component/s: None
Affects Version/s: None
Fix Version/s: None

Type: Feature Priority: Minor
Reporter: Kevin Downey Assignee: Ghadi Shayban
Resolution: Unresolved Votes: 5
Labels: None

Attachments: Text File 0001-CLJ-1906-add-ingeminate-function.patch     Text File 0001-CLJ-1906-add-unfold-function.patch     Text File 0001-CLJ-1906-transducer-enabled-iterate.patch     File CLJ-1906-seqable-reducible.diff     Text File CLJ-1906-successions.patch    

 Description   

Many apis (elasticsearch, github, s3, etc) have parts of the api
which, in usage, end up being used in an interative way. You make an
api call, and you use the result to make another api call, and so
on. This most often shows up in apis have some concept of pages of
results that you page through, and is very prevalent in http apis.

This appears to be such a common pattern that it would be great if
Clojure had in built support for it.

You may think Clojure already does have support for it, after all,
Clojure has `iterate`. In fact the docstring for `iterate`
specifically says the function you give it must be free of side
effects.

I propose adding a function `unfold` to clojure.core to support this
use case. `unfold` would return an implementation of ReduceInit. The
name `unfold` matches what would be a similar Haskell function
(https://hackage.haskell.org/package/base-4.8.2.0/docs/Data-List.html#v:unfoldr)
and also matches the name for a similar function used in some existing
Clojure libraries
(https://github.com/amalloy/useful/blob/develop/src/flatland/useful/seq.clj#L128-L147).

`unfold` in some ways looks like a combination of `take-while` and
`iterate`, except for the fact that `iterate` requires a pure
function. Another possible solution would be a version of `iterate`
that doesn't require a pure function.

It seems like given the use case I envision for `unfold`, a
non-caching reducible would be perfect. But that would leave those
that prefer seqs high and dry, so maybe at least some consideration
should be given to seqs.

Mailing list discussion is here
(https://groups.google.com/forum/#!topic/clojure-dev/89RNvkLdYc4)

A sort of dummy api you might want to interact with would look something like

(import '(java.util UUID))

(def uuids (repeatedly 1000 #(UUID/randomUUID)))

(def uuid-index
  (loop [uuids uuids
         index  {}]
    (if (seq uuids)
      (recur (rest uuids) (assoc index (first uuids) (rest uuids)))
      index)))

(defn api
  "pages through uuids, 10 at a time. a list-from of :start starts the listing"
  [list-from]
  (let [page (take 10 (if (= :start list-from)
                        uuids
                        (get uuid-index list-from)))]
    {:page page
     :next (last page)}))

given the above api, if you had an implementation of `unfold` that took a predicate that decided when to continue unfolding, a producer which given a value in a sequence produced the next value, and an initial value, you could do something like this:

(= uuids (into [] (mapcat :page) (unfold :next (comp api :next) (api :start))))

and the result would be true.

The equivilant take-while + iterate would be something like:

;; the halting condition is not strictly the same
(= uuids (into [] (mapcat :page) (take-while (comp seq :page) (iterate (comp api :next) (api :start)))))


 Comments   
Comment by Kevin Downey [ 31/Mar/16 4:21 PM ]

I made two patches, one adds unfold as discussed above, one adds ingeminate which is like iterate but without the function purity restrictions, and doesn't return a seq.

Comment by Ghadi Shayban [ 11/Apr/16 10:46 AM ]

Though syntax is less important than the semantics, may I propose the name `progression` for this? Clojure's fold is called reduce, so unfold is too much like Haskell. Other names I was considering include evolve & derivations.

Comment by Alex Miller [ 11/Apr/16 11:23 AM ]

Another option would be `productions` (reminiscent of `reductions`).

Comment by Kevin Downey [ 11/Apr/16 9:32 PM ]

productions has a nice ring to it. emanate could work too, would sort near eduction

Comment by Ghadi Shayban [ 12/Apr/16 10:08 PM ]

Adding a patch with a generator impl that is clojure.lang.{Seqable,IReduceInit}.

Generative tests assert that the seq and reduce halves are equivalent.

Tests assert basic functionality, obeying reduced, and maximal laziness of the seq impl.

Docstring has been wordsmithed and the function named `productions`.

Comment by Kevin Downey [ 18/Apr/16 3:21 PM ]

apparently unfold is part of SRFI 1: List Library in scheme land http://srfi.schemers.org/srfi-1/srfi-1.html#FoldUnfoldMap

it looks like their unfold is take-while + iterate + map

Comment by Ghadi Shayban [ 23/Apr/16 11:06 PM ]

Main differences between Scheme's impl and this proposed one:
Predicate reversed (stop? vs continue?)
Scheme has a "mapping function" to produce a different value from the current seed, Clojure doesn't (but has transducers)
Scheme has an extra optional arg to build the tail of the list

Now I'm partial to the name successions.

Comment by Michał Marczyk [ 10/May/16 11:07 AM ]

I can confirm that I found unfold quite useful in my Scheme days.

In Clojure, this general pattern can be expressed using transducers at a modest cost in keystrokes:

(def numbers (doall (range 1000)))

(defn api [list-from]
  (if list-from
    (let [page (vec
                 (take 10 (if (= :start list-from)
                            numbers
                            (drop list-from numbers))))]
      {:page page
       :next (some-> (last page) inc)})))

(= numbers
   (sequence (comp (take-while some?)
                   (mapcat :page))
             (iterate (comp api :next)
                      (api :start))))
;= true

Maybe this could be simplified with an xform-enabled version of iterate?

(defn iterate*
  ([f seed]
   (iterate f seed))
  ([xform f seed]
   (sequence xform (iterate f seed))))

(= numbers
   (iterate*
     (comp (take-while some?) (mapcat :page))
     (comp api :next)
     (api :start)))
;= true

Admittedly this takes more characters, but is quite generic and a transducer-enabled overload in iterate feels pretty natural to me. Attaching a simple patch implementing this in clojure.core/iterate – I'll look at clojure.lang.Iterate to see if it's worth implementing direct support later, unless of course nobody wants this.

Comment by Michał Marczyk [ 10/May/16 11:08 AM ]

0001-CLJ-1906-transducer-enabled-iterate.patch adds a ternary overload to iterate that delegates to the binary overload and sequence.

Comment by Ghadi Shayban [ 10/May/16 12:56 PM ]

A few unsatisfactory things about overloading {iterate}
1) iterate's docstring says {f must be free of side-effects}
2) There is boilerplate and subtlety around the terminating item. In this case the final API call is made unconditionally, leading to an extra empty/marker item that is filtered by take-while. With the current proposal, the predicate controls iteration from the inside out

Comment by Ghadi Shayban [ 06/Jun/16 8:40 AM ]

updated patch to apply cleanly to core

Comment by Ghadi Shayban [ 18/Sep/16 11:40 PM ]

I'm not sure I'm sold on this anymore, and have suggested a different approach on the mailing list https://groups.google.com/d/msg/clojure-dev/89RNvkLdYc4/PAJh8gfmDAAJ

Comment by Ghadi Shayban [ 17/Apr/17 1:40 PM ]

I have been marinating upon two generator functions that cover use-cases including the one listed above.

One of them is similar to Scheme's unfold but with some deviations more appropriate to Clojure. The other function takes a side-effecting producer and a sentinel value.

Ignore the naming and examine the semantics. https://gist.github.com/ghadishayban/902373e247e920855139902912d237f0





[CLJ-1905] loop should retain primitive int or float without widening Created: 29/Mar/16  Updated: 15/Apr/16

Status: Open
Project: Clojure
Component/s: None
Affects Version/s: Release 1.8
Fix Version/s: None

Type: Enhancement Priority: Minor
Reporter: Renzo Borgatti Assignee: Unassigned
Resolution: Unresolved Votes: 1
Labels: compiler, performance, primitives
Environment:

Possibly older Clojure versions (but not verified).


Attachments: Text File 0001-CLJ-1905-remove-useless-widening-on-loop-bindings.patch    
Patch: Code and Test
Approval: Prescreened

 Description   

In the following example:

(defn incrementer [n]
  (let [n (int n)]
    (loop [i (int 0)]
      (if (< i n)
        (recur (unchecked-inc-int i))
        i))))

the loop-local starts as an int but is widened to a local but is widened to a long in the recur. It should be possible to retain the primitive int (or alternately float) type on the recur, rather than widening to long (or double).

The compiler code that is promoting the int seems to be:
https://github.com/clojure/clojure/blob/master/src/jvm/clojure/lang/Compiler.java#L6270-L6282

Proposed: remove useless widening on loop bindings

Patch: 0001-CLJ-1905-remove-useless-widening-on-loop-bindings.patch

Prescreening comments: My main question here is: do we want to support primitive int/float loop vars?



 Comments   
Comment by Alex Miller [ 29/Mar/16 10:54 AM ]

I don't think anything but primitive longs or doubles are intended to be supported in loop/recur. Presuming that is correct, this would be the expected behavior.

The last major round of numeric/primitive refactoring made the decision to focus only on supporting primitive longs and doubles. One consequence of this is that primitive int loops are difficult to optimize - the main time I run into this is when working with Java interop in a tight loop on (for example) String, collection, or array operations (all of which are int-indexed).

Re unchecked-inc vs unchecked-inc-int, the primary reason to have these two variants is not performance but behavior. In particular, hashing operations often expect to get 32-bit int overflow semantics, not 64-bit int overflow semantics.

In summary, I think in the example given I would not write it with either int or unchecked-inc-int but with long and unchecked-inc if you are looking for best performance.

Comment by Nicola Mometto [ 29/Mar/16 11:01 AM ]

Alex Miller I don't think that's correct, as (AFAIK) it's only in fn params/returns that primitive types are supposed to be restricted to longs and doubles.
Note that for example, char, byte, boolean, short etc work just fine in both let and loop, while int and float work fine in let but not in loop.

This is caused by the following 4 lines in Compiler.java https://github.com/clojure/clojure/blob/master/src/jvm/clojure/lang/Compiler.java#L6278-L6281

As far as I can tell, there's no reason for those 4 lines to be there at this point in time, and removing them makes int and float locals to be emitted correctly inside loops

This example in clojure.org itself seems to assume that ints should work in loops http://clojure.org/reference/java_interop#primitives

Also from that same paragraph:

All Java primitive types are supported
let/loop-bound locals can be of primitive types, having the inferred, possibly primitive type of their init-form

Comment by Alex Miller [ 29/Mar/16 12:07 PM ]

I agree that it should be possible to let-bound primitives of other types - I'm really talking about what happens on recur.

What would you expect to happen for a fn recur target? I wouldn't expect primitives other than long or double to work there since they can't occur in the function signature.

Note that I haven't closed this ticket, still talking through this.

Comment by Alex Miller [ 29/Mar/16 12:10 PM ]

I've definitely run into cases where creating a primitive int loop/recur would be useful for tight interop loops given how common int-indexing is in Java (some of the alioth benchmarks in particular would benefit from this). I think the argument is far weaker for float though.

Comment by Nicola Mometto [ 29/Mar/16 12:19 PM ]

I don't think we need to worry about fn recur targets at all, given that the only possible primitive bindings there are either long or double, and int/floats would get widened anyway, but good point, the tests in a patch for this ticket need to be sure that case is indeed handled.

RE: floats – I recall people complaining about bad support for floats when using clojure for graphical processing.

Even if admittedly a weak argument, I'm always of the idea that we should strike to be as consistent as possible. I don't think anybody would expect let/loop locals to behave differently, or differences between primitive types (other than the documented limitation about long/double being the only working prim types for fn arguments/return vals)

Comment by Alex Miller [ 29/Mar/16 12:30 PM ]

I'll leave this one in here but I'm going to treat it as an enhancement to current behavior. I think there's a good chance that Rich will just say this is working as intended.

I don't think the example is a very good one though and would welcome a better example. The reservations regarding unchecked-inc-int do not seem correct or valid to me (as usage should be fine on longs and is not designed for perf reasons anyways). A good example would should usage of a Java api in a loop where int-indexing and int-math gives better performance.

Comment by Nicola Mometto [ 30/Mar/16 8:51 AM ]

I edited the title as the bug is in `loop`, not `recur`

Comment by Nicola Mometto [ 02/Apr/16 9:55 AM ]

Attached a patch that removes the useless widenings done by the compiler on loop bindings, here's a benchmark demonstrating the speedup gained when areducing over int-arrays:

Before patch:

Clojure 1.8.0
user=> (use 'criterium.core)
nil
user=> (let [xs (int-array (range 1e6))] (bench (areduce xs i ret 0 (+ ret (aget xs i)))))
Evaluation count : 64260 in 60 samples of 1071 calls.
             Execution time mean : 954.009929 µs
    Execution time std-deviation : 20.292809 µs
   Execution time lower quantile : 926.331747 µs ( 2.5%)
   Execution time upper quantile : 1.009189 ms (97.5%)
                   Overhead used : 1.840681 ns

Found 4 outliers in 60 samples (6.6667 %)
	low-severe	 4 (6.6667 %)
 Variance from outliers : 9.4244 % Variance is slightly inflated by outliers
nil

After patch:

Clojure 1.9.0-master-SNAPSHOT
user=> (use 'criterium.core)
nil
user=> (let [xs (int-array (range 1e6))] (bench (areduce xs i ret 0 (+ ret (aget xs i)))))
Evaluation count : 68640 in 60 samples of 1144 calls.
             Execution time mean : 870.462532 µs
    Execution time std-deviation : 13.100790 µs
   Execution time lower quantile : 852.357513 µs ( 2.5%)
   Execution time upper quantile : 896.531529 µs (97.5%)
                   Overhead used : 1.844045 ns

Found 1 outliers in 60 samples (1.6667 %)
	low-severe	 1 (1.6667 %)
 Variance from outliers : 1.6389 % Variance is slightly inflated by outliers
nil




[CLJ-1890] enhance pprint to print type for defrecord (as in pr) Created: 05/Feb/16  Updated: 15/May/17

Status: Open
Project: Clojure
Component/s: None
Affects Version/s: Release 1.8
Fix Version/s: None

Type: Feature Priority: Minor
Reporter: Daniel Compton Assignee: Unassigned
Resolution: Unresolved Votes: 1
Labels: pprint, print

Attachments: Text File CLJ-1890-pprint-records.patch    
Approval: Prescreened

 Description   

pprint currently doesn't print the names of defrecord types, instead printing just the underlying map. This is in contrast to pr-str/println. This ticket proposes that the behaviour of pprint is changed to match pr-str and println's.

More discussion at https://groups.google.com/forum/#!topic/clojure-dev/lRDG6a5eE-s

user=> (defrecord myrec [a b])
user.myrec
user=> (->myrec 1 2)
#user.myrec{:a 1, :b 2}
user=> (pr-str (->myrec 1 2))
"#user.myrec{:a 1, :b 2}"
user=> (println (->myrec 1 2))
#user.myrec{:a 1, :b 2}
nil
user=> (pprint (->myrec 1 2))
{:a 1, :b 2}
nil

Approach: Add new IRecord case to pprint's simple-dispatch mode. Extract guts of pprint-map to pprint-map-kvs and call it from both existing pprint-map and new pprint-record. Set multimethod preference for IRecord version. Added test.

user=> (pprint (->myrec 1 2))
#user.myrec{:a 1, :b 2}

Patch: CLJ-1890-pprint-records.patch

Prescreened by: Alex Miller



 Comments   
Comment by Daniel Compton [ 05/Feb/16 1:51 PM ]

The fix for this will needed to be ported to ClojureScript too.

Comment by Steve Miner [ 06/Feb/16 11:55 AM ]

Added patch to pprint records with classname.

Comment by Steve Miner [ 06/Feb/16 12:03 PM ]

Open question: How should pprint handle a record that has a print-method defined for it? Should the print-method be used instead of the pprint default?

The current release and my patch do not consider the print-method when calling pprint.

Comment by Alex Miller [ 08/Feb/16 4:33 PM ]

I do not think pprint should check for or use print-method. pprint has it's own simple-dispatch multimethod that you can extend, or it will ultimately fall through to pr (which can be extended via print-dup).

Example of the former:

user=> (defrecord R [a])
user=> (def r (->R 1))
user=> (pprint r)
{:a 1}

user=> (use 'clojure.pprint)
user=> (defmethod simple-dispatch user.R [r] (pr r))
#object[clojure.lang.MultiFn 0x497470ed "clojure.lang.MultiFn@497470ed"]
user=> (pprint r)
#user.R{:a 1}
Comment by Steve Miner [ 09/Feb/16 6:27 AM ]

Right, clojure.pprint/simple-dispatch is there for user code to customize pprint, independent of whatever they might do with print-method. No need to conflate the two. So the patch is ready to review.





[CLJ-1889] Add optional predicate to string trim functions that determines if a character should be trimmed Created: 27/Jan/16  Updated: 15/May/17

Status: Open
Project: Clojure
Component/s: None
Affects Version/s: Release 1.7, Release 1.8
Fix Version/s: None

Type: Feature Priority: Minor
Reporter: Tamas Szabo Assignee: Unassigned
Resolution: Unresolved Votes: 0
Labels: string

Attachments: Text File CLJ-1889-trim-enhancement.patch    
Approval: Triaged

 Description   

The proposal is that the trim functions (trim, triml, and trimr) would get a second arity with a function trim?:

[trim? ^CharSequence s]

trim? comes first to support partial.

New doc string would be:

"Removes characters from both ends of string. 
 If trim? is omitted white space is removed. When supplied it accepts 
 a character and returns true if the character should be removed."

Example test:

(deftest t-trim 
  (is (= "foo" (s/trim "  foo  \r\n"))) 
  (is (= "bar" (s/trim "\u2000bar\t \u2002"))) 

  ;; Additional test 
  (is (= "bar" (s/trim "$%#\u2000bar\t \u2002%$#" 
                       #(or (Character/isWhitespace %) ((set "$#%") %))))))

Similar to Python's strip - https://docs.python.org/2/library/stdtypes.html#str.strip

Approach: The proposed solution isn't very DRY but it follows the design guidelines at the top of the file, more exactly point 3:

"3. Functions take advantage of String implementation details to
write high-performing loop/recurs instead of using higher-order
functions. (This is not idiomatic in general-purpose application
code.)"

First I had a solution in which I replaced Character/isWhitespace from the current implementation by calling pred. pred was defaulted to an is-whitespace? function.
That code is of course nicer, even trim-newline could just call into trimr, removing a lot of duplication, but it adds the overhead of always calling a function, instead of calling Character/isWhitespace directly.

The only way I can see to have optimised and DRYer code is to use macros, but I don't think that it will necessary lead to nicer code.

Given the existing design style of the other functions in string.clj I felt that the best solution would be to just simply duplicate in favour of optimised code.



 Comments   
Comment by Tamas Szabo [ 27/Jan/16 2:44 PM ]

Proposed solution. Code + tests.

Comment by Tamas Szabo [ 27/Jan/16 3:42 PM ]

Added new patch that renames pred to trim?

Comment by Andy Fingerhut [ 27/Jan/16 6:27 PM ]

Note that Java, and thus Clojure/Java, uses UTF-16 encoding for strings in memory. Thus if you wanted to trim a set of Unicode code points from the beginning and/or end of a string, the API of trim? taking a single 16-bit Java character is not enough information to determine whether it should be trimmed or not.

If you want to handle that generality, it would require a more complex implementation, which checks whether the first/last character is one half of a code point that is encoded as 2 16-bit Java characters, and pass a 32-bit int to trim?, or something similar to that.

I have no objections if these API enhancements are made without enabling testing against an arbitrary Unicode code point. In the past, similar suggestions have been rejected in Clojure's built-in lib, e.g. CLJ-945

Comment by Tamas Szabo [ 28/Jan/16 1:56 AM ]

Yes, the UTF-16 encoding and Character representing either a codepoint or a half-codepoint is a bit of a mess, isn't it?

In the Java String and Character API's the methods that accept char, handle only characters in the Basic Multilingual Plane.
trim? accepts a character, so following the same behavior it will work only for removing characters in the Basic Multilingual Plane.

I think even this would be fine, but additionally because the high/low surrogates and the BMP characters are disjoint, you could actually use the same implementation to remove Unicode code points that aren't in the BMP. You can just say that both the high and low code unit of the codepoint are "unwanted".

Ex:
𝄞 is "\uD834\uDD1E"

user=> (trimr (set " \uD834\uDD1E") "example string  𝄞  ")
"example string"
Comment by Andy Fingerhut [ 28/Jan/16 5:11 AM ]

Agreed, but probably better to anti-recommend such an implementation of trimr for removing such things, because it would also remove only one UTF-16 Java character out of 2 high/low surrogates if it matched a member of the set, even if the other surrogate didn't match anything in the set, which would leave behind a malformed UTF-16 string.

Again, probably best to either not include this in the implementation at all, and at most warn about it in the docs, or to handle it in the implementation by checking for high/low surrogates in the loop(s).

Comment by Tamas Szabo [ 28/Jan/16 6:00 AM ]

Yes, you're right. That solution won't work in all cases, so it can't be recommended.

I am slightly inclined towards having trim? accept chars and work only for removing BMP characters. This will arguably be enough for the majority of the use cases.
The other solution can be used for all use cases, but then trim? will have to accept int, or 2 chars, or a string, so trim? would be less intuitive (although closer to the real world ), and writing those trim? functions would be less user friendly.

That being said, I am happy to change the implementation to do that if it is required.

Currently, I'm not even sure if the enhancement will be accepted or rejected or what the process for that is.





[CLJ-1886] AOT compilation can cause java.lang.NoSuchFieldError: __thunk__0__ Created: 25/Jan/16  Updated: 14/Jul/17

Status: Open
Project: Clojure
Component/s: None
Affects Version/s: Release 1.7, Release 1.8
Fix Version/s: None

Type: Defect Priority: Minor
Reporter: Ryan Fowler Assignee: Unassigned
Resolution: Unresolved Votes: 0
Labels: aot


 Description   

In some very specific situation that I don't understand, the aot compiler can create class files with an inconsistent idea of a field called _thunk0_.

I've created a project at https://github.com/ryfow/weird-aot that reproduces the problem with `lein run`.

The ingredients for reproduction seem to be slf4j-timbre, tools.analyzer, and core.async.

I suspect that slf4j-timbre being aot compiled but not directly loaded by clojure code is a factor.

Note that the weird-aot timbre version differs from the version compiled in slf4j-timbre.

It's unclear to me why tools.analyzer and core.async are required to exhibit the problem.

Here's the stacktrace I get when I run `lein run` on the weird-aot project.

Exception.txt
Exception in thread "main" java.lang.NoSuchFieldError: __thunk__0__, compiling:(/private/var/folders/2q/tk7cywk93217_d4pxn_5kft40000gn/T/form-init7490372454812250103.clj:1:125)
        at clojure.lang.Compiler.load(Compiler.java:7239)
        at clojure.lang.Compiler.loadFile(Compiler.java:7165)
        at clojure.main$load_script.invoke(main.clj:275)
        at clojure.main$init_opt.invoke(main.clj:280)
        at clojure.main$initialize.invoke(main.clj:308)
        at clojure.main$null_opt.invoke(main.clj:343)
        at clojure.main$main.doInvoke(main.clj:421)
        at clojure.lang.RestFn.invoke(RestFn.java:421)
        at clojure.lang.Var.invoke(Var.java:383)
        at clojure.lang.AFn.applyToHelper(AFn.java:156)
        at clojure.lang.Var.applyTo(Var.java:700)
        at clojure.main.main(main.java:37)
Caused by: java.lang.NoSuchFieldError: __thunk__0__
        at clojure.tools.analyzer.jvm.utils__init.load(Unknown Source)
        at clojure.tools.analyzer.jvm.utils__init.<clinit>(Unknown Source)
        at java.lang.Class.forName0(Native Method)
        at java.lang.Class.forName(Class.java:340)
        at clojure.lang.RT.classForName(RT.java:2154)
        at clojure.lang.RT.classForName(RT.java:2163)
        at clojure.lang.RT.loadClassForName(RT.java:2182)
        at clojure.lang.RT.load(RT.java:436)
        at clojure.lang.RT.load(RT.java:412)
        at clojure.core$load$fn__5448.invoke(core.clj:5866)
        at clojure.core$load.doInvoke(core.clj:5865)
        at clojure.lang.RestFn.invoke(RestFn.java:408)
        at clojure.core$load_one.invoke(core.clj:5671)
        at clojure.core$load_lib$fn__5397.invoke(core.clj:5711)
        at clojure.core$load_lib.doInvoke(core.clj:5710)
        at clojure.lang.RestFn.applyTo(RestFn.java:142)
        at clojure.core$apply.invoke(core.clj:632)
        at clojure.core$load_libs.doInvoke(core.clj:5749)
        at clojure.lang.RestFn.applyTo(RestFn.java:137)
        at clojure.core$apply.invoke(core.clj:632)
        at clojure.core$require.doInvoke(core.clj:5832)
        at clojure.lang.RestFn.invoke(RestFn.java:703)
        at clojure.tools.analyzer.jvm$loading__5340__auto____1677.invoke(jvm.clj:9)
        at clojure.tools.analyzer.jvm__init.load(Unknown Source)
        at clojure.tools.analyzer.jvm__init.<clinit>(Unknown Source)
        at java.lang.Class.forName0(Native Method)
        at java.lang.Class.forName(Class.java:340)
        at clojure.lang.RT.classForName(RT.java:2154)
        at clojure.lang.RT.classForName(RT.java:2163)
        at clojure.lang.RT.loadClassForName(RT.java:2182)
        at clojure.lang.RT.load(RT.java:436)
        at clojure.lang.RT.load(RT.java:412)
        at clojure.core$load$fn__5448.invoke(core.clj:5866)
        at clojure.core$load.doInvoke(core.clj:5865)
        at clojure.lang.RestFn.invoke(RestFn.java:408)
        at clojure.core$load_one.invoke(core.clj:5671)
        at clojure.core$load_lib$fn__5397.invoke(core.clj:5711)
        at clojure.core$load_lib.doInvoke(core.clj:5710)
        at clojure.lang.RestFn.applyTo(RestFn.java:142)
        at clojure.core$apply.invoke(core.clj:632)
        at clojure.core$load_libs.doInvoke(core.clj:5749)
        at clojure.lang.RestFn.applyTo(RestFn.java:137)
        at clojure.core$apply.invoke(core.clj:632)
        at clojure.core$require.doInvoke(core.clj:5832)
        at clojure.lang.RestFn.invoke(RestFn.java:421)
        at weird_aot.core$loading__5340__auto____81.invoke(core.clj:1)
        at weird_aot.core__init.load(Unknown Source)
        at weird_aot.core__init.<clinit>(Unknown Source)
        at java.lang.Class.forName0(Native Method)
        at java.lang.Class.forName(Class.java:340)
        at clojure.lang.RT.classForName(RT.java:2154)
        at clojure.lang.RT.classForName(RT.java:2163)
        at clojure.lang.RT.loadClassForName(RT.java:2182)
        at clojure.lang.RT.load(RT.java:436)
        at clojure.lang.RT.load(RT.java:412)
        at clojure.core$load$fn__5448.invoke(core.clj:5866)
        at clojure.core$load.doInvoke(core.clj:5865)
        at clojure.lang.RestFn.invoke(RestFn.java:408)
        at clojure.core$load_one.invoke(core.clj:5671)
        at clojure.core$load_lib$fn__5397.invoke(core.clj:5711)
        at clojure.core$load_lib.doInvoke(core.clj:5710)
        at clojure.lang.RestFn.applyTo(RestFn.java:142)
        at clojure.core$apply.invoke(core.clj:632)
        at clojure.core$load_libs.doInvoke(core.clj:5749)
        at clojure.lang.RestFn.applyTo(RestFn.java:137)
        at clojure.core$apply.invoke(core.clj:632)
        at clojure.core$require.doInvoke(core.clj:5832)
        at clojure.lang.RestFn.invoke(RestFn.java:408)
        at user$eval65$fn__67.invoke(form-init7490372454812250103.clj:1)
        at user$eval65.invoke(form-init7490372454812250103.clj:1)
        at clojure.lang.Compiler.eval(Compiler.java:6782)
        at clojure.lang.Compiler.eval(Compiler.java:6772)
        at clojure.lang.Compiler.load(Compiler.java:7227)
        ... 11 more


 Comments   
Comment by Kevin Downey [ 26/Jan/16 1:58 AM ]

run.sh in the linked github repo throws:

Exception in thread "main" java.lang.RuntimeException: Method code too large!, compiling:(weird_aot/jetty.clj:4:1)

and fails to compile the required java source

EDIT it does compile the java source, but doesn't create the default compiler output directory for clojure or create it

Comment by Kevin Downey [ 26/Jan/16 2:03 AM ]

`lein compile` with a checkout of the linked github project completes without error for me

Comment by Kevin Downey [ 26/Jan/16 2:20 AM ]

fiddling a little, a number of deps, and their transient dependencies seem to be AOT compiled, likely with different versions of Clojure, which is not intended to work as far as I am aware. Code aot compiled with Clojure version A will fail to link with code being compiled with Clojure version B

Comment by Nicola Mometto [ 26/Jan/16 2:55 AM ]

I agree with Kevin here. The issue is highly likely caused by dependencies being distributed AOT and a dependency clash.

Comment by Kevin Downey [ 26/Jan/16 3:01 AM ]

com.fzakaria/slf4j-timbre "0.2.2" is the issue. the library is aot compiled, which transitively aot compiles its dependencies, which are older versions of a bunch of timbre libraries, which in turn depend on an old version of tools.reader, so the jar for com.fzakaria/slf4j-timbre "0.2.2" contains an old compiled version of `tools.reader`. org.clojure/tools.analyzer.jvm "0.6.9" was aot compiled against a newer version of `tools.reader` so everything explodes

Comment by Alex Miller [ 26/Jan/16 8:54 AM ]

Publishing a jar with AOT'ed dependencies is for sure a problem. I realize this is a bit painful due to CLJ-322 (which I'm hoping to actually make some headway on this year).

Is there something else that should be done on this ticket?

Comment by Nicola Mometto [ 26/Jan/16 8:56 AM ]

I don't think there's anything that we can do other than pushing CLJ-322 and discouraging users to publish AOT compiled libs

Comment by Ryan Fowler [ 26/Jan/16 9:11 AM ]

The problem for me is the error message. It's fine that I can't depend on AOT compiled libraries. It doesn't seem ok that the error message when I do this is "java.lang.NoSuchFieldError: _thunk0_" or "java.lang.RuntimeException: Method code too large!"

Comment by Alex Miller [ 26/Jan/16 9:28 AM ]

I hear you. Unfortuantely, I'm not sure there's any way to detect this is what is happening in a generic way and produce a better error. The same kinds of weirdness can happen in Java as well when using a mixture of library versions.

Comment by Arnout Roemers [ 14/Jul/17 6:47 AM ]

I'm not sure the issue is (only) due to having an AOT compiled library. I started rewriting parts of the slf4j-timbre library, in order to remove the need for AOT [1]. But I noticed the problem in that version remains when the TimbreLoggerAdapter loads the slf4j-timbre.adapter namespace, instead of a Clojure namespace, during compilation. So as soon as a SLF4J log statement is performed during compilation, the `_thunk_` issue still ensued when running the result.

A workaround however is to require the slf4j-timbre.adapter namespace as one of the first in the AOT compiled namespaces. This also works with the current AOT compiled version of slf4j-timbre library. So maybe solving this issue has more to do with transitive loading of namespaces through Java classes during AOT compilation, instead of with AOT compiled libraries per se?

[1] https://github.com/hellodata-org/slf4j-timbre/blob/f9a2f4469fd92063c261276479593de39fffbee3/src-java/com/github/fzakaria/slf4j/TimbreLoggerAdapter.java





[CLJ-1885] data/diff does not return a tuple when comparing different maps Created: 16/Jan/16  Updated: 16/Jan/16

Status: Open
Project: Clojure
Component/s: None
Affects Version/s: Release 1.7, Release 1.8
Fix Version/s: None

Type: Defect Priority: Minor
Reporter: Eric Dvorsak Assignee: Unassigned
Resolution: Unresolved Votes: 0
Labels: None
Environment:

all


Attachments: Text File CLJ-1885.patch     Text File CLJ-1885-tests.patch    
Approval: Triaged

 Description   

Problem: clojure.data/diff inconsistently returns a lazy seq when comparing different maps, but a vector otherwise.

user> (data/diff {:a 1 :b 2} {:a 1})
({:b 2} nil {:a 1})

This is inconsistent with doc and normal behavior :

user> (data/diff {:a 1 :b 2} {:a 1 :b 2})
[nil nil {:a 1, :b 2}]
user> (data/diff #{1 2 3} #{1 2 3})
[nil nil #{1 3 2}]
user> (data/diff #{1 2 3} #{1 2})
[#{3} nil #{1 2}]

The docstring states: "Recursively compares a and b, returning a tuple of [things-only-in-a things-only-in-b things-in-both]", implying that it should always return a vector.



 Comments   
Comment by Eric Dvorsak [ 16/Jan/16 10:02 AM ]

Fixing it just requires to vectorize diff-associative output like this :

(defn- diff-associative
  "Diff associative things a and b, comparing only keys in ks."
  [a b ks]
  (vec (reduce
   (fn [diff1 diff2]
     (doall (map merge diff1 diff2)))
   [nil nil nil]
   (map
    (partial diff-associative-key a b)
    ks))))
Comment by Alex Miller [ 16/Jan/16 10:10 AM ]

There are other potential ways to address this, such as by using transducers instead. Not sure if that's worth doing, but seems reasonable to consider while we're making changes.

Comment by Eric Dvorsak [ 16/Jan/16 10:15 AM ]

Maybe this could be done as an improvement and proposed in an other ticket.

Vec is already used to vectorize the lists in diff-sequential. I would suggest to just fix the bug and add the test cases that should have screen it.

Comment by Eric Dvorsak [ 16/Jan/16 10:20 AM ]

There is a test case that should already fail :

[{:a #{2}} {:a #{4}} {:a #{3}}] {:a #{2 3}} {:a #{3 4}}

I get

({:a #{2}} {:a #{4}} {:a #{3}})
Comment by Alex Miller [ 16/Jan/16 10:33 AM ]

The test may need to be made more strict, checking not just for sequential equality but also for a returned vector.

Just curious - was this issue causing a problem in your code or did you just notice it and find it surprising?

Comment by Eric Dvorsak [ 16/Jan/16 11:05 AM ]

Simple patch that just does for maps what is done for lists : Creates a new vector with the vec function.

Comment by Eric Dvorsak [ 16/Jan/16 11:08 AM ]

@Alex Miller : I noticed a bug in my program behavior and traced it down to a (get diff 2) instead of (nth diff 2), but I realized that it was only buggy in some cases so I looked further and found out if was coming from diff.

Comment by Eric Dvorsak [ 16/Jan/16 11:27 AM ]

More strict tests checking for a returned vector.





[CLJ-1884] Add support for two parameters to rand and rand-int Created: 14/Jan/16  Updated: 15/May/17

Status: Open
Project: Clojure
Component/s: None
Affects Version/s: Release 1.7, Release 1.8
Fix Version/s: None

Type: Feature Priority: Minor
Reporter: Juan José Conti Assignee: Unassigned
Resolution: Unresolved Votes: 1
Labels: None

Attachments: Text File rand_with_two_params1.patch    

 Description   

I'd like to propose a change for rand and rand-int. I'd like to add the possibility to call them with two parameters having the result is in that range.

I think it would be a useful change as now, when you want to get a random number in a range you have to google how to do it because you don't remember it and end up with something that is not obvious what it's doing:

(+ a (rand (- b a))

The change is simple (I've done it locally, build and tested it). Patch attached.

https://groups.google.com/forum/#!topic/clojure-dev/hl2XtXNGb8w






[CLJ-1881] Can :or destructuring refer to previous sequential bindings? Created: 11/Jan/16  Updated: 11/Jan/16

Status: Open
Project: Clojure
Component/s: None
Affects Version/s: Release 1.7
Fix Version/s: None

Type: Enhancement Priority: Minor
Reporter: Alex Miller Assignee: Unassigned
Resolution: Unresolved Votes: 0
Labels: destructuring, docs

Approval: Triaged

 Description   

The following code works, but it is unspecified in the docs whether `(inc a)` can rely on `a` being bound.

user=> (defn foo [a {:keys [b] :or {b (inc a)}}]
  [a b])
user=> (foo 1 {:b 99})
[1 99] ;; :or not needed
user=> (foo 1 {})
[1 2]  ;; :or binds b to (inc a)

In sequential destructuring, are bindings bound in order such that subsequent :or value expressions can rely on prior sequential bindings?

This is true based on the current implementation of destructure, but looking for a statement to this effect in the docs and/or tests.






[CLJ-1875] Parameter names in docstring for `into` Created: 06/Jan/16  Updated: 20/Jan/16

Status: Open
Project: Clojure
Component/s: None
Affects Version/s: Release 1.7
Fix Version/s: None

Type: Enhancement Priority: Minor
Reporter: Marc O'Morain Assignee: Unassigned
Resolution: Unresolved Votes: 0
Labels: docstring

Attachments: Text File CLJ-1875.patch    
Patch: Code
Approval: Triaged

 Description   

The docstring for into does not have the correct names for the parameters. The parameter names in the arglist are to, from and xform, but the doctring refers to them as to-coll from-coll, and the docstring does not refer to the optional transducer, xform by name.

Approach: update docstring to reflect param names
Patch: CLJ-1875.patch
Screened by:



 Comments   
Comment by Alex Miller [ 20/Jan/16 3:58 PM ]

The additional phrase at the end doesn't make sense to me. How about instead:

"A transducer, xform, may be supplied as an optional second argument."





[CLJ-1871] Add the ability to rename columns in clojure.pprint/print-table Created: 24/Dec/15  Updated: 15/May/17

Status: Open
Project: Clojure
Component/s: None
Affects Version/s: Backlog
Fix Version/s: None

Type: Feature Priority: Minor
Reporter: Aviad Reich Assignee: Unassigned
Resolution: Unresolved Votes: 0
Labels: print

Attachments: Text File CLJ-1871.patch    

 Description   

I suggest adding the ability to rename columns in clojure.pprint/print-table, with the following interface:

(print-table [[:b "column b"] [:a "a"]]
                 [{:a 1 :b {:a 'is-a} :c ["hi" "there"]}
                  {:b 5 :a 7 :c "dog" :d -700}])

|  column b |  a |
|-----------+----|
| {:a is-a} |  1 |
|         5 |  7 |


 Comments   
Comment by Aviad Reich [ 24/Dec/15 12:28 AM ]

patch





[CLJ-1867] with-redefs used on a macro permanently changes it to a function Created: 10/Dec/15  Updated: 10/Dec/15

Status: Open
Project: Clojure
Component/s: None
Affects Version/s: Release 1.7
Fix Version/s: None

Type: Defect Priority: Minor
Reporter: Gary Fredericks Assignee: Unassigned
Resolution: Unresolved Votes: 0
Labels: None

Approval: Triaged

 Description   

If you use with-redefs to redefine a macro (which is likely a mistake), the macro loses its macro status after the with-redefs call completes.

Presumably the fix depends on whether we think there is a valid use of with-redefs on a macro (which would only work if you're calling eval or equivalent in the body, and would require knowing enough about what you're doing to add the two extra macro args to your function) – if so, we would keep it from losing the macro status; if not, we might also have it throw an exception if you accidentally use it on a macro.

Demonstration of the effect:

user> (defmacro kwote [arg] `(quote ~arg))
#'user/kwote
user> (kwote hello)
hello
user> kwote
CompilerException java.lang.RuntimeException: Can't take value of a macro: #'user/kwote, compiling:(/tmp/form-init6222001939841513290.clj:1:18983)

;; Everything above is as expected

user> (with-redefs [kwote (constantly :in-with-redefs)] (kwote with-redefs-body))
with-redefs-body
user> (kwote hello)
CompilerException java.lang.RuntimeException: Unable to resolve symbol: hello in this context, compiling:(/tmp/form-init6222001939841513290.clj:1:1) 
user> (kwote :arg-1)
ArityException Wrong number of args (1) passed to: user/kwote  clojure.lang.AFn.throwArity (AFn.java:429)
user> (kwote :arg-1 :arg-2 :arg-3)
(quote :arg-3)
user> kwote
#object[user$kwote 0x37e32ff6 "user$kwote@37e32ff6"]


 Comments   
Comment by Gary Fredericks [ 10/Dec/15 12:04 PM ]

Looks like the root cause is that with-redefs uses Var#bindRoot which intentionally clears the macro flag: https://github.com/clojure/clojure/blob/5cfe5111ccb5afec4f9c73b46bba29ecab6a5899/src/jvm/clojure/lang/Var.java#L270





[CLJ-1866] Optimise argument boxing during reflection Created: 08/Dec/15  Updated: 11/Dec/15

Status: Open
Project: Clojure
Component/s: None
Affects Version/s: None
Fix Version/s: None

Type: Enhancement Priority: Minor
Reporter: Mike Anderson Assignee: Unassigned
Resolution: Unresolved Votes: 0
Labels: performance, reflection

Attachments: Text File clj-1866.patch    

 Description   

Currently argument boxing is clojure.lang.Reflector is inefficient for the following two reasons:
1. It makes an unnecessary call to Class.cast(..) when the parameter type is non-primitive
2. It allocates an unnecessary extra Object[] array when boxing arguments

This patch fixes these issues, without otherwise changing behaviour. All tests pass.



 Comments   
Comment by Alex Miller [ 09/Dec/15 7:23 AM ]

Example code where this is an issue?

Benchmark before/after ?

Comment by Mike Anderson [ 09/Dec/15 9:08 PM ]

Hi alex, I'm trying to improve the fast path for reflection, this will be a bunch of changes, together with clj-1784 and a few other ideas I have.

I don't think it will be productive to have an extensive debate / benchmarking over every single change. Would it be better to make a new ticket for all changes together with benchmarking for the overall impact?

Happy to do this, but it would be helpful if you could give an indication that this stuff will be accepted on the assumption that an overall improvement is demonstrated. I don't want to waste effort on Clojure dev if you guys are not interested in performance improvements in this space. And I don't have time to have an extensive debate over every individual change.

Comment by Alex Miller [ 10/Dec/15 7:51 AM ]

Each ticket should identify a specific problem and propose a solution with evidence that it helps. If there are multiple issues it is quite likely that they may move at different rates.

If you identify a hot spot, then we are happy to look at it. But we have to start from a problem to solve not just: "here are some changes". If the change makes things better, then you should be able to demonstrate that.

Comment by Alex Miller [ 10/Dec/15 7:53 AM ]

I would say that I am still dubious that the right answer to a problem with reflection is not just removing the reflection. But I can't evaluate that until you provide a problem.

Comment by Mike Anderson [ 10/Dec/15 7:10 PM ]

The problem is that it is inefficient (and therefore slow for users of reflection), as stated in the description. If you have an alternative way that you would like to see it worded, can you suggest?

Whether or not people should be using reflection is orthogonal to making reflection itself faster. I agree people should use type hints where they can but plenty of people don't have time to figure it out / don't know how to do properly / don't realise it is happening so surely any improvement for these cases should be welcome?

Also you haven't answered my question: would you like everything rolled into a single reflection performance improvement ticket, or not?

Comment by Alex Miller [ 11/Dec/15 10:11 AM ]

The title of this ticket is "Optimise argument boxing during reflection". That is a solution, not a problem. What I'm looking for is a title like "Reflection with boxed args is slow" and a description that starts with some example code demonstrating the problem. (That example code often makes for a particularly good template for a test that should be in the patch as well.)

I am then looking for evidence that the change you are suggesting improves the problem. For a performance issue, I am specifically looking for a before/after benchmark, preferably using a testing tool like criterium that gives me some confidence that the gains are real.

From a prioritization standpoint, I do not consider reflection performance to be a high priority because the best answer is probably: don't use reflection. That said, I'm willing to consider it, particularly if there is a compelling example where it may be difficult to remove the reflection or where it is particularly non-obvious that the reflection is happening.

Regarding your final question, we prefer to consider individual problems rather "a big bunch of changes", so separate would be better.





[CLJ-1858] Transducer for partition-all with step Created: 28/Nov/15  Updated: 15/May/17

Status: Open
Project: Clojure
Component/s: None
Affects Version/s: Release 1.7
Fix Version/s: None

Type: Feature Priority: Minor
Reporter: Jeremy Apthorp Assignee: Unassigned
Resolution: Unresolved Votes: 0
Labels: transducers


 Description   

The docs for partition-all[1] mention that when a coll is not provided, it returns a transducer. This is true for the form (partition-all n), but not true for (partition-all n step). There's no clear way that I can see to combine transducers from core to produce this sort of "sliding window" transducer, i.e.

user=> (into [] (partition-all 2 1) [1 2 3 4 5])
((1 2) (2 3) (3 4) (4 5) (5))

Of course, there's an arity collision between the above hypothesized (partition-all n step) and the concrete, non-transducer-producing (partition-all n coll), which could be resolved by switching on the type of the second argument, or less hackily by providing the functionality in a separate function, e.g. (sliding window-size step-size), or perhaps by some other means.

I implemented this function with reference to the existing (partition-all n) transducer here: https://gist.github.com/nornagon/03b85fbc22b3613087f6

Would it make sense to work on getting something like this into core?

[1]: http://clojuredocs.org/clojure.core/partition-all






[CLJ-1848] update! for transients Created: 13/Nov/15  Updated: 15/May/17

Status: Open
Project: Clojure
Component/s: None
Affects Version/s: Release 1.7
Fix Version/s: None

Type: Feature Priority: Minor
Reporter: A. R Assignee: Unassigned
Resolution: Unresolved Votes: 0
Labels: None
Environment:

-



 Description   

Now that we have `update` we should possibly also think of having `update!` for transients for consistency.

Thoughts?






[CLJ-1843] Add =to function exposing Util/equivPred Created: 06/Nov/15  Updated: 15/May/17

Status: Open
Project: Clojure
Component/s: None
Affects Version/s: None
Fix Version/s: None

Type: Feature Priority: Minor
Reporter: Nicola Mometto Assignee: Unassigned
Resolution: Unresolved Votes: 1
Labels: None

Attachments: Text File 0001-CLJ-1843-add-to-for-faster-equality-check-against-kn.patch    
Approval: Triaged

 Description   

Description:
It is sometimes useful to compare a collection of values against one value, clojure internally defines a predicate for this exact purpose which has some nice performance improvements over just a partial applied =.

Prior discussion with Rich: https://groups.google.com/forum/#!topic/clojure-dev/0c-VNhEKVkI

Example usage:

;;before:
(map (partial = 3) coll)
;;after:
(map (=to 3) coll)

Benchmarks:

test (partial = 'foo) #(= 'foo %) (=to 'foo)
small homogeneous coll 217ns 165ns 39ns
small eterogeneous coll, 192ns 167ns 41ns
big homogeneous coll 66us 52us 8us
big eterogeneous coll 82us 66us 27us

Full benchmarks output:

(use 'criterium.core)

(defn benchmark-f [f]
  (let [colls [['foo 'foo 'foo]
               [1 :foo 'foo]
               (doall (repeat 1e3 'foo))
               (doall (take 1e3 (cycle [1 :foo 'foo])))]]
    (doseq [c colls]
      (quick-bench (run! f c)))))

(benchmark-f (partial = 'foo))
ARNING: Final GC required 1.405293826432628 % of runtime
WARNING: Final GC required 10.202923149112559 % of runtime
Evaluation count : 3116130 in 6 samples of 519355 calls.
Execution time mean : 217.723199 ns
Execution time std-deviation : 29.425291 ns
Execution time lower quantile : 189.944710 ns ( 2.5%)
Execution time upper quantile : 261.717351 ns (97.5%)
Overhead used : 1.863362 ns
WARNING: Final GC required 4.2579397621583315 % of runtime
Evaluation count : 3138636 in 6 samples of 523106 calls.
Execution time mean : 198.985418 ns
Execution time std-deviation : 12.691848 ns
Execution time lower quantile : 182.441245 ns ( 2.5%)
Execution time upper quantile : 207.839280 ns (97.5%)
Overhead used : 1.863362 ns
WARNING: Final GC required 6.631646134523004 % of runtime
Evaluation count : 10038 in 6 samples of 1673 calls.
Execution time mean : 66.977712 µs
Execution time std-deviation : 10.411821 µs
Execution time lower quantile : 59.620690 µs ( 2.5%)
Execution time upper quantile : 84.483254 µs (97.5%)
Overhead used : 1.863362 ns

Found 1 outliers in 6 samples (16.6667 %)
low-severe  1 (16.6667 %)
Variance from outliers : 47.3059 % Variance is moderately inflated by outliers
WARNING: Final GC required 5.272721959665122 % of runtime
Evaluation count : 7908 in 6 samples of 1318 calls.
Execution time mean : 82.588512 µs
Execution time std-deviation : 5.215537 µs
Execution time lower quantile : 75.977936 µs ( 2.5%)
Execution time upper quantile : 86.849982 µs (97.5%)
Overhead used : 1.863362 ns


(benchmark-f #(= 'foo %))
WARNING: Final GC required 1.284421364203217 % of runtime
WARNING: Final GC required 10.04376144830405 % of runtime
Evaluation count : 3643032 in 6 samples of 607172 calls.
             Execution time mean : 165.393131 ns
    Execution time std-deviation : 1.041355 ns
   Execution time lower quantile : 164.277060 ns ( 2.5%)
   Execution time upper quantile : 166.849951 ns (97.5%)
                   Overhead used : 1.605524 ns
WARNING: Final GC required 6.258680973295933 % of runtime
Evaluation count : 3584574 in 6 samples of 597429 calls.
             Execution time mean : 167.659014 ns
    Execution time std-deviation : 3.821817 ns
   Execution time lower quantile : 164.175156 ns ( 2.5%)
   Execution time upper quantile : 173.210781 ns (97.5%)
                   Overhead used : 1.605524 ns

Found 1 outliers in 6 samples (16.6667 %)
	low-severe	 1 (16.6667 %)
 Variance from outliers : 13.8889 % Variance is moderately inflated by outliers
WARNING: Final GC required 6.914389197005716 % of runtime
Evaluation count : 11196 in 6 samples of 1866 calls.
             Execution time mean : 52.593759 µs
    Execution time std-deviation : 834.220092 ns



(benchmark-f (=to 'foo))
WARNING: Final GC required 7.40391654943877 % of runtime
Evaluation count : 15169068 in 6 samples of 2528178 calls.
Execution time mean : 39.937424 ns
Execution time std-deviation : 2.782661 ns
Execution time lower quantile : 37.393937 ns ( 2.5%)
Execution time upper quantile : 42.780432 ns (97.5%)
Overhead used : 1.863362 ns
WARNING: Final GC required 5.986859953402835 % of runtime
Evaluation count : 15199992 in 6 samples of 2533332 calls.
Execution time mean : 41.229082 ns
Execution time std-deviation : 2.815533 ns
Execution time lower quantile : 37.371527 ns ( 2.5%)
Execution time upper quantile : 43.208673 ns (97.5%)
Overhead used : 1.863362 ns
WARNING: Final GC required 5.039484046472016 % of runtime
Evaluation count : 69462 in 6 samples of 11577 calls.
Execution time mean : 8.976972 µs
Execution time std-deviation : 587.089991 ns
Execution time lower quantile : 8.505317 µs ( 2.5%)
Execution time upper quantile : 9.744296 µs (97.5%)
Overhead used : 1.863362 ns
WARNING: Final GC required 5.773010947849351 % of runtime
Evaluation count : 23352 in 6 samples of 3892 calls.
Execution time mean : 27.277376 µs
Execution time std-deviation : 2.115666 µs
Execution time lower quantile : 25.719322 µs ( 2.5%)
Execution time upper quantile : 30.123547 µs (97.5%)
Overhead used : 1.863362 ns

Patch: 0001-CLJ-1843-add-to-for-faster-equality-check-against-kn.patch



 Comments   
Comment by Alex Miller [ 06/Nov/15 9:15 AM ]

Did you look at (apply = 3 coll) ? Just curious.

Comment by Nicola Mometto [ 06/Nov/15 9:19 AM ]

The advantage of Util/equivPred over Util/equiv is that it can assume the type of the provided argument, avoiding the runtime cost of doing the multiple instance checks that Util/equiv has to do to figure out what comparator to use internally

Comment by Alex Miller [ 06/Nov/15 10:08 AM ]

Could you quantify the difference between these approaches on 2-3 collection sizes?

Comment by Nicola Mometto [ 06/Nov/15 2:53 PM ]

With the following setup:

(use 'criterium.core)

(defn =to [x]
  (let [pred (clojure.lang.Util/equivPred x)]
    (fn [y]
      (.equiv pred x y))))

(defn benchmark-f [f]
  (let [colls [['foo 'foo 'foo]
               [1 :foo 'foo]
               (doall (repeat 1e3 'foo))
               (doall (take 1e3 (cycle [1 :foo 'foo])))]]
    (doseq [c colls]
      (quick-bench (run! f c)))))

The results for (benchmark-f (partial = 'foo) are:

WARNING: Final GC required 1.405293826432628 % of runtime
WARNING: Final GC required 10.202923149112559 % of runtime
Evaluation count : 3116130 in 6 samples of 519355 calls.
Execution time mean : 217.723199 ns
Execution time std-deviation : 29.425291 ns
Execution time lower quantile : 189.944710 ns ( 2.5%)
Execution time upper quantile : 261.717351 ns (97.5%)
Overhead used : 1.863362 ns
WARNING: Final GC required 4.2579397621583315 % of runtime
Evaluation count : 3138636 in 6 samples of 523106 calls.
Execution time mean : 198.985418 ns
Execution time std-deviation : 12.691848 ns
Execution time lower quantile : 182.441245 ns ( 2.5%)
Execution time upper quantile : 207.839280 ns (97.5%)
Overhead used : 1.863362 ns
WARNING: Final GC required 6.631646134523004 % of runtime
Evaluation count : 10038 in 6 samples of 1673 calls.
Execution time mean : 66.977712 µs
Execution time std-deviation : 10.411821 µs
Execution time lower quantile : 59.620690 µs ( 2.5%)
Execution time upper quantile : 84.483254 µs (97.5%)
Overhead used : 1.863362 ns

Found 1 outliers in 6 samples (16.6667 %)
low-severe  1 (16.6667 %)
Variance from outliers : 47.3059 % Variance is moderately inflated by outliers
WARNING: Final GC required 5.272721959665122 % of runtime
Evaluation count : 7908 in 6 samples of 1318 calls.
Execution time mean : 82.588512 µs
Execution time std-deviation : 5.215537 µs
Execution time lower quantile : 75.977936 µs ( 2.5%)
Execution time upper quantile : 86.849982 µs (97.5%)
Overhead used : 1.863362 ns

The results fore (benchmark-f (=to 'foo)) are:

WARNING: Final GC required 7.40391654943877 % of runtime
Evaluation count : 15169068 in 6 samples of 2528178 calls.
Execution time mean : 39.937424 ns
Execution time std-deviation : 2.782661 ns
Execution time lower quantile : 37.393937 ns ( 2.5%)
Execution time upper quantile : 42.780432 ns (97.5%)
Overhead used : 1.863362 ns
WARNING: Final GC required 5.986859953402835 % of runtime
Evaluation count : 15199992 in 6 samples of 2533332 calls.
Execution time mean : 41.229082 ns
Execution time std-deviation : 2.815533 ns
Execution time lower quantile : 37.371527 ns ( 2.5%)
Execution time upper quantile : 43.208673 ns (97.5%)
Overhead used : 1.863362 ns
WARNING: Final GC required 5.039484046472016 % of runtime
Evaluation count : 69462 in 6 samples of 11577 calls.
Execution time mean : 8.976972 µs
Execution time std-deviation : 587.089991 ns
Execution time lower quantile : 8.505317 µs ( 2.5%)
Execution time upper quantile : 9.744296 µs (97.5%)
Overhead used : 1.863362 ns
WARNING: Final GC required 5.773010947849351 % of runtime
Evaluation count : 23352 in 6 samples of 3892 calls.
Execution time mean : 27.277376 µs
Execution time std-deviation : 2.115666 µs
Execution time lower quantile : 25.719322 µs ( 2.5%)
Execution time upper quantile : 30.123547 µs (97.5%)
Overhead used : 1.863362 ns
Comment by Nicola Mometto [ 06/Nov/15 3:07 PM ]

Using #(= 'foo %) rather than (partial = 'foo) allows for inlining of = and makes performance a bit better, but =to still wins noticeably

WARNING: Final GC required 1.284421364203217 % of runtime
WARNING: Final GC required 10.04376144830405 % of runtime
Evaluation count : 3643032 in 6 samples of 607172 calls.
             Execution time mean : 165.393131 ns
    Execution time std-deviation : 1.041355 ns
   Execution time lower quantile : 164.277060 ns ( 2.5%)
   Execution time upper quantile : 166.849951 ns (97.5%)
                   Overhead used : 1.605524 ns
WARNING: Final GC required 6.258680973295933 % of runtime
Evaluation count : 3584574 in 6 samples of 597429 calls.
             Execution time mean : 167.659014 ns
    Execution time std-deviation : 3.821817 ns
   Execution time lower quantile : 164.175156 ns ( 2.5%)
   Execution time upper quantile : 173.210781 ns (97.5%)
                   Overhead used : 1.605524 ns

Found 1 outliers in 6 samples (16.6667 %)
	low-severe	 1 (16.6667 %)
 Variance from outliers : 13.8889 % Variance is moderately inflated by outliers
WARNING: Final GC required 6.914389197005716 % of runtime
Evaluation count : 11196 in 6 samples of 1866 calls.
             Execution time mean : 52.593759 µs
    Execution time std-deviation : 834.220092 ns
   Execution time lower quantile : 51.510161 µs ( 2.5%)
   Execution time upper quantile : 53.367649 µs (97.5%)
                   Overhead used : 1.605524 ns
WARNING: Final GC required 6.179040224498723 % of runtime
Evaluation count : 9162 in 6 samples of 1527 calls.
             Execution time mean : 66.527357 µs
    Execution time std-deviation : 2.119652 µs
   Execution time lower quantile : 65.308835 µs ( 2.5%)
   Execution time upper quantile : 70.201570 µs (97.5%)
                   Overhead used : 1.605524 ns

small homogeneous coll, (partial = 'foo): 217ns, #(= 'foo %): 165ns, (=to 'foo): 39ns
small eterogeneous coll, (partial = 'foo): 192ns, #(= 'foo %): 167ns, (=to 'foo): 41ns
big homogeneous coll, (partial = 'foo): 66us, #(= 'foo %): 52us, (=to 'foo): 8us
big eterogeneous coll, (partial = 'foo: 82us, #(= 'foo %): 66us, (=to 'foo): 27us

Comment by Nicola Mometto [ 17/Dec/15 5:13 PM ]

Apparently this was something that was already discussed a couple of years ago and Rich seemed ok with this https://groups.google.com/forum/#!topic/clojure-dev/0c-VNhEKVkI





[CLJ-1836] Expose clojure.repl/doc as a function call Created: 28/Oct/15  Updated: 15/May/17

Status: Open
Project: Clojure
Component/s: None
Affects Version/s: Release 1.7, Release 1.8
Fix Version/s: None

Type: Feature Priority: Minor
Reporter: Terje Dahl Assignee: Unassigned
Resolution: Unresolved Votes: 0
Labels: repl

Attachments: Text File doc-fn-1.patch    
Approval: Triaged

 Description   

Restructure the printing function clojure.repl/doc so that it calls a fuction clojure.repl/doc-fn for its data - in the same way as dir calls dir-fn. Make doc-fn public so that it can be called directly and allow developers to parse and display the data as needed.

Use case: I am making a namespace inspector (using JavaFX) (somewhat like the Swing-based tree-inspector in Clojure), and when getting a function, I would like to display the same meta-information as "doc" prints in the REPL - including the special forms data coded in a private var/map in Clojure.

Patch: doc-fn.patch



 Comments   
Comment by Alex Miller [ 28/Oct/15 12:34 PM ]

A few comments:

1) Patch authors must sign the contributor's agreement, see http://clojure.org/contributing

2) The patch is not in the correct format - see http://dev.clojure.org/display/community/Developing+Patches for more info.

3) Patch should include a test for the new doc-fn.

Comment by Terje Dahl [ 21/Nov/15 6:29 AM ]

1. Agreement signed.

2. Tests added.
(That was useful! I had to fix a couple of things.)

3. Patch created as per instructions and attached:
"doc-fn-1.patch"





[CLJ-1821] Move map-invert from clojure.set to clojure.core Created: 28/Sep/15  Updated: 28/Sep/15

Status: Open
Project: Clojure
Component/s: None
Affects Version/s: Release 1.8
Fix Version/s: None

Type: Enhancement Priority: Minor
Reporter: Lars Andersen Assignee: Unassigned
Resolution: Unresolved Votes: 0
Labels: None


 Description   

map-invert is hard to find when it lives in the clojure.set namespace, because it acts on maps and not sets. To my eyes set/map-invert also looks strange when reading code, but this is the preferred way to bring in the clojure.set namespace.

This is one of the minor warts I'd like to see fixed in clojure 2.0.






[CLJ-1813] Improve use-fixtures docstring Created: 10/Sep/15  Updated: 13/Oct/15

Status: Open
Project: Clojure
Component/s: None
Affects Version/s: Release 1.7
Fix Version/s: None

Type: Enhancement Priority: Minor
Reporter: Daniel Compton Assignee: Unassigned
Resolution: Unresolved Votes: 0
Labels: clojure.test, docstring

Attachments: Text File fixture-docstring.patch    

 Description   

The docstring for use-fixtures says

Wrap test runs in a fixture function to perform setup and teardown. 
Using a fixture-type of :each wraps every test individually, 
while: once wraps the whole run in a single function

I think it would be helpful to explain what a fixture function is and how it performs setup and teardown. I know because I've looked at examples, but I don't think the docstring explains this at all.

Is this something Core is interested in taking a patch on?



 Comments   
Comment by Alex Miller [ 10/Sep/15 9:08 PM ]

It's explained in the clojure.test ns docstring in more depth:

Fixtures are attached to namespaces in one of two ways.  \"each\"
   fixtures are run repeatedly, once for each test function created
   with \"deftest\" or \"with-test\".  \"each\" fixtures are useful for
   establishing a consistent before/after state for each test, like
   clearing out database tables.

   \"each\" fixtures can be attached to the current namespace like this:
   (use-fixtures :each fixture1 fixture2 ...)
   The fixture1, fixture2 are just functions like the example above.
   They can also be anonymous functions, like this:
   (use-fixtures :each (fn [f] setup... (f) cleanup...))

   The other kind of fixture, a \"once\" fixture, is only run once,
   around ALL the tests in the namespace.  \"once\" fixtures are useful
   for tasks that only need to be performed once, like establishing
   database connections, or for time-consuming tasks.

   Attach \"once\" fixtures to the current namespace like this:
   (use-fixtures :once fixture1 fixture2 ...)

I'm not really answering your question, just wondering if you saw that and whether it changes your question.

Comment by Daniel Compton [ 10/Sep/15 9:15 PM ]

Perhaps just pointing the user towards the ns docstring would be a good alternative? I had forgotten about the docstring on the ns, and I'm not sure whether duplicating the docs about fixtures in use-fixtures is a great idea.

Perhaps something like this?

Wrap test runs in a fixture function to perform setup and teardown. 
Using a fixture-type of :each wraps every test individually, 
while :once wraps the whole run in a single function

See the clojure.test docstring for more details.




[CLJ-1811] test line reporting doesn't always report test's file & line number Created: 02/Sep/15  Updated: 03/Dec/15

Status: Open
Project: Clojure
Component/s: None
Affects Version/s: Release 1.7, Release 1.8
Fix Version/s: None

Type: Enhancement Priority: Minor
Reporter: Ryan Fowler Assignee: Unassigned
Resolution: Unresolved Votes: 0
Labels: test

Attachments: Text File clj-1811.patch     File error_reporting.clj     Text File LINE_REPORING_1.patch     Text File LINE_REPORING_2.patch    
Patch: Code and Test
Approval: Triaged

 Description   

If an Exception is thrown during test execution, the filename and line number are frequently not helpful for finding the problem. For instance, this code:

error_reporting.clj
(require '[clojure.test :refer [deftest test-var]])

(deftest foo
  (meta))

(test-var #'foo)

Will output an error at AFn.java:429.

ERROR in (foo) (AFn.java:429)
Uncaught exception, not in assertion.
expected: nil
  actual: clojure.lang.ArityException: Wrong number of args (0) passed to: core/meta--4144
 at clojure.lang.AFn.throwArity (AFn.java:429)
    clojure.lang.AFn.invoke (AFn.java:28)
    user/fn (error_reporting.clj:4)
    clojure.test$test_var$fn__7670.invoke (test.clj:704)
    clojure.test$test_var.invoke (test.clj:704)
    user$eval6.invoke (error_reporting.clj:6)
    clojure.lang.Compiler.eval (Compiler.java:6782)
    ...etc

Rich's Comment 24016 on CLJ-377 says that he thinks the message should report the test file line rather than where the exception was thrown.

Approach: Filter the stacktrace class prefix clojure.lang.AFn from the top of error stacktraces.

After applying the patch, the above example outputs error_reporting.clj:4:

ERROR in (foo) (error_reporting.clj:4)
Uncaught exception, not in assertion.
expected: nil
  actual: clojure.lang.ArityException: Wrong number of args (0) passed to: core/meta--4141
 at clojure.lang.AFn.throwArity (AFn.java:429)
    clojure.lang.AFn.invoke (AFn.java:28)
    user$fn__3.invokeStatic (error_reporting.clj:4)
    user/fn (error_reporting.clj:3)
    clojure.test$test_var$fn__114.invoke (test.clj:705)
    clojure.test$test_var.invokeStatic (test.clj:705)
    clojure.test$test_var.invoke (test.clj:-1)
    user$eval6.invokeStatic (error_reporting.clj:6)
    user$eval6.invoke (error_reporting.clj:-1)
    clojure.lang.Compiler.eval (Compiler.java:6939)
    ...etc

Patch: clj-1811.patch



 Comments   
Comment by Ryan Fowler [ 02/Sep/15 5:36 PM ]

example file from description

Comment by Alex Miller [ 04/Sep/15 10:04 AM ]

A quick search on Github shows many cases where people call into the (admittedly private) file-and-line function. These users would be broken by the patch. Perhaps it would be better to create a new function or a new arity rather than removing the existing arity.

Just eyeballing it, but I suspect you've introduced reflection in a couple places in the new code, particularly these might need another type hint:

1. (.getName (.getClass (:test (meta test-var))))
2. #(= (.getClassName %) test-var-class-name)

I need to look at more of the code to make a judgement on everything else. Seeing testing-vars in there means that this function is now dependent on external state, so need to think carefully to be sure that every calling context has that global state (or won't fail in bad ways if it doesn't). It would be helpful to see a discussion of your thinking about that in the "approach" section of the ticket description.

Comment by Ryan Fowler [ 30/Sep/15 3:41 PM ]

Second attempt at a patch to resolve this issue. Corrects issues Alex pointed out

Comment by Ryan Fowler [ 30/Sep/15 3:53 PM ]

I've filled in some detail in the approach section.

I also added a new patch LINE_REPORTING_2.patch that addresses reflection warnings, restores the old arity of file-and-line and adds protection from people calling file-and-line from outside a testing context.

Comment by Ryan Fowler [ 30/Sep/15 6:20 PM ]

While discussing an issue with my coworker James, I realized that this fix helps with shared functions calling (is). Notice how the run of this sample code reports line 7 with LINE_REPORTING_2.patch applied. This test line is generally much more useful than the shared function line.

example_2
ryans-mbp:~/oss/clojure% cat -n error_reporting.clj
     1  (require '[clojure.test :refer [deftest test-var is]])
     2
     3  (defn shared-code [arg]
     4    (is arg))
     5
     6  (deftest test-shared-code
     7    (shared-code false))
     8
     9  (test-var #'test-shared-code)
ryans-mbp:~/oss/clojure% java -jar ~/.m2/repository/org/clojure/clojure/1.7.0/clojure-1.7.0.jar ./error_reporting.clj

FAIL in (test-shared-code) (error_reporting.clj:4)
expected: arg
  actual: false
ryans-mbp:~/oss/clojure% java -jar target/clojure-1.8.0-master-SNAPSHOT.jar ./error_reporting.clj

FAIL in (test-shared-code) (error_reporting.clj:7)
expected: arg
  actual: false
Comment by Michael Blume [ 02/Dec/15 5:55 PM ]

Patch doesn't apply anymore, but maybe this has been sorted by CLJ-1856?

Comment by Alex Miller [ 03/Dec/15 9:57 AM ]

This is not fixed by CLJ-1856, but could be if clojure.lang.AFn was filtered out of error stacktraces when finding the location. This is pretty specific - it might make sense to broaden to other calling paths too, not sure.

Attached a new patch that applies this filtering.





[CLJ-1808] map-invert should use transients and reduce-kv instead of reduce Created: 30/Aug/15  Updated: 18/Jan/16

Status: Open
Project: Clojure
Component/s: None
Affects Version/s: None
Fix Version/s: None

Type: Enhancement Priority: Minor
Reporter: Nikita Prokopov Assignee: Unassigned
Resolution: Unresolved Votes: 0
Labels: ft

Attachments: Text File clj-1808-map-invert-should-use-reduce-kv-and-transient.patch    
Patch: Code
Approval: Prescreened

 Description   

Two performance enhancements on clojure.set/map-invert:

1) Use reduce-kv to avoid realizing mapentry's from input map
2) Use transients to create the output map

Patch: clj-1808-map-invert-should-use-reduce-kv-and-transient.patch



 Comments   
Comment by Alex Miller [ 04/Sep/15 10:42 AM ]

Would be nice to see a quick perf test that compared before/after times.





[CLJ-1806] group-by as reducer / reduction fn Created: 29/Aug/15  Updated: 15/May/17

Status: Open
Project: Clojure
Component/s: None
Affects Version/s: Release 1.8
Fix Version/s: None

Type: Feature Priority: Minor
Reporter: Karsten Schmidt Assignee: Unassigned
Resolution: Unresolved Votes: 1
Labels: reducers, transducers

Attachments: Text File clj-1806-1.patch    

 Description   

Whilst working on a query engine heavily based on transducers, I noticed that it'd be great to have group-by able to be used as reduction fn. The attached patch adds a single arity version to group-by which enables this use case and also includes a few tests.



 Comments   
Comment by Karsten Schmidt [ 29/Aug/15 8:08 PM ]

patch w/ described changes





[CLJ-1804] take transducer optimization Created: 25/Aug/15  Updated: 26/Aug/15

Status: Open
Project: Clojure
Component/s: None
Affects Version/s: Release 1.8
Fix Version/s: None

Type: Enhancement Priority: Minor
Reporter: Karsten Schmidt Assignee: Unassigned
Resolution: Unresolved Votes: 0
Labels: performance, transducers

Attachments: Text File clj-1804-1.patch     Text File clj-1804-2.patch    
Patch: Code
Approval: Triaged

 Description   

A basic refactoring to remove the let form and only requires a single counter check for each iteration, yields an 25% performance increase. With the patch, 2 checks are only required for the last iteration (in case counter arg was <= 0)...

;; master
(quick-bench (into [] (take 1000) (range 2000)))
WARNING: Final GC required 34.82584189073624 % of runtime
Evaluation count : 13050 in 6 samples of 2175 calls.
             Execution time mean : 46.921254 µs
    Execution time std-deviation : 1.904733 µs
   Execution time lower quantile : 45.124921 µs ( 2.5%)
   Execution time upper quantile : 49.427201 µs (97.5%)
                   Overhead used : 2.367243 ns

;; w/ patch
(quick-bench (into [] (take 1000) (range 2000)))
WARNING: Final GC required 34.74448252054369 % of runtime
Evaluation count : 18102 in 6 samples of 3017 calls.
             Execution time mean : 34.301193 µs
    Execution time std-deviation : 1.714105 µs
   Execution time lower quantile : 32.341349 µs ( 2.5%)
   Execution time upper quantile : 37.046851 µs (97.5%)
                   Overhead used : 2.367243 ns


 Comments   
Comment by Karsten Schmidt [ 25/Aug/15 10:35 AM ]

Proposed patch, passes all existing tests

Comment by Alex Miller [ 25/Aug/15 11:28 AM ]

From a superficial skim, I see checks for pos? and then neg? which both exclude 0 and that gives me doubts about that branch. That may actually be ok but I will have to read it more closely.

Comment by Karsten Schmidt [ 25/Aug/15 11:58 AM ]

Hi Alex, try running the tests... AFAICT it's all still working as advertised: For (take 0) or (take -1) then the pos? check fails, but we must ensure to not call rf for that iteration. For all other (take n) cases only the pos? check is executed apart from the last iteration (which causes a single superfluous neg? call) The current path/impl always does 2 checks for all iterations and hence is (much) slower.

Comment by Alex Miller [ 25/Aug/15 7:22 PM ]

The only time that neg? case will be hit is if take is passed n <= 0, so I think this is correct. However, you might consider some different way to handle that particular case - for example, it could be pulled out of the transducer function entirely and be created as a separate transducer function entirely. I'm not sure that's worth doing, but it's an idea.

Comment by Karsten Schmidt [ 26/Aug/15 6:01 AM ]

Good idea, Alex! This 2nd patch removes the neg? check and adds a quick bail transducer for cases where n <= 0. It also made it slightly faster still:

(quick-bench (into [] (take 1000) (range 2000)))
Evaluation count : 20370 in 6 samples of 3395 calls.
             Execution time mean : 30.302673 µs

(now ~35% faster than original)





[CLJ-1803] Enable destructuring of sequency maps Created: 22/Aug/15  Updated: 26/Aug/15

Status: Open
Project: Clojure
Component/s: None
Affects Version/s: None
Fix Version/s: None

Type: Enhancement Priority: Minor
Reporter: Reid McKenzie Assignee: Unassigned
Resolution: Unresolved Votes: 0
Labels: destructuring

Attachments: Text File 0001-Enable-destructuring-of-seq-map-types.patch    
Patch: Code
Approval: Triaged

 Description   

At present, Clojure's destructuring implementation will create a hash-map from any encountered value satisfying clojure.core/seq? This has the I argue undesirable side effect of making it impossible to employ destructuring on a custom Associative type which is also a Seq. This came up when trying to destructure instances of a tagged value class which for the purpose of pattern matching behave as [k v] seqs, but since the v is known to be a map, are also associative on the map part so as to avoid the syntactic overhead of updates preserving the tag.

;; A sketch of such a type
(deftype ATaggedVal [t v]
  clojure.lang.Indexed
  (nth [self i]    (nth self i nil))
  (nth [self i o] (case i (0) t (1) v o))

  clojure.lang.Sequential
  clojure.lang.ISeq
  (next  [this]     (seq this))
  (first [this]     t)
  (more  [this]     (.more (seq this)))
  (count [this]     2)
  (equiv [this obj] (= (seq this) obj))
  (seq   [self]     (cons t (cons v nil)))

  clojure.lang.Associative
  (entryAt [self key] (.entryAt v key))
  (assoc [_ sk sv]    (ATaggedVal. t (.assoc v sk sv)))

  clojure.lang.ILookup
  (valAt [self k]   (.valAt v k))
  (valAt [self k o] (.valAt v k o))

  clojure.lang.IPersistentMap
  (assocEx [_ sk sv] (ATaggedVal. t (.assocEx v sk sv)))
  (without [_ sk]    (ATaggedVal. t (.without v sk))))

So using such a thing,

(let [{:keys [x]} (ATaggedVal. :foo {:x 3 :y 4})] x)
;; expect 3
=> nil

Since for any type T such that clojure.core/get will behave, T should satisfy clojure.core/map? it should be correct simply to change the behavior of destructure to only build a hash-map if map? isn't already satisfied.

The attached patch makes this change.



 Comments   
Comment by Alex Miller [ 22/Aug/15 1:21 PM ]

Probably worth watching CLJ-1778 too which might cause this not to apply anymore.

Could you add an example of what doesn't work to the description?

Comment by Ragnar Dahlen [ 24/Aug/15 1:20 AM ]

The current patch for CLJ-1778 does not address this issue.

The idea seems sound to me, if we're map destructuring a value that's
already a map (as determined by map?), we don't need to create a
new one by calling seq and HashMap/create, unless there's a
really good reason it should be exactly that map implementation (I
don't see one).

I don't think the current patch is OK though as it makes an
(unneccessary) breaking change to the behaviour of map destructuring.
Previously, destructuring a non-seqable value returned nil, but with
patch, seq is always called on value and for non-seqble types this
will instead throw an exception. It should be trivial to change the
patch to retain the original behaviour.

1.8.0-master:

user=> (let [{:keys [x]} (java.util.Date.)] x)
nil

with 0001-Enable-destructuring-of-seq-map-types.patch:

user=> (let [{:keys [x]} (java.util.Date.)] x)
IllegalArgumentException Don't know how to create ISeq from: java.util.Date  clojure.lang.RT.seqFrom (RT.java:528)
Comment by Reid McKenzie [ 26/Aug/15 1:15 PM ]

I contend that the behavior broken is, at best, undefined behavior consequent from the implementation and that the failure to cast exception is at least clearer than the silent nil behavior of the original implementation.

I would personally prefer to extend the destructuring checks logically to (cond map? x seq? (hash-map (seq x)) :else (throw "Failed to destructure non-map type") but I think that change is sufficiently large that it would meaningfully decrease the chances of this patch being accepted.





[CLJ-1800] Doc that lazy-seq with-meta forces realization Created: 13/Aug/15  Updated: 19/Aug/15

Status: Open
Project: Clojure
Component/s: None
Affects Version/s: Release 1.7
Fix Version/s: None

Type: Enhancement Priority: Minor
Reporter: Max Penet Assignee: Unassigned
Resolution: Unresolved Votes: 0
Labels: docstring

Attachments: Text File CLJ-1800-no-realize-v1.patch     Text File CLJ-1800-v2.patch    
Approval: Triaged

 Description   

Applying meta to a lazy-seq causes realization:

(def x (vary-meta (lazy-seq (prn :realized)) assoc :foo :bar))
:realized

This might be surprising, so modify docstring of lazy-seq to mention it.

Patch:



 Comments   
Comment by Alex Miller [ 13/Aug/15 9:02 AM ]

I think it's likely that seq() is called here so that the old LazySeq instance and the new one share the sequence. Otherwise the pre-meta and post-meta versions would be performing the same function calls on the same inputs but would be disconnected, which seems bad.

Comment by Alex Miller [ 13/Aug/15 9:03 AM ]

I'm not really sure where this would be documented. Maybe on the http://clojure.org/metadata page?

Comment by Max Penet [ 13/Aug/15 9:18 AM ]

That would make sense yes and on the docstring of lazy-seq as well.

Comment by Alex Miller [ 13/Aug/15 9:47 AM ]

I added a sentence to the metadata page and updated the description to be more applicable here to a docstring change.

Comment by Michael Blume [ 13/Aug/15 1:29 PM ]

With this patch, with-meta doesn't realize the seq, but realization still only happens once – would this be an acceptable approach?

Comment by Michael Blume [ 19/Aug/15 4:46 PM ]

Added test





[CLJ-1799] Replace refs in pprint Created: 13/Aug/15  Updated: 28/Sep/17

Status: Open
Project: Clojure
Component/s: None
Affects Version/s: Release 1.7
Fix Version/s: None

Type: Enhancement Priority: Minor
Reporter: Alex Miller Assignee: Unassigned
Resolution: Unresolved Votes: 0
Labels: performance, print

Attachments: Text File CLJ-1799.patch    
Patch: Code

 Description   

I noticed that pprint uses refs and dosync transactions in a number of places, which seems unlikely to be necessary. It seems like these could be replaced by atoms, or even volatiles, given that printing typically happens in a single thread. Presumably this would improve performance of pprint significantly.



 Comments   
Comment by dennis zhuang [ 14/Aug/15 11:28 AM ]

I develop a patch to fix this issue.I run all the tests in clojure and clojure.data.json, and no one fails.

Use criterium to do a simple benchmark as below:

(use 'criterium.core)
(require '[clojure.data.json :as json])
(bench (json/write-str 
  {:a 1 :b 2 :c (range 10) :d "hello world"
   :e (apply hash-set (range 10))}))

before patch:

Evaluation count : 6180060 in 60 samples of 103001 calls.
             Execution time mean : 10.302604 µs
    Execution time std-deviation : 597.958933 ns
   Execution time lower quantile : 9.631444 µs ( 2.5%)
   Execution time upper quantile : 11.618551 µs (97.5%)
                   Overhead used : 1.724553 ns

After patch:

Evaluation count : 6000900 in 60 samples of 100015 calls.
             Execution time mean : 10.212543 µs
    Execution time std-deviation : 564.874941 ns
   Execution time lower quantile : 9.528383 µs ( 2.5%)
   Execution time upper quantile : 11.334033 µs (97.5%)
                   Overhead used : 1.827143 ns




[CLJ-1798] The RetryEx in LockingTransaction should be static Created: 13/Aug/15  Updated: 13/Aug/15

Status: Open
Project: Clojure
Component/s: None
Affects Version/s: Release 1.7
Fix Version/s: None

Type: Enhancement Priority: Minor
Reporter: dennis zhuang Assignee: Unassigned
Resolution: Unresolved Votes: 0
Labels: STM, performance
Environment:

java version "1.7.0_17"
Java(TM) SE Runtime Environment (build 1.7.0_17-b02)
Java HotSpot(TM) 64-Bit Server VM (build 23.7-b01, mixed mode)

Mac OSX 10.10.4


Attachments: PNG File profile.png     Text File static_retryex.patch    
Patch: Code
Approval: Triaged

 Description   

We are using clojure.data.json, and we profiled our project with jprofiler, it shows that there is a hotspot in LockingTransaction.Too many RetryEx instances were created during the profiling.
The retryex instance variable should be static as a class variable to avoid creating when a new STM transaction begins.

The attacments are the profile result screen snapshot and the patch to make the retryex to be static.
I don't do any benchmark right now,but maybe a clojure.data.json performance benchmark will approve the patch works better.



 Comments   
Comment by Alex Miller [ 13/Aug/15 8:04 AM ]

I think the suggestion here is sound, but I have a hard time believing it will make much of a difference. My real question is why pprint is using dosync.

Comment by dennis zhuang [ 13/Aug/15 9:03 AM ]

I found it uses many dosync in writer as below:

https://github.com/clojure/clojure/blob/master/src/clj/clojure/pprint/pretty_writer.clj

It seems that using the dosync to change some mutable states.Maybe they can be rewrote into other forms such as atom,java object/lock etc.
But it's another story.





[CLJ-1797] Mention cljc when require fails Created: 10/Aug/15  Updated: 09/Mar/16

Status: Open
Project: Clojure
Component/s: None
Affects Version/s: Release 1.7
Fix Version/s: None

Type: Enhancement Priority: Minor
Reporter: Mike Fikes Assignee: Cezary Kosko
Resolution: Unresolved Votes: 1
Labels: errormsgs

Attachments: Text File clj-1797.patch    
Patch: Code
Approval: Prescreened

 Description   

If you attempt to require a namespace for which the code cannot be located, now that cljc files are considered when locating code, it could be mentioned in the list of files considered on the classpath.

$ java -jar clojure-1.7.0.jar 
Clojure 1.7.0
user=> (require 'foo.bar)
FileNotFoundException Could not locate foo/bar__init.class or foo/bar.clj on classpath.  clojure.lang.RT.load (RT.java:449)

Note: FWIW, ClojureScript has similar error messages, and the order listed in the error message is the order tried when loading. If this were followed, I suspect the text of the error message above would end up looking like:

Could not locate foo/bar__init.class, foo/bar.clj, or foo/bar.cljc on classpath.

Patch: clj-1797.patch
Approach: add info about cljcfile not found in exception thrown from load
Screened by: Alex Miller



 Comments   
Comment by Cezary Kosko [ 18/Jan/16 6:21 AM ]

Hey,

I'm a newbie, so I thought I'd do that one (even though it's not vetted yet, but seems it 's bound to be), to get a tad more familiar w/ the sources.

Kind regards,
Cezary

Comment by Alex Miller [ 18/Jan/16 6:53 AM ]

Go for it!





[CLJ-1796] Protocol functions fail to find future extensions when assigned to a local or new var Created: 08/Aug/15  Updated: 10/Aug/15

Status: Open
Project: Clojure
Component/s: None
Affects Version/s: Release 1.7
Fix Version/s: None

Type: Defect Priority: Minor
Reporter: Nathan Marz Assignee: Unassigned
Resolution: Unresolved Votes: 0
Labels: protocols

Approval: Triaged

 Description   
(defprotocol TestProtocol
  (tester [o]))

(let [t tester]
  (defn another-tester [o]
  	(t o)))

(def another-tester2 tester)

(extend-protocol TestProtocol
  String
  (tester [o] (println "Strings work!")))

(another-tester "A") ;; Error
(another-tester2 "A") ;; Error
(tester "A") ;; Works fine

(let [t tester]
  (defn another-tester [o]
  	(t o)))

(another-tester "A") ;; Works fine

(def another-tester2 tester)

(another-tester2 "A") ;; Works fine

(extend-protocol TestProtocol
  Long
  (tester [o] (println "Longs work!")))

(another-tester "A") ;; Works fine
(another-tester 3) ;; Error
(another-tester2 3) ;; Error


 Comments   
Comment by Nathan Marz [ 08/Aug/15 12:47 PM ]

This issue appears to be Clojure specific – I did some testing in CLJS and was unable to reproduce the issue.

Comment by Ghadi Shayban [ 09/Aug/15 9:51 AM ]

Nathan,
Not sure if you tried this, but using:

(def another-handle #'the-protocol-function)
rather than dereffing outright.

Comment by Nathan Marz [ 09/Aug/15 6:25 PM ]

That's a good workaround but it does seem that my test case should work. I ran into this because I was passing around functions dynamically and saving them for later execution – and this issue popped up with protocol methods. Having to pass around protocol methods differently than regular functions doesn't seem right.

Comment by Kevin Downey [ 10/Aug/15 11:21 AM ]

this is a result of the protocol implementation in clojure, protocol extension mutates the vars, once you have taken then value of the var (which happens once for top level forms) you will not see further mutations of the var so no more protocol extension





[CLJ-1792] array-map and hash-map differ in handling of keys comparing identical but not equal Created: 04/Aug/15  Updated: 04/Aug/15

Status: Open
Project: Clojure
Component/s: None
Affects Version/s: None
Fix Version/s: None

Type: Defect Priority: Minor
Reporter: Nicola Mometto Assignee: Unassigned
Resolution: Unresolved Votes: 0
Labels: None

Attachments: Text File 0001-do-pointer-check-in-Util.EquivPred.patch    
Patch: Code

 Description   
user=> (let [a (array-map Double/NaN 1)] (assoc a (key (first a)) "foo"))
{NaN 1, NaN "foo"}
user=> (let [a (hash-map Double/NaN 1)] (assoc a (key (first a)) "foo"))
{NaN "foo"}

Approach: Array-map's comparison skips identity-check and always delegates to .equals calls. The attached patch alignes array-map behaviour with hash-map by doing the appropriate pointer checks before delegating to .equals






[CLJ-1791] Issue defining a defrecord protocol method named "clear" Created: 04/Aug/15  Updated: 01/Apr/16

Status: Open
Project: Clojure
Component/s: None
Affects Version/s: None
Fix Version/s: None

Type: Enhancement Priority: Minor
Reporter: Mike Anderson Assignee: Unassigned
Resolution: Unresolved Votes: 0
Labels: None


 Description   

There seems to be a problem in trying to define a protocol with a method named "clear"

(defprotocol PClear
(clear [o]))
=> PClear

(defrecord Foo []
PClear
(clear [o] o))
=> CompilerException java.lang.ClassFormatError: Duplicate method name&signature in class file xxxx/Foo, compiling:(NO_SOURCE_PATH:1:1)

I assume this is due to a name conflict with the Java method Collection.clear() in the underlying implementation. However the error is very unclear about this, and the potential for conflict appears to be undocumented as far as I can see.

There seem to be two possible approaches to fixing this:
a) Disallow the use of "clear" as a protocol method name (in which case the error should be more informative, and the rule should be documented)
b) Find a way to support this in the class file format (possibly by overloading on JVM return types, since Collection.clear() returns void??)



 Comments   
Comment by Nicola Mometto [ 04/Aug/15 6:58 AM ]

Mike, the jvm doesn't support return type overloading so your second suggestion is not technically possible.

Reading the doc for defrecord

The class will have implementations of several (clojure.lang)
interfaces generated automatically: IObj (metadata support) and
IPersistentMap, and all of their superinterfaces.

Perharps java.util.Collection (or even better, java.util.Map) should be mentioned here.

Comment by Alex Miller [ 04/Aug/15 7:46 AM ]

I think this should be a doc enhancement request.

Comment by Shogo Ohta [ 08/Aug/15 3:42 AM ]

It might be out of the scope of this ticket, but protocol method conflicts can cause some other kinds of errors:

user=> (defprotocol P1 (finalize [this]))
P1
user=> (defrecord R1 [] P1 (finalize [this]))

CompilerException java.lang.VerifyError: (class: user/R1, method: finalize signature: ()Ljava/lang/Object;) Unable to pop operand off an empty stack, compiling: ...
user=> (defprotocol P2 (wait [this]))
P2
user=> (defrecord R2 [] P2 (wait [this]))
user.R2
user=> (def r (->R2))
#'user/r
user=> (wait r)
CompilerException java.lang.IllegalArgumentException: No single method: wait of interface: user.P2 found for function: wait of protocol: P2, compiling: ...
user=>

IMHO it would be nicer if defprotocol would warn method conflicts with a more informative message.

Comment by Mike Anderson [ 31/Mar/16 7:53 PM ]

@Nicola Mometto : I believe the JVM does in fact support return type overloading:

"Note that there may be more than one matching method in a class because while the Java language forbids a class to declare multiple methods with the same signature but different return types, the Java virtual machine does not. This increased flexibility in the virtual machine can be used to implement various language features. For example, covariant returns can be implemented with bridge methods; the bridge method and the method being overridden would have the same signature but different return types."

See : http://docs.oracle.com/javase/8/docs/api/java/lang/Class.html#getMethod-java.lang.String-java.lang.Class...-

Comment by Nicola Mometto [ 01/Apr/16 3:55 AM ]

Ah, yes of course, thanks.





[CLJ-1789] Use transients with select-keys if possible Created: 28/Jul/15  Updated: 04/Oct/17

Status: Open
Project: Clojure
Component/s: None
Affects Version/s: None
Fix Version/s: None

Type: Enhancement Priority: Minor
Reporter: Alex Miller Assignee: Unassigned
Resolution: Unresolved Votes: 1
Labels: performance

Attachments: Text File 0001-CLJ-1789-Use-transients-and-reduce-with-select-keys.patch    
Patch: Code

 Description   

Problem: Currently select-keys uses conj to add entries. If the map is editable, conj! could be used instead to improve select-keys performance.

Additionally keyseq is traversed as a seq but could be traversed via reduce instead, which might be faster.

Approach 1: Use a transient map and conj!, keeping loop/recur
Approach 2: Reimplement select-keys to use reduce instead of loop/recur
Approach 3: Combine approach one and two

selected key size loop/recur transient reduce transient + reduce
1 243 ns 256 ns 161 ns 188 ns
7 1.1 ms
885 ns 454 ns

From these numbers, approach 3 was chosen.

Note: In order to implement select-keys in terms of reduce, select-keys needed to be moved until after the definition of reduce. This forced a (declare select-keys) since it's used before the definition of reduce

Patch: 0001-CLJ-1789-Use-transients-and-reduce-with-select-keys.patch



 Comments   
Comment by Erik Assum [ 28/Sep/17 2:28 PM ]

Standard Clojure select-keys:

(bench (clojure.core/select-keys {:a "b" :c "d"} [:a]))
Evaluation count : 246382440 in 60 samples of 4106374 calls.
             Execution time mean : 243.245536 ns
    Execution time std-deviation : 2.714803 ns
   Execution time lower quantile : 238.473675 ns ( 2.5%)
   Execution time upper quantile : 248.544255 ns (97.5%)
                   Overhead used : 1.845047 ns

With transients:

(bench (select-keys {:a "b" :c "d"} [:a]))
Evaluation count : 232727220 in 60 samples of 3878787 calls.
             Execution time mean : 256.937568 ns
    Execution time std-deviation : 10.025123 ns
   Execution time lower quantile : 249.951872 ns ( 2.5%)
   Execution time upper quantile : 276.251590 ns (97.5%)
                   Overhead used : 1.845047 ns

Found 5 outliers in 60 samples (8.3333 %)
	low-severe	 3 (5.0000 %)
	low-mild	 2 (3.3333 %)
 Variance from outliers : 25.4503 % Variance is moderately inflated by outliers

With reduce

(bench (select-keys {:a "b" :c "d"} [:a]))
Evaluation count : 364807860 in 60 samples of 6080131 calls.
             Execution time mean : 161.582833 ns
    Execution time std-deviation : 2.212659 ns
   Execution time lower quantile : 158.027524 ns ( 2.5%)
   Execution time upper quantile : 167.673682 ns (97.5%)
                   Overhead used : 1.845047 ns

Found 3 outliers in 60 samples (5.0000 %)
	low-severe	 3 (5.0000 %)
 Variance from outliers : 1.6389 % Variance is slightly inflated by outliers

Reduce + transient

(bench (select-keys {:a "b" :c "d"} [:a]))
Evaluation count : 318075720 in 60 samples of 5301262 calls.
             Execution time mean : 188.656164 ns
    Execution time std-deviation : 3.024952 ns
   Execution time lower quantile : 183.867285 ns ( 2.5%)
   Execution time upper quantile : 195.466784 ns (97.5%)
                   Overhead used : 1.845047 ns

Found 4 outliers in 60 samples (6.6667 %)
	low-severe	 4 (6.6667 %)
 Variance from outliers : 1.6389 % Variance is slightly inflated by outliers

On bigger map/selection

(bench (clojure.core/select-keys {:a "b" :c "d" :b "b" :d "d" :e "e" :f "f" :g "g"} [:a :c :b :d :e :f :g]))
Evaluation count : 56147160 in 60 samples of 935786 calls.
             Execution time mean : 1.104653 µs
    Execution time std-deviation : 36.366516 ns
   Execution time lower quantile : 1.048257 µs ( 2.5%)
   Execution time upper quantile : 1.142031 µs (97.5%)
                   Overhead used : 1.845047 ns

Found 5 outliers in 60 samples (8.3333 %)
	low-severe	 4 (6.6667 %)
	low-mild	 1 (1.6667 %)
 Variance from outliers : 19.0389 % Variance is moderately inflated by outliers

reduce

(bench (select-keys {:a "b" :c "d" :b "b" :d "d" :e "e" :f "f" :g "g"} [:a :c :b :d :e :f :g]))
Evaluation count : 67723500 in 60 samples of 1128725 calls.
             Execution time mean : 885.840664 ns
    Execution time std-deviation : 11.503115 ns
   Execution time lower quantile : 864.403495 ns ( 2.5%)
   Execution time upper quantile : 905.721942 ns (97.5%)
                   Overhead used : 1.845047 ns

Transient + reduce

(bench (select-keys {:a "b" :c "d" :b "b" :d "d" :e "e" :f "f" :g "g"} [:a :c :b :d :e :f :g]))
Evaluation count : 134119380 in 60 samples of 2235323 calls.
             Execution time mean : 454.587795 ns
    Execution time std-deviation : 15.681611 ns
   Execution time lower quantile : 439.822498 ns ( 2.5%)
   Execution time upper quantile : 485.797378 ns (97.5%)
                   Overhead used : 1.845047 ns

Found 3 outliers in 60 samples (5.0000 %)
	low-severe	 3 (5.0000 %)
 Variance from outliers : 20.6393 % Variance is moderately inflated by outliers

The attached patch is using both transients and reduce

Comment by Alex Miller [ 04/Oct/17 8:54 AM ]

The proposed approach seems good to me. The description needs to reflect what was considered and chosen better.





[CLJ-1776] Test that collections are valid statements Created: 08/Jul/15  Updated: 10/Oct/17

Status: Open
Project: Clojure
Component/s: None
Affects Version/s: None
Fix Version/s: None

Type: Enhancement Priority: Minor
Reporter: Michael Blume Assignee: Michael Blume
Resolution: Unresolved Votes: 0
Labels: compiler, test

Attachments: Text File clj-1776-v1.patch     Text File CLJ-1776-v2.patch    
Patch: Code and Test

 Description   

It's possible to break the compiler such that vectors are emitted incorrectly when they're in statement position (I accidentally did this). This doesn't break any part of the Clojure test suite, but does break valid Clojure code (for me it hit taoensso's encore). Add tests to the test suite so defects of this kind are caught.



 Comments   
Comment by Daniel Compton [ 09/Feb/17 8:34 PM ]

Is there a compiler change required as well here?

Comment by Alex Miller [ 09/Feb/17 10:56 PM ]

I think he is saying that the compiler is fine but that if it weren't, no test would tell you that.

Comment by Michael Blume [ 09/Oct/17 6:30 PM ]

Yes, that.

Comment by Andy Fingerhut [ 09/Oct/17 10:21 PM ]

Is it an accident that the Clojure compiler accepts 'code' like this? If I saw Clojure code like this in a project, I think I would assume it was an error.

Comment by Alex Miller [ 09/Oct/17 10:24 PM ]

Seems like it should be valid code to me (although there is not much reason to ever do this unless it's the last expression in a body).

Comment by Andy Fingerhut [ 09/Oct/17 10:35 PM ]

[facepalm] Yes, of course this should be valid in return position, for returning such a vector or map. Away from return position is the odd-looking thing, but as you say, no reason the compiler should reject it.

Comment by Alex Miller [ 10/Oct/17 8:43 AM ]

Note that the elements of a collection will be evaluated, so side effects triggered by code in the collection will be executed.

user=> (let [] 
         [(future (println "hi")) (future (println "there"))] 
         1)
hi
there
1




[CLJ-1768] quote of an empty lazyseq produces an error when evaled Created: 24/Jun/15  Updated: 26/Apr/16

Status: Open
Project: Clojure
Component/s: None
Affects Version/s: Release 1.7
Fix Version/s: None

Type: Defect Priority: Minor
Reporter: Tim Engler Assignee: Unassigned
Resolution: Unresolved Votes: 0
Labels: None

Approval: Triaged

 Description   
user=> (eval `'())
()
user=> `'~(map identity ())
(quote ())
user=> (eval `'~(map identity ()))    ;; expected: ()
CompilerException java.lang.UnsupportedOperationException: Unknown Collection type, compiling:(NO_SOURCE_PATH:5:1)
user=> (prn *e)
#error {
 :cause "Unknown Collection type"
 :via
 [{:type clojure.lang.Compiler$CompilerException
   :message "java.lang.UnsupportedOperationException: Unknown Collection type, compiling:(NO_SOURCE_PATH:5:1)"
   :at [clojure.lang.Compiler analyzeSeq "Compiler.java" 6730]}
  {:type java.lang.UnsupportedOperationException
   :message "Unknown Collection type"
   :at [clojure.lang.Compiler$EmptyExpr emit "Compiler.java" 2929]}]
 :trace
 [[clojure.lang.Compiler$EmptyExpr emit "Compiler.java" 2929]
  [clojure.lang.Compiler$BodyExpr emit "Compiler.java" 5905]
  [clojure.lang.Compiler$FnMethod doEmit "Compiler.java" 5453]
  [clojure.lang.Compiler$FnMethod emit "Compiler.java" 5311]
  [clojure.lang.Compiler$FnExpr emitMethods "Compiler.java" 3843]
  [clojure.lang.Compiler$ObjExpr compile "Compiler.java" 4489]
  [clojure.lang.Compiler$FnExpr parse "Compiler.java" 3983]
  [clojure.lang.Compiler analyzeSeq "Compiler.java" 6721]
  [clojure.lang.Compiler analyze "Compiler.java" 6524]
  [clojure.lang.Compiler eval "Compiler.java" 6779]
  [clojure.lang.Compiler eval "Compiler.java" 6745]
  [clojure.core$eval invoke "core.clj" 3081]
  ;; elided rest
nil
user=> (eval `'~(map identity '(x)))
(x)

Cause: In the empty list case, the compiler here sees a LazySeq. I suspect something earlier in the stack should be producing an empty list instead, but haven't tracked it back yet.



 Comments   
Comment by Tim Engler [ 26/Apr/16 4:17 AM ]

Still exists in clojure 1.8





[CLJ-1767] Documentation Issues with sort Created: 22/Jun/15  Updated: 20/Aug/17

Status: Open
Project: Clojure
Component/s: None
Affects Version/s: Release 1.6
Fix Version/s: None

Type: Defect Priority: Minor
Reporter: Marc O'Morain Assignee: Unassigned
Resolution: Unresolved Votes: 0
Labels: docstring


 Description   

The documentation for sort seems to be incomplete:

  • sort can return nil in some situations. There is a discussion in CTYP-228 with more backstory. There is a repro case here: http://sprunge.us/VIFc?clj supplied by Nicola Mometto. The doc string states that sort "Returns a sorted sequence of the items in coll.", which does not indicate that sort can return nil.
  • It is stated that the "comparator must implement java.util.Comparator.", but this is not true - the comparator can be any IFn that can accept 2 arguments and return either a Boolean or a Number.

For the first issue (nil return), changing the implementation to never return nil might be the neatest fix. For the second issue, the docstring could reference a description of how what functions are valid comparison functions, which can be referenced by other functions that also accept comparators (e.g., sort-by, sorted-set-by, sorted-map-by).



 Comments   
Comment by Alex Miller [ 22/Jun/15 7:51 AM ]

The first issue is being covered by CLJ-1763, so I would remove it from the ticket.

The second is technically true - Arrays.sort() will invoked and it takes a Comparator. The tricky bit is that AFn base class implements Comparator so all function implementations that extend from it that support a 2-arg function satisfy this constraint. But it might be helpful to call that out better.

Comment by Marc O'Morain [ 19/Aug/17 3:36 PM ]

I was thinking that a neat fix here would be to just state that sort takes a comparator function, and let the user refer to the reference manual for details of comparator functions. I could not find anything in the reference manual, but the Clojure guide has a good description of comparators - https://clojure.org/guides/comparators. I'm thinking of writing a small entry for the reference manual to document comparators - how does that sound?

Comment by Alex Miller [ 20/Aug/17 4:56 PM ]

This issue should be rewritten to remove the first issue. For the second one, I think it would be ok to add a clause/sentence to cover that a comparator can be a 2-arg function that returns either a boolean or a number.

If you want to update the web site, you should do that as an issue or PR on https://github.com/clojure/clojure-site.





[CLJ-1764] partition-by runs infinite loop when one element of infinite partition is accessed Created: 19/Jun/15  Updated: 01/Feb/16

Status: Open
Project: Clojure
Component/s: None
Affects Version/s: Release 1.7
Fix Version/s: None

Type: Defect Priority: Minor
Reporter: Leon Grapenthin Assignee: Unassigned
Resolution: Unresolved Votes: 0
Labels: None

Attachments: Text File clj-1764.patch    
Patch: Code and Test
Approval: Prescreened

 Description   
(def x (partition-by zero? (range)))
(first x)
;-> (0)
(first (second x))
;-> infinite loop

The reason is that partition-by counts and thus realizes each current partition to call itself recursively.

It seems like unexpected behavior, since the user may not intend to realize the entire partition.

Approach: Change from using seq to lazy-seq in its last line

Patch: clj-1764.patch

Screened-by: Alex Miller - I did a perf check too and seemed to be about the same, possibly even faster on average (gc effects mean there is a lot of deviation).

(def r (range 100000))
(dotimes [_ 20]
  (time (count (partition-by odd? r))))


 Comments   
Comment by Matthew Gilliard [ 23/Jul/15 2:36 PM ]

Patch as suggested by Leon, + test.





[CLJ-1763] clojure.core/sort is not thread-safe on Java collections with backing arrays Created: 19/Jun/15  Updated: 30/Jul/15

Status: Open
Project: Clojure
Component/s: None
Affects Version/s: None
Fix Version/s: None

Type: Enhancement Priority: Minor
Reporter: Nicola Mometto Assignee: Unassigned
Resolution: Unresolved Votes: 0
Labels: collections

Attachments: Text File 0001-CLJ-1763-make-sort-thread-safe.patch    
Patch: Code
Approval: Triaged

 Description   

If a (mutable) Java collection that exposes it's backing array is passed to c.c/sort in multiple threads, the collection will be concurrently modified in multiple threads.

user=> (def q (java.util.concurrent.ArrayBlockingQueue. 1))
#'user/q
user=> (future (loop [] (.add q 1) (.remove q 1) (recur)))
#object[clojure.core$future_call$reify__4393 0x4769b07b {:status :pending, :val nil}]
user=> (take 3 (distinct (repeatedly #(sort q))))
((1) () nil)

Approach: Convert coll to a seq before converting it to an array, thus preserving the original collection.

Patch: 0001-CLJ-1763-make-sort-thread-safe.patch

Alternate approaches:

1. Document in sort that, like Java arrays, Java collections backed by arrays are modified in-place.
2. Change RT.toArray() to defensively copy the array returned from a (non-IPersistentCollection) Java collection. This has a number of potential ramifications as this method is called from several paths.
3. For non-Clojure collections, could also use Collections.sort() instead of dumping to array and using Arrays.sort().



 Comments   
Comment by Alex Miller [ 19/Jun/15 10:55 AM ]

The docstring says "If coll is a Java array, it will be modified. To avoid this, sort a copy of the array." which also seems like solid advice in this case.

Creating a sequence view of the input collection would significantly alter the performance characteristics.

Comment by Alex Miller [ 19/Jun/15 10:59 AM ]

I guess what I'm saying is, we should not make the performance worse for persistent collections in order to make it safer for arbitrary Java collections. But it may still be useful to make it safer without affecting persistent collection performance and/or updating the docstring.

Comment by Nicola Mometto [ 19/Jun/15 11:02 AM ]

Alex, no additional sequence is being created, the seq call was already there

Comment by Alex Miller [ 19/Jun/15 11:53 AM ]

Well, that's kind of true. The former use did not force realization of the whole seq, just the first element. That said, from a quick test the extra cost seems small on a set (vector seqs are actually faster due to their structure).

Comment by Stuart Halloway [ 30/Jul/15 9:11 AM ]

I think this should be a docstring change, if anything at all.





[CLJ-1762] Implement IKVReduce for java.util.map Created: 18/Jun/15  Updated: 18/Jun/15

Status: Open
Project: Clojure
Component/s: None
Affects Version/s: Release 1.8
Fix Version/s: None

Type: Enhancement Priority: Minor
Reporter: Chen Guo Assignee: Unassigned
Resolution: Unresolved Votes: 1
Labels: interop, performance

Attachments: Text File reduce-kv-java-map.patch    
Patch: Code
Approval: Triaged

 Description   

reduce works on Java maps, but reduce-kv does not:

user=> (def clojure-map {1 2 3 4})
#'user/clojure-map
user=> (def java-map (java.util.HashMap. {1 2 3 4}))
#'user/java-map
user=> (reduce (fn [sum [k v]] (+ sum k v)) 0 java-map)
10
user=> (reduce-kv + 0 clojure-map)
10
user=> (reduce-kv + 0 java-map)

IllegalArgumentException No implementation of method: :kv-reduce of protocol: #'clojure.core.protocols/IKVReduce found for class: java.util.HashMap  clojure.core/-cache-protocol-fn\
 (core_deftype.clj:544)

It's trivial to destructure arguments in a regular reduce, but there are performance implications. The following example yields a 7x speed up when run with the implementation of reduce-kv for java.util.Map as implemented in this patch:

user=> (def big-clojure-map (into {} (map #(vector % %) (range 10000))))
#'user/big-clojure-map
user=> (def big-java-map (java.util.HashMap. big-clojure-map))
Reflection warning, /tmp/form-init7130245387362554027.clj:1:19 - call to java.util.HashMap ctor can't be resolved.
#'user/big-java-map
user=> (defn reduce-sum [m] (reduce (fn [sum [k v]] (+ sum k v)) 0 m))
#'user/reduce-sum
user=> (defn reduce-sum-kv [m] (reduce-kv (fn [sum k v] (+ sum k v)) 0 m))
#'user/reduce-sum-kv
user=> (time (dotimes [_ 1000] (reduce-sum big-java-map)))
"Elapsed time: 2624.692113 msecs"
nil
user=> (time (dotimes [_ 1000] (reduce-sum-kv big-java-map)))
"Elapsed time: 376.802454 msecs"
nil





[CLJ-1752] realized? return true for an instance that is not IPending Created: 09/Jun/15  Updated: 09/Jun/15

Status: Open
Project: Clojure
Component/s: None
Affects Version/s: Release 1.7
Fix Version/s: None

Type: Enhancement Priority: Minor
Reporter: Logan Linn Assignee: Unassigned
Resolution: Unresolved Votes: 1
Labels: None

Approval: Triaged

 Description   

To safely test if an arbitrary seq is realized (non-lazy), we need a wrapper like:

(defn seq-realized? [s]
  (if (instance? clojure.lang.IPending s)
    (realized? s)
    true))

If realized? returned true for an (ISeq?) instance that is not IPending there would be less surprising behavior for cases such as, (realized? (range 10)) which throws exception.

NB: A follow-up to CLJ-1751.






[CLJ-1748] Change clojure.core/reverse to return rseq for args that are Reversible Created: 07/Jun/15  Updated: 07/Jun/15

Status: Open
Project: Clojure
Component/s: None
Affects Version/s: None
Fix Version/s: None

Type: Enhancement Priority: Minor
Reporter: Andy Fingerhut Assignee: Unassigned
Resolution: Unresolved Votes: 0
Labels: None


 Description   

There may be issues with this suggestion about concrete types of return values, or doc strings that promise things that you want to preserve that cannot be preserved with this suggested change.

However, if those issues are not show stoppers, changing clojure.core/reverse to check if its arg is Reversible, and if so, return rseq, else do as reverse does today, could be faster in many situations.






[CLJ-1747] eduction's printer requires/promises that its source collection is Iterable Created: 07/Jun/15  Updated: 12/Apr/17

Status: Open
Project: Clojure
Component/s: None
Affects Version/s: Release 1.7
Fix Version/s: None

Type: Defect Priority: Minor
Reporter: Ghadi Shayban Assignee: Unassigned
Resolution: Unresolved Votes: 1
Labels: None

Attachments: Text File CLJ-1747-eduction-print.patch    
Patch: Code

 Description   

eduction expects its source collection to be Iterable [1], and its print-method goes through print-sequential [2]. This implies a promise that may restrict the use-case of an eduction over a virtual collection, e.g. a pure IReduceInit source that may be backed by I/O or some other resource. I have found it useful to construct these I/O reducibles and wrap them with an eduction. But when interacting at the REPL, printing out an eduction-wrapped IReduceInit will fail. Does the print-method impl for eduction require too much? This is a only minor inconvenience more than anything else, obviously I could create my own flavor of eduction.

Totally hypothetical example:

(defn database-index
  [name]
  (reify clojure.lang.IReduceInit
    (reduce [_ f init]
      (with-open [rdr (fressian/create-reader (io/input-stream name))]
       (loop [] ...reduce impl...)))))

(eduction (filter (as-of #inst "2012-01-01")) (database-index "eavt.fress"))
;; ^ throws when printed by repl

[1] https://github.com/clojure/clojure/blob/master/src/clj/clojure/core.clj#L7336-L7338
[2] https://github.com/clojure/clojure/blob/master/src/clj/clojure/core.clj#L7360

Proposed Approach:
Since we have #object tagged literals, let eduction print like an opaque #object






[CLJ-1746] new keyword for `require` that both refers other namespace's symbol and exclude the same in clojure.core Created: 06/Jun/15  Updated: 15/May/17

Status: Open
Project: Clojure
Component/s: None
Affects Version/s: None
Fix Version/s: None

Type: Feature Priority: Minor
Reporter: Hoang Minh Thang Assignee: Unassigned
Resolution: Unresolved Votes: 1
Labels: enhancement


 Description   

I find myself repeat code like this

(ns foo.bar
(:refer-clojure :exclude [doseq])
(:require [clojure.core.typed :refer [doseq]]))

and just think why not something like:

(ns foo.bar
(:require [clojure.core.typed :override [doseq]]))



 Comments   
Comment by Mike Anderson [ 09/Jun/15 10:40 AM ]

I agree this is very annoying.

I think it is a duplicate of my issue though: The patch for CLJ-1257 would make this unnecessary (it would allow the user to override any vars, without getting an exception).





[CLJ-1737] Omit java exception class from CompilerException message Created: 23/May/15  Updated: 14/Dec/15

Status: Open
Project: Clojure
Component/s: None
Affects Version/s: Release 1.5, Release 1.6, Release 1.7
Fix Version/s: None

Type: Enhancement Priority: Minor
Reporter: John Hume Assignee: Unassigned
Resolution: Unresolved Votes: 0
Labels: compiler, errormsgs, patch, usability

Attachments: Text File clearer-CompilerException-messase.patch     File compiler_exception_examples.clj    
Patch: Code
Approval: Triaged

 Description   

A CompilerException is always created with a cause exception. Currently the message is built using cause.toString(), which for all examples I've examined is the cause class, followed by a colon, followed by the cause message. In all those examples, the message of the cause is informative, and the class name provides no additional help.

I propose to switch to using cause.getMessage() rather than cause.toString(). This would make it easier for tools to present compiler errors that don't leak implementation details that may confuse a new user. The cause class would still be shown in the stack trace.

Here are the examples I looked at, with the output from before the attached patch:

Example source '(ns foo)

(def'
Exception message:
 java.lang.RuntimeException: EOF while reading, starting at line 3, compiling:(/Users/jhume/Projects/clojure/clojure/temp.clj:3:1)

Example source ':foo}'
Exception message:
 java.lang.RuntimeException: Unmatched delimiter: }, compiling:(/Users/jhume/Projects/clojure/clojure/temp.clj:1:6)

Example source 'foo'
Exception message:
 java.lang.RuntimeException: Unable to resolve symbol: foo in this context, compiling:(/Users/jhume/Projects/clojure/clojure/temp.clj:14:1)

Example source 'clojure.core/firstt'
Exception message:
 java.lang.RuntimeException: No such var: clojure.core/firstt, compiling:(/Users/jhume/Projects/clojure/clojure/temp.clj:15:1)

Example source '(nil 1)'
Exception message:
 java.lang.IllegalArgumentException: Can't call nil, compiling:(/Users/jhume/Projects/clojure/clojure/temp.clj:1:1)

Example source '("hi" 1)'
Exception message:
 java.lang.ClassCastException: java.lang.String cannot be cast to clojure.lang.IFn, compiling:(/Users/jhume/Projects/clojure/clojure/temp.clj:1:1)

Example source '{:foo}'
Exception message:
 java.lang.RuntimeException: Map literal must contain an even number of forms, compiling:(/Users/jhume/Projects/clojure/clojure/temp.clj:1:7)

Example source '1st'
Exception message:
 java.lang.NumberFormatException: Invalid number: 1st, compiling:(/Users/jhume/Projects/clojure/clojure/temp.clj:1:1)

And the output with the attached patch applied:

Example source '(ns foo)

(def'
Exception message:
 EOF while reading, starting at line 3, compiling:(/Users/jhume/Projects/clojure/clojure/temp.clj:3:1)

Example source ':foo}'
Exception message:
 Unmatched delimiter: }, compiling:(/Users/jhume/Projects/clojure/clojure/temp.clj:1:6)

Example source 'foo'
Exception message:
 Unable to resolve symbol: foo in this context, compiling:(/Users/jhume/Projects/clojure/clojure/temp.clj:14:1)

Example source 'clojure.core/firstt'
Exception message:
 No such var: clojure.core/firstt, compiling:(/Users/jhume/Projects/clojure/clojure/temp.clj:15:1)

Example source '(nil 1)'
Exception message:
 Can't call nil, compiling:(/Users/jhume/Projects/clojure/clojure/temp.clj:1:1)

Example source '("hi" 1)'
Exception message:
 java.lang.String cannot be cast to clojure.lang.IFn, compiling:(/Users/jhume/Projects/clojure/clojure/temp.clj:1:1)

Example source '{:foo}'
Exception message:
 Map literal must contain an even number of forms, compiling:(/Users/jhume/Projects/clojure/clojure/temp.clj:1:7)

Example source '1st'
Exception message:
 Invalid number: 1st, compiling:(/Users/jhume/Projects/clojure/clojure/temp.clj:1:1)





[CLJ-1734] Display more descriptive error message when trying to use reader conditionals in a non-cljc file Created: 19/May/15  Updated: 06/Oct/16

Status: Open
Project: Clojure
Component/s: None
Affects Version/s: Release 1.7
Fix Version/s: None

Type: Enhancement Priority: Minor
Reporter: Daniel Compton Assignee: Unassigned
Resolution: Unresolved Votes: 3
Labels: errormsgs, reader


 Description   

I spent a few puzzled minutes trying to understand the following message from the Clojure compiler:

CompilerException java.lang.RuntimeException: Conditional read not allowed, compiling: <filename>

Eventually I realised it was because I was trying to use reader conditionals in a .clj file that I hadn't renamed to cljc. I think it would be really helpful for people working in mixed clj and cljc codebases to have this error message extended to something like:

"Conditional read not allowed because file does not have extension .cljc"



 Comments   
Comment by Alex Miller [ 19/May/15 11:45 PM ]

The reader doesn't know this - it can be called in multiple ways (from repl, via clojure.core/read, via clojure.core/read-string, load/compile .cljc, load/compile .clj) so that description would actually be wrong in some of those. It seems like you're getting a pretty good error message already - it told you the problem and gave you the file name.

The message could be tweaked to something like "Reader conditionals not allowed in this context" which might give you a better clue.

Comment by Daniel Compton [ 20/May/15 3:12 PM ]

Perhaps I'm not understanding how the reader determines whether reader conditionals are allowed or not, but those would all seem to have different reasons for not being allowed and would be caught by different checks. Each of these checks could give a more specific warning explaining why the read wasn't allowed?

Counteracting my point, it looks like there is only one place where this exception is thrown - https://github.com/clojure/clojure/blob/7b9c61d83304ff9d5f9feddecf23e620c0b33c6e/src/jvm/clojure/lang/LispReader.java#L1406. I'm not sure if this could be extended to give more details in different error cases or if that information isn't available at that point?

Comment by Alex Miller [ 20/May/15 3:36 PM ]

The reader is invoked with an options map which will (or will not) have {:read-cond :allow} or {:read-cond :preserve}. That's the only info the reader has - if either of those is set and a reader conditional is encountered, it throws.

The compiler decides how to initialize these options when it's calling the reader. Users of read and read-string similarly decide which options are allowed when they call it. It would be possible to pass more info into the reader or to catch and rethrow in the compiler where more context is known, but both of those complicate the code for what is already a decent error imho.

Comment by Daniel Compton [ 20/May/15 4:03 PM ]

I agree it is a reasonable error message, I guess we can wait and see how other people find it once 1.7 is released. If it turns out to be an issue for lots of other people then we could revisit this then?

Comment by Marshall Abrams [ 06/Oct/16 10:09 AM ]

I encountered the problem with using reader conditionals in a .clj file. I would have had no idea of how to fix the error meant except for Daniel Compton's blog post about it. I understand that it's not just a filename issue, and that there can be other sources of this error, but when it is a filename issue, it's very mysterious, and this seems like an easy mistake to make. A contributing context is that Clojurescript macros can be defined in a .clj file, so it can be natural to use that extension as a reminder about when the macros will be compiled.

I do think that something like "Reader conditionals not allowed ..." would be a lot easier to understand than "Conditional read not allowed". That would at least make it obvious that it's the reader conditional that's causing the problem. Once you know that this is what "conditional read" means, it's obvious, but "conditional read" is not a common phrase in the Clojure world, afaik, and I wondered why it didn't say "reader conditional" if that's what the message was about.

(An option would be to add a comment in the error string stating that the cause might be reader conditionals used outside of a .cljc file. Not sure if you want to go down that road of "might be"/"maybe" in error messages, since that kind of thing could get out of hand and lead to long error messages. However, this sort of thing has been done elsewhere, and it might be very helpful in this case; mistakenly using .clj with a reader conditional seems like something that would occur often.)





[CLJ-1733] print-dup form unreadable for sorted sets and maps Created: 19/May/15  Updated: 06/Jul/16

Status: Reopened
Project: Clojure
Component/s: None
Affects Version/s: Release 1.6, Release 1.7
Fix Version/s: None

Type: Defect Priority: Minor
Reporter: Nikita Prokopov Assignee: Unassigned
Resolution: Unresolved Votes: 0
Labels: None
Environment:

Clojure 1.6.0
Clojure 1.7.0-alpha5
Clojure 1.7.0-beta3

java version "1.8.0"
Java(TM) SE Runtime Environment (build 1.8.0-b132)
Java HotSpot(TM) 64-Bit Server VM (build 25.0-b70, mixed mode)


Attachments: Text File clj-1733-tagged-literals-throw-on-sorted-set.patch    
Patch: Code and Test
Approval: Triaged

 Description   

print-dup for sorted sets and maps presume a nonexistent static create method that takes an IPersistentCollection

Printing

user=> (print-dup (sorted-set 1) *out*)
#=(clojure.lang.PersistentTreeSet/create [1])

Can't read back

(read-string "#=(clojure.lang.PersistentTreeSet/create [1])")
ClassCastException Cannot cast clojure.lang.PersistentVector to clojure.lang.ISeq  java.lang.Class.cast (Class.java:3356)

Possible Fixes

  • add create methods taking IPersistentVector to collections
  • emit something different from print-dup


 Comments   
Comment by Alex Miller [ 19/May/15 4:55 PM ]

It's trying to invoke PersistentTreeSet.create(ISeq) with ["123"]. It's not clear to me where the vector comes from?

Comment by Nikita Prokopov [ 19/May/15 5:04 PM ]

It’s a particular case of CLJ-1461. Vector comes from reading output of print-dup:

(defrecord Rec [f])

(binding [*print-dup* true]
  (prn (Rec. (sorted-set 1))))
;; => #tonsky.Rec[#=(clojure.lang.PersistentTreeSet/create [1])]

I already have a patch for PersistentTreeSet (attached here). Can look into CLJ-1461 later.

Comment by Mike Rodriguez [ 05/Jul/16 11:29 PM ]

This won't work for sorted sets (or maps) that are defined with a custom Comparator though via fn's like sorted-set-by etc. I think the round-trip print to read result would then be confusing and incorrect right?

Even more troublesome to me here is that I see no clear way to make print-dup capable of handling the case of a custom Comparator correctly. Arbitrary functions are black boxes and we have no generally, effective way to print-dup them (based on my research I assume this to be correct). We can always make special wrapped fn's for that, but again, not general.





[CLJ-1732] Add docstring explanation of (isa? [x1 x2...] [parent1 parent2...]) Created: 17/May/15  Updated: 17/May/15

Status: Open
Project: Clojure
Component/s: None
Affects Version/s: Release 1.6
Fix Version/s: None

Type: Enhancement Priority: Minor
Reporter: Phill Wolf Assignee: Unassigned
Resolution: Unresolved Votes: 0
Labels: docstring

Approval: Triaged

 Description   

The "Multimethods and Hierarchies" page mentions that "isa?" has special behavior when aimed at two vectors[1]. But the docstring of "isa?" does not mention it[2].

[1] http://clojure.org/multimethods
[2] http://clojure.github.io/clojure/clojure.core-api.html#clojure.core/isa?






[CLJ-1715] Use AFn.applyToHelper rather than IFn.applyTo in InvokeExpr.eval Created: 23/Apr/15  Updated: 23/Apr/15

Status: Open
Project: Clojure
Component/s: None
Affects Version/s: None
Fix Version/s: None

Type: Enhancement Priority: Minor
Reporter: Nicola Mometto Assignee: Unassigned
Resolution: Unresolved Votes: 0
Labels: None

Attachments: Text File 0001-CLJ-1715-Use-AFn.applyToHelper-rather-than-IFn.apply.patch    
Patch: Code

 Description   

Because of implementation details of how def forms are compiled, invokations in its args are evaluated using applyTo rather than directly using the invoke method. This forces IFn implementors to implement applyTo even when not necessary.

The proposed patch changes InvokeExpr.eval to use AFn.applyToHelper, so that invoke rather than applyTo is used when possible.

Example of currently failing code that will work with patch:

user=> (deftype x [] clojure.lang.IFn (invoke [_] 1))
user.x
user=> (def a ((x.)))
AbstractMethodError   clojure.lang.Compiler$InvokeExpr.eval (Compiler.java:3553)





[CLJ-1708] Volatile mutable in deftype is not settable when using try..finally and returning this Created: 17/Apr/15  Updated: 31/Jul/15

Status: Open
Project: Clojure
Component/s: None
Affects Version/s: Release 1.6, Release 1.7
Fix Version/s: None

Type: Defect Priority: Minor
Reporter: Patrick Gombert Assignee: Unassigned
Resolution: Unresolved Votes: 0
Labels: compiler, deftype
Environment:

clojure 1.6.0, clojure 1.7.0-beta1


Approval: Triaged

 Description   

Reproducible Code: https://gist.github.com/patrickgombert/1bcb8a051aeb3e82d855

When using a volatile-mutable field in deftype, compilation fails if the field is set! in a method call that uses both try..finally and returns itself from the method call. Leaving out either the try..finally or returning itself from the method causes compilation to succeed.

Expected behavior: set! should set the volatile-mutable variable and compilation should succeed.



 Comments   
Comment by Kevin Downey [ 17/Apr/15 7:15 PM ]

this must be the same issue as CLJ-1422 and CLJ-701, it has nothing to do with returning `this`, but with the try being in a tail position or not. if the try is not in a tail position the compiler hoists it out in to a thunk. effectively the code is

(deftype SomeType [^:volatile-mutable foo]
  SomeProtocol
  (someFn [_] ((fn [] (try (set! foo 1))))))

which the compiler also rejects, because it doesn't let you mutate fields from functions that are not the immediate protocol functions





[CLJ-1704] Clarify cond documentation to explain about using :else Created: 14/Apr/15  Updated: 14/Apr/15

Status: Open
Project: Clojure
Component/s: None
Affects Version/s: Release 1.6
Fix Version/s: None

Type: Enhancement Priority: Minor
Reporter: Daniel Compton Assignee: Unassigned
Resolution: Unresolved Votes: 0
Labels: docstring


 Description   

The documentation for cond doesn't explicitly mention that you can use :else (or any other keyword) to catch any values that don't match the previous conditions. While it is true that the documentation does say that a test will evaluate and return the value of logical true, it could be more helpful by pointing out that a keyword like :else will always be logical true.

I'm not 100% sure about whether this is necessary, but wanted to see what others thought and whether it would be helpful or not.






[CLJ-1693] into: merge metadata Created: 03/Apr/15  Updated: 03/Apr/15

Status: Open
Project: Clojure
Component/s: None
Affects Version/s: Release 1.7
Fix Version/s: None

Type: Enhancement Priority: Minor
Reporter: Gregg Reynolds Assignee: Unassigned
Resolution: Unresolved Votes: 0
Labels: function, interop
Environment:

all


Approval: Triaged

 Description   

Currently (into to from) preserves to's metadata but discards from's metadata. The enhancement would be to have 'into' do something like (merge (meta to) (meta from)). Justification: as with data, so with metadata. Use case: Using deftype, I have a class EntityMap that clojurizes a native Java class (App Engine's Entity class), making it behave just like a clojure map. This includes using into to convert an EntityMap to an ordinary PersistentMap; the problem is that key information for the EntityMap is really metadata, so I need (into {} em) to put that metadata into the new PersistentMap.

See also CLJ-916






[CLJ-1688] Object instance members should resolve to Object Created: 30/Mar/15  Updated: 30/Mar/15

Status: Open
Project: Clojure
Component/s: None
Affects Version/s: None
Fix Version/s: None

Type: Enhancement Priority: Minor
Reporter: Michael Blume Assignee: Unassigned
Resolution: Unresolved Votes: 0
Labels: reflection, typehints


 Description   
(defn unparse-pattern ^String [pattern] (.toString pattern))
Reflection warning, ring/swagger/coerce.clj:22:41 - reference to field toString can't be resolved.

Reflection isn't really necessary here, we could just special-case the methods on Object.






[CLJ-1687] Clojure doesn't resolve static calls even when it has all information needed to do so Created: 30/Mar/15  Updated: 30/Mar/15

Status: Open
Project: Clojure
Component/s: None
Affects Version/s: None
Fix Version/s: None

Type: Enhancement Priority: Minor
Reporter: Michael Blume Assignee: Unassigned
Resolution: Unresolved Votes: 0
Labels: reflection, typehints


 Description   

If I create a class with two methods, one of which takes (String, String), and the other taking (String, Number), and then write a function

(defn foo
  [x ^String y]
  (Thing/hello x y))

it seems obvious that I'm trying to call the first method and not the second. But on lein check, clojure prints

Reflection warning, resolve_fail/core.clj:6:3 - call to static method hello on resolve_fail.Thing can't be resolved (argument types: unknown, java.lang.String).

unless I also type-hint x.



 Comments   
Comment by Nicola Mometto [ 30/Mar/15 2:32 PM ]

I have looked at this countless times while working on tools.analyzer and hacking the reflector and found out that there doesn't seem to be a way to make things like this "work" without breaking other cases





[CLJ-1682] clojure.set/intersection occasionally allows non-set arguments. Created: 24/Mar/15  Updated: 14/Jul/15

Status: Open
Project: Clojure
Component/s: None
Affects Version/s: None
Fix Version/s: None

Type: Defect Priority: Minor
Reporter: Valerie Houseman Assignee: Unassigned
Resolution: Unresolved Votes: 1
Labels: checkargs

Approval: Triaged

 Description   

clojure.set/intersection, by intent and documentation, is meant to be operations between two sets. However, it sometimes allows (and returns correct opreations upon) non-set arguments. This confuses the intention that non-set arguments are not to be used.

Here's an example with Set vs. KeySeq:
If there happens to be an intersection, you'll get a result. This may lead someone coding this to think that's okay, or to not notice they've used an incompatible data type. As soon as the intersection is empty, however, an appropriate type error ensues, albeit by accident because the first argument to clojure.core/disj should be a set.

user=> (require '[clojure.set :refer [intersection]])
nil
user=> (intersection #{:key_1 :key_2} (keys {:key_1 "na"}))   ;This works, but shouldn't
(:key_1)
user=> (intersection #{:key_1 :key_2} (keys {:key_3 "na"}))   ;This fails, because intersection assumes the second argument was a Set
ClassCastException clojure.lang.APersistentMap$KeySeq cannot be cast to clojure.lang.IPersistentSet  clojure.core/disj (core.clj:1449)

(disj (keys {:key_1 "na"}) #{:key_1 :key_2})   ;The assumption that intersection made
ClassCastException clojure.lang.APersistentMap$KeySeq cannot be cast to clojure.lang.IPersistentSet  clojure.core/disj (core.clj:1449)

Enforcing type security on a library that's clearly meant for a particular type seems like the responsible thing to do. It prevents buggy code from being unknowingly accepted as correct, until the right data comes along to step on the bear trap.



 Comments   
Comment by Andy Fingerhut [ 24/Mar/15 7:19 PM ]

CLJ-810 was similar, except it was for function clojure.set/difference. That one was declined with the comment "set/difference's behavior is not documented if you don't pass in a set." I do not know what core team will judge ought to be done with this ticket, but wanted to provide some history.

Dynalint [1] and I think perhaps Dire [2] can be used to add dynamic argument checking to core functions.

[1] https://github.com/frenchy64/dynalint
[2] https://github.com/MichaelDrogalis/dire

Comment by Alex Miller [ 24/Mar/15 9:00 PM ]

Now that `set` is faster for sets, I think we could actually add checking for sets in some places where we might not have before. So, it's worth looking at with fresh eyes.

Comment by Jason Wolfe [ 28/May/15 2:54 AM ]

Back in 2009 I submitted a patch to the set functions with explicit `set?` checks and Rich's response was "the fact that these functions happen to work when the second argument is not a set is an implementation artifact and not a promise of the interface, so I'm not in favor of the set? testing or any other accommodation of that." Not sure if that is still accurate though.