<< Back to previous view

[CLJ-2160] LispReader allows no-ops macros to sneak in certain other forms (namespaced maps, tagged literals and anonymous arguments) Created: 25/Apr/17  Updated: 25/Apr/17

Status: Open
Project: Clojure
Component/s: None
Affects Version/s: Release 1.9
Fix Version/s: None

Type: Defect Priority: Minor
Reporter: Christophe Grand Assignee: Unassigned
Resolution: Unresolved Votes: 0
Labels: reader

Attachments: Text File clj2160-2.patch     Text File clj2160.patch    
Patch: Code
Approval: Triaged

 Description   

No-op macros are line comments (starting with #! or ;), #_, reader conditional (splicing or not) with no matching feature.
Furthermore once a no-op macro has been read regular whitespace are allowed anew.

Examples:
Namespaced map #foo{:bar :baz}

#:#_()#! bang bang
#?(:whatever 42); now a blank line

#?@(:default ())foo
{:bar :baz}

Tagged literal #inst "2017-04-24T09:11:29.878-00:00"

##_()#! bang bang
#?(:whatever 42); now a blank line

inst "2017-04-24T09:11:29.878-00:00"

Anonymous argument: #(do %1)

#(do %#_()#! bang bang
#?(:whatever 42); now a blank line

#?@(:default ())1)

In addition anonymous arguments implementation is leaky (any %n is accepted as long as n is (-2.0 -1.0] (mapping to %&) and [1.0 Infinity) and any representation can be used (bigdec or bigint or float or integers in any basis).

#(list %#_(first arg)1.00000001 %#_(secong arg)2r10 %#_(rest arg)-1.5)


 Comments   
Comment by Christophe Grand [ 25/Apr/17 6:31 AM ]

The patch extracts the body from the read loop to expose a readSome method that returns either a form or the reader (if no valued form has been read starting at the current position).
This patch also adds a regex pattern to validate anonymous args.

Comment by Christophe Grand [ 25/Apr/17 7:35 AM ]

clj2160-2 is clj2160 + two redundant checks that were not removed





[CLJ-2159] Disambiguate behavior of def with doc-string Created: 23/Apr/17  Updated: 23/Apr/17

Status: Open
Project: Clojure
Component/s: None
Affects Version/s: Release 1.8, Release 1.9
Fix Version/s: None

Type: Enhancement Priority: Trivial
Reporter: Christopher Brown Assignee: Unassigned
Resolution: Unresolved Votes: 0
Labels: docstring, documentation
Environment:

REPL


Attachments: Text File clarify-def-forms.patch    
Patch: Code

 Description   

As far as I can tell, it's impossible to use `def` to create a var that's unbound but has `:doc` metadata (or to change the `:doc` metadata of an existing var without also binding / changing the bound value).

This change clarifies the possible usages of `def`; i.e., if you supply `doc-string`, you must also supply `init`.






[CLJ-2158] multi-spec retag generator in conflict with user tag spec/gen Created: 22/Apr/17  Updated: 25/Apr/17

Status: Open
Project: Clojure
Component/s: None
Affects Version/s: None
Fix Version/s: None

Type: Defect Priority: Major
Reporter: Leon Grapenthin Assignee: Unassigned
Resolution: Unresolved Votes: 0
Labels: generator, spec
Environment:

JVM


Approval: Triaged

 Description   

Problem: multi-spec does generate retag values on its own from values for which methods have been implemented. However a user can have purposefully speced the retag key differently and multi-spec will generate incompatible values (resulting in a such-that failure). An example is hierarchy dispatch where methods dispatch values aren't necessarily the valid tag values.

Proposed solution: When generating the retag value, multi-spec should first try the existing spec for that key and generate "such-that" it is a possible dispatch value for the multimethod, only generate direclty from the multimethod based mechanism iff there is no spec for the tag key.



 Comments   
Comment by Leon Grapenthin [ 25/Apr/17 3:00 AM ]

Improved proposed solution to cover both "user spec is a subset of dispatch values" and vice versa.





[CLJ-2157] multi-spec doesn't generate possible tags from hierarchy Created: 22/Apr/17  Updated: 22/Apr/17

Status: Open
Project: Clojure
Component/s: None
Affects Version/s: Release 1.9
Fix Version/s: None

Type: Enhancement Priority: Major
Reporter: Leon Grapenthin Assignee: Unassigned
Resolution: Unresolved Votes: 0
Labels: generator, spec


 Description   

Problem: Even though multi-spec supports hierarchy dispatch of multi-methods, its generator only generates tags that have direct method implementations.

Proposed solution: It should also generate from hierarchy.






[CLJ-2156] Document char[] input support in clojure.java.io/copy Created: 22/Apr/17  Updated: 22/Apr/17

Status: Open
Project: Clojure
Component/s: None
Affects Version/s: Release 1.9
Fix Version/s: None

Type: Defect Priority: Trivial
Reporter: Yegor Timoshenko Assignee: Unassigned
Resolution: Unresolved Votes: 0
Labels: docstring

Attachments: Text File copy-doc.patch    
Patch: Code
Approval: Prescreened

 Description   

https://github.com/clojure/clojure/blob/a26dfc1390c53ca10dba750b8d5e6b93e846c067/src/clj/clojure/java/io.clj#L393

copy actually supports char[] as input (see lines 373-380), so add to docstring.

Prescreened by: Alex Miller - tests already exist for this behavior, just not in docstring






[CLJ-2155] clojure.string/index-of has some ^long type hints on let bindings that don't actually do anything Created: 19/Apr/17  Updated: 19/Apr/17

Status: Open
Project: Clojure
Component/s: None
Affects Version/s: None
Fix Version/s: None

Type: Defect Priority: Trivial
Reporter: Kevin Downey Assignee: Unassigned
Resolution: Unresolved Votes: 1
Labels: None

Approval: Triaged

 Description   

^long type hints on let binding values don't do anything:

user=> (def x 1)
#'user/x
user=> (set! *warn-on-reflection* true)
true
user=> (let [w ^long x] (Long/valueOf w))
Reflection warning, NO_SOURCE_PATH:13:18 - call to static method valueOf on java.lang.Long can't be resolved (argument types: unknown).
1
user=> (let [w (long x)] (Long/valueOf w))
1
user=>

but clojure.string/index-of has at least two cases of them, and even if they did do something, there is no reflective code that would take advantage of those type hints.






[CLJ-2154] Spec macros keys and keys* silently ignores non-keywords given in the vectors for named arguments :req and :req-un Created: 17/Apr/17  Updated: 17/Apr/17

Status: Open
Project: Clojure
Component/s: None
Affects Version/s: Release 1.9
Fix Version/s: None

Type: Defect Priority: Minor
Reporter: Rovanion Luckey Assignee: Unassigned
Resolution: Unresolved Votes: 0
Labels: spec
Environment:

Ubuntu 16.04 32-bit, Clojure 1.9.0-alpha15


Approval: Triaged

 Description   

If we try to pass a non-keyword to `clojure.spec/keys` or `clojure.spec/keys*` on the named argument `:opt` or `:opt-un` we get an assertion error:

(spec/valid? (spec/keys :opt ['a 5]) {})
;1. Caused by java.lang.AssertionError
;   Assert failed: all keys must be namespace-qualified keywords
;   (every? (fn* [p1__13667#] (c/and (keyword? p1__13667#) (namespace
;   p1__13667#))) (concat req-keys req-un-specs opt opt-un))

(spec/valid? (spec/keys* :opt-un ['a 5]) {})
;1. Caused by java.lang.AssertionError
;   Assert failed: all keys must be namespace-qualified keywords                    
;   (every? (fn* [p1__13667#] (c/and (keyword? p1__13667#) (namespace               
;   p1__13667#))) (concat req-keys req-un-specs opt opt-un))

But if we do the same for the named arguments `:req` and `:req-un` the arguments are silently ignored and the call to `keys` returns a spec matching any map without any requirements:

(spec/valid? (spec/keys :req ['a 5]) {})
=> true
(spec/valid? (spec/keys :req-un ['a 5]) {})
=> true
(spec/valid? (spec/keys* :req ['a 5]) {})
=> true
(spec/valid? (spec/keys* :req-un ['a 5]) {})
=> true

An assertion should probably be thrown for the `:req(-un)?` args too.






[CLJ-2153] Documentation for int-in and int-in-range? does not mention that they're limited to fixed precision integers Created: 12/Apr/17  Updated: 13/Apr/17

Status: Open
Project: Clojure
Component/s: None
Affects Version/s: Release 1.9
Fix Version/s: Release 1.9

Type: Enhancement Priority: Trivial
Reporter: Rovanion Luckey Assignee: Unassigned
Resolution: Unresolved Votes: 0
Labels: docstring, spec

Attachments: Text File fixed-precision-doc-3.patch     Text File fixed-precision-doc.patch     Text File fixed-precision-doc.patch    
Patch: Code
Approval: Screened

 Description   

The documentation for clojure.spec/int-in and clojure.spec/int-in-range? does not explicitly mention that they only accept and produce fixed precision integers as opposed to all integer types including BigInt.

Patch: fixed-precision-doc-3.patch

Screened by: Alex Miller



 Comments   
Comment by Alex Miller [ 12/Apr/17 10:00 AM ]

To consider the patch, you need to sign the contributors agreement, which you can find here: https://clojure.org/community/contributing

Comment by Alex Miller [ 13/Apr/17 8:31 AM ]

Thanks for signing the CA. Please update the patch so it applies cleanly without whitespace warnings:

$ git apply ~/Downloads/fixed-precision-doc.patch
/Users/alex/Downloads/fixed-precision-doc.patch:32: trailing whitespace.
  "Return true if start <= val, val < end and val is a fixed
/Users/alex/Downloads/fixed-precision-doc.patch:40: trailing whitespace.
  "Returns a spec that validates fixed precision integers in the
warning: 2 lines add whitespace errors.

While this is admittedly not consistent within this file, I think it's best if the docstring lines are indented (with 2 spaces) after the first line, and that's what I'd like to see in the patch. Text changes are fine.

Comment by Rovanion Luckey [ 13/Apr/17 8:40 AM ]

Docstring now indented with two spaces after the first line.

Comment by Rovanion Luckey [ 13/Apr/17 8:42 AM ]

Now with even less trailing whitespaces.





[CLJ-2152] clojure.spec: s/& has a broken form Created: 12/Apr/17  Updated: 12/Apr/17

Status: Open
Project: Clojure
Component/s: None
Affects Version/s: Release 1.9
Fix Version/s: Release 1.9

Type: Defect Priority: Major
Reporter: Tommi Reiman Assignee: Alex Miller
Resolution: Unresolved Votes: 0
Labels: spec
Environment:

1.9-alpha15


Approval: Vetted

 Description   
(require '[clojure.spec :as s])

(s/form (s/& integer?))
; (clojure.spec/& #object[clojure.core$integer_QMARK_ 0x5536db54 "clojure.core$integer_QMARK_@5536db54"])





[CLJ-2151] clojure.spec: s/merge reports errors multiple times with qualified keywords Created: 11/Apr/17  Updated: 11/Apr/17  Resolved: 11/Apr/17

Status: Closed
Project: Clojure
Component/s: None
Affects Version/s: Release 1.9
Fix Version/s: None

Type: Defect Priority: Major
Reporter: Tommi Reiman Assignee: Unassigned
Resolution: Declined Votes: 0
Labels: spec
Environment:

1.9-alpha15



 Description   

With unqualified keys, the problems look as expected. With qualified keys, the problems are shown from all branches?

(require '[clojure.spec :as s])

(s/def ::a int?)
(s/def ::b string?)

(s/explain-data
  (s/merge
    (s/keys :req-un [::a])
    (s/keys :req-un [::b]))
  {:a 1 :b 1})
; #:clojure.spec{:problems ({:path [:b], :pred string?, :val 1, :via [:user/b], :in [:b]})}

(s/explain-data
  (s/merge
    (s/keys :req [::a])
    (s/keys :req [::b]))
  {::a 1 ::b 1})
;#:clojure.spec{:problems ({:path [:user/b], :pred string?, :val 1, :via [:user/b], :in [:user/b]}
;                          {:path [:user/b], :pred string?, :val 1, :via [:user/b], :in [:user/b]})}


 Comments   
Comment by Alex Miller [ 11/Apr/17 3:42 PM ]

Spec explain reports every error it found - in this case it found that the required key ::b is missing in both branches of the s/merge. So I think this is the expected behavior.





[CLJ-2150] c.l Created: 11/Apr/17  Updated: 11/Apr/17  Resolved: 11/Apr/17

Status: Closed
Project: Clojure
Component/s: None
Affects Version/s: None
Fix Version/s: None

Type: Defect Priority: Major
Reporter: Tommi Reiman Assignee: Unassigned
Resolution: Duplicate Votes: 0
Labels: None


 Comments   
Comment by Tommi Reiman [ 11/Apr/17 7:35 AM ]

keybord typing error created multiple issues. Closing the unfinished duplicates.





[CLJ-2149] c.l Created: 11/Apr/17  Updated: 11/Apr/17  Resolved: 11/Apr/17

Status: Closed
Project: Clojure
Component/s: None
Affects Version/s: None
Fix Version/s: None

Type: Defect Priority: Major
Reporter: Tommi Reiman Assignee: Unassigned
Resolution: Duplicate Votes: 0
Labels: None


 Comments   
Comment by Tommi Reiman [ 11/Apr/17 7:35 AM ]

keybord typing error created multiple issues. Closing the unfinished duplicates.





[CLJ-2148] c.l Created: 11/Apr/17  Updated: 11/Apr/17  Resolved: 11/Apr/17

Status: Closed
Project: Clojure
Component/s: None
Affects Version/s: None
Fix Version/s: None

Type: Defect Priority: Major
Reporter: Tommi Reiman Assignee: Unassigned
Resolution: Duplicate Votes: 0
Labels: None


 Comments   
Comment by Tommi Reiman [ 11/Apr/17 7:35 AM ]

keybord typing error created multiple issues. Closing the unfinished duplicates.





[CLJ-2147] clojure.spec: s/keys* doesn't return a spec Created: 11/Apr/17  Updated: 11/Apr/17  Resolved: 11/Apr/17

Status: Closed
Project: Clojure
Component/s: None
Affects Version/s: Release 1.9
Fix Version/s: None

Type: Defect Priority: Major
Reporter: Tommi Reiman Assignee: Unassigned
Resolution: Declined Votes: 0
Labels: spec
Environment:

1.9.0-alpha15



 Description   
(s/spec? (s/keys* :req-un [::a ::b]))
; nil


 Comments   
Comment by Alex Miller [ 11/Apr/17 3:46 PM ]

This is pretty subtle but the regex ops return something that is a `regex?` but not a `spec?`. The reason for this has to do with being able to merge things that are regexes, but not merge across specs. (There is also some further history based in prior implementations.) In any case, this is currently the expected result.





[CLJ-2146] partition-by and partition-all transducers should ensure visibility of state changes Created: 09/Apr/17  Updated: 17/Apr/17

Status: Open
Project: Clojure
Component/s: None
Affects Version/s: Release 1.7
Fix Version/s: None

Type: Defect Priority: Major
Reporter: Alex Miller Assignee: Unassigned
Resolution: Unresolved Votes: 1
Labels: transducers

Approval: Vetted

 Description   

The partition-by and partition-all transducers use state stored in an ArrayList. This state should be protected (for example, by volatile) to ensure visibility if used in a transducing process that moves computations across threads.



 Comments   
Comment by Léo NOEL [ 13/Apr/17 1:00 PM ]

Discussion here : https://groups.google.com/forum/m/#!topic/clojure/VQj0E9TJWYY

Comment by Léo NOEL [ 16/Apr/17 9:47 AM ]

Note that following this logic, transients are as much broken, as they make use of plain arrays.
This paragraph https://clojure.org/reference/transients#_concurrent_use makes me believe this very problem has already been tackled before. Is the discussion available somewhere ?
In my opinion, documentation should be more precise about what is meant by thread isolation, and explain why it is OK to use unsynchronized mutable objects when they're owned by something that enforces sequential processing (agents, go blocks, channels, single-threading, etc).

Comment by Alex Miller [ 17/Apr/17 9:24 AM ]

Transients originally enforced thread isolation by recording and validating the originating thread. This was weakened to allow for transients passed around go blocks in Clojure 1.7 and has been through some rounds of fixes (like CLJ-1580). If there is an issue with them now, please file a separate ticket.





[CLJ-2145] locals closed over by a ^:once fn aren't cleared if the fn is in a branch Created: 08/Apr/17  Updated: 11/Apr/17

Status: Open
Project: Clojure
Component/s: None
Affects Version/s: None
Fix Version/s: None

Type: Defect Priority: Major
Reporter: Nicola Mometto Assignee: Unassigned
Resolution: Unresolved Votes: 1
Labels: compiler, locals-clearing

Attachments: Text File 0001-CLJ-2145-fix-clearing-of-locals-closed-over-by-a-FNO.patch    
Patch: Code
Approval: Triaged

 Description   

Minimal case:

(fn foo [x]
  (if true
    (^:once fn* []
     ;; x is not cleared here
     x)))

This is a severe bug as it means that every local used inside a loop or try/catch expression that the clojure compiler internally hoists in a FNONCE, in a conditional branch, cannot be cleared at the moment.

As a concrete example reported in slack,

;; THIS OOMs
(defn test1 [x]
  (if true
    (do
      (try (doseq [_ x] _))
      1)
    0))

(test1 (take 1000000 (range)))

;; THIS DOESN'T OOM 
(defn test2 [x]
  (do
    (try (doseq [_ x] _))
    1))

(test2 (take 1000000 (range)))

Approach: don't set a new clearing frame if the fn is ^:once and there's an existing clearing frame
Patch: 0001-CLJ-2145-fix-clearing-of-locals-closed-over-by-a-FNO.patch






[CLJ-2144] clojure.walk/keywordize-keys wants ns support for clojure.spec utility Created: 08/Apr/17  Updated: 11/Apr/17

Status: Open
Project: Clojure
Component/s: None
Affects Version/s: Release 1.9
Fix Version/s: None

Type: Enhancement Priority: Major
Reporter: Aaron Brooks Assignee: Unassigned
Resolution: Unresolved Votes: 0
Labels: None

Attachments: Text File 0001-CLJ-2144-Add-namespace-arg-to-walk-keywordize-keys.patch    
Patch: Code and Test

 Description   

keywordize-keys currently takes a single argument, a nested structure presumably containing maps, turning all string keys into un-namespaced keys. I've found that I've needed to maintain my own modified version of keywordize-keys that allows me to pass a namespace so I can import JSON objects and use them with clojure.spec (which strongly prefers namespaced keys).

The addition of an additional 2-arity invocation with a namespace string argument is a non-breaking change. I'll attach a patch once I have a JIRA number.



 Comments   
Comment by Aaron Brooks [ 08/Apr/17 1:51 PM ]

This patch also includes a test but I accidentally selected "Code" when I opened the ticket. I don't think I can change that. Can a maintainer update the "Patch" field to "Code and test"?

Comment by Alex Miller [ 11/Apr/17 4:10 PM ]

Another route to go with this btw is to first do CLJ-1899 to build on.





[CLJ-2143] The result of s/form for s/keys* is different from the original form Created: 05/Apr/17  Updated: 05/Apr/17

Status: Open
Project: Clojure
Component/s: None
Affects Version/s: Release 1.9
Fix Version/s: Release 1.9

Type: Defect Priority: Major
Reporter: OHTA Shogo Assignee: Alex Miller
Resolution: Unresolved Votes: 0
Labels: spec

Approval: Vetted

 Description   

If s/form is applied to s/keys*, it returns a value completely different from the original form:

user=> (s/form (s/keys* :req-un [::x ::y]))
(clojure.spec/& (clojure.spec/* (clojure.spec/cat :clojure.spec/k clojure.core/keyword? :clojure.spec/v clojure.core/any?)) :clojure.spec/kvs->map mspec__14270__auto__)
user=>


 Comments   
Comment by Alex Miller [ 05/Apr/17 8:57 AM ]

Thanks for logging - I've been working on an approach for this one but never got around to actually logging it.





[CLJ-2142] Namespace map syntax prevents duplicate key check Created: 02/Apr/17  Updated: 03/Apr/17

Status: Open
Project: Clojure
Component/s: None
Affects Version/s: Release 1.9
Fix Version/s: Release 1.9

Type: Defect Priority: Major
Reporter: Gregg Reynolds Assignee: Unassigned
Resolution: Unresolved Votes: 1
Labels: reader

Attachments: Text File 0001-CLJ-2142-throw-on-duplicate-keys-in-namespaced-map-l.patch     Text File 0001-CLJ-2142-throw-on-duplicate-keys-in-namespaced-map-l-v2.patch     Text File clj-2142-3.patch    
Patch: Code and Test
Approval: Vetted

 Description   
user> #::{:a 1 :a 2}
#:user{:a 2}

Cause: In the namespace map reader, a map is built by repeated assoc rather than via createWithCheck. Thus, assoc of same key replaces prior key rather than throwing an error.

Approach: Build an array and invoke RT.map(a), to echo same code path without namespace map literal syntax.

After:

user=> #::{:a 1 :a 2}
IllegalArgumentException Duplicate key: :user/a  clojure.lang.PersistentArrayMap.createWithCheck (PersistentArrayMap.java:71)

Patch: clj-2142-3.patch



 Comments   
Comment by Nicola Mometto [ 02/Apr/17 12:06 PM ]

updated patch also fixes EdnReader

Comment by Gregg Reynolds [ 02/Apr/17 1:23 PM ]

wow, that was fast, Nicola!





[CLJ-2141] New qualified-* predicates can return true, nil, and false Created: 31/Mar/17  Updated: 11/Apr/17

Status: Open
Project: Clojure
Component/s: None
Affects Version/s: Release 1.9
Fix Version/s: Release 1.9

Type: Defect Priority: Major
Reporter: Daniel Compton Assignee: Unassigned
Resolution: Unresolved Votes: 89
Labels: function

Attachments: Text File clj-2141.patch    
Patch: Code
Approval: Vetted

 Description   

As of Clojure 1.9-alpha15, the new qualifed-keyword?, qualified-symbol?, and qualified-ident? functions can return true, nil, or false depending on the input passed to them, e.g.:

=> (qualified-keyword? ::abc)
true
=> (qualified-keyword? :abc)
nil
=> (qualified-keyword? 'abc)
false

Returning nil rather than false for unqualified keywords has the following issues:

  • Many sources of Clojure documentation note that a trailing '?' in a Clojure function generally implies a true/false return so this violates a common expectation.
  • Returning true/false/nil complicates the ability of ClojureScript to optimize boolean returns. Theoretically, this might affect Clojure JIT optimization as well, turning a call taking the return value into polymorphic rather than bimorphic.
  • Returning true/false/nil can yield confusing results for functions that partition by value, like group-by

The prior implementation during development used `boolean` to coerce the return value to a null and that was removed for performance reasons (would require a call through another var).

Alternatives:

Perf benchmark (w/ criterium bench, Java 1.8, AOT, direct-linked)
Example: (bench (dotimes [_ 100] (qualified-keyword? :a)))

Impl :a :a/b note
(and (keyword? x) (namespace x) true) 96.749127 ns 96.412551 ns current impl (returns nil for :a)
(and (keyword? x) (some? (namespace x))) 99.014677 ns 96.149567 ns uses existing preds
(and (keyword? x) (not (nil? (namespace x)))) 97.512916 ns 95.008061 ns nil? inlines
(if (and (keyword? x) (namespace x)) true false) 98.617369 ns 97.866673 ns if-based
(if (keyword? x) (if (namespace x) true false) false) 99.000727 ns 99.474754 ns  

Note that these vary by less than 1 ns per invocation, so perf is not significantly different.

Approach: In my opinion, the best combination of returning logical values, using a logical expression, and performance is:
(and (keyword? x) (not (nil? (namespace x))))

Patch: clj-2141.patch - implements the discussed approach



 Comments   
Comment by Didier A. [ 03/Apr/17 5:14 AM ]

Agreed, either make it truthy/falsey and drop the ?, or keep the convention, one of the fee ones that still does not have an exception to its rule.

Comment by Alex Miller [ 03/Apr/17 12:19 PM ]

Moving to vetted so we remember to look at this before 1.9 releases.

Comment by Nikita Prokopov [ 04/Apr/17 2:38 AM ]

Just wanted to add my 2cents:

  • Even if nobody ever promised explicitly that predicates strictly return true/false, it has become an established convention, people rely on it.
  • Performance benefits are debatable at best. The function itself might become slightly faster but the coercion to boolean is still happening at the call site (e.g. inside if/when condition).
  • I think problem is important enough that it must get at least a second look. If there’s no good reason to break the convention, it’ll be better for everybody if we keep following it.
  • Having predicates that return three possible values is, well, weird by all means.
  • Even worse is that they look exactly like the old predicates. There’s no way to tell. It adds nuance and complexity to otherwise pretty simple and straightforward part of Clojure

Sorry but I really think this was overlooked. I only wish best for Clojure ind its future. Please at least consider this.

Thanks!

Nikita.

Comment by Ertuğrul Çetin [ 04/Apr/17 6:25 AM ]

Totally agreed!

Comment by Murat Berk [ 05/Apr/17 9:38 AM ]

It should NOT violate a common expectation/convention.

Comment by Bozhidar Batsov [ 06/Apr/17 1:50 AM ]

Btw, this is also an opportunity to improve the docstrings of those functions. Apart from the fact that for the millionth time they're missing a final `.`, now we can add something like `otherwise it returns false` or something like this. I assume this was originally omitted due the possibility to return either nil or false.

Comment by Alex Miller [ 07/Apr/17 3:29 PM ]

Re the docstrings, no plans to update those. The "otherwise..." was omitted because it's omitted on all of the other predicate docstrings as well. That makes sense (if the common implication is that the return value is false).

Comment by Bozhidar Batsov [ 11/Apr/17 2:22 AM ]

Perhaps. On a more general note - I really think some docstring standard should be imposed at some point. I find it odd that some docstrings are terminated with `.`, some are not. Not to mention it's really hard to figure out in docstrings what's a reference to a parameter, what's a reference to some other var, etc. And inconsistent docstrings are hard to format/present - e.g. in Elisp and CL it's common to have the first line of the docstring as a separate sentence, so you could have a good one-line overview of the thing it describes. Anyways, that's completely off-topic. One day I might blog/speak about this.





[CLJ-2140] surprising output from s/explain for a s/coll-of spec that encounters a map Created: 31/Mar/17  Updated: 31/Mar/17  Resolved: 31/Mar/17

Status: Closed
Project: Clojure
Component/s: None
Affects Version/s: Release 1.9
Fix Version/s: None

Type: Defect Priority: Major
Reporter: David Chelimsky Assignee: Unassigned
Resolution: Declined Votes: 0
Labels: spec


 Description   

When a coll-of spec encounters a map, the explain output does not align with the validation issue:

(s/def :some/keywords (s/coll-of keyword?))
=> :some/keywords
(s/explain-data :some/keywords [:foo])
=> nil
(s/explain-data :some/keywords ["foo"])
=> #:clojure.spec{:problems ({:path [], :pred keyword?, :val "foo", :via [:some/keywords], :in [0]})}
(s/explain-data :some/keywords :foo)
=> #:clojure.spec{:problems [{:path [], :pred coll?, :val :foo, :via [:some/keywords], :in []}]}
(s/explain-data :some/keywords {:foo :bar})
=> #:clojure.spec{:problems ({:path [], :pred keyword?, :val [:foo :bar], :via [:some/keywords], :in [0]})}

I'd expect this from the last expression:

=> #:clojure.spec{:problems [{:path [], :pred coll?, :val {:foo :bar}, :via [:some/keywords], :in []}]}


 Comments   
Comment by Alex Miller [ 31/Mar/17 9:13 AM ]

Maps are collections and it is possible and occasionally useful to use s/coll-of or s-every to spec maps as collections of tuples. So, I think this is the expected and correct behavior.

There are some related cases in CLJ-2080 where this can go awry but the changes in the patch there would not affect this case.





[CLJ-2139] Couldn't satisfy such-that predicate after 100 tries error when using int? and int-in together Created: 29/Mar/17  Updated: 30/Mar/17  Resolved: 30/Mar/17

Status: Closed
Project: Clojure
Component/s: None
Affects Version/s: Release 1.9
Fix Version/s: None

Type: Defect Priority: Minor
Reporter: Didier A. Assignee: Unassigned
Resolution: Declined Votes: 0
Labels: newbie, spec
Environment:

Windows 7
Clojure 1.9.0-alpha15



 Description   

If you spec :args with (s/and int? (s/int-in x y)) you will get "ExceptionInfo Couldn't satisfy such-that predicate after 100 tries."

Here's code to reproduce:

Unable to find source-code formatter for language: clj. Available languages are: javascript, sql, xhtml, actionscript, none, html, xml, java
(ns spec-test.core
  (:require [clojure.spec :as s]
            [clojure.spec.test :as st]))

(defn aaa [a] (.charAt "abcdef" a))
(s/fdef aaa
        :args (s/cat :a (s/and int? (s/int-in 0 5)))
        :ret any?)

(st/check `aaa)


 Comments   
Comment by Alex Miller [ 30/Mar/17 8:59 AM ]

The problem here is that when you use (s/and int? (s/int-in 0 5)) as the spec, the generator will generate random ints from the first subspec, then filter based on the second one. Because most of the numbers don't fall in the range, it will fail to generate. You can find more info about and generators at http://blog.cognitect.com/blog/2016/8/24/combining-specs-with-and.

In this particular case, it's easy to adjust this by simply reducing your spec to (s/int-in 0 5) which is designed to generate only values in that range.





[CLJ-2138] Can't use an aliased keyword in the same form as alias definition Created: 25/Mar/17  Updated: 26/Mar/17  Resolved: 26/Mar/17

Status: Closed
Project: Clojure
Component/s: None
Affects Version/s: Release 1.8
Fix Version/s: None

Type: Defect Priority: Major
Reporter: Yegor Timoshenko Assignee: Unassigned
Resolution: Declined Votes: 0
Labels: None
Environment:

java version "1.8.0_111"
Java(TM) SE Runtime Environment (build 1.8.0_111-b14)
Java HotSpot(TM) 64-Bit Server VM (build 25.111-b14, mixed mode)



 Description   

user=> (require '[clojure.string :as str])
nil
user=> ::str/hello
:clojure.string/hello
user=> (do (require '[clojure.string :as str2]) ::str2/hello)

RuntimeException Invalid token: ::str2/hello clojure.lang.Util.runtimeException (Util.java:221)
RuntimeException Unmatched delimiter: ) clojure.lang.Util.runtimeException (Util.java:221)

At the same time, creating an alias and using a function from the aliased namespace is possible:

user=> (do (require '[clojure.string :as str3]) (str3/blank? ""))
true



 Comments   
Comment by Alex Miller [ 26/Mar/17 11:05 AM ]

Autoresolved keywords are a feature of the reader, which reads one form at a time. In this case, the whole do block is read prior to being evaluated so the alias context does not yet exist when that keyword is read.





[CLJ-2137] Clojure REPL doesn't ask for new input if line contains a keyword with a colon Created: 25/Mar/17  Updated: 26/Mar/17  Resolved: 26/Mar/17

Status: Closed
Project: Clojure
Component/s: None
Affects Version/s: Release 1.6, Release 1.8, Release 1.9
Fix Version/s: None

Type: Defect Priority: Major
Reporter: Yegor Timoshenko Assignee: Unassigned
Resolution: Not Reproducible Votes: 0
Labels: reader, repl
Environment:

java version "1.8.0_111"
Java(TM) SE Runtime Environment (build 1.8.0_111-b14)
Java HotSpot(TM) 64-Bit Server VM (build 25.111-b14, mixed mode)



 Description   
user=> :a
:a
user=> :a:z
:a:z
user=> [:a
  #_=> ]
[:a]
user=> [:a:z

RuntimeException EOF while reading, starting at line 1  clojure.lang.Util.runtimeException (Util.java:221)


 Comments   
Comment by Alex Miller [ 26/Mar/17 7:44 PM ]

This is not reproducible with the standard Clojure repl (invoking clojure.main). I can reproduce it with Leiningen so I'm guessing that's where you're seeing it. You can file a bug report for Leiningen at https://github.com/technomancy/leiningen.

Comment by Yegor Timoshenko [ 26/Mar/17 7:48 PM ]

Couldn't imagine that a build tool can affect the REPL. Thank you!





[CLJ-2136] Reloading multi-arity macros fails Created: 24/Mar/17  Updated: 24/Mar/17  Resolved: 24/Mar/17

Status: Closed
Project: Clojure
Component/s: None
Affects Version/s: Release 1.7, Release 1.8
Fix Version/s: None

Type: Defect Priority: Major
Reporter: Joel Kaasinen Assignee: Unassigned
Resolution: Declined Votes: 0
Labels: macro
Environment:

Tested against clojure 1.7 and 1.8



 Description   

When a multi-arity macro is defined using &form and &env, reloading the namespace fails.

Steps to reproduce:

File macro.clj:

(ns macro)

(defmacro macro
  ([x] (macro &form &env x 2))
  ([x y] `(prn ~x ~y)))

REPL:

user=> (require 'macro)
nil
user=> (macro/macro 1)
1 2
nil
user=> (require 'macro :reload)

CompilerException clojure.lang.ArityException: Wrong number of args (4) passed to: macro/macro, compiling:(macro.clj:4:8)

PS. a workaround is to define the macro like

(defmacro macro
  ([x] `(macro ~x 2))
  ([x y] `(prn ~x ~y)))


 Comments   
Comment by Alex Miller [ 24/Mar/17 8:26 AM ]

This macro definition is incorrect as it's passing 4 args from one arity to the other, not 2 (which is what the error tells you). The "workaround" fixes that problem. I don't see a bug here.





[CLJ-2135] clojure.spec/Spec implementations that don't implement IObj get silently dropped in s/def Created: 23/Mar/17  Updated: 23/Mar/17

Status: Open
Project: Clojure
Component/s: None
Affects Version/s: None
Fix Version/s: None

Type: Enhancement Priority: Minor
Reporter: Thomas Heller Assignee: Unassigned
Resolution: Unresolved Votes: 0
Labels: spec

Attachments: Text File clj-2135.patch    
Patch: Code
Approval: Triaged

 Description   
(deftype MySpec []
  s/Spec
  (conform* [_ x]
    ::s/invalid))

(s/def ::x (MySpec.))

(s/explain ::x :foo)

This will fail with a "Unable to resolve spec: :user/x" exception, but the def succeeded. Switching the deftype to defrecord fixes the problem.

Cause: The with-name function has cond options for ident?, regex?, and IObj. If none of these succeed, there is no fallthrough case and the s/def will silently return nil.

Proposed: Throw an error in the fallthrough case.

Patch: clj-2135.patch






[CLJ-2134] Update the docstring of `with-redefs` to reflect how little the macro should be used Created: 23/Mar/17  Updated: 23/Mar/17

Status: Open
Project: Clojure
Component/s: None
Affects Version/s: None
Fix Version/s: None

Type: Enhancement Priority: Minor
Reporter: Timothy Baldridge Assignee: Unassigned
Resolution: Unresolved Votes: 0
Labels: docstring

Approval: Triaged

 Description   

Currently the docstring for `with-redefs` recommends itself for use in testing. However there are a number of reasons why using this macro for testing is suboptimal:

  • `with-redefs` "bindings" are not transferred to new threads since it's a global mutation
  • users can get runtime errors if they redef a primitive type-hinted function to a function taking only objects
  • If parts of the body of `with-redefs` is delayed (via a delay, go block, etc.) that code may not see the new root
  • The mutation is global so it "leaks" outside the current scope into other code that may currently be running in another thread
  • Clojure tends to shun global mutation, and yet this macro isn't marked with a `!` nor properly warns users about the dangers mentioned here

Due to these reasons I often encounter new users using `with-redefs` without understanding the ramifications of doing so. All this behavior makes sense if a user understands how Vars work, but that's a lot of knowledge to take on for a new user.

Suggestion:
Remove the suggestion that `with-redefs` be used in testing
Add a few notes of warning about global mutation, and concurrency issues with the macro.






[CLJ-2133] Clarify documentation for the satisfies? function. Created: 23/Mar/17  Updated: 23/Mar/17

Status: Open
Project: Clojure
Component/s: None
Affects Version/s: Release 1.8, Release 1.9
Fix Version/s: None

Type: Enhancement Priority: Minor
Reporter: David Chelimsky Assignee: Unassigned
Resolution: Unresolved Votes: 1
Labels: docstring
Environment:

N/A



 Description   

The docs for satisfies? says "Returns true if x satisfies the protocol", but does not define the meaning of "satisfies". The function returns true when type and protocol are referenced in the same call to either extend-type or extend-protocol even when none of the protocol functions are implemented. I think the doc should be specific about this to avoid confusion.






[CLJ-2132] Maven pom requires artifact signing to install locally Created: 23/Mar/17  Updated: 20/Apr/17  Resolved: 20/Apr/17

Status: Closed
Project: Clojure
Component/s: None
Affects Version/s: Release 1.9
Fix Version/s: Release 1.9

Type: Enhancement Priority: Major
Reporter: Alex Miller Assignee: Unassigned
Resolution: Completed Votes: 0
Labels: build, regression

Attachments: Text File clj-2132.patch    
Patch: Code
Approval: Ok

 Description   

The recent pom changes inadvertently made artifact signing a requirement for locally installing a Clojure build via:

mvn clean install

The attached patch moves the GPG plugin back into a profile (named "sign").






[CLJ-2131] partition-with Created: 19/Mar/17  Updated: 19/Mar/17  Resolved: 19/Mar/17

Status: Closed
Project: Clojure
Component/s: None
Affects Version/s: None
Fix Version/s: None

Type: Enhancement Priority: Trivial
Reporter: Derek Troy-West Assignee: Unassigned
Resolution: Declined Votes: 0
Labels: None


 Description   

Any interest in introducing a partition fn that sits somewhere between partition-by and split-with?

(defn partition-with
"Applies f to each value in coll, splitting it each time f returns truthy
Returns a lazy seq of partitions."
[f coll]
(lazy-seq
(when-let [s (seq coll)]
(let [run (cons (first s) (take-while (complement f) (next s)))]
(cons run (partition-with f (seq (drop (count run) s))))))))

e.g

(partition-with #(= (rem % 3) 0) [1 2 3 6 7 8 9 12 13 15 16 17 18])
=> ((1 2) (3) (6 7 8) (9) (12 13) (15 16 17) (18))

I've used this occasionally and I notice it popped up on StackOverflow recently.

Not included thus far: the transducer arity or tests, but I'm happy to supply a patch if you're interested.



 Comments   
Comment by Derek Troy-West [ 19/Mar/17 6:31 AM ]

Apols for the formatting, I don't seem to be able to edit.

Comment by Derek Troy-West [ 19/Mar/17 7:12 AM ]

On reflection the special case of a seq of delimited sub-sequences is probably too narrow for core, which explains its current absence. Please ignore (I would kill the Jira myself, but..)

Comment by Alex Miller [ 19/Mar/17 12:32 PM ]

closed per request





[CLJ-2130] classof data-reader Created: 17/Mar/17  Updated: 17/Mar/17  Resolved: 17/Mar/17

Status: Closed
Project: Clojure
Component/s: None
Affects Version/s: None
Fix Version/s: None

Type: Enhancement Priority: Minor
Reporter: Greg Chapman Assignee: Unassigned
Resolution: Declined Votes: 0
Labels: None


 Description   

I think it would be useful to have a #classof default data-reader which would read the supplied form as a Java class. Examples:

#classof String -> java.lang.String
#classof java.util.Map -> java.util.Map
#classof [String] -> [Ljava.lang.String;
#classof long -> long (aka Long/TYPE)
#classof [long] -> [J
#classof [[int]] -> [[I

So the idea is that a symbol would be read as the class to which it resolves (with support for primitives), and vectors would be read as the class of arrays of the classof the first (and only) item in the vector.

The main reason for wanting this is to have a readable form for array classes.



 Comments   
Comment by Alex Miller [ 17/Mar/17 4:02 PM ]

This is a solution, not a problem. The description only lightly mentions the problem at the end but does not demonstrate (preferably in an example) where this problem is encountered or why the current solution for these problems is not sufficient. The suggestion here is one idea, but no other alternatives are suggested, and tradeoffs are not considered.

Comment by Greg Chapman [ 17/Mar/17 8:01 PM ]

This is a fair criticism, and I regret having posted the idea without exploring it more fully. Especially as the above idea will not work for type-hinting (for some reason, I thought instances of Class could be used in :tags, but I now see that only Symbols and Strings are allowed).

Anyway, the thing that got me thinking along these lines is the annoyance involved with extending protocols to arrays. In particular, extend-type uses its t parameter in two ways: 1) emitted as-is to be evaluated (resolving to a Class) and passed as the first parameter to extend, and 2) in the unemitted metadata to type-hint the protocol functions. I don't think there's a way to satisfy both these uses with a single form representing an array class, so you have to fall back on extend and do your own type-hinting.

But that's not really that big a deal. I apologize for wasting your time with this.

Comment by Alex Miller [ 17/Mar/17 10:05 PM ]

You might also want to keep an eye on CLJ-1381.





[CLJ-2129] Enhance CompilerException to optionally return the invalid form Created: 16/Mar/17  Updated: 16/Mar/17

Status: Open
Project: Clojure
Component/s: None
Affects Version/s: None
Fix Version/s: None

Type: Enhancement Priority: Major
Reporter: Alex Miller Assignee: Unassigned
Resolution: Unresolved Votes: 1
Labels: compiler, error-reporting

Approval: Triaged

 Description   

Currently CompilerException wraps errors that occur at compile time and adds file/line/col info. Some tools could do more with the failing form, particularly if it includes useful meta.

Proposal is to add a CompilerException constructor that also takes the form (Object) and conveys it. Existing uses like CLI and REPL would do nothing different, but a tooling user of the Compiler could use that information.

Possible issue: if the form is lazy or large?



 Comments   
Comment by Thomas Heller [ 16/Mar/17 12:29 PM ]

I added the latest form to the CompilerException when available but that form contains basically no metadata.

user=> (defn x
  :foo)
CompilerException clojure.lang.ExceptionInfo: Call to clojure.core/defn did not conform to spec:
...

user=> (binding [*print-meta* true] (prn (.-form *e)))
^{:line 17, :column 1} (defn x :foo)

That information is already present in the CompilerException. Having the form provides minimal benefit in this case since it has lost all other source information and we cannot tell how this look in the the source. To get a source mapping for tools so they can highlight the correct area it would still need to read the form itself (and probably not using the form from the Exception).

Comment by Thomas Heller [ 16/Mar/17 12:37 PM ]

Looking into LINE_BEFORE, COLUMN_BEFORE, LINE_AFTER, COLUMN_AFTER now to potentially provide the exact boundaries of the form if possible.

Comment by Thomas Heller [ 16/Mar/17 2:00 PM ]

I added the current reader location to the Compiler Exception [1].

user=> (load-file "/Users/zilence/code/shadow-devtools/src/dev/demo/defn_error.clj")
CompilerException clojure.lang.ExceptionInfo: Call to clojure.core/defn did not conform to spec:
...
user=> (.. *e -location -lineBefore)
4
user=> (.. *e -location -lineAfter)
6
;; column is 1 before/after, source is
(defn x
  :foo)

This would at least allow extracting the entire source string of the form easily. When using a reader however it could start at the location already provided by the CompilerException and it would find the end on its own. The bindings required for the reader location are also lost when working in a REPL, so it always points to 0/0-0/0. Not sure this is worth pursuing.

[1] https://github.com/clojure/clojure/compare/master...thheller:CLJ-2128





[CLJ-2128] spec error during macroexpand no longer throws compiler exception with location Created: 16/Mar/17  Updated: 16/Mar/17

Status: Open
Project: Clojure
Component/s: None
Affects Version/s: Release 1.9
Fix Version/s: Release 1.9

Type: Defect Priority: Major
Reporter: Alex Miller Assignee: Unassigned
Resolution: Unresolved Votes: 1
Labels: error-reporting, regression, spec
Environment:

1.9.0-alpha12-1.9.0-alpha15


Attachments: Text File clj-2128.patch    
Patch: Code
Approval: Vetted

 Description   

This used to work but got out of sync with the commit https://github.com/clojure/clojure/commit/b3d3a5d6ff0a2f435bb6a5326da2b960038adad4, which changed the IllegalArgumentException to an ex-info, but didn't change the corresponding catch in the Compiler. That change is visible as of 1.9.0-alpha12.

Before alpha12:

...
Exception in thread "main" java.lang.IllegalArgumentException: Call to clojure.core/ns did not conform to spec:
In: [1] val: ((:required clojure.string)) fails at: [:args] predicate: (cat :docstring (? string?) :attr-map (? map?) :clauses :clojure.core.specs/ns-clauses),  Extra input
:clojure.spec/args  (th.core (:required clojure.string))
, compiling:(th/core.clj:1:1)
...

^^ note the "th/core.clj:1:1"

After alpha12:

...
Exception in thread "main" clojure.lang.ExceptionInfo: Call to clojure.core/ns did not conform to spec:
In: [1] val: ((:required clojure.string)) fails at: [:args] predicate: (cat :docstring (? string?) :attr-map (? map?) :clauses :clojure.core.specs/ns-clauses),  Extra input
:clojure.spec/args  (th.core (:required clojure.string))
 #:clojure.spec{:problems [{:path [:args], :reason "Extra input", :pred (cat :docstring (? string?) :attr-map (? map?) :clauses :clojure.core.specs/ns-clauses), :val ((:required clojure.string)), :via [], :in [1]}], :args (th.core (:required clojure.string))}, compiling:(/private/var/folders/7r/_1fj0f517rgcxwx79mn79mfc0000gn/T/form-init4120559363887828149.clj:1:125)
	at clojure.lang.Compiler.load(Compiler.java:7441)
...

Patch: clj-2128.patch






[CLJ-2127] clojure.spec/def requires literal keyword as first argument Created: 15/Mar/17  Updated: 15/Mar/17  Resolved: 15/Mar/17

Status: Closed
Project: Clojure
Component/s: None
Affects Version/s: Release 1.9
Fix Version/s: None

Type: Defect Priority: Minor
Reporter: Anders Hessellund Jensen Assignee: Unassigned
Resolution: Declined Votes: 0
Labels: None


 Description   

I expected the following to work:
(clojure.spec/def (keyword "foo" "bar") string?)

It fails, because the def macro does not evaluate the first argument. Is this intentional? If so, perhaps the documentation or error message could be updated to reflect that the macro only accepts literal keywords.

As an aside, as a clojure beginner I found the semantics of clojure.spec/def confusing. The regular def macro and its variants creates bindings in the current namespace. clojure.spec/def does not modify the namespace, it registers specs in the spec registry under the provided key, even though it also accepts a symbol as an argument. I would have been less confused if the function was named clojure.spec/register!, the first argument was evaluated, and registering a symbol would require quoting of the symbol. That would make it behave like a regular function where arguments are evaluated as usual.



 Comments   
Comment by Alex Miller [ 15/Mar/17 9:37 AM ]

The docstring starts "Given a namespace-qualified keyword or resolvable symbol k" which seems to say the accurate thing. The error seems pretty clear to me too:

user=> (clojure.spec/def (keyword "foo" "bar") string?)
AssertionError Assert failed: k must be namespaced keyword or resolvable symbol
(c/and (ident? k) (namespace k))

If you really need to do something like this, it's easy enough to wrap with another macro like:

user=> (defmacro mydef [ns n s] `(s/def ~(keyword ns n) ~s))
#'user/mydef
user=> (mydef "foo" "bar" string?)
:foo/bar

Rich considered all the ramifications of the name when he named it, so that's not likely to change.

Comment by Anders Hessellund Jensen [ 15/Mar/17 10:49 AM ]

I still fail to see how I can infer from the documentation that k has to be a literal qualified keyword, and that an expression evaluating to a qualified keyword is not accepted. I assume it is my lack of Clojure experience.

Thanks for your response, and apologise for wasting your time.

Comment by Alex Miller [ 15/Mar/17 11:18 AM ]

This is the situation with many core macros (which are after all taking code and returning code and thus are a bit more literal-minded than functions). No apologies necessary - while I'm declining here, it's still useful feedback and might affect decisions in the future.





[CLJ-2126] Can set! to fields of a defrecord Created: 14/Mar/17  Updated: 14/Mar/17

Status: Open
Project: Clojure
Component/s: None
Affects Version/s: Release 1.8, Release 1.9
Fix Version/s: None

Type: Defect Priority: Major
Reporter: Francis Avila Assignee: Unassigned
Resolution: Unresolved Votes: 0
Labels: compiler

Attachments: Text File 0001-CLJ-2126-don-t-assign-final-fields.patch    
Patch: Code
Approval: Triaged

 Description   

It is possible to set! to fields of a defrecord even though they are final.

(defprotocol SetA (seta [x a]))
=> SetA
(defrecord X [a]
  SetA
  (seta [this newa]
    ; Next line should error at compile time, does not.
    ; (However (set! a newa) does error correctly.)
    (set! (.a this) newa)))
=> user.X
(def x (->X 0))
=> #'user/x
x
=> #user.X{:a 0}
(seta x 1) ;; This should not run.
=> 1
x
=> #user.X{:a 1}

There are two issues here:

  1. The Clojure compiler does not detect that (set! (.a this) x) is assignment to a final field. This could be enhanced. Nicola Mometto has discovered why and believes he has a straightforward patch.
  2. The JVM bytecode verifier only performs the necessary final-assignment check on classfiles version 9 and above: https://bugs.openjdk.java.net/browse/JDK-8159215 (Clojure generates version 6 classfiles.) This is out of our hands.

Approach: make the compiler fail at compile time if trying to set! a field that's final

Patch: 0001-CLJ-2126-don-t-assign-final-fields.patch






[CLJ-2125] fspec doesn't generate pure functions. Created: 12/Mar/17  Updated: 12/Mar/17  Resolved: 12/Mar/17

Status: Closed
Project: Clojure
Component/s: None
Affects Version/s: Release 1.9
Fix Version/s: None

Type: Enhancement Priority: Minor
Reporter: N/A Assignee: Unassigned
Resolution: Declined Votes: 0
Labels: spec


 Description   

fspec doesn't generate pure functions.

(defn foo
[f]
(let [mf (memoize f)]
(= (mf 0) (mf 0))))

(s/fdef foo
:args (s/cat :f (s/fspec :args (s/cat :y int?)
:ret int?))
:ret true?)

(stest/check `foo)

=>
({:spec #object[clojure.spec$fspec_impl$reify__14282
0x20877912
"clojure.spec$fspec_impl$reify__14282@20877912"],
:clojure.spec.test.check/ret {:result true,
:num-tests 1000,
:seed 1489361298159},
:sym spike-spec.core/foo})

(defn bar
[f]
(= (f 0) (f 0)))

(s/fdef bar
:args (s/cat :f (s/fspec :args (s/cat :y int?)
:ret int?))
:ret true?)

(stest/check `bar)

=>
({:spec #object[clojure.spec$fspec_impl$reify__14282
0x56f3d7c7
"clojure.spec$fspec_impl$reify__14282@56f3d7c7"],
:clojure.spec.test.check/ret {:result #error{:cause "Specification-based check failed",
:data {:clojure.spec/problems [{:path [:ret],
:pred true?,
:val false,
:via [],
:in []}],
:clojure.spec.test/args (#object[clojure.spec$fspec_impl$reify_14282$fn_14285
0x7cf479f8
"clojure.spec$fspec_impl$reify_14282$fn_14285@7cf479f8"]),
:clojure.spec.test/val false,
:clojure.spec/failure :check-failed},
:via [{:type clojure.lang.ExceptionInfo,
:message "Specification-based check failed",
:data {:clojure.spec/problems [{:path [:ret],
:pred true?,
:val false,
:via [],
:in []}],
:clojure.spec.test/args (#object[clojure.spec$fspec_impl$reify_14282$fn_14285
0x7cf479f8
"clojure.spec$fspec_impl$reify_14282$fn_14285@7cf479f8"]),
:clojure.spec.test/val false,
:clojure.spec/failure :check-failed},
:at [clojure.core$ex_info
invokeStatic
"core.clj"
4725]}],
:trace [[clojure.core$ex_info
invokeStatic
"core.clj"
4725]
[clojure.spec.test$explain_check
invokeStatic
"test.clj"
279]
[clojure.spec.test$check_call
invokeStatic
"test.clj"
295]
[clojure.spec.test$quick_check$fn__13414
invoke
"test.clj"
308]
[clojure.lang.AFn
applyToHelper
"AFn.java"
154]
[clojure.lang.AFn
applyTo
"AFn.java"
144]
[clojure.core$apply
invokeStatic
"core.clj"
657]
[clojure.core$apply
invoke
"core.clj"
652]
[clojure.test.check.properties$apply_gen$fn_1500$fn_1501
invoke
"properties.cljc"
16]
[clojure.test.check.properties$apply_gen$fn__1500
invoke
"properties.cljc"
16]
[clojure.test.check.rose_tree$fmap
invokeStatic
"rose_tree.cljc"
78]
[clojure.test.check.rose_tree$fmap
invoke
"rose_tree.cljc"
74]
[clojure.test.check.generators$fmap$fn__629
invoke
"generators.cljc"
89]
[clojure.test.check.generators$gen_fmap$fn__579
invoke
"generators.cljc"
55]
[clojure.test.check.generators$call_gen
invokeStatic
"generators.cljc"
41]
[clojure.test.check.generators$call_gen
invoke
"generators.cljc"
38]
[clojure.test.check$quick_check
invokeStatic
"check.cljc"
62]
[clojure.test.check$quick_check
doInvoke
"check.cljc"
37]
[clojure.lang.RestFn
invoke
"RestFn.java"
425]
[clojure.lang.AFn
applyToHelper
"AFn.java"
156]
[clojure.lang.RestFn
applyTo
"RestFn.java"
132]
[clojure.core$apply
invokeStatic
"core.clj"
657]
[clojure.spec.gen$quick_check
invokeStatic
"gen.clj"
27]
[clojure.spec.gen$quick_check
doInvoke
"gen.clj"
27]
[clojure.lang.RestFn
applyTo
"RestFn.java"
137]
[clojure.core$apply
invokeStatic
"core.clj"
661]
[clojure.spec.test$quick_check
invokeStatic
"test.clj"
306]
[clojure.spec.test$check_1
invokeStatic
"test.clj"
334]
[clojure.spec.test$check$fn__13433
invoke
"test.clj"
410]
[clojure.core$pmap$fn_9385$fn_9386
invoke
"core.clj"
6897]
[clojure.core$binding_conveyor_fn$fn__6772
invoke
"core.clj"
2020]
[clojure.lang.AFn
call
"AFn.java"
18]
[java.util.concurrent.FutureTask
run
"FutureTask.java"
266]
[java.util.concurrent.ThreadPoolExecutor
runWorker
"ThreadPoolExecutor.java"
1142]
[java.util.concurrent.ThreadPoolExecutor$Worker
run
"ThreadPoolExecutor.java"
617]
[java.lang.Thread
run
"Thread.java"
745]]},
:seed 1489361392805,
:failing-size 0,
:num-tests 1,
:fail [(#object[clojure.spec$fspec_impl$reify_14282$fn_14285
0x7cf479f8
"clojure.spec$fspec_impl$reify_14282$fn_14285@7cf479f8"])],
:shrunk {:total-nodes-visited 0,
:depth 0,
:result #error{:cause "Specification-based check failed",
:data {:clojure.spec/problems [{:path [:ret],
:pred true?,
:val false,
:via [],
:in []}],
:clojure.spec.test/args (#object[clojure.spec$fspec_impl$reify_14282$fn_14285
0x7cf479f8
"clojure.spec$fspec_impl$reify_14282$fn_14285@7cf479f8"]),
:clojure.spec.test/val false,
:clojure.spec/failure :check-failed},
:via [{:type clojure.lang.ExceptionInfo,
:message "Specification-based check failed",
:data {:clojure.spec/problems [{:path [:ret],
:pred true?,
:val false,
:via [],
:in []}],
:clojure.spec.test/args (#object[clojure.spec$fspec_impl$reify_14282$fn_14285
0x7cf479f8
"clojure.spec$fspec_impl$reify_14282$fn_14285@7cf479f8"]),
:clojure.spec.test/val false,
:clojure.spec/failure :check-failed},
:at [clojure.core$ex_info
invokeStatic
"core.clj"
4725]}],
:trace [[clojure.core$ex_info
invokeStatic
"core.clj"
4725]
[clojure.spec.test$explain_check
invokeStatic
"test.clj"
279]
[clojure.spec.test$check_call
invokeStatic
"test.clj"
295]
[clojure.spec.test$quick_check$fn__13414
invoke
"test.clj"
308]
[clojure.lang.AFn
applyToHelper
"AFn.java"
154]
[clojure.lang.AFn
applyTo
"AFn.java"
144]
[clojure.core$apply
invokeStatic
"core.clj"
657]
[clojure.core$apply
invoke
"core.clj"
652]
[clojure.test.check.properties$apply_gen$fn_1500$fn_1501
invoke
"properties.cljc"
16]
[clojure.test.check.properties$apply_gen$fn__1500
invoke
"properties.cljc"
16]
[clojure.test.check.rose_tree$fmap
invokeStatic
"rose_tree.cljc"
78]
[clojure.test.check.rose_tree$fmap
invoke
"rose_tree.cljc"
74]
[clojure.test.check.generators$fmap$fn__629
invoke
"generators.cljc"
89]
[clojure.test.check.generators$gen_fmap$fn__579
invoke
"generators.cljc"
55]
[clojure.test.check.generators$call_gen
invokeStatic
"generators.cljc"
41]
[clojure.test.check.generators$call_gen
invoke
"generators.cljc"
38]
[clojure.test.check$quick_check
invokeStatic
"check.cljc"
62]
[clojure.test.check$quick_check
doInvoke
"check.cljc"
37]
[clojure.lang.RestFn
invoke
"RestFn.java"
425]
[clojure.lang.AFn
applyToHelper
"AFn.java"
156]
[clojure.lang.RestFn
applyTo
"RestFn.java"
132]
[clojure.core$apply
invokeStatic
"core.clj"
657]
[clojure.spec.gen$quick_check
invokeStatic
"gen.clj"
27]
[clojure.spec.gen$quick_check
doInvoke
"gen.clj"
27]
[clojure.lang.RestFn
applyTo
"RestFn.java"
137]
[clojure.core$apply
invokeStatic
"core.clj"
661]
[clojure.spec.test$quick_check
invokeStatic
"test.clj"
306]
[clojure.spec.test$check_1
invokeStatic
"test.clj"
334]
[clojure.spec.test$check$fn__13433
invoke
"test.clj"
410]
[clojure.core$pmap$fn_9385$fn_9386
invoke
"core.clj"
6897]
[clojure.core$binding_conveyor_fn$fn__6772
invoke
"core.clj"
2020]
[clojure.lang.AFn
call
"AFn.java"
18]
[java.util.concurrent.FutureTask
run
"FutureTask.java"
266]
[java.util.concurrent.ThreadPoolExecutor
runWorker
"ThreadPoolExecutor.java"
1142]
[java.util.concurrent.ThreadPoolExecutor$Worker
run
"ThreadPoolExecutor.java"
617]
[java.lang.Thread
run
"Thread.java"
745]]},
:smallest [(#object[clojure.spec$fspec_impl$reify_14282$fn_14285
0x7cf479f8
"clojure.spec$fspec_impl$reify_14282$fn_14285@7cf479f8"])]}},
:sym spike-spec.core/bar,
:failure #error{:cause "Specification-based check failed",
:data {:clojure.spec/problems [{:path [:ret],
:pred true?,
:val false,
:via [],
:in []}],
:clojure.spec.test/args (#object[clojure.spec$fspec_impl$reify_14282$fn_14285
0x7cf479f8
"clojure.spec$fspec_impl$reify_14282$fn_14285@7cf479f8"]),
:clojure.spec.test/val false,
:clojure.spec/failure :check-failed},
:via [{:type clojure.lang.ExceptionInfo,
:message "Specification-based check failed",
:data {:clojure.spec/problems [{:path [:ret],
:pred true?,
:val false,
:via [],
:in []}],
:clojure.spec.test/args (#object[clojure.spec$fspec_impl$reify_14282$fn_14285
0x7cf479f8
"clojure.spec$fspec_impl$reify_14282$fn_14285@7cf479f8"]),
:clojure.spec.test/val false,
:clojure.spec/failure :check-failed},
:at [clojure.core$ex_info invokeStatic "core.clj" 4725]}],
:trace [[clojure.core$ex_info invokeStatic "core.clj" 4725]
[clojure.spec.test$explain_check
invokeStatic
"test.clj"
279]
[clojure.spec.test$check_call
invokeStatic
"test.clj"
295]
[clojure.spec.test$quick_check$fn__13414
invoke
"test.clj"
308]
[clojure.lang.AFn applyToHelper "AFn.java" 154]
[clojure.lang.AFn applyTo "AFn.java" 144]
[clojure.core$apply invokeStatic "core.clj" 657]
[clojure.core$apply invoke "core.clj" 652]
[clojure.test.check.properties$apply_gen$fn_1500$fn_1501
invoke
"properties.cljc"
16]
[clojure.test.check.properties$apply_gen$fn__1500
invoke
"properties.cljc"
16]
[clojure.test.check.rose_tree$fmap
invokeStatic
"rose_tree.cljc"
78]
[clojure.test.check.rose_tree$fmap
invoke
"rose_tree.cljc"
74]
[clojure.test.check.generators$fmap$fn__629
invoke
"generators.cljc"
89]
[clojure.test.check.generators$gen_fmap$fn__579
invoke
"generators.cljc"
55]
[clojure.test.check.generators$call_gen
invokeStatic
"generators.cljc"
41]
[clojure.test.check.generators$call_gen
invoke
"generators.cljc"
38]
[clojure.test.check$quick_check
invokeStatic
"check.cljc"
62]
[clojure.test.check$quick_check
doInvoke
"check.cljc"
37]
[clojure.lang.RestFn invoke "RestFn.java" 425]
[clojure.lang.AFn applyToHelper "AFn.java" 156]
[clojure.lang.RestFn applyTo "RestFn.java" 132]
[clojure.core$apply invokeStatic "core.clj" 657]
[clojure.spec.gen$quick_check
invokeStatic
"gen.clj"
27]
[clojure.spec.gen$quick_check doInvoke "gen.clj" 27]
[clojure.lang.RestFn applyTo "RestFn.java" 137]
[clojure.core$apply invokeStatic "core.clj" 661]
[clojure.spec.test$quick_check
invokeStatic
"test.clj"
306]
[clojure.spec.test$check_1
invokeStatic
"test.clj"
334]
[clojure.spec.test$check$fn__13433
invoke
"test.clj"
410]
[clojure.core$pmap$fn_9385$fn_9386
invoke
"core.clj"
6897]
[clojure.core$binding_conveyor_fn$fn__6772
invoke
"core.clj"
2020]
[clojure.lang.AFn call "AFn.java" 18]
[java.util.concurrent.FutureTask
run
"FutureTask.java"
266]
[java.util.concurrent.ThreadPoolExecutor
runWorker
"ThreadPoolExecutor.java"
1142]
[java.util.concurrent.ThreadPoolExecutor$Worker
run
"ThreadPoolExecutor.java"
617]
[java.lang.Thread run "Thread.java" 745]]}})



 Comments   
Comment by Alex Miller [ 12/Mar/17 7:22 PM ]

The generated function uses the ret spec to generate results so I don't this does not seem like a goal.





[CLJ-2124] Catch multiple exceptions in a single catch block Created: 12/Mar/17  Updated: 12/Mar/17

Status: Open
Project: Clojure
Component/s: None
Affects Version/s: None
Fix Version/s: None

Type: Enhancement Priority: Minor
Reporter: Daniel Compton Assignee: Unassigned
Resolution: Unresolved Votes: 2
Labels: exceptions, try-catch

Approval: Triaged

 Description   

Java 7 and up support multi-catch exceptions (http://www.oracle.com/technetwork/articles/java/java7exceptions-486908.html). It would be handy if Clojure also supported them to prevent catching something like Exception and then writing manual logic to check the Exception type, or duplicating the logic over multiple catch blocks.

A possible syntax for this could be:

(try (fn-that-throws)
     (catch (UnknownHostException NoRouteToHostException) e
       (go-offline)))

Prior art for this is a try* macro: https://gist.github.com/Gonzih/5814945.

One nuance to handle is

Edit: Note that in Java 7, you cannot both catch ExceptionA& ExceptionB in the same time, if ExceptionB is inherited(directly or indirectly) from ExceptionA. Compiler will complain: The exception ExceptionB is already caught by the alternative ExceptionA. - http://stackoverflow.com/a/3495968/826486

I tried searching to see if this had been asked already, but got mountains of results. I didn't see anything in the first few pages though.






[CLJ-2123] Lighter-weight aliasing for keywords Created: 10/Mar/17  Updated: 10/Mar/17

Status: Open
Project: Clojure
Component/s: None
Affects Version/s: Release 1.9
Fix Version/s: Release 1.9

Type: Enhancement Priority: Major
Reporter: Alex Miller Assignee: Unassigned
Resolution: Unresolved Votes: 0
Labels: keywords, namespace

Approval: Vetted

 Description   

It is useful to make qualified keywords, and particularly so with spec. Using namespace aliases helps a lot in working with a lot of qualified keywords. However, currently creating an aliased namespace requires that the namespace actually exists.

This ticket is a placeholder to do something more with lighter-weight aliasing for keywords. Details TBD.






[CLJ-2122] flatten docstring does not describe lazy result Created: 07/Mar/17  Updated: 07/Mar/17

Status: Open
Project: Clojure
Component/s: None
Affects Version/s: Release 1.8
Fix Version/s: None

Type: Enhancement Priority: Trivial
Reporter: Alan Thompson Assignee: Unassigned
Resolution: Unresolved Votes: 0
Labels: docstring
Environment:

All


Approval: Triaged

 Description   

clojure.core/flatten uses tree-seq to return a lazy result. The lazy nature of the result is not described in the docstring.

Original docstring:
--------------------------------
"Takes any nested combination of sequential things (lists, vectors,
etc.) and returns their contents as a single, flat sequence.
(flatten nil) returns an empty sequence."

Proposed docstring:
--------------------------------
"Takes any nested combination of sequential things (lists, vectors,
etc.) and returns their contents as a single, flat lazy sequence.
(flatten nil) returns an empty sequence."



 Comments   
Comment by Alex Miller [ 07/Mar/17 6:28 PM ]

Seems reasonable.





[CLJ-2121] Parameter names in docstring for `pos?` Created: 04/Mar/17  Updated: 05/Mar/17  Resolved: 05/Mar/17

Status: Closed
Project: Clojure
Component/s: None
Affects Version/s: Release 1.1, Release 1.9
Fix Version/s: None

Type: Enhancement Priority: Minor
Reporter: Marc O'Morain Assignee: Unassigned
Resolution: Duplicate Votes: 0
Labels: docstring

Attachments: Text File 0001-CLJ-2121-update-docstring-to-reflect-param-name.patch    
Patch: Code
Approval: Triaged

 Description   

The docstring for `pos?` is as follows:

Usage: (pos? x)
Returns true if num is greater than zero, else false

This should be either:

Usage: (pos? num)
Returns true if num is greater than zero, else false

or

Usage: (pos? x)
Returns true if x is greater than zero, else false


 Comments   
Comment by Marc O'Morain [ 04/Mar/17 8:50 AM ]

(`neg?` and `zero?` have the same issue)

Comment by Alex Miller [ 04/Mar/17 10:58 AM ]

Patch welcome - change should update docstring, not arg.

Comment by Erik Assum [ 04/Mar/17 2:32 PM ]

Duplicate of CLJ-1859?

Comment by Alex Miller [ 05/Mar/17 7:08 PM ]

Closed as duplicate





[CLJ-2120] (for) works in REPL, but not in a file Created: 28/Feb/17  Updated: 28/Feb/17  Resolved: 28/Feb/17

Status: Closed
Project: Clojure
Component/s: None
Affects Version/s: Release 1.8
Fix Version/s: None

Type: Defect Priority: Major
Reporter: Vadim Liventsev Assignee: Unassigned
Resolution: Completed Votes: 0
Labels: None
Environment:

elemenary OS 0.4 (Ubuntu 16.04.2 LTS / Linux 4.4.0-64-generic)
Java HotSpot(TM) 64-Bit Server VM 1.8.0_121-b13
Loading clojure 1.8.0 with Leiningen



 Description   

I have the following file:

for.clj
(println "for.clj loaded")
(for [fruit ["apple" "orange" "watermelon"]] (println fruit))

REPL is running in the same folder.
Calling

(for [fruit ["apple" "orange" "watermelon"]] (println fruit))
yields the expected result:

apple
orange
watermelon
(nil nil nil)

Calling

(load "for")
yields:

for.clj loaded
nil

The same happens when the file is loaded by Leiningen. It is also not limited to just (println), I have a project with a bunch of (for), it works in REPL, but grinds to a halt when loaded from a file.



 Comments   
Comment by Alex Miller [ 28/Feb/17 7:12 AM ]

for is lazy and won't evaluate its body unless something uses the resulting lazy seq. In the repl, the printing will do so but when you load, nothing will be doing the forcing.

You can use (doall (for ...)) to force the lazy seq around the for to be evaluated. Also see dorun and run! for some other fns that are used to force side effects.





[CLJ-2119] clojure.core/extends? inconsistent (erroroneous) between 1.7, 1.8/1.9.0-alpha14 Created: 28/Feb/17  Updated: 01/Mar/17  Resolved: 28/Feb/17

Status: Closed
Project: Clojure
Component/s: None
Affects Version/s: Release 1.8
Fix Version/s: None

Type: Defect Priority: Major
Reporter: Tom S Assignee: Unassigned
Resolution: Declined Votes: 0
Labels: bug, extends?
Environment:

windows 7/ 10
java version "1.8.0_111"
Java(TM) SE Runtime Environment (build 1.8.0_111-b14)
Java HotSpot(TM) 64-Bit Server VM (build 25.111-b14, mixed mode)



 Description   

Up to clojure 1.7, clojure.core/extends? served as a predicate as an alternative to the much much slower clojure.core/satisfies? pred for relatively quick type-based operations. I recently migrated to 1.8, and testing showed this no longer holds; clojure.core/extends? appears to be unable to detect that an object actually extends a protocol.

Simple example (1.7):

;user=> (defn ikv? [obj] (extends? clojure.core.protocols/IKVReduce (type obj)))
;#'user/ikv?
;user=> (ikv? {:a 2})
;true

Called against a clojure.lang.PersistentArrayMap, which implements IKVReduce in its java class definition, this makes sense.

The same example reports false on clojure 1.8 (and 1.9.0-alpha14).
However, clojure.core/satisfies? is true in 1.8 (and 1.9.0-alpha14).

My hacked workaround is to use Alex Miller's example of memoizing the call to satisfies? (slightly slower than extends? but reasonable for my use cases.

However, the fundamental behavior of extends? seems to be broken, which may be a deeper issue going forward. Plus, it's a otherwise "silent" error, since it happily returns false, leading to silent, possibly obscure runtime errors.



 Comments   
Comment by Alex Miller [ 28/Feb/17 7:30 AM ]

I think you're relying on implementation details that are subject to change.

extends? can only catch protocol extensions that happen in the type definition, not ones that are applied later. So checking a protocol with satisfies? is the correct way to do it.

That said, I'll double check and make sure nothing is amiss.

Comment by Alex Miller [ 28/Feb/17 8:50 AM ]

Yeah, this change in what you're seeing is due to the refactoring in https://github.com/clojure/clojure/commit/684cd4117bb2cf463c95293855a0dff52134bdcd. Again, the fact that extends? worked before was relying on unreliable details of the specific way things are implemented and what you're really trying to check is something about the protocol, which requires satisfies?.

Comment by Tom S [ 01/Mar/17 4:11 AM ]

I completely missed the fact that clojure.lang.IKVReduce popped up during
the refactor, and is used as a fast-path for classes that implement it
during kv-reduce.

I chased my tail trying to figure out why maps don't extend
clojure.core.protocols/IKVReduce, which is simply because the interface
clojure.core.protocols.IKVReduce is no longer implemented, leaving
isAssignAbleFrom reflection call to return false now, while providing a
protocol implementation for clojure.lang.IKVReduce. The protocol
implementation delegates to the class's implementation of
clojure.lang.IKVReduce if possible.

So, satisfies? traces through superclasses trying to find a protocol
implementation, which is far more durable (depending on the protocol)
rather than class-specific interface implementations I was using to
cheat out a fastpath.

Just for grins, here's the microbenchmark associated with the "fast path"
vs cached method lookup:
(def the-map {:a 2})

;;note the clojure.lang.KVReduce
(defn ikv? [x] (instance? clojure.lang.KVReduce x))

(let [cache (java.util.HashMap.)]
(defn sat? [protocol x]
(let [c (class x)]
(if-let [res (.get cache c)] ;we know.
res
(let [res (satisfies? protocol x)
_ (.put cache c res)]
res)))))

;;Microbenchmarks.

;;(time (dotimes [i 1000000] (sat? clojure.core.protocols/IKVReduce the-map)))
;;"Elapsed time: 13.31948 msecs"

;;(time (dotimes [i 1000000] (ikv? the-map))
;;"Elapsed time: 6.816777 msecs"

Negligible cost for caching (preferably using a concurrent map in production).

Thanks for tracking down the refactoring change. Sorry for the false alarm.

Comment by Tom S [ 01/Mar/17 4:43 AM ]

;typo correction.
(defn ikv? [x] (instance? clojure.lang.IKVReduce x))





[CLJ-2118] extend-type doesn't identify that a protocol is a protocol Created: 23/Feb/17  Updated: 23/Feb/17  Resolved: 23/Feb/17

Status: Closed
Project: Clojure
Component/s: None
Affects Version/s: Release 1.8
Fix Version/s: None

Type: Defect Priority: Major
Reporter: Maurício Eduardo Chicupo Szabo Assignee: Unassigned
Resolution: Declined Votes: 0
Labels: protocols
Environment:

Ubuntu 16.04 x86_64, OpenJDK 8



 Description   

I'm trying to implement this tutorial: http://www.lucagrulla.com/posts/server-sent-events-with-ring-and-compojure/

In one part, I need to extend a core.async type with the following protocol: https://github.com/ring-clojure/ring/blob/master/ring-core/src/ring/core/protocols.clj#L19.

So, I've tried to implement something like this:

src/async_test/protocol_ext.clj
(ns async-test.protocol-ext
  (:require [clojure.core.async :as async]
            [clojure.java.io :as io])
  (:import [ring.core.protocols StreamableResponseBody]
           [clojure.core.async.impl.channels ManyToManyChannel]))

(extend-type ManyToManyChannel
  StreamableResponseBody
  (write-body-to-stream [channel response output-stream]
                        (async/go (with-open [writer (io/writer output-stream)]
                                    (async/loop []
                                                (when-let [msg (async/<! channel)]
                                                  (doto writer (.write msg) (.flush))
                                                  (recur)))))))

But this fails with interface ring.core.protocols.StreamableResponseBody is not a protocol.

Then, I tried to monkey-patch ring: opened a new file with the correct path, and added the following lines on the bottom of the protocol declaration, inside `extend-protocol` call:

src/ring/core/protocols.clj
ManyToManyChannel
  (write-body-to-stream [channel response output-stream]
                        (async/go (with-open [writer (io/writer output-stream)]
                                    (async/loop []
                                                (when-let [msg (async/<! channel)]
                                                  (doto writer (.write msg) (.flush))
                                                  (recur)))))))

Then, it worked. What's happening? Is Clojure ignoring that StreamableResponseBody is a protocol if there's already an `extend-protocol` call?



 Comments   
Comment by Maurício Eduardo Chicupo Szabo [ 23/Feb/17 3:00 PM ]

Okay, my bad. Seems that I need to `:require` the protocol, not `:import` it. Sorry about that...

Comment by Alex Miller [ 23/Feb/17 3:04 PM ]

The issue here is that you are importing StreamableResponseBody as a Java class (really an interface). While the protocol does generate a Java interface, you should use the protocol, not the interface, when you extend. This is confusing because they both have the same name.

So, your ns declaration should instead be:

(ns async-test.protocol-ext
  (:require [clojure.core.async :as async]
            [clojure.java.io :as io]
						[ring.core.protocols :refer [StreamableResponseBody]])
  (:import [clojure.core.async.impl.channels ManyToManyChannel]))

The monkeypatch you tried worked because you were properly referring to the protocol in that case.





[CLJ-2117] Support typical polymorphic maps in spec more directly Created: 23/Feb/17  Updated: 23/Feb/17  Resolved: 23/Feb/17

Status: Closed
Project: Clojure
Component/s: None
Affects Version/s: Release 1.9
Fix Version/s: None

Type: Enhancement Priority: Major
Reporter: Leon Grapenthin Assignee: Unassigned
Resolution: Declined Votes: 0
Labels: spec


 Description   

Problem: A common scenario is that one wants to spec this:

(def a-poly-map
  {::type :t1
   <kvs required/optional in t1>
   <kvs required/optional in all>
   })

To do this, you usually write lots of boiler plate code like this:

(s/def ::type #{:t1 <more-types>})

(defmulti type-dispatch ::type)
(defmethod type-dispatch :t1 [_] (s/keys :req [<t1-keys>]))
<more defmethods for more-types>

(s/def ::poly-map (s/merge (s/keys :req [::type <poly-map-keys-in-all>])
                           (s/multi-spec type-dispatch ::type))

As a workaround one can (and we did) of course write a macro to write all that code.

From my experience this usecase is the 90% multi-spec usecase and thus I'd prefer if a more convenient to use spec existed.

Proposed solution: Implement this spec:

(s/def ::poly-map (s/merge (s/keys :req [<poly-map-keys-in-all>]
                           (s/poly-map ::type
                             :t1 {:req [<t1-keys>]} ; :req, :opt etc. as in s/keys
                             <more requirements for more types>)))

Limitations: This syntax does not support to reuse existing specs in for dispatch values (:t1, :t2, ...) intentionally, because it seems an non-existing usecase to spec incomplete s/keys for such maps separately anyway.

Further enhancement: While the <poly-map-keys-in-all> usecase is rare, this syntax could also directly support it:

(s/def ::poly-map (s/poly-map ::type
                    {:req [<poly-map-keys-in-all>]} ; optional
                    :t1 {:req [<t1-keys>]}
                    <more requirements for more types>))

This new spec would automatically check for the presence of ::type and conform/explain like a regular s/keys.



 Comments   
Comment by Alex Miller [ 23/Feb/17 11:50 AM ]

Thanks, but I don't think we expect to do anything like this.

Comment by Leon Grapenthin [ 23/Feb/17 12:03 PM ]

Why?

Comment by Alex Miller [ 23/Feb/17 12:51 PM ]

Because spec provides the tools to do it already.

Comment by Leon Grapenthin [ 23/Feb/17 1:55 PM ]

This ticket is asking for a syntactic simplification of a typical use case. I understand that there already is s/multi-spec. s/multi-spec however is a generic and in comparison quite advanced tool that is not tied to maps. Thus it requires a substantial amount of boilerplate to "do it".

After much practical experience with spec in my observation - and I do assume others would agree - this is the most common application for it.

There are many things in clojure.core which aren't the only "tools to do it" for the exact same reason, so please reconsider not closing this right away.

Comment by Alex Miller [ 23/Feb/17 2:32 PM ]

My job is to weigh the balance of several factors and decide if an enhancement request seems reasonably likely to remain as an open ticket and at some point be further evaluated.

Some factors I try to think about:

  • Is this a good problem? Does it describe a pain point that many people are likely to encounter? You claim it is "most common" use but seems way too early to judge this (from what I've seen, I wouldn't agree right now).
  • Is the pain significant enough that it prevents you from doing your work? In this case, clearly not - you've already built it and it would be easy to release it independently from core.
  • Is this a fundamental (simple) feature that belongs in core? Clojure has a small library and wishes to keep it that way. There are ample examples where this constraint was not considered enough in Clojure. From what I can tell, this is appears to be a composite of existing things, not a fundamental part.
  • If we were to solve this problem in core, is this the proposed solution a likely way we would do it? Based on things we've talked about during dev, my suspicion is, probably not.

On balance, it does not seem to be something worth leaving open in the tracker right now. But as the saying goes, "no is temporary, yes is forever". If a while down the road it's clear that this is a prominent problem that needs attention, there is nothing preventing us from considering it again (presumably with more data and experience at that time).

Comment by Leon Grapenthin [ 23/Feb/17 3:11 PM ]

Thanks for explaining the decision. I understand that you have priorities and probably a better plan to address this problem in the future.





[CLJ-2116] Support for selective conforming with clojure.spec Created: 22/Feb/17  Updated: 28/Feb/17

Status: Open
Project: Clojure
Component/s: None
Affects Version/s: Release 1.9
Fix Version/s: None

Type: Enhancement Priority: Major
Reporter: Tommi Reiman Assignee: Unassigned
Resolution: Unresolved Votes: 5
Labels: spec
Environment:

[org.clojure/clojure "1.9.0-alpha14"]



 Description   

Problem

using clojure.spec in runtime border validation supporting multiple exchange formats is hard.

Details

Currently in clojure.spec (alpha-14), conformers are attached to Spec instances at creation time and they are invoked on every conform. This is not very useful in system border validation, where conforming/coercion functions should be selected based on runtime data, e.g. the exchange format.

Examples:

  • a keyword? spec:
    • with EDN, no coercion should be done (it can present Keywords)
    • with JSON, String->Keyword coercion should be applied
    • with String-based formats (CSV, query-params, ...), String->Keyword coercion should be applied
  • a integer? spec:
    • with EDN, no coercion should be done (it can present numbers)
    • with JSON, no coercion should be done (it can present numbers)
    • with String-based formats (CSV, query-params, ...), String->Long coercion should be applied

Here is a more complete example:

(s/def ::id integer?)
(s/def ::name string?)
(s/def ::title keyword?)
(s/def ::person (s/keys :opt [::id], :req-un [::name ::title]))

;; this is how we see the data over different exchange formats
(def edn-person {::id 1, :name "Tiina", :title :boss})
(def json-person {::id 1, :name "Tiina", :title "boss"})
(def string-person {::id "1", :name "Tiina", :title "boss"})

;; here's what we want
(def conformed-person edn-person)

To use this today, one needs to manually create new border specs with different conformers for all different exchange formats. Non-qualified keywords could be mapped in s/keys to work (e.g. ::title => ::title$JSON), but this wont work if fully qualified keys are exposed over the border (like ::id in the example) - one can't register multiple, differently conforming version of the spec with same name.

Suggestion

Support selective conforming in the Spec Protocol with a new 3-arity conform* and clojure.spec/conform, both taking a extra user-provided callback/visitor function. If the callback is provided, it's called from within the Specs conform* with the current spec as argument and it will return either nil or a 2-arity conformer function that should be used for the actual confrom.

Actual conforming-matcher implementations can be maintained in 3rd party libraries, like spec-tools[1].

Using it would look like this:

;; edn
(assert (= conformed-person (s/conform ::person edn-person)))
(assert (= conformed-person (s/conform ::person edn-person nil)))

;; json
(assert (= conformed-person (s/conform ::person json-person json-conforming-matcher)))

;; string
(assert (= conformed-person (s/conform ::person string-person string-conforming-matcher)))

Alternative

Another option to support this would be to allow Specs to be extended with Protocols. 3rd party libs could have a new Conforming protocol with 3-arity conform and add implementations for it on all current specs. Currently this is not possible.

[1] https://github.com/metosin/spec-tools



 Comments   
Comment by Alex Miller [ 22/Feb/17 3:33 PM ]

I don't think we are interested in turning spec into a transformation engine via conformers, so I suspect we are probably not interested. However, I'll leave it for Rich to assess.

Comment by Tommi Reiman [ 23/Feb/17 1:26 AM ]

Currently, Plumatic Schema is the tool used at the borders. Now, people are starting to move to Spec and it would really bad for the Clojure Web Developement Story if one had to use two different modelling libraries for their apps. If Spec doesn't want to be a tranformation engine via conformers, I hope for the Alternative suggestion to allow 3rd parties to write this kind of extensions: exposing Specs as Records/Types instead of reified protocols would do the job?

Comment by ken restivo [ 28/Feb/17 9:43 PM ]

I could see why the Clojure core developers might not want Spec to support this kind of coercion, but the practical reality is that someone will have to. If it isn't in Spec itself, it'll have to be done libraries built upon it like Tommi's.

The use case here is: I have a conf file that is YAML. I'm parsing the YAML using a Clojure library, turning it into a map. Now I have to validate the map, but YAML doesn't support keywords, for example, and the settings structure goes directly into Component/Mount/etc as part of the app state, so it makes sense to run s/conform on it as the first step in app startup after reading configuration. Add to this the possibility of other methods of merging in configuration (env vars, .properties files, etc) and this coercion will be necessary somewhere.





[CLJ-2115] Support data conveying conform errors (alternative/complement to :clojure.spec/invalid) Created: 22/Feb/17  Updated: 22/Feb/17

Status: Open
Project: Clojure
Component/s: None
Affects Version/s: Release 1.9
Fix Version/s: None

Type: Enhancement Priority: Major
Reporter: Max Penet Assignee: Unassigned
Resolution: Unresolved Votes: 5
Labels: spec


 Description   

At the moment if a conform calls fails (returning :clojure.spec/invalid) there is no way to supply extra information about why it failed. We do have the possibility to get explain-data, but at best this would return a spec form, the original value and some metadata.

While this is fine in most cases, some conformer functions upon failure can provide extra data that'd be useful to the consumer. A practical example we had was a spec that can contain values for a String based DSL (think SQL like), that would conform these values to their parsed AST. When the conform wrapped function fails it would throw an ex-info with line/col info and more metadata about the failure. But all this data was lost since we can only return :clojure.spec/invalid. All this happened inside a rule engine schema, that can contain hundreds of these; re-parsing all the failing values for error reporting is something we wanted to avoid.

The proposal would be to to support a new return value that'd allow conveying data about the conform failure, or to support both this new value and :clojure.spec/invalid.
This could take the form of (explain-info {..}) potentially returned by conformer function for later consumption by explain, to match clojure semantics with exceptions (ex-info/ex-data, explain-info/explain-data).

A more naive implementation could just allow to throw inside the conformer function and have the error merged/assoced into the explain map (but that might be a bit too invasive in my opinion).






[CLJ-2114] ::defn-args spec incorrectly parses map body as a prepost rather than function body Created: 16/Feb/17  Updated: 14/Mar/17  Resolved: 14/Mar/17

Status: Closed
Project: Clojure
Component/s: None
Affects Version/s: Release 1.9
Fix Version/s: Release 1.9

Type: Defect Priority: Major
Reporter: Nicola Mometto Assignee: Unassigned
Resolution: Completed Votes: 0
Labels: spec

Attachments: Text File 0001-CLJ-2144-conform-map-fn-bodies-as-body-rather-than-a.patch    
Patch: Code
Approval: Ok

 Description   

Reported by Claire Alvis in #clojure-spec:

user> (s/conform :clojure.core.specs/defn-args
                 '(foo [bar] {:baz 42}))
{:name foo, :bs [:arity-1 {:args {:args [[:sym bar]]}, :prepost {:baz 42}}]}

The current spec conforms function bodies with single maps as prepost conditions rather than function bodies, after the patch:

user=> (s/conform :clojure.core.specs/defn-args '(foo [bar] {:baz 42}))
{:name foo, :bs [:arity-1 {:args {:args [[:sym bar]]}, :body [:body [{:baz 42}]]}]}
user=> (s/conform :clojure.core.specs/defn-args '(foo [bar] {:baz 42} 1))
{:name foo, :bs [:arity-1 {:args {:args [[:sym bar]]}, :body [:prepost+body {:prepost {:baz 42}, :body [1]}]}]}

Patch: 0001-CLJ-2144-conform-map-fn-bodies-as-body-rather-than-a.patch

Screened by: Alex Miller



 Comments   
Comment by Nicola Mometto [ 16/Feb/17 5:42 PM ]

An open question is whether we also want to make `:prepost` stricter as part of this patch, so that it will ensure that `:pre` and `:post` are a collection





[CLJ-2113] Update Clojure maven for latest on CI server Created: 16/Feb/17  Updated: 14/Mar/17  Resolved: 14/Mar/17

Status: Closed
Project: Clojure
Component/s: None
Affects Version/s: None
Fix Version/s: Release 1.9

Type: Enhancement Priority: Critical
Reporter: Alex Miller Assignee: Unassigned
Resolution: Completed Votes: 0
Labels: build

Attachments: Text File mvn3.patch    
Patch: Code
Approval: Ok

 Description   

Update maven build infrastructure to stop using oss-parent and use updated Maven 3 and sonatype plugins (similar to other changes made recently in all contrib projects).

  • Removed oss-parent parent pom. This has been deprecated for years and is no longer recommended for use.
  • Add snapshot repo (was previously pulled in via oss-parent)
  • maven-compiler-plugin - update to latest version
  • maven-release-plugin - update to latest version
  • add nexus-staging-maven-plugin - current recommended plugin for releases to maven central, replaces most of the maven-release-plugin's work (old version of this previously in oss-parent)
  • add maven-gpg-plugin for signing (previously in oss-parent)
  • remove old release profile which was activated by oss-parent pom

Patch: mvn3.patch

It's difficult to test this completely outside the context of actually building and deploying snapshots and releases but the changes are very similar to those made for all contrib projects recently.






[CLJ-2112] Add specs for spec forms Created: 15/Feb/17  Updated: 15/Feb/17

Status: Open
Project: Clojure
Component/s: None
Affects Version/s: Release 1.9
Fix Version/s: Release 1.9

Type: Enhancement Priority: Major
Reporter: Alex Miller Assignee: Alex Miller
Resolution: Unresolved Votes: 11
Labels: spec

Attachments: Text File spec-forms.patch    
Patch: Code
Approval: Incomplete

 Description   

It would be useful to have specs that described spec forms, such that it was possible to go from a spec form like (s/keys :req [::a ::b] :opt [::c]) to a conformed version that allowed you to grab the parts without parsing the s-expression. This can be done by creating specs, thus allowing:

user=> (require '[clojure.spec :as s] '[clojure.spec.specs])
user=> (s/def ::aspec (s/keys :req [::a ::b] :opt [::c]))
user=> (def aspec-data (s/conform :clojure.spec.specs/spec (s/form ::aspec)))
user=> (pr aspec-data)
[:form {:s clojure.spec/keys, 
        :args {:req [[:key :clojure.spec.specs/a] [:key :clojure.spec.specs/b]], 
               :opt [:clojure.spec.specs/c]}}]
user=> (map val (-> aspec-data val :args :req))
(:clojure.spec.specs/a :clojure.spec.specs/b)

Patch: spec-forms.patch (a work in progress)






[CLJ-2111] s/every :kind does not work as intended Created: 14/Feb/17  Updated: 14/Feb/17

Status: Open
Project: Clojure
Component/s: None
Affects Version/s: Release 1.9
Fix Version/s: None

Type: Defect Priority: Major
Reporter: Leon Grapenthin Assignee: Unassigned
Resolution: Unresolved Votes: 0
Labels: spec

Approval: Vetted

 Description   
user=> (s/valid? (s/every number? :kind (s/or :vector vector? :list list?))
          [])
ClassCastException clojure.spec$or_spec_impl$reify__13891 cannot be cast to clojure.lang.IFn  dev/eval44499/fn--44501 (form-init3178965928127409998.clj:22)

user=> (pst *e)
ClassCastException clojure.spec$or_spec_impl$reify__13903 cannot be cast to clojure.lang.IFn
	user/eval20/fn--22 (NO_SOURCE_FILE:13)
	clojure.spec/every-impl/reify--14039 (spec.clj:1225)
	clojure.spec/valid? (spec.clj:744)
	clojure.spec/valid? (spec.clj:740)

Expected: true



 Comments   
Comment by Alex Miller [ 14/Feb/17 5:06 PM ]

Certainly a function like this works (s/every number? :kind #(or (vector? %) (list? %))). The question is whether the s/every doc that states "pred/spec" means only a predicate function or "predicate function OR spec". I'm not sure what the intention was. Certainly the code in every seems to be wrapping the kind into a function and then invoking it in every-impl, so it's not written to accept a spec currently.

Comment by Alex Miller [ 14/Feb/17 5:06 PM ]

Marking vetted to either resolve, update docstring, or decline. Need more info from Rich.





[CLJ-2110] sorted map returns nil value for existing key Created: 14/Feb/17  Updated: 14/Feb/17  Resolved: 14/Feb/17

Status: Closed
Project: Clojure
Component/s: None
Affects Version/s: Release 1.8
Fix Version/s: None

Type: Defect Priority: Major
Reporter: jonnik Assignee: Unassigned
Resolution: Declined Votes: 0
Labels: sorted-map
Environment:

Oracle Java 1.8.0_112



 Description   

Then i try to get value by key from sorted map with comparator by value it returns nil.

(def tmap {1 {:v 1} 2 {:v 2} 3 {:v 3}})
(def tmap-sorted (apply sorted-map-by #(let [val-comp (- (compare (get-in tmap [%1 :v]) (get-in tmap [%2 :v])))]
                                         (if (= val-comp 0) 1 val-comp))
                        (flatten (vec tmap))))
; => {3 {:v 3} 2 {:v 2} 1 {:v 1}}
(get tmap-sorted 3)
;=> nil

Expected: {:v 3}
Actual: nil



 Comments   
Comment by Andy Fingerhut [ 14/Feb/17 11:31 AM ]

Your comparison function never returns 0, by the way you have written it. If it never returns 0, it will never 'think' that it has found an equal element when searching for a match using 'get'. By replacing (if (= val-comp 0) 1 val-comp) with val-comp, the get calls will work:

(def tmap-sorted (apply sorted-map-by #(let [val-comp (- (compare (get-in tmap [%1 :v]) (get-in tmap [%2 :v])))]
                                         val-comp)
                        (flatten (vec tmap))))
tmap-sorted
; => {3 {:v 3}, 2 {:v 2}, 1 {:v 1}}
(get tmap-sorted 3)
; => {:v 3}

You are getting a descending sorted order by negating the return value of compare. I would recommend to follow the advice on this page: https://clojure.org/guides/comparators particularly "Reverse the sort by reversing the order that you give the arguments to an existing comparator." That helps avoid corner cases with some integer values.

I would also recommend (into my-empty-sorted-map tmap) in place of your (apply my-empty-sorted-map (flatten (vec tmap)). Putting all of those recommendations together would result in code like this:

(def tmap-sorted2 (into (sorted-map-by #(compare (get-in tmap [%2 :v]) (get-in tmap [%1 :v])))
                        tmap))
tmap-sorted2
; => {3 {:v 3}, 2 {:v 2}, 1 {:v 1}}
(get tmap-sorted2 3)
; => {:v 3}
Comment by saintech [ 14/Feb/17 11:41 AM ]

Ok. But how about this?:

tmap-sorted
; => {3 {:v 3} 2 {:v 2} 1 {:v 1}}
(first tmap-sorted)
; => [3 {:v 3}]
(get tmap-sorted 3)
;=> nil

Is it OK?

Comment by Andy Fingerhut [ 14/Feb/17 12:43 PM ]

I believe that calling clojure.core/first on a sorted-map does not cause its comparison function to be called at all. It is already stored in a sorted tree in memory, and first just finds the earliest one.

clojure.core/get does call the comparison function, perhaps several times, to find an item with an equal key. The original comparison function given in the description never returns equality (i.e. the integer 0) when comparing two items.

Comment by Alex Miller [ 14/Feb/17 1:32 PM ]

Agreed with Andy, declining





[CLJ-2109] Protocol methods not instrumented Created: 13/Feb/17  Updated: 13/Feb/17

Status: Open
Project: Clojure
Component/s: None
Affects Version/s: Release 1.9
Fix Version/s: None

Type: Defect Priority: Minor
Reporter: Stuart Sierra Assignee: Unassigned
Resolution: Unresolved Votes: 1
Labels: spec
Environment:

Clojure 1.9.0-alpha14


Approval: Triaged

 Description   

Spec instrument does not work on protocol methods. Invalid arguments will be accepted silently with no error. Protocol vars are included in the return value of (instrument).

Steps to reproduce

(require
 '[clojure.spec :as s]
 '[clojure.spec.test :as test])

(defprotocol P
  (method [this arg]))

(defrecord R []
  P
  (method [this arg]
    (str "R.method called with " (pr-str arg))))

(s/fdef method
  :args (s/cat :this any?
               :arg number?))

(defn wrapped [this arg]
  (method this arg))

(s/fdef wrapped
  :args (s/cat :this any?
               :arg number?))

(test/instrument)

(println (method (->R) "not a number"))

(println (wrapped (->R) "not a number"))

This code produces the output:

R.method called with "not a number"
clojure.lang.ExceptionInfo: Call to #'user/wrapped did not conform to spec:
In: [1] val: "not a number" fails at: [:args :arg] predicate: number?
...

Possible resolutions

1. Add support to instrument for protocol methods
2. Document that instrument does not work on protocol methods, do not return protocol method Vars from (instrument), throw exception if protocol method Vars are included in the symbols passed to (instrument syms)

See also

CLJ-1941 describes a different case where instrument does not work. This issue was identified in a comment.

Workarounds

This issue can be avoided by wrapping protocol methods in normal functions and spec'ing the functions. This is already common practice.






[CLJ-2108] Loading core specs affects startup time Created: 13/Feb/17  Updated: 13/Feb/17

Status: Open
Project: Clojure
Component/s: None
Affects Version/s: Release 1.9
Fix Version/s: Release 1.9

Type: Enhancement Priority: Major
Reporter: Alex Miller Assignee: Alex Miller
Resolution: Unresolved Votes: 0
Labels: spec

Approval: Vetted

 Description   

Adding the loading of spec itself + the clojure.core.specs namespace containing specs for core makes start time worse (and will only get longer as we add more core specs).



 Comments   
Comment by Ghadi Shayban [ 13/Feb/17 4:58 PM ]

Locally,
alpha14 takes 1.02s

time java -jar clojure-1.9.0-alpha14.jar -e ':foo'

Master with this line commented out takes 0.92s

time java -jar target/clojure-1.9.0-master-SNAPSHOT.jar -e ':foo'




[CLJ-2107] s/explain of a predicate defined s/with-gen yields s/unknown Created: 07/Feb/17  Updated: 08/Feb/17  Resolved: 08/Feb/17

Status: Closed
Project: Clojure
Component/s: None
Affects Version/s: Release 1.9
Fix Version/s: None

Type: Defect Priority: Major
Reporter: Tamas Herman Assignee: Unassigned
Resolution: Duplicate Votes: 0
Labels: generator, spec


 Description   

This explanation is clear:

(s/def ::some-string string?)
(s/explain ::some-string 123)

val: 123 fails spec: :boot.user/some-string predicate: string?

However if the failing spec has a custom generator, the explanation is not so helpful:

(s/def ::some-string-with-custom-generator (s/with-gen string? gen/string-alphanumeric))
(s/explain ::some-string-with-custom-generator nil)

=> :boot.user/some-string-with-custom-generator
val: nil fails spec: :boot.user/some-string-with-custom-generator predicate: :clojure.spec/unknown

I would expect the same predicate: string? explanation.

Based on the symptom I suspect this issue is related to http://dev.clojure.org/jira/browse/CLJ-2068



 Comments   
Comment by Tamas Herman [ 08/Feb/17 2:21 AM ]

Explanation of a workaround from Slack from @hiredman:

with-gen is a function, so it's arguments are evaluated, so the string? argument to with-gen is a function object, when spec is trying to find the name to report it does some stuff, which for symbols and keywords reports a good name, but for other Objects (including function objects) you get :clojure.spec/unknown.

wrapping with s/spec allows the spec macro to capture the meaningful, the symbol before evaluation:

(s/def ::some-string-with-custom-generator (s/with-gen (s/spec string?) gen/string-alphanumeric))
(s/explain ::some-string-with-custom-generator nil)

val: nil fails spec: :boot.user/some-string-with-custom-generator predicate: string?

I don't see any simple solution to make the original example yield a good explanation,
so maybe we should just explain how it works in https://clojure.org/guides/spec#_custom_generators
and wrap the with-gen examples with spec.

Comment by Alex Miller [ 08/Feb/17 8:49 AM ]

As you mentioned, this is a dup of CLJ-2068 and is fixed by the patch there.





[CLJ-2106] satisfies? is quite slow when returning false Created: 06/Feb/17  Updated: 09/Feb/17  Resolved: 09/Feb/17

Status: Closed
Project: Clojure
Component/s: None
Affects Version/s: Release 1.8
Fix Version/s: None

Type: Enhancement Priority: Minor
Reporter: Mike Kaplinskiy Assignee: Unassigned
Resolution: Duplicate Votes: 0
Labels: performance


 Description   
(defprotocol SatisfiesTest (testThing [this]))
(defrecord MyRecord [] Object SatisfiesTest (testThing [this]))
(def r (MyRecord.))

(time (dotimes [_ 30000] (instance? user.SatisfiesTest r)))
"Elapsed time: 1.715 msecs"

(time (dotimes [_ 30000] (satisfies? SatisfiesTest r)))
"Elapsed time: 3.944 msecs"

(time (dotimes [_ 30000] (instance? user.SatisfiesTest {})))
"Elapsed time: 2.304 msecs"

(time (dotimes [_ 30000] (satisfies? SatisfiesTest {})))
"Elapsed time: 718.949 msecs"

It would be nice if satisfies? memoized negative return values by class (though that cache would need to be cleared by `extend-type` and friends).



 Comments   
Comment by Alex Miller [ 06/Feb/17 1:56 PM ]

This is covered in an existing ticket - http://dev.clojure.org/jira/browse/CLJ-1814

Comment by Alex Miller [ 06/Feb/17 2:02 PM ]

Actually, read too quickly - CLJ-1814 covers the positive case only. I don't think we're going to cache a negative return value though. Managing and cleaning that cache is likely to not be worth it. If you need this kind of thing, you should probably consider a different kind of conditional check before-hand.

Comment by Alex Miller [ 06/Feb/17 2:05 PM ]

For example, you can just memoize this call like this:

user=> (def check (memoize #(satisfies? SatisfiesTest %)))
#'user/check
user=> (time (dotimes [_ 30000] (check {})))
"Elapsed time: 3.512 msecs"
Comment by Alex Miller [ 06/Feb/17 2:13 PM ]

I tweaked the test to remove the use of range and the construction of the record so the numbers are more useful.

Comment by Nicola Mometto [ 08/Feb/17 6:52 PM ]

Alex Miller CLJ-1814 handles the negative cases too, and doesn't keep a cache of negative return values. It keeps a cache of all the protocols extended by a certain class and invalidates that cache on every call to extend, the benchmarks on the ticket description showcase both a positive and a negative test

Comment by Alex Miller [ 09/Feb/17 3:09 PM ]

Nicola - cool! I didn't realize that. Will mark this as a dupe then.





[CLJ-2105] incorrect spec conform when an optional (?) is inside of a one or more (+) Created: 03/Feb/17  Updated: 03/Feb/17

Status: Open
Project: Clojure
Component/s: None
Affects Version/s: Release 1.9
Fix Version/s: None

Type: Defect Priority: Major
Reporter: Anthony D'Ambrosio Assignee: Unassigned
Resolution: Unresolved Votes: 0
Labels: spec
Environment:

1.9.0-alpha14


Approval: Vetted

 Description   
(s/def ::thing (s/cat :a (s/? string?) :b (s/+ number?))) 
(s/def ::seq-of (s/+ ::thing))

(s/conform ::seq-of (list "foo" 1 "bar" 2 3 "qux" 4)) 
;=> [{:a "foo", :b [1]} [{:a "bar", :b [2 3]} {:a "qux", :b [4]}]]

;EXPECTED 
;=> [{:a "foo", :b [1]} {:a "bar", :b [2 3]} {:a "qux", :b [4]}]

;; ONLY 2nd thing matters? 
(s/conform ::seq-of (list "foo" 1 2 "bar" 3)) 
;=> [{:a "foo", :b [1 2]} {:a "bar", :b [2]}]

;; NO OPTIONAL 
(s/def ::thing (s/cat :a string? :b (s/+ number?))) 
(s/def ::seq-of (s/+ ::thing))
(s/conform ::seq-of (list "foo" 1 "bar" 2 3 "qux" 4)) 
;=> [{:a "foo", :b [1]} {:a "bar", :b [2 3]} {:a "qux", :b [4]}]

This also only shows up if there are 2+ numbers in the 2nd or later ::thing
AND the problem goes away if I make :a not optional...

Could be related to http://dev.clojure.org/jira/browse/CLJ-2003 ?






[CLJ-2104] Typo in pprint docstring Created: 29/Jan/17  Updated: 29/Jan/17

Status: Open
Project: Clojure
Component/s: None
Affects Version/s: Release 1.9
Fix Version/s: None

Type: Enhancement Priority: Major
Reporter: Alex Miller Assignee: Unassigned
Resolution: Unresolved Votes: 0
Labels: docstring, typo

Attachments: Text File clj-2104.patch    
Patch: Code
Approval: Prescreened

 Description   

Typo in pprint ns docstring: "complete documentation on the the clojure web site on github"






[CLJ-2103] s/coll-of and s/every take unnecessary long to generate if :into not provided Created: 28/Jan/17  Updated: 17/Feb/17

Status: Open
Project: Clojure
Component/s: None
Affects Version/s: Release 1.9
Fix Version/s: None

Type: Defect Priority: Major
Reporter: Leon Grapenthin Assignee: Unassigned
Resolution: Unresolved Votes: 1
Labels: generator, spec
Environment:

alpha14



 Description   
(s/def ::bar (s/coll-of number? :kind vector?))

(defn foo [a]
  (reduce + 0 a))

(s/fdef foo
  :args (s/cat :a ::bar)
  :ret number?)

(first (stest/check `foo))

Doesn't terminate in a reasonable amount of time.
One observes that changing :gen-max doesn't affect computation time.

One observes that it can be fixed by adding :into [] to the s/coll-of spec.

The reason is this: If :into is not provided, s/every and s/coll-of has to generate a vector (via gen of vector?) to call empty on it, to then fill it up. This is quite clearly documented in the docstring of `s/every`:

:kind - a pred/spec that the collection type must satisfy, e.g. vector?
         (default nil) Note that if :kind is specified and :into is
         not, this pred must generate in order for every to generate.

Assumedly the vector? generates quite large vectors at a certain point which significantly slows down the generation.

The responsible code is in gen* of every-impl

(gen/bind
 (cond
  gen-into (gen/return (empty gen-into))
  kind (gen/fmap #(if (empty? %) % (empty %))
                 (gensub kind overrides path rmap form))  ;; creating and mapping gen of :kind
 :else (gen/return []))
 (fn [init]
    ....


 Comments   
Comment by Leon Grapenthin [ 14/Feb/17 3:03 PM ]

I see two approaches to improve this behavior:

1. The gen uses the gen of kind to generate one value with the smallest size, calls empty on it to determine :into. This would lead to a surprise when your :kind is e. g. (s/or :a-vec vector? :a-list list?) (which currently throws, anyway)
2. We use an internal lookup table to assume :into. {clojure.core/vector? [], clojure.core/set? #{} ...}





[CLJ-2102] Reduce collection generator default size from 20 Created: 25/Jan/17  Updated: 07/Apr/17

Status: Open
Project: Clojure
Component/s: None
Affects Version/s: Release 1.9
Fix Version/s: Release 1.9

Type: Enhancement Priority: Major
Reporter: Alex Miller Assignee: Unassigned
Resolution: Unresolved Votes: 3
Labels: generator, spec

Attachments: Text File clj-2102-2.patch     Text File clj-2102.patch    
Patch: Code
Approval: Vetted

 Description   

In general I find that it is very easy (especially with nested or recursive collections) to have a check run OOME due to generating very large nested collections. Currently the default is 20 - I think we should change it to 3.

The attached patch just changes the default from 20 to 3. An alternate approach would be to change it to a dynvar setting.

Patch: clj-2102-2.patch



 Comments   
Comment by Alex Miller [ 07/Apr/17 3:45 PM ]

Updated patch to apply to master





[CLJ-2101] Request for a way to forget specs Created: 24/Jan/17  Updated: 25/Jan/17  Resolved: 24/Jan/17

Status: Closed
Project: Clojure
Component/s: None
Affects Version/s: Release 1.9
Fix Version/s: None

Type: Enhancement Priority: Trivial
Reporter: Johan Gall Assignee: Unassigned
Resolution: Duplicate Votes: 0
Labels: spec


 Description   

Hello,

I have a server that loads code, and specs, at runtime in generated namespaces.

Thus I have a spec leak.

It would be nice if there was an official API to forget specs.



 Comments   
Comment by Johan Gall [ 24/Jan/17 7:01 AM ]

(or better, a way to create a temporary spec world, but then that is probably not trivial)

Comment by Alex Miller [ 24/Jan/17 7:51 AM ]

Is this covered by http://dev.clojure.org/jira/browse/CLJ-2060 ?

Comment by Alex Miller [ 24/Jan/17 12:51 PM ]

Marking as a dup for now, but will re-open if this goes beyond what's in CLJ-2060.

Comment by Johan Gall [ 25/Jan/17 5:20 AM ]

ah sorry, indeed it's a dup





[CLJ-2100] s/nilable form should include the original spec, not the resolved spec Created: 19/Jan/17  Updated: 14/Mar/17  Resolved: 14/Mar/17

Status: Closed
Project: Clojure
Component/s: None
Affects Version/s: Release 1.9
Fix Version/s: Release 1.9

Type: Defect Priority: Major
Reporter: Alex Miller Assignee: Unassigned
Resolution: Completed Votes: 0
Labels: spec

Attachments: Text File clj-2100.patch    
Patch: Code
Approval: Ok

 Description   
=> (clojure.spec/def :test/name string?)
:test/name

=> (clojure.spec/form (clojure.spec/and :test/name))
(clojure.spec/and :test/name)

=> (clojure.spec/form (clojure.spec/nilable :test/name))
(clojure.spec/nilable clojure.core/string?)

In the final line, s/nilable form has the resolved spec rather than the original spec.

Proposed: Instead of getting the internal spec description, resolve the original spec form.

After:

user=> (clojure.spec/form (clojure.spec/nilable :test/name))
(clojure.spec/nilable :test/name)

Patch: CLJ-2100.patch






[CLJ-2099] Keywords with aliased namespaces cannot be read when the namespace is required in a reader conditional Created: 13/Jan/17  Updated: 25/Jan/17  Resolved: 13/Jan/17

Status: Closed
Project: Clojure
Component/s: None
Affects Version/s: Release 1.9
Fix Version/s: None

Type: Defect Priority: Minor
Reporter: Viktor Magyari Assignee: Unassigned
Resolution: Declined Votes: 0
Labels: keywords, reader, readerconditionals


 Description   

The title in itself isn't entirely true, but I couldn't find a way to describe it succintly (feel free to change).

The issue is easier to demonstrate with an example:

(ns foo
  #?(:cljs (:require [clojure.core :as c])))

#?(:cljs ::c/x)

When reading this in a :clj context, the reader cannot read ::c/x ("Invalid token: ::c/x"), despite the code being correct (presumably).
The same thing happens if the reader conditional branches are :clj and the source is read in a :cljs context.



 Comments   
Comment by Alex Miller [ 13/Jan/17 8:07 AM ]

This looks like expected behavior to me. Auto-resolved keywords rely on a resolution context and there isn't one at the point where ::c/x is being read.

Comment by Viktor Magyari [ 13/Jan/17 9:05 AM ]

To me it seems reasonable to expect the resolution context to include the clojure.core alias - more generally, include <platform> specific aliases in the <platform> branches of reader conditionals. Maybe consider this as an enhancement ticket?

Comment by Alex Miller [ 13/Jan/17 9:18 AM ]

To do this would require adding special handling specifically for ns or other things that create aliases, which implies conditional evaluation of some forms at read-time. You would also need some place (other than the current platform's namespace maps) to track other platform's namespace aliases. That's a lot of very special, custom stuff.

We're not going to add this.

Comment by Leon Grapenthin [ 25/Jan/17 2:13 PM ]

1. Why does the reader try to read the :cljs branch in Viktors example?
2. The original intent of reader conditionals was that the code could be read independently of the lang. Have aliases not been considered then due to a lack of popularity of ::a/n syntax?

My suggestion would be a knob that reads unresolvable alias kws as a special object #aliased-keyword{:alias "a", :name "n"}. This would solve both Viktors problem and also pay its debt to the reader conditional intent.

Comment by Alex Miller [ 25/Jan/17 5:16 PM ]

1. The reader reads everything, it's just selective about which parts of an expression gets returned. Without reading, it wouldn't know where the end of that form was.

2. I think this is a little more subtle in that reading the second thing in a conditional requires remembering something read and discarded in a prior conditional.

That said, maybe it would be possible to either read in a mode where this particular case doesn't throw or where this particular exception was known and discarded when inside a conditional.





[CLJ-2098] autodoc fails to load clojure/spec.clj Created: 12/Jan/17  Updated: 07/Apr/17

Status: Open
Project: Clojure
Component/s: None
Affects Version/s: Release 1.9
Fix Version/s: Release 1.9

Type: Defect Priority: Critical
Reporter: Alex Miller Assignee: Unassigned
Resolution: Unresolved Votes: 0
Labels: spec

Attachments: Text File clj-2098.patch    
Approval: Vetted

 Description   

Reported by Tom F for the autodoc process. The following (essentially) is what autodoc is doing that currently blows up:

tom@renoir:~/src/clj/autodoc-work-area/clojure/src$ java -cp clojure.jar clojure.main
Clojure 1.9.0-master-SNAPSHOT
user=> (load-file "src/clj/clojure/spec.clj")
CompilerException java.lang.IllegalArgumentException: No implementation of method: :conform* \
of protocol: #'clojure.spec/Spec found for class: clojure.spec$regex_spec_impl$reify__14279, \
compiling:(/home/tom/src/clj/autodoc-work-area/clojure/src/src/clj/clojure/spec.clj:684:1)

Cause: There is code in Compiler.macroexpand() with the intention to suspend spec checking inside clojure.spec. However, it is currently doing an exact path match on SOURCE_PATH. In the load-file call above, this ends up being an absolute path.

Approach: Check for a path suffix rather than an exact match in Compiler.

Patch: clj-2098.patch



 Comments   
Comment by Alex Miller [ 07/Apr/17 3:38 PM ]

Removing "code" tag for patch as this is not good to screen yet. While the patch kind of works, it would be good if there were a more reliable way to omit spec checking in the scope of the spec namespace. Rather than checking the source path, it would be much better to instead check the current ns. This works EXCEPT during the macro expansion of ns (which has a spec) in spec itself. At that point you have not yet evaluated the (in-ns 'clojure.spec) (since that's what the ns macro is expanding to). Really, all of this is a problem at exactly one point - compiling clojure.spec itself. Another option might be some way to suspend macro spec checking altogether (which might be a generally useful feature).





[CLJ-2097] Improve generation failure exception Created: 10/Jan/17  Updated: 11/Jan/17

Status: Open
Project: Clojure
Component/s: None
Affects Version/s: None
Fix Version/s: None

Type: Enhancement Priority: Major
Reporter: Russell Mull Assignee: Unassigned
Resolution: Unresolved Votes: 0
Labels: generator, spec


 Description   

It's pretty easy to write a spec whose generator fails like this:

Couldn't satisfy such-that predicate after 100 tries.

This is of course expected in many ways, but it's a very unhelpful error. Some things that could make this better include:

  • Including the spec that failed in the exception. I only see one invocation of gen/such-that in spec.clj, and it appears to have the spec's form at hand. gen/such-that takes an exception constructor where this could be used.
  • Allow max-tries to be changed from the hardcoded value of 100. When dealing with an intermittent failure, it can be useful to crank down max-tries to a very small number, making the failure easier to reproduce.


 Comments   
Comment by Alex Miller [ 11/Jan/17 8:41 AM ]

These are reasonable suggestions and this area is likely to evolve in tandem with test.check to provide better info.





[CLJ-2096] "Key must be integer" thrown when vector lookup done with short or byte Created: 04/Jan/17  Updated: 06/Jan/17  Resolved: 05/Jan/17

Status: Closed
Project: Clojure
Component/s: None
Affects Version/s: Release 1.8, Release 1.9
Fix Version/s: None

Type: Defect Priority: Major
Reporter: Aaron Cummings Assignee: Unassigned
Resolution: Declined Votes: 0
Labels: None
Environment:

clojure-1.9.0-alpha14 and clojure-1.8.0 tested.



 Description   

Looking up a value in a vector with a long or integer index works:
([:a :b :c] (long 1)) => :b
([:a :b :c] (int 1)) => :b

However, lookups with short or byte fail with "IllegalArgumentException Key must be integer"
([:a :b :c] (short 1))
([:a :b :c] (byte 1))

Root cause seems to be clojure.lang.Util.isInteger() which returns true only if the argument is an instance of Integer, Long, clojure.lang.BigInt, or BigInteger. I think it would be more correct for clojure.lang.Util.isInteger() to be consistent with clojure.core/integer? which additionally returns true for both Short and Byte.



 Comments   
Comment by Alex Miller [ 05/Jan/17 8:24 AM ]

I don't see any good reason to make changes in this area so declining.

Comment by Aaron Cummings [ 05/Jan/17 8:44 AM ]

Hi Alex - the case where we ran into this exception was in parsing a binary file where the record descriptor is a byte, and we used this byte to do a lookup in a vector (the vector holding keywords which describe the record type). The workaround is pretty simple though; just cast the byte to an int.

Curiously, a map lookup like ({0 :a, 1 :b, 2 :c} (byte 1)) does work.

I wondering though if the non-support of short and byte lookups in vectors is intentional, and if so, the reason for this choice (I don't see any obvious problems, so perhaps Rich knows something I don't here). If instead this is an oversight, and is deemed not broken enough to fix, then I can accept that.

Comment by Alex Miller [ 06/Jan/17 8:26 AM ]

I would say that given that the check and error message exists, it is intentional. Certainly there is a performance impact to checking for a broader set of types.





[CLJ-2095] Doc s/gen overrides do not take effect inside custom generators Created: 03/Jan/17  Updated: 03/Jan/17

Status: Open
Project: Clojure
Component/s: None
Affects Version/s: Release 1.9
Fix Version/s: None

Type: Enhancement Priority: Minor
Reporter: Russell Mull Assignee: Unassigned
Resolution: Unresolved Votes: 0
Labels: docstring, generator, spec
Environment:

clojure 1.9.0-alpha14



 Description   

Custom generators may build (via fmap/bind) on spec generators. Generator overrides at the top level will not take effect inside custom generators:

(require '[clojure.spec :as s])
(require '[clojure.test.check.generators :as gen])

;; A map that holds a single integer value
(s/def ::val integer?)
(s/def ::body (s/keys :req [::val]))

;; This spec matches stringified versions of 'body'.
;; (read-string is for demonstration purposes only, of course)
(s/def ::stringy-body
  (s/with-gen
    (s/and string? #(s/valid? ::body (read-string %)))
    #(gen/fmap pr-str (s/gen ::body))))

(s/valid? ::stringy-body "{:user/val 37}") ;; => true

;; This makes various stringified maps, as expected
(take 3 (gen/sample (s/gen ::stringy-body)))
;; => ("#:user{:val -1}" "#:user{:val 0}" "#:user{:val -1}")

;; *** But the overrides don't get passed through ***
(take 3 (gen/sample (s/gen ::stringy-body {::val #(s/gen #{42})})))
;; ("#:user{:val -1}" "#:user{:val 0}" "#:user{:val 0}")

Should consider documenting this in s/gen, s/with-gen, etc.



 Comments   
Comment by Alex Miller [ 03/Jan/17 5:39 PM ]

When you use with-gen, you're basically overriding the built-in gen mechanism (which supports overrides) and providing your own (opaque to spec) generator. You should not expect overrides to take effect inside a custom generator.

Comment by Russell Mull [ 03/Jan/17 5:41 PM ]

That makes sense, but in lieu of that I expected (and went looking for) some way to get at the overrides map from the function passed to s/with-gen, and found none.

Comment by Russell Mull [ 03/Jan/17 5:48 PM ]

... I didn't fully parse your comment the first time around. I can see from the implementation that a custom generator (gfn internally) is never passed any of the contextual information that the builtin specs have at hand. As it sounds like this is intentional, it would be useful to note this limitation in the docstring for s/gen or perhaps s/with-gen.

Comment by Alex Miller [ 03/Jan/17 5:58 PM ]

It's not a crazy idea, but it doesn't seem like there's any way this could be done in the current impl without some pretty significant changes.





[CLJ-2094] clojure.spec: bug with protocols and extend-type Created: 01/Jan/17  Updated: 03/Jan/17  Resolved: 03/Jan/17

Status: Closed
Project: Clojure
Component/s: None
Affects Version/s: Release 1.9
Fix Version/s: None

Type: Defect Priority: Major
Reporter: John Schmidt Assignee: Unassigned
Resolution: Declined Votes: 0
Labels: spec


 Description   

I have the following two clj files (I've tried to come up with a minimal example):

core.clj
--------------
(ns spec-test.core
  (:require [clojure.spec :as s]))

(defprotocol Game
  (move [game]))

(s/def ::game1 #(satisfies? Game %))
(s/def ::game2 (partial satisfies? Game))

foo.clj
--------------
(ns spec-test.foo
  (:require [spec-test.core :refer [Game]]))

(defrecord Foo [])

(extend-type Foo
  Game
  (move [game]))

Here's a REPL session that explains my problem:

user=> (ns spec-test.core)
nil
spec-test.core=> (require 'spec-test.core :reload)
nil
spec-test.core=> (require 'spec-test.foo :reload)
nil
spec-test.core=> (satisfies? Game (spec-test.foo/->Foo))
true
spec-test.core=> ((partial satisfies? Game) (spec-test.foo/->Foo))
true
spec-test.core=> (s/explain ::game1 (spec-test.foo/->Foo))
Success!
nil
spec-test.core=> (s/explain ::game2 (spec-test.foo/->Foo))
val: #spec_test.foo.Foo{} fails spec: :spec-test.core/game2 predicate: (partial satisfies? Game) <---- WAAAAAT
nil

I would expect ::game1 and ::game2 to be equivalent, but somehow they're not.

Also see: https://groups.google.com/forum/#!topic/clojure/igBlMpqGU3A



 Comments   
Comment by Steve Miner [ 02/Jan/17 12:43 PM ]

I gave some incorrect advice on the mailing list so I'll try to correct myself here. Basically, the protocol is stored in a var, Game. If a predicate captures a particular state of the protocol, it won't respond correctly when additions are made to the protocol (with extend-type, etc.) So the problem with the usage of `partial` is that it evaluates the current value of the protocol at that point in time, before it has been extended to cover Foo.

The #(...) function definition would have used the var itself, not its current value. Naturally, the var allows the protocol to be updated such that the function sees the updated value. Basically, this is just normal Clojure evaluation. A `defn` style function would have worked fine as well. It's just the `partial` that evaluated its args that leads to the problem.

The same kind of issue could come up if you passed any var to partial and captured the current value of the var. Later changes to the var would not affect the result of partial.

I'll say the bug is the confusing error message. It seems that s/explain is implying it is using the var Game, but it really captured an "old" value of Game.

Comment by Alex Miller [ 03/Jan/17 2:10 PM ]

I think Steve's assessment is all correct.

I'm not sure that it's possible for spec to know what's happened here though wrt giving a better error message.

I don't think I really see anything that can be "fixed", so I'm going to mark this as declined.





[CLJ-2093] partial and fn behave differently with eval Created: 28/Dec/16  Updated: 31/Dec/16  Resolved: 31/Dec/16

Status: Closed
Project: Clojure
Component/s: None
Affects Version/s: None
Fix Version/s: None

Type: Defect Priority: Major
Reporter: N/A Assignee: Unassigned
Resolution: Declined Votes: 0
Labels: None


 Description   

(eval (list map (partial + 1) [0]))
CompilerException java.lang.ExceptionInInitializerError
(eval (list map (fn [x] (+ 1 x)) [0]))
=> (1)



 Comments   
Comment by Kevin Downey [ 29/Dec/16 9:22 PM ]

same issue as http://dev.clojure.org/jira/browse/CLJ-1206 and all the issues connected to that

Comment by Alex Miller [ 31/Dec/16 11:17 AM ]

You should not expect either of these to work as the expressions contain function objects (not function forms).

You should be doing something like this:

(eval (list 'map '(partial + 1) [0]))




[CLJ-2092] deftype instances with mutable fields cannot be compiled Created: 24/Dec/16  Updated: 31/Dec/16

Status: Open
Project: Clojure
Component/s: None
Affects Version/s: None
Fix Version/s: None

Type: Defect Priority: Major
Reporter: Herwig Hochleitner Assignee: Unassigned
Resolution: Unresolved Votes: 0
Labels: compiler, deftype

Approval: Triaged

 Description   

When evaluating or compiling an implementer of clojure.lang.IType, the compiler tries to reflectively access its fields. This fails, when a field is marked mutable (hence private):

Clojure 1.9.0-master-SNAPSHOT
user=> (deftype T [^:unsynchronized-mutable t])
user.T
user=> (T. :t)
#object[user.T 0x2654635 "user.T@2654635"]
user=> (eval (T. :t))
CompilerException java.lang.IllegalArgumentException: No matching field found: t for class user.T
            Reflector.java:  271  clojure.lang.Reflector/getInstanceField
             Compiler.java: 4724  clojure.lang.Compiler$ObjExpr/emitValue
             Compiler.java: 4851  clojure.lang.Compiler$ObjExpr/emitConstants
             Compiler.java: 4529  clojure.lang.Compiler$ObjExpr/compile
             Compiler.java: 4049  clojure.lang.Compiler$FnExpr/parse
             Compiler.java: 6866  clojure.lang.Compiler/analyzeSeq
             Compiler.java: 6669  clojure.lang.Compiler/analyze
             Compiler.java: 6924  clojure.lang.Compiler/eval
             Compiler.java: 6890  clojure.lang.Compiler/eval
                  core.clj: 3105  clojure.core/eval
...

For classes that don't implement IType, no such problem exists.

user> (deftype* user/U user.U
        [^:unsynchronized-mutable u]
        :implements [])
nil
user> (eval (user.U. :u))
#object[user.U 0x34699051 "user.U@34699051"]

This problem commonly occurs, when implementing a tagged literal for a deftype with cached hash.



 Comments   
Comment by Alex Miller [ 31/Dec/16 12:01 PM ]

Yeah, this is interesting. The compiler compiles a deftype into a call to the constructor with the current values of the fields, but mutable fields are not accessible. One alternative would be to provide some standard method to "read" the field set rather than relying on reflection. (Another would be changing the access modifiers for mutable fields but I think that's probably a non-starter.)





[CLJ-2091] clojure.lang.APersistentVector#hashCode is not thread-safe Created: 24/Dec/16  Updated: 18/Apr/17

Status: Open
Project: Clojure
Component/s: None
Affects Version/s: None
Fix Version/s: Release 1.9

Type: Defect Priority: Major
Reporter: thurston n Assignee: Unassigned
Resolution: Unresolved Votes: 0
Labels: collections, concurrency

Attachments: File clj-2091-0.diff     File clj-2091-default-initialization.diff    
Patch: Code
Approval: Ok

 Description   

Problem: clojure.lang.APersistentVector#hashCode contain a deliberate data race on hash computation. However, the code as written does not follow safe practices for the intended data race. Specifically, the problem arises because the hashCode() (and hasheq()) method make multiple reads of the (unsynchronized) _hash field. The JMM permits these reads to return different values. Specifically, the last read in the return may return the pre-computed value -1, which is not the desired hash value. This problem also applies to APersistentMap, APersistentSet, and PersistentQueue.

See: http://jeremymanson.blogspot.com/2008/12/benign-data-races-in-java.html for a good description of the problem.

Fix: The main fix is to read the cached hash field only once and return the value of the local computation, not the value of the field.

A secondary change that is also beneficial is to use the default initializer value (which has special ordering in the JMM to the beginning of the thread) rather than setting and using -1 as the sentinel value.

In both cases these changes follow the canonical idioms used in java.lang.String for lazy hash computation. The patch covers both.

Patch: clj-2091-default-initialization.diff - note that this patch will indicate whitespace errors when applied due to the wacky line endings in PersistentQueue. The problem here is really the PQ formatting, not the patch.

Prescreened by: Alex Miller

There are some hash-related tests already but I also spot-checked that hash computations are returning the same value with and without the patch for the collections in question.



 Comments   
Comment by thurston n [ 24/Dec/16 4:38 PM ]

I can of course provide a patch, but as this is my first issue and am generally unfamiliar with clojure's practices and because this issue is not restricted to APersistentVector#hashCode, I thought it best to hold off and let the stewards decide how to proceed and how I could best help

Comment by Alex Miller [ 31/Dec/16 12:47 PM ]

Patch welcome (but please sign the contributor's agreement first - http://clojure.org/community/contributing). Also, see the processes for developing patches at http://dev.clojure.org/display/community/Developing+Patches.

AFAICT, the affected classes are APersistentVector, APersistentMap, APersistentSet, and PersistentQueue? Would be ok by me to cover all of them in a single patch.

Comment by thurston n [ 03/Jan/17 4:00 PM ]

AFAICT, the affected classes are APersistentVector, APersistentMap, APersistentSet, and PersistentQueue?

  • Dunno. However my experience tells me that the broken idiom (racy cache/memoization) is likely elsewhere; but I know of no systematic way of finding them. Regardless, I'll just focus on those 4 classes.
  • My plan is to also fix #hasheq(). It's the same problem; if you don't want that then just let me know and I'll refrain.
  • I'm not planning to deal with the initialization of #_hash and #_hasheq (currently inline-initialized to -1); that's a separate (although related) thread-safety problem. Might they be just what we refer to as a "legacy idiosyncrasy"? If so, then they really should be changed to just be default-initialized. I did, as an experiment, change one to default initialization, and the tests passed - that should be enough, but given that the persistent classes are serializable, code-coverage, et al., I can't say for sure. So I'll leave it to others more familiar with the codebase to make that determination. I note that if it is in future determined to change them, then #hashCode() and #hasheq() will need to be modified (trivially) accordingly.

That's the plan. Sound good?

Comment by Alex Miller [ 03/Jan/17 9:49 PM ]

I thought the non-default initialization was part of what you were describing, so now I'm not sure we're on the same page. Maybe you can just patch one so we have something concrete to talk about.

Comment by thurston n [ 03/Jan/17 9:56 PM ]

I'm not sure what you mean by "patch one" - I just submitted a patch, did you look at that?

Comment by Alex Miller [ 03/Jan/17 10:13 PM ]

I meant one class - sorry, I didn't see the patch. I will look at it tomorrow with fresh eyes.

Comment by thurston n [ 03/Jan/17 11:02 PM ]

Sure.

To be clear, as I mentioned in today's earlier comment, I would advise removing the inline-initialization, viz.

int _hash = -1;
int _hasheq = -1;

with

int _hash;
int _hasheq;

As I wrote, the extant tests would pass (of course, changing #hashCode() and #hasheq() appropriately)

But the initialization issue is a different, although certainly not orthogonal, issue than the one my patch addresses.

Currently, (i.e. pre-patch), #hashCode() can return a spurious -1 even if an APersistentVector instance is safely published - my patch fixes that.

However, because of the inline-initialization, an APersistentVector instance that is not safely published could return a spurious 0 from #hashCode(), even with my patch.

Now if the inline-initialization is just a "legacy idiosyncrasy" (and we all do that at one time or another), then it could be safely replaced (along with the appropriate modification to my patch) and all APersistentVector instances (safely published or not), would have #hashCode() implementations that are correct.

Comment by Alex Miller [ 06/Jan/17 3:19 PM ]

Ok, I went over all this again and it makes sense to me. I think you should proceed and also make the initializer change (remove the -1 as sentinel and replace with no initializer and 0 for the comparison checks in the methods).

Comment by thurston n [ 06/Jan/17 11:36 PM ]

Combines the 2 commits into a single commit patch
So incorporates the original patch changes (single read) with default initialization and checks for zero
Don't know what to do with PersistentQueue's mixed line-endings – that I'll leave to you to deal with

Comment by thurston n [ 09/Jan/17 8:16 PM ]

Problem also in core.lang.ASeq#hashCode() and core.lang.ASeq#hasheq() - although thankfully without inline initialization

Surely not the last place either

Comment by Alex Miller [ 10/Jan/17 8:33 AM ]

Feel free to update the patch if you like





[CLJ-2090] Improve clojure.core/distinct perf by using transient set Created: 23/Dec/16  Updated: 04/Jan/17

Status: Open
Project: Clojure
Component/s: None
Affects Version/s: None
Fix Version/s: None

Type: Enhancement Priority: Major
Reporter: Nikita Prokopov Assignee: Unassigned
Resolution: Unresolved Votes: 0
Labels: performance, transducers, transient

Attachments: Text File clj-2090-use-transient-set-in-distinct-2.patch    
Patch: Code
Approval: Triaged

 Description   

Current implementation of clojure.core/distinct uses persistent set. This patch improves performance of lazy arity by ~25%-30% and transducer by ~40%-50% by using transient set instead.

10 elements
(doall (distinct coll)) 	 5.773439 µs => 4.179092 µs (-27%)
(into [] (distinct) coll) 	 3.238236 µs => 1.943254 µs (-39%)

100 elements
(doall (distinct coll)) 	 67.725764 µs => 42.129993 µs (-37%)
(into [] (distinct) coll) 	 35.702741 µs => 16.495947 µs (-53%)

1000 elements
(doall (distinct coll)) 	 540.652739 µs => 399.053873 µs (-26%)
(into [] (distinct) coll) 	 301.423077 µs => 164.025500 µs (-45%)

10000 elements
(doall (distinct coll)) 	 3.439137 ms => 3.058872 ms (-11%)
(into [] (distinct) coll) 	 1.437390 ms => 848.277178 µs (-40%)

Benchmarking code: https://gist.github.com/tonsky/97dfe1f9c48eccafc983a49c7042fb21



 Comments   
Comment by Alex Miller [ 23/Dec/16 8:52 AM ]

You can't remove the volatile - you still need that for safe publication in multi threaded transducing contexts.

Comment by Nikita Prokopov [ 23/Dec/16 11:50 AM ]

Alex Miller How do you mean?

  • I don’t update seen link because transient set can be mutated in-place
  • Are transducers meant to be used from multiple threads? Because even existing implementation clearly has race condition. I imagine fixing that would be costly (we’ll need a synchronized section), so maybe it should be a specialized transducer that you use only when needed?
Comment by Alex Miller [ 23/Dec/16 12:26 PM ]

Transient sets can NOT be mutated in place - you must use the return value.

Yes, transducers are used from multiple threads in (for example) transducer chans in core.async go blocks.

Comment by Alex Miller [ 23/Dec/16 12:28 PM ]

I should also say transducers are not expected to be used from more than one thread at a time, so there are no race problems. But being used from multiple threads over time requires proper safe publication.

Comment by Nikita Prokopov [ 24/Dec/16 3:07 AM ]

But being used from multiple threads over time requires proper safe publication.

Does that imply that no transients could be used in transducers (because underlying arrays on which transient impl is based are mutated in place, so different threads could potentially see different states of transient object)?

Does that also mean that partition-by and partition-all should be fixed (they use java.util.ArrayList which, being array of references, has no safe publication semantics)?

Transient sets can NOT be mutated in place - you must use the return value.

I was thinking that clojure/core.clj and clojure.lang.ATransientSet.java are both part of Clojure internals, colocated, so can share a little bit of internal knowledge about each other. It seems safe to do that, because that knowledge does not leak outside, and, if at any point impl of ATransientSet would change, core.clj could be updated accordingly in the same release. I wouldn’t do that in any third-party library, of course.

Comment by Alex Miller [ 24/Dec/16 9:13 AM ]

Does that imply that no transients could be used in transducers (because underlying arrays on which transient impl is based are mutated in place, so different threads could potentially see different states of transient object)?

Transients require only that they are asked by no more than a single thread at a time and so are safe to use in a transducer. However, they should guarantee safe publication. core.async channels already do this as an artifact of their implementation, but other transducing contexts may not.

Transients should NEVER be used as "mutate in place", regardless of concurrency. While they will appear to "work" in some circumstances, this is never correct (eventually an update operation will return a new instance and if you are mutating in place, your data will then be missing). This is discussed and correct examples are shown at http://clojure.org/reference/transients.

Does that also mean that partition-by and partition-all should be fixed (they use java.util.ArrayList which, being array of references, has no safe publication semantics)?

That's something Rich and I are discussing but, probably.

Comment by Nikita Prokopov [ 24/Dec/16 12:56 PM ]

Alex Miller Here’s quick test that shows that changes to transient set (which is nothing more but a wrapper around transient map) made in one thread are not always visible from another thread.

https://gist.github.com/tonsky/62a7ec6d539fc013186bee2df0812cf6

That means that if we try to use transients for e.g. distinct it will miss duplicate items

Comment by Nikita Prokopov [ 24/Dec/16 1:02 PM ]

Removed transients from transducer arity of distincts because transducers might be accessed from multiple threads

Comment by Nikita Prokopov [ 24/Dec/16 1:12 PM ]

Maybe that doc http://clojure.org/reference/transients should be updated re: transients are not safe to use from multiple threads because changes made by one thread are not necessarily visible to another. Even if they don’t compete

Comment by Alex Miller [ 31/Dec/16 12:54 PM ]

I would say that test is demonstrating a bug in transient sets/maps and you should file a ticket for that as it's a lot more important than this enhancement.

distinct should be able to use transients in both the transducer and lazy seq impls. The issue with contains? not working on transients is actually a separate ticket - http://dev.clojure.org/jira/browse/CLJ-700 that will likely require some class hierarchy rearrangement. I don't think we would take this change until that is fixed (so that you can avoid relying on the class and Java method variants).

Comment by Nikita Prokopov [ 04/Jan/17 11:47 AM ]

I have to admit my test was demonstrating something else: there were no proper thread isolation. So it was a concurrency issue, not “safe publication” issue. My current understanding is this:

Transients require thread isolation. Use of a particular transient instance should be controlled either by using it in an single-threaded scope, or in a framework that enforces this.

That guarantee implicitly presumes that there’s happens-before relation between transient usage from multiple threads. There’s no other way to define “only one thread is in this section at a time”.

That, in turn, means that all writes that happened in thread 1 are visible in thread 2, regardless to volatility of the variables involved. In fact, we can remove all volatiles from transients implementation and probably make them faster, because, by asking “no more than one thread at a time” we enforce users to establish happens-before between sections, and that would give us all the safe publication guarantees we need.

Is my understanding correct? Am I missing something?

Comment by Nikita Prokopov [ 04/Jan/17 11:55 AM ]

Also, long-living transients (e.g. in a transducers associated with a queue, for example) will hold a reference to a thread that created them. Is that a bad thing? Should we switch to boolean flag instead?





[CLJ-2089] Sorted colls with default comparator don't check that first element is Comparable Created: 19/Dec/16  Updated: 13/Jan/17

Status: Open
Project: Clojure
Component/s: None
Affects Version/s: Release 1.9
Fix Version/s: Release 1.9

Type: Defect Priority: Major
Reporter: Alex Miller Assignee: Unassigned
Resolution: Unresolved Votes: 0
Labels: collections

Attachments: Text File clj-2089.patch    
Patch: Code and Test
Approval: Vetted

 Description   

Sorted maps and sets use the default comparator. The default comparator requires elements to be Comparable. PersistentTreeMap will not actually invoke the comparator until the second element is added to the map/set, so it is possible to create an invalid single element sorted map/set that will blow up only on later use.

The following examples create invalid "timebomb" collections that will throw ClassCastException if another element is added or if they're used in other ways (because sets are not comparable):

(sorted-set #{1})
(conj (sorted-set) #{1})
(sorted-map #{} 1)
(assoc (sorted-map) #{} 1)

Example:

(def s (sorted-set #{1}))  ;; this doesn't fail
(conj s #{2})              ;; first conj triggers error
ClassCastException clojure.lang.PersistentHashSet cannot be cast to java.lang.Comparable  clojure.lang.Util.compare (Util.java:153)

Cause: In PersistentTreeMap.add(), in the case where the existing tree is null, the comparator is never invoked and so the default comparator will never throw the ClassCastException seen on later compares. Note that none of this applies for a custom comparator (sorted-set-by and sorted-map-by) - those comparators can do whatever.

Proposed: In add(), if the map is empty AND the comparator is the default comparator, then check whether the added key is Comparable and throw CCE if not. Note that PersistentTreeMap is also the impl used by PersistentTreeSet so this covers both. The error message is customized to give you a better hint about the problem (and written to be applicable for both maps and sets). In the patch some existing (bad) tests had to be adjusted.

user=> (def s (sorted-set #{1}))
ClassCastException Default comparator requires nil, Number, or Comparable: #{1}  clojure.lang.PersistentTreeMap.add (PersistentTreeMap.java:335)

One downside of the current patch is that does not catch the equivalent problem if using a custom comparator and a bad first element. You could maybe do this by calling comp.compare(k,k) (although many comparators, like the default comparator, have an early identity check that would not fail on this). For now, decided that if you supply a custom comparator, it's up to you to not use it with elements that don't work with that comparator.

Patch: clj-2089.patch






[CLJ-2088] 'into' can make maps from collection of lists, but vectors are ok. Created: 19/Dec/16  Updated: 20/Dec/16  Resolved: 19/Dec/16

Status: Closed
Project: Clojure
Component/s: None
Affects Version/s: Release 1.8
Fix Version/s: None

Type: Defect Priority: Major
Reporter: Eugene Aksenov Assignee: Unassigned
Resolution: Declined Votes: 0
Labels: bug
Environment:

OS: Ubuntu 16.04, x86_64, Linux 4.2.0-42-generic



 Description   

'into' can make maps from collection of lists, but vectors are ok.

Good behavior:
(into {} [[:a "a"] [:b "b"]])
;;=> {:a "a", :b "b"}

(into {} '([1 "a"] [2 "b"]))
;;=> {1 "a", 2 "b"}

Bad examples:
(into {} ['(:a "a") '(:b "b")])
;;=> ClassCastException clojure.lang.Keyword cannot be cast to java.util.Map$Entry clojure.lang.ATransientMap.conj (ATransientMap.java:44)

(into {} [(list [:a "a"]) [:b "b"]])
;;=> ClassCastException clojure.lang.PersistentVector cannot be cast to java.util.Map$Entry clojure.lang.ATransientMap.conj (ATransientMap.java:44)

(into {} ['(1 "a") '(2 "b")])
;;=> ClassCastException java.lang.Long cannot be cast to java.util.Map$Entry clojure.lang.ATransientMap.conj (ATransientMap.java:44)

(into {} '('(1 "a") '(2 "b")))
;;=> ClassCastException clojure.lang.Symbol cannot be cast to java.util.Map$Entry clojure.lang.ATransientMap.conj (ATransientMap.java:44)



 Comments   
Comment by Alex Miller [ 19/Dec/16 11:53 AM ]

into is built on conj (takes a sequential collection of elements to invoke with conj). conj for maps takes a map entry - a subclass of java.util.Map$MapEntry or a 2-element vector. Both of these choices can access keys and vals directly without "traversing" the entry (going through the key to get to the value).

We don't want to support that for performance reasons, so lists or seqs or not valid as map entries (and conj on a map does not support them).

into takes a sequential collection of these entries though, so vector, or list, or seq are all fine as the src collection for into.

So, all of this is working as intended and I am declining the ticket.

Comment by Eugene Aksenov [ 20/Dec/16 2:09 AM ]

Ok, live and learn
I've just added this code case and brief explanation to clojuredocs.org. Hope no one will be confused anymore.





[CLJ-2087] map does not work with core.async/<! Created: 17/Dec/16  Updated: 17/Dec/16  Resolved: 17/Dec/16

Status: Closed
Project: Clojure
Component/s: None
Affects Version/s: None
Fix Version/s: None

Type: Defect Priority: Major
Reporter: Roy Varghese Assignee: Unassigned
Resolution: Declined Votes: 0
Labels: None


 Description   

The construct
(map <! [chan1 chan2 chan3])

does not work for reasons outlined here https://github.com/clojure/core.async/wiki/Go-Block-Best-Practices

This leads to very obscure bugs.

If this cannot be fixed in map, please consider throwing an exception when it cannot handle the passed function.



 Comments   
Comment by Alex Miller [ 17/Dec/16 5:08 PM ]

It's unlikely that we're going to "fix" this as it part of the design of Clojure.

When I try your example I see:

AssertionError Assert failed: <! used not in (go ...) block




[CLJ-2086] A macro in a nested syntax-quote causes an exception Created: 15/Dec/16  Updated: 03/Jan/17  Resolved: 15/Dec/16

Status: Closed
Project: Clojure
Component/s: None
Affects Version/s: None
Fix Version/s: None

Type: Defect Priority: Major
Reporter: N/A Assignee: Unassigned
Resolution: Declined Votes: 0
Labels: None


 Description   

(defmacro foo
[x y]
``(~~x ~~y))

(foo and true)

CompilerException java.lang.RuntimeException: Can't take value of a macro: #'clojure.core/and



 Comments   
Comment by Alex Miller [ 15/Dec/16 10:15 PM ]
user=> (macroexpand '(foo and true))
(clojure.core/seq (clojure.core/concat (clojure.core/list and) (clojure.core/list true)))

What do you expect this to do?

Comment by Alex Miller [ 15/Dec/16 10:36 PM ]

Declining till more info about a) what problem you are trying to solve and b) what the expected behavior is.

Comment by N/A [ 22/Dec/16 8:30 PM ]

a) I'm trying to define a macro that defines a macro.
(defmacro make-defmacro
[macro-name macro]
`(defmacro ~macro-name
x# & more#
(do (~macro x#)
`(~~macro-name ~@more#))))

(make-defmacro bar and)
CompilerException java.lang.RuntimeException: Can't take value of a macro

b) (defmacro foo
[x y]
``(~~x ~~y))

(macroexpand '(foo and true))
=> (and true)

Comment by Viktor Magyari [ 03/Jan/17 10:52 AM ]

Try

(defmacro foo [x y]
  ``(~'~x ~~y))




[CLJ-2085] Add additional info to explain-data to help explain printers Created: 15/Dec/16  Updated: 03/Apr/17

Status: Open
Project: Clojure
Component/s: None
Affects Version/s: Release 1.9
Fix Version/s: Release 1.9

Type: Enhancement Priority: Major
Reporter: Alex Miller Assignee: Unassigned
Resolution: Unresolved Votes: 3
Labels: spec

Attachments: Text File explain-data.patch    
Patch: Code
Approval: Screened

 Description   

Right now, the explain-data provided to the explain printer function only has the list of problem data. There are many interesting things a printer could do with access to the root spec and value but those are not currently available.

Proposed: Provide these values in the explain-data map (as ::spec and ::value).

Patch: explain-data.patch



 Comments   
Comment by David Chelimsky [ 03/Apr/17 9:26 AM ]

We're using explain-data to generate non-dev-readable messages in a browser. Access to the root structure would be helpful in cases where understanding the output requires specific knowledge of Clojure's data structures e.g. when a spec for a collection of maps is used to validate a map, in which case the :val key is bound to a MapEntry.





[CLJ-2084] Support MapEquivalence for defrecords Created: 13/Dec/16  Updated: 13/Dec/16  Resolved: 13/Dec/16

Status: Closed
Project: Clojure
Component/s: None
Affects Version/s: None
Fix Version/s: None

Type: Enhancement Priority: Major
Reporter: Herwig Hochleitner Assignee: Unassigned
Resolution: Declined Votes: 0
Labels: defrecord


 Description   

defrecord equality implies matching types. This has been discussed and decided upon such that people should only bother with defrecords when they want an equality partition. There are, however, use cases for defrecords as small structmaps to save memory and those require map-based equivalence. Clojure has a marker interface for map equivalence: clojure.lang.MapEquivalence, which gets used in = to decide whether something qualifies as a persistent map. This can be reused to change the equiv implementation of a defrecord.

Current Behavior

When MapEquivalence is implemented in a defrecord, = becomes non-reflexive

(= {} (->Rec)), (not= (->Rec) {})

Proposed Behavior

Add clojure.lang.MapEquivalence to implemented interfaces in a defrecord, to treat it as a PersistenMap in equality, disregarding the type tag.
APersistentMap.equiv already does this, defrecord implementation can call APersistentMap/mapEquals



 Comments   
Comment by Alex Miller [ 13/Dec/16 10:27 AM ]

Wouldn't this be a breaking change for anyone relying on records to compare not equals based on type?

Comment by Herwig Hochleitner [ 13/Dec/16 4:35 PM ]

The key phrase is "Add clojure.lang.MapEquivalence to implemented interfaces". Any existing defrecords would not be affected. Au contraire, having a defrecord implement MapEquivalence right now results in a non-reflective-equals clojure-gigo scenario, so it's completely uncharted territory and ripe for definition. Sorry for not being clearer in the description, would you consider reopening?

Comment by Alex Miller [ 13/Dec/16 5:15 PM ]

It's not uncharted territory. Records intentionally compare different for equality based on type.

If you don't want that, don't use records. Or pour your record into a map before comparing via into or select-keys.

Comment by Herwig Hochleitner [ 13/Dec/16 5:51 PM ]

Yes, and I'm not proposing to change a single bit of that, agreed?
I am proposing the possibility to have specific defrecords implement map-like equality by using an option that a) naturally fits with what's already there b) compiles but yields GIGO right now:

(defrecord Rec []
  clojure.lang.MapEquivalence)

How is this already charted? If it is, then this ticket needs to be refiled as a Defect, because then the current behavior makes no sense.

So, if you're not declining this on technical grounds, what is it then? If it is not wanting to add use cases to defrecords for philosophical reasons, you're using the term "breaking change" very loosely here. There can't be any existing expectations on defrecords implementing MapEquivalence right now, because such defrecords are invalid programs right now.
Are you arguing some sort of "general expectation" on defrecords? One that would stand without looking at specific definitions? What use would that be to anybody. Who works with a defrecord without having a general idea of what it definition is?

Comment by Herwig Hochleitner [ 13/Dec/16 6:19 PM ]

Just to be clear, I did my due diligence on this and I'm aware that the current equality situation for records is completely intentional, hence I'm proposing growing it. The argument might still be that we don't want to grow it in that particular direction, but it's definitely not breakage.

Comment by Alex Miller [ 13/Dec/16 6:34 PM ]

I thought you were asking to add MapEquivalence to all records, which would be a breaking change.

The docstring for defrecord says:

{pre}
In addition, defrecord will define type-and-value-based =,
and will defined Java .hashCode and .equals consistent with the
contract for java.util.Map.{pre}

The important thing there is "type-and-value-based", and that's part of the design of records. Asking for MapEquivalence on a map means value-based only. These two things are in conflict and thus I would agree that asking for MapEquivalence on a record makes no sense.

deftype exists so that you can make your own types with whatever equality/hash/behavior you want - if you want something like this, then build it on top of deftype.

Comment by Herwig Hochleitner [ 13/Dec/16 6:41 PM ]

yeah, changing defrecord to implement MapEquivalence by default would make no sense.

However, declaring a (specific) defrecord to be in the MapEquivalence partition makes perfect sense, doesn't it?
That way, we can type our records an structmap them too.

Comment by Herwig Hochleitner [ 13/Dec/16 6:49 PM ]

Anyway, I'll build this on deftype for data.xml in any case, because we want this to work on current and older clojure versions. I just thought it would fit neatly with clojure itself (since there is already a marker interface for that purpose, only its use currently leads to breakage) and wanted to make you aware of the idea. I'll recycle it through the mailing list, as soon as the deftype based thing is in data.xml





[CLJ-2083] spec for printable/readable/edn data Created: 12/Dec/16  Updated: 12/Dec/16

Status: Open
Project: Clojure
Component/s: None
Affects Version/s: Release 1.9
Fix Version/s: None

Type: Enhancement Priority: Major
Reporter: Brandon Bloom Assignee: Unassigned
Resolution: Unresolved Votes: 0
Labels: None


 Description   

When spec'ing some things, I've used `any?` in a few cases where it is overly permissive. In particular, sometimes I need to specify a value must be printable/readable, such as when a value may wind up in an edn file. Similarly, I've needed to spec something must have non-generative value-identity, ie. ban closures, etc. Printable/readable or simply `edn?` would be a much better approximation than `any?`.



 Comments   
Comment by Brandon Bloom [ 12/Dec/16 1:08 PM ]

I realize that an edn? predicate would have O(N) runtime, vs an edn spec that could take advantage of every/every-kv etc for sampling conformance.

Comment by Herwig Hochleitner [ 12/Dec/16 11:12 PM ]

Related: CLJ-1527





[CLJ-2082] Improve documentation of clojure.walk/walk Created: 11/Dec/16  Updated: 11/Dec/16

Status: Open
Project: Clojure
Component/s: None
Affects Version/s: Release 1.8
Fix Version/s: None

Type: Enhancement Priority: Minor
Reporter: David Cook Assignee: Unassigned
Resolution: Unresolved Votes: 0
Labels: docs, docstring, documentation


 Description   

The documentation for the clojure.walk module isn't clear on which methods recurse through data structures, and which only operate on the outermost layer. The documentation for clojure.walk/walk and clojure.walk/postwalk both use forms of the word "traverse," and there's nothing calling out that clojure.walk/walk, unlike the rest of the functions in the namespace, doesn't recurse through the provided form.






[CLJ-2081] for macro spec should know :let can't go in the first position Created: 09/Dec/16  Updated: 09/Dec/16

Status: Open
Project: Clojure
Component/s: None
Affects Version/s: Release 1.9
Fix Version/s: None

Type: Enhancement Priority: Minor
Reporter: lvh Assignee: Unassigned
Resolution: Unresolved Votes: 0
Labels: spec

Approval: Triaged

 Description   

The for macro does not support :let in the first position. This was reported in CLJ-207 and a patch was produced but rejected. Since we have spec for macros now, it might be an opportunity to provide a better error message. (I think first-place let should be fine, but that's neither here nor there.)






[CLJ-2080] clojure.spec/every-kv does not work correctly on vectors Created: 08/Dec/16  Updated: 31/Mar/17

Status: Open
Project: Clojure
Component/s: None
Affects Version/s: Release 1.9
Fix Version/s: Release 1.9

Type: Defect Priority: Major
Reporter: Brandon Bloom Assignee: Unassigned
Resolution: Unresolved Votes: 0
Labels: spec
Environment:

alpha 14


Attachments: Text File clj-2080-2.patch     Text File clj-2080-3.patch     Text File clj-2080-4.patch     Text File clj-2080.patch    
Patch: Code
Approval: Vetted

 Description   

The every-kv doc states "takes separate key and val preds and works on associative collections". Vectors return true for associative? but do not currently work:

user=> (s/conform (s/every-kv any? any?) [])
[]
user=> (s/conform (s/every-kv any? any?) [1 2 3])
:clojure.spec/invalid
user=> (s/conform (s/every-kv integer? string?) [])
[]
user=> (s/conform (s/every-kv integer? string?) ["x"])
:clojure.spec/invalid

Another similar problem:

(s/explain-data (s/every-kv int? int?) [{:a :b}])
UnsupportedOperationException nth not supported on this type: PersistentArrayMap  clojure.lang.RT.nthFrom (RT.java:903)

Cause: The combination of every-kv and every-impl assume that the collection passed to every-kv can provide a seq of map entries (which is the intention). In the explain case, the ::kfn created by every-kv is used to create a better path segment using the key rather than the element index. The kfn assumes that an element of the collection can call `(nth entry 0)` on the element. In the explain failure above, the map {:a :b} will throw when invoked with nth.

Proposed: Do the following to be clearer about the requirement that the coll elements are map entries:

  • Modify the docstring to say "seqs to map entries" rather than "is a map"
  • Modify the kfn to add a check that the element is an entry and if so, use it's key. If not, use the index (of the element itself). In this case, when passed an entry it will report with the actual key but when passed something that's not an entry, it will report the collection based index of the non-entry.

Patch: clj-2080-4.patch



 Comments   
Comment by Alex Miller [ 09/Dec/16 8:35 AM ]

At the moment, I'm inclined to say the doc in every-kv should be tightened to say "map" instead of "associative collection" but will check with Rich.

Comment by Alex Miller [ 31/Mar/17 9:07 AM ]

Updated to apply to master





[CLJ-2079] Generator overrides for spec aliases are not respected Created: 08/Dec/16  Updated: 08/Dec/16

Status: Open
Project: Clojure
Component/s: None
Affects Version/s: Release 1.9
Fix Version/s: Release 1.9

Type: Defect Priority: Major
Reporter: Nate Smith Assignee: Unassigned
Resolution: Unresolved Votes: 4
Labels: generator, spec

Approval: Vetted

 Description   

Generator overrides for spec aliases are not respected.

Unable to find source-code formatter for language: clojure. Available languages are: javascript, sql, xhtml, actionscript, none, html, xml, java
(require '[clojure.spec :as s])
(require '[clojure.spec.gen :as gen])
(s/def ::original number?)
(s/def ::alias ::original)

(every? double? (gen/sample (s/gen ::alias {::alias gen/double})))
;; => false

Providing a generator override for the original spec works as expected:

Unable to find source-code formatter for language: clojure. Available languages are: javascript, sql, xhtml, actionscript, none, html, xml, java
(every? double? (gen/sample (s/gen ::alias {::original gen/double})))
;; => true


 Comments   
Comment by Alex Miller [ 08/Dec/16 5:02 PM ]

Probably a missing delay in the alias case - there's another ticket that has the same cause.

Comment by Nate Smith [ 08/Dec/16 6:43 PM ]

Looks like it might be because gensub looks for matching overrides by calling spec-name, which returns the wrong value for spec aliases.

(require '[clojure.spec :as s])
(s/def ::original number?)
(s/def ::alias ::original)
(@#'clojure.spec/spec-name (s/get-spec ::alias))
;; => :user/original




[CLJ-2078] Add a maven profile to connect with an nrepl client Created: 07/Dec/16  Updated: 07/Dec/16  Resolved: 07/Dec/16

Status: Closed
Project: Clojure
Component/s: None
Affects Version/s: None
Fix Version/s: None

Type: Enhancement Priority: Trivial
Reporter: Herwig Hochleitner Assignee: Unassigned
Resolution: Declined Votes: 0
Labels: development

Attachments: Text File clj-2078-0.patch    
Patch: Code

 Description   

Opening an nrepl port into a project is a popular development strategy with clojure projects. It should also work with the clojure project.

Maven profiles make it possible to add development tools without affecting the rest of the build.

clojure-maven-plugin has this tool built right in as the clojure:nrepl goal and cider provides middlewares for many advanced ide functionalities.

Attached patch provides and documents a -P cider profile that lets you jack into a clojure source checkout.



 Comments   
Comment by Herwig Hochleitner [ 07/Dec/16 7:38 PM ]

Assuming that non-contrib covered code is OK for optional dev tools.

Comment by Alex Miller [ 07/Dec/16 8:44 PM ]

Thanks, I don't think we're going to do this.





[CLJ-2077] Clojure can't be loaded from the boot classpath under java 9 Created: 06/Dec/16  Updated: 25/Apr/17

Status: Open
Project: Clojure
Component/s: None
Affects Version/s: Release 1.9
Fix Version/s: Release 1.9

Type: Defect Priority: Major
Reporter: Toby Crawley Assignee: Unassigned
Resolution: Unresolved Votes: 5
Labels: java9

Approval: Vetted

 Description   

As part of the changes for the jigsaw module system in Java 9, the
java packages available to the boot classloader are now a subset of
the the full java distribution. This means that classes loaded via the
boot classloader cannot access any classes outside of that subset.

The list of packages not visible to the boot classloader are (see
module/package mapping in http://cr.openjdk.java.net/~mr/jigsaw/ea/module-summary.html - java.base module is loaded in boot classloader):

java.activation
java.annotations.common
java.compact1
java.compact2
java.compact3
java.compiler
java.corba
java.scripting
java.se
java.se.ee
java.sql
java.sql.rowset
java.transaction
java.xml.bind
java.xml.ws
jdk.accessibility
jdk.charsets
jdk.crypto.ec
jdk.crypto.pkcs11
jdk.dynalink
jdk.jsobject
jdk.localedata
jdk.naming.dns
jdk.scripting.nashorn
jdk.xml.dom
jdk.zipfs
jdk.attach
jdk.compiler
jdk.hotspot.agent
jdk.internal.le
jdk.internal.opt
jdk.jartool
jdk.javadoc
jdk.jconsole
jdk.jdeps
jdk.jdi
jdk.jlink
jdk.jshell
jdk.jstatd
jdk.jvmstat

Clojure itself only uses one package on that list: java.sql. It is
used in clojure.instant to provide print-method and
print-dup implementations for java.sql.Timestamp, and in
clojure.core/resultset-seq.

This can be seen with (using Clojure 1.4.0 or higher, and a early-access build
of Java 9, most recently tested with 9-ea+147):

java -Xbootclasspath/a:clojure.jar clojure.main -e "(require 'clojure.instant)"

This affects any clojure-based tool that puts itself on the boot
classpath in order to gain a startup time boost (both lein
and boot are affected currently).



 Comments   
Comment by Toby Crawley [ 06/Dec/16 12:34 PM ]

More details on the underlying change that is triggering this are available at http://openjdk.java.net/jeps/261 (search for java.sql to find the relevant section).

Comment by Alex Miller [ 06/Dec/16 8:41 PM ]

Does this need to be a ticket here? Or is this really an issue for build tools?

Comment by Toby Crawley [ 08/Dec/16 4:30 PM ]

That depends on if we want using Clojure from the boot classpath to be an acceptable use case. If not, then I agree, it is just an issue for tooling.

Comment by Toby Crawley [ 09/Dec/16 2:21 PM ]

I realized today that this issue doesn't actually affect boot, since it doesn't use the bootclasspath. So lein is the only tooling I know of that is affected by this.

Comment by Ivan Pierre [ 12/Dec/16 4:59 AM ]

https://docs.oracle.com/javase/8/docs/api/java/sql/Timestamp.html
Would it be possible to use java.util.Date instead. Alas it's not possible to downcast :
Due to the differences between the Timestamp class and the java.util.Date class mentioned above, it is recommended that code not views Timestamp values generically as an instance of java.util.Date. The inheritance relationship between Timestamp and java.util.Date really denotes implementation inheritance, and not type inheritance.

Comment by Ivan Pierre [ 12/Dec/16 8:22 AM ]

The problem with Date is that it doesn't deal with nanoseconds. But Timestamp is created by a long giving the TimeDate value in milliseconds.
The use of setNano and getNano are only to store the SQL TIMESTAMP. Wouldn't it be better to deal with this value in another way?

The other way is to take just what we need from TimeStamp, and it's just a little thing, I'll try to compile with that to see if some other thing comes after...

Test code : https://gist.github.com/ivanpierre/b0ea937dac97d910a7c3c1e5774028e0

Comment by Ivan Pierre [ 12/Dec/16 1:13 PM ]

Ok, I pass to the GNU version of Timestamp. The code is neater. I mixed some of Sun's for more consistency. I dropped the string management of dates as Clojure will do it in clojure.instant.

It still works. I had a doubt...

If I type (clojure.lang.TimeStamp. 3678141) the response will be :
==> #inst "1970-01-01T01:01:18.141000000-00:00" with a nano of
141000000

But is if set nano to 1 : (.setNanos (clojure.lang.TimeStamp. 3678141) 1) the response is : #inst "1970-01-01T01:01:18.000000001-00:00"

This is correct, but it's a little disturbing to see my nice .141 disappear...

I put a fork on my GitHub. Last commit : https://github.com/ivanpierre/clojure/commit/749a0184ee7409290dad9ff353605fcaabd64f69

So, good, now pass to Leinigen...

Comment by Alex Miller [ 13/Dec/16 10:32 AM ]

I think the first question here is: do we expect that Clojure should be loadable from the bootclasspath?

While I realize this is a hack people use, my initial answer would be no (that was never a design constraint afaik). But will need to defer to Rich on that.

Comment by Ivan Pierre [ 13/Dec/16 10:41 AM ]

Well Leiningen do it, even with the lein repl. I test to see if TimeStamp is the only thing.
Alas Leinigen is not 1.9 compatible, so I have to go down to version 1.8 to make the tests. (problem of conflict between clojure.spec and hara library)
A funny thing would be to pass the whole Clojure test in bootstrap so we would know...

Comment by Ivan Pierre [ 13/Dec/16 1:06 PM ]

The java Timestamp could be directly integrated into clojure.instant as it's a new datatype. So no need to worry about copyright stuff, and integer it in a complete manner to accept SQL TIMESTAMP, and some correct protocols.
The worst is that looking across the DBs documentation doesn't help a lot and some are quite contradictory.

Comment by Phil Hagelberg [ 03/Apr/17 1:05 PM ]

> I think the first question here is: do we expect that Clojure should be loadable from the bootclasspath?

I don't know if it's the right thing for Clojure to be able to be loaded from the bootclasspath, but as an additional point of data: Leiningen takes 2.5 seconds to load on my machine from the bootclasspath on Java 8. Without the bootclasspath, it takes 3.4 seconds on Java 8 and 4.5 seconds on Java 9. (This is just for one Clojure instance, not launching a project subprocess.) So in this case it nearly doubles the time, which is consistently in the top 5 complaints about working with Clojure.

I would not be surprised if other Clojure-based tools (IDEs/editors perhaps) would want to use the bootclasspath but I don't have any data on that. I know there are people using the bootclasspath for production servers, but they probably wouldn't be as upset about adding a few seconds as people using it on their own machine.

I would be happy to write the few lines of code needed to defer all references to the inaccessible classes until runtime if it's decided that's the way to go.

Comment by Ghadi Shayban [ 03/Apr/17 3:13 PM ]

Phil - would you mind sharing your environment and testing code? I see a slowdown without bootclasspath but not nearly as dramatic as what you see. The Java 9 builds you are using probably have extra debugging enabled.

Comment by Phil Hagelberg [ 25/Apr/17 10:08 PM ]

Ghadi: I'm running on a Zulu build of Java 9 since I wasn't able to find binaries for Debian of the official OpenJDK ones. It's possible there's extra debugging overhead, but it's more likely it's just because I'm running on old hardware.

$ java -version
openjdk version "9-ea"
OpenJDK Runtime Environment (Zulu 9.0.0.10-linux64) (build 9-ea+155-pre-release-eval)
OpenJDK 64-Bit Server VM (Zulu 9.0.0.10-linux64) (build 9-ea+155, mixed mode, pre-release-eval)

Comment by Alex Miller [ 25/Apr/17 10:52 PM ]

I'm moving this into Clojure 1.9. Not implying any decisions, just so we look at it before any 1.9 release decisions.





[CLJ-2076] s/coll-of and s/map-of do not unform their elements Created: 05/Dec/16  Updated: 07/Apr/17

Status: Open
Project: Clojure
Component/s: None
Affects Version/s: Release 1.9
Fix Version/s: Release 1.9

Type: Defect Priority: Major
Reporter: Alex Miller Assignee: Unassigned
Resolution: Unresolved Votes: 1
Labels: spec

Attachments: Text File clj-2076-2.patch     Text File clj-2076-3.patch     Text File clj-2076.patch    
Patch: Code and Test
Approval: Screened

 Description   

s/coll-of and s/map-of unform with identity but should unform their elements:

(s/def ::o (s/coll-of (s/or :i integer? :s string?)))
(->> [1 2 "blah"] (s/conform ::o) (s/unform ::o))
=> [[:i 1] [:i 2] [:s "blah"]]

Expected: [1 2 "blah"]

Cause: every-impl unform* just returns x

Approach: Use the init/add/complete fns to generate an unformed value (when needed)

Patch: clj-2076-3.patch



 Comments   
Comment by Alex Miller [ 07/Dec/16 5:50 PM ]

This needs tests and a bunch of verification, but first pass at fixing.

Comment by Alex Miller [ 09/Dec/16 8:03 AM ]

Added tests, ready to screen

Comment by Alex Miller [ 07/Apr/17 3:49 PM ]

Added patch -3, only updated to apply cleanly to master.





[CLJ-2075] Add three-arities to < <= > >= = == not= Created: 03/Dec/16  Updated: 23/Dec/16

Status: Open
Project: Clojure
Component/s: None
Affects Version/s: None
Fix Version/s: None

Type: Enhancement Priority: Major
Reporter: Nikita Prokopov Assignee: Unassigned
Resolution: Unresolved Votes: 5
Labels: None

Attachments: Text File clj-2075-add-three-arities-to-comparisons.patch     Text File clj-2075-over-clj-1912.patch    

 Description   

In my practice, using three-arities of less/greater operations is pretty common for e.g. checking a number is in range:

(< 0 temp 100)

The problem is, it is almost three times as slow compared to (and (< 0 temp) (< temp 100)).

This happens because three-arities are handled by the generic vararg arity branch:

(defn <
  "Returns non-nil if nums are in monotonically increasing order,
  otherwise false."
  {:inline (fn [x y] `(. clojure.lang.Numbers (lt ~x ~y)))
   :inline-arities #{2}
   :added "1.0"}
  ([x] true)
  ([x y] (. clojure.lang.Numbers (lt x y)))
  ([x y & more]
    (if (< x y)
     (if (next more)
       (recur y (first more) (next more))
       (< y (first more)))
     false)

This patch adds special handling for three-arities to these fns: < <= > >= = == not=

(defn <
  "Returns non-nil if nums are in monotonically increasing order,
  otherwise false."
  {:inline (fn [x y] `(. clojure.lang.Numbers (lt ~x ~y)))
   :inline-arities #{2}
   :added "1.0"}
  ([x] true)
  ([x y] (. clojure.lang.Numbers (lt x y)))
  ([x y z] (and (. clojure.lang.Numbers (lt x y))
                (. clojure.lang.Numbers (lt y z))))
  ([x y z & more]
   (if (< x y)
     (let [nmore (next more)]
       (if nmore
         (recur y z (first more) nmore)
         (< y z (first more))))
     false)))

The performance gains are quite significant:

(= 5 5 5) 	 24.508635 ns => 4.802783 ns (-80%)
(not= 1 2 3) 	 122.085793 ns => 21.828776 ns (-82%)
(< 1 2 3) 	 30.842993 ns => 6.714757 ns (-78%)
(<= 1 2 2) 	 30.712399 ns => 6.011326 ns (-80%)
(> 3 2 1) 	 22.577751 ns => 6.893885 ns (-69%)
(>= 3 2 2) 	 21.593219 ns => 6.233540 ns (-71%)
(== 5 5 5) 	 19.700540 ns => 6.066265 ns (-69%)

Higher arities also become faster, mainly because there's one less iteration now:

(= 5 5 5 5) 	 50.264580 ns => 31.361655 ns (-37%)
(< 1 2 3 4) 	 68.059758 ns => 43.684409 ns (-35%)
(<= 1 2 2 4) 	 65.653826 ns => 45.194730 ns (-31%)
(> 3 2 1 0) 	 119.239733 ns => 44.305519 ns (-62%)
(>= 3 2 2 0) 	 65.738453 ns => 44.037442 ns (-33%)
(== 5 5 5 5) 	 50.773521 ns => 33.725097 ns (-33%)

This patch also changes vararg artity of not= to use next/recur instead of apply:

(defn not=
  "Same as (not (= obj1 obj2))"
  {:tag Boolean
   :added "1.0"
   :static true}
  ([x] false)
  ([x y] (not (= x y)))
  ([x y z] (not (= x y z)))
  ([x y z & more]
   (if (= x y)
     (let [nmore (next more)]
       (if nmore
         (recur y z (first more) nmore)
         (not= y z (first more))))
     true)))

Results are good:

(not= 1 2 3 4) 	 130.517439 ns => 29.675640 ns (-77%)

I'm also doing what Jozef Wagner did in CLJ-1912 (calculating (next more) just once), although perf gains from that alone are not that big.

My point here is that optimizing three-arities makes sence because they appear in the real code quite often. Higher arities (4 and more) are much less widespread.



 Comments   
Comment by Nikita Prokopov [ 03/Dec/16 2:32 AM ]

Benchmark code here https://gist.github.com/tonsky/442eda3ba6aa4a71fd67883bb3f61d99

Comment by Alex Miller [ 03/Dec/16 8:24 AM ]

It might make more sense to combine this with CLJ-1912, otherwise these patches will fight.

Comment by Nikita Prokopov [ 03/Dec/16 1:02 PM ]

Use this patch if CLJ-1912 would be applied first

Comment by Nikita Prokopov [ 23/Dec/16 7:50 AM ]

I found a problem with previous patches that during defining = (equality), and is not yet defined. Replaced with if





[CLJ-2074] ::keys spec conflicts with destructuring spec Created: 02/Dec/16  Updated: 11/Jan/17

Status: Open
Project: Clojure
Component/s: None
Affects Version/s: None
Fix Version/s: None

Type: Defect Priority: Major
Reporter: Nicola Mometto Assignee: Alex Miller
Resolution: Unresolved Votes: 1
Labels: destructuring, spec

Attachments: File close-destructuring-keys-specs.diff    
Patch: Code

 Description   

As a consequence of the destructuring specs being implemented in terms of `s/keys`, defining a spec for `::keys` or `::strs` is problematic at the moment, because it will conflict with trying to use `::keys` for destructuring:

user=> (require '[clojure.spec :as s])
nil
user=> (s/def ::keys nil?)
:user/keys
user=> (let [{::keys [a]} {::a 1}] a)
ExceptionInfo Call to clojure.core/let did not conform to spec:
In: [0 0] val: #:user{:keys [a]} fails spec: :clojure.core.specs/local-name at: [:args :bindings :binding :sym] predicate: simple-symbol?
In: [0 0 0] val: ([:user/keys [a]]) fails spec: :clojure.core.specs/seq-binding-form at: [:args :bindings :binding :seq] predicate: (cat :elems (* :clojure.core.specs/binding-form) :rest (? (cat :amp #{(quote &)} :form :clojure.core.specs/binding-form)) :as (? (cat :as #{:as} :sym :clojure.core.specs/local-name))),  Extra input
In: [0 0 :user/keys] val: [a] fails spec: :user/keys at: [:args :bindings :binding :map :user/keys] predicate: nil?
:clojure.spec/args  ([#:user{:keys [a]} #:user{:a 1}] a)
  clojure.core/ex-info (core.clj:4725)

This feels like an implementation detail leak.



 Comments   
Comment by Brandon Bloom [ 10/Jan/17 5:36 PM ]

I also just ran in to this problem. Just wanted to say that I'd like to see a fix, but I'm not quite sure about the proposed solution. Or, at least, the name "closed?" seems to imply a non-extensible map, when in reality the flag more or less means "not a map that participates in the global keys system", for which I do not have a better name suggestion.

Comment by Alex Miller [ 11/Jan/17 8:35 AM ]

The proposed patch is a non-starter. I have some ideas on how to address this, but just haven't gotten around to working on it yet.

Comment by Alex Miller [ 11/Jan/17 8:37 AM ]

Removed proposal and patch from the ticket as we will not be going this direction. Captured here for reference:

"The attached patch implements a proposed solution to this issue, by adding a `:closed?` option to `s/keys` and using it for the destructuring spec. If `s/keys` is used with `:closed?` set to true, `conform` will only validate declared specs as opposed to the default behaviour of `s/keys` of validating all namespaced keywords with existing specs.

After this patch, the above example runs fine and usages of `s/keys` without `:closed?` set to true will validate against `::keys` as per current behaviour.

Patch: close-destructuring-keys-specs.diff"





[CLJ-2073] AOT compilation can result in spurious ClassCastException during compile Created: 02/Dec/16  Updated: 02/Dec/16

Status: Open
Project: Clojure
Component/s: None
Affects Version/s: Release 1.7, Release 1.8
Fix Version/s: None

Type: Defect Priority: Major
Reporter: Paul Mooser Assignee: Unassigned
Resolution: Unresolved Votes: 0
Labels: aot, compiler
Environment:

java version "1.8.0_112"
Java(TM) SE Runtime Environment (build 1.8.0_112-b16)
Java HotSpot(TM) 64-Bit Server VM (build 25.112-b16, mixed mode)


Attachments: File consumer.clj     File implementer.clj     File protocol.clj    

 Description   

If you try to compile the attached files as follows (assuming they are in "src"):

java -Dclojure.compile.path=out -cp "./clojure-1.8.0.jar:out:src" clojure.lang.Compile implementer protocol consumer

an exception will be thrown:

Exception in thread "main" java.lang.ClassCastException: implementer.Obj cannot be cast to protocol.Dependent, compiling:(consumer.clj:5:1)
	at clojure.lang.Compiler$InvokeExpr.eval(Compiler.java:3657)
	at clojure.lang.Compiler.compile1(Compiler.java:7474)
	at clojure.lang.Compiler.compile(Compiler.java:7541)
	at clojure.lang.RT.compile(RT.java:406)
	at clojure.lang.RT.load(RT.java:451)
	at clojure.lang.RT.load(RT.java:419)
	at clojure.core$load$fn__5677.invoke(core.clj:5893)
	at clojure.core$load.invokeStatic(core.clj:5892)
	at clojure.core$load.doInvoke(core.clj:5876)
	at clojure.lang.RestFn.invoke(RestFn.java:408)
	at clojure.core$load_one.invokeStatic(core.clj:5697)
	at clojure.core$compile$fn__5682.invoke(core.clj:5903)
	at clojure.core$compile.invokeStatic(core.clj:5903)
	at clojure.core$compile.invoke(core.clj:5895)
	at clojure.lang.Var.invoke(Var.java:379)
	at clojure.lang.Compile.main(Compile.java:67)
Caused by: java.lang.ClassCastException: implementer.Obj cannot be cast to protocol.Dependent
	at protocol$fn__12$G__8__14.invoke(protocol.clj:3)
	at protocol$fn__12$G__7__17.invoke(protocol.clj:3)
	at protocol$expand_deps.invokeStatic(protocol.clj:8)
	at protocol$expand_deps.invoke(protocol.clj:6)
	at clojure.lang.AFn.applyToHelper(AFn.java:154)
	at clojure.lang.AFn.applyTo(AFn.java:144)
	at clojure.lang.Compiler$InvokeExpr.eval(Compiler.java:3652)
	... 15 more
  • This does not occur with 1.6 or earlier versions
  • This does not occur if you do not try to invoke AOT
  • This may not occur for some orderings of the arguments

This appears to be related to the class being loaded by two different class loaders, and also may result in the namespace being compiled more than once. This issue has popped up for us multiple times in our production build, but it took a while to realize it was a compiler issue and to find a minimal example.






[CLJ-2072] Primitive type aliases do not always work due to meta data evaluation Created: 01/Dec/16  Updated: 01/Dec/16

Status: Open
Project: Clojure
Component/s: None
Affects Version/s: Release 1.8
Fix Version/s: None

Type: Enhancement Priority: Minor
Reporter: Volkert Oakley Jurgens Assignee: Unassigned
Resolution: Unresolved Votes: 0
Labels: typehints


 Description   

Primitive alias do not work when the meta data is evaluated, for example in the case of def.

In this example, char is interpreted to be the function char rather than the type alias. This is because clojure.lang.Compiler$DefExpr$Parser.parse evaluates the meta data on the symbol.

(def ^char x \space)
(String/valueOf x)
=> CompilerException java.lang.IllegalArgumentException: Unable to resolve classname: clojure.core$char@7b82f7d1

Instead, this has to be written as

(def ^Character/TYPE x \space)

However, when using primitive type hints in-line, they work fine:

(def x \space)
(String/valueOf ^char x)

Primitive type aliases should be handled consistently.

Googling shows that this seems to be a well-known problem, but I have not found a Jira issue for it.



 Comments   
Comment by Alex Miller [ 01/Dec/16 7:48 PM ]

Var meta is evaluated. Meta on function signatures and other locations is not. Those two things are that way for historical reasons but would at this point be breaking changes if we changed either of them, so they are not going to change.

One thing that could potentially be done is to detect this particular problem when it happens and create a warning or error. In particular, this would present as a var whose meta :tag is a function.

So, if you want to re-write this ticket as a request for error message, that is something worth considering.





[CLJ-2071] Unexpected behavior with clojure.spec/tuple and clojure.spec.test/instrument Created: 01/Dec/16  Updated: 01/Dec/16  Resolved: 01/Dec/16

Status: Closed
Project: Clojure
Component/s: None
Affects Version/s: Release 1.9
Fix Version/s: None

Type: Defect Priority: Major
Reporter: Ben Rady Assignee: Unassigned
Resolution: Declined Votes: 0
Labels: None
Environment:

Clojure 1.9, JDK 8



 Description   

It looks like stest/instrument is comparing a sequence of actual args to a vector created by spec/tuple and it doesn't match, however clojure.spec.test/instrument appears to work fine. Reading the clojure.spec guide I would think the two approaches would be equivalent.

(ns sample
  (:require [clojure.spec.test :as stest]
            [clojure.spec :as spec]))

    (defn myinc [i]
      (inc i))

    (spec/fdef myinc
               :args (spec/tuple integer?) ; Fails with the error below
               ; :args (spec/cat :i integer?) ; This works
               :ret integer?)

    (stest/instrument `myinc)
    (myinc 1)
    ; clojure.lang.ExceptionInfo: Call to #'specific.core-spec/myinc did not conform to spec:
    ; val: (1) fails at: [:args] predicate: vector?
    ; :clojure.spec/args  (1)
    ; :clojure.spec/failure  :instrument
    ; :clojure.spec.test/caller  {:file "core_spec.clj", :line 28, :var-scope specific.core-spec/fn--6046}


 Comments   
Comment by Ben Rady [ 01/Dec/16 8:11 AM ]

Note that the namespace in this example was changed. It used to be specific.core-spec, which is shown in the error.

Comment by Alex Miller [ 01/Dec/16 10:22 AM ]

Spec will create a list or a seq of the args for checking the :args spec. Tuples can only be used on vectors (because they match by index). So, this is not currently expected to work. It is recommended that you use a regex spec (s/cat) instead.





[CLJ-2070] Faster clojure.core/delay implementation Created: 29/Nov/16  Updated: 30/Nov/16

Status: Open
Project: Clojure
Component/s: None
Affects Version/s: Release 1.9
Fix Version/s: None

Type: Enhancement Priority: Major
Reporter: dennis zhuang Assignee: Unassigned
Resolution: Unresolved Votes: 0
Labels: performance
Environment:

macOS Sierra
intel Core i7 2.5G Hz
Memory 16GB

java version "1.8.0_66"
Java(TM) SE Runtime Environment (build 1.8.0_66-b17)
Java HotSpot(TM) 64-Bit Server VM (build 25.66-b17, mixed mode)


Attachments: Text File fast_delay_with_volatile_fn2.patch    
Patch: Code and Test
Approval: Prescreened

 Description   

clojure.lang.Delay uses a synchronized lock to protect the deref method, because it must make sure the Delay object is realized exactly once.

As we known synchronized lock plays worse performance under heavy concurrency. Instead, using volatile and double-checking lock in this situation improves the performance. The benchmark code is at test-delay.clj. The benchmark-delay function accepts thread count number as an argument, and it will start as many threads as the argument to access delay object concurrently as in (time (benchmark-delay 1)).

threads 1.9.0-alpha14 + patch % better
1 0.570196 ms 0.499905 ms 12
10 19.66194 ms 1.313828 ms 93
20 40.740032 ms 2.149794 ms 95
100 184.041421 ms 8.317295 ms 95

Patch: fast_delay_with_volatile_fn2.patch
Prescreened by: Alex Miller



 Comments   
Comment by Alex Miller [ 29/Nov/16 8:52 AM ]

A faster version of delay would be helpful - these are used extensively inside spec so would actually help the average case spec performance.

These whitespace errors need to be cleaned up...

$ git apply ~/Downloads/fast_delay.patch
/Users/alex/Downloads/fast_delay.patch:67: trailing whitespace.
	                try
/Users/alex/Downloads/fast_delay.patch:105: trailing whitespace.

/Users/alex/Downloads/fast_delay.patch:115: space before tab in indent.
        	    (fn []
/Users/alex/Downloads/fast_delay.patch:116: space before tab in indent.
          		    (.await barrier)
/Users/alex/Downloads/fast_delay.patch:117: space before tab in indent.
          		    (dotimes [_ 10000]
warning: squelched 8 whitespace errors
warning: 13 lines add whitespace errors.

More importantly, the double-check is on fn, so it's critical that fn is marked as volatile. You should re-run the perf test afterwards.

Comment by dennis zhuang [ 29/Nov/16 9:13 AM ]

Sorry, white spaces errors should be fixed before my attached.

But the fn doesn't need to be marked as volatile, because it's protected by synchronized in all blocks. And writing it to be null is fine here.

fn=null;

It's not like double-checking in singleton example, there is no reordering here.

Comment by Alex Miller [ 29/Nov/16 9:25 AM ]

fn is read at the top before the synchronized block - it needs to be volatile or one thread may not see the write inside the synchronized block from another thread.

Comment by dennis zhuang [ 29/Nov/16 9:41 AM ]

Yep ,but it's fine here.
If one thread can't see the writing null for fn at the top, it will enter the locking block.
The double-checking on fn!=null will make sure the fn is called at most once, and if the fn was called by another thread and was set to be null ,then current thread will fail on second checking on fn!=null and exit the locking to go on reading value or exception.

So, in the worst situation, maybe two or more threads would enter the locking block,but they all will fail on second checking on fn!=null except one thread of them success.

I don't want to declare fn to be volatile, because volatile also has a cost on reading. The fn variable may be flushed into main memory too late, but it's acceptable and safe here, and we avoid the cost of volatile reading.

Comment by Alex Miller [ 29/Nov/16 9:45 AM ]

I think you're wrong, and I'm not screening it without it being marked as volatile.

Comment by dennis zhuang [ 29/Nov/16 9:54 AM ]

The patch which mark fn volatile.

Comment by dennis zhuang [ 29/Nov/16 9:54 AM ]

The patch which does't mark fn volatile.

Comment by dennis zhuang [ 29/Nov/16 9:59 AM ]

Hi, alex.

I understand your opinion here. Though i still insist that fn doesn't need to be marked as volatile, but it's not a critical concern here.

I uploaded two patches, one is marked fn volatile, the other is not. All of them fixed the whitespace errors and update the benchmark result in ticket description.

Thanks.

Comment by dennis zhuang [ 29/Nov/16 10:15 AM ]

Rebase master.

Comment by Nicola Mometto [ 30/Nov/16 11:53 AM ]

dennis, here's an article describing why fn needs to be volatile: https://www.cs.umd.edu/~pugh/java/memoryModel/DoubleCheckedLocking.html

Comment by dennis zhuang [ 30/Nov/16 6:01 PM ]

@Nicola

I knew the double-checking issue in old JVM memory model, but it is different here.
There is no instance constructed, it's only assigning fn to be null, so it doesn't have instruments reordering. And we don't have a partial constructed instance that escapes.

But it's not critical concern here, it seems that volatile doesn't compact performance of this patch.

Thanks.





[CLJ-2069] lazy seq that encounters an exception has differing behavior on repeated use Created: 27/Nov/16  Updated: 29/Nov/16

Status: Open
Project: Clojure
Component/s: None
Affects Version/s: Release 1.8, Release 1.9
Fix Version/s: None

Type: Defect Priority: Major
Reporter: TianJun Assignee: Unassigned
Resolution: Unresolved Votes: 0
Labels: lazy
Environment:

OS X EI Capitan, Java HotSpot(TM) 64-Bit Server VM 1.8.0_101-b13


Attachments: Text File 0001-CLJ-2069-cache-exceptions-thrown-during-lazy-seq-rea.patch    
Patch: Code
Approval: Prescreened

 Description   

It seems the below does not compile with 1.8.0 and 1.9.0-alpha14, the same errors appear in both versions.

user=> (def fibonacci-1
  ((fn fib  [a b]
    (lazy-seq  (cons a  (fib b  (+ a b)))))
    0 1))

user=> (filter #(< % 100) fibonacci-1)

ArithmeticException integer overflow  clojure.lang.Numbers.throwIntOverflow (Numbers.java:1501)

user=> (filter #(< % 100) fibonacci-1)

NullPointerException   clojure.lang.Numbers.ops (Numbers.java:1013)

user=> (def fibonacci-2
         (lazy-cat [0 1] (map + (rest fibonacci-2) fibonacci-2)))

user=> (filter #(< % 100) fibonacci-2)

ArithmeticException integer overflow  clojure.lang.Numbers.throwIntOverflow (Numbers.java:1501)

user=> (filter #(< % 100) fibonacci-2)
(0 1 1 2 3 5 8 13 21 34 55 89)

Patch: 0001-CLJ-2069-cache-exceptions-thrown-during-lazy-seq-rea.patch

Proposal: Cache exceptions thrown during lazy-seq realization, to avoid re-running bodyfn which is declared as `^:once`

Prescreened by: Alex Miller



 Comments   
Comment by TianJun [ 27/Nov/16 10:42 AM ]

Maybe I should use take-while instead of filter.

However, can anyone explain why I get ArithmeticException while running

(filter #(< % 100) fibonacci-2)

for the first time and get the right result at the second time?

Comment by Nicola Mometto [ 28/Nov/16 3:27 AM ]

The NPE is caused by the interaction between:

  • lazy-seq throwing an exception while realizing part of the sequence
  • lazy-seq internally using ^:once for locals clearing

lazy-seq expects the bodyfn to be run exactly once and then the result to be cached, but if an exception gets thrown during the execution of bodyfn, the function will get run again when the sequence tries to be realized a second time. However if locals clearing has already happened (even partially) this means some locals in bodyfn will now be nil rather than holding their actual value.

Comment by Nicola Mometto [ 28/Nov/16 3:36 AM ]

Attached patch that fixes this issue





[CLJ-2068] s/explain of evaluated predicate yields :s/unknown Created: 23/Nov/16  Updated: 10/Mar/17

Status: Open
Project: Clojure
Component/s: None
Affects Version/s: Release 1.9
Fix Version/s: Release 1.9

Type: Defect Priority: Major
Reporter: Alex Miller Assignee: Alex Miller
Resolution: Unresolved Votes: 1
Labels: spec

Attachments: Text File clj-2068-2.patch     Text File clj-2068.patch    
Patch: Code and Test
Approval: Screened

 Description   

Got:

(s/explain #{1 2 3} 4)
val: 4 fails predicate: :clojure.spec/unknown

(s/explain odd? 10)
val: 10 fails predicate: :clojure.spec/unknown

Expected to receive a description of the failing predicate as in:

(s/def ::s #{1 2 3})
(s/explain ::s 4)
;; val: 4 fails spec: :user/s predicate: #{1 3 2}

(s/def ::o odd?)
(s/explain ::o 10)
val: 10 fails spec: :user/o predicate: odd?

Cause: specize was falling through on these cases to Object and just returning unknown.

Proposed:
Special handling for 2 cases:
1. Sets - explictly catch IPersistentSet and use the set as the form.
2. Functions - demunge the function name and use the qualified function name symbol as the form. Add a special check for anonymous functions and revert to ::unknown for those (not much we can do with an eval'ed anonymous function).

Patch: clj-2068-2.patch



 Comments   
Comment by Alex Miller [ 13/Dec/16 6:52 PM ]

Simplified anon fn check and added a few basic tests.

Comment by Stuart Halloway [ 10/Mar/17 11:22 AM ]

Could the Specize protocol be extended to IFn, reducing the iffiness?

Comment by Alex Miller [ 10/Mar/17 11:39 AM ]

Yes, I think that would make sense.

Comment by Ghadi Shayban [ 10/Mar/17 12:57 PM ]

Extending Specize to IFn may incur the wrath of CLJ-1152





[CLJ-2067] (s/def ::a ::b) throws unable to resolve error if ::b is not defined Created: 22/Nov/16  Updated: 22/Nov/16

Status: Open
Project: Clojure
Component/s: None
Affects Version/s: Release 1.9
Fix Version/s: Release 1.9

Type: Defect Priority: Minor
Reporter: Oliver George Assignee: Unassigned
Resolution: Unresolved Votes: 0
Labels: spec

Approval: Vetted

 Description   

It should be possible to do `(s/def ::a ::b)` before declaring ::b.

Currently this throws an "Unable to resolve" error.

Alex indicated that everything should have delays but that they are missing in some cases. This seems like one of those cases.

Examples where things work fine are `(s/def ::a (s/and ::b))` and `(s/def ::a (s/keys :req [::b]))`.






[CLJ-2066] Reflection on internal classes fails under Java 9 Created: 22/Nov/16  Updated: 06/Apr/17

Status: Open
Project: Clojure
Component/s: None
Affects Version/s: Release 1.8
Fix Version/s: Release 1.9

Type: Defect Priority: Major
Reporter: Toby Crawley Assignee: Unassigned
Resolution: Unresolved Votes: 1
Labels: interop, reflection, regression

Attachments: Text File tcrawley.CLJ-2066.2017-03-16.patch    
Patch: Code and Test
Approval: Incomplete

 Description   

Due to changes in reflective access for the Jigsaw module system in Java 9, the Reflector will now fail on some cases that worked in previous Java versions.

(def fac (javax.xml.stream.XMLInputFactory/newInstance))
(.createXMLStreamReader fac (java.io.StringReader. ""))

Here fac will be an instance of com.sun.xml.internal.stream.XMLInputFactoryImpl, which is an implementation of javax.xml.stream.XMLInputFactory. In the new java.xml module, javax.xml.stream is an exported package, but the XMLInputFactoryImpl is an internal implementation of the public interface in that package. The invocation of createXMLStreamReader will be reflective and the Reflector will attempt to invoke the method based on the implementation class, which is not accessible outside the module, yielding:

IllegalAccessException class clojure.lang.Reflector cannot access class com.sun.xml.internal.stream.XMLInputFactoryImpl (in module java.xml) because module java.xml does not export com.sun.xml.internal.stream to unnamed module @4722ef0c
	jdk.internal.reflect.Reflection.throwIllegalAccessException (Reflection.java:423)
	jdk.internal.reflect.Reflection.throwIllegalAccessException (Reflection.java:414)
	jdk.internal.reflect.Reflection.ensureMemberAccess (Reflection.java:112)
	java.lang.reflect.AccessibleObject.slowCheckMemberAccess (AccessibleObject.java:632)
	java.lang.reflect.AccessibleObject.checkAccess (AccessibleObject.java:624)
	java.lang.reflect.Method.invoke (Method.java:539)
	clojure.lang.Reflector.invokeMatchingMethod (Reflector.java:93)
	clojure.lang.Reflector.invokeInstanceMethod (Reflector.java:28)

One workaround here is to avoid the reflective call by type-hinting to the public exported interface:

(.createXMLStreamReader ^javax.xml.stream.XMLInputFactory fac (java.io.StringReader. ""))

Another (undesirable) workaround is to export the private package from java.xml to the unnamed module (which is the module used when code is loaded from the classpath rather than from a module) when invoking java/javac:

java --add-exports=java.xml/com.sun.xml.internal.stream=ALL-UNNAMED --add-exports=java.xml/com.sun.xml.internal.stream.writers=ALL-UNNAMED --add-exports=java.xml/com.sun.org.apache.xerces.internal.impl=ALL-UNNAMED ...

Proposed: The attached patch will check for and catch IllegalAccessException in the Reflector. When it occurs, the Reflector will attempt to invoke the method on all super-classes and super-interfaces in case one of them can succeed.

Patch: tcrawley.CLJ-2066.2017-02-14.patch

More info:



 Comments   
Comment by Toby Crawley [ 22/Nov/16 10:02 AM ]

This is the root cause of http://dev.clojure.org/jira/browse/DXML-32

Comment by Toby Crawley [ 14/Feb/17 3:48 PM ]

I don't see any indication that Java 9 will change in our favor, so I'm attaching a patch to fix this in Clojure itself.

Comment by Alex Miller [ 07/Mar/17 4:04 PM ]

Screening comments:

I think by just inspecting the patch that the new code will attempt to invoke the method on all super classes and interfaces, many of which will not have the applicable method and will thus throw IllegalArgumentException, possibly before finding the correct one. Can we reduce the effort here by only attempting the call on the super types that actually have the method? It would be good to expand the tests to also include this case if possible if I'm just mis-reading things (I did not attempt to construct this scenario).

Also, is there a method we can call to determine whether the method is accessible before invoking and needing to catch IllegalAccessException? (Obviously, we would want something that is not new in Java 9 so that this call worked on older JDKs too.)

Comment by Herwig Hochleitner [ 07/Mar/17 5:37 PM ]

This may be a stupid question, but has there been any research, whether [1] and [2] could be utilized to handle this without catching exceptions?

[1] http://download.java.net/java/jdk9/docs/api/java/lang/Class.html#getModule--
[2] http://download.java.net/java/jdk9/docs/api/java/lang/reflect/Module.html#isExported-java.lang.String-java.lang.reflect.Module-

Comment by Alex Miller [ 07/Mar/17 6:27 PM ]

Nope, that's the kind of thing I was asking about in my comment. However, those are jdk 9 only methods and right now Clojure supports jdk 6+. That's not impossible to deal with but does increase the difficulty.

Comment by Toby Crawley [ 08/Mar/17 2:30 PM ]

Alex: you are correct, an intermediate ancestor that does not provide the method will cause the lookup to fail - I'll rework and provide a new patch with better tests.

Herwig: we could build Clojure as a multi-release jar in order to have code that can use Java 9 features, but that would definitely complicate the build. However, there may be other Java 9 issues that may require us to take that option (I'm investigating one potential issue now).

Comment by Alex Miller [ 08/Mar/17 3:28 PM ]

Yeah, I was looking at the multi release jar stuff yesterday. I would really like to avoid that if at all possible as it adds a bunch of work for build, ci, and test.

Comment by Toby Crawley [ 13/Mar/17 9:42 PM ]

I've attached a new patch (tcrawley.CLJ-2066.2017-03-13.patch) and removed the old one (tcrawley.CLJ-2066.2017-02-14.patch). The new patch checks for any matching methods before calling invokeMatchingMethod; this will prevent the lack of the method on an ancestor from throwing IllegalArgumentException and aborting the lookup process.

I wasn't able to provide a test for the case where the method was missing on an ancestor, since that requires trying to invoke the method through an interface that doesn't provide it (all subclasses will provide methods from parent classes, so you need an ancestor tree that pulls in an interface), and I couldn't find anything in Java's stdlib that did that and was a non-exposed class. To implement a proper test for this, I believe we'd need to construct a module ourselves, which would require changes to the build process to build that module (and run the test) only under Java 9. I did, however, confirm locally that the old code was broken in this regard, and that the new patch fixes it.

Comment by Toby Crawley [ 14/Mar/17 12:00 PM ]

After reviewing tcrawley.CLJ-2066.2017-03-13.patch, I can see a new issue with it: if the method being invoked exists only on the non-public class, then the patch will throw an IllegalArgumentException claiming that the method doesn't exist instead of the original, more accurate IllegalAccessException. I'll rework and provide another patch.

Comment by Toby Crawley [ 16/Mar/17 9:22 AM ]

New patch attached (tcrawley.CLJ-2066.2017-03-16.patch) that will properly throw the original IllegalAccessException when the method only exists on the non-public class. Patch tcrawley.CLJ-2066.2017-03-13.patch has been deleted.

Comment by Ghadi Shayban [ 05/Apr/17 6:56 PM ]

In JDK9 public classes housed in a non-exported package are now inaccessible. Previously public classes might not be public any more. Clojure should link to them neither non-reflectively nor reflectively – perhaps a slightly larger adjustment to the compiler/reflector method lookup?

Comment by Herwig Hochleitner [ 06/Apr/17 8:07 AM ]

Ghadi: does that mean, that we can't even get at the Class object to call http://download.java.net/java/jdk9/docs/api/java/lang/Class.html#getModule-- ?

Comment by Alex Miller [ 06/Apr/17 9:07 AM ]

My understanding (which may be faulty) is that the new module system is independent from the classloader setup (although the base classloader hierarchy has changed a bit) and that reflection is still largely the same with respect to examining classes and methods. But you may run into issues with invoking constructors or methods that are not allowed according to access restrictions. I'm not sure if the ability to load classes is affected (but I didn't think so).





[CLJ-2065] reduce-kv fails on subvec Created: 20/Nov/16  Updated: 09/Mar/17

Status: Open
Project: Clojure
Component/s: None
Affects Version/s: Release 1.9
Fix Version/s: None

Type: Defect Priority: Minor
Reporter: Steve Miner Assignee: Unassigned
Resolution: Unresolved Votes: 0
Labels: collections

Approval: Triaged

 Description   

reduce-kv works as expected on vectors with the element index passed as the "key" argument. However, it fails with a subvec because clojure.lang.APersistentVector$SubVector does not implement IKVReduce.

(reduce-kv + 0 [1 2 3])
9

(reduce-kv + 0 (subvec [1 2 3] 1))
IllegalArgumentException No implementation of method: :kv-reduce of protocol: #'clojure.core.protocols/IKVReduce found for class: clojure.lang.APersistentVector$SubVector clojure.core/-cache-protocol-fn (core_deftype.clj:583)

One work around is to copy the subvec into a vector:

(reduce-kv + 0 (into [] (subvec [1 2 3] 1)))
6

Note however, the `vec` would not work here. Since Clojure 1.7, vec will return a subvec rather than copying.

(reduce-kv + 0 (vec (subvec [1 2 3] 1)))
IllegalArgumentException No implementation of method: :kv-reduce of protocol: #'clojure.core.protocols/IKVReduce found for class: clojure.lang.APersistentVector$SubVector clojure.core/-cache-protocol-fn (core_deftype.clj:583)



 Comments   
Comment by Steve Miner [ 20/Nov/16 12:53 PM ]

Here is my current work-around:

(extend-type clojure.lang.APersistentVector$SubVector
  clojure.core.protocols/IKVReduce
  (kv-reduce [subv f init]
    (transduce (map-indexed vector)
               (fn ([ret] ret) ([ret [k v]] (f ret k v)))
               init
               subv)))

In my tests it was usually faster to copy the subvec into a regular vector but I like the look of the transduce fix. It would probably be faster to add a native Java implementation in APersistentVector.java. I'm willing to do the work if the Clojure/core team wants a patch.

Comment by Steve Miner [ 09/Mar/17 1:38 PM ]

Revised work-around using more interop for better performance. Comparable to the speed of normal vector reduce-kv.

(when-not (satisfies?   clojure.core.protocols/IKVReduce (subvec [1] 0))
  (extend-type clojure.lang.APersistentVector$SubVector
    clojure.core.protocols/IKVReduce
    (kv-reduce
      [subv f init]
      (let [cnt (.count subv)]
        (loop [k 0 ret init]
          (if (< k cnt)
            (let [val (.nth subv k)
                  ret (f ret k val)]
              (if (reduced? ret)
                @ret
                (recur (inc k) ret)))
            ret))))))




[CLJ-2063] Show longest path explain error first Created: 17/Nov/16  Updated: 17/Nov/16

Status: Open
Project: Clojure
Component/s: None
Affects Version/s: Release 1.9
Fix Version/s: Release 1.9

Type: Enhancement Priority: Major
Reporter: Alex Miller Assignee: Unassigned
Resolution: Unresolved Votes: 0
Labels: error-reporting, spec

Attachments: Text File longest-explain.patch    
Patch: Code
Approval: Vetted

 Description   

It is observed that the explain problem with the longest path is most likely the one that parsed the furthest and is thus the closest to the user's actual intent.

Proposed: Sort the explain problems with longest path first.

Patch: longest-explain.patch






[CLJ-2062] Spec import and refer-clojure macros Created: 17/Nov/16  Updated: 14/Mar/17  Resolved: 14/Mar/17

Status: Closed
Project: Clojure
Component/s: None
Affects Version/s: Release 1.9
Fix Version/s: Release 1.9

Type: Enhancement Priority: Major
Reporter: Alex Miller Assignee: Unassigned
Resolution: Completed Votes: 0
Labels: core.specs, spec

Attachments: Text File import-referclj-2.patch    
Patch: Code
Approval: Ok

 Description   

Add specs for import and refer-clojure.

Patch:

  • Fixes some indentation of previous specs
  • Factors out ::filters spec from ::ns-refer-clojure
  • Factors out ::import-list from ::ns-import
  • Reuses ::filters in ::ns-refer
  • Reuses ::filters in ::use-prefix-list
  • Removes :ret any? in ::ns-use (no need for it)
  • Adds clojure.core/import spec
  • Adds clojure.core/refer-clojure spec

Patch: import-referclj-2.patch






[CLJ-2061] Better error message when exercise-fn called on fn without :args spec Created: 17/Nov/16  Updated: 17/Nov/16

Status: Open
Project: Clojure
Component/s: None
Affects Version/s: Release 1.9
Fix Version/s: Release 1.9

Type: Enhancement Priority: Major
Reporter: Alex Miller Assignee: Unassigned
Resolution: Unresolved Votes: 0
Labels: errormsgs, spec

Attachments: Text File clj-2061.patch    
Patch: Code
Approval: Vetted

 Description   
;; no spec
user=> (s/exercise-fn str)
IllegalArgumentException No implementation of method: :specize* of protocol: #'clojure.spec/Specize found for class: nil  clojure.core/-cache-protocol-fn (core_deftype.clj:583)

;; no :args spec
user=> (s/fdef str :ret string?)
user=> (s/exercise-fn str)
IllegalArgumentException No implementation of method: :specize* of protocol: #'clojure.spec/Specize found for class: nil  clojure.core/-cache-protocol-fn (core_deftype.clj:583)

Proposed: Check for missing :args spec and throw better error

user=> (s/exercise-fn str)
Exception Unable to resolve args spec  clojure.spec/exercise-fn (spec.clj:1811)

user=> (s/fdef str :ret string?)
user=> (s/exercise-fn str)
Exception Unable to resolve args spec  clojure.spec/exercise-fn (spec.clj:1811)

Patch: clj-2061.patch






[CLJ-2060] Add undef to remove a spec Created: 16/Nov/16  Updated: 16/Nov/16

Status: Open
Project: Clojure
Component/s: None
Affects Version/s: Release 1.9
Fix Version/s: Release 1.9

Type: Enhancement Priority: Major
Reporter: Alex Miller Assignee: Unassigned
Resolution: Unresolved Votes: 1
Labels: spec

Attachments: Text File clj-2060.patch    
Patch: Code
Approval: Vetted

 Description   

Currently there is no provided way to remove a spec from the registry. During interactive development, particularly when working on complicated or recursive specs, it would be useful to have this ability.

Proposed: Add s/undef that removes a spec from the registry. In the patch it returns the updated registry, although that may be more harmful than helpful at the repl (where receiving nil would probably be less noise). Another option would be to return true/false indicating whether the key was in the registry. However, as the registry is held in an atom, this has a race and would be more involved to implement (so I didn't).

Patch: clj-2060.patch



 Comments   
Comment by Alex Miller [ 16/Nov/16 11:55 AM ]

Moving to 1.9 so it will get looked at, may not be added.





[CLJ-2059] explain-data problems don't provide resolved symbols under :pred Created: 15/Nov/16  Updated: 16/Nov/16

Status: Open
Project: Clojure
Component/s: None
Affects Version/s: Release 1.9
Fix Version/s: Release 1.9

Type: Defect Priority: Major
Reporter: David Nolen Assignee: Unassigned
Resolution: Unresolved Votes: 0
Labels: error-reporting, spec

Attachments: Text File clj-2059.patch    
Patch: Code and Test
Approval: Vetted

 Description   

Currently, explain-data returns unresolved preds. This is a problem when trying to write a custom explain print function that chooses what to do based on the predicate as it does not have enough information.

(require '[clojure.spec :as s])

(defn password-valid-length? [pass]
  (> (count pass) 12))

(s/def ::password (s/and string? password-valid-length?))

(-> (s/explain-data ::password "foobar")
  ::s/problems first :pred) 
;;=> password-valid-length?  
;;expected: user/password-valid-length?

Cause: Currently, explain* returns preds in the abbrev form for all spec impls.

Proposed: Have explain* return resolved preds. In cases where the abbreviated form should be used (anything for human consumption at either the repl or an error message), convert to it. For example, explain-printer should (and already does) do this.

Patch: clj-2059.patch

  • Changes all spec impls to avoid calling abbrev on preds when building explain-data
  • Undoes op-describe change for s/? in CLJ-2042 and fixes this at a higher level by calling res on the incoming pred (this is a better fix)
  • Changes the expected test output for spec tests to expect the resolved pred





[CLJ-2058] s/keys doesn't check non-keyword elements in :req and :req-un vectors Created: 15/Nov/16  Updated: 15/Nov/16  Resolved: 15/Nov/16

Status: Closed
Project: Clojure
Component/s: None
Affects Version/s: Release 1.9
Fix Version/s: None

Type: Defect Priority: Minor
Reporter: Eugene Pakhomov Assignee: Unassigned
Resolution: Not Reproducible Votes: 0
Labels: spec


 Description   

As can be seen here https://github.com/clojure/clojure/blob/master/src/clj/clojure/spec.clj#L430 the s/keys macro filters out any non-keyword element from :req and :req-un before checking it, but later it still uses them in the arguments to map-spec-impl.
Compare the behavior when passing non-keyword elements to :opt and :opt-un.



 Comments   
Comment by Alex Miller [ 15/Nov/16 9:29 AM ]

:req and :req-un support `and` and `or` connectives, :opt and :opt-un do not. So this all seems right to me.

I don't see any bug here?

Comment by Eugene Pakhomov [ 15/Nov/16 9:58 AM ]

I see your point. But how are `and` and `or` related to "all keys must be namespace-qualified keywords"?
And why to do the check at all for :opt and :opt-un that are not even used, when the check for :req and :req-un just allows any forms, not just documented `and` and `or`?

Comment by Alex Miller [ 15/Nov/16 10:08 AM ]

and and or are connectors, not keys.

I'd be fine if the ticket showed something invalid - no actual bug is being shown here. If you improve the ticket, I will re-open.





[CLJ-2057] Function spec missing :ret can yield wrong answer for valid? Created: 14/Nov/16  Updated: 16/Nov/16

Status: Open
Project: Clojure
Component/s: None
Affects Version/s: Release 1.9
Fix Version/s: Release 1.9

Type: Defect Priority: Major
Reporter: James Gatannah Assignee: Unassigned
Resolution: Unresolved Votes: 0
Labels: spec

Attachments: Text File clj-2057.patch    
Patch: Code
Approval: Vetted

 Description   

Create a spec on a function, leaving off the :ret.

user> (s/fdef ::foo :args (s/cat :n integer?))
=> :user:foo
user> (defn f [n] (* n 2))
=> #'user/f
;; Need org.clojure/test.check on classpath
user> (s/valid? ::foo f)
=> false
user> (s/explain-data ::foo f)
=> nil

Cause: Originally, :ret spec was required. We loosened that requirement, but parts of the implementation still assume that the :ret spec exists (valid-fn, etc). Here, s/valid? is incorrectly returning false because the returned value does not match the non-existent :ret spec, even though f should be fine. explain-data is doing the right thing (it's not failing).

Proposed: Patch in any? as the default :ret spec if it's missing. Another way to go would be to verify that all of the existing fspec conform and explain code worked as intended when :ret spec is missing - it seems like we would effectively be swapping in an any? spec in all of those cases though, so the proposed path seemed easier.

Patch: clj-2057.patch



 Comments   
Comment by Alex Miller [ 15/Nov/16 9:35 AM ]

fyi, fdef should take a qualified symbol, not a qualified keyword. To do what you're doing here, I would do:

(s/def ::foo (s/fspec :args (s/cat :n integer?)))
(defn f [n] (* n 2))
(s/valid? ::foo f)
(s/explain-data ::foo f)

Not that you will get a different result, but that's really the intent of the api. You're leaning a bit too much on implementation details that may change (namely that fdef is effectively def of an fspec - this didn't used to be the case and may not be the case in the future).





[CLJ-2056] Efficient shortcut for (first (filter pred coll)) idiom Created: 11/Nov/16  Updated: 30/Jan/17

Status: Open
Project: Clojure
Component/s: None
Affects Version/s: None
Fix Version/s: None

Type: Enhancement Priority: Major
Reporter: Nikita Prokopov Assignee: Unassigned
Resolution: Unresolved Votes: 36
Labels: None

Attachments: Text File clj-2056-clojure-core-seek-2.patch    
Patch: Code and Test
Approval: Prescreened

 Description   

It's a common task to look up for an item in a collection based on predicate. Currently Clojure has no direct support for that in clojure.core. Instead, our options are:

1. (first (filter pred coll)) will create intermediate lazy sequence and might evaluate pred up to 31 extra times in case of chunked sequence

2. (some #(when (pred %) %) coll) will short-circuit on first match, but won't catch false value in something like (some #(when false? %) [true false true])

Additionally, both of these workarounds a) obscure the purpose of the code, and b) do not handle custom not-found values.

Attached is a patch that makes use of efficiency of reduce-able collections, handles edge cases like looking for false? or nil?, and supports optional not-found value.

Examples:

(seek odd? (range)) => 1
(seek pos? [-1 1]) => 1
(seek pos? [-1 -2] ::not-found) => ::not-found
(seek nil? [1 2 nil 3] ::not-found) => nil

Patch: clj-2056-clojure-core-seek-2.patch

Prescreening notes: I think the general approach is good. Is it necessary to support nil? and false? preds? Or would a transduce formulation like the one in comments be sufficient.



 Comments   
Comment by Alex Miller [ 11/Nov/16 8:54 AM ]

Just as an interesting aside, the new halt-when transducer could actually be used to create something like this too (if you set aside the desire to support nil? and false? preds).

(transduce (comp (filter pred) (halt-when any?)) identity nil coll)

Patch has some trailing whitespace in the test code - could you clean that up?

Comment by Nikita Prokopov [ 12/Nov/16 3:46 AM ]

Attaching patch with trailing whitespace cleaned

Comment by Nikita Prokopov [ 12/Nov/16 3:46 AM ]

Thanks Alex! Attached new patch with whitespace cleaned

Comment by Moritz Heidkamp [ 30/Jan/17 12:34 PM ]

I had a similar train of thought today and arrived at the idea of adding transducer support to first, e.g. (first (filter pred) coll). This could potentially be as efficient as the special-purpose seek function proposed above but would of course not support the not-found value. However, it seems like a natural extension to first and could be useful for other purposes, too.





[CLJ-2055] binding-form spec parses symbol-only maps incorrectly Created: 08/Nov/16  Updated: 14/Mar/17  Resolved: 14/Mar/17

Status: Closed
Project: Clojure
Component/s: None
Affects Version/s: Release 1.9
Fix Version/s: Release 1.9

Type: Defect Priority: Major
Reporter: Brandon Bloom Assignee: Unassigned
Resolution: Completed Votes: 0
Labels: spec
Environment:

1.9.0-alpha14


Attachments: Text File CLJ-2055-01.patch    
Patch: Code
Approval: Ok

 Description   

The :clojure.core.specs/binding-form spec incorrectly treats some maps as sequential bindings.

Actual:

user=> (s/conform :clojure.core.specs/binding-form '{x y})
[:seq {:elems [[:seq {:elems [[:sym x] [:sym y]]}]]}]

Expected:

user=> (s/conform :clojure.core.specs/binding-form '{x y})
[:map {x y}]

Cause:

When there is no :keys, :strs, or :syms from :clojure.core.specs/map-special-binding, then :clojure.core.specs/seq-binding-form treats a map as sequential.

Proposed fix:

Include an (s/and vector? ...) check. See patch.

Patch: CLJ-2055-01.patch
Screened by: Alex Miller






[CLJ-2054] generator for `any?` occasionally generates `Double/NaN` for which equality semantics don't apply, and that is a problem for the :ret spec of many functions. Created: 07/Nov/16  Updated: 14/Dec/16

Status: Open
Project: Clojure
Component/s: None
Affects Version/s: Release 1.9
Fix Version/s: None

Type: Enhancement Priority: Major
Reporter: Dimitrios Piliouras Assignee: Unassigned
Resolution: Unresolved Votes: 0
Labels: generator, spec
Environment:

Ubuntu 16.10 - Oracle Java 8


Approval: Triaged

 Description   

The generator for `any?` will occasionally give back Double/NaN value(s). Since NaNs & equality (via `=`) don't get along, :ret spec'ing a fn which transforms/processes a collection according to a predicate, becomes rather problematic. That's because the most obvious thing to check under :ret (the case where the predicate didn't return true for any value, and so the output coll should be equal to the input coll because nothing was transformed/processed), cannot be expressed trivially.

The workaround I've come up with in my own specs is to spec the elements of the collection with `(s/and any? (complement double-NaN?))` instead of just `any?`, and it works. However, even though I can live without NaNs in the tests, I must admit it still feels sort of hacky.

Ideas:

1) The generator for `any?` could be hardcoded to never return Double/NaN. Sounds rather invasive.
2) The generator for `any?` could be reworked to somehow be configurable wrt allowing/prohibiting Double/NaNs. Then perhaps a dynamic-var and/or a macro (e.g. `without-NaNs`) could expose this (just brainstorming here).
3) The generator for `any? could stay as is, but a new equality operator could be added (e.g. `clojure.spec/===`), which somehow ignores NaNs (a naive implementation for instance might walk the data-structures and replace all NaNs with keywords, and only then perform a regular comparison).



 Comments   
Comment by Alex Miller [ 08/Nov/16 10:29 AM ]

Should consider whether this change is more appropriate in test.check or in the spec generator for any?.

Comment by Dimitrios Piliouras [ 11/Nov/16 12:31 PM ]

It turns out that my workaround does not fully work. I literally just stumbled in the following case:

{nil {[] {NaN 0}}}

which is a conforming value for:

(s/def ::persistent-map
(s/map-of ::anything-but-NaN ::anything-but-NaN)) ;; (s/and any? (complement double-NaN?))

So basically, the inner collections can still have NaNs. So far I've got 4 specs that I've written and faced this problem on all of them.





[CLJ-2052] .class vs .clj isn't picked correctly when either .class or .clj are not in a jar Created: 03/Nov/16  Updated: 04/Nov/16  Resolved: 04/Nov/16

Status: Closed
Project: Clojure
Component/s: None
Affects Version/s: None
Fix Version/s: None

Type: Defect Priority: Major
Reporter: Mike Kaplinskiy Assignee: Unassigned
Resolution: Not Reproducible Votes: 0
Labels: None


 Description   

The code that figures out the last modification date (https://github.com/clojure/clojure/blob/master/src/jvm/clojure/lang/RT.java#L389) uses URLConnection.getLastModified. For `file://` URLs this always returns 0.

One of the resulting bugs: when both .class and .clj files are present on a directory-based classpath, .clj files are always preferred, regardless of modification time.



 Comments   
Comment by Alex Miller [ 03/Nov/16 6:15 PM ]

Can you provide OS and JVM info? This might be env-specific.

Comment by Mike Kaplinskiy [ 03/Nov/16 6:20 PM ]

Sure - I'm on macOS Sierra on Oracle Java 8:

$ java -version
java version "1.8.0_74"
Java(TM) SE Runtime Environment (build 1.8.0_74-b02)
Java HotSpot(TM) 64-Bit Server VM (build 25.74-b02, mixed mode)
$ uname -a
Darwin mikekap-mbp.local 16.1.0 Darwin Kernel Version 16.1.0: Thu Oct 13 21:26:57 PDT 2016; root:xnu-3789.21.3~60/RELEASE_X86_64 x86_64 i386 MacBookPro11,5 Darwin
Comment by Alex Miller [ 03/Nov/16 6:28 PM ]

It's prob not too fun but some example code would be awfully handy.

Comment by Mike Kaplinskiy [ 03/Nov/16 7:13 PM ]

Sorry this looks like it was my fault - the path I was looking at when testing didn't exist (seems .getLastModified always returns 0 on those instead of throwing an exception).

Sorry for the noise.

Comment by Alex Miller [ 04/Nov/16 9:26 AM ]

Closed per comments.





[CLJ-2051] Typo in clojure.instant/validated docstring Created: 03/Nov/16  Updated: 03/Nov/16

Status: Open
Project: Clojure
Component/s: None
Affects Version/s: Release 1.8
Fix Version/s: None

Type: Defect Priority: Trivial
Reporter: Greg Leppert Assignee: Unassigned
Resolution: Unresolved Votes: 0
Labels: docstring, typo

Attachments: Text File 58f6bca6ba64ca343b43585463eff1be74aeb965.patch    
Patch: Code
Approval: Prescreened

 Description   
  • "Return a function which constructs and instant by calling constructor
    + "Return a function which constructs an instant by calling constructor

Prescreened by: Alex Miller






[CLJ-2050] Remove redundant key comparisons in HashCollisionNode Created: 03/Nov/16  Updated: 03/Nov/16

Status: Open
Project: Clojure
Component/s: None
Affects Version/s: None
Fix Version/s: None

Type: Enhancement Priority: Minor
Reporter: Kwang Yul Seo Assignee: Unassigned
Resolution: Unresolved Votes: 0
Labels: collections, performance

Attachments: Text File 0001-Remove-redundant-key-comparisons-in-HashCollisionNod.patch    
Patch: Code
Approval: Prescreened

 Description   

Comparing key to array[idx] is redundant as findIndex already performed key comparison to find the index.

Prescreened by: Alex Miller






[CLJ-2049] Improve clojure.zip documentation Created: 25/Oct/16  Updated: 27/Oct/16

Status: Open
Project: Clojure
Component/s: None
Affects Version/s: Release 1.8, Release 1.9
Fix Version/s: None

Type: Enhancement Priority: Minor
Reporter: Brighid M Assignee: Unassigned
Resolution: Unresolved Votes: 0
Labels: docstring, zip
Environment:

All


Attachments: File improve-zip-docs.diff    
Patch: Code
Approval: Triaged

 Description   

The clojure.zip module has extremely terse docstrings that are helpful as reference material but completely unhelpful to someone approaching the module cold. Expanded docstrings would make this piece of code much more visible to users who might find it helpful.



 Comments   
Comment by Brighid M [ 25/Oct/16 9:03 PM ]

A patch that improves clojure.zip's docstrings.

Comment by Alex Miller [ 27/Oct/16 8:42 AM ]

I would prefer if we could minimize the size of the changes by removing diffs that just add periods, change line breaks, add whitespace, or make other non-essential changes. That will help focus on the things that matter like changes in the ns docstring, zipper, etc. Additionally this ticket talks about doc strings but also makes changes in exception messages - I'd prefer code changes to be in a separate ticket. Also, please don't remove the comment sections in the files.

If you would like to convert the test comment section into actual tests, that would be a great separate ticket (I feel like I might have even written a patch to do that at one point but I don't see a ticket for it!).

Keeping this patch entirely focused on essential docstring changes is the best way to ensure its timely inclusion.

Comment by Brighid M [ 27/Oct/16 4:32 PM ]

Alex — to clarify, I'm hearing "this ticket should include two patches: one with the major prose additions and one with small proofreading-ish changes" and "this ticket's patches should not include the changes to exception messages nor the comment-move"?

Supplemental question: is there a style guide hanging around for the code & documentation style of the Clojure core? I poked around for such a guide on clojure.org and in JIRA: I didn't find one, but maybe I overlooked it. I was looking for a guideline about the comment section. It would be good to have a hint about "please don't touch these," because AFAICT they don't communicate that by themselves.

Comment by Alex Miller [ 27/Oct/16 5:43 PM ]

Actually, I'd prefer to have just one patch with the major prose additions. I don't think the minor changes are worth doing and will be a distraction from the more important prose.

Unfortunately, there is no style guide for Clojure core, and in fact it's been written by many people over many years so there generally isn't a consistent style throughout the code. Generally Rich prefers that debug code or comments that trace to him are left intact (git blame can help there). More generally, the simpler a patch is to review, the easier it is for it to stay good and be easily reviewed.

Thanks!

Comment by Brighid M [ 27/Oct/16 6:34 PM ]

> Actually, I'd prefer to have just one patch with the major prose additions.
Okay, I'll produce that patch and attach it to this ticket.

> I don't think the minor changes are worth doing and will be a distraction from the more important prose. Unfortunately, there is no style guide for Clojure core, and in fact it's been written by many people over many years so there generally isn't a consistent style throughout the code.
I think you just made an argument for why it is worth doing minor changes: consistency generally makes things more accessible. However, now is clearly not the time to litigate that, so I'm gonna come back with just the ns/docstring version of this patch.





[CLJ-2048] java.lang.ClassCastException: [Ljava.lang.Object; cannot be cast to [Ljava.lang.StackTraceElement Created: 21/Oct/16  Updated: 21/Oct/16

Status: Open
Project: Clojure
Component/s: None
Affects Version/s: Release 1.8, Release 1.9
Fix Version/s: None

Type: Defect Priority: Major
Reporter: Gerrit Jansen van Vuuren Assignee: Unassigned
Resolution: Unresolved Votes: 0
Labels: None

Attachments: Text File CLJ-2048-b.patch    
Patch: Code
Approval: Prescreened

 Description   

clojure.core/throw-if creates an array to call Exception.setStracktrace() without specifying the array type. This works fine when passed at least one StackTraceElement, but in the case where passed no stack trace elements (all are filtered or stack traces are being elided by the JVM), this will be an Object[] which results in a ClassCastException:

Exception in thread "main" java.lang.ClassCastException: [Ljava.lang.Object; cannot be cast to [Ljava.lang.StackTraceElement;
        at clojure.core$throw_if.invokeStatic(core.clj:5649)
        at clojure.core$load_one.invokeStatic(core.clj:5698)
        at clojure.core$compile$fn__5682.invoke(core.clj:5903)
        at clojure.core$compile.invokeStatic(core.clj:5903)
        at clojure.core$compile.invoke(core.clj:5895)
        at clojure.lang.Var.invoke(Var.java:379)
        at clojure.lang.Compile.main(Compile.java:67)

This is tricky to reproduce because it involves stack trace filtering so there is no reproducing case here.

Patch: CLJ-2048-b.patch
Prescreened by: Alex Miller



 Comments   
Comment by Gerrit Jansen van Vuuren [ 21/Oct/16 4:18 AM ]

patch calls into-array with StackTraceElement type

Comment by Alex Miller [ 21/Oct/16 8:01 AM ]

How do you cause this to occur?

Comment by Alex Miller [ 21/Oct/16 8:11 AM ]

into-array will create a typed array based on the first element of the seq it is passed, so generally you should see a StackTraceElement[] here. I think the only time this would fail is if it was passed no stack trace elements.

Comment by Alex Miller [ 21/Oct/16 8:19 AM ]

I'd be happy to move this through screening, but the patch needs to be in the proper format (see http://dev.clojure.org/display/community/Developing+Patches).

Comment by Gerrit Jansen van Vuuren [ 21/Oct/16 8:42 AM ]

I'm trying to reproduce this in a way that can be presented here, but I got the compile error just after doing some serious package renaming, and can't reproduce it outside of the project itself.

Comment by Gerrit Jansen van Vuuren [ 21/Oct/16 8:45 AM ]

ok, I'll reformat the patch after reading (http://dev.clojure.org/display/community/Developing+Patches)

Comment by Gerrit Jansen van Vuuren [ 21/Oct/16 9:15 AM ]

I've created a new patch based on the guidelines, attached as file: CLJ-2048-b.patch.

Just to summarise:
The into-array returns the correct type if its provided with a none empty sequence, but if the sequence is empty it cannot know the type and then returns an object array. Because we set the array here to a java method Exception::setStackTrace passing it an object array causes a ClassCastException. One fix is to check for an empty sequence, but a less verbose way is just to pass the type which is known as part of the call to into-array, this is what is done in the patch CLJ-2048-b.patch.





[CLJ-2046] generate random subsets of or'd required keys in map specs Created: 17/Oct/16  Updated: 07/Apr/17

Status: Open
Project: Clojure
Component/s: None
Affects Version/s: Release 1.9
Fix Version/s: Release 1.9

Type: Enhancement Priority: Major
Reporter: David Chelimsky Assignee: Unassigned
Resolution: Unresolved Votes: 1
Labels: generator, spec

Attachments: Text File clj-2046-2.patch     Text File map-spec-gen-enhancements.patch    
Patch: Code and Test
Approval: Screened

 Description   

(s/keys :req [(or ::x ::y)]) always generates maps with both ::x and ::y but it should also generate maps with either ::x or ::y.

The attached patch supports arbitrarily deeply nested or and and expressions within the values of :req and :req-un in map specs. It also uses the same 'or' mechanism for :opt and :opt-un keys, thereby replacing the use of clojure.core/shuffle with clojure.test.check.generators/shuffle, ensuring repeatability of the generators.

Patch: clj-2046-2.patch

Screened by: Alex Miller



 Comments   
Comment by Alex Miller [ 07/Apr/17 3:59 PM ]

Patch updated to apply to current master, no semantic changes, attribution retained.





[CLJ-2045] bean function not working Created: 17/Oct/16  Updated: 17/Oct/16  Resolved: 17/Oct/16

Status: Closed
Project: Clojure
Component/s: None
Affects Version/s: Release 1.9
Fix Version/s: None

Type: Defect Priority: Major
Reporter: Roy Varghese Assignee: Unassigned
Resolution: Duplicate Votes: 0
Labels: None
Environment:

(defproject hello-jcloud "0.1.0-SNAPSHOT"
:description "FIXME: write description"
:url "http://example.com/FIXME"
:license {:name "Eclipse Public License"
:url "http://www.eclipse.org/legal/epl-v10.html"}
:dependencies [#_[org.clojure/clojure "1.8.0"]
[org.clojure/clojure "1.9.0-alpha13"]
[org.clojure/tools.logging "0.1.0"]
[org.apache.jclouds/jclouds-all "2.0.0-SNAPSHOT"]]
:repositories {"jclouds-snapshots" "https://repository.apache.org/content/repositories/snapshots"})



 Description   

user> (bean (java.util.Date.))
UnsupportedOperationException empty clojure.core.proxy$clojure.lang.APersistentMap$ff19274a.empty (:-1)
user>

It works when clojure/1.8.0 is uncommented.



 Comments   
Comment by Alex Miller [ 17/Oct/16 7:38 AM ]

This is a dupe of http://dev.clojure.org/jira/browse/CLJ-2027 which will probably be in the next alpha.





[CLJ-2044] Four functions in clojure.instant have incomplete documentation Created: 15/Oct/16  Updated: 24/Oct/16

Status: Open
Project: Clojure
Component/s: None
Affects Version/s: Release 1.4, Release 1.5, Release 1.6, Release 1.7, Release 1.8, Release 1.9
Fix Version/s: None

Type: Enhancement Priority: Major
Reporter: Bruce Adams Assignee: Bruce Adams
Resolution: Unresolved Votes: 0
Labels: docstring, instant

Attachments: Text File defns-for-instant-def-timestamp.patch     Text File defns-for-instant.patch    
Patch: Code
Approval: Prescreened

 Description   

Of the five public functions defined in clojure.instant, these four:

  • parse-timestamp
  • read-instant-calendar
  • read-instant-date
  • read-instant-timestamp

are each declared as a Var without any arglists metadata. This means documentation does not contain function calling information.

In http://clojure.github.io/clojure/clojure.instant-api.html, each of these functions is described as a var and there is no Usage: ... information given.

The output of doc does not include argument list information. For example:

user=> (doc clojure.instant/read-instant-date)
-------------------------
clojure.instant/read-instant-date
  To read an instant as a java.util.Date, bind *data-readers* to a map with
this var as the value for the 'inst key. The timezone offset will be used
to convert into UTC.

A related problem is that stack traces show anonymous functions instead of the names for any of these functions. For example:

user=> (clojure.instant/read-instant-timestamp "123")
RuntimeException Unrecognized date/time syntax: 123  clojure.instant/fn--6879/fn--6880 (instant.clj:107)

Proposed: Refactor the code into defn functions which makes the code clearer and addresses the documentation issue. An alternate approach would be to apply :arglists metadata to the vars.

Patch: defns-for-instant-def-timestamp.patch
Prescreened: Alex Miller



 Comments   
Comment by Bruce Adams [ 15/Oct/16 12:40 PM ]

Proposed solution: refactor the definitions of the four problematic functions to be defined using defn.

Comment by Bruce Adams [ 16/Oct/16 5:24 PM ]

Some of my thinking leading to the solution I propose.

Initially, when I realized that I didn't know what arguments parse-timestamp required, I assumed the appropriate fix was to enhance the docstring. Then I noticed that the on-line documentation for functions was formatted quite differently from the text output by doc. Any decent fix was going to have to feed function information into different variants of documentation formatting code.

I can guess, from other examples such as first, that :arglists metadata is what indicates that a var is a function. One possible solution would be to add :arglists to each of the four functions. It felt cleaner to refactor the code into simple defn functions. Refactoring code just for the side effect of documentation seems a bit odd, but the code itself strikes me as more legible after my refactoring.

Comment by Alex Miller [ 17/Oct/16 9:53 AM ]

This seems reasonable to me. I would move the timestamp regex into a separate (private) var instead of creating it in parse-timestamp.

It's possible the way these functions were defined was designed to minimize startup time or something like that, but I don't have any background on that.

Comment by Bruce Adams [ 23/Oct/16 4:06 PM ]

Updated patch based on Alex's great suggestion. This adds a separate, private, def for the timestamp regex.





[CLJ-2043] s/form of conformer is broken Created: 14/Oct/16  Updated: 14/Mar/17  Resolved: 14/Mar/17

Status: Closed
Project: Clojure
Component/s: None
Affects Version/s: Release 1.9
Fix Version/s: Release 1.9

Type: Defect Priority: Major
Reporter: Alex Miller Assignee: Unassigned
Resolution: Completed Votes: 3
Labels: spec

Attachments: Text File clj-2043.patch    
Patch: Code
Approval: Ok

 Description   

s/form of s/conformer is wrong:

(s/form (s/conformer str))
=> str

Proposed: Fix the form for conformer to match the conformer call.

Patch: clj-2043.patch






[CLJ-2042] s/form of s/? does not resolve pred Created: 14/Oct/16  Updated: 28/Oct/16  Resolved: 28/Oct/16

Status: Closed
Project: Clojure
Component/s: None
Affects Version/s: Release 1.9
Fix Version/s: Release 1.9

Type: Defect Priority: Major
Reporter: Alex Miller Assignee: Unassigned
Resolution: Completed Votes: 0
Labels: spec

Attachments: Text File clj-2042.patch    
Patch: Code
Approval: Ok

 Description   
user=> (require '[clojure.spec :as s])
nil
user=> (s/form (s/? int?))
(clojure.spec/? int?)

Patch: clj-2042.patch






[CLJ-2041] clojure.spec/keys requires input collections conform to clojure.core/map? Created: 11/Oct/16  Updated: 08/Mar/17

Status: Open
Project: Clojure
Component/s: None
Affects Version/s: Release 1.9
Fix Version/s: Release 1.9

Type: Defect Priority: Minor
Reporter: Timothy Baldridge Assignee: Unassigned
Resolution: Unresolved Votes: 6
Labels: spec

Approval: Vetted

 Description   

I would like to use specs to validate Datomic entities. However, `s/keys` is too restrictive in that it requires input collections to conform to `clojure.core/map?` instead of some more primitive interface (for example clojure.lang.ILookup or clojure.lang.Associative).



 Comments   
Comment by Alex Miller [ 04/Nov/16 8:21 AM ]

s/keys uses IPersistentMap's Iterable support for iterating through all entries for validation. ILookup and Associative do not support iteration. So, that's why it is the way it is. But, understand the desire.

Comment by Alex Miller [ 04/Nov/16 8:34 AM ]

Datomic entities are seqable so maybe that's a potential path (would be slower for actual PHMs though).

Comment by Odin Standal [ 04/Nov/16 8:35 AM ]

Thanks for following up. So any ideas or guidance on how to use clojure.spec with Datomic entities?

Comment by Alex Miller [ 04/Nov/16 8:37 AM ]

For now, you could use into to pour an entity into a PHM before validating. I hesitate to suggest it, but that could even be in the spec with a leading conformer.

Comment by Alex Miller [ 14/Nov/16 12:16 PM ]

Moving this into 1.9 for the moment just so we don't lose it. Not sure whether we can or will actually do anything with this though.

Comment by Alex Miller [ 08/Mar/17 10:00 AM ]

This is related to CLJ-2080 as it's the same basic issue of what map-of and every-kv actually expect as valid inputs, which is: something that seqs to map entries. We can't really write a predicate for it without an interface "ISeqsToMapEntries" (intentionally bad name) to indicate this. java.util.Map, IPersistentMap, etc imply this, but an object can seq to map entries without meeting all the constraints of those much broader interfaces. ILookup does not and should not imply this.





[CLJ-2040] Allow runtime modification of REPL exception handling Created: 11/Oct/16  Updated: 15/Feb/17

Status: Open
Project: Clojure
Component/s: None
Affects Version/s: Release 1.9
Fix Version/s: None

Type: Enhancement Priority: Major
Reporter: Luke VanderHart Assignee: Unassigned
Resolution: Unresolved Votes: 0
Labels: repl

Attachments: Text File CLJ-2040-dynamic-repl-exceptions.patch    
Patch: Code
Approval: Triaged

 Description   

Problem Statement

Clojure's REPL is capable of paramterizing almost every aspect of its functionality, including how uncaught exceptions are printed. In the current implementation, these customization hooks are passed in as arguments and closed over, meaning that they cannot be changed once the REPL is started.

Many development tools want to override how the REPL handles uncaught errors. Examples of useful customizations include (but are not limited to):

  • Formatted exception messages (including whitespace and ANSI coloring)
  • Alternative representations for certain types of exceptions (e.g, Spec errors)
  • Dropping into a graphical interaction mode to better inspect ex-data.

Currently, this type of customization must be applied before a REPL is started, meaning that changing how a REPL displays errors requires support from (or plugins to) a third-party tool such as Boot or Leiningen.

Alternatives


1. Take no action.

Third-party tool support is required to create customized exception handling in the REPL. Tools have different techniques for doing this:

  • nREPL can intercept the exception on the wire and passes it through middleware
  • Leiningen plugins alter the root binding of clojure.main/repl-caught.
  • Boot allows users to build a task to invoke clojure.main/repl with the desired arguments.

Users will continue to select one of these according to their tooling preferences.

Benefits:
1. No effort or changes to the existing code.

Tradeoffs:
1. Tools will continue to implement their own diverse, sometimes hacky techniques for printing custom exceptions.
2. Any library intended to provide alternative exception handling will be tied to a specific launcher tool.

2. Make the REPL exception handler dynamically rebindable

If the REPL exception handler were a dynamic, thread-local var, users and libraries could change the behavior of the currently running REPL.

Benefits:
1. Users and libraries can freely override how exceptions are printed, regardless of how Clojure was launched.
2. Fully backwards compatible with existing tools.

Tradeoffs:
1. It will be possible for library authors to provide "bad" or poorly reasoned error printers. This is still possible with launch tools, but the barrier of entry is even lower with libraries.

The attached patch implements this option.

3. Encourage users to start new REPLs instead

In many Clojure environments, it's possible to explicitly launch a REPL from within another REPL. This sub-REPL could have the desired :caught hook.

Benefits:
1. No effort or changes to the existing code.
2. "Functionally pure", and in alignment with the evident design of the current REPL.

Tradeoffs:
1. There is a non-trivial subset of Clojure developers who do not know exactly how REPLs work. They are likely to be confused or subject to increased cognitive load. Insofar as this set of beginner/intermediate developers are precisely who enhanced error messages are meant to help in the first place, this solution is counterproductive.
2. For better or for worse, many existing and widely used tools do not support this. This does not work at all in nREPL, for example. However, even the simplest command-line REPLs behavior would change for the worse; sending a EOF (accidentally or otherwise) would always kill the sub-REPL with no feedback as to what just happened.



 Comments   
Comment by Alex Miller [ 15/Feb/17 9:39 AM ]

On the repl-caught var, it would be good to mention the signature of the handler, namely that it takes an exception, is expected to print or otherwise handle the exception, and that its return will be ignored.

Rather than the changes in repl, what if you added a dynamic-repl-caught and change that to be the default caught handler in repl? Or even just changed repl-caught itself.





[CLJ-2039] typo in deftype doc string Created: 09/Oct/16  Updated: 11/Oct/16

Status: Open
Project: Clojure
Component/s: None
Affects Version/s: Release 1.8
Fix Version/s: None

Type: Enhancement Priority: Trivial
Reporter: Steve Miner Assignee: Unassigned
Resolution: Unresolved Votes: 0
Labels: docstring, typo

Attachments: Text File CLJ-2039-deftype-typo.patch    
Patch: Code
Approval: Prescreened

 Description   

The doc string for deftype refers to "record class" where it should say "type".






[CLJ-2038] Clojure.spec/exercise-fn should accept custom generator map Created: 08/Oct/16  Updated: 07/Nov/16

Status: Open
Project: Clojure
Component/s: None
Affects Version/s: Release 1.9
Fix Version/s: None

Type: Enhancement Priority: Minor
Reporter: Andrea Richiardi Assignee: Unassigned
Resolution: Unresolved Votes: 1
Labels: spec

Attachments: File CLJ-2038-exercise-fn-should-accept-custom-generator.diff    
Approval: Triaged

 Description   

I tried to generate some data with exercise-fn but could not do it because I need to carry my own custom generators.
At the moment though, there is no way to add them, like an optional parameter.



 Comments   
Comment by Alex Miller [ 11/Oct/16 12:09 PM ]

Patch would be welcome for this!

Comment by Laszlo Török [ 07/Nov/16 9:23 AM ]

First attempt to form a patch.

I am unsure whether it is ok to use the overloaded 3rd argument approach or take a keyword arg approach for the optional arguments.

i.e. ([sym-or-fn n & {:keys [fspec gen]}])

On the other hand, it only make sense to pass one or the other and given sym-or-fn, fspec-or-gen seems appropriate.

Currently the test suite for c.spec is lacking, I'm happy to add a few example-based tests for exercise-fn, once the approach is approved.
Also





[CLJ-2037] specs in registry lack :file metadata despite having :line, :column Created: 08/Oct/16  Updated: 01/Dec/16

Status: Open
Project: Clojure
Component/s: None
Affects Version/s: Release 1.9
Fix Version/s: None

Type: Enhancement Priority: Major
Reporter: Felix Andrews Assignee: Unassigned
Resolution: Unresolved Votes: 0
Labels: spec

Approval: Triaged

 Description   

As of 1.9.0-alpha13, specs in the registry lack :file metadata despite having :line, :column

user=> (require '[clojure.spec :as s])
user=> (-> (s/registry) (get :clojure.core.specs/arg-list) (meta))
{:line 1118, :column 5, :clojure.spec/name :clojure.core.specs/arg-list}
user=> (-> (s/registry) (get 'clojure.core/let) (meta))
{:line 1675, :column 5, :clojure.spec/name clojure.core/let}

This would be useful because:

  • we could list all the specs defined in a project, by filtering the registry.
  • we could read the source of a spec, like clojure.repl/source, for pretty formatting.

(specifically, for use in Codox https://github.com/weavejester/codox/pull/134 )

I had a quick look but couldn't see where the metadata is set.
Cheers



 Comments   
Comment by Alex Miller [ 08/Oct/16 11:12 AM ]

You can use s/describe or s/form to grab the source of a spec now, btw.

Comment by Felix Andrews [ 12/Oct/16 11:29 PM ]

The following works in my tests. (For testing I used in-ns, @#'registry-ref, #'ns-qualify)).

The approach is to set the registry item metadata after a def. It is not enough to set metadata on the def'd value because it is subsequently altered inside def.

(ns clojure.spec)
(alias 'c 'clojure.core)

(defmacro def
  [k spec-form]
  (let [k (if (symbol? k) (ns-qualify k) k)
        m (assoc (meta &form) :file *file*)]
    `(do
       (def-impl '~k '~(res spec-form) ~spec-form)
       (swap! registry-ref update '~k vary-meta c/merge ~m)
       '~k)))

(defmacro fdef
  [fn-sym & specs]
  (let [k (ns-qualify fn-sym)
        m (assoc (meta &form) :file *file*)]
    `(do
       (clojure.spec/def ~fn-sym (clojure.spec/fspec ~@specs))
       (swap! registry-ref update '~k vary-meta c/merge ~m)
       '~k)))

You can use s/describe or s/form to grab the source of a spec now, btw.

Yes, that's nice except for longer specs when line wrapping and indentation would help.

Comment by Jozef Wagner [ 01/Dec/16 12:31 PM ]

Note that current :line and :column meta are not pointing to the place where the spec was defined but to the clojure/spec.clj file, e.g. second example (c.c/let) points to fspec-impl





[CLJ-2036] Generators for some? and any? only return collections or nil Created: 07/Oct/16  Updated: 11/Oct/16

Status: Open
Project: Clojure
Component/s: None
Affects Version/s: Release 1.9
Fix Version/s: Release 1.9

Type: Defect Priority: Major
Reporter: Sean Corfield Assignee: Unassigned
Resolution: Unresolved Votes: 0
Labels: generator, spec

Approval: Vetted

 Description   

Both of these should generate a better variety of values (strings, keywords, symbols, numbers) in addition to collections and nil.



 Comments   
Comment by Sean Corfield [ 07/Oct/16 11:55 AM ]

See al