<< Back to previous view

[CLJ-2251] Generic spec walking for clojure.spec Created: 11/Oct/17  Updated: 18/Apr/18

Status: Open
Project: Clojure
Component/s: None
Affects Version/s: Release 1.9
Fix Version/s: None

Type: Feature Priority: Major
Reporter: Tommi Reiman Assignee: Unassigned
Resolution: Unresolved Votes: 6
Labels: None
Environment:

[org.clojure/spec.alpha "0.1.134"]



 Description   

Problem

To do runtime coercion, specs need to be walked twice to strip away the branching information: s/conform + s/unform. This introduced extra latency (see the sample below).

Proposal

New versatile s/walk* to support generic spec walking.

Current status

Still, when running s/conform + s/unform, we walk the specs twice - which is performance-wise suboptimal. Below is a sample, with Late 2013 MacBook Pro with 2,5 GHz i7, with JVM running as -server.

(require '[clojure.spec.alpha :as s])

(s/def ::id int?)
(s/def ::name string?)
(s/def ::languages (s/coll-of #{:clj :cljs} :into #{}))
(s/def ::street string?)
(s/def ::zip string?)
(s/def ::number int?)

(s/def ::address (s/keys
                   :req-un [::street ::zip ::number]))

(s/def ::user (s/keys
                :req [::id]
                :req-un [::name ::address]
                :opt-un [::languages]))

(def value {::id 1
            :name "Liisa"
            :languages #{:clj :cljs}
            :address {:street "Hämeenkatu"
                      :number 24
                      :zip "33200"}})

; 2.0 µs
(cc/quick-bench
  (s/conform ::user value))

; 6.2 µs
(cc/quick-bench
  (s/unform ::user (s/conform ::user value)))

Despite s/conform is relatively fast, we triple the latency in the sample when running also s/unform. As we know already that we are not interested in the branching info, we could just not emit those.

Suggestion

s/walk* to replace both s/confrom* and s/unform*, maybe even s/explain*. It would take extra mode argument, which would be a Keyword of one of the following:

  • :validate - return false on first failing spec
  • :conform - like the current s/conform*, maybe also return s/explain results?
  • :unform - like the current s/unform*
  • :coerce - s/conform* + s/unform*, could be optimized (e.g. if no branching info, just return the value)

The public apis could be remain the same (+ optional extra argument with CLJ-2116), and a new s/coerce to call the s/walk* with :coerce.

Results

Single sweep validation & coercion. Happy runtime.



 Comments   
Comment by Tommi Reiman [ 17/Apr/18 7:40 AM ]

Renamed the issue. Instead of Keyword argument, it should take a function to walk the spec to support arbitrary walking applications.





[CLJ-2201] proxy-super is not threadsafe, it should be made safe or documented to be unsafe Created: 05/Jul/17  Updated: 17/Apr/18

Status: Open
Project: Clojure
Component/s: None
Affects Version/s: None
Fix Version/s: None

Type: Enhancement Priority: Major
Reporter: Kevin Downey Assignee: Unassigned
Resolution: Unresolved Votes: 0
Labels: None


 Description   

Coming from java you might expect proxy-super to be pretty innocuous, but proxy-super operates by mutating the proxy object then restoring it after the call to proxy-super is invoked. This can lead to very weird behavior. If you have a proxy with method M, which invokes proxy-super, then while that proxy-super is running all calls to M on that proxy object will immediately invoke the super M not the proxied M.

Actually making proxy-super safe (not just threadsafe, but also safe when invoked later on in the same callstack) seems like it might be really hard, but it would be nice. Alternatively some blinking hazard lights in the docstring might be a good idea.



 Comments   
Comment by Freek Paans [ 17/Apr/18 12:58 AM ]

I ran into this while trying to wrap calls to a queue: https://stackoverflow.com/questions/49862954/clojure-proxy-multithreading-issue





[CLJ-2348] 'check' has inconsistent behavior for required and optional map keys Created: 16/Apr/18  Updated: 16/Apr/18

Status: Open
Project: Clojure
Component/s: None
Affects Version/s: Release 1.9
Fix Version/s: None

Type: Defect Priority: Minor
Reporter: Ben Brinckerhoff Assignee: Unassigned
Resolution: Unresolved Votes: 0
Labels: spec
Environment:

org.clojure/spec.alpha "0.1.143"


Approval: Triaged

 Description   

Repro:

(s/def :ex/f fn?)
(s/def :ex/m (s/keys :opt [:ex/f]))
(s/fdef my-fn
        :args (s/cat :m :ex/m))
(defn my-fn [f])
(clojure.spec.test.alpha/check `my-fn)

Actual: Exception is thrown - "Unable to construct gen at: [:m :ex/f] for: :ex/f"

Expected: A value should be returned containing the failure. This is the behavior that will occur if you replace the ":opt" with a ":req" in the keys spec.

I would expect this value to contain a failure such that:

(ex-data (:result (:clojure.spec.test.check/ret (first (clojure.spec.test.alpha/check `my-fn))))) ;; => #:clojure.spec.alpha{:path [:m :ex/f], :form :ex/f, :failure :no-gen}





[CLJ-2347] 'inst?' spec generator produces unreadable instants Created: 15/Apr/18  Updated: 15/Apr/18

Status: Open
Project: Clojure
Component/s: None
Affects Version/s: Release 1.9
Fix Version/s: None

Type: Defect Priority: Minor
Reporter: David Bürgin Assignee: Unassigned
Resolution: Unresolved Votes: 0
Labels: spec
Environment:

Clojure 1.9.0
org.clojure/spec.alpha 0.1.143
org.clojure/test.check 0.9.0


Approval: Triaged

 Description   

The spec generator associated with inst? may produce instants that are technically valid but cannot be read back (and also are not practical in reality).

(require '[clojure.spec.alpha :as s])

(second (last (s/exercise inst? 100)))
;; => #inst "883641-02-19T16:17:26.482-00:00"

#inst "883641-02-19T16:17:26.482-00:00"
;; => RuntimeException Unrecognized date/time syntax: 883641-02-19T16:17:26.482-00:00  clojure.instant/fn--7987/fn--7988 (instant.clj:107)

Minor issue, but I ran into this while interacting with inst generator/generated values in the REPL.






[CLJ-2346] Improve chunked sequence processing Created: 08/Apr/18  Updated: 08/Apr/18

Status: Open
Project: Clojure
Component/s: None
Affects Version/s: Release 1.9
Fix Version/s: None

Type: Enhancement Priority: Major
Reporter: Renzo Borgatti Assignee: Unassigned
Resolution: Unresolved Votes: 1
Labels: performance

Attachments: Text File CLJ-2346.patch    
Patch: Code

 Description   

Chunked sequential processing might benefit from invoking reduce on ArrayChunk with the required transformation instead of the current dotimes. The functions affected are map, filter and keep. The following table shows the relevant benchmarks in ms (number in [brackets] are benchmarks on the patch that are worse than the original).

long range

f 1.10.0-master doall ms 1.10.0+patch doall ms 1.10.0-master chunk-last ms 1.10.0+patch chunk-last ms
(map inc lr) 4.05 3.34 2.69 2.55
(keep identity lr) 2.88 2.69 1.09 0.76
(filter odd? lr) 2.43 2.26 1.61 1.35

range

f 1.10.0-master doall ms 1.10.0+patch doall ms 1.10.0-master chunk-last ms 1.10.0+patch chunk-last ms
(map inc lr) 4.38 [4.85] 2.02 [2.43]
(keep identity lr) 2.62 2.36 0.97 0.86
(filter odd? lr) 2.46 2.30 1.85 1.49

vector

f 1.10.0-master doall ms 1.10.0+patch doall ms 1.10.0-master chunk-last ms 1.10.0+patch chunk-last ms
(map inc lr) 3.94 [4.14] 2.20 [2.62]
(keep identity lr) 2.68 2.49 0.99 0.80
(filter odd? lr) 2.61 2.49 1.67 1.55

gvec

f 1.10.0-master doall ms 1.10.0+patch doall ms 1.10.0-master chunk-last ms 1.10.0+patch chunk-last ms
(map inc lr) 5.17 4.50 3.18 2.98
(keep identity lr) 3.73 3.27 1.90 [2.02]
(filter odd? lr) 3.38 3.23 2.56 [2.92]

All benchmarks executed with Criterium using the form: {{(let [xs chunked-seq] (bench (walk (f xs))))}} where:

  • "chunked-seq" is one of: (range 100000), (range 1e5), (vec (range 1e5)) or (into (vector-of :int) (range 1e5))
  • "walk" is either doall or chunk-last (see below for definition)
  • "f" is one of: (map inc xs), (filter odd? xs) or (keep identity xs).

Observations:

  • The more and larger the chunks the more the benefits. Custom (chunked) sequences with larger chunks (than 32) could additionally benefit from the changes.
  • keep-indexed and map-indexed also use the same pattern but the change make things worse by requiring an different way to increment the index.
  • for is also chunk-aware, but it uses an explicit loop to handle :let, :when, :while cases that is difficult to separate from the chunk-buffer changes.
  • chunk-last is a chunk-aware function to access the last element. Compared to doall that walks the sequence one by one, chunk-last is more efficient for both current code and changes. The function is: {{(defn chunk-last [xs] (when-let [xs (seq xs)] (if-let [cn (chunk-next xs)] (recur cn) (last xs))))}}
  • The initial pre-definition of dotimes can go.


 Comments   
Comment by Alex Miller [ 08/Apr/18 6:45 PM ]

Can you squash the patch?

Comment by Renzo Borgatti [ 08/Apr/18 6:50 PM ]

Sure done.

Comment by Alex Miller [ 08/Apr/18 10:19 PM ]

Thanks!





[CLJ-2181] try accepts multiple catch blocks for the same class Created: 07/Jun/17  Updated: 08/Apr/18

Status: Open
Project: Clojure
Component/s: None
Affects Version/s: Release 1.8, Release 1.9
Fix Version/s: None

Type: Defect Priority: Minor
Reporter: Justin Glenn Smith Assignee: Unassigned
Resolution: Unresolved Votes: 1
Labels: compiler
Environment:

any


Attachments: Text File 0001-Disallow-unreachable-code-in-catch-exprs.patch    
Approval: Triaged

 Description   

try silently accepts multiple catch blocks for the same class, but only the first one gets called

user=> (try (/ 1 0) (catch Exception _ (println "a")) (catch Exception _ (println "b")))
a
nil


 Comments   
Comment by John Schmidt [ 08/Apr/18 11:49 AM ]

I've attached a patch which fixes this problem by bringing Clojure's try/catch semantics more inline with Java. For each catch clause, the compiler checks if a previous catch clause is a supertype if the current one. If it is, an IllegalArgumentException is thrown. This is O(n^2) in the number of catch clauses, but since n will be low in the vast majority of cases, and since the work is done at parse time, it should not pose any performance issues.

I had to modify an unrelated test since it actually had multiple catch clauses that caught Exception which is no longer allowed with this patch.





[CLJ-2345] Catching non-Throwable produces invalid bytecode Created: 07/Apr/18  Updated: 08/Apr/18

Status: Open
Project: Clojure
Component/s: None
Affects Version/s: Release 1.9
Fix Version/s: None

Type: Defect Priority: Major
Reporter: John Schmidt Assignee: Unassigned
Resolution: Unresolved Votes: 0
Labels: compiler

Attachments: Text File 0001-Error-when-parsing-try-catch-if-catch-target-is-not-.patch    
Patch: Code and Test

 Description   

In Clojure 1.9, if you throw an object that is not an instance of Throwable, you get the following error message:

user=> (try "something" (catch Object o "oops"))

VerifyError (class: user$eval1432, method: invokeStatic signature: ()Ljava/lang/Object;) catch_type not a subclass of Throwable  java.lang.Class.getDeclaredConstructors0 (Class.java:-2)

The error message isn't too bad: it tells you most of what you want to know (except the actual class that you tried to catch).

But in a not to distant future Clojure might upgrade to Java 1.8 bytecode, and when that happens the error will look like this:

user=> (try "something" (catch Object o "oops"))

VerifyError Catch type is not a subclass of Throwable in exception handler 6
Exception Details:
  Location:
    user$eval1444.invokeStatic()Ljava/lang/Object; @6: astore_1
  Reason:
    Type 'java/lang/Object' (constant pool 17) is not assignable to 'java/lang/Throwable'
  Bytecode:
    0000000: 120d 4ba7 000a 4c12 0f4b a700 032a b0
  Exception Handler Table:
    bci [0, 3] => handler: 6
  Stackmap Table:
    same_locals_1_stack_item_frame(@6,Object[#17])
    append_frame(@13,Object[#21])
  java.lang.Class.getDeclaredConstructors0 (Class.java:-2)

To me this looks like an internal compiler error and I don't think it should be exposed to users.

The patch fixes the issue by adding a check at try expr parse time that the thing you are catching is a subclass of Throwable.



 Comments   
Comment by Alex Miller [ 08/Apr/18 7:55 AM ]

The compiler should never produce byte code that has VerifyErrors so this is definitely a bug.





[CLJ-2325] = on sorted collections with different key types still incorrectly throws Created: 20/Feb/18  Updated: 06/Apr/18

Status: Open
Project: Clojure
Component/s: None
Affects Version/s: Release 1.9
Fix Version/s: None

Type: Defect Priority: Major
Reporter: Daniel Compton Assignee: Unassigned
Resolution: Unresolved Votes: 0
Labels: collections


 Description   

CLJ-1242 partially fixed = throwing an exception when comparing with sorted collections, but there are still cases where this will throw.

(= {:todos 1} (sorted-map 1 2))

ClassCastException java.lang.Long cannot be cast to clojure.lang.Keyword  clojure.lang.Keyword.compareTo (Keyword.java:114)

Some of the tests in the original patch on CLJ-1242 also still throw exceptions. - https://dev.clojure.org/jira/secure/attachment/12084/0001-fix-for-CLJ-1242-tests.patch.

Note that this is not symmetrical, the reverse doesn't throw.

(def a {:todos 1})
(def b (sorted-map 1 2))

(= a b)
=> <throws>
(= b a)
=> false


 Comments   
Comment by Daniel Compton [ 06/Mar/18 2:47 PM ]

This issue also exists in ClojureScript (which is where we first discovered it).

Comment by Steve Miner [ 06/Apr/18 10:35 AM ]

The rejected fix for CLJ-1242 that protected entryAt from ClassCastException would have fixed this as well. The root of the issue is that the exception can come from checking keys so it makes sense to fix it there. The accepted fix for CLJ-1242 protected only the = test with the sorted-map first. The more basic key access question was punted. I expect this to work:

(= (get (sorted-map 1 2) :a :missing) :missing)

I think it's worth taking another look at entryAt. Most users would expect the following to succeed:

(every? (fn [create] (= (get (create 1 2) :a :missing) :missing)) [hash-map array-map sorted-map])

If there's a performance concern, then the restriction on testing keys should be documented.





[CLJ-2344] 'Exception Not enough arguments for format definition' deep inside pprint Created: 22/Jan/18  Updated: 01/Apr/18

Status: Open
Project: Clojure
Component/s: None
Affects Version/s: None
Fix Version/s: None

Type: Defect Priority: Minor
Reporter: Jonathan Rees Assignee: David Miller
Resolution: Unresolved Votes: 0
Labels: printing
Environment:

Clojure 1.9.0. MacOS 10.12.5



 Description   

My first time submitting a bug. I didn't understand Jira very well; it asked me which project and I couldn't figure out the right project (I assumed ClojureCLR because the pprint issues seem to be tagged 'CLR', but I don't know if ClojureCLR = CLR or even what CLR stands for) and I couldn't figure out if the bug had already been reported (every search seemed to turn up lots of results that were not relevant, didn't include all my search terms). And I didn't see any "Read this before submitting a bug report!!" page like I'd expect for a project of this size. So my apologies if I got this wrong.

Expected behavior:

user> (use 'clojure.pprint)
user> (with-pprint-dispatch code-dispatch
         (write '(ns bleh
                   (:refer-clojure :only [])
                   (:require [blah :refer :all]))
                :pretty true))
(ns bleh
  (:refer-clojure :only [])
  (:require [blah :refer :all]))
nil
user>

Actual behavior:

user> (use 'clojure.pprint)
user> (with-pprint-dispatch code-dispatch
         (write '(ns bleh
                   (:refer-clojure :only [])
                   (:require [blah :refer :all]))
                :pretty true))
Exception Not enough arguments for format definition  clojure.pprint/next-arg (cl_format.clj:93)
(ns


 Comments   
Comment by Jonathan Rees [ 23/Jan/18 10:08 AM ]

Another similar situation: try pretty printing (ns) - you get a stack overflow.

Comment by David Miller [ 01/Apr/18 1:56 PM ]

This behavior is consistent with ClojureJVM.
That doesn't mean it is correct.
However, I prefer errors that originate with ClojureJVM be fixed there first and then migrate over to ClojureCLR as I update to their patches.
I am inquiring about transferring this over. If there is no direct way to transfer, I will post a JIRA issue on this with them.





[CLJ-2343] define and load classes in memory with gen-class Created: 30/Mar/18  Updated: 30/Mar/18

Status: Open
Project: Clojure
Component/s: None
Affects Version/s: None
Fix Version/s: None

Type: Enhancement Priority: Major
Reporter: Nicola Mometto Assignee: Unassigned
Resolution: Unresolved Votes: 1
Labels: gen-class

Attachments: Text File 0001-CLJ-2343-define-and-load-class-while-JITing-gen-clas.patch    
Patch: Code
Approval: Triaged

 Description   

Currently gen-class only works while AOT compiling, but just evaluates to nil while JIT loading.
The reason for this behaviour is historical and no longer relvant since CLJ-979 landed in 1.7, which fixed the dynamic classloader definition issues and also made this exact same change for gen-interface. The only reason why this wasn't also done for gen-class is that I forgot about it.

This patch fixes this inconsistency

Patch: 0001-CLJ-2343-define-and-load-class-while-JITing-gen-clas.patch






[CLJ-2342] Add `invoke` function Created: 30/Mar/18  Updated: 30/Mar/18

Status: Open
Project: Clojure
Component/s: None
Affects Version/s: None
Fix Version/s: None

Type: Feature Priority: Minor
Reporter: Nicola Mometto Assignee: Unassigned
Resolution: Unresolved Votes: 4
Labels: None

Attachments: Text File 0001-CLJ-2342-add-invoke.patch    
Approval: Triaged

 Description   

When writing code that needs to do predicate dispatch (such as code analysis libraries), it's quite common to need an `invoke` function for a pattern that looks like:

(defn classify [x]
  (condp invoke x
    string? (string-f x ..)
    number? (number-f x ..)
    ..))

This is definitely expressible without `invoke`, but the `condp invoke` pattern reads quite nicely.

Additionally `invoke` is also useful in the cases where one has a sliding window of functions and values, for patterns like:

(apply map invoke funs vals)

Patch: 0001-CLJ-2342-add-invoke.patch



 Comments   
Comment by Rick Moynihan [ 30/Mar/18 11:22 AM ]

Some use cases for this are also when you have partially applied all the arguments to a function already and want to call it.

(map (comp :result invoke) fns)

apply can't be used for these cases as it requires arguments to be provided.





[CLJ-2341] apply should support application without having to supply args Created: 30/Mar/18  Updated: 30/Mar/18  Resolved: 30/Mar/18

Status: Closed
Project: Clojure
Component/s: None
Affects Version/s: Release 1.9
Fix Version/s: None

Type: Enhancement Priority: Minor
Reporter: Rick Moynihan Assignee: Unassigned
Resolution: Declined Votes: 1
Labels: None


 Description   

e.g. The following doesn't work because apply currently requires a sequence of arguments:

(apply println)

It would be useful if apply supported this arity, so you could write things like (comp :foo apply) instead of (comp :foo #(%)) or (map apply println)



 Comments   
Comment by Alex Miller [ 30/Mar/18 8:14 AM ]

apply is clear about requiring a final args list argument.

(apply println []) is the correct usage and I don't think it makes sense to make this a special case.





[CLJ-1857] clojure.string/split docstring does not match the behavior of parameter "limit" Created: 27/Nov/15  Updated: 29/Mar/18  Resolved: 29/Mar/18

Status: Closed
Project: Clojure
Component/s: None
Affects Version/s: Release 1.7
Fix Version/s: None

Type: Enhancement Priority: Trivial
Reporter: Miikka Koskinen Assignee: Unassigned
Resolution: Duplicate Votes: 0
Labels: string

Attachments: File CLJ_1857_split_docs_update.diff    

 Description   
clojure.string/split
([s re] [s re limit])
  Splits string on a regular expression.  Optional argument limit is
  the maximum number of splits. Not lazy. Returns vector of the splits.

What happens is that limit is the maximum number of parts returned, not the number of splits done. If limit is 1, no splits are done, while I'd expect at most one split to be done. It's a bit of a matter of terminology, but I think that the text could be clarified. Based on ClojureDocs examples, I'm not the only one who was confused.

user=> (str/split "1 2 3" #" ")
["1" "2" "3"]
user=> (str/split "1 2 3" #" " 1)
["1 2 3"]
user=> (str/split "1 2 3" #" " 2)
["1" "2 3"]


 Comments   
Comment by Alex Miller [ 29/Nov/15 2:52 PM ]

To me, the last sentence indicates that "split" (as a noun) is being used to refer to the parts resulting from splitting but there is some ambiguity in the prior sentence. Would "parts" be better?

I don't get your point on the clojuredocs examples - those make sense to me.

Comment by Stephen Hopper [ 09/Feb/16 10:00 PM ]

The docs are a bit ambiguous as "splits" is a verb in the first sentence, but a noun in the other two occurrences. I believe the ClojureDocs example being referred to is likely this one (mostly because of the comment in it):

; Note that the 'limit' arg is the maximum number of strings to
; return (not the number of splits)
user=> (str/split "q1w2e3r4t5y6u7i8o9p0" #"\d+" 5)
["q" "w" "e" "r" "t5y6u7i8o9p0"]

Because split is the name of the function and the action being performed, I think it makes sense to leave it as the verb in the first sentence and replace the other two occurrences with "parts". Does that sound reasonable?

Comment by Alex Miller [ 29/Mar/18 3:46 PM ]

Incorporated these changes into the existing pending patch for the same function at CLJ-1360.





[CLJ-1360] clojure.string/split strips trailing delimiters Created: 18/Feb/14  Updated: 29/Mar/18

Status: Open
Project: Clojure
Component/s: None
Affects Version/s: None
Fix Version/s: None

Type: Defect Priority: Minor
Reporter: Tim McCormack Assignee: Unassigned
Resolution: Unresolved Votes: 2
Labels: docstring, string

Attachments: Text File clj-1360-2.patch     Text File clj-1360.patch    
Patch: Code
Approval: Prescreened

 Description   

clojure.string/split and clojure.string/split-lines inherit the bizarre default behavior of java.lang.String#split(String,int) in stripping trailing consecutive delimiters:

(clojure.string/split "banana" #"an")
⇒ ["b" "" "a"]
(clojure.string/split "banana" #"na")
⇒ ["ba"]
(clojure.string/split "nanabanana" #"na")
⇒ ["" "" "ba"]

In the case of split-lines, processing a file line by line and rejoining results in truncation of trailing newlines in the file. In both cases, the behavior is surprising and cannot be inferred from the docstrings. A workaround for split is to pass a limit of -1.

Proposed: As current users may be relying on the current behavior, the attached merely updates the docstring to warn of this behavior and suggest use of -1 as a limit to workaround.

Patch: clj-1360-2.patch



 Comments   
Comment by Andy Fingerhut [ 18/Feb/14 10:51 AM ]

Probably documenting would be safer than changing the behavior at this point, given that some people may actually rely on the current behavior after testing, deploying, etc.

I don't currently have a suggestion for a modified doc string, but note that there are examples of this behavior and how one can use an extra "-1" limit argument at the end to get all split strings: http://clojuredocs.org/clojure_core/clojure.string/split

Comment by Crispin Wellington [ 21/May/15 10:46 PM ]

This bug just bit me. +1 to be fixed. If we just document and leave the behavior as is, then we have a surprising and inconsistent behaving split (why are inner empty values kept, but outer ones stripped?) that is different to every other split you've ever used. The optional -1 limit argument looks hacky but a fix could keep this -1 argument working.

EDIT: this looks to be java's string class behavior: http://stackoverflow.com/questions/2170557/split-method-of-string-class-does-not-include-trailing-empty-strings
Would be nice if limit defaulted to -1 on that type of clojure.string/split call.

Comment by Stuart Halloway [ 19/Jul/15 8:03 AM ]

This is really gross, and the original developer has been punched in the neck. (Ow.)

I hate the Java leakage, but given that this is already out there, and that people are likely already relying on both the default and the negative-arg behavior, I think the least bad bet is to document precisely the semantics we have.

Comment by Alex Miller [ 29/Mar/18 3:45 PM ]

Incorporate changes from CLJ-1857 in clj-1360-2.patch.





[CLJ-2271] "caller" information missing in explain-data during macro instrumentation failure Created: 19/Nov/17  Updated: 27/Mar/18

Status: Open
Project: Clojure
Component/s: None
Affects Version/s: Release 1.10
Fix Version/s: None

Type: Defect Priority: Major
Reporter: Ben Brinckerhoff Assignee: Unassigned
Resolution: Unresolved Votes: 7
Labels: error-reporting, spec
Environment:

org.clojure/spec.alpha "0.1.143"


Approval: Triaged

 Description   

When there is a instrumentation failure for a function, the explain-data includes "caller" information. However, this information is missing if the instrumentation failure is for a macro.

This comment has led me to believe that the intended behavior is for explain-data to contain this info, so third-party error printers can display it.

In the repro below, I'm setting up a custom printer just to capture the raw explain-data (it's not a useful printer, just a means to show what is happenening)

Repro:

(require '[clojure.spec.alpha :as s])
  (require '[clojure.spec.test.alpha :as st])
  (require '[clojure.specs.alpha :as s])


  (s/fdef my-fn
          :args (s/cat :x int?))
  (defn my-fn [x]
    x)

  (s/fdef my-macro
          :args (s/cat :x int?))
  (defmacro my-macro [x]
    x)

  (st/instrument)
  (def !ed (atom nil))
  (set! s/*explain-out* (fn [ed]
                          (reset! !ed ed)))
  (my-fn "")
  @!ed
  ;; {:clojure.spec.alpha/problems [{:path [:args :x], :pred clojure.core/int?, :val "", :via [], :in [0]}], :clojure.spec.alpha/spec #object[clojure.spec.alpha$regex_spec_impl$reify__2436 0x72029b0e "clojure.spec.alpha$regex_spec_impl$reify__2436@72029b0e"], :clojure.spec.alpha/value (""), :clojure.spec.alpha/args (""), :clojure.spec.alpha/failure :instrument, :clojure.spec.test.alpha/caller {:file "form-init8333540581183382896.clj", :line 548, :var-scope expound.alpha/eval27394}}

  ;; ^--- Note there is an entry for :clojure.spec.test.alpha/caller

  (my-macro "")
  @!ed

  ;; #:clojure.spec.alpha{:problems [{:path [:args :x], :pred clojure.core/int?, :val "", :via [], :in [0]}], :spec #object[clojure.spec.alpha$regex_spec_impl$reify__2436 0x479a6a73 "clojure.spec.alpha$regex_spec_impl$reify__2436@479a6a73"], :value (""), :args ("")}

  ;; ^--- No caller information


 Comments   
Comment by Alex Miller [ 27/Nov/17 8:39 AM ]

You can't instrument a macro, so that part of the ticket doesn't make sense as written. But I expect you mean the spec check during macro expansion.

In the macro check case, the caller info is known by the compiler and included in the wrapper CompilerException. I suppose that info could be passed into s/macroexpand-check from the Compiler and possibly produce similar results as with instrumented function calls.

Comment by Ben Brinckerhoff [ 19/Mar/18 6:39 PM ]

Ah, you're correct, thanks for clarifying.

If caller information was added, it would also be very useful to add a specific `:clojure.spec.alpha/failure` value for specs that fail during macro expansion. That would allow 3rd party tools to show specific types of error messages when macro expansion failed.

Comment by Ben Brinckerhoff [ 19/Mar/18 6:41 PM ]

Additionally, having access to the symbol for the macro name would assist in error reporting.

Comment by Shogo Ohta [ 27/Mar/18 9:54 PM ]

IMHO explain-data for instrumentation failures and macro spec errors should contain the fspec of the function (or macro), as I suggested before in CLJ-2218.

An fspec has some useful info (such as the ns-qualified name symbol and the line number etc.) of the spec'ed function in its metadata, so spec error reporters can use them for richer and more precise error messages in a uniform way for both instrumentation failures and macro spec errors.





[CLJ-1665] take-nth transducer could be faster without rem Created: 20/Feb/15  Updated: 27/Mar/18

Status: Open
Project: Clojure
Component/s: None
Affects Version/s: Release 1.7
Fix Version/s: None

Type: Enhancement Priority: Major
Reporter: Steve Miner Assignee: Unassigned
Resolution: Unresolved Votes: 2
Labels: performance
Environment:

Mac OS X 10.10.2, JDK 1.8.0_31


Attachments: Text File CLJ-1665-faster-take-nth-transducer-without-rem.patch    
Patch: Code
Approval: Triaged

 Description   

The take-nth transducer is calling rem on each index, which is relatively expensive compared to a zero? test. It could just count down from N instead as the step size is fixed.



 Comments   
Comment by Steve Miner [ 20/Feb/15 12:34 PM ]

Patch attached. It's about 25% faster on a simple test like:

(time (transduce (take-nth 13) + (range 1e7)))
Comment by Steve Miner [ 20/Feb/15 12:41 PM ]

I didn't worry about (take-nth 0) case, but my patch does give a different result. The current implementation gets a divide by zero error (from rem). My patched version returns just the first element once. The regular collection version returns an infinite sequence of the first element. I doubt anyone expects a sensible answer from the 0 case so I didn't try to do anything special with it.

Comment by Michael Blume [ 20/Feb/15 12:55 PM ]

Nice =)

I would say that the transducer version really ought to match the collection version as closely as possible, but I don't think there's actually a way to write a transducer that transforms a finite sequence into an infinite sequence, so no luck there.

Maybe while we're at it we should change both the transducer and the collection arities to throw on zero?

Comment by Renzo Borgatti [ 27/Mar/18 11:07 AM ]

GIGO case, but rem is also responsible for:

user=> (take-nth 2.5 (range 10))
(0 3 6 9)
user=> (sequence (take-nth 2.5) (range 10))
(0 5)

The patch by Steve (CLJ-1665-faster-take-nth-transducer-without-rem.patch) is just missing a cast to int to solve the above:

(defn take-nth [n]
  (fn [rf]
    (let [n (int n)
          iv (volatile! 1)]
      (fn
        ([] (rf))
        ([result] (rf result))
        ([result input]
         (let [i (vswap! iv dec)]
           (if (zero? i)
             (do (vreset! iv n)
                 (rf result input))
             result)))))))




[CLJ-2339] map-entry constructor function Created: 26/Mar/18  Updated: 27/Mar/18

Status: Open
Project: Clojure
Component/s: None
Affects Version/s: None
Fix Version/s: None

Type: Enhancement Priority: Minor
Reporter: jcr Assignee: Unassigned
Resolution: Unresolved Votes: 1
Labels: None

Approval: Triaged

 Description   

Since 1.8, we have map-entry? predicate, but no corresponding constructor function. Implementation could be as simple as:

(defn map-entry
  "Creates a new map entry with key k and value v. 

  See also: map-entry?, key, val"
  [k v]
  (clojure.lang.MapEntry/create k v))


 Comments   
Comment by Mike Fikes [ 26/Mar/18 10:14 AM ]

Perhaps the existence of other kinds of map entries would matter to users of such a constructor.

user=> (let [me (first (sorted-map :a 1))]
         [(map-entry? me) (type me)])
[true clojure.lang.PersistentTreeMap$BlackVal]
Comment by Ghadi Shayban [ 26/Mar/18 10:35 AM ]

I don't see the point. What would having a map entry constructor achieve?

Comment by Alex Miller [ 26/Mar/18 5:13 PM ]

Mike: no, I don't think that should matter

Ghadi: there is code out in the wild that invokes MapEntry/create because it's faster in some cases - this would certainly be preferable.

Comment by jcr [ 27/Mar/18 10:45 AM ]

Ghadi: another consideration is clj\cljs portability. Speaking of which...

Alex: should I create a ticket for clojurescript too? Seems like the body of the function would be:

(cljs.core/MapEntry. k v nil)




[CLJ-2340] Add inline arities to clojure.core/not= Created: 26/Mar/18  Updated: 26/Mar/18

Status: Open
Project: Clojure
Component/s: None
Affects Version/s: Release 1.9
Fix Version/s: None

Type: Enhancement Priority: Minor
Reporter: John Schmidt Assignee: Unassigned
Resolution: Unresolved Votes: 1
Labels: performance

Attachments: Text File 0001-add-inline-arities.patch    
Patch: Code
Approval: Triaged

 Description   

The docstring for clojure.core/not= states "Same as (not (= obj1 obj2))". But since clojure.core/= has inline arities the emitted code can differ:

user=> (defn v1 [] (not= (bit-and 0 0) 0))
#'user/v1
user=> (println (disassemble v1))
...
  public static java.lang.Object invokeStatic();
     0  getstatic indy_type_hint.core$v1.const__0 : clojure.lang.Var [15]
     3  invokevirtual clojure.lang.Var.getRawRoot() : java.lang.Object [20]
     6  checkcast clojure.lang.IFn [22]
     9  lconst_0
    10  lconst_0
    11  invokestatic clojure.lang.Numbers.and(long, long) : long [28]       <<<<<<< doesn't use land intrinsic
    14  invokestatic clojure.lang.Numbers.num(long) : java.lang.Number [32]
    17  getstatic indy_type_hint.core$v1.const__2 : java.lang.Object [36]
    20  invokeinterface clojure.lang.IFn.invoke(java.lang.Object, java.lang.Object) : java.lang.Object [40] [nargs: 3]
    25  areturn
...

user=> (defn v2 [] (not (= (bit-and 0 0) 0)))
#'user/v1
user=> (println (disassemble v2))
...
  public static java.lang.Object invokeStatic();
     0  getstatic indy_type_hint.core$v2.const__0 : clojure.lang.Var [15]
     3  invokevirtual clojure.lang.Var.getRawRoot() : java.lang.Object [20]
     6  checkcast clojure.lang.IFn [22]
     9  lconst_0
    10  lconst_0
    11  land               <<<<<<< uses land intrinsic
    12  lconst_0
    13  invokestatic clojure.lang.Util.equiv(long, long) : boolean [28]
    16  ifeq 25
    19  getstatic java.lang.Boolean.TRUE : java.lang.Boolean [34]
    22  goto 28
    25  getstatic java.lang.Boolean.FALSE : java.lang.Boolean [37]
    28  invokeinterface clojure.lang.IFn.invoke(java.lang.Object) : java.lang.Object [41] [nargs: 2]
    33  areturn
...

The patch adds the same inline arities to clojure.core/not= as clojure.core/=. With the patch applied the compiler emits the v2 bytecode for both v1 and v2.



 Comments   
Comment by John Schmidt [ 26/Mar/18 3:10 PM ]

This is the first time I've submitted a patch, please let me know if I did something incorrectly. Also, I haven't signed the CLA yet, but I will be able to do so in the next few weeks.





[CLJ-2338] clojure.core.reducers/reducer is not aligned with transducer behavior Created: 26/Mar/18  Updated: 26/Mar/18

Status: Open
Project: Clojure
Component/s: None
Affects Version/s: Release 1.5, Release 1.6, Release 1.7, Release 1.8, Release 1.9
Fix Version/s: None

Type: Defect Priority: Major
Reporter: shulang Assignee: Unassigned
Resolution: Unresolved Votes: 0
Labels: reducers

Attachments: Text File reducer.patch    
Patch: Code
Approval: Triaged

 Description   

These two suppose to be the same:

(transduce (comp (take 10) (partition-all 3)) conj (range))
;; => [[0 1 2] [3 4 5] [6 7 8] [9]]

(reduce conj (r/reducer (range) (comp (take 10) (partition-all 3))))
;; => [[0 1 2] [3 4 5] [6 7 8]]

Reason being r/reducer is not currently respecting the 1-arity case of xf.



 Comments   
Comment by David Bürgin [ 26/Mar/18 2:54 AM ]

It’s the same with plain reduce:

(reduce ((partition-all 3) conj) [] (range 10))
; => [[0 1 2] [3 4 5] [6 7 8]]

I believe completion is a feature of transduce, not necessarily of other transducible processes.





[CLJ-2337] Clojure master tests fails with JDK 10 early access builds Created: 20/Mar/18  Updated: 20/Mar/18  Resolved: 20/Mar/18

Status: Closed
Project: Clojure
Component/s: None
Affects Version/s: Release 1.10
Fix Version/s: None

Type: Defect Priority: Major
Reporter: Karthikeyan Singaravelan Assignee: Unassigned
Resolution: Duplicate Votes: 0
Labels: jdk-10
Environment:

Operating system details :

$ uname -a
Linux ubuntu-s-1vcpu-1gb-blr1-01 4.4.0-116-generic #140-Ubuntu SMP Mon Feb 12 21:23:04 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux

Java version :

$ java -version
java version "10" 2018-03-20
Java(TM) SE Runtime Environment 18.3 (build 10+46)
Java HotSpot(TM) 64-Bit Server VM 18.3 (build 10+46, mixed mode)


Attachments: Text File clojure-9-jdk-10.log     Text File clojure-jdk-10.log    

 Description   

I have tested Clojure master with the latest JDK 10 early access build. I could find some failures. I have attached the output of `mvn package` as an attachment.

Steps to install JDK 10 :

$ wget https://raw.githubusercontent.com/sormuras/bach/master/install-jdk.sh
$ chmod +x install-jdk.sh
$ ./install-jdk.sh -F 10 -L BCL
$ export JAVA_HOME=/home/ubuntu/jdk-10/ # current jdk 10 path
$ export PATH=$JAVA_HOME/bin:$PATH
$ java -version
java version "10" 2018-03-20
Java(TM) SE Runtime Environment 18.3 (build 10+46)
Java HotSpot(TM) 64-Bit Server VM 18.3 (build 10+46, mixed mode)

Steps to reproduce :

$ git clone https://github.com/clojure/clojure
$ cd clojure
$ mvn package

Feel free to close this if this is too early to test and file this bug. I have tried testing since JDK 10 since it has reached RC phase.
JDK 10 release notes : http://jdk.java.net/10/release-notes



 Comments   
Comment by Karthikeyan Singaravelan [ 20/Mar/18 2:16 AM ]

I just tried `mvn package` with Clojure 1.9 release commit and could see the same test case failing. Attached a log of the same with the ticket.

Commit hash : 841fa60b41bc74367fb16ec65d025ea5bde7a617

Comment by Nicola Mometto [ 20/Mar/18 4:35 AM ]

dupe of CLJ-2330

Comment by Karthikeyan Singaravelan [ 20/Mar/18 4:58 AM ]

This can be closed. I tried "JDK 10" in JIRA search and reported this since I couldn't find any related issues. Sorry for the noise.





[CLJ-2284] Incorrect bytecode generated for static methods on interfaces Created: 09/Dec/17  Updated: 19/Mar/18

Status: Open
Project: Clojure
Component/s: None
Affects Version/s: Release 1.9
Fix Version/s: None

Type: Defect Priority: Major
Reporter: Zach Tellman Assignee: Unassigned
Resolution: Unresolved Votes: 9
Labels: asm, compiler, java19
Environment:

JDK 9 or higher


Attachments: Text File 0001-unbundle-ASM.patch     Text File 0002-update-classfile-version-to-9.patch     Text File 0003-CLJ-2284.patch     Text File v2-0001-vendor-asm-to-sha-648c512e38e6cf58c09e22b324a9c99.patch     Text File v2-0002-bump-javac-output-to-1.8-classfiles.patch     Text File v2-0003-Compiler-genclass-emit-jdk8-classfiles.patch     Text File v2-0004-CLJ-2284-enable-calls-to-static-interface-methods.patch     Text File v2-0005-CLJ-2284-test-cases.patch    
Approval: Triaged

 Description   

Calls to static interface methods compile on Java 8 but not on Java 9 because of a new bytecode restriction. As the usage of static interface methods increases, this problem will manifest more. I ran into it while using the AWS SDK (2.0 preview) which makes extensive usage of these methods.

public interface JDK8InterfaceMethods {
    // cannot call either of these from Clojure on Java 9
    // throws IncompatibleClassChangeError per JVMS
    public static long staticMethod0(long v) { return v; }
    public static String staticMethod1(String s) { return s; }
}

JVMS Section 5.4.3.3 and 5.4.3.4 describe method resolution
https://docs.oracle.com/javase/specs/jvms/se9/html/jvms-5.html#jvms-5.4.3.3

The v2 patches have a full description in each commit. The basic approach is to link the method description correctly (a one-line change) but in order to do that we must emit JDK8 bytecode, not the current JDK5 bytecode. If this approach is decent I recommend getting in the change early in order to enjoy extended testing.

The original ticket had a repro here: https://github.com/ztellman/java9-failure



 Comments   
Comment by Ghadi Shayban [ 09/Dec/17 11:08 PM ]

Static or default interface method invocations are not supported when the bytecode emitted is < 1.8 bytecode (Clojure currently emits 1.6 bytecode). I'm mildly surprised this compiled.

Comment by David Bürgin [ 21/Dec/17 12:54 PM ]

Not sure Ghadi’s assertion is correct? No new bytecodes were introduced for static and default interface methods. Or at least I have been using JDK 8 utilities like CharSequence.codePoints() and Comparator.reverseOrder() in Clojure for a long time. (And if this usage were not supported today that would seem like a critical issue.)

Comment by Zach Tellman [ 22/Dec/17 9:08 PM ]

Yeah, I'd expect default interface methods to just be filled in on classes which don't provide their own implementation, and empirically they don't seem to cause any issues, but I'm not an authority on JVM bytecode.

Comment by Nicola Mometto [ 23/Dec/17 2:10 PM ]

while it is true that no bytecodes were introduced, the JVM spec mandates that invocations to static interface methods refer to an InterfaceMethodref entry in the constant pool rather than simply to a Methodref – as to why this worked in jvm 1.8 and fails with a verify error in jvm 9 , I can only assume the bytecode verifier became stricter since version 9

See https://docs.oracle.com/javase/specs/jvms/se9/html/jvms-5.html#jvms-5.4.3.3, describing the resolution rules for Methodrefs:

When resolving a method reference:

    If C is an interface, method resolution throws an IncompatibleClassChangeError.
Comment by Nicola Mometto [ 23/Dec/17 2:21 PM ]

Indeed here's the openjdk ticket tightening bytecode verification to adhere to the JVM spec: https://bugs.openjdk.java.net/browse/JDK-8145148
According to the last comment on that ticket, upgrading the ASM version provided with clojure to 5.1 should fix this

Comment by Nicola Mometto [ 24/Dec/17 12:31 PM ]

The following 3 patches work to fix this problem, I've kept them split for now so that we can be evaluated separately, but in order to fix this all three are required.

1st patch simply updates the version of ASM that clojure uses to 6.0, removes the bundled ASM and adds ASM as a separate dependency. If this is not desirable we could just update the bundled version of ASM but it seems like now that clojure requires external deps, this is up for debate.

2nd patch updates the classfile version that clojure emits from 1_5 to 9 so that ASM can emit interface method refs, because of changes in bytecode verification since v1.5, several additional changes were needed:
1- it's not possible anymore to assign final fields from methods other than <clinit>, so it was necessary to mark the const__ fields as non final as they are assigned by init__n methods
2- stack map frames are not optional anymore so we need to tell ASM to compute them for us
3- a custom ClassWriter overriding getCommonSuperClass has been implemented, using DCL to load classes and short circuiting on classes that are being defined (as per that method's javadoc)

finally the third patch addresses this issue by emitting `invokeStatic` through the new `visitmethodInsn` arity that supports interface methods

Comment by Nicola Mometto [ 24/Dec/17 12:41 PM ]

Note that I'm not proposing the 3 patches as they are as a fix for this, given that bumping classfile version means discontinuing support for running clojure on java versions earlier than what's specified, which is undesiderable (I've bumped it to V9 but what's needed here should be just V1_8, but still).

That said, if we want to fix this, we definitely need the fixes I've put in those patches and we likely also want to conditionally emit different classfile versions (e.g. defaulting to 1_5 but bumping it to a higher version for specific classfiles when needed)

Comment by Vladimir Tsanev [ 06/Feb/18 3:41 AM ]

Nicola Mometto are sure you sure upgrade the V1_8 is needed to address interface static and/or default methods? CONSTANT_InterfaceMethodref is not new it was in the instruction set since forever (and in V1_6 particularly https://docs.oracle.com/javase/specs/jvms/se6/html/ClassFile.doc.html#42041).

Comment by Nicola Mometto [ 06/Feb/18 9:23 AM ]

Vladimir Tsanev InterfaceMethodref is not new, but supporting `invokeStatic` on an InterfaceMethodref is only supported since V1_8

Comment by Ghadi Shayban [ 17/Feb/18 3:57 PM ]

Attached v2- patches. Please read patch-3 commit message carefully.

Comment by Ghadi Shayban [ 17/Feb/18 4:04 PM ]

I consider this a very high priority defect when running Clojure on Java 9. Some libraries (notably the new AWS SDK) are beginning to make use of static methods on interfaces.

The v2- patches enable JDK8 bytecode, which allows Clojure's bytecode to correctly call these methods without throwing an exception. This sheds compatibility for JDK 7. Android should no longer have problems with JDK8 bytecode.

ASM is now hosted on git, and the first patch's commit message contains a useful vendoring script.

As a side-effect loading Clojure is consistently about 5% faster across a few machines I tested (x86 & ARM). I did not test compilation speed.

Comment by Vladimir Tsanev [ 13/Mar/18 4:56 AM ]

asm 6.1 is released which does not have any new features (except java 10 bytecode version). It should be binary compatible, but have few runtime changes - probably worth checking if everything is ok against the new version. They also changed the code style - another good reason to unbundle asm since merging will become harder.

Comment by Ghadi Shayban [ 19/Mar/18 10:15 AM ]

The patch in v2-0001 is already based on the ASM-6.1 branch but not the final release. Can bump it with the script in the commit comment.

Comment by Ghadi Shayban [ 19/Mar/18 11:23 AM ]

Workaround macro:

(defmacro interface-static-call
  [sym & argtypes]
  `(let [m# (.getMethod ~(symbol (namespace sym))
                        ~(name sym)
                        (into-array Class ~argtypes))]
     (fn [& args#]
       (.invoke m# nil (to-array args#)))))

(def client
  (interface-static-call S3Client/create))




[CLJ-1594] Colons followed by spaces are not ignored within (comment ...) Created: 19/Nov/14  Updated: 19/Mar/18  Resolved: 19/Mar/18

Status: Closed
Project: Clojure
Component/s: None
Affects Version/s: Release 1.6
Fix Version/s: None

Type: Defect Priority: Major
Reporter: Ed Ward Assignee: Unassigned
Resolution: Declined Votes: 0
Labels: bug, comment, function
Environment:

Ubuntu Linux 14.10
Clojure 1.6.0
OpenJDK 64-Bit Server VM 1.7.0_65-b32



 Description   

Running

(comment abc:def)
works, while running
(comment abc: def)
and
(comment abc : def)
fail with the exception

RuntimeException Invalid token: :  clojure.lang.Util.runtimeException (Util.java:221)
CompilerException java.lang.RuntimeException: Unable to resolve symbol: def in this context, compiling:(/tmp/form-init1585368677683647130.clj:1:1010) 
RuntimeException Unmatched delimiter: )  clojure.lang.Util.runtimeException (Util.java:221)


 Comments   
Comment by Nicola Mometto [ 19/Nov/14 11:19 AM ]

`comment` is a macro, it doesn't bypess the reader.
":" is not a regular clojure token so it's expected that this throws.
The only way to include non clojure tokens in a source code is after a ";" comment

Comment by Andy Fingerhut [ 19/Nov/14 3:37 PM ]

I've just added some notes about this to ClojureDocs.org here: http://clojuredocs.org/clojure.core/comment

Comment by Alex Miller [ 22/Nov/14 9:12 AM ]

agreed with Nicola's comment above





[CLJ-2265] deftype generates new types when evaluation is expected to be suppressed Created: 13/Nov/17  Updated: 19/Mar/18  Resolved: 19/Mar/18

Status: Closed
Project: Clojure
Component/s: None
Affects Version/s: Release 1.8
Fix Version/s: None

Type: Defect Priority: Minor
Reporter: Ramsey Nasser Assignee: Unassigned
Resolution: Completed Votes: 0
Labels: bug, deftype
Environment:

macOs 10.12.6



 Description   

There does not seem to be a consistent way to prevent deftype from emitting a new type. This was discovered in ClojureCLR when trying to write a defonce-style macro wrapper for deftype, but it affects ClojureJVM as well. It seems to emit types on analysis, which defies Clojure's evaluation semantics.

The following REPL session highlights expected and unexpected behavior, namely that when false is unable to prevent deftype from emitting a new type into memory. comment seems to work, however.

nREPL server started on port 52921 on host 127.0.0.1 - nrepl://127.0.0.1:52921
REPL-y 0.3.7, nREPL 0.2.12
Clojure 1.8.0
Java HotSpot(TM) 64-Bit Server VM 1.8.0_60-b27
    Docs: (doc function-name-here)
          (find-doc "part-of-name-here")
  Source: (source function-name-here)
 Javadoc: (javadoc java-object-or-class-here)
    Exit: Control+D or (exit) or (quit)
 Results: Stored in vars *1, *2, *3, an exception in *e

user=> (deftype TypeTest [a b])
user.TypeTest
user=> (->> user.TypeTest .getDeclaredFields (map #(.getName %)))
("a" "b")
user=> (deftype TypeTest [a b c])
user.TypeTest
user=> (->> user.TypeTest .getDeclaredFields (map #(.getName %)))
("a" "b" "c")  ;; <= expected redefinition 
user=> (when false (deftype TypeTest [a]))
nil
user=> (->> user.TypeTest .getDeclaredFields (map #(.getName %)))
("a") ;; <= when false fails to prevent redefinition
user=> (comment (deftype TypeTest [a b c]))
nil
user=> (->> user.TypeTest .getDeclaredFields (map #(.getName %)))
("a")  ;; <= comment manages to prevent redefinition
user=>


 Comments   
Comment by Kevin Downey [ 13/Nov/17 1:26 PM ]

not on analysis, on compilation.

deftype has a compilation time effect. similarly def has a compilation time effect.

so

(when false (def a 1))

will result in the var for a being interned, but it won't have a root value.

the reason wrapping in (comment ...) stops whatever from happening is compilation happens after macroexpansions, and the comment macro replaces whatever form with nil.

I am not sure this would be considered a bug. The only way to avoid this would be to have deftype generate types at runtime instead of at compile time, but I am pretty sure the types are generated at compile time to 1. avoid generating a new type every time the compiled expression is run 2. to make class generation fairly predictable so clojure can be used in environments with unusual classloader restrictions.

Comment by Kevin Downey [ 13/Nov/17 1:29 PM ]

maybe take a look at compile-if https://github.com/clojure/clojure/blob/master/src/clj/clojure/core/reducers.clj#L24-L35

Comment by Alex Miller [ 13/Nov/17 5:23 PM ]

As Kevin commented, this is the intended behavior.

Comment by Ramsey Nasser [ 14/Nov/17 10:41 AM ]

Noted. compile-if is what we need. Apologies for the noise.





[CLJ-1406] Libs are blindly added into loaded-libs even if an error occurs during loading Created: 17/Apr/14  Updated: 18/Mar/18

Status: Open
Project: Clojure
Component/s: None
Affects Version/s: Release 1.5, Release 1.6
Fix Version/s: Release 1.10

Type: Defect Priority: Major
Reporter: Shogo Ohta Assignee: Unassigned
Resolution: Unresolved Votes: 0
Labels: None

Attachments: Text File 0001-modify-clojure.core-load-lib-so-that-it-removes-the-.patch     Text File CLJ-1406.patch    
Patch: Code
Approval: Incomplete

 Description   

Suppose you have a lib that causes some errors during loading, like the following:

(ns broken-lib)

(} ; this line will cause a reader error

And then, if you require the lib, it would be added into loaded-libs in spite of the reader error, which makes require succeed silently after that.

user=> (contains? (loaded-libs) 'broken-lib)
false
user=> (require 'broken-lib)
CompilerException java.lang.RuntimeException: Unmatched delimiter: }, compiling:(broken_lib.clj:3:3) 
user=> (contains? (loaded-libs) 'broken-lib)
true
user=> (require 'broken-lib)
nil
user=>

Things get worse if you have another namespace that requires a broken lib (say here broken-lib.core):

(ns broken-lib.core
  (:require [broken-lib :as lib]))

Although you'll get the actual error the first time you load the depending namespace, after that you'll find a wrong compiler exception thrown every time you try to reload it. The situation will last even after you actually do fix the code causing the original error.

user=> (require 'broken-lib.core)

CompilerException java.lang.RuntimeException: Unmatched delimiter: }, compiling:(broken_lib.clj:3:3) 

user=> (require 'broken-lib.core :reload)
CompilerException java.lang.Exception: namespace 'broken-lib' not found, compiling:(broken_lib/core.clj:1:1) 

user=> (require 'broken-lib.core :reload) ;; reload after fix the bug in broken-lib

CompilerException java.lang.Exception: namespace 'broken-lib' not found, compiling:(broken_lib/core.clj:1:1) 
user=>

Cause:
The patch for CLJ-1116 made the ns macro blindly add the lib being defined into loaded-libs even if an error occurs during loading.

Approach:
Modify clojure.core/load-lib so that it removes the lib from loaded-libs on error.



 Comments   
Comment by Alex Miller [ 17/Apr/14 9:07 AM ]

This patch seems somewhat removed from the cause - is there some way to instead prevent the lib from being added to loaded-libs in the first place?

Comment by Shogo Ohta [ 17/Apr/14 9:21 AM ]

To do so, I think we need to revert CLJ-1116.

Comment by Alex Miller [ 22/Sep/16 1:41 PM ]

I don't think this solution is good as is so moving to Incomplete.

Comment by Shogo Ohta [ 15/Oct/16 1:34 AM ]

What kind of solution are you expecting for this problem?

To prevent the lib from being added to loaded-libs in the first place, I think ns macro needs to know where it is used from. It should immediately add the ns when used in the REPL for CLJ-1116, while it should defer adding the ns until completing loading whole the file without errors when used within a file.

Comment by Ghadi Shayban [ 18/Mar/18 2:05 PM ]

Attached a similar approach as Shogo Ohta's original. Given the way ns now works, I see this approach as tenable.

To support dynamic ns creation at the REPL, CLJ-1116 wrote to loaded-libs inside the ns macro, and at the same time made some logic in load-lib and load-libs unnecessary. This set of patches undoes the side-effect when loading fails, and cleans up some dead conditionals.

New behavior: Given ns A which requires #{B, C}, if B loads but C doesn't, only B will be written to loaded-libs, and A and C will be undone. This improves the experience with load and fixes all of the annoying behavior in the ticket description.

NB: the 'require' patch entails:
If a lib is specified in an ns :require or :use, loading that lib the first time must result in an ns that corresponds exactly to the lib name. (There is not and has never been this restriction when calling `load` directly, but only via `ns`) I was surprised to find in the test suite an accidental lib with underscores where the loaded code results in a namespace with a dash. (Perhaps c.c.specs should specify a tighter spec for libs than 'simple-symbol?')

The commits are separated, with accompanying short explanatory comments.





[CLJ-1852] Clojure-generated class names length exceed file-system limit Created: 20/Nov/15  Updated: 17/Mar/18

Status: Open
Project: Clojure
Component/s: None
Affects Version/s: Release 1.7, Release 1.8
Fix Version/s: None

Type: Defect Priority: Major
Reporter: Martin Raison Assignee: Unassigned
Resolution: Unresolved Votes: 11
Labels: compiler
Environment:

tested on CentOS 6


Attachments: Text File 0001-CLJ-1852-Create-jar-if-class-name-length-exceed-MAX_.patch    
Approval: Triaged

 Description   

Class names generated by the Clojure compiler can be arbitrarily long, exceeding the file system's maximum allowed file name length. For example it happens when you nest functions a bit too deeply:

(defmacro nestfn [n & body]
  (if (> n 0)
    `(fn [] (nestfn ~(- n 1) ~@body))
    body))

(def myf (nestfn 100 "body"))

Compiling this produces a java.io.IOException: File name too long exception.



 Comments   
Comment by Martin Raison [ 20/Nov/15 9:32 PM ]

The Scala community found this issue a while ago, and now the compiler has a max-classfile-name parameter (defaulting to 255). Hashing is used when the limit is exceeded. Maybe we should consider something similar?

Comment by Philipp Neumann [ 16/Oct/17 10:53 AM ]

I tried clojure.core.match with 13 patterns and the compiliation failed under Windows. I assume this problem is the root cause of it.

Comment by Dr. Christian Betz [ 08/Mar/18 4:47 AM ]

Some more info on that:

A colleague of mine just ran into that problem because he's using Linux / eCryptfs (where the limit of 143 is rather small, compared to our FileVault encrypted macOS used otherwise): see https://bugs.launchpad.net/ecryptfs/+bug/344878.

However, Clojure's not alone with that problem, Scala is also hit hard: https://issues.scala-lang.org/browse/SI-3623

There's no "easy" solution to this, and truncating the filename (as Scala does) bears a lot of other problems, obviously.

For all of you bitten by this problem, one possible workaround might be the one proposed by Mario Pastorelli (https://issues.scala-lang.org/secure/ViewProfile.jspa?name=melrief) in comment https://issues.scala-lang.org/browse/SI-3623?focusedCommentId=76104&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-76104:

Use an unencrypted filesystem to temporarily store classfiles, maybe by having a tempFS to keep stuff in memory. Not the best way, because we do not like leaving important stuff unencrypted, ...

Comment by Alex Vong [ 16/Mar/18 4:55 PM ]

Hello,

I run into the same problem while using clojure.core.match. I macroexpand the function definition and observe that the macro-expanded definition is deeply nested.

I think one way to solve it is to provide an option to wrap the class file as a jar with some shorter file name so that the the class name can remain the same (zipped file name can be as long as you like, right?). WDYT?

Btw, I am using clojure 1.9.0 so I think we should say that this bug affects 1.9 as well.

Comment by Alex Vong [ 17/Mar/18 11:50 AM ]

I come up with a proof-of-concept patch. The compiler now outputs jar instead of class if the class name is longer than 255. The jar name is simply a (left) truncation of the class name.

Suppose *compile-path* is set to "build", then you need to add build/* to your class path, so that the jars can be found.

This patch is only a proof of concept, ideally all classes with long name should be put into one big jar to avoid having to decompress many files. Also, the user should be able to specify *compile-name-max* and *compile-jar-name*. Finally, the code is quite ugly, I should have spitted things into several functions.

Comment by Alex Miller [ 17/Mar/18 12:19 PM ]

Alex - we're not going to output jars. This is at odds with many facets of the Clojure runtime.

Comment by Steve Miner [ 17/Mar/18 4:49 PM ]

Sorry if this comment is off topic, but the original example is a bit confusing to me. Maybe it should be:

(defmacro nestfn [n & body]
  (if (> n 0)
    `(fn [] (nestfn ~(dec n) ~@body))
    `(do ~@body)))

That way (trampoline (nestfn 10 "foo")) would return "foo". However, I do get a CompilerException java.lang.StackOverflowError for n=1000 on macOS.





[CLJ-2165] #clojure/var tag for transmitting var identity Created: 22/May/17  Updated: 17/Mar/18

Status: Open
Project: Clojure
Component/s: None
Affects Version/s: Release 1.9
Fix Version/s: Release 1.10

Type: Feature Priority: Major
Reporter: Alex Miller Assignee: Unassigned
Resolution: Unresolved Votes: 1
Labels: print, reader, var

Attachments: Text File vartag2.patch    
Approval: Vetted

 Description   

Currently one can't send vars around in edn. #' is clojure reader specific. Objective is to transmit var identity and bind to same-named var on reading side (a la var serialization support).

Proposed: This is not generic enough to add to edn, so use #clojure/var for tag. Printing may print #clojure/var instead of #' (perhaps via a flag) - needs more assessment. #clojure/var tag reader should be installed in data readers.

Patch: vartag2.patch



 Comments   
Comment by Christophe Grand [ 11/Jul/17 10:14 AM ]

Should unnamed vars (eg created by with-local-vars) print to #clojure/var nil or throw an exception? (exception is the print-dup behavior)

Comment by Steven Yi [ 08/Aug/17 11:49 AM ]

I think the vartag2.patch has an issue in that the test for print-var-tagged in missing an assertion. I think it is supposed to have something like:

(is (and ...))

within the last let-binding.

Comment by Ghadi Shayban [ 17/Mar/18 2:21 PM ]

Printing unnamed vars has little utility (nothing distinguishes them from each other without knowing their code context), seems like it would be fine to have #clojure/var nil or #clojure/var :unnamed





[CLJ-2336] Args to defn can be invalid, but produce nil explain-data Created: 16/Mar/18  Updated: 17/Mar/18

Status: Open
Project: Clojure
Component/s: None
Affects Version/s: Release 1.9
Fix Version/s: None

Type: Defect Priority: Major
Reporter: Ben Brinckerhoff Assignee: Unassigned
Resolution: Unresolved Votes: 0
Labels: spec

Approval: Triaged

 Description   
~  clj
Clojure 1.9.0
user=> (require '[clojure.spec.alpha :as s])
nil
user=> (require '[clojure.core.specs.alpha])
nil
(let [spec (:args (s/get-spec `defn))
        form `(~'foo ([~'a] 1) [~'a ~'b])]
    {:valid (s/valid?
             spec
             form)
     :explain-data (s/explain-data
                    spec
                    form)})
{:valid false, :explain-data nil}
user=>

Expected: If form is invalid, explain-data should not be nil






[CLJ-2335] clojure.lang.PersistentArrayMap keys function doesn't preserve order when number of pairs exceeds 8 Created: 12/Mar/18  Updated: 12/Mar/18  Resolved: 12/Mar/18

Status: Closed
Project: Clojure
Component/s: None
Affects Version/s: Release 1.8
Fix Version/s: None

Type: Defect Priority: Major
Reporter: Peter Ullah Assignee: Unassigned
Resolution: Declined Votes: 0
Labels: core
Environment:

OSX 10.12.6
https://repo1.maven.org/maven2/org/clojure/clojure/1.8.0/clojure-1.8.0.jar



 Description   

java -cp ~/Downloads/clojure-1.8.0.jar clojure.main

```
user=> (def one-to-eight {
:one "One"
:two "Two"
:three "Three"
:four "Four"
:five "Five"
:six "Six"
:seven "Seven"
:eight "Eight"})

user=> (def one-to-nine {
:one "One"
:two "Two"
:three "Three"
:four "Four"
:five "Five"
:six "Six"
:seven "Seven"
:eight "Eight"
:nine "Nine"})

user=> (keys one-to-eight)
(:one :two :three :four :five :six :seven :eight)

user=> (keys one-to-nine)
(:one :eight :three :five :four :nine :two :seven :six)

```



 Comments   
Comment by Peter Ullah [ 12/Mar/18 12:43 PM ]

Incidentally, first notice using using 1.10.0-alpha4. 1.8 was the first .jar I came to that didn't cause a spec related exception when attempting to run java -cp etc.

Comment by Alex Miller [ 12/Mar/18 1:10 PM ]

Clojure literal maps make no guarantees on entry order. Which map implementation is used is undefined and you should have no expectation about which one that is. (For maps >8 entries, you're seeing hash map used here instead as array maps are only efficient for small maps.) So, this is the expected behavior. If you really need to retain insertion order, you can do that by using the explicit `array-map` constructor.

user=> (def one-to-nine (array-map :one "One" :two "Two" :three "Three" :four "Four" :five "Five" :six "Six" :seven "Seven" :eight "Eight" :nine "Nine"))
#'user/one-to-nine
user=> (keys one-to-nine)
(:one :two :three :four :five :six :seven :eight :nine)

Since Clojure 1.9, Clojure has a dependency on two external jars and starting it with just -cp and the clojure jar will not work. You can find instructions on how to build an all-included local jar at https://clojure.org/guides/getting_started#_other_ways_to_run_clojure.

Comment by Peter Ullah [ 12/Mar/18 1:48 PM ]

Thank you for the explanation, Alex.





[CLJ-2073] AOT compilation can result in spurious ClassCastException during compile Created: 02/Dec/16  Updated: 12/Mar/18

Status: Open
Project: Clojure
Component/s: None
Affects Version/s: Release 1.7, Release 1.8
Fix Version/s: None

Type: Defect Priority: Major
Reporter: Paul Mooser Assignee: Unassigned
Resolution: Unresolved Votes: 0
Labels: aot, compiler
Environment:

java version "1.8.0_112"
Java(TM) SE Runtime Environment (build 1.8.0_112-b16)
Java HotSpot(TM) 64-Bit Server VM (build 25.112-b16, mixed mode)


Attachments: File consumer.clj     File implementer.clj     File protocol.clj    
Approval: Triaged

 Description   

If you try to compile the attached files as follows (assuming they are in "src"):

java -Dclojure.compile.path=out -cp "./clojure-1.8.0.jar:out:src" clojure.lang.Compile implementer protocol consumer

an exception will be thrown:

Exception in thread "main" java.lang.ClassCastException: implementer.Obj cannot be cast to protocol.Dependent, compiling:(consumer.clj:5:1)
	at clojure.lang.Compiler$InvokeExpr.eval(Compiler.java:3657)
	at clojure.lang.Compiler.compile1(Compiler.java:7474)
	at clojure.lang.Compiler.compile(Compiler.java:7541)
	at clojure.lang.RT.compile(RT.java:406)
	at clojure.lang.RT.load(RT.java:451)
	at clojure.lang.RT.load(RT.java:419)
	at clojure.core$load$fn__5677.invoke(core.clj:5893)
	at clojure.core$load.invokeStatic(core.clj:5892)
	at clojure.core$load.doInvoke(core.clj:5876)
	at clojure.lang.RestFn.invoke(RestFn.java:408)
	at clojure.core$load_one.invokeStatic(core.clj:5697)
	at clojure.core$compile$fn__5682.invoke(core.clj:5903)
	at clojure.core$compile.invokeStatic(core.clj:5903)
	at clojure.core$compile.invoke(core.clj:5895)
	at clojure.lang.Var.invoke(Var.java:379)
	at clojure.lang.Compile.main(Compile.java:67)
Caused by: java.lang.ClassCastException: implementer.Obj cannot be cast to protocol.Dependent
	at protocol$fn__12$G__8__14.invoke(protocol.clj:3)
	at protocol$fn__12$G__7__17.invoke(protocol.clj:3)
	at protocol$expand_deps.invokeStatic(protocol.clj:8)
	at protocol$expand_deps.invoke(protocol.clj:6)
	at clojure.lang.AFn.applyToHelper(AFn.java:154)
	at clojure.lang.AFn.applyTo(AFn.java:144)
	at clojure.lang.Compiler$InvokeExpr.eval(Compiler.java:3652)
	... 15 more
  • This does not occur with 1.6 or earlier versions
  • This does not occur if you do not try to invoke AOT
  • This may not occur for some orderings of the arguments

This appears to be related to the class being loaded by two different class loaders, and also may result in the namespace being compiled more than once. This issue has popped up for us multiple times in our production build, but it took a while to realize it was a compiler issue and to find a minimal example.






[CLJ-2311] Spec generator override won't work on multi-spec dispatch key Created: 12/Jan/18  Updated: 12/Mar/18

Status: Open
Project: Clojure
Component/s: None
Affects Version/s: Release 1.9
Fix Version/s: None

Type: Defect Priority: Minor
Reporter: Andreas Liljeqvist Assignee: Unassigned
Resolution: Unresolved Votes: 0
Labels: generator, spec
Environment:

[org.clojure/clojure "1.9.0"]
[org.clojure/spec.alpha "0.1.143"]


Attachments: Text File CLJ-2311.patch     File multi_spec_bug.clj    
Patch: Code
Approval: Triaged

 Description   

Generator override doesn't work as expected on multi-specs.
Code below illustrates the problem.

(s/def ::obj-type #{:a :b})

(s/def ::base-obj (s/keys :req [::obj-type]))

(defmulti obj-type ::obj-type)
(defmethod obj-type :a [_]
  ::base-obj)
(defmethod obj-type :b [_]
  ::base-obj)

(s/def ::obj (s/multi-spec obj-type ::obj-type))
(gen/sample (s/gen ::obj {::obj-type #(gen/return :a)}))

In the example above the dispatch-fn ::obj-type for the multimethod is given a generator override.
It is expected to return only colls of {::obj-type :a}
Actually it will also return {::obj-type :b}.
That is a generator cannot be used to constrain the set of dispatch-keys to sample from.

Current method:

In the case of a multimethod a generator is constructed for every possible dispatch value.
One is then chosen randomly without paying any attention to overrides for the dispatch-fn(key).

Patched method:

Commit available here
Patched version constructs generators for dispatch values exactly as original.
After that a check is made to see if there exists an override for the dispatch-fn.
If so a gen/bind is done using the override generator.
The bind function generates a value from the override generator.
That value is then used to lookup and return the correct multimethod generator.

Test case



 Comments   
Comment by Andreas Liljeqvist [ 09/Mar/18 9:35 AM ]

Patch provided for issue.

Comment by Alex Miller [ 09/Mar/18 9:48 AM ]

Can you move the code for the problem into the description and explain the problem and the change?

Comment by Andreas Liljeqvist [ 09/Mar/18 10:17 AM ]

I can't figure out how to edit the description...

description:
In the case of a multimethod a generator is constructed for every possible dispatch value.
One is then choosen randomly without paying any attention to overrides for the dispatch-fn(key).

commit: https://github.com/bonega/spec.alpha/commit/9cb42478b52eac275d496ec29669e2bf4b3e8e1f
My patch hopefully fixes that by checking if there exists an override for the dispatch-fn.
If so a gen/bind is done to the override generator and the result is used to lookup the correct multimethod generator.

Test case: https://pastebin.com/62ZT5Zfc
The test is expected to pass.
Basically I want to generate ::obj with an overriding generator like `(gen/generate (s/gen ::obj {::obj-type #(gen/return :a)}))`.
Expected result should always be `{::obj-type :a}`.
Previously :b was possible as output even though an override was specified.

Comment by Alex Miller [ 09/Mar/18 2:01 PM ]

I've given you edit right for the ticket so you can make the updates...

Comment by Andreas Liljeqvist [ 12/Mar/18 7:35 AM ]

I have updated the description.
Let me know if anything is unclear.





[CLJ-2334] Equality not working for Double/NaN Created: 10/Mar/18  Updated: 10/Mar/18  Resolved: 10/Mar/18

Status: Closed
Project: Clojure
Component/s: None
Affects Version/s: Release 1.8, Release 1.9
Fix Version/s: None

Type: Defect Priority: Minor
Reporter: Bruno Bonacci Assignee: Unassigned
Resolution: Declined Votes: 0
Labels: None
Environment:

Clojure 1.9, 1.8, etc



 Description   

the equals `=` function returns `false` when comparing Double/NaN

(.equals Double/NaN Double/NaN) ;;=> true
(= Double/NaN Double/NaN)       ;;=> false

and even

;; this is ok
(/ 0.0 0.0) ;;=> NaN
(.equals Double/NaN (/ 0.0 0.0)) ;;=> true

;; this is wrong
(= Double/NaN (/ 0.0 0.0)) ;; false


 Comments   
Comment by Bruno Bonacci [ 10/Mar/18 9:47 AM ]

Obviously the consequence of the missed equality is that you can break many other contracts

(-> (hash-map)
    (assoc Double/NaN 1)
    (assoc Double/NaN 2)
    (assoc Double/NaN 3))
;;=> {NaN 1, NaN 2, NaN 3}

(set [Double/NaN Double/NaN Double/NaN])
;;=> #{NaN NaN NaN}
Comment by David Bürgin [ 10/Mar/18 3:29 PM ]

Bruno Bonacci Double.NaN is not equal itself in Java – what you’re seeing is a peculiarity of Double.equals, see https://docs.oracle.com/javase/9/docs/api/java/lang/Double.html#equals-java.lang.Object- Clojure seems to be doing the right thing here.

Comment by Alex Miller [ 10/Mar/18 3:57 PM ]

Yes, this is the expected behavior according to the IEEE spec.





[CLJ-2304] CLONE - spec passing nil as second argument to `ExceptionInfo` constructor... Created: 02/Jan/18  Updated: 09/Mar/18

Status: Open
Project: Clojure
Component/s: None
Affects Version/s: Release 1.9
Fix Version/s: None

Type: Defect Priority: Major
Reporter: Jonathan Leonard Assignee: Unassigned
Resolution: Unresolved Votes: 0
Labels: spec
Environment:

Mac OS X 10.13.2

jonathan$ java -version
java version "1.8.0_131"
Java(TM) SE Runtime Environment (build 1.8.0_131-b11)
Java HotSpot(TM) 64-Bit Server VM (build 25.131-b11, mixed mode)


Attachments: Text File better-messaging.patch    

 Description   

Exception thrown from this line:
https://github.com/clojure/clojure/blob/clojure-1.9.0/src/jvm/clojure/lang/ExceptionInfo.java#L31

Due to this call:
https://github.com/clojure/spec.alpha/blob/spec.alpha-0.1.143/src/main/clojure/clojure/spec/test/alpha.clj#L279

if `when-not` returns nil, `ExceptionInfo` throws exception on line 31.

A naive fix may be:
(apply ex-info (remove nil? ["Specification-based check failed" (when-not ...)]))

although that is really just masking over the issue and provides no actionable info to the sufferer of the failure.

Unfortunately I have no minimal test case as this is proprietary (and complicated code).

Perhaps the author from reading the code will have more insight.

Here's the full stack trace:

[[clojure.lang.ExceptionInfo <init> "ExceptionInfo.java" 31]
[clojure.lang.ExceptionInfo <init> "ExceptionInfo.java" 22]
[clojure.core$ex_info invokeStatic "core.clj" 4739]
[clojure.core$ex_info invoke "core.clj" 4739]
[clojure.spec.test.alpha$explain_check invokeStatic "alpha.clj" 277]
[clojure.spec.test.alpha$explain_check invoke "alpha.clj" 275]
[clojure.spec.test.alpha$check_call invokeStatic "alpha.clj" 295]
[clojure.spec.test.alpha$check_call invoke "alpha.clj" 285]
[clojure.spec.test.alpha$quick_check$fn__2986 invoke "alpha.clj" 308]
[clojure.lang.AFn applyToHelper "AFn.java" 154]
[clojure.lang.AFn applyTo "AFn.java" 144]
[clojure.core$apply invokeStatic "core.clj" 657]
[clojure.core$apply invoke "core.clj" 652]
[clojure.test.check.properties$apply_gen$fn__16139$fn__16140 invoke "properties.cljc" 30]
[clojure.test.check.properties$apply_gen$fn__16139 invoke "properties.cljc" 29]
[clojure.test.check.rose_tree$fmap invokeStatic "rose_tree.cljc" 77]
[clojure.test.check.rose_tree$fmap invoke "rose_tree.cljc" 73]
[clojure.test.check.generators$fmap$fn__9199 invoke "generators.cljc" 101]
[clojure.test.check.generators$gen_fmap$fn__9173 invoke "generators.cljc" 57]
[clojure.test.check.generators$call_gen invokeStatic "generators.cljc" 41]
[clojure.test.check.generators$call_gen invoke "generators.cljc" 37]
[clojure.test.check$quick_check invokeStatic "check.cljc" 94]
[clojure.test.check$quick_check doInvoke "check.cljc" 37]
[clojure.lang.RestFn invoke "RestFn.java" 425]
[clojure.lang.AFn applyToHelper "AFn.java" 156]
[clojure.lang.RestFn applyTo "RestFn.java" 132]
[clojure.core$apply invokeStatic "core.clj" 657]
[clojure.core$apply invoke "core.clj" 652]
[clojure.spec.gen.alpha$quick_check invokeStatic "alpha.clj" 29]
[clojure.spec.gen.alpha$quick_check doInvoke "alpha.clj" 27]
[clojure.lang.RestFn applyTo "RestFn.java" 137]
[clojure.core$apply invokeStatic "core.clj" 661]
[clojure.core$apply invoke "core.clj" 652]
[clojure.spec.test.alpha$quick_check invokeStatic "alpha.clj" 309]
[clojure.spec.test.alpha$quick_check invoke "alpha.clj" 302]
[clojure.spec.test.alpha$check_1 invokeStatic "alpha.clj" 335]
[clojure.spec.test.alpha$check_1 invoke "alpha.clj" 323]
[clojure.spec.test.alpha$check$fn__3005 invoke "alpha.clj" 411]
[clojure.core$pmap$fn__8105$fn__8106 invoke "core.clj" 6942]
[clojure.core$binding_conveyor_fn$fn__5476 invoke "core.clj" 2022]
[clojure.lang.AFn call "AFn.java" 18]
[java.util.concurrent.FutureTask run "FutureTask.java" 266]
[java.util.concurrent.ThreadPoolExecutor runWorker "ThreadPoolExecutor.java" 1149]
[java.util.concurrent.ThreadPoolExecutor$Worker run "ThreadPoolExecutor.java" 624]
[java.lang.Thread run "Thread.java" 748]]



 Comments   
Comment by Jonathan Leonard [ 02/Jan/18 12:47 PM ]

So, I was able to finally track this down to an actual problem where my function value returned from the function under test would throw and exception on a particular input.

This patch helps point people in that direction (although an even better one would be to surface/explain the underlying failure rather than just allude to it).

Comment by Jonathan Leonard [ 01/Feb/18 2:50 PM ]

Anyone going to respond to this? What's the usual expected response time for feedback on a patch/pull request?

Comment by Alex Miller [ 01/Feb/18 2:58 PM ]

There is not enough information here for this ticket to be actionable. If (s/valid? spec data) is false, then (s/explain-data spec data) should never be nil. The fact that you're seeing that is indication of a bug in the spec implementation, but without more info about the spec or the data, there's nothing I can do at the moment.

Comment by Jonathan Leonard [ 01/Feb/18 3:35 PM ]

The "actionable" part is the submitted patch. This code is more resilient and friendly to sufferers of the issue.

The actual text of this ticket is merely a CLONE of the original (since I could not re-open or edit the original).

Comment by Jonathan Leonard [ 01/Feb/18 3:37 PM ]

i.e., the only way I could submit this patch was to create a CLONE.

Comment by Andy Fingerhut [ 02/Feb/18 12:37 AM ]

Jonathan, since you are on the list of Clojure contributors, I have bumped up your permissions on Clojure JIRA so you should be able to edit tickets now.

I do not think there is a 'usual time' by which tickets are responded to. It can vary by orders of magnitude, depending upon clarity and perceived criticality of the issue.

Alex would probably appreciate if the description included an example that can be evaluated in order to reproduce the problem. Often by seeing the problem, they may consider other solutions to the problem.

Comment by Alex Miller [ 02/Feb/18 7:18 AM ]

The patch is addressing the symptom, not the problem. We really need the spec and data value causing the issue.

Comment by Jonathan Leonard [ 02/Feb/18 3:38 PM ]

The patch does exactly what it claims to do: provide better messaging for this scenario.

Unfortunately my attempt at constructing a minimal repro from scratch was not successful. I understand that you would like me to do additional work but all work that I've done up to this point is pro bono and I cannot donate any more of my time to this cause at this time.

We should discuss the merits of this patch on its own-- is it an improvement over the previous code or not? [And static human analysis of the code can determine that it is in fact an improvement]. The root "problem" here is that `s/valid?` and `s/explain-data` are not pure, total functions-- they can fail probabilistically depending on the arguments (i.e., when there is a higher-order function involved [which due to a fundamental limitation of spec can only be "verified" by passing random data through it]). In fact, I cannot imagine any better way to address this issue given that limitation (except bubbling up the exception which was certainly swallowed in my original, real-world reproduction of this but which was not in my attempt at the minimal repro).

Comment by Jonathan Leonard [ 08/Mar/18 1:19 PM ]

I would appreciate a response to the provided reasoning.

Comment by Ghadi Shayban [ 08/Mar/18 7:33 PM ]

Without a reproduction case, I'm afraid this will not move forward. If there is an invariant being broken (e.g. invalid data but missing explain-data) provide a example. It doesn't have to be minimal, just needs to be reproducible.

[1] https://dev.clojure.org/display/community/Creating+Tickets
[2] https://dev.clojure.org/display/community/Developing+Patches (see Before You Code)

Comment by Jonathan Leonard [ 08/Mar/18 7:49 PM ]

So, are you saying that the Clojure project does not accept bug reports that were discovered by manual human inspection? Because if it does accept such, then you should consider this bug to be a case of such.

Also, if you would allow yourselves to merely think about the two functions in question, you can "discover" the bug for yourselves. Do you accept that `s/explain-data` and `s/valid?` are not purely deterministic? i.e., they can return different results than both each other and themselves given the exact same input on two successive runs. If you do accept this, then you have convinced yourselves of the validity of this bug claim (and I strongly urge you to accept that because it is the reality of the situation).

Comment by Ghadi Shayban [ 08/Mar/18 8:20 PM ]

I am trying to help you formulate a better, actionable ticket. I don't speak on behalf of 'the Clojure project'. I implore you to tone down the antagonism. Language like "if you would allow yourselves to merely think about the two functions in question" doesn't belong here.

I don't doubt that you've uncovered an issue, but as it stands this is unactionable.

Comment by Jonathan Leonard [ 08/Mar/18 9:01 PM ]

I think you should go back and re-read my previous message. Imagine that it were spoken calmly, plainly and matter-of-factly because that is how it was written/intended (instead of however you have chosen to read it). Also, it was not new information— merely a rewording of previous communications (which do not seem to have been understood).

And I really do not appreciate the sanctimonious, condescending, patronizing in your previous message to me. Save the moral grandstanding for somewhere more appropriate (wherever that may be). It has no place in a technical bug report.

Comment by Alex Miller [ 09/Mar/18 8:47 AM ]

Regarding the non-determinism, that's a known issue being tracked in CLJ-1949. Assuming that's addressed, I don't see the need for this change. However, I'm not sure where all that would go so I'm just going to leave this ticket open and defer to later when that becomes clearer.

Comment by Jonathan Leonard [ 09/Mar/18 3:07 PM ]

Ah, yes, if CLJ-1949 were fixed then this would not be an issue. Thanks for the (reasonable) explanation.





[CLJ-2333] Support java.nio.file.Path in clojure.java.io Created: 07/Mar/18  Updated: 08/Mar/18

Status: Open
Project: Clojure
Component/s: None
Affects Version/s: Release 1.9
Fix Version/s: None

Type: Enhancement Priority: Minor
Reporter: Michael Nygard Assignee: Unassigned
Resolution: Unresolved Votes: 0
Labels: io

Approval: Triaged

 Description   

java.nio.file.Path objects are largely equivalent to java.io.File. They represent a location in a (possibly-remote) filesystem.

Coercions in clojure.java.io don't recognize java.nio.file.Path. It would be nice if they did.






[CLJ-2054] generator for `any?` occasionally generates `Double/NaN` for which equality semantics don't apply, and that is a problem for the :ret spec of many functions. Created: 07/Nov/16  Updated: 06/Mar/18

Status: Open
Project: Clojure
Component/s: None
Affects Version/s: Release 1.9
Fix Version/s: None

Type: Enhancement Priority: Major
Reporter: Dimitrios Piliouras Assignee: Unassigned
Resolution: Unresolved Votes: 3
Labels: generator, spec
Environment:

Ubuntu 16.10 - Oracle Java 8


Attachments: Text File exclude_NaN_from_any_and_some_generators.patch    
Approval: Triaged

 Description   

The generator for `any?` will occasionally give back Double/NaN value(s). Since NaNs & equality (via `=`) don't get along, :ret spec'ing a fn which transforms/processes a collection according to a predicate, becomes rather problematic. That's because the most obvious thing to check under :ret (the case where the predicate didn't return true for any value, and so the output coll should be equal to the input coll because nothing was transformed/processed), cannot be expressed trivially.

The workaround I've come up with in my own specs is to spec the elements of the collection with `(s/and any? (complement double-NaN?))` instead of just `any?`, and it works. However, even though I can live without NaNs in the tests, I must admit it still feels sort of hacky.

Ideas:

1) The generator for `any?` could be hardcoded to never return Double/NaN. Sounds rather invasive.
2) The generator for `any?` could be reworked to somehow be configurable wrt allowing/prohibiting Double/NaNs. Then perhaps a dynamic-var and/or a macro (e.g. `without-NaNs`) could expose this (just brainstorming here).
3) The generator for `any? could stay as is, but a new equality operator could be added (e.g. `clojure.spec/===`), which somehow ignores NaNs (a naive implementation for instance might walk the data-structures and replace all NaNs with keywords, and only then perform a regular comparison).



 Comments   
Comment by Alex Miller [ 08/Nov/16 10:29 AM ]

Should consider whether this change is more appropriate in test.check or in the spec generator for any?.

Comment by Dimitrios Piliouras [ 11/Nov/16 12:31 PM ]

It turns out that my workaround does not fully work. I literally just stumbled in the following case:

{nil {[] {NaN 0}}}

which is a conforming value for:

(s/def ::persistent-map
(s/map-of ::anything-but-NaN ::anything-but-NaN)) ;; (s/and any? (complement double-NaN?))

So basically, the inner collections can still have NaNs. So far I've got 4 specs that I've written and faced this problem on all of them.

Comment by Wes Morgan [ 06/Mar/18 3:37 PM ]

exclude_NaN_from_any_and_some_generators.patch is my attempt at addressing this by preventing NaNs from appearing in any? and some? generated values. It seemed to solve the problem in my testing.





[CLJ-2329] partition-all docstring should mention it returns vectors in transducer case Created: 02/Mar/18  Updated: 06/Mar/18

Status: Open
Project: Clojure
Component/s: None
Affects Version/s: Release 1.9
Fix Version/s: None

Type: Enhancement Priority: Minor
Reporter: A. R Assignee: Unassigned
Resolution: Unresolved Votes: 0
Labels: docstring

Approval: Triaged

 Description   

partition-all currently states that it returns a collection of lists. Though, when using as a transducer it returns vectors.



 Comments   
Comment by Phill Wolf [ 06/Mar/18 6:00 AM ]

There is more than a decorative difference between lists and vectors!

This does not seem like a doc bug. The transducer flavor of the function fails to meet a well-stated contract.

In the alternative, that it is just a doc bug and it's OK for partition-all to produce lists sometimes and vectors sometimes, based on a subtle configuration of your program, which you might change for computational reasons, with the nature of the returned structures farthest from your mind, thereby inadvertently up-ending all of your downstream conj's... then the doc bug is essentially that the docs have, all these years, over-stated an internal implementation detail by promising lists, and the safe way to consume the result of partition-all would be with sequence functions only.





[CLJ-2332] remove-tap docstring repetition Created: 05/Mar/18  Updated: 05/Mar/18

Status: Open
Project: Clojure
Component/s: None
Affects Version/s: None
Fix Version/s: None

Type: Defect Priority: Trivial
Reporter: Mike Fikes Assignee: Unassigned
Resolution: Unresolved Votes: 0
Labels: docstring

Approval: Triaged

 Description   

"Remove f from the tap set the tap set." should probably be "Remove f from the tap set."






[CLJ-2079] Generator overrides for spec aliases are not respected Created: 08/Dec/16  Updated: 04/Mar/18

Status: Open
Project: Clojure
Component/s: None
Affects Version/s: Release 1.9
Fix Version/s: Release 1.10

Type: Defect Priority: Major
Reporter: Nate Smith Assignee: Unassigned
Resolution: Unresolved Votes: 6
Labels: generator, spec

Approval: Vetted

 Description   

Generator overrides for spec aliases are not respected.

Unable to find source-code formatter for language: clojure. Available languages are: javascript, sql, xhtml, actionscript, none, html, xml, java
(require '[clojure.spec :as s])
(require '[clojure.spec.gen :as gen])
(s/def ::original number?)
(s/def ::alias ::original)

(every? double? (gen/sample (s/gen ::alias {::alias gen/double})))
;; => false

Providing a generator override for the original spec works as expected:

Unable to find source-code formatter for language: clojure. Available languages are: javascript, sql, xhtml, actionscript, none, html, xml, java
(every? double? (gen/sample (s/gen ::alias {::original gen/double})))
;; => true


 Comments   
Comment by Alex Miller [ 08/Dec/16 5:02 PM ]

Probably a missing delay in the alias case - there's another ticket that has the same cause.

Comment by Nate Smith [ 08/Dec/16 6:43 PM ]

Looks like it might be because gensub looks for matching overrides by calling spec-name, which returns the wrong value for spec aliases.

(require '[clojure.spec :as s])
(s/def ::original number?)
(s/def ::alias ::original)
(@#'clojure.spec/spec-name (s/get-spec ::alias))
;; => :user/original
Comment by Charles Despointes [ 20/Jun/17 1:19 PM ]

I've a somewhat similar issue. I think it is related.
I'm trying to do something like :

(require '[clojure.spec.alpha :as s])
(require '[clojure.spec.gen.alpha :as gen])
(s/def ::bar any?)
(s/def ::foo (s/with-gen any? (fn [] (s/gen ::bar))))
(gen/generate (s/gen ::foo {::bar (fn [] (s/gen int?))}))

I'm somewhat expecting it generates me an integer like it would have with a direct aliasing to ::bar in ::foo definition. But it doesn't and keep the with-gen binded generator.
Is that the same issue or is that an expected behaviour or should i fill a new issue ?

Comment by James Gatannah [ 04/Mar/18 12:07 AM ]

I think I'm probably running into this, except that there seems to be some non-determinism involved.

It seems to behave differently, depending on whether I run it from within a deftest or from the REPL. (Running it from the REPL seems to fail every time. Running it inside deftest seems much more reliable).

My really long-winded description is at https://groups.google.com/forum/#!topic/clojure/zPWPmQGm94w

The sample where I tried to document exactly what I'm seeing is at https://gist.github.com/jimrthy/21851c52a8cd6b04a31ed08b1d0a7f04 (when I wrote that, running inside deftest seemed to work every time. That is not the case).

If nothing else, it would be nice to have better error messages that include the name of the spec it's failing to generate. Actually, that would be generally helpful and possibly worth its own ticket. Please let me know if you'd like me to create one.





[CLJ-2331] Socket REPL :args should accept an edn string Created: 03/Mar/18  Updated: 03/Mar/18  Resolved: 03/Mar/18

Status: Closed
Project: Clojure
Component/s: None
Affects Version/s: None
Fix Version/s: None

Type: Enhancement Priority: Major
Reporter: Andrea Richiardi Assignee: Unassigned
Resolution: Declined Votes: 0
Labels: None


 Description   

Given the new developments of both cli and cljs.main command line parameters, it feels now natural to encapsulate data in edn files that can then be $(cat opt.edn)-ed at the command line.

It would be great if the same could be applied to the :args socket REPL argument, like:

cli -J-Dclojure.server.repl="{:port 5555 :accept clojure.core.server/repl :args $(cat socket-opts.edn)}"

Now that I am writing the rationale for this, I see that the whole map could be included in an .edn, but I am going to throw the idea out there so that we can start discussing about it.

My main use case for now would be to pass the ClojureScript compiler option map down to the socket repl without converting it to a vector of strings.



 Comments   
Comment by Andrea Richiardi [ 03/Mar/18 5:05 PM ]

Actually a completely different argument might be a better non-breaking idea.

Comment by Andrea Richiardi [ 03/Mar/18 11:01 PM ]

Basically I had completely missed the call to read-string at https://github.com/clojure/clojure/blob/master/src/clj/clojure/core/server.clj#L153.
After realizing that :args is already converted to Clojure data, David (Nolen) basically pushed a patch that solves the use case I was describing in the ticket. Closing.





[CLJ-2330] Clojure compiler build fails with Java 10 early access Created: 02/Mar/18  Updated: 02/Mar/18

Status: Open
Project: Clojure
Component/s: None
Affects Version/s: Release 1.9
Fix Version/s: None

Type: Defect Priority: Major
Reporter: John Schmidt Assignee: Chouser
Resolution: Unresolved Votes: 1
Labels: compiler

Approval: Triaged

 Description   

When building clojure-1.10.0-alpha4 with Java 10 early access the test test-proxy-non-serializable fails with the following error:

[java] ERROR in (test-proxy-non-serializable) (ObjectStreamClass.java:689)
     [java] That proxy classes refuse serialization and deserialization
     [java] expected: (thrown? java.io.NotSerializableException (-> serialized-proxy (.getBytes "ISO-8859-1") java.io.ByteArrayInputStream. java.io.ObjectInputStream. .readObject))
     [java]   actual: java.io.InvalidClassException: javax.swing.event.EventListenerList; local class incompatible: stream classdesc serialVersionUID = -5677132037850737084, local class serialVersionUID = -7977902244297240866
     [java]  at java.io.ObjectStreamClass.initNonProxy (ObjectStreamClass.java:689)
     [java]     java.io.ObjectInputStream.readNonProxyDesc (ObjectInputStream.java:1894)
     [java]     java.io.ObjectInputStream.readClassDesc (ObjectInputStream.java:1763)
     [java]     java.io.ObjectInputStream.readOrdinaryObject (ObjectInputStream.java:2051)
     [java]     java.io.ObjectInputStream.readObject0 (ObjectInputStream.java:1585)
     [java]     java.io.ObjectInputStream.defaultReadFields (ObjectInputStream.java:2346)
     [java]     java.io.ObjectInputStream.readSerialData (ObjectInputStream.java:2240)
     [java]     java.io.ObjectInputStream.readOrdinaryObject (ObjectInputStream.java:2078)
     [java]     java.io.ObjectInputStream.readObject0 (ObjectInputStream.java:1585)
     [java]     java.io.ObjectInputStream.readObject (ObjectInputStream.java:422)
     [java]     clojure.test_clojure.java_interop$fn__12007.invokeStatic (java_interop.clj:201)
     [java]     clojure.test_clojure.java_interop/fn (java_interop.clj:187)
     [java]     clojure.test$test_var$fn__9377.invoke (test.clj:716)
     [java]     clojure.test$test_var.invokeStatic (test.clj:716)
     [java]     clojure.test$test_var.invoke (test.clj:707)
     [java]     clojure.test$test_vars$fn__9403$fn__9408.invoke (test.clj:734)
     [java]     clojure.test$default_fixture.invokeStatic (test.clj:686)
     [java]     clojure.test$default_fixture.invoke (test.clj:682)
     [java]     clojure.test$test_vars$fn__9403.invoke (test.clj:734)
     [java]     clojure.test$default_fixture.invokeStatic (test.clj:686)
     [java]     clojure.test$default_fixture.invoke (test.clj:682)
     [java]     clojure.test$test_vars.invokeStatic (test.clj:730)
     [java]     clojure.test$test_all_vars.invokeStatic (test.clj:736)
     [java]     clojure.test$test_ns.invokeStatic (test.clj:757)
     [java]     clojure.test$test_ns.invoke (test.clj:742)
     [java]     clojure.core$map$fn__5675.invoke (core.clj:2747)
     [java]     clojure.lang.LazySeq.sval (LazySeq.java:40)
     [java]     clojure.lang.LazySeq.seq (LazySeq.java:49)
     [java]     clojure.lang.Cons.next (Cons.java:39)
     [java]     clojure.lang.RT.next (RT.java:706)
     [java]     clojure.core$next__5196.invokeStatic (core.clj:64)
     [java]     clojure.core$reduce1.invokeStatic (core.clj:936)
     [java]     clojure.core$reduce1.invokeStatic (core.clj:926)
     [java]     clojure.core$merge_with.invokeStatic (core.clj:3051)
     [java]     clojure.core$merge_with.doInvoke (core.clj:3043)
     [java]     clojure.lang.RestFn.applyTo (RestFn.java:139)
     [java]     clojure.core$apply.invokeStatic (core.clj:659)
     [java]     clojure.test$run_tests.invokeStatic (test.clj:767)
     [java]     clojure.test$run_tests.doInvoke (test.clj:767)
     [java]     clojure.lang.RestFn.applyTo (RestFn.java:137)
     [java]     clojure.core$apply.invokeStatic (core.clj:657)
     [java]     user$eval31031.invokeStatic (run_test.clj:8)
     [java]     user$eval31031.invoke (run_test.clj:8)
     [java]     clojure.lang.Compiler.eval (Compiler.java:7059)
     [java]     clojure.lang.Compiler.load (Compiler.java:7511)
     [java]     clojure.lang.Compiler.loadFile (Compiler.java:7449)
     [java]     clojure.main$load_script.invokeStatic (main.clj:278)
     [java]     clojure.main$script_opt.invokeStatic (main.clj:338)
     [java]     clojure.main$script_opt.invoke (main.clj:333)
     [java]     clojure.main$main.invokeStatic (main.clj:424)
     [java]     clojure.main$main.doInvoke (main.clj:387)
     [java]     clojure.lang.RestFn.applyTo (RestFn.java:137)
     [java]     clojure.lang.Var.applyTo (Var.java:702)
     [java]     clojure.main.main (main.java:37)

Alex Miller suggested on Slack that the test is now invalid due to a serialVersionUID change.

Output of java -version:

➜  clojure git:(master) ✗ java -version
openjdk version "10" 2018-03-20
OpenJDK Runtime Environment 18.3 (build 10+45)
OpenJDK 64-Bit Server VM 18.3 (build 10+45, mixed mode)


 Comments   
Comment by Alex Miller [ 02/Mar/18 5:10 PM ]

Test failure related to CLJ-2204





Generated at Thu Apr 26 17:59:13 CDT 2018 using JIRA 4.4#649-r158309.