<< Back to previous view

[CTYP-99] Checking an ns becomes significantly slower as the number of optional keys in a HMap increases Created: 03/Dec/13  Updated: 14/Feb/14  Resolved: 14/Feb/14

Status: Resolved
Project: core.typed
Component/s: None
Affects Version/s: None
Fix Version/s: None

Type: Defect Priority: Blocker
Reporter: Gordon Syme Assignee: Ambrose Bonnaire-Sergeant
Resolution: Completed Votes: 1
Labels: None

Attachments: File core.clj     File core.clj     File project.clj     GZip Archive test-hmap.tar.gz    

 Description   

I'm using clojure.core.typed 0.2.19 with the slim classifier but I've observed the same without slim.

My suspicion is that the latent filters associated with functions grow in size exponentially with each extra optional key to a HMap (based on the output when you have a type error). I think it's generating all combinations of present and absent keys for the HMap when calculating latent filters for a function.

I've attached a tarball with a lein project with ten namespaces that all contain the same ten simple functions in the form

(t/ann f [(HMap :optional {:a String}) -> (t/Option String)])
(defn f [m]
  (:a m))

The type annotations vary in the number of optional keywords.

(test-hmap.core/go) checks all the namespaces. The time to check each namespace grows non-linearly. The first namespace gets penalised by core.typed initialisation the first time it's run.

E.g. on my local machine:

test-hmap.one
Start collecting test-hmap.one
Finished collecting test-hmap.one
Collected 1 namespaces in 20.743 msecs
Start checking test-hmap.one
Checked test-hmap.one in 46.231 msecs
Checked 1 namespaces (approx. 34 lines) in 67.672 msecs


test-hmap.two
Start collecting test-hmap.two
Finished collecting test-hmap.two
Collected 1 namespaces in 18.751 msecs
Start checking test-hmap.two
Checked test-hmap.two in 45.525 msecs
Checked 1 namespaces (approx. 35 lines) in 64.814 msecs


test-hmap.three
Start collecting test-hmap.three
Finished collecting test-hmap.three
Collected 1 namespaces in 19.106 msecs
Start checking test-hmap.three
Checked test-hmap.three in 57.346 msecs
Checked 1 namespaces (approx. 35 lines) in 77.055 msecs


test-hmap.four
Start collecting test-hmap.four
Finished collecting test-hmap.four
Collected 1 namespaces in 21.222 msecs
Start checking test-hmap.four
Checked test-hmap.four in 51.339 msecs
Checked 1 namespaces (approx. 35 lines) in 73.155 msecs


test-hmap.five
Start collecting test-hmap.five
Finished collecting test-hmap.five
Collected 1 namespaces in 22.215 msecs
Start checking test-hmap.five
Checked test-hmap.five in 63.415 msecs
Checked 1 namespaces (approx. 35 lines) in 86.309 msecs


test-hmap.six
Start collecting test-hmap.six
Finished collecting test-hmap.six
Collected 1 namespaces in 23.76 msecs
Start checking test-hmap.six
Checked test-hmap.six in 99.407 msecs
Checked 1 namespaces (approx. 35 lines) in 123.881 msecs


test-hmap.seven
Start collecting test-hmap.seven
Finished collecting test-hmap.seven
Collected 1 namespaces in 26.519 msecs
Start checking test-hmap.seven
Checked test-hmap.seven in 213.515 msecs
Checked 1 namespaces (approx. 35 lines) in 240.639 msecs


test-hmap.eight
Start collecting test-hmap.eight
Finished collecting test-hmap.eight
Collected 1 namespaces in 32.581 msecs
Start checking test-hmap.eight
Checked test-hmap.eight in 626.76 msecs
Checked 1 namespaces (approx. 35 lines) in 659.975 msecs


test-hmap.nine
Start collecting test-hmap.nine
Finished collecting test-hmap.nine
Collected 1 namespaces in 43.478 msecs
Start checking test-hmap.nine
Checked test-hmap.nine in 2634.468 msecs
Checked 1 namespaces (approx. 35 lines) in 2678.716 msecs


test-hmap.ten
Start collecting test-hmap.ten
Finished collecting test-hmap.ten
Collected 1 namespaces in 84.394 msecs
Start checking test-hmap.ten
Checked test-hmap.ten in 9277.54 msecs
Checked 1 namespaces (approx. 35 lines) in 9362.613 msecs


 Comments   
Comment by Ambrose Bonnaire-Sergeant [ 03/Dec/13 12:50 PM ]

Yes, the primitives are :mandatory, :absent-keys and :complete; :optional expands to be in terms of those.

Thanks for the report, I haven't done performance testing on this strategy. It will probably need to be reconsidered.

Comment by Cees van Kemenade [ 26/Jan/14 1:04 PM ]

code of project ctcrash

Comment by Cees van Kemenade [ 26/Jan/14 1:05 PM ]

+1 For this issue.

I've spend quite some time to learn core.typed on a real use case, and I have to say that I'm amazed by the power of the analysis and the potential to prevent errors that otherwise would be detected to late! Apart from that the type-annotations are valuable and precise information about the interface when doing maintenance at code you did not see for some time.
Two big reasons to use core.typed, however, as I turned more and more code into typed code the number of optional keys increased in my core data-record increase, and core.typed (check-ns) slowed down to the extreme (using version 0.2.25 (current stable)). I refactored the code to get a single line of code that triggers the issue on my core data structure a (Seqable DbMsg). Attached you find the project.clj and code (core.clj vs 8:09pm).

When I reduce the number of optional keys to 4-5 the (check-ns) runs in a few seconds. But having 9-11 optional keys kills the (check-ns) process.

I hope this one-liner helps making the issues easy to reproduce.
Currently I don't know a work-around for this issue,
so I can not use core.typed to check my project.

I guess that many people/projects should run into the same issues,
as the pattern is returning quite often in clojure:

1. start with data-set with small Hashmaps
2. the hashmaps evolve in a number steps (meaning more keys are added and some of the existing keys are removed)
I guess the workaround would be to use limited or no optional keys, and prepare a custom HMap definition for each stage of analysis. Although this would work, the amount of bookkeeping already significant for a all records are running through the same stages (linear process), and will be a showstopper if it is a non-linear process (not all records follow the same path (sequence of analysis stages)).
I will investigate whether splitting in multiple types (for different stages) will rescue my day.

Comment by Ambrose Bonnaire-Sergeant [ 26/Jan/14 7:51 PM ]

Upgraded to 'Blocker'.

Comment by Ambrose Bonnaire-Sergeant [ 27/Jan/14 6:34 AM ]

This is theoretically trivial to implement, and just involves shuffling around where optional keys are handled. However, this is spread out all over the code base, so I may not complete the patch for a week.

Great to hear core.typed is working out for you!

Comment by Cees van Kemenade [ 27/Jan/14 6:44 AM ]

Thanks for picking up this issue fast.
I'll wait patiently.

Comment by Ambrose Bonnaire-Sergeant [ 14/Feb/14 2:26 AM ]

This is fixed in 0.2.31.

I've added both of your tests in the test suite - thanks! https://github.com/typedclojure/core.typed-test-suite/tree/master/src/hmap

Please confirm and I'll close the issue.

Comment by Cees van Kemenade [ 14/Feb/14 4:11 AM ]

This resolve the issue and also resolves CT-102.
Thanks!

The code in hmap/big-options.clj (from your test-suite) still contains a workaround to make core.typed proceed without errors.
The (if (string? notifs) ... on line 107 is introduced to prevent a type-error (core.typed infers it that notifs might als be a (sequable String). However, as this is the else-branch of line 104 (if (or (seq? notifs) (vector notifs)) ... the case that notifs is a (Seqable string) can be excluded already).
Also using on line 104 the more general (if (sequential? notifs) ... does not resolve this issues.

Is this inference error already on the list, or would you recommend posting a seperate JIRA-case for this issue?

Comment by Gordon Syme [ 14/Feb/14 5:56 AM ]

Our core.typed checks run an order of magnitude quicker now (~400s --> ~80s). And they keep that speed with all the extra optionals now. That's really awesome, thanks Ambrose!





[CTYP-161] "Cannot assoc args" with HMap intersections Created: 09/Jul/14  Updated: 20/Jul/14  Resolved: 20/Jul/14

Status: Resolved
Project: core.typed
Component/s: None
Affects Version/s: None
Fix Version/s: None

Type: Defect Priority: Blocker
Reporter: Timo Mihaljov Assignee: Ambrose Bonnaire-Sergeant
Resolution: Completed Votes: 0
Labels: None
Environment:

[org.clojure/core.typed "0.2.53"]



 Description   
=> (t/cf (assoc (t/ann-form {:foo 1, :bar 2}
                            (t/HMap :mandatory {:foo Number
                                                :bar Number}))
                :baz 3))
[(t/HMap :mandatory {:baz (t/Value 3), :foo java.lang.Number, :bar java.lang.Number}) {:then tt, :else ff}]

=> (t/cf (assoc (t/ann-form {:foo 1, :bar 2}
                            (t/I (t/HMap :mandatory {:foo Number})
                                 (t/HMap :mandatory {:bar Number})))
                :baz 3))
Type Error (/private/var/folders/x2/47j5hlbs01b9b4mjz_8fyf05m8hy4p/T/form-init2093510643541319345.clj:1:7) Cannot assoc args `[(clojure.core.typed/Value :baz) {:then tt, :else ff}] [(clojure.core.typed/Value 3) {:then tt, :else ff}]` on (clojure.core.typed/I (clojure.core.typed/HMap :mandatory {:foo java.lang.Number}) (clojure.core.typed/HMap :mandatory {:bar java.lang.Number}))
ExceptionInfo Type Checker: Found 1 error  clojure.core/ex-info (core.clj:4327)



 Comments   
Comment by Timo Mihaljov [ 09/Jul/14 6:15 AM ]

As a workaround, assoc can be replaced with assoc-in. It loses the type and returns Any, but the type can be recovered with an assertion or a postcondition (e.g. {:post [((t/pred MyMap) %)]}).

Comment by Ambrose Bonnaire-Sergeant [ 20/Jul/14 4:42 AM ]

Fixed, will appear in 0.2.59

https://github.com/clojure/core.typed/commit/e74a533e61cd9e9431d4e16f7b793ed3a50237d9





[CTYP-83] check-ns takes exponentially longer on each invocation Created: 04/Oct/13  Updated: 01/Sep/14  Resolved: 01/Sep/14

Status: Resolved
Project: core.typed
Component/s: None
Affects Version/s: None
Fix Version/s: None

Type: Defect Priority: Blocker
Reporter: Hugo Duncan Assignee: Ambrose Bonnaire-Sergeant
Resolution: Completed Votes: 0
Labels: None


 Description   

Running `(check-ns pathological)` repeatedly on the following namespace, `check-ns` takes longer and longer to complete.

(ns pathological
  (:require
   [clojure.core.typed :refer [ann def-alias loop>
                               NonEmptyVec NilableNonEmptySeq]]))

(def-alias VersionVector (NonEmptyVec Number))
(ann version-less [(U nil VersionVector) (U nil VersionVector) -> boolean])
(defn version-less
  "Compare two version vectors."
  [v1 v2]
  (loop> [v1 :- (NilableNonEmptySeq Number) (seq v1)
          v2 :- (NilableNonEmptySeq Number) (seq v2)]
    (let [fv1 (first v1)
          fv2 (first v2)]
      (cond
       (and (not v1) (not v2)) false
       (and v1 (not v2)) false
       (or (and (not v1) v2)
           (and fv1 fv2 (< fv1 fv2))) true
           (and fv1 fv2 (> fv1 fv2)) false
           :else (recur (next v1) (next v2))))))


 Comments   
Comment by Ambrose Bonnaire-Sergeant [ 01/Sep/14 3:41 PM ]

Fix will be included in 0.2.68 https://github.com/clojure/core.typed/commit/0065e05fa20c3c33dea49937558af0933c363125





[CTYP-91] polymorphic ann-protocol doesn't work Created: 30/Oct/13  Updated: 05/Nov/13  Resolved: 05/Nov/13

Status: Resolved
Project: core.typed
Component/s: None
Affects Version/s: None
Fix Version/s: None

Type: Defect Priority: Major
Reporter: Russell Mull Assignee: Ambrose Bonnaire-Sergeant
Resolution: Completed Votes: 0
Labels: None
Environment:

Clojure 1.5.1, Core.typed 0.2.14 and 0.2.15



 Description   

The example of a polymorphic defprotocol annotation from the documentation is:

(ann-protocol [[x :variance :covariant]]
  IFoo
  bar
  [IFoo -> Any]
  baz
  [IFoo -> Number])

It's implied that there exists a corresponding protocol like this:

(defprotocol> IFoo
  (bar [this])
  (baz [this]))

Type checking this code with check-ns gives an error:

AssertionError Assert failed: ((u/hash-c? (every-pred symbol? (complement namespace)) Type?) methods)
	clojure.core.typed.type-rep/Protocol-maker (type_rep.clj:289)
	clojure.core.typed.type-ctors/Protocol* (type_ctors.clj:448)
	clojure.core.typed.collect-phase/gen-protocol* (collect_phase.clj:443)
	clojure.core.typed.collect-phase/fn--14235 (collect_phase.clj:458)
	clojure.lang.MultiFn.invoke (MultiFn.java:227)
	clojure.core.typed.collect-phase/collect -COLON-invoke14011 (collect_phase.clj:189)
	clojure.lang.MultiFn.invoke (MultiFn.java:227)
	clojure.core.typed.collect-phase/collect-asts (collect_phase.clj:119)
	clojure.core.typed.collect-phase/collect-ns (collect_phase.clj:107)
	clojure.core.typed/check-ns (typed.clj:1459)
	clojure.core.typed/check-ns (typed.clj:1429)

Incidentally, it would be nice if the example in the documentation actually used the declared type parameter in some way.



 Comments   
Comment by Russell Mull [ 30/Oct/13 10:01 PM ]

The above stack trace is from 0.2.14. This is the trace for 0.2.15:

AssertionError Assert failed: ((u/hash-c? (every-pred symbol? (complement namespace)) Type?) methods)
	clojure.core.typed.type-rep/Protocol-maker (type_rep.clj:289)
	clojure.core.typed.type-ctors/Protocol* (type_ctors.clj:448)
	clojure.core.typed.collect-phase/gen-protocol* (collect_phase.clj:443)
	clojure.core.typed.collect-phase/fn--14237 (collect_phase.clj:458)
	clojure.lang.MultiFn.invoke (MultiFn.java:227)
	clojure.core.typed.collect-phase/collect -COLON-invoke14013 (collect_phase.clj:189)
	clojure.lang.MultiFn.invoke (MultiFn.java:227)
	clojure.core.typed.collect-phase/collect-asts (collect_phase.clj:119)
	clojure.core.typed.collect-phase/collect-ns (collect_phase.clj:107)
	clojure.core.typed/check-ns (typed.clj:1459)
	clojure.core.typed/check-ns (typed.clj:1429)
Comment by Ambrose Bonnaire-Sergeant [ 05/Nov/13 9:52 AM ]

This exposes that polymorphic protocols are completely untested :/

This issue is fixed in 0.2.16: https://github.com/clojure/core.typed/commit/4474f76eea1e966523914c5455761f1e7b8542c8

Also updated the doc: https://github.com/clojure/core.typed/commit/216454c8c292d68dfa1ac1cd5ce36e00f43cc1c8

This doesn't mean polymorphic protocols/datatypes are usable yet, but it's progress. Working on it.





[CTYP-105] derive/subtype failure on hashmap Created: 19/Feb/14  Updated: 19/Feb/14  Resolved: 19/Feb/14

Status: Resolved
Project: core.typed
Component/s: None
Affects Version/s: None
Fix Version/s: None

Type: Defect Priority: Major
Reporter: Cees van Kemenade Assignee: Ambrose Bonnaire-Sergeant
Resolution: Completed Votes: 0
Labels: None
Environment:

linux/java + core.typed 0.2.31


Attachments: File failDerive2.clj    

 Description   

The attached code fails when checking the namespace, as core.typed incorrectly assumes that the return-type is incorrect. Part of error-output is included as string ErrInfo and compared for difference with the return-type from the proto-type of the failing function.

When you load the file the analysis of the error-message is shown (run check-ns to see the message)



 Comments   
Comment by Ambrose Bonnaire-Sergeant [ 19/Feb/14 3:00 AM ]

Fixed in 0.2.32: https://github.com/clojure/core.typed/commit/0e0a41c917ef027bf47ab6ed90955881975a7a13

Comment by Cees van Kemenade [ 19/Feb/14 12:16 PM ]

Great!

Thanks for the rapid fix.
Tested it it and it works fine.

C.





[CTYP-146] java.lang.AssertionError: Assert failed: (<= (count s) cnt) Created: 26/May/14  Updated: 27/May/14  Resolved: 27/May/14

Status: Resolved
Project: core.typed
Component/s: None
Affects Version/s: 0.2.0
Fix Version/s: None

Type: Defect Priority: Major
Reporter: Kris Jenkins Assignee: Ambrose Bonnaire-Sergeant
Resolution: Completed Votes: 0
Labels: None
Environment:

OSX


Attachments: Text File core-typed-bug.txt    

 Description   

With core.typed 0.2.50 I get this exception:

Exception in thread "main" java.lang.AssertionError: Assert failed: (<= (count s) cnt)
at clojure.core.typed.cs_gen$pad_right.invokePrim(cs_gen.clj:1252)
at clojure.core.typed.cs_gen$cs_gen_Function.invoke(cs_gen.clj:1296)
at clojure.core.typed.cs_gen$any_impl_RBRACK_11316$iter_1131711321$fn11322$fn11323$fn11324$iter1132511329$fn11330$fn11331$fn11332$fn_11333.invoke(cs_gen.clj:864)
at clojure.core.typed.cs_gen$any_impl_RBRACK_11316$iter_1131711321$fn11322$fn11323$fn11324$iter1132511329$fn11330$fn11331$fn_11332.invoke(cs_gen.clj:863)
at clojure.core.typed.cs_gen$any_impl_RBRACK_11316$iter_1131711321$fn11322$fn11323$fn11324$iter1132511329$fn11330$fn_11331.invoke(cs_gen.clj:861)
at clojure.core.typed.cs_gen$any_impl_RBRACK_11316$iter_1131711321$fn11322$fn11323$fn11324$iter1132511329$fn_11330.invoke(cs_gen.clj:861)
...

When type-checking this code (either from lein typed (0.3.4) or the Emacs plugin):

(ns demo
(:require [clojure.core.typed :refer [ann Seq Option]]))

(ann distinct-except
(All [x]
[[x -> Boolean] (Option (Seq x)) -> (Option (Seq x))]))
(defn distinct-except
"Same as distinct, but keeps duplicates if they pass exception?"
[exception? [head & tail :as coll]]
(lazy-seq
(when head
(cons head
(distinct-except exception?
(if (exception? head)
tail
(remove (partial = head) tail)))))))

I get the same exception if I just (cf ...) the defn too. I've attached the full stacktrace. Let me know if there's anything else I can helpfully supply.



 Comments   
Comment by Kris Jenkins [ 26/May/14 2:49 PM ]

Oh, sorry - Jira code formatting. Here's that code again:

Unable to find source-code formatter for language: clojure. Available languages are: javascript, sql, xhtml, actionscript, none, html, xml, java
(ns demo
  (:require [clojure.core.typed :refer [ann Seq Option]]))

(ann distinct-except
  (All [x]
       [[x -> Boolean] (Option (Seq x)) -> (Option (Seq x))]))
(defn distinct-except
  "Same as distinct, but keeps duplicates if they pass exception?"
  [exception? [head & tail :as coll]]
  (lazy-seq
   (when head
     (cons head
           (distinct-except exception?
                            (if (exception? head)
                              tail
                              (remove (partial = head) tail)))))))
Comment by Ambrose Bonnaire-Sergeant [ 27/May/14 11:48 AM ]

Fixed with commit: https://github.com/clojure/core.typed/commit/7599eb8601c12f625340889e4a8a6351d39d95d8





[CTYP-157] Fails to properly load protocols as soon as a ns containing a protocol is required from another checked ns Created: 17/Jun/14  Updated: 20/Jul/14  Resolved: 20/Jul/14

Status: Resolved
Project: core.typed
Component/s: None
Affects Version/s: None
Fix Version/s: None

Type: Defect Priority: Major
Reporter: Daniel Ziltener Assignee: Ambrose Bonnaire-Sergeant
Resolution: Completed Votes: 0
Labels: None
Environment:

Linux, Java 8.



 Description   

Minimal test case: https://gist.github.com/zilti/ac127170605600d008bb



 Comments   
Comment by Ambrose Bonnaire-Sergeant [ 20/Jul/14 5:30 AM ]

I believe this is fixed with lein-typed 0.3.5.

Please try it.





[CTYP-145] Offer separate zero-dependency artifact Created: 26/May/14  Updated: 20/Jul/14  Resolved: 20/Jul/14

Status: Resolved
Project: core.typed
Component/s: None
Affects Version/s: None
Fix Version/s: None

Type: Enhancement Priority: Major
Reporter: Chas Emerick Assignee: Ambrose Bonnaire-Sergeant
Resolution: Completed Votes: 0
Labels: None

Attachments: File CTYP-145-1.diff     File CTYP-145.diff    
Patch: Code

 Description   

The objective here is to be able to apply typed-clojure to libraries, without having their downstream dependents require the typed-clojure dependency as well. The libraries in question would then be able to fold in a separate TC dependency (which would bring in e.g. analyzer, ClojureScript, etc) for doing the actual type checking at test-time, in addition to their "regular" tests which would be run without the TC machinery. (Leiningen profiles would be ideal for setting up these different sorts of testing contexts.)

Tasks include:

1. Defining a "top-level" zero-dependency namespace(s) that provides all of the user-facing type-annotation macros (which must be no-ops when evaluated outside of a type-checking process).
2. Modifying the TC pom.xml and other project configuration so that two dependencies are produced; one without dependencies containing only the above-noted namespace(s), and another that depends upon the first and which also carries the necessary dependencies for actually performing type checking.

Namespaces to be included in no-dep jar:

src/main/clojure/clojure/core/typed.clj
src/main/clojure/clojure/core/typed/internal.clj
src/main/clojure/clojure/core/typed/current_impl.clj
src/main/clojure/clojure/core/typed/profiling.clj
src/main/clojure/clojure/core/typed/util_vars.clj
src/main/clojure/clojure/core/typed/profiling.clj
src/main/clojure/clojure/core/typed/errors.clj
src/main/clojure/clojure/core/typed/coerce_utils.clj
src/main/clojure/clojure/core/typed/ast_utils.clj
src/main/clojure/clojure/core/typed/type_contract.clj
src/main/clojure/clojure/core/typed/load_if_needed.clj
;; TODO: ClojureScript stuff


 Comments   
Comment by Ambrose Bonnaire-Sergeant [ 26/May/14 6:30 AM ]

Namespaces that should go in the zero-dep jar:

src/main/clojure/clojure/core/typed.clj
src/main/clojure/clojure/core/typed/internal.clj
src/main/clojure/clojure/core/typed/current_impl.clj
src/main/clojure/clojure/core/typed/profiling.clj
src/main/clojure/clojure/core/typed/util_vars.clj
src/main/clojure/clojure/core/typed/errors.clj
src/main/clojure/clojure/core/typed/coerce_utils.clj
src/main/clojure/clojure/core/typed/ast_utils.clj
Comment by Chas Emerick [ 26/May/14 7:25 AM ]

From ambrosebs in #typed-clojure: the name of the no-deps artifact will be core.typed.rt.

Comment by Ambrose Bonnaire-Sergeant [ 26/May/14 11:02 AM ]

Removed tools.analyzer dep https://github.com/clojure/core.typed/commit/b35131e503547d46e13ec9d50f7def84ebd8db04

Comment by Chas Emerick [ 28/May/14 8:59 AM ]

It looks like we need to produce four artifacts now: core.typed and core.typed.rt, in "regular" and "slim". Does this sound right?

Comment by Ambrose Bonnaire-Sergeant [ 28/May/14 9:14 AM ]

I guess so. I'm not sure we need core.typed.rt "regular", but it might be useful.

Comment by Chas Emerick [ 28/May/14 12:23 PM ]

For those looking on, the WIP patch is here:

https://github.com/cemerick/core.typed/commit/0cd50d09fa17b9c833066b4cf9db7457b8e7f959

A project that uses the new "rt" zero-dep artifact to enable annotations without requiring downstream dependents to depend upon core-typed is here:

https://github.com/cemerick/pprng/tree/WIP-typed-clojure

Official patch to be attached shortly.

Comment by Ambrose Bonnaire-Sergeant [ 28/May/14 1:55 PM ]

Added load_if_needed.clj (latest master).

Comment by Chas Emerick [ 29/May/14 6:15 AM ]

Patch attached. When additional namespaces should be added to the core.typed.rt artifact, just add them to the listing in module-rt/pom.xml.

core.typed installs locally just fine with this patch, but there's no easy way for me to test it in the Jenkins/build.clojure.org environment. Only one way to find out if it plays nice there.

Comment by Chas Emerick [ 29/May/14 10:56 AM ]

Attached an additional patch ({{CTYP-145-1.diff}}) that adds a minimal test to ensure that core.typed.rt can load. This will catch cases when that subset of typed-clojure expands, but e.g. the file list in module-rt/pom.xml hasn't been updated to match.

Comment by Ambrose Bonnaire-Sergeant [ 20/Jul/14 5:32 AM ]

org.clojure/core.typed.rt now lives!





[CTYP-106] Else branch does not tighten scope to non-sequential Created: 19/Feb/14  Updated: 20/Jul/14  Resolved: 20/Jul/14

Status: Resolved
Project: core.typed
Component/s: None
Affects Version/s: None
Fix Version/s: None

Type: Defect Priority: Major
Reporter: Cees van Kemenade Assignee: Ambrose Bonnaire-Sergeant
Resolution: Completed Votes: 0
Labels: None

Attachments: File failDerive1a.clj     File failDerive1.clj    

 Description   

This error is related to the example longOptions.clj of CTYPE-102 and the derived hmap/big-options mentioned as CTYPE-99. I've prepared a new example to clearly show the issue.

The first function passes type-checking. However, it contains a redundant check on line 83 and a redundant base-case on line 85.

The second function does not pass the (check-ns) as it assumes that notifs can be a (seqable String). However, as this is the else-branch of line 112 (if (or (sequential? notifs) (seq? notifs) (vector notifs)) ... so the notifs can not be a (Seqable String) in this branch.

NOTE: core.typed does not know clojure.core/sequential? yet. Therefore the seq? and vector? tests are added too.



 Comments   
Comment by Ambrose Bonnaire-Sergeant [ 19/Feb/14 5:03 AM ]

Seqable doesn't extend Sequential, ISeq or IPersistentVector, so it's expected that this fails.

I suspect that the type for :st_notif is always an immutable clojure collection? If so, annotate :st_notif as (U (t/Coll String) String nil), and use (coll? notifs) as the predicate. Otherwise, try (not (string? notifs)) as the predicate.

Comment by Cees van Kemenade [ 19/Feb/14 2:18 PM ]

I tried to turn it into clojure.core.typed/Coll and that worked fine.
Thanks for the suggestion/solution!

Out of curiosity I tried as an alternative to check for (= (class notifs) clojure.lang.Seqable) as a second solution, but that did not solve the issue.
So I will stick to solution 1.

Comment by Ambrose Bonnaire-Sergeant [ 19/Feb/14 7:21 PM ]

= isn't useful for narrowing types, especially in the else branch. instance? and isa? would be more useful.

Comment by Cees van Kemenade [ 20/Feb/14 9:48 AM ]

Sorry, it was a typo from my side. I used (isa? ....) see line 166 of the attached fail failDerive1a.clj
This code triggers a type-error on line 171.

Comment by Ambrose Bonnaire-Sergeant [ 20/Jul/14 5:42 AM ]

I think this is resolved.





[CTYP-31] Error with unannotated datatypes. Created: 23/Jul/13  Updated: 20/Jul/14  Resolved: 20/Jul/14

Status: Resolved
Project: core.typed
Component/s: None
Affects Version/s: None
Fix Version/s: None

Type: Defect Priority: Major
Reporter: Alan Busby Assignee: Ambrose Bonnaire-Sergeant
Resolution: Completed Votes: 0
Labels: None
Environment:

[org.clojure/core.typed "0.1.17"]



 Description   

From mail;
> > > Hmm. Is LocalConnection a datatype? What's the error you get?
> >
> > This: (t/ann ^:nocheck datomic.api/connect [String -> datomic.peer.LocalConnection])
> > Produces: AssertionError Assert failed: ((u/array-map-c? symbol? (some-fn Scope? Type?)) fields) clojure.core.typed.type-rep/->DataType (type_rep.clj:265)
>
> Thanks.
>
> core.typed doesn't like unannotated datatypes right now. This is probably a defect, please open a ticket.



 Comments   
Comment by Ambrose Bonnaire-Sergeant [ 20/Jul/14 5:45 AM ]

I believe this is more graceful now.





[CTYP-130] add HSequential type Created: 31/Mar/14  Updated: 10/Apr/14  Resolved: 10/Apr/14

Status: Resolved
Project: core.typed
Component/s: None
Affects Version/s: None
Fix Version/s: None

Type: Enhancement Priority: Minor
Reporter: Di Xu Assignee: Ambrose Bonnaire-Sergeant
Resolution: Completed Votes: 0
Labels: None

Attachments: Text File ambrose-hsequential-v3-1.patch     Text File ambrose-various-fixes.patch     File hsequential-demo.diff     File hsequential.diff     File hsequential-v2.diff     File hsequential-v3.diff    

 Description   

Hi, Ambrose

I've implemented part of HSequential as we discussed in CTYP-126.

But currently

(t/cf (fn [& y] (if (empty? y) nil (first y))))

will got Type error, saying

(I (HSequential [Any *]) clojure.core.typed/NonEmptyCount)

can't be applied to

(HSequential [x Any *])

it's easy to fix via adding special case in `upcast-to-HSequential` in patch, but this maybe not very general, I also think even `upcast-to-HSequential` isn't general at all.

How should I solve this error?



 Comments   
Comment by Ambrose Bonnaire-Sergeant [ 31/Mar/14 6:01 AM ]

You should call subtype directly with those two types and add printlns until the culprit is obvious.

There's a nice sub? macro in c.c.t.test.core

There's a bug in your second subtype? case; the test should be (HSequential? s), not (HSequential? t).

Comment by Ambrose Bonnaire-Sergeant [ 31/Mar/14 6:03 AM ]

I believe if you fix the subtype case, the left hand side will have a CountRange that satisfies the right.

Comment by Ambrose Bonnaire-Sergeant [ 31/Mar/14 6:08 AM ]

Hmm I see what you were trying to do. I suggest following HMap's subtyping.

Make `upcast-HSequential` which is [HSequential -> Type]. It always returns a slightly less accurate type, which is still useful to continue the subtyping check.

The subtyping case should test (HSequential? s) and then (subtype (upcast-HSequential s) t).

Comment by Ambrose Bonnaire-Sergeant [ 31/Mar/14 6:12 AM ]

I think upcast-HSequential should do something like

(HSequential [Any])
=>
(I (t/Coll Any) clojure.lang.Sequential (ExactCount 1))

Comment by Ambrose Bonnaire-Sergeant [ 31/Mar/14 6:23 AM ]

A nice thing we can do with HSequential is move most of the custom logic for (subtype HVec HVec), (subtype HList HList) and (subtype HSeq HSeq) into one place.

All we need is {HVec,Hlist,HSeq}->HSequential functions, and then we can have subtyping cases like

(and (HVec? s) (HVec? t))
(subtype (HVec->HSequential s) (HVec->HSequential t))

This also applies to cs_gen.

Comment by Di Xu [ 31/Mar/14 8:19 AM ]

Or, could we add special handler in `In` to handle the case of intersection of HSequential and CountRange, like change (In (HSequential [a b c *]) (CountRange 1 5)) -> (HSequential [a b c c c]) and (In (HSequential [a *]) (CountRange 1)) -> (HSequential [a a *])

something like that?

Comment by Ambrose Bonnaire-Sergeant [ 31/Mar/14 9:08 AM ]

I'm not sure. The first example would actually be (U (HSequential [a b]) (HSequential [a b c]) (HSequential [a b c c]) (HSequential [a b c c c]).

It might generate very large types.

Either way the subtype style I described is still needed.

Comment by Di Xu [ 01/Apr/14 10:55 PM ]

Well, I think we're talking about two different problem here, the problem you're talking about is how to compare HSequential with other type, the one I'm talking about is how to use CountRange to extends HSequential type, because

(HSequential [Any *])

can never be the subtype of

(HSequential [Any Any *])

, only when we put constraint that the count of first one is at least 1. Right?

So, I'll first extends `In` as I described here, and leave your solution to future. For the efficience consideration, I think it's not a big problem, because most CountRange just specified low bound as `empty?`, so we don't need to generate a large set very often.

Comment by Ambrose Bonnaire-Sergeant [ 03/Apr/14 6:36 AM ]

I was responding that (HSequential [a b c *]) (CountRange 1 5)) -> (HSequential [a b c c c])
is unsound, and should be (U (HSequential [a b]) (HSequential [a b c]) (HSequential [a b c c]) (HSequential [a b c c c]).

Comment by Ambrose Bonnaire-Sergeant [ 03/Apr/14 6:43 AM ]

Perhaps a better idea is to add an extra field in HSequential/HList etc. for a CountRange. Then (HSequential [a b c *]) (CountRange 1 5)) -> (HSequential [a b c *] :count (CountRange 1 5))

That way there's no chance of generating massive types, and we can utilise that information in subtyping to simplify the HSequential at the last minute.

Comment by Di Xu [ 03/Apr/14 9:59 AM ]

Wow, that's a good idea. So when (> (:lower c) (count (:types t))) we should extends the types with (repeat (:rest t)), but I don't know what to do with drest. I will mark it unsupport.

There're a problem makes me complete lost. With hsequential-demo.diff patch, after changing the r/-hsequential into r/-hvec there'll be no problem testing

lein test :only clojure.core.typed.test.core/annotate-user-defined-ploydot

but after changing back to r/-hsequential, there'll be a type error. I don't know what I'm missing, I've debugged it for a whole day and still feel lost. I've greped the code that support HVec in fold_default.clj, frees.clj, promote_demote.clj and subst.clj. And copy the code to implement HSequential, but still got type error. I don't know what's going on here.

ps. the bug you mentioned in the first comment is not a bug, I was going to test if t is HSequential or not, and upcast s to HSequential if so, just like the {HVec,Hlist,HSeq}->HSequential you menntioned.

Comment by Ambrose Bonnaire-Sergeant [ 03/Apr/14 1:18 PM ]

I found a bunch of (pre-existing) bugs and added a few cases you were missing. I started getting the same type error for HVec/HSequential; hopefully the attached patch will help you continue.

Comment by Di Xu [ 07/Apr/14 3:02 AM ]

Well, turns out we don't need CountRange to pass the test case, but we also can't pass it with `first`, because both function and arguments have free variable, so change the test case to concrete type.

Comment by Ambrose Bonnaire-Sergeant [ 07/Apr/14 5:07 AM ]

The rest arguments of a function are actually ISeq's, not just Sequentials. We should be calling -hseq instead of -hsequential in the :fn check method.

If you `mvn test` you'll see what this breaks.

Next step:

  • add rest/drest to HSeq (and HList if you're enthusiastic)
  • change -hsequential to -hseq
Comment by Di Xu [ 08/Apr/14 9:06 AM ]

added, but failed in core/mapentry-first-test, core/hvec-ops and core/variable-hvec-test

I think those cases is beyond my current ability, it involve infer type variable in HSequential.

also failed in clojure.core.typed.test.fail.recur-non-seq-rest and clojure.core.typed.test.fail.recur-empty-seq

I don't know why should these cases fail. And confused by check-recur, which requires rest-arg-type has at least one element.

Comment by Ambrose Bonnaire-Sergeant [ 08/Apr/14 9:44 AM ]

Thanks, having a look.

You've added a few subtyping cases that are unsound. I'll fix them in my next patch.

It's important that (subtype? S T) only returns true if we can insert S in all places T is accepted.

(subtype? (HVec [1]) (HSequential [1]))
;=> true

(subtype? (HSequential [1]) (HVec [1]))
;=> false

(subtype? (HList [1] (HSeq [1]))
;=> true

(subtype? (HSeq [1]) (HList [1]))
;=> false

Comment by Ambrose Bonnaire-Sergeant [ 08/Apr/14 10:39 AM ]

Added ambrose-hsequential-v3-1.patch

Should pass all the tests. Please try it out.

Comment by Di Xu [ 09/Apr/14 2:27 AM ]

Oh, right.. it's unsound to do that in subtype, I was foolish that moment, sorry.

It's seems that you solved (first [1 'a]) by

(and (r/RClass? S)
     ((some-fn r/HeterogeneousVector? r/HSequential?) T))
         (if-let [[Sv] (seq
                         (filter (some-fn r/HeterogeneousVector? r/HSequential?)
                                 (map c/fully-resolve-type (c/RClass-supers* S))))]

right?

Well, I think we could close this ticket now. It spend such long time

Comment by Ambrose Bonnaire-Sergeant [ 09/Apr/14 2:40 AM ]

Actually I believe I added some extra cases for (cs-gen HVec HSeq) to fix (first [1 'a]).

That particular addition fixed (cf (-> {:a 1} first second) Number), because AMapEntry has a HVec ancestor and we used to rely on `second` having a HVec arity. Now `second` has a HSequential arity, we want to trigger this case if there's a (cs-gen RClass HSequential), instead of just (cs-gen RClass HVec).

Fantastic work, I'll merge this soon.

Comment by Ambrose Bonnaire-Sergeant [ 10/Apr/14 10:47 AM ]

Merged.





[CTYP-163] Corrections to a few type signatures for clojure.core functions Created: 19/Jul/14  Updated: 20/Jul/14  Resolved: 20/Jul/14

Status: Resolved
Project: core.typed
Component/s: None
Affects Version/s: None
Fix Version/s: None

Type: Enhancement Priority: Trivial
Reporter: Andy Fingerhut Assignee: Ambrose Bonnaire-Sergeant
Resolution: Completed Votes: 0
Labels: None

Attachments: File ctyp-163-v1.diff    
Patch: Code

 Description   

Was reading through the clojure.core type signatures, and noticed that read-line and printf were incorrect, and format was in there twice. Small patch to correct these things attached.



 Comments   
Comment by Ambrose Bonnaire-Sergeant [ 20/Jul/14 3:59 AM ]

Thanks merged https://github.com/clojure/core.typed/commit/87c5812ddd1ec3f336a9e53c9df53f436ec48cc6

read-line also returns nil according to Javadoc, fixed that myself https://github.com/clojure/core.typed/commit/e2eacf39bd494b026cbaccad3e9531cdce4b3332





[CTYP-128] Trim down the AOT jar Created: 28/Mar/14  Updated: 20/Jul/14  Resolved: 20/Jul/14

Status: Resolved
Project: core.typed
Component/s: None
Affects Version/s: None
Fix Version/s: None

Type: Task Priority: Trivial
Reporter: Ambrose Bonnaire-Sergeant Assignee: Ambrose Bonnaire-Sergeant
Resolution: Declined Votes: 0
Labels: None


 Description   

Started some work here https://github.com/clojure/core.typed/commit/d5c1d23f15301100e36f0290df43ad09e4b377d1

All tests started failing.



 Comments   
Comment by Ambrose Bonnaire-Sergeant [ 20/Jul/14 5:54 AM ]

Clojure doesn't make this easy by any means.





[CTYP-164] core.typed: Enhance annotation for clojure.string/blank? Created: 28/Jul/14  Updated: 28/Jul/14  Resolved: 28/Jul/14

Status: Resolved
Project: core.typed
Component/s: None
Affects Version/s: None
Fix Version/s: None

Type: Enhancement Priority: Trivial
Reporter: Jochen Hinrichsen Assignee: Unassigned
Resolution: Completed Votes: 0
Labels: enhancement

Patch: Code

 Description   

The latest core.typed release 0.2.64-SNAPSHOT contains clojure.string/blank [String -> boolean], this should read clojure.string/blank [(U nil String) -> String].

-clojure.string/blank? [String -> Boolean]
+clojure.string/blank? [(U nil String) -> Boolean]

Pull request available: https://github.com/clojure/core.typed/pull/7/files



 Comments   
Comment by Ambrose Bonnaire-Sergeant [ 28/Jul/14 7:08 AM ]

Added myself, thanks!





[CTYP-162] A few annotations were added for clojure.string namespace. Created: 18/Jul/14  Updated: 06/Aug/14  Resolved: 06/Aug/14

Status: Resolved
Project: core.typed
Component/s: None
Affects Version/s: None
Fix Version/s: None

Type: Enhancement Priority: Trivial
Reporter: Aravind K N Assignee: Ambrose Bonnaire-Sergeant
Resolution: Completed Votes: 0
Labels: None

Attachments: File string.diff     File string_updated.diff    
Patch: Code and Test

 Description   

Typed annotations were added for blank?,capitalize,lower-case,replace,reverse and trim.



 Comments   
Comment by Andy Fingerhut [ 19/Jul/14 11:16 AM ]

Aravind, triml and trimr have the same type as trim, and should be easy to add.

clojure.string/replace has a more complex type than [String -> String]. Take a look at the documentation string to get a better idea of what it does. clojure.string/replace-first has the same type signature, so once you get one right you can copy it to the other.

Comment by Aravind K N [ 24/Jul/14 9:32 AM ]

Added annotations for replace-first, triml,trimr and altered the annotation for replace.

Comment by Aravind K N [ 06/Aug/14 8:25 AM ]

This is cleared.
Could someone change it to fixed?

Comment by Ambrose Bonnaire-Sergeant [ 06/Aug/14 9:11 AM ]

Merged, thanks all!





Generated at Thu Oct 23 06:15:20 CDT 2014 using JIRA 4.4#649-r158309.