[CTYP-99] Checking an ns becomes significantly slower as the number of optional keys in a HMap increases Created: 03/Dec/13 Updated: 14/Feb/14 Resolved: 14/Feb/14
|Reporter:||Gordon Syme||Assignee:||Ambrose Bonnaire-Sergeant|
|Attachments:||core.clj core.clj project.clj test-hmap.tar.gz|
I'm using clojure.core.typed 0.2.19 with the slim classifier but I've observed the same without slim.
My suspicion is that the latent filters associated with functions grow in size exponentially with each extra optional key to a HMap (based on the output when you have a type error). I think it's generating all combinations of present and absent keys for the HMap when calculating latent filters for a function.
I've attached a tarball with a lein project with ten namespaces that all contain the same ten simple functions in the form
The type annotations vary in the number of optional keywords.
(test-hmap.core/go) checks all the namespaces. The time to check each namespace grows non-linearly. The first namespace gets penalised by core.typed initialisation the first time it's run.
E.g. on my local machine:
|Comment by Ambrose Bonnaire-Sergeant [ 03/Dec/13 12:50 PM ]|
Yes, the primitives are :mandatory, :absent-keys and :complete; :optional expands to be in terms of those.
Thanks for the report, I haven't done performance testing on this strategy. It will probably need to be reconsidered.
|Comment by Cees van Kemenade [ 26/Jan/14 1:04 PM ]|
code of project ctcrash
|Comment by Cees van Kemenade [ 26/Jan/14 1:05 PM ]|
+1 For this issue.
I've spend quite some time to learn core.typed on a real use case, and I have to say that I'm amazed by the power of the analysis and the potential to prevent errors that otherwise would be detected to late! Apart from that the type-annotations are valuable and precise information about the interface when doing maintenance at code you did not see for some time.
When I reduce the number of optional keys to 4-5 the (check-ns) runs in a few seconds. But having 9-11 optional keys kills the (check-ns) process.
I hope this one-liner helps making the issues easy to reproduce.
I guess that many people/projects should run into the same issues,
1. start with data-set with small Hashmaps
|Comment by Ambrose Bonnaire-Sergeant [ 26/Jan/14 7:51 PM ]|
Upgraded to 'Blocker'.
|Comment by Ambrose Bonnaire-Sergeant [ 27/Jan/14 6:34 AM ]|
This is theoretically trivial to implement, and just involves shuffling around where optional keys are handled. However, this is spread out all over the code base, so I may not complete the patch for a week.
Great to hear core.typed is working out for you!
|Comment by Cees van Kemenade [ 27/Jan/14 6:44 AM ]|
Thanks for picking up this issue fast.
|Comment by Ambrose Bonnaire-Sergeant [ 14/Feb/14 2:26 AM ]|
This is fixed in 0.2.31.
I've added both of your tests in the test suite - thanks! https://github.com/typedclojure/core.typed-test-suite/tree/master/src/hmap
Please confirm and I'll close the issue.
|Comment by Cees van Kemenade [ 14/Feb/14 4:11 AM ]|
This resolve the issue and also resolves CT-102.
The code in hmap/big-options.clj (from your test-suite) still contains a workaround to make core.typed proceed without errors.
Is this inference error already on the list, or would you recommend posting a seperate JIRA-case for this issue?
|Comment by Gordon Syme [ 14/Feb/14 5:56 AM ]|
Our core.typed checks run an order of magnitude quicker now (~400s --> ~80s). And they keep that speed with all the extra optionals now. That's really awesome, thanks Ambrose!