core.typed

Checking an ns becomes significantly slower as the number of optional keys in a HMap increases

Details

  • Type: Defect Defect
  • Status: Resolved Resolved
  • Priority: Blocker Blocker
  • Resolution: Completed
  • Affects Version/s: None
  • Fix Version/s: None
  • Component/s: None
  • Labels:
    None

Description

I'm using clojure.core.typed 0.2.19 with the slim classifier but I've observed the same without slim.

My suspicion is that the latent filters associated with functions grow in size exponentially with each extra optional key to a HMap (based on the output when you have a type error). I think it's generating all combinations of present and absent keys for the HMap when calculating latent filters for a function.

I've attached a tarball with a lein project with ten namespaces that all contain the same ten simple functions in the form

(t/ann f [(HMap :optional {:a String}) -> (t/Option String)])
(defn f [m]
  (:a m))

The type annotations vary in the number of optional keywords.

(test-hmap.core/go) checks all the namespaces. The time to check each namespace grows non-linearly. The first namespace gets penalised by core.typed initialisation the first time it's run.

E.g. on my local machine:

test-hmap.one
Start collecting test-hmap.one
Finished collecting test-hmap.one
Collected 1 namespaces in 20.743 msecs
Start checking test-hmap.one
Checked test-hmap.one in 46.231 msecs
Checked 1 namespaces (approx. 34 lines) in 67.672 msecs


test-hmap.two
Start collecting test-hmap.two
Finished collecting test-hmap.two
Collected 1 namespaces in 18.751 msecs
Start checking test-hmap.two
Checked test-hmap.two in 45.525 msecs
Checked 1 namespaces (approx. 35 lines) in 64.814 msecs


test-hmap.three
Start collecting test-hmap.three
Finished collecting test-hmap.three
Collected 1 namespaces in 19.106 msecs
Start checking test-hmap.three
Checked test-hmap.three in 57.346 msecs
Checked 1 namespaces (approx. 35 lines) in 77.055 msecs


test-hmap.four
Start collecting test-hmap.four
Finished collecting test-hmap.four
Collected 1 namespaces in 21.222 msecs
Start checking test-hmap.four
Checked test-hmap.four in 51.339 msecs
Checked 1 namespaces (approx. 35 lines) in 73.155 msecs


test-hmap.five
Start collecting test-hmap.five
Finished collecting test-hmap.five
Collected 1 namespaces in 22.215 msecs
Start checking test-hmap.five
Checked test-hmap.five in 63.415 msecs
Checked 1 namespaces (approx. 35 lines) in 86.309 msecs


test-hmap.six
Start collecting test-hmap.six
Finished collecting test-hmap.six
Collected 1 namespaces in 23.76 msecs
Start checking test-hmap.six
Checked test-hmap.six in 99.407 msecs
Checked 1 namespaces (approx. 35 lines) in 123.881 msecs


test-hmap.seven
Start collecting test-hmap.seven
Finished collecting test-hmap.seven
Collected 1 namespaces in 26.519 msecs
Start checking test-hmap.seven
Checked test-hmap.seven in 213.515 msecs
Checked 1 namespaces (approx. 35 lines) in 240.639 msecs


test-hmap.eight
Start collecting test-hmap.eight
Finished collecting test-hmap.eight
Collected 1 namespaces in 32.581 msecs
Start checking test-hmap.eight
Checked test-hmap.eight in 626.76 msecs
Checked 1 namespaces (approx. 35 lines) in 659.975 msecs


test-hmap.nine
Start collecting test-hmap.nine
Finished collecting test-hmap.nine
Collected 1 namespaces in 43.478 msecs
Start checking test-hmap.nine
Checked test-hmap.nine in 2634.468 msecs
Checked 1 namespaces (approx. 35 lines) in 2678.716 msecs


test-hmap.ten
Start collecting test-hmap.ten
Finished collecting test-hmap.ten
Collected 1 namespaces in 84.394 msecs
Start checking test-hmap.ten
Checked test-hmap.ten in 9277.54 msecs
Checked 1 namespaces (approx. 35 lines) in 9362.613 msecs
  1. core.clj
    26/Jan/14 1:09 PM
    2 kB
    Cees van Kemenade
  2. core.clj
    26/Jan/14 1:04 PM
    7 kB
    Cees van Kemenade
  3. project.clj
    26/Jan/14 1:09 PM
    0.4 kB
    Cees van Kemenade
  4. test-hmap.tar.gz
    03/Dec/13 12:32 PM
    7 kB
    Gordon Syme

Activity

Cees van Kemenade made changes -
Field Original Value New Value
Attachment core.clj [ 12721 ]
Cees van Kemenade made changes -
Attachment project.clj [ 12723 ]
Attachment core.clj [ 12722 ]
Cees van Kemenade made changes -
Comment [ Project and file with one-lines (excluding type definitions) that kills (check-ns).

I hope this one-liner helps making the issues easy to reproduce.
Currently I don't know a work-around for this issue,
so I can not use core.typed to check my project.

I guess that many people/projects should run into the same issues,
as the pattern is returning quite often in clojure:
   - start with data-set with small Hashmaps
   - the hashmaps evolve in a number steps (meaning more keys are added and some of the existing keys are removed)
I guess the workaround would be to use limited or no optional keys, and prepare a custom HMap definition for each stage of analysis. Although this would work, the amount of bookkeeping already significant for a all records are running through the same stages (linear process), and will be a showstopper if it is a non-linear process (not all records follow the same path (sequence of analysis stages)).


]
Ambrose Bonnaire-Sergeant made changes -
Priority Major [ 3 ] Blocker [ 1 ]
Ambrose Bonnaire-Sergeant made changes -
Resolution Completed [ 1 ]
Status Open [ 1 ] Resolved [ 5 ]

People

Vote (1)
Watch (1)

Dates

  • Created:
    Updated:
    Resolved: