Clojure

Loading core specs affects startup time

Details

  • Type: Enhancement Enhancement
  • Status: Closed Closed
  • Priority: Major Major
  • Resolution: Completed
  • Affects Version/s: Release 1.9
  • Fix Version/s: Release 1.9
  • Component/s: None
  • Labels:
  • Patch:
    Code
  • Approval:
    Ok

Description

Adding the loading of spec itself + the clojure.core.specs namespace containing specs for core makes start time worse (and will only get longer as we add more core specs).

$ export REPO=$HOME/.m2/repository
$ time java -cp $REPO/org/clojure/clojure/1.8.0/clojure-1.8.0.jar clojure.main -e nil
real	0m0.842s

$ time java -cp $REPO/org/clojure/clojure/1.9.0-alpha19/clojure-1.9.0-alpha19.jar:$REPO/org/clojure/spec.alpha/0.1.123/spec.alpha-0.1.123.jar:$REPO/org/clojure/core.specs.alpha/0.1.24/core.specs.alpha-0.1.24.jar clojure.main -e nil
real	0m1.286s

Proposed: Turn off macro spec checking during RT initialization. On first macroexpand after initialization, lazily load clojure.spec.alpha and clojure.core.specs.alpha.

Patch details:

  • RT - add flag CHECK_SPECS, initialized to false. Set flag after init is done.
  • RT.doInit() - Don't preemptively load clojure.spec.alpha or clojure.core.specs.alpha.
  • Compiler.macroexpand1() - when macroexpanding, call checkSpecs()
  • Compiler.checkSpecs() - if RT.CHECK_SPECS and not MACRO_CHECK_LOADING, then get cached var for clojure.spec.alpha/macroexpand-check and invoke it with the form to check the spec
  • Compiler.ensureMacroCheck() - if MACRO_CHECK (the cached var) is null, then enter the loading block and check that again (double-checked locking). Set the MACRO_CHECK_LOADING flag while loading (this prevents reentrant checking during loading - where spec itself uses macros that can trigger this code). Load clojure.spec.alpha and clojure.core.specs.alpha. Set the cached var.
  • Compile - used in the Clojure build process. The problem that can occur during building is that the lazy load of clojure.core.specs.alpha causes it to get compiled (CLJ-322) and this leads to class loading problems later in the test phase (via CLJ-1544 ish stuff). So instead, eagerly load clojure.core.specs.alpha before compilation starts. Maybe this should happen in compile or elsewhere instead?
  • clojure.main - binds clojure.spec.alpha/explain-out, which requires an explicit load of clojure.spec.alpha - this was previously relying on the implicit load in RT. This has its own performance implications during startup, but can address that separately.

With just this patch, the impact of avoiding the load of clojure.core.specs.alpha can be seen:

$ time java -cp $REPO/org/clojure/clojure/1.9.0-master-SNAPSHOT/clojure-1.9.0-master-SNAPSHOT.jar:$REPO/org/clojure/spec.alpha/0.1.123/spec.alpha-0.1.123.jar:$REPO/org/clojure/core.specs.alpha/0.1.24/core.specs.alpha-0.1.24.jar clojure.main -e nil
real	0m0.990s

However, additional changes are required to avoid loading spec.alpha entirely - that relies on other tickets:

  • CLJ-1891 - clojure.core.server is being unnecessarily loaded even when no servers are specified (this is due to changes in 1.8) which loads clojure.main, which loads clojure.spec.alpha
  • TODO - clojure.main only loads spec.alpha to bind clojure.spec.alpha/explain-out - there are a couple ways to potentially remove this linkage

Applying CLJ-1891 and removing the load of spec.alpha in clojure.main gave me about 0m0.830s for the call above for comparison.

Patch: clj-2108.patch

Activity

Hide
Ghadi Shayban added a comment -

Locally,
alpha14 takes 1.02s

time java -jar clojure-1.9.0-alpha14.jar -e ':foo'

Master with this line commented out takes 0.92s

time java -jar target/clojure-1.9.0-master-SNAPSHOT.jar -e ':foo'
Show
Ghadi Shayban added a comment - Locally, alpha14 takes 1.02s
time java -jar clojure-1.9.0-alpha14.jar -e ':foo'
Master with this line commented out takes 0.92s
time java -jar target/clojure-1.9.0-master-SNAPSHOT.jar -e ':foo'

People

Vote (0)
Watch (3)

Dates

  • Created:
    Updated:
    Resolved: