Versions Compared

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.

Need different build artifacts:

  • production: lean and mean
    • no source code
    • (eventually?) reduced metadata
  • development
    • source code 
  • batteries included
    • includes latest core-approved version of all contribs
    • easy way to distinguish level of maturity
      • Clojure and a few others: 1.x or later
      • many things < 1.0
    • does batteries included need both production and development versions?
      • guessing development only would be ok

Thoughts on build profiles (SS 2011-03-17)

There are two aspects, each with two possible approaches:

  • Packaging aspect: what goes in the JAR?
    • "before" approach: modules
      • Split "Clojure" into smaller modules
        • E.g., "data structures", "runtime support", "compiler"
        • All can live in the same Git repository
      • Build and release each module on its own
      • Release "Clojure" as the combination of all of them
      • Users can combine them in different ways
        • E.g., an Android app can omit the compiler
      • Requires significant source-code reorganization
    • "after" approach: multiple build artifacts
      • Build "Clojure" as one thing
      • Filter different subsets of .class files into different JARs
      • Leaves current source-code structure in place
      • Trickier to get right
        • E.g., ensuring no missing dependencies among compiled artifacts
  • Compiling aspect: what goes in each .class file?
    • "before" approach: compiler flags
      • Compile-time flags to include/omit certain things
        • Set once for an entire build? Or per-namespace?
      • Raises question: what is the interface to the Clojure compiler?
        • clojure.lang.Compile is a command-line interface
        • clojure.core/compile is the Clojure interface
        • For better tooling we need an official Java interface
    • "after" approach: filtering
      • Process compiler's output to remove things like metadata
      • How would we do this?

Problem

Clojure today exists at a balance point between performance and dynamic flexibility. Some users would be willing to sacrifice dynamic features to gain more performance. Other users would be willing to sacrifice performance to gain better development error messages or debugging. The problem we aim to solve is to alter the Clojure build such that we can produce alternate versions of Clojure that are useful to smaller audiences while retaining Clojure as it is today.

Example builds we may want to produce (examples):

  • Direct linking
    • Off - allows redefs, good for interactive dev tooling or modifying Clojure internals
    • On - better performance and optimization for long-running more static programs
  • Lazy vars
    • On - faster bootstrap for scripts or dev reloading
    • Off - faster var invocation for long-running programs
  • Metadata (:doc, :file, :line, :added) - removing metadata makes .class files smaller and faster to load, but removes metadata for doc
    • Full - elide nothing
    • Minimal - elide #{:doc, :file, :line, :added}
  • Dev assistance
    • Validation and error messages that can guide development at the expense of runtime performance

Build profiles

Below are some proposed build variants.

ProfileDescriptionCompiled?Direct linking?Lazy vars?MetadataDev assistance?
devFor developmentYesOffOnFullOn
dynamicFor dynamic production useYesOffOffMinimalOff
staticFor static production useYesOnOffMinimalOff
slimSource onlyNon/an/an/an/a
1.8 clojure jarFor comparisonYesOnOffFullOff

OPEN: Are these the right profiles and the right settings?

OPEN: Which one is the default no-classifier?

Constraints 

Clojure users:

  • Build profile artifacts should be available in public Maven repositories for use with existing build systems (Leiningen, Maven, Boot), alternatives include:
    • Different artifacts:  [org.clojure/clojure-foo "1.8.0"]
    • Different qualifiers: [org.clojure/clojure "1.8.0-foo"]
    • Different classifiers: [org.clojure/clojure "1.8.0" :classifier "foo"] - seems like the best choice
  • Some features may require further compiler flags or metadata to activate even if using the specialized build
    • For example, you may need to supply special var metadata on a function to get access to a feature
    • Ideally, use of the special feature should be simply ignored if not using the specialized build (or a warning should direct you)

Build:

  • Produce multiple output jars each time we build a version (snapshot or release)
  • The jars will contain different versions of the classes
  • Specialized build flags or env variables will need to be set during the build for each variant
  • Testing should occur for each variant
Integration / release infrastructure:
  • CI must be able to build and test custom builds as part of the dependency matrix
  • CI must be able to perform releases for all of the variants
  • CI must be able to test contribs against custom builds

Compiler:

  • The compiler will need a standard way to determine which variant is active to make decisions during compilation

External builds

Builds that use Clojure may also need to make these kinds of compiler choices to produce their own build profiles. 

It would be useful to produce a guide on how to enable combinations of compiler options to produce builds of different flavors for libraries with the common build tools.

Implementation options

Hardest part of this is getting the build to sign and release all of the artifacts together.

Full build would consist of:

  • javac Java source
  • jar slim jar (without class files)
  • filter resources - update version.properties
  • repeat for each variant
    • aot compile Clojure source (using Compiler we just built)
    • jar compiled clojure jar (with class files)
    • javac test Java
    • aot compile test Clojure
    • run clojure.test runner 
    • run generative test runner
  • jar source files
  • javadoc generate
  • jar javadoc files
  • zip package of full distribution
  • install to local repo
  • deploy
    • sign all artifacts
    • push all artifacts to Maven central
    • git tag release
    • update version to next SNAPSHOT on prod deploy

OPEN: which implementation would be best? 

Maven custom assembly

  • The Maven assembly plugin allows for arbitrary customization of the artifacts created by the build. However, creating multiple artifacts with different Maven coordinates will likely require multiple builds to produce each artifact for deployment. This has issues with signing and release.
  • How-to in Maven
    • http://stackoverflow.com/questions/3092085/building-same-project-in-maven-with-different-artifactid-based-on-jdk-used
    • Create one Maven profile per output artifact, in each profile:
      • Set a property that can be used as the <classifier> in the maven-jar-plugin
      • Set <activation><active-by-default> to false
      • Set system property compiler flags that need to be active during Clojure compilation for the foo variant
      • Turn off generation of the -sources and -javadoc classifier artifacts in these variant builds
    • Run the release once per profile - this can be automated in Maven with a matrix build
    • Much more difficult to produce N variants as part of a single build (in a way that works with CI well)

Boot

  • Boot provides a great deal of customization capability and might also be a good option for this level of customization
  • Unknown: signed deployment to Maven central repos? Yes: https://github.com/boot-clj/boot/blob/master/doc/boot.task.built-in.md#push
    • Need to figure out how to load encrypted creds on CI server - might require some hackery
  • In theory, this seems feasible with existing plugins, unknown what complications would arise

Custom script (ant, make, etc)

  • Would prefer something with greater familiarity to Clojure devs as patch developers need to run the build too
  • How-to in Ant
    • Define property that can be set to activate a particular variant
    • Set that property when running the build

Coordinate Research

Maven coordinate options

  • Different artifactId
    • Releasing different artifacts would probably require running the build multiple times - there are probably some complexities in the CI and release process as a result.
    • NOT recommended
  • Different qualifier
    • Qualifiers are really designed to give you more information about the purpose of a released version (alpha, beta, GA, build number), so this would be departing from standard Maven expectations.
    • NOT recommended
  • Different classifier
    • Classifiers are designed for this use case - they allow you to create variants of the same group/artifact/version specialized for a particular environment (JDK variants is one example given)
    • Lein and Maven can both specify a classifier - in lein with [org.clojure/clojure "1.6.0" :classifier "foo"]
    • It seems good that the main build and variant share Maven co-ords - there is then a standard way to get from main artifact to (for example) the dev variant for tooling.