Versions Compared

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.

We need a convenient, uniform way to build contrib projects. Problems are

  • Team will work across many projects and will not want to lean a different approach to building each one.
  • Projects will need to work with a variety of tools: Hudson, leiningen, maven, and IDEs at a minimum. (Note that these are to some degree overlapping.)
  • Users will want an easy way to find released versions.
  • Users will want sufficiently fine granularity of releases to get things they need, without a ton of stuff they don't.
  • Team will need automation: one step to trigger an entire push/build/release cycle when necessary.
  • Clojure/core will probably want to package multiple libs into easy-to-consume distributions.

Here is a possible direction. Please shred this and come up with something better!

  • Treat the pom.xml as the data of record. All the other tools listed above are capable of producing/consuming this as necessary.
  • Register all projects with build.clojure.org to get CI and snapshot builds.
  • Configure build.clojure.org to cut releases when it sees a commit with version metadata. 
    • As an example, the clojure build currently uses boolean flag name "interim", when this is false, Clojure could cut a release.
    • Should automate both release and follow-on commit that resets the "interim" flag.
  • Document everything and verify that a new contrib author can get started easily.

Issues

  • Is the build box a good maven repos, or should something different be used for releases?
  • Does whatever solution we choose imply any feature requests for leiningen?
  • What about Clojars? The proposal above assumes (I think correctly) that the official Clojure and Contrib builds need to come from a official site that we control, not Clojars.

October 26, 2010 - Brainstorm

SDH notes added in blue below

Overview:

  • Properly formatted pom.xml required
  • Hudson polls master branch on git repo hourly and runs tests.
  • Nightly Snapshots are built at 2:00AM ET
  • Releases are built as requested (see below)

Pom:

  • Must have groupId of org.clojure
  • artifactId is the remaining package name hyphenated (e.g. data.finger-tree is data-finger-tree). Should there be a "clojure" prefix in artifactId, e.g. clojure-data-finger-tree? Could matter if artifactId ever travels alone.
  • Must have version which is the current SNAPSHOT version number. (X.Y.Z-SNAPSHOT where X.Y.Z is greater than last release's version)
  • Must have `test' and `package' goals.
  • This (and any additional requirements as of yet undetermined) will be validated before maven touches the file. Failing validation fails the build. This (and possibly some other items) could be missing in the initial release. So long we requirements are documented, enforcement can always come later.

Versioning:

  • If there exists a git tag for the release (format TBD) a release package is created, pending a passing build. Please respect, if possible, the tagging scheme already used by Clojure.** The version number is the one in pom.xml without -SNAPSHOT.
    • Potential mechanism: gitattributes. We can define a filter to remove -SNAPSHOT from the pom when it is pulled so maven will version correctly.
    • This would be required to build packages in all cases. Otherwise unintended inconsistencies could arise.
  • Otherwise, a snapshot is created based on the version element in pom.xml.
  • Builds must target Java 5 until/unless we ever agree to move the ecosystem to a later version of Java.
  • Builds should prefer source-only distribution of jars, unless there is a requirement for compilation. (We should also keep a list of things forcing compilation on us and try to eliminate them.)

Repository Management:

  • We still need to come up with a way to have hudson place artifact files in /snapshots or /releases accordingly.
  • The current hudson plugin requires a hard-coded URI.
  • Some options:
    • Shell script which does the copy
    • Possibly a better hudson plugin (not yet investigated)
    • (Insert other, more brilliant idea here)
  • We could dodge the issue entirely and not separate snapshots from full releases. There should be no risk of colliding version numbers given a nightly, timestamped snapshot build.
    • This will break some Maven-aware tools, possibly including IDEs

Response to Issues (above):

  • For now, the build box is probably fine. We will probably want to investigate a Nexus solution as load/usage increases. (That may be considerably sooner than this bullet would lead you to believe.)
  • A leiningen task which generates the contrib-specific pom.xml would probably be welcome, but is not required by the process. Unless manually editing XML is unacceptable, in which case it is.
  • Agreed about Clojars, at least in the context of contrib.
  • Consider a pre-release "staging" repository (supported by Nexus Professional)

Open Questions:

  • How exactly we would pre-validate pom.xml files
  • Whether to create and maintain a parent pom.xml file for clojure-contrib and require lib pom.xml files to refer to it
  • Whether leiningen can participate fully in the scheme we're developing. Possible answers include yes, no, and yes with modifications and plugins.

What We're Ruling Out:

  • Inventing / reinventing any tools. There is a consensus among lib authors that using existing tools, while not perfect, is desirable.

The situation

  • Multi-module clojure-contrib in one repository has not worked well
    • Build is complicated and slow
    • Still hard for authors to work on their libs
  • We will have more projects under the org.clojure umbrella
    • New contrib libraries: nrepl, unify, finger-tree
    • More on the way
  • Current Clojure release process involves manual steps

The goals, in order of priority

  • For users
    • 1. Find release versions
    • 2. IDE / tool support
    • 3. Get just the stuff they need, without stuff they don't
    • 4. Easy-to-use distributions containing multiple libraries known to work together
  • For developers
    • 1. Build / release automation
    • 2. Unified build / release process across projects
    • 3. IDE / tool support

The plan

Clojure, the language

  • patch CLJ-681has been applied
    • Keeps the Ant build the way it is now
      • Developers can continue to use Ant for local builds
    • Maven controls the build/test/release process on Hudson
      • Real pom.xml
      • Declares Sonatype's oss-parent POM as a parent
        • Enables releases to public open-source repositories managed by Sonatype
      • Calls out to Ant for Clojure-specific compile and test phases
  • SNAPSHOT releases automatically deployed by hudson
  • Numbered releases controlled by Hudson/Maven
    • Only Clojure language committers may initiate a release

New contrib projects

  • To promote new contrib projects, see Guidelines for Clojure Contrib committers
  • To release, see How to Make Releases
  • Git repositories
    • New contrib libs get their own repos under the Clojure github organization
    • Each lib gets one or more "owners" who have commit access
    • Library authors still use JIRA + patches to accept contributions
    • All contributors must have a CA
  • JIRA
    • Each contrib library/project will get its own JIRA
  • Maven
    • Contrib projects must be built with Maven 2
    • All contrib POMs must declare org.clojure:pom.contribas a parent
      • Inherits from Sonatype's OSS POM
      • Defines release process (OSS snapshots + staging)
      • Set dependency on latest Clojure release (1.3.0-alpha5)
        • May be overridden on a per-project basis
        • Support building with a local Clojure JAR
      • Inherits from pom.oss-deploy
      • Sets license to EPL
      • Configures Java 1.5 as a target
      • Sets common plugin/build configurations
      • Uses com.theoryinpractise:clojure-maven-plugin
      • Defaults to source-only JAR, override to AOT-compile only when necessary

"Classic" / old clojure-contrib

  • The old monolithic build structure is deprecated
  • The interim multi-module build structure is deprecated
    • Module releases up to 1.3.0-alpha4 are available in the build.clojure.org release repository
      • These are source-only JARs wherever possible
      • They should be compatible with any version of Clojure
    • The multi-module build does not work with Clojure 1.3.0-alpha5 and later
  • Libraries with community demand and willing maintainers will become "new" contrib projects as above
  • The "classic" clojure-contrib JIRAis deprecated
    • Important issues should be reposted in the new per-project JIRAs

Work to be done

  • DONE Clean up / finalize pom.contrib
    • Replace 3-level inheritance (pom.oss-deploy -> pom.baseline -> pom.contrib) with just pom.contrib
  • DONE Finalize contrib project Hudson config
  • DONE Figure out how to give Hudson permissions to contrib authors (Stuart Sierra & Aaron Bedra)
  • DONE Create new Hudson job for finger-tree
  • DONE Figure out how to manage many similar Hudson configurations (Chas Emerick & Stuart Sierra)
  • TODO verify Sonatype release goals do the right thing in the presence of multiple staging repositories
  • TODO Get older releases into Maven Central?
  • TODO Redirect repositories at build.clojure.org to OSS Sonatype
    • Assuming we roll all existing releases into central, we only need to set up redirects for /snapshots

Possible future development

  • Improve the Maven build process; see Clojure Maven Plugin
  • Better Clojure-from-Java interface (see CLJ-452)
  • Aggregate projects that package many contrib libraries in one distribution
  • Testing harness to build/test many Clojure libraries with a locally-built Clojure JAR
  • Making the build process better
    • Improve clojure.lang.Compile
      • Investigate why compilation sometimes pauses for a long time (minutes) at the end
      • Investigate if core needs explicit compilation order
    • Investigate Maven 3
    • Investigate Polyglot Maven
    • Documentation: how do I do X, Y, or Z?
    • Write more plugins
    • Developing other non-Maven tools
      • Will require a lot of work to match what Maven can do
      • Not a Clojure/core priority