Versions Compared

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.

Goals

We need a convenient, uniform way to build Clojure and contrib projects. Problems are

  • Team will work across many projects and will not want to lean a different approach to building each one.
  • Projects will need to work with a variety of tools: Hudson, leiningen, maven, and IDEs at a minimum. (Note that these are to some degree overlapping.)
  • Users will want an easy way to find released versions.
  • Users will want sufficiently fine granularity of releases to get things they need, without a ton of stuff they don't.
  • Team will need automation: one step to trigger an entire push/build/release cycle when necessary.
  • Clojure/core will probably want to package multiple libs into easy-to-consume distributions.

Near-term Process

  • Clojure and all contrib projects will be built using Maven 2
  • All POMs will inherit from pom.baseline or pom.contrib
  • pom.contrib includes support for specifying a local clojure.jar
  • Only Hudson will deploy releases or SNAPSHOTs
  • All builds must be configured for JDK 1.5 in Hudson
  • Stable and SNAPSHOT releases will be deployed to Sonatype's OSS repositories
  • Stable releases will be initiated by Hudson (the "Perform Maven Release" link)
  • Contrib projects will produce source-only JARs whenever possible
  • ZIP distributions will be generated by Maven 2 and manually uploaded to clojure.org

Notes

  • In "classic" clojure-contrib, only 4 modules require AOT-compilation: condition, fnmap, jmx, repl-ln
    • All other clojure-contrib modules are building source-only JARs

The inheritance diagram of POMs:

Image Removed

Benefits

  • Common build configuration across org.clojure projects
  • Ensure released JARs are built with JDK 1.5
  • SNAPSHOT releases automatically uploaded to public OSS repositories
  • Stable releases automatically staged for promotion into Maven Central repository
  • Better continuous integration?

Release Process

"Releasing" Snapshots

Deployment of SNAPSHOTs to the OSS repo is entirely straightforward:

  • SCM changes in the corresponding github repos prompts hudson build(s) automatically,
  • successful SNAPSHOT builds triggers downstream hudson builds, and
  • deployment of SNAPSHOT artifacts to the OSS snapshots repo can be done without any concern for coordination or planning

Releasing Releases

The release process involves some additional steps, and some added process/policy may be desirable.

Right now, the build.poms and tools.nrepl Hudson jobs are configured to support the most straightforward/naïve release process possible:

  1. Click "Perform Maven Release" on the job's main page.
  2. Specify the release and development versions to be provided to maven-release-plugin:
    1. release version The version of the release to be built and deployed to the OSS repo, thereupon to central
    2. development version The version that the project's POM(s) will be updated to in git HEAD via a post-release push.

Suggestions for these are usually provided, based on the last version known to Hudson, e.g. if HEAD has a version of 1.0.0-SNAPSHOT, release and dev versions of 1.0.0 and 1.0.1-SNAPSHOT respectively.

  1. The maven release plugin does a few things worth noting:
    1. in the hudson-local git repo, it:
      1. commits an updated project POM with the release version, tagging that changeset with a corresponding tag
      2. commits another updated project POM with the development version, leaving this changeset as master
    2. it performs a clean clone of the hudson-local git repo, checking out the tagged release changeset
    3. in rough terms, performs an mvn clean deploy, targeting the Sonatype OSS releases repository
  2. The deploy operation will create or add to an org.clojure-specific release staging repository in the OSS Nexus instance.
  3. The naïve part: after the deploy operation completes, we automatically close and promote the staging repository's contents to the OSS releases repo, and therefore to central
  4. Once the mvn invocation completes successfully, a post-maven build step pushes master and the release tag to the upstream github repo

This process may or may not be sufficient.  The uncertainty centers on the interaction with the Nexus release staging repository.  Some thoughts:

  • In contrast to deploying to a snapshots repository (which exists solely to provide a way to refer to potentially- / frequently-changing artifacts with a stable identifier), once an artifact gets into central, it can never be changed or removed.  Thus, there is a desire to have a final opportunity to vet release artifacts.
    • To be clear: this policy (and all that flows from it) is central's; release repositories in general (Nexus-hosted or not) are not required to be deployed to via staging repos.
  • This is where the staging repos come in: once one is created to contain deployed artifacts, those artifacts can be accessed via the Nexus management site, and subjected to whatever final testing / verification a project's author(s) deem necessary.
  • Apache, among others, runs additional tests on the artifacts deployed to a given closed staging repository, the results of which are used as part of their release voting process.

Given the above, questions include:

  1. Will we want to vet individual release artifacts in any way just prior to their staging repository being promoted to central (similar to e.g. Apache)?
  2. Will we want to attempt to coordinate releases in any way, at the point of staging repository promotion?
    1. This would imply not closing our staging repository automatically as part of our projects' release builds.
    2. Once we've deployed whichever projects we aim to release in a coordinated fashion to the staging repository, we then manually close that repo.
    3. After verifying that the batch of staged artifacts pass our final criteria, we then manually promote their staging repo.
    4. If those artifacts are not acceptable, we can then drop the staging repository, and revisit whatever issues were exposed in our final testing.
      1. Note that this would require us backing out the versioning changes in the projects' POM(s) in SCM (again, presumably a manual operation) without stomping on any substantive patches applied in the interim.

I've written before about how coordinated releases should be off the table; the dynamics of staging repository closing and promotion would seem to make that prospect even less attractive, if only because of the serious pain that would result if we were to back out committed and pushed versioning changes (the automatic handling of which is a no-brainer IMO).

Vetting individual release artifacts would seem to be a more feasible option, if so desired (at least the pain of backing out project version changes would be localized to the project with the problematic release).

However, in both cases, I'd suggest that having multiple alpha and RC releases should eliminate all technical reasons for vetting of staged release artifacts.  At the very least, I'd say that it's worth trying to use the naïve approach through one full release cycle, if only because it's the simplest fully-automated solution available.

- Chas

Possible Future Development

...

The situation

  • Multi-module clojure-contrib in one repository has not worked well
    • Build is complicated and slow
    • Still hard for authors to work on their libs
  • We will have more projects under the org.clojure umbrella
    • New contrib libraries: nrepl, unify, finger-tree
    • More on the way
  • Current Clojure release process involves manual steps

The goals, in order of priority

  • For users
    • 1. Find release versions
    • 2. IDE / tool support
    • 3. Get just the stuff they need, without stuff they don't
    • 4. Easy-to-use distributions containing multiple libraries known to work together
  • For developers
    • 1. Build / release automation
    • 2. Unified build / release process across projects
    • 3. IDE / tool support

The plan

Clojure, the language

  • patch CLJ-681has been applied
    • Keeps the Ant build the way it is now
      • Developers can continue to use Ant for local builds
    • Maven controls the build/test/release process on Hudson
      • Real pom.xml
      • Declares Sonatype's oss-parent POM as a parent
        • Enables releases to public open-source repositories managed by Sonatype
      • Calls out to Ant for Clojure-specific compile and test phases
  • SNAPSHOT releases automatically deployed by hudson
  • Numbered releases controlled by Hudson/Maven
    • Only Clojure language committers may initiate a release

New contrib projects

  • To promote new contrib projects, see Guidelines for Clojure Contrib committers
  • To release, see How to Make Releases
  • Git repositories
    • New contrib libs get their own repos under the Clojure github organization
    • Each lib gets one or more "owners" who have commit access
    • Library authors still use JIRA + patches to accept contributions
    • All contributors must have a CA
  • JIRA
    • Each contrib library/project will get its own JIRA
  • Maven
    • Contrib projects must be built with Maven 2
    • All contrib POMs must declare org.clojure:pom.contribas a parent
      • Inherits from Sonatype's OSS POM
      • Defines release process (OSS snapshots + staging)
      • Set dependency on latest Clojure release (1.3.0-alpha5)
        • May be overridden on a per-project basis
        • Support building with a local Clojure JAR
      • Inherits from pom.oss-deploy
      • Sets license to EPL
      • Configures Java 1.5 as a target
      • Sets common plugin/build configurations
      • Uses com.theoryinpractise:clojure-maven-plugin
      • Defaults to source-only JAR, override to AOT-compile only when necessary

"Classic" / old clojure-contrib

  • The old monolithic build structure is deprecated
  • The interim multi-module build structure is deprecated
    • Module releases up to 1.3.0-alpha4 are available in the build.clojure.org release repository
      • These are source-only JARs wherever possible
      • They should be compatible with any version of Clojure
    • The multi-module build does not work with Clojure 1.3.0-alpha5 and later
  • Libraries with community demand and willing maintainers will become "new" contrib projects as above
  • The "classic" clojure-contrib JIRAis deprecated
    • Important issues should be reposted in the new per-project JIRAs

Work to be done

  • DONE Clean up / finalize pom.contrib
    • Replace 3-level inheritance (pom.oss-deploy -> pom.baseline -> pom.contrib) with just pom.contrib
  • DONE Finalize contrib project Hudson config
  • DONE Figure out how to give Hudson permissions to contrib authors (Stuart Sierra & Aaron Bedra)
  • DONE Create new Hudson job for finger-tree
  • DONE Figure out how to manage many similar Hudson configurations (Chas Emerick & Stuart Sierra)
  • TODO verify Sonatype release goals do the right thing in the presence of multiple staging repositories
  • TODO Get older releases into Maven Central?
  • TODO Redirect repositories at build.clojure.org to OSS Sonatype
    • Assuming we roll all existing releases into central, we only need to set up redirects for /snapshots

Possible future development

  • Improve the Maven build process; see Clojure Maven Plugin
  • Better Clojure-from-Java interface (see CLJ-452)
  • Aggregate projects that package many contrib libraries in one distribution
  • Testing harness to build/test many Clojure libraries with a locally-built Clojure JARRedirect build.
  • clojure.org/snapshots and /releases to Central or Sonatype repository Assuming we roll all existing releases into central, we only need to set up redirects for /snapshotsMaking the build process better
    • Improve clojure.lang.Compile
      • Investigate why compilation sometimes pauses for a long time (minutes) at the end
      • Investigate if core needs explicit compilation order
    • Investigate Maven 3
    • Investigate Polyglot Maven
    • Documentation: how do I do X, Y, or Z?
    • Write more plugins
    • Developing other non-Maven tools
      • Will require a lot of work to match what Maven can do
      • Not a Clojure/core priority