We need a convenient, uniform way to build Clojure and contrib projects. Problems are
- Team will work across many projects and will not want to lean a different approach to building each one.
- Projects will need to work with a variety of tools: Hudson, leiningen, maven, and IDEs at a minimum. (Note that these are to some degree overlapping.)
- Users will want an easy way to find released versions.
- Users will want sufficiently fine granularity of releases to get things they need, without a ton of stuff they don't.
- Team will need automation: one step to trigger an entire push/build/release cycle when necessary.
- Clojure/core will probably want to package multiple libs into easy-to-consume distributions.
- Clojure and all contrib projects will be built using Maven 2
- All POMs will inherit from pom.baseline or pom.contrib
- pom.contrib includes support for specifying a local clojure.jar
- Only Hudson will deploy releases or SNAPSHOTs
- All builds must be configured for JDK 1.5 in Hudson
- Stable and SNAPSHOT releases will be deployed to Sonatype's OSS repositories
- Stable releases will be initiated by Hudson (the "Perform Maven Release" link)
- Contrib projects will produce source-only JARs whenever possible
- ZIP distributions will be generated by Maven 2 and manually uploaded to clojure.org
- In "classic" clojure-contrib, only 4 modules require AOT-compilation: condition, fnmap, jmx, repl-ln
- All other clojure-contrib modules are building source-only JARs
The inheritance diagram of POMs:
- Common build configuration across org.clojure projects
- Ensure released JARs are built with JDK 1.5
- SNAPSHOT releases automatically uploaded to public OSS repositories
- Stable releases automatically staged for promotion into Maven Central repository
- Better continuous integration?
Deployment of SNAPSHOTs to the OSS repo is entirely straightforward:
- SCM changes in the corresponding github repos prompts hudson build(s) automatically,
- successful SNAPSHOT builds triggers downstream hudson builds, and
- deployment of SNAPSHOT artifacts to the OSS snapshots repo can be done without any concern for coordination or planning
The release process involves some additional steps, and some added process/policy may be desirable.
Right now, the build.poms and tools.nrepl Hudson jobs are configured to support the most straightforward/naïve release process possible:
- Click "Perform Maven Release" on the job's main page.
- Specify the release and development versions to be provided to maven-release-plugin:
- release version The version of the release to be built and deployed to the OSS repo, thereupon to central
- development version The version that the project's POM(s) will be updated to in git HEAD via a post-release push.
Suggestions for these are usually provided, based on the last version known to Hudson, e.g. if HEAD has a version of 1.0.0-SNAPSHOT, release and dev versions of 1.0.0 and 1.0.1-SNAPSHOT respectively.
- The maven release plugin does a few things worth noting:
- in the hudson-local git repo, it:
- commits an updated project POM with the release version, tagging that changeset with a corresponding tag
- commits another updated project POM with the development version, leaving this changeset as master
- it performs a clean clone of the hudson-local git repo, checking out the tagged release changeset
- in rough terms, performs an
mvn clean deploy, targeting the Sonatype OSS releases repository
- The deploy operation will create or add to an org.clojure-specific release staging repository in the OSS Nexus instance.
- The naïve part: after the deploy operation completes, we automatically close and promote the staging repository's contents to the OSS releases repo, and therefore to central
- Once the
mvn invocation completes successfully, a post-maven build step pushes master and the release tag to the upstream github repo
This process may or may not be sufficient. The uncertainty centers on the interaction with the Nexus release staging repository. Some thoughts:
- In contrast to deploying to a snapshots repository (which exists solely to provide a way to refer to potentially- / frequently-changing artifacts with a stable identifier), once an artifact gets into central, it can never be changed or removed. Thus, there is a desire to have a final opportunity to vet release artifacts.
- To be clear: this policy (and all that flows from it) is central's; release repositories in general (Nexus-hosted or not) are not required to be deployed to via staging repos.
- This is where the staging repos come in: once one is created to contain deployed artifacts, those artifacts can be accessed via the Nexus management site, and subjected to whatever final testing / verification a project's author(s) deem necessary.
- Apache, among others, runs additional tests on the artifacts deployed to a given closed staging repository, the results of which are used as part of their release voting process.
Given the above, questions include:
- Will we want to vet individual release artifacts in any way just prior to their staging repository being promoted to central (similar to e.g. Apache)?
- Will we want to attempt to coordinate releases in any way, at the point of staging repository promotion?
- This would imply not closing our staging repository automatically as part of our projects' release builds.
- Once we've deployed whichever projects we aim to release in a coordinated fashion to the staging repository, we then manually close that repo.
- After verifying that the batch of staged artifacts pass our final criteria, we then manually promote their staging repo.
- If those artifacts are not acceptable, we can then drop the staging repository, and revisit whatever issues were exposed in our final testing.
- Note that this would require us backing out the versioning changes in the projects' POM(s) in SCM (again, presumably a manual operation) without stomping on any substantive patches applied in the interim.
I've written before about how coordinated releases should be off the table; the dynamics of staging repository closing and promotion would seem to make that prospect even less attractive, if only because of the serious pain that would result if we were to back out committed and pushed versioning changes (the automatic handling of which is a no-brainer IMO).
Vetting individual release artifacts would seem to be a more feasible option, if so desired (at least the pain of backing out project version changes would be localized to the project with the problematic release).
However, in both cases, I'd suggest that having multiple alpha and RC releases should eliminate all technical reasons for vetting of staged release artifacts. At the very least, I'd say that it's worth trying to use the naïve approach through one full release cycle, if only because it's the simplest fully-automated solution available.
Possible Future Development
- Leiningen support
- Ant build scripts for local development/testing
- Aggregate projects that package many contrib libraries in one distribution
- Testing harness to build/test many Clojure libraries with a locally-built Clojure JAR
- Redirect build.clojure.org/snapshots and /releases to Central or Sonatype repository
- Assuming we roll all existing releases into central, we only need to set up redirects for