Versions Compared

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.

We need a convenient, uniform way to build contrib projects. Problems are

  • Team will work across many projects and will not want to lean a different approach to building each one.
  • Projects will need to work with a variety of tools: Hudson, leiningen, maven, and IDEs at a minimum. (Note that these are to some degree overlapping.)
  • Users will want an easy way to find released versions.
  • Users will want sufficiently fine granularity of releases to get things they need, without a ton of stuff they don't.
  • Team will need automation: one step to trigger an entire push/build/release cycle when necessary.
  • Clojure/core will probably want to package multiple libs into easy-to-consume distributions.

Here is a possible direction. Please shred this and come up with something better!

  • Treat the pom.xml as the data of record. All the other tools listed above are capable of producing/consuming this as necessary.
  • Register all projects with build.clojure.org to get CI and snapshot builds.
  • Configure build.clojure.org to cut releases when it sees a commit with version metadata. 
    • As an example, the clojure build currently uses boolean flag name "interim", when this is false, Clojure could cut a release.
    • Should automate both release and follow-on commit that resets the "interim" flag.
  • Document everything and verify that a new contrib author can get started easily.

Issues

  • Is the build box a good maven repos, or should something different be used for releases?
  • Does whatever solution we choose imply any feature requests for leiningen?
  • What about Clojars? The proposal above assumes (I think correctly) that the official Clojure and Contrib builds need to come from a official site that we control, not Clojars.

October 26, 2010 - Brainstorm

Overview:

  • Properly formatted pom.xml required
  • Hudson polls master branch on git repo hourly and runs tests.
  • Nightly Snapshots are built at 2:00AM ET
  • Releases are built as requested (see below)

Pom:

  • Must have groupId of org.clojure
  • artifactId is the remaining package name hyphenated (e.g. stu.library is stu-library)
  • Must have version which is the current SNAPSHOT version number. (X.Y.Z-SNAPSHOT where X.Y.Z is greater than last release's version)
  • Must have `test' and `package' goals.
  • This (and any additional requirements as of yet undetermined) will be validated before maven touches the file. Failing validation fails the build.

Versioning:

  • If there exists a git tag for the release (format TBD) a release package is created, pending a passing build.
    • The version number is the one in pom.xml without -SNAPSHOT.
    • Potential mechanism: gitattributes. We can define a filter to remove -SNAPSHOT from the pom when it is pulled so maven will version correctly.
    • This would be required to build packages in all cases. Otherwise unintended inconsistencies could arise.
  • Otherwise, a snapshot is created based on the version element in pom.xml.

Repository Management:

  • We still need to come up with a way to have hudson place artifact files in /snapshots or /releases accordingly.
  • The current hudson plugin requires a hard-coded URI.
  • Some options:
    • Shell script which does the copy
    • Possibly a better hudson plugin (not yet investigated)
    • (Insert other, more brilliant idea here)
  • We could dodge the issue entirely and not separate snapshots from full releases. There should be no risk of colliding version numbers given a nightly, timestamped snapshot build.

Response to Issues (above):

  • For now, the build box is probably fine. We will probably want to investigate a Nexus solution as load/usage increases. (That may be considerably sooner than this bullet would lead you to believe.)
  • A leiningen task which generates the contrib-specific pom.xml would probably be welcome, but is not required by the process. Unless manually editing XML is unacceptable, in which case it is.
  • Agreed about Clojars, at least in the context of contrib.

The situation

  • Multi-module clojure-contrib in one repository has not worked well
    • Build is complicated and slow
    • Still hard for authors to work on their libs
  • We will have more projects under the org.clojure umbrella
    • New contrib libraries: nrepl, unify, finger-tree
    • More on the way
  • Current Clojure release process involves manual steps

The goals, in order of priority

  • For users
    • 1. Find release versions
    • 2. IDE / tool support
    • 3. Get just the stuff they need, without stuff they don't
    • 4. Easy-to-use distributions containing multiple libraries known to work together
  • For developers
    • 1. Build / release automation
    • 2. Unified build / release process across projects
    • 3. IDE / tool support

The plan

Clojure, the language

  • patch CLJ-681has been applied
    • Keeps the Ant build the way it is now
      • Developers can continue to use Ant for local builds
    • Maven controls the build/test/release process on Hudson
      • Real pom.xml
      • Declares Sonatype's oss-parent POM as a parent
        • Enables releases to public open-source repositories managed by Sonatype
      • Calls out to Ant for Clojure-specific compile and test phases
  • SNAPSHOT releases automatically deployed by hudson
  • Numbered releases controlled by Hudson/Maven
    • Only Clojure language committers may initiate a release

New contrib projects

  • To promote new contrib projects, see Guidelines for Clojure Contrib committers
  • To release, see How to Make Releases
  • Git repositories
    • New contrib libs get their own repos under the Clojure github organization
    • Each lib gets one or more "owners" who have commit access
    • Library authors still use JIRA + patches to accept contributions
    • All contributors must have a CA
  • JIRA
    • Each contrib library/project will get its own JIRA
  • Maven
    • Contrib projects must be built with Maven 2
    • All contrib POMs must declare org.clojure:pom.contribas a parent
      • Inherits from Sonatype's OSS POM
      • Defines release process (OSS snapshots + staging)
      • Set dependency on latest Clojure release (1.3.0-alpha5)
        • May be overridden on a per-project basis
        • Support building with a local Clojure JAR
      • Inherits from pom.oss-deploy
      • Sets license to EPL
      • Configures Java 1.5 as a target
      • Sets common plugin/build configurations
      • Uses com.theoryinpractise:clojure-maven-plugin
      • Defaults to source-only JAR, override to AOT-compile only when necessary

"Classic" / old clojure-contrib

  • The old monolithic build structure is deprecated
  • The interim multi-module build structure is deprecated
    • Module releases up to 1.3.0-alpha4 are available in the build.clojure.org release repository
      • These are source-only JARs wherever possible
      • They should be compatible with any version of Clojure
    • The multi-module build does not work with Clojure 1.3.0-alpha5 and later
  • Libraries with community demand and willing maintainers will become "new" contrib projects as above
  • The "classic" clojure-contrib JIRAis deprecated
    • Important issues should be reposted in the new per-project JIRAs

Work to be done

  • DONE Clean up / finalize pom.contrib
    • Replace 3-level inheritance (pom.oss-deploy -> pom.baseline -> pom.contrib) with just pom.contrib
  • DONE Finalize contrib project Hudson config
  • DONE Figure out how to give Hudson permissions to contrib authors (Stuart Sierra & Aaron Bedra)
  • DONE Create new Hudson job for finger-tree
  • DONE Figure out how to manage many similar Hudson configurations (Chas Emerick & Stuart Sierra)
  • TODO verify Sonatype release goals do the right thing in the presence of multiple staging repositories
  • TODO Get older releases into Maven Central?
  • TODO Redirect repositories at build.clojure.org to OSS Sonatype
    • Assuming we roll all existing releases into central, we only need to set up redirects for /snapshots

Possible future development

  • Improve the Maven build process; see Clojure Maven Plugin
  • Better Clojure-from-Java interface (see CLJ-452)
  • Aggregate projects that package many contrib libraries in one distribution
  • Testing harness to build/test many Clojure libraries with a locally-built Clojure JAR
  • Making the build process better
    • Improve clojure.lang.Compile
      • Investigate why compilation sometimes pauses for a long time (minutes) at the end
      • Investigate if core needs explicit compilation order
    • Investigate Maven 3
    • Investigate Polyglot Maven
    • Documentation: how do I do X, Y, or Z?
    • Write more plugins
    • Developing other non-Maven tools
      • Will require a lot of work to match what Maven can do
      • Not a Clojure/core priority