We need a convenient, uniform way to build contrib projects. Problems are
- Team will work across many projects and will not want to lean a different approach to building each one.
- Projects will need to work with a variety of tools: Hudson, leiningen, maven, and IDEs at a minimum. (Note that these are to some degree overlapping.)
- Users will want an easy way to find released versions.
- Users will want sufficiently fine granularity of releases to get things they need, without a ton of stuff they don't.
- Team will need automation: one step to trigger an entire push/build/release cycle when necessary.
- Clojure/core will probably want to package multiple libs into easy-to-consume distributions.
- Treat the pom.xml as the data of record. All the other tools listed above are capable of producing/consuming this as necessary.
- Register all projects with build.clojure.org to get CI and snapshot builds.
- Builds must target Java 5
- Nice to have: facilitate source-only jars whenever that can possibly work
- No Inventing / reinventing any tools. There is a consensus among lib authors that using existing tools, while not perfect, is desirable.
- We will use a "parent pom" approach
- Neither build box or clojars are good maven repos
- ...Chas: please describe Sonatype OSS release process here...
- Document everything and verify that a new contrib author can get started easily.
Building Clojure Itself
Switching to Maven would be acceptable if it could support the following things that Rich needs (and already has via ant):
- Rich please fill in the blanks...
Another option: configure Hudson to build using "lein test, deploy"
- This would require adding a "deploy" task to Leiningen, but I don't think this would be difficult
- It will allow Leiningen plugins to be used at test time, which some contrib authors may want?
- Allows autogenerated pom.xml files to not be checked in
- But the first option above may be the simplest solution.
- Perhaps we should only consider "lein test, deploy" if issues arise with the pom or if contrib authors want specific lein features at build time?
- PNH: It sounds like Leiningen can produce a pom.xml file that should be capable of driving a Hudson build via Maven; if we run into missing features needed, I'd be willing to work towards adding them. It would also be possible to run Leiningen in the Hudson build as well, but it would require adding a "deploy" task. (should not be difficult)
Everything below here needs editing...
True or False?
- Properly formatted pom.xml required
- Hudson polls master branch on git repo hourly and runs tests.
- Nightly Snapshots are built at 2:00AM ET
- Releases are built as requested (see below)
- Must have groupId of org.clojure
- artifactId is the remaining package name hyphenated (e.g. data.finger-tree is data-finger-tree). Should there be a "clojure" prefix in artifactId, e.g. clojure-data-finger-tree? Could matter if artifactId ever travels alone.
- Must have version which is the current SNAPSHOT version number. (X.Y.Z-SNAPSHOT where X.Y.Z is greater than last release's version)
- Must have `test' and `package' goals.
- This (and any additional requirements as of yet undetermined) will be validated before maven touches the file. Failing validation fails the build. This (and possibly some other items) could be missing in the initial release. So long we requirements are documented, enforcement can always come later.
There's no good reason to not deploy artifacts to a proper repository. AFAIK, the only advantage to e.g. clojure's scp deployment to a filesystem location is that it helps to avoid running a repository app.
If we deployed project artifacts to a repository like Nexus (free version will do, though I suspect sonatype would set us up with a free "Pro" license), then we would gain:
- a passable artifact discovery UI (and various web services that would make building a more attractive frontend ostensibly straightforward)
- ability to proxy third-party artifacts (important insofar as contrib projects may have external dependencies)
- ability to provide artifact indexes to downstream proxies (aids in visibility of artifacts in users' repos as well as in IDEs)
- all sorts of user management facilities that would allow for the delegation of management activities (e.g. no need to chase down the one or two guys that have access to the build box if there's a problem, when there's a reasonably-sized group of trusted folk that have various admin rights on the nexus).
If that's convincing, then there's the further question of whether we'd like to deploy artifacts to Sonatype's OSS repo (which provides for easy syncing of releases to central) rather than to our own nexus install. I think one will be necessary eventually anyway – assuming we want to provide a saner community repository – but that's a separate topic.
If we decide that getting artifacts into central is important (I think it's hard to argue that it's not), then that makes the question of supporting lein-driven build processes a difficult one. The entire OSS deployment toolchain is maven-based. There may be a reasonable way to connect the dots, but someone will have to do some serious digging.
Given the options considered above for deploying artifacts directly on build.clojure.org, I'd vote (if I were to have a vote) for doing it right from the start, and getting a nexus set up before we deploy any artifacts anywhere.
- Consider a pre-release "staging" repository (supported by Nexus Professional)
November 3, 2010 (Chas)
Snapshots of parent POMs and nREPL are now deploying to Sonatype OSS.
Other contrib projects can use the parent contrib POM as nREPL is now doing to get there as well; the respective Hudson jobs just need to invoke the deploy phase (ideally,
clean deploy). I can help lib authors with the former as needed.
This is a good intermediate stable state, presuming contrib projects stay on Clojure 1.2.0.
I have yet to look at what's needed to get clojure proper going into OSS. That, and getting releases staged and pushed out to central are the next steps. There's a number of issues and process questions that need to be resolved before that can happen though. Here's a list of those questions, as well as basic TODOs:
- Need to gpg key info to sign artifacts for staging of releases into OSS (and thereby into central). Key info is of the form "Name Here (some comment) <firstname.lastname@example.org>". This information will be available to anyone inspecting clojure/contrib releases, and will be pushed into pgp.mit.edu. What name, comment/org, and email should I use?
- All hudson builds currently use the user-local maven repository. I would recommend that at some point we configure all jobs (or twiddling this globally) to use a private m2 repo during their build. This will eliminate the potential for contaminating builds with dependencies available locally, but not available given the repositories defined in each project's POM.
- Doing this will require having a local nexus instance to cache external dependencies.
- We need to settle on the release process for artifacts.
- Assuming we're still "meeting at the POM", the discussion above about using tags to drive releases, git attributes to update pom.xml version numbers, etc. seems crazy to me.
- Use the tools we've got: release and versions plugins, with the attendant Hudson frontends.
- This requires the hudson user to have a proper rsa key that is granted access on the various repos in github, so it can push released version tags and incremented version numbers.
- Slight tangent: worth noting is that breaking apart contrib – while making for a much more pleasant/accessible development process – implies that coordinated releases will essentially not be possible anymore. i.e. contrib lib A depends on B; the latter needs to be released before the former can be released using the latter's released artifact as a dependency, and since everything's a separate project, there's no reactor or other coordinating mechanism to properly order each project's release process.
- Clojure core:
- IMO, the existing release process is not going to cut it anymore; given the credentials needed to deploy to OSS and the gpg key needed to sign all released artifacts going there, having deployed releases (or snapshots, even) happen anywhere other than build.clojure.org seems like a horrible idea.
- The easiest path (from a build & CI perspective) would be to replace ant with maven. I'll assume that's not going to happen (Rich, I've got a crate of monkey-picked oolongs for you if we can switch! ;-) )
- Short of that, I would recommend using the promotions plugin for hudson to drive whatever scripting we put together to twiddle the necessary version (version.properties?), run an ant build, sign jars as necessary, deploy the result to OSS, and push the released version tag and incremented version numbers back into github.
- build.clojure.org/snapshots is defined as an always-available repository in
settings.xml. I suspect this was to avoid having parent POMs; in any case, surely we can eliminate that now?
- There is no JDK 1.5 on build.clojure.org. Obviously needs to be there, and set as the default JDK; hudson install OK here?