We need a convenient, uniform way to build contrib projects. Problems are
- Team will work across many projects and will not want to lean a different approach to building each one.
- Projects will need to work with a variety of tools: Hudson, leiningen, maven, and IDEs at a minimum. (Note that these are to some degree overlapping.)
- Users will want an easy way to find released versions.
- Users will want sufficiently fine granularity of releases to get things they need, without a ton of stuff they don't.
- Team will need automation: one step to trigger an entire push/build/release cycle when necessary.
- Clojure/core will probably want to package multiple libs into easy-to-consume distributions.
Here is a possible direction. Please shred this and come up with something better!
- Treat the pom.xml as the data of record. All the other tools listed above are capable of producing/consuming this as necessary.
- Register all projects with build.clojure.org to get CI and snapshot builds.
- Configure build.clojure.org to cut releases when it sees a commit with version metadata.
- As an example, the clojure build currently uses boolean flag name "interim", when this is false, Clojure could cut a release.
- Should automate both release and follow-on commit that resets the "interim" flag.
- Document everything and verify that a new contrib author can get started easily.
Another option: configure Hudson to build using "lein test, deploy"
- This would require adding a "deploy" task to Leiningen, but I don't think this would be difficult
- It will allow Leiningen plugins to be used at test time, which some contrib authors may want?
- Allows autogenerated pom.xml files to not be checked in
- But the first option above may be the simplest solution.
- Perhaps we should only consider "lein test, deploy" if issues arise with the pom or if contrib authors want specific lein features at build time?
- Is the build box a good maven repos, or should something different be used for releases?
- Does whatever solution we choose imply any feature requests for leiningen?
- What about Clojars? The proposal above assumes (I think correctly) that the official Clojure and Contrib builds need to come from a official site that we control, not Clojars.
October 26, 2010 - Brainstorm
SDH notes added in blue and green below
- Properly formatted pom.xml required
- Hudson polls master branch on git repo hourly and runs tests.
- Nightly Snapshots are built at 2:00AM ET
- Releases are built as requested (see below)
- Must have groupId of org.clojure
- artifactId is the remaining package name hyphenated (e.g. data.finger-tree is data-finger-tree). Should there be a "clojure" prefix in artifactId, e.g. clojure-data-finger-tree? Could matter if artifactId ever travels alone.
- Must have version which is the current SNAPSHOT version number. (X.Y.Z-SNAPSHOT where X.Y.Z is greater than last release's version)
- Must have `test' and `package' goals.
- This (and any additional requirements as of yet undetermined) will be validated before maven touches the file. Failing validation fails the build. This (and possibly some other items) could be missing in the initial release. So long we requirements are documented, enforcement can always come later.
- If there exists a git tag for the release (format TBD) a release package is created, pending a passing build. Please respect, if possible, the tagging scheme already used by Clojure.** I don't think we can use git tags to trigger the release build, unless we use a separate tag from the actual release tag. The actual release tag means "this was release", so tagging that way before the build would be backwards.** We need something that says to Hudson "release candidate", and then Hudson should apply the "release" tag if the release succeeds.
- The version number is the one in pom.xml without -SNAPSHOT.
- Potential mechanism: gitattributes. We can define a filter to remove -SNAPSHOT from the pom when it is pulled so maven will version correctly.
- This would be required to build packages in all cases. Otherwise unintended inconsistencies could arise.
- Otherwise, a snapshot is created based on the version element in pom.xml.
- Builds must target Java 5 until/unless we ever agree to move the ecosystem to a later version of Java.
- Builds should prefer source-only distribution of jars, unless there is a requirement for compilation. (We should also keep a list of things forcing compilation on us and try to eliminate them.)
- We still need to come up with a way to have hudson place artifact files in /snapshots or /releases accordingly.
- The current hudson plugin requires a hard-coded URI.
- Some options:
- Shell script which does the copy
- Possibly a better hudson plugin (not yet investigated)
- (Insert other, more brilliant idea here)
- We could dodge the issue entirely and not separate snapshots from full releases. There should be no risk of colliding version numbers given a nightly, timestamped snapshot build.
- This will break some Maven-aware tools, possibly including IDEs
There's no good reason to not deploy artifacts to a proper repository. AFAIK, the only advantage to e.g. clojure's scp deployment to a filesystem location is that it helps to avoid running a repository app.
If we deployed project artifacts to a repository like Nexus (free version will do, though I suspect sonatype would set us up with a free "Pro" license), then we would gain:
- a passable artifact discovery UI (and various web services that would make building a more attractive frontend ostensibly straightforward)
- ability to proxy third-party artifacts (important insofar as contrib projects may have external dependencies)
- ability to provide artifact indexes to downstream proxies (aids in visibility of artifacts in users' repos as well as in IDEs)
- all sorts of user management facilities that would allow for the delegation of management activities (e.g. no need to chase down the one or two guys that have access to the build box if there's a problem, when there's a reasonably-sized group of trusted folk that have various admin rights on the nexus).
If that's convincing, then there's the further question of whether we'd like to deploy artifacts to Sonatype's OSS repo (which provides for easy syncing of releases to central) rather than to our own nexus install. I think one will be necessary eventually anyway – assuming we want to provide a saner community repository – but that's a separate topic.
If we decide that getting artifacts into central is important (I think it's hard to argue that it's not), then that makes the question of supporting lein-driven build processes a difficult one. The entire OSS deployment toolchain is maven-based. There may be a reasonable way to connect the dots, but someone will have to do some serious digging.
Given the options considered above for deploying artifacts directly on build.clojure.org, I'd vote (if I were to have a vote) for doing it right from the start, and getting a nexus set up before we deploy any artifacts anywhere.
Response to Issues (above)
- For now, the build box is probably fine. We will probably want to investigate a Nexus solution as load/usage increases. (That may be considerably sooner than this bullet would lead you to believe.)
- A leiningen task which generates the contrib-specific pom.xml would probably be welcome, but is not required by the process. Unless manually editing XML is unacceptable, in which case it is.
- Agreed about Clojars, at least in the context of contrib.
- Consider a pre-release "staging" repository (supported by Nexus Professional)
- How exactly we would pre-validate pom.xml files
- Whether to create and maintain a parent pom.xml file for clojure-contrib and require lib pom.xml files to refer to it
- Whether leiningen can participate fully in the scheme we're developing. Possible answers include yes, no, and yes with modifications and plugins.
- PNH: It sounds like Leiningen can produce a pom.xml file that should be capable of driving a Hudson build via Maven; if we run into missing features needed, I'd be willing to work towards adding them. It would also be possible to run Leiningen in the Hudson build as well, but it would require adding a "deploy" task. (should not be difficult)
What We're Ruling Out
- Inventing / reinventing any tools. There is a consensus among lib authors that using existing tools, while not perfect, is desirable.