Error formatting macro: pagetree: java.lang.NullPointerException
Skip to end of metadata
Go to start of metadata
You are viewing an old version of this page. View the current version. Compare with Current  |   View Page History

Our Story Thus Far

It is a publicly stated goal of Clojure not to become an "island" apart from the Java ecosystem. We want JVM developers, including large enterprises, to be able to use Clojure and integrate it with their existing Java projects.

When it comes to distributing Java libraries, there's only one game in town, and that is the Maven central repository. I am told that some businesses will not even consider using Java libraries that are not available in the Maven central repository.

So we want Clojure uploaded to the Maven central repository.

Prior to now, this process has been manual, resulting in a long delay between the public announcement of a release and its appearance in the repository. We want to shorten that lead time.

Clojure-contrib projects have never been uploaded to the central repository; we want to change this.

For deploying Java projects to Maven repositories, there are only two games in town: Ant and Maven. We have been down the road with Ant. Clojure's old build.xml script was long, complex, and understood by few. The deployment tasks were understood by no one, frequently broken, and heavily dependent on the layout of the server at build.clojure.org. The Maven repository we maintained at build.clojure.org did not contain the metadata files expected by Java IDEs and other Maven-aware tools.

This speaks to the more fundamental problem with script-based build systems. Since the days of Make, every software project has rolled its own build and deployment process. In the Java world, every Ant build.xml tends to repeat the same patterns, with minor variations. Ant does not provide a convenient way to abstract or reuse these patterns. As projects grow, they develop ever-more-complex build scripts that are difficult to maintain and highly dependent on the build environment. This is exactly the problem we encountered with Clojure.

So we do not want to use Ant.

Why Maven?

Maven was invented to solve the problems of script-based builds by abstracting out common patterns of software development into reusable components, or plugins. It achieves this by being opinionated. It imposes a rigid model of software building and deployment, codified in the Project Object Model (POM). The POM is not a build script. It is not a script at all. Rather, it is a tree of parameters describing which plugins should be used to build a project, and with what configuration options.

The second reason Maven was invented was to make build configurations reusable across multiple projects. This is achieved through POM inheritance. Every POM has a parent, either specified explicitly or defaulting to a "Super POM" configured for a typical Java project. The POM inheritance model is independent from the Maven dependency model, and is single-inheritance-only. Clojure's new POM inherits from a publicly-available POM which defines configuration parameters for deploying to Sonatype's open-source Maven repositories, and from there to the Maven central repositories.

Necessity Is the Mother of Invention

Mark Derricutt was a developer scratching an itch when he wrote and released a Maven plugin, whose full name is com.theoryinpractise:clojure-maven-plugin. Because it was convenient, a number of Clojure library developers started using it, including clojure-contrib developers. There are problems with this plugin, starting with the fact that it was not developed under the Clojure CA. We have started the process of obtaining the necessary permissions to place it under the CA, but with over 30 committers we are not hopeful that this will be quick.

Meanwhile, Mark and Stuart S. have discussed the possibility of writing a better plugin from scratch, under the CA. It would be smaller and simpler than Mark's old plugin, doing just the minimum necessary to build and release Clojure libraries with Maven. Extra features, such as running a SWANK server, will be left out, perhaps to be developed in other plugins.

Writing Maven plugins, while not inherently complex, is tricky to get right. We are building a tool not only for ourselves but for anyone using Clojure in a large JVM project. A plugin should be usable with any version of Maven post-2.0.0, and should work in all operating systems and environments in which Maven is used. It should not conflict with any other plugin, or make too many assumptions about how the project is structured.

This last point is a flaw in Mark's old plugin: it makes too many assumptions about how Clojure code will be written, such as the assumption that compilation order is not significant. In Clojure core, compilation order is significant, making Mark's old plugin inappropriate for compiling core.

We were able to work around this problem in Clojure core by injecting Ant back into the build process through a Maven Ant plugin. It happens to work because building Clojure is not very complicated. But this technique cannot be reused without copying the same scripts, with modifications, into every project that needs them.

To be absolutely clear: the mixture of Ant and Maven in Clojure's build is a temporary hack, one which we do not want to maintain indefinitely. Furthermore, no one is willing to maintain Ant build scripts across multiple clojure-contrib projects.

Since Mark released com.theoryinpractise:clojure-maven-plugin to the Maven central repository, it is now part of the Maven / Java ecosystem, on equal footing with the Apache Commons libraries and Maven itself. Since we have chosen to participate in the Maven ecosystem, there are no rational grounds for excluding one small piece of that ecosystem. While not ideal, this plugin is adequate for building and releasing Clojure libraries, including Clojure contrib projects.

Mothers Are the Necessity of Invention

We do not want clojure-contrib library authors to be burdened with configuring their build processes, and we do not want to repeat the same configuration settings in every project. We are using POM inheritance to achieve this. In the clojure-contrib parent POM, currently named org.clojure:pom.contrib, we configure Mark's old plugin with our preferred defaults: compile and test all Clojure namespaces, but do not include AOT-compiled .class files in the output JAR. We have successfully released and org.clojure:tools.nrepl under this process. A similar process has driven all 1.3.0-alpha* releases of the old, monolithic clojure-contrib.

Like the use of Ant in core, the use of Mark's old plugin in contrib is temporary. It is the shortest route to getting contrib projects building on Hudson and deployed to the Maven central repository. Once we have developed a replacement plugin, we will configure it once in the org.clojure:pom.contrib parent POM and all contrib projects will start using it without needing further changes.

The design discussion around a new plugin, to be named org.clojure:clojure-maven-plugin or similar, is ongoing. The scope of the project remains indeterminate, in part because there is no written specification for how Java programs should interact with the Clojure compiler. (For example, one suggestion is to implement the javax.scripting interfaces defined in JSR-223.) There are likely multiple plugins to be written: at a minimum, one for compilation and one for integration with the Surefire testing plugin. Once we have determined the correct approach we will be able to estimate the amount of developer time necessary to implement these plugins. Until then, using com.theoryinpractise:clojure-maven-plugin is the only viable solution.

Further Reading

Introduction to the Build Lifecycle: description of the standard phases in a Maven build

Maven: The Complete Reference: free book from Sonatype

Labels: