Skip to end of metadata
Go to start of metadata

Problems to Solve

  1. Publishing: Make Clojure and its libraries available to build tools
  2. Building: Compile, test, and package Clojure projects for release
  3. Mixing: Combine Clojure and other languages in one build
  4. Fetching: Download & use external dependencies in Clojure projects

These are all independent! But we expect a build system to do all of them.

Potential Solutions

Maven 2/3 with com.theoryinpractise:clojure-maven-plugin

  • Publishing: Good
    • Already exists
    • Hudson integration
    • Easy deployment to Sonatype OSS
    • Not a Clojure contrib project
  • Building: Fair
    • Inheritance reduces repitition across projects
    • Initial configuration is messy
    • Bad design forces unnecessary compilation steps
    • Slow
    • Not extensible in Clojure
  • Mixing: Good
  • Fetching: Good

Maven 2/3 with redesigned clojure-maven-plugin

  • Publishing: Good
  • Building: Good
    • Opportunity to improve the plugin design
  • Mixing: Good
  • Fetching: Good

Ant with maven-ant-tasks

  • Publishing: Fair
    • Hard to configure deployments
  • Building: Fair
    • No inheritance or reuse
  • Fetching: Poor
  • Mixing: Fair

Clojure-specific tool wrapping Maven (e.g., Leiningen)

  • Publishing: Poor
    • No automatic Sonatype OSS deployment
    • Not a Clojure contrib project
  • Building: Fair
    • Configuration syntax in Clojure
    • Easy to get started
    • Extensible by writing Clojure functions
    • Reinvents the wheel of common Maven plugins
    • No inheritance or reuse
  • Fetching: Good
  • Mixing: Poor
    • Clojure-specific, doesn't support mixed-language projects well

Clojure-specific tool that can use Maven repositories

  • Publishing: Good
  • Building: Fair
    • Configuration syntax in Clojure
    • Opportunity to get the design right
    • Could use Aether for dependency resolution
  • Fetching: Good
  • Mixing: Poor
    • Won't support other languages
    • Reinventing the wheel

Polyglot Maven

  • Publishing: Good
  • Building: Good
    • Configuration syntax in Clojure
    • Not released yet
    • Not a Clojure contrib project
    • Still needs a Maven plugin to build Clojure projects
  • Fetching: Good
  • Mixing: Good

Alternate-language tool (e.g., Gradle)

  • Publishing: Fair?
  • Building: Fair
    • New & unfamiliar syntax
    • Likely needs plugins to build Clojure projects
  • Fetching: Good
  • Mixing: Fair?

Issues to Consider

  • Compatibility with Maven 2.0.x, 2.1.x, and 3.x
  • Compatibility with other popular Maven plugins
    • Javadoc
    • Jar
    • Sources
    • See Clojure's pom.xml for examples
  • Proper integration with Maven Surefire testing plugin?
  • Clojure autodoc support?
  • Fork a new Java process (like Mark Derricutt's plugin) or run in a Classloader?
    • Configurable?
    • Maven Toolchain API support?
    • JVM command-line argument support?
  • What language to write in?
    • Clojure
      • Clojure startup time slows down builds?
      • Makes the plugin depend on the language
      • Have to create plugin metadata XML by hand
      • More extensible?
    • Java
      • Automatically generates plugin metadata XML from source
      • Requires calling Clojure through Java
      • Less flexible?
      • Faster?
  • How to support interactive development?
    • Maven wasn't really designed for that
    • But other languages, e.g. Scala, have made it work
  • Sensible defaults
    • AOT-compile .clj sources?
    • Auto-discovery of namespaces for compilation and/or testing?

Current clojure-maven-plugin

  • Source at github.com/talios/clojure-maven-plugin
  • Good:
    • Deployed to Maven Central
    • Does not depend on any specific version of Clojure
      • Written in Java
    • Handles different JDK versions with the Maven toolchain API
    • Handles different operating systems fairly well
    • Does many things
      • compile, test, swank, nailgun, gendoc, autodoc, etc.
  • Not so good:
    • Not developed under the Clojure CA
      • May be unable to get permission from all contributors
    • Tries to do too many things?
    • Forks a new Java sub-process on every execution
    • Confusing configuration variables
    • Bad defaults
      • AOT-compile everything

What do we need?

  • Essential tasks for building & releasing Clojure projects with Maven
    • AOT-compile
    • Run tests
    • Add Clojure sources to a JAR
  • Useful features for developers
    • Automatic namespace discovery from .clj sources
    • Run a REPL, possibly with wrappers like readline
    • Run a SWANK server
    • Run a Nailgun server
    • Execute arbitrary Clojure code with the project's dependencies on the classpath
    • Execute Clojure in-process or forked
  • The ideal
    • Generic way to execute Clojure code during a Maven build
    • With declarative configuration syntax for common tasks

Ideas

  • Use maven-exec-plugin
    • Supports running any Java class in-process or forked
    • Manages project classpath / dependencies
    • Using it makes the build more script-like, less declarative
  • Use maven-antrun-plugin
    • Supports anything Ant can do
    • Even more flexible than maven-exec-plugin
    • Even more script-like
  • Create a plugin that calls maven-exec-plugin
    • BUT Maven plugins may not depend on other plugins
  • Rewrite clojure-maven-plugin in Java, duplicating some functionality of maven-exec-plugin
    • Configuration option to run in-process or fork
  • Rewrite clojure-maven-plugin in Clojure
    • Can it still launch a process with a different version of Clojure?

Radical Change Thought

Instead of having just "one" plugin, maybe it would be better to break it up to multiple artifacts/plugins, with specific functionality.  For ultimate flexibility I'm guessing each of these artifacts would have to be a separate git repo, with their own lifecycles.

  • org.clojure.mojo/nsdiscovery
    • Namespace discovery code, the existing code, split out.
  • org.clojure.mojo/clojure-maven-embedder
    • Support code to actually execute clojure code, be it in process, forked.
    • Make use of clojure-jsr223 from Armando Blancas ( has signed contributor agreement ), project hasn't been updated since 2009, could be good to also move this under the org.clojure umbrella and give it some new life.  By using a javax.scripting/forking model we keep the direct dependency on clojure outside of the plugin, allowing independent choice of version the end developers project.  This would work well with my idea of having the mojo just a thin umbrella which delegates out to the actual implementation in clojure.
      • Use of jsr223 could probably even be used by the forked VM model as well.
      • Non-forked VMs would preclude any toolchain support/mixed VM
      • jsr223 support would also play well into embedded clojure scripts inside a POM somehow.
  • org.clojure.mojo/clojure-maven-plugin
    • The core of the original plugin
    • A single configurable source directory, with NO namespace configuration settings.  We still use the nsdicovery artifact however, the original reason for namespace selection/filtering was when pulling in multiple clojure projects as git submodules etc.  In a more modular world that we've moved to, with clojars and dependency management, I think this is no longer really needed.  Maybe we need a compiler-directive metadata element that the compiler looks at to ignore/use, which all tools can then use?
    • Stripped down to maybe just the compile goal, and nothing else, default to a temporary output path so we get the strictness of compilation, but no AOT artifacts by default.
    • If we keep each goal, the mojo code should just delegate to an implementation in a supporting artifact - which may or may not be java or clojure based.
  • org.clojure.mojo/clojure-surefire-driver
    • Implements the running of clojure tests using the new surefire 2.7 pluggable provides code, allows reuse of surefires forking model, configuration, debug support etc.
    • The original goal in clojure-maven-plugin could now delegate to this code.
  • org.clojure.mojo/swank-maven-plugin
    • I remember the original code for this goal was actually a separate plugin, maybe it should return to being one?
  • org.clojure.mojo/clojure-archetype
    • I believe Stuart had set one up a long time ago, would be good to pull it back in with everything and update it for the custom packaging etc.

What do other JVM languages do?

  • Scala Maven Plugin
    • add-source goal to add project.build.sourceDirectory/../scala
    • compile and testCompile goals
    • "continuous compilation" goal (infinite loop, compiles on file save)
    • "continuous testing" goal
    • "console" goal starts a REPL
    • "run" goal supports multiple configurations with different main class and arguments
  • JRuby Maven Plugin
    • Still "alpha" as of "Last Published: 07/06/2006"
    • See also JRUBY-434
  • GMaven (Groovy)
    • Includes a Maven archetype for Groovy projects
    • Allows shell-style glob to select files to compile
  • maven-jython-plugin
    • Executes Jython as a sub-process
Labels:
  1. Jan 02, 2011

    I'm conflicted about the idea of splitting the plugin into multiple plugins. On the one hand, it makes a lot of sense to have developer tools, like SWANK, separated from the essential build components. On the other hand, multiple plugins means more maintenance work and more potential for duplication.

    What are the implications of plugins having dependencies? How would the compiler plugin, for example, use the nsdiscovery module?

  2. Jan 02, 2011

    Using JSR 223 (javax.scripting) seems like a good idea.

    One important question: Does the JSR 223 specification require that each ScriptEngine instance provide an isolated execution context and unique global scope? If it does, Clojure cannot implement it correctly without substantial changes to the Clojure runtime.

    Assuming that Clojure can support JSR 223, it has broader applications beyond the Maven plugin. It should become the standard interface for accessing Clojure from Java code, perhaps packaged with the core language itself.

    1. Jan 03, 2011

      There is a JSR 223 compliant fork of Clojure here: https://github.com/pmf/clojure-jsr223

    2. Jan 10, 2011

      I've never used javax.script stuff, but IMO, that looks like a nightmare for Java-->Clojure interop, unless one happens to be committed to using javax.script already.  By all means, let's support it for those that are, but I'd much prefer something more direct and straightforward (as proposed here).

    3. Jan 12, 2011

      I've been doing some experimentation with the clojure-jsr223 jar I originally mentioned.  I inserted that jar into my local maven repo and also added a dependency on 1.3-alpha4 and ran the class I'd posted to https://gist.github.com/f0937f7e7e2e3ea2f16d - so far with just this small tinkering I'm quite pleasantly surprised by the results.

      jsr223 provides what it calls a "binding", which can be pluggable against calls to provide different scopes.  It doesn't appear that jsr223 requires separate global scopes tho.  That the current library does is remap any "binding" into the user ns prior to calling your eval statement - so whilst other namespaces don't disappear there is an element of flexibility.  Ideally I'd probably like this not being the "user" namespace tho, but something specific and unique to jsr223 integration.

      I must say I do like being to just go engine.eval("(+ 5 6)" and get a Long out.

      Also the fact that it just worked out of the box is fairly awesome!

  3. Jan 10, 2011

    If we're really looking at a reimplementation, then I like the new direction, especially the embedder (for those times where I would really rather not set up profiles to run a script or two at different times).  Making it easier for others to bring additional functionality into maven written in Clojure seems like an obvious good.

    This is probably treading too far from the stated scope here, but it'd be even better if writing maven plugins in Clojure were made easier.  An appropriate archetype seems like an obvious first step, but I wonder if there's something more clever and lightweight that could be done.

    Splitting out non-build-related bits seems particularly wise, just so that they can be managed by those that have a vested interest and expertise in the other parts involved (e.g. swank, nailgun, maybe nREPL in the future, etc).  Re: integration of test with surefire, does that imply that there wouldn't be a separate goal execution for clojure:test, or are the separate results just aggregated?

  4. Jan 12, 2011

    The advantage to writing plugins in Java is that they can use the preprocessor to generate all the XML metadata for things like POM configuration tags. We could generate that ourselves, of course, it's just more work.

    The other advantage to writing Java is we avoid potential clashes the version of Java Clojure the plugin was compiled with and the version used in the current project. Especially if we want to compile core!

    1. Jan 11, 2011

      I think a fair bit of the work needed to build mojos in Clojure has been done already by Hugo Duncan (with an example of the results of that work here).  Anyway, I'd be interested in writing Maven plugins in Clojure for a variety of reasons -- I wasn't necessarily suggesting that a c-m-p replacement be implemented using Clojure.

      Why would the version of Java be an issue?  Ensure that your toolchain is running on 1.5, target 1.5, and you're done, right?

      1. Jan 12, 2011

        Sorry, I meant the version of Clojure. Corrected.

  5. Jan 13, 2011

    What implications would multiple plugins have for prefix resolution at the command line?

    What happens if two plugins in the same project have the same goalPrefix?

  6. Jan 13, 2011

    I want to get the essential components done first. In my current thinking, that's two things:

    • One plugin for essential build tasks, with 3 goals:
      • add-source (add Clojure source directory to the POM)
      • compile
      • test-compile
    • One plugin for Surefire testing integration
  7. Jan 18, 2011

    Another question: should we define a new packaging type to define custom lifecycle bindings?

    Recent versions of com.theoryinpractise:clojure-maven-plugin define the "clojure" packaging type as a duplicate of "jar" with additional executions for clojure-maven-plugin.