Versions Compared

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.

Problem

Writing programs that target Clojure and ClojureScript involves a lot of copy and pasting. The usual approach is to copy the whole code of one implementation to a source file of the other implementation and to modify the platform dependent forms until they work on the other platform. Depending on the kind of program the platform specific code is often a fraction of the code that works on both platforms. A change to platform independent code requires a modification of two source files that have to be kept in sync. To solve this problem branching by target platform on a form level would help a lot.

See also ticket CLJS-27.

Current Solutions

Tools like lein-cljsbuild provide some help using "crossover" files. Crossover files are usually written in Clojure and contain only code that works on both platforms. On compilation the crossover files are run through a preprocessor that removes the special comment ;*CLJSBUILD-REMOVE*; from the source file to allow platform specific references to macros. This is explained in detail over here:

This works quite well, but is probably not the right granularity. This approach requires one to split a project into platform specific files, which is not a bad practice per-se. However, (my) experience shows that this can quickly lead to the situation where one has to think a lot about in which file to put a specific function, in order to go though the whole preprocessing machinery. Functions are split into namespaces because of conditional compilation, and not because they belong to the same part or module of the program.

As mentioned above the granularity where this solution solves the problem is probably not at the right level. Having only one function with a platform specific form puts one into the business of splitting it into two files maintaining two different versions of the function. A branching granularity on a form level would probably be much better.

Potential Solution

A solution to this problem that is used by Common Lisp implementations are feature expressions. There are Clojure dialects (Clojure, ClojureScript, ClojureCLR) hosted on several different platforms. We wish to write libraries that can share as much portable code as possible while leaving flexibility to provide platform-specific bits as needed, then have this code run on all of them.

Use cases for platform-specific functionality:

Proposal: Feature expressions

Common Lisp approaches this problem using feature expressions.

Each platform has a variable called *features* that contains keywords that indicate the supported features of the platform the code is running under. The branching on a platform or a platform specific feature is done via the reader macros #+ and #- followed by a feature condition. The feature condition is either a symbol or a form that combines symbols with the or, and or not operators. The feature condition is evaluated by looking up the symbols in the *features* variable. If the feature condition evaluates to true the next form will be passed through the reader and evaluated, otherwise it will be discarded.The patches attached to the CLJS-27 ticket contain a proof of concept implementation of these feature expressions for Clojure and ClojureScript. With this extension one can branch on a form level and write code like illustrated by the following exampleis a set of keywords to indicate supported features. 

The Reader understands a new kind of "feature expression". The reader macros #+ and #- are used to include or skip a form based on a feature expression:

  • #+feature-expr form - include next form if feature-expr evaluates to true
  • #-feature-expr form - skip next form if feature-expr evaluates to true

Feature expressions evaluate as booleans and are defined as follows:

  • feature - a symbol, evaluated as if:

    Code Block
    (contains? *features* (keyword feature))
  • (not feature-expr) - returns logical not of feature-expr
  • (and feature-expr*) - returns true if all feature-expr are true, otherwise false.
  • (or feature-expr*) - returns true if any of its feature-expr are true, otherwise false.

Skipping in the reader is performed by binding *read-suppressed* to true for the next form.

Example:

No Format
(ns feature.expressions
  #+clojurescriptcljs (:require [goog.string :as gstring]))

(defn my-trim [s]
  #+clojureclj (.. s toString trim)
  #+clojurescriptcljs (gstring/trim s))

(my-trim " Hello CL? ")

The patches add a dynamic variable called *features* to the clojure.core and cljs.core namespaces, that should contain the supported features of the platform in question as keywords. Unlike in Common Lisp, the variable is a Clojure set and not a list. In Clojure the set contains at the moment the :clojure keyword, and in ClojureScript the :clojurescript keyword.

I would like to get feedback on the following issues: 

  • Are those keywords ok? Is :jvm for Clojure and :js for ClojureScript better? 
  • Should ClojureScript add something like :rhino, :v8 or :browser as well?
  • Someone mentioned that this should support Clojure's namespaces. How should this be done?
  • How should the compilation process work with this extension? 
  • Will the ClojureScript compiler read *.clj files?  What happens to *.cljs files?

To run the ClojureScript tests, drop a JAR named "clojure.jar" that has the Clojure patch applied into ClojureScript's lib directory.

Features

The platform feature will be one of: clj, cljs, or clr.  Users may supply their own features, which should, by convention, use namespaces. Platform-specified features will not be namespaced.

In Clojure, the initial feature set may be specified at start time with the system property clojure.features, which is a comma-delimited list of symbols to add as features. The platform feature will always be added, regardless of whether it is set in the system property. 

For example:

Code Block
-Dclojure.features=arch/osx,my.app/prod,my.app/strictmath

will yield the runtime feature set: #{:clj :arch/osx :my.app/prod :my.app/strictmath}

In ClojureScript, there is a new build option with key :features that takes a set of keywords defining the features. The platform feature :cljs is always added to this set, as in Clojure.

In addition to setting the features initially, users may bind *features* around explicit calls to the reader.

Reading Unreadable Things

There may be tagged literals or classes that are not known or available on all platforms. The reader must be able to read but avoid constructing these entities.

Example:

Code Block
(def init #+cljs #js {} 
          #-cljs nil)

The #js tagged literal is not known on the Clojure platform, but the reader should read and skip it without failure.

CLR extended symbol problem: Clojure CLR uses an expanded set of valid symbols in the reader. The #| reader extension from Common Lisp was implement in ClojureCLR to delimit symbols containing otherwise invalid characters. The corresponding symbols are not currently readable by Clojure or ClojureScript.  The #| extension support is described here: https://github.com/clojure/clojure-clr/wiki/Specifying-types. The #| reader extension would need to be supported in Clojure and ClojureScript readers to support the full set of valid ClojureCLR symbols in feature expressions.

Loading and File Extensions

When code needs to be loaded based on a namespace:

  • Clojure will continue to read only .clj files. The file will be read by the Clojure reader and feature expressions will be applied.
  • ClojureScript will first look for .cljs files, then for .clj files. The .clj file will be read by the ClojureScript reader and feature expressions will be applied.
  • ClojureCLR - ???
CLJS source extension problem: ClojureScript currently expects source files to end (only) in .cljs. There are a number of places in the code where this assumption is made:
  • clj.cljs.compiler/rename-to-js - renames .cljs file name to .js file name. 
    • regex is easily fixed for this to cover both .clj and .cljs
  • clj.cljs.compiler/cljs-files-in - given a directory, finds all .cljs files to compile
    • used by clj.cljs.repl/analyze-source, which is used by clj.cljs.repl.browser/-setup and clj.cljs.repl/repl
    • can grab .clj too, but may include macro files not previously included - how do we distinguish here?? see next question.
  • clj.cljs.analyzer/ns->relpath - given a namespace, find the resource path
    • can check first for .cljs path, then .clj path (adds resource check where prior was simple string manipulation)
  • clj.cljs.closure/cljs-source-for-namespace - given a ns, find path and url of ns

    • can check first for .cljs resource, then .clj resource
  • clj.cljs.repl.rhino/goog-require - given ns, makes cljs resource path

    • can check first for .cljs resource, then .clj resource
  • clj.cljs.repl.browser/send-static - serves static resources

    • add support for serving .clj as well as .cljs
 
CLJS accidental macro compilation problem: when we collect all the ClojureScript code to compile, we will (accidentally) include .clj macro files that should only be read for macro expansion.

We use a single classpath (often merged from several directories within a library) to load several kinds of ClojureScript resources:
  • ClojureScript-only files: .cljs - compiled as CLJS
  • Mixed source files: .clj (presumably with feature expressions) - compiled as CLJS (NEW)
  • ClojureScript macro files: .clj (loaded as needed for macro support in CLJS)
Because mixed and macro files have the same extension, we need a way to indicate that macro files should not be compiled as ClojureScript. Possible solutions:
  1. Modify the way ClojureScript loads code to specify separate paths for ClojureScript code and macro code  NO
    1. Likely breaks many existing projects which mingle these into the same source path.
    2. Also breaks library publishing as a single jar.
  2. Add build option to specify names to skip during compilation (the macro namespaces).   NO
    1. require-macros would ignore this directive so require-macros would still load the ns'es
    2. Existing CLJS projects would still work if the code in the macro files happened to be compile-able by ClojureScript
    3. This capability would also make it easier to mingle Clojure and ClojureScript projects in the same project in a single source tree (with capability to ignore certain files for CLJS for reasons other than avoiding compilation of macro files)
    4. Main problem is that if published as a library, there is no way to get this ns exclusion list to users of the published library. Would have to also invent some way to specify that list of namespaces in jar metadata etc.
  3. Add marker in macro namespaces to identify them as being skipped for compilation  INVESTIGATING
    1. Add namespace meta or other indicator that this .clj file is only included for macros not for compilation - use existing parse-ns for this
    2. Could automatically be picked up by downstream users of code published as a library

Open Extension

One consequence of feature expressions is that a library must encode solutions for all of the supported platforms in the source code at packaging time (in other words, extension is closed to external users).
 
An alternate solution to specifying all variants in a single file with feature expressions is to swap whole namespaces in and out for different platforms. This allows others to provide implementations for different platforms (if, for example, the library author lacked the expertise to do so). This strategy is orthogonal to feature expressions but made possible by the extension preference checking as specified in the previous section.

Example:

No Format
Library files:
 src/my/core.clj - ns requires my.thing
 src/my/thing.clj - Clojure implementation of my.thing (loaded by Clojure)
 src/my/thing.cljs - ClojureScript implementation of my.thing (loaded by ClojureScript)

ClojureScript Tooling Support

ClojureScript tooling needs to be aware of the change in supported file extension names and possibly new build options. Need to ensure tools are able to support the new mixed-language projects well:

  • lein-cljsbuild - most ClojureScript projects today are built with the lein-cljsbuild plugin. 
    • compiler.clj - needs to properly account for ClojureScript .clj files
    • features - might need to add support for specifying non-default feature set when testing
  • austin?
  • what else?

 

FAQ

  1. What about non-boolean expressions for things like Clojure version, JDK version, etc?  Out of scope. The "compile-if" trick covers many of those (relatively rare) cases already.

Patches

JIRA Tickets and patches:

Alternate Approaches

Copy / paste

One approach is to maintain two versions of the same file that are largely the same but modify the platform-specific parts in each copy. This obviously works but is gross.

cljx

cljx is an implementation of feature expressions that:

  • rewrites .cljx files containing feature expression-annotated code into external files based on well-known tags
    • The most common use is to use clj and cljs tags and write .clj and .cljs files for consumption by other tools/compilers/etc
  • optionally applies the same transformation interactively via installation of a REPL extension

It has been used successfully by a number of projects (see the cljx README for a partial list).  cljx's limitations include:

  • It does not address portability of macros at all; it is strictly a source-to-source transformation.  Macros continue to be written in Clojure, and must be rewritten or implemented conditionally on the contents of &env.
  • It does not provide any runtime customization of the "features" you can use; these are set either within build configuration (via cljx' Leiningen integration), or via the configuration of the cljx REPL extension.  The latter technically is available for modification, but is not in practical use.
  • The set of provided "features" is limited to one for Clojure (#+clj) and one for ClojureScript (#+cljs).  Further discrimination based on target runtime (e.g. rhino vs. node vs. v8 vs. CLR) would be trivial, but has not been implemented to date. 

cljx expressions are typically applied:

  • Inside ns macro
  • Top-level forms
  • Occasionally internal forms where it's concise
    • (.getTime #+clj (java.util.Date.) #+cljs (js/Date.))
lein-cljsbuild "crossovers"

lein-cljsbuild provides a (deprecated, to be removed) feature called "crossovers" that provides a very limited preprocessing of certain files during the cljsbuild build process; a special comment string is removed, allowing one to work around the -macros declarations required in ClojureScript ns forms.  Crossover files must otherwise be fully portable.  Language/runtime-specific code must be maintained in separate files.  However, (my) experience shows that this can quickly lead to the situation where one has to think a lot about in which file to put a specific function, in order to go though the whole preprocessing machinery. Functions are split into namespaces because of conditional compilation, and not because they belong to the same part or module of the program.

Tagged Literal

Define a custom tagged literal that implements conditional read-time expressions:

#feature/condf [ (and jdk-1.6+ clj-1.5.*) 
 (call-my-fast-reducer-code) 
 else (some-old-fashioned-code) ]

Proof of concept here:  https://github.com/miner/wilkins

References

The Common Lisp Hyperspec about the Sharp Sign macros:

...

Maintaining Portable Lisp Programs:

...

ClojureScript JIRA Ticket and patches with a proof of concept implementation of CL's feature expressions: