Problem

There are Clojure dialects (Clojure, ClojureScript, ClojureCLR) hosted on several different platforms. We wish to write libraries that can share as much portable code as possible while leaving flexibility to provide platform-specific bits as needed, then have this code run on all of them.

Use cases for platform-specific functionality currently handled by cljx:

Use caseExample (cljx)Example (feature expr)
Platform-specific require/import

In .cljx file which is preprocessed into .clj and .cljs files:

(ns cemerick.pprng
 #+cljs (:require math.seedrandom
 [cljs.core :as lang])
 #+clj (:require [clojure.core :as lang])
 #+clj (:import java.util.Random)
 (:refer-clojure :exclude (double float int long boolean)))

In .clj file loadable in both Clojure and ClojureScript:

(ns cemerick.pprng
 #+cljs (:require math.seedrandom
 [cljs.core :as lang])
 #+clj (:require [clojure.core :as lang])
 #+clj (:import java.util.Random)
 (:refer-clojure :exclude (double float int long boolean)))

Exception handling, catching "all" - see issue below on why this is not a complete solution in either case.

In .cljx file which is preprocessed into .clj and .cljs files:

(try
 (>! c msg)
 (catch #+clj Exception
        #+cljs js/Object 
        e)
    .... )) 

In .clj file loadable in both Clojure and ClojureScript:

(try
 (>! c msg)
 (catch #+clj Exception
        #+cljs js/Object 
        e)
 .... )) 

 

Platform-specific calls for "hosty" library things (strings, math, dates, random numbers, uris, reflection warnings)

In .cljx file which is preprocessed into .clj and .cljs files:

;; dates
 (.getTime 
  #+clj (java.util.Date.)
  #+cljs (js/Date.))

;; uris 
#+clj
(defn url-encode
 [string]
 (some-> string str (URLEncoder/encode "UTF-8") (.replace "+" "%20")))
#+cljs
(defn url-encode
 [string]
 (some-> string str (js/encodeURIComponent) (.replace "+" "%20")))

;; reflection warnings
#+clj (set! *warn-on-reflection* true)   

 

 

In .clj file loadable in both Clojure and ClojureScript:

;; dates
 (.getTime 
 #+clj (java.util.Date.)
 #+cljs (js/Date.))

;; uris   
#+clj
(defn url-encode
 [string]
 (some-> string str (URLEncoder/encode "UTF-8") (.replace "+" "%20")))
#+cljs
(defn url-encode
 [string]
 (some-> string str (js/encodeURIComponent) (.replace "+" "%20"))) 

;; reflection warnings
#+clj (set! *warn-on-reflection* true)   

 

 

 

Protocol invocation (CLJS internal protocol names start with -, CLJ interfaces do not)

In .cljx file which is preprocessed into .clj and .cljs files:

#+cljs
(deftype With [name unique-name body value bang]
 SpecComponent
 (install [this description]
 (swap! (.-withs description) conj this))
 cljs.core/IDeref
 (-deref [this]
 (when (= ::none @value)
 (reset! value (body)))
 @value))

 (See "-deref" in CLJS vs "deref" in CLJ.)

 

In .clj file loadable in both Clojure and ClojureScript:

#+cljs
(deftype With [name unique-name body value bang]
 SpecComponent
 (install [this description]
 (swap! (.-withs description) conj this))
 cljs.core/IDeref
 (-deref [this]
 (when (= ::none @value)
 (reset! value (body)))
 @value))
Extending protocols to implementation-specific classes

In .cljx file which is preprocessed into .clj and .cljs files:

#+clj
(extend-protocol SpecComponent
 java.lang.Object
 (install [this description] ...)) 

In .clj file loadable in both Clojure and ClojureScript:

#+clj
(extend-protocol SpecComponent
 java.lang.Object
 (install [this description] ...)) 

Proposal summary

The feature expression proposal includes the following items:

WhereItemDescriptionExample
allAdds *features* dynamic variable

Set of keywords indicating supported features. Can be changed in a binding around reading. The feature set is open and can be dynamically modified in binding or modified on startup. Non-namespaced features are reserved by the platform. User features should be namespaced.

allDefines well-known platform feature namesEach platform guarantees that *features* includes the platform feature.clj, cljs, clr
CLJAdds system property to set custom features at startupUse system property clojure.features on JVM startup to specify additional custom features. The platform feature will always be included, regardless of this setting.-Dclojure.features=arch/osx
CLJSAdds build option to set custom features at build timeUse build option :features which should be a set of keywords. The platform feature will always be included, regardless of this setting.:build #{:js/node}
allNew reader syntax for feature include or exclude

Both feature include (#+) and exclude (#-) include a conditional expression and a form. The conditional expression is made from features (boolean value based on membership in feature set), and, or, and not. This is the same syntax used in CL feature expressions and cljx.

(defn my-trim [s]
  #+clj (.. s toString trim)
  #+cljs (gstring/trim s))
allExcluded tagged literals do not need to have a data readerIt is allowed for an excluded feature to use an undefined tagged literal. For example, the #js tagged literal is not defined in Clojure but may used in a #+cljs feature expression.
(def init
  #+cljs #js {}
  #+clj nil)
CLJSAllow .clj extension for ClojureScript source filesBoth .cljs and .clj are now acceptable file extensions for CLJS. Portable files that can be loaded in either Clojure or ClojureScript (typically using feature expressions) must have the .clj extension (or they cannot be loaded in Clojure). 

Issues

Below are important issues considered and current status.

NameDescriptionProposedStatus
Version checksShould feature expressions support broader kinds of expressions (JDK or Clojure version ranges, etc)? Not included
ClojureCLR delimited symbolsClojureCLR supports a wider set of symbols via a |-delimited extension which cannot be read if in feature expressions from the CLJ or CLJS readers.Add support for |-delimited symbols in CLJ, CLJS. Issues involved relate toNot included (reconsider in future)
Portable "catch all" errors

It is currently difficult even with feature expressions to easily "catch all" in CLJS (like catch Throwable).

Note: This is orthogonal to feature expressions but important to the greater goal of portable code.

Proposal exists at Platform Errors with related tickets CLJS-661 and CLJ-1293.OPEN
CLJS namespace collisionsExisting code that uses a particular ns for macros (.clj file) may have conflicts if they use the same .clj for Clojure code in the same library.This scenario is not too common and must be resolved by using non-overlapping namespaces for macros and ClojureScript code in a namespace.No change needed
CLJS file loadingclj.cljs.compiler/cljs-files-in will find all files recursively under a directory via file extension. Expanding this to include .clj will snare .clj macro files. This code is used by the cljsc tool and both command line and browser repl. lein cljsbuild has similar issues.See next section.OPEN
Tooling on raw source of portable files is more complicated.Editors and other tools that work on the raw source files cannot determine a specific dialect of Clojure based on the file extensions. This makes it more complicated to enable advanced editor features that require analyzing code. See Colin Fleming's comments (re Cursive).???OPEN

CLJS File Loading Issue

Adding .clj files as valid file extensions means that parts of ClojureScript that currently specify a directory of .cljs files will include any .clj macro files already in that source tree and interpret them as .clj source files.

Some alternatives and their impact to existing file types or code:

AlternativeDescriptionCLJS .cljsMacro .cljCLJS .cljCLJS buildCLJS ToolsCLJ Tools
Separate macro pathSpecify separate paths for ClojureScript code and macros. The .clj files in the code path are then unambiguously CLJS code.-Move to new macro dir-Pass macro path through stackNeed additional macro path-
Exclude namespacesSpecify a build option to exclude a set of namespaces from compilation. Macros would be explicitly specified there and could then be intermingled but ignored when pulling CLJS source from a directory.---Pass excluded ns list through stackNeed list of excluded macro ns'es-
Mark macro namespacesSpecify ns meta or other marker in .clj macro files so that they can be excluded based on file inspection.-Mark as macro ns-Needs to parse all .clj files to check if macro ns--
Mark regular portable namespacesSpecify ns meta or other marker in .clj source files so that they can be included based on file inspection.--Mark as portableNeeds to parse all .clj file to check if portable ns-?
Specify a portable file extensionDefine a new unambiguous file extension (.cljc) to indicate a portable source file to be read in multiple host envs. (The .cljx extension is somewhat analogous although it only exists during preprocessing.) Explored in more detail in next table.

-

-New file extension .cljcPull in .cljc files as CLJS sourceLook for .cljc filesLook for .cljc files

 Portable file extension consequences

This table explores the consequences and changes required if we added a new portable file extension across Clojure and ClojureScript (a proposed solution for the prior issue CLJS File Loading).

WhereDescription of ImpactChange
CLJSClojureScript finds CLJS source files in a dir by .cljs extensionLook for both .cljs and .cljc as source
CLJClojure looks for CLJ source files by .clj extensionLook for both .clj and .cljc as source
CLJS and CLJIf both .cljc and either .cljs or .clj exist, which will be loaded?Should prefer platform-specific file to common file (.cljc over .cljs or .clj). This enables the open extension case described below in Portable Libraries
lein, mvn, other build toolsFind CLJ source files by .clj extensionLook for both .clj and .cljc as source
lein cljsbuild and other CLJS toolsFind CLJS source files in a dir by .cljs extensionLook for both .cljs and .cljc as source
CLJ or CLJS IDEs (CIDER, Fireplace, Light Table, Cursive, CCW, etc)Need to recognize .cljc as potentially both .clj and .cljs filesUpdate editors and IDEs to be aware of portable Clojure files
   

 

Detailed proposal

Feature Sets

The platform feature will be one of: cljcljs, or clr.  Users may supply their own features, which should, by convention, use namespaces. Platform-specified features will not be namespaced.

In Clojure, the initial feature set may be specified at start time with the system property clojure.features, which is a comma-delimited list of symbols to add as features. The platform feature will always be added, regardless of whether it is set in the system property. 

For example:

-Dclojure.features=arch/osx,my.app/prod,my.app/strictmath

will yield the runtime feature set: #{:clj :arch/osx :my.app/prod :my.app/strictmath}

In ClojureScript, there is a new build option with key :features that takes a set of keywords defining the features. The platform feature :cljs is always added to this set, as in Clojure.

In addition to setting the features initially, users may bind *features* around explicit calls to the reader.

Feature Expression Syntax

Common Lisp approaches this problem using feature expressions.

Each platform has a variable called *features* that is a set of keywords to indicate supported features. 

The Reader understands a new kind of "feature expression". The reader macros #+ and #- are used to include or skip a form based on a feature expression:

Feature expressions evaluate as booleans and are defined as follows:

Skipping in the reader is performed by binding *read-suppressed* to true for the next form.

Example:

(ns feature.expressions
  #+cljs (:require [goog.string :as gstring]))

(defn my-trim [s]
  #+clj (.. s toString trim)
  #+cljs (gstring/trim s))

(my-trim " Hello CL? ")

 

Reading Unknown Tagged Literals

There may be tagged literals or classes that are not known or available on all platforms. The reader must be able to read but avoid constructing these entities when they are in excluded feature expressions.

Example:

(def init #+cljs #js {} 
          #-cljs nil)

The #js tagged literal is not known on the Clojure platform, but the reader should read and avoid trying to instantiate it.

ClojureScript File Loading

When code needs to be loaded based on a namespace, ClojureScript will first look for .cljs files, then for .clj files. 

ClojureScript currently expects source files to end (only) in .cljs. Changes must be made in several places to allow reading .clj files:
 
ClojureScript Tooling Support

ClojureScript tooling needs to be aware of the change in supported file extension names and possibly new build options. Need to ensure tools are able to support the new mixed-language projects well:

 

Patches

JIRA Tickets and patches:

 

Building Portable Libraries

Case 1: Independent CLJS and CLJ files packaged in the same jar
 
Some libraries may provide independent implementations of a library for both CLJ and CLJS and be packaged in the same jar. In this case, namespaces should not overlap between CLJ and CLJS:
 

 

myproj/clj/mylib.clj
myproj/cljs/mylib.cljs

 

Case 2: Mixed portable and non-portable CLJS and CLJ packaged in the same jar
 

 

myproj/mylib.clj - portable code with feature expressions - both CLJ and CLJS load myproj.mylib
myproj/foo.clj - helper clojure ns
myproj/bar.cljs - helper clojure ns

 

Case 3: Library designed for open extension
 
A library designed for open extension isolates platform-specific code in one or more namespaces. The library code may compile against (but not include) a stub namespace or use a .clj mixed mode file to contain a default library implementation. The non-isolated code should all be .clj files that are portable across platforms.

 

myproj/mylib.clj 
myproj/open.clj - a ns used by mylib but designed to be overridden or replaced by a platform-specific version of the ns

 

A separate jar would supply the platform-specific version of the ns:

 

myproj/open.cljs

 

Other additional jars would be used to supply other platform-specific versions of the ns. A consumer would use the first jar + the selected extension jar.

Alternate Approaches

Copy / paste

One approach is to maintain two versions of the same file that are largely the same but modify the platform-specific parts in each copy. This obviously works but is gross.

cljx

cljx is an implementation of feature expressions that:

It has been used successfully by a number of projects (see the cljx README for a partial list).  cljx's limitations include:

cljx expressions are typically applied:

lein-cljsbuild "crossovers"

lein-cljsbuild provides a (deprecated, to be removed) feature called "crossovers" that provides a very limited preprocessing of certain files during the cljsbuild build process; a special comment string is removed, allowing one to work around the -macros declarations required in ClojureScript ns forms.  Crossover files must otherwise be fully portable.  Language/runtime-specific code must be maintained in separate files.  However, (my) experience shows that this can quickly lead to the situation where one has to think a lot about in which file to put a specific function, in order to go though the whole preprocessing machinery. Functions are split into namespaces because of conditional compilation, and not because they belong to the same part or module of the program.

Tagged Literal

Define a custom tagged literal that implements conditional read-time expressions:

#feature/condf [ (and jdk-1.6+ clj-1.5.*) 
 (call-my-fast-reducer-code) 
 else (some-old-fashioned-code) ]

Proof of concept here:  https://github.com/miner/wilkins

References

The Common Lisp Hyperspec about the Sharp Sign macros:

Examples of Common Lisp's Feature Expresions:

Maintaining Portable Lisp Programs:

Crossover files in lein-cljsbuild: