Versions Compared

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.

Problem

Writing programs that target Clojure and ClojureScript involves a lot of copy and pasting. The usual approach is to copy the whole code of one implementation to a source file of the other implementation and to modify the platform dependent forms until they work on the other platform. Depending on the kind of program the platform specific code is often a fraction of the code that works on both platforms. A change to platform independent code requires a modification of two source files that have to be kept in sync. To solve this problem branching by target platform on a form level would help a lot.

See also ticket CLJS-27.

Current Solutions

Tools like lein-cljsbuild provide some help using "crossover" files. Crossover files are usually written in Clojure and contain only code that works on both platforms. On compilation the crossover files are run through a preprocessor that removes the special comment ;*CLJSBUILD-REMOVE*; from the source file to allow platform specific references to macros. This is explained in detail over here:

This works quite well, but is probably not the right granularity. This approach requires one to split a project into platform specific files, which is not a bad practice per-se. However, (my) experience shows that this can quickly lead to the situation where one has to think a lot about in which file to put a specific function, in order to go though the whole preprocessing machinery. Functions are split into namespaces because of conditional compilation, and not because they belong to the same part or module of the program.

As mentioned above the granularity where this solution solves the problem is probably not at the right level. Having only one function with a platform specific form puts one into the business of splitting it into two files maintaining two different versions of the function. A branching granularity on a form level would probably be much better.

Potential Solution

A solution to this problem that is used by Common Lisp implementations are feature expressions. There are Clojure dialects (Clojure, ClojureScript, ClojureCLR) hosted on several different platforms. We wish to write libraries that can share as much portable code as possible while leaving flexibility to provide platform-specific bits as needed, then have this code run on all of them.

Use cases for platform-specific functionality currently handled by cljx:

Use caseExample (cljx)Example (feature expr)
Platform-specific require/import

In .cljx file which is preprocessed into .clj and .cljs files:

Code Block
(ns cemerick.pprng
 #+cljs (:require math.seedrandom
 [cljs.core :as lang])
 #+clj (:require [clojure.core :as lang])
 #+clj (:import java.util.Random)
 (:refer-clojure :exclude (double float int long boolean)))

In .clj file loadable in both Clojure and ClojureScript:

Code Block
(ns cemerick.pprng
 #+cljs (:require math.seedrandom
 [cljs.core :as lang])
 #+clj (:require [clojure.core :as lang])
 #+clj (:import java.util.Random)
 (:refer-clojure :exclude (double float int long boolean)))

Exception handling, catching "all" - see issue below on why this is not a complete solution in either case.

In .cljx file which is preprocessed into .clj and .cljs files:

Code Block
(try
 (>! c msg)
 (catch #+clj Exception
        #+cljs js/Object 
        e)
    .... )) 

In .clj file loadable in both Clojure and ClojureScript:

Code Block
(try
 (>! c msg)
 (catch #+clj Exception
        #+cljs js/Object 
        e)
 .... )) 

 

Platform-specific calls for "hosty" library things (strings, math, dates, random numbers, uris, reflection warnings)

In .cljx file which is preprocessed into .clj and .cljs files:

Code Block
;; dates
 (.getTime 
  #+clj (java.util.Date.)
  #+cljs (js/Date.))

;; uris 
#+clj
(defn url-encode
 [string]
 (some-> string str (URLEncoder/encode "UTF-8") (.replace "+" "%20")))
#+cljs
(defn url-encode
 [string]
 (some-> string str (js/encodeURIComponent) (.replace "+" "%20")))

;; reflection warnings
#+clj (set! *warn-on-reflection* true)   

 

 

In .clj file loadable in both Clojure and ClojureScript:

Code Block
;; dates
 (.getTime 
 #+clj (java.util.Date.)
 #+cljs (js/Date.))

;; uris   
#+clj
(defn url-encode
 [string]
 (some-> string str (URLEncoder/encode "UTF-8") (.replace "+" "%20")))
#+cljs
(defn url-encode
 [string]
 (some-> string str (js/encodeURIComponent) (.replace "+" "%20"))) 

;; reflection warnings
#+clj (set! *warn-on-reflection* true)   

 

 

 

Protocol invocation (CLJS internal protocol names start with -, CLJ interfaces do not)

In .cljx file which is preprocessed into .clj and .cljs files:

Code Block
#+cljs
(deftype With [name unique-name body value bang]
 SpecComponent
 (install [this description]
 (swap! (.-withs description) conj this))
 cljs.core/IDeref
 (-deref [this]
 (when (= ::none @value)
 (reset! value (body)))
 @value))

 (See "-deref" in CLJS vs "deref" in CLJ.)

 

In .clj file loadable in both Clojure and ClojureScript:

Code Block
#+cljs
(deftype With [name unique-name body value bang]
 SpecComponent
 (install [this description]
 (swap! (.-withs description) conj this))
 cljs.core/IDeref
 (-deref [this]
 (when (= ::none @value)
 (reset! value (body)))
 @value))
Extending protocols to implementation-specific classes

In .cljx file which is preprocessed into .clj and .cljs files:

Code Block
#+clj
(extend-protocol SpecComponent
 java.lang.Object
 (install [this description] ...)) 

In .clj file loadable in both Clojure and ClojureScript:

Code Block
#+clj
(extend-protocol SpecComponent
 java.lang.Object
 (install [this description] ...)) 

Proposal summary

The feature expression proposal includes the following items:

WhereItemDescriptionExample
allAdds *features* dynamic variable

Set of keywords indicating supported features. Can be changed in a binding around reading. The feature set is open and can be dynamically modified in binding or modified on startup. Non-namespaced features are reserved by the platform. User features should be namespaced.

allDefines well-known platform feature namesEach platform guarantees that *features* includes the platform feature.clj, cljs, clr
CLJAdds system property to set custom features at startupUse system property clojure.features on JVM startup to specify additional custom features. The platform feature will always be included, regardless of this setting.-Dclojure.features=arch/osx
CLJSAdds build option to set custom features at build timeUse build option :features which should be a set of keywords. The platform feature will always be included, regardless of this setting.:build #{:js/node}
allNew reader syntax for feature include or exclude

Both feature include (#+) and exclude (#-) include a conditional expression and a form. The conditional expression is made from features (boolean value based on membership in feature set), and, or, and not. This is the same syntax used in CL feature expressions and cljx.

(defn my-trim [s]
  #+clj (.. s toString trim)
  #+cljs (gstring/trim s))
allExcluded tagged literals do not need to have a data readerIt is allowed for an excluded feature to use an undefined tagged literal. For example, the #js tagged literal is not defined in Clojure but may used in a #+cljs feature expression.
(def init
  #+cljs #js {}
  #+clj nil)
CLJSAllow .clj extension for ClojureScript source filesBoth .cljs and .clj are now acceptable file extensions for CLJS. Portable files that can be loaded in either Clojure or ClojureScript (typically using feature expressions) must have the .clj extension (or they cannot be loaded in Clojure). 

Issues

Below are important issues considered and current status.

NameDescriptionProposedStatus
Version checksShould feature expressions support broader kinds of expressions (JDK or Clojure version ranges, etc)? Not included
ClojureCLR delimited symbolsClojureCLR supports a wider set of symbols via a |-delimited extension which cannot be read if in feature expressions from the CLJ or CLJS readers.Add support for |-delimited symbols in CLJ, CLJS. Issues involved relate toNot included (reconsider in future)
Portable "catch all" errors

It is currently difficult even with feature expressions to easily "catch all" in CLJS (like catch Throwable).

Note: This is orthogonal to feature expressions but important to the greater goal of portable code.

Proposal exists at Platform Errors with related tickets CLJS-661 and CLJ-1293.OPEN
CLJS namespace collisionsExisting code that uses a particular ns for macros (.clj file) may have conflicts if they use the same .clj for Clojure code in the same library.This scenario is not too common and must be resolved by using non-overlapping namespaces for macros and ClojureScript code in a namespace.No change needed
CLJS file loadingclj.cljs.compiler/cljs-files-in will find all files recursively under a directory via file extension. Expanding this to include .clj will snare .clj macro files. This code is used by the cljsc tool and both command line and browser repl. lein cljsbuild has similar issues.See next section.OPEN
Tooling on raw source of portable files is more complicated.Editors and other tools that work on the raw source files cannot determine a specific dialect of Clojure based on the file extensions. This makes it more complicated to enable advanced editor features that require analyzing code. See Colin Fleming's comments (re Cursive).???OPEN

CLJS File Loading Issue

Adding .clj files as valid file extensions means that parts of ClojureScript that currently specify a directory of .cljs files will include any .clj macro files already in that source tree and interpret them as .clj source files.

Some alternatives and their impact to existing file types or code:

AlternativeDescriptionCLJS .cljsMacro .cljCLJS .cljCLJS buildCLJS ToolsCLJ Tools
Separate macro pathSpecify separate paths for ClojureScript code and macros. The .clj files in the code path are then unambiguously CLJS code.-Move to new macro dir-Pass macro path through stackNeed additional macro path-
Exclude namespacesSpecify a build option to exclude a set of namespaces from compilation. Macros would be explicitly specified there and could then be intermingled but ignored when pulling CLJS source from a directory.---Pass excluded ns list through stackNeed list of excluded macro ns'es-
Mark macro namespacesSpecify ns meta or other marker in .clj macro files so that they can be excluded based on file inspection.-Mark as macro ns-Needs to parse all .clj files to check if macro ns--
Mark regular portable namespacesSpecify ns meta or other marker in .clj source files so that they can be included based on file inspection.--Mark as portableNeeds to parse all .clj file to check if portable ns-?
Specify a portable file extensionDefine a new unambiguous file extension (.cljc) to indicate a portable source file to be read in multiple host envs. (The .cljx extension is somewhat analogous although it only exists during preprocessing.) Explored in more detail in next table.

-

-New file extension .cljcPull in .cljc files as CLJS sourceLook for .cljc filesLook for .cljc files

 Portable file extension consequences

This table explores the consequences and changes required if we added a new portable file extension across Clojure and ClojureScript (a proposed solution for the prior issue CLJS File Loading).

WhereDescription of ImpactChange
CLJSClojureScript finds CLJS source files in a dir by .cljs extensionLook for both .cljs and .cljc as source
CLJClojure looks for CLJ source files by .clj extensionLook for both .clj and .cljc as source
CLJS and CLJIf both .cljc and either .cljs or .clj exist, which will be loaded?Should prefer platform-specific file to common file (.cljc over .cljs or .clj). This enables the open extension case described below in Portable Libraries
lein, mvn, other build toolsFind CLJ source files by .clj extensionLook for both .clj and .cljc as source
lein cljsbuild and other CLJS toolsFind CLJS source files in a dir by .cljs extensionLook for both .cljs and .cljc as source
CLJ or CLJS IDEs (CIDER, Fireplace, Light Table, Cursive, CCW, etc)Need to recognize .cljc as potentially both .clj and .cljs filesUpdate editors and IDEs to be aware of portable Clojure files
   

 

Detailed proposal

Feature Sets

The platform feature will be one of: cljcljs, or clr.  Users may supply their own features, which should, by convention, use namespaces. Platform-specified features will not be namespaced.

In Clojure, the initial feature set may be specified at start time with the system property clojure.features, which is a comma-delimited list of symbols to add as features. The platform feature will always be added, regardless of whether it is set in the system property. 

For example:

Code Block
-Dclojure.features=arch/osx,my.app/prod,my.app/strictmath

will yield the runtime feature set: #{:clj :arch/osx :my.app/prod :my.app/strictmath}

In ClojureScript, there is a new build option with key :features that takes a set of keywords defining the features. The platform feature :cljs is always added to this set, as in Clojure.

In addition to setting the features initially, users may bind *features* around explicit calls to the reader.

Feature Expression Syntax

Common Lisp approaches this problem using feature expressions.

Each platform has a variable called *features* that contains keywords that indicate the supported features of the platform the code is running under. The branching on a platform or a platform specific feature is done via the reader macros #+ and #- followed by a feature condition. The feature condition is either a symbol or a form that combines symbols with the or, and or not operators. The feature condition is evaluated by looking up the symbols in the *features* variable. If the feature condition evaluates to true the next form will be passed through the reader and evaluated, otherwise it will be discarded.The patches attached to the CLJS-27 ticket contain a proof of concept implementation of these feature expressions for Clojure and ClojureScript. With this extension one can branch on a form level an write code like illustrated by the following exampleis a set of keywords to indicate supported features. 

The Reader understands a new kind of "feature expression". The reader macros #+ and #- are used to include or skip a form based on a feature expression:

  • #+feature-expr form - include next form if feature-expr evaluates to true
  • #-feature-expr form - skip next form if feature-expr evaluates to true

Feature expressions evaluate as booleans and are defined as follows:

  • feature - a symbol, evaluated as if:

    Code Block
    (contains? *features* (keyword feature))
  • (not feature-expr) - returns logical not of feature-expr
  • (and feature-expr*) - returns true if all feature-expr are true, otherwise false.
  • (or feature-expr*) - returns true if any of its feature-expr are true, otherwise false.

Skipping in the reader is performed by binding *read-suppressed* to true for the next form.

Example:

No Format
(ns feature.expressions
  #+clojurescriptcljs (:require [goog.string :as gstring]))

(defn my-trim [s]
  #+clojureclj (.. s toString trim)
  #+clojurescriptcljs (gstring/trim s))

(my-trim " Hello CL? ")

The patches add a dynamic variable called *features* to the clojure.core and cljs.core namespaces, that should contain the supported features of the platform in question as keywords. Unlike in Common Lisp, the variable is a Clojure set and not a list. In Clojure the set contains at the moment the :clojure keyword, and in ClojureScript the :clojurescript keyword.

I would like to get feedback on the following issues: 

  • Are those keywords ok? Is :jvm for Clojure and :js for ClojureScript better? 
  • Should ClojureScript add something like :rhino, :v8 or :browser as well?
  • Someone mentioned that this should support Clojure's namespaces. How should this be done?
  • How should the compilation process work with this extension? 
  • Will the ClojureScript compiler read *.clj files?  What happens to *.cljs files?

To run the ClojureScript tests, drop a JAR named "clojure.jar" that has the Clojure patch applied into ClojureScript's lib directory.

 

Reading Unknown Tagged Literals

There may be tagged literals or classes that are not known or available on all platforms. The reader must be able to read but avoid constructing these entities when they are in excluded feature expressions.

Example:

Code Block
(def init #+cljs #js {} 
          #-cljs nil)

The #js tagged literal is not known on the Clojure platform, but the reader should read and avoid trying to instantiate it.

ClojureScript File Loading

When code needs to be loaded based on a namespace, ClojureScript will first look for .cljs files, then for .clj files. 

ClojureScript currently expects source files to end (only) in .cljs. Changes must be made in several places to allow reading .clj files:
  • cljs.compiler/rename-to-js - renames .cljs file name to .js file name. 
    • regex must cover both .clj and .cljs
  • cljs.compiler/cljs-files-in - given a directory, finds all .cljs files to compile
    • can grab .clj too, but may include macro files not previously included - how do we distinguish here??
    • called-by backtraces:
      • cljs.repl/analyze-source
        • cljs.repl.browser/-setup - to support code reflection in the browser repl
        • cljs.repl/repl - to support code reflection in the repl, populates :cljs.analyzer/namespaces in compiler-env
      • cljs.compiler/compile-root
        • cljs.closure/compile-dir
        • cljs.closure/-compile (protocol Compilable on File)
          • cljs.closure/build
            • cljsc - main tooling entry point (often used to compile a directory of cljs)
  • cljs.analyzer/ns->relpath - given a namespace, find the resource path
    • check first for .cljs path, then .clj path (adds resource check where prior was simple string manipulation)
  • cljs.closure/cljs-source-for-namespace - given a ns, find path and url of ns

    • check first for .cljs resource, then .clj resource
  • cljs.repl.rhino/goog-require - given ns, makes cljs resource path

    • check first for .cljs resource, then .clj resource
  • cljs.repl.browser/send-static - serves static resources

    • add support for serving .clj as well as .cljs
 
ClojureScript Tooling Support

ClojureScript tooling needs to be aware of the change in supported file extension names and possibly new build options. Need to ensure tools are able to support the new mixed-language projects well:

  • lein-cljsbuild - most ClojureScript projects today are built with the lein-cljsbuild plugin. 
    • compiler.clj - needs to properly account for ClojureScript .clj files
    • features - might need to add support for specifying non-default feature set when testing
  • austin?
  • what else?

 

Patches

JIRA Tickets and patches:

 

Building Portable Libraries

Case 1: Independent CLJS and CLJ files packaged in the same jar
 
Some libraries may provide independent implementations of a library for both CLJ and CLJS and be packaged in the same jar. In this case, namespaces should not overlap between CLJ and CLJS:
 

 

Code Block
myproj/clj/mylib.clj
myproj/cljs/mylib.cljs

 

Case 2: Mixed portable and non-portable CLJS and CLJ packaged in the same jar
 

 

Code Block
myproj/mylib.clj - portable code with feature expressions - both CLJ and CLJS load myproj.mylib
myproj/foo.clj - helper clojure ns
myproj/bar.cljs - helper clojure ns

 

Case 3: Library designed for open extension
 
A library designed for open extension isolates platform-specific code in one or more namespaces. The library code may compile against (but not include) a stub namespace or use a .clj mixed mode file to contain a default library implementation. The non-isolated code should all be .clj files that are portable across platforms.

 

Code Block
myproj/mylib.clj 
myproj/open.clj - a ns used by mylib but designed to be overridden or replaced by a platform-specific version of the ns

 

A separate jar would supply the platform-specific version of the ns:

 

Code Block
myproj/open.cljs

 

Other additional jars would be used to supply other platform-specific versions of the ns. A consumer would use the first jar + the selected extension jar.

Alternate Approaches

Copy / paste

One approach is to maintain two versions of the same file that are largely the same but modify the platform-specific parts in each copy. This obviously works but is gross.

cljx

cljx is an implementation of feature expressions that:

  • rewrites .cljx files containing feature expression-annotated code into external files based on well-known tags
    • The most common use is to use clj and cljs tags and write .clj and .cljs files for consumption by other tools/compilers/etc
  • optionally applies the same transformation interactively via installation of a REPL extension

It has been used successfully by a number of projects (see the cljx README for a partial list).  cljx's limitations include:

  • It does not address portability of macros at all; it is strictly a source-to-source transformation.  Macros continue to be written in Clojure, and must be rewritten or implemented conditionally on the contents of &env.
  • It does not provide any runtime customization of the "features" you can use; these are set either within build configuration (via cljx' Leiningen integration), or via the configuration of the cljx REPL extension.  The latter technically is available for modification, but is not in practical use.
  • The set of provided "features" is limited to one for Clojure (#+clj) and one for ClojureScript (#+cljs).  Further discrimination based on target runtime (e.g. rhino vs. node vs. v8 vs. CLR) would be trivial, but has not been implemented to date. 

cljx expressions are typically applied:

  • Inside ns macro
  • Top-level forms
  • Occasionally internal forms where it's concise
    • (.getTime #+clj (java.util.Date.) #+cljs (js/Date.))
lein-cljsbuild "crossovers"

lein-cljsbuild provides a (deprecated, to be removed) feature called "crossovers" that provides a very limited preprocessing of certain files during the cljsbuild build process; a special comment string is removed, allowing one to work around the -macros declarations required in ClojureScript ns forms.  Crossover files must otherwise be fully portable.  Language/runtime-specific code must be maintained in separate files.  However, (my) experience shows that this can quickly lead to the situation where one has to think a lot about in which file to put a specific function, in order to go though the whole preprocessing machinery. Functions are split into namespaces because of conditional compilation, and not because they belong to the same part or module of the program.

Tagged Literal

Define a custom tagged literal that implements conditional read-time expressions:

#feature/condf [ (and jdk-1.6+ clj-1.5.*) 
 (call-my-fast-reducer-code) 
 else (some-old-fashioned-code) ]

Proof of concept here:  https://github.com/miner/wilkins

References

The Common Lisp Hyperspec about the Sharp Sign macros:

...

Maintaining Portable Lisp Programs:

...

ClojureScript JIRA Ticket and patches with a proof of concept implementation of CL's feature expressions: