Versions Compared

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.

Problem

There are Clojure dialects (Clojure, ClojureScript, ClojureCLR) hostd hosted on several different platforms. We wish to write libraries that can share as much portable code as possible while leaving flexibility to provide platform-specific bits as needed, then have this code run on all of them.

Use cases for platform-specific functionality currently handled by cljx:

Use caseExample (cljx)Example (feature expr)
Platform-specific require/import

...

In .cljx file which is preprocessed into .clj and .cljs files:

Code Block
(ns cemerick.pprng
 #+cljs (:require math.seedrandom
 [cljs.core :as lang])
 #+clj (:require [clojure.core :as lang])
 #+clj (:import java.util.Random)
 (:refer-clojure :exclude (double float int long boolean)))

In .clj file loadable in both Clojure and ClojureScript:

Code Block
(ns cemerick.pprng
 #+cljs (:require math.seedrandom
 [cljs.core :as lang])
 #+clj (:require [clojure.core :as lang])
 #+clj (:import java.util.Random)
 (:refer-clojure :exclude (double float int long boolean)))

Exception handling, catching "all" - see issue below on why this is not a complete solution in either case.

In .cljx file which is preprocessed into .clj and .cljs files:

Code Block
(try
 (>! c msg)
 (catch #+clj Exception
        #+cljs js/Object 
        e)
    .... )) 

In .clj file loadable in both Clojure and ClojureScript:

Code Block
(try
 (>! c msg)
 (catch #+clj Exception
        #+cljs js/Object 
        e)
 .... )) 

 

Platform-specific calls for "hosty" library things (strings, math, dates, random numbers, uris, reflection warnings

...

Proposal: Feature expressions

)

In .cljx file which is preprocessed into .clj and .cljs files:

Code Block
;; dates
 (.getTime 
  #+clj (java.util.Date.)
  #+cljs (js/Date.))

;; uris 
#+clj
(defn url-encode
 [string]
 (some-> string str (URLEncoder/encode "UTF-8") (.replace "+" "%20")))
#+cljs
(defn url-encode
 [string]
 (some-> string str (js/encodeURIComponent) (.replace "+" "%20")))

;; reflection warnings
#+clj (set! *warn-on-reflection* true)   

 

 

In .clj file loadable in both Clojure and ClojureScript:

Code Block
;; dates
 (.getTime 
 #+clj (java.util.Date.)
 #+cljs (js/Date.))

;; uris   
#+clj
(defn url-encode
 [string]
 (some-> string str (URLEncoder/encode "UTF-8") (.replace "+" "%20")))
#+cljs
(defn url-encode
 [string]
 (some-> string str (js/encodeURIComponent) (.replace "+" "%20"))) 

;; reflection warnings
#+clj (set! *warn-on-reflection* true)   

 

 

 

Protocol invocation (CLJS internal protocol names start with -, CLJ interfaces do not)

In .cljx file which is preprocessed into .clj and .cljs files:

Code Block
#+cljs
(deftype With [name unique-name body value bang]
 SpecComponent
 (install [this description]
 (swap! (.-withs description) conj this))
 cljs.core/IDeref
 (-deref [this]
 (when (= ::none @value)
 (reset! value (body)))
 @value))

 (See "-deref" in CLJS vs "deref" in CLJ.)

 

In .clj file loadable in both Clojure and ClojureScript:

Code Block
#+cljs
(deftype With [name unique-name body value bang]
 SpecComponent
 (install [this description]
 (swap! (.-withs description) conj this))
 cljs.core/IDeref
 (-deref [this]
 (when (= ::none @value)
 (reset! value (body)))
 @value))
Extending protocols to implementation-specific classes

In .cljx file which is preprocessed into .clj and .cljs files:

Code Block
#+clj
(extend-protocol SpecComponent
 java.lang.Object
 (install [this description] ...)) 

In .clj file loadable in both Clojure and ClojureScript:

Code Block
#+clj
(extend-protocol SpecComponent
 java.lang.Object
 (install [this description] ...)) 

Proposal summary

The feature expression proposal includes the following items:

WhereItemDescriptionExample
allAdds *features* dynamic variable

Set of keywords indicating supported features. Can be changed in a binding around reading. The feature set is open and can be dynamically modified in binding or modified on startup. Non-namespaced features are reserved by the platform. User features should be namespaced.

allDefines well-known platform feature namesEach platform guarantees that *features* includes the platform feature.clj, cljs, clr
CLJAdds system property to set custom features at startupUse system property clojure.features on JVM startup to specify additional custom features. The platform feature will always be included, regardless of this setting.-Dclojure.features=arch/osx
CLJSAdds build option to set custom features at build timeUse build option :features which should be a set of keywords. The platform feature will always be included, regardless of this setting.:build #{:js/node}
allNew reader syntax for feature include or exclude

Both feature include (#+) and exclude (#-) include a conditional expression and a form. The conditional expression is made from features (boolean value based on membership in feature set), and, or, and not. This is the same syntax used in CL feature expressions and cljx.

(defn my-trim [s]
  #+clj (.. s toString trim)
  #+cljs (gstring/trim s))
allExcluded tagged literals do not need to have a data readerIt is allowed for an excluded feature to use an undefined tagged literal. For example, the #js tagged literal is not defined in Clojure but may used in a #+cljs feature expression.
(def init
  #+cljs #js {}
  #+clj nil)
CLJSAllow .clj extension for ClojureScript source filesBoth .cljs and .clj are now acceptable file extensions for CLJS. Portable files that can be loaded in either Clojure or ClojureScript (typically using feature expressions) must have the .clj extension (or they cannot be loaded in Clojure). 

Issues

Below are important issues considered and current status.

NameDescriptionProposedStatus
Version checksShould feature expressions support broader kinds of expressions (JDK or Clojure version ranges, etc)? Not included
ClojureCLR delimited symbolsClojureCLR supports a wider set of symbols via a |-delimited extension which cannot be read if in feature expressions from the CLJ or CLJS readers.Add support for |-delimited symbols in CLJ, CLJS. Issues involved relate toNot included (reconsider in future)
Portable "catch all" errors

It is currently difficult even with feature expressions to easily "catch all" in CLJS (like catch Throwable).

Note: This is orthogonal to feature expressions but important to the greater goal of portable code.

Proposal exists at Platform Errors with related tickets CLJS-661 and CLJ-1293.OPEN
CLJS namespace collisionsExisting code that uses a particular ns for macros (.clj file) may have conflicts if they use the same .clj for Clojure code in the same library.This scenario is not too common and must be resolved by using non-overlapping namespaces for macros and ClojureScript code in a namespace.No change needed
CLJS file loadingclj.cljs.compiler/cljs-files-in will find all files recursively under a directory via file extension. Expanding this to include .clj will snare .clj macro files. This code is used by the cljsc tool and both command line and browser repl. lein cljsbuild has similar issues.See next section.OPEN
Tooling on raw source of portable files is more complicated.Editors and other tools that work on the raw source files cannot determine a specific dialect of Clojure based on the file extensions. This makes it more complicated to enable advanced editor features that require analyzing code. See Colin Fleming's comments (re Cursive).???OPEN

CLJS File Loading Issue

Adding .clj files as valid file extensions means that parts of ClojureScript that currently specify a directory of .cljs files will include any .clj macro files already in that source tree and interpret them as .clj source files.

Some alternatives and their impact to existing file types or code:

AlternativeDescriptionCLJS .cljsMacro .cljCLJS .cljCLJS buildCLJS ToolsCLJ Tools
Separate macro pathSpecify separate paths for ClojureScript code and macros. The .clj files in the code path are then unambiguously CLJS code.-Move to new macro dir-Pass macro path through stackNeed additional macro path-
Exclude namespacesSpecify a build option to exclude a set of namespaces from compilation. Macros would be explicitly specified there and could then be intermingled but ignored when pulling CLJS source from a directory.---Pass excluded ns list through stackNeed list of excluded macro ns'es-
Mark macro namespacesSpecify ns meta or other marker in .clj macro files so that they can be excluded based on file inspection.-Mark as macro ns-Needs to parse all .clj files to check if macro ns--
Mark regular portable namespacesSpecify ns meta or other marker in .clj source files so that they can be included based on file inspection.--Mark as portableNeeds to parse all .clj file to check if portable ns-?
Specify a portable file extensionDefine a new unambiguous file extension to indicate a portable source file to be read in multiple host envs. (The .cljx extension is somewhat analogous although it only exists during preprocessing.)

-

-New file extensionPull in additional extension files as CLJS sourceLook for additional extensionLook for additional extension

 

Detailed proposal

Feature Sets

The platform feature will be one of: cljcljs, or clr.  Users may supply their own features, which should, by convention, use namespaces. Platform-specified features will not be namespaced.

In Clojure, the initial feature set may be specified at start time with the system property clojure.features, which is a comma-delimited list of symbols to add as features. The platform feature will always be added, regardless of whether it is set in the system property. 

For example:

Code Block
-Dclojure.features=arch/osx,my.app/prod,my.app/strictmath

will yield the runtime feature set: #{:clj :arch/osx :my.app/prod :my.app/strictmath}

In ClojureScript, there is a new build option with key :features that takes a set of keywords defining the features. The platform feature :cljs is always added to this set, as in Clojure.

In addition to setting the features initially, users may bind *features* around explicit calls to the reader.

Feature Expression Syntax

Common Lisp approaches this problem using feature expressions.

Each platform has a variable called *features* that is a set of keywords to indicate supported features. 

...

No Format
(ns feature.expressions
  #+cljs (:require [goog.string :as gstring]))

(defn my-trim [s]
  #+clj (.. s toString trim)
  #+cljs (gstring/trim s))

(my-trim " Hello CL? ")

Features

The platform feature will be one of: clj, cljs, or clr.  Users may supply their own features, which should, by convention, use namespaces. Platform-specified features will not be namespaced.

In Clojure, the initial feature set may be specified at start time with the system property clojure.features, which is a comma-delimited list of symbols to add as features. The platform feature will always be added, regardless of whether it is set in the system property. 

For example:

Code Block
-Dclojure.features=arch/osx,my.app/prod,my.app/strictmath

will yield the runtime feature set: #{:clj :arch/osx :my.app/prod :my.app/strictmath}

In ClojureScript, there is a new build option with key :features that takes a set of keywords defining the features. The platform feature :cljs is always added to this set, as in Clojure.

In addition to setting the features initially, users may bind *features* around explicit calls to the reader.

Reading Unreadable ThingsReading Unknown Tagged Literals

There may be tagged literals or classes that are not known or available on all platforms. The reader must be able to read but avoid constructing these entities when they are in excluded feature expressions.

Example:

Code Block
(def init #+cljs #js {} 
          #-cljs nil)

The #js tagged literal is not known on the Clojure platform, but the reader should read and skip it without failure.

Potential issue: Clojure CLR needed to expand the valid set of symbols and uses the #| reader extension from Common Lisp to support this in the CLR. The corresponding symbols are not currently readable by Clojure or ClojureScript.  The #| extension support is described here: https://github.com/clojure/clojure-clr/wiki/Specifying-types. The #| reader extension may need to be supported in Clojure and ClojureScript to support the full set of valid ClojureCLR symbols in feature expressions.

...

avoid trying to instantiate it.

ClojureScript File Loading

When code needs to be loaded based on a namespace:

...

ClojureScript will first look for .cljs files, then for .clj files.

...

Potential ClojureScript issue: existing projects may have macro .clj files in the same namespace where a file with shared code should go. In this case, the project will need to be refactored, likely with the macros in a different location.

Open Extension

One consequence of feature expressions is that a library must encode solutions for all of the supported platforms in the source code at packaging time (in other words, extension is closed to external users).
 
An alternate solution to specifying all variants in a single file with feature expressions is to swap whole namespaces in and out for different platforms. This allows others to provide implementations for different platforms (if, for example, the library author lacked the expertise to do so). This strategy is orthogonal to feature expressions but made possible by the extension preference checking as specified in the previous section.

Example:

No Format
Library files:
 src/my/core.clj - ns requires my.thing
 src/my/thing.clj - Clojure implementation of my.thing (loaded by Clojure)
 src/my/thing.cljs - ClojureScript implementation of my.thing (loaded by ClojureScript)

...

 

ClojureScript currently expects source files to end (only) in .cljs. Changes must be made in several places to allow reading .clj files:
  • cljs.compiler/rename-to-js - renames .cljs file name to .js file name. 
    • regex must cover both .clj and .cljs
  • cljs.compiler/cljs-files-in - given a directory, finds all .cljs files to compile
    • can grab .clj too, but may include macro files not previously included - how do we distinguish here??
    • called-by backtraces:
      • cljs.repl/analyze-source
        • cljs.repl.browser/-setup - to support code reflection in the browser repl
        • cljs.repl/repl - to support code reflection in the repl, populates :cljs.analyzer/namespaces in compiler-env
      • cljs.compiler/compile-root
        • cljs.closure/compile-dir
        • cljs.closure/-compile (protocol Compilable on File)
          • cljs.closure/build
            • cljsc - main tooling entry point (often used to compile a directory of cljs)
  • cljs.analyzer/ns->relpath - given a namespace, find the resource path
    • check first for .cljs path, then .clj path (adds resource check where prior was simple string manipulation)
  • cljs.closure/cljs-source-for-namespace - given a ns, find path and url of ns

    • check first for .cljs resource, then .clj resource
  • cljs.repl.rhino/goog-require - given ns, makes cljs resource path

    • check first for .cljs resource, then .clj resource
  • cljs.repl.browser/send-static - serves static resources

    • add support for serving .clj as well as .cljs
 
ClojureScript Tooling Support

ClojureScript tooling needs to be aware of the change in supported file extension names and possibly new build options. Need to ensure tools are able to support the new mixed-language projects well:

  • lein-cljsbuild - most ClojureScript projects today are built with the lein-cljsbuild plugin. 
    • compiler.clj - needs to properly account for ClojureScript .clj files
    • features - might need to add support for specifying non-default feature set when testing
  • austin?
  • what else?

 

Patches

JIRA Tickets and patches:

 

Building Portable Libraries

Case 1: Independent CLJS and CLJ files packaged in the same jar
 
Some libraries may provide independent implementations of a library for both CLJ and CLJS and be packaged in the same jar. In this case, namespaces should not overlap between CLJ and CLJS:
 

 

Code Block
myproj/clj/mylib.clj
myproj/cljs/mylib.cljs

 

Case 2: Mixed portable and non-portable CLJS and CLJ packaged in the same jar
 

 

Code Block
myproj/mylib.clj - portable code with feature expressions - both CLJ and CLJS load myproj.mylib
myproj/foo.clj - helper clojure ns
myproj/bar.cljs - helper clojure ns

 

Case 3: Library designed for open extension
 
A library designed for open extension isolates platform-specific code in one or more namespaces. The library code may compile against (but not include) a stub namespace or use a .clj mixed mode file to contain a default library implementation. The non-isolated code should all be .clj files that are portable across platforms.

 

Code Block
myproj/mylib.clj 
myproj/open.clj - a ns used by mylib but designed to be overridden or replaced by a platform-specific version of the ns

 

A separate jar would supply the platform-specific version of the ns:

 

Code Block
myproj/open.cljs

 

Other additional jars would be used to supply other platform-specific versions of the ns. A consumer would use the first jar + the selected extension jar.

Alternate Approaches

Copy / paste

...