Problem
Writing programs that target Clojure and ClojureScript involves a lot of copy and pasting. The usual approach is to copy the whole code of one implementation to a source file of the other implementation and to modify the platform dependent forms until they work on the other platform. Depending on the kind of program the platform specific code is often a fraction of the code that works on both platforms. A change to platform independent code requires a modification of two source files that have to be kept in sync. To solve this problem branching by target platform on a form level would help a lot.
See also ticket CLJS-27.
Current Solutions
Tools like lein-cljsbuild provide some help using "crossover" files. Crossover files are usually written in Clojure and contain only code that works on both platforms. On compilation the crossover files are run through a preprocessor that removes the special comment ;*CLJSBUILD-REMOVE*; from the source file to allow platform specific references to macros. This is explained in detail over here:
This works quite well, but is probably not the right granularity. This approach requires one to split a project into platform specific files, which is not a bad practice per-se. However, (my) experience shows that this can quickly lead to the situation where one has to think a lot about in which file to put a specific function, in order to go though the whole preprocessing machinery. Functions are split into namespaces because of conditional compilation, and not because they belong to the same part or module of the program.
As mentioned above the granularity where this solution solves the problem is probably not at the right level. Having only one function with a platform specific form puts one into the business of splitting it into two files maintaining two different versions of the function. A branching granularity on a form level would probably be much better.
Potential Solution
A solution to this problem that is used by Common Lisp implementations are feature expressions. Each platform has a variable called *features* that contains keywords that indicate the supported features of the platform the code is running under. The branching on a platform or a platform specific feature is done via the reader macros #+ and #- followed by a feature condition. The feature condition is either a symbol or a form that combines symbols with the or, and or not operators. The feature condition is evaluated by looking up the symbols in the *features* variable. If the feature condition evaluates to true the next form will be passed through the reader and evaluated, otherwise it will be discarded.
The patches attached to the CLJS-27 ticket contain a proof of concept implementation of these feature expressions for Clojure and ClojureScript. With this extension one can branch on a form level and write code like illustrated by the following example:
(ns feature.expressions #+clojurescript (:require [goog.string :as gstring])) (defn my-trim [s] #+clojure (.. s toString trim) #+clojurescript (gstring/trim s)) (my-trim " Hello CL? ")
The patches add a dynamic variable called *features* to the clojure.core and cljs.core namespaces, that should contain the supported features of the platform in question as keywords. Unlike in Common Lisp, the variable is a Clojure set and not a list. In Clojure the set contains at the moment the :clojure keyword, and in ClojureScript the :clojurescript keyword.
I would like to get feedback on the following issues:
- Are those keywords ok? Is :jvm for Clojure and :js for ClojureScript better?
- Should ClojureScript add something like :rhino, :v8 or :browser as well?
- Someone mentioned that this should support Clojure's namespaces. How should this be done?
- How should the compilation process work with this extension?
- Will the ClojureScript compiler read
*.cljfiles? What happens to*.cljsfiles?
To run the ClojureScript tests, drop a JAR named "clojure.jar" that has the Clojure patch applied into ClojureScript's lib directory.
References
The Common Lisp Hyperspec about the Sharp Sign macros:
Examples of Common Lisp's Feature Expresions:
- http://www.lispworks.com/documentation/lw50/CLHS/Body/24_abaa.htm
- http://www.gigamonkeys.com/book/practical-a-portable-pathname-library.html
Maintaining Portable Lisp Programs:
Crossover files in lein-cljsbuild:
ClojureScript JIRA Ticket and patches with a proof of concept implementation of CL's feature expressions:
26 Comments
Hide/Show CommentsJul 22, 2012
Kevin Downey
I am not wild about the conditional stuff happening at read time, it means the conditionals have no data representation and cannot be manipulated as such
Jul 22, 2012
Stuart Halloway
Kevin,
Can you elaborate with an example?
Jul 27, 2012
Kevin Downey
with this design if you read
on clojure you get the list (+ 1 2) and on clojurescript the list (+ 2 3) and there is no indication that a compile time conditional was read. this means the conditionals will not be visible to anything that operates on datastructures rather than text.
how will macros be shared? I guess they would check `*features*` directly to decide if they want to emit something different for clojure or clojurescript
another thing is there is no indication that the above is basically a cond, besides the forms being written next to each other. compare to something like:
it is possible that we have to have conditionals at read time because some forms are readable on one host but another (vars? syntax quote in the reader does some class lookups I think, on the otherhand, syntax quote should be rewritten as a macro)
if issues with readable/unreadable forms are addressed between hosts, feature-cond above could be a macro that checks `*features*` so no change to the compiler or reader
Mar 05, 2013
Brandon Bloom
At first, I wanted to agree with you that it would be nicer to just have a Macro rather than introducing a new feature into the reader. However, after more thought, I think that this absolutely needs to occur at read time. The main reason is that, if it's a macro, then it will be visible to outer macro forms when it shouldn't be. Consider some macro which expects a symbol, but instead gets a (feature-cond ...) form. It will (assert (symbol? ...)) and fail, even though the feature-cond form might expand to a symbol, the outer macro has no way of knowing.
It seems like the expected thing is for this to happen almost equivalent to the C preprocessor, so the reader is the right place to do that.
Jul 27, 2012
Timothy Baldridge
The complaint I have with this feature is that now all the separate implementations of a given expression are housed in a single file. This closes off the system, disallowing future expansion (without getting a patch submitted upstream to the library provider). This gets to be a bigger issue when we start porting libraries to other platforms.
What I'd like to see is some sort of "patch" based system. I'd like to see some way where we can put JVM specific code into a single source file, and then allow users of the library to replace those JVM versions with CLJS versions, CLR, Python, or C versions as needed.
The common lisp method shown here is simple, but I have to say, it feels like a hack.
Jul 27, 2012
Kevin Downey
load-file is an expression, so any kind of feature expressions would allow for providing implementations in different files
Mar 05, 2013
Brandon Bloom
That assumes you have a runtime compiler, which ClojureScript does not. You'd need to run load-file in the context of the compiler, but ClojureScript (currently) emits top-level forms as JavaScript, rather than running them in the compiler's environment.
Jul 23, 2012
Roman Scherer
I received an email from Kevin Lynagh. He could not write to the wiki, so I post it here:
Hi Roman,
Jul 27, 2012
Stuart Halloway
Jul 28, 2012
Brandon Bloom
> Are those keywords ok? Is :jvm for Clojure and :js for ClojureScript better?
> Should ClojureScript add something like :rhino, :v8 or :browser as well?
:jvm isn't a good idea for Clojure, since :rhino is also on the JVM. If you were writing some ClojureScript code, and wanted to test for a Java package, you'd want to check for :jvm. Similarly, you'd want a :gclosure member as well, if there was ever an implementation that targeted JS without it.
I'd imagine the Clojure set could look like #{:clojure :java :jvm}
And the ClojureScript one could be #{:clojurescript :js :gclosure} potentially conj :jvm for Rhino
Because it will soon be possible to have #{:clojurescript :lua}
This raises an interesting point: Why a set? Why not a map? Here's an example Clojure map:
{:dialect :clojure, :clojure {:version "1.2.3"}, :target :java, :runtime :jvm, :jvm {}}
And one for ClojureScript:
{:dialect :clojurescript, :clojurescript {:version "2.3.4"}, :target :js, :js {}, :runtime :browser} ; maybe something for :gclosure version
This way the #+clojurescript and #-js things will still work like set memberships. We could come up with some way to include an arbitrary predicate against map values, or we could leave that to be feature not available to the reader, only to macros against *features*.
A few issues: Since this happens at read time, it may be too late to decide :v8 vs :rhino, etc. Although, some JavaScript shops/frameworks/teams/whatever do multi-compile sources for different browsers. ie. Do conditional logic by browser version, server side, rather than client side, serve up the browser-specific javascript. In that case, you even could support a :browser key with the various information there.
I feel like this needs some kind of cond macro. I should be able to check for "do I have a runtime that optimizes foo? if so, do this, otherwise, do it the slow way like this" I realize that I could use #+foo, then #-foo, but what if I had two possible optimizations? I'd need #+foo(x) #+bar(y) #-foo(#-bar(y)) ;; seems like I need some kind of "else" expression.
RE: read-time vs macro-expansion-time. Seems like the right thing for this is read-time because of symbol resolution, but we should also consider some compile-time macro support against the same dynamic var. In theory, it should be workable with all the standard macros and stuff, since it's just a set or map, but we should still explore it a bit, so we know we're not missing something.
Aug 01, 2012
Roman Scherer
I asked the Common Lisper's for some input over here:
https://groups.google.com/forum/?hl=en&fromgroups#!topic/comp.lang.lisp/x2k3XbW3LmA
Elias Mårtenson responded to one of my questions on
comp.alt.lisp. Here's what he wrote:
I can address one of the quest, whether they are better
implemented as a macro. The answer to this is clearly no. In
fact, it couldn't be implemented as a macro, because at the time
the macro is evaluated, the symbols making up the expression has
already been interned by the reader. Thus, if the form references
any packages that does not exist in your environment, you would
get an error in the reader, before your conditional expression
has had the chance to run.
A somehow related thread I found on this topic is here:
https://groups.google.com/forum/?hl=en&fromgroups#!topic/comp.lang.lisp/CBB4hzqRCS8
What are the next steps?
Aug 01, 2012
Kevin Downey
I would be careful conflating those issues with common lisp with clojure. clojure's reader is not the same as common lisp's and clojure's symbols are not the same as common lisp's. I think the only part of the clojure reader that has those kind of issues in syntax quote, which tries to resolve classes.
Aug 25, 2012
Brandon Bloom
Kevin is right. Try this out, it works:
(def ^:dynamic *features* #{'clojure})
(defmacro feature-cond [& exprs]
`~(-> (filter (fn [[feature expr]]
(contains? *features* feature))
(partition 2 exprs))
first
second))
(feature-cond
clojure (def x 1)
clojurescript (def y 2))
(feature-cond
clojurescript (* y 3)
clojure (* x 3))
Aug 02, 2012
Hugo Duncan
I would like to see the feature system being open to use by libraries. In pallet, we recently introduced features to be able to write providers that work with multiple versions of pallet, and I think it would make sense to replace this with whatever feature system ends up in clojure.
The current proposal seems to support this, but it would be good to clarify this as a requirement of the feature system.
Aug 05, 2012
Brandon Bloom
Again, if 'features (or whatever it is called) is a map, then you might be able to do something like (load-file (str "foo/" (:dialect features) ".clj")) or something like that... That wouldn't be possible as a bit-flag set.
Aug 23, 2012
Stuart Halloway
What problems, if any, could this change cause for existing code? Obviously pre-feature-expression code will be unable to read the forms. In particular:
Aug 26, 2012
Kevin Downey
feature-cond above would be readable in previous versions of clojure, and even other lisps for whatever that is worth.
Mar 06, 2013
Brandon Bloom
What if we provided read-time variants of unquote and unquote-splicing?
Here's a splicing example:
The code inside the #~ and #~@ forms would be run at read-time and subject to *read-eval*. This gives you the full power of Clojure for conditional compilation at read time. We could provide some standard function and macro utilities for conditional compilation that are available both at read and macro expansion time. Allowing for something like:
This proposal gives you the full power of Clojure for controlling conditional reading and compilation without adding dramatically new ideas (it's basically just syntax-quote at read-time).
May 06, 2013
Herwig Hochleitner
I love the consistency and minimalism of this approach.
One Issue however: It probably won't be possible to read-splice on the toplevel. Consider: What should (read-str "#~@(range 5)") yield?
I'm still +1 on this one and just forbid toplevel splices.
May 08, 2013
Kevin Downey
have you looked at what it would take to allow the reader to splice in forms at any point? with syntax quote it is a little easier because the expression has to be synax quoted to begin with before you can splice in to it.
May 09, 2013
Brandon Bloom
I guess at the top level, a splice would just assume an implicit "do".
May 09, 2013
Kevin Downey
the top level is the easiest case though, the gnarly cases are splicing in to nested data structures, that may be spliced together themselves.
May 09, 2013
Brandon Bloom
Oh, I though you were asking with respect to Herwig's comment.
Doesn't syntax quote already have this problem? Try `{1 ~@(list 2 3 4)} for example. You can do that, but you can't do `{1 2 ~@(list 3 4)}
Seems like a separate issue all together.
May 09, 2013
Kevin Downey
syntax quote at least can limit the scope of splicing and unsplicing to syntax quoted forms. if any recursive call to read can (some how?) returning multiple values I think the entire reader would be much more complex. I have no evidence of that, which is what I asked if you had looked in to what it would take.
I have no doubt it is possible to do, I just don't thing I'd care for the result
May 09, 2013
Kevin Downey
pushing conditionals in to the reader does not save us from requiring the code to be readable on all platforms. it has to be readable for the reader to even recognize it as a form to throw away
May 11, 2013
Herwig Hochleitner
I guess recursive invokations of read could be replaced with read-seq-of-forms + concat et al.
That would buy us read-splice, but: Is it worth the performance cost? If yes, should read-seq-of-forms be made public?
I'm kind of meh on (= (read-str "#~@(range 5)") '(do 1 2 3 4 5))
If I want a do, I'll write (a regular form generating) it. `~@ sets a nice precedent.
Those complications make me wonder if #~@ serves any conceivable purpose, that #~ together `(~@) can't (or shouldn't) serve.
True, this should be an intended design constraint. We want to have the same basic clojure syntax on every platform, i.e. a very small superset of edn, right?
The reason we even need conditionals in the reader is that the reader already does some possibly platform-dependent side-effects, i.e. reader tags, defrecord instances and friends.
So let's just KISS the syntactic considerations goodbye for a moment:
That gets us to the real meat of the problem:
Say we have:
Under current reader rules, would the #platform tag have a chance to remove the form before the reader tried to apply the #asm tag?
I suspect that's not the case, which makes me wonder if it's worth changing the reader to allow reader tags to omit reader tags from their child form. This would imply the ability to generate new reader tags (and other reader side effects) in code a reader tag emits.
I also suspect that if we try to just allocate a couple of new reader table characters and work downwards from there, we will end up with pretty much the same code it would take to apply reader tags in a second pass, except it would be less general purpose.
We still might not want to expose it through reader tags, since it would alter their semantics on the input side. Opinions on this one?