Versions Compared

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.

 

UPDATE 06052013 - Herwig

Updated the page with input from David Nolen, the description of the prototype is left intact at the end of the page.

I concur with giving specify an extend-like interface, which can be passed functions.

UPDATE 06052013-2 - Herwig

I pushed a commit to github that makes most of clojurescript runnable again with the refactoring: https://github.com/bendlas/clojurescript/commit/5d827ee8f11b2a575dff2647f7321bc2c47db1de

There is still an issue with accessing locals from reify, when implementing IFn. Due to the call method hack.

Added replys to comments.

Problem statement

The actual prototype of js objects is not settable. Thus it's not possible to assign a protocol implementation a live objects, e.g. input from jsonp.

The fields needed to let a javascript object participate in a protocol could simply be assigned, however the algorithm to get the name-mangled protocol fields is woven into extend-type, tighly coupled to target the .protocol property.

Name, Interface?

Even with namespacing extend-type is a global modification of a Clojure program. While powerful, this feature comes at some cost to modularity. On dynamic hosts like JavaScript, Clojure can provide an efficient modification of individual instances so that they may participate in a protocol. Such a feature would have the additional benefit of actually being more general than extend-type - that is the global modification (via JavaScript prototypes in the case of ClojureScript) is a special case of instance modification.

Proposal

Add two new macros: specify and specify!. Both have a syntax similar to clojure.core/extend, but take a target object in place of the type and are macros.

specify! will mutate the target object by assigning protocol methods as keys. This is mostly useful to create new prototypes, but can occasionally be useful for external objects from libraries or eval.

specify will clone the target object before specify! ing the clone.

Code Block
(specify! x
 IFoo 
 {:-bar existing-fn
  :-baz (fn ([x] ...) ([x y] ..)})

The function implementing the method (existing-fn) will be called method-fn in this.

Costs

Giving specify an interface like extend has following difficulties

  1. Since there is one slot per method, you need to pass a multi-arity fn when implementing a multi-arity method.

    In a naive implementation, that would cost another arity dispatch, which already has been done at the method call site.

    This can be mitigated by having specify reuse the existing infrastructure for arity optimization: properties on the fn object called .cljs$lang$arity$*

    specify can peel off those properties and use them as methods, which leads to the next cost
  2. A redundant fn object when implementing inline (the one that would do the arity dispatch otherwise)

    this could be mitigated by having specify recognize fn forms and emitting the arity-methods directly 
  3. specify will have to dispatch on the type(s) of method-fn

    the type being whether it has .cljs$lang$arity$*

    see Issues > Method arities 

Issues

Method arities

Since the arity dispatch is already done at the method call site, we want to directly call the appropriate arity of the method-fn. Since method-fns are lexically bound, we cannot statically know in all cases, how to get the target arity thunk from method-fn.

As the fallback, specify will have to look at the method-fn in every call (of specify!), but this could be mitigated, when the type of a method-fn is known from analysis.

Compatibility with external objects / Name clashes

Since specify may come into contact with external data, there is a risk of name clashes, since under advanced compilation method names are shortened to few characters.

specify should check if a property exists on the target object before assigning and should throw if so. This will also guard against double specification with possibly different method-fns.

For external data, such conflicts can and should be resolved with externs files.

We need to think about this some more - we really don't want any checks for modifying prototypes. We need specify! to work elegantly for both scenarios - David

Ok, that's true. Maybe a warning would be more appropriate but the warning strings would already equal half a reflection layer. In production, people really shouldn't specify the same object more than once. For development and macros that will use specify on the prototype (extend-type), there could be another metadata flag ^:force-specify - Herwig

I'm inclined to say that specify! need not warn at all. If you're going to specify! foreign data you don't control that's a user error. For foreign data I think it's probably far wiser to use reify.

It's a user error, alright, but it's one that might manifest as a heisenbug from compilation to compilation, because the set of clobbered property names will change randomly. It won't show up under normal compilation and not even under advanced compilation with minification disabled (there is an option in gclosure for that, we should expose it). A truly horrifying perspective in my eyes. We should at least warn about it. - Herwig

The docstring of specify! should make it clear that it's not intended for data objects outside the ClojureScript compilation process and suggest reify for those cases. - David

I'm not exactly happy with that, but I think its manageable, given that clojurescript has still some way to go towards 1.0 - Herwig

Object methods and IFn

As per the refactoring from CLJS-414, generating (unmangled) Object methods is possible with (the private version of) specify.

When implementing IFn, extend-type generates a multi-arity fn and uses that as a call method. This hack was carried over into CLJS-414.

specify will generate arity methods, call and apply directly, when implementing IFn.

David, can you comment on how .apply vs .cljs$lang$applyTo are used?

.apply is for regular JavaScript fns and simple ClojureScript fns, .cljs$lang$applyTo is for everything else, .cljs$lang$applyTo gets checked first - David

I will need to unify the code for emitting a function with the code for specifying IFn. That move might even make fn* redundant in the analysis format, since it can then be implemented as (specify! (Function.) IFn ...) - Herwig

I don't see how since you still want the general version that dispatches via argument.length - David

Well, my example presupposed that specifying an Object as IFn would generate .call and .apply, as requested, which would supposedly need to dispatch on argument.lenght and do all the other things fn is doing, won't it? - Herwig

I don't see why you need to generate .apply instead of cljs$lang$applyTo. We do probably need to generate .call, this is mostly needed for compilation mode other than advanced - David

1. pass a function-like object into jquery 2. jquery calls .apply on it. ... We don't need to generate .apply for every specify IFn; a single .apply method that calls .call or .cljs$lang$applyTo can be assigned to all implementors of IFn. Maybe just Function.prototype.apply - Herwig

It's an entirely unrealistic expection to think you can pass instances of IFn to external JavaScript library and think they will work. The point of adding a .call property is about a uniform ClojureScript calling convention. JS libs will not generally invoke functions via .call or .apply. Adding .apply serves no purpose for ClojureScript that clj$lang$applyTo doesn't solve. - David

Right, adding .call doesn't automatically make an object callable. Thanks for catching that!

But we do want cljs$core$IFn$_invoke$arity$variadic and cljs$lang$maxFixedArity, right? - Herwig

Update 290513: I've implemented generating .cljs$core$IFn$_invoke$arity$ methods and .cljs$lang$applyTo along with .-cljs$lang$maxFixedArity and a wrapper for .call. Surprisingly, this seems to slow down IFn invokations slightly, at least on my machine, despite the generated code looking pretty optimal: https://github.com/bendlas/clojurescript/blob/8a7b4d75708ed8c7ac35ccb1dbd5627b1fce97dc/src/clj/cljs/core.clj#L583 and https://gist.github.com/bendlas/5664542#file-specify-js-L77 - Herwig

The slowdown issue has been resolved by type hinting. An area I want to explore is the possibility of using a single .call method for all fns (containing a switch and pulling the arity off this). - Herwig

Method names

Should the method names be written as symbols instead of keywords? extend has keywords, but it's a function.

specify might become a function on clojurescript if and when cljs gets reflection, which it probably won't.

Still, other dynamic hosts, that have reflection built in might profit, if we stick to keywords now.

there's also the added benefit of specifying maps of fns at the macro level and merging them there. This one is probably worth asking the community about once the proposal is more fully baked - David

Protocol method calling convention

Right now there is a special case, where IFn$_invoke is called without the otherwise usual first this parameter. I will explore the possibility to remove the redundant first parameters from other protocol methods aswell. Specify will need to introduce the binding via this-as or a wrapper function in the case of external implementations. - Herwig

Implementation

Herwig

I'm willing to implement the optimization for cost 1: to pull arity methods from the method-fn, if available. That would leave the overhead of the type dispatch(es) when invoking specify + a couple of redundant fn objects per specify call site.

I don't see why we need type dispatches, redundant fns for inline specifications do not add any more cost than what we're currently paying - which is nonexistent as far as I can tell from benchmarking - David

By type dispatch I mean checking for arity-methods on method-fn / other type specific optimisations. Currently extend-type just generates the arity-methods. A full fn would also generate an arity dispatch and apply that will never be used by specify. - Herwig

Right we should probably avoid generating the full fn - David

My point is that we don't generate the full fn, the user does:

Code Block
(extend-type
 IFoo
 (-m1 ; this only generates IFoo$._m1$arity methods
  ([o] :m1) 
  ([o x] [:m1 x])))

(specify (js-obj)
  IFoo
  {:-m1 (fn ; this generates an unused arity-dispatch, ... unless we do something about it 
         ([o] :m1)
         ([o x] [:m1 x]))})

- Herwig

Yes, I think we should special case this since in this case that the fn arities map back to the protocol, there's no need for the wrapper. - David

==

Both could be thought of as confined costs, but I would like to make sure that they won't be show-stoppers, if nobody gets around to figure out how the specify macro could interact with the analyzer in such a way that they are removed. 

analyzer/resolve-existing-var is all you need, this will contain all the known information about the fn, arities, max-arity, whether it's variadic etc - David

Does this also work for lexically bound names? What about expressions? Can I analyze them from a macro? - Herwig

It does work for lexically bound names though I'm not sure why you need that for specify. Expression themselves don't generally have any information. I don't see why you couldn't analyze from a macro, you're passed the environment so I think you should be able to call analyzer from the macro yes. - David

 

Code Block
(specify x IFoo {:m1 existing-fn}) ; existing-fn is lexically bound and could be an arbitrary expression. Remember, I'm referring to existing-fn as method-fn - Herwig
UPDATE 06052013-3 - Herwig

Most major decisions seem to be ironed out. I'll start with the ground work for:

  1. Implementing protocol methods with the arity-methods of a regular fn
  2. Emitting arity-dispatch for .call outside of cljs.compiler/emit for use in specify!

There is still time to chime in on the design; it might take a week or two before I can allocate another time block like this to start pushing patches

Proposal Description of prototype implementation as part of refactoring: CLJS-414

The operation to extend an instance will be called specify in the next proposal and have an interface similar to deftype. See discussion at CLJS-398

...

 

...

UPDATE 06052013

The refactoring done as part of this implementation will be rebased to current master with names specify and specify* set to private, until the design for specify is fully fleshed out.

Outline of the implementation

...

specify expands to specify* of protocols and method-arity-maps with fn expressions.

Since the whole protocol/type stack in clojurescript is based on modifying the prototype anyway, the patches from CLJS-414 basically just pull out the redundancies and label them.

Kernel operation: specify*

...

Code Block
(deftype A
  ILookup
  (-lookup ([o k] (aget s k))
           ([o k nf] (if (.hasOwnProperty o k) (aget o k) nf))) 
;; expands to

(specify (.-prototype A)
  ILookup
  ...)

;; expands to

(specify* (.-prototype A)
  ILookup {-lookup {2 (fn [o k] ..)
                    3 (fn [o k nf] ..)}})

;; and if we want to specify a whole batch
;; we can have the same method instances attached to every object
  
(let [m2 (fn [o k] ..)
      m3 (fn [o k nf] ..)] 
  (defn specify-alookup! [o]
    (specify* o
      ILookup {-lookup {2 m2 3 m3}})))

(defn jsonp-callback [osjson-list]
  (process (map specify-alookup! osjson-list))) 

 

Issues

  • specify* can't-be used with IFn in the current patchset to CLJS-414 as of writing 
  • specify expects to see each method only once, should error when user tries to do

...