<< Back to previous view

[CLJS-1613] Preserve Structural Sharing for clj->js Created: 03/Apr/16  Updated: 08/Apr/16  Resolved: 08/Apr/16

Status: Closed
Project: ClojureScript
Component/s: None
Affects Version/s: 1.7.228
Fix Version/s: None

Type: Enhancement Priority: Minor
Reporter: Tom Raithel Assignee: Unassigned
Resolution: Declined Votes: 0
Labels: clj->js

java version "1.8.0_74"
Java(TM) SE Runtime Environment (build 1.8.0_74-b02)
Java HotSpot(TM) 64-Bit Server VM (build 25.74-b02, mixed mode)


Hi, is there any possibility to keep structural sharing intact after calling clj->js on a collection?

It seems like clj->js just makes a deep copy from the collection on every call. I´m new to ClojureScript but it is really confusing that the equality-check against the clojure set is true and the check against the js object equals false.

(def set1 {:a [1 2 3]})
(def set2 (assoc set1 :b "foo"))
(.log js/console (== (get set1 :a) (get set2 :a)))
; -> true

(def set1js (clj->js2 set1))
(def set2js (clj->js2 set2))
(.log js/console (== (.-a set1js) (.-a set2js)))
; -> false

Comment by Thomas Heller [ 04/Apr/16 3:25 AM ]

This is just JS being JS. clj->js converts the entire structure, JS doesn't have a set type so we get a [1,2,3] javascript array.

[1,2,3] == [1,2,3]

is false in JS, nothing we can do about that. Generally you want to stay away from JS native collections as much as possible.

Comment by Tom Raithel [ 04/Apr/16 3:46 AM ]

Thanks for your response.

Yes, the two arrays are not the same because clj->js copies all the values over. I was just wondering if there may be a solution to pick up the real reference to the underlying JS object instead of just copying them over.

For example in this case, if I access the [1 2 3] array from the maps directly via js (not calling clj->js), the actual JS objects are equal:

(def a1 (.-tail (nth (.-arr set1) 1)))
(def a2 (.-tail (nth (.-arr set2) 1)))
(.log js/console a1)
(.log js/console a2)
(.log js/console (== a1 a2))
; -> true

I know that example above is hacky, but I just wanted to demonstrate that the structural sharing is already in place, it´s just not preserved when using clj->js.

This would be very helpful for applications, where only part of the code - for example the business layer - are written in ClojureScript and data is shared via plain JS objects.

Comment by Thomas Heller [ 04/Apr/16 4:22 AM ]

That this works is just pure coincidence. If you put a few more elements into the initial vector (eg. (def set1 {:a (into [] (range 500))})) the (def a1 (.-tail (nth (.-arr set1) 1))) will only contain a small part of the values. If you want all values in JS you would need to create a new array and lose the "shareable" reference.

Comment by Tom Raithel [ 04/Apr/16 4:34 AM ]

Ah I see, didn´t know that they are lazy evaluated. I will probably have to come up with another solution to that specific problem. Thx for your help Thomas.

Comment by Francis Avila [ 05/Apr/16 2:09 PM ]

Tom, the issue is not lazy evaluation, it's that a vector internally divides its values into multiple js arrays whose values never change after initialization.

I don't see how this idea is in any way feasible. clj->js cannot ever safely reuse the same underlying arrays from clojure collection objects because any mutation to the structure returned by clj->js would mutate the original clj collections.

At most what could be done is ensuring that clj objects of equal value produce identical js objects. However, even this I don't think is desirable in the general case because it would completely frustrate expectations from javascript consumers of your object. It also has other problems.

Suppose this did exist and you had a clj->js with this behavior:

(def a (clj->js [#{1 2 3} [1 2 3]]))
(def b (clj->js [#{1 2 3} [1 2 3]]))
(def c (clj->js #{1 2 3}))
(assert (identical? a b))
(assert (identical? (aget a 0) c)
(assert (not (identical? (aget a 0) (aget a 1))) ; I assume?

Now how would you avoid this scenario if you mutate something?

(aset a 0 3 "new")
; a is now #js[#js[1 2 3 "new"] #js[1 2 3]]
; b is now the same
; c is now #js[#js[1 2 3 "new"]]

; and subsequent clj->js calls are broken unless you go to extraordinary lengths to invalidate cached entries
(clj->js #{1 2 3}) ; => #js[1 2 3 "new"]

;; But suppose you don't like the above answer. Your options are either:
;; 1. Create a new cached copy with the correct mapping, breaking the identity with already-created values
(def d (clj->js #{1 2 3}))
; #js[1 2 3] again
(assert (not (identical? c d)) ; no more sharing

;; or 2. mutate the cached value back to the original value!
(def d (clj->js #{1 2 3}))
; a, b, c, d are now what they were before the (aset a 0 3 "new")

The core problem is that js data structures are all mutable so the only safe thing to do is clone everything. If you have some special case where you know that the js consumer of the clj->js call will not mutate anything and you don't mind the memory overhead of memoization to get the reference-equality you want, you can always implement a clj->js variant with the properties you desire.

Comment by Tom Raithel [ 07/Apr/16 2:19 PM ]

Hey Francis,

your first code block is not quite what I meant. I was proposing something that behaves exactly like the identical? check of clj objects. That means - regarding your example - I would expect to (assert (identical? (aget a 0) c) equals false because c has no relation to the a object.

I hacked together an extended version of clj->js which is basically storing all the already converted JS objects on to the clj object, so that the second clj->js call on the identical data returns the same object that was already transformed.

It may be a little bit hacky and probably agains all laws of cljs (I´m still new to it), but it works for me. I assume it would also be a bit faster then clj->js because if the value is already cached, there is no need for transformation.

(defn clj->cached-js
  "Recursively transforms ClojureScript values to JavaScript and cache its values.
sets/vectors/lists become Arrays, Keywords and Symbol become Strings,
Maps become Objects. Arbitrary keys are encoded to by key->js."
  (when-not (nil? state)
    (if (satisfies? IEncodeJS state)
      (-clj->js state)
      (if (aget state "__cachedJs")
        (aget state "__cachedJs")
        (let [obj (cond
                    (keyword? state) (name state)
                    (symbol? state) (str state)
                    (map? state) (let [m (js-obj)]
                               (doseq [[k v] state]
                                 (aset m (key->js k) (clj->cached-js v)))
                    (coll? state) (let [arr (array)]
                                (doseq [state (map clj->cached-js state)]
                                  (.push arr state))
                    :else state)]
          (aset state "__cachedJs" obj)

Here are some tests that I wrote:

(def some-symbol "my symbol")
(def state {
  :symbol some-symbol
  :key :some-key
  :id 1234
  :user {
    :name "Tom",
    :age 33,
    :city "Wiesbaden"
  :hobbies [
    {:name "code"}
    {:name "sleep"}
  :tags '("foo" "bar" "baz")

(t/deftest test-matches-original-behaviour
  (let [obj (core/clj->cached-js state)
        original-obj (clj->js state)]
    (t/is (= (js->clj original-obj) (js->clj obj)))))

(t/deftest test-change-hash-value
  (let [obj (core/clj->cached-js state)
        new-state (assoc-in state [:user :name] "Bert")
        new-obj (core/clj->cached-js new-state)]
    (t/is (not= new-obj obj))
    (t/is (not= (.-user new-obj) (.-user obj)))
    (t/is (= (.-user.name new-obj) "Bert"))
    (t/is (= (.-hobbies new-obj) (.-hobbies obj)))
    (t/is (= (.-tags new-obj) (.-tags obj)))

(t/deftest test-change-vector-value
  (let [obj (core/clj->cached-js state)
        new-state (assoc-in state [:hobbies 1 :name] "drink")
        new-obj (core/clj->cached-js new-state)]
    (t/is (not= new-obj obj))
    (t/is (= (.-user new-obj) (.-user obj)))
    (t/is (not= (.-hobbies new-obj) (.-hobbies obj)))
    (t/is (= (aget new-obj "hobbies" 0) (aget obj "hobbies" 0)))
    (t/is (not= (aget new-obj "hobbies" 1) (aget obj "hobbies" 1)))
    (t/is (= (aget new-obj "hobbies" 1 "name") "drink"))
    (t/is (= (.-tags new-obj) (.-tags obj)))

(t/deftest test-add-value-to-vector
  (let [obj (core/clj->cached-js state)
        new-state (assoc-in state
                            (let [hobbies (get state :hobbies)
                                 [before after] (split-at 1 hobbies)]
                              (vec (concat before [{:name "kid"}] after))))
        new-obj (core/clj->cached-js new-state)]
    (t/is (not= (.-hobbies new-obj) (.-hobbies obj)))
    (t/is (= (aget new-obj "hobbies" 0) (aget obj "hobbies" 0)))
    (t/is (not= (aget new-obj "hobbies" 1) (aget obj "hobbies" 1)))
    (t/is (= (aget new-obj "hobbies" 2) (aget obj "hobbies" 1)))

Of course it does not prevent you from manipulating the converted JS objects which subsequently would change other JS references, but that is - like Thomas Heller mentioned - just JS beeing JS. It is just not immutable, but you should treat it as such.

Don´t get me wrong, I believe that is not something that should be implemented into the ClojureScript itself. But this ticket helped me a lot in understanding the principle of clj->js. For now I will go with this solution. If you see some problems with this approach, please let me know.

Thanks for your help!

Comment by Francis Avila [ 08/Apr/16 1:48 PM ]

Consider using (def clj->cached-js (memoize clj->js)).

[CLJS-536] clj->js trims the namespace prefix from keywords while writing them to string Created: 12/Jul/13  Updated: 09/Jul/17  Resolved: 02/Dec/13

Status: Closed
Project: ClojureScript
Component/s: None
Affects Version/s: None
Fix Version/s: None

Type: Defect Priority: Major
Reporter: Vasile Assignee: Unassigned
Resolution: Declined Votes: 0
Labels: clj->js

Attachments: Text File 0001-CLJS-536-Add-support-for-namespaced-keywords.patch     Text File 0001-CLJS-536-Add-support-for-namespaced-keywords.patch    


The following behavior was observed and confirmed from the code:

(clj->js :ns/n) => "n"

I believe this is a limitation and the namespace of the keyword should be kept while writing it to string.
The code in core.js does this while handling keywords:

(keyword? x) (name x)

while it should do this (or something similar):

(keyword? x) (str (namespace x) "/" (name x))

Comment by Vasile [ 12/Jul/13 12:03 PM ]

a better (working) fix: (keyword? x) (str (if (namespace x) (str (namespace x) "/")) (name x))

Comment by David Nolen [ 16/Jul/13 6:22 AM ]

I'd be willing to take a patch that allows this behavior to be configured.

Comment by Niklas Närhinen [ 01/Nov/13 7:33 AM ]

Proposed fix

Comment by Niklas Närhinen [ 01/Nov/13 7:37 AM ]

Fixed version of patch

Comment by David Nolen [ 01/Nov/13 9:23 AM ]

Excellent, Niklas I don't see you on the list of contributors, please send in your Contributor Agreement, http://clojure.org/contributing, so we can apply the patch.

Comment by David Nolen [ 02/Dec/13 8:52 PM ]

I looked more closely at the clj->js source, the system is already customizable. We can't determine ahead of time how you might want to emit keywords and symbols - thus you can extend Keyword and Symbol to IEncodeJS yourself and get the desired behavior.

Comment by Andrea Richiardi [ 17/Jan/17 2:52 PM ]

I have just stumbled across this one, shall we at least say it in the docstring of clj->js that we are losing the namespace part?

Comment by Jozef Wagner [ 19/Feb/17 4:11 AM ]

With the introduction of specs, the namespaced keywords are being used more and more. This issue prevents streamlined edn->json->edn transformation. I think it should be reopened. IMO the 'lossy' method should never be a default one.

Comment by Paulus Esterhazy [ 19/Feb/17 5:53 AM ]

Unless we are willing to break existing code, I don't think it will be possible to change the default behavior.

I'm also not sure that extending IEncodeJS is the best solution, as it affects every call to clj->js, including calls to libraries which may rely on the ns-stripping behavior.

However, the attached patch allows you to make the decision on a per-call basis.

One quibble with the patch: perhaps it would be better to use kwargs style `(clj->js v :preserve-namespaces true)` in line with `js->clj`?

Comment by Le Wang [ 13/Jun/17 9:26 AM ]

How would extending IEncodeJS work if I want to only include namespaces sometimes, and don't want to change the default behaviour for all libraries I've included? If this is not possible, it seems this is not a valid solution so this problem.

Comment by Enzzo Cavallo [ 09/Jul/17 8:00 PM ]

I would like to propose a solution
(First I made a patch, but I saw that the jira does not accept any more attachments, so I published in the gist)
It allows to process keywords with any function
Dont break any existing code
Use named arguments (kwargs style)

Generated at Sun Jan 21 05:16:26 CST 2018 using JIRA 4.4#649-r158309.