Clojure 1.2 and 1.3, Mac OS X, Java 1.6

Patch to fix DJSON-3.

I'm not an expert on JSON, so feel free to correct me if I am wrong, but it seems from reading RFC 4627 that strings in JSON, if they are encoded with the \uxxxx format for hex digits xxxx, should have their 16-bit UTF-16 code units encoded in that way. There should always be exactly 4 hex digits after the \u.

Here are a couple of test cases with the current code, without either of the patches fix_unicode_non_bmp.patch or fix_unicode_non_bmp_2.patch:

user=> (in-ns 'clojure.data.json)
#<Namespace clojure.data.json>

;; Incorrect.
clojure.data.json=> (map #(format "%x" (int %)) (json-str "abc\ud834\udd1e" :escape-unicode false))
("22" "61" "62" "63" "d834" "dd1e" "dd1e" "22")

;; Also incorrect.
clojure.data.json=> (json-str "abc\ud834\udd1e")
"\"abc\\u1d11e\\udd1e\""

Here is the behavior with patch fix_unicode_non_bmp.patch:

; This is correct
clojure.data.json=> (map #(format "%x" (int %)) (json-str "abc\ud834\udd1e" :escape-unicode false))
("22" "61" "62" "63" "d834" "dd1e" "22")

;; but this is not. JSON specifies that UTF-16 code units should be
;; included in string in form \uxxxx, for hex encoding xxxx of 16-bit
;; code units.
clojure.data.json=> (json-str "abc\ud834\udd1e")
"\"abc\\u1d11e\""

Here is the behavior with patch fix_unicode_non_bmp_2.patch:

;; correct
clojure.data.json=> (map #(format "%x" (int %)) (json-str "abc\ud834\udd1e" :escape-unicode false))
("22" "61" "62" "63" "d834" "dd1e" "22")

;; also correct.
clojure.data.json=> (json-str "abc\ud834\udd1e")
"\"abc\\ud834\\udd1e\""

fix_unicode_non_bmp_3.patch is same as fix_unicode_non_bmp_2.patch, except it includes the unit test Matthew Phillips had in his patch, plus another one to test the :escape-unicode true case.

Eventually I may even get the hang of this. djson-3-non-bmp-chars-patch.txt contains patch in proper format.

patch applied.

Hi Stuart, it looks like you've only applied Andy Fingerhut's last patch. My fix is in the first one: "fix_unicode_non_bmp.patch"

Matthew, did you see my comment from Jan 12, 2012? I show a test case there that appears not to work with your patch, unless I am misunderstanding the correct JSON desired behavior. That same test does work with my latest patch, I think. Take a look and see what you think.

Oh I see - I misunderstood what your were doing there, thinking it only applied to the escape-unicode == true case.

And of course the Clojure string is already in UTF-16, so there's no reason to turn it into 32 bit code points and then expand it back to UTF-16 pairs.

According to clojuresphere.com, data.json is relied on by 340+ projects: http://www.clojuresphere.com/org.clojure/data.json

Alternative suggestion: Mark the library as "under development/API subject to change" while < v1.0.

@Tim McCormack: if a project is used by over 300 other projects and has been around for about a year, it is just lame to use excuses like that. It is too late, too many people already
use c.d.j.

Old APIs restored, documented as "DEPRECATED", in release 0.2.1.

Not a bug

clojure.data.json 0.2.0

djson-7-dont-write-unnecessary-commas-v1.txt dated Jan 21 2013 modifies write-object so that it only writes out a comma if :value-fn does not cause the key/value pair to be omitted.

Patch applied.

Patch djson-8-correct-write-json-patch-v1.txt dated Mar 27 2013 is a simple one-line fix.

Already fixed in commit d3cce5b200031cf22603c13a9a39e3939651d344