http://dev.clojure.org/jira This file is an XML representation of an issue en-us 4.4 649 25-07-2011 http://dev.clojure.org/jira/browse/DJSON-3 data.json <p><span class="error">&#91;This ticket filed as a follow on from pull request 2 at github&#93;</span></p> <p>One of my users decided to start sending Emoji happy faces over a network protocol encoded using data.json, which caused it to promptly roll over and die. It turned out that write-json-string, while aware of the code point vs. character issue, was still iterating over a character count not a code point count.</p> <p>I have demonstrated the problem in a test, and fixed this: will file a patch when my contributor agreement is in place.</p> Clojure 1.2 and 1.3, Mac OS X, Java 1.6 DJSON-3

Error writing strings with characters outside the BMP

Defect Major Resolved Completed Stuart Sierra Matthew Phillips Fri, 16 Dec 2011 17:53:29 -0600 Fri, 9 Mar 2012 19:41:03 -0600 Wed, 7 Mar 2012 07:58:48 -0600 1 2 <p>Patch to fix <a href="http://dev.clojure.org/jira/browse/DJSON-3" title="Error writing strings with characters outside the BMP"><del>DJSON-3</del></a>.</p> <p>I'm not an expert on JSON, so feel free to correct me if I am wrong, but it seems from reading RFC 4627 that strings in JSON, if they are encoded with the \uxxxx format for hex digits xxxx, should have their 16-bit UTF-16 code units encoded in that way. There should always be exactly 4 hex digits after the \u.</p> <p>Here are a couple of test cases with the current code, without either of the patches fix_unicode_non_bmp.patch or fix_unicode_non_bmp_2.patch:</p> <p>user=&gt; (in-ns 'clojure.data.json)<br/> #&lt;Namespace clojure.data.json&gt;</p> <p>;; Incorrect.<br/> clojure.data.json=&gt; (map #(format "%x" (int %)) (json-str "abc\ud834\udd1e" :escape-unicode false))<br/> ("22" "61" "62" "63" "d834" "dd1e" "dd1e" "22")</p> <p>;; Also incorrect.<br/> clojure.data.json=&gt; (json-str "abc\ud834\udd1e")<br/> "\"abc\\u1d11e\\udd1e\""</p> <p>Here is the behavior with patch fix_unicode_non_bmp.patch:</p> <p>; This is correct<br/> clojure.data.json=&gt; (map #(format "%x" (int %)) (json-str "abc\ud834\udd1e" :escape-unicode false))<br/> ("22" "61" "62" "63" "d834" "dd1e" "22")</p> <p>;; but this is not. JSON specifies that UTF-16 code units should be<br/> ;; included in string in form \uxxxx, for hex encoding xxxx of 16-bit<br/> ;; code units.<br/> clojure.data.json=&gt; (json-str "abc\ud834\udd1e")<br/> "\"abc\\u1d11e\""</p> <p>Here is the behavior with patch fix_unicode_non_bmp_2.patch:</p> <p>;; correct<br/> clojure.data.json=&gt; (map #(format "%x" (int %)) (json-str "abc\ud834\udd1e" :escape-unicode false))<br/> ("22" "61" "62" "63" "d834" "dd1e" "22")</p> <p>;; also correct.<br/> clojure.data.json=&gt; (json-str "abc\ud834\udd1e")<br/> "\"abc\\ud834\\udd1e\""</p> <p>fix_unicode_non_bmp_3.patch is same as fix_unicode_non_bmp_2.patch, except it includes the unit test Matthew Phillips had in his patch, plus another one to test the :escape-unicode true case.</p> <p>Eventually I may even get the hang of this. djson-3-non-bmp-chars-patch.txt contains patch in proper format.</p> <p>patch applied.</p> <p>Hi Stuart, it looks like you've only applied Andy Fingerhut's last patch. My fix is in the first one: "fix_unicode_non_bmp.patch"</p> <p>Matthew, did you see my comment from Jan 12, 2012? I show a test case there that appears not to work with your patch, unless I am misunderstanding the correct JSON desired behavior. That same test does work with my latest patch, I think. Take a look and see what you think.</p> <p>Oh I see - I misunderstood what your were doing there, thinking it only applied to the escape-unicode == true case.</p> <p>And of course the Clojure string is already in UTF-16, so there's no reason to turn it into 32 bit code points and then expand it back to UTF-16 pairs.</p> Global Rank Patch Code and Test