Clojure JIRA

[DXML-1] Stack overflow when parsing huge XML file

Tue, 20 Mar 2012 08:05:15 -0500

This is using Ryan Senior's new 0.0.3-SNAPSHOT.

While trying to parse a huge XML file (7.5 GB compressed, a dump of Wikipedia), got a stack overflow error. Some digging turned up this bug:

http://bugs.sun.com/bugdatabase/view_bug.do?bug_id=6440214

Modifying clojure.data.xml/source-seq to disable the IS_COALESCING property got rid of the error.

The old lazy-xml contrib code worked (although used up tons more memory).

Attached is a patch that adds keyword options to source-seq, parse, and parse-str, allowing the consumer to disable coalescing and sidestep the upstream bug.

[DXML-5] OutOfMemory errors when emitting large XML documents

Tue, 26 Jun 2012 13:37:51 -0500

Emitting large XML documents, fed from lazy-seqs in data.xml does not work. Currently, the lazy-seq is held in a defrecord, which holds onto the head of the lazy-seq and will force it to all be in memory (eventually consuming all available memory). Example code to reproduce the issue below:

Unable to find source-code formatter for language: clojure. Available languages are: javascript, sql, xhtml, actionscript, none, html, xml, java

(with-open [fw (java.io.FileWriter. "/tmp/lots-of-foo.xml")]
    (xml/emit
       (Element. :some-tags
           {}
           (map #(Element. :foo {} [(str "foo" %)])
                (range 0 10000000)))
       fw))

[DXML-3] Build release on JDK 1.6

Fri, 24 Feb 2012 15:13:18 -0600

See https://groups.google.com/d/topic/clojure-dev/Z-wrRTcUs6U/discussion

[DXML-6] data.xml tests fail on clojure 1.2.0 and 1.2.1

Tue, 22 May 2012 12:54:41 -0500

See the test matrix here: http://build.clojure.org/job/data.xml-test-matrix/. Looks like the mixed-quotes test is to blame, just a reordering of attributes when they are emitted to a string.

[DXML-2] lein deps fails

Tue, 22 May 2012 10:56:32 -0500

C:\Users\ralphm\workspace\dbxml-env>lein version
Leiningen 1.6.2 on Java 1.7.0_02 Java HotSpot(TM) Client VM
C:\Users\ralphm\workspace\dbxml-env>lein deps
Downloading: org/clojure/data.xml/0.0.2-SNAPSHOT/data.xml-0.0.2-SNAPSHOT.pom from repository clojars at http://clojars.org/repo/
Unable to locate resource in repository
[INFO] Unable to find resource 'org.clojure:data.xml:pom:0.0.2-SNAPSHOT' in repository clojars (http://clojars.org/repo/)
Downloading: org/clojure/data.xml/0.0.2-SNAPSHOT/data.xml-0.0.2-SNAPSHOT.jar from repository clojars at http://clojars.org/repo/
Unable to locate resource in repository
[INFO] Unable to find resource 'org.clojure:data.xml:jar:0.0.2-SNAPSHOT' in repository clojars (http://clojars.org/repo/)
An error has occurred while processing the Maven artifact tasks.
Diagnosis:

Unable to resolve artifact: Missing:
----------
1) org.clojure:data.xml:jar:0.0.2-SNAPSHOT

Try downloading the file manually from the project website.

Then, install it using the command:
mvn install:install-file -DgroupId=org.clojure -DartifactId=data.xml -Dversion=0.0.2-SNAPSHOT
-Dpackaging=jar -Dfile=/path/to/file

Alternatively, if you host your own repository you can deploy the file there:
mvn deploy:deploy-file -DgroupId=org.clojure -DartifactId=data.xml -Dversion=0.0.2-SNAPSHOT -Dpackaging=jar -Dfile=/path/to/file -Durl=[url] -DrepositoryId=[id]

Path to dependency:
1) org.apache.maven:super-pom:pom:2.0
2) org.clojure:data.xml:jar:0.0.2-SNAPSHOT

----------
1 required artifact is missing.

for artifact:
org.apache.maven:super-pom:pom:2.0

from the specified remote repositories:
central (http://repo1.maven.org/maven2),
clojars (http://clojars.org/repo/)

[DXML-8] Cannot pass strings when keywords are expected

Wed, 14 Nov 2012 07:28:10 -0600

This error does not present till you attempt to emit xml that has a string where the element function is expecting a keyword. This is double hard to figure out at first because the error message is vague. I am requesting that the element function is allowed to use strings instead of keywords or the error message is cleaned up so that the "user" error is clear.

[DXML-11] Support cdata with sexp-as-element

Tue, 8 Jan 2013 22:07:06 -0600

prxml allowed something like this:

(prxml [:foo [:cdata! "all my cdata"]])

It doesn't look like that is currently allowed in data.xml. It looked like maybe I could extend the AsElements protocol to get this behavior, but I couldn't quite figure it out, seems like I'd have to have access to the XmlStreamWriter to get the string representation of the cdata.

[DXML-12] Do the right thing if cdata content contains the cdata end-tag "]]>"

Tue, 8 Jan 2013 22:07:25 -0600

(xml/emit-str (xml/cdata "fooo]]>bar"))

"<?xml version=\"1.0\" encoding=\"UTF-8\"?><![CDATA[fooo]]>bar]]>"

This is invalid xml. The contract for cdata states that it cannot contain the end tag "]]>", so if the cdata function gets passed content that contains it, it should do the right thing, which is probably this:

http://stackoverflow.com/questions/223652/is-there-a-way-to-escape-a-cdata-end-token-in-xml

(split the content so it is emitted as multiple cdata blocks, none of which contain the entire end-tag "]]>").

This is not a purely academic bug report - I actually hit this problem in prxml and fixed it on my fork.

[DXML-7] cannot change encoding when using the indent function

Tue, 9 Oct 2012 22:46:20 -0500

When using the Indent function, and trying to change the encoding, an exception is thrown.

java.lang.IllegalArgumentException: No value supplied for key: [:encoding "UTF-8"]

This seems to be that the options are not being passed from indent to emit correctly.

[DXML-9] Remove some use of reflection in data.xml

Wed, 14 Nov 2012 07:26:09 -0600

There are a couple of occurrences of reflection in the data.xml library

[DXML-10] Support for DOCTYPE when emitting XML

Wed, 14 Nov 2012 02:06:10 -0600

Some consumers of XML files require an explicit DOCTYPE to accept an XML file. data.xml does not currently support the specification of doctypes when emitting XML. When XML is parsed, I believe DOCTYPEs are silently ignored, so there is no representation in the data model for them. The best design is possibly an :doctype option in clojure.data.xml/emit ?

I've attached a web.xml as example.

[DXML-13] Support for preserving whitespace between tags

Sun, 10 Feb 2013 04:35:46 -0600

XML parsers can support preserving white space nodes, but clojure.data.xml does not seem to support this functionality.

For example, the following should be able to return true (perhaps with an option to parse-str):

Desired Behavior

(= (clojure.data.xml/element :x {}
                             (clojure.data.xml/element :a {} "foo")
                             " "
                             (clojure.data.xml/element :a {} "bar"))
   (clojure.data.xml/parse-str
     (str "<?xml version=\"1.0\" encoding=\"UTF-8\"?>"
          "<x>"
          "<a>foo</a>"
          " "
          "<a>bar</a>"
          "</x>")))
;=> true

This is the current behavior:

Current Behavior

(= (clojure.data.xml/element :x {}
                             (clojure.data.xml/element :a {} "foo")
                             (clojure.data.xml/element :a {} "bar"))
   (clojure.data.xml/parse-str
     (str "<?xml version=\"1.0\" encoding=\"UTF-8\"?>"
          "<x>"
          "<a>foo</a>"
          " "
          "<a>bar</a>"
          "</x>")))
;=> true

[DXML-14] IllegalArgumentException when trying to emit a boolean value

Thu, 7 Mar 2013 05:39:08 -0600

I can create an element with a boolean value but I can't emit it...

user=> (emit-str (element :something {} false))
IllegalArgumentException No implementation of method: :gen-event of protocol: #'clojure.data.xml/EventGeneration found for class: java.lang.Boolean clojure.core/-cache-protocol-fn (core_deftype.clj:541)

[DXML-15] data.xml can't parse own output if there's a colon in an attribute name

Wed, 3 Apr 2013 20:00:02 -0500

Observe:

> (x/emit-str (x/element :NC {"xmlns" "http://example.com" "xmlns:xsi" "http://www.w3.org/2001/XMLSchema-instance" "xsi:schemaLocation" "http://www.example.com/schema.xsd"} (x/element :Foo {} "bar")))
"<?xml version=\"1.0\" encoding=\"UTF-8\"?><NC xsi:schemaLocation=\"http://www.example.com/schema.xsd\" xmlns:xsi=\"http://www.w3.org/2001/XMLSchema-instance\" xmlns=\"http://example.com\"><Foo>bar</Foo></NC>"
> (x/parse-str *1)
#clojure.data.xml.Element{:tag :NC, :attrs {:xsi/schemaLocation "http://www.example.com/schema.xsd"}, :content (#clojure.data.xml.Element{:tag :Foo, :attrs {}, :content ("bar")})}
a> (x/emit-str *1)
XMLStreamException Prefix cannot be null com.sun.xml.internal.stream.writers.XMLStreamWriterImpl.writeAttribute (XMLStreamWriterImpl.java:574)
app.services.external.experian.internal.test-data>

(a) the xmlns and xmlns:xsi attributes have disappeared. Not the point of this issue but worth pointing out.
(b) "xsi:schemaLocation" has become :xsi/schemaLocation
(c) emitting a string blows up.

[DXML-4] Namespaces support

Sun, 21 Apr 2013 19:04:27 -0500

Add support for both parsing and emitting namespace qualified tags and namespaces URI declarations.
It basically follows the underlying Java XML API in giving xmlns:foo attributes "special" treatment.

[DXML-16] Eliminate reflection in emit-cdata

Thu, 25 Apr 2013 13:30:53 -0500

Solvable with a type hint on emit-cdata arg 'writer'