[DXML-5] OutOfMemory errors when emitting large XML documents Created: 27/Apr/12 Updated: 26/Jun/12 Resolved: 26/Jun/12 |
|
| Status: | Resolved |
| Project: | data.xml |
| Component/s: | None |
| Affects Version/s: | None |
| Fix Version/s: | None |
| Type: | Defect | Priority: | Major |
| Reporter: | Ryan Senior | Assignee: | Ryan Senior |
| Resolution: | Completed | Votes: | 0 |
| Labels: | None | ||
| Description |
|
Emitting large XML documents, fed from lazy-seqs in data.xml does not work. Currently, the lazy-seq is held in a defrecord, which holds onto the head of the lazy-seq and will force it to all be in memory (eventually consuming all available memory). Example code to reproduce the issue below: Unable to find source-code formatter for language: clojure. Available languages are: javascript, sql, xhtml, actionscript, none, html, xml, java (with-open [fw (java.io.FileWriter. "/tmp/lots-of-foo.xml")] (xml/emit (Element. :some-tags {} (map #(Element. :foo {} [(str "foo" %)]) (range 0 10000000))) fw)) |
| Comments |
| Comment by Ryan Senior [ 22/May/12 10:57 AM ] |
|
Fixed |
| Comment by Ryan Senior [ 26/Jun/12 12:18 PM ] |
|
Found this to be fixed only in the simplest case. If you have a large lazy-seq nested below 2+ tags it will hold onto the head of the lazy-seq and consume memory. |
| Comment by Ryan Senior [ 26/Jun/12 1:37 PM ] |
|
Added an intermediate step to emitting elements to the stream writer. Now elements get flattened to a stream of events that get written to the stream writer. |
| Comment by Ryan Senior [ 26/Jun/12 1:37 PM ] |
|
Not sure how to set a "Fix Version" in Jira, but this was fixed in 0.0.5 |