<< Back to previous view

[DXML-5] OutOfMemory errors when emitting large XML documents Created: 27/Apr/12  Updated: 26/Jun/12  Resolved: 26/Jun/12

Status: Resolved
Project: data.xml
Component/s: None
Affects Version/s: None
Fix Version/s: None

Type: Defect Priority: Major
Reporter: Ryan Senior Assignee: Ryan Senior
Resolution: Completed Votes: 0
Labels: None


 Description   

Emitting large XML documents, fed from lazy-seqs in data.xml does not work. Currently, the lazy-seq is held in a defrecord, which holds onto the head of the lazy-seq and will force it to all be in memory (eventually consuming all available memory). Example code to reproduce the issue below:

Unable to find source-code formatter for language: clojure. Available languages are: javascript, sql, xhtml, actionscript, none, html, xml, java
(with-open [fw (java.io.FileWriter. "/tmp/lots-of-foo.xml")]
    (xml/emit
       (Element. :some-tags
           {}
           (map #(Element. :foo {} [(str "foo" %)])
                (range 0 10000000)))
       fw))


 Comments   
Comment by Ryan Senior [ 22/May/12 10:57 AM ]

Fixed

Comment by Ryan Senior [ 26/Jun/12 12:18 PM ]

Found this to be fixed only in the simplest case. If you have a large lazy-seq nested below 2+ tags it will hold onto the head of the lazy-seq and consume memory.

Comment by Ryan Senior [ 26/Jun/12 1:37 PM ]

Added an intermediate step to emitting elements to the stream writer. Now elements get flattened to a stream of events that get written to the stream writer.

Comment by Ryan Senior [ 26/Jun/12 1:37 PM ]

Not sure how to set a "Fix Version" in Jira, but this was fixed in 0.0.5

Generated at Mon Sep 01 06:34:32 CDT 2014 using JIRA 4.4#649-r158309.