data.xml

OutOfMemory errors when emitting large XML documents

Details

  • Type: Defect Defect
  • Status: Resolved Resolved
  • Priority: Major Major
  • Resolution: Completed
  • Affects Version/s: None
  • Fix Version/s: None
  • Component/s: None
  • Labels:
    None

Description

Emitting large XML documents, fed from lazy-seqs in data.xml does not work. Currently, the lazy-seq is held in a defrecord, which holds onto the head of the lazy-seq and will force it to all be in memory (eventually consuming all available memory). Example code to reproduce the issue below:

Unable to find source-code formatter for language: clojure. Available languages are: javascript, sql, xhtml, actionscript, none, html, xml, java
(with-open [fw (java.io.FileWriter. "/tmp/lots-of-foo.xml")]
    (xml/emit
       (Element. :some-tags
           {}
           (map #(Element. :foo {} [(str "foo" %)])
                (range 0 10000000)))
       fw))

Activity

Hide
Ryan Senior added a comment -

Fixed

Show
Ryan Senior added a comment - Fixed
Hide
Ryan Senior added a comment -

Found this to be fixed only in the simplest case. If you have a large lazy-seq nested below 2+ tags it will hold onto the head of the lazy-seq and consume memory.

Show
Ryan Senior added a comment - Found this to be fixed only in the simplest case. If you have a large lazy-seq nested below 2+ tags it will hold onto the head of the lazy-seq and consume memory.
Hide
Ryan Senior added a comment -

Added an intermediate step to emitting elements to the stream writer. Now elements get flattened to a stream of events that get written to the stream writer.

Show
Ryan Senior added a comment - Added an intermediate step to emitting elements to the stream writer. Now elements get flattened to a stream of events that get written to the stream writer.
Hide
Ryan Senior added a comment -

Not sure how to set a "Fix Version" in Jira, but this was fixed in 0.0.5

Show
Ryan Senior added a comment - Not sure how to set a "Fix Version" in Jira, but this was fixed in 0.0.5

People

Vote (0)
Watch (0)

Dates

  • Created:
    Updated:
    Resolved: