[CONTRIB-88] clojure.contrib.sql runs out of memory on very large datasets

Tue, 24 Aug 2010 20:09:00 -0500

If you have a query that results in a very large dataset you will run out of memory because by default, with-query-results pulls the entire result set into memory (even though it creates a lazy-seq using resultset-seq).

This issue has been discussed previously here:
http://groups.google.com/group/clojure/browse_thread/thread/7b0c250e0ba6c9eb/fb9001522b49c20a

The fix is simple, just call (.setFetchSize stmt 1). But, with-query-results doesn't give you a way to do that currently. I propose adding an optional parameter after sql-params that is a hash of attributes to set on the PreparedStatement before executing it.

So you can do this:

(with-connection {���} 
    (.setAutoCommit (sql/connection) false) ;; needed for postgres 
    (with-query-results results ["SELECT id, data FROM nodes"] 
      {:fetch-size 1000} 
      (doseq [r results] 
        ���)))

The new code in clojure.contrib.sql is very simple, but it depends on a new contrib namespace called clojure.contrib.bean. Here is more info on my proposed fix:
http://groups.google.com/group/clojure-dev/browse_thread/thread/d8334759f10f3f45

Clojure JIRA

[CONTRIB-88] clojure.contrib.sql runs out of memory on very large datasets