[CLJ-669] clojure.java.io/do-copy: use java.nio for Files Created: 01/Nov/10 Updated: 22/Nov/13 Resolved: 22/Nov/13
|Affects Version/s:||Release 1.3|
|Fix Version/s:||Release 1.6|
|Attachments:||0001-use-java.nio-in-do-copy-method-for-Files.patch clj-669-use-java.nio-in-do-copy-for-files-patch-v2.txt clj-669-use-java.nio-in-do-copy-for-files-patch-v3.diff clj-669-use-java.nio-in-do-copy-for-files-patch-v3.txt|
NIO Channels reduce CPU/Disk load when copying Files (by using syscalls like sendfile internally on Linux/Solaris).
CPU-Load goes from 100% to 0% on my system when copying large files, because no userspace-copying is involved.
Screened by: Alex Miller
|Comment by Jürgen Hötzel [ 27/Nov/10 5:05 AM ]|
Patch instead of linking to github
|Comment by Andy Fingerhut [ 24/May/13 12:45 PM ]|
Patch clj-669-use-java.nio-in-do-copy-for-files-patch-v2.txt dated May 24 2013 is identical to patch 0001-use-java.nio-in-do-copy-method-for-Files.patch dated Nov 27 2010 except it applies cleanly to latest master. The older patch conflicts with a recent commit for
|Comment by Stuart Sierra [ 02/Aug/13 2:05 PM ]|
Marked as incomplete pending a question: do we know that transferTo will always copy the entire file in one call? Nothing in the Javadoc for FileChannel.transferTo guarantees that it will.
Just to be safe, I think we should call transferTo in a loop, keeping track of how many bytes have been copied.
As a side note, I verified that the transferTo method is available in JDK 1.5.
|Comment by Andy Fingerhut [ 10/Aug/13 3:37 AM ]|
Good catch, Stuart. I verified by testing the -v2.txt patch on a 7 Gbyte file, and at least with a couple of OS/JDK combos transferTo never copied more than just under 2 Gbytes per call. It definitely needs a loop around transferTo calls to be correct.
Patch clj-669-use-java.nio-in-do-copy-for-files-patch-v3.txt dated Aug 10 2013 implements such a loop. I've tested it with OSX10.8.4+OracleJDK1.7.0_15, Ubuntu 12.04.1+OpenJDK1.6.0_27, and Ubuntu 12.04.1+OracleJDK1.7.0_25. In all cases it worked correctly on the 7 Gbyte file, and it reduced CPU load in the Java process as compared to the original version of do-copy.