clojure.java.io/do-copy: use java.nio for Files

Description

NIO Channels reduce CPU/Disk load when copying Files (by using syscalls like sendfile internally on Linux/Solaris).

CPU-Load goes from 100% to 0% on my system when copying large files, because no userspace-copying is involved.

Patch: clj-669-use-java.nio-in-do-copy-for-files-patch-v3.diff

Screened by: Alex Miller

Environment

None

Attachments

4

Activity

Show:

Andy Fingerhut August 10, 2013 at 9:37 AM

Good catch, Stuart. I verified by testing the -v2.txt patch on a 7 Gbyte file, and at least with a couple of OS/JDK combos transferTo never copied more than just under 2 Gbytes per call. It definitely needs a loop around transferTo calls to be correct.

Patch clj-669-use-java.nio-in-do-copy-for-files-patch-v3.txt dated Aug 10 2013 implements such a loop. I've tested it with OSX10.8.4+OracleJDK1.7.0_15, Ubuntu 12.04.1+OpenJDK1.6.0_27, and Ubuntu 12.04.1+OracleJDK1.7.0_25. In all cases it worked correctly on the 7 Gbyte file, and it reduced CPU load in the Java process as compared to the original version of do-copy.

Stuart Sierra August 2, 2013 at 8:05 PM

Marked as incomplete pending a question: do we know that transferTo will always copy the entire file in one call? Nothing in the Javadoc for FileChannel.transferTo guarantees that it will.

Just to be safe, I think we should call transferTo in a loop, keeping track of how many bytes have been copied.

As a side note, I verified that the transferTo method is available in JDK 1.5.

Andy Fingerhut May 24, 2013 at 6:45 PM

Patch clj-669-use-java.nio-in-do-copy-for-files-patch-v2.txt dated May 24 2013 is identical to patch 0001-use-java.nio-in-do-copy-method-for-Files.patch dated Nov 27 2010 except it applies cleanly to latest master. The older patch conflicts with a recent commit for CLJ-1072 that updates the obsolete #^ syntax for metadata.

Jürgen Hötzel November 27, 2010 at 11:05 AM

Patch instead of linking to github

Completed

Details

Assignee

Reporter

Approval

Ok

Patch

Code

Priority

Affects versions

Fix versions

Created November 1, 2010 at 10:03 PM
Updated November 23, 2013 at 1:06 AM
Resolved November 23, 2013 at 1:06 AM