<< Back to previous view

[CLJ-669] clojure.java.io/do-copy: use java.nio for Files Created: 01/Nov/10  Updated: 22/Nov/13  Resolved: 22/Nov/13

Status: Closed
Project: Clojure
Component/s: None
Affects Version/s: Release 1.3
Fix Version/s: Release 1.6

Type: Enhancement Priority: Major
Reporter: Jürgen Hötzel Assignee: Unassigned
Resolution: Completed Votes: 0
Labels: performance

Attachments: Text File 0001-use-java.nio-in-do-copy-method-for-Files.patch     Text File clj-669-use-java.nio-in-do-copy-for-files-patch-v2.txt     File clj-669-use-java.nio-in-do-copy-for-files-patch-v3.diff     Text File clj-669-use-java.nio-in-do-copy-for-files-patch-v3.txt    
Patch: Code
Approval: Ok

 Description   

NIO Channels reduce CPU/Disk load when copying Files (by using syscalls like sendfile internally on Linux/Solaris).

CPU-Load goes from 100% to 0% on my system when copying large files, because no userspace-copying is involved.

Patch: clj-669-use-java.nio-in-do-copy-for-files-patch-v3.diff

Screened by: Alex Miller



 Comments   
Comment by Jürgen Hötzel [ 27/Nov/10 5:05 AM ]

Patch instead of linking to github

Comment by Andy Fingerhut [ 24/May/13 12:45 PM ]

Patch clj-669-use-java.nio-in-do-copy-for-files-patch-v2.txt dated May 24 2013 is identical to patch 0001-use-java.nio-in-do-copy-method-for-Files.patch dated Nov 27 2010 except it applies cleanly to latest master. The older patch conflicts with a recent commit for CLJ-1072 that updates the obsolete #^ syntax for metadata.

Comment by Stuart Sierra [ 02/Aug/13 2:05 PM ]

Marked as incomplete pending a question: do we know that transferTo will always copy the entire file in one call? Nothing in the Javadoc for FileChannel.transferTo guarantees that it will.

Just to be safe, I think we should call transferTo in a loop, keeping track of how many bytes have been copied.

As a side note, I verified that the transferTo method is available in JDK 1.5.

Comment by Andy Fingerhut [ 10/Aug/13 3:37 AM ]

Good catch, Stuart. I verified by testing the -v2.txt patch on a 7 Gbyte file, and at least with a couple of OS/JDK combos transferTo never copied more than just under 2 Gbytes per call. It definitely needs a loop around transferTo calls to be correct.

Patch clj-669-use-java.nio-in-do-copy-for-files-patch-v3.txt dated Aug 10 2013 implements such a loop. I've tested it with OSX10.8.4+OracleJDK1.7.0_15, Ubuntu 12.04.1+OpenJDK1.6.0_27, and Ubuntu 12.04.1+OracleJDK1.7.0_25. In all cases it worked correctly on the 7 Gbyte file, and it reduced CPU load in the Java process as compared to the original version of do-copy.

Generated at Tue Sep 30 19:08:49 CDT 2014 using JIRA 4.4#649-r158309.