Clojure

clojure.java.io/do-copy: use java.nio for Files

Details

  • Type: Enhancement Enhancement
  • Status: Closed Closed
  • Priority: Major Major
  • Resolution: Completed
  • Affects Version/s: Release 1.3
  • Fix Version/s: Release 1.6
  • Component/s: None
  • Labels:
  • Patch:
    Code
  • Approval:
    Ok

Description

NIO Channels reduce CPU/Disk load when copying Files (by using syscalls like sendfile internally on Linux/Solaris).

CPU-Load goes from 100% to 0% on my system when copying large files, because no userspace-copying is involved.

Patch: clj-669-use-java.nio-in-do-copy-for-files-patch-v3.diff

Screened by: Alex Miller

Activity

Hide
Jürgen Hötzel added a comment -

Patch instead of linking to github

Show
Jürgen Hötzel added a comment - Patch instead of linking to github
Jürgen Hötzel made changes -
Field Original Value New Value
Attachment 0001-use-java.nio-in-do-copy-method-for-Files.patch [ 10032 ]
Rich Hickey made changes -
Fix Version/s Release 1.5 [ 10150 ]
Andy Fingerhut made changes -
Patch Code [ 10001 ]
Stuart Halloway made changes -
Fix Version/s Release 1.5 [ 10150 ]
Fix Version/s Release 1.6 [ 10157 ]
Hide
Andy Fingerhut added a comment -

Patch clj-669-use-java.nio-in-do-copy-for-files-patch-v2.txt dated May 24 2013 is identical to patch 0001-use-java.nio-in-do-copy-method-for-Files.patch dated Nov 27 2010 except it applies cleanly to latest master. The older patch conflicts with a recent commit for CLJ-1072 that updates the obsolete #^ syntax for metadata.

Show
Andy Fingerhut added a comment - Patch clj-669-use-java.nio-in-do-copy-for-files-patch-v2.txt dated May 24 2013 is identical to patch 0001-use-java.nio-in-do-copy-method-for-Files.patch dated Nov 27 2010 except it applies cleanly to latest master. The older patch conflicts with a recent commit for CLJ-1072 that updates the obsolete #^ syntax for metadata.
Andy Fingerhut made changes -
Alex Miller made changes -
Approval Vetted [ 10003 ]
Hide
Stuart Sierra added a comment -

Marked as incomplete pending a question: do we know that transferTo will always copy the entire file in one call? Nothing in the Javadoc for FileChannel.transferTo guarantees that it will.

Just to be safe, I think we should call transferTo in a loop, keeping track of how many bytes have been copied.

As a side note, I verified that the transferTo method is available in JDK 1.5.

Show
Stuart Sierra added a comment - Marked as incomplete pending a question: do we know that transferTo will always copy the entire file in one call? Nothing in the Javadoc for FileChannel.transferTo guarantees that it will. Just to be safe, I think we should call transferTo in a loop, keeping track of how many bytes have been copied. As a side note, I verified that the transferTo method is available in JDK 1.5.
Stuart Sierra made changes -
Approval Vetted [ 10003 ] Incomplete [ 10006 ]
Hide
Andy Fingerhut added a comment -

Good catch, Stuart. I verified by testing the -v2.txt patch on a 7 Gbyte file, and at least with a couple of OS/JDK combos transferTo never copied more than just under 2 Gbytes per call. It definitely needs a loop around transferTo calls to be correct.

Patch clj-669-use-java.nio-in-do-copy-for-files-patch-v3.txt dated Aug 10 2013 implements such a loop. I've tested it with OSX10.8.4+OracleJDK1.7.0_15, Ubuntu 12.04.1+OpenJDK1.6.0_27, and Ubuntu 12.04.1+OracleJDK1.7.0_25. In all cases it worked correctly on the 7 Gbyte file, and it reduced CPU load in the Java process as compared to the original version of do-copy.

Show
Andy Fingerhut added a comment - Good catch, Stuart. I verified by testing the -v2.txt patch on a 7 Gbyte file, and at least with a couple of OS/JDK combos transferTo never copied more than just under 2 Gbytes per call. It definitely needs a loop around transferTo calls to be correct. Patch clj-669-use-java.nio-in-do-copy-for-files-patch-v3.txt dated Aug 10 2013 implements such a loop. I've tested it with OSX10.8.4+OracleJDK1.7.0_15, Ubuntu 12.04.1+OpenJDK1.6.0_27, and Ubuntu 12.04.1+OracleJDK1.7.0_25. In all cases it worked correctly on the 7 Gbyte file, and it reduced CPU load in the Java process as compared to the original version of do-copy.
Andy Fingerhut made changes -
Alex Miller made changes -
Approval Incomplete [ 10006 ] Vetted [ 10003 ]
Andy Fingerhut made changes -
Description NIO Channels reduce CPU/Disk load when copying Files (by using
syscalls like sendfile internally on Linux/Solaris).

CPU-Load goes from 100% to 0% on my system when copying large files, because no userspace-copying is involved:

Patch: http://github.com/juergenhoetzel/clojure/commit/2b5ab103cbcfe6c49236ac6966c032d3c922233d
NIO Channels reduce CPU/Disk load when copying Files (by using
syscalls like sendfile internally on Linux/Solaris).

CPU-Load goes from 100% to 0% on my system when copying large files, because no userspace-copying is involved:

Patch: clj-669-use-java.nio-in-do-copy-for-files-patch-v3.txt
Alex Miller made changes -
Approval Vetted [ 10003 ] Screened [ 10004 ]
Description NIO Channels reduce CPU/Disk load when copying Files (by using
syscalls like sendfile internally on Linux/Solaris).

CPU-Load goes from 100% to 0% on my system when copying large files, because no userspace-copying is involved:

Patch: clj-669-use-java.nio-in-do-copy-for-files-patch-v3.txt
NIO Channels reduce CPU/Disk load when copying Files (by using syscalls like sendfile internally on Linux/Solaris).

CPU-Load goes from 100% to 0% on my system when copying large files, because no userspace-copying is involved.

*Patch:* clj-669-use-java.nio-in-do-copy-for-files-patch-v3.txt

*Screened by:* Alex Miller
Alex Miller made changes -
Labels performance
Alex Miller made changes -
Priority Minor [ 4 ] Major [ 3 ]
Alex Miller made changes -
Description NIO Channels reduce CPU/Disk load when copying Files (by using syscalls like sendfile internally on Linux/Solaris).

CPU-Load goes from 100% to 0% on my system when copying large files, because no userspace-copying is involved.

*Patch:* clj-669-use-java.nio-in-do-copy-for-files-patch-v3.txt

*Screened by:* Alex Miller
NIO Channels reduce CPU/Disk load when copying Files (by using syscalls like sendfile internally on Linux/Solaris).

CPU-Load goes from 100% to 0% on my system when copying large files, because no userspace-copying is involved.

*Patch:* clj-669-use-java.nio-in-do-copy-for-files-patch-v3.diff

*Screened by:* Alex Miller
Attachment clj-669-use-java.nio-in-do-copy-for-files-patch-v3.diff [ 12373 ]
Rich Hickey made changes -
Approval Screened [ 10004 ] Ok [ 10007 ]
Stuart Halloway made changes -
Resolution Completed [ 1 ]
Status Open [ 1 ] Closed [ 6 ]

People

Vote (0)
Watch (2)

Dates

  • Created:
    Updated:
    Resolved: