Clojure

Clojure-generated class names length exceed file-system limit

Details

  • Type: Defect Defect
  • Status: Open Open
  • Priority: Major Major
  • Resolution: Unresolved
  • Affects Version/s: Release 1.7, Release 1.8
  • Fix Version/s: None
  • Component/s: None
  • Labels:
  • Environment:
    tested on CentOS 6
  • Approval:
    Triaged

Description

Class names generated by the Clojure compiler can be arbitrarily long, exceeding the file system's maximum allowed file name length. For example it happens when you nest functions a bit too deeply:

(defmacro nestfn [n & body]
  (if (> n 0)
    `(fn [] (nestfn ~(- n 1) ~@body))
    body))

(def myf (nestfn 100 "body"))

Compiling this produces a java.io.IOException: File name too long exception.

Activity

Hide
Martin Raison added a comment -

The Scala community found this issue a while ago, and now the compiler has a max-classfile-name parameter (defaulting to 255). Hashing is used when the limit is exceeded. Maybe we should consider something similar?

Show
Martin Raison added a comment - The Scala community found this issue a while ago, and now the compiler has a max-classfile-name parameter (defaulting to 255). Hashing is used when the limit is exceeded. Maybe we should consider something similar?
Hide
Philipp Neumann added a comment -

I tried clojure.core.match with 13 patterns and the compiliation failed under Windows. I assume this problem is the root cause of it.

Show
Philipp Neumann added a comment - I tried clojure.core.match with 13 patterns and the compiliation failed under Windows. I assume this problem is the root cause of it.
Hide
Dr. Christian Betz added a comment -

Some more info on that:

A colleague of mine just ran into that problem because he's using Linux / eCryptfs (where the limit of 143 is rather small, compared to our FileVault encrypted macOS used otherwise): see https://bugs.launchpad.net/ecryptfs/+bug/344878.

However, Clojure's not alone with that problem, Scala is also hit hard: https://issues.scala-lang.org/browse/SI-3623

There's no "easy" solution to this, and truncating the filename (as Scala does) bears a lot of other problems, obviously.

For all of you bitten by this problem, one possible workaround might be the one proposed by Mario Pastorelli (https://issues.scala-lang.org/secure/ViewProfile.jspa?name=melrief) in comment https://issues.scala-lang.org/browse/SI-3623?focusedCommentId=76104&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-76104:

Use an unencrypted filesystem to temporarily store classfiles, maybe by having a tempFS to keep stuff in memory. Not the best way, because we do not like leaving important stuff unencrypted, ...

Show
Dr. Christian Betz added a comment - Some more info on that: A colleague of mine just ran into that problem because he's using Linux / eCryptfs (where the limit of 143 is rather small, compared to our FileVault encrypted macOS used otherwise): see https://bugs.launchpad.net/ecryptfs/+bug/344878. However, Clojure's not alone with that problem, Scala is also hit hard: https://issues.scala-lang.org/browse/SI-3623 There's no "easy" solution to this, and truncating the filename (as Scala does) bears a lot of other problems, obviously. For all of you bitten by this problem, one possible workaround might be the one proposed by Mario Pastorelli (https://issues.scala-lang.org/secure/ViewProfile.jspa?name=melrief) in comment https://issues.scala-lang.org/browse/SI-3623?focusedCommentId=76104&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-76104: Use an unencrypted filesystem to temporarily store classfiles, maybe by having a tempFS to keep stuff in memory. Not the best way, because we do not like leaving important stuff unencrypted, ...
Hide
Alex Vong added a comment - - edited

Hello,

I run into the same problem while using clojure.core.match. I macroexpand the function definition and observe that the macro-expanded definition is deeply nested.

I think one way to solve it is to provide an option to wrap the class file as a jar with some shorter file name so that the the class name can remain the same (zipped file name can be as long as you like, right?). WDYT?

Btw, I am using clojure 1.9.0 so I think we should say that this bug affects 1.9 as well.

Show
Alex Vong added a comment - - edited Hello, I run into the same problem while using clojure.core.match. I macroexpand the function definition and observe that the macro-expanded definition is deeply nested. I think one way to solve it is to provide an option to wrap the class file as a jar with some shorter file name so that the the class name can remain the same (zipped file name can be as long as you like, right?). WDYT? Btw, I am using clojure 1.9.0 so I think we should say that this bug affects 1.9 as well.
Hide
Alex Vong added a comment - - edited

I come up with a proof-of-concept patch. The compiler now outputs jar instead of class if the class name is longer than 255. The jar name is simply a (left) truncation of the class name.

Suppose *compile-path* is set to "build", then you need to add build/* to your class path, so that the jars can be found.

This patch is only a proof of concept, ideally all classes with long name should be put into one big jar to avoid having to decompress many files. Also, the user should be able to specify *compile-name-max* and *compile-jar-name*. Finally, the code is quite ugly, I should have spitted things into several functions.

Show
Alex Vong added a comment - - edited I come up with a proof-of-concept patch. The compiler now outputs jar instead of class if the class name is longer than 255. The jar name is simply a (left) truncation of the class name. Suppose *compile-path* is set to "build", then you need to add build/* to your class path, so that the jars can be found. This patch is only a proof of concept, ideally all classes with long name should be put into one big jar to avoid having to decompress many files. Also, the user should be able to specify *compile-name-max* and *compile-jar-name*. Finally, the code is quite ugly, I should have spitted things into several functions.
Hide
Alex Miller added a comment -

Alex - we're not going to output jars. This is at odds with many facets of the Clojure runtime.

Show
Alex Miller added a comment - Alex - we're not going to output jars. This is at odds with many facets of the Clojure runtime.
Hide
Steve Miner added a comment -

Sorry if this comment is off topic, but the original example is a bit confusing to me. Maybe it should be:

(defmacro nestfn [n & body]
  (if (> n 0)
    `(fn [] (nestfn ~(dec n) ~@body))
    `(do ~@body)))

That way (trampoline (nestfn 10 "foo")) would return "foo". However, I do get a CompilerException java.lang.StackOverflowError for n=1000 on macOS.

Show
Steve Miner added a comment - Sorry if this comment is off topic, but the original example is a bit confusing to me. Maybe it should be:
(defmacro nestfn [n & body]
  (if (> n 0)
    `(fn [] (nestfn ~(dec n) ~@body))
    `(do ~@body)))
That way (trampoline (nestfn 10 "foo")) would return "foo". However, I do get a CompilerException java.lang.StackOverflowError for n=1000 on macOS.
Hide
Ivan Kryvoruchko added a comment -

@Alex Miller Hello! Would a patch with Scala-like approach (hashing of name in case it's too long) be considered? As it breaks binary compatibility this workaround would definitely be disabled by default, but it would be possible to enable name hashing by using new compiler option. This approach would definitely help in our particular case, but I'm not sure if it's useful/generic enough to be included in compiler, so I haven't started working on patch yet.

Show
Ivan Kryvoruchko added a comment - @Alex Miller Hello! Would a patch with Scala-like approach (hashing of name in case it's too long) be considered? As it breaks binary compatibility this workaround would definitely be disabled by default, but it would be possible to enable name hashing by using new compiler option. This approach would definitely help in our particular case, but I'm not sure if it's useful/generic enough to be included in compiler, so I haven't started working on patch yet.
Hide
Alex Miller added a comment -

Not going to make any breaking changes. Plan should be to switch second strategy when current strategy doesn’t work.

Show
Alex Miller added a comment - Not going to make any breaking changes. Plan should be to switch second strategy when current strategy doesn’t work.

People

Vote (11)
Watch (10)

Dates

  • Created:
    Updated: