Clojure

Enhance AOT compilation process to emit classfiles only for explicitly-specified namespaces

Details

  • Type: Feature Feature
  • Status: Closed Closed
  • Priority: Major Major
  • Resolution: Declined
  • Affects Version/s: None
  • Fix Version/s: None
  • Component/s: None
  • Labels:

Description

Why This Was Declined

The current behavior of transitive compilation is a valuable semantic. It is arguably essential to e.g. AOT protocols ahead of their call sites.

On the other hand, you should not ship AOT compiled classes for which your project is not the source. As worded in the original ticket:
"Having the option of shipping either all AOT-compiled classfiles or mixed source/AOT depending upon one's distribution requirements would make that phase of work with a clojure codebase significantly easier and less error-prone."

This latter goal can be accomplished by keeping "what you compile" and "what you ship" cleanly separate. The compiler does not know or care about the latter, and tools can automate this with e.g. boot's sift.

Historical Context

This was originally/erroneously reported by Howard Lewis Ship in the clojure-contrib assembla:

My build file specifies the namespaces to AOT compile but if I include another namespace
(even from a JAR dependency) that is not AOT compiled, the other namespace will be compiled as well.

In my case, I was using clojure-contrib's clojure.contrib.str-utils2 namespace, and I got a bunch of
clojure/contrib/str_utils2 classes in my output directory.

I think that the AOT compiler should NOT precompile any namespaces that are transitively reached,
only namespaces in the set specified by the command line are appropriate.

As currently coded, you will frequently find unwanted third-party dependencies in your output JARs;
further, if multiple parties depend on the same JARs, this could cause bloating and duplication in the
eventual runtime classpath.

The only question in my mind is what the default should be. We're all used to the current behaviour, but I'd guess that any nontrivial project where the form of the distributable matters (i.e. the source/AOT mix), providing as much control as possible by default makes the most sense. Given the tooling that most people are using, it's trivial (and common practice, IIUC) to provide a comprehensive list of namespaces one wishes to compile, so making that the default shouldn't be a hurdle to anyone. If an escape hatch is desired, a --transitive switch to clojure.lang.Compile could be added.

  1. 0322-limit-aot-resolved.patch
    29/Nov/10 7:18 AM
    11 kB
    Stuart Halloway
  2. CLJ-322.diff
    19/Nov/10 9:28 PM
    11 kB
    Chas Emerick
  3. compile-interop-1.patch
    29/Nov/11 6:54 PM
    1 kB
    Stuart Sierra
  4. write-classes-1.diff.gz
    10/Dec/10 3:34 PM
    2 kB
    Stuart Sierra

Activity

Chas Emerick made changes -
Field Original Value New Value
Reporter Chas Emerick [ cemerick ]
Priority Blocker [ 1 ]
Patch Code
Chas Emerick made changes -
Patch Code Code and Test
Chas Emerick made changes -
Attachment CLJ-322.diff [ 10024 ]
Stuart Halloway made changes -
Attachment 0322-limit-aot-resolved.patch [ 10035 ]
Priority Blocker [ 1 ] Major [ 3 ]
Stuart Halloway made changes -
Approval Test Incomplete
Waiting On cemerick
Alexander Redington made changes -
Fix Version/s Approved Backlog [ 10034 ]
Fix Version/s Release.Next [ 10038 ]
Stuart Sierra made changes -
Attachment write-classes-1.diff.gz [ 10050 ]
Stuart Sierra made changes -
Attachment compile-interop-1.patch [ 10723 ]
Approval Incomplete [ 10006 ] Vetted [ 10003 ]
Stuart Sierra made changes -
Assignee Chas Emerick [ cemerick ] Stuart Sierra [ stuart.sierra ]
Stuart Sierra made changes -
Assignee Stuart Sierra [ stuart.sierra ]
Rich Hickey made changes -
Fix Version/s Release 1.6 [ 10157 ]
Fix Version/s Approved Backlog [ 10034 ]
Stuart Halloway made changes -
Description This was [originally/erroneously reported|https://www.assembla.com/spaces/clojure-contrib/tickets/23] by Howard Lewis Ship in the clojure-contrib assembla:

{code}My build file specifies the namespaces to AOT compile but if I include another namespace
(even from a JAR dependency) that is not AOT compiled, the other namespace will be compiled as well.

In my case, I was using clojure-contrib's clojure.contrib.str-utils2 namespace, and I got a bunch of
clojure/contrib/str_utils2 classes in my output directory.

I think that the AOT compiler should NOT precompile any namespaces that are transitively reached,
only namespaces in the set specified by the command line are appropriate.

As currently coded, you will frequently find unwanted third-party dependencies in your output JARs;
further, if multiple parties depend on the same JARs, this could cause bloating and duplication in the
eventual runtime classpath.{code}

Having the option of shipping either all AOT-compiled classfiles or mixed source/AOT depending upon one's distribution requirements would make that phase of work with a clojure codebase significantly easier and less error-prone. The only question in my mind is what the default should be. We're all used to the current behaviour, but I'd guess that any nontrivial project where the form of the distributable matters (i.e. the source/AOT mix), providing as much control as possible by default makes the most sense. Given the tooling that most people are using, it's trivial (and common practice, IIUC) to provide a comprehensive list of namespaces one wishes to compile, so making that the default shouldn't be a hurdle to anyone. If an escape hatch is desired, a --transitive switch to clojure.lang.Compile could be added.
Summary: still needs decision on implementation approach.

This was [originally/erroneously reported|https://www.assembla.com/spaces/clojure-contrib/tickets/23] by Howard Lewis Ship in the clojure-contrib assembla:

{code}My build file specifies the namespaces to AOT compile but if I include another namespace
(even from a JAR dependency) that is not AOT compiled, the other namespace will be compiled as well.

In my case, I was using clojure-contrib's clojure.contrib.str-utils2 namespace, and I got a bunch of
clojure/contrib/str_utils2 classes in my output directory.

I think that the AOT compiler should NOT precompile any namespaces that are transitively reached,
only namespaces in the set specified by the command line are appropriate.

As currently coded, you will frequently find unwanted third-party dependencies in your output JARs;
further, if multiple parties depend on the same JARs, this could cause bloating and duplication in the
eventual runtime classpath.{code}

Having the option of shipping either all AOT-compiled classfiles or mixed source/AOT depending upon one's distribution requirements would make that phase of work with a clojure codebase significantly easier and less error-prone. The only question in my mind is what the default should be. We're all used to the current behaviour, but I'd guess that any nontrivial project where the form of the distributable matters (i.e. the source/AOT mix), providing as much control as possible by default makes the most sense. Given the tooling that most people are using, it's trivial (and common practice, IIUC) to provide a comprehensive list of namespaces one wishes to compile, so making that the default shouldn't be a hurdle to anyone. If an escape hatch is desired, a --transitive switch to clojure.lang.Compile could be added.
Approval Vetted [ 10003 ] Incomplete [ 10006 ]
Alex Miller made changes -
Labels aot
Alex Miller made changes -
Fix Version/s Release 1.6 [ 10157 ]
Alex Miller made changes -
Approval Incomplete [ 10006 ] Vetted [ 10003 ]
Alex Miller made changes -
Issue Type Enhancement [ 4 ] Feature [ 5 ]
Stuart Halloway made changes -
Approval Vetted [ 10003 ]
Description Summary: still needs decision on implementation approach.

This was [originally/erroneously reported|https://www.assembla.com/spaces/clojure-contrib/tickets/23] by Howard Lewis Ship in the clojure-contrib assembla:

{code}My build file specifies the namespaces to AOT compile but if I include another namespace
(even from a JAR dependency) that is not AOT compiled, the other namespace will be compiled as well.

In my case, I was using clojure-contrib's clojure.contrib.str-utils2 namespace, and I got a bunch of
clojure/contrib/str_utils2 classes in my output directory.

I think that the AOT compiler should NOT precompile any namespaces that are transitively reached,
only namespaces in the set specified by the command line are appropriate.

As currently coded, you will frequently find unwanted third-party dependencies in your output JARs;
further, if multiple parties depend on the same JARs, this could cause bloating and duplication in the
eventual runtime classpath.{code}

Having the option of shipping either all AOT-compiled classfiles or mixed source/AOT depending upon one's distribution requirements would make that phase of work with a clojure codebase significantly easier and less error-prone. The only question in my mind is what the default should be. We're all used to the current behaviour, but I'd guess that any nontrivial project where the form of the distributable matters (i.e. the source/AOT mix), providing as much control as possible by default makes the most sense. Given the tooling that most people are using, it's trivial (and common practice, IIUC) to provide a comprehensive list of namespaces one wishes to compile, so making that the default shouldn't be a hurdle to anyone. If an escape hatch is desired, a --transitive switch to clojure.lang.Compile could be added.
h3. Why This Was Declined

The current behavior of transitive compilation is a valuable semantic. It is arguably essential to e.g. AOT protocols ahead of their call sites.

On the other hand, you should not ship AOT compiled classes for which your project is not the source. As worded in the original ticket:
"Having the option of shipping either all AOT-compiled classfiles or mixed source/AOT depending upon one's distribution requirements would make that phase of work with a clojure codebase significantly easier and less error-prone."

This latter goal can be accomplished by keeping "what you compile" and "what you ship" cleanly separate. The compiler does not know or care about the latter, and tools can automate this with e.g. boot's {{sift}}.

h3. Historical Context

This was [originally/erroneously reported|https://www.assembla.com/spaces/clojure-contrib/tickets/23] by Howard Lewis Ship in the clojure-contrib assembla:

{code}My build file specifies the namespaces to AOT compile but if I include another namespace
(even from a JAR dependency) that is not AOT compiled, the other namespace will be compiled as well.

In my case, I was using clojure-contrib's clojure.contrib.str-utils2 namespace, and I got a bunch of
clojure/contrib/str_utils2 classes in my output directory.

I think that the AOT compiler should NOT precompile any namespaces that are transitively reached,
only namespaces in the set specified by the command line are appropriate.

As currently coded, you will frequently find unwanted third-party dependencies in your output JARs;
further, if multiple parties depend on the same JARs, this could cause bloating and duplication in the
eventual runtime classpath.{code}

  The only question in my mind is what the default should be. We're all used to the current behaviour, but I'd guess that any nontrivial project where the form of the distributable matters (i.e. the source/AOT mix), providing as much control as possible by default makes the most sense. Given the tooling that most people are using, it's trivial (and common practice, IIUC) to provide a comprehensive list of namespaces one wishes to compile, so making that the default shouldn't be a hurdle to anyone. If an escape hatch is desired, a --transitive switch to clojure.lang.Compile could be added.
Stuart Halloway made changes -
Resolution Declined [ 2 ]
Status In Progress [ 3 ] Closed [ 6 ]

People

Vote (29)
Watch (22)

Dates

  • Created:
    Updated:
    Resolved: