<< Back to previous view

[CLJ-1761] clojure.core/run! does not always return nil Created: 17/Jun/15  Updated: 17/Jul/15  Resolved: 17/Jul/15

Status: Closed
Project: Clojure
Component/s: None
Affects Version/s: None
Fix Version/s: Release 1.8

Type: Defect Priority: Major
Reporter: Jonas Enlund Assignee: Unassigned
Resolution: Completed Votes: 1
Labels: None

Attachments: Text File clj-1761.patch     Text File clj-1761-with-tests.patch    
Patch: Code and Test
Approval: Ok

 Description   

According to the documentation clojure.core/run! should return nil. This is not the case as seen by the following examples:

user=> (run! identity [1])
1
user=> (run! reduced (range))
0

Approach: return 'nil'

Patch: clj-1761-with-tests.patch

Screened by: Alex Miller






[CLJ-1659] compile leaks files Created: 16/Feb/15  Updated: 17/Jul/15  Resolved: 17/Jul/15

Status: Closed
Project: Clojure
Component/s: None
Affects Version/s: Release 1.6
Fix Version/s: Release 1.8

Type: Defect Priority: Major
Reporter: Ralf Schmitt Assignee: Unassigned
Resolution: Completed Votes: 3
Labels: compiler, ft

Attachments: Text File clj-1659.patch     Text File clj-1659-v2.patch     Text File clj-1659-v3.patch    
Patch: Code
Approval: Ok

 Description   

clojure's compile function leaks file descriptors, i.e. it relies on garbage collection to close the files. I'm trying to use boot [1] on windows and ran into the problem, that files could not be deleted intermittently [2]. The problem is that clojure's compile function, or rather clojure.lang.RT.lastModified() relies on garbage collection to close files. lastModified() looks like:

static public long lastModified(URL url, String libfile) throws IOException{
	if(url.getProtocol().equals("jar")) {
		return ((JarURLConnection) url.openConnection()).getJarFile().getEntry(libfile).getTime();
	}
	else {
		return url.openConnection().getLastModified();
	}
}

Here's the stacktrace from file leak detector [3]:

#205 C:\Users\ralf\.boot\tmp\Users\ralf\home\steinmetz\2mg\-x24pa9\steinmetz\fx\config.clj by thread:clojure-agent-send-off-pool-0 on Sat Feb 14 19:58:46 UTC 2015
    at java.io.FileInputStream.(FileInputStream.java:139)
    at java.io.FileInputStream.(FileInputStream.java:93)
    at sun.net.www.protocol.file.FileURLConnection.connect(FileURLConnection.java:90)
    at sun.net.www.protocol.file.FileURLConnection.initializeHeaders(FileURLConnection.java:110)
    at sun.net.www.protocol.file.FileURLConnection.getLastModified(FileURLConnection.java:178)
    at clojure.lang.RT.lastModified(RT.java:390)
    at clojure.lang.RT.load(RT.java:421)
    at clojure.lang.RT.load(RT.java:411)
    ...

Cause: getLastModified() opens the URLConnection's InputStream but does not close it.

Approach: On Stackoverflow [4] there's a discussion on how to close the URLConnection correctly.

On non-Windows operating systems this shouldn't be much of a problem. But on windows this hurts very much, since you can't delete files that are opened by some process.

Patch: clj-1659-v3.patch

Screened by: Alex Miller

[1] http://boot-clj.com/
[2] https://github.com/boot-clj/boot/issues/117
[3] http://file-leak-detector.kohsuke.org/
[4] http://stackoverflow.com/questions/9150200/closing-urlconnection-and-inputstream-correctly



 Comments   
Comment by Pietro Menna [ 06/May/15 11:10 AM ]

First attempt Patch

Comment by Pietro Menna [ 06/May/15 11:13 AM ]

Hi Alex,

This is my first patch to Clojure and to any OSS. So maybe I will need a little guidance. I follow the steps on how to generate the patch and just uploaded the patch to this thread.

The link from Stack Overflow was good, but unfortunately it is not possible to cast to HttpURLConnection in order to have the .disconnect() method.

Please, let me know if I should attempt anything else.

Kind regards,

Pietro

Comment by Andy Fingerhut [ 06/May/15 11:20 AM ]

Seems that creating a test for this to be run on every build might be difficult.

Have you verified that on Windows it has the desired effect?

Comment by Ralf Schmitt [ 06/May/15 4:49 PM ]

I don't understand how the patch solves that issue. It just sets connection to null. Or am I missing something?

You can test for the file leak with the following program. This works on windows and Linux for me.

leak-test.clj
;; test file leak
;; on UNIX-like systems set hard limit on open files via
;; ulimit -H -n 200 and then run
;; java -jar clojure-1.6.0.jar leak-test.clj

(let [file (java.io.File. "test-leak.txt")
      url (.. file toURI toURL)]
  (doseq [x (range 2000)]
    (print x " ")
    (flush)
    (spit file "")
    (clojure.lang.RT/lastModified url nil)
    (assert (.delete file) "delete failed")))
Comment by Ralf Schmitt [ 06/May/15 5:21 PM ]

dammit, the formatting is wrong. but this patch seems to fix the problem for me (tested on linux).

Comment by Ralf Schmitt [ 06/May/15 6:16 PM ]

indentation fixed

Comment by Andy Fingerhut [ 07/May/15 8:45 AM ]

Ralf, thanks for the patch. I can't say if or when this ticket will be considered for a change to Clojure, but I do know that patches are only considered if they were written by someone who has signed a CA. Were you considering doing so? You can do it on-line here if you wish: http://clojure.org/contributing

Also, patches should be in a slightly different format that include the author's name, email, date of change, etc. Instructions for creating a patch in that format are here: http://dev.clojure.org/display/community/Developing+Patches

Comment by Ralf Schmitt [ 07/May/15 8:50 AM ]

Thanks for the explanation. I've signed the CA and I will update the patch.

Comment by Ralf Schmitt [ 07/May/15 9:34 AM ]

patch vs current master, created with git format-patch

Comment by Andy Fingerhut [ 26/May/15 8:41 AM ]

Ralph, it would be good if all attached files on a ticket have different names, for clarity when referring to them. Could you remove your clj-1659.patch and upload it with a different name, e.g. clj-1659-v2.patch ?

Comment by Ralf Schmitt [ 26/May/15 8:49 AM ]

upload my last patch with non-conflicting filename

Comment by Ralf Schmitt [ 26/May/15 8:52 AM ]

yes, sorry about that. I don't know how to remove my previous patches. (ok, found it now)

Comment by Sven Richter [ 08/Jun/15 2:57 AM ]

This one is keeping me from using boot on windows and seeing how far the 1.7 release process is I would like to express my strong wish that this one makes it into the 1.7 release.

Thanks everyone for their hard work

Comment by Alex Miller [ 18/Jun/15 4:09 PM ]

The clj-1659-v3.patch is same as v2, just removed unneeded braces to match rest of compiler.





[CLJ-1645] 'javap -v' on protocol class reveals no source file Created: 16/Jan/15  Updated: 17/Jul/15  Resolved: 17/Jul/15

Status: Closed
Project: Clojure
Component/s: None
Affects Version/s: Release 1.6
Fix Version/s: Release 1.8

Type: Defect Priority: Minor
Reporter: Fabio Tudone Assignee: Unassigned
Resolution: Completed Votes: 0
Labels: compiler, ft, protocols, source
Environment:

Mac OS X Yosemite.

java version "1.8.0_25"
Java(TM) SE Runtime Environment (build 1.8.0_25-b17)
Java HotSpot(TM) 64-Bit Server VM (build 25.25-b02, mixed mode)


Attachments: Text File CLJ-1645-protocol-class-has-no-source-file-information.patch     Text File CLJ-1645-protocol-class-has-no-source-file-information-w-repl.patch    
Patch: Code and Test
Approval: Ok

 Description   

Through "javap -v" I can find source filename information in Clojure-generated datatype class files but not in protocol ones.

Approach: In gen-interface, if the *source-path* indicates this is not a REPL-generated interface, invoke the proper ASM method to set the source file. Per JVM expectations, this is the actual file name, not the file path.

Patch: CLJ-1645-protocol-class-has-no-source-file-information-w-repl.patch

Screened by: Alex Miller



 Comments   
Comment by Fabio Tudone [ 22/Jun/15 11:45 AM ]

Any chances this will get into Clojure 1.7.0?

Comment by Alex Miller [ 22/Jun/15 12:28 PM ]

No, sorry.

Comment by Alex Miller [ 22/Jun/15 1:10 PM ]

It looks like the patch does not handle the repl case where the file will be "NO_SOURCE_FILE". Can you add a (when (not= "NO_SOURCE_FILE" source-path) ...) check around calling the visitSource invocation?

Comment by Fabio Tudone [ 22/Jun/15 3:16 PM ]

As requested

Comment by Alex Miller [ 22/Jun/15 3:57 PM ]

My last comment formatted my "earmuff stars" around source-path and you included that in the patch, which doesn't compile. Should be *source-path*, not source-path in the condition.

Comment by Fabio Tudone [ 22/Jun/15 4:02 PM ]

Right, re-attaching.

Comment by Alex Miller [ 22/Jun/15 4:18 PM ]

Fabio, this looks good and I would like to move it forward, but I just realized we don't have a signed Contributor Agreement from you. If you could do so, that would be great - the information and electronic form can be found at http://clojure.org/contributing .

Comment by Fabio Tudone [ 22/Jun/15 4:24 PM ]

Sure, with pleasure. Please also consider that the original patch is not mine though, but Yanxiang Lou's.

Comment by Fabio Tudone [ 22/Jun/15 4:27 PM ]

Signed.





[CLJ-1644] into-array fails for sequences starting with nil Created: 15/Jan/15  Updated: 17/Jul/15  Resolved: 17/Jul/15

Status: Closed
Project: Clojure
Component/s: None
Affects Version/s: None
Fix Version/s: Release 1.8

Type: Defect Priority: Minor
Reporter: Michael Blume Assignee: Michael Blume
Resolution: Completed Votes: 0
Labels: arrays, ft

Attachments: Text File CLJ-1644-array-first-nil-v1.patch     Text File CLJ-1644-array-first-nil-v2.patch    
Patch: Code and Test
Approval: Ok

 Description   

The into-array doc string implies that into-array will fall back to an Object array if aseq is supplied, but if the first element of aseq is nil, an NPE occurs.

user=> (doc into-array)
-------------------------
clojure.core/into-array
([aseq] [type aseq])
  Returns an array with components set to the values in aseq. The array's
  component type is type if provided, or the type of the first value in
  aseq if present, or Object. All values in aseq must be compatible with
  the component type. Class objects for the primitive types can be obtained
  using, e.g., Integer/TYPE.
user=> (into-array [nil 1 2])
NullPointerException   clojure.lang.RT.seqToTypedArray (RT.java:1691)

Approach: Check for nil and use Object as the array type.

Patch: CLJ-1644-array-first-nil-v2.patch

Screened by: Alex Miller



 Comments   
Comment by Michael Blume [ 15/Jan/15 1:45 PM ]

uploading patch v1 (adding nil as a cons-able element in CLJ-1643 will also bring this out, but I don't want to make the one patch depend on the other)

Comment by Phill Wolf [ 12/Mar/15 7:06 PM ]

Searching through the sequence for a non-null item is not consistent with into-array's docstring. The docstring says, "The array's component type is type if provided, or the type of the first value in aseq if present, or Object." In keeping with the docstring, wouldn't a null first item suggest an array of Object?

Working harder than that (by searching the sequence) only delays the inevitable: a whole sequence of nulls producing an Object array, instead of an array of the type the programmer expected, and triggering a run-time crash.

In summary: this patch goes farther than necessary, but even so, it does not cure the risk of unexpected results from nulls. A simpler remedy – returning an array of Object if the first item is null – would be consistent with the docstring and avoid raising unfounded expectations. Adding a statement that null is of type Object to the docstring could help programmers avoid falling into the trap.

Comment by Alex Miller [ 12/Mar/15 8:29 PM ]

No search through the sequence will pass screening, please just add a nil check.

Comment by Michael Blume [ 13/Mar/15 4:39 PM ]

done





[CLJ-1588] StackOverflow in clojure.test macroexpand with `are` and anonymous `fn` Created: 13/Nov/14  Updated: 17/Jul/15  Resolved: 17/Jul/15

Status: Closed
Project: Clojure
Component/s: None
Affects Version/s: Release 1.6
Fix Version/s: Release 1.8

Type: Defect Priority: Major
Reporter: Nikita Prokopov Assignee: Unassigned
Resolution: Completed Votes: 0
Labels: ft, template

Attachments: Text File clj-1588-2.patch     Text File clj-1588.patch     Text File clojure-test.recursion.patch    
Patch: Code and Test
Approval: Ok

 Description   

I noticed that this happens when an argument in anonymous `fn` is named the same as one of the binding in `are` form

(use 'clojure.test)
(deftest x
  (are [x y] (= x y)
    ((fn [x] (inc x)) 1) 2))
=>
clojure.lang.Compiler$CompilerException: java.lang.StackOverflowError, compiling:(/Users/nprokopov/Dropbox/ws/clojure.unicode/test/clojure/test_unicode.clj:54:3)

Approach: Replace prewalk-replace with postwalk-replace. prewalk-replace first replaces the form and then goes inside it looking for more replacements. postwalk-replace avoids that.

Patch: clj-1588-2.patch

Screened by: Alex Miller



 Comments   
Comment by Michael Blume [ 13/Nov/14 11:45 PM ]

yep, I bet it's trying to replace the x with (fn [x] (inc x)) and then replacing the x in that and...

this doesn't seem like that much of a defect? Like why write the code with the same variable names in the first place?

Comment by Nikita Prokopov [ 13/Nov/14 11:49 PM ]

Well, logically these are two completely separate, isolated Xes. It can be avoided, sure, but this behavior is not expected and I’m sure can be easily fixed.

Comment by Nikita Prokopov [ 13/Nov/14 11:53 PM ]

I mean, there shouldn’t be any recursion at all in the first place, right?

Comment by Nikita Prokopov [ 14/Nov/14 2:12 AM ]

Patch

Comment by Nikita Prokopov [ 14/Nov/14 2:13 AM ]

I fixed the issue by replacing prewalk with postwalk. It was caused by prewalk-replace that first replaced the form and then goes inside it looking for more replacements. Postwalk-replace avoids that.

Comment by Alex Miller [ 14/Nov/14 9:27 AM ]

1) Needs tests.
2) As a change in template, this affects many possible users. Can you assess other possible users either in core or in other external projects and whether they are affected? I have found http://crossclj.info to be helpful for questions like this.

Comment by Alex Miller [ 14/Nov/14 11:42 AM ]

Please include tests in a single combined patch and update the description to include a line specifying the current active patch for consideration.

Comment by Nikita Prokopov [ 14/Nov/14 11:46 AM ]

Alex, I attached a test case.

I also took a look at all use-cases of apply-template and do-template in both clojure.core and third-party projects. It’s not used very often, and in all cases use of do-template is pretty straightforward (take [x y z] and just replace it with some completely different forms), it does not depend on recursion and change from prewalk to postwalk will not cause any change in behavior. I think this patch is safe.

Comment by Nikita Prokopov [ 14/Nov/14 12:00 PM ]

I’m afraid I cannot edit description...

src/clj/clojure/template.clj => line 43
test/clojure/test_clojure/test.clj => lines 83-85

Comment by Alex Miller [ 14/Nov/14 10:14 PM ]

You should have edit rights now on jira issues.

Comment by Alex Miller [ 11/Apr/15 7:40 AM ]

Updated patch to proper format. The test in the patch does not pass:

[java] FAIL in (clj-1588-symbols-in-are-isolated-from-test-clauses) (:)
     [java]  but got :pass
     [java] expected: (= ((fn [x] (inc x)) 1) 2)
     [java]   actual: (#object[clojure.core$_EQ_ 0x7e2e5d92 "clojure.core$_EQ_@7e2e5d92"] 2 2)
Comment by Nikita Prokopov [ 12/Apr/15 6:14 AM ]

clojure.test-clojure.test now uses custom test reporter that relies on specific message. Updated patch to work around that





[CLJ-1565] pprint issues infinite output for a protocol Created: 15/Oct/14  Updated: 17/Jul/15  Resolved: 17/Jul/15

Status: Closed
Project: Clojure
Component/s: None
Affects Version/s: Release 1.6
Fix Version/s: Release 1.8

Type: Defect Priority: Major
Reporter: Michael Nygard Assignee: Unassigned
Resolution: Completed Votes: 5
Labels: ft, print, protocols

Attachments: Text File CLJ-1565.patch     File fix-pprint-var.diff    
Patch: Code
Approval: Ok

 Description   

Using pprint with a protocol name generates an unending stream of output. pprint appears to recurse through the Var reference as the value of the :var key in the protocol definition itself.

To reproduce:

user=> (defprotocol Foo (foo-you [this]))
Foo
user=> (prn Foo)
{:on user.Foo, :on-interface user.Foo, :sigs {:foo-you {:name foo-you, :arglists ([this]), :doc nil}}, :var #'user/Foo, :method-map {:foo-you :foo-you}, :method-builders {#'user/foo-you #object[user$eval1261$fn__1262 0x629ea441 "user$eval1261$fn__1262@629ea441"]}}
user=> (pprint Foo)
{:on user.Foo,
 :on-interface user.Foo,
 :sigs {:foo-you {:doc nil, :arglists ([this]), :name foo-you}},
 :var
 #<Var@6a3b02d8:
   {:on user.Foo,
    :on-interface user.Foo,
    :sigs {:foo-you {:doc nil, :arglists ([this]), :name foo-you}},
    :var
    #<Var@6a3b02d8:
      {:on user.Foo,
       :on-interface user.Foo,
       :sigs {:foo-you {:doc nil, :arglists ([this]), :name foo-you}},
       :var
       #<Var@6a3b02d8:
         {:on user.Foo,
          :on-interface user.Foo,
          :sigs
          {:foo-you {:doc nil, :arglists ([this]), :name foo-you}},
          :var
          #<Var@6a3b02d8:
            {:on user.Foo,
             :on-interface user.Foo,
             :sigs
             {:foo-you {:doc nil, :arglists ([this]), :name foo-you}},
             :var
;; forever

Approach: Add a dispatch method which doesn't deref the Var

user=> (defprotocol Foo (foo-you [this]))
Foo
user=> (prn Foo)
{:on user.Foo, :on-interface user.Foo, :sigs {:foo-you {:name foo-you, :arglists ([this]), :doc nil}}, :var #'user/Foo, :method-map {:foo-you :foo-you}, :method-builders {#'user/foo-you #object[user$eval10$fn__11 0x2452a793 "user$eval10$fn__11@2452a793"]}}
user=> (pprint Foo)
{:on user.Foo,
 :on-interface user.Foo,
 :sigs {:foo-you {:name foo-you, :arglists ([this]), :doc nil}},
 :var #'user/Foo,
 :method-map {:foo-you :foo-you},
 :method-builders
 {#'user/foo-you
  #object[user$eval10$fn__11 0x2452a793 "user$eval10$fn__11@2452a793"]}}
nil

Patch: CLJ-1565.patch
Screened by: Alex Miller



 Comments   
Comment by Andy Fingerhut [ 12/Nov/14 1:26 PM ]

I've run across this issue while debugging Eastwood. It probably does more than what you want in terms of modifying pprint behavior, but check out eastwood.util/pprint-meta here: https://github.com/jonase/eastwood/blob/master/src/eastwood/util.clj#L206

Comment by Daniel Marjenburgh [ 12/Nov/14 2:29 PM ]

The issue is that the simple-dispatch multifn dispatched a clojure.lang.Var to clojure.lang.IDeref, which dereferenced the Var before printing it. We have created a patch which dispatches a Var to the default print fn.

– With regards from the Amsterdam Clojure meetup group

Comment by Nicola Mometto [ 12/Nov/14 7:13 PM ]

The patch for this ticket also addressed http://dev.clojure.org/jira/browse/CLJ-1576

Comment by Alex Miller [ 29/Apr/15 11:28 AM ]

Patch needs a test.

Comment by Aspasia Beneti [ 01/May/15 6:49 AM ]

I added a test to the fix. Not sure if it is the right place for the test, any feedback is welcome.





[CLJ-1562] some->,some->>,cond->,cond->> and as-> doesn't work with (recur) Created: 11/Oct/14  Updated: 17/Jul/15  Resolved: 17/Jul/15

Status: Closed
Project: Clojure
Component/s: None
Affects Version/s: Release 1.6
Fix Version/s: Release 1.8

Type: Defect Priority: Critical
Reporter: Nahuel Greco Assignee: Unassigned
Resolution: Completed Votes: 0
Labels: ft

Attachments: Text File 1562-tests.patch     Text File fix-CLJ-1418_and_1562.patch    
Patch: Code and Test
Approval: Ok

 Description   

some-> and his friends doesn't work with recur, because they never place the last expression in tail position. For example:

(loop [l [1 2 3]] 
  (some-> l 
          next 
          recur))

raises UnsupportedOperationException: Can only recur from tail position

This is similar to the bug reported for as-> at http://dev.clojure.org/jira/browse/CLJ-1418 (see the comment at http://dev.clojure.org/jira/browse/CLJ-1418?focusedCommentId=35702&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-35702)

It can be fixed by changing the some-> definition to:

(defmacro some->
  "When expr is not nil, threads it into the first form (via ->),
  and when that result is not nil, through the next etc"
  {:added "1.5"}
  [expr & forms]
  (let [g (gensym)
        pstep (fn [step] `(if (nil? ~g) nil (-> ~g ~step)))]
    `(let [~g ~expr
           ~@(interleave (repeat g) (map pstep (butlast forms)))]
       ~(if forms
          (pstep (last forms))
          g))))

Similar fixes can be done for some->>, cond->, cond->> and as->.

Note -> supports recur without problems, fixing this will homogenize *-> macros behaviour.

Patch: fix-CLJ-1418_and_1562.patch (code) and 1562-tests.patch (tests)

Screened by: Alex Miller

Examples before/after:

1) (as-> 0 x (inc x))

;; before
(clojure.core/let [x 0 x (inc x)] x)

;; after
(clojure.core/let [x 0] (inc x))

2) (some-> 1 (- 2))

;; before
(clojure.core/let
 [G__5 1
  G__5 (if (clojure.core/nil? G__5) nil (clojure.core/-> G__5 (- 2)))]
 G__5)

;; after
(clojure.core/let
 [G__21 1]
 (if (clojure.core/nil? G__21) nil (clojure.core/-> G__21 (- 2))))

3) (some->> 1 (- 2))

;; before
(clojure.core/let
 [G__8 1
  G__8 (if (clojure.core/nil? G__8) nil (clojure.core/->> G__8 (- 2)))]
 G__8)

;; after
(clojure.core/let
 [G__24 1]
 (if (clojure.core/nil? G__24) nil (clojure.core/->> G__24 (- 2))))

4) (cond-> 0 true inc true (- 2))

;; before
(clojure.core/let
 [G__14 0
  G__14 (if true (clojure.core/-> G__14 inc) G__14)
  G__14 (if true (clojure.core/-> G__14 (- 2)) G__14)]
 G__14)

;; after
(clojure.core/let
 [G__30 0 
  G__30 (if true (clojure.core/-> G__30 inc) G__30)]
 (if true (clojure.core/-> G__30 (- 2)) G__30))

5) (cond->> 0 true inc true (- 2))

;; before
(clojure.core/let
 [G__11 0
  G__11 (if true (clojure.core/->> G__11 inc) G__11)
  G__11 (if true (clojure.core/->> G__11 (- 2)) G__11)]
 G__11)

;; after
(clojure.core/let
 [G__27 0 
  G__27 (if true (clojure.core/->> G__27 inc) G__27)]
 (if true (clojure.core/->> G__27 (- 2)) G__27))


 Comments   
Comment by Alex Miller [ 29/Apr/15 11:15 AM ]

Would be great if there was a patch here to consider that covered the set of affected macros.

Comment by Nahuel Greco [ 03/May/15 12:39 PM ]

Attached a patch from https://github.com/nahuel/clojure/compare/fix-CLJ-1418/1562 . This patch also fixes CLJ-1418

Comment by Alex Miller [ 04/May/15 11:17 AM ]

This patch looks like a great start - I will need to look at it further. One thing I just noticed is that none of these macros has any tests. I would love to take this opportunity to rectify that by adding tests for all of them.

Comment by Rich Hickey [ 14/Jul/15 12:44 PM ]

It would be helpful to see before and after macroexpansions

Comment by Alex Miller [ 14/Jul/15 1:58 PM ]

Added example expansions before/after





[CLJ-1533] Oddity in type tag usage for primInvoke Created: 24/Sep/14  Updated: 17/Jul/15  Resolved: 17/Jul/15

Status: Closed
Project: Clojure
Component/s: None
Affects Version/s: Release 1.6
Fix Version/s: Release 1.8

Type: Defect Priority: Critical
Reporter: Andy Fingerhut Assignee: Unassigned
Resolution: Completed Votes: 0
Labels: ft, typehints

Attachments: Text File 0001-CLJ-1533-inject-original-var-form-meta-in-constructe.patch     Text File clj-1533-2.patch    
Patch: Code and Test
Approval: Ok

 Description   

Some odd behavior demonstrated in Clojure 1.6.0 REPL below. Why does the (Math/abs (f2 -3)) call issue a reflection warning? It seems like perhaps it should not, given the other examples.

user=> (clojure-version)
"1.6.0"
user=> (set! *warn-on-reflection* true)
true
user=> (defn ^{:tag 'long} f1 [x] (inc x))
#'user/f1
user=> (Math/abs (f1 -3))
2
user=> (defn ^{:tag 'long} f2 [^long x] (inc x))
#'user/f2
user=> (Math/abs (f2 -3))
Reflection warning, NO_SOURCE_PATH:6:1 - call to static method abs on java.lang.Math can't be resolved (argument types: java.lang.Object).
2
user=> (defn ^{:tag 'long} f3 ^long [^long x] (inc x))
#'user/f3
user=> (Math/abs (f3 -3))
2

Cause: invokePrim path does not take into account var or form meta

Approach: apply var and form meta to invokePrim expression

Patch: clj-1533-2.patch

Screened by: Alex Miller



 Comments   
Comment by Nicola Mometto [ 25/Sep/14 9:47 AM ]

The issue is similar to http://dev.clojure.org/jira/browse/CLJ-1491

Comment by Nicola Mometto [ 25/Sep/14 9:58 AM ]

The root cause was also almost the same, the proposed patch is a superset of the one proposed for CLJ-1491

Comment by Alex Miller [ 25/Sep/14 10:09 AM ]

Can we include 1491 cases in this ticket and mark 1491 a duplicate?

Comment by Alex Miller [ 25/Sep/14 10:09 AM ]

Also needs tests in the patch.

Comment by Nicola Mometto [ 25/Sep/14 10:23 AM ]

Updated the patch with testcases for both issues, I agree that CLJ-1491 should be closed as duplicate

Comment by Alex Miller [ 04/May/15 8:52 AM ]

New patch is identical, just refreshed to apply to master





[CLJ-1528] clojure.test/inc-report-counter is not thread safe Created: 19/Sep/14  Updated: 17/Jul/15  Resolved: 17/Jul/15

Status: Closed
Project: Clojure
Component/s: None
Affects Version/s: Release 1.7
Fix Version/s: Release 1.8

Type: Defect Priority: Critical
Reporter: Alexander Redington Assignee: Alexander Redington
Resolution: Completed Votes: 1
Labels: ft, test
Environment:

OS X, Clojure 1.7, Macbook pro


Attachments: File fix-CLJ-1528.diff    
Patch: Code
Approval: Ok

 Description   

clojure.test/inc-report-counter function combines dereferencing the report-counters ref and operating on the previous state of the ref, leading to race conditions during concurrent use.

(dosync (commute *report-counters* assoc name
                 (inc (or (*report-counters* name) 0))))

Approach: Rewrite update function to be entirely in terms of the old state:

(dosync (commute *report-counters* update-in [name] (fnil inc 0)))

Patch: fix-CLJ-1528.diff
Screened by: Alex Miller



 Comments   
Comment by Alexander Redington [ 19/Sep/14 10:58 AM ]

Fixes 1528





[CLJ-1399] missing field munging when recreating deftypes serialized into byte code Created: 02/Apr/14  Updated: 17/Jul/15  Resolved: 17/Jul/15

Status: Closed
Project: Clojure
Component/s: None
Affects Version/s: Release 1.7
Fix Version/s: Release 1.8

Type: Defect Priority: Critical
Reporter: Kevin Downey Assignee: Unassigned
Resolution: Completed Votes: 2
Labels: compiler, deftype, ft

Attachments: File clj-1399.diff     File clj-1399-with-test.diff    
Patch: Code and Test
Approval: Ok

 Description   

deftypes with fields whose names get munged fail when constructed in data reader functions.

user=> (deftype Foo [hello-world])
user.Foo
user=> (alter-var-root #'default-data-readers assoc 'foo (fn [x] (->Foo x)))
{inst #'clojure.instant/read-instant-date, uuid #'clojure.uuid/default-uuid-reader, foo #object[user$eval12$fn__13 0x23c89df9 "user$eval12$fn__13@23c89df9"]}
user=> #foo "1"
CompilerException java.lang.IllegalArgumentException: No matching field found: hello-world for class user.Foo, compiling:(NO_SOURCE_PATH:0:0)

Cause: To embed deftypes in the bytecode the compiler emits the value of each field, then emits a call to the deftypes underlying class's constructor. To get a list of fields the compiler calls .getBasis. The getBasis fields are the "clojure" level field names of the deftype, which the actual "jvm" level field names have been munged (replacing - with _, etc), so the compiler tries to generate code to set values on non-existent fields.

Approach: Munge the field name before emitting it in bytecode.
Patch: clj-1399-with-test.diff
Screened by: Alex Miller



 Comments   
Comment by Kevin Downey [ 02/Apr/14 4:26 PM ]

reproducing case

$ rlwrap java -server -Xmx1G -Xms1G -jar /Users/hiredman/src/clojure/target/clojure-1.6.0-master-SNAPSHOT.jar
Clojure 1.6.0-master-SNAPSHOT
user=> (deftype Foo [hello-world])
user.Foo
user=> (alter-var-root #'default-data-readers assoc 'foo (fn [x] (Foo. x)))
{foo #<user$eval6$fn__7 user$eval6$fn__7@2f953efd>, inst #'clojure.instant/read-instant-date, uuid #'clojure.uuid/default-uuid-reader}
user=> #foo "1"
CompilerException java.lang.IllegalArgumentException: No matching field found: hello-world for class user.Foo, compiling:(NO_SOURCE_PATH:0:0)
user=>
Comment by Kevin Downey [ 02/Apr/14 4:39 PM ]

this patch fixes the issue on the latest master for me

Comment by Chas Emerick [ 02/Apr/14 4:57 PM ]

FWIW, this was precipitated by real experience (I think I created the refheap paste). The workaround is easy (don't use dashes in field names of deftypes you want to return from data reader functions), but I wouldn't expect anyone to guess that that wasn't already oversensitized to munging edge cases.

Comment by Alex Miller [ 29/Apr/15 11:36 AM ]

Could the patch have a test?

Comment by Kevin Downey [ 29/Apr/15 1:20 PM ]

clj-1399-with-test.diff adds a test





[CLJ-1313] Correct a few unit tests Created: 23/Dec/13  Updated: 17/Jul/15  Resolved: 17/Jul/15

Status: Closed
Project: Clojure
Component/s: None
Affects Version/s: Release 1.5, Release 1.6
Fix Version/s: Release 1.8

Type: Defect Priority: Major
Reporter: Andy Fingerhut Assignee: Unassigned
Resolution: Completed Votes: 1
Labels: ft

Attachments: File clj-1313-v1.diff     File clj-1313-v2.diff     Text File clj-1313-v3.patch    
Patch: Code and Test
Approval: Ok

 Description   

Several unit tests do not test what they appear to have been intended to test, because of missing is statements around (= expr1 expr2) expressions, or because of use of (is (thrown? ...)) instead of (is (thrown-with-msg? ...))

Patch: clj-1313-v3.patch

Screened by: Alex Miller



 Comments   
Comment by Andy Fingerhut [ 23/Dec/13 3:11 AM ]

Patch clj-1313-v1.diff wraps (is ...) around several = expressions in unit tests that appeared to have been missing them, and changes several thrown? to thrown-with-msg? when there were regexes that were unused.

Comment by Stuart Halloway [ 31/Jan/14 12:36 PM ]

please update to apply cleanly on master

Comment by Andy Fingerhut [ 31/Jan/14 3:29 PM ]

clj-1313-v2.diff is identical to clj-1313-v1.diff except that it removes the portion that conflicts with the latest Clojure master. That portion needs updating for a different reason anyway (ticket CLJ-1328), and is probably best put into a patch for that ticket.

Comment by Andy Fingerhut [ 04/Apr/15 1:36 PM ]

clj-1313-v3.patch adds two other test corrections/improvements that were not in the -v2 patch. The change from doall to dorun is not really a correction, as much as cleaning up the fact that the return value of doall was discarded.





[CLJ-1250] Reducer (and folder) instances hold onto the head of seqs Created: 03/Sep/13  Updated: 17/Jul/15  Resolved: 17/Jul/15

Status: Closed
Project: Clojure
Component/s: None
Affects Version/s: Release 1.5
Fix Version/s: Release 1.8

Type: Defect Priority: Major
Reporter: Christophe Grand Assignee: Unassigned
Resolution: Completed Votes: 10
Labels: compiler, memory, reducers

Attachments: Text File after-change.txt     Text File before-change.txt     Text File CLJ-1250-08-29.patch     Text File CLJ-1250-08-29-ws.patch     Text File CLJ-1250-20131211.patch     Text File clj-1250-2.patch     Text File CLJ-1250-AllInvokeSites-20140204.patch     Text File CLJ-1250-AllInvokeSites-20140320.patch     Text File clj1250.patch    
Patch: Code and Test
Approval: Ok

 Description   

Problem Statement
A shared function #'clojure.core.reducers/reducer holds on to the head of a reducible collection, causing it to blow up when the collection is a lazy sequence.

Reproduction steps:
Compare the following calls:

(time (reduce + 0 (map identity (range 1e8))))
(time (reduce + 0 (r/map identity (range 1e8))))

The second call should fail on a normal or small heap.

(If reducers are faster than seqs, increase the range.)

Cause: #'reducer closes over a collection when in order to reify CollReduce, and the closed-over is never cleared. When code attempts to reduce over this anonymous transformed collection, it will realize the tail while the head is stored in the closed-over.

Patch: clj-1250-2.patch

Approach:

Clear the reference to 'this' on the stack just before a tail call occurs

Removes calls to emitClearLocals(), which is a no-op.

When the context is RETURN (indicating a tail call), and the operation
is an InvokeExpr, StaticMethodExpr, or InstanceMethodExpr, clear the
reference to 'this' which is in slot 0 of the locals.

Edge-case: Inside the body of a try block, we cannot clear 'this' at the tail
position as we might need to keep refs around for use in the catch or finally
clauses. Introduces another truthy dynamic binding var to track position being
inside a try block.

Adds two helpers to emitClearThis and inTailCall.

Advantages: Fixes this case with no user code changes. Enables GC to do reclaim closed-overs references earlier.
Disadvantages: A compiler change.

Screened by: Alex Miller

Alternate Approaches:

1) Reimplement the #'reducer (and #'folder) transformation fns similar to the manner that Christophe proposes here:

(defrecord Reducer [coll xf])

(extend-protocol 
  clojure.core.protocols/CollReduce
  Reducer
      (coll-reduce [r f1]
                   (clojure.core.protocols/coll-reduce r f1 (f1)))
      (coll-reduce [r f1 init]
                   (clojure.core.protocols/coll-reduce (:coll r) ((:xf r) f1) init)))

(def rreducer ->Reducer) 

(defn rmap [f coll]
  (rreducer coll (fn [g] 
                   (fn [acc x]
                     (g acc (f x))))))

Advantages: Relatively non-invasive change.
Disadvantages: Not evident code. Additional protocol dispatch, though only incurred once

2) Alternate approach

from Christophe Grand:
Another way would be to enhance the local clearing mechanism to also clear "this" but it's complex since it may be needed to keep a reference to "this" for caches long after obvious references to "this" are needed.

Advantages: Fine-grained
Disadvantages: Complex, invasive, and the compiler is hard to hack on.

Mitigations
Avoid reducing on lazy-seqs and instead operate on vectors / maps, or custom reifiers of CollReduce or CollFold. This could be easier with some implementations of common collection functions being available (like iterate and partition).

See https://groups.google.com/d/msg/clojure-dev/t6NhGnYNH1A/2lXghJS5HywJ for previous discussion.



 Comments   
Comment by Gary Fredericks [ 03/Sep/13 8:53 AM ]

Fixed indentation in description.

Comment by Ghadi Shayban [ 11/Dec/13 11:08 PM ]

Adding a patch that clears "this" before tail calls. Verified that Christophe's repro case is fixed.

Will upload a diff of the bytecode soon.

Any reason this juicy bug was taken off 1.6?

Comment by Ghadi Shayban [ 11/Dec/13 11:17 PM ]

Here's the bytecode for the clojure.core.reducers/reducer reify before and after the change... Of course a straight diff isn't useful because all the line numbers changed. Kudos to Gary Trakhman for the no.disassemble lein plugin.

Comment by Christophe Grand [ 12/Dec/13 6:58 AM ]

Ghadi, I'm a bit surprised by this part of the patch: was the local clearing always a no-op here?

-		if(context == C.RETURN)
+		if(shouldClear)
 			{
-			ObjMethod method = (ObjMethod) METHOD.deref();
-			method.emitClearLocals(gen);
+                            gen.visitInsn(Opcodes.ACONST_NULL);
+                            gen.visitVarInsn(Opcodes.ASTORE, 0);
 			}

The problem with this approach (clear this on tail call) is that it adds yet another special case. To me the complexity stem from having to keep this around even if the user code doesn't refer to it.

Comment by Ghadi Shayban [ 12/Dec/13 7:19 AM ]

Thank you - I failed to mention this in the commit message: it appears that emitClearLocals() belonging to both ObjMethod and FnMethod (its child) are empty no-ops. I believe the actual local clearing is on line 4855.

I agree re: another special case in the compiler.

Comment by Alex Miller [ 12/Dec/13 8:56 AM ]

Ghadi re 1.6 - this ticket was never in the 1.6 list, it has not yet been vetted by Rich but is ready to do so when we open up again after 1.6.

Comment by Ghadi Shayban [ 12/Dec/13 8:59 AM ]

Sorry I confused the critical list with the Rel1.6 list.

Comment by Ghadi Shayban [ 14/Dec/13 11:16 AM ]

New patch 20131214 that handles all tail invoke sites (InvokeExpr + StaticMethodExpr + InstanceMethodExpr). 'StaticInvokeExpr' seems like an old remnant that had no active code path, so that was left as-is.

The approach taken is still the same as the original small patch that addressed only InvokeExpr, except that it is now using a couple small helpers. The commit message has more details.

Also a 'try' block with no catch or finally clause now becomes a BodyExpr. Arguably a user error, historically accepted, and still accepted, but now they are a regular BodyExpr, instead of being wrapped by a the no-op try/catch mechanism. This second commit can be optionally discarded.

With this patch on my machine (4/8 core/thread Ivy Bridge) running on bare clojure.main:
Christophe's test cases both run i 3060ms on a artificially constrained 100M max heap, indicating a dominant GC overhead. (But they now both work!)

When max heap is at a comfortable 2G the reducers version outpaces the lazyseq at 2100ms vs 2600ms!

Comment by Ghadi Shayban [ 13/Jan/14 10:48 AM ]

Updating stale patch after latest changes to master. Latest is CLJ-1250-AllInvokeSites-20140113

Comment by Ghadi Shayban [ 04/Feb/14 3:50 PM ]

Updating patch after murmur changes

Comment by Tassilo Horn [ 13/Feb/14 4:52 AM ]

Ghadi, I suffer from the problem of this issue. Therefore, I've applied your patch CLJ-1250-AllInvokeSites-20140204.patch to the current git master. However, then I get lots of "java.lang.NoSuchFieldError: array" errors when the clojure tests are run:

     [java] clojure.test-clojure.clojure-set
     [java] 
     [java] java.lang.NoSuchFieldError: array
     [java] 	at clojure.core.protocols$fn__6026.invoke(protocols.clj:123)
     [java] 	at clojure.core.protocols$fn__5994$G__5989__6003.invoke(protocols.clj:19)
     [java] 	at clojure.core.protocols$fn__6023.invoke(protocols.clj:147)
     [java] 	at clojure.core.protocols$fn__5994$G__5989__6003.invoke(protocols.clj:19)
     [java] 	at clojure.core.protocols$seq_reduce.invoke(protocols.clj:31)
     [java] 	at clojure.core.protocols$fn__6017.invoke(protocols.clj:48)
     [java] 	at clojure.core.protocols$fn__5968$G__5963__5981.invoke(protocols.clj:13)
     [java] 	at clojure.core$reduce.invoke(core.clj:6213)
     [java] 	at clojure.set$difference.doInvoke(set.clj:61)
     [java] 	at clojure.lang.RestFn.invoke(RestFn.java:442)
     [java] 	at clojure.test_clojure.clojure_set$fn__1050$fn__1083.invoke(clojure_set.clj:109)
     [java] 	at clojure.test_clojure.clojure_set$fn__1050.invoke(clojure_set.clj:109)
     [java] 	at clojure.test$test_var$fn__7123.invoke(test.clj:704)
     [java] 	at clojure.test$test_var.invoke(test.clj:704)
     [java] 	at clojure.test$test_vars$fn__7145$fn__7150.invoke(test.clj:721)
     [java] 	at clojure.test$default_fixture.invoke(test.clj:674)
     [java] 	at clojure.test$test_vars$fn__7145.invoke(test.clj:721)
     [java] 	at clojure.test$default_fixture.invoke(test.clj:674)
     [java] 	at clojure.test$test_vars.invoke(test.clj:718)
     [java] 	at clojure.test$test_all_vars.invoke(test.clj:727)
     [java] 	at clojure.test$test_ns.invoke(test.clj:746)
     [java] 	at clojure.core$map$fn__2665.invoke(core.clj:2515)
     [java] 	at clojure.lang.LazySeq.sval(LazySeq.java:40)
     [java] 	at clojure.lang.LazySeq.seq(LazySeq.java:49)
     [java] 	at clojure.lang.Cons.next(Cons.java:39)
     [java] 	at clojure.lang.RT.boundedLength(RT.java:1655)
     [java] 	at clojure.lang.RestFn.applyTo(RestFn.java:130)
     [java] 	at clojure.core$apply.invoke(core.clj:619)
     [java] 	at clojure.test$run_tests.doInvoke(test.clj:761)
     [java] 	at clojure.lang.RestFn.applyTo(RestFn.java:137)
     [java] 	at clojure.core$apply.invoke(core.clj:617)
     [java] 	at clojure.test.generative.runner$run_all_tests$fn__527.invoke(runner.clj:255)
     [java] 	at clojure.test.generative.runner$run_all_tests$run_with_counts__519$fn__523.invoke(runner.clj:251)
     [java] 	at clojure.test.generative.runner$run_all_tests$run_with_counts__519.invoke(runner.clj:251)
     [java] 	at clojure.test.generative.runner$run_all_tests.invoke(runner.clj:253)
     [java] 	at clojure.test.generative.runner$test_dirs.doInvoke(runner.clj:304)
     [java] 	at clojure.lang.RestFn.applyTo(RestFn.java:137)
     [java] 	at clojure.core$apply.invoke(core.clj:617)
     [java] 	at clojure.test.generative.runner$_main.doInvoke(runner.clj:312)
     [java] 	at clojure.lang.RestFn.invoke(RestFn.java:408)
     [java] 	at user$eval564.invoke(run_tests.clj:3)
     [java] 	at clojure.lang.Compiler.eval(Compiler.java:6657)
     [java] 	at clojure.lang.Compiler.load(Compiler.java:7084)
     [java] 	at clojure.lang.Compiler.loadFile(Compiler.java:7040)
     [java] 	at clojure.main$load_script.invoke(main.clj:274)
     [java] 	at clojure.main$script_opt.invoke(main.clj:336)
     [java] 	at clojure.main$main.doInvoke(main.clj:420)
     [java] 	at clojure.lang.RestFn.invoke(RestFn.java:408)
     [java] 	at clojure.lang.Var.invoke(Var.java:379)
     [java] 	at clojure.lang.AFn.applyToHelper(AFn.java:154)
     [java] 	at clojure.lang.Var.applyTo(Var.java:700)
     [java] 	at clojure.main.main(main.java:37)
Comment by Ghadi Shayban [ 13/Feb/14 8:23 AM ]

Can you give some details about your JVM/environment that can help reproduce? I'm not encountering this error.

Comment by Tassilo Horn [ 13/Feb/14 9:41 AM ]

Sure. It's a 64bit ThinkPad running GNU/Linux.

% java -version
java version "1.7.0_51"
OpenJDK Runtime Environment (IcedTea 2.4.5) (ArchLinux build 7.u51_2.4.5-1-x86_64)
OpenJDK 64-Bit Server VM (build 24.51-b03, mixed mode)
Comment by Ghadi Shayban [ 13/Feb/14 10:19 AM ]

Strange, that is exactly my mail env, OpenJDK7 on Arch, 64-bit. I have also tested on JDK 6/7/8 on OSX mavericks. Are you certain that the git tree is clean besides the patch? (Arch users unite!)

Comment by Tassilo Horn [ 14/Feb/14 1:13 AM ]

Yes, the tree is clean. But now I see that I get the same error also after resetting to origin/master, so it's not caused by your patch at all. Oh, now the error vanished after doing a `mvn clean`! So problem solved.

Comment by Nicola Mometto [ 19/Feb/14 12:32 PM ]

Ghandi, FnExpr.parse should bind IN_TRY_BLOCK to false before analyzing the fn body, consider the case

(try (do something (fn a [] (heap-consuming-op a))) (catch Exception e ..))

Here in the a function the this local will never be cleared even though it's perfectly safe to.
Admittedly this is an edge case but we should cover this possibility too.

Comment by Ghadi Shayban [ 19/Feb/14 2:06 PM ]

You may have auto-corrected my name to Ghandi instead of Ghadi. I wish I were that wise =)

I will update the patch for FnExpr (that seems reasonable), but maybe after 1.6 winds down and the next batch of tickets get scrutiny. It would be nice to get input on a preferred approach from Rich or core after it gets vetted – or quite possibly not vetted.

Comment by Nicola Mometto [ 19/Feb/14 6:11 PM ]

hah, sorry for the typo on the name

Seems reasonable to me, in the meantime I just pushed to tools.analyzer/tools.emitter complete support for "this" clearing, I'll test this a bit in the next few days to make sure it doesn't cause unexpected problems.

Comment by Andy Fingerhut [ 24/Feb/14 12:13 PM ]

Patch CLJ-1250-AllInvokeSites-20140204.patch no longer applies cleanly to latest master as of Feb 23, 2014. It did on Feb 14, 2014. Most likely some of its context lines are changed by the commit to Clojure master on Feb 23, 2014 – I haven't checked in detail.

Comment by Ghadi Shayban [ 20/Mar/14 4:39 PM ]

Added a patch that 1) applies cleanly, 2) binds the IN_TRY_EXPR to false initially when analyzing FnExpr and 3) uses RT.booleanCast

Comment by Alex Miller [ 22/Aug/14 9:31 AM ]

Can you squash the patch and add tests to cover all this stuff?

Comment by Ghadi Shayban [ 22/Aug/14 9:47 AM ]

Sure. Have any ideas for how to test proper behavior of reference clearing? Know of some prior art in the test suite?

Comment by Alex Miller [ 22/Aug/14 10:25 AM ]

Something like the test in the summary would be a place to start. I don't know of any test that actually inspects bytecode or anything but that's probably not wise anyways. Need to make that kind of a test but get coverage on the different kinds of scenarios you're covering - try/catch, etc.

Comment by Ghadi Shayban [ 22/Aug/14 12:13 PM ]

Attached new squashed patch with a couple of tests.

Removed (innocuous but out-of-scope) second commit that analyzed try blocks missing a catch or finally clause as BodyExprs

Comment by Ghadi Shayban [ 29/Aug/14 11:43 AM ]

Rebased to latest master. Current patch CLJ-1250-08-29

Comment by Jozef Wagner [ 29/Aug/14 2:40 PM ]

CLJ-1250-08-29.patch is fishy, 87k size and it includes many unrelated commits

Comment by Alex Miller [ 29/Aug/14 2:44 PM ]

Agreed, Ghadi that last rebase looks wrong.

Comment by Ghadi Shayban [ 29/Aug/14 3:06 PM ]

Oops. Used format-patch against the wrong base. Updated.

Apologies that ticket is longer than War & Peace

Comment by Alex Miller [ 08/Sep/14 7:02 PM ]

I have not had enough time to examine all the bytecode diffs that I want to on this yet but preliminary feedback:

Compiler.java

  • need to use tabs instead of spaces to blend into the existing code better
  • why do StaticFieldExpr and InstanceFieldExpr not need this same logic?

compilation.clj

  • has some whitespace diffs that you could get rid of
  • there seem to be more cases in the code than are covered in the tests here?
Comment by Ghadi Shayban [ 08/Sep/14 11:19 PM ]

The germ of the issue is to clear the reference to 'this' (arg 0) when transferring control to another activation frame. StaticFieldExpr and InstanceFieldExpr do not transfer control to another frame. (StaticMethod and InstanceMethod do transfer control, and are covered by the patch)

Comment by Alex Miller [ 25/Sep/14 9:03 AM ]

Makes sense - can you address the tabs and whitespace issues?

Comment by Ghadi Shayban [ 26/Sep/14 12:51 PM ]

Latest patch CLJ-1250-08-29-ws.patch with whitespace issues fixed.

Comment by Michael Blume [ 17/Jun/15 4:43 PM ]

Patch doesn't apply, will attempt to fix and upload

Comment by Michael Blume [ 17/Jun/15 4:58 PM ]

Nope, sorry, admitting defeat on this one. Ghadi, can you update?

Comment by Ghadi Shayban [ 22/Jun/15 9:40 PM ]

I've updated the patch for first 1.8 merge window.

Comment by Alex Miller [ 23/Jun/15 7:37 AM ]

Added clj-1250-2.patch which is same as clj1250.patch but squashes commits and fixes whitespace tab/space issues.

Comment by Nicola Mometto [ 17/Jul/15 8:24 PM ]

Not sure what it means but the testcase is failing with an OOM exception in the IBM JDK 1.6 instance http://build.clojure.org/job/clojure-test-matrix/284/jdk=IBM%20JDK%201.6/console

Comment by Alex Miller [ 17/Jul/15 10:29 PM ]

CLJ-1780 created to track the ibm failure. maybe just difference in gc speed or something?





[CLJ-1208] Namespace is not loaded on defrecord class init Created: 03/May/13  Updated: 17/Jul/15  Resolved: 17/Jul/15

Status: Closed
Project: Clojure
Component/s: None
Affects Version/s: None
Fix Version/s: Release 1.8

Type: Enhancement Priority: Major
Reporter: Tim McCormack Assignee: Unassigned
Resolution: Completed Votes: 5
Labels: compiler, defrecord, deftype

Attachments: Text File 0001-CLJ-1208-load-own-namespace-in-deftype-defrecord-cla.patch     Text File 0001-CLJ-1208-load-own-namespace-in-deftype-defrecord-cla-v2.patch     Text File 0001-CLJ-1208-load-own-namespace-in-deftype-defrecord-cla-v3.patch     Text File 0001-CLJ-1208-load-own-namespace-in-deftype-defrecord-cla-v4.patch     Text File 0001-CLJ-1208-load-own-namespace-in-deftype-defrecord-cla-v5-no-opts.patch     Text File 0001-CLJ-1208-load-own-namespace-in-deftype-defrecord-cla-v5.patch    
Patch: Code and Test
Approval: Ok

 Description   

As a user of Clojure interop from Java, I want defrecords (and deftypes?) to load their namespaces upon class initialization so that I can simply construct and use AOT'd record classes without manually requiring their namespaces first.

Calling the defrecord's constructor may or may not result in "Attempting to call unbound fn" exceptions, depending on what code has already been run.

This issue has been raised several times over the years, but I could not find an existing ticket for it:

Screened: Approach 2, which makes loading the namespace optional. This makes the change purely additive, whereas Approach 1 changes semantics (load ordering) of existing programs. I don't know why we would want that risk, especially when it is not clear that most users of deftype would even want ns loading side effect.

Approach 1: require the namespace a record/type belongs to during the record/type class init
Patch: 0001-CLJ-1208-load-own-namespace-in-deftype-defrecord-cla-v5-no-opts.patch

Approach 2: like Approach 1 but does the automatic loading only when a :load-ns option is set to true in the deftype/defrecord
Patch: 0001-CLJ-1208-load-own-namespace-in-deftype-defrecord-cla-v5.patch

Note: patch for Approach 1 causes some generative tests to fail since the namespace used to evalaute a defrecord is immediately destroyed thus impossible to load



 Comments   
Comment by Nicola Mometto [ 18/Jan/15 7:10 AM ]

The attached patch approaches this issue by adding a :load-ns options to deftype/defrecord which defaults to false.
When true, the type/record be compiled with a call to clojure.core/require to its originating namespace in its static initializer.

The patch has two known limitations:

  • clojure.core deftypes/defrecords cannot have :load-ns since we use clojure.core/require to load the namespaces so clojure.core needs to be loaded manually anyway
  • clojure.lang.Compiler/demunge is used to get the originating namespace from the deftype/defrecord class name, this means that namespaces like foo_bar are not supported since they get demunged into foo-bar. If this is something that needs to be addressed, it shouldn't be too hard to just pass the unmunged namespace name in the opts map.
Comment by Nicola Mometto [ 22/Jan/15 12:59 PM ]

Updated patch fixing a whitespace error and mentionint :load-ns in the docstrings of deftype/defrecord

Comment by Nicola Mometto [ 11/Mar/15 6:12 AM ]

Updated patch so it applies on lastest HEAD

Comment by Michael Blume [ 17/Jun/15 12:12 PM ]

No longer applies I'm afraid

Comment by Nicola Mometto [ 17/Jun/15 12:22 PM ]

Thanks Michael, updated the patch. I have to say it's getting kind of annoying having to maintain a patch for months without any feedback.

Comment by Alex Miller [ 17/Jun/15 1:03 PM ]

What are the negative impacts if this is always done, rather than being an option?

Comment by Alex Miller [ 17/Jun/15 1:06 PM ]

Also, you should never rely on demunge - it's best-effort for printing purposes only.

Comment by Nicola Mometto [ 17/Jun/15 1:09 PM ]

Does extra bytecode emitted count as a negative impact?

Comment by Alex Miller [ 17/Jun/15 1:15 PM ]

No, I'm not concerned about that.

Comment by Nicola Mometto [ 17/Jun/15 1:48 PM ]

Attached patch that doesn't use demunge but change the macroexpansion of defrecord and deftype to include the namespace segment in the tagsym in deftype* special form

Comment by Nicola Mometto [ 17/Jun/15 2:02 PM ]

Alex, I attached two versions of the last patch, one with :load-ns and one without.
Making :load-ns the default behaviour causes some generative tests to fail since they immediately eliminate the namespace used to defrecord making the record class fail when trying to load said namespace.

I can try to change those tests if necessary.

Comment by Michael Blume [ 10/Jul/15 6:09 PM ]

v5-no-opts breaks generative tests on my box:

test-generative:
     [java] java.util.concurrent.ExecutionException: java.lang.ExceptionInInitializerError, compiling:(/Users/michael.blume/workspace/clojure/src/script/run_test_generative.clj:5:1)
     [java] 	at java.util.concurrent.FutureTask.report(FutureTask.java:122)
     [java] 	at java.util.concurrent.FutureTask.get(FutureTask.java:192)
     [java] 	at clojure.core$deref_future.invoke(core.clj:2186)
     [java] 	at clojure.core$future_call$reify__6736.deref(core.clj:6683)
     [java] 	at clojure.core$deref.invoke(core.clj:2206)
     [java] 	at clojure.core$map$fn__4553.invoke(core.clj:2622)
     [java] 	at clojure.lang.LazySeq.sval(LazySeq.java:40)
     [java] 	at clojure.lang.LazySeq.seq(LazySeq.java:56)
     [java] 	at clojure.lang.RT.seq(RT.java:507)
     [java] 	at clojure.core$seq__4128.invoke(core.clj:137)
     [java] 	at clojure.core$concat$cat__4217$fn__4218.invoke(core.clj:700)
     [java] 	at clojure.lang.LazySeq.sval(LazySeq.java:40)
     [java] 	at clojure.lang.LazySeq.seq(LazySeq.java:49)
     [java] 	at clojure.lang.ChunkedCons.chunkedNext(ChunkedCons.java:59)
     [java] 	at clojure.core$chunk_next.invoke(core.clj:673)
     [java] 	at clojure.core.protocols$fn__6518.invoke(protocols.clj:139)
     [java] 	at clojure.core.protocols$fn__6478$G__6473__6487.invoke(protocols.clj:19)
     [java] 	at clojure.core.protocols$seq_reduce.invoke(protocols.clj:31)
     [java] 	at clojure.core.protocols$fn__6506.invoke(protocols.clj:101)
     [java] 	at clojure.core.protocols$fn__6452$G__6447__6465.invoke(protocols.clj:13)
     [java] 	at clojure.core$reduce.invoke(core.clj:6519)
     [java] 	at clojure.test.generative.runner$run_suite.invoke(runner.clj:185)
     [java] 	at clojure.test.generative.runner$_main.doInvoke(runner.clj:208)
     [java] 	at clojure.lang.RestFn.invoke(RestFn.java:408)
     [java] 	at user$eval644.invoke(run_test_generative.clj:5)
     [java] 	at clojure.lang.Compiler.eval(Compiler.java:6797)
     [java] 	at clojure.lang.Compiler.load(Compiler.java:7242)
     [java] 	at clojure.lang.Compiler.loadFile(Compiler.java:7180)
     [java] 	at clojure.main$load_script.invoke(main.clj:275)
     [java] 	at clojure.main$script_opt.invoke(main.clj:337)
     [java] 	at clojure.main$main.doInvoke(main.clj:421)
     [java] 	at clojure.lang.RestFn.invoke(RestFn.java:408)
     [java] 	at clojure.lang.Var.invoke(Var.java:379)
     [java] 	at clojure.lang.AFn.applyToHelper(AFn.java:154)
     [java] 	at clojure.lang.Var.applyTo(Var.java:700)
     [java] 	at clojure.main.main(main.java:37)
     [java] Caused by: java.lang.ExceptionInInitializerError, compiling:(/Users/michael.blume/workspace/clojure/src/script/run_test_generative.clj:5:1)
     [java] 	at clojure.lang.Compiler.analyzeSeq(Compiler.java:6745)
     [java] 	at clojure.lang.Compiler.analyze(Compiler.java:6539)
     [java] 	at clojure.lang.Compiler.analyze(Compiler.java:6500)
     [java] 	at clojure.lang.Compiler$BodyExpr$Parser.parse(Compiler.java:5880)
     [java] 	at clojure.lang.Compiler$FnMethod.parse(Compiler.java:5311)
     [java] 	at clojure.lang.Compiler$FnExpr.parse(Compiler.java:3925)
     [java] 	at clojure.lang.Compiler.analyzeSeq(Compiler.java:6736)
     [java] 	at clojure.lang.Compiler.analyze(Compiler.java:6539)
     [java] 	at clojure.lang.Compiler.analyzeSeq(Compiler.java:6726)
     [java] 	at clojure.lang.Compiler.analyze(Compiler.java:6539)
     [java] 	at clojure.lang.Compiler.access$300(Compiler.java:38)
     [java] 	at clojure.lang.Compiler$DefExpr$Parser.parse(Compiler.java:578)
     [java] 	at clojure.lang.Compiler.analyzeSeq(Compiler.java:6738)
     [java] 	at clojure.lang.Compiler.analyze(Compiler.java:6539)
     [java] 	at clojure.lang.Compiler.analyzeSeq(Compiler.java:6726)
     [java] 	at clojure.lang.Compiler.analyze(Compiler.java:6539)
     [java] 	at clojure.lang.Compiler.analyze(Compiler.java:6500)
     [java] 	at clojure.lang.Compiler$BodyExpr$Parser.parse(Compiler.java:5878)
     [java] 	at clojure.lang.Compiler$LetExpr$Parser.parse(Compiler.java:6194)
     [java] 	at clojure.lang.Compiler.analyzeSeq(Compiler.java:6738)
     [java] 	at clojure.lang.Compiler.analyze(Compiler.java:6539)
     [java] 	at clojure.lang.Compiler.analyze(Compiler.java:6500)
     [java] 	at clojure.lang.Compiler$BodyExpr$Parser.parse(Compiler.java:5880)
     [java] 	at clojure.lang.Compiler$FnMethod.parse(Compiler.java:5311)
     [java] 	at clojure.lang.Compiler$FnExpr.parse(Compiler.java:3925)
     [java] 	at clojure.lang.Compiler.analyzeSeq(Compiler.java:6736)
     [java] 	at clojure.lang.Compiler.analyze(Compiler.java:6539)
     [java] 	at clojure.lang.Compiler.eval(Compiler.java:6794)
     [java] 	at clojure.lang.Compiler.eval(Compiler.java:6760)
     [java] 	at clojure.core$eval.invoke(core.clj:3081)
     [java] 	at clojure.test_clojure.generators$generate_namespaces$make_in_ns__830.invoke(generators.clj:30)
     [java] 	at clojure.test_clojure.generators$generate_namespaces$fn__836$fn__837.invoke(generators.clj:41)
     [java] 	at clojure.core$map$fn__4553.invoke(core.clj:2622)
     [java] 	at clojure.lang.LazySeq.sval(LazySeq.java:40)
     [java] 	at clojure.lang.LazySeq.seq(LazySeq.java:49)
     [java] 	at clojure.lang.RT.seq(RT.java:507)
     [java] 	at clojure.core$seq__4128.invoke(core.clj:137)
     [java] 	at clojure.core$concat$fn__4215.invoke(core.clj:691)
     [java] 	at clojure.lang.LazySeq.sval(LazySeq.java:40)
     [java] 	at clojure.lang.LazySeq.seq(LazySeq.java:49)
     [java] 	at clojure.lang.RT.seq(RT.java:507)
     [java] 	at clojure.core$seq__4128.invoke(core.clj:137)
     [java] 	at clojure.core$concat$cat__4217$fn__4218.invoke(core.clj:700)
     [java] 	at clojure.lang.LazySeq.sval(LazySeq.java:40)
     [java] 	at clojure.lang.LazySeq.seq(LazySeq.java:49)
     [java] 	at clojure.lang.RT.seq(RT.java:507)
     [java] 	at clojure.core$seq__4128.invoke(core.clj:137)
     [java] 	at clojure.core.protocols$seq_reduce.invoke(protocols.clj:30)
     [java] 	at clojure.core.protocols$fn__6506.invoke(protocols.clj:101)
     [java] 	at clojure.core.protocols$fn__6452$G__6447__6465.invoke(protocols.clj:13)
     [java] 	at clojure.core$reduce.invoke(core.clj:6519)
     [java] 	at clojure.core$into.invoke(core.clj:6600)
     [java] 	at clojure.test_clojure.generators$generate_namespaces.invoke(generators.clj:44)
     [java] 	at clojure.test_clojure.generators$fn__841.invoke(generators.clj:50)
     [java] 	at clojure.lang.Delay.deref(Delay.java:37)
     [java] 	at clojure.core$deref.invoke(core.clj:2206)
     [java] 	at clojure.test_clojure.generators$var.invoke(generators.clj:58)
     [java] 	at clojure.test.generative.runner$eval559$fn__560$fn__561$fn__562$fn__563.invoke(runner.clj:66)
     [java] 	at clojure.core$map$fn__4553.invoke(core.clj:2622)
     [java] 	at clojure.lang.LazySeq.sval(LazySeq.java:40)
     [java] 	at clojure.lang.LazySeq.seq(LazySeq.java:49)
     [java] 	at clojure.lang.RT.seq(RT.java:507)
     [java] 	at clojure.core$seq__4128.invoke(core.clj:137)
     [java] 	at clojure.core.protocols$seq_reduce.invoke(protocols.clj:30)
     [java] 	at clojure.core.protocols$fn__6506.invoke(protocols.clj:101)
     [java] 	at clojure.core.protocols$fn__6452$G__6447__6465.invoke(protocols.clj:13)
     [java] 	at clojure.core$reduce.invoke(core.clj:6519)
     [java] 	at clojure.core$into.invoke(core.clj:6600)
     [java] 	at clojure.test.generative.runner$eval559$fn__560$fn__561$fn__562.invoke(runner.clj:66)
     [java] 	at clojure.core$repeatedly$fn__5111.invoke(core.clj:4921)
     [java] 	at clojure.lang.LazySeq.sval(LazySeq.java:40)
     [java] 	at clojure.lang.LazySeq.seq(LazySeq.java:49)
     [java] 	at clojure.lang.RT.seq(RT.java:507)
     [java] 	at clojure.lang.RT.nthFrom(RT.java:924)
     [java] 	at clojure.lang.RT.nth(RT.java:883)
     [java] 	at clojure.test.generative.runner$run_one$fn__588$fn__589$fn__590.invoke(runner.clj:94)
     [java] 	at clojure.test.generative.runner$run_one$fn__588$fn__589.invoke(runner.clj:93)
     [java] 	at clojure.core$binding_conveyor_fn$fn__4444.invoke(core.clj:1916)
     [java] 	at clojure.lang.AFn.call(AFn.java:18)
     [java] 	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
     [java] 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
     [java] 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
     [java] 	at java.lang.Thread.run(Thread.java:745)
     [java] Caused by: java.lang.ExceptionInInitializerError
     [java] 	at java.lang.Class.forName0(Native Method)
     [java] 	at java.lang.Class.forName(Class.java:344)
     [java] 	at clojure.lang.RT.classForName(RT.java:2154)
     [java] 	at clojure.lang.RT.classForName(RT.java:2163)
     [java] 	at clojure.lang.Compiler$HostExpr.maybeClass(Compiler.java:1017)
     [java] 	at clojure.lang.Compiler$HostExpr.access$600(Compiler.java:795)
     [java] 	at clojure.lang.Compiler$NewExpr$Parser.parse(Compiler.java:2609)
     [java] 	at clojure.lang.Compiler.analyzeSeq(Compiler.java:6738)
     [java] 	... 82 more
     [java] Caused by: java.io.FileNotFoundException: Could not locate clojure/generated/ns0__init.class or clojure/generated/ns0.clj on classpath.
     [java] 	at clojure.lang.RT.load(RT.java:449)
     [java] 	at clojure.lang.RT.load(RT.java:412)
     [java] 	at clojure.core$load$fn__5448.invoke(core.clj:5866)
     [java] 	at clojure.core$load.doInvoke(core.clj:5865)
     [java] 	at clojure.lang.RestFn.invoke(RestFn.java:408)
     [java] 	at clojure.core$load_one.invoke(core.clj:5671)
     [java] 	at clojure.core$load_lib$fn__5397.invoke(core.clj:5711)
     [java] 	at clojure.core$load_lib.doInvoke(core.clj:5710)
     [java] 	at clojure.lang.RestFn.applyTo(RestFn.java:142)
     [java] 	at clojure.core$apply.invoke(core.clj:632)
     [java] 	at clojure.core$load_libs.doInvoke(core.clj:5749)
     [java] 	at clojure.lang.RestFn.applyTo(RestFn.java:137)
     [java] 	at clojure.core$apply.invoke(core.clj:632)
     [java] 	at clojure.core$require.doInvoke(core.clj:5832)
     [java] 	at clojure.lang.RestFn.invoke(RestFn.java:408)
     [java] 	at clojure.generated.ns0.ExampleRecord-0.<clinit>(run_test_generative.clj:5)
     [java] 	... 90 more
Comment by Nicola Mometto [ 11/Jul/15 3:25 AM ]

Michael, that's already noted in the ticket description and the screened patch is v5.patch, not v5-no-opts.patch

Comment by Michael Blume [ 11/Jul/15 10:45 PM ]

Aha, missed that, sorry

Comment by Michael Blume [ 12/Jul/15 7:18 PM ]

The screened patch seems to break 'lein check' for clj-http

Compiling namespace clj-http.client
Exception in thread "main" java.lang.NoClassDefFoundError: IllegalName: compile__stub.clj_http.headers.clj-http.headers/HeaderMap, compiling:(clj_http/headers.clj:105:1)
	at clojure.lang.Compiler.analyzeSeq(Compiler.java:6749)
	at clojure.lang.Compiler.analyze(Compiler.java:6543)
	at clojure.lang.Compiler.analyze(Compiler.java:6504)
	at clojure.lang.Compiler$BodyExpr$Parser.parse(Compiler.java:5882)
	at clojure.lang.Compiler$LetExpr$Parser.parse(Compiler.java:6198)
	at clojure.lang.Compiler.analyzeSeq(Compiler.java:6742)
	at clojure.lang.Compiler.analyze(Compiler.java:6543)
	at clojure.lang.Compiler.analyze(Compiler.java:6504)
	at clojure.lang.Compiler$BodyExpr$Parser.parse(Compiler.java:5884)
	at clojure.lang.Compiler$FnMethod.parse(Compiler.java:5315)
	at clojure.lang.Compiler$FnExpr.parse(Compiler.java:3926)
	at clojure.lang.Compiler.analyzeSeq(Compiler.java:6740)
	at clojure.lang.Compiler.analyze(Compiler.java:6543)
	at clojure.lang.Compiler.eval(Compiler.java:6798)
	at clojure.lang.Compiler.eval(Compiler.java:6790)
	at clojure.lang.Compiler.load(Compiler.java:7246)
	at clojure.lang.RT.loadResourceScript(RT.java:371)
	at clojure.lang.RT.loadResourceScript(RT.java:362)
	at clojure.lang.RT.load(RT.java:446)
	at clojure.lang.RT.load(RT.java:412)
	at clojure.core$load$fn__5448.invoke(core.clj:5866)
	at clojure.core$load.doInvoke(core.clj:5865)
	at clojure.lang.RestFn.invoke(RestFn.java:408)
	at clojure.core$load_one.invoke(core.clj:5671)
	at clojure.core$load_lib$fn__5397.invoke(core.clj:5711)
	at clojure.core$load_lib.doInvoke(core.clj:5710)
	at clojure.lang.RestFn.applyTo(RestFn.java:142)
	at clojure.core$apply.invoke(core.clj:632)
	at clojure.core$load_libs.doInvoke(core.clj:5749)
	at clojure.lang.RestFn.applyTo(RestFn.java:137)
	at clojure.core$apply.invoke(core.clj:632)
	at clojure.core$require.doInvoke(core.clj:5832)
	at clojure.lang.RestFn.invoke(RestFn.java:482)
	at clj_http.core$eval237$loading__5340__auto____238.invoke(core.clj:1)
	at clj_http.core$eval237.invoke(core.clj:1)
	at clojure.lang.Compiler.eval(Compiler.java:6801)
	at clojure.lang.Compiler.eval(Compiler.java:6790)
	at clojure.lang.Compiler.load(Compiler.java:7246)
	at clojure.lang.RT.loadResourceScript(RT.java:371)
	at clojure.lang.RT.loadResourceScript(RT.java:362)
	at clojure.lang.RT.load(RT.java:446)
	at clojure.lang.RT.load(RT.java:412)
	at clojure.core$load$fn__5448.invoke(core.clj:5866)
	at clojure.core$load.doInvoke(core.clj:5865)
	at clojure.lang.RestFn.invoke(RestFn.java:408)
	at clojure.core$load_one.invoke(core.clj:5671)
	at clojure.core$load_lib$fn__5397.invoke(core.clj:5711)
	at clojure.core$load_lib.doInvoke(core.clj:5710)
	at clojure.lang.RestFn.applyTo(RestFn.java:142)
	at clojure.core$apply.invoke(core.clj:632)
	at clojure.core$load_libs.doInvoke(core.clj:5749)
	at clojure.lang.RestFn.applyTo(RestFn.java:137)
	at clojure.core$apply.invoke(core.clj:632)
	at clojure.core$require.doInvoke(core.clj:5832)
	at clojure.lang.RestFn.invoke(RestFn.java:805)
	at clj_http.client$eval83$loading__5340__auto____84.invoke(client.clj:1)
	at clj_http.client$eval83.invoke(client.clj:1)
	at clojure.lang.Compiler.eval(Compiler.java:6801)
	at clojure.lang.Compiler.eval(Compiler.java:6790)
	at clojure.lang.Compiler.load(Compiler.java:7246)
	at clojure.lang.RT.loadResourceScript(RT.java:371)
	at clojure.lang.RT.loadResourceScript(RT.java:362)
	at clojure.lang.RT.load(RT.java:446)
	at clojure.lang.RT.load(RT.java:412)
	at clojure.core$load$fn__5448.invoke(core.clj:5866)
	at clojure.core$load.doInvoke(core.clj:5865)
	at clojure.lang.RestFn.invoke(RestFn.java:408)
	at user$eval60$fn__71.invoke(form-init8647608528776892632.clj:1)
	at user$eval60.invoke(form-init8647608528776892632.clj:1)
	at clojure.lang.Compiler.eval(Compiler.java:6801)
	at clojure.lang.Compiler.eval(Compiler.java:6791)
	at clojure.lang.Compiler.load(Compiler.java:7246)
	at clojure.lang.Compiler.loadFile(Compiler.java:7184)
	at clojure.main$load_script.invoke(main.clj:275)
	at clojure.main$init_opt.invoke(main.clj:280)
	at clojure.main$initialize.invoke(main.clj:308)
	at clojure.main$null_opt.invoke(main.clj:343)
	at clojure.main$main.doInvoke(main.clj:421)
	at clojure.lang.RestFn.invoke(RestFn.java:421)
	at clojure.lang.Var.invoke(Var.java:383)
	at clojure.lang.AFn.applyToHelper(AFn.java:156)
	at clojure.lang.Var.applyTo(Var.java:700)
	at clojure.main.main(main.java:37)
Caused by: java.lang.NoClassDefFoundError: IllegalName: compile__stub.clj_http.headers.clj-http.headers/HeaderMap
	at java.lang.ClassLoader.preDefineClass(ClassLoader.java:654)
	at java.lang.ClassLoader.defineClass(ClassLoader.java:758)
	at java.lang.ClassLoader.defineClass(ClassLoader.java:642)
	at clojure.lang.DynamicClassLoader.defineClass(DynamicClassLoader.java:46)
	at clojure.lang.Compiler$NewInstanceExpr.compileStub(Compiler.java:7767)
	at clojure.lang.Compiler$NewInstanceExpr.build(Compiler.java:7632)
	at clojure.lang.Compiler$NewInstanceExpr$DeftypeParser.parse(Compiler.java:7542)
	at clojure.lang.Compiler.analyzeSeq(Compiler.java:6742)
	... 82 more
Failed.

Will try to create a minimal failing repo.

Comment by Michael Blume [ 12/Jul/15 7:31 PM ]

Looks like it just breaks potemkin's def-map-type.

Comment by Nicola Mometto [ 13/Jul/15 8:47 AM ]

This is a bug in potemkin caused by the fact that it assumes deftype expands to an unqualified second argument.
The deftype* special form takes as first argument the tagsym and as a second argument the class name, potemkin passes a wrong second argument, it expands to :

(deftype* user/Foo user.user/Foo ..)

rather than

(deftype* user/Foo user.Foo ..)

I'll open a PR fixing this issue in potemkin and link back to this ticket.

Comment by Nicola Mometto [ 13/Jul/15 1:15 PM ]

Opened PR: https://github.com/ztellman/potemkin/pull/40





[CLJ-703] Improve writeClassFile performance Created: 04/Jan/11  Updated: 17/Jul/15  Resolved: 17/Jul/15

Status: Closed
Project: Clojure
Component/s: None
Affects Version/s: Release 1.3
Fix Version/s: Release 1.8

Type: Enhancement Priority: Critical
Reporter: Jürgen Hötzel Assignee: Unassigned
Resolution: Completed Votes: 27
Labels: Compiler, ft, performance

Attachments: Text File 0001-use-File.mkdirs-instead-of-mkdir-every-single-direct.patch     Text File 0002-Ensure-atomic-creation-of-class-files-by-renaming-a-.patch     Text File clj-703-remove-sync-in-write-class-file.patch     Text File improve-writeclassfile-perf.patch     Text File remove-flush-and-sync-only.patch     Text File remove-sync-only.patch    
Patch: Code
Approval: Ok

 Description   

When writing class files to disk, Clojure currently calls flush on the FileOutputStream used, and sync on the underlying FileDescriptor for each class file it writes. This ticket proposes to remove the sync call as it provides little value, but has a detrimental effect on performance on some operating systems.

Background

The call to sync in the current implementation of Compiler.writeClassFile was added in SVN-1252 to address an issue where compile allegedly returned but changes to class files weren't visible[1]. The original report lacks detail and it's hard to discern what the actual issue was.

Calling FileDescriptor.sync tells the operating system to flush any buffered data to disk (to its best capability). This means that in the event of a system failure, the kernel buffers will not hold any data that hasn't been persisted to disk, and thus no data would be lost.

However, sync does not provide any special guarantees around visbility. Any data written using to a file descriptor that is subsequently closed is already guaranteed to be visible to all other processes, as soon as the close call returns. When not using sync, the data returned may reside only in kernel buffers, but the reading process will see no difference.

Syncing is an expensive operation and doing so after each class file is written can make compilation significantly slower.

Laurent Petit (who made the original request for this) reports that it would not make any difference for him now in CCW (https://groups.google.com/d/msg/clojure-dev/Vz9h8hqAvFk/cQvip5j_zoQJ).

Proposed solution

Remove the call to FileDescriptor.sync when writing class files to disk.

Latest patch: http://dev.clojure.org/jira/secure/attachment/14214/clj-703-remove-sync-in-write-class-file.patch

Performance implications

The performance gains of removing the sync call will vary depending on OS and file system. Some operating systems provide strong guarantees that your data will actually end up on the disk, others less so. See for example differences between fsync in Linux[2] and OSX[3].

Preliminary tests on Linux have shown compile times going from:

  • aleph.http: 13s -> 4s
  • incanter.core: 14s -> 4s

I've created a test for this and have asked the community for help testing and gathering data: http://goo.gl/forms/0b8SVt2pyN

Will update this ticket once more data is available.

Tradeoffs

In short, this change trades some degree of on-disk consistency for compilation speed. If a fault (process, OS, hardware) occurs whilst compiling, this patch may result in class files not being properly written to disk. This however, is still the case today, depending on the type of fault, the OS you're using and the fsync guarantees provided. If a compilation fails with a fatal error, it is unreasonable to expect that the output is in a consistent state.

[1]: https://groups.google.com/forum/#!topic/clojure/C-DEWzpPPUY
[2]: "Note that while fsync() will flush all data from the host to the drive (i.e. the "permanent storage device"), the drive itself may not physically write the data to the platters for quite some time and it may be written in an out-of-order sequence.": https://developer.apple.com/library/mac/documentation/Darwin/Reference/ManPages/man2/fsync.2.html
[3]: "[...] includes writing through or flushing a disk cache if present." http://man7.org/linux/man-pages/man2/fsync.2.html[2]:



 Comments   
Comment by David Powell [ 17/Jan/11 2:16 PM ]

Removing sync makes clojure build much faster. I wonder why it was added in the first place? I guess only Rich knows? I assume that it is not necessary.

If we are removing sync though, I wouldn't bother with the atomic rename stuff. Doing that sort of thing can cause problems on some platforms, eg with search indexers and virus checkers momentarily locking files after they are created.

The patch seems to be assuming that sync is there for some reason, but my initial assumption would be that sync isn't necessary - perhaps it was working around some issue that no longer exists?

Comment by Jürgen Hötzel [ 19/Jan/11 2:05 PM ]

Although its unlikely: there is a possible race condition "loading a paritally written classfile"?:

https://github.com/clojure/clojure/blob/master/src/jvm/clojure/lang/RT.java#L393

Comment by John Szakmeister [ 25/May/12 4:22 AM ]

The new improve-writeclassfile-perf version of the patch combines the two previous patches into a single patch file and brings them up-to-date with master. I can split the two changes back out into separate patch files if desired, but I figured out current tooling is more geared towards a single patch being applied.

Comment by John Szakmeister [ 25/May/12 4:36 AM ]

FWIW, both fixes look sane. The first one is a nice cleanup. The second one is a little more interesting in that uses a rename operation to put the final file into place. It removes the sync call, which does make things faster. In general, if we're concerned about on-disk consistency, we should really have a combination of the two: write the full contents to a tmp file, sync it, and atomically rename to the destination file name.

Neither the current master, nor the current patch will guarantee on-disk consistency across a machine wide crash. The current master could crash before the sync() occurs, leaving the file in an inconsistent state. With the patch, the OS may not get the file from file cache to disk before an OS level crash occurs, and leave the file in an inconsistent state as well. The benefit of the patch version is that the whole file does atomically come into view at once. It does have a nasty side effect of leaving around a temp file if the compiler crashes just before the rename though.

Perhaps a little more work to catch an exception and clean up is in order? In general, I like the patched version better.

Comment by Ivan Kozik [ 05/Oct/12 7:07 PM ]

File.renameTo returns false on (most?) errors, but the patch doesn't check for failure. Docs say "The return value should always be checked to make sure that the rename operation was successful." Failure might be especially likely on Windows, where files are opened by others without FILE_SHARE_DELETE.

Comment by Dave Della Costa [ 23/Jun/14 11:37 PM ]

We've been wondering why our compilation times on linux were so slow. It became the last straw when we walked away from one project and came back after 15 minutes and it was not done yet.

After some fruitless investigation into our linux configuration and lein java args, we stumbled upon this issue via the associated Clojure group thread. Upon commenting out the flush() and sync() lines (https://github.com/clojure/clojure/blob/master/src/jvm/clojure/lang/Compiler.java#L7171-L7172) and compiling Clojure 1.6 ourselves, our projects all started compiling in under a minute.

Point being, can we at least provide some flag to allow for "unsafe compilation" or something? As it is, this is bad enough that we've manually modified all our local versions of Clojure to work around the issue.

Comment by Tim McCormack [ 30/Sep/14 4:10 PM ]

Additional motivation: This becomes really unpleasant on an encrypted filesystem, since write and read latency become higher.

As a partial workaround, I've been using this script to mount a ramdisk over top of target, which speeds up compilation 2-4x: https://gist.github.com/timmc/6c397644668bcf41041f (but removing flush() and sync() entirely would probably speed things up even more, if safe)

Comment by Ragnar Dahlen [ 06/Oct/14 12:30 PM ]

I'd like to explore this issue further as I also don't think the flush and sync calls add any value, but do have a severe impact on performance.

To resurrect the discussion, I've attached a new patch with the following approach:

  • create a temporary file
  • write class bytecode to temporary file, no flush or sync
  • close temporary file
  • atomic rename of temporary file to class file name

It is different to previous patches in that:

  • it applies cleanly to master
  • it checks return value from File.renameTo
  • it omits proposed File.mkdirs change as the current implementation is actually converting from an "internal name", where forward slashes are assumed (splits on "/"), to a platform specific path using File.separator. I'm not convinced that the previous patch is safe on all versions of Windows, and I think it's separate from the main issue here.

I opted for the atomic rename of a temp file to avoid leaving empty class files with a correct expected class file name in case of failure.

It is my understanding that this patch will guarantee that:

  • when writeClassFile returns successfully, a class file with the expected name will exists, and subsequent reads from that file name will return the expected bytecode (possibly from kernel buffers).
  • when writeClassFile fails, a class file with the expected name will not have been created (or updated if it previously existed).

Anything preventing the operating system from flushing its buffers to disk (power failure etc) might leave a corrupt (empty, partially written) class file. It's my opinion that that is acceptable behaviour, and worth the performance improvement (I'm seeing AOT compilation reduced from 1m20s -> 22s on one of our codebases, would be happy to provide more benchmarks if requested).

Would be grateful for feedback/testing of this patch.

Comment by Ragnar Dahlen [ 08/Oct/14 6:00 AM ]

We're testing this patch on various projects/platforms at my company. So far we've seen:

  • Significantly reduced compilation times on Linux (two typical examples: 30s to 15s, 1m30s to 30s)
  • No significant change in compilation times on Mac OSX.
  • File.renameTo consistently failing on a Windows machine.

My understanding is that the performance difference between Linux and OSX is due to differences in how these platforms implement fsync. OSX by default does not actually tell the drive to flush its buffers (requires fcntl F_FULLSYNC for this, not used by JVMs) [1], whereas Linux does [2].

Our very limited test shows (as was previously pointed out) that File.renameTo is problematic on Windows. I've attached a new patch that doesn't use rename, and only has the the sync call removed (flush is a no-op for FileOutputStreams). We're currently testing this patch.

The drawback of this patch is that it may leave correctly named, but empty class files if the write fails. One option would be to try and delete the file in the catch block. Personally, I wouldn't expect a compilation that failed because of OS/IO reasons to leave my classfiles in a consistent state.

[1]: "Note that while fsync() will flush all data from the host to the drive (i.e. the "permanent storage device"), the drive itself may not physically write the data to the platters for quite some time and it may be written in an out-of-order sequence.": https://developer.apple.com/library/mac/documentation/Darwin/Reference/ManPages/man2/fsync.2.html

[2]: "[...] includes writing through or flushing a disk cache if present."
http://man7.org/linux/man-pages/man2/fsync.2.html

Comment by Alex Miller [ 17/May/15 8:24 AM ]

Currently this ticket is not a state where it can be evaluated, and I would like it to get there before we consider tickets for 1.8.

The description for this ticket needs to be something that a screener read to fully understand the problem, proposed solution, performance implications, and tradeoffs. Right now it does not seem up to date with information discussed in comments, which patch should be considered, performance data, and what we might lose by making the change. It would be great if someone could help get this in shape.

Comment by Andy Fingerhut [ 17/May/15 10:16 AM ]

Alex, you will probably get a wider audience of contributors willing to help if you sent an email to clojure-dev every once in a while with a list of tickets you want help updating. Right now I think it is this one and CLJ-1449, but it would be nice if there were a single report link contributors could click on to find out what you are looking for help with.

Normally there is the "Needs Patch" state for the next release, but for CLJ-1449 you are looking for a patch for the next-next release, so "Needs Patch" won't do it.

Perhaps if there were 'standard' labels created for 'Alex requests ticket updates' for such tickets, and filter for them?

Comment by Ragnar Dahlen [ 17/May/15 10:47 AM ]

I'd be happy to help improve the state of this ticket. I'm not the original reporter, is it still OK for me to change description etc. to address your concerns?

Comment by Alex Miller [ 17/May/15 11:02 AM ]

Andy - I will probably do so soon but I thought previous commenters would see this now.

Ragnar - absolutely!

Comment by Ragnar Dahlen [ 30/May/15 6:24 PM ]

Updated for RC1

Comment by Ragnar Dahlen [ 30/May/15 6:25 PM ]

Alex - Updated description, please let me know if I can improve it further.

Comment by Alex Miller [ 30/May/15 8:49 PM ]

Ragnar - much better - thank you! Could you also add a pointer to the patch for consideration?

Comment by Ragnar Dahlen [ 31/May/15 3:40 AM ]

Alex - Done





Generated at Wed Jul 29 23:06:56 CDT 2015 using JIRA 4.4#649-r158309.