<< Back to previous view

[CLJ-1785] Reader conditionals throws when they have nil expressions Created: 21/Jul/15  Updated: 31/Jul/15  Resolved: 31/Jul/15

Status: Closed
Project: Clojure
Component/s: None
Affects Version/s: Release 1.7
Fix Version/s: Release 1.8

Type: Defect Priority: Critical
Reporter: Jozef Wagner Assignee: Jozef Wagner
Resolution: Completed Votes: 1
Labels: reader, readerconditionals

Attachments: Text File clj-1785.patch     Text File clj-1785-v2.patch    
Patch: Code and Test
Approval: Ok

 Description   

Reader conditional that has nil as an expression fails.

e.g. (read-string {:read-cond :allow} "#?(:default nil)")

The fact that nil values are valid expressions are documented at both official documentation and design page.

Patch: clj-1785-v2.patch

Screened by: Alex Miller



 Comments   
Comment by Jozef Wagner [ 21/Jul/15 3:53 PM ]

Added patch with tests

Comment by Jozef Wagner [ 21/Jul/15 4:06 PM ]

v2 patch that uses static final sentinel value

Comment by Nicola Mometto [ 22/Jul/15 7:23 AM ]

CLJ-1138 reports a similar bug with data readers.





[CLJ-1772] Spelling mistake in clojure.test/use-fixtures Created: 01/Jul/15  Updated: 31/Jul/15  Resolved: 31/Jul/15

Status: Closed
Project: Clojure
Component/s: None
Affects Version/s: Release 1.7
Fix Version/s: Release 1.8

Type: Defect Priority: Trivial
Reporter: Daniel Compton Assignee: Unassigned
Resolution: Completed Votes: 0
Labels: docstring, ft

Attachments: Text File clj-1772.patch    
Patch: Code
Approval: Ok

 Description   

Part of the docstring for `use-fixtures` is:

individually, while:once wraps the whole run in a single function.

this should be

individually, while :once wraps the whole run in a single function.

Screened by: Alex Miller



 Comments   
Comment by Alex Miller [ 01/Jul/15 12:42 AM ]

If you can get me a patch, happy to pre-screen for next release.

Comment by Daniel Compton [ 01/Jul/15 12:43 AM ]

2015-07-01 17:43:02





[CLJ-1769] Docstrings for *' and +' refer to * and + instead of *' and +' Created: 28/Jun/15  Updated: 31/Jul/15  Resolved: 31/Jul/15

Status: Closed
Project: Clojure
Component/s: None
Affects Version/s: Release 1.6, Release 1.7
Fix Version/s: Release 1.8

Type: Defect Priority: Trivial
Reporter: Mark Simpson Assignee: Unassigned
Resolution: Completed Votes: 0
Labels: docstring, ft

Attachments: Text File docstringfix.patch    
Patch: Code
Approval: Ok

 Description   

The docstrings for *' and +' refer to the behavior of * and + if they are passed no arguments. The docstrings should refer to the behavior of *' and +' instead.



 Comments   
Comment by Andy Fingerhut [ 02/Jul/15 3:49 PM ]

Mark, your patch "patch.txt" is not in the expected format for a patch, and please use a file name ending with ".diff" or ".patch", for the convenience of patch reviewers. See this link for instructions on creating a patch in the expected format: http://dev.clojure.org/display/community/Developing+Patches

Comment by Mark Simpson [ 02/Jul/15 4:36 PM ]

Sorry about that. Hopefully I got things right this time.





[CLJ-1766] Serializing+deserializing lists breaks their hash Created: 21/Jun/15  Updated: 24/Aug/15

Status: Open
Project: Clojure
Component/s: None
Affects Version/s: Release 1.6, Release 1.7
Fix Version/s: Release 1.8

Type: Defect Priority: Major
Reporter: Chris Vermilion Assignee: Unassigned
Resolution: Unresolved Votes: 0
Labels: interop
Environment:

Tested on OS X 10.10.3, Clojure 1.6.0, Java 1.8.0_45-b14, but don't think this is a factor.


Attachments: Text File clj-1766-2.patch     Text File CLJ-1766.patch     File serialization_test_mod.diff     File serialize-test.clj    
Patch: Code and Test
Approval: Ok

 Description   

clojure.lang.PersistentList implements Serializable but a serialization roundtrip results in a hash of 0. This appears to be caused by ASeq marking its _hash property as transient; other collection base classes don't do this. I don't know if there is a rationale for the distinction, but the result is that serializing, then deserializing, a list results in its _hash property being 0 instead of either its previous, calculated value, or -1, which would trigger recalculation.

This means that any code that relies on a list's hash value can break in subtle ways.

Examples:

(import '[java.io ByteArrayInputStream ByteArrayOutputStream ObjectInputStream ObjectOutputStream])

(defn obj->bytes [obj]
  (let [bytestream (ByteArrayOutputStream.)]
    (doto (ObjectOutputStream. bytestream) (.writeObject obj))
    (.toByteArray bytestream)))

(defn bytes->obj [bs]
  (.readObject (ObjectInputStream. (ByteArrayInputStream. bs))))

(defn round-trip [x] (bytes->obj (obj->bytes x)))
user=> (hash '(1))
-1381383523
user=> (hash (round-trip '(1)))
0
user=> #{'(1) (round-trip '(1))}
#{(1) (1)}
user=> (def m {'(1) 1})
#'user/m
user=> (= m (round-trip m))
true
user=> (= (round-trip m) m)
false

Approach: Use 0 as the "uncomputed" hash value, not -1. The auto initialized value on serialization construction will then automatically trigger a re-computation.

Alternate approaches:

  • Implement a readObject method that sets the _hash property to -1. This is somewhat complicated in the face of subclasses, etc.

Note: Also need to consider other classes that use -1 instead of 0 as the uncomputed hash value: APersistentMap, APersistentSet, APersistentVector, PersistentQueue - however I believe in those cases they are not transient and thus may avoid the issue. Perhaps they should be transient though.

Patch: clj-1766-2.patch
Screened by: Alex Miller



 Comments   
Comment by Chris Vermilion [ 21/Jun/15 1:10 PM ]

Yikes, sorry about the formatting above; JIRA newbie and looks like I can't edit. Also, I should have noted that the above functions require an import: (import '[java.io ByteArrayInputStream ByteArrayOutputStream ObjectInputStream ObjectOutputStream]).

Comment by Alex Miller [ 22/Jun/15 7:55 AM ]

Yes, this is a bug. My preference would be to use 0 to indicate "not computed" and thus sidestep this issue. The downside of changing from -1 to 0 is that serialization/deserialization is then broken between versions of Clojure (although maybe if it was already broken, that's not an issue). -1 is used like this for lazily computed hashes in many classes in Clojure so the issue should really be fixed across the board.

Comment by Alex Miller [ 22/Jun/15 8:22 AM ]

There are some serialization round-trip tests in clojure.test-clojure.serialization - seems like those should be updated to compare the .hashCode and hash before/after, which should catch this. I attached a diff (not a proper patch) with that change just as a demonstration, which indeed produces a bunch of failures. That should be incorporated into any fix, possibly along with other failures.

Comment by Alex Miller [ 31/Jul/15 9:04 AM ]

Also, don't worry about crediting me on that test change, please just incorporate it into whatever gets made here.

Comment by Andrew Rosa [ 31/Jul/15 9:31 AM ]

Alex Miller I hope to work on this issue this weekend. There are some conflict with the alpha release schedule?

Comment by Alex Miller [ 31/Jul/15 9:47 AM ]

No conflict - when it's ready we'll look at it.

Comment by Andrew Rosa [ 04/Aug/15 8:21 AM ]

As Alex Miller recomended, I followed the change-default-to-zero path.
Not only it sidesteps the serialization issue, it looks more aligned with
the semantics of transient. Of course, there is no guarantees
about how different JVM implementations will deal with it - sometimes
they will be serialized, sometimes they will not.

So, together with the patch I've made some manual tests between some versions. The script used is attached for further inspection.

Running on a 1.6 REPL has shown on the original described issue:

serialize-test=> *clojure-version*
{:major 1, :minor 6, :incremental 0, :qualifier nil}
serialize-test=> (def f "seq-1-6.dump")
#'serialize-test/f
serialize-test=> (def x {'(1) 1})
#'serialize-test/x
serialize-test=> (dump-bytes f (serialize x))
nil
serialize-test=> (deserialize (slurp-bytes f))
{(1) 1}
serialize-test=> (hash x)
202919476
serialize-test=> (hash (deserialize (serialize x)))
202919476
serialize-test=> (hash (deserialize (slurp-bytes f)))
-1619826937
serialize-test=> (= x (deserialize (slurp-bytes f)))
true
serialize-test=> (= (deserialize (slurp-bytes f)) x)
false

Running on 1.8 master. Despite of the = behavior looking ok, the
hashes are different between roundtrips, like we saw on 1.6:

serialize-test=> *clojure-version*
{:major 1, :minor 8, :incremental 0, :qualifier "master", :interim true}
serialize-test=> (def f "seq-1-8-master.dump")
#'serialize-test/f
serialize-test=> (def x {'(1) 1})
#'serialize-test/x
serialize-test=> (dump-bytes f (serialize x))
nil
serialize-test=> (deserialize (slurp-bytes f))
{(1) 1}
serialize-test=> (hash x)
202919476
serialize-test=> (hash (deserialize (serialize x)))
202919476
serialize-test=> (hash (deserialize (slurp-bytes f)))
-1619826937
serialize-test=> (= x (deserialize (slurp-bytes f)))
true
serialize-test=> (= (deserialize (slurp-bytes f)) x)
true

Running on 1.8 after patch the hashes are correctly computed on both cases:

serialize-test=> *clojure-version*
{:major 1, :minor 8, :incremental 0, :qualifier "master", :interim true}
serialize-test=> (def f "seq-1-8-patch.dump")
#'serialize-test/f
serialize-test=> (def x {'(1) 1})
#'serialize-test/x
serialize-test=> (dump-bytes f (serialize x))
nil
serialize-test=> (deserialize (slurp-bytes f))
{(1) 1}
serialize-test=> (hash x)
202919476
serialize-test=> (hash (deserialize (serialize x)))
202919476
serialize-test=> (hash (deserialize (slurp-bytes f)))
202919476
serialize-test=> (= x (deserialize (slurp-bytes f)))
true
serialize-test=> (= (deserialize (slurp-bytes f)) x)
true

Please note I've dumped each serialized data into different files, so I could test the interoperability between those versions. What I found:

  • 1.6 serialization is already incompatible with 1.8, on both directions;
  • 1.8 data could be exchange between master and patched versions, not affecting version behavior (hashes are computed only on patched REPL).

Did I miss something or everything looks correct for you too? I'm open to do some more exhaustive testing if anyone could give me some directions about where to explore.

Comment by Rich Hickey [ 08/Aug/15 9:46 AM ]

This patch both fixes the serialization problem and changes the implementation for no reason related to the problem. The implementation works in place in order to avoid head-holding, which the implementation change reintroduces. Screeners - please make sure patches contain the minimal code to address the problem and don't incorporate other extraneous 'improvements'.

Comment by Ghadi Shayban [ 08/Aug/15 11:59 AM ]

Rich Hickey The only change I see is the removal of a commented-out impl.

Comment by Alex Miller [ 08/Aug/15 10:33 PM ]

I agree with Ghadi - there is no change in implementation here, just some commented (non-used) code was removed. Moving back to Screened.

Comment by Alex Miller [ 18/Aug/15 12:26 PM ]

Latest patch is identical, just does not remove the commented out code.





[CLJ-1761] clojure.core/run! does not always return nil Created: 17/Jun/15  Updated: 17/Jul/15  Resolved: 17/Jul/15

Status: Closed
Project: Clojure
Component/s: None
Affects Version/s: None
Fix Version/s: Release 1.8

Type: Defect Priority: Major
Reporter: Jonas Enlund Assignee: Unassigned
Resolution: Completed Votes: 1
Labels: None

Attachments: Text File clj-1761.patch     Text File clj-1761-with-tests.patch    
Patch: Code and Test
Approval: Ok

 Description   

According to the documentation clojure.core/run! should return nil. This is not the case as seen by the following examples:

user=> (run! identity [1])
1
user=> (run! reduced (range))
0

Approach: return 'nil'

Patch: clj-1761-with-tests.patch

Screened by: Alex Miller






[CLJ-1722] Typo in the doc string of `with-bindings` Created: 03/May/15  Updated: 31/Jul/15  Resolved: 31/Jul/15

Status: Closed
Project: Clojure
Component/s: None
Affects Version/s: Release 1.6
Fix Version/s: Release 1.8

Type: Defect Priority: Trivial
Reporter: Blake West Assignee: Unassigned
Resolution: Completed Votes: 0
Labels: docstring, ft

Attachments: Text File fixwithbindingsdocs.patch    
Patch: Code
Approval: Ok

 Description   

The doc string says "the execute body". It should say "then execute body".



 Comments   
Comment by Andy Fingerhut [ 26/May/15 8:47 AM ]

Alex, this one 'falls off the JIRA state chart' since Rich hasn't assigned it a Fix Version. Should Approval be Triaged instead?





[CLJ-1659] compile leaks files Created: 16/Feb/15  Updated: 10/Aug/15  Resolved: 17/Jul/15

Status: Closed
Project: Clojure
Component/s: None
Affects Version/s: Release 1.6
Fix Version/s: Release 1.8

Type: Defect Priority: Major
Reporter: Ralf Schmitt Assignee: Unassigned
Resolution: Completed Votes: 3
Labels: compiler, ft

Attachments: Text File clj-1659.patch     Text File clj-1659-v2.patch     Text File clj-1659-v3.patch    
Patch: Code
Approval: Ok

 Description   

clojure's compile function leaks file descriptors, i.e. it relies on garbage collection to close the files. I'm trying to use boot [1] on windows and ran into the problem, that files could not be deleted intermittently [2]. The problem is that clojure's compile function, or rather clojure.lang.RT.lastModified() relies on garbage collection to close files. lastModified() looks like:

static public long lastModified(URL url, String libfile) throws IOException{
	if(url.getProtocol().equals("jar")) {
		return ((JarURLConnection) url.openConnection()).getJarFile().getEntry(libfile).getTime();
	}
	else {
		return url.openConnection().getLastModified();
	}
}

Here's the stacktrace from file leak detector [3]:

#205 C:\Users\ralf\.boot\tmp\Users\ralf\home\steinmetz\2mg\-x24pa9\steinmetz\fx\config.clj by thread:clojure-agent-send-off-pool-0 on Sat Feb 14 19:58:46 UTC 2015
    at java.io.FileInputStream.(FileInputStream.java:139)
    at java.io.FileInputStream.(FileInputStream.java:93)
    at sun.net.www.protocol.file.FileURLConnection.connect(FileURLConnection.java:90)
    at sun.net.www.protocol.file.FileURLConnection.initializeHeaders(FileURLConnection.java:110)
    at sun.net.www.protocol.file.FileURLConnection.getLastModified(FileURLConnection.java:178)
    at clojure.lang.RT.lastModified(RT.java:390)
    at clojure.lang.RT.load(RT.java:421)
    at clojure.lang.RT.load(RT.java:411)
    ...

Cause: getLastModified() opens the URLConnection's InputStream but does not close it.

Approach: On Stackoverflow [4] there's a discussion on how to close the URLConnection correctly.

On non-Windows operating systems this shouldn't be much of a problem. But on windows this hurts very much, since you can't delete files that are opened by some process.

Patch: clj-1659-v3.patch

Screened by: Alex Miller

[1] http://boot-clj.com/
[2] https://github.com/boot-clj/boot/issues/117
[3] http://file-leak-detector.kohsuke.org/
[4] http://stackoverflow.com/questions/9150200/closing-urlconnection-and-inputstream-correctly



 Comments   
Comment by Pietro Menna [ 06/May/15 11:10 AM ]

First attempt Patch

Comment by Pietro Menna [ 06/May/15 11:13 AM ]

Hi Alex,

This is my first patch to Clojure and to any OSS. So maybe I will need a little guidance. I follow the steps on how to generate the patch and just uploaded the patch to this thread.

The link from Stack Overflow was good, but unfortunately it is not possible to cast to HttpURLConnection in order to have the .disconnect() method.

Please, let me know if I should attempt anything else.

Kind regards,

Pietro

Comment by Andy Fingerhut [ 06/May/15 11:20 AM ]

Seems that creating a test for this to be run on every build might be difficult.

Have you verified that on Windows it has the desired effect?

Comment by Ralf Schmitt [ 06/May/15 4:49 PM ]

I don't understand how the patch solves that issue. It just sets connection to null. Or am I missing something?

You can test for the file leak with the following program. This works on windows and Linux for me.

leak-test.clj
;; test file leak
;; on UNIX-like systems set hard limit on open files via
;; ulimit -H -n 200 and then run
;; java -jar clojure-1.6.0.jar leak-test.clj

(let [file (java.io.File. "test-leak.txt")
      url (.. file toURI toURL)]
  (doseq [x (range 2000)]
    (print x " ")
    (flush)
    (spit file "")
    (clojure.lang.RT/lastModified url nil)
    (assert (.delete file) "delete failed")))
Comment by Ralf Schmitt [ 06/May/15 5:21 PM ]

dammit, the formatting is wrong. but this patch seems to fix the problem for me (tested on linux).

Comment by Ralf Schmitt [ 06/May/15 6:16 PM ]

indentation fixed

Comment by Andy Fingerhut [ 07/May/15 8:45 AM ]

Ralf, thanks for the patch. I can't say if or when this ticket will be considered for a change to Clojure, but I do know that patches are only considered if they were written by someone who has signed a CA. Were you considering doing so? You can do it on-line here if you wish: http://clojure.org/contributing

Also, patches should be in a slightly different format that include the author's name, email, date of change, etc. Instructions for creating a patch in that format are here: http://dev.clojure.org/display/community/Developing+Patches

Comment by Ralf Schmitt [ 07/May/15 8:50 AM ]

Thanks for the explanation. I've signed the CA and I will update the patch.

Comment by Ralf Schmitt [ 07/May/15 9:34 AM ]

patch vs current master, created with git format-patch

Comment by Andy Fingerhut [ 26/May/15 8:41 AM ]

Ralph, it would be good if all attached files on a ticket have different names, for clarity when referring to them. Could you remove your clj-1659.patch and upload it with a different name, e.g. clj-1659-v2.patch ?

Comment by Ralf Schmitt [ 26/May/15 8:49 AM ]

upload my last patch with non-conflicting filename

Comment by Ralf Schmitt [ 26/May/15 8:52 AM ]

yes, sorry about that. I don't know how to remove my previous patches. (ok, found it now)

Comment by Sven Richter [ 08/Jun/15 2:57 AM ]

This one is keeping me from using boot on windows and seeing how far the 1.7 release process is I would like to express my strong wish that this one makes it into the 1.7 release.

Thanks everyone for their hard work

Comment by Alex Miller [ 18/Jun/15 4:09 PM ]

The clj-1659-v3.patch is same as v2, just removed unneeded braces to match rest of compiler.

Comment by Erik Dannenberg [ 06/Aug/15 12:09 PM ]

Issue still persists with Clojure 1.8.0-alpha4 and latest boot. Running Windows 8.1.

Example app that should reproduce the error: https://github.com/danielsz/sente-boot

clone, run boot dev, change a source file => file exists errors

Any chances to get a fix for this into Clojure 1.7? It's pretty much a show stopper for us as it prevents the reloaded workflow for our windows devs.

Comment by Ralf Schmitt [ 06/Aug/15 4:21 PM ]

This must be another issue. Please try to find the leak with http://file-leak-detector.kohsuke.org/

(I once dug into a similar issue in boot's bug tracker and the problem was somewhere in the cljs compiler).

Comment by Erik Dannenberg [ 10/Aug/15 11:46 AM ]

I attached the file leak detector and ran boot with clojure 1.7 and 1.8-alpha4, the stacktraces for both are a bit different but from what i can tell the leak cause is still the same and looking very similiar to the stacktrace posted in the issue description. I do however see the cljs compiler further down in the stacktrace.

Should i open an issue on CLJS tracker?

Comment by Ralf Schmitt [ 10/Aug/15 11:51 AM ]

This is hard to tell without a stacktrace...

Comment by Erik Dannenberg [ 10/Aug/15 11:59 AM ]
1 descriptors are open
#1 C:\Users\ed\.boot\cache\tmp\Users\ed\projekte\sente-system\5ho\-c4hx1l\adzerk\boot_reload.cljs by thread:clojure-agent-send-off-pool-0 on Mon Aug 10 17:54:39 CEST 2015
	at java.io.FileInputStream.<init>(FileInputStream.java:139)
	at java.io.FileInputStream.<init>(FileInputStream.java:93)
	at sun.net.www.protocol.file.FileURLConnection.connect(FileURLConnection.java:90)
	at sun.net.www.protocol.file.FileURLConnection.initializeHeaders(FileURLConnection.java:110)
	at sun.net.www.protocol.file.FileURLConnection.getLastModified(FileURLConnection.java:178)
	at cljs.util$last_modified.invokeStatic(util.cljc:136)
	at cljs.util$last_modified.invoke(util.cljc)
	at cljs.analyzer$requires_analysis_QMARK_.invokeStatic(analyzer.cljc:2179)
	at cljs.analyzer$requires_analysis_QMARK_.invoke(analyzer.cljc)
	at cljs.analyzer$requires_analysis_QMARK_.invokeStatic(analyzer.cljc:2164)
	at cljs.analyzer$requires_analysis_QMARK_.invoke(analyzer.cljc)
	at cljs.analyzer$analyze_file.invokeStatic(analyzer.cljc:2220)
	at cljs.analyzer$analyze_file.invoke(analyzer.cljc)
	at cljs.analyzer$analyze_deps.invokeStatic(analyzer.cljc:1291)
	at cljs.analyzer$analyze_deps.invoke(analyzer.cljc)
	at cljs.analyzer$eval1868$fn__1870.invoke(analyzer.cljc:1545)
	at clojure.lang.MultiFn.invoke(MultiFn.java:251)
	at cljs.analyzer$analyze_seq.invokeStatic(analyzer.cljc:1900)
	at cljs.analyzer$analyze_seq.invoke(analyzer.cljc)
	at cljs.analyzer$analyze$fn__2118.invoke(analyzer.cljc:1992)
	at cljs.analyzer$analyze.invokeStatic(analyzer.cljc:1985)
	at cljs.analyzer$analyze.invoke(analyzer.cljc)
	at cljs.compiler$compile_file_STAR_$fn__3216.invoke(compiler.cljc:1027)
	at cljs.compiler$with_core_cljs.invokeStatic(compiler.cljc:968)
	at cljs.compiler$with_core_cljs.invoke(compiler.cljc)
	at cljs.compiler$compile_file_STAR_.invokeStatic(compiler.cljc:988)
	at cljs.compiler$compile_file_STAR_.invoke(compiler.cljc)
	at cljs.compiler$compile_file$fn__3248.invoke(compiler.cljc:1129)
	at cljs.compiler$compile_file.invokeStatic(compiler.cljc:1109)
	at cljs.compiler$compile_file.invoke(compiler.cljc)
	at cljs.compiler$compile_root.invokeStatic(compiler.cljc:1181)
	at cljs.compiler$compile_root.invoke(compiler.cljc)
	at cljs.closure$compile_dir.invokeStatic(closure.clj:385)
	at cljs.closure$compile_dir.invoke(closure.clj)
	at cljs.closure$eval3614$fn__3615.invoke(closure.clj:425)
	at cljs.closure$eval3567$fn__3568$G__3558__3575.invoke(closure.clj:331)
	at cljs.closure$eval3627$fn__3628.invoke(closure.clj:439)
	at cljs.closure$eval3567$fn__3568$G__3558__3575.invoke(closure.clj:331)
	at adzerk.boot_cljs.impl.CljsSourcePaths$fn__4011.invoke(impl.clj:14)
Comment by Alex Miller [ 10/Aug/15 12:30 PM ]

Looks like similar problem in CLJS to me - I would open a ticket there.

Comment by Erik Dannenberg [ 10/Aug/15 1:35 PM ]

Done. CLJS-1416





[CLJ-1657] proxy creates bytecode that calls super methods of abstract classes Created: 08/Feb/15  Updated: 31/Jul/15  Resolved: 31/Jul/15

Status: Closed
Project: Clojure
Component/s: None
Affects Version/s: Release 1.6
Fix Version/s: Release 1.8

Type: Defect Priority: Major
Reporter: Alexander Yakushev Assignee: Stuart Halloway
Resolution: Completed Votes: 1
Labels: None
Environment:

Everywhere, but so far relevant only on Android 5.0


Attachments: File CLJ-1657-patch.diff    
Patch: Code
Approval: Ok

 Description   

When proxy is used to extend abstract classes (e.g. java.io.Writer), the bytecode it produces include the call to non-existing super methods. For example, here's decompiled method from clojure/pprint/column_writer.clj:

public void close()
    {
        Object obj;
label0:
        {
            obj = RT.get(__clojureFnMap, "close");
            if(obj == null)
                break label0;
            ((IFn)obj).invoke(this);
            break MISSING_BLOCK_LABEL_31;
        }
        JVM INSTR pop ;
        super.close();
    }

As you can see on the last line, super.close() tries to call a non-defined method (because close() is abstract in Writer).

This hasn't been an issue anywhere until Android 5.0 came out. Its bytecode optimizer is very aggressive and rejects such code. Google guys claim that it is a bug in their code, which they already fixed[1]. Still I wonder if having faulty bytecode, that is not valid by Java standards, might cause issues in future (not only on Android, but in other enviroments too).

[1] https://code.google.com/p/android/issues/detail?id=80687



 Comments   
Comment by Alexander Yakushev [ 18/Mar/15 5:31 AM ]

I attached a patch that resolves the issue. The change makes `generate-proxy` treat abstract methods like interface methods. Which means, if the implementation for the method is not provided, it will throw unsupported exception rather than try to call the parent method (which doesn't exist).

Comment by Michael Blume [ 18/Mar/15 12:50 PM ]

Alexander: Awesome, thanks =)

Note: If you use git format-patch after making a commit, you can generate a patch file with your name/e-mail and a commit message that a clojure maintainer can apply directly to clojure as a new commit.

Comment by Alex Miller [ 18/Mar/15 12:53 PM ]

The patch process is documented here: http://dev.clojure.org/display/community/Developing+Patches

Comment by Alexander Yakushev [ 18/Mar/15 4:38 PM ]

Sorry, I should have checked the guidelines first. I uploaded a new patch, hope it is correct now.

Comment by Alexander Yakushev [ 18/Jul/15 10:59 AM ]

Anything that holds this back? Now that Clojure is in volatile after-release state it's a good time to deal with it.

Comment by Alex Miller [ 18/Jul/15 12:25 PM ]

No, wasn't in triaged but is now.





[CLJ-1645] 'javap -v' on protocol class reveals no source file Created: 16/Jan/15  Updated: 17/Jul/15  Resolved: 17/Jul/15

Status: Closed
Project: Clojure
Component/s: None
Affects Version/s: Release 1.6
Fix Version/s: Release 1.8

Type: Defect Priority: Minor
Reporter: Fabio Tudone Assignee: Unassigned
Resolution: Completed Votes: 0
Labels: compiler, ft, protocols, source
Environment:

Mac OS X Yosemite.

java version "1.8.0_25"
Java(TM) SE Runtime Environment (build 1.8.0_25-b17)
Java HotSpot(TM) 64-Bit Server VM (build 25.25-b02, mixed mode)


Attachments: Text File CLJ-1645-protocol-class-has-no-source-file-information.patch     Text File CLJ-1645-protocol-class-has-no-source-file-information-w-repl.patch    
Patch: Code and Test
Approval: Ok

 Description   

Through "javap -v" I can find source filename information in Clojure-generated datatype class files but not in protocol ones.

Approach: In gen-interface, if the *source-path* indicates this is not a REPL-generated interface, invoke the proper ASM method to set the source file. Per JVM expectations, this is the actual file name, not the file path.

Patch: CLJ-1645-protocol-class-has-no-source-file-information-w-repl.patch

Screened by: Alex Miller



 Comments   
Comment by Fabio Tudone [ 22/Jun/15 11:45 AM ]

Any chances this will get into Clojure 1.7.0?

Comment by Alex Miller [ 22/Jun/15 12:28 PM ]

No, sorry.

Comment by Alex Miller [ 22/Jun/15 1:10 PM ]

It looks like the patch does not handle the repl case where the file will be "NO_SOURCE_FILE". Can you add a (when (not= "NO_SOURCE_FILE" source-path) ...) check around calling the visitSource invocation?

Comment by Fabio Tudone [ 22/Jun/15 3:16 PM ]

As requested

Comment by Alex Miller [ 22/Jun/15 3:57 PM ]

My last comment formatted my "earmuff stars" around source-path and you included that in the patch, which doesn't compile. Should be *source-path*, not source-path in the condition.

Comment by Fabio Tudone [ 22/Jun/15 4:02 PM ]

Right, re-attaching.

Comment by Alex Miller [ 22/Jun/15 4:18 PM ]

Fabio, this looks good and I would like to move it forward, but I just realized we don't have a signed Contributor Agreement from you. If you could do so, that would be great - the information and electronic form can be found at http://clojure.org/contributing .

Comment by Fabio Tudone [ 22/Jun/15 4:24 PM ]

Sure, with pleasure. Please also consider that the original patch is not mine though, but Yanxiang Lou's.

Comment by Fabio Tudone [ 22/Jun/15 4:27 PM ]

Signed.





[CLJ-1644] into-array fails for sequences starting with nil Created: 15/Jan/15  Updated: 17/Jul/15  Resolved: 17/Jul/15

Status: Closed
Project: Clojure
Component/s: None
Affects Version/s: None
Fix Version/s: Release 1.8

Type: Defect Priority: Minor
Reporter: Michael Blume Assignee: Michael Blume
Resolution: Completed Votes: 0
Labels: arrays, ft

Attachments: Text File CLJ-1644-array-first-nil-v1.patch     Text File CLJ-1644-array-first-nil-v2.patch    
Patch: Code and Test
Approval: Ok

 Description   

The into-array doc string implies that into-array will fall back to an Object array if aseq is supplied, but if the first element of aseq is nil, an NPE occurs.

user=> (doc into-array)
-------------------------
clojure.core/into-array
([aseq] [type aseq])
  Returns an array with components set to the values in aseq. The array's
  component type is type if provided, or the type of the first value in
  aseq if present, or Object. All values in aseq must be compatible with
  the component type. Class objects for the primitive types can be obtained
  using, e.g., Integer/TYPE.
user=> (into-array [nil 1 2])
NullPointerException   clojure.lang.RT.seqToTypedArray (RT.java:1691)

Approach: Check for nil and use Object as the array type.

Patch: CLJ-1644-array-first-nil-v2.patch

Screened by: Alex Miller



 Comments   
Comment by Michael Blume [ 15/Jan/15 1:45 PM ]

uploading patch v1 (adding nil as a cons-able element in CLJ-1643 will also bring this out, but I don't want to make the one patch depend on the other)

Comment by Phill Wolf [ 12/Mar/15 7:06 PM ]

Searching through the sequence for a non-null item is not consistent with into-array's docstring. The docstring says, "The array's component type is type if provided, or the type of the first value in aseq if present, or Object." In keeping with the docstring, wouldn't a null first item suggest an array of Object?

Working harder than that (by searching the sequence) only delays the inevitable: a whole sequence of nulls producing an Object array, instead of an array of the type the programmer expected, and triggering a run-time crash.

In summary: this patch goes farther than necessary, but even so, it does not cure the risk of unexpected results from nulls. A simpler remedy – returning an array of Object if the first item is null – would be consistent with the docstring and avoid raising unfounded expectations. Adding a statement that null is of type Object to the docstring could help programmers avoid falling into the trap.

Comment by Alex Miller [ 12/Mar/15 8:29 PM ]

No search through the sequence will pass screening, please just add a nil check.

Comment by Michael Blume [ 13/Mar/15 4:39 PM ]

done





[CLJ-1609] Fix an edge case in the Reflector's search for a public method declaration Created: 05/Dec/14  Updated: 24/Aug/15

Status: Open
Project: Clojure
Component/s: None
Affects Version/s: Release 1.6
Fix Version/s: Release 1.8

Type: Defect Priority: Major
Reporter: Jeremy Heiler Assignee: Unassigned
Resolution: Unresolved Votes: 1
Labels: interop

Attachments: Text File clj-1609-2.patch     Text File clj-1609-3.patch     Text File clj-1609-test.patch     Text File reflector_method_bug.patch    
Patch: Code and Test
Approval: Ok

 Description   
// all: package demo;

public interface IBar {
  void stuff();
}

class Bar {
  public void stuff() { System.out.println("stuff"); }
}

class SubBar extends Bar implements IBar {
  // implements IBar, via non-public superclass Bar
}

public class Factory {
  public static IBar get() { return new SubBar(); }
}

Can't invoke from Clojure:

(import '[demo Factory IBar])
(def inst (Factory/get))  ;; instance of SubBar, inferred as IBar
(.stuff inst)
;; IllegalArgumentException Can't call public method of non-public class: public void demo.Bar.stuff()  clojure.lang.Reflector.invokeMatchingMethod (Reflector.java:88)

Cause: The Reflector is not taking into account that a non-public class can implement an interface, and have a non-public parent class contain an implementation of a method on that interface.

Approach: The solution I took is to pass in the target object's class instead of the declaring class of the method object.

Patch: clj-1609-3.patch + clj-1609-test.patch

Screened by: Fogus



 Comments   
Comment by Alex Miller [ 05/Dec/14 8:48 AM ]

Probably a dupe of CLJ-259. I'll probably dupe that one to this though.

Comment by Jeremy Heiler [ 05/Dec/14 1:33 PM ]

Thanks. Sorry for not finding that myself. CLJ-259 refers to CLJ-126, which I think is covered with this patch.

Comment by Jeremy Heiler [ 06/Dec/14 12:37 PM ]

I was looking into whether or not the target could be null, and it can be when invoking a static method. However, I don't think that code path would make it to the target.getClass() because non-public static methods aren't returned by getMethods().

Comment by Stuart Halloway [ 31/Jul/15 9:24 AM ]

Jeremy,

The compile-tests phase of the build provides a point for compiling Java classes that that tests can then consume. Does that provide the hook you would need to add a test?

Comment by Alex Miller [ 07/Aug/15 9:10 AM ]

afaict, this ticket is incomplete pending a test from Stu's last comment so moving there

Comment by Alex Miller [ 18/Aug/15 4:58 PM ]

clj-1609-2.patch is identical to previous patch, just adds jira id to beginning of commit message

clj-1609-test.patch adds a reflection test that fails without the patch and passes with it.

Comment by Alex Miller [ 18/Aug/15 4:58 PM ]

should be ready to screen now

Comment by Fogus [ 21/Aug/15 12:53 PM ]

The patch and test are straight-forward and clean.

Comment by Rich Hickey [ 24/Aug/15 9:54 AM ]

Can I get some more context in this diff please?

Comment by Alex Miller [ 24/Aug/15 12:11 PM ]

Added clj-1609-3.patch which is identical to prior but has 15 lines of context instead.





[CLJ-1588] StackOverflow in clojure.test macroexpand with `are` and anonymous `fn` Created: 13/Nov/14  Updated: 17/Jul/15  Resolved: 17/Jul/15

Status: Closed
Project: Clojure
Component/s: None
Affects Version/s: Release 1.6
Fix Version/s: Release 1.8

Type: Defect Priority: Major
Reporter: Nikita Prokopov Assignee: Unassigned
Resolution: Completed Votes: 0
Labels: ft, template

Attachments: Text File clj-1588-2.patch     Text File clj-1588.patch     Text File clojure-test.recursion.patch    
Patch: Code and Test
Approval: Ok

 Description   

I noticed that this happens when an argument in anonymous `fn` is named the same as one of the binding in `are` form

(use 'clojure.test)
(deftest x
  (are [x y] (= x y)
    ((fn [x] (inc x)) 1) 2))
=>
clojure.lang.Compiler$CompilerException: java.lang.StackOverflowError, compiling:(/Users/nprokopov/Dropbox/ws/clojure.unicode/test/clojure/test_unicode.clj:54:3)

Approach: Replace prewalk-replace with postwalk-replace. prewalk-replace first replaces the form and then goes inside it looking for more replacements. postwalk-replace avoids that.

Patch: clj-1588-2.patch

Screened by: Alex Miller



 Comments   
Comment by Michael Blume [ 13/Nov/14 11:45 PM ]

yep, I bet it's trying to replace the x with (fn [x] (inc x)) and then replacing the x in that and...

this doesn't seem like that much of a defect? Like why write the code with the same variable names in the first place?

Comment by Nikita Prokopov [ 13/Nov/14 11:49 PM ]

Well, logically these are two completely separate, isolated Xes. It can be avoided, sure, but this behavior is not expected and I’m sure can be easily fixed.

Comment by Nikita Prokopov [ 13/Nov/14 11:53 PM ]

I mean, there shouldn’t be any recursion at all in the first place, right?

Comment by Nikita Prokopov [ 14/Nov/14 2:12 AM ]

Patch

Comment by Nikita Prokopov [ 14/Nov/14 2:13 AM ]

I fixed the issue by replacing prewalk with postwalk. It was caused by prewalk-replace that first replaced the form and then goes inside it looking for more replacements. Postwalk-replace avoids that.

Comment by Alex Miller [ 14/Nov/14 9:27 AM ]

1) Needs tests.
2) As a change in template, this affects many possible users. Can you assess other possible users either in core or in other external projects and whether they are affected? I have found http://crossclj.info to be helpful for questions like this.

Comment by Alex Miller [ 14/Nov/14 11:42 AM ]

Please include tests in a single combined patch and update the description to include a line specifying the current active patch for consideration.

Comment by Nikita Prokopov [ 14/Nov/14 11:46 AM ]

Alex, I attached a test case.

I also took a look at all use-cases of apply-template and do-template in both clojure.core and third-party projects. It’s not used very often, and in all cases use of do-template is pretty straightforward (take [x y z] and just replace it with some completely different forms), it does not depend on recursion and change from prewalk to postwalk will not cause any change in behavior. I think this patch is safe.

Comment by Nikita Prokopov [ 14/Nov/14 12:00 PM ]

I’m afraid I cannot edit description...

src/clj/clojure/template.clj => line 43
test/clojure/test_clojure/test.clj => lines 83-85

Comment by Alex Miller [ 14/Nov/14 10:14 PM ]

You should have edit rights now on jira issues.

Comment by Alex Miller [ 11/Apr/15 7:40 AM ]

Updated patch to proper format. The test in the patch does not pass:

[java] FAIL in (clj-1588-symbols-in-are-isolated-from-test-clauses) (:)
     [java]  but got :pass
     [java] expected: (= ((fn [x] (inc x)) 1) 2)
     [java]   actual: (#object[clojure.core$_EQ_ 0x7e2e5d92 "clojure.core$_EQ_@7e2e5d92"] 2 2)
Comment by Nikita Prokopov [ 12/Apr/15 6:14 AM ]

clojure.test-clojure.test now uses custom test reporter that relies on specific message. Updated patch to work around that





[CLJ-1586] Compiler doesn't preserve metadata for LazySeq literals Created: 12/Nov/14  Updated: 24/Aug/15

Status: Open
Project: Clojure
Component/s: None
Affects Version/s: None
Fix Version/s: Release 1.8

Type: Defect Priority: Major
Reporter: Nicola Mometto Assignee: Unassigned
Resolution: Unresolved Votes: 0
Labels: compiler, metadata, typehints

Attachments: Text File 0001-CLJ-1586-Compiler-doesn-t-preserve-metadata-for-lazy.patch     Text File 0001-Compiler-doesn-t-preserve-metadata-for-lazyseq-liter.patch     Text File clj-1586-2.patch    
Patch: Code and Test
Approval: Ok

 Description   

The analyzer in Compiler.java forces evaluation of lazyseq literals, but loses the compile time original metadata of that form, meaning that a type hint will be lost.

Example demonstrating this issue:

user=> (set! *warn-on-reflection* true)
true
user=> (list '.substring (with-meta (concat '(identity) '("foo")) {:tag 'String}) 0)
(.substring (identity "foo") 0)
user=> (eval (list '.substring (with-meta (concat '(identity) '("foo")) {:tag 'String}) 0))
Reflection warning, NO_SOURCE_PATH:6:1 - call to method substring on java.lang.Object can't be resolved (no such method).
"foo"

Forcing the concat call to an ASeq rather than a LazySeq fixes this issue:

user=> (eval (list '.hashCode (with-meta (seq (concat '(identity) '("foo"))) {:tag 'String})))
101574

This ticket blocks http://dev.clojure.org/jira/browse/CLJ-1444 since clojure.core/sequence might return a lazyseq.

This bug affected both tools.analyzer and tools.reader and forced me to commit a fix in tools.reader to work around this issue, see: http://dev.clojure.org/jira/browse/TANAL-99

The proposed patch trivially preserves the form metadata after realizing the lazyseq

Approach: Keep a copy of the original form, and apply its metadata to the realized lazyseq
Patch: clj-1586-2.patch
Screened by: Alex Miller



 Comments   
Comment by Stuart Halloway [ 31/Jul/15 9:19 AM ]

Please add a test case.

Comment by Nicola Mometto [ 31/Jul/15 9:32 AM ]

Updated patch with testcase

Comment by Alex Miller [ 11/Aug/15 10:55 AM ]

clj-1586-2.patch is same as prior, just updated to apply to current master





[CLJ-1565] pprint issues infinite output for a protocol Created: 15/Oct/14  Updated: 17/Jul/15  Resolved: 17/Jul/15

Status: Closed
Project: Clojure
Component/s: None
Affects Version/s: Release 1.6
Fix Version/s: Release 1.8

Type: Defect Priority: Major
Reporter: Michael Nygard Assignee: Unassigned
Resolution: Completed Votes: 5
Labels: ft, print, protocols

Attachments: Text File CLJ-1565.patch     File fix-pprint-var.diff    
Patch: Code
Approval: Ok

 Description   

Using pprint with a protocol name generates an unending stream of output. pprint appears to recurse through the Var reference as the value of the :var key in the protocol definition itself.

To reproduce:

user=> (defprotocol Foo (foo-you [this]))
Foo
user=> (prn Foo)
{:on user.Foo, :on-interface user.Foo, :sigs {:foo-you {:name foo-you, :arglists ([this]), :doc nil}}, :var #'user/Foo, :method-map {:foo-you :foo-you}, :method-builders {#'user/foo-you #object[user$eval1261$fn__1262 0x629ea441 "user$eval1261$fn__1262@629ea441"]}}
user=> (pprint Foo)
{:on user.Foo,
 :on-interface user.Foo,
 :sigs {:foo-you {:doc nil, :arglists ([this]), :name foo-you}},
 :var
 #<Var@6a3b02d8:
   {:on user.Foo,
    :on-interface user.Foo,
    :sigs {:foo-you {:doc nil, :arglists ([this]), :name foo-you}},
    :var
    #<Var@6a3b02d8:
      {:on user.Foo,
       :on-interface user.Foo,
       :sigs {:foo-you {:doc nil, :arglists ([this]), :name foo-you}},
       :var
       #<Var@6a3b02d8:
         {:on user.Foo,
          :on-interface user.Foo,
          :sigs
          {:foo-you {:doc nil, :arglists ([this]), :name foo-you}},
          :var
          #<Var@6a3b02d8:
            {:on user.Foo,
             :on-interface user.Foo,
             :sigs
             {:foo-you {:doc nil, :arglists ([this]), :name foo-you}},
             :var
;; forever

Approach: Add a dispatch method which doesn't deref the Var

user=> (defprotocol Foo (foo-you [this]))
Foo
user=> (prn Foo)
{:on user.Foo, :on-interface user.Foo, :sigs {:foo-you {:name foo-you, :arglists ([this]), :doc nil}}, :var #'user/Foo, :method-map {:foo-you :foo-you}, :method-builders {#'user/foo-you #object[user$eval10$fn__11 0x2452a793 "user$eval10$fn__11@2452a793"]}}
user=> (pprint Foo)
{:on user.Foo,
 :on-interface user.Foo,
 :sigs {:foo-you {:name foo-you, :arglists ([this]), :doc nil}},
 :var #'user/Foo,
 :method-map {:foo-you :foo-you},
 :method-builders
 {#'user/foo-you
  #object[user$eval10$fn__11 0x2452a793 "user$eval10$fn__11@2452a793"]}}
nil

Patch: CLJ-1565.patch
Screened by: Alex Miller



 Comments   
Comment by Andy Fingerhut [ 12/Nov/14 1:26 PM ]

I've run across this issue while debugging Eastwood. It probably does more than what you want in terms of modifying pprint behavior, but check out eastwood.util/pprint-meta here: https://github.com/jonase/eastwood/blob/master/src/eastwood/util.clj#L206

Comment by Daniel Marjenburgh [ 12/Nov/14 2:29 PM ]

The issue is that the simple-dispatch multifn dispatched a clojure.lang.Var to clojure.lang.IDeref, which dereferenced the Var before printing it. We have created a patch which dispatches a Var to the default print fn.

– With regards from the Amsterdam Clojure meetup group

Comment by Nicola Mometto [ 12/Nov/14 7:13 PM ]

The patch for this ticket also addressed http://dev.clojure.org/jira/browse/CLJ-1576

Comment by Alex Miller [ 29/Apr/15 11:28 AM ]

Patch needs a test.

Comment by Aspasia Beneti [ 01/May/15 6:49 AM ]

I added a test to the fix. Not sure if it is the right place for the test, any feedback is welcome.





[CLJ-1562] some->,some->>,cond->,cond->> and as-> doesn't work with (recur) Created: 11/Oct/14  Updated: 17/Jul/15  Resolved: 17/Jul/15

Status: Closed
Project: Clojure
Component/s: None
Affects Version/s: Release 1.6
Fix Version/s: Release 1.8

Type: Defect Priority: Critical
Reporter: Nahuel Greco Assignee: Unassigned
Resolution: Completed Votes: 0
Labels: ft

Attachments: Text File 1562-tests.patch     Text File fix-CLJ-1418_and_1562.patch    
Patch: Code and Test
Approval: Ok

 Description   

some-> and his friends doesn't work with recur, because they never place the last expression in tail position. For example:

(loop [l [1 2 3]] 
  (some-> l 
          next 
          recur))

raises UnsupportedOperationException: Can only recur from tail position

This is similar to the bug reported for as-> at http://dev.clojure.org/jira/browse/CLJ-1418 (see the comment at http://dev.clojure.org/jira/browse/CLJ-1418?focusedCommentId=35702&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-35702)

It can be fixed by changing the some-> definition to:

(defmacro some->
  "When expr is not nil, threads it into the first form (via ->),
  and when that result is not nil, through the next etc"
  {:added "1.5"}
  [expr & forms]
  (let [g (gensym)
        pstep (fn [step] `(if (nil? ~g) nil (-> ~g ~step)))]
    `(let [~g ~expr
           ~@(interleave (repeat g) (map pstep (butlast forms)))]
       ~(if forms
          (pstep (last forms))
          g))))

Similar fixes can be done for some->>, cond->, cond->> and as->.

Note -> supports recur without problems, fixing this will homogenize *-> macros behaviour.

Patch: fix-CLJ-1418_and_1562.patch (code) and 1562-tests.patch (tests)

Screened by: Alex Miller

Examples before/after:

1) (as-> 0 x (inc x))

;; before
(clojure.core/let [x 0 x (inc x)] x)

;; after
(clojure.core/let [x 0] (inc x))

2) (some-> 1 (- 2))

;; before
(clojure.core/let
 [G__5 1
  G__5 (if (clojure.core/nil? G__5) nil (clojure.core/-> G__5 (- 2)))]
 G__5)

;; after
(clojure.core/let
 [G__21 1]
 (if (clojure.core/nil? G__21) nil (clojure.core/-> G__21 (- 2))))

3) (some->> 1 (- 2))

;; before
(clojure.core/let
 [G__8 1
  G__8 (if (clojure.core/nil? G__8) nil (clojure.core/->> G__8 (- 2)))]
 G__8)

;; after
(clojure.core/let
 [G__24 1]
 (if (clojure.core/nil? G__24) nil (clojure.core/->> G__24 (- 2))))

4) (cond-> 0 true inc true (- 2))

;; before
(clojure.core/let
 [G__14 0
  G__14 (if true (clojure.core/-> G__14 inc) G__14)
  G__14 (if true (clojure.core/-> G__14 (- 2)) G__14)]
 G__14)

;; after
(clojure.core/let
 [G__30 0 
  G__30 (if true (clojure.core/-> G__30 inc) G__30)]
 (if true (clojure.core/-> G__30 (- 2)) G__30))

5) (cond->> 0 true inc true (- 2))

;; before
(clojure.core/let
 [G__11 0
  G__11 (if true (clojure.core/->> G__11 inc) G__11)
  G__11 (if true (clojure.core/->> G__11 (- 2)) G__11)]
 G__11)

;; after
(clojure.core/let
 [G__27 0 
  G__27 (if true (clojure.core/->> G__27 inc) G__27)]
 (if true (clojure.core/->> G__27 (- 2)) G__27))


 Comments   
Comment by Alex Miller [ 29/Apr/15 11:15 AM ]

Would be great if there was a patch here to consider that covered the set of affected macros.

Comment by Nahuel Greco [ 03/May/15 12:39 PM ]

Attached a patch from https://github.com/nahuel/clojure/compare/fix-CLJ-1418/1562 . This patch also fixes CLJ-1418

Comment by Alex Miller [ 04/May/15 11:17 AM ]

This patch looks like a great start - I will need to look at it further. One thing I just noticed is that none of these macros has any tests. I would love to take this opportunity to rectify that by adding tests for all of them.

Comment by Rich Hickey [ 14/Jul/15 12:44 PM ]

It would be helpful to see before and after macroexpansions

Comment by Alex Miller [ 14/Jul/15 1:58 PM ]

Added example expansions before/after





[CLJ-1533] Oddity in type tag usage for primInvoke Created: 24/Sep/14  Updated: 17/Jul/15  Resolved: 17/Jul/15

Status: Closed
Project: Clojure
Component/s: None
Affects Version/s: Release 1.6
Fix Version/s: Release 1.8

Type: Defect Priority: Critical
Reporter: Andy Fingerhut Assignee: Unassigned
Resolution: Completed Votes: 0
Labels: ft, typehints

Attachments: Text File 0001-CLJ-1533-inject-original-var-form-meta-in-constructe.patch     Text File clj-1533-2.patch    
Patch: Code and Test
Approval: Ok

 Description   

Some odd behavior demonstrated in Clojure 1.6.0 REPL below. Why does the (Math/abs (f2 -3)) call issue a reflection warning? It seems like perhaps it should not, given the other examples.

user=> (clojure-version)
"1.6.0"
user=> (set! *warn-on-reflection* true)
true
user=> (defn ^{:tag 'long} f1 [x] (inc x))
#'user/f1
user=> (Math/abs (f1 -3))
2
user=> (defn ^{:tag 'long} f2 [^long x] (inc x))
#'user/f2
user=> (Math/abs (f2 -3))
Reflection warning, NO_SOURCE_PATH:6:1 - call to static method abs on java.lang.Math can't be resolved (argument types: java.lang.Object).
2
user=> (defn ^{:tag 'long} f3 ^long [^long x] (inc x))
#'user/f3
user=> (Math/abs (f3 -3))
2

Cause: invokePrim path does not take into account var or form meta

Approach: apply var and form meta to invokePrim expression

Patch: clj-1533-2.patch

Screened by: Alex Miller



 Comments   
Comment by Nicola Mometto [ 25/Sep/14 9:47 AM ]

The issue is similar to http://dev.clojure.org/jira/browse/CLJ-1491

Comment by Nicola Mometto [ 25/Sep/14 9:58 AM ]

The root cause was also almost the same, the proposed patch is a superset of the one proposed for CLJ-1491

Comment by Alex Miller [ 25/Sep/14 10:09 AM ]

Can we include 1491 cases in this ticket and mark 1491 a duplicate?

Comment by Alex Miller [ 25/Sep/14 10:09 AM ]

Also needs tests in the patch.

Comment by Nicola Mometto [ 25/Sep/14 10:23 AM ]

Updated the patch with testcases for both issues, I agree that CLJ-1491 should be closed as duplicate

Comment by Alex Miller [ 04/May/15 8:52 AM ]

New patch is identical, just refreshed to apply to master





[CLJ-1528] clojure.test/inc-report-counter is not thread safe Created: 19/Sep/14  Updated: 17/Jul/15  Resolved: 17/Jul/15

Status: Closed
Project: Clojure
Component/s: None
Affects Version/s: Release 1.7
Fix Version/s: Release 1.8

Type: Defect Priority: Critical
Reporter: Alexander Redington Assignee: Alexander Redington
Resolution: Completed Votes: 1
Labels: ft, test
Environment:

OS X, Clojure 1.7, Macbook pro


Attachments: File fix-CLJ-1528.diff    
Patch: Code
Approval: Ok

 Description   

clojure.test/inc-report-counter function combines dereferencing the report-counters ref and operating on the previous state of the ref, leading to race conditions during concurrent use.

(dosync (commute *report-counters* assoc name
                 (inc (or (*report-counters* name) 0))))

Approach: Rewrite update function to be entirely in terms of the old state:

(dosync (commute *report-counters* update-in [name] (fnil inc 0)))

Patch: fix-CLJ-1528.diff
Screened by: Alex Miller



 Comments   
Comment by Alexander Redington [ 19/Sep/14 10:58 AM ]

Fixes 1528





[CLJ-1485] clojure.test.junit/with-junit-output doesn't handle multiple expressions Created: 29/Jul/14  Updated: 31/Jul/15  Resolved: 31/Jul/15

Status: Closed
Project: Clojure
Component/s: None
Affects Version/s: Release 1.6
Fix Version/s: Release 1.8

Type: Defect Priority: Minor
Reporter: Howard Lewis Ship Assignee: Unassigned
Resolution: Completed Votes: 0
Labels: clojure.test, ft

Attachments: Text File clj-1485.patch    
Patch: Code
Approval: Ok

 Description   

From the docstring description, and the use of ~@body, the intent of with-junit-output was to support a body containing multiple forms (for side-effects). However, calling it with multiple expressions will yield an error about the bindings in the let form.

(defmacro with-junit-output
  "Execute body with modified test-is reporting functions that write
  JUnit-compatible XML output."
  {:added "1.1"}
  [& body]
  `(binding [t/report junit-report
             *var-context* (list)
             *depth* 1]
     (t/with-test-out
       (println "<?xml version=\"1.0\" encoding=\"UTF-8\"?>")
       (println "<testsuites>"))
     (let [result# ~@body]
       (t/with-test-out (println "</testsuites>"))
       result#)))

Cause: The ~@body in the macro is spliced into code expecting a single expression.

Approach: Wrap a (do ) around the ~@body.

Patch: clj-1485.patch

Screened by: Alex Miller



 Comments   
Comment by Howard Lewis Ship [ 29/Jul/14 4:59 PM ]

Patch for issue





[CLJ-1449] Add clojure.string functions for portability to ClojureScript Created: 19/Jun/14  Updated: 30/Aug/15

Status: Open
Project: Clojure
Component/s: None
Affects Version/s: Release 1.6
Fix Version/s: Release 1.8

Type: Enhancement Priority: Major
Reporter: Bozhidar Batsov Assignee: Nola Stowe
Resolution: Unresolved Votes: 29
Labels: ft, string

Attachments: Text File add_functions_to_strings-2.patch     Text File add_functions_to_strings-3.patch     Text File add_functions_to_strings-4.patch     Text File add_functions_to_strings-5.patch     Text File add_functions_to_strings-6.patch     Text File add_functions_to_strings-7.patch     Text File add_functions_to_strings.patch     Text File clj-1449-more-v1.patch    
Patch: Code and Test
Approval: Ok

 Description   

It would be useful if a few common functions from Java's String were available as Clojure functions for the purposes of increasing portability to other Clojure platforms like ClojureScript.

The functions below also cover the vast majority of cases where Clojure users currently drop into Java interop for String calls - this tends to be an issue for discoverability and learning. While the goal of this ticket is increased portability, improving that is a nice secondary benefit.

Proposed clojure.string fn java.lang.String method
index-of indexOf
last-index-of lastIndexOf
starts-with? startsWith
ends-with? endsWith
includes? contains

Patch:

  • clj-1449-7.patch - uses nil to indicate not-found in index-of

Performance: Tested the following with criterium for execution mean time (all times in ns).

Java Clojure Java Clojure (-4 patch) Clojure (-6 patch)
(.indexOf "banana" "n") (clojure.string/index-of "banana" "n") 8.70 9.03 9.27
(.indexOf "banana" "n" 1) (clojure.string/index-of "banana" "n" 1) 7.29 7.61 7.66
(.indexOf "banana" (int \n)) (clojure.string/index-of "banana" \n) 5.34 6.20 6.20
(.indexOf "banana" (int \n) 1) (clojure.string/index-of "banana" \n 1) 5.38 6.19 6.24
(.startsWith "apple" "a") (clojure.string/starts-with? "apple" "a") 4.84 5.71 5.65
(.endsWith "apple" "e") (clojure.string/ends-with? "apple" "e") 4.82 5.68 5.70
(.contains "baked apple pie" "apple") (clojure.string/includes? "baked apple pie" "apple") 10.78 11.99 12.17

Screened by: Stu, who prefers nil over -1 - add_functions_to_strings-6.patch

Note: In both Java and JavaScript, indexOf will return -1 to indicate not-found, so this is (at least in these two cases) the status quo. The benefit of nil return in Clojure is that it can be used as a found/not-found condition. The benefit of a -1 return is that it matches Java/JavaScript and that the return type can be hinted as a primitive long, allowing you to feed it into some other arithmetic or string operation without boxing.



 Comments   
Comment by Alex Miller [ 19/Jun/14 12:53 PM ]

Re substring, there is a clojure.core/subs for this (predates the string ns I believe).

clojure.core/subs
([s start] [s start end])
Returns the substring of s beginning at start inclusive, and ending
at end (defaults to length of string), exclusive.

Comment by Jozef Wagner [ 20/Jun/14 3:21 AM ]

As strings are collection of characters, you can use Clojure's sequence facilities to achieve such functionality:

user=> (= (first "asdf") \a)
true
user=> (= (last "asdf") \a)
false
Comment by Alex Miller [ 20/Jun/14 8:33 AM ]

Jozef, String.startsWith() checks for a prefix string, not just a prefix char.

Comment by Bozhidar Batsov [ 20/Jun/14 9:42 AM ]

Re substring, I know about subs, but it seems very odd that it's not in the string ns. After all most people will likely look for string-related functionality in clojure.string. I think it'd be best if `subs` was added to clojure.string and clojure.core/subs was deprecated.

Comment by Pierre Masci [ 01/Aug/14 5:27 AM ]

Hi, I was thinking the same about starts-with and .ends-with, as well as (.indexOf s "c") and (.lastIndexOf "c").

I read the whole Java String API recently, and these 4 functions seem to be the only ones that don't have an equivalent in Clojure.
It would be nice to have them.

Andy Fingerhut who maintains the Clojure Cheatsheet told me: "I maintain the cheatsheet, and I put .indexOf and .lastIndexOf on there since they are probably the most common thing I saw asked about that is in the Java API but not the Clojure API, for strings."
Which shows that there is a demand.

Because Clojure is being hosted on several platforms, and might be hosted on more in the future, I think these functions should be part of the de-facto ecosystem.

Comment by Andy Fingerhut [ 30/Aug/14 3:39 PM ]

Updating summary line and description to add contains? as well. I can back this off if it changes your mind about triaging it, Alex.

Comment by Andy Fingerhut [ 30/Aug/14 3:40 PM ]

Patch clj-1449-basic-v1.patch dated Aug 30 2014 adds starts-with? ends-with? contains? functions to clojure.string.

Patch clj-1449-more-v1.patch is the same, except it also replaces several Java method calls with calls to these Clojure functions.

Comment by Andy Fingerhut [ 05/Sep/14 1:02 PM ]

Patch clj-1449-basic-v1.patch dated Sep 5 2014 is identical to the patch I added recently called clj-1149-basic-v1.patch. It is simply renamed without the typo'd ticket number in the file name.

Comment by Yehonathan Sharvit [ 02/Dec/14 3:09 PM ]

What about an implementation that works also in cljs?

Comment by Bozhidar Batsov [ 02/Dec/14 3:11 PM ]

Once this is added to Clojure it will be implemented in ClojureScript as well.

Comment by Yehonathan Sharvit [ 02/Dec/14 3:22 PM ]

Great! Any idea when it will be added to Clojure?
Also, will it be automatically added to Clojurescript or someone will have to write a particular code for it.
The suggested patch relies on Java so I am curious to understand who is going to port the patch to cljs.

Comment by Bozhidar Batsov [ 02/Dec/14 3:27 PM ]

No idea when/if this will get merged. Upvote the ticket to improve the odds of this happening sooner.
Someone on the ClojureScript team will have to implement this in terms of JavaScript.

Comment by Alex Miller [ 02/Dec/14 4:01 PM ]

Some things that would be helpful:

1) It would be better to combine the two patches into a single patch - I think changing current uses into new users is a good thing to include. Also, please keep track of the "current" patch in the description.
2) Patch needs tests.
3) Per the instructions at the top of the clojure.string ns (and the rest of the functions), the majority of these functions are implemented to take the broader CharSequence interface. Similar to those implementations, you will need to provide a CharSequence implementation while also calling into the String functions when you actually have a String.
4) Consider return type hints - I'm not sure they're necessary here, but I would examine bytecode for typical calling situations to see if it would be helpful.
5) Check performance implications of the new versions vs the old with a good tool (like criterium). You've put an additional var invocation and (soon) type check in the calling path for these functions. I think providing a portable target is worth a small cost, but it would be good to know what the cost actually is.

I don't expect we will look at this until after 1.7 is released.

Comment by Andy Fingerhut [ 02/Dec/14 8:05 PM ]

Alex, all your comments make sense.

If you think a ready-and-waiting patch that does those things would improve the odds of the ticket being vetted by Rich, please let us know.

My guess is that his decision will be based upon the description, not any proposed patches. If that is your belief also, I'll wait until he makes that decision before working on a patch. Of course, any other contributor is welcome to work on one if they like.

Comment by Alex Miller [ 02/Dec/14 8:40 PM ]

Well nothing is certain of course, but I keep a special report of things I've "screened" prior to vetting that makes possible moving something straight from Triaged all the way through into Screened/Ok when Rich is able to look at them. This is a good candidate if things were in pristine condition.

That said, I don't know whether Rich will approve it or not, so it's up to you. I think the argument for portability is a strong one and complements the feature expression.

Comment by Alex Miller [ 14/May/15 8:55 AM ]

I'd love to have a really high-quality patch on this ticket to consider for 1.8 that took into account my comments above.

Also, it occurred to me that I don't think this should be called "contains?", overlapping the core function with a different meaning (contains value vs contains key). Maybe "includes?"?

Comment by Andy Fingerhut [ 14/May/15 9:14 AM ]

clojure.string already has 2 name conflicts with clojure.core, for which most people probably do something like (require '[clojure.string :as str]) to avoid that:

user=> (use 'clojure.string)
WARNING: reverse already refers to: #'clojure.core/reverse in namespace: user, being replaced by: #'clojure.string/reverse
WARNING: replace already refers to: #'clojure.core/replace in namespace: user, being replaced by: #'clojure.string/replace
nil
Comment by Alex Miller [ 14/May/15 10:05 AM ]

I'm not concerned about overlapping the name. I'm concerned about overlapping it and meaning something different, particularly vs one of the most confusing functions in core already. clojure.core/contains? is better than linear time key search. clojure.string/whatever will be a linear time subsequence match.

Comment by Alex Miller [ 14/May/15 10:18 AM ]

Ruby uses "include?" for this.

Comment by Devin Walters [ 19/May/15 4:56 PM ]

I agree with Alex's comment about the overlap. Personally, I prefer the way "includes?" reads over "include?", but IMO either one is better than adding to the "contains?" confusion.

Comment by Bozhidar Batsov [ 20/May/15 12:10 AM ]

I'm fine with `includes?`. Ruby is famous for the bad English used in its core library.

Comment by Andrew Rosa [ 17/Jun/15 12:29 PM ]

Andy Fingerhut, since you are the author of the original patch do you desire to continue the work on it? If you prefer (or even need) to focus on different stuff, I would like to tackle this issue.

Comment by Andy Fingerhut [ 17/Jun/15 1:08 PM ]

Andrew Rosa, I am perfectly happy if someone else wants to work on a revised patch for this. If you are interested, go for it! And from Alex's comments, I believe my initial patch covers such a small fraction of what he wants, do not worry about keeping my name associated with any patch you submit – if yours is accepted, my patch will not have helped you in getting there.

Comment by Nola Stowe [ 03/Jul/15 9:01 AM ]

Andrew Rosa, I am interested on working on this. Are you currently working it?

Comment by Andrew Rosa [ 03/Jul/15 12:08 PM ]

Thanks Andy Fingerhut, didn't saw the answer here. I'm happy that Nola Stowe could take this one.

Comment by Nola Stowe [ 08/Jul/15 12:46 PM ]

Updated patch to rename methods as specified and add tests.

Comment by Bozhidar Batsov [ 08/Jul/15 3:26 PM ]

A few remarks:

1. added should say 1.8, not 1.9.
2. The docstrings could be improved a bit - normally they should end with a . and refer to all the params of the function in question.

Comment by Nola Stowe [ 08/Jul/15 3:29 PM ]

Thanks Bozhidar Batsov .. I will, and I also got feedback from the slack clojure-dev group (Alex, Ghadi, AndrewHr) so I will be submitting an improved patch soon

Comment by Andy Fingerhut [ 09/Jul/15 11:57 AM ]

Also, I am not sure, but Alex Miller made this comment earlier: "Per the instructions at the top of the clojure.string ns, functions should take the broader CharSequence interface instead of String. Similar to existing clojure.string functions, you will need to provide a CharSequence implementation while also calling into the String functions when you actually have a String."

Looking at the Java classes and interfaces, one of the classes that implements CharSequence is CharBuffer, and it has no indexOf or lastIndexOf methods. If one were to call the proposed index-of or last-index-of with a CharBuffer, I believe you would get an exception for an unknown method.

I don't know if Alex's sentence is essential to any patch to be considered, but wanted to point out that it seems like it effectively means writing your own implementation of each of these methods for some of the classes implementing the CharSequence interface.

Comment by Nola Stowe [ 09/Jul/15 4:15 PM ]

Andy, yes you are right. Alex helped me with my current changes (not posted yet) and where needed I cast the value to a string so that it would have the indexOf methods.

Comment by Nola Stowe [ 10/Jul/15 7:43 AM ]

Added type hints, using StringBuffer instead of string in tests.

Comment by Nola Stowe [ 10/Jul/15 11:53 PM ]

Use critium and did some bench marking, I am not familiar enough to say "hey its good or no, too slow". I tried an optimization but it didn't make much difference.

notes here:

https://www.evernote.com/l/ABLYZbDr8ElIs6fkODAd3poBfIm_Bfr5kLo

Appreciate any feedback

Comment by Alex Miller [ 13/Jul/15 10:37 AM ]

Nola, some comments:

1) In index-of and last-index-of, I think we should use (instance? Character value) instead of char? - this will avoid a function invocation.

2) In index-of and last-index-of, in the branches where you know you have a Character, we should just invoke the proper function on the character instance directly rather than calling int. In this case that is (.charValue ^Character value). However, this function returns char so you still want the ^int hint.

3) In index-of and last-index-of, we should do a couple of things to maximally utilize primitive support for from-index. We should type-hint the incoming value as a ^long (only long and double are specialized cases in Clojure, not int). We should then use the function unchecked-int to optimally convert the long primitive to an int primitive.

4) We should type-hint the return value to a primitive long to avoid boxing the return.

Putting 1+2+3+4 together:

(defn index-of
  "Return index of value (string or char) in s, optionally searching forward from from-index."
  {:added "1.8"}
  (^long [^CharSequence s value]
   (if (instance? Character value)
     (.indexOf (.toString s) ^int (.charValue ^Character value))
     (.indexOf (.toString s) ^String value)))
  (^long [^CharSequence s value ^long from-index]
  (if (instance? Character value)
    (.indexOf (.toString s) ^int (.charValue ^Character value) (unchecked-int from-index))
    (.indexOf (.toString s) ^String value (unchecked-int from-index)))))

Similar changes in last-index-of.

5) In starts-with? and ends-with?, I would move the type hint for substr to the parameter declaration. This is more of a style preference on my part, no functional difference I believe.

6) On includes? I would do the same as #5, except .contains actually takes a CharSequence for the substr, so I would change the type hint from ^String to the broader ^CharSequence.

7) Can you edit the description and add a table with the "execution time mean" number for direct Java vs new Clojure fn for the patch after these changes? Please add tests for the from-index variants of index-of and last-index-of as well.

Comment by Nola Stowe [ 14/Jul/15 5:46 PM ]

optimizations as suggested by Alex Miller

Comment by Andy Fingerhut [ 14/Jul/15 5:50 PM ]

Nola, not a big deal, but Rich has requested that patch file names end with ".patch" or ".diff", I think because whatever he uses to view them recognizes that suffix and puts it in a different viewing/editing mode.

Comment by Nola Stowe [ 14/Jul/15 6:03 PM ]

Andy, so no need to keep around patches when used in comparisons?

Comment by Andy Fingerhut [ 14/Jul/15 8:31 PM ]

It is ok to have multiple versions of patches attached, but names like add_functions_to_strings-2.patch etc. would be preferred over names like add_functions_to_strings.patch-2, so the suffix is ".patch".

Comment by Nola Stowe [ 14/Jul/15 8:55 PM ]

original patch submitted by Nola

Comment by Nola Stowe [ 14/Jul/15 8:56 PM ]

Optimizations suggested by Alex

Comment by Alex Miller [ 14/Jul/15 9:44 PM ]

Nola, I re-did the timings on my machine - these were done with Java 1.8 and no Leiningen involved.

Comment by Alex Miller [ 14/Jul/15 9:57 PM ]

-4 patch is identical to -3 but removes one errant ^long type hint

Otherwise, looks good to me!

Comment by Alex Miller [ 17/Jul/15 10:24 PM ]

Rich, since you didn't put any comments on here, I'm not sure if there's anything else you wanted, so marking screened.

Comment by Stuart Halloway [ 19/Jul/15 7:36 AM ]

I dislike the approach here for several reasons:

  • floats Java-isms (-1 for missing instead of nil?) up into the Clojure API
  • makes narrow string fns out of things that could/should be seq fns

Stepping back, this is all library work. Why should it be in core? If this started in a contrib, it could evolve much more freely.

Comment by Alex Miller [ 19/Jul/15 8:49 AM ]

re Java-isms: totally fair comment worth changing

re narrow string fns: if these were seq fns, they would presumably take seqables of chars and return seqables of chars. When I have strings, I want to work with strings (as with clojure.string/join and clojure.string/reverse vs their seq equivalents).

re library: these particular functions are the most commonly used string functions where current users drop to Java interop. Note that all 5 of these functions have over time been added to the cheatsheet (http://clojure.org/cheatsheet) because they are commonly used. I believe there is significant value in including them in core and standardizing their name and behavior across to CLJS.

Comment by Bozhidar Batsov [ 19/Jul/15 10:29 AM ]

I'm obviously biased, but I totally agree with everything Alex said. Having one core string library and a separate contrib string library is just impractical. With clojure.string already part of the language it's much more sensible to extend it where/when needed instead of encouraging the proliferation of tons of alternative libraries (there are already a couple of those out there, mostly because essential functions are missing). Initial designs are rarely perfect, there's nothing wrong in revisiting and improving them.

I believe this is one of the top voted tickets, which clearly indicates that the Clojure community is quite supportive of this additions.

Comment by Michael Blume [ 19/Jul/15 12:26 PM ]

None of these functions return Strings so they could very easily be seq functions which use the String methods as a fast path when called with Strings, the trouble is then they wouldn't necessarily belong in clojure.string.

Comment by Bozhidar Batsov [ 19/Jul/15 1:49 PM ]

Such an argument can be made for existing functions like clojure.core/subs and clojure.string/reverse - we could have just went with generic functions that operate on seqs and converted the results to strings, but we didn't. It might be just me, but I'm really thinking this is being overanalysed. Sometimes it's really preferable to deal with some things as strings (not to mention more efficient).

Comment by Rich Hickey [ 20/Jul/15 9:39 AM ]

I completely agree with Alex and Bozhidar re: strings vs seqs. Other langs that have only lists and use them for strings are really bad at strings. So - string-specific fns are ok. What about the Java-isms?

Comment by Bozhidar Batsov [ 20/Jul/15 10:08 AM ]

We should address them for sure. Let's wait for a couple of days for Nola to update the patch. If she's too busy I'll do the update myself.

Comment by Nola Stowe [ 20/Jul/15 10:50 AM ]

I will be happy to work on it Saturday. Is nil an acceptable response when index-of "apple" "p" is not found? The other functions return booleans and I think those are ok.

Comment by Bozhidar Batsov [ 20/Jul/15 10:55 AM ]

Yep, we should go with nil.

Comment by Nola Stowe [ 25/Jul/15 4:33 PM ]

Changed index-of and last-index-of to return nil instead of -1 (java-ism), thus had to remove the type hint for the return value.

Seems to have slowed down the function compared to having the function return -1. My benchmarks: https://www.evernote.com/l/ABL2wead73BJZYAkpH5yxsfX9JM8cpUiNhw

Comment by Nola Stowe [ 25/Jul/15 9:05 PM ]

Updated patch to include return value in doc string for index-of last-index-of, otherwise same as last update.

Comment by Alex Miller [ 26/Jul/15 8:40 AM ]

Removing the hint causes the return type to be Object, which forces boxing in the "found" cases.

Comment by Nola Stowe [ 26/Jul/15 8:56 AM ]

Is it worth it to remove the java-ism of returning -1 when not found? Personally, I don't really like a function returning one of two things... found index or nil ... I prefer it being an int at all times.

Comment by Alex Miller [ 26/Jul/15 9:00 AM ]

There is one whitespace issue in the -5 patch:

[~/code/clojure (master)]$ git apply ~/Downloads/add_functions_to_strings-5.patch
/Users/alex/Downloads/add_functions_to_strings-5.patch:34: trailing whitespace.
  (let [result
warning: 1 line adds whitespace errors.

I updated the timings in the description to include both -4 and -5 - the boxing there does have a significant impact.

Comment by Nola Stowe [ 26/Jul/15 9:28 AM ]

Fixed whitespace issue.

Comment by Alex Miller [ 26/Jul/15 9:32 AM ]

Added -6 patch and timings that address most of the performance impact.

Comment by Stuart Halloway [ 27/Jul/15 9:35 AM ]

Alex: Note that all of these functions return numbers and booleans, not strings nor seqs of anything. This is partially why they were left out of the original string implementation.

It isn't clear to me why index-of should be purely a string fn, I regularly need index-of with other things, and both my book and JoC use index-of as an example of a nice-to-have.

Comment by Nicola Mometto [ 27/Jul/15 9:44 AM ]

Stuart, I don't think the fact that those functions don't return string is really relevant, the important fact is that they operate on strings (just like c.set/superset? or subset? don't return sets) and are frequently used on them.

Comment by Bozhidar Batsov [ 27/Jul/15 9:51 AM ]

I'm with Nicola on this one - we have to be practical. There's a reason why this is the top-voted JIRA ticket - Clojure programmers want those functions. That's says plenty.

Anyways, having a generic `index-of` is definitely not a bad idea.

Comment by Stuart Halloway [ 31/Jul/15 11:27 AM ]

How does this compare to making them all seq fns and adding them to clojure.core?

Comment by Andy Fingerhut [ 31/Jul/15 11:39 AM ]

Given Rich's comment earlier "I completely agree with Alex and Bozhidar re: strings vs seqs. Other langs that have only lists and use them for strings are really bad at strings. So - string-specific fns are ok.", do we need to write a separate patch that makes these all into seq fns and do performance comparisons between those vs. the string-specific ones?

Comment by Stuart Halloway [ 31/Jul/15 1:26 PM ]

Andy,

Definitely not! I want to have a design conversation.

We have protocols, and could use them to provide core fns that are specialized to strings, so it isn't clear to me why we would have to be really bad at it.

Comment by Stuart Halloway [ 07/Aug/15 2:19 PM ]

Discussed with Rich. He does not want sequence fns with these performance characteristics, so string-specific fns it shall be.

Comment by Rich Hickey [ 08/Aug/15 10:11 AM ]

So which patch is screened?

Comment by Alex Miller [ 08/Aug/15 11:40 AM ]

Stu screened add_functions_to_strings-6.patch where index-of returns nil on not-found. The add_functions_to_strings-4.patch is almost identical but returns -1 from index-of (which matches Java and JavaScript index-of behavior and may be slightly faster for callers that can be monomorphic on primitive longs?).

Comment by Rich Hickey [ 24/Aug/15 11:27 AM ]

nil-returning index fns should document that return when not found

Comment by Nola Stowe [ 24/Aug/15 11:56 AM ]

adds better doc string to indicated nil is return when not found for index-of and last-index-of.

Comment by Alex Miller [ 24/Aug/15 11:59 AM ]

Added clj-1449-7.patch that is identical to -6 but adds statement of return expectation on index-of and last-index-of.

Comment by Alex Miller [ 24/Aug/15 12:19 PM ]

jinx - same patch! Replacing with Nola's.

Comment by Colin Taylor [ 30/Aug/15 6:48 PM ]

Bit late on this - but .equalsIgnoreCase (=ic? equals-ic ? ) is currently a visible wart for our portability.

Comment by Alex Miller [ 30/Aug/15 10:03 PM ]

Hi Colin, that's a good idea but you have missed the window on this ticket - feel free to file a different one.





[CLJ-1399] missing field munging when recreating deftypes serialized into byte code Created: 02/Apr/14  Updated: 17/Jul/15  Resolved: 17/Jul/15

Status: Closed
Project: Clojure
Component/s: None
Affects Version/s: Release 1.7
Fix Version/s: Release 1.8

Type: Defect Priority: Critical
Reporter: Kevin Downey Assignee: Unassigned
Resolution: Completed Votes: 2
Labels: compiler, deftype, ft

Attachments: File clj-1399.diff     File clj-1399-with-test.diff    
Patch: Code and Test
Approval: Ok

 Description   

deftypes with fields whose names get munged fail when constructed in data reader functions.

user=> (deftype Foo [hello-world])
user.Foo
user=> (alter-var-root #'default-data-readers assoc 'foo (fn [x] (->Foo x)))
{inst #'clojure.instant/read-instant-date, uuid #'clojure.uuid/default-uuid-reader, foo #object[user$eval12$fn__13 0x23c89df9 "user$eval12$fn__13@23c89df9"]}
user=> #foo "1"
CompilerException java.lang.IllegalArgumentException: No matching field found: hello-world for class user.Foo, compiling:(NO_SOURCE_PATH:0:0)

Cause: To embed deftypes in the bytecode the compiler emits the value of each field, then emits a call to the deftypes underlying class's constructor. To get a list of fields the compiler calls .getBasis. The getBasis fields are the "clojure" level field names of the deftype, which the actual "jvm" level field names have been munged (replacing - with _, etc), so the compiler tries to generate code to set values on non-existent fields.

Approach: Munge the field name before emitting it in bytecode.
Patch: clj-1399-with-test.diff
Screened by: Alex Miller



 Comments   
Comment by Kevin Downey [ 02/Apr/14 4:26 PM ]

reproducing case

$ rlwrap java -server -Xmx1G -Xms1G -jar /Users/hiredman/src/clojure/target/clojure-1.6.0-master-SNAPSHOT.jar
Clojure 1.6.0-master-SNAPSHOT
user=> (deftype Foo [hello-world])
user.Foo
user=> (alter-var-root #'default-data-readers assoc 'foo (fn [x] (Foo. x)))
{foo #<user$eval6$fn__7 user$eval6$fn__7@2f953efd>, inst #'clojure.instant/read-instant-date, uuid #'clojure.uuid/default-uuid-reader}
user=> #foo "1"
CompilerException java.lang.IllegalArgumentException: No matching field found: hello-world for class user.Foo, compiling:(NO_SOURCE_PATH:0:0)
user=>
Comment by Kevin Downey [ 02/Apr/14 4:39 PM ]

this patch fixes the issue on the latest master for me

Comment by Chas Emerick [ 02/Apr/14 4:57 PM ]

FWIW, this was precipitated by real experience (I think I created the refheap paste). The workaround is easy (don't use dashes in field names of deftypes you want to return from data reader functions), but I wouldn't expect anyone to guess that that wasn't already oversensitized to munging edge cases.

Comment by Alex Miller [ 29/Apr/15 11:36 AM ]

Could the patch have a test?

Comment by Kevin Downey [ 29/Apr/15 1:20 PM ]

clj-1399-with-test.diff adds a test





[CLJ-1390] pprint a GregorianCalendar results in Arity exception Created: 25/Mar/14  Updated: 31/Jul/15  Resolved: 31/Jul/15

Status: Closed
Project: Clojure
Component/s: None
Affects Version/s: None
Fix Version/s: Release 1.8

Type: Defect Priority: Minor
Reporter: Steve Suehs Assignee: Unassigned
Resolution: Completed Votes: 1
Labels: ft, print

Attachments: Text File 0001-CLJ-1390.patch     Text File 0002-CLJ-1390-test2.patch     Text File 0002-CLJ-1390-test.patch     Text File CLJ-1390-pprint-GregorianCalendar.patch    
Patch: Code and Test
Approval: Ok

 Description   

What I was doing: attempting to pretty-print nested structures from Things mac app, which include instances of java.util.GregorianCalendar.
What I expected to happen: output should have an #inst very much like printing java.util.Date.
What happened instead: ArityException Wrong number of args (4) passed to: pprint$pretty-writer$fn

thingsplay.core=> (def nowish (java.util.GregorianCalendar.))
#'thingsplay.core/nowish
thingsplay.core=> nowish
#inst "2014-03-25T22:43:29.240-05:00"
thingsplay.core=> (use 'clojure.pprint)
nil
thingsplay.core=> (pprint nowish)
ArityException Wrong number of args (4) passed to: pprint$pretty-writer$fn  clojure.lang.AFn.throwArity (AFn.java:437)
#inst "
thingsplay.core=> (simple-dispatch nowish)
#inst "2014-03-25T22:43:29.240-05:00"nil
thingsplay.core=> nowish
#inst "2014-03-25T22:43:29.240-05:00"
thingsplay.core=> (write nowish)
ArityException Wrong number of args (4) passed to: pprint$pretty-writer$fn  clojure.lang.AFn.throwArity (AFn.java:437)
#inst "

Cause: clojure.pprint/pretty-writer does not fully implement the Writer interface used in printing the calendar.

Approach: Better implement Writer interface.

Patch: CLJ-1390-pprint-GregorianCalendar.patch

Screened by: Alex Miller



 Comments   
Comment by Norman Richards [ 29/Mar/14 4:15 PM ]

The print-calendar function introduced in CLJ-928 doesn't work with clojure.pprint/pretty-writer since pretty-writer does not correctly implement the java.io.Writer interface. It only implements write(String) but print-calendar wants write(String,int,int) among others. Although pretty-writer should probably correctly implement java.io.Writer, it's pretty easy to rewrite print-calendar to use the supported subset of java.io.Writer that is implemented.

Comment by Steve Suehs [ 29/Mar/14 4:20 PM ]

Thank you, "random person at the Austin Clojure Hack Day" "that I don't know" that has a CA in place. You are awesome!

See you at the next Austin Clojure Meetup.

-s

Comment by Andy Fingerhut [ 04/Apr/14 2:51 PM ]

Norman, it would be good if the patch included a few test cases, especially ones that fail without the patch, and succeed with the patch.

Comment by Norman Richards [ 04/Apr/14 3:09 PM ]

Absolutely. I have no idea how test cases work on Clojure core, but I assume it should be easy enough to do.

Comment by Andy Fingerhut [ 04/Apr/14 3:15 PM ]

I would recommend looking at the following file in the Clojure repo, which already contains other pprint tests. It shouldn't be too difficult to get an idea from one or more of the tests there. Actually those might be slightly unusual in that many of them use a simple-tests macro defined in file test_helper.clj that most of the Clojure tests do not use, but ask questions if you have trouble, e.g. on the Clojure Google group or IRC channel.

test/clojure/test_clojure/pprint/test_pretty.clj

Comment by Steve Suehs [ 04/Apr/14 7:50 PM ]

Alright...you two are inspiring me to go work on getting my CA in.

Comment by Norman Richards [ 04/Apr/14 8:31 PM ]

Test case attached. Apply the test patch, "mvn test" fails. Apply the fix, test passes.

Comment by Andy Fingerhut [ 06/Apr/14 3:32 PM ]

It would be better if the "is" were of the form:

(is (= calculated-value "constant string to compare against") "string to show if test fails")

rather than just (is calculated-value "string to show if test fails"). The second form will fail if calculating the value throws an exception, but only the first form will calculate it, and then verify that the value is the expected one (and fail if it is not the expected one).

Comment by Norman Richards [ 07/Apr/14 10:49 AM ]

Ok - here's an alternative test case. I'm less happy with this test case, since I have to add the TimeZone and make assumptions about how the specifics of how the pretty printer formats. But, it does test the fix adequately, so if you like the test2 patch better, that's perfectly fine with me.

Comment by Steve Miner [ 10/Apr/14 4:23 PM ]

I would rather fix the actual bug in pretty_writer.clj. The proxy needs to support more of the java.io.Writer interface. I think adding another arity to the write method would work. Something like:

(write 
   ...
  ([x off len]
      (.write this (subs (str x) off (+ off len)))))
Comment by Steve Miner [ 10/Apr/14 4:38 PM ]

CLJ-1390-pprint-GregorianCalendar.patch fixes the pretty_writer.clj proxy to support the missing version of the write method. Includes the same test as the previous patch.

Comment by Alex Miller [ 28/Dec/14 10:53 AM ]

Agreed with Steve on approach.





[CLJ-1361] Pretty printing code using pprint with code-dispatch incorrectly prints a simple ns macro call. Created: 18/Feb/14  Updated: 31/Jul/15  Resolved: 31/Jul/15

Status: Closed
Project: Clojure
Component/s: None
Affects Version/s: Release 1.5, Release 1.6
Fix Version/s: Release 1.8

Type: Defect Priority: Minor
Reporter: Russ Olsen Assignee: Unassigned
Resolution: Completed Votes: 0
Labels: ft, print
Environment:

java version "1.6.0_65"
Java(TM) SE Runtime Environment (build 1.6.0_65-b14-462-11M4609)
Java HotSpot(TM) 64-Bit Server VM (build 20.65-b04-462, mixed mode)

Mac OS X 10.9.1


Attachments: Text File simple-ns-pprint-fix.patch    
Patch: Code and Test
Approval: Ok

 Description   

Pretty printing code using pprint with code-dispatch incorrectly prints a simple ns macro call.
The problem is that "nil" is added to the output after the namespace name.

user> (use 'clojure.pprint)
nil
user> (def code '(ns foo.bar))
#'user/code
user> (with-out-str (with-pprint-dispatch code-dispatch (pprint code)))
"(ns foo.barnil)\n"   ;; Expected: {{"(ns foo.bar)\n"}}

Cause: In clojure.pprint/pprint-ns-reference, reference is printed regardless, but may be nil.
Solution: Omit printing reference if nil.
Patch: simple-ns-pprint-fix.patch
Screened by: Alex Miller






[CLJ-1319] array-map fails lazily if passed an odd number of arguments Created: 08/Jan/14  Updated: 03/Aug/15

Status: Reopened
Project: Clojure
Component/s: None
Affects Version/s: None
Fix Version/s: Release 1.8

Type: Defect Priority: Minor
Reporter: Stuart Sierra Assignee: Unassigned
Resolution: Unresolved Votes: 1
Labels: errormsgs, ft

Attachments: Text File 0001-CLJ-1319-Throw-on-odd-arguments-to-PersistentArrayMa.patch     Text File 0002-CLJ-1319-Throw-on-odd-arguments-to-PersistentArrayMa.patch    
Patch: Code and Test
Approval: Ok

 Description   

If called with an odd number of arguments, array-map does not throw an exception until the map is realized, when it throws the confusing ArrayIndexOutOfBoundsException.

Example, in 1.5.1 and 1.6.0-alpha3:

user=> (def m (hash-map :a 1 :b))
IllegalArgumentException No value supplied for key: :b  clojure.lang.PersistentHashMap.create (PersistentHashMap.java:77)

user=> (def m (array-map :a 1 :b))
#'user/m
user=> (prn m)
ArrayIndexOutOfBoundsException 3
  clojure.lang.PersistentArrayMap$Seq.first
  (PersistentArrayMap.java:313)

Approach: Catch on construction and throw same error as hash-map.

user=> (def m (array-map :a 1 :b))
IllegalArgumentException No value supplied for key: :b  clojure.lang.PersistentArrayMap.createAsIfByAssoc (PersistentArrayMap.java:78)

Patch: 0002-CLJ-1319-Throw-on-odd-arguments-to-PersistentArrayMa.patch
Screened by: Alex Miller



 Comments   
Comment by Alex Miller [ 08/Jan/14 11:01 AM ]

PersistentArrayMap.createAsIfByAssoc could check length is even to catch this

Comment by Jason Felice [ 27/Jan/14 1:01 PM ]

A better error message would be nice... this is the best I could think of.

Comment by OHTA Shogo [ 14/May/15 7:06 AM ]

I suffered from this problem recently. Anything to resolve before applying the patch?

Comment by Alex Miller [ 14/May/15 8:21 AM ]

The error message should match the message for hash-map:

user=> (hash-map 1 2 3)
IllegalArgumentException No value supplied for key: 3  clojure.lang.PersistentHashMap.create (PersistentHashMap.java:77)
user=> (array-map 1 2 3)
IllegalArgumentException Odd number of arguments when creating map  clojure.lang.PersistentArrayMap.createAsIfByAssoc (PersistentArrayMap.java:78)
Comment by OHTA Shogo [ 14/May/15 8:49 PM ]

Let me have a try. I just updated the patch!

0002-CLJ-1319-Throw-on-odd-arguments-to-PersistentArrayMa.patch

Comment by Alex Miller [ 15/May/15 9:10 AM ]

Looks good.

Comment by Alex Miller [ 03/Aug/15 8:05 AM ]

Reopened - does not seem to have correct behavior in 1.8.0-alpha4.

user=> (array-map 1 2 3)
ArrayIndexOutOfBoundsException 3  clojure.lang.PersistentArrayMap$Seq.first (PersistentArrayMap.java:321)
user=> (def m (array-map :a 1 :b))
#'user/m
Comment by Alex Miller [ 03/Aug/15 8:17 AM ]

Doesn't like this was merged so moving to Ok but not closed for next round.





[CLJ-1313] Correct a few unit tests Created: 23/Dec/13  Updated: 17/Jul/15  Resolved: 17/Jul/15

Status: Closed
Project: Clojure
Component/s: None
Affects Version/s: Release 1.5, Release 1.6
Fix Version/s: Release 1.8

Type: Defect Priority: Major
Reporter: Andy Fingerhut Assignee: Unassigned
Resolution: Completed Votes: 1
Labels: ft

Attachments: File clj-1313-v1.diff     File clj-1313-v2.diff     Text File clj-1313-v3.patch    
Patch: Code and Test
Approval: Ok

 Description   

Several unit tests do not test what they appear to have been intended to test, because of missing is statements around (= expr1 expr2) expressions, or because of use of (is (thrown? ...)) instead of (is (thrown-with-msg? ...))

Patch: clj-1313-v3.patch

Screened by: Alex Miller



 Comments   
Comment by Andy Fingerhut [ 23/Dec/13 3:11 AM ]

Patch clj-1313-v1.diff wraps (is ...) around several = expressions in unit tests that appeared to have been missing them, and changes several thrown? to thrown-with-msg? when there were regexes that were unused.

Comment by Stuart Halloway [ 31/Jan/14 12:36 PM ]

please update to apply cleanly on master

Comment by Andy Fingerhut [ 31/Jan/14 3:29 PM ]

clj-1313-v2.diff is identical to clj-1313-v1.diff except that it removes the portion that conflicts with the latest Clojure master. That portion needs updating for a different reason anyway (ticket CLJ-1328), and is probably best put into a patch for that ticket.

Comment by Andy Fingerhut [ 04/Apr/15 1:36 PM ]

clj-1313-v3.patch adds two other test corrections/improvements that were not in the -v2 patch. The change from doall to dorun is not really a correction, as much as cleaning up the fact that the return value of doall was discarded.





[CLJ-1232] Functions with non-qualified return type hints force import of hinted classes when called from other namespace Created: 18/Jul/13  Updated: 24/Aug/15

Status: Open
Project: Clojure
Component/s: None
Affects Version/s: Release 1.5, Release 1.6, Release 1.7
Fix Version/s: Release 1.8

Type: Defect Priority: Major
Reporter: Tassilo Horn Assignee: Unassigned
Resolution: Unresolved Votes: 8
Labels: compiler, typehints

Attachments: Text File 0001-auto-qualify-arglists-class-names.patch     Text File 0001-auto-qualify-arglists-class-names-v2.patch     Text File 0001-auto-qualify-arglists-class-names-v3.patch     Text File 0001-auto-qualify-arglists-class-names-v4.patch     Text File 0001-CLJ-1232-auto-qualify-arglists-class-names.patch     Text File 0001-throw-on-non-qualified-class-names-that-are-not-auto.patch    
Patch: Code and Test
Approval: Ok

 Description   

You can add a type hint to function arglists to indicate the return type of a function like so.

user> (import '(java.util List))
java.util.List
user> (defn linkedlist ^List [] (java.util.LinkedList.))
#'user/linkedlist
user> (.size (linkedlist))
0

The problem is that now when I call `linkedlist` exactly as above from another namespace, I'll get an exception because java.util.List is not imported in there.

user> (in-ns 'user2)
#<Namespace user2>
user2> (refer 'user)
nil
user2> (.size (linkedlist))
CompilerException java.lang.IllegalArgumentException: Unable to resolve classname: List, compiling:(NO_SOURCE_PATH:1:1)
user2> (import '(java.util List)) ;; Too bad, need to import List here, too.
java.util.List
user2> (.size (linkedlist))
0

There are two workarounds: You can import the hinted type also in the calling namespace, or you always use fully qualified class names for return type hints. Clearly, the latter is preferable.

Approach: Resolve the tags in the defn macro.

Patch: 0001-CLJ-1232-auto-qualify-arglists-class-names.patch

Screened by: Fogus

Alternate approach: Make the compiler resolve the return tags when necessary (tag is not a string, primitive tag (^long) or array tag (^longs)) and update the Var's :arglist appropriately. Patch: 0001-auto-qualify-arglists-class-names-v4.patch. Note that this patch had problems with Hystrix, which was using non-conformant arglists - Hystrix has since been patched.



 Comments   
Comment by Andy Fingerhut [ 16/Apr/14 3:47 PM ]

To make sure I understand, Nicola, in this ticket you are asking that the Clojure compiler change behavior so that the sample code works correctly with no exceptions, the same way as it would work correctly without exceptions if one of the workarounds were used?

Comment by Tassilo Horn [ 17/Apr/14 12:18 AM ]

Hi Andy. Tassilo here, not Nicola. But yes, the example should work as-is. When I'm allowed to use type hints with simple imported class names for arguments, then doing so for return values should work, too.

Comment by Rich Hickey [ 10/Jun/14 10:41 AM ]

Type hints on function params are only consumed by the function definition, i.e. in the same module as the import/alias. Type hints on returns are just metadata, they don't get 'compiled' and if the metadata is not useful to consumers in other namespaces, it's not a useful hint. So, if it's not a type in the auto-imported set (java.lang), it should be fully qualified.

Comment by Alex Miller [ 10/Jun/14 11:55 AM ]

Based on Rich's comment, this ticket should probably morph into an enhancement request on documentation, probably on http://clojure.org/java_interop#Java Interop-Type Hints.

Comment by Andy Fingerhut [ 10/Jun/14 3:13 PM ]

I would suggest something like the following for a documentation change, after this part of the text on the page Alex links in the previous comment:

For function return values, the type hint can be placed before the arguments vector:

(defn hinted
(^String [])
(^Integer [a])
(^java.util.List [a & args]))

-> #user/hinted

If the return value type hint is for a class that is outside of java.lang, which is the part auto-imported by Clojure, then it must be a fully qualified class name, e.g. java.util.List, not List.

Comment by Nicola Mometto [ 10/Jun/14 4:02 PM ]

I don't understand why we should enforce this complexity to the user.
Why can't we just make the Compiler (or even defn itself) update all the arglists tags with properly resolved ones? (that's what I'm doing in tools.analyzer.jvm)

Comment by Alexander Kiel [ 19/Jul/14 10:02 AM ]

I'm with Nicola here. I also think that defn should resolve the type hint according the imports of the namespace defn is used in.

Comment by Max Penet [ 22/Jul/14 7:06 AM ]

Same here, I was bit by this in the past. The current behavior is clearly counterintuitive.

Comment by Nicola Mometto [ 28/Aug/14 12:58 PM ]

Attached two patches implementing two different solutions:

  • 0001-auto-qualify-arglists-class-names.patch makes the compiler automatically qualify all the tags in the :arglists
  • 0001-throw-on-non-qualified-class-names-that-are-not-auto.patch makes the compiler throw an exception for all public defs whose return tag is a symbol representing a non-qualified class that is not in the auto-import list (approach proposed in IRC by Alex Miller)
Comment by Tassilo Horn [ 29/Aug/14 1:49 AM ]

For what it's worth, I'd prefer the first patch because the second doesn't help in situations where the caller lives in a namespace where the called function's return type hinted class is `ns-unmap`-ed. And there a good reasons for doing that. For example, Process is a java.lang class and Process is a pretty generic name. So in some namespace, I want to define my own Process deftype or defrecord. Without unmapping 'Process first to get rid of the java.lang.Process auto-import, I'd get an exception:

user> (deftype Process [])
IllegalStateException Process already refers to: class java.lang.Process in namespace: user  clojure.lang.Namespace.referenceClass (Namespace.java:140)

Now when I call some function from some library that has a `^Process` return type hint (meaning java.lang.Process there), I get the same exception as in my original report.

I can even get into troubles when only using standard Clojure functions because those have `^String` and `^Class` type hints. IMO, Class is also a pretty generic name I should be able to name my custom deftype/defrecord. And I might also want to have a custom String type/record in my astrophysics system.

Comment by Andy Fingerhut [ 30/Sep/14 4:39 PM ]

Not sure whether the root cause of this behavior is the same as the example in the description or not, but seems a little weird that even for fully qualified Java class names hinting the arg vector, it makes a difference whether it is done with defn or def:

Clojure 1.6.0
user=> (set! *warn-on-reflection* true)
true
user=> (defn f1 ^java.util.LinkedList [coll] (java.util.LinkedList. coll))
#'user/f1
user=> (def f2 (fn ^java.util.LinkedList [coll] (java.util.LinkedList. coll)))
#'user/f2
user=> (.size (f1 [2 3 4]))
3
user=> (.size (f2 [2 3 4]))
Reflection warning, NO_SOURCE_PATH:5:1 - reference to field size can't be resolved.
3
Comment by Alex Miller [ 30/Sep/14 6:21 PM ]

Andy, can you file that as a separate ticket?

Comment by Andy Fingerhut [ 30/Sep/14 9:08 PM ]

Created ticket CLJ-1543 for the issue raised in my comment earlier on 30 Sep 2014.

Comment by Andy Fingerhut [ 01/Oct/14 12:38 PM ]

Tassilo (or anyone), is there a reason to prefer putting the tag on the argument vector in your example? It seems that putting it on the Var name instead avoids this issue:

user=> (clojure-version)
"1.6.0"
user=> (set! *warn-on-reflection* true)
true
user=> (import '(java.util List))
java.util.List
user=> (defn ^List linkedlist [] (java.util.LinkedList.))
#'user/linkedlist
user=> (.size (linkedlist))
0
user=> (ns user2)
nil
user2=> (refer 'user)
nil
user2=> (.size (linkedlist))
0

I suppose that only allows a single type tag, rather than an independent one for each arity.

Comment by Tassilo Horn [ 02/Oct/14 3:16 AM ]

I wasn't aware of the fact that you can put it on the var's name. That's not documented at http://clojure.org/java_interop#Java Interop-Type Hints. But IMHO the documented version with putting the tag on the argument vector is more general since it supports different return type hints for the different arity version. In any case, if both forms are permitted then they should be equivalent in the case the function has only one arity.

Comment by Rich Hickey [ 16/Mar/15 12:02 PM ]

Please work on the simplest patch that resolves the names

Comment by Alex Miller [ 16/Mar/15 4:34 PM ]

Nicola, in this:

if (tag != null &&
                        !(tag instanceof String) &&
                        primClass((Symbol)tag) == null &&
                        !tagClass((Symbol) tag).getName().startsWith("["))
                        {
                            argvec = (IPersistentVector)((IObj)argvec).withMeta(RT.map(RT.TAG_KEY, Symbol.intern(tagClass((Symbol)tag).getName())));
                        }

doesn't tagClass already handle most of these cases properly already? Can this be simplified? Is there an optimization case in avoiding lookup for a dotted name?

Comment by Nicola Mometto [ 16/Mar/15 5:10 PM ]

Patch 0001-auto-qualify-arglists-class-names-v2.patch avoids doing unnecessary lookups (dotted names, special tags (primitive tags, array tags)) and adds a testcase

Comment by Michael Blume [ 20/May/15 1:13 PM ]

I'm seeing an odd failure with this patch and hystrix defcommands, will post a small reproduction shortly

Comment by Michael Blume [ 20/May/15 1:20 PM ]

https://github.com/MichaelBlume/hystrix-demo

passes lein check with 1.7 beta3, fails with v3 patch applied

Comment by Nicola Mometto [ 20/May/15 1:40 PM ]

During analysis the compiler understands only arglists in the form of (quote ([..]*)) (see https://github.com/clojure/clojure/blob/master/src/jvm/clojure/lang/Compiler.java#L557-L558), hystrix emits arglists in the form of (list (quote [..])*).

Not sure what to do about this.

Comment by Michael Blume [ 20/May/15 1:51 PM ]

Possibly just ask Hystrix not to do that?

Comment by Alex Miller [ 20/May/15 2:30 PM ]

test.generative uses non-standard arglists too. I haven't looked at the patch, but if it's sensitive to that, it's probably not good enough.

Comment by Nicola Mometto [ 20/May/15 6:51 PM ]

test.generative uses non-standard :tag, not :arglists

Comment by Alex Miller [ 20/May/15 10:22 PM ]

ah, yes. sorry.

Comment by Stuart Halloway [ 10/Jul/15 12:15 PM ]

Opened https://github.com/Netflix/Hystrix/issues/831 to see what is up with hystrix.

Comment by Stuart Halloway [ 31/Jul/15 9:16 AM ]

Can somebody with context update this patch to apply to latest master?

Comment by Nicola Mometto [ 31/Jul/15 9:37 AM ]

patch updated

Comment by Matt Jacobs [ 06/Aug/15 12:59 PM ]

I've released hystrix-clj 1.4.14, with a fix for https://github.com/Netflix/Hystrix/issues/831: https://github.com/Netflix/Hystrix/releases/tag/v1.4.14. If anything else is needed on the Hystrix side, please open another issue.

Comment by Fogus [ 07/Aug/15 11:03 AM ]

Applied the 0001-CLJ-1232-auto-qualify-arglists-class-names.patch to master (1.8) and ran through the example scenario to check the veracity of the change. Additionally, I ran a modified snippet using def and verified that it too worked. Finally, after checking the code it seems reasonable in implementation and scope.

Comment by Rich Hickey [ 08/Aug/15 9:56 AM ]

I'd rather not change the compiler and it seems hystrix was broken. Also please make it clear what the single strategy and patch is at the top of the ticket.

Comment by Alex Miller [ 11/Aug/15 5:01 PM ]

Made preferred approach clear in the description and moved back to vetted. I believe the other ticket was screened by Fogus so this needs to be re-screened with the preferred approach ticket.

Comment by Alex Miller [ 11/Aug/15 5:08 PM ]

Nicola and Fogus, would appreciate an eye as to whether I just made the proper alterations in the description.

Comment by Nicola Mometto [ 12/Aug/15 6:17 AM ]

Alex Miller Reading Fogus' last comment, it seems to me that he actually screened the current patch (the one who resolves tags in defn rather than in the compiler) "Applied the 0001-CLJ-1232-auto-qualify-arglists-class-names.patch to master [..]"

Comment by Fogus [ 14/Aug/15 9:18 AM ]

It would be better if the description said the word "preferred," but based on my reading resolving the tags in defn is the winner (0001-CLJ-1232-auto-qualify-arglists-class-names.patch). Thankfully that's the one that I screened as I agree that non-trivial changes to the compiler are to be avoided. That said, of course that patch does modify the compiler but since it's a change that pulls out some code into a useful method I let it through. To my mind that was a trivial (and justifiable change). Of course, if we want to avoid any compiler changes then this patch would have to be reworked.

Comment by Alex Miller [ 18/Aug/15 5:03 PM ]

Moving back to screened as fogus screened the preferred (by Rich) version.





[CLJ-1225] quot overflow issues around Long/MIN_VALUE for BigInt Created: 25/Jun/13  Updated: 31/Jul/15  Resolved: 31/Jul/15

Status: Closed
Project: Clojure
Component/s: None
Affects Version/s: Release 1.5
Fix Version/s: Release 1.8

Type: Defect Priority: Minor
Reporter: Andy Fingerhut Assignee: Unassigned
Resolution: Completed Votes: 1
Labels: ft, math

Attachments: Text File clj-1225-2.txt     Text File clj-1225-fix-division-overflow-patch-v1.txt    
Patch: Code and Test
Approval: Ok

 Description   

In Clojure 1.5.1, see the following undesirable behavior regarding incorrect quot results for BigInts:

user=> (quot Long/MIN_VALUE -1N)
-9223372036854775808N
user=> (quot (bigint Long/MIN_VALUE) -1)
-9223372036854775808N

Similar issue to CLJ-1222. The root cause is that Java division of longs gives a numerically incorrect answer of Long.MIN_VALUE for (Long.MIN_VALUE / -1), because the numerically correct answer does not fit in a long. I believe this is the only pair of arguments for long division that gives a numerically incorrect answer, because division with a denominator having an absolute value of 2 or more gives a result closer to 0 than the numerator, and everything works fine for a denominator of 1 or -1, except this one case.

Related issues: CLJ-1222 for multiply, CLJ-1253 for / on longs, CLJ-1254 for quot on longs

Patch: clj-1225-2.txt
Screened by: Alex Miller



 Comments   
Comment by Andy Fingerhut [ 25/Jun/13 11:03 AM ]

Patch clj-1225-fix-division-overflow-patch-v1.txt dated Jun 25 2013 may be one good way to address this issue. It modifies quot and / to return the numerically correct (BigInt) answer when given args Long/MIN_VALUE and -1.

It also removes the quotient intrinsic that does a JVM LDIV operation on longs for quot, since that operation is one of those that gives the incorrect result. I have not done any performance testing with this patch yet, but I have verified that it does not introduce any new reflection warnings when compiling Clojure itself.

Comment by Andy Fingerhut [ 25/Jun/13 11:13 AM ]

Another possible approach would be to create unchecked-quotient and quot', which together with quot would correspond to the existing unchecked-multiply, *' and *. That is a more significant change. One potential concern it addresses that patch clj-1225-fix-division-overflow-patch-v1.txt does not is that patch leaves a Clojure developer with no way to do a primitive Java long division except by writing Java code.

Comment by Rich Hickey [ 05/Sep/13 8:39 AM ]

this is two separate issues, one with longs and one with bigints. long problem should throw

Comment by Andy Fingerhut [ 06/Sep/13 12:37 AM ]

Updating description for BigInt issue only. Will create separate ticket for incorrect behavior of / and quot on long type args Long/MIN_VALUE and -1.

Comment by Andy Fingerhut [ 06/Sep/13 12:41 AM ]

Patch clj-1225-2.txt fixes this issue with quot on BigInts, with tests for quot and / on these values. / on BigInt worked fine before, but added the tests in case someone decides to change the implementation and forgets this corner case.





[CLJ-1208] Namespace is not loaded on defrecord class init Created: 03/May/13  Updated: 17/Jul/15  Resolved: 17/Jul/15

Status: Closed
Project: Clojure
Component/s: None
Affects Version/s: None
Fix Version/s: Release 1.8

Type: Enhancement Priority: Major
Reporter: Tim McCormack Assignee: Unassigned
Resolution: Completed Votes: 5
Labels: compiler, defrecord, deftype

Attachments: Text File 0001-CLJ-1208-load-own-namespace-in-deftype-defrecord-cla.patch     Text File 0001-CLJ-1208-load-own-namespace-in-deftype-defrecord-cla-v2.patch     Text File 0001-CLJ-1208-load-own-namespace-in-deftype-defrecord-cla-v3.patch     Text File 0001-CLJ-1208-load-own-namespace-in-deftype-defrecord-cla-v4.patch     Text File 0001-CLJ-1208-load-own-namespace-in-deftype-defrecord-cla-v5-no-opts.patch     Text File 0001-CLJ-1208-load-own-namespace-in-deftype-defrecord-cla-v5.patch    
Patch: Code and Test
Approval: Ok

 Description   

As a user of Clojure interop from Java, I want defrecords (and deftypes?) to load their namespaces upon class initialization so that I can simply construct and use AOT'd record classes without manually requiring their namespaces first.

Calling the defrecord's constructor may or may not result in "Attempting to call unbound fn" exceptions, depending on what code has already been run.

This issue has been raised several times over the years, but I could not find an existing ticket for it:

Screened: Approach 2, which makes loading the namespace optional. This makes the change purely additive, whereas Approach 1 changes semantics (load ordering) of existing programs. I don't know why we would want that risk, especially when it is not clear that most users of deftype would even want ns loading side effect.

Approach 1: require the namespace a record/type belongs to during the record/type class init
Patch: 0001-CLJ-1208-load-own-namespace-in-deftype-defrecord-cla-v5-no-opts.patch

Approach 2: like Approach 1 but does the automatic loading only when a :load-ns option is set to true in the deftype/defrecord
Patch: 0001-CLJ-1208-load-own-namespace-in-deftype-defrecord-cla-v5.patch

Note: patch for Approach 1 causes some generative tests to fail since the namespace used to evalaute a defrecord is immediately destroyed thus impossible to load



 Comments   
Comment by Nicola Mometto [ 18/Jan/15 7:10 AM ]

The attached patch approaches this issue by adding a :load-ns options to deftype/defrecord which defaults to false.
When true, the type/record be compiled with a call to clojure.core/require to its originating namespace in its static initializer.

The patch has two known limitations:

  • clojure.core deftypes/defrecords cannot have :load-ns since we use clojure.core/require to load the namespaces so clojure.core needs to be loaded manually anyway
  • clojure.lang.Compiler/demunge is used to get the originating namespace from the deftype/defrecord class name, this means that namespaces like foo_bar are not supported since they get demunged into foo-bar. If this is something that needs to be addressed, it shouldn't be too hard to just pass the unmunged namespace name in the opts map.
Comment by Nicola Mometto [ 22/Jan/15 12:59 PM ]

Updated patch fixing a whitespace error and mentionint :load-ns in the docstrings of deftype/defrecord

Comment by Nicola Mometto [ 11/Mar/15 6:12 AM ]

Updated patch so it applies on lastest HEAD

Comment by Michael Blume [ 17/Jun/15 12:12 PM ]

No longer applies I'm afraid

Comment by Nicola Mometto [ 17/Jun/15 12:22 PM ]

Thanks Michael, updated the patch. I have to say it's getting kind of annoying having to maintain a patch for months without any feedback.

Comment by Alex Miller [ 17/Jun/15 1:03 PM ]

What are the negative impacts if this is always done, rather than being an option?

Comment by Alex Miller [ 17/Jun/15 1:06 PM ]

Also, you should never rely on demunge - it's best-effort for printing purposes only.

Comment by Nicola Mometto [ 17/Jun/15 1:09 PM ]

Does extra bytecode emitted count as a negative impact?

Comment by Alex Miller [ 17/Jun/15 1:15 PM ]

No, I'm not concerned about that.

Comment by Nicola Mometto [ 17/Jun/15 1:48 PM ]

Attached patch that doesn't use demunge but change the macroexpansion of defrecord and deftype to include the namespace segment in the tagsym in deftype* special form

Comment by Nicola Mometto [ 17/Jun/15 2:02 PM ]

Alex, I attached two versions of the last patch, one with :load-ns and one without.
Making :load-ns the default behaviour causes some generative tests to fail since they immediately eliminate the namespace used to defrecord making the record class fail when trying to load said namespace.

I can try to change those tests if necessary.

Comment by Michael Blume [ 10/Jul/15 6:09 PM ]

v5-no-opts breaks generative tests on my box:

test-generative:
     [java] java.util.concurrent.ExecutionException: java.lang.ExceptionInInitializerError, compiling:(/Users/michael.blume/workspace/clojure/src/script/run_test_generative.clj:5:1)
     [java] 	at java.util.concurrent.FutureTask.report(FutureTask.java:122)
     [java] 	at java.util.concurrent.FutureTask.get(FutureTask.java:192)
     [java] 	at clojure.core$deref_future.invoke(core.clj:2186)
     [java] 	at clojure.core$future_call$reify__6736.deref(core.clj:6683)
     [java] 	at clojure.core$deref.invoke(core.clj:2206)
     [java] 	at clojure.core$map$fn__4553.invoke(core.clj:2622)
     [java] 	at clojure.lang.LazySeq.sval(LazySeq.java:40)
     [java] 	at clojure.lang.LazySeq.seq(LazySeq.java:56)
     [java] 	at clojure.lang.RT.seq(RT.java:507)
     [java] 	at clojure.core$seq__4128.invoke(core.clj:137)
     [java] 	at clojure.core$concat$cat__4217$fn__4218.invoke(core.clj:700)
     [java] 	at clojure.lang.LazySeq.sval(LazySeq.java:40)
     [java] 	at clojure.lang.LazySeq.seq(LazySeq.java:49)
     [java] 	at clojure.lang.ChunkedCons.chunkedNext(ChunkedCons.java:59)
     [java] 	at clojure.core$chunk_next.invoke(core.clj:673)
     [java] 	at clojure.core.protocols$fn__6518.invoke(protocols.clj:139)
     [java] 	at clojure.core.protocols$fn__6478$G__6473__6487.invoke(protocols.clj:19)
     [java] 	at clojure.core.protocols$seq_reduce.invoke(protocols.clj:31)
     [java] 	at clojure.core.protocols$fn__6506.invoke(protocols.clj:101)
     [java] 	at clojure.core.protocols$fn__6452$G__6447__6465.invoke(protocols.clj:13)
     [java] 	at clojure.core$reduce.invoke(core.clj:6519)
     [java] 	at clojure.test.generative.runner$run_suite.invoke(runner.clj:185)
     [java] 	at clojure.test.generative.runner$_main.doInvoke(runner.clj:208)
     [java] 	at clojure.lang.RestFn.invoke(RestFn.java:408)
     [java] 	at user$eval644.invoke(run_test_generative.clj:5)
     [java] 	at clojure.lang.Compiler.eval(Compiler.java:6797)
     [java] 	at clojure.lang.Compiler.load(Compiler.java:7242)
     [java] 	at clojure.lang.Compiler.loadFile(Compiler.java:7180)
     [java] 	at clojure.main$load_script.invoke(main.clj:275)
     [java] 	at clojure.main$script_opt.invoke(main.clj:337)
     [java] 	at clojure.main$main.doInvoke(main.clj:421)
     [java] 	at clojure.lang.RestFn.invoke(RestFn.java:408)
     [java] 	at clojure.lang.Var.invoke(Var.java:379)
     [java] 	at clojure.lang.AFn.applyToHelper(AFn.java:154)
     [java] 	at clojure.lang.Var.applyTo(Var.java:700)
     [java] 	at clojure.main.main(main.java:37)
     [java] Caused by: java.lang.ExceptionInInitializerError, compiling:(/Users/michael.blume/workspace/clojure/src/script/run_test_generative.clj:5:1)
     [java] 	at clojure.lang.Compiler.analyzeSeq(Compiler.java:6745)
     [java] 	at clojure.lang.Compiler.analyze(Compiler.java:6539)
     [java] 	at clojure.lang.Compiler.analyze(Compiler.java:6500)
     [java] 	at clojure.lang.Compiler$BodyExpr$Parser.parse(Compiler.java:5880)
     [java] 	at clojure.lang.Compiler$FnMethod.parse(Compiler.java:5311)
     [java] 	at clojure.lang.Compiler$FnExpr.parse(Compiler.java:3925)
     [java] 	at clojure.lang.Compiler.analyzeSeq(Compiler.java:6736)
     [java] 	at clojure.lang.Compiler.analyze(Compiler.java:6539)
     [java] 	at clojure.lang.Compiler.analyzeSeq(Compiler.java:6726)
     [java] 	at clojure.lang.Compiler.analyze(Compiler.java:6539)
     [java] 	at clojure.lang.Compiler.access$300(Compiler.java:38)
     [java] 	at clojure.lang.Compiler$DefExpr$Parser.parse(Compiler.java:578)
     [java] 	at clojure.lang.Compiler.analyzeSeq(Compiler.java:6738)
     [java] 	at clojure.lang.Compiler.analyze(Compiler.java:6539)
     [java] 	at clojure.lang.Compiler.analyzeSeq(Compiler.java:6726)
     [java] 	at clojure.lang.Compiler.analyze(Compiler.java:6539)
     [java] 	at clojure.lang.Compiler.analyze(Compiler.java:6500)
     [java] 	at clojure.lang.Compiler$BodyExpr$Parser.parse(Compiler.java:5878)
     [java] 	at clojure.lang.Compiler$LetExpr$Parser.parse(Compiler.java:6194)
     [java] 	at clojure.lang.Compiler.analyzeSeq(Compiler.java:6738)
     [java] 	at clojure.lang.Compiler.analyze(Compiler.java:6539)
     [java] 	at clojure.lang.Compiler.analyze(Compiler.java:6500)
     [java] 	at clojure.lang.Compiler$BodyExpr$Parser.parse(Compiler.java:5880)
     [java] 	at clojure.lang.Compiler$FnMethod.parse(Compiler.java:5311)
     [java] 	at clojure.lang.Compiler$FnExpr.parse(Compiler.java:3925)
     [java] 	at clojure.lang.Compiler.analyzeSeq(Compiler.java:6736)
     [java] 	at clojure.lang.Compiler.analyze(Compiler.java:6539)
     [java] 	at clojure.lang.Compiler.eval(Compiler.java:6794)
     [java] 	at clojure.lang.Compiler.eval(Compiler.java:6760)
     [java] 	at clojure.core$eval.invoke(core.clj:3081)
     [java] 	at clojure.test_clojure.generators$generate_namespaces$make_in_ns__830.invoke(generators.clj:30)
     [java] 	at clojure.test_clojure.generators$generate_namespaces$fn__836$fn__837.invoke(generators.clj:41)
     [java] 	at clojure.core$map$fn__4553.invoke(core.clj:2622)
     [java] 	at clojure.lang.LazySeq.sval(LazySeq.java:40)
     [java] 	at clojure.lang.LazySeq.seq(LazySeq.java:49)
     [java] 	at clojure.lang.RT.seq(RT.java:507)
     [java] 	at clojure.core$seq__4128.invoke(core.clj:137)
     [java] 	at clojure.core$concat$fn__4215.invoke(core.clj:691)
     [java] 	at clojure.lang.LazySeq.sval(LazySeq.java:40)
     [java] 	at clojure.lang.LazySeq.seq(LazySeq.java:49)
     [java] 	at clojure.lang.RT.seq(RT.java:507)
     [java] 	at clojure.core$seq__4128.invoke(core.clj:137)
     [java] 	at clojure.core$concat$cat__4217$fn__4218.invoke(core.clj:700)
     [java] 	at clojure.lang.LazySeq.sval(LazySeq.java:40)
     [java] 	at clojure.lang.LazySeq.seq(LazySeq.java:49)
     [java] 	at clojure.lang.RT.seq(RT.java:507)
     [java] 	at clojure.core$seq__4128.invoke(core.clj:137)
     [java] 	at clojure.core.protocols$seq_reduce.invoke(protocols.clj:30)
     [java] 	at clojure.core.protocols$fn__6506.invoke(protocols.clj:101)
     [java] 	at clojure.core.protocols$fn__6452$G__6447__6465.invoke(protocols.clj:13)
     [java] 	at clojure.core$reduce.invoke(core.clj:6519)
     [java] 	at clojure.core$into.invoke(core.clj:6600)
     [java] 	at clojure.test_clojure.generators$generate_namespaces.invoke(generators.clj:44)
     [java] 	at clojure.test_clojure.generators$fn__841.invoke(generators.clj:50)
     [java] 	at clojure.lang.Delay.deref(Delay.java:37)
     [java] 	at clojure.core$deref.invoke(core.clj:2206)
     [java] 	at clojure.test_clojure.generators$var.invoke(generators.clj:58)
     [java] 	at clojure.test.generative.runner$eval559$fn__560$fn__561$fn__562$fn__563.invoke(runner.clj:66)
     [java] 	at clojure.core$map$fn__4553.invoke(core.clj:2622)
     [java] 	at clojure.lang.LazySeq.sval(LazySeq.java:40)
     [java] 	at clojure.lang.LazySeq.seq(LazySeq.java:49)
     [java] 	at clojure.lang.RT.seq(RT.java:507)
     [java] 	at clojure.core$seq__4128.invoke(core.clj:137)
     [java] 	at clojure.core.protocols$seq_reduce.invoke(protocols.clj:30)
     [java] 	at clojure.core.protocols$fn__6506.invoke(protocols.clj:101)
     [java] 	at clojure.core.protocols$fn__6452$G__6447__6465.invoke(protocols.clj:13)
     [java] 	at clojure.core$reduce.invoke(core.clj:6519)
     [java] 	at clojure.core$into.invoke(core.clj:6600)
     [java] 	at clojure.test.generative.runner$eval559$fn__560$fn__561$fn__562.invoke(runner.clj:66)
     [java] 	at clojure.core$repeatedly$fn__5111.invoke(core.clj:4921)
     [java] 	at clojure.lang.LazySeq.sval(LazySeq.java:40)
     [java] 	at clojure.lang.LazySeq.seq(LazySeq.java:49)
     [java] 	at clojure.lang.RT.seq(RT.java:507)
     [java] 	at clojure.lang.RT.nthFrom(RT.java:924)
     [java] 	at clojure.lang.RT.nth(RT.java:883)
     [java] 	at clojure.test.generative.runner$run_one$fn__588$fn__589$fn__590.invoke(runner.clj:94)
     [java] 	at clojure.test.generative.runner$run_one$fn__588$fn__589.invoke(runner.clj:93)
     [java] 	at clojure.core$binding_conveyor_fn$fn__4444.invoke(core.clj:1916)
     [java] 	at clojure.lang.AFn.call(AFn.java:18)
     [java] 	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
     [java] 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
     [java] 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
     [java] 	at java.lang.Thread.run(Thread.java:745)
     [java] Caused by: java.lang.ExceptionInInitializerError
     [java] 	at java.lang.Class.forName0(Native Method)
     [java] 	at java.lang.Class.forName(Class.java:344)
     [java] 	at clojure.lang.RT.classForName(RT.java:2154)
     [java] 	at clojure.lang.RT.classForName(RT.java:2163)
     [java] 	at clojure.lang.Compiler$HostExpr.maybeClass(Compiler.java:1017)
     [java] 	at clojure.lang.Compiler$HostExpr.access$600(Compiler.java:795)
     [java] 	at clojure.lang.Compiler$NewExpr$Parser.parse(Compiler.java:2609)
     [java] 	at clojure.lang.Compiler.analyzeSeq(Compiler.java:6738)
     [java] 	... 82 more
     [java] Caused by: java.io.FileNotFoundException: Could not locate clojure/generated/ns0__init.class or clojure/generated/ns0.clj on classpath.
     [java] 	at clojure.lang.RT.load(RT.java:449)
     [java] 	at clojure.lang.RT.load(RT.java:412)
     [java] 	at clojure.core$load$fn__5448.invoke(core.clj:5866)
     [java] 	at clojure.core$load.doInvoke(core.clj:5865)
     [java] 	at clojure.lang.RestFn.invoke(RestFn.java:408)
     [java] 	at clojure.core$load_one.invoke(core.clj:5671)
     [java] 	at clojure.core$load_lib$fn__5397.invoke(core.clj:5711)
     [java] 	at clojure.core$load_lib.doInvoke(core.clj:5710)
     [java] 	at clojure.lang.RestFn.applyTo(RestFn.java:142)
     [java] 	at clojure.core$apply.invoke(core.clj:632)
     [java] 	at clojure.core$load_libs.doInvoke(core.clj:5749)
     [java] 	at clojure.lang.RestFn.applyTo(RestFn.java:137)
     [java] 	at clojure.core$apply.invoke(core.clj:632)
     [java] 	at clojure.core$require.doInvoke(core.clj:5832)
     [java] 	at clojure.lang.RestFn.invoke(RestFn.java:408)
     [java] 	at clojure.generated.ns0.ExampleRecord-0.<clinit>(run_test_generative.clj:5)
     [java] 	... 90 more
Comment by Nicola Mometto [ 11/Jul/15 3:25 AM ]

Michael, that's already noted in the ticket description and the screened patch is v5.patch, not v5-no-opts.patch

Comment by Michael Blume [ 11/Jul/15 10:45 PM ]

Aha, missed that, sorry

Comment by Michael Blume [ 12/Jul/15 7:18 PM ]

The screened patch seems to break 'lein check' for clj-http

Compiling namespace clj-http.client
Exception in thread "main" java.lang.NoClassDefFoundError: IllegalName: compile__stub.clj_http.headers.clj-http.headers/HeaderMap, compiling:(clj_http/headers.clj:105:1)
	at clojure.lang.Compiler.analyzeSeq(Compiler.java:6749)
	at clojure.lang.Compiler.analyze(Compiler.java:6543)
	at clojure.lang.Compiler.analyze(Compiler.java:6504)
	at clojure.lang.Compiler$BodyExpr$Parser.parse(Compiler.java:5882)
	at clojure.lang.Compiler$LetExpr$Parser.parse(Compiler.java:6198)
	at clojure.lang.Compiler.analyzeSeq(Compiler.java:6742)
	at clojure.lang.Compiler.analyze(Compiler.java:6543)
	at clojure.lang.Compiler.analyze(Compiler.java:6504)
	at clojure.lang.Compiler$BodyExpr$Parser.parse(Compiler.java:5884)
	at clojure.lang.Compiler$FnMethod.parse(Compiler.java:5315)
	at clojure.lang.Compiler$FnExpr.parse(Compiler.java:3926)
	at clojure.lang.Compiler.analyzeSeq(Compiler.java:6740)
	at clojure.lang.Compiler.analyze(Compiler.java:6543)
	at clojure.lang.Compiler.eval(Compiler.java:6798)
	at clojure.lang.Compiler.eval(Compiler.java:6790)
	at clojure.lang.Compiler.load(Compiler.java:7246)
	at clojure.lang.RT.loadResourceScript(RT.java:371)
	at clojure.lang.RT.loadResourceScript(RT.java:362)
	at clojure.lang.RT.load(RT.java:446)
	at clojure.lang.RT.load(RT.java:412)
	at clojure.core$load$fn__5448.invoke(core.clj:5866)
	at clojure.core$load.doInvoke(core.clj:5865)
	at clojure.lang.RestFn.invoke(RestFn.java:408)
	at clojure.core$load_one.invoke(core.clj:5671)
	at clojure.core$load_lib$fn__5397.invoke(core.clj:5711)
	at clojure.core$load_lib.doInvoke(core.clj:5710)
	at clojure.lang.RestFn.applyTo(RestFn.java:142)
	at clojure.core$apply.invoke(core.clj:632)
	at clojure.core$load_libs.doInvoke(core.clj:5749)
	at clojure.lang.RestFn.applyTo(RestFn.java:137)
	at clojure.core$apply.invoke(core.clj:632)
	at clojure.core$require.doInvoke(core.clj:5832)
	at clojure.lang.RestFn.invoke(RestFn.java:482)
	at clj_http.core$eval237$loading__5340__auto____238.invoke(core.clj:1)
	at clj_http.core$eval237.invoke(core.clj:1)
	at clojure.lang.Compiler.eval(Compiler.java:6801)
	at clojure.lang.Compiler.eval(Compiler.java:6790)
	at clojure.lang.Compiler.load(Compiler.java:7246)
	at clojure.lang.RT.loadResourceScript(RT.java:371)
	at clojure.lang.RT.loadResourceScript(RT.java:362)
	at clojure.lang.RT.load(RT.java:446)
	at clojure.lang.RT.load(RT.java:412)
	at clojure.core$load$fn__5448.invoke(core.clj:5866)
	at clojure.core$load.doInvoke(core.clj:5865)
	at clojure.lang.RestFn.invoke(RestFn.java:408)
	at clojure.core$load_one.invoke(core.clj:5671)
	at clojure.core$load_lib$fn__5397.invoke(core.clj:5711)
	at clojure.core$load_lib.doInvoke(core.clj:5710)
	at clojure.lang.RestFn.applyTo(RestFn.java:142)
	at clojure.core$apply.invoke(core.clj:632)
	at clojure.core$load_libs.doInvoke(core.clj:5749)
	at clojure.lang.RestFn.applyTo(RestFn.java:137)
	at clojure.core$apply.invoke(core.clj:632)
	at clojure.core$require.doInvoke(core.clj:5832)
	at clojure.lang.RestFn.invoke(RestFn.java:805)
	at clj_http.client$eval83$loading__5340__auto____84.invoke(client.clj:1)
	at clj_http.client$eval83.invoke(client.clj:1)
	at clojure.lang.Compiler.eval(Compiler.java:6801)
	at clojure.lang.Compiler.eval(Compiler.java:6790)
	at clojure.lang.Compiler.load(Compiler.java:7246)
	at clojure.lang.RT.loadResourceScript(RT.java:371)
	at clojure.lang.RT.loadResourceScript(RT.java:362)
	at clojure.lang.RT.load(RT.java:446)
	at clojure.lang.RT.load(RT.java:412)
	at clojure.core$load$fn__5448.invoke(core.clj:5866)
	at clojure.core$load.doInvoke(core.clj:5865)
	at clojure.lang.RestFn.invoke(RestFn.java:408)
	at user$eval60$fn__71.invoke(form-init8647608528776892632.clj:1)
	at user$eval60.invoke(form-init8647608528776892632.clj:1)
	at clojure.lang.Compiler.eval(Compiler.java:6801)
	at clojure.lang.Compiler.eval(Compiler.java:6791)
	at clojure.lang.Compiler.load(Compiler.java:7246)
	at clojure.lang.Compiler.loadFile(Compiler.java:7184)
	at clojure.main$load_script.invoke(main.clj:275)
	at clojure.main$init_opt.invoke(main.clj:280)
	at clojure.main$initialize.invoke(main.clj:308)
	at clojure.main$null_opt.invoke(main.clj:343)
	at clojure.main$main.doInvoke(main.clj:421)
	at clojure.lang.RestFn.invoke(RestFn.java:421)
	at clojure.lang.Var.invoke(Var.java:383)
	at clojure.lang.AFn.applyToHelper(AFn.java:156)
	at clojure.lang.Var.applyTo(Var.java:700)
	at clojure.main.main(main.java:37)
Caused by: java.lang.NoClassDefFoundError: IllegalName: compile__stub.clj_http.headers.clj-http.headers/HeaderMap
	at java.lang.ClassLoader.preDefineClass(ClassLoader.java:654)
	at java.lang.ClassLoader.defineClass(ClassLoader.java:758)
	at java.lang.ClassLoader.defineClass(ClassLoader.java:642)
	at clojure.lang.DynamicClassLoader.defineClass(DynamicClassLoader.java:46)
	at clojure.lang.Compiler$NewInstanceExpr.compileStub(Compiler.java:7767)
	at clojure.lang.Compiler$NewInstanceExpr.build(Compiler.java:7632)
	at clojure.lang.Compiler$NewInstanceExpr$DeftypeParser.parse(Compiler.java:7542)
	at clojure.lang.Compiler.analyzeSeq(Compiler.java:6742)
	... 82 more
Failed.

Will try to create a minimal failing repo.

Comment by Michael Blume [ 12/Jul/15 7:31 PM ]

Looks like it just breaks potemkin's def-map-type.

Comment by Nicola Mometto [ 13/Jul/15 8:47 AM ]

This is a bug in potemkin caused by the fact that it assumes deftype expands to an unqualified second argument.
The deftype* special form takes as first argument the tagsym and as a second argument the class name, potemkin passes a wrong second argument, it expands to :

(deftype* user/Foo user.user/Foo ..)

rather than

(deftype* user/Foo user.Foo ..)

I'll open a PR fixing this issue in potemkin and link back to this ticket.

Comment by Nicola Mometto [ 13/Jul/15 1:15 PM ]

Opened PR: https://github.com/ztellman/potemkin/pull/40





[CLJ-1157] Classes generated by gen-class aren't loadable from remote codebase for mis-implementation of static-initializer Created: 04/Feb/13  Updated: 31/Jul/15  Resolved: 31/Jul/15

Status: Closed
Project: Clojure
Component/s: None
Affects Version/s: Release 1.4, Release 1.5, Release 1.6
Fix Version/s: Release 1.8

Type: Defect Priority: Minor
Reporter: Tsutomu Yano Assignee: Unassigned
Resolution: Completed Votes: 1
Labels: gen-class
Environment:

Tested on Mac OS X 10.9.1 and Oracle JVM 1.7.0_51 with Clojure 1.6 master SNAPSHOT


Attachments: File 20140121_fix_classloader.diff     File clj-1157-v2.diff    
Patch: Code
Approval: Ok

 Description   

When a genclass'ed object is serialized and sent to a remote system, the remote system throws an exception loading the object:

Exception in thread "main" java.lang.ExceptionInInitializerError
        at java.io.ObjectStreamClass.hasStaticInitializer(Native Method)
        at java.io.ObjectStreamClass.computeDefaultSUID(ObjectStreamClass.java:1723)
        at java.io.ObjectStreamClass.access$100(ObjectStreamClass.java:69)
        at java.io.ObjectStreamClass$1.run(ObjectStreamClass.java:247)
        at java.io.ObjectStreamClass$1.run(ObjectStreamClass.java:245)
        at java.security.AccessController.doPrivileged(Native Method)
        at java.io.ObjectStreamClass.getSerialVersionUID(ObjectStreamClass.java:244)
        at java.io.ObjectStreamClass.initNonProxy(ObjectStreamClass.java:600)
        at java.io.ObjectInputStream.readNonProxyDesc(ObjectInputStream.java:1601)
        at java.io.ObjectInputStream.readClassDesc(ObjectInputStream.java:1514)
        at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1750)
        at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1347)
        at java.io.ObjectInputStream.readObject(ObjectInputStream.java:369)
        at sun.rmi.server.UnicastRef.unmarshalValue(UnicastRef.java:324)
        at sun.rmi.server.UnicastRef.invoke(UnicastRef.java:173)
        at java.rmi.server.RemoteObjectInvocationHandler.invokeRemoteMethod(RemoteObjectInvocationHandler.java:194)
        at java.rmi.server.RemoteObjectInvocationHandler.invoke(RemoteObjectInvocationHandler.java:148)
        at $Proxy0.getResult(Unknown Source)
        at client.SampleClient$_main.doInvoke(SampleClient.clj:12)
        at clojure.lang.RestFn.invoke(RestFn.java:397)
        at clojure.lang.AFn.applyToHelper(AFn.java:159)
        at clojure.lang.RestFn.applyTo(RestFn.java:132)
        at client.SampleClient.main(Unknown Source)
 Caused by: java.io.FileNotFoundException: Could not locate remoteserver/SampleInterfaceImpl__init.class or remoteserver/SampleInterfaceImpl.clj on classpath: 
        at clojure.lang.RT.load(RT.java:434)
        at clojure.lang.RT.load(RT.java:402)
        at clojure.core$load$fn__5039.invoke(core.clj:5520)
        at clojure.core$load.doInvoke(core.clj:5519)
        at clojure.lang.RestFn.invoke(RestFn.java:408)
        at clojure.lang.Var.invoke(Var.java:415)
        at remoteserver.SampleInterfaceImpl.<clinit>(Unknown Source)
        ... 23 more

Reproduce:

// build
git clone git://github.com/tyano/clojure_genclass_fix.git
cd clojure_genclass_fix
sh build.sh
// start rmiregistry
rmiregistry -J-Djava.rmi.server.useCodebaseOnly=false &
// start server
cd remoteserver
sh start.sh
// Start client
cd ../client
sh start.sh

Cause:

A gen-classed class (in this case, SampleInterfaceImpl.class) uses a static-initializer for loading SampleInterfaceImpl__init.class like:

static
  {
    RT.var("clojure.core", "load").invoke("/remoteserver/SampleInterfaceImpl");
  }

RT.load in default uses a context-classloader for loading __init.class but all classes depending on a gen-classed class must be loaded from the same classloader. In this case, RT.load must use a remote URLClassLoader which loads the main class.

Proposed:

Instead produce the equivalent of this in the static initializer:

static
  {
    Var.pushThreadBindings(RT.map(new Object[] { Compiler.LOADER, SampleInterfaceImpl.class.getClassLoader() }));
    try {
        RT.var("clojure.core", "load").invoke("/remoteserver/SampleInterfaceImpl");
    }
    finally 
    {
        Var.popThreadBindings();
    }
  }

With this code, RT.load will uses a same classloader which load SampleInterfaceImpl.class.

Patch: clj-1157-v2.diff

Screened by:



 Comments   
Comment by Stuart Halloway [ 01/Mar/13 10:20 AM ]

This sounds reasonable, but anything touching classloaders must be considered very carefully.

Comment by Stuart Halloway [ 01/Mar/13 12:12 PM ]

It seems overly complex to have the patch do so much code generation. Why not implement a method that does this job, and have the generated code call that?

Comment by Andy Fingerhut [ 11/Jan/14 2:47 PM ]

Patch 20130204_fix_classloader.diff dated Feb 3, 2013 no longer applies cleanly as of the latest commits to Clojure master on Jan 11, 2014. The only conflict in applying the patch appears to be in the file src/jvm/clojure/asm/commons/GeneratorAdapter.java. This is probably due to the commit for ticket CLJ-713 that was committed today, updating the ASM library.

Comment by Tsutomu Yano [ 21/Jan/14 3:01 AM ]

I put a new patch applicable on the latest master branch.
This new patch is simpler and robust because the code-generation becomes very simple. Now It just call a method implemented with Java.

And I fixed my sample program and the 'HOW TO REPRODUCT THIS ISSUE' section of this ticket, because old description is not runnable on newest JVM. It is because the specification of remote method call of the newest JVM was changed from the old one. In the newest JVM, we need a 'java.rmi.server.useCodebaseOnly=false' option for making the behavior of remote call same as old one.
pull the newest sample.

Comment by Alex Miller [ 25/Sep/14 10:16 AM ]

Added almost the same patch that does not have whitespace error.





[CLJ-1137] Metadata on a def gets evaluated twice Created: 21/Dec/12  Updated: 31/Jul/15  Resolved: 31/Jul/15

Status: Closed
Project: Clojure
Component/s: None
Affects Version/s: None
Fix Version/s: Release 1.8

Type: Defect Priority: Minor
Reporter: Ghadi Shayban Assignee: Unassigned
Resolution: Completed Votes: 2
Labels: compiler, ft, meta

Attachments: File CLJ-1137-eval-metadata-once.diff    
Patch: Code
Approval: Ok

 Description   

Metadata on the symbol of a def special form is evaluated twice.

(def ^{:foo (println "HA")} a [])

prints out HA HA. Offending line is in Compiler$DefExpr, fixed.

Patch: CLJ-1137-eval-metadata-once.diff

Screened by: Alex Miller






[CLJ-1134] star-directive in clojure.pprint/cl-format with an at-prefix ("~n@*") do not obey its specifications Created: 18/Dec/12  Updated: 31/Jul/15  Resolved: 31/Jul/15

Status: Closed
Project: Clojure
Component/s: None
Affects Version/s: Release 1.4, Release 1.5
Fix Version/s: Release 1.8

Type: Defect Priority: Minor
Reporter: Jean Niklas L'orange Assignee: Unassigned
Resolution: Completed Votes: 0
Labels: ft, print

Attachments: Text File clj-1134-star-directive-in-cl-format.txt    
Patch: Code and Test
Approval: Ok

 Description   

The star-directive in clojure.pprint/cl-format with an at-prefix (~n@*) does not obey its specifications according to Common Lisp the Language, 2nd Edition. There are two bugs within ~n@* as of right now:

  1. When ~n@* is supposed to jump forward over more than one argument, it jumps one step backward as if it had seen ~:*. For instance, (cl-format nil "~D ~3@*~D" 0 1 2 3) will return "0 0" and not "0 3" as expected.
  2. When ~@* is seen, the formatter is supposed to jump to the first argument (as n defaults to 0, see specification linked above). However, whenever a ~@*-directive is seen, the formatter jumps to the second argument instead.

Inside a clean Clojure repl, perform these steps:

user=> (require '[clojure.pprint :refer [cl-format]])
nil
user=> (cl-format nil "~D ~3@*~D" 0 1 2 3)
"0 0"                                           ;; Expected: "0 3"
user=> (cl-format nil "~D~D~D~D ~@*~D" 0 1 2 3)
"0123 1"                                        ;; Expected: "0123 0"

The format strings which reproduce the problem has been compared with the format function from the Common Lisp implementations SBCL, CLisp and Clozure. All of them print the expected output.

Patch: clj-1134-star-directive-in-cl-format.txt

Screened by: Alex Miller



 Comments   
Comment by Jean Niklas L'orange [ 18/Dec/12 9:28 PM ]

Patch attached.

It may be easier to read the changes the patch does from within JIRA instead from the commit message, so I've added it here:

This solves two issues as specified by #CLJ-1134. Issue #1 is solved by doing a
relative jump forward within absolute-reposition in cl_format.clj, line 114 by
switching (- (:pos navigator) position) with (- position (:pos navigator)).

Issue #2 is handled by changing the default n-parameter to * depending on
whether the @-prefix is placed or not. If it is placed, then n defaults to
0, otherwise it defaults to 1.

In addition, new tests have been appended to test_cl_format.clj to ensure the
correctness of this patch. The tests have been tested on the Common Lisp
implementation GNU CLISP 2.49, which presumably handle the ~n@*
correctly. This patch and GNU CLISP returns the same output for each format
call, sans case for printed symbols; Common Lisp has case-insensitive symbols,
whereas Clojure has not.

Comment by Tom Faulhaber [ 14/Apr/14 11:12 AM ]

I walked through this patch and it looks just right. Thanks!

Let's get it applied for 1.7.

Comment by Jean Niklas L'orange [ 23/Jun/15 10:53 AM ]

I'm assuming this will not get in 1.7 as the RC2 is out right now, but I wish it could be prioritised into 1.8.

As it is a triaged bugfix that applies cleanly, I'm not sure there's anything more I can do here. But if there is, don't hesitate to ask for it.

Comment by Alex Miller [ 23/Jun/15 11:20 AM ]

Pre-screened for next release.





[CLJ-1060] 'list*' returns not a list Created: 03/Sep/12  Updated: 31/Jul/15  Resolved: 31/Jul/15

Status: Closed
Project: Clojure
Component/s: None
Affects Version/s: Release 1.4
Fix Version/s: Release 1.8

Type: Defect Priority: Trivial
Reporter: Andrei Zhlobich Assignee: Unassigned
Resolution: Completed Votes: 0
Labels: ft

Attachments: File list-star-docstring.diff     File list-star-fix.diff    
Patch: Code
Approval: Ok

 Description   

Function 'list*' returns sequence, but not a list.
It is a bit confusing.

user=> (list? (list* 1 '(2 3)))
false

Approach: Change the doc string to say that it returns a seq, not a list.

Patch: list-star-docstring.diff



 Comments   
Comment by Stuart Halloway [ 17/Sep/12 6:52 AM ]

should the docstring for list* change to say it returns a seq?

Comment by Timothy Baldridge [ 27/Nov/12 11:58 AM ]

Is there a reason why we can't have list* actually return a list? The cost of creating a list vs a cons is negligible.

Comment by Marek Srank [ 04/Jan/13 2:02 PM ]

The question is what to do with the one-argument case of list*, because in cases like: (list* {:a 1 :b 2}) it doesn't return IPersistentList as well. I propose just applying 'list'.

I added patch 'list-star-fix.diff' (dated 04/Jan/2013) with Cons implementing IPersistentList and doing (apply list args) in one-argument case of list*. To be able to use 'apply' in list* I had to declare it before the definition of list* in the source code. The apply itself also uses list*, but luckily not the one-argument version of list*, so it should be safe... The patch also contains simple unit tests.

Discussion is also here: https://groups.google.com/forum/#!topic/clojure/co8lcKymfi8

Comment by Michał Marczyk [ 04/Jan/13 4:11 PM ]

(apply list args) would make (list* (range)) hang, where currently it returns a partially-realized lazy seq. Also, even for non-lazy seqs – possibly lists themselves – it would always build a new list from scratch, right?

Also, if I'm reading the patch correctly, it would make 2+-ary list* overloads and cons return lists – that is, IPersistentList instances – always (Conses would now be lists), but repeatedly calling next on such a list might eventually produce a non-list. The only way around that would be to make all seqs into IPersistentLists – that doesn't seem desirable at first glance...?

On a side note, for the case where the final "args" argument is in fact a list, we already have a "prepend many items" function, namely conj. list* currently acts as the vararg variant of cons (in line with Lisp tradition); I'm actually quite happy with that.

Comment by Michał Marczyk [ 04/Jan/13 4:19 PM ]

I'm in favour of the "list" -> "seq" tweak to the docstring though, assuming impl remains unchanged.

Comment by Marek Srank [ 04/Jan/13 6:13 PM ]

Yep, these are all valid points, thanks! I see this as a question whether the list* function is a list constructor or not. If yes (and it would be possible to implement it in a satisfactory way), it should probably return a list.

We could avoid building a new list by sth like:

(if (list? args)
  args
  (apply list args))

(btw, 'vec' also creates a new vector even when the argument itself is a vector)

The contract of next seems to be to return a seq, not a list - its docstring reads: "Returns a seq of the items after the first. Calls seq on its argument. If there are no more items, returns nil."

Btw, in some Lisp/Scheme impls I checked, cons seems to be a list as well. E.g. in CLisp (and similar in Guile and Racket):

> (listp (cons 2 '()))
T
> (listp (list* 1 2 '()))
T
Comment by Steve Miner [ 26/Feb/15 3:11 PM ]

I bump into this every once in a while and it bothers my pedantic side.

I think it's too late to change the implementation of `list*`. There's a risk of breaking existing code (dealing with lazy-seqs, etc.) It would be good to change the doc string to say it returns a seq, not a list.

But the real issue is the name of the function implies that it will return a list. You could deprecate `list*` (but keep it forever for backwards compatibility.) A better name for the same implementation might be `seq*`.

Comment by Andy Fingerhut [ 02/Jul/15 3:13 PM ]

To Andy Sheldon, author of patch list-star-docstring.diff: Clojure only accepts patches written by those who have signed a contributor agreement. If you were interested in doing that, details on how are here: http://clojure.org/contributing

Comment by Andy Sheldon [ 02/Jul/15 8:34 PM ]

@AndyFingerhut - Thanks for the reminder. Signed, sealed, delivered.





[CLJ-703] Improve writeClassFile performance Created: 04/Jan/11  Updated: 17/Jul/15  Resolved: 17/Jul/15

Status: Closed
Project: Clojure
Component/s: None
Affects Version/s: Release 1.3
Fix Version/s: Release 1.8

Type: Enhancement Priority: Critical
Reporter: Jürgen Hötzel Assignee: Unassigned
Resolution: Completed Votes: 27
Labels: Compiler, ft, performance

Attachments: Text File 0001-use-File.mkdirs-instead-of-mkdir-every-single-direct.patch     Text File 0002-Ensure-atomic-creation-of-class-files-by-renaming-a-.patch     Text File clj-703-remove-sync-in-write-class-file.patch     Text File improve-writeclassfile-perf.patch     Text File remove-flush-and-sync-only.patch     Text File remove-sync-only.patch    
Patch: Code
Approval: Ok

 Description   

When writing class files to disk, Clojure currently calls flush on the FileOutputStream used, and sync on the underlying FileDescriptor for each class file it writes. This ticket proposes to remove the sync call as it provides little value, but has a detrimental effect on performance on some operating systems.

Background

The call to sync in the current implementation of Compiler.writeClassFile was added in SVN-1252 to address an issue where compile allegedly returned but changes to class files weren't visible[1]. The original report lacks detail and it's hard to discern what the actual issue was.

Calling FileDescriptor.sync tells the operating system to flush any buffered data to disk (to its best capability). This means that in the event of a system failure, the kernel buffers will not hold any data that hasn't been persisted to disk, and thus no data would be lost.

However, sync does not provide any special guarantees around visbility. Any data written using to a file descriptor that is subsequently closed is already guaranteed to be visible to all other processes, as soon as the close call returns. When not using sync, the data returned may reside only in kernel buffers, but the reading process will see no difference.

Syncing is an expensive operation and doing so after each class file is written can make compilation significantly slower.

Laurent Petit (who made the original request for this) reports that it would not make any difference for him now in CCW (https://groups.google.com/d/msg/clojure-dev/Vz9h8hqAvFk/cQvip5j_zoQJ).

Proposed solution

Remove the call to FileDescriptor.sync when writing class files to disk.

Latest patch: http://dev.clojure.org/jira/secure/attachment/14214/clj-703-remove-sync-in-write-class-file.patch

Performance implications

The performance gains of removing the sync call will vary depending on OS and file system. Some operating systems provide strong guarantees that your data will actually end up on the disk, others less so. See for example differences between fsync in Linux[2] and OSX[3].

Preliminary tests on Linux have shown compile times going from:

  • aleph.http: 13s -> 4s
  • incanter.core: 14s -> 4s

I've created a test for this and have asked the community for help testing and gathering data: http://goo.gl/forms/0b8SVt2pyN

Will update this ticket once more data is available.

Tradeoffs

In short, this change trades some degree of on-disk consistency for compilation speed. If a fault (process, OS, hardware) occurs whilst compiling, this patch may result in class files not being properly written to disk. This however, is still the case today, depending on the type of fault, the OS you're using and the fsync guarantees provided. If a compilation fails with a fatal error, it is unreasonable to expect that the output is in a consistent state.

[1]: https://groups.google.com/forum/#!topic/clojure/C-DEWzpPPUY
[2]: "Note that while fsync() will flush all data from the host to the drive (i.e. the "permanent storage device"), the drive itself may not physically write the data to the platters for quite some time and it may be written in an out-of-order sequence.": https://developer.apple.com/library/mac/documentation/Darwin/Reference/ManPages/man2/fsync.2.html
[3]: "[...] includes writing through or flushing a disk cache if present." http://man7.org/linux/man-pages/man2/fsync.2.html[2]:



 Comments   
Comment by David Powell [ 17/Jan/11 2:16 PM ]

Removing sync makes clojure build much faster. I wonder why it was added in the first place? I guess only Rich knows? I assume that it is not necessary.

If we are removing sync though, I wouldn't bother with the atomic rename stuff. Doing that sort of thing can cause problems on some platforms, eg with search indexers and virus checkers momentarily locking files after they are created.

The patch seems to be assuming that sync is there for some reason, but my initial assumption would be that sync isn't necessary - perhaps it was working around some issue that no longer exists?

Comment by Jürgen Hötzel [ 19/Jan/11 2:05 PM ]

Although its unlikely: there is a possible race condition "loading a paritally written classfile"?:

https://github.com/clojure/clojure/blob/master/src/jvm/clojure/lang/RT.java#L393

Comment by John Szakmeister [ 25/May/12 4:22 AM ]

The new improve-writeclassfile-perf version of the patch combines the two previous patches into a single patch file and brings them up-to-date with master. I can split the two changes back out into separate patch files if desired, but I figured out current tooling is more geared towards a single patch being applied.

Comment by John Szakmeister [ 25/May/12 4:36 AM ]

FWIW, both fixes look sane. The first one is a nice cleanup. The second one is a little more interesting in that uses a rename operation to put the final file into place. It removes the sync call, which does make things faster. In general, if we're concerned about on-disk consistency, we should really have a combination of the two: write the full contents to a tmp file, sync it, and atomically rename to the destination file name.

Neither the current master, nor the current patch will guarantee on-disk consistency across a machine wide crash. The current master could crash before the sync() occurs, leaving the file in an inconsistent state. With the patch, the OS may not get the file from file cache to disk before an OS level crash occurs, and leave the file in an inconsistent state as well. The benefit of the patch version is that the whole file does atomically come into view at once. It does have a nasty side effect of leaving around a temp file if the compiler crashes just before the rename though.

Perhaps a little more work to catch an exception and clean up is in order? In general, I like the patched version better.

Comment by Ivan Kozik [ 05/Oct/12 7:07 PM ]

File.renameTo returns false on (most?) errors, but the patch doesn't check for failure. Docs say "The return value should always be checked to make sure that the rename operation was successful." Failure might be especially likely on Windows, where files are opened by others without FILE_SHARE_DELETE.

Comment by Dave Della Costa [ 23/Jun/14 11:37 PM ]

We've been wondering why our compilation times on linux were so slow. It became the last straw when we walked away from one project and came back after 15 minutes and it was not done yet.

After some fruitless investigation into our linux configuration and lein java args, we stumbled upon this issue via the associated Clojure group thread. Upon commenting out the flush() and sync() lines (https://github.com/clojure/clojure/blob/master/src/jvm/clojure/lang/Compiler.java#L7171-L7172) and compiling Clojure 1.6 ourselves, our projects all started compiling in under a minute.

Point being, can we at least provide some flag to allow for "unsafe compilation" or something? As it is, this is bad enough that we've manually modified all our local versions of Clojure to work around the issue.

Comment by Tim McCormack [ 30/Sep/14 4:10 PM ]

Additional motivation: This becomes really unpleasant on an encrypted filesystem, since write and read latency become higher.

As a partial workaround, I've been using this script to mount a ramdisk over top of target, which speeds up compilation 2-4x: https://gist.github.com/timmc/6c397644668bcf41041f (but removing flush() and sync() entirely would probably speed things up even more, if safe)

Comment by Ragnar Dahlen [ 06/Oct/14 12:30 PM ]

I'd like to explore this issue further as I also don't think the flush and sync calls add any value, but do have a severe impact on performance.

To resurrect the discussion, I've attached a new patch with the following approach:

  • create a temporary file
  • write class bytecode to temporary file, no flush or sync
  • close temporary file
  • atomic rename of temporary file to class file name

It is different to previous patches in that:

  • it applies cleanly to master
  • it checks return value from File.renameTo
  • it omits proposed File.mkdirs change as the current implementation is actually converting from an "internal name", where forward slashes are assumed (splits on "/"), to a platform specific path using File.separator. I'm not convinced that the previous patch is safe on all versions of Windows, and I think it's separate from the main issue here.

I opted for the atomic rename of a temp file to avoid leaving empty class files with a correct expected class file name in case of failure.

It is my understanding that this patch will guarantee that:

  • when writeClassFile returns successfully, a class file with the expected name will exists, and subsequent reads from that file name will return the expected bytecode (possibly from kernel buffers).
  • when writeClassFile fails, a class file with the expected name will not have been created (or updated if it previously existed).

Anything preventing the operating system from flushing its buffers to disk (power failure etc) might leave a corrupt (empty, partially written) class file. It's my opinion that that is acceptable behaviour, and worth the performance improvement (I'm seeing AOT compilation reduced from 1m20s -> 22s on one of our codebases, would be happy to provide more benchmarks if requested).

Would be grateful for feedback/testing of this patch.

Comment by Ragnar Dahlen [ 08/Oct/14 6:00 AM ]

We're testing this patch on various projects/platforms at my company. So far we've seen:

  • Significantly reduced compilation times on Linux (two typical examples: 30s to 15s, 1m30s to 30s)
  • No significant change in compilation times on Mac OSX.
  • File.renameTo consistently failing on a Windows machine.

My understanding is that the performance difference between Linux and OSX is due to differences in how these platforms implement fsync. OSX by default does not actually tell the drive to flush its buffers (requires fcntl F_FULLSYNC for this, not used by JVMs) [1], whereas Linux does [2].

Our very limited test shows (as was previously pointed out) that File.renameTo is problematic on Windows. I've attached a new patch that doesn't use rename, and only has the the sync call removed (flush is a no-op for FileOutputStreams). We're currently testing this patch.

The drawback of this patch is that it may leave correctly named, but empty class files if the write fails. One option would be to try and delete the file in the catch block. Personally, I wouldn't expect a compilation that failed because of OS/IO reasons to leave my classfiles in a consistent state.

[1]: "Note that while fsync() will flush all data from the host to the drive (i.e. the "permanent storage device"), the drive itself may not physically write the data to the platters for quite some time and it may be written in an out-of-order sequence.": https://developer.apple.com/library/mac/documentation/Darwin/Reference/ManPages/man2/fsync.2.html

[2]: "[...] includes writing through or flushing a disk cache if present."
http://man7.org/linux/man-pages/man2/fsync.2.html

Comment by Alex Miller [ 17/May/15 8:24 AM ]

Currently this ticket is not a state where it can be evaluated, and I would like it to get there before we consider tickets for 1.8.

The description for this ticket needs to be something that a screener read to fully understand the problem, proposed solution, performance implications, and tradeoffs. Right now it does not seem up to date with information discussed in comments, which patch should be considered, performance data, and what we might lose by making the change. It would be great if someone could help get this in shape.

Comment by Andy Fingerhut [ 17/May/15 10:16 AM ]

Alex, you will probably get a wider audience of contributors willing to help if you sent an email to clojure-dev every once in a while with a list of tickets you want help updating. Right now I think it is this one and CLJ-1449, but it would be nice if there were a single report link contributors could click on to find out what you are looking for help with.

Normally there is the "Needs Patch" state for the next release, but for CLJ-1449 you are looking for a patch for the next-next release, so "Needs Patch" won't do it.

Perhaps if there were 'standard' labels created for 'Alex requests ticket updates' for such tickets, and filter for them?

Comment by Ragnar Dahlen [ 17/May/15 10:47 AM ]

I'd be happy to help improve the state of this ticket. I'm not the original reporter, is it still OK for me to change description etc. to address your concerns?

Comment by Alex Miller [ 17/May/15 11:02 AM ]

Andy - I will probably do so soon but I thought previous commenters would see this now.

Ragnar - absolutely!

Comment by Ragnar Dahlen [ 30/May/15 6:24 PM ]

Updated for RC1

Comment by Ragnar Dahlen [ 30/May/15 6:25 PM ]

Alex - Updated description, please let me know if I can improve it further.

Comment by Alex Miller [ 30/May/15 8:49 PM ]

Ragnar - much better - thank you! Could you also add a pointer to the patch for consideration?

Comment by Ragnar Dahlen [ 31/May/15 3:40 AM ]

Alex - Done





[CLJ-130] Namespace metadata lost in AOT compile Created: 19/Jun/09  Updated: 31/Jul/15  Resolved: 31/Jul/15

Status: Closed
Project: Clojure
Component/s: None
Affects Version/s: None
Fix Version/s: Release 1.8

Type: Defect Priority: Major
Reporter: Stuart Sierra Assignee: Unassigned
Resolution: Completed Votes: 5
Labels: aot, metadata

Attachments: Text File 0001-CLJ-130-preserve-metadata-for-AOT-compiled-namespace.patch     File aot-drops-metadata-demo.sh     File aot-drops-metadata-demo.sh     File aot-drops-metadata-demo-w-direct-linkage.sh    
Patch: Code
Approval: Ok

 Description   

AOT-compilation drops namespace metadata.

This also affects all of the namespaces packaged with Clojure, except clojure.core, for which metadata is explicitly added in core.clj.

Cause of the bug:

  • a namespace inherits the metadata of the symbol used to create that namespace the first time
  • the namespace is created in the load() method, that is invoked after the __init() method
  • the __init0() method creates all the Vars of the namespace
  • interning a Var in a namespace that doesn't exist forces that namespace to be created

This means that the namespace will have been already created (with nil metadata) by the time the load() method gets invoked and thus the call to in-ns will be a no-op and the metadata will be lost.

Approach: The attached patch fixes this issue by explicitely attaching the metadata to the namespace after its creation (via ns) using a .resetMeta call
Patch: 0001-CLJ-130-preserve-metadata-for-AOT-compiled-namespace.patch
Screened by:



 Comments   
Comment by Assembla Importer [ 24/Aug/10 6:45 AM ]

Converted from http://www.assembla.com/spaces/clojure/tickets/130

Comment by Assembla Importer [ 24/Aug/10 6:45 AM ]

richhickey said: Updating tickets (#127, #128, #129, #130)

Comment by Assembla Importer [ 24/Aug/10 6:45 AM ]

juergenhoetzel said: This is still a issue on

Clojure 1.2.0-master-SNAPSHOT

Any progress, hints? I prefer interactive documentiation via slime/repl

Comment by Howard Lewis Ship [ 09/Sep/14 9:44 AM ]

This is of great concern to me, as the Rook web services framework we're building depends on availability of namespace metadata at runtime.

Comment by Howard Lewis Ship [ 09/Sep/14 9:53 AM ]

BTW, I verified that this still exists in 1.6.0.

Comment by Howard Lewis Ship [ 09/Sep/14 10:11 AM ]

For me personally, I would raise the priority of this issue. And I think in general, anything that works differently with AOT vs. non-AOT should be major, if not blocker, priority.

Comment by Howard Lewis Ship [ 09/Sep/14 10:25 AM ]

Alex Miller:

@hlship I think the question is where it would go. note no one has suggested a solution in last 5 yrs.

Alas, I have not delved into the AOT compilation code (since, you know, I value my sanity). But it seems to me like the __init class for the namespace could construct the map and update the Namespace object.

Comment by Howard Lewis Ship [ 09/Sep/14 4:27 PM ]

Just playing with javap, I can see that the meta data is being assembled in some way, so it's a question of why it is not accessible ...

  public static void __init0();
    Code:
       0: ldc           #108                // String clojure.core
       2: ldc           #110                // String in-ns
       4: invokestatic  #116                // Method clojure/lang/RT.var:(Ljava/lang/String;Ljava/lang/String;)Lclojure/lang/Var;
       7: checkcast     #12                 // class clojure/lang/Var
      10: putstatic     #10                 // Field const__0:Lclojure/lang/Var;
      13: aconst_null
      14: ldc           #118                // String fan.auth
      16: invokestatic  #122                // Method clojure/lang/Symbol.intern:(Ljava/lang/String;Ljava/lang/String;)Lclojure/lang/Symbol;
      19: checkcast     #124                // class clojure/lang/IObj
      22: iconst_4
      23: anewarray     #4                  // class java/lang/Object
      26: dup
      27: iconst_0
      28: aconst_null
      29: ldc           #126                // String meta-foo
      31: invokestatic  #130                // Method clojure/lang/RT.keyword:(Ljava/lang/String;Ljava/lang/String;)Lclojure/lang/Keyword;
      34: aastore
      35: dup
      36: iconst_1
      37: aconst_null
      38: ldc           #132                // String meta-bar
      40: invokestatic  #130                // Method clojure/lang/RT.keyword:(Ljava/lang/String;Ljava/lang/String;)Lclojure/lang/Keyword;
      43: aastore
      44: dup
      45: iconst_2
      46: aconst_null
      47: ldc           #134                // String doc
      49: invokestatic  #130                // Method clojure/lang/RT.keyword:(Ljava/lang/String;Ljava/lang/String;)Lclojure/lang/Keyword;
      52: aastore
      53: dup
      54: iconst_3
      55: ldc           #136                // String Defines the resources for the authentication service.
      57: aastore
      58: invokestatic  #140                // Method clojure/lang/RT.map:([Ljava/lang/Object;)Lclojure/lang/IPersistentMap;
      61: checkcast     #64                 // class clojure/lang/IPersistentMap
      64: invokeinterface #144,  2          // InterfaceMethod clojure/lang/IObj.withMeta:(Lclojure/lang/IPersistentMap;)Lclojure/lang/IObj;

If I'm reading the code correctly, a Symbol named after the namespace is interned, and the meta-data for the namespace is applied to the symbol, so it's just a question of commuting that meta data to the Namespace object. I must be missing something.

Comment by Nicola Mometto [ 30/Sep/14 6:45 PM ]

Attached patch fixes this issue by explicitely attaching the metadata to the namespace after its creation using a .resetMeta call.

Comment by Nicola Mometto [ 30/Sep/14 7:46 PM ]

Here's an explaination of why this bug happens:

  • a namespace inherits the metadata of the symbol used to create that namespace the first time
  • the namespace is created in the load() method, that is invoked after the __init() method
  • the __init0() method creates all the Vars of the namespace
  • interning a Var in a namespace that doesn't exist forces that namespace to be created

This means that the namespace will have been already created (with nil metadata) by the time the load() method gets invoked and thus the call to in-ns will be a no-op and the metadata will be lost.

Comment by Fogus [ 31/Jul/15 11:06 AM ]

Added new test script that works with 1.8.0-master. Also, added a script that will enable direct linkage before testing.

Comment by Fogus [ 31/Jul/15 11:08 AM ]

Tested with the latest 1.8.0-master source and ran modified versions of the test script to ensure that the patch worked with and without direct linking.





Generated at Mon Aug 31 14:49:58 CDT 2015 using JIRA 4.4#649-r158309.