<< Back to previous view

[CLJ-1550] Classes generated by deftype and defrecord don't play nice with .getPackage Created: 07/Oct/14  Updated: 07/Oct/14

Status: Open
Project: Clojure
Component/s: None
Affects Version/s: Release 1.6
Fix Version/s: None

Type: Defect Priority: Major
Reporter: Bozhidar Batsov Assignee: Unassigned
Resolution: Unresolved Votes: 0
Labels: bug


 Description   
(.getPackage String)
;; => #<Package package java.lang, Java Platform API Specification, version 1.7>
(deftype T [])
(.getPackage T)
;; => nil

This seems like a bug to me as it's not obvious why the class generated by deftype should exhibit different behaviour.



 Comments   
Comment by Alex Miller [ 07/Oct/14 8:54 AM ]

According to http://docs.oracle.com/javase/7/docs/api/java/lang/Class.html#getPackage() this method returns the package information found by the class loader or null if there is none. Its not clear to me that the current behavior is wrong per the spec. I would need to experiment more to see if this is unusual or not.

Comment by Bozhidar Batsov [ 07/Oct/14 9:05 AM ]

A bit of background for the issue. I'm no expert on the topic, but being able to procure all the class information except its package definitely looks strange to me.

Comment by Kevin Downey [ 07/Oct/14 11:46 AM ]

if you AOT compile(generate a class file on disk for a deftype), getPackage works fine, which suggests to me it is a jvm issue

Comment by Kevin Downey [ 07/Oct/14 11:49 AM ]

actually, it must just be that dynamicclassloader doesn't define a package for classes it loads

Comment by Alex Miller [ 07/Oct/14 12:13 PM ]

Yep, I believe that's correct.





[CLJ-1541] System/getProperty "user.dir" gives wrong output Created: 30/Sep/14  Updated: 30/Sep/14  Resolved: 30/Sep/14

Status: Closed
Project: Clojure
Component/s: None
Affects Version/s: Release 1.6
Fix Version/s: None

Type: Defect Priority: Major
Reporter: Khuram U. Khalid Assignee: Unassigned
Resolution: Not Reproducible Votes: 0
Labels: System/getProperty, bug, user.dir
Environment:

Windows 8.1 - Java version 1.8.0 - Clojure 1.6.0 - IntelliJ IDEA - Maven 3.0.5



 Description   

;; For example if current project is in C:\Projects\My Project
;; Following gives...

(ns my.project.com.core)
(defn -main [] (println (System/getProperty "user.dir"))

;;=> C:\Projects\My Project\src\main\my\project\com

;; While when same Clojure code is run from a Java project gives...
public static void main(String[] args) { my.project.com.core.main(); }
;;=> C:\Projects\My Project

Expected same behavior and hence correct output in Clojure.



 Comments   
Comment by Alex Miller [ 30/Sep/14 9:18 AM ]

I tried this on a simple project at the command line and saw no difference in behavior between Java and Clojure. Clojure does not modify the user.dir system property and you are calling directly into the System class, just like Java does, so the only difference is in how your environment is configured when running in these two contexts.

It's possible that your environment (IDEA) is configuring Java and Clojure projects differently, but you should ask on the mailing list or issue tracker for the tool.





[CLJ-1531] inc always warns when *unchecked-math* is set Created: 23/Sep/14  Updated: 23/Sep/14  Resolved: 23/Sep/14

Status: Closed
Project: Clojure
Component/s: None
Affects Version/s: Release 1.7
Fix Version/s: None

Type: Defect Priority: Major
Reporter: Pierre-Yves Ritschard Assignee: Pierre-Yves Ritschard
Resolution: Not Reproducible Votes: 0
Labels: bug, errormsgs


 Description   

While testing 1.7-alpha2 I stumbled this (affects clojure.data.codec amongst others). inc inlines a call to clojure.lang.Numbers's inc method which according to the rules of CLJ-1325 will warn.

I can't find a way around it for now, except maybe having a primitive-inc and primitive-dec java method which would be inlined in that case.

Happy to work on a patch but would prefer discussing it first.



 Comments   
Comment by Nicola Mometto [ 23/Sep/14 12:42 PM ]

I cannot reproduce this:

Clojure 1.7.0-master-SNAPSHOT
user=> (set! *unchecked-math* true)
true
user=> (inc 1)
2

Looking at Numbers.java I see both unchecked_inc and inc have long/double taking methods.

Comment by Pierre-Yves Ritschard [ 23/Sep/14 1:49 PM ]

you're right, i must have been confused.

Comment by Pierre-Yves Ritschard [ 23/Sep/14 1:50 PM ]

not a bug





[CLJ-1502] Clojure Inspector navigation error Created: 12/Aug/14  Updated: 15/Aug/14

Status: Open
Project: Clojure
Component/s: None
Affects Version/s: Release 1.6
Fix Version/s: None

Type: Defect Priority: Minor
Reporter: Dan Campbell Assignee: Unassigned
Resolution: Unresolved Votes: 0
Labels: bug, inspector, navigation
Environment:

Windows 7 and 8, Java 7, Clojure repl


Attachments: Text File clj-1502-v1.patch    
Patch: Code

 Description   

With Clojure 1.6.0 on some platforms (details below), if you create an object such as

(def nst (vec '((3 7 22) 99 (123 18 225 437))))

and then you inspect the tree representing the object

(inspect-tree nst)

Most of the navigation with the keyboard proceeds fine. However, when you point to an individual value - e.g. the 99 or the 437 - and press the right arrow key, there is an error

Exception in thread "AWT-EventQueue-0" java.lang.UnsupportedOperationException: count not supported on this type: Long
	at clojure.lang.RT.countFrom(RT.java:556)
	at clojure.lang.RT.count(RT.java:530)
	at clojure.inspector$fn__6907.invoke(inspector.clj:40)
	at clojure.lang.MultiFn.invoke(MultiFn.java:227)
	at clojure.inspector$tree_model$fn__6929.invoke(inspector.clj:63)
	at clojure.inspector.proxy$java.lang.Object$TreeModel$775afa87.getChildCount(Unknown Source)
	at javax.swing.plaf.basic.BasicTreeUI$Actions.traverse(BasicTreeUI.java:4395)
	at javax.swing.plaf.basic.BasicTreeUI$Actions.actionPerformed(BasicTreeUI.java:4052)
	at javax.swing.SwingUtilities.notifyAction(SwingUtilities.java:1662)
	at javax.swing.JComponent.processKeyBinding(JComponent.java:2878)
	at javax.swing.JComponent.processKeyBindings(JComponent.java:2925)
	at javax.swing.JComponent.processKeyEvent(JComponent.java:2841)
	at java.awt.Component.processEvent(Component.java:6282)
	at java.awt.Container.processEvent(Container.java:2229)
	at java.awt.Component.dispatchEventImpl(Component.java:4861)
	at java.awt.Container.dispatchEventImpl(Container.java:2287)
	at java.awt.Component.dispatchEvent(Component.java:4687)
	at java.awt.KeyboardFocusManager.redispatchEvent(KeyboardFocusManager.java:1895)
	at java.awt.DefaultKeyboardFocusManager.dispatchKeyEvent(DefaultKeyboardFocusManager.java:762)
	at java.awt.DefaultKeyboardFocusManager.preDispatchKeyEvent(DefaultKeyboardFocusManager.java:1027)
	at java.awt.DefaultKeyboardFocusManager.typeAheadAssertions(DefaultKeyboardFocusManager.java:899)
	at java.awt.DefaultKeyboardFocusManager.dispatchEvent(DefaultKeyboardFocusManager.java:727)
	at java.awt.Component.dispatchEventImpl(Component.java:4731)
	at java.awt.Container.dispatchEventImpl(Container.java:2287)
	at java.awt.Window.dispatchEventImpl(Window.java:2719)
	at java.awt.Component.dispatchEvent(Component.java:4687)
	at java.awt.EventQueue.dispatchEventImpl(EventQueue.java:735)
	at java.awt.EventQueue.access$200(EventQueue.java:103)
	at java.awt.EventQueue$3.run(EventQueue.java:694)
	at java.awt.EventQueue$3.run(EventQueue.java:692)
	at java.security.AccessController.doPrivileged(Native Method)
	at java.security.ProtectionDomain$1.doIntersectionPrivilege(ProtectionDomain.java:76)
	at java.security.ProtectionDomain$1.doIntersectionPrivilege(ProtectionDomain.java:87)
	at java.awt.EventQueue$4.run(EventQueue.java:708)
	at java.awt.EventQueue$4.run(EventQueue.java:706)
	at java.security.AccessController.doPrivileged(Native Method)
	at java.security.ProtectionDomain$1.doIntersectionPrivilege(ProtectionDomain.java:76)
	at java.awt.EventQueue.dispatchEvent(EventQueue.java:705)
	at java.awt.EventDispatchThread.pumpOneEventForFilters(EventDispatchThread.java:242)
	at java.awt.EventDispatchThread.pumpEventsForFilter(EventDispatchThread.java:161)
	at java.awt.EventDispatchThread.pumpEventsForHierarchy(EventDispatchThread.java:150)
	at java.awt.EventDispatchThread.pumpEvents(EventDispatchThread.java:146)
	at java.awt.EventDispatchThread.pumpEvents(EventDispatchThread.java:138)
	at java.awt.EventDispatchThread.run(EventDispatchThread.java:91)

Environments where this has been reproduced:
+ Windows 7 Enterprise, SP1, Oracle JDK 1.7.0_51, Clojure 1.6.0
+ Ubuntu Linux 14.04.1, Oracle JDK 1.7.0_65, Clojure 1.6.0

Environments where the same sequence of events does not cause an exception:
+ Mac OS X 10.8.5, Oracle JDK 1.7.0_51, Clojure 1.6.0



 Comments   
Comment by Andy Fingerhut [ 13/Aug/14 6:08 PM ]

Patch clj-1502-v1.patch avoids the exception in the situation reported. Tested manually on OS X, Linux, and Windows 7 versions mentioned in the patch comment. I suspect it is not worth the effort to write an automated test for this.

Comment by Dan Campbell [ 15/Aug/14 6:40 PM ]

Thanks, Andy

  • DC




[CLJ-1370] ((println)) should throw CompilerException instead of NPE? Created: 06/Mar/14  Updated: 06/Mar/14  Resolved: 06/Mar/14

Status: Closed
Project: Clojure
Component/s: None
Affects Version/s: Release 1.5
Fix Version/s: None

Type: Defect Priority: Minor
Reporter: The Alchemist Assignee: Unassigned
Resolution: Declined Votes: 0
Labels: bug


 Description   

How to Reproduce

=> ((println))

NullPointerException   hello.core/eval4117 (NO_SOURCE_FILE:1)

What's Wrong?

I might be completely wrong, in which case don't hesitate to close this defect, but I wish this would throw a CompilerException with an IllegalArgumentException, just like ((nil)):

=> ((nil))
CompilerException java.lang.IllegalArgumentException: Can't call nil, compiling:(NO_SOURCE_PATH:1:2)


 Comments   
Comment by Alex Miller [ 06/Mar/14 12:37 PM ]

There is no compilation error here. The error occurs during evaluation.

user> (defn x [] ((println)))  ;; compiles just fine
#'user/x
user> (x)   ;; fails in evaluation
NullPointerException   user/x (NO_SOURCE_FILE:1)

The error is thrown when trying to evaluate (nil) where NullPointerException is a perfectly valid error.





[CLJ-1306] Cannot reduce over short[] arrays Created: 14/Dec/13  Updated: 14/Dec/13  Resolved: 14/Dec/13

Status: Closed
Project: Clojure
Component/s: None
Affects Version/s: Release 1.5
Fix Version/s: None

Type: Defect Priority: Major
Reporter: Mike Anderson Assignee: Unassigned
Resolution: Duplicate Votes: 0
Labels: bug

Attachments: File clj-1306.diff    

 Description   

Reducing over a short array currently causes an error:

(reduce + (seq (short-array 10)))
=> ClassCastException [S cannot be cast to [Ljava.lang.Object; clojure.core.protocols/fn--6037 (protocols.clj:126)

This appears to occur because ArraySeq is assumed by protocols.clj to contain an Object[] array in the ".array" field, when in fact it is a short[] array.

Proposed solution to to create ArraySeq_short (analogous to the other primitive types ArraySeq_long etc.) to handle short arrays.



 Comments   
Comment by Mike Anderson [ 14/Dec/13 7:37 AM ]

Fix for CLJ-1306

Comment by Alex Miller [ 14/Dec/13 8:17 AM ]

This was also discovered in CLJ-1200 and the patch for that issue includes ArraySeq_short as you propose. It is expected that CLJ-1200 will be included in 1.6.

Comment by Mike Anderson [ 14/Dec/13 9:29 AM ]

OK, thanks Alex!

Just to be sure: Can you confirm that a fix will definitely go into 1.6? This is a defect, and as such should have a higher priority than CLJ-1200 (which appears to be presented as an performance enhancement patch).

My patch also includes a regression test which I think could helpfully be included in the test suite.

Comment by Alex Miller [ 14/Dec/13 11:24 AM ]

Yes, I fully expect CLJ-1200 to be included and talked to Rich about it as recently as yesterday. I could split things out of that patch and pull both of these in separately. That would be objectively better but definitely more work to do all the ticket, patch, and screening work so I'd rather not. If you want to attach just the regression test to 1200, I think we could include it that way as 1200 hasn't been screened yet. Stuart Sierra is planning to screen it in the next few days.





[CLJ-1267] Reader Bug for making vector of numbers Created: 26/Sep/13  Updated: 26/Sep/13  Resolved: 26/Sep/13

Status: Closed
Project: Clojure
Component/s: None
Affects Version/s: Release 1.5
Fix Version/s: None

Type: Defect Priority: Major
Reporter: Amin Razavi Assignee: Unassigned
Resolution: Declined Votes: 0
Labels: bug
Environment:

Windows 8 , Core i7



 Description   

When Trying To Make a Vector of Numbers Like 01 04 ... it's OK!
=> (vector 04)
; [4]
But When Trying To Make a Vector of "09" It Says :
NumberFormatException Invalid Number: 09



 Comments   
Comment by Alex Miller [ 26/Sep/13 10:56 AM ]

Numbers with a leading 0 are read as octal (valid digits = 0-7). Example:

> 0100
64

So, this is the expected behavior.

Comment by Amin Razavi [ 26/Sep/13 7:49 PM ]

shame on me , sorry.





[CLJ-1258] (apply and [false true]) gives CompilerException Created: 08/Sep/13  Updated: 09/Sep/13  Resolved: 09/Sep/13

Status: Closed
Project: Clojure
Component/s: None
Affects Version/s: Release 1.4, Release 1.5
Fix Version/s: None

Type: Defect Priority: Major
Reporter: Cynthia Qiu Assignee: Unassigned
Resolution: Declined Votes: 0
Labels: bug
Environment:

Ubuntu 13.04. I run 1.4.0 by "apt-get install", and 1.5.1 by "java -cp clojure-1.5.1.jar clojure.main".



 Description   

user=> (apply and [false true])
CompilerException java.lang.RuntimeException: Can't take value of a macro: #'clojure.core/and, compiling:(NO_SOURCE_PATH:1)
user=> (apply and (list false true))
CompilerException java.lang.RuntimeException: Can't take value of a macro: #'clojure.core/and, compiling:(NO_SOURCE_PATH:2)



 Comments   
Comment by Cynthia Qiu [ 08/Sep/13 10:18 PM ]

I got it. "and" is not a function but a special form. Sorry for that.

Comment by Andy Fingerhut [ 08/Sep/13 11:57 PM ]

Minor comment on yours: Clojure and other Lisps often distinguish between "special forms" and macros. I may be missing some of the distinction, but I am pretty sure the main part is that special forms are usually built into the implementation of Lisp, whereas macros are typically defined by the user via defmacro.

In any case, neither special forms nor macros may be used with apply, only functions.

Comment by Alex Miller [ 09/Sep/13 1:26 AM ]

As per Andy's comment, this is expected behavior when trying to apply to macros.





[CLJ-1179] distinct? does not accept zero arguments Created: 09/Mar/13  Updated: 04/Jan/14  Resolved: 12/Apr/13

Status: Closed
Project: Clojure
Component/s: None
Affects Version/s: Release 1.4, Release 1.5
Fix Version/s: None

Type: Enhancement Priority: Minor
Reporter: Jean Niklas L'orange Assignee: Unassigned
Resolution: Declined Votes: 0
Labels: bug

Attachments: Text File clj-1179-distinct-zero-arguments.txt    
Patch: Code

 Description   

distinct? cannot be invoked with zero arguments. When using the pattern (apply distinct? x), this is bothersome as you have to check whether x is empty or not. It is also logical that distinct? should return true if no arguments are given, since there are no duplicates.

What (small set of) steps will reproduce the problem?

user=> (apply distinct? [])
ArityException Wrong number of args (0) passed to: core$distinct-QMARK-  clojure.lang.AFn.throwArity (AFn.java:437)

What is the expected output? What do you see instead?

I would expect distinct? to return true whenever given zero arguments.

What version are you using?

This was tested under Clojure 1.4 and Clojure 1.5.



 Comments   
Comment by Jean Niklas L'orange [ 09/Mar/13 6:08 AM ]

Attached the straightforward patch which solves this issue.

Comment by Devin Walters [ 04/Jan/14 1:32 PM ]

Is there a reason this was closed without a comment?

Comment by Alex Miller [ 04/Jan/14 2:12 PM ]

Rich declined it, implying that it was not a change he wanted. Not sure of the reason.





[CLJ-1028] (compile) crashes with NullPointerException if public function 'load' is defined Created: 20/Jul/12  Updated: 20/Jul/12  Resolved: 20/Jul/12

Status: Closed
Project: Clojure
Component/s: None
Affects Version/s: Release 1.4
Fix Version/s: None

Type: Defect Priority: Minor
Reporter: Ben Kelly Assignee: Unassigned
Resolution: Declined Votes: 0
Labels: Compiler, bug
Environment:

Linux, OpenJDK 1.6.0 64bit


Attachments: File stack-trace    

 Description   

When performing AOT compilation, if the namespace being compiled or one of the namespaces :required by it defines a public function named 'load', the compiler will crash with a NullPointerException.

The following code demonstrates this:

(ns ca.ancilla.kessler.core (:gen-class)) (defn load [x] x) (defn -main [& args] 0)

When run directly, with 'clojure -m ca.ancilla.kessler.core' or 'clojure ca/ancilla/kessler/core.clj', it runs as expected. When loaded with 'clojure -i' and (compile)d, however, or when automatically compiled by 'lein run', it results in a NullPointerException (the complete stack trace is attached).

This occurs whether or not 'load' or actually called. It does not, however, occur if 'load' is private.



 Comments   
Comment by Ben Kelly [ 20/Jul/12 12:43 PM ]

If you add (:refer-clojure :exclude [load]) to the (ns), it works fine:

(ns ca.ancilla.kessler.core (:refer-clojure :exclude [load]) (:gen-class))
(defn load [x] x)
(defn -main [& args] 0)

Thanks to llasram on IRC for discovering this.

Comment by Stuart Halloway [ 20/Jul/12 4:35 PM ]

You should not replace functions in clojure.core. This is left legal (with a loud CONSOLE warning) for compatibility, but programs that do it are in error.

Comment by Ben Kelly [ 20/Jul/12 10:06 PM ]

So, just to make sure that I have this right, then...

If I want to create a namespace with a public function that shares a name with a function in clojure.core, the only supported way of doing this is to (:refer-clojure :exclude [list of all such functions])?

If so, it would be nice if the warning were replaced with an error, rather than having the compiler emit an error and then crash.





[CLJ-1006] Quotient on bigdec may produce wrong result Created: 01/Jun/12  Updated: 01/Mar/13  Resolved: 09/Nov/12

Status: Closed
Project: Clojure
Component/s: None
Affects Version/s: Release 1.4
Fix Version/s: None

Type: Defect Priority: Major
Reporter: laurent joigny Assignee: Unassigned
Resolution: Declined Votes: 0
Labels: bug
Environment:

Linux 3.2.0-24-generic #39-Ubuntu SMP i686 GNU/Linux


Attachments: Java Source File TestBigDecimalQuotient.java    

 Description   

Hi,

As discussed on the mailing list in the message "When arithmetic on a computer bite back" (01/jun)

There may be bug in the way quotient is implemented for bigdec.

user> (quot 1.4411518807585587E17 2) ;; correct with doubles
7.2057594037927936E16
user> (quot 1.4411518807585587E+17M 2) ;; wrong with BigDecs
72057594037927935M


Laurent



 Comments   
Comment by laurent joigny [ 01/Jun/12 5:48 PM ]

I can reproduce the bug when using BigDecimal constructor on String.
See attached file for a test class.

More infos :
java version "1.6.0_24"
OpenJDK Runtime Environment (IcedTea6 1.11.1) (6b24-1.11.1-4ubuntu3)
OpenJDK Client VM (build 20.0-b12, mixed mode, sharing)

Comment by laurent joigny [ 01/Jun/12 5:49 PM ]

A simple test file, that you can drop in Clojure sources and execute to reproduce the bug on BigDecimal constructor using String as argument.

Comment by Tassilo Horn [ 03/Jun/12 4:30 AM ]

Seems to be a general precision problem. Note that in

user> (quot 1.4411518807585587E17 2) ;; correct with doubles
7.2057594037927936E16
user> (quot 1.4411518807585587E+17M 2) ;; wrong with BigDecs
72057594037927935M

the double result is actually wrong and the bigdec one is correct. The problem which lead to the wrong conclusion is that in your calculation the input number is already wrong.

So the moral is: don't use any floating points (neither doubles nor bigdecs) for computations involving divisibility tests.

For bigdecs, you can set the math context for making computations throw exceptions if they lose precision, though:

user> (binding [*math-context* (java.math.MathContext. 1 java.math.RoundingMode/UNNECESSARY)]
	       (quot (bigdec (Math/pow 2 58)) 2))
;Division impossible
;  [Thrown class java.lang.ArithmeticException]
Comment by Stuart Sierra [ 09/Nov/12 8:49 AM ]

Not a bug. Just floating-point arithmetic.





[CLJ-988] the locking in MultiFn.java (synchronized methods) can cause lots of contention in multithreaded programs Created: 08/May/12  Updated: 21/Sep/12  Resolved: 21/Sep/12

Status: Closed
Project: Clojure
Component/s: None
Affects Version/s: None
Fix Version/s: Release 1.5

Type: Enhancement Priority: Major
Reporter: Kevin Downey Assignee: Stuart Sierra
Resolution: Completed Votes: 10
Labels: bug, performance

Attachments: Text File clj-988-tests-only-patch-v1.txt     File issue988-lockless-multifn+tests-120817.diff     File issue988-lockless-multifn+tests.diff    
Patch: Code and Test
Approval: Ok

 Description   

if you call a single multimethod a lot in multithreaded code you get lots of contention for the lock on the multimethod. this contention slows things down a lot.

this is due to getMethod being synchronized. it would be great if there was some fast non-locking path through the multimethod.



 Comments   
Comment by Kevin Downey [ 08/May/12 11:30 AM ]

http://groups.google.com/group/clojure-dev/browse_thread/thread/6a8219ae3d4cd0ae?hl=en

Comment by David Santiago [ 11/May/12 6:38 AM ]

Here's a stab in the dark attempt at rewriting MultiFn to use atoms to swap between immutable copies of its otherwise mutable state.

The four pieces of the MultiFn's state that are mutable and protected by locks are now contained in the MultiFn.State class, which is immutable and contains convenience functions for creating a new one with one field changed. An instance of this class is now held in an atom in the MultiFn called state. Changes to any of these four members are now done with an atomic swap of these State objects.

The getMethod/findAndCacheBestMethod complex was rewritten to better enable the atomic logic. findAndCacheBestMethod was replaced with findBestMethod, which merely looks up the method; the caching logic was moved to getMethod so that it can be retried easily as part of the work that method does.

As it was findAndCacheBestMethod seemingly had the potential to cause a stack overflow in a perfect storm of heavy concurrent modification, since it calls itself recursively if it finds that the hierarchy has changed while it has done its work. This logic is now done in the CAS looping of getMethod, so hopefully that is not even an unlikely possibility anymore.

There is still seemingly a small race here, since the check is done of a regular variable against the value in a ref. Now as before, the ref could be updated just after you do the check, but before the MultiFn's state is updated. Of course, only the method lookup part of a MultiFn call was synchronized before; it could already change after the lookup but before the method itself executed, having a stale method finish seemingly after the method had been changed. Things are no different now in general, with the atom-based approach, so perhaps this race is not a big deal, as a stale value can't persist for long.

The patch passes the tests and Clojure and multimethods seems to work.

Comment by Kevin Downey [ 12/May/12 8:59 PM ]

this patch gets rid of the ugly lock contention issues. I have not been able to benchmark it vs. the old multimethod implementation, but looking at it, I would not be surprised if it is faster when the system is in a steady state.

Comment by Stuart Halloway [ 08/Jun/12 12:11 PM ]

This looks straightforward, except for getMethod. Can somebody with context add more discussion of how method caching works?

Also, it would be great to have tests with this one.

Comment by David Santiago [ 15/Jun/12 4:44 AM ]

Obviously I didn't write the original code, so I'm not the ideal
person to explain this stuff. But I did work with it a bit recently,
so in the hopes that I can be helpful, I'm writing down my
understanding of the code as I worked with it. Since I came to the
code and sort of reverse engineered my way to this understanding,
hopefully laying this all out will make any mistakes or
misunderstandings I may have made easier to catch and correct. To
ensure some stability, I'll talk about the original MultiFn code as it
stands at this commit:
https://github.com/clojure/clojure/blob/8fda34e4c77cac079b711da59d5fe49b74605553/src/jvm/clojure/lang/MultiFn.java

There are four pieces of state that change as the multimethod is either
populated with methods or called.

  • methodTable: A persistent map from a dispatch value (Object) to
    a function (IFn). This is the obvious thing you think it is,
    determining which dispatch values call which function.
  • preferTable: A persistent map from a dispatch value (Object) to
    another value (Object), where the key "is preferred" to the value.
  • methodCache: A persistent map from a dispatch value (Object) to
    function (IFn). By default, the methodCache is assigned the same
    map value as the methodTable. If values are calculated out of the
    hierarchy during a dispatch, the originating value and the
    ultimate method it resolves to are inserted as additional items in
    the methodCache so that subsequent calls can jump right to the
    method without recursing down the hierarchy and preference table.
  • cachedHierarchy: An Object that refers to the hierarchy that is
    reflected in the latest cached values. It is used to check if the
    hierarchy has been updated since we last updated the cache. If it
    has been updated, then the cache is flushed.

I think reset(), addMethod(), removeMethod(), preferMethod(),
prefers(), isA(), dominates(), and resetCache() are extremely
straightforward in both the original code and the patch. In the
original code, the first four of those are synchronized, and the other
four are only called from methods that are synchronized (or from
methods only called from methods that are synchronized).

Invoking a multimethod through its invoke() function will call
getFn(). getFn() will call the getMethod() function, which is
synchronized. This means any call of the multimethod will wait for and
take a lock as part of method invocation. The goal of the patch in
this issue is to remove this lock on calls into the multimethod. It in
fact removes the locks on all operations, and instead keeps its
internal mutable state by atomically swapping a private immutable
State object held in an Atom called state.

The biggest change in the patch is to the
getFn()>getMethod()>findAndCacheBestMethod() complex from the
original code. I'll describe how that code works first.

In the original code, getFn() does nothing but bounce through
getMethod(). getMethod() tries three times to find a method to call,
after checking that the cache is up to date and flushing it if it
isn't:

1. It checks if there's a method for the dispatch value in the
methodCache.

2. If not, it calls findAndCacheBestMethod() on the
dispatch value. findAndCacheBestMethod() does the following:

1. It iterates through every map entry in the method table,
keeping at each step the best entry it has found so far
(according to the hierarchy and preference tables).

2. If it did not find a suitable entry, it returns null.

3. Otherwise, it checks if the hierarchy has been changed since the cache
was last updated. If it has not changed, it inserts the method
into the cache and returns it. If it has been changed, it
resets the cache and calls itself recursively to repeat the process.

3. Failing all else, it will return the method for the default
dispatch value.

Again, remember everything in the list above happens in a call to a
synchronized function. Also note that as it is currently written,
findAndCacheBestMethod() uses recursion for iteration in a way that
grows the stack. This seems unlikely to cause a stack overflow unless
the multimethod is getting its hierarchy updated very rapidly for a
sustained period while someone else tries to call it. Nonetheless, the
hierarchy is held in an IRef that is updated independently of the
locking of the MultiFn. Finally, note that the multimethod is locked
only while the method is being found. Once it is found, the lock is
released and the method actually gets called afterwards without any
synchronization, meaning that by the time the method actually
executes, the multimethod may have already been modified in a way that
suggests a different method should have been called. Presumably this
is intended, understood, and not a big deal.

Moving on now to the patch in this issue. As mentioned, the main
change is updating this entire apparatus to work with a single atomic
swap to control concurrency. This means that all updates to the
multimethod's state have to happen at one instant in time. Where the
original code could make multiple changes to the state at different
times, knowing it was safely protected by an exclusive lock, rewriting
for atom swaps requires us to reorganize the code so that all updates
to the state happen at the same time with a single CAS.

To implement this change, I pulled the implicit looping logic from
findAndCacheBestMethod() up into getMethod() itself, and broke the
"findBestMethod" part into its own function, findBestMethod(), which
makes no update to any state while implementing the same
logic. getMethod() now has an explicit loop to avoid stack-consuming
recursion on retries. This infinite for loop covers all of the logic
in getMethod() and retries until a method is successfully found and a
CAS operation succeeds, or we determine that the method cannot be
found and we return the default dispatch value's implementation.

I'll now describe the operation of the logic in the for loop. The
first two steps in the loop correspond to things getMethod() does
"before" its looping construct in the original code, but we have to do
in the loop to get the latest values.

1. First we dereference our state, and assign this value to both
oldState and newState. We also set a variable called needWrite to
false; this is so we can avoid doing a CAS (they're not free) when
we have not actually updated the state.

2. Check if the cache is stale, and flush it if so. If the cache
gets flushed, set needWrite to true, as the state has changed.

3. Check if the methodCache has an entry for this dispatch
value. If so, we are "finished" in the sense that we found the
value we wanted. However, we may need to update the state. So,

  • If needWrite is false, we can return without a CAS, so just
    break out of the loop and return the method.
  • Otherwise, we need to update the state object with a CAS. If
    the CAS is successful, break out of the loop and return the
    target function. Otherwise, continue on the next iteration
    of the loop, skipping any other attempts to fetch the method
    later in the loop (useless work, at this point).

4. The value was not in the methodCache, so call the function
findBestMethod() to attempt to find a suitable method based on the
hierarchy and preferences. If it does find us a suitable method,
we now need to cache it ourselves. We create a new state object
with the new method cache and attempt to update the state atom
with a CAS (we definitely need a write here, so no need to check
needWrite at this point).

The one thing that is possibly questionable is the check at this
point to make sure the hierarchy has not been updated since the
beginning of this method. I inserted this here to match the
identical check at the corresponding point in
findAndCacheBestMethod() in the original code. That is also a
second check, since the cache is originally checked for freshness
at the very beginning of getMethod() in the original code. That
initial check happens at the beginning of the loop in the
patch. Given that there is no synchronization with the method
hierarchy, it is not clear to me that this second check is needed,
since we are already proceeding with a snapshot from the beginning
of the loop. Nonetheless, it can't hurt as far as I can tell, it
is how the original code worked, and I assume there was some
reason for that, so I kept the second check.

5. Finally, if findBestMethod() failed to find us a method for the
dispatch value, find the method for the default dispatch value and
return that by breaking out of the loop.

So the organization of getMethod() in the patch is complicated by two
factors: (1) making the retry looping explicit and stackless, (2)
skipping the CAS when we don't need to update state, and (3) skipping
needless work later in the retry loop if we find a value but are
unable to succeed in our attempt to CAS. Invoking a multimethod that
has a stable hierarchy and a populated cache should not even have a
CAS operation (or memory allocation) on this code path, just a cache
lookup after the dispatch value is calculated.

Comment by David Santiago [ 15/Jun/12 4:45 AM ]

I've updated this patch (removing the old version, which is entirely superseded by this one). The actual changes to MultiFn.java are identical (modulo any thing that came through in the rebase), but this newer patch has tests of basic multimethod usage, including defmulti, defmethod, remove-method, prefer-method and usage of these in a hierarchy that updates in a way interleaved with calls.

Comment by David Santiago [ 15/Jun/12 6:38 AM ]

I created a really, really simple benchmark to make sure this was an improvement. The following tests were on a quad-core hyper-threaded 2012 MBP.

With two threads contending for a simple multimethod:
The lock-based multifns run at an average of 606ms, with about 12% user, 15% system CPU at around 150%.
The lockless multifns run at an average of 159ms, with about 25% user, 3% system CPU at around 195%.

With four threads contending for a simple multimethod:
The lock-based multifns run at an average of 1.2s, with about 12% user, 15% system, CPU at around 150%.
The lockless multifns run at an average of 219ms, with about 50% user, 4% system, CPU at around 330%.

You can get the code at https://github.com/davidsantiago/multifn-perf

Comment by David Santiago [ 14/Aug/12 10:02 PM ]

It's been a couple of months, and so I just wanted to check in and see if there was anything else needed to move this along.

Also, Alan Malloy pointed out to me that my benchmarks above did not mention single-threaded performance. I actually wrote this into the tests above, but I neglected to report them at the time. Here are the results on the same machine as above (multithreaded versions are basically the same as the previous comment).

With a single thread performing the same work:
The lock-based multifns run at an average of 142ms.
The lockless multifns run at an average of 115ms.

So the lockless multimethods are still faster even in a single-threaded case, although the speedup is more modest compared to the speedups in the multithreaded cases above. This is not surprising, but it is good to know.

Comment by Stuart Sierra [ 17/Aug/12 2:58 PM ]

Screened. The approach is sound.

I can confirm similar performance measurements using David Santiago's benchmark, compared with Clojure 1.5 master as of commit f5f4faf.

Mean runtime (ms) of a multimethod when called repeatedly from N threads:

|            | N=1 | N=2 | N=4 |
|------------+-----+-----+-----|
| 1.5 master |  80 | 302 | 765 |
| lockless   |  63 |  88 | 125 |

My only concern is that the extra allocations of the State class will create more garbage, but this is probably not significant if we are already using persistent maps. It would be interesting to compare this implementation with one using thread-safe mutable data structures (e.g. ConcurrentHashMap) for the cache.

Comment by David Santiago [ 17/Aug/12 7:05 PM ]

I think your assessment that it's not significant compared to the current implementation using persistent maps is correct. Regarding the extra garbage, note that the new State is only created when the hierarchy has changed or there's a cache miss (related, obviously)... situations where you're already screwed. Then it won't have to happen again for the same method (until another change to the multimethod). So for most code, it won't happen very often.

ConcurrentHashMap might be faster, it'd be interesting to see. My instinct is to keep it as close to being "made out of Clojure" as possible. In fact, it's hard to see why this couldn't be rewritten in Clojure itself some day, as I believe Chas Emerick has suggested. Also, I would point out that two of the three maps are used from the Clojure side in core.clj. I assume they would be happier if they were persistent maps.

Funny story: I was going to point out the parts of the code that were called from the clojure side just now, and alarmingly cannot find two of the functions. I think I must have misplaced them while rewriting the state into an immutable object. Going to attach a new patch with the fix and some tests for it in a few minutes.

Comment by David Santiago [ 17/Aug/12 7:44 PM ]

Latest patch for this issue. Supersedes issue988-lockless-multifn+tests.diff as of 08/17/2012.

Comment by David Santiago [ 17/Aug/12 7:49 PM ]

As promised, I reimplemented those two functions. I also added more multimethod tests to the test suite. The new tests should definitely prevent a similar mistake. While I was at it, I went through core.clj and found all the multimethod API functions I could and ensured that there were at least some basic functionality tests for all of them. The ones I found were: defmulti, defmethod, remove-all-methods, remove-method, prefer-method, methods, get-method, prefers (Some of those already had tests from the earlier version of the patch).

Really sorry about catching this right after you vetted the patch. 12771 test assertions were apparently not affected by prefers and methods ceasing to function, but now there are 12780 to help prevent a similar error. Since you just went through it, I'm leaving the older version of the patch up so you can easily see the difference to what I've added.

Comment by Rich Hickey [ 15/Sep/12 9:05 AM ]

https://github.com/clojure/clojure/commit/83ebf814d5d6663c49c1b2d0d076b57638bff673 should fix these issues. The patch here was too much of a change to properly vet.

If you could though, I'd appreciate a patch with just the multimethod tests.

Comment by Andy Fingerhut [ 15/Sep/12 10:59 AM ]

Patch clj-988-tests-only-patch-v1.txt dated Sep 15 2012 is a subset of David Santiago's
patch issue988-lockless-multifn+tests-120817.diff dated Aug 17 2012. It includes only the tests from that patch. Applies cleanly and passes tests with latest master after Rich's read/write lock change for multimethods was committed.

Comment by Rich Hickey [ 17/Sep/12 9:20 AM ]

tests-only patch ok





Generated at Wed Oct 22 23:40:07 CDT 2014 using JIRA 4.4#649-r158309.