<< Back to previous view

[CTYP-99] Checking an ns becomes significantly slower as the number of optional keys in a HMap increases Created: 03/Dec/13  Updated: 14/Feb/14  Resolved: 14/Feb/14

Status: Resolved
Project: core.typed
Component/s: None
Affects Version/s: None
Fix Version/s: None

Type: Defect Priority: Blocker
Reporter: Gordon Syme Assignee: Ambrose Bonnaire-Sergeant
Resolution: Completed Votes: 1
Labels: None

Attachments: File core.clj     File core.clj     File project.clj     GZip Archive test-hmap.tar.gz    

 Description   

I'm using clojure.core.typed 0.2.19 with the slim classifier but I've observed the same without slim.

My suspicion is that the latent filters associated with functions grow in size exponentially with each extra optional key to a HMap (based on the output when you have a type error). I think it's generating all combinations of present and absent keys for the HMap when calculating latent filters for a function.

I've attached a tarball with a lein project with ten namespaces that all contain the same ten simple functions in the form

(t/ann f [(HMap :optional {:a String}) -> (t/Option String)])
(defn f [m]
  (:a m))

The type annotations vary in the number of optional keywords.

(test-hmap.core/go) checks all the namespaces. The time to check each namespace grows non-linearly. The first namespace gets penalised by core.typed initialisation the first time it's run.

E.g. on my local machine:

test-hmap.one
Start collecting test-hmap.one
Finished collecting test-hmap.one
Collected 1 namespaces in 20.743 msecs
Start checking test-hmap.one
Checked test-hmap.one in 46.231 msecs
Checked 1 namespaces (approx. 34 lines) in 67.672 msecs


test-hmap.two
Start collecting test-hmap.two
Finished collecting test-hmap.two
Collected 1 namespaces in 18.751 msecs
Start checking test-hmap.two
Checked test-hmap.two in 45.525 msecs
Checked 1 namespaces (approx. 35 lines) in 64.814 msecs


test-hmap.three
Start collecting test-hmap.three
Finished collecting test-hmap.three
Collected 1 namespaces in 19.106 msecs
Start checking test-hmap.three
Checked test-hmap.three in 57.346 msecs
Checked 1 namespaces (approx. 35 lines) in 77.055 msecs


test-hmap.four
Start collecting test-hmap.four
Finished collecting test-hmap.four
Collected 1 namespaces in 21.222 msecs
Start checking test-hmap.four
Checked test-hmap.four in 51.339 msecs
Checked 1 namespaces (approx. 35 lines) in 73.155 msecs


test-hmap.five
Start collecting test-hmap.five
Finished collecting test-hmap.five
Collected 1 namespaces in 22.215 msecs
Start checking test-hmap.five
Checked test-hmap.five in 63.415 msecs
Checked 1 namespaces (approx. 35 lines) in 86.309 msecs


test-hmap.six
Start collecting test-hmap.six
Finished collecting test-hmap.six
Collected 1 namespaces in 23.76 msecs
Start checking test-hmap.six
Checked test-hmap.six in 99.407 msecs
Checked 1 namespaces (approx. 35 lines) in 123.881 msecs


test-hmap.seven
Start collecting test-hmap.seven
Finished collecting test-hmap.seven
Collected 1 namespaces in 26.519 msecs
Start checking test-hmap.seven
Checked test-hmap.seven in 213.515 msecs
Checked 1 namespaces (approx. 35 lines) in 240.639 msecs


test-hmap.eight
Start collecting test-hmap.eight
Finished collecting test-hmap.eight
Collected 1 namespaces in 32.581 msecs
Start checking test-hmap.eight
Checked test-hmap.eight in 626.76 msecs
Checked 1 namespaces (approx. 35 lines) in 659.975 msecs


test-hmap.nine
Start collecting test-hmap.nine
Finished collecting test-hmap.nine
Collected 1 namespaces in 43.478 msecs
Start checking test-hmap.nine
Checked test-hmap.nine in 2634.468 msecs
Checked 1 namespaces (approx. 35 lines) in 2678.716 msecs


test-hmap.ten
Start collecting test-hmap.ten
Finished collecting test-hmap.ten
Collected 1 namespaces in 84.394 msecs
Start checking test-hmap.ten
Checked test-hmap.ten in 9277.54 msecs
Checked 1 namespaces (approx. 35 lines) in 9362.613 msecs


 Comments   
Comment by Ambrose Bonnaire-Sergeant [ 03/Dec/13 12:50 PM ]

Yes, the primitives are :mandatory, :absent-keys and :complete; :optional expands to be in terms of those.

Thanks for the report, I haven't done performance testing on this strategy. It will probably need to be reconsidered.

Comment by Cees van Kemenade [ 26/Jan/14 1:04 PM ]

code of project ctcrash

Comment by Cees van Kemenade [ 26/Jan/14 1:05 PM ]

+1 For this issue.

I've spend quite some time to learn core.typed on a real use case, and I have to say that I'm amazed by the power of the analysis and the potential to prevent errors that otherwise would be detected to late! Apart from that the type-annotations are valuable and precise information about the interface when doing maintenance at code you did not see for some time.
Two big reasons to use core.typed, however, as I turned more and more code into typed code the number of optional keys increased in my core data-record increase, and core.typed (check-ns) slowed down to the extreme (using version 0.2.25 (current stable)). I refactored the code to get a single line of code that triggers the issue on my core data structure a (Seqable DbMsg). Attached you find the project.clj and code (core.clj vs 8:09pm).

When I reduce the number of optional keys to 4-5 the (check-ns) runs in a few seconds. But having 9-11 optional keys kills the (check-ns) process.

I hope this one-liner helps making the issues easy to reproduce.
Currently I don't know a work-around for this issue,
so I can not use core.typed to check my project.

I guess that many people/projects should run into the same issues,
as the pattern is returning quite often in clojure:

1. start with data-set with small Hashmaps
2. the hashmaps evolve in a number steps (meaning more keys are added and some of the existing keys are removed)
I guess the workaround would be to use limited or no optional keys, and prepare a custom HMap definition for each stage of analysis. Although this would work, the amount of bookkeeping already significant for a all records are running through the same stages (linear process), and will be a showstopper if it is a non-linear process (not all records follow the same path (sequence of analysis stages)).
I will investigate whether splitting in multiple types (for different stages) will rescue my day.

Comment by Ambrose Bonnaire-Sergeant [ 26/Jan/14 7:51 PM ]

Upgraded to 'Blocker'.

Comment by Ambrose Bonnaire-Sergeant [ 27/Jan/14 6:34 AM ]

This is theoretically trivial to implement, and just involves shuffling around where optional keys are handled. However, this is spread out all over the code base, so I may not complete the patch for a week.

Great to hear core.typed is working out for you!

Comment by Cees van Kemenade [ 27/Jan/14 6:44 AM ]

Thanks for picking up this issue fast.
I'll wait patiently.

Comment by Ambrose Bonnaire-Sergeant [ 14/Feb/14 2:26 AM ]

This is fixed in 0.2.31.

I've added both of your tests in the test suite - thanks! https://github.com/typedclojure/core.typed-test-suite/tree/master/src/hmap

Please confirm and I'll close the issue.

Comment by Cees van Kemenade [ 14/Feb/14 4:11 AM ]

This resolve the issue and also resolves CT-102.
Thanks!

The code in hmap/big-options.clj (from your test-suite) still contains a workaround to make core.typed proceed without errors.
The (if (string? notifs) ... on line 107 is introduced to prevent a type-error (core.typed infers it that notifs might als be a (sequable String). However, as this is the else-branch of line 104 (if (or (seq? notifs) (vector notifs)) ... the case that notifs is a (Seqable string) can be excluded already).
Also using on line 104 the more general (if (sequential? notifs) ... does not resolve this issues.

Is this inference error already on the list, or would you recommend posting a seperate JIRA-case for this issue?

Comment by Gordon Syme [ 14/Feb/14 5:56 AM ]

Our core.typed checks run an order of magnitude quicker now (~400s --> ~80s). And they keep that speed with all the extra optionals now. That's really awesome, thanks Ambrose!





[CTYP-91] polymorphic ann-protocol doesn't work Created: 30/Oct/13  Updated: 05/Nov/13  Resolved: 05/Nov/13

Status: Resolved
Project: core.typed
Component/s: None
Affects Version/s: None
Fix Version/s: None

Type: Defect Priority: Major
Reporter: Russell Mull Assignee: Ambrose Bonnaire-Sergeant
Resolution: Completed Votes: 0
Labels: None
Environment:

Clojure 1.5.1, Core.typed 0.2.14 and 0.2.15



 Description   

The example of a polymorphic defprotocol annotation from the documentation is:

(ann-protocol [[x :variance :covariant]]
  IFoo
  bar
  [IFoo -> Any]
  baz
  [IFoo -> Number])

It's implied that there exists a corresponding protocol like this:

(defprotocol> IFoo
  (bar [this])
  (baz [this]))

Type checking this code with check-ns gives an error:

AssertionError Assert failed: ((u/hash-c? (every-pred symbol? (complement namespace)) Type?) methods)
	clojure.core.typed.type-rep/Protocol-maker (type_rep.clj:289)
	clojure.core.typed.type-ctors/Protocol* (type_ctors.clj:448)
	clojure.core.typed.collect-phase/gen-protocol* (collect_phase.clj:443)
	clojure.core.typed.collect-phase/fn--14235 (collect_phase.clj:458)
	clojure.lang.MultiFn.invoke (MultiFn.java:227)
	clojure.core.typed.collect-phase/collect -COLON-invoke14011 (collect_phase.clj:189)
	clojure.lang.MultiFn.invoke (MultiFn.java:227)
	clojure.core.typed.collect-phase/collect-asts (collect_phase.clj:119)
	clojure.core.typed.collect-phase/collect-ns (collect_phase.clj:107)
	clojure.core.typed/check-ns (typed.clj:1459)
	clojure.core.typed/check-ns (typed.clj:1429)

Incidentally, it would be nice if the example in the documentation actually used the declared type parameter in some way.



 Comments   
Comment by Russell Mull [ 30/Oct/13 10:01 PM ]

The above stack trace is from 0.2.14. This is the trace for 0.2.15:

AssertionError Assert failed: ((u/hash-c? (every-pred symbol? (complement namespace)) Type?) methods)
	clojure.core.typed.type-rep/Protocol-maker (type_rep.clj:289)
	clojure.core.typed.type-ctors/Protocol* (type_ctors.clj:448)
	clojure.core.typed.collect-phase/gen-protocol* (collect_phase.clj:443)
	clojure.core.typed.collect-phase/fn--14237 (collect_phase.clj:458)
	clojure.lang.MultiFn.invoke (MultiFn.java:227)
	clojure.core.typed.collect-phase/collect -COLON-invoke14013 (collect_phase.clj:189)
	clojure.lang.MultiFn.invoke (MultiFn.java:227)
	clojure.core.typed.collect-phase/collect-asts (collect_phase.clj:119)
	clojure.core.typed.collect-phase/collect-ns (collect_phase.clj:107)
	clojure.core.typed/check-ns (typed.clj:1459)
	clojure.core.typed/check-ns (typed.clj:1429)
Comment by Ambrose Bonnaire-Sergeant [ 05/Nov/13 9:52 AM ]

This exposes that polymorphic protocols are completely untested :/

This issue is fixed in 0.2.16: https://github.com/clojure/core.typed/commit/4474f76eea1e966523914c5455761f1e7b8542c8

Also updated the doc: https://github.com/clojure/core.typed/commit/216454c8c292d68dfa1ac1cd5ce36e00f43cc1c8

This doesn't mean polymorphic protocols/datatypes are usable yet, but it's progress. Working on it.





[CTYP-105] derive/subtype failure on hashmap Created: 19/Feb/14  Updated: 19/Feb/14  Resolved: 19/Feb/14

Status: Resolved
Project: core.typed
Component/s: None
Affects Version/s: None
Fix Version/s: None

Type: Defect Priority: Major
Reporter: Cees van Kemenade Assignee: Ambrose Bonnaire-Sergeant
Resolution: Completed Votes: 0
Labels: None
Environment:

linux/java + core.typed 0.2.31


Attachments: File failDerive2.clj    

 Description   

The attached code fails when checking the namespace, as core.typed incorrectly assumes that the return-type is incorrect. Part of error-output is included as string ErrInfo and compared for difference with the return-type from the proto-type of the failing function.

When you load the file the analysis of the error-message is shown (run check-ns to see the message)



 Comments   
Comment by Ambrose Bonnaire-Sergeant [ 19/Feb/14 3:00 AM ]

Fixed in 0.2.32: https://github.com/clojure/core.typed/commit/0e0a41c917ef027bf47ab6ed90955881975a7a13

Comment by Cees van Kemenade [ 19/Feb/14 12:16 PM ]

Great!

Thanks for the rapid fix.
Tested it it and it works fine.

C.





[CTYP-107] automatic pre- and post-conditions for unchecked code Created: 19/Feb/14  Updated: 04/Mar/14  Resolved: 27/Feb/14

Status: Resolved
Project: core.typed
Component/s: None
Affects Version/s: None
Fix Version/s: None

Type: Enhancement Priority: Minor
Reporter: Cees van Kemenade Assignee: Ambrose Bonnaire-Sergeant
Resolution: Completed Votes: 0
Labels: None


 Description   

Currently it is fairly difficult to use core.typed to achieve/ensure production quality code. Often it happens that part of your own code does not pass the core.typed checker. Of course you can add the :no-check to your function, but in that case all type-checking is bypassed for this function and it can inject arbitrary data into the rest of your program (the type-signature is not enforced anymore.

It would be possible to prevent this issue if core.typed offered the opportunity to derive pre- and post-conditions based on the type-signature. This way unchecked code would use run-time checking as a fall-back scenario to prevent unchecked code could inject arbitrary data into the rest of your program (like the schema-library by prismatic, but using the core.typed annotations instead of schemas).
One step beyond would be to:
1. Use dynalint for the unchecked code (if this is possible)
2. Generate wrappers with pre- and post-conditions for external libaries.



 Comments   
Comment by Ambrose Bonnaire-Sergeant [ 19/Feb/14 7:08 AM ]

This is highly desirable. Typed Racket does an excellent job, so it's simple enough to follow their lead.

The biggest issue is actually the time it takes to load the type checker. I don't want to impose this on anyone who isn't generating contracts from types.

I've pondered writing a simpler intermediate AST representation of types that requires no further loading of the type system. It would look very much like the output of tools.analyzer or CLJS analysis. When a global annotation is added, it would be converted to an AST (very cheaply). Converting this AST to a contract would be less robust than using the type manipulating utilities of the full type checker, but the idea is that a type contract might generate incorrectly, but a "check-ns" always tells you after the fact.

It sucks, but some people rely on the zero load-time of core.typed (it's a pretty sweet property too!).

In short, I've thought a lot about this, and it's a glaring missing component in a gradual type system.

Comment by Cees van Kemenade [ 19/Feb/14 1:37 PM ]

I can imagine the dilemma. Core-typed has zero slow-down as all checking is taken off-line. I can image that the launch of the type-system at runtime is a significant overhead. To prevent this overhead the preparations of the type-checking should be brought to compile-time. Your suggestion of generating AST representations does this.

I was considering generation of source-code at compile time to generate ordinary pre- and post-conditions, or to wrap a function symbol with some checks. However, it seemed to be less than trivial to locate where the ann-macro store the type-annotations. I suppose the limited documention on this type of internal.

I guess the manual insertion of clojure pre- and post-conditions on unchecked functionscurrently is the best way out for the time being.

Comment by Ambrose Bonnaire-Sergeant [ 27/Feb/14 8:46 AM ]

Added clojure.core.typed/pred in 0.2.34. This is about as good as it's going to get for now.

Comment by Cees van Kemenade [ 28/Feb/14 2:31 AM ]

Thanks, this is a great tool to implement a postcondition for a function that has a ^:no-check directive. This way you can prevent that unchecked functions insert incorrect data into your process-flow.
Cool!

Comment by Cees van Kemenade [ 04/Mar/14 2:58 PM ]

Note:
When using this I noticed that (core.typed/pred T) returns an internal error when your type is (Seqable X) or your type contains a (Seqable X) somewhere at a nested level.

However, doing the transformation (Seqable X) --> (core.typed/Coll X)
works nicely.





[CTYP-130] add HSequential type Created: 31/Mar/14  Updated: 10/Apr/14  Resolved: 10/Apr/14

Status: Resolved
Project: core.typed
Component/s: None
Affects Version/s: None
Fix Version/s: None

Type: Enhancement Priority: Minor
Reporter: Di Xu Assignee: Ambrose Bonnaire-Sergeant
Resolution: Completed Votes: 0
Labels: None

Attachments: Text File ambrose-hsequential-v3-1.patch     Text File ambrose-various-fixes.patch     File hsequential-demo.diff     File hsequential.diff     File hsequential-v2.diff     File hsequential-v3.diff    

 Description   

Hi, Ambrose

I've implemented part of HSequential as we discussed in CTYP-126.

But currently

(t/cf (fn [& y] (if (empty? y) nil (first y))))

will got Type error, saying

(I (HSequential [Any *]) clojure.core.typed/NonEmptyCount)

can't be applied to

(HSequential [x Any *])

it's easy to fix via adding special case in `upcast-to-HSequential` in patch, but this maybe not very general, I also think even `upcast-to-HSequential` isn't general at all.

How should I solve this error?



 Comments   
Comment by Ambrose Bonnaire-Sergeant [ 31/Mar/14 6:01 AM ]

You should call subtype directly with those two types and add printlns until the culprit is obvious.

There's a nice sub? macro in c.c.t.test.core

There's a bug in your second subtype? case; the test should be (HSequential? s), not (HSequential? t).

Comment by Ambrose Bonnaire-Sergeant [ 31/Mar/14 6:03 AM ]

I believe if you fix the subtype case, the left hand side will have a CountRange that satisfies the right.

Comment by Ambrose Bonnaire-Sergeant [ 31/Mar/14 6:08 AM ]

Hmm I see what you were trying to do. I suggest following HMap's subtyping.

Make `upcast-HSequential` which is [HSequential -> Type]. It always returns a slightly less accurate type, which is still useful to continue the subtyping check.

The subtyping case should test (HSequential? s) and then (subtype (upcast-HSequential s) t).

Comment by Ambrose Bonnaire-Sergeant [ 31/Mar/14 6:12 AM ]

I think upcast-HSequential should do something like

(HSequential [Any])
=>
(I (t/Coll Any) clojure.lang.Sequential (ExactCount 1))

Comment by Ambrose Bonnaire-Sergeant [ 31/Mar/14 6:23 AM ]

A nice thing we can do with HSequential is move most of the custom logic for (subtype HVec HVec), (subtype HList HList) and (subtype HSeq HSeq) into one place.

All we need is {HVec,Hlist,HSeq}->HSequential functions, and then we can have subtyping cases like

(and (HVec? s) (HVec? t))
(subtype (HVec->HSequential s) (HVec->HSequential t))

This also applies to cs_gen.

Comment by Di Xu [ 31/Mar/14 8:19 AM ]

Or, could we add special handler in `In` to handle the case of intersection of HSequential and CountRange, like change (In (HSequential [a b c *]) (CountRange 1 5)) -> (HSequential [a b c c c]) and (In (HSequential [a *]) (CountRange 1)) -> (HSequential [a a *])

something like that?

Comment by Ambrose Bonnaire-Sergeant [ 31/Mar/14 9:08 AM ]

I'm not sure. The first example would actually be (U (HSequential [a b]) (HSequential [a b c]) (HSequential [a b c c]) (HSequential [a b c c c]).

It might generate very large types.

Either way the subtype style I described is still needed.

Comment by Di Xu [ 01/Apr/14 10:55 PM ]

Well, I think we're talking about two different problem here, the problem you're talking about is how to compare HSequential with other type, the one I'm talking about is how to use CountRange to extends HSequential type, because

(HSequential [Any *])

can never be the subtype of

(HSequential [Any Any *])

, only when we put constraint that the count of first one is at least 1. Right?

So, I'll first extends `In` as I described here, and leave your solution to future. For the efficience consideration, I think it's not a big problem, because most CountRange just specified low bound as `empty?`, so we don't need to generate a large set very often.

Comment by Ambrose Bonnaire-Sergeant [ 03/Apr/14 6:36 AM ]

I was responding that (HSequential [a b c *]) (CountRange 1 5)) -> (HSequential [a b c c c])
is unsound, and should be (U (HSequential [a b]) (HSequential [a b c]) (HSequential [a b c c]) (HSequential [a b c c c]).

Comment by Ambrose Bonnaire-Sergeant [ 03/Apr/14 6:43 AM ]

Perhaps a better idea is to add an extra field in HSequential/HList etc. for a CountRange. Then (HSequential [a b c *]) (CountRange 1 5)) -> (HSequential [a b c *] :count (CountRange 1 5))

That way there's no chance of generating massive types, and we can utilise that information in subtyping to simplify the HSequential at the last minute.

Comment by Di Xu [ 03/Apr/14 9:59 AM ]

Wow, that's a good idea. So when (> (:lower c) (count (:types t))) we should extends the types with (repeat (:rest t)), but I don't know what to do with drest. I will mark it unsupport.

There're a problem makes me complete lost. With hsequential-demo.diff patch, after changing the r/-hsequential into r/-hvec there'll be no problem testing

lein test :only clojure.core.typed.test.core/annotate-user-defined-ploydot

but after changing back to r/-hsequential, there'll be a type error. I don't know what I'm missing, I've debugged it for a whole day and still feel lost. I've greped the code that support HVec in fold_default.clj, frees.clj, promote_demote.clj and subst.clj. And copy the code to implement HSequential, but still got type error. I don't know what's going on here.

ps. the bug you mentioned in the first comment is not a bug, I was going to test if t is HSequential or not, and upcast s to HSequential if so, just like the {HVec,Hlist,HSeq}->HSequential you menntioned.

Comment by Ambrose Bonnaire-Sergeant [ 03/Apr/14 1:18 PM ]

I found a bunch of (pre-existing) bugs and added a few cases you were missing. I started getting the same type error for HVec/HSequential; hopefully the attached patch will help you continue.

Comment by Di Xu [ 07/Apr/14 3:02 AM ]

Well, turns out we don't need CountRange to pass the test case, but we also can't pass it with `first`, because both function and arguments have free variable, so change the test case to concrete type.

Comment by Ambrose Bonnaire-Sergeant [ 07/Apr/14 5:07 AM ]

The rest arguments of a function are actually ISeq's, not just Sequentials. We should be calling -hseq instead of -hsequential in the :fn check method.

If you `mvn test` you'll see what this breaks.

Next step:

  • add rest/drest to HSeq (and HList if you're enthusiastic)
  • change -hsequential to -hseq
Comment by Di Xu [ 08/Apr/14 9:06 AM ]

added, but failed in core/mapentry-first-test, core/hvec-ops and core/variable-hvec-test

I think those cases is beyond my current ability, it involve infer type variable in HSequential.

also failed in clojure.core.typed.test.fail.recur-non-seq-rest and clojure.core.typed.test.fail.recur-empty-seq

I don't know why should these cases fail. And confused by check-recur, which requires rest-arg-type has at least one element.

Comment by Ambrose Bonnaire-Sergeant [ 08/Apr/14 9:44 AM ]

Thanks, having a look.

You've added a few subtyping cases that are unsound. I'll fix them in my next patch.

It's important that (subtype? S T) only returns true if we can insert S in all places T is accepted.

(subtype? (HVec [1]) (HSequential [1]))
;=> true

(subtype? (HSequential [1]) (HVec [1]))
;=> false

(subtype? (HList [1] (HSeq [1]))
;=> true

(subtype? (HSeq [1]) (HList [1]))
;=> false

Comment by Ambrose Bonnaire-Sergeant [ 08/Apr/14 10:39 AM ]

Added ambrose-hsequential-v3-1.patch

Should pass all the tests. Please try it out.

Comment by Di Xu [ 09/Apr/14 2:27 AM ]

Oh, right.. it's unsound to do that in subtype, I was foolish that moment, sorry.

It's seems that you solved (first [1 'a]) by

(and (r/RClass? S)
     ((some-fn r/HeterogeneousVector? r/HSequential?) T))
         (if-let [[Sv] (seq
                         (filter (some-fn r/HeterogeneousVector? r/HSequential?)
                                 (map c/fully-resolve-type (c/RClass-supers* S))))]

right?

Well, I think we could close this ticket now. It spend such long time

Comment by Ambrose Bonnaire-Sergeant [ 09/Apr/14 2:40 AM ]

Actually I believe I added some extra cases for (cs-gen HVec HSeq) to fix (first [1 'a]).

That particular addition fixed (cf (-> {:a 1} first second) Number), because AMapEntry has a HVec ancestor and we used to rely on `second` having a HVec arity. Now `second` has a HSequential arity, we want to trigger this case if there's a (cs-gen RClass HSequential), instead of just (cs-gen RClass HVec).

Fantastic work, I'll merge this soon.

Comment by Ambrose Bonnaire-Sergeant [ 10/Apr/14 10:47 AM ]

Merged.





Generated at Fri Apr 18 23:31:32 CDT 2014 using JIRA 4.4#649-r158309.