core.match

Non deterministic match behavior for seqs when AOT

Details

  • Type: Defect Defect
  • Status: Closed Closed
  • Priority: Major Major
  • Resolution: Completed
  • Affects Version/s: None
  • Fix Version/s: None
  • Component/s: None
  • Labels:
    None
  • Environment:
    Clojure 1.3

    Leiningen 1.7.1 on Java 1.6.0_26

    core.match "0.2.0-alpha9"

Description

The following source example:

https://github.com/narkisr/match-issue/blob/master/src/match_issue/core.clj

The defrule macro calls a function called lhs that use the match library,

this function calls it self recursively and print out the rest of body left to be matched,

When running lein compile two different results appear (in non deterministic fashion):

(when $message :> Message (== level 6) :from (entry-point event-stream))
($message :> Message (== level 6) :from (entry-point event-stream))
(Message (== level 6) :from (entry-point event-stream))
((== level 6) :from (entry-point event-stream))

This is the bug, as (== level 6) should be matched by:

[([([(o :when operator?) f s] :seq) :as c & r] :seq)]

In other cases the output is (the correct one):

(when $message :> Message (== level 6) :from (entry-point event-stream))
($message :> Message (== level 6) :from (entry-point event-stream))
(Message (== level 6) :from (entry-point event-stream))
((== level 6) :from (entry-point event-stream))
(:from (entry-point event-stream))
(entry-point event-stream)
()
()

Activity

Hide
David Nolen added a comment -

All AOT related issues w/o patches are low priority for the foreseeable future. They are simply too time consuming to track down and I don't have the bandwidth at the moment. I will get to them eventually, but if you want movement on this please submit a patch. I will happily apply it!

Show
David Nolen added a comment - All AOT related issues w/o patches are low priority for the foreseeable future. They are simply too time consuming to track down and I don't have the bandwidth at the moment. I will get to them eventually, but if you want movement on this please submit a patch. I will happily apply it!
David Nolen made changes -
Field Original Value New Value
Priority Critical [ 2 ] Minor [ 4 ]
Hide
Ronen Narkis added a comment -

Hey David,

After going through the source code and some more in depth look I managed to understand what was going on, first iv stepped through the different stages

(def m (emit-matrix ['body]
'( [(['when & r] :seq)] (lhs r)
[(['accumulate & r] :seq)] (<< "accumulate(~(lhs r))")
[([bind ':> & r] :seq)] (<< "~{bind}:~(lhs r)"); pattern with bind
[([(t :when is-type?) & r] :seq)] (<< "{t}(lhs r)"); pattern
[([([(f :when acc-fn?) & args] :seq) & r] :seq)] (<< ",{f}{args}~(lhs r)" ); accumulate function
[([':from dest & r] :seq)] (<< "from ~(lhs dest) ~(lhs r)")
[([':over win & r] :seq)] (<< "over ~(window win) ~(lhs r)")
[([([(o :when operator?) f s] :seq) :as c & r] :seq)] (<< "(~(reduce str (map pr-str (to-infix c)))) ~(lhs r)")
[(['entry-point point & r] :seq)] (<< "entry-point \"~{point}\"~(lhs r)")
:else ""
)))

(def c (cm/compile m))

(pprint c)

(pprint (executable-form c))

The last pprint failed aot compilation:

$ lein compile

...
Caused by: java.lang.ClassCastException: clojure.core.match.WildcardPattern cannot be cast to java.lang.String

Which was very weird as I could clearly see that:

(deftype WildcardPattern [sym _meta]
IWildcardPattern
(sym [_] sym)
clojure.lang.IObj
(meta [_] _meta)
(withMeta [_ new-meta]
(WildcardPattern. sym new-meta))
Object
(toString [_]
(str sym)))

Then iv also tried to enabled trace:

(set-trace! true)

Which resulted with:

$ lein compile

...

TRACE: DAG: Column 0 : [#<SeqPattern #<LiteralPattern window> #<LiteralPattern :time> #<WildcardPattern t> #<WildcardPattern unit>>]
TRA at clojure.lang.Compiler.analyzeSeq(Compiler.java:6416)
at clojure.lang.Compiler.analyze(Compiler.java:6216)
at clojure.lang.Compiler.analyze(Compiler.java:6177)
at clojure.lang.Compiler$BodyExpr$Parser.parse(Compiler.java:5572)
at clojure.lang.Compiler$FnMethod.parse(Compiler.java:5008)
at clojure.lang.Compiler$FnExpr.parse(Compiler.java:3629)
at clojure.lang.Compiler.analyzeSeq(Compiler.java:6407)
at clojure.lang.Compiler.analyze(Compiler.java:6216)
at clojure.lang.Compiler.analyzeSeq(Compiler.java:6397)
at clojure.lang.Compiler.analyze(Compiler.java:6216)
at clojure.lang.Compiler.access$100(Compiler.java:37)
at clojure.lang.Compiler$DefExpr$Parser.parse(Compiler.java:492)
at clojure.lang.Compiler.analyzeSeq(Compiler.java:6409)
at clojure.lang.Compiler.analyze(Compiler.java:6216)
at clojure.lang.Compiler.analyze(Compiler.java:6177)
at clojure.lang.Compiler.compile1(Compiler.java:6980)
at clojure.lang.Compiler.compile(Compiler.java:7046)
at clojure.lang.RT.compile(RTCE: DAG: Add switch-node on occurrence exp

The most weird thing was that on second compile it all went well, this raised a flag and my suspicion was that the core.match classes got compiled on the first run:

$ ls classes/

match$analyze_actions$analyze_action__1166.class
match$analyze_actions.class
match$backtrack_expr.class
match$bind_node.class
match$catch_error.class
match$check_matrix_args$check_pattern__1150.class
...

Then iv decided to AOT compile clojure.core match:

$ git checkout core.match-0.2.0-alpha9
$ git diff project.clj
-(defproject match "0.2.0-alpha10-SNAPSHOT"
+(defproject match "0.2.0-alpha10-aot"
:description "Optimized pattern matching and predicate dispatch for Clojure"
:source-path "src/main/clojure"

  • :test-path "src/test/clojure")
    + :test-path "src/test/clojure"
    + :aot [clojure.core.match]
    + )

After using the aot jar it all went smooth including passing my non deterministic case (which is deterministic but very confusing to track down),

In order to reproduce it its important to run:

$ lein clean
$ rm -rf lib

And only then run

$ lein compile

Otherwise the classes dir might contain AOT'ed classes and make it seem that even non-aot jar works fine (its enough to run lein compile once to cause the classes to get compiled),

The fix in my case is simple, just AOT ns.

Im not sure if this Is a bug in the way Clojure AOT its classes but I think that adding the AOT'ed classes to the default core match distro is a reasonable workaround

BTW when using master (and not the tag) iv stumbled upon:

Exception in thread "main" java.lang.AssertionError: Assert failed: Unknown predicate in [is-type?]

Which is a pred defined in my ns.

Show
Ronen Narkis added a comment - Hey David, After going through the source code and some more in depth look I managed to understand what was going on, first iv stepped through the different stages (def m (emit-matrix ['body] '( [(['when & r] :seq)] (lhs r) [(['accumulate & r] :seq)] (<< "accumulate(~(lhs r))") [([bind ':> & r] :seq)] (<< "~{bind}:~(lhs r)"); pattern with bind [([(t :when is-type?) & r] :seq)] (<< "{t}(lhs r)"); pattern [([([(f :when acc-fn?) & args] :seq) & r] :seq)] (<< ",{f}{args}~(lhs r)" ); accumulate function [([':from dest & r] :seq)] (<< "from ~(lhs dest) ~(lhs r)") [([':over win & r] :seq)] (<< "over ~(window win) ~(lhs r)") [([([(o :when operator?) f s] :seq) :as c & r] :seq)] (<< "(~(reduce str (map pr-str (to-infix c)))) ~(lhs r)") [(['entry-point point & r] :seq)] (<< "entry-point \"~{point}\"~(lhs r)") :else "" ))) (def c (cm/compile m)) (pprint c) (pprint (executable-form c)) The last pprint failed aot compilation: $ lein compile ... Caused by: java.lang.ClassCastException: clojure.core.match.WildcardPattern cannot be cast to java.lang.String Which was very weird as I could clearly see that: (deftype WildcardPattern [sym _meta] IWildcardPattern (sym [_] sym) clojure.lang.IObj (meta [_] _meta) (withMeta [_ new-meta] (WildcardPattern. sym new-meta)) Object (toString [_] (str sym))) Then iv also tried to enabled trace: (set-trace! true) Which resulted with: $ lein compile ... TRACE: DAG: Column 0 : [#<SeqPattern #<LiteralPattern window> #<LiteralPattern :time> #<WildcardPattern t> #<WildcardPattern unit>>] TRA at clojure.lang.Compiler.analyzeSeq(Compiler.java:6416) at clojure.lang.Compiler.analyze(Compiler.java:6216) at clojure.lang.Compiler.analyze(Compiler.java:6177) at clojure.lang.Compiler$BodyExpr$Parser.parse(Compiler.java:5572) at clojure.lang.Compiler$FnMethod.parse(Compiler.java:5008) at clojure.lang.Compiler$FnExpr.parse(Compiler.java:3629) at clojure.lang.Compiler.analyzeSeq(Compiler.java:6407) at clojure.lang.Compiler.analyze(Compiler.java:6216) at clojure.lang.Compiler.analyzeSeq(Compiler.java:6397) at clojure.lang.Compiler.analyze(Compiler.java:6216) at clojure.lang.Compiler.access$100(Compiler.java:37) at clojure.lang.Compiler$DefExpr$Parser.parse(Compiler.java:492) at clojure.lang.Compiler.analyzeSeq(Compiler.java:6409) at clojure.lang.Compiler.analyze(Compiler.java:6216) at clojure.lang.Compiler.analyze(Compiler.java:6177) at clojure.lang.Compiler.compile1(Compiler.java:6980) at clojure.lang.Compiler.compile(Compiler.java:7046) at clojure.lang.RT.compile(RTCE: DAG: Add switch-node on occurrence exp The most weird thing was that on second compile it all went well, this raised a flag and my suspicion was that the core.match classes got compiled on the first run: $ ls classes/ match$analyze_actions$analyze_action__1166.class match$analyze_actions.class match$backtrack_expr.class match$bind_node.class match$catch_error.class match$check_matrix_args$check_pattern__1150.class ... Then iv decided to AOT compile clojure.core match: $ git checkout core.match-0.2.0-alpha9 $ git diff project.clj -(defproject match "0.2.0-alpha10-SNAPSHOT" +(defproject match "0.2.0-alpha10-aot" :description "Optimized pattern matching and predicate dispatch for Clojure" :source-path "src/main/clojure"
  • :test-path "src/test/clojure") + :test-path "src/test/clojure" + :aot [clojure.core.match] + )
After using the aot jar it all went smooth including passing my non deterministic case (which is deterministic but very confusing to track down), In order to reproduce it its important to run: $ lein clean $ rm -rf lib And only then run $ lein compile Otherwise the classes dir might contain AOT'ed classes and make it seem that even non-aot jar works fine (its enough to run lein compile once to cause the classes to get compiled), The fix in my case is simple, just AOT ns. Im not sure if this Is a bug in the way Clojure AOT its classes but I think that adding the AOT'ed classes to the default core match distro is a reasonable workaround BTW when using master (and not the tag) iv stumbled upon: Exception in thread "main" java.lang.AssertionError: Assert failed: Unknown predicate in [is-type?] Which is a pred defined in my ns.
David Nolen made changes -
Priority Minor [ 4 ] Major [ 3 ]
Hide
David Nolen added a comment -

This sounds very similar to the other AOT bug reported by Kevin Lynagh MATCH-53 which is fixed in master, unless I hear different this is resolved.

Show
David Nolen added a comment - This sounds very similar to the other AOT bug reported by Kevin Lynagh MATCH-53 which is fixed in master, unless I hear different this is resolved.
David Nolen made changes -
Resolution Completed [ 1 ]
Status Open [ 1 ] Resolved [ 5 ]
David Nolen made changes -
Status Resolved [ 5 ] Closed [ 6 ]

People

Vote (0)
Watch (0)

Dates

  • Created:
    Updated:
    Resolved: