<< Back to previous view

[CLJ-1947] Add vec-of spec Created: 05/Jun/16  Updated: 28/Jun/16  Resolved: 28/Jun/16

Status: Closed
Project: Clojure
Component/s: None
Affects Version/s: Release 1.9
Fix Version/s: None

Type: Enhancement Priority: Minor
Reporter: Maarten Truyens Assignee: Unassigned
Resolution: Completed Votes: 0
Labels: spec


 Description   

It would be great to add a "vec-of" (and perhaps also a "set-of") Spec, similar to the already existing map-of. I find myself writing (coll-of ::foo []) writing too often.



 Comments   
Comment by Alex Miller [ 28/Jun/16 10:01 PM ]

With 1.9.0-alpha8, you can now get this same effect using:

(s/coll-of ::foo :kind vector?)
(s/coll-of ::foo :kind set?)




[CLJ-1946] improve error messages for `map-of` spec Created: 04/Jun/16  Updated: 28/Jun/16  Resolved: 28/Jun/16

Status: Closed
Project: Clojure
Component/s: None
Affects Version/s: None
Fix Version/s: None

Type: Enhancement Priority: Major
Reporter: Chris Price Assignee: Unassigned
Resolution: Completed Votes: 0
Labels: errormsgs, spec


 Description   

When using a map-of spec where the value predicate refers to another spec, the error message if a value does not conform does not seem like it is as helpful as it could be:

(spec/def ::myint integer?)
(spec/explain
 (spec/map-of string? ::myint)
 {"x" 1 "y" "not an int!"})
=> :user.swagger-ui-service/myint
val: {"x" 1, "y" "not an int!"} fails predicate: (coll-checker (tuple string? :user.swagger-ui-service/myint))

The explain result doesn't indicate which key/value pair in the map failed to conform, and it also doesn't make it clear that the integer? predicate is ultimately the one that caused the failure.

From reading through the introduction and docs, it seemed like error messaging was a primary motivation for spec, so any improvements that might be possible to this error message would be extremely valuable.



 Comments   
Comment by Chris Price [ 04/Jun/16 11:22 AM ]

This becomes particularly important with deeply-nested specs and/or large maps.

Comment by Sean Corfield [ 04/Jun/16 11:53 PM ]

In my opinion, the failure should use the spec name, not the underlying predicate function, in the message – isn't that the whole point of using named specs in this?

(as for explaining the value that failed, I agree on the surface but haven't looked at the implementation in detail to see if there might be a good reason)

Comment by Chris Price [ 06/Jun/16 12:23 PM ]

If nothing else, it's inconsistent with what gets logged for other types of specs:

(spec/explain
 (spec/cat :myint ::myint)
 ["hi"])
 
In: [0] val: "hi" fails spec: :puppetlabs.trapperkeeper.main/myint at: [:myint] predicate: integer?
=> nil
(spec/explain
 (spec/tuple ::myint)
 ["hi"])
 
In: [0] val: "hi" fails spec: :puppetlabs.trapperkeeper.main/myint at: [0] predicate: integer?
=> nil

Having spent a good chunk of a day hacking on a project that involved some very deeply-nested specs, I can definitively say that the logging from cat and tuple was much easier to debug. With those, I could keep poking at my "production" code via the REPL, tweaking things until the specs were correct. With map-of, the only way I was able to debug was to copy/paste the entire failed val into the REPL and cobble together a one-off spec/explain form to repro, and then delete things from it until I got past the map-of so that the error was less opaque. Then once I fixed the actual issue I'd need to copy/paste the REPL stuff back into my "production" code. It wasn't impossible, but it was a much more tedious workflow than when dealing with errors from cat or tuple.

Comment by Alex Miller [ 13/Jun/16 3:50 PM ]

The issues here is due to the way map-of and coll-of sample their contents rather than fully conforming all of them. This is done for performance but is not what most people expect. It's likely there will be some more additions in this area.

Comment by Alex Miller [ 28/Jun/16 10:00 PM ]

As of 1.9.0-alpha8, map-of now conforms all entries and the error you'll see is:

In: ["y" 1] val: "not an int!" fails spec: :user/myint at: [1] predicate: integer?




[CLJ-1964] rmap / *recursion-limit* not respected through custom generators Created: 19/Jun/16  Updated: 28/Jun/16  Resolved: 28/Jun/16

Status: Closed
Project: Clojure
Component/s: None
Affects Version/s: Release 1.9
Fix Version/s: None

Type: Defect Priority: Major
Reporter: Leon Grapenthin Assignee: Unassigned
Resolution: Completed Votes: 0
Labels: generator, spec

Approval: Triaged

 Description   

In all cases where a custom generator is used, the recursion limit is not respected.

This limitation becomes clear when one tries to build a recursive spec around e. g. s/coll-of because it uses a custom generator via s/coll-gen. Running s/exercise on it quickly blows the stack.

Here is an example for illustration with s/map-of

(s/def ::map-tree (s/map-of keyword? (s/or :tree ::map-tree :leaf nil?)))

Even though s/or implements recursion checking, it is deceived here and not able to detect itself being called subsequently because the custom generator of s/coll-of (used in s/map-of) doesn't/can't pass rmap (keeping track of recursion calls) through to s/or's gen*.

For the concrete case, coll-of-impl could be implemented that would implement a gen* that passes on rmap.

For the general case of custom generators the challenge would be that test.check generators don't take and pass on rmap to generators of specs they potentially reuse and that there is no well-defined behavior for what they themselves should do when the recursion-limit has been reached. Ideas are:

  • reduce generator size when recursion is detected (this is the strategy used in recursive specs of test.check use(d)?)
  • expose the recursion-limit / rmap mechanism to the user so that custom generators can pass it on to subsequent calls of specs. E. g. a custom generator is passed a context object that it should pass to s/gen as an optional argument


 Comments   
Comment by Leon Grapenthin [ 19/Jun/16 5:08 PM ]

I have changed the issue because in the former description I had made some assumptions that I could prove incorrect by studying the implementation a bit more.

Comment by Alex Miller [ 25/Jun/16 9:37 AM ]

Please re-check this after the next alpha - there are a lot of changes happening in this area.

Comment by Alex Miller [ 28/Jun/16 9:58 PM ]

As of Clojure 1.9.0-alpha8, due to changes in map-of etc, s/exercise now works on this example.





[CLJ-1963] clojure.spec/map-of has confusing error message when input is not a map Created: 15/Jun/16  Updated: 28/Jun/16  Resolved: 28/Jun/16

Status: Closed
Project: Clojure
Component/s: None
Affects Version/s: Release 1.9
Fix Version/s: None

Type: Defect Priority: Major
Reporter: Russell Mull Assignee: Unassigned
Resolution: Completed Votes: 0
Labels: errormsgs, spec


 Description   

When using map-of specs, the error message given when checking a non-map value is less than enlightening:

user> (require '[clojure.spec :as s])
nil
user> (s/def ::int-map (s/map-of integer? integer?))
:user/int-map
user> (s/explain ::int-map :not-a-map)
val: :not-a-map fails spec: :user/int-map predicate: (coll-checker (tuple integer? integer?))
nil

This can be worked around to some degree by requiring that the value be a map explicitly:

user> (s/def ::fancy-int-map (s/and map? ::int-map))
:user/fancy-int-map
user> (s/explain ::fancy-int-map :not-a-map)
val: :not-a-map fails spec: :user/fancy-int-map predicate: map?
nil

The definition

map-of
looks like it's trying to do just this, but the
map?
predicate comes second for some reason.



 Comments   
Comment by Alex Miller [ 28/Jun/16 9:54 PM ]

map-of implementation changed a lot in alpha8 and you will now see the error for your first example as:

user=> (s/explain ::int-map :not-a-map)
val: :not-a-map fails spec: :user/int-map predicate: map?




[CLJ-1971] Update docstring of empty? to suggest not-empty instead of seq Created: 27/Jun/16  Updated: 28/Jun/16  Resolved: 28/Jun/16

Status: Closed
Project: Clojure
Component/s: None
Affects Version/s: None
Fix Version/s: None

Type: Enhancement Priority: Trivial
Reporter: Daniel Compton Assignee: Unassigned
Resolution: Declined Votes: 0
Labels: None


 Description   

The docstring for empty? says

clojure.core/empty? [coll]
Returns true if coll has no items - same as (not (seq coll)). Please use the idiom (seq x) rather than (not (empty? x))

Would it make more sense to suggest using not-empty, instead of seq here?



 Comments   
Comment by Alex Miller [ 28/Jun/16 9:20 AM ]

No, the recommended idiom is still to use seq as a termination condition in this case.





[CLJ-1969] :as form is unbound when no optional keyword arguments is passed even though :or form is provided Created: 24/Jun/16  Updated: 24/Jun/16  Resolved: 24/Jun/16

Status: Closed
Project: Clojure
Component/s: None
Affects Version/s: Release 1.7
Fix Version/s: None

Type: Defect Priority: Trivial
Reporter: Lars Andersen Assignee: Unassigned
Resolution: Declined Votes: 0
Labels: None


 Description   
(defn fn-with-kw-opts [& {:keys [opt] :or {opt 1} :as options}]
  (println "opt " opt "options " options))

user> (fn-with-kw-opts)
;;=> opt  1 options  nil

user> (fn-with-kw-opts :opt 2)
;;=> opt  2 options  {:opt 2}

I would expect options to be bound to the default value when no keyword argument is passed.



 Comments   
Comment by Alex Miller [ 24/Jun/16 7:36 AM ]

:as binds to the original value passed in and will never include values from :or. :or is used to provide defaults for each local being bound when that local is missing in the input.

In the case of (fn-with-kw-opts), the incoming value is nil so options is bound to nil.

Comment by Alex Miller [ 24/Jun/16 7:38 AM ]

Working as designed.





[CLJ-1919] Destructuring support for namespaced keys and syms Created: 27/Apr/16  Updated: 23/Jun/16  Resolved: 23/Jun/16

Status: Closed
Project: Clojure
Component/s: None
Affects Version/s: None
Fix Version/s: Release 1.9

Type: Enhancement Priority: Critical
Reporter: Alex Miller Assignee: Alex Miller
Resolution: Completed Votes: 1
Labels: destructuring

Attachments: Text File clj-1919-2.patch     Text File clj-1919.patch    
Patch: Code and Test
Approval: Ok

 Description   

Expand destructuring to better support a set of keys (or syms) from a map when the keys share the same namespace.

Example:

(def m {:person/first "Darth" :person/last "Vader" :person/email "darth@death.star"})

(let [{:keys [person/first person/last person/email]} m]
  (format "%s %s - %s" first last email))

Proposed: The special :keys and :syms keywords used in associative destructuring may now have a namespace (eg :person/keys). That namespace will be applied during lookup to all listed keys or syms when they are retrieved from the input map.

Example (also uses the new literal syntax for namespaced maps from CLJ-1910):

(def m #:person{:first "Darth" :last "Vader" :email "darth@death.star"})

(let [{:person/keys [first last email]} m]
  (format "%s %s - %s" first last email))
  • The key list after :ns/keys should contain either non-namespaced symbols or non-namespaced keywords. Symbols are preferred.
  • The key list after :ns/syms should contain non-namespaced symbols.
  • As :ns/keys and :ns/syms are read as normal keywords, auto-resolved keywords work as well: ::keys, ::alias/keys, etc.
  • Clarification - the :or defaults map always uses non-namespaced symbols as keys - that is, they are always the same as the locals being created (not the keys being looked up in the map). No change in behavior here, just trying to be explicit - this was not previously well-documented for namespaced key lookup and was broken. The attached patch fixes this behavior.

Patch: clj-1919-2.patch



 Comments   
Comment by Alex Miller [ 06/Jun/16 7:26 AM ]

This patch now needs to be re-worked on top of https://github.com/clojure/clojure/commit/0aa346766c4b065728cde9f9fcb4b2276a6fe7b5

Comment by Alex Miller [ 14/Jun/16 9:34 AM ]

Rebased patch to current master. No semantic changes as they didn't actually conflict, just were close enough to confuse git.





[CLJ-1910] Namespaced maps Created: 07/Apr/16  Updated: 23/Jun/16  Resolved: 23/Jun/16

Status: Closed
Project: Clojure
Component/s: None
Affects Version/s: None
Fix Version/s: Release 1.9

Type: Enhancement Priority: Critical
Reporter: Alex Miller Assignee: Alex Miller
Resolution: Completed Votes: 1
Labels: print, reader

Attachments: Text File clj-1910-2.patch     Text File clj-1910.patch    
Patch: Code and Test
Approval: Ok

 Description   

A common usage of namespaced keywords and symbols is in providing attribute disambiguation in map contexts:

{:person/first "Han" :person/last "Solo" :person/ship 
  {:ship/name "Millenium Falcon" :ship/model "YT-1300f light freighter"}}

The namespaces provide value (disambiguation) but have the downside of being repetitive and verbose.

Namespaced maps are a reader (and printer) feature to specify a namespace context for a map.

  • Namespaced maps combine a default namespace with a map and yield a map.
  • Namespaced maps are reader macros starting with #: or #::, followed by a normal map definition.
    • #:sym indicates that sym is the default namespace for the map to follow.
    • #:: indicates that the default namespace auto-resolves to the current namespace.
    • #::sym indicates that sym should be auto-resolved using the current namespace's aliases OR any fully-qualified loaded namespace.
      • These rules match the rules for auto-resolved keywords.
  • A namespaced map is read with the following differences from normal maps:
    • A keyword or symbol key without a namespace is read with the default namespace as its namespace.
    • Keys that are not symbols or keywords are not affected.
    • Keys that specify an explicit namespace are not affected EXCEPT the special namespace _, which is read with NO namespace. This allows the specification of bare keys in a namespaced map.
    • Values are not affected.
    • Nested map keys are not affected.
  • The edn reader supports #: but not #:: with the same rules as above.
  • Maps will be printed in namespaced map form only when:
    • All map keys are keywords or symbols
    • All map keys are namespaced
    • All map keys have the same namespace

Examples:

;; same as above - notice you can nest #: maps and this is a case where the printer roundtrips
user=> #:person{:first "Han" :last "Solo" :ship #:ship{:name "Millenium Falcon" :model "YT-1300f light freighter"}}
#:person{:first "Han" :last "Solo" :ship #:ship{:name "Millenium Falcon" :model "YT-1300f light freighter"}}

;; effects on keywords with ns, without ns, with _ ns, and non-kw
user=> #:foo{:kw 1, :n/kw 2, :_/bare 3, 0 4}
{:foo/kw 1, :n/kw 2, :bare 3, 0 4}

;; auto-resolved namespaces (will use user as the ns)
user=> #::{:kw 1, :n/kw 2, :_/bare 3, 0 4}
:user/kw 1, :n/kw 2, :bare 3, 0 4}

;; auto-resolve alias s to clojure.string
user=> (require '[clojure.string :as s])
nil
user=> #::s{:kw 1, :n/kw 2, :_/bare 3, 0 4}
{:clojure.string/kw 1, :n/kw 2, :bare 3, 0 4}

;; to show symbol changes, we'll quote the whole thing to avoid evaluation
user=> '#::{a 1, n/b 2, _/c 3}
{user/a 1, n/b 2, c 3}

;; edn reader also supports (only) the #: syntax
user=> (clojure.edn/read-string "#:person{:first \"Han\" :last \"Solo\" :ship #:ship{:name \"Millenium Falcon\" :model \"YT-1300f light freighter\"}}")
#:person{:first "Han", :last "Solo", :ship #:ship{:name "Millenium Falcon", :model "YT-1300f light freighter"}}

Patch: clj-1910-2.patch

Screener notes:

  • Autoresolution supports fully-qualified loaded namespaces (like auto-resolved keywords)
  • TODO: pprint support for namespaced maps
  • TODO: printer flag to suppress printing namespaced maps


 Comments   
Comment by Nicola Mometto [ 08/Apr/16 3:57 AM ]

1- yes please. that's consistent with how tagged literals work.
2- no please. that would make the proposed syntax useless for e.g. Datomic schemas, for which I think this would be a good fit to reduce noise
3- yes please
4- yes please, consistency over print methods is important

Comment by Nicola Mometto [ 08/Apr/16 4:00 AM ]

Quoting from a post I wrote on the clojure-dev ML:

  • I really don't like the idea of special-casing `_` here, users are already confused about idioms like `[.. & _]` thinking that `_` is some special token/variable. Making it an actual special token in some occasion wouldn't help.
  • I also don't like how we're making the single `:` auto-qualify keywords when used within the context of a qualifying-map. Auto-qualifying keywords has always been the job of the double `::`, changing this would introduce (IMO) needless cognitive overhead.
  • The current impl treats `#:foo{'bar 1}` and `'#:foo{bar 1}` differently. I can see why is that, but the difference might be highly unintuitive to some.
  • The current proposal makes it feel like quote is auto-qualifying symbols , when that has always been the job of syntax-quote. I know that's not correct, but that's how it's perceived.

Here's an alternative syntax proposal that handles all those issues:

  • No #::, only #:foo or #::foo
  • No auto-resolution of symbols when the namespaced-map is quoted, only when syntax-quoted
  • No special-casing of `_`
  • No auto-resolution of single-colon keywords

Here's how the examples in the ticket description would look:

#:person{::first "Han", ::last "Solo", ::ship #:ship{::name "Millenium Falcon", ::model "YT-1300f light freighter"}}
;=> {:person/first "Han" :person/last "Solo" :person/ship {:ship/name "Millenium Falcon" :ship/model "YT-1300f light freighter"}}

#:foo{::kw 1, :n/kw 2, :bare 3, 0 4}
;=> {:foo/kw 1, :n/kw 2, :bare 3, 0 4}

{::kw 1, :n/kw 2, bare 3, 0 4}
;=> {:user/kw 1, :n/kw 2, :bare 3, 0 4}

Note in the previous example how we don't need `#::` at all – `::` already does that job for us

(require '[clojure.string :as s])
#::s{::kw 1, :n/kw 2, bare 3, 0 4}
;=> {:clojure.string/kw 1, :n/kw 2, :bare 3, 0 4}

`{a 1, n/b 2, ~'c 3}
;=> {user/a 1, n/b 2, c 3}

Again, no need for `#::` here, we can just rely on the existing auto-qualifying behaviour of `.

`#:foo{a 1, n/b 2}
;=> {foo/a 1, n/b 2}

I think this would be more consistent with the existing behaviour – it's basically just making `#:foo` or `#::foo` mean: in the top-level keys of the following map expression, resolve keywords/symbols as if ns was bound to `foo`, rather than introducing new resolution rules and special tokens.

I realize that this proposal wouldn't work with EDNReader as-is, given its lack of support for `::` and "`". I don't have a solution to that other than "let's just bite the bullet and implement them there too", but maybe that's not acceptable.

Comment by Alex Miller [ 08/Apr/16 8:45 AM ]

Nicola, thanks for the proposal, we talked through it. We share your dislike for :_/kw syntax and you should consider that a placeholder for this behavior for the moment - it may be removed or replaced before we get to a published release.

For the rest of it:

  • requiring syntax quote is a non-starter
  • supporting a mixture of default ns and the current ns is important and this is not possible with your proposal. Like #:foo{:bar 1 ::baz 2}.
  • there is a lot of value to changing the scope of a map without modifying the contents, which is an advantage of the syntax in the ticket
Comment by Christophe Grand [ 08/Apr/16 10:31 AM ]

Why restrict this feature to a single namespace? (this doesn't preclude a shorthand for the single mapping) I'd like to locally define aliases (and default ns).

Comment by Alex Miller [ 08/Apr/16 11:02 AM ]

We already have namespace level aliases. You can use :: in the map to leverage those aliases (independently from the default ns):

(ns app 
  (:require [my.domain :as d]
            [your.domain :as y]))

#::{:svc 1, ::d/name 2, ::y/name 3}

;;=> {:app/svc 1, :my.domain/name 2, :your.domain/y 3}
Comment by Christophe Grand [ 11/Apr/16 4:03 AM ]

Alex, if existing namespace level aliases are enough when there's more than one namespace used in the key set I fail to understand the real value of this proposal.

Okay I'm lying a little: there are no aliases in edn, so this would bring aliases to edn (and allows printers to factor/alias namespaces out). And for Clojure code you can't define an alias to a non-existing namespace – and I believe that this implementation wouldn't check namespace existence when resolving the default ns #:person{:name}.

Still my points hold for edn (and that's where the value of this proposal seems to be): why not allows local aliases too?

#:person #:employee/e {:name "John Smith", :e/eid "012345"}
;=> {:person/name "John Smith", :employee/eid "012345"}

I have another couple of questions:

  • should it apply to other datatypes?
  • should it be transitive?
Comment by Alex Miller [ 28/Apr/16 1:33 PM ]

New patch rev supports spaces between the namespace part #:foo and the map in both LispReader and EdnReader.





Generated at Tue Jun 28 22:41:10 CDT 2016 using JIRA 4.4#649-r158309.