<< Back to previous view

[CLJ-2011] clojure.walk.macroexpand-all will not properly expand macros that depend on &env Created: 23/Aug/16  Updated: 23/Aug/16

Status: Open
Project: Clojure
Component/s: None
Affects Version/s: Release 1.8, Release 1.9
Fix Version/s: None

Type: Defect Priority: Major
Reporter: Collin Bell Assignee: Unassigned
Resolution: Unresolved Votes: 0
Labels: macro, walk
Environment:

MacOSX, Clojure 1.9.0-alpha10, Java 1.8.0_45, CIDER 0.13.0snapshot (package: 20160602.809), nREPL 0.2.12



 Description   

(clojure.walk/macroexpand-all '(defn foo [a] (go [] a)))

Unhandled clojure.lang.ExceptionInfo
Could not resolve var: a
{:var a}

This is because go depends on &env and macroexpand-all does not handle &env.

The reason this issue is important is because it breaks the cider debugger for async.






[CLJ-1899] Add function transform-keys to clojure.walk Created: 08/Mar/16  Updated: 14/Mar/16

Status: Open
Project: Clojure
Component/s: None
Affects Version/s: Release 1.9
Fix Version/s: None

Type: Enhancement Priority: Major
Reporter: Rafal Szalanski Assignee: Unassigned
Resolution: Unresolved Votes: 3
Labels: walk
Environment:

OS X, Java 8, Clojure 1.8


Attachments: Text File clj1899.patch     Text File clj1899-review1.patch    
Patch: Code and Test
Approval: Prescreened

 Description   

In CLJ-1894 I proposed a patch to change clojure.walk/stringify-keys to include namespace if keywords use namespaces. I made a wrong assumption about backwards compatibility of that change, however I still think the behaviour is not exactly what it should be.

Interesting thing Alex Miller pointed out in his comment to CLJ-1894 is that stringify-keys and keywordize-keys are essentially the same function with a different transformation. I think having one function which does a deep transformation of map keys using a transformation supplied by user is a good idea and it could be used to simplify some Clojure libraries.

Proposal:

  • add clojure.walk/transform-keys to walk a map and transform all keys
  • use transform-keys in clojure.walk/stringify-keys & clojure.walk/keywordize-keys

Patch: clj1899-review1.patch

Screened by: Alex Miller



 Comments   
Comment by Rafal Szalanski [ 08/Mar/16 6:37 AM ]

CLJ-1899 patch

Comment by Alex Miller [ 10/Mar/16 9:20 AM ]

In the patch, transform-keys should take the arguments in the reverse order [m f] - generally for any function that is collection -> collection, the collection should be the first arg.

Comment by Rafal Szalanski [ 14/Mar/16 9:34 AM ]

CLJ-1899 patch addressing issues pointed by Alex miller.





[CLJ-1894] Include namespace when stringifying keys in maps Created: 22/Feb/16  Updated: 24/Feb/16  Resolved: 24/Feb/16

Status: Closed
Project: Clojure
Component/s: None
Affects Version/s: Release 1.9
Fix Version/s: None

Type: Enhancement Priority: Major
Reporter: Rafal Szalanski Assignee: Unassigned
Resolution: Declined Votes: 0
Labels: walk
Environment:

OS X, Java 8, Clojure 1.8


Attachments: Text File full-name.patch    
Patch: Code and Test

 Description   

I noticed that if one wants to stringify keys in map using clojure.walk/stringify-keys and the keywords or symbols contain namespaces, for example:

(clojure.walk/stringify-keys {:a 1 :b/c 2})

then the result will be equal to {"a" 1 "c" 2}. The docstring for clojure.walk/stringify-keys says:

Recursively transforms all map keys from keywords to strings.

which to my mind implies the namespace should be included in the result map so that reverse operation clojure.walk/keywordize-keys can re-create the initial map.

The patch I am proposing adds a function full-name to clojure.core namespace which returns full string representation of a keyword including the namespace (if it's present). I also modify clojure.walk/stringify-keys to use that function instead of clojure.core/name.

The change should be 100% compatible with any Clojure code out there. I am making an assumption that people who came up against this problem found a different way of solving that problem, even re-design everything to use keywords instead of strings. Keywords are one of the most commonly used parts of the language and have clear benefits over strings (i.e. they are functions).



 Comments   
Comment by Alex Miller [ 24/Feb/16 1:53 PM ]

I disagree with your assumption re current use - it's possible that the current behavior is desired (or at least relied-upon) by some existing caller. I'm not willing to change this default behavior.

Instead, I would note that stringify-keys and keywordize-keys are the same function with a different transformation. It would be better to extract that more generic function (transform-keys?), refactor stringify-keys and keywordize-keys in terms of it, and let users supply any transformation function they want.

I'm going to decline this ticket as is, as will not make the suggested change. The other idea would be a reasonable alternative ticket. (Although it does risk running into an expansion of purpose to include shallow/deep alternative and value transformations - all things I think are valuable and in common use).





[CLJ-1847] clojure.walk/walk returns a PersistentVector when the input is of type IMapEntry Created: 13/Nov/15  Updated: 15/Nov/15  Resolved: 15/Nov/15

Status: Closed
Project: Clojure
Component/s: None
Affects Version/s: Release 1.7
Fix Version/s: None

Type: Defect Priority: Minor
Reporter: Yehonathan Sharvit Assignee: Unassigned
Resolution: Declined Votes: 0
Labels: walk

Attachments: Text File fix_walk.patch    
Patch: Code

 Description   

`walk` is supposed to build up a data structure of the same type as its input.
With `clojure.lang.IMapEntry`, it doesn't work as expected.
The output is of type: `clojure.lang.PersisentVector`

For instance,
(class (walk/walk identity identity (first {:a 1})) ); clojure.lang.PersisentVector



 Comments   
Comment by Alex Miller [ 15/Nov/15 12:54 PM ]

Thanks for filing this. I spent some time looking at it and I don't think there is anything that needs to be done at this time.

Re your description, "`walk` is supposed to build up a data structure of the same type as its input." Regarding IMapEntry.... prior to 1.8, IMapEntry concrete types were descendants of AMapEntry, in particular the most common concrete type was MapEntry, which were used in PersistentHashMap.

In 1.8, PHM now returns PersistentVectors, which now implement IMapEntry. So, walk now takes a map entry (which is a PV) and returns a PV, which is an IMapEntry. So I believe the contract is still satisfied.

The new map-entry? predicate can be used to catch only 2-element PVs (not other counts) as map entries, however it is important to still consider whether you will incorrectly snare 2 element vectors that aren't in a map in this particular case. The current clojure.walk code however will have the same effective result in either case so the clojure.walk code does not need to change (and in fact non-PV entries still exist in sorted maps).





[CLJ-1240] Clarify limits of clojure.walk/macroexpand-all in docstring Created: 29/Jul/13  Updated: 03/Sep/13

Status: Open
Project: Clojure
Component/s: None
Affects Version/s: None
Fix Version/s: None

Type: Enhancement Priority: Trivial
Reporter: Stuart Sierra Assignee: Stuart Sierra
Resolution: Unresolved Votes: 0
Labels: walk

Attachments: Text File 0001-CLJ-1240-Note-limits-of-clojure.walk-macroexpand-all.patch    
Patch: Code

 Description   

The clojure.walk/macroexpand-all function appears to be a general recursive macroexpansion, but it is not because it doesn't understand special forms such as let.

Current patch: 0001-CLJ-1240-Note-limits-of-clojure.walk-macroexpand-all.patch

The modified docstring in this patch notes that clojure.walk/macroexpand-all is not identical to macroexpansion by the Clojure compiler and should be used for development only.






[CLJ-1239] faster, more flexible dispatch for clojure.walk Created: 29/Jul/13  Updated: 26/Jan/16

Status: Open
Project: Clojure
Component/s: None
Affects Version/s: None
Fix Version/s: None

Type: Enhancement Priority: Critical
Reporter: Stuart Sierra Assignee: Stuart Sierra
Resolution: Unresolved Votes: 8
Labels: walk

Attachments: Text File 0001-CLJ-1239-protocol-dispatch-for-clojure.walk.patch     Text File 0002-CLJ-1239-protocol-dispatch-for-clojure.walk.patch    
Patch: Code
Approval: Triaged

 Description   

The conditional dispatch in clojure.walk is slow and not open to extension. Prior to CLJ-1105 it did not support records.

Approach: Reimplement clojure.walk using protocols. The public API does not change. Users can extend the walk protocol to other types (for example, Java collections) if desired. As in CLJ-1105, this version of clojure.walk supports records.

Patch: 0002-CLJ-1239-protocol-dispatch-for-clojure.walk.patch

Performance: My tests indicate this is 1.5x-2x the speed of the original clojure.walk. See https://github.com/stuartsierra/clojure.walk2 for benchmarks.

Risks: This approach carries some risk of breaking user code that relied on type-specific behavior of the old clojure.walk. When running the full Clojure test suite, I discovered (and fixed) some breakages that did not show up in clojure.walk's unit tests. See, for example, commit 730eb75 in clojure.walk2



 Comments   
Comment by Vjeran Marcinko [ 19/Oct/13 12:32 PM ]

It looks, as it is now, that walking the tree and replacing forms doesn't preserve original meta-data contained in data structures.

Comment by Andy Fingerhut [ 23/Nov/13 1:11 AM ]

Patch 0001-CLJ-1239-protocol-dispatch-for-clojure.walk.patch no longer applies cleanly to latest Clojure master since the patch for CLJ-1105 was committed on Nov 22, 2013. From the description, it appears the intent was either that patch or this one, not both, so I am not sure what should happen with this patch, or even this ticket.

Comment by Alex Miller [ 23/Nov/13 1:52 AM ]

This patch and ticket are still candidates for future release.

Comment by Stuart Sierra [ 20/Dec/13 9:14 AM ]

Added new patch that applies on latest master after CLJ-1105.

Comment by Chouser [ 27/Feb/14 10:26 AM ]

The way this patch behaves can be surprising compared to regular maps:

(clojure.walk/prewalk-replace {[:a 1] nil} {:a 1, :b 2})
;=> {:b 2}

(defrecord Foo [a b])
(clojure.walk/prewalk-replace {[:a 1] nil} (map->Foo {:a 1, :b 2}))
;=> #user.Foo{:a 1, :b 2}

Note how the [:a 1] entry is removed from the map, but not from the record.

Here's an implementation that doesn't suffer from that problem, though it does scary class name munging instead: https://github.com/LonoCloud/synthread/blob/a315f861e04fd33ba5398adf6b5e75579d18ce4c/src/lonocloud/synthread/impl.clj#L66

Perhaps we could add to the defrecord abstraction to support well the kind of things that synthread code is doing clumsily, and then walk could take advantage of that.

Comment by Alex Miller [ 27/Feb/14 2:11 PM ]

@Chouser, can you file a new ticket related to this? It's hard to manage work on something from comments on a closed ticket.

Comment by Alex Miller [ 27/Feb/14 3:54 PM ]

@Chouser - Never mind! I was thinking this was the change that went into 1.6. Carry on.

Comment by Nicola Mometto [ 27/Feb/14 5:17 PM ]

Alex, for what it matters clojure-1.6.0 after CLJ-1105 exibits the same behaviour as described by Chouser for this patch





[CLJ-1105] clojure.walk should support records Created: 08/Nov/12  Updated: 14/Feb/14  Resolved: 22/Nov/13

Status: Closed
Project: Clojure
Component/s: None
Affects Version/s: Release 1.4
Fix Version/s: Release 1.6

Type: Enhancement Priority: Minor
Reporter: Jouni K. Seppänen Assignee: Unassigned
Resolution: Completed Votes: 2
Labels: walk

Attachments: Text File 0001-CLJ-1105-Support-records-in-clojure.walk.patch    
Patch: Code and Test
Approval: Ok

 Description   

Problem: clojure.walk throws exceptions if used on records.

user=> (defrecord Foo [x])
user.Foo
user=> (def f (Foo. :x))
#'user/f
user=> (use 'clojure.walk)
nil
user=> (postwalk identity {:foo f})
UnsupportedOperationException Can't create empty: user.Foo  user.Foo (NO_SOURCE_FILE:1)

Current Patch: 0001-CLJ-1105-Support-records-in-clojure.walk.patch adds a special case for records.

See also: CLJ-1239 "faster, more flexible dispatch for clojure.walk" which could supersede this ticket.

Screened by: Alex Miller - I think this will likely be superseded by CLJ-1239 in the future, but this is a reasonable short-term step.



 Comments   
Comment by Nicola Mometto [ 08/Nov/12 2:35 PM ]

maybe clojure should follow clojurescript's footsteps and move empty out of IPersistentCollection and create an
interface IEmptyableCollection extends IPersistentCollection { IEmptyableCollection empty(); }

Comment by Stuart Halloway [ 25/Nov/12 6:39 PM ]

Can whoever claims this please consider walk's behavior in the face of all different collection types? I think it also fails with Java collections.

Also, the collection partitioning code in clojure.data may be of use.

Comment by Stuart Sierra [ 26/Jul/13 7:39 AM ]

clojure.walk can be made to support records directly, see https://github.com/stuartsierra/clojure.walk2

Comment by Alex Miller [ 26/Jul/13 10:18 AM ]

Update title and adjust description slightly to focus this ticket on being an enhancement to make clojure.walk work with records.

Comment by Stuart Sierra [ 29/Jul/13 12:15 PM ]

Attachment 0001-CLJ-1105-Support-records-in-clojure.walk.patch is a minimal patch adding support for records without altering the design of clojure.walk.

Comment by Nicola Mometto [ 29/Jul/13 12:45 PM ]

clojure.walk is not required during bootstrapping, so I don't see why we couldn't just substitute clojure.walk with clojure.walk2 given that it only changes the internal implementation to a faster one

Comment by Chouser [ 14/Feb/14 1:50 PM ]

The way this patch behaves can be surprising compared to regular maps:

(clojure.walk/prewalk-replace {[:a 1] nil} {:a 1, :b 2})
;=> {:b 2}

(defrecord Foo [a b])
(clojure.walk/prewalk-replace {[:a 1] nil} (map->Foo {:a 1, :b 2}))
;=> #user.Foo{:a 1, :b 2}

Note how the [:a 1] entry is removed from the map, but not from the record.

I believe CLJ-1239 also demonstrates this behavior.

Comment by Andy Fingerhut [ 14/Feb/14 1:54 PM ]

This comment would probably get more visibility on CLJ-1239, given that this one is closed and CLJ-1239 is still open.





[CLJ-916] into loses metadata Created: 21/Jan/12  Updated: 18/Aug/12  Resolved: 18/Aug/12

Status: Closed
Project: Clojure
Component/s: None
Affects Version/s: Release 1.3
Fix Version/s: Release 1.5

Type: Defect Priority: Major
Reporter: Mark Swanson Assignee: Unassigned
Resolution: Completed Votes: 4
Labels: into, metadata, walk

Attachments: Text File clj-916-make-into-and-others-preserve-metadata-patch2.txt     Text File clj-916-make-into-preserve-metadata-patch1.txt    
Patch: Code and Test
Approval: Ok

 Description   

The into fn loses metadata.

I'm seeing some team members (myself included) rely on metadata. Metadata has been incredibly useful in some cases, however the silent destruction of metadata by core clojure fns (into, walk, etc) have been a source of increasing complexity and confusion.

A team member could start relying on a 3rd party library that used 'into'. Actually, the into fn could essentially be added to the code base at any time in many ways. Wrt metadata the potential for incidental complexity increases exponentially as more developers and libraries enter the mix.

One of the reasons Clojure has worked for us so well is because it has greatly reduced incidental complexity.
IMHO the 'into metadata bug' seriously limits the usefulness of metadata and would love to see the into fn fixed in 1.4.



 Comments   
Comment by Andy Fingerhut [ 01/Mar/12 4:42 AM ]

Someone please correct this if it is wrong, but it seems that into loses metadata because it is implemented using transients, and transients lose metadata. If true, then one way to fix this ticket is to implement metadata for transients.

Are there any fundamental reasons why it would be difficult to add metadata to transients, as long as the metadata itself was immutable?

Comment by Andy Fingerhut [ 16/Mar/12 4:57 AM ]

First rough cut of a patch that makes not only into, but also select-keys, clojure.set/project, and clojure.set/rename preserve metadata. Added tests for these and many other functions. Still some open questions about other functions not tested in comments. This patch does not attempt to make transients preserve metadata.

Comment by Andy Fingerhut [ 23/Mar/12 8:03 PM ]

clj-916-make-into-and-others-preserve-metadata-patch2.txt is semantically same patch and comments as March 16, 2012. Merely touched up to apply cleanly to latest master as of Mar 23, 2012.

Comment by Aaron Brooks [ 26/May/12 10:58 PM ]

I just ran into this issue myself. Is there anything that I could do to help get this fixed? Test writing? Patch checks? I'm happy to help in any way I can.

Comment by Andy Fingerhut [ 27/May/12 1:11 AM ]

You could examine the existing patch, including its tests, and see if it would have done what you were hoping it would do. Add a comment here regarding whether the changes look good to you. The attached patch is already on my weekly list of prescreened patches, but it is only one among many.

Comment by Fogus [ 15/Aug/12 1:14 PM ]

The code is clean and tests test what they should test. Regarding the questions in the test file:

  • I recommend (royal) we create another ticket for the remaining clojure.set functions and discuss them there. Once resolved the questions in the metadata.clj test file should be removed.
  • remove returns a LazySeq, but the intent of the question is clear. I vote that remove preserve metadata as well, but that should be the topic of yet another ticket.

One final point. I'd prefer that tickets be addressed in isolation and not contain extra fixes. This is a personal preference, but one that I'd like to take up on the clojure-dev list.

Comment by Andy Fingerhut [ 16/Aug/12 12:38 AM ]

clj-916-make-into-preserve-metadata-patch1.txt dated Aug 15 2012 only changes the behavior of into so that it preserves metadata. One or more other tickets will be created for some other functions that currently do not preserve metadata, but perhaps should.





Generated at Thu Aug 25 20:44:02 CDT 2016 using JIRA 4.4#649-r158309.