<< Back to previous view

[CLJ-1240] Clarify limits of clojure.walk/macroexpand-all in docstring Created: 29/Jul/13  Updated: 03/Sep/13

Status: Open
Project: Clojure
Component/s: None
Affects Version/s: None
Fix Version/s: None

Type: Enhancement Priority: Trivial
Reporter: Stuart Sierra Assignee: Stuart Sierra
Resolution: Unresolved Votes: 0
Labels: walk

Attachments: Text File 0001-CLJ-1240-Note-limits-of-clojure.walk-macroexpand-all.patch    
Patch: Code

 Description   

The clojure.walk/macroexpand-all function appears to be a general recursive macroexpansion, but it is not because it doesn't understand special forms such as let.

Current patch: 0001-CLJ-1240-Note-limits-of-clojure.walk-macroexpand-all.patch

The modified docstring in this patch notes that clojure.walk/macroexpand-all is not identical to macroexpansion by the Clojure compiler and should be used for development only.






[CLJ-1239] faster, more flexible dispatch for clojure.walk Created: 29/Jul/13  Updated: 27/Feb/14

Status: Open
Project: Clojure
Component/s: None
Affects Version/s: None
Fix Version/s: None

Type: Enhancement Priority: Minor
Reporter: Stuart Sierra Assignee: Stuart Sierra
Resolution: Unresolved Votes: 4
Labels: walk

Attachments: Text File 0001-CLJ-1239-protocol-dispatch-for-clojure.walk.patch     Text File 0002-CLJ-1239-protocol-dispatch-for-clojure.walk.patch    
Patch: Code

 Description   

The conditional dispatch in clojure.walk is slow and not open to extension. Prior to CLJ-1105 it did not support records.

Approach: Reimplement clojure.walk using protocols. The public API does not change. Users can extend the walk protocol to other types (for example, Java collections) if desired. As in CLJ-1105, this version of clojure.walk supports records.

Patch: 0002-CLJ-1239-protocol-dispatch-for-clojure.walk.patch

Performance: My tests indicate this is 1.5x-2x the speed of the original clojure.walk. See https://github.com/stuartsierra/clojure.walk2 for benchmarks.

Risks: This approach carries some risk of breaking user code that relied on type-specific behavior of the old clojure.walk. When running the full Clojure test suite, I discovered (and fixed) some breakages that did not show up in clojure.walk's unit tests. See, for example, commit 730eb75 in clojure.walk2



 Comments   
Comment by Vjeran Marcinko [ 19/Oct/13 12:32 PM ]

It looks, as it is now, that walking the tree and replacing forms doesn't preserve original meta-data contained in data structures.

Comment by Andy Fingerhut [ 23/Nov/13 1:11 AM ]

Patch 0001-CLJ-1239-protocol-dispatch-for-clojure.walk.patch no longer applies cleanly to latest Clojure master since the patch for CLJ-1105 was committed on Nov 22, 2013. From the description, it appears the intent was either that patch or this one, not both, so I am not sure what should happen with this patch, or even this ticket.

Comment by Alex Miller [ 23/Nov/13 1:52 AM ]

This patch and ticket are still candidates for future release.

Comment by Stuart Sierra [ 20/Dec/13 9:14 AM ]

Added new patch that applies on latest master after CLJ-1105.

Comment by Chouser [ 27/Feb/14 10:26 AM ]

The way this patch behaves can be surprising compared to regular maps:

(clojure.walk/prewalk-replace {[:a 1] nil} {:a 1, :b 2})
;=> {:b 2}

(defrecord Foo [a b])
(clojure.walk/prewalk-replace {[:a 1] nil} (map->Foo {:a 1, :b 2}))
;=> #user.Foo{:a 1, :b 2}

Note how the [:a 1] entry is removed from the map, but not from the record.

Here's an implementation that doesn't suffer from that problem, though it does scary class name munging instead: https://github.com/LonoCloud/synthread/blob/a315f861e04fd33ba5398adf6b5e75579d18ce4c/src/lonocloud/synthread/impl.clj#L66

Perhaps we could add to the defrecord abstraction to support well the kind of things that synthread code is doing clumsily, and then walk could take advantage of that.

Comment by Alex Miller [ 27/Feb/14 2:11 PM ]

@Chouser, can you file a new ticket related to this? It's hard to manage work on something from comments on a closed ticket.

Comment by Alex Miller [ 27/Feb/14 3:54 PM ]

@Chouser - Never mind! I was thinking this was the change that went into 1.6. Carry on.

Comment by Nicola Mometto [ 27/Feb/14 5:17 PM ]

Alex, for what it matters clojure-1.6.0 after CLJ-1105 exibits the same behaviour as described by Chouser for this patch





[CLJ-1105] clojure.walk should support records Created: 08/Nov/12  Updated: 14/Feb/14  Resolved: 22/Nov/13

Status: Closed
Project: Clojure
Component/s: None
Affects Version/s: Release 1.4
Fix Version/s: Release 1.6

Type: Enhancement Priority: Minor
Reporter: Jouni K. Seppänen Assignee: Unassigned
Resolution: Completed Votes: 2
Labels: walk

Attachments: Text File 0001-CLJ-1105-Support-records-in-clojure.walk.patch    
Patch: Code and Test
Approval: Ok

 Description   

Problem: clojure.walk throws exceptions if used on records.

user=> (defrecord Foo [x])
user.Foo
user=> (def f (Foo. :x))
#'user/f
user=> (use 'clojure.walk)
nil
user=> (postwalk identity {:foo f})
UnsupportedOperationException Can't create empty: user.Foo  user.Foo (NO_SOURCE_FILE:1)

Current Patch: 0001-CLJ-1105-Support-records-in-clojure.walk.patch adds a special case for records.

See also: CLJ-1239 "faster, more flexible dispatch for clojure.walk" which could supersede this ticket.

Screened by: Alex Miller - I think this will likely be superseded by CLJ-1239 in the future, but this is a reasonable short-term step.



 Comments   
Comment by Nicola Mometto [ 08/Nov/12 2:35 PM ]

maybe clojure should follow clojurescript's footsteps and move empty out of IPersistentCollection and create an
interface IEmptyableCollection extends IPersistentCollection { IEmptyableCollection empty(); }

Comment by Stuart Halloway [ 25/Nov/12 6:39 PM ]

Can whoever claims this please consider walk's behavior in the face of all different collection types? I think it also fails with Java collections.

Also, the collection partitioning code in clojure.data may be of use.

Comment by Stuart Sierra [ 26/Jul/13 7:39 AM ]

clojure.walk can be made to support records directly, see https://github.com/stuartsierra/clojure.walk2

Comment by Alex Miller [ 26/Jul/13 10:18 AM ]

Update title and adjust description slightly to focus this ticket on being an enhancement to make clojure.walk work with records.

Comment by Stuart Sierra [ 29/Jul/13 12:15 PM ]

Attachment 0001-CLJ-1105-Support-records-in-clojure.walk.patch is a minimal patch adding support for records without altering the design of clojure.walk.

Comment by Nicola Mometto [ 29/Jul/13 12:45 PM ]

clojure.walk is not required during bootstrapping, so I don't see why we couldn't just substitute clojure.walk with clojure.walk2 given that it only changes the internal implementation to a faster one

Comment by Chouser [ 14/Feb/14 1:50 PM ]

The way this patch behaves can be surprising compared to regular maps:

(clojure.walk/prewalk-replace {[:a 1] nil} {:a 1, :b 2})
;=> {:b 2}

(defrecord Foo [a b])
(clojure.walk/prewalk-replace {[:a 1] nil} (map->Foo {:a 1, :b 2}))
;=> #user.Foo{:a 1, :b 2}

Note how the [:a 1] entry is removed from the map, but not from the record.

I believe CLJ-1239 also demonstrates this behavior.

Comment by Andy Fingerhut [ 14/Feb/14 1:54 PM ]

This comment would probably get more visibility on CLJ-1239, given that this one is closed and CLJ-1239 is still open.





[CLJ-916] into loses metadata Created: 21/Jan/12  Updated: 18/Aug/12  Resolved: 18/Aug/12

Status: Closed
Project: Clojure
Component/s: None
Affects Version/s: Release 1.3
Fix Version/s: Release 1.5

Type: Defect Priority: Major
Reporter: Mark Swanson Assignee: Unassigned
Resolution: Completed Votes: 4
Labels: into, metadata, walk

Attachments: Text File clj-916-make-into-and-others-preserve-metadata-patch2.txt     Text File clj-916-make-into-preserve-metadata-patch1.txt    
Patch: Code and Test
Approval: Ok

 Description   

The into fn loses metadata.

I'm seeing some team members (myself included) rely on metadata. Metadata has been incredibly useful in some cases, however the silent destruction of metadata by core clojure fns (into, walk, etc) have been a source of increasing complexity and confusion.

A team member could start relying on a 3rd party library that used 'into'. Actually, the into fn could essentially be added to the code base at any time in many ways. Wrt metadata the potential for incidental complexity increases exponentially as more developers and libraries enter the mix.

One of the reasons Clojure has worked for us so well is because it has greatly reduced incidental complexity.
IMHO the 'into metadata bug' seriously limits the usefulness of metadata and would love to see the into fn fixed in 1.4.



 Comments   
Comment by Andy Fingerhut [ 01/Mar/12 4:42 AM ]

Someone please correct this if it is wrong, but it seems that into loses metadata because it is implemented using transients, and transients lose metadata. If true, then one way to fix this ticket is to implement metadata for transients.

Are there any fundamental reasons why it would be difficult to add metadata to transients, as long as the metadata itself was immutable?

Comment by Andy Fingerhut [ 16/Mar/12 4:57 AM ]

First rough cut of a patch that makes not only into, but also select-keys, clojure.set/project, and clojure.set/rename preserve metadata. Added tests for these and many other functions. Still some open questions about other functions not tested in comments. This patch does not attempt to make transients preserve metadata.

Comment by Andy Fingerhut [ 23/Mar/12 8:03 PM ]

clj-916-make-into-and-others-preserve-metadata-patch2.txt is semantically same patch and comments as March 16, 2012. Merely touched up to apply cleanly to latest master as of Mar 23, 2012.

Comment by Aaron Brooks [ 26/May/12 10:58 PM ]

I just ran into this issue myself. Is there anything that I could do to help get this fixed? Test writing? Patch checks? I'm happy to help in any way I can.

Comment by Andy Fingerhut [ 27/May/12 1:11 AM ]

You could examine the existing patch, including its tests, and see if it would have done what you were hoping it would do. Add a comment here regarding whether the changes look good to you. The attached patch is already on my weekly list of prescreened patches, but it is only one among many.

Comment by Fogus [ 15/Aug/12 1:14 PM ]

The code is clean and tests test what they should test. Regarding the questions in the test file:

  • I recommend (royal) we create another ticket for the remaining clojure.set functions and discuss them there. Once resolved the questions in the metadata.clj test file should be removed.
  • remove returns a LazySeq, but the intent of the question is clear. I vote that remove preserve metadata as well, but that should be the topic of yet another ticket.

One final point. I'd prefer that tickets be addressed in isolation and not contain extra fixes. This is a personal preference, but one that I'd like to take up on the clojure-dev list.

Comment by Andy Fingerhut [ 16/Aug/12 12:38 AM ]

clj-916-make-into-preserve-metadata-patch1.txt dated Aug 15 2012 only changes the behavior of into so that it preserves metadata. One or more other tickets will be created for some other functions that currently do not preserve metadata, but perhaps should.





Generated at Fri Aug 29 19:09:12 CDT 2014 using JIRA 4.4#649-r158309.