defrecord improvements

rh says:

Please don't work on any other aspect of this until the problem section is good.

Main Problems:

The Java unification of records prevents them from being first class, in either the data or fn sense:

• record data is not first class
  • can't read/write them
    • crummy choice: maps are good as data, need records for protocol polymorphism
  • user code cannot fix this
    • anything that requires EvalRead is not a fix
• record creation is not first class
  • no per-record factory fn (or access to any associated fn plumbing, e.g. apply)
  • Clojure level use/require doesn't get you access to records
  • user code can mostly fix this (defrecord+factory macro)

Lesser Problems (Not Yet Under Consideration):

• generic factory fn
  • like factory fn, but generic with name
  • introduces weak-referencing, modularity issues, etc.
  • don't have a good problem statement, so ignoring this for now
  • which comes first: generic or specific?
• support for common creation patterns
  • named arguments
    • with more than a few slots, record construction is difficult to read
  • default values
    • maybe needs to be a property of factory fn, not record
    • different factory fns can have different defaults
  • validations
  • are the patterns truly common?
  • very solvable in user space, esp. if per-record factory fn available
• application code needing to know record fields
  • synthesizing data
  • creating factory fns if we don't provide them

Challenges:

• how evaluative should record read/write be?
  • option 1: records are data++: no EvalReader needed, no non-data semantics
  • option 2: records are more:
    • maybe EvalReader required?
    • maybe special eval loopholes for constructor fns?
  • option 1 wins
• what happens when readers and writers disagree about a record's fields?
  • positional approach would either fail or silently do the wrong thing
  • k/v approach lets you get back to the data
    • still on you to fix it
• does this have be a breaking change?
  • data print/read: no
  • constructor fn: yes
    • any good generated name likely to collide with what people are using
• what if defrecord is not present on the read side?
  • fail?
  • create a plan map instead
    • plus tag in data?
    • plus tag in metadata?
    • reify in a tagging interface
  • attempt to load
    • no – could lead to arbitrary code injection during read

Some Options:

• create reader/writer positional syntax, no constructor fn
  • pros
    • easy to deliver efficiently
    • non-breaking
    • introduces no logic (user or clojure) into print/read
  • cons
    • what happens if defrecord field count changes?
    • what happens if field names change?
      • no way to know
  • feels like a non-starter
• create reader/writer kv syntax, no constructor fn
  • pros
    • non-breaking
    • introduces no logic (user or Clojure) into print read
    • can still recover data if defrecord structure has changed
  • cons
    • how to deliver read efficiently?
      • create empty object + merge
        • cache the empty object we merge against?
      • reflect against object and manufacture reader fn
        • who keeps track of this?
        • how would this interact with constructor, if we add that separately?
      • add a map-based constructor to defrecord classes
        • what would its signature be?
      • add a static map based factory fn to defrecord classes
• reader/writer syntax that depends on a new factory fn
  • pros
    • can be efficient
    • can implement any policy in handling defrecord changes
  • cons
    • likely breaking (what will the fn names be?)
    • read/write now depends on fns
• positional constructor fn
  • no
  • replicates the weakness of existing constructors
• kv constructor fn
  • open questions
    • autogenerated for all defrecords?
    • optional?
    • conveniences (defaults, etc.)
      • no

Tentative Proposal 1:

Define a k/v syntax for read and write that does not require a factory fn.

• adopt the existing print syntax as legal read syntax?
  • "#:user.P{:x 1, :y 2}"
• get Rich's input on efficient reader approach (4 possibilities listed above)
• if reader defrecord fields are different, merge and move on
• Undecided: if record class not loaded:
  • TBD: error or make a plain ol map?
  • hm, could fix on writer side: option to dumb records down to maps?

Tentative Proposal 2:

Autogenerate a k/v factory fn for all defrecords.

• (new-foo :x 1 :y 2)
• class constructor is an interop detail
• factory fn is the Clojure way
• people can build their own defaults, validation, etc. easily with macros, given this

Some history:

The record multimethod was almost ready to go when Rich raised the GC issue. What happens when somebody creates a ton of record classes over time? GC can collect records that are not longer in use, but doesn't clean up the old multimethod functions.

Additional Reading

Some (non-contributed) code that demonstrates people's need for this:

• cemerick's defrecord slot defaults
• David McNeil's enhanced clojure records