Since memo-swap! does behave like the documentation says – and explicitly rebuilds the seed with deref-able values – I'm inclined to believe this is a bug in memo and all the other functions that accept a seed or a base and people simply don't use those forms.
I believe the correct solution, that would not break any existing code, would be to preprocess all the seed / base maps in all calls to build-memoizer to rebuild the map and wrap any non-IDeref values in a deref like memo-swap! does.
The documentation gives an example of passing a seed value to memo as a simple number. However, that does not work. The documentation is here: http://clojure.github.io/core.memoize/#clojure.core.memoize/memo
This does work:
((clojure.core.memoize/memo + {[42 1] (delay 99)}) 42 1)
This fails when it tries to deref 99:
((clojure.core.memoize/memo + {[42 1] 99}) 42 1)