Done, however the names where changed to traced? and traceable?.

Done is 0.7.4

Done, "good" version is 0.7.5 (readme & comments need an update in 0.7.4")

CA received it seems (I am listed on http://clojure.org/contributing ).

I'm not sure about this. If you're only using c.t.n.find in isolation, it's fine. But if you're using code-reloading and c.t.n.repl, it could incorrectly try to reload .cljs files in JVM Clojure.

We really need [Feature Expressions]http://dev.clojure.org/display/design/Feature+Expressions or something like it to get away from multiple file extensions.

Until then, I think it has to be optional. I don't know how best to achieve this. The APIs in c.t.n.repl and c.t.n.dir are not amenable to extension. I'll think about it.

True, I didn't realize that.

Maybe using a dynamic var to hold the regex pattern (or a predicate?) could be a reasonable solution in the meantime, this would allow to rebind it in the case of codox & similar libs.
I don't really "like" to use dynamic vars (or regexes!), but in this case it might make sense.
Not to mention it would also allow more flexibility on projects where you don't want to have your clj codoxed, but only your cljs for instance.

Any thoughts on my last comment/edit? I don't think there is a single doc lib that works with cljs at the moment, it is a bit painful to be honest.

Yes, I think a dynamic var would be OK. However, I would like to know of a real use case, not just a potential one.

The idea was to be able to use codox on cljs files, I tried locally but there are other problems with this approach to be able to get to cljs vars metadata. So in the end I think you were right, maybe it's better to wait for feature expressions for this.

Pierre-Yves, there are instructions for creating patches under the headings "Development" and "Adding patches" on this page: http://dev.clojure.org/display/design/JIRA+workflow

Submissions to this module do require the author to sign a CA. Instructions here: http://clojure.org/contributing

Suggested Patch

Hello, now that I'm a registered contributor, I attached a file as suggested in the workflow wiki

I'm surprised that this doesn't already work. We have support for unifying expressions in the pattern already. Look at line 1230 in tests.clj in the master branch.

So this should just work, no need to explicitly support macros as far as I can tell. If it's not working, then there's a bug.

At least on 0.7.5, matching against a macro gives a runtime error:

Exception in thread "main" java.lang.ClassCastException: clojure.core.logic.LVar cannot be cast to clojure.lang.IFn
	at rl.core$glyph_$fn__123$fn__144$fn__165$fn__166$_inc__167$fn__168.invoke(core.clj:61)
	at clojure.core.logic.Substitutions.bind(logic.clj:211)
	at rl.core$glyph_$fn__123$fn__144$fn__165$fn__166$_inc__167.invoke(core.clj:58)
	at clojure.core.logic$fn__1056$_inc__1057.invoke(logic.clj:1160)
	at clojure.core.logic$fn__1056$_inc__1057.invoke(logic.clj:1160)
	at clojure.core.logic$fn__898$_inc__899.invoke(logic.clj:823)
	at clojure.core.logic$fn__890$fn__891.invoke(logic.clj:828)

Using a fn instead of a macro gives the same:

Exception in thread "main" java.lang.ClassCastException: clojure.core.logic.LVar cannot be cast to clojure.lang.IFn
	at rl.core$drawable_$fn__235$fn__248$fn__249$_inc__250$fn__251.invoke(core.clj:67)
	at clojure.core.logic.Substitutions.bind(logic.clj:211)
	at rl.core$drawable_$fn__235$fn__248$fn__249$_inc__250.invoke(core.clj:65)
	at clojure.core.logic$fn__1056$_inc__1057.invoke(logic.clj:1160)
	at clojure.core.logic$fn__894$_inc__895.invoke(logic.clj:826)
	at clojure.core.logic$fn__1056$_inc__1057.invoke(logic.clj:1160)
	at clojure.core.logic$fn__898$_inc__899.invoke(logic.clj:823)
	at clojure.core.logic$fn__898$_inc__899.invoke(logic.clj:823)
	at clojure.core.logic$fn__898$_inc__899.invoke(logic.clj:823)
	at clojure.core.logic$fn__890$fn__891.invoke(logic.clj:828)

Here's (glyph-) for reference (don't mind all the extra [], I have a weird key/value thing because of some conveniences for maintaining fact identity in a temporal database):

(defna glyph- [Key Val]
	([[Thing] [Glyph]] (thing- [Thing]) (on-fire_ *turn* [Thing]) (== Glyph \δ))
	([[Thing] [Glyph]] (thing- [Thing]) (fresh [Type] (type- [Thing] [Type]) (glyph- [Type] [Glyph])))
	([[(type-enum :player)] [Glyph]] (== Glyph \@))
	([[(type-enum :dragon)] [Glyph]] (== Glyph \D))
	([[Type] [Glyph]] (== Glyph \?)))

and type-enum as a macro:

(defmacro type-enum [v] `[:enum :type ~v])

and as a fn:

(defn type-enum [v] [:enum :type ~v])

I'll mess around and see if my example works in HEAD.

Same exception with this test case in HEAD (sorry for all the facts):

(defrel thing- [Thing])
(defrel type- [Thing] [Type])
(fact thing- [0])
(fact thing- [1])
(fact thing- [2])
(fact type- [0] [:player])
(fact type- [1] [:dragon])
(fact type- [2] [:pig])
(defn type-enum [t] [:type t])
(defna drawable- [Key]
  ([[Thing]] (thing- [Thing]) (fresh [Type] (type- [Thing] [Type]) (drawable- [Type])))
  ([[(type-enum :player)]] succeed)
  ([[(type-enum :dragon)]] succeed))

(deftest do-fns-work
	(is (= (run* [q] (drawable- [q])) '(0 1))))

Now that I look at it, I may be expecting a wrong-format return value, but the point is that I don't even get that far.

Using the REPL, I checked out how (defna drawable- . . .) expands (tidied up slightly):

(def drawable- (clojure.core/fn ([Key] 
  (clojure.core.logic/conda 
	  ((clojure.core.logic/fresh [Thing] (clojure.core.logic/== [Thing] Key) (thing- [Thing]) (fresh [Type] (type- [Thing] [Type]) (drawable- [Type])))) 
		((clojure.core.logic/fresh [type-enum] 
		  (clojure.core.logic/== [(type-enum :player)] Key) succeed))
		((clojure.core.logic/fresh [type-enum] 
		  (clojure.core.logic/== [(type-enum :dragon)] Key) succeed))))))

Note the (clojure.core.logic/fresh [type-enum] . . .) forms, which are exactly what I would not want to see in this case.

I'm not really sure why this doesn't work here yet works for the matche test case.

[(type-enum :dragon)]

This pattern make it seem like you want to match:

[[:type :dragon]]

Note extra level of brackets here. Is this the case?

Even so I agree that the expansion doesn't look quite right. We should never descend into a seq form like that.

Yes, that's exactly the desired outcome in this case--a tagged value in my naive interpretation. Is the reason it fails whereas the test on :1230 doesn't the fact that it's producing a vector and not a list? Changing the fn to return a list instead of a vector didn't seem to help.

My patch, fwiw, doesn't exhibit that behavior (at least for macros, haven't tested it with fns).

What I mean is don't you want the following instead?

(defna drawable- [Key]
  ([[Thing]] (thing- [Thing]) (fresh [Type] (type- [Thing] [Type]) (drawable- [Type])))
  ([(type-enum :player)] succeed)
  ([(type-enum :dragon)] succeed))

Note that I removed a layer of square brackets.

Nope! I actually want both. I'm doing some temporal logic stuff and I wanted some conveniences for "updating" a fluent, so I wanted to distinguish between the "key part" and the "value part" of the arguments. It looks silly for facts with no "value part", but it lets me write procedures and fns something like this:

(defrel heldo Time Fluent)
(defrel ¬heldo Time Fluent)
(declare fluent-obtainedo) ; most recent 'held' not terminated by a '¬held', or fail
(defn alter-fluent [Time Rel Key NewVal]
  ;todo: error check, ensure old != new, old obtains, new does not obtain
  (doseq [old-val (run* [old-val] (fluent-obtainedo Time [Rel Key old-val]))]
    (fact ¬heldo Time [Rel Key old-val]))
  (fact heldo Time [Rel Key NewVal]))
. . .
(fact heldo 0 ['pos [0] [0 0]])
. . .
(alter-fluent 1 'pos [0] [1 1])

And I write all the non-temporal fluents that way too for consistency and to help prevent mistakes.

I'll try to give a closer look at this issue over the weekend.

We're thinking about a more general solution here: http://github.com/clojure/core.logic/wiki/Better-syntax-support

I actually did manage to tack on a prototype that covers the basic behavior I would like to see: https://github.com/werg/herpderp/blob/master/src/herpderp/multo.clj

I use a set stored in a ref in the defne's metadata to manage dynamic changes to the clauses. Upon changing that set using defclause I use eval to re-define the var using defne.

This might not be nice, but allows me to continue developing features against it.

I don't think the current implementation can really support this and I don't think it's wise to try to hack around the current implementation. I'd be willing to consider a comprehensive solution if someone is willing to do the legwork.

Declined. 'write-string' is an implementation detail, not something I will commit to as a public API. Use the :value-fn option of 'write' to handle extension to new types. Or copy the implementation of 'write-string' into your namespace.

clojurescript in browsers, wrapper for Google Closures WebSocket.

Have you verified that this doesn't break browser REPL? Thanks!

This is a known issue which will have to wait for if and when Keywords and Symbols become proper types in ClojureScript.

Extending js/Object is not recommended, if you actually need to add functionality to the base JS native types the convention is different from Clojure: default instead of Object, string instead of js/String. Please refer to core.cljs if you want more examples.

Thanks for the pointer about default and string, I didn't know about that.

I reported the issue at the demand of bbloom. It does seem to be difficult to address without taking a (major) performance hit unfortunately.

every environment

This could easily be done by adding support for this in closure.clj - patch welcome!

I propose to add a new keyword/value in the optimization option map, namely :exclude-path. the value of this option should be a subdir of the source-dir.

Top level scenario:

1. you use lein-cljsbuild to define your cljs project
2. you define more builds, e.g. a :dev build and a :prod build.
(defproject ...
:cljsbuild {:builds
{:dev {:source-path "src/cljs" {:compiler {:output-to "resources/public/js/main_dbg.js"
:optimizations :whitespace
:pretty-print true}}}
:prod {:source-path "src/cljs" {:compiler {:output-to "resources/public/js/main_dbg.js"
:optimizations :advanced
:exclude-path "some-path"}}}})

3. lein-cljsbuild plugin will instruct CLJ compiler by passing it the soruce-dir (e.g. "src/cljs") and the options map which now can contain also an optional :exclude-path keyword.

4. During compilation the compiler will exclude from source-dir any cljs source which is contained in the named excluded directory.

I'll start bottom-up from 4. then I'll try to patch lein-cljsbuild too.

Mimmo

Hi David, here is the flattened patch relative. The two guys which worked on the patch under my direction are interns in my own company. Next monday we'll send their signed CA.

My best
Mimmo

In general, we should not complicate the compiler with additional options when functionality can be provided by external tools.

I think this feature can be provided by external tools. The compiler will only automatically pull in things that are actually dependencies of the sources that we provide to the compiler. External tools should provide ways to limit what is handed to the compiler.

I would first attempt to modify lein-cljsbuild to do what you want.

When using the compiler directly, you can provide your own implementation of Compilable which, when given a directory, will filter out sources based on some criteria you provide. In my projects I have custom implementations of Compilable that do just this. You should be able to do the same thing in lein-cljsbuild.

-Brenton

Thanks for the advice Brenton. I'll try to understand from the maintainer of `lein-cljsbuild` where to start from. I agree with you about keeping the compiler clean from options that can be implemented by the tools. But I'm no so sure that patching lein-cljsbuild we'll be as easy as adding `:exclude` option to the compiler.

Mimmo

It doesn't matter which one is easier to do. Every new option and special case that we add to the compiler makes it more difficult to understand how changes will impact users.

I agree that anything that can be solved at higher level tools is better - it wasn't clear to me from the implementation that Compilable could be used to control this - but I see now.

Mimmo, cljs.closure/build takes a Compilable and a map of options. So lein-cljsbuild could construct the custom Compilable that understands :excludes and pass it along.

FWIW, I agree with Brenton that this should be in lein-cljsbuild.

I didn't know that cljs.closure/build was flexible enough to do this already. I always thought that it needed to be extended so that a vector of files could be passed in, or something, but it sounds like the Compilable approach should work just fine.

I will happily accept a patch for this. One thing to keep in mind, though, is that the :exclude entry should not be in the :compiler map if lein-cljsbuild is handling it. The :compiler map is passed straight through as options to cljs.closure/build. So, the :exclude entry should be a neighbor of the :compiler entry.

Hi all,
we all agree with Brenton and yes, the :exclude option has to be at the same hierarchical level of :source-path. I'll verify if we can also extend :source-path from taking a String to taking a vector of String, in such a way that the user could ask to compile more cljs src-dir.

Mimmo

Hi all,
just a small add on to the previous comment. I don't think we're going to update cljsc/cljsc.clj, which I consider a kind of tool, much more fragile and limited than cljsbuild plugin, to interoperate with CLJS compiler.

My best

Mimmo

Should be resolved by tools that use the compiler.

Auto-generation from the build script of version_autogen.clj and version_autogen.cljs files
that both define the cljs.version-autogen/clojurescript-version
with the current version info for both the clj and cljs environments

Why a separate namespace for this?

Good point - probably better to add the clojurescript-version var to the cljs.core namespace.

New patch change the build script to auto-generates the clojurescript-version assignment statements in both cljs/core.clj and cljs/core.cljs to reflect the correct version info.

In that way it resembles more the clojure.core use of clojure-version.

Note that the two separate assignments for clj and cljs are needed to detect out-of-sync of repl-compiler version with the compiler used to generate the js-code that was downloaded thru the webpage.

Difficult to write a test-script, but it seems to work in my repl:

—
user=> (require 'cljs.core)
nil
user=> cljs.core/*clojurescript-version*
{:minor 0, :revision 1515, :major 0}
user=> (run-repl-listen)
"Type: " :cljs/quit " to quit"
ClojureScript:cljs.user> *clojurescript-version*
{:revision 1515, :major 0, :minor 0}
ClojureScript:cljs.user>
—

CLJS-402: build script auto-generates the clojurescript-version assignment statements in both cljs/core.clj and cljs/core.cljs to reflect the correct version info

When I tried to implement a "clojurescript-version" function like the "clojure-version" one, I (re)discovered that cljs/core.clj is not your average cli-file but some funky file used by the clojurescript compiler to bootstrap - in other words it's not a good file to require and run normal functions from your clj-environment.

So I moved the *clojurescript-version* var to cljs/compiler.clj as it seemed to make sense to associate the version with the compiler.

On the cljs site, I left the *clojurescript-version* in cljs/core.cljs, although I was tempted to create a cljs.compiler namespace to store that var on the cljs side to make it symmetric .

Also added a (clojurescript-version) function to both the clj and cljs sides, with an added special-fn in the cljs/repl.clj such that you can ask for the compiler version of the repl-server from the cljs-repl.

Updated 3rd patch file replaces the previous one.

Just wanted to record an example of a mismatch in compilers:

user=> (run-repl-listen)
"Type: " :cljs/quit " to quit"
ClojureScript:cljs.user> (clojurescript-version)
"0.0.1515"
ClojureScript:cljs.user> (clj-clojurescript-version)
"0.0.1516"
ClojureScript:cljs.user>

This happens when you generate the javascript from your cljs with a "lein cljsbuild once" command with compiler version "0.0.1515",
then upgrade the compiler to "0.0.1516" and then run the cljs-repl without regenerating the javascript for the webpage-download.
Very obscure things can happen after that...

build script auto-generates the clojurescript-version assignment statements in both cljs/compiler.clj and cljs/core.cljs to reflect the correct version info
Function "clojurescript-version" is added to both compiler.clj and core.cljs to mimic equivalent "clojure-version"
Special-fn "clj-clojurescript-version was added to repl.clj to obtain the compiler's clojurescript-compiler version from the cljs-repl

The patch uses sed, I'm not sure this is such a great idea since whatever the patch does should probably work with whatever build setup we have on Hudson.

Would it make sense to call this 'clojure-version' as well, instead of clojurescript-version? It's a map, and so various flags could be added to differentiate jvm, js, python, c, etc. runtimes. What does ClojureCLR do?

sed was already used in the build script... a few lines down, the pom file is transformed with:

sed -e s/CLOJURESCRIPT_VERSION/0.0-$REVISION/ < "$POM_TEMPLATE" > "$POM_FILE"

A quick scan of the src-code showed that ClojureCRL uses clojure-version and *clojure-version*.

However, there is less chance for confusion as you are supposedly running on the CLR already.

With ClojureScript you can have these three intertwined "clojure" environments that are all live at the same time: the clojure version, the repl clojurescript version, and the clojurescript version used to compile the initially loaded js. Not sure if overloading the clojure-version fn-name and relying on namespaces to differentiate will be helpful... but it's not that big of a deal and I'm easily persuaded otherwise - somebody should make the call and I'll change it.

Sorry just now coming back to this ticket. I think it's probably preferable that the map of properties be called the same thing everywhere. I would apply the patch with this change. I note it no longer applies to master.

Rich Hickey already had a good name for an operation like this - specify.

OK, I've declined the ticket, since not only the name but also the patches should be disregarded.

The problem with above impl is that the generated implementing fns get instantiated, every time extend-instance is evaluated. That is not acceptable on a per-instance basis.

I will work on a new macro called specify, which allows passing implementing fns.

Is a syntax similar to clojure.core/extend right for specify?
Should I create a new ticket?

As far as I understand it the interface for specify should be the same as extend-type.

Can we get the ticket # in the commit message? Thanks!

Of course, sorry! Here is a new patch, if you please.

Thanks. I'm confused about the second aspect of the patch, what do you mean by bogus error messages?

Well, the repl uses a :statement context to generate the javascript which gets printed out as part of a stack trace. When setting :expr back to :statement in the repl, you can try the following scenario:

• enter into the repl (def val :worked) (if (> (Math/rand) 0.5) val (throw ""))
• half of the time you will get :worked
• half of the time you will get a stacktrace citing 'ns.val = ":worked"; if (Math.rand() > 0.5){}{throw ""}'
• that makes no sense

With the full patch, the printout will be smth like '(function(){ns.val = ":worked"; if (Math.rand() > 0.5){return ns.val}{throw ""}})()', which is noisier, but more like what actually got evaluated by the repl client.

##EDIT#APPEND##

The reason the repl still works, even when analyzing in a statement context with the patch is, that in order to get the return value, the repl statement has to be evaluated in an :expr context anyway.
This happens by unquoting it into a wrapper form. With the patch the wrapper form also gets analyzed in an :expr context, which doesn't hurt.
Only when an exception gets caught, the js of the unwrapped expr is used, which now always matches the its counterpart in the wrapper form.

I'm still confused. Does this additional modification have anything to do with the ticket? By that I mean does the patch require the second modification? If it does can you explain a further? Thanks.

The patch doesn't require the second modification in the sense that everything still works if it's left out.

The patch requires the modification in the sense that it would break stack traces on the REPL otherwise + the two changes should be reverted together if and when we move such optimizations out of the emitter into an optimization pass.

So the trade off is in always having the correct code in a REPL stacktrace in exchange for making it more verbose. Again, this doesn't influence behavior, so it's a matter of prioritizing requirements.

I still don't understand why/how the part of the patch that addresses the ticket breaks stack traces. Can you explain?

1) var read statements get omitted
2) typing "var" into the repl still works, because it's analyzed in a wrapper form
3) if it throws, the printed generated js is the original form in a statement context, you can observe this with above code sample.

Ok I understand now, I don't think the second part of the patch is relevant. The user entered two statements, not one combined in an implicit do which is what the other part of the patch does. Can we get a new patch that only includes the part that addresses the ticket? Thanks!

For the record: There was some talk in the chat, where I laid down my rationale for changing the repl specifically: "Technically repl inputs have to be in an expr context, to capture the return val. And they are, by virtue of the wrapping form. The change makes that fact explicit, thereby keeping stack traces sane."

I also discovered, that two other ops, beside a var read, don't emit in a statement context either and therefore have same issue. I created http://dev.clojure.org/jira/browse/CLJS-403 for the REPL change.

Updated attached patch contains just the optimization.

fixed, http://github.com/clojure/clojurescript/commit/97e5fbd1e1597d58be35fd8320c8044ccc9d3a3d

clojure/clojurescript "0.0-1450"

This patch no longer applies, mind updating it?

This patch should apply to master version on Mon, 15 Oct 2012 22:03:19 -0700 (4defcbcf19112b9be6a4a27b5d8855552bf94948)

Excellent, fixed http://github.com/clojure/clojurescript/commit/bef56a74f2eeecabfe0c0a28d89b455dce576ea3

Please at the ticket # to the commit message though, thanks!

Why don't you return a vector like Clojure does?

Return a vector like Clojure.

The patch is not correctly formatted with attribution.

Use git format-patch instead of git diff, to hopefully provide the right patch format.

fixed, http://github.com/clojure/clojurescript/commit/b06905863a9b0ce3618634d8fe0effb3aefd4063

Patch 0001-Add-hasheq-based-fallback-to-Util.compare.patch dated Apr 17 2013 applies cleanly to latest master, but causes several unit tests in data_structures.clj to fail. These are the kinds of things you would expect to fail with the change made in the patch, because the failing tests expect an exception to be thrown when comparing objects that don't implement Comparable, and with this patch's changes they no longer do. If this patch's change is desired, those tests should probably be removed.

Thanks Andy, I can't believe I've forgotten to re-run the tests.

Anyway, I'm attaching a new version of the patch that deletes the few sorted-set and sorted-set-by invocations that check that an exception is thrown when creating sorted sets containing non-Comparables.

New version of the patch.

Tassilo, you say that one of your use cases is sorted collections. Note that any compare function that returns 0 for two values will cause sorted sets and maps to treat the two compared values as equal, and at most one of them will get into the ordered set/map, the second treated as a duplicate, even though the values are not equal. See https://github.com/jafingerhut/thalia/blob/master/doc/other-topics/comparators.md#comparators-for-sorted-sets-and-maps-are-easy-to-get-wrong for an example (not based on your modified compare, but on a comparator that returns 0 for non-equal items).

With your proposed change to compare, this occurrence would become very dependent upon the hash function used. Probably quite rare, but it would crop up from time to time, and could potentially be very difficult to detect and track down the source.

Hi Andy, you are right. It's possible to add an explicit equals()-check in cases the hashcodes are the same, but I think there's nothing you can do in the hashcodes-equal-but-objects-are-not case. That is, there's no way to ensure the rule sgn(compare(x, y)) == -sgn(compare(y, x)), the transitivity rule, and the compare(x, y)==0 ==> sgn(compare(x, z))==sgn(compare(y, z)) for all z rule.

I'm closing that ticket.

Below is a workaround, at least. I don't know, but perhaps the as-file method for URLs in io.clj of Clojure, the part that converts %hh sequences to a character with code point in the range 0 through 255, is at least partly at fault here. I don't know right now if it is possible to modify that code to handle the general case of whatever character encoding munging is going on here to when .getResource creates the URL object.

clojure.java.io/resource is documented to return a Java object of type java.net.URL, which seems like it does %hh escaping of many characters. Reference [1] to a Java bug from 2001 where a Java user was surprised by the then-recent change in behavior of the getResource method [2].

Doing a little searching I found this StackOverflow question [3], which has what might be a workaround. I tried it on my Mac OS X 10.6 system running JDK 1.6 and it seemed to work:

(slurp (.getContent (clojure.java.io/resource "abcíd/foo.txt")))

That getContent is a method for class java.net.URL [4]

[1] http://bugs.sun.com/bugdatabase/view_bug.do?bug_id=4466485
[2] http://docs.oracle.com/javase/1.5.0/docs/api/java/lang/Class.html#getResource%28java.lang.String%29
[3] http://stackoverflow.com/questions/13013629/best-international-alternative-to-javas-getclass-getresource
[4] http://docs.oracle.com/javase/1.5.0/docs/api/java/net/URL.html#getContent%28%29

Hi Andy,

Thanks for the background and suggestions, that's very helpful.

I'm gradually learning Clojure with no Java experience. In this case I was searching for the preferred Clojure way to access items in directories declared under :resource-paths in a Leiningen project.clj file. Perhaps clojure.java.io/resource isn't the best way to do this as it's possibly too tied to the expectation for a URI instead of a more general IRI.

You're suggested workaround did work for my use case:

(slurp (.getContent (clojure.java.io/resource "abcíd/foo.txt")))

but hopefully there would be more native/direct Clojure way to accomplish the same eventually.

I don't know if java.net.IDN would be useful internally as a fix in clojure.java.io/resource — I'm assuming not since it wasn't added until Java 6.[1]

user=> (import 'java.net.IDN)
java.net.IDN
user=> (java.net.IDN/toASCII "/dir/déf")
"xn--/dir/df-gya"
user=> (java.net.IDN/toUnicode "xn--/dir/df-gya")
"/dir/déf"

[1]: http://docs.oracle.com/javase/6/docs/api/java/net/IDN.html

Patch clj-1177-patch-v1.txt dated Mar 8 2013 is an attempt to solve this issue, in what I think may be a correct way. As specified in RFC 3986, when taking a Unicode string and making a URL of it, it should be encoded in UTF-8 and then each individual byte is subject to the %HH hex encoding. This patch reverses that to turn URLs into file names.

Tested on Mac OS X 10.6 with a command line like this (it doesn't work without the -Dfile.encoding=UTF-8 option on my Mac, probably because the default encoding is MacRoman):

% java -cp clojure.jar:path/to/resource -Dfile.encoding=UTF-8 clojure.main
user=> (require '[clojure.java.io :as io])
nil
user=> (io/resource "abcíd/foo.txt")
#<URL file:/Users/jafinger/clj/clj-ns-browser/resource/abc%c3%add/foo.txt>
user=> (slurp (io/resource "abcíd/foo.txt"))
"The quick brown fox jumped over the lázy dög!\n"

I think that this patch would be much more helpful to users if it reported the problem form (both name and params).

(And I wonder if we should be using ex-info for all errors going forward.)

New version of the patch that mentions both method name and argument vector, and uses ex-info as Stu suggested.

Presumuptuously changing Approval from Incomplete back to None, since the reason for marking it Incomplete seems to have been addressed with a new patch.

It is easy for tools that consume Clojure to hide stack traces for those who do not want to see them. If Clojure itself eats stack traces, it is impossible for those of us who do want to see them to get them back.

Please do this kind of work in tool (if at all) and make it optional for users.

Please close and reject. The patch is not working if val has non final fields.

Ian, thanks for the patch. It is Rich Hickey's policy only to consider applying patches to Clojure from those who have signed a Clojure contributor agreement: http://clojure.org/contributing

Were you interested in doing so, or perhaps it is already in progress?

I wasn't aware that this was necessary. I'm mailing the form in.

Patch clj-1042-negative-subs-patch2.txt dated Aug 27 2012 is identical to Ian Eure's negative-subs.patch, except it is in the desired git format.

Ian, for future reference on creating patches in the desired format, see the instructions under the heading "Development" on this page: http://dev.clojure.org/display/design/JIRA+workflow

Thanks, will do.

If Clojure decides to support Python-style negative indices, you should also consider adding support to subvec.

Patch extended to support negative indices on (subvec) as well.

The arg to rindex should probably be tagged with ^clojure.lang.Counted instead of ^String now.

Regarding the previous comment, String is a Java class so it isn't a clojure.lang.Counted. Is the type hint necessary? Maybe it should be on the call rather than the defn.

Ignoring the type hinting, I'll suggest a slightly simpler way to implement the rindex logic:

(defn rindex [coll i]
(if (neg? i) (+ (count coll) i) i))

In any case, I'm not sure rindex should be public even if you want the subs and subvec enhancements. Someone needs to make the case for adding a new function to core.

The Pythonic negative index is a debatable feature since it's pretty easy to implement for yourself if you want it.

Sorry, the type hint on rindex args isn't necessary at all. Just looked up in the source, calling count should never be reflective, since (count ..) emits calls to clojure.lang.RT/count.

Your solution looks good.

Negative indices were considered and rejected a long time ago. (I am merely conveying information--I have no strong opinion on this one.)

Note: If some people really like negative index behavior as in Perl or Python, it is straightforward to create a library of functions in a different namespace, perhaps with different names, that can do it. Perhaps a "pythonisms" library?

Would this be accepted as part of clojure.string instead? I considered putting it there, but thought it would be confusing to have multiple substring functions in different namespaces.

This is very helpful in practice, and I'd really like to see at least the (subs) stuff in Clojure.

Disclaimer: I'm no Clojure/core member, so can't speak authoritatively on whether something would or would not be accepted into clojure.string.

However, given that clojure.string is distributed with clojure.core, my guess is it would not be accepted. You'd be able to get things like this out for you and others as a separate library distributed on Clojars. That would also make it easy to include other Python-like things that you don't find in Clojure already.

This isn't about "python-like things," this is about a useful feature. Lots of languages support this: Perl, PHP, Ruby, Python, JavaScript, to name a few. Are you really suggesting that I should create a whole package for a version of a function in clojure.core with slightly different semantics? That's insane.

Anyway, I'm done wasting my time trying to navigate this hopelessly broken process. Maybe it would have been accepted if I included yet another way to interoperate with Java types.

Stuart, do you remember why specifically negative indexes were rejected? Developing a separate library for a minor improvement to an existing function sounds unreasonable.

some explanation about this topic was given by Rich Hickey himself here: http://news.ycombinator.com/item?id=2053908

citation:
"...Yes, there is a backlog from when it was just me, and it will take a while to whittle down. We have finite resources and have to prioritize. I can assure you we have more important things to concentrate on than your negative index substring enhancement, and are doing so. You'll just have to be patient. Or, if you insist, I'll just reject it now because a) IMO it's goofy, b) you can make your own function that works that way c) we don't get a free ride from j.l.String, d) it begs the question of negative indices elsewhere..."

i've been following this thread hoping this feature would be included. but whatever the reason was for the rejection, i'm sure it was thoughtful. great thanks for this wonderful language, and thanks Ian Eure for his effort.

That HN link eventually leads back to CLJ-688 which was rejected.

Michael: A proposal for negative indexes would need to be systematic in considering implications for all functions that have index arguments.

Ian: Clojure is driven by design, not incremental piling of features.

All: clojure.string is incomplete in more important and fundamental ways than negative indexes. This sucks now, and will suck worse as more code is written in different dialects. I find myself wishing string was a contrib, so we could iterate faster.

Stuart: Any specific proposals for how you'd like to see clojure.string improve? If it can be made a contrib, that would be cool, but understood if that would be considered too breaking of a change. Even if it isn't made a contrib, having tickets for improvement ideas you are willing to see means patches might get written, and they'll get in some time.

Philip, thanks for writing that patch. Could you please attach it as a file to this ticket, instead of copying and pasting it into a comment? Instructions are under the heading "Adding patches" on this page:

http://dev.clojure.org/display/design/JIRA+workflow

Rich Hickey's policy is only to include code in Clojure that is written by those who have signed a contributor agreement. You can find out more at http://clojure.org/contributing

Thanks Andy. Patch attached:

0001-clojure.data-diff-cope-with-falsey-values-in-maps.patch

I'll send in a CA.

CA snail-mailed yesterday, I guess it will take a week to arrive.

I now have a CA in place.

Yeah, this should be fixed.

Patch applies cleanly against 4004d267e124f12b65b0d7fb6522f32a75e3c4fb. Submitter is a confirmed CA signer.

Here's an implementation.

There's something strange with my patch. The creation of a composition of a huge seq of functions is much faster with `comp*` than with `comp`, which is expected, because the seq doesn't need to be reversed.

The strange thing however is that the compositions created with `comp` evaluate about 10-20% faster than those created with `comp*` although the arbitrary-arity version of `comp` is defined in terms of `comp*`: `(apply comp* (reverse (list* f1 f2 f3 fs)))`.

For some benchmarking details, see: https://groups.google.com/d/msg/clojure/MizwTxHwLE4/hGLrMfetlP8J

Here's some benchmark:

user> (use 'criterium.core)
nil
user> (let [coll (doall (take 1000000 (repeat inc)))
	   f1 (apply comp* coll) 
	   f2 (apply comp coll)] 
	   (bench (f1 0) :verbose) 
	   (println "---------------------------------------")
	   (bench (f2 0) :verbose))
amd64 Linux 3.4.2-gentoo 2 cpu(s)
OpenJDK 64-Bit Server VM 22.0-b10
Runtime arguments: -agentlib:jdwp=transport=dt_socket,server=y,suspend=n -XX:+TieredCompilation -Xmx1G -Dclojure.compile.path=/home/horn/Repos/clj/testi/target/classes -Dtesti.version=0.1.0-SNAPSHOT -Dclojure.debug=false
Evaluation count             : 600
             Execution time mean : 112.324465 ms  95.0% CI: (112.247218 ms, 112.380682 ms)
    Execution time std-deviation : 6.513809 ms  95.0% CI: (6.477450 ms, 6.553029 ms)
         Execution time lower ci : 105.609401 ms  95.0% CI: (105.609401 ms, 105.622918 ms)
         Execution time upper ci : 122.353763 ms  95.0% CI: (122.353763 ms, 122.405315 ms)
---------------------------------------
amd64 Linux 3.4.2-gentoo 2 cpu(s)
OpenJDK 64-Bit Server VM 22.0-b10
Runtime arguments: -agentlib:jdwp=transport=dt_socket,server=y,suspend=n -XX:+TieredCompilation -Xmx1G -Dclojure.compile.path=/home/horn/Repos/clj/testi/target/classes -Dtesti.version=0.1.0-SNAPSHOT -Dclojure.debug=false
Evaluation count             : 1440
             Execution time mean : 43.519663 ms  95.0% CI: (43.516732 ms, 43.524062 ms)
    Execution time std-deviation : 492.299089 us  95.0% CI: (490.829889 us, 494.198137 us)
         Execution time lower ci : 42.781398 ms  95.0% CI: (42.781398 ms, 42.781398 ms)
         Execution time upper ci : 44.157311 ms  95.0% CI: (44.157311 ms, 44.158513 ms)
nil

Ok, I've tracked down the performance difference. This comes from the fact that `reverse` creates a clojure.lang.PersistentList which can be iterated much faster than a (fully realized) LazySeq. If you provide your fncoll in `(apply comp* fncoll)` as vector or list, the application of the created composition is as fast as for `comp`.

So for me, the patch is good and makes sense.

I'd like to see something much faster than this.

For reference here is a benchmark of a non-reducers (seq based) process that uses partition-by

user=> (def x (vec (range 1e6)))
#'user/x
user=> (bench (reduce + (map count (partition-by #(or (zero? (mod % 3)) (zero? (mod % 5))) x))))
Evaluation count             : 60
             Execution time mean : 1.072157 sec  95.0% CI: (1.070606 sec, 1.073381 sec)
    Execution time std-deviation : 165.818282 ms  95.0% CI: (163.873585 ms, 168.271261 ms)
         Execution time lower ci : 972.562000 ms  95.0% CI: (972.562000 ms, 973.301850 ms)
         Execution time upper ci : 1.419148 sec  95.0% CI: (1.419148 sec, 1.419148 sec)

Found 7 outliers in 60 samples (11.6667 %)
	low-severe	 2 (3.3333 %)
	low-mild	 5 (8.3333 %)
 Variance from outliers : 85.8489 % Variance is severely inflated by outliers
nil
user=>

Same again using r/partition-by from reducer-partition-by.diff

user=> (bench (r/reduce + (r/map count (r/partition-by #(or (zero? (mod % 3)) (zero? (mod % 5))) x))))
Evaluation count             : 60
             Execution time mean : 1.418350 sec  95.0% CI: (1.417738 sec, 1.418948 sec)
    Execution time std-deviation : 66.736477 ms  95.0% CI: (66.186568 ms, 67.610777 ms)
         Execution time lower ci : 1.370419 sec  95.0% CI: (1.370419 sec, 1.370419 sec)
         Execution time upper ci : 1.544151 sec  95.0% CI: (1.544151 sec, 1.544156 sec)

Found 10 outliers in 60 samples (16.6667 %)
	low-severe	 2 (3.3333 %)
	low-mild	 8 (13.3333 %)
 Variance from outliers : 33.5591 % Variance is moderately inflated by outliers
nil
user=> 

(using https://github.com/hugoduncan/criterium for benchmarking)

same again for r/partition-by from reducers-partition-by2.diff

user=> (bench (r/reduce + (r/map count (r/partition-by #(or (zero? (mod % 3)) (zero? (mod % 5))) x))))
Evaluation count             : 180
             Execution time mean : 307.596806 ms  95.0% CI: (307.271339 ms, 307.961550 ms)
    Execution time std-deviation : 34.060809 ms  95.0% CI: (33.613169 ms, 34.416837 ms)
         Execution time lower ci : 285.339333 ms  95.0% CI: (285.339333 ms, 285.339333 ms)
         Execution time upper ci : 385.087950 ms  95.0% CI: (385.087950 ms, 385.087950 ms)

Found 10 outliers in 60 samples (16.6667 %)
	low-severe	 4 (6.6667 %)
	low-mild	 6 (10.0000 %)
 Variance from outliers : 73.8053 % Variance is severely inflated by outliers
nil
user=> 

same again driven using r/fold (for grins) instead of r/reduce

user=> (bench (r/fold + (r/map count (r/partition-by #(or (zero? (mod % 3)) (zero? (mod % 5))) x))))
Evaluation count             : 360
             Execution time mean : 215.214486 ms  95.0% CI: (214.915417 ms, 215.664236 ms)
    Execution time std-deviation : 36.588464 ms  95.0% CI: (36.305548 ms, 36.847846 ms)
         Execution time lower ci : 185.575000 ms  95.0% CI: (185.575000 ms, 185.575000 ms)
         Execution time upper ci : 287.605175 ms  95.0% CI: (286.547833 ms, 287.605175 ms)

Found 6 outliers in 60 samples (10.0000 %)
	low-severe	 3 (5.0000 %)
	low-mild	 3 (5.0000 %)
 Variance from outliers : 87.6303 % Variance is severely inflated by outliers
nil
user=> 

reducers-partition-by2.diff is faster, but I am not wild about introducing a type and a protocol. also not sure about the CollFold impl.

Let's leave fold out for this first patch, please.

reducer-partition-by3.diff is a cleaned up version of reducer-partition-by2.diff without fold

Per Andy Fingerhut's email reducer-partition-by3.diff was failing to apply. This patch should apply cleanly to current master.

Presumptuously changing Approval from Incomplete to None, since the reason for its being marked Incomplete seems to have been addressed with the latest patch.

should this be assigned to someone?

Any

I would prefer to see this, and other interop/reflective items, in a separate contrib lib.

Thanks for the feedback, Stu. Is there an existing contrib lib for stuff like this? Or should I create one?

Any environment that supports FreeDesktop, so Linux and BSD mainly.

Is this a duplicate of CLJ-896? Can you look at the patch there and see if there is any advantage to the patch for it vs. the one here? Or perhaps some combination that is an improvement on them both?

The issue is definitely a duplicate of CLJ-896. The proposed patches were different. CLJ-896 now has an attached patch that is an enhancement of what Jeremy Heiler created.

Tested with 1.3.0 and 1.4.0

Duplicate of CLJ-1021. Later ticket kept in preference to this one, because it has a patch and this one does not.

Mac Oracle JDK, Linux OpenJDK

I take back the part about (apply) being slow. I think it's more the fact that it's doing a linear scan on keys every single time.

(apply) does show up a lot in a profiler, but I haven't figured out why. (apply + (range big)) seems to even be slightly faster than (reduce +) on the same range.

Patch in proper format

Changed to :ttl-ms

Completed in f4d1bf9f1069ba875a7a6a8a65646b35c6fbfd8f