When calling (read-line) from lein repl or ccw repl, waiting around four minutes (on my machine),
evaluating any form causes the following exception:

ERROR: Unhandled REPL handler exception processing message {:code (+ 1 1), :id 08e30b70-ab0f-4a47-8937-3c040ce06f4f, :ns user, :op eval, :session 916200cc-e476-4ac4-98f6-76057a9020be}
java.io.IOException: Write end dead
	at java.io.PipedReader.ready(Unknown Source)
	at clojure.tools.nrepl.middleware.session.proxy$java.io.PipedReader$0.ready(Unknown Source)
	at java.io.BufferedReader.ready(Unknown Source)
	at java.io.FilterReader.ready(Unknown Source)
	at java.io.PushbackReader.ready(Unknown Source)
	at clojure.tools.nrepl.middleware.session$add_stdin$fn__1103.invoke(session.clj:197)
	at clojure.tools.nrepl.middleware$wrap_conj_descriptor$fn__840.invoke(middleware.clj:17)
	at clojure.tools.nrepl.middleware.session$session$fn__1093.invoke(session.clj:165)
	at clojure.tools.nrepl.middleware$wrap_conj_descriptor$fn__840.invoke(middleware.clj:17)
	at clojure.tools.nrepl.server$handle_STAR_.invoke(server.clj:18)
	at clojure.tools.nrepl.server$handle$fn__1132.invoke(server.clj:27)
	at clojure.core$binding_conveyor_fn$fn__4130.invoke(core.clj:1836)
	at clojure.lang.AFn.call(AFn.java:18)
	at java.util.concurrent.FutureTask$Sync.innerRun(Unknown Source)
	at java.util.concurrent.FutureTask.run(Unknown Source)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
	at java.lang.Thread.run(Unknown Source)

While googling a bit on this and without concrete knowledge of the implementation in nrepl,
it seems like closing a thread that wrote to PipedWriter, before closing the writer or writing to it from another thread,
causes PipedReader#ready to throw that exception.

Syntax errors happen when the user has bound things like *print-length*, *print-depth* and those take precedence.

See https://github.com/technomancy/leiningen/issues/952

Based on discussion in that ticket, I think the right approach is to rebind these vars when doing pr-str in load_file.clj - but I haven't yet been able to reproduce locally yet.

When trying out remote eval using your sample in the README, ie

(with-open [conn (repl/connect :port 59258)]
(-> (repl/client conn 1000) ; message receive timeout required
(repl/message {:op "eval" :code "(+ 2 3)"})
repl/response-values))

I'm noticing that hosting process leaking a thread each time the remote eval is called. Jconsole shows a clojure-agent-send-off-pool-xxx thread got spawn as result of the call. The stack appears to be pointing to the "while true" loop inside fn_transport.

java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:458)
java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:359)
java.util.concurrent.SynchronousQueue.put(SynchronousQueue.java:878)
clojure.tools.nrepl.transport$fn_transport$fn__3912.invoke(transport.clj:44)
clojure.core$binding_conveyor_fn$fn__3989.invoke(core.clj:1819)
clojure.lang.AFn.call(AFn.java:18)
java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
java.util.concurrent.FutureTask.run(FutureTask.java:166)
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
java.lang.Thread.run(Thread.java:722)

What do you recommend as way to free the thread? I have server code that calls nrepl on behave of client connections and the number of call can pile up fairly quickly.

I was accidentally printing the reference returned by start-server by calling it as the final function in my main method, which causes this error:

java.lang.IllegalArgumentException: Multiple methods in multimethod 'print-method' match dispatch value: class clojure.tools.nrepl.server.Server -> interface clojure.lang.IDeref and interface clojure.lang.IRecord, and neither is preferred
at clojure.lang.MultiFn.findAndCacheBestMethod(MultiFn.java:136)
at clojure.lang.MultiFn.getMethod(MultiFn.java:111)
at clojure.lang.MultiFn.getFn(MultiFn.java:119)
at clojure.lang.MultiFn.invoke(MultiFn.java:167)
at clojure.core$pr_on.invoke(core.clj:3266)
at clojure.core$pr.invoke(core.clj:3278)
at clojure.lang.AFn.applyToHelper(AFn.java:161)
at clojure.lang.RestFn.applyTo(RestFn.java:132)
at clojure.core$apply.invoke(core.clj:601)
at clojure.core$prn.doInvoke(core.clj:3311)
at clojure.lang.RestFn.invoke(RestFn.java:408)
at clojure.main$eval_opt.invoke(main.clj:299)
at clojure.main$initialize.invoke(main.clj:316)
at clojure.main$null_opt.invoke(main.clj:349)
at clojure.main$main.doInvoke(main.clj:427)
at clojure.lang.RestFn.invoke(RestFn.java:421)
at clojure.lang.Var.invoke(Var.java:419)
at clojure.lang.AFn.applyToHelper(AFn.java:163)
at clojure.lang.Var.applyTo(Var.java:532)
at clojure.main.main(main.java:37)

If I try to run lein repl outside of a project folder, I get Clojure 1.4.0.
I then run

(println (javax.xml.bind.DatatypeConverter/parseDateTime "2008-07-21T19:17:29"))

which produces

IndexOutOfBoundsException start 26, end 2, s.length() 28 java.lang.AbstractStringBuilder.append (AbstractStringBuilder.java:476)
#inst "2008-07-21T19:17:29.000+01:

Note the absence of a closing '"'.

My apologies in advance if this is impossible to reproduce.

Not sure whether this ticket belongs here or in the Clojure-proper JIRA, so feel free to close if this is an inappropriate location. clojure.main/repl creates a new DynamicClassLoader on every execution, so it looks like the stack of classloaders grows without bounds. Seems a bit similar to http://dev.clojure.org/jira/browse/NREPL-31 in that clojure.main/repl has another assumption about when clojure.main/repl will run.

See https://groups.google.com/forum/?fromgroups=#!topic/clojure/firG9zTVecU%5B1-25%5D for the original report.

...maybe push the protocol details into a /doc directory.

When specifying middleware, it would be much easier for the user to be able to override a default middleware without having to specify a handler.

For example, if there is a default middleware providing the "complete" operation, the user should be able to just specify their preferred completion middleware, without having to specify all middleware as a handler.

One way to do this might be to check metadata for the provided operations of the specified metadata, and ensure that either the default middleware for that operation is removed, or that the specified middleware takes precendence (which may be simpler when a middleware provides multiple operations).

There are several uses of reflection in tools.nrepl that can be eliminated with suitable type hints.

The refers of REPL utilities (pst, pprint, etc) happens in clojure.main/repl. In a terminal/single-threaded REPL, this is called just once, so it only ever affects the user namespace. Every expression sent for evaluation by nREPL invokes clojure.main/repl though, so moving *ns* around will inadvertently cause those refers to happen over and over, into non-user namespaces. (I've been enjoying having pprint and pp available all the time, but I'd never thought much about why they were always there.)

In the end, this is an nREPL bug.

I don't see any easy way out off the top of my head. I think nREPL will end up having to stop using clojure.main/repl, and maintain a modified version of it itself (something I wanted to avoid exactly so as to benefit from the changes to clojure.main/repl from version to version of Clojure).

Suggestions most welcome.

(Originally reported here.)

7888 is "free", at least in IANA.

Most users want to put an nREPL port on their app, and that should always be on a typical port. Auto-selection of a port is only desirable in tooling scenarios, and tooling authors can pass an argument to start-server.

This will change the behaviour of (start-server).

...particularly important where less sophisticated clients and polling-oriented transports are involved (e.g. HTTP).

When running the tests, I'm occasionally getting SocketExceptions (which don't fail the test suite) with the message "Broken pipe". I think these stem from one end of the nREPL socket trying to write to the socket after the other end has gone away.

There's an existing test (ensure-closeable) that explicitly expects a SocketException to be raised if the socket has had .close called directly on it, rather than having the other end hang up, which I think is what the "Broken pipe" tells us. I'm curious as to whether this is important for some reason I haven't seen, or whether catching all SocketExceptions on transport/bencode's write function and returning nil would be appropriate.

If the end doing the socket closing actually should affect the behavior, we could sniff the exception message, but that feels a bit yucky.

I can actually reproduce this in "real life" by firing up an nREPL server / client (`reply --nrepl --port 9999`), then attaching to it (`reply --nrepl --attach 9999`), and in the attached client, run (Thread/sleep 10000) and immediately kill the process. Then the "Broken pipe" exception shows up in the original REPL with the server in the same process.

I've got a patch for this in my bitbucket fork (socket-exception branch): https://bitbucket.org/trptcolin/nrepl/changeset/4a432f3ce95a

Simply interpolating contents of files to be loaded into a Compiler/load expression to be evaluated does not work with very large files. (See http://code.google.com/p/counterclockwise/issues/detail?id=429)

ritz re-uses some of nrepl's private functions to avoid duplication. The uses are listed below.

Would it be possible to make these functions public? More subjectively, it might also be worth considering factoring out the session functionality into it's own namespace (including msg and possibly queue-eval), so the functionality is not split across the session middleware and the interruptible-eval middleware.

The debug nrepl server:
https://github.com/pallet/ritz/blob/develop/nrepl/src/ritz/nrepl.clj#L189

This uses clojure.tools.nrepl.middleware.session/create-session and clojure.tools.nrepl.middleware.session/session-out

ritz provides an eval op that tracks source forms:
https://github.com/pallet/ritz/blob/develop/nrepl-middleware/src/ritz/nrepl/middleware/tracking_eval.clj

This uses clojure.tools.nrepl.middleware.interruptible-eval/queue-eval and clojure.tools.nrepl.middleware.interruptible-eval/configure-executor

See https://github.com/trptcolin/reply/issues/84#issuecomment-8481753 and maybe http://code.google.com/p/counterclockwise/issues/detail?id=405

Basically, (send-off some-agent println) & co. should get what's printed to out in that agent's thread back to the nREPL client — not silently let it dump out to System/out.

Portal ostensibly does this well. Examine their approach, see if it is compatible with nREPL's objectives.

Ill-formed brain dump:

• multiplex new out's to System/out
  • (still won't solve clojure.test/test-out content will disappearing into the ether when it's loaded when out is bound to an nREPL out; maybe we should ensure out is bound to System/out while code is being loaded?)
• optionally multiplex System/out and System/err
• optionally join multiplexed S/out and S/err, receive :stdout, :stderr msgs

The session middleware takes the :session key from the message, and uses it as an id to lookup session data, which it then places on the same key.

Using a separate key for the session id and the session data would be less confusing, and would allow easier checking of the availability of the session data in the message map.

If you want to use nREPL on Android (currently only possible with sattvik's fork of Clojure) then you need bigger stack sizes than Android's default 8k to eval even moderately complex expressions. For comparison: A regular JVM on a PC has default stack sizes around 256k ... 512k depending on platform.

In nrepl.clj, modify

(doto (Thread. r)
(.setDaemon true))))))

to read

(doto (Thread. (.getThreadGroup (Thread/currentThread)) r "nREPL" 524288) ; 512k generous stack size
(.setDaemon true))))))

Note: There are warnings here http://bit.ly/u86tF1 about choosing a good stack size but since in practice nREPL seems to have <10 threads running with 1 active client connection we can be generous here.

The more advanced version would be to make this somehow configurable but that'd probably be over-engineering things.

In particular, no boxing of individual bytes or multiple en/decoding steps should be involved.

This implies that:

1. all string values should remain byte[] coming out of the bencode protocol support (which makes it more compliant with bencode anyway)
2. the string decoding should occur in the bencode transport impl
3. all string values should be decoded, except for those named in a reserved slot (say, "-unencoded")

All map keys should always be decoded to Strings.

While nREPL should continue to be fully compatible with Clojure 1.2.x, let's not kill ourselves trying to optimize for 1.2.x as well as Clojure >= 1.3.0; if necessary, just make sure the latter is fast. If 1.2.x benefits as well, then good.

If you want to use nREPL on Android (currently only possible with sattvik's fork of Clojure) then you need bigger stack sizes than Android's default 8k to eval even moderately complex expressions. For comparison: A regular JVM on a PC has default stack sizes around 256k ... 512k depending on platform.

In nrepl.clj, modify

(doto (Thread. r)
(.setDaemon true))))))

to read

(doto (Thread. (.getThreadGroup (Thread/currentThread)) r "nREPL" 524288) ; 512k generous stack size
(.setDaemon true))))))

Note: There are warnings here http://bit.ly/u86tF1 about choosing a good stack size but since in practice nREPL seems to have <10 threads running with 1 active client connection we can be generous here.

The more advanced version would be to make this somehow configurable but that'd probably be over-engineering things.

Not sure how this should be configured, but it would be nice if doing something like (println large-seq-here) would flush output automatically after some number of characters. This is so that output could stream constantly across the nrepl connection instead of waiting for the entire output to send as one message (assuming no manual flushes).

Running -main from clojure.tools.nrepl.cmdline gives the following exception:

Exception in thread "main" java.lang.IllegalArgumentException: No value supplied for key: 0
at clojure.lang.PersistentHashMap.createWithCheck(PersistentHashMap.java:89)
at clojure.core$hash_map.doInvoke(core.clj:355)
at clojure.lang.RestFn.applyTo(RestFn.java:137)
at clojure.core$apply.invoke(core.clj:600)
at clojure.tools.nrepl.server$start_server.doInvoke(server.clj:65)
at clojure.lang.RestFn.invoke(RestFn.java:408)
at clojure.tools.nrepl.cmdline$_main.doInvoke(cmdline.clj:74)
at clojure.lang.RestFn.invoke(RestFn.java:397)
at clojure.lang.Var.invoke(Var.java:397)
at clojure.lang.AFn.applyToHelper(AFn.java:159)
at clojure.lang.Var.applyTo(Var.java:518)
at clojure.core$apply.invoke(core.clj:600)
at clojure.main$main_opt.invoke(main.clj:323)
at clojure.main$main.doInvoke(main.clj:426)
at clojure.lang.RestFn.invoke(RestFn.java:421)
at clojure.lang.Var.invoke(Var.java:405)
at clojure.lang.AFn.applyToHelper(AFn.java:163)
at clojure.lang.Var.applyTo(Var.java:518)
at clojure.main.main(main.java:37)

It looks like the repl/connect and start-server functions have changed since cmdline.clj was written. I attached a patch that updates their usage.

When I start the nREPL on the Android device (by calling `(clojure.tools.nrepl.server/start-server :port 9999)`) it all goes well. But when I try to connect to this REPL using any of the clients I find myself in an empty namespace (or something like that). The var ns is unbound, no functions from the clojure.core are available. At the beginning REPLy tries to perform some actions but they fail (says that it cannot find symbol `defn` - because nothing from clojure.core is being mapped).

However I can do (in-ns 'anywhere) it works. Everything else in the REPL works correctly (as far as I see). The issue itself is minor but I'm afraid that it is caused by some crash during nREPL initialization that might lead to other problems in future.

As described here: https://github.com/technomancy/leiningen/issues/631, the *out*, *err*, and value printing are ordered differently than expected sometimes.

There are 2 things left here:
(1) The return value can sometimes print before synchronous prints to err/out. Synchronizing all the printing on the REPLy side doesn't seem to help. An example case:

=> (do (.print *err* "one") (print "two") (print "three"))
niltwothreeone

(2) Calling .println() on the *out*/*err* streams doesn't necessarily flush the stream. Changing this may have performance/network ramifications, but generally it seems like surprising behavior. An example case:

user=> (do (.println *out* "one") (.print *err* "two") (.print *err* "three") 1)
twothreeone
1

I'd expect two .print() calls to different streams to have undefined ordering, but the other orderings were all surprising.

The attached patch forces flushes to avoid both of these, for (1) immediately before sending the value message, and for (2) on the "normal" PrintStream [autoFlush] cases of println, printf, or format.

Side note: I couldn't get any tests to fail in this scenario, even after doing some reworking of the streams used in tests, but I kept one new test in this patch anyway. Turns out it's hard to write tests that really make sure .flush() gets called.

Failures like these are happening sometimes in the test suite, mainly around stdin:

FAIL in (request-multiple-read-objects-in) (nrepl_test.clj:211)
expected: (= (quote (:ohai)) (response-values (for [resp (repl-eval session "(read)")] (do (when (-> resp :status set (contains? "need-input")) (session {:op :stdin, :stdin ":ohai :kthxbai\n"})) resp))))
actual: (not (= (:ohai) nil))

FAIL in (request-multiple-read-objects-in) (nrepl_test.clj:217)
expected: (= [" :kthxbai"] (repl-values session "(read-line)"))
actual: (not (= [" :kthxbai"] nil))

I've done a ton of debugging and thinking about this, and haven't gotten far. A few theories I had that seemed decent at the time, but ultimately haven't gotten rid of these occasional failures:

• in middleware.session/add-stdin, flush the :stdin-writer after writing to it on :stdin messages (see http://stackoverflow.com/questions/2843555/better-alternative-for-pipedreader-pipedwriter)
• in server/handler, when a message is received on the transport, farm the handling out to a future (this is half-baked & caused additional intermittent failures)

The debugging I've been doing (print statements galore) suggests that when this problem happens, the client does send back the proper :stdin, but it doesn't get picked up by the add-stdin handler in time to write to the :stdin-writer and avoid the expiring timeout.

I'm pretty much stumped at this point, unfortunately.

There's a stray "#" in clojure.tools.nrepl.middleware.session/session-out which is turning the body of the (not off) case into a lambda that's never executed:

(cond
(number? x) (.append buf (char x))
(not off) #(.append buf x) ; <----
(instance? CharSequence x) (.append buf ^CharSequence x off len)
:else (.append buf ^chars x off len))

https://github.com/clojure/tools.nrepl/blob/master/src/main/clojure/clojure/tools/nrepl/middleware/session.clj#L34

This is causing the newline character in (.println out) and (.println err) to be lost:

user=> (with-open [conn (repl/connect :port 7888)]
(-> (repl/client conn 1000)
(repl/message {:op :eval :code "(do (.println out \"1\") (.println out \"2\") (.println out \"3\")(flush))"})
doall
pprint))
({:out "123", ;; <--- where are my "\n"s?
...}, ...)

Removing the stray "#" fixes the problem:

{:out "1\n2\n3\n",
...}

Reported downstream by jaceklaskowski in https://github.com/technomancy/leiningen/issues/631

Patch with the trivial fix and unit test here: https://wok.meshy.org/~ato/0001-Remove-stray-hash-in-session-out-that-breaks-println-out.patch

Couldn't attach the patch to this issue as JIRA told me I don't have permission. I (Alex Osborne) have signed the Clojure CA and contribute this patch as per the terms of it.

If the call to clojure.main/repl in `evaluate' raises an exception before the :init part gets evaluated (so the expression `(push-thread-bindings @bindings)' is executed) then nREPL crashes with the following exception:

java.lang.IllegalStateException: Pop without matching push

It happens because after the underlining exception in clojure.main/repl is caught by the try block in `evaluate', the `finally' black calls (pop-thread-bindings) which were not actually "pushed".

We talked about this briefly on IRC/github: https://github.com/overtone/overtone/issues/85. I guess the nicest way to fix it would be to stop using agents in nREPL. There's only one spot where agents are created, and a few send-off calls (no sends), all internal, so any solution will be non-intrusive from an API perspective.

One implementation idea is to use a simple blocking queue and thread pool in java-land.

Another is to use nearly the same implementation as Agent, but remove the nesting restriction (call it NestableAgent?). Also get rid of the STM interaction and validator business since those aren't available outside the clojure.lang package - and then provide a parallel version of send-off. I've spiked a bit on this one, but I'm starting to think a plain queue/thread pool is cleaner.

Thoughts?

nrepl currently accepts a :port to bind to as an option, and uses that with the single-arg ServerSocket constructor, which binds to that port on all interfaces. I propose we add a :host option as well that can be used to specify the hostname or address of the interface to bind to.

Here is a potential implementation:

diff --git a/src/main/clojure/clojure/tools/nrepl/server.clj b/src/main/clojure/clojure/tools/nrepl/server.clj
index 9dbac40..b6f625b 100644
--- a/src/main/clojure/clojure/tools/nrepl/server.clj
+++ b/src/main/clojure/clojure/tools/nrepl/server.clj
@@ -8,7 +8,7 @@
                                             pr-values
                                             session))
   (:use [clojure.tools.nrepl.misc :only (returning response-for log)])
-  (:import (java.net Socket ServerSocket)))
+  (:import (java.net InetAddress Socket ServerSocket)))
 
 (defn unknown-op
   "Sends an :unknown-op :error for the given message."
@@ -64,7 +64,9 @@
 
 (defn start-server
   "Starts a socket-based nREPL server.  Configuration options include:
- 
+
+   * :host — the address or hostname of the interface that should be used;
+       defaults to all interfaces
    * :port — defaults to 0, which autoselects an open port on localhost
    * :handler — the nREPL message handler to use for each incoming connection;
        defaults to the result of (default-handler)
@@ -77,8 +79,10 @@
 
    Returns a handle to the server that is started, which may be stopped
    either via `stop-server`, (.close server), or automatically via `with-open`."
-  [& {:keys [port transport-fn handler ack-port greeting-fn] :or {port 0}}]
-  (let [ss (ServerSocket. port)
+  [& {:keys [host port transport-fn handler ack-port greeting-fn] :or {port 0}}]
+  (let [ss  (if host
+              (ServerSocket. port 0 (InetAddress/getByName host))
+              (ServerSocket. port))
         smap {:ss ss
               :transport (or transport-fn t/bencode)
               :greeting greeting-fn

I'm putting a CA in the mail today.

Swank does this by polling back to the connected client to request input. In the face of asynchronous operations, this may be the only option.

Will require adding to the protocol.

The let in send-ack should be be with-open.

This:

=> (println " ")
nil

Should be this:

=> (println " ")
 
nil

Perhaps :out isn't being sent if the content {{.trim}}s down to a zero-length string?

When (read-line) is preceded by (read), it already has a newline in the stream, and returns "" immediately rather than prompting for input.

(read) then (read-line) works fine in the stock Clojure repl because the REPL shares STDIN with the in used by the normal (read) and (read-line) calls. So when the REPL's loop does clojure.main/repl-read (to get my evaluation input "(read-line)"), it skips whitespace before the read for the thing to eval, reads that thing, and then skips a newline, if that's the next thing: https://github.com/clojure/clojure/blob/master/src/clj/clojure/main.clj#L145-161

That allows the .read calls generated by the (read-line) execution to start after a trailing newline left in the stream.

We can do the same thing in nrepl.middleware.session/add-stdin on an eval message, but we need to check .ready first, to avoid sending unnecessary :need-input requests back to the client. I think this does belong in add-stdin, since it's essentially a stdin concern, even though it acts on the eval message.

I've got one test exposing the problem, and one proving we only eat newlines, not any old trailing character. I had to stick :stdin-reader on the session to enable consuming the newline from the middleware - not sure whether there's a way to do without it that I'm not seeing.

Running "mvn install" on tools.nrepl results in a build failure:

[INFO] Missing:
[INFO] ----------
[INFO] 1) org.clojure:clojure:jar:1.3.0-master-SNAPSHOT

Changing 1.3.0-master-SNAPSHOT to 1.3.0 in src/integration/clojure-1.3.0/pom.xml seems to fix it.