Patch available: unifies entry points, fixes (read-line) for clojure.main/repl, provides source-only jar
Stephen C. Gilardi
changes discussed here recently. Details below. Feedback welcome.
--Steve
[1] clojure.main no longer calls gen-class. Instead there is now a
stub clojure.main class written in Java so it is always available with
Clojure's runtime. Each of the entry points provided by class
clojure.main ensures that namespace clojure.main is loaded (using
"require") and then calls the corresponding fn in namepace clojure.main.
class clojure.main entry points:
- main : unified entry point for clojure. Can run a repl or a
script, calls clojure.main/main
- legacy_repl : called by the new, stubbed clojure.lang.Repl,
calls clojure.main/legacy-repl
- legacy_script : called by the new, stubbed clojure.lang.Script,
calls clojure.main/legacy-script
Because the clojure.main class is always compiled along with the rest
of the Clojure runtime, it can be Clojure's unified entry point
whether or not ahead-of-time compiled Clojure code is available.
The pre-unification Clojure entry points (clojure.lang.Repl,
clojure.lang.Script) are still available and now run via fns in
namespace clojure.main. As a result, those fns will get a good workout
even if users don't make any changes to how they drive clojure.jar.
[2] clojure.main/repl's handling of (read-line) was different than
that of clojure.lang.Repl and not useful. This was because
clojure.main/repl *always* reads from *in*, while clojure.lang.Repl
created its own LineNumberingPushbackReader on System.in instead. In
order to handle (read-line) well while still using *in* exclusively,
clojure.main/repl needed to become aware of lines in the input stream
rather than treating all whitespace the same. I've implemented that,
notably without any change to LispReader.java so loading from files
remains exactly the same.
Some visible changes to the repl:
- The repl now only prompts when the input stream is pointing to the
start of a new line of input.
- Multiple expressions on a line will only yield a single prompt.
Blank lines (or lines containing only whitespace) will be consumed and
yield a new prompt.
user=> 1 2
1
2
user=>
user=>
- (read-line) will read a fresh line if there is nothing (or only
whitespace) left on the current input line, or the (post-whitespace)
rest of the input line otherwise.
user=> (read-line)
123
"123"
user=> (read-line) 456
"456"
user=>
In the first example, the repl waited on the line after (read-line)
and I typed 123. This should be the most common case an gives the same
effect as clojure.lang.Repl used to. (Note that it doesn't have to be
a naked call to (read-line) as shown here, (read-line) from within
another function down the stack would behave the same way.)
In the second example, (read-line) read 456 from the rest of the
input line and printed the string that it read. It then waited for
input without prompting on the next (blank) line. This is unfortunate,
but I don't see a clean way around it--it's a consequence of code
running during "eval" (in this case the read-line) changing the input
stream in a way the repl can't predict. (read-line) consumes the
newline after 456 on the input line. Whatever is typed on the line
after "456" will be evaluated and printed. Just pressing return will
produce a prompt (as shown).
- Exceptions caught by the repl now cause the rest of the input line
to be skipped.
user=> 1 2 3 (/ 1 0) 4 5 6
1
2
3
java.lang.ArithmeticException: Divide by zero (NO_SOURCE_FILE:0)
user=>
- Comments to end of line (;) are now supported at the repl. This can
be convenient when using a session transcript to demonstrate something.
user=> (use 'clojure.set) ; pull in the set functions
nil
user=> 3 4 ; 5 6
3
4
user=>
[3] clojure.lang.LineNumberingPushbackReader extends
java.io.PushbackReader, but was not always honoring the latter's
contract that when a character is successfully "unread", it will be
the next character read. In the case of LNPBR's readLine, the pushed
back character was ignored. Most of the time this had no significant
effect, but I needed to fix it to support (read-line) from the same
stream. These examples show the problem and the fixed behavior:
WAS:
user=> (with-in-str "12\n3" [(read-line) (read)])
["12" 3]
user=> (with-in-str "12\n3" (.unread *in* (.read *in*)) [(read-line)
(read)])
["2" 13]
user=>
IS:
user=> (with-in-str "12\n3" [(read-line) (read)])
["12" 3]
user=> (with-in-str "12\n3" (.unread *in* (.read *in*)) [(read-line)
(read)])
["12" 3]
[4] The "prompt" hook for clojure.main/repl now takes an argument: a
keyword that indicates where on the line the input stream is pointing.
The "only prompt on a fresh line" behavior is implemented by only
printing the prompt when the "where" argument is :line-start. Anyone
using the prompt hook will need to update.
[5] The "-e" and "--eval" options for the command line of clojure.main
now evaluate all the expressions present in their string argument and
print all non-nil return values using prn.
% java -cp clojure.jar clojure.main -e "(use 'clojure.set) (union
#{:a} #{:b})"
#{:b :a}
%
[6] Updated clojure/build.xml with these changes:
- changed the example in description to avoid "java -jar". It now
recommends:
java -cp clojure.jar clojure.main
- Changed names of compile_java and compile_clojure targets to
compile-java and compile-clojure
- Renamed "jar" target to "clojure" -- it builds clojure.jar
- Added a "clojure-slim" target that builds clojure-slim.jar. That
jar contains compiled Java code, but no compiled Clojure code. It is
functionally equivalent to clojure.jar, but it's much smaller--trading
reduced size for increased startup/execution time until all the
Clojure code a program uses is JIT compiled into memory by Clojure.
- Added a legacy "jar" target that (also) builds clojure.jar
- Added an "all" target that builds both clojure.jar and clojure-
slim.jar
- Made "all" the default target
Notes:
- In contrast to most files in Clojure,
LineNumberingPushbackReader.java has DOS line endings. If you have
trouble applying this patch to that file, that may be the cause. Rich,
please consider converting LNPBR.java to use unix line endings.
- src/jvm/clojure/main.java is a new file produced by the patch
Rich Hickey
Wow, that's a lot of changes. I'm not sure about this one:
- Exceptions caught by the repl now cause the rest of the input line
to be skipped.
I'm quite concerned about compatibility, especially with all the read-
line stuff. I remember some nightmares:
http://groups.google.com/group/clojure/browse_frm/thread/e818dabbd9ce7cb5/cfcc9d37934f844a
http://groups.google.com/group/clojure/browse_frm/thread/16843146286cc16c/795e028ed81905c0
Are these changes compatible with the interactions done via, say,
emacs inferior lisp mode? It's important to consider the fact that the
repl can be interacted with by programs, not just people.
Has anyone tried this patch under Windows?
There also seems to be a difference in behavior between clojure.main
and Repl regarding shutdown:
http://groups.google.com/group/clojure/msg/eea16032c5b6b591
is that difference in the patch?
I guess I'm looking for recommendations about how best to test this
short of a push-it-and-(briefly)-break-all-svn-users.
Rich
Stephen C. Gilardi
Thanks!
Wow, that's a lot of changes. I'm not sure about this one:
- Exceptions caught by the repl now cause the rest of the input line to be skipped.
In what way is that the right thing to do?
I'm quite concerned about compatibility, especially with all the read-line stuff. I remember some nightmares:
Are these changes compatible with the interactions done via, say, emacs inferior lisp mode? It's important to consider the fact that the repl can be interacted with by programs, not just people.
Has anyone tried this patch under Windows?
There also seems to be a difference in behavior between clojure.main and Repl regarding shutdown:
http://groups.google.com/group/clojure/msg/eea16032c5b6b591
is that difference in the patch?
I guess I'm looking for recommendations about how best to test this short of a push-it-and-(briefly)-break-all-svn-users.
Stephen C. Gilardi
I am aware of in its predecessor.
On Dec 17, 2008, at 6:10 PM, Stephen C. Gilardi wrote:
> On Dec 17, 2008, at 2:27 PM, Rich Hickey wrote:
>
>> In what way is that the right thing to do?
>
>
> The idea was that the sequence of top level forms may each depend on
> the success of the previous one. It can easily be changed to
> consider them all independent and I'll be happy to do that.
Top level forms are now independently evaluated. As always, only the
most recent exception is saved in *e so stack trace information can be
lost in the case of multiple exceptions on the same input line. I
think that's ok.
>> I'm quite concerned about compatibility, especially with all the
>> read-line stuff. I remember some nightmares:
>
> I know you're talking about unexpected nightmares. I think testing
> by others would be wise and I welcome it. These specific ones came
> up in my own testing I believe the patched code handles them well.
>
>>
> [details]
(read-line) now works in an unsurprising, easy to explain way. All
kinds of reading from the input stream during eval are now supported
and they no longer interfere with prompting.
The behavior is that after reading a top level form and before
evaluating it, if the input stream is pointing at a newline character,
the repl will consume that newline before evaluating the form. If the
top level form doesn't end the input line, any reading done by the
code being eval'd begins right after the form and all subsequent
characters including whitespace and newlines will be seen.
Without this special handling, (read-line) from the repl would always
return "".
(Alternative special handling would skip whitespace after the top
level form rather than only a single newline character. I think the
latter is preferable, but it would be easy to change to the other
option.)
Here are some examples with notes below:
Clojure
1 user=> (read-line)
2 123
3 "123"
4 user=> (read-line)123
5 "123"
6 user=> (read-line) 1 2 3
7 " 1 2 3"
8 user=> [(.read *in*)(.read *in*)(.read *in*)(.read *in*)]
9 1234
10 [49 50 51 52]
11 user=> [(.read *in*)(.read *in*)(.read *in*)(.read *in*)]
12 12
13 34
14 [49 50 10 51]
15 4
16 user=> [(.read *in*)(.read *in*)(.read *in*)(.read *in*)]1
17 23456
18 [49 10 50 51]
19 456
20 user=> (.read *in*)
21 1 2 3
22 49
23 2
24 3
25 user=>
Notes:
1. The newline the user entered after ")" is eaten. The repl blocks on
the next line waiting for input. A string containing the entire line
of input is returned as the value of the (read-line).
4. There is no newline immediately after ")", so the rest of the line
is returned by (read-line)
6. Same situation here. Note that the returned value begins with
whitespace.
8. Reading 4 character codes into a vector. They are the codes for 1,
2, 3, and 4.
11. Reading 4 character codes into a vector. They are the codes for 1,
2, newline, and 3. The subsequent 4 remains on the input stream and is
read/eval/printed.
16. Reading 4 character codes into a vector. The first comes from
after the top level form. Then newline, 2, and 3. Then 456 is read/
eval/printed.
20. One character code is read, then 2 and 3 are read/eval/printed.
>> Are these changes compatible with the interactions done via, say,
>> emacs inferior lisp mode? It's important to consider the fact that
>> the repl can be interacted with by programs, not just people.
In those I've tested, it has worked fine. The "prompt only when
there's nothing left to read/eval/print from the most recent user
input" behavior is now configurable via a ":need-prompt" (function)
argument to the repl. There is an opportunity to prompt after each top
level read/eval/print is complete. If (need-prompt) returns true, the
prompt will be printed. The default need-prompt function cooperates
with LineNumberingPushbackReader to determine when we're about to wait
for the user to type on a new blank line and prompts only then. If
some software requires prompting for every top level read/eval/print,
code that launches the associated repl can pass in #(identity true)
for need-prompt.
> Could someone please try it with Gorilla?
It appears nobody has tried this.
>> Has anyone tried this patch under Windows?
I tried it on Windows and it works fine. I learned something important
here. LineNumberingPushbackReader wraps a LineNumberReader stream. The
latter collapses all three supported line terminators: CR, LF, and
CRLF into a single \n character. We've been running with that all
along. I've taken advantage of that to simplify clojure.main and my
modifications to LineNumberingPushbackReader and noted that I'm using
that behavior in comments and doc strings.
>> There also seems to be a difference in behavior between
>> clojure.main and Repl regarding shutdown:
>>
>> http://groups.google.com/group/clojure/msg/eea16032c5b6b591
>>
>> is that difference in the patch?
clojure.main/main now calls (flush) and (System/exit 0) to address the
reported shutdown issues.
>> I guess I'm looking for recommendations about how best to test this
>> short of a push-it-and-(briefly)-break-all-svn-users.
>
> I'm open to any ideas along those lines. Are there volunteers to
> test? to review the code? to suggest a better way to fix read-line?
I think there won't be much if any breakage and I'll be quick to
respond if there is. In this thread, we've given folks the opportunity
to give it a try and give feedback. Unfortunately there hasn't been
any. I would encourage interested folks to give this patch a try,
review the code, and speak up.
I'll be happy to fix any problems anyone finds with this.
--Steve
Mark McGranaghan
Thanks for your work on polishing up Clojure's entry point situation.
I've applied your patch and tried a series of typical entry point
scenarios; all but one of them worked for me. Just for the record,
here is what did work:
- Using "-h" to get help
- Using no options to boot into a repl
- Running the repl with jLine
- Using "-" to read from stdin
- Running a script located based on a filesystem path
- Running a script located relative to classpath
- Using "-i" to load files before running a script
- Using set! in one of these init files
- Accessing *command-line-args* from a script
The one that didn't work was booting into a line-numbered repl, which
is the setup I use now with trunk Clojure and Clojure-Contrib:
java -cp clojure.jar:/Users/mmcgrana/Clojure/jars/clojure-contrib.jar
clojure.main -e "(use 'clojure.contrib.repl-ln)" -e "(repl)"
This just hangs without giving a prompt, and then when I press enter
it goes into an infinite loop of error messages.
Perhaps I'm just using the wrong approach to booting in the line-numbering repl?
- Mark
Stephen C. Gilardi
Thanks for your work on polishing up Clojure's entry point situation.
I've applied your patch and tried a series of typical entry point
scenarios; all but one of them worked for me. Just for the record,
here is what did work:
- Using "-h" to get help
- Using no options to boot into a repl
- Running the repl with jLine
- Using "-" to read from stdin
- Running a script located based on a filesystem path
- Running a script located relative to classpath
- Using "-i" to load files before running a script
- Using set! in one of these init files
- Accessing *command-line-args* from a script
The one that didn't work was booting into a line-numbered repl, which
is the setup I use now with trunk Clojure and Clojure-Contrib:
java -cp clojure.jar:/Users/mmcgrana/Clojure/jars/clojure-contrib.jar
clojure.main -e "(use 'clojure.contrib.repl-ln)" -e "(repl)"
Mark McGranaghan
Your version of repl_ln.clj works for me:
1:1 user=> (+ 1 2)
3
1:2 user=> (throw (Exception. "test"))
java.lang.Exception: test (repl-1:2)
Thanks again,
- Mark
Mark McGranaghan
The (System/exit 0) bit at the end of the main function changes the
behavior of some scripts I have for starting Jetty servers. The
scripts basically end with (.start server), were server is a Jetty
Server instance, and previously would run until control-C'd from the
command line. I see in the linked thread above that this change in
behavior is required by other aspects of Clojure, which makes senes to
me. Do you know of any straiforward change I could make to these
scripts to keep the servers running? Perhaps this is a matter of the
Jetty APIs and not Clojure.
I'm not using Compujre here, but looking quickly at Compojure it seems
like it uses (.start server) as well from scripts.
- Mark
Stephen C. Gilardi
--~--~---------~--~----~------------~-------~--~----~
You received this message because you are subscribed to the Google Groups "Clojure" group.
To post to this group, send email to clo...@googlegroups.com
To unsubscribe from this group, send email to clojure+u...@googlegroups.com
For more options, visit this group at http://groups.google.com/group/clojure?hl=en
-~----------~----~----~----~------~----~------~--~---
Rich Hickey
when run in repl or script modes respectively, it needs to. Repl
calls exit, Script does not.
Rich
Stephen C. Gilardi
If main doesn't match the behavior of Repl and Script in this area
when run in repl or script modes respectively, it needs to. Repl
calls exit, Script does not.
Rich Hickey
Rich