Re: IDE agnostic question on user assistance (emacs/vimClojure users help appreciated too !)

[Chas Emerick]

Automatically reloading namespaces into a REPL that I'm actively using is a very bad idea -- I expect to control what happens within REPLs I start, and subverting that is decidedly impolite IMO.  I can expect to see odd behaviour when that happens, insofar as I assume new code I've written hasn't yet been put into play.

(I actually didn't realize that this is how ccw's completion, etc were implemented, but I've not been able to verify that behaviour -- adding a new def to a source file and saving it does not seem to add the defined var to the REPL, so perhaps I'm misunderstanding things).

To answer your open question directly: if an IDE/plugin/whatever stops REPLs I've created (and potentially poured hours of time into to get data into particular states, etc), then I'll never touch that environment again.  Your suggestion (a) that code be automatically loaded in an IDE-managed and -started REPL is the right direction -- that's an implementation detail that I rightfully don't care about one way or the other.

The "desynchronization" issue that you're worried about is not a concern as far as I'm concerned, and I suspect that'd be a typical reaction.  A REPL should have only one point of input in the base case -- the user of the REPL.  Tinkering with that is violating a variety of expectations.  If the user wants to have some new code she's written loaded into the REPL, she'll do that of her own accord -- and if there's a variety of changed files, then having a "reload all" command associated with each directory or project makes sense.

Cheers,

- Chas

On Jul 19, 2010, at 8:35 AM, Laurent PETIT wrote:

Hi, 

I'm currently thinking about the next step for better user-assistance in Eclipse/counterclockwise.

But the questions I'm facing are general - enough so that they can be posted here.

Preliminary info:

user assistance (code completion, var documentation, etc.) is mainly obtained from a running instance of a REPL for the project.

Currently in ccw, I "try to be smart" by reacting as such when a file is saved by the user: I automatically reload all the project's namespaces into the project's running REPL (if there is one).

That way, user-assistance in the IDE is as accurate as possible, and I avoid desynchronisation (technically speaking, I currently "obtain" this feature by AOT-compiling the project's namespaces, but that's an implementation detail) between what is saved in the project's files, and what is loaded in the REPL. 

In the future, I intend to be even smarter :-) 

  * By being more "incremental" regarding which namespaces to call "reload" on. 

  * And also by providind ways for people with "corner-case projects" to disable the "automatic reload" feature for the whole project, or for specific namespaces.

  * But one question(*) remains open: should I stop to use the user-created REPL as the target of these "automatic reloads" ? Currently, if the user has not launched any REPL, he cannot benefit from IDE assistance requiring a running REPL. 

    a. So having an "IDE-dedicated live server REPL" seems like I could relieve the user from explicitly launching a REPL (it's weird and counter-intuitive for a user to have to somehow "manually" trigger the IDE user assistance !). 

    b. But if I do so, then the "automatic reloads" will now happen on the "IDE-dedicated server REPL", not on the REPL(s) the user will manually launch. And again there will be a desynchronization between the user's REPL loaded code and the project's saved files content ...

Users of Counterclockwise, Enclojure, La Clojure, please speak ! What behaviour do you expect from your IDE in this area ? (please do not answer in general terms, but try to the same precision-level of this email, or even more precise).

Users of Emacs / swank, vimClojure (etc.), please speak ! Share with us your workflows, why you think the goal I'm trying to achieve is good or not, so that we could think of better workflows to provide to IDE users if it seems appropriate.

Thanks in advance,

-- 

Laurent

(*) and many more, but I'd like to start with this one :-)

--
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clo...@googlegroups.com
Note that posts from new members are moderated - please be patient with your first post.
To unsubscribe from this group, send email to
clojure+u...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en

[Meikel Brandmeyer]

Hello Laurent,

please let me explain my philosophy before I go into details how VimClojure handles the clojure connection.

1. Vim is not an IDE like Eclipse or Netbeans. It is an editor which can be used with other tools to form an IDE, but on its own it is not.
2. The tools should only do what the user tells them to do. No more, no less.

While Vim's reality started deviate from these ideals long ago, I still feel that this is True Way™. And this is reflected in VimClojure's way of handling things.

So what does VimClojure's architecture look like:

• There is of course Vim itself. It sits there. Does nothing. Waits for user input.
• Then there is the server process running separately. It listens for connections from the Vim side and runs the requested commands. In particular you can arbitrary many repl, doc lookup, evaluation, whatever connections in parallel. You can forward the connection via ssh and you can embed the server in any application you can think of. I had both success with a web app and a swing app.
• The client, which Vim shells out to, to plumb both parties together. The client-server connection works via a modified nailgun, but that's an implementation detail.
• All this degrades gracefully and is not required to run a vanilla vim with syntax highlighting, indentation and builtin static completion. Doc lookup, code eval, macroexpand, etc. are just supplementary to the core tasks available also without the server.

One consequence of this architecture is that there is no continuous connection between both sides. However this is the only way Vim can handle such communication in a portable and stable way. The downside is that we cannot use interactive commands since the interaction is split over multiple connections. The upside is that one side going nuts doesn't affect the other too much. I can simply restart the server in the background with the Vim side noticing it, if the server goes banana. Some reloads and I can go on.

Starting the server is up to the user. Rule 1: Vim is not an IDE. There is a plethora of tools for handling classpaths questions. I personally use gradle; before that I used simple shell scripts with project relative CLASSPATH and that's it. ant+ivy, maven, leiningen, … all will they do a sufficient job. Vim is an editor. It has no business with Java classpaths. (For eclipse this is different: eclipse is expected to handle the classpath, no?) Besides that, you cannot portable start a background process from inside vim. (At least I'm not aware of how to do that)

I'm not sure I understand what you mean with „REPL“. I suppose it means „server instance“ in my settings. In that case there is only one. But it is not a repl. You can connect to it to obtain a repl connection (keeping book of the repl state etc.). But it is not in itself a repl.

As for a typical workflow of mine:

1. gradle runNailgun
2. Start vim
3. open clojure file
4. edit edit edit
5. lookup some docs
6. edit edit
7. save ; Note: nothing happens here. The file gets saved. That's it.
8. edit edit edit
9. Send some code via \et ; „eval toplevel“
10. Open a repl \sR ; „start repl“ R meaning in the namespace of the current buffer
11. repeat  3. - 10. as needed

You might note point 7: I do nothing on the server side on file safe. If I added function, it just isn't there for completion, yet. Only when I tell VC to evaluate the function via one of the \e* functions it is available for omni completion. I can live with such an inconsistent state, because I normally work on a very restricted set of functions. As soon as they are done, I do a complete sync by reloading the whole file etc if necessary. VC tries hard to keep line numbers as close to a synced state as possible. And as I said: if I mess up too much, I nuke the server and restart it without impact on my running vim. On the next interaction the namespaces are loaded cleanly from disk.

I agree with Chas, that strange things can happen. Imagine a pre-defonce-semantics-for-multimethods clojure version: you could loose your whole method definitions by a innocent looking file save. I prefer the „Launch rockets“-button to be more prominent. ;)

Puuh. I'm sorry this got so long, but I think I had to provide the necessary context to understand the why things are as they are with VimClojure. I hope I could answer your question.

Sincerely

Meikel

[Laurent PETIT]

That's interesting ! Thanks for the feedback.
Please note that it's not "impolite" in the sense that there's a policy: the project is reloaded in the REPL when the file is saved - that's a kind of "invitation" made by the user :-).
If you don't work from the files, but from the REPL, nothing will happen automatically.
If you work from the files, then you simply do not save the files until you're "happy" with the codebase. You can make several roundtrips between the file's content and the REPL (even via keyboard shortcuts) without saving the files. Ultimately, when you want to quit, you have to quit the REPL anyway, and you have the option to save the files or not.

Does this answer make sense ?

There's also the problem of "I save file A", and all project namespaces are reloaded as well. By maintaining a dependency graph of the namespaces relationships (easily obtained dynamically via the REPL), I may be able to only reload the expected namespaces. Some may see this as a feature - ensuring that all functions which depend on namespace A macros are recompiled -, I can see that some may see this as a recipe for disaster if reloading the namespaces also reloads global-var-as-data contents. Proper use of defonce may help there ?

Anyway, I didn't expect you reaction to be soo opinionated, and it's refreshing to hear others thoughts.

Let's see how others react to what you wrote !

Cheers,

--
Laurent

2010/7/19 Chas Emerick <ceme...@snowtide.com>

[Meikel Brandmeyer]

Hi,

Am 19.07.2010 um 20:50 schrieb Laurent PETIT:

> Please note that it's not "impolite" in the sense that there's a policy: the project is reloaded in the REPL when the file is saved - that's a kind of "invitation" made by the user :-).
> If you don't work from the files, but from the REPL, nothing will happen automatically.
> If you work from the files, then you simply do not save the files until you're "happy" with the codebase. You can make several roundtrips between the file's content and the REPL (even via keyboard shortcuts) without saving the files. Ultimately, when you want to quit, you have to quit the REPL anyway, and you have the option to save the files or not.
>
> Does this answer make sense ?

It does make sense, but it doesn't work for me. I work a lot in the repl but only with files. %) If I have correctly set up file in the classpath, where I edit my function and which a save quite often. (Sometimes in a broken state.) In the repl I only define small mock functions for quick testing of the functions which were defined in the file and sent over via code evaluation.

If you don't want to settle on one way, I would suggest, that you make the behaviour configurable.

> There's also the problem of "I save file A", and all project namespaces are reloaded as well. By maintaining a dependency graph of the namespaces relationships (easily obtained dynamically via the REPL), I may be able to only reload the expected namespaces. Some may see this as a feature - ensuring that all functions which depend on namespace A macros are recompiled -, I can see that some may see this as a recipe for disaster if reloading the namespaces also reloads global-var-as-data contents. Proper use of defonce may help there ?

This is something I would support. I think it should be easy to get a clean system after some changes. Then all changed macros would be recompiled and the multimethod wouldn't require the defonce semantics, since it's easy to reconstruct the methods. I could imagine to provide such a „reload dependees“ feature also in VimClojure. Then everyone can choose whether to use it, or not.

Sincerely
Meikel

[Barry Dahlberg]

As a Clojure newbie coming from Microsoft land I was surprised by the answers here.

Eclipse / CCW is the IDE I've used most so far and I've always wondered why it didn't just open the REPL for me and have it always there.  Having it up to date with my latest code files is great because I mostly use it to test things I've just written and saved.  Reloading all the namespaces etc is something I haven't got my head around yet, especially since my brain has a very small amount of RAM and I can't remember what is current.

Do people work with more than one REPL?  What do you do with them?  Are there any good articles or videos around talking about this?

Cheers
Barry

[Peter Schuller]

> Users of Emacs / swank, vimClojure (etc.), please speak ! Share with us your
> workflows, why you think the goal I'm trying to achieve is good or not, so
> that we could think of better workflows to provide to IDE users if it seems
> appropriate.

To the limited extent that I've developed with Slime (and this goes
for both Common Lisp and clojure), one annoyance I have perceived is
de-normalization of the image/running repl. While having a running
live repl is great in many ways, when it comes specifically to
developing your "files on disk" code, I tend to want my testing
environment to be as normalized as possible with respect to my source
code. So for example, ideally the removal of a function from a
namespace in the code would be reflected when reloading/recompiling
that file.

I understand why this doesn't happen automatically by default
(reloading just re-sets or adds what's defined in the file), in an
ideal world that is something that I'd like an IDE to "somehow" get
around. If it is possible to do that while at the same time allowing
my nice dynamic repl to be active, so much the better.

Examples of things affected in practice:

* (deftest ...) which mutates the vars in the namespace; removal of
tests don't "stick"
* moving functions between namespaces; things can either break for
bogus reasons or *not* break for bogus reasons that you don't notice
until you run 'mvn test' (or similar) separately

I also sympathize and agree that it can be hugely problematic if an
environment stomps on your carefully prepared REPL in ways you don't
expect. I'm not sure how to best reconcile these two concerns.

Possibly easy access to a fresh re-spawn:ed REPL with command line
history retained might make a fresh normalized REPL easily accessible
in a way that doesn't involve re-typing lots of stuff (while clearly
not preserving actual state other than history)?

--
/ Peter Schuller

[Meikel Brandmeyer]

Hi,

Am 19.07.2010 um 21:56 schrieb Peter Schuller:

> To the limited extent that I've developed with Slime (and this goes
> for both Common Lisp and clojure), one annoyance I have perceived is
> de-normalization of the image/running repl. While having a running
> live repl is great in many ways, when it comes specifically to
> developing your "files on disk" code, I tend to want my testing
> environment to be as normalized as possible with respect to my source
> code. So for example, ideally the removal of a function from a
> namespace in the code would be reflected when reloading/recompiling
> that file.

It seems it does that already:

Clojure 1.2.0-master-SNAPSHOT
user=> (require 'test.file2)
WARNING: prn already refers to: #'clojure.core/prn in namespace: test.file2, being replaced by: #'test.file2/prn
nil
; function prn is removed here from the file
user=> (require :reload 'test.file2)
nil
user=> (test.file2/prn 5)
java.lang.Exception: No such var: test.file2/prn (NO_SOURCE_FILE:4)

I haven't tested what happens to the deftest stuff, but the „moving function“ could be addressed.

Sincerely
Meikel

[Mark Rathwell]

:reload reloads the specified namespace

There is also :reload-all, which reloads the specified namespace as well as all namespaces import'ed, use'd, and require'd in the specified namespace.

ex:

(require '[some.ns.foo :as foo] :reload-all)

[Peter Schuller]

> I haven't tested what happens to the deftest stuff, but the „moving function“ could be addressed.

Cool. I had not yet run into the moving function case with clojure (I
did several times with Common Lisp), so that might be a non-issue. I
was just presuming that no magic was going on.

--
/ Peter Schuller

[Meikel Brandmeyer]

Hi,

It turns out you shouldn't also listen to me. I think I will stop writing emails for today. I got caught by the last-var-wins feature.

Clojure 1.2.0-master-SNAPSHOT
user=> (require 'foo.bar)
nil
user=> (foo.bar/baz)
5
; Remove function baz here from file
user=> (require :reload 'foo.bar)
nil
user=> (foo.bar/baz)
5

In my stupidity I chose prn as function name, which removed from the namespace on reload, because clojure.core/prn „overwrites“ the Var again. Maybe such a functionality could be also implemented for normal reload? But then what happens to other namespaces which refer to this Var? I admit, this is not a trivial question.

Sincerely
Meikel

[Laurent PETIT]

You got me thinking that all the stuff I and you were talking about concerning namespace dependency graph, etc. for only reloading the appropriate namespaces should be possible to be run from the running REPL, and thus provided as a regular tool to be available to all environments: REPL, any IDE / back-end server etc.

In its "pure" form it could recompute the dependency graph of the namespaces every time "on-the-fly".

With this function, even from the REPL could you be sure that you have all the functions depending on the macros of some modified namespace up-to-date.
It is even possible to solve the "functions should be removed if no more in the namespace" issue, by adding temporary watchers to the namespace vars, and then removing all the vars whose watchers were not triggered and are not defonce vars (this defonce issue being the last problem I currently see: how to know that defonce vars have been removed ? The only solution I temporarily envision is to rebind the defonce macro during the reload with a decorator which will store information about defonced vars).

Of course, some corner cases will still exist: direct access to vars in other namespaces via other-ns/the-var while not explicitly requiring/using other-ns (maybe "oddly" relying on other-ns to be loaded preliminary, by other means).

2010/7/19 Meikel Brandmeyer <m...@kotka.de>

[Mark Rathwell]

>>In my stupidity I chose prn as function name, which removed from the namespace on >>reload, because clojure.core/prn „overwrites“ the Var again. Maybe such a functionality >>could be also implemented for normal reload? But then what happens to other >>namespaces which refer to this Var? I admit, this is not a trivial question.

I don't think anything gets removed (unloaded), at least that's the way I remember it.

Things get a little tricky with :reload-all.  For example (I don't have a repl handy to verify, but this is how I remember it):

---------

original files:

---------

(ns some.ns.foo1

  "namespace foo 1"

  (:gen-class)

  (:require [some.ns.foo2 :as foo2]))

(defn func-a

  []

  (foo2/func-x "abc"))

(defn func-b

  []

  (str "123"))

(ns some.ns.foo2

  "namespace foo 2"

  (:gen-class))

(defn func-x

  [z]

  (str z))

(defn func-y

  []

  (str "456"))

---------

save and go to REPL:

---------

user=> (require '[some.ns.foo1 :as foo1])

user=> (require '[some.ns.foo2 :as foo2])

user=> (foo1/func-a)

"abc"

user=> (foo1/func-b)

"123"

user=> (foo2/func-x "def")

"def"

user=> (foo2/func-y)

"456"

---------

change files:

---------

(ns some.ns.foo1

  "namespace foo 1"

  (:gen-class)

  (:require [some.ns.foo2 :as foo2]))

(defn func-a

  []

  (foo2/func-x "abc"))

(ns some.ns.foo2

  "namespace foo 2"

  (:gen-class))

(defn func-x

  [z]

  (str z "new"))

(defn func-y

  []

  (str "456" "new"))

---------

save and go back to REPL:

---------

user=> (require '[some.ns.foo1 :as foo1] :reload-all)

user=> (foo1/func-a)

"abcnew"

user=> (foo1/func-b)

"123"

user=> (foo2/func-x "def")

"def"

user=> (foo2/func-y)

"456"

Notice, the removed function in foo1 still exists, because it was not overwritten by anything.  Also, notice that we specified :reload-all for foo1, which then reloads foo2 in the context of foo1, so when calling a function in foo1 that calls foo2, we get the updated output.  But, when calling foo2 functions directly, we get the old functions.

This could all be BS, so verify before taking my word for it, as stated I don't have a repl to verify, but from memory it seems that this is the behavior.

 - Mark

[Laurent PETIT]

2010/7/19 Peter Schuller <peter.s...@infidyne.com>

Yes, that's currently the most important point I'd missed so far. So I think IDEs may not simply have to provide "white/black" behaviour concerning dynamic feature, but also a middle path (though I guess by reading the comments that I may still make the default behaviour be "automatically reload", and for people wanting the maximum safety, "don't automatically reload").

Possibly easy access to a fresh re-spawn:ed REPL with command line
history retained might make a fresh normalized REPL easily accessible
in a way that doesn't involve re-typing lots of stuff (while clearly
not preserving actual state other than history)?

--
/ Peter Schuller

[Meikel Brandmeyer]

Hi,

Am 19.07.2010 um 23:01 schrieb Mark Rathwell:

> Notice, the removed function in foo1 still exists, because it was not overwritten by anything.

This is true if the function name is not used in anything the namespace refers to. Take eg. my first example with prn. There the reload overwrites the previous Var (ie. the Var is removed from the namespace) by the clojure.core/prn function. If the function was still there it would again overwrite the c.c/prn. This is the new last-var-wins behaviour. (which I generally like, but which also didn't bite me up to now besides this example)

> Also, notice that we specified :reload-all for foo1, which then reloads foo2 in the context of foo1, so when calling a function in foo1 that calls foo2, we get the updated output. But, when calling foo2 functions directly, we get the old functions.

No. This is wrong. There is only one some.ns.foo2. It doesn't matter where things are loaded from.

Sincerely
Meikel

[Meikel Brandmeyer]

Laurent,

Am 19.07.2010 um 23:16 schrieb Laurent PETIT:

> Well, by activating the "automatic builder" feature of an IDE, I /suppose/ users typically think this is a way to tell the IDE to do automatic rebuilds. "rebuild" here meaning AOT compilation of the strict necessary files, and reload of what has been changed :-)

Yes. That's fine. „Every automatic system is only as good as you can turn it off.“ ;)

Meikel

[Mark Rathwell]

>>  Also, notice that we specified :reload-all for foo1, which then reloads foo2 

>> in the context of foo1, so when calling a function in foo1 that calls foo2, 

>> we get the updated output.  But, when calling foo2 functions directly, we get the old functions.

>No. This is wrong. There is only one some.ns.foo2. It doesn't matter where things are loaded from.

You are correct, my mistake.  For some reason I had been operating under this assumption.

Sincerely
Meikel

[j-g-faustus]

I haven't tried CCW, but I am happy with Enclojure 6.8. (In 6.9 the
file navigator which lists the functions and vars in the file doesn't
work yet, and the navigator is perhaps 50% of the value I derive from
an IDE.)

> Am 19.07.2010 um 20:50 schrieb Laurent PETIT:
> > If you work from the files, then you simply do not save the files until you're "happy" with the codebase.

I wouldn't want to have the REPL autoload anything just because I save
a file, simply because most of the files I am editing are in a broken
state most of the time, and could stay that way for days.

Saving is something I do out of habit, because I'm going out for
lunch, because I want access to the files from another tool (backup,
version control, move to different editor or computer, whatever).
I'm thinking of "save" as "save the text in this file" rather than
"recompile and reload everything", I don't want "save" to give me
exception stacktraces and a boatload of warnings.

I once used a Java IDE that tried to recompile the project every time
I saved. I never got used to that either and found it a constant
nuisance. And it's even worse in a REPL-based language where a reload
may mess up carefully arranged global state.


I'm working similarly to Meikel, using the REPL for testing of simple
draft/mock functions and using "evaulate expression" at least as often
as "load file" - the file as a whole is broken and won't load, but I
just finished this particular function and want to test it.

I normally start out with a sketch or skeleton of the whole project,
with multiple files and lots of partial drafts, and fill out the
details over time (days or weeks).
Being able to finish and test functions one by one is one of the great
benefits of REPL-based development IMHO. I wouldn't want anything that
forces me to have every file in a loadable state all the time.

Perhaps relatedly, someone doing a usability study of how newspapers
journalists work told me that journalists (at least the ones she were
studying) didn't write articles by starting at the beginning and
finishing at the end.
Instead they wrote down snippets, sentence fragments, nifty turns of
phrase etc. in whatever order they happened to think of them, spread
the fragments out on the screen and copy/pasted the parts together to
assemble a complete article.
According to her this was a surprisingly efficient way of working,
they could finish an article in no time compared to writing it in a
more traditional start-to-finish fashion. Perhaps because the time you
need to think of what to say about X can be spent to say something
about Y.

This is somewhat similar to my preferred way of working with code, I
start out with a jumble of fragments and assemble them into complete
code over time.
The Java development style where you need the whole file (or even
worse, the whole project) to compile in order to test a single method
is a lot slower, at least for me.

> a. So having an "IDE-dedicated live server REPL" seems like I
> could relieve the user from explicitly launching a REPL (it's weird
> and counter-intuitive for a user to have to somehow "manually"
> trigger the IDE user assistance !).

I can't tell. I have the REPL running 100% of the time I am using a
Clojure IDE, so this is an issue I never encounter. But if you need a
"server REPL" for internal reasons that's fine by me, as long as it
doesn't mess with the REPL I am using.

> I may still make the default behaviour be "automatically
> reload", and for people wanting the maximum safety, "don't automatically
> reload").

I'm perfectly fine with that :)


For what I would like:

* I would be happy to have a one-click option (button or keyboard
shortcut) to reload everything that is currently loaded.

* If there is some way to undefine vars that were loaded from file
when the vars are deleted from the file and the file is reloaded, I
think I would like that too. It would save many a REPL restart to
check that newly refactored code doesn't depend on something that was
deleted. (Although it might come back to bite me if I have overridden
the file definition in the REPL and a reload removes both of them, I
expect I would eventually learn not to avoid that.)
Or perhaps a one-click manual "restart REPL and reload current files",
somewhat analogous to a Java "clean and build".


I should mention that my working habits were formed with Emacs and
Allegro Common Lisp and a decade with various Java IDEs.
We adjust to the tools we are using, at least to some extent, and
people coming to Clojure from elsewhere may have different
expectations.

If we want to move the state of Clojure IDEs forward, we may have to
break some expectations for people like me, and "but that's how we did
it in X twenty years ago" is not a sufficient reason to keep something
in and of itself.

I don't think it is possible to please everyone simultaneously, so if
you think you have a better idea and can find at least a few people
who prefer it your way, I'd say go for it :)

Thanks for raising the question, it will be interesting to see what
comes out of it.


jf

[Lee Spector]

I'm using Eclipse/Counterclockwise these days but one of the first things I did was to figure out how to turn off the automatic build-on-save behavior. I save all of the time -- usually every "complete thought" which may be as little as a symbol or a line or a small expression -- and when I do that it's because I want to save (so that my changes won't be lost if my machine crashes, etc.), not because I want to trigger any other behaviors. The vast majority of my saves occur in states in which my code is incomplete and probably even nonsensical. The idea of not saving until everything is perfect doesn't make any kind of reasonable sense to me.

I would certainly hope that it remains possible to disable the automatic building, but I would also argue that it'd be most sensible for it to be off by default. When someone wants to build isn't it reasonable for them to say so? They could do that with another single keystroke or click -- just not the same one used for saving. I understand that auto-completion and documentation and other features depend on the most recent build, but it seems natural enough to me to require a rebuild if I want that information to be updated. Tying saving and building together is cute, but it doesn't seem right to me that it's the default (and I'd probably always want it off).

-Lee

On Jul 19, 2010, at 2:50 PM, Laurent PETIT wrote:

> That's interesting ! Thanks for the feedback.
> Please note that it's not "impolite" in the sense that there's a policy: the project is reloaded in the REPL when the file is saved - that's a kind of "invitation" made by the user :-).
> If you don't work from the files, but from the REPL, nothing will happen automatically.
> If you work from the files, then you simply do not save the files until you're "happy" with the codebase. You can make several roundtrips between the file's content and the REPL (even via keyboard shortcuts) without saving the files. Ultimately, when you want to quit, you have to quit the REPL anyway, and you have the option to save the files or not.
>
> Does this answer make sense ?
>
> There's also the problem of "I save file A", and all project namespaces are reloaded as well. By maintaining a dependency graph of the namespaces relationships (easily obtained dynamically via the REPL), I may be able to only reload the expected namespaces. Some may see this as a feature - ensuring that all functions which depend on namespace A macros are recompiled -, I can see that some may see this as a recipe for disaster if reloading the namespaces also reloads global-var-as-data contents. Proper use of defonce may help there ?
>
> Anyway, I didn't expect you reaction to be soo opinionated, and it's refreshing to hear others thoughts.
>
> Let's see how others react to what you wrote !
>
> Cheers,

--
Lee Spector, Professor of Computer Science
School of Cognitive Science, Hampshire College
893 West Street, Amherst, MA 01002-3359
lspe...@hampshire.edu, http://hampshire.edu/lspector/
Phone: 413-559-5352, Fax: 413-559-5438

Check out Genetic Programming and Evolvable Machines:
http://www.springer.com/10710 - http://gpemjournal.blogspot.com/

[Lee Spector]

On Jul 19, 2010, at 5:22 PM, Laurent PETIT wrote:
> Some people have reported that they launch several REPLs when the first one is busy for a long time. What they really do, though, is that they launch a new java environment with the project files and dependencies. When we have a proper REPL client in ccw, we'll also have a way to launch several concurrent REPLs for the same java environment. This will be *much* better.

I'm in this camp. I often have a long process running in the REPL and want to try out other things at the same time. In fact, I sometimes use the REPL just for the sake of calling doc or find-doc, and I may want to do that halfway through editing an expression in a REPL... so I need another REPL for that. When I've wanted to do this kind of thing in my recent CCW work I've actually run the second REPL in a completely different way, from a clj script in a Terminal window. Obviously silly, but I'm still wrapping my head around the way things work in CCW and Clojure more generally. In the Lisp environments I've used it was generally easy to open a second REPL ("Listener") that acted just like the first (with access to the same state, etc.). I've never known or had to care about how this was being done internally.

-Lee

[Lee Spector]

On sketching-in-nonlinear-order, that's definitely how I write too, both code and prose (and email!). This has implications not only for the automatic build behavior we were discussing but also for the structure-based editing discussion from a week or two ago. If you piece together your code in a nonlinear way then the assumptions that structure-based editors sometimes make -- e.g. that whenever you type "(" you want an immediately following ")" -- will usually be wrong and it will be a nuisance to undo all of the system's "helpfulness."

-Lee

--

[Rick Moynihan]

On 19 July 2010 22:25, Laurent PETIT <lauren...@gmail.com> wrote:
>
> 2010/7/19 Peter Schuller <peter.s...@infidyne.com>

>> I also sympathize and agree that it can be hugely problematic if an
>> environment stomps on your carefully prepared REPL in ways you don't
>> expect. I'm not sure how to best reconcile these two concerns.
>
> Yes, that's currently the most important point I'd missed so far. So I think
> IDEs may not simply have to provide "white/black" behaviour concerning
> dynamic feature, but also a middle path (though I guess by reading the
> comments that I may still make the default behaviour be "automatically
> reload", and for people wanting the maximum safety, "don't automatically
> reload").

The middle way, seems like a nice option. How about, having save as
just save, providing a save-all-and-reload-all option; and using
editor overlays to indicate un-evaluated sexpressions (that haven't
yet been sent to the repl). In light of this, you might also be able
to show vars that exist in the current repl environment, that don't
yet exist in a file (though this is arguably more complex).

Just my 2 pence worth; but I agree with others that save should be
just save, and evaluate should be separate... I come from the
SLIME/Emacs camp, and yes desynchronisation is sometimes a problem,
but its pretty easy to clean the environment and evaluate everything
again.

R.

[Chas Emerick]

Saving a file is not an invitation to do anything with the REPLs that I started -- it's an invitation to save the file! :-D

And no, I write a lot of code in the REPL at all.  I *load* a lot of code into REPLs, yes, multiple active REPLs, all the time.  Actually writing code in what is almost always a 2nd-class environment (compared to a proper editor) doesn't make any sense to me.

I see three issues here:

(a) Should ccw provide the richest possible editing experience, e.g. code completion, symbol navigation, etc, etc?

(b) Should the implementation of (a) automatically and silently modify the state of a running program initiated by the user?

(c) To what extent should running REPLs be "synchronized" with the codebase in question?

I'd answer an emphatic "yes", "no", and "probably not much", respectively.

What's odd to me about this "debate" is that issue (c) would appear to be a very difficult problem to solve in general for very little gain.  Certainly at the edges, people are going to have unusual and/or long-running build processes that either the tool will not know how to invoke "properly" (i.e. as they would be in the normal course of development), or that will be simply too long-running to fit within an interactive setting.  This is to say nothing of automatically ns-unmap'ing deleted vars, dealing with files that are in an interim state (which, as many have mentioned, is the case 98% of the time), and other stuff that I'm surely not thinking of.

As far as I can tell, the advantage of all the work associated with tackling (c) is....to help people avoid understanding how to work with REPLs (perhaps to somehow nudge the environment as a whole a hair closer to an image-based environment?...though that's speculation on my part).  Knowing how to work with REPLs, and understanding the relationship between them and source files and (if one uses AOT) classfiles is paramount to being able to use Clojure effectively IMO.  If anyone were to get the impression that REPLs are really just an editor feature, are managed automatically, and are not a natural outcropping of Clojure being a lisp, they'd be at a disadvantage.

Totally FWIW, I think enclojure's REPL support is stellar (some are probably tired of hearing me say that).  I think ccw (or any other "integrated environment") would do well to ape it as much as possible (something I aim to help with, but I'm underwater at the moment).

- Chas

On Jul 19, 2010, at 2:50 PM, Laurent PETIT wrote:

[Chas Emerick]

Forgot one thing:

On Jul 19, 2010, at 2:50 PM, Laurent PETIT wrote:

Anyway, I didn't expect you reaction to be soo opinionated, and it's refreshing to hear others thoughts.

Strong opinions, weakly held, 24/7. ;-)

- Chas

[Meikel Brandmeyer]

Hi,

On Jul 19, 11:09 pm, Laurent PETIT <laurent.pe...@gmail.com> wrote:

> You got me thinking that all the stuff I and you were talking about
> concerning namespace dependency graph, etc. for only reloading the
> appropriate namespaces should be possible to be run from the running REPL,
> and thus provided as a regular tool to be available to all environments:
> REPL, any IDE / back-end server etc.
>
> In its "pure" form it could recompute the dependency graph of the namespaces
> every time "on-the-fly".
>
> With this function, even from the REPL could you be sure that you have all
> the functions depending on the macros of some modified namespace up-to-date.
> It is even possible to solve the "functions should be removed if no more in
> the namespace" issue, by adding temporary watchers to the namespace vars,
> and then removing all the vars whose watchers were not triggered and are not
> defonce vars (this defonce issue being the last problem I currently see: how
> to know that defonce vars have been removed ? The only solution I
> temporarily envision is to rebind the defonce macro during the reload with a
> decorator which will store information about defonced vars).
>
> Of course, some corner cases will still exist: direct access to vars in
> other namespaces via other-ns/the-var while not explicitly requiring/using
> other-ns (maybe "oddly" relying on other-ns to be loaded preliminary, by
> other means).

I had hopes that this discussion
http://groups.google.com/group/clojure-dev/browse_thread/thread/b9d15561e8a3b5aa/0541da4e70ad682d
triggers something, but Stephen obviously didn't have the time to work
further on this topic.

Sincerely
Meikel

[Laurent PETIT]

Hi !

First of all, thanks to you (and to others as well, I will not repeat the many thanks I would like to write everywhere in the thread !) for the detailed answer.

Definitely food for thought !

2010/7/20 j-g-faustus <johannes...@gmail.com>

I haven't tried CCW, but I am happy with Enclojure 6.8. (In 6.9 the
file navigator which lists the functions and vars in the file doesn't
work yet, and the navigator is perhaps 50% of the value I derive from
an IDE.)

> Am 19.07.2010 um 20:50 schrieb Laurent PETIT:

> > If you work from the files, then you simply do not save the files until you're "happy" with the codebase.

I wouldn't want to have the REPL autoload anything just because I save
a file, simply because most of the files I am editing are in a broken
state most of the time, and could stay that way for days.

Wow, if you can work efficiently by having broken files saved for several days in your code base, and effectively work out the situation, you're way smarter than me !

I generally try to keep my files in a good shape at the end of a coding session, because my RAM is limited.

This generally invokes either placing the wrong things within single line comments, or the (comment ) form, or (sometimes) the new definition immediately followed by the old definition (so that reloading the ns eventually loads the old definition).

And generally, I also write a comment to remind me of what I was trying to achieve (ok, not always doing this one, and always regretting so).

Now, if the "automatic reload" tries to reload a broken namespace, the reload will stop at the first compilation error. You'll have an error marker in the source code to indicate where things start to go wrong.

I simply cannot (yet) understand how "bad" it is for the REPL. I certainly need more "use case" examples to really grasp the annoyance pointed out by you, Chas, etc. Having this use cases in mind will help me build a better IDE, so please don't hesitate !

 

Saving is something I do out of habit, because I'm going out for
lunch, because I want access to the files from another tool (backup,
version control, move to different editor or computer, whatever).
I'm thinking of "save" as "save the text in this file" rather than
"recompile and reload everything", I don't want "save" to give me
exception stacktraces and a boatload of warnings.

I once used a Java IDE that tried to recompile the project every time
I saved. I never got used to that either and found it a constant
nuisance. And it's even worse in a REPL-based language where a reload
may mess up carefully arranged global state.


I'm working similarly to Meikel, using the REPL for testing of simple
draft/mock functions and using "evaulate expression" at least as often
as "load file" - the file as a whole is broken and won't load, but I
just finished this particular function and want to test it.

I normally start out with a sketch or skeleton of the whole project,
with multiple files and lots of partial drafts, and fill out the
details over time (days or weeks).
Being able to finish and test functions one by one is one of the great
benefits of REPL-based development IMHO. I wouldn't want anything that
forces me to have every file in a loadable state all the time.

I don't think you deliberately leave your namespaces in non loadable states. Do you ?

 

Oh, maybe I understand: by "partial drafts", you mean "pseudo-code", directly written in clojure, is it this ?

If there was a shortcut key to quickly wrap/unwrap a top level defn with (comment), and if the keyboard shortcut for "evaluate top level form in the REPL" would be smart enough to detect top level comment and send to the REPL the unwrapped form, would it help you ?

That way, your files would visually show the "pseudo-code" parts (those within comments), and your namespaces will remain in a loadable shape, eventually relieving you from manually load your project "piece by piece" every time you must restart a REPL ?

Perhaps relatedly, someone doing a usability study of how newspapers
journalists work told me that journalists (at least the ones she were
studying) didn't write articles by starting at the beginning and
finishing at the end.
Instead they wrote down snippets, sentence fragments, nifty turns of
phrase etc. in whatever order they happened to think of them, spread
the fragments out on the screen and copy/pasted the parts together to
assemble a complete article.
According to her this was a surprisingly efficient way of working,
they could finish an article in no time compared to writing it in a
more traditional start-to-finish fashion. Perhaps because the time you
need to think of what to say about X can be spent to say something
about Y.

This is somewhat similar to my preferred way of working with code, I
start out with a jumble of fragments and assemble them into complete
code over time.
The Java development style where you need the whole file (or even
worse, the whole project) to compile in order to test a single method
is a lot slower, at least for me.

I understand what you're saying, but I'm wary of pushing metaphors too far. I don't exactly see where the REPL sits in your's :-)

 

>     a. So having an "IDE-dedicated live server REPL" seems like I
> could relieve the user from explicitly launching a REPL (it's weird
> and counter-intuitive for a user to have to somehow "manually"
> trigger the IDE user assistance !).

I can't tell. I have the REPL running 100% of the time I am using a
Clojure IDE, so this is an issue I never encounter. But if you need a
"server REPL" for internal reasons that's fine by me, as long as it
doesn't mess with the REPL I am using.

> I may still make the default behaviour be "automatically
> reload", and for people wanting the maximum safety, "don't automatically
> reload").

I'm perfectly fine with that :)


For what I would like:

* I would be happy to have a one-click option (button or keyboard
shortcut) to reload everything that is currently loaded.

That's definitely something I'd like to provide as a user convenience. One question : when you write "reload everything that is currently loaded", what are precisely your expectations ? by "that is currently loaded", do you expect a "per var" granularity level, or is a "per namespace" granularity level good enough ? (because I see how to do the "per namespace" level, but not the "per var" - without having to patch clojure for enhancing the 'def special form - and I do not want to do that)

 

* If there is some way to undefine vars that were loaded from file
when the vars are deleted from the file and the file is reloaded, I
think I would like that too. It would save many a REPL restart to
check that newly refactored code doesn't depend on something that was
deleted. (Although it might come back to bite me if I have overridden
the file definition in the REPL and a reload removes both of them, I
expect I would eventually learn not to avoid that.)
Or perhaps a one-click manual "restart REPL and reload current files",
somewhat analogous to a Java "clean and build".


I should mention that my working habits were formed with Emacs and
Allegro Common Lisp and a decade with various Java IDEs.
We adjust to the tools we are using, at least to some extent, and
people coming to Clojure from elsewhere may have different
expectations.

If we want to move the state of Clojure IDEs forward, we may have to
break some expectations for people like me, and "but that's how we did
it in X twenty years ago" is not a sufficient reason to keep something
in and of itself.

I don't think it is possible to please everyone simultaneously, so if
you think you have a better idea and can find at least a few people
who prefer it your way, I'd say go for it :)

Thanks for raising the question, it will be interesting to see what
comes out of it.

Indeed, that's why I'm making the above suggestions to "challenge" your way of working :-)

[Laurent PETIT]

Hi Lee!

2010/7/20 Lee Spector <lspe...@hampshire.edu>

I'm using Eclipse/Counterclockwise these days but one of the first things I did was to figure out how to turn off the automatic build-on-save behavior. I save all of the time -- usually every "complete thought" which may be as little as a symbol or a line or a small expression -- and when I do that it's because I want to save (so that my changes won't be lost if my machine crashes, etc.), not because I want to trigger any other behaviors. The vast majority of my saves occur in states in which my code is incomplete and probably even nonsensical. The idea of not saving until everything is perfect doesn't make any kind of reasonable sense to me.

Lee, did you remove the automatic build-on-save behavior because you "feared" that it would give annoyances to you, or because you encountered annoyances and wanted to get rid of them ? If the latter, could you please expand on what the annoyances were for you ? (maybe your annoyance inherently imply to trigger the behavior of, maybe there are alternative solutions to be found to get the best of both worlds).

[Laurent PETIT]

2010/7/20 Lee Spector <lspe...@hampshire.edu>

On sketching-in-nonlinear-order, that's definitely how I write too, both code and prose

 

(and email!).

This explains a lot of things :-p (just kidding)

 

--

[Laurent PETIT]

2010/7/20 Rick Moynihan <rick.m...@gmail.com>

On 19 July 2010 22:25, Laurent PETIT <lauren...@gmail.com> wrote:
>
> 2010/7/19 Peter Schuller <peter.s...@infidyne.com>

>> I also sympathize and agree that it can be hugely problematic if an
>> environment stomps on your carefully prepared REPL in ways you don't
>> expect. I'm not sure how to best reconcile these two concerns.
>
> Yes, that's currently the most important point I'd missed so far. So I think
> IDEs may not simply have to provide "white/black" behaviour concerning
> dynamic feature, but also a middle path (though I guess by reading the
> comments that I may still make the default behaviour be "automatically
> reload", and for people wanting the maximum safety, "don't automatically
> reload").

The middle way, seems like a nice option.  How about, having save as
just save, providing a save-all-and-reload-all option; and using
editor overlays to indicate un-evaluated sexpressions (that haven't
yet been sent to the repl).  In light of this, you might also be able
to show vars that exist in the current repl environment, that don't
yet exist in a file (though this is arguably more complex).

Yes, I'm starting to thing about this as the final target. Not surprisingly, it's not the easiest one to reach :-)

[Laurent PETIT]

2010/7/20 Chas Emerick <ceme...@snowtide.com>

Saving a file is not an invitation to do anything with the REPLs that I started -- it's an invitation to save the file! :-D

And no, I write a lot of code in the REPL at all.  I *load* a lot of code into REPLs, yes, multiple active REPLs, all the time.  Actually writing code in what is almost always a 2nd-class environment (compared to a proper editor) doesn't make any sense to me.

I see three issues here:

(a) Should ccw provide the richest possible editing experience, e.g. code completion, symbol navigation, etc, etc?

(b) Should the implementation of (a) automatically and silently modify the state of a running program initiated by the user?

You're right. When ccw will be remotely connected to web server instances, the current behaviour will be a recipe for disaster. The "middle way" is definitely the final target, but providing good defaults for the "smart-reload-on-save" should be studied. 

Here are my thoughts for the defaults:

  a. jvm loaded by ccw: 99% of the times it's in "dev mode" => smart-reload-on-save true by default

  b. connection to a remote REPL: 99% it's for "touching" more "sensible" environments (test serveur, pre-production server) => smart-reload-on-save false by default

 

(c) To what extent should running REPLs be "synchronized" with the codebase in question?

Well, the IDE should assist the user, right ? :-) So the value of c) must be compared relatively to the values of other potential enhancements, not thougt about in isolation.

 

I'd answer an emphatic "yes", "no", and "probably not much", respectively.

What's odd to me about this "debate" is that issue (c) would appear to be a very difficult problem to solve in general for very little gain.  Certainly at the edges, people are going to have unusual and/or long-running build processes that either the tool will not know how to invoke "properly" (i.e. as they would be in the normal course of development), or that will be simply too long-running to fit within an interactive setting.  This is to say nothing of automatically ns-unmap'ing deleted vars, dealing with files that are in an interim state (which, as many have mentioned, is the case 98% of the time), and other stuff that I'm surely not thinking of.

As far as I can tell, the advantage of all the work associated with tackling (c) is....to help people avoid understanding how to work with REPLs (perhaps to somehow nudge the environment as a whole a hair closer to an image-based environment?...though that's speculation on my part).  Knowing how to work with REPLs, and understanding the relationship between them and source files and (if one uses AOT) classfiles is paramount to being able to use Clojure effectively IMO.  If anyone were to get the impression that REPLs are really just an editor feature, are managed automatically, and are not a natural outcropping of Clojure being a lisp, they'd be at a disadvantage.

Totally FWIW, I think enclojure's REPL support is stellar (some are probably tired of hearing me say that).  I think ccw (or any other "integrated environment") would do well to ape it as much as possible (something I aim to help with, but I'm underwater at the moment).

Chas, that's not a problem if you cannot work on this as you offered to do. I just wish you had told me that sooner, because that's an area on which I also can (and would like to) work, but I had stopped touching it since you volunteered (my work on paredit enhancements is slow currently, so I want to start something else in parallel).

[Chas Emerick]

On Jul 20, 2010, at 4:29 AM, Laurent PETIT wrote:

> You're right. When ccw will be remotely connected to web server
> instances, the current behaviour will be a recipe for disaster. The
> "middle way" is definitely the final target, but providing good
> defaults for the "smart-reload-on-save" should be studied.
> Here are my thoughts for the defaults:
> a. jvm loaded by ccw: 99% of the times it's in "dev mode" => smart-
> reload-on-save true by default
> b. connection to a remote REPL: 99% it's for "touching" more
> "sensible" environments (test serveur, pre-production server) =>
> smart-reload-on-save false by default

I'll have to strenuously disagree again. Defaults matter, insofar as
they recommend preferred behaviour. By all means, use a background
REPL to support editor features, but don't step on users' REPLs --
doing so isn't actually doing them any favors. I have to repeat this
for emphasis:

Knowing how to work with REPLs, and understanding the relationship
between them and source files and (if one uses AOT) classfiles is
paramount to being able to use Clojure effectively IMO. If anyone
were to get the impression that REPLs are really just an editor
feature, are managed automatically, and are not a natural outcropping
of Clojure being a lisp, they'd be at a disadvantage.

I'm curious: are there any other lisp environments where reload-on-
save is the default?

> Totally FWIW, I think enclojure's REPL support is stellar (some are
> probably tired of hearing me say that). I think ccw (or any other
> "integrated environment") would do well to ape it as much as
> possible (something I aim to help with, but I'm underwater at the
> moment).
>
> Chas, that's not a problem if you cannot work on this as you offered
> to do. I just wish you had told me that sooner, because that's an
> area on which I also can (and would like to) work, but I had stopped
> touching it since you volunteered (my work on paredit enhancements
> is slow currently, so I want to start something else in parallel).

It's not that I cannot work on it, just delayed from doing so. I'm
sorry I didn't say so last week, but I didn't think it was relevant --
I don't think I've ever worked on an open source project where there
was such a schedule! ;-)

Cheers,

- Chas

[Laurent PETIT]

2010/7/20 Chas Emerick <ceme...@snowtide.com>

On Jul 20, 2010, at 4:29 AM, Laurent PETIT wrote:

You're right. When ccw will be remotely connected to web server instances, the current behaviour will be a recipe for disaster. The "middle way" is definitely the final target, but providing good defaults for the "smart-reload-on-save" should be studied.
Here are my thoughts for the defaults:

 a. jvm loaded by ccw: 99% of the times it's in "dev mode" => smart-reload-on-save true by default

 b. connection to a remote REPL: 99% it's for "touching" more "sensible" environments (test serveur, pre-production server) => smart-reload-on-save false by default

I'll have to strenuously disagree again.  Defaults matter, insofar as they recommend preferred behaviour.  

 

By all means, use a background REPL to support editor features,

So e.g. code completion while in the file editor works on a set of possible completions, and code completion while in the REPL works on another set of possible completions ? e.g. if I dynamically define a new  var in the REPL, it's not suggested in the editor. If someone asks the editor for "macro-expansion", it will macroexpand with the definition found in the "background-instance", not the one the user may just have modified/corrected in the REPL but not yet re-saved in the file ...

Hmmm ....

 

but don't step on users' REPLs -- doing so isn't actually doing them any favors.  I have to repeat this for emphasis:

Knowing how to work with REPLs, and understanding the relationship between them and source files and (if one uses AOT) classfiles is paramount to being able to use Clojure effectively IMO.  If anyone were to get the impression that REPLs are really just an editor feature, are managed automatically, and are not a natural outcropping of Clojure being a lisp, they'd be at a disadvantage.

I'm curious: are there any other lisp environments where reload-on-save is the default?

Totally FWIW, I think enclojure's REPL support is stellar (some are probably tired of hearing me say that).  I think ccw (or any other "integrated environment") would do well to ape it as much as possible (something I aim to help with, but I'm underwater at the moment).

Chas, that's not a problem if you cannot work on this as you offered to do. I just wish you had told me that sooner, because that's an area on which I also can (and would like to) work, but I had stopped touching it since you volunteered (my work on paredit enhancements is slow currently, so I want to start something else in parallel).

It's not that I cannot work on it, just delayed from doing so.  I'm sorry I didn't say so last week, but I didn't think it was relevant -- I don't think I've ever worked on an open source project where there was such a schedule! ;-)

Cheers,

- Chas

[Lee Spector]

On Jul 20, 2010, at 4:15 AM, Laurent PETIT wrote:

> Oh, maybe I understand: by "partial drafts", you mean "pseudo-code", directly written in clojure, is it this ?
>
> If there was a shortcut key to quickly wrap/unwrap a top level defn with (comment), and if the keyboard shortcut for "evaluate top level form in the REPL" would be smart enough to detect top level comment and send to the REPL the unwrapped form, would it help you ?
> That way, your files would visually show the "pseudo-code" parts (those within comments), and your namespaces will remain in a loadable shape, eventually relieving you from manually load your project "piece by piece" every time you must restart a REPL ?

Laurent,

This strikes me as an instance (somewhat like structure-based editing :-0) in which you are being admirably clever, but in which simplicity and clarity and obviousness can be more helpful than even very clever cleverness.

I can't speak for Johannes but sometimes my broken stuff is pseudocode, sometimes it's just incomplete, sometimes it's English or abbreviations, etc. Sometimes I plan to come back to it in a minute before evaluating it (but I may want to save in the meantime anyway), while sometimes I may want to evaluate it even if I know it's broken because I want to see how the system thinks it's broken in particular.

All of this works fine as long as the system evaluates what I tell it to evaluate and leaves it alone otherwise. Automatic evaluation disrupts this and adding more complexity to locally disable the automatic evaluation in some cases, and then even more complexity to get the system TO evaluate that stuff anyway (even if it's marked as a comment!) when the user says to evaluate the file, is just making things that could be simple and clear and obvious really confusing.

-Lee

[Lee Spector]

On Jul 20, 2010, at 4:18 AM, Laurent PETIT wrote:
>
> Lee, did you remove the automatic build-on-save behavior because you "feared" that it would give annoyances to you, or because you encountered annoyances and wanted to get rid of them ? If the latter, could you please expand on what the annoyances were for you ? (maybe your annoyance inherently imply to trigger the behavior of, maybe there are alternative solutions to be found to get the best of both worlds).
>

It annoyed me inherently that behaviors were being automatically triggered. FWIW I sometimes include top-level calls in my files, not just definitions (maybe this is a Lisp-world practice that I should get over? It seems not to be done much in the Clojure world (?)), and I was seeing all of my top-level calls being run over and over again every time I saved a file. Definitely annoying, from my perspective.

-Lee

[Lee Spector]

On Jul 20, 2010, at 7:43 AM, Chas Emerick wrote:
>
> I'm curious: are there any other lisp environments where reload-on-save is the default?

I don't think I've ever seen one. I can't say I've tried them all, but I've worked in a bunch over several decades...

[Lee Spector]

On Jul 20, 2010, at 9:15 AM, Lee Spector wrote:
> On Jul 20, 2010, at 7:43 AM, Chas Emerick wrote:
>>
>> I'm curious: are there any other lisp environments where reload-on-save is the default?
>
> I don't think I've ever seen one. I can't say I've tried them all, but I've worked in a bunch over several decades...

Ooo -- maybe just thought of an exception, Interlisp-D, which I mentioned here previously. But it wasn't really "reload" because it wasn't file-based at all, as far as I recall (it has been a long time!). You had a window for each definition, and I guess when you saved a definition it would be "active"... But anyway, this was such a radically different framework overall that I don't think it's really relevant. In every file-based Lisp system I've used saving a file meant saving a file, and evaluating/reloading was a distinct process.

[Chas Emerick]

On Jul 20, 2010, at 9:14 AM, Lee Spector wrote:

> On Jul 20, 2010, at 4:18 AM, Laurent PETIT wrote:
>>
>> Lee, did you remove the automatic build-on-save behavior because
>> you "feared" that it would give annoyances to you, or because you
>> encountered annoyances and wanted to get rid of them ? If the
>> latter, could you please expand on what the annoyances were for
>> you ? (maybe your annoyance inherently imply to trigger the
>> behavior of, maybe there are alternative solutions to be found to
>> get the best of both worlds).
>>
>
> It annoyed me inherently that behaviors were being automatically
> triggered. FWIW I sometimes include top-level calls in my files, not
> just definitions (maybe this is a Lisp-world practice that I should
> get over? It seems not to be done much in the Clojure world (?)),
> and I was seeing all of my top-level calls being run over and over
> again every time I saved a file. Definitely annoying, from my
> perspective.

Top-level calls are definitely frowned upon (and almost never
necessary). Almost by definition, they're side-effecting, and you're
not always in control of when code is loaded (e.g. if someone is using
your code as a library, top-level calls are decidedly irritating).

- Chas

[Chas Emerick]

On Jul 20, 2010, at 8:34 AM, Laurent PETIT wrote:

> So e.g. code completion while in the file editor works on a set of
> possible completions, and code completion while in the REPL works on
> another set of possible completions ? e.g. if I dynamically define a
> new var in the REPL, it's not suggested in the editor. If someone
> asks the editor for "macro-expansion", it will macroexpand with the
> definition found in the "background-instance", not the one the user
> may just have modified/corrected in the REPL but not yet re-saved in
> the file ...

Yes, that's just about right. Files on disk are due their own
treatment, separate from user REPLs. By all means, a "Reload all in
current REPL" command would be handy to bring a REPL "up to date".

- Chas

[Laurent PETIT]

2010/7/20 Chas Emerick <ceme...@snowtide.com>

On Jul 20, 2010, at 8:34 AM, Laurent PETIT wrote:

So e.g. code completion while in the file editor works on a set of possible completions, and code completion while in the REPL works on another set of possible completions ? e.g. if I dynamically define a new  var in the REPL, it's not suggested in the editor. If someone asks the editor for "macro-expansion", it will macroexpand with the definition found in the "background-instance", not the one the user may just have modified/corrected in the REPL but not yet re-saved in the file ...

Yes, that's just about right.  Files on disk are due their own treatment, separate from user REPLs.  

Yes. So this would be a "simple to understand" behavior. What still bothers me in the end, is that's is the kind of "simple to understand behavior" that'll bit anybody from time to time ( after having spent 15 minutes understanding why this doesn't work anymore: "OH YES, I have forgotten to synchronize the REPL ...")

Easy to understand, easy to be bitten by ...

... anyway, since there's a strong point from many different voices for making this configurable, configurable it will be :-)

Unless something really knew is taken on the table by somebody else, I'll consider that we've achieved a round on this topic !
 

By all means, a "Reload all in current REPL" command would be handy to bring a REPL "up to date".

Yeah. But again, please (I've asked this before, nobody answered), what do you precisely mean by "reload all in current REPL": a reload at the namespace level ? Or really *only* reloading the vars you carefully and manually sent to the REPL over time ?

[Nicolas Oury]

On Tue, Jul 20, 2010 at 8:02 PM, Laurent PETIT <lauren...@gmail.com> wrote:
>
> Yes. So this would be a "simple to understand" behavior. What still bothers
> me in the end, is that's is the kind of "simple to understand behavior"
> that'll bit anybody from time to time ( after having spent 15 minutes
> understanding why this doesn't work anymore: "OH YES, I have forgotten to
> synchronize the REPL ...")
>

It happens to me multiple times every day. Not 15 minutes but like 1-2
minutes each time.
If I can be sure that what I evaluate in the REPL has the latest
version of all my files could make me switch
from emacs+swank to ccw.
Another thing that happens sometimes is:
- I refactor (for example function foo become function bar)
- I revaluate the files that I change
- I test a few functions and everything works fine
- I leave swank relaunch and it breaks (of course foo was still loaded)

A feature that allow to quit and reload quickly would be great.
Especially, if it comes with a configuration local to the project
telling what file to evaluate when the REPL start/restart and which
ns to be in.

Best,

Nicolas.

[Lee Spector]

On Jul 20, 2010, at 2:27 PM, Chas Emerick wrote:
> Top-level calls are definitely frowned upon (and almost never necessary). Almost by definition, they're side-effecting, and you're not always in control of when code is loaded (e.g. if someone is using your code as a library, top-level calls are decidedly irritating).
>

Okay. FWIW in my case I'm generally not providing a library and I generally AM in control of when code is loaded, as long as my IDE isn't doing it behind my back :-).

I guess in most cases I can get used to putting all of my top-level stuff in a concisely-named function that I later call, although my main working mode is to have a bunch of top-level calls in a short file that I re-load after reloading definitions in my other files... And it'd be a little bit of a pain to have to type calls rather than just hitting the reload key sequence... not a huge pain, but I'll probably try to keep doing this as long as my tools let me.

In any event, I find it hard to imagine any circumstance in which I'll want reloading to happen independent of my request.

[j-g-faustus]

On Jul 20, 9:02 pm, Laurent PETIT <laurent.pe...@gmail.com> wrote:
> Yeah. But again, please (I've asked this before, nobody answered), what do
> you precisely mean by "reload all in current REPL": a reload at the
> namespace level ? Or really *only* reloading the vars you carefully and
> manually sent to the REPL over time ?

I'm thinking in terms of files rather than namespaces or vars:
If I start a REPL and have loaded files A and B, which in turn may
have loaded other files and libs, I expect that the files will be
reloaded in the same way as if I selected files A and B in the editor
and did a "load file" on each of them and all their dependencies. Or
used (require... :reload-all) from the REPL.

It's just a convenience for when there are many files to load. And no,
I don't expect individual "evaluate expression" to be rerun.

And it would be nice with a button/shortcut that reloads the same
files, but restarts the REPL first.

> Indeed, that's why I'm making the above suggestions to "challenge" your way
> of working :-)

It's certainly interesting. I don't expect to have accidentally hit
upon the most efficient workflow possible in any conceivable world, so
there may very well be room for improvements.
I appreciate hearing what others are thinking around these issues, and
might even pick up a trick or two :)

After some more thinking, the autoload-on-save thing may not be so bad
after all, although it would require some adaptions on my end (like
storing expensive state in uniquely named REPL vars rather than file
vars so they don't get clobbered by the autoload). But it sounds like
the issue has been settled, so I'll leave it at that.


jf

[Jeff Rose]

On Jul 19, 8:19 pm, Meikel Brandmeyer <m...@kotka.de> wrote:

> Starting the server is up to the user. Rule 1: Vim is not an IDE. There is a plethora of tools for handling classpaths questions. I personally use gradle; before that I used simple shell scripts with project relative CLASSPATH and that's it. ant+ivy, maven, leiningen, … all will they do a sufficient job. Vim is an editor. It has no business with Java classpaths. (For eclipse this is different: eclipse is expected to handle the classpath, no?) Besides that, you cannot portable start a background process from inside vim. (At least I'm not aware of how to do that)

Really, there isn't a way to start processes from VIM? How about just
opening a temporary buffer for the output of the nailgun server, and
then start it with a bang!?

How about offering a variable that we can set in our .vimrc files with
a command to run on plugin load? That way I can start with "lein
nailgun", you gravy, shellscripts, or whatever. Having it start
automatically would be very nice because now you have to have multiple
terminals open and run multiple commands to just pop open a clojure
file with a repl.

-Jeff

[Meikel Brandmeyer]

Hi,

On Jul 21, 1:35 pm, Jeff Rose <ros...@gmail.com> wrote:

> Really, there isn't a way to start processes from VIM?  How about just
> opening a temporary buffer for the output of the nailgun server, and
> then start it with a bang!?

I was a but unclear on what I mean with background: I can start
processes from Vim. That's not the problem. But then Vim waits for
them to complete. And no. Adding & doesn't help, because it doesn't
work in Windows. So you fire up your server and have a waiting Vim
which cannot be used anymore. Ah, the attainments of single-threaded
applications. There is certainly no race condition. Bleh. :(

If this was not the case I could offer such a starter command.
Although I doubt it has much value, because Vim's started from the
graphical user interface don't know your project directory. So
starting at plugin load would not work. So you need another command
(maybe key bindings) to start the server after changing the vim cwd
(or providing it as an argument). Complexity creeps in for a non-
issue.

> Having it start automatically would be very nice because now you
> have to have multiple terminals open and run multiple commands
> to just pop open a clojure file with a repl.

This is a gross exaggeration. You can just as well send the process
into the background with shell job control. Then you don't block a the
terminal and still have control over the server process. Which you
don't have with the fire-and-forgot processes started from Vim. And
even if you don't use the job control, you can minimise the window.
This terminal is certainly not an issue.

You should be more concerned about the integration of your VCS system.
There you run several commands more than once all during your
development work. Not only one in the morning.

Believe me: This is a non-issue. I will include such functionality
only if someone provides me a working, portable patch. Otherwise
things stay as they are.

Sincerely
Meikel

[j-g-faustus]

On Jul 20, 10:15 am, Laurent PETIT <laurent.pe...@gmail.com> wrote:
> Oh, maybe I understand: by "partial drafts", you mean "pseudo-code",
> directly written in clojure, is it this ?

It could be pseudo-code or partial implementations or code that is
more or less complete but won't compile because the functions or libs
it is using aren't written yet. Or a number of other things that might
cause compilation to fail.

The non-compiling code tends to be syntactically correct, at least
enough to get the Enclojure paren-matching, bulk formatting and code
navigator to work - these are all things I want to help me quickly
navigate the drafts.

> If there was a shortcut key to quickly wrap/unwrap a top level defn with
> (comment), and if the keyboard shortcut for "evaluate top level form in the
> REPL" would be smart enough to detect top level comment and send to the REPL
> the unwrapped form, would it help you ?
> That way, your files would visually show the "pseudo-code" parts (those
> within comments), and your namespaces will remain in a loadable shape,
> eventually relieving you from manually load your project "piece by piece"
> every time you must restart a REPL ?

Actually, it's an interesting idea.

If I were to think aloud on a possible feature in a future IDE, it
could go something like this:

- Some special markup for pseudo-code. Not (comment) as it is already
used for comments, but perhaps something like (pseudo) or (draft). It
would act as a comment from the compiler's perspective but the IDE
could treat it differently.
Or perhaps a keyword argument to comment, ignored by the compiler but
read by the IDE - like (comment :pseudo ....).

- The contents of (pseudo) would be parsed and displayed in the code
navigator window (I assume CCW has one as well?) as a distinct group
or marked in some other way, so you could at a glance see which
functions/vars are implemented and which ones are still at the draft
stage.
It would be a code-level TODO-list of sorts.

It would have the advantage over what I am doing in Enclojure today
that I could load the non-draft functions without resorting to
"evaluate expression", and the navigator window would explicitly show
me what remains to be implemented. Very handy for large files.

I can't see the immediate need to send pseudo-code to the REPL - it's
in the pseudo-tag because it doesn't work, isn't it? (Copy/paste works
fine for those rare occasions where I might want to do so anyway.)
I would probably edit it in the editor, move it out of the pseudo-tag
when it is ready for testing and evaluate it then.

I guess something like this would take a few iterations to get right,
but I imagine I would find it very useful.


Regards
jf

[Aaron Cohen]

On Wed, Jul 21, 2010 at 9:33 AM, Meikel Brandmeyer <m...@kotka.de> wrote:

Hi,

On Jul 21, 1:35 pm, Jeff Rose <ros...@gmail.com> wrote:

> Really, there isn't a way to start processes from VIM?  How about just
> opening a temporary buffer for the output of the nailgun server, and
> then start it with a bang!?

I was a but unclear on what I mean with background: I can start
processes from Vim. That's not the problem. But then Vim waits for
them to complete. And no. Adding & doesn't help, because it doesn't
work in Windows. So you fire up your server and have a waiting Vim
which cannot be used anymore. Ah, the attainments of single-threaded
applications. There is certainly no race condition. Bleh. :(

If this was not the case I could offer such a starter command.
Although I doubt it has much value, because Vim's started from the
graphical user interface don't know your project directory. So
starting at plugin load would not work. So you need another command
(maybe key bindings) to start the server after changing the vim cwd
(or providing it as an argument). Complexity creeps in for a non-
issue.

Apologies if you know all this Meikel, but have you seen http://vim.wikia.com/wiki/Execute_external_programs_asynchronously_under_Windows?

--Aaron 

[Meikel Brandmeyer]

Hi,

On Jul 22, 8:45 am, Aaron Cohen <aa...@assonance.org> wrote:

> Apologies if you know all this Meikel, but have you seenhttp://vim.wikia.com/wiki/Execute_external_programs_asynchronously_un...
> ?

Thanks for the pointer. In fact I haven't seen this. I'm obviously
googling the wrong keywords.

Sincerely
Meikel

[Zmitro Lapcjonak]

On Jul 20, 10:02 pm, Laurent PETIT wrote:

> that'll bit anybody from time to time ( after having spent 15 minutes
> understanding why this doesn't work anymore: "OH YES, I have forgotten to
> synchronize the REPL ...")

thus i do enter nothing in the CCW repl.
all code, both business logic and all tests, tries, doc and
calculations i type
in the current file (having highlighting, code-completion and paredit
- missing in repl),
then select and send it to repl.
before save or just after try eval's - i comment or delete the not
needed lines.
(still waiting for implementation of Ctrl+/ for multiple-line
comments :)

--
Zmi La

[Lee Spector]

I too often use a file as a pseudo-REPL in this way, but note that this means putting top-level forms in the file, which is apparently discouraged, and in the context of CCW's default setting of rebuild-on-save will cause your top-level forms to be re-evaluated each time you save (which you say that you are, but still I think this is a hinderance).

--