How to implement "go to definition" and "find all references"
Peter Wolf
The next feature I want to implement is "references". That is, one
selects a symbol, and then can go to the location where that symbol was
defined (e.g. def, defn, let, etc.). One can also get a list of all the
locations where that symbol is referenced. This feature is very nice
for navigating code, and is also the core of many automatic refactorings
(e.g. "rename").
Implementing references are pretty straightforward in a static language
like Java, were all the references are resolved at compile time.
However, in a language like Clojure some references get resolved at run
time.
How do other IDEs handle this? Is there a recommended set of rules for
what references can and can not be resolved by the editor? How does one
detect a possible non-static reference, or how does one ensure that a
reference will always refer to the same location?
Note that I need a 100% reliable solution, if I am going to implement
automatic refactoring on top of it. No one wants refactoring that
messes up the code 5% of the time.
Thanks
Peter
Christophe Grand
> Note that I need a 100% reliable solution, if I am going to implement
> automatic refactoring on top of it. No one wants refactoring that
> messes up the code 5% of the time.
>
too dynamic (reflection, identifers hidden in XML or .properties files
etc.).
Christophe
Allen Rohner
+ Lisp has had a solution for this for years by directly asking the
running lisp process. The high level description is that the process
keeps track of all of the functions it has loaded/compiled, and the
list of functions each of those functions calls. Then to find out who
calls foo, you just ask the process. The answer is pretty reliable and
doesn't require writing your own parser. Of course, it too can't deal
with eval'ing and identifiers in XML, but I'm not sure that anything
can.
I've been toying with the idea of implementing this in clojure.
Allen
It would be fairly straightforward to modify the running clojure
Stuart Sierra
> The next feature I want to implement is "references". That is, one
> selects a symbol, and then can go to the location where that symbol was
> defined (e.g. def, defn, let, etc.).
-Stuart Sierra
lpetit
While I understand this solution has been long in place for Lips, I
don't think it looks like the ideal solution, e.g. in a world where
the "source code" is still in files, and not managed by the "lisp
image". I'm aware of just smalltalk that does this cleanly (it is even
managing versions of modifications as they are made !).
This to say, that, as an IDE provider, I feel uneasy to offer text
source code in an editor, while not being able to say to the user :
"what you are currently seeing is/is not in sync with what is in the
running lisp you see in the REPL below".
How does Slime handle that ?
Thanks in advance for your answers, (I'm not an experience emacs/slime
user, so feedback welcome !)
--
Laurent
Peter Wolf
How does SLIME handle this case?
user=> (def foo 1)
#'user/foo
user=> (defn bah [] (let [foo 2] foo))
#'user/bah
user=> (bah)
2
If I select the "foo" inside the let, I want the local one
How does the running image figure that out? What does the API to the
LISP process look like?
Also what happens if you have the following in a file? How does the
image figure out which (def...) maps to which reference?
(def foo 1)
foo
(def foo 2)
foo
Thanks
Peter
Stuart Sierra
On Jan 16, 5:37 pm, Peter Wolf <opus...@gmail.com> wrote:
> Hi and thanks for all the feedback
>
> How does SLIME handle this case?
>
> user=> (def foo 1)
> #'user/foo
> user=> (defn bah [] (let [foo 2] foo))
> #'user/bah
> user=> (bah)
> 2
process (Clojure, in this case) and gets strings back.
> If I select the "foo" inside the let, I want the local one.
> How does the running image figure that out? What does the API to the
> LISP process look like?
>
> Also what happens if you have the following in a file? How does the
> image figure out which (def...) maps to which reference?
>
> (def foo 1)
> foo
>
> (def foo 2)
> foo
forms in order as it encounters them. So first "foo" is defined to be
1, then it's redefined to be 2.
Remember, Clojure is a compiler, not an interpreter. The compiler
doesn't remember syntax. There is no "running image" in the Smalltalk
sense.
So the 100% perfect refactoring you have in mind may not be possible
without reimplementing a large portion of Clojure itself.
-Stuart Sierra
Peter Wolf
means that all references are resolved at compile time? Do I ever have
to run the code to figure out the context of a reference? Or, does the
lexical context give me all the information I need?
I have already reimplemented the Clojure parser to do the syntax
checking, folding and brace matching. Reimplementing references might
not be so bad.
In brief, I parse the Clojure program into a tree structure (of
course). Defs, defns, lets etc are all nodes on this tree. Symbols are
leafs. The nodes in the tree are sorted by the order the text appeared
in the file. Used code from other files is treated as being textually
inserted.
Can I always resolve a reference by walking back up the tree. Walk back
at the current level, if not found, go up a level and walk back, repeat.
Thanks
P
Stephen C. Gilardi
On Jan 17, 2009, at 8:40 AM, Peter Wolf wrote:
> Actually, the observation below might be really good news. Does it
> means that all references are resolved at compile time? Do I ever
> have
> to run the code to figure out the context of a reference? Or, does
> the
> lexical context give me all the information I need?
My understanding is that all references are resolved at compile time.
It's one of the design choices that helps make Clojure fast.
--Steve
lpetit
As I understand, you've made what I also began to make for clojuredev
(clojure dev environment for eclipse me and other folks are working on
on our spare time) : a static source code parser. Mine is currently
not very tested (and maybe not very usefull as is, because it has not
yet be faced to real-world problem).
Do you think it could be possible to reuse your parser for the needs
of clojuredev , or is it too tied to the intelliJ framework/
infrastructure ?
Thanks in advance,
--
Laurent
Peter Wolf
I think much of the parser, such as the JFlex lexer is certainly
reusable. The recursive descent parser outputs Intellij objects, but
with pretty minor changes could be made reuseable.
Please feel free to take anything you want.
http://code.google.com/p/clojure-intellij-plugin/source/browse/
Peter Wolf
How is the Clojure compiler tested? Is there a set Clojure code that
serves as Unit tests? I need something with all the corner cases both
for syntax and references.
Thanks
P
Meikel Brandmeyer
Am 17.01.2009 um 16:22 schrieb Peter Wolf:
> I think much of the parser, such as the JFlex lexer is certainly
> reusable. The recursive descent parser outputs Intellij objects, but
> with pretty minor changes could be made reuseable.
>
> Please feel free to take anything you want.
>
> http://code.google.com/p/clojure-intellij-plugin/source/browse/
There is lots of such things going at the moment.
- Enclojure
- Clojuredev
- the IntelliJ Plugin
- the swank/SLIME/emacs thingy
- my Vim Gorilla
Is there some interest to bundle the efforts?
I'm thinking about a project, which provides such common
things, like the Parser mentioned above. Or Chouser's or
cgrand's javadoc. Everything in a neutral way, so that the
specific frontend projects just provide the interface to the
IDE in question and use the same backend functions.
This would allow a faster development for the different
platforms, since re-inventing the wheel is not necessary.
I don't know the requirements of the different platforms,
let alone how to implement all the features like refactoring
and stuff. So I don't even know, whether this is possible
or not.
So what do you think?
Sincerely
Meikel
Meikel Brandmeyer
Am 17.01.2009 um 18:12 schrieb Meikel Brandmeyer:
> things, like the Parser mentioned above. Or Chouser's or
I'm sorry. I meant Chouser's show.
Sincerely
Meikel
Peter Wolf
As a first cut, I think we need to separate those tools written in JVM
languages (Clojure/Java) and those written in something else.
I certainly think the JVM based projects can, and should, share
components. BTW the most important JVM project is Clojure itself. The
tools should share as much as possible with the Clojure core sources.
Tools such as SLIME and (I think) Gorilla, on the other hand, are not
written in language that makes sharing easy.
However, I would be very much in favor of a common test set. A
collection of Clojure code that can be used to test tools, and ensure
common behavior. These would be useful for all tools written in all
languages.
My 2 cents
P
Matt Revelle
>
> Sure, good idea. I'm in!
>
> As a first cut, I think we need to separate those tools written in JVM
> languages (Clojure/Java) and those written in something else.
>
> I certainly think the JVM based projects can, and should, share
> components. BTW the most important JVM project is Clojure itself.
> The
> tools should share as much as possible with the Clojure core sources.
>
> Tools such as SLIME and (I think) Gorilla, on the other hand, are not
> written in language that makes sharing easy.
running Lisp process (in this case, Clojure), essentially all the
integration between Emacs/SLIME and Clojure is written in Clojure.
The component of SLIME that runs in the Lisp process is called SWANK.
I recall that a Common Lisp plugin for Eclipse, CUSP, used swank so it
may be useful environments other than Emacs/SLIME.
The main repo for swank-clojure is: http://github.com/jochu/swank-clojure/tree/master
lpetit
The problem is, I couldn't figure out quickly the detail of the swank
client they implemented, nor did I figure out quickly the detail of
the interface between slime and swank.
I wished there were already a client for swank written in clojure, but
I couldn't find one.
So, there is swank/slime to learn, and there is eclipse IDE plugin
writing to learn.
The former can be bypassed by doing something quick and not-so-dirty.
The latter can't be bypassed !
So I made the choice to dot it iteratively, and concentrate on eclipse
integration first.
At the end of the day, I know I may well have finally faced all the
problems the slime/swank protocol faced and solved them differently.
But sometimes, you have to do the trip yourself to really understand
what it is all about ! :-)
Meikel Brandmeyer
Am 17.01.2009 um 20:03 schrieb Matt Revelle:
>> Tools such as SLIME and (I think) Gorilla, on the other hand, are not
>> written in language that makes sharing easy.
>
> This is not entirely correct. SLIME works by communicating with the
> running Lisp process (in this case, Clojure), essentially all the
> integration between Emacs/SLIME and Clojure is written in Clojure.
> The component of SLIME that runs in the Lisp process is called SWANK.
Yes. Gorilla also consists basically of some part on the Vim side,
which sends stuff to a Clojure server. That part is written in Clojure.
The result is sent back to Vim.
So stuff like function completion etc. can be easily shared.
Sincerely
Meikel
lpetit
> Sure, good idea. I'm in!
>
> As a first cut, I think we need to separate those tools written in JVM
> languages (Clojure/Java) and those written in something else.
>
> I certainly think the JVM based projects can, and should, share
> components.
But alas, I'm not sure I have sufficient free time to be able to
follow the rythm.
And that's also one of the reasons why I did'nt try to spend time with
others code. If I did so, I wouldn't have produced anything myself
yet :-(
I think we could maybe share the code that manipulates text and does
static parsing. It could be very interesting, because it could also
serve later as the basis for command-line tools for static analysis
(simple refactorings such as renaming, formatting/indenting).
These tools deserve to be written in clojure or java so they can be
easily integrated in any java based IDE, tool (maven, ant, cruise
control, ...).
--
Laurent
Stuart Sierra
> > Tools such as SLIME and (I think) Gorilla, on the other hand, are not
> > written in language that makes sharing easy.
>
> This is not entirely correct. SLIME works by communicating with the
> running Lisp process (in this case, Clojure), essentially all the
> integration between Emacs/SLIME and Clojure is written in Clojure.
> The component of SLIME that runs in the Lisp process is called SWANK.
specific. There are multiple implementations of SWANK for different
Lisps, including Clojure. SLIME communicates with SWANK via a well-
defined socket interface, see <http://common-lisp.net/project/slime/>
swank-clojure <http://github.com/jochu/swank-clojure> is a little
confusing because it includes both an implementation of SWANK in
Clojure and some SLIME extensions in Emacs Lisp.
swank-clojure could be a place to implement shared-backend features.
Then again, you might not even need to modify swank-clojure. Since
SWANK can send arbitrary expressions to the Clojure process, you could
implement your introspection/reflection/refactoring features in pure
Clojure (like show, source, javadoc) and just call them through SWANK.
-Stuart Sierra
Stuart Sierra
> How is the Clojure compiler tested? Is there a set Clojure code that
> serves as Unit tests? I need something with all the corner cases both
> for syntax and references.
clojure
-Stuart Sierra
lpetit
decided to not follow slime/swank which, I guess, imposes as a middle
language something closer to emacs-lisp than to clojure for the
exchanged data structures.
And we could indeed also share the whole code of the server side.
I'm in the process of refactoring the code (client and server) for
clojuredev.
We have currently one single function, that returns a big map with all
the information for the namespace.
This has allowed me to do a namespace browser for clojuredev :
http://code.google.com/p/clojure-dev/wiki/NamespaceBrowser
--
Laurent
> 5KViewDownload
Meikel Brandmeyer
Am 17.01.2009 um 21:16 schrieb Stuart Sierra:
> SLIME communicates with SWANK via a well-defined
> socket interface, see <http://common-lisp.net/project/slime/>
Hmm.. I looked there before, but I couldn't find a definition
of the interface protocol.
> swank-clojure could be a place to implement shared-backend features.
> Then again, you might not even need to modify swank-clojure. Since
> SWANK can send arbitrary expressions to the Clojure process, you could
> implement your introspection/reflection/refactoring features in pure
> Clojure (like show, source, javadoc) and just call them through SWANK.
Unfortunately, I can't use swank due to the restrictions
imposed by the GPL.
Sincerely
Meikel
Meikel Brandmeyer
Am 17.01.2009 um 21:40 schrieb lpetit:
> Now that's interesting. It may be easier to share code because you too
> decided to not follow slime/swank which, I guess, imposes as a middle
> language something closer to emacs-lisp than to clojure for the
> exchanged data structures.
Well. I'm not sure I understand you correctly. The current layout
is as follows:
A Ruby interface in Vim, which extracts data etc. and sends
via the Ruby telnet client simple clojure expressions to some
TCP port.
There a Clojure server listens and simply executes the expressions
in Repl and sends back the result.
There is no real protocol defined at the moment and I'm somewhat
limited with the results, since implementing parser in Vim is no
fun.... But this simple setup already allows gems like a remote
clojure server or a Repl in a Vim buffer.
At the moment I'm investigating nailgun to eliminate the Ruby
stuff...
> And we could indeed also share the whole code of the server side.
That would be a tremendous win. I think interfacing to the IDE/editor
is already quite some work.
> I'm in the process of refactoring the code (client and server) for
> clojuredev.
>
> We have currently one single function, that returns a big map with all
> the information for the namespace.
> This has allowed me to do a namespace browser for clojuredev :
> http://code.google.com/p/clojure-dev/wiki/NamespaceBrowser
Uh. Nice. I'll give that a try. Let's see whether I can interface
to that.
Sincerely
Meikel
lpetit
really simple, so instead of pointing you in the svn repo, I'll put it
right after the following explanations.
I've made simple yet powerful (I think :-) choices : when I want to
display data as nodes, every node will be represented as a map. One
key of the map gives the name of the node. By convention, always with
key :name. And for allowing different types (for different behaviour
based on different data types), a :type key by convention.
If the node has children, its corresponding map will have a :children
key, which will be a vector of maps, each map will be a subnode ...
etc...
Ah, and I've made every value but the children vector strings, to be
sure that it is plain clojure data structure for read/pr
for example :
{ :name "namespaces" :type "namespaces" :children [
{ :name "clojure.core" :type "ns" :children [
{ :name "defn" :type "var" ... key/values from (meta )
Now the code :
;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
; support code
(defn- meta-info [v]
(reduce (fn [m e] (merge m { (first e) (str (second e)) })) {} (meta
v)))
(defn- symbol-info [s]
(merge { :type "symbol" :name (str s) } (meta-info (find-var s))))
(defn- var-info [v]
(merge { :type "var" :name (str v) } (meta-info v)))
(defn- ns-info [n]
{ :name ((comp str ns-name) n)
:type "ns"
:children (apply vector (map #(var-info (second %)) (ns-interns
n))) })
(defn namespaces-info []
{ :name "namespaces" :type "namespaces"
:children (apply vector (map ns-info (all-ns))) })
Regards,
--
Laurent
Bill Clementson
On Sat, Jan 17, 2009 at 12:47 PM, Meikel Brandmeyer <m...@kotka.de> wrote:
> Hi Stuart,
>
> Am 17.01.2009 um 21:16 schrieb Stuart Sierra:
>
>> SLIME communicates with SWANK via a well-defined
>> socket interface, see <http://common-lisp.net/project/slime/>
>
> Hmm.. I looked there before, but I couldn't find a definition
> of the interface protocol.
To better understand the SLIME/SWANK communications protocol, you can:
1. Read the source code (search for "Communication protocol" in
slime.el and read the code from that point)
2. Look at Tobias Rittweiler's excellent SLIME presentation
(http://trittweiler.blogspot.com/2008/12/last-wednesday-i-gave-talk-to-munich.html)
3. Examine the contents of the *slime-events* buffer in Emacs (it
contains the actual exchanges between the SLIME/SWANK components)
My recent SLIME blog post has a lot of links to other material that
you might find useful: http://bc.tech.coop/blog/081209.html
--
Bill Clementson