Save current namespace like a Smalltalk image
Robert Campbell
Sometimes I have pretty long REPL sessions where I'm trying to flesh
out some ideas. When I close my instance of Clojure Box (Emacs based)
I lose all the definitions I had worked out over time. Is there any
way to dump namespace(s) to an image? It would be great to be able to
load up some workspace image and pick up where I left off.
Rob
Daniel Lyons
Something similar was discussed recently but didn't come to a solid
conclusion: http://groups.google.com/group/clojure/browse_thread/thread/4efaee2e67a272c6/f24578bfa06e6b9c?lnk=gst&q=printing+and+reading+a+function#f24578bfa06e6b9c
This is kind of a cop-out, but in general my advice would be to work
from a file. Get your Clojure and Emacs set up so that you can compile
your stuff pretty easily and your files are in the namespace-
appropriate folder underneath your classpath. For example, I keep my
Clojure code in ~/Projects/Languages/Clojure and my Emacs config looks
like this:
(setq swank-clojure-extra-classpaths
(cons "/Users/fusion/Projects/Languages/Clojure/classes"
(cons "/Users/fusion/Projects/Languages/Clojure"
(directory-files "~/.clojure" t "\.jar$"))))
(eval-after-load 'clojure-mode '(clojure-slime-config))
(setq swank-clojure-extra-vm-args '("-Dclojure.compile.path=/Users/
fusion/Projects/Languages/Clojure/classes"))
Now if I want to load or compile a Clojure file, it just works.
Next, when I start doodling I make a file in the aforementioned
directory and put my stuff in there and open up a slime session in
another window. C-c C-c sends the current form over Slime to the
running session. Then I do my interactive testing and exploration in
the slime session. Whenever I hit on a form I want to keep, I copy and
paste it over to the file and make it into a function over there. I
might make a function with a dumb name like demo or test and put a
bunch of forms in there, and eventually they get refactored into unit
tests (or not). If I close Emacs and reopen it on a file that doesn't
yet have a namespace and whatnot, I select the stuff I want to
evaluate and do C-c C-r to evaluate the region. It's handy, if less
transparent.
The main advantage to this, apart from keeping the code clean, is that
you avoid the dirty image problem that can happen with Common Lisp or
(I assume) Smalltalk, where the code seems to work but accidentally
depends on cruft in the image that never made it into the source file.
I've had this happen in CL and found it very frustrating. I had a
tumblog which I tried to make faster by saving images and found one
day to my surprise that I couldn't make a fresh image on my server,
which was running a different architecture, because the only reason it
was able to make images on my box was because the first image had some
crap in it that never made it to the source file. Maybe this problem
isn't as prevalent in Smalltalk; maybe the JVM can circumvent this by
being cross-platform, but it's happened more than once in CL. IIRC,
CMU CL for a long time was self-hosting to such a degree it couldn't
be built at all without a running binary of a previous version. That
kind of thing makes porting to new architectures quite difficult.
Just my $0.02,
—
Daniel Lyons
Robert Campbell
random - and forgotten - code in the image. I have a lot of this
during my exploration sessions.
The main reason this is an issue for me is during development I
sometimes find I need another library added to my classpath. Right now
the only way I know how to modify the classpath in Emacs is to change
the .emacs file with an add-to-list 'swank-clojure-extra-classpaths
and reboot. I think my looking for an image solution might be a
cop-out itself; I need to learn Emacs better so I can figure out how
to modify the classpath without rebooting. Then I wouldn't be
rebooting so often and I wouldn't need to be making images to save
I'm-in-the-middle-of-a-thought state
Jurjen
come to a final solution yet. The main reason I wanted to achieve it
was that I do my developing / tinkering / brainstorming spread over
several work boxes spread out through several locations, and a clojure
REPL is cheap and easy, whereas maintaining several IDEs in synch (3
locations at work, 2 at home) can be a bit of a nightmare.
The compromise I've got at the moment is that I've made a custom
wrapper around [defn] that records the code used to create the
instance, and stores it in the metadata of the var that points to the
function. I can then cycle through the namespace definitions using ns-
interns / ns-publics and see the definition of each function, and can
save it to a file.
I tried to create a print-dup method so that the entire contents of a
namespace could be dumped to a file, but as chouser pointed out, print-
dup works on the function itself, whereas the code is stored in the
metadata of the var that points to the function (and there's no back-
link from the function to the var), so now it is a multi-stage process
to port current code in its entirety, but as I'm generally only
working on fairly limited areas of code it isn't a huge deal.
Also, any closures are not captured by capturing the source, so
there's still issues there, but for me the function definition is
generally good enough. Still have to implement it for macros as well,
but haven't needed that as much.
Incidentally, I find the easiest way to port my code around is to
print it to the repl, then cut-and-paste it to etherpad, which I can
then access from anywhere (without having to save). Now if only there
was a hosted REPL that integrated an IDE nicely I would really be set.
Lord-of-all-repls comes close, but is not pure clojure or JVM.
Jurjen
Robert Campbell
That wrapper for defn is something I was looking for in another post;
I previously asked how I can inspect the definitions of my functions
on the REPL. When I'm exploring stuff I'll be redefining many
functions many times and sometimes I lose track things. I basically
have to scroll around searching my REPL for the last definition. It
sounds like you have a solution to this problem. It seems strange to
me that Clojure doesn't support this concept natively. At some point
the function definition is compiled into bytecode to run on the JVM,
why not just automatically safe the original definition in metadata
when this is done? Have you should about adding your wrapper code to
Contrib?
Rob
John Harrop
Thanks Daniel, that makes perfect sense, especially about having
random - and forgotten - code in the image. I have a lot of this
during my exploration sessions.
Robert Campbell
> of the REPL inputs? Then you could rescue code from this, or find any cruft
> your image had become dependent on, or whatever.
the user (me) to sort though this transcript and try to reproduce the
latest version of the image. A function may be redefined many times
and the only version I'd _usually_ care about is the latest. This is
typically because the latest definition encompasses the latest
understanding of the problem or solution.
Having said that, it's better than nothing. There are certainly times
when I need to backtrack to previous definitions when I've down down
the wrong path. Having a transcript is better than where I'm at today.
Michael Wood
One way of doing this (although you can decide for yourself whether
it's acceptable) is to run clojure inside of the Unix "script"
command.
--
Michael Wood <esio...@gmail.com>
Josh Daghlian
1) trying to always work from a file rather than entering stuff by
hand at the repl
2) always working in a "scrap" namespace whose contents I can inspect
using ns-interns and clojure.inspector/inspect-tree, like so:
user=> (ns scrap
(:refer-clojure)
(:use (clojure inspector set)))
scrap=> (inspect-tree (ns-interns 'scrap))
;;shows me an empty inspector window
scrap=> (def x 10)
scrap=> (defn hello [s] (str "Hello, " s "!"))
scrap=> (inspect-tree (ns-interns 'scrap))
;;now shows me a list of what I've defined in my namespace
At some point I'll improve the gui inspectors to more closely resemble
(in functionality) the object inspectors from Eclipse or Idea, but
this suffices for now to at least identify what things I've defined so
I know to shove their definitions into a file before restarting the
repl.
Cheers
--josh
Daniel Lyons
On Jul 8, 2009, at 5:46 AM, Robert Campbell wrote:
> It seems strange to me that Clojure doesn't support this concept
> natively
Comments are part of the problem. Clojure's reader strips them out
while parsing before compiling the function, so you would lose them
during the first round-trip of read and print. Even a macro doesn't
really solve this problem; the only possible solution is to accept
comments that aren't really comments because the reader sees and
preserves them, or decide that they aren't an important part of the
function definition.
I'm sure there are other hang-ups (closures are probably another) but
this is the first one that comes to mind for me.
—
Daniel Lyons
Sean Devlin
Phil Hagelberg
> The main reason this is an issue for me is during development I
> sometimes find I need another library added to my classpath. Right now
> the only way I know how to modify the classpath in Emacs is to change
> the .emacs file with an add-to-list 'swank-clojure-extra-classpaths
> and reboot. I think my looking for an image solution might be a
> cop-out itself; I need to learn Emacs better so I can figure out how
> to modify the classpath without rebooting.
Unfortunately this is impossible due to the way the classloaders in the
JVM work; you can't modify your classpath at runtime and have it work
consistently.
The solution I've settled on is to have a static classpath; no matter
what the project is, my classpath is always the same:
src/:target/dependency/:target/classes/:test/
When you need to add a new library, instead of putting the jar on the
classpath, just unpack the jar in the target/dependency directory. Then
you can access it without restarting the JVM.
-Phil
Daniel Lyons
On Jul 8, 2009, at 10:22 AM, Sean Devlin wrote:
> Isn't this why you would use a doc string, and not a comment?
Docstrings aren't the only comments in my code. :)
—
Daniel Lyons
mccraigmccraig
i've seen remarks to the effect that it doesn't work well, but i don't
understand why...
c
Robert Campbell
>> the
>> JVM work; you can't modify your classpath at runtime and have it work
>> consistently.
Eclipse and a little IntelliJ and both IDEs allow you to modify
classpath at any time. It seems like it would be a basic requirement
for any IDE. Think of how often you do this. If I had to reboot
Eclipse every time I'd go crazy. Then throw in the lack of image
support and it becomes a real obstacle.
Does anyone know exactly when the JVM is booted for the REPL? Is it
from swank or clojure-mode or one of these systems? Couldn't there be
a way to reboot this subsystem instead of all Emacs?
Maybe from the command line you could execute the (add-to-list
'swank-clojure-extra-classpaths "/whatever") then reboot the subsystem
which invokes the JVM? Or modify the .emacs file and just reboot the
subsystem?
It's tough because I'm completely new to Emacs and I'm not clear on
how all the pieces fit together yet.
Phil Hagelberg
> Doesn't this seem a little crazy though? My day job is Java dev in
> Eclipse and a little IntelliJ and both IDEs allow you to modify
> classpath at any time. It seems like it would be a basic requirement
> for any IDE. Think of how often you do this. If I had to reboot
> Eclipse every time I'd go crazy. Then throw in the lack of image
> support and it becomes a real obstacle.
Oh, sorry; sounds like I was unclear. You don't have to restart Emacs,
just the JVM subprocess.
M-x slime-restart-inferior-lisp
You don't run into this problem with Java dev in Eclipse because Java
doesn't have a REPL. I agree that it sucks though. Unpacking your
dependencies all in one place is a lot nicer anyway; no restarts required.
-Phil
Richard Newman
> transcript
> of the REPL inputs? Then you could rescue code from this, or find
> any cruft
> your image had become dependent on, or whatever.
For the record, this is usually termed "dribbling" in the Lisp world.
It's very handy for debugging customer interaction -- just tell them
to turn on dribble, and have them send you the file rather than
copying and pasting:
http://www.franz.com/support/documentation/8.1/doc/introduction.htm#reporting-bugs-1
Robert Campbell
a directory, and any jar you throw in there gets exploded to the
classpath dir like you use. That would make it pretty painless.
> For the record, this is usually termed "dribbling" in the Lisp world.
> It's very handy for debugging customer interaction -- just tell them
> to turn on dribble, and have them send you the file rather than
> copying and pasting:
are in Lisp? When I was reading a post by Steve Yegge he wrote
"As far as I know, Lisp had it first, and it's called advice in Lisp.
Advice is a mini-framework that provides before, around, and after
hooks by which you can programmatically modify the behavior of some
action or function call in the system."
http://steve-yegge.blogspot.com/2007/01/pinocchio-problem.html
Searching turns up some pretty thin results, maybe the better one
being details on Emacs Lisp advices:
http://www.delorie.com/gnu/docs/elisp-manual-21/elisp_212.html
Does Clojure support something like this? I didn't come across it in
Stuart's book, website, etc. and AOP certainly helps manage complexity
in Java
Richard Newman
> are in Lisp? When I was reading a post by Steve Yegge he wrote
CLOS (the Common Lisp Object System) has before, around, and after
methods. Advice is similar, and certainly informed the CL standard. So
far as I understand it, advice (in its various historical forms)
provides a way to intercept the call to a function to modify arguments
or otherwise "advise" the function about what to do. The function
continues to do the heavy lifting, but is unaware of the advice.
CLOS around methods can do the same thing, with two differences:
* Advice is -- I think -- intended to be narrower in scope (you don't
put big functionality in advice: you add individual features). Emacs
doesn't follow this, of course :)
* Advice is more modular -- around/before/after methods in CLOS are
run in a defined order, each controlling the invocation of the next
around method; multiple pieces of advice can be toggled on or off.
You'd do things like turn on tracing/logging/profiling using advice.
Interesting reading:
http://p-cos.blogspot.com/2007/12/origin-of-advice.html
> Does Clojure support something like this? I didn't come across it in
> Stuart's book, website, etc. and AOP certainly helps manage complexity
> in Java.
IME AOP in OO languages usually helps because they lack higher-order
functions, first-order functions, and other linguistic features. It's
much easier to define global triggers in Lisps: just have some global
list of function objects that you can call, call a method and allow
other people to define their own arounds, etc.
You can implement something like method combinations in Clojure, but
it's not built in. People have worked on CLOS-style object systems in
Clojure:
http://code.google.com/p/explorersguild/wiki/XGModelAndGF
HTH,
-R
Daniel
>
> Robert Campbell <rrc...@gmail.com> writes:
>
>> The main reason this is an issue for me is during development I
>> sometimes find I need another library added to my classpath. Right now
>> the only way I know how to modify the classpath in Emacs is to change
>> the .emacs file with an add-to-list 'swank-clojure-extra-classpaths
>> and reboot. I think my looking for an image solution might be a
>> cop-out itself; I need to learn Emacs better so I can figure out how
>> to modify the classpath without rebooting.
>
> Unfortunately this is impossible due to the way the classloaders in the
> JVM work; you can't modify your classpath at runtime and have it work
> consistently.
>
> The solution I've settled on is to have a static classpath; no matter
> what the project is, my classpath is always the same:
> src/:target/dependency/:target/classes/:test/
>
> When you need to add a new library, instead of putting the jar on the
> classpath, just unpack the jar in the target/dependency directory. Then
> you can access it without restarting the JVM.
>
</snip>
> You don't run into this problem with Java dev in Eclipse because Java
> doesn't have a REPL. I agree that it sucks though. Unpacking your
> dependencies all in one place is a lot nicer anyway; no restarts required.
Uhm, that's not exactly true. The classpath is what you give your VM
as initial set of available classes to what it defines internally. The
definition is static, but there are other ways to inject code into the
class lookup mechanism inside the JVM which is where the classloaders
come in. It's perfectly fine to reload any class at runtime, including
the ones in a jar if they are defined in a classloader that you can
control. There are two problems you have to overcome though.
1) If you want to change classes that are already loaded, one strategy
is to hot swap the code (either via JVMTI, that's what you can do when
you're in debug mode in an IDE, or you use JavaRebel (commercial),
which goes a bit further in what you're able to redefine without
killing the JVM, and losing state). Not sure if that's needed for
Clojure code, since I haven't looked much at the bytecode so far, but
it's really handy for Java and Scala for instance).
2) If you really need to add new things to the classpath (e.g. a new
Jar), or you just want to reload a whole bunch of stuff, then you have
to ditch the classloader that has these classes loaded (does not apply
if you're adding new stuff), and add a new classloader that finds the
new dependency at runtime. This is not that hard to do (check
classloader javadoc), but it has quite a few cornercases, so it's not
recommended to try it adhoc. There are systems that implement the
whole system for you. Eclipse itself (the IDE, not the projects in it)
can install plugins at runtime (although it asks you whether you want
to restart), as can Maven (plugins are installed at runtime). The
mechanism that supports this in Eclipse is OSGi (in Maven it's
homegrown - classworlds), which is a nifty specification for dynamic
service management inside the JVM. It can do a lot of things, but what
is most interesting here is that it tracks versioned dependencies of
runtime components. Think Maven or Ivy at runtime, and it's
dynamically loading/unloading code based on what you specify. All of
the OSGi implementations feature a management console (a REPL) where
you can interact with the system at runtime (load and unload
dependencies, it will warn if there are unmet dependencies etc.).
I think making Clojure work well with OSGi is under 'longterm' on the
roadmap, but I don't know what the current status is. I'm definitely
giving it a vote though.
Just to be clear, the classpath you define on the commandline is just
a shortcut to define a classloader that looks up classes inside the
jars and directories you define on the commandline. If you have your
own classloaders (instantiated) you can load your code from a special
directory (that's what the webcontainers (tomcat etc.) do to support
hot redeployment of apps), the web (that's what applets do), a
database (don't know who's doing that), an uncompiled script file
(that's what scripting languages do unless they have precompiled to
class files), or any other storage you can think of. The only
requirement for a classloader is to return a bytearray that is valid
java bytecode according to the spec, and it can only load the code
once (it's cached after that, that's why you have to ditch the
classloader).
(And although the -cp argument supports wildcards since Java 1.6, they
are expanded early and cannot be reevaluated during runtime :( )
More information available at:
http://java.sun.com/javase/6/docs/technotes/tools/windows/classpath.html
http://java.sun.com/j2se/1.5.0/docs/api/java/lang/ClassLoader.html
So depending on your approach and what you want to be able to do
(redefine/reload or add), it's a bit of work. The easiest thing is to
unpack your dependencies, but for my taste it's pretty messy.
Cheers,
Daniel
(today seems to be the day of long posts. Sorry guys)
Jurjen
worth, the code I use to capture function definitions is:
(defmacro mkfn
"Create a function and record the source in the var metadata"
[name & fdecl]
(list*
`defn
(with-meta name
(assoc (meta name) :source (str (list* `mkfn name fdecl))))
fdecl))
which is based on the macro [defn-] (from clojure.core) so will work
exactly as [defn] does. To get the source back, you need to access the
metadata of the var that points to the function, so for (mkfn test [a
b] (+ a b)) you would use (meta (var test)).
I also thought about keeping previous definitions (like version
control) because I tend to occasionally destroy perfectly fine
functions. This could be done by changing the assoc line to keep a
list of definitions, but tends to cause a bit of clutter...
I can then dump any namespace to a string (to write to a file / post
to net / etc)
(mkfn get-ns
"Prints the public contents of the given namespace"
[ns]
(map #(if (:source (meta (second %1))) ; check if source was
recorded
(:source (meta (second %1)))
(if (:macro (meta (second %1))) '() ; macros not handled
yet...
(list 'def
(binding [*print-dup* true] ; else use generic print-
dup
(pr-str (second %1)) (pr-str (first %1))) (eval (first
%1)))))
(ns-publics ns))) ; over all public vars in ns
This still has some issues, especially if you have defined some
unbound vars, so is still very much a work-in-progress. As you might
note, macros aren't handled yet as print-dup tends to throw exceptions
on them when it encounters (eval (first %1)).
Also note that any functions defined by [defn] will be printed as
(defn func #<user$func__134 user$func__134@1570c24>) so when reading
back in, the reader will choke on the #<...> form (CompilerException
java.lang.RuntimeException: java.lang.Exception: Unreadable form)
Hope its useful for you, your mileage may vary (greatly, I expect)
If I get it down to something useful and reasonably functional, I'll
submit it to clojure.contrib, but for now that's about all I use.
Jurjen