Literate programming in emacs - any experience?
Colin Yates
Sam Ritchie
--
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clo...@googlegroups.com
Note that posts from new members are moderated - please be patient with your first post.
To unsubscribe from this group, send email to
clojure+u...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
daly
> Hi all,
>
>
> There are some excellent resources on this mailing list regarding
> literate resources, but they are more based around the theory rather
> than actual use.
>
>
> Has anybody got any "real world usage reports" regarding using
> literate programming in emacs? In particular, does paredit and slime
> work inside the clojure "fragments" when using org.babel for example?
I've been using literate programming for years in Axiom.
Basically I write in a buffer containing latex, my literate tool
of choice. I have a *shell* buffer open running Lisp.
The lisp code is in a "chunk" delimited by
\begin{chunk}{chunkname}
(this is the lisp code)
\end{chunk}
All I have to do is clip the lisp code, yank it into the *shell*
buffer, and it gets evaluated.
Once I've developed a small piece of program (about 20 lines
or so), I do a complete system rebuild and run all of the tests.
If all of the tests pass I git-commit the result, push it to
the public servers, and start writing again.
>
>
> Finally - how are people finding practising TDD with literate
> programming? I imagine that Clojure's excellent REPL (+ evaluating
> clojure forms from within a buffer) mean there are far less "type,
> extract tangled code, run tests" needed. Hmmm, not sure that is
> clear. What I mean is, do people find that the ability to evaluate a
> clojure form from within org.babel (assuming that is possible!) is
> sufficient for TDD or do you find you need to type, extract the
> tangled code and then run lein (for example) to run the tests?
Axiom has a directory of tests, all of which are also literate
documents. When I create a new test I write the new tests, run
the tests, and capture the output. The captured output is part
of the literate test document.
When I do a system build the new test is run as part of the whole
test suite. The test results are compared against the stored
results and failures are noted.
I've attached an example of a literate test case.
>
>
> Basically - how do y'all get on with TDDing in emacs following the
> approach of literate programming - any advice welcome!
>
>
> Thanks all.
>
>
> Col
>
daly
> I've been wondering this as well -- I'm specifically curious about how
> one might jump back and forth between literate source like this and
> the REPL. I know you can evaluate code snippets into the repl; I'm
> thinking of early steps like loading an entire namespace into the repl
> when its code exists in separate code blocks. Some command like
> "evaluate all code snippets that'll be tangled into this file."
It is trivial to write a lisp program which reads a literate file,
collects the chunks into a hash table, and writes out the chunks
into a standard text file which you can load.
I've attached a common lisp file that I use for that purpose.
It accepts either noweb syntax for chunks, as in:
<<chunkname>>=
(this is lisp code)
@
or latex syntax for chunks, as in:
\begin{chunk}{chunkname}
(this is lisp code)
\end{chunk}
You could also write one to process straight HTML, as in:
http://axiom-developer.org/axiom-website/litprog.html
I would even be trivial to write an elisp version that runs
as an emacs command, although I've never felt the need.
Tim Daly
Mark Engelberg
> programming in emacs? In particular, does paredit and slime work inside the
> clojure "fragments" when using org.babel for example?
I had major problems trying to get paredit to work with org.babel
mode, especially when you throw my preference for CUA mode into the
mix. Too many of the key bindings conflict with one another. I also
couldn't figure out how to make any of the typical keystrokes (C-c
c-k, for example, to compile a whole file) do anything useful in org
mode.
My top three wishlist for more blog reports:
1. Literate programming workflow within a typical Clojure setup (Tim's
workflow description is interesting, but as far as I can tell his way
of doing things wouldn't transfer over to my setup. Would also be
nice to see reports from users of Eclipse, etc.).
2. "Here's how I debugged this problem" reports in various IDEs and/or
emacs setups.
3. "Here's how I profiled this program" reports in various IDEs and/or
emacs setups.
--Mark
Colin Yates
Cedric Greevey
> It accepts either noweb syntax for chunks, as in:
> <<chunkname>>=
> (this is lisp code)
> @
A rather unfortunate choice of delimiter if you wanted to use this
with Clojure code, since Clojure code frequently has internal @-signs
for other purposes. I don't suppose that noweb syntax was developed
with Clojure in mind, though, or vice-versa.
Andy Fingerhut
Of course, there is nothing magic about the syntax or the tangle.lisp source code that parses it that says a developer must use them -- I think one of Tim's points is that these tangle programs are so simple you can cook up one of your own very easily. That can be an advantage or a disadvantage, depending upon whether you want a de facto standard used by many developers for this purpose.
Andy
daly
The noweb syntax was chosen by Norman Ramsey, the author.
Since the tangle program processes the literate document,
the REPL would never see the chunk syntax. The unfortunate
side effect is that latex DOES see the chunk syntax so you
have to use a "weave" program to get straight latex.
Instead I implemented a new latex environment called "chunk"
so I could use \begin{chunk} as the delimiter. This means
that the weave step is no longer required and my document
is straight latex.
In the HTML version I used <pre id="chunkname"> so that
the weave step is not required either.
For Eclipse I suppose we could invent a chunk syntax
and create a plugin. If people are interested perhaps we
could create a Literate Clojure plugin for Eclipse. See
http://www.eclipse.org/articles/Article-Your%20First%
20Plug-in/YourFirstPlugin.html
That would make Clojure and Literate much more useful
to Eclipse users.
Tim Daly
Cedric Greevey
> On Mon, 2012-01-23 at 16:17 -0500, Cedric Greevey wrote:
>> On Mon, Jan 23, 2012 at 11:19 AM, daly <da...@axiom-developer.org> wrote:
>> > It accepts either noweb syntax for chunks, as in:
>> > <<chunkname>>=
>> > (this is lisp code)
>> > @
>>
>> A rather unfortunate choice of delimiter if you wanted to use this
>> with Clojure code, since Clojure code frequently has internal @-signs
>> for other purposes. I don't suppose that noweb syntax was developed
>> with Clojure in mind, though, or vice-versa.
>>
>
> The noweb syntax was chosen by Norman Ramsey, the author.
> Since the tangle program processes the literate document,
> the REPL would never see the chunk syntax.
But the tangle program would, and would interpret the chunk as
stopping at the first use of @my-atom or @some-ref or @a-future...
> Instead I implemented a new latex environment called "chunk"
> so I could use \begin{chunk} as the delimiter. This means
> that the weave step is no longer required and my document
> is straight latex.
That works much better, since \end{chunk} is not valid Clojure, as
\end is not a valid character constant, and {chunk} is not a valid map
(one fewer values than keys!), so the end delimiter should never occur
accidentally inside of Clojure code. Well, there's the minor matter of
the slight chance of "\\end{chunk}" or some similar string literal, I
suppose, but it's a lot less likely than an @ character! And such a
string literal can be broken up using e.g. (str "\\en" "d{chunk}").
Though that's a bit of an ugly hack, it's only likely to arise in
exactly one instance: the Clojure code used to implement tangle
itself.
> In the HTML version I used <pre id="chunkname"> so that
> the weave step is not required either.
</pre> being the delimiter, I presume. Another one that ought to be
rare in Clojure code and absent outside of string literals, though
given the frequent use of Clojure with Compojure/etc. to emit HTML,
it'll be more common there than \end{chunk}. I suppose it might also
be a valid symbol, but you'd have to be crazy to have (defn </pre>
[foo] ...) in your code base. :)
> For Eclipse I suppose we could invent a chunk syntax
> and create a plugin. If people are interested perhaps we
> could create a Literate Clojure plugin for Eclipse. See
> http://www.eclipse.org/articles/Article-Your%20First%
> 20Plug-in/YourFirstPlugin.html
> That would make Clojure and Literate much more useful
> to Eclipse users.
Based on CCW, or a de novo effort?
daly
> > For Eclipse I suppose we could invent a chunk syntax
> > and create a plugin. If people are interested perhaps we
> > could create a Literate Clojure plugin for Eclipse. See
> > http://www.eclipse.org/articles/Article-Your%20First%
> > 20Plug-in/YourFirstPlugin.html
> > That would make Clojure and Literate much more useful
> > to Eclipse users.
>
> Based on CCW, or a de novo effort?
>
I don't use Eclipse. I was just following the principle that
"advocacy is volunteering" and, since I'm advocating doing
literate programming and the topic is literate Clojure
under Eclipse, I felt I needed to set up some kind of
solution. I try to implement what I advocate (e.g. showing
tangle for lisp code and HTML code). Otherwise I'd be
expecting someone else to do stuff I want and that's not
really how open source should work.
So, no, if CCW is already doing this then follow their lead.
Tim Daly
Cedric Greevey
CCW isn't, to my knowledge, specifically doing literate-support. But
it is an existing Clojure IDE plugin for Eclipse, so building upon
that might make more sense than creating a whole new Eclipse Clojure
IDE plugin from scratch.
Colin Yates
Stuart Sierra
You need the latest versions of org-mode, SLIME, and clojure-mode. My .emacs has the relevant elisp snippets. http://github.com/stuartsierra/dotfiles
-S
Colin Yates
Ulises
thread: http://www.jstatsoft.org/v46/i03/paper
Cheers
U
Folcon
daly
Sort of "extreme pair programming with everybody"? :-)
There is no such thing as a simple job but I'll see what I can do.
A watchable video takes time to compose.
Tim Daly
Colin Yates
daly
> I know I would find it incredibly helpful, and would consider paying a
> token sum of money (£5?)...
An amusing thought but no thanks.
Buy yourself a pint and swear you'll at least try to write
your next program in some form of literate programming.
At the next conj I'll buy you a pint.
Colin Yates
Folcon
daly
I am composing the notes for this video and I realized I have
a piece of programming information about myself that you probably
do not have about yourself.
Years ago I wrote a program (Archive) to allow users of a time
shared mainframe to back up programs to magnetic tape. This was
a single person project. As an experiment I copied every input
at the keyboard and every output of every command. I also kept
a journal of everything I did, why I did it, and what failed.
The key result was that I discovered what I call my personal
"irreducible error rate". If I do 100 things I will make 3 errors.
This was independent of the task. So typing 100 characters has
3 wrong letters which were mostly caught while typing. Writing
100 lines of code had 3 errors somewhere. Composing email
introduces 3 errors per 100 lines of code.
Most of the errors were trivial (spelling), some were syntax
(missing delimiter), some were semantic (wrong action), some
were design (wrong solution).
As a result of this measurement I have formed habits to look
for the 3 mistakes I know exist. For example, I am writing a
book of Axiom graphics images (a "gallery"). I have written
679 equations so far, which are going to create the images.
I know there are at least 21 uncaught errors lurking in those
equations. So before I attempt to generate the images I will
examine them in detail with an eye toward finding what I know
is there.
Literate programming is both a source of errors and a help.
It is a source of errors because I have more typing to do
and will generate bad latex and meaningless sentences, as
well as the "standard" coding errors. So in some sense I have
added to the number of errors I WILL introduce. But the tools,
like the compiler, tex and spellcheck will at least warn me of
these failures.
However, literate programming helps because I am explaining
what I am doing to myself as well as others. This really
reduces the "big", "costly" errors, those in the design and
implementation. There are no tools to help so I have to create
a discipline that will highlight these errors as soon as
possible.
I KNOW my intrinsic error rate. What is yours?
What discipline can you apply to your work to find your errors?
I'm not sure how to get this important piece of self-knowledge
into the video so I thought I'm mention it here.
Tim Daly
Sam Aaron
On 30 Jan 2012, at 17:07, daly wrote:
>
> The key result was that I discovered what I call my personal
> "irreducible error rate". If I do 100 things I will make 3 errors.
> This was independent of the task. So typing 100 characters has
> 3 wrong letters which were mostly caught while typing. Writing
> 100 lines of code had 3 errors somewhere. Composing email
> introduces 3 errors per 100 lines of code.
I wonder if that rate has changed since the time you measured it.
Sam
daly
errors that I continue to make. The 2/3 keys get mistyped.
The _/+ keys get mistyped. I seem unable to overcome this.
Which is really bad considering my hobby is computer algebra.
The largest contribution to my errors is using copy/paste.
It is responsible for about 50% of all errors. If I could
convince myself to stop using it my error rate would drop.
On good days I don't use it but I get lazy.
Curiously I do have a lower error rate in Lisp than I do in
other languages. In C, for instance, I get caught by things
like float to double conversions on calls, despite knowing
about it. In Java I miss the null case checks despite being
aware that Java is brain-dead about null. In Python I skip
"self"ing things. In Javascript I miss the 'var' occasionally.
Lisp "just works" and works just as I expect. It eliminates
whole categories of errors. (The most annoying thing about
Clojure is the "null pointer exceptions" from Java.) When
I want solid, correct code I reach for Lisp. For random
segment faults, C. For heap exhaustion or null pointers,
Java, etc. Rich did a marvelous thing by forcing alter to
require dosync. It eliminates a whole class of errors.
When I find a mistake I still try to find the root cause.
Then I try to change what I do so the mistake cannot exist.
This changes the type of possible errors but the 3% is still
there. I just make "more sophisticated, higher level errors".
I am hoping that Literate Programming will raise my errors
to truly epic proportions :-)
One of my personal motivations for literate programming is
to eliminate errors. I have no tool that will catch errors
in reasoning, missing cases, bad design, mindless stupidity,
and horrible inefficiency. Writing an explanation of the
code is the best way I have found to catch errors of this
kind.
One of Rich's stated motivations for Clojure was that he
found concurrent programming in various languages to be
very error prone and wanted to create a language that would
eliminate concurrent errors. In some sense, we are trying
to achieve similar goals.
So Literate Programming is not about tools. It is about a
change in mindset. I want to be a better programmer and this
is an effective tool to help me generate higher quality (ie
less buggy) code.
Tim Daly
Hugo Duncan
> Has anybody got any "real world usage reports" regarding using literate
> programming in emacs? In particular, does paredit and slime work inside
> the clojure "fragments" when using org.babel for example?
For the update in Pallet docs [1], we've been using Jekyll with
markdown. To edit code blocks within the markdown, I've been using
mumamo [2] in emacs, with a variant of jekyll-mumamo.el [3]. An example
of what markdown with clojure blocks looks like is the how-to for using
a server-rack with Pallet [4] (unfortunately, Jekyll doesn't use the
same fencing for code blocks as GitHub).
SLIME works fully within the code blocks. For example C-x C-e can be
used to evaluate expressions. Paredit also works.
Obviously this is not a full literate programming environment, but
someone might find it useful.
Hugo
[1] http://palletops.com/doc/
[2] http://ourcomments.org/Emacs/nXhtml/doc/nxhtml.html
[3] https://gist.github.com/1666286
[4] https://raw.github.com/pallet/pallet.github.com/master/doc/how-tos/_posts/2012-01-26-using-pallet-with-existing-servers.md
Mark Engelberg
> SLIME works fully within the code blocks. For example C-x C-e can be
> used to evaluate expressions. Paredit also works.
My understanding is that unless you use C-c C-k to evaluate the entire
file (which I don't think works in a literate program) the expressions
evaluate in a way that doesn't provide Clojure with line numbers,
making stacktraces even more obtuse. Am I wrong, or has anyone found
a way around this issue?
Hugo Duncan
I use ritz [1], which keeps track of of both repl source forms, and sets
correct line numbers when compiling functions with C-c C-c.
Each REPL source form is numbered, and you can list them with M-x
slime-list-repl-forms.
When running a function compiled from a markdown file with C-c C-c
(which is M-x slime-compile-defun), any exception ends up showing the
line number in the markdown file. eg.
Backtrace:
0: user$f.invoke (2012-01-14-script.md:27)
Also, exceptions land you in the debugger, so it is quite easy to see
where a problem is occurring.
Fogus
Hugo Duncan
> I would love to see your .emacs setup around these tools.
I'll put together a blog post - my .emacs files could do with a cleanup,
so this sounds like a good excuse to get it done.
daly
Sorry for the delay. Here is the answer to your request.
Note that the source and PDF are at:
src: http://daly.axiom-developer.org/clojure.pamphlet
pdf: http://daly.axiom-developer.org/clojure.pdf
I have a quick video of my normal literate programming workflow
(ignoring the usual git commits).
It shows 3 things:
1) Extracting Clojure from the book and rebuilding the book PDF.
2) adding code chunks and rebuilding clojure and the book
3) making text-only modifications and rebuilding only the PDF.
Tim
Tim Dysinger
On Monday, January 23, 2012 3:14:09 AM UTC-10, Colin Yates wrote:
Hi all,There are some excellent resources on this mailing list regarding literate resources, but they are more based around the theory rather than actual use.
Has anybody got any "real world usage reports" regarding using literate programming in emacs? In particular, does paredit and slime work inside the clojure "fragments" when using org.babel for example?
Finally - how are people finding practising TDD with literate programming? I imagine that Clojure's excellent REPL (+ evaluating clojure forms from within a buffer) mean there are far less "type, extract tangled code, run tests" needed. Hmmm, not sure that is clear. What I mean is, do people find that the ability to evaluate a clojure form from within org.babel (assuming that is possible!) is sufficient for TDD or do you find you need to type, extract the tangled code and then run lein (for example) to run the tests?Basically - how do y'all get on with TDDing in emacs following the approach of literate programming - any advice welcome!Thanks all.Col
Michael O'Keefe
martin_clausen
daly
> I'm using org-mode, org-babel & swank for a "living" document I'm
> writing. I'm generating a PDF which includes documentation, working
> clojure code that generates incanter graphs. Also the generated
> incanter graphs are included in the generated (latex) PDF.
>
> On Monday, January 23, 2012 3:14:09 AM UTC-10, Colin Yates wrote:
> Hi all,
>
>
> There are some excellent resources on this mailing list
> regarding literate resources, but they are more based around
> the theory rather than actual use.
Literate Programming example with Clojure
http://youtu.be/mDlzE9yy1mk
It is a quick video of my normal literate programming workflow
(ignoring the usual git commits)
It shows 3 things:
1) Extracting Clojure from the book and rebuilding the book PDF
2) adding code chunks and rebuilding clojure and the book
3) making text-only modifications and rebuilding only the PDF
>
>
> Has anybody got any "real world usage reports" regarding using
> literate programming in emacs? In particular, does paredit
> and slime work inside the clojure "fragments" when using
> org.babel for example?
>
>
> Finally - how are people finding practising TDD with literate
> programming? I imagine that Clojure's excellent REPL (+
> evaluating clojure forms from within a buffer) mean there are
> far less "type, extract tangled code, run tests" needed.
> Hmmm, not sure that is clear. What I mean is, do people find
> that the ability to evaluate a clojure form from within
> org.babel (assuming that is possible!) is sufficient for TDD
> or do you find you need to type, extract the tangled code and
> then run lein (for example) to run the tests?
When you run 'make', as shown in the video, it rebuilds the program
and runs all the tests.
>
>
> Basically - how do y'all get on with TDDing in emacs following
> the approach of literate programming - any advice welcome!
Here is an interesting quote from Greg Wilson (http://vimeo.com/9270320)
>From a IBM 1970s study: "Hour for hour - the most effective way to get
bugs out of code is to sit and read the code, not to run it and not to
write unit tests. Code review is the best technique known for fixing
bugs. Get somebody else to read your code. Sixty to Ninety percent of
errors can be removed before the very first run of the program. And
hour for hour, if you have a choice of writing unit tests and reading
code, read the code."
Thus, literate programming, which explains the reasoning behind the
code and the issues involved, would seem to make code reviews be much
more effective. I would love to do a study about this but so far I
have not found any professors interested.
Tim Daly
da...@axiom-developer.org