Knuth's literate programming "tangle" function in Clojure
Tim Daly
; 1 ABSTRACT and USE CASES
; 2 THE LATEX SUPPORT CODE
; 3 IMPORTS
; 4 THE TANGLE COMMAND
; 5 SAY
; 6 READ-FILE
; 7 ISCHUNK
; 8 HASHCHUNKS
; 9 EXPAND
; 10 TANGLE
;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
;;; 0 AUTHOR and LICENSE
;;; Timothy Daly (da...@axiom-developer.org)
;;; License: Public Domain
;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
;;; 1 ABSTRACT and USE CASES
;;; Don Knuth has defined literate programming as a combination of
;;; documentation and source code in a single file. The TeX language
;;; is documented this way in books. Knuth defined two functions
;;; tangle -> extract the source code from a literate file
;;; weave -> extract the latex from a literate file
;;; This seems unnecessarily complex. Latex is a full programming
;;; language and is capable of defining "environments" that can
;;; handle code directly in Latex. Here we define the correct environment
;;; macros. Thus, the "weave" function is not needed.
;;; If this "tangle" function were added to Clojure then Clojure could
;;; read literate files in Latex format and extract the code. We create
;;; the necessary "tangle" function here.
;;; This program will extract the source code from a literate file.
;;; A literate lisp file contains a mixture of latex and lisp sources code.
;;; The file is intended to be in standard latex format. In order to
;;; delimit code chunks we define a latex "chunk" environment.
;;; Latex format files defines a newenvironment so that code chunks
;;; can be delimited by \begin{chunk}{name} .... \end{chunk} blocks
;;; This is supported by the following latex code.
;;; So a trivial example of a literate latex file might look like
;;; (ignore the prefix semicolons. that's for lisp)
; this is a file that is in a literate
; form it has a chunk called
; \begin{chunk}{first chunk}
; THIS IS THE FIRST CHUNK
; \end{chunk}
; and this is a second chunk
; \begin{chunk}{second chunk}
; THIS IS THE SECOND CHUNK
; \end{chunk}
; and this is more in the first chunk
; \begin{chunk}{first chunk}
; \getchunk{second chunk}
; THIS IS MORE IN THE FIRST CHUNK
; \end{chunk}
; \begin{chunk}{all}
; \getchunk{first chunk}
; \getchunk{second chunk}
; \end{chunk}
; and that's it
;;; From a file called "testcase" that contains the above text
;;; we want to extract the chunk names "second chunk". We do this with:
; (tangle "testcase" "second chunk")
; which yields:
; THIS IS THE SECOND CHUNK
;;; From the same file we might extract the chunk named "first chunk".
;;; Notice that this has the second chunk embedded recursively inside.
;;; So we execute:
; (tangle "testcase" "first chunk")
; which yields:
; THIS IS THE FIRST CHUNK
; THIS IS THE SECOND CHUNK
; THIS IS MORE IN THE FIRST CHUNK
;;; There is a third chunk called "all" which will extract both chunks:
; (tangle "testcase" "all")
; which yields
; THIS IS THE FIRST CHUNK
; THIS IS THE SECOND CHUNK
; THIS IS MORE IN THE FIRST CHUNK
; THIS IS THE SECOND CHUNK
;;; The tangle function takes a third argument which is the name of
;;; an output file. Thus, you can write the same results to a file with:
; (tangle "testcase" "all" "outputfile")
;;; It is also worth noting that all chunks with the same name will be
;;; merged into one chunk so it is possible to split chunks in mulitple
;;; parts and have them extracted as one. That is,
; \begin{chunk}{a partial chunk}
; part 1 of the partial chunk
; \end{chunk}
; not part of the chunk
; \begin{chunk}{a partial chunk}
; part 2 of the partial chunk
; \end{chunk}
;;; These will be combined on output as a single chunk. Thus
; (tangle "testmerge" "a partial chunk")
; will yield
; part 1 of the partial chunk
; part 2 of the partial chunk
;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
;;; 2 THE LATEX SUPPORT CODE
;;; The verbatim package quotes everything within its grasp and is used to
;;; hide and quote the source code during latex formatting. The verbatim
;;; environment is built in but the package form lets us use it in our
;;; chunk environment and it lets us change the font.
;;;
;;; \usepackage{verbatim}
;;;
;;; Make the verbatim font smaller
;;; Note that we have to temporarily change the '@' to be just a character
;;; because the \verbatim@font name uses it as a character
;;;
;;; \chardef\atcode=\catcode`\@
;;; \catcode`\@=11
;;; \renewcommand{\verbatim@font}{\ttfamily\small}
;;; \catcode`\@=\atcode
;;; This declares a new environment named ``chunk'' which has one
;;; argument that is the name of the chunk. All code needs to live
;;; between the \begin{chunk}{name} and the \end{chunk}
;;; The ``name'' is used to define the chunk.
;;; Reuse of the same chunk name later concatenates the chunks
;;; For those of you who can't read latex this says:
;;; Make a new environment named chunk with one argument
;;; The first block is the code for the \begin{chunk}{name}
;;; The second block is the code for the \end{chunk}
;;; The % is the latex comment character
;;; We have two alternate markers, a lightweight one using dashes
;;; and a heavyweight one using the \begin and \end syntax
;;; You can choose either one by changing the comment char in column 1
;;; \newenvironment{chunk}[1]{% we need the chunkname as an argument
;;; {\ }\newline\noindent% make sure we are in column 1
;;; %{\small $\backslash{}$begin\{chunk\}\{{\bf #1}\}}% alternate begin mark
;;; \hbox{\hskip 2.0cm}{\bf --- #1 ---}% mark the beginning
;;; \verbatim}% say exactly what we see
;;; {\endverbatim% process \end{chunk}
;;; \par{}% we add a newline
;;; \noindent{}% start in column 1
;;; \hbox{\hskip 2.0cm}{\bf ----------}% mark the end
;;; %$\backslash{}$end\{chunk\}% alternate end mark (commented)
;;; \par% and a newline
;;; \normalsize\noindent}% and return to the document
;;; This declares the place where we want to expand a chunk
;;; Technically we don't need this because a getchunk must always
;;; be properly nested within a chunk and will be verbatim.
;;; \providecommand{\getchunk}[1]{%
;;; \noindent%
;;; {\small $\backslash{}$begin\{chunk\}\{{\bf #1}\}}}% mark the reference
;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
;;; 3 IMPORTS
(import [java.io BufferedReader FileReader BufferedWriter FileWriter])
(import [java.lang String])
;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
;;; 4 THE TANGLE COMMAND
;;;
;;; The tangle command does all of the work of extracting code.
;;;
;;; In latex form the code blocks are delimited by
;;; \begin{chunk}{name}
;;; ... (code for name)...
;;; \end{chunk}
;;;
;;; and referenced by \getchunk{name} which gets replaced by the code
;;; There are several ways to invoke the tangle function.
;;;
;;; The first argument is always the file from which to extract code
;;;
;;; The second argument is the name of the chunk to extract
;;; (tangle "clweb.pamphlet" "name")
;;;
;;; The standard chunk name is ``*'' but any name can be used.
;;;
;;; The third arument is the name of an output file:
;;; (tangle "clweb.pamphlet" "clweb.chunk" "clweb.spadfile")
;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
;;; 5 SAY
;;; This function will either write to the output file, or if null,
;;; to the console
(defn say [where what]
(if where
(do (.write where what) (.write where "\n"))
(println what)))
;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
;;; 6 READ-FILE
;;; Here we return a lazy sequence that will fetch lines as we need them
;;; from the file.
(defn read-file [streamname]
^{:doc "Implement read-sequence in GCL"}
(let [stream (BufferedReader. (FileReader. streamname))]
(line-seq stream)))
;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
;;; 7 ISCHUNK
;;; There is a built-in assumption (in the ischunk functions)
;;; that the chunks occur on separate lines and that the indentation
;;; of the chunk reference has no meaning.
;;;
;;; ischunk recognizes chunk names in latex convention
;;;
;;; There are 3 cases to recognize:
;;; \begin{chunk}{thechunkname} ==> 'define thechunkname
;;; \end{chunk} ==> 'end nil
;;; \getchunk{thechunkname} ==> 'refer thechunkname
;;; The regex pattern #"^\\begin\{chunk\}\{.*\}$" matches
;;; \begin{chunk}{anything here}
;;; The regex pattern #"^\\end\{chunk\}$" matches
;;; \end{chunk}
;;; The regex pattern #"^\\getchunk\{.*\}$" matches
;;; \getchunk{anything here}
(defn ischunk [line]
^{:doc "Find chunks delimited by latex syntax"}
(let [ begin #"^\\begin\{chunk\}\{.*\}$"
end #"^\\end\{chunk\}$"
get #"^\\getchunk\{.*\}$"
trimmed (.trim line) ]
(cond
(re-find begin trimmed)
(list 'define (apply str (butlast (drop 14 trimmed))))
(re-find end trimmed)
(list 'end nil)
(re-find get trimmed)
(list 'refer trimmed)
:else
(list nil trimmed))))
;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
;;; 8 HASHCHUNKS
;;; hashchunks gathers the chunks and puts them in the hash table
;;;
;;; if we find the chunk syntax and it is a
;;; define ==> parse the chunkname and start gathering lines onto a stack
;;; end ==> push the completed list of lines into a stack of chunks
;;; already in the hash table
;;; otherwise ==> if we are gathering, push the line onto the stack
;;; a hash table entry is a list of lists such as
;;; (("6" "5") ("4" "3") ("2" "1"))
;;; each of the sublists is a set of lines in reverse (stack) order
;;; each sublist is a single chunk of lines.
;;; there is a new sublist for each reuse of the same chunkname
;;; Calls to ischunk can have 4 results (define, end, refer, nil) where
;;; define ==> we found a \begin{chunk}{...}
;;; end ==> we found a \end{chunk}
;;; refer ==> we found a \getchunk{...}
;;; nil ==> ordinary text or program text
;;;
;;; gather is initially false, implying that we are not gathering code.
;;; gather is true if we are gathering a chunk
(defn hashchunks [lines]
^{:doc "Gather all of the chunks and put them into a hash table"}
(loop [ line lines
gather false
hash (hash-map)
chunkname "" ]
(if (not (empty? line))
(let [[key value] (ischunk (first line))]
(condp = key
'define
(recur (rest line) true hash value)
'end
(recur (rest line) false hash chunkname)
'refer
(if gather
(recur (rest line) gather
(assoc hash chunkname (conj (get hash chunkname) value))
chunkname)
(recur (rest line) gather hash chunkname))
nil
(if gather
(recur (rest line) gather
(assoc hash chunkname (conj (get hash chunkname) value))
chunkname)
(recur (rest line) gather hash chunkname))))
hash)))
;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
;;; 9 EXPAND
;;; expand will recursively expand chunks in the hash table
;;;
;;; latex chunk names are just the chunkname itself e.g. chunkname
;;; a hash table key is the chunk name and the value is a reverse
;;; list of all of the text in that chunk.
;;; To process the chunk we reverse the main list and
;;; for each sublist we reverse the sublist and process the lines
;;; if a chunk name reference is encountered in a line we call expand
;;; recursively to expand the inner chunkname.
(defn expand [chunkname where table]
^{:doc recursively expand latex getchunk tags}
(let [chunk (reverse (get table chunkname))]
(when chunk
(loop [lines chunk]
(when (not (empty? lines))
(let [line (first lines)]
(let [[key value] (ischunk line)]
(if (= key 'refer)
(do
(expand (apply str (butlast (drop 10 value))) where table)
(recur (rest lines)))
(do (say where line)
(recur (rest lines)))))))))))
;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
;;; 10 TANGLE
;;; We expand all of the lines in the file that are surrounded by the
;;; requested chunk name. These chunk names are looked up in the hash
;;; table built by hashchunks, given the input filename.
;;; then we recursively expand the ``topchunk'' to the output stream
(defn tangle
^{:doc "Extract the source code from a pamphlet file, optional file
output"}
([filename topchunk] (tangle filename topchunk nil))
([filename topchunk file]
(if (string? file)
(with-open [where (BufferedWriter. (FileWriter. file))]
(expand topchunk where (hashchunks (read-file filename))))
(expand topchunk nil (hashchunks (read-file filename))))))
Robert McIntyre
http://www.mail-archive.com/gard...@lispniks.com/msg01006.html and
wondering about how hard something like this would be to write. If
possible, could you expand on how one might use this in a development
work-flow with emacs or texmacs? Is there some sort of lisp-pamphlet
mode that could be extended to clojure? I'm assuming you've already
solved all these problems and more while working with Axiom. I'd
appreciate any pointers or best practices you might have found.
Thanks so much for your work!
--Robert McIntyre
> --
> You received this message because you are subscribed to the Google
> Groups "Clojure" group.
> To post to this group, send email to clo...@googlegroups.com
> Note that posts from new members are moderated - please be patient with your
> first post.
> To unsubscribe from this group, send email to
> clojure+u...@googlegroups.com
> For more options, visit this group at
> http://groups.google.com/group/clojure?hl=en
Praki Prakash
This approach is very interesting. My choice of mode for LP has always been noweb-mode but it doesn't seem to work with my version of emacs anymore. My current approach is to embed prose and clojure code in a latex document and generate a .tex file with formatted clojure code and .clj containing only clojure code. Needless to say, it is a hack and I would like to see if I can adopt your approach.
However, I have a question on mapping of line numbers in clojure stacktrace to its source. AFAIK, there is no support in clojure compiler for #LINE directive. In my case, a code is always in one location and I just replace latex lines with empty lines. How do you address this issue?
Thanks,
Praki
Tim Daly
On 12/26/2010 9:56 PM, Praki Prakash wrote:
> Tim,
>
> This approach is very interesting. My choice of mode for LP has always
> been noweb-mode but it doesn't seem to work with my version of emacs
> anymore. My current approach is to embed prose and clojure code in a
> latex document and generate a .tex file with formatted clojure code
> and .clj containing only clojure code. Needless to say, it is a hack
> and I would like to see if I can adopt your approach.
Well, I'm something of a primitivist so I don't use modes in Emacs.
I know there is a noweb mode and a clojure mode and a latex mode but
I've never used any of them so I can't comment.
In the Knuth web approach (ala noweb) the chunk markup is not valid
latex. Therefore you need to run "weave" to extract valid latex.
I found this pointless so I wrote the latex environment macros to
wrap the code chunks. This means that the document you are writing
is always pure latex. So you might find it useful to use the chunk
environment and skip the "generate .tex file" step.
All that is left is to extract the clojure code. While writing the
code I have a REPL open so I can kill/yank changes directly. The
full build/test cycle uses the attached code to extract from the
literate document (which I call a "pamphlet").
>
> However, I have a question on mapping of line numbers in clojure
> stacktrace to its source. AFAIK, there is no support in clojure
> compiler for #LINE directive. In my case, a code is always in one
> location and I just replace latex lines with empty lines. How do you
> address this issue?
I don't use the #LINE directive. I usually use a split screen
with lisp or clojure in a shell buffer (not a slime setup) and
my code in the other buffer. I've hand-checked the code in the
REPL before adding to or changing the document. After each change
I do a complete system rebuild/test cycle.
Thus, if it fails I know exactly what I broke and where I broke it.
It is broken at the last change, which is likely still in a buffer.
When it works, the code is already documented in latex and pdf (since
the build/test automatically regenerates the pdf).
I would like to see Clojure move toward literate programming and
have direct support for reading and compiling latex documents.
It is surprisingly efficient and effective. In my last 3 year
project I implemented over 60k lines of lisp and 6k pages of
documentation (with the embedded lisp code in literate form, of
course). When the "program was done", the "documentation was done".
In fact, that was true from the first day of the project and was
an invariant throughout. I highly recommend it.
Tim
Tim Daly
On 12/26/2010 8:33 PM, Robert McIntyre wrote:
> That's really cool. I was _just reading_ your comments from 2006 at
> http://www.mail-archive.com/gard...@lispniks.com/msg01006.html and
> wondering about how hard something like this would be to write. If
> possible, could you expand on how one might use this in a development
> work-flow with emacs or texmacs? Is there some sort of lisp-pamphlet
> mode that could be extended to clojure? I'm assuming you've already
> solved all these problems and more while working with Axiom. I'd
> appreciate any pointers or best practices you might have found.
Wow. That's an old reference. Most of those links are dead. Try
http://en.wikipedia.org/wiki/Axiom_computer_algebra_system
or http://axiom-developer.org
As I mentioned in my other email on this subject I don't use Emacs
modes. I don't want the computer editing my text. I just want it to
do exactly what I told it to do. But I'm a curmudgeon :-)
My style of programming is to change one thing and then do a complete
system rebuild and test, which takes about 1 hour on my fastest machine,
5 on my slowest. I "work a pipeline" of changes so the build/test time
is overlapped with changes. I can have up to 5 machines doing build/test
while I'm working on the 6th change. (Git helps a LOT here. I strongly
recommend git also.)
I usually use a split screen with the literate latex document (I call
it a "pamphlet") in one buffer (not a slime setup) and the REPL in the
other buffer. I use the REPL to hand-check the code before modifying
the pamphlet.
Thus, if (when :-)) it fails I know what I broke and where I broke it.
When it works, the code is already documented in latex and pdf.
The build/test automatically regenerates the pdf.
The goal is to make the smallest change possible and concentrate on getting
it right. It might require writing or updating test cases which are also
in the same literate document under a "test" chunk name. The automated build
extracts and runs all "test" chunks.
So the essential loop is:
edit the pamphlet in a split buffer
define the functions in the REPL buffer
change the function
hand-test the change in the REPL
update the pamphlet with the changed function
write/update the test code
save the pamphlet
do a full system build and test
check that the test pass
check the pdf documentation
The second step would be a lot easier if Clojure knew how to load
a latex file. I'm thinking of writing a load-pamphlet function to
do this. It could specify a chunk name to extract as in:
(defn
(load-pamphlet [file] (load-pamphlet file "*"))
(load-pamphlet [file chunk] (tangle file chunk)))
Thus, to do step 2 above I just need to (load-pamphlet...) the file.
I could run the tests with (load-pamphlet file "test")
\begin{rant}{
\tl;dr{Let's move programming out of the dark ages.}
We of the Clojure community are trying hard to move Lisp out of
the 60s programming model. We can also move programming out of
the 60s code-then-maybe-document model. The Knuth technology is
30+ years old.
When I started 40 years ago (http://www.dilbert.com/fast/2010-12-23/)
we had limits like 8k of memory so no file was bigger than 4k. Thus,
C programs were "tiny piles of sand" with include/overlay linkers/etc.
Few of you have ever worked on a fully loaded PDP 11/40. Why do you
program like you only have 4k?
Now we can have a machine with 128Gig of memory. Its about time for
programmers to lose the "one function-one file" idea based on keeping
the machine happy. We need to think about one-program-one-document to
keep the humans happy. For a perfect example, see the book:
"Lisp In Small Pieces by Christian Queinnec" [1]
Queinnec's book is my ultimate example. He MOTIVATES every piece of
code that gets introduced and he talks about everything. The code
contains a complete lisp system including the interpreter, compiler,
and garbage collector. If you want to know lisp, read this book.
If you want to learn literate programming, study this book.
I want "Clojure In Small Pieces", a literate form of Clojure that
I can execute from the book. Then I can open to the chapter on
PersistentHashMap and read all about log32 tries, why they matter,
and how Clojure implements them, with literature references and a
good index. Oh, yeah, and the ACTUAL code that gets executed.
Forget the jar file. Send me the pamphlet containing Clojure.
Programming will come out of the dark ages when we employ English
majors as project leads with the title "Editor in Chief". If you
can explain it to an English major in text, you probably understand
the problem :-)
Let's move programming out of the dark ages.
\end{rant}
Tim Daly
[1] http://www.amazon.com/Lisp-Small-Pieces-Christian-Queinnec/dp/0521545668
Seth
excellent place to start to do literate programming. Interesting
ideas ... maybe i will try this in my own code ...
Hubert Iwaniuk
Yes I did play with org-mode + babel for clojure.
It works great :-)
Just make sure you are using latest and greatest of org-mode.
Cheers,
Hubert.
Robert McIntyre
it with clojure for a total newbie?
sincerely,
--Robert McIntyre
Hubert Iwaniuk
http://orgmode.org/worg/org-contrib/babel/languages/ob-doc-clojure.html
Cheers,
Hubert
Tim Daly
directly from the book. It dynamically builds the source
tree from the book, runs tests, creates the pdf, and
starts the REPL.
At least in theory. I am stuck with running a couple
tests. The only real change I've made to the sources
is to make it fit a printable page which involves
changing a line to make it shorter.
I've run into a syntax for strings that I don't understand.
The string #"some string" is used in the test files. The
documentation on the reader does not list this as a possible
input case. What does it mean?
Once I cross this hurdle everything else works and I can
post a new version for your experiments.
Tim
Jeff Valk
> I've run into a syntax for strings that I don't understand.
> The string #"some string" is used in the test files. The
> documentation on the reader does not list this as a possible
> input case. What does it mean?
It's reader syntax for a regular expression.
user=> (type #"some string")
java.util.regex.Pattern
It and its reader macro friends can be found here:
http://clojure.org/reader
> Once I cross this hurdle everything else works and I can
> post a new version for your experiments.
Thanks for your efforts on this. I'm quite interested in its potential!
- Jeff
Robert McIntyre
hope that helps,
--Robert McIntyre
Tim Daly
of string, which I thought was odd. I'll look again.
Thanks for the answers.
Tim Daly
Note that 'literate programming' involves writing literature
for other people to read. The executable code is included as
a 'reduction to practice' but the emphasis is on describing
the ideas. Rich has some powerful ideas that he has reduced
to running code. What we need to do is start with a description
of the ideas and bridge the gap to the actual implementation.
Ideally you can read a literate program like a novel, from
beginning to end, and find that every line of code has a
'motivation' for being introduced. The side-effect is that
there is a reason why the idea is implemented in a particular
way rather than 'just because it worked'. Literate programming
tends to improve code quality because you have to explain it.
Emacs org-mode, on the other hand, is a useful development
technology but it really isn't literate programming.
Tim Daly
Tim Daly
http://daly.axiom-developer.org/clojure.pdf
http://daly.axiom-developer.org/clojure.pamphlet
http://daly.axiom-developer.org/clojure.sty
This version of the literate document contains a
complete, working system. The steps for building
it are in the preface.
Essentially you compile the tangle function from
the document (or use the same source code here:
http://daly.axiom-developer.org/tangle.c )
Then you run tangle to extract the Makefile.
Then you type make.
Or, for the programmers:
1) edit the file, clip out and save tangle.c
2) gcc -o tangle tangle.c
3) tangle clojure.pamphlet Makefile >Makefile
4) make
It should extract the sources, build Clojure,
test it, build the pdf, and leave you at a
REPL prompt.
The source tree lives under the 'tpd' directory.
You can put it anywhere with an argument to make, e.g.
4) make WHERE=myplace
This means that you only need the latex document
to develop (resist the urge to edit the other
files).
Now the problem is to write the ideas and connect
them to the code. I started doing this for the
Red Black tree idea and PersistentTreeMap. Feel
free to pick an idea (or suggest one) and work
out the details.
I urge you to try the edit/build cycle using
literate tools as a possible different way to
work. My usual command line after every change is:
rm -rf tpd && tangle clojure.pamphlet Makefile >Makefile && make
A complete rebuild from scratch takes less than a
minute on a fast machine.
Tim
Seth
http://orgmode.org/worg/org-contrib/babel/
Ive actually used it to create my emacs.el, by having code in
emacs.org and have init.el tangle out the emacs code. Of course i
never documented
anything and did it for the novelty of being able to organize all that
code in one file, instead of expanding it to other files :)
Seth
>it with clojure for a total newbie?
it could be used. You have the ability to embed arbitrary code (from
many different languages). You can edit the code in its own emacs
major mode and then it will automatically be saved back once done. You
can then document it using org-modes awesome abilities.
However, this is sort of clumsy.
I would rather be able to have all of my code in all of its 'little
files' arranged in directories. And when im editing the clojure files,
i would like to be like 'oh, i want to document this better/introduce
the motivation etc! And then automatically have the code, or parts of
the code, copied to the org file and then i could document it. And
then jump back to the code to continue developing. And have changes in
the clojure file automatically reflected in the org file. I was
thinking that 'chunk' labels could be embedded in the source code
(like in marginalia in github: just comments like ;;##Block Name) so
that we wouldn't have to have all code in one file in one chunk, but
could split it up.
Tim Daly
On 1/5/2011 9:27 AM, Seth wrote:
>> Just discovered org-mode myself --- does anyone know of guide to using
>> it with clojure for a total newbie?
> I havent actually used it for clojure per se. I was just imagining how
> it could be used. You have the ability to embed arbitrary code (from
> many different languages). You can edit the code in its own emacs
> major mode and then it will automatically be saved back once done. You
> can then document it using org-modes awesome abilities.
> However, this is sort of clumsy.
>
> I would rather be able to have all of my code in all of its 'little
> files' arranged in directories.
Just out of curiosity, what is the advantage of maintaining code
in 'little files'? The main reason people use an IDE is that they
get a whole-project view. The IDE lets you move around and find
things as though your project was effectively one big file. It really
is a kind of patch on the little-file organization. What is it about
'little file format' that is actually useful? Except for habit, is
there a real advantage?
Conceptually separate efforts, such as the Clojure-contrib effort
could be done in separate volumes. But you would expect this kind of
natural organization, just as you might expect a multi-volume story
like the Asimov Foundation series. Wouldn't a well organized contrib
literate document be useful? When you program and reach for your
books, don't you find a good index and cross-reference the best way
to find the idea you need?
> And when im editing the clojure files,
> i would like to be like 'oh, i want to document this better/introduce
> the motivation etc! And then automatically have the code, or parts of
> the code, copied to the org file and then i could document it. And
> then jump back to the code to continue developing.
Viewing the step of 'documenting the code' as just another step in
development is one of the reasons that documentation is rarely done
and even more rarely done well. The target audience is usually another
developer so tools like Javadoc exist to make it easy to look up class
details. What tools exist to support whole-project, idea-to-implementation
documentation?
Viewing documentation as a phase of development is conceptually and
actually very different from viewing the project as a literate effort.
Development targets the machine whereas a literate effort targets people.
The difference in target audience makes a qualitative difference in what
you write, when you write it, and why you write it. There is nothing
about the Javadoc organization (to pick on one tool) that encourages or
even supports the 'idea-to-implementation' flow. Are there open source
examples of documentation from the ideas to the implementation?
Literate programming allows you to reorganize your code by ideas. For
instance, in the clojure example, the PersistentTreeMap class is split
into its subclasses like Node. Okasaki's work starts with the idea of
Nodes so we highlight and explain the Node structure of PersistentTreeMap
before we get into the top level class details. In this way you motivate
the need for the Node class and 'bring the reader along' so that when
they get to PersistentTreeMap they already understand Nodes.
Because of the way Java forces you to organize your code you have to
introduce the PersistentTreeMap class before you introduce the Node class.
This is the late 90s and we ought to be able to organize our code any
way we want rather than be forced to organize it for the compilers,
linkers, and loaders. Why would we want to organize our code for the
convenience of our tools?
> And have changes in
> the clojure file automatically reflected in the org file. I was
> thinking that 'chunk' labels could be embedded in the source code
> (like in marginalia in github: just comments like ;;##Block Name) so
> that we wouldn't have to have all code in one file in one chunk, but
> could split it up.
>
Having literate code in more than one file is certainly possible because
Latex supports an 'include' command. You could include the code chunks or
you could include the chapters and keep them in 'litte file format'.
I'm not sure what advantage this confers. Working in a single file or
working in multiple files is pretty transparent. Emacs lets me split
buffers in the same file as easily as having two files in split windows.
Finding things is SO much easier (hey, its certainly in THIS file :-) )
The hardest initial part is using apt-get texlive.
The difference is not the physical organization but the mental organization.
A single file format gives the impression of a book and with that comes the
skills of organizing the material for presenting the ideas in a logical
fashion. It is this change of mental viewpoint that is the critical part
of literate programming rather than the tools like emacs-org-mode or latex.
The mental transition to this style of programming is at least as hard as
the mental transition from Object Oriented programming to Functional
programming.
In my experience, the gain is worth the effort.
Clojure code is nearly documentation-free at the moment. We could certainly
create some Javadoc-like tools or IDEs or Emacs-modes that fit our normal
routines or we could experiment with a whole new mindset. Clojure users
are early-adopting, risk-taking, leading-edge developers. Literate
programming
is just the right kind of challenge for innovating away from the past.
Besides, wouldn't it be great if the community could point at "the Clojure
book", like the Lisp community points to "the Steele book"? Steele managed
to organize an unreadable standard into a readable and very useful book.
Surely we can do the same.
Tim Daly
Eric Schulte
> The literate programming is actually a contrib to org-mode.
> http://orgmode.org/worg/org-contrib/babel/
>
This has been moved out of contrib and into the Org-mode core, so with
recent versions of Org-mode the code block "Literate Programming" and
"Reproducible Research" support is built in.
In fact when Emacs 24 is released this will be part of the Emacs core.
>
> Ive actually used it to create my emacs.el, by having code in
> emacs.org and have init.el tangle out the emacs code. Of course i
> never documented
> anything and did it for the novelty of being able to organize all that
> code in one file, instead of expanding it to other files :)
I do this myself and find it very convenient. In fact I maintain a
Literate fork of Phil Hagelberg's emacs-starter-kit which does exactly
this, allowing you to keep you emacs customizations in either .org files
or .el files.
The git repo for this is here
https://github.com/eschulte/emacs-starter-kit
and the documentation (exported from the literate .org files) is here
http://eschulte.github.com/emacs-starter-kit/
Cheers -- Eric
Eric Schulte
Seth <wbu...@gmail.com> writes:
>>Just discovered org-mode myself --- does anyone know of guide to using
>>it with clojure for a total newbie?
>
> I havent actually used it for clojure per se. I was just imagining how
> it could be used. You have the ability to embed arbitrary code (from
> many different languages). You can edit the code in its own emacs
> major mode and then it will automatically be saved back once done. You
> can then document it using org-modes awesome abilities.
> However, this is sort of clumsy.
>
There are a variety of options here
- you can write *all* of your code in a single large Org-mode file, and
tangle out .clj files for compilation.
- you can write *all* of your code in .clj files, and simply link to the
code from your .org files
- you can write some code in external .clj files, and some embedded in
.org files
- with current versions of Org-mode it is even possible to propagate
changes from a tangled .clj file back into the code blocks in a .org
file if e.g. you are working on a project with non-org users who would
rather edit the .clj files directly. See the `org-babel-detangle'
function.
>
> I would rather be able to have all of my code in all of its 'little
> files' arranged in directories. And when im editing the clojure files,
> i would like to be like 'oh, i want to document this better/introduce
> the motivation etc! And then automatically have the code, or parts of
> the code, copied to the org file and then i could document it. And
> then jump back to the code to continue developing. And have changes in
> the clojure file automatically reflected in the org file. I was
> thinking that 'chunk' labels could be embedded in the source code
> (like in marginalia in github: just comments like ;;##Block Name) so
> that we wouldn't have to have all code in one file in one chunk, but
> could split it up.
I am a grad student and spend much of my time writing code and running
experiments in Clojure. I do all of this in an environment of mixed
.org and .clj files. I find I prefer to write larger libraries directly
in .clj files, but then I often embed the snippets of code required for
running experiments, generating tables/graphs and analyzing experimental
results in code blocks embedded in Org-mode files. From these code
blocks I can either tangle the clojure code out into executable scripts,
or execute it /in situ/ in the .org file with the results dumped
directly into my org-mode buffer.
I find this to be a *very* comfortable research and development
environment, although as one of the main developers of the code block
support for Org-mode I'm certainly biased.
Cheers -- Eric
Eric Schulte
I'm confused as to what parts of LP practice are not supported by
Org-mode. Are you aware that Org-mode files can be exported to formats
more suitable for publication and human consumption (e.g. woven). See
http://orgmode.org/manual/Exporting.html
Tim Daly <da...@axiom-developer.org> writes:
> I looked at org-mode.
>
> Note that 'literate programming' involves writing literature
> for other people to read. The executable code is included as
> a 'reduction to practice' but the emphasis is on describing
> the ideas. Rich has some powerful ideas that he has reduced
> to running code. What we need to do is start with a description
> of the ideas and bridge the gap to the actual implementation.
>
> Ideally you can read a literate program like a novel, from
> beginning to end, and find that every line of code has a
> 'motivation' for being introduced. The side-effect is that
> there is a reason why the idea is implemented in a particular
> way rather than 'just because it worked'. Literate programming
> tends to improve code quality because you have to explain it.
>
> Emacs org-mode, on the other hand, is a useful development
> technology but it really isn't literate programming.
>
I would be interested to hear your thoughts as to why Org-mode is not a
literate programming tool.
Thanks -- Eric
Eric Schulte
Org-mode to work with code blocks (e.g. Literate Programming or
Reproducible Research) the online manual is also very useful.
http://orgmode.org/manual/Working-With-Source-Code.html
also, for a good review of Org-mode's support for the practices of
Literate Programming and Reproducible Research, see this draft
manuscript (currently in submission).
http://cs.unm.edu/~eschulte/org-paper/
both the .org source and the exported .pdf are available
Cheers -- Eric
Tim Daly
On 1/5/2011 7:37 PM, Eric Schulte wrote:
> Hi Tim,
>
> I'm confused as to what parts of LP practice are not supported by
> Org-mode. Are you aware that Org-mode files can be exported to formats
> more suitable for publication and human consumption (e.g. woven). See
> http://orgmode.org/manual/Exporting.html
I am truly impressed with the number of formats org-mode can
support. Of course, I expect nothing less as a heavy emacs user.
> Tim Daly<da...@axiom-developer.org> writes:
>
>> I looked at org-mode.
>>
>> Note that 'literate programming' involves writing literature
>> for other people to read. The executable code is included as
>> a 'reduction to practice' but the emphasis is on describing
>> the ideas. Rich has some powerful ideas that he has reduced
>> to running code. What we need to do is start with a description
>> of the ideas and bridge the gap to the actual implementation.
>>
>> Ideally you can read a literate program like a novel, from
>> beginning to end, and find that every line of code has a
>> 'motivation' for being introduced. The side-effect is that
>> there is a reason why the idea is implemented in a particular
>> way rather than 'just because it worked'. Literate programming
>> tends to improve code quality because you have to explain it.
>>
>> Emacs org-mode, on the other hand, is a useful development
>> technology but it really isn't literate programming.
>>
> I would be interested to hear your thoughts as to why Org-mode is not a
> literate programming tool.
I never said org-mode wasn't a 'literate programming tool'. It is clearly an
outstanding version of a literate programming tool. What I said was that
org-mode "really isn't literate programming".
I am trying to distinguish between tool and task.
Literate programming, as a tool, can be done with notepad.
Literate programming, as a task, is a radical change of mindset.
It is at least as difficult as going from Object Oriented programming
to Functional programming.
The point of the clojure.pamphlet file isn't to highlight how it
was created (emacs, fundamental mode). The point is to begin to
think about documentation as an "ideas to implementation", speaking
from "human to human", way of looking at the problem.
I made the machinery as simple as possible so people could experiment
with a new way of creating software. It is hardly new, and it isn't
my idea (see Knuth). I just have come to understand that it is a very
efficient and effective way to develop software that can "live".
Clojure breaks with the past in many ways. I am advocating breaking
with the past in terms of the 'little files' idea, 'javadoc', and
other ways of documenting. And, since Advocacy is Volunteering, I
pretty much put myself into a position where I had to demonstrate
what I was advocating. Thus, the Clojure in Small Pieces book.
Mark Engelberg
> For the most up-to-date and comprehensive documentation of using
> Org-mode to work with code blocks (e.g. Literate Programming or
> Reproducible Research) the online manual is also very useful.
In literate programming org-mode, will Clojure code be properly
highlighted and indented?
Is there a keystroke (like Ctrl-c,Ctrl-k) that will evaluate the
Clojure code in the entire file and send it to the swank REPL?
Will stacktraces point at the correct line number?
Tim Daly
It seems that your real question is whether Clojure knows about
a literate document. It does not. But it would be possible to
modify the reader behavior when given a pamphlet file. The REPL
uses a line numbering reader. Anything between the last
\end{chunk} and the next \begin{chunk} could be considered
as comments to be ignored but the line numbers for the function
would be correct and therefore the stack traces would be correct.
I suppose it would be reasonably easy (there is no such thing
as a simple job) to write a literate reader for the REPL. All
it would need to know is where to turn-on and turn-off the normal
read semantics.
When I get to documenting the REPL I will look at how reading
is done and think about writing a LiterateReader class.
Tim
Ken Wesson
If we had custom reader macros in Clojure we wouldn't even be having
this discussion; you would probably have already implemented it by
now. :)
Tim Daly
LispReader is a class that appears to have a read function that does
Clojure s-expression parsing. Wrapping that around a LiterateReader
stream would seem to do the job. The LiterateReader stream only has
to change any non-chunk line into a Clojure comment by prepending
a semicolon. The code to scan a line that begins with a \begin{chunk}
or \end{chunk} does not seem all that challenging.
In fact, a slightly smarter LiterateReader stream could be given the
particular line as an argument and only call LispReader on that
s-expression. So a LiterateReader with an optional line number
argument would allow an editor to specify where to start .read.
A reader macro would require special syntax. This may be reasonable
but it seems that a simple (load-literate "filename" N) would be
all that is needed, requiring no special syntax.
Alternatively the reader could use the file extension so that
pamphlet files would invoke the LiterateReader stream automatically.
Or if the parse of the first line begins with \ then use LiterateReader.
I'm documenting Bit-partitioned hash tries at the moment. I'll
see if I can document the reader next and get an idea of exactly
how it works and what it would take to change it.
Tim Daly
Seth
productivity in writing code that affected my statement that i liked
the little files. Im used to, i suppose, developing code for a
specific function in a file, being able to compile, goto line numbers
where there are errors,
send code to slime, etc. Looking over your example made things much
clearer. Its like your guiding your reader to specific parts of the
'little files', describing the theory behind them, moving on, etc. And
each code fragment has a chunk name associated with it, and all of
them are combined into the final .clj file using the code fragment
names (in a separate chunk).
At first, i thought this would be less productive than simply putting
all of the code in one clj file, but now that i think about it i think
it would, with the appropriate tools. And it wouldn't even be too
difficult, with org-mode (prefer it over latex any day!)
Im going to start transferring a subsection of my program to literate
programming, using org-mode. See how it goes...
Oh, and Tim, you might want to take a closer look at org-mode. Instead
of having to tangle out the code that builds everything, you could
create an executable shell script block in org-mode - the makefile
script could be tangled into a string using noweb syntax, and then
everything could go from there. You can execute the block by hitting C-
c c-c in the org file (or something like that). Pretty cool, in my
opinion!
Eric Schulte
> On Wed, Jan 5, 2011 at 4:44 PM, Eric Schulte <schult...@gmail.com> wrote:
>> For the most up-to-date and comprehensive documentation of using
>> Org-mode to work with code blocks (e.g. Literate Programming or
>> Reproducible Research) the online manual is also very useful.
>
> In literate programming org-mode, will Clojure code be properly
> highlighted and indented?
yes, Clojure code is displayed using the Emacs major mode for Clojure
code, so the appearance is as you would expect. See this screenshot
from the file I am working on at the moment, the upper frame is a .clj
file and the lower is a .org file. http://i.imgur.com/kdbDp.png
>
> Is there a keystroke (like Ctrl-c,Ctrl-k) that will evaluate the
> Clojure code in the entire file and send it to the swank REPL?
Yes, C-c C-c evaluates the code block under the point, and many other
keystrokes bind to various other functions [1], specifically C-c C-v b
executes the entire buffer.
> Will stacktraces point at the correct line number?
No, this is one of the reasons that I currently tend to do large-scale
development in .clj files and reserve embedded code for shorter chunks
of code. That said I have successfully completed large clojure projects
in which the entirety of the code was tangled from a single literate
.org file.
Cheers -- Eric
Footnotes:
[1] M-x org-babel-describe-bindings
Major Mode Bindings Starting With C-c C-v:
key binding
--- -------
C-c C-v a org-babel-sha1-hash
C-c C-v b org-babel-execute-buffer
C-c C-v d org-babel-demarcate-block
C-c C-v e org-babel-execute-maybe
C-c C-v f org-babel-tangle-file
C-c C-v g org-babel-goto-named-src-block
C-c C-v h org-babel-describe-bindings
C-c C-v i org-babel-lob-ingest
C-c C-v l org-babel-load-in-session
C-c C-v n org-babel-next-src-block
C-c C-v o org-babel-open-src-block-result
C-c C-v p org-babel-previous-src-block
C-c C-v r org-babel-goto-named-result
C-c C-v s org-babel-execute-subtree
C-c C-v t org-babel-tangle
C-c C-v u org-babel-goto-src-block-head
C-c C-v v org-babel-expand-src-block
C-c C-v x org-babel-do-key-sequence-in-edit-buffer
C-c C-v z org-babel-switch-to-session-with-code
Eric Schulte
>>> Emacs org-mode, on the other hand, is a useful development
>>> technology but it really isn't literate programming.
>>>
>> I would be interested to hear your thoughts as to why Org-mode is not a
>> literate programming tool.
> I never said org-mode wasn't a 'literate programming tool'. It is clearly an
> outstanding version of a literate programming tool. What I said was that
> org-mode "really isn't literate programming".
>
Ok, then it sounds like we're in agreement. I just was confused by the
use of "org-mode" as a verb to describe one particular task when it
supports many disparate tasks, including LP as you have defined it.
Thanks for the explanation.
Cheers -- Eric
Tim Daly
On 1/5/2011 10:58 PM, Seth wrote:
> Now that i think of it, it is mostly a fear of having decreased
> productivity in writing code that affected my statement that i liked
> the little files. Im used to, i suppose, developing code for a
> specific function in a file, being able to compile, goto line numbers
> where there are errors,
Try inserting a syntax error anywhere in the code. Then type
'make' to the shell. You'll get a traceback that shows you the exact
line in the file that failed since the literate document is really
feeding Java files to the compiler. This is forced by Java since
there is a (bogus) connection between filename and contents.
In any case, you still get the same traceback you always got.
> send code to slime, etc. Looking over your example made things much
> clearer. Its like your guiding your reader to specific parts of the
> 'little files', describing the theory behind them, moving on, etc. And
> each code fragment has a chunk name associated with it, and all of
> them are combined into the final .clj file using the code fragment
> names (in a separate chunk).
The REPL makes it much easier to develop lisp code in a literate
style because you can kill/yank an s-expression into a shell buffer
(or use slime).
My usual method of working is to build and test a function in the
REPL. Once it works I have the source already in the file so I can
just save it, build the whole system, run the tests, and make sure
I didn't break anything (it takes less than a minute).
> At first, i thought this would be less productive than simply putting
> all of the code in one clj file, but now that i think about it i think
> it would, with the appropriate tools. And it wouldn't even be too
> difficult, with org-mode (prefer it over latex any day!)
You see all of the source as one file. The compiler sees all of the
source in little files. The beauty of literate programming is that you
no longer have to pay attention to the compiler's failings.
> Im going to start transferring a subsection of my program to literate
> programming, using org-mode. See how it goes...
>
> Oh, and Tim, you might want to take a closer look at org-mode. Instead
> of having to tangle out the code that builds everything, you could
> create an executable shell script block in org-mode - the makefile
> script could be tangled into a string using noweb syntax, and then
> everything could go from there. You can execute the block by hitting C-
> c c-c in the org file (or something like that). Pretty cool, in my
> opinion!
I used noweb (ref: Norman Ramsey) for years in Axiom. It is a useful
tool. However I finally understood that I can get rid of the 'weave'
function with a couple latex macros I wrote (see a prior post) and
I could get rid of the 'tangle' function by modifying the reader.
I patched lisp and build tangle directly into the image.
Thus, with some simple changes I no longer need any special tools.
That makes life simple and I like simple.
Org-mode sounds great and from what I've seen from the docs it does
everything but cook rice. I would highly recommend it as a tool if you
like that sort of thing. It would integrate well in a slime environment
if you like that sort of thing.
ANYTHING that helps you write literate programs is a win in my book.
I'm afraid that I have two personal failing that make org-mode unlikely.
I don't like modes (My emacs mode table is smashed to be fundamental mode
for everything). Editors should not change my source code.
Thus, org-mode is "right-out", to quote Monty Python.
Second, I'm addicted to Latex. Latex is an amazing program, simply
stunning. I cannot imagine trying to write Clojure in Small Pieces
without it. It is just a markup language like HTML and thus not hard
to learn but it is also a turing complete language that has a huge
ecosystem of tools and techniques. I am creating the graphics for
CISP at the moment in Latex. I could do it using some other tool such
as gimp but life would not be as simple and the clojure.pamphlet file
would now need image files (more 'little files' cruft).
But whatever works for you is great. Please post an example of
a literate document using org-mode. We can then compare and contrast,
as my English teacher used to say. It would be interesting to see another
example of a literate document for Clojure. Slime and org-mode may
be the proper way to go.
Tim Daly
Tim Daly
On 1/5/2011 11:18 PM, Eric Schulte wrote:
> Mark Engelberg<mark.en...@gmail.com> writes:
>
>> On Wed, Jan 5, 2011 at 4:44 PM, Eric Schulte<schult...@gmail.com> wrote:
>>> For the most up-to-date and comprehensive documentation of using
>>> Org-mode to work with code blocks (e.g. Literate Programming or
>>> Reproducible Research) the online manual is also very useful.
>> In literate programming org-mode, will Clojure code be properly
>> highlighted and indented?
> yes, Clojure code is displayed using the Emacs major mode for Clojure
> code, so the appearance is as you would expect. See this screenshot
> from the file I am working on at the moment, the upper frame is a .clj
> file and the lower is a .org file. http://i.imgur.com/kdbDp.png
>
>> Is there a keystroke (like Ctrl-c,Ctrl-k) that will evaluate the
>> Clojure code in the entire file and send it to the swank REPL?
> Yes, C-c C-c evaluates the code block under the point, and many other
> keystrokes bind to various other functions [1], specifically C-c C-v b
> executes the entire buffer.
>
>> Will stacktraces point at the correct line number?
> No, this is one of the reasons that I currently tend to do large-scale
> development in .clj files and reserve embedded code for shorter chunks
> of code. That said I have successfully completed large clojure projects
> in which the entirety of the code was tangled from a single literate
> .org file.
Can you post examples of these? I'd love to see some other examples.
Tim Daly
So my comment was unclear. I so badly need an Editor-in-Chief to
scan my emails for clarity, punctuation, and other things literate
people ought have mastered by now. Sorry about that.
Eric Schulte
Sure thing, check out this old version of a file which tangles out into
the directory layout expected by lein.
http://gitweb.adaptive.cs.unm.edu/?p=asm.git;a=blob;f=asm.org;h=f043a8c8b0a917f58b62bdeac4c0dca441b8e2cb;hb=HEAD
Also, this project has an org-mode front page with code examples, the
html woven from this front page is shown at
http://repo.or.cz/w/neural-net.git
and the raw org file is available here
http://repo.or.cz/w/neural-net.git/blob/HEAD:/neural-net.org
I'll have to check out clojure.pamphlet, it sounds like an elegant
alternative. It's always interesting to see other solutions in this
space. For example I think scribble is a nice tool from the scheme
world. http://lambda-the-ultimate.org/node/4017
Best -- Eric
Tim Daly
On 1/6/2011 12:03 AM, Eric Schulte wrote:
>> Can you post examples of these? I'd love to see some other examples.
> Sure thing, check out this old version of a file which tangles out into
> the directory layout expected by lein.
> http://gitweb.adaptive.cs.unm.edu/?p=asm.git;a=blob;f=asm.org;h=f043a8c8b0a917f58b62bdeac4c0dca441b8e2cb;hb=HEAD
I see that this file is using chunk markup but I don't see
the "ideas-to-implementation" (I2I) explanation. That is, I
don't see text that is written so someone like me can read the
text, understand the ideas and how they translate to code.
> Also, this project has an org-mode front page with code examples, the
> html woven from this front page is shown at
> http://repo.or.cz/w/neural-net.git
> and the raw org file is available here
> http://repo.or.cz/w/neural-net.git/blob/HEAD:/neural-net.org
I love the graphics in this example! But, alas, this also seems
to be disconnected from the I2I property. But this is VERY
encouraging since it shows that you can get beautifully formatted
documents from the source.
> I'll have to check out clojure.pamphlet, it sounds like an elegant
> alternative. It's always interesting to see other solutions in this
> space. For example I think scribble is a nice tool from the scheme
> world. http://lambda-the-ultimate.org/node/4017
Take a look at http://daly.axiom-developer.org/clojure.pdf
The key difference, at least from my perspective, is that the CISP
is (well, is slowly being) organized around ideas. The goal is to
introduce the ideas and explain them in enough detail to *motivate*
the code that follows the explanation.
If you handed your printed document (or pdf) to someone will they
come away with enough understanding of neural networks to be able
to explain your code? Can they read the pdf and know what to change
and where to change it? Do they have literature references to follow
for deeper information? Is there a usable index with proper cross
references so they can hyperlink to the explanation and its associated
code block? These things seem, in my opinion, necessary (but not
sufficient) for the I2I property to hold.
I think that the tool you have is very, very nice. It shows me that
it would make a proper substitute for my beloved Latex as a viable
alternative for literate programming.
Tim Daly
Eric Schulte
> On 1/6/2011 12:03 AM, Eric Schulte wrote:
>>> Can you post examples of these? I'd love to see some other examples.
>> Sure thing, check out this old version of a file which tangles out into
>> the directory layout expected by lein.
>> http://gitweb.adaptive.cs.unm.edu/?p=asm.git;a=blob;f=asm.org;h=f043a8c8b0a917f58b62bdeac4c0dca441b8e2cb;hb=HEAD
> I see that this file is using chunk markup but I don't see
> the "ideas-to-implementation" (I2I) explanation. That is, I
> don't see text that is written so someone like me can read the
> text, understand the ideas and how they translate to code.
>
True, unfortunately I don't have any good examples where I write in an
I2I style. I think the discipline needed to sit down and document ones
initial ideas may be one of the largest hurdles to a wider adoption of
LP practices.
However there are more examples of org-mode usage available online at
http://orgmode.org/worg/org-contrib/babel/uses.html
>
>> Also, this project has an org-mode front page with code examples, the
>> html woven from this front page is shown at
>> http://repo.or.cz/w/neural-net.git
>> and the raw org file is available here
>> http://repo.or.cz/w/neural-net.git/blob/HEAD:/neural-net.org
> I love the graphics in this example! But, alas, this also seems
> to be disconnected from the I2I property. But this is VERY
> encouraging since it shows that you can get beautifully formatted
> documents from the source.
Thanks, you may also notice that the graphics are actually produced from
inline LaTeX (using tikz) avoiding the need for external binary files
and allowing everything to be handled nicely by git.
>
>> I'll have to check out clojure.pamphlet, it sounds like an elegant
>> alternative. It's always interesting to see other solutions in this
>> space. For example I think scribble is a nice tool from the scheme
>> world. http://lambda-the-ultimate.org/node/4017
> Take a look at http://daly.axiom-developer.org/clojure.pdf
>
Very nice, I think this has become my new reference for code in the
Clojure core. Is the .clj or .tex file available as well?
>
> The key difference, at least from my perspective, is that the CISP
> is (well, is slowly being) organized around ideas. The goal is to
> introduce the ideas and explain them in enough detail to *motivate*
> the code that follows the explanation.
>
> If you handed your printed document (or pdf) to someone will they
> come away with enough understanding of neural networks to be able
> to explain your code? Can they read the pdf and know what to change
> and where to change it? Do they have literature references to follow
> for deeper information? Is there a usable index with proper cross
> references so they can hyperlink to the explanation and its associated
> code block? These things seem, in my opinion, necessary (but not
> sufficient) for the I2I property to hold.
>
> I think that the tool you have is very, very nice. It shows me that
> it would make a proper substitute for my beloved Latex as a viable
> alternative for literate programming.
>
Thanks, and I agree the tool should support LP/I2I style composition,
the main hurdle is getting users (including myself) to write code in
that style (regardless of the tool used).
Best Regards -- Eric
>
> Tim Daly
Michael Wood
On 6 January 2011 07:33, Tim Daly <da...@axiom-developer.org> wrote:
[...]
> Take a look at http://daly.axiom-developer.org/clojure.pdf
I like it :)
Some simple corrections:
You have a typo on the front page:
"Based on Version 1.3.0-alphs4" (alphs4 instead of alpha4).
In the foreword, "This is a literate fork of Clojure, invented by Rich
Hickey." sounds a little like Rich Hickey invented this fork of
Clojure rather than Clojure itself.
I have no idea what that "((iHy))" means after the date at the end of
the foreword.
Page 2:
Last sentence of 1.3.2 says "We will look now look...".
In 1.3.3 the methods for updating the right member are identical to
the ones for the left member. I suspect "left" should be replace with
"right" in all three lines?
Page 5:
Fourth paragraph of 1.3.5 contains a sentence with a missing "a":
"Since it is copy operation...".
Also in 1.3.5, the list of Black node types duplicates the information
in 1.3.3. (Same with the list of Red nodes in 1.3.11.)
In the introductions to the Black, BlackVal, BlackBranch and
BlackBranchVal descriptions you say "This is constructed by
PersistenTreeMap's black method." Are all of them constructed with
the black method? If so, perhaps you should say that somewhere
instead of saying it separately for each one? Perhaps 1.3.10 is
enough. Just when I was reading through the introductions to Black,
BlackVal etc., I kept wondering if the next one was going to say it
was constructed by some method other than black, since you were
explicitly mentioning it every time.
Page 9:
There seems to be a spurious "new" in the first sentence: "If either
of the children exist but new have no value for this node..."
The second sentence of 1.3.11 is confusing to me. It seems to me that
the colour of the node returned has nothing to do with whether or not
subtrees are balanced, so it doesn't make sense to me to contrast
these two things. (Of course I know nothing about Red Black trees
other than what I've read so far in this document.)
The next paragraph is a little confusing to me too. Why would Red
nodes have to be returned only because the default colour is Black?
Surely even if the default colour were Red, Red nodes would need to be
returned? Perhaps I'm missing something fundamental about how Red
Black trees work.
Page 13:
1.3.16 seems to be more or less a copy of 1.3.10 with Black replaced
with Red, so it also has the extraneous "new".
Page 14:
The second sentence of 1.3.17 says "... all four cases all
balance...". It would be better without the second "all".
The next paragraph says "Each cases shows the ML code...". This
should of course be "Each case...". Also, what is "ML code"?
Page 16:
There's a missing "'s" in the second sentence: "... a black node (p5)
removeRight method..."
Anyway, that's as far as I've got so far.
--
Michael Wood <esio...@gmail.com>
Tim Daly
The source documents are
http://daly.axiom-developer.org/clojure.pamphlet
http://daly.axiom-developer.org/clojure.sty
The instructions for running tangle are in the preface.
The tangle.c function is in the document but it is standalone here:
http://daly.axiom-developer.org/tangle.c
A pamphlet file IS a tex file. I just chose the name to distinguish
it as a literate document from other tex files I have.
You might find it interesting to change the pamphlet to use org-mode
(or change org-mode to read pamphlets :-) )
Tim Daly
On 1/6/2011 12:07 PM, Michael Wood wrote:
> Hi
>
> On 6 January 2011 07:33, Tim Daly<da...@axiom-developer.org> wrote:
> [...]
>> Take a look at http://daly.axiom-developer.org/clojure.pdf
> I like it :)
>
> Some simple corrections:
>
> You have a typo on the front page:
>
> "Based on Version 1.3.0-alphs4" (alphs4 instead of alpha4).
>
> In the foreword, "This is a literate fork of Clojure, invented by Rich
> Hickey." sounds a little like Rich Hickey invented this fork of
> Clojure rather than Clojure itself.
You're right. Rich did not invent the fork. The spoon, perhaps.....
This is why projects need an Editor-in-Chief.
> I have no idea what that "((iHy))" means after the date at the end of
> the foreword.
It is an lisp joke between myself and certain others. Ignore it.
One of the struggles about these paragraphs is that I am trying
to reverse engineer Rich's code. There are no comments to guide the
connection between ideas and code. I have to stare at it and guess
his intentions and mindset. Ideally code would be "born literate"
but that is not the case here.
The posted version
> Page 13:
>
> 1.3.16 seems to be more or less a copy of 1.3.10 with Black replaced
> with Red, so it also has the extraneous "new".
I did a self-study of errors once and found that 50 percent of all
the errors I ever make are caused by copy-pasting. You think I would
learn but apparently I don't.
> Page 14:
>
> The second sentence of 1.3.17 says "... all four cases all
> balance...". It would be better without the second "all".
>
> The next paragraph says "Each cases shows the ML code...". This
> should of course be "Each case...". Also, what is "ML code"?
You're right. I should define all my abbreviations like ML before
they are used. I added a line to define it along with two literature
references.
> Page 16:
>
> There's a missing "'s" in the second sentence: "... a black node (p5)
> removeRight method..."
>
> Anyway, that's as far as I've got so far.
Very useful feedback. Thanks.
Tim Daly
Ken Wesson
Aside from the grammatical and spelling nitty-gritty, there are some
larger scale concerns, ones that unfortunately go to the heart of the
entire "literate programming" concept.
Namely, at the start it jumps very quickly into a red-black tree
implementation without really motivating it.
Now, it mentions wanting a binary search tree, and that a red-black
tree maintains certain invariants that limit how unbalanced it can
get, but the explicit motivation for this (avoiding the linear
worst-case performance of a fully unbalanced tree in favor of a tree
that gives guaranteed logarithmic bounds on its operations, other than
the obviously-linear-best-case full traversal of course).
More significantly, there's no explanation for introducing a binary
search tree in the first place -- and mention of Java but no direct
motivation for not using java.util.TreeMap or similar instead of
rolling our own. There's a brief mention of immutability and
persistence, without the connecting thoughts -- namely we want clojure
to have an immutable sorted-map type, and while we could use a factory
function that spits out (Collections/unmodifiableMap (doto (TreeMap.)
#(doseq [[k v] kvs] (.put % k v)))), every assoc would require copying
the entire map, in linear time and space, etc. etc. etc., vs. using a
"persistent" data structure specifically designed for structure
sharing with modified versions.
And then of course the overarching purpose of Clojure itself isn't
even given, let alone why it has sorted-map, and why it's an immutable
sorted-map ...
I know, I know, tall order, right? I hope you're at least thinking
about this stuff for the eventual final version, though. (And why
start with the red-black tree, anyway? After introducing Clojure I'd
probably start with getting a minimalist implementation that is
runnable first -- so the compiler and evaluator and repl and
s-expressions, lists and any other infrastructure these need under the
hood, and whatever else is needed for this -- some of the numerics
stuff, symbols and symbol interning, etc.; then with already a basic
Lisp environment in play, adding the other functions and data
structures in core that weren't needed just to get things up and
running with a basically functional repl. Maybe sorted-map (or a
persistent immutable binary search tree implemented in Java) is one of
the things that is needed to get the compiler working, but if so, I'd
think a truly literate version is going to explain what Clojure is,
and then how we're gonna implement it, and then details of the planned
compiler implementation, and why this needs a persistent immutable
binary search tree, and how that will also be useful later for making
the user-visible sorted-map and sorted-set features, and THEN get down
into the weeds of how we make a persistent immutable binary search
tree, how we can avoid the linear worst-time behavior of unbalanced
trees by ensuring the tree never gets too far out of balance (and even
what "balanced" means in this context), and that red-black trees are
among several kinds that have such properties, and then why red-black
trees were chosen over, say, two-three trees, etc.
Of course, you might need to ask Rich some questions to get some of
this information. It's pretty clear how a persistent immutable
balanced search tree is needed to make sorted-map and sorted-set work
well and in a pure-functional way; it may or may not be obvious on
close examination that a red-black tree is a superior choice over the
alternatives; maybe it isn't and Rich's choice was more or less
arbitrary, or based on what he already knew how to implement or had a
reference handy for; or Rich tried several alternatives and found that
he could make a Java implementation of a persistent, immutable
red-black tree faster than any of the others, or better at
structure-sharing; to really be sure, unless it really does become
obvious on looking into the code itself why, you'd have to ask Rich
why he chose red-black trees for this job. And there will probably be
many other similar points in the code where there's an implementation
choice that may not be obvious without Rich's input. And all of them
will need some explanation, if the program is to be truly literate --
even if there turn out to be cases where that explanation will be
something like "tests showed that X and Y were equally good
alternatives for implementing this, so Rich just flipped a coin and it
came up tails so he used Y". :)
Tim Daly
On 1/6/2011 9:15 PM, Ken Wesson wrote:
> I've now had a quick look at this too.
>
> Aside from the grammatical and spelling nitty-gritty, there are some
> larger scale concerns, ones that unfortunately go to the heart of the
> entire "literate programming" concept.
>
> Namely, at the start it jumps very quickly into a red-black tree
> implementation without really motivating it.
Actually the document was released because it contained a complete
working version of Clojure so it was the initial starting point.
I started on the Red Black tree section so I could show an example
of how to reorganize the code presentation so that it fit the
presentation in text rather than fitting the requirements of Java.
Red Black trees are mentioned in one of Rich's lectures as an
implementation detail so I decided to try and reverse engineer
their use.
Clearly there needs to be some discussion about Persistence and
Okasaki and Rich's research references motivating why they were
chosen.
I did this with the section on Bit-partitioned hash tries.
That version has not been published yet.
> Now, it mentions wanting a binary search tree, and that a red-black
> tree maintains certain invariants that limit how unbalanced it can
> get, but the explicit motivation for this (avoiding the linear
> worst-case performance of a fully unbalanced tree in favor of a tree
> that gives guaranteed logarithmic bounds on its operations, other than
> the obviously-linear-best-case full traversal of course).
>
> More significantly, there's no explanation for introducing a binary
> search tree in the first place -- and mention of Java but no direct
> motivation for not using java.util.TreeMap or similar instead of
> rolling our own. There's a brief mention of immutability and
> persistence, without the connecting thoughts -- namely we want clojure
> to have an immutable sorted-map type, and while we could use a factory
> function that spits out (Collections/unmodifiableMap (doto (TreeMap.)
> #(doseq [[k v] kvs] (.put % k v)))), every assoc would require copying
> the entire map, in linear time and space, etc. etc. etc., vs. using a
> "persistent" data structure specifically designed for structure
> sharing with modified versions.
>
> And then of course the overarching purpose of Clojure itself isn't
> even given, let alone why it has sorted-map, and why it's an immutable
> sorted-map ...
There is very little that needs to be said about the overarching
purpose of Clojure for the expected target audience. Since this
includes detailed source code in two languages it is reasonable to
assume that it is being read by a programmer and, more specifically,
it is being read by a programmer who is interested in the details
of Clojure's implementation.
> I know, I know, tall order, right? I hope you're at least thinking
> about this stuff for the eventual final version, though. (And why
> start with the red-black tree, anyway? After introducing Clojure I'd
> probably start with getting a minimalist implementation that is
> runnable first
The included implementation is runnable. I was not advocating
literate program at the introduction of Clojure 1.0 so the first
version is 1.3 which is not minimalist in any sense.
> -- so the compiler and evaluator and repl and
> s-expressions, lists and any other infrastructure these need under the
> hood, and whatever else is needed for this -- some of the numerics
> stuff, symbols and symbol interning, etc.; then with already a basic
> Lisp environment in play, adding the other functions and data
> structures in core that weren't needed just to get things up and
> running with a basically functional repl.
I wish Clojure was "factored" in terms of the implementation
to suit the I2I (ideas-to-implementation) approach but it isn't.
The literate goal here isn't to rewrite or refactor it but to
explain how the ideas are realized in code. A different goal
would be to factor Clojure in layers but that's someone else's book.
> Maybe sorted-map (or a
> persistent immutable binary search tree implemented in Java) is one of
> the things that is needed to get the compiler working, but if so, I'd
> think a truly literate version is going to explain what Clojure is,
> and then how we're gonna implement it, and then details of the planned
> compiler implementation, and why this needs a persistent immutable
> binary search tree, and how that will also be useful later for making
> the user-visible sorted-map and sorted-set features, and THEN get down
> into the weeds of how we make a persistent immutable binary search
> tree, how we can avoid the linear worst-time behavior of unbalanced
> trees by ensuring the tree never gets too far out of balance (and even
> what "balanced" means in this context), and that red-black trees are
> among several kinds that have such properties, and then why red-black
> trees were chosen over, say, two-three trees, etc.
>
> Of course, you might need to ask Rich some questions to get some of
> this information.
Rich obviously knows the answers to these questions but he is
a valuable and highly constrained resource. It would be nice if
he provided some motivating material but I think that we, as programmers,
can figure out how bit-partitioning works and how the code implements
it. Every new developer needs to do that to make changes. Literate
source code would decrease the time it takes to get up to speed.
So while it would be nice if Rich contributed I think he has better
things to do.
> It's pretty clear how a persistent immutable
> balanced search tree is needed to make sorted-map and sorted-set work
> well and in a pure-functional way; it may or may not be obvious on
> close examination that a red-black tree is a superior choice over the
> alternatives; maybe it isn't and Rich's choice was more or less
> arbitrary, or based on what he already knew how to implement or had a
> reference handy for; or Rich tried several alternatives and found that
> he could make a Java implementation of a persistent, immutable
> red-black tree faster than any of the others, or better at
> structure-sharing; to really be sure, unless it really does become
> obvious on looking into the code itself why, you'd have to ask Rich
> why he chose red-black trees for this job. And there will probably be
> many other similar points in the code where there's an implementation
> choice that may not be obvious without Rich's input. And all of them
> will need some explanation, if the program is to be truly literate --
> even if there turn out to be cases where that explanation will be
> something like "tests showed that X and Y were equally good
> alternatives for implementing this, so Rich just flipped a coin and it
> came up tails so he used Y". :)
>
Rich has provided a lot of discussion about his motivations
in his videos. There was a discussion a while back about providing
this same information in textual form and as a result of that
I ended up advocating literate programming.... which led to this.
I am hoping that showing an example of literate Clojure will
motivate others to pick a section (e.g. agents) and work through
the details. Once a section is written it can benefit from the
"many eyes" of open source to improve the style and accuracy
of presentation. In that way we all end up developing a document
that quickly brings up the average quality material for developers.
Tim Daly
Ken Wesson
> On 1/6/2011 9:15 PM, Ken Wesson wrote:
>> And then of course the overarching purpose of Clojure itself isn't
>> even given, let alone why it has sorted-map, and why it's an immutable
>> sorted-map ...
>
> There is very little that needs to be said about the overarching
> purpose of Clojure for the expected target audience. Since this
> includes detailed source code in two languages it is reasonable to
> assume that it is being read by a programmer and, more specifically,
> it is being read by a programmer who is interested in the details
> of Clojure's implementation.
Perhaps, but I think it might be worth considering broadening the
potential target audience to include people with some Java knowledge
that are curious about Clojure, but don't necessarily know much about
Clojure.
>> I know, I know, tall order, right? I hope you're at least thinking
>> about this stuff for the eventual final version, though. (And why
>> start with the red-black tree, anyway? After introducing Clojure I'd
>> probably start with getting a minimalist implementation that is
>> runnable first
>
> The included implementation is runnable. I was not advocating
> literate program at the introduction of Clojure 1.0 so the first
> version is 1.3 which is not minimalist in any sense.
Oh, I wasn't suggesting to leave anything out entirely; just to
organize the text to put the bootstrap-necessary material first, until
the code up to that point can get you a working repl with some
functionality, and then after that fleshing out the rest of the
feature-set and standard (clojure.core; contrib would be biting off
more than anyone could probably chew) library.
>> -- so the compiler and evaluator and repl and
>> s-expressions, lists and any other infrastructure these need under the
>> hood, and whatever else is needed for this -- some of the numerics
>> stuff, symbols and symbol interning, etc.; then with already a basic
>> Lisp environment in play, adding the other functions and data
>> structures in core that weren't needed just to get things up and
>> running with a basically functional repl.
>
> I wish Clojure was "factored" in terms of the implementation
> to suit the I2I (ideas-to-implementation) approach but it isn't.
I thought you had been saying all kinds of stuff about how your
literate-programming tools allow you to organize the material for the
benefit of human readers, rather than the requirements of the
compiling tools? This seems somewhat at odds with that.
>> Of course, you might need to ask Rich some questions to get some of
>> this information.
>
> Rich obviously knows the answers to these questions but he is
> a valuable and highly constrained resource. It would be nice if
> he provided some motivating material but I think that we, as programmers,
> can figure out how bit-partitioning works and how the code implements
> it. Every new developer needs to do that to make changes. Literate
> source code would decrease the time it takes to get up to speed.
> So while it would be nice if Rich contributed I think he has better
> things to do.
...
> Rich has provided a lot of discussion about his motivations
> in his videos.
Then those videos should prove a valuable resource for you.
> There was a discussion a while back about providing
> this same information in textual form and as a result of that
> I ended up advocating literate programming.... which led to this.
I seem to recall being involved in that discussion. :)
> I am hoping that showing an example of literate Clojure will
> motivate others to pick a section (e.g. agents) and work through
> the details. Once a section is written it can benefit from the
> "many eyes" of open source to improve the style and accuracy
> of presentation. In that way we all end up developing a document
> that quickly brings up the average quality material for developers.
That seems to me to suggest creating a wiki or similar for this
project. Why stop with half-measures? Let anyone contribute. Sections
can be marked as code-only, code and partial text, and code with
completed text; encourage people to flesh out the first two and
proofread and correct errors in the latter. Sections with "completed
text" would also have an uneditable "canonical version" that could be
added to the .pamphlet file, though still updated from time to time
with typo fixes and other corrections. (And the whole thing would
retain an uneditable history, like wikis commonly do, as a defense
against vandalism. Think of it as persistent, immutable text editing.
:))
There's also the need though for motivating sections that may not have
any code themselves but would preface some series of code-sections
related to implementing a particular larger element of the
architecture, such as the compiler, maps, or lazy sequences.
Any such wiki should probably allow such sections to be created, and
provide a discussion forum of some sort for debating which such
sections are needed, creations/deletions, any other wiki
administrivia, and the question of how to sequence and organize the
thing as a whole. TOC pages could exist that provide the organization
as editable sequences of wikilinks to smaller chunks of the thing.
Perhaps these should be the same as the motivating-sections, with the
motivating text at the top and the TOC of subsections at the bottom.
(In the .pamphlet file these sub-TOCs would be removed, of course,
leaving just the motivating text as chapter X section 1 and the
subsequent subsections as sections 2 onward of chapter X, and LaTeX
would generate its own, global TOC.)
Tim Daly
On 1/6/2011 11:42 PM, Ken Wesson wrote:
> On Thu, Jan 6, 2011 at 11:05 PM, Tim Daly<da...@axiom-developer.org> wrote:
>> On 1/6/2011 9:15 PM, Ken Wesson wrote:
>>> And then of course the overarching purpose of Clojure itself isn't
>>> even given, let alone why it has sorted-map, and why it's an immutable
>>> sorted-map ...
>> There is very little that needs to be said about the overarching
>> purpose of Clojure for the expected target audience. Since this
>> includes detailed source code in two languages it is reasonable to
>> assume that it is being read by a programmer and, more specifically,
>> it is being read by a programmer who is interested in the details
>> of Clojure's implementation.
> Perhaps, but I think it might be worth considering broadening the
> potential target audience to include people with some Java knowledge
> that are curious about Clojure, but don't necessarily know much about
> Clojure.
I'm open to including such a section at the beginning of the book.
I'm not a Java programmer though, I come from the Lisp side.
If you're willing to write it, we can add it (Advocacy... :-) )
Hmmm. I may have misunderstood your point. I thought you were suggesting
writing code that is not part of the distribution in order to get a
minimal running system and then working from that. If that is not what
you're suggesting then I'm confused.
>>> Of course, you might need to ask Rich some questions to get some of
>>> this information.
>> Rich obviously knows the answers to these questions but he is
>> a valuable and highly constrained resource. It would be nice if
>> he provided some motivating material but I think that we, as programmers,
>> can figure out how bit-partitioning works and how the code implements
>> it. Every new developer needs to do that to make changes. Literate
>> source code would decrease the time it takes to get up to speed.
>> So while it would be nice if Rich contributed I think he has better
>> things to do.
> ...
>> Rich has provided a lot of discussion about his motivations
>> in his videos.
> Then those videos should prove a valuable resource for you.
Yes, I am reviewing the video and slides so the explanation
is as close to Rich's message as possible. These are Rich's
ideas. The goal is to write them down.
>> There was a discussion a while back about providing
>> this same information in textual form and as a result of that
>> I ended up advocating literate programming.... which led to this.
> I seem to recall being involved in that discussion. :)
>
>> I am hoping that showing an example of literate Clojure will
>> motivate others to pick a section (e.g. agents) and work through
>> the details. Once a section is written it can benefit from the
>> "many eyes" of open source to improve the style and accuracy
>> of presentation. In that way we all end up developing a document
>> that quickly brings up the average quality material for developers.
> That seems to me to suggest creating a wiki or similar for this
> project. Why stop with half-measures? Let anyone contribute. Sections
> can be marked as code-only, code and partial text, and code with
> completed text; encourage people to flesh out the first two and
> proofread and correct errors in the latter. Sections with "completed
> text" would also have an uneditable "canonical version" that could be
> added to the .pamphlet file, though still updated from time to time
> with typo fixes and other corrections. (And the whole thing would
> retain an uneditable history, like wikis commonly do, as a defense
> against vandalism. Think of it as persistent, immutable text editing.
> :))
The pamphlet sources are in a git repository so they are immutable.
Wikis are fine for a lot of things but not for linearizing the
ideas into a readable literate form. Books fulfill that role.
> There's also the need though for motivating sections that may not have
> any code themselves but would preface some series of code-sections
> related to implementing a particular larger element of the
> architecture, such as the compiler, maps, or lazy sequences.
There is a non-code section on the topic of identity versus state
in the next version. If you send a Java section that would also be
non-code.
Ken Wesson
> Hmmm. I may have misunderstood your point. I thought you were suggesting
> writing code that is not part of the distribution in order to get a
> minimal running system and then working from that. If that is not what
> you're suggesting then I'm confused.
No, I was just suggesting that the order of the material put the stuff
in the distribution that's necessary to bootstrap a minimally
functional repl first, culminating in the eval function and the
command-line repl class, then flesh out the rest of Clojure's
feature-set with the rest of the stuff in the distribution. No new
code.
> The pamphlet sources are in a git repository so they are immutable.
>
> Wikis are fine for a lot of things but not for linearizing the
> ideas into a readable literate form. Books fulfill that role.
I suggested *maybe* letting the wiki users try to decide,
collectively, on a linearization; maybe that would prove workable and
maybe not. If not, you'd have to linearize it yourself to make the
book version. But if you're looking for section submissons and user
proof-reading a wiki can at least organize that activity, and can
provide "seeds" by having unwritten sections in there with just the
source code that is to be explained. And without potential
contributors maybe being put off by having to learn a whole extra set
of tools (namely, github and whatever client software) and get a login
at some site (github). Some might not have used git. A few might not
have used any code repository system. A wiki on the other hand can be
edited by anyone who can type stuff into a web form and can be
configured not to require a login (ala Wikipedia itself).
Robert McIntyre
useful for a clear explanation of PersistentVectors. Maybe even get
in touch with the guy for an addition to the book?
Thanks for your work on a literate clojure.
sincerely,
--Robert McIntyre
Tim Daly
It is a challenge to reverse-engineer someone else's code.
It is even more challenging to understand how it relates
to the algorithm and the idea. This will help a lot.
Tim Daly