Literate programming
da...@axiom-developer.org
traction. I would encourage you to bring examples. We can discuss the
merits and possibly gain some new insights. If nothing else, please
sign up for the Literate Software session at Clojure-Conj. I promise
to keep it short.
Literate programming can take various forms. I am working on a survey
of literate software. I came across an interesting non-latex example
worth sharing:
http://jashkenas.github.com/coffee-script/documentation/docs/nodes.html
Notice how well they have documented an apparently simple line as in:
exports.Base = class Base
The Base is an abstract base class for all nodes in the syntax tree.
Each subclass implements the compileNode method, which performs the
code generation for that node. To compile a node to JavaScript, call
compile on it, which wraps compileNode in some generic extra smarts,
to know when the generated code needs to be wrapping up in a
closure. An options hash is passed and cloned throughout, containing
information about the environment from higher in the tree (such as
if a returned value is being requested by the surrounding function),
information about the current scope, and indentation level.
Notice how this is not only giving trivial information (e.g. Base is
an abstract base class) but WHY it exists (..as a base for all nodes in
the syntax tree). It gives operational information (to compile a node..)
as well as information about the effect (..which wraps...). It shows
how global information is used (An options hash..) and WHY (containing
information about the environment...)
Code only tells you HOW something is done at the time it is done.
It's like having a recipe without an idea what you would make.
If our standards of documentation were raised to this level then large
systems like Axiom, Clojure, and ClojureScript would be much easier to
maintain and modify in the long term.
If you want your code to live beyond you, make it literate.
Tim Daly
da...@literatesoftware.com
jaime
no ideas about this stuff.
daly
http://en.wikipedia.org/wiki/Literate_programming
or the original paper
www.literateprogramming.com/knuthweb.pdf
or from the master himself:
http://www-cs-faculty.stanford.edu/~uno/lp.html
Tim Daly
Mark Engelberg
I recall that at some point you described your setup for doing Clojure
literate programming, and if I recall correctly, you were primarily
working in LaTeX, relying on incremental compilation to test little
snippets of code as you wrote them.
Have there been any new developments on the literate programming /
Clojure front, in terms of tools that leverage existing build tools,
test suites, generating meaningful line numbers for stack traces and
debugging?
daly
> Tim,
>
> I recall that at some point you described your setup for doing Clojure
> literate programming, and if I recall correctly, you were primarily
> working in LaTeX, relying on incremental compilation to test little
> snippets of code as you wrote them.
Yes, the idea is to write a literate version of Clojure, see
http://daly.literatesoftware.com/clojure.pamphlet
http://daly.literatesoftware.com/clojure.pdf
similar in style to "Lisp in Small Pieces". (The effort has been
stalled temporarily while I try to find new employment.)
>From the above document you extract the Makefile, type 'make'
and end up with the newly formatted PDF and a running REPL.
So any edits to the document text are immediately reflected in
the PDF and any edits to the code are immediately reflected in
the compiled results.
>
> Have there been any new developments on the literate programming /
> Clojure front, in terms of tools that leverage existing build tools,
> test suites, generating meaningful line numbers for stack traces and
> debugging?
>
The question is ill-posed in sense that literate programming is not
a tool or technology but a change in mindset. To quote Knuth:
I believe that the time is ripe for significantly better documentation
of programs, and that we can best achieve this by considering programs
to be works of literature. Hence, my title "Literate Programming".
Let us change our traditional attitude to the construction of
programs: Instead of imagining that our main task is to instruct
a computer what to do, let us concentrate rather on explaining to
human beings what we want a computer to do.
-- Donald Knuth "Literate Programming (1984) CSLI 1992 pg. 99
So imagine a world where the eloquence of Rich Hickey was expressed in
book form. Imagine the whole immutability lecture written down and
decorated with the actual running code as illustration. Imagine that
the original authors clearly explained software transactional memory
and illustrated it with the actual code. Imagine a discussion of
argument destructuring with running code illustrations.
How many more people would be able to dive into the details of Clojure
to maintain and modify the code? How many people would find it much
easier to understand prototypes, defrecords, macros, and all of the
other struggles that populate the current email forums? How are
infinite sequences supported and what code supports it? What does
NIL mean and why was it defined that way?
We do not capture the ideas. We do not rewrite the explanations to
clarify the fine points of confusion. We do not illustrate how the
ideas of Red-Black tries are moved "from ideas to implementation".
We do not communicate the ideas to each other. We code for the machine.
So we end up with the traditional "tree of sand" pile of little files.
In order to overcome this we construct tools (IDEs) that know how to
navigate this pile, tools to store the changes (SVN, git, hg), tools
to put the sandpiles together (#include), tools to build guardrails,
tools, tools, tools... (You can, of course, keep each chapter and
section and subsection in separate files and "include" them in the
book.)
Now imagine that the "book" is multimedia where you can include
animation (watch the tree balance), video, and a running REPL.
Think Khan Academy for Clojure.
This community is willing to overthrow the traditional ideas of lisp
in order to make progress on a new path. Yet we continue to structure
the development as though we worked on a PDP 11/40 with 4k file size
limits. We continue to let the most valuable information which is the
meat of Clojure disappear while we keep only the code, the bones of
Clojure.
Open the source code. Stare at it. Ask yourself if you understand
exactly why it was needed, why it is structured that way, what would
happen if you changed it and what else depends on this code. Imagine
your job is to maintain and modify it but Rich is not available for
questions and answers.
Ultimately that is what matters. In the long term the code will be
the only remaining artifact after Rich leaves the project. Look at
Sourceforge and you will see thousands of dead projects that will
never be picked up because they are just trees of code, dead code.
Et tu, Clojure?
Literate programming is about making code live.
I like Clojure and I really want it to live.
Think long term. Imagine a better way.
Tim Daly
Daniel Jomphe
On Thursday, October 27, 2011 1:58:52 PM UTC-4, TimDaly wrote:
So imagine a world where the eloquence of Rich Hickey was expressed in
book form. Imagine the whole immutability lecture written down and
decorated with the actual running code as illustration. Imagine that
the original authors clearly explained software transactional memory
and illustrated it with the actual code. Imagine a discussion of
argument destructuring with running code illustrations.How many more people would be able to dive into the details of Clojure
to maintain and modify the code? How many people would find it much
easier to understand prototypes, defrecords, macros, and all of the
other struggles that populate the current email forums? How are
infinite sequences supported and what code supports it? What does
NIL mean and why was it defined that way?We do not capture the ideas. We do not rewrite the explanations to
clarify the fine points of confusion. We do not illustrate how the
ideas of Red-Black tries are moved "from ideas to implementation".
We do not communicate the ideas to each other. We code for the machine.
bernardH
the only lurker on this list that you manage to get interested in LP.
I wanted to share two thought that I had when looking at your LP take
on Clojure code :
1°) The infrsastructure should really lower the barrier to
participation as wiki did for wikipedia.
I gain some insight into the persistent data structures in Clojure
implementation as I ported persistent vector to C++ : I wish I could
just jump in and add that insight as a few paragraphs. Programs
nowadays can be massive hence the need of collaborative developpement.
A github account would be more appropriate imho that a http server.
2°) It was my understanding that Clojure implementation was both large
and poised for some overhaul (Clojure in Clojure ?). On the other
hand, I heard that Clojurescript implementation was damn small and
maybe the experience carried from Clojure implementation means that
its foundation will last : maybe it could be an opportunity for a LP
project ?
Best Regards,
B
PS: I'm seeing awsome potential in LP as a teaching tool ! I long for
the day where I can direct my students to LP projects to help them
understand the tought processes that went into great programs.
On Oct 27, 7:58 pm, daly <d...@axiom-developer.org> wrote:
> On Thu, 2011-10-27 at 00:17 -0700, Mark Engelberg wrote:
> > Tim,
>
> > I recall that at some point you described your setup for doing Clojure
> > literate programming, and if I recall correctly, you were primarily
> > working in LaTeX, relying on incremental compilation to test little
> > snippets of code as you wrote them.
>
> stalled temporarily while I try to find new employment.)
>
> exactly why it was needed, why it is structured that way, what would
> happen if you changed it and what else depends on this code. Imagine
> your job is to maintain and modify it but Rich is not available for
> questions and answers.
>
> Ultimately that is what matters. In the long term the code will be
> the only remaining artifact after Rich leaves the project. Look at
> Sourceforge and you will see thousands of dead projects that will
> never be picked up because they are just trees of code, dead code.
> Et tu, Clojure?
>
> Literate programming is about making code live.
> I like Clojure and I really want it to live.
>
> Think long term. Imagine a better way.
>
> Tim Daly
the only lurker on this list that you manage to get interested in LP.
I wanted to share two thought that I had when looking at your LP take
on Clojure code :
1°) The infrsastructure should really lower the barrier to
participation as wiki did for wikipedia.
I gain some insight into the persistent data structures in Clojure
implementation as I ported persistent vector to C++ : I wish I could
just jump in and add that insight as a few paragraphs. Programs
nowadays can be massive hence the need of collaborative developpement.
A github account would be more appropriate imho that a http server.
2°) It was my understanding that Clojure implementation was both large
and poised for some overhaul (Clojure in Clojure ?). On the other
hand, I heard that Clojurescript implementation was damn small and
maybe the experience carried from Clojure implementation means that
its foundation will last : maybe it could be an opportunity for a LP
project ?
Best Regards,
B
PS: I'm seeing awsome potential in LP as a teaching tool ! I long for
the day where I can direct my students to LP projects to help them
understand the tought processes that went into great programs.
PS2: I'd be willing to help fund LP work on Clojure[Script], now that
donations to Clojure are not possible anymore .
Larry Johnson
Larry
--
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clo...@googlegroups.com
Note that posts from new members are moderated - please be patient with your first post.
To unsubscribe from this group, send email to
clojure+u...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
--
Off the Beaten Path in Technology
http://otbeatenpath.wordpress.com
Tim Sally
http://www.cs.tufts.edu/~nr/cs257/archive/don-knuth/pearls-1.pdf
Larry Johnson
I haven't been working with it for awhile, but I did a somewhat primitive modification to the XML Docbook markup language (I just added a few appropriate tags for "tangling" the executable source code, and "weaving" the well formatted article documenting the code) which I used as the source language, then wrote a tangle and weave in perl. I got the idea from Norman Walsh's article Literate Programming in XML which can be found at http://nwalsh.com/docs/articles/xml2002/lp/paper.html
The advantage of this was that given the array of tools for rendering Docbook "weaving" was a piece of cake, and perl had a good range of modules for doing the "tangle".
As I stated, I'm very new to clojure, but I've always been fascinated with LP, and I'm very happy to see this discussion going on here.
Larry
--
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clo...@googlegroups.com
Note that posts from new members are moderated - please be patient with your first post.
To unsubscribe from this group, send email to
clojure+u...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
daly
On Thu, 2011-10-27 at 20:11 -0400, Larry Johnson wrote:
> My two favorite articles on Literate Programming are both from Donald
> Knuth's book Literate Programming. One is "Computer Programming as an
> Art", and the other is "Literate Programming". When I was preparing
> to interview Knuth a bit over a year ago I re-read the entire book. I
> expected it to be a somewhat outdated description of WEB, TANGLE, and
> WEAVE. On the contrary it was wonderfully timeless. When I mentioned
> that to Knuth he sort of grumbled something to the effect of "Well,
> yes, some things in computer science have a long shelf life" (that's a
> paraphrase, but it was something like that).
Knuth's invention of literate programming is one of the unrecognized
pearls of computer science. I have yet to see a programming team that
has an Editor-in-Chief who does patch-review for clarity, sentence
structure, punctuation, relevance, location in the book, proper
citations and index terms, etc. Maybe someday.
>
> I haven't been working with it for awhile, but I did a somewhat
> primitive modification to the XML Docbook markup language (I just
> added a few appropriate tags for "tangling" the executable source
> code, and "weaving" the well formatted article documenting the code)
> which I used as the source language, then wrote a tangle and weave in
> perl. I got the idea from Norman Walsh's article Literate Programming
> in XML which can be found at
> http://nwalsh.com/docs/articles/xml2002/lp/paper.html
>
> The advantage of this was that given the array of tools for rendering
> Docbook "weaving" was a piece of cake, and perl had a good range of
> modules for doing the "tangle".
Any means of publication can be the medium for literate programming.
As I rule I prefer Latex but anything will do.
All you need is a distinguished means of quoting and naming the
chunks. In html this could be as simple as:
<pre id="somename">
your code
</pre>
and you need a program, often called "tangle", to extract the chunk
tangle mywebpage.html somename >mysomename.file
The machinery of literate programming is dirt simple.
Poof! You're done.
The hardest part of literate programming is the mindset.
In order to do literate programming you need to change your focus
from traditional programming to writing for humans and, as a side
effect, writing for the machine.
>
> As I stated, I'm very new to clojure, but I've always been fascinated
> with LP, and I'm very happy to see this discussion going on here.
Java has taken the PDP 11/40 sand files to their logical extreme where
we have wired the name of the tiny file to the name of the tiny object.
We have packaged the automobile into a crate with labeled bags of
screws, hoses, switches, etc. and are expected to understand the car.
Clojure is a very early adoptive, open minded community willing to
challenge old assumptions. It helps to highlight what those old
assumptions are, as Rich has done for Lisp, since they can be
difficult to see and hard to displace. I am hoping the community
will disrupt the tiny-files, javadoc, IDE, code-for-the-machine,
mindset and start communicating with humans. Rich has great ideas
in Clojure and all we get is the dried bones of source code.
We can do so much better.
Tim Daly
Robert McIntyre
in clojure.
http://orgmode.org/
git://orgmode.org/org-mode.git
Here are some things I've used it for:
http://aurellem.com/abomination/html/no_parens.html
http://aurellem.com/pokemon-types/html/lpsolve.html
http://aurellem.com/thoughts/html/man-years.html
It's possible to automatically monitor the org files in a project,
regenerate the html and source code whenever they are changed, and get
a very tight write-debug-test cycle. Try it out it's fun!
sincerely,
--Robert McIntyre
Stefan Kamphausen
just to be sure: are you are aware of Marginalia?
https://github.com/fogus/marginalia
Regards,
Stefan
Michael Jaaka
Larry Johnson
Any means of publication can be the medium for literate programming.
As I rule I prefer Latex but anything will do.
All you need is a distinguished means of quoting and naming the
chunks. In html this could be as simple as:
<pre id="somename">
your code
</pre>
and you need a program, often called "tangle", to extract the chunk
tangle mywebpage.html somename >mysomename.file
Yep. That's all my Rube Goldberg-ish system did. The chunk of machine recognizable code was named as an attribute in a modified <sourcecode> tag. The "weave" just stripped it out so you were left with "typeset ready" Docbook markup. The "tangle" just yanked out the target source code and wrote it to a file in the proper order.
For programmers who are resistant to even commenting code LP must seem nightmarish (at least at first). Not only are you required to provide comments, but those comments have to be a complete, well formatted, human readable, journal article. Also, it takes some time getting one's head wrapped around the fact that the order in which the target language source code appears in the article is not necessarily the order in which the machine needs it for compilation and execution purposes.
When LP is practiced consistently it prevents a number of bad habits, and I can't see that those good effects change whether one is using a "top down" or "bottom up" approach to programming. Since you're writing a complete, well formatted, human readable, article on how the code works, you can't just cut-and-paste code into a project without really understanding what the code is doing (well, I guess you can, but that would become obvious while you or another reader were trying to follow the article).
Tassilo Horn
while I agree that good documentation is important for maintaining and
developing further a given code base, I always wonder how literate
programming deals with refactoring and larger restructuring. I mean, in
basically all software projects I'm involved in, developers have a hard
time in keeping at least the most integral documentation up-to-date,
e.g., docstrings, JavaDocs, etc.
Now I consider a some literate code base, i.e., a book that describes
the complete system in a very detailed manner including the reasons to
design it that way. That's awesome for developers joining the project,
but doesn't it hinder any further development?
For example, I'm currently reworking some paper that includes code for
some model transformation. If I had the right tools, I could extract
the source code from the latex files and have a running transformation.
But now, I had to do some minor modifications to the code which took me
about 5 minutes, but adapting the surrounding text takes me hours over
hours. I think, that's an experience many people have made: changing
source code of a system that is well-designed and understood by the
developer is much easier to do than changing a cohesive text in natural
language. In code, I have an automated, picky lector that always reads
the complete "book" after each change I make (also known as compiler).
In plain text, only a human can verify consistency and only in very
narrow borders.
Well, that's more text than I intended to write. ;-) So short story
long: how do you manage conistency of docs and code in literate code
bases?
Bye,
Tassilo
daly
Hard work? Book authors face the same problem. Every new change to
Clojure makes more work for the authors.
One side-effect of Literate Programming is that you re-arrange your
code to fit the presentation. As a result you bring together chunks
that are all related (e.g. the code to support Red-Black tries).
I would claim that refactoring "in the large" is unlikely in a
literate program because you have to think through the code in order
to explain it clearly. But if you do a large refactor then it is
just hard work. For small refactorings you might not have to change
the explanation at all.
Clojure is pretty well "firmed up" at this point. I don't know if
there is going to be a large rewrite of the fundamentals.
Tim Daly
Mark Engelberg
>> Have there been any new developments on the literate programming /
>> Clojure front, in terms of tools that leverage existing build tools,
>> test suites, generating meaningful line numbers for stack traces and
>> debugging?
>>
>
> The question is ill-posed in sense that literate programming is not
> a tool or technology but a change in mindset. To quote Knuth:
I think my question is legitimate.
To take an extreme example, if you tell me literate programming is
great, but that I have to write out my programs by hand on paper and
scan them into a computer, I'd laugh at you. Why would I want to give
up the convenience of typing in my programs via keyboard to do
literate programming?
So I'd like an honest appraisal of the state of the art in literate
programming of Clojure -- what tools and conveniences do I have to
give up to do LP. Do I have to give up things like meaningful line
numbers when I get an error message?
daly
> On Thu, Oct 27, 2011 at 10:58 AM, daly <da...@axiom-developer.org> wrote:
> >> Have there been any new developments on the literate programming /
> >> Clojure front, in terms of tools that leverage existing build tools,
> >> test suites, generating meaningful line numbers for stack traces and
> >> debugging?
> >>
> >
> > The question is ill-posed in sense that literate programming is not
> > a tool or technology but a change in mindset. To quote Knuth:
>
> I think my question is legitimate.
>
> To take an extreme example, if you tell me literate programming is
> great, but that I have to write out my programs by hand on paper and
> scan them into a computer, I'd laugh at you. Why would I want to give
> up the convenience of typing in my programs via keyboard to do
> literate programming?
I'm not sure what tools you use for development. Straight emacs is
sufficient for my development. Emacs doesn't care what style is used
for development.
>
> So I'd like an honest appraisal of the state of the art in literate
> programming of Clojure -- what tools and conveniences do I have to
> give up to do LP. Do I have to give up things like meaningful line
> numbers when I get an error message?
>
State of the art in literate programming in Clojure?
Hmmm. The pamphlet link is my only example of literate programming
related to Clojure. In order to develop you edit the pamphlet, type
"make" at a command line, and the world is rebuilt. Since the code is
extracted into the same source tree as it originally had, the line
numbers will be related to the file and line as before.
My development style involves changing about 10 lines of code or less
followed by a complete system rebuild and test cycle. Overlapping
rebuild and test is time for writing the literate explanations.
Almost always, when something fails it is perfectly clear what lines
caused the error. When that isn't the case then a little "navel
debugging" (contemplating what might be wrong) is used.
I don't use IDEs but I suppose it would be possible to write a
literate plugin of some kind that ignored things that were not in
code chunks. Line numbers could then be interpreted with respect
to the start of the chunk. Most IDEs allow plugins.
To your point, though, it is true that we may have reached a local
optimum of tool development for tiny files. It might be necessary
to move away from this peak in order to climb a different hill.
This would imply making things "worse" for development until the
tools adapt. Clojure moved off the local optimum of common lisp
and broke all my code-walking software, violated quite a few of my
assumptions, and invalidated my manuals but that isn't an argument
for abandoning Clojure.
You might feel that the current toolset is optimal and that
literate programming is not worth the time and effort.
De gustibus non disputandum. (There is no disputing taste).
A literate Clojure would only be of benefit to those who come after.
All it offers now is a lot of time and effort to communicate ideas
to people we will never meet. However the thing that draws people
to Clojure is the ideas, not the code.
We can debate it at length over beers at the Conj.
Tim Daly
Damion Junk
Stefan Kamphausen
Clojure moved off the local optimum of common lisp
I can't help, but I really like that quote. Nice meme :-)
Stefan
Adam Getchell
I have also been using Emacs/Org-mode/Babel/R lately, mostly as a way to have easily modifiable write up and source code for assignments in statistics courses. I suppose this is one valid use, but I'm using it less to communicate code meaning and more as a convenient way to perform analysis, look at the results, and talk about them all in the same place, and then to easily generate documentation of my efforts [LaTeX output].
Damion
"Invincibility is in oneself, vulnerability in the opponent." -- Sun Tzu
daly
>
>
> On Fri, Oct 28, 2011 at 2:45 PM, Damion Junk <jun...@gmail.com> wrote:
> I have also been using Emacs/Org-mode/Babel/R lately, mostly
> as a way to have easily modifiable write up and source code
> for assignments in statistics courses. I suppose this is one
> valid use, but I'm using it less to communicate code meaning
> and more as a convenient way to perform analysis, look at the
> results, and talk about them all in the same place, and then
> to easily generate documentation of my efforts [LaTeX output].
>
>
> I'm interested in how you're including LaTeX in your code, as I'm
> writing models which require LaTeX to express what they're modelling.
>
>
> BTW I looked at marginalia+leiningen, which seems to work for my
> purposes except I'm unclear where to put the .tex files.
I'm not sure what this sentence means. The .tex files ARE the
literate program. By analogy, you seem to be asking something like
"I'm writing a book but I don't know where to put the text".
>
>
> The combination of literate + TDD seems forbidding.
Are you finding it hard to explain why you wrote a test?
Tim Daly
Larry Johnson
Are you finding it hard to explain why you wrote a test?
>
> The combination of literate + TDD seems forbidding.
Tim Daly
I decided awhile back when trying to answer questions about literate programming, that people get caught up in the moving parts, and not in what the approach actually yields. Your statement above puts it nicely and succinctly, and hearkens back to Knuths original articles. Lately I emphasize the woven text (without inititially calling it that) and ask the person I'm talking with to imagine writing an article or book about their code, how it works, with proofs where appropriate. That article should be written as a work of literature. Not all literature has to be Hugo or Melville (or Jack Kerouac, or Gertrude Stein for that matter). Some programs are more appropriately Mickey Spillane or Terry Pratchett, or even in the style of a manual for an electric razor. The point is that it should be satisfying to read and comprehensively informative.
Test code, and descriptions of external libraries are no different from any other sections of the article or book. The most important thing is to introduce them into the work at point most conducive to the reader's understanding.
I really believe that there's no programming or engineering methodology which doesn't lend itself to literate programming. If it can be described, it can be presented in the form of an article. If it can't be describe in human language it's probably terrible code.
Larry
daly
>
>
> On Thu, Dec 22, 2011 at 2:54 PM, daly <da...@axiom-developer.org>
> wrote:
>
> >
> > The combination of literate + TDD seems forbidding.
>
>
> Are you finding it hard to explain why you wrote a test?
>
> Tim Daly
>
>
>
> I decided awhile back when trying to answer questions about literate
> programming, that people get caught up in the moving parts, and not in
> what the approach actually yields. Your statement above puts it
> nicely and succinctly, and hearkens back to Knuths original articles.
> Lately I emphasize the woven text (without inititially calling it
> that)
Untangling woven text is trivial. It can be done in any language. See
http://axiom-developer.org/axiom-website/litprog.html
> and ask the person I'm talking with to imagine writing an article or
> book about their code, how it works, with proofs where appropriate.
> That article should be written as a work of literature. Not all
> literature has to be Hugo or Melville (or Jack Kerouac, or Gertrude
> Stein for that matter). Some programs are more appropriately Mickey
> Spillane or Terry Pratchett, or even in the style of a manual for an
> electric razor. The point is that it should be satisfying to read and
> comprehensively informative.
>
> Test code, and descriptions of external libraries are no different
> from any other sections of the article or book. The most important
> thing is to introduce them into the work at point most conducive to
> the reader's understanding.
If a test is testing something worthwhile, as opposed to the "blindly
testing that 3=3" mindset, then it probably only takes a couple lines
to motivate why you need the test. A test paragraph could contain the
details. So if you're writing a hash table implementation, the test
code of overflow chaining could explain what problems might occur (e.g.
what happens when the table is actually full? How do you detect that
the overflow chain has looped?, etc.)
So if you wrote the hash table implementation code in literate form
then each test could explain (and test) a particular boundary case or
issue that might arise.
This has the effect of motivating tests that are detailed, based on
real issues, and handle boundary cases clearly.
>
> I really believe that there's no programming or engineering
> methodology which doesn't lend itself to literate programming. If it
> can be described, it can be presented in the form of an article. If
> it can't be describe in human language it's probably terrible code.
Test cases don't have to appear as "test cases". They appear in the
text as either example code (useful for the reader but also testing)
or as explanations of possible failures, misuse, misapplication, or
boundary cases.
This is especially powerful if you focus your tests on
precondition/postcondition testing. See Hoare's paper
"An Axiomatic Basis for Computer Programming"
http://citeseer.ist.psu.edu/viewdoc/summary?doi=10.1.1.116.2392
Using Hoare's ideas to focus the development of your test cases means
that your tests have a good theoretical basis. They check for the
conditions that should survive refactoring. Requiring precondition
postcondition tests, along with clearly written natural language
explanations, could be part of the peer code review prior to
accepting new or revised code into the main line.
Better yet, in a literate program, you can explain what the invariant
is, why it is important, and what breaks when you violate it. So
testing now has a good formal basis rather than random 3=3 tests.
We can be so much better programmers.
Raise your game.
Be literate.
Tim Daly
Larry Johnson
nchurch
think; and it helps me to "program" other people to think like me.
But I would never have learned English unless along the way it gave me
near-term results. It should follow, then, that telling people to
write literate programs for the sake of posterity is never going to be
enough to change mindsets. Taking an illiterate [sic] program and
making it literate as you are doing with Clojure will be a tour-de-
force, and an intimidating one at that. I think that making literate
programming incrementally useful, however, is \inevitably going to
involve tooling.
Of two kinds, if you follow the logic above: social, and personal.
Firstly, there really needs to be something like a Github for literate
programming. Why? Because a literate program will be easier to
\understand, but also easier to \find (see example below). Imagine a
sort of "function search engine"----code liberated from namespaces and
leiningen projects, annotated and independently useable. If there
were a community that actually \took \advantage of findability and
understandability, it would greatly speed coding (think of how much
time you have to spend hunting for documentation), and in the process
not only \promote literate programming but make it \advantageous.
On the personal side, writing English as part of coding has to be
maximally efficient for us humans: no retyping. The identifiers we
use in English should flow into the code, and vice-versa. Take the
following example:
To CAPITALIZE EVERY WORD in a STRING:
Split the STRING on SPACE into WORDS;
CAPITALIZE the first letter of every WORD, producing CAPITALIZED
WORDS.
Re-assemble[JOIN] the string on SPACE, to make the CAPITALIZED
STRING.
Using functions from clojure.string: split, join, capitalize; see
documentation:
http://docs.oracle.com/javase/1.5.0/docs/api/java/util/regex/Pattern.html
http://clojure.org/other_functions
http://www.fatvat.co.uk/2009/01/regular-expressions-in-clojure.html
http://clojure.github.com/clojure/clojure.string-api.html
(defn capitalize-every-word [m-string]
(let [m-space " "
regex-space #"\s"
words (split m-string regex-space)
capitalized-words (map capitalize words)
capitalized-string (join m-space capitalized-words)]
capitalized-string))
(defn c-e-w [m-string]
(join " " (map capitalize (split m-string #"\s"))))
Now there is actually not that much novel text here, and typing it out
was an enlightening exercise (to really see how enlightening, you'd
have to see the full example; but I've omitted it because the
Internets are already complaining about a long post not written in
their native Twitterese). But it was very cumbersome and time-
consuming to do it. This cumbersomeness was incidental: because my
activities were split among editing, browsing, and coding, for one
thing; and also (not a complaint, Rich!) because our programming
languages do not incorporate basic English concepts like abbreviation
and definition-in-place.
(One of the big advantages of writing English along with code is that
you come up with sensible identifiers. Without English, it's a
puzzling exercise done completely out of context.)
You can't tell me this isn't a problem for IDE's to handle. At a
minimum, I want my IDE to link up the identifiers between code and
English. If they do, look at what happens: the identifiers get
enriched by natural language context (notice, for instance, how JOIN
gets associated with 'assemble'), and if we push everything to our
social tool, the code ends up with more paths to be located by.
We don't yet have Google-level findability for code, of course; but I
would argue that this is not a technological defect, it is simply a
failure of description! Once we have literate code, Google will build
a function search engine for us.
daly
> Firstly, there really needs to be something like a Github for literate
> programming.
What a great idea!
I'll see what I can do.
Tim Daly
nchurch
might guess); but I haven't demonstrated myself to be a great coder,
yet. I feel like I'm a coder who needs something like literate
programming to be great, so it's kind of a chicken-and-egg problem.
I'm already partway there with the existence of Clojure, but although
it's the most intelligent language I've every come across (and it is
at least Lisp), it still isn't enough.
nchurch
that we might want to go just a little beyond Github to the way we
write code itself. Look again at the capitalize-every-word function I
defined above. In my original unshortened commentary, I included also
this:
.............
To split a string into words (in the simplest case):
Split the string on a separator, the canonical one being space, but
also including punctuation characters such as ",';:./(){}-+_|\
BUT: Re-assembling the string will not reproduce the old string in
general unless space is the only character that splits. In order to
preserve this simple abstraction, we would need to rewrite split and
assemble. We could also change the string in-place.
Ossia: To split a string into words:
Split the string on space. Use clojure.string/split.
.............
You can see that I am writing here not only about the current good-
enough implementation, but a \future implementation, should it ever be
needed. In fact, I discovered a bug in my code before I ever wrote
the code; but the question is what to do about the bug!
What I really want you to notice is how we are faced with a choice:
either we improve the library code (presumably by making split
optionally return an array of separators matched by the regex for
later reassembly by join, perhaps in metadata of some sort), or we go
with a less felicitous abstraction. There probably should be some
sort of formal analogue to this decision point in the natural language
text.
What does this mean for a Github? It may mean----and no doubt this
will be controversial----that once we have the capability of a
function search engine, we don't really worry about there being
canonical library code any more. If I face this particular problem,
maybe I decide to re-write Join and Split to my own specifications. I
then put them out there under my own namespace, and when someone needs
to use them they just type (use '[nchurch :only join split]) wherever
they are in their file, or whatever the exact incantation is (this is
one place I always need to go to the docs....). They have no trouble
browsing multiple alternatives, because they can see at a glance how
each one differs.
Now we have possibly multiple joins and splits; and so the community
no longer has a standard! Well, it would seem to me this fits well
with Github's always-fork philosophy. I do think there is only one
way to write a good library function, but it may be that there will
need to be a proliferation of alternatives----each making its own
argument and tradeoffs across various levels of abstraction, to return
to the theme of literate programming----before an optimal basis is
discovered. (There must have been a time before phillips head vs.
regular.....) And when that happens, perhaps it goes into
clojure.string (modulo legalities about contributor agreements----
something that might also be handled at the Github level, by the
way.).
Cedric Greevey
> What I really want you to notice is how we are faced with a choice:
> either we improve the library code (presumably by making split
> optionally return an array of separators matched by the regex for
> later reassembly by join, perhaps in metadata of some sort), or we go
> with a less felicitous abstraction. There probably should be some
> sort of formal analogue to this decision point in the natural language
> text.
Seems to me that the structure is this:
A string that's written language is some alternating form like
word separator word separator ...
where a "word" consists of consecutive alphanumeric characters and a
"separator" consists of consecutive nonalphanumeric characters. The
separators in this paragraph include " ", " \"", "\" ", ". ", and " \"
\", \" \\\"\", \"\\\" \", \". \", ", as well as one more that I can't
add here without generating one more still.
One tends sometimes to want to split a string in such a fashion,
transform the words only, and then reassemble.
This in turn suggests tools like:
(wordify string) -> a seq of strings starting with a separator and
alternating words and separators. The initial separator may be "".
Something like (map #(apply str %) (group-by alphanumeric? string))
with alphanumeric? testing a character, but if the first character of
the first string in the output seq is alphanumeric, a (cons "" %)
needs to be done on the output before returning it.
(map-words* f wordified-string) -> (interleave (take-nth 2
wordified-string) (map f (take-nth 2 (next wordified-string))))
(map-words f string) -> (apply str (map-words* f (wordify string)))
or similar.
daly
Github does not publish its source code. Other things like gitorious
are available but the setup is a challenge.
Tim Daly
daly
>might guess); but I haven't demonstrated myself to be a great coder,
>yet. I feel like I'm a coder who needs something like literate
>programming to be great, so it's kind of a chicken-and-egg problem.
>I'm already partway there with the existence of Clojure, but although
>it's the most intelligent language I've every come across (and it is
>at least Lisp), it still isn't enough.
Do yourself a favor and lose the "great coder" meme. Or get a job
at google and remain blissfully unaware.
One of the best books I've ever read about programming is called
"Practicing: A Musician's Return to Music" where the author talks
about his development as a musician. He would receive compliments
on how great he was at playing the guitar. At one point he replies
"How would you know?". The better he got, the worse he knew he was.
Your opinion of how great you are at programming will follow a
bell curve. You'll start off coming out of college thinking you're
ok, memorize a few algorithms and order theory ("the google disease")
and think you're "great" ("google only hires great coders"). But as
you learn more you'll discover that you have SO much more to learn
and as you work on larger projects you'll discover the musician's
insight. People would rate you "great" but you'll be able to say
"How would you know?". At which point, the better you get, the worse
you'll know you are.
Anybody who rates themselves as "great" is probably on the uphill
side of the learning curve.
The fact that you're trying to learn Clojure, moving into areas
that are beyond your comfort zone, and trying to learn literate
programming to improve your game, all points to the fact that you
will likely reach a point where you feel that being labeled "great"
is a sign that the speaker is clueless. Give it 10000 hours.
On Thu, 2011-12-22 at 20:59 -0800, nchurch wrote:
> You can see that I am writing here not only about the current good-
> enough implementation, but a \future implementation, should it ever be
> needed. In fact, I discovered a bug in my code before I ever wrote
> the code; but the question is what to do about the bug!
You found it. This is the magic hidden in literate programming.
Writing the natural language explanation, either before or during
coding exposes bugs before they get into the code. It exposes
hidden assumptions that will break boundary cases.
It is hard to demonstrate this effect. You have to try to write a
literate program and wait for the aha! moment to occur.
Lisps in general, and Clojure in particular, have the same aha!
moment where you transition from "how could anyone code like this?"
to "how could anyone code any other way?".
Pick some useful task, like a code-walker that will list all of
the called functions in a Clojure s-expression, and try to write
a literate version. Odds are good that you will learn more than
you ever wanted to know about Clojure. And, oh-by-the-way,
everyone else can read your literate code and learn Clojure too.
How sweet it that?
Tim Daly
Adam Getchell
On Tue, 2011-12-20 at 00:26 -0800, Adam Getchell wrote:I'm not sure what this sentence means. The .tex files ARE the
> BTW I looked at marginalia+leiningen, which seems to work for my
> purposes except I'm unclear where to put the .tex files.
literate program. By analogy, you seem to be asking something like
"I'm writing a book but I don't know where to put the text".
> The combination of literate + TDD seems forbidding.Are you finding it hard to explain why you wrote a test?
Jay Edwards
You can see how they handle it (from a Ruby-centric perspective) here: https://github.com/brighterplanet/numbers/blob/gh-pages/_posts/2010-12-02-github-pages-rocco-and-rake-file-tasks.markdown
The same approach should work fine for a Clojure project.
J.