Contrib directory structure

[Sean Devlin]

Hello everyone,

People have been putting a lot of good ideas into contrib. A lot of
people. I was wondering if the library could use a little re-
organization. I'd like to run a few ideas past everyone. Here are my
current thoughts.

1. Create a test directory, and all tests live in test/clojure/contrib

2. Each test suite file is named my_ns_test.clj

3. Each test suite is in the clojure.contrib.my-ns-test namespace.

4. Create a docs directory for manually written *usage* documentation.

5. Each namespace has a my-ns.markdown file in docs. Just about every
function gets asked "How does this work?" at least once on the mailing
list. Simply copying this down into the docs dir will make thing more
accessible to new users. This hopefully will cut down redundant
chatter, and give Mr. Halloway a head start on his next book :)

6. Modify the autodocs script to prepend the manually written docs to
the output. Here's roughly what the new autodocs routine would look
like

(str (slurp "docs/my-ns.markdown")
"\n-----\nReference\n"
(toms-autodoc-routine ...))

As an example, let's add the awesome.clj library to contrib. The
additions would be as follows

src/clojure/contrib/awesome.clj
test/clojure/contrib/awesome_test.clj
docs/awesome.markdown

Would anyone else find this type of structure useful, or does the
current solution work?

[Richard Newman]

> Would anyone else find this type of structure useful, or does the
> current solution work?

I think that would certainly help to make contrib more scalable and
navigable going forward...

[Howard Lewis Ship]

On Thu, Aug 6, 2009 at 10:02 AM, Sean Devlin<francoi...@gmail.com> wrote:
>
> Hello everyone,
>
> People have been putting a lot of good ideas into contrib.  A lot of
> people.  I was wondering if the library could use a little re-
> organization.  I'd like to run a few ideas past everyone.  Here are my
> current thoughts.
>
> 1. Create a test directory, and all tests live in test/clojure/contrib
>

+1

Although the :test keyword implies co-locating tests with the code
being tested, I'm still happier seperating out the test code from the
production code (and only shipping the production code).

> 2. Each test suite file is named my_ns_test.clj

_1

>
> 3. Each test suite is in the clojure.contrib.my-ns-test namespace.
>
> 4. Create a docs directory for manually written *usage* documentation.
>
> 5. Each namespace has a my-ns.markdown file in docs.  Just about every
> function gets asked "How does this work?" at least once on the mailing
> list.  Simply copying this down into the docs dir will make thing more
> accessible to new users.  This hopefully will cut down redundant
> chatter, and give Mr. Halloway a head start on his next book :)
>
> 6. Modify the autodocs script to prepend the manually written docs to
> the output.  Here's roughly what the new autodocs routine would look
> like
>
> (str (slurp "docs/my-ns.markdown")
>      "\n-----\nReference\n"
>      (toms-autodoc-routine ...))
>
> As an example, let's add the awesome.clj library to contrib.  The
> additions would be as follows
>
> src/clojure/contrib/awesome.clj
> test/clojure/contrib/awesome_test.clj
> docs/awesome.markdown
>
> Would anyone else find this type of structure useful, or does the
> current solution work?
> >
>

--
Howard M. Lewis Ship

Creator of Apache Tapestry
Director of Open Source Technology at Formos

[Sean Devlin]

Howard,
I am confused with the _1 in response to my second point. Is that
supposed to be +1 or -1?

Thanks,
Sean

PS - If it's -1, I'd like to know why. What in your experience makes
this a move in the wrong direction?

On Aug 6, 3:43 pm, Howard Lewis Ship <hls...@gmail.com> wrote:

[Lauri Pesonen]

Sean,

2009/8/7 Sean Devlin <francoi...@gmail.com>:

>
> Howard,
> I am confused with the _1 in response to my second point.  Is that
> supposed to be +1 or -1?

I think it's meant to be '+1' (he hit '-' instead of '=' which
produces '_' when shifted).

And +1 for the test code restructure from me, too.

> Thanks,
> Sean

--
! Lauri

[Richard Newman]

> I think it's meant to be '+1' (he hit '-' instead of '=' which
> produces '_' when shifted).

I thought it was an amusing pun on
how_package_names_translate_to_filenames. :)

[Tom Faulhaber]

Sean,

I generally like these proposals, but I want to be careful about being too limiting in a few cases:

1. Create a test directory, and all tests live in test/clojure/contrib

There already is a test directory (src/clojure/contrib/test_contrib), so the actual proposal is to move it around to be more sensible. Sounds OK to me.
 

2. Each test suite file is named my_ns_test.clj

Not a bad default, but there will be not always be a one-to-one relationship between namespaces and  tests. pprint already has two test files and a helper files with more to come. We may also have test files that perform tests across more than one namespace.

So, I would add that in addition to my_ns_test.clj, you could have directories of the form my_ns_test/ and files that aren't named after any ns that do cross system testing.

3. Each test suite is in the clojure.contrib.my-ns-test namespace.

Sounds good, where it makes sense. (Which will be most of the time.)
 

4. Create a docs directory for manually written *usage* documentation.

5. Each namespace has a my-ns.markdown file in docs.  Just about every
function gets asked "How does this work?" at least once on the mailing
list.  Simply copying this down into the docs dir will make thing more
accessible to new users.  This hopefully will cut down redundant
chatter, and give Mr. Halloway a head start on his next book :)

6. Modify the autodocs script to prepend the manually written docs to
the output.  Here's roughly what the new autodocs routine would look
like

(str (slurp "docs/my-ns.markdown")
     "\n-----\nReference\n"
     (toms-autodoc-routine ...))

Overall, this sounds like a great idea. I've been preparing to add this kind of info to the autodoc world (as we had some for datalog and pprint/cl-format in the old version). I was wondering how to store it and, now that you suggest it, I like the idea of having a specific doc directory in clojure-contrib (the name doc/ seems more traditional than docs/, but I can't get to worked up about that).

I think that basic usage info should go on the namespace metadata (autodoc already supports this), where it would in theory be available to other tools operating on clojure as well.  (I say in theory because of clojure issue #130.)

I'm not sure that the resulting documentation should go *into* the API reference page as you suggest, but perhaps be referenced from the page. Those pages are laid out as reference pages and I don't want folks to feel like they have to restrict themselves to a small form document. (Also, the formatting is somewhat more complex than you suggest since I'm working in html and not markdown, but that's not an issue.)

There's no reason that we shouldn't allow sets of web pages for a single namespace (for example, the documentation for how to build custom dispatch for the pretty printer will probably end up as it's own page).

But those are all nits...

Finally, we should be clear that (for the moment at least), tests and long form doc are *optional*, but if they're included, they should follow this convention. For example, in duck-streams the namespace doc is probably plenty of usage doc.

Well, that's enough nattering on for now,

Tom

[Sean Devlin]

Tom,

1. Excellent point about the one-to-one relationship. I think you
pprint lib is an excellent example. So, if understand what you're
saying correctly, the pprint test dir would look like

test/clojure/contrib/pprint/ns_1_test.clj
test/clojure/contrib/pprint/ns_2_test.clj
etc...

and the namespaces would be

clojure.contrib.pprint.ns-1-test
clojure.contrib.pprint.ns-2-test
etc...

2. If you doc is a more idiomatic directory name than docs, go for
it.

3. Storing usage docs in the ns docstring seems like a good idea,
especially to start. As some namespaces get more involved (think duck-
streams, str-utils-2, seq-utils), the documentation will grow too.
These would be good candidates to have their own entry in the doc
directory. For now, I think this is best handled on a case by case
basis.

4. I personally like markdown because github will automatically
convert it to html for me. Also, I hate worrying about closing tags,
but I can get over it. As an example, check out the docs directory on
my personal clojure utils: http://github.com/francoisdevlin/devlinsf-clojure-utils/tree/master

5. I think there should be a stronger emphasis on testing, so that
design intent is captured. I agree that keeping contrib as a
playground is important for now. However, I would ask everyone here
to please include tests with you submission, even if it's only a few
cases captured from the REPL. Otherwise, you end up with ambiguities
like this http://groups.google.com/group/clojure/browse_thread/thread/8d85d6a57b294611
.

[Tom Faulhaber]

On Mon, Aug 10, 2009 at 7:32 AM, Sean Devlin <francoi...@gmail.com> wrote:

Tom,

1.  Excellent point about the one-to-one relationship.  I think you
pprint lib is an excellent example.  So, if understand what you're
saying correctly, the pprint test dir would look like

test/clojure/contrib/pprint/ns_1_test.clj
test/clojure/contrib/pprint/ns_2_test.clj
etc...

and the namespaces would be

clojure.contrib.pprint.ns-1-test
clojure.contrib.pprint.ns-2-test
etc...

Yeah, although I think that once you're in the pprint directory, you won't be restricted to any particular form of the name (except perhaps for the trailing  -test) so that the test files/namespaces can be meaningful. For example, a set of tests of custom dispatch might be in the file test/clojure/contrib/dispatch_test.clj and the namespace clojure.contrib.pprint.dispatch-test.
 

3.  Storing usage docs in the ns docstring seems like a good idea,
especially to start.  As some namespaces get more involved (think duck-
streams, str-utils-2, seq-utils), the documentation will grow too.
These would be good candidates to have their own entry in the doc
directory.  For now, I think this is best handled on a case by case
basis.

Hopefully, we have pretty good docs in the namespace doc strings already. In most of the examples you cite, the functions in the namespace are pretty independent of each other and usage doc probably belongs on the function and not the namespace.

None of which prevents the author (or others) from writing a long form discussion of the namespace and how to use it. I just don't want to put that in-line on the reference page, but rather have it be linked. (You'll see what I mean when I do it and we can discuss more - both pprint and datalog have long form usage docs that I will port into the new environment.)
 

4.  I personally like markdown because github will automatically
convert it to html for me.  Also, I hate worrying about closing tags,
but I can get over it.  As an example, check out the docs directory on
my personal clojure utils:  http://github.com/francoisdevlin/devlinsf-clojure-utils/tree/master

I love markdown, too, and I will support it (and textile as well) for the source of documents. But when I push the  documents to the github pages I'll compile it to html. There are three reasons for this:

1) Github doesn't support raw html in the markdown source (last I looked) and the standard markdown utility does, which gives the author a lot more flexibility.
2) I want to have a way that your markdown can be wrapped in standard layout template so they look like the other pages on the autodoc (with TOC links, header, footer, etc.).
3) I want to be have the autodoc that's checked in be a doc tree that can be downloaded and used directly as a local HTML tree to support offline use and doc for historical versions.
 

5.  I think there should be a stronger emphasis on testing, so that
design intent is captured.  I agree that keeping contrib as a
playground is important for now.  However, I would ask everyone here
to please include tests with you submission, even if it's only a few
cases captured from the REPL.  Otherwise, you end up with ambiguities
like this http://groups.google.com/group/clojure/browse_thread/thread/8d85d6a57b294611
.

Personally, I agree with you. pprint/cl-format have about 250 tests and I have plans for a bunch more. But I don't feel like I'm in a position to dictate this.

Tom

[Sean Devlin]

On Aug 12, 12:29 pm, Tom Faulhaber <tomfaulha...@gmail.com> wrote:
> On Mon,

>
> Yeah, although I think that once you're in the pprint directory, you won't
> be restricted to any particular form of the name (except perhaps for the
> trailing  -test) so that the test files/namespaces can be meaningful. For
> example, a set of tests of custom dispatch might be in the file
> test/clojure/contrib/dispatch_test.clj and the namespace
> clojure.contrib.pprint.dispatch-test.
>

Tom, good point about freedom in the sub directory. If a library
needs a sub directory, the test writer should be free to design the
tests as he sees fit.

>
> Hopefully, we have pretty good docs in the namespace doc strings already. In
> most of the examples you cite, the functions in the namespace are pretty
> independent of each other and usage doc probably belongs on the function and
> not the namespace.
>
> None of which prevents the author (or others) from writing a long form
> discussion of the namespace and how to use it. I just don't want to put that
> in-line on the reference page, but rather have it be linked. (You'll see
> what I mean when I do it and we can discuss more - both pprint and datalog
> have long form usage docs that I will port into the new environment.)

Looking forward to your examples :)

>
> I love markdown, too, and I will support it (and textile as well) for the
> source of documents. But when I push the  documents to the github pages I'll
> compile it to html. There are three reasons for this:
>
> 1) Github doesn't support raw html in the markdown source (last I looked)
> and the standard markdown utility does, which gives the author a lot more
> flexibility.
> 2) I want to have a way that your markdown can be wrapped in standard layout
> template so they look like the other pages on the autodoc (with TOC links,
> header, footer, etc.).
> 3) I want to be have the autodoc that's checked in be a doc tree that can be
> downloaded and used directly as a local HTML tree to support offline use and
> doc for historical versions.

Great points. As long as your HTML renderer supports markdown, I'll
be happy.

>
>
>
> Personally, I agree with you. pprint/cl-format have about 250 tests and I
> have plans for a bunch more. But I don't feel like I'm in a position to
> dictate this.

True, we can and should only provide test cases and raise issues right
now. I know with a little more structure, I personally will be more
confident in providing test cases for other libs in contrib.

>
> Tom

Let us know when you're finished, I'd love to see what you've got.

Sean

[Tom Faulhaber]

I've added support for the doc/ directory for markdown only (so far).
I've added the pprint docs there and you can see them linked off the
pprint autodoc page (http://richhickey.github.com/clojure-contrib/
pprint-api.html).

If anyone wants to add doc to the doc/ directory in a different format
(textile, raw html) let me know and I'll add support.

I'll document this more completely pretty soon, but I just wanted to
let you guys know to encourage you to write doc.

Tom

[Sean Devlin]

Tom,
Excellent work getting this up. I really like what you did.

A quick question. Where does the markdown file go? Is it in

doc/a-namespace/my-file.markdown

or

doc/a-namespace.markdown

Thank you for taking the time to make this a reality.

Sean

[Tom Faulhaber]

Thanks.

Either way should work fine, You can even have both, if you want :-).

pprint has two different doc files (representing two of the three total files written so far about clojure-contrib), so I used the subdirectory approach. You can see that by pulling the latest contrib source and looking in doc/ (if you haven't already).

Enjoy,

Tom

[Howard Lewis Ship]

Sorry, it's +1. That's a good naming convention. Clumsy fingers.